Awesome
Shrine::Tus
Provides tools for integrating Shrine with tus-ruby-server.
Installation
gem "shrine-tus", "~> 2.0"
Usage
When the file is uploaded to the tus server, you'll probably want to attach it to a database record and move it to permanent storage. The storage that tus-ruby-server uses should be considered temporary, as some uploads might never be finished, and tus-ruby-server needs to clear expired uploads.
Shrine-tus provides three ways of moving the file uploaded into the tus server to permanent storage, which differ in performance and level of decoupling. But regardless of the approach you choose, the code for attaching the uploaded file works the same:
class VideoUploader < Shrine
# ...
end
class Movie < Sequel::Model
include VideoUploader::Attachment.new(:video)
end
file_data #=> {"id":"http://tus-server.org/68db42638388ae645ab747b36a837a79", "storage":"cache", "metadata":{...}}
Movie.create(video: file_data)
See shrine-tus-demo for an example application that uses shrine-tus.
Note that by default Shrine won't extract metadata from directly upload files, instead it will just copy metadata that was extracted on the client side. See this section for the rationale and instructions on how to opt in.
Approach A: Downloading through tus server
Conceptionally the simplest setup is to have Shrine download the uploaded file through your tus server app. This is also the most decoupled option, because your main app can remain completely oblivious to what storage the tus server app uses internally, or in which programming language is it written (e.g. you could swap it for tusd and everything should continue to work the same).
To use this approach, you need to assign Shrine::Storage::Tus as your
temporary storage:
require "shrine/storage/tus"
Shrine.storages = {
cache: Shrine::Storage::YourTemporaryStorage.new(...),
store: Shrine::Storage::YourPermanentStorage.new(...),
tus: Shrine::Storage::Tus.new
}
class VideoUploader < Shrine
storages[:cache] = storages[:tus] # set Shrine::Storage::Tus as temporary storage
end
Shrine::Storage::Tus is a subclass of Shrine::Storage::Url, which uses
Down for downloading. By default, the Down::Http backend is used, which is
implemented using HTTP.rb.
If you're experiencing a lot of network hiccups while downloading, you might
want to consider switching to the Down::Wget backend, as wget automatically
resumes the download in case of network hiccups.
Shrine::Storage::Tus.new(downloader: :wget)
Approach B: Downloading directly from storage
While the appoach A is decoupled from the tus server implementation, it might not be the most performant depending on the overhead between your main app and the tus server app, how well the web server that you use for the tus server app handles streaming downloads (whether it blocks the web worker, thus affecting the app's request throughput), and whether you have hard request timeout limits like on Heroku. Down will also temporarily cache downloaded content to disk while copying to the permanent storage, so that adds I/O and disk usage.
Instead of downloading through the tus server app, you can download directly
from the underlying storage that it uses. This requires that you have the tus
storage configured in your main app (which might already by the case if you're
running tus server in the same process as your main app), and you can pass that
storage to Shrine::Storage::Tus:
Shrine::Storage::Tus.new(tus_storage: Tus::Server.opts[:storage])
Approach C: Tus storage equals Shrine storage
Approach B internally utilizes the common interface of each tus storage
object for streaming the file uploaded to that storage, via #each. However,
certain Shrine storage classes have optimizations when copying/moving a file
between two storages of the same kind.
So if you want to use the same kind of permanent storage as your tus server
uses, you can reap those performance benefits. In order to do this, instead of
using Shrine::Storage::Tus as your temporary Shrine storage as we did in
approaches A and B, we will be using a regular Shrine storage which will match
the storage that the tus server uses. In other words, your Shrine storage and
your tus storage would reference the same files. So, if your tus server app is
configured with either of the following storages:
Tus::Server.opts[:storage] = Tus::Storage::Filesystem.new("data")
Tus::Server.opts[:storage] = Tus::Storage::Gridfs.new(client: mongo, prefix: "tus")
Tus::Server.opts[:storage] = Tus::Storage::S3.new(prefix: "tus", **s3_options)
Then Shrine should be configured with the corresponding temporary storage:
Shrine.storages[:cache] = Shrine::Storage::FileSystem.new("data")
Shrine.storages[:cache] = Shrine::Storage::Gridfs.new(client: mongo, prefix: "tus")
Shrine.storages[:cache] = Shrine::Storage::S3.new(prefix: "tus", **s3_options)
In approaches A and B we didn't need to change the file data received
from the client, because we were using a subclass of Shrine::Storage::Url,
which accepts the id field as a URL. But with this approach the id field
will need to be translated from the tus URL to the correct ID for your
temporary Shrine storage, using the tus plugin that ships with shrine-tus.
Shrine.storages = {
cache: Shrine::Storage::YourTemporaryStorage.new(...),
store: Shrine::Storage::YourPermanentStorage.new(...),
}
class VideoUploader < Shrine
plugin :tus
end
Note that it's not recommended to use the delete_promoted Shrine plugin
with this this approach, because depending on the tus storage implementation
it could cause HEAD requests to the tus server app to return a success for files
that were deleted by Shrine.
These are the performance advantages for each of the official storages:
Filesystem
Shrine::Storage::FileSystem will have roughly the same performance as in
option B, though it will allocate less memory. However, if you load the
moving plugin, Shrine will execute a mv command between the tus storage
and permanent storage, which executes instantaneously regardless of the
filesize.
Shrine.plugin :moving
Mongo GridFS
Shrine::Storage::Gridfs will use more efficient copying, resulting in up to
2x speedup according to my benchmarks.
AWS S3
Shrine::Storage::S3 will issue a single S3 COPY request for files smaller
than 100MB, while files 100MB or larger will be divided into multiple chunks
which will be copied individually and in parallel using S3's multipart API.
Contributing
$ bundle exec rake test