Awesome

feathers-s3

feathers-s3 allows to deal with AWS S3 API compatible storages to manage file upload/download in a FeathersJS application.

Unlike the solution feathers-blob, which provides a store abstraction, feathers-s3 is limited to be used with stores providing a S3 compatible API. However, it takes advantage of the S3 API by using presigned URLs to manage (upload, share) objects on a store in a more reliable and secure way.

Using Presigned URL has different pros and cons:

• Pros
  • It decreases the necessary resources of the server because the transfer process is established between the client and the S3 service.
  • It speeds up the transfer process because it can be easily parallelized.
  • It reduces the risk of your server becoming a bottleneck.
  • It supports multipart upload by design.
  • It is inherently secure.
• Cons
  • It involves extra complexity on the client side.
  • It requires your S3 bucket to have CORS enabled.
  • It requires your provider to support S3 Signature Version 4.
  • The access to the object is limited to a short time.

To address these drawbacks, feathers-s3 provides:

• Helper functions to simplify usage from a client application.
• An Express middleware to directly access objects based on URLs without using presignedl url. There is no time constraint unlike with presigned url and you can also access only a portion of an object using range requests.
• A proxy mode that let you use service methods that don't rely on presigned URL in case your S3 provider doesn't support CORS settings or you'd like to process data on your backend. In this case the objects are always transferred through your backend.

Principle

The following sections illustrate the different process implemented by feathers-s3:

Upload

The upload process can be a singlepart upload or a multipart upload depending on the size of the object to be uploaded. If the size is greater than a chunkSize (by default 5MB), feathers-s3 performs a multipart upload. Otherwise it performs a singlepart upload.

Singlepart upload

Multipart upload

Download

Usage

Installation

npm install @kalisio/feathers-s3 --save

or

yarn add @kalisio/feathers-s3

Example

The provided example illustrates how to setup:

• a server app

https://github.com/kalisio/feathers-s3/blob/fa8adde8181ccad740ed7fcfb92af1a5c43f9281/example/app.mjs#L1-L44

• a browser client app

https://github.com/kalisio/feathers-s3/blob/184802600263e62b4d8d910d8781cbf3b7b03966/example/public/index.html#L1-L74

As a general rule you can also have a look at the tests in order to have different use cases of the library.

Data processing

Some use cases might require you directly process the data on your server before sending it to the object storage, e.g. if you'd like to resize an image. You can do that by:

1. registering a before hook on the putObject custom method to process the data before sending it to the object storage,
2. using the proxy mode on the client side service to send the data to your server instead of the object storage,
3. defining the appropriate chunkSize on the client service to not use multipart upload as processing usually requires the whole content to be sent.

Here is a simple example relying on sharp to resize an image:

async function resizeImage (hook) {
  hook.data.buffer = await sharp(hook.data.buffer)
    .resize(128, 48, { fit: 'contain', background: '#00000000' }).toBuffer()
}

app.service('s3').hooks({
  before: {
    putObject: [resizeImage]
  }
})

// Here you can proceed as usual from server side
service.putObject({ id, buffer, type })
// or client side
clientService.upload(id, blob, options)

API

feathers-s3 consists of three parts:

• Service that provides basic methods for using S3 API.
• Middlewares that provides an Express middleware to access an object from the store.
• Client that provides helper functions to simplify the upload and download logic for the client side.

Service

constructor (options)

Create an instance of the service with the given options:

find (params)

Lists some objects in a bucket according given criteria provided in the params.query object.

Check the ListObjectsCommandInput documentation to have the list of supported poperties.

create (data, params)

Generates a presigned URL for the following commands:

• PutObjectCommand
• GetObjectCommand
• UploadPartCommnad

The payload data must contain the following properties:

get (id, params)

Get an object content from a bucket.

[!NOTE] The object will be entirely read and transferred to the client, for large files consider using presigned URLs instead.

remove (id, params)

Remove an object fromt the bucket.

createMultipartUpload (data, params)

Initiate a multipart upload.

It wraps the CreateMultipartUploadCommand.

The payload data must contain the following properties:

Any optional properties are forwarded to the underlying CreateMultipartUploadCommand command parameters.

completeMultipartUpload (data, params)

Finalize a multipart upload.

It wraps the CompleteMultipartUploadCommand.

The payload data must contain the following properties:

Any optional properties are forwarded to the underlying CompleteMultipartUploadCommand command parameters.

uploadPart (data, params)

Upload a part to a bucket.

The payload data must contain the following properties:

putObject (data, params)

Upload an object to a bucket.

The payload data must contain the following properties:

getObjectCommand (data, params)

Execute the GetObjectCommand and returns the response.

[!NOTE] This method is not declared on the client side.

The payload data must contain the following properties:

uploadFile (data, params)

Convenient method to upload a file.

[!NOTE] This method is not declared on the client side.

The payload data must contain the following properties:

[!NOTE] You can also provide an id property to override the computed object key.

downloadFile (data, params)

Convenient method to download a file.

[!NOTE] This method is not declared on the client side.

The payload data must contain the following properties:

Middlewares

getObject (service)

It expect to be setup on a route path like path/to/get/* where the last parameter is the path to the object in the target bucket. It is associated with an S3 service to use the same configuration (s3client, bucket, etc...).

// How to setup the route
app.get('/s3-objects/*', getObject(service))
// How to use it with any request agent like superagent
const response = await superagent.get('your.domain.com/s3-objects/my-file.png')

You can also simply use the target object URL in your HTML: <img src="your.domain.com/s3-objects/my-file.png">.

If you'd like to authenticate the route on your backend you will have to do something like this:

import { authenticate } from '@feathersjs/express'

app.get('/s3-objects/*', authenticate('jwt'), getObject(service))

In this case, if you need to reference the object by the URL it will require you to add the JWT as a query parameter like this: <img src="your.domain.com/s3-objects/my-file.png?jwt=TOKEN">. The JWT can then be extracted by a middleware:

import { authenticate } from '@feathersjs/express'

app.get('/s3-objects/*', (req, res, next) => {
  req.feathers.authentication = {
    strategy: 'jwt',
    accessToken: req.query.jwt
  }
  next()
}, authenticate('jwt'), getObject(service))

Client

getClientService (app, options)

Return the client service interface. The client service exposes the custom methods defined in the Service and is also decorated with 2 helper functions that really simplify the logic when implementing a client application, notably for multipart upload.

The options are:

upload (id, blob, options)

Upload a Blob object to the bucket with the given key id.

According the size of chunk you set when instanciang the client service and the size of the blob, the method will automatically perform a singlepart upload or a mulitpart upload.

If the proxy option is undefined. The client performs the upload action directly using fetch. Otherwise, it uses the proxyUpload custom method.

download (key, type, options)

Download an object from the bucket with the given key id.

If the proxy option is undefined. The client performs the download action directly using fetch. Otherwise, it uses the getObject custom method.

License

Copyright (c) 2017-20xx Kalisio

Licensed under the MIT license.

Authors

This project is sponsored by