Awesome

vertx-rest-storage

Persistence for REST resources in the filesystem, Aws S3 storage or a redis database.

Stores resources in a hierarchical way according to their URI. It actually implements a generic CRUD REST service.

It uses usual mime mapping to determine content type, so you can also use it as a web server. Without extension, JSON is assumed.

The following methods are supported on leaves (documents):

• GET: Returns the content of the resource.
• PUT: Stores the request body in the resource.
• DELETE: Deletes the resource.

The following methods are supported on intermediate nodes (collections):

• GET: Returns the list of collection members. Serves JSON and HTML representations.
• POST (StorageExpand): Returns the expanded content of the sub resources of the (collection) resource. The depth is limited to 1 level. See description below
• DELETE: Delete the collection and all its members.

Runs either as a module or can be integrated into an existing application by instantiating the RestStorageHandler class directly.

Run it

1. clone the repository
2. Install and start Redis

• Debian/Ubuntu: apt-get install redis-server
• Fedora/RedHat/CentOS: yum install redis
• OS X: brew install redis
• Windows
• Other

3. run mvn install -Dmaven.test.skip=true
4. run the fatjar with `java -jar build/libs/rest-storage-x.x.x-all.jar
5. you get a rest-storage, that stores to the filesystem in the directory where you started it. If you want to use the rest-storage with redis, you have to pass the configuration over a json file with -conf conf.json

Features

GET

Invoking GET request on a leave (document) returns the content of the resource.

GET /storage/resources/resource_1

Invoking GET request on a collection returns a list of collection members.

GET /storage/resources/

Parameters

Examples

Given a collection of ten items (res1-res10) under the path /server/tests/offset/resources/

The returned json response look like this:

{
  "resources": [
    "res1",
    "res10",
    "res2",
    "res3",
    "res4"
  ]
}

DELETE

Invoking DELETE request on a leave (document) deletes the resource.

DELETE /storage/resources/resource_1

Invoking DELETE request on a collection deletes the collection and all its children.

DELETE /storage/resources/

Parameters

StorageExpand

The StorageExpand feature expands the hierarchical resources and returns them as a single concatenated json resource.

Having the following resources in the storage:

key: data:test:collection:resource1     value: {"myProp1": "myVal1"}
key: data:test:collection:resource2     value: {"myProp2": "myVal2"}
key: data:test:collection:resource3     value: {"myProp3": "myVal3"}

would lead to this result

{
    "collection" : {
        "resource1" : {
            "myProp1": "myVal1"
        },
        "resource2" : {
            "myProp2": "myVal2"
        },
        "resource3" : {
            "myProp3": "myVal3"
        }
    }
}

Usage

To use the StorageExpand feature you have to make a POST request to the desired collection to expand having the url parameter storageExpand=true. Also, you wil have to send the names of the sub resources in the body of the request. Using the example above, the request would look like this:

POST /yourStorageURL/collection?storageExpand=true with the body:

{
    "subResources" : ["resource1", "resource2", "resource3"]
}

The amount of sub resources that can be provided is defined in the configuration by the property maxStorageExpandSubresources.

To override this for a single request, add the following request header with an appropriate value:

x-max-expand-resources: 1500

Reject PUT requests on low memory (redis only)

The redis storage provides a feature to reject PUT requests when the memory gets low. The information about the used memory is provided by the redis INFO command.

Attention: The stats received by the INFO command depend on the redis version. The required stats are used_memory and total_system_memory. Without these stats, the feature is disabled!

Configuration

To enable the feature, set the rejectStorageWriteOnLowMemory property (ModuleConfiguration) to true. Additionally, the freeMemoryCheckIntervalMs property can be changed to modify the interval for current memory usage calculation.

Usage

To define the importance level of PUT requests, add the following header:

x-importance-level: 75

The value defines the percentage of used memory which is not allowed to be exceeded to accept the PUT request. In the example above, the Request will only be acepted when the currently used memory is lower than 75%. When the currently used memory is higher than 75%, the request will be rejected with a status code 507 Insufficient Storage.

The higher the x-importance-level value, the more important the request. When no x-importance-level header is provided, the request is handled with the highest importance.

Lock Mechanism

The lock mechanism allows you to lock a resource for a specified time. This way only the owner of the lock is able to write or delete the given resource. To lock a resource, you have to add the following headers to your PUT / DELETE request.

Warning: The lock will always be removed if you perform a DELETE on a collection containing a locked resource. There is no check for locks in collections.

Store data compressed

In order to optimize the memory usage when using the redis storage, it's possible to store resources compressed using the gzip compression algorithm.

To store a resource compressed, add the following header to the PUT request:

x-stored-compressed: true

When making a GET request to a compressed resource, the resource will be uncompressed before returning. No additional header is required!

Restrictions

The data compression feature is not compatible with all vertx-rest-storage features. The following listing contains the restrictions of this feature:

• Data compression is available in redis storage only
• Data compression cannot be used with merge=true url parameter concurrently. Such PUT requests will be rejected.
• Compressed resources cannot be used in storageExpand requests. storageExpand requests to a collection containing a compressed resource will be rejected.
• If a resource is already stored in a different compression state (state = not compressed, compressed) as the compression of sent resource, the stored resource will be overwritten in every case. Like this we prevent unexpected behaviour considering the etag mechanism.

Configuration

The following configuration values are available:

Configuration util

The configurations have to be passed as JsonObject to the module. For a simplified configuration the ModuleConfigurationBuilder can be used.

Example:

ModuleConfiguration config = with()
		.redisHost("anotherhost")
		.redisPort(1234)
		.editorConfig(new JsonObject().put("myKey", "myValue"))
		.build();

JsonObject json = config.asJsonObject();

Properties not overridden will not be changed. Thus remaining default.

To use default values only, the ModuleConfiguration constructor without parameters can be used:

JsonObject json  = new ModuleConfiguration().asJsonObject();

Storage types

Currently, there are three storage types supported. File system storage, S3 storage and redis storage.

File System Storage

The data is stored hierarchically on the file system. This is the default storage type when not overridden in the configuration.

S3 storage

The data is stored in a S3 instance.

AWS S3

See https://aws.amazon.com/s3 it is also possible to use a local instance using https://docs.localstack.cloud/user-guide/aws/s3/

docker run --rm -p 4566:4566 -v ./s3:/var/lib/localstack localstack/localstack:s3-latest

Redis Storage

The data is stored in a redis database. Caution: The redis storage implementation does not currently support streaming. Avoid transferring too big payloads since they will be entirely copied in memory.

Dependencies

• Starting from 2.6.x rest-storage requires Java 11.

• This module uses Vert.x v3.3.3 (or later), so Java 8 is required.