Awesome
level-transcoder
Encode data with built-in or custom encodings. The successor to level-codec
that transcodes encodings from and to internal data formats supported by a database. This allows a database to store data in a format of its choice (Buffer, Uint8Array or String) with zero-effort support of known encodings. That includes other encoding interfaces in the ecosystem like abstract-encoding
and multiformats
.
Usage
Create a transcoder, passing a desired format:
const { Transcoder } = require('level-transcoder')
const transcoder1 = new Transcoder(['view'])
const transcoder2 = new Transcoder(['buffer'])
const transcoder3 = new Transcoder(['utf8'])
Then select an encoding and encode some data:
// Uint8Array(3) [ 49, 50, 51 ]
console.log(transcoder1.encoding('json').encode(123))
// <Buffer 31 32 33>
console.log(transcoder2.encoding('json').encode(123))
// '123'
console.log(transcoder3.encoding('json').encode(123))
If the Transcoder
constructor is given multiple formats then Transcoder#encoding()
selects an encoding with the best fitting format. Consider a database like leveldown
which has the ability to return data as a Buffer or string. If an encoding.decode(data)
function needs a string, we'll want to fetch that data
from the database as a string. This avoids the cost of having to convert a Buffer to a string. So we'd use the following transcoder:
const transcoder = new Transcoder(['buffer', 'utf8'])
Then, knowing for example that the return value of JSON.stringify(data)
is a UTF-8 string which matches one of the given formats, the 'json'
encoding will return a string here:
// '123'
console.log(transcoder.encoding('json').encode(123))
In contrast, data encoded as a 'view'
(for now that just means Uint8Array) would get transcoded into the 'buffer'
encoding. Copying of data is avoided where possible, like how the underlying ArrayBuffer of a view can be passed to Buffer.from(..)
without a copy.
Lastly, encodings returned by Transcoder#encoding()
have a format
property to be used to forward information to an underlying store. For example: an input value of { x: 3 }
using the 'json'
encoding which has a format
of 'utf8'
, can be forwarded as value '{"x":3}'
with encoding 'utf8'
. Vice versa for output.
Encodings
Built-in Encodings
These encodings can be used out of the box and are to be selected by name.
In this table, the input is what encode()
accepts. The format is what encode()
returns as well as what decode()
accepts. The output is what decode()
returns. The TypeScript typings of level-transcoder
have generic type parameters with matching names: TIn
, TFormat
and TOut
.
Name | Input | Format | Output |
---|---|---|---|
'buffer' <sup>1</sup> | Buffer, Uint8Array, String | 'buffer' | Buffer |
'view' | Uint8Array, Buffer, String | 'view' | Uint8Array |
'utf8' | String, Buffer, Uint8Array | 'utf8' | String |
'json' | Any JSON type | 'utf8' | As input |
'hex' | String (hex), Buffer | 'buffer' | String (hex) |
'base64' | String (base64), Buffer | 'buffer' | String (base64) |
<sup>1</sup> Aliased as 'binary'
. Use of this alias does not affect the ability to transcode.
Transcoder Encodings
It's not necessary to use or reference the below encodings directly. They're listed here for implementation notes and to show how input and output is the same; it's the format that differs.
Custom encodings are transcoded in the same way and require no additional setup. For example: if a custom encoding has { name: 'example', format: 'utf8' }
then level-transcoder
will create transcoder encodings on demand with names 'example+buffer'
and 'example+view'
.
Name | Input | Format | Output |
---|---|---|---|
'buffer+view' | Buffer, Uint8Array, String | 'view' | Buffer |
'view+buffer' | Uint8Array, Buffer, String | 'buffer' | Uint8Array |
'utf8+view' | String, Buffer, Uint8Array | 'view' | String |
'utf8+buffer' | String, Buffer, Uint8Array | 'buffer' | String |
'json+view' | Any JSON type | 'view' | As input |
'json+buffer' | Any JSON type | 'buffer' | As input |
'hex+view' <sup>1</sup> | String (hex), Buffer | 'view' | String (hex) |
'base64+view' <sup>1</sup> | String (base64), Buffer | 'view' | String (base64) |
<sup>1</sup> Unlike other encodings that transcode to 'view'
, these depend on Buffer at the moment and thus don't work in browsers if a shim is not included by JavaScript bundlers like Webpack and Browserify.
Ecosystem Encodings
Various modules in the ecosystem, in and outside of Level, can be used with level-transcoder
although they follow different interfaces. Common between the interfaces is that they have encode()
and decode()
methods. The terms "codec" and "encoding" are used interchangeably in the ecosystem. Passing these encodings through Transcoder#encoding()
(which is done implicitly when used in an abstract-level
database) results in normalized encoding objects as described further below.
Module | Format | Interface | Named |
---|---|---|---|
protocol-buffers | buffer | level-codec | ❌ |
charwise | utf8 | level-codec | ✅ |
bytewise | buffer | level-codec | ✅ |
lexicographic-integer-encoding | buffer , utf8 | level-codec | ✅ |
abstract-encoding | buffer | abstract-encoding | ❌ |
multiformats | view | multiformats | ✅ |
Those marked as not named are modules that export or generate encodings that don't have a name
property (or type
as an alias). We call these anonymous encodings. They can only be used as objects and not by name. Passing an anonymous encoding through Transcoder#encoding()
does give it a name
property for compatibility, but the value of name
is not deterministic.
API
Transcoder
transcoder = new Transcoder(formats)
Create a new transcoder, providing the formats that are supported by a database (or other). The formats
argument must be an array containing one or more of 'buffer'
, 'view'
, 'utf8'
. The returned transcoder
instance is stateful, in that it contains a set of cached encoding objects.
encoding = transcoder.encoding(encoding)
Returns the given encoding
argument as a normalized encoding object that follows the level-transcoder
encoding interface. The encoding
argument may be:
- A string to select a known encoding by its name
- An object that follows one of the following interfaces:
level-transcoder
,level-codec
,abstract-encoding
,multiformats
- A previously normalized encoding, such that
encoding(x)
equalsencoding(encoding(x))
.
Results are cached. If the encoding
argument is an object and it has a name then subsequent calls can refer to that encoding by name.
Depending on the formats
provided to the Transcoder
constructor, this method may return a transcoder encoding that translates the desired encoding from / to a supported format. Its encode()
and decode()
methods will have respectively the same input and output types as a non-transcoded encoding, but its name
property will differ.
encodings = transcoder.encodings()
Get an array of encoding objects. This includes:
- Encodings for the
formats
that were passed to theTranscoder
constructor - Custom encodings that were passed to
transcoder.encoding()
- Transcoder encodings for either.
Encoding
data = encoding.encode(data)
Encode data.
data = encoding.decode(data)
Decode data.
encoding.name
Unique name. A string.
encoding.commonName
Common name, computed from name
. If this encoding is a transcoder encoding, name
will be for example 'json+view'
and commonName
will be just 'json'
. Else name
will equal commonName
.
encoding.format
Name of the (lower-level) encoding used by the return value of encode()
. One of 'buffer'
, 'view'
, 'utf8'
. If name
equals format
then the encoding can be assumed to be idempotent, such that encode(x)
equals encode(encode(x))
.
Encoding Interface
Custom encodings must follow the following interface:
interface IEncoding<TIn, TFormat, TOut> {
name: string
format: 'buffer' | 'view' | 'utf8'
encode: (data: TIn) => TFormat
decode: (data: TFormat) => TOut
}
Install
With npm do:
npm install level-transcoder
Contributing
Level/transcoder
is an OPEN Open Source Project. This means that:
Individuals making significant and valuable contributions are given commit-access to the project to contribute as they see fit. This project is more like an open wiki than a standard guarded open source project.
See the Contribution Guide for more details.
Donate
Support us with a monthly donation on Open Collective and help us continue our work.