Awesome

Java Multicodec

By identityfoundry.com

About Multiformats

Multicodec is part of the Multiformats collection of protocols. Multiformats is a collection of protocols which aim to future-proof systems, today. They do this mainly by allowing data to be self-describable. See:

• https://github.com/multiformats/multiformats

About Multicodec

The Multicodec is an agreed way for encoding bytes with a prefix that specifies the type of encoding. The format is therefore a portable and self describing way of expressing an encoding of bytes that does not assume a specific context. See:

• https://github.com/multiformats/multicodec

Examples

For example, the multicodec specification is used in the specifications for Decentralised Identifiers (DIDs) in regard to the DID Method 'key' specification. Therefore, any work on did:key implementations needs to also solve for the use of multicodecs. See:

• https://w3c-ccg.github.io/did-method-key

Why A Java Implementation?

It is noted that the https://github.com/multiformats/multiformats lists technology implementations, however a java implementation is not available. This library addresses that gap to provide an implementation of the multicodec specification in java.

Dependencies

The code of a multicodec is encoded as unsigned varint as defined by multiformats/unsigned-varint. It is then used as a prefix to identify the data that follows. This represents some additional binary operations.

This project also solves for a java implementation of unsigned varints so may also be of use for implementors.

Sample Usage

//Demonstrate the encoding of data for an example ED25519 public key...
byte[] raw = getEd25519PublicKeyData();
byte[] multicodecEncoding = Multicodec.encode(Multicodec.Codec.ED25519_PUB, raw);

//Demonstrate the decoding...
DecodedData decodedData = null;
try {
    decodedData = MulticodecEncoder.decode(multicodecEncoding);
} catch (AmbiguousCodecEncodingException exAmbiguousCodecEncoding) {
    System.err.println("AmbiguousCodecEncodingException:" + exAmbiguousCodecEncoding.getMessage());
}
if (decodedData != null) {
    System.out.printf("Codec:%s(%s)%n", decodedData.getCodec().name(), decodedData.getCodec().code);
    System.out.println("byte data length:" + decodedData.getDataAsBytes().length);
    System.out.println("hex data:" + decodedData.getDataAsHex());
}

Known Problems

Clash of Codecs - Decoding

There are two basic classes of the codecs, single bytes codes (e.g cidv1(0x01), sha2_256(0x12)) and multibyte codecs (e.g. upd(0x0111), p256_pub(0x1200)).

The way that the codec codes are configured it is possible to encode to them, but there are situations where it is not possible to decode.

Example 1:

1. Consider the sample data as hex: 0xA1E9D3D8EC
2. When it is multicodec cidv1 encoded we have : 0xA1E9D3D8EC --(cidv1)--> 0x01A1E9D3D8EC
3. When it is multicodec upd encoded we have : 0xA1E9D3D8EC --(upd)----> 0x0111A1E9D3D8EC
4. Notice that both encoded strings for different encoding types start with 0x01.
5. Therefore, when comes to decoding the udp value 0x0111A1E9D3D8EC it is not possible to determine if the string is
   1. cidv1 encoded with data: 0x11A1E9D3D8EC
   2. upd encoded with data: 0xA1E9D3D8EC

Example 2:

Another example also applies to sha2_256 and p256_pub encoding.

1. When it is multicodec sha2_256 encoded we have : 0xA1E9D3D8EC --(sha2_256)--> 0x12A1E9D3D8EC
2. When it is multicodec p256_pub encoded we have : 0xA1E9D3D8EC --(p256_pub)--> 0x1200A1E9D3D8EC
3. Notice that both encoded strings for different encoding types start with 0x12.
4. Therefore, when comes to decoding the p256_pub value 0x1200A1E9D3D8EC it is not possible to determine if the string is
   1. sha2_256 encoded with data: 0x00A1E9D3D8EC
   2. p256_pub encoded with data: 0x1200A1E9D3D8EC

List of clashes

These clashes occur in a number of situations and are listed below.

Current Workarounds

The Multicodec.decode() method makes the assumption that the decoding only supports single byte codec codes, to prevent the "clash of codecs" issue from presenting itself. This means that while Multicodec.encode() method can encode for any codec code, the Multicodec.decode() does not support decoding of multibyte codec codes.

If Multicodec.decode() is used to decode a multibyte codec code then AmbiguousCodecEncodingException is thrown.

Possible Solutions

Solution A : Use of a reserved character delimiter

The bytes used in the codec prefix need to be unambiguous. Pick a used reserved byte (e.g. 0x00) and use as a delimiter between the codec code bytes and the data

Examples

Pros

• Codec prefixes are explicit and unambiguous
• Common simple rule applied across all codecs

Cons

• Breaking change for the in-use single byte codecs
• Relies on no codecs ending in the reserved byte (e.g. 0x00), which is an implicit rule that is too easily broken for new codec additions.
• Any codec that ends in the reserved byte (e.g. 0x00) must be reassigned to a value that does not. e.g. p256_pub(0x1200) must be reassigned.

Recommendation:

• Not recommended.
• This will break a number of existing implementations that assume single byte codec in order to optimise to a simple rule.
• The optimisation is in favour of the newer multibyte codecs that are in a draft over single byte codecs that are in permanent state.

Solution B : Careful selection of codecs that avoids clashes

The bytes used in the codec prefix need to be unambiguous. The multibyte codec values that are in draft need to be re-assigned so there is no possible clashes.

Pros

• Codec prefixes are explicit and unambiguous
• Non-breaking change for the in-use single byte codecs

Cons

• Any multibyte codec that clashes with a single byte codec must be reassigned to a value that does not clash. Large reassignment exercise of multibyte codecs
• Relies on an implicit subtle rule that is embedded into the codec selections. The rule will likely be broken for new codec additions that are unaware of the clashing issue.

Recommendation:

• Weakly viable, but not recommended.
• Will result in a significant refactoring and bulk reassignment process for a significant proportion of the multibyte codecs.

Solution C : Use of a reserved character delimiter, but only for multibyte codecs

The bytes used in the codec prefix need to be unambiguous.

1. Pick an unused reserved byte (rb) (e.g. 0x00)
2. Create rule that allows single byte codecs, so long as they do not start with the reserved byte (0x00).
3. If a multiple byte codec is adopted then it must start and end with the reserved byte (0x00).

Examples

Pros

• Codec prefixes are explicit and unambiguous.
• Non-breaking change for the in-use single byte codecs.
• Does not rely on clever implicit rules to stagger codec selections that will be easily broken.
• Does not need a large reassignment exercise of the multibyte codecs.
• The impact is limited only to multibyte codecs, which are mostly in draft status.

Cons

• Any existing multibyte codecs implementations must adopt the rule.

Recommendation:

• Recommended as most viable.
• Does not break most permanent implementations.
• Ring fences change to the more problematic multibyte codecs that are in draft.