Home

Awesome

micro-packed

Less painful binary encoding / decoding

Define complex binary structures using composable primitives. Comes with a friendly debugger.

Used in:

Usage

npm install micro-packed

import * as P from 'micro-packed';
const s = P.struct({
  field1: P.U32BE, // 32-bit unsigned big-endian integer
  field2: P.string(P.U8), // String with U8 length prefix
  field3: P.bytes(32), // 32 bytes
  field4: P.array(
    P.U16BE,
    P.struct({
      // Array of structs with U16BE length
      subField1: P.U64BE, // 64-bit unsigned big-endian integer
      subField2: P.string(10), // 10-byte string
    })
  ),
});

Table of contents:

Basics

There are 3 main interfaces:

Coder and BytesCoder use encode / decode methods

BytesCoderStream use encodeStream and decodeStream

Flexible size

Many primitives accept length / size / len as their argument. It represents their size. There are four different types of size:

Primitive types

P.bytes

Bytes CoderType with a specified length and endianness.

ParamDescription
lenLength CoderType, number, Uint8Array (for terminator), or null.
leWhether to use little-endian byte order.
// Dynamic size bytes (prefixed with P.U16BE number of bytes length)
const dynamicBytes = P.bytes(P.U16BE, false);
const fixedBytes = P.bytes(32, false); // Fixed size bytes
const unknownBytes = P.bytes(null, false); // Unknown size bytes, will parse until end of buffer
const zeroTerminatedBytes = P.bytes(new Uint8Array([0]), false); // Zero-terminated bytes

Following shortcuts are also available:

P.string

String CoderType with a specified length and endianness.

ParamDescription
lenCoderType, number, Uint8Array (terminator) or null
leWhether to use little-endian byte order.
const dynamicString = P.string(P.U16BE, false); // Dynamic size string (prefixed with P.U16BE number of string length)
const fixedString = P.string(10, false); // Fixed size string
const unknownString = P.string(null, false); // Unknown size string, will parse until end of buffer
const nullTerminatedString = P.cstring; // NUL-terminated string
const _cstring = P.string(new Uint8Array([0])); // Same thing

P.hex

Hexadecimal string CoderType with a specified length, endianness, and optional 0x prefix.

Returns: CoderType representing the hexadecimal string.

ParamDescription
lenLength CoderType (dynamic size), number (fixed size), Uint8Array (for terminator), or null (will parse until end of buffer)
isLEWhether to use little-endian byte order.
with0xWhether to include the 0x prefix.
const dynamicHex = P.hex(P.U16BE, { isLE: false, with0x: true }); // Hex string with 0x prefix and U16BE length
const fixedHex = P.hex(32, { isLE: false, with0x: false }); // Fixed-length 32-byte hex string without 0x prefix

P.constant

Creates a CoderType for a constant value. The function enforces this value during encoding, ensuring it matches the provided constant. During decoding, it always returns the constant value. The actual value is not written to or read from any byte stream; it's used only for validation.

Returns: CoderType representing the constant value.

ParamDescription
cConstant value.
// Always return 123 on decode, throws on encoding anything other than 123
const constantU8 = P.constant(123);

P.pointer

Pointer to a value using a pointer CoderType and an inner CoderType. Pointers are scoped, and the next pointer in the dereference chain is offset by the previous one. By default (if no 'allowMultipleReads' in ReaderOpts is set) is safe, since same region of memory cannot be read multiple times.

Returns: CoderType representing the pointer to the value.

ParamDescription
ptrCoderType for the pointer value.
innerCoderType for encoding/decoding the pointed value.
sizedWhether the pointer should have a fixed size.
const pointerToU8 = P.pointer(P.U16BE, P.U8); // Pointer to a single U8 value

Complex types

P.array

Array of items (inner type) with a specified length.

ParamDescription
lenLength CoderType (dynamic size), number (fixed size), Uint8Array (terminator), or null (will parse until end of buffer)
innerCoderType for encoding/decoding each array item.
const a1 = P.array(P.U16BE, child); // Dynamic size array (prefixed with P.U16BE number of array length)
const a2 = P.array(4, child); // Fixed size array
const a3 = P.array(null, child); // Unknown size array, will parse until end of buffer
const a4 = P.array(new Uint8Array([0]), child); // zero-terminated array (NOTE: terminator can be any buffer)

P.struct

Structure of composable primitives (C/Rust struct)

Returns: CoderType representing the structure.

ParamDescription
fieldsObject mapping field names to CoderTypes.
// Define a structure with a 32-bit big-endian unsigned integer, a string, and a nested structure
const myStruct = P.struct({
  id: P.U32BE,
  name: P.string(P.U8),
  nested: P.struct({
    flag: P.bool,
    value: P.I16LE,
  }),
});

P.tuple

Tuple (unnamed structure) of CoderTypes. Same as struct but with unnamed fields.

ParamDescription
fieldsArray of CoderTypes.
const myTuple = P.tuple([P.U8, P.U16LE, P.string(P.U8)]);

P.map

Mapping between encoded values and string representations.

Returns: CoderType representing the mapping.

ParamDescription
innerCoderType for encoded values.
variantsObject mapping string representations to encoded values.
// Map between numbers and strings
const numberMap = P.map(P.U8, {
  one: 1,
  two: 2,
  three: 3,
});

// Map between byte arrays and strings
const byteMap = P.map(P.bytes(2, false), {
  ab: Uint8Array.from([0x61, 0x62]),
  cd: Uint8Array.from([0x63, 0x64]),
});

P.tag

Tagged union of CoderTypes, where the tag value determines which CoderType to use. The decoded value will have the structure { TAG: number, data: ... }.

ParamDescription
tagCoderType for the tag value.
variantsObject mapping tag values to CoderTypes.
// Tagged union of array, string, and number
// Depending on the value of the first byte, it will be decoded as an array, string, or number.
const taggedUnion = P.tag(P.U8, {
  0x01: P.array(P.U16LE, P.U8),
  0x02: P.string(P.U8),
  0x03: P.U32BE,
});

const encoded = taggedUnion.encode({ TAG: 0x01, data: 'hello' }); // Encodes the string 'hello' with tag 0x01
const decoded = taggedUnion.decode(encoded); // Decodes the encoded value back to { TAG: 0x01, data: 'hello' }

P.mappedTag

Mapping between encoded values, string representations, and CoderTypes using a tag CoderType.

ParamDescription
tagCoderCoderType for the tag value.
variantsObject mapping string representations to [tag value, CoderType] pairs.
const cborValue: P.CoderType<CborValue> = P.mappedTag(P.bits(3), {
  uint: [0, cborUint], // An unsigned integer in the range 0..264-1 inclusive.
  negint: [1, cborNegint], // A negative integer in the range -264..-1 inclusive
  bytes: [2, P.lazy(() => cborLength(P.bytes, cborValue))], // A byte string.
  string: [3, P.lazy(() => cborLength(P.string, cborValue))], // A text string (utf8)
  array: [4, cborArrLength(P.lazy(() => cborValue))], // An array of data items
  map: [5, P.lazy(() => cborArrLength(P.tuple([cborValue, cborValue])))], // A map of pairs of data items
  tag: [6, P.tuple([cborUint, P.lazy(() => cborValue)] as const)], // A tagged data item ("tag") whose tag number
  simple: [7, cborSimple], // Floating-point numbers and simple values, as well as the "break" stop code
});

Padding, prefix, magic

P.padLeft

Pads a CoderType with a specified block size and padding function on the left side.

Returns: CoderType representing the padded value.

ParamDescription
blockSizeBlock size for padding (positive safe integer).
innerInner CoderType to pad.
padFnPadding function to use. If not provided, zero padding is used.
// Pad a U32BE with a block size of 4 and zero padding
const paddedU32BE = P.padLeft(4, P.U32BE);

// Pad a string with a block size of 16 and custom padding
const paddedString = P.padLeft(16, P.string(P.U8), (i) => i + 1);

P.padRight

Pads a CoderType with a specified block size and padding function on the right side.

Returns: CoderType representing the padded value.

ParamDescription
blockSizeBlock size for padding (positive safe integer).
innerInner CoderType to pad.
padFnPadding function to use. If not provided, zero padding is used.
// Pad a U16BE with a block size of 2 and zero padding
const paddedU16BE = P.padRight(2, P.U16BE);

// Pad a bytes with a block size of 8 and custom padding
const paddedBytes = P.padRight(8, P.bytes(null), (i) => i + 1);

P.ZeroPad

Shortcut to zero-bytes padding

P.prefix

Prefix-encoded value using a length prefix and an inner CoderType.

Returns: CoderType representing the prefix-encoded value.

ParamDescription
lenLength CoderType, number, Uint8Array (for terminator), or null.
innerCoderType for the actual value to be prefix-encoded.
const dynamicPrefix = P.prefix(P.U16BE, P.bytes(null)); // Dynamic size prefix (prefixed with P.U16BE number of bytes length)
const fixedPrefix = P.prefix(10, P.bytes(null)); // Fixed size prefix (always 10 bytes)

P.magic

Magic value CoderType that encodes/decodes a constant value. This can be used to check for a specific magic value or sequence of bytes at the beginning of a data structure.

Returns: CoderType representing the magic value.

ParamDescription
innerInner CoderType for the value.
constantConstant value.
checkWhether to check the decoded value against the constant.
// Always encodes constant as bytes using inner CoderType, throws if encoded value is not present
const magicU8 = P.magic(P.U8, 0x42);

P.magicBytes

Magic bytes CoderType that encodes/decodes a constant byte array or string.

Returns: CoderType representing the magic bytes.

ParamDescription
constantConstant byte array or string.
// Always encodes undefined into byte representation of string 'MAGIC'
const magicBytes = P.magicBytes('MAGIC');

Flags

P.flag

Flag CoderType that encodes/decodes a boolean value based on the presence of a marker.

Returns: CoderType representing the flag value.

ParamDescription
flagValueMarker value.
xorWhether to invert the flag behavior.
const flag = P.flag(new Uint8Array([0x01, 0x02])); // Encodes true as u8a([0x01, 0x02]), false as u8a([])
const flagXor = P.flag(new Uint8Array([0x01, 0x02]), true); // Encodes true as u8a([]), false as u8a([0x01, 0x02])
// Conditional encoding with flagged
const s = P.struct({ f: P.flag(new Uint8Array([0x0, 0x1])), f2: P.flagged('f', P.U32BE) });

P.flagged

Conditional CoderType that encodes/decodes a value only if a flag is present.

Returns: CoderType representing the conditional value.

ParamDescription
pathPath to the flag value or a CoderType for the flag.
innerInner CoderType for the value.
defOptional default value to use if the flag is not present.

P.optional

Optional CoderType that encodes/decodes a value based on a flag.

Returns: CoderType representing the optional value.

ParamDescription
flagCoderType for the flag value.
innerInner CoderType for the value.
defOptional default value to use if the flag is not present.
// Will decode into P.U32BE only if flag present
const optional = P.optional(P.flag(new Uint8Array([0x0, 0x1])), P.U32BE);
// If no flag present, will decode into default value
const optionalWithDefault = P.optional(P.flag(new Uint8Array([0x0, 0x1])), P.U32BE, 123);
const s = P.struct({
  f: P.flag(new Uint8Array([0x0, 0x1])),
  f2: P.flagged('f', P.U32BE),
});
const s2 = P.struct({
  f: P.flag(new Uint8Array([0x0, 0x1])),
  f2: P.flagged('f', P.U32BE, 123),
});

Wrappers

P.apply

Applies a base coder to a CoderType.

Returns: CoderType representing the transformed value.

ParamDescription
innerThe inner CoderType.
bThe base coder to apply.
import { hex } from '@scure/base';
const hex = P.apply(P.bytes(32), hex); // will decode bytes into a hex string

P.wrap

Wraps a stream encoder into a generic encoder and optionally validation function

ParamDescription
innerBytesCoderStream & { validate?: Validate<T> }.
const U8 = P.wrap({
  encodeStream: (w: Writer, value: number) => w.byte(value),
  decodeStream: (r: Reader): number => r.byte()
});

const checkedU8 = P.wrap({
  encodeStream: (w: Writer, value: number) => w.byte(value),
  decodeStream: (r: Reader): number => r.byte()
  validate: (n: number) => {
   if (n > 10) throw new Error(`${n} > 10`);
   return n;
  }
});

P.lazy

Lazy CoderType that is evaluated at runtime.

Returns: CoderType representing the lazy value.

ParamDescription
fnA function that returns the CoderType.
type Tree = { name: string; children: Tree[] };
const tree = P.struct({
  name: P.cstring,
  children: P.array(
    P.U16BE,
    P.lazy((): P.CoderType<Tree> => tree)
  ),
});

Bit fiddling

Bit fiddling is implementing using primitive called Bitset: a small structure to store position of ranges that have been read. Can be more efficient when internal trees are utilized at the cost of complexity. Needs O(N/8) memory for parsing. Purpose: if there are pointers in parsed structure, they can cause read of two distinct ranges: [0-32, 64-128], which means 'pos' is not enough to handle them

P.bits

CoderType for parsing individual bits. NOTE: Structure should parse whole amount of bytes before it can start parsing byte-level elements.

Returns: CoderType representing the parsed bits.

ParamDescription
lenNumber of bits to parse.
const s = P.struct({ magic: P.bits(1), version: P.bits(1), tag: P.bits(4), len: P.bits(2) });

P.bitset

Bitset of boolean values with optional padding.

Returns: CoderType representing the bitset.

ParamDescription
namesAn array of string names for the bitset values.
padWhether to pad the bitset to a multiple of 8 bits.
const myBitset = P.bitset(['flag1', 'flag2', 'flag3', 'flag4'], true);

utils

P.validate

Validates a value before encoding and after decoding using a provided function.

Returns: CoderType which check value with validation function.

ParamDescription
innerThe inner CoderType.
fnThe validation function.
const val = (n: number) => {
  if (n > 10) throw new Error(`${n} > 10`);
  return n;
};

const RangedInt = P.validate(P.U32LE, val); // Will check if value is <= 10 during encoding and decoding

coders.dict

Base coder for working with dictionaries (records, objects, key-value map) Dictionary is dynamic type like: [key: string, value: any][]

Returns: base coder that encodes/decodes between arrays of key-value tuples and dictionaries.

const dict: P.CoderType<Record<string, number>> = P.apply(
 P.array(P.U16BE, P.tuple([P.cstring, P.U32LE] as const)),
 P.coders.dict()
);

coders.decimal

Base coder for working with decimal numbers.

Returns: base coder that encodes/decodes between bigints and decimal strings.

ParamDefaultDescription
precisionNumber of decimal places.
round<code>false</code>Round fraction part if bigger than precision (throws error by default)
const decimal8 = P.coders.decimal(8);
decimal8.encode(630880845n); // '6.30880845'
decimal8.decode('6.30880845'); // 630880845n

coders.match

Combines multiple coders into a single coder, allowing conditional encoding/decoding based on input. Acts as a parser combinator, splitting complex conditional coders into smaller parts.

encode = [Ae, Be]; decode = [Ad, Bd] -> match([{encode: Ae, decode: Ad}, {encode: Be; decode: Bd}])

Returns: Combined coder for conditional encoding/decoding.

ParamDescription
lstArray of coders to match.

coders.reverse

Reverses direction of coder

Debugger

There is a second optional module for debugging into console.

import * as P from 'micro-packed';
import * as PD from 'micro-packed/debugger';

const debugInt = PD.debug(P.U32LE); // Will print info to console
// PD.decode(<coder>, data);
// PD.diff(<coder>, actual, expected);

Decode

Diff

License

MIT (c) Paul Miller (https://paulmillr.com), see LICENSE file.