Home

Awesome

MemoryPack

NuGet GitHub Actions Releases

Zero encoding extreme performance binary serializer for C# and Unity.

image

Compared with System.Text.Json, protobuf-net, MessagePack for C#, Orleans.Serialization. Measured by .NET 7 / Ryzen 9 5950X machine. These serializers have IBufferWriter<byte> method, serialized using ArrayBufferWriter<byte> and reused to avoid measure buffer copy.

For standard objects, MemoryPack is x10 faster and x2 ~ x5 faster than other binary serializers. For struct array, MemoryPack is even more powerful, with speeds up to x50 ~ x200 greater than other serializers.

MemoryPack is my 4th serializer, previously I've created well known serializers, ZeroFormatter, Utf8Json, MessagePack for C#. The reason for MemoryPack's speed is due to its C#-specific, C#-optimized binary format and a well tuned implementation based on my past experience. It is also a completely new design utilizing .NET 7 and C# 11 and the Incremental Source Generator (.NET Standard 2.1 (.NET 5, 6) and there is also Unity support).

Other serializers perform many encoding operations such as VarInt encoding, tag, string, etc. MemoryPack format uses a zero-encoding design that copies as much C# memory as possible. Zero-encoding is similar to FlatBuffers, but it doesn't need a special type, MemoryPack's serialization target is POCO.

Other than performance, MemoryPack has these features.

Installation

This library is distributed via NuGet. For best performance, recommend to use .NET 7. Minimum requirement is .NET Standard 2.1.

PM> Install-Package MemoryPack

And also a code editor requires Roslyn 4.3.1 support, for example Visual Studio 2022 version 17.3, .NET SDK 6.0.401. For details, see the Roslyn Version Support document.

For Unity, the requirements and installation process are completely different. See the Unity section for details.

Quick Start

Define a struct or class to be serialized and annotate it with the [MemoryPackable] attribute and the partial keyword.

using MemoryPack;

[MemoryPackable]
public partial class Person
{
    public int Age { get; set; }
    public string Name { get; set; }
}

Serialization code is generated by the C# source generator feature which implements the IMemoryPackable<T> interface. In Visual Studio you can check a generated code by using a shortcut Ctrl+K, R on the class name and select *.MemoryPackFormatter.g.cs.

Call MemoryPackSerializer.Serialize<T>/Deserialize<T> to serialize/deserialize an object instance.

var v = new Person { Age = 40, Name = "John" };

var bin = MemoryPackSerializer.Serialize(v);
var val = MemoryPackSerializer.Deserialize<Person>(bin);

Serialize method supports a return type of byte[] as well as it can serialize to IBufferWriter<byte> or Stream. Deserialize method supports ReadOnlySpan<byte>, ReadOnlySequence<byte> and Stream. And there are alse non-generics versions.

Built-in supported types

These types can be serialized by default:

Define [MemoryPackable] class / struct / record / record struct

[MemoryPackable] can annotate to any class, struct, record, record struct and interface. If a type is struct or record struct which contains no reference types (C# Unmanaged types) any additional annotation (ignore, include, constructor, callbacks) is not used, that serialize/deserialize directly from the memory.

Otherwise, by default, [MemoryPackable] serializes public instance properties or fields. You can use [MemoryPackIgnore] to remove serialization target, [MemoryPackInclude] promotes a private member to serialization target.

[MemoryPackable]
public partial class Sample
{
    // these types are serialized by default
    public int PublicField;
    public readonly int PublicReadOnlyField;
    public int PublicProperty { get; set; }
    public int PrivateSetPublicProperty { get; private set; }
    public int ReadOnlyPublicProperty { get; }
    public int InitProperty { get; init; }
    public required int RequiredInitProperty { get; init; }

    // these types are not serialized by default
    int privateProperty { get; set; }
    int privateField;
    readonly int privateReadOnlyField;

    // use [MemoryPackIgnore] to remove target of a public member
    [MemoryPackIgnore]
    public int PublicProperty2 => PublicProperty + PublicField;

    // use [MemoryPackInclude] to promote a private member to serialization target
    [MemoryPackInclude]
    int privateField2;
    [MemoryPackInclude]
    int privateProperty2 { get; set; }
}

MemoryPack's code generator adds information about what members are serialized to the <remarks /> section. This can be viewed by hovering over the type with Intellisense.

image

All members must be memorypack-serializable, if not the code generator will emit an error.

image

MemoryPack has 35 diagnostics rules (MEMPACK001 to MEMPACK035) to be defined comfortably.

If target type is defined MemoryPack serialization externally and registered, use [MemoryPackAllowSerialize] to silent diagnostics.

[MemoryPackable]
public partial class Sample2
{
    [MemoryPackAllowSerialize]
    public NotSerializableType? NotSerializableProperty { get; set; }
}

Member order is important, MemoryPack does not serialize the member-name or other information, instead serializing fields in the order they are declared. If a type is inherited, serialization is performed in the order of parent → child. The order of members can not change for the deserialization. For the schema evolution, see the Version tolerant section.

The default order is sequential, but you can choose the explicit layout with [MemoryPackable(SerializeLayout.Explicit)] and [MemoryPackOrder()].

// serialize Prop0 -> Prop1
[MemoryPackable(SerializeLayout.Explicit)]
public partial class SampleExplicitOrder
{
    [MemoryPackOrder(1)]
    public int Prop1 { get; set; }
    [MemoryPackOrder(0)]
    public int Prop0 { get; set; }
}

Constructor selection

MemoryPack supports both parameterized and parameterless constructors. The selection of the constructor follows these rules. (Applies to classes and structs).

[MemoryPackable]
public partial class Person
{
    public readonly int Age;
    public readonly string Name;

    // You can use a parameterized constructor - parameter names must match corresponding members name (case-insensitive)
    public Person(int age, string name)
    {
        this.Age = age;
        this.Name = name;
    }
}

// also supports record primary constructor
[MemoryPackable]
public partial record Person2(int Age, string Name);

public partial class Person3
{
    public int Age { get; set; }
    public string Name { get; set; }

    public Person3()
    {
    }

    // If there are multiple constructors, then [MemoryPackConstructor] should be used
    [MemoryPackConstructor]
    public Person3(int age, string name)
    {
        this.Age = age;
        this.Name = name;
    }
}

Serialization callbacks

When serializing/deserializing, MemoryPack can invoke a before/after event using the [MemoryPackOnSerializing], [MemoryPackOnSerialized], [MemoryPackOnDeserializing], [MemoryPackOnDeserialized] attributes. It can annotate both static and instance (non-static) methods, and public and private methods.

[MemoryPackable]
public partial class MethodCallSample
{
    // method call order is static -> instance
    [MemoryPackOnSerializing]
    public static void OnSerializing1()
    {
        Console.WriteLine(nameof(OnSerializing1));
    }

    // also allows private method
    [MemoryPackOnSerializing]
    void OnSerializing2()
    {
        Console.WriteLine(nameof(OnSerializing2));
    }

    // serializing -> /* serialize */ -> serialized
    [MemoryPackOnSerialized]
    static void OnSerialized1()
    {
        Console.WriteLine(nameof(OnSerialized1));
    }

    [MemoryPackOnSerialized]
    public void OnSerialized2()
    {
        Console.WriteLine(nameof(OnSerialized2));
    }

    [MemoryPackOnDeserializing]
    public static void OnDeserializing1()
    {
        Console.WriteLine(nameof(OnDeserializing1));
    }

    // Note: instance method with MemoryPackOnDeserializing, that not called if instance is not passed by `ref`
    [MemoryPackOnDeserializing]
    public void OnDeserializing2()
    {
        Console.WriteLine(nameof(OnDeserializing2));
    }

    [MemoryPackOnDeserialized]
    public static void OnDeserialized1()
    {
        Console.WriteLine(nameof(OnDeserialized1));
    }

    [MemoryPackOnDeserialized]
    public void OnDeserialized2()
    {
        Console.WriteLine(nameof(OnDeserialized2));
    }
}

Callbacks allows parameterless method and ref reader/writer, ref T value method. For example, ref callbacks can write/read custom header before serialization process.

[MemoryPackable]
public partial class EmitIdData
{
    public int MyProperty { get; set; }

    [MemoryPackOnSerializing]
    static void WriteId<TBufferWriter>(ref MemoryPackWriter<TBufferWriter> writer, ref EmitIdData? value)
        where TBufferWriter : IBufferWriter<byte> // .NET Standard 2.1, use where TBufferWriter : class, IBufferWriter<byte>
    {
        writer.WriteUnmanaged(Guid.NewGuid()); // emit GUID in header.
    }

    [MemoryPackOnDeserializing]
    static void ReadId(ref MemoryPackReader reader, ref EmitIdData? value)
    {
        // read custom header before deserialize
        var guid = reader.ReadUnmanaged<Guid>();
        Console.WriteLine(guid);
    }
}

If set a value to ref value, you can change the value used for serialization/deserialization. For example, instantiate from ServiceProvider.

// before using this formatter, set ServiceProvider
// var options = MemoryPackSerializerOptions.Default with { ServiceProvider = provider };
// MemoryPackSerializer.Deserialize(value, options);

[MemoryPackable]
public partial class InstantiateFromServiceProvider
{
    static IServiceProvider serviceProvider = default!;

    public int MyProperty { get; private set; }

    [MemoryPackOnDeserializing]
    static void OnDeserializing(ref MemoryPackReader reader, ref InstantiateFromServiceProvider value)
    {
        if (value != null) return;
        value = reader.Options.ServiceProvider!.GetRequiredService<InstantiateFromServiceProvider>();
    }
}

Define custom collection

By default, annotated [MemoryPackObject] type try to serialize its members. However, if a type is a collection (ICollection<>, ISet<>, IDictionary<,>), use GenerateType.Collection to serialize it correctly.

[MemoryPackable(GenerateType.Collection)]
public partial class MyList<T> : List<T>
{
}

[MemoryPackable(GenerateType.Collection)]
public partial class MyStringDictionary<TValue> : Dictionary<string, TValue>
{

}

Static constructor

MemoryPackable class can not define static constructor because generated partial class uses it. Instead, you can define a static partial void StaticConstructor() to do the same thing.

[MemoryPackable]
public partial class CctorSample
{
    static partial void StaticConstructor()
    {
    }
}

Polymorphism (Union)

MemoryPack supports serializing interface and abstract class objects for polymorphism serialization. In MemoryPack this feature is called Union. Only interfaces and abstracts classes are allowed to be annotated with [MemoryPackUnion] attributes. Unique union tags are required.

// Annotate [MemoryPackable] and inheritance types with [MemoryPackUnion]
// Union also supports abstract class
[MemoryPackable]
[MemoryPackUnion(0, typeof(FooClass))]
[MemoryPackUnion(1, typeof(BarClass))]
public partial interface IUnionSample
{
}

[MemoryPackable]
public partial class FooClass : IUnionSample
{
    public int XYZ { get; set; }
}

[MemoryPackable]
public partial class BarClass : IUnionSample
{
    public string? OPQ { get; set; }
}
// ---

IUnionSample data = new FooClass() { XYZ = 999 };

// Serialize as interface type.
var bin = MemoryPackSerializer.Serialize(data);

// Deserialize as interface type.
var reData = MemoryPackSerializer.Deserialize<IUnionSample>(bin);

switch (reData)
{
    case FooClass x:
        Console.WriteLine(x.XYZ);
        break;
    case BarClass x:
        Console.WriteLine(x.OPQ);
        break;
    default:
        break;
}

tag allows 0 ~ 65535, it is especially efficient for less than 250.

If an interface and derived types are in different assemblies, you can use MemoryPackUnionFormatterAttribute instead. Formatters are generated the way that they are automatically registered via ModuleInitializer in C# 9.0 and above.

Note that ModuleInitializer is not supported in Unity, so the formatter must be manually registered. To register your union formatter invoke {name of your union formatter}Initializer.RegisterFormatter() manually in Startup. For example UnionSampleFormatterInitializer.RegisterFormatter().

// AssemblyA
[MemoryPackable(GenerateType.NoGenerate)]
public partial interface IUnionSample
{
}

// AssemblyB define definition outside of target type
[MemoryPackUnionFormatter(typeof(IUnionSample))]
[MemoryPackUnion(0, typeof(FooClass))]
[MemoryPackUnion(1, typeof(BarClass))]
public partial class UnionSampleFormatter
{
}

Union can be assembled in code via DynamicUnionFormatter<T>.

// (ushort, Type)[]
var formatter = new DynamicUnionFormatter<IFooBarBaz>(
    (0, typeof(Foo)),
    (1, typeof(Bar)),
    (2, typeof(Baz))
);

MemoryPackFormatterProvider.Register(formatter);

Serialize API

Serialize has three overloads.

// Non generic API also available, these version is first argument is Type and value is object?
byte[] Serialize<T>(in T? value, MemoryPackSerializerOptions? options = default)
void Serialize<T, TBufferWriter>(in TBufferWriter bufferWriter, in T? value, MemoryPackSerializerOptions? options = default)
async ValueTask SerializeAsync<T>(Stream stream, T? value, MemoryPackSerializerOptions? options = default, CancellationToken cancellationToken = default)

For performance, the recommended API uses BufferWriter. This serializes directly into the buffer. It can be applied to PipeWriter in System.IO.Pipelines, BodyWriter in ASP .NET Core, etc.

If a byte[] is required (e.g. RedisValue in StackExchange.Redis), the return byte[] API is simple and almost as fast.

Note that SerializeAsync for Stream is asynchronous only for Flush; it serializes everything once into MemoryPack's internal pool buffer and then writes using WriteAsync. Therefore, the BufferWriter overload, which separates and controls buffer and flush, is better.

If you want to do a complete streaming write, see the Streaming Serialization section.

MemoryPackSerializerOptions

MemoryPackSerializerOptions configures whether strings are serialized as UTF16 or UTF8. This can be configured by passing MemoryPackSerializerOptions.Utf8 for UTF8 encoding, MemoryPackSerializerOptions.Utf16 for UTF16 encoding or MemoryPackSerializerOptions.Default which defaults to UTF8. Passing null or using the default parameter results in UTF8 encoding.

Since C#'s internal string representation is UTF16, UTF16 performs better. However, the payload tends to be larger; in UTF8, an ASCII string is one byte, while in UTF16 it is two bytes. Because the difference in size of this payload is so large, UTF8 is set by default.

If the data is non-ASCII (e.g. Japanese, which can be more than 3 bytes, and UTF8 is larger), or if you have to compress it separately, UTF16 may give better results.

While UTF8 or UTF16 can be selected during serialization, it is not necessary to specify it during deserialization. It will be automatically detected and deserialized normally.

Additionaly you can get/set IServiceProvider? ServiceProvider { get; init; } from options. It is useful to get DI object(such as ILogger<T>) from serialization process(MemoryPackReader/MemoryPackWriter has .Options property).

Deserialize API

Deserialize has ReadOnlySpan<byte> and ReadOnlySequence<byte>, Stream overload and ref support.

T? Deserialize<T>(ReadOnlySpan<byte> buffer)
int Deserialize<T>(ReadOnlySpan<byte> buffer, ref T? value)
T? Deserialize<T>(in ReadOnlySequence<byte> buffer)
int Deserialize<T>(in ReadOnlySequence<byte> buffer, ref T? value)
async ValueTask<T?> DeserializeAsync<T>(Stream stream)

ref overload overwrites an existing instance, for details see the Overwrite section.

DeserializeAsync(Stream) is not a complete streaming read operation, first it reads into MemoryPack's internal pool up to the end-of-stream, then it deserializes.

If you want to do a complete streaming read operation, see the Streaming Serialization section.

Overwrite

To reduce allocations, MemoryPack supports deserializing to an existing instance, overwriting it. This can be used with the Deserialize(ref T? value) overload.

var person = new Person();
var bin = MemoryPackSerializer.Serialize(person);

// overwrite data to existing instance.
MemoryPackSerializer.Deserialize(bin, ref person);

MemoryPack will attempt to overwrite as much as possible, but if the following conditions do not match, it will create a new instance (as in normal deserialization).

Version tolerant

In default(GenerateType.Object), MemoryPack supports limited schema evolution.

[MemoryPackable]
public partial class VersionCheck
{
    public int Prop1 { get; set; }
    public long Prop2 { get; set; }
}

// Add is OK.
[MemoryPackable]
public partial class VersionCheck
{
    public int Prop1 { get; set; }
    public long Prop2 { get; set; }
    public int? AddedProp { get; set; }
}

// Remove is NG.
[MemoryPackable]
public partial class VersionCheck
{
    // public int Prop1 { get; set; }
    public long Prop2 { get; set; }
}

// Change order is NG.
[MemoryPackable]
public partial class VersionCheck
{
    public long Prop2 { get; set; }
    public int Prop1 { get; set; }
}

In use-case, store old data (to file, to redis, etc...) and read to new schema is always ok. In the RPC scenario, schema exists both on the client and the server side, the client must be updated before the server. An updated client has no problem connecting to the old server but an old client can not connect to a new server.

By default, when the old data read to new schema, any members not on the data side are initialized with the default literal. If you want to avoid this and use initial values of field/properties, you can use [SuppressDefaultInitialization].

[MemoryPackable]
public partial class DefaultValue
{
    public string Prop1 { get; set; }

    [SuppressDefaultInitialization]
    public int Prop2 { get; set; } = 111; // < if old data is missing, set `111`.
    
    public int Prop3 { get; set; } = 222; // < if old data is missing, set `default`.
}

[SuppressDefaultInitialization] has following limitation:

The next Serialization info section shows how to check for schema changes, e.g., by CI, to prevent accidents.

When using GenerateType.VersionTolerant, it supports full version-tolerant.

// Ok to serialize/deserialize both 
// VersionTolerantObject1 -> VersionTolerantObject2 and 
// VersionTolerantObject2 -> VersionTolerantObject1

[MemoryPackable(GenerateType.VersionTolerant)]
public partial class VersionTolerantObject1
{
    [MemoryPackOrder(0)]
    public int MyProperty0 { get; set; } = default;

    [MemoryPackOrder(1)]
    public long MyProperty1 { get; set; } = default;

    [MemoryPackOrder(2)]
    public short MyProperty2 { get; set; } = default;
}

[MemoryPackable(GenerateType.VersionTolerant)]
public partial class VersionTolerantObject2
{
    [MemoryPackOrder(0)]
    public int MyProperty0 { get; set; } = default;

    // deleted
    //[MemoryPackOrder(1)]
    //public long MyProperty1 { get; set; } = default;

    [MemoryPackOrder(2)]
    public short MyProperty2 { get; set; } = default;

    // added
    [MemoryPackOrder(3)]
    public short MyProperty3 { get; set; } = default;
}

// If set SerializeLayout.Sequential explicitly, allows automatically order.
// But it can not remove any member for versoin-tolerant.
[MemoryPackable(GenerateType.VersionTolerant, SerializeLayout.Sequential)]
public partial class VersionTolerantObject3
{
    public int MyProperty0 { get; set; } = default;
    public long MyProperty1 { get; set; } = default;
    public short MyProperty2 { get; set; } = default;
}

GenerateType.VersionTolerant is slower than GenerateType.Object in serializing. Also, the payload size will be slightly larger.

Serialization info

You can check IntelliSense in type what members are serialized. There is an option to write that information to a file at compile time. Set MemoryPackGenerator_SerializationInfoOutputDirectory as follows.

<!-- output memorypack serialization info to directory -->
<ItemGroup>
    <CompilerVisibleProperty Include="MemoryPackGenerator_SerializationInfoOutputDirectory" />
</ItemGroup>
<PropertyGroup>
    <MemoryPackGenerator_SerializationInfoOutputDirectory>$(MSBuildProjectDirectory)\MemoryPackLogs</MemoryPackGenerator_SerializationInfoOutputDirectory>
</PropertyGroup>

The following info is written to the file.

image

If the type is unmanaged, showed unmanaged before type name.

unmanaged FooStruct
---
int x
int y

By checking the differences in this file, dangerous schema changes can be prevented. For example, you may want to use CI to detect the following rules

Circular Reference

MemoryPack also supports circular reference. This allows the tree objects to be serialized as is.

// to enable circular-reference, use GenerateType.CircularReference
[MemoryPackable(GenerateType.CircularReference)]
public partial class Node
{
    [MemoryPackOrder(0)]
    public Node? Parent { get; set; }
    [MemoryPackOrder(1)]
    public Node[]? Children { get; set; }
}

For example, System.Text.Json preserve-references code will become like here.

// https://learn.microsoft.com/en-us/dotnet/standard/serialization/system-text-json/preserve-references?pivots=dotnet-7-0
Employee tyler = new()
{
    Name = "Tyler Stein"
};

Employee adrian = new()
{
    Name = "Adrian King"
};

tyler.DirectReports = new List<Employee> { adrian };
adrian.Manager = tyler;

var bin = MemoryPackSerializer.Serialize(tyler);
Employee? tylerDeserialized = MemoryPackSerializer.Deserialize<Employee>(bin);

Console.WriteLine(tylerDeserialized?.DirectReports?[0].Manager == tylerDeserialized); // true

[MemoryPackable(GenerateType.CircularReference)]
public partial class Employee
{
    [MemoryPackOrder(0)]
    public string? Name { get; set; }
    [MemoryPackOrder(1)]
    public Employee? Manager { get; set; }
    [MemoryPackOrder(2)]
    public List<Employee>? DirectReports { get; set; }
}

GenerateType.CircularReference has the same characteristics as version-tolerant. However, as an additional constraint, only parameterless constructors are allowed. Also, object reference tracking is only done for objects marked with GenerateType.CircularReference. If you want to track any other object, wrap it.

CustomFormatter

If implements MemoryPackCustomFormatterAttribute<T> or MemoryPackCustomFormatterAttribute<TFormatter, T>(more performant, but complex), you can configure to use custom formatter to MemoryPackObject's member.

[AttributeUsage(AttributeTargets.Field | AttributeTargets.Property, AllowMultiple = false, Inherited = false)]
public abstract class MemoryPackCustomFormatterAttribute<T> : Attribute
{
    public abstract IMemoryPackFormatter<T> GetFormatter();
}

MemoryPack provides the following formatting attributes: Utf8StringFormatterAttribute, Utf16StringFormatterAttribute, InternStringFormatterAttribute, OrdinalIgnoreCaseStringDictionaryFormatterAttribute<TValue>, BitPackFormatterAttribute, BrotliFormatter, BrotliStringFormatter, BrotliFormatter<T>, MemoryPoolFormatter<T>, ReadOnlyMemoryPoolFormatter<T>.

[MemoryPackable]
public partial class Sample
{
    // serialize this member as UTF16 String, it is performant than UTF8 but in ASCII, size is larger(but non ASCII, sometimes smaller).
    [Utf16StringFormatter]
    public string? Text { get; set; }

    // In deserialize, Dictionary is initialized with StringComparer.OrdinalIgnoreCase.
    [OrdinalIgnoreCaseStringDictionaryFormatter<int>]
    public Dictionary<string, int>? Ids { get; set; }
    
    // In deserialize time, all string is interned(see: String.Intern). If similar values come repeatedly, it saves memory.
    [InternStringFormatter]
    public string? Flag { get; set; }
}

In order to configure a set/dictionary's equality comparer, all built-in formatters have a comparer constructor overload. You can easily create custom equality-comparer formatters.

public sealed class OrdinalIgnoreCaseStringDictionaryFormatter<TValue> : MemoryPackCustomFormatterAttribute<Dictionary<string, TValue?>>
{
    static readonly DictionaryFormatter<string, TValue?> formatter = new DictionaryFormatter<string, TValue?>(StringComparer.OrdinalIgnoreCase);

    public override IMemoryPackFormatter<Dictionary<string, TValue?>> GetFormatter()
    {
        return formatter;
    }
}

BitPackFormatter compresses bool[] types only. bool[] is normally serialized as 1 byte per boolean value, however BitPackFormatter serializes bool[] like a BitArray storing each bool as 1 bit. Using BitPackFormatter, 8 bools become 1 byte where they would normally be 8 bytes, resulting in a 8x smaller size.

[MemoryPackable]
public partial class Sample
{
    public int Id { get; set; }

    [BitPackFormatter]
    public bool[]? Data { get; set; }
}

BrotliFormatter is for byte[], for example you can compress large payload by Brotli.

[MemoryPackable]
public partial class Sample
{
    public int Id { get; set; }

    [BrotliFormatter]
    public byte[]? Payload { get; set; }
}

BrotliStringFormatter is for string, serialize compressed string (UTF16) by Brotli.

[MemoryPackable]
public partial class Sample
{
    public int Id { get; set; }

    [BrotliStringFormatter]
    public string? LargeText { get; set; }
}

BrotliFormatter<T> is for any type, serialized data compressed by Brotli. If a type is byte[] or string, you should use BrotliFormatter or BrotliStringFormatter for performance.

[MemoryPackable]
public partial class Sample
{
    public int Id { get; set; }

    [BrotliFormatter<ChildType>]
    public ChildType? Child { get; set; }
}

Deserialize array pooling

In order to deserialize a large array (any T), MemoryPack offers multiple efficient pooling methods. The most effective way is to use the #Overwrite function. In particular List<T> is always reused.

[MemoryPackable]
public partial class ListBytesSample
{
    public int Id { get; set; }
    public List<byte> Payload { get; set; }
}

// ----

// List<byte> is reused, no allocation in deserialize.
MemoryPackSerializer.Deserialize<ListBytesSample>(bin, ref reuseObject);

// for efficient operation, you can get Span<T> by CollectionsMarshal
var span = CollectionsMarshal.AsSpan(value.Payload);

A convenient way is to deserialize to an ArrayPool at deserialization time. MemoryPack provides MemoryPoolFormatter<T> and ReadOnlyMemoryPoolFormatter<T>.

[MemoryPackable]
public partial class PoolModelSample : IDisposable
{
    public int Id { get; }

    [MemoryPoolFormatter<byte>]
    public Memory<byte> Payload { get; private set; }

    public PoolModelSample(int id, Memory<byte> payload)
    {
        Id = id;
        Payload = payload;
    }

    // You must write the return code yourself, here is snippet.

    bool usePool;

    [MemoryPackOnDeserialized]
    void OnDeserialized()
    {
        usePool = true;
    }

    public void Dispose()
    {
        if (!usePool) return;

        Return(Payload); Payload = default;
    }

    static void Return<T>(Memory<T> memory) => Return((ReadOnlyMemory<T>)memory);

    static void Return<T>(ReadOnlyMemory<T> memory)
    {
        if (MemoryMarshal.TryGetArray(memory, out var segment) && segment.Array is { Length: > 0 })
        {
            ArrayPool<T>.Shared.Return(segment.Array, clearArray: RuntimeHelpers.IsReferenceOrContainsReferences<T>());
        }
    }
}

// ---

using(var value = MemoryPackSerializer.Deserialize<PoolModelSample>(bin))
{
    // do anything...
}   // return to ArrayPool

Performance

See the my blog post How to make the fastest .NET Serializer with .NET 7 / C# 11, case of MemoryPack

Payload size and compression

Payload size depends on the target value; unlike JSON, there are no keys and it is a binary format, so the payload size is likely to be smaller than JSON.

For those with varint encoding, such as MessagePack and Protobuf, MemoryPack tends to be larger if ints are used a lot (in MemoryPack, ints are always 4 bytes due to fixed size encoding, while MessagePack is 1~5 bytes).

float and double are 4 bytes and 8 bytes in MemoryPack, but 5 bytes and 9 bytes in MessagePack. So MemoryPack is smaller, for example, for Vector3 (float, float, float) arrays.

String is UTF8 by default, which is similar to other serializers, but if the UTF16 option is chosen, it will be of a different nature.

In any case, if the payload size is large, compression should be considered. LZ4, ZStandard and Brotli are recommended.

Compression

MemoryPack provides an efficient helper for Brotli compression via BrotliEncoder and BrotliDecoder. MemoryPack's BrotliCompressor and BrotliDecompressor provide compression/decompression optimized for MemoryPack's internal behavior.

using MemoryPack.Compression;

// Compression(require using)
using var compressor = new BrotliCompressor();
MemoryPackSerializer.Serialize(compressor, value);

// Get compressed byte[]
var compressedBytes = compressor.ToArray();

// Or write to other IBufferWriter<byte>(for example PipeWriter)
compressor.CopyTo(response.BodyWriter);

using MemoryPack.Compression;

// Decompression(require using)
using var decompressor = new BrotliDecompressor();

// Get decompressed ReadOnlySequence<byte> from ReadOnlySpan<byte> or ReadOnlySequence<byte>
var decompressedBuffer = decompressor.Decompress(buffer);

var value = MemoryPackSerializer.Deserialize<T>(decompressedBuffer);

Both BrotliCompressor and BrotliDecompressor are struct, it does not allocate memory on heap. Both store compressed or decompressed data in an internal memory pool for Serialize/Deserialize. Therefore, it is necessary to release the memory pooling, don't forget to use using.

Compression level is very important. The default is set to quality-1 (CompressionLevel.Fastest), which is different from the .NET default (CompressionLevel.Optimal, quality-4).

Fastest (quality-1) will be close to the speed of LZ4, but 4 is much slower. This was determined to be critical in the serializer use scenario. Be careful when using the standard BrotliStream (quality-4 is the default). In any case, compression/decompression speeds and sizes will result in very different results for different data. Please prepare the data to be handled by your application and test it yourself.

Note that there is a several-fold speed penalty between MemoryPack's uncompressed and Brotli's added compression.

Brotli is also suppored in a custom formatter. BrotliFormatter can compress a specific member.

[MemoryPackable]
public partial class Sample
{
    public int Id { get; set; }

    [BrotliFormatter]
    public byte[]? Payload { get; set; }
}

Serialize external types

If you want to serialize external types, you can make a custom formatter and register it to provider, see Formatter/Provider API for details. However, creating a custom formatter is difficult. Therefore, we recommend making a wrapper type. For example, if you want to serialize an external type called AnimationCurve.

// Keyframe: (float time, float inTangent, float outTangent, int tangentMode, int weightedMode, float inWeight, float outWeight)
[MemoryPackable]
public readonly partial struct SerializableAnimationCurve
{
    [MemoryPackIgnore]
    public readonly AnimationCurve AnimationCurve;

    [MemoryPackInclude]
    WrapMode preWrapMode => AnimationCurve.preWrapMode;
    [MemoryPackInclude]
    WrapMode postWrapMode => AnimationCurve.postWrapMode;
    [MemoryPackInclude]
    Keyframe[] keys => AnimationCurve.keys;

    [MemoryPackConstructor]
    SerializableAnimationCurve(WrapMode preWrapMode, WrapMode postWrapMode, Keyframe[] keys)
    {
        var curve = new AnimationCurve(keys);
        curve.preWrapMode = preWrapMode;
        curve.postWrapMode = postWrapMode;
        this.AnimationCurve = curve;
    }

    public SerializableAnimationCurve(AnimationCurve animationCurve)
    {
        this.AnimationCurve = animationCurve;
    }
}

The type to wrap is public, but excluded from serialization (MemoryPackIgnore). The properties you want to serialize are private, but included (MemoryPackInclude). Two patterns of constructors should also be prepared. The constructor used by the serializer should be private.

As it is, it must be wrapped every time, which is inconvenient. And also strcut wrapper can not represents null. So let's create a custom formatter.

public class AnimationCurveFormatter : MemoryPackFormatter<AnimationCurve>
{
    // Unity does not support scoped and TBufferWriter so change signature to `Serialize(ref MemoryPackWriter writer, ref AnimationCurve value)`
    public override void Serialize<TBufferWriter>(ref MemoryPackWriter<TBufferWriter> writer, scoped ref AnimationCurve? value)
    {
        if (value == null)
        {
            writer.WriteNullObjectHeader();
            return;
        }

        writer.WritePackable(new SerializableAnimationCurve(value));
    }

    public override void Deserialize(ref MemoryPackReader reader, scoped ref AnimationCurve? value)
    {
        if (reader.PeekIsNull())
        {
            reader.Advance(1); // skip null block
            value = null;
            return;
        }
        
        var wrapped = reader.ReadPackable<SerializableAnimationCurve>();
        value = wrapped.AnimationCurve;
    }
}

Finally, register the formatter in startup.

MemoryPackFormatterProvider.Register<AnimationCurve>(new AnimationCurveFormatter());

Note: Unity's AnimationCurve can serializable by default so does not needs this custom formatter for AnimationCurve

Packages

MemoryPack has these packages.

MemoryPack is the main library, it provides full support for high performance serialization and deserialization of binary objects. It depends on MemoryPack.Core for the core base libraries and MemoryPack.Generator for code generation. MemoryPack.Streaming adds additional extensions for Streaming Serialization. MemoryPack.AspNetCoreMvcFormatter adds input/output formatters for ASP.NET Core. MemoryPack.UnityShims adds Unity shim types and formatters for share type between .NET and Unity.

TypeScript and ASP.NET Core Formatter

MemoryPack supports TypeScript code generation. It generates class and serialization code from C#, In other words, you can share types with the Browser without using OpenAPI, proto, etc.

Code generation is integrated with Source Generator, the following options(MemoryPackGenerator_TypeScriptOutputDirectory) set the output directory for TypeScript code. Runtime code is output at the same time, so no additional dependencies are required.

<!-- output memorypack TypeScript code to directory -->
<ItemGroup>
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptOutputDirectory" />
</ItemGroup>
<PropertyGroup>
    <MemoryPackGenerator_TypeScriptOutputDirectory>$(MSBuildProjectDirectory)\wwwroot\js\memorypack</MemoryPackGenerator_TypeScriptOutputDirectory>
</PropertyGroup>

A C# MemoryPackable type must be annotated with [GenerateTypeScript].

[MemoryPackable]
[GenerateTypeScript]
public partial class Person
{
    public required Guid Id { get; init; }
    public required int Age { get; init; }
    public required string FirstName { get; init; }
    public required string LastName { get; init; }
    public required DateTime DateOfBirth { get; init; }
    public required Gender Gender { get; init; }
    public required string[] Emails { get; init; }
}

public enum Gender
{
    Male, Female, Other
}

Runtime code and TypeScript type will be generated in the target directory.

image

The generated code is as follows, with simple fields and static methods for serialize/serializeArray and deserialize/deserializeArray.

import { MemoryPackWriter } from "./MemoryPackWriter.js";
import { MemoryPackReader } from "./MemoryPackReader.js";
import { Gender } from "./Gender.js"; 

export class Person {
    id: string;
    age: number;
    firstName: string | null;
    lastName: string | null;
    dateOfBirth: Date;
    gender: Gender;
    emails: (string | null)[] | null;

    constructor() {
        // snip...
    }

    static serialize(value: Person | null): Uint8Array {
        // snip...
    }

    static serializeCore(writer: MemoryPackWriter, value: Person | null): void {
        // snip...
    }

    static serializeArray(value: (Person | null)[] | null): Uint8Array {
        // snip...
    }

    static serializeArrayCore(writer: MemoryPackWriter, value: (Person | null)[] | null): void {
        // snip...
    }
    static deserialize(buffer: ArrayBuffer): Person | null {
        // snip...
    }

    static deserializeCore(reader: MemoryPackReader): Person | null {
        // snip...
    }

    static deserializeArray(buffer: ArrayBuffer): (Person | null)[] | null {
        // snip...
    }

    static deserializeArrayCore(reader: MemoryPackReader): (Person | null)[] | null {
        // snip...
    }
}

You can use this type like following.

let person = new Person();
person.id = crypto.randomUUID();
person.age = 30;
person.firstName = "foo";
person.lastName = "bar";
person.dateOfBirth = new Date(1999, 12, 31, 0, 0, 0);
person.gender = Gender.Other;
person.emails = ["foo@bar.com", "zoo@bar.net"];

// serialize to Uint8Array
let bin = Person.serialize(person);

let blob = new Blob([bin.buffer], { type: "application/x-memorypack" })

let response = await fetch("http://localhost:5260/api",
    { method: "POST", body: blob, headers: { "Content-Type": "application/x-memorypack" } });

let buffer = await response.arrayBuffer();

// deserialize from ArrayBuffer 
let person2 = Person.deserialize(buffer);

The MemoryPack.AspNetCoreMvcFormatter package adds MemoryPack input and output formatters for ASP.NET Core MVC. You can add MemoryPackInputFormatter, MemoryPackOutputFormatter to ASP.NET Core MVC with the following code.

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddRazorPages();

builder.Services.AddControllers(options =>
{
    options.InputFormatters.Insert(0, new MemoryPackInputFormatter());
    // If checkContentType: true then can output multiple format(JSON/MemoryPack, etc...). default is false.
    options.OutputFormatters.Insert(0, new MemoryPackOutputFormatter(checkContentType: false));
});

If you call from HttpClient, you can set application/x-memorypack to content-header.

var content = new ByteArrayContent(bin)
content.Headers.ContentType = new MediaTypeHeaderValue("application/x-memorypack");

TypeScript Type Mapping

There are a few restrictions on the types that can be generated. Among the primitives, char and decimal are not supported. Also, OpenGenerics type cannot be used.

C#TypeScriptDescription
boolboolean
bytenumber
sbytenumber
intnumber
uintnumber
shortnumber
ushortnumber
longbigint
ulongbigint
floatnumber
doublenumber
stringstring | null
GuidstringIn TypeScript, represents as string but serialize/deserialize as 16byte binary
DateTimeDateDateTimeKind will be ignored
enumconst enumlong and ulong underlying type is not supported
T?T | null
T[]T[] | null
byte[]Uint8Array | null
: ICollection<T>T[] | nullSupports all ICollection<T> implemented type like List<T>
: ISet<T>Set<T> | nullSupports all ISet<T> implemented type like HashSet<T>
: IDictionary<K,V>Map<K, V> | nullSupports all IDictionary<K,V> implemented type like Dictionary<K,V>.
[MemoryPackable]classSupports class only
[MemoryPackUnion]abstract class

[GenerateTypeScript] can only be applied to classes and is currently not supported by struct.

Configure import file extension and member name casing

In default, MemoryPack generates file extension as .js like import { MemoryPackWriter } from "./MemoryPackWriter.js";. If you want to change other extension or empty, use MemoryPackGenerator_TypeScriptImportExtension to configure it. Also the member name is automatically converted to camelCase. If you want to use original name, use MemoryPackGenerator_TypeScriptConvertPropertyName to false.

<ItemGroup>
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptOutputDirectory" />
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptImportExtension" />
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptConvertPropertyName" />
    <CompilerVisibleProperty Include="MemoryPackGenerator_TypeScriptEnableNullableTypes" />
</ItemGroup>
<PropertyGroup>
    <MemoryPackGenerator_TypeScriptOutputDirectory>$(MSBuildProjectDirectory)\wwwroot\js\memorypack</MemoryPackGenerator_TypeScriptOutputDirectory>
    <!-- allows empty -->
    <MemoryPackGenerator_TypeScriptImportExtension></MemoryPackGenerator_TypeScriptImportExtension>
    <!-- default is true -->
    <MemoryPackGenerator_TypeScriptConvertPropertyName>false</MemoryPackGenerator_TypeScriptConvertPropertyName>
    <!-- default is false -->
    <MemoryPackGenerator_TypeScriptEnableNullableTypes>true</MemoryPackGenerator_TypeScriptEnableNullableTypes>
</PropertyGroup>

MemoryPackGenerator_TypeScriptEnableNullableTypes allows C# nullable annotations to be reflected in TypeScript code. The default is false, making everything nullable.

Streaming Serialization

MemoryPack.Streaming provides MemoryPackStreamingSerializer, which adds additional support for serializing and deserializing collections with streams.

public static class MemoryPackStreamingSerializer
{
    public static async ValueTask SerializeAsync<T>(PipeWriter pipeWriter, int count, IEnumerable<T> source, int flushRate = 4096, CancellationToken cancellationToken = default)
    public static async ValueTask SerializeAsync<T>(Stream stream, int count, IEnumerable<T> source, int flushRate = 4096, CancellationToken cancellationToken = default)
    public static async IAsyncEnumerable<T?> DeserializeAsync<T>(PipeReader pipeReader, int bufferAtLeast = 4096, int readMinimumSize = 8192, [EnumeratorCancellation] CancellationToken cancellationToken = default)
    public static IAsyncEnumerable<T?> DeserializeAsync<T>(Stream stream, int bufferAtLeast = 4096, int readMinimumSize = 8192, CancellationToken cancellationToken = default)
}

Formatter/Provider API

If you want to implement formatter manually, inherit MemoryPackFormatter<T> and override the Serialize and Deserialize methods.

public class SkeltonFormatter : MemoryPackFormatter<Skelton>
{
    public override void Serialize<TBufferWriter>(ref MemoryPackWriter<TBufferWriter> writer, scoped ref Skelton? value)
    {
        if (value == null)
        {
            writer.WriteNullObjectHeader();
            return;
        }

        // use writer method.
    }

    public override void Deserialize(ref MemoryPackReader reader, scoped ref Skelton? value)
    {
        if (!reader.TryReadObjectHeader(out var count))
        {
            value = null;
            return;
        }

        // use reader method.
    }
}

The created formatter is registered with MemoryPackFormatterProvider.

MemoryPackFormatterProvider.Register(new SkeltonFormatter());

Note: unmanged struct(doesn't contain reference types) can not use custom formatter, it always serializes native memory layout.

MemoryPackWriter/ReaderOptionalState

Initializing MemoryPackWriter/MemoryPackReader requires OptionalState. It is wrapper of MemoryPackSerializerOptions, it can create form MemoryPackWriterOptionalStatePool.

// when disposed, OptionalState will return to pool.
using(var state = MemoryPackWriterOptionalStatePool.Rent(MemoryPackSerializerOptions.Default))
{
    var writer = new MemoryPackWriter<T>(ref t, state);
}

// for Reader
using (var state = MemoryPackReaderOptionalStatePool.Rent(MemoryPackSerializerOptions.Default))
{
    var reader = new MemoryPackReader(buffer, state);
}

Target framework dependency

MemoryPack provides netstandard2.1 and net7.0 but both are not compatible. For example, MemoryPackable types under netstandard2.1 project and use it from net7.0 project, throws runtime exception like this

Unhandled exception. System.TypeLoadException: Virtual static method '' is not implemented on type '' from assembly '*'.

Since net7.0 uses static abstract members (Virtual static method), that does not support netstandard2.1, this behavior is a specification.

.NET 7 project shouldn't use the netstandard 2.1 dll. In other words, if the Application is a .NET 7 Project, all the dependencies that use MemoryPack must support .NET 7. So if a library developer has a dependency on MemoryPack, you need to configure dual target framework.

<TargetFrameworks>netstandard2.1;net7.0</TargetFrameworks>

RPC

Cysharp/MagicOnion is a code-first grpc-dotnet framework using MessagePack instead of protobuf. MagicOnion now supports MemoryPack as a serialization layer via MagicOnion.Serialization.MemoryPack package(preview). See details: MagicOnion#MemoryPack support

Unity

Minimum supported Unity version is 2022.3.12f1.

The MemoryPack core package is provided by nuget. It is also available in Unity. If you want to get Unity built-in type support, we additionally provide MemoryPack.Unity extension.

  1. Install MemoryPack from NuGet using NuGetForUnity
  1. Install the MemoryPack.Unity package by referencing the git URL

MemoryPack uses the ..* release tag, so you can specify a version like #1.0.0. For example: https://github.com/Cysharp/MemoryPack.git?path=src/MemoryPack.Unity/Assets/MemoryPack.Unity#1.0.0

As with the .NET version, the code is generated by a code generator (MemoryPack.Generator.dll). Reflection-free implementation also provides the best performance in IL2CPP.

For more information on Unity and Source Generator, please refer to the Unity documentation.

Source Generator is also used officially by Unity by com.unity.properties and com.unity.entities. In other words, it is the standard for code generation in the next generation of Unity.

You can serialize all unmanaged types (such as Vector3, Rect, etc...) and some classes(AnimationCurve, Gradient, RectOffset). If you want to serialize other Unity-specific types, see Serialize external types section.

In Unity performance, MemoryPack is x3~x10 faster than JsonUtility.

image

If shared code has Unity's type(Vector2, etc...), MemoryPack provides MemoryPack.UnityShims package in NuGet.

The MemoryPack.UnityShims package provides shims for Unity's standard structs (Vector2, Vector3, Vector4, Quaternion, Color, Bounds, Rect, Keyframe, WrapMode, Matrix4x4, GradientColorKey, GradientAlphaKey, GradientMode, Color32, LayerMask, Vector2Int, Vector3Int, RangeInt, RectInt, BoundsInt) and some classes(AnimationCurve, Gradient, RectOffset).

[!WARNING] Currently, the following limitations exist for use in Unity

  1. Unity version does not support CustomFormatter.
  2. If you are using .NET7 or later, MemoryPack binary format is not fully compatible with Unity.
    • This problem occurs with value types that [StructLayout(LayoutKind.Auto)] is explicitly specified. (The default for struct is LayoutKind.Sequencil.) For such types, binaries serialized in .NET cannot be deserialized in Untiy. Similarly, a binary serialized in Unity cannot be serialized in .NET side.
    • The affected types typically include the following types.
      • DateTimeOffset
      • ValueTuple
    • Currently, the simple solution is to not use these types.

Native AOT

Unfortunately, .NET 7 Native AOT causes crash (Generic virtual method pointer lookup failure) when use MemoryPack due to a runtime bug. It is going to be fixed in .NET 8. Using ``Microsoft.DotNet.ILCompiler` preview version, will fix it in .NET 7. Please see issue's comment how setup it.

Binary wire format specification

The type of T defined in Serialize<T> and Deserialize<T> is called C# schema. MemoryPack format is not self-described format. Deserialize requires the corresponding C# schema. These types exist as internal representations of binaries, but types cannot be determined without a C# schema.

Endian must be Little Endian. However, reference C# implementation does not care about endianness so can not use on big-endian machine. However, modern computers are usually little-endian.

There are eight types of format.

Unmanaged struct

Unmanaged struct is C# struct that doesn't contain reference types, similar constraint of C# Unmanaged types. Serializing struct layout as it is, includes padding.

Object

(byte memberCount, [values...])

Object has 1byte unsigned byte as member count in header. Member count allows 0 to 249, 255 represents object is null. Values store memorypack value for the number of member count.

Version Tolerant Object

(byte memberCount, [varint byte-length-of-values...], [values...])

Version Tolerant Object is similar as Object but has byte length of values in header. varint follows these spec, first sbyte is value or typeCode and next X byte is value. 0 to 127 = unsigned byte value, -1 to -120 = signed byte value, -121 = byte, -122 = sbyte, -123 = ushort, -124 = short, -125 = uint, -126 = int, -127 = ulong, -128 = long.

Circular Reference Object

(byte memberCount, [varint byte-length-of-values...], varint referenceId, [values...])
(250, varint referenceId)

Circular Reference Object is similar as Version Tolerant Object but if memberCount is 250, next varint(unsigned-int32) is referenceId. If not, after byte-length-of-values, varint referenceId is written.

Tuple

(values...)

Tuple is fixed-size, non-nullable value collection. In .NET, KeyValuePair<TKey, TValue> and ValueTuple<T,...> are serialized as Tuple.

Collection

(int length, [values...])

Collection has 4 byte signed integer as data count in header, -1 represents null. Values store memorypack value for the number of length.

String

(int utf16-length, utf16-value)
(int ~utf8-byte-count, int utf16-length, utf8-bytes)

String has two-forms, UTF16 and UTF8. If first 4byte signed integer is -1, represents null. 0, represents empty. UTF16 is same as collection(serialize as ReadOnlySpan<char>, utf16-value's byte count is utf16-length * 2). If first signed integer <= -2, value is encoded by UTF8. utf8-byte-count is encoded in complement, ~utf8-byte-count to retrieve count of bytes. Next signed integer is utf16-length, it allows -1 that represents unknown length. utf8-bytes store bytes for the number of utf8-byte-count.

Union

(byte tag, value)
(250, ushort tag, value)

First unsigned byte is tag that for discriminated value type or flag, 0 to 249 represents tag, 250 represents next unsigned short is tag, 255 represents union is null.

License

This library is licensed under the MIT License.