Awesome

ECS.CSharp.Benchmark - Common use-cases

Motivation of this benchmark project:

• Compare performance of common uses cases of multiple ECS projects.
  "I want to compare the performance of two C# ECS projects"?
  Use the example command shown in the Benchmark CLI examples below.

• Utilize a common ECS operation of a specific ECS project in most simple & performant way.

• "I want to migrate from one C# ECS to another"?
  Navigate to a specific benchmark in both projects an use their implementations as migration guide.
  Benchmark implementations are intended to be as simple as possible.

• Having an alternative to the popular Ecs.CSharp.Benchmark.
  As the mentioned project is currently not active maintained.

See comments about this benchmark at reddit announcement post

Contents

• Tested projects
• ECS implementation
• Benchmarks
  • Overview
  • Basic
  • Relations
  • Command buffer
  • Events
  • Search
• Setup
• Contribution
• Benchmark CLI
• Other C# ECS Benchmarks

Tested projects

• All tested projects are engine agnostic.
• 🔒 C# - dll uses only verifiable safe code. This allows execution in trusted environments.
  See Unsafe code ⋅ Microsoft and C# Language specification ⋅ Microsoft using <AllowUnsafeBlocks>.

Projects ordered by dll size

ECS	ECS type	Entity	🔒 C#	size kb	tested	nuget latest
Leopotam.EcsLite	Sparse Set	int	✅	18	1.0.1	⁽¹⁾
Morpeh	?	class		76	2023.1.0
fennecs	Archetype	struct		157	0.5.10-beta
DefaultEcs	Sparse Set	struct		169	0.18.0-beta01
Friflo.Engine.ECS	Archetype	struct	✅	307	3.0.0-preview.10
Friflo.Engine.ECS.Boost	📦			+12	3.0.0-preview.10
TinyEcs	Archetype	struct		487	1.4.0
Myriad.ECS	Archetype	struct		631	21.2.1
Arch	Archetype	struct		765	1.2.8
Arch.Relationships	📦		✅	+9	1.0.1
Flecs.NET	Arch/Sparse	struct		1600	4.0.0

⁽¹⁾ nuget package not published by project owner

ECS implementation

The are typically two types used by many ECS projects

Archetype

Entities are stored in single array. Components as stored in "tables" aka Archetypes.
An Archetype contains arrays of components for a specific set of component types.

Pros: Enable fast iteration of component queries.
Cons: Add/remove operations require a structural change.

Sparse Set

A sparse Set based ECS stores each component in its own sparse set which is has the entity id as key.

Pros: Fast add/remove operations.
Cons: Each component type requires an array with the size of all components.

Benchmarks

Feature Matrix

• Watch - Watch / navigate entity components ad-hoc in debugger. <details> <summary>See example screenshot</summary> <img src="docs/images/Debugger-Watch.png" width="634" height="178"/> </details>

¹ Ensures a cycle free entity hierarchy. See CheckTreeCycles()

Run benchmarks always on an Apple Mac Mini M2.
Its hardware specs are fixed and can be compared with benchmarks you run by your own on this machine.

BenchmarkDotNet v0.13.12, macOS Sonoma 14.5 (23F79) [Darwin 23.5.0]
Apple M2, 1 CPU, 8 logical and 8 physical cores
.NET SDK 8.0.100
  [Host]   : .NET 8.0.0 (8.0.23.53103), Arm64 RyuJIT AdvSIMD
  ShortRun : .NET 8.0.0 (8.0.23.53103), Arm64 RyuJIT AdvSIMD

Overview

Readme benchmark update: 2024-08-09 using this run

Ratio: Average Performance Ratio - see C# ECS Benchmark Overview ⋅ Google Sheets

Basic

Add & Remove 1 / 5 component on 100 entities

Note: See impact of structural changes in Archetype based ECS projects.

Create 100 entities with 1 / 3 components

Created entities are initialized with custom values as recommended here.
"in a real-world environment creating a large number of default entities isn't very useful and you'll always want to set component values!"

Some frameworks support bulk creation of entities.
If available bulk creation is faster than the approach creating entities ony by one.

Create World

Delete 100.000 entities with 5 components

Get & Set 1 / 5 component on 100 entities

Note: Sparse Set based ECS projects are in lead because of viewer array lookups.

Query 100 / 100.000 entities with 1 / 5 components

Note: Archetype based ECS projects are in lead if querying multiple components.
Returned components are sequentially stored in memory providing a high cache hit rate.

Rule: Query implementations must not use vectorization (SIMD instructions).
If doing so this case requires a new benchmark category: QueryVectorization.

Query performance comparison: C++ vs C#

Performance reference vanilla C++ implementation

Relations

Some ECS projects have support for Entity Relationships.
Compared to relational databases: Entity relationships are similar to foreign keys referencing primary keys in other tables. ECS implementations typically ensure referential integrity. This means there are never links to entities which doesn't exist.

Relations enable directed links between entities aka entity relationships.
Directed link means that a link points from a source entity to a target entity.
A single entity can have multiple links to other target entities.

Add & Remove 1 / 100 link relations on 100 entities

When dealing with an ECS following question arises at some point:
"Is it okay for performance to use an array, List<> or Dictionary<> as a component field"?
No, its not 😲. Now each component has a one or more reference types.
As a result there is no cache locality anymore and GC requires much more CPU & memory resources.
This is the reason why many ECS projects have relations.

A typical limitation of an ECS is that an entity can only contain one component of a certain type.
Relations can be used to add multiple components of the same type to a single entity.
To differentiate relations added to the same entity following mechanisms are used:

• Friflo.Engine.ECS - A component field is used as discriminator specified in IRelationComponent<TKey>.
• Flecs.NET & TinyEcs - Tags are used as discriminator.
• Arch.Relationships - An additional entity with a component is used as discriminator.
• fennecs - Reference type instances are used as discriminator.

Add & Remove 1 / 100 relations on 100 entities

Add & Remove 10 child entities on 100 parent entities

Child / parent entity relationships are used to build a hierarchy / tree of entities.
It is, among other things, a use case for scene trees, entity parenting or character rig skeletons.

Command buffer

A command buffer is used to record entity changes in a buffer.
While recording the state of entities remains unchanged.
These changes are applied to these entities when calling either
Playback(), Execute(), Commit() or DeferEnd()

Add & Remove 2 components on 100 entities using a command buffer

1. Add components. Apply changes.
2. Remove components. Apply changes.

Events

ECS implementations supporting callbacks for specific events are called reactive.
Typical event types are:

• Add / Update / Remove component
• Add / Remove tag
• Create / Delete entity

Get callback event on Add & Remove 1 component on 100 entities

Search

A search can be used to get all entities with a specific component field value.
This type of search is typically executed in O(1) .
E.g. to find all entities having a Player component where Player.name == "Bob"

struct Player { string name; } 

A search can also be used for range queries to find all entities with a component field value in a [min, max] range.
E.g. a range query return all entities with a Health component where Health.value is between 10 and 100.

struct Health { int value; } 

Search and Range Queries of component fields are explained at this Wiki.

Search component field in 1.000.000 entities

Execute 1000 searches for different search values in a data set of 1.000.000 entities.
Each result has 1 match.

Search range of component fields in 1.000.000 entities

Execute 1000 range queries with different [min, max] in a data set of 1.000.000 entities.
Each result has 100 matches.

Setup

The benchmark project can be build and executed on Windows, macOS & Linux.
All popular IDE's can be used to run and debug the project: Rider, Visual Studio Code & Visual Studio.

Benchmark constraints

• Each benchmark is simple and uses the fastest single threaded variant available.
  To obtain clarity a project must not have multiple variants of the same benchmark type.
• A ECS framework must implement all Basic benchmarks. Reasons
  • Operations used in basic benchmarks are required to build a rudimentary application.
  • Avoid cherry picking of benchmarks with good performance.
• Each Benchmark shares no state or code with any other benchmarks.
• Adding or removing a benchmark implementation has no effect on all others.
• Each project has an extension class BenchUtils with two methods to used by its benchmarks.
  BenchUtils.CreateEntities(int count)
BenchUtils.AddComponents(this Entity[] entities)
• A package of the ECS must be available on nuget.

The benchmarks CreateEntity and DeleteEntity are changing the state of World which has influence on the benchmark measurement.
If executing their [Benchmark] method multiple times the number of entities will grow / shrink for each method iteration.
This would slow down the execution over time and give wrong measurement results.
To avoid this these benchmarks are executed with 100.000 entities every time on a new World instance.

Contribution

Contributions are welcome.
Only requirement: Ensure it compiles.

• How to add a single benchmark?
  Copy an existing benchmark and make adaptations.

• How to add benchmarks for a new ECS project?
  Copy an existing project and make adaptations.

• Adding a new benchmark category?
  Open a new issue or discussion to explain the feature.

Benchmark CLI

Currently ~ 100 benchmarks

• Running all benchmarks ~ 40 minutes
• Running all benchmarks with --job Short ~ 10 minutes
• Running all benchmarks with --job Dry ~ 1 minute (used by CI)

The published benchmarks are executed without: --job argument.
The measurement difference when using --job Short were 2x in some benchmarks.

For documentation of --job argument see BenchmarkDotNet CLI args

Windows CLI

cd ./src

dotnet run -c Release --filter *                                # run all benchmarks
dotnet run -c Release --filter *AddRemoveComponents_Friflo*     # run a specific benchmark
dotnet run -c Release --filter *AddRemoveComponents*            # run benchmarks of single category
dotnet run -c Release --filter *Friflo*                         # run benchmarks of single project
dotnet run -c Release --filter *Friflo* *Arch*                  # compare benchmarks of two projects

# run basic benchmarks
dotnet run -c Release --filter *AddRemoveComponents* *GetSetComponents* *CreateEntity* *CreateWorld* *DeleteEntity* *Query*
dotnet run -c Release --filter *Links* *Relations*              # run relation benchmarks
dotnet run -c Release --filter *CommandBuffer*                  # run command buffer benchmarks
dotnet run -c Release --filter *Events*                         # run component events benchmarks
dotnet run -c Release --filter *Search*                         # run search benchmarks

macOS / Linux CLI

cd ./src

dotnet run -c Release --filter \*                               # run all benchmarks
dotnet run -c Release --filter \*AddRemoveComponents_Friflo\*   # run a specific benchmark
dotnet run -c Release --filter \*AddRemoveComponents\*          # run benchmarks of single category
dotnet run -c Release --filter \*Friflo\*                       # run benchmarks of single project
dotnet run -c Release --filter \*Friflo\* \*Arch\*              # compare benchmarks of two projects

# run basic benchmarks
dotnet run -c Release --filter \*AddRemoveComponents\* \*GetSetComponents\* \*CreateEntity\* \*CreateWorld\* \*DeleteEntity\* \*Query\*
dotnet run -c Release --filter \*Links\* \*Relations\*          # run relation benchmarks
dotnet run -c Release --filter \*CommandBuffer\*                # run command buffer benchmarks
dotnet run -c Release --filter \*Events\*                       # run component events benchmarks
dotnet run -c Release --filter \*Search\*                       # run search benchmarks

Other C# ECS Benchmarks

• Ecs.CSharp.Benchmark
  Popular C# ECS Benchmark with focus on large data sets: Create & query entities

• ecs-benchmark-runner-dotnet
  C# ECS benchmark frameworks with a near-real-world scenario test scenario.
  Benchmarks runs on multiple platforms: .NET, Native AOT & Unity.