Awesome

[!WARNING] Benchmark is under active development all API, integrations and set of benchmarks is subject to change!

Latest run

updated with actions and can be found here

What is all about?

General idea is to hide implementation of each ECS under context abstraction and work with it from benchmark implementations.

Benchmarks design follow 2 rules which I try to balance with:

Strict usage - to ensure all benchmarks are running with same flow to avoid cheating.
Features utilization - to allow implementations to run in perfomant way.

General flow of any benchmark execution is divided into 3 steps:

Preparation
- Creating world
- Creating initial entities if needed
- Initialize filters and queries or other stuff which used to gain perfomance
Benchmark call
- Aquiring lock of world
- Run main logic
- Commiting changes
Cleanup - mostly omited

[!IMPORTANT] Don't search truth here. There won't be any.

Implemented contexts

ECS	Version	Implemented	Verified	Notes
Arch	1.3.3-alpha	✅	❌	N/A
fennecs	0.5.14-beta	✅	❌	N/A
Morpeh	stage-2024.1.0	✅	❌	N/A
DragonECS	0.8.61	✅	❌	N/A
LeoECS	2023.6.22	✅	❌	N/A
LeoECSLite	2024.5.22	✅	❌	N/A
DefaultECS	0.18.0-beta01	✅	❌	Analyzer 0.17.8
FlecsNET	4.0.3	✅	❌	N/A
TinyEcs	1.4.0	✅	❌	N/A
Xeno	0.1.6	✅	✅	N/A
FriFlo	3.0.0-preview.18	✅	❌	N/A
StaticEcs	0.9.0	✅	✅	N/A

Implemented benchmarks

Benchmark	Description
Create Empty Entity	Creates [EntityCount] empty entities
Create Entity With N Components	Creates [EntityCount] entitites with N components
Add N Components	Adds N components to [EntityCount] entities
Remove N Components	Adds N components to [EntityCount] entities
System with N Components	Performs simple operations on entities (numbers sum)
System with N Components Multiple Composition	Same as System with N Components but with mixture of other components

Running

Just call Benchmark.sh from terminal.

Command line args:

arg	description	sample
benchmark	allow to specify single benchmark to run	`benchmarks=CreateEmptyEntities,Add1Component`
benchmarks	allow to specify benchmarks to run	`benchmark=CreateEmptyEntities`
contexts	allow to specify contexts to run	`contexts=Morpeh,Fennecs,...`
--list	prints all benchmarks name	`--list`

Since all comparisons is made by string contains you can simply write something like contexts=Morpeh instead of context=MorpehContext and benchmarks=With1,With2 to launch subset of benchmarks. Selected benchmarks and contexts will be logged to console. BUT benchmark arg requires exact name match with those printed with --list

Contribution

Fork
Implement
Create PR

Problems

Because of nature of BenchmarkDotNet there's sequential iteration of creating entities happening. This leads to case where, for example we creating 100k entities in benchmark, it's properly cleared in Setup and Cleanup but benchmaXeno.SourceGenerator.csprojrk itself will be called multiple times which will lead to creating 100k entities, then another 100k and in some cases lead to millions of entities in the world which can affect perfomance of creation and deletion on certain ECS implementations.
System benchmarks which uses Padding property produces up to 1.100.000 entities each because of logic of padding generation. It affects runs duration but for now i'm not sure about correct way do fix that (maybe keep entire entities count up to EntityCount so it'll not affect speed but it'll reduce actual entity count to about 9.9k so archetype ecs implementation will gain significant boost).
Because some framework deleting entity on delete last components there are differences in behaviours in tests and benchmarks. For example RemoveComponent benchmark will work faster with Arch and FennECS because they're not deleting entity. Because of that special property called DeletesEntityOnLastComponentDeletion is required to be implemented in each context.
TinyECS - Deleting entities during lock causing stackoverflow on merge so get rid of lock during deletions.