Home

Awesome

CircleCI

js-framework-benchmark

This is a simple benchmark for several javascript frameworks. The benchmarks creates a large table with randomized entries and measures the time for various operations including rendering duration.

Screenshot

About the benchmarks

The following operations are benchmarked for each framework:

For all benchmarks the duration is measured including rendering time. You can read some details on this article. The results of this benchmark is outlined on my blog (round 1, round 2, round 3, round 4, round 5, round 6, round 7 and round 8).

Snapshot of the results

Official results are posted on the blog mentioned above. The current snapshot that may not have the same quality (i.e. results might be for mixed browser versions, number of runs per benchmark may vary) can be seen here Results

How to get started - building and running

There are currently ~60 framework entries in this repository. Installing (and maintaining) those can be challenging, but here are simplified instructions how to get started.

1. Prerequisites

Have node.js (>=7.6) installed. If you want to do yourself a favour use nvm for that and install yarn. The benchmark has been tested with node 8.4.0. For some frameworks you'll also need java (>=8, e.g. openjdk-8-jre on ubuntu). Please make sure that the following command work before trying to build:

> npm
npm -version
5.0.0
> node --version
v8.0.0
> echo %JAVA_HOME% / echo $JAVA_HOME
> java -version
java version "1.8.0_131" ...
> javac -version
javac 1.8.0_131

2. Start installing

As stated above building and running the benchmarks for all frameworks can be challenging, thus we start step by step...

Install global dependencies This installs just a few top level dependencies for the building the frameworks and a http-server.

npm install

We start the http-server in the root directory

npm start

Verify that the http-server works: Try to open http://localhost:8080/index.html. If you see something like that you're on the right track: Index.html

Now open a new terminal window and keep http-server running in background.

3. Building and running a single framework

We now try to build the first framework. Go to the vanillajs reference implementation

cd frameworks/keyed/vanillajs

and install the dependencies

npm install

and build the framework

npm run build-prod

There should be no build errors and we can open the framework in the browser: http://localhost:8080/frameworks/keyed/vanillajs/

Some frameworks like binding.scala or ember can't be opened that way, because they need a 'dist' or 'target/web/stage' or something in the URL. You can find out the correct URL in the index.html you've opened before or take a look whether there's a customURL property under js-framework-benchmark in the package.json that represents the url.

Open the browser console and click a bit on the buttons and you should see some measurements printed on the console. First Run

What is printed on the console is not what is actually measured by the automated benchmark driver. The benchmark driver extracts events from chrome's timeline to calculate the duration for the operations. What get's printed on the console above is an approximation of the actual duration which is pretty close to the actual duration.

Optional 3.1: Contributing a new implementation

For contributions it is basically sufficient to create a new directory for your framework that supports npm install and npm run build-prod and can be then opened in the browser. All other steps are optional. Let's simulate that by copying vanillajs.

cd ../frameworks/keyed
cp -r vanillajs super-vanillajs
cd super-vanillajs

Then we edit super-vanillajs/index.html to have a correct index.html:

<title>Super-VanillaJS-"keyed"</title>
...
                    <h1>Super-VanillaJS-"keyed"</h1>

In most cases you'll need npm install and npm run build-prod and then check whether it works in the browser on http://localhost:8080/frameworks/keyed/super-vanillajs/.

(Of course in reality you'd rather throw out the javascript source files and use your framework there instead of only changing the html file.)

4. Running a single framework with the automated benchmark driver

As mentioned above the benchmark uses an automated benchmark driver using chromedriver to measure the duration for each operation using chrome's timeline. Here are the steps to run is for a single framework:

cd ../../..
cd webdriver-ts

and install the dependencies

npm install

and build the benchmark driver

npm run build-prod

now run the benchmark driver for the vanillajs-keyed framework:

npm run selenium -- --count 3 --framework vanillajs-keyed

Just lean back and watch chrome run the benchmarks. It runs each benchmark 3 times for the vanillajs-keyed framework.

You should keep the chrome window visible since otherwise it seems like paint events can be skipped leading to wrong results. On the terminal will appear various log statements.

The results for that run will be saved in the webdriver-ts/results directory. We can take a look at the results of a single result:

cat results/vanillajs-keyed_01_run1k.json
{"framework":"vanillajs-keyed","benchmark":"01_run1k","type":"cpu","min":135.532,"max":154.821,"mean":143.79166666666666,"median":141.022,"geometricMean":143.56641695989177,"standardDeviation":8.114582360718808,"values":[154.821,135.532,141.022]}

As you can see the mean duration for create 1000 rows was 144 msecs.

6. Building the result table

Install libraries

cd ..
cd webdriver-ts-results
npm install
cd ..
cd webdriver-ts

In the webdriver-ts directory issue the follwing command:

npm run results

Now a result table should have been created which can be opened on http://localhost:8080/webdriver-ts-results/table.html. There's nothing in table except for the column vanillajs-keyed at the right end of the first table. First Run Results

Optional 6.1 Adding your new implementation to the results table.

(Notice: Updating common.ts is no longer necessary, super-vanillajs is visible in the result table)

Your package.json must include some information for the benchmark. Since you copied it, the important section is already there:

  ...
  "js-framework-benchmark": {
    "frameworkVersion": ""
  },
  ...

This one is a bit exceptional since vanillajs has no version. If you use a normal framework like react it carries a version information. For most frameworks you'll add a dependency to your framework in package.json. The benchmark can automatically determine the correct version information from package.json and package-lock.json if you specify the package name like that:

  "js-framework-benchmark": {
    "frameworkVersionFromPackage": "react"
  },

Now the benchmark will fetch the installed react version from package-lock.json in the react directory and use that version number to compute the correct version string. If your library has multiple important packages like react + redux you can put them separated with a colon there like "react:redux". If you don't pull your framework from npm you can hardcode a version like "frameworkVersion": "0.0.1". The other important, but optional properties for js-framework-benchmark are shown in the following example:

"customURL": "/target/web/stage",
"useShadowRoot": true

You can set an optional different URL if needed or specify that your DOM uses a shadow root.

Optional 6.2 Updating the index.html file

With

npm run index

you include Super-VanillaJS-keyed in http://localhost:8080/index.html

Optional 7. Building and running the benchmarks for all frameworks

This is not for the faint at heart. You can build all frameworks simply by issuing

cd ..
npm run build-prod

After downloading the whole internet it starts building it. Basically there should be no errors during the build, but I can't guarantee that the dependencies won't break. You can now run selenium for all frameworks by invoking npm run selenium in the root directory.

After that you can check all results in http://localhost:8080/webdriver-ts/table.html.

Tips and tricks

How to contribute

Contributions are very welcome. Please use the following rules:

This work is derived from a benchmark that Richard Ayotte published on https://gist.github.com/RichAyotte/a7b8780341d5e75beca7 and adds more framework and more operations. Thanks for the great work.

Thanks to Baptiste Augrain for making the benchmarks more sophisticated and adding frameworks.