Awesome

README

This contains the metadata for comprehensive benchmarks to analyze potential compiler regressions

Dependencies

This requires pycpuinfo for the diagnostic CPU information.

pip install py-cpuinfo --user

Basic Use

The diagnostics can then be exported using the flatbench and metadata.py scripts:

# this is the directory to where the rust benchmark data is
repository="../rust-lexical"
target="${repository}/lexical-benchmark/target"
python flatbench.py --target "${target}"
python metadata.py --target "${target}"
python plot.py --repository "${repository}"

Results

The data will be output in the results directory with each result in a directory containing the shorthash of the file. For example, for commit 9b7cac6, the results are stored under results/9b7cac6.

An example of the profiling info is below. This contains the mean benchmark, the upper and lower bound, and the confidence, all grouped by the bench groups and the benchmark name. The results are in nanoseconds.

Raw Benchmarks

{
  "mean": {
    "canada": {
      "parse_canada_core": 1299484.0578760111,
      "parse_canada_lexical": 1174099.1108035871
    },
    "contrived_denormal10": {
      "parse_f64_core": 9.547369661223229,
      "parse_f64_lexical": 9.024892765168747
    }
  },
  "lower": {
    "canada": {
      "parse_canada_core": 1294971.510482425,
      "parse_canada_lexical": 1166243.1561483613
    },
    "contrived_denormal10": {
      "parse_f64_core": 9.526261257980039,
      "parse_f64_lexical": 8.994481446498016
    }
  },
  "upper": {
    "canada": {
      "parse_canada_core": 1304157.4730729826,
      "parse_canada_lexical": 1182712.22545152
    },
    "contrived_denormal10": {
      "parse_f64_core": 9.570402076208493,
      "parse_f64_lexical": 9.055119922622316
    }
  },
  "confidence": {
    "canada": {
      "parse_canada_core": 0.95,
      "parse_canada_lexical": 0.95
    },
    "contrived_denormal10": {
      "parse_f64_core": 0.95,
      "parse_f64_lexical": 0.95
    }
  },
  "std_dev": {
    "canada": {
      "parse_canada_core": 23543.600828805113,
      "parse_canada_lexical": 41980.392749475715
    },
    "contrived_denormal10": {
      "parse_f64_core": 0.1127078393110621,
      "parse_f64_lexical": 0.1549991088288157
    }
  }
}

An example of the metadata is:

{
  "commit": "9b7cac6",
  "platform": {
    "processor": "Intel64 Family 6 Model 183 Stepping 1, GenuineIntel",
    "machine": "AMD64",
    "version": "10.0.22631",
    "release": "10",
    "system": "Windows"
  },
  "cpuinfo": {
    "cpuinfo_version": [
      9,
      0,
      0
    ],
    "cpuinfo_version_string": "9.0.0",
    "arch": "X86_64",
    "bits": 64,
    "count": 28,
    "arch_string_raw": "AMD64",
    "vendor_id_raw": "GenuineIntel",
    "brand_raw": "Intel(R) Core(TM) i7-14700K",
    "hz_actual_friendly": "3.4000 GHz",
    "hz_actual": [
      3400000000,
      0
    ],
    "l2_cache_size": 29360128,
    "stepping": 1,
    "model": 183,
    "family": 6,
    "l3_cache_size": 34603008,
    "hz_advertised_friendly": "3.4180 GHz",
    "hz_advertised": [
      3418000000,
      0
    ],
    "flags": [
      "3dnow"
    ],
    "l2_cache_line_size": 2048,
    "l2_cache_associativity": 7
  },
  "rust": {
    "version": "1.81.0 (eeb90cda1 2024-09-04)",
    "binary": " rustc",
    "commit-hash": " eeb90cda1969383f56a2637cbd3037bdf598841c",
    "commit-date": " 2024-09-04",
    "host": " x86_64-pc-windows-msvc",
    "release": " 1.81.0",
    "LLVM version": " 18.1.7"
  }
}

Plots

The plots are each will be in the plot subdirectory, for example, under results/edceaca/plot for commit edceaca. Examples from this benchmark include:

JSON Simple - Parse Integer

Earth - Parse Float

JSON Simple - Write Integer

Random Big Integers - Write Float

Latest Results

The latest result is for commit edceaca.

Benchmarks — Parse Integer

Json_Random:

Json_Simple:

Random_Large:

Random_Large_Signed:

Random_Simple:

Random_Simple_Signed:

Random_Uniform:

Benchmarks — Parse Float

Canada:

Contrived Denormal10:

Contrived Denormal100:

Contrived Denormal1600:

Contrived Denormal20:

Contrived Denormal200:

Contrived Denormal30:

Contrived Denormal3200:

Contrived Denormal40:

Contrived Denormal400:

Contrived Denormal50:

Contrived Denormal6400:

Contrived Denormal800:

Contrived Disguised:

Contrived Fast:

Contrived Halfway:

Contrived Large10:

Contrived Large100:

Contrived Large1600:

Contrived Large20:

Contrived Large200:

Contrived Large30:

Contrived Large3200:

Contrived Large40:

Contrived Large400:

Contrived Large50:

Contrived Large6400:

Contrived Large800:

Contrived Moderate:

Earth:

Json:

Mesh:

Random Big Int Dot Int:

Random Big Ints:

Random Int E Int:

Random One Over Rand32:

Random Simple Int32:

Random Simple Int64:

Random Simple Uniform32:

Benchmarks — Write Integer

Json Chain Random:

Json Random:

Json Simple:

Random Large:

Random Large Signed:

Random Simple:

Random Simple Signed:

Random Uniform:

Benchmarks — Write Float

Json:

Random Big Int Dot Int:

Random Big Ints:

Random One Over Rand32:

Random Simple Int32:

Random Simple Int64:

Random Simple Uniform32:

Random Special: