Home

Awesome

toml-bench

deps

Which toml package to use in python?

See also: toml-lang and PEP 680

Report

Version

The verions of the packages tested in this report.

Version
<a target="_blank" href="https://github.com/uiri/toml">toml</a>0.10.2
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>2.0.1; tomli_w: 1.0.0
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>0.12.5
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>0.11.0
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>0.3.1
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>(Python 3.12.2)

Dumping None value

How the package dumps None value in python

Literally <package>.dumps(None)

Dumped value or error
<a target="_blank" href="https://github.com/uiri/toml">toml</a>'NoneType' object is not iterable
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>'NoneType' object has no attribute 'items'
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>Expecting Mapping or TOML Container, <class 'NoneType'> given
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>"null"<br />---<br />rtoml v0.11+ supports dumping None to a desired string:<br />rtoml.dumps(data, none_value='@None'):<br />"@None"
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>'NoneType' object has no attribute 'items'
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>module 'tomllib' has no attribute 'dumps'

Dumping key-None pair

How the package dumps key-value pair with value None

Literally <package>.dumps({"key": None})

Dumped value or error
<a target="_blank" href="https://github.com/uiri/toml">toml</a>
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>Object of type <class 'NoneType'> is not TOML serializable
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>Invalid type <class 'NoneType'>
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>key = "null"<br /><br />---<br />rtoml v0.11+ supports dumping None to a desired string:<br />rtoml.dumps(data, none_value='@None'):<br />key = "@None"<br />
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>TOML cannot encode None
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>module 'tomllib' has no attribute 'dumps'

Dumping list with None value

How the package dumps a list with None value in it.

Literally <package>.dumps({"key": [1, 2, 3, None, 5]})

Dumped value or error
<a target="_blank" href="https://github.com/uiri/toml">toml</a>key = [ 1, 2, 3, "None", 5,]<br />
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>Object of type <class 'NoneType'> is not TOML serializable
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>Invalid type <class 'NoneType'>
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>key = [1, 2, 3, "null", 5]<br /><br />---<br />rtoml v0.11+ supports dumping None to a desired string:<br />rtoml.dumps(data, none_value='@None'):<br />key = [1, 2, 3, "@None", 5]<br />
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>bad type '<class 'NoneType'>' for dump_value
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>module 'tomllib' has no attribute 'dumps'

Loading None-like values

How the package loads None-like value in string

Literally <package>.loads('v1 = "null" v2 = "None"')

Loaded as
<a target="_blank" href="https://github.com/uiri/toml">toml</a>{'v1': 'null', 'v2': 'None'}
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>module 'tomli_w' has no attribute 'loads'
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>{'v1': 'null', 'v2': 'None'}
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>{'v1': 'null', 'v2': 'None'}<br />---<br />rtoml v0.11+ supports loading custom None values:<br />rtoml.loads(data, none_value='None'):<br />{'v1': 'null', 'v2': None}<br />rtoml.loads(data, none_value='null'):<br />{'v1': None, 'v2': 'None'}
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>{'v1': 'null', 'v2': 'None'}
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>{'v1': 'null', 'v2': 'None'}

Dumping a heterogenous array

How the package dumps a python dictionary with a heterogenous array.

Literally <package>.dumps({"v": [1, 1.2, True, "string"]})

Dumped value or error
<a target="_blank" href="https://github.com/uiri/toml">toml</a>v = [ 1, 1.2, true, "string",]<br />
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>v = [<br />    1,<br />    1.2,<br />    true,<br />    "string",<br />]<br />
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>v = [1, 1.2, true, "string"]<br />
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>v = [1, 1.2, true, "string"]<br />
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>v = [1, 1.2, true, 'string']<br />
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>Dumping not supported

Loading a heterogenous array

How the package loads a toml string with a heterogenous array.

Literally <package>.loads('v = [1, 1.2, True, "string"]')

Loaded as
<a target="_blank" href="https://github.com/uiri/toml">toml</a>Not a homogeneous array (line 2 column 1 char 1)
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>{'v': [1, 1.2, True, 'string']}
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>{'v': [1, 1.2, True, 'string']}
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>{'v': [1, 1.2, True, 'string']}
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>{'v': [1, 1.2, True, 'string']}
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>{'v': [1, 1.2, True, 'string']}

Dumping a nested array

How the package dumps a python dictionary with a nested array.

Literally <package>.dumps({"v": [[1], [1, 2]]})

Dumped value or error
<a target="_blank" href="https://github.com/uiri/toml">toml</a>v = [ [ 1,], [ 1, 2,],]<br />
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>v = [<br />    [<br />        1,<br />    ],<br />    [<br />        1,<br />        2,<br />    ],<br />]<br />
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>v = [[1], [1, 2]]<br />
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>v = [[1], [1, 2]]<br />
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>v = [[1], [1, 2]]<br />
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>Dumping not supported

Loading a nested array

How the package loads a toml string with a nested array.

Literally <package>.loads('v = [[1], [1, 2]]')

Loaded as
<a target="_blank" href="https://github.com/uiri/toml">toml</a>{'v': [[1], [1, 2]]}
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>{'v': [[1], [1, 2]]}
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>{'v': [[1], [1, 2]]}
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>{'v': [[1], [1, 2]]}
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>{'v': [[1], [1, 2]]}
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>{'v': [[1], [1, 2]]}

Dumping keeps order of keys?

Whether the package preserves the order of the keys while dumps a python dictionary.

Thus, whether <package>.dumps({"c": 1, "a": 2, "b": 3}) yields a string like c = 1\na = 2\nb = 3\n.

Order kept?
<a target="_blank" href="https://github.com/uiri/toml">toml</a>Kept
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>Kept
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>Kept
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>Kept
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>Kept
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>Dumping not supported

Loading keeps order of keys?

Whether the package preserves the order of the keys while loads a TOML string.

Thus, whether <package>.loads('c = 1\na = 2\nb = 3\n') yields a dictionary with keys in the order of ['c', 'a', 'b'].

Order kept?
<a target="_blank" href="https://github.com/uiri/toml">toml</a>Kept
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>Kept
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>Kept
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>Kept
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>Kept
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>Kept

Dumping unicode

How the package dumps Unicode in python

Literally, <package>.dumps({"你好": "世界"})

Dumped value
<a target="_blank" href="https://github.com/uiri/toml">toml</a>"你好" = "世界"<br />
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>"你好" = "世界"<br />
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>"你好" = "世界"<br />
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>"你好" = "世界"<br />
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>'你好' = '世界'<br />
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>Dumping not supported

Loaded unicode

How the package loads a file with unicode.

The file was created by:

## Create a file with unicode content
with open(self.datafile, "w", encoding="utf-8") as f:
    f.write('"你好" = "世界"\n')

## Use `<package>.load()` to load the file
with open(self.datafile, "r", encoding="utf-8") as f:
    loaded = <package>.load(f)
Loaded as
<a target="_blank" href="https://github.com/uiri/toml">toml</a>{'你好': '世界'}
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>File must be opened in binary mode, e.g. use open('foo.toml', 'rb')<br />When loaded with rb:<br />{'你好': '世界'}
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>{'你好': '世界'}
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>{'你好': '世界'}
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>{'你好': '世界'}
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>File must be opened in binary mode, e.g. use open('foo.toml', 'rb')<br />When loaded with rb:<br />{'你好': '世界'}

Compliance with valid tests in toml-test

Test the compliance with the standard test suites for valid toml files here:

https://github.com/BurntSushi/toml-test/

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

Result (toml-test v1.5.0)
<a target="_blank" href="https://github.com/uiri/toml">toml</a>spec/array-0.toml Not a homogeneous array (line 8 column 1 char 261)<br />spec/keys-4.toml Found invalid character in key name: 'c'. Try quoting the key name. (line 2 column 8 char 57)<br />spec/local-time-0.toml Parsed as unexpected data.<br />datetime/no-seconds.toml invalid literal for int() with base 0: '13:37' (line 2 column 1 char 46)<br />datetime/local-time.toml Parsed as unexpected data.<br />datetime/datetime.toml Parsed as unexpected data.<br />comment/tricky.toml Parsed as unexpected data.<br />key/dotted-1.toml Parsed as unexpected data.<br />key/unicode.toml Found invalid character in key name: '‍'. Try quoting the key name. (line 5 column 2 char 67)<br />key/dotted-2.toml Found invalid character in key name: '"'. Try quoting the key name. (line 7 column 11 char 166)<br />key/quoted-unicode.toml Duplicate keys! (line 3 column 1 char 19)<br />key/dotted-empty.toml Duplicate keys! (line 2 column 1 char 17)<br />key/escapes.toml Parsed as unexpected data.<br />table/empty-name.toml Can't have a keygroup with an empty name (line 1 column 1 char 0)<br />string/raw-multiline.toml Unbalanced quotes (line 20 column 50 char 532)<br />string/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28)<br />string/hex-escape.toml Reserved escape sequence used (line 3 column 1 char 35)<br />string/escape-esc.toml Reserved escape sequence used (line 1 column 1 char 0)<br />string/multiline-quotes.toml Unterminated string found. Reached end of file. (line 27 column 1 char 664)<br />float/zero.toml Weirdness with leading zeroes or underscores in your number. (line 4 column 1 char 47)<br />array/mixed-int-string.toml Not a homogeneous array (line 1 column 1 char 0)<br />array/nested-double.toml Not a homogeneous array (line 1 column 1 char 0)<br />array/string-with-comma-2.toml string index out of range<br />array/mixed-int-float.toml Not a homogeneous array (line 1 column 1 char 0)<br />array/mixed-string-table.toml list index out of range<br />array/mixed-int-array.toml Not a homogeneous array (line 1 column 1 char 0)<br />inline-table/multiline.toml Invalid inline table value encountered (line 1 column 1 char 0)<br />inline-table/key-dotted-1.toml Parsed as unexpected data.<br />inline-table/key-dotted-5.toml Not a homogeneous array (line 2 column 1 char 20)<br />inline-table/newline.toml Key name found without value. Reached end of line. (line 5 column 2 char 98)<br />157/187 (83.96%) passed
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>datetime/no-seconds.toml Expected newline or end of document after a statement (at line 2, column 23)<br />key/unicode.toml Invalid statement (at line 3, column 1)<br />string/hex-escape.toml Unescaped '' in a string (at line 3, column 22)<br />string/escape-esc.toml Unescaped '' in a string (at line 1, column 10)<br />inline-table/newline.toml Invalid initial character for a key part (at line 3, column 21)<br />182/187 (97.33%) passed
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>datetime/no-seconds.toml Invalid number at line 2 col 25<br />key/unicode.toml Empty key at line 3 col 0<br />string/hex-escape.toml Invalid character 'x' in string at line 3 col 20<br />string/escape-esc.toml Invalid character 'e' in string at line 1 col 8<br />inline-table/newline.toml Empty key at line 3 col 20<br />182/187 (97.33%) passed
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>spec/table-9.toml duplicate key: apple for key fruit at line 8 column 1<br />datetime/no-seconds.toml expected a colon, found a newline at line 2 column 26<br />key/unicode.toml unexpected character found: \u{20ac} at line 3 column 1<br />table/array-within-dotted.toml duplicate key: apple for key fruit at line 4 column 1<br />string/hex-escape.toml invalid escape character in string: x at line 3 column 21<br />string/escape-esc.toml invalid escape character in string: e at line 1 column 9<br />inline-table/newline.toml expected a table key, found a newline at line 3 column 21<br />180/187 (96.26%) passed
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>spec/string-4.toml Didn't find expected newline (line 7, column 62)<br />spec/string-7.toml Didn't find expected newline (line 7, column 50)<br />datetime/no-seconds.toml can't parse type (line 2, column 20)<br />datetime/milliseconds.toml Didn't find expected newline (line 2, column 27)<br />datetime/datetime.toml Didn't find expected newline (line 4, column 18)<br />comment/after-literal-no-ws.toml can't parse type (line 1, column 4)<br />comment/tricky.toml can't parse type (line 11, column 7)<br />key/unicode.toml '€' cannot begin key (line 3, column 0)<br />string/raw-multiline.toml Didn't find expected newline (line 22, column 3)<br />string/hex-escape.toml \x not a valid escape (line 3, column 43)<br />string/escape-esc.toml \e not a valid escape (line 1, column 33)<br />string/multiline-quotes.toml Didn't find expected newline (line 4, column 26)<br />inline-table/newline.toml ' ' cannot begin key (line 3, column 20)<br />174/187 (93.05%) passed
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>datetime/no-seconds.toml Expected newline or end of document after a statement (at line 2, column 23)<br />key/unicode.toml Invalid statement (at line 3, column 1)<br />string/hex-escape.toml Unescaped '' in a string (at line 3, column 22)<br />string/escape-esc.toml Unescaped '' in a string (at line 1, column 10)<br />inline-table/newline.toml Invalid initial character for a key part (at line 3, column 21)<br />182/187 (97.33%) passed

Compliance with invalid tests in toml-test

Test the compliance with the standard test suites for invalid toml files here:

https://github.com/BurntSushi/toml-test/

Result (toml-test v1.5.0)
<a target="_blank" href="https://github.com/uiri/toml">toml</a>Not OK: integer/double-sign-plus.toml incorrectly parsed.<br />Not OK: integer/us-after-bin.toml incorrectly parsed.<br />Not OK: integer/double-sign-nex.toml incorrectly parsed.<br />Not OK: integer/us-after-hex.toml incorrectly parsed.<br />Not OK: integer/us-after-oct.toml incorrectly parsed.<br />Not OK: spec/inline-table-2-0.toml incorrectly parsed.<br />Not OK: datetime/offset-overflow-minute.toml incorrectly parsed.<br />Not OK: datetime/offset-overflow-hour.toml incorrectly parsed.<br />Not OK: control/comment-del.toml incorrectly parsed.<br />Not OK: control/string-del.toml incorrectly parsed.<br />Not OK: 63 more items incorrectly parsed.<br />298/371 (80.32%) passed
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18)<br /> 371/371 (100%) passed
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>Not OK: control/comment-cr.toml incorrectly parsed.<br />Not OK: control/multi-cr.toml incorrectly parsed.<br />Not OK: control/rawmulti-cd.toml incorrectly parsed.<br />Not OK: control/bare-cr.toml incorrectly parsed.<br />Not OK: table/append-with-dotted-keys-1.toml incorrectly parsed.<br />Not OK: table/overwrite-array-in-parent.toml incorrectly parsed.<br />Not OK: table/append-to-array-with-dotted-keys.toml incorrectly parsed.<br />Not OK: table/append-with-dotted-keys-2.toml incorrectly parsed.<br />Not OK: array/extend-defined-aot.toml incorrectly parsed.<br />Not OK: inline-table/overwrite-09.toml incorrectly parsed.<br />361/371 (97.30%) passed
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>Not OK: integer/positive-hex.toml incorrectly parsed.<br />Not OK: integer/positive-bin.toml incorrectly parsed.<br />Not OK: integer/positive-oct.toml incorrectly parsed.<br />Not OK: datetime/offset-overflow-minute.toml incorrectly parsed.<br />Not OK: datetime/offset-overflow-hour.toml incorrectly parsed.<br />Not OK: control/comment-del.toml incorrectly parsed.<br />Not OK: control/comment-cr.toml incorrectly parsed.<br />Not OK: control/multi-cr.toml incorrectly parsed.<br />Not OK: control/rawmulti-cd.toml incorrectly parsed.<br />Not OK: control/bare-cr.toml incorrectly parsed.<br />361/371 (97.30%) passed
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>Not OK: spec/inline-table-2-0.toml incorrectly parsed.<br />Not OK: spec/table-9-1.toml incorrectly parsed.<br />Not OK: spec/table-9-0.toml incorrectly parsed.<br />Not OK: datetime/offset-overflow-minute.toml incorrectly parsed.<br />Not OK: control/comment-del.toml incorrectly parsed.<br />Not OK: control/comment-lf.toml incorrectly parsed.<br />Not OK: control/comment-null.toml incorrectly parsed.<br />Not OK: control/comment-ff.toml incorrectly parsed.<br />Not OK: control/comment-cr.toml incorrectly parsed.<br />Not OK: control/multi-cr.toml incorrectly parsed.<br />Not OK: 14 more items incorrectly parsed.<br />347/371 (93.53%) passed
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>OK: inline-table/linebreak-1.toml Unclosed inline table (at line 3, column 18)<br /> 371/371 (100%) passed

Compliance with valid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for valid toml files here:

https://github.com/python/cpython/tree/3.11/Lib/test/test_tomllib/data/valid

The tests come up with a JSON counterpart that can be used to valid whether loading the toml file yields the same result as the JSON counterpart.

Result (cpython tag 3.12.4)
<a target="_blank" href="https://github.com/uiri/toml">toml</a>apostrophes-in-literal-string.toml Unbalanced quotes (line 1 column 50 char 49)<br />five-quotes.toml Unterminated string found. Reached end of file. (line 7 column 1 char 97)<br />dates-and-times/datetimes.toml Parsed as unexpected data.<br />multiline-basic-str/ends-in-whitespace-escape.toml Reserved escape sequence used (line 6 column 1 char 28)<br />8/12 (66.67%) passed
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>OK, 12/12 (100%) passed
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>OK, 12/12 (100%) passed
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>OK, 12/12 (100%) passed
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>apostrophes-in-literal-string.toml Didn't find expected newline (line 3, column 3)<br />five-quotes.toml Didn't find expected newline (line 3, column 3)<br />dates-and-times/datetimes.toml Didn't find expected newline (line 1, column 19)<br />9/12 (75.00%) passed
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>OK, 12/12 (100%) passed

Compliance with invalid tests in python tomllib test data

Test the compliance with python tomllib test data (since python 3.11) for invalid toml files here:

https://github.com/python/cpython/tree/main/Lib/test/test_tomllib/data/invalid

Result (cpython tag 3.12.4)
<a target="_blank" href="https://github.com/uiri/toml">toml</a>Not OK: invalid-comment-char.toml incorrectly parsed.<br />Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.<br />Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.<br />Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.<br />Not OK: array/unclosed-empty.toml incorrectly parsed.<br />Not OK: array/file-end-after-val.toml incorrectly parsed.<br />Not OK: array/unclosed-after-item.toml incorrectly parsed.<br />Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed.<br />Not OK: inline-table/unclosed-empty.toml incorrectly parsed.<br />41/50 (82.00%) passed
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21)<br /> 50/50 (100%) passed
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>Not OK: array-of-tables/overwrite-array-in-parent.toml incorrectly parsed.<br />Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.<br />Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.<br />Not OK: dotted-keys/extend-defined-aot.toml incorrectly parsed.<br />Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.<br />Not OK: inline-table/override-val-in-table.toml incorrectly parsed.<br />44/50 (88.00%) passed
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.<br />49/50 (98.00%) passed
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>Not OK: non-scalar-escaped.toml incorrectly parsed.<br />Not OK: invalid-comment-char.toml incorrectly parsed.<br />Not OK: table/redefine-2.toml incorrectly parsed.<br />Not OK: table/redefine-1.toml incorrectly parsed.<br />Not OK: multiline-basic-str/carriage-return.toml incorrectly parsed.<br />Not OK: dotted-keys/extend-defined-table.toml incorrectly parsed.<br />Not OK: dotted-keys/extend-defined-table-with-subtable.toml incorrectly parsed.<br />Not OK: inline-table/overwrite-value-in-inner-table.toml incorrectly parsed.<br />Not OK: inline-table/override-val-with-table.toml incorrectly parsed.<br />41/50 (82.00%) passed
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>OK: inline-table/overwrite-implicitly.toml Cannot overwrite a value (at line 1, column 21)<br /> 50/50 (100%) passed

Running speed with data provided by rtoml

Test the speed of loading and dumping the loaded using data provided by rtoml

https://github.com/samuelcolvin/rtoml/raw/main/tests/data.toml

Loading speedDumping speed
<a target="_blank" href="https://github.com/uiri/toml">toml</a>Excluded (heterogeneous arrays not supported)Excluded (heterogeneous arrays not supported)
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>2.14s (5000 iterations)0.73s (5000 iterations)
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>39.78s (5000 iterations)0.98s (5000 iterations)
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>0.37s (5000 iterations)0.08s (5000 iterations)
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>4.99s (5000 iterations)1.87s (5000 iterations)
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>2.04s (5000 iterations)Dumping not supported

Running speed with data provided by tomli

Test the speed of loading and dumping the loaded using data provided by tomli

https://github.com/hukkin/tomli/raw/master/benchmark/data.toml

Loading speedDumping speed
<a target="_blank" href="https://github.com/uiri/toml">toml</a>Excluded (heterogeneous arrays not supported)Excluded (heterogeneous arrays not supported)
<a target="_blank" href="https://github.com/hukkin/tomli">tomli/tomli_w</a>1.41s (5000 iterations)0.46s (5000 iterations)
<a target="_blank" href="https://github.com/sdispater/tomlkit">tomlkit</a>24.55s (5000 iterations)0.51s (5000 iterations)
<a target="_blank" href="https://github.com/samuelcolvin/rtoml">rtoml</a>0.32s (5000 iterations)0.16s (5000 iterations)
<a target="_blank" href="https://github.com/alethiophile/qtoml">qtoml</a>3.63s (5000 iterations)1.25s (5000 iterations)
<a target="_blank" href="https://docs.python.org/3/library/tomllib.html">tomllib</a>1.44s (5000 iterations)Dumping not supported

Other reports

Run your own report

Install

pip install -U toml-bench

Generate your own report

toml-bench

Use a different data directory than the default one

toml-bench --datadir /tmp/toml-bench

Write the report to a markdown file

toml-bench --report ./README.md

Test with a different version of compliance set (BurntSushi/toml-test)

toml-bench --comver 1.0.0

Use a different number of iterations in speed tests

toml-bench --iter 5000

Test with different versions of packages

git clone https://github.com/pwwang/toml-bench.git
cd toml-bench
# See https://python-poetry.org/docs/cli/#add
# for how to specify a version constraint
poetry add "tomli=2.0.0"
poetry update
poetry install
poetry run toml-bench