Awesome

jackson-jq

Pure Java jq Implementation for Jackson JSON Processor

Usage

First, you need Java 8 or later.

If you use Maven, add the following snippet to the <dependencies> section of your POM. For instructions for other build tools (Gradle, etc.), visit jackson-jq on search.maven.org.

<dependency>
	<groupId>net.thisptr</groupId>
	<artifactId>jackson-jq</artifactId>
	<version>1.2.0</version>
</dependency>

See jackson-jq/src/test/java/examples/Usage.java for the API usage.

Using a jackson-jq command line tool

To test a query quickly, we provide jackson-jq CLI.

Please note that jackson-jq is a Java library and the CLI is provided solely for debugging/testing purpose (and not for production). The command-line options might change without notice.

$ curl -LO https://repo1.maven.org/maven2/net/thisptr/jackson-jq-cli/1.2.0/jackson-jq-cli-1.2.0.jar

$ java -jar jackson-jq-cli-1.2.0.jar --help
usage: jackson-jq [OPTIONS...] QUERY
 -c,--compact      compact instead of pretty-printed output
 -h,--help         print this message
    --jq <arg>     specify jq version
 -n,--null-input   use `null` as the single input value
 -r,--raw          output raw strings, not JSON texts

$ java -jar jackson-jq-cli-1.2.0.jar '.foo'
{"foo": 42}
42

To test a query with a specific jq version,

$ java -jar jackson-jq-cli-1.2.0.jar --jq 1.5 'join("-")'
["1", 2]
jq: error: string ("-") and number (2) cannot be added

$ java -jar jackson-jq-cli-1.2.0.jar --jq 1.6 'join("-")' # jq-1.6 can join any values, not only strings
["1", 2]
"1-2"

Homebrew (or Linuxbrew) users can alternatively run brew tap eiiches/jackson-jq && brew install jackson-jq to install the CLI. jackson-jq will be available on your $PATH.

Branches and versioning

There are currently two development branches.

develop/1.x: This branch (you are viewing), which is currently under development for the future 1.0 release. You can find preview releases at Releases page (tags: 1.0.0-preview.yyyyMMdd). Although the API is not stable yet, I recommend new users to use these releases insetad of 0.x versions, because these releases have more features, better compatibility, and better performance.
develop/0.x: The development branch for 0.x versions. Features that need breaking API changes will no longer be added. Go to Releases and find the latest 0.x.y version.

PRs can be sent to any of the develop/* branches. The patch will be ported to the other branch(es) if necessary.

We use Semantic Versioning 2.0.0 for Java API versioning, 1.0.0 onwards. A jq behavior fix (even if it may possibly affect users) will not be considered a major change if the fix is to make the bahavior compatible with ./jq; these kind of incompatible changes are documented in the release note.

If you get different results between ./jq and jackson-jq, please file an issue. That is a bug on jackson-jq side.

Implementation Status

jackson-jq aims to be a compatible jq implementation. However, not every feature is available; some are intentionally omitted because thay are not relevant as a Java library; some may be incomplete, have bugs or are yet to be implemented.

List of Features

<details> <summary>Click to see the list</summary> <br />

This table illustrates which features (picked from jq-1.5 manual) are supported and which are not in jackson-jq. We try to keep this list accurate and up to date. If you find something is missing or wrong, please file an issue.

Language Features / Functions	jackson-jq
Basic filters	○
• `.`	○
• `.foo`, `.foo.bar`	○
• `.foo?`	○
• `.[<string>]`, `.[2]`, `.[10:15]`	○
• `.[]`	○
• `.[]?`	○
• `,`	○
• `ǀ`	○
Types and Values	○
• Array construction - `[]`	○
• Objects - `{}`	○<sup>*4</sup>
Builtin operators and functions	○
• Addition - `+`	○
• Subtraction - `-`	○
• Multiplication, division, modulo - `*`, `/`, and `%`	○<sup>*5</sup>
• `length`	○
• `keys`, `keys_unsorted`	○
• `has(key)`	○
• `in`	○
• `path(path_expression)`	○<sup>*7</sup>
• `del(path_expression)`	○
• `to_entries`, `from_entries`, `with_entries`	○
• `select(boolean_expression)`	○
• `arrays`, `objects`, `iterables`, `booleans`, `numbers`, `normals`, `finites`, `strings`, `nulls`, `values`, `scalars`	○
• `empty`	○
• `error(message)`	○
• `$__loc__`	×
• `map(x)`, `map_values(x)`	○
• `paths`, `paths(node_filter)`, `leaf_paths`	○
• `add`	○
• `any`, `any(condition)`, `any(generator; condition)`	○
• `all`, `all(condition)`, `all(generator; condition)`	○
• `flatten`, `flatten(depth)`	○
• `range(upto)`, `range(from;upto)` `range(from;upto;by)`	○
• `floor`	○
• `sqrt`	○
• `tonumber`	○
• `tostring`	○
• `type`	○
• `infinite`, `nan`, `isinfinite`, `isnan`, `isfinite`, `isnormal`	○
• `sort, sort_by(path_expression)`	○
• `group_by(path_expression)`	○
• `min`, `max`, `min_by(path_exp)`, `max_by(path_exp)`	○
• `unique`, `unique_by(path_exp)`	○
• `reverse`	○
• `contains(element)`	○
• `indices(s)`	○<sup>*9</sup>
• `index(s)`, `rindex(s)`	○
• `inside`	○
• `startswith(str)`	○
• `endswith(str)`	○
• `combinations`, `combinations(n)`	○
• `ltrimstr(str)`	○
• `rtrimstr(str)`	○
• `explode`	○
• `implode`	○
• `split`	○
• `join(str)`	○
• `ascii_downcase`, `ascii_upcase`	○
• `while(cond; update)`	○
• `until(cond; next)`	○
• `recurse(f)`, `recurse`, `recurse(f; condition)`, `recurse_down`	○
• `..`	○
• `env`	○<sup>*6</sup>
• `transpose`	○
• `bsearch(x)`	×
• String interpolation - `\(foo)`	○
• Convert to/from JSON	○
• Format strings and escaping	○
• Dates	×
Conditionals and Comparisons	○
• `==`, `!=`	○
• if-then-else	○
• `>, >=, <=, <`	○
• and/or/not	○
• Alternative operator - `//`	○
• try-catch	○<sup>*1</sup>
• Breaking out of control structures	○<sup>*2</sup>
• `?` operator	○
Regular expressions (PCRE)	○
• `test(val)`, `test(regex; flags)`	○
• `match(val)`, `match(regex; flags)`	○
• `capture(val)`, `capture(regex; flags)`	○
• `scan(regex)`, `scan(regex; flags)`	○
• `split(regex; flags)`	○
• `splits(regex)`, `splits(regex; flags)`	○
• `sub(regex; tostring)` `sub(regex; string; flags)`	○
• `gsub(regex; string)`, `gsub(regex; string; flags)`	○
Advanced features	○
• Variables	○<sup>*11</sup>
• Destructuring Alternative Operator: ?//	✕ (#44)
• Defining Functions	○<sup>*3</sup>
• Reduce	○
• `limit(n; exp)`	○
• `first(expr)`, `last(expr)`, `nth(n; expr)`	○
• `first`, `last`, `nth(n)`	○
• `foreach`	○
• Recursion	○
• Generators and iterators	○
Math	△
I/O	N/A
• `input`	N/A
• `inputs`	N/A
• `debug`	N/A
• `input_filename`	N/A
• `input_line_number`	N/A
Streaming	N/A
• `truncate_stream(stream_expression)`	N/A
• `fromstream(stream_expression)`	N/A
• `tostream`	N/A
Assignment	○
• `=`	○
• `ǀ=`	○<sup>*8</sup>
• `+=`, `-=`, `*=`, `/=`, `%=`, `//=`	○
• Complex assignments	○
Modules	△
• `import RelativePathString as NAME [<metadata>];`	○
• `include RelativePathString [<metadata>];`	○
• `import RelativePathString as $NAME [<metadata>];`	○
• `module <metadata>;`	○
• `modulemeta`	×

</details>

Known Compatibility Issues / Differences

Category: BUG

<details> <summary>(*11) Operator Precedences in <code>1 + 3 as $a | ($a * 2)</code></summary>

Description

The presence of as $a affects precedence of | and other operators in jq:

$ jq -n '1 + 3 | (. * 2)' # interpreted as (1 + 3) | (. * 2)
8
$ jq -n '1 + 3 as $a | ($a * 2)' # interpreted as 1 + (3 as $a | ($a * 2))
7

whereas jackson-jq consistently interprets them as (1 + 3) whether as $a is used or not:

$ java -jar jackson-jq-cli-1.2.0.jar -n '1 + 3 | (. * 2)' # interpreted as (1 + 3) | (. * 2)
8
$ java -jar jackson-jq-cli-1.2.0.jar -n '1 + 3 as $a | ($a * 2)' # interpreted as (1 + 3) as $a | ($a * 2)
8

Examples

$ jq -n '1 + 3 as $a | ($a * 2)' # interpreted as 1 + (3 as $a | ($a * 2))
7
$ java -jar jackson-jq-cli-1.2.0.jar -n '1 + 3 as $a | ($a * 2)' # interpreted as (1 + 3) as $a | ($a * 2)
8

Workaround

Use explicit parentheses.

Description

If the function with the same is defined more than once at the same scope, jackson-jq uses the last one.

Examples

$ jq -n 'def f: 1; def g: f; def f: 2; g'
1
$ java -jar jackson-jq-cli-1.2.0.jar -n 'def f: 1; def g: f; def f: 2; g'
2

Workaround

Avoid using the duplicate function name.

$ java -jar jackson-jq-cli-1.2.0.jar -n 'def f1: 1; def g: f1; def f2: 2; g'
1

</details>

Category: BY DESIGN

<details> <summary>(*1) Error Message Wording</summary>

Description

Error messages differ between jq and jackson-jq and they also tend to change between versions.

Workaround

None. This is by design and will not be fixed.

</details> <details> <summary>(*6) <code>env/0</code> is not available by default.</summary>

Description

env/0 is not available by default for security reasons and must be added manually to the scope.

Workaround

Add env/0 manually into the scope:

SCOPE.addFunction("env", 0, new EnvFunction())

</details> <details> <summary>(*4) Field Ordering in JSON Object</summary>

Description

The order of the keys in JSON is not preserved. It was a design decision but we are slowly trying to fix this in order to improve the compatibility with jq.

Workaround

None. Use array if the order is important.

</details> <details> <summary>(*5) <code>0 / 0</code> is an error in jackson-jq.</summary>

Description

jq evaluates 0 / 0, if hard-coded, to NaN without any errors, whereas 0 | 0 / . results in a zero-division error. jackson-jq always raises an error in both cases.

Examples

$ jq -n '0 / 0'
null
$ jq -n '10 / 0'
jq: error: Division by zero? at <top-level>, line 1:
10 / 0
jq: 1 compile error
$ jq '. / 0' <<< 0
jq: error (at <stdin>:1): number (0) and number (0) cannot be divided because the divisor is zero
$ java -jar jackson-jq-cli-1.2.0.jar -n '0 / 0'
jq: error: number (0) and number (0) cannot be divided because the divisor is zero

Workaround

If you need NaN, use nan instead of 0 / 0.

</details> <details> <summary>(*8) <code>... |= empty</code> is an error in jackson-jq.</summary>

Description

.foo |= empty always throws an error in jackson-jq instead of producing an unexpected result. jq-1.5 and jq-1.6 respectively produces a different and incorrect result for [1,2,3] | ((.[] | select(. > 1)) |= empty). jq#897 says "empty in the RHS is undefined". You can still use _modify/2 directly if you really want to emulate the exact jq-1.5 or jq-1.6 behavior.

Examples

$ jq-1.6 -n '[1,2,3] | ((.[] | select(. > 1)) |= empty)'
[
  1,
  3
]
$ jq-1.5 -n '[1,2,3] | ((.[] | select(. > 1)) |= empty)'
null
$ jq-1.2 -n '[1,2,3] | ((.[] | select(. > 1)) |= empty)'
[
  1,
  2,
  3
]
$ java -jar jackson-jq-cli-1.2.0.jar --jq 1.6 -n '[1,2,3] | ((.[] | select(. > 1)) |= empty)'
jq: error: `|= empty` is undefined. See https://github.com/stedolan/jq/issues/897
$ java -jar jackson-jq-cli-1.2.0.jar --jq 1.5 -n '[1,2,3] | ((.[] | select(. > 1)) |= empty)'
jq: error: `|= empty` is undefined. See https://github.com/stedolan/jq/issues/897

Workaround

You can use _modify/2 if you really want to the original behavior.

$ java -jar jackson-jq-cli-1.2.0.jar --jq 1.6 -n '[1,2,3] | _modify((.[] | select(. > 1)); empty)'
[ 1, 3 ]
$ java -jar jackson-jq-cli-1.2.0.jar --jq 1.5 -n '[1,2,3] | _modify((.[] | select(. > 1)); empty)'
null

</details> <details> <summary>(*7) Variables don't carry path information even in jq 1.5 compat mode.</summary>

Description

path(.foo as $a | $a) always throws an error as $variables in jackson-jq do not carry path information like jq-1.5 accidentally? did. The behavior is fixed in jq-1.6 whose documentation explicitly states them as "not a valid or useful path expression". So, I dicided not to implement it even in jq-1.5 compatible mode.

Examples

jq 1.5

$ jq-1.5 -c 'path(.foo as $a | $a)' <<< '{"foo": 1}'
["foo"]
$ java -jar jackson-jq-cli-1.2.0.jar --jq 1.5 -c 'path(.foo as $a | $a)' <<< '{"foo": 1}'
jq: error: Invalid path expression with result 1

jq 1.6

$ jq-1.6 -c 'path(.foo as $a | $a)' <<< '{"foo": 1}'
jq: error (at <stdin>:1): Invalid path expression with result 1
$ java -jar jackson-jq-cli-1.2.0.jar --jq 1.6 -c 'path(.foo as $a | $a)' <<< '{"foo": 1}'
jq: error: Invalid path expression with result 1

Workaround

None

</details> <details> <summary>(*2) <code>try (break $label) catch .</code> always produces <code>{"__jq": 0}</code>.</summary>

Description

<code>try (break $label) catch .</code> always produces <code>{"__jq": 0}</code> in jackson-jq, while __jq should contain the index of the label the break statement jumps to.

Examples

$ jq -n 'label $a | label $b | try (break $b) catch .'
{
  "__jq": 1
}
$ java -jar jackson-jq-cli-1.2.0.jar -n 'label $a | label $b | try (break $b) catch .'
{
  "__jq" : 0
}

Workaround

None. Tell us your use case if you need this feature.

</details>

Category: BUGFIX

<details> <summary>(*9) <code>indices("")</code> returns <code>[]</code> (empty array) in jackson-jq.</summary>

Description

indices/1 implementation in jq-1.5 and jq-1.6 had a bug that caused indices("") to end up in infinite loop which eventually leads to OOM. The bug is fixed and likely to be in jq-1.7 (not released yet). jackson-jq chose not to simulate this bug.

Examples

$ jq-1.5 -n '"x" | indices("")' # stuck in infinite loop
^C
$ jq-1.6 -n '"x" | indices("")' # stuck in infinite loop
^C
$ jq-1.6-83-gb52fc10 -n '"x" | indices("")'
[]
$ java -jar jackson-jq-cli-1.2.0.jar -n '"x" | indices("")'
[ ]

</details>

Using jackson-jq/extras module

The jackson-jq/extras module is a jq module that provides some useful functions that do not exist in jq.

To use this module, you need to add the following Maven dependency and set BuiltinModuleLoader (see jackson-jq/src/test/java/examples/Usage.java) to the scope.

<dependency>
	<groupId>net.thisptr</groupId>
	<artifactId>jackson-jq-extra</artifactId>
	<version>1.2.0</version>
</dependency>

Now, you can import the module in jq:

import "jackson-jq/extras" as extras;

extras::uuid4

For a historical reason, adding the Maven dependency also makes the functions directly available to jq. This behavior is deprecated and will be removed at some point in the future.

<details> <summary>List of Functions</summary>

uuid4/0

jackson-jq -n 'uuid4' #=> "a69cf146-f40e-42e1-ae88-12590bdae947"

random/0

jackson-jq -n 'random' #=> 0.43292159535427466

timestamp/0, strptime/{1, 2}, strftime/{1, 2}

jackson-jq -n 'timestamp' #=> 1477162056362
jackson-jq -n '1477162342372 | strftime("yyyy-MM-dd HH:mm:ss.SSSXXX")' #=> "2016-10-23 03:52:22.372+09:00"
jackson-jq -n '1477162342372 | strftime("yyyy-MM-dd HH:mm:ss.SSSXXX"; "UTC")' #=> "2016-10-22 18:52:22.372Z"
jackson-jq -n '"2016-10-23 03:52:22.372+09:00" | strptime("yyyy-MM-dd HH:mm:ss.SSSXXX")' #=> 1477162342372
jackson-jq -n '"2016-10-22 18:52:22.372" | strptime("yyyy-MM-dd HH:mm:ss.SSS"; "UTC")' #=> 1477162342372

uriparse/0

jackson-jq -n '"http://user@www.example.com:8080/index.html?foo=1&bar=%20#hash" | uriparse' #=>

{
  "scheme" : "http",
  "user_info" : "user",
  "raw_user_info" : "user",
  "host" : "www.example.com",
  "port" : 8080,
  "authority" : "user@www.example.com:8080",
  "raw_authority" : "user@www.example.com:8080",
  "path" : "/index.html",
  "raw_path" : "/index.html",
  "query" : "foo=1&bar= ",
  "raw_query" : "foo=1&bar=%20",
  "query_obj" : {
    "bar" : " ",
    "foo" : "1"
  },
  "fragment" : "hash",
  "raw_fragment" : "hash"
}

uridecode/0

jackson-jq -n '"%66%6f%6f" | uridecode' #=> "foo"

hostname/0

jackson-jq -n 'hostname' #=> "jenkins-slave01"

</details>

Contributing

If you are planning to send a PR and the change is not small, please open an issue and discuss it with the authors first.
Other than bug reports or patches, documentation improvements (including small grammatical or wording corrections) would be greatly appreciated.

License

This software is licensed under Apache Software License, Version 2.0, with some exceptions:

jackson-jq/src/test/resources contains test cases from stedolan/jq.
jackson-jq/src/main/resources/net/thisptr/jackson/jq/jq.json contains function definitions extracted from stedolan/jq.

See COPYING for details.

Awesome

jackson-jq

Usage

Using a jackson-jq command line tool

Branches and versioning

Implementation Status

List of Features

Known Compatibility Issues / Differences

Category: BUG

Description

Examples

Workaround

Links

Description

Examples

Workaround

Category: BY DESIGN

Description

Workaround

Description

Workaround

Description

Workaround

Description

Examples

Workaround

Description

Examples

Workaround

Description

Examples

Workaround

Description

Examples

Workaround

Category: BUGFIX

Description

Examples

Using jackson-jq/extras module

uuid4/0

random/0

timestamp/0, strptime/{1, 2}, strftime/{1, 2}

uriparse/0

uridecode/0

hostname/0

Contributing

License