Awesome
gojq
Pure Go implementation of jq
This is an implementation of jq command written in Go language. You can also embed gojq as a library to your Go products.
Usage
$ echo '{"foo": 128}' | gojq '.foo'
128
$ echo '{"a": {"b": 42}}' | gojq '.a.b'
42
$ echo '{"id": "sample", "10": {"b": 42}}' | gojq '{(.id): .["10"].b}'
{
"sample": 42
}
$ echo '[{"id":1},{"id":2},{"id":3}]' | gojq '.[] | .id'
1
2
3
$ echo '{"a":1,"b":2}' | gojq '.a += 1 | .b *= 2'
{
"a": 2,
"b": 4
}
$ echo '{"a":1} [2] 3' | gojq '. as {$a} ?// [$a] ?// $a | $a'
1
2
3
$ echo '{"foo": 4722366482869645213696}' | gojq .foo
4722366482869645213696 # keeps the precision of large numbers
$ gojq -n 'def fact($n): if $n < 1 then 1 else $n * fact($n - 1) end; fact(50)'
30414093201713378043612608166064768844377641568960512000000000000 # arbitrary-precision integer calculation
Nice error messages.
$ echo '[1,2,3]' | gojq '.foo & .bar'
gojq: invalid query: .foo & .bar
.foo & .bar
^ unexpected token "&"
$ echo '{"foo": { bar: [] } }' | gojq '.'
gojq: invalid json: <stdin>
{"foo": { bar: [] } }
^ invalid character 'b' looking for beginning of object key string
Installation
Homebrew
brew install gojq
Zero Install
0install add gojq https://apps.0install.net/utils/gojq.xml
Build from source
go install github.com/itchyny/gojq/cmd/gojq@latest
Docker
docker run -i --rm itchyny/gojq
docker run -i --rm ghcr.io/itchyny/gojq
Difference to jq
- gojq is purely implemented with Go language and is completely portable. jq depends on the C standard library so the availability of math functions depends on the library. jq also depends on the regular expression library and it makes build scripts complex.
- gojq implements nice error messages for invalid query and JSON input. The error message of jq is sometimes difficult to tell where to fix the query.
- gojq does not keep the order of object keys. I understand this might cause problems for some scripts but basically, we should not rely on the order of object keys. Due to this limitation, gojq does not have
keys_unsorted
function and--sort-keys
(-S
) option. I would implement when ordered map is implemented in the standard library of Go but I'm less motivated. - gojq supports arbitrary-precision integer calculation while jq does not; jq loses the precision of large integers when calculation is involved. Note that even with gojq, all mathematical functions, including
floor
andround
, convert integers to floating-point numbers; only addition, subtraction, multiplication, modulo, and division operators (when divisible) keep the integer precision. To calculate floor division of integers without losing the precision, usedef idivide($n): (. - . % $n) / $n;
. To round down floating-point numbers to integers, usedef ifloor: floor | tostring | tonumber;
, but note that this function does not work with large floating-point numbers and also loses the precision of large integers. - gojq behaves differently than jq in some features, hoping that jq will fix the behaviors in the future. gojq consistently counts by characters (not by bytes) in
index
,rindex
, andindices
functions;"12345" | .[index("3"):]
results in"345"
(jq#1430, jq#1624). gojq supports string indexing;"abcde"[2]
(jq#1520). gojq fixes handling files with no newline characters at the end (jq#2374). gojq consistently truncates down floating-point number indices both in indexing ([0] | .[0.5]
results in0
), and slicing ([0,1,2] | .[0.5:1.5]
results in[0]
). gojq parses unary operators with higher precedence than variable binding ([-1 as $x | 1,$x]
results in[1,-1]
not[-1,-1]
) (jq#3053). gojq fixes@base64d
to allow binary string as the decoded string (jq#1931). gojq improves time formatting and parsing; deals with%f
instrftime
andstrptime
(jq#1409), parses timezone offsets withfromdate
andfromdateiso8601
(jq#1053), supports timezone name/offset with%Z
/%z
instrptime
(jq#929, jq#2195), and looks up correct timezone during daylight saving time on formatting with%Z
(jq#1912). gojq supports nanoseconds in date and time functions. - gojq does not support some functions intentionally;
get_jq_origin
,get_prog_origin
,get_search_list
(unstable, not listed in jq document),input_line_number
,$__loc__
(performance issue). gojq does not support some flags;--ascii-output, -a
(performance issue),--seq
(not used commonly),--sort-keys, -S
(sorts by default becausemap[string]any
does not keep the order),--unbuffered
(unbuffered by default). gojq does not parse JSON extensions supported by jq;NaN
,Infinity
, and[000]
. gojq normalizes floating-point numbers to fit to double-precision floating-point numbers. gojq does not support some regular expression metacharacters, backreferences, look-around assertions, and some flags (regular expression engine differences). gojq does not support BOM (encoding/json
does not support this). gojq disallows using keywords for function names (def true: .; true
is a confusing query), and module name prefixes in function declarations (using module prefixes likedef m::f: .;
is undocumented). - gojq supports reading from YAML input (
--yaml-input
) while jq does not. gojq also supports YAML output (--yaml-output
). gojq supports@urid
format string (jq#798, jq#2261).
Color configuration
The gojq command automatically disables coloring output when the output is not a tty.
To force coloring output, specify --color-output
(-C
) option.
When NO_COLOR
environment variable is present or --monochrome-output
(-M
) option is specified, gojq disables coloring output.
Use GOJQ_COLORS
environment variable to configure individual colors.
The variable is a colon-separated list of ANSI escape sequences of null
, false
, true
, numbers, strings, object keys, arrays, and objects.
The default configuration is 90:33:33:36:32:34;1
.
Usage as a library
You can use the gojq parser and interpreter from your Go products.
package main
import (
"fmt"
"log"
"github.com/itchyny/gojq"
)
func main() {
query, err := gojq.Parse(".foo | ..")
if err != nil {
log.Fatalln(err)
}
input := map[string]any{"foo": []any{1, 2, 3}}
iter := query.Run(input) // or query.RunWithContext
for {
v, ok := iter.Next()
if !ok {
break
}
if err, ok := v.(error); ok {
if err, ok := err.(*gojq.HaltError); ok && err.Value() == nil {
break
}
log.Fatalln(err)
}
fmt.Printf("%#v\n", v)
}
}
- Firstly, use
gojq.Parse(string) (*Query, error)
to get the query from a string.- Use
gojq.ParseError
to get the error position and token of the parsing error.
- Use
- Secondly, get the result iterator
- using
query.Run
orquery.RunWithContext
- or alternatively, compile the query using
gojq.Compile
and thencode.Run
orcode.RunWithContext
. You can reuse the*Code
against multiple inputs to avoid compilation of the same query. But for arguments ofcode.Run
, do not give values sharing same data between multiple calls. - In either case, you cannot use custom type values as the query input. The type should be
[]any
for an array andmap[string]any
for a map (just like decoded to anany
using the encoding/json package). You can't use[]int
ormap[string]string
, for example. If you want to query your custom struct, marshal to JSON, unmarshal toany
and use it as the query input.
- using
- Thirdly, iterate through the results using
iter.Next() (any, bool)
. The iterator can emit an error so make sure to handle it. The method returnstrue
with results, andfalse
when the iterator terminates.- The return type is not
(any, error)
because the iterator may emit multiple errors. Thejq
andgojq
commands stop the iteration on the first error, but the library user can choose to stop the iteration on errors, or to continue until it terminates.- In any case, it is recommended to stop the iteration on
gojq.HaltError
, which is emitted byhalt
andhalt_error
functions, although these functions are rarely used. The error implementsgojq.ValueError
, and if the error value isnil
, stop the iteration without handling the error. Technically speaking, we can fix the iterator to terminate on the halting error, but it does not terminate at the moment. Thehalt
function in jq not only stops the iteration, but also terminates the command execution, even if there are still input values. So, gojq leaves it up to the library user how to handle the halting error.
- In any case, it is recommended to stop the iteration on
- Note that the result iterator may emit infinite number of values;
repeat(0)
andrange(infinite)
. It may stuck with no output value;def f: f; f
. UseRunWithContext
when you want to limit the execution time.
- The return type is not
gojq.Compile
allows to configure the following compiler options.
gojq.WithModuleLoader
allows to load modules. By default, the module feature is disabled. If you want to load modules from the file system, usegojq.NewModuleLoader
.gojq.WithEnvironLoader
allows to configure the environment variables referenced byenv
and$ENV
. By default, OS environment variables are not accessible due to security reasons. You can usegojq.WithEnvironLoader(os.Environ)
if you want.gojq.WithVariables
allows to configure the variables which can be used in the query. Pass the values of the variables tocode.Run
in the same order.gojq.WithFunction
allows to add a custom internal function. An internal function can return a single value (which can be an error) each invocation. To add a jq function (which may include a comma operator to emit multiple values,empty
function, accept a filter for its argument, or call another built-in function), useLoadInitModules
of the module loader.gojq.WithIterFunction
allows to add a custom iterator function. An iterator function returns an iterator to emit multiple values. You cannot define both iterator and non-iterator functions of the same name (with possibly different arities). You can usegojq.NewIter
to convert values or an error to agojq.Iter
.gojq.WithInputIter
allows to useinput
andinputs
functions. By default, these functions are disabled.
Bug Tracker
Report bug at Issues・itchyny/gojq - GitHub.
Author
itchyny (https://github.com/itchyny)
License
This software is released under the MIT License, see LICENSE.