Home

Awesome

jsn

Build Status

Hex.pm version

jsn is a tool for working with JSON representations in erlang--complex, nested JSON objects in particular.

In the spirit of ej, it supports the common formats output by JSON decoders such as jsone, jiffy, and mochijson2. Unlike ej, however, it supports all four common JSON representations in Erlang:

In addition to supporting the additional proplist and map formats, jsn's path input structure is somewhat more flexible, allowing for input of period-delimited binary strings or atoms to indicate a path through a deeply nested structure. This support is similar to kvc's path format, and also likely to be familiar to users of erlson.

This code base was originally developed as a wrapper around ej, adding support for the 'syntactic sugar' of the period-delimited keys. However, a need arose for the library to be proplist-compatible, then map-compatible, so it has been refactored to be a nearly standalone library.

Caveats & known issues

Proplist format concerns

It should be noted that the proplist format supported by jsn is compatible with the abandoned jsonx library, and is not compatible with the proplist format used in jsx and jsone. Specifically, jsn uses the empty list ([]) like jsonx to represent an empty object, whereas jsx and jsone use an empty tuple in a list ([{}]) to represent an empty object.

jsx and jsone use this representation to disambiguate empty JSON objects from empty arrays, which cannot be distinguished in the jsonx proplist format used by jsn. jsn is incompatible with this format. While the getter (jsn:get/2,3) functions are generally functional, most other library functions are not, and may result in unpredictable behaviors.

jsn does not plan to support a jsx and jsone compatible proplist format; long-term, clients are strongly encouraged to use the map format instead. It a vastly more performant data structure that maps naturally to JSON objects without ambiguity.

Roadmap

Future improvements to this library are TBD at this time.

Changelog

29 March 2022 - 2.2.2

4 November 2020 - 2.2.1

9 June 2020 - 2.2.0

9 February 2020 - 2.1.4

19 August 2019 - 2.1.3

18 August 2019 - 2.1.2

27 February 2018 - 2.1.1

6 November 2017 - 2.1.0

6 October 2017 - 2.0.0

3 October 2017 - 1.2.1

26 September 2017 - 1.2.0

24 August 2017 - 1.1.0

23 August 2017 - 1.0.3

22 August 2017 - 1.0.2

6 November 2016 - 1.0.1

5 February 2016 - 1.0.0

Running

To run this library locally, build the source code:

make

Then start an erlang shell:

make start

jsn is an OTP library, and does not really need to be started as such.

Application integration

jsn has been published to hex.pm; to add jsn to your Erlang OTP application, simply add it to your rebar.config:

{deps, [
    %% ... your deps ...
    jsn
]}.

and your applications src/<application>.app.src or ebin/<application>.app file:

{application, <application>, [
    {description, "An application that uses nested JSON interaction"},
    {applications, [
        kernel,
        stdlib,
        %% ... your deps ...
        jsn
    ]}
]}

After you re-compile, you will have full access to jsn from your local console.

Paths & indexes

Paths are pointers into a (potentially nested) jsn object. An object may contain sub-objects or arrays at any layer, and as such, a path may include both keys (as binary strings and sometimes atoms) as well as array indexes. Indexes can be provided as positive integers (i.e., 1 is the first element of an array) or as the shortcut atoms 'first' and 'last'.

There are 3 different supported path styles, each with different tradeoffs:

  1. List of binary/atom keys, e.g. [<<"person">>, 'id']. Mixing and matching atom and binary string keys is supported, but using binary keys only is most performant (and matches the 'native' path format). However, this format does not support array indexes.
  2. Tuple of binary keys and array indexes, e.g., {<<"users">>, last}. Atom keys are not supported due to potential ambiguity. This is the only possible path format to use if you want to leverage the array index feature.
  3. Period-delimited atom or binary string, e.g. <<"user.id">> or 'user.id'. This format is the most compact and readable, but only supports keys (no array indexes).

Library functions

jsn provides functions to create, append, delete, and transform objects in all supported formats (map, proplist, eep18, and struct). This section contains a reference for the primary library functions available.

new/0,1,2 - Create a new object

Examples

% create an empty object
jsn:new().
% #{}

% create an object using a single path, value pair.
jsn:new({'user.id', <<"123">>}).
% #{<<"user">> => #{<<"id">> => <<"123">>}}

% create an object using a list of path, value pairs.
jsn:new([{'user.id', <<"123">>}, {<<"user.name">>, <<"John">>}]).
% #{<<"user">> => #{<<"id">> => <<"123">>,
%                   <<"name">> => <<"John">>}}

% create a jsn object in proplist format
jsn:new([{'user.id', <<"123">>},
         {<<"user.name">>, <<"John">>}], [{format, proplist}]).
% [{<<"user">>, [{<<"id">>,<<"123">>},
%                {<<"name">>,<<"John">>}]}]

% create a jsn object in eep18 format
jsn:new([{'user.id', <<"123">>},
         {<<"user.name">>, <<"John">>}], [{format, eep18}]).
% {[{<<"user">>, {[{<<"id">>,<<"123">>},
%                  {<<"name">>,<<"John">>}]}}]}

% create a jsn object in struct (mochijson2) format
jsn:new([{'user.id', <<"123">>},
         {<<"user.name">>, <<"John">>}], [{format, struct}]).
% {struct, [{<<"user">>, {struct, [{<<"id">>,<<"123">>},
%                                  {<<"name">>,<<"John">>}]}}]}

get/2,3, get_list/2,3, find/3,4, select/2,3, with/2, and without/2 - Extract data from objects

<a name="selections-conditions"/>Selections and Conditions in select/2,3

The functions select/2 and select/3 accept selection and conditional specifications defined in jsn.hrl.

Selections can be passed singularly or as a list in select/2 and select/3. If it is passed as a list of selections, the output result from the select call will contain a symmetrically ordered list of results for each element in the input list. A Selection is one of the following:

Conditions can be passed singularly or as a list in select/3. A Condition is one of the following:

See below for examples.

<a name="extract-examples"/>Examples

User = jsn:new([{'user.id', <<"123">>},
                {'user.activated', true},
                {'user.name.first', <<"Jane">>},
                {'user.name.last', <<"Doe">>}]).
% #{<<"user">> => #{<<"activated">> => true,
%                   <<"id">> => <<"123">>,
%                   <<"name">> => #{<<"first">> => <<"Jane">>,
%                                   <<"last">> => <<"Doe">>}}}

% get the user id
UserId = jsn:get('user.id', User).
% <<"123">>

% get a non-existent field, with and without a custom default
jsn:get(<<"user.deleted">>, User).
% undefined
jsn:get([<<"user">>, <<"deleted">>], User, false).
% false

% get several fields in a single call:
jsn:get_list(['user.name.first', 'user.name.last', 'user.name.middle'], User).
% [<<"Jane">>,<<"Doe">>,undefined]
jsn:get_list(['user.activated', 'user.deleted'], User, false).
% [true,false]

User2 = jsn:new([{'user.id', <<"456">>},
                 {'user.name.first', <<"Eve">>},
                 {'user.name.middle', <<"L.">>},
                 {'user.name.last', <<"Doer">>}]).
% #{<<"user">> => #{<<"id">> => <<"456">>,
%                   <<"name">> => #{<<"first">> => <<"Eve">>,
%                                    <<"last">> => <<"Doer">>,
%                                    <<"middle">> => <<"L.">>}}}

% find the first user by id:
[User] = jsn:find({<<"user">>, <<"id">>}, <<"123">>, [User, User2]).
% [#{<<"user">> => #{<<"activated">> => true,
%                    <<"id">> => <<"123">>,
%                    <<"name">> => #{<<"first">> => <<"Jane">>,
%                                    <<"last">> => <<"Doe">>}}}]

% select the first name from the users:
jsn:select({value, <<"user.name.first">>}, [User, User2]).
% [<<"Jane">>,<<"Eve">>]

% select the user id and whole object from the users:
jsn:select([{value, [<<"user">>, <<"id">>]}, identity], [User, User2]).
% [[<<"123">>, #{<<"user">> =>
%                    #{<<"activated">> => true,
%                      <<"id">> => <<"123">>,
%                      <<"name">> => #{<<"first">> => <<"Jane">>,
%                                      <<"last">> => <<"Doe">>}}}],
%  [<<"456">>, #{<<"user">> =>
%                    #{<<"id">> => <<"456">>,
%                      <<"name">> => #{<<"first">> => <<"Eve">>,
%                                      <<"last">> => <<"Doer">>,
%                                      <<"middle">> => <<"L.">>}}}]]

% select the user id and first name from the users whose last name is <<"Doe">>:
jsn:select([{value, [<<"user">>, <<"id">>]},
            {value, [<<"user">>, <<"name">>, <<"first">>]}],
           {<<"user.name.last">>, <<"Doe">>},
           [User, User2]).
% [[<<"123">>,<<"Jane">>]]

% select the user id from the users whose middle name is missing:
jsn:select({value, [<<"user">>, <<"id">>]},
           {<<"user.name.middle">>, fun(undefined) -> true; (_) -> false end},
           [User, User2]).
% [<<"123">>]

% select the user id from the users whose whose first name is < 4 characters
% and whose last name is > 3 characters
ConditionFun = fun(Object) ->
                   [First, Last] = jsn:get_list([<<"user.name.first">>,
                                                 <<"user.name.last">>],
                                                Object),
                   (byte_size(First) < 4) andalso (byte_size(Last) > 3)
                end.
jsn:select({value, [<<"user">>, <<"id">>]},
           [ConditionFun],
           [User, User2]).
% [<<"456">>]

<a name="with-without"/>Using with/2 and without/2

The functions with/2 and without/3 accept a list of path()s or a path_elements_map() defined in jsn.hrl. These functions are inspired by maps:with/2 and maps:without/2, but support the nested path formats of this library. Essentially, you can pass a list of arbitrarily nested paths to either function, and they will return your input with those paths applied, either by including only those paths in the input in the case of with/2, or removing those paths from the input in the case of without/2.

However, because nested paths are a little more complex than working with the flat keys list of the maps analogues of these functions, there are some caveats and specific aspects to be aware of when using these functions.

See below for examples.

<a name="with-without-examples"/>Examples

User = jsn:new([{'user.activated', true},
                {'user.hobbies', [#{<<"type">> => <<"food">>,
                                    <<"name">> => <<"bread">>},
                                  #{<<"type">> => <<"drink">>,
                                    <<"name">> => <<"wine">>}]},
                {'user.id', <<"123">>},
                {'user.name.first', <<"Jane">>},
                {'user.name.last', <<"Doe">>},
                {'user.password_hash', <<"9S+9MrKzuG/4jvbEkGKChfSCrxXdyylUH5S89Saj9sc=">>}]).
% #{<<"user">> =>
%       #{<<"activated">> => true,
%         <<"hobbies">> =>
%             [#{<<"name">> => <<"bread">>,
%                <<"type">> => <<"food">>},
%              #{<<"name">> => <<"wine">>,
%                <<"type">> => <<"drink">>}],
%         <<"id">> => <<"123">>,
%         <<"name">> =>
%             #{<<"first">> => <<"Jane">>,
%               <<"last">> => <<"Doe">>},
%         <<"password_hash">> =>
%             <<"9S+9MrKzuG/4jvbEkGKChfSCrxXdyylUH5S89Saj9sc=">>}}

% get the user with just the user's name
UserWithName = jsn:with(['user.name'], User).
% #{<<"user">> =>
%       #{<<"name">> =>
%             #{<<"first">> => <<"Jane">>,
%               <<"last">> => <<"Doe">>}}}

% get the user with just the user's id and first hobby name
UserWithIdAndHobby = jsn:with(['user.id',
                               {<<"user">>, <<"hobbies">>, 1, <<"name">>}],
                              User).
% #{<<"user">> =>
%       #{<<"hobbies">> => [#{<<"name">> => <<"bread">>}],
%         <<"id">> => <<"123">>}}

% get the user without her password hash or hobby types
UserWithoutPassHash = jsn:without(['user.password_hash',
                                   {<<"user">>, <<"hobbies">>, 1, <<"type">>},
                                   {<<"user">>, <<"hobbies">>, 2, <<"type">>}], User).
% #{<<"user">> =>
%       #{<<"activated">> => true,
%         <<"hobbies">> =>
%             [#{<<"name">> => <<"bread">>},
%              #{<<"name">> => <<"wine">>}],
%         <<"id">> => <<"123">>,
%         <<"name">> =>
%             #{<<"first">> => <<"Jane">>,
%               <<"last">> => <<"Doe">>}}}

% pre-build a tree of path elements that includes atoms
PathMap1 = jsn:path_elements_map([[user, id],
                                  [user, name],
                                  {<<"user">>, <<"hobbies">>, first}]).
% #{user =>
%       #{hobbies => #{first => true},
%         id => true,
%         name => true,
%         <<"hobbies">> => #{first => true},
%         <<"id">> => true,
%         <<"name">> => true},
%   <<"user">> =>
%       #{hobbies => #{first => true},
%         id => true,
%         name => true,
%         <<"hobbies">> => #{first => true},
%         <<"id">> => true,
%         <<"name">> => true}}

% use a pre-built path elements map
UserWithNameAndIdAndFirstHobby = jsn:with(PathMap1, User).
% #{<<"user">> =>
%       #{<<"hobbies">> =>
%             [#{<<"name">> => <<"bread">>,
%                <<"type">> => <<"food">>}],
%         <<"id">> => <<"123">>,
%         <<"name">> =>
%             #{<<"first">> => <<"Jane">>,
%               <<"last">> => <<"Doe">>}}}

set/3 and set_list/2 - Add to and update existing objects

Examples

User = jsn:new([{'user.id', <<"123">>},
                {'user.activated', true},
                {'user.name.first', <<"Jane">>},
                {'user.name.last', <<"Doe">>}]).
% #{<<"user">> => #{<<"activated">> => true,
%                   <<"id">> => <<"123">>,
%                   <<"name">> => #{<<"first">> => <<"Jane">>,
%                                   <<"last">> => <<"Doe">>}}}

% Set Jane's middle name
jsn:set([<<"user">>, <<"name">>, <<"middle">>], User, <<"Jacqueline">>).
% #{<<"user">> => #{<<"activated">> => true,
%                   <<"id">> => <<"123">>,
%                   <<"name">> => #{<<"first">> => <<"Jane">>,
%                                   <<"last">> => <<"Doe">>,
%                                   <<"middle">> => <<"Jacqueline">>}}}

% Deactivate Jane's User, and change her middle name
jsn:set_list([{'user.activated', false},
              {'user.name.middle', <<"Jay">>}], User).
% #{<<"user">> => #{<<"activated">> => false,
%                   <<"id">> => <<"123">>,
%                   <<"name">> => #{<<"first">> => <<"Jane">>,
%                                   <<"last">> => <<"Doe">>,
%                                   <<"middle">> => <<"Jay">>}}}

delete/2, delete_list/2, and delete_if_equal/2 - Remove data from existing objects

Examples

Company = jsn:new([{'company.name', <<"Foobar, Inc.">>},
                   {'company.created.by', <<"00000000">>},
                   {'company.created.at', 469778436},
                   {'company.location', <<"U.S. Virgin Islands">>},
                   {{<<"company">>, <<"employees">>}, []},
                   {{<<"company">>, <<"employees">>, 1, <<"id">>}, <<"00000000">>},
                   {{<<"company">>, <<"employees">>, 1, <<"name">>}, <<"Alice">>},
                   {{<<"company">>, <<"employees">>, 1, <<"position">>}, <<"CEO">>},
                   {{<<"company">>, <<"employees">>, 2, <<"id">>}, <<"00000001">>},
                   {{<<"company">>, <<"employees">>, 2, <<"name">>}, <<"Bob">>},
                   {{<<"company">>, <<"employees">>, 2, <<"position">>}, <<"CTO">>}]).
% #{<<"company">> =>
%       #{<<"created">> => #{<<"at">> => 469778436,
%                            <<"by">> => <<"00000000">>},
%         <<"employees">> => [#{<<"id">> => <<"00000000">>,
%                               <<"name">> => <<"Alice">>,
%                               <<"position">> => <<"CEO">>},
%                             #{<<"id">> => <<"00000001">>,
%                               <<"name">> => <<"Bob">>,
%                               <<"position">> => <<"CTO">>}],
%         <<"location">> => <<"U.S. Virgin Islands">>,
%         <<"name">> => <<"Foobar, Inc.">>}}

% remove the location from Company
jsn:delete('company.location', Company).
% #{<<"company">> =>
%       #{<<"created">> => #{<<"at">> => 469778436,
%                            <<"by">> => <<"00000000">>},
%         <<"employees">> => [#{<<"id">> => <<"00000000">>,
%                               <<"name">> => <<"Alice">>,
%                               <<"position">> => <<"CEO">>},
%                             #{<<"id">> => <<"00000001">>,
%                               <<"name">> => <<"Bob">>,
%                               <<"position">> => <<"CTO">>}],
%         <<"name">> => <<"Foobar, Inc.">>}}

% delete Bob and the location in one call
jsn:delete_list(['company.location', {<<"company">>, <<"employees">>, last}], Company).
% #{<<"company">> =>
%       #{<<"created">> => #{<<"at">> => 469778436,
%                            <<"by">> => <<"00000000">>},
%         <<"employees">> => [#{<<"id">> => <<"00000000">>,
%                               <<"name">> => <<"Alice">>,
%                               <<"position">> => <<"CEO">>}],
%         <<"name">> => <<"Foobar, Inc.">>}}

% conditionally delete the company Location
SecretLocations = [<<"Nevada">>, <<"Luxembourg">>, <<"U.S. Virgin Islands">>].
jsn:delete_if_equal('company.location', SecretLocations, Company).
% #{<<"company">> =>
%       #{<<"created">> => #{<<"at">> => 469778436,
%                            <<"by">> => <<"00000000">>},
%         <<"employees">> => [#{<<"id">> => <<"00000000">>,
%                               <<"name">> => <<"Alice">>,
%                               <<"position">> => <<"CEO">>},
%                             #{<<"id">> => <<"00000001">>,
%                               <<"name">> => <<"Bob">>,
%                               <<"position">> => <<"CTO">>}],
%         <<"name">> => <<"Foobar, Inc.">>}}

copy/3,4 and transform/2 - Re-shaping existing objects

Examples

Source = jsn:new([{'key1', <<"value1">>},
                  {'key2', <<"value2">>},
                  {'key3', <<"value3">>},
                  {'key3', <<"value3">>}]).
% #{<<"key1">> => <<"value1">>,
%   <<"key2">> => <<"value2">>,
%   <<"key3">> => <<"value3">>}

Destination = jsn:new({'key4', <<"value4">>}).
% #{<<"key4">> => <<"value4">>}

% copy some of the paths from source to destination
[NewDestination] = jsn:copy(['key1', 'key2'], Source, Destination).
% [#{<<"key1">> => <<"value1">>,
%    <<"key2">> => <<"value2">>,
%    <<"key4">> => <<"value4">>}]

T1 = fun(<<"value", N/binary>>) -> N end.

% transform all the keys of NewDestination
jsn:transform([{key1, T1},{key2, T1},{key4, T1}], NewDestination).
% #{<<"key1">> => <<"1">>,
%   <<"key2">> => <<"2">>,
%   <<"key4">> => <<"4">>}

equal/3,4 - Path-wise object comparison

Examples

Object1 = jsn:new([{<<"path1">>, <<"thing1">>},
                   {<<"path2">>, <<"thing2">>},
                   {<<"path3">>, <<"thing3">>}]).
% #{<<"path1">> => <<"thing1">>,
%   <<"path2">> => <<"thing2">>,
%   <<"path3">> => <<"thing3">>}

Object2 = jsn:new([{<<"path1">>, <<"thing1">>},
                   {<<"path2">>, <<"notthing2">>}]).
% #{<<"path1">> => <<"thing1">>, <<"path2">> => <<"notthing2">>}

% by path1, these objects are equal
jsn:equal([<<"path1">>], Object1, Object2).
% ok

% by path2, not so much
jsn:equal([<<"path2">>], Object1, Object2).
% {error,{not_equal,<<"mismatch of requested and existing field(s): path2">>}}

% same story with path3
jsn:equal([<<"path3">>], Object1, Object2).
% {error,{not_equal,<<"mismatch of requested and existing field(s): path3">>}}

% but if you use soft mode, path 3 works (because it's missing in Object2)
jsn:equal([<<"path3">>], Object1, Object2, soft).
% ok

% you can also use a list of objects for the 3rd argument.
jsn:equal([<<"path1">>, <<"path3">>], Object1, [Object1, Object2], soft).
% ok

is_equal/2 and is_subset/2 - Boolean object comparison

Examples

Object1 = jsn:new([{<<"path1">>, <<"thing1">>},
                   {<<"path2">>, <<"thing2">>}]).
% #{<<"path1">> => <<"thing1">>,<<"path2">> => <<"thing2">>}

Object2 = jsn:new([{<<"path1">>, <<"thing1">>},
                   {<<"path2">>, <<"thing2">>}], [{format, struct}]).
% {struct,[{<<"path1">>,<<"thing1">>},
%          {<<"path2">>,<<"thing2">>}]}

Object3 = [{path1, <<"thing1">>},
           {path2, <<"thing2">>}].
% [{path1,<<"thing1">>},{path2,<<"thing2">>}]

jsn:is_equal(Object1, Object2).
% true

jsn:is_equal(Object1, Object3).
% true

Object4 = jsn:set(path1, Object1, 1).
% #{<<"path1">> => 1,<<"path2">> => <<"thing2">>}

jsn:is_equal(Object1, Object4).
% false

Object5 = jsn:set_list([{path3, <<"thing3">>}], Object1).
% #{<<"path1">> => <<"thing1">>,
%   <<"path2">> => <<"thing2">>,
%   <<"path3">> => <<"thing3">>}

Object6 = jsn:set_list([{path3, <<"thing3">>}], Object2).
% {struct,[{<<"path3">>,<<"thing3">>},
%          {<<"path1">>,<<"thing1">>},
%          {<<"path2">>,<<"thing2">>}]}

jsn:is_subset(Object1, Object1).
% true

jsn:is_subset(Object1, Object2).
% true

jsn:is_subset(Object1, Object3).
% true

jsn:is_subset(Object3, Object1).
% false

as_map/1, from_map/1,2, as_proplist/1, and from_proplist/1,2 - Object format conversion

Examples

Object1 = jsn:new([{<<"path1">>, <<"thing1">>},
                   {<<"path2">>, <<"thing2">>}]).
% #{<<"path1">> => <<"thing1">>,<<"path2">> => <<"thing2">>}

Object2 = jsn:as_proplist(Object1).
% [{<<"path2">>,<<"thing2">>},{<<"path1">>,<<"thing1">>}]

Object3 = jsn:from_proplist(Object2, [{format, struct}]).
% {struct,[{<<"path2">>,<<"thing2">>},
%          {<<"path1">>,<<"thing1">>}]}

Object1 = jsn:from_proplist(Object2).
% #{<<"path1">> => <<"thing1">>,<<"path2">> => <<"thing2">>}

Object1 = jsn:as_map(Object2).
% #{<<"path1">> => <<"thing1">>,<<"path2">> => <<"thing2">>}

Object2 = jsn:from_map(Object1, [{format, proplist}]).
% [{<<"path2">>,<<"thing2">>},{<<"path1">>,<<"thing1">>}]

Object3 = jsn:from_map(Object1, [{format, struct}]).
% {struct,[{<<"path2">>,<<"thing2">>},
%          {<<"path1">>,<<"thing1">>}]}