Home

Awesome

LQN - Lisp Query Notation

About

LQN is a Common Lisp libary, query language and terminal utility to query and transform text files such as JSON and CSV, as well as Lisp data (LDN), The terminal utilities will parse the input data to internal lisp structures according to the input mode. Then the lqn query language can be used for queries and transformations.

lqn started as an experiment and programming exercise. But it has turned into a little language i find rather useful. Both in the terminal, and more interestingly, as a meta language for writing macros in CL. The main purpose of the design is to make something that is intuitive, terse, yet flexible enough that you can write generic CL if you need to. I also wanted to make something that requres a relatively simple compiler.

Here is a small tutorial: https://inconvergent.net/2024/lisp-query-notation/.

An expanded version of the tutorial can be seen in this paper: https://zenodo.org/records/11001584

When using LQN on the terminal there are three terminal commands, or input modes: jqn, tqn and lqn. For JSON, text and lisp data respectively. (For installation see below.) You can find some terminal command examples in bin/lqn-sh.lisp, bin/jqn-sh.lisp, and bin/tqn-sh.lisp.

Symbol documentation can be seen in docs/lqn.md.

Object Representation

Internally JSON arrays/lists are represented as vectors. and JSON objects/dicts are represented as hash-tables (ht). Thus a text file is a vector of strings.We use object in the context of Operators and other LQN utilities to refer to either a vector or a ht. Lisp data is read directly.

Operators

The following operators have special behaviour. You can also write generic CL code in almost all contexts, as we demonstrate soon. In operators we use _ to refer to the current value.

In the following sections [d] represents an optional default value. E.g. if key/index is missing, or if a functon would otherwise return nil. k is an initial counter value. Whereas .. means that there can be arbitrary arguments/expr. expr denotes any expression or operator; like (+ 1 _) or #[:id].

Strings and :keywords

In operators, and several functions, :keywords can be used to represent lowercase strings. This is useful in the terminal to avoid escaping strings. Particularly when using Selector operators. You can use "Strings" instead, if you need case or whitespace.

Pipe Operator - ||

(|| expr ..) pipes the results from the first expr to the second, and so on. Returns the result of the last expr. The Pipe operator surrounds all queries by default. So it is usually not neccessary to use it explicitly.

For convenience the pipe has the following default translations:

Get Operator - @

Select :keys, indexes or paths from nested structure:

Paths support wildcards (*) and numerical indices for nested structures. E.g. this is a valid path: :*/0/things.

Map Operator - #()

Map operations over vector; or over the values of a ht:

Selector Operators - {}, #{}, #[]

Select from one structure into a new data structure. using selectors:

A selector is a triple (mode key expr). Only key is required. If expr is not provided the expr is _, that is: the value of the key. The modes are as follows:

Selectors can either be written out in full, or they can be be written in short form depending on what you want to achieve. The @ in the following examples is used to append a mode to a key without having to wrap the Selector in parenthesis. If you need eg. case or spaces you can use "strings". Here are some examples using {}. It behaves the same for the other Selector operators:

{_}               ; select all keys.
{_ :-@key1}       ; select all keys except "key1".
{:key1 "Key2"}    ; select "key1" and "Key2".
{:+@key}          ; same as :key [+ mode is default].
{"+@Key"}         ; select "Key".
{:?@key }         ; select "key" if the value is not nil.
{(:%@key expr)}   ; select "key" if expr is not nil.
{("?@Key" expr)}  ; select "Key" if the value is not nil.
{("%@Key" expr)}  ; select "Key" if expr is not nil.
{(:+ "Key" expr)} ; same as ("+@Key" expr).

; Use `_` in `expr` to refer to the value of the selected key:
{(:key1 sup))          ; convert value of "key1" to uppercase
 (:key3 (or _ "That")) ; select the value of "key3", or literally "That".
 (:key2 (+ 33 _))}     ; add 33 to value of "key2"

; override and drop keys:
{_                ; select all keys, then override these:
 (:key2 (sdwn _)) ; lowercase the value of "key2"
  :-@key3}        ; drop "key3"

We use {} in the examples but all Selector operators have the same behaviour.

Filter Operator - [] TODO TODO TODO

Filter vector; or the values of a ht:

The filter operator behaves somewhat similar to the Selector operators. They are used with [], ?srch, ?xpr, ?txpr, ?mxpr operators. The modes behave like this:

If this is not what you need, you can compose boolean expressions with regular CL boolen operators. Here are some examples:

[:hello]               ; strings containing "hello".
[:hi "Hello"]          ; strings containing either "Hello" OR "hi".
[:+@hi :+@hello]       ; strings containing "hi" AND "hello".
[:+@hi :+@hello "OH"]  ; strings containing ("hi" AND "hello") OR "OH".
[int!?]                ; items that can be parsed as int.
[(> _ 3)]              ; numbers larger than 3.
[_ :-@hi]              ; strings except those that contain "hi".
[(+@pref? _ "start")   ; strings that start with "start" and end with "end".
 (+@post? _ "end")]
[(fx1 _)]              ; items where this expression is not nil.
[(or (fx1 _) (fx2 _))] ; ...

Fold Operator - ?fld

Reduce vector; or the values of a ht:

Group by Operator - ?grp

Group input into a new ht:

Recursion Operator - ?rec

Repeat the same expression while something is true:

Search Operator - ?srch

Iterate a datastructure (as if with ?txpr) and collect the matches in a new vector:

Transformer Operators - ?xpr, ?txpr, ?mxpr

Perform operation when pattern or condition is satisfied:

Recursively traverse a nested structure of sequences and hts and return a new value for each match:

Query Utilities

The internal representation of in lqn means you can use the regular CL utilities such as gethash, aref, subseq, length etc. But for convenience there are some utility functions/macros in defined in lqn. Some of them are described below. There are more in the documentation.

Global Query Context Fxs

Defined in the query scope:

Operator Context Fxs

Defined in all operators:

Generic Utilities

General utilities:

Hash-table / Strings / Vectors / Sequences

For all sequences and hts:

Make or join hts:

Primarily for sequences (string, vector, list):

Primarily for string searching. [i] means case insensitive:

String maniuplation:

Type Coercion and Tests

(is? o [d]) returns o if not nil, empty sequence, or empty ht; or d.

These functions return the argument if the argument is the corresponding type: flt?, int?, ht?, lst?, num?, str?, vec?, seq?.

These functions return the argument parsed as the corresponding type if possible; otherwise they return the optional second argument: int!?, flt!?, num!?, str!?, vec!?, seq!?.

The following functions will coerce the argument, or fail if the coercion is not supported: str!, int!, flt!, lst! sym!,

Install

lqn requires SBCL. And is pretty easy to install via quicklisp. SBCL is available in most package managers. And you can get quicklisp at https://www.quicklisp.org/beta/. Make sure lqn is available in your quicklisp local-projects folder. Mine is at ~/quicklisp/local-projects/.

Then create an alias for SBCL to execute shell wrappers e.g:

alias jqn="sbcl --script ~/path/to/lqn/bin/jqn-sh.lisp"
alias tqn="sbcl --script ~/path/to/lqn/bin/tqn-sh.lisp"
alias lqn="sbcl --script ~/path/to/lqn/bin/lqn-sh.lisp"

Unfortunately this will tend to have a high startup time. To make it run faster you can create an SBCL image/core that has lqn preloaded and dump it using sb-ext:save-lisp-and-die. Then use the core in the alias instead of SBCL.

is an example script for creating your own core. You can also preload your own libraries which will be available to lqn.

You can see an example bash script for making your own core herebin/core.sh