Home

Awesome

A standard library for mattn/go-sqlite3

As an alternative to compiling C extensions like extension-functions.c and sqlean into mattn/go-sqlite3, this package implements many of these functions (and more from PostgreSQL) in Go.

These are in addition to all builtin functions provided by SQLite.

Continue reading for all functions, notes and examples.

Why would I use this?

This library is used in DataStation and dsq to simplify and power data analysis in SQL.

Analyzing logs with SQL in DataStation

Read the DataStation blog post to better understand the background.

Example

package main

import (
	"fmt"
	"database/sql"

	_ "github.com/mattn/go-sqlite3"
	stdlib "github.com/multiprocessio/go-sqlite3-stdlib"
)

func main() {
	stdlib.Register("sqlite3_ext")
	db, err := sql.Open("sqlite3_ext", ":memory:")
	if err != nil {
		panic(err)
	}

	var s string
	err = db.QueryRow("SELECT repeat('x', 2)").Scan(&s)
	if err != nil {
		panic(err)
	}

	fmt.Println(s)
}

Alternatively if you want to be able to add your own additional extensions you can just use the ConnectHook:

package main

import (
	"database/sql"
	"fmt"

	sqlite3 "github.com/mattn/go-sqlite3"
	stdlib "github.com/multiprocessio/go-sqlite3-stdlib"
)

func main() {
	sql.Register("sqlite3_ext",
		&sqlite3.SQLiteDriver{
			ConnectHook: stdlib.ConnectHook,
		})
	db, err := sql.Open("sqlite3_ext", ":memory:")
	if err != nil {
		panic(err)
	}

	var s string
	err = db.QueryRow("SELECT repeat('x', 2)").Scan(&s)
	if err != nil {
		panic(err)
	}

	fmt.Println(s)
}

Functions

Strings

Name(s)NotesExample
repeat, replicaterepeat('f', 5) = 'fffff'
strpos, charindex0-indexed position of substring in stringstrpos('abc', 'b') = 1
reversereverse('abc') = 'cba'
lpadOmit the third argument to default to padding with spaceslpad('22', 3, '0') = '022'
rpadOmit the third argument to default to padding with spacesrpad('22', 3, '0') = '220'
lenShorthand for lengthlen('my string') = '9'
split_partSplit string an take nth split piecesplit('1,2,3', ',', 0) = '1', split('1,2,3', ',' -1) = '3'
regexpGo's regexp package, not PCREx REGEXP '[a-z]+$', REGEXP('[a-z]+$', x)
regexp_countNumber of times the regexp matches in stringregexp_count('abc1', '[a-z]1') = '1'
regexp_split_partRegexp equivalent of split_partregexp_split_part('ab12', '[a-z]1', 0) = 'a'

Aggregation

Most of these are implemented as bindings to gonum.

Name(s)NotesExample
stddev, stdev, stddev_popstddev(n)
modemode(n)
medianmedian(n)
percentile, percDiscreteperc(response_time, 95)
percentile_25, perc_25, percentile_50, perc_50, percentile_75, perc_75, percentile_90, perc_90, percentile_95, perc_95, percentile_99, perc_99Discreteperc_99(response_time)
percentile_cont, perc_contContinuousperc_cont(response_time, 95)
percentile_cont_25, perc_cont_25, percentile_cont_50, perc_cont_50, percentile_cont_75, perc_cont_75, percentile_cont_90, perc_cont_90, percentile_cont_95, perc_cont_95, percentile_cont_99, perc_cont_99Continuousperc_cont_99(response_time)

Net

Name(s)NotesExample
url_schemeurl_scheme('https://x.com:90/home.html') = 'https'
url_hosturl_host('https://x.com:90/home.html') = 'x.com:90'
url_porturl_port('https://x.com:90/home.html') = '90'
url_pathurl_path('https://x.com/some/path.html?p=123') = '/some/path.html'
url_paramurl_param('https://x.com/home.html?p=123&z=%5B1%2C2%5D#section-1', 'z') = '[1,2]'
url_fragmenturl_fragment('https://x.com/home.html?p=123&z=%5B1%2C2%5D#section-1') = 'section-1'

Date

Best effort family of date parsing (uses dateparse) and date part retrieval. Results will differ depending on your computer's timezone.

Name(s)NotesExample
date_yeardate_year('2021-04-05') = 2021
date_monthJanuary is 1, not 0date_month('May 6, 2021') = 5
date_daydate_day('May 6, 2021') = 6
date_yeardayDay offset in yeardate_yearday('May 6, 2021') = 127
date_hour24-hourdate_hour('May 6, 2021 4:50 PM') = 16
date_minutedate_minute('May 6, 2021 4:50') = 50
date_seconddate_second('May 6, 2021 4:50:20') = 20
date_unixdate_unix('May 6, 2021 4:50:20') = 1588740620
date_rfc3339date_rfc3339('May 6, 2021 4:50:20') = 2020-05-06T04:50:20Z

Math

Name(s)NotesExample
acosacos(n)
acoshacosh(n)
asinasin(n)
asinhasinh(n)
atanatan(n)
atanhatanh(n)
ceil, ceilingceil(n)
cosceil(n)
coshcosh(n)
degreesdegrees(radians)
expe^nexp(n)
floorfloor(n)
ln, loglog(x)
log10log10(x)
log2log2(x)
modmod(num, denom)
pipi()
pow, powerpow(base, exp)
radiansradians(degrees)
sinsin(n)
sinhsinh(n)
sqrtsqrt(n)
tantan(n)
tanhtanh(n)
trunc, truncateRounds up to zero if negative, down to zero if positive.trunc(-10.9) = -10, trunc(10.4) = 10.0

Encoding

Name(s)NotesExample
base64Convert string to base64base64(s)
from_base64Convert string from base64from_base64(s)
base32Convert string to base32base32(s)
from_base32Convert string from base32from_base32(s)
md5Hex md5 sum of stringmd5(s)
sha1Hex sha1 sum of stringsha1(s)
sha256Hex sha256 sum of stringsha256(s)
sha512Hex sha512 sum of stringsha512(s)
sha3_256Hex sha3_256 sum of stringsha3_256(s)
sha3_512Hex sha3_512 sum of stringsha3_512(s)
blake2b_256Hex blake2b_256 sum of stringblake2b_256(s)
blake2b_512Hex blake2b_512 sum of stringblake2b_512(s)

How is this tested?

There is 95% test coverage and automated tests on Windows, macOS and Linux.

I just want to use it as a CLI or GUI

See dsq (a command-line tool for executing SQL on data files) and DataStation, a GUI application for querying and building reports with data from databases, servers, and files.

Contribute

Join the #dev channel on the Multiprocess Labs Discord.

If you have an idea for a new function, say so on the Discord channel or open an issue here.

Make sure the function doesn't already exist in dsq (or the sqlite3 CLI).

License

This software is licensed under an Apache 2.0 license.