Home

Awesome

sqlite-ecosystem

All the SQLite extensions, tools, and guides that I (Alex Garcia šŸ‘‹šŸ¼) have written and open sourced.

Feel free to start a discussion in this repo about meta-level SQLite extension and tooling talk! Though if you're having a specific issue with a particular extension or tool, then file an issue in that project's repository.

Overview

These are all the loadable SQLite extensions I have built, along with which programming language they are written in and where they are distributed.

ExtensionDescriptionLanguagePythonNode.jsDenoRubyDatasettesqlite-utils
sqlite-httpMake HTTP requestsGoāœ…āœ…āœ…āœ…āœ…āœ…
sqlite-htmlparse HTML documentsGoāœ…āœ…āœ…āœ…āœ…āœ…
sqlite-linesRead files/blob line-by-lineCāœ…āœ…āœ…āœ…āœ…āœ…
sqlite-pathParse and generate filepathsCāœ…āœ…āœ…āœ…āœ…āœ…
sqlite-urlParse and generate URLsCāœ…āœ…āœ…āœ…āœ…āœ…
sqlite-xsvQuery CSVsRustāœ…āœ…āœ…āœ…šŸš§šŸš§
sqlite-regexRegular expression functionsRustāœ…āœ…āœ…āœ…āœ…āœ…
sqlite-ulidWork with ULIDsRustāœ…āœ…āœ…āœ…āœ…āœ…
sqlite-jsonschemaValidate JSON objects with JSON SchemaRustāœ…āœ…āœ…āœ…āœ…āœ…
sqlite-fastrandGenerate fast numbers/blobs quicklyRustāœ…āœ…āœ…āœ…āœ…āœ…
sqlite-vssVector search in SQLiteC++āœ…āœ…āœ…āœ…āœ…āœ…

Extensions

sqlite-http

A SQLite extension for making HTTP requests purely in SQL.

select http_get_body('https://text.npr.org/');
select
  http_get_body('https://api.github.com/repos/sqlite/sqlite') ->> '$.description';
Language/PlatformInstall
Pythonpip install sqlite-httpPyPI
Node.jsnpm install sqlite-httpnpm
Denodeno.land/x/sqlite_httpdeno.land/x release
Datasettedatasette install datasette-sqlite-httpPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-httpPyPI
Rubygem install sqlite-httpGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-html

A SQLite extension for querying, manipulating, and creating HTML elements.

select html_extract('<p> Anakin <b>Skywalker</b> </p>', 'b'); -- "<b>Skywalker</b>"
Language/PlatformInstall
Pythonpip install sqlite-htmlPyPI
Node.jsnpm install sqlite-htmlnpm
Denodeno.land/x/sqlite_htmldeno.land/x release
Datasettedatasette install datasette-sqlite-htmlPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-htmlPyPI
Rubygem install sqlite-htmlGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-lines

A SQLite extension for efficiently reading large files or blobs line-by-line.

select line from lines_read('logs.txt');
Language/PlatformInstall
Pythonpip install sqlite-linesPyPI
Node.jsnpm install sqlite-linesnpm
Denodeno.land/x/sqlite_linesdeno.land/x release
Datasettedatasette install datasette-sqlite-linesPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-linesPyPI
Rubygem install sqlite-linesGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-path

A SQLite extension for parsing, generating, and querying paths.

select path_dirname('foo/bar.txt'); -- 'foo/'
select path_basename('foo/bar.txt'); -- 'bar.txt'
select path_extension('foo/bar.txt'); -- '.txt'
Language/PlatformInstall
Pythonpip install sqlite-pathPyPI
Node.jsnpm install sqlite-pathnpm
Denodeno.land/x/sqlite_pathdeno.land/x release
Datasettedatasette install datasette-sqlite-pathPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-pathPyPI
Rubygem install sqlite-pathGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-url

A SQLite extension for parsing, generating, and querying URLs and query strings

select url_valid('https://sqlite.org'); -- 1
select url_scheme('https://www.sqlite.org/vtab.html#usage'); -- 'https'
select url_host('https://www.sqlite.org/vtab.html#usage'); -- 'www.sqlite.org'
Language/PlatformInstall
Pythonpip install sqlite-urlPyPI
Node.jsnpm install sqlite-urlnpm
Denodeno.land/x/sqlite_urldeno.land/x release
Datasettedatasette install datasette-sqlite-urlPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-urlPyPI
Rubygem install sqlite-urlGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-xsv

A fast and performant SQLite extension for CSV files, written in Rust!

create virtual table temp.students using csv(
  filename="students.csv"
);

select * from temp.students;

create virtual table temp.students_gz using csv(
  filename="students.csv.gz"
);

create virtual table temp.students_reader using csv_reader(
  id integer,
  name text,
  age integer,
  progess real
);
select * from temp.students_reader('./target/students.csv');
Language/PlatformInstall
Pythonpip install sqlite-xsvPyPI
Node.jsnpm install sqlite-xsvnpm
Denodeno.land/x/sqlite_xsvdeno.land/x release
Rubygem install sqlite-xsvGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-regex

A fast SQLite extension for regular expressions.

select regex_find(
  '[0-9]{3}-[0-9]{3}-[0-9]{4}',
  'phone: 111-222-3333'
);

select rowid, *
from regex_find_all(
  '\b\w{13}\b',
  'Retroactively relinquishing remunerations is reprehensible.'
);
/*
ā”Œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”¬ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”
ā”‚ rowid ā”‚ start ā”‚ end ā”‚     match     ā”‚
ā”œā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¼ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¼ā”€ā”€ā”€ā”€ā”€ā”¼ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”¤
ā”‚ 0     ā”‚ 0     ā”‚ 13  ā”‚ Retroactively ā”‚
ā”‚ 1     ā”‚ 14    ā”‚ 27  ā”‚ relinquishing ā”‚
ā”‚ 2     ā”‚ 28    ā”‚ 41  ā”‚ remunerations ā”‚
ā”‚ 3     ā”‚ 45    ā”‚ 58  ā”‚ reprehensible ā”‚
ā””ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”“ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”€ā”˜
*/
Language/PlatformInstall
Pythonpip install sqlite-regexPyPI
Node.jsnpm install sqlite-regexnpm
Denodeno.land/x/sqlite_regexdeno.land/x release
Datasettedatasette install datasette-sqlite-regexPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-regexPyPI
Rubygem install sqlite-regexGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-ulid

A SQLite extension for generating and working with ULIDs.

select ulid(); -- '01gqr4j69cc7w1xdbarkcbpq17'
select ulid_bytes(); -- X'0185310899dd7662b8f1e5adf9a5e7c0'
select ulid_with_prefix('invoice'); -- 'invoice_01gqr4jmhxhc92x1kqkpxb8j16'
select ulid_datetime('01gqr4j69cc7w1xdbarkcbpq17') -- '2023-01-26 22:53:20.556'
Language/PlatformInstall
Pythonpip install sqlite-ulidPyPI
Node.jsnpm install sqlite-ulidnpm
Denodeno.land/x/sqlite_uliddeno.land/x release
Datasettedatasette install datasette-sqlite-ulidPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-ulidPyPI
Rubygem install sqlite-ulidGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-jsonschema

A SQLite extension for validating JSON objects with JSON Schema.

select jsonschema_matches('{"maxLength": 5}', json_quote('alex')); -- 1
Language/PlatformInstall
Pythonpip install sqlite-jsonschemaPyPI
Node.jsnpm install sqlite-jsonschemanpm
Denodeno.land/x/sqlite_jsonschemadeno.land/x release
Datasettedatasette install datasette-sqlite-jsonschemaPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-jsonschemaPyPI
Rubygem install sqlite-jsonschemaGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-fastrand

A SQLite extension for quickly generating random numbers, booleans, characters, and blobs

select fastrand_int(); -- 556823563
select fastrand_alphabetic(); -- 's'
select fastrand_uppercase();-- 'M'
select fastrand_double(); -- 0.740834390248454
Language/PlatformInstall
Pythonpip install sqlite-fastrandPyPI
Node.jsnpm install sqlite-fastrandnpm
Denodeno.land/x/sqlite_fastranddeno.land/x release
Datasettedatasette install datasette-sqlite-fastrandPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-fastrandPyPI
Rubygem install sqlite-fastrandGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-vss

A SQLite extension for efficient vector search, based on Faiss!

create virtual table vss_articles using vss0(
  headline_embedding(384),
  description_embedding(384),
);

insert into vss_articles(rowid, headline_embedding)
  select rowid, headline_embedding from articles;

select rowid, distance
from vss_articles
where vss_search(
  headline_embedding,
  (select headline_embedding from articles where rowid = 123)
)
limit 100;
Language/PlatformInstall
Pythonpip install sqlite-vssPyPI
Node.jsnpm install sqlite-vssnpm
Denodeno.land/x/sqlite_vssdeno.land/x release
Datasettedatasette install datasette-sqlite-vssPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-vssPyPI
Rubygem install sqlite-vssGem
Github ReleaseGitHub tag (latest SemVer pre-release)

sqlite-hello

The smallest "Hello world" SQLite extension possible.

sqlite> .load ./hello0
sqlite> select hello('Alex');
'Hello, Alex!'
Language/PlatformInstall
Pythonpip install sqlite-helloPyPI
Node.jsnpm install sqlite-hellonpm
Denodeno.land/x/sqlite_hellodeno.land/x release
Datasettedatasette install datasette-sqlite-helloPyPI
sqlite-utilssqlite-utils install sqlite-utils-sqlite-helloPyPI
Rubygem install sqlite-helloGem
Github ReleaseGitHub tag (latest SemVer pre-release)

Distribution

SQLite extensions have historically been hard to shared with other people. Installing and using them can be tricky and confusing, but these are some strategies I've used to make that easier!

pip for Python Developers

Most of my SQLite extensions are additional placed in a small Python library and distributed on PyPi. Python developers can then install these extensions with a pip install command like so:

pip install sqlite-regex

The actual Python library is small and meant to be used with the builtin sqlite3 Python module, like so:

import sqlite3
import sqlite_regex

db = sqlite3.connect(':memory:')

db.enable_load_extension(True)
sqlite_regex.load(db)

db.execute("select regex_version(), '[abc]' regexp 'a';").fetchone()
# ('v0.1.0', 1)

See _Making SQLite extensions pip install-able _ (February 2023) for more details.

npm for Node.js Developers

These extensions are also distributed on npm for use in Node.js. they can be install with a npm install command like so:

npm install sqlite-regex

And then used in Node.js programs with better-sqlite3 or node-sqlite3 like so:

import Database from "better-sqlite3";
import * as sqlite_regex from "sqlite-regex";

const db = new Database(":memory:");

db.loadExtension(sqlite_regex.getLoadablePath());

const version = db.prepare("select regex_version()").pluck().get();
console.log(version); // "v0.2.0"

See Making SQLite extensions npm install'able for Node.js, and on deno.land/x for Deno (March 2023) for more details.

deno.land/x for Deno Developers

These extensions are also distributed on deno.land/x for use in Deno programs. It's meant to work with the x/sqlite3 client like so:

import { Database } from "https://deno.land/x/sqlite3@0.8.0/mod.ts";
import * as sqlite_regex from "https://deno.land/x/sqlite_regex@v0.2.3-alpha.2/mod.ts";

const db = new Database(":memory:");

db.enableLoadExtension = true;
sqlite_regex.load(db);

const [version] = db.prepare("select regex_version()").value<[string]>()!;

console.log(version);

See the "Deno" section of Making SQLite extensions npm install'able for Node.js, and on deno.land/x for Deno (March 2023) for more details.

gem for ruby Developers

These extensions are also distributed on rubygems.org for use in Ruby applications. It's meant to work with the sqlite3 client like so:

require 'sqlite3'
require 'sqlite_regex'

db = SQLite3::Database.new(':memory:')
db.enable_load_extension(true)
SqliteRegex.load(db)
db.enable_load_extension(false)

result = db.execute("select regex_version(), '[abc]' regexp 'a';")
puts result.first.first # "v0.2.3-alpha.7"
puts result.first.last  # 1

See Making SQLite extension gem install'able for Ruby Developers (June 2023) for more details.

As Datasette Plugins

Most of these SQLite extensions are also distributed as Datasette Plugins. They are small wrappers around their corresponding pip packages.

They can be installed like so:

datasette install datasette-sqlite-regex

Now all future Datasette instances will include sqlite-regex functions. Here we test it using datasette --get.

$ datasette --get '/_memory.csv?sql=select+regex_version()'
regex_version()
v0.1.0

As sqlite-utils Plugins

Most of these SQLite extensions are also distributed as sqlite-utils plugins. They are small wrappers around their corresponding pip packages.

They can be installed like so:

sqlite-utils install sqlite-utils-sqlite-regex

Now when using the sqlite-utils CLI, the sqlite-regex functions will be available.

$ sqlite-utils memory 'select regex_version()'
[{"regex_version()": "v0.2.3"}]

Roadmap

Various extensions and tools that I plan to build and open source in the future! Most of the development of these happen in private repos, to make my life easier.

If you're interested in any of these, just send me a message!

Future SQLite Extensions

Future Extension Bindings

Future Tooling