Awesome

sqlite-http

A SQLite extension for making HTTP requests purely in SQL.

Create GET, POST, and other HTTP requests, like curl, wget, and fetch
Download response bodies, header, status codes, timing info
Set rate limits, timeouts

Usage

.load ./http0
select http_get_body('https://text.npr.org/');
/*
<!DOCTYPE html>
<html lang="en">
<head>
  <title>NPR : National Public Radio</title>
  ....
*/

Query for all custom headers in an endpoint.

select name, value
from http_headers_each(
  http_get_headers('https://api.github.com/')
)
where name like 'X-%';
/*
┌────────────────────────┬────────────────────────────────────┐
│          name          │               value                │
├────────────────────────┼────────────────────────────────────┤
│ X-Ratelimit-Limit      │ 60                                 │
│ X-Ratelimit-Used       │ 8                                  │
│ X-Content-Type-Options │ nosniff                            │
│ X-Github-Media-Type    │ github.v3; format=json             │
│ X-Github-Request-Id    │ CCCA:5FDF:1014BC2:10965F9:62F3DE4E │
│ X-Ratelimit-Remaining  │ 52                                 │
│ X-Ratelimit-Resource   │ core                               │
│ X-Frame-Options        │ deny                               │
│ X-Ratelimit-Reset      │ 1660152798                         │
│ X-Xss-Protection       │ 0                                  │
└────────────────────────┴────────────────────────────────────┘
*/

Scrape data from a JSON endpoint.

select http_get_body('https://api.github.com/repos/sqlite/sqlite')
  ->> '$.description' as description;
/*
┌───────────────────────────────────────────────┐
│                  description                  │
├───────────────────────────────────────────────┤
│ Official Git mirror of the SQLite source tree │
└───────────────────────────────────────────────┘
*/

Pass in specific headers in a request.

select
  value
from json_each(
  http_get_body(
    'https://api.github.com/issues',
    http_headers(
      'Authorization', 'token ghp_16C7e42F292c6912E7710c8'
    )
  )
);

Documentation

See docs.md for a full API reference.

Installing

Language	Install
Python	`pip install sqlite-http`
Datasette	`datasette install datasette-sqlite-http`
Node.js	`npm install sqlite-http`
Deno	`deno.land/x/sqlite_http`
Ruby	`gem install sqlite-http`
Github Release

The Releases page contains pre-built binaries for Linux amd64, MacOS amd64 (no arm), and Windows.

As a loadable extension

If you want to use sqlite-http as a Runtime-loadable extension, Download the http0.dylib (for MacOS), http0.so (Linux), or http0.dll (Windows) file from a release and load it into your SQLite environment.

Note: The 0 in the filename (http0.dylib/ http0.so/http0.dll) denotes the major version of sqlite-http. Currently sqlite-http is pre v1, so expect breaking changes in future versions.

For example, if you are using the SQLite CLI, you can load the library like so:

.load ./http0
select http_version();
-- v0.0.1

Or in Python, using the builtin sqlite3 module:

import sqlite3

con = sqlite3.connect(":memory:")

con.enable_load_extension(True)
con.load_extension("./http0")

print(con.execute("select http_version()").fetchone())
# ('v0.0.1',)

Or in Node.js using better-sqlite3:

const Database = require("better-sqlite3");
const db = new Database(":memory:");

db.loadExtension("./http0");

console.log(db.prepare("select http_version()").get());
// { 'http_version()': 'v0.0.1' }

Or with Datasette, with the "no network" option to limit DDoS attacks: