Home

Awesome

<table style="width:100%"> <tr> <td> <table style="width:100%"> <tr> <td> key </td> <td> value </td> </tr> <tr> <td><a href="https://github.com/sponsors/spacejam">buy a coffee for us to convert into databases</a></td> <td><a href="https://github.com/sponsors/spacejam"><img src="https://img.shields.io/github/sponsors/spacejam"></a></td> </tr> <tr> <td><a href="https://docs.rs/sled">documentation</a></td> <td><a href="https://docs.rs/sled"><img src="https://docs.rs/sled/badge.svg"></a></td> </tr> <tr> <td><a href="https://discord.gg/Z6VsXds">chat about databases with us</a></td> <td><a href="https://discord.gg/Z6VsXds"><img src="https://img.shields.io/discord/509773073294295082.svg?logo=discord"></a></td> </tr> </table> </td> <td> <p align="center"> <img src="https://raw.githubusercontent.com/spacejam/sled/main/art/tree_face_anti-transphobia.png" width="40%" height="auto" /> </p> </td> </tr> </table>

sled - it's all downhill from here!!!

An embedded database.

let tree = sled::open("/tmp/welcome-to-sled")?;

// insert and get, similar to std's BTreeMap
let old_value = tree.insert("key", "value")?;

assert_eq!(
  tree.get(&"key")?,
  Some(sled::IVec::from("value")),
);

// range queries
for kv_result in tree.range("key_1".."key_9") {}

// deletion
let old_value = tree.remove(&"key")?;

// atomic compare and swap
tree.compare_and_swap(
  "key",
  Some("current_value"),
  Some("new_value"),
)?;

// block until all operations are stable on disk
// (flush_async also available to get a Future)
tree.flush()?;

If you would like to work with structured data without paying expensive deserialization costs, check out the structured example!

features

expectations, gotchas, advice

performance

a note on lexicographic ordering and endianness

If you want to store numerical keys in a way that will play nicely with sled's iterators and ordered operations, please remember to store your numerical items in big-endian form. Little endian (the default of many things) will often appear to be doing the right thing until you start working with more than 256 items (more than 1 byte), causing lexicographic ordering of the serialized bytes to diverge from the lexicographic ordering of their deserialized numerical form.

interaction with async

If your dataset resides entirely in cache (achievable at startup by setting the cache to a large enough value and performing a full iteration) then all reads and writes are non-blocking and async-friendly, without needing to use Futures or an async runtime.

To asynchronously suspend your async task on the durability of writes, we support the flush_async method, which returns a Future that your async tasks can await the completion of if they require high durability guarantees and you are willing to pay the latency costs of fsync. Note that sled automatically tries to sync all data to disk several times per second in the background without blocking user threads.

We support async subscription to events that happen on key prefixes, because the Subscriber struct implements Future<Output=Option<Event>>:

let sled = sled::open("my_db").unwrap();

let mut sub = sled.watch_prefix("");

sled.insert(b"a", b"a").unwrap();

extreme::run(async move {
    while let Some(event) = (&mut sub).await {
        println!("got event {:?}", event);
    }
});

minimum supported Rust version (MSRV)

We support Rust 1.62 and up.

architecture

lock-free tree on a lock-free pagecache on a lock-free log. the pagecache scatters partial page fragments across the log, rather than rewriting entire pages at a time as B+ trees for spinning disks historically have. on page reads, we concurrently scatter-gather reads across the log to materialize the page from its fragments. check out the architectural outlook for a more detailed overview of where we're at and where we see things going!

philosophy

  1. don't make the user think. the interface should be obvious.
  2. don't surprise users with performance traps.
  3. don't wake up operators. bring reliability techniques from academia into real-world practice.
  4. don't use so much electricity. our data structures should play to modern hardware's strengths.

known issues, warnings

priorities

  1. A full rewrite of sled's storage subsystem is happening on a modular basis as part of the komora project, in particular the marble storage engine. This will dramatically lower both the disk space usage (space amplification) and garbage collection overhead (write amplification) of sled.
  2. The memory layout of tree nodes is being completely rewritten to reduce fragmentation and eliminate serialization costs.
  3. The merge operator feature will change into a trigger feature that resembles traditional database triggers, allowing state to be modified as part of the same atomic writebatch that triggered it for retaining serializability with reactive semantics.

fund feature development

Like what we're doing? Help us out via GitHub Sponsors!