Home

Awesome

Lakekeeper Catalog for Apache Iceberg

License Unittests Spark Integration Pyiceberg Integration Trino Integration Starrocks Integration Artifact Hub Docker on quay Helm Chart Discord

This is Lakekeeper: A Rust-native implementation of the Apache Iceberg REST Catalog specification based on apache/iceberg-rust.

If you have questions, feature requests or just want a chat, we are hanging around in Discord!

<p align="center"> <img src="https://github.com/lakekeeper/lakekeeper/raw/main/assets/Lakekeeper-Overview.png" width="500"> </p>

Next Steps

The catalog is evolving quickly. Especially internal rust APIs are not stable and subject to change. External REST APIs are kept as stable as possible, especially the /catalog API is stable as of today.

Quickstart

A Docker Container is available on quay.io. We have prepared a self-contained docker-compose file to demonstrate the usage of spark with our catalog:

git clone https://github.com/lakekeeper/lakekeeper.git
cd lakekeeper/examples/self-contained
docker compose up

Then open your browser and head to localhost:8888.

For more information on deployment, please check the User Guide.

Scope and Features

The Iceberg Catalog REST interface has become the standard for catalogs in open Lakehouses. It natively enables multi-table commits, server-side deconflicting and much more. It is figuratively the (TIP) of the Iceberg.

We have started this implementation because we were missing customizability, support for on-premise deployments and other features that are important for us in existing Iceberg Catalogs. Please find following some of our focuses with this implementation:

Please find following an overview of currently supported features. Please also check the Issues if you are missing something.

Status

Supported Operations - Iceberg-Rest

OperationStatusDescription
NamespacedoneAll operations implemented
TabledoneAll operations implemented - additional integration tests in development
ViewsdoneRemove unused files and log entries
MetricsopenEndpoint is available but doesn't store the metrics

Storage Profile Support

StorageStatusComment
S3 - AWSsemi-donevended-credentials & remote-signing, assume role missing
S3 - Customdonevended-credentials & remote-signing, tested against minio
Azure ADLS Gen2done
Azure Blobopen
Microsoft OneLakeopen
Google Cloud Storagedone

Details on how to configure the storage profiles can be found in the Storage Guide.

Supported Catalog Backends

BackendStatusComment
Postgresdone
MongoDBopen

Supported Secret Stores

BackendStatusComment
Postgresdone
kv2 (hcp-vault)doneuserpass auth

Supported Event Stores

BackendStatusComment
Natsdone
Kafkaopen

Supported Operations - Management API

OperationStatusDescription
Warehouse ManagementdoneCreate / Update / Delete a Warehouse
AuthZopenManage access to warehouses, namespaces and tables
More to come!open

Auth(N/Z) Handlers

OperationStatusDescription
OIDC (AuthN)doneSecure access to the catalog via OIDC
Custom (AuthZ)doneIf you are willing to implement a single rust Trait, the AuthZHandler can be implement to connect to your system
OpenFGA (AuthZ)openInternal Authorization management

Multiple Projects / Multi-Tenancy

Lakekeeper can host multiple independent projects that each host multiple warehouses. The overall structure looks like this:

<project-1-uuid>/
├─ foo-warehouse
├─ bar-warehouse
<project-2-uuid>/
├─ foo-warehouse
├─ bas-warehouse

Each Project in Lakekeeper corresponds to a distinct Application Object within your IdP, enabling genuine Multi-Tenancy. A single instance of Lakekeeper can host multiple Projects, ensuring separation at all levels, including Authentication. Our Multi-Tenancy definition is broader than most catalogs. While other catalogs may consider hosting multiple warehouses under a single IdP Application as Multi-Tenancy, a single Lakekeeper Project can achieve that and more.

As this additional layer isn't necessary for most deployments, the LAKEKEEPER__ENABLE_DEFAULT_PROJECT configuration option is set to true by default. This setting effectively disables multi-tenancy by defaulting to a project with the NIL UUID ("00000000-0000-0000-0000-000000000000") as the project-id.

To enable multi-project setups, simply set LAKEKEEPER__ENABLE_DEFAULT_PROJECT to false. Multi-project setups require additional configuration to include project-ids in HTTP headers and separate authentication audiences. This documentation does not cover these steps yet. Of course, hosting multiple warehouses within a single Lakekeeper Project is also possible.

Undrop Tables

When a table or view is dropped, it is not immediately deleted from the catalog. Instead, it is marked as dropped and a job for its cleanup is scheduled. The table, including its data if purgeRequested=True, is then deleted after the configured expiration delay has passed. This will allow for a recovery of tables that have been dropped by accident.

Undrop of a table is only possible if soft-deletes are enabled for a Warehouse.

Configuration

The basic setup of the Catalog is configured via environment variables. As this catalog supports a multi-tenant setup, each catalog ("warehouse") also comes with its own configuration options including its Storage Configuration. The documentation of the Management-API for warehouses is hosted at the unprotected /swagger-ui endpoint.

Following options are global and apply to all warehouses:

General

Previous to Lakekeeper Version 0.5.0 please prefix all environment variables with ICEBERG_REST__ instead of LAKEKEEPER__.

VariableExampleDescription
LAKEKEEPER__BASE_URIhttps://example.com:8080Base URL where the catalog is externally reachable. Default: https://localhost:8080
LAKEKEEPER__ENABLE_DEFAULT_PROJECTtrueIf true, the NIL Project ID ("00000000-0000-0000-0000-000000000000") is used as a default if the user does not specify a project when connecting. This option is enabled by default, which we recommend for all single-project (single-tenant) setups. Default: true.
LAKEKEEPER__RESERVED_NAMESPACESsystem,examplesReserved Namespaces that cannot be created via the REST interface
LAKEKEEPER__METRICS_PORT9000Port where the metrics endpoint is reachable. Default: 9000
LAKEKEEPER__LISTEN_PORT8080Port the server listens on. Default: 8080
LAKEKEEPER__SECRET_BACKENDpostgresThe secret backend to use. If kv2 is chosen, you need to provide additional parameters found under Default: postgres, one-of: [postgres, kv2]

Self-signed certificates in dependencies (e.g. minio)

You may be running Lakekeeper in your own environment which uses self-signed certificates for e.g. minio. Lakekeeper is built with reqwest's rustls-tls-native-roots feature activated, this means SSL_CERT_FILE and SSL_CERT_DIR are respected. If both are not set, the system's default CA store is used. If you want to use a custom CA store, set SSL_CERT_FILE to the path of the CA file or SSL_CERT_DIR to the path of the CA directory. The certificate used by the server cannot be a CA. It needs to be an end entity certificate, else you may run into CaUsedAsEndEntity errors.

Task queues

Currently, the catalog uses two task queues, one to ultimately delete soft-deleted tabulars and another to purge tabulars which have been deleted with the purgeRequested=True query parameter. The task queues are configured as follows:

VariableExampleDescription
LAKEKEEPER__QUEUE_CONFIG__MAX_RETRIES5Number of retries before a task is considered failed Default: 5
LAKEKEEPER__QUEUE_CONFIG__MAX_AGE3600Amount of seconds before a task is considered stale and could be picked up by another worker. Default: 3600
LAKEKEEPER__QUEUE_CONFIG__POLL_INTERVAL10Amount of seconds between polling for new tasks. Default: 10

The queues are currently implemented using the sqlx Postgres backend. If you want to use a different backend, you need to implement the TaskQueue trait.

Postgres

Configuration parameters if Postgres is used as a backend, you may either provide connection strings or use the PG_* environment variables, connection strings take precedence:

VariableExampleDescription
LAKEKEEPER__PG_DATABASE_URL_READpostgres://postgres:password@localhost:5432/icebergPostgres Database connection string used for reading
LAKEKEEPER__PG_DATABASE_URL_WRITEpostgres://postgres:password@localhost:5432/icebergPostgres Database connection string used for writing.
LAKEKEEPER__PG_ENCRYPTION_KEY<This is unsafe, please set a proper key>If LAKEKEEPER__SECRET_BACKEND=postgres, this key is used to encrypt secrets. It is required to change this for production deployments.
LAKEKEEPER__PG_READ_POOL_CONNECTIONS10Number of connections in the read pool
LAKEKEEPER__PG_WRITE_POOL_CONNECTIONS5Number of connections in the write pool
LAKEKEEPER__PG_HOST_RlocalhostHostname for read operations
LAKEKEEPER__PG_HOST_WlocalhostHostname for write operations
LAKEKEEPER__PG_PORT5432Port number
LAKEKEEPER__PG_USERpostgresUsername for authentication
LAKEKEEPER__PG_PASSWORDpasswordPassword for authentication
LAKEKEEPER__PG_DATABASEicebergDatabase name
LAKEKEEPER__PG_SSL_MODErequireSSL mode (disable, allow, prefer, require)
LAKEKEEPER__PG_SSL_ROOT_CERT/path/to/root/certPath to SSL root certificate
LAKEKEEPER__PG_ENABLE_STATEMENT_LOGGINGtrueEnable SQL statement logging
LAKEKEEPER__PG_TEST_BEFORE_ACQUIREtrueTest connections before acquiring from the pool
LAKEKEEPER__PG_CONNECTION_MAX_LIFETIME1800Maximum lifetime of connections in seconds

KV2 (HCP Vault)

Configuration parameters if a KV2 compatible storage is used as a backend. Currently, we only support the userpass authentication method. You may provide the envs as single values like LAKEKEEPER__KV2__URL=http://vault.local etc. or as a compound value like: LAKEKEEPER__KV2='{url="http://localhost:1234", user="test", password="test", secret_mount="secret"}'

VariableExampleDescription
LAKEKEEPER__KV2__URLhttps://vault.localURL of the KV2 backend
LAKEKEEPER__KV2__USERadminUsername to authenticate against the KV2 backend
LAKEKEEPER__KV2__PASSWORDpasswordPassword to authenticate against the KV2 backend
LAKEKEEPER__KV2__SECRET_MOUNTkv/data/icebergPath to the secret mount in the KV2 backend

Nats

If you want the server to publish events to a NATS server, set the following environment variables:

VariableExampleDescription
LAKEKEEPER__NATS_ADDRESSnats://localhost:4222The URL of the NATS server to connect to
LAKEKEEPER__NATS_TOPICicebergThe subject to publish events to
LAKEKEEPER__NATS_USERtest-userUser to authenticate against nats, needs LAKEKEEPER__NATS_PASSWORD
LAKEKEEPER__NATS_PASSWORDtest-passwordPassword to authenticate against nats, needs LAKEKEEPER__NATS_USER
LAKEKEEPER__NATS_CREDS_FILE/path/to/file.credsPath to a file containing nats credentials
LAKEKEEPER__NATS_TOKENxyzNats token to authenticate against server

OpenID Connect

If you want to limit ac cess to the API, set LAKEKEEPER__OPENID_PROVIDER_URI to the URI of your OpenID Connect Provider. The catalog will then verify access tokens against this provider. The provider must have the .well-known/openid-configuration endpoint under ${LAKEKEEPER__OPENID_PROVIDER_URI}/.well-known/openid-configuration and the openid-configuration needs to have the jwks_uri and issuer defined.

If LAKEKEEPER__OPENID_PROVIDER_URI is set, every request needs have an authorization header, e.g.

curl {your-catalog-url}/catalog/v1/transactions/commit -X POST -H "authorization: Bearer {your-token-here}" -H "content-type: application/json" -d ...
VariableExampleDescription
LAKEKEEPER__OPENID_PROVIDER_URIhttps://keycloak.local/realms/{your-realm}OpenID Provider URL, with keycloak this is the url pointing to your realm, for Azure App Registration it would be something like https://login.microsoftonline.com/{your-tenant-id-here}/v2.0/. If this variable is not set, endpoints are not secured

License

Licensed under the Apache License, Version 2.0