Awesome

Datadex is a fully open-source, serverless, and local-first Data Platform that improves how communities collaborate on Open Data. Datadex is not a new tool, it is a pattern showing an opinionated bridge between existing ones.

🚀 Implementations

Check other real-world production Open Data Portals of the Datadex pattern in the following repositories:

• LUNG-SARG. The Open Data Platform for Sustainable, Accessible Lung Radiogenomics.
• Datania. An Open Data Platform at national level that unifies and harmonizes information from different sources.
• Gitcoin Grants Data Portal. A Data hub for Gitcoin Grants data and related models.
• Filecoin Data Portal. A data portal for data related to the Filecoin network and ecosystem.

💡 Principles

Make Open Data compatible with the Modern Data Ecosystem.

• Open: Code, standards, infrastructure, and data, all public and open source. Rely on open source tools, standards, public infrastructure, and accessible data formats.
• Modular and Interoperable: Easy to replace, extend or remove components of the pattern. Environment flexibility (your laptop, in a cluster, or from the browser) when running and when deploying (S3 + GH Pages, IPFS, Hugging Face).
• Permissionless: Any improvement is one Pull Request away. Update pipelines, add datasets, or improve documentation. When consuming, there are no API limits, just plain files.
• Data as Code: Reproducible datasets with declarative stateless transformations tracked in git. Data is versioned alongside the code. Models are reusable, packaged, and versioned.
• Glue: Be a bridge between tools and approaches. E.g: Use software engineering good practices like types, tests, materialized views, and more.

⚙️ Setup

Datadex is mainly a Python project, so you'll need to have Python installed. If you hit any issue, please open an issue! The easiest way to get started is using a Python virtual environment, but a development container is also provided.

🐍 Python Virtual Environment

The recommended way is to install uv and let it manage the Python environment. The following commands will install the dependencies and create a virtual environment in the project's folder.

make setup

Alternatively, you can rely on your system's Python installation to create a virtual environment and install the dependencies.

# Create a virtual environment
python3 -m venv .venv
source .venv/bin/activate

# Install the package and dependencies
pip install -e ".[dev]"

Now, you should be able to spin up Dagster UI (make dev or dagster dev) and access it locally.

🐳 Docker / Dev Containers

You can use VSCode Remote Containers to get started with Datadex too. If you have Docker running, open the project in VSCode and click on the bottom right corner to open the project in a container.

Once inside the develpment environment, you'll only need to run make dev to spin up the Dagster UI locally. You'll also have a few extra extensions installed and configured to work with the project.

The development environment can also run in your browser thanks to GitHub Codespaces!

📜 License

Datadex is licensed under the MIT License. See the LICENSE file for details.