Awesome
<img align="centre" alt="overlapping arrows" height="92" src="icons/mabel.svg" />mabel is a Data Engineering platform designed to run in serverless environments.
mabel just runs when you need it, scaling to zero, making it efficient and ideal for deployments to platforms like Kubernetes, GCP Cloud Run, AWS Fargate and Knative.
- Documentation GitHub Wiki
- Bug Reports GitHub Issues
- Feature Requests GitHub Issues
- Source Code GitHub
- Discussions GitHub Discussions
Focus on What Matters
We've built mabel to enable Data Analysts to write complex data engineering tasks quickly and easily, so they could get on with doing what they do best.
from mabel import Reader
data = Reader(dataset="test_data")
print(data.count())
Key Features
- On-the-fly compression
- Low-memory requirements, even with terabytes of data
- Indexing and partitioning of data for fast reads
- Cursors for tracking reading position between processes
- Partial SQL DQL (Data Query Language) support
- Schema and data_expectations validation
Installation
From PyPI (recommended)
pip install --upgrade mabel
From GitHub
pip install --upgrade git+https://github.com/mabel-dev/mabel
Guides
Dependencies
- orjson for JSON (de)serialization
- orso for data Schemas
- zstandard for real-time on disk compression
- LZ4 for real-time in memory compression
There are a number of optional dependencies which are usually only required for specific features and functionality. These are listed in tests/requirements.txt.
Integrations
mabel comes with adapters for the following data services:
| Service | |
|---|---|
| <img align="centre" alt="GCP Storage" height="48" src="icons/gcs-logo.png" /> | Google Cloud Storage |
| <img align="centre" alt="MinIo" height="48" src="icons/minio-logo.png" /> | MinIO |
| <img align="centre" alt="AWS S3" height="48" src="icons/s3-logo.png" /> | AWS S3 |
| <img align="centre" alt="Azure" height="48" src="icons/azure.svg" /> | Azure Blob Storage |
| <img align="centre" alt="Local" height="48" src="icons/local-storage.png" /> | Local Storage |
Mabel is extensible with adapters for other data services as required.
Deployment and Execution
mabel supports running on a range of platforms, including:
| Platform | |
|---|---|
| <img align="centre" alt="Docker" height="48" src="icons/docker-logo.png" /> | Docker |
| <img align="centre" alt="Kubernetes" height="48" src="icons/kubernetes-logo.svg" /> | Kubernetes |
| <img align="centre" alt="Windows" height="48" src="icons/windows-logo.png" /> | Windows (<img align="centre" alt="Notice" height="12" src="icons/note.svg" />1) |
| <img align="centre" alt="Linux" height="48" src="icons/linux-logo.jpg" /> | Linux (<img align="centre" alt="Notice" height="12" src="icons/note.svg" />2) |
| <img align="centre" alt="Linux" height="48" src="icons/mac-os.png" /> | Mac (<img align="centre" alt="Notice" height="12" src="icons/note.svg" />3) |
<img align="centre" alt="Notice" height="12" src="icons/note.svg" />1 - Some non-core features are not available on Windows.
<img align="centre" alt="Notice" height="12" src="icons/note.svg" />2 - Tested on Debian (WSL) and Ubuntu.
<img align="centre" alt="Notice" height="12" src="icons/note.svg" />3 - Tested on Apple Silicon Macs.
How Can I Contribute?
All contributions, bug reports, bug fixes, documentation improvements, enhancements, and ideas are welcome.
If you have a suggestion for an improvement or a bug, raise a ticket or start a discussion.
Want to help build mabel? See the contribution guidance.
License
