Awesome
<h1 align="center"> <a target="_blank" href="https://mage.ai" > <img align="center" alt="Mage" src="https://github.com/mage-ai/assets/blob/main/mascots/mascots-shorter.jpeg?raw=true" style="width:100%;" /> </a> </h1> <p align="center"> ๐ง A modern replacement for Airflow. </p> <p align="center"> <a href="https://docs.mage.ai" target="_blank" ><b>Documentation</b></a> ๐ช๏ธ <a href="https://youtu.be/GswOdShLGmg" target="_blank" ><b>Get a 5 min overview</b></a> ๐ <a href="https://demo.mage.ai" target="_blank" ><b>Play with live tool</b></a> ๐ฅ <a href="https://www.mage.ai/chat" target="_blank" > <b>Get instant help</b> </a> </p> <div align="center"> <a href="https://pypi.org/project/mage-ai/" target="_blank" > <img alt="PyPi" src="https://img.shields.io/pypi/v/mage-ai?color=orange" /> </a> <a href="https://anaconda.org/conda-forge/mage-ai" target="_blank" > <img src="https://anaconda.org/conda-forge/mage-ai/badges/version.svg" /> </a> <a href="https://opensource.org/licenses/Apache-2.0" target="_blank" > <img alt="License" src="https://img.shields.io/github/license/mage-ai/mage-ai?color=red" /> </a> <a href="https://www.mage.ai/chat" target="_blank" > <img alt="Slack" src="https://img.shields.io/badge/Slack-Join%20Slack-blueviolet?logo=slack" /> </a> <a href="https://github.com/mage-ai/mage-ai" target="_blank" > <img alt="GitHub Stars" src="https://img.shields.io/github/stars/mage-ai/mage-ai?logo=github"> </a> <a href="https://hub.docker.com/r/mageai/mageai" target="_blank" > <img alt="Docker pulls" src="https://img.shields.io/docker/pulls/mageai/mageai.svg"> </a> <a href="https://pepy.tech/project/mage-ai" target="_blank" > <img alt="pip installs" src="https://static.pepy.tech/personalized-badge/mage-ai?period=total&units=international_system&left_color=grey&right_color=blue&left_text=pip%20installs"> </a> </div> <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=b3c96d79-b8f0-414b-a687-8bfc164b4b7a" /> <div align="center">Give your data team magical
powers
</div>
<p align="center">
<b>Integrate</b> and synchronize data from 3rd party sources
</p>
<p align="center">
Build real-time and batch pipelines to <b>transform</b> data using Python, SQL, and R
</p>
<p align="center">
Run, monitor, and <b>orchestrate</b> thousands of pipelines without losing sleep
</p>
<br />
<p align="center">1๏ธโฃ ๐๏ธ</p>
<h1 align="center">Build</h1>
<p align="center">
Have you met anyone who said they loved developing in Airflow?
<br />
Thatโs why we designed an easy developer experience that youโll enjoy.
</p>
<b>Easy developer experience</b><br />Start developing locally with a single command or launch a dev environment in your cloud using Terraform.<br /><br/><b>Language of choice</b><br />Write code in Python, SQL, or R in the same data pipeline for ultimate flexibility.<br /><br /><b>Engineering best practices built-in</b><br />Each step in your pipeline is a standalone file containing modular code thatโs reusable and testable with data validations. No more DAGs with spaghetti code. | <img src="https://github.com/mage-ai/assets/blob/main/overview/mage-build.gif?raw=true" /> |
<b>Interactive code</b><br />Immediately see results from your codeโs output with an interactive notebook UI.<br /><br/><b>Data is a first-class citizen</b><br />Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.<br /><br /><b>Collaborate on cloud</b><br />Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment. | <img src="https://github.com/mage-ai/assets/blob/main/overview/mage-preview.gif?raw=True" /> |
<b>Fast deploy</b><br />Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.<br /><br/><b>Scaling made simple</b><br />Transform very large datasets directly in your data warehouse or through a native integration with Spark.<br /><br /><b>Observability</b><br />Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI. | <img src="https://github.com/mage-ai/assets/blob/main/overview/observability.gif?raw=True" /> |
๐ง Intro
Mage is an open-source data pipeline tool for transforming and integrating data.
<br />๐โโ๏ธ Install
The recommended way to install the latest version of Mage is through Docker with the following command:
docker pull mageai/mageai:latest
You can also install Mage using pip or conda, though this may cause dependency issues without the proper environment.
pip install mage-ai
conda install -c conda-forge mage-ai
Looking for help? The fastest way to get started is by checking out our documentation here.
Looking for quick examples? Open a demo project right in your browser or check out our guides.
๐ฎ Demo
Live demo
Build and run a data pipeline with our <b>demo app</b>.
WARNING
The live demo is public to everyone, please donโt save anything sensitive (e.g. passwords, secrets, etc).
Demo video (5 min)
<sub><i>Click the image to play video</i></sub>
<br />๐ฉโ๐ซ Tutorials
- Load data from API, transform it, and export it to PostgreSQL
- Integrate Mage into an existing Airflow project
- Train model on Titanic dataset
- Set up dbt models and orchestrate dbt runs
๐ฎ Features
๐ถ | <b>Orchestration</b> | Schedule and manage data pipelines with observability. |
๐ | <b>Notebook</b> | Interactive Python, SQL, & R editor for coding data pipelines. |
๐๏ธ | <b>Data integrations</b> | Synchronize data from 3rd party sources to your internal destinations. |
๐ฐ | <b>Streaming pipelines</b> | Ingest and transform real-time data. |
โ | <b>dbt</b> | Build, run, and manage your dbt models with Mage. |
<b>A sample data pipeline defined across 3 files โ</b>
- Load data โ
@data_loader def load_csv_from_file(): return pd.read_csv('default_repo/titanic.csv')
- Transform data โ
@transformer def select_columns_from_df(df, *args): return df[['Age', 'Fare', 'Survived']]
- Export data โ
@data_exporter def export_titanic_data_to_disk(df) -> None: df.to_csv('default_repo/titanic_transformed.csv')
<b>What the data pipeline looks like in the UI โ</b>
<img alt="data pipeline overview" src="https://github.com/mage-ai/assets/blob/main/data-pipeline-overview.png?raw=True" />
New? We recommend reading about <b>blocks</b> and learning from a <b>hands-on tutorial</b>.
<br />๐๏ธ Core design principles
Every user experience and technical design decision adheres to these principles.
๐ป | Easy developer experience | Open-source engine that comes with a custom notebook UI for building data pipelines. |
๐ข | Engineering best practices built-in | Build and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts. |
๐ณ | Data is a first-class citizen | Designed from the ground up specifically for running data-intensive workflows. |
๐ช | Scaling is made simple | Analyze and process large data quickly for rapid iteration. |
๐ธ Core abstractions
These are the fundamental concepts that Mage uses to operate.
Project | Like a repository on GitHub; this is where you write all your code. |
Pipeline | Contains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code. |
Block | A file with code that can be executed independently or within a pipeline. |
Data product | Every block produces data after it's been executed. These are called data products in Mage. |
Trigger | A set of instructions that determine when or how a pipeline should run. |
Run | Stores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc. |
๐โโ๏ธ Contributing and developing
Add features and instantly improve the experience for everyone.
Check out the <b>contributing guide</b> to set up your development environment and start building.
<br />๐จโ๐ฉโ๐งโ๐ฆ Community
Individually, weโre a mage.
๐ง Mage
Magic is indistinguishable from advanced technology. A mage is someone who uses magic (aka advanced technology). Together, weโre Magers!
๐งโโ๏ธ๐ง Magers (
/หmฤjษr/
)A group of mages who help each other realize their full potential! Letโs hang out and chat together โ
For real-time news, fun memes, data engineering topics, and more, join us on โ
<img alt="Twitter" height="20" src="https://user-images.githubusercontent.com/78053898/198755056-a15c4439-c07f-41ea-ba35-bc4bfdd09f1a.png" /> | |
<img alt="LinkedIn" height="20" src="https://user-images.githubusercontent.com/78053898/198755052-2777d6ae-c161-4a4b-9ece-4fd7bd458e26.png" /> | |
<img alt="GitHub" height="20" src="https://user-images.githubusercontent.com/78053898/198755053-5c3971b1-9c49-4888-8a8e-1599f0fc6646.png" /> | GitHub |
<img alt="Slack" height="20" src="https://user-images.githubusercontent.com/78053898/198755054-03d47bfc-18b6-45a5-9593-7b496eb927f3.png" /> | Slack |
๐ค Frequently Asked Questions (FAQs)
Check out our FAQ page to find answers to some of our most asked questions.
<br />๐ชช License
See the LICENSE file for licensing information.
<br />