Home

Awesome

<h1 align="center"> <a target="_blank" href="https://mage.ai" > <img align="center" alt="Mage" src="https://github.com/mage-ai/assets/blob/main/mascots/mascots-shorter.jpeg?raw=true" style="width:100%;" /> </a> </h1> <p align="center"> ๐Ÿง™ A modern replacement for Airflow. </p> <p align="center"> <a href="https://docs.mage.ai" target="_blank" ><b>Documentation</b></a>&nbsp;&nbsp;&nbsp;๐ŸŒช๏ธ&nbsp;&nbsp;&nbsp; <a href="https://youtu.be/GswOdShLGmg" target="_blank" ><b>Get a 5 min overview</b></a>&nbsp;&nbsp;&nbsp;๐ŸŒŠ&nbsp;&nbsp;&nbsp; <a href="https://demo.mage.ai" target="_blank" ><b>Play with live tool</b></a>&nbsp;&nbsp;&nbsp;๐Ÿ”ฅ&nbsp;&nbsp;&nbsp; <a href="https://www.mage.ai/chat" target="_blank" > <b>Get instant help</b> </a> </p> <div align="center"> <a href="https://pypi.org/project/mage-ai/" target="_blank" > <img alt="PyPi" src="https://img.shields.io/pypi/v/mage-ai?color=orange" /> </a> <a href="https://anaconda.org/conda-forge/mage-ai" target="_blank" > <img src="https://anaconda.org/conda-forge/mage-ai/badges/version.svg" /> </a> <a href="https://opensource.org/licenses/Apache-2.0" target="_blank" > <img alt="License" src="https://img.shields.io/github/license/mage-ai/mage-ai?color=red" /> </a> <a href="https://www.mage.ai/chat" target="_blank" > <img alt="Slack" src="https://img.shields.io/badge/Slack-Join%20Slack-blueviolet?logo=slack" /> </a> <a href="https://github.com/mage-ai/mage-ai" target="_blank" > <img alt="GitHub Stars" src="https://img.shields.io/github/stars/mage-ai/mage-ai?logo=github"> </a> <a href="https://hub.docker.com/r/mageai/mageai" target="_blank" > <img alt="Docker pulls" src="https://img.shields.io/docker/pulls/mageai/mageai.svg"> </a> <a href="https://pepy.tech/project/mage-ai" target="_blank" > <img alt="pip installs" src="https://static.pepy.tech/personalized-badge/mage-ai?period=total&units=international_system&left_color=grey&right_color=blue&left_text=pip%20installs"> </a> </div> <img referrerpolicy="no-referrer-when-downgrade" src="https://static.scarf.sh/a.png?x-pxid=b3c96d79-b8f0-414b-a687-8bfc164b4b7a" /> <div align="center">

Give your data team magical powers

</div> <p align="center"> <b>Integrate</b> and synchronize data from 3rd party sources </p> <p align="center"> Build real-time and batch pipelines to <b>transform</b> data using Python, SQL, and R </p> <p align="center"> Run, monitor, and <b>orchestrate</b> thousands of pipelines without losing sleep </p> <br /> <p align="center">1๏ธโƒฃ ๐Ÿ—๏ธ</p> <h1 align="center">Build</h1> <p align="center"> Have you met anyone who said they loved developing in Airflow? <br /> Thatโ€™s why we designed an easy developer experience that youโ€™ll enjoy. </p>
<b>Easy developer experience</b><br />Start developing locally with a single command or launch a dev environment in your cloud using Terraform.<br /><br/><b>Language of choice</b><br />Write code in Python, SQL, or R in the same data pipeline for ultimate flexibility.<br /><br /><b>Engineering best practices built-in</b><br />Each step in your pipeline is a standalone file containing modular code thatโ€™s reusable and testable with data validations. No more DAGs with spaghetti code.<img src="https://github.com/mage-ai/assets/blob/main/overview/mage-build.gif?raw=true" />
<p align="center"> โ†“ </p> <p align="center">2๏ธโƒฃ ๐Ÿ”ฎ</p> <h1 align="center">Preview</h1> <p align="center"> Stop wasting time waiting around for your DAGs to finish testing. <br /> Get instant feedback from your code each time you run it. </p>
<b>Interactive code</b><br />Immediately see results from your codeโ€™s output with an interactive notebook UI.<br /><br/><b>Data is a first-class citizen</b><br />Each block of code in your pipeline produces data that can be versioned, partitioned, and cataloged for future use.<br /><br /><b>Collaborate on cloud</b><br />Develop collaboratively on cloud resources, version control with Git, and test pipelines without waiting for an available shared staging environment.<img src="https://github.com/mage-ai/assets/blob/main/overview/mage-preview.gif?raw=True" />
<p align="center"> โ†“ </p> <p align="center">3๏ธโƒฃ ๐Ÿš€</p> <h1 align="center">Launch</h1> <p align="center"> Donโ€™t have a large team dedicated to Airflow? <br /> Mage makes it easy for a single developer or small team to scale up and manage thousands of pipelines. </p>
<b>Fast deploy</b><br />Deploy Mage to AWS, GCP, or Azure with only 2 commands using maintained Terraform templates.<br /><br/><b>Scaling made simple</b><br />Transform very large datasets directly in your data warehouse or through a native integration with Spark.<br /><br /><b>Observability</b><br />Operationalize your pipelines with built-in monitoring, alerting, and observability through an intuitive UI.<img src="https://github.com/mage-ai/assets/blob/main/overview/observability.gif?raw=True" />
<br />

๐Ÿง™ Intro

Mage is an open-source data pipeline tool for transforming and integrating data.

  1. Install
  2. Demo
  3. Tutorials
  4. Documentation
  5. Features
  6. Core design principles
  7. Core abstractions
  8. Contributing
<br />

๐Ÿƒโ€โ™€๏ธ Install

The recommended way to install the latest version of Mage is through Docker with the following command:

docker pull mageai/mageai:latest

You can also install Mage using pip or conda, though this may cause dependency issues without the proper environment.

pip install mage-ai
conda install -c conda-forge mage-ai

Looking for help? The fastest way to get started is by checking out our documentation here.

Looking for quick examples? Open a demo project right in your browser or check out our guides.

๐ŸŽฎ Demo

Live demo

Build and run a data pipeline with our <b>demo app</b>.

WARNING

The live demo is public to everyone, please donโ€™t save anything sensitive (e.g. passwords, secrets, etc).

Demo video (5 min)

Mage quick start demo

<sub><i>Click the image to play video</i></sub>

<br />

๐Ÿ‘ฉโ€๐Ÿซ Tutorials

<img alt="Fire mage" height="160" src="https://github.com/mage-ai/assets/blob/main/mage-fire-charging-up.svg?raw=True" /> <br />

๐Ÿ”ฎ Features

๐ŸŽถ<b>Orchestration</b>Schedule and manage data pipelines with observability.
๐Ÿ““<b>Notebook</b>Interactive Python, SQL, & R editor for coding data pipelines.
๐Ÿ—๏ธ<b>Data integrations</b>Synchronize data from 3rd party sources to your internal destinations.
๐Ÿšฐ<b>Streaming pipelines</b>Ingest and transform real-time data.
โŽ<b>dbt</b>Build, run, and manage your dbt models with Mage.

<b>A sample data pipeline defined across 3 files โž</b>

  1. Load data โž
    @data_loader
    def load_csv_from_file():
        return pd.read_csv('default_repo/titanic.csv')
    
  2. Transform data โž
    @transformer
    def select_columns_from_df(df, *args):
        return df[['Age', 'Fare', 'Survived']]
    
  3. Export data โž
    @data_exporter
    def export_titanic_data_to_disk(df) -> None:
        df.to_csv('default_repo/titanic_transformed.csv')
    

<b>What the data pipeline looks like in the UI โž</b>

<img alt="data pipeline overview" src="https://github.com/mage-ai/assets/blob/main/data-pipeline-overview.png?raw=True" />

New? We recommend reading about <b>blocks</b> and learning from a <b>hands-on tutorial</b>.

Ask us questions on Slack

<br />

๐Ÿ”๏ธ Core design principles

Every user experience and technical design decision adheres to these principles.

๐Ÿ’ปEasy developer experienceOpen-source engine that comes with a custom notebook UI for building data pipelines.
๐ŸšขEngineering best practices built-inBuild and deploy data pipelines using modular code. No more writing throwaway code or trying to turn notebooks into scripts.
๐Ÿ’ณData is a first-class citizenDesigned from the ground up specifically for running data-intensive workflows.
๐ŸชScaling is made simpleAnalyze and process large data quickly for rapid iteration.
<br />

๐Ÿ›ธ Core abstractions

These are the fundamental concepts that Mage uses to operate.

ProjectLike a repository on GitHub; this is where you write all your code.
PipelineContains references to all the blocks of code you want to run, charts for visualizing data, and organizes the dependency between each block of code.
BlockA file with code that can be executed independently or within a pipeline.
Data productEvery block produces data after it's been executed. These are called data products in Mage.
TriggerA set of instructions that determine when or how a pipeline should run.
RunStores information about when it was started, its status, when it was completed, any runtime variables used in the execution of the pipeline or block, etc.
<br />

๐Ÿ™‹โ€โ™€๏ธ Contributing and developing

Add features and instantly improve the experience for everyone.

Check out the <b>contributing guide</b> to set up your development environment and start building.

<br />

๐Ÿ‘จโ€๐Ÿ‘ฉโ€๐Ÿ‘งโ€๐Ÿ‘ฆ Community

Individually, weโ€™re a mage.

๐Ÿง™ Mage

Magic is indistinguishable from advanced technology. A mage is someone who uses magic (aka advanced technology). Together, weโ€™re Magers!

๐Ÿง™โ€โ™‚๏ธ๐Ÿง™ Magers (/หˆmฤjษ™r/)

A group of mages who help each other realize their full potential! Letโ€™s hang out and chat together โž

Hang out on Slack

For real-time news, fun memes, data engineering topics, and more, join us on โž

<img alt="Twitter" height="20" src="https://user-images.githubusercontent.com/78053898/198755056-a15c4439-c07f-41ea-ba35-bc4bfdd09f1a.png" />Twitter
<img alt="LinkedIn" height="20" src="https://user-images.githubusercontent.com/78053898/198755052-2777d6ae-c161-4a4b-9ece-4fd7bd458e26.png" />LinkedIn
<img alt="GitHub" height="20" src="https://user-images.githubusercontent.com/78053898/198755053-5c3971b1-9c49-4888-8a8e-1599f0fc6646.png" />GitHub
<img alt="Slack" height="20" src="https://user-images.githubusercontent.com/78053898/198755054-03d47bfc-18b6-45a5-9593-7b496eb927f3.png" />Slack
<br />

๐Ÿค” Frequently Asked Questions (FAQs)

Check out our FAQ page to find answers to some of our most asked questions.

<br />

๐Ÿชช License

See the LICENSE file for licensing information.

<img alt="Water mage casting spell" height="300" src="https://github.com/mage-ai/assets/blob/main/mage-water-charging-up.svg?raw=True" />

<br />