Awesome
Data Engineering Zoomcamp
<p align="center"> <a href="https://airtable.com/shr6oVXeQvSI5HuWD"><img src="https://user-images.githubusercontent.com/875246/185755203-17945fd1-6b64-46f2-8377-1011dcb1a444.png" height="50" /></a> </p>- Register in DataTalks.Club's Slack
- Join the
#course-data-engineeringchannel - Join the course Telegram channel with announcements
- The videos are published on DataTalks.Club's YouTube channel in the course playlist
- Frequently asked technical questions
Syllabus
- Module 1: Containerization and Infrastructure as Code
- Module 2: Workflow Orchestration
- Workshop 1: Data Ingestion
- Module 3: Data Warehouse
- Module 4: Analytics Engineering
- Module 5: Batch processing
- Module 6: Streaming
- Workshop 2: Stream Processing with SQL
- Project
Taking the course
2025 Cohort
- Start: 13 January 2025
- Registration link: https://airtable.com/shr6oVXeQvSI5HuWD
Self-paced mode
All the materials of the course are freely available, so that you can take the course at your own pace
- Follow the suggested syllabus (see below) week by week
- You don't need to fill in the registration form. Just start watching the videos and join Slack
- Check FAQ if you have problems
- If you can't find a solution to your problem in FAQ, ask for help in Slack
Syllabus
We encourage Learning in Public
Note: NYC TLC changed the format of the data we use to parquet. In the course we still use the CSV files accessible here.
Module 1: Containerization and Infrastructure as Code
- Course overview
- Introduction to GCP
- Docker and docker-compose
- Running Postgres locally with Docker
- Setting up infrastructure on GCP with Terraform
- Preparing the environment for the course
- Homework
Module 2: Workflow Orchestration
- Data Lake
- Workflow orchestration
- Workflow orchestration with Mage
- Homework
Workshop 1: Data Ingestion
- Reading from apis
- Building scalable pipelines
- Normalising data
- Incremental loading
- Homework
Module 3: Data Warehouse
- Data Warehouse
- BigQuery
- Partitioning and clustering
- BigQuery best practices
- Internals of BigQuery
- BigQuery Machine Learning
Module 4: Analytics engineering
- Basics of analytics engineering
- dbt (data build tool)
- BigQuery and dbt
- Postgres and dbt
- dbt models
- Testing and documenting
- Deployment to the cloud and locally
- Visualizing the data with google data studio and metabase
Module 5: Batch processing
- Batch processing
- What is Spark
- Spark Dataframes
- Spark SQL
- Internals: GroupBy and joins
Module 6: Streaming
- Introduction to Kafka
- Schemas (avro)
- Kafka Streams
- Kafka Connect and KSQL
Workshop 2: Stream Processing with SQL
Project
Putting everything we learned to practice
- Week 1 and 2: working on your project
- Week 3: reviewing your peers
Overview
<img src="images/architecture/arch_v3_workshops.jpg" />Prerequisites
To get the most out of this course, you should feel comfortable with coding and command line and know the basics of SQL. Prior experience with Python will be helpful, but you can pick Python relatively fast if you have experience with other programming languages.
Prior experience with data engineering is not required.
Instructors
Past instructors:
Asking for help in Slack
The best way to get support is to use DataTalks.Club's Slack. Join the #course-data-engineering channel.
To make discussions in Slack more organized:
- Follow these recommendations when asking for help
- Read the DataTalks.Club community guidelines
Supporters and partners
Thanks to the course sponsors for making it possible to run this course
<p align="center"> <a href="https://mage.ai/"> <img height="120" src="images/mage.svg"> </a> </p> <p align="center"> <a href="https://dlthub.com/"> <img height="90" src="images/dlthub.png"> </a> </p> <p align="center"> <a href="https://risingwave.com/"> <img height="90" src="images/rising-wave.png"> </a> </p>Do you want to support our course and our community? Please reach out to alexey@datatalks.club