Learn and practice each part of the data engineering process and apply your acquired knowledge and skills to develop an end-to-end data pipeline from the ground up.
Register on Slack • Join the #course-data-engineering Slack channel • Telegram Announcements Channel • Course Playlist • Frequently Asked Questions (FAQ)
- Start Date: 13 January 2025
- Registration Link: Sign up here
- Materials: Cohort-specific materials
The course materials are open for self-paced learning. Simply follow the suggested syllabus week by week:
- Start watching the videos.
- Join the Slack community.
- Refer to the FAQ document for common issues.
This course consists of modules, workshops, and a project that helps you apply the concepts and tools learned during the course. The syllabus is structured to guide you step-by-step through the world of data engineering.
- Module 1: Containerization and Infrastructure as Code
- Module 2: Workflow Orchestration
- Workshop 1: Data Ingestion
- Module 3: Data Warehouse
- Module 4: Analytics Engineering
- Module 5: Batch Processing
- Module 6: Streaming
- Project
To get the most out of this course, you should feel comfortable with coding and the command line and know the basics of SQL. Prior experience with Python will be helpful, but you can pick Python is relatively fast if you have experience with other programming languages.
Prior experience with data engineering is not required.
We encourage Learning in Public
Note: NYC TLC changed the format of the data we use to parquet. In the course we still use the CSV files accessible here.
- Course overview
- Introduction to GCP
- Docker and docker-compose
- Running Postgres locally with Docker
- Setting up infrastructure on GCP with Terraform
- Preparing the environment for the course
- Homework
- Data Lake
- Workflow orchestration
- Workflow orchestration with Kestra
- Homework
- Reading from apis
- Building scalable pipelines
- Normalising data
- Incremental loading
- Homework
- Data Warehouse
- BigQuery
- Partitioning and clustering
- BigQuery best practices
- Internals of BigQuery
- BigQuery Machine Learning
- Basics of analytics engineering
- dbt (data build tool)
- BigQuery and dbt
- Postgres and dbt
- dbt models
- Testing and documenting
- Deployment to the cloud and locally
- Visualizing the data with google data studio and metabase
- Batch processing
- What is Spark
- Spark Dataframes
- Spark SQL
- Internals: GroupBy and joins
- Introduction to Kafka
- Schemas (avro)
- Kafka Streams
- Kafka Connect and KSQL
Putting everything we learned to practice
- Week 1 and 2: working on your project
- Week 3: reviewing your peers
Past instructors:
The best way to get support is to use DataTalks.Club's Slack. Join the #course-data-engineering channel.
To make discussions in Slack more organized:
- Follow these recommendations when asking for help
- Read the DataTalks.Club community guidelines
Thanks to the course sponsors for making it possible to run this course
Do you want to support our course and our community? Please reach out to [email protected]
