š± Beginner Data Engineer Roadmap
Start from zero and become job-ready. A step-by-step learning path covering SQL, Python, ETL basics, cloud fundamentals, and your first data pipeline projects.
ā Expert-Designed Learning Pathā¢ Industry-Validated Curriculumā¢ Real-World Application Focus
This roadmap was created by data engineering professionals with 51 hands-on tasks covering production-ready skills used by companies like Netflix, Airbnb, and Spotify. Master Python, SQL, PostgreSQL and 5 more technologies.
Beginner
11 sections ā¢ 51 tasks
Skills You'll Learn
- SQL
- Python
- ETL fundamentals
- Cloud basics
- Data modeling
- Version control
Tools You'll Use
- Python
- SQL
- PostgreSQL
- Docker
- Git
- DuckDB
- Airflow
- dbt
Projects to Build
- Local Data Engineering Environment with dlt, DuckDB & Jupyter
Set up a local development environment for data processing and analytics using Jupyter notebooks, dlt, and DuckDB. All tools are open-source and run locally.
- Scheduled GitHub ETL with Polars, DLT & DuckDB
Build a scheduled ETL pipeline that extracts GitHub repository data, transforms it with Polars, and stores results in DuckDB
- End-to-End Analytics Platform with DuckDB + Metabase
Build a modern, low-cost analytics stack using DuckDB, Metabase, and GitHub Actions for automated data updates and business-ready dashboards.
Learning Resources
Step 0: Prerequisites
-Understand basic computer science concepts: how the internet works, client-server model, and file systems
-Get comfortable with the command line: navigate directories, create files, and run scripts
-Learn how to use a code editor (VS Code recommended) and install useful extensions
-Understand what data engineering is and how it fits in the data ecosystem alongside analytics and data science
Step 1: SQL Fundamentals
-Learn SELECT, WHERE, ORDER BY, and LIMIT to query data from tables
-Master JOIN types: INNER, LEFT, RIGHT, and FULL OUTER joins across multiple tables
-Use GROUP BY and aggregate functions (COUNT, SUM, AVG, MIN, MAX) for data summarization
-Write subqueries and Common Table Expressions (CTEs) for complex queries
-Learn window functions (ROW_NUMBER, RANK, LAG, LEAD, running totals) for advanced analytics
Step 2: Python for Data
-Install Python, set up a virtual environment, and learn core syntax (variables, loops, functions)
-Work with Python data structures: lists, dictionaries, sets, and comprehensions
-Read and write files: CSV, JSON, and Parquet using pandas or polars
-Make HTTP requests to REST APIs and parse JSON responses
-Handle errors gracefully with try/except and implement basic logging
Step 3: Version Control and CLI
-Install Git and learn the basics: init, add, commit, push, pull, and branching
-Create a GitHub account and push your first repository
-Practice essential bash commands: pipes, redirects, grep, awk, and cron
-Set up SSH keys for secure access to remote servers and GitHub
Step 4: Databases and Data Modeling
-Install PostgreSQL locally and practice creating databases, tables, and inserting data
-Understand relational database design: primary keys, foreign keys, and constraints
-Learn normalization (1NF, 2NF, 3NF) and when to denormalize for performance
-Draw Entity-Relationship (ER) diagrams to model a real-world business domain
-Explore [Data Modeling fundamentals](/fundamentals/data-modeling) for deeper schema design skills
Step 5: Docker and Development Environment
-Install Docker and understand containers vs virtual machines
-Write your first Dockerfile to containerize a Python script
-Use Docker Compose to spin up PostgreSQL and pgAdmin as a local data stack
-Mount local volumes for code hot-reloading and data persistence
Step 6: Your First ETL Pipeline
-Extract data from a public REST API (e.g., weather, financial, or open government data)
-Transform the raw data using Python: clean, filter, enrich, and reshape
-Load the transformed data into a PostgreSQL database
-Add logging, error handling, and idempotency to make the pipeline production-ready
-Schedule the pipeline to run daily using cron or a simple Python scheduler
Step 7: Cloud Fundamentals
-Create a free-tier account on AWS, GCP, or Azure and explore the console
-Learn object storage (S3 / GCS): upload files, set permissions, and organize with prefixes
-Understand IAM basics: users, roles, policies, and the principle of least privilege
-Provision a managed database (RDS / Cloud SQL) and connect from your local machine
Step 8: Orchestration Basics
-Understand what orchestration is and why it matters for data pipelines
-Install Apache Airflow locally using Docker Compose
-Write your first DAG with tasks, dependencies, and a schedule
-Use Airflow operators to run Python functions, execute SQL, and transfer data
-Monitor DAG runs, handle failures, and set up retries and alerts
Step 9: Analytics Engineering
-Understand what analytics engineering is and where dbt fits in the modern data stack
-Install dbt Core and initialize a project connected to PostgreSQL or DuckDB
-Create staging and mart models using SQL and Jinja templating
-Add data tests (not_null, unique, accepted_values, relationships) to validate transformations
-Generate and serve dbt documentation to share your data lineage with stakeholders
Step 10: Portfolio and Job Search
-Build a portfolio with 2-3 end-to-end data pipeline projects on GitHub
-Write clear README files with architecture diagrams for each project
-Tailor your resume to highlight data engineering skills, tools, and measurable outcomes
-Practice common data engineering interview topics: SQL, system design, and pipeline architecture
-Explore the [Interview Prep](/interview-prep) section for real questions from top companies