Data Engineering Roadmap for Beginners: A 9-Month Plan from Zero to Job-Ready (2026)

A data engineering roadmap for a beginner in India takes about nine months at 10 to 12 hours a week. You learn SQL first, then Python, Linux and Git, data modelling, one cloud, Spark and Databricks, a cloud warehouse, orchestration, streaming, dbt and data quality, governance, and finally the data layer that AI products run on. Each stage names the tools to learn and one project to finish before you move on, so you reach the job market with a GitHub profile that answers the questions hiring managers ask.

What a data engineer does

A data engineer builds and runs the pipelines that move data from where it is created (an app database, a payment gateway, a toll plaza sensor, a government API) into a lake or warehouse, cleans it, shapes it into tables people can query, and keeps it fresh, tested and governed. Analysts, dashboards, machine learning models and AI assistants all sit on top of that work.

The role is large and growing in India. AIM Research estimated that India hosts around 140,000 data engineers, with 6,288 open positions in Bengaluru alone at the time of its June 2025 report, and BFSI employing 57% of them. Banks, fintechs and insurers, together with global capability centres, are where most of these roles sit.

The roadmap at a glance

Stage	Weeks	What you learn	Main tools	Project you finish
1	1–4	SQL for data	PostgreSQL, DBeaver	Timed SQL set on a business schema
2	5–8	Python for data	Python, pandas, pytest, requests	A tested data-cleaning script
3	9–10	Linux, Git, an AI coding assistant	bash, cron, Git, GitHub	Kirana Ledger, your first scheduled pipeline
4	11–13	Data modelling and storage	Parquet, dbdiagram, MongoDB	A dimensional model with slowly changing dimensions
5	14–17	One cloud, end to end	AWS, Azure or GCP	Highway Toll Data Pipeline
6	18–21	Spark and Databricks	PySpark, Delta Lake, Unity Catalog	A Databricks job reading your lake
7	22–23	Cloud warehouse and cost	Snowflake, Snowpipe	A monthly cost model
8	24–26	Orchestration and containers	Airflow, Docker	Mandi Price Pipeline
9	27–28	Streaming	Kafka, Spark Structured Streaming	Real-Time Payment Monitor
10	29–30	dbt and data quality	dbt Core, dbt-expectations	Data Quality Test Suite
11	31–32	Governance and DataOps	GitHub Actions, Terraform, a data catalog	Runbook and a governed dataset
12	33–34	The data layer under AI	pgvector, LangChain, MLflow	Data Pipeline for an AI Assistant
13	35–36	Job-ready: design, rounds, portfolio	Excalidraw, GitHub Pages	An end-to-end data platform and a portfolio page

Before you start: time, setup and the order

Plan for 10 to 12 hours a week: two hours on four weekdays and a longer block at the weekend for the project. A laptop with 8 GB of RAM is enough, because the heavy work runs on free cloud tiers and trial credits. You need no computer science degree and no prior Python. You do need to finish each stage's project before you start the next one. The order matters because every later tool assumes the earlier ones: Spark DataFrames are SQL in disguise, Airflow DAGs are Python files, and every cloud console assumes you are comfortable in a terminal.

Stage 1 (weeks 1–4): SQL

SQL is the one skill tested in every data engineering hiring round, so it comes first and gets the most time. Learn SELECT, filters and joins in week 1; aggregates and GROUP BY in week 2; subqueries and CTEs in week 3; and window functions, indexes and reading a query plan in week 4. Window functions deserve extra hours: as the PostgreSQL documentation puts it, they calculate across related rows while each row keeps its own identity, which is exactly what running totals, rankings and deduplication need.

• Tools: PostgreSQL (or MySQL), DBeaver as your query client.
• Project: load a small business schema (orders, customers, products, payments) and solve 25 timed questions on it, from "top three products per city" to "customers whose spend fell month on month". Save the queries in a repository with a one-line note on each.

Stage 2 (weeks 5–8): Python for data

You need working Python, not computer science Python. Cover types, functions, files and error handling; calling APIs and parsing JSON with requests; testing with pytest; then pandas and NumPy for cleaning and reshaping tables. Write a test for every cleaning function you build. Hiring managers notice tests faster than they notice clever code.

• Tools: Python, pandas, NumPy, pytest, requests, Jupyter, VS Code.
• Project: a tested data-cleaning script that pulls a public dataset (an open government dataset works well), fixes types, removes duplicates, standardises names and dates, and writes a clean file, with at least ten pytest tests.

Stage 3 (weeks 9–10): Linux, Git and an AI coding assistant

Teams expect you to move around a terminal, schedule a job and send a clean pull request. Learn bash basics, file permissions and cron; Git branches, pull requests and GitHub; and how to use an AI coding assistant such as Copilot, Cursor or Claude Code inside a repository while you review every line it writes.

• Tools: bash, cron, Git, GitHub, one AI coding assistant.
• Project: Kirana Ledger. Take a neighbourhood store's daily transactions in a database, clean them into an analysis-ready table, and run the whole thing on a cron schedule. This is your first pipeline: source to clean table, on a timer, with a clean commit history.

Stage 4 (weeks 11–13): Data modelling and storage

Before you build bigger pipelines, learn how data should be stored and shaped. Cover OLTP versus OLAP systems; lake, warehouse and lakehouse; the medallion layers (bronze for raw data, silver for cleaned and validated data, gold for business-ready aggregates, in Databricks' own description); file formats such as Parquet, Avro and ORC; partitioning; dimensional modelling with facts and dimensions; and slowly changing dimensions. Kimball's Type 2 adds a new row for every change, with an effective date, an expiry date and a current-row flag. Finish with idempotent loads and backfills, and a short look at MongoDB so you know where document stores fit.

• Tools: PostgreSQL, Parquet, dbdiagram, MongoDB.
• Project: a dimensional model for the Kirana Ledger data, with one Type 2 dimension and a load you can re-run without creating duplicates.

Stage 5 (weeks 14–17): One cloud, end to end

Pick one cloud and go deep. Learning three at once gives you three shallow answers in an interview. A practical way to choose: open 20 data engineer job listings in the city you want to work in and count which cloud they name. Then learn identity and access (IAM), object storage as a data lake, serverless ingest, managed ETL with a data catalog, querying the lake, the cloud warehouse, secrets management, budgets and cost alerts, and monitoring.

Layer	AWS	Azure	GCP
Lake storage	S3	ADLS	Cloud Storage
Serverless ingest	Lambda	Functions	Cloud Functions
Managed ETL	Glue	Fabric Data Factory	Dataflow
Query and warehouse	Athena, Redshift	Fabric Lakehouse and Warehouse	BigQuery

• Project: Highway Toll Data Pipeline. Ingest toll plaza transaction files into object storage, catalog them, transform them into partitioned Parquet, and query daily traffic and revenue per plaza from the warehouse, with a budget alert switched on.

Stage 6 (weeks 18–21): Spark and Databricks

Spark is how you process data too large for one machine. Learn the architecture (driver, executors, partitions), DataFrames and Spark SQL, and why joins cause shuffles. The Spark documentation calls the shuffle "a complex and costly operation" because it copies data across executors and machines. Then learn to handle skew, read the Spark UI, and use Delta Lake or Iceberg tables. On Databricks, cover Unity Catalog, Auto Loader, Lakeflow pipelines, jobs and Asset Bundles. A short look at Hadoop and Hive helps if you join a team with older systems.

• Tools: PySpark, Databricks, Delta Lake, Iceberg, Unity Catalog.
• Project: a scheduled Databricks job, governed by Unity Catalog, that reads the toll data from your lake and writes bronze, silver and gold Delta tables.

Stage 7 (weeks 22–23): Cloud warehouse and cost

Warehouses are where most business questions get answered, and where cloud bills grow quietly. Learn loading and Snowpipe, clustering and query performance, time travel, roles and access, semi-structured data, and how credits turn into money.

• Tools: Snowflake, Snowsight, Snowpipe.
• Project: a monthly cost model: load a dataset, run a realistic daily workload, and show in a short report what it costs per month and how you cut it.

Stage 8 (weeks 24–26): Orchestration and containers

Pipelines have to run on a schedule, retry when they fail and tell someone when they break. Learn Airflow DAGs, sensors and retries; Docker images and Compose; alerting to Slack or email; and managed Airflow on your cloud. Apply the Airflow rule that tasks should behave like database transactions: a re-run must give the same result, so you use UPSERT rather than INSERT and read a fixed data interval, never "the latest data".

• Tools: Airflow, Docker, managed Airflow (MWAA, Fabric or Composer), Slack alerts.
• Project: Mandi Price Pipeline. Pull daily agricultural market (mandi) prices, clean and store them, and publish a price table every day, with retries and a failure alert. You can see how a finished version is laid out on the Mandi Price Pipeline project page.

Stage 9 (weeks 27–28): Streaming

Some data cannot wait for a nightly batch: payments, fraud signals, delivery tracking. Learn Kafka topics, partitions and consumer groups; Schema Registry; delivery semantics (Kafka's default is at least once, so your consumer must handle duplicates); event time and windows; and Spark Structured Streaming or your cloud's managed stream (Kinesis, Event Hubs or Pub/Sub).

• Project: Real-Time Payment Monitor. Stream simulated UPI-style payment events through Kafka, compute per-minute failure rates by bank, and flag spikes as they happen.

Stage 10 (weeks 29–30): dbt and data quality

This stage turns raw tables into tested, documented models the business can trust. Learn dbt models and materialisations, incremental models and snapshots, tests and freshness checks, lineage and documentation. dbt ships with four generic tests (unique, not_null, accepted_values and relationships), and dbt-expectations adds many more. Put a Power BI or Looker Studio report on top so you can show the result to a non-technical person.

• Project: Data Quality Test Suite. A dbt project over one of your earlier datasets with tests on every model, a freshness check and generated documentation.

Stage 11 (weeks 31–32): Governance and DataOps

Once pipelines are live, keeping them correct is the job. Learn monitoring, runbooks and root-cause write-ups; data contracts; catalogs and lineage; CI/CD for data with GitHub Actions; and an introduction to Terraform. India adds a specific reason to learn PII masking: the Digital Personal Data Protection Rules were notified on 13 November 2025, and the obligations on consent, security safeguards, breach reporting and retention apply from 13 May 2027. Teams handling customer data will expect you to know how to mask and restrict it.

• Project: a runbook, a root-cause write-up for a failure you cause on purpose, and a governed dataset with masked personal fields and role-based access.

Stage 12 (weeks 33–34): The data layer under AI

AI products run on pipelines, and job titles in India now combine the two. Learn how to call an LLM inside a pipeline for extraction and classification with structured output, the machine learning basics a data engineer needs, training-data pipelines and feature stores, embeddings and vector stores, RAG ingestion, and handing models over with MLflow.

• Tools: pgvector, ChromaDB, LangChain, MLflow, Bedrock, Azure OpenAI or Vertex AI.
• Project: Data Pipeline for an AI Assistant. Ingest a set of documents, chunk and embed them into a vector store on a schedule, and keep the index fresh when documents change.

Stage 13 (weeks 35–36): Job-ready

Spend the last stretch on what the hiring rounds test. Practise data-platform system design (including the Lambda versus Kappa architectures), timed SQL and Python rounds, and the "walk me through your project" conversation. Then build one end-to-end data platform that combines batch and streaming, is costed and governed, and publish a portfolio page that links every project.

What hiring managers in India check

• SQL under time pressure. Window functions, deduplication, top-N per group and month-on-month change, written live.
• A GitHub profile that reads like work. Clean commit history, a README with an architecture diagram, how to run it, and what it costs.
• Re-runs and failures. Can you explain what happens when your pipeline runs twice, or fails halfway? Idempotency and backfills come up in almost every technical round.
• Tests. pytest on your Python, dbt tests on your models.
• One cloud, explained clearly. Why S3 and not a database, why Parquet and not CSV, what the monthly bill is.

Certifications: when they help

A certification works best after you have finished stage 5, as proof alongside a project, not as a substitute for one. Note what each exam expects:

Certification	What the provider says
AWS Certified Data Engineer – Associate (DEA-C01)	50 questions that count towards the result; ingestion and transformation 34%, data store management 26%, operations 22%, security and governance 18%; aimed at 2–3 years of data engineering experience
Microsoft Fabric Data Engineer Associate (DP-700)	Three skill areas of 30–35% each: implement and manage an analytics solution, ingest and transform data, monitor and optimise
Google Cloud Professional Data Engineer	40–50 questions in 2 hours; aimed at 3+ years of industry experience, including 1+ year on Google Cloud

For a fresher, the AWS or Microsoft associate exam is the realistic target once your stage 5 project is done.

Data engineer salary in India: what the data says

Pay rises quickly with experience. Experienced data engineers earn up to ₹40 LPA in base pay at PayScale's 90th percentile. Each figure is one source's own number, as published; the full breakdown by experience, cloud skill and role is in our data engineer salary guide.

Measure	Figure	Source
Entry level (under 1 year), average total pay	₹518,398	PayScale, updated 14 Jul 2026
Early career (1–4 years), average total pay	₹810,624	PayScale, updated 14 Jul 2026
Median base pay, all levels	₹975k	PayScale, 1,284 profiles
Average base pay in job postings	₹9,68,560	Indeed India, updated 20 Sep 2026
Average salary	₹11,00,000	Glassdoor India, Sep 2026

How SkilD teaches this roadmap

The PG Program in Data Engineering follows this order over 34 weeks: 206 live hours and 17 projects, including Kirana Ledger, the Highway Toll Data Pipeline, the Mandi Price Pipeline and the Real-Time Payment Monitor. You choose AWS, Azure or GCP as your track. Classes run live on Saturdays and Sundays from 10 AM to 1 PM IST with industry experts, including faculty from Mastercard and Paytm, every class is recorded and up within 24 hours, and placement support runs until you have an offer. The first class is on Saturday 28 November 2026. SkilD has trained 4,200+ learners, now working at 39+ companies, and the highest recorded salary hike among them is 82%.

If you want a shorter start (SQL, Python, Linux and Git, ending with Kirana Ledger), Data Engineering Fundamentals is an eight-week course with 48 live hours. The quickest way to judge the teaching is to take Week 1 of the program free.

Frequently asked questions