Can your pipeline rerun last week without double-counting a row?
Market (Mandi) Price Pipeline: a nightly Airflow pipeline that never double-counts a row.
Buyers, traders and price dashboards read this table every morning. The pipeline checks each night’s prices before loading them, and a rerun leaves the table exactly as it was.
Core level · built in weeks 21 and 22 of the program.
Taught live by industry experts from Mastercard and Paytm.
alert overnight_batch FAILED · check · ds 2026-08-12 · try 4
11 Aug loaded again464 rows before464 after
The problem: a crop-price table that has to be right every morning.
Market (mandi) prices move every day, and people decide what to buy, sell and stock from the table they open each morning. A pipeline that fails quietly leaves a gap; one that reruns carelessly counts a night twice and skews every average. You build the nightly load that does neither, and prove it.
Who reads the table
A procurement team pricing tomorrow's purchase, a trader comparing markets (mandis), and the price dashboard both of them open first.
The data
Daily market (mandi) crop prices: commodity, market, date and price. By default a fixed test source returns the same prices for any given date, which is what lets you prove the rerun. The Agmarknet daily file swaps in with one function.
What's messy on purpose
Empty fetches, missing values, prices at or below zero and prices out of range. Each one must stop the run before it reaches the warehouse.
Where it goes next
Weeks 23 to 25 make the data move in real time, and the end-to-end project puts batch and streaming into one platform.
- Python and SQL basics; weeks 1 to 20 of the program build everything this project uses
- Docker on your laptop, free
- A Slack webhook, if you want live alerts
- Weeks 21 and 22 of the PG Program in Data Engineering
PG Program in Data Engineering: the program this project comes from.
Build it live on weekends with the faculty, then the projects after it and your own end-to-end data platform, with placement support to the first offer. Data Engineers earn up to ₹80 LPA in India (recrew.ai, 2026).
Thirteen nights in Airflow: nine backfilled, two on schedule, one stopped by the check, one still running.
Fetch, check, load, and one alert when a night fails.
Checked before it loads. Safe to run again.
Each night’s run works from that night’s date, checks the prices before they reach the warehouse, and replaces only that night when it runs twice.
- Data
- Daily market (mandi) crop prices.
- Built with
- Python, Apache Airflow, Docker, DuckDB or Snowflake, and a Slack alert.
- Key challenge
- Rerunning any past night without double-counting a row.
- Done when
- The same night loaded twice leaves the row count unchanged.
What the pipeline must do: five rules from the project brief.
You will use these five rules in every pipeline you build at work.
Same night, same result
Run a night twice and the table stays exactly as it was, proven with row counts.
For engineers idempotent per logical date, by delete-partition-then-insert or MERGE.
Any past week, on demand
Rerun a whole week with one command, because every step works from the date of the night it is running for, never today's date.
For engineers catchup=True, logic driven by the logical date ds, small payloads passed by file path rather than XCom; airflow dags backfill -s … -e ….
Stop before loading bad data
An empty fetch, missing values, or a price at or below zero or out of range stops the run, so the warehouse only ever sees checked rows.
For engineers a validation task that raises before load.
Retry, then give up cleanly
Three retries, each after a longer wait, and one night at a time.
For engineers capped exponential backoff, max_active_runs=1.
One alert you can act on
It names the step, the night, the try number and the log link.
For engineers a failure callback to a Slack webhook.
Six tools, each here for one job, and ₹0 to run on a laptop.
Airflow schedules and retries the run. Docker starts it with one command. DuckDB is the free warehouse on a laptop, and Snowflake the warehouse in the cohort. Slack carries the alert; Python holds the logic.
Where it runs
On your laptop, for ₹0
Airflow 2.9.3 · Docker · DuckDB · the fixed test source
The default source returns the same prices for any given date, every time, which is what lets you prove the rerun. Airflow, Docker and DuckDB are free.
In the cohort, on Snowflake
Airflow · Docker · Snowflake
The cohort version loads Snowflake in week 22, on the cloud account you set up in week 11.
On live market data
Agmarknet daily file · the same pipeline
One function, src/tasks.py::fetch_prices, supplies the prices. Replace it with a download of the day's Agmarknet file and keep the same output shape; every other line stays as it is. A live source can fail on any night, and handling that is exactly what the retry design is for.
Run it on your laptop in four steps.
- 1Copy the settings
cp .env.example .env· add a Slack webhook if you want live alerts - 2Start Airflow
docker compose up - 3Open the Airflow screen
http://localhost:8080· the password is in the log - 4Backfill a week
docker compose exec airflow airflow dags backfill mandi_overnight -s 2026-08-01 -e 2026-08-07
No Docker? The logic runs on its own too: python3 -c against src/tasks.py, exactly as the run log shows.
The files
- BRIEF.md, with the review checklist
- dags/mandi_overnight.py: wiring only (the schedule, retries, alert and step order)
- src/tasks.py: fetch, check, load and alert, each testable on its own and safe to rerun
- docker-compose.yml: one command starts Airflow on your laptop
- output/verification_run.log: the committed proof run
What you hand in
- The pipeline file and the tasks file
- The run log: the pipeline loading, three nights, the rerun proof, the same-data check and the stopped bad data
- The compose file
- The README
How this project reads on your resume.
Name it, list the tools, prove it with one number, and link the code. This is the entry a hiring manager expects.
Market (Mandi) Price Pipeline1Python, Apache Airflow, Docker, Snowflake2
github.com/your-name/mandi-price-pipeline4
- Built a nightly Apache Airflow pipeline that loads daily market (mandi) crop prices, in the Agmarknet format, into Snowflake; containerised with Docker.
- Made every load idempotent: backfills and reruns add zero duplicate rows.3
- Added validation and retries that stop empty, missing and out-of-range prices before load, with a Slack alert on any failed night.
What a hiring manager checks
- 1A title that says what it doesThe name tells them the problem before they read a line.
- 2The tools they search forWritten the way job posts write them, so a recruiter’s search finds you.
- 3One number that proves itA count, a score or a time, measured on your own run.
- 4A link they can openYour repository, with a README and a diagram of how the parts fit.
Shipped my first production-style pipeline: market crop prices into Snowflake every night, checked before they load, and safe to rerun. The code and the run log are on GitHub.
inFeatured on her profile
The brief, the tools and the run log for the Market (Mandi) Price Pipeline, in one file.
- BRIEF.md: the five things, with the engineering detail
- The six tools, each with its job
- The committed run log, line by line
- The other Data Engineering project briefs
Download it now. A program advisor calls within one working day to walk through it and answer anything it leaves open.
Your brief pack is ready.
A program advisor calls within one working day to walk through it.
Built live with industry experts who work with data every day.
The faculty teach this project live, review your build, and answer in the cohort chat between sessions, by name.
Meet the faculty before you apply: try a free live build, ninety minutes, live and free.
Where it sits in the Data Engineering program.
The project before it makes a slow job fast; the one after it makes the data move live. Everything leads into the end-to-end data platform.
Festival Sales Spark Pipeline
Festival-season sales too big for pandas, processed in Spark as partitioned Delta tables.
Proves you can make a slow job fast, and explain the speed-up.
→Real-Time Payment Monitor
A live stream of payments with a running total and an anomaly flag.
Proves you can keep a stream correct when a message arrives twice.
→Data Quality Test Suite
Tables broken on purpose, and a test suite that catches every fault.
Proves you can stop a bad load before anyone reads it.
→Your own data platform
Batch and real time in one platform on your cloud, with the AI layer on top and a monthly running-cost statement, presented to a hiring panel.
Proves you can build, run, cost and explain a data platform.
→Start your application.
The PG Program in Data Engineering, where you build this project. Three short steps. Step one takes a minute; finish the rest now or later.
Details saved.
Two short steps finish the application, now or later. A program advisor calls within one working day either way.
Your application is in.
A program advisor calls you within one working day with the timetable, the faculty list and the cohort calendar.
Aiming at something next to data engineering? Take the skill check or see the other programs, and we'll map the route with you.
The Market (Mandi) Price Pipeline project: your questions, answered.
How long does the Market (Mandi) Price Pipeline data engineering project take?
Weeks 21 and 22 of the PG Program in Data Engineering, built in the live sessions with the faculty and finished by the end of week 22.
What do I need to know before building this Airflow project?
Python and SQL basics, and Docker installed. In the program, weeks 1 to 20 build everything it uses, from SQL at production depth to Spark on the cloud.
Which data does this data engineering project use?
Daily market (mandi) crop prices. By default a fixed test source returns the same prices for any given date, so a rerun can be proven; the Agmarknet daily file swaps in by replacing one function.
Does this Airflow data pipeline project run on my laptop?
Yes: docker compose up starts Airflow, and DuckDB is the free warehouse. The cohort version loads Snowflake in week 22.
Can I build this data engineering project without joining the program?
You can start from the brief in the free pack. The live weeks with the faculty, the weekly reviews, the Saturday doubt clinic and the panel come with the PG Program in Data Engineering.
What does idempotent mean in a data pipeline?
Running the same load twice leaves the table exactly as one run would. Here it is done by deleting the night's partition and inserting again, or with MERGE, and proven with row counts before and after a rerun.
How do you backfill an Airflow DAG without double-counting rows?
Make every step work from the run's logical date (ds) instead of today's date, keep catch-up on, and make each load idempotent. Then airflow dags backfill reruns any range, and the row counts stay the same.
Why pass file paths between Airflow tasks instead of data through XCom?
XCom is built for small values. Passing a file path keeps Airflow's metadata database light and each task testable on its own, and the Airflow file stays wiring only.
How should retries be set for a nightly data pipeline?
A few retries with exponential backoff and a cap, so a flaky source gets time to recover, and one run at a time (max_active_runs=1), so a slow night never overlaps the next.
What should a data pipeline failure alert contain?
The step, the night, the try number and a link to the log: enough to act on at once, without searching for anything else.
Which data engineering job does this Airflow project prepare me for?
ETL / ELT Developer is the title this project opens, and the project is a core line in a Data Engineer's portfolio.
How does this project help with data engineering placement?
It gives you one resume entry with three bullets. The portfolio review checks it against what an interviewer opens first, and the mock interviews ask about it. Placement support runs from week 20 to the first offer, with introductions into the 39+ companies our learners work at.
What does the PG Program in Data Engineering cost?
The program fee is ₹1,14,000 including GST, from ₹3,516 a month on EMI, and phase discounts apply while a phase is open. It covers all 34 weeks, every project, cloud credits, mock interviews and placement support.
Build this one live, and the data platform it leads to.
Thirty-four weeks from the first class, this pipeline is one line in a portfolio that ends in your own data platform, presented to a panel.
Not ready to apply yet? Request the project brief and tools or save a seat at the free live build.
Data and AI openings on WhatsApp, and the hiring market twice a month.
The roles the companies our learners work at are hiring for, and the dates of the next free live build. Read three issues of the Brief and you’ll know whether we’re your kind of school.
You’re on the list.
The first alert lands the next time our network is hiring, and the next Brief lands in your email.