Phase 12, in depth

By Week 21 you already have a working extract, validate, transform, load, and dbt chain. Airflow is the phase where that chain stops depending on you being awake. Here's the full 7-stage breakdown of how this phase actually teaches that.

Where this fits

GAP Commerce's day-to-day pipeline runs on Fabric Data Factory, the platform's own orchestrator, taught first in Week 21. This deep dive covers Airflow specifically because it's still the orchestrator most job postings name, and everything here (DAGs, dependencies, retries) maps directly onto the Fabric pipeline you already built.

Where Airflow sits in a data platform

Every piece of data GAP Commerce generates still moves through the same system. Airflow doesn't move any of it. It sits above the pipeline, deciding when each stage runs, what waits for what, and what happens the moment one stage fails.

  1. sourcesfiles · APIs · DBs
  2. ingestionpython
  3. processingpandas
  4. orchestrationairflowyou are here
  5. storageADLS Gen2 · OneLake
  6. warehousesnowflake
  7. consumptionBI · ML

Why Airflow, specifically

A hand-written script has no concept of "run this every morning" or "don't start step three until steps one and two both succeed." Someone has to answer that, and it can't keep being a person checking a terminal at 2 AM.

That's a deliberately narrow claim. Airflow governs order, retries, scheduling, and timeouts. It doesn't know or care what a task does, which is exactly what makes it reusable across every pipeline you'll ever build.

  • Not the transformation engine. Airflow triggers dbt_run. It doesn't write the SQL inside it.

  • Not the storage layer. Azure and OneLake hold the data. Airflow just decides when a task touches it.

  • Not built for continuous streams. A DAG runs on a schedule. Data that never stops arriving is Kafka's job.

  • Not a query engine. Airflow can trigger a query. It can't answer one. That's the warehouse.

The 7 stages, stage by stage

Week 21 of the published programme, opened up. Every stage follows the same shape: why a data engineer needs it, and what you actually build.

What Stage 06 actually feels like

Orchestration is mostly about recovering from failure correctly, not watching a DAG turn green. Here's a real decision your on-call self has to make. Try one, then see the reasoning.

Stage 06 · Debugging a failed DAG

Your GAP Commerce DAG has five tasks: extract, validate, transform, load, and dbt_run. extract and validate succeeded. transform just failed.

What should your pipeline do?

What Airflow doesn't solve

None of these are weaknesses in Airflow. They're the reasons the rest of the data engineering stack exists. Every card below follows the same shape, on purpose: what it is, why it exists, when you'd reach for it, and why it's next.

Streaming (Eventstream + Kafka)

Coming next

What happens when data doesn't arrive as a file once a day, but continuously, all day, forever?

What is it?
Microsoft Fabric's Eventstream and Eventhouse route continuous events into the same platform as everything else; Apache Kafka is the industry-standard alternative, a distributed, durable log that producers append events to and consumers read from independently.
Why do I need it?
A file represents data that already happened. Both Eventstream and Kafka represent data that is continuously happening, so the platform's own data lands there first, with Kafka taught hands-on because it's still the most job-relevant standalone streaming tool.
When will I use it?
When 'once a day' stops being good enough: clickstreams, live orders, sensor data.
Why am I learning it next?
Generators (Stage 12) are the closest thing Python has to this idea. Eventstream and Kafka both take it further, in two different tools.

Cloud (Azure + Fabric)

Coming next

What happens when your pipeline has to leave your laptop and run in production?

What is it?
Managed infrastructure on Microsoft Azure, plus Microsoft Fabric, the unified platform that ties Data Factory, OneLake, Lakehouse/Warehouse, and Power BI together.
Why do I need it?
A local file becomes Azure Blob Storage / ADLS Gen2. A local database becomes a managed service. A local script becomes a scheduled, monitored job. Fabric is where those pieces meet as one first-party product.
When will I use it?
The day your pipeline needs to run whether or not your laptop is open.
Why am I learning it next?
Every concept transfers. This is about where the same pipeline actually lives once it's real, and it's the platform GAP Commerce actually runs on.

Warehouse (Fabric + Snowflake)

Coming next

Cloud storage holds files. How does an analyst get an answer in 3 seconds, not by scanning files by hand?

What is it?
A columnar, compute/storage-separated system built to answer analytical questions across huge datasets fast. Fabric Warehouse queries the OneLake data that's already there; Snowflake is the industry-standard alternative, taught hands-on as a real comparison.
Why do I need it?
An OLTP database like PostgreSQL is built for many small, fast transactions. A warehouse is built for the opposite: fewer, much bigger analytical queries, and the same lesson holds in either tool.
When will I use it?
The moment 'query the data lake directly' stops being fast enough for the people who actually need answers.
Why am I learning it next?
You've already modeled fact and dimension tables (Week 14). Fabric Warehouse is where that model gets queried at scale first, with Snowflake taught right alongside it.

dbt

Coming next

The same revenue calculation has been copy-pasted into four scripts, and they've quietly drifted apart.

What is it?
A SQL-based transformation framework that turns warehouse SQL into versioned, tested, documented models instead of scattered scripts.
Why do I need it?
SQL transformation logic that isn't centralized gets duplicated, and duplicated logic always eventually disagrees with itself.
When will I use it?
The moment more than one person, or more than one script, needs the same transformed table.
Why am I learning it next?
You already write the SQL. dbt is what makes that SQL reusable, tested, and traceable instead of copy-pasted.

What "job-ready" means for this phase

Not "wrote a DAG once." A specific, checkable bar: what you can build, defend in an interview, and point to on GitHub by the end of Week 21.

  • A DAG that represents extract, validate, transform, load, and dbt_run as a real dependency graph, not a script that runs top to bottom.

  • A retry and backfill policy you can defend in an interview, not just recite the syntax for.

  • The instinct to clear only the failed task and its downstream tasks, not the whole DAG, when something breaks.

  • dags/gap_commerce_pipeline.py: the artifact an interviewer will actually ask you to walk through.

Airflow is only one phase of the programme.

SQL, Python, pipelines, and cloud build up to this. Testing, CI/CD, Spark, and Kafka build outward from here, the rest of the 24-week core plus a 4-week production capstone picking up exactly where this phase leaves off.

Book your free career call
See the full programme