Phase 02, in depth
Where Python sits in a data platform
Every piece of data GAP Commerce generates (a customer signing up, an order being placed, a payment clearing) moves through a system that looks roughly like this. Python is where that system starts: the tool that takes data from wherever it begins and hands it off clean to everything after it.
- sourcesfiles · APIs · DBs
- ingestionpythonyou are here
- processingpandas
- orchestrationairflow
- storageADLS Gen2 · OneLake
- warehousesnowflake
- consumptionBI · ML
Why Python, specifically
Most of the transformation logic in a real data platform lives in SQL. Python is the glue: it pulls data from files, APIs, and databases, validates and reshapes it, and automates the workflow that moves it forward.
That's a deliberately narrower claim than "Python is everything." It isn't. It's the entry point into building, which is exactly why it's taught early, and taught deep.
Not the query language. SQL runs inside the warehouse. Python hands data to it, not the other way round.
Not the distributed engine. Once data outgrows one machine's memory, Spark takes over the processing.
Not the durable log. Data that never stops arriving is Fabric Eventstream's job (or Kafka's), not a Python script's.
Not the scheduler. Fabric Data Factory (or Airflow) decides when your Python functions actually run, and what happens if they fail.
The 15 stages, stage by stage
Weeks 5-7 of the published programme, opened up. Every stage follows the same shape: why a data engineer needs it, and what you actually build.
What Stage 06 actually feels like
Production data engineering is mostly about anticipating failure, not writing the happy path. Here's a real decision your pipeline has to make. Try one, then see the reasoning.
Stage 06 · Error handling
Your pipeline just read the next record from customers.csv. The email field is empty.
What should your pipeline do?
Where Python stops scaling
None of these are weaknesses in Python. They're the reasons the rest of the data engineering stack exists. Every card below follows the same shape, on purpose: what it is, why it exists, when you'd reach for it, and why it's next.
SQL
Coming nextPython can manipulate data. But how do you query millions of records that already live inside a database?
- What is it?
- The language every warehouse and most operational databases are actually queried in.
- Why do I need it?
- Python and SQL are co-required in the large majority of data engineer job postings. They're learned together, not sequentially.
- When will I use it?
- The moment your pipeline needs to read from or write to a real warehouse, not just a file.
- Why am I learning it next?
- It's introduced inside this Python module already, at Stage 08: Databases.
Spark / PySpark
Coming nextWhat happens when the dataset no longer fits comfortably in memory on one machine?
- What is it?
- A distributed processing engine that splits work across a cluster of machines instead of one.
- Why do I need it?
- Pandas loads a dataset entirely into one machine's memory and computes eagerly. That's a real, practical ceiling, not a metaphor.
- When will I use it?
- When a job that used to run in Pandas starts running out of memory, or would take hours a distributed job takes minutes.
- Why am I learning it next?
- You already know the data structures Spark manipulates. PySpark is largely the same ideas, at a different scale.
Streaming (Eventstream + Kafka)
Coming nextWhat happens when data doesn't arrive as a file once a day, but continuously, all day, forever?
- What is it?
- Microsoft Fabric's Eventstream and Eventhouse route continuous events into the same platform as everything else; Apache Kafka is the industry-standard alternative, a distributed, durable log that producers append events to and consumers read from independently.
- Why do I need it?
- A file represents data that already happened. Both Eventstream and Kafka represent data that is continuously happening, so the platform's own data lands there first, with Kafka taught hands-on because it's still the most job-relevant standalone streaming tool.
- When will I use it?
- When 'once a day' stops being good enough: clickstreams, live orders, sensor data.
- Why am I learning it next?
- Generators (Stage 12) are the closest thing Python has to this idea. Eventstream and Kafka both take it further, in two different tools.
Orchestration (Fabric + Airflow)
Read the deep diveWho runs your pipeline tomorrow at 2 AM? What happens if step two fails but step one already succeeded?
- What is it?
- Microsoft Fabric Data Factory is GAP Commerce's own orchestrator, a pipeline of activities with dependencies, retries, and a schedule; Apache Airflow does the same job as code, DAGs of tasks instead of a visual pipeline, and is still the orchestrator most job postings name.
- Why do I need it?
- A hand-written script has no built-in concept of 'retry this three times' or 'don't start step three until steps one and two both succeed.' Both tools answer that the same way, with different syntax.
- When will I use it?
- The moment one script becomes several, and 'someone runs it manually' stops being an acceptable answer.
- Why am I learning it next?
- You've already written extract(), transform(), and load() as separate functions. Fabric Data Factory (and Airflow, taught alongside it) is what decides when each of them runs.
Cloud (Azure + Fabric)
Coming nextWhat happens when your pipeline has to leave your laptop and run in production?
- What is it?
- Managed infrastructure on Microsoft Azure, plus Microsoft Fabric, the unified platform that ties Data Factory, OneLake, Lakehouse/Warehouse, and Power BI together.
- Why do I need it?
- A local file becomes Azure Blob Storage / ADLS Gen2. A local database becomes a managed service. A local script becomes a scheduled, monitored job. Fabric is where those pieces meet as one first-party product.
- When will I use it?
- The day your pipeline needs to run whether or not your laptop is open.
- Why am I learning it next?
- Every concept transfers. This is about where the same pipeline actually lives once it's real, and it's the platform GAP Commerce actually runs on.
What "job-ready" actually means here
Not "completed the module." A specific, checkable bar for every core competency this module builds.
| Competency | Beginner | Developing | Job-ready |
|---|---|---|---|
| Python fundamentals | Can read simple code | Writes working scripts with guidance | Writes clean, correct scripts unaided |
| Data structures | Recognizes lists and dictionaries | Manipulates them with loops | Uses comprehensions and nested structures fluently |
| Functions | Can call a function | Writes basic functions | Designs reusable, testable pipeline functions |
| File processing | Can open a file | Reads and writes CSV and JSON | Handles multiple formats and malformed input |
| APIs | Understands what an API is | Makes a basic GET request | Handles auth, pagination, and retries |
| Databases | Knows what SQL is | Runs a query from Python | Uses parameterized queries and transactions |
| Error handling | Knows try/except exists | Catches known exceptions | Designs for failure: custom exceptions, retry logic |
| Testing & logging | Has run a test once | Writes a basic pytest test | Tests and logs a pipeline as a habit, not an afterthought |
| Pipeline thinking | Sees Python as a script | Sees Python as connected steps | Sees Python as one component in a larger data system |
Python is only one phase of the programme.
SQL came first. Pipelines, cloud, Spark, and Kafka build outward from here, the rest of the 24-week core plus a 4-week production capstone picking up exactly where this phase leaves off.