Airflow: Fundamentals to Your First DAG

Modern workflows are a pile of interconnected tasks with dependencies and schedules. Managing that by hand, or with cron, works right up until it doesn’t, and the failure is gradual enough that you don’t notice you’ve built a distributed system out of crontabs until something breaks at 2am.

I wrote these three as a series. Merged here.

What Airflow is

An open-source platform for developing, scheduling, and monitoring batch-oriented workflows, with a Python framework and a web UI for watching things run and fail.

The organizing idea is workflows as code. Everything is defined in Python, which buys you version control, code review, testing, and the ability to generate workflows dynamically rather than clicking them into existence.

It scales from one machine to distributed execution, connects to most things through providers and plugins, and uses Jinja templating for parameterization.

When not to use it

More useful than the feature list, and more often ignored.

Real-time streaming. Airflow is batch. It schedules at intervals. It is not Kafka and won’t pretend to be. Using both together is normal: Kafka ingests, Airflow orchestrates the batch work downstream.

Sub-minute scheduling. The scheduler is built for minutes and longer. If you need something every few seconds, you want Celery Beat or an event-driven serverless setup.

Heavy computation. Airflow orchestrates, it doesn’t process. Let Spark, Flink, or Dask do the work and let Airflow decide when they run. People get this wrong and end up with worker nodes doing pandas operations on datasets that should never have been on a worker node.

Low-code preference. It’s code-first. If your team wants a UI, Step Functions or Cloud Workflows will make everyone happier.

simple workflows. Three sequential tasks with no dependencies is a cron job. Don’t install a scheduler to run a scheduler.

Core concepts

DAG. The workflow blueprint. Directed because tasks flow one way, acyclic because nothing can depend on itself. A collection of tasks plus their execution order.

Operator. What a task does. A template for a unit of work: run a Python function, execute a shell command, run SQL. You supply the arguments, Airflow handles execution.

Task. An instance of an operator inside a DAG. The operator defines what, the task defines when and with what.

Sensor. An operator that waits for something external before letting the workflow continue. A file appearing, a row changing, an API returning what you expect. Useful, and also the most common source of workflows that quietly hang.

Core components

Scheduler. Decides what runs and when, and hands work to the executor. The brain.

Executor. Determines how tasks actually run. Local for development, Celery or Kubernetes for anything real.

Webserver. The UI. Trigger, pause, rerun, and read logs without touching a terminal.

Metadata database. Where state lives. Every task instance, every run, every connection. Back it up, because losing it means losing all your execution history.

DAG files directory. Where your Python lives and where the scheduler goes looking.

Running it locally

Docker Compose is the path of least resistance. The official compose file brings up scheduler, webserver, database, and executor together, and you mount a local dags/ folder so files you write appear in the UI without a rebuild.

The Astro CLI is the faster route if you’d rather not tune the compose file yourself. astro dev init generates a configured project with an example DAG to poke at.

Either way, the first thing to check is that your dags/ mount is actually working. A surprising amount of early Airflow confusion is a file the scheduler cannot see.

Anatomy of a DAG

default_args holds the settings shared across tasks: owner, retry behavior, start date, alerting. Centralizing them keeps tasks consistent.

The DAG object takes an ID, a description, those default args, and a schedule. It’s the container.

Tasks and operators are the work itself, each with a unique task ID.

Dependencies set execution order, usually with bitshift operators (task1 >> task2), or set_upstream and set_downstream if you prefer the verbose form.

The schedule takes a cron expression, a timedelta, or None for manual triggering only.

A working example

The one I built was an ETL against a public cocktail API: call the endpoint, check it responded, flatten the JSON, write to Postgres. Small enough to follow, complete enough to be real.

The shape:

from airflow import DAG
from airflow.providers.postgres.operators.postgres import PostgresOperator
from airflow.providers.http.sensors.http import HttpSensor
from airflow.providers.http.operators.http import SimpleHttpOperator  # renamed HttpOperator in later providers
from airflow.operators.python import PythonOperator
from airflow.providers.postgres.hooks.postgres import PostgresHook
from datetime import datetime, timedelta
from pandas import json_normalize
import json, logging

Tasks run in order: create the target table, sensor waits for the API to be available, HTTP operator pulls the data, a Python callable normalizes it to CSV, a second Python callable copies it into Postgres.

Two things worth pulling out, because they’re the parts people trip on.

XCom is how one task hands data to the next. ti.xcom_pull(task_ids=...) retrieves what an upstream task pushed. It works well for small payloads like an API response and badly for anything large, because it goes through the metadata database. If you’re moving real volume, write to storage and pass the path.

Validate what you got. The processing function should check the structure before assuming it and raise clearly when the shape is wrong. An ETL that fails loudly on bad input is worth considerably more than one that succeeds with nothing in it, and the second failure mode is much harder to notice.

Helper functions live outside the DAG context and get called by PythonOperators. Keeping them separate also makes them testable without Airflow running, which is the main reason to do it.

---

A note from 2026

The code above targets Airflow 2.x. Airflow 3 landed after these posts and changed enough that copying this directly will cause problems.

The specifics worth knowing before you do: the provider import paths moved and several operators were reorganized, schedule_interval has given way to schedule, and the execution model around dataset and asset-driven scheduling changed meaningfully. Check the current documentation rather than trusting the imports in this post.

The concepts all survived. DAGs, operators, tasks, sensors, XCom, and the scheduler-executor-database split are the same ideas they were. The “when not to use Airflow” section in particular has aged well, and remains the part I’d most want someone to read before they install anything.