Apache Airflow: scheduling data pipelines written in Python

Updated November 2026

Apache Airflow runs workflows that you write as Python code: it starts each step on schedule, in the right order, retries failures and shows everything in a web UI. As of November 2026 the current release is Airflow 3.3.2, and the 2.x line has reached end of life.

What Airflow is

The project describes Airflow as a platform to programmatically author, schedule and monitor workflows. A workflow is a Dag (directed acyclic graph): a set of tasks with arrows between them showing what must finish before what. "Acyclic" means the arrows never loop back, so every run has a clear start and end.

Airflow does not move or process data itself. It is an orchestrator: each task tells some other system what to do, such as run a SQL query, start a Spark job, copy files to object storage or call an API. Ready-made integrations for clouds, databases and other services come as separately installed provider packages.

Typical uses

The documentation calls Airflow a platform for orchestrating batch workflows: it fits best when a job has a clear start and end and runs on a schedule or after an event. Continuous streams of events are better handled by a streaming system such as Kafka, with Airflow perhaps scheduling the batch jobs around it. And because every workflow is code, Airflow suits teams that prefer writing Python to clicking through a designer.

How it works

A minimal Airflow 3 installation has these parts, according to the project's architecture overview:

Scheduler
Decides when each Dag run and task should start and hands tasks to the executor. The executor is a setting of the scheduler, not a separate program: LocalExecutor runs tasks on the same machine, CeleryExecutor sends them to worker machines through a queue, KubernetesExecutor starts a pod per task.
Dag processor
Reads your Dag files from a Dag bundle (by default a local folder, or a versioned source such as Git) and stores a serialized copy in the database. In Airflow 3 it always runs as its own process, so the scheduler never runs code written by Dag authors.
API server
Serves the web UI and REST API. Workers also report task state through it, so task code no longer talks to the database directly.
Metadata database
Stores the state of every Dag, run, task and variable. PostgreSQL or MySQL in production.
Optional parts
Workers (for Celery or Kubernetes executors), a triggerer that waits on deferred tasks without holding a worker slot, and a plugins folder.
Dag files Dag processor Scheduler Metadata database API server (UI, REST API) Workers run tasks dashed: hands out tasks
In Airflow 3, workers report back through the API server and never get database credentials; only the API server, scheduler, Dag processor and triggerer use the database.

Here is a small Dag in Airflow 3 style, using the airflow.sdk imports that replace the older airflow.decorators paths:

from datetime import datetime
from airflow.sdk import dag, task

@dag(schedule="@daily", start_date=datetime(2026, 1, 1), catchup=False)
def daily_orders():
    @task
    def extract():
        # for example, the number of orders found
        return 1250

    @task
    def report(count):
        print(f"Loaded {count} orders")

    report(extract())

daily_orders()

Save it in the dags folder. Airflow sees that report depends on extract and runs them in that order once a day.

Current versions

Airflow version life cycle, as of November 2026
MajorLatestStateFirst releaseEnd of life
33.3.2MaintainedApril 22, 2025Not yet set
22.11.2EOL (limited maintenance from October 22, 2025)December 17, 2020April 22, 2026
1.101.10.15EOLAugust 27, 2018June 17, 2021

Airflow 3.3.2 is tested with PostgreSQL 14 to 18, MySQL 8.0, 8.4 and Innovation releases, and Kubernetes 1.30 to 1.35. SQLite is for tests and local development only. MariaDB is explicitly not supported as a backend. The project recommends at least 4 GB of memory, and production should run on Linux; on Windows use WSL2 or containers.

Install and try it

With pip and a constraints file

Airflow has hundreds of dependencies, so the project publishes a constraints file per Airflow and Python version that pins a tested set. Always install with it. On Debian or Ubuntu:

sudo apt update
sudo apt install python3-pip python3-venv
python3 -m venv ~/airflow-venv
source ~/airflow-venv/bin/activate
pip install --upgrade pip

export AIRFLOW_HOME=~/airflow
AIRFLOW_VERSION=3.3.2
PYTHON_VERSION="$(python -c 'import sys; print(f"{sys.version_info.major}.{sys.version_info.minor}")')"
pip install "apache-airflow==${AIRFLOW_VERSION}" \
  --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-${AIRFLOW_VERSION}/constraints-${PYTHON_VERSION}.txt"
airflow version
airflow standalone

airflow standalone creates the database, a user and all components in one terminal. Open http://localhost:8080. The generated admin password is in $AIRFLOW_HOME/simple_auth_manager_passwords.json.generated if it does not appear in the output. The project also supports uv as a faster alternative to pip; Poetry and pip-tools are not supported installers.

For a production-style setup you run the parts separately: airflow db migrate, then airflow api-server, airflow scheduler, airflow dag-processor and, if needed, airflow triggerer, usually as systemd services or containers.

With Docker Compose

The project's Compose file starts a CeleryExecutor setup with PostgreSQL and Redis (see also our Docker guide). It needs Docker Compose v2.14.0 or newer and at least 4 GB of memory for Docker:

mkdir airflow-docker && cd airflow-docker
curl -LfO 'https://airflow.apache.org/docs/apache-airflow/3.3.2/docker-compose.yaml'
mkdir -p ./dags ./logs ./plugins ./config
echo -e "AIRFLOW_UID=$(id -u)" > .env
docker compose up airflow-init
docker compose up

Log in at http://localhost:8080 with user airflow, password airflow. Remove everything with docker compose down --volumes --remove-orphans. The project warns that this file is for learning, has no production security guarantees, and recommends Kubernetes with the community Helm chart for production.

Upgrading

  1. Back up the metadata database. If you cannot take a hot backup, stop Airflow first so the copy is consistent. A failed migration without a backup can leave the database half-migrated.
  2. Reinstall with the new version and its constraints file, using the same pip command with a new AIRFLOW_VERSION.
  3. Run airflow db migrate. It is safe to run even when there is nothing to migrate. (airflow db upgrade is the old, deprecated name.) Add --show-sql-only to review the SQL first.
  4. Restart all components. Helm-chart deployments with post-upgrade hooks and managed services run the migration for you.

From 2.x to 3

Airflow 3 has breaking changes. The project's upgrade guide says: be on 2.7 or later (ideally the latest 2.x), back up, trim old data with airflow db clean, and make sure airflow dags reserialize runs without errors. Check your Dag code with Ruff's Airflow rules:

ruff check dags/ --select AIR301 --show-fixes
ruff check dags/ --select AIR301 --fix

The biggest change for Dag authors: task code can no longer open database sessions or import Airflow models directly. Everything goes through the Task SDK and the API server.

Security notes

Security fixes only land in the newest minor line. Airflow releases patches only for the latest MINOR version. Once 3.3.0 shipped, no more 3.2.x releases follow. To get security fixes you must run the latest 3.3.x. Providers are versioned and patched separately, so upgrade them too.

Sources