Apache Airflow DAG Patterns

Production-ready patterns for Apache Airflow including DAG design, operators, sensors, testing, and deployment strategies.

When to Use This Skill

• Creating data pipeline orchestration with Airflow
• Designing DAG structures and dependencies
• Implementing custom operators and sensors
• Testing Airflow DAGs locally
• Setting up Airflow in production
• Debugging failed DAG runs

Core Concepts

1. DAG Design Principles

2. Task Dependencies

# Linear
task1 >> task2 >> task3

# Fan-out
task1 >> [task2, task3, task4]

# Fan-in
[task1, task2, task3] >> task4

# Complex
task1 >> task2 >> task4
task1 >> task3 >> task4

Quick Start

# dags/example_dag.py
from datetime import datetime, timedelta
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.operators.empty import EmptyOperator

default_args = {
    'owner': 'data-team',
    'depends_on_past': False,
    'email_on_failure': True,
    'email_on_retry': False,
    'retries': 3,
    'retry_delay': timedelta(minutes=),
    : ,
    : timedelta(hours=),
}

 DAG(
    dag_id=,
    default_args=default_args,
    description=,
    schedule=,  
    start_date=datetime(, , ),
    catchup=,
    tags=[, ],
    max_active_runs=,
)  dag:

    start = EmptyOperator(task_id=)

     ():
        execution_date = context[]
        
         {: }

    extract = PythonOperator(
        task_id=,
        python_callable=extract_data,
    )

    end = EmptyOperator(task_id=)

    start >> extract >> end