Apache Airflow

This section explains how to connect an Apache Airflow 2.x instance to DataDios as a data source. DataDios reads Airflow through its REST API. It can also create a new DAG by writing the DAG file into Airflow's DAGs folder (see Creating DAGs); it never triggers, pauses, edits or deletes a DAG.

Airflow 3 (/api/v2) and managed Airflow services that authenticate through cloud IAM (MWAA, Cloud Composer) are not supported yet.

Before you connect

DataDios talks to Airflow's stable REST API (/api/v1). Airflow only accepts Basic-auth requests on that API when the basic-auth backend is enabled. Set this on the Airflow webserver:

AIRFLOW__API__AUTH_BACKENDS=airflow.api.auth.backend.basic_auth

Without it, every connection fails with 401 Invalid User Credentials, which looks exactly like a wrong password. If your Airflow fronts the API with a bearer-token authentication backend, choose the token auth type instead; the setting above is then not needed.

Test Connection reports the Airflow status code rather than echoing Airflow's response body, which can contain a failing task's rendered config. The ones you are most likely to see:

Message	What it usually means
`401 Invalid User Credentials`	Wrong username or password, or the basic-auth backend above is not enabled
`403 Insufficient Privileges`	Credentials are valid but the account lacks read access to DAGs, runs, task instances, datasets or DAG source
`404 Resource Not Found`	The URL does not point at an Airflow 2.x `/api/v1` endpoint — check for a missing or extra path prefix
`502 Gateway Error`	A proxy or load balancer in front of Airflow did not pass the request through

The Airflow account DataDios uses only needs read access to DAGs, DAG runs, task instances, datasets and DAG source, plus XComs if you set Records XCom Key. Airflow's built-in Viewer role covers all of these.

The DataDios server must be able to reach the Airflow webserver URL over HTTP(S). Test Connection first calls Airflow's /api/v1/health, which answers without credentials, and then reads one DAG. So a passing test proves both that Airflow is reachable and that the account can read DAGs.

What you can and cannot do

An Airflow source is for exploring pipelines and drawing lineage. These DataDios features are not available for it:

SQL editor and running queries
Data quality rules, scores and the DQ Hub
AI Studio
Importing or deleting objects (creating a DAG is supported — see Creating DAGs)
Cost analysis

Workload metrics are available — see Workload analysis below.

Metadata history is also available: DataDios records changes to DAG and task metadata, so an edited DAG shows up as a new version in the timeline. See Metadata Timeline.

Steps to Create an Airflow Data Source

Navigate to the Data Sources tab in DataDios and click + CREATE DS
Select Apache Airflow (listed under the Transform category)
Fill in the connection details below
Click Test Connection — a healthy connection returns Status: OK
Click Create to save the data source

Connection details

Field	Required	Description
Airflow URL	Yes	Base URL of the Airflow webserver, for example `https://airflow.example.com`
Auth Type	Yes	`basic` (username and password) or `token` (bearer token)
Username / Password	For `basic`	Airflow account credentials. The password is stored encrypted and never shown again
Token	For `token`	Bearer token. Stored encrypted
Verify SSL	No	`Yes` (default) verifies the server certificate. Set `No` only for an instance behind a self-signed certificate — it applies to this data source alone
DAG Filter	No	Comma-separated DAG ids (`dag_one,dag_two`) to show instead of every DAG
Dataset URI to Table	No	JSON map from an Airflow dataset URI to the DataDios table it names. Needed for lineage — see Dataset lineage
Records XCom Key	No	XCom key your tasks push a row count under, to collect the Records Processed task metric. Leave blank to skip XCom reads — it costs one extra request per task instance
Folder connection holding the DAGs folder	No	One of your own Folder connections, in Remote SSH mode, whose folder is Airflow's DAGs folder. DataDios uploads new DAGs over SSH through it; nothing needs mounting. A Folder connection shared with you is not listed. See Creating DAGs
DAGs Folder	No	Airflow's DAGs folder as mounted into the DataDios server. Needed only to create DAGs from DataDios; left blank, creating a DAG is refused with a message — see Creating DAGs

Creating DAGs

Airflow has no API for creating a DAG: a DAG exists once its Python file is in the scheduler's DAGs folder. DataDios creates a DAG by writing that file there.

Setup (once). Choose one of these:

Over SSH (recommended when Airflow runs on another server):
1. Create a Folder data source in Remote SSH mode: the Airflow server's host, an SSH user that can write to the DAGs folder, and that user's password or key. Set Folder Path to Airflow's DAGs folder itself, as an absolute path, for example /home/airflow/dags. It must be a connection you own; one shared with you does not work here.
2. On the Airflow data source, pick that connection in Folder connection holding the DAGs folder.
Mounted folder (by your DataDios administrator):
1. Mount Airflow's DAGs folder into every DataDios backend server, under the directory the server's AIRFLOW_DAGS_ROOT setting names, in the sub-folder for your tenant: <AIRFLOW_DAGS_ROOT>/<tenant id>/.... DataDios refuses any other folder, so one tenant can never write into another tenant's Airflow.
2. Set DAGs Folder on the Airflow data source to that path as the DataDios server sees it.
If the server has no AIRFLOW_DAGS_ROOT set, creating a DAG is refused with "DAG creation is not enabled on this DataDios server". Ask your DataDios administrator to set it up.

If both are set, Folder connection holding the DAGs folder is used.

Who can create DAGs: only a tenant admin, including on an Airflow source shared with you — what counts is your own role, not the source owner's. A DAG is code that runs inside your Airflow with Airflow's own access to connections and secrets, so creating one is treated like committing to your DAGs repository.

Create a DAG from the Airflow source's object list: click the Create Item icon, enter the DAG ID, and write, paste or upload (.py) the DAG code.

DataDios checks the code is valid Python, that the DAG ID uses only letters, digits, _, . and -, that it matches the dag_id the code declares (when it declares exactly one literal dag_id), and that no DAG file with that name exists yet — it never overwrites a DAG. If one DAG in a request cannot be written, DataDios removes the ones it already wrote. If it cannot remove one, the message names it; delete that file from the DAGs folder before you retry. The new DAG appears in the explorer after Airflow's next scan for new files (dag_dir_list_interval, 5 minutes by default); click refresh then.

Deleting or editing DAGs, and deploying through a Git repository, are not supported yet — use your usual DAG deployment for those.

Exploring an Airflow data source

Airflow is a pipeline orchestrator, not a database, so the explorer tree follows Airflow's own structure instead of database → schema → table:

DAG
└── Tasks     (one row per task in the DAG)

DAG runs are not rows in the tree — they change on every schedule tick.

The DAG row carries whether the DAG is paused, whether it is active, whether it has import errors, and the state of its latest run. These are Airflow-specific extras, so they arrive as columns that are hidden by default. Turn on the ones you want from the column selector in the table toolbar — the icon whose tooltip reads Select Columns.

A latest-run cell is empty either when the DAG has not run, or when its runs are older than the recent-run window DataDios samples across the instance. The two cases are not distinguished in the table.

Selecting a DAG opens its details (schedule, owners, tags and so on). Values that Airflow treats as secret, such as connection strings inside a DAG's default arguments, are redacted before they are shown or stored.

Selecting a task shows the task's attributes as Airflow reports them: its operator, retries, trigger rule, pool, owner, downstream tasks and so on. Secret values are redacted the same way.

Dataset lineage

Airflow expresses lineage through Datasets: a task declares outlets=[Dataset(uri)] and a downstream DAG schedules on schedule=[Dataset(uri)]. DataDios reads these from Airflow and draws source table → DAG → target table.

A dataset URI is an opaque string chosen by the DAG author (s3://bucket/orders, postgres://...), so DataDios cannot infer which registered table it refers to. You tell it in the Dataset URI to Table field, using data_source_name:schema:table values:

{
  "s3://bucket/orders": "warehouse_ds:analytics:orders",
  "s3://bucket/orders_clean": "warehouse_ds:analytics:orders_clean"
}

A dataset URI that is not in the map is skipped with a warning, never guessed. The DAG still appears in lineage; only the edge that depends on the unmapped dataset is missing.
DAGs on a plain schedule_interval that declare no datasets contribute no lineage. That is a correct empty result, not a failure.
The mapped data sources and tables must already be registered and synced in DataDios.
From the DataDios release after v1.13.0, if DataDios cannot read Airflow's datasets during a lineage build, the DAG edges from the last successful build are kept, not removed.
From the DataDios release after v1.13.0, a DAG that only consumes datasets (none it produces maps to a table) is joined to the tables it consumes, so impact analysis on those tables reaches it.

Run a lineage sync from the data source to refresh the graph after you change the map.

Workload analysis

DataDios can collect DAG-run, task and DAG-health metrics from Airflow and chart them in the Workload Analyzer.

To start collecting, expand Workload in the left sidebar, click Extract, then + EXTRACT WORKLOAD. Select your Airflow source and click START EXTRACTION. The full steps are in Landing On The Page.

What is collected

DataDios extracts three hourly metric groups. Run and task metrics are recorded once a run or task instance finishes. DAG health is a snapshot of every DAG taken at each extraction, so a paused or never-run DAG still appears.

Metric group	Metrics
Airflow DAG Run Metrics	Run Duration, Schedule Lag, Queued Duration, Data Freshness, Success Count, Fail Count
Airflow Task Instance Metrics	Task Duration, Queue Wait, Retry Count, Retries Exhausted, Pool Slots, Records Processed
Airflow DAG Health Metrics	Active, Paused and Running DAGs, DAGs With Import Errors, Failing DAGs, Never Run DAGs, Consecutive Failures, Success Rate (recent runs), Hours Since Last Run, Oldest Active Run, Usual Run Interval, Stuck DAGs, Overdue DAGs

The less obvious ones:

Schedule Lag — from the end of the run's data interval to the moment the run started.
Queued Duration / Queue Wait — from queued to started, for a run and for a task instance.
Data Freshness — from the end of the run's data interval to the moment the run finished: how old the data is when it lands.
Records Processed — only collected when the Records XCom Key field is set; see Connection details.
Usual Run Interval — the median gap between a DAG's recent run starts. It needs at least three runs.
Stuck DAGs — a run has been in flight for more than twice the usual run interval.
Overdue DAGs — an active, unpaused DAG with nothing running whose last run started more than twice the usual run interval ago. This catches a DAG that quietly stopped running, which no failure count shows.
Failing DAGs — an active, unpaused DAG whose latest finished run failed.

Schedule Lag and Data Freshness are collected for scheduled runs only. A manual or backfill run has no meaningful data interval, so both are empty for it.

The Airflow dashboard

In the Workload Analyzer, an Airflow source has five dashboard tabs:

Tab	What it shows
DAG Performance	Run duration, run count and daily success-rate trends; a per-DAG summary; Getting Slower — DAGs whose average run time grew between the earlier and later half of the period
Fleet Health	The latest fleet snapshot; Needs Attention — failing, broken, stuck or overdue DAGs; active vs failing DAGs over time
Reliability	Flakiest Tasks by failures and retries; problem DAGs grouped by owner and tag
Scheduler	Schedule lag, DAG-run queue time and data freshness trends; task queue wait by pool and queue
Time Consumers	DAGs and tasks by total time consumed; the longest single runs

Troubleshooting

What you see	What to check
Test Connection says the Airflow REST API did not respond at `/api/v1/health`	The DataDios server cannot reach the Airflow URL (check the address, any path prefix, and firewalls), or the server uses a self-signed certificate. For a self-signed certificate, set Verify SSL to `No` for this data source
Test Connection says the Airflow REST API rejected the credentials, although the password is right	The basic-auth backend is not enabled on the Airflow webserver, or the account cannot read DAGs — see Before you connect
A DAG is missing from the explorer	If DAG Filter is set, only the DAG ids listed there are shown, and they must match exactly (case included). Otherwise check that the Airflow account can read that DAG
A DAG appears but has no tasks under it	The Airflow account cannot read that DAG's tasks. The rest of the instance still loads
A lineage edge is missing	The dataset URI is not in Dataset URI to Table, or it maps to a table that is not registered and synced in DataDios. Fix the map, then run a lineage sync — see Dataset lineage
Records Processed is empty	Records XCom Key is blank, the account cannot read XComs, your tasks do not push a value under that key, or the hourly 2,000-task-instance cap was reached
Schedule Lag and Data Freshness are empty for some runs	Those runs were manual or backfill runs. Both metrics are collected for scheduled runs only

Limits

Very large instances: the explorer lists up to 10,000 DAGs. Beyond that the listing fails with an explicit message rather than showing an incomplete tree. DAG Filter narrows what the explorer shows, but DataDios still reads the whole DAG list from Airflow, so it does not raise or work around this cap.
Records Processed is collected for at most 2,000 task instances per hourly extraction. Past that the metric is empty for the rest of the window.
DAG code created from DataDios is capped at 1 MB per DAG file.

Before you connect​

What you can and cannot do​

Steps to Create an Airflow Data Source​

Connection details​

Creating DAGs​

Exploring an Airflow data source​

Dataset lineage​

Workload analysis​

What is collected​

The Airflow dashboard​

Troubleshooting​

Limits​

See also​