DataDios Lineage

Overview

Lineage is the live map of how data flows through your stack, drawn on the Graph tab as a single canvas. DataDios discovers it automatically from the warehouse, the orchestrator, and your code, then keeps it current as those sources change — so you can trace any column back to its origin and forward to every consumer without maintaining a manual catalog.

Key Capabilities

The graph is complete and live

Every dataset, every job, and every column-level dependency lives in one graph. It updates as your warehouse, dbt manifest, and orchestrator change — there is no separate catalog to keep current. Stale lineage is worse than none, so the graph is refreshed continuously rather than snapshotted once.

• Granularity: column-level
• Refresh: continuous
• Curation: the catalog is built from continuous discovery, not hand-entered — but the Entity Catalog below also lets you mint entity types and instances, rename, edit attributes, delete, and group by hand where discovery doesn't cover it

From the DataDios release after v1.13.0, a dbt model or an Airflow DAG whose output is not in the workspace still shows the tables it reads, so impact analysis on those tables reaches it.

Lineage the platform records

Some sources record which tables a job read and wrote while it ran. DataDios draws those jobs between their tables and marks the edges Discovered: observed when the job ran, never Declared. Databricks jobs and pipelines are drawn this way from the DataDios release after v1.13.0, from the last 90 days of runs — see Databricks: job and pipeline lineage.

Swimlanes group the graph by the kind of system

The canvas lays your sources out in lanes, left to right, in the direction data travels: Operational DBs → Pipelines & Staging → Warehouses → BI & Reports. A Postgres database sits in the first lane, dbt and your S3 landing buckets in the second, Snowflake or Databricks in the third. Reading the canvas left to right is reading your pipeline in order.

Four more lanes describe systems that sit beside the flow rather than in it: Policies (Collibra, Atlas, IGC), Identities (AWS Secrets Manager, Azure Key Vault), AI & Models (the LLM providers and vector stores) and Infrastructure (cloud accounts, hosts, logs). Anything DataDios has no lane for lands in Other / Custom.

• Flow lanes: always shown
• Context lanes: shown when populated
• Policies / Identities: click the lane chip to add them

Two behaviours are worth knowing, because a lane you expect can be absent:

• A context lane appears only when you actually have a connector in it, so an account with no model or host connector does not carry blank columns.
• Policies and Identities start switched off even when populated — governance and secret-store connectors are usually reference material rather than part of the flow you came to read. Click their lane chip to bring them onto the canvas. Every lane chip toggles, so you can also narrow the canvas to just the two lanes you are comparing.

A connector's lane is decided by the backend's connector registry, not by a list kept inside the UI, so a newly supported source appears in the right lane with no UI change. The lane follows what DataDios does with a source rather than how the vendor is categorised: an object store is staging because data lands there and is read out by something else, while Hadoop is a warehouse because you query it. DataDios dashboards are the one exception: they have no connector, so they always sit in BI & Reports (see Dashboards show which tables they read).

If that registry cannot be reached, DataDios says so rather than guessing: every surface that shows a lane reports Lanes are unavailable. In the entity and relationship tables the lane cells are left blank, so an unknown lane is never mistaken for a real one. The canvas has to draw each source somewhere, so it places them all in Other / Custom — which is why it carries the same notice, to say that the grouping you are looking at is not the real one.

Dashboards show which tables they read

From the DataDios release after v1.13.0, your DataDios dashboards appear on the graph as cards in the BI & Reports lane, linked to the tables and data sources their widgets use. So before you change or drop a table, you can see which dashboards read it. Each link says how, one line per widget, for example Widget "Revenue by region" runs SQL on snow-prod: reads sales.orders, sales.customers.

• SQL widgets, pinned from an AI Studio worksheet with Add to dashboard, link from each table their query reads. If DataDios finds no table in the SQL, the widget links only to its data source.
• Workload charts, insights and metadata change timelines link from their data source, not from tables.
• Widgets with no data source, such as a pinned chat answer, are not linked.

Hover a link on the canvas, or read the How column in the Relationships tab, to see these lines. Select a dashboard card and choose Open dashboard → to go to the dashboard.

Why a dashboard or a link is missing

• The graph hasn't been rebuilt since you pinned the widget. Dashboards are added when the graph is rebuilt. As the owner, select Rebuild to run one now.
• It isn't your graph, or your data source. Only the graph owner's dashboards are drawn, and only their widgets on data sources in the graph that the owner registered. People you share a graph with see none of your dashboards. A dashboard with no such widget, such as a new empty one, is not drawn.
• DataDios couldn't find the table. A table links only when DataDios finds it under that data source, in both the Entity Catalog and the graph; syncing the source's metadata puts it there. In the Entities tab, the dashboard's row says what is missing, for example 3 of 4 widgets linked · 1 table unmatched; select +N more under it to list the unmatched tables.
• The name fits more than one table. DataDios never guesses. For orders in two schemas of one database, write sales.orders in the SQL. For matches in different databases of one data source, the widget's database picks one. From the DataDios release after v1.13.0, Add to dashboard in AI Studio records the database the query ran in, or the catalog on Databricks, for source types where a worksheet runs in a chosen database, such as PostgreSQL, Snowflake and DuckDB. A full name in the SQL, such as mydb.sales.orders, wins over the recorded database. A widget pinned in AI Studio before this release has none and is not updated: pin the result again, or write the full name in the SQL.
• The dashboard reads more than 200 tables. Lineage takes the first 200 table names, in the order the widgets are laid out. A widget left with none of its tables is skipped entirely: no link to its tables or to its data source.
• The lane is full. The BI & Reports lane shows at most ten cards, data sources first, then dashboards by name; its footer then reads 10 of N shown. Find the rest in the Entities and Relationships tabs.

You can't draw a link to or from a dashboard by hand; its links come only from its widgets.

Workflows carve the graph into ownership

A workflow is a named slice of the graph that a team owns — its sources, its outputs, and the intermediate nodes between them. Two teams can have overlapping workflows; ownership here means responsibility, not exclusivity, so a shared table can belong to more than one team's view.

• Sources: per workflow
• Outputs: owned
• Overlap: allowed

Propose connections discovery can't see

Some dependencies automated discovery can't observe — a Lambda that reads from S3, a manual CSV upload. In free-form mode the graph's owner can drag a connector between two nodes, and the connection counts as soon as it is drawn: only the owner can add one. If a connection you added runs through a transform, such as a dbt model, it is drawn only while that transform's data source is in the lineage graph. Remove the data source and, once the graph rebuilds, the connection leaves the map, but it is not deleted: add the data source back and it returns. Renaming that data source hides the connection until you draw it again.

Impact: what breaks if this changes?

Select any node and the Impact view lists every downstream consumer. Before you ship a schema change, Impact tells you exactly who depends on the column you're about to drop or rename — so a breaking change becomes a conversation with the right teams instead of an outage.

When column lineage is cut short

A column's lineage is read by walking its connections, starting from the column you selected. On a very large graph the walk stops at a limit instead of slowing everything down. From the DataDios release after v1.13.0, when that happens the lineage view says so with an amber note — Some lineage isn't shown — this result is partial. — rather than showing part of the answer as if it were all of it.

The note is not an error. What is listed beside it is real lineage; there is more that was not read. It appears in two places:

• The side panel, above a selected column's Upstream and Downstream lists. While the note shows, an empty list reads None shown instead of None — leaf column or None — terminal column: none were read, which does not mean the column has none.
• Column mapping, between the drawing and the mappings table. If nothing could be drawn, it reads No column mapping shown. instead of No downstream column lineage for this column.

The limits that cut a walk short:

• Upstream and Downstream lists: one step either way, up to 499 neighbours.
• Column mapping: up to five steps from the column; at most 10,000 objects in one walk, and at most 5,000 connections read for each batch of 100 objects.

The note can also appear on a small graph. The walk does not go level by level, so an object it first reaches by a route longer than five steps is left out, along with the connection to it.

There is no Load more: a walk that is cut short ends there. To see further, click a column in the Upstream or Downstream list to select it, and the walk starts again from that column.

Through the API, the same thing is reported as page_info.truncated on GET /data-lineage/entities/<id>/graph (on any page of a walk) and truncated on GET /data-lineage/entities/<id>/neighbors.

Pairs with drift and quality

Lineage is most useful next to the rest of the platform. Cross-reference it with the Metadata Timeline to see when a column changed, and with data quality checks to see whether a failing rule's root cause is an upstream table that shifted.

Entity Catalog

Overview

The Catalog tab on the Lineage page is a flat, searchable list of the structural objects DataDios has catalogued — sources, databases, schemas, objects and columns in one table, sorted by name, with each row's full path beneath it. Your DataDios dashboards are listed too, under the Dashboard level.

From the DataDios release after v1.13.0, the instances of any entity type your team creates are listed as well, each under its type's own name, and the name search finds them.

Tags, domains, environments, classifications and groups are entities too, but they are curation vocabulary rather than data objects, so they appear as values you pick in an attribute cell rather than as rows of their own. Domain, Env, Classification and Tags are also the filter rail's four dimensions, once a structural row actually carries the value; Groups is not one of the rail's dimensions.

Where Workflows shows how data moves, the Catalog answers "what do we have, and how is it classified". Within those levels it is not scoped to a workflow: it spans everything in the tenant.

Before you start

The catalog is populated by the entity sync, which publishes through the tenant's default queue. If no queue is configured the catalog stays empty and nothing reports an error — see Queue Configuration. Dashboard rows are the exception: they don't use the queue, so they appear even when no queue is configured (see Dashboards in the catalog).

Dashboards in the catalog

From the DataDios release after v1.13.0, each of your DataDios dashboards has one row, at the Dashboard level. You don't create or manage these rows; DataDios keeps them in step with your dashboards:

• Create a dashboard and its row is added in the same step. If the row can't be written, the create fails and no dashboard is made. A rename or delete works the same way.
• Rename a dashboard and the row's name follows.
• Delete a dashboard and its row leaves the table.

Your Default dashboard has a row from the start: DataDios creates it, with its row, when your user is set up. Dashboards made before this feature are added when DataDios is upgraded.

If you created your own entity type named dashboard before this release, DataDios leaves that type as yours and adds no dashboard rows, not even when it is upgraded. Creating, renaming and deleting dashboards still works but writes nothing to the catalog, and nothing on screen says so. Your dashboards still appear in lineage, with no catalog row behind them. To get the rows, delete your own dashboard type (the delete is refused while anything still uses it). The next time you create or rename a dashboard, DataDios adds its own type back and writes that dashboard's row. To add rows for all your dashboards at once, rebuild a lineage graph you own that has at least one data source in it.

A row belongs to the dashboard, not to its name. Renaming keeps the row, with its tags, domain and other attributes. A new dashboard that reuses a deleted dashboard's name gets a new row and starts with none of the old row's attributes.

You curate a dashboard row like any other: click its Domain, Env, Classification, Tags or Groups cell. The same row is the dashboard's card in lineage.

Navigation

• Open Lineage and select the Catalog tab, or go straight to /lineage?tab=catalog.

Reading the table

Column	What it shows
Entity	The object's name, with its full path beneath it (source/database/schema/object/column)
Entity type	What the row is — Source, Database, Schema, Table, Column, or any other type, including one your team created
Kind	What the row is within its type. For a database object: a view, a materialized view, a procedure or a function — a plain table shows a dash. For a file: its extension, such as csv. Any other row shows a dash, because its type already says everything
Domain · Env · Classification · Tags	Curated attributes, where they have been set. Click a cell to edit it
Groups	The groups this entity belongs to. Click to add or remove

Columns, in the toolbar, turns any column off and on again except the entity name and the row actions. The choice is remembered in the browser you made it in, so it does not follow you to another machine or to a colleague.

A path shown with a leading …/ is incomplete — one of its ancestors has been retired or removed. An attribute cell reading not shown means the row carries more values than the table resolves at once; it does not mean the row has none.

Finding things

• Search matches the start of an entity's name and ignores case, so ord finds both orders and ORDERS. It does not match the middle of a name, and it does not search the path.
• Entity type, at the top of the filter rail, narrows the table to the kinds of thing you pick. It is split in two. Default holds the types DataDios knows about — Source, Database, Schema, Table, Column and anything a connector reported — and you can pick several at once. Custom holds the types your team created, and you can pick only one at a time: picking a custom type adds that type's own attributes to the table as extra columns, and two types' attributes have no columns in common. Picking in one group clears the other. Clicking a type you already picked clears it.
• The filter rail on the left also filters by classification, domain, environment and tags. Its counts cover every entity in the tenant. Selecting an Entity type narrows the table but not these counts; applying a rail filter or a search does narrow them, and the counts can then be minimums rather than exact — the rail says Counts are minimums when they are. Each dimension shows its eight most-used values; Show all lists the rest, up to fifty. A dimension with more than fifty values says More values than shown, and the others can't be reached from the rail — use search instead. A value you've selected stays visible either way.

Search, entity type, rail filters and rows-per-page are kept in the page address, so you can bookmark or share a filtered view — the link opens on the first page of results, not the page you were on. Your browser's Back button undoes the last entity-type, rail or rows-per-page change, but not a search: Back after a search leaves the catalog instead of clearing it.

Paging

The catalog uses Back and Next rather than page numbers. The count above the pager shows 500+ once a result set passes five hundred — the exact total is not computed, because counting every match costs more than showing the page.

Rows per page, beside the pager, shows 25, 50, 100 or 200 rows. It starts at 25.

Managing entities

Create covers four kinds, chosen from one dropdown:

Kind	What it makes	Appears in the table?
Structural entity	A source, database, schema, object or column instance — the same kind discovery produces	Yes
Vocabulary value	A tag, domain, environment, classification or group value to curate with	No — it becomes a value you can pick in an attribute cell. Tag, domain, environment and classification values also become a filter-rail entry once applied to a row; group values do not, since Groups isn't a filter-rail dimension
Instance of a declared type	A row of a type you've minted below	Yes — under the type's own name
Entity type	A new type, with its own attribute definitions, to hold instances	No — the type itself is not a row

Dashboards are not in the dropdown: a dashboard's row is created with the dashboard. Through the API, creating an entity of the dashboard type is refused with 409 type_instances_managed.

Creating a name that already exists never duplicates it, and what happens next depends on the kind. A structural entity or a custom-type instance is refused: nothing is created, the dialog stays open and says so, and anything you typed alongside the name — a parent, attribute values — is not applied to the existing row. Edit that row to change its values. A vocabulary value is not refused, because the request carries nothing but the name: you are told it already exists, and the dialog closes, since the value you wanted is there and ready to pick.

When you create an entity type, each attribute definition takes a name, a value type (Text), and optionally Multi (holds several values), Required, and a Default value. A required attribute must be filled in when you create an instance, unless it has a default — then the default is written for you. After the type is created the dialog stays open with the new type selected, so you can create its first instance straight away.

To add an attribute to a type you created earlier, pick that type in Create and choose Define attribute. Every instance of the type gets the new field. A definition can't be edited or removed once it's added, so check the name before you save it. Define attribute is offered only on types you created, not on the built-in levels or vocabularies.

The first and third kinds produce a row you will see here. The table lists the structural levels, dashboards and the types your team created, so a vocabulary value cannot be found by searching or filtering the Catalog. Create one when something else needs it — a tag you are about to apply — not as a way to add a row to this table.

Edit shows the entity's type as a read-only field — a type can't be changed after creation — and renames an entity you created by hand (origin: user_defined). On a discovered row the name belongs to the source system, so the field is shown read-only with the reason underneath — most rows here are discovered, so for those Edit is for attributes only. A rename changes only the display name — existing filters, curation and counts keep pointing at the same row. A dashboard row's name is read-only too, with the reason "The name comes from the dashboard. Rename it on the dashboard's page."

Attributes are edited from the table, not from a form. Click any Domain, Env, Classification, Tags or Groups cell and a picker opens on the spot:

• Click a value to add it; click it again to take it off. Domain and Env hold one value, so picking a second replaces the first. Classification, Tags and Groups hold as many as you like.
• Type in the box to narrow the list, then press Enter to attach a value that isn't there yet — it is created and applied in one step, so there's no separate "add a tag first".
• A cell shows one value plus a count (+2) when there are more; hover the count to see the rest.
• A cell reading not shown carries more values than the table resolves at once. It can't be edited from here, because the picker would have to replace a list it never fully loaded.

The list of existing values shows the first 200. Past that, type the value's exact name and press Enter — it resolves to the existing one rather than making a duplicate.

Delete runs in the background — you'll see the row disappear once it finishes, not immediately. It's permanent. There's no audit trail and no undo. It's only offered on rows you created by hand (origin: user_defined) — discovered rows can't be deleted from here. That includes dashboard rows, which go when you delete the dashboard. A row with children can't be deleted until they are.

Groups are a lightweight way to cluster entities across levels — the groups attribute is seeded by DataDios like Domain, Env, Classification and Tags, though no group values come pre-populated. Add a row to a group, or take it out again, from the Groups cell.

Entity details

Click an entity's name to open its quick view — beside the table on a wide screen, as a drawer over it on a narrower one: its type, how many children it has (with a + once the count is capped), its attributes, and Edit. Press Escape, click X, or click the name again to close it.

For a page of its own, choose Full details in the quick view, or Open entity from the row's actions menu. The entity page shows:

• The trail of parents above it — source, database, schema and so on — each one a link to its own page.
• The entity's name and full path.
• Its attributes, in three groups: Identity (type, and kind when it has one), Governance (Domain, Env, Classification, Tags and Groups, editable in place as in the table), and the attributes of its own type.
• Edit entity and Back to the catalog.

The address of an entity page stays valid after changes in its source:

• If the entity was renamed — confirmed on the Lineage page's Review tab, or matched automatically during the sync — the old address forwards to the new entity and a banner shows the old name and the date of the rename. Renaming a table forwards every column's old address too.
• If it was renamed and the new entity was later retired, the page says This entity was retired and offers a way back to the catalog.
• Otherwise the page says No such entity. That includes an entity that simply stopped appearing in its source, even though the message suggests the link may be mistyped.

Bulk actions

Check the box on any row to select it, or use the checkbox in the table header to select every row on the current page. A toolbar appears above the table showing how many rows are checked, with Add to group, Edit attributes, and Clear to drop the selection.

Both dialogs write as you go — there's no separate save step. Every value you click writes immediately; Done only closes the dialog, it never holds anything back to submit. The write itself runs as a background job, split into one job per entity type in the selection (a table and a column write separately, for example) — a different mechanism from the immediate save a single-row edit uses, so the table can take a few seconds to catch up, and a mixed-type selection can partially succeed: one type's job failing does not undo another type's that already wrote.

• Add to group works like the per-row Groups cell, but for the whole selection at once. The pills shown are the union of groups across the selection, so a pill isn't necessarily on every checked row — removing it takes it off every row that has it.
• Edit attributes offers the same curated attributes as the per-row cells, except Groups, which has its own dialog — usually Domain, Env, Classification and Tags, though a deployment can define fewer. An attribute this deployment can't bulk-edit is called out rather than silently hidden.
• Domain and Env replace, not add — across the whole selection, with no undo. Same rule as the per-row cell (a new pick replaces the old one), but here it applies to every checked row at once: picking a new Domain for 400 checked rows overwrites 400 existing values in one click.
• A checked row with no resolvable type is left out of the write and called out in the dialog — it isn't silently skipped.
• The selection survives Back/Next, so you can build a larger selection across pages — but a search, an Entity type change, or a filter-rail change clears it, the same way any of those resets the table itself.
• A single write accepts at most 25,000 entities of one type. Past that, the dialog's controls disable with an explanation rather than failing the write.