DataDios Lineage

Overview

Lineage is the live map of how data flows through your stack. DataDios discovers it automatically from the warehouse, the orchestrator, and your code, then keeps it current as those sources change — so you can trace any column back to its origin and forward to every consumer without maintaining a manual catalog.

Key Capabilities

The graph is complete and live

Every dataset, every job, and every column-level dependency lives in one graph. It updates as your warehouse, dbt manifest, and orchestrator change — there is no separate catalog to keep current. Stale lineage is worse than none, so the graph is refreshed continuously rather than snapshotted once.

• Granularity: column-level
• Refresh: continuous
• Curation: the catalog is built from continuous discovery, not hand-entered — but the Entity Catalog below also lets you mint entity types and instances, rename, edit attributes, delete, and group by hand where discovery doesn't cover it

From the DataDios release after v1.13.0, a dbt model or an Airflow DAG whose output is not in the workspace still shows the tables it reads, so impact analysis on those tables reaches it.

Lineage the platform records

Some sources record which tables a job read and wrote while it ran. DataDios draws those jobs between their tables as observed when the job ran, not declared. Databricks jobs and pipelines are drawn this way from the DataDios release after v1.13.0, from the last 90 days of runs — see Databricks: job and pipeline lineage.

Workflows carve the graph into ownership

A workflow is a named slice of the graph that a team owns — its sources, its outputs, and the intermediate nodes between them. Two teams can have overlapping workflows; ownership here means responsibility, not exclusivity, so a shared table can belong to more than one team's view.

• Sources: per workflow
• Outputs: owned
• Overlap: allowed

Propose connections discovery can't see

Some dependencies automated discovery can't observe — a Lambda that reads from S3, a manual CSV upload. In free-form mode the graph's owner can drag a connector between two nodes, and the connection counts as soon as it is drawn: only the owner can add one. If a connection you added runs through a transform, such as a dbt model, it is drawn only while that transform's data source is in the lineage graph. Remove the data source and, once the graph rebuilds, the connection leaves the map, but it is not deleted: add the data source back and it returns. Renaming that data source hides the connection until you draw it again.

Impact: what breaks if this changes?

Select any node and the Impact view lists every downstream consumer. Before you ship a schema change, Impact tells you exactly who depends on the column you're about to drop or rename — so a breaking change becomes a conversation with the right teams instead of an outage.

Pairs with drift and quality

Lineage is most useful next to the rest of the platform. Cross-reference it with the Metadata Timeline to see when a column changed, and with data quality checks to see whether a failing rule's root cause is an upstream table that shifted.

Entity Catalog

Overview

The Catalog tab on the Lineage page is a flat, searchable list of the structural objects DataDios has catalogued — sources, databases, schemas, objects and columns in one table, sorted by name, with each row's full path beneath it. Your DataDios dashboards are listed too, under the Dashboard level.

From the DataDios release after v1.13.0, the instances of any entity type your team creates are listed as well, each under its type's own name, and the name search finds them.

Tags, domains, environments, classifications and groups are entities too, but they are curation vocabulary rather than data objects, so they appear as values you pick in an attribute cell rather than as rows of their own. Domain, Env, Classification and Tags are also the filter rail's four dimensions, once a structural row actually carries the value; Groups is not one of the rail's dimensions.

Where Workflows shows how data moves, the Catalog answers "what do we have, and how is it classified". Within those levels it is not scoped to a workflow: it spans everything in the tenant.

Before you start

The catalog is populated by the entity sync, which publishes through the tenant's default queue. If no queue is configured the catalog stays empty and nothing reports an error — see Queue Configuration. Dashboard rows are the exception: they don't use the queue, so they appear even when no queue is configured (see Dashboards in the catalog).

Dashboards in the catalog

From the DataDios release after v1.13.0, each of your DataDios dashboards has one row, at the Dashboard level. You don't create or manage these rows; DataDios keeps them in step with your dashboards:

• Create a dashboard and its row is added in the same step. If the row can't be written, the create fails and no dashboard is made. A rename or delete works the same way.
• Rename a dashboard and the row's name follows.
• Delete a dashboard and its row leaves the table.

Your Default dashboard has a row from the start: DataDios creates it, with its row, when your user is set up. Dashboards made before this feature are added when DataDios is upgraded.

If you created your own entity type named dashboard before this release, DataDios leaves that type as yours and adds no dashboard rows, not even when it is upgraded. Creating, renaming and deleting dashboards still works but writes nothing to the catalog, and nothing on screen says so. Your dashboards still appear in lineage, with no catalog row behind them. To get the rows, delete your own dashboard type (the delete is refused while anything still uses it). The next time you create or rename a dashboard, DataDios adds its own type back and writes that dashboard's row. To add rows for all your dashboards at once, rebuild a lineage graph you own that has at least one data source in it.

A row belongs to the dashboard, not to its name. Renaming keeps the row, with its tags, domain and other attributes. A new dashboard that reuses a deleted dashboard's name gets a new row and starts with none of the old row's attributes.

You curate a dashboard row like any other: click its Domain, Env, Classification, Tags or Groups cell.

Navigation

• Open Lineage and select the Catalog tab, or go straight to /lineage?tab=catalog.

Reading the table

Column	What it shows
Entity	The object's name, with its full path beneath it (source/database/schema/object/column)
Entity type	What the row is — Source, Database, Schema, Table, Column, or any other type, including one your team created
Kind	What the row is within its type. For a database object: a view, a materialized view, a procedure or a function — a plain table shows a dash. For a file: its extension, such as csv. Any other row shows a dash, because its type already says everything
Domain · Env · Classification · Tags	Curated attributes, where they have been set. Click a cell to edit it
Groups	The groups this entity belongs to. Click to add or remove

Columns, in the toolbar, turns any column off and on again except the entity name and the row actions. The choice is remembered in the browser you made it in, so it does not follow you to another machine or to a colleague.

A path shown with a leading …/ is incomplete — one of its ancestors has been retired or removed. An attribute cell reading not shown means the row carries more values than the table resolves at once; it does not mean the row has none.

Finding things

• Search matches the start of an entity's name and ignores case, so ord finds both orders and ORDERS. It does not match the middle of a name, and it does not search the path.
• Entity type, at the top of the filter rail, narrows the table to the kinds of thing you pick. It is split in two. Default holds the types DataDios knows about — Source, Database, Schema, Table, Column and anything a connector reported — and you can pick several at once. Custom holds the types your team created, and you can pick only one at a time: picking a custom type adds that type's own attributes to the table as extra columns, and two types' attributes have no columns in common. Picking in one group clears the other. Clicking a type you already picked clears it.
• The filter rail on the left also filters by classification, domain, environment and tags. Its counts cover every entity in the tenant. Selecting an Entity type narrows the table but not these counts; applying a rail filter or a search does narrow them, and the counts can then be minimums rather than exact — the rail says Counts are minimums when they are. Each dimension shows its eight most-used values; Show all lists the rest, up to fifty. A dimension with more than fifty values says More values than shown, and the others can't be reached from the rail — use search instead. A value you've selected stays visible either way.

Search, entity type, rail filters and rows-per-page are kept in the page address, so you can bookmark or share a filtered view — the link opens on the first page of results, not the page you were on. Your browser's Back button undoes the last entity-type, rail or rows-per-page change, but not a search: Back after a search leaves the catalog instead of clearing it.

Paging

The catalog uses Back and Next rather than page numbers. The count above the pager shows 500+ once a result set passes five hundred — the exact total is not computed, because counting every match costs more than showing the page.

Rows per page, beside the pager, shows 25, 50, 100 or 200 rows. It starts at 25.

Managing entities

Create covers four kinds, chosen from one dropdown:

Kind	What it makes	Appears in the table?
Structural entity	A source, database, schema, object or column instance — the same kind discovery produces	Yes
Vocabulary value	A tag, domain, environment, classification or group value to curate with	No — it becomes a value you can pick in an attribute cell. Tag, domain, environment and classification values also become a filter-rail entry once applied to a row; group values do not, since Groups isn't a filter-rail dimension
Instance of a declared type	A row of a type you've minted below	Yes — under the type's own name
Entity type	A new type, with its own attribute definitions, to hold instances	No — the type itself is not a row

Dashboards are not in the dropdown: a dashboard's row is created with the dashboard. Through the API, creating an entity of the dashboard type is refused with 409 type_instances_managed.

Creating a name that already exists never duplicates it, and what happens next depends on the kind. A structural entity or a custom-type instance is refused: nothing is created, the dialog stays open and says so, and anything you typed alongside the name — a parent, attribute values — is not applied to the existing row. Edit that row to change its values. A vocabulary value is not refused, because the request carries nothing but the name: you are told it already exists, and the dialog closes, since the value you wanted is there and ready to pick.

When you create an entity type, each attribute definition takes a name, a value type (Text), and optionally Multi (holds several values), Required, and a Default value. A required attribute must be filled in when you create an instance, unless it has a default — then the default is written for you. After the type is created the dialog stays open with the new type selected, so you can create its first instance straight away.

To add an attribute to a type you created earlier, pick that type in Create and choose Define attribute. Every instance of the type gets the new field. A definition can't be edited or removed once it's added, so check the name before you save it. Define attribute is offered only on types you created, not on the built-in levels or vocabularies.

The first and third kinds produce a row you will see here. The table lists the structural levels, dashboards and the types your team created, so a vocabulary value cannot be found by searching or filtering the Catalog. Create one when something else needs it — a tag you are about to apply — not as a way to add a row to this table.

Edit shows the entity's type as a read-only field — a type can't be changed after creation — and renames an entity you created by hand (origin: user_defined). On a discovered row the name belongs to the source system, so the field is shown read-only with the reason underneath — most rows here are discovered, so for those Edit is for attributes only. A rename changes only the display name — existing filters, curation and counts keep pointing at the same row. A dashboard row's name is read-only too, with the reason "The name comes from the dashboard. Rename it on the dashboard's page."

Attributes are edited from the table, not from a form. Click any Domain, Env, Classification, Tags or Groups cell and a picker opens on the spot:

• Click a value to add it; click it again to take it off. Domain and Env hold one value, so picking a second replaces the first. Classification, Tags and Groups hold as many as you like.
• Type in the box to narrow the list, then press Enter to attach a value that isn't there yet — it is created and applied in one step, so there's no separate "add a tag first".
• A cell shows one value plus a count (+2) when there are more; hover the count to see the rest.
• A cell reading not shown carries more values than the table resolves at once. It can't be edited from here, because the picker would have to replace a list it never fully loaded.

The list of existing values shows the first 200. Past that, type the value's exact name and press Enter — it resolves to the existing one rather than making a duplicate.

Delete runs in the background — you'll see the row disappear once it finishes, not immediately. It's permanent. There's no audit trail and no undo. It's only offered on rows you created by hand (origin: user_defined) — discovered rows can't be deleted from here. That includes dashboard rows, which go when you delete the dashboard. A row with children can't be deleted until they are.

Groups are a lightweight way to cluster entities across levels — the groups attribute is seeded by DataDios like Domain, Env, Classification and Tags, though no group values come pre-populated. Add a row to a group, or take it out again, from the Groups cell.

Entity details

Click an entity's name to open its quick view — beside the table on a wide screen, as a drawer over it on a narrower one: its type, how many children it has (with a + once the count is capped), its attributes, and Edit. Press Escape, click X, or click the name again to close it.

For a page of its own, choose Full details in the quick view, or Open entity from the row's actions menu. The entity page shows:

• The trail of parents above it — source, database, schema and so on — each one a link to its own page.
• The entity's name and full path.
• Its attributes, in three groups: Identity (type, and kind when it has one), Governance (Domain, Env, Classification, Tags and Groups, editable in place as in the table), and the attributes of its own type.
• Edit entity and Back to the catalog.

The address of an entity page stays valid after changes in its source:

• If the entity was renamed — confirmed on the Lineage page's Review tab, or matched automatically during the sync — the old address forwards to the new entity and a banner shows the old name and the date of the rename. Renaming a table forwards every column's old address too.
• If it was renamed and the new entity was later retired, the page says This entity was retired and offers a way back to the catalog.
• Otherwise the page says No such entity. That includes an entity that simply stopped appearing in its source, even though the message suggests the link may be mistyped.

Bulk actions

Check the box on any row to select it, or use the checkbox in the table header to select every row on the current page. A toolbar appears above the table showing how many rows are checked, with Add to group, Edit attributes, and Clear to drop the selection.

Both dialogs write as you go — there's no separate save step. Every value you click writes immediately; Done only closes the dialog, it never holds anything back to submit. The write itself runs as a background job, split into one job per entity type in the selection (a table and a column write separately, for example) — a different mechanism from the immediate save a single-row edit uses, so the table can take a few seconds to catch up, and a mixed-type selection can partially succeed: one type's job failing does not undo another type's that already wrote.

• Add to group works like the per-row Groups cell, but for the whole selection at once. The pills shown are the union of groups across the selection, so a pill isn't necessarily on every checked row — removing it takes it off every row that has it.
• Edit attributes offers the same curated attributes as the per-row cells, except Groups, which has its own dialog — usually Domain, Env, Classification and Tags, though a deployment can define fewer. An attribute this deployment can't bulk-edit is called out rather than silently hidden.
• Domain and Env replace, not add — across the whole selection, with no undo. Same rule as the per-row cell (a new pick replaces the old one), but here it applies to every checked row at once: picking a new Domain for 400 checked rows overwrites 400 existing values in one click.
• A checked row with no resolvable type is left out of the write and called out in the dialog — it isn't silently skipped.
• The selection survives Back/Next, so you can build a larger selection across pages — but a search, an Entity type change, or a filter-rail change clears it, the same way any of those resets the table itself.
• A single write accepts at most 25,000 entities of one type. Past that, the dialog's controls disable with an explanation rather than failing the write.