Skip to main content

DataDios Data Source

Overview​

Data Sources in DataDios are configured connections to external databases, APIs, files, and other data systems. They serve as the foundation for all data operations within the platform, enabling you to connect to, explore, and manage various types of data repositories.

Accessing Data Sources​

  1. Navigate to the Data Source section from the left navigation menu
  2. Click on Sources to view all available data sources
  3. The main interface displays a table with columns for Name, Ownership, Type, and Actions

Creating a New Data Source​

Step 1: Access Creation Dialog​

  1. Click the + CREATE SOURCE button in the top-right corner of the Data Sources page
  2. The "Create Data Sources" dialog will open

Step 2: Select Data Source Type​

  1. In the "Select DS type" dropdown, choose from available options:
    • API HUB: For API-based data connections
    • Data Quality: For data quality monitoring sources
    • Database: For traditional database connections
    • Folder: For file-based data sources
    • Gen AI Models: For AI model integrations (e.g. Claude, Gemini, OpenAI, Ollama)
    • Issue Tracker: For issue trackers such as Jira Cloud
    • Knowledge Base: For document stores such as Google Drive, SharePoint, and Confluence
    • Messaging: For send-only notification targets such as Microsoft Teams and Slack
    • Payment Gateway: For payment system connections
    • Secret Store: For secure credential storage
    • Service: For web service connections

Step 3: Configure Data Source​

  1. Name: Enter a descriptive name for your data source
  2. Connection Parameters: Fill in the required configuration details based on the selected type
  3. Choose between Form or Json input methods using the toggle buttons

Step 4: Save and Test​

  1. Click Save to create the data source
  2. Use Test Connection to verify the configuration is working correctly

Importing Data Sources​

Using the Import Feature​

  1. Click the IMPORT button in the top navigation
  2. Select your import file (typically JSON or configuration file)
  3. Follow the import wizard to complete the process
  4. Verify imported data sources appear in the main list

Managing Existing Data Sources​

Viewing Data Source Details​

  1. Click the arrow (▷) next to any data source name to expand and view sub-items
  2. For hierarchical data sources like "Gen AI Models", you'll see individual items listed with their ownership and type information

Updating Data Source Configuration​

Step 1: Access Update Dialog​

  1. Click the edit icon (pencil) in the Actions column for the desired data source
  2. The "Update Data Source" dialog will open showing current configuration

Step 2: Modify Settings​

Available configuration options include:

  • project_name: Update the associated project
  • model_type: Change the model type (e.g., gpt-4o)
  • embedding_model: Modify embedding settings. Changing it doesn't change knowledge bases already enabled
  • schedule_sync: Adjust synchronization schedule
  • Select: Update selection parameters
  • API_KEY: Update authentication credentials
  • secret_name: Modify secret store references

Step 3: Apply Changes​

Use the available action buttons:

  • COPY: Duplicate the current configuration
  • DELETE: Remove the data source (use with caution)
  • SAVE: Apply and save changes
  • TEST CONNECTION: Verify updated configuration works

Data Explorer​

Accessing Data Explorer​

  1. Click on any data source name to open the Data Explorer
  2. The explorer provides multiple views of your data source information

Catalog attribute columns​

The Data Explorer table can show four columns from the entity catalog: Domain, Env, Classification and Tags. They are off by default. Click Columns at the top right of the table and turn them on.

Editing these cells works from the DataDios release after v1.13.0.

Each row is matched to its catalog entity, so a value set in the catalog shows here, and a value set here shows in the catalog. The columns work the same way on the databases, schemas and tables inside a source as on the list of sources.

To change a value, click the cell. A picker opens in place:

  • Tick a value to add it, or untick it to remove it. Domain and Env hold one value, so ticking another replaces the current one.
  • Type a new value and press Enter to add it.

The cell updates when the save returns. If the save fails, an error message appears and the picker stays open. This is the same picker the entity catalog table uses. See Managing entities for how many values each attribute holds.

A greyed dash means no catalog entity was found for that row, and the cell cannot be edited. Hover over the dash or click it to see the reason, for example "No catalog entity was found for sales_db, so its domain can't be set." An entity appears after the source has been synced and catalogued. After that, the cell can keep its greyed dash for up to five minutes. Reload the page to see the change immediately.

On a source that someone else shared with you, the owner sets these values, so the cells always show the greyed dash.

A Retry button in a cell means its value could not be loaded. Click it to load the value again.

When you open an object from Semantic Search, Data Explorer preserves the complete object path in the breadcrumb and header. The path can include the data source, database or catalog, schema, table or view, column, and rule or other child object. Long names are shortened visually but remain available from the full-path hover text.

Available Views​

Metadata Tab​

  • Attributes: View data structure and column information
  • Constraints: See data constraints and relationships
  • Indexes: Review database indexes and performance optimizations
  • Partitions: Understand data partitioning strategies

The main table displays:

  • ATTR_SNO: Attribute sequence number
  • ATTR_NAME: Column or attribute names
  • DATA_TYPE: Data types (VARCHAR, INTEGER, BOOLEAN, etc.)
  • DATA_LENGTH: Maximum length constraints
  • DATA_SCALE: Precision settings
  • NULLABLE: Whether null values are allowed
  • GENERIC_TYPE: Standardized data type classification

Data Preview Tab​

  • View actual data content from the data source
  • Browse records and understand data patterns
  • Useful for data validation and exploration

The Data Preview tab is available only for object types that expose readable records. If the connector or object does not support preview, the object remains accessible through its supported tabs.

Lineage Tab​

The object's relationships, drawn as a graph: the tables it is joined to by foreign key, the views and procedures whose SQL references it, and the queries that have actually run against it.

  • See what a table is connected to, and how.
  • Trace what a change here would affect before you make it.
  • Read a foreign key as a rule the database enforces, and a parsed reference or a captured query as something we observed — the graph distinguishes them.
Captured queries on the graph​

The queries from the Queries Tab also appear here as their own kind of node, beside the tables the object is joined to. They are labelled Query and carry no expand arrow, because unlike a table there is nothing inside one to open. Each is named by its Query ID; the Queries tab is where you read the statement itself.

A busy table is touched by far more queries than a diagram can hold, so the graph draws a page of them and reports the total in a control at the bottom of the canvas — "1–10 of 120" — with arrows to move through the set. The tables, views and procedures around the object are not paged and do not move as you turn the page. If a page cannot be loaded, or the captured queries are unavailable, the graph says so beside that control and keeps everything you have already explored on screen.

Data Quality Rules Tab​

Shows the Data Quality rules that target whatever you have selected, and how each one has been running over the last 30 days. The tab appears at every level of the tree — the data source, a database or catalog, a schema, or a single table — and narrows to what you clicked. Previously every level above a table showed the whole data source's rules.

  • Three figures at the top: Rules on the object, how many are Failing, and the object's overall Pass rate
  • One row per rule, with how many times it has run and its pass rate. On a table, each row also shows the rule's hub, dimension and severity
  • A rule that has not run in the window reads No runs in 30d rather than passing — there is no result to report, so it is left out of the pass rate as well
  • If no rule on the object has ever run, the pass rate reads — rather than 0%
  • The table shows the 100 worst rules — lowest pass rate first, with unmeasured rules last — and a line below it gives the total. If more rules exist than could be read in one request, that line says which of them the figures above cover

Rules are created and edited in the Data Quality product — this tab is read-only.

Above a table — on a data source, database or schema — the figures currently cover the whole data source, and the tab says so above the list. Hub, dimension and severity are not shown at those levels.

note

How the level is matched. Each engine qualifies a table differently, and the tab follows the engine rather than assuming one hierarchy: Trino narrows by catalog and schema, Oracle by container database, database and schema, MySQL by database only (it has no schema level), and PostgreSQL, SQL Server, Snowflake and Azure Synapse by database and schema. Data sources other than these seven do not record those details on their rules, so their tab narrows by data source and table name only.

Queries Tab​

The queries that have actually run against this object, newest first. Where the schema tells you what could depend on a table, this tells you what really touched it — which is the question you need answered before changing or retiring anything.

Each row has five columns:

  • Query — the captured SQL. Some connectors record only the query's hash, not its text; where that happens the hash is shown instead, so the row still identifies a distinct query.
  • Type — how the source classified the statement. PostgreSQL, MySQL and SQL Server report a coarse kind such as SELECT or DML; Oracle and Snowflake record every entry as QUERY.
  • Association — how the query relates to the object. Connectors record USED_OBJECTS today, meaning the query referenced the object; the column exists because that will not always be the only value, and the tab does not yet distinguish a read from a write.
  • Last Run — when the query was last seen touching this object.
  • Query ID — the identifier the source recorded the query under. Hidden by default; turn it on with the Columns button. It is the same identifier a query node carries on the Lineage tab, so it is how you match one to the other.

The Columns button chooses which of these to show, and Rows per page sets the page size (25, 50, 100 or 200).

Two things are worth knowing about this tab:

  • It only appears for data sources that capture query activity. Where a connector cannot observe queries, the tab is not offered at all rather than shown empty.
  • An empty table means nothing has been recorded yet, not that nothing uses this object. Query activity is gathered by workload extraction, so an object in a data source whose extraction has never run shows an empty table. That is the normal state of a newly added data source, not a sign that the object is unused.

The table pages through everything the object has, a page at a time, and the count beside the buttons — "1–25 of 4,820" — is the whole set rather than the page in front of you. An object with nothing recorded reads "No queries captured for this object.", and a failed load says so and offers a retry — neither is an empty grid you have to interpret.

On a phone the same rows render as cards rather than a table, with the pager beneath them.

These same queries are also drawn on the Lineage Tab, as nodes beside the tables the object is joined to.

Data Quality Score Tab​

  • View overall data quality metrics
  • Track data quality trends over time
  • Identify areas for improvement

AI Sources for the Whole Tenant​

A tenant administrator can make one AI model data source available to every user in the tenant, so users don't each need their own model source and API key. They do it by binding the source to AI · reasoning (chat, SQL AI, insights, data quality rules, semantic search) or AI · embeddings (knowledge base RAG) in Manage → Integrations. That screen is the only place an AI source is set for the tenant.

tip

If an AI feature tells you it "needs an AI source", ask your tenant administrator to bind AI · reasoning in Manage → Integrations.

The DataDios Data Source​

A data source of type DataDios signs in to a DataDios account, usually your own, and reads its data sources, workflows and schedulers. You give it the address of the DataDios instance (by default https://prod.datadios.com), an email and a password. When someone signs up a new organization, DataDios creates one for that person, named datadios_self, which reads their own account, including its API usage. Users added to the organization later do not get one.

From the DataDios release after v1.13.0, a DataDios data source cannot be shared, and neither can a workload chart that reads one. See A DataDios data source cannot be shared.

When the account has two-step authentication on​

The totp_secret field is available from the DataDios release after v1.13.0. Earlier versions do not show it.

If the account the data source signs in with uses two-step authentication, an email and a password are not enough. The data source also needs the account's authenticator setup key, entered in the totp_secret field. With it, the data source makes the 6-digit code itself when it signs in, so scheduled extractions keep running without anyone typing a code. Where the deployment has caching turned on (the IS_CACHE_ENABLE setting, together with a Redis host in REDIS_CACHE_HOST; ask your administrator whether yours does), it reuses that sign-in until it expires (two days by default), so it needs a code only about once every two days. Without caching it makes a new code on every run. The key is stored encrypted, like the password.

The setup key is the text behind Copy setup key on the screen that shows the QR code when you set up two-step authentication. It is shown only at that moment. DataDios cannot show it again later. If you did not save it, an administrator has to reset your two-step authentication. Then set it up again and save the key before you finish.

What happens without a working key:

  • No key entered: Test Connection fails with a message asking for the setup key. Scheduled extractions of the data source are stopped. Add the key, then resume them.
  • The account has not set up two-step authentication yet, but must: Test Connection fails. Scheduled extractions of the data source are stopped. Sign in to DataDios once and set it up, add the key, then resume them.
  • Wrong key: each attempt fails, and each wrong code counts toward the account's two-step lockout. After 6 wrong codes the second factor locks for 30 minutes, and during the lock even a correct code is refused, so you cannot sign in to DataDios yourself either (see If your code is not accepted). Once the account is locked, the data source's scheduled extractions are stopped, so a wrong key cannot keep locking you out. Correct the key, then resume them. A key with a character a setup key cannot contain (0, 1, 8, 9 or punctuation) stops them on the first run. Spaces in the key are fine.
  • Two sign-ins in the same 30 seconds: a code can be used only once. When the data source does need a new code and you sign in with your phone in the same 30 seconds, one of the two is rejected: enter the next code, or the data source tries again on its next run. A failed run like this does not stop the schedules. Without caching, when the data source signs in twice within 30 seconds (Test Connection, then Create), the second usually waits up to 30 seconds for a new code. If one is still rejected, try again.

A wrong key and two sign-ins in the same 30 seconds show the same message: "DataDios 2-step verification failed (401): the setup key is wrong, or another sign-in for this user used the same 30-second code." To tell them apart, look at the next run. If it succeeds, two sign-ins used the same code. If the message comes back on every run, the key is wrong: correct it before the lockout.

datadios_self does not need the key. It reads the same DataDios instance it runs on without signing in, so it keeps working when you turn on two-step authentication.

tip

On an on-prem deployment, you can avoid storing a second factor in a data source by signing in with a dedicated account that the deployment exempts from two-step authentication (the MFA_EXEMPT_EMAILS setting, which DataDios Cloud customers cannot change). That account skips the second factor at every sign-in, not only for this data source, so give it a long password and only the rights the data source needs. See Two things to be aware of.

Best Practices​

Data Source Configuration​

  1. Use Descriptive Names: Choose clear, meaningful names for easy identification
  2. Test Connections: Always test connections after configuration changes
  3. Secure Credentials: Use the Secret Store for sensitive information like API keys
  4. Regular Updates: Keep connection parameters current as systems change