Skip to main content

DataDios Data Source

Overview​

Data Sources in DataDios are configured connections to external databases, APIs, files, and other data systems. They serve as the foundation for all data operations within the platform, enabling you to connect to, explore, and manage various types of data repositories.

Accessing Data Sources​

  1. Navigate to the Data Source section from the left navigation menu
  2. Click on Sources to view all available data sources
  3. The main interface displays a table with columns for Name, Ownership, Type, and Actions

Creating a New Data Source​

Step 1: Access Creation Dialog​

  1. Click the + CREATE SOURCE button in the top-right corner of the Data Sources page
  2. The "Create Data Sources" dialog will open

Step 2: Select Data Source Type​

  1. In the "Select DS type" dropdown, choose from available options:
    • API HUB: For API-based data connections
    • Data Quality: For data quality monitoring sources
    • Database: For traditional database connections
    • Folder: For file-based data sources
    • Gen AI Models: For AI model integrations
    • Payment Gateway: For payment system connections
    • Secret Store: For secure credential storage
    • Service: For web service connections

Step 3: Configure Data Source​

  1. Name: Enter a descriptive name for your data source
  2. Connection Parameters: Fill in the required configuration details based on the selected type
  3. Choose between Form or Json input methods using the toggle buttons

Step 4: Save and Test​

  1. Click Save to create the data source
  2. Use Test Connection to verify the configuration is working correctly

Importing Data Sources​

Using the Import Feature​

  1. Click the IMPORT button in the top navigation
  2. Select your import file (typically JSON or configuration file)
  3. Follow the import wizard to complete the process
  4. Verify imported data sources appear in the main list

Managing Existing Data Sources​

Viewing Data Source Details​

  1. Click the arrow (▷) next to any data source name to expand and view sub-items
  2. For hierarchical data sources like "Gen AI Models", you'll see individual items listed with their ownership and type information

Updating Data Source Configuration​

Step 1: Access Update Dialog​

  1. Click the edit icon (pencil) in the Actions column for the desired data source
  2. The "Update Data Source" dialog will open showing current configuration

Step 2: Modify Settings​

Available configuration options include:

  • project_name: Update the associated project
  • model_type: Change the model type (e.g., gpt-4o)
  • embedding_model: Modify embedding settings
  • schedule_sync: Adjust synchronization schedule
  • Select: Update selection parameters
  • API_KEY: Update authentication credentials
  • secret_name: Modify secret store references

Step 3: Apply Changes​

Use the available action buttons:

  • COPY: Duplicate the current configuration
  • DELETE: Remove the data source (use with caution)
  • SAVE: Apply and save changes
  • TEST CONNECTION: Verify updated configuration works

Data Explorer​

Accessing Data Explorer​

  1. Click on any data source name to open the Data Explorer
  2. The explorer provides multiple views of your data source information

Available Views​

Object Meta Data Tab​

  • Attributes: View data structure and column information
  • Constraints: See data constraints and relationships
  • Indexes: Review database indexes and performance optimizations
  • Partitions: Understand data partitioning strategies

The main table displays:

  • ATTR_SNO: Attribute sequence number
  • ATTR_NAME: Column or attribute names
  • DATA_TYPE: Data types (VARCHAR, INTEGER, BOOLEAN, etc.)
  • DATA_LENGTH: Maximum length constraints
  • DATA_SCALE: Precision settings
  • NULLABLE: Whether null values are allowed
  • GENERIC_TYPE: Standardized data type classification

Object Data Tab​

  • View actual data content from the data source
  • Browse records and understand data patterns
  • Useful for data validation and exploration

Graph Tab​

  • Visual representation of data relationships
  • Understand data lineage and dependencies
  • Explore connections between different data elements

Data Quality Rules Tab​

  • Review configured data quality rules
  • Monitor data quality standards
  • Set up validation criteria

Data Quality Score Tab​

  • View overall data quality metrics
  • Track data quality trends over time
  • Identify areas for improvement

Best Practices​

Data Source Configuration​

  1. Use Descriptive Names: Choose clear, meaningful names for easy identification
  2. Test Connections: Always test connections after configuration changes
  3. Secure Credentials: Use the Secret Store for sensitive information like API keys
  4. Regular Updates: Keep connection parameters current as systems change