Skip to main content

DataDios Metadata Synchronization

Overview​

DataDios Metadata Synchronization is a comprehensive metadata management feature within the DataDios Cloud Accelerator platform. It enables seamless migration and synchronization of metadata between any data sources, including databases and data governance systems. The feature supports all types of metadata and governance processes, working efficiently across both on-premises and cloud platforms.

Key Capabilities​

Smart Delta Processing​

The intelligent delta capability reduces redundancy by identifying and migrating only the changes, rather than processing entire datasets. This approach significantly improves efficiency and reduces processing time.

Universal Compatibility​

  • Supports all types of metadata and governance systems
  • Works seamlessly on on-premises and cloud platforms
  • Compatible with databases and data governance systems
  • Handles complex metadata relationships and dependencies

Importance in Data Migration Projects​

Critical Component for System Synchronization​

Metadata Sync serves as a crucial component in data source migration projects by ensuring target data source systems remain synchronized with source systems.

Change Tracking Without Migration​

The feature allows tracking of metadata changes in a data source without requiring migration to another system. This is particularly valuable when you need to:

  • Maintain records of metadata changes
  • Monitor system evolution over time
  • Ensure compliance and audit requirements
  • Track data lineage and dependencies

Accuracy and Currency​

By continuously tracking metadata changes, organizations can ensure their target data sources remain up-to-date and accurate, supporting better decision-making and system reliability.

Core Features​

Efficient Metadata Migration​

  • Smart Processing: Migrate metadata and synchronize differences without reimporting entire datasets
  • Delta Identification: Automatically identify and process only changed metadata
  • Time and Effort Savings: Streamlined approach reduces manual work and processing time

Comprehensive Synchronization Logging​

  • Detailed Tracking: Maintains logs of each synchronization operation
  • Timestamped Records: Easy tracking and review of updates by date and time
  • Audit Trail: Ensures accountability and supports compliance requirements
  • Historical Analysis: Review past synchronizations for troubleshooting and optimization

Data Lineage Visualization​

  • Complete Overview: Full visibility into data flow throughout the data source
  • Dependency Mapping: Clear understanding of relationships between data items
  • Source and Structure Insights: Better comprehension of data origins and organization
  • Migration Support: Valuable information for data migration and management projects

Smart Delta Feature​

Redundancy Reduction​

The Smart Delta feature minimizes redundancy in metadata migration by intelligently identifying and processing only the changes rather than entire datasets.

Change-Only Migration​

  • Selective Processing: Move only modified metadata to target systems
  • Resource Optimization: Save processing time and system resources
  • Accuracy Maintenance: Ensure target systems remain current and accurate
  • Efficiency Maximization: Optimize migration workflows for better performance

Data Lineage Graph​

Built-in Visualization​

DataDios Metadata Sync includes an integrated lineage graph feature providing visual representation of overall data flow patterns.

Graph Components​

  • Nodes: Each node represents a corresponding item or object in the data source
  • Edges: Each edge illustrates relationships and connections between nodes
  • Visual Mapping: Clear representation of data flow and dependencies

Enhanced Understanding​

The lineage graph provides deeper insights into data source and structure, proving invaluable for:

  • Data migration planning and execution
  • Data management optimization
  • Compliance and governance initiatives
  • System architecture documentation

Comprehensive Logging System​

Metadata Tracking​

  • Complete Capture: All metadata, changes, and lineage details with precise timestamps
  • Easy Review: Simple tracking and review of all data source updates
  • Historical Access: Ability to examine metadata states at specific points in time

Time-Stamped Documentation​

Timestamps enable detailed analysis of:

  • Source metadata evolution over time
  • Previously imported data states
  • Change patterns and trends
  • Troubleshooting and debugging support

Workflow Process​

The Metadata Synchronization process follows a structured 6-step workflow:

Step 1: Create a Workflow​

  • Source and Target Selection: Choose source and target data sources
  • Configuration: Provide additional information as needed for migration
  • Optional Target: Target data source is optional for metadata tracking only scenarios
  • Flexibility: Steps 4 and 5 can be skipped when only source tracking is required

Step 2: Export Metadata​

  • Metadata Export: Extract source metadata and relationship data
  • Progress Tracking: Monitor export process completion
  • Multiple Views: Access exported data through three different perspectives:
    • Summary: Graphical overview of exported data
    • List Items: Detailed item listings with comprehensive information
    • Data Lineage: Clear visualization of data relationships and hierarchies

Step 3: Find Delta Changes​

  • Critical Analysis: Essential step for identifying metadata differences
  • Smart Delta Engine: Intelligent difference detection for current vs. previous exports
  • Visual Representation: Color-coded differences with drill-down capabilities
  • Column-Level Detail: Granular analysis down to individual column changes

Step 4: Convert Target Data​

  • Format Conversion: Transform source changes into target-compatible formats
  • Selective Processing: Convert only identified changes, not entire datasets
  • Visual Feedback: Color-coded display showing conversion differences
  • Efficiency Focus: Optimize processing by handling only necessary changes

Step 5: Import Metadata​

  • Target Integration: Import converted metadata into target database
  • Additional Configuration: Provide supplementary information as needed
  • Import Verification: Review list of successfully imported items
  • Quality Assurance: Final validation of import process completion

Step 6: Synchronize Workflows (Optional)​

  • Ongoing Synchronization: Create sync workflows for continuous updates
  • Change-Only Processing: Move only new changes to maintain synchronization
  • Automated Workflow: Execute steps 2-6 automatically for delta processing
  • System Alignment: Ensure both source and target systems remain synchronized

How to Use Metadata Sync​

Creating a New Workflow​

Initial Setup​

  1. Navigate to 'Metadata Sync' section from the left side panel
  2. Click 'CREATE WORKFLOW' button to start new workflow
  3. Select source and target data sources (target is optional)
  4. Choose from available data sources (e.g., Azure SQL Server as source)
  5. Click 'NEXT' button to proceed to the next step

Workflow Execution Steps​

Step 1: Sync Metadata View​

  1. Review side-by-side 'Sync Metadata' view
  2. Note: Previous sync data will be empty for first-time workflows
  3. Navigate through sections using column indicators:
    • Summary of Data: Overview and statistics
    • List of Items: Detailed item inventory
    • Lineage Graphs: Visual relationship mapping
  4. Click 'NEXT' to proceed to Delta Finding

Step 2: Delta Finding​

  1. Review delta changes between current and previous sync
  2. If no target data source: This becomes the final workflow step
  3. With target data source: Click 'NEXT' for Delta to Target Conversion

Step 3: Delta to Target Conversion (If Target Selected)​

  1. Review conversion of delta changes to target format
  2. Click 'NEXT' to proceed to final step

Step 4: Target Import (If Target Selected)​

  1. Complete the import process to target system
  2. Review final import results and confirmation

Creating Sync Workflows​

Ongoing Synchronization Setup​

  1. Navigate to 'Metadata Sync' > 'Workflows'
  2. Locate desired workflow in the list
  3. Click on 'Sync Workflows' column for the workflow
  4. Click 'CREATE SYNC WORKFLOW' in the top right

Sync Workflow Execution​

  1. Sync Meta Data Step: Review new sync workflow metadata step
  2. Navigation: Use 'NEXT' button to progress through steps
  3. Delta Finding: Review only changes since last sync
    • Note: No data displays if no changes occurred since last sync
  4. Conversion and Import: Process delta data to target system
  5. Completion: Verify successful synchronization of changes

Best Practices​

Planning Phase​

  • Requirement Analysis: Clearly define metadata synchronization requirements
  • Source Assessment: Evaluate source system metadata complexity and volume
  • Target Preparation: Ensure target systems are ready to receive metadata
  • Timeline Planning: Account for initial sync and ongoing synchronization needs

Implementation Phase​

  • Test Workflows: Start with smaller datasets to validate processes
  • Monitor Progress: Use built-in logging and reporting for progress tracking
  • Validate Results: Verify synchronization accuracy through lineage graphs
  • Document Processes: Maintain records of workflow configurations and results

Ongoing Operations​

  • Regular Sync Workflows: Establish routine synchronization schedules
  • Monitor Delta Changes: Review delta findings to understand system evolution
  • Performance Optimization: Use Smart Delta features to minimize processing overhead
  • Audit and Compliance: Leverage comprehensive logging for audit requirements

Troubleshooting​

Common Issues and Solutions​

Connection Problems​

  • Verify data source connectivity and credentials
  • Check network access between systems
  • Validate data source configurations

Sync Failures​

  • Review error logs for specific failure details
  • Verify source and target system availability
  • Check for metadata schema changes

Performance Issues​

  • Monitor system resources during large synchronizations
  • Consider breaking large workflows into smaller batches
  • Optimize delta processing for better performance

Data Inconsistencies​

  • Use lineage graphs to identify relationship issues
  • Review delta findings for unexpected changes
  • Validate source system metadata integrity

Getting Support​

For technical assistance and advanced troubleshooting, contact the DataDios support team with:

  • Detailed error descriptions
  • Workflow configuration information
  • System environment details
  • Log files and error messages