Skip to main content

Datadios On-Premises Installation Guide

Deploy Datadios in your own infrastructure for complete data sovereignty and customization control.

System Requirements​

Hardware Requirements (Minimum)​

ComponentSpecification
ProcessorUp to 3.1 GHz Intel Xeon Platinum
Memory (RAM)16GB
Storage Volume128GB

Supported Operating Systems​

  • Ubuntu: 24.04 LTS or newer (recommended - primary support platform)
  • RHEL: 8+ (supported)
  • Windows: 10+ Pro/Home Edition (supported)

Software Prerequisites​

For Ubuntu​

  • Latest version of Docker (Ensure Docker is usable as a non-root user)
  • Docker Compose

For Windows​

  • Docker Desktop software

For RHEL​

  • Podman
  • Podman Compose

Installation Commands​

Ubuntu/Debian Systems​

# Update package index
sudo apt update

# Install Docker
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh

# Install Docker Compose
sudo apt install docker-compose-plugin

# Add user to docker group
sudo usermod -aG docker $USER
newgrp docker

RHEL/CentOS Systems​

# Install Podman and Podman Compose
sudo dnf install podman podman-compose

# Enable Podman socket
systemctl --user enable podman.socket
systemctl --user start podman.socket

Windows Systems​

  1. Install Docker Desktop for Windows
  2. Enable WSL 2 backend
  3. Restart system after installation

Pre-Installation Setup​

Access for datadios​

Datadios need access to all the data sources you will be connecting through datadios.

  1. If datadios is used for migration validation then it need access to the source and destination databases used for validation (eg. snowflake, oracle, sqlserver etc..)
  2. For ports see the docker compose file ( datadios can be setup for http or https )

Docker Hub Access​

The necessary Docker images can be pulled from ​Datadios Docker Hub​. The Datadios team will provide login credentials to access the required images.

Required Docker Images​

  • npod/on-prem (Main Datadios container)
  • dd10user/pg_docker (Optional, required for running PostgreSQL locally)

Docker Hub Login​

# Login with provided credentials
docker login -u docker-hub-username -p docker-hub-password

PostgreSQL Database Configuration​

Datadios uses PostgreSQL as its backend database to store configuration and user data. The database can be set up either on the cloud or locally.

Cloud-based PostgreSQL Server​

The PostgreSQL server should be set up with the pg_partman extension.

Local PostgreSQL Server​

The Datadios team provides a pre-configured Docker image.

Configuring Datadios​

Create a YAML file named docker-compose.yaml and paste the following content. Modify the configurations as needed.

Configuring with a Cloud-based PostgreSQL Server​

services:
  datadios:
    image: npod/on-prem
    restart: always
    container_name: dd
    ports:
      - "443:443" # Adjust as needed (if 443 then you need to provide the certificates, if certificates not provided use 80:80)
    deploy:
      resources:
        limits:
          cpus: "0.8" # Set to 80% of available system CPU cores (if you have 4 CPU core then it should be 3.2)
          memory: 6G # Set to 75% of available system memory
    environment:
      - S3_BUCKET=docker_local
      - DD_ROOT=/opt/datadios
      - DB_TYPE=postgresql
      - db_url=url-of-your-cloud-postgresql-server
      - db_username=username-of-your-cloud-postgresql-server
      - db_password=password-of-your-cloud-postgresql-server
      - SERVER_TYPE=datadios_on_prem
      - GUNICORN_TYPE=multiple_worker
      - PORT=443 # Ensure this matches the host mapping
      - KEY_FILE_NAME=your_server.key # note this change (you need to provide in case of https)
      - CRT_FILE_NAME=your_server.crt # Note this change (you need to provide in case of https)
    volumes:
      - /home/dell/work/ssl_test/server_certs:/opt/datadios/https_data:Z # note this change (location should be your certificates location, change it accordingly)

Configuring with Local PostgreSQL Server​

services:
  datadios:
    image: npod/on-prem
    restart: always
    container_name: dd
    ports:
      - "443:443" # Adjust as needed (if 443 then you need to provide the certificates, if certificates not provided use 80:80)
    deploy:
      resources:
        limits:
          cpus: "0.8" # Set to 80% of available system CPU cores (if you have 4 CPU core then it should be 3.2)
          memory: 6G # Set to 75% of available system memory
    environment:
      - S3_BUCKET=docker_local
      - DD_ROOT=/opt/datadios
      - DB_TYPE=postgresql
      - db_url=postgresql-user # datadios team will provide the credentials for postgresql
      - db_username=username-postgresql # datadios team will provide the credentials for postgresql
      - db_password=password-postgresql # datadios team will provide the credentials for postgresql
      - SERVER_TYPE=datadios_on_prem
      - GUNICORN_TYPE=multiple_worker
      - PORT=443 # Ensure this matches the host mapping
      - KEY_FILE_NAME=your_server.key # note this change (you need to provide in case of https)
      - CRT_FILE_NAME=your_server.crt # Note this change (you need to provide in case of https)
    volumes:
      - /home/dell/work/ssl_test/server_certs:/opt/datadios/https_data:Z # note this change (location should be your certificates location, change it accordingly)
    networks:
      - dd_network  
      
  postgres:
    restart: always
    image: dd10user/pg_docker
    container_name: pg_docker
    shm_size: '256m'
    expose:
      - "5432"
    networks:
      - dd_network    

networks:
      dd_network:
        driver: bridge
PostgreSQL credentials can be configured as well. :::

Running Datadios​

Run the following command to start the Datadios service:

docker compose up -d

This will pull the necessary images from Docker Hub and start the required containers.

Verifying the Installation​

Once the containers are running, verify the installation:

  1. Open a web browser
  2. Visit http://localhost
  3. If a different port was configured, access http://localhost:<port_number>
  4. If https enabled, accesss https://<your-server-address/dns>

Monitoring Resource Usage​

To check CPU, memory, and disk usage for Docker:

docker stats

Updating Datadios​

To update Datadios periodically, run the following commands:

docker compose down
Ensure this command is executed in the directory containing docker-compose.yaml. :::
docker compose up -d

Backing Up PostgreSQL Database​

Details for database backup procedures are yet to be added. :::

Frequently Asked Questions (FAQs)​

Q: What should I do if docker compose up -d fails?​

A: Check if Docker is installed and running. Run docker --version and docker compose version to verify. If using Ubuntu, ensure your user is in the docker group:

sudo usermod -aG docker $USER

Then restart your session.

Q: How do I change the default port for Datadios?​

A: To change the default port, follow these steps:

  1. Open the docker-compose.yaml file

  2. Modify the PORT environment variable under dd_mul_worker, e.g., change:

    - PORT=80

    to:

    - PORT=8080

  3. Update the port mapping under ports, changing:

    ports:    - "80:80"

    to:

    ports:    - "8080:80"

  4. Restart the container with:

    docker compose down && docker compose up -d

Q: What should I do if the Docker login fails?​

A: If you encounter a Docker login failure, try the following steps:

  1. ​Check your credentials​: Ensure you are using the correct username and password provided by the Datadios team
  2. ​Retry login​: Run the command again: 
    docker login -u docker-hub-username -p docker-hub-password