Remote Folder (SFTP)

This section explains how to create, configure, and test a Remote Folder data source in DataDios. Remote Folder allows you to connect to file systems on remote servers via SSH/SFTP protocol.

Overview

The Remote Folder data source enables secure access to files and directories on remote Linux/Unix servers. It supports:

Multiple authentication methods: Password or SSH private keys (RSA, ECDSA, Ed25519, DSA)
Various file formats: CSV, JSON, Parquet, ORC, Avro, XML, TSV, photos, video, archives, and more
Hierarchical browsing: Navigate through folder structures
Metadata extraction: View file attributes and data schemas

Steps to Create and Test a Remote Folder Data Source

Step 1: Create a Data Source

Navigate to the Data Sources tab in DataDios
Click + CREATE DS
From the list of available data source types, select Remote Folder

Step 2: Fill Connection Details

In the Connection Details form, provide the required parameters:

Required Parameters

Parameter	Description	Example
Host	Remote server hostname or IP address	`192.168.1.100` or `myserver.example.com`
Username	SSH username for authentication	`ubuntu`, `ec2-user`
Folder Path	Absolute path to the folder on remote server	`/home/user/data` or `/var/data/files`

Authentication (Choose One)

Option A: Password Authentication

Parameter	Description
Password	SSH password for the user

Option B: SSH Private Key Authentication

Parameter	Description
PEM Data	SSH private key content (paste the entire key including BEGIN/END headers)
Passphrase	(Optional) Passphrase if the private key is encrypted

Supported Key Types

DataDios supports multiple SSH key algorithms:

RSA (most common)
ECDSA (elliptic curve)
Ed25519 (modern, recommended)
DSA (legacy)

Optional Parameters

Parameter	Description	Example
Group	Grouping for organizing data sources	`Production`, `Development`
Object Types	Filter files by type (comma-separated)	`CSV,JSON,PARQUET` or `*` for all
Secret Name	Reference to secret stores (AWS Secrets Manager, Azure Key Vault)	`my-sftp-credentials`

Step 3: Test Connection

After entering details, click Test Connection
Ensure the connection is validated successfully
If using SSH keys, verify the key format is correct (PEM format with proper line breaks)

Trusting the server's host key (Remote SSH)

A new Remote SSH connection only connects to a server whose host key is pinned on the connection, so DataDios can tell if the server it reaches is ever swapped for another one. The first time you connect:

Click Test Connection. It fails with "Server '…' not found in known_hosts" and shows the key the server presented, for example SHA256:3hQpYkqv….
Check that key against the server's real key. On the server, ssh-keygen -lf /etc/ssh/ssh_host_ed25519_key.pub prints it.
Click Trust this key. A note confirms the key is trusted and will be pinned on save. Trusting a key is not a passing test.
Click Test Connection again. It should now connect.
Click Save (or Create) to pin the key.

Every later connection compares the server's key with the pinned one and refuses to connect on a mismatch.

If the server's key changes, Test Connection says it does not match the pinned key and offers Replace pinned key. Replace it only if you know the server's key was changed, for example after a rebuild; an unexpected change can mean you are not reaching the server you think.

To skip the check, turn on Accept unknown host keys: DataDios then trusts whatever key the server presents. Connections saved before host-key checking was added keep behaving that way until you edit them.

Troubleshooting Connection Issues

Authentication failed: Verify username and password/key are correct
Host key verification: "not found in known_hosts" means the server's key is not pinned yet, and "does not match the pinned key" means it changed. See Trusting the server's host key
Permission denied: Ensure the user has read access to the specified folder path
Key format error: Make sure the private key includes -----BEGIN ... KEY----- and -----END ... KEY----- headers

Step 4: Save Data Source

If the test succeeds, click Create to save the data source
You will be redirected to the Datasource Listing Page, where the Remote Folder data source will appear

Step 5: Explore Data Source Items

Expand the Remote Folder data source to view all items (folders and files)
The file browser displays:
- Folders: Click to expand and view contents
- Files: Shows file name, type, and last modified date
To view metadata about any item:
- Click the item name
- Click the three stacked lines icon to open the Object Metadata pop-up
You can also explore additional features in the Metadata Explorer:
- Object Data
  - View the actual data present in the selected file (e.g., CSV rows, JSON content)
- Attributes
  - View column names and inferred data types for structured files

Supported File Formats

Format	Extension	Features
CSV	`.csv`	Comma-separated values, auto-detect schema
TSV	`.tsv`	Tab-separated values
JSON	`.json`	JSON documents
Parquet	`.parquet`	Columnar storage format
ORC	`.orc`	Optimized Row Columnar format
Avro	`.avro`	Apache Avro data serialization
AVSC	`.avsc`	Avro schema files
XML	`.xml`	XML documents
Text	`.txt`, `.text`	Plain text files
Photos	`.png`, `.jpg`, `.jpeg`, `.gif`, `.webp`	Previews inline in Object Data, up to 20 MB per file
Video	`.mp4`, `.mov`	Stored only — no preview
Archives	`.zip`, `.tar`, `.gz`, `.7z`	Stored only — no preview

A photo larger than the 20 MB preview limit still uploads and stays stored — DataDios just can't render an automatic preview for it. There is currently no route to retrieve a stored file's original bytes back out of DataDios (video and archives included), so "stored" here means the platform keeps the file, not that it offers a download link.

Unviewable binary formats (legacy image formats like .bmp/.tiff/.svg, audio/video formats other than mp4/mov, and executable formats) are refused. Creating or uploading objects only works when the Folder datasource's Connection Mode is set to Local Folder; it is not available for Remote SSH (SFTP) connections. In Local Folder mode, each user can store up to 100 MB in total.

Connection Configuration Examples

Example 1: Password Authentication

{
  "host": "192.168.1.100",
  "username": "datauser",
  "password": "SecurePassword123",
  "folder_path": "/home/datauser/datasets",
  "object_types": "CSV,PARQUET,JSON"
}

Example 2: SSH Private Key (RSA)

{
  "host": "myserver.example.com",
  "username": "ubuntu",
  "pem_data": "-----BEGIN RSA PRIVATE KEY-----\nMIIEpAIBAAKCAQEA...\n-----END RSA PRIVATE KEY-----",
  "folder_path": "/var/data/files",
  "object_types": "*"
}

Example 3: SSH Private Key with Passphrase (Ed25519)

{
  "host": "secure-server.example.com",
  "username": "admin",
  "pem_data": "-----BEGIN OPENSSH PRIVATE KEY-----\nb3BlbnNzaC1rZXktdjEAAAA...\n-----END OPENSSH PRIVATE KEY-----",
  "passphrase": "my-key-passphrase",
  "folder_path": "/data/analytics"
}

Best Practices

Use SSH Keys instead of passwords for enhanced security
Use Ed25519 Keys for modern, secure, and fast authentication
Store Credentials in Secret Stores (AWS Secrets Manager, Azure Key Vault) to avoid hardcoding
Always Test Connection before saving to ensure configuration is correct
Use Specific Object Types to filter only relevant file types and improve performance
Organize with Groups for easier management of multiple Remote Folder data sources
Use Absolute Paths for folder_path to avoid ambiguity

Security Considerations

All connections are made over SSH (port 22 by default), ensuring encrypted data transfer
Private keys are stored securely and never exposed in logs
Use encrypted private keys with passphrases for additional security
Consider using bastion hosts or jump servers for accessing servers in private networks

Generating SSH Keys

Linux / macOS

Generate Ed25519 Key (Recommended)

ssh-keygen -t ed25519 -C "your-email@example.com"

Generate RSA Key

ssh-keygen -t rsa -b 4096 -C "your-email@example.com"

Copy Public Key to Server

ssh-copy-id username@remote-server

View Private Key (for pasting into DataDios)

cat ~/.ssh/id_ed25519

Windows

Option 1: Using PowerShell (Windows 10/11)

# Generate Ed25519 key
ssh-keygen -t ed25519 -C "your-email@example.com"

# Generate RSA key
ssh-keygen -t rsa -b 4096 -C "your-email@example.com"

Keys are saved to: C:\Users\YourUsername\.ssh\

Option 2: Using Git Bash

ssh-keygen -t ed25519 -C "your-email@example.com"

After Generating Keys

Copy public key to server - Add contents of id_ed25519.pub (or id_rsa.pub) to server's ~/.ssh/authorized_keys
Use private key in DataDios - Copy contents of id_ed25519 (or id_rsa) and paste into PEM Data field

Example: View and copy private key

# Linux/macOS/Git Bash
cat ~/.ssh/id_ed25519

# Windows PowerShell
Get-Content $env:USERPROFILE\.ssh\id_ed25519

tip

When prompted during key generation:

File location: Press Enter to use default
Passphrase: Optional but recommended for extra security

Troubleshooting

Common Issues

Issue	Possible Cause	Solution
Connection timeout	Server unreachable or firewall blocking	Verify network connectivity and firewall rules
Authentication failed	Wrong credentials	Double-check username and password/key
Permission denied	User lacks read access	Ensure user has permissions on the folder
Key format error	Invalid PEM format	Verify key has proper headers and line breaks
File not found	Path doesn't exist	Verify folder_path exists on the remote server

Key Format Tips

When pasting SSH private keys, ensure:

The key includes -----BEGIN ... KEY----- header
The key includes -----END ... KEY----- footer
Line breaks are preserved (if pasting from web form, the system will auto-fix)

Overview​

Steps to Create and Test a Remote Folder Data Source​

Step 1: Create a Data Source​

Step 2: Fill Connection Details​

Required Parameters​

Authentication (Choose One)​

Optional Parameters​

Step 3: Test Connection​

Trusting the server's host key (Remote SSH)​

Step 4: Save Data Source​

Step 5: Explore Data Source Items​

Supported File Formats​

Connection Configuration Examples​

Example 1: Password Authentication​

Example 2: SSH Private Key (RSA)​

Example 3: SSH Private Key with Passphrase (Ed25519)​

Best Practices​

Security Considerations​

Generating SSH Keys​

Linux / macOS​

Windows​

After Generating Keys​

Troubleshooting​

Common Issues​

Key Format Tips​