Skip to main content

Load Data & Create Object

DataDios lets you push data into a datasource directly from the product — either by loading rows into an object (table, blob, file) that already exists, by creating a brand-new object, or by doing both in a single step. This page covers the three related actions, the write modes available when loading, upload limits, and which datasources support each capability.

What you can do​

ActionWhat it does
Create ObjectCreates a new object (for example, a SQL table or an object-store blob/key) in the selected datasource, from a schema you define or infer from a file.
Load DataLoads rows from an uploaded file (or a file fetched from a URL) into an object that already exists.
Create and LoadCreates a new object and loads the uploaded file into it in one step.
Infer SchemaReads an uploaded file's columns/types without creating or loading anything — used to prefill the column list when you're building a new object from a file.

Before loading into an existing object, the Load Plan step lets you preview how the file's columns map onto the target object's columns, and flags any required column that isn't mapped or any value DataDios can't safely convert to the target's type, before you commit to the load.

Sync vs. async loads​

A load (via Load Data or Create and Load) runs synchronously if the file is small enough — the request completes and DataDios returns the outcome (rows inserted/updated, or a failure) directly in the response, with HTTP status 200.

Once the uploaded file is larger than the sync threshold (see Upload limits below), DataDios instead accepts the request, hands the load off to a background job, and responds immediately with HTTP status 202 and a request ID. Your client is expected to poll DataDios' async status-check endpoint with that request ID until the job reaches a terminal state.

Status values you'll see while polling:

StatusMeaning
PROCESSINGThe load is still running. The response also carries a completion percentage and a rows-processed count you can show as progress.
SUCCESSThe load finished and all rows were written.
FAILThe load did not complete — no rows (or no further rows) were written.
PARTIAL_SUCCESSSee Partial success below — don't treat this as a full success.
note

If you are polling from the DataDios UI, you may also see a Stopped Watching state on the screen after about 10 minutes. This is a UI-side timeout that stops the browser from continuing to poll — it does not mean the backend job failed. The load may still be running or may have already finished; refresh or re-check the object to see its current state.

Write modes​

When loading into an existing object, you choose how the incoming rows are applied:

  • Append — every row in the file is added to the object as a new row. Append has no concept of "already loaded this file" — re-uploading the same file with append adds a second copy of every row rather than detecting or skipping the duplicates. If you need to reload a file without doubling its rows, use truncate (replace everything) or upsert (update matching rows, insert the rest) instead.

  • Truncate — the object's existing contents are cleared first, then the file's rows are written in.

  • Upsert — you choose one or more key columns. For each incoming row, DataDios looks for an existing row whose key columns match:

    • If a match is found and at least one non-key column actually differs from the incoming row, the existing row is updated to the new values.
    • If a match is found and every value already agrees with the incoming row, the row is left alone (counted as skipped, not updated).
    • If no match is found, the row is inserted as new.

    This means upsert is a genuine update-or-insert: a row whose key already exists is overwritten with the new values, not skipped and not duplicated. The response reports rows_inserted, rows_updated (rows that changed), and rows_skipped (rows that matched and needed no change) separately, so you can tell exactly what happened.

    Upsert requires the source file to have a unique value per row for the chosen key columns — if the file contains the same key twice, the load is rejected before anything is written to the target object (which row would "win" is otherwise undefined).

Example

Loading a file of customer records with upsert and key column customer_id: a row for a customer_id that's new to the table is inserted; a row for a customer_id already in the table, where an address changed, updates just that row; a row that's identical to what's already stored is left as-is.

Not every datasource supports every write mode — see the capability table below. Object-store datasources in particular do not support upsert (see Object-store limitations).

Upload limits​

  • Files up to 5 MB load synchronously — the response comes back on the same request with the outcome (HTTP 200).
  • Files larger than that, up to a hard cap of 100 MB, are accepted and processed asynchronously (HTTP 202 + a request ID to poll, see Sync vs. async loads).
  • Uploads over 100 MB are rejected outright (HTTP 413).

These thresholds are the same regardless of which datasource you're loading into or from (upload, or a source URL DataDios fetches on your behalf) — DataDios does not currently vary the sync/async cutoff or the hard size cap per connector.

Partial success​

Occasionally a create-and-load's data load fails after the object was already created, and the automatic rollback that removes that object also fails — so the object is still there, empty or partly filled, rather than cleaned up. DataDios reports this outcome as PARTIAL_SUCCESS, distinct from both SUCCESS and FAIL, and the HTTP response uses status 207 (Multi-Status) rather than 200. (When the rollback succeeds, the outcome is a plain FAIL instead — the object is gone, not partially there.)

If you only check the HTTP status code, a plain success check (< 400) will treat 207 as fine — be sure your integration checks the status field in the response body and treats PARTIAL_SUCCESS as not a clean success. It means the object may now exist with partial, or no, data in it, and you should check the object before relying on it.

Object-store limitations​

For object-store datasources (Amazon S3, Azure Blob Storage), loading works differently from a SQL database because the underlying storage has no native append or row-level update operation:

  • Upsert is not supported. These datasources only support append and truncate — there's no per-row key to upsert against.
  • Append is a read-modify-write of the whole object: DataDios reads the existing file, adds the new rows, and writes the object back in full. Because of this, append is only available while the existing object is under the platform's upload size cap (100 MB). Once an object has grown past that size, further appends to it are refused, since doing the read-modify-write would require materializing an object over the limit.

Which datasources support what​

The table below reflects which datasources currently support Create Object and Load Data (with which write modes and file formats), as configured in the platform today. This includes DataDios' own platform datasources (DQ HUB, API Collection, Command Registry) alongside customer warehouse/storage connectors — both kinds show up in your datasource list and both are covered here.

DatasourceCreate ObjectLoad DataWrite modesFormats
Amazon S3YesYesappend, truncatecsv
Azure Blob StorageYesYesappend, truncatecsv, parquet
PostgreSQLYesYesappend, truncate, upsertcsv
SQL ServerYesYesappend, truncate, upsertcsv
SnowflakeYesYesappend, truncate, upsertcsv
MySQLYesYesappend, truncate, upsertcsv
MariaDBYesYesappend, truncate, upsertcsv
Oracle DatabaseYesYesappend, truncate, upsertcsv
DB2 DatabaseYesYesappend, truncate, upsertcsv
SQLiteYesYesappend, truncate, upsertcsv
BigQueryYesYesappend, truncate, upsertcsv
TrinoYesYesappend onlycsv
DQ HUBYesYesappend, truncatecsv
API CollectionYesYesappend, truncatecsv
Command RegistryYesYesappend onlycsv
FolderYesNo——
DuckDBYesNo——
RedisYesNo——
ArcadeDBYesNo——
HadoopNoNo——
IBM Db2 for iNoNo——
NetezzaNoNo——
DatabricksNoNo——
RedshiftNoNo——
TeradataNoNo——
Azure SynapseNoNo——
Google Cloud StorageNoNo——
MongoDBNoNo——

Datasources without Create Object or Load Data support — including Redis, ArcadeDB (create only, no load), Hadoop, Netezza, Databricks, Redshift, Teradata, Azure Synapse, Google Cloud Storage, and MongoDB — are not offered for this feature in the current release; the create/load actions are hidden or unavailable for these connection types.

Google Cloud Storage

Create Object and Load Data support for Google Cloud Storage is implemented but not yet enabled — it has not been verified against a real bucket. It will be turned on once that verification is complete.

Azure Synapse

Azure Synapse is not currently exposed for Create Object or Load Data at all, on either serverless or dedicated pools. (A serverless Synapse SQL pool has no writable storage of its own and could never support this regardless — its tables are read-only views over external storage — but at present the feature is off for Synapse connections generally, not only for serverless pools.)