Skip to main content

Load Data & Create Object

DataDios lets you push data into a datasource directly from the product — either by loading rows into an object (table, blob, file) that already exists, by creating a brand-new object, or by doing both in a single step. This page covers the three related actions, the write modes available when loading, upload limits, and which datasources support each capability.

What you can do​

ActionWhat it does
Create ObjectCreates a new object (for example, a SQL table or an object-store blob/key) in the selected datasource, from a schema you define or infer from a file.
Load DataLoads rows from an uploaded file (or a file fetched from a URL) into an object that already exists.
Create and LoadCreates a new object and loads the uploaded file into it in one step.
Infer SchemaReads an uploaded file's columns/types without creating or loading anything — used to prefill the column list when you're building a new object from a file.
Edit itemUpdates an existing object in place (for example, a single ticket or record) — see the datasource list below; not the same as Load Data's file-based row loading.

Before loading into an existing object, the Load Plan step lets you preview how the file's columns map onto the target object's columns, and flags any required column that isn't mapped or any value DataDios can't safely convert to the target's type, before you commit to the load.

Load Data (and Edit item) are only offered on regular writable objects — not on views, procedures, functions, materialized views, sequences, indexes, foreign tables, databases, schemas, or individual columns, even on a datasource that otherwise supports Load Data.

Edit item is further limited to datasources that support updating an existing object in place — today that's API Collection, Command Registry, DQ HUB, and ServiceNow. On every other datasource, editing a row isn't offered; use Load Data to add or replace rows instead.

Sync vs. async loads​

Async loading is temporarily unavailable

Every load (via Load Data or Create and Load) currently runs synchronously only — a file above the sync limit (see Upload limits below) is rejected outright rather than accepted and handed off to a background job. This is temporary; the rest of this section describes the async behavior once it's re-enabled.

A load (via Load Data or Create and Load) runs synchronously if the file is small enough — the request completes and DataDios returns the outcome (rows inserted/updated, or a failure) directly in the response, with HTTP status 200.

Once the uploaded file is larger than the sync threshold (see Upload limits below), DataDios instead accepts the request, hands the load off to a background job, and responds immediately with HTTP status 202 and a request ID. Your client is expected to poll DataDios' async status-check endpoint with that request ID until the job reaches a terminal state.

Status values you'll see while polling (once async loading is re-enabled):

StatusMeaning
PROCESSINGThe load is still running. The response also carries a completion percentage and a rows-processed count you can show as progress.
SUCCESSThe load finished and all rows were written.
FAILThe load did not complete — no rows (or no further rows) were written.
PARTIAL_SUCCESSSee Partial success below — don't treat this as a full success.
note

If you are polling from the DataDios UI, you may also see a Stopped Watching state on the screen after about 10 minutes. This is a UI-side timeout that stops the browser from continuing to poll — it does not mean the backend job failed. The load may still be running or may have already finished; refresh or re-check the object to see its current state.

Write modes​

When loading into an existing object, you choose how the incoming rows are applied:

  • Append — every row in the file is added to the object as a new row. Append has no concept of "already loaded this file" — re-uploading the same file with append adds a second copy of every row rather than detecting or skipping the duplicates. If you need to reload a file without doubling its rows, use truncate (replace everything) or upsert (update matching rows, insert the rest) instead.

  • Truncate — the object's existing contents are cleared first, then the file's rows are written in.

  • Upsert — you choose one or more key columns. For each incoming row, DataDios looks for an existing row whose key columns match:

    • If a match is found and at least one non-key column actually differs from the incoming row, the existing row is updated to the new values.
    • If a match is found and every value already agrees with the incoming row, the row is left alone (counted as skipped, not updated).
    • If no match is found, the row is inserted as new.

    This means upsert is a genuine update-or-insert: a row whose key already exists is overwritten with the new values, not skipped and not duplicated. The response reports rows_inserted, rows_updated (rows that changed), and rows_skipped (rows that matched and needed no change) separately, so you can tell exactly what happened.

    Upsert requires the source file to have a unique value per row for the chosen key columns — if the file contains the same key twice, the load is rejected before anything is written to the target object (which row would "win" is otherwise undefined).

Example

Loading a file of customer records with upsert and key column customer_id: a row for a customer_id that's new to the table is inserted; a row for a customer_id already in the table, where an address changed, updates just that row; a row that's identical to what's already stored is left as-is.

Not every datasource supports every write mode — see the capability table below. Object-store datasources in particular do not support upsert (see Object-store limitations).

Upload limits​

  • Files up to 5 MB load synchronously — the response comes back on the same request with the outcome (HTTP 200).
  • Uploads over 5 MB are rejected outright (HTTP 413) — async loading for larger files is temporarily unavailable, see Sync vs. async loads above.

This threshold is the same regardless of which datasource you're loading into or from (upload, or a source URL DataDios fetches on your behalf) — DataDios does not currently vary the sync cutoff per connector.

This 5 MB figure is the cap on the incoming file you're loading. It's a separate number from the 100 MB cap on an object-store target's existing size, described in Object-store limitations below — that limit governs whether an already-stored blob is small enough to append to, not how large a file you can upload.

Partial success​

Occasionally a create-and-load's data load fails after the object was already created, and the automatic rollback that removes that object also fails — so the object is still there, empty or partly filled, rather than cleaned up. DataDios reports this outcome as PARTIAL_SUCCESS, distinct from both SUCCESS and FAIL, and the HTTP response uses status 207 (Multi-Status) rather than 200. (When the rollback succeeds, the outcome is a plain FAIL instead — the object is gone, not partially there.)

If you only check the HTTP status code, a plain success check (< 400) will treat 207 as fine — be sure your integration checks the status field in the response body and treats PARTIAL_SUCCESS as not a clean success. It means the object may now exist with partial, or no, data in it, and you should check the object before relying on it.

Object-store limitations​

For object-store datasources (Amazon S3, Azure Blob Storage), loading works differently from a SQL database because the underlying storage has no native append or row-level update operation:

  • Upsert is not supported. These datasources only support append and truncate — there's no per-row key to upsert against.
  • Append is a read-modify-write of the whole object: DataDios reads the existing file, adds the new rows, and writes the object back in full. Because of this, append is only available while the existing object is under the platform's upload size cap (100 MB). Once an object has grown past that size, further appends to it are refused, since doing the read-modify-write would require materializing an object over the limit.

Which datasources support what​

The table below reflects which datasources currently support Create Object and Load Data (with which write modes and file formats), as configured in the platform today. This includes DataDios' own platform datasources (DQ HUB, API Collection, Command Registry) alongside customer warehouse/storage connectors — both kinds show up in your datasource list and both are covered here.

DatasourceCreate ObjectLoad DataWrite modesFormats
Amazon S3YesYesappend, truncatecsv
Azure Blob StorageYesYesappend, truncatecsv, parquet
PostgreSQLYesYesappend, truncate, upsertcsv
SQL ServerYesYesappend, truncate, upsertcsv
SnowflakeYesYesappend, truncate, upsertcsv
MySQLYesYesappend, truncate, upsertcsv
MariaDBYesYesappend, truncate, upsertcsv
Oracle DatabaseYesYesappend, truncate, upsertcsv
DB2 DatabaseYesYesappend, truncate, upsertcsv
SQLiteYesYesappend, truncate, upsertcsv
BigQueryYesYesappend, truncate, upsertcsv
DatabricksYesYesappend, truncate, upsertcsv
TrinoYesYesappend onlycsv
DQ HUBYesYesappend, truncatecsv
API CollectionYesYesappend, truncatecsv
Command RegistryYesYesappend onlycsv
FolderYesNo——
DuckDBYesNo——
RedisYesNo——
ArcadeDBYesNo——
HadoopNoNo——
IBM Db2 for iNoNo——
NetezzaNoNo——
RedshiftNoNo——
TeradataNoNo——
Azure SynapseNoNo——
Google Cloud StorageNoNo——
MongoDBNoNo——

Datasources without Create Object or Load Data support — including Redis, ArcadeDB (create only, no load), Hadoop, Netezza, Redshift, Teradata, Azure Synapse, Google Cloud Storage, and MongoDB — are not offered for the missing action in the current release; the create/load actions are hidden or unavailable for these connection types.

Databricks

Databricks supports Create Object for catalogs, schemas, tables and columns, and Load Data and Create and Load into tables — see Databricks for the grants each needs. Loads are CSV only. If the load in a Create and Load fails, the new table is kept (empty), and the outcome is PARTIAL_SUCCESS — see Partial success. A Create and Load into a table name that already exists loads into that existing table. If a load fails partway, the rows written before the failure stay in the table, and the error says how many.

Google Cloud Storage

Create Object and Load Data support for Google Cloud Storage is implemented but not yet enabled — it has not been verified against a real bucket. It will be turned on once that verification is complete.

Azure Synapse

Azure Synapse is not currently exposed for Create Object or Load Data at all, on either serverless or dedicated pools. (A serverless Synapse SQL pool has no writable storage of its own and could never support this regardless — its tables are read-only views over external storage — but at present the feature is off for Synapse connections generally, not only for serverless pools.)

Folder

Create Object on a Folder datasource lets you create either a file or a folder. It only works when the datasource's Connection Mode is Local Folder; it is not available for Remote SSH (SFTP) connections.

A file/folder toggle on the Create Item form switches between the two. Creating a folder inside another folder places it there, not at the root.

A file's extension is checked against a blocklist, not an allowlist: unviewable binary, legacy image, audio/video, archive, and executable formats are refused (for example .bmp, .tiff, .svg, .mp3, .wav, .avi, .rar, .exe, .dll), while text-based and structured formats — CSV, JSON, PARQUET, XLSX, PDF, DOCX, TXT, XML, and others — are accepted, along with photos (PNG, JPG, JPEG, GIF, WEBP), video (MP4, MOV), and archives (ZIP, TAR, GZ, 7Z). Photos up to 20 MB preview inline in Object Data — a larger photo still uploads and stays stored, it just skips the automatic preview. Video and archives are stored only, with no preview and currently no way to retrieve the original file back out of DataDios.