rsync

[rsync] is a utility for efficiently transferring and synchronizing files between a local and a remote system. omniload supports rsync as a data source through the [oc-rsync] binary, which must be installed separately.

The connector stages remote files into a local directory, then reads them through the shared filesystem reader. All supported file formats work the same as with file://.

URI formats

Two transport schemes are supported, both handled by the same connector.

SSH transport

rsync+ssh://[<user>@]<host>[:<ssh_port>]/path/to/data.csv

Reaches the remote through an SSH shell. The remote path uses rsync’s single-colon host:path wire format internally.

rsync daemon transport

rsync://[<user>[:<password>]@]<host>[:<port>]/module/path/to/data.csv

Connects directly to an rsync daemon on TCP port 873 (default).

URI parameters

Parameters are passed as query-string values on the source URI.

Connection (SSH)

user:

SSH username. Can also be placed in the URI authority (user@host).

ssh_port:

SSH port. Can also be placed in the URI authority (host:2222). Type: int. Default: 22.

ssh_key:

Path to an SSH private key file.

ssh_options:

Additional SSH options, shell-split (e.g. ?ssh_options=-o%20StrictHostKeyChecking%3Dno).

rsh:

Override the entire remote-shell command. When set, ssh_key, ssh_port, and ssh_options are ignored.

Connection (daemon)

user:

Daemon username. Can also be placed in the URI authority.

password:

Inline daemon password. Promoted to the RSYNC_PASSWORD environment variable so it never appears on the command line. Can also be placed in the URI authority.

password_file:

Path to an rsync password file. Takes precedence over an inline password.

port:

Daemon port. Can also be placed in the URI authority. Type: int. Default: 873.

no_motd:

Suppress the daemon message-of-the-day. Type: bool. Default: true.

Transfer tuning

compress:

Enable transfer compression (-z). Type: bool. Default: true.

timeout:

I/O timeout in seconds (--timeout). Type: int.

contimeout:

Connection timeout in seconds (--contimeout). Type: int.

bwlimit:

Bandwidth limit (--bwlimit), e.g. 2m for 2 MiB/s.

rsync_path:

Path to rsync on the remote machine (--rsync-path).

staging_dir:

Local directory for staging transferred files. Defaults to a deterministic subdirectory under the system temp directory. Repeated runs reuse the same staging cache so rsync transfers only the delta.

extra_args:

Escape hatch for arbitrary rsync flags, shell-split (e.g. ?extra_args=--chmod%3DD755%20--numeric-ids).

Remote path and file selection

The remote path is supplied via –source-table. When empty, the path component of the source URI is used as a fallback. The path may contain glob patterns:

┌─────────────────────────┬─────────────────────────────────┐
│         Pattern         │           Description           │
├─────────────────────────┼─────────────────────────────────┤
│ module/exports/*.csv    │ All CSV files at the top level. │
├─────────────────────────┼─────────────────────────────────┤
│ module/exports/**/*.csv │ All CSV files recursively.      │
├─────────────────────────┼─────────────────────────────────┤
│ /srv/data/report.jsonl  │ A single file (SSH transport).  │
└─────────────────────────┴─────────────────────────────────┘

A #format or #key=value fragment on the table selects the reader and passes reader hints, exactly as with file://.

Incremental loading

The rsync connector manages incrementality through file modification time. Unchanged files are skipped by rsync on re-sync, and the reader’s mtime-based selection filters files on the staged tree. Row-level --incremental-key is not supported and will be rejected.

Examples

Load CSV from an rsync daemon into DuckDB.

omniload ingest \
    --source-uri   'rsync://user@data.example.com' \
    --source-table 'module/exports/*.csv' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.exports'

Load JSONL from a remote server via SSH.

omniload ingest \
    --source-uri   'rsync+ssh://deploy@host?ssh_key=/keys/id_ed25519' \
    --source-table '/srv/data/**/*.jsonl' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.data'

Load with bandwidth limit and custom port.

omniload ingest \
    --source-uri   'rsync://host:8730/mod?bwlimit=2m&timeout=30' \
    --source-table 'mod/reports/*.parquet' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.reports'

Load Excel with a reader hint.

omniload ingest \
    --source-uri   'rsync+ssh://host' \
    --source-table '/data/book.xlsx#sheet_name=Sheet1' \
    --dest-uri     'duckdb:///demo.duckdb' \
    --dest-table   'testdrive.staff'

Tip

The --source-table option carries both the remote path and optional reader hints. When the URI already contains a path, the table value takes precedence.

Note

rsync or oc-rsync must be installed and available on the system PATH.