Universal uploader of the tabular data to the database

Source code

Tool. Runs: nsaph.loader.data_loader

Description

This tool ingests tabular data, usually in CSV format into the database

Inputs

Name

Type

Default

Description

registry

File

A path to the data model file

table

string

the name of the table to be created

database

File

Path to database connection file, usually database.ini

connection_name

string

The name of the section in the database.ini file

domain

string

input

[‘File’, ‘File[]’]

A path the downloaded data files

pattern

string

*.csv*

threads

int

4

number of threads, concurrently writing into the database

page_size

int

1000

explicit page size for the database

log_frequency

long

100000

informational logging occurs every specified number of records

limit

long?

if specified, the process will stop after ingesting the specified number of records

depends_on

Any?

a special field used to enforce dependencies and execution order

Outputs

Name

Type

Description

log

File

errors

stderr