Skip to content

ingest

Usage:

$ ingest [OPTIONS] COMMAND [ARGS]...

Options:

  • --version
  • --install-completion: Install completion for the current shell.
  • --show-completion: Show completion for the current shell, to copy it or customize the installation.
  • --help: Show this message and exit.

Commands:

  • download: Downloads data defined in download.yaml...
  • download-release-metadata: Fetch release-metadata.yaml from each...
  • build-receipt: Aggregate per-ingest release-metadata.yaml...
  • transform: Run Koza transformation on specified...
  • merge: Merge nodes and edges into kg
  • closure
  • jsonl
  • neo4j-csv
  • sqlite
  • solr
  • export
  • report: Run Koza QC on specified Monarch ingests
  • graph-stats: Generate graph statistics from merged KG...
  • connectivity: Generate connectivity (connected...
  • release: Copy data to Monarch GCP data buckets

ingest download

Downloads data defined in download.yaml and runs post-download processing

Usage:

$ ingest download [OPTIONS]

Options:

  • -i, --ingest TEXT: Run a single ingest (see download.yaml for a list)
  • --ingests TEXT: Which ingests to download data for
  • --ingest_file PATH: A yaml file which has a newline seperated list of ingests to perform.
  • --all / --no-all: Download all ingest datasets [default: all]
  • --help: Show this message and exit.

ingest download-release-metadata

Fetch release-metadata.yaml from each ingest repo's latest GitHub release.

Usage:

$ ingest download-release-metadata [OPTIONS]

Options:

  • -o, --output-dir TEXT: Directory to write per-repo release-metadata.yaml files [default: data/release-metadata]
  • --help: Show this message and exit.

ingest build-receipt

Aggregate per-ingest release-metadata.yaml files into output/metadata.yaml.

Usage:

$ ingest build-receipt [OPTIONS]

Options:

  • -i, --input-dir TEXT: Directory of per-ingest release-metadata.yaml files [default: data/release-metadata]
  • -o, --output-dir TEXT: Directory to write metadata.yaml [default: output]
  • --kg-name TEXT: Name of the KG being built [default: monarch-kg]
  • --kg-version TEXT: Version tag (defaults to today's date)
  • --help: Show this message and exit.

ingest transform

Run Koza transformation on specified Monarch ingests

Usage:

$ ingest transform [OPTIONS]

Options:

  • -o, --output-dir TEXT: Directory to output data [default: output]
  • -i, --ingest TEXT: Run a single ingest (see ingests.yaml for a list)
  • --ingests TEXT: Which ingests to download data for
  • --ingest_file PATH: A yaml file which has a newline seperated list of ingests to perform.
  • --phenio / --no-phenio: Run the phenio transform [default: no-phenio]
  • -a, --all: Ingest all sources
  • -f, --force: Force ingest, even if output exists (on by default for single ingests)
  • -d, --debug / -q, --quiet: Use --quiet to suppress log output, --debug for verbose, including Koza logs
  • -l, --log: Write DEBUG level logs to ./logs/ for each ingest
  • -n, --row-limit INTEGER: Number of rows to process
  • --help: Show this message and exit.

ingest merge

Merge nodes and edges into kg

Usage:

$ ingest merge [OPTIONS]

Options:

  • --input-dir TEXT: Directory with nodes and edges to be merged [default: output/transform_output]
  • --output-dir TEXT: Directory to output data [default: output]
  • -d, --debug / -q, --quiet: Use --quiet to suppress log output, --debug for verbose
  • --closure / --no-closure: Apply closure to merged graph [default: no-closure]
  • --kg-name, --kg_name TEXT: The name of the kg being produced. Merge artificat will be ultimately be stored in output/$KG_NAME.tar.gz [default: monarch-kg]
  • --help: Show this message and exit.

ingest closure

Usage:

$ ingest closure [OPTIONS]

Options:

  • --help: Show this message and exit.

ingest jsonl

Usage:

$ ingest jsonl [OPTIONS]

Options:

  • --help: Show this message and exit.

ingest neo4j-csv

Usage:

$ ingest neo4j-csv [OPTIONS]

Options:

  • --help: Show this message and exit.

ingest sqlite

Usage:

$ ingest sqlite [OPTIONS]

Options:

  • --help: Show this message and exit.

ingest solr

Usage:

$ ingest solr [OPTIONS]

Options:

  • --help: Show this message and exit.

ingest export

Usage:

$ ingest export [OPTIONS]

Options:

  • --help: Show this message and exit.

ingest report

Run Koza QC on specified Monarch ingests

Usage:

$ ingest report [OPTIONS]

Options:

  • --help: Show this message and exit.

ingest graph-stats

Generate graph statistics from merged KG database

Usage:

$ ingest graph-stats [OPTIONS]

Options:

  • -i, --input-db TEXT: Path to input DuckDB database [default: output/monarch-kg.duckdb]
  • -o, --output TEXT: Output YAML file path [default: output/merged_graph_stats.yaml]
  • -b, --backend TEXT: Backend to use: 'koza' (new) or 'kgx' (legacy) [default: koza]
  • --help: Show this message and exit.

ingest connectivity

Generate connectivity (connected component) report from merged KG database

Usage:

$ ingest connectivity [OPTIONS]

Options:

  • -i, --input-db TEXT: Path to input DuckDB database [default: output/monarch-kg.duckdb]
  • -o, --output TEXT: Output YAML summary path [default: output/connectivity_summary.yaml]
  • --help: Show this message and exit.

ingest release

Copy data to Monarch GCP data buckets

Usage:

$ ingest release [OPTIONS]

Options:

  • --dir TEXT: Directory with kg to be released [default: output]
  • --kghub / --no-kghub: Also release to kghub S3 bucket [default: no-kghub]
  • --help: Show this message and exit.