Skip to content

monarch-ingest

Details
GitHub monarch-initiative/monarch-ingest
Language Python
Description Data ingest application for Monarch Initiative knowledge graph using Koza

Dependencies

External Dependencies

Package Version
biolink-model >=4.4.2
bmt >=1.0.15,<2
kghub-downloader >=0.4.5,<1
kgx >=2.6.0,<3
linkml >=1.9,<2
linkml-solr ==0.2.3
multi-indexer ==0.0.5
botocore >=1.31,<2
importlib-metadata >=4.6.1
loguru
pydantic >=2.5,<3
sh >=1.14.3,<2
typer >=0.20,<1
yamllint >=1.35.1,<2
linkml-runtime >=1.7.5,<2
pandas >=2.0.3
notebook >=7.3.2,<8
duckdb >=1.3.0,<2
pystow >=0.5.4,<1
koza[grape]>=2.6.2,<3

Documentation

monarch-ingest

documentation

Monarch Ingest generates KGX formatted files conforming to the BioLink Model from a wide variety of biomedical data sources.

The output of the Monarch Ingest process is the Monarch KG, available at data.monarchinitiative.org.

Installation

monarch-ingest is a Python 3.10+ package, managed with uv.

  1. Install uv: 

    curl -LsSf https://astral.sh/uv/install.sh | sh

  2. Clone and install: 

    git clone git@github.com:monarch-initiative/monarch-ingest
cd monarch-ingest
uv sync

  3. Run commands with uv run (or activate the venv with source .venv/bin/activate): 

    uv run ingest --help

Quick Start

Run the full pipeline:

# Download source data
ingest download --all

# Run all transforms
ingest transform --all

# Merge into single node/edge files
ingest merge

# Upload to Monarch bucket
ingest release

CLI Reference

ingest [OPTIONS] COMMAND [ARGS]...

Commands:
  download   Downloads data defined in download.yaml
  transform  Run Koza transformation on specified Monarch ingests
  merge      Merge nodes and edges into kg
  release    Copy data to Monarch GCP data buckets
  closure    Generate closure files
  jsonl      Convert to JSONL format
  solr       Load Solr index
  sqlite     Create SQLite database

Common Options

download: - --ingests TEXT - Which ingests to download data for - --all - Download all ingest datasets

transform: - -i, --ingest TEXT - Run a single ingest - -a, --all - Ingest all sources - -o, --output-dir TEXT - Directory to output data (default: output) - -f, --force - Force ingest even if output exists - -n, --row-limit INTEGER - Number of rows to process - -l, --log - Write DEBUG logs to ./logs/

merge: - --input-dir TEXT - Directory with nodes/edges (default: output/transform_output) - --output-dir TEXT - Output directory (default: output)

Creating an Ingest

An ingest has two main steps: 1. Download - Fetch source data 2. Transform - Convert to KGX format

To run an existing ingest: 

# Download data for a specific ingest
ingest download --ingests ncbi_gene

# Transform with row limit for testing
ingest transform --ingest ncbi_gene --row-limit 20 --log

For detailed ingest creation instructions, see the Create an Ingest documentation.

KG Build Process

The build process includes:

  1. Download - Weekly job fetches source data to cloud storage
  2. Transform - Each ingest produces KGX TSV and RDF output
  3. Merge - Join all transforms, normalize IDs via SSSOM mappings, prune dangling edges
  4. Post-processing - Generate Neo4j dump, denormalized edges, SQLite, and Solr index

For detailed architecture, see the KG Build Process documentation.

Modular Ingests

Some ingests are maintained in separate repositories and referenced as pass-through URLs:

Ingest Repository
Xenbase xenbase-ingest
ZFIN zfin-ingest
Alliance alliance-ingest
ClinGen clingen-ingest
ClinVar clinvar-ingest
OMIM omim-ingest
GO go-ingest
BioGRID biogrid-ingest
PantherDB pantherdb-orthologs-ingest

Resources