Evals

The eval framework should answer two questions:

1. Are results statistically consistent with trusted implementations on frozen inputs?
2. Is the engine faster on workloads we care about?

Layers

Unit tests validate local pieces: bitsets, Fisher exact p-values, loaders, and annotation propagation.

CLI integration tests validate the executable surface and TSV output shape.

Eval datasets validate end-to-end workflows. These should be explicit, versioned, and reproducible.

Benchmarks validate speed on synthetic and real workloads.

Expression20 Starter Eval

The preferred real-data smoke panel is evals/expression20. It names 20 human MSigDB/MyGeneset.info expression-derived signatures selected across:

• in vivo or primary immune cell-type contrasts;
• patient or clinical disease-state contrasts;
• vaccination and infection-response contrasts;
• cytokine-stimulated primary cells;
• drug response and resistance signatures;
• cell-line stimulation/differentiation signatures.

Fetch the current GMT snapshot:

python3 scripts/fetch_mygeneset_eval.py \
  --manifest evals/expression20/sets.tsv \
  --out-dir evals/expression20/generated

Run expression-vs-expression overlap enrichment:

genesets-rs run evals/expression20/config.yaml

Run Expression20 against official GO:

python3 scripts/run_disease20_go_eval.py \
  --manifest evals/expression20/sets.tsv \
  --out-dir evals/expression20_vs_go/generated \
  --eval-name expression20_vs_go \
  --description "Twenty expression-derived human MSigDB/MyGeneset.info signatures enriched against official GOA human GO annotations."

Disease20 Legacy Eval

Disease20 is retained as a legacy/artificial knowledge-curation contrast. It names 20 human Disease Ontology gene sets from MyGeneset.info. The manifest is intentionally small and hand-reviewed, but it is not representative of differential-expression query workflows.

Fetch the current GMT snapshot:

python3 scripts/fetch_mygeneset_eval.py \
  --manifest evals/disease20/sets.tsv \
  --out-dir evals/disease20/generated

This writes:

• queries.gmt: disease gene sets in GMT format;
• metadata.json: source URL, fetch time, gene counts, and skipped IDs.

Generated files are not required for cargo test. The goal is to keep normal CI deterministic while making real-data evals one command away.

Disease20 vs GO

The second eval uses the same 20 MyGeneset.info disease query sets against official GOA human GO annotations.

Run it:

python3 scripts/run_disease20_go_eval.py

This downloads official GO inputs, prepares ontology-neutral tables, runs four annotation-filter variants, and writes companion metadata:

• all: all evidence codes, NOT filtered, contributes_to retained;
• no_contributes_to: all evidence codes, NOT and contributes_to filtered;
• iba: IBA only, NOT filtered;
• iba_iea: IBA or IEA only, NOT filtered.

The prep stage writes metadata.yaml with source URLs, file digests, GAF header details, closure relation policy, and annotation filter counts. The run stage writes run_metadata.yaml with command timings and result summaries.

These filters are intentionally outside the Rust core engine. The core consumes prepared gene_id, term_id tables; eval helpers decide how to derive those tables from GAF.

By default, generated eval configs include max_p_adjust: 0.05, so result TSVs contain Bonferroni-significant rows. The companion metadata records this cutoff and the per-variant CLI runtime.

GO Timepoints

The go_timepoints eval runs Disease20 against official GO archive releases:

• 2026-03-25;
• 2021-05-01;
• 2016-05-01.

Run it:

python3 scripts/run_go_timepoints_eval.py

The summary metadata contains per-timepoint ontology sizes, GAF metadata, per-variant row counts, run timing, and pairwise diffs over (query_id, target_id) result keys.

Expression500 GO All vs IBA

The IBA-impact eval compares current GOA human all-evidence annotations against the IBA-only annotation variant over the same 500 expression-derived query sets used by the GO impact report:

genesets-workflows go-impact evals/go_iba_impact_expression500.yaml

This uses the same prepared GO ontology, closure, background, and query sets on both sides. Only the annotation table changes:

• left side: all, NOT-filtered GOA human, all evidence codes, contributes_to retained;
• right side: iba, NOT-filtered GOA human, IBA evidence only.

The local June 2026 run used the current 2026-03-25 GO/GOA prep. The annotation variant sizes were:

At Bonferroni-adjusted p <= 0.05, the report produced:

The threshold-crossing diff classified:

The same run now records GO term coverage, which asks which propagated GO targets are scorable but never become significant for this query collection:

The paired coverage comparison classified 10,338 terms as scorable but never significant in both variants, 9,536 as scorable but never significant on only one side, and 1,176 as significant in at least one variant. This is useful ontology “dark matter” for a fixed query collection: terms can be fully annotated and still never show up in enrichment output.

This supports the expected contraction from all evidence to IBA-only, but it also shows why post-processing matters. Because IBA changes target sizes and the correction universe, some very broad terms become significant only in the IBA run. With a simple specificity guard of target size <= 1,000, the diff becomes 3,699 lost, 1,401 retained, and 350 gained rows.

Interpretation: IBA is a useful vetted subset, but not an automatic “less-biased answer set.” It is also a lower-coverage subset, and it can shift enrichment toward terms whose IBA target sizes shrink dramatically. Reports should therefore present all-vs-IBA threshold crossings together with target size deltas and a configurable slim/antislim or size-based term filter.

Qualitative spot checks support that caution:

The working interpretation is therefore: IBA is more curated and useful as a contrast set, but many “lost” calls are plausible biology that lacks IBA coverage. Many “gained” calls are not discoveries; they are threshold crossings caused by smaller IBA target sets and a changed correction universe. The report should expose target-size deltas and representative overlap genes before promoting a gained/lost term as biologically meaningful.

The losses are not all equivalent. Useful report examples should separate desirable losses from concerning losses:

Good losses, where IBA removes broad or weakly interpretable all-evidence calls:

Bad losses, where IBA removes biologically plausible calls for the query set:

The report should therefore label losses as “possibly useful pruning” only when they are broad, recurrent, and replaceable by more specific retained terms. Losses that match the experimental contrast and have coherent overlap genes should be highlighted as IBA coverage gaps.

Some bad losses have partial compensation, but the compensation is usually from retained IBA-significant neighbors rather than from newly gained IBA-only terms:

These spot checks are still too immune-heavy. That is partly a fixture problem: the first 500 MyGeneset/MSigDB GSE* hits are dominated by C7-style immunologic signatures. The IBA report should therefore avoid presenting those examples as representative of all expression data. Less immune-centric checks from the same run are more mixed:

This suggests a useful display rule: for each lost term, show whether the same query still has retained or gained terms in the same ontology neighborhood. A loss with strong neighboring retained terms is less damaging than a loss where the entire biological neighborhood disappears.

For the next expression-scale fixture, sample by source family or keywords rather than taking the first GSE* results. The eval set should deliberately include immune, cancer, stromal, epithelial, neuronal, developmental, cell-cycle, drug-response, and stress-response signatures so qualitative examples are not all drawn from immune contrasts.

The stratified 5,000-set plan lives at evals/expression_like/msigdb_diverse_5k.yaml and can be fetched with:

genesets-workflows fetch-mygeneset-stratified \
  evals/expression_like/msigdb_diverse_5k.yaml

The same source can drive temporal GO and all-vs-IBA reports:

genesets-workflows go-impact evals/go_impact_5y_expression5000_diverse.yaml
genesets-workflows go-impact evals/go_iba_impact_expression5000_diverse.yaml

Benchmark composition matters. The refreshed source reserves modern MSigDB families rather than relying on first-N GSE* or old pathway hits:

This is a benchmark-suite rule, not just a convenience for one notebook: large gene-set benchmarks should report source-family composition and reserve space for newer collections such as MSigDB C8, C9, and 3CA.

A local synthetic scale probe used the current 500-set fixture repeated with unique query IDs. This is not a biological eval, but it answers the engineering question about 10x query-set scale:

The full 5,000-set synthetic all-vs-IBA path was about 31s on this laptop. The all-evidence side wrote 208,220 significant rows; the IBA side wrote 99,290; the diff wrote 237,560 rows. The row count is not exactly 10x the 500-set run because Bonferroni becomes stricter as query count increases.

The actual stratified 5,000-set all-vs-IBA report completed in 52.6s after refreshing the query source to include explicit C8/C9/3CA coverage:

The diff classes were 337,357 lost, 136,246 shared, and 27,620 gained significant pairs. This is a better scale signal than the repeated synthetic run because the query collection is actually diverse and has a broader gene background.

The same stratified source against the five-year temporal GO/GOA diff completed in 47.3s:

The temporal diff classes were 73,369 lost, 215,638 shared, and 257,965 gained significant pairs. This is the current best estimate for a coarse two-snapshot GO impact report over 5,000 diversified expression-like query sets.

Expression-Like Query Sets

Disease Ontology gene sets are convenient but artificial. For scale and realism, use MyGeneset/MSigDB signatures:

genesets-workflows fetch-mygeneset \
  --query 'GSE*' \
  --source-filter msigdb \
  --limit 2000 \
  --out-dir evals/expression_like/generated/msigdb_gse_2k

MSigDB GSE, *_UP, *_DN, and vs signatures are closer to differential-expression outputs than disease ontology gene sets. These should become the main large-query workloads for timing and GO-version diffing.

Expression500 vs Reactome Flat

Reactome can be tested without any new Rust code because the core engine already supports flat GMT target libraries. The current flat Reactome eval uses the same 500 expression-derived query sets as the GO impact report and enriches them against the official Reactome pathway GMT.

Run it:

genesets-workflows reactome-flat

This performs two steps:

1. genesets-workflows prepare-reactome-flat downloads ReactomePathways.gmt.zip from the official Reactome download directory and normalizes it from name, id, genes... to id, name, genes....
2. genesets-rs run evals/reactome_flat/config.yaml enriches the expression query sets against that flat pathway library.

The local May 2026 run prepared 2,557 Reactome pathways over 11,963 gene symbols and produced 509 Bonferroni-significant rows across 85 query sets. The enrichment step took about 0.22 seconds on this laptop.

This is intentionally the flat baseline. A hierarchy-aware Reactome workflow should prepare the standard ontology-style trio:

• terms.tsv from ReactomePathways.txt;
• closure.tsv from ReactomePathwaysRelation.txt;
• gene_terms.tsv from a lowest-level mapping such as NCBI2Reactome.txt, with a gene ID strategy matched to the query sets.

Planned Comparison Evals

The most useful comparison suite will freeze:

• ontology release;
• association release;
• closure relation policy;
• gene universe;
• query gene sets;
• exact correction method;
• expected significant terms from each comparator.

Comparator adapters should run each external tool in containers where possible and emit normalized TSV for comparison.