pheval
Details
| GitHub | monarch-initiative/pheval |
| Language | Python |
| Description | A framework for empirical evaluation of phenotype matching and prioritisation |
Dependencies
External Dependencies
| Package | Version |
|---|---|
| python | >=3.9,<4.0.0 |
| jaydebeapi | >=1.2.3 |
| tqdm | >=4.64.1 |
| pandas | >=1.5.1 |
| deprecation | >=2.1.0 |
| click | >=8.1.3 |
| class-resolver | >=0.4.2 |
| phenopackets | ^2.0.2 |
| oaklib | >=0.5.6 |
| ^3.0.0 | |
| pyaml | ^21.10.1 |
| plotly | ^5.13.0 |
| seaborn | ^0.12.2 |
| matplotlib | ^3.7.0 |
| pyserde | ^0.9.8 |
| polars | ^0.19.15 |
| scikit-learn | ^1.4.0 |
| duckdb | ^1.0.0 |
Documentation
PhEval - Phenotypic Inference Evaluation Framework
Overview
The absence of standardised benchmarks and data standardisation for Variant and Gene Prioritisation Algorithms (VGPAs) presents a significant challenge in the field of genomic research. To address this, we developed PhEval, a novel framework designed to streamline the evaluation of VGPAs that incorporate phenotypic data. PhEval offers several key benefits:
- Automated Processes: Reduces manual effort by automating various evaluation tasks, thus enhancing efficiency.
- Standardisation: Ensures consistency and comparability in evaluation methodologies, leading to more reliable and standardised assessments.
- Reproducibility: Facilitates reproducibility in research by providing a standardised platform, allowing for consistent validation of algorithms.
- Comprehensive Benchmarking: Enables thorough benchmarking of algorithms, providing well-founded comparisons and deeper insights into their performance.
PhEval is a valuable tool for researchers looking to improve the accuracy and reliability of VGPA evaluations through a structured and standardised approach.
For more information please see the full documentation.
Download and Installation
- Ensure you have Python 3.8 or greater installed.
- Install with
pip:pip install pheval - See list of all PhEval utility commands:
pheval-utils --help
Usage
The PhEval CLI offers a variety of commands categorised into two main types: Runner Implementations and Utility Commands. Below is an overview of each category, detailing how they can be utilised to perform various tasks within PhEval.
Runner Implementations
The primary command used within PhEval is pheval run. This command is responsible for executing concrete VGPA runner implementations, that we sometimes term as plugins. By using pheval run, users can leverage these runner implementations to: execute the VGPA on a set of test corpora, produce tool-specific result outputs, and post-process tool-specific outputs to PhEval standardised TSV outputs.
Some concrete PhEval runner implementations include the Exomiser runner and the Phen2Gene runner. The full list of currently implemented runners can be found here
Please read the documentation for a step-by-step for creating your own PhEval plugin.
Utility Commands
In addition to the main run command, PhEval provides a set of utility commands designed to enhance the overall functionality of the CLI. These commands can be used to set up and configure experiments, streamline data preparation, and benchmark the performance of various VGPA runner implementations. By utilising these utilities, users can optimise their experimental workflows, ensure reproducibility, and compare the efficiency and accuracy of different approaches. The utility commands offer a range of options that facilitate the customisation and fine-tuning to suit diverse research objectives.
Example Usage
To add noise to an existing corpus of phenopackets, this could be used to assess the robustness of VGPAs when less relevant or unreliable phenotype data is introduced:
pheval-utils scramble-phenopackets --phenopacket-dir /phenopackets --scramble-factor 0.5 --output-dir /scrambled_phenopackets_0.5
To update the gene symbols and identifiers to a specific namespace:
pheval-utils update-phenopackets --phenopacket-dir /phenopackets --output-dir /updated_phenopackets --gene-identifier ensembl_id
To prepare VCF files for a corpus of phenopackets, spiking in the known causative variants:
pheval-utils create-spiked-vcfs --phenopacket-dir /phenopackets --hg19-template-vcf /template_hg19.vcf --hg38-template-vcf /template_hg38.vcf --output-dir /vcf
Alternatively, you can wrap all corpus preparatory commands into a single step. Specifying --variant-analysis/--gene-analysis/--disease-analysis will check the phenopackets for complete records documenting the known entities. If template vcf(s) are provided this will spike VCFs with the known variant for the corpus. If a --gene-identifier is specified then the corpus of phenopackets is updated.
pheval-utils prepare-corpus \
--phenopacket-dir /phenopackets \
--variant-analysis \
--gene-analysis \
--gene-identifier ensembl_id \
--hg19-template-vcf /template_hg19.vcf \
--hg38-template-vcf /template_hg38.vcf \
--output-dir /vcf
See the documentation for instructions on benchmarking and evaluating the performance of various VGPAs.