Last update: 20240920

Variant features to RDF concept metadata

This process is starting with sequencing_assay, which includes library_preparation, sequencing_run, etc. We will continue through the pipeline until we reach the final end-point required to report a pathogenic variant.

This documentation outlines the transformation of variant information from whole genome sequence (WGS) data to a format adhering to RDF structure data concepts. The aim is to ensure that the omic output from genomic analyses can be seamlessly integrated into clinical data warehouses with high fidelity and clarity.

Number of variables:

• All SPHN RDF concept info (see SPHN_dataset_release_2024_2_20240502.xlsx) = 1503
• Subset of relevant concepts = 76 (see example_subset_concepts.tsv)
• Relevant WGS pipeline logs = 62 (see example_report.tsv)

• Currently automated match = 13 (see example_report_concepts.Rds)

  This repository uses a public dataset of example genetic variants and sequencing/analysis log data.

Overview

The process begins with the extraction of variant data from a genomic study, (no sensitive data is included in the public example set). The key variant features such as Chromosome (CHROM), Position (POS), Reference Allele (REF), and Alternate Allele (ALT) are formatted alongside metadata that describes their relationship to RDF concepts. This ensures downstream users can map these data accurately within clinical and research frameworks.

This document is to be updated as we improve the linking of result terms to SPHN_dataset_release_2024_2_20240502.xlsx which is critical so that downstream users can correctly map data.

Aims

1. Data preparation: Start with the extracted variant information from the genomic pipeline.
2. Key term identification: Focus on essential genomic terms like CHROM, POS, REF, and ALT, Sequencing run, Sequencing instrument.
3. Metadata addition: Attach metadata columns that specify RDF concept requirements such as type and cardinality.
4. Validation checklist:
   • Do we have all necessary variant descriptors present?
   • Is there inclusion and accuracy of all metadata explanations?
   • Is there alignment of metadata with SPHN omic concepts?
   • Downstream users (mapping) can choose from TSV, HTML, JSON, and Rds. Any others needed?

Current version

The observation column is highlighted in GREEN. It contains the data which we report as output from the pipeline for use in our database Here is the completed concept observations (this is file example_report_concepts.html):

Semantic evidence network

I have added the following method to automatically plot semantic evidence networks which show how evidence provenance has been generated. The dataset is organised into three hierarchical grouping levels based on the column concept_or_concept_compositions_or_inherited. The top level, Level 1, includes entries where this column equals “concept”. The subsequent levels, Level 2 and Level 3, contain entries where this column does not equal “concept”. The distinction between Levels 2 and 3 lies in the presence of distinct observations; Level 3 specifically represents the final observation associated with the general concept names from Level 2, differentiated further by non-empty values in the observation column, making Level 3 essentially a detailed continuation of Level 2.

Nodes within the network are structured with the following attributes: general_concept_name, id, group, name, and observation, where general_concept_name recurs in both Level 2 and Level 3 but differs based on the associated observation. Edges within this hierarchical setup link nodes from Level 1 to Level 2 and from Level 2 to Level 3 using general_concept_name as a consistent link identifier, facilitating a connection between the initial abstract concept level and its more detailed observational breakdowns.

Downloads

Example output (in mutiple filetypes) can be downloaded from the public set:

Example inputs can be downloaded from the public set:

Process Steps for Variant Features to RDF Concept Mapping

This section outlines the sequential processing steps from data extraction through to the final merged dataset, prepared for RDF concept mapping. Each step corresponds to a specific script and handles distinct data types or stages in data preparation and merging.

1. Export Variant Data from Study
   • Extracts variant data from genomic projects focusing on specific genes and filtering for high-impact variants, saving them in formats like RDS and TSV for further processing.
2. Read Variant Report Data
   • Loads and transforms variant data into a long format to facilitate metadata annotation, preparing the data by adding a column for metadata requirements.
3. Read Sequencing Assay Data
   • Extracts key sequencing assay data such as identifiers, read depth, and file formats from logs or metadata files, providing crucial context for sequencing parameters.
4. Read BWA Read Group Data
   • Parses BWA and samtools log files to extract detailed read group information, including metadata about the sequencing run such as machine, file paths, and read group specifications.
5. Read Fastq Header Data - Analyzes headers from FASTQ files to extract sequencing instrument details and run metrics, offering a granular look at the sequencing runs which is instrumental in validating sequencing quality and parameters.
6. Merge Datasets
   • Combines all processed data from the previous steps into a single dataset, aligning them by common identifiers and ensuring consistency across data types.
7. Map Pipeline Output to SPHN Concepts
   • Maps the merged dataset to standard SPHN RDF concepts, ensuring each data point is correctly classified according to standardized ontology, thus aligning detailed genomic data with broader healthcare data standards.

Terms used in WGS logging

Descriptions for sequencing assay (WGS) terms

Descriptions for FASTQ/BAM readgroup terms

Descriptions for genetic variants terms