Releases: FINNGEN/kanta_lab_preprocessing
Release list
V3
Thiis pull request makes significant improvements to the README.md documentation, clarifying column descriptions, updating technical notes, and expanding explanations of the data processing pipeline. The changes focus on enhancing transparency around harmonization, extraction, and QC steps, as well as updating mappings and file references to reflect the current version of the pipeline.
Enhancements to column descriptions and harmonization process:
Expanded descriptions for MEASUREMENT_VALUE_MERGED, OUTCOME_POS_EXTRACTED, TEST_OUTCOME_TEXT_EXTRACTED, QC_PASS, and QC_NOTES columns, including details on ad-hoc harmonization, post-merging QC, blacklist/whitelist tables, and new urine measurement extraction logic. [1] [2]
Clarified that MEASUREMENT_VALUE_MERGED now merges harmonized values and extracted values, and described the QC process using multiple mapping and threshold tables. [1] [2]
Updates to pipeline and technical documentation:
Improved explanations of extraction filters, including new functions for extracting positive results and plus abnormality values from free text, and updated mapping file references.
Expanded QC section to describe new post-harmonization fixes, blacklist masking, and conflict flagging between outcome columns, with references to relevant mapping and QC tables.
Corrections and clarifications to mapping and metadata:
Updated raw-to-output column mapping table to correct column names, clarify mappings, and remove obsolete columns such as SERVICE_PROVIDER_ID. [1] [2]
Renamed and clarified the metadata file section to EXTENDED_COLUMNS FILE, explaining its purpose and unique columns for debugging and QC.
Minor improvements and terminology updates:
Improved terminology for new columns and clarified that certain columns cannot be shared due to privacy reasons.
Fixed typos and improved formatting for technical notes and column descriptions throughout the documentation. [1] [2]
These updates make the documentation more comprehensive and user-friendly, providing clearer guidance for downstream analysis and QC.