Pipeline rehaul to decide how to deal with extracted data

[piotor87]

Historically we decided to keep the extraction process (core pipeline) separate as a post-harmonization step since initially we did not trust the extracted data in the first place.

Now that this mistrust has faded/is fading, as we've seen that the extracted data is statistically coherent with the source data, the question is how to integrate it better in the pipeline. There are two main approaches

1. inject units for each extracted test name. THe idea is to take each significant test name and do a ks test comparison with the harmonized data and come up with a candidate unit. After that the data can be harmonized normally. PROS: conceptually very clean, resolves the ambiguity of extracted data altogether. CONS: there are 1k test names with 100 entries, 400 with 1000 entries. Plus the extracted data tends to also have mismatch of units so the list will always require a lot of manual work
2. treat unitless source units equally as extracted data. Which makes sense since ultimately they are the same thing once you trust the numerical entry. THis means that we could replace the extracted column with an unitless column --> assign target unit --> QC as before. N.B. The Qcing will still be required also in case 1 because of the unit mismatches that will happen regardless. PROS: much less work in maintaining injection lists CONS: more hackish and it will still require having a separate column

[piotor87]

WE shuold also add unit tests so that every commit needs to pass them

[piotor87]

Possible solution i've been exploring.

For each omop id get extracted data and run a bimodality test and, if bimodal, return (both normal and lognormal test) the threshold to separate. We then split the test into two subtests and for each we compare to harmonizeddata. If KS pass then --> assign target unit. Else find in source data a candidate distribution ks wise. For the best one --> assign that unit

[piotor87]

E.g. for
cloZAPine [Moles/volume] in Serum or Plasma Substance Concentration
​s-klotsa [] (Extracted)
[ 1.89, 332.85, 686.46, 972.8, 1236.66, 1499.27, 1784.23, 2108.76, 2573.77 ]
Selected model: lognormal (lowest BIC) --> Separator: 35.151 --> BIC: 12689.68, BC: 0.906, DIP: 0.051071772936887604 (p=0.0)

[piotor87]

But what do you do when the threshold is not clear?

From du-prot 3020876 Protein [Mass/time] in 24 hour Urine

[piotor87]

We are trying this.

[piotor87]

Addressed.

An overview can be found here https://finngen.github.io/kanta_lab_preprocessing/injection/