Refactor issue90

[marrip]

Hey Xiaoli,

My first attempt at compressing the changes you made by using pandas. I would like to refactor a bit more but I want you to be on board first so let me know if this looks ok. 🙏

OBS! The branch comes from my fork again so when merged we still need to merge your branch, just fyi 😉

[xiaoliz0]

Hi Martin, I start to check and test your codes now.
Emm, a small bug I fixed is in line 1354 in Script/PRONTO.py "pandas.read.csv" to "pandas.read_csv". More errors come from reading the small_variant_table file due to the formatting issue?

@tonjegul It is probably better we prepare a testing sample data to Martin, then he could test and fix the codes in a much shorter time.

[marrip]

Hi Martin, I start to check and test your codes now. Emm, a small bug I fixed is in line 1354 in Script/PRONTO.py "pandas.read.csv" to "pandas.read_csv". More errors come from reading the small_variant_table file due to the formatting issue?

Hey Xiaoli! Sorry about that, I think I fixed both issues now. But I agree, some testdata would be highly appreciated ☺️

[xiaoliz0]

Hi @marrip, I added a fake sample data files in the "test_data/ous/test_files_for_PRONTO_newFunctions" which was designed by Tonje. The ppt report in the output folder is the output. Could you use it to test your codes for debugging? Thanks! :)

[marrip]

Hi @marrip, I added a fake sample data files in the "test_data/ous/test_files_for_PRONTO_newFunctions" which was designed by Tonje. The ppt report in the output folder is the output. Could you use it to test your codes for debugging? Thanks! :)

🍐 fect! Great work, guys, will test and get back to you 👍

[marrip]

Hey, we tested with the test data locally and that worked fine now. We mitigated all bugs we found. There seems to be a discrepancy between the test raw data and the expected output ppt or maybe I misunderstood something. Please let me know.

[xiaoliz0]

Hey, we tested with the test data locally and that worked fine now. We mitigated all bugs we found. There seems to be a discrepancy between the test raw data and the expected output ppt or maybe I misunderstood something. Please let me know.

Thanks Martin! Great work! Emm, about the output ppt for the testing sample, I will take a further check. Maybe this is due to the name of the small_variant_table file. Which makes the scripts did not read it, so only the meta data is updated in the output ppt. I will upload a new output ppt later.

[xiaoliz0]

Hi @marrip, I try to test your new codes in our side. PRONTO failes when there is variant in small_variant_table_forQC.tsv fitting for the FILTER0-1 conditions in configure_PRONTO.ini
The error is like: "root:Column CPSR_ACMG_class not found in data".

[marrip]

Hi @marrip, I try to test your new codes in our side. PRONTO failes when there is variant in small_variant_table_forQC.tsv fitting for the FILTER0-1 conditions in configure_PRONTO.ini The error is like: "root:Column CPSR_ACMG_class not found in data".

I think I have fixed this on my branch?

[xiaoliz0]

Hey, we tested with the test data locally and that worked fine now. We mitigated all bugs we found. There seems to be a discrepancy between the test raw data and the expected output ppt or maybe I misunderstood something. Please let me know.

Thanks Martin! Great work! Emm, about the output ppt for the testing sample, I will take a further check. Maybe this is due to the name of the small_variant_table file. Which makes the scripts did not read it, so only the meta data is updated in the output ppt. I will upload a new output ppt later.

Just did a quick check. The scripts did read small_variant_table_forQC.tsv file. This probably caused by the issue you mentioned earlier " all_data will always be filtered according to the last item in the list".

[xiaoliz0]

Hi @marrip, I try to test your new codes in our side. PRONTO failes when there is variant in small_variant_table_forQC.tsv fitting for the FILTER0-1 conditions in configure_PRONTO.ini The error is like: "root:Column CPSR_ACMG_class not found in data".

I think I have fixed this on my branch?

I checked out the codes with this PR106. Do you have some codes not committed here?

[marrip]

Hi @marrip, I try to test your new codes in our side. PRONTO failes when there is variant in small_variant_table_forQC.tsv fitting for the FILTER0-1 conditions in configure_PRONTO.ini The error is like: "root:Column CPSR_ACMG_class not found in data".

I think I have fixed this on my branch?

I checked out the codes with this PR106. Do you have some codes not committed here?

ah, I might have an idea. Are you using your test data or the test data in the repo? There was some formatting problems.

[xiaoliz0]

Hi @marrip, I try to test your new codes in our side. PRONTO failes when there is variant in small_variant_table_forQC.tsv fitting for the FILTER0-1 conditions in configure_PRONTO.ini The error is like: "root:Column CPSR_ACMG_class not found in data".

I think I have fixed this on my branch?

I checked out the codes with this PR106. Do you have some codes not committed here?

ah, I might have an idea. Are you using your test data or the test data in the repo? There was some formatting problems.

Yes, I use the test data which is the same in the repo. What is the formatting problem? Could you provide a right format file I could take a further check?

[xiaoliz0]

Hi @marrip thanks for your updates. I tested with the new codes and got some errors like:

I can not find '0,657' in the "_TMB_coding.txt" file, but only see a '657' in the "AF_tumor_DNA" column. Is this something about the way to read the files with panda? Or some settings need to be changed from European numbers?