Join WQP data with the TADA criteria table

[wokenny13]

This is the first PR step for joining the criteria table to the WQP data for analysis. This new function will automatically join the WQP data frame with the TADA compatible criteria table dependent on what has been filled out in the criteria table.

Join WQP results to a criteria table by the best available key:

1. TADA.ComparableDataIdentifier (if present in both and non-NA in criteria)
2. TADA.CharacteristicName + TADA.ResultSampleFractionText + TADA.MethodSpeciationName
3. TADA.CharacteristicName + TADA.ResultSampleFractionText
4. TADA.CharacteristicName + TADA.MethodSpeciationName
5. TADA.CharacteristicName (or when byChar = TRUE)

Users can make the criteria table join only by the TADA.CharacteristicName regardless of what has been filled out in the criteria table.

For testing, please look at the example tests and test out example WQP data retrievals with criteria table filled out. You can use the autoassign method in TADA_DefineCriteriaMethodology() if desired for testing or use a criteria table from the TADACommunityHub.

Ex.

Data_Nutrients_UT2 <- TADA_RunKeyFlagFunctions(Data_Nutrients_UT)
Data_Nutrients_UT_Harmonized <- TADA_HarmonizeSynonyms(Data_Nutrients_UT2)
UTAH_Criteria <- TADA_DefineCriteriaMethodology(Data_Nutrients_UT_Harmonized, org_id = "UTAHDWQ", auto_assign = TRUE, displayUniqueId = TRUE)[[1]]
UTAH_with_criteria <- TADA_Analysis_Join_WQP_Criteria(Data_Nutrients_UT_Harmonized, UTAH_Criteria)

[github-actions]

coverage-report

File	Coverage		Missing
All files	67%	✅
R/ATTAINSCrosswalks.R	53%	✅	32-876 1101-1105 1110-1115 1120-1123 1128-1159 1165-1168 1174-1177 1191-1194 1205-1220 1266-1269 1513-1514 1521-1522 1527-1528 1555-1562 1693 2016-2019 2024 2053-2057 2069-2072 2080 2089-2096 2111-2120 2125-2133 2139-2162 2182-2197 2202-2205 2210-2241 2254-2257 2330-2334 2429-2438 2654-2655 2724 2895-2901 3121-3699 3866 3869-3874 3892-3895 3933-3945 3994-4132 4136-4139 4202-4203 4239 4315-4321 4440 4517-4519 4523-4525 4561-4606 4663-4665
R/CriteriaAnalysis.R	71%	✅	112 119-132 177-181 255 258 271 281 286 292 396-398 400 409 422 448-486 511-518 543-570
R/CriteriaMethods.R	96%	✅	309 313-317 333-362 586 751-754 771-774 919-922 931-934 1005-1013 1183-1189 1193-1199 1383-1384 1389-1390 1463-1469 1482 1561 1739-1741 1772 2179
R/TADARefTables.R	91%	✅	75-78 306-308 370-383 394 866-878 966 976-978 992 1013-1015 1020-1037 1051 1199-1203 1396-1402 1419-1432
R/Utilities.R	67%	✅	547 665-666 670 675-677 772 784-813 901-912 1056-1058 1126-1127 1181-1185 1295-1296 1300-1301 1311-1315 1320-1325 1367-1521 1615 1672 1680 1741 1749-1769 1795-1796 1805-1819 1843-1846 1910-1911 1928-1930 1935 1938-1940 2024-2102 2133-2135 2171-2339 2366-2370 2379-2383 2600 2653 2659-2662 2666 2677 2696-2697 2699-2714 2716 2718 2724 2727-2729 2731 2766 2781 2787 2795 2800 2808-2813 2842 2852-2854 2875-2879 2889-2891 2894-2899 2903-2908 2916 2926-2927 2938 2954 2986-3020 3135 3144 3183-3187

Minimum allowed coverage is 20%

Generated by 🐒 cobertura-action against 4d4b19a

[wokenny13]

@cristinamullin @hillarymarler This is ready for review

[cristinamullin]

@wokenny13 I tried to run your example in the PR description and ran into this error:

> Data_Nutrients_UT2 <- TADA_RunKeyFlagFunctions(Data_Nutrients_UT)
All characteristic/fraction combinations are valid in your dataframe. Returning input dataframe with TADA.SampleFraction.Flag column for tracking.
TADA_FlagSpeciation: Rows with Suspect speciations have been flagged but retained. Review these rows using the new TADA.MethodSpeciation.Flag column before proceeding and/or set clean = 'suspect_only' or 'both'.
> Data_Nutrients_UT_Harmonized <- TADA_HarmonizeSynonyms(Data_Nutrients_UT2)
Warning: Your dataframe contains suspect metadata combinations in the following flag columns:
                 Flag_Column Result Count
1 TADA.MethodSpeciation.Flag            4
2       TADA.ResultUnit.Flag            4
> UTAH_Criteria <- TADA_DefineCriteriaMethodology(Data_Nutrients_UT_Harmonized, org_id = "UTAHDWQ", auto_assign = TRUE, displayUniqueId = TRUE)
TADA_DefineCriteriaMethodology: auto_assign = TRUE was selected but no MLSummaryRef. Generating TADA_MLSummary with default assignment.
TADA_DefineCriteriaMethodology: auto_assign = TRUE was selected. Running TADA_ParametersForAnalysis with default assignment.
TADA_DefineCriteriaMethodology: auto_assign = TRUE was selected. Running TADA_UsesForAnalysis with default assignment.
TADA_UsesForAnalysis: 
  NAs were found in ATTAINS.ParameterName. 
  Please ensure that you have inputted all field values of interest in 
  the ATTAINS.ParameterName column generated from TADA_ParametersForAnalysis() function.
TADA_UsesForAnalysis: auto_assign == TRUE was selected, 
  assigning all unique ATTAINS.UseName, by ATTAINS.OrganizationIdentifier, to any ATTAINS.ParameterName that an 
  organization have not done assessments for in prior ATTAINS cycle. Please review carefully and Exclude rows as needed.
TADA_DefineCriteriaMethodology: auto_assign = TRUE was selected.
  Finding an alias match between ATTAINS parameter name and Criteria Search Tool (CST) standardized pollutant names.
  Finding an alias match between ATTAINS use name and Criteria Search Tool (CST) uses.
  If an ATTAINS.ParameterName and ATTAINS.UseName alias was found, populating these rows with the CST magnitude values.
  A many-to-many match is likely. User review is needed to ensure accuracy in crosswalk method.
Warning in TADA_DefineCriteriaMethodology:  There are 2 TADA.CharacteristicName units that do not match with the CST autoassign MagnitudeUnit values. Converting these MagnitudeUnit Values from the CST to match the TADA.ResultMeasure.MeasureUnitCode in your dataframe. Please review these conversions.
Warning message:
In TADA_MLSummary(.data, displayNA = TRUE, org_id = org_id, usesRef = TADA_usesRef,  :
  TADA_MLSummary: displayNA = TRUE was selected: Too many sites or uses and parameters. Cannot assign all uses and parameters to each monitoring sites in the output. Defaulting to displayNA = FALSE

> UTAH_with_criteria <- TADA_Analysis_Join_WQP_Criteria(Data_Nutrients_UT_Harmonized, UTAH_Criteria)
Error in TADA_Analysis_Join_WQP_Criteria(Data_Nutrients_UT_Harmonized,  : 
  is.data.frame(criteria) is not TRUE
Called from: TADA_Analysis_Join_WQP_Criteria(Data_Nutrients_UT_Harmonized, 
    UTAH_Criteria)

[wokenny13]

@wokenny13 I tried to run your example in the PR description and ran into this error:

> Data_Nutrients_UT2 <- TADA_RunKeyFlagFunctions(Data_Nutrients_UT)
All characteristic/fraction combinations are valid in your dataframe. Returning input dataframe with TADA.SampleFraction.Flag column for tracking.
TADA_FlagSpeciation: Rows with Suspect speciations have been flagged but retained. Review these rows using the new TADA.MethodSpeciation.Flag column before proceeding and/or set clean = 'suspect_only' or 'both'.
> Data_Nutrients_UT_Harmonized <- TADA_HarmonizeSynonyms(Data_Nutrients_UT2)
Warning: Your dataframe contains suspect metadata combinations in the following flag columns:
                 Flag_Column Result Count
1 TADA.MethodSpeciation.Flag            4
2       TADA.ResultUnit.Flag            4
> UTAH_Criteria <- TADA_DefineCriteriaMethodology(Data_Nutrients_UT_Harmonized, org_id = "UTAHDWQ", auto_assign = TRUE, displayUniqueId = TRUE)
TADA_DefineCriteriaMethodology: auto_assign = TRUE was selected but no MLSummaryRef. Generating TADA_MLSummary with default assignment.
TADA_DefineCriteriaMethodology: auto_assign = TRUE was selected. Running TADA_ParametersForAnalysis with default assignment.
TADA_DefineCriteriaMethodology: auto_assign = TRUE was selected. Running TADA_UsesForAnalysis with default assignment.
TADA_UsesForAnalysis: 
  NAs were found in ATTAINS.ParameterName. 
  Please ensure that you have inputted all field values of interest in 
  the ATTAINS.ParameterName column generated from TADA_ParametersForAnalysis() function.
TADA_UsesForAnalysis: auto_assign == TRUE was selected, 
  assigning all unique ATTAINS.UseName, by ATTAINS.OrganizationIdentifier, to any ATTAINS.ParameterName that an 
  organization have not done assessments for in prior ATTAINS cycle. Please review carefully and Exclude rows as needed.
TADA_DefineCriteriaMethodology: auto_assign = TRUE was selected.
  Finding an alias match between ATTAINS parameter name and Criteria Search Tool (CST) standardized pollutant names.
  Finding an alias match between ATTAINS use name and Criteria Search Tool (CST) uses.
  If an ATTAINS.ParameterName and ATTAINS.UseName alias was found, populating these rows with the CST magnitude values.
  A many-to-many match is likely. User review is needed to ensure accuracy in crosswalk method.
Warning in TADA_DefineCriteriaMethodology:  There are 2 TADA.CharacteristicName units that do not match with the CST autoassign MagnitudeUnit values. Converting these MagnitudeUnit Values from the CST to match the TADA.ResultMeasure.MeasureUnitCode in your dataframe. Please review these conversions.
Warning message:
In TADA_MLSummary(.data, displayNA = TRUE, org_id = org_id, usesRef = TADA_usesRef,  :
  TADA_MLSummary: displayNA = TRUE was selected: Too many sites or uses and parameters. Cannot assign all uses and parameters to each monitoring sites in the output. Defaulting to displayNA = FALSE

> UTAH_with_criteria <- TADA_Analysis_Join_WQP_Criteria(Data_Nutrients_UT_Harmonized, UTAH_Criteria)
Error in TADA_Analysis_Join_WQP_Criteria(Data_Nutrients_UT_Harmonized,  : 
  is.data.frame(criteria) is not TRUE
Called from: TADA_Analysis_Join_WQP_Criteria(Data_Nutrients_UT_Harmonized, 
    UTAH_Criteria)

Hi Cristina,

the output of TADA_DefineCriteriaMethodology now has 3 df, so this line should be updated below:

UTAH_Criteria <- TADA_DefineCriteriaMethodology(Data_Nutrients_UT_Harmonized, org_id = "UTAHDWQ", auto_assign = TRUE, displayUniqueId = TRUE)[[1]]

[cristinamullin]

The ATTAINS and CST uses crosswalk function, TADA_GetTADAUsesAliasRef, needs review/edits. Too many rows are being generated with the auto assign method.

Currently, for the ATTAINS.UseName there is also a UseGroup that these uses would fall under. The CST has an Aquatic Life or Human Health indicator column. If a CST use is NA for that Aquatic Life or Human Health indicator column or if that ATTAINS.UseName is NA, it will get assigned to all unique CST uses. For example, this means many CST uses are getting matched to "Domestic Source" for UT even though they are not all related to domestic source...

To do:

1. Suggestion from Cristina: Greg provided a crosswalk that was manually reviewed. These matches should be labeled as approved.
2. Suggestion from Kenny: The current TADA_GetTADAUsesAliasRef only leverages the CRITERIATYPEAQUAHUMHLTH in its crosswalk. I think additional labeling of these ATTAINS Use Groups with the CRITERIATYPE_WATERORG from the CST could be very useful to reduce this list. Ex. Labeling Domestic Source (which falls under the ATTAINS Use Group of DRINKINGWATER_USE) to W could help narrow this list down.
3. How many are left for manual review after completing 1 and 2 above... anything else we can do to automate the matching so there are less left to review manually?
4. Finally, review and modify the TADAUsesAlias table to specify which rows should be rejected. Make this a a team priority!

[wokenny13]

I compared the "like match" method currently used in TADA vs Greg's table. Greg's table after filtering only had 156 ATTAINS Uses that matched to CST Uses (filtering out NAs, filtering out any changes in ATTAINS.Uses domain as well as CST uses domain.)

From there, the TADA method was able to match 147 out of those 156 matches in Greg's table. The TADA method is looking at % like matches to avoid any manual review needed, and the tolerance is meant to be adjustable. TADA identifies 7,708 close matches out of the 17281 total matches. The remaining 10k matches are based on the Human health/Aquatic Life and Organism & water or water only indicator crosswalk of ATTAINS Use Groups.

2. From adding O & W indicator, this reduced the table from over 30k aliases to 17281 aliases. Reduced table comparison for Domestic Source:

3. We could increase the tolerance to the maximum and have these as "very certain matches" if we would like. These very certain matches would be if there is an exact match of word in any order.

[wokenny13]

@cristinamullin This PR is ready to review. We can work on additional approval or rejection of rows in a future PR by adding ways to differentiate the methods of the uses alias (percent match method vs A/H and O/W categories crosswalk).

The use alias from Greg's table was added as approved and adding on the O/W indicator to the ATTAINS Use Group was added to help reduce the list of potential use alias match to those that are more certain. I added arg inputs to ensure that the csv file does not replace any rows that got approved (reviewing these additional inputs would be nice to test).