ARS_Test_Runner output

[ShervinAbd92]

currently if we run the ARS_test_Runner with the following command:

ARS_Test_Runner --env 'ci' --query_type 'treats_creative' --expected_output '["TopAnswer","TopAnswer"]' --input_curie 'MONDO:0005301' --output_curie  '["PUBCHEM.COMPOUND:107970","UNII:3JB47N2Q2P"]' 

the json report is outputted like bellow:

{
    "pks": {
        "parent_pk": "e29c5051-d8d7-4e82-a1a1-b3cc9b8c9657",
        "merged_pk": "56e3d5ac-66b4-4560-9f56-7a4d117e8003",
        "aragorn": "14953570-7451-4d1b-a817-fc9e7879b477",
        "arax": "8c88ead6-6cbf-4c9a-9570-ca76392ddb7a",
        "unsecret": "bd084e27-2a0e-4df4-843c-417bfac6f8c7",
        "bte": "d28a4146-9486-4e98-973d-8cdd33270595",
        "improving": "d8d3c905-ec07-491f-a078-7ef0f489a409"
    },
    "results": [
        {
            "PUBCHEM.COMPOUND:107970": {
                "aragorn": "Fail",
                "arax": "Pass",
                "unsecret": "Fail",
                "bte": "Pass",
                "improving": "Pass",
                "ars": "Pass"
            }
        },
        {
            "UNII:3JB47N2Q2P": {
                "aragorn": "Fail",
                "arax": "Pass",
                "unsecret": "Fail",
                "bte": "Pass",
                "improving": "Pass",
                "ars": "Pass"
            }
        }
    ]
}

Is this format acceptable for the Test Harness?

[RichardBruskiewich]

Thanks @ShervinAbd92 (cc: @maximusunc, @sierra-moxon).

From a test-agnostic perspective, I note that this JSON is a multilevel dictionary data structure with specified keys corresponding to the basic reporting requirements and context of the ARS_test_runner, which is totally understandable and fine. The results are simply pass/fail for specified ARA's, with keys simply corresponding to the InputID (only) of the Implied TestAsset, but seems opaque to other TestCase context. The JSON report also assumes that the ARS is the main entry point of the testing.

Not too surprisingly, this result output model is not quite generic enough to cover the full landscape of testing, including the "OneHop" TestRunner.

That said, though, I guess we can iterate towards a shared TestResult model by comparing the various 'native' outcomes of all the current flavors of TestRunners.

[RichardBruskiewich]

Maybe let me take a baby step forward towards a 'generic' model: could the 'results' dictionary keys be generalized into a kind of "subject-predicate'object" string pattern?

For example, the above sample "results" section could be generalized as follows (feel free to correct this perception):

...
    "results": [
        {
            "PUBCHEM.COMPOUND:107970--?->?": {
                "aragorn": "Fail",
                "arax": "Pass",
                "unsecret": "Fail",
                "bte": "Pass",
                "improving": "Pass",
                "ars": "Pass"
            }
        },
...
    ]

...

which simply means that the predicate and subject of the TestAsset are 'wild card' values. It this is not the case, then they should be specified in the results. For example, if we know that that source TestAsset had the predicate biolink:treats, then it should be part of the key, namely:

...
    "results": [
        {
            "PUBCHEM.COMPOUND:107970--biolink:treats->?": {
                "aragorn": "Fail",
                "arax": "Pass",
                "unsecret": "Fail",
                "bte": "Pass",
                "improving": "Pass",
                "ars": "Pass"
            }
        },
  ...
    ]

...

which simply implies that the TestCase is 'get me any object target which the given subject 'treats'.

If, in fact, the actual TestCase was to validate that this particular chemical treats a particular disease (in this case, "Fingolimod treats "relapsing-remitting multiple sclerosis", then the following full TestAsset triple could perhaps be specified:

...

...
    "results": [
        {
            "PUBCHEM.COMPOUND:107970--biolink:treats->MONDO:0005314": {
                "aragorn": "Fail",
                "arax": "Pass",
                "unsecret": "Fail",
                "bte": "Pass",
                "improving": "Pass",
                "ars": "Pass"
            }
        },
...    ]

...

By now, I can hear @maximusunc saying "...no, no, no..." then proceeding to suggest the obvious: that we simply use a globally resolvable CURIE back to the correct TestAsset (or full TestCase) in the 'Tests' repository. corresponding to the indicated result.

To do this, though, requires a quick agreement on URI identification of TestAsset/Case/Suite etc. components in the Tests repository.

Note for example, the static TestCase JSON file number 103 which is similar in structure (just doesn't have PUBCHEM.COMPOUND:107970 as its subject).

If the 'TestCase' base URI were something like https://raw.githubusercontent.com/NCATSTranslator/Tests/main/test_cases/TestCase_ and the locally defined prefix, something like 'TestCase:', then the CURIE TestCase:103 could be used as a results dictionary key (above), which could be resolved by pertinent testing infrastructure software into https://github.com/NCATSTranslator/Tests/blob/main/test_cases/TestCase_103.json (right Max?).

[ShervinAbd92]

hi @RichardBruskiewich, I originally had that type of result's key in my report and Max suggested we change it to only showing output curie, but i can do either way. we just have to come to an agreement on the format.

[maximusunc]

😆 At least I'm predictable!

So here are my current thoughts/questions:

• The Test Harness knows what test is being run (it's the one that sent it off), so does the Runner need to give back the entire input that it got, or is this abbreviated version ok? I wanted this short version for simplicity, but I guess it wouldn't be too hard to work with the former.
• The Test Harness is who does the reporting back to the Information Radiator anyways, it could add the test case uid to the report so test reviewers can easily go back to the Tests repo to look at the entire Test Case.
• I think this comes down to how much info we need to be passing around to the different components. Is this a closed system or do we keep things completely modular at the expense of payloads between the different services? BTW, this ARS Test Runner won't ever return the actual responses from the ARS and ARAs, it will only pass along PKs. We can't and probably never should store these hundreds of MB responses in multiple locations.
• We have an interesting situation here and with the Benchmarks Runner where we're hitting the ARS, but we also have test results for all the ARAs as well. Do we treat each ARA response as its own test and the ARS is just another one in the list? This has more implication on the Information Radiator as to how we show the results. If the ARS test failed, do we just show the whole thing as failed?

[RichardBruskiewich]

OK, all your points generally make good sense, @maximusunc:

1. Agreed, the original TestCase is likely fully recoverable outside of the TestRunner. The only possible caveat there relates to the funny issue I've mentioned that the some TestRunners (e.g. the SRI_Testing... evolving into the OneHopTests Testrunner? Maybe, to some extend, Andrew's Testrunner(?)) may internally configure TestCases based on the cross product of TestAsset and internal templating of various TestCases. I've still got to ponder whether or not this is something that can be transformed into the external Tests accessioning of TestCases (I have a few vague thoughts about this..)

2. if the PK's are being passed around, then yes, the (TRAPI etc.) query/response content is already persisted elsewhere and could, in principle, be independently retrieved (i.e. from the ARS or ARAX UI)

3. On your final point, let's certainly consider system developer needs for the resolution of test failures: can we give them enough granular information about the test failures to facilitate debugging of the software or back end knowledge graphs? This suggests that some kind of query audit trail from UI <-> ARS <-> ARA<-> KP <-> back end (3rd party) knowledge sources will somehow be recoverable from every test failure.

[sierra-moxon]

@ShervinAbd92 - have your tests (e.g. --input_curie 'MONDO:0005301' should return --output_curie '["PUBCHEM.COMPOUND:107970","UNII:3JB47N2Q2P"]' as "TopAnswers") been converted to the TestCase format in the Tests repo yet?

[maximusunc]

Actually, could I make one more suggestion? Could the results be either a dict or a list, not a list of dicts? The dict keys are going to be unique, so I would just make it a dict. Each key is an output id containing its test results.

[maximusunc]

Actually, we also need to handle the cases where an output id is not provided. The Test Runner needs to throw an error for these cases so that the Information Radiator can show that those tests failed and to give that reason.

[ShervinAbd92]

@maximusunc if we make the results as dict with output_id be the keys and pass/fail the value, what happens to the pks? (parent_pk, merged_pk, or inidividual ARAs who returned)

[maximusunc]

I'm only talking about the results key here. The pks would stay the same.

[ShervinAbd92]

{
    "pks": {
        "parent_pk": "e29c5051-d8d7-4e82-a1a1-b3cc9b8c9657",
        "merged_pk": "56e3d5ac-66b4-4560-9f56-7a4d117e8003",
        "aragorn": "14953570-7451-4d1b-a817-fc9e7879b477",
        "arax": "8c88ead6-6cbf-4c9a-9570-ca76392ddb7a",
        "unsecret": "bd084e27-2a0e-4df4-843c-417bfac6f8c7",
        "bte": "d28a4146-9486-4e98-973d-8cdd33270595",
        "improving": "d8d3c905-ec07-491f-a078-7ef0f489a409"
    },
    "PUBCHEM.COMPOUND:107970": {
                "aragorn": "Fail",
                "arax": "Pass",
                "unsecret": "Fail",
                "bte": "Pass",
                "improving": "Pass",
                "ars": "Pass"
    },
    "UNII:3JB47N2Q2P": {
                "aragorn": "Fail",
                "arax": "Pass",
                "unsecret": "Fail",
                "bte": "Pass",
                "improving": "Pass",
                "ars": "Pass"
    }
}

[maximusunc]

{
    "pks": {
        "parent_pk": "e29c5051-d8d7-4e82-a1a1-b3cc9b8c9657",
        "merged_pk": "56e3d5ac-66b4-4560-9f56-7a4d117e8003",
        "aragorn": "14953570-7451-4d1b-a817-fc9e7879b477",
        "arax": "8c88ead6-6cbf-4c9a-9570-ca76392ddb7a",
        "unsecret": "bd084e27-2a0e-4df4-843c-417bfac6f8c7",
        "bte": "d28a4146-9486-4e98-973d-8cdd33270595",
        "improving": "d8d3c905-ec07-491f-a078-7ef0f489a409"
    },
    "results": {
        "PUBCHEM.COMPOUND:107970": {
            "aragorn": "Fail",
            "arax": "Pass",
            "unsecret": "Fail",
            "bte": "Pass",
            "improving": "Pass",
            "ars": "Pass"
        },
        "UNII:3JB47N2Q2P": {
            "aragorn": "Fail",
            "arax": "Pass",
            "unsecret": "Fail",
            "bte": "Pass",
            "improving": "Pass",
            "ars": "Pass"
        }
    }
}

[ShervinAbd92]

and in the case of having empty "" for curie id

{
    "pks": {
        "parent_pk": "e29c5051-d8d7-4e82-a1a1-b3cc9b8c9657",
        "merged_pk": "56e3d5ac-66b4-4560-9f56-7a4d117e8003",
        "aragorn": "14953570-7451-4d1b-a817-fc9e7879b477",
        "arax": "8c88ead6-6cbf-4c9a-9570-ca76392ddb7a",
        "unsecret": "bd084e27-2a0e-4df4-843c-417bfac6f8c7",
        "bte": "d28a4146-9486-4e98-973d-8cdd33270595",
        "improving": "d8d3c905-ec07-491f-a078-7ef0f489a409"
    },
    "results": {
        "PUBCHEM.COMPOUND:107970": {
            "aragorn": "Fail",
            "arax": "Pass",
            "unsecret": "Fail",
            "bte": "Pass",
            "improving": "Pass",
            "ars": "Pass"
        },
        "":{ "empty output id, ignoring analysis"
        },
        "UNII:3JB47N2Q2P": {
            "aragorn": "Fail",
            "arax": "Pass",
            "unsecret": "Fail",
            "bte": "Pass",
            "improving": "Pass",
            "ars": "Pass"
        }
    }
}

[maximusunc]

But this won't work because the output_ids could be empty strings and are therefore not unique. The results key might need to be a list. Maybe something like this?

{
    "pks": {
        "parent_pk": "e29c5051-d8d7-4e82-a1a1-b3cc9b8c9657",
        "merged_pk": "56e3d5ac-66b4-4560-9f56-7a4d117e8003",
        "aragorn": "14953570-7451-4d1b-a817-fc9e7879b477",
        "arax": "8c88ead6-6cbf-4c9a-9570-ca76392ddb7a",
        "unsecret": "bd084e27-2a0e-4df4-843c-417bfac6f8c7",
        "bte": "d28a4146-9486-4e98-973d-8cdd33270595",
        "improving": "d8d3c905-ec07-491f-a078-7ef0f489a409"
    },
    "results": [
        {
            "aragorn": "Fail",
            "arax": "Pass",
            "unsecret": "Fail",
            "bte": "Pass",
            "improving": "Pass",
            "ars": "Pass"
        },
        {
            "error": "No output id provided."
        },
        {
            "aragorn": "Fail",
            "arax": "Pass",
            "unsecret": "Fail",
            "bte": "Pass",
            "improving": "Pass",
            "ars": "Pass"
        },
        {
            "error": "No output id provided."
        }
    ]
}

[ShervinAbd92]

got it, i will work on it