Feature templates

[beykyle]

Pending PR tasks

• Consistent use of type hints based on clear developer guidelines
• Get bfrescox tests passing. Please document and justify all test design decisions in TestData/README (e.g., adjusting threshold test value)
• Update bfrescoxpro testing to run the same tests as bfrescox
• Get all code passing standard adherence check (flake8 / check tox task)
• Since this is a large PR, some clean-up efforts could be skipped in this PR to keep manageable. Determine what will be skipped and create an Issue for each distinct task and add these as tasks in alpha release Issue

PR Self-review

• Review all changes
• Confirm all tox tasks running locally
• Build source distribution for both and universal wheel for bfrescox and confirm that contents are correct and minimal.
• Assess if code coverage levels are acceptable and assess test update needs to address insufficiencies
  • Is it acceptable that a test file not have 100% coverage?
• Confirm all actions passing

@jared321 PR Review

• Review all changes
• Confirm all tox tasks running locally and, for bfrescoxpro, on a variety of different machines with modules and different MPI implementations
• Confirm correct synchronization with main
• Confirm all actions passing

[codecov-commenter]

Codecov Report

❌ Patch coverage is 79.45205% with 210 lines in your changes missing coverage. Please review.
✅ Project coverage is 78.81%. Comparing base (2e69dc9) to head (15387e9).

Files with missing lines	Patch %	Lines
common/parse_parallelization_setup.py	6.25%	60 Missing ⚠️
common/_run_frescox_simulation.py	58.88%	37 Missing ⚠️
common/parse_performance_results.py	12.82%	34 Missing ⚠️
common/Configuration.py	70.27%	11 Missing ⚠️
common/generate_inelastic_template.py	83.92%	9 Missing ⚠️
...src/bfrescox/tests/TestRunUserProvidedTemplates.py	91.22%	5 Missing ⚠️
.../bfrescoxpro/tests/TestRunUserProvidedTemplates.py	92.42%	5 Missing ⚠️
common/generate_elastic_template.py	82.75%	5 Missing ⚠️
...escox_pypkg/src/bfrescox/tests/TestElasticSuite.py	93.75%	4 Missing ⚠️
...cox_pypkg/src/bfrescox/tests/TestInelasticSuite.py	93.75%	4 Missing ⚠️
... and 13 more

Additional details and impacted files

@@             Coverage Diff             @@
##             main      #19       +/-   ##
===========================================
+ Coverage   58.75%   78.81%   +20.06%     
===========================================
  Files          17       35       +18     
  Lines         320     1213      +893     
===========================================
+ Hits          188      956      +768     
- Misses        132      257      +125     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[beykyle]

At this point, tests are passing for user defined templates, including tests which run a simulation and compare cross sections parsed from the output to expected values. See bfrescox_pypkg/src/bfrescox/tests/TestRunUserProvidedTemplates.py.

It should be fairly simple to copy and modify this test file to apply to a new test suite. We need test suites for problems with the template types that can be generated in bfrescox:

• elastic
• inelastic

There is an additional remaining task:

• get GH actions to pass

[beykyle]

Also, we could probably simplify testing with https://docs.python.org/3/library/tempfile.html

[jared321]

As presently written, a bug could render out tests useless. In particular, they do not print out or produce any output to indicate that they have actually run any tests. GH actions would then blissfully run do-nothing tests and report success.

What do you think about hardcoding the number of expected test cases in each test and having the test confirm that that many tests were actually run?

I've now reviewed all the files changed in this PR. @beykyle Can you please review my changes.

After that, I'll still want to do a quick high-level review of the PR to ensure that we've thought of everything and everything is functioning as expected.

[jared321]

My two cents:

I suspect that this README is different from what #44 requests. In particular, this README contains lab notes that explain the provenance of the different tests that you all have decided to use to build up your test suites. They provide a record of why, how, what these different tests are so that people designing and maintaining the test suite can do a good job filling out the test suite so that it is effective without being too large or redundant. Therefore, I don't think that the README needs to be placed in a very public place. Rather, putting these notes alongside the test suite definitions makes good sense.

However, as part of #44 we should probably put information in the Developer Guide that explains the structure of the testing infrastructure and how to use it. This could include pointing to this README and emphasizing the need to keep it up-to-date.

[beykyle]

What do you think about hardcoding the number of expected test cases in each test and having the test confirm that that many tests were actually run?

this is fine with me

[beykyle]

I can smell a merge coming

[jared321]

I have reviewed the test suite setup and have the following comments/requests

• The bfrescoxpro MPI/OpenMP setups all specify just a single MPI process, which means that we aren't adequately testing MPI functionality. Running these with 2 MPI processes should be sufficient without running afoul of the limited number of processors in GH action runners.
• When I run the 48Ca user-provided with bfrescoxpro and 2 MPI processes, the test runs without stopping. It appears that Frescox gets stuck writing data endlessly to fort.38. Please determine if this is due to how we are running/configuring the test setup or if it is a bug in Frescox. If it cannot be fixed and we have to run this setup with just one MPI process, please note this in the test suite README notes and add details.
• We presently only test data results loaded from fort.16, which is the official source of Frescox results. However, the data in those files suffers from serious round-off error. This error is so large that we should be able to set our error checking tolerances to zero absolute difference. Please lower these tolerances as much as possible and add details to the test suite README notes about how threshold values were selected. Also include details regarding this testing limitation of fort.16.
• Tests should also load data from stdout and check it as well since the same data is written to stdout with several orders of magnitude more precision.

[beykyle]

* [ ]  When I run the 48Ca user-provided with `bfrescoxpro` and 2 MPI processes, the test runs without stopping.  It appears that Frescox gets stuck writing data endlessly to `fort.38`.  Please determine if this is due to how we are running/configuring the test setup or if it is a bug in Frescox.  If it cannot be fixed and we have to run this setup with just one MPI process, please note this in the test suite README notes and add details.

I can reproduce this, investigating