ID-45: Session CLI and Scripted Execution

[karlnaden]

Summary

Upgrades the ability to integrate execution of running Inferno tests into CI/CD pipelines and capture current state test suite execution for regression testing. Specifically

• Creates session CLI commands to interact with Inferno sessions on local and remote Inferno instances using the command line. Under the hood, these use the existing json web API which has not changed.
• Creates a execute_script CLI along with a corresponding script configuration yaml format to allow the specification and automated execution of Inferno test sessions along with the comparison of the results against those of a "known good" run.
• Adds derivative tools for executing scripts in github workflows and includes these in all new inferno test kits.

See doc site PR 111 for full documentation on the changes.

Testing Guidance

The best way to test is to run and create scripts, using the PRs for the first set of scripts that target g10 and its dependencies us-core and smart. Specific places to look:

• The execution scripts in this repository that are now run as a part of the github workflows on this PR
• Execution scripts for the specific test kits in the following draft PRs (dependent on this one, but demonstrating that it works). The PRs have instructions for setting them up to be run.
  • smart app launch
  • us core
  • g10

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 84.77%. Comparing base (52e5ad7) to head (8189830).
⚠️ Report is 1 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #760      +/-   ##
==========================================
+ Coverage   84.69%   84.77%   +0.08%     
==========================================
  Files         296      297       +1     
  Lines       13904    13968      +64     
  Branches     1937     1955      +18     
==========================================
+ Hits        11776    11842      +66     
+ Misses       2120     2118       -2     
  Partials        8        8              

Flag	Coverage Δ
backend	92.44% <100.00%> (+0.11%)	⬆️
frontend	79.39% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[ljtucker]

Changes look good overall, but a few notes of possible enhancements to make before merging (all optional).

TO test the validity of the sample scripts, I ran:

• bundle exec rake execute_scripts:run_all
• bundle exec inferno execute_script execution_scripts/demo/demo_individual_tests.yaml
• bundle exec inferno execute_script --allow-commands execution_scripts/demo/demo_wait_group_with_commands.yaml

All tests passed and claimed to match the expected results. I then changed the expecteed json for the demo_individual_tests suite and confirmed that the test then failed and generated diff files. All seems to be working as expected