feat: Improve testing of IR bundles with something like `opa test`

[philipaconrad]

Description

We'd like for folks to be able to test their IR plan bundles the same way they would with opa test.

Proposed solution

In my local experiments, I've observed that while only explicitly-marked entrypoint rules get a plan in the resulting IR, ALL rules appear to get at least a func in the final IR JSON blob.

This means that we could theoretically "fish out" the test rules for running in something like a swift-opa test command. I think we might need to generate an ad-hoc IR plan wrapper, but that might be a doable way of re-implementing opa test over here in Swift OPA. 🤔

Requirements List

Here is what opa test -h prints out for OPA v1.18.2:

Raw text of `opa test -h` (click to expand)

% opa test -h
Execute Rego test cases.

The 'test' command takes a file or directory path as input and executes all
test cases discovered in matching files. Test cases are rules whose names have the prefix "test_".

If the '--bundle' option is specified the paths will be treated as policy bundles
and loaded following standard bundle conventions. The path can be a compressed archive
file or a directory which will be treated as a bundle. Without the '--bundle' flag OPA
will recursively load ALL *.rego, *.json, and *.yaml files for evaluating the test cases.

Test cases under development may be prefixed "todo_" in order to skip their execution,
while still getting marked as skipped in the test results.

Example policy (example/authz.rego):

	package authz

	allow if {
		input.path == ["users"]
		input.method == "POST"
	}

	allow if {
		input.path == ["users", input.user_id]
		input.method == "GET"
	}

Example test (example/authz_test.rego):

	package authz_test

	import data.authz.allow

	test_post_allowed if {
		allow with input as {"path": ["users"], "method": "POST"}
	}

	test_get_denied if {
		not allow with input as {"path": ["users"], "method": "GET"}
	}

	test_get_user_allowed if {
		allow with input as {"path": ["users", "bob"], "method": "GET", "user_id": "bob"}
	}

	test_get_another_user_denied if {
		not allow with input as {"path": ["users", "bob"], "method": "GET", "user_id": "alice"}
	}

	todo_test_user_allowed_http_client_data if {
		false # Remember to test this later!
	}

Example test run:

	$ opa test ./example/

If used with the '--bench' option then tests will be benchmarked.

Example benchmark run:

	$  opa test --bench ./example/

The optional "gobench" output format conforms to the Go Benchmark Data Format.

The --watch flag can be used to monitor policy and data file-system changes. When a change is detected, OPA reloads
the policy and data and then re-runs the tests. Watching individual files (rather than directories) is generally not
recommended as some updates might cause them to be dropped by OPA.

Usage:
  opa test <path> [path [...]] [flags]

Flags:
      --bench                              benchmark the unit tests
      --benchmem                           report memory allocations with benchmark results (default true)
  -b, --bundle                             load paths as bundle files or root directories
      --capabilities string                set capabilities version or capabilities.json file path
      --count int                          number of times to repeat each test (default 1)
  -c, --coverage                           report coverage (overrides debug tracing)
  -z, --exit-zero-on-skipped               skipped tests return status 0
      --explain {fails,full,notes,debug}   enable query explanations (default fails)
      --fail-on-empty                      Whether to fail the test when no test was run
  -f, --format {pretty,json,gobench}       set output format (default pretty)
  -h, --help                               help for test
      --ignore strings                     set file and directory names to ignore during loading (e.g., '.*' excludes hidden files)
  -m, --max-errors int                     set the number of errors to allow before compilation fails early (default 10)
  -p, --parallel int                       the number of tests that can run in parallel, defaulting to the number of CPUs (explicitly set with 0). Benchmarks are always run sequentially. (default 16)
  -r, --run string                         run only test cases matching the regular expression
  -s, --schema string                      set schema file path or directory path
      --sort {none,duration}               sort the JSON formatted test output (default none)
  -t, --target {rego,wasm}                 set the runtime to exercise (default rego)
      --threshold float                    set coverage threshold and exit with non-zero status if coverage is less than threshold %
      --timeout duration                   set test timeout (default 5s, 30s when benchmarking)
      --v0-compatible                      opt-in to OPA features and behaviors prior to the OPA v1.0 release
      --var-values                         show local variable values in test output
  -v, --verbose                            set verbose reporting mode
  -w, --watch                              watch command line files for changes

From that, I propose the following requirements:

Core requirements

• Search through the folder or files provided, and out of any discovered IR plans, extract all test_* / todo_test_* func names. These correspond to rules of the same prefixes in Rego.
  • Once the rules are extracted, Swift OPA will need to generate a basic plan wrapper for each rule so that it can be run in isolation.
• Need support for loading JSON (and eventually YAML) data files in the files/folders list.

Extra requirements from flags

flag	needed for MVP of feature?
--bench	❌
--benchmem	❌
--bundle	✅
--capabilities	❌?
--count	✅
--coverage	❌
--exit-zero-on-skipped	✅
--explain	❌
--format	❌
--help	✅
--ignore	✅
--max-errors	✅
--parallel	✅
--run	✅
--schema	❌
--sort	✅
--target	✅
--threshold	✅
--timeout	✅
--v0-compatible	❌
--var-values	❌
--verbose	✅
--watch	❌