Add initial implementation of the functest engine by elizashurov · Pull Request #1000 · HCK-CI/AutoHCK

elizashurov · 2026-06-15T12:23:47Z

Introduces a new functest engine that runs JSON-driven functional tests
directly on a client VM.

Extend QMP with argument support, wait_for event polling, and QMPError
Add FunctestEngine with JSON-driven test loading, execution, and step handling
Add FunctestClient to boot and prepare the client VM
Add FunctestTools subclassing AutoHCK::Tools for WinRM access
Add an example balloon driver test suite

gemini-code-assist

Code Review

This pull request introduces a new JSON-driven functional test engine (functest) to AutoHCK, adding CLI options, a test executor, step handlers, and test definitions for balloon service and driver verification. It also extends the QEMU setup manager to support hypervisor-level QMP commands and events. The review feedback suggests several key improvements: buffering asynchronous QMP events to prevent them from being lost, adding a connection timeout and custom port support to the WinRM port check, failing fast on engine configuration load errors, handling nil exit statuses for signaled host commands, serializing test timestamps as ISO 8601 strings to avoid JSON generation issues, and adding safety checks for missing keys in the test loader.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

Copilot

Pull request overview

Adds a new functest engine to run JSON-defined functional test cases directly on a client VM (no HLK Studio), including hypervisor-level QMP steps and a sample balloon driver suite.

Changes:

Extend QMP support with command arguments, event buffering, event polling (wait_for), and a QMPError.
Introduce the FunctestEngine workflow (suite/test loading, step execution, results output) plus FunctestClient + FunctestTools for WinRM-based guest control.
Add example functest suites/cases and PowerShell scripts for driver sign/update and balloon service/device checks.

Reviewed changes

Copilot reviewed 29 out of 29 changed files in this pull request and generated 9 comments.

Show a summary per file

File	Description
lib/setupmanagers/qemuhck/qmp.rb	Adds QMP command arguments, event buffering, and event waiting support.
lib/setupmanagers/qemuhck/qemuhck.rb	Tracks client VM runners and exposes QMP command/event helpers for functest steps.
lib/setupmanagers/qemuhck/qemu_machine.rb	Exposes the per-runner QMP instance via `attr_reader :qmp`.
lib/setupmanagers/qemuhck/exceptions.rb	Introduces `QMPError` for hypervisor/QMP failures.
lib/setupmanagers/functest_client.rb	New setup manager to boot/prepare a client VM for functest runs.
lib/engines/functest/tests/suites/balloon_driver_tests.json	Example functest suite definition for balloon driver validation.
lib/engines/functest/tests/scripts/verify_driver_signed.ps1	Script to validate driver Authenticode signature in DriverStore.
lib/engines/functest/tests/scripts/remove_driver_from_store.ps1	Script to remove a driver from DriverStore via `pnputil`.
lib/engines/functest/tests/scripts/find_driver_in_store.ps1	Script to locate a driver `.sys` in DriverStore.
lib/engines/functest/tests/scripts/check_driver_installed.ps1	Script to confirm a driver is present via `driverquery`.
lib/engines/functest/tests/scripts/balloon/verify_balloon_service.ps1	Script to validate balloon service state.
lib/engines/functest/tests/scripts/balloon/install_balloon_service.ps1	Script to install balloon service from uploaded package.
lib/engines/functest/tests/scripts/balloon/check_balloon_pnp_device.ps1	Script to validate balloon PnP device presence/health.
lib/engines/functest/tests/cases/driver_update.json	Test case for uninstall/reinstall driver flow with reboots and validations.
lib/engines/functest/tests/cases/driver_sign_check.json	Test case for post-install signature validation flow.
lib/engines/functest/tests/cases/balloon/balloon_service.json	Test case for balloon service install/verify/uninstall flow.
lib/engines/functest/test_loader.rb	Loads suite/test JSON definitions from the functest tests directory.
lib/engines/functest/test_executor.rb	Executes tests/steps, handles cleanup, and records summary results.
lib/engines/functest/test_context.rb	Variable substitution/capture support for JSON-driven steps.
lib/engines/functest/test_case.rb	Sorbet models for functest test cases and step types (incl. QMP steps).
lib/engines/functest/suite.rb	Sorbet model for suites and suite requirement blocks.
lib/engines/functest/step_handler.rb	Implements per-step execution: guest commands, file transfer, reboot, QMP, etc.
lib/engines/functest/functest.rb	Main `FunctestEngine` implementation and results generation.
lib/engines/functest/functest.json	Engine config (definitions path, default timeout, workspace subdir).
lib/engines/functest/functest_tools.rb	WinRM tools wrapper for functest without HLK Studio dependency.
lib/engines/engine.rb	Registers `functest` in the engine selector.
lib/cli.rb	Adds `functest` mode + options for suite/category and testcase selection.
lib/all.rb	Adds autoloads for `FunctestClient`, `FunctestEngine`, and `QMPError`.
config.json	Adds `functest_engine` mapping so `Project` can select the new engine.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

- Add argument support, event waiting, and QMPError to QMP class - Expose qmp reader from QemuMachine::Runner - Track client VM runners to enable per-client QMP access Signed-off-by: Elizabeth Ashurov <eashurov@redhat.com>

- Add FunctestEngine with JSON-driven test loading, execution, and step handling - Add FunctestTools subclassing AutoHCK::Tools for studio-less WinRM access - Add FunctestClient to boot and prepare the client VM - Register functest in the engine selector, CLI, and QemuHCK setup manager Signed-off-by: Elizabeth Ashurov <eashurov@redhat.com>

Demonstrates functest engine usage with a partial balloon driver test suite. Not a full migration of the KAR balloon tests - remaining cases will be added when migrating KAR tests to hck-ci. Signed-off-by: Elizabeth Ashurov <eashurov@redhat.com>

Move minidump collection logic out of Tests into a shared MemoryDumpCollector class. Wire hcktest to use the new class and add minidump collection to functest's TestExecutor so BSOD is captured after every functest test as well. Signed-off-by: Elizabeth Ashurov <eashurov@redhat.com>

Introduce TestResult to unify reporting for both hcktest and functest. This allows functest to use the existing JUnit/HTML reports by converting results to a common format. - hcktest: converts HLK::Test to TestResult - functest: converts result hashes to TestResult Signed-off-by: Elizabeth Ashurov <eashurov@redhat.com>

kostyanf14 · 2026-06-17T13:47:45Z

+  module Models
+    # A single test result used by JUnit, HTML report, and project status.
+    # Both engines produce this via from_hlk_test or from_functest.
+    class TestResult


Why not a Sorbet structure?

No reason, will change

kostyanf14 · 2026-06-17T13:48:33Z

+        result.last_result    = test.last_result
+        result.dump_path      = test.dump_path
+        result.url            = test.url
+        result.estimatedruntime = test.estimatedruntime


Do we use executionstate in any results?

Yes, in result_report.rb

Copilot

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 2 comments.

+      def execute_step_action(step, desc)
+        key, handler = STEP_HANDLERS.find { |k, _| step.send(k.to_sym) }
+        return @logger.warn("Unknown step type in: #{desc}") unless key
+
+        send(handler, step, desc)
+      end


Copilot

Pull request overview

Copilot reviewed 34 out of 34 changed files in this pull request and generated 4 comments.

+      def execute_step_action(step, desc)
+        key, handler = STEP_HANDLERS.find { |k, _| step.send(k.to_sym) }
+        return @logger.warn("Unknown step type in: #{desc}") unless key
+
+        send(handler, step, desc)
+      end


+    def initialize(project)
+      @project = project
+      @logger = project.logger
+      @config = load_engine_config
+      @drivers = load_drivers
+      prepare_extra_sw
+      @logger.info('Functest engine initialized')
+    end


+      drv = @drivers.first
+      return unless drv
+


+      driver_path = @project.options.test.driver_path
+      drivers = @project.engine.drivers
+      return if driver_path.nil? || drivers.empty?


elizashurov requested review from YanVugenfirer and kostyanf14 June 15, 2026 12:23

github-advanced-security AI found potential problems Jun 15, 2026

View reviewed changes

Comment thread lib/engines/functest/step_handler.rb Fixed

Comment thread lib/engines/functest/step_handler.rb Fixed

Comment thread lib/engines/functest/step_handler.rb Fixed

Comment thread lib/engines/functest/step_handler.rb Fixed

gemini-code-assist Bot reviewed Jun 15, 2026

View reviewed changes

elizashurov force-pushed the VIRTWINKVM-2179 branch from 9cdc5e2 to 46f6df7 Compare June 15, 2026 12:51

kostyanf14 reviewed Jun 15, 2026

View reviewed changes

github-advanced-security AI found potential problems Jun 15, 2026

View reviewed changes

elizashurov force-pushed the VIRTWINKVM-2179 branch from 46f6df7 to 72493b6 Compare June 15, 2026 12:56

github-advanced-security AI found potential problems Jun 15, 2026

View reviewed changes

Comment thread lib/engines/functest/step_handler.rb Fixed

elizashurov force-pushed the VIRTWINKVM-2179 branch 2 times, most recently from 26f5e45 to 59c642c Compare June 16, 2026 14:18

kostyanf14 requested a review from Copilot June 16, 2026 14:22

Copilot started reviewing on behalf of kostyanf14 June 16, 2026 14:23 View session

Copilot AI reviewed Jun 16, 2026

View reviewed changes

elizashurov force-pushed the VIRTWINKVM-2179 branch from 59c642c to 26a4e8e Compare June 17, 2026 12:48

github-advanced-security AI found potential problems Jun 17, 2026

View reviewed changes

Comment thread lib/engines/functest/step_handler.rb Fixed

Comment thread lib/engines/functest/step_handler.rb Fixed

elizashurov force-pushed the VIRTWINKVM-2179 branch 3 times, most recently from b53f014 to 8736438 Compare June 17, 2026 13:07

github-advanced-security AI found potential problems Jun 17, 2026

View reviewed changes

Comment thread lib/engines/functest/step_handler.rb Fixed

elizashurov added 4 commits June 17, 2026 16:19

functest: add example balloon test suite

767e536

Demonstrates functest engine usage with a partial balloon driver test suite. Not a full migration of the KAR balloon tests - remaining cases will be added when migrating KAR tests to hck-ci. Signed-off-by: Elizabeth Ashurov <eashurov@redhat.com>

elizashurov force-pushed the VIRTWINKVM-2179 branch from 8736438 to 22df25f Compare June 17, 2026 13:20

kostyanf14 requested a review from Copilot June 17, 2026 13:44

Copilot started reviewing on behalf of kostyanf14 June 17, 2026 13:45 View session

kostyanf14 reviewed Jun 17, 2026

View reviewed changes

Copilot AI reviewed Jun 17, 2026

View reviewed changes

kostyanf14 requested a review from Copilot June 17, 2026 13:58

Copilot started reviewing on behalf of kostyanf14 June 17, 2026 13:58 View session

Copilot AI reviewed Jun 17, 2026

View reviewed changes

Conversation

elizashurov commented Jun 15, 2026

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kostyanf14 Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

elizashurov Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

kostyanf14 Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

elizashurov Jun 17, 2026

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants