Skip to content

Improve upstream-test-suite robustness and diagnostics#52

Open
prikryla wants to merge 1 commit into
masterfrom
upstream-test-suite-robustness
Open

Improve upstream-test-suite robustness and diagnostics#52
prikryla wants to merge 1 commit into
masterfrom
upstream-test-suite-robustness

Conversation

@prikryla

@prikryla prikryla commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Add socat and softhsm to recommended packages so meson tests that depend on tang (via socat) and PKCS#11 (via softhsm) run instead of being silently skipped. Increase TMT duration from 10m to 30m to prevent slow architectures (s390x, aarch64) from being killed mid-run. Use --timeout-multiplier 3 for meson test to accommodate resource-constrained VMs. Preserve meson-logs/testlog.txt via rlFileSubmit so future failures can be diagnosed without re-running.

Summary by Sourcery

Improve robustness and diagnostics of the upstream test suite run.

Enhancements:

  • Increase meson test timeout via a timeout multiplier to better handle slow or resource-constrained environments.
  • Preserve meson test logs and submit them as artifacts for easier diagnosis of failures.

Add socat and softhsm to recommended packages so meson tests that
depend on tang (via socat) and PKCS#11 (via softhsm) run instead of
being silently skipped.  Increase TMT duration from 10m to 30m to
prevent slow architectures (s390x, aarch64) from being killed
mid-run.  Use --timeout-multiplier 3 for meson test to accommodate
resource-constrained VMs.  Preserve meson-logs/testlog.txt via
rlFileSubmit so future failures can be diagnosed without re-running.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@prikryla prikryla self-assigned this Jul 8, 2026
@sourcery-ai

sourcery-ai Bot commented Jul 8, 2026

Copy link
Copy Markdown
Reviewer's guide (collapsed on small PRs)

Reviewer's Guide

Makes upstream-test-suite runs more robust and diagnosable by relaxing meson test timeouts and preserving logs, and by adjusting the TMT metadata (packages and duration) so tests that depend on tang/PKCS#11 run reliably on slower architectures and constrained environments.

File-Level Changes

Change Details Files
Relax meson test execution to better handle slow or resource-constrained environments and preserve logs for debugging.
  • Run meson tests with a higher timeout multiplier to reduce spurious timeouts on slow VMs or architectures.
  • Copy the meson test log from the build directory into a temporary directory after the test run, allowing post-run inspection even if the build directory is cleaned.
  • Submit the copied meson test log as an artifact so failures can be diagnosed without re-running the suite.
Sanity/upstream-test-suite/runtest.sh
Adjust TMT metadata to ensure dependent tests run and the overall test run is not prematurely killed on slow architectures.
  • Recommend installation of additional packages (socat, softhsm) so tang- and PKCS#11-dependent meson tests are executed instead of skipped.
  • Increase the TMT test duration limit from 10 minutes to 30 minutes to avoid timeouts on slower architectures such as s390x and aarch64.
Sanity/upstream-test-suite/main.fmf

Tips and commands

Interacting with Sourcery

  • Trigger a new review: Comment @sourcery-ai review on the pull request.
  • Continue discussions: Reply directly to Sourcery's review comments.
  • Generate a GitHub issue from a review comment: Ask Sourcery to create an
    issue from a review comment by replying to it. You can also reply to a
    review comment with @sourcery-ai issue to create an issue from it.
  • Generate a pull request title: Write @sourcery-ai anywhere in the pull
    request title to generate a title at any time. You can also comment
    @sourcery-ai title on the pull request to (re-)generate the title at any time.
  • Generate a pull request summary: Write @sourcery-ai summary anywhere in
    the pull request body to generate a PR summary at any time exactly where you
    want it. You can also comment @sourcery-ai summary on the pull request to
    (re-)generate the summary at any time.
  • Generate reviewer's guide: Comment @sourcery-ai guide on the pull
    request to (re-)generate the reviewer's guide at any time.
  • Resolve all Sourcery comments: Comment @sourcery-ai resolve on the
    pull request to resolve all Sourcery comments. Useful if you've already
    addressed all the comments and don't want to see them anymore.
  • Dismiss all Sourcery reviews: Comment @sourcery-ai dismiss on the pull
    request to dismiss all existing Sourcery reviews. Especially useful if you
    want to start fresh with a new review - don't forget to comment
    @sourcery-ai review to trigger a new review!

Customizing Your Experience

Access your dashboard to:

  • Enable or disable review features such as the Sourcery-generated pull request
    summary, the reviewer's guide, and others.
  • Change the review language.
  • Add, remove or edit custom review instructions.
  • Adjust other review settings.

Getting Help

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • Consider guarding the rlFileSubmit call with a file existence check (or aligning its expected return codes) so that a missing meson-logs/testlog.txt from a partially failed run does not cause a secondary failure when the log copy step returns 1.
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- Consider guarding the `rlFileSubmit` call with a file existence check (or aligning its expected return codes) so that a missing `meson-logs/testlog.txt` from a partially failed run does not cause a secondary failure when the log copy step returns 1.

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants