Skip to content

NO-JIRA: isolate NewSession tests from ambient AWS env vars#8911

Open
ironcladlou wants to merge 1 commit into
openshift:mainfrom
ironcladlou:fix-aws-session-test-env-isolation
Open

NO-JIRA: isolate NewSession tests from ambient AWS env vars#8911
ironcladlou wants to merge 1 commit into
openshift:mainfrom
ironcladlou:fix-aws-session-test-env-isolation

Conversation

@ironcladlou

@ironcladlou ironcladlou commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Clear AWS_PROFILE and AWS_DEFAULT_PROFILE in TestNewSession setup so config.LoadDefaultConfig doesn't pick up the host's profile and fail silently (the error is discarded on util.go:116, producing a zero-value config with empty Region and nil Credentials)

Test plan

  • go test ./cmd/infra/aws/util/ -run TestNewSession -v passes with AWS_PROFILE set

Summary by CodeRabbit

  • Tests
    • Improved AWS session test setup to clear additional environment variables, reducing interference from local credentials and making file-based credential tests more reliable.

Clear AWS_PROFILE and AWS_DEFAULT_PROFILE in test setup so
config.LoadDefaultConfig does not pick up the host's profile and
fail silently (the error is discarded, producing a zero-value config).

Signed-off-by: Dan Mace <dmace@redhat.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@ironcladlou ironcladlou changed the title fix(aws): isolate NewSession tests from ambient AWS env vars NO-JIRA: isolate NewSession tests from ambient AWS env vars Jul 2, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@ironcladlou: This pull request explicitly references no jira issue.

Details

In response to this:

Summary

  • Clear AWS_PROFILE and AWS_DEFAULT_PROFILE in TestNewSession setup so config.LoadDefaultConfig doesn't pick up the host's profile and fail silently (the error is discarded on util.go:116, producing a zero-value config with empty Region and nil Credentials)

Test plan

  • go test ./cmd/infra/aws/util/ -run TestNewSession -v passes with AWS_PROFILE set

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 2, 2026
@coderabbitai

coderabbitai Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor
📝 Walkthrough

Walkthrough

The change modifies a test file for AWS session creation. In TestNewSession, the environment variable clearing logic used before testing credential-file loading is expanded to clear a broader set of AWS-related environment variables, including additional credential, config, profile, and token variables, across two test cases.

Changes

Compact Metadata

  • Type: Test modification
  • Scope: Single test file
  • Files affected: 1

Related Issues: None specified

Related PRs: None specified

Suggested Labels: test, aws

Suggested Reviewers: None specified

Poem
A rabbit hopped through env vars small,
Clearing tokens, profiles, one and all,
So credentials from files could truly shine,
No stray env var crossing the line.
Test cases clean, the burrow's just fine. 🐰

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed No Ginkgo titles were added; all t.Run names are static strings with no dynamic data or generated identifiers.
Test Structure And Quality ✅ Passed Not applicable: these are plain Go subtests, not Ginkgo; the PR only broadens env cleanup in setup and adds no timeout/cleanup issues.
Topology-Aware Scheduling Compatibility ✅ Passed Only cmd/infra/aws/util/util_test.go changed, adding AWS env cleanup in tests; no scheduling manifests or topology constraints were introduced.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed Not applicable: this PR only changes Go unit tests in cmd/infra/aws/util/util_test.go, with no Ginkgo e2e tests or network assumptions.
No-Weak-Crypto ✅ Passed Changed test/setup code only clears AWS env vars; targeted scans of cmd/infra/aws/util found no MD5/SHA1/DES/RC4/3DES/Blowfish/ECB or secret-compare usage.
Container-Privileges ✅ Passed Only cmd/infra/aws/util/util_test.go changed; it’s a unit test and contains no container/K8s privilege settings.
No-Sensitive-Data-In-Logs ✅ Passed Only test env cleanup changed; no new logging or print paths expose passwords, tokens, PII, or host/customer data.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: isolating NewSession tests from ambient AWS environment variables.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@openshift-ci openshift-ci Bot requested review from cblecker and jparrill July 2, 2026 20:19
@openshift-ci openshift-ci Bot added the area/cli Indicates the PR includes changes for CLI label Jul 2, 2026
@openshift-ci

openshift-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: ironcladlou
Once this PR has been reviewed and has the lgtm label, please assign enxebre for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added area/platform/aws PR/issue for AWS (AWSPlatform) platform and removed do-not-merge/needs-area labels Jul 2, 2026
@ironcladlou

Copy link
Copy Markdown
Contributor Author

Without this, AWS environment variables interact with and break the test execution:

      --- FAIL: TestNewSession/When_given_credentials_file,_it_should_load_config_from_file (0.00s)
          util_test.go:60: Expected region us-west-2, got
      --- FAIL: TestNewSession/When_given_credentials_file,_it_should_read_credentials_from_the_file_not_from_env_vars (0.00s)
  panic: runtime error: invalid memory address or nil pointer dereference [recovered, repanicked]
  [signal SIGSEGV: segmentation violation code=0x2 addr=0x18 pc=0x1049f3508]

  goroutine 115 [running]:
  testing.tRunner.func1.2({0x105041140, 0x10600bb90})
          /Users/dmace/.local/share/mise/installs/go/1.25.7/src/testing/testing.go:1872 +0x2b4
  testing.tRunner.func1()
          /Users/dmace/.local/share/mise/installs/go/1.25.7/src/testing/testing.go:1875 +0x460
  panic({0x105041140?, 0x10600bb90?})
          /Users/dmace/.local/share/mise/installs/go/1.25.7/src/runtime/panic.go:783 +0x120
  github.com/openshift/hypershift/cmd/infra/aws/util.TestNewSession.func5(0xc000b256c0, 0xc000a4a908)
          /Users/dmace/Projects/hypershift-tmp/cmd/infra/aws/util/util_test.go:90 +0x48
  github.com/openshift/hypershift/cmd/infra/aws/util.TestNewSession.func8(0xc000b256c0)
          /Users/dmace/Projects/hypershift-tmp/cmd/infra/aws/util/util_test.go:157 +0x2dc
  testing.tRunner(0xc000b256c0, 0xc0002d6b20)
          /Users/dmace/.local/share/mise/installs/go/1.25.7/src/testing/testing.go:1934 +0x168
  created by testing.(*T).Run in goroutine 80
          /Users/dmace/.local/share/mise/installs/go/1.25.7/src/testing/testing.go:1997 +0x6e4
  FAIL    github.com/openshift/hypershift/cmd/infra/aws/util      4.498s

@ironcladlou

Copy link
Copy Markdown
Contributor Author

/verified with manual testing

@openshift-ci-robot

Copy link
Copy Markdown

@ironcladlou: The /verified command must be used with one of the following actions: by, later, remove, or bypass. See https://docs.ci.openshift.org/docs/architecture/jira/#premerge-verification for more information.

Details

In response to this:

/verified with manual testing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@ironcladlou

Copy link
Copy Markdown
Contributor Author

/verified by manual testing

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 2, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@ironcladlou: This PR has been marked as verified by manual testing.

Details

In response to this:

/verified by manual testing

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci

openshift-ci Bot commented Jul 2, 2026

Copy link
Copy Markdown
Contributor

@ironcladlou: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@codecov

codecov Bot commented Jul 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 35.92%. Comparing base (654bad7) to head (8a68a38).
⚠️ Report is 1127 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8911      +/-   ##
==========================================
+ Coverage   26.56%   35.92%   +9.36%     
==========================================
  Files        1087     1137      +50     
  Lines      105042   115488   +10446     
==========================================
+ Hits        27902    41489   +13587     
+ Misses      74731    71101    -3630     
- Partials     2409     2898     +489     
Flag Coverage Δ
cmd-support 34.08% <ø> (?)
cpo-hostedcontrolplane 45.31% <ø> (?)
cpo-other 45.10% <ø> (?)
hypershift-operator 50.71% <ø> (?)
other 12.54% <ø> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@hypershift-jira-solve-ci

Copy link
Copy Markdown

Test Failure Analysis Complete

Job Information

  • Prow Job: Red Hat Konflux / hypershift-operator-main-enterprise-contract / hypershift-operator-main
  • Build ID: check-run 84869577133 (pipeline: hypershift-operator-main-enterprise-contract-7j9fw)
  • Second Job: Red Hat Konflux / hypershift-operator-enterprise-contract / hypershift-operator-main
  • Second Build ID: check-run 84869576355 (pipeline: hypershift-operator-enterprise-contract-crgb9)
  • Snapshot: hypershift-operator-20260702-201819-000
  • PR: #8911 — NO-JIRA: isolate NewSession tests from ambient AWS env vars
  • Commit: 8a68a38c60c3422426a3aad81bf4b4439ace2515

Test Failure Analysis

Error

Integration test for component hypershift-operator-main snapshot
hypershift-operator-20260702-201819-000 and scenario
hypershift-operator-main-enterprise-contract has failed

verify task: 258 success(es), 6 warning(s), 64 failure(s)

Summary

These two Konflux Enterprise Contract (EC) failures are not caused by the PR's code change and are not reproducible across other PRs. The PR only modifies cmd/infra/aws/util/util_test.go (+4/−2 lines), adding AWS_PROFILE and AWS_DEFAULT_PROFILE to the list of environment variables cleared during test setup. Since _test.go files are excluded from Go binary builds, this change has zero effect on the built container image. The EC verify task reported 64 failures against snapshot hypershift-operator-20260702-201819-000, while every other recent PR (8908–8915, excluding cancelled 8913) consistently passes the same EC scenarios with 264 successes and 34 warnings. This is a Konflux infrastructure transient failure specific to this snapshot's EC evaluation.

Root Cause

The root cause is a transient Konflux Enterprise Contract verification failure affecting only snapshot hypershift-operator-20260702-201819-000. The evidence points to an infrastructure-level issue during the EC evaluation window (20:18–20:46 UTC on July 2, 2026):

  1. Anomalous check count: Passing PRs consistently show 298 total EC checks (264 success + 34 warning). PR NO-JIRA: isolate NewSession tests from ambient AWS env vars #8911 shows 328 total checks (258 success + 6 warning + 64 failure) — 30 additional checks appeared, and 34 previously-warning checks shifted to a mix of warnings and failures. This is consistent with the EC evaluator receiving an incomplete or inconsistent policy/attestation state.

  2. Timing anomaly: The hypershift-operator-enterprise-contract scenario took 28 minutes to complete (vs the typical ~25–52 seconds for the verify task itself), suggesting infrastructure delays or retries during evaluation.

  3. Scope isolation: Both failing EC scenarios evaluate the same snapshot but are independent pipeline runs. Both fail with identical 258/6/64 patterns, confirming the issue is with the snapshot's attestation state, not the policy configuration.

  4. Concurrent PRs unaffected: PR CNTRLPLANE-3626: feat(ignition-server, ignition-server-proxy): inject centralized TLS configuration #8910 (Konflux suite started at 20:34, during NO-JIRA: isolate NewSession tests from ambient AWS env vars #8911's run) passed with the normal 264/34 pattern on the same EC scenarios, ruling out a global policy change or platform-wide outage.

The most likely mechanism is that Tekton Chains did not complete SLSA provenance attestation for this specific build before the EC verification tasks attempted to validate it, or there was a transient registry/storage issue accessing the attestation artifacts for this snapshot. This is a known class of Konflux transient failures.

Recommendations
  1. Re-trigger the Konflux checks: Push an empty commit or close/reopen the PR to trigger a fresh Konflux build and EC evaluation. The new snapshot will get a fresh attestation and should pass.

    git commit --allow-empty -m "chore: re-trigger Konflux checks"
    git push
  2. No code changes needed: The PR's change is correct and test-only. The EC failures are unrelated to the code change.

  3. This is not a blocking issue for merge: The enterprise-contract checks are evaluating the Konflux-built image's supply chain provenance, not the code quality. The PR's actual CI checks (unit tests, lint, verify, codespell, envtest) all passed.

  4. If the failure recurs after re-trigger: Report to the Konflux team (Red Hat AppStudio / Stonesoup) with the pipeline run names (hypershift-operator-main-enterprise-contract-7j9fw, hypershift-operator-enterprise-contract-crgb9) for investigation of the attestation pipeline.

Evidence
Evidence Detail
PR code change cmd/infra/aws/util/util_test.go only (+4/−2 lines, test-only)
Build status All build tasks succeeded (hypershift-operator-main-on-pull-request-gddwt)
EC result (PR #8911) 258 success, 6 warning, 64 failure (328 total)
EC result (PR #8910) 264 success, 34 warning, 0 failure (298 total) — same scenarios
EC result (PR #8912) 264 success, 34 warning, 0 failure (298 total) — same scenarios
EC result (PR #8914) 264 success, 34 warning, 0 failure (298 total) — same scenarios
EC result (PR #8915) 264 success, 34 warning, 0 failure (298 total) — same scenarios
EC result (PR #8908) 264 success, 34 warning, 0 failure (298 total) — same scenarios
EC result (PR #8909) 264 success, 34 warning, 0 failure (298 total) — same scenarios
Unique to #8911 Only PR with failure conclusion; all 7 other checked PRs pass
Snapshot affected hypershift-operator-20260702-201819-000
mce-217 EC on same PR Passed (neutral/warning) — only hypershift-operator EC scenarios failed
GitHub Actions checks All 15 GitHub Actions checks passed (tests, lint, verify, codespell, envtest)
Timing EC ran 20:18–20:46 UTC; concurrent PR #8910 EC ran 20:34–21:04 and passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/cli Indicates the PR includes changes for CLI area/platform/aws PR/issue for AWS (AWSPlatform) platform jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants