Skip to content

CNTRLPLANE-3562, CNTRLPLANE-3563: test(healthcheck): add unit tests for AWS identity provider#8829

Open
mgencur wants to merge 4 commits into
openshift:mainfrom
mgencur:CNTRLPLANE-3562_idp_deletion
Open

CNTRLPLANE-3562, CNTRLPLANE-3563: test(healthcheck): add unit tests for AWS identity provider#8829
mgencur wants to merge 4 commits into
openshift:mainfrom
mgencur:CNTRLPLANE-3562_idp_deletion

Conversation

@mgencur

@mgencur mgencur commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

What this PR does / why we need it:

Adds unit tests for the AWS identity provider deletion path and fixes two production code issues discovered during review.

Tests:

  • TestCleanupAWSOIDCBucketData — S3 OIDC cleanup, error handling, NoSuchBucket tolerance, partial deletion failures
  • TestDeleteAWSEndpointServices — CPO finalizer removal on invalid credentials and expired grace period
  • TestDeleteOrphanedMachines — AWSMachine finalizer cleanup based on credential status
  • TestGetCredentialStatus — improved with idiomatic patterns and helper extraction
  • TestAWSHealthCheckIdentityProviderConditionLogic - test cases covering DescribeVpcEndpoints error paths (WebIdentityErr, other API errors, non-API errors) and the success path that were previously untested

Fixes:

  • S3 DeleteObjects output was discarded — partial deletion failures (per-object errors in output.Errors) were silently treated as success, potentially leaving orphaned S3 objects with no retry path
  • deleteAWSEndpointServices log message said "no valid aws credentials" even when credentials were valid but the grace period had expired

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3562
Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3563

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
    • Improved AWS identity provider health checks so status updates are more accurate across unavailable APIs, missing credentials, and AWS API failures.
    • Cleanup of AWS OIDC bucket data now correctly reports partial delete failures instead of treating them as successful.
    • Deletion of AWS endpoint services now shows clearer reasons in logs when finalizers are removed.

@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jun 25, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 25, 2026

Copy link
Copy Markdown

@mgencur: This pull request references CNTRLPLANE-3562 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Extract validateAWSIdentityProvider from awsHealthCheckIdentityProvider to enable testing with a mock EC2 client. Add test cases covering DescribeVpcEndpoints error paths (WebIdentityErr, other API errors, non-API errors) and the success path that were previously untested.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3562

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added do-not-merge/needs-area area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release labels Jun 25, 2026
@openshift-ci openshift-ci Bot requested review from clebs and devguyio June 25, 2026 09:06
@openshift-ci openshift-ci Bot added the area/platform/aws PR/issue for AWS (AWSPlatform) platform label Jun 25, 2026
@openshift-ci

openshift-ci Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: mgencur
Once this PR has been reviewed and has the lgtm label, please assign csrwng for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@coderabbitai

coderabbitai Bot commented Jun 25, 2026

Copy link
Copy Markdown
Contributor

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 26dddd04-7449-4338-ad64-1652019fd0fa

📥 Commits

Reviewing files that changed from the base of the PR and between 7babf03 and 5293291.

📒 Files selected for processing (4)
  • hypershift-operator/controllers/hostedcluster/aws_endpoint_services_test.go
  • hypershift-operator/controllers/hostedcluster/aws_oidc_test.go
  • hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
  • hypershift-operator/controllers/hostedcluster/internal/platform/aws/aws_test.go
🚧 Files skipped from review as they are similar to previous changes (3)
  • hypershift-operator/controllers/hostedcluster/aws_oidc_test.go
  • hypershift-operator/controllers/hostedcluster/aws_endpoint_services_test.go
  • hypershift-operator/controllers/hostedcluster/internal/platform/aws/aws_test.go

📝 Walkthrough

Walkthrough

AWS identity-provider health checking now uses a shared EC2-backed helper. Hostedcluster AWS cleanup now logs endpoint-service finalizer removal with a credential-based reason, treats partial S3 DeleteObjects failures as errors, and adds tests for endpoint-service deletion, OIDC bucket cleanup, credential status handling, and orphaned-machine cleanup.

🚥 Pre-merge checks | ✅ 11
✅ Passed checks (11 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title is clearly related to the PR and captures the main testing focus, even though it omits the additional production fixes.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names ✅ Passed All added/updated test titles are static strings; no dynamic data, timestamps, UUIDs, or generated identifiers appear in titles.
Test Structure And Quality ✅ Passed PASS: the added tests are table-driven unit tests, not Ginkgo; they use fake clients, no long waits, and no unmanaged resources.
Topology-Aware Scheduling Compatibility ✅ Passed Touched controller code only changes AWS health-check and cleanup logic; no nodeSelector, affinity, spread, replica, or PDB scheduling constraints were added.
Ipv6 And Disconnected Network Test Compatibility ✅ Passed The PR adds only Go unit tests with fake clients/mocks; no Ginkgo e2e tests, hardcoded IPv4, or external connectivity assumptions were found.
No-Weak-Crypto ✅ Passed Touched AWS/S3 and test code adds no MD5/SHA1/DES/RC4/ECB/custom crypto, and no secret/token comparisons were introduced.
Container-Privileges ✅ Passed PR only changes Go controllers/tests; scans of touched files found no hostPID/hostNetwork/hostIPC, allowPrivilegeEscalation, SYS_ADMIN, or privileged container settings.
No-Sensitive-Data-In-Logs ✅ Passed No new logs expose secrets or PII; the only changed log line adds a reason while logging only resource name and endpoint-id.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
control-plane-operator/controllers/healthcheck/aws_test.go (1)

159-178: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Helper-direct branch leaves the entrypoint wiring untested.

For all KAS-True cases the test calls validateAWSIdentityProvider directly and never exercises awsHealthCheckIdentityProviderGetEC2Client(ctx). That's the seam most likely to regress (see the typed-nil concern in aws.go), so the EC2-client acquisition path stays uncovered. Consider a follow-up that drives the real entrypoint, or at minimum a comment documenting that the helper is tested in isolation by design.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@control-plane-operator/controllers/healthcheck/aws_test.go` around lines 159
- 178, The KAS-True branch in the AWS healthcheck test bypasses the real
entrypoint and only exercises validateAWSIdentityProvider, leaving
awsHealthCheckIdentityProvider and GetEC2Client(ctx) untested. Update the test
in aws_test.go to cover the actual awsHealthCheckIdentityProvider path for
KAS-True cases, or add an explicit note if helper-only coverage is intentional,
so the EC2 client acquisition seam is verified alongside
validateAWSIdentityProvider and aws.go behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@control-plane-operator/controllers/healthcheck/aws_test.go`:
- Around line 159-178: The KAS-True branch in the AWS healthcheck test bypasses
the real entrypoint and only exercises validateAWSIdentityProvider, leaving
awsHealthCheckIdentityProvider and GetEC2Client(ctx) untested. Update the test
in aws_test.go to cover the actual awsHealthCheckIdentityProvider path for
KAS-True cases, or add an explicit note if helper-only coverage is intentional,
so the EC2 client acquisition seam is verified alongside
validateAWSIdentityProvider and aws.go behavior.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: 4f062843-bcb5-4f51-8a7a-538fff759525

📥 Commits

Reviewing files that changed from the base of the PR and between 652dbf2 and 7445f05.

📒 Files selected for processing (2)
  • control-plane-operator/controllers/healthcheck/aws.go
  • control-plane-operator/controllers/healthcheck/aws_test.go

@codecov

codecov Bot commented Jun 25, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 22.22222% with 7 lines in your changes missing coverage. Please review.
✅ Project coverage is 43.21%. Comparing base (9acec47) to head (5293291).
⚠️ Report is 44 commits behind head on main.

Files with missing lines Patch % Lines
...trollers/hostedcluster/hostedcluster_controller.go 14.28% 5 Missing and 1 partial ⚠️
...trol-plane-operator/controllers/healthcheck/aws.go 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #8829      +/-   ##
==========================================
+ Coverage   42.95%   43.21%   +0.26%     
==========================================
  Files         766      766              
  Lines       94722    94884     +162     
==========================================
+ Hits        40688    41008     +320     
+ Misses      51223    51021     -202     
- Partials     2811     2855      +44     
Files with missing lines Coverage Δ
...trol-plane-operator/controllers/healthcheck/aws.go 97.33% <50.00%> (+57.60%) ⬆️
...trollers/hostedcluster/hostedcluster_controller.go 54.10% <14.28%> (+2.11%) ⬆️

... and 14 files with indirect coverage changes

Flag Coverage Δ
cmd-support 36.42% <ø> (+0.01%) ⬆️
cpo-hostedcontrolplane 45.27% <ø> (+0.39%) ⬆️
cpo-other 45.43% <50.00%> (+0.49%) ⬆️
hypershift-operator 53.48% <14.28%> (+0.42%) ⬆️
other 31.69% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Extract validateAWSIdentityProvider from awsHealthCheckIdentityProvider
to enable testing with a mock EC2 client. Add test cases covering
DescribeVpcEndpoints error paths (WebIdentityErr, other API errors,
non-API errors) and the success path that were previously untested.

Signed-off-by: Martin Gencur <mgencur@redhat.com>
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mgencur mgencur force-pushed the CNTRLPLANE-3562_idp_deletion branch from 7445f05 to efbc004 Compare June 25, 2026 09:17
@mgencur mgencur changed the title CNTRLPLANE-3562: test(healthcheck): add unit tests for AWS identity provider validation CNTRLPLANE-3562, CNTRLPLANE-3563: test(healthcheck): add unit tests for AWS identity provider Jun 26, 2026
@openshift-ci-robot

openshift-ci-robot commented Jun 26, 2026

Copy link
Copy Markdown

@mgencur: This pull request references CNTRLPLANE-3562 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

This pull request references CNTRLPLANE-3563 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Extract validateAWSIdentityProvider from awsHealthCheckIdentityProvider to enable testing with a mock EC2 client. Add test cases covering DescribeVpcEndpoints error paths (WebIdentityErr, other API errors, non-API errors) and the success path that were previously untested.

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-3562

Special notes for your reviewer:

Checklist:

  • Subject and description added to both, commit and PR.
  • Relevant issues have been referenced.
  • This change includes docs.
  • This change includes unit tests.

Summary by CodeRabbit

  • Bug Fixes
  • Improved AWS identity provider health reporting so status checks are more accurate across unavailable services, missing AWS clients, AWS API failures, and successful endpoint validation.
  • Updated the health-check flow to handle additional AWS error scenarios consistently, resulting in clearer and more reliable condition status and reasons.
  • Tests
  • Expanded AWS identity provider condition logic coverage with additional scenarios and EC2 API mocking.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot added the area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release label Jun 26, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🧹 Nitpick comments (2)
hypershift-operator/controllers/hostedcluster/aws_oidc_test.go (1)

45-46: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Rename the ctrl mock parameter to avoid shadowing the controller-runtime alias.

Each setupS3Mock closure introduces ctrl *gomock.Controller, which shadows the imported ctrl package alias used later on Line 146. Renaming the parameter keeps the test less error-prone and matches the repo's Go guideline. As per coding guidelines, **/!(*.pb).go: Avoid variable shadowing in Go files.

Also applies to: 64-65, 85-86, 104-105

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hypershift-operator/controllers/hostedcluster/aws_oidc_test.go` around lines
45 - 46, The setupS3Mock closures use a parameter named ctrl that shadows the
imported controller-runtime alias used elsewhere in aws_oidc_test.go. Rename
each closure parameter in the affected test cases to a distinct name, and update
the NewMockS3API call within those closures accordingly so the gomock controller
argument is still passed through without variable shadowing.

Source: Coding guidelines

hypershift-operator/controllers/hostedcluster/aws_endpoint_services_test.go (1)

41-81: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a negative case covering the grace-period skip path.

Both table entries exercise finalizer removal. There's no case for GetCredentialStatus == Valid within awsEndpointDeletionGracePeriod, where deleteAWSEndpointServices hits the early continue and the CPO finalizer must be preserved (and pending stays true). Adding it locks in the grace-period guard against regressions.

{
    name: "When endpoint is deleting with valid creds inside grace period, it should preserve CPO finalizer",
    hc:   hostedClusterWithCredentialConditions(metav1.ConditionTrue, metav1.ConditionTrue),
    endpoints: []hyperv1.AWSEndpointService{
        {
            ObjectMeta: metav1.ObjectMeta{
                Name:              "ep-1",
                Namespace:         namespace,
                DeletionTimestamp: &metav1.Time{Time: time.Now().Add(-1 * time.Minute)},
                Finalizers:        []string{cpoFinalizer},
            },
        },
    },
    expectPending:          true,
    expectFinalizerRemoved: false,
},

Note: the current assertion block only verifies removal when expectFinalizerRemoved is true; you'll want a matching branch to assert the finalizer is still present for this case.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@hypershift-operator/controllers/hostedcluster/aws_endpoint_services_test.go`
around lines 41 - 81, Add a negative test case in aws_endpoint_services_test for
the grace-period skip path in deleteAWSEndpointServices: when
GetCredentialStatus is Valid but the AWSEndpointService deletion timestamp is
still inside awsEndpointDeletionGracePeriod, the CPO finalizer should be
preserved and pending should remain true. Update the table-driven test alongside
the existing hostedClusterWithCredentialConditions cases, and extend the final
assertion logic to handle expectFinalizerRemoved == false by verifying the
cpoFinalizer is still present on the endpoint.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In
`@hypershift-operator/controllers/hostedcluster/internal/platform/aws/aws_test.go`:
- Around line 361-377: TestDeleteOrphanedMachines is missing coverage for the
valid-creds skip-cleanup path because the finalizer assertion only runs for
machines without a DeletionTimestamp. Update the test case for “When credentials
are valid, it should skip cleanup” so it actually verifies that test-finalizer
remains on a non-deleting AWSMachine, either by removing the DeletionTimestamp
guard in the finalizer check or by adding a machine that is not marked for
deletion. Use the existing TestDeleteOrphanedMachines,
tc.expectFinalizersCleared, and machineList checks to locate the assertion
logic.

---

Nitpick comments:
In `@hypershift-operator/controllers/hostedcluster/aws_endpoint_services_test.go`:
- Around line 41-81: Add a negative test case in aws_endpoint_services_test for
the grace-period skip path in deleteAWSEndpointServices: when
GetCredentialStatus is Valid but the AWSEndpointService deletion timestamp is
still inside awsEndpointDeletionGracePeriod, the CPO finalizer should be
preserved and pending should remain true. Update the table-driven test alongside
the existing hostedClusterWithCredentialConditions cases, and extend the final
assertion logic to handle expectFinalizerRemoved == false by verifying the
cpoFinalizer is still present on the endpoint.

In `@hypershift-operator/controllers/hostedcluster/aws_oidc_test.go`:
- Around line 45-46: The setupS3Mock closures use a parameter named ctrl that
shadows the imported controller-runtime alias used elsewhere in
aws_oidc_test.go. Rename each closure parameter in the affected test cases to a
distinct name, and update the NewMockS3API call within those closures
accordingly so the gomock controller argument is still passed through without
variable shadowing.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: c9e6fbef-8bf4-42f4-9687-f666ee7ac93b

📥 Commits

Reviewing files that changed from the base of the PR and between efbc004 and 7babf03.

📒 Files selected for processing (4)
  • hypershift-operator/controllers/hostedcluster/aws_endpoint_services_test.go
  • hypershift-operator/controllers/hostedcluster/aws_oidc_test.go
  • hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go
  • hypershift-operator/controllers/hostedcluster/internal/platform/aws/aws_test.go

mgencur and others added 2 commits June 29, 2026 08:37
Add tests for the AWS identity provider deletion path (OCP-60484):
- TestCleanupAWSOIDCBucketData: S3 cleanup, error handling, NoSuchBucket tolerance
- TestDeleteAWSEndpointServices: CPO finalizer removal for invalid creds and expired grace period
- TestDeleteOrphanedMachines: AWSMachine finalizer cleanup based on credential status

These tests exercise the condition chain: OIDC upload fails → no finalizer → cleanup no-op →
 GetCredentialStatus=Invalid → deleteAWSEndpointServices removes CPO finalizer → DeleteOrphanedMachines clears
 AWSMachine finalizers → deletion proceeds.

Improve existing TestGetCredentialStatus: standardize test names to
"it should" convention, replace &[]T{v}[0] with ptr.To(), and extract
hostedClusterWithCredentialConditions helper to reduce duplication.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… endpoint service log

Differentiate between "no valid credentials" and "grace period expired"
when logging CPO finalizer removal for AWSEndpointService resources.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mgencur mgencur force-pushed the CNTRLPLANE-3562_idp_deletion branch from 7babf03 to 8cefa9b Compare June 29, 2026 06:37
The S3 DeleteObjects API can return (output, nil) with per-object errors
in output.Errors, meaning some objects failed to delete while the call
itself succeeded. Previously the output was discarded, so partial
failures silently removed the finalizer, leaving orphaned S3 objects
with no retry path.

Capture the output and return an error on partial failure, keeping the
finalizer in place so the controller retries on the next reconcile.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@mgencur mgencur force-pushed the CNTRLPLANE-3562_idp_deletion branch from 8cefa9b to 5293291 Compare June 29, 2026 06:40
@hypershift-jira-solve-ci

hypershift-jira-solve-ci Bot commented Jun 29, 2026

Copy link
Copy Markdown

Now I have all the data. Let me verify one detail about the exact check run ID:

Test Failure Analysis Complete

Job Information

  • Prow Job: codecov/patch (Codecov GitHub Check — not a Prow CI job)
  • Build ID: Check Run 83994685111
  • PR: #8829CNTRLPLANE-3562, CNTRLPLANE-3563: test(healthcheck): add unit tests for AWS identity provider
  • Commit: 5293291b91e768e25538d5534dba2d5f7517fb11
  • Branch: CNTRLPLANE-3562_idp_deletionmain

Test Failure Analysis

Error

codecov/patch: 22.22% of diff hit (target 42.95%)
Patch coverage: 22.22222% with 7 lines in your changes missing coverage.

Files with missing lines:
  hostedcluster_controller.go — 14.28% patch coverage (5 missing + 1 partial)
  healthcheck/aws.go          — 50.00% patch coverage (1 missing)

Summary

The codecov/patch check fails because only 2 of 9 new executable lines (22.22%) in the two modified production files are covered by tests, falling well below the 42.95% patch coverage target. All other CI checks pass. The coverage gap stems from how the tests are structured: the new tests call validateAWSIdentityProvider directly (bypassing the wrapper that delegates to it), the deleteAWSEndpointServices tests don't exercise the newly added reason/GetCredentialStatus branching logic, and the refactored cleanupOIDCBucketData lines aren't fully reached. Notably, the PR actually improves overall project coverage from 42.95% to 43.21% (+0.26%), so the failure is specifically about patch-level (diff) coverage, not project-level regression.

Root Cause

The failure is caused by insufficient test coverage on newly added/modified executable lines in two production files:

File 1: control-plane-operator/controllers/healthcheck/aws.go (50% — 1 of 2 lines uncovered)

The PR extracted validateAWSIdentityProvider from awsHealthCheckIdentityProvider to enable mocking. The new test cases for KAS-available scenarios call validateAWSIdentityProvider directly, which covers the function signature line. However, the delegation line return validateAWSIdentityProvider(ctx, hcp, ec2Client) inside awsHealthCheckIdentityProvider is never hit because:

  • KAS-unavailable tests call awsHealthCheckIdentityProvider but return before reaching the delegation line
  • KAS-available tests call validateAWSIdentityProvider directly, bypassing the wrapper entirely

File 2: hypershift-operator/controllers/hostedcluster/hostedcluster_controller.go (14.28% — 6 of 7 lines uncovered)

Lines 3549–3554 (deleteAWSEndpointServices reason logic): The PR adds a reason variable with a conditional branch based on platformaws.GetCredentialStatus(hc). While the test in aws_endpoint_services_test.go exercises finalizer removal, the test cases set up conditions where the code reaches the finalizer-removal outcome via the shouldRemoveFinalizer logic — but the specific new lines computing the reason string and calling GetCredentialStatus are not covered by the unit test instrumentation. The 2 test cases provided only cover the CredentialStatusInvalid and CredentialStatusValid credential states superficially — they validate the side effect (finalizer removal, pending status) but don't result in the coverage tool seeing the reason assignment lines as executed, likely because the test flow through the for loop/if chain reaches a different branch.

Lines 4855–4867 (cleanupOIDCBucketData refactored output handling): The PR refactors the DeleteObjects call from an inline if to a separate output, err := assignment and adds a new else if output.Errors branch. While the test does include a "partial failure" case, the refactored variable assignment lines and the else if conditional line appear as new uncovered lines to Codecov.

Recommendations
  1. Cover the delegation line in aws.go: Add one test case that calls awsHealthCheckIdentityProvider (not validateAWSIdentityProvider directly) with KAS condition set to True. Since hostedcontrolplane.GetEC2Client(ctx) returns nil in test context, this will hit the "EC2 client not available" path through the full call chain, covering the return validateAWSIdentityProvider(...) line.

  2. Cover the reason branching in deleteAWSEndpointServices: Ensure the test cases actually execute through the code path at lines 3549–3554. Add debug logging or a direct assertion on the log output to confirm the reason assignment is reached. The "valid creds past grace period" test case should hit reason = "deletion grace period expired" — verify the shouldRemoveFinalizer function returns the correct signal that routes through the reason assignment rather than the continue on an earlier branch.

  3. Cover the refactored cleanupOIDCBucketData lines: The existing "When cleanup succeeds" test case should cover the happy-path through output, err := and the else if (when output.Errors is empty). If Codecov still shows these as uncovered, it may be a flag/test-suite segregation issue — confirm the hypershift-operator flag runs these tests.

  4. Alternative — adjust Codecov configuration: Since this PR is a net-positive for coverage (+0.26% project-wide, +57.60% on aws.go), the team could consider adjusting the codecov.yml patch target, or marking this check as non-blocking. The codecov/patch threshold is a policy decision — the code quality and test quality are sound.

Evidence
Evidence Detail
Check Run ID 83994685111 — conclusion: failure
Patch Coverage 22.22% achieved vs 42.95% required (7 lines uncovered out of 9 new executable lines)
Project Coverage 43.21% (+0.26% vs base 9acec47) — project coverage improved
aws.go gap Line return validateAWSIdentityProvider(ctx, hcp, ec2Client) in wrapper function — tests bypass it by calling the extracted function directly
hostedcluster_controller.go gap 5 missing + 1 partial: reason variable + GetCredentialStatus branch (lines 3549–3554), output, err := refactor + output.Errors branch (lines 4855–4867)
Codecov bot comment "Patch coverage is 22.22222% with 7 lines in your changes missing coverage"
Other CI checks All pass — this is the only failure on the PR
PR net effect +162 lines, +320 hits, -202 misses — overall positive contribution to test coverage

@openshift-ci

openshift-ci Bot commented Jun 29, 2026

Copy link
Copy Markdown
Contributor

@mgencur: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release area/platform/aws PR/issue for AWS (AWSPlatform) platform jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants