Skip to content

fix(grafana): comment out grafana steps to unblock rollouts (ARO-28044)#5838

Merged
openshift-merge-bot[bot] merged 5 commits into
Azure:mainfrom
raelga:rael/aro-28044-grafana-mitigation
Jun 29, 2026
Merged

fix(grafana): comment out grafana steps to unblock rollouts (ARO-28044)#5838
openshift-merge-bot[bot] merged 5 commits into
Azure:mainfrom
raelga:rael/aro-28044-grafana-mitigation

Conversation

@raelga

@raelga raelga commented Jun 29, 2026

Copy link
Copy Markdown
Collaborator

ARO-28044

What

Comments out the Grafana rollout resources and steps that reconcile the shared Grafana instance and integrations:

  • grafanaWorkspaceIdLookup, grafana, and Grafana AFD permissions in global-infra.bicep
  • grafana-monitoring-reader and grafana-dashboards in the global pipelines
  • add-grafana-datasource in the regional pipeline

Why

Prod rollouts are currently blocked by the AME deployment-script storage policy used by the Grafana workspace lookup. This is a non-destructive mitigation: the existing Grafana instance and existing integrations remain in place, while EV2 stops reconciling Grafana until the proper reconcile fix lands in #5837 / ARO-28040.

Testing

  • cd config && make materialize
  • make validate-config-pipelines
  • /home/rael/bin/bicep build dev-infrastructure/templates/global-infra.bicep (passes with expected unused Grafana parameter/import warnings while Grafana is commented out)

No unit, integration, or E2E tests were added; this only disables EV2 Grafana rollout steps/resources.

Special notes for your reviewer

@jboll this is intended as a temporary ARO-28044 mitigation only. Each commented block has a TODO([ARO-28044](https://redhat.atlassian.net/browse/ARO-28044)): re-enable after [ARO-28040](https://redhat.atlassian.net/browse/ARO-28040) marker.

PR Checklist

  • PR is scoped to a single task (no mixed concerns)
  • Title follows Conventional Commits format
  • Summary explains the "Why" behind the change
  • Linked to relevant ticket/issue
  • Screenshots included (if graph/UI/metrics changes)
  • Self-reviewed the diff
  • CI/CD checks are passing (ignore Tide)
  • Draft PR used for WIP (if applicable)
  • Commit history is clean (rebased/squashed)
  • Tricky code blocks are commented
  • Specific reviewers tagged
  • All comment threads resolved before merge

If E2E tests are included:

  • E2E tests follow Principles of Good E2E Test Case Design
  • If new E2E use case is covered (via a new test or new check/verifier),
    demonstrate that the test is able to detect a defect/error and fail with
    proper error message and logs which communicates nature of the problem.

Copilot AI review requested due to automatic review settings June 29, 2026 09:30

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Temporarily disables EV2 reconciliation of the shared Managed Grafana instance and related integrations to unblock production rollouts impacted by Azure Policy restrictions on ARM deploymentScripts (ARO-28044). This is positioned as a non-destructive mitigation until the longer-term fix in ARO-28040 lands.

Changes:

  • Comments out the Grafana workspace lookup module, Grafana instance module, and AFD permission module wiring in global-infra.bicep.
  • Comments out Grafana subscription RBAC and dashboard reconciliation steps in the global pipelines.
  • Comments out the regional pipeline step that adds the Grafana datasource.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.

File Description
dev-infrastructure/templates/global-infra.bicep Disables Grafana module deployments (workspace lookup, instance creation, AFD permissions) via TODO-commented blocks.
dev-infrastructure/region-pipeline.yaml Disables the regional Grafana datasource reconciliation step.
dev-infrastructure/global-pipeline.yaml Disables Grafana monitoring-reader RBAC and dashboard reconciliation steps in the global pipeline.
dev-infrastructure/global-pipeline-stg.yaml Disables Grafana monitoring-reader RBAC and dashboard reconciliation steps in the staging global pipeline.

Comment thread dev-infrastructure/global-pipeline.yaml
Copilot AI review requested due to automatic review settings June 29, 2026 10:07

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 6 out of 6 changed files in this pull request and generated 2 comments.

Comment thread dev-infrastructure/region-pipeline.yaml Outdated
Comment thread dev-infrastructure/templates/global-infra.bicep
Copilot AI review requested due to automatic review settings June 29, 2026 10:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

@janboll janboll left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please do not delete more than needed to solve this

Comment thread dev-infrastructure/svc-pipeline.yaml
Comment thread dev-infrastructure/mgmt-pipeline.yaml
Comment thread dev-infrastructure/global-pipeline.yaml
Comment thread dev-infrastructure/global-pipeline.yaml
Comment thread dev-infrastructure/global-pipeline-stg.yaml
Comment thread dev-infrastructure/global-pipeline-stg.yaml

@janboll janboll left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

after discussion, we readd this after doing refactoring
/lgtm
/approve

@openshift-ci

openshift-ci Bot commented Jun 29, 2026

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: janboll, raelga

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

raelga added a commit to raelga/ARO-HCP that referenced this pull request Jun 29, 2026
The ARO-28044 mitigation (Azure#5838) commented out the grafana rollout and
reader steps to unblock pipelines. This change supersedes that mitigation
with a redesigned, deploymentScript-free grafana reconcile, so re-enable
grafana first by reverting the mitigation in full.
@openshift-merge-bot openshift-merge-bot Bot merged commit 8cf3517 into Azure:main Jun 29, 2026
17 checks passed
raelga added a commit to raelga/ARO-HCP that referenced this pull request Jun 29, 2026
The ARO-28044 mitigation (Azure#5838) commented out the grafana rollout and
reader steps to unblock pipelines. This change supersedes that mitigation
with a redesigned, deploymentScript-free grafana reconcile, so re-enable
grafana first by reverting the mitigation in full.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants