fix: Refactor grafana provisioning by janboll · Pull Request #5843 · Azure/ARO-HCP

janboll · 2026-06-29T13:20:03Z

https://redhat.atlassian.net/browse/ARO-28040

What

This change refactors grafana provisioning and management. Instead of bicep the lifecycle of grafana is managed via grafanactl.

Update to version 12, since version 11 is not available anymore and causes API errors.

Why

Ever since we can not use deployment scripts in bicep anymore, we can not run "side-effects", fetching input from outside the biceps managment scope. But since we for grafana need to lookup all the AMWs from all the subscriptions we need so programmatic way of doing this. Hence move this over to ARO-Tools/Grafanactl and manage the whole lifecycle using it. In that tool we can do the required API calls to fetch the AMWs.

Testing

This was manually tested using the also added Grafana pipeline.

Special notes for your reviewer

PR Checklist

If E2E tests are included:

E2E tests follow Principles of Good E2E Test Case Design
If new E2E use case is covered (via a new test or new check/verifier),
demonstrate that the test is able to detect a defect/error and fail with
proper error message and logs which communicates nature of the problem.

openshift-ci · 2026-06-29T13:20:15Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

Copilot

Pull request overview

This PR refactors Grafana provisioning in the dev-infrastructure pipelines by moving Grafana instance reconciliation out of Bicep and into a templatize pipeline step backed by grafanactl manage, while keeping Bicep responsible for role assignments and related outputs. It also bumps multiple tooling modules to a newer github.com/Azure/ARO-Tools revision that includes the new Grafana manage functionality.

Changes:

Add a new templatize pipeline step (GrafanaManageStep) and wire it into the pipeline runner.
Update global/stg global pipelines and add a dev-only Grafana pipeline plus supporting Bicep templates/params for Grafana role assignments.
Update output-global.bicep to read Grafana outputs from an existing Grafana resource; bump ARO-Tools deps across tooling/test modules.

Reviewed changes

Copilot reviewed 28 out of 38 changed files in this pull request and generated 8 comments.

Show a summary per file

File	Description
topology.yaml	Registers a new dev-only Grafana serviceGroup pipeline.
tooling/yamlwrap/go.mod	Bumps `ARO-Tools/tools/yamlwrap` dependency revision.
tooling/yamlwrap/go.sum	Dependency hash updates for the yamlwrap tool module.
tooling/templatize/pkg/pipeline/run.go	Adds execution support for `GrafanaManageStep`.
tooling/templatize/pkg/pipeline/grafana_reconcile.go	Implements the Grafana manage/reconcile step via `grafanactl` reconcile options.
tooling/templatize/go.mod	Bumps `ARO-Tools` module revisions and adds indirect deps needed by new code paths.
tooling/templatize/go.sum	Dependency hash updates corresponding to the new `ARO-Tools` revision.
tooling/secret-sync/go.mod	Bumps `ARO-Tools/tools/secret-sync` and related indirect deps.
tooling/secret-sync/go.sum	Dependency hash updates for secret-sync module.
tooling/pipeline-documentation/go.mod	Bumps `ARO-Tools/pipelines` revision.
tooling/pipeline-documentation/go.sum	Dependency hash updates for pipeline-documentation module.
tooling/helmtest/go.mod	Bumps `ARO-Tools` deps used by helmtest.
tooling/helmtest/go.sum	Dependency hash updates for helmtest module.
tooling/hcpctl/go.mod	Updates `cmdutils` and `armresourcegraph` dependency versions.
tooling/hcpctl/go.sum	Dependency hash updates for hcpctl module.
tooling/grafanactl/main.go	Adds the `manage` subcommand to the grafanactl CLI wrapper.
tooling/grafanactl/go.mod	Bumps grafanactl wrapper deps and adds resourcegraph indirect dep.
tooling/grafanactl/go.sum	Dependency hash updates for grafanactl wrapper module.
tooling/aro-hcp-exporter/go.mod	Updates `armresourcegraph` dependency version.
tooling/aro-hcp-exporter/go.sum	Dependency hash updates for aro-hcp-exporter module.
test/go.mod	Bumps `ARO-Tools` deps and adds resourcegraph indirect dep for tests module.
test/go.sum	Dependency hash updates for test module.
mgmt-agent/go.mod	Bumps `ARO-Tools/testutil` dependency revision.
mgmt-agent/go.sum	Dependency hash updates for mgmt-agent module.
dev-infrastructure/templates/output-global.bicep	Changes Grafana outputs to read from an existing Grafana resource; still outputs global MSI id.
dev-infrastructure/templates/grafana-roles.bicep	New template to apply Grafana-related role assignments (and optional AFD permissions).
dev-infrastructure/templates/global-infra.bicep	Removes Grafana provisioning (now handled elsewhere) and adds `globalMSIId` output; currently contains unresolved merge markers.
dev-infrastructure/modules/grafana/integration-lookup.bicep	Removes the prior deployment-script based integration lookup module.
dev-infrastructure/modules/grafana/instance.bicep	Converts Grafana resource to `existing` and limits module responsibilities to RBAC assignments.
dev-infrastructure/modules/grafana/grafanaIntegrationsLookup.ps1	Removes the prior PowerShell resource graph lookup script.
dev-infrastructure/grafana-pipeline.yaml	New dev-only pipeline for Grafana reconciliation and RBAC/monitoring-reader setup.
dev-infrastructure/global-pipeline.yaml	Adds Grafana manage + roles + monitoring-reader steps; currently contains unresolved merge markers.
dev-infrastructure/global-pipeline-stg.yaml	Adds Grafana manage + roles + monitoring-reader steps for STG; currently contains unresolved merge markers.
dev-infrastructure/configurations/grafana-roles.tmpl.bicepparam	New param file for grafana-roles template.
dev-infrastructure/configurations/grafana-roles-stg.tmpl.bicepparam	New STG param file variant for grafana-roles template.
dev-infrastructure/configurations/global-infra.tmpl.bicepparam	Removes Grafana-related params now that global-infra no longer provisions Grafana.
dev-infrastructure/configurations/global-infra-stg.tmpl.bicepparam	Removes Grafana-related params now that global-infra no longer provisions Grafana (STG variant).
config/config.yaml	Updates Grafana name/version values in defaults and dev cloud config (currently set to test/personal names).

+  - name: output
+    action: ARM
+    template: templates/output-global.bicep
+    parameters: configurations/output-global.tmpl.bicepparam
+    deploymentLevel: ResourceGroup
+    outputOnly: true
+  - name: grafana-reconcile


janboll · 2026-06-30T13:12:18Z

This requires: Azure/ARO-Tools#260

Copilot

Pull request overview

Copilot reviewed 34 out of 44 changed files in this pull request and generated 2 comments.

+resource grafana 'Microsoft.Dashboard/grafana@2024-10-01' existing = {
+  name: grafanaName
+}
+
+output grafanaResourceId string = grafana.id
+output grafanaPrincipalId string = grafana.identity.principalId


Copilot

Pull request overview

Copilot reviewed 34 out of 44 changed files in this pull request and generated 1 comment.

This commit re-enables grafana management, previously disabled. With this change bicep steps to manage grafana are removed. This is moved over into grafanactl. The goal is to remove the need to provide input into bicep (AMW[].id, for creating integrations). Also adding a grafana pipeline that can be used to test out grafana changes in dev.

Also: update grafana version to version 12, since 11 is deprecated

Copilot

Pull request overview

Copilot reviewed 33 out of 43 changed files in this pull request and generated 1 comment.

Copilot

Pull request overview

Copilot reviewed 35 out of 45 changed files in this pull request and generated 1 comment.

+    identityFrom:
+      resourceGroup: singleton
+      step: output
+      name: globalMSIId


raelga

Nice refactor — grafanactl is clearly the right direction, and dropping the PowerShell AMW lookup is exactly what ARO-28038/SFI wants. Two things I think are correctness blockers once this lands on current main (which now includes #5847 / AROSLSRE-1362):

1. STG-global V2 reconcile pins the wrong version → regresses AROSLSRE-1362.
In global-pipeline-stg.yaml, the V2 grafana-reconcile step uses:

grafanaName: "{{ .stgGlobalV2.grafanaName }}"         # V2 ✔
majorVersion: "{{ .monitoring.grafanaMajorVersion }}" # shared ✘ — should be stgGlobalV2

#5847 added a V2-only stgGlobalV2.grafanaMajorVersion: "13" specifically because the new V2 workspace can't be born on the shared value (Azure retired v11 for new-workspace creation on 2026-06-15; only 12/13 accepted). Sourcing majorVersion from monitoring.grafanaMajorVersion here bypasses that pin — if the shared value resolves to 11 for stg, the new workspace hits the original GrafanaMajorVersionNotSupported again. Suggest pointing this step's majorVersion at {{ .stgGlobalV2.grafanaMajorVersion }}. (This is the re-point #5847's PR body flagged would be needed when creation moved to grafanactl.)

2. Dangling bicep param after rebase — and CI won't catch it.
This PR removes param grafanaMajorVersion from templates/global-infra.bicep, but #5847 (merged) left configurations/global-infra-stg.tmpl.bicepparam assigning param grafanaMajorVersion = '{{ .stgGlobalV2.grafanaMajorVersion }}'. Once rebased on main, that param file assigns a parameter the template no longer declares → the STG-global global deploy fails at rollout with an undeclared-parameter error.

Heads-up that ci/prow/bicep-lint will not flag this: it's az bicep lint --file over .bicep templates/modules only, it filters out *.tmpl.bicepparam, and nothing in the repo runs az bicep build-params, so param↔template drift isn't validated in CI — it only surfaces in the INT/STG EV2 rollout. Please also strip grafanaMajorVersion from global-infra-stg.tmpl.bicepparam and remove the now-orphaned stgGlobalV2.grafanaMajorVersion config field + its required entry in config.schema.json as part of this PR.

raelga

/lgtm
/approve

openshift-ci · 2026-07-01T14:04:01Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: janboll, raelga

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [janboll,raelga]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci Bot added do-not-merge/work-in-progress approved labels Jun 29, 2026

janboll changed the title ~~WIP~~ WIP refactor grafana provisioning Jun 29, 2026

janboll force-pushed the grafana-refactor-grafana branch from d09655b to 9c0960e Compare June 29, 2026 13:36

janboll mentioned this pull request Jun 30, 2026

chore(grafana): move workspace lookup to pipeline shell step (ARO-28040) #5837

Closed

14 tasks

janboll force-pushed the grafana-refactor-grafana branch from 9c0960e to a99db21 Compare June 30, 2026 08:30

openshift-ci Bot added the needs-rebase label Jun 30, 2026

janboll force-pushed the grafana-refactor-grafana branch from a99db21 to c5803e3 Compare June 30, 2026 12:50

openshift-ci Bot removed the needs-rebase label Jun 30, 2026

janboll changed the title ~~WIP refactor grafana provisioning~~ fix: Refactor grafana provisioning Jun 30, 2026

janboll marked this pull request as ready for review June 30, 2026 13:07

Copilot AI review requested due to automatic review settings June 30, 2026 13:07

openshift-ci Bot removed the do-not-merge/work-in-progress label Jun 30, 2026

openshift-ci Bot requested review from ashishmax31 and bennerv June 30, 2026 13:07

Copilot started reviewing on behalf of janboll June 30, 2026 13:08 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

janboll force-pushed the grafana-refactor-grafana branch from daa0d03 to 3472c82 Compare June 30, 2026 13:21

Copilot AI review requested due to automatic review settings June 30, 2026 13:21

Copilot started reviewing on behalf of janboll June 30, 2026 13:21 View session

janboll force-pushed the grafana-refactor-grafana branch from 3472c82 to cbb3f8f Compare June 30, 2026 13:22

janboll mentioned this pull request Jun 30, 2026

Add manage grafana command Azure/ARO-Tools#260

Merged

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Copilot AI review requested due to automatic review settings June 30, 2026 13:26

Copilot started reviewing on behalf of janboll June 30, 2026 13:26 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Comment thread dev-infrastructure/grafana-pipeline.yaml Outdated

openshift-ci Bot added the needs-rebase label Jun 30, 2026

janboll added 2 commits June 30, 2026 14:17

Add missing merge fixes, also revert dummy config changes.

e949964

Also: update grafana version to version 12, since 11 is deprecated

Fix test pipeline

44295f2

Copilot AI review requested due to automatic review settings June 30, 2026 14:17

janboll force-pushed the grafana-refactor-grafana branch from b7d7c8f to 44295f2 Compare June 30, 2026 14:17

openshift-ci Bot removed the needs-rebase label Jun 30, 2026

Copilot started reviewing on behalf of janboll June 30, 2026 14:18 View session

Copilot AI reviewed Jun 30, 2026

View reviewed changes

Comment thread dev-infrastructure/grafana-pipeline.yaml

janboll added 2 commits June 30, 2026 14:27

Fix output, only output required data

b50560b

Update ARO Tools dependencies

d25c0a5

Copilot AI review requested due to automatic review settings July 1, 2026 08:15

Copilot started reviewing on behalf of janboll July 1, 2026 08:16 View session

Copilot AI reviewed Jul 1, 2026

View reviewed changes

Comment thread dev-infrastructure/grafana-pipeline.yaml

Comment on lines +28 to +31

identityFrom:

resourceGroup: singleton

step: output

name: globalMSIId

Update docs

85cdf81

raelga requested changes Jul 1, 2026

View reviewed changes

raelga approved these changes Jul 1, 2026

View reviewed changes

openshift-ci Bot assigned raelga Jul 1, 2026

openshift-ci Bot added the lgtm label Jul 1, 2026

openshift-merge-bot Bot merged commit 84f391e into main Jul 2, 2026
18 checks passed

openshift-merge-bot Bot deleted the grafana-refactor-grafana branch July 2, 2026 00:53

This was referenced Jul 2, 2026

revert: grafana provisioning refactor — unsupported templated enum step field (AROSLSRE-1379) #5887

Merged

fix(config): set grafanaZoneRedundantMode for public clouds (AROSLSRE-1379) #5886

Closed

Uh oh!

Conversation

janboll commented Jun 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Testing

Special notes for your reviewer

PR Checklist

Uh oh!

openshift-ci Bot commented Jun 29, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

janboll commented Jun 30, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

raelga left a comment

Choose a reason for hiding this comment

Uh oh!

raelga left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci Bot commented Jul 1, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

janboll commented Jun 29, 2026 •

edited

Loading