feat(templatize): add multi-stamp parallel execution for entrypoint runs#5697
Open
geoberle wants to merge 1 commit into
Open
feat(templatize): add multi-stamp parallel execution for entrypoint runs#5697geoberle wants to merge 1 commit into
geoberle wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: geoberle The full list of commands accepted by this bot can be found here. The pull request process is described here DetailsNeeds approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Collaborator
Author
|
/hold |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates tooling/templatize to support running stamped service-group graphs concurrently within a single entrypoint invocation, while isolating per-stamp execution state (outputs/config/pipelines) to prevent cross-stamp collisions.
Changes:
- Introduces stamp-layered step outputs via
ExecutionState.RecordOutputandExecutionState.GetOutputs(stamp)and updates step runners (ARM/ARMStack/Helm/Shell) to resolve inputs against the correct stamp view. - Switches entrypoint graph generation to
graph.ForStampedEntrypoint(...)and adds astampConfigExecutorwrapper to apply per-stamp configuration during execution. - Adds multi-stamp unit tests and updates CLI options/helpers + Makefile wiring to drive stamp-count-based runs/cleanup.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| tooling/templatize/pkg/pipeline/shell.go | Shell step input resolution now reads outputs via GetOutputs(id.Stamp) for stamp isolation. |
| tooling/templatize/pkg/pipeline/run.go | Core execution updated for stamped graphs, layered outputs, per-stamp config executor, and stamp-aware graph lookup. |
| tooling/templatize/pkg/pipeline/run_test.go | Adds unit tests covering multi-stamp execution, output isolation, subscription resolution, and error propagation. |
| tooling/templatize/pkg/pipeline/helm.go | Helm runner now resolves replacements/inputs using stamp-scoped outputs. |
| tooling/templatize/pkg/pipeline/arm.go | ARM runner now passes stamp context through and resolves inputs using stamp-scoped outputs. |
| tooling/templatize/pkg/pipeline/arm_stack.go | ARMStack input resolution switched to GetOutputs(id.Stamp); stack name logic remains stamp-influenced via options.Stamp. |
| tooling/templatize/internal/well_formed_test.go | Updates well-formed validation iteration to match new stamped step keying in the graph. |
| tooling/templatize/go.mod | Bumps ARO-Tools config / pipelines versions needed for stamped graph support. |
| tooling/templatize/go.sum | Updates checksums for new ARO-Tools dependency versions. |
| tooling/templatize/cmd/rolloutoptions.go | Persists ConfigResolver in completed rollout options to enable per-stamp config resolution. |
| tooling/templatize/cmd/entrypoint/run/options.go | Entry-point run now precomputes stamped graph inputs (stamp configs/pipelines) and passes them into pipeline execution. |
| tooling/templatize/cmd/entrypoint/entrypointutils/stamps.go | New helpers for stamp list derivation, per-stamp config resolution, and stamped graph assembly. |
| tooling/templatize/cmd/entrypoint/entrypointutils/options.go | Adds --stamp-count-config-ref and updates regional RG precheck extraction to span all stamp configs. |
| tooling/templatize/cmd/entrypoint/cleanup/options.go | Cleanup now builds stamped graph + uses all stamp configs to discover RGs for deletion. |
| Makefile | Updates local-run and cleanup invocations to use --stamp-count-config-ref=mgmt.stamps.count. |
| dev-infrastructure/templates/mgmt-cluster.bicep | Makes module deployment name stamp-unique via uniqueString(resourceGroup().name) to avoid cross-stamp ARM conflicts. |
geoberle
force-pushed
the
templatize-stamping
branch
from
7d0be4a to
407a03f
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
407a03f to
e03c3df
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
e03c3df to
6c2440e
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
6c2440e to
b519c4b
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
b519c4b to
a5c75d9
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
a5c75d9 to
7660073
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
7660073 to
dd52a09
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
dd52a09 to
9579113
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
34702ea to
7f66844
Compare
Collaborator
Author
|
/test e2e-parallel |
geoberle
force-pushed
the
templatize-stamping
branch
from
7f66844 to
377239d
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
377239d to
3df2f94
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
3df2f94 to
b8f5f84
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
b8f5f84 to
ef467e1
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
ef467e1 to
8451726
Compare
Comment on lines
+32
to
+34
| // RegionalResourceGroupNames extracts the regional resource group names from the | ||
| // base configuration and all per-stamp configurations. This ensures stamped RG | ||
| // names (which vary per stamp) are included. |
| cmd.Flags().StringArrayVar(&opts.TopologyFiles, "topology-config", opts.TopologyFiles, "Path to a topology configuration file. Can be specified multiple times.") | ||
| cmd.Flags().StringVar(&opts.Entrypoint, "entrypoint", opts.Entrypoint, "Name of the entrypoint to create Ev2 manifests for. Exclusive with --service-group.") | ||
| cmd.Flags().StringVar(&opts.ServiceGroup, "service-group", opts.ServiceGroup, "Name of the service group to create Ev2 manifests for. Exclusive with --entrypoint.") | ||
| cmd.Flags().StringVar(&opts.StampCountConfigRef, "stamp-count-config-ref", opts.StampCountConfigRef, "Configuration path where the stamp count is stored (e.g. mgmt.stamps.count). When provided, stamped service groups are executed once per stamp in parallel.") |
Collaborator
Author
|
/test e2e-parallel |
geoberle
force-pushed
the
templatize-stamping
branch
from
8451726 to
d5eee41
Compare
geoberle
force-pushed
the
templatize-stamping
branch
from
d5eee41 to
19f4318
Compare
ARO HCP deploys management clusters as "stamps" within Azure regions. Previously templatize handled one stamp per invocation. This change makes the execution engine natively iterate stamps in parallel. Key changes: Execution state isolation: ExecutionState gains layered outputs (StampOutputs) with RecordOutput/GetOutputs methods. Each stamp's step outputs are stored separately, with GetOutputs returning a merged snapshot (stamp layer over base) to prevent cross-stamp output collision. All step runners (ARM, ARM stack, Helm, shell) use GetOutputs(id.Stamp) for stamp-aware input resolution. Graph merge via ARO-Tools: graph.MergeStamped() combines per-stamp graphs into a unified execution graph. ResourceGroups and Steps use stamp-keyed types (ResourceGroupKey, StepKey) so per-stamp metadata (different Azure RG names, subscriptions, AKS cluster names) doesn't collide. GetResourceGroup/GetStep provide stamp-aware lookup. Unstamped parent nodes get their Children fixed up to reference all stamped children. Stamp resolution: BuildStampList determines stamps from either --stamp (single, e.g. from dev settings) or --stamp-count-config-ref (multi, reads count from config). The count flag wins when both are present. Shared helpers in entrypointutils/stamps.go serve both the run and cleanup subcommands. Cleanup support: cleanup subcommand uses BuildStampedGraph to discover all per-stamp resource groups for deletion. RegionalResourceGroupNames accepts per-stamp configs to include all stamp-variant RG names. Other: stampConfigExecutor swaps config per stamp with missing-key validation. Stamped step errors include stamp context. Logs include stamp as structured field. ARM deployment names in mgmt-cluster.bicep use stamp-unique suffixes to prevent cross-stamp conflicts.
geoberle
force-pushed
the
templatize-stamping
branch
from
19f4318 to
32c078a
Compare
Collaborator
Author
|
/test e2e-parallel |
|
|
||
| // collectResourceGroupTargets extracts all unique (resourceGroup, subscription) pairs across | ||
| // every stamp's pipelines. This avoids relying on the execution graph which, in service-group | ||
| // mode, only contains resource groups from the first stamp. |
Contributor
There was a problem hiding this comment.
I might prefer some other mechanism that requires that we're resolving the graph the same way here that we did during deployment - what is it that causes this to be in "service-group" mode and requires this workaround?
|
PR needs rebase. DetailsInstructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
ARO HCP deploys management clusters as "stamps" within Azure regions. Previously templatize handled one stamp per invocation. This change makes the execution engine natively iterate stamps in parallel.
Key changes:
Execution state isolation: ExecutionState gains layered outputs (StampOutputs) with RecordOutput/GetOutputs methods. Each stamp's step outputs are stored separately, with GetOutputs returning a merged snapshot (stamp layer over base) to prevent cross-stamp output collision. All step runners (ARM, ARM stack, Helm, shell) use GetOutputs(id.Stamp) for stamp-aware input resolution.
Graph merge via ARO-Tools: graph.MergeStamped() combines per-stamp graphs into a unified execution graph. ResourceGroups and Steps use stamp-keyed types (ResourceGroupKey, StepKey) so per-stamp metadata (different Azure RG names, subscriptions, AKS cluster names) doesn't collide. GetResourceGroup/GetStep provide stamp-aware lookup. Unstamped parent nodes get their Children fixed up to reference all stamped children.
Stamp resolution: BuildStampList determines stamps from either --stamp (single, e.g. from dev settings) or --stamp-count-config-ref (multi, reads count from config). The count flag wins when both are present. Shared helpers in entrypointutils/stamps.go serve both the run and cleanup subcommands.
Cleanup support: cleanup subcommand uses BuildStampedGraph to discover all per-stamp resource groups for deletion. RegionalResourceGroupNames accepts per-stamp configs to include all stamp-variant RG names.
Other: stampConfigExecutor swaps config per stamp with missing-key validation. Stamped step errors include stamp context. Logs include stamp as structured field. ARM deployment names in mgmt-cluster.bicep use stamp-unique suffixes to prevent cross-stamp conflicts.
Why
Testing
Special notes for your reviewer
PR Checklist