feat(graph): support multi-stamp execution graphs#250
Open
geoberle wants to merge 3 commits into
Open
Conversation
geoberle
force-pushed
the
stamp-validation
branch
from
ea75893 to
9a6cf99
Compare
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends the pipelines graph tooling to represent and execute multi-stamp topologies by duplicating stamped services per stamp while deduplicating unstamped nodes. It also changes Service.Stamped to *bool to preserve explicit user intent (stamped: false) and introduces per-stamp keying in graph identifiers/metadata.
Changes:
- Changed
topology.Service.Stampedfromboolto*bool, updated stamping propagation, and added validation for explicit opt-outs under stamped parents. - Added stamp-aware graph identity (
Identifier.Stamp) plus composite keys for steps/resource groups, and introducedForStampedEntrypoint+mergeStampedto merge per-stamp graphs. - Added
ConfigResolver.GetRegionStampConfigurationto re-resolve configuration templates using a different stamp value.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| pipelines/topology/types.go | Switches Stamped to pointer semantics and updates propagation/validation logic. |
| pipelines/topology/types_test.go | Adds/updates tests for pointer-based stamped propagation and validation cases. |
| pipelines/graph/graph.go | Introduces Stamp in identifiers and refactors graph lookups to use composite keys; adds ForStampedEntrypoint. |
| pipelines/graph/merge.go | Adds per-stamp graph merge implementation to combine per-stamp graphs into one execution graph. |
| pipelines/graph/merge_test.go | Adds tests for per-stamp merge behavior (node duplication, rewiring, and RG/step selection). |
| config/config.go | Adds API to resolve config for a region using an alternate stamp and stores resolver inputs for re-resolution. |
| config/config_test.go | Adds test coverage for stamp-specific config resolution. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
geoberle
force-pushed
the
stamp-validation
branch
from
9a6cf99 to
5ad5152
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
5ad5152 to
cb0be5e
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
cb0be5e to
fc1c8c7
Compare
12 tasks
geoberle
force-pushed
the
stamp-validation
branch
from
fc1c8c7 to
5f52dbb
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
3e08b9f to
1d214ad
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
1d214ad to
e63e2cb
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
2933842 to
f60a66d
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
f60a66d to
58826b5
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
58826b5 to
79b6037
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
79b6037 to
f8b5e32
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
f8b5e32 to
ba9356b
Compare
geoberle
force-pushed
the
stamp-validation
branch
from
7b31b0b to
961e693
Compare
Stamp-aware graph construction: when a service is marked as stamped in the
topology, accumulate() expands it once per stamp during a single tree walk.
Unstamped services are processed once. No post-hoc merge pass needed.
Key changes:
- ForEntrypoints accepts stamp-keyed pipelines (map[string]map[string]*Pipeline).
Non-stamped callers use ForEntrypoint which wraps pipelines as {"": pipelines}.
ForStampedEntrypoint is removed — callers pass stamp pipelines directly.
- nodesFor bakes stamp into all Identifiers at creation (nodes, edges, steps,
validation steps, resource group keys).
- Leaf-to-root wiring between services is stamp-scoped: stamped parent leaves
connect only to same-stamp child roots. Unstamped parent leaves fan out to
all stamps.
- External dependency resolution inherits stamp from the declaring node when
the target service is stamped; unstamped targets keep empty stamp.
- StepKey removed — Identifier used as map key for Steps.
- GetResourceGroup narrowed to accept ResourceGroupKey instead of Identifier.
- ConfigResolver.GetRegionStampConfiguration resolves config with stamp dimension.
- Service.Stamped changed to *bool — explicit false preserved and validated,
distinguishing "unset" from "intentionally not stamped".
Breaking changes on Graph, ForEntrypoints, and ForStampedEntrypoint (removed).
…nctions Split ForEntrypoints into ForEntrypoints (unstamped, flat pipelines map) and ForStampedEntrypoints (stamp-expanded, stamp→pipelines map). Callers that don't use stamps no longer see the map[string]map[string]*Pipeline type. ForEntrypoint remains as a single-entrypoint convenience wrapper. Extract resolveIterations, accumulateIteration, registerResourceGroups, and wireInterServiceEdges from the monolithic accumulate method. Add stamp and regionShort parameters to GetRegionConfiguration and ValueProvenance so they re-template per-stamp instead of using the resolver's initial stamp. Extract resolveOverrides to share the preprocessing logic.
Signed-off-by: Gerd Oberlechner <goberlec@redhat.com>
geoberle
force-pushed
the
stamp-validation
branch
from
961e693 to
74e83f2
Compare
Comment on lines
+804
to
+813
| func dotLabel(shortSG string, id Identifier, resourceGroups map[ResourceGroupKey]*types.ResourceGroupMeta) string { | ||
| rgName := id.ResourceGroup | ||
| if rg, ok := resourceGroups[id.ResourceGroupKey()]; ok { | ||
| rgName = rg.ResourceGroup | ||
| } | ||
| if id.Stamp != "" { | ||
| return fmt.Sprintf("%s/%s/%s (stamp=%s)", shortSG, rgName, id.Step, id.Stamp) | ||
| } | ||
| return fmt.Sprintf("%s/%s/%s", shortSG, rgName, id.Step) | ||
| } |
| } | ||
|
|
||
| // GetResourceGroup returns the resource group metadata for the given key. | ||
| func (c *Graph) GetResourceGroup(key ResourceGroupKey) (*types.ResourceGroupMeta, bool) { |
Contributor
There was a problem hiding this comment.
Probably just a nit, so if there are no other reasons to update the PR maybe ignore - but IMO having these kinds of helpers are strictly worse than omitting them. The member of c that you do the lookup in is already exported, so the user can already just replace c.GetResourceGroup(key) with c.ResourceGroups[key] without loss of functionality. Having the helper in addition to the member only adds confusion on what is the correct thing to use.
| node [fontname="Helvetica,Arial,sans-serif"] | ||
| edge [fontname="Helvetica,Arial,sans-serif"] | ||
| "Grandparent_global_acrs" [label="Grandparent/global/acrs"]; | ||
| "Grandparent_global_acrs" [label="Grandparent/global-rg/acrs"]; |
Contributor
There was a problem hiding this comment.
Could we pull this into a prior commit, so you can prove in the stamped commit that the existing tests are unaffected?
|
|
||
| var stamps []string | ||
| for stamp := range stampPipelines { | ||
| if stamp != "" { |
Contributor
There was a problem hiding this comment.
By treating "" as a special string value, we're making the code in here a bit more fragile than it needs to be, and not letting the Go type system work for us by distinguishing clearly between stamped and not-stamped would make it much less likely/possible to make mistakes when coding
| // safely identifies root nodes, and this is the only time any actor will add parents to these roots. | ||
| // When both parent and child are stamped, only matching-stamp roots are returned. | ||
| func (c *Graph) findChildRoots(parent, child *topology.Service, parentStamp string) []Identifier { | ||
| var roots []Identifier |
Contributor
There was a problem hiding this comment.
We tried to use sets.Set before so doing lookups on these is not O(N)
| "MgmtDB_db-rg_deploy_2" [label="MgmtDB/db-rg/deploy (stamp=2)" style=filled fillcolor="#FFD9B3"]; | ||
| "MgmtDB_db-rg_deploy_2" -> "MgmtNet_net-rg_deploy_2"; | ||
| "MgmtNet_net-rg_deploy_1" [label="MgmtNet/net-rg/deploy (stamp=1)" style=filled fillcolor="#B3D9FF"]; | ||
| "MgmtNet_net-rg_deploy_2" [label="MgmtNet/net-rg/deploy (stamp=2)" style=filled fillcolor="#FFD9B3"]; |
Contributor
There was a problem hiding this comment.
I would want to see:
- a child of the stamped pipelines to show fan-in on dependency
- an example of external dependency
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Stamp-aware graph construction: when a service is marked as stamped in the topology,
accumulate()expands it once per stamp during a single tree walk. Unstamped services are processed once. No post-hoc merge pass needed.Service.Stampedchanged to*bool— explicitfalseis preserved and validated, distinguishing "unset" from "intentionally not stamped".What changed
map[string]*Pipeline— stamped services appear once with an empty Stamp, suitable for runtimes that handle stamp expansion themselves (e.g. EV2).map[string]map[string]*Pipeline) and expands stamped services once per stamp in the graph, suitable for runtimes where the graph drives per-stamp execution (e.g. templatize).Identifierused as map key forSteps.ResourceGroupKeyinstead ofIdentifier.Breaking changes
Breaking changes on
Graph,ForEntrypoints(signature changed), andForStampedEntrypoint(replaced byForStampedEntrypoints).Companion PRs to adapt: