Harden Azure provisioning recovery

[davidfowl]

Description

This PR introduces AzureProvisioningController, a serialized control loop that coordinates all run-mode Azure provisioning operations. It replaces the inline provisioning logic that previously lived in AzureProvisioner with a channel-based queue that serializes startup provisioning, dashboard commands, CLI commands, per-resource recovery actions, and background drift detection through a single processing loop.

Demos

Dashboard command menu: opens the storage resource commands and shows Cancel deployment disabled once provisioning has completed.

CLI demos:

• Resource command discovery and get-azure-resource
• Completed deployment cancel guard
• Combined CLI flow
• CLI discovery screenshot GIF
• CLI cancel guard screenshot GIF

Controller architecture

The controller uses a Channel<QueuedOperation> with a single reader. Every operation — provision, reprovision, reset, change-location, change-context, cancel-deployment, delete-resource, delete-environment, drift-check — is modeled as a typed intent record that gets enqueued and processed one at a time. This eliminates races between concurrent dashboard commands, CLI commands, and the periodic drift monitor.

Within a provisioning pass, individual resources fan out concurrently but are ordered by dependency. Each resource gets a per-resource ProvisioningTaskCompletionSource that downstream resources await before starting their own deployment. The TCS is completed through the controller lifecycle paths, so dependents unblock as soon as their prerequisites finish rather than waiting for the entire batch.

What the provisioning stack can do now

Resource commands (per-resource):

• Get Azure resource — returns cached deployment state, Azure metadata, portal links, outputs, and live existence status for the backing resource
• Reprovision — clears cached deployment state for a resource and its children/role-assignments, then redeploys
• Change location — prompts for a new Azure region, deletes the existing ARM resource if it conflicts, sets a per-resource location override, and reprovisions
• Cancel deployment — requests cancellation for an active cached ARM deployment and marks the local resource tree canceled only when an active deployment was actually targeted
• Delete Azure resource — cancels any cached active deployment, deletes cached/live ARM resource IDs including partial deployment operation targets, then resets local state
• Forget state — clears cached deployment state without deleting Azure resources or reprovisioning

Environment commands (all resources):

• Reset provisioning state — wipes all cached deployment state and resets the environment to NotStarted
• Change Azure context — re-prompts for subscription/tenant/resource-group/location, then reprovisions all resources with the new context
• Reprovision all — clears and redeploys all Azure resources while preserving location overrides
• Delete Azure resources — deletes the resource group and resets state

Command state invariants:

• Diagnostic commands such as Get Azure resource remain enabled during resource operations
• Mutating commands for a target resource are disabled while that resource has an active queued operation
• Unrelated resource commands remain enabled while another resource is being reprovisioned/deleted
• Environment commands are disabled while any Azure operation is active
• Cancel deployment is only enabled while the resource is waiting on an active ARM deployment, and direct execution against completed deployments fails without rewriting local state to Canceled
• Change location and Reprovision are disabled during active provisioning/deleting states; recovery commands remain available for failed/stale resources

Background drift detection:

• Periodic timer probes ARM to verify each running resource still exists
• Marks missing resources as "Missing in Azure" and the environment as "Drifted"
• Non-overlapping — queues at most one check through the same serialized channel

Azure resource metadata:

• Both fresh and cached-state resources now expose: azure.subscription.id, azure.resource.group, azure.tenant.id, azure.tenant.domain, azure.location, and resource.source (full ARM deployment id)
• Failed resources stamp the predicted deployment id before the ARM call, so agents and tools can still query Azure even when provisioning fails

Location overrides:

• Per-resource overrides are persisted in deployment state and survive resets/reprovisioning
• When changing location, the controller deletes the existing Azure resource first to avoid ARM InvalidResourceLocation conflicts
• Stale overrides are cleared when the effective location changes

Other changes

• BicepProvisioner — hardened checksum reuse validation, unified Azure identity metadata across fresh/cached paths, predicted deployment-id stamping for failed resources
• RunModeProvisioningContextProvider — refactored Azure context acquisition and interactive prompting
• AzureResourcePreparer — wires per-resource commands into the app model with command-specific enabled/disabled state
• Only registers AzureProvisioningController in run mode (fixes DI failures in publish/test scenarios)

Test coverage

• Controller regression tests covering: reprovision, reset, change-location, change-context, cancel-deployment, delete-resource, delete-environment, drift detection, dependency ordering, command state management, location override preservation
• Command-state invariant tests covering resource lifecycle states, targeted active operations, unrelated resources, environment command locking, and completed-deployment cancel behavior
• Provisioner regression tests covering: checksum validation, cached-state identity properties, stale location overrides, failed-resource metadata stamping

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
    • If yes, did you have an API Review for it?
      • Yes
      • No
    • Did you add <remarks /> and <code /> elements on your triple slash comments?
      • Yes
      • No
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
    • If yes, have you done a threat model and had a security review?
      • Yes
      • No
  • No
• Does the change require an update in our Aspire docs?
  • Yes
    • Link to aspire.dev issue:
      • New issue
  • No

[github-actions]

🚀 Dogfood this PR with:

⚠️ WARNING: Do not do this without first carefully reviewing the code of this PR to satisfy yourself it is safe.

curl -fsSL https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.sh | bash -s -- 15697

Or

• Run remotely in PowerShell:

iex "& { $(irm https://raw.githubusercontent.com/microsoft/aspire/main/eng/scripts/get-aspire-cli-pr.ps1) } 15697"

[github-actions]

Re-running the failed jobs in the CI workflow for this pull request because 1 job was identified as retry-safe transient failures in the CI run attempt.
GitHub was asked to rerun all failed jobs for that attempt, and the rerun is being tracked in the rerun attempt.
The job links below point to the failed attempt jobs that matched the retry-safe transient failure rules.

• Tests / Hosting-1 / Hosting-1 (windows-latest) - Job-level runner or infrastructure failure matched the transient allowlist.

[davidfowl]

@copilot fix the failing test

[Copilot]

@copilot fix the failing test

I investigated all Azure test classes locally and all tests pass (BicepProvisionerTests, EnvironmentResourceExtensionsTests, ResourcePreparerTests, ProvisioningContextProviderTests, DeployerTests, BicepResourceTests — 178+ tests total). I was unable to access the CI logs to identify the specific failing test due to DNS restrictions in this environment. The previous two commits (95026c9, 7bbaf1f) addressed DI resolution failures for AzureProvisioningController in publish/test mode — could you point to the specific failing test or share the CI run URL so I can investigate further?

Warning

Firewall rules blocked me from connecting to one or more addresses (expand for details)

I tried to connect to the following addresses, but was blocked by firewall rules:

• azcliprod.blob.core.windows.net
  • Triggering command: /usr/bin/../../opt/az/bin/python3 /usr/bin/../../opt/az/bin/python3 -Im azure.cli account get-access-token --output json --resource REDACTED (dns block)
• https://api.github.com/graphql
  • Triggering command: /usr/bin/gh gh pr list --repo microsoft/aspire --head davidfowl/azure-provisioning-refactor --json number,title committer.name ptables docker netw�� ls --format /usr/bin/docker HEAD (http block)
• https://api.github.com/repos/microsoft/aspire/actions/runs
  • Triggering command: /usr/bin/gh gh run list --repo microsoft/aspire --branch davidfowl/azure-provisioning-refactor --limit 5 --json databaseId,status,conclusion,name docker netw�� ls --format /usr/bin/docker --verify davidfowl/azure--p rosoft.developercontrolplane.linls docker (http block)
  • Triggering command: /usr/bin/gh gh run list --repo microsoft/aspire --branch davidfowl/azure-provisioning-refactor --limit 3 get-access-tokenrun-controllers /home/REDACTED/.ca--kubeconfig get/packages/mic/tmp/aspire-dcp7J1czA/kubeconfig netw�� ls oj /usr/bin/docker account TR7N1/kubeconfignetwork /home/REDACTED/.cols docker (http block)
• https://api.github.com/repos/microsoft/aspire/commits/7bbaf1f11e709efb1696d30fc76e2795e57bd97a/check-runs
  • Triggering command: /usr/bin/gh gh api /repos/microsoft/aspire/commits/7bbaf1f11e709efb1696d30fc76e2795e57bd97a/check-runs?per_page=100 controlplane.linget-access-token netw�� ls oj /usr/bin/docker --local artifacts/bin/As-rs /home/REDACTED/.dotnet/tools echo ls docker netw�� ls --format rosoft.developercontrolplane.linux-amd64/0.22.9/tools/dcp --noprofile (http block)
  • Triggering command: /usr/bin/gh gh api /repos/microsoft/aspire/commits/7bbaf1f11e709efb1696d30fc76e2795e57bd97a/check-runs?per_page=100 docker netw�� ls --format /usr/bin/docker set-url origin name docker _.py�� ls --format /usr/bin/docker AzureProvisioninuname tests/Aspire.Ora-rs t/lsb_release docker (http block)
• https://api.github.com/repos/microsoft/aspire/pulls
  • Triggering command: /usr/bin/gh gh api repos/microsoft/aspire/pulls?head=microsoft:davidfowl/azure-provisioning-refactor docker netw�� ls --format /usr/bin/docker --local committer.email get/packages/mic--filter-not-trait docker netw�� ls --format /usr/bin/docker --oneline 2hMCk/kubeconfig/usr/bin/az tnet/tools/unameaccount docker (http block)
  • Triggering command: /usr/bin/gh gh api repos/microsoft/aspire/pulls?head=microsoft:davidfowl/azure-provisioning-refactor&state=open docker netw�� ls --format /usr/bin/docker ls --format /usr/bin/docker docker netw�� k/aspire/aspire/artifacts/log/De-- --format /usr/bin/docker ls --format es docker (http block)
• https://api.github.com/repos/microsoft/aspire/statuses/7bbaf1f11e709efb1696d30fc76e2795e57bd97a
  • Triggering command: /usr/bin/gh gh api /repos/microsoft/aspire/statuses/7bbaf1f11e709efb1696d30fc76e2795e57bd97a docker netw�� ls --format /usr/bin/docker azd auth token -/home/REDACTED/.nuget/packages/microsoft.developercontrolplane.linux-amd64/0.22.9/--format (http block)

If you need me to access, download, or install something from one of these locations, you can either:

• Configure Actions setup steps to set up my environment, which run before the firewall is enabled
• Add the appropriate URLs or hosts to the custom allowlist in this repository's Copilot coding agent settings (admins only)

[davidfowl]

/deployment-test

[github-actions]

🚀 Deployment tests starting on PR #15697...

This will deploy to real Azure infrastructure. Results will be posted here when complete.

View workflow run

[JamesNK]

How do you feel about test coverage? I think new code is greater than tests, which probably means there are gaps.

However, it is difficult to unit test Azure APIs. Maybe if they were behind a client with an interface that could be mocks then the surrounding code can be well tested.

[davidfowl]

We have an end to end deployment test that actually passes, this code had 0 test coverage before so I'm feeling pretty good 😄

[davidfowl]

However, it is difficult to unit test Azure APIs. Maybe if they were behind a client with an interface that could be mocks then the surrounding code can be well tested.

Some of them are that's why we can write unit tests to begin with but at some point we deploy a bicep file and that's the end of it.

I'll see if I can get most of the scenarios unit tested. Though I care more about the ux right now. I just ran through the ux and there were some gaps to fill and experiences to improve.

BTW we could really use a progress primitive for long running ops like this.

[github-actions]

❓ CLI E2E Tests unknown — 113 passed, 0 failed, 2 unknown (commit 297fbea)

View all recordings

-	Test	Detail
✅	AddPackageInteractiveWhileAppHostRunningDetached	Recording · Job · CLI logs
✅	AddPackageWhileAppHostRunningDetached	Recording · Job · CLI logs
✅	AgentCommands_AllHelpOutputs_AreCorrect	Recording · Job · CLI logs
✅	AgentInitCommand_DefaultSelection_InstallsDefaultSkills	Recording · Job · CLI logs
✅	AgentInitCommand_MigratesDeprecatedConfig	Recording · Job · CLI logs
✅	AgentInit_NonInteractive_BundleOnlySkillsNotInCatalog	Recording · Job · CLI logs
✅	AgentMcpListStructuredLogsReturnsLogsFromStarterApp	Recording · Job · CLI logs
✅	AgentMcpListStructuredLogsReturnsLogsFromStarterApp_DevLocalhost	Recording · Job · CLI logs
✅	AgentMcpListStructuredLogsReturnsLogsFromStarterApp_Isolated	Recording · Job · CLI logs
✅	AllPublishMethodsBuildDockerImages	Recording · Job · CLI logs
✅	AspireAddAndStartWorkAgainstLegacyAppHostTs	Recording · Job · CLI logs
✅	AspireAddPackageVersionToDirectoryPackagesProps	Recording · Job · CLI logs
✅	AspireInitSingleFileAppHostRunsViaDotnetRunAppHost	Recording · Job · CLI logs
✅	AspireInit_ExistingAppHostDir_RecreatesNuGetConfigKeepsFiles	Recording · Job · CLI logs
❔	AspireInit_SolutionFile_BuildsAgainstChannelHive	Recording · Job · CLI logs
✅	AspireStartUpdatesStaleTypeScriptAppHostPath	Recording · Job · CLI logs
✅	AspireUpdateRemovesAppHostPackageVersionFromDirectoryPackagesProps	Recording · Job · CLI logs
✅	AspireUpdateRemovesOrphanAppHostPackageVersionWhenSdkAlreadyCurrent	Recording · Job · CLI logs
✅	Banner_DisplayedOnFirstRun	Recording · Job · CLI logs
✅	Banner_DisplayedWithExplicitFlag	Recording · Job · CLI logs
✅	Banner_NotDisplayedWithNoLogoFlag	Recording · Job · CLI logs
✅	CertificatesClean_RemovesCertificates	Recording · Job · CLI logs
✅	CertificatesTrust_WithNoCert_CreatesAndTrustsCertificate	Recording · Job · CLI logs
✅	CertificatesTrust_WithUntrustedCert_TrustsCertificate	Recording · Job · CLI logs
✅	ConfigSetGet_CreatesNestedJsonFormat	Recording · Job · CLI logs
✅	CreateAndRunAspireStarterProject	Recording · Job · CLI logs
✅	CreateAndRunAspireStarterProjectWithBundle	Recording · Job · CLI logs
✅	CreateAndRunEmptyAppHostProject	Recording · Job · CLI logs
✅	CreateAndRunJavaEmptyAppHostProject	Recording · Job · CLI logs
✅	CreateAndRunJsReactProject	Recording · Job · CLI logs
✅	CreateAndRunPolyglotAppHostWithDevLocalhostUrls	Recording · Job · CLI logs
✅	CreateAndRunPythonReactProject	Recording · Job · CLI logs
✅	CreateAndRunTypeScriptEmptyAppHostProject	Recording · Job · CLI logs
✅	CreateAndRunTypeScriptStarterProject	Recording · Job · CLI logs
✅	CreateJavaAppHostWithViteApp	Recording · Job · CLI logs
✅	CreateTypeScriptAppHostWithViteApp_UsesConfiguredToolchain	Recording · Job · CLI logs
✅	DashboardRunWithAgentMcpListTracesReturnsNoTraces	Recording · Job · CLI logs
✅	DashboardRunWithAgentMcpListTracesReturnsNoTraces_DevLocalhost	Recording · Job · CLI logs
✅	DashboardRunWithOtelTracesReturnsNoTraces	Recording · Job · CLI logs
✅	DashboardRunWithOtelTracesReturnsNoTraces_DevLocalhost	Recording · Job · CLI logs
✅	DeployK8sBasicApiService	Recording · Job · CLI logs
✅	DeployK8sWithExternalHelmChart	Recording · Job · CLI logs
✅	DeployK8sWithGarnet	Recording · Job · CLI logs
✅	DeployK8sWithMongoDB	Recording · Job · CLI logs
✅	DeployK8sWithMySql	Recording · Job · CLI logs
✅	DeployK8sWithPostgres	Recording · Job · CLI logs
✅	DeployK8sWithRabbitMQ	Recording · Job · CLI logs
✅	DeployK8sWithRedis	Recording · Job · CLI logs
✅	DeployK8sWithSqlServer	Recording · Job · CLI logs
✅	DeployK8sWithValkey	Recording · Job · CLI logs
✅	DeployTypeScriptAppToKubernetes	Recording · Job · CLI logs
✅	DescribeCommandResolvesReplicaNames	Recording · Job · CLI logs
✅	DescribeCommandShowsRunningResources	Recording · Job · CLI logs
✅	DetachFormatJsonProducesValidJson	Recording · Job · CLI logs
✅	DetachFormatJsonProducesValidJsonWhenRestartingExistingInstance	Recording · Job · CLI logs
✅	DoPublishAndDeployListStepsWork	Recording · Job · CLI logs
✅	DocsCommand_RendersInteractiveMarkdownFromLocalSource	Recording · Job · CLI logs
✅	DoctorCommand_DetectsDeprecatedAgentConfig	Recording · Job · CLI logs
✅	DoctorCommand_TypeScriptAppHostReportsMissingConfiguredToolchain	Recording · Job · CLI logs
✅	DoctorCommand_WithSslCertDir_ShowsTrusted	Recording · Job · CLI logs
✅	DoctorCommand_WithoutSslCertDir_ShowsPartiallyTrusted	Recording · Job · CLI logs
✅	DotNetRunFileBasedAppHostUsesAspireCliBundle	Recording · Job · CLI logs
✅	DotNetRunProjectAppHostUsesAspireCliBundle	Recording · Job · CLI logs
✅	GatewayWithoutExternalEndpoint_FailsPublishWithGuidance	Recording · Job · CLI logs
✅	GeneratedAspireDevScript_StartsWatchMode_WithConfiguredToolchain	Recording · Job · CLI logs
✅	GlobalMigration_HandlesCommentsAndTrailingCommas	Recording · Job · CLI logs
✅	GlobalMigration_HandlesMalformedLegacyJson	Recording · Job · CLI logs
✅	GlobalMigration_PreservesAllValueTypes	Recording · Job · CLI logs
✅	GlobalMigration_SkipsWhenNewConfigExists	Recording · Job · CLI logs
✅	GlobalSettings_MigratedFromLegacyFormat	Recording · Job · CLI logs
✅	IngressWithoutExternalEndpoint_FailsPublishWithGuidance	Recording · Job · CLI logs
✅	InitTypeScriptAppHost_AugmentsExistingViteRepoInWorkspaceSubdirectory	Recording · Job · CLI logs
✅	InteractiveCSharpInitCreatesExpectedFiles	Recording · Job · CLI logs
✅	InvalidAppHostPathWithComments_IsHealedOnRun	Recording · Job · CLI logs
✅	JavaScriptHostingApisRunFromTypeScriptAppHost	Recording · Job · CLI logs
✅	LatestCliCanStartStableChannelAppHost	Recording · Job · CLI logs
✅	LatestCliCanStartStableChannelTypeScriptAppHost	Recording · Job · CLI logs
✅	LegacySettingsMigration_AdjustsRelativeAppHostPath	Recording · Job · CLI logs
✅	LogsCommandShowsResourceLogs	Recording · Job · CLI logs
✅	OtelLogsReturnsStructuredLogsFromStarterApp	Recording · Job · CLI logs
✅	OtelLogsReturnsStructuredLogsFromStarterAppIsolated	Recording · Job · CLI logs
✅	ProcessCommandCallbackReceivesCliArguments	Recording · Job · CLI logs
✅	PsCommandListsRunningAppHost	Recording · Job · CLI logs
✅	PsFormatJsonOutputsOnlyJsonToStdout	Recording · Job · CLI logs
✅	PublishJavaScriptPatternsGeneratesExpectedDockerComposeArtifacts	Recording · Job · CLI logs
✅	PublishWithConfigureEnvFileUpdatesEnvOutput	Recording · Job · CLI logs
✅	PublishWithDockerComposeServiceCallbackSucceeds	Recording · Job · CLI logs
✅	PublishWithoutOutputPathUsesAppHostDirectoryDefault	Recording · Job · CLI logs
✅	ResourceCommand_FailedExec_ShowsLogPathAndLogHasEntries	Recording · Job · CLI logs
✅	ResourceCommand_SetAndDeleteParameterUpdatesDescribeOutput	Recording · Job · CLI logs
✅	RestoreGeneratesSdkFiles	Recording · Job · CLI logs
✅	RestoreGeneratesSdkFiles_WithConfiguredToolchain	Recording · Job · CLI logs
✅	RestoreRefreshesGeneratedSdkAfterAddingIntegration	Recording · Job · CLI logs
✅	RestoreSupportsConfigOnlyHelperPackageAndCrossPackageTypes	Recording · Job · CLI logs
✅	RunFromParentDirectory_UsesExistingConfigNearAppHost	Recording · Job · CLI logs
✅	RunReportsSyntaxErrorsForDotNetAppHost	Recording · Job · CLI logs
✅	RunReportsSyntaxErrorsForTypeScriptAppHost	Recording · Job · CLI logs
✅	SecretCrudOnDotNetAppHost	Recording · Job · CLI logs
✅	SecretCrudOnTypeScriptAppHost	Recording · Job · CLI logs
✅	StagingChannel_ConfigureAndVerifySettings_ThenSwitchChannels	Recording · Job · CLI logs
✅	StartAndWaitForTypeScriptSqlServerAppHostWithNativeAssets	Recording · Job · CLI logs
✅	StartReportsSyntaxErrorsForDotNetAppHost	Recording · Job · CLI logs
✅	StartReportsSyntaxErrorsForTypeScriptAppHost	Recording · Job · CLI logs
✅	StopAllAppHostsFromAppHostDirectory	Recording · Job · CLI logs
✅	StopJavaPolyglotAppHostUsingApphostDirectory	Recording · Job · CLI logs
✅	StopNonInteractiveSingleAppHost	Recording · Job · CLI logs
✅	StopTypeScriptPolyglotAppHostUsingApphostDirectory	Recording · Job · CLI logs
✅	StopWithNoRunningAppHostExitsSuccessfully	Recording · Job · CLI logs
✅	TypeScriptAppHostRunDoesNotDeadlockWhenLazyOptionsInvokeAsyncCallback	Recording · Job · CLI logs
✅	TypeScriptAppHostWithVite_AllowsDifferentGuestPkgManager	Recording · Job · CLI logs
✅	UnAwaitedChainsCompileWithAutoResolvePromises	Recording · Job · CLI logs
❔	UpdateToStable_CSharpEmptyAppHost_KeepsConfigChannel	Recording · Job · CLI logs
✅	UpdateToStable_CSharpSingleFileInit_KeepsConfigChannel	Recording · Job · CLI logs
✅	UpdateToStable_TypeScriptSingleFileInit_KeepsConfigChannel	Recording · Job · CLI logs
✅	UpdateToStable_TypeScript_PreviewsStablePkgsAndKeepsChannel	Recording · Job · CLI logs

---

_{📹 Recordings uploaded automatically from CI run #27225236780}