Batch small Helix work items to reduce per-item overhead (-53% compute)

[mmitche]

Batch small Helix work items to reduce per-item overhead

Problem

Each ASP.NET Core CI build sends ~503 Helix work items, but most have only 2-10 seconds of actual test execution with ~20-40 seconds of per-item overhead (tool installs, vstest startup, result upload). This wastes ~5.6 compute-hours per build.

Solution

Batch compatible small test assemblies into groups of ~20 per Helix work item. Assemblies with special dependencies (IIS, Playwright, Java, Node, MSSQL) remain as individual items.

Measured Results

Metric	Baseline	Batched	Change
Work items	503	47	-91%
Total compute	634 min (10.6 hrs)	300 min (5.0 hrs)	-53%

Per-queue breakdown:

Queue	Baseline (items/min)	Batched (items/min)	Savings
Windows (vs2026)	189 / 300 min	23 / 148 min	-51%
macOS (osx.15)	156 / 239 min	11 / 88 min	-63%
Ubuntu (2404)	158 / 95 min	13 / 64 min	-32%

How it works

1. eng/helix/helix.proj - new BatchSmallWorkItems MSBuild inline task runs after the existing Gather target. Groups eligible items by TFM, creates combined payload directories, writes a targets.txt manifest.

2. eng/tools/HelixTestRunner - accepts --targets-file targets.txt to run dotnet test sequentially for each assembly. Tool installs happen once per batch. Test results are merged.

3. eng/helix/content/runtests.cmd / runtests.sh - detect @targets.txt prefix for batched mode. Fully backward compatible.

Batching rules

• Eligible: Items with no special PreCommands (no IIS, Playwright, Java, Node, MSSQL deps)
• Group by: Target framework (net11.0, net472, etc.)
• Max batch size: 20 assemblies per work item
• Failed tests: Batched items exit 0 so all test results are reported. Individual failures visible via test results XML.

CI validation

• Build 1432072 - Windows pass
• Build 1431657 - Ubuntu + Windows pass

[dotnet-policy-service]

Hey @dotnet/aspnet-build, looks like this PR is something you want to take a look at.

[mmitche]

@wtgodbe THis is an attempt to reduce the test overhead in aspnetcore. It is NOT ready (prototype included some hacks for testing)

[wtgodbe]

Love this idea! Will take a closer look next week, but worth noting that the helix tests are currently broken into 2 subsets:

aspnetcore/eng/Build.props

Lines 236 to 297 in 27c660e

	<ProjectsWithTestsSubset1 Include="
	$(RepoRoot)src\Framework\AspNetCoreAnalyzers\test\Microsoft.AspNetCore.App.Analyzers.Test.csproj;
	$(RepoRoot)src\Framework\test\Microsoft.AspNetCore.App.UnitTests.csproj;
	$(RepoRoot)src\Caching\**\*.*proj;
	$(RepoRoot)src\DefaultBuilder\**\*.*proj;
	$(RepoRoot)src\Features\**\*.*proj;
	$(RepoRoot)src\DataProtection\**\*.*proj;
	$(RepoRoot)src\Antiforgery\**\*.*proj;
	$(RepoRoot)src\Hosting\**\*.*proj;
	$(RepoRoot)src\Http\**\*.*proj;
	$(RepoRoot)src\HttpClientFactory\**\*.*proj;
	$(RepoRoot)src\Html.Abstractions\**\*.*proj;
	$(RepoRoot)src\Identity\**\*.*proj;
	$(RepoRoot)src\Servers\**\*.csproj;
	$(RepoRoot)src\Security\**\*.*proj;
	$(RepoRoot)src\SiteExtensions\Microsoft.Web.Xdt.Extensions\**\*.csproj;
	$(RepoRoot)src\SiteExtensions\LoggingAggregate\test\**\*.csproj;
	$(RepoRoot)src\Shared\**\*.*proj;
	$(RepoRoot)src\Tools\**\*.*proj;
	$(RepoRoot)src\Logging.AzureAppServices\**\src\*.csproj;
	$(RepoRoot)src\Middleware\**\*.csproj;
	"
	Exclude="
	@(ProjectToBuild);
	@(ProjectToExclude);
	$(RepoRoot)**\node_modules\**\*;
	$(RepoRoot)**\bin\**\*;
	$(RepoRoot)**\obj\**\*;"
	Condition=" '$(BuildMainlyReferenceProviders)' != 'true' " />
	<ProjectsWithTestsSubset2 Include="
	$(RepoRoot)src\Razor\**\*.*proj;
	$(RepoRoot)src\Mvc\**\*.*proj;
	$(RepoRoot)src\Azure\**\*.*proj;
	$(RepoRoot)src\SignalR\**\*.csproj;
	$(RepoRoot)src\StaticAssets\**\*.csproj;
	$(RepoRoot)src\Components\**\*.csproj;
	$(RepoRoot)src\Analyzers\**\*.csproj;
	$(RepoRoot)src\FileProviders\**\*.csproj;
	$(RepoRoot)src\Configuration.KeyPerFile\**\*.csproj;
	$(RepoRoot)src\Localization\**\*.csproj;
	$(RepoRoot)src\ObjectPool\**\*.csproj;
	$(RepoRoot)src\JSInterop\**\*.csproj;
	$(RepoRoot)src\WebEncoders\**\*.csproj;
	$(RepoRoot)src\HealthChecks\**\*.csproj;
	$(RepoRoot)src\Testing\**\*.csproj;
	$(RepoRoot)src\Grpc\**\*.csproj;
	$(RepoRoot)src\ProjectTemplates\**\*.csproj;
	$(RepoRoot)src\Extensions\**\*.csproj;
	$(RepoRoot)src\OpenApi\**\*.csproj;
	$(RepoRoot)src\Validation\**\*.csproj;
	"
	Exclude="
	@(ProjectToBuild);
	@(ProjectToExclude);
	$(RepoRoot)**\node_modules\**\*;
	$(RepoRoot)**\bin\**\*;
	$(RepoRoot)**\obj\**\*;"
	Condition=" '$(BuildMainlyReferenceProviders)' != 'true' " />
	<DotNetProjects Condition=" '$(HelixSubset)' == '' OR '$(HelixSubset)' == '1'" Include="@(ProjectsWithTestsSubset1)" />
	<DotNetProjects Condition=" '$(HelixSubset)' == '' OR '$(HelixSubset)' == '2'" Include="@(ProjectsWithTestsSubset2)" />

,

aspnetcore/.azure/pipelines/ci-public.yml

Lines 562 to 591 in 27c660e

	# Helix x64 subset 1
	- template: jobs/default-build.yml
	parameters:
	jobName: Helix_x64_Subset_1
	jobDisplayName: 'Tests: Helix x64 Subset 1'
	agentOs: Windows
	timeoutInMinutes: 240
	steps:
	# Build the shared framework
	- script: ./eng/build.cmd -ci -prepareMachine -nativeToolsOnMachine -all -pack -arch x64
	/p:CrossgenOutput=false /p:ASPNETCORE_TEST_LOG_DIR=artifacts/log $(_InternalRuntimeDownloadArgs)
	/p:VsTestUseMSBuildOutput=false $(HelixSubset1LogArgs)
	displayName: Build shared fx
	# -noBuildNative -noBuild to avoid repeating work done in the previous step.
	- script: ./eng/build.cmd -ci -prepareMachine -nativeToolsOnMachine -all -noBuildNative -noBuild -test
	-projects eng\helix\helix.proj /p:IsHelixPRCheck=true /p:IsHelixJob=true
	/p:CrossgenOutput=false /p:ASPNETCORE_TEST_LOG_DIR=artifacts/log $(_InternalRuntimeDownloadArgs)
	/p:VsTestUseMSBuildOutput=false /p:RunTemplateTests=false /p:HelixSubset=1
	displayName: Run build.cmd helix target
	env:
	HelixApiAccessToken: $(HelixApiAccessToken) # Needed for internal queues
	SYSTEM_ACCESSTOKEN: $(System.AccessToken) # We need to set this env var to publish helix results to Azure Dev Ops
	artifacts:
	- name: Helix_Subset_1_Logs_Attempt_$(System.JobAttempt)
	path: artifacts/log/
	publishOnError: true
	includeForks: true
	# Helix x64 subset 2

. We did this because parallelizing the helix tests cut about 40 minutes off the build - but the 2 groups were chosen completely arbitrarily. There might be some extra time to be saved by re-subsetting based on whatever "compatibility" heuristic you're currently using