Add multi-control transfer inference#69
Merged
Merged
Conversation
Port multi-control transfer inference from imaginaire4
(commit 0aafd95958) into cosmos_framework, remapping
projects/cosmos3/vfm/* -> cosmos_framework/{model,data}/vfm/* and
packages/cosmos3/cosmos3/* -> cosmos_framework/inference/*.
N independent maskless SDPA passes (one per control), with the noisy
output as the weighted sum across controls; N=1, weight=1.0 reproduces
two_way_attention exactly. Adds a per-control `weight` arg/override,
threads control_weights through the batch -> GenerationDataClean ->
PackedSequence -> SplitInfo, and routes to the new
multi_control_two_way_attention in dispatch_attention.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OverridesBase._build resolves overrides via model_validate(model_dump()) without exclude_none, so an unset `weight` arrives as None at the resolved TransferArgs.weight (typed float), failing pydantic validation (edge.weight: float_type, input_value=None). This broke the generator-inference-smoke transfer_edge run, whose JSON sets control_path and preset_edge_threshold but not weight. Add a before-validator on TransferArgs.weight that maps None -> 1.0 (neutral / single-control parity), keeping the I4 field signatures intact. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
yy-code-nv
approved these changes
Jun 30, 2026
Dinghow
approved these changes
Jun 30, 2026
foreverlms
approved these changes
Jun 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Port multi-control transfer inference from imaginaire4 (commit 0aafd95958) into cosmos_framework, remapping projects/cosmos3/vfm/* -> cosmos_framework/{model,data}/vfm/* and packages/cosmos3/cosmos3/* -> cosmos_framework/inference/*.
N independent maskless SDPA passes (one per control), with the noisy output as the weighted sum across controls; N=1, weight=1.0 reproduces two_way_attention exactly. Adds a per-control
weightarg/override, threads control_weights through the batch -> GenerationDataClean -> PackedSequence -> SplitInfo, and routes to the new multi_control_two_way_attention in dispatch_attention.