fix: tighten context compression prompt and token budget - #1306
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adjust context compression so the model no longer spends extra time generating transient analysis, clamp compression-request output tokens with a temporary override, and simplify the retained compression contract wording.
Fixes #
Type and Areas
Type:
bug fix
Areas:
Rust core, AI adapters, runtime ports
Motivation / Impact
Context compression had two practical issues: the prompt required an
<analysis>section that was discarded anyway, and compression requests inherited the active modelmax_tokenswithout a dedicated cap, which could push some requests past the effective context window and trigger provider-side errors. This change makes compression summaries summary-only, caps compression output tokens tomin(model.max_tokens, 8192)with an8192fallback when the model has no explicit limit, and removes redundant contract duplication from the compression prompt. The retained contract text is also phrased more naturally for future rounds.Verification
pnpm run fmt:rscargo test -p bitfun-ai-adapters with_max_tokens_overrides_output_limit -- --nocapturecargo test -p bitfun-core model_summary_prompt_does_not_inline_compaction_contract -- --nocapturecargo test -p bitfun-core model_summary_prompt_requires_summary_only -- --nocapturecargo test -p bitfun-core compression_request_max_tokens_clamps_to_global_cap -- --nocapturecargo test -p bitfun-core build_compression_ai_client_overrides_only_max_tokens -- --nocapturecargo test -p bitfun-runtime-ports compression_contract_renders_model_visible_fields -- --nocaptureReviewer Notes
Checklist