Skip to content

Add cache balance weight for cache-aware routing - #1

Closed
weireweire wants to merge 1 commit into
pd-router-dp-awarefrom
pd-router-cache-balance-weight
Closed

Add cache balance weight for cache-aware routing#1
weireweire wants to merge 1 commit into
pd-router-dp-awarefrom
pd-router-cache-balance-weight

Conversation

@weireweire

Copy link
Copy Markdown
Owner

Summary

This stacked draft PR adds an optional cache_balance_weight knob to SMG cache-aware routing. A value of 0.0 preserves the existing strict best-prefix behavior. Positive values subtract a shadow-cache-size penalty from cache-match scoring, allowing common prefixes to spread across workers when the caller wants better cache footprint balance.

Changes include:

  • Adds cache_balance_weight to cache-aware policy config, validation, CLI, and Python bindings.
  • Adds per-tenant shadow cache footprint helpers used by cache-balanced selection.
  • Keeps low-match fallback on minimum worker load.
  • Adds unit coverage for default compatibility, argument plumbing, balance spreading, and strong-hit preservation.
  • Updates SMG docs for the new option.

Relationship

This is stacked on pd-router-dp-aware and should be rebased onto upstream main after the DP-aware PD router change lands.

Validation

  • git diff HEAD~1..HEAD --check

@weireweire

Copy link
Copy Markdown
Owner Author

Closing this fork-local PR; the change is being opened against sgl-project/sglang instead.

@weireweire weireweire closed this May 25, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant