Skip to content

fix(vision): replace image numbering with content-hash ids#19

Open
flobernd wants to merge 1 commit into
profile-routingfrom
image-id-single-source
Open

fix(vision): replace image numbering with content-hash ids#19
flobernd wants to merge 1 commit into
profile-routingfrom
image-id-single-source

Conversation

@flobernd

@flobernd flobernd commented Jul 17, 2026

Copy link
Copy Markdown
Owner

Summary

Replaces the image agent's sequential [Image #N] numbering with content-derived <image id="..."/> markers. Each stripped image's id is the first 16 hex chars of the SHA-256 of its image_url, so ids are stable across requests and history rewrites (e.g. client-side compaction), identical bytes dedupe to one cache entry, and a client's own numeric markers can never collide with a gateway id.

Intention

Clients like Claude Code write session-continuous [Image #N] markers into user text, while the strip renumbered image parts from 1 per request. Two numbering schemes of identical shape in the same turn made the model follow the client's numbers and call analyzeImage with ids that missed the per-session cache (or, in a long session, exact-hit the wrong image), burning analysis rounds. Content-hash ids remove the collision class entirely: user text passes through verbatim (no harness-specific rewriting), and a wrong id can only miss the cache, never silently resolve to the wrong image.

Changes

  • The strip replaces each image with an <image id="..."/> marker carrying its content-derived id; the placeholder text, system-prompt instruction, and imageId tool-parameter description consistently instruct one batched analyzeImage call with ids copied verbatim, ignoring any other numbering in the text.
  • Resolution is exact-only: VisionRequest reports unresolved_ids, and the executor injects a model-visible note listing the session's valid ids in document order so the model corrects its call in one round. The note hedges for the rare case where a correctly copied id is no longer cached (LRU-evicted or TTL-expired). Partial misses analyze the resolved images and append the correction; miss rounds still count against the analysis ceiling.
  • README: the image analysis section now states the strip's actual scope - images are stripped from every user message, and residual images are file_id images or images in non-user messages.

Testing

Full cargo test green (1070 passed, 0 failed across all suites), cargo fmt --check clean, cargo clippy --all-targets zero warnings.

Notes

Stacked on local-inference-lab#36's branch; retarget or rebase once that merges.

@flobernd
flobernd force-pushed the image-id-single-source branch from 5d0e5db to 3492e9b Compare July 18, 2026 07:49
Clients like Claude Code write their own session-continuous [Image #N]
markers into user text, while the strip renumbered image parts per
request starting at 1. Two numbering schemes of identical shape in the
same turn made the model follow the client's numbers and call
analyzeImage with ids that missed the per-session cache (or, in a long
session, exact-hit the WRONG image), burning analysis rounds.

Give each stripped image an <image id="..."/> marker whose id is the
first 16 hex chars of the SHA-256 of its image URL. A client's numeric
markers can never collide with such an id, so user text is left
verbatim (no harness-specific rewriting), and a wrong id can only MISS
the cache, never silently resolve to the wrong image. Misses inject a
model-visible note listing the session's valid ids so the model
corrects itself in one round; the note also hedges for the rare case
where a correctly copied id is no longer cached (LRU-evicted or
TTL-expired). Miss rounds still count against the analysis ceiling.
Content-derived ids are also stable across requests and history
rewrites (e.g. client-side compaction), and identical bytes dedupe to
one cached entry.

Also align the README's image analysis section with the strip's actual
scope: images are stripped from every user message, and residuals are
file_id images or images in non-user messages.
@flobernd
flobernd force-pushed the image-id-single-source branch from 3492e9b to d555f9d Compare July 18, 2026 10:36
@flobernd flobernd changed the title fix(vision): make stripped placeholders the single image-numbering source fix(vision): replace image numbering with content-hash ids Jul 18, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant