Skip to content

feat: support image URLs (fileUri) in image input fields#522

Open
gennadijbgl wants to merge 3 commits into
ax-llm:mainfrom
gennadijbgl:feat/image-url-support
Open

feat: support image URLs (fileUri) in image input fields#522
gennadijbgl wants to merge 3 commits into
ax-llm:mainfrom
gennadijbgl:feat/image-url-support

Conversation

@gennadijbgl

Copy link
Copy Markdown

What

Adds URL support to image input fields, mirroring the existing file field's data / fileUri pattern. Images can now be supplied by URL instead of only inline base64.

const analyze = ax('photo:image -> answer:string');

// existing — inline base64
await analyze.forward(llm, { photo: { mimeType: 'image/jpeg', data } });

// new — by URL
await analyze.forward(llm, {
  photo: { mimeType: 'image/jpeg', fileUri: 'https://example.com/photo.jpg' },
});

An image field value now accepts either { mimeType, data } (base64, unchanged) or { mimeType, fileUri } (URL). Supplying both, or neither, is rejected — same rule the file type already enforces.

How

The internal image content block gains a second union member carrying fileUri (parallel to how file is modeled). Each provider maps the URL variant to its native form:

Provider base64 (existing) URL (new)
OpenAI image_url.url = data URI image_url.url = the URL directly
Anthropic source: { type: 'base64', … } source: { type: 'url', url }
Gemini inlineData: { data } fileData: { fileUri } (e.g. gs:// / Files API URIs)

Validation (dsp/util.ts, dsp/prompt.ts, ai/validate.ts), content hashing (ai/base.ts), logging (dsp/loggers.ts) and the unsupported-provider content processor (ai/processor.ts) were updated to narrow on the URL variant. AxFieldValue exposes the new shape to callers.

Note on Gemini: fileData.fileUri accepts Google-hosted URIs (Files API / gs://), consistent with the existing file behavior — not arbitrary public http(s) URLs. OpenAI and Anthropic accept public URLs.

Tests

  • dsp/prompt.test.ts — image render with data, with fileUri, mixed arrays, and validation (missing mimeType / missing both / both present).
  • google-gemini/api.test.ts — request mapping: base64 → inlineData, fileUrifileData.
  • Type-check clean; full unit suite green (the 2 unrelated jsRuntime sandbox failures are a Windows-only Node permission-model artifact).

🤖 Generated with Claude Code

gennadijbgl and others added 3 commits May 29, 2026 16:13
Mirror the existing `file` field's data/fileUri pattern for `image`
fields so images can be supplied by URL instead of inline base64.

An image field value now accepts either `{ mimeType, data }` (base64,
existing) or `{ mimeType, fileUri }` (URL). Providers map the URL
variant natively:
- OpenAI: passes the URL straight to `image_url.url`
- Anthropic: emits `source: { type: 'url', url }`
- Gemini: emits `fileData: { fileUri }` (e.g. gs:// / Files API URIs)

Validation, hashing, logging and content-processing paths updated to
handle the URL variant. Adds prompt-render/validation tests and Gemini
request-mapping tests.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Fold the now-identical image/file validators in dsp/util.ts into a
single `validMediaContent` (mimeType + exactly one of data|fileUri),
and collapse the six duplicated Gemini image/file content blocks into
one `toGeminiMediaPart` helper.

No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Collapse the duplicated `'fileUri' in validated ? … : …` ternary in the
array and non-array image-render branches into one local `toImagePart`,
matching the `toGeminiMediaPart` style. No behavior change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dosco

dosco commented Jun 20, 2026

Copy link
Copy Markdown
Collaborator

sorry was busy with big changes to multi language codebase will review this in a day or so.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants