Feature request: optional LLM image captioning for markitdown evidence prep

[rmonier]

Follows on from issue #3, "Markitdown integration for external-document evidence" — that issue's markitdown integration shipped without LLM image captioning; this file tracks that specific follow-up separately, as agreed when #3 was implemented.

Status

Not implemented, intentionally. .agents/skills/agent-ready-context/scripts/prepare_external_evidence.py
accepts image sources (.jpg/.jpeg/.png) today, but they carry limited value: only
EXIF/IPTC metadata is extracted (via the optional exiftool binary), with no visual
content extraction. This is a deliberate no-op, not a bug — this issue tracks turning it
into a real capability later, with full user disclosure, once the project has a
consistent story for provider credentials (see Resources). No hints or scaffolding for
this are wired into the skill yet; that is deliberate too, until this is actually adopted.

Why this isn't a regression

Verified before filing, not assumed: the OpenKB-era pipeline (predecessor to the current
OpenWiki-based pipeline) never had PDF/Office/image ingestion of any kind, and never had
LLM-based image analysis. Confirmed against:

• git show c275aca:.agents/skills/agent-ready-context/references/external-docs.md — the
  OpenKB-era doc covers URL-text evidence only (web tool fetch, or openkb add <url>); no
  mention of PDF, Office, images, OCR, or vision anywhere.
• The OpenKB migration project memory (sessions from 2026-07-07 through 2026-07-10) — an
  extensive, session-by-session record of OpenKB's actual capabilities; no image/vision/OCR
  feature appears anywhere in it.

So this is a net-new capability request, not a gap the OpenWiki migration reopened.
The two regressions that migration did reopen (no PDF/Office ingestion, no URL ingestion)
are already closed by the markitdown integration that shipped alongside this issue.

What's already verified (do not re-derive)

Checked directly against the installed markitdown 0.1.6 package source, not just its
docs:

• The CLI (markitdown <file>) has no flag for LLM image captioning. Confirmed by
  markitdown --help and by grepping __main__.py for llm_client/llm_model/
  llm_prompt — zero matches. The feature exists only on the Python MarkItDown class
  constructor: _markitdown.py
  (__init__/enable_builtins, around the llm_client/llm_model/llm_prompt kwargs).
• The actual call site is
  converters/_llm_caption.py:
  client.chat.completions.create(model=model, messages=[...]) — the standard OpenAI SDK
  shape, image passed as a base64 data URI. Any object duck-typed to that interface works,
  including openai.OpenAI(base_url=..., api_key=...) pointed at a local OpenAI-compatible
  engine (Ollama/vLLM/llama.cpp) — the same shape OpenWiki already uses for its air-gapped
  mode, so this can mirror that pattern structurally.
• Non-LLM image handling (what runs today) is in
  converters/_image_converter.py:
  EXIF/IPTC metadata only (ImageSize, Title, Caption, Description, Keywords,
  Artist, Author, DateTimeOriginal, CreateDate, GPSPosition), via the optional
  exiftool binary (auto-detected via the EXIFTOOL_PATH env var or PATH). No OCR
  anywhere in the package — verified with a repo-wide grep for ocr/tesseract; the only
  hit is inside the unwired Azure Document Intelligence converter, which does OCR remotely
  in Azure, never locally. Those metadata fields are usually empty for the
  screenshots/diagrams this pipeline actually processes, so today's image support is
  low-value by design, not by bug.
• Related, already shipped, for contrast: YouTube URL support (added alongside this issue)
  needs no LLM at all —
  converters/_youtube_converter.py
  uses youtube_transcript_api.YouTubeTranscriptApi plus a plain page fetch (bs4), no
  openai/llm_client anywhere in that file. It just retrieves YouTube's own pre-existing
  caption track over the network — same trust tier as fetching any webpage, no credentials
  needed. Do not conflate the two when reasoning about this issue.

Proposed shape (decided in principle, not built)

• New, separate optional script (not a flag on prepare_external_evidence.py) — keep
  that script dependency-free and CLI-only (its deliberate existing design); the new script
  would need real third-party deps (markitdown, openai) via PEP 723
  dependencies = [...], which has precedent in this skill (run_openwiki_staged.py
  already declares pyyaml).
• Credentials: standard OpenAI SDK env vars (OPENAI_API_KEY / OPENAI_BASE_URL),
  read internally by the real openai SDK (openai.OpenAI() with no explicit args) — this
  repo's own code never reads or touches the key value, matching the hands-off guarantee
  already used for OpenWiki credentials. Model name via an explicit --llm-model flag,
  never silently defaulted.
• Config loading: the user wants this sourced from project/user-scoped config rather
  than requiring the caller to export shell vars on every invocation — but wants it
  designed once, consistently, alongside the config mechanism planned for the langmem
  work in discussions#4, not as a
  markitdown-specific one-off. Resolve that shared mechanism first, then wire this script
  to it, rather than inventing a second config convention.
• Disclosure: before every call, state tool pin (markitdown + openai SDK versions),
  provider/model (from config, name only), endpoint family (host, not full URL/creds),
  credential source (env var / config key name only, never the value), content sent (the
  image file), and cost boundary — mirroring
  references/privacy-and-data-flows.md's existing OpenWiki disclosure rule. Never invoke
  without fresh consent; never silently pick a provider.
• Draft/review contract stays identical to the rest of this pipeline: still writes to
  okf/.okf-build/external/, never okf/external/ directly; same review checklist
  (prompt injection, PII, size) before promotion.

Open questions for the adopting session

1. The exact shared config mechanism (once the langmem discussion lands) — file format,
   precedence order (project vs. user vs. shell env), and whether it needs a new tiny
   loader or can reuse an existing library (e.g. python-dotenv).
2. Exact new script name and CLI shape, e.g.
   caption_image_with_llm.py --repo . --source <image> --llm-model <model> --resource <canonical-uri>.
3. Whether to cap image size/dimensions before base64-encoding (cost and payload-size
   guard) — _llm_caption.py sends the raw file bytes as-is today.
4. Whether images markitdown itself doesn't accept for this converter
   (ACCEPTED_FILE_EXTENSIONS = [".jpg", ".jpeg", ".png"] in _image_converter.py; no
   .gif/.webp/.bmp) should be pre-converted, or stay out of scope.

Files likely touched at adoption time

• NEW .agents/skills/agent-ready-context/scripts/<name>.py
• .agents/skills/agent-ready-context/references/external-docs.md — new section
• .agents/skills/agent-ready-context/references/dependencies.md — new pin row(s) for
  openai (and a version bump if markitdown's own pin needs to move)
• .agents/skills/agent-ready-context/references/privacy-and-data-flows.md — new
  data-flow row + disclosure text
• .agents/skills/agent-ready-context/SKILL.md — workflow step + command block
• .agents/skills/agent-ready-context/scripts/check_prereqs.py — optional tool check
• tests/test_openwiki_adapter.py — new test class, CLI/LLM calls mocked, never required
  in CI
• AGENTS.md — new pin row + disclosure note, at adoption time only; no version literals
  belong in the skill files themselves

Resources

• markitdown repo: https://github.com/microsoft/markitdown (MIT; pinned version
  currently 0.1.6 per this repo's AGENTS.md)
• Verified source, pinned to v0.1.6:
  • https://github.com/microsoft/markitdown/blob/v0.1.6/packages/markitdown/src/markitdown/_markitdown.py
  • https://github.com/microsoft/markitdown/blob/v0.1.6/packages/markitdown/src/markitdown/converters/_llm_caption.py
  • https://github.com/microsoft/markitdown/blob/v0.1.6/packages/markitdown/src/markitdown/converters/_image_converter.py
  • https://github.com/microsoft/markitdown/blob/v0.1.6/packages/markitdown/src/markitdown/converters/_youtube_converter.py
    (for contrast: no LLM used, network-only via youtube_transcript_api)
• OpenAI Python SDK (the client shape _llm_caption.py expects):
  https://github.com/openai/openai-python
• This repo's shared-config discussion — resolve before wiring credentials here:
  Proposal: add harness-captured, LangMem-assisted repository dreaming to the `agent-ready-context` lifecycle #4
• This repo's existing disclosure pattern to mirror:
  .agents/skills/agent-ready-context/references/privacy-and-data-flows.md,
  .agents/skills/agent-ready-context/references/openwiki-providers.md

Acceptance criteria (draft, refine at adoption time)

1. LLM image captioning is strictly optional; every existing flow and gate passes with it
   absent, same as markitdown itself is optional today.
2. No credential value is ever read, printed, or logged by this repo's own code — the
   openai SDK (or equivalent) reads it directly from the resolved config/env.
3. Full disclosure (tool pin, provider/model, endpoint, credential source, content sent,
   cost boundary) before every call; consent is per-call, not standing.
4. Tests pass without the real markitdown/openai packages installed (mocked), never
   required in CI.
5. No version literals in skill files themselves — pins live only in the consuming
   repository's AGENTS.md, same policy as every other tool here.