EricSun0218 · EricSun0218 · Aug 15, 2026 · Aug 15, 2026 · Aug 15, 2026
@@ -2,13 +2,16 @@
 
 ## Unreleased
 
-- Add the optional `TaskPlanExtension` for session/actor-scoped persistent ordered checklists, revision-checked mutations, host-validated evidence, per-input advancement guards, pending-work routing, typed UI projection events, and bounded terminal retention.
-- Add typed, model-free host queries for persisted goals and task plans, including session revisions, and scope goal-change events with their session/actor key and input ID.
-- Add batched, payload-free mailbox pending-status queries that distinguish ready work from active leases without claiming delivery or incrementing attempts.
-- Add backward-compatible durable task-plan pause/resume with revision checks, preserved in-progress steps, non-runnable paused routing, typed change reasons, and restart coverage.
+No changes yet.
 
 ## 0.3.0-alpha.2
 
+- Add durable image input for game observations: bounded PNG/JPEG/WebP/GIF decode admission, immutable content-addressed local objects, reference-only transcripts, provider/model preflight before reads, tool-result image persistence, JSON/SSE transport, and owner-authorized retrieval.
+- Document the recommended large-world perception stack: bounded structured state, sparse BEV/topological summaries, selective screenshots, exact query tools, and deterministic game-owned execution rather than raw voxel dumps.
+- Add the optional `TaskPlanExtension` for session/actor-scoped persistent ordered checklists, revision-checked mutations, host-validated evidence, per-input advancement guards, pending-work routing, typed UI projection events, and bounded terminal retention.
+- Add typed, model-free host queries for persisted goals and task plans, including session revisions, and scope goal-change events with their session/actor key and input ID.
+- Add batched, payload-free mailbox pending-status queries that distinguish ready work from active leases without claiming delivery or incrementing attempts.
+- Add durable task-plan pause/resume with revision checks, preserved in-progress steps, non-runnable paused routing, typed change reasons, and restart coverage.
 - Add the optional `OpenGameAgent.Memory` package with a model-agnostic embedding provider contract, authoritative-save verification, rebuildable local vector indexes, hybrid lexical/vector recall, structured diagnostics, and game-time-aware reranking.
 - Add deterministic authoritative memory snapshots for in-memory and local-file stores so derived indexes can be rebuilt explicitly after embedding model or preprocessing changes.
 - Document local source references and game-provided local embedding integration, including BGE-M3-compatible query/document adapters and save boundaries.

@@ -102,6 +102,12 @@ Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "OpenGameAgent.Memory", "src
 EndProject
 Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "OpenGameAgent.Memory.Tests", "tests\OpenGameAgent.Memory.Tests\OpenGameAgent.Memory.Tests.csproj", "{5E8B096B-DD5F-4463-B841-7675F560B52D}"
 EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "OpenGameAgent.Attachments", "src\OpenGameAgent.Attachments\OpenGameAgent.Attachments.csproj", "{02764E37-C515-48D2-BBFA-BA9C8C71425C}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "OpenGameAgent.Attachments.Local", "src\OpenGameAgent.Attachments.Local\OpenGameAgent.Attachments.Local.csproj", "{967FB56B-BC1A-4FEF-A2C8-289CED38A733}"
+EndProject
+Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "OpenGameAgent.Attachments.Tests", "tests\OpenGameAgent.Attachments.Tests\OpenGameAgent.Attachments.Tests.csproj", "{F0ECE44A-ABCF-482C-9F81-D878B28C250D}"
+EndProject
 Global
 	GlobalSection(SolutionConfigurationPlatforms) = preSolution
 		Debug|Any CPU = Debug|Any CPU
@@ -300,6 +306,18 @@ Global
 		{5E8B096B-DD5F-4463-B841-7675F560B52D}.Debug|Any CPU.Build.0 = Debug|Any CPU
 		{5E8B096B-DD5F-4463-B841-7675F560B52D}.Release|Any CPU.ActiveCfg = Release|Any CPU
 		{5E8B096B-DD5F-4463-B841-7675F560B52D}.Release|Any CPU.Build.0 = Release|Any CPU
+		{02764E37-C515-48D2-BBFA-BA9C8C71425C}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{02764E37-C515-48D2-BBFA-BA9C8C71425C}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{02764E37-C515-48D2-BBFA-BA9C8C71425C}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{02764E37-C515-48D2-BBFA-BA9C8C71425C}.Release|Any CPU.Build.0 = Release|Any CPU
+		{967FB56B-BC1A-4FEF-A2C8-289CED38A733}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{967FB56B-BC1A-4FEF-A2C8-289CED38A733}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{967FB56B-BC1A-4FEF-A2C8-289CED38A733}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{967FB56B-BC1A-4FEF-A2C8-289CED38A733}.Release|Any CPU.Build.0 = Release|Any CPU
+		{F0ECE44A-ABCF-482C-9F81-D878B28C250D}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
+		{F0ECE44A-ABCF-482C-9F81-D878B28C250D}.Debug|Any CPU.Build.0 = Debug|Any CPU
+		{F0ECE44A-ABCF-482C-9F81-D878B28C250D}.Release|Any CPU.ActiveCfg = Release|Any CPU
+		{F0ECE44A-ABCF-482C-9F81-D878B28C250D}.Release|Any CPU.Build.0 = Release|Any CPU
 	EndGlobalSection
 	GlobalSection(NestedProjects) = preSolution
 		{01759D73-7B80-47A2-9D7D-154CC64C6851} = {EA3AF59A-9A1C-4197-B2A3-F93894D131B8}
@@ -334,5 +352,8 @@ Global
 		{5697E98C-2249-4D4C-894B-CB0A8732238E} = {86AE6217-BFEE-4349-945A-70ECEC211437}
 		{00AE7836-01FA-4151-A38A-8263D9164A75} = {EA3AF59A-9A1C-4197-B2A3-F93894D131B8}
 		{5E8B096B-DD5F-4463-B841-7675F560B52D} = {86AE6217-BFEE-4349-945A-70ECEC211437}
+		{02764E37-C515-48D2-BBFA-BA9C8C71425C} = {EA3AF59A-9A1C-4197-B2A3-F93894D131B8}
+		{967FB56B-BC1A-4FEF-A2C8-289CED38A733} = {EA3AF59A-9A1C-4197-B2A3-F93894D131B8}
+		{F0ECE44A-ABCF-482C-9F81-D878B28C250D} = {86AE6217-BFEE-4349-945A-70ECEC211437}
 	EndGlobalSection
 EndGlobal
@@ -22,7 +22,7 @@ OpenGameAgent is a compact, hackable C# runtime that lets game characters observ
 
 OpenGameAgent starts with a small, composable agent kernel. The stateful core streams model output, executes validated tools, accepts steering while running, and continues the model/tool loop until work is complete. Use the kernel by itself, add the game runtime for game time and durable state, then opt into extensions for memory, goals, host-verified task plans, artifacts, delegation, external tools, structured interaction, and workflow graphs.
 
-Inputs are bounded JSON. They may represent dialogue, combat observations, simulation ticks, UI events, plans, sensor state, or any other game-owned data; natural language is optional. No model is bundled. Cloud and local API endpoints are both supported.
+Inputs are bounded JSON plus optional durable image observations. They may represent dialogue, combat observations, simulation ticks, UI events, plans, sensor state, screenshots, or any other game-owned data; natural language is optional. No model is bundled. Cloud and local API endpoints are both supported.
 
 ## A programmable agent runtime built for games
 
@@ -57,6 +57,7 @@ Install the complete game runtime from NuGet:
 ```bash
 dotnet add package OpenGameAgent --version 0.3.0-alpha.2
 dotnet add package OpenGameAgent.Memory --version 0.3.0-alpha.2 # optional semantic memory
+dotnet add package OpenGameAgent.Attachments.Local --version 0.3.0-alpha.2 # optional durable image input
 ```
 
 The kernel, persistence, providers, and engine-compatible client are also published as separate `OpenGameAgent.*` packages. Godot, Unity, and portable server archives are available on the [Releases](https://github.com/EricSun0218/OpenGameAgent/releases) page. See [Getting started](docs/getting-started.md) and [Engine integration](docs/engine-integration.md) before connecting a game.
@@ -69,6 +70,7 @@ OpenGameAgent keeps the reusable agent machinery independent from the game while
 
 - named timelines and integer ticks, with optional calendar JSON;
 - structured observations and context slices with floating-point values intact;
+- content-addressed screenshot/image input with decode validation, model-capability preflight, and session-authorized retrieval;
 - quick-response, full-agent, and deterministic-workflow routes;
 - per-actor serialization with bounded cross-actor concurrency;
 - journaled action intents and authoritative game receipts;
@@ -116,6 +118,7 @@ Read [Architecture](docs/architecture.md) for the ownership and failure boundari
 | Agent kernel | Streaming typed messages, tool loop, typed partial tool results, steering, follow-up, hooks, cancellation, strict transcript validation, provider failures as results |
 | Tool execution | Provider-request schema preflight plus execution-time validation over a bounded JSON Schema subset, guaranteed result for every accepted call, safe parallel reads, conflict-key serialization, policy blocking/termination, timeouts, uncertain write outcomes |
 | Game runtime | Arbitrary JSON input, game clocks/timelines, fast/full/workflow routing, optimistic sessions, duplicate-input protection, actor concurrency, active-run steering/abort |
+| Image input | PNG/JPEG/WebP/GIF admission, immutable content-addressed storage, reference-only transcripts, capability preflight, tool-result images, and authorized server retrieval |
 | Extension API | Immutable builder; prompt/context/tool/skill/route/workflow/hook/provider/service registration; typed lifecycle events and channels; namespaced persistent state |
 | Official extensions | Tool policy and search, structured player questions/recommended replies, goals, host-verified ordered task plans with durable pause/resume, memory, artifacts, knowledge, delegation, tracing, and durable parallel workflow graphs |
 | World primitives | Durable actions, resumable workflows, memories, skills, signals, game-time schedules, actor mailboxes with batch read-only pending status |
@@ -224,6 +227,7 @@ Real-editor gates are documented in [Engine integration](docs/engine-integration
 - [Engine integration](docs/engine-integration.md)
 - [Deployment and security](docs/deployment-and-security.md)
 - [Generated media](docs/media.md)
+- [Image input and game perception](docs/image-input.md)
 
 ## Project boundary
 

@@ -22,7 +22,7 @@ OpenGameAgent 是一个紧凑、可修改的 C# Runtime，让游戏角色能够
 
 OpenGameAgent 从小型、可组合的 Agent 内核出发。有状态核心会流式接收模型输出、执行经过校验的工具、在运行中接受 steering，并持续进行模型/工具循环直到任务结束。开发者既可以只使用内核，也可以叠加游戏 Runtime 获得游戏时间与可靠状态，再按需加入记忆、目标、宿主证据校验的任务清单、产物、委派、外部工具、结构化交互和工作流图等扩展。
 
-输入是有大小限制的 JSON，可以表示对话、战斗观察、模拟 Tick、UI 事件、计划、传感状态或任意游戏数据，不要求是自然语言。项目不捆绑模型，同时支持云端和本地 API。
+输入是有大小限制的 JSON，并可携带持久化图片观察，可以表示对话、战斗观察、模拟 Tick、UI 事件、计划、传感状态、截图或任意游戏数据，不要求是自然语言。项目不捆绑模型，同时支持云端和本地 API。
 
 ## 为游戏构建的可编程 Agent Runtime
 
@@ -57,6 +57,7 @@ OpenGameAgent 不绑定任何模型或 Provider。角色通过开发者定义的
 ```bash
 dotnet add package OpenGameAgent --version 0.3.0-alpha.2
 dotnet add package OpenGameAgent.Memory --version 0.3.0-alpha.2 # 可选语义记忆
+dotnet add package OpenGameAgent.Attachments.Local --version 0.3.0-alpha.2 # 可选持久图片输入
 ```
 
 内核、持久化、模型提供方和引擎兼容客户端也分别提供 `OpenGameAgent.*` 包。Godot、Unity 与可移植服务端压缩包可以从 [Releases](https://github.com/EricSun0218/OpenGameAgent/releases) 页面下载。接入游戏前请阅读[快速开始](docs/getting-started.md)和[引擎接入](docs/engine-integration.md)。
@@ -69,6 +70,7 @@ OpenGameAgent 不替游戏规定玩法，而是提供可复用的游戏坐标与
 
 - 命名时间线、整数 Tick 和可选日历 JSON；
 - 保留浮点数的结构化观察与上下文；
+- 经真实解码校验、内容寻址持久化、模型能力预检与会话授权读取的截图/图片输入；
 - 快速回复、完整 Agent、确定性 Workflow 三种路由；
 - 同一角色串行、不同角色有界并行；
 - 先记日志的动作意图与游戏权威回执；
@@ -114,6 +116,7 @@ GameAgentRuntime
 | Agent 内核 | 流式类型化消息、工具循环、类型化工具中间结果、steering、follow-up、hooks、取消、严格会话校验、提供方错误结果化 |
 | 工具执行 | provider 请求前 schema 预检及执行期有界 JSON Schema 子集校验、每个已接受调用都有结果、安全并行读、冲突键串行、策略拦截/终止、超时与写入结果未知语义 |
 | 游戏 Runtime | 任意 JSON 输入、游戏时钟/时间线、快速/完整/Workflow 路由、乐观并发会话、输入去重、角色并发、运行中 steering/abort |
+| 图片输入 | PNG/JPEG/WebP/GIF 准入、不可变内容寻址存储、仅引用会话、模型能力预检、工具结果图片与授权服务端读取 |
 | 扩展 API | 不可变构建器；提示词/上下文/工具/Skills/路由/Workflow/Hooks/提供方/服务注册；类型化生命周期事件与通道；命名空间持久状态 |
 | 官方扩展 | 工具策略与搜索、玩家结构化提问/推荐回复、目标、支持持久暂停/恢复且由宿主校验证据的有序任务清单、记忆、产物、外部知识、委派、追踪和可持久并行工作流图 |
 | 世界原语 | 可恢复动作、可续跑 Workflow、记忆、Skills、信号、游戏时间调度、支持批量只读待处理状态的角色邮箱 |
@@ -223,6 +226,7 @@ dotnet test OpenGameAgent.sln -c Release --no-build --no-restore
 - [引擎集成](docs/engine-integration.md)
 - [部署与安全](docs/deployment-and-security.md)
 - [生成式媒体](docs/media.md)
+- [图片输入与游戏感知](docs/image-input.md)
 
 ## 项目边界
 

@@ -17,7 +17,7 @@ The kernel owns one stateful model/tool loop:
 7. apply steering or follow-up messages;
 8. continue until the model stops, a hook stops the run, cancellation occurs, or a limit is reached.
 
-It knows nothing about NPCs, worlds, inventories, or engines. Its canonical values are typed content parts (`text`, `json`, `resource`, `reasoning`, and `tool_call`), messages, model requests, tools, and events.
+It knows nothing about NPCs, worlds, inventories, or engines. Its canonical values are typed content parts (`text`, `json`, `resource`, durable `image_attachment`, `reasoning`, and `tool_call`), messages, model requests, tools, and events. Inline image bytes are request-boundary input; canonical history stores only immutable attachment references.
 
 `Agent` owns mutable transcript and queue state. `AgentLoop` is the lower-level execution function. A host that already owns state can call the loop directly; most integrations should keep an `Agent` or use `GameAgentRuntime`.
 
@@ -45,6 +45,7 @@ It does not own a universal world model. Context remains opaque JSON supplied by
 - `OpenGameAgent.Models` adds provider/model catalogs, capability-aware selection, reasoning levels, cost metadata, dynamic refresh, and replaceable authentication.
 - `OpenGameAgent.Models.BuiltIn` turns the bundled directory into an executable multi-provider model runtime; `OpenGameAgent.Models.Auth.BuiltIn` adds explicitly configured browser and device authorization flows.
 - `OpenGameAgent.ProviderTransport` centralizes bounded response observations, header guards, and retry metadata without adding HTTP concepts to the kernel.
+- `OpenGameAgent.Attachments` defines immutable image references and storage admission; `OpenGameAgent.Attachments.Local` provides a content-addressed local implementation with real decode and integrity checks.
 - `OpenGameAgent.Media` routes image, audio, and video generation by provider/model capability while keeping generation jobs outside the text/tool protocol.
 - `OpenGameAgent.Connectors.Mcp` exposes external tool servers through one lazy, searchable tool by default. Direct tool exposure is an explicit opt-in.
 - Provider, persistence, engine, client, and server packages stay replaceable and do not change kernel semantics.
@@ -106,6 +107,8 @@ Transcript compaction is also a provider-view operation. The included summarizin
 
 Context admission runs before the first request, after tool turns, and again after final request hooks. A hook therefore cannot accidentally bypass the configured context window. Large text or JSON tool results can be moved into the artifact store and replaced with a bounded handle and preview. This keeps canonical results recoverable without repeatedly paying their full context cost.
 
+Image admission follows the same canonical/request-view split. Inline user or tool-result images are fully validated and persisted before they enter session history. The active provider/model is preflighted, then immutable references are resolved into bytes only for the outgoing model request. System and assistant images are rejected; generated assets use the media pipeline. See [Image input and game perception](image-input.md).
+
 The system prompt keeps the most reusable bytes first: base instructions, then selected skills, then mutable authoritative game context. This ordering preserves the longest possible provider-cache prefix when world state changes, without moving dynamic state out of the game-owned context boundary.
 
 After a tool turn, `GameAgentRuntime` refreshes authoritative context, tools, and selected skills by default before the next model request. A configured next-turn hook can supply an explicit replacement context instead. Active game-layer runs can also be steered or aborted by `GameSessionKey`; messages never cross actor lanes.