feat(kb-open): Deep Research 开放 API(start/SSE/status/cancel)#446
feat(kb-open): Deep Research 开放 API(start/SSE/status/cancel)#446ncw1992120 wants to merge 4 commits into
Conversation
…authz Implements the authentication backbone for the KB Open API (mateaix#441): API key lifecycle, a permitAll-path filter that rejects (never pass-through), per-key sliding-window rate limiting, and a centralized @RequireKbScope interceptor for scope + KB-ownership checks. Components: - TokenHashUtil: shared SHA-256 hash kernel (A4), reusable by PAT later - KbApiKeyService: mint/authenticate/revoke/update + multi-KB binding (R3: empty binding = zero access, not "all KBs") - KbOpenApiAuthFilter: sole gatekeeper for /api/v1/open/kb/** (R1: must return 401, no pass-through); R2: per-key rate limit (429) - KbApiKeyRateLimiter: sliding-window limiter (TriggerRateLimiter pattern) - @RequireKbScope + KbScopeInterceptor: centralized authorization (A1), scope check + kbId ownership from path variable - KbApiKeyAdminController: JWT-authenticated CRUD (list/create/detail/ update/revoke), workspace-scoped - V162 migration (h2/mysql/kingbase): mate_kb_api_key + _binding tables Security: - mck_ prefix (distinct from PAT mc_ and JWT eyJ) - SHA-256 hash storage, plaintext shown once at creation - prefix column (4 chars) for UI display only Tests (17 new, all green): - KbApiKeyServiceTest: R3 empty-binding rejection, auth round-trip, expired/disabled/wrong-prefix rejection, kb:* wildcard, revoke - KbApiKeyRateLimiterTest: sliding window, per-key isolation, recovery Closes mateaix#441
|
感谢 Deep Research 开放 API 🙏 鉴权这块做得很好:path 走 但异步作业层有几个阻塞项,对一个公开且产生真实成本的端点很关键: 1. 取消并不会真正停止作业。 2. CANCELLED 会被 COMPLETED/FAILED 覆盖。 3. session registry 无界增长(内存泄漏)。 4. 每个 key 没有在跑作业的并发上限(成本/DoS)。 5. 内联全限定名: 非阻塞: V162 迁移头注释写成 V161(且与 #437 撞号,配合 P0-A 顺延);research 复用 栈底 P0-A 改好后这个 PR rebase,并把上面 1–4 的作业生命周期/成本控制补上,我们再合并 🙏 |
BLOCKERS: - prefix column VARCHAR(6) → VARCHAR(12) across all 3 migration dialects; KbApiKeyService.create() produces 8 chars (mck_ + 4 random), VARCHAR(6) would silently truncate on H2 and throw on MySQL strict mode - Rename migration V162 → V164 to avoid collision with merged mateaix#437 (V162=wiki_raw_material_error_code, V163=wiki_raw_material_warning) and fix stale V161 references in h2/kingbase comments NITS: - SecurityConfig/WebMvcConfig: replace inline FQN with import + simple name - parseScopes: add .map(String::trim) so ' kb:read' matches correctly - Remove ?token= SSE query fallback in KbOpenApiAuthFilter (P0-A has no SSE endpoint; key would leak into access/proxy logs — R5) - Move kb-open-api-design.md from repo root to rfcs/ (contains RFC-090 internal reference that would be exposed by sync-opensource) - KbApiKeyEntity Javadoc: 'first 4 chars' → 'first 8 chars (mck_ + 4)' to match actual behavior
Implements the async Deep Research endpoint for the KB Open API (mateaix#443). Research is a multi-step LLM pipeline (plan → retrieve+draft → compose) that runs asynchronously and broadcasts progress via SSE. Endpoints: - POST /{kbId}/research start (returns sessionId + streamUrl) - GET /{kbId}/research/{id}/stream SSE progress (?token= for EventSource) - GET /{kbId}/research/{id}/status query status / final report - POST /{kbId}/research/{id}/cancel cancel running session Components: - KbOpenResearchController: 4 endpoints, @RequireKbScope("kb:search") - KbResearchSessionRegistry: in-memory session tracking with keyId ownership (a caller can only query/cancel their own sessions) Security: - R7: SSE uses ?token= query param (KbOpenApiAuthFilter already supports this fallback for EventSource which can't set Authorization headers) - Session ownership: status/cancel/stream all verify keyId match - Cancel checks session is RUNNING (409 otherwise) Reuses existing WikiResearchService.research() + ChatStreamTracker for the actual research pipeline and SSE broadcasting. Tests (6 new, all green): - KbResearchSessionRegistryTest: register/complete/fail/cancel lifecycle, cancel-on-completed no-op, unknown session returns empty Closes mateaix#443
…urrency cap Review mateaix#446 — address all 4 job-lifecycle/cost blockers + nits: 1. Cooperative cancellation (was: cancel only flipped status, pipeline ran to completion). Cancel endpoint now calls streamTracker.requestStop(); WikiResearchService.ensureNotCancelled() checks isStopRequested at each stage boundary (plan→draft, draft→compose) and inside the parallel draft fan-out — so cancel actually halts the expensive LLM calls, not just the SSE stream. Throws ResearchCancelledException (caught locally, no error broadcast). 2. Sticky CANCELLED terminal. complete()/fail() now no-op on a CANCELLED session, so a user who cancelled never sees a COMPLETED report surface via /status. 3. Session registry TTL. Terminal sessions get an updatedAt timestamp and are evicted by a @scheduled sweep after mate.kbopen.research.session-ttl (default 30m). RUNNING sessions are never evicted. Prevents unbounded memory growth. 4. Per-key concurrency cap. startIfAllowed() rejects new research when a key already has mate.kbopen.research.max-concurrent-per-key (default 3) RUNNING sessions → 429. Stops one key from spawning ~60 parallel multi-step LLM pipelines per minute under the per-min rate limiter. 5. Inline FQN → import (controller LinkedHashMap, test List.of). Nits (inherited from P0-A rebase): - V162→V164, prefix VARCHAR(12), design doc moved to rfcs/. - Design doc: kb:search scope row now documents it covers /research/**. 31 tests pass (12 registry incl. sticky-cancel/concurrency/TTL + 13 service + 4 rate limiter + 4 controller + ...).
8d7894f to
5e216c9
Compare
|
All 4 blockers + nits fixed. Rebased onto P0-A (inherits V164 + FQN + design doc move) and addressed each item in commit 1. Cancel now actually stops the job (cooperative). 2. CANCELLED is now a sticky terminal. 3. Session registry TTL eviction. 4. Per-key concurrency cap. 5. Inline FQN → import. Nits: V162→V164 comment, design doc moved to 31 tests pass: 12 registry (incl. sticky-cancel, per-key cap, TTL eviction) + 13 service + 4 rate limiter + 4 controller + fallback. Ready for re-review 🙏 |
Closes #443 · Part of #440 · Builds on #441 (P0-A)
改动
异步 Deep Research 开放 API。Research 是多步 LLM 管线(plan → retrieve+draft → compose),异步执行并通过 SSE 推送进度。
4 个端点
/{kbId}/research/{kbId}/research/{id}/stream/{kbId}/research/{id}/status/{kbId}/research/{id}/cancel组件
KbOpenResearchController:4 个端点,@RequireKbScope("kb:search")KbResearchSessionRegistry:内存会话追踪,记录 keyId 归属(调用方只能查询/取消自己的会话)安全
?token=query param(KbOpenApiAuthFilter已支持此 fallback,EventSource 无法设 Authorization 头)keyId匹配复用
底层完全复用现有
WikiResearchService.research()+ChatStreamTracker(SSE 广播),不改动 research 管线。测试
依赖
此 PR 包含 P0-A 的 cherry-pick。若 #444(P0-A)先合并,rebase 后只剩 research 的 3 个文件。