diff --git a/.gitignore b/.gitignore index 3b88f71..0fee9ee 100644 --- a/.gitignore +++ b/.gitignore @@ -1,5 +1,14 @@ # Datasets -datasets/ +datasets/* +!datasets/llm/ +datasets/llm/* +!datasets/llm/README.md + +# Local LLM models +models/* +!models/llm/ +models/llm/* +!models/llm/README.md # Python __pycache__/ @@ -33,4 +42,4 @@ configs/agent_experiment_runs/ configs/agent_config_*.yaml configs/cached/ -.pnpm-store/ \ No newline at end of file +.pnpm-store/ diff --git a/README.md b/README.md index 04784de..cc2f869 100644 --- a/README.md +++ b/README.md @@ -136,6 +136,21 @@ cd apps/frontend && pnpm install && pnpm dev - Frontend: `http://localhost:5173` - Backend API: `http://localhost:8000/docs` +### LLM PEFT smoke run + +The LLM PEFT smoke config uses a tiny local Hugging Face-compatible model so it can run without downloading a base model. Generate it once before selecting `configs/smoke/llm_peft_simulation.yaml` in the Simulation UI: + +```bash +uv run python scripts/create_tiny_llm_fixture.py +``` + +For a direct PowerShell smoke run: + +```powershell +$env:PYTHONPATH="libs;apps/backend/runners" +uv run python -c "from pathlib import Path; from runtime_dispatcher import run_runtime; raise SystemExit(0 if run_runtime(Path('configs/smoke/llm_peft_simulation.yaml')) else 1)" +``` + ### Distributed deployment ```bash @@ -194,8 +209,12 @@ figaro/ │ ├── models/ # CNN / ResNet │ ├── data/ # Data loading & partitioning │ ├── privacy/ # CKKS encryption -│ └── compression/ # Top-K sparsification +│ ├── compression/ # Top-K sparsification +│ └── llm/ # LLM PEFT runtime utilities ├── configs/ # Experiment configs +├── datasets/llm/ # Local LLM SFT/evaluation JSONL files +├── models/llm/ # Local Hugging Face-compatible LLM directories +├── skill/ # Local operational skill guides ├── scripts/ # Docker deployment scripts └── .github/workflows/ # CI pipelines ``` @@ -210,14 +229,16 @@ PRs welcome! Figaro is meant to be a readable, research-friendly FL platform. - [x] **Interactive Agent Planning** — Multi-turn dialogue support for refining experiments, plus visual topology previews (Plan Preview) before execution. - [x] **Execution Transparency** — Real-time tracking of node-level status during execution and automated natural-language interpretation of results. - [x] **Strict Configuration Engine** — Implement strict Pydantic/JSON Schema validation to resolve historical inconsistencies between `config_schema` and underlying algorithms. -- [ ] **Advanced Experiment Tracking** — Multi-dimensional search filtering (by metrics, hyperparameters, status) and configuration version control (diffing). - -**Phase 2: LLM & LoRA Federated Fine-Tuning** -- [ ] **Native LLM Ecosystem Integration** — Seamless Hugging Face model loading (e.g., Llama 3, Qwen) and efficient parsing of JSONL instruction-tuning datasets. -- [ ] **Parameter-Efficient Runtime** — Deep integration with LoRA/PEFT, including support for QLoRA (4-bit/8-bit quantization) to lower client-side memory barriers. -- [ ] **Specialized Adapter Aggregation** — Custom aggregation mechanisms for LoRA adapters, exploring support for heterogeneous LoRA ranks across clients. -- [ ] **LLM Evaluation Metrics** — Built-in evaluation for generative tasks (Rouge, BLEU, Perplexity) and automated LLM-as-a-Judge capabilities. -- [ ] **Hardware Guardrails** — Pre-run dynamic GPU memory estimation (OOM prevention) and automated tuning of gradient accumulation and checkpointing. +- [x] **Advanced Experiment Tracking** — Multi-dimensional search filtering (by metrics, hyperparameters, status) and configuration version control (diffing). + +**Phase 2: LLM Federated PEFT Fine-Tuning** +- [x] **Simulation LoRA/PEFT SFT Route** — `task.type=llm_peft_sft` dispatches to a dedicated single-machine simulation runtime that loads a Hugging Face or local causal LM, applies LoRA adapters, and runs per-client supervised fine-tuning from JSONL data. +- [x] **LLM/PEFT Configuration Surface** — `config_schema` and runtime normalization now cover base model, tokenizer, max sequence length, precision, SFT dataset path/format, prompt template, LoRA hyperparameters, target modules, quantization mode, and adapter resume path. +- [x] **JSONL SFT Data Pipeline** — Supports prompt/completion and chat messages JSONL formats, deterministic client splitting, and prompt rendering for plain/chat-style templates. +- [x] **Adapter-Only Federated Aggregation** — Aggregates LoRA/PEFT adapter tensors by client example count, persists global adapter artifacts, records SHA-256 lineage, and supports warm-starting from a previous global adapter. +- [x] **LLM Runtime Dependencies & Metrics** — Core project dependencies include `transformers`, `peft`, `accelerate`, `safetensors`, and `bitsandbytes`; backend metrics include train loss, perplexity, token throughput, adapter size, runtime status, dataset summary, and adapter artifact lineage. +- [x] **Frontend & Agent UX for LLM Runs** — Exposes the LLM route in the schema-driven Simulation and Agent planning flows, with task-aware field visibility, LLM-specific metric charts, local model/dataset selection, and Adapter Lineage artifact views. +- [x] **Evaluation Harness** — Adds validation JSONL configuration, per-round evaluation loss/perplexity calculation after global adapter aggregation, normalized evaluation metrics, and a generated tiny-model smoke config for LLM PEFT runs. **Phase 3: Enterprise & Team Collaboration** - [ ] **Multi-Tenant Workspaces** — Isolated project environments with Role-Based Access Control (RBAC) and comprehensive audit logging. diff --git a/README.zh-CN.md b/README.zh-CN.md index 67c527d..5679a4e 100644 --- a/README.zh-CN.md +++ b/README.zh-CN.md @@ -195,8 +195,12 @@ figaro/ │ ├── models/ # CNN / ResNet │ ├── data/ # 数据加载与划分 │ ├── privacy/ # CKKS 加密 -│ └── compression/ # Top-K 稀疏化 +│ ├── compression/ # Top-K 稀疏化 +│ └── llm/ # LLM PEFT 运行时工具 ├── configs/ # 实验配置 +├── datasets/llm/ # 本地 LLM SFT / 评测 JSONL 文件 +├── models/llm/ # 本地 Hugging Face 兼容 LLM 目录 +├── skill/ # 本地操作说明 skill ├── scripts/ # Docker 部署脚本 └── .github/workflows/ # CI 流水线 ``` @@ -211,14 +215,16 @@ figaro/ - [x] **Agent 交互体验升级** —— 支持多轮对话微调实验计划,提供实验执行前的Plan Preview。 - [x] **执行与分析透明化** —— 支持实验节点的实时状态追踪,以及 Agent 驱动的运行结果自动化图表解释。 - [x] **配置引擎重构** —— 引入基于 Pydantic/JSON Schema 的严格强校验,彻底修复 `config_schema` 与底层算法实现不一致的问题。 -- [ ] **高阶实验管理** —— 支持按指标、超参等多维度搜索过滤实验历史,支持配置文件版本控制与 Diff 差异对比。 - -**Phase 2:LLM / LoRA 联邦微调支持** -- [ ] **大模型生态原生接入** —— 内置 Hugging Face 适配层,一键加载主流开源模型,支持 JSONL 格式的指令微调数据集高效解析。 -- [ ] **高效分布式微调** —— 深度集成 LoRA/PEFT 训练环境,支持 QLoRA (4-bit/8-bit 量化) 以显著降低边缘节点的显存门槛。 -- [ ] **专属权重聚合策略** —— 针对 LLM 微调定制的 Adapter 权重聚合机制,探索支持客户端异构 LoRA Rank 的聚合方案。 -- [ ] **大模型专项评估体系** —— 集成生成式 NLP 指标,并引入基于大模型的自动化指令跟随能力评估 (LLM-as-a-Judge)。 -- [ ] **资源护栏与预判** —— 训练任务启动前进行动态 GPU 显存预估(防 OOM 机制),并支持梯度累积与 Checkpointing 的自动调优。 +- [x] **高阶实验管理** —— 支持按指标、超参等多维度搜索过滤实验历史,支持配置文件版本控制与 Diff 差异对比。 + +**Phase 2:基于现有框架的 LLM 联邦 PEFT 微调支持** +- [x] **单机仿真 LoRA/PEFT SFT 路线** —— `task.type=llm_peft_sft` 会分发到专用的单机仿真运行时,加载 Hugging Face 或本地 Causal LM,注入 LoRA Adapter,并基于 JSONL 数据执行每个客户端的监督微调。 +- [x] **LLM/PEFT 配置面扩展** —— `config_schema` 与运行时规范化已覆盖基础模型、Tokenizer、最大序列长度、精度、SFT 数据路径/格式、Prompt 模板、LoRA 超参、目标模块、量化模式和 Adapter 续跑路径。 +- [x] **JSONL SFT 数据管线** —— 支持 prompt/completion 与 messages 两类 JSONL 格式,支持确定性的客户端数据切分,并提供 plain/chat 风格的文本渲染。 +- [x] **Adapter 权重联邦聚合** —— 按客户端样本数聚合 LoRA/PEFT Adapter Tensor,持久化全局 Adapter 产物,记录 SHA-256 血缘,并支持从历史全局 Adapter warm start 续跑。 +- [x] **LLM 运行依赖与指标** —— 主项目依赖已包含 `transformers`、`peft`、`accelerate`、`safetensors`、`bitsandbytes`;后端指标已包含 train loss、perplexity、token throughput、adapter size、运行状态、数据摘要和 Adapter artifact lineage。 +- [x] **前端与 Agent 的 LLM 运行体验** —— 已在 schema-driven Simulation UI 与 Agent 规划流中开放 LLM 路线,支持按任务类型显示/隐藏字段、LLM 指标曲线、本地模型/数据集选择,以及 Adapter Lineage 产物视图。 +- [x] **评测与 Smoke 配置** —— 已增加验证集 JSONL 配置、全局 Adapter 聚合后的逐轮 evaluation loss/perplexity 计算、标准化评测指标,以及 tiny-model LLM PEFT smoke fixture/config。 **Phase 3:平台化与企业级协作** - [ ] **多租户与细粒度权限** —— 构建多用户隔离的项目空间 (Workspaces),引入基于角色的访问控制 (RBAC) 和完整的操作审计日志。 diff --git a/apps/backend/app/api/v1/endpoints/agent.py b/apps/backend/app/api/v1/endpoints/agent.py index 29f6f1b..9d0e902 100644 --- a/apps/backend/app/api/v1/endpoints/agent.py +++ b/apps/backend/app/api/v1/endpoints/agent.py @@ -4,12 +4,16 @@ from __future__ import annotations -from fastapi import APIRouter, status, HTTPException +from datetime import datetime + +from fastapi import APIRouter, status, HTTPException, Query import json from app.api.deps import AsyncSessionDep from app.schemas.agent import ( + AgentConfigDiffResponse, AgentConfigChangeResponse, + AgentConfigVersionResponse, AgentCurrentExperimentResponse, AgentCurrentPlanResponse, AgentExperimentResponse, @@ -27,7 +31,14 @@ from app.services.agent import AgentExperimentRunService, AgentExperimentService, AgentOptimizationHistoryService, AgentRuntimeService, AgentState from app.services.agent.graph import FederatedAgentGraphBuilder from app.services.agent.objectives import AgentOptimizationObjective, resolve_objective -from app.services.agent.planning import build_schema_prompt_context, dumps_for_prompt +from app.services.agent.planning import ( + build_initial_config, + build_schema_prompt_context, + deep_merge_config, + dumps_for_prompt, + lock_structured_constraints, + merge_llm_resource_constraints, +) from app.services.llm import LLMRegistry, LLMService from app.schemas.agent import AgentPlanPreviewRequest, AgentPlanPreviewResponse, ExperimentPlanPreview, AgentPlanReviseRequest import uuid @@ -241,12 +252,52 @@ def _build_history_summary(item) -> AgentOptimizationJobSummaryResponse: status_code=status.HTTP_200_OK, summary="List persisted agent optimization jobs", ) -async def list_optimization_jobs(session: AsyncSessionDep) -> list[AgentOptimizationJobSummaryResponse]: +async def list_optimization_jobs( + session: AsyncSessionDep, + job_status: str | None = Query(default=None, alias="status"), + q: str | None = Query(default=None), + model_name: str | None = Query(default=None), + objective: str | None = Query(default=None), + best_score_min: float | None = Query(default=None), + best_score_max: float | None = Query(default=None), + created_from: datetime | None = Query(default=None), + created_to: datetime | None = Query(default=None), + dataset: str | None = Query(default=None), + config_model: str | None = Query(default=None), + aggregation: str | None = Query(default=None), + num_clients: int | None = Query(default=None), + num_rounds: int | None = Query(default=None), + dataset_name_alias: str | None = Query(default=None, alias="dataset.name"), + config_model_alias: str | None = Query(default=None, alias="model.name"), + aggregation_alias: str | None = Query(default=None, alias="federated.aggregation"), + num_clients_alias: int | None = Query(default=None, alias="federated.num_clients"), + num_rounds_alias: int | None = Query(default=None, alias="federated.num_rounds"), +) -> list[AgentOptimizationJobSummaryResponse]: """ Return persisted Agent optimization jobs ordered by latest update time. """ service = AgentOptimizationHistoryService(session) - jobs = await service.list_jobs() + config_filters = { + "dataset.name": dataset_name_alias or dataset, + "model.name": config_model_alias or config_model, + "federated.aggregation": aggregation_alias or aggregation, + "federated.num_clients": num_clients_alias if num_clients_alias is not None else num_clients, + "federated.num_rounds": num_rounds_alias if num_rounds_alias is not None else num_rounds, + } + try: + jobs = await service.list_jobs( + status=job_status, + q=q, + model_name=model_name, + objective=objective, + best_score_min=best_score_min, + best_score_max=best_score_max, + created_from=created_from, + created_to=created_to, + config_filters=config_filters, + ) + except ValueError as exc: + raise HTTPException(status_code=422, detail=str(exc)) from exc return [_build_history_summary(item) for item in jobs] @@ -284,6 +335,67 @@ async def get_optimization_job( return _build_progress_response(snapshot) +def _to_config_version_response(version) -> AgentConfigVersionResponse: + return AgentConfigVersionResponse( + id=version.id, + optimization_job_id=version.optimization_job_id, + run_id=version.run_id, + iteration=version.iteration, + source=version.source, + label=version.label, + config_hash=version.config_hash, + config_json=version.config_json if isinstance(version.config_json, dict) else {}, + diff_json=[ + AgentConfigChangeResponse.model_validate(item) + for item in (version.diff_json if isinstance(version.diff_json, list) else []) + ], + created_at=version.created_at, + ) + + +@agent_router.get( + "/optimization-jobs/{optimization_job_id}/config-versions", + response_model=list[AgentConfigVersionResponse], + status_code=status.HTTP_200_OK, + summary="List persisted config versions for an agent optimization job", +) +async def list_optimization_job_config_versions( + optimization_job_id: int, + session: AsyncSessionDep, +) -> list[AgentConfigVersionResponse]: + """Return config versions captured during planning and experiment execution.""" + service = AgentOptimizationHistoryService(session) + versions = await service.list_config_versions(optimization_job_id) + return [_to_config_version_response(version) for version in versions] + + +@agent_router.get( + "/optimization-jobs/{optimization_job_id}/config-diff", + response_model=AgentConfigDiffResponse, + status_code=status.HTTP_200_OK, + summary="Diff two persisted config versions for an agent optimization job", +) +async def diff_optimization_job_config_versions( + optimization_job_id: int, + session: AsyncSessionDep, + to_version_id: int = Query(...), + from_version_id: int | None = Query(default=None), +) -> AgentConfigDiffResponse: + """Return a normalized config diff between two persisted versions.""" + service = AgentOptimizationHistoryService(session) + changes = await service.diff_config_versions( + optimization_job_id=optimization_job_id, + from_version_id=from_version_id, + to_version_id=to_version_id, + ) + return AgentConfigDiffResponse( + optimization_job_id=optimization_job_id, + from_version_id=from_version_id, + to_version_id=to_version_id, + changes=[AgentConfigChangeResponse.model_validate(item) for item in changes], + ) + + @agent_router.post( "/optimize/start", response_model=AgentOptimizeProgressResponse, @@ -675,13 +787,39 @@ async def revise_plan_preview( except Exception as e: raise HTTPException(status_code=500, detail=f"Failed to parse LLM response: {str(e)}") - snapshot["draft_experiments"] = updated_experiments + experiment_service = AgentExperimentService(session) + schema = experiment_service.get_config_schema() + constrained_base = experiment_service.normalize_simulation_config( + deep_merge_config(build_initial_config(schema), config_constraints) + ) + effective_constraints = merge_llm_resource_constraints(constrained_base, config_constraints) + normalized_experiments = [] + for idx, exp in enumerate(updated_experiments): + if not isinstance(exp, dict): + continue + raw_config = exp.get("config_patch", exp.get("config", {})) + if not isinstance(raw_config, dict): + raw_config = {} + merged = lock_structured_constraints( + deep_merge_config(constrained_base, raw_config), + effective_constraints, + ) + normalized_experiments.append( + { + **exp, + "name": exp.get("name", f"exp-{idx + 1}"), + "plan_summary": exp.get("plan_summary", snapshot.get("goal", "")), + "config_patch": experiment_service.normalize_simulation_config(merged), + } + ) + + snapshot["draft_experiments"] = normalized_experiments await history_service.update_job_snapshot(task_id=job.task_id, snapshot=snapshot) return AgentPlanPreviewResponse( optimization_job_id=job.id, goal=snapshot.get("goal", ""), - experiments=[ExperimentPlanPreview.model_validate(exp) for exp in updated_experiments], + experiments=[ExperimentPlanPreview.model_validate(exp) for exp in normalized_experiments], system_mode=snapshot.get("system_mode", "simulation"), config_constraints=config_constraints, ) diff --git a/apps/backend/app/models/__init__.py b/apps/backend/app/models/__init__.py index e8852bb..9155124 100644 --- a/apps/backend/app/models/__init__.py +++ b/apps/backend/app/models/__init__.py @@ -1,8 +1,10 @@ """Model package marker — import all models so SQLModel registers them.""" from app.models.agent import ( # noqa: F401 + AgentConfigVersion, AgentExperiment, AgentExperimentRun, AgentExperimentRunLog, AgentExperimentRunResult, + AgentOptimizationJob, ) diff --git a/apps/backend/app/models/agent/__init__.py b/apps/backend/app/models/agent/__init__.py index 3f6f494..8fe0be4 100644 --- a/apps/backend/app/models/agent/__init__.py +++ b/apps/backend/app/models/agent/__init__.py @@ -10,7 +10,7 @@ AgentExperimentRunStatus, AgentExperimentStatus, ) -from app.models.agent.optimization_jobs import AgentOptimizationJob, AgentOptimizationJobStatus +from app.models.agent.optimization_jobs import AgentConfigVersion, AgentOptimizationJob, AgentOptimizationJobStatus __all__ = [ "AgentExperiment", @@ -21,4 +21,5 @@ "AgentExperimentRunResult", "AgentOptimizationJob", "AgentOptimizationJobStatus", + "AgentConfigVersion", ] diff --git a/apps/backend/app/models/agent/optimization_jobs.py b/apps/backend/app/models/agent/optimization_jobs.py index 8848516..650c05b 100644 --- a/apps/backend/app/models/agent/optimization_jobs.py +++ b/apps/backend/app/models/agent/optimization_jobs.py @@ -93,3 +93,45 @@ class AgentOptimizationJob(SQLModel, table=True): default=None, sa_column=Column(DateTime(timezone=True), nullable=True), ) + + +class AgentConfigVersion(SQLModel, table=True): + __tablename__ = "agent_config_versions" + + id: int | None = Field( + default=None, + sa_column=Column(Integer, primary_key=True, nullable=False), + ) + optimization_job_id: int = Field( + sa_column=Column(Integer, nullable=False, index=True), + ) + run_id: str | None = Field( + default=None, + sa_column=Column(String(64), nullable=True, index=True), + ) + iteration: int = Field( + default=0, + sa_column=Column(Integer, nullable=False, index=True), + ) + source: str = Field( + default="experiment", + sa_column=Column(String(32), nullable=False, index=True), + ) + label: str = Field( + sa_column=Column(String(255), nullable=False), + ) + config_hash: str = Field( + sa_column=Column(String(64), nullable=False, index=True), + ) + config_json: dict[str, Any] = Field( + default_factory=dict, + sa_column=Column(JSON().with_variant(JSONB, "postgresql"), nullable=False), + ) + diff_json: list[dict[str, Any]] = Field( + default_factory=list, + sa_column=Column(JSON().with_variant(JSONB, "postgresql"), nullable=False), + ) + created_at: datetime = Field( + default_factory=utcnow, + sa_column=Column(DateTime(timezone=True), nullable=False), + ) diff --git a/apps/backend/app/repositories/agent/__init__.py b/apps/backend/app/repositories/agent/__init__.py index dec957e..c02fe11 100644 --- a/apps/backend/app/repositories/agent/__init__.py +++ b/apps/backend/app/repositories/agent/__init__.py @@ -1,6 +1,6 @@ """Agent repository exports.""" from app.repositories.agent.experiment_repository import AgentExperimentRepository -from app.repositories.agent.optimization_job_repository import AgentOptimizationJobRepository +from app.repositories.agent.optimization_job_repository import AgentConfigVersionRepository, AgentOptimizationJobRepository -__all__ = ["AgentExperimentRepository", "AgentOptimizationJobRepository"] +__all__ = ["AgentExperimentRepository", "AgentOptimizationJobRepository", "AgentConfigVersionRepository"] diff --git a/apps/backend/app/repositories/agent/optimization_job_repository.py b/apps/backend/app/repositories/agent/optimization_job_repository.py index 6fa8f95..7953833 100644 --- a/apps/backend/app/repositories/agent/optimization_job_repository.py +++ b/apps/backend/app/repositories/agent/optimization_job_repository.py @@ -4,13 +4,14 @@ from __future__ import annotations +from datetime import datetime from typing import Any -from sqlalchemy import func +from sqlalchemy import func, or_ from sqlalchemy.ext.asyncio import AsyncSession from sqlmodel import select -from app.models.agent import AgentOptimizationJob, AgentOptimizationJobStatus +from app.models.agent import AgentConfigVersion, AgentOptimizationJob, AgentOptimizationJobStatus from app.models.base import utcnow @@ -62,8 +63,38 @@ async def get_job_by_name(self, job_name: str) -> AgentOptimizationJob | None: result = await self.session.execute(stmt) return result.scalars().first() - async def list_jobs(self) -> list[AgentOptimizationJob]: + async def list_jobs( + self, + *, + status: AgentOptimizationJobStatus | None = None, + q: str | None = None, + model_name: str | None = None, + best_score_min: float | None = None, + best_score_max: float | None = None, + created_from: datetime | None = None, + created_to: datetime | None = None, + ) -> list[AgentOptimizationJob]: stmt = select(AgentOptimizationJob).order_by(AgentOptimizationJob.updated_at.desc(), AgentOptimizationJob.id.desc()) + if status is not None: + stmt = stmt.where(AgentOptimizationJob.status == status) + if q: + like_value = f"%{q.lower()}%" + stmt = stmt.where( + or_( + func.lower(AgentOptimizationJob.goal).like(like_value), + func.lower(AgentOptimizationJob.job_name).like(like_value), + ) + ) + if model_name: + stmt = stmt.where(func.lower(AgentOptimizationJob.model_name) == model_name.lower()) + if best_score_min is not None: + stmt = stmt.where(AgentOptimizationJob.best_score >= best_score_min) + if best_score_max is not None: + stmt = stmt.where(AgentOptimizationJob.best_score <= best_score_max) + if created_from is not None: + stmt = stmt.where(AgentOptimizationJob.created_at >= created_from) + if created_to is not None: + stmt = stmt.where(AgentOptimizationJob.created_at <= created_to) result = await self.session.execute(stmt) return list(result.scalars().all()) @@ -108,3 +139,72 @@ async def update_job( self.session.add(job) await self.session.flush() return job + + +class AgentConfigVersionRepository: + """CRUD helpers for persisted agent configuration versions.""" + + def __init__(self, session: AsyncSession): + self.session = session + + async def create_version( + self, + *, + optimization_job_id: int, + run_id: str | None, + iteration: int, + source: str, + label: str, + config_hash: str, + config_json: dict[str, Any], + diff_json: list[dict[str, Any]], + ) -> AgentConfigVersion: + version = AgentConfigVersion( + optimization_job_id=optimization_job_id, + run_id=run_id, + iteration=iteration, + source=source, + label=label, + config_hash=config_hash, + config_json=config_json, + diff_json=diff_json, + ) + self.session.add(version) + await self.session.flush() + return version + + async def find_version( + self, + *, + optimization_job_id: int, + run_id: str | None, + iteration: int, + source: str, + config_hash: str, + ) -> AgentConfigVersion | None: + stmt = select(AgentConfigVersion).where( + AgentConfigVersion.optimization_job_id == optimization_job_id, + AgentConfigVersion.iteration == iteration, + AgentConfigVersion.source == source, + AgentConfigVersion.config_hash == config_hash, + ) + if run_id is None: + stmt = stmt.where(AgentConfigVersion.run_id.is_(None)) + else: + stmt = stmt.where(AgentConfigVersion.run_id == run_id) + result = await self.session.execute(stmt) + return result.scalars().first() + + async def get_version(self, version_id: int) -> AgentConfigVersion | None: + return await self.session.get(AgentConfigVersion, version_id) + + async def list_versions(self, optimization_job_id: int, *, source: str | None = None) -> list[AgentConfigVersion]: + stmt = ( + select(AgentConfigVersion) + .where(AgentConfigVersion.optimization_job_id == optimization_job_id) + .order_by(AgentConfigVersion.created_at.asc(), AgentConfigVersion.id.asc()) + ) + if source is not None: + stmt = stmt.where(AgentConfigVersion.source == source) + result = await self.session.execute(stmt) + return list(result.scalars().all()) diff --git a/apps/backend/app/schemas/__init__.py b/apps/backend/app/schemas/__init__.py index b793e9f..ad3ca51 100644 --- a/apps/backend/app/schemas/__init__.py +++ b/apps/backend/app/schemas/__init__.py @@ -29,6 +29,8 @@ ) from app.schemas.message import Message from app.schemas.agent import ( + AgentConfigDiffResponse, + AgentConfigVersionResponse, AgentCurrentPlanResponse, AgentExperimentSummary, AgentOptimizationJobSummaryResponse, @@ -66,4 +68,6 @@ "AgentOptimizationJobSummaryResponse", "AgentCurrentPlanResponse", "AgentExperimentSummary", + "AgentConfigVersionResponse", + "AgentConfigDiffResponse", ] diff --git a/apps/backend/app/schemas/agent.py b/apps/backend/app/schemas/agent.py index 3baec67..2425c04 100644 --- a/apps/backend/app/schemas/agent.py +++ b/apps/backend/app/schemas/agent.py @@ -67,6 +67,30 @@ class AgentConfigChangeResponse(BaseModel): new_value: Any = None +class AgentConfigVersionResponse(BaseModel): + """One persisted config version for a historical Agent optimization.""" + + id: int + optimization_job_id: int + run_id: str | None = None + iteration: int + source: str + label: str + config_hash: str + config_json: dict[str, Any] = Field(default_factory=dict) + diff_json: list[AgentConfigChangeResponse] = Field(default_factory=list) + created_at: datetime + + +class AgentConfigDiffResponse(BaseModel): + """Diff payload between two persisted config versions.""" + + optimization_job_id: int + from_version_id: int | None = None + to_version_id: int + changes: list[AgentConfigChangeResponse] = Field(default_factory=list) + + class AgentCurrentPlanResponse(BaseModel): """Planner output for the iteration currently being prepared or executed.""" diff --git a/apps/backend/app/schemas/simulation/runs.py b/apps/backend/app/schemas/simulation/runs.py index 9d8f23e..45e2dd3 100644 --- a/apps/backend/app/schemas/simulation/runs.py +++ b/apps/backend/app/schemas/simulation/runs.py @@ -68,9 +68,21 @@ class SimulationRunGlobalResults(BaseModel): global_accuracy: list[float] = Field(default_factory=list) +class SimulationRunLlmResults(BaseModel): + """LLM PEFT metric series.""" + + rounds: list[int] = Field(default_factory=list) + train_loss: list[float] = Field(default_factory=list) + validation_loss: list[float] = Field(default_factory=list) + perplexity: list[float] = Field(default_factory=list) + token_throughput: list[float] = Field(default_factory=list) + adapter_size_bytes: list[int] = Field(default_factory=list) + + class SimulationRunClientSeries(BaseModel): """Per-client metric series in simulation mode.""" + rounds: list[int] = Field(default_factory=list) train_loss: list[float] = Field(default_factory=list) train_acc: list[float] = Field(default_factory=list) test_loss: list[float] = Field(default_factory=list) @@ -90,4 +102,9 @@ class SimulationRunMetricsResponse(BaseModel): experiment_info: SimulationRunExperimentInfo = Field(default_factory=SimulationRunExperimentInfo) global_results: SimulationRunGlobalResults = Field(default_factory=SimulationRunGlobalResults) + llm_results: SimulationRunLlmResults = Field(default_factory=SimulationRunLlmResults) + llm_dataset: dict[str, Any] = Field(default_factory=dict) + llm_evaluation: dict[str, Any] = Field(default_factory=dict) + llm_runtime: dict[str, Any] = Field(default_factory=dict) + llm_artifacts: list[dict[str, Any]] = Field(default_factory=list) client_results: dict[str, SimulationRunClientSeries] = Field(default_factory=dict) diff --git a/apps/backend/app/services/agent/capabilities.py b/apps/backend/app/services/agent/capabilities.py index 4b973e2..a6ebfa5 100644 --- a/apps/backend/app/services/agent/capabilities.py +++ b/apps/backend/app/services/agent/capabilities.py @@ -1,118 +1,34 @@ """ -Helpers for inspecting the current federated learning stack capabilities. +Helpers for exposing executable simulation capabilities to the Agent. -Used by the Agent to ground its proposals in the actually supported -datasets, models, and aggregations. +The Agent should plan only combinations that the simulation runtime can +execute. The source of truth lives in libs/fl_core/simulation_registry.py; +this module is a thin backend-side adapter. """ from __future__ import annotations -import ast -import re from pathlib import Path -from typing import Any, Dict, List +from typing import Any, Dict - -def _read_text(path: Path) -> str: - return path.read_text(encoding="utf-8", errors="replace") - - -def _extract_list_literal(text: str) -> List[Any]: - """ - Extract a Python list literal from a string like: [ "a", "b" ]. - Returns [] on failure. - """ - try: - value = ast.literal_eval(text) - if isinstance(value, list): - return value - except Exception: - pass - return [] - - -def _extract_supported_datasets(project_root: Path) -> List[str]: - path = project_root / "libs" / "fl_core" / "data" / "data_loader.py" - if not path.exists(): - return [] - text = _read_text(path) - match = re.search(r"supported_datasets\s*=\s*(\[[^\]]*\])", text, flags=re.DOTALL) - if not match: - return [] - items = _extract_list_literal(match.group(1)) - out: List[str] = [] - for item in items: - if isinstance(item, str): - out.append(item) - return out - - -def _extract_model_registry_keys(project_root: Path) -> List[str]: - path = project_root / "libs" / "fl_core" / "models" / "model_manager.py" - if not path.exists(): - return [] - text = _read_text(path) - match = re.search(r"model_registry\s*=\s*\{(.*?)\}\s*\n", text, flags=re.DOTALL) - if not match: - return [] - block = match.group(1) - keys = re.findall(r"['\"]([a-zA-Z0-9_]+)['\"]\s*:", block) - seen: set[str] = set() - out: List[str] = [] - for key in keys: - lower = key.lower() - if lower not in seen: - seen.add(lower) - out.append(lower) - return out - - -def _extract_aggregation_strategies(project_root: Path) -> List[str]: - """ - Returns aggregation strategies from federated/aggregation.py without imports. - """ - path = project_root / "libs" / "fl_core" / "federated" / "aggregation.py" - if not path.exists(): - return [] - text = _read_text(path) - - match = re.search(r"_strategies\s*=\s*\{(.*?)\}", text, flags=re.DOTALL) - if not match: - return [] - aggregations = re.findall(r"['\"]([a-zA-Z0-9_]+)['\"]\s*:", match.group(1)) - - seen: set[str] = set() - out: List[str] = [] - for item in aggregations: - lower = item.lower() - if lower not in seen: - seen.add(lower) - out.append(lower) - return out +from app.services.simulation.compatibility import runtime_capabilities def get_platform_capabilities(project_root: Path | None = None) -> Dict[str, Any]: """ - Inspect the current codebase and return the supported configuration options - for datasets, models, and aggregations. + Return supported simulation datasets, models, aggregations, and compatibility + rules. The project_root argument is kept for compatibility with older tests. """ - root = project_root or Path(__file__).resolve().parents[4] - - datasets = _extract_supported_datasets(root) - models = _extract_model_registry_keys(root) - aggregations = _extract_aggregation_strategies(root) - - metrics = { - "global_results": ["rounds", "global_loss", "global_accuracy"], - "client_results": ["train_loss", "train_acc", "test_loss", "test_acc"], - } - - return { - "datasets": datasets, - "distributions": ["iid", "non_iid"], - "models": models, - "aggregations": aggregations, - "metrics": metrics, + _ = project_root + capabilities = runtime_capabilities() + capabilities["training_routes"] = { + "classic_fl": { + "modes": ["simulation", "distributed"], + "description": "Classic federated learning route for the built-in small-model datasets.", + }, + "llm_peft_sft": { + "modes": ["simulation"], + "description": "Simulation-only route for federated LoRA/PEFT supervised fine-tuning on JSONL SFT data.", + }, } - - + return capabilities diff --git a/apps/backend/app/services/agent/experiment_service.py b/apps/backend/app/services/agent/experiment_service.py index a6bbd8d..5b42b2b 100644 --- a/apps/backend/app/services/agent/experiment_service.py +++ b/apps/backend/app/services/agent/experiment_service.py @@ -32,6 +32,15 @@ AgentExperimentStatus, ) from app.repositories.agent import AgentExperimentRepository +from app.services.llm_resources import augment_config_schema_with_llm_resources +from app.services.run_artifacts import ( + legacy_live_results_filename, + live_results_filename, + run_artifact_timestamp, + run_config_filename, + run_log_filename, +) +from app.services.simulation.compatibility import canonicalize_runtime_config, validate_runtime_config_or_raise from app.core.logger import get_logger @@ -42,7 +51,7 @@ # Shared constants (mirrored from simulation run_service / run_metrics_service) # --------------------------------------------------------------------------- -RESULT_PATH_PATTERN = re.compile(r"结果已保存到[::]\s*(.+)$") +RESULT_PATH_PATTERN = re.compile(r"\u7ed3\u679c\u5df2\u4fdd\u5b58\u5230[:\uff1a]\s*(.+)$") LOG_LEVEL_PREFIX_PATTERN = re.compile( r"^\s*(DEBUG|INFO|WARNING|WARN|ERROR|CRITICAL)\b[\s:\-]", re.IGNORECASE, @@ -205,7 +214,7 @@ def get_config_schema(cls) -> dict[str, Any]: loaded = yaml.safe_load(schema_path.read_text(encoding="utf-8")) if not isinstance(loaded, dict): raise exceptions.InternalServiceError("Config schema must be a YAML object.") - return loaded + return augment_config_schema_with_llm_resources(loaded, cls._project_root()) @classmethod def normalize_simulation_config(cls, config: dict[str, Any]) -> dict[str, Any]: @@ -235,7 +244,9 @@ def _normalize_simulation_config(cls, config: dict[str, Any]) -> dict[str, Any]: system["mode"] = "simulation" system["node_role"] = "server" + canonicalize_runtime_config(merged) cls._validate_config_node(merged, schema, path_prefix="") + validate_runtime_config_or_raise(merged) return merged @classmethod @@ -310,6 +321,12 @@ def _cleanup_run_local_files(self, run_id: str) -> None: self._project_root() / "configs" / "agent_experiment_runs" / f"{run_id}.json", self._project_root() / "results" / f"live_results_{run_id}.json", ] + candidates.extend((self._project_root() / "configs" / "agent_experiment_runs").glob(f"{run_id}*.json")) + candidates.extend((self._project_root() / "configs" / "agent_experiment_runs").glob(f"*_{run_id}.json")) + candidates.extend((self._project_root() / "results").glob(f"live_results_{run_id}*.json")) + candidates.extend((self._project_root() / "results").glob(f"*_{run_id}_live_results.json")) + candidates.extend((self._project_root() / "logs" / "agent_experiment_runs").glob(f"{run_id}*_server.log")) + candidates.extend((self._project_root() / "logs" / "agent_experiment_runs").glob(f"*_{run_id}_server.log")) for path in candidates: try: if path.exists() and path.is_file(): @@ -476,14 +493,12 @@ async def get_run_metrics(self, run_id: str) -> dict[str, Any]: run = await self.repo.get_run(run_id) if not run: raise exceptions.RunNotFound("Run not found") - # If terminal and already has metrics, return them - if run.status in TERMINAL_RUN_STATUSES and isinstance(run.metrics_json, dict) and run.metrics_json: - return self._normalize_metrics_payload(run.metrics_json) # Try live result file - live_result_path = self._results_dir() / self._build_live_results_filename(run_id) - loaded = self._load_metrics_file(live_result_path) - if loaded is not None: - return loaded + for live_result_path in self._live_result_candidates_for_run(run): + loaded = self._load_metrics_file(live_result_path) + if loaded is not None: + await self._persist_run_metrics_if_changed(run, loaded) + return loaded # Try result artifacts results = await self.repo.list_results(run_id) @@ -492,6 +507,7 @@ async def get_run_metrics(self, run_id: str) -> dict[str, Any]: artifact_path = self._resolve_result_artifact_path(artifact.path) loaded = self._load_metrics_file(artifact_path) if loaded is not None: + await self._persist_run_metrics_if_changed(run, loaded) return loaded if isinstance(run.metrics_json, dict) and run.metrics_json: @@ -499,6 +515,15 @@ async def get_run_metrics(self, run_id: str) -> dict[str, Any]: return self._empty_metrics_payload() + async def _persist_run_metrics_if_changed(self, run: AgentExperimentRun, metrics: dict[str, Any]) -> None: + current = run.metrics_json if isinstance(run.metrics_json, dict) else {} + if self._stable_json(current) == self._stable_json(metrics): + return + + await self.repo.update_run_metrics(run, metrics) + await self.session.commit() + await self.session.refresh(run) + async def delete_run(self, run_id: str) -> None: run = await self.get_run(run_id) if run.status in {AgentExperimentRunStatus.QUEUED, AgentExperimentRunStatus.RUNNING}: @@ -522,18 +547,22 @@ async def _start_run_with_config(self, *, experiment_id: int, config_json: dict[ ) await self.repo.create_run(run) - config_path = self._write_run_config(run.id, config_json) + artifact_timestamp = run_artifact_timestamp(run.created_at) + config_path = self._write_run_config(run.id, config_json, artifact_timestamp=artifact_timestamp) await self.repo.add_result( run_id=run.id, artifact_type=RUN_CONFIG_ARTIFACT, path=str(config_path), - metadata_json={"source": "run_config"}, + metadata_json={"source": "run_config", "artifact_timestamp": artifact_timestamp}, ) command = self._build_train_command(config_path) run.command = " ".join(command) - live_results_path = self._results_dir() / self._build_live_results_filename(run.id) + live_results_path = self._results_dir() / self._build_live_results_filename( + run.id, + artifact_timestamp=artifact_timestamp, + ) child_env = self._build_run_environment(live_results_path) process = await self._spawn_subprocess(command, env=child_env) @@ -548,13 +577,21 @@ async def _start_run_with_config(self, *, experiment_id: int, config_json: dict[ await self.session.refresh(run) self._processes[run.id] = process - self._tasks[run.id] = asyncio.create_task(self._watch_process(run.id, process)) + self._tasks[run.id] = asyncio.create_task( + self._watch_process(run.id, process, artifact_timestamp=artifact_timestamp) + ) return run - async def _watch_process(self, run_id: str, process: asyncio.subprocess.Process) -> None: + async def _watch_process( + self, + run_id: str, + process: asyncio.subprocess.Process, + *, + artifact_timestamp: str | None = None, + ) -> None: """Consume subprocess streams and persist final run status/artifacts.""" result_path: str | None = None - log_path = self._run_log_path(run_id) + log_path = self._run_log_path(run_id, artifact_timestamp=artifact_timestamp) async def consume(stream: asyncio.StreamReader, level: str, prefix: str) -> None: nonlocal result_path @@ -566,7 +603,7 @@ async def consume(stream: asyncio.StreamReader, level: str, prefix: str) -> None message = line.decode("utf-8", errors="replace").rstrip() if not message: continue - file_obj.write(f"[{prefix}] {message}\\n") + file_obj.write(f"[{prefix}] {message}\n") file_obj.flush() match = RESULT_PATH_PATTERN.search(message) @@ -601,21 +638,35 @@ async def consume(stream: asyncio.StreamReader, level: str, prefix: str) -> None ) await repo.add_log(run_id, f"run finished with exit_code={exit_code}") + metrics_persisted = False if result_path: await repo.add_result( run_id=run_id, artifact_type=TRAINING_RESULT_ARTIFACT, path=result_path, - metadata_json={"source": "process_output"}, + metadata_json={ + "source": "process_output", + "artifact_timestamp": artifact_timestamp, + }, ) resolved = self._resolve_result_artifact_path(result_path) normalized = self._load_metrics_file(resolved) if normalized is not None: await repo.update_run_metrics(run, normalized) + metrics_persisted = True - if not run.metrics_json: + if not metrics_persisted: guessed = self._guess_live_result_file_for_run(run) if guessed is not None: + await repo.add_result( + run_id=run_id, + artifact_type=TRAINING_RESULT_ARTIFACT, + path=str(guessed), + metadata_json={ + "source": "live_results_fallback", + "artifact_timestamp": artifact_timestamp, + }, + ) normalized = self._load_metrics_file(guessed) if normalized is not None: await repo.update_run_metrics(run, normalized) @@ -645,17 +696,33 @@ def _runtime_log_dir(self) -> Path: path.mkdir(parents=True, exist_ok=True) return path - def _run_log_path(self, run_id: str) -> Path: - return self._runtime_log_dir() / f"{run_id}_server.log" + def _run_log_path(self, run_id: str, *, artifact_timestamp: str | None = None) -> Path: + if artifact_timestamp is None: + return self._runtime_log_dir() / f"{run_id}_server.log" + return self._runtime_log_dir() / run_log_filename( + run_id, + timestamp=artifact_timestamp, + role="server", + ) - def _build_live_results_filename(self, run_id: str) -> str: - return f"live_results_{run_id}.json" + def _build_live_results_filename(self, run_id: str, *, artifact_timestamp: str | None = None) -> str: + if artifact_timestamp is None: + return f"live_results_{run_id}.json" + return live_results_filename(run_id, timestamp=artifact_timestamp) - def _run_config_path(self, run_id: str) -> Path: - return self._job_config_dir() / f"{run_id}.json" + def _run_config_path(self, run_id: str, *, artifact_timestamp: str | None = None) -> Path: + if artifact_timestamp is None: + return self._job_config_dir() / f"{run_id}.json" + return self._job_config_dir() / run_config_filename(run_id, timestamp=artifact_timestamp) - def _write_run_config(self, run_id: str, config_json: dict[str, Any]) -> Path: - path = self._run_config_path(run_id) + def _write_run_config( + self, + run_id: str, + config_json: dict[str, Any], + *, + artifact_timestamp: str | None = None, + ) -> Path: + path = self._run_config_path(run_id, artifact_timestamp=artifact_timestamp) path.write_text(json.dumps(config_json, ensure_ascii=False, indent=2), encoding="utf-8") return path @@ -705,17 +772,23 @@ def _derive_final_status(current_status: AgentExperimentRunStatus, exit_code: in return AgentExperimentRunStatus.SUCCEEDED if exit_code == 0 else AgentExperimentRunStatus.FAILED def _cleanup_run_local_files(self, run_id: str) -> None: - for path in [ - self._run_config_path(run_id), - self._results_dir() / self._build_live_results_filename(run_id), - self._run_log_path(run_id), - ]: + for path in self._matching_run_local_files(run_id): try: if path.exists() and path.is_file(): path.unlink() except OSError: continue + def _matching_run_local_files(self, run_id: str) -> list[Path]: + return [ + *self._job_config_dir().glob(f"{run_id}*.json"), + *self._job_config_dir().glob(f"*_{run_id}.json"), + *self._results_dir().glob(f"live_results_{run_id}*.json"), + *self._results_dir().glob(f"*_{run_id}_live_results.json"), + *self._runtime_log_dir().glob(f"{run_id}*_server.log"), + *self._runtime_log_dir().glob(f"*_{run_id}_server.log"), + ] + # -- metrics helpers (mirrored from SimulationRunMetricsService) ------ @staticmethod @@ -731,6 +804,18 @@ def _empty_metrics_payload() -> dict[str, Any]: "global_loss": [], "global_accuracy": [], }, + "llm_results": { + "rounds": [], + "train_loss": [], + "validation_loss": [], + "perplexity": [], + "token_throughput": [], + "adapter_size_bytes": [], + }, + "llm_dataset": {}, + "llm_evaluation": {}, + "llm_runtime": {}, + "llm_artifacts": [], "client_results": {}, } @@ -755,27 +840,41 @@ def _guess_live_result_file_for_run(self, run: AgentExperimentRun) -> Path | Non directory = self._results_dir() if not directory.exists(): return None - expected = directory / self._build_live_results_filename(run.id) - if expected.exists() and expected.is_file(): - return expected + for expected in self._live_result_candidates_for_run(run): + if expected.exists() and expected.is_file(): + return expected started_at = run.started_at or run.created_at if started_at is None: return None start_ts = started_at.timestamp() candidates: list[Path] = [] - for path in directory.glob("live_results_*.json"): - try: - mtime = path.stat().st_mtime - except OSError: - continue - if mtime >= start_ts - 5: - candidates.append(path) + for pattern in ("*_live_results.json", "live_results_*.json"): + for path in directory.glob(pattern): + try: + mtime = path.stat().st_mtime + except OSError: + continue + if mtime >= start_ts - 5: + candidates.append(path) if not candidates: return None candidates.sort(key=lambda p: p.stat().st_mtime, reverse=True) return candidates[0] + def _live_result_candidates_for_run(self, run: AgentExperimentRun) -> list[Path]: + directory = self._results_dir() + timestamped = directory / self._build_live_results_filename( + run.id, + artifact_timestamp=run_artifact_timestamp(run.created_at), + ) + legacy_timestamped = directory / legacy_live_results_filename( + run.id, + timestamp=run_artifact_timestamp(run.created_at), + ) + legacy = directory / self._build_live_results_filename(run.id) + return list(dict.fromkeys([timestamped, legacy_timestamped, legacy])) + def _normalize_metrics_payload(self, payload: dict[str, Any]) -> dict[str, Any]: normalized = self._empty_metrics_payload() @@ -794,6 +893,35 @@ def _normalize_metrics_payload(self, payload: dict[str, Any]) -> dict[str, Any]: "global_accuracy": self._to_float_list(global_results.get("global_accuracy")), } + llm_results = payload.get("llm_results") + if isinstance(llm_results, dict): + normalized["llm_results"] = { + "rounds": self._to_int_list(llm_results.get("rounds")), + "train_loss": self._to_float_list(llm_results.get("train_loss")), + "validation_loss": self._to_float_list(llm_results.get("validation_loss")), + "perplexity": self._to_float_list(llm_results.get("perplexity")), + "token_throughput": self._to_float_list(llm_results.get("token_throughput")), + "adapter_size_bytes": self._to_int_list(llm_results.get("adapter_size_bytes")), + } + + llm_dataset = payload.get("llm_dataset") + if isinstance(llm_dataset, dict): + normalized["llm_dataset"] = llm_dataset + + llm_evaluation = payload.get("llm_evaluation") + if isinstance(llm_evaluation, dict): + normalized["llm_evaluation"] = llm_evaluation + + llm_runtime = payload.get("llm_runtime") + if isinstance(llm_runtime, dict): + normalized["llm_runtime"] = llm_runtime + + llm_artifacts = payload.get("llm_artifacts") + if isinstance(llm_artifacts, list): + normalized["llm_artifacts"] = [ + artifact for artifact in llm_artifacts if isinstance(artifact, dict) + ] + client_results = payload.get("client_results") if isinstance(client_results, dict): normalized_clients: dict[str, dict[str, list[float]]] = {} @@ -801,6 +929,7 @@ def _normalize_metrics_payload(self, payload: dict[str, Any]) -> dict[str, Any]: if not isinstance(client_name, str) or not isinstance(client_data, dict): continue normalized_clients[client_name] = { + "rounds": self._to_int_list(client_data.get("rounds")), "train_loss": self._to_float_list(client_data.get("train_loss")), "train_acc": self._to_float_list(client_data.get("train_acc")), "test_loss": self._to_float_list(client_data.get("test_loss")), @@ -834,6 +963,10 @@ def _to_int_list(value: Any) -> list[int]: continue return output + @staticmethod + def _stable_json(value: Any) -> str: + return json.dumps(value, sort_keys=True, ensure_ascii=False) + # -- log level inference --------------------------------------------- @staticmethod diff --git a/apps/backend/app/services/agent/graph.py b/apps/backend/app/services/agent/graph.py index f9d305f..f704065 100644 --- a/apps/backend/app/services/agent/graph.py +++ b/apps/backend/app/services/agent/graph.py @@ -31,11 +31,13 @@ build_schema_prompt_context, collect_disabled_option_errors, deep_merge_config, + lock_structured_constraints, + merge_llm_resource_constraints, select_llm_model, ) from .prompts import build_plan_prompt, build_plan_system_instructions from .state import AgentState, ExperimentPlan, ExperimentRecord -from .summary import build_results_table, build_summary_text, get_last_global_accuracy +from .summary import build_results_table, build_summary_text, get_agent_run_score class FederatedAgentGraphBuilder: @@ -86,6 +88,7 @@ async def _node_parse(self, state: AgentState) -> AgentState: constrained_base = experiment_service.normalize_simulation_config( deep_merge_config(base_config, state.config_constraints) ) + effective_constraints = merge_llm_resource_constraints(constrained_base, state.config_constraints) disabled_errors = collect_disabled_option_errors(constrained_base, schema) if disabled_errors: raise exceptions.BadRequestError("; ".join(disabled_errors)) @@ -97,8 +100,12 @@ async def _node_parse(self, state: AgentState) -> AgentState: raw_config = exp.get("config_patch", exp.get("config", {})) if not isinstance(raw_config, dict): raw_config = {} + merged = lock_structured_constraints( + deep_merge_config(constrained_base, raw_config), + effective_constraints, + ) normalized = experiment_service.normalize_simulation_config( - deep_merge_config(constrained_base, raw_config) + merged ) disabled_errors = collect_disabled_option_errors(normalized, schema) if disabled_errors: @@ -127,7 +134,7 @@ async def _node_parse(self, state: AgentState) -> AgentState: capabilities=capabilities, base_config=constrained_base, schema_context=schema_context, - config_constraints=state.config_constraints, + config_constraints=effective_constraints, ) logger.info("llm input instructions=%s", instructions) @@ -165,8 +172,12 @@ async def _node_parse(self, state: AgentState) -> AgentState: patch_config = exp.get("config", {}) if not isinstance(patch_config, dict): patch_config = {} - # Merge with constrained base config and normalize. - merged = deep_merge_config(constrained_base, patch_config) + # Merge with constrained base config, then re-apply + # structured controls so prompt drift cannot override them. + merged = lock_structured_constraints( + deep_merge_config(constrained_base, patch_config), + effective_constraints, + ) try: normalized = experiment_service.normalize_simulation_config(merged) disabled_errors = collect_disabled_option_errors(normalized, schema) @@ -222,6 +233,7 @@ async def _node_run_sequential(self, state: AgentState) -> AgentState: base_config = experiment_service.normalize_simulation_config( deep_merge_config(build_initial_config(schema), state.config_constraints) ) + effective_constraints = merge_llm_resource_constraints(base_config, state.config_constraints) max_wait_seconds = 3600 poll_interval = 2 @@ -232,7 +244,10 @@ async def _node_run_sequential(self, state: AgentState) -> AgentState: state.iteration = idx + 1 # -- Build config -- - merged = deep_merge_config(base_config, plan.config_patch) + merged = lock_structured_constraints( + deep_merge_config(base_config, plan.config_patch), + effective_constraints, + ) config = experiment_service.normalize_simulation_config(merged) disabled_errors = collect_disabled_option_errors(config, schema) if disabled_errors: @@ -295,7 +310,7 @@ async def _node_run_sequential(self, state: AgentState) -> AgentState: name=plan.name, iteration_goal=plan.iteration_goal, plan_summary=plan.plan_summary, - score=get_last_global_accuracy(metrics), + score=get_agent_run_score(metrics), decision="recorded", ) state.experiment_results.append(record) diff --git a/apps/backend/app/services/agent/history_service.py b/apps/backend/app/services/agent/history_service.py index 7dce611..a9b4b96 100644 --- a/apps/backend/app/services/agent/history_service.py +++ b/apps/backend/app/services/agent/history_service.py @@ -5,6 +5,8 @@ from __future__ import annotations import copy +import hashlib +import json from datetime import datetime from typing import Any @@ -12,10 +14,11 @@ from sqlalchemy.ext.asyncio import AsyncSession from app.core import exceptions -from app.models.agent import AgentOptimizationJob, AgentOptimizationJobStatus -from app.repositories.agent import AgentOptimizationJobRepository +from app.models.agent import AgentConfigVersion, AgentOptimizationJob, AgentOptimizationJobStatus +from app.repositories.agent import AgentConfigVersionRepository, AgentOptimizationJobRepository -from .summary import get_last_global_accuracy +from .memory import compute_config_diff +from .summary import get_agent_run_score class AgentOptimizationHistoryService: @@ -26,6 +29,7 @@ class AgentOptimizationHistoryService: def __init__(self, session: AsyncSession): self.session = session self.repository = AgentOptimizationJobRepository(session) + self.config_version_repository = AgentConfigVersionRepository(session) async def create_job( self, @@ -47,6 +51,7 @@ async def create_job( resolved_status = AgentOptimizationJobStatus(status.lower()) try: + safe_snapshot = self._to_json_safe(snapshot) job = await self.repository.create_job( task_id=task_id, job_name=job_name, @@ -55,8 +60,9 @@ async def create_job( model_name=model_name, status=resolved_status, max_iterations=max_iterations, - snapshot_json=self._to_json_safe(snapshot), + snapshot_json=safe_snapshot, ) + await self._sync_config_versions(job, safe_snapshot) await self.session.commit() await self.session.refresh(job) return job @@ -64,8 +70,38 @@ async def create_job( await self.session.rollback() raise exceptions.JobAlreadyExists(self.JOB_NAME_CONFLICT_MESSAGE) from exc - async def list_jobs(self) -> list[AgentOptimizationJob]: - return await self.repository.list_jobs() + async def list_jobs( + self, + *, + status: AgentOptimizationJobStatus | str | None = None, + q: str | None = None, + model_name: str | None = None, + objective: str | None = None, + best_score_min: float | None = None, + best_score_max: float | None = None, + created_from: datetime | None = None, + created_to: datetime | None = None, + config_filters: dict[str, Any] | None = None, + ) -> list[AgentOptimizationJob]: + resolved_status = self._resolve_status(status) + jobs = await self.repository.list_jobs( + status=resolved_status, + q=q, + model_name=model_name, + best_score_min=best_score_min, + best_score_max=best_score_max, + created_from=created_from, + created_to=created_to, + ) + return [ + job + for job in jobs + if self._matches_snapshot_filters( + job.snapshot_json if isinstance(job.snapshot_json, dict) else {}, + objective=objective, + config_filters=config_filters or {}, + ) + ] async def get_job_or_raise(self, optimization_job_id: int) -> AgentOptimizationJob: job = await self.repository.get_job(optimization_job_id) @@ -93,9 +129,10 @@ async def update_job_snapshot( status = AgentOptimizationJobStatus(str(snapshot.get("status", AgentOptimizationJobStatus.QUEUED.value))) finished_at = snapshot.get("finished_at") if snapshot.get("finished_at") is not None else ... best_metrics = snapshot.get("best_metrics") - best_score = get_last_global_accuracy(best_metrics) if isinstance(best_metrics, dict) else None + best_score = get_agent_run_score(best_metrics) if isinstance(best_metrics, dict) else None try: + safe_snapshot = self._to_json_safe(snapshot) await self.repository.update_job( job, job_name=str(job_name) if job_name is not None else None, @@ -105,9 +142,10 @@ async def update_job_snapshot( completed_iterations=int(snapshot.get("completed_iterations") or 0), simulation_job_id=self._extract_simulation_job_id(snapshot), best_score=best_score, - snapshot_json=self._to_json_safe(snapshot), + snapshot_json=safe_snapshot, finished_at=finished_at, ) + await self._sync_config_versions(job, safe_snapshot) await self.session.commit() await self.session.refresh(job) return job @@ -132,6 +170,241 @@ async def _ensure_unique_job_name(self, job_name: str, *, exclude_job_id: int | await self.session.flush() else: raise exceptions.JobAlreadyExists(self.JOB_NAME_CONFLICT_MESSAGE) + + async def list_config_versions(self, optimization_job_id: int) -> list[AgentConfigVersion]: + job = await self.get_job_or_raise(optimization_job_id) + versions = await self.config_version_repository.list_versions(optimization_job_id, source="experiment") + if versions: + return versions + snapshot = job.snapshot_json if isinstance(job.snapshot_json, dict) else {} + if not self._build_config_version_candidates(snapshot): + return [] + await self._sync_config_versions(job, snapshot) + await self.session.commit() + return await self.config_version_repository.list_versions(optimization_job_id, source="experiment") + + async def get_config_version_or_raise(self, optimization_job_id: int, version_id: int) -> AgentConfigVersion: + await self.get_job_or_raise(optimization_job_id) + version = await self.config_version_repository.get_version(version_id) + if version is None or version.optimization_job_id != optimization_job_id: + raise exceptions.ResourceNotFound("Agent config version not found") + return version + + async def diff_config_versions( + self, + *, + optimization_job_id: int, + from_version_id: int | None, + to_version_id: int, + ) -> list[dict[str, Any]]: + to_version = await self.get_config_version_or_raise(optimization_job_id, to_version_id) + previous_config = None + if from_version_id is not None: + from_version = await self.get_config_version_or_raise(optimization_job_id, from_version_id) + previous_config = from_version.config_json if isinstance(from_version.config_json, dict) else {} + changes = compute_config_diff(previous_config, to_version.config_json if isinstance(to_version.config_json, dict) else {}) + return [self._serialize_config_change(change) for change in changes] + + async def _sync_config_versions(self, job: AgentOptimizationJob, snapshot: dict[str, Any]) -> None: + if job.id is None: + return + for candidate in self._build_config_version_candidates(snapshot): + existing = await self.config_version_repository.find_version( + optimization_job_id=job.id, + run_id=candidate["run_id"], + iteration=candidate["iteration"], + source=candidate["source"], + config_hash=candidate["config_hash"], + ) + if existing is not None: + continue + await self.config_version_repository.create_version( + optimization_job_id=job.id, + run_id=candidate["run_id"], + iteration=candidate["iteration"], + source=candidate["source"], + label=candidate["label"], + config_hash=candidate["config_hash"], + config_json=candidate["config_json"], + diff_json=candidate["diff_json"], + ) + + @classmethod + def _build_config_version_candidates(cls, snapshot: dict[str, Any]) -> list[dict[str, Any]]: + candidates: list[dict[str, Any]] = [] + previous_config: dict[str, Any] | None = None + + experiments = snapshot.get("experiments") + if isinstance(experiments, list): + for index, item in enumerate(experiments, start=1): + if not isinstance(item, dict): + continue + config = item.get("config") + if not isinstance(config, dict): + continue + candidates.append( + cls._build_config_version_candidate( + config=config, + previous_config=previous_config, + iteration=int(item.get("iteration") or index), + source="experiment", + run_id=str(item.get("run_id")) if item.get("run_id") is not None else None, + label=str(item.get("name") or item.get("plan_summary") or f"Experiment {index}"), + diff_json=item.get("config_diff"), + ) + ) + previous_config = config + + return candidates + + @classmethod + def _build_config_version_candidate( + cls, + *, + config: dict[str, Any], + previous_config: dict[str, Any] | None, + iteration: int, + source: str, + run_id: str | None, + label: str, + diff_json: Any, + ) -> dict[str, Any]: + safe_config = cls._to_json_safe(config) + if isinstance(diff_json, list): + safe_diff = cls._to_json_safe(diff_json) + else: + safe_diff = [ + cls._serialize_config_change(change) + for change in compute_config_diff(previous_config, safe_config) + ] + return { + "run_id": run_id, + "iteration": iteration, + "source": source, + "label": label[:255], + "config_hash": cls._config_hash(safe_config), + "config_json": safe_config, + "diff_json": safe_diff, + } + + @staticmethod + def _best_snapshot_experiment(snapshot: dict[str, Any]) -> dict[str, Any] | None: + experiments = snapshot.get("experiments") + if not isinstance(experiments, list): + return None + best_record = None + best_score = None + for item in experiments: + if not isinstance(item, dict): + continue + score = item.get("score") + if score is None: + continue + try: + numeric_score = float(score) + except (TypeError, ValueError): + continue + if best_score is None or numeric_score > best_score: + best_record = item + best_score = numeric_score + return best_record + + @staticmethod + def _serialize_config_change(change: Any) -> dict[str, Any]: + if isinstance(change, dict): + return { + "path": str(change.get("path", "$")), + "change_type": str(change.get("change_type", "updated")), + "old_value": copy.deepcopy(change.get("old_value")), + "new_value": copy.deepcopy(change.get("new_value")), + } + return { + "path": change.path, + "change_type": change.change_type, + "old_value": copy.deepcopy(change.old_value), + "new_value": copy.deepcopy(change.new_value), + } + + @staticmethod + def _config_hash(config: dict[str, Any]) -> str: + payload = json.dumps(config, sort_keys=True, separators=(",", ":"), default=str) + return hashlib.sha256(payload.encode("utf-8")).hexdigest() + + @classmethod + def _resolve_status(cls, status: AgentOptimizationJobStatus | str | None) -> AgentOptimizationJobStatus | None: + if status is None: + return None + if isinstance(status, AgentOptimizationJobStatus): + return status + status_text = str(status).strip().lower() + if not status_text or status_text == "all": + return None + return AgentOptimizationJobStatus(status_text) + + @classmethod + def _matches_snapshot_filters( + cls, + snapshot: dict[str, Any], + *, + objective: str | None, + config_filters: dict[str, Any], + ) -> bool: + if objective and str(snapshot.get("objective", "")).lower() != objective.lower(): + return False + for path, expected in config_filters.items(): + if expected is None or str(expected).strip() == "": + continue + if not cls._snapshot_has_config_value(snapshot, path, expected): + return False + return True + + @classmethod + def _snapshot_has_config_value(cls, snapshot: dict[str, Any], path: str, expected: Any) -> bool: + for config in cls._iter_snapshot_configs(snapshot): + value = cls._get_value_by_path(config, path) + if value is not None and cls._values_equal(value, expected): + return True + return False + + @classmethod + def _iter_snapshot_configs(cls, snapshot: dict[str, Any]): + for key in ("best_config", "config_constraints"): + value = snapshot.get(key) + if isinstance(value, dict): + yield value + current_experiment = snapshot.get("current_experiment") + if isinstance(current_experiment, dict) and isinstance(current_experiment.get("config"), dict): + yield current_experiment["config"] + for collection_key in ("experiments", "draft_experiments"): + collection = snapshot.get(collection_key) + if not isinstance(collection, list): + continue + for item in collection: + if not isinstance(item, dict): + continue + for config_key in ("config", "config_patch"): + config = item.get(config_key) + if isinstance(config, dict): + yield config + + @staticmethod + def _get_value_by_path(obj: dict[str, Any], path: str) -> Any: + current: Any = obj + for part in path.split("."): + if not isinstance(current, dict) or part not in current: + return None + current = current[part] + return current + + @staticmethod + def _values_equal(actual: Any, expected: Any) -> bool: + if isinstance(actual, (int, float)) and isinstance(expected, str): + try: + return float(actual) == float(expected) + except ValueError: + return False + return str(actual).lower() == str(expected).lower() + @staticmethod def _extract_simulation_job_id(snapshot: dict[str, Any]) -> int | None: current_experiment = snapshot.get("current_experiment") diff --git a/apps/backend/app/services/agent/objectives.py b/apps/backend/app/services/agent/objectives.py index 583b0cd..56d6724 100644 --- a/apps/backend/app/services/agent/objectives.py +++ b/apps/backend/app/services/agent/objectives.py @@ -1,14 +1,13 @@ """ Objective helpers for the bench-mode experiment executor. -Bench mode does not optimize — it runs experiments and reports results. -The objective system is kept minimal for compatibility. +Bench mode runs a planned batch of experiments and reports comparable scores. +The objective surface is intentionally small for API compatibility. """ from __future__ import annotations from enum import Enum -from typing import Any class AgentOptimizationObjective(str, Enum): @@ -35,7 +34,11 @@ def resolve_objective( goal: str, requested_objective: str | AgentOptimizationObjective | None, ) -> AgentOptimizationObjective: - """Resolve the effective objective — always accuracy in bench mode.""" + """Resolve the effective objective. + + LLM PEFT runs are scored by negative loss in the summary layer, but the + public objective enum remains accuracy-only for the current Agent API. + """ return AgentOptimizationObjective.ACCURACY @@ -57,7 +60,7 @@ def is_better_score( candidate_score: float | None, reference_score: float | None, ) -> bool: - """Higher accuracy is always better in bench mode.""" + """Higher comparable score is better.""" if candidate_score is None: return False if reference_score is None: diff --git a/apps/backend/app/services/agent/planning.py b/apps/backend/app/services/agent/planning.py index c66570c..20e0615 100644 --- a/apps/backend/app/services/agent/planning.py +++ b/apps/backend/app/services/agent/planning.py @@ -48,8 +48,9 @@ def build_initial_config(schema: dict[str, Any]) -> dict[str, Any]: """ if not schema: return { + "task": {"type": "classic_fl"}, "dataset": {"name": "CIFAR-10", "distribution": "non_iid", "alpha": 0.5}, - "model": {"name": "CNN"}, + "model": {"name": "Auto"}, "federated": { "num_clients": 3, "num_rounds": 10, @@ -88,6 +89,77 @@ def deep_merge_config(base: dict[str, Any], override: dict[str, Any] | None) -> return merged +def lock_structured_constraints(config: dict[str, Any], constraints: dict[str, Any] | None) -> dict[str, Any]: + """Apply user-selected structured constraints as final locked values.""" + if not isinstance(constraints, dict) or not constraints: + return copy.deepcopy(config) + return deep_merge_config(config, constraints) + + +def infer_sft_settings_for_dataset_path(dataset_path: str) -> dict[str, str]: + """Infer hidden SFT format selectors from a concrete dataset path.""" + normalized = dataset_path.strip().lower() + if normalized.endswith(".jsonl"): + file_format = "jsonl" + elif normalized.endswith(".parquet"): + file_format = "parquet" + else: + file_format = "auto" + + if "alpaca" in normalized: + data_format = "alpaca" + elif "messages" in normalized or "chat" in normalized: + data_format = "messages" + else: + data_format = "prompt_completion" + + return {"file_format": file_format, "format": data_format} + + +def llm_resource_constraints_from_config(config: dict[str, Any]) -> dict[str, Any]: + """Lock concrete LLM PEFT resources from the effective base config.""" + task = config.get("task") + if not isinstance(task, dict) or task.get("type") != "llm_peft_sft": + return {} + + constraints: dict[str, Any] = {"task": {"type": "llm_peft_sft"}} + llm = config.get("llm") + if isinstance(llm, dict) and isinstance(llm.get("base_model"), str) and llm["base_model"].strip(): + constraints["llm"] = {"base_model": llm["base_model"]} + + sft = config.get("sft") + if isinstance(sft, dict) and isinstance(sft.get("dataset_path"), str) and sft["dataset_path"].strip(): + constraints["sft"] = { + "dataset_path": sft["dataset_path"], + **infer_sft_settings_for_dataset_path(sft["dataset_path"]), + } + + evaluation = config.get("evaluation") + if isinstance(evaluation, dict): + evaluation_constraints: dict[str, Any] = {} + if isinstance(evaluation.get("enable"), bool): + evaluation_constraints["enable"] = evaluation["enable"] + if ( + evaluation.get("enable") is True + and isinstance(evaluation.get("dataset_path"), str) + and evaluation["dataset_path"].strip() + ): + evaluation_constraints["dataset_path"] = evaluation["dataset_path"] + if evaluation_constraints: + constraints["evaluation"] = evaluation_constraints + + return constraints + + +def merge_llm_resource_constraints( + config: dict[str, Any], + constraints: dict[str, Any] | None, +) -> dict[str, Any]: + """Merge effective LLM resource defaults with explicit user constraints.""" + resource_constraints = llm_resource_constraints_from_config(config) + return deep_merge_config(resource_constraints, constraints or {}) + + def _option_ui(definition: dict[str, Any]) -> dict[str, Any]: ui = definition.get("ui") if not isinstance(ui, dict): diff --git a/apps/backend/app/services/agent/prompts/__init__.py b/apps/backend/app/services/agent/prompts/__init__.py index 73a3276..a7ebb78 100644 --- a/apps/backend/app/services/agent/prompts/__init__.py +++ b/apps/backend/app/services/agent/prompts/__init__.py @@ -48,7 +48,12 @@ def build_plan_prompt( f"Default base config after applying user constraints:\n{dumps_for_prompt(base_config)}\n\n" f"User-selected structured constraints:\n{dumps_for_prompt(constraints)}\n\n" "Parse the request into experiment configurations.\n" - "Every experiment must inherit the structured constraints unless the user's request explicitly changes them.\n" + "User-selected structured constraints are locked controls and have higher priority than natural-language text.\n" + "Every experiment must inherit the structured constraints. If text conflicts with them, keep the structured constraints.\n" "Use executable_options for select fields. Do not use disabled_options.\n" + "For LLM PEFT simulation requests, set task.type=\"llm_peft_sft\" and use llm/sft/peft/federated fields.\n" + "For LLM validation or perplexity requests, set evaluation.enable=true and include evaluation.dataset_path when available.\n" + "Respect dataset_model_compatibility. Use model.name=\"Auto\" when the user changes dataset without naming a model.\n" + "Respect privacy/compression compatibility rules from Runtime capabilities.\n" "Return ONLY a JSON object with plan_summary and experiments list, no commentary." ) diff --git a/apps/backend/app/services/agent/prompts/plan_instructions.txt b/apps/backend/app/services/agent/prompts/plan_instructions.txt index 316aaa1..b624d30 100644 --- a/apps/backend/app/services/agent/prompts/plan_instructions.txt +++ b/apps/backend/app/services/agent/prompts/plan_instructions.txt @@ -5,9 +5,15 @@ Rules: - Treat config_schema.yaml as the source of truth for fields, defaults, selectable options, and disabled future options. - If the user mentions multiple parameter values, generate one experiment per value. - If the user mentions multiple dimensions, generate the Cartesian product. -- If the user does not specify a field, inherit the default base config and structured constraints from the prompt. +- User-selected structured constraints are locked controls. Every experiment must inherit them, even if the natural-language text or an example mentions a different value. +- If the user does not specify a field, inherit the default base config from the prompt. - Each experiment's "config" should contain the actual parameter values that make that experiment distinct. - Do not use any option listed under disabled_options. +- For LLM PEFT simulation requests, set task.type="llm_peft_sft" and use the llm, sft, peft, and federated fields. Do not vary classic dataset/model/privacy/compression fields unless the user explicitly asks for the classic FL route. +- For LLM PEFT validation or perplexity requests, set evaluation.enable=true and provide evaluation.dataset_path when the user names a validation JSONL file. +- Respect Runtime capabilities, especially dataset_model_compatibility and privacy/compression/aggregation compatibility. +- Never combine CKKS with compression or secure aggregation. Secure aggregation is only valid with fedavg/simple_avg in simulation. Differential privacy can be combined with supported compression/encryption. +- If the user changes dataset but does not ask for a model, use model.name="Auto" or the dataset's default_model. - `federated.seed` controls random seed for full reproducibility. Do not vary it unless the user asks for repeats, variance, or specific seeds. - Return ONLY a JSON object with: - "plan_summary": brief description of what will be tested @@ -15,7 +21,7 @@ Rules: Example: user says "test alpha=0.1, 0.3, 0.5": { - "plan_summary": "Compare non-IID alpha values 0.1, 0.3, 0.5 on CIFAR-10", + "plan_summary": "Compare non-IID alpha values 0.1, 0.3, 0.5 on the selected dataset", "experiments": [ {"name": "alpha_0.1", "config": {"dataset": {"alpha": 0.1}}}, {"name": "alpha_0.3", "config": {"dataset": {"alpha": 0.3}}}, diff --git a/apps/backend/app/services/agent/summary.py b/apps/backend/app/services/agent/summary.py index 04e0b5c..16a3d59 100644 --- a/apps/backend/app/services/agent/summary.py +++ b/apps/backend/app/services/agent/summary.py @@ -19,12 +19,63 @@ def get_last_global_accuracy(metrics: dict[str, Any]) -> float | None: return float(last) if isinstance(last, (int, float)) else None +def get_last_llm_train_loss(metrics: dict[str, Any]) -> float | None: + """Extract the final LLM train loss from run metrics.""" + llm_results = metrics.get("llm_results") or {} + train_loss = llm_results.get("train_loss") or [] + if not train_loss: + return None + last = train_loss[-1] + return float(last) if isinstance(last, (int, float)) else None + + +def get_last_llm_validation_loss(metrics: dict[str, Any]) -> float | None: + """Extract the final non-zero LLM validation loss from run metrics.""" + llm_results = metrics.get("llm_results") or {} + validation_loss = llm_results.get("validation_loss") or [] + for value in reversed(validation_loss): + if isinstance(value, (int, float)) and float(value) > 0: + return float(value) + return None + + +def get_last_llm_loss(metrics: dict[str, Any]) -> float | None: + """Return validation loss when present, otherwise train loss.""" + return get_last_llm_validation_loss(metrics) or get_last_llm_train_loss(metrics) + + +def get_agent_run_score(metrics: dict[str, Any]) -> float | None: + """Return the comparable score used by Agent result ranking. + + Classic FL keeps higher-is-better accuracy. LLM PEFT uses negative loss + so lower loss ranks higher while preserving the same max() flow. + """ + llm_loss = get_last_llm_loss(metrics) + if llm_loss is not None: + return -llm_loss + return get_last_global_accuracy(metrics) + + def _format_score(value: float | None) -> str: if value is None: return "-" return f"{value:.4f}" +def _is_llm_record(record: ExperimentRecord) -> bool: + return get_last_llm_loss(record.metrics) is not None + + +def _format_record_metric(record: ExperimentRecord) -> str: + validation_loss = get_last_llm_validation_loss(record.metrics) + if validation_loss is not None: + return f"val_loss={validation_loss:.4f}" + llm_loss = get_last_llm_train_loss(record.metrics) + if llm_loss is not None: + return f"train_loss={llm_loss:.4f}" + return _format_score(record.score) + + def _get_federated_summary(config: dict[str, Any]) -> str: federated = config.get("federated") or {} if not isinstance(federated, dict): @@ -49,15 +100,17 @@ def build_results_table(state: AgentState) -> str: return "No experiment results available." lines: list[str] = [] - header = f"{'Name':<30} {'Accuracy':<12} {'Rounds':<8} {'Config Summary'}" + metric_header = "Metric" if any(_is_llm_record(record) for record in state.experiment_results) else "Accuracy" + header = f"{'Name':<30} {metric_header:<12} {'Rounds':<8} {'Config Summary'}" lines.append(header) lines.append("-" * len(header)) for record in state.experiment_results: name = record.name or f"exp-{record.iteration}" - score = _format_score(record.score) + score = _format_record_metric(record) global_results = record.metrics.get("global_results") or {} - rounds = len(global_results.get("rounds") or []) + llm_results = record.metrics.get("llm_results") or {} + rounds = len((llm_results.get("rounds") if isinstance(llm_results, dict) else None) or global_results.get("rounds") or []) config_summary = _get_federated_summary(record.config) lines.append(f"{name:<30} {score:<12} {rounds:<8} {config_summary}") @@ -78,12 +131,20 @@ def build_summary_text(*, state: AgentState) -> str: if scored: best = max(scored, key=lambda r: r.score) # type: ignore[arg-type] worst = min(scored, key=lambda r: r.score) # type: ignore[arg-type] - lines.append( - f"Best accuracy: {_format_score(best.score)} ({best.name or f'exp-{best.iteration}'})" - ) - lines.append( - f"Worst accuracy: {_format_score(worst.score)} ({worst.name or f'exp-{worst.iteration}'})" - ) + if any(_is_llm_record(record) for record in scored): + lines.append( + f"Best LLM loss: {_format_record_metric(best)} ({best.name or f'exp-{best.iteration}'})" + ) + lines.append( + f"Worst LLM loss: {_format_record_metric(worst)} ({worst.name or f'exp-{worst.iteration}'})" + ) + else: + lines.append( + f"Best accuracy: {_format_score(best.score)} ({best.name or f'exp-{best.iteration}'})" + ) + lines.append( + f"Worst accuracy: {_format_score(worst.score)} ({worst.name or f'exp-{worst.iteration}'})" + ) lines.append("") lines.append(build_results_table(state)) @@ -156,4 +217,4 @@ def get_best_experiment(state: AgentState) -> ExperimentRecord | None: scored = [r for r in state.experiment_results if r.score is not None] if not scored: return None - return max(scored, key=lambda r: r.score) \ No newline at end of file + return max(scored, key=lambda r: r.score) diff --git a/apps/backend/app/services/distributed/config_service.py b/apps/backend/app/services/distributed/config_service.py index cd8e14e..bf091d9 100644 --- a/apps/backend/app/services/distributed/config_service.py +++ b/apps/backend/app/services/distributed/config_service.py @@ -10,6 +10,7 @@ from typing import Any from app.core import exceptions +from app.services.simulation.compatibility import validate_llm_simulation_mode_or_raise from app.services.simulation.job_service import SimulationJobService @@ -41,6 +42,7 @@ def normalize_server_config(cls, config_json: dict[str, Any]) -> dict[str, Any]: expected_clients = cls.extract_expected_clients(normalized) cls._set_value_by_path(normalized, "federated.num_clients", expected_clients) cls._set_value_by_path(normalized, "dataset.num_clients", expected_clients) + validate_llm_simulation_mode_or_raise(normalized) return normalized @@ -86,6 +88,7 @@ def build_participant_runtime_config( cls._set_value_by_path(normalized, "distributed.client_id", assigned_participant_id) cls._set_value_by_path(normalized, "federated.num_clients", expected_clients) cls._set_value_by_path(normalized, "dataset.num_clients", expected_clients) + validate_llm_simulation_mode_or_raise(normalized) return normalized @@ -117,7 +120,7 @@ def _init_defaults_from_schema(cls, schema: dict[str, Any]) -> dict[str, Any]: """ output: dict[str, Any] = {} for key, raw_definition in schema.items(): - if key in {"role", "depends_on", "hidden"}: + if key in {"role", "depends_on", "hidden", "ui"}: continue if not isinstance(raw_definition, dict): continue diff --git a/apps/backend/app/services/llm_resources.py b/apps/backend/app/services/llm_resources.py new file mode 100644 index 0000000..8452621 --- /dev/null +++ b/apps/backend/app/services/llm_resources.py @@ -0,0 +1,215 @@ +"""Local LLM model and dataset discovery for schema-backed UI options.""" + +from __future__ import annotations + +from pathlib import Path +from typing import Any + + +LLM_MODEL_ROOT = Path("models") / "llm" +LLM_DATASET_ROOT = Path("datasets") / "llm" +DEFAULT_LLM_BASE_MODEL = "Qwen/Qwen2.5-0.5B-Instruct" +DEFAULT_LLM_DATASET_PATH = "./datasets/llm/train.jsonl" +DEFAULT_LLM_EVALUATION_DATASET_PATH = "./datasets/llm/validation.jsonl" +SFT_DATASET_SUFFIXES = {".jsonl", ".parquet"} +EVALUATION_DATASET_MARKERS = ("eval", "valid", "validation") + + +def discover_llm_model_options(project_root: Path) -> list[str]: + """Return local model directory paths under models/llm as schema option values.""" + root = project_root / LLM_MODEL_ROOT + if not root.exists() or not root.is_dir(): + return [] + + candidates: list[Path] = [] + for child in root.iterdir(): + if child.is_dir() and _is_visible_relative(child, project_root): + candidates.append(child) + + for config_file in root.rglob("config.json"): + model_dir = _model_option_dir_for_config(config_file, root) + if model_dir.is_dir() and _is_visible_relative(model_dir, project_root): + candidates.append(model_dir) + + return _unique_sorted_relative_options(candidates, project_root) + + +def discover_llm_dataset_options(project_root: Path) -> list[str]: + """Return local SFT training files and dataset directories under datasets/llm.""" + return _discover_llm_dataset_options(project_root, include_evaluation_files=False) + + +def discover_llm_evaluation_dataset_options(project_root: Path) -> list[str]: + """Return local evaluation files and dataset directories under datasets/llm.""" + return _discover_llm_dataset_options(project_root, include_evaluation_files=True) + + +def get_llm_resource_options(project_root: Path) -> dict[str, Any]: + """Return dynamic local LLM resource options for API and schema callers.""" + model_options = discover_llm_model_options(project_root) + dataset_options = discover_llm_dataset_options(project_root) + evaluation_dataset_options = discover_llm_evaluation_dataset_options(project_root) + return { + "models": model_options, + "datasets": dataset_options, + "evaluation_datasets": evaluation_dataset_options, + "default_model": model_options[0] if model_options else "", + "default_dataset": dataset_options[0] if dataset_options else "", + "default_evaluation_dataset": _preferred_evaluation_dataset(evaluation_dataset_options), + } + + +def augment_config_schema_with_llm_resources(schema: dict[str, Any], project_root: Path) -> dict[str, Any]: + """Inject discovered local LLM resource options into the mutable config schema.""" + resources = get_llm_resource_options(project_root) + + _set_field_options( + schema, + ("llm", "base_model"), + resources["models"], + source_dir=f"./{LLM_MODEL_ROOT.as_posix()}", + default_value=resources["default_model"], + empty_message="No local models found", + ) + _set_field_options( + schema, + ("sft", "dataset_path"), + resources["datasets"], + source_dir=f"./{LLM_DATASET_ROOT.as_posix()}", + default_value=resources["default_dataset"], + empty_message="No training datasets found", + ) + _set_field_options( + schema, + ("evaluation", "dataset_path"), + resources["evaluation_datasets"], + source_dir=f"./{LLM_DATASET_ROOT.as_posix()}", + default_value=resources["default_evaluation_dataset"], + empty_message="No evaluation datasets found", + ) + _set_field_default(schema, ("evaluation", "enable"), bool(resources["default_evaluation_dataset"])) + return schema + + +def _discover_llm_dataset_options(project_root: Path, *, include_evaluation_files: bool) -> list[str]: + root = project_root / LLM_DATASET_ROOT + if not root.exists() or not root.is_dir(): + return [] + + candidates: list[Path] = [] + for path in root.rglob("*"): + if ( + not path.is_file() + or path.suffix.lower() not in SFT_DATASET_SUFFIXES + or not _is_visible_relative(path, project_root) + ): + continue + if not include_evaluation_files and _is_evaluation_dataset(path): + continue + if path.suffix.lower() == ".parquet": + dataset_dir = _dataset_option_dir_for_parquet(path, root) + if dataset_dir is not None and _is_visible_relative(dataset_dir, project_root): + candidates.append(dataset_dir) + continue + candidates.append(path) + return _unique_sorted_relative_options(candidates, project_root) + + +def _model_option_dir_for_config(config_file: Path, model_root: Path) -> Path: + try: + relative = config_file.relative_to(model_root) + except ValueError: + return config_file.parent + + parts = relative.parts + if "snapshots" in parts: + snapshot_index = parts.index("snapshots") + if snapshot_index > 0: + return model_root.joinpath(*parts[:snapshot_index]) + return config_file.parent + + +def _dataset_option_dir_for_parquet(path: Path, dataset_root: Path) -> Path | None: + try: + relative = path.relative_to(dataset_root) + except ValueError: + return None + if len(relative.parts) <= 1: + return None + return dataset_root / relative.parts[0] + + +def _set_field_options( + schema: dict[str, Any], + path: tuple[str, ...], + options: list[str], + *, + source_dir: str, + default_value: str, + empty_message: str, +) -> None: + node: Any = schema + for part in path: + if not isinstance(node, dict): + return + node = node.get(part) + if not isinstance(node, dict): + return + + node["options"] = options + node["default"] = default_value + ui = node.setdefault("ui", {}) + if isinstance(ui, dict): + ui["option_source"] = { + "type": "local_directory", + "path": source_dir, + } + ui["empty_message"] = empty_message + + +def _preferred_evaluation_dataset(dataset_options: list[str]) -> str: + for option in dataset_options: + if _is_evaluation_option(option): + return option + return "" + + +def _is_evaluation_option(option: str) -> bool: + stem = Path(option).stem.lower() + return any(marker in stem for marker in EVALUATION_DATASET_MARKERS) + + +def _is_evaluation_dataset(path: Path) -> bool: + stem = path.stem.lower() + return any(marker in stem for marker in EVALUATION_DATASET_MARKERS) + + +def _set_field_default(schema: dict[str, Any], path: tuple[str, ...], default_value: Any) -> None: + node: Any = schema + for part in path: + if not isinstance(node, dict): + return + node = node.get(part) + if isinstance(node, dict): + node["default"] = default_value + + +def _unique_sorted_relative_options(paths: list[Path], project_root: Path) -> list[str]: + options = {_relative_option(path, project_root) for path in paths} + return sorted(options, key=str.lower) + + +def _relative_option(path: Path, project_root: Path) -> str: + try: + relative = path.relative_to(project_root) + except ValueError: + return path.as_posix() + return f"./{relative.as_posix()}" + + +def _is_visible_relative(path: Path, project_root: Path) -> bool: + try: + relative = path.relative_to(project_root) + except ValueError: + return False + return all(part and not part.startswith(".") for part in relative.parts) diff --git a/apps/backend/app/services/run_artifacts.py b/apps/backend/app/services/run_artifacts.py new file mode 100644 index 0000000..ae2907d --- /dev/null +++ b/apps/backend/app/services/run_artifacts.py @@ -0,0 +1,39 @@ +"""Shared naming helpers for run-local config, log, and result files.""" + +from __future__ import annotations + +from datetime import datetime, timezone + + +def run_artifact_timestamp(value: datetime | None = None) -> str: + """Return a stable UTC timestamp prefix for one run's local artifacts.""" + moment = value or datetime.now(timezone.utc) + if moment.tzinfo is None: + moment = moment.replace(tzinfo=timezone.utc) + else: + moment = moment.astimezone(timezone.utc) + return moment.strftime("%Y%m%d_%H%M%SZ") + + +def run_artifact_stem(run_id: str, *, timestamp: str) -> str: + return f"{timestamp}_{run_id}" + + +def legacy_run_artifact_stem(run_id: str, *, timestamp: str) -> str: + return f"{run_id}_{timestamp}" + + +def run_config_filename(run_id: str, *, timestamp: str) -> str: + return f"{run_artifact_stem(run_id, timestamp=timestamp)}.json" + + +def run_log_filename(run_id: str, *, timestamp: str, role: str) -> str: + return f"{run_artifact_stem(run_id, timestamp=timestamp)}_{role}.log" + + +def live_results_filename(run_id: str, *, timestamp: str) -> str: + return f"{run_artifact_stem(run_id, timestamp=timestamp)}_live_results.json" + + +def legacy_live_results_filename(run_id: str, *, timestamp: str) -> str: + return f"live_results_{legacy_run_artifact_stem(run_id, timestamp=timestamp)}.json" diff --git a/apps/backend/app/services/simulation/compatibility.py b/apps/backend/app/services/simulation/compatibility.py new file mode 100644 index 0000000..a280543 --- /dev/null +++ b/apps/backend/app/services/simulation/compatibility.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +import sys +from pathlib import Path +from typing import Any + +from app.core import exceptions + +CLASSIC_FL_TASK = "classic_fl" +LLM_PEFT_SFT_TASK = "llm_peft_sft" +LLM_SIMULATION_ONLY_MESSAGE = "LLM PEFT is currently supported in simulation mode only" + + +def _project_root() -> Path: + return Path(__file__).resolve().parents[5] + + +def _ensure_fl_core_path() -> None: + libs_path = _project_root() / "libs" + if str(libs_path) not in sys.path: + sys.path.insert(0, str(libs_path)) + + +def _registry(): + _ensure_fl_core_path() + from fl_core import simulation_registry + + return simulation_registry + + +def runtime_task_type(config: dict[str, Any]) -> str: + """Return the training route encoded in config, defaulting legacy configs to classic FL.""" + task = config.get("task") + if not isinstance(task, dict): + return CLASSIC_FL_TASK + raw_task_type = task.get("type", CLASSIC_FL_TASK) + if not isinstance(raw_task_type, str) or not raw_task_type.strip(): + return CLASSIC_FL_TASK + return raw_task_type.strip() + + +def runtime_mode(config: dict[str, Any]) -> str: + """Return the selected runtime mode, defaulting missing legacy configs to simulation.""" + system = config.get("system") + if not isinstance(system, dict): + return "simulation" + raw_mode = system.get("mode", "simulation") + if not isinstance(raw_mode, str) or not raw_mode.strip(): + return "simulation" + return raw_mode.strip() + + +def validate_llm_simulation_mode_or_raise(config: dict[str, Any]) -> None: + if runtime_task_type(config) == LLM_PEFT_SFT_TASK and runtime_mode(config) != "simulation": + raise exceptions.BadRequestError(LLM_SIMULATION_ONLY_MESSAGE) + + +def canonicalize_runtime_config(config: dict[str, Any]) -> None: + if runtime_task_type(config) != CLASSIC_FL_TASK: + return + + registry = _registry() + + dataset = config.get("dataset") + if not isinstance(dataset, dict): + return + raw_dataset_name = dataset.get("name") + if isinstance(raw_dataset_name, str): + dataset["name"] = registry.canonical_dataset_name(raw_dataset_name) + + spec = registry.get_dataset_spec(dataset.get("name")) + + model = config.get("model") + if not isinstance(model, dict): + return + raw_model_name = model.get("name") + if isinstance(raw_model_name, str) and raw_model_name.strip().lower() != "auto": + model["name"] = registry.canonical_model_name(raw_model_name) + if spec is not None: + model["input_shape"] = list(spec.input_shape) + model["num_classes"] = spec.num_classes + + +def validate_runtime_config_or_raise(config: dict[str, Any]) -> None: + validate_llm_simulation_mode_or_raise(config) + + if runtime_task_type(config) != CLASSIC_FL_TASK: + return + + registry = _registry() + errors = registry.validate_training_combination(config) + if errors: + raise exceptions.BadRequestError("; ".join(errors)) + + +def runtime_capabilities() -> dict[str, Any]: + return _registry().capabilities_payload() diff --git a/apps/backend/app/services/simulation/job_service.py b/apps/backend/app/services/simulation/job_service.py index 59b4386..1cb7b4d 100644 --- a/apps/backend/app/services/simulation/job_service.py +++ b/apps/backend/app/services/simulation/job_service.py @@ -15,6 +15,8 @@ from app.core import exceptions from app.models.simulation import SimulationJob, SimulationJobStatus, SimulationRunStatus from app.repositories.simulation import SimulationJobRepository, SimulationRunRepository +from app.services.llm_resources import augment_config_schema_with_llm_resources +from app.services.simulation.compatibility import canonicalize_runtime_config, validate_runtime_config_or_raise class SimulationJobService: @@ -192,7 +194,7 @@ def get_config_schema(cls) -> dict[str, Any]: loaded = yaml.safe_load(schema_path.read_text(encoding="utf-8")) if not isinstance(loaded, dict): raise exceptions.InternalServiceError("Config schema must be a YAML object.") - return loaded + return augment_config_schema_with_llm_resources(loaded, cls._project_root()) @classmethod def normalize_simulation_config(cls, config: dict[str, Any]) -> dict[str, Any]: @@ -230,7 +232,9 @@ def _normalize_simulation_config(cls, config: dict[str, Any]) -> dict[str, Any]: system["mode"] = "simulation" system["node_role"] = "server" + canonicalize_runtime_config(merged) cls._validate_config_node(merged, schema, path_prefix="") + validate_runtime_config_or_raise(merged) return merged @classmethod @@ -349,6 +353,12 @@ def _cleanup_run_local_files(self, run_id: str) -> None: self._project_root() / "config" / "runs" / f"{run_id}.json", self._project_root() / "results" / f"live_results_{run_id}.json", ] + candidates.extend((self._project_root() / "configs" / "simulation_runs").glob(f"{run_id}*.json")) + candidates.extend((self._project_root() / "configs" / "simulation_runs").glob(f"*_{run_id}.json")) + candidates.extend((self._project_root() / "results").glob(f"live_results_{run_id}*.json")) + candidates.extend((self._project_root() / "results").glob(f"*_{run_id}_live_results.json")) + candidates.extend((self._project_root() / "logs" / "simulation_runs").glob(f"{run_id}*_server.log")) + candidates.extend((self._project_root() / "logs" / "simulation_runs").glob(f"*_{run_id}_server.log")) for path in candidates: try: if path.exists() and path.is_file(): diff --git a/apps/backend/app/services/simulation/run_metrics_service.py b/apps/backend/app/services/simulation/run_metrics_service.py index 6da54fe..253e393 100644 --- a/apps/backend/app/services/simulation/run_metrics_service.py +++ b/apps/backend/app/services/simulation/run_metrics_service.py @@ -15,6 +15,7 @@ from app.core import exceptions from app.models.simulation import SimulationRun, SimulationRunStatus from app.repositories.simulation import SimulationRunRepository +from app.services.run_artifacts import legacy_live_results_filename, live_results_filename, run_artifact_timestamp TRAINING_RESULT_ARTIFACT = "training_result" TERMINAL_RUN_STATUSES = { @@ -43,14 +44,11 @@ async def get_run_metrics(self, run_id: str) -> dict[str, Any]: if not run: raise exceptions.RunNotFound("Run not found") - if run.status in TERMINAL_RUN_STATUSES and isinstance(run.metrics_json, dict) and run.metrics_json: - return self._normalize_metrics_payload(run.metrics_json) - - live_result_path = self._results_dir() / self._build_live_results_filename(run_id) - loaded = self._load_metrics_file(live_result_path) - if loaded is not None: - await self._persist_run_metrics_if_changed(run, loaded) - return loaded + for live_result_path in self._live_result_candidates_for_run(run): + loaded = self._load_metrics_file(live_result_path) + if loaded is not None: + await self._persist_run_metrics_if_changed(run, loaded) + return loaded results = await self.run_repository.list_results(run_id) training_results = [result for result in results if result.artifact_type == TRAINING_RESULT_ARTIFACT] @@ -120,6 +118,18 @@ def empty_metrics_payload() -> dict[str, Any]: "global_loss": [], "global_accuracy": [], }, + "llm_results": { + "rounds": [], + "train_loss": [], + "validation_loss": [], + "perplexity": [], + "token_throughput": [], + "adapter_size_bytes": [], + }, + "llm_dataset": {}, + "llm_evaluation": {}, + "llm_runtime": {}, + "llm_artifacts": [], "client_results": {}, } @@ -147,11 +157,32 @@ def _results_dir(self) -> Path: return path - def _build_live_results_filename(self, run_id: str) -> str: + def _build_live_results_filename( + self, + run_id: str, + *, + artifact_timestamp: str | None = None, + ) -> str: """ Build live result filename for one run ID. """ - return f"live_results_{run_id}.json" + if artifact_timestamp is None: + return f"live_results_{run_id}.json" + return live_results_filename(run_id, timestamp=artifact_timestamp) + + + def _live_result_candidates_for_run(self, run: SimulationRun) -> list[Path]: + directory = self._results_dir() + timestamped = directory / self._build_live_results_filename( + run.id, + artifact_timestamp=run_artifact_timestamp(run.created_at), + ) + legacy_timestamped = directory / legacy_live_results_filename( + run.id, + timestamp=run_artifact_timestamp(run.created_at), + ) + legacy = directory / self._build_live_results_filename(run.id) + return list(dict.fromkeys([timestamped, legacy_timestamped, legacy])) def _load_metrics_file(self, file_path: Path) -> dict[str, Any] | None: @@ -179,9 +210,9 @@ def _guess_live_result_file_for_run(self, run: SimulationRun) -> Path | None: if not directory.exists(): return None - expected = directory / self._build_live_results_filename(run.id) - if expected.exists() and expected.is_file(): - return expected + for expected in self._live_result_candidates_for_run(run): + if expected.exists() and expected.is_file(): + return expected started_at = run.started_at or run.created_at if started_at is None: @@ -189,13 +220,14 @@ def _guess_live_result_file_for_run(self, run: SimulationRun) -> Path | None: start_ts = started_at.timestamp() candidates: list[Path] = [] - for path in directory.glob("live_results_*.json"): - try: - mtime = path.stat().st_mtime - except OSError: - continue - if mtime >= start_ts - 5: - candidates.append(path) + for pattern in ("*_live_results.json", "live_results_*.json"): + for path in directory.glob(pattern): + try: + mtime = path.stat().st_mtime + except OSError: + continue + if mtime >= start_ts - 5: + candidates.append(path) if not candidates: return None @@ -231,6 +263,35 @@ def _normalize_metrics_payload(self, payload: dict[str, Any]) -> dict[str, Any]: "global_accuracy": self._to_float_list(global_results.get("global_accuracy")), } + llm_results = payload.get("llm_results") + if isinstance(llm_results, dict): + normalized["llm_results"] = { + "rounds": self._to_int_list(llm_results.get("rounds")), + "train_loss": self._to_float_list(llm_results.get("train_loss")), + "validation_loss": self._to_float_list(llm_results.get("validation_loss")), + "perplexity": self._to_float_list(llm_results.get("perplexity")), + "token_throughput": self._to_float_list(llm_results.get("token_throughput")), + "adapter_size_bytes": self._to_int_list(llm_results.get("adapter_size_bytes")), + } + + llm_dataset = payload.get("llm_dataset") + if isinstance(llm_dataset, dict): + normalized["llm_dataset"] = llm_dataset + + llm_evaluation = payload.get("llm_evaluation") + if isinstance(llm_evaluation, dict): + normalized["llm_evaluation"] = llm_evaluation + + llm_runtime = payload.get("llm_runtime") + if isinstance(llm_runtime, dict): + normalized["llm_runtime"] = llm_runtime + + llm_artifacts = payload.get("llm_artifacts") + if isinstance(llm_artifacts, list): + normalized["llm_artifacts"] = [ + artifact for artifact in llm_artifacts if isinstance(artifact, dict) + ] + client_results = payload.get("client_results") if isinstance(client_results, dict): normalized_clients: dict[str, dict[str, list[float]]] = {} @@ -238,6 +299,7 @@ def _normalize_metrics_payload(self, payload: dict[str, Any]) -> dict[str, Any]: if not isinstance(client_name, str) or not isinstance(client_data, dict): continue normalized_clients[client_name] = { + "rounds": self._to_int_list(client_data.get("rounds")), "train_loss": self._to_float_list(client_data.get("train_loss")), "train_acc": self._to_float_list(client_data.get("train_acc")), "test_loss": self._to_float_list(client_data.get("test_loss")), diff --git a/apps/backend/app/services/simulation/run_service.py b/apps/backend/app/services/simulation/run_service.py index eedb92a..ef107b4 100644 --- a/apps/backend/app/services/simulation/run_service.py +++ b/apps/backend/app/services/simulation/run_service.py @@ -19,9 +19,15 @@ from app.core.db import AsyncSessionLocal from app.models.simulation import SimulationRun, SimulationRunLog, SimulationRunResult, SimulationRunStatus from app.repositories.simulation import SimulationJobRepository, SimulationRunRepository +from app.services.run_artifacts import ( + live_results_filename, + run_artifact_timestamp, + run_config_filename, + run_log_filename, +) from app.services.simulation.run_metrics_service import SimulationRunMetricsService -RESULT_PATH_PATTERN = re.compile(r"结果已保存到[::]\s*(.+)$") +RESULT_PATH_PATTERN = re.compile(r"\u7ed3\u679c\u5df2\u4fdd\u5b58\u5230[:\uff1a]\s*(.+)$") LOG_LEVEL_PREFIX_PATTERN = re.compile( r"^\s*(DEBUG|INFO|WARNING|WARN|ERROR|CRITICAL)\b[\s:\-]", re.IGNORECASE, @@ -199,18 +205,22 @@ async def _start_run_with_config(self, *, job_id: int, config_json: dict[str, An ) await self.run_repository.create_run(run) - config_path = self._write_run_config(run.id, config_json) + artifact_timestamp = run_artifact_timestamp(run.created_at) + config_path = self._write_run_config(run.id, config_json, artifact_timestamp=artifact_timestamp) await self.run_repository.add_result( run_id=run.id, artifact_type=RUN_CONFIG_ARTIFACT, path=str(config_path), - metadata_json={"source": "run_config"}, + metadata_json={"source": "run_config", "artifact_timestamp": artifact_timestamp}, ) command = self._build_train_command(config_path) run.command = " ".join(command) - live_results_path = self._results_dir() / self._build_live_results_filename(run.id) + live_results_path = self._results_dir() / self._build_live_results_filename( + run.id, + artifact_timestamp=artifact_timestamp, + ) child_env = self._build_run_environment(live_results_path) process = await self._spawn_subprocess(command, env=child_env) @@ -225,16 +235,24 @@ async def _start_run_with_config(self, *, job_id: int, config_json: dict[str, An await self.session.refresh(run) self._processes[run.id] = process - self._tasks[run.id] = asyncio.create_task(self._watch_process(run.id, process)) + self._tasks[run.id] = asyncio.create_task( + self._watch_process(run.id, process, artifact_timestamp=artifact_timestamp) + ) return run - async def _watch_process(self, run_id: str, process: asyncio.subprocess.Process) -> None: + async def _watch_process( + self, + run_id: str, + process: asyncio.subprocess.Process, + *, + artifact_timestamp: str | None = None, + ) -> None: """ Consume subprocess streams and persist final run status/artifacts. """ result_path: str | None = None - log_path = self._simulation_log_path(run_id) + log_path = self._simulation_log_path(run_id, artifact_timestamp=artifact_timestamp) async def consume(stream: asyncio.StreamReader, level: str, prefix: str) -> None: nonlocal result_path @@ -248,7 +266,7 @@ async def consume(stream: asyncio.StreamReader, level: str, prefix: str) -> None if not message: continue - file_obj.write(f"[{prefix}] {message}\\n") + file_obj.write(f"[{prefix}] {message}\n") file_obj.flush() match = RESULT_PATH_PATTERN.search(message) @@ -283,22 +301,36 @@ async def consume(stream: asyncio.StreamReader, level: str, prefix: str) -> None ) await repository.add_log(run_id, f"run finished with exit_code={exit_code}") + metrics_persisted = False if result_path: await repository.add_result( run_id=run_id, artifact_type=TRAINING_RESULT_ARTIFACT, path=result_path, - metadata_json={"source": "process_output"}, + metadata_json={ + "source": "process_output", + "artifact_timestamp": artifact_timestamp, + }, ) normalized = self.metrics_service.load_metrics_file( self.metrics_service.resolve_result_artifact_path(result_path) ) if normalized is not None: await repository.update_run_metrics(run, normalized) + metrics_persisted = True - if not run.metrics_json: + if not metrics_persisted: guessed = self.metrics_service.guess_live_result_file_for_run(run) if guessed is not None: + await repository.add_result( + run_id=run_id, + artifact_type=TRAINING_RESULT_ARTIFACT, + path=str(guessed), + metadata_json={ + "source": "live_results_fallback", + "artifact_timestamp": artifact_timestamp, + }, + ) normalized = self.metrics_service.load_metrics_file(guessed) if normalized is not None: await repository.update_run_metrics(run, normalized) @@ -343,33 +375,55 @@ def _runtime_log_dir(self) -> Path: return path - def _simulation_log_path(self, run_id: str, client_id: int | None = None) -> Path: + def _simulation_log_path( + self, + run_id: str, + client_id: int | None = None, + *, + artifact_timestamp: str | None = None, + ) -> Path: """ Build simulation log path for server or client role. """ suffix = "server" if client_id is None else f"client_{client_id}" - return self._runtime_log_dir() / f"{run_id}_{suffix}.log" + if artifact_timestamp is None: + return self._runtime_log_dir() / f"{run_id}_{suffix}.log" + return self._runtime_log_dir() / run_log_filename( + run_id, + timestamp=artifact_timestamp, + role=suffix, + ) - def _build_live_results_filename(self, run_id: str) -> str: + def _build_live_results_filename(self, run_id: str, *, artifact_timestamp: str | None = None) -> str: """ Build live result filename for one run ID. """ - return f"live_results_{run_id}.json" + if artifact_timestamp is None: + return f"live_results_{run_id}.json" + return live_results_filename(run_id, timestamp=artifact_timestamp) - def _run_config_path(self, run_id: str) -> Path: + def _run_config_path(self, run_id: str, *, artifact_timestamp: str | None = None) -> Path: """ Build local run config path. """ - return self._job_config_dir() / f"{run_id}.json" + if artifact_timestamp is None: + return self._job_config_dir() / f"{run_id}.json" + return self._job_config_dir() / run_config_filename(run_id, timestamp=artifact_timestamp) - def _write_run_config(self, run_id: str, config_json: dict[str, Any]) -> Path: + def _write_run_config( + self, + run_id: str, + config_json: dict[str, Any], + *, + artifact_timestamp: str | None = None, + ) -> Path: """ Write run config payload to JSON file. """ - path = self._run_config_path(run_id) + path = self._run_config_path(run_id, artifact_timestamp=artifact_timestamp) path.write_text(json.dumps(config_json, ensure_ascii=False, indent=2), encoding="utf-8") return path @@ -457,9 +511,19 @@ def _cleanup_run_local_files(self, run_id: str) -> None: """ Cleanup local config/results/log files for a run ID. """ - self._delete_file_if_exists(self._run_config_path(run_id)) - self._delete_file_if_exists(self._results_dir() / self._build_live_results_filename(run_id)) - self._delete_file_if_exists(self._simulation_log_path(run_id)) + for path in self._matching_run_local_files(run_id): + self._delete_file_if_exists(path) + + + def _matching_run_local_files(self, run_id: str) -> list[Path]: + return [ + *self._job_config_dir().glob(f"{run_id}*.json"), + *self._job_config_dir().glob(f"*_{run_id}.json"), + *self._results_dir().glob(f"live_results_{run_id}*.json"), + *self._results_dir().glob(f"*_{run_id}_live_results.json"), + *self._runtime_log_dir().glob(f"{run_id}*_server.log"), + *self._runtime_log_dir().glob(f"*_{run_id}_server.log"), + ] @staticmethod diff --git a/apps/backend/runners/classic_fl_runtime.py b/apps/backend/runners/classic_fl_runtime.py new file mode 100644 index 0000000..7bb1ef7 --- /dev/null +++ b/apps/backend/runners/classic_fl_runtime.py @@ -0,0 +1,12 @@ +from __future__ import annotations + +from pathlib import Path + +from core_runtime import FederatedLearningFramework + + +def run_classic_fl_runtime(config_path: Path) -> bool: + """Run the existing classic small-model federated learning runtime.""" + framework = FederatedLearningFramework(config_path=str(config_path)) + return framework.run() + diff --git a/apps/backend/runners/core_runtime.py b/apps/backend/runners/core_runtime.py index 48fce22..732eac3 100644 --- a/apps/backend/runners/core_runtime.py +++ b/apps/backend/runners/core_runtime.py @@ -22,11 +22,14 @@ from fl_core.data.data_loader import DataManager from fl_core.data.data_splitter import DataSplitter, FederatedDataManager from fl_core.models.model_manager import ModelManager +from fl_core.simulation_registry import resolve_model_name_for_dataset from fl_core.federated.server import FederatedServer from fl_core.federated.client import FederatedClient from fl_core.federated.client_manager import ClientManager -from fl_core.compression.sparsification import GlobalTopKSparsifier +from fl_core.compression.sparsification import CompressionFactory +from fl_core.privacy.differential_privacy import DifferentialPrivacyManager from fl_core.privacy.encryption import CKKSManager +from fl_core.privacy.secure_aggregation import SecureAggregationMasker from fl_core.federated.communication import GRPCServer, GRPCWorker, GRPCClientProxy WebApp = None @@ -96,14 +99,36 @@ def initialize_system(self) -> bool: self.sparsifier = None if sparsify_conf.get('enable', False): + method = sparsify_conf.get('method', 'global_topk') ratio = sparsify_conf.get('ratio', 0.5) - self.sparsifier = GlobalTopKSparsifier(ratio=ratio) + threshold = sparsify_conf.get('threshold', 1e-3) + self.sparsifier = CompressionFactory.create(method=method, ratio=ratio, threshold=threshold) print(f"✓ 通信压缩: 稀疏化已启用 (Ratio: {ratio})") privacy_config = self.config_manager.config.get('privacy', {}) ckks_conf = privacy_config.get('homomorphic_encryption', {}) - + dp_conf = privacy_config.get('differential_privacy', {}) + secure_agg_conf = privacy_config.get('secure_aggregation', {}) + + self.dp_manager = None + if dp_conf.get('enable', False): + self.dp_manager = DifferentialPrivacyManager( + clipping_norm=dp_conf.get('clipping_norm', 1.0), + noise_multiplier=dp_conf.get('noise_multiplier', 0.0), + ) + print( + "Differential privacy enabled: " + f"clip={self.dp_manager.clipping_norm}, noise={self.dp_manager.noise_multiplier}" + ) + + self.secure_aggregation = None + if secure_agg_conf.get('enable', False): + self.secure_aggregation = SecureAggregationMasker( + mask_std=secure_agg_conf.get('mask_std', 1.0), + ) + print(f"Secure aggregation masking enabled: std={self.secure_aggregation.mask_std}") + self.ckks_manager = None if ckks_conf.get('enable', False): # 暂不支持同时开启 @@ -155,7 +180,10 @@ def _get_experiment_info(self) -> Dict[str, Any]: "dataset_name": dataset_cfg.get('name', 'Unknown'), "distribution": dataset_cfg.get('distribution', 'iid'), "alpha": dataset_cfg.get('alpha', 'N/A'), - "model_name": model_cfg.get('name', 'Unknown'), + "model_name": resolve_model_name_for_dataset( + model_cfg.get('name'), + dataset_cfg.get('name', 'CIFAR-10'), + ), "num_classes": model_cfg.get('num_classes', 10) }, "federated": { @@ -169,13 +197,23 @@ def _get_experiment_info(self) -> Dict[str, Any]: "security": { "encryption": { "enabled": privacy_cfg.get('homomorphic_encryption', {}).get('enable', False), - "type": "CKKS", + "type": privacy_cfg.get('homomorphic_encryption', {}).get('method', 'ckks'), "poly_modulus_degree": privacy_cfg.get('homomorphic_encryption', {}).get('poly_modulus_degree', 0) }, + "differential_privacy": { + "enabled": privacy_cfg.get('differential_privacy', {}).get('enable', False), + "clipping_norm": privacy_cfg.get('differential_privacy', {}).get('clipping_norm', 1.0), + "noise_multiplier": privacy_cfg.get('differential_privacy', {}).get('noise_multiplier', 0.0), + }, + "secure_aggregation": { + "enabled": privacy_cfg.get('secure_aggregation', {}).get('enable', False), + "mask_std": privacy_cfg.get('secure_aggregation', {}).get('mask_std', 1.0), + }, "compression": { "enabled": compression_cfg.get('sparsification', {}).get('enable', False), - "type": "Global Top-K", - "ratio": compression_cfg.get('sparsification', {}).get('ratio', 0.0) + "type": compression_cfg.get('sparsification', {}).get('method', 'global_topk'), + "ratio": compression_cfg.get('sparsification', {}).get('ratio', 0.0), + "threshold": compression_cfg.get('sparsification', {}).get('threshold', 1e-3), } } } @@ -248,7 +286,9 @@ def run_distributed_server(self): self.client_manager = ClientManager( clients=[], sparsifier=self.sparsifier, - ckks_manager=self.ckks_manager + ckks_manager=self.ckks_manager, + dp_manager=self.dp_manager, + secure_aggregation=self.secure_aggregation, ) self.logger.log_info(f"等待 {federated_config.get('num_clients')} 个远程客户端连接...") @@ -422,11 +462,16 @@ def setup_models(self) -> bool: self.model_manager = ModelManager(model_config) dataset_info = self.data_manager.get_dataset_info() + model_name = resolve_model_name_for_dataset( + model_config.get('name'), + dataset_info['name'], + ) global_model = self.model_manager.create_model( - model_name=model_config.get('name'), + model_name=model_name, input_shape=dataset_info['input_shape'], - num_classes=dataset_info['num_classes'] + num_classes=dataset_info['num_classes'], + dataset_info=dataset_info, ) self.global_model = global_model @@ -479,6 +524,8 @@ def setup_federated_components(self) -> bool: clients=clients, sparsifier=self.sparsifier, ckks_manager=self.ckks_manager, + dp_manager=self.dp_manager, + secure_aggregation=self.secure_aggregation, selection_strategy="random" ) # Parallel client training races the global torch RNG inside @@ -524,7 +571,7 @@ def run_federated_training(self) -> bool: self.client_manager.broadcast_model_to_clients(all_clients, global_params) training_results = self.client_manager.train_clients( - selected_clients=all_clients, + selected_clients=selected_clients, epochs=local_epochs, learning_rate=learning_rate, round_num=round_num diff --git a/apps/backend/runners/experiment_runner.py b/apps/backend/runners/experiment_runner.py index bded743..e5ccdc5 100644 --- a/apps/backend/runners/experiment_runner.py +++ b/apps/backend/runners/experiment_runner.py @@ -9,7 +9,7 @@ import yaml -from core_runtime import FederatedLearningFramework +from runtime_dispatcher import run_runtime def _set_value_by_path(obj: dict[str, Any], path: str, value: Any) -> None: @@ -49,8 +49,7 @@ def _apply_runtime_profile(config: dict[str, Any], *, mode: str, role: str | Non def _run_framework(config_path: Path) -> int: - framework = FederatedLearningFramework(config_path=str(config_path)) - return 0 if framework.run() else 1 + return 0 if run_runtime(config_path) else 1 def main() -> int: diff --git a/apps/backend/runners/llm_peft_runtime.py b/apps/backend/runners/llm_peft_runtime.py new file mode 100644 index 0000000..e13cd65 --- /dev/null +++ b/apps/backend/runners/llm_peft_runtime.py @@ -0,0 +1,402 @@ +from __future__ import annotations + +import json +import os +import random +import re +import sys +import time +from pathlib import Path +from typing import Any + +import torch +import yaml + +PROJECT_ROOT = Path(__file__).resolve().parents[3] +if str(PROJECT_ROOT) not in sys.path: + sys.path.insert(0, str(PROJECT_ROOT)) +FL_CORE_PARENT = PROJECT_ROOT / "libs" +if str(FL_CORE_PARENT) not in sys.path: + sys.path.insert(0, str(FL_CORE_PARENT)) + +from fl_core.llm.config import LlmPeftRuntimeConfig, normalize_llm_peft_config +from fl_core.llm.data import load_sft_records, split_records_by_client +from fl_core.llm.aggregation import aggregate_adapter_state_dicts +from fl_core.llm.artifacts import ( + adapter_artifact_record, + adapter_state_size_bytes, + load_adapter_artifact, + save_adapter_artifact, + sha256_file, +) +from fl_core.llm.metrics import append_llm_round_metrics, empty_llm_metrics_payload +from fl_core.llm.modeling import missing_llm_runtime_dependencies +from fl_core.llm.trainer import LlmPeftTrainer + + +class LlmPeftRuntimeNotImplemented(RuntimeError): + """Raised when the planned LLM PEFT route is selected before implementation.""" + + +SFT_DATA_FORMATS = ("prompt_completion", "messages", "alpaca") + + +def run_llm_peft_runtime(config_path: Path) -> bool: + """ + Bootstrap the Phase 2 LLM PEFT federated fine-tuning route. + + This route already owns config/data/metrics preparation. The heavy + Transformers/PEFT training is routed through this entrypoint while keeping + a clear metrics record when the deployed runtime environment is incomplete. + """ + try: + raw_config = _load_config(config_path) + runtime_config = normalize_llm_peft_config(raw_config) + metrics = empty_llm_metrics_payload(runtime_config) + _write_metrics(runtime_config, metrics) + + print("LLM_PEFT_RUNTIME_SELECTED") + print(f"LLM_BASE_MODEL: {runtime_config.llm.base_model}") + print(f"SFT_DATASET: {runtime_config.sft.dataset_path}") + print(f"PEFT_ADAPTER: method={runtime_config.peft.method} rank={runtime_config.peft.rank}") + + dataset_path = _resolve_runtime_path(runtime_config.sft.dataset_path) + dataset_file_format = _effective_sft_file_format( + dataset_path, + configured_file_format=runtime_config.sft.file_format, + ) + records = load_sft_records( + dataset_path, + data_format=runtime_config.sft.format, + file_format=runtime_config.sft.file_format, + ) + client_splits = split_records_by_client( + records, + num_clients=runtime_config.federated.num_clients, + seed=runtime_config.federated.seed, + ) + metrics["llm_dataset"] = { + "num_records": len(records), + "client_record_counts": [len(client_records) for client_records in client_splits], + "format": runtime_config.sft.format, + "file_format": dataset_file_format, + "prompt_template": runtime_config.sft.prompt_template, + "path": str(dataset_path), + } + evaluation_records = None + if runtime_config.evaluation.enabled: + if runtime_config.evaluation.dataset_path is None: + raise ValueError("evaluation.dataset_path must be set when evaluation.enable is true") + evaluation_path = _resolve_runtime_path(runtime_config.evaluation.dataset_path) + evaluation_file_format = _effective_sft_file_format( + evaluation_path, + configured_file_format="auto", + ) + evaluation_records, evaluation_data_format = _load_sft_records_with_format_fallback( + evaluation_path, + preferred_format=runtime_config.sft.format, + file_format="auto", + ) + evaluation_records = evaluation_records[: runtime_config.evaluation.max_samples] + metrics["llm_evaluation"] = { + "enabled": True, + "path": str(evaluation_path), + "file_format": evaluation_file_format, + "format": evaluation_data_format, + "num_records": len(evaluation_records), + "batch_size": runtime_config.evaluation.batch_size, + "max_samples": runtime_config.evaluation.max_samples, + } + print( + "LLM_EVALUATION_ENABLED: " + f"path={evaluation_path} records={len(evaluation_records)} " + f"format={evaluation_data_format} batch_size={runtime_config.evaluation.batch_size}" + ) + else: + metrics["llm_evaluation"] = {"enabled": False} + print("LLM_EVALUATION_DISABLED") + _write_metrics(runtime_config, metrics) + + missing = missing_llm_runtime_dependencies() + if missing: + print( + "LLM_PEFT_RUNTIME_BLOCKED: missing LLM runtime dependencies: " + + ", ".join(missing) + ) + metrics["llm_runtime"] = { + "status": "blocked", + "reason": "missing_llm_runtime_dependencies", + "missing_dependencies": missing, + } + _write_metrics(runtime_config, metrics) + print(f"\u7ed3\u679c\u5df2\u4fdd\u5b58\u5230: {_result_path(runtime_config)}") + return False + + trainer = LlmPeftTrainer(runtime_config) + global_adapter_state: dict[str, torch.Tensor] | None = None + parent_adapter_path: str | None = None + parent_adapter_sha256: str | None = None + if runtime_config.peft.resume_adapter_path is not None: + resume_path = _resolve_runtime_path(runtime_config.peft.resume_adapter_path) + global_adapter_state, resume_metadata = load_adapter_artifact(resume_path) + parent_adapter_path = str(resume_path) + parent_adapter_sha256 = sha256_file(resume_path) + metrics["llm_runtime"] = { + "status": "resumed", + "resume_adapter_path": parent_adapter_path, + "resume_adapter_sha256": parent_adapter_sha256, + "resume_metadata": resume_metadata, + } + _write_metrics(runtime_config, metrics) + print(f"LLM_PEFT_RESUME_ADAPTER: {resume_path}") + + rng = random.Random(runtime_config.federated.seed) + work_dir = _runtime_work_dir(runtime_config) + adapter_dir = _adapter_dir(runtime_config) + nonempty_client_ids = [idx for idx, client_records in enumerate(client_splits) if client_records] + if not nonempty_client_ids: + raise ValueError("No clients have SFT records") + + for round_num in range(1, runtime_config.federated.num_rounds + 1): + started_at = time.perf_counter() + selected_ids = _select_clients( + nonempty_client_ids, + clients_per_round=runtime_config.federated.clients_per_round, + rng=rng, + ) + print(f"LLM_PEFT_ROUND_START: round={round_num} clients={selected_ids}") + updates = [] + for client_id in selected_ids: + print( + "LLM_PEFT_CLIENT_START: " + f"round={round_num} client={client_id} records={len(client_splits[client_id])}" + ) + update = trainer.train_client( + client_id=client_id, + records=client_splits[client_id], + round_num=round_num, + initial_adapter_state=global_adapter_state, + output_dir=work_dir / f"round_{round_num}" / f"client_{client_id}", + ) + updates.append(update) + _append_client_metrics(metrics, client_id=client_id, round_num=round_num, update=update) + print( + "LLM_PEFT_CLIENT_DONE: " + f"round={round_num} client={client_id} " + f"train_loss={float(update.metrics.get('train_loss', 0.0)):.6f} " + f"records={update.num_examples} tokens={update.num_tokens}" + ) + + global_adapter_state = aggregate_adapter_state_dicts( + updates, + strategy=runtime_config.federated.aggregation, + ) + adapter_size = adapter_state_size_bytes(global_adapter_state) + adapter_path = save_adapter_artifact( + adapter_dir / f"round_{round_num}_global_adapter.pt", + global_adapter_state, + metadata={ + "round": round_num, + "selected_clients": selected_ids, + "aggregation": runtime_config.federated.aggregation, + "base_model": runtime_config.llm.base_model, + "peft_method": runtime_config.peft.method, + "parent_adapter_path": parent_adapter_path, + "parent_adapter_sha256": parent_adapter_sha256, + }, + ) + artifact_record = adapter_artifact_record( + adapter_path, + round_num=round_num, + size_bytes=adapter_size, + selected_clients=selected_ids, + parent_path=parent_adapter_path, + parent_sha256=parent_adapter_sha256, + ) + metrics.setdefault("llm_artifacts", []).append(artifact_record) + parent_adapter_path = artifact_record["path"] + parent_adapter_sha256 = artifact_record["sha256"] + elapsed = max(time.perf_counter() - started_at, 1e-9) + total_tokens = sum(update.num_tokens for update in updates) + train_loss = _weighted_train_loss(updates) + validation_loss = None + if evaluation_records is not None: + evaluation_metrics = trainer.evaluate_adapter( + records=evaluation_records, + adapter_state=global_adapter_state, + output_dir=work_dir / f"round_{round_num}" / "evaluation", + ) + validation_loss = float(evaluation_metrics.get("validation_loss", 0.0)) + metrics["llm_evaluation"] = { + **metrics.get("llm_evaluation", {}), + "last_round": round_num, + "last_validation_loss": validation_loss, + "last_num_examples": int(evaluation_metrics.get("num_examples", len(evaluation_records))), + "last_num_tokens": int(evaluation_metrics.get("num_tokens", 0)), + } + print( + "LLM_PEFT_EVAL_DONE: " + f"round={round_num} validation_loss={validation_loss:.6f} " + f"records={int(evaluation_metrics.get('num_examples', len(evaluation_records)))}" + ) + append_llm_round_metrics( + metrics, + round_num=round_num, + train_loss=train_loss, + validation_loss=validation_loss, + token_throughput=total_tokens / elapsed, + adapter_size_bytes=adapter_size, + ) + metrics["llm_runtime"] = { + "status": "running", + "last_round": round_num, + "latest_adapter_path": artifact_record["path"], + "latest_adapter_sha256": artifact_record["sha256"], + } + _write_metrics(runtime_config, metrics) + print(f"LLM_PEFT_ROUND_DONE: round={round_num} train_loss={train_loss:.6f}") + + metrics["llm_runtime"] = { + "status": "completed", + "completed_rounds": runtime_config.federated.num_rounds, + "latest_adapter_path": parent_adapter_path, + "latest_adapter_sha256": parent_adapter_sha256, + } + _write_metrics(runtime_config, metrics) + print("LLM_PEFT_RUNTIME_COMPLETED") + print(f"\u7ed3\u679c\u5df2\u4fdd\u5b58\u5230: {_result_path(runtime_config)}") + return True + except Exception as exc: + print(f"LLM_PEFT_RUNTIME_FAILED: {exc}") + return False + + +def _load_config(path: Path) -> dict[str, Any]: + loaded = yaml.safe_load(path.read_text(encoding="utf-8")) + if loaded is None: + return {} + if not isinstance(loaded, dict): + raise ValueError("Config must be a YAML/JSON object") + return loaded + + +def _write_metrics(config: LlmPeftRuntimeConfig, payload: dict[str, Any]) -> None: + result_path = _result_path(config) + result_path.parent.mkdir(parents=True, exist_ok=True) + result_path.write_text(json.dumps(payload, ensure_ascii=False, indent=2), encoding="utf-8") + + +def _result_path(config: LlmPeftRuntimeConfig) -> Path: + override_name = os.environ.get("FIGARO_RESULTS_FILE", "").strip() + if override_name: + return PROJECT_ROOT / config.results_dir / Path(override_name).name + return PROJECT_ROOT / config.results_dir / "live_results_llm_peft.json" + + +def _runtime_work_dir(config: LlmPeftRuntimeConfig) -> Path: + path = PROJECT_ROOT / config.results_dir / "llm_peft_work" / _run_artifact_id() + path.mkdir(parents=True, exist_ok=True) + return path + + +def _adapter_dir(config: LlmPeftRuntimeConfig) -> Path: + path = PROJECT_ROOT / config.results_dir / "llm_peft_adapters" / _run_artifact_id() + path.mkdir(parents=True, exist_ok=True) + return path + + +def _load_sft_records_with_format_fallback( + path: Path, + *, + preferred_format: str, + file_format: str, +) -> tuple[list[Any], str]: + tried_formats: list[str] = [] + last_error: ValueError | None = None + for data_format in (preferred_format, *SFT_DATA_FORMATS): + if data_format in tried_formats: + continue + tried_formats.append(data_format) + try: + return load_sft_records(path, data_format=data_format, file_format=file_format), data_format + except ValueError as exc: + last_error = exc + formats = ", ".join(tried_formats) + raise ValueError(f"Unable to load SFT records from {path} using formats: {formats}") from last_error + + +def _resolve_runtime_path(path: Path) -> Path: + return path if path.is_absolute() else PROJECT_ROOT / path + + +def _effective_sft_file_format(path: Path, *, configured_file_format: str) -> str: + if configured_file_format != "auto": + return configured_file_format + if path.is_file(): + detected = _sft_file_format_from_suffix(path) + return detected or "auto" + if path.is_dir(): + formats = { + detected + for candidate in path.rglob("*") + if candidate.is_file() + and (detected := _sft_file_format_from_suffix(candidate)) is not None + } + if len(formats) == 1: + return next(iter(formats)) + if len(formats) > 1: + return "mixed" + return "auto" + + +def _sft_file_format_from_suffix(path: Path) -> str | None: + suffix = path.suffix.lower() + if suffix == ".jsonl": + return "jsonl" + if suffix == ".parquet": + return "parquet" + return None + + +def _run_artifact_id() -> str: + override_name = os.environ.get("FIGARO_RESULTS_FILE", "").strip() + if not override_name: + return "manual" + stem = Path(override_name).name + if stem.endswith(".json"): + stem = stem[:-5] + prefix = "live_results_" + if stem.startswith(prefix): + stem = stem[len(prefix):] + sanitized = re.sub(r"[^A-Za-z0-9_.-]+", "_", stem).strip("._-") + return sanitized or "manual" + + +def _select_clients(client_ids: list[int], *, clients_per_round: int, rng: random.Random) -> list[int]: + selected_count = min(clients_per_round, len(client_ids)) + return sorted(rng.sample(client_ids, selected_count)) + + +def _append_client_metrics(metrics: dict[str, Any], *, client_id: int, round_num: int, update) -> None: + client_key = f"client_{client_id}" + clients = metrics.setdefault("client_results", {}) + series = clients.setdefault( + client_key, + {"rounds": [], "train_loss": [], "train_acc": [], "test_loss": [], "test_acc": []}, + ) + train_loss = float(update.metrics.get("train_loss", 0.0)) + series.setdefault("rounds", []).append(int(round_num)) + series["train_loss"].append(train_loss) + series["train_acc"].append(0.0) + series["test_loss"].append(0.0) + series["test_acc"].append(0.0) + + +def _weighted_train_loss(updates) -> float: + total_examples = sum(max(0, int(update.num_examples)) for update in updates) + if total_examples <= 0: + return 0.0 + weighted = 0.0 + for update in updates: + weighted += float(update.metrics.get("train_loss", 0.0)) * max(0, int(update.num_examples)) + return weighted / total_examples diff --git a/apps/backend/runners/runtime_dispatcher.py b/apps/backend/runners/runtime_dispatcher.py new file mode 100644 index 0000000..e92b2d8 --- /dev/null +++ b/apps/backend/runners/runtime_dispatcher.py @@ -0,0 +1,68 @@ +from __future__ import annotations + +from pathlib import Path +from typing import Any + +import yaml + + +CLASSIC_FL_TASK = "classic_fl" +LLM_PEFT_SFT_TASK = "llm_peft_sft" +SUPPORTED_TASK_TYPES = {CLASSIC_FL_TASK, LLM_PEFT_SFT_TASK} +LLM_SIMULATION_ONLY_MESSAGE = "LLM PEFT is currently supported in simulation mode only" + + +def load_runtime_config(config_path: Path) -> dict[str, Any]: + loaded = yaml.safe_load(config_path.read_text(encoding="utf-8")) + if loaded is None: + return {} + if not isinstance(loaded, dict): + raise ValueError("Config must be a YAML/JSON object") + return loaded + + +def get_task_type(config: dict[str, Any]) -> str: + """Return the selected training route, defaulting legacy configs to classic FL.""" + task = config.get("task") + if not isinstance(task, dict): + return CLASSIC_FL_TASK + + task_type = task.get("type", CLASSIC_FL_TASK) + if not isinstance(task_type, str) or not task_type.strip(): + return CLASSIC_FL_TASK + return task_type.strip() + + +def get_runtime_mode(config: dict[str, Any]) -> str: + """Return the runtime mode, defaulting missing legacy configs to simulation.""" + system = config.get("system") + if not isinstance(system, dict): + return "simulation" + mode = system.get("mode", "simulation") + if not isinstance(mode, str) or not mode.strip(): + return "simulation" + return mode.strip() + + +def validate_runtime_route(config: dict[str, Any]) -> None: + if get_task_type(config) == LLM_PEFT_SFT_TASK and get_runtime_mode(config) != "simulation": + raise ValueError(LLM_SIMULATION_ONLY_MESSAGE) + + +def run_runtime(config_path: Path) -> bool: + config = load_runtime_config(config_path) + validate_runtime_route(config) + task_type = get_task_type(config) + + if task_type == CLASSIC_FL_TASK: + from classic_fl_runtime import run_classic_fl_runtime + + return run_classic_fl_runtime(config_path) + + if task_type == LLM_PEFT_SFT_TASK: + from llm_peft_runtime import run_llm_peft_runtime + + return run_llm_peft_runtime(config_path) + + supported = ", ".join(sorted(SUPPORTED_TASK_TYPES)) + raise ValueError(f"Unsupported task.type={task_type!r}. Supported task types: {supported}") diff --git a/apps/backend/tests/api/v1/endpoints/test_agent.py b/apps/backend/tests/api/v1/endpoints/test_agent.py index 44d5b2b..ecd32f9 100644 --- a/apps/backend/tests/api/v1/endpoints/test_agent.py +++ b/apps/backend/tests/api/v1/endpoints/test_agent.py @@ -22,7 +22,41 @@ def test_agent_config_schema_endpoint_returns_ui_metadata(client): assert response.status_code == 200 payload = response.json() - assert payload["model"]["name"]["options"] == ["CNN", "LeNet", "ResNet"] + assert "Auto" in payload["model"]["name"]["options"] + assert "FedAvgCNN" in payload["model"]["name"]["options"] + assert "TextDNN" in payload["model"]["name"]["options"] + assert "CharLSTM" in payload["model"]["name"]["options"] + llm_task_ui = payload["task"]["type"]["ui"]["options"]["llm_peft_sft"] + assert llm_task_ui["badge"] == "simulation" + assert "disabled" not in llm_task_ui + assert payload["llm"]["base_model"]["type"] == "text" + assert "Qwen/Qwen2.5-0.5B-Instruct" not in payload["llm"]["base_model"]["options"] + if payload["llm"]["base_model"]["options"]: + assert payload["llm"]["base_model"]["default"] in payload["llm"]["base_model"]["options"] + else: + assert payload["llm"]["base_model"]["default"] == "" + assert payload["llm"]["base_model"]["ui"]["empty_message"] == "No local models found" + assert payload["llm"]["base_model"]["ui"]["option_source"]["path"] == "./models/llm" + if payload["sft"]["dataset_path"]["options"]: + assert payload["sft"]["dataset_path"]["default"] in payload["sft"]["dataset_path"]["options"] + else: + assert payload["sft"]["dataset_path"]["default"] == "" + assert payload["sft"]["dataset_path"]["ui"]["empty_message"] == "No training datasets found" + assert payload["sft"]["dataset_path"]["ui"]["option_source"]["path"] == "./datasets/llm" + assert payload["sft"]["file_format"]["options"] == ["auto", "jsonl", "parquet"] + assert payload["sft"]["validation_split"]["default"] == 0.0 + assert "alpaca" in payload["sft"]["format"]["options"] + if payload["evaluation"]["dataset_path"]["options"]: + if payload["evaluation"]["dataset_path"]["default"]: + assert payload["evaluation"]["dataset_path"]["default"] in payload["evaluation"]["dataset_path"]["options"] + assert payload["evaluation"]["enable"]["default"] is True + else: + assert payload["evaluation"]["enable"]["default"] is False + else: + assert payload["evaluation"]["dataset_path"]["default"] == "" + assert payload["evaluation"]["dataset_path"]["ui"]["empty_message"] == "No evaluation datasets found" + assert payload["evaluation"]["enable"]["default"] is False + assert payload["evaluation"]["dataset_path"]["ui"]["option_source"]["path"] == "./datasets/llm" aggregation_ui = payload["federated"]["aggregation"]["ui"] assert aggregation_ui["featured"] is True assert aggregation_ui["options"]["fedprox"]["disabled"] is True @@ -367,7 +401,7 @@ class _FakeHistoryService: def __init__(self, _session): self._session = _session - async def list_jobs(self): + async def list_jobs(self, **_filters): return [history_job] async def get_job_or_raise(self, optimization_job_id): @@ -390,3 +424,104 @@ async def get_job_or_raise(self, optimization_job_id): assert detail_payload["optimization_job_id"] == 7 assert detail_payload["job_name"] == "cifar10-alpha-sweep" assert detail_payload["resolved_objective"] == "accuracy" + + +def test_agent_optimization_jobs_history_filters_are_forwarded(client, monkeypatch): + import app.api.v1.endpoints.agent as agent_module + + captured = {} + + class _FakeHistoryService: + def __init__(self, _session): + self._session = _session + + async def list_jobs(self, **filters): + captured.update(filters) + return [] + + monkeypatch.setattr(agent_module, "AgentOptimizationHistoryService", _FakeHistoryService) + + response = client.get( + "/api/v1/agent/optimization-jobs" + "?status=completed" + "&q=cifar" + "&model_name=gpt-test" + "&objective=accuracy" + "&best_score_min=0.8" + "&best_score_max=0.95" + "&dataset=CIFAR-10" + "&config_model=FedAvgCNN" + "&aggregation=fedavg" + "&num_clients=3" + "&num_rounds=10" + ) + + assert response.status_code == 200 + assert captured["status"] == "completed" + assert captured["q"] == "cifar" + assert captured["model_name"] == "gpt-test" + assert captured["objective"] == "accuracy" + assert captured["best_score_min"] == 0.8 + assert captured["best_score_max"] == 0.95 + assert captured["config_filters"] == { + "dataset.name": "CIFAR-10", + "model.name": "FedAvgCNN", + "federated.aggregation": "fedavg", + "federated.num_clients": 3, + "federated.num_rounds": 10, + } + + +def test_agent_config_version_endpoints(client, monkeypatch): + import app.api.v1.endpoints.agent as agent_module + + now = datetime.now(timezone.utc) + version = SimpleNamespace( + id=3, + optimization_job_id=7, + run_id="run-1", + iteration=1, + source="experiment", + label="alpha 0.1", + config_hash="abc123", + config_json={"dataset": {"alpha": 0.1}}, + diff_json=[ + { + "path": "dataset.alpha", + "change_type": "updated", + "old_value": 0.5, + "new_value": 0.1, + } + ], + created_at=now, + ) + + class _FakeHistoryService: + def __init__(self, _session): + self._session = _session + + async def list_config_versions(self, optimization_job_id): + assert optimization_job_id == 7 + return [version] + + async def diff_config_versions(self, *, optimization_job_id, from_version_id, to_version_id): + assert optimization_job_id == 7 + assert from_version_id is None + assert to_version_id == 3 + return version.diff_json + + monkeypatch.setattr(agent_module, "AgentOptimizationHistoryService", _FakeHistoryService) + + versions_response = client.get("/api/v1/agent/optimization-jobs/7/config-versions") + assert versions_response.status_code == 200 + versions_payload = versions_response.json() + assert versions_payload[0]["id"] == 3 + assert versions_payload[0]["label"] == "alpha 0.1" + assert versions_payload[0]["diff_json"][0]["path"] == "dataset.alpha" + + diff_response = client.get("/api/v1/agent/optimization-jobs/7/config-diff?to_version_id=3") + assert diff_response.status_code == 200 + diff_payload = diff_response.json() + assert diff_payload["optimization_job_id"] == 7 + assert diff_payload["to_version_id"] == 3 + assert diff_payload["changes"][0]["new_value"] == 0.1 diff --git a/apps/backend/tests/llm/test_llm_core.py b/apps/backend/tests/llm/test_llm_core.py new file mode 100644 index 0000000..e41c508 --- /dev/null +++ b/apps/backend/tests/llm/test_llm_core.py @@ -0,0 +1,295 @@ +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import pytest +import torch + +PROJECT_ROOT = Path(__file__).resolve().parents[4] +LIBS_DIR = PROJECT_ROOT / "libs" +if str(LIBS_DIR) not in sys.path: + sys.path.insert(0, str(LIBS_DIR)) + +from fl_core.llm.aggregation import AdapterClientUpdate, aggregate_adapter_state_dicts +from fl_core.llm.artifacts import ( + adapter_artifact_record, + load_adapter_artifact, + save_adapter_artifact, + sha256_file, +) +from fl_core.llm.config import normalize_llm_peft_config, parse_target_modules +from fl_core.llm.data import SftRecord, format_sft_record_text, load_jsonl_sft_records, load_sft_records, split_records_by_client +from fl_core.llm.metrics import append_llm_round_metrics, empty_llm_metrics_payload +from fl_core.llm import trainer as llm_trainer +from fl_core.llm.trainer import TokenizedSftDataset + + +def _base_config(tmp_path: Path) -> dict: + dataset_path = tmp_path / "train.jsonl" + dataset_path.write_text( + "\n".join( + [ + json.dumps({"prompt": "Q1", "completion": "A1"}), + json.dumps({"prompt": "Q2", "completion": "A2"}), + ] + ), + encoding="utf-8", + ) + return { + "task": {"type": "llm_peft_sft"}, + "llm": { + "base_model": "Qwen/Qwen2.5-0.5B-Instruct", + "tokenizer": "auto", + "max_seq_length": 512, + "precision": "bf16", + }, + "sft": { + "dataset_path": str(dataset_path), + "format": "prompt_completion", + "prompt_template": "plain", + }, + "peft": { + "method": "lora", + "rank": 8, + "alpha": 16, + "dropout": 0.05, + "target_modules": "q_proj, v_proj", + "quantization": "none", + }, + "federated": { + "num_clients": 2, + "num_rounds": 1, + "clients_per_round": 1, + "local_epochs": 1, + "learning_rate": 0.0002, + "aggregation": "fedavg", + "seed": 7, + }, + } + + +def test_normalize_llm_peft_config_parses_nested_sections(tmp_path): + resume_path = tmp_path / "adapter.pt" + config = _base_config(tmp_path) + config["peft"]["resume_adapter_path"] = str(resume_path) + + normalized = normalize_llm_peft_config(config) + + assert normalized.task_type == "llm_peft_sft" + assert normalized.llm.base_model == "Qwen/Qwen2.5-0.5B-Instruct" + assert normalized.sft.dataset_path.name == "train.jsonl" + assert normalized.sft.per_device_train_batch_size == 1 + assert normalized.sft.gradient_accumulation_steps == 1 + assert normalized.sft.file_format == "auto" + assert normalized.sft.validation_split == 0.0 + assert normalized.peft.target_modules == ("q_proj", "v_proj") + assert normalized.peft.resume_adapter_path == resume_path + assert normalized.federated.num_clients == 2 + assert normalized.evaluation.enabled is False + assert normalized.evaluation.batch_size == 1 + + +def test_parse_target_modules_rejects_empty_values(): + with pytest.raises(ValueError, match="target_modules"): + parse_target_modules(" , ") + + +def test_normalize_llm_peft_config_parses_evaluation_section(tmp_path): + evaluation_path = tmp_path / "validation.jsonl" + evaluation_path.write_text(json.dumps({"prompt": "Q", "completion": "A"}), encoding="utf-8") + config = _base_config(tmp_path) + config["evaluation"] = { + "enable": True, + "dataset_path": str(evaluation_path), + "batch_size": 2, + "max_samples": 8, + } + + normalized = normalize_llm_peft_config(config) + + assert normalized.evaluation.enabled is True + assert normalized.evaluation.dataset_path == evaluation_path + assert normalized.evaluation.batch_size == 2 + assert normalized.evaluation.max_samples == 8 + + +def test_load_jsonl_sft_records_prompt_completion_and_split(tmp_path): + config = normalize_llm_peft_config(_base_config(tmp_path)) + records = load_jsonl_sft_records(config.sft.dataset_path, data_format=config.sft.format) + splits = split_records_by_client(records, num_clients=2, seed=123) + + assert len(records) == 2 + assert sorted(len(split) for split in splits) == [1, 1] + assert format_sft_record_text(records[0], data_format="prompt_completion", prompt_template="plain") == "Q1A1" + + +def test_load_jsonl_sft_records_messages_format(tmp_path): + dataset_path = tmp_path / "messages.jsonl" + dataset_path.write_text( + json.dumps( + { + "messages": [ + {"role": "user", "content": "Hello"}, + {"role": "assistant", "content": "Hi"}, + ] + } + ), + encoding="utf-8", + ) + + [record] = load_jsonl_sft_records(dataset_path, data_format="messages") + + assert record.messages[0]["role"] == "user" + assert "<|im_start|>user" in format_sft_record_text(record, data_format="messages", prompt_template="chatml") + + +def test_load_sft_records_jsonl_still_works(tmp_path): + dataset_path = tmp_path / "train.jsonl" + dataset_path.write_text(json.dumps({"prompt": "Q", "completion": "A"}), encoding="utf-8") + + [record] = load_sft_records(dataset_path, data_format="prompt_completion", file_format="auto") + + assert record.prompt == "Q" + assert record.completion == "A" + + +def test_load_alpaca_parquet_sft_records(tmp_path): + import pyarrow as pa + import pyarrow.parquet as pq + + dataset_dir = tmp_path / "alpaca" + data_dir = dataset_dir / "data" + data_dir.mkdir(parents=True) + parquet_path = data_dir / "train-00000-of-00001-a09b74b3ef9c3b56.parquet" + table = pa.Table.from_pylist( + [ + { + "instruction": "Summarize the note", + "input": "The run finished with lower loss.", + "output": "The run improved loss.", + }, + { + "instruction": "Say hello", + "input": "", + "output": "Hello.", + }, + ] + ) + pq.write_table(table, parquet_path) + + records = load_sft_records(dataset_dir, data_format="alpaca", file_format="auto") + + assert len(records) == 2 + assert records[0].prompt == ( + "Instruction: Summarize the note\n" + "Input: The run finished with lower loss.\n" + "Answer:" + ) + assert records[0].completion == " The run improved loss." + assert records[1].prompt == "Instruction: Say hello\nAnswer:" + assert records[1].completion == " Hello." + assert format_sft_record_text(records[0], data_format="alpaca", prompt_template="plain").endswith( + " The run improved loss." + ) + + +def test_aggregate_adapter_state_dicts_weighted_by_examples(): + update_a = AdapterClientUpdate( + adapter_state={"lora_A": torch.tensor([1.0, 3.0])}, + num_examples=1, + ) + update_b = AdapterClientUpdate( + adapter_state={"lora_A": torch.tensor([3.0, 5.0])}, + num_examples=3, + ) + + aggregated = aggregate_adapter_state_dicts([update_a, update_b]) + + assert torch.allclose(aggregated["lora_A"], torch.tensor([2.5, 4.5])) + + +def test_aggregate_adapter_state_dicts_rejects_shape_mismatch(): + update_a = AdapterClientUpdate(adapter_state={"lora_A": torch.tensor([1.0])}, num_examples=1) + update_b = AdapterClientUpdate(adapter_state={"lora_A": torch.tensor([[1.0]])}, num_examples=1) + + with pytest.raises(ValueError, match="mismatched shape"): + aggregate_adapter_state_dicts([update_a, update_b]) + + +def test_adapter_artifact_round_trip_and_lineage_record(tmp_path): + artifact_path = save_adapter_artifact( + tmp_path / "adapter.pt", + {"lora_A": torch.tensor([1.0, 2.0])}, + metadata={"round": 1}, + ) + + adapter_state, metadata = load_adapter_artifact(artifact_path) + record = adapter_artifact_record( + artifact_path, + round_num=1, + size_bytes=8, + selected_clients=[0, 1], + parent_path="previous.pt", + parent_sha256="abc", + ) + + assert torch.allclose(adapter_state["lora_A"], torch.tensor([1.0, 2.0])) + assert metadata["round"] == 1 + assert record["sha256"] == sha256_file(artifact_path) + assert record["parent_sha256"] == "abc" + + +def test_empty_llm_metrics_payload_and_round_append(tmp_path): + config = normalize_llm_peft_config(_base_config(tmp_path)) + payload = empty_llm_metrics_payload(config) + append_llm_round_metrics(payload, round_num=1, train_loss=2.0, validation_loss=1.0, token_throughput=12.5) + + assert payload["experiment_info"]["basic"]["task_type"] == "llm_peft_sft" + assert payload["llm_results"]["rounds"] == [1] + assert payload["llm_artifacts"] == [] + assert payload["llm_evaluation"] == {} + assert payload["llm_results"]["perplexity"][0] == pytest.approx(2.71828, rel=1e-4) + assert payload["global_results"]["global_loss"] == [1.0] + + +def test_llm_round_append_without_validation_keeps_validation_series_empty(tmp_path): + config = normalize_llm_peft_config(_base_config(tmp_path)) + payload = empty_llm_metrics_payload(config) + append_llm_round_metrics(payload, round_num=1, train_loss=2.0, token_throughput=12.5) + + assert payload["llm_results"]["rounds"] == [1] + assert payload["llm_results"]["train_loss"] == [2.0] + assert payload["llm_results"]["validation_loss"] == [] + assert payload["llm_results"]["perplexity"][0] == pytest.approx(7.38905, rel=1e-4) + assert payload["global_results"]["global_loss"] == [2.0] + + +def test_tokenized_sft_dataset_uses_rendered_text(): + class FakeTokenizer: + def __call__(self, text, *, truncation, max_length, padding): + assert truncation is True + assert padding is False + input_ids = [ord(ch) for ch in text[:max_length]] + return {"input_ids": input_ids, "attention_mask": [1] * len(input_ids)} + + dataset = TokenizedSftDataset( + [SftRecord(prompt="abc", completion="def")], + tokenizer=FakeTokenizer(), + data_format="prompt_completion", + prompt_template="plain", + max_seq_length=4, + ) + + item = dataset[0] + assert item["input_ids"] == [97, 98, 99, 100] + assert item["labels"] == [-100, -100, -100, 100] + assert dataset.num_tokens == 4 + + +def test_llm_peft_progress_callback_handles_trainer_lifecycle_methods(): + callback = llm_trainer._LlmPeftProgressCallback(round_num=1, client_id=0, total_steps=100) + + assert callback.on_init_end(None, None, None) is None + assert callback.on_train_begin(None, None, None) is None diff --git a/apps/backend/tests/runners/test_llm_peft_runtime.py b/apps/backend/tests/runners/test_llm_peft_runtime.py new file mode 100644 index 0000000..df6e4dd --- /dev/null +++ b/apps/backend/tests/runners/test_llm_peft_runtime.py @@ -0,0 +1,416 @@ +from __future__ import annotations + +import json +import sys +from pathlib import Path + +import yaml +import torch + +RUNNERS_DIR = Path(__file__).resolve().parents[2] / "runners" +if str(RUNNERS_DIR) not in sys.path: + sys.path.insert(0, str(RUNNERS_DIR)) + +import llm_peft_runtime # noqa: E402 +from fl_core.llm.aggregation import AdapterClientUpdate # noqa: E402 +from fl_core.llm.artifacts import save_adapter_artifact, sha256_file # noqa: E402 + + +def test_llm_peft_runtime_writes_blocked_metrics_for_missing_dependencies(tmp_path, monkeypatch): + dataset_path = tmp_path / "train.jsonl" + dataset_path.write_text( + "\n".join( + [ + json.dumps({"prompt": "hello", "completion": "world"}), + json.dumps({"prompt": "foo", "completion": "bar"}), + ] + ), + encoding="utf-8", + ) + config_path = tmp_path / "config.yaml" + config_path.write_text( + yaml.safe_dump( + { + "task": {"type": "llm_peft_sft"}, + "llm": {"base_model": "tiny-model", "tokenizer": "auto", "max_seq_length": 128, "precision": "fp32"}, + "sft": {"dataset_path": str(dataset_path), "format": "prompt_completion", "prompt_template": "plain"}, + "peft": { + "method": "lora", + "rank": 4, + "alpha": 8, + "dropout": 0.0, + "target_modules": "q_proj", + "quantization": "none", + }, + "federated": { + "num_clients": 2, + "num_rounds": 1, + "clients_per_round": 1, + "local_epochs": 1, + "learning_rate": 0.0002, + "aggregation": "fedavg", + "seed": 1, + }, + "logging": {"results_dir": "results"}, + }, + sort_keys=False, + ), + encoding="utf-8", + ) + monkeypatch.setattr(llm_peft_runtime, "PROJECT_ROOT", tmp_path) + monkeypatch.setenv("FIGARO_RESULTS_FILE", "live_results_test.json") + monkeypatch.setattr(llm_peft_runtime, "missing_llm_runtime_dependencies", lambda: ["transformers", "peft"]) + + assert llm_peft_runtime.run_llm_peft_runtime(config_path) is False + + metrics_path = tmp_path / "results" / "live_results_test.json" + metrics = json.loads(metrics_path.read_text(encoding="utf-8")) + assert metrics["llm_dataset"]["num_records"] == 2 + assert metrics["llm_dataset"]["format"] == "prompt_completion" + assert metrics["llm_dataset"]["file_format"] == "jsonl" + assert metrics["llm_dataset"]["client_record_counts"] == [1, 1] + assert metrics["llm_runtime"]["reason"] == "missing_llm_runtime_dependencies" + assert metrics["llm_runtime"]["missing_dependencies"] == ["transformers", "peft"] + + +def test_llm_peft_runtime_runs_client_training_and_aggregates_adapters(tmp_path, monkeypatch): + dataset_path = tmp_path / "train.jsonl" + dataset_path.write_text( + "\n".join( + [ + json.dumps({"prompt": "p0", "completion": "c0"}), + json.dumps({"prompt": "p1", "completion": "c1"}), + ] + ), + encoding="utf-8", + ) + config_path = tmp_path / "config.yaml" + config_path.write_text( + yaml.safe_dump( + { + "task": {"type": "llm_peft_sft"}, + "llm": {"base_model": "tiny-model", "tokenizer": "auto", "max_seq_length": 128, "precision": "fp32"}, + "sft": { + "dataset_path": str(dataset_path), + "format": "prompt_completion", + "prompt_template": "plain", + "per_device_train_batch_size": 1, + "gradient_accumulation_steps": 1, + }, + "peft": { + "method": "lora", + "rank": 4, + "alpha": 8, + "dropout": 0.0, + "target_modules": "q_proj", + "quantization": "none", + }, + "federated": { + "num_clients": 2, + "num_rounds": 1, + "clients_per_round": 2, + "local_epochs": 1, + "learning_rate": 0.0002, + "aggregation": "fedavg", + "seed": 1, + }, + "logging": {"results_dir": "results"}, + }, + sort_keys=False, + ), + encoding="utf-8", + ) + + class FakeTrainer: + def __init__(self, config): + self.config = config + + def train_client(self, *, client_id, records, round_num, initial_adapter_state=None, output_dir=None): + assert round_num == 1 + assert initial_adapter_state is None + assert output_dir is not None + value = float(client_id + 1) + return AdapterClientUpdate( + adapter_state={"lora_A": torch.tensor([value])}, + num_examples=len(records), + num_tokens=10, + metrics={"train_loss": value}, + ) + + monkeypatch.setattr(llm_peft_runtime, "PROJECT_ROOT", tmp_path) + monkeypatch.setenv("FIGARO_RESULTS_FILE", "live_results_test.json") + monkeypatch.setattr(llm_peft_runtime, "missing_llm_runtime_dependencies", lambda: []) + monkeypatch.setattr(llm_peft_runtime, "LlmPeftTrainer", FakeTrainer) + + assert llm_peft_runtime.run_llm_peft_runtime(config_path) is True + + metrics_path = tmp_path / "results" / "live_results_test.json" + metrics = json.loads(metrics_path.read_text(encoding="utf-8")) + assert metrics["llm_results"]["rounds"] == [1] + assert metrics["llm_results"]["train_loss"] == [1.5] + assert metrics["llm_results"]["validation_loss"] == [] + assert metrics["client_results"]["client_0"]["rounds"] == [1] + assert metrics["client_results"]["client_1"]["rounds"] == [1] + assert metrics["llm_runtime"]["status"] == "completed" + assert len(metrics["llm_artifacts"]) == 1 + assert metrics["llm_artifacts"][0]["round"] == 1 + assert metrics["llm_artifacts"][0]["selected_clients"] == [0, 1] + adapter_path = Path(metrics["llm_runtime"]["latest_adapter_path"]) + assert adapter_path.exists() + assert metrics["llm_runtime"]["latest_adapter_sha256"] == sha256_file(adapter_path) + + +def test_llm_peft_runtime_evaluates_global_adapter_when_enabled(tmp_path, monkeypatch): + dataset_path = tmp_path / "train.jsonl" + dataset_path.write_text( + "\n".join( + [ + json.dumps({"prompt": "p0", "completion": "c0"}), + json.dumps({"prompt": "p1", "completion": "c1"}), + ] + ), + encoding="utf-8", + ) + evaluation_path = tmp_path / "validation.jsonl" + evaluation_path.write_text( + "\n".join( + [ + json.dumps({"prompt": "v0", "completion": "a0"}), + json.dumps({"prompt": "v1", "completion": "a1"}), + ] + ), + encoding="utf-8", + ) + config_path = tmp_path / "config.yaml" + config_path.write_text( + yaml.safe_dump( + { + "task": {"type": "llm_peft_sft"}, + "llm": {"base_model": "tiny-model", "tokenizer": "auto", "max_seq_length": 128, "precision": "fp32"}, + "sft": { + "dataset_path": str(dataset_path), + "format": "prompt_completion", + "prompt_template": "plain", + }, + "peft": { + "method": "lora", + "rank": 4, + "alpha": 8, + "dropout": 0.0, + "target_modules": "q_proj", + "quantization": "none", + }, + "evaluation": { + "enable": True, + "dataset_path": str(evaluation_path), + "batch_size": 1, + "max_samples": 1, + }, + "federated": { + "num_clients": 2, + "num_rounds": 1, + "clients_per_round": 2, + "local_epochs": 1, + "learning_rate": 0.0002, + "aggregation": "fedavg", + "seed": 1, + }, + "logging": {"results_dir": "results"}, + }, + sort_keys=False, + ), + encoding="utf-8", + ) + + class FakeTrainer: + def __init__(self, config): + self.config = config + + def train_client(self, *, client_id, records, round_num, initial_adapter_state=None, output_dir=None): + return AdapterClientUpdate( + adapter_state={"lora_A": torch.tensor([float(client_id + 1)])}, + num_examples=len(records), + num_tokens=10, + metrics={"train_loss": float(client_id + 1)}, + ) + + def evaluate_adapter(self, *, records, adapter_state, output_dir=None): + assert len(records) == 1 + assert adapter_state is not None + assert torch.allclose(adapter_state["lora_A"], torch.tensor([1.5])) + assert output_dir is not None + return {"validation_loss": 0.25, "num_examples": len(records), "num_tokens": 4} + + monkeypatch.setattr(llm_peft_runtime, "PROJECT_ROOT", tmp_path) + monkeypatch.setenv("FIGARO_RESULTS_FILE", "live_results_eval.json") + monkeypatch.setattr(llm_peft_runtime, "missing_llm_runtime_dependencies", lambda: []) + monkeypatch.setattr(llm_peft_runtime, "LlmPeftTrainer", FakeTrainer) + + assert llm_peft_runtime.run_llm_peft_runtime(config_path) is True + + metrics = json.loads((tmp_path / "results" / "live_results_eval.json").read_text(encoding="utf-8")) + assert metrics["llm_evaluation"]["enabled"] is True + assert metrics["llm_evaluation"]["num_records"] == 1 + assert metrics["llm_evaluation"]["last_validation_loss"] == 0.25 + assert metrics["llm_results"]["validation_loss"] == [0.25] + assert metrics["global_results"]["global_loss"] == [0.25] + + +def test_llm_peft_runtime_allows_prompt_completion_eval_for_alpaca_training(tmp_path, monkeypatch): + dataset_path = tmp_path / "alpaca.jsonl" + dataset_path.write_text( + "\n".join( + [ + json.dumps({"instruction": "Do one thing", "input": "", "output": "done"}), + json.dumps({"instruction": "Do another thing", "input": "x", "output": "done x"}), + ] + ), + encoding="utf-8", + ) + evaluation_path = tmp_path / "validation.jsonl" + evaluation_path.write_text( + json.dumps({"prompt": "v0", "completion": "a0"}), + encoding="utf-8", + ) + config_path = tmp_path / "config.yaml" + config_path.write_text( + yaml.safe_dump( + { + "task": {"type": "llm_peft_sft"}, + "llm": {"base_model": "tiny-model", "tokenizer": "auto", "max_seq_length": 128, "precision": "fp32"}, + "sft": { + "dataset_path": str(dataset_path), + "format": "alpaca", + "file_format": "auto", + "prompt_template": "plain", + }, + "peft": { + "method": "lora", + "rank": 4, + "alpha": 8, + "dropout": 0.0, + "target_modules": "q_proj", + "quantization": "none", + }, + "evaluation": { + "enable": True, + "dataset_path": str(evaluation_path), + "batch_size": 1, + "max_samples": 1, + }, + "federated": { + "num_clients": 2, + "num_rounds": 1, + "clients_per_round": 2, + "local_epochs": 1, + "learning_rate": 0.0002, + "aggregation": "fedavg", + "seed": 1, + }, + "logging": {"results_dir": "results"}, + }, + sort_keys=False, + ), + encoding="utf-8", + ) + + class FakeTrainer: + def __init__(self, config): + self.config = config + + def train_client(self, *, client_id, records, round_num, initial_adapter_state=None, output_dir=None): + assert records[0].prompt.startswith("Instruction:") + return AdapterClientUpdate( + adapter_state={"lora_A": torch.tensor([float(client_id + 1)])}, + num_examples=len(records), + num_tokens=10, + metrics={"train_loss": float(client_id + 1)}, + ) + + def evaluate_adapter(self, *, records, adapter_state, output_dir=None): + assert records[0].prompt == "v0" + return {"validation_loss": 0.5, "num_examples": len(records), "num_tokens": 4} + + monkeypatch.setattr(llm_peft_runtime, "PROJECT_ROOT", tmp_path) + monkeypatch.setenv("FIGARO_RESULTS_FILE", "live_results_eval_mixed.json") + monkeypatch.setattr(llm_peft_runtime, "missing_llm_runtime_dependencies", lambda: []) + monkeypatch.setattr(llm_peft_runtime, "LlmPeftTrainer", FakeTrainer) + + assert llm_peft_runtime.run_llm_peft_runtime(config_path) is True + + metrics = json.loads((tmp_path / "results" / "live_results_eval_mixed.json").read_text(encoding="utf-8")) + assert metrics["llm_dataset"]["format"] == "alpaca" + assert metrics["llm_evaluation"]["format"] == "prompt_completion" + assert metrics["llm_results"]["validation_loss"] == [0.5] + + +def test_llm_peft_runtime_resumes_from_adapter_artifact(tmp_path, monkeypatch): + dataset_path = tmp_path / "train.jsonl" + dataset_path.write_text(json.dumps({"prompt": "p0", "completion": "c0"}), encoding="utf-8") + resume_path = save_adapter_artifact( + tmp_path / "resume_adapter.pt", + {"lora_A": torch.tensor([9.0])}, + metadata={"round": 7}, + ) + config_path = tmp_path / "config.yaml" + config_path.write_text( + yaml.safe_dump( + { + "task": {"type": "llm_peft_sft"}, + "llm": {"base_model": "tiny-model", "tokenizer": "auto", "max_seq_length": 128, "precision": "fp32"}, + "sft": { + "dataset_path": str(dataset_path), + "format": "prompt_completion", + "prompt_template": "plain", + }, + "peft": { + "method": "lora", + "rank": 4, + "alpha": 8, + "dropout": 0.0, + "target_modules": "q_proj", + "resume_adapter_path": str(resume_path), + "quantization": "none", + }, + "federated": { + "num_clients": 1, + "num_rounds": 1, + "clients_per_round": 1, + "local_epochs": 1, + "learning_rate": 0.0002, + "aggregation": "fedavg", + "seed": 1, + }, + "logging": {"results_dir": "results"}, + }, + sort_keys=False, + ), + encoding="utf-8", + ) + + class FakeTrainer: + def __init__(self, config): + self.config = config + + def train_client(self, *, client_id, records, round_num, initial_adapter_state=None, output_dir=None): + assert initial_adapter_state is not None + assert torch.allclose(initial_adapter_state["lora_A"], torch.tensor([9.0])) + return AdapterClientUpdate( + adapter_state={"lora_A": torch.tensor([10.0])}, + num_examples=len(records), + num_tokens=5, + metrics={"train_loss": 0.5}, + ) + + monkeypatch.setattr(llm_peft_runtime, "PROJECT_ROOT", tmp_path) + monkeypatch.setenv("FIGARO_RESULTS_FILE", "live_results_resume.json") + monkeypatch.setattr(llm_peft_runtime, "missing_llm_runtime_dependencies", lambda: []) + monkeypatch.setattr(llm_peft_runtime, "LlmPeftTrainer", FakeTrainer) + + assert llm_peft_runtime.run_llm_peft_runtime(config_path) is True + + metrics = json.loads((tmp_path / "results" / "live_results_resume.json").read_text(encoding="utf-8")) + artifact = metrics["llm_artifacts"][0] + assert artifact["parent_path"] == str(resume_path) + assert artifact["parent_sha256"] == sha256_file(resume_path) + assert metrics["llm_runtime"]["latest_adapter_sha256"] == artifact["sha256"] diff --git a/apps/backend/tests/runners/test_runtime_dispatcher.py b/apps/backend/tests/runners/test_runtime_dispatcher.py new file mode 100644 index 0000000..3239be5 --- /dev/null +++ b/apps/backend/tests/runners/test_runtime_dispatcher.py @@ -0,0 +1,82 @@ +from __future__ import annotations + +import sys +import types +from pathlib import Path + +import pytest +import yaml + +RUNNERS_DIR = Path(__file__).resolve().parents[2] / "runners" +if str(RUNNERS_DIR) not in sys.path: + sys.path.insert(0, str(RUNNERS_DIR)) + +from runtime_dispatcher import get_runtime_mode, get_task_type, run_runtime # noqa: E402 + + +def _write_config(tmp_path: Path, config: dict) -> Path: + config_path = tmp_path / "config.yaml" + config_path.write_text(yaml.safe_dump(config, sort_keys=False), encoding="utf-8") + return config_path + + +def test_get_task_type_defaults_legacy_configs_to_classic_fl(): + assert get_task_type({}) == "classic_fl" + assert get_task_type({"task": {}}) == "classic_fl" + + +def test_get_runtime_mode_defaults_legacy_configs_to_simulation(): + assert get_runtime_mode({}) == "simulation" + assert get_runtime_mode({"system": {}}) == "simulation" + + +def test_run_runtime_dispatches_classic_route(tmp_path, monkeypatch): + called = {} + + def fake_run(path: Path) -> bool: + called["path"] = path + return True + + fake_module = types.SimpleNamespace(run_classic_fl_runtime=fake_run) + monkeypatch.setitem(sys.modules, "classic_fl_runtime", fake_module) + + config_path = _write_config(tmp_path, {"task": {"type": "classic_fl"}}) + + assert run_runtime(config_path) is True + assert called["path"] == config_path + + +def test_run_runtime_dispatches_llm_peft_route(tmp_path, monkeypatch): + called = {} + + def fake_run(path: Path) -> bool: + called["path"] = path + return False + + fake_module = types.SimpleNamespace(run_llm_peft_runtime=fake_run) + monkeypatch.setitem(sys.modules, "llm_peft_runtime", fake_module) + + config_path = _write_config(tmp_path, {"task": {"type": "llm_peft_sft"}}) + + assert run_runtime(config_path) is False + assert called["path"] == config_path + + +def test_run_runtime_rejects_llm_peft_distributed_route(tmp_path): + config_path = _write_config( + tmp_path, + { + "task": {"type": "llm_peft_sft"}, + "system": {"mode": "distributed"}, + }, + ) + + with pytest.raises(ValueError, match="simulation mode only"): + run_runtime(config_path) + + +def test_run_runtime_rejects_unknown_task_type(tmp_path): + config_path = _write_config(tmp_path, {"task": {"type": "unknown"}}) + + with pytest.raises(ValueError, match="Unsupported task.type"): + run_runtime(config_path) diff --git a/apps/backend/tests/schemas/test_contracts.py b/apps/backend/tests/schemas/test_contracts.py index 488a136..27dbc57 100644 --- a/apps/backend/tests/schemas/test_contracts.py +++ b/apps/backend/tests/schemas/test_contracts.py @@ -85,6 +85,10 @@ def test_metrics_and_progress_schema_defaults_and_parsing(): assert metrics.global_results.rounds == [] assert metrics.global_results.global_loss == [] assert metrics.global_results.global_accuracy == [] + assert metrics.llm_results.rounds == [] + assert metrics.llm_dataset == {} + assert metrics.llm_evaluation == {} + assert metrics.llm_artifacts == [] assert metrics.client_results == {} parsed = SimulationRunMetricsResponse.model_validate( diff --git a/apps/backend/tests/services/agent/test_experiment_run_artifacts.py b/apps/backend/tests/services/agent/test_experiment_run_artifacts.py new file mode 100644 index 0000000..8522b76 --- /dev/null +++ b/apps/backend/tests/services/agent/test_experiment_run_artifacts.py @@ -0,0 +1,20 @@ +from app.services.agent.experiment_service import AgentExperimentRunService + + +class _DummySession: + pass + + +def test_agent_experiment_run_artifact_paths_share_timestamp(tmp_path): + service = AgentExperimentRunService(session=_DummySession()) # type: ignore[arg-type] + service._job_config_dir = lambda: tmp_path / "configs" # type: ignore[method-assign] + service._results_dir = lambda: tmp_path / "results" # type: ignore[method-assign] + service._runtime_log_dir = lambda: tmp_path / "logs" # type: ignore[method-assign] + timestamp = "20260603_151122Z" + run_id = "abc12345" + + assert service._run_config_path(run_id, artifact_timestamp=timestamp).name == f"{timestamp}_{run_id}.json" + assert service._build_live_results_filename(run_id, artifact_timestamp=timestamp) == ( + f"{timestamp}_{run_id}_live_results.json" + ) + assert service._run_log_path(run_id, artifact_timestamp=timestamp).name == f"{timestamp}_{run_id}_server.log" diff --git a/apps/backend/tests/services/agent/test_history_service.py b/apps/backend/tests/services/agent/test_history_service.py new file mode 100644 index 0000000..5a86db6 --- /dev/null +++ b/apps/backend/tests/services/agent/test_history_service.py @@ -0,0 +1,19 @@ +from app.services.agent.history_service import AgentOptimizationHistoryService + + +def test_config_version_candidates_only_include_experiments(): + snapshot = { + "draft_experiments": [ + {"name": "draft alpha", "config_patch": {"dataset": {"alpha": 0.2}}}, + ], + "experiments": [ + {"run_id": "run-1", "iteration": 1, "name": "alpha_0.1", "config": {"dataset": {"alpha": 0.1}}}, + {"run_id": "run-2", "iteration": 2, "name": "alpha_0.3", "config": {"dataset": {"alpha": 0.3}}}, + ], + "best_config": {"dataset": {"alpha": 0.3}}, + } + + candidates = AgentOptimizationHistoryService._build_config_version_candidates(snapshot) + + assert [candidate["source"] for candidate in candidates] == ["experiment", "experiment"] + assert [candidate["label"] for candidate in candidates] == ["alpha_0.1", "alpha_0.3"] diff --git a/apps/backend/tests/services/agent/test_planning.py b/apps/backend/tests/services/agent/test_planning.py index 0fa66db..9470397 100644 --- a/apps/backend/tests/services/agent/test_planning.py +++ b/apps/backend/tests/services/agent/test_planning.py @@ -3,6 +3,10 @@ build_initial_config, build_schema_prompt_context, collect_disabled_option_errors, + deep_merge_config, + infer_sft_settings_for_dataset_path, + lock_structured_constraints, + merge_llm_resource_constraints, select_llm_model, ) @@ -43,12 +47,70 @@ def test_build_initial_config_uses_schema_defaults(): def test_build_initial_config_hardcoded_defaults_on_empty_schema(): cfg = build_initial_config({}) + assert cfg["task"]["type"] == "classic_fl" assert cfg["dataset"]["name"] == "CIFAR-10" - assert cfg["model"]["name"] == "CNN" + assert cfg["model"]["name"] == "Auto" assert cfg["federated"]["num_clients"] == 3 assert cfg["federated"]["num_rounds"] == 10 +def test_lock_structured_constraints_override_llm_patch_values(): + constrained_base = { + "dataset": {"name": "FEMNIST", "alpha": 0.5}, + "model": {"name": "LeNet"}, + "federated": {"aggregation": "fedavg"}, + } + llm_patch = { + "dataset": {"name": "CIFAR-10", "alpha": 0.1}, + "model": {"name": "Auto"}, + } + constraints = {"dataset": {"name": "FEMNIST"}, "model": {"name": "LeNet"}} + + merged = deep_merge_config(constrained_base, llm_patch) + locked = lock_structured_constraints(merged, constraints) + + assert locked["dataset"]["name"] == "FEMNIST" + assert locked["dataset"]["alpha"] == 0.1 + assert locked["model"]["name"] == "LeNet" + + +def test_llm_resource_constraints_lock_effective_dataset_defaults(): + constrained_base = { + "task": {"type": "llm_peft_sft"}, + "llm": {"base_model": "./models/llm/tiny"}, + "sft": {"dataset_path": "./datasets/llm/alpaca"}, + "evaluation": {"enable": True, "dataset_path": "./datasets/llm/validation.jsonl"}, + } + llm_patch = { + "sft": {"dataset_path": "./datasets/llm/train.jsonl"}, + "evaluation": {"dataset_path": "./datasets/llm/train.jsonl"}, + } + + constraints = merge_llm_resource_constraints(constrained_base, {}) + locked = lock_structured_constraints(deep_merge_config(constrained_base, llm_patch), constraints) + + assert locked["llm"]["base_model"] == "./models/llm/tiny" + assert locked["sft"]["dataset_path"] == "./datasets/llm/alpaca" + assert locked["sft"]["file_format"] == "auto" + assert locked["sft"]["format"] == "alpaca" + assert locked["evaluation"]["dataset_path"] == "./datasets/llm/validation.jsonl" + + +def test_infer_sft_settings_from_dataset_path(): + assert infer_sft_settings_for_dataset_path("./datasets/llm/train.jsonl") == { + "file_format": "jsonl", + "format": "prompt_completion", + } + assert infer_sft_settings_for_dataset_path("./datasets/llm/alpaca") == { + "file_format": "auto", + "format": "alpaca", + } + assert infer_sft_settings_for_dataset_path("./datasets/llm/chat.parquet") == { + "file_format": "parquet", + "format": "messages", + } + + def test_schema_prompt_context_marks_disabled_options(): schema = { "federated": { diff --git a/apps/backend/tests/services/agent/test_prompts.py b/apps/backend/tests/services/agent/test_prompts.py index 7e22388..b7916ba 100644 --- a/apps/backend/tests/services/agent/test_prompts.py +++ b/apps/backend/tests/services/agent/test_prompts.py @@ -17,6 +17,8 @@ def test_build_plan_system_instructions_includes_global_prompt(): instructions = build_plan_system_instructions("test alpha=0.1,0.3") assert "User experiment request" in instructions assert "test alpha=0.1,0.3" in instructions + assert "LLM PEFT simulation" in instructions + assert "validation or perplexity" in instructions def test_build_plan_prompt_includes_history_and_context(): @@ -49,5 +51,6 @@ def test_build_plan_prompt_includes_history_and_context(): assert "Schema context from config_schema.yaml" in prompt assert "User-selected structured constraints" in prompt assert "disabled_options" in prompt + assert "evaluation.enable=true" in prompt assert "ResNet" in prompt assert "Return ONLY a JSON object" in prompt diff --git a/apps/backend/tests/services/agent/test_summary.py b/apps/backend/tests/services/agent/test_summary.py index 1fbab80..43ed771 100644 --- a/apps/backend/tests/services/agent/test_summary.py +++ b/apps/backend/tests/services/agent/test_summary.py @@ -2,7 +2,10 @@ from app.services.agent.summary import ( build_results_table, build_summary_text, + get_agent_run_score, get_last_global_accuracy, + get_last_llm_train_loss, + get_last_llm_validation_loss, ) @@ -16,6 +19,18 @@ def test_get_last_global_accuracy_returns_none_for_empty(): assert get_last_global_accuracy({"global_results": {}}) is None +def test_agent_run_score_uses_negative_llm_train_loss(): + metrics = {"llm_results": {"train_loss": [1.2, 0.8]}} + assert get_last_llm_train_loss(metrics) == 0.8 + assert get_agent_run_score(metrics) == -0.8 + + +def test_agent_run_score_prefers_llm_validation_loss(): + metrics = {"llm_results": {"train_loss": [0.8], "validation_loss": [0.6]}} + assert get_last_llm_validation_loss(metrics) == 0.6 + assert get_agent_run_score(metrics) == -0.6 + + def test_build_results_table_formats_experiments(): state = AgentState(goal="test experiments") state.experiment_results = [ @@ -89,3 +104,31 @@ def test_build_summary_text_reports_best_and_worst(): assert "Completed 2 experiments" in summary assert "Best accuracy: 0.8500 (high)" in summary assert "Worst accuracy: 0.3300 (low)" in summary + + +def test_build_summary_text_reports_llm_loss_when_present(): + state = AgentState(goal="test llm") + state.experiment_results = [ + ExperimentRecord( + iteration=1, + run_id="run-1", + job_id=10, + name="rank-8", + config={"task": {"type": "llm_peft_sft"}, "federated": {}}, + metrics={"llm_results": {"rounds": [1], "train_loss": [0.9]}}, + score=-0.9, + ), + ExperimentRecord( + iteration=2, + run_id="run-2", + job_id=11, + name="rank-16", + config={"task": {"type": "llm_peft_sft"}, "federated": {}}, + metrics={"llm_results": {"rounds": [1], "train_loss": [0.7]}}, + score=-0.7, + ), + ] + + summary = build_summary_text(state=state) + assert "Best LLM loss: train_loss=0.7000 (rank-16)" in summary + assert "Worst LLM loss: train_loss=0.9000 (rank-8)" in summary diff --git a/apps/backend/tests/services/distributed/test_distributed_job_service.py b/apps/backend/tests/services/distributed/test_distributed_job_service.py index c3f5657..70244fe 100644 --- a/apps/backend/tests/services/distributed/test_distributed_job_service.py +++ b/apps/backend/tests/services/distributed/test_distributed_job_service.py @@ -129,6 +129,18 @@ async def _run(): asyncio.run(_run()) +def test_normalize_server_config_rejects_llm_peft_route(): + service = DistributedJobService(session=_DummySession()) # type: ignore[arg-type] + + with pytest.raises(exceptions.BadRequestError, match="simulation mode only"): + service.config_service.normalize_server_config( + { + "task": {"type": "llm_peft_sft"}, + "federated": {"num_clients": 2}, + } + ) + + def test_update_job_config_normalizes_and_persists(monkeypatch): async def _run(): session = _DummySession() diff --git a/apps/backend/tests/services/test_job_service.py b/apps/backend/tests/services/test_job_service.py index 9bd3220..02662c6 100644 --- a/apps/backend/tests/services/test_job_service.py +++ b/apps/backend/tests/services/test_job_service.py @@ -6,6 +6,7 @@ def test_normalize_simulation_config_public_api_returns_invariants(): normalized = SimulationJobService.normalize_simulation_config({}) + assert normalized["task"]["type"] == "classic_fl" assert normalized["system"]["mode"] == "simulation" assert normalized["system"]["node_role"] == "server" @@ -25,3 +26,95 @@ def test_normalize_simulation_config_keeps_dataset_and_federated_client_counts_a assert normalized["federated"]["num_clients"] == 5 assert normalized["dataset"]["num_clients"] == 5 + + +def test_normalize_simulation_config_allows_auto_model_for_dataset_switch(): + normalized = SimulationJobService.normalize_simulation_config( + { + "dataset": {"name": "MNIST"}, + "model": {"name": "Auto"}, + } + ) + + assert normalized["dataset"]["name"] == "MNIST" + assert normalized["model"]["name"] == "Auto" + assert normalized["model"]["input_shape"] == [1, 28, 28] + assert normalized["model"]["num_classes"] == 10 + + +def test_normalize_simulation_config_rejects_incompatible_model(): + with pytest.raises(exceptions.BadRequestError, match="not compatible"): + SimulationJobService.normalize_simulation_config( + { + "dataset": {"name": "AG News"}, + "model": {"name": "ResNet18"}, + } + ) + + +def test_normalize_simulation_config_bypasses_classic_compatibility_for_llm_peft_route(): + normalized = SimulationJobService.normalize_simulation_config( + { + "task": {"type": "llm_peft_sft"}, + "dataset": {"name": "AG News"}, + "model": {"name": "ResNet18"}, + } + ) + + assert normalized["task"]["type"] == "llm_peft_sft" + assert normalized["dataset"]["name"] == "AG News" + assert normalized["model"]["name"] == "ResNet18" + + +def test_normalize_simulation_config_rejects_ckks_with_sparsification(): + with pytest.raises(exceptions.BadRequestError, match="not compatible"): + SimulationJobService.normalize_simulation_config( + { + "privacy": {"homomorphic_encryption": {"enable": True}}, + "compression": {"sparsification": {"enable": True}}, + } + ) + + +def test_normalize_simulation_config_accepts_dp_with_compression(): + normalized = SimulationJobService.normalize_simulation_config( + { + "privacy": { + "differential_privacy": { + "enable": True, + "clipping_norm": 1.0, + "noise_multiplier": 0.1, + } + }, + "compression": { + "sparsification": { + "enable": True, + "method": "random_k", + "ratio": 0.25, + } + }, + } + ) + + assert normalized["privacy"]["differential_privacy"]["enable"] is True + assert normalized["compression"]["sparsification"]["method"] == "random_k" + + +def test_normalize_simulation_config_rejects_secure_aggregation_with_weighted_avg(): + with pytest.raises(exceptions.BadRequestError, match="equal-weight"): + SimulationJobService.normalize_simulation_config( + { + "federated": {"aggregation": "weighted_avg"}, + "privacy": {"secure_aggregation": {"enable": True}}, + } + ) + + +def test_normalize_simulation_config_rejects_secure_aggregation_with_compression(): + with pytest.raises(exceptions.BadRequestError, match="not compatible"): + SimulationJobService.normalize_simulation_config( + { + "privacy": {"secure_aggregation": {"enable": True}}, + "compression": {"sparsification": {"enable": True}}, + } + ) diff --git a/apps/backend/tests/services/test_llm_resources.py b/apps/backend/tests/services/test_llm_resources.py new file mode 100644 index 0000000..14b2ea0 --- /dev/null +++ b/apps/backend/tests/services/test_llm_resources.py @@ -0,0 +1,196 @@ +from app.services.llm_resources import ( + DEFAULT_LLM_BASE_MODEL, + DEFAULT_LLM_DATASET_PATH, + DEFAULT_LLM_EVALUATION_DATASET_PATH, + augment_config_schema_with_llm_resources, + discover_llm_dataset_options, + discover_llm_evaluation_dataset_options, + discover_llm_model_options, + get_llm_resource_options, +) + + +def test_discovers_local_llm_model_directories(tmp_path): + model_dir = tmp_path / "models" / "llm" / "tiny-model" + model_dir.mkdir(parents=True) + nested_model_dir = tmp_path / "models" / "llm" / "nested" / "snapshot" + nested_model_dir.mkdir(parents=True) + (nested_model_dir / "config.json").write_text("{}", encoding="utf-8") + hidden_model_dir = tmp_path / "models" / "llm" / ".hidden-model" + hidden_model_dir.mkdir(parents=True) + + options = discover_llm_model_options(tmp_path) + + assert "./models/llm/tiny-model" in options + assert "./models/llm/nested/snapshot" in options + assert "./models/llm/.hidden-model" not in options + + +def test_discovers_hf_snapshot_model_directory(tmp_path): + snapshot_dir = tmp_path / "models" / "llm" / "Qwen2.5-7B-Instruct" / "snapshots" / "abc" + snapshot_dir.mkdir(parents=True) + (snapshot_dir / "config.json").write_text("{}", encoding="utf-8") + + options = discover_llm_model_options(tmp_path) + + assert "./models/llm/Qwen2.5-7B-Instruct" in options + assert "./models/llm/Qwen2.5-7B-Instruct/snapshots/abc" not in options + + +def test_discovers_local_llm_jsonl_datasets(tmp_path): + dataset_dir = tmp_path / "datasets" / "llm" + dataset_dir.mkdir(parents=True) + (dataset_dir / "train.jsonl").write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + (dataset_dir / "custom.jsonl").write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + (dataset_dir / "validation.jsonl").write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + nested_dir = dataset_dir / "alpaca" + nested_dir.mkdir() + (nested_dir / "train.jsonl").write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + (dataset_dir / "notes.txt").write_text("ignore me", encoding="utf-8") + + options = discover_llm_dataset_options(tmp_path) + + assert options == [ + "./datasets/llm/alpaca/train.jsonl", + "./datasets/llm/custom.jsonl", + "./datasets/llm/train.jsonl", + ] + + +def test_discovers_parquet_dataset_directory(tmp_path): + parquet_path = ( + tmp_path + / "datasets" + / "llm" + / "alpaca" + / "data" + / "train-00000-of-00001.parquet" + ) + parquet_path.parent.mkdir(parents=True) + parquet_path.write_bytes(b"parquet-placeholder") + + options = discover_llm_dataset_options(tmp_path) + + assert "./datasets/llm/alpaca" in options + assert "./datasets/llm/alpaca/data/train-00000-of-00001.parquet" not in options + + +def test_discovers_evaluation_dataset_options_including_validation(tmp_path): + dataset_dir = tmp_path / "datasets" / "llm" + dataset_dir.mkdir(parents=True) + (dataset_dir / "train.jsonl").write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + (dataset_dir / "validation.jsonl").write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + + options = discover_llm_evaluation_dataset_options(tmp_path) + + assert options == [ + "./datasets/llm/train.jsonl", + "./datasets/llm/validation.jsonl", + ] + + +def test_augments_schema_with_default_and_discovered_llm_options(tmp_path): + (tmp_path / "models" / "llm" / "tiny-model").mkdir(parents=True) + dataset_dir = tmp_path / "datasets" / "llm" + dataset_dir.mkdir(parents=True) + (dataset_dir / "train.jsonl").write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + (dataset_dir / "custom.jsonl").write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + (dataset_dir / "validation.jsonl").write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + schema = { + "llm": { + "base_model": { + "type": "text", + "default": DEFAULT_LLM_BASE_MODEL, + "ui": {}, + } + }, + "sft": { + "dataset_path": { + "type": "text", + "default": DEFAULT_LLM_DATASET_PATH, + "ui": {}, + } + }, + "evaluation": { + "enable": { + "type": "bool", + "default": False, + "ui": {}, + }, + "dataset_path": { + "type": "text", + "default": DEFAULT_LLM_EVALUATION_DATASET_PATH, + "ui": {}, + } + }, + } + + augmented = augment_config_schema_with_llm_resources(schema, tmp_path) + + assert augmented["llm"]["base_model"]["options"] == [ + "./models/llm/tiny-model", + ] + assert augmented["llm"]["base_model"]["default"] == "./models/llm/tiny-model" + assert augmented["llm"]["base_model"]["ui"]["option_source"]["path"] == "./models/llm" + assert augmented["sft"]["dataset_path"]["options"] == [ + "./datasets/llm/custom.jsonl", + "./datasets/llm/train.jsonl", + ] + assert augmented["sft"]["dataset_path"]["default"] == "./datasets/llm/custom.jsonl" + assert augmented["sft"]["dataset_path"]["ui"]["option_source"]["path"] == "./datasets/llm" + assert augmented["evaluation"]["dataset_path"]["options"] == [ + "./datasets/llm/custom.jsonl", + "./datasets/llm/train.jsonl", + "./datasets/llm/validation.jsonl", + ] + assert augmented["evaluation"]["dataset_path"]["default"] == "./datasets/llm/validation.jsonl" + assert augmented["evaluation"]["dataset_path"]["ui"]["option_source"]["path"] == "./datasets/llm" + assert augmented["evaluation"]["enable"]["default"] is True + + +def test_get_llm_resource_options_returns_defaults_and_discovered_values(tmp_path): + (tmp_path / "models" / "llm" / "tiny-model" / "config.json").parent.mkdir(parents=True) + (tmp_path / "models" / "llm" / "tiny-model" / "config.json").write_text("{}", encoding="utf-8") + dataset_path = tmp_path / "datasets" / "llm" / "custom.jsonl" + dataset_path.parent.mkdir(parents=True) + dataset_path.write_text('{"prompt":"p","completion":"c"}\n', encoding="utf-8") + + resources = get_llm_resource_options(tmp_path) + + assert resources["default_model"] == "./models/llm/tiny-model" + assert resources["models"] == ["./models/llm/tiny-model"] + assert resources["datasets"] == ["./datasets/llm/custom.jsonl"] + assert resources["evaluation_datasets"] == ["./datasets/llm/custom.jsonl"] + assert resources["default_evaluation_dataset"] == "" + + +def test_augments_schema_with_empty_resource_messages_when_no_local_resources(tmp_path): + schema = { + "llm": {"base_model": {"type": "text", "default": DEFAULT_LLM_BASE_MODEL, "ui": {}}}, + "sft": {"dataset_path": {"type": "text", "default": DEFAULT_LLM_DATASET_PATH, "ui": {}}}, + "evaluation": { + "enable": { + "type": "bool", + "default": True, + "ui": {}, + }, + "dataset_path": { + "type": "text", + "default": DEFAULT_LLM_EVALUATION_DATASET_PATH, + "ui": {}, + } + }, + } + + augmented = augment_config_schema_with_llm_resources(schema, tmp_path) + + assert augmented["llm"]["base_model"]["options"] == [] + assert augmented["llm"]["base_model"]["default"] == "" + assert augmented["llm"]["base_model"]["ui"]["empty_message"] == "No local models found" + assert augmented["sft"]["dataset_path"]["options"] == [] + assert augmented["sft"]["dataset_path"]["default"] == "" + assert augmented["sft"]["dataset_path"]["ui"]["empty_message"] == "No training datasets found" + assert augmented["evaluation"]["dataset_path"]["options"] == [] + assert augmented["evaluation"]["dataset_path"]["default"] == "" + assert augmented["evaluation"]["dataset_path"]["ui"]["empty_message"] == "No evaluation datasets found" + assert augmented["evaluation"]["enable"]["default"] is False diff --git a/apps/backend/tests/services/test_run_metrics_service.py b/apps/backend/tests/services/test_run_metrics_service.py new file mode 100644 index 0000000..5deafbc --- /dev/null +++ b/apps/backend/tests/services/test_run_metrics_service.py @@ -0,0 +1,41 @@ +from __future__ import annotations + +from app.services.simulation.run_metrics_service import SimulationRunMetricsService + + +def test_normalize_metrics_payload_preserves_llm_sections(): + service = SimulationRunMetricsService.__new__(SimulationRunMetricsService) + + normalized = service._normalize_metrics_payload( + { + "llm_results": { + "rounds": [1, "2"], + "train_loss": [1.5], + "validation_loss": [1.2], + "perplexity": [3.32], + "token_throughput": [42], + "adapter_size_bytes": [1024], + }, + "llm_dataset": {"num_records": 12}, + "llm_evaluation": {"enabled": True, "num_records": 4}, + "llm_runtime": {"status": "blocked"}, + "llm_artifacts": [{"round": 1, "path": "adapter.pt"}, "not-an-object"], + "client_results": { + "client_0": { + "rounds": [1, "3"], + "train_loss": [1.0, 0.8], + "train_acc": [], + "test_loss": [], + "test_acc": [], + } + }, + } + ) + + assert normalized["llm_results"]["rounds"] == [1, 2] + assert normalized["llm_results"]["token_throughput"] == [42.0] + assert normalized["llm_dataset"]["num_records"] == 12 + assert normalized["llm_evaluation"]["num_records"] == 4 + assert normalized["llm_runtime"]["status"] == "blocked" + assert normalized["llm_artifacts"] == [{"round": 1, "path": "adapter.pt"}] + assert normalized["client_results"]["client_0"]["rounds"] == [1, 3] diff --git a/apps/backend/tests/services/test_run_service.py b/apps/backend/tests/services/test_run_service.py index b04fa63..2d7cf02 100644 --- a/apps/backend/tests/services/test_run_service.py +++ b/apps/backend/tests/services/test_run_service.py @@ -1,4 +1,5 @@ import asyncio +import re from pathlib import Path from app.models.simulation import SimulationJob, SimulationJobStatus, SimulationRun, SimulationRunStatus @@ -17,6 +18,11 @@ def test_run_service_start_run_flow_without_real_subprocess(monkeypatch, tmp_pat async def _run_test(): service = SimulationRunService(session=_DummySession()) service._job_config_dir = lambda: Path(tmp_path) # type: ignore[method-assign] + service._results_dir = lambda: Path(tmp_path) / "results" # type: ignore[method-assign] + service._runtime_log_dir = lambda: Path(tmp_path) / "logs" # type: ignore[method-assign] + service._results_dir().mkdir(parents=True, exist_ok=True) + service._runtime_log_dir().mkdir(parents=True, exist_ok=True) + captured: dict[str, object] = {"results": []} class _FakeProcess: def __init__(self): @@ -42,9 +48,11 @@ async def readline(self): return b"" async def _fake_spawn(self, _command, env=None): + captured["env"] = env return _FakeProcess() - async def _fake_watch(self, run_id, process): + async def _fake_watch(self, run_id, process, *, artifact_timestamp=None): + captured["artifact_timestamp"] = artifact_timestamp self._processes.pop(run_id, None) self._tasks.pop(run_id, None) @@ -66,6 +74,7 @@ async def _add_log(_run_id, _msg, level="INFO"): return None async def _add_result(**_kwargs): + captured["results"].append(_kwargs) return None monkeypatch.setattr(SimulationRunService, "_spawn_subprocess", _fake_spawn) @@ -81,5 +90,19 @@ async def _add_result(**_kwargs): assert run is not None assert run.status == SimulationRunStatus.RUNNING assert run.process_id == 111 + env = captured["env"] + assert isinstance(env, dict) + result_filename = env["FIGARO_RESULTS_FILE"] + result_match = re.fullmatch(rf"(\d{{8}}_\d{{6}}Z)_{run.id}_live_results\.json", result_filename) + assert result_match is not None + timestamp = result_match.group(1) + + results = captured["results"] + assert isinstance(results, list) + config_artifact = results[0] + config_path = Path(config_artifact["path"]) + assert config_path.name == f"{timestamp}_{run.id}.json" + assert config_artifact["metadata_json"]["artifact_timestamp"] == timestamp + assert service._simulation_log_path(run.id, artifact_timestamp=timestamp).name == f"{timestamp}_{run.id}_server.log" asyncio.run(_run_test()) diff --git a/apps/backend/tests/test_project_dependencies.py b/apps/backend/tests/test_project_dependencies.py new file mode 100644 index 0000000..df1b6a2 --- /dev/null +++ b/apps/backend/tests/test_project_dependencies.py @@ -0,0 +1,13 @@ +from __future__ import annotations + +from pathlib import Path +import tomllib + + +def test_llm_peft_dependencies_are_core_project_dependencies(): + project_root = Path(__file__).resolve().parents[3] + pyproject = tomllib.loads((project_root / "pyproject.toml").read_text(encoding="utf-8")) + dependencies = "\n".join(pyproject["project"]["dependencies"]) + + for package in ("transformers", "peft", "accelerate", "safetensors", "bitsandbytes"): + assert package in dependencies diff --git a/apps/frontend/src/api/agent.ts b/apps/frontend/src/api/agent.ts index 0e0cda6..876d06d 100644 --- a/apps/frontend/src/api/agent.ts +++ b/apps/frontend/src/api/agent.ts @@ -21,6 +21,42 @@ export type AgentConfigChange = { new_value: unknown; }; +export type AgentHistoryFilters = { + q?: string; + status?: string; + model_name?: string; + objective?: AgentOptimizationObjective | "all"; + best_score_min?: string; + best_score_max?: string; + created_from?: string; + created_to?: string; + dataset?: string; + config_model?: string; + aggregation?: string; + num_clients?: string; + num_rounds?: string; +}; + +export type AgentConfigVersion = { + id: number; + optimization_job_id: number; + run_id: string | null; + iteration: number; + source: string; + label: string; + config_hash: string; + config_json: Record; + diff_json: AgentConfigChange[]; + created_at: string; +}; + +export type AgentConfigDiffResponse = { + optimization_job_id: number; + from_version_id: number | null; + to_version_id: number; + changes: AgentConfigChange[]; +}; + export type AgentCurrentPlan = { iteration: number; iteration_goal: string; @@ -192,6 +228,33 @@ async function readJson(response: Response): Promise { return (await response.json()) as T; } +function addOptionalParam(params: URLSearchParams, key: string, value: unknown): void { + if (value === undefined || value === null) return; + const text = String(value).trim(); + if (!text || text === "all") return; + params.set(key, text); +} + +function historyFiltersToParams(filters?: AgentHistoryFilters): string { + const params = new URLSearchParams(); + if (!filters) return ""; + addOptionalParam(params, "q", filters.q); + addOptionalParam(params, "status", filters.status); + addOptionalParam(params, "model_name", filters.model_name); + addOptionalParam(params, "objective", filters.objective); + addOptionalParam(params, "best_score_min", filters.best_score_min); + addOptionalParam(params, "best_score_max", filters.best_score_max); + addOptionalParam(params, "created_from", filters.created_from); + addOptionalParam(params, "created_to", filters.created_to); + addOptionalParam(params, "dataset", filters.dataset); + addOptionalParam(params, "config_model", filters.config_model); + addOptionalParam(params, "aggregation", filters.aggregation); + addOptionalParam(params, "num_clients", filters.num_clients); + addOptionalParam(params, "num_rounds", filters.num_rounds); + const query = params.toString(); + return query ? `?${query}` : ""; +} + export const agentApi = { async getConfigSchema(): Promise> { const response = await fetch(`${baseUrl}/api/v1/agent/config/schema`); @@ -226,8 +289,8 @@ export const agentApi = { return readJson(response); }, - async listOptimizationJobs(): Promise { - const response = await fetch(`${baseUrl}/api/v1/agent/optimization-jobs`); + async listOptimizationJobs(filters?: AgentHistoryFilters): Promise { + const response = await fetch(`${baseUrl}/api/v1/agent/optimization-jobs${historyFiltersToParams(filters)}`); return readJson(response); }, @@ -303,4 +366,18 @@ export const agentApi = { } return response.json(); }, + + async listConfigVersions(optimizationJobId: number): Promise { + const response = await fetch(`${baseUrl}/api/v1/agent/optimization-jobs/${encodeURIComponent(optimizationJobId)}/config-versions`); + return readJson(response); + }, + + async getConfigDiff(optimizationJobId: number, toVersionId: number, fromVersionId?: number | null): Promise { + const params = new URLSearchParams({ to_version_id: String(toVersionId) }); + if (fromVersionId != null) { + params.set("from_version_id", String(fromVersionId)); + } + const response = await fetch(`${baseUrl}/api/v1/agent/optimization-jobs/${encodeURIComponent(optimizationJobId)}/config-diff?${params.toString()}`); + return readJson(response); + }, }; diff --git a/apps/frontend/src/api/runs.ts b/apps/frontend/src/api/runs.ts index 799cddd..a0446b3 100644 --- a/apps/frontend/src/api/runs.ts +++ b/apps/frontend/src/api/runs.ts @@ -4,6 +4,7 @@ import { getHttpErrorMessage } from "./error"; type Run = components["schemas"]["RunResponse"]; export type RunClientMetricSeries = { + rounds?: number[]; train_loss: number[]; train_acc: number[]; test_loss: number[]; @@ -21,6 +22,18 @@ export type RunMetrics = { global_loss: number[]; global_accuracy: number[]; }; + llm_results?: { + rounds: number[]; + train_loss: number[]; + validation_loss: number[]; + perplexity: number[]; + token_throughput: number[]; + adapter_size_bytes: number[]; + }; + llm_dataset?: Record; + llm_evaluation?: Record; + llm_runtime?: Record; + llm_artifacts?: Record[]; client_results: Record; }; diff --git a/apps/frontend/src/features/agent/components/AgentExperimentStudio.tsx b/apps/frontend/src/features/agent/components/AgentExperimentStudio.tsx index ef8e760..fe49c69 100644 --- a/apps/frontend/src/features/agent/components/AgentExperimentStudio.tsx +++ b/apps/frontend/src/features/agent/components/AgentExperimentStudio.tsx @@ -1,5 +1,5 @@ -import { ArrowRight, Loader2, SlidersHorizontal, Sparkles, X } from "lucide-react"; -import { useMemo } from "react"; +import { ArrowRight, Loader2, RefreshCw, SlidersHorizontal, Sparkles, X } from "lucide-react"; +import { useMemo, useState } from "react"; import { Badge } from "../../../components/ui/badge"; import { Button } from "../../../components/ui/button"; @@ -12,17 +12,23 @@ import { Textarea } from "../../../components/ui/textarea"; import type { AgentPageProps } from "../../../pages/types"; import { getValueByPath } from "../../simulation/utils"; import { + booleanDisabledForConfig, + booleanDisableReasonForConfig, + agentFieldVisibleForConfig, buildAgentConfig, - checkDependency, coerceFieldValue, collectAgentSchemaFields, + fieldEmptyMessage, + fieldCompatibilityHint, formatFieldValue, - optionDisabled, + optionDisableReasonForConfig, + optionDisabledForConfig, optionLabel, optionMeta, selectedConstraintChips, type AgentSchemaField, } from "../schema"; +import { isModelCompatibleWithDataset } from "../../config/compatibility"; export function AgentExperimentStudio(props: AgentPageProps) { const { @@ -37,7 +43,10 @@ export function AgentExperimentStudio(props: AgentPageProps) { presets, busy, handleGeneratePlan, + notifyError, + refreshLlmResources, } = props; + const [refreshingResources, setRefreshingResources] = useState(false); const fields = useMemo(() => collectAgentSchemaFields(configSchema, { featuredOnly: true }), [configSchema]); const effectiveConfig = useMemo( @@ -45,10 +54,21 @@ export function AgentExperimentStudio(props: AgentPageProps) { [configConstraints, configSchema], ); const chips = useMemo( - () => selectedConstraintChips(configSchema, configConstraints), - [configConstraints, configSchema], + () => selectedConstraintChips(configSchema, configConstraints, effectiveConfig), + [configConstraints, configSchema, effectiveConfig], ); - const visibleFields = fields.filter((field) => checkDependency(effectiveConfig, field.definition.depends_on)); + const visibleFields = fields.filter((field) => agentFieldVisibleForConfig(field, effectiveConfig)); + + async function handleRefreshResources(): Promise { + setRefreshingResources(true); + try { + await refreshLlmResources(); + } catch (error) { + notifyError(error, "agent-llm-resources-refresh"); + } finally { + setRefreshingResources(false); + } + } return (
@@ -72,7 +92,7 @@ export function AgentExperimentStudio(props: AgentPageProps) {