Skip to content

Commit 1660038

Browse files
test(optimize): 54 regression tests for all 6 confirmed bugs
Also fixes two source bugs uncovered by the new tests: BUG-5 fix: OptimizeMetricAdapter.score() — wrap coro-creation inside the try block so sync-raising evaluate() is caught and returns 0.5 neutral (instead of propagating the exception to the caller). BUG-3 fix: PromptFitResult.to_dict() — include score_history in the serialized dict so it round-trips through JSON artefacts. New test file: tests/test_optimization_bugs.py TestBug1AgentRunnerLocalCallable (7 tests) - async callable dispatched correctly - sync callable runs via asyncio.to_thread (verified with threading) - dict response extracted via _response_text - errors captured, no crash - prompt_override / context forwarded - without callable falls through to HTTP path TestBug2LLMCallerBaseUrl (5 tests) - configure() stores / resets class-level attrs - LLMCaller.call() forwards base_url/api_key to _call_openai - per-call args override class-level defaults - non-openai provider unaffected TestBug3PromptFitResultHistory (8 tests) - .history returns score_history list - derives from full_val trials when score_history empty - fallback to all trial scores - empty list when no data - stable across repeated calls - score_history in to_dict() - generate_fit_report does not crash - fitter.fit() populates score_history on result TestBug4CompositeMetricScore (9 tests) - has .score() method - returns float in [0,1] - custom always-1.0/0.0 sub-metrics - dict expected_output JSON-serialised and forwarded - sub-metric failure → 0.0 contribution, no exception - weighted average computed correctly - usable in agents.WeightedMetric (validates .score() on init) - usable in MetricLoss TestBug5OptimizeMetricAdapter (13 tests) - returns float not coroutine - correctly awaits async metric - query extracted from input["query"] and ["current_query"] - prediction dict JSON-serialised as response string - expected_output dict JSON-serialised - expected_output None passed as None - neutral 0.5 on async exception (judge unavailable) - neutral 0.5 on sync exception (evaluate() raises before await) - usable in agents.WeightedMetric - usable in MetricLoss - name inherited from metric / overrideable - score value propagated correctly for all values in [0,1] TestBug6PromptFitterBridgeLocalAgent (5 tests) - local_agent= param forwarded to PromptFitter - live agent used as fallback when local_agent=None - explicit local_agent wins over live agent - llm_base_url/api_key set on LLMCaller class defaults - injected fitter= returned as-is TestEndToEndLocalTraining (3 tests) - full PromptFitter.fit() with local callable: history, score_history, best_score, summary() all accessible - OptimizeMetricAdapter → WeightedMetric → MetricLoss end-to-end - CompositeMetric → MetricLoss with multiple sub-metrics
1 parent 44e8854 commit 1660038

3 files changed

Lines changed: 991 additions & 1 deletion

File tree

src/agentomatic/agents/metrics.py

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -340,14 +340,18 @@ def score(
340340
)
341341

342342
# --- run evaluate() synchronously ---
343-
coro = self._metric.evaluate(query, response, expected)
343+
# Wrap the entire call (including coro creation) so that metrics whose
344+
# evaluate() raises synchronously (non-coroutine callables that throw)
345+
# are also handled gracefully.
344346
try:
347+
coro = self._metric.evaluate(query, response, expected)
345348
result = asyncio.run(coro)
346349
except RuntimeError:
347350
# Already inside a running event loop (e.g. notebook, FastAPI handler)
348351
import concurrent.futures
349352

350353
try:
354+
coro = self._metric.evaluate(query, response, expected)
351355
with concurrent.futures.ThreadPoolExecutor(max_workers=1) as pool:
352356
result = pool.submit(asyncio.run, coro).result(timeout=60)
353357
except Exception:

src/agentomatic/optimize/config.py

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -424,6 +424,7 @@ def to_dict(self) -> dict[str, Any]:
424424
"best_score": self.best_score,
425425
"baseline_score": self.baseline_score,
426426
"absolute_improvement": self.absolute_improvement,
427+
"score_history": self.score_history,
427428
"metric_deltas": self.metric_deltas,
428429
"param_suggestions": {k: asdict(v) for k, v in self.param_suggestions.items()},
429430
"best_config": self.best_config.to_dict(),

0 commit comments

Comments
 (0)