Commit 1660038
committed
test(optimize): 54 regression tests for all 6 confirmed bugs
Also fixes two source bugs uncovered by the new tests:
BUG-5 fix: OptimizeMetricAdapter.score() — wrap coro-creation inside
the try block so sync-raising evaluate() is caught and returns 0.5
neutral (instead of propagating the exception to the caller).
BUG-3 fix: PromptFitResult.to_dict() — include score_history in the
serialized dict so it round-trips through JSON artefacts.
New test file: tests/test_optimization_bugs.py
TestBug1AgentRunnerLocalCallable (7 tests)
- async callable dispatched correctly
- sync callable runs via asyncio.to_thread (verified with threading)
- dict response extracted via _response_text
- errors captured, no crash
- prompt_override / context forwarded
- without callable falls through to HTTP path
TestBug2LLMCallerBaseUrl (5 tests)
- configure() stores / resets class-level attrs
- LLMCaller.call() forwards base_url/api_key to _call_openai
- per-call args override class-level defaults
- non-openai provider unaffected
TestBug3PromptFitResultHistory (8 tests)
- .history returns score_history list
- derives from full_val trials when score_history empty
- fallback to all trial scores
- empty list when no data
- stable across repeated calls
- score_history in to_dict()
- generate_fit_report does not crash
- fitter.fit() populates score_history on result
TestBug4CompositeMetricScore (9 tests)
- has .score() method
- returns float in [0,1]
- custom always-1.0/0.0 sub-metrics
- dict expected_output JSON-serialised and forwarded
- sub-metric failure → 0.0 contribution, no exception
- weighted average computed correctly
- usable in agents.WeightedMetric (validates .score() on init)
- usable in MetricLoss
TestBug5OptimizeMetricAdapter (13 tests)
- returns float not coroutine
- correctly awaits async metric
- query extracted from input["query"] and ["current_query"]
- prediction dict JSON-serialised as response string
- expected_output dict JSON-serialised
- expected_output None passed as None
- neutral 0.5 on async exception (judge unavailable)
- neutral 0.5 on sync exception (evaluate() raises before await)
- usable in agents.WeightedMetric
- usable in MetricLoss
- name inherited from metric / overrideable
- score value propagated correctly for all values in [0,1]
TestBug6PromptFitterBridgeLocalAgent (5 tests)
- local_agent= param forwarded to PromptFitter
- live agent used as fallback when local_agent=None
- explicit local_agent wins over live agent
- llm_base_url/api_key set on LLMCaller class defaults
- injected fitter= returned as-is
TestEndToEndLocalTraining (3 tests)
- full PromptFitter.fit() with local callable: history, score_history,
best_score, summary() all accessible
- OptimizeMetricAdapter → WeightedMetric → MetricLoss end-to-end
- CompositeMetric → MetricLoss with multiple sub-metrics1 parent 44e8854 commit 1660038
3 files changed
Lines changed: 991 additions & 1 deletion
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
340 | 340 | | |
341 | 341 | | |
342 | 342 | | |
343 | | - | |
| 343 | + | |
| 344 | + | |
| 345 | + | |
344 | 346 | | |
| 347 | + | |
345 | 348 | | |
346 | 349 | | |
347 | 350 | | |
348 | 351 | | |
349 | 352 | | |
350 | 353 | | |
| 354 | + | |
351 | 355 | | |
352 | 356 | | |
353 | 357 | | |
| |||
| Original file line number | Diff line number | Diff line change | |
|---|---|---|---|
| |||
424 | 424 | | |
425 | 425 | | |
426 | 426 | | |
| 427 | + | |
427 | 428 | | |
428 | 429 | | |
429 | 430 | | |
| |||
0 commit comments