ASTRA export can record a rejected option as the chosen one, and silently drops unparseable decisions

[lfnothias]

Observation

Two behaviours in the decision extractor let the exported capsule state something the run did not do, without surfacing that anything went wrong.

1. A rejected option can be recorded as the chosen one. decision_extractor.py:206-209:

if not (isinstance(chosen, str) and chosen in valid_ids):
    # Fall back to the first listed option (the prompt lists the chosen one
    # first) so a missing/typo'd chosen_option_id doesn't drop the decision.
    chosen = options[0].id

The justification is that the prompt lists the chosen option first. sources/transparency/prompts/decision_extraction.md contains no such ordering instruction. So when the model omits or misspells chosen_option_id, the capsule silently records options[0] as the realized decision, which may be an alternative that was considered and rejected.

2. A malformed extraction is indistinguishable from "no decision". _parse_response (decision_extractor.py:103) returns None both for a legitimate null and for unparseable or schema-violating output, logging only at DEBUG. _warn_on_failures counts only raised exceptions. A model returning prose, truncated JSON, or a wrong schema therefore yields a capsule with no decisions and no warning.

Commit 1d7645a moved extraction onto the cheaper smolagent_model_id, which makes both paths more likely rather than less.

Why it matters

The capsule's stated purpose is to let a reviewer see what was chosen over what. Recording a rejected alternative as the decision, or emitting an empty decision set, is not a degraded export — it is a confident and wrong provenance record, which is worse than an absent one. Neither case is currently visible to the person reading the capsule.

Pathway

1. Do not guess the chosen option. If chosen_option_id is missing or unknown, either drop the decision and count it as a failure, or keep the decision and mark it explicitly (for example chosen_option_id: null with a chosen_unresolved: true flag) so the reviewer sees the gap. Remove the ordering assumption from the comment either way.
2. Make the prompt earn the assumption if you prefer to keep it: state in decision_extraction.md that the chosen option must be listed first, and validate it.
3. Separate "no decision" from "could not parse". Give _parse_response distinct return values, count parse failures alongside _EXTRACT_FAILED, and let _warn_on_failures report them.
4. Record the failure count in the capsule itself, so an export produced from a degraded extraction is self-describing.

Step 1 and step 3 are independent and each removes one silent-wrong path.

Note

tests/astra_exporter_test.py currently fails on mimosa_v2 — three tests construct Decision with the option_id / option_label / option_description keywords that 18529e8 replaced with chosen_option_id + options. Production code uses the new shape, so the exporter itself is fine; the tests are stale and the new multi-option emission has lost its coverage. Happy to send that as a separate small pull request.

[lfnothias]

The test half is sent as #177. It repairs the three stale constructions and adds coverage for what the new shape exists for: that alternatives are emitted beside the chosen option, and that build_universe reads chosen_option_id rather than the first listed option.

That second test is deliberately independent of whatever is decided here about the extractor fallback. The writer should never depend on option order, and now a test holds it to that. The fallback itself, and the silent dropping of unparseable decisions, remain open in this issue.