Fix correctness bugs found in code review

[hherb]

Summary

A thorough code review surfaced a number of genuine correctness bugs — cases where the library would silently produce wrong or unintended results. This PR fixes each one, with regression tests, across 11 focused commits. All 336 tests pass (was 308; +28 new). No new lint errors are introduced.

Fixes

High severity (silent data loss / systematically wrong results)

• Full-text sources silently dropped during sync (publications/sync.py). _record_to_fulltext_sources accessed each entry as a dict (.get()/[]), but fetchers populate FetchedRecord.fulltext_sources with FullTextSourceEntry dataclasses. Every record carrying a full-text source raised inside the swallowed except, was counted as failed, and the publication was never stored. The bug was masked by a test fixture that used plain dicts — now fixed to use the real type.
• Transparency COI detection read the wrong field (transparency/). COI/data-availability statements live in full text, not the abstract, so reading them from abstractText produced near-universal false negatives that drove almost every paper to HIGH risk via the missing-COI downgrade. Now fetches EuropePMC full text when available and scans that; coi_disclosed is tri-state (True/False/None) and only an explicit False triggers the downgrade.
• Ollama token accounting clobbered legitimate zero counts (llm/providers/ollama.py). value or estimate replaced a real 0 (cached prompt / empty completion) with a fabricated estimate, corrupting usage/cost. Uses is None now.

Medium severity (wrong classification / parsing / atomicity)

• Study-design misclassification (quality/metadata_filter.py). Re-ordered priority so cohort designs outrank case-control (a paper tagged both was downgraded to the weaker design); dropped Multicenter Study → RCT and Comparative Study → cross-sectional mappings (organisational/generic attributes, not designs — they misclassified observational papers as RCTs with high confidence).
• ClinicalTrials.gov results check (transparency/analyzer.py). Requested a ResultsSection field but read a resultsSection key, under-detecting posted results. Uses the v2 hasResults boolean with a fallback.
• SQLite migrations were not atomic (db/operations.py). executescript() issues an implicit COMMIT, committing the migration's BEGIN early; a later failure left a half-applied, unrecorded schema. Statements are now executed individually (SQLite supports transactional DDL) so failures roll back cleanly.
• JATS parser (fulltext/jats_parser.py): header-less tables no longer misclassify a data row with a leading <th> row-label as a header; titled-but-empty abstract subsections are no longer dropped; multi-paragraph captions are joined with a space.

Low severity

• Full-text cache key collisions (fulltext/service.py): distinct identifiers could sanitise to the same filename and serve the wrong article's text — now append a short hash. Also: PubMed fallback mislabelled source="doi" → "pubmed"; URL-encode the Unpaywall email.
• LLM provider casing (llm/client.py): a mixed-case default_provider/Provider:model desynced routing, cost aggregation, and the tool-capability allowlist — normalised to lowercase.
• bioRxiv PDF version hard-coded v1 → uses the record's actual version; PubMed non-numeric months (e.g. Winter) no longer produce invalid dates like 2024-Winter.
• JSON extraction (llm/utils.py, agents/base.py): greedy \{.*\} spanned first-to-last brace, swallowing prose between objects — replaced with balanced-brace scanning that respects string literals and returns the first parseable object.
• OpenAI reasoning models (llm/providers/openai_compat.py): o1/o3 reject max_tokens/temperature — now use max_completion_tokens via an overridable hook.

Investigated, not a bug

• A reported missing case_series value in the classifier prompts was a false positive — both prompts and the score/tier mappings already support it. No change.

Testing

• pytest tests/ -v → 336 passed.
• Each fix ships a regression test that fails against the old behaviour.

🤖 Generated with Claude Code

---

Generated by Claude Code