fix(acc-scanner): skip @acc: matches inside string literals

[jhetchan]

Summary

The @acc: scanner in exo/stdlib/requirements.py previously used a plain regex over each line of text to find acceptance-criteria annotations in test files. This approach failed for Python test files whose fixtures quote the marker as string-literal data — e.g. write_text("# @acc: ACC-001\n…") or triple-quoted blocks where # @acc: appears at the start of a source line inside the literal. Those occurrences were being matched and reported as acc_orphan errors, causing exo trace-reqs --check-tests (and the session-finish verify gate) to fail spuriously. The fix uses tokenize.generate_tokens for .py files and matches @acc: only inside COMMENT tokens, making string literals invisible to the scanner. Non-Python files (.ts, .js, .rb, etc.) continue using the existing line-regex path because their comment syntax does not produce the same false-positive collision.

Repro

Before (plain regex — .py file string literals matched):

With a requirements.yaml that defines ACC IDs, running exo trace-reqs --check-tests against the repo would produce ~23 acc_orphan errors, all pointing at lines inside tests/test_requirements.py that quote the @acc: marker inside Python string literals (e.g. test fixture data written to temp files). None of those occurrences were real test annotations.

errors (23):
  - [acc_orphan] tests/test_requirements.py:949: @acc: ACC-001 is not defined in any requirement's acceptance list
  - [acc_orphan] tests/test_requirements.py:962: @acc: ACC-001 is not defined in any requirement's acceptance list
  ... (21 more)

After (tokenize — only COMMENT tokens matched):

exo trace-reqs --check-tests
Requirement Traceability: PASS
  acceptance criteria: N defined, N tested
  (no acc_orphan violations)

Test plan

• TestAccScannerStringLiteralImmunity::test_string_literal_yields_no_refs — Python source with f = "# @acc: ACC-001" yields zero refs
• TestAccScannerStringLiteralImmunity::test_real_comment_yields_one_ref — Python source with # @acc: ACC-002 comment yields exactly one ref at ACC-002
• TestAccScannerStringLiteralImmunity::test_mixed_source_yields_only_comment_ref — source mixing both yields exactly one ref (the comment)
• TestAccScannerStringLiteralImmunity::test_tokenize_error_returns_empty — syntactically broken Python (""" unterminated) does not raise; returns empty list
• TestAccScannerStringLiteralImmunity::test_non_python_file_line_regex_preserved — .ts file with // @acc: ACC-003 still yields one ref (non-Python path preserved)
• python3 -m pytest tests/test_requirements.py -v — 97/97 pass including new tests
• ruff check exo/ tests/ and ruff format --check exo/ tests/ — clean
• python3 -m pytest — 1583 passing; 62 pre-existing failures on main unchanged

---

Governed by ExoProtocol · ticket TKT-20260502-093139-P9I2 · intent INT-20260502-093114-N3NR

https://claude.ai/code/session_012d1U3o5ej6rQ9Gm447LjiL

---

Generated by Claude Code

[exoprotocol-governance]

❌ ExoProtocol Health Report

Verdict: FAIL  ·  Drift: n/a  ·  Ungoverned commits: 1  ·  Scope issues: 2

PR Governance Check: FAIL
  range: a42d5cce15f5d39003b63501f43ebcf9eacf0d26..50dd76c6c76a22c6d00a6077f3ea181235c4aca2
  commits: 1 total, 0 governed, 1 ungoverned
  governance: DRIFT DETECTED
  changed files: 2
  scope violations: ['exo/stdlib/requirements.py', 'tests/test_requirements.py']
  reasons:
    - Governance integrity check failed — constitution drift or missing lock
    - 2 file(s) not covered by any session scope: exo/stdlib/requirements.py, tests/test_requirements.py
    - 1 commit(s) made outside any governed session

---

View full report & drift trends →  ·  ExoProtocol