fix(tables): flatten page-number tables of contents instead of gridding

[abimaelmartell]

Summary

A title-based table-of-contents page — About the Publisher vii, Experiment #1: Hydrostatic Pressure 3 — with no dot leaders and no section numbers was being detected as a 2-column data table and rendered as a markdown grid. That scrambles the page's linear reading order (a leading cause of reading-order/NID loss on those docs) and scores zero on table structure.

The codebase already renders detected TOCs as a flat per-row list (TableKind::Toc → format_toc_as_list), but is_table_of_contents only recognized dot-leader and section-numbered TOCs. This adds the missing page-number-column case.

is_page_number_toc matches a narrow (2–3 column) list where:

• the last column is page numbers (short integers or roman numerals) on ≥70% of rows, and those values are mostly non-decreasing (the monotonic run is what separates a contents list from an incidental numeric column),
• the first column is text titles,
• there is no header row — a TOC's first row is already an entry with a page number, whereas a data table opens with a column header.

The narrow-width, no-header, and monotonic guards keep real tables intact — verified on a 4-column regional data table and a 2-column Mineral | CEC table (header row + ascending values) that both correctly stay gridded.

Impact (opendataloader-bench, 200 docs)

Metric	Before	After
Overall	0.830	0.834
Reading order (NID)	0.883	0.888
Tables (TEDS)	0.656	0.656
Headings (MHS)	0.736	0.742

NID (reading order) and MHS improve with no TEDS regression — the change removes phantom-table scrambling without touching real tables.

Testing

• Unit tests: title-based TOC matches; a numeric data table, a non-monotonic label/number table, a prose last column, a header-row Mineral | CEC table, and a 4-column regional grid all correctly rejected.
• cargo fmt / cargo clippy -- -D warnings / cargo test pass.
• Corpus FP sweep: table flattening only hits genuine contents/index pages (e.g. a financial-statements TOC in an annual report); real data tables are untouched.

🤖 Generated with Claude Code

---

Summary by cubic

Fixes misdetected page-number-only tables of contents by flattening them into lists instead of grids, restoring reading order. Benchmarks/docs updated: NID 0.89 (rounded); TEDS unchanged.

• Bug Fixes
  • Added is_page_number_toc and wired it into is_table_of_contents.
  • Detection: 2–3 cols; ≥5 rows; last col is page numbers (short ints or canonical roman ≤8 chars) on ≥70% of filled rows; values mostly non-decreasing; first col has text; no header row (checks the first row’s last cell). Accept any page sequence with a gap; for perfectly dense runs, flatten only when first-col titles read like multi‑word headings, otherwise keep as a table.
  • Shared canonical_roman_value/to_roman_lower in tables::mod and use them in both detector and format::is_page_number_cell; formatter now recognizes canonical roman numerals for clean title/page splits.
  • Guards keep real tables intact (e.g., 4‑col regional grid; 2‑col Mineral | CEC with header).

^{Written for commit 5a2bad7. Summary will update on new commits.}

[cubic-dev-ai]

1 issue found across 1 file

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/tables/detect_heuristic.rs">

<violation number="1" location="src/tables/detect_heuristic.rs:930">
P3: Roman front-matter pages are detected as TOCs but not formatted as page numbers, so they lose the intended title/page separation. Extend `format::is_page_number_cell` with the same Roman-token recognition.</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P3: Roman front-matter pages are detected as TOCs but not formatted as page numbers, so they lose the intended title/page separation. Extend format::is_page_number_cell with the same Roman-token recognition.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/tables/detect_heuristic.rs, line 930:

<comment>Roman front-matter pages are detected as TOCs but not formatted as page numbers, so they lose the intended title/page separation. Extend `format::is_page_number_cell` with the same Roman-token recognition.</comment>

<file context>
@@ -914,7 +914,109 @@ fn looks_like_number(s: &str) -> bool {
+    if t.chars().all(|c| c.is_ascii_digit()) && t.len() <= 4 {
+        return t.parse().ok();
+    }
+    let lower = t.to_ascii_lowercase();
+    if !lower.is_empty() && lower.len() <= 6 && lower.chars().all(|c| "ivxlc".contains(c)) {
+        let mut total = 0i32;
</file context>

[abimaelmartell]

All four addressed:

1. Roman lookalike words (P2) — valid. page_number_value now re-encodes the parsed value to canonical roman and requires an exact match, so civil, mix, ill, lil are rejected while vii/ix/xii still parse. Test added.
2. Blank final header cell (P2) — valid. The no-header guard checked the first non-empty last cell, so Category | (blank header) slipped through. It now checks the actual first row's last cell; a blank/non-page-number header rejects the heuristic. Test added.
3. Roman pages not formatted (P3) — valid. format::is_page_number_cell now recognizes canonical roman numerals, so roman front-matter pages get the title\tpage separation in the flat list (doc 108 now renders About the Publisher\tvii).
4. Weak monotonicity test (P3) — valid. The test had 4 rows and exited on the < 5 row-count guard before monotonicity ran; bumped to 5 rows with a sanity assert that the earlier guards pass, so the non-decreasing check is what rejects it.

Benchmark unchanged (overall 0.834, NID 0.888, TEDS 0.656, MHS 0.742) — the fixes tighten detection without altering corpus outcomes. fmt/clippy/616 tests green.

[abimaelmartell]

@cubic-dev-ai

[cubic-dev-ai]

@cubic-dev-ai

@abimaelmartell I have started the AI code review. It will take a few minutes to complete.

[cubic-dev-ai]

All reported issues were addressed across 4 files

<sub>Reply with feedback, questions, or to request a fix.

Fix all with cubic | Re-trigger cubic</sub>

[abimaelmartell]

All three addressed:

1. Roman length ≤6 vs ≤8 (P2) — valid. Roman parsing is now a single shared canonical_roman_value (see below) that accepts ≤8 chars, so 7–8 char numerals like xxxviii flatten consistently on both the detector and formatter sides.

2. Monotonicity too weak for ordinal/rank/ID columns (P2) — valid. Added a page-span guard: real page numbers skip through a document, so their range far exceeds the entry count, whereas a dense consecutive sequence (1,2,3,…) has range ≈ length. is_page_number_toc now requires max − min > entry_count, which rejects rank/ordinal/ID data tables that monotonicity alone passed. New test page_number_toc_rejects_dense_ordinal_column.

3. Duplicated roman code (P2) — valid. Extracted canonical_roman_value + to_roman_lower into tables/mod.rs (pub(super)); both detect_heuristic.rs and format.rs now call it, removing the duplicated table/loop.

Cost of the stronger signal is ~0.001 aggregate (overall 0.834→0.833, NID 0.888→0.887) — a precision gain worth it, and still a clear win over baseline main (NID 0.883, TEDS unchanged at 0.656). Doc 108 still flattens; 617 tests + clippy green.

[cubic-dev-ai]

All reported issues were addressed across 3 files (changes from recent commits).

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[abimaelmartell]

Valid — fixed. The strict page-span rule did reject legitimate one-page-per-entry TOCs (range ≈ entry count), which is the ~0.001 it had cost.

Reworked is_page_number_toc's final check:

• Any page sequence with a gap (range > entry count) → TOC. Covers essentially all real contents.
• A perfectly dense consecutive run (1,2,3,… or 100,101,…) is the rank/ID/ordinal signature — but a chapter-per-page TOC can produce it too, so it falls back to the title signal you suggested: flatten when the first-column entries average ≥1.8 alphabetic words (multi-word headings), keep as a table when they're the short single-word labels typical of leaderboards/ID lists.

This recovers the consecutive-page TOC case (NID back to 0.888, overall 0.834) while still rejecting dense ordinal data tables (page_number_toc_rejects_dense_ordinal_column — single-word labels — still passes). Added page_number_toc_matches_consecutive_pages_with_titles. 618 tests + clippy green; TEDS unchanged at 0.656.

[cubic-dev-ai]

1 issue found across 1 file (changes from recent commits).

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="src/tables/detect_heuristic.rs">

<violation number="1" location="src/tables/detect_heuristic.rs:1017">
P1: Headerless rank/ID tables with a skipped or repeated value now flatten even when labels are ordinary multi-word names. `!dense_consecutive` treats those common data-table sequences as contents without applying the new title signal; retain a TOC-specific guard on this path instead of accepting every imperfect sequence.</violation>
</file>

<sub>Tip: Review your code locally with the cubic CLI to iterate faster.

Fix all with cubic | Re-trigger cubic</sub>

[cubic-dev-ai]

P1: Headerless rank/ID tables with a skipped or repeated value now flatten even when labels are ordinary multi-word names. !dense_consecutive treats those common data-table sequences as contents without applying the new title signal; retain a TOC-specific guard on this path instead of accepting every imperfect sequence.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At src/tables/detect_heuristic.rs, line 1017:

<comment>Headerless rank/ID tables with a skipped or repeated value now flatten even when labels are ordinary multi-word names. `!dense_consecutive` treats those common data-table sequences as contents without applying the new title signal; retain a TOC-specific guard on this path instead of accepting every imperfect sequence.</comment>

<file context>
@@ -996,14 +996,44 @@ pub(super) fn is_page_number_toc(cells: &[Vec<String>]) -> bool {
+        sorted.dedup();
+        sorted.len() == page_vals.len()
+    };
+    if !dense_consecutive {
+        // Narrow range but with a gap or repeat — still contents-like.
+        return true;
</file context>