Skip to content

idx=12: freeze (412 records) — Triton Container Ninth Restated Improve IWW detection in signature-page explosion logicedit Agreement (widened IWW + ancestor up-walk) #85

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
arthrod wants to merge 1 commit into
base: redo/idx-11
Choose a base branch
from
Open

idx=12: freeze (412 records) — Triton Container Ninth Restated Improve IWW detection in signature-page explosion logicedit Agreement (widened IWW + ancestor up-walk) #85

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
412 changes: 412 additions & 0 deletions data/auto_parse/level_freeze/frozen/idx_12.jsonl
View file
Open in desktop

Large diffs are not rendered by default.

9 changes: 8 additions & 1 deletion data/auto_parse/level_freeze/state.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@
8,
9,
10,
11
11,
12
],
"history": [
{
Expand Down Expand Up @@ -199,6 +200,12 @@
"action": "freeze",
"idx": 11,
"n_records": 90
},
{
"ts": "2026-05-17T08:44:03",
"action": "freeze",
"idx": 12,
"n_records": 412
}
]
}
83 changes: 72 additions & 11 deletions scripts/parse_doc2dict_with_config.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4195,11 +4195,20 @@ def _chain_ancestor_node_ids(node: dict[str, Any]) -> set[int]:
actives.sort(key=lambda r: r["node_id"])

# Pin the IWW operating clause depth=1.
#
# We detect IWW in either the combined span (title+body) OR in the
# body alone. The body-alone check handles records where doc2dict
# placed page-chrome filler text in the title (e.g. "[Remainder of
# page intentionally left blank]") and the IWW operating sentence
# sits in the body. The combined-span check (`_is_iww_clause(span)`)
# only fires when IWW is at the start of the span, so a non-IWW
# title prefix would otherwise hide the IWW anchor.
iww_present = False
iww_carriers: list[dict[str, Any]] = []
for r in actives:
span = _span_text(r)
if _is_iww_clause(span):
body = (r.get("body_direct") or "").strip()
if _is_iww_clause(span) or _is_iww_clause(body):
r["depth"] = 1 + (r.get("subdoc_penalty") or 0)
Comment on lines +4210 to 4212

@coderabbitai coderabbitai Bot May 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Apply span-or-body IWW guard in the /s/ PASS-3 branch too.

This change widens IWW detection here, but Line 4535 still uses span-only exclusion. In documents where IWW is in body_direct and title has filler text, the /s/ branch can still demote the IWW carrier to L2.

Suggested fix 
-        if _is_iww_clause(_span_text(r)):
+        r_body = (r.get("body_direct") or "").strip()
+        if _is_iww_clause(_span_text(r)) or _is_iww_clause(r_body):
             continue
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/parse_doc2dict_with_config.py` around lines 4210 - 4212, In the /s/
PASS-3 branch currently using a span-only IWW check, update the conditional to
use the same span-or-body guard as earlier: replace uses of _is_iww_clause(span)
with (_is_iww_clause(span) or _is_iww_clause(body)) (reusing the existing body =
(r.get("body_direct") or "").strip() value) so that the branch that sets
r["depth"] for PASS-3 respects IWW found in body_direct as well; keep the rest
of the branch logic (r["depth"] assignments and subdoc_penalty) unchanged.

iww_present = True
iww_carriers.append(r)
Expand Down Expand Up @@ -4247,10 +4256,59 @@ def _walk_descendants_local(nid: int) -> list[dict[str, Any]]:
return out

# Set of parent_node_ids to consider — every parent that contains
# an IWW record as a child.
sig_area_parent_ids: set[int | None] = {
iww.get("parent_node_id") for iww in iww_carriers
}
# an IWW record as a child, plus an UP-walk extension when none of
# the IWW carrier's immediate siblings are sig-shape (e.g. the IWW
# was packed inside an all-caps continuation node that is itself a
# sibling of the real sig parties).
sig_area_parent_ids: set[int | None] = set()
for iww in iww_carriers:
cur_pid = iww.get("parent_node_id")
iww_seen: set[int | None] = set()
walked = 0
while cur_pid is not None and cur_pid not in iww_seen and walked < 4:
Comment on lines +4263 to +4268

@gemini-code-assist gemini-code-assist Bot May 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

high

The initialization of sig_area_parent_ids as an empty set, combined with the while cur_pid is not None condition, introduces a regression where root-level IWW carriers (those with parent_node_id=None) no longer have their siblings checked for signature shapes. The original implementation correctly included None in the set of parent IDs to consider.

To fix this, initialize sig_area_parent_ids with the immediate parents of all IWW carriers, which restores the original behavior for root nodes, and then perform the up-walk for non-root parents.

Suggested change
sig_area_parent_ids: set[int | None] = set()
for iww in iww_carriers:
cur_pid = iww.get("parent_node_id")
iww_seen: set[int | None] = set()
walked = 0
while cur_pid is not None and cur_pid not in iww_seen and walked < 4:
sig_area_parent_ids: set[int | None] = {
iww.get("parent_node_id") for iww in iww_carriers
}
for iww in iww_carriers:
cur_pid = iww.get("parent_node_id")
if cur_pid is None:
continue
iww_seen: set[int | None] = set()
walked = 0
while cur_pid is not None and cur_pid not in iww_seen and walked < 4:

iww_seen.add(cur_pid)
parent_rec = by_node_id.get(cur_pid)
Comment on lines +4265 to +4270

@coderabbitai coderabbitai Bot May 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major | ⚡ Quick win

Handle root-level IWW carriers in the up-walk seed.

If an IWW carrier has parent_node_id=None, the current loop never adds a search base, so root-level sig-shape siblings are never discovered in the IWW-only (no /s/) path.

Suggested fix 
         sig_area_parent_ids: set[int | None] = set()
         for iww in iww_carriers:
             cur_pid = iww.get("parent_node_id")
+            if cur_pid is None:
+                # Root-level IWW: scan root siblings as signature-area candidates.
+                sig_area_parent_ids.add(None)
+                continue
             iww_seen: set[int | None] = set()
             walked = 0
             while cur_pid is not None and cur_pid not in iww_seen and walked < 4:
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
cur_pid = iww.get("parent_node_id")
iww_seen: set[int | None] = set()
walked = 0
while cur_pid is not None and cur_pid not in iww_seen and walked < 4:
iww_seen.add(cur_pid)
parent_rec = by_node_id.get(cur_pid)
cur_pid = iww.get("parent_node_id")
if cur_pid is None:
# Root-level IWW: scan root siblings as signature-area candidates.
sig_area_parent_ids.add(None)
continue
iww_seen: set[int | None] = set()
walked = 0
while cur_pid is not None and cur_pid not in iww_seen and walked < 4:
iww_seen.add(cur_pid)
parent_rec = by_node_id.get(cur_pid)
🤖 Prompt for AI Agents 
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@scripts/parse_doc2dict_with_config.py` around lines 4265 - 4270, The up-walk
loop starting from cur_pid = iww.get("parent_node_id") skips adding a search
base when parent_node_id is None, so root-level IWW carriers never seed sibling
discovery; fix by detecting if iww.get("parent_node_id") is None before the
while and explicitly add the IWW's node id (e.g., iww.get("node_id")) or an
appropriate root search key into the up-walk seed collection so root-level
sig-shape siblings are included, then proceed with the existing while using
cur_pid, iww_seen, walked and by_node_id as before.

# Always include this parent as a search base.
sig_area_parent_ids.add(cur_pid)
# Stop the up-walk at the L0 agreement title or at a
# section-marker ancestor (a real agreement clause). The
# title-as-root rubric forbids demoting top-level body
# clauses to sig-line depth.
if parent_rec is None:
break
if parent_rec.get("depth") == 0:
break
if _has_section_marker_title(parent_rec):
break
# If the current parent already has sig-shape siblings of
# the IWW carrier visible at this level, stop — no need
# to walk further. Sig-shape detection here mirrors the
# sibling test below.
has_sig_sib = False
for sib in children_of_local.get(cur_pid, []):
if sib.get("node_id") == iww.get("node_id"):
continue
if sib.get("is_envelope") or sib.get("scope") == "trailer":
continue
if _has_section_marker_title(sib):
continue
s_body = (sib.get("body_direct") or "").strip()
if _is_iww_clause(_span_text(sib)) or _is_iww_clause(s_body):
continue
s_title = (sib.get("title") or "").strip()
if (
_SIG_FIELD_RE.match(s_title)
or _SIG_FIELD_RE.match(s_body)
or (s_title and _SIG_BLOCK_LABEL_RE.match(s_title))
or (s_title and _CORP_SUFFIX_LABEL_RE.match(s_title))
or (not s_title and s_body and _SIG_FIELD_RE.match(s_body))
):
Comment on lines +4299 to +4305

@gemini-code-assist gemini-code-assist Bot May 17, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

medium

The logic for detecting signature-shaped records is now duplicated in multiple places within this function (the up-walk, the sibling loop, and the descendant loop). This increases the risk of inconsistencies if the signature detection rules need to be updated. Consider extracting this logic into a local helper function to improve maintainability.

has_sig_sib = True
break
if has_sig_sib:
break
cur_pid = parent_rec.get("parent_node_id")
walked += 1
for parent_id in sig_area_parent_ids:
siblings = children_of_local.get(parent_id, [])
for sib in siblings:
Expand All @@ -4260,11 +4318,13 @@ def _walk_descendants_local(nid: int) -> list[dict[str, Any]]:
# A numbered/lettered section is agreement content
# (a top body clause), not a sig fragment.
continue
# Skip the IWW record itself — it stays L1.
if _is_iww_clause(_span_text(sib)):
# Skip the IWW record itself — it stays L1. Detect IWW
# in the combined span OR in the body alone (matches the
# PASS-2 IWW carrier detection above).
sib_body = (sib.get("body_direct") or "").strip()
if _is_iww_clause(_span_text(sib)) or _is_iww_clause(sib_body):
continue
sib_title = (sib.get("title") or "").strip()
sib_body = (sib.get("body_direct") or "").strip()
looks_sig = (
_SIG_FIELD_RE.match(sib_title)
or _SIG_FIELD_RE.match(sib_body)
Expand All @@ -4282,10 +4342,10 @@ def _walk_descendants_local(nid: int) -> list[dict[str, Any]]:
continue
if _has_section_marker_title(d):
continue
if _is_iww_clause(_span_text(d)):
d_body = (d.get("body_direct") or "").strip()
if _is_iww_clause(_span_text(d)) or _is_iww_clause(d_body):
continue
d_title = (d.get("title") or "").strip()
d_body = (d.get("body_direct") or "").strip()
d_looks_sig = (
_SIG_FIELD_RE.match(d_title)
or _SIG_FIELD_RE.match(d_body)
Expand All @@ -4307,7 +4367,8 @@ def _walk_descendants_local(nid: int) -> list[dict[str, Any]]:
continue
if r.get("depth") == 0:
continue
if _is_iww_clause(_span_text(r)):
r_body = (r.get("body_direct") or "").strip()
if _is_iww_clause(_span_text(r)) or _is_iww_clause(r_body):
continue
l2_depth = 2 + (r.get("subdoc_penalty") or 0)
r["depth"] = l2_depth
Expand Down