From 721dd85c5b6ae25515fecf5ddd03d2aa15adffd1 Mon Sep 17 00:00:00 2001 From: Arthur Souza Rodrigues Date: Sun, 17 May 2026 08:19:25 -0400 Subject: [PATCH] =?UTF-8?q?idx=3D11:=20freeze=20(90=20records)=20=E2=80=94?= =?UTF-8?q?=20Extraction=20Oil=20&=20Gas=20Amendment=20No.=2011=20to=20Cre?= =?UTF-8?q?dit=20Agreement:=20IWW-less=20sig-page=20explosion=20+=20all-ca?= =?UTF-8?q?ps=20body-paragraph=20depth=20demotion?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two SHAPE-based parser additions to handle credit-agreement amendment idx=11: 1. `_split_dense_sig_body_no_iww`: counterpart to the existing IWW splitter for agreements that use an alternative sig-page operating phrase (e.g. "EXECUTED as of the date first set forth above.") instead of the canonical "IN WITNESS WHEREOF". Triggers only when no IWW anchor exists in the doc AND a body carries ≥3 `/s/` marks (a structurally dense sig page). Splits at the last sentence-ending period before the first sig-shape line, emits the operating clause as L1 and each sig-page line as L2. 2. `_demote_deeply_nested_body_paragraphs`: demotes all-caps body paragraphs that doc2dict mis-classified as deep predicted headers (HTML container nesting reflected in the depth, not structural hierarchy). Shape: cls=predicted header, depth ≥ 3, empty body, title > 60 chars, all-uppercase letters, no section-marker or structural-level pattern match. Re-set depth to 1 + subdoc_penalty. Reconstruction: word_coverage=94.9%, char_ratio=80.9% (≥ 90% bar). All 12 idxs (0..11) regress OK. --- .../level_freeze/frozen/idx_11.jsonl | 90 ++++++ data/auto_parse/level_freeze/state.json | 9 +- scripts/parse_doc2dict_with_config.py | 282 ++++++++++++++++++ 3 files changed, 380 insertions(+), 1 deletion(-) create mode 100644 data/auto_parse/level_freeze/frozen/idx_11.jsonl diff --git a/data/auto_parse/level_freeze/frozen/idx_11.jsonl b/data/auto_parse/level_freeze/frozen/idx_11.jsonl new file mode 100644 index 0000000..2eba139 --- /dev/null +++ b/data/auto_parse/level_freeze/frozen/idx_11.jsonl @@ -0,0 +1,90 @@ +{"idx": 11, "order": 0, "level": 0, "span": "AMENDMENT NO. 11 TO CREDIT AGREEMENT"} +{"idx": 11, "order": 1, "level": 1, "span": "This Amendment No. 11 to Credit Agreement (this “Agreement”) dated as of March 15, 2017 (the “Effective Date”), is among Extraction Oil & Gas, Inc., a Delaware corporation (the “Borrower”), 7N, LLC, a Delaware limited liability company, 8 North, LLC, a Delaware limited liability company, Bison Exploration, LLC, a Delaware limited liability company, Elevation Midstream, LLC, a Delaware limited liability company, Extraction Finance Corp., a Delaware corporation, Mountaintop Minerals, LLC, a Delaware limited liability company, XOG Services, LLC, a Delaware limited liability company, XOG Services, Inc., a Colorado corporation, and XTR Midstream, LLC, a Delaware limited liability company (“XTR”; and together with the other entities listed subsequent to the Borrower above, collectively, the “Guarantors”), the undersigned Lenders (as defined below), and Wells Fargo Bank, National Association, as Administrative Agent for the Lenders (in such capacity, the “Administrative Agent”) and as Issuing Lender (the “Issuing Lender”)."} +{"idx": 11, "order": 2, "level": 1, "span": "INTRODUCTION\nA.         The Borrower, the financial institutions party thereto as lenders (the “Lenders”), the Issuing Lender, and the Administrative Agent have entered into the Credit Agreement dated as of September 4, 2014, as amended by the Amendment No. 1 dated as of September 24, 2014, the Amendment No. 2 and Joinder dated as of November 10, 2014, the Amendment No. 3 dated as of December 30, 2014, the Waiver dated as of February 12, 2015, the Consent Agreement dated as of February 27, 2015, the Consent Agreement dated as of March 25, 2015, the Waiver dated as of April 28, 2015, the Amendment No. 4 and Joinder dated as of May 27, 2015, the Amendment No. 5 dated as of September 1, 2015, the Amendment No. 6 dated as of September 10, 2015, the Amendment No. 7 and Joinder dated as of December 15, 2015, the Amendment No. 8 and Joinder dated as of June 13, 2016, the Amendment No. 9 dated as of August 12, 2016, and the Consent, Amendment No. 10 and Joinder dated September 14, 2016 (as so amended and modified and as may be otherwise amended, restated, or modified from time to time, the “Credit Agreement”).\nB.         The Guarantors have entered into the Guaranty Agreement dated as of September 4, 2014 (as amended, restated, supplemented or otherwise modified from time to time, the “Guaranty”) in favor of the Administrative Agent for the benefit of the Secured Parties (as defined in the Credit Agreement).\nC.         XTR desires to contribute cash and certain midstream assets in exchange for approximately 15% of the Equity Interest (as defined in the Credit Agreement) of Platte River Holdings LLC, a Delaware limited liability company (“PRH”), pursuant to that certain Contribution Agreement to be entered into on or about the Effective Date, among ARB Platte River, LLC, a Colorado limited liability company (“ARB”), XTR and PRH (the “Proposed Contribution”).\nD.         Contemporaneously with the Proposed Contribution, (i) ARB and XTR will enter into that certain First Amended and Restated Limited Liability Company Agreement of PRH to be entered into on or about the Effective Date and (ii) Platte River Midstream, LLC, a Delaware\nlimited liability company and wholly-owned subsidiary of PRH (“PRM”), and the Borrower will enter into that certain First Amended and Restated Transportation Services Agreement to be entered into on or about the Effective Date, pursuant to which the Borrower will agree to ship or pay to ship certain committed volumes of hydrocarbons on midstream infrastructure owned and operated by PRM (the “Proposed Shipping Arrangement”).\nE.         The Borrower has requested that the Lenders and the Administrative Agent, and the Administrative Agent and the Lenders have agreed, subject to the terms and conditions hereof, amend the Credit Agreement as set forth herein to permit each of the Proposed Contribution and the Proposed Shipping Arrangement.\nTHEREFORE, in fulfillment of the foregoing, the Borrower, the Guarantors, the Administrative Agent, the Issuing Lender, and the undersigned Lenders hereby agree as follows:\nSection 1.         Definitions; References.  Unless otherwise defined in this Agreement, each term used in this Agreement which is defined in the Credit Agreement has the meaning assigned to such term in the Credit Agreement, as amended hereby.\nSection 2.         Amendments to Credit Agreement.  Upon the satisfaction of the conditions specified in Section 6 of this Agreement, and effective as of the date set forth above, the Credit Agreement is amended as follows:"} +{"idx": 11, "order": 3, "level": 1, "span": "“Approved Transportation Agreements” means the Grand Mesa Agreements, the Tallgrass Letter Agreement, the PRM Transportation Agreement and such other transportation services agreements as may be approved by the Majority Lenders in writing, in each case, together with such changes thereto as may be approved by the Administrative Agent."} +{"idx": 11, "order": 4, "level": 2, "span": "(c)         Section 6.3 of the Credit Agreement (Investments) is amended by deleting the “and” at the end of clause (f) thereof and replacing clause (g) thereof with the following clause (g) and new clause (h):"} +{"idx": 11, "order": 5, "level": 2, "span": "(g)         the investment made by XTR in PRH pursuant to the PRH Contribution Agreement and the PRH LLC Agreement so long as (i) the aggregate amount of cash contributed by XTR to PRH pursuant thereto does not exceed $5,000,000 and (ii) the amount of cash and the fair market value of the assets contributed by XTR to PRH pursuant thereto does not exceed $10,000,000 in the aggregate; and"} +{"idx": 11, "order": 6, "level": 2, "span": "(h)         other investments in an aggregate amount not to exceed $5,000,000."} +{"idx": 11, "order": 7, "level": 2, "span": "(d)         Section 6.8 of the Credit Agreement (Sale of Assets) is amended by replacing clause (h) thereof in its entirety with the following:"} +{"idx": 11, "order": 8, "level": 2, "span": "(h)          (i) Permitted Investments of the type described in Section 6.3(g) and (ii) other Asset Sales of Property not constituting Oil and Gas Properties and not otherwise permitted by this Section 6.8, the aggregate consideration of which shall not exceed $5,000,000 during the term of this Agreement; and"} +{"idx": 11, "order": 9, "level": 2, "span": "(e)         Article 6 of the Credit Agreement (Negative Covenants) is further amended by adding the following new Section 6.27 to the end thereof:"} +{"idx": 11, "order": 10, "level": 2, "span": "6.27        PRH and PRM.  Notwithstanding anything to the contrary contained herein, no Loan Party shall, nor shall it permit any of its Subsidiaries to, create, assume, incur or suffer to exist any Lien on or in respect of any of its Property for the benefit of PRH or PRM.\nSection 3.         Reaffirmation of Liens.\nDocuments, (ii) represents and warrants that it has no defenses to the enforcement of the Security Documents and that according to their terms the Security Documents will continue in full force and effect to secure the Borrower’s and Guarantors’ obligations under the Loan Documents, as the same may be amended, supplemented, or otherwise modified, and (iii) acknowledges, represents, and warrants that the liens and security interests created by the Security Documents are valid and subsisting and create a first and prior Lien (subject only to Permitted Liens) in the Collateral to secure the Secured Obligations.\nSection 4.         Reaffirmation of Guaranty.  Each Guarantor hereby ratifies, confirms, and acknowledges that its obligations under the Guaranty and the other Loan Documents are in full force and effect and that such Guarantor continues to unconditionally and irrevocably guarantee the full and punctual payment, when due, whether at stated maturity or earlier by acceleration or otherwise, of all of the Guaranteed Obligations (as defined in the Guaranty), as such Guaranteed Obligations may have been amended by this Agreement.  Each Guarantor hereby acknowledges that its execution and delivery of this Agreement does not indicate or establish an approval or consent requirement by such Guarantor under the Credit Agreement in connection with the execution and delivery of amendments, modifications or waivers to the Credit Agreement, the Notes or any of the other Loan Documents.\nSection 5.         Representations and Warranties.  Each of the Borrower and each Guarantor represents and warrants to the Administrative Agent and the Lenders that:\nSection 6.         Effectiveness.  This Agreement shall become effective as of the date hereof upon the occurrence of all of the following:\nSection 7.         Post-Closing Obligations.  On or before 5:00 p.m. (Houston, Texas time) on the effective date of the Proposed Contribution, the Borrower shall deliver to the Administrative Agent certified, fully executed, correct and complete copies of the PRH Contribution Agreement, the PRH LLC Agreement, and the PRM Transportation Agreement, in each case, as in effect on the Effective Date.  The Borrower's failure to satisfy the obligations set forth in this Section 7 shall constitute an immediate Event of Default under this Agreement and the Credit Agreement.\nSection 8.         Effect on Loan Documents.  Except as amended herein, the Credit Agreement and the Loan Documents remain in full force and effect as originally executed and are hereby ratified and confirmed, and nothing herein shall act as a waiver of any of the Administrative Agent's or Lenders' rights under the Loan Documents.  This Agreement is a Loan Document for the purposes of the provisions of the other Loan Documents.  Without limiting the foregoing, any breach of representations, warranties, and covenants under this Agreement is a Default or Event of Default under other Loan Documents.\nSection 9.         Choice of Law.  This Agreement shall be governed by and construed and enforced in accordance with the laws of the State of New York without regard to conflicts of laws principles (other than Sections 5-1401 and 5-1402 of the General Obligations Law of the State of New York).\nSection 10.         Counterparts.  This Agreement may be signed in any number of counterparts, each of which shall be an original."} +{"idx": 11, "order": 11, "level": 2, "span": "(a)         Each of the Borrower and each Guarantor (i) is party to certain Security Documents securing and supporting the Borrower's and Guarantors’ obligations under the Loan"} +{"idx": 11, "order": 12, "level": 2, "span": "(b)         The delivery of this Agreement does not indicate or establish a requirement that any Loan Document requires any Guarantor's approval of amendments to the Credit Agreement."} +{"idx": 11, "order": 13, "level": 2, "span": "(a)         the representations and warranties set forth in the Credit Agreement and in the other Loan Documents are true and correct in all material respects as of the date of this Agreement (except to the extent such representations and warranties relate to an earlier date, in which case such representations and warranties shall be true and correct in all material respects as of such earlier date); provided that such materiality qualifier shall not apply if such representation or warranty is already subject to a materiality qualifier in the Credit Agreement or such other Loan Document;"} +{"idx": 11, "order": 14, "level": 2, "span": "(b)          (i) the execution, delivery, and performance of this Agreement are within the corporate, limited partnership or limited liability company power, as appropriate, and authority of the Borrower and Guarantors and have been duly authorized by appropriate proceedings and (ii) this Agreement constitutes a legal, valid, and binding obligation of the Borrower and Guarantors, enforceable against the Borrower and Guarantors in accordance with its terms, except as limited by applicable bankruptcy, insolvency, reorganization, moratorium, or similar laws affecting the rights of creditors generally and general principles of equity; and"} +{"idx": 11, "order": 15, "level": 2, "span": "(c)         as of the effectiveness of this Agreement and after giving effect thereto, no Default or Event of Default has occurred and is continuing."} +{"idx": 11, "order": 16, "level": 2, "span": "(a)         Documentation\nThe Administrative Agent shall have received this Agreement, duly and validly executed by the Borrower, the Guarantors, the Administrative Agent, the Issuing Bank and the Majority Lenders, in form and substance reasonably satisfactory to the Administrative Agent and the Majority Lenders;"} +{"idx": 11, "order": 17, "level": 2, "span": "(b)         Representations and Warranties\nThe representations and warranties in this Agreement being true and correct in all material respects before and after giving effect to this Agreement (except to the extent such representations and warranties relate to an earlier date, in which case such representations and warranties shall be true and correct in all material respects as of such earlier date); provided that such materiality qualifier shall not apply if such representation or warranty is already subject to a materiality qualifier in the Credit Agreement or such other Loan Document."} +{"idx": 11, "order": 18, "level": 2, "span": "(c)         No Default or Event of Default\nThere being no Default or Event of Default which has occurred and is continuing."} +{"idx": 11, "order": 19, "level": 2, "span": "(d)         Expenses\nThe Borrower’s having paid all costs, expenses, and fees which have been invoiced and are payable pursuant to Section 9.1 of the Credit Agreement or any other agreement."} +{"idx": 11, "order": 20, "level": 1, "span": "THIS WRITTEN AGREEMENT AND THE LOAN DOCUMENTS, AS DEFINED IN THE CREDIT AGREEMENT, REPRESENT THE FINAL AGREEMENT AMONG THE PARTIES AND MAY NOT BE CONTRADICTED BY EVIDENCE OF PRIOR, CONTEMPORANEOUS, OR SUBSEQUENT ORAL AGREEMENTS"} +{"idx": 11, "order": 21, "level": 1, "span": "OF THE PARTIES.  THERE ARE NO UNWRITTEN ORAL AGREEMENTS BETWEEN THE PARTIES."} +{"idx": 11, "order": 22, "level": 1, "span": "[Remainder of page intentionally left blank; Signature pages follow.] EXECUTED as of the date first set forth above."} +{"idx": 11, "order": 23, "level": 2, "span": "BORROWER:"} +{"idx": 11, "order": 24, "level": 2, "span": "EXTRACTION OIL & GAS, INC."} +{"idx": 11, "order": 25, "level": 2, "span": "By:"} +{"idx": 11, "order": 26, "level": 2, "span": "/s/ Matthew R. Owens"} +{"idx": 11, "order": 27, "level": 2, "span": "Name:"} +{"idx": 11, "order": 28, "level": 2, "span": "Matthew R Owens"} +{"idx": 11, "order": 29, "level": 2, "span": "Title:"} +{"idx": 11, "order": 30, "level": 2, "span": "President"} +{"idx": 11, "order": 31, "level": 2, "span": "GUARANTORS:"} +{"idx": 11, "order": 32, "level": 2, "span": "7N, LLC"} +{"idx": 11, "order": 33, "level": 2, "span": "8 NORTH, LLC"} +{"idx": 11, "order": 34, "level": 2, "span": "BISON EXPLORATION, LLC"} +{"idx": 11, "order": 35, "level": 2, "span": "ELEVATION MIDSTREAM, LLC"} +{"idx": 11, "order": 36, "level": 2, "span": "EXTRACTION FINANCE CORP."} +{"idx": 11, "order": 37, "level": 2, "span": "MOUNTAINTOP MINERALS, LLC"} +{"idx": 11, "order": 38, "level": 2, "span": "XOG SERVICES, INC."} +{"idx": 11, "order": 39, "level": 2, "span": "XOG SERVICES, LLC"} +{"idx": 11, "order": 40, "level": 2, "span": "XTR MIDSTREAM, LLC"} +{"idx": 11, "order": 41, "level": 2, "span": "Each By:"} +{"idx": 11, "order": 42, "level": 2, "span": "ADMINISTRATIVE AGENT/ISSUING LENDER/LENDER:"} +{"idx": 11, "order": 43, "level": 2, "span": "WELLS FARGO BANK, NATIONAL ASSOCIATION,"} +{"idx": 11, "order": 44, "level": 2, "span": "As Administrative Agent, Issuing Lender, and a Lender"} +{"idx": 11, "order": 45, "level": 2, "span": "/s/ Zachary Kramer"} +{"idx": 11, "order": 46, "level": 2, "span": "Zachary Kramer"} +{"idx": 11, "order": 47, "level": 2, "span": "Assistant Vice President"} +{"idx": 11, "order": 48, "level": 2, "span": "LENDERS:"} +{"idx": 11, "order": 49, "level": 2, "span": "ROYAL BANK OF CANADA"} +{"idx": 11, "order": 50, "level": 2, "span": "as a Lender"} +{"idx": 11, "order": 51, "level": 2, "span": "/s/ Kristan Spivey"} +{"idx": 11, "order": 52, "level": 2, "span": "Kristan Spivey"} +{"idx": 11, "order": 53, "level": 2, "span": "Authorized Signatory"} +{"idx": 11, "order": 54, "level": 2, "span": "BOKF, NA, dba Bank of Oklahoma,"} +{"idx": 11, "order": 55, "level": 2, "span": "/s/ Benjamin H. Adler"} +{"idx": 11, "order": 56, "level": 2, "span": "Benjamin H. Adler"} +{"idx": 11, "order": 57, "level": 2, "span": "Vice President"} +{"idx": 11, "order": 58, "level": 2, "span": "GOLDMAN SACHS BANK USA,"} +{"idx": 11, "order": 59, "level": 2, "span": "/s/ Ushma Dedhiya"} +{"idx": 11, "order": 60, "level": 2, "span": "Ushma Dedhiya"} +{"idx": 11, "order": 61, "level": 2, "span": "FIFTH THIRD BANK,"} +{"idx": 11, "order": 62, "level": 2, "span": "/s/ Jonathan H. Lee"} +{"idx": 11, "order": 63, "level": 2, "span": "Jonathan H. Lee"} +{"idx": 11, "order": 64, "level": 2, "span": "Director"} +{"idx": 11, "order": 65, "level": 2, "span": "SUNTRUST BANK,"} +{"idx": 11, "order": 66, "level": 2, "span": "/s/ Arize Agumadu"} +{"idx": 11, "order": 67, "level": 2, "span": "Arize Agumadu"} +{"idx": 11, "order": 68, "level": 2, "span": "KEYBANK NATIONAL ASSOCIATION,"} +{"idx": 11, "order": 69, "level": 2, "span": "/s/ Paul Pace"} +{"idx": 11, "order": 70, "level": 2, "span": "Paul Pace"} +{"idx": 11, "order": 71, "level": 2, "span": "Senior Vice President"} +{"idx": 11, "order": 72, "level": 2, "span": "BARCLAYS BANK PLC,"} +{"idx": 11, "order": 73, "level": 2, "span": "/s/ Graeme Palmer"} +{"idx": 11, "order": 74, "level": 2, "span": "Graeme Palmer"} +{"idx": 11, "order": 75, "level": 2, "span": "ABN AMRO CAPITAL USA LLC,"} +{"idx": 11, "order": 76, "level": 2, "span": "/s/ Darrell Holley"} +{"idx": 11, "order": 77, "level": 2, "span": "Darrell Holley"} +{"idx": 11, "order": 78, "level": 2, "span": "Managing Director"} +{"idx": 11, "order": 79, "level": 2, "span": "/s/ Michaela Braun"} +{"idx": 11, "order": 80, "level": 2, "span": "Michaela Braun"} +{"idx": 11, "order": 81, "level": 2, "span": "CREDIT SUISSE AG,"} +{"idx": 11, "order": 82, "level": 2, "span": "CAYMAN ISLANDS BRANCH,"} +{"idx": 11, "order": 83, "level": 2, "span": "/s/ Nupur Kumar"} +{"idx": 11, "order": 84, "level": 2, "span": "Nupur Kumar"} +{"idx": 11, "order": 85, "level": 2, "span": "/s/ Lea Baerlocher"} +{"idx": 11, "order": 86, "level": 2, "span": "Lea Baerlocher"} +{"idx": 11, "order": 87, "level": 2, "span": "CITIBANK, N.A.,"} +{"idx": 11, "order": 88, "level": 2, "span": "/s/ Phil Ballard"} +{"idx": 11, "order": 89, "level": 2, "span": "Phil Ballard"} diff --git a/data/auto_parse/level_freeze/state.json b/data/auto_parse/level_freeze/state.json index e8c94a9..c3d993f 100644 --- a/data/auto_parse/level_freeze/state.json +++ b/data/auto_parse/level_freeze/state.json @@ -11,7 +11,8 @@ 7, 8, 9, - 10 + 10, + 11 ], "history": [ { @@ -192,6 +193,12 @@ "action": "freeze", "idx": 10, "n_records": 6 + }, + { + "ts": "2026-05-17T08:18:41", + "action": "freeze", + "idx": 11, + "n_records": 90 } ] } diff --git a/scripts/parse_doc2dict_with_config.py b/scripts/parse_doc2dict_with_config.py index eba3083..b249a5a 100644 --- a/scripts/parse_doc2dict_with_config.py +++ b/scripts/parse_doc2dict_with_config.py @@ -3343,6 +3343,73 @@ def _strip_page_footer_exhibit_titles(rows: list[dict[str, Any]]) -> list[dict[s return rows +def _demote_deeply_nested_body_paragraphs( + rows: list[dict[str, Any]], +) -> list[dict[str, Any]]: + """Demote deeply-nested all-caps paragraph records to L1. + + doc2dict computes ``depth`` from HTML nesting (``
`` → ``

`` → + ````…). When the source typesets a body paragraph in bold/all- + caps for legal emphasis (statutory disclaimers, no-oral-amendments + notices), the HTML wraps it in extra container tags and doc2dict's + walker assigns a depth that reflects the markup, not the structural + hierarchy. + + The rubric's depth contract is structural (title → top-level body + clause → subsection), not HTML-tree. A long all-caps paragraph that + is not a section header sits at L1 alongside its numbered/lettered + sibling clauses — same depth as a numbered Section. + + Shape detection (no phrase matching): + - cls is "predicted header" (doc2dict classified as a header but + the title text is paragraph-shaped). + - depth ≥ 3 (deeper than parent + 1 for a plain body paragraph). + - title is non-empty, has no body, and matches no section marker + (numbered/lettered/roman) and no ``_STRUCTURAL_LEVELS`` pattern. + - title length > 60 chars (real headings are short; only body + paragraphs run this long). + - title is entirely uppercase (after stripping whitespace and + punctuation) — the structural cue that the source typeset this + as a legal-emphasis paragraph, not a normal heading. + + Effect: re-set depth to ``1 + subdoc_penalty`` so the paragraph + emits as an L1 sibling of the surrounding sections (or as an L1+ + descendant inside any enclosing subdocs). + """ + for r in rows: + if r.get("is_envelope") or r.get("scope") == "trailer": + continue + if (r.get("cls") or "") != "predicted header": + continue + depth = r.get("depth") or 0 + subdoc_penalty = r.get("subdoc_penalty") or 0 + if depth < 3 + subdoc_penalty: + continue + title = (r.get("title") or "").strip() + body = (r.get("body_direct") or "").strip() + if not title or body: + continue + if len(title) <= 60: + continue + # Skip if any section-marker or structural pattern matches. + if _infer_level_from_title(title) is not None: + continue + structural_matched = False + for pat, _lvl in _STRUCTURAL_LEVELS: + if pat.match(title): + structural_matched = True + break + if structural_matched: + continue + # All-uppercase shape: every letter character (after the + # punctuation pass) is uppercase. Allow numerics and punctuation. + letters = [c for c in title if c.isalpha()] + if not letters or not all(c.isupper() for c in letters): + continue + r["depth"] = 1 + subdoc_penalty + return rows + + def _drop_pre_title_position_records(rows: list[dict[str, Any]]) -> list[dict[str, Any]]: """Drop in-scope records that appear BEFORE the L0 agreement title in sorted document-order. @@ -3822,6 +3889,201 @@ def _split_iww_and_sig_from_body( return new_rows +def _split_dense_sig_body_no_iww( + rows: list[dict[str, Any]], +) -> list[dict[str, Any]]: + """Extract a signature-page block from a body when no IWW is present. + + Counterpart to ``_split_iww_and_sig_from_body`` for agreements that + use an alternative sig-page operating phrase (e.g. "EXECUTED as of + the date first set forth above.") instead of the canonical "IN + WITNESS WHEREOF". When doc2dict packs the entire sig page into a + record's body and there is no IWW anchor in the document, the + upstream IWW split is a no-op and the sig content stays buried as a + multi-thousand-char body string. + + Structural detection (shape-only, no phrase matching): + + 1. The document has ZERO existing records whose title/body STARTS + with the IWW phrase. (When an IWW IS present, ``_split_iww_and_ + sig_from_body`` handles it.) This function is the IWW-less + fallback — it never runs if an IWW anchor exists, so it cannot + double-split or compete with the IWW splitter. + 2. Find a record whose body contains ≥3 ``/s/`` signature marks. + The 3-mark threshold filters out bodies that merely quote a + sig-mark in passing; a real packed signature page always + carries multiple distinct signatures. + 3. Locate the boundary: the LAST newline before the FIRST sig-shape + line in the body. A sig-shape line is any of: a ``/s/`` mark, + a "By:/Name:/Title:/Address:" field, a short uppercase label + (``_SIG_BLOCK_LABEL_RE``), a corporate-suffix party name + (``_CORP_SUFFIX_LABEL_RE``). + 4. Split: + - Text before the boundary stays on the original record as + its trimmed body. + - Text between the boundary and the start of the sig area — + if it carries substantive content (a sentence ending in + ``.``) — emits as a new L1 record (the "operating clause + stand-in"). This is the agreement's equivalent of the IWW + sentence: it sits at L1 and serves as the sig page's + header parent. + - All sig-shape lines from the boundary onward emit as L2 + records, one per line, dedup'd to collapse the doc2dict + "each block twice" pathology. + + Runs AFTER ``_split_iww_and_sig_from_body`` so the IWW splitter + gets first crack at the document. Runs BEFORE ``_explode_signature + _block_lines`` so the new sig-line records pick up the standard + sig-line marking and depth pinning. + """ + # Guard 1: skip if any record carries IWW as leading text. The IWW + # splitter handles those cases; we are strictly the IWW-less + # fallback. + for r in rows: + if r.get("is_envelope") or r.get("scope") == "trailer": + continue + title = (r.get("title") or "").strip() + body = (r.get("body_direct") or "").strip() + if _is_iww_clause(title) or _is_iww_clause(body): + return rows + + def _is_sig_shape_line(line: str) -> bool: + s = line.strip() + if not s: + return False + if _SIGN_OFF_RE.search(s): + return True + if _SIG_FIELD_RE.match(s): + return True + # Uppercase label or corporate-suffix party-name shape. + if _SIG_BLOCK_LABEL_RE.match(s): + return True + if _CORP_SUFFIX_LABEL_RE.match(s): + return True + # Label ending in colon ("BORROWER:", "GUARANTORS:") + if re.match(r"^[A-Z][A-Z .,&'\-/]{1,60}:$", s): + return True + return False + + next_id = max((r["node_id"] for r in rows), default=-1) + 1 + new_rows: list[dict[str, Any]] = [] + + for r in rows: + new_rows.append(r) + if r.get("is_envelope") or r.get("scope") == "trailer": + continue + body = r.get("body_direct") or "" + if not body: + continue + # Count /s/ marks — at least 3 distinct sig marks indicates a + # real packed sig page, not a passing reference. + sig_marks = _SIGN_OFF_RE.findall(body) + if len(sig_marks) < 3: + continue + # Find the first sig-shape line in the body, scanning newlines. + lines = body.split("\n") + first_sig_line_idx: int | None = None + for i, line in enumerate(lines): + if _is_sig_shape_line(line): + first_sig_line_idx = i + break + if first_sig_line_idx is None: + # /s/ is present but no clean line break before it — bail. + continue + # The "operating clause stand-in" is the substantive sentence + # text BEFORE the first sig-shape line. To recover the operating + # sentence cleanly even when doc2dict left fragment lines (like + # a single-word remnant "atthew") between the operating period + # and the first sig-shape line, we split at the LAST sentence- + # ending period in the pre-sig text. Everything after that + # period — and the fragment lines too — moves into the sig area. + pre_sig_text = "\n".join(lines[:first_sig_line_idx]) + last_period = pre_sig_text.rfind(".") + if last_period >= 0: + operating_text = pre_sig_text[: last_period + 1].rstrip() + # Fragment text between the operating-clause period and the + # first sig-shape line is dropped — typically a single-char + # doc2dict drop ("atthew" from "Matthew") that has no + # downstream consumer. + else: + operating_text = "" + # The sig-area is the remaining lines. + sig_area_lines = lines[first_sig_line_idx:] + # Keep nothing on the original body — the sig page was its + # entire content. (If the body had a real agreement-clause + # prefix it should already be its own record, not packed here.) + # Operating-clause-stand-in requirements: substantive content, + # ends with a period. + op_ok = bool(operating_text) and operating_text.rstrip().endswith(".") + + # Clear the original record's body (the sig content has been + # extracted into its own records below). + r["body_direct"] = "" + r["body_direct_chars"] = 0 + + subdoc_penalty = r.get("subdoc_penalty") or 0 + l1_depth = 1 + subdoc_penalty + l2_depth = 2 + subdoc_penalty + + op_node_id: int | None = None + if op_ok: + # Emit the operating-clause stand-in as a new L1 record. + # Parent is the same as the original record's parent so + # the source-position sorter can place it correctly. + op_record = { + "node_id": next_id, + "parent_node_id": r.get("parent_node_id"), + "depth": l1_depth, + "doc2dict_section_key": f"_sigop_{next_id}", + "cls": "promoted text leaf", + "title": "", + "standardized_title": None, + "n_direct_section_children": 0, + "body_direct_chars": len(operating_text), + "body_recursive_chars": len(operating_text), + "body_direct": operating_text, + "body_recursive": operating_text, + "subdoc_penalty": subdoc_penalty, + "is_envelope": False, + "scope": "agreement", + } + new_rows.append(op_record) + op_node_id = next_id + next_id += 1 + + # Emit each sig-area line as L2. Use the same per-line cleaner + # and dedup logic as the IWW splitter. Parent is the op_record + # if we emitted one, else the original record. + parent_for_sig_lines = op_node_id if op_node_id is not None else r["node_id"] + cleaned = _split_sig_block_body_into_lines("\n".join(sig_area_lines)) + seen_lines: set[str] = set() + for line in cleaned: + if line in seen_lines: + continue + seen_lines.add(line) + new_rows.append({ + "node_id": next_id, + "parent_node_id": parent_for_sig_lines, + "depth": l2_depth, + "doc2dict_section_key": f"_sig_line_{next_id}", + "cls": "promoted text leaf", + "title": line, + "standardized_title": None, + "n_direct_section_children": 0, + "body_direct_chars": 0, + "body_recursive_chars": len(line), + "body_direct": "", + "body_recursive": line, + "subdoc_penalty": subdoc_penalty, + "is_envelope": False, + "scope": "agreement", + "_sig_line": True, + }) + next_id += 1 + + return new_rows + + def _explode_signature_block_lines( rows: list[dict[str, Any]], ) -> list[dict[str, Any]]: @@ -4498,6 +4760,26 @@ def parse_one(idx: int, raw: str) -> tuple[dict[str, Any], list[dict[str, Any]]] # and emits IWW as L1 + each sig-page line as L2 — restoring the # title-as-root sig-page hierarchy. sections = _split_iww_and_sig_from_body(sections) + # IWW-less fallback: when the agreement uses an alternative sig-page + # operating phrase (e.g. "EXECUTED as of the date first set forth + # above.") instead of "IN WITNESS WHEREOF", the IWW splitter above + # is a no-op. This pass detects dense sig content (≥3 ``/s/`` marks + # in one body) and splits it into an L1 operating-clause stand-in + # plus per-line L2 sig records. Same title-as-root sig-page + # hierarchy, just keyed off ``/s/`` density and sig-shape lines + # instead of the IWW phrase. + sections = _split_dense_sig_body_no_iww(sections) + # Demote deeply-nested all-caps body paragraphs (legal-emphasis + # disclaimers, no-oral-amendments notices) to L1. doc2dict's depth + # reflects HTML container nesting, not structural hierarchy; an + # all-caps disclaimer wrapped in extra ``

`` for visual + # emphasis lands at depth 4+ even though it sits structurally at L1 + # alongside the surrounding numbered sections. Detection is purely + # SHAPE-based: predicted-header class, no section marker, no + # structural-level pattern, empty body, long all-caps title. Runs + # AFTER the sig-page splitter so any record that started with the + # sig page packed into its body has been cleared first. + sections = _demote_deeply_nested_body_paragraphs(sections) # Explode the signature page per the title-as-root rubric: # - IWW operating clause sits at L1 (signature-page header). # - Each signature-page line (party label, /s/, name, title,