Springer/Nature "Buy article PDF" paywall misdetected as logged-in — Buy button clicked instead of institutional SSO

[noking-xdj]

Summary

On closed-access nature.com / link.springer.com articles that show a "Buy article PDF USD 39.95" paywall CTA, instsci papers judges the page as already logged-in, so it never enters the institutional-login flow. Instead the generic PDF-button clicker matches the Buy button (:has-text('PDF')), clicks it, and the run ends with a nature.com .pdf 404 / pdf_not_captured. This violates AGENTS.md step 5 ("let the user complete institution login; wait + resume"), which never happens because the SSO path is short-circuited upstream.

Reproduce

• instsci 0.1.1, Windows, CloakBrowser 0.4.9, Shibboleth/CARSI via university IdP.
• Command:

  instsci papers dois.txt --publisher auto --institution "<your institution>" --no-broker

• dois.txt contains the closed-access DOI 10.1007/s11630-023-1652-y (Journal of Thermal Science, Springer).
• Observed event sequence:

  article_open
  -> pdf_button_clicked  (target text: "Buy article PDF USD 39.95")
  -> nature.com .pdf 404
  -> pdf_not_captured
  

• Expected: the "Buy article PDF" paywall is detected as a hard auth wall; instsci clicks nature.com's "Access through your institution" link (→ wayf.springernature.com), waits for the user to complete SSO, then downloads the entitled PDF.

Root cause

All references are against instsci/publisher_batch.py unless noted. Verified against the current main.

A. _looks_logged_out returns False for the Buy-PDF paywall

_looks_logged_out (defined line 2684) has a success-article branch whose return preempts the sso_text_markers fallback:

# publisher_batch.py:2702-2721
if self._is_success_article_url(getattr(page, "url", "")):   # 2702
    hard_wall_markers = (...)                                 # 2703-2719
    return any(marker in text for marker in hard_wall_markers)  # 2720 — exits the function
return any(marker in text for marker in self.profile.sso_text_markers)  # 2721 — unreachable for success-article URLs

The hard_wall_markers tuple (lines 2703–2719) contains "purchase pdf" (2714) and "purchase this article" (2715), but not "buy article pdf", "buy pdf", "add to cart", or any price/usd/$ pattern. So a Nature page whose body text contains only Buy article PDF USD 39.95 matches nothing in hard_wall_markers → _looks_logged_out returns False → the page is judged logged-in.

Because line 2720 returns inside the if, the sso_text_markers fallback at line 2721 is never consulted for a success-article URL — even though SPRINGER_PROFILE.sso_text_markers (publisher_profiles.py:454–459) does include "access through your institution", which is exactly the link we want.

B. _ensure_login short-circuits on the bad signal

_ensure_login (defined line 678) early-returns when _looks_logged_out is False:

# publisher_batch.py:684-685
if not self._looks_logged_out(page):
    return True

With the paywall misjudged as logged-in, _ensure_login returns True here and never reaches _complete_login_from_current_page (line 687). No SSO entry is ever clicked, and no institutional-login wait is entered.

C. _click_pdf_entry's generic selectors match the Buy button

With the run now believing it is logged in, _click_pdf_entry (defined line 2052) is reached. Springer/Nature is not one of the publisher-specific branches (lines 2053–2061 guard on elsevier/aps/wiley only), so execution falls into the generic catch-all tuple:

# publisher_batch.py:2069-2070
"button:has-text('PDF')",
"a:has-text('PDF')",

The loop (lines 2072–2092) clicks the first visible match with no purchase/buy denylist — unlike the Elsevier path, which penalizes purchase pdf (score -= 80 at line 1034). Playwright :has-text() is a substring match, so a Buy article PDF button satisfies both selectors and gets clicked. The second-stage isPdfControl JS (lines 2094–2142) is never reached because the generic selector already matched first.

Why no path partially fixes this

Repo-wide, the only "purchase" handling is: "purchase pdf"/"purchase this article" in hard_wall_markers (2714–2715); the Elsevier-only score -= 80 (1034); and the Elsevier-only entitlement check _elsevier_lacks_pdf_entitlement (line 2741, gated on profile.name.lower() == "elsevier"). None of these apply to the Springer/Nature path. There are zero occurrences of buy, add to cart, usd, shopping cart, or a price regex anywhere in the repo.

Why this is wrong vs. AGENTS.md

AGENTS.md step 5 instructs the harness to "let the user complete institution login; wait + resume" when a closed-access page requires SSO. That step is only reached via _complete_login_from_current_page, which is only called when _looks_logged_out returns True. Because the Buy-PDF paywall makes _looks_logged_out return False, the SSO wait is skipped entirely and the Buy button is clicked instead — the opposite of the documented flow.

Suggested fix

Non-prescriptive; happy to take either direction.

1. Recognize the Buy/Cart paywall (root cause A). Either
   • add "buy article pdf", "buy pdf", "add to cart" (and optionally a price-with-intent pattern like usd\s?\d / \$\d) to hard_wall_markers (lines 2703–2719); or, more robustly,
   • restructure the success-article branch so the sso_text_markers fallback is also consulted, e.g. return hard_wall_match or any(marker in text for marker in self.profile.sso_text_markers). This also catches future paywall phrasings that overlap with SSO link text.
2. Never click a Buy/Cart/price button as the PDF entry (root cause C). Add a purchase denylist to _click_pdf_entry's generic loop (lines 2072–2092) — skip any :has-text('PDF') match whose visible text/href contains buy, purchase, add to cart, or a price. The Elsevier path already does the equivalent at line 1034.
3. Prefer the working SSO link for Springer/Nature. _click_sso_entry (line 814) already has generic a:has-text('Access through your institution') selectors (lines 830–838) that route to wayf.springernature.com; consider also guarding against the dead /institutional-login (404) href. This is secondary — _click_sso_entry is unreachable until fix Request for Collaboration Quotation #1 unblocks _ensure_login.
4. Late-render nuance. The Buy CTA appears after domcontentloaded, so detection may need a short wait / locator poll before _looks_logged_out is evaluated, otherwise the markers may not yet be in the DOM.

Environment

• instsci 0.1.1 (PyPI == GitHub main, commit 2026-06-23)
• cloakbrowser 0.4.9
• Windows 11
• Shibboleth / CARSI via university IdP

Workaround (current)

We run a small monkey-patch wrapper around the PublisherBatch class that overrides:

• _ensure_login — wait for late render, detect the Buy-PDF paywall, and if present call _complete_login_from_current_page to drive SSO;
• _click_pdf_entry — reject any candidate whose text/href matches buy/purchase/add to cart/a price;
• _click_sso_entry — prefer the wayf.springernature.com WAYF link and skip the dead /institutional-login URL;

launched with --no-broker. Verified to download the entitled article PDF (1.68 MB) after manual SSO completion. Happy to attach the patch if useful.

Relevant files

• instsci/publisher_batch.py — _ensure_login (678), _click_pdf_entry (2052), _looks_logged_out (2684), success-article branch (2702–2720), sso fallback (2721)
• instsci/publisher_profiles.py — SPRINGER_PROFILE (438), success_url_markers (445), sso_text_markers (454–459)
• tests/test_acs_batch.py — _looks_logged_out exercised only with RSC/Elsevier/ACS/Wiley profiles; no Springer/Nature paywall-detection coverage

Notes for tests

Two existing intentional behaviors worth preserving when fixing this:

• test_publisher_article_title_is_not_treated_as_auth_wall (test_acs_batch.py:277) asserts _looks_logged_out is False for an RSC page whose body contains the benign phrase "purchase options". This is likely why "buy article pdf" was never added. A fix should still pass this case — "Buy article PDF USD 39.95" (price + CTA verb) is a distinct, harder signal than the soft "purchase options" wording.
• A new test exercising _looks_logged_out with SPRINGER_PROFILE on a nature.com/articles/... page containing "Buy article PDF USD 39.95" would directly cover this gap.