fix(openapi_fdw): handle RFC 8288 Link header pagination#599
Merged
burmecia merged 4 commits intoMay 19, 2026
Conversation
handle_pagination only read JSON-body pagination paths (/links/next, /meta/pagination/next, etc.) and never inspected the HTTP Link response header. APIs that paginate exclusively via Link (GitHub, GitLab, most RFC 5988 / RFC 8288 REST services) never auto-paginated beyond the first page, regardless of max_pages, page_size, or page_size_param. Thread response headers into handle_pagination and parse rel="next" entries before the existing JSON-body lookups. Cross-origin protection is preserved by the existing resolve_pagination_url path. Precedence (first match wins): 1. Configured cursor_path 2. RFC 8288 Link header (rel="next") 3. JSON-body next-URL paths 4. has_more flag + cursor paths Verified end-to-end against the real GitHub API: count(*) on repo_pulls for supabase/wrappers fetches 468 PRs across 16 chained HTTP requests, each following the previous response's Link header. Closes supabase#598
Contributor
Author
|
Note: I did not bump the version on this PR, but can if the change is approved. |
Contributor
There was a problem hiding this comment.
Pull request overview
Adds RFC 8288 Link header support to openapi_fdw pagination so APIs that paginate exclusively via Link: <...>; rel="next" (e.g., GitHub/GitLab) can automatically fetch subsequent pages, while preserving existing pagination behaviors and cross-origin protections.
Changes:
- Thread HTTP response headers into
handle_paginationand addLinkheader parsing (with precedence between cursor_path / header / body heuristics). - Add unit tests covering a variety of
Linkheader formats and precedence rules. - Update OpenAPI FDW docs and GitHub example README to document
Linkheader pagination support.
Reviewed changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| wasm-wrappers/fdw/openapi_fdw/src/response.rs | Adds Link header parsing helpers and updates pagination precedence to consider RFC 8288 headers. |
| wasm-wrappers/fdw/openapi_fdw/src/response_tests.rs | Adds/updates unit tests for Link header pagination and new handle_pagination signature. |
| wasm-wrappers/fdw/openapi_fdw/src/request.rs | Passes response headers into pagination handling at request time. |
| wasm-wrappers/fdw/openapi_fdw/examples/github/README.md | Documents GitHub pagination behavior and provides a multi-page count(*) example. |
| docs/catalog/wasm/index.md | Updates OpenAPI Wasm wrapper tag/source links to v0.2.0. |
| docs/catalog/openapi.md | Documents Link header pagination as a supported pagination style. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
split_link_entries toggled in_quotes on every `"` character without accounting for backslash-escaped quotes inside an HTTP quoted-string (e.g., title="a \"quoted, comma\" page"). With escapes unhandled the quote state could desync and a comma inside a quoted parameter value could be treated as an entry separator, causing rel="next" to be missed. Track quoted-pair escapes when scanning, so `\"` inside a quoted parameter is consumed as a literal and does not flip in_quotes. Addresses Copilot review feedback on supabase#599.
apply_headers now always adds a default user-agent when none is provided. Six tests still asserted the old header counts and indices from before that change was introduced.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #598
Summary
handle_paginationonly inspected JSON-body pagination paths (/links/next,/meta/pagination/next, etc.) and never read the HTTPLinkresponse header. APIs that paginate exclusively viaLink— GitHub, GitLab, and most REST services that follow RFC 5988 / RFC 8288 — never auto-paginated past the first page.handle_paginationand parserel="next"entries before the existing JSON-body lookups. Cross-origin protection is preserved automatically by the existingresolve_pagination_urlpath.cursor_path→Linkheader → JSON-body next-URL →has_more+ cursor.Verification
Tested end-to-end against the real GitHub API:
Debug trace shows exactly 16 chained HTTP requests, each one following the previous response's
Link: <...>; rel="next"header (GitHub even rewrites the path to/repositories/558211706/pulls?...&page=N— the FDW transparently follows it).Test plan
response_tests.rscover: GitHub-style multi-rel headers, case-insensitive header name, unquoted rel values, multi-rel values (rel="next prev"), extra params, no-match returns None, array bodies + Link header, Link beats body autodetect, configuredcursor_pathbeats Link, fallback to body when norel="next", multiple separateLinkheaders, malformed input doesn't panic.Docs
docs/catalog/openapi.md— addedLinkheader to the supported pagination styles list.docs/catalog/wasm/index.md— bumped OpenAPI tag/source links from v0.1.4 → v0.2.0 (was stale; mainopenapi.mdalready lists 0.2.0 as current).wasm-wrappers/fdw/openapi_fdw/examples/github/README.md— added a "Pagination across pages" subsection in §6 with the count(*) demo as a real-world test.