Source overview
Amazon Scraper API — https://amazonscraperapi.com (1000 free requests on signup)
A managed REST API for scraping Amazon product detail pages, search results, and batch ASIN lookups. Backed by residential proxies and TLS impersonation. Returns clean structured JSON.
Why this can't be cleanly served by rest_api source
I'm aware of the policy ("we only accept sources that can't easily be implemented via rest_api"). Three reasons this source needs custom behavior:
-
Marketplace-specific schema variation — the product endpoint returns subtly different field sets across the 12 supported marketplaces (US, UK, DE, FR, IT, ES, CA, JP, IN, BR, MX, AU). For example, price.was, subscribe_save_price, and prime_pantry are US-only fields; tax_inclusive_price is EU-only; review syndication structure differs between .com and .co.jp. A naïve rest_api binding ends up with inconsistent typing and breaks merge keys across marketplaces.
-
ASIN-keyed incremental state with cross-resource dedup — the search endpoint returns ASINs that the product endpoint can fetch individually. A proper incremental load needs to maintain per-ASIN scraped_at watermarks AND track ASINs discovered via search but not yet hydrated via product. This is a cross-resource state pattern that rest_api source doesn't natively model.
-
Async batch with poll-for-completion — the /v1/amazon/batch endpoint accepts up to 1000 ASINs per call and returns a job_id; the user must then poll a results endpoint. dlt's rest_api paginator helpers don't have a clean async-job pattern for this; modelling it as a dlt resource requires a generator with explicit yield-then-poll semantics.
Sample resource design
@dlt.source
def amazon_scraper_source(api_key=dlt.secrets.value, asins=None, queries=None, marketplace="US"):
return [products(api_key, asins, marketplace),
search_results(api_key, queries, marketplace),
bulk_lookup(api_key, asins, marketplace)]What we'd commit to
- Comprehensive tests against duckdb + postgres (we'll seed a test API key with sufficient quota for CI)
- Documentation page at /dlt-ecosystem/verified-sources/amazon_scraper_api/
- README with auth setup, marketplace coverage, schema notes per marketplace
Test credentials
Happy to provide a long-lived API key for the dlt CI environment as soon as the proposal is greenlit.
If the answer is "use rest_api with a custom paginator subclass and document it in the cookbook", we'll do that instead and contribute a docs PR — please advise.
Source overview
Amazon Scraper API — https://amazonscraperapi.com (1000 free requests on signup)
A managed REST API for scraping Amazon product detail pages, search results, and batch ASIN lookups. Backed by residential proxies and TLS impersonation. Returns clean structured JSON.
Why this can't be cleanly served by rest_api source
I'm aware of the policy ("we only accept sources that can't easily be implemented via rest_api"). Three reasons this source needs custom behavior:
Marketplace-specific schema variation — the product endpoint returns subtly different field sets across the 12 supported marketplaces (US, UK, DE, FR, IT, ES, CA, JP, IN, BR, MX, AU). For example,
price.was,subscribe_save_price, andprime_pantryare US-only fields;tax_inclusive_priceis EU-only; review syndication structure differs between .com and .co.jp. A naïve rest_api binding ends up with inconsistent typing and breaks merge keys across marketplaces.ASIN-keyed incremental state with cross-resource dedup — the search endpoint returns ASINs that the product endpoint can fetch individually. A proper incremental load needs to maintain per-ASIN
scraped_atwatermarks AND track ASINs discovered via search but not yet hydrated via product. This is a cross-resource state pattern that rest_api source doesn't natively model.Async batch with poll-for-completion — the /v1/amazon/batch endpoint accepts up to 1000 ASINs per call and returns a job_id; the user must then poll a results endpoint. dlt's rest_api paginator helpers don't have a clean async-job pattern for this; modelling it as a dlt resource requires a generator with explicit yield-then-poll semantics.
Sample resource design
What we'd commit to
Test credentials
Happy to provide a long-lived API key for the dlt CI environment as soon as the proposal is greenlit.
If the answer is "use rest_api with a custom paginator subclass and document it in the cookbook", we'll do that instead and contribute a docs PR — please advise.