Books to Scrape — Production-Grade Web Scraper

A clean, polite, production-style Python scraper for the public scraping sandbox books.toscrape.com (a site built specifically for scraping practice — no Terms-of-Service issues).

It crawls all 50 catalogue pages (~1,000 books), optionally enriches each record from its product page, and outputs a clean products.csv plus a summary.

Portfolio demo. The same architecture (pagination, detail enrichment, polite requests, retries, monitoring) is what I use for real client scrapers on e-commerce sites, directories and marketplaces.

Features

• Full pagination — follows the "next" link until the last page automatically.
• Detail enrichment — visits each product page for category, UPC and exact stock count (toggle off for speed).
• Polite & compliant — reads and respects robots.txt, custom User-Agent, configurable delay between requests.
• Robust — automatic retries with backoff; a failed item is logged and skipped instead of crashing the run.
• Clean output — structured products.csv (UTF-8) + a readable summary (totals, average price, rating distribution, top categories).
• Testable — parsing logic is pure (HTML in → data out), with an offline test that runs against saved fixtures, no network required.

Extracted fields

Field	Example	Source
title	A Light in the Attic	listing
price_gbp	51.77	listing
availability	In stock	listing
stock_count	22	product page
rating	3	listing (CSS)
category	Poetry	product page
upc	a897fe39b1053632	product page
url	http://books.toscrape.com/catalogue/...	listing

Install

pip install -r requirements.txt

Usage

python scraper.py                 # all pages + detail enrichment -> products.csv
python scraper.py --max-pages 2   # quick demo on the first 2 listing pages
python scraper.py --no-details    # listing fields only (much faster)
python scraper.py --out books.csv # custom output path

Sample output

====================================================
  TOTAL BOOKS:        1000
  AVG PRICE (GBP):    35.07
  PRICE RANGE:        10.00 - 59.99
  RATING COUNTS:      1*:226, 2*:196, 3*:203, 4*:179, 5*:196
  TOP CATEGORIES:     Default (152), Nonfiction (110), Sequential Art (75)
====================================================

Run the offline test

PYTHONPATH=. python tests/test_offline.py

Verifies the parsers against saved HTML fixtures (price/rating/pagination/ category/UPC/stock) with assertions — no network needed.

Tech stack

Python · httpx · BeautifulSoup (lxml) · pandas · tqdm

Responsible scraping

This project targets a sandbox built for scraping. For any real-world target I always check robots.txt and the site's Terms of Service, apply rate limiting, and never collect personal data unlawfully (GDPR / KVKK).

---

Need a scraper for a site that keeps breaking your tools, or data trapped in scanned PDFs? I build production pipelines that keep running.