📚 Python Web Scraper

A professional intermediate-level web scraping project built with Python, BeautifulSoup, Requests, SQLAlchemy, and SQLite.

---

Features

• Web scraping using Requests
• HTML parsing using BeautifulSoup
• Multi-page scraping
• Rate limiting
• Retry mechanism with exponential backoff
• Logging with Loguru
• Data validation
• CSV export
• JSON export
• SQLite database storage
• Duplicate prevention
• Command-line interface (CLI)

---

Project Architecture

Website
   │
   ▼
Fetcher
   │
   ▼
Parser
   │
   ▼
Extractor
   │
   ▼
Validator
   │
   ▼
SQLite Database
   │
   ├── CSV Export
   └── JSON Export

---

Folder Structure

web_scraper/
│
├── config/
│   └── settings.py
│
├── scraper/
│   ├── fetcher.py
│   ├── parser.py
│   ├── extractor.py
│   └── pipeline.py
│
├── database/
│   ├── db.py
│   ├── models.py
│   ├── crud.py
│   └── init_db.py
│
├── utils/
│   ├── exporter.py
│   ├── logger.py
│   └── validator.py
│
├── data/
│   └── exports/
│
├── logs/
│
├── screenshots/
│
├── main.py
├── check_db.py
├── requirements.txt
├── README.md
└── .gitignore

---

Installation

Clone repository:

git clone <repository-url>
cd web_scraper

Create virtual environment:

python -m venv venv

Activate:

Windows:

venv\Scripts\activate

Install dependencies:

pip install -r requirements.txt

---

Usage

Scrape 5 pages:

python main.py --pages 5

Scrape 10 pages:

python main.py --pages 10

Display help:

python main.py --help

---

Database

Initialize database:

python -m database.init_db

Check stored records:

python check_db.py

---

Example Output

Fetching...
Found 20 books
Waiting 2.14s

Fetching...
Found 20 books

Saved 100 books

---

Technologies Used

• Python
• Requests
• BeautifulSoup4
• SQLAlchemy
• SQLite
• Pandas
• Loguru
• Git
• GitHub

---

Future Improvements

• Playwright Integration
• Proxy Rotation
• Docker Support
• GitHub Actions CI/CD
• MongoDB Support
• Async Scraping

---

Author

Samyak Mahatme