🕷️ WebScraperPro

A powerful, async web scraping toolkit with multiple export formats

Features • Installation • Quick Start • CLI Usage • Examples • API Reference

---

✨ Features

• 🚀 Async/Concurrent Scraping - Scrape hundreds of pages simultaneously
• 🔄 Smart Retry Logic - Exponential backoff with configurable retries
• 🎭 User-Agent Rotation - Avoid detection with rotating browser signatures
• 🌐 Proxy Support - Built-in proxy rotation for anonymity
• 📊 Multiple Export Formats - JSON, CSV, Excel, SQLite, Markdown
• 🎯 Schema-Based Extraction - Define what you want, get structured data
• 🪝 Request/Response Hooks - Custom processing at every step
• 📈 Built-in Statistics - Track success rates and response times
• 🖥️ Powerful CLI - Full-featured command-line interface

---

📦 Installation

# Clone the repository
git clone https://github.com/yourusername/web-scraper-pro.git
cd web-scraper-pro

# Install dependencies
pip install -r requirements.txt

# Or install with pip (editable mode)
pip install -e .

Optional Dependencies

# For Excel export support
pip install openpyxl

# For faster HTML parsing
pip install lxml

---

🚀 Quick Start

Basic Usage

import asyncio
from scraper import WebScraper, HTMLParser

async def main():
    # Create scraper
    scraper = WebScraper()
    
    # Scrape a single page
    result = await scraper.scrape_single("https://example.com")
    
    if result.success:
        # Parse the HTML
        parser = HTMLParser(result.html)
        
        # Extract data
        title = parser.get_page_title()
        links = parser.get_links()
        
        print(f"Title: {title}")
        print(f"Found {len(links)} links")

asyncio.run(main())

Scrape Multiple URLs

import asyncio
from scraper import WebScraper

async def main():
    scraper = WebScraper()
    
    urls = [
        "https://example.com",
        "https://httpbin.org/html",
        "https://quotes.toscrape.com",
    ]
    
    # Scrape all URLs concurrently
    results = await scraper.scrape(urls)
    
    for result in results:
        if result.success:
            print(f"✅ {result.url} ({result.response_time:.2f}s)")
        else:
            print(f"❌ {result.url} - {result.error}")
    
    # Get statistics
    print(scraper.get_stats())

asyncio.run(main())

Synchronous Wrapper

from scraper import WebScraper

scraper = WebScraper()

# Use .run() for synchronous execution
results = scraper.run([
    "https://example.com",
    "https://httpbin.org/html",
])

---

🖥️ CLI Usage

The CLI provides a powerful command-line interface:

# Scrape a single URL
python cli.py https://example.com -o output.json

# Scrape multiple URLs
python cli.py https://site1.com https://site2.com -o results.csv -f csv

# Scrape URLs from a file
python cli.py -i urls.txt -o data.json -c 20

# Extract specific data using selectors
python cli.py https://example.com --extract "title:h1,links:a@href[]"

# Extract links and images
python cli.py https://example.com --links --images -o data.json

# Configure scraping behavior
python cli.py https://example.com -c 20 -d 0.5 -t 60 -r 5

CLI Options

Option	Description	Default
-o, --output	Output file path	output.json
-f, --format	Output format (json, csv, excel, sqlite, markdown)	json
-c, --concurrent	Max concurrent requests	10
-d, --delay	Delay between requests (seconds)	1.0
-t, --timeout	Request timeout (seconds)	30
-r, --retries	Max retry attempts	3
-i, --input-file	File containing URLs	-
-e, --extract	Extraction schema	-
--links	Extract all links	-
--images	Extract all images	-
--meta	Extract meta tags	-

---

📖 Examples

1. Scrape Hacker News

from scraper import WebScraper, HTMLParser
import asyncio

async def scrape_hn():
    scraper = WebScraper()
    result = await scraper.scrape_single("https://news.ycombinator.com")
    
    parser = HTMLParser(result.html)
    
    stories = []
    for link in parser.select(".titleline > a")[:10]:
        stories.append({
            "title": link.get_text(strip=True),
            "url": link.get("href"),
        })
    
    return stories

stories = asyncio.run(scrape_hn())

2. Schema-Based Extraction

parser = HTMLParser(html)

# Define what you want to extract
schema = {
    "title": "h1",
    "description": "meta[name='description']@content",
    "all_links": "a@href[]",
    "prices": ".price[]",
    "main_image": "img.hero@src",
}

# Extract everything at once
data = parser.extract(schema)

3. Export to Multiple Formats

from scraper.exporters import JSONExporter, CSVExporter, ExcelExporter, SQLiteExporter

data = [{"name": "Product 1", "price": 29.99}, ...]

# JSON
JSONExporter().export(data, "output/data.json")

# CSV
CSVExporter().export(data, "output/data.csv")

# Excel (requires openpyxl)
ExcelExporter(sheet_name="Products").export(data, "output/data.xlsx")

# SQLite Database
SQLiteExporter(table_name="products").export(data, "output/data.db")

4. Advanced: Hooks and Proxies

from scraper import WebScraper
from scraper.core import ScraperConfig

config = ScraperConfig(
    max_concurrent_requests=20,
    request_delay=0.5,
    random_delay=True,
)

scraper = WebScraper(config)

# Add proxies
scraper.add_proxies([
    "http://proxy1:8080",
    "http://user:pass@proxy2:8080",
])

# Add hooks
@scraper.before_request
def log_request(url):
    print(f"Requesting: {url}")

@scraper.after_response  
def process_response(result):
    print(f"Got {result.status_code} for {result.url}")

---

📚 API Reference

WebScraper

The main scraping class with async support.

from scraper import WebScraper
from scraper.core import ScraperConfig

# Configuration options
config = ScraperConfig(
    max_concurrent_requests=10,  # Simultaneous requests
    request_delay=1.0,           # Delay between requests
    timeout=30,                  # Request timeout
    max_retries=3,               # Retry attempts
    retry_delay=2.0,             # Initial retry delay
    random_delay=True,           # Randomize delays
)

scraper = WebScraper(config)

# Methods
await scraper.scrape(urls)        # Scrape multiple URLs
await scraper.scrape_single(url)  # Scrape single URL
scraper.run(urls)                 # Sync wrapper
scraper.get_stats()               # Get statistics
scraper.add_proxies(proxies)      # Add proxy servers

HTMLParser

Powerful HTML parsing with multiple extraction methods.

from scraper import HTMLParser

parser = HTMLParser(html)

# Selection
parser.select(css_selector)       # Select all matching
parser.select_one(css_selector)   # Select first match

# Text extraction
parser.get_text(selector)         # Get text from element
parser.get_all_text(selector)     # Get text from all matches
parser.get_attribute(sel, attr)   # Get attribute value

# Special extractions
parser.get_links(base_url)        # Extract all links
parser.get_images(base_url)       # Extract all images
parser.get_meta_tags()            # Extract meta tags
parser.get_open_graph()           # Extract OG tags
parser.get_json_ld()              # Extract JSON-LD
parser.get_tables()               # Extract all tables
parser.get_page_title()           # Get page title

# Pattern matching
parser.find_emails()              # Find email addresses
parser.find_phone_numbers()       # Find phone numbers
parser.find_by_regex(pattern)     # Custom regex

# Schema extraction
parser.extract(schema)            # Extract using schema

Exporters

Export data to various formats.

from scraper.exporters import (
    JSONExporter,
    CSVExporter,
    ExcelExporter,
    SQLiteExporter,
    MarkdownExporter,
)

# All exporters have the same interface
exporter.export(data, filepath)

---

🧪 Practice Sites

Test the scraper with these practice sites (designed for learning):

• Quotes to Scrape - Quotes and pagination
• Books to Scrape - E-commerce simulation
• httpbin.org - HTTP testing

---

⚠️ Legal & Ethical Considerations

• Always check a website's robots.txt and Terms of Service
• Respect rate limits and add delays between requests
• Don't scrape personal or sensitive data without permission
• Use this tool responsibly and ethically

---

📄 License

MIT License - see LICENSE for details.

---

🤝 Contributing

Contributions are welcome! Please feel free to submit a Pull Request.

1. Fork the repository
2. Create your feature branch (git checkout -b feature/amazing)
3. Commit your changes (git commit -m 'Add amazing feature')
4. Push to the branch (git push origin feature/amazing)
5. Open a Pull Request

---

By Mustafa Faham

⭐ Star this repo if you find it useful!