Art Wikipedia Knowledge Acquisition System

A sophisticated art knowledge extraction and aggregation platform that curates comprehensive data from Wikipedia's extensive art collections, enabling precision-targeted acquisition of painting and sculpture metadata.

🎨 Overview

This system provides a robust pipeline for extracting, processing, and structuring high-fidelity art historical data from Wikipedia. It combines web intelligence with efficient data aggregation to deliver enriched artwork documentation including visual assets, contextual information, and temporal metadata.

✨ Core Features

• Dual-Category Curation: Targeted extraction of paintings and sculptures with comprehensive coverage
• Multi-Modal Data Acquisition: Full-resolution image procurement and encyclopedic full-text retrieval
• Structured Output Formats: Native JSON serialization and embedded-image Excel export with aspect ratio preservation
• Resilient Processing: Checkpoint-based fault tolerance enabling interrupted crawls to resume seamlessly
• Asynchronous Concurrency: Non-blocking request orchestration for optimized throughput
• Automatic Persistence: Incremental checkpoint saves (every 100 records) ensuring data integrity

📦 Installation

pip install -r requirements.txt

🚀 Usage

python crawler.py

📁 Output Structure

output/
├── artworks.json      # Comprehensive dataset (all acquired records)
├── artworks.xlsx      # Formatted spreadsheet with embedded imagery
└── checkpoint.json    # Execution checkpoint for recovery

images/                # Downloaded image assets
logs/
└── crawl.log          # Detailed execution logs

📊 Data Schema

Each artwork record contains:

{
  "title": "Mona Lisa",
  "image_url": "https://...",
  "image_width": 1920,
  "image_height": 1080,
  "image_path_local": "images/...",
  "page_text": "Complete encyclopedic content...",
  "page_url": "https://en.wikipedia.org/...",
  "art_type": "Painting",
  "source_category": "Category:Paintings",
  "crawled_at": "2026-03-05T..."
}

⚙️ Configuration

Customize acquisition parameters via config.py:

Parameter	Description	Default
TARGET_COUNT	Total artworks to acquire	10000
MAX_CONCURRENT_REQUESTS	Parallel request limit	10
REQUEST_DELAY	Inter-request delay (seconds)	0.1
DOWNLOAD_IMAGES	Enable image asset retrieval	True

🔄 Checkpoint Recovery

The system automatically maintains progress snapshots in checkpoint.json. Resume interrupted sessions by executing the crawler again—it will intelligently continue from the last recorded state.

To restart from scratch, remove output/checkpoint.json and re-run.

⏱️ Performance Metrics

• Estimated Duration: 10,000 artworks in 30-60 minutes (network and concurrency dependent)
• Resource Requirements: ~10-50GB disk space (images + metadata)
• Rate Limits: Wikipedia API enforces throttling; avoid excessive concurrency escalation
• Bandwidth: Optimize based on network conditions and concurrent request count

📝 Notes

• Ensure adequate storage capacity before initiating large-scale crawls
• Respect Wikipedia's rate-limiting policies and Terms of Service
• Monitor execution logs for detailed acquisition metrics and diagnostic information
• Consider network stability for extended acquisitions

---

Art Wikipedia Knowledge Acquisition System — Precision data extraction for art historical research and knowledge infrastructure development.