OpenAg-DB: Open Agricultural Equipment Database

A public, community-driven agricultural equipment database using a Python-centric stack and Lakehouse architecture. OpenAg-DB provides comprehensive specifications for tractors, combines, implements, and other agricultural equipment.

🚀 Live Demo

Frontend: https://adam133.github.io/equipment-testing/
Backend API: Deployed on Google Cloud Run (us-central1)

The frontend is connected to a live FastAPI backend running on Google Cloud Platform, which queries Unity Catalog Delta tables for real agricultural equipment data.

🌾 About

OpenAg-DB is designed to be the go-to resource for agricultural equipment specifications, built on modern data infrastructure:

• Data Model: Pydantic models with polymorphic support for different equipment types
• Storage: Unity Catalog Delta tables for versioned, queryable data
• API: FastAPI backend for serving equipment data
• Scrapers: Polite, scheduled Scrapy spiders for data collection
• Frontend: React + Vite + Shadcn UI for searchable interface
• Community: Open-source contribution model via GitHub

🏗️ Architecture

OpenAg-DB/
├── src/
│   ├── api/           # FastAPI backend
│   ├── scrapers/      # Scrapy spiders for data collection
│   ├── core/          # Pydantic models & Unity Catalog utilities
│   └── frontend/      # React/Vite static site
├── .github/workflows/ # CI/CD and scheduled scrapers
└── tests/            # Pytest test suite

Data Flow

1. Collection: Scrapy spiders collect equipment data from manufacturer websites
2. Validation: All data validated against Pydantic models
3. Storage: Data written to Delta tables in Unity Catalog
4. Query: FastAPI serves data to frontend
5. Contribution: Users suggest corrections via GitHub Issues

📊 Equipment Types

OpenAg-DB supports multiple equipment categories with specialized models:

Tractors

• Horsepower (PTO/Engine)
• Transmission Type
• Hydraulic Flow & Pressure
• Three-Point Hitch Specifications

Combines

• Grain Tank Capacity
• Separator Type (Conventional/Rotary/Hybrid)
• Unloading Rate
• Engine Specifications

Implements

• Working Width
• Weight
• Required HP Range
• Row Configuration (for planters/cultivators)

Quick Start

Backend (Python)

# Clone the repository
git clone https://github.com/adam133/equipment-testing.git
cd equipment-testing

# Install dependencies and pre-commit hooks (recommended)
make setup
# Or: uv sync --all-extras --dev && uv run pre-commit install

# Start the API server
uv run openagdb-api
# Visit http://localhost:8000/docs for interactive API documentation

# Run tests
make test
# Or: uv run pytest

Frontend (React)

# Navigate to frontend directory
cd src/frontend

# Install dependencies
npm install

# Start development server
npm run dev
# Visit http://localhost:5173

# Build for production
npm run build

Prerequisites

• Python 3.12 or higher
• uv - Fast Python package installer
• Node.js 22+ (for frontend development)
• Unity Catalog endpoint with access token (for data storage in production)

Installation

Install uv

If you don't have uv installed:

On macOS and Linux:

curl -LsSf https://astral.sh/uv/install.sh | sh

On Windows:

powershell -c "irm https://astral.sh/uv/install.ps1 | iex"

Install Project Dependencies

# Clone the repository
git clone https://github.com/adam133/equipment-testing.git
cd equipment-testing

# Sync Python dependencies
uv sync

# For development with all optional dependencies
uv sync --all-extras

Usage

Running the API Server

# Start the FastAPI development server
uv run openagdb-api

# The API will be available at http://localhost:8000
# Interactive API docs at http://localhost:8000/docs

Using the Equipment Models

from core.models import Tractor, EquipmentCategory, TransmissionType

# Create a tractor instance
tractor = Tractor(
    make="John Deere",
    model="5075E",
    series="5E Series",
    year_start=2014,
    pto_hp=65,
    engine_hp=75,
    transmission_type=TransmissionType.POWERSHIFT,
)

# Validate and export
print(tractor.model_dump())

Running Scrapers

# Run a specific spider
uv run scrapy crawl quality_farm_supply -o tractors.json

# List all available spiders
uv run scrapy list

Available Spiders

• quality_farm_supply: Scrapes tractor specifications from Quality Farm Supply
  • Supports multiple makes and models
  • Exports to JSON, CSV, and other formats
  • See src/scrapers/spiders/README.md for details
• tractordata: Template spider (placeholder)

Development

Project Structure

equipment-testing/
├── .github/workflows/
│   ├── ci.yml           # Testing, linting, building
│   └── scraper.yml      # Scheduled data collection
├── src/
│   ├── api/             # FastAPI REST API
│   │   └── main.py      # API endpoints
│   ├── core/            # Core data models
│   │   ├── models.py    # Pydantic equipment models
│   │   └── databricks_utils.py  # Unity Catalog utilities
│   ├── scrapers/        # Scrapy data collection
│   │   ├── spiders/     # Spider implementations
│   │   ├── pipelines.py # Data processing pipelines
│   │   └── settings.py  # Scrapy configuration
│   ├── frontend/        # React frontend (future)
│   └── equipment_testing/  # Legacy CLI
├── tests/               # Pytest test suite
│   ├── test_models.py   # Model tests
│   └── test_api.py      # API tests
├── pyproject.toml       # Project metadata & dependencies
└── README.md

Adding Dependencies

To add new dependencies to the project:

# Add a production dependency
uv add <package-name>

# Add a development dependency
uv add --dev <package-name>

# Add to specific optional group
uv add --optional iceberg <package-name>

Development Commands

This project includes a Makefile with common development tasks:

# Show all available commands
make help

# Install dependencies and pre-commit hooks (first-time setup)
make setup

# Run tests with coverage
make test

# Run linter
make lint

# Format code
make format

# Check formatting without changes
make format-check

# Run type checker
make type-check

# Run all pre-commit hooks
make pre-commit

# Clean build artifacts
make clean

Running Tests

# Run all tests
make test
# Or: uv run pytest

# Run with coverage
uv run pytest --cov=core --cov=api --cov-report=term

# Run specific test file
uv run pytest tests/test_models.py

# Run specific test
uv run pytest tests/test_models.py::test_tractor_creation

Code Quality

This project uses pre-commit hooks to automatically check code quality before commits. Pre-commit hooks are automatically installed when you run make setup or make dev.

Pre-commit Hooks

# Install pre-commit hooks (if not already installed)
make install-hooks
# Or: uv run pre-commit install

# Run hooks manually on all files
make pre-commit
# Or: uv run pre-commit run --all-files

Important: Pre-commit hooks will automatically run before each commit. If they find issues:

• Auto-fixable issues (formatting, import sorting) will be fixed automatically
• The commit will be blocked so you can review the fixes
• Stage the fixes with git add and commit again

Manual Code Quality Checks

# Format code with ruff
uv run ruff format src/ tests/

# Check formatting without making changes
uv run ruff format --check src/ tests/

# Lint code with ruff
uv run ruff check src/ tests/

# Fix linting issues automatically
uv run ruff check --fix src/ tests/

# Type checking with mypy
uv run mypy src/

Creating a New Virtual Environment

If you need to recreate the virtual environment:

# Remove existing environment
rm -rf .venv

# Create new environment and sync dependencies
uv sync

Building and Distribution

Building the Package

# Build distribution packages
uv build

This will create wheel and source distributions in the dist/ directory.

Installing the Package Locally

# Install in development mode
uv pip install -e .

API Documentation

Once the API server is running, interactive documentation is available at:

• Swagger UI: http://localhost:8000/docs
• ReDoc: http://localhost:8000/redoc

Example API Endpoints

# Health check
curl http://localhost:8000/health

# List all equipment
curl http://localhost:8000/equipment?limit=10

# Filter tractors by manufacturer
curl http://localhost:8000/equipment/tractors?make=John%20Deere

# Submit a contribution
curl -X POST http://localhost:8000/contributions \
  -H "Content-Type: application/json" \
  -d '{
    "field_name": "engine_hp",
    "proposed_value": "105",
    "notes": "Updated from manufacturer specs"
  }'

Data Model

OpenAg-DB uses Pydantic models for data validation and serialization:

• CommonEquipment: Base model with fields shared across all equipment
• Tractor: Extends CommonEquipment with tractor-specific fields
• Combine: Extends CommonEquipment with combine-specific fields
• Implement: Extends CommonEquipment with implement-specific fields

All models support:

• Automatic validation
• Type safety
• JSON serialization
• Time-travel queries (via Iceberg snapshots)

Contributing

We welcome contributions! Here's how you can help:

1. Submit Equipment Data: Use the web interface to suggest corrections
2. Improve Scrapers: Add new spiders for additional data sources
3. Enhance Models: Propose new equipment types or fields
4. Fix Bugs: Report and fix issues on GitHub

See CONTRIBUTING.md for detailed guidelines.

Roadmap

Phase 1: Core Setup ✅

• Initialize project structure
• Define Pydantic models
• Create FastAPI application
• Set up Scrapy framework

Phase 2: Data Collection

• Implement manufacturer-specific spiders (Quality Farm Supply spider)
• Add more data sources and spiders
• Add Unity Catalog Delta table integration
• Configure Unity Catalog connection
• Set up automated scraping workflow

Phase 3: API & Query Layer

• Integrate DuckDB for queries
• Add authentication/rate limiting
• Implement contribution workflow
• Deploy API to production

Phase 4: Frontend

• Build React search interface
• Add filtering and sorting
• Deploy to GitHub Pages
• Implement contribution form
• Connect to real API backend

Phase 5: Community Features

• Automated contribution review
• Data quality metrics
• Community guidelines
• Public API keys

Technology Stack

• Language: Python 3.12+
• Package Manager: uv
• Data Models: Pydantic
• API Framework: FastAPI
• Scraping: Scrapy
• Storage: Unity Catalog Delta tables
• Query Engine: DuckDB
• Frontend: React + Vite + Shadcn UI
• CI/CD: GitHub Actions
• Hosting: TBD (API) + GitHub Pages (Frontend)

Why uv?

• Fast: 10-100x faster than pip
• Reliable: Deterministic dependency resolution
• Compatible: Drop-in replacement for pip and pip-tools
• Modern: Built with Rust for performance
• All-in-one: Package management, virtual environments, and more

License

This project is licensed under the MIT License - see the LICENSE file for details.

Resources

• uv Documentation
• Python Packaging Guide
• Project Repository