fccentrummap

FC Centrum map data pipeline and visualization app.

Single source of truth

Canonical project behavior and architectural constraints live in:

• openspec/specs/

Use other Markdown files for explanation and workflow help, not as competing sources of truth.

Repo layout

.
├── data/        # local/generated pipeline artifacts
├── docs/        # supporting documentation
├── openspec/    # canonical specs and change history
├── scraper/     # Go module for scraping and export
└── viz/         # frontend map application

Canonical commands

make scrape
make export
make build
make check
make setup-hooks

Unified scrape CLI (stage-based)

The unified CLI is wired with urfave/cli/v3 and uses subcommands plus idiomatic --help output.

cd scraper

go run ./cmd/scrape --help
go run ./cmd/scrape <stage> --help

# Preflight env + schema init (fails fast on missing required API keys)
go run ./cmd/scrape init --db-path ../data/spots.db --reset

# Current Gemini-first SQLite stages
go run ./cmd/scrape collect-article-urls --io sqlite --db-path ../data/spots.db --article-url "<FCCENTRUM_ARTICLE_URL>"
go run ./cmd/scrape fetch-articles --io sqlite --db-path ../data/spots.db
# To regenerate Gemini-derived data while preserving fetched article sources:
go run ./cmd/scrape init --db-path ../data/spots.db --reset-gemini-derived
# Prompt/raw/parsed artifacts are written as deterministic diagnostics.
# Existing parsed artifacts are reused; add --force to regenerate and call Gemini again.
go run ./cmd/scrape extract-spots-gemini-direct --io sqlite --db-path ../data/spots.db --out-dir ../data/gemini-direct --model gemini-3.1-pro-preview
go run ./cmd/scrape geocode-spots --io sqlite --db-path ../data/spots.db

# Optional file-mode geocode input path
go run ./cmd/scrape geocode-spots --io file --in ../data/stages/geocode-input/<identity>.json

# Export smoke test (can succeed even with null payload during scaffold phase)
go run ./cmd/scrape export-data --io sqlite --db-path ../data/spots.db --out ../viz/public/data/spots.json --spot-source gemini-direct

Stage mode support matrix

Stage	--io sqlite	--io file
init	Supported	Not supported (error)
collect-article-urls	Supported	Not implemented (error; use sqlite)
fetch-articles	Supported	Not implemented (error; use sqlite)
extract-spots-gemini-direct	Supported	Not implemented (error; use sqlite)
geocode-spots	Supported	Supported
export-data	Supported	Not implemented (error; use sqlite)

Notes

Legacy standalone command entrypoints were removed in favor of cmd/scrape stage subcommands. Deprecated stdlib flag parsing paths and legacy invocation shims were intentionally dropped during the urfave/cli v3 migration.

Commit messages

Use Conventional Commits:

• <type>[optional scope][!]: <description>
• Example: feat(scraper): add youtube audio acquisition stage

Enable local commit-message validation hooks with:

make setup-hooks

Key spec areas

• openspec/specs/project-layout/spec.md
• openspec/specs/developer-workflow/spec.md
• openspec/specs/web-scraper/spec.md
• openspec/specs/sqlite-storage/spec.md
• openspec/specs/static-data/spec.md
• openspec/specs/frontend-portability/spec.md

Supporting docs

• docs/architecture.md
• docs/development.md
• docs/scraper-v2-schema-migration.md
• docs/frontend-portability.md
• viz/README.md