Web ingestion engine: point it at a wiki, a site, or a URL list and it slices everything into typed JSON records, one page at a time — each run a native dagonizer DAG.
The full documentation is published at https://studnicky.github.io/Ripperoni/.
- Getting Started: scaffold → state → run
- Walk-through: a complete AONPRD scrape end to end
- Authoring a DAG:
DAGBuilder, placement types, builtin nodes - Configuration: the
state.jsonrun document - Plugins: declare DAGs as documents + register nodes
- Crawler: the native embedded
crawl:discoverDAG - Architecture: the runner, services, and scrapers
- DAG Diagrams: the live orchestration / crawl / page DAGs
A scrape is two authored documents — a single orchestration <name>.dag.jsonld and a <name>.state.json — run by one command. The orchestration imports plugin DAGs as embedded-dag / scatter; plugins ship their DAGs as JSON-LD documents and register their node instances. Built on @studnicky/dagonizer; output feeds Squashage, which graph-squashes the JSON into deterministic RDF.
Node.js >= 24 (matches engines.node in package.json).
Published to GitHub Packages under the @studnicky scope:
echo '@studnicky:registry=https://npm.pkg.github.com' >> .npmrc
npm install @studnicky/ripperoni# write a starter orchestration + state pair
ripperoni scaffold mysite
# edit mysite.dag.jsonld + mysite.state.json, then run it
ripperoni run mysite.dag.jsonld --state mysite.state.jsonMIT. See LICENSE.
See CHANGELOG.md and the GitHub releases.