Skip to content

Latest commit

 

History

History
662 lines (511 loc) · 21.9 KB

File metadata and controls

662 lines (511 loc) · 21.9 KB

Tests Release Coverage License: MIT

ReqPack

ReqPack is universal package-manager orchestrator. It gives you one CLI for installing, updating, auditing, snapshotting, and inspecting packages across multiple ecosystems while keeping real package-manager work inside dedicated plugins.

Examples in this README use rqp, because that is built binary name.

What ReqPack Does Well

  • One command surface for multiple ecosystems.
  • Manifest-based installs via reqpack.lua.
  • Plugin wrappers and registry-backed plugin refresh.
  • Built-in audit and SBOM export flows.
  • Remote command server and remote client profiles.
  • XDG-friendly config, cache, and data locations.
  • Coverage, test, and release automation in GitHub Actions.

Quick Start

rqp install apt curl git
rqp install npm express
rqp install apt:curl npm:express
rqp update --all
rqp list apt
rqp outdated
rqp audit .
rqp sbom --format cyclonedx-json --output sbom.json
rqp snapshot --output reqpack.lua

CLI Cheat Sheet

Task Command / flags
Show help rqp --help, rqp <command> -h
Dry-run before changes rqp install ... --dry-run
CI-safe run --non-interactive --stop-on-first-failure
Use more parallelism --jobs 8 or --jobs-max
Use custom config rqp --config ~/.config/reqpack/dev.lua ...
Use custom registry or plugin dir rqp --registry ./registry --plugin-dir ./plugins ...
Override proxy target rqp -Dproxy.java.default=gradle install java ...
Strict security gate --abort-on-unsafe --severity-threshold high --fail-on-unresolved-version
Write structured logs --verbose --log-level debug --structured-log-file ./reqpack.jsonl
Export audit file rqp audit . --format sarif --output audit.sarif
Export SBOM file rqp sbom . --format cyclonedx-json --output sbom.json
Refresh host cache rqp host refresh
Start remote server rqp serve --remote --token secret

Full flag inventory later in this README under CLI Overrides And Terminal Flags.

Requirements

ReqPack core currently targets:

  • Linux and macOS.
  • CMake 3.15+.
  • C++20 compiler.
  • Lua 5.4 development files.
  • CLI11.
  • libcurl.
  • Boost.
  • OpenSSL.
  • fmt.
  • spdlog.
  • zstd.
  • LMDB.
  • sol2 headers.
  • Git for Git-backed plugin refresh.

Self-update note: rqp update without package-manager arguments downloads matching release binary from configured release source. Release-binary users no longer need local build toolchain for self-update.

Source Layout

ReqPack source is organized by runtime area first, then by domain where core code gets larger:

  • src/main/cpp/main.cpp for entrypoint and runtime mode selection
  • src/main/cpp/cli and src/main/include/cli for CLI parsing
  • src/main/cpp/output and src/main/include/output for display, logging, and command rendering
  • src/main/cpp/plugins and src/main/include/plugins for plugin bridge and native plugin interfaces
  • src/main/cpp/core and src/main/include/core for domain-oriented core logic

Inside core, files are grouped by domain so navigation is direct:

  • config, host, manifest, archive, download
  • planning, execution, registry, packages, state
  • security, export, remote, history, plugins, common

Examples:

  • src/main/include/core/registry/registry.h
  • src/main/cpp/core/security/validator.cpp
  • src/main/cpp/core/remote/serve_remote.cpp
  • tests/unit/core/security/*
  • tests/integration/core/execution/*

Installation

Option 1: Use Release Binary

Tagged releases publish archives named rqp-<tag>-<target>.tar.gz. Each archive currently contains rqp binary and SHA256SUMS is published alongside release assets.

curl -fsSL https://raw.githubusercontent.com/Coditary/ReqPack/main/install.sh | sh

install.sh detects OS and architecture, downloads matching release binary, installs it to ~/.local/bin/rqp, writes default self-update config if missing, then runs initial rqp update --all and rqp host refresh.

If you prefer manual asset installation:

tar -xzf "rqp-vX.Y.Z-x86_64-linux.tar.gz"
chmod +x rqp
mkdir -p ~/.local/bin
cp rqp ~/.local/bin/rqp

Option 2: Use Container Image

Tagged releases also publish Linux container images to ghcr.io/coditary/reqpack for linux/amd64 and linux/arm64. Those images are now validated in Podman-backed system tests before release publish.

docker pull ghcr.io/coditary/reqpack:vX.Y.Z
docker run --rm -v "$PWD:/workspace" ghcr.io/coditary/reqpack:vX.Y.Z --help
docker run --rm -v "$PWD:/workspace" ghcr.io/coditary/reqpack:vX.Y.Z sbom .

To keep ReqPack config, cache, and self-update data between runs, mount:

docker run --rm \
  -v "$PWD:/workspace" \
  -v "$HOME/.config/reqpack:/root/.config/reqpack" \
  -v "$HOME/.cache/reqpack:/root/.cache/reqpack" \
  -v "$HOME/.local/share/reqpack:/root/.local/share/reqpack" \
  ghcr.io/coditary/reqpack:vX.Y.Z host refresh

Container note: ReqPack inside Docker can only use package-manager tools available inside container, not package managers installed on your host.

Option 3: Build From Source On Ubuntu Or Debian

sudo apt-get update
sudo apt-get install -y --no-install-recommends \
  build-essential ca-certificates cmake curl git pkg-config \
  libboost-dev libcli11-dev libcurl4-openssl-dev libfmt-dev \
  liblua5.4-dev libspdlog-dev libssl-dev libzstd-dev

git clone https://github.com/Coditary/ReqPack.git
cd ReqPack
git clone --depth 1 -b v3.3.0 https://github.com/ThePhD/sol2.git /tmp/reqpack-sol2
cmake -S . -B build -DSOL2_INCLUDE_DIR=/tmp/reqpack-sol2/include
cmake --build build --parallel --target ReqPack reqpack_test_targets
ctest --test-dir build --output-on-failure

Run binary with:

./build/rqp --help

Option 4: Build From Source On macOS

brew install cli11 fmt spdlog boost zstd openssl@3 lua@5.4 ccache

git clone https://github.com/Coditary/ReqPack.git
cd ReqPack
git clone --depth 1 -b v3.3.0 https://github.com/ThePhD/sol2.git /tmp/reqpack-sol2

BREW_PREFIX="$(brew --prefix)"
OPENSSL_PREFIX="$(brew --prefix openssl@3)"
LUA_PREFIX="$(brew --prefix lua@5.4)"
ZSTD_PREFIX="$(brew --prefix zstd)"

cmake -S . -B build \
  -DSOL2_INCLUDE_DIR=/tmp/reqpack-sol2/include \
  -DCMAKE_PREFIX_PATH="${BREW_PREFIX};${OPENSSL_PREFIX};${LUA_PREFIX};${ZSTD_PREFIX}" \
  -DOPENSSL_ROOT_DIR="${OPENSSL_PREFIX}" \
  -DREQPACK_ZSTD_LIBRARY="${ZSTD_PREFIX}/lib/libzstd.dylib" \
  -DLUA_INCLUDE_DIR="${LUA_PREFIX}/include/lua" \
  -DLUA_LIBRARIES="${LUA_PREFIX}/lib/liblua.5.4.dylib"

cmake --build build --parallel --target ReqPack reqpack_test_targets
ctest --test-dir build --output-on-failure

Everyday Usage

Common Commands

Command Purpose Example
install Install packages, local targets, or manifest contents rqp install apt curl git
remove Remove packages rqp remove npm express
update Self-update, refresh plugin wrappers, or update packages rqp update --all
search Search available packages rqp search apt curl
list List installed packages rqp list dnf
info Show package metadata rqp info brew jq
outdated Show packages with newer versions rqp outdated
ensure Ensure plugin requirements are installed rqp ensure apt brew
audit Audit packages or manifests for vulnerabilities rqp audit .
sbom Export SBOM for installed packages rqp sbom --format cyclonedx-json --output sbom.json
snapshot Write installed-package state to reqpack.lua rqp snapshot --output reqpack.lua
host refresh Refresh cached host metadata rqp host refresh
test-plugin Run hermetic plugin conformance cases rqp test-plugin --plugin demo --preset core
serve Run stdin or remote command server rqp serve --remote --token secret
remote Connect to configured remote profile rqp remote dev list apt

Install Packages

rqp install apt curl git
rqp install npm express lodash brew jq
rqp install apt:curl npm:express
rqp install brew ./my-formula.rb

ReqPack also supports manifest installs from current directory or any directory containing reqpack.lua:

rqp install .
rqp install ./myproject
rqp install /absolute/path/to/project

Batch mode from stdin is useful for automation:

printf 'install dnf curl\ninstall npm express\n' | rqp install --stdin

Remove And Update

rqp remove apt curl
rqp remove apt:curl npm:lodash

rqp update
rqp update --all
rqp update pip
rqp update pip --all
rqp update sys pip
rqp update apt:curl npm:express

Update behavior is worth knowing:

  • rqp update refreshes local registry first, then downloads ReqPack release binary for current host from configured release source.
  • rqp update --all refreshes all known plugin wrappers.
  • rqp update <system> without package names refreshes that plugin wrapper.
  • rqp update <system> --all updates all packages for that system.
  • rqp update sys <tool> updates package-manager binary itself through ReqPack wrapper layer.

Search, List, Info, Outdated

rqp search dnf python3 --arch noarch --arch x86_64
rqp list
rqp list apt
rqp info npm express
rqp outdated dnf --type doc

search, list, and outdated support repeatable --arch and --type filters where plugin supports them.

CLI Overrides And Terminal Flags

ReqPack can override many runtime settings directly in terminal instead of only through config.lua.

Global runtime overrides accepted by CLI parser:

  • config and registry: --config <path>, --registry <path>, --registry-path <path>, --plugin-dir <path>, --no-auto-load-plugins, --no-proxy-expansion, -Dproxy.<name>.default=<target>
  • execution and interaction: --dry-run, --jobs <n>, --jobs-max, --stop-on-first-failure, --no-transaction-db, --non-interactive, --archive-password <value>
  • security and audit policy: --prompt-on-unsafe, --abort-on-unsafe, --severity-threshold <low|medium|high|critical>, --score-threshold <0.0-10.0>, --osv-db <path>, --osv-feed <url-or-path>, --osv-refresh <manual|periodic|always>, --osv-refresh-interval <seconds>, --osv-overlay <path>, --ignore-vuln <id>, --allow-vuln <id>, --fail-on-unresolved-version, --prompt-on-unresolved-version, --strict-ecosystem-mapping, --include-withdrawn-in-report, --report, --report-format <none|json|cyclonedx>, --report-output <path>
  • logging: --log-level <trace|debug|info|warn|error|critical>, --log-console, --no-log-console, --verbose, --log-pattern <value>, --log-file <path>, --structured-log-file <path>, --log-capture-display, --no-log-capture-display, --log-category <name>, --backtrace
  • SBOM defaults and export behavior: --sbom-format <table|json|cyclonedx-json>, --sbom-output <path>, --sbom-no-pretty, --sbom-no-dependency-edges, --sbom-skip-missing-packages, --sbom-fail-on-missing-package

Command-specific flags you can set in terminal:

Command Flags
install --stdin
update --all
search, list, outdated --arch <value>, --type <value>
audit `--format <table
sbom `--format <table
snapshot --output <path>, --force
pack --output <path>, --payload-dir <path>, --force
test-plugin --plugin <path-or-id>, --preset <name>, --case <file.lua>, --cases <directory>, --report <file.json>
serve --stdin, --remote, --json, --http, --https, --bind <addr>, --port <n>, --token <value>, --username <name>, --password <value>, --readonly, --max-connections <n>

Important CLI notes:

  • --verbose currently turns logger console output on; it does not raise log level by itself
  • --jobs and --jobs-max cannot be combined
  • archive password can also come from REQPACK_ARCHIVE_PASSWORD
  • --token and --username/--password cannot be combined for serve --remote
  • --http and --https are reserved for future serve modes, not production-ready server protocols yet
  • --report* flags are accepted today, but current validator report hook is still partial
  • --sbom-no-pretty is accepted today, but current exporter code does not read it yet

Examples:

rqp --config ~/.config/reqpack/ci.lua --log-level debug install npm react --prompt-on-unsafe
rqp update --all --jobs-max --structured-log-file ./reqpack.jsonl
rqp audit . --osv-feed ./test-data/osv --strict-ecosystem-mapping --output audit.sarif --format sarif
rqp sbom npm react --sbom-skip-missing-packages --output sbom.json
rqp serve --remote --json --bind 127.0.0.1 --port 4545 --readonly --max-connections 4

For full per-command help, run rqp --help or rqp <command> -h.

Snapshot And Restore

snapshot writes packages tracked in ReqPack history into portable reqpack.lua manifest.

rqp snapshot --output reqpack.lua
rqp install .

Plugin Testing

test-plugin runs hermetic plugin conformance cases against one Lua plugin without touching a real package manager.

rqp test-plugin --plugin ./plugins/demo --case ./cases/install.lua
rqp test-plugin --plugin demo --cases ./tests/plugins/demo
rqp test-plugin --plugin demo --preset core --report ./plugin-test-report.json

Command notes:

  • --plugin accepts plugin id, plugin directory, or direct script path.
  • --case adds one Lua case file.
  • --cases loads all *.lua files from a directory.
  • --preset core loads built-in preset cases from <plugin>/.reqpack-test/core/.
  • --report writes JSON summary including commands, stdout, stderr, artifacts, and event payloads.

Minimal case shape:

return {
  name = "install success",
  request = {
    action = "install",
    system = "demo",
    packages = {
      { name = "curl", version = "8.0" }
    }
  },
  fakeExec = {
    {
      match = "demo-pm install curl",
      exitCode = 0,
      stdout = "ok\n",
      stderr = "",
      success = true,
    }
  },
  expect = {
    success = true,
    commands = { "demo-pm install curl" },
    stdout = { "ok\n" },
    events = { "installed", "success" },
    eventPayloads = {
      installed = "{name=curl}",
      success = "ok",
    },
  }
}

Available expect checks in MVP+:

  • success
  • commands
  • stdout
  • stderr
  • events
  • eventPayloads
  • artifacts
  • resultCount
  • resultName
  • resultVersion

Audit

ReqPack can audit installed packages, explicit package lists, or manifests.

rqp audit
rqp audit npm react
rqp audit npm:react maven:org.junit:junit
rqp audit .
rqp audit ./reqpack.lua
rqp audit --format cyclonedx-vex-json --output audit.json
rqp audit --format sarif --output audit.sarif

Audit behavior:

  • Without --output, exit code is 1 when findings exist.
  • With --output, export still succeeds with exit code 0.
  • Output formats: table, json, cyclonedx-vex-json, sarif.

SBOM

ReqPack can export installed-package inventory as terminal table, JSON, or CycloneDX JSON.

rqp sbom
rqp sbom apt npm
rqp sbom --format json
rqp sbom --format cyclonedx-json --output sbom.json

SBOM output formats:

  • table
  • json
  • cyclonedx-json

Useful SBOM flags:

  • --wide
  • --no-wrap
  • --force
  • --sbom-skip-missing-packages

Remote Mode

ReqPack has two remote pieces:

  • rqp serve --remote starts remote TCP command server.
  • rqp remote <profile> connects to profile from remote.lua.

Today, text and JSON protocols are usable. --http and --https are reserved for future server mode and should not be documented as production-ready yet.

Minimal ~/.config/reqpack/remote.lua example:

profiles = {
  dev = {
    host = "127.0.0.1",
    port = 4545,
    protocol = "auto",
    token = "secret",
  },
}

users = {
  admin = {
    token = "secret",
    isAdmin = true,
  },
}

Examples:

rqp serve --remote --bind 127.0.0.1 --port 4545 --token secret
rqp serve --remote --json --readonly --max-connections 4
rqp remote dev list apt
rqp remote dev install apt curl

Manifest Example

Project manifests use reqpack.lua.

return {
  packages = {
    { system = "dnf", name = "curl" },
    { system = "npm", name = "express", version = "4.18.0" },
  }
}

Minimal Config Example

Minimal ~/.config/reqpack/config.lua example:

return {
  interaction = {
    interactive = true,
  },
  execution = {
    jobs = 4,
    jobsMode = "fixed",
  },
  security = {
    onUnsafe = "prompt",
    severityThreshold = "critical",
  },
  registry = {
    remoteUrl = "https://github.com/Coditary/rqp-registry.git",
  },
  selfUpdate = {
    releaseApiBaseUrl = "https://api.github.com",
    releaseTag = "latest",
    linkPath = "~/.local/bin/rqp",
  },
}

Useful config areas for daily use:

  • interaction.interactive
  • execution.jobs and execution.jobsMode
  • security.onUnsafe, security.severityThreshold, security.scoreThreshold
  • registry.remoteUrl, registry.pluginDirectory, registry.sources
  • selfUpdate.repoUrl, selfUpdate.releaseApiBaseUrl, selfUpdate.releaseTag, selfUpdate.linkPath
  • sbom.defaultFormat, sbom.defaultOutputPath

Config And File Locations

ReqPack follows XDG base-directory rules.

Config:

  • $XDG_CONFIG_HOME/reqpack/config.lua
  • $XDG_CONFIG_HOME/reqpack/remote.lua
  • fallback: ~/.config/reqpack/...

Data:

  • $XDG_DATA_HOME/reqpack/plugins
  • $XDG_DATA_HOME/reqpack/repos
  • $XDG_DATA_HOME/reqpack/registry
  • $XDG_DATA_HOME/reqpack/history
  • $XDG_DATA_HOME/reqpack/rqp/state
  • $XDG_DATA_HOME/reqpack/self/bin
  • $XDG_DATA_HOME/reqpack/security/index
  • $XDG_DATA_HOME/reqpack/security/osv
  • fallback: ~/.local/share/reqpack/...

Cache:

  • $XDG_CACHE_HOME/reqpack/transactions
  • $XDG_CACHE_HOME/reqpack/security/cache
  • $XDG_CACHE_HOME/reqpack/host/info.v1.json
  • fallback: ~/.cache/reqpack/...

Binary link:

  • self-update symlink default: ~/.local/bin/rqp

Registry notes:

  • default registry remote: https://github.com/Coditary/rqp-registry.git
  • local workspace plugins/ directory is auto-used when you run rqp from repo root and no custom plugin directory is configured

Bundled or checked-in plugin examples in this repository:

  • plugins/dnf
  • plugins/maven
  • plugins/java
  • plugins/sys

Development Shortcuts

make build
make test
make test-unit
make test-smoke
make test-coverage
make profile-tests

Coverage writes Coverage.xml below build/coverage/Testing/... and summary is derived from src/main/cpp sources. Profiling writes reports below build/profile/profile-data/.

System-Test Helpers

ReqPack now has dedicated artifact-level system tests in addition to CTest-based unit and integration tests.

  • Linux system tests package rqp, build temporary runtime image in Podman, then run end-to-end fixture checks.
  • macOS system tests package rqp, extract bundle, then run same fixture flow natively.

Local examples:

bash .github/scripts/configure-build.sh build linux -DSOL2_INCLUDE_DIR=/tmp/reqpack-sol2/include
bash scripts/package_release_bundle.sh build/rqp /tmp/rqp-test.tar.gz linux
bash .github/scripts/run-system-tests-linux.sh /tmp/rqp-test.tar.gz "$PWD" x86_64-linux
bash .github/scripts/configure-build.sh build macos \
  -DSOL2_INCLUDE_DIR=/tmp/reqpack-sol2/include \
  -DCMAKE_PREFIX_PATH="${BREW_PREFIX};${OPENSSL_PREFIX};${LUA_PREFIX};${ZSTD_PREFIX}" \
  -DOPENSSL_ROOT_DIR="${OPENSSL_PREFIX}" \
  -DREQPACK_ZSTD_LIBRARY="${ZSTD_PREFIX}/lib/libzstd.dylib" \
  -DLUA_INCLUDE_DIR="${LUA_PREFIX}/include/lua" \
  -DLUA_LIBRARIES="${LUA_PREFIX}/lib/liblua.5.4.dylib"
bash scripts/package_release_bundle.sh build/rqp /tmp/rqp-test.tar.gz macos
bash .github/scripts/run-system-tests-macos.sh /tmp/rqp-test.tar.gz "$PWD"

CI/CD

Repo ships three documentation-visible automation signals:

  • Tests: artifact-first build matrix for x86_64-linux, aarch64-linux, and aarch64-darwin, then fan-out unit, integration, and system verification.
  • Release: tag-driven artifact-first packaging flow for x86_64-linux, aarch64-linux, and aarch64-darwin plus SHA256SUMS and Linux GHCR image publication.
  • Coverage: Linux coverage run that updates README badge from GitHub Actions on pushes to main and uploads report artifacts for pull requests.

Release workflow builds each target once per workflow run, then reuses those exact tested outputs for:

  • archived build-tree CTest replay
  • Linux Podman system tests
  • Darwin native system tests
  • GHCR image publication
  • GitHub release asset publication

Coverage badge note: it will show pending until first successful Coverage workflow run on main writes .github/badges/coverage.json.

Relevant workflow files:

  • .github/workflows/tests.yml
  • .github/workflows/release.yml
  • .github/workflows/coverage.yml

Repository Layout

  • src/main/cpp: core implementation.
  • src/main/include: public and internal headers.
  • plugins/: built-in or locally checked-in plugins.
  • tests/: unit, integration, system, coverage, and profiling helpers.
  • .github/workflows/: CI, coverage, and release automation.

Contributing

Contribution guide lives in CONTRIBUTING.md. If you want to improve core behavior, add tests, report bugs, or propose plugin/runtime changes, start there.

License

ReqPack is licensed under 0BSD. Use it, modify it, fork it, ship it. No attribution requirements, no field-of-use restrictions, no warranty.