Skip to content

pg push service#547

Merged
wesm merged 2 commits into
kenn-io:mainfrom
mxfeinberg:pg-push-service
May 29, 2026
Merged

pg push service#547
wesm merged 2 commits into
kenn-io:mainfrom
mxfeinberg:pg-push-service

Conversation

@mxfeinberg
Copy link
Copy Markdown
Contributor

@mxfeinberg mxfeinberg commented May 26, 2026

Problem

I track token usage across AI tools with agentsview, and lately I've been
running more remote agent tools on machines other than my primary dev box.
Each of those machines records its own sessions locally. The Postgres support
(pg push / pg serve) already lets me review everything together, but
keeping the shared database current meant either remembering to run pg push
by hand on each machine or wiring up a bespoke cron job per host. I wanted a
touchless way to sync data from multiple machines into one place so remote
agent token usage shows up in my primary dashboard automatically, without
maintaining custom cron entries everywhere.

I also made some updates to hermes specific code because although tokens
were being passed through, the associated costs were not being aggregated
properly.

pg push service

A long-running auto-push daemon and a first-class way to install it as an OS
service, so a recorder machine keeps the shared Postgres current on its own.

agentsview pg push --watch start a foreground daemon that pushes to Postgres
shortly after sessions change, with a periodic floor as a safety net:

  • A file watcher (reusing the same machinery as serve) coalesces change
    events over a debounce window (--debounce, default 30s) and triggers a
    local sync + incremental push.
  • A periodic floor (--interval, default 15m) pushes regardless of file
    events and covers any directories the watch budget couldn't cover.
  • Pushes are serialized by a single goroutine and connect to Postgres lazily,
    reconnecting on error. A transiently unreachable database is logged and
    retried on the next trigger rather than crashing the daemon.
  • A single-instance lock prevents two watchers from racing on the push
    watermarks. On shutdown (SIGINT/SIGTERM) it performs one bounded final flush.
  • It reads the same [pg] config and project filters as pg push, and
    honors result_content_blocked_categories so the push path no longer
    diverges from serve.

agentsview pg service install|uninstall|status|start|stop|logs
installs and manages the daemon as a per-user OS service:

  • launchd LaunchAgent on macOS, systemd --user unit on Linux, behind a small
    platform abstraction with pure (golden-tested) unit-file rendering.
  • install validates that the Postgres DSN is resolvable before creating the
    service, and surfaces the systemd linger requirement so the service keeps
    running on headless boxes after logout.
  • status reports the manager state plus the last successful push time;
    logs -f tails the daemon log and survives log rotation.

The daemon reads the DSN from the existing config file (no credentials are
copied into unit files), so protecting config.toml is sufficient.

Notes

  • Platform support targets macOS (launchd) and Linux (systemd --user); the
    daemon command itself is cross-platform Go.
  • The generated unit always pins AGENTSVIEW_DATA_DIR to the install-time data
    dir, so the service resolves the same config regardless of the environment it
    starts in.
  • The watch daemon's initial resync is push-oriented and skips the serve-side
    Vacuum/BackfillSignals steps, since signal recomputation happens on the
    serve side.
  • A windows service was not added because I don't currently have access to a
    windows machine that I could use for testing.

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (6a35195)

No Medium, High, or Critical issues found.

All review outputs are clean; no actionable findings to include.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (5101e30)

High confidence issues remain; PR should not merge until the import break is fixed.

High

  • cmd/agentsview/pg_service.go:14
    New files import go.kenn.io/agentsview/internal/..., but the module is github.com/wesm/agentsview, with no module-path change in the diff. These packages will not resolve.
    Fix: Use the repository’s existing import path, e.g. github.com/wesm/agentsview/internal/config, in all new files.

Medium

  • cmd/agentsview/pg_service_manager.go:26
    buildServiceSpec validates pg.url using the installer’s current environment, but the rendered launchd/systemd units only preserve AGENTSVIEW_DATA_DIR. A config like pg.url = "${PG_URL}" installs successfully, then the service fails at runtime because PG_URL is absent.
    Fix: Either reject env-dependent PG URLs during service install or render the required PG environment variables into the service unit.

  • internal/pidlock/pidlock.go:26
    Acquire is not atomic: two concurrent processes can both see no live lock, write the shared .tmp file, rename over the lock path, and both return success.
    Fix: Acquire with an atomic primitive such as os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, ...), removing stale locks and retrying as needed.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (93270cc)

Medium-risk issues remain around single-instance locking and service environment propagation.

Medium

  • internal/pidlock/pidlock.go:41
    Acquire is not atomic: after checking/removing a stale lock, it writes a temp file and renames it over the target. Two near-simultaneous starts can both acquire the “single-instance” lock, allowing multiple pg push --watch daemons and potentially concurrent resyncs against the same temp DB path.
    Fix: Create the lock with an atomic primitive such as os.OpenFile(path, os.O_CREATE|os.O_EXCL|os.O_WRONLY, ...) or an OS file lock, and avoid releasing a lock file that no longer belongs to the current process.

  • cmd/agentsview/pg_service_manager.go:27
    Service install validates the resolved PG URL using the current environment, but the generated launchd/systemd units only preserve AGENTSVIEW_DATA_DIR. Configurations that use AGENTSVIEW_PG_URL or ${VAR} expansion can pass install and then fail when the background service starts without those variables.
    Fix: Persist the required resolved PG settings into the service environment/unit, support an environment file, or reject env-dependent PG URLs during service install with a clear error.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (9b511a2)

High severity issue found.

High

  • cmd/agentsview/pg_service_systemd.go:40

    systemdManager.render interpolates spec.BinPath and spec.DataDir directly into a systemd unit using quoted strings. systemd unit files are not shell scripts, but newlines and quote characters inside these values can break out of the intended directive and inject additional unit directives.

    Since spec.DataDir comes from configuration, a crafted data dir containing quote/newline-injected directives could cause agentsview pg service install to write, enable, and start a user service that runs attacker-chosen commands as the current user.

    Suggested remediation: escape or quote values using systemd-safe syntax, or reject unsafe path characters such as \n, \r, ", and control bytes before writing the unit. Cover both ExecStart and Environment with tests.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (1929c73)

High: PR has compile-breaking import path issues and a service environment persistence bug that can make installed services run with different config than validated.

High

  • cmd/agentsview/pg_watch.go:13
    The new command files import internal packages via go.kenn.io/agentsview/..., but this repository’s module path is github.com/wesm/agentsview. Because these are internal packages, the mismatched import path will break compilation for the added watch/service code.
    Fix: Change the new imports in pg_watch.go, pg_service.go, pg_service_manager.go, and their tests to github.com/wesm/agentsview/internal/....

Medium

  • cmd/agentsview/pg_service_manager.go:20
    Service installation only rejects env-dependent pg.url, but the installed service environment only preserves AGENTSVIEW_DATA_DIR. If the install-time config depends on other env vars such as AGENTSVIEW_PG_SCHEMA, AGENTSVIEW_PG_MACHINE, or agent session directory env vars, foreground install validation can succeed while the background service later runs with different schema, machine, or source directories.
    Fix: Either reject all env-derived config that affects service runtime, or render the required resolved environment values into both launchd and systemd units.

Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (b559140)

No Medium, High, or Critical findings were reported.

All completed reviews found no actionable issues; one review was skipped due to quota exhaustion.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

Note: gemini review skipped (agent quota exhausted)

@mxfeinberg mxfeinberg marked this pull request as ready for review May 26, 2026 21:57
@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 26, 2026

roborev: Combined Review (bfd3d7d)

No Medium, High, or Critical findings were reported.

All completed reviews found no issues; one review was skipped due to quota exhaustion.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

Note: gemini review skipped (agent quota exhausted)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 27, 2026

roborev: Combined Review (84da010)

High: PR has a compile-blocking module import issue; also includes Medium concerns around Linux service logs and a flaky stdin test.

High

  • cmd/agentsview/pg_service.go:13
    New files import go.kenn.io/agentsview/internal/..., but this repository’s module path is github.com/wesm/agentsview. Go will treat those as external internal-package imports and fail to compile.

    Fix: Change the new imports to the repository module path, e.g. github.com/wesm/agentsview/internal/config, consistently across the added files and tests.

Medium

  • cmd/agentsview/pg_service_systemd.go:33
    The systemd unit ignores spec.LogPath, while pg service logs tails pg-watch.log. stderr/stdout fatal messages from the service go to journald instead of that file, so Linux users can get an empty or incomplete service log.

    Fix: Either configure StandardOutput/StandardError to append to spec.LogPath, or make the systemd log command read from journald.

  • cmd/agentsview/pg_service_test.go in TestPromptYesNo
    The test temporarily overrides the global os.Stdin variable. This introduces a race condition and can cause flaky failures if other tests or goroutines access os.Stdin concurrently.

    Fix: Refactor promptYesNo in cmd/agentsview/pg_service.go to accept an io.Reader, passing os.Stdin in production code and a pipe/mock reader in tests.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 27, 2026

roborev: Combined Review (0084777)

Medium: tailFile can miss new log data after truncation or get stuck after rotation.

Medium

  • Location: cmd/agentsview/pg_service.go:696-724
    Problem: tailFile only checks the already-open file descriptor. If the log is truncated and rewritten past the current offset, the prefix of the new log can be skipped. If the log is rotated, the open descriptor can keep pointing at the old unlinked inode and remain at EOF indefinitely.
    Fix: Periodically stat the path itself and reopen when the path’s file identity changes or when truncation is detected.

Low-severity findings were omitted per instructions.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 28, 2026

roborev: Combined Review (ad32c91)

No Medium, High, or Critical findings were reported across the reviews.

All reported findings were below the requested severity threshold and have been omitted.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@mxfeinberg
Copy link
Copy Markdown
Contributor Author

@wesm , I just pushed the last couple of tweaks to this work. As mentioned in my top comment, I added a system service (for MacOS and Linux) intended to push data from remote machines to a central pg instance without the need for a dedicated cron or manual pushes. It was motivated by a deployment of hermes that I just started using. I also added some other tweaks to the hermes parser because none of its tokens were being priced properly via my deployment.

Let me know if there's any interest in merging these changes or changes like this! No worries if not, but I've been a fan of using agentsview and roborev per a recommendation from Hugh, and I thought I could contribute back.

@wesm wesm force-pushed the pg-push-service branch from ad32c91 to b4493cb Compare May 28, 2026 19:35
@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 28, 2026

roborev: Combined Review (b4493cb)

No Medium, High, or Critical findings were reported.

The only finding was Low severity and has been omitted per instructions.


Synthesized from 3 reviews (agents: codex, gemini | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 28, 2026

roborev: Combined Review (3345c2b)

Summary verdict: One medium issue needs attention before merge.

Medium

  • internal/pricing/fallback.go:5: FallbackPricing() adds new gpt-5.5 and openrouter/owl-alpha fallback rows, but FallbackVersion was not bumped. Existing databases that already stored the current fallback seed version will skip re-seeding, so these new rows will not be added and Hermes gpt-5.5 usage can remain unpriced.

    Fix: Bump FallbackVersion whenever changing the fallback pricing list, and consider adding a test that fails when fallback rows change without a version change.


Synthesized from 2 reviews (agents: codex | types: default, security)

@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 29, 2026

roborev: Combined Review (8a04578)

No Medium, High, or Critical findings were reported.

No actionable findings to include under the requested severity threshold.


Synthesized from 2 reviews (agents: codex | types: default, security)

mxfeinberg and others added 2 commits May 29, 2026 15:30
Adds an auto-push mode that keeps PostgreSQL in sync without manual runs:

- pg push --watch: long-lived daemon that does an initial catch-up push,
  then pushes on debounced file changes plus a periodic floor tick. It
  lazily connects and reconnects after errors so a transiently
  unreachable database never crashes the daemon, and bounds the shutdown
  flush so process exit cannot hang.
- pg service: install/uninstall/status/start/stop/logs that run the
  watcher as a launchd LaunchAgent (macOS) or systemd --user unit
  (Linux). Service units only receive AGENTSVIEW_DATA_DIR, so install
  rejects env-dependent pg.url values and warns about other uninherited
  runtime env vars. Service unit values are validated to prevent
  injection.
- internal/pidlock: single-instance file lock so only one watcher runs
  per data dir.
- postgres push: sync usage_events independently of transcript messages
  so zero-message sessions' token/cost accounting still reaches PG.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ates

- hermes: stop treating cost_status "included" as a confident $0 when
  cost_source is none/empty (Hermes's default for models it does not
  price, e.g. gpt-5.5). Such rows now leave cost_usd nil so they are
  catalog-priced.
- hermes: populate the session-aggregate token columns (total output,
  peak context) from Hermes's own state.db accounting.
- pricing: add gpt-5.5 and openrouter/owl-alpha fallback rates and bump
  FallbackVersion so the rates re-seed.
- db: bump dataVersion to 30 so existing Hermes rows re-parse and pick up
  catalog pricing.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@wesm wesm force-pushed the pg-push-service branch from 8a04578 to e54f4e7 Compare May 29, 2026 21:19
@roborev-ci
Copy link
Copy Markdown

roborev-ci Bot commented May 29, 2026

roborev: Combined Review (e54f4e7)

Verdict: No Medium, High, or Critical findings were reported.

No actionable findings at or above Medium severity.


Synthesized from 2 reviews (agents: codex | types: default, security)

@wesm
Copy link
Copy Markdown
Member

wesm commented May 29, 2026

Merging this! Thank you

@wesm wesm merged commit f47f17d into kenn-io:main May 29, 2026
14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants