Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions .gitleaks.toml
Original file line number Diff line number Diff line change
Expand Up @@ -39,3 +39,18 @@ ever existed. Matched by `generic-api-key` because the fixtures are named `secre
Keyed on the literal and nothing else — the file is still scanned for everything else.
"""
regexes = ['''AKIAABCDEFGHIJKLMNOP''']

[[allowlists]]
description = """
A high-entropy nonsense string in the keyword-proximity tests
(packages/detect/tests/test_signals.py, introduced in e6b1317). It is the *subject*
of those tests rather than a credential: they assert that `oauth` clipped by a window
edge does not count as the keyword `auth`, which needs a token entropic enough for
`generic-api-key` to match — so the fixture and the finding have the same cause.

Allowlisted here rather than with an inline `# gitleaks:allow` because e6b1317 is
already in history and the CI job scans all of it; editing the file today would not
change what that commit contains. Keyed on the literal, so the file is still scanned
for everything else.
"""
regexes = ['''8f3Kd0aQ2mVx7Lp9Zr4Ts6''']
8 changes: 8 additions & 0 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
@@ -1,3 +1,11 @@
# Vendored frontend assets and the brand marks are byte-pinned: static/assets.lock.json
# records a sha384 of each, base.html emits it as an `integrity=` attribute, and
# apps/api/tests/test_web_invariants.py checks the two agree. A whitespace hook that
# "fixes" a minified bundle changes those bytes, so the browser blocks the script and
# the page silently loses its behaviour — with a diff that looks like nothing happened.
# Leave them exactly as web/vendor_assets.py wrote them.
exclude: ^apps/api/src/iceberg_api/web/static/(js/vendor|css/vendor|fonts|img)/

repos:
- repo: https://github.com/pre-commit/pre-commit-hooks
rev: v6.0.0
Expand Down
5 changes: 3 additions & 2 deletions ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,8 +160,9 @@ connector credentials, full audit trail on finding state changes, and RBAC on ev
- **M1 — Control plane MVP:** auth, sources/schedules, scan orchestration, findings/triage.
- **M2 — Detection + Confluence:** detection engine, connector framework, Confluence connector,
engine worker.
- **M3 — Web UI:** HTMX/Alpine screens.
- **M4 — Notifications & prod deploy:** notifications, Helm, hardening.
- **M3 — Web UI:** HTMX/Alpine screens, served by the API at the application root and driving the
same route handlers the JSON API exposes ([`docs/web.md`](./docs/web.md)).
- **M4 — Notifications & prod deploy:** notification dispatch, Helm, hardening.

Non-goals for MVP: Jira/SMB connectors, incremental/delta scanning, image OCR, external ticket
creation, multi-tenancy.
21 changes: 16 additions & 5 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,13 @@ IcebergSST — a secret-scanning platform for non-git enterprise sources (Conflu
shares). See [`ARCHITECTURE.md`](./ARCHITECTURE.md) for the design spec and [`docs/adr/`](./docs/adr/)
for decision rationale.

**Where the code is:** M0–M2 are built. `packages/core` (config, DB session, secret store,
**Where the code is:** M0–M3 are built. `packages/core` (config, DB session, secret store,
redaction, fingerprinting, the SQLModel schema), `packages/detect` (rule packs, the detection
engine), `packages/connectors` (Confluence, extraction, the sandbox), `apps/api` (auth, sources,
scans, findings, engine-facing routes, scheduler/maintenance), `apps/engine` (the Dramatiq
worker), Alembic, and the docker-compose stack all exist and are tested. The HTMX/Alpine web UI
(M3) and notifications/Helm (M4) are still to come — `docs/backlog.md` tracks those.
scans, findings, notification channels, engine-facing routes, scheduler/maintenance, and the
server-rendered web console under `apps/api/src/iceberg_api/web/`), `apps/engine` (the Dramatiq
worker), Alembic, and the docker-compose stack all exist and are tested. Notification *dispatch*
and Helm (M4) are still to come — `docs/backlog.md` tracks those.

## Non-negotiable invariants

Expand All @@ -28,8 +29,13 @@ change the corresponding ADR:
3. **Detection logic in code, tuning in data.** Rules live in versioned rule packs
(`packages/detect`). Suppressions/allowlists live in the DB and are analyst-editable.
4. **API-first.** The FastAPI OpenAPI schema is the contract; the HTMX/Alpine UI is just another
client of the same routes.
client of the same routes. Concretely: a web route resolves the RBAC dependency its API
counterpart declares and then calls that handler as a function. Nothing under
`iceberg_api.web` queries the database (see `docs/web.md`).
5. **Single-org.** No `tenant_id`, no row-level tenancy.
6. **No inline script or style in a template.** The console runs under
`script-src 'self'` with no `'unsafe-inline'`/`'unsafe-eval'`; Alpine components go in
`static/js/tags.js` and server data travels through JSON islands.

## Stack

Expand All @@ -53,6 +59,11 @@ Prefer extending these over adding prose:
configuration, no master key, and stay scalable; `.env.example` documents every variable compose
interpolates.
- `apps/api/tests/test_migrations.py` — migrations apply, match the metadata, and reverse.
- `apps/api/tests/test_web_invariants.py` — no `iceberg_api.web` module imports the ORM or builds a
query, every mutating browser route is CSRF-protected, no HTML endpoint appears in the OpenAPI
document, and every vendored asset matches its `assets.lock.json` integrity.
- `apps/api/tests/test_web_shell.py` — no template carries an inline `<script>`/`<style>` or an
`onclick=`-style handler, and the CSP never grows `'unsafe-inline'`/`'unsafe-eval'` for scripts.

## Working style

Expand Down
18 changes: 12 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -7,10 +7,11 @@ Most secret-scanning tooling is git-centric. IcebergSST targets the *non-git* lo
credentials quietly accumulate — **Confluence**, **Jira**, and **network file shares** — with an
API-first management plane and horizontally scalable, isolated scanner engines.

> ⚠️ **Status: M0 foundations.** The workspace, core package (config, secret store, redaction,
> fingerprinting, schema), and the local container stack exist; the API routes, detection engine,
> connectors, and UI do not yet. See [`ARCHITECTURE.md`](./ARCHITECTURE.md) and [`docs/`](./docs/)
> for the spec, and the GitHub milestones/issues for what is left.
> ⚠️ **Status: M0–M3 built.** The core package, the control-plane API, the detection engine, the
> Confluence connector, the engine worker, the local container stack, and the web console all exist
> and are tested. Notification *dispatch* and the Helm chart (M4) do not yet. See
> [`ARCHITECTURE.md`](./ARCHITECTURE.md) and [`docs/`](./docs/) for the spec, and the GitHub
> milestones/issues for what is left.

## What it does

Expand All @@ -30,7 +31,7 @@ API-first management plane and horizontally scalable, isolated scanner engines.
|-------|------------|
| Control-plane API | FastAPI (API-first; OpenAPI is the contract) |
| ORM / database | SQLModel over PostgreSQL |
| Web UI | Server-rendered HTML driven by HTMX + Alpine.js |
| Web console | Server-rendered Jinja + HTMX + Alpine, under a strict CSP ([`docs/web.md`](./docs/web.md)) |
| Job queue | Redis + Dramatiq |
| Scanner engines | Separate Dramatiq worker processes; consume jobs from Redis, **POST results back to the API** (no database credentials) |
| Auth | OIDC/SSO + RBAC (admin / analyst / viewer) |
Expand All @@ -51,6 +52,10 @@ make scale N=3 # more engine replicas
make down # stop; `make destroy` also drops the data volume
```

The console is then at <http://localhost:8000/> and the OpenAPI docs at `/docs`. Signing in needs
OIDC configured in `.env`; the first person to sign in lands as a viewer unless
`ICEBERG_BOOTSTRAP_ADMIN_SUBJECT` names them.

`make check` runs what CI runs: `ruff`, `mypy`, and `pytest`. `make help` lists every target.

## Repository layout
Expand All @@ -61,7 +66,8 @@ apps/engine Dramatiq scanner worker
packages/core shared models, config, secret-store, fingerprinting, redaction
packages/detect rule packs + detection engine
packages/connectors connector interface + Confluence (Jira/SMB later)
web/ HTMX templates + Alpine components (M3)
apps/api/…/web the console: Jinja templates, Alpine components, design system
web/ frontend asset vendoring (no Node toolchain)
deploy/compose docker-compose development stack
deploy/docker role Dockerfiles (api, engine)
deploy/helm Helm chart (M4)
Expand Down
3 changes: 3 additions & 0 deletions apps/api/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,9 @@ dependencies = [
"croniter>=6.0.0",
"fastapi>=0.139.2",
"iceberg-core",
# The M3 web UI's template engine. Imported directly by iceberg_api.web —
# declared rather than leaned on transitively through fastapi's extras.
"jinja2>=3.1.6",
"psycopg[binary]>=3.2.12",
# Starlette needs it to parse form bodies: the CSRF token arrives that way
# from HTMX posts, and the M3 UI is all forms.
Expand Down
24 changes: 22 additions & 2 deletions apps/api/src/iceberg_api/app.py
Original file line number Diff line number Diff line change
@@ -1,11 +1,14 @@
"""FastAPI application factory.

Establishes what every route inherits — JSON logs with a per-request correlation
id (#67) and the Prometheus exposition endpoint — and mounts the routers.
id (#67), the browser security headers (#53), and the Prometheus exposition
endpoint — and mounts the routers.

Human-facing routes live under ``/api/v1`` from day one (docs/api.md
§ Conventions); ``/healthz`` and ``/metrics`` stay unversioned because operators
and scrapers are not API clients.
and scrapers are not API clients. The M3 web UI mounts at the root and is
excluded from the OpenAPI schema: the schema is the API's contract, and HTML
endpoints in it would describe a second one (docs/web.md).
"""

import re
Expand All @@ -15,6 +18,7 @@

import structlog
from fastapi import FastAPI, Request, Response
from fastapi.staticfiles import StaticFiles
from iceberg_core import metrics as _metrics # noqa: F401 # registers iceberg_* series
from iceberg_core.config import ApiSettings
from iceberg_core.logging import configure_logging
Expand All @@ -25,11 +29,16 @@
from iceberg_api.findings.routes import router as findings_router
from iceberg_api.findings.suppression_routes import router as suppressions_router
from iceberg_api.maintenance import background_maintenance
from iceberg_api.notifications.routes import router as notifications_router
from iceberg_api.rules import router as rules_router
from iceberg_api.scans.routes import router as scans_router
from iceberg_api.sources.routes import router as sources_router
from iceberg_api.sources.schedule_routes import router as schedules_router
from iceberg_api.users.routes import router as users_router
from iceberg_api.web import errors as web_errors
from iceberg_api.web.assets import STATIC_DIR
from iceberg_api.web.routes import router as web_router
from iceberg_api.web.security import security_headers_middleware

API_PREFIX = "/api/v1"

Expand Down Expand Up @@ -70,6 +79,10 @@ async def lifespan(_app: FastAPI) -> AsyncIterator[None]:

app = FastAPI(title="IcebergSST API", lifespan=lifespan)

# Registered before the request-context middleware, so it runs outermost and
# stamps the headers on error responses the inner layers produce too.
app.middleware("http")(security_headers_middleware(settings))

@app.middleware("http")
async def request_context(
request: Request, call_next: Callable[[Request], Awaitable[Response]]
Expand All @@ -94,11 +107,18 @@ async def metrics() -> Response:
app.include_router(auth_router, prefix=API_PREFIX)
app.include_router(engines_router, prefix=API_PREFIX)
app.include_router(findings_router, prefix=API_PREFIX)
app.include_router(notifications_router, prefix=API_PREFIX)
app.include_router(rules_router, prefix=API_PREFIX)
app.include_router(scans_router, prefix=API_PREFIX)
app.include_router(schedules_router, prefix=API_PREFIX)
app.include_router(sources_router, prefix=API_PREFIX)
app.include_router(suppressions_router, prefix=API_PREFIX)
app.include_router(users_router, prefix=API_PREFIX)

# The browser surface, last: its routes live at the root, and mounting them
# after the API guarantees no web path can shadow an /api/v1 one.
app.mount("/static", StaticFiles(directory=STATIC_DIR), name="static")
app.include_router(web_router)
web_errors.register(app)

return app
1 change: 1 addition & 0 deletions apps/api/src/iceberg_api/notifications/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
"""Notification channels: where newly-opened findings get announced (#72)."""
Loading
Loading