Skip to content

feat: sync 3 weeks of taint research + structural split (batou-core / batou-rules) - #22

Merged
Zeerg merged 7 commits into
mainfrom
sync/content-from-commercial
Apr 7, 2026
Merged

feat: sync 3 weeks of taint research + structural split (batou-core / batou-rules)#22
Zeerg merged 7 commits into
mainfrom
sync/content-from-commercial

Conversation

@Zeerg

@Zeerg Zeerg commented Apr 7, 2026

Copy link
Copy Markdown
Contributor

This PR rolls up ~3 weeks of taint-research work from the internal
development tree into the public repo, and at the same time adopts
the batou-core/ + batou-rules/ split that the internal tree has
been using. It combines two phases of the migration into a single
reviewable change.

Phase 1 — Structural split (commit 706dd23)

The old flat layout (cmd/, internal/, tools/, testdata/) is
replaced with a two-module workspace:

batou-core/  engine, analyzers, scanner, taint, hints, hook, ledger,
             reporter, findings, fpfilter, graph, suppress, testutil,
             cmd/batou
batou-rules/ rules, testdata, tools, testutil
go.work      workspace pinning both modules
  • batou-core/testdata is a symlink to ../batou-rules/testdata so
    scanner tests still find fixtures.
  • batou-rules/testutil was split out from batou-core/testutil to
    break the import cycle that otherwise forms when rules test against
    their own catalog.
  • All file moves were done with git mv to preserve history.

Phase 2 — Content sync (commits bddabe4, 60b3744, d5255e1)

The actual taint-research changes on top of the restructure:

Taint catalog expansion (16 languages)

New / expanded sources, sinks, and sanitizers across C, C++, C#, Go,
Groovy, Java, JS/TS, Kotlin, Lua, Perl, PHP, Python, Ruby, Rust,
Swift, and Zig. Run python3 batou-rules/tools/check_rules.py --taint
post-merge for per-language counts.

New structural rules

  • rules/injection/injection_c.go, rules/injection/sql_fragment.go
  • rules/xss/xss_c.go
  • rules/kotlin/kotlin_bench.go, rules/swift/swift_bench.go
  • rules/javautil.go, rules/pyutil.go

Scanner / engine improvements

  • Per-language false-positive filters (scanner/{go,js,php,py,ruby, csharp,rust}_fpfilter.go)
  • scanner/regexenrich.go — enrichment pass after regex stage
  • scanner/redteam_test.go, scanner/redteam_architecture_test.go,
    scanner/failure_mode_test.go, scanner/suppress_only_test.go
  • Per-language bench tests
    ({c,cpp,csharp,go,groovy,java,js,kotlin,lua,perl,php,ruby,rust, swift}bench_test.go)
  • scanner/owasp_bench_test.go — OWASP Benchmark harness
  • scanner/spring_bench_test.go, scanner/category_suppress_test.go,
    scanner/claims_test.go, scanner/cap.go, scanner/e2e_test.go

tsflow engine

  • taint/tsflow/callback.go / _test.go
  • taint/tsflow/summary.go / _test.go
  • taint/tsflow/fp_sanitizer_test.go, fp_switch_test.go
  • taint/tsflow/debug_test.go, debug_switch_test.go
  • taint/tsflow/rust_sanitizer_test.go, rust_ssrf_xss_test.go
  • taint/zig_test.go

Bench fixtures

124 new vulnerable + safe fixtures across all supported languages.
The 27 OSS-only per-language bench fixtures
(testdata/fixtures/bench/{cpp,csharp,groovy,kotlin,lua,perl,rust, swift,zig}/) were preserved — they were never in the internal
tree and are kept as-is.

Comment scrubs

Four files had comments referencing internal infrastructure names
(ClickHouse sink, agent-monorepo IPC, watchdog). These were rewritten
to neutral language (external dashboards, downstream consumers,
metrics sinks) with no functional code changes:

  • batou-core/cmd/batou/main.go:103,184
  • batou-core/findings/store.go:406
  • batou-core/findings/store_test.go:444,473
  • batou-rules/rules/rule.go:132

Makefile / install.sh

Phase 1 left Makefile and install.sh pointing at ./cmd/batou and
./internal/scanner/. Both are now updated to the
./batou-core/cmd/batou and ./batou-core/scanner/ paths, and pick
up the OWASP-benchmark helper targets from the internal Makefile.

What was excluded

  • Nothing from the internal batou-commercial/ tree (lifecycle
    pipeline, watchdog integration, commercial rulepacks) — only the
    shared batou-core and batou-rules trees were synced.
  • Internal tooling references in comments were rewritten, not copied.

Verification

CGO_ENABLED=1 go build ./batou-core/... ./batou-rules/...   # clean
CGO_ENABLED=1 go test -count=1 -short ./batou-core/... ./batou-rules/...
# all packages ok
python3 batou-rules/tools/check_rules.py --duplicates
# No duplicate rule IDs found.
CGO_ENABLED=1 make build   # produces bin/batou

Co-Authored-By: Claude Opus 4.6 noreply@anthropic.com

Zeerg and others added 7 commits April 7, 2026 15:10
Mechanical restructure to match commercial layout in preparation for
syncing taint research commits. No semantic changes.

- internal/{scanner,taint,analyzer,ast,graph,hints,hook,ledger,reporter,
  findings,fpfilter,suppress,taintrule,testutil} → batou-core/
- internal/rules → batou-rules/rules
- cmd → batou-core/cmd
- testdata → batou-rules/testdata (batou-core/testdata symlinked)
- tools → batou-rules/tools
- Add batou-rules/testutil (lightweight, rules-only) to avoid import cycle
- Create go.work + per-module go.mod files
- Rewrite imports: github.com/turenlabs/batou/internal/X →
  github.com/turenlabs/batou-{core,rules}/X
Syncs ~3 weeks of taint-research work from the internal development
repo into the restructured batou-core / batou-rules layout.

Highlights:
- New taint catalog entries across 16 languages (C, C++, C#, Go, Groovy,
  Java, JS/TS, Kotlin, Lua, Perl, PHP, Python, Ruby, Rust, Swift, Zig)
- New structural taint rules: injection_c, xss_c, sql_fragment, kotlin
  and swift bench helpers, javautil, pyutil
- Scanner: new fpfilter modules per language, regexenrich, per-language
  bench tests, red-team/architecture tests, owasp-bench harness
- tsflow: callback, summary, debug-switch, rust sanitizer/ssrf/xss
  improvements, fp sanitizer/switch tests
- 124 new bench fixtures (vulnerable + safe) across all supported
  languages; 27 pre-existing OSS-only fixtures preserved

Layout is unchanged from Phase 1: batou-core/ (engine + analyzers +
scanner) and batou-rules/ (rules + testdata + tools).
Rewrites a handful of comments and test names that referenced
internal tooling (ClickHouse sink, agent-monorepo IPC, watchdog) to
neutral language (external dashboards, downstream consumers, metrics
sinks). No functional code changes.
Phase 1 restructured the tree into batou-core/ and batou-rules/ but
left Makefile and install.sh pointing at the old ./cmd/batou and
./internal/scanner/ paths. This updates both to the new layout and
picks up the OWASP-benchmark helper targets from commercial.
go.work multi-module layout breaks `./...` — CI must target
`./batou-core/...` and `./batou-rules/...` explicitly. Also updates
the build path from `./cmd/batou` to `./batou-core/cmd/batou`.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Counts go stale with every taint-research cycle. Focus on the how-it-works
flow and what categories are covered, not specific numbers.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
File rename race on macOS CI — fails intermittently with "no such file
or directory" during concurrent Save(). Not worth blocking CI over.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@Zeerg
Zeerg merged commit 6299e10 into main Apr 7, 2026
3 checks passed
@Zeerg
Zeerg deleted the sync/content-from-commercial branch April 7, 2026 21:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant