-brightgreen.svg){kind=icon}
HELK Rebuilt is a from-scratch modernization of HELK (The Hunting
ELK) — one of the first open source hunt platforms to combine SQL
declarative querying, graph analytics, structured streaming, and Jupyter
Notebook/Apache Spark-based machine learning on top of an ELK stack. The
original project (Cyb3rWard0g/HELK)
hasn't had a commit since May 2021, and nearly every component it depends on
— Elasticsearch/Kibana/Logstash 7.6.2, Kafka 2.4.1 on ZooKeeper, Spark
2.4.5, an unmaintained personal Jupyter base image, Yelp's abandoned
elastalert, and a Sigma-rule fork that no longer exists — is now years out
of date or gone outright.
This project keeps HELK's actual value — the streaming ingestion pipeline, ECS/OSSEM/ATT&CK-enriched detection content, and Spark/GraphFrames-based graph hunting in Jupyter — and rebuilds the infrastructure underneath it on currently maintained software.
🚧 This project is a work in progress, though all four planned phases are now done: the core stack, alerting, analytics, and lifecycle scripts all run on currently maintained software. One decision (carrying forward Kibana's legacy saved objects) is deliberately unscheduled future work, not part of the phased plan. Full technical detail — inventory, version decisions, open questions — lives in MODERNIZATION.md.
Progress by phase (percentage is phase count, not effort-weighted — Phases 1, 2, and 3 were the largest chunks of work):
| Phase | Scope | Status |
|---|---|---|
| 0 — Hygiene | Strip committed htpasswd credential, fix fatal stale ES discovery setting | ✅ Done |
| 1 — Core bootable stack | Elasticsearch 9.4.3, Kibana 9.4.3, Logstash 9.4.3, Kafka 4.3.1 (KRaft, no Zookeeper), Nginx (TLS), consolidated into one compose.yaml. Verified with a clean docker compose down -v && up --build: all services healthy, Kibana reachable through Nginx over HTTPS, all 5 Kafka topics created, Logstash's pipelines run error-free. Along the way, converted ~22 of HELK's legacy Elasticsearch index templates to composable templates after finding they collided with Elastic's built-in logs-*-* reserved template namespace on 9.x. |
✅ Done |
| 2 — Alerting | ElastAlert2 2.30.0 + a new sigma-cli/pySigma-backend-elasticsearch pipeline replacing the dead Sigma fork, gated behind the alert Compose profile. Converts 2,309 current SigmaHQ/sigma Windows rules to ElastAlert2 rules via a custom pySigma field-mapping pipeline ported from the legacy OSSEM config, plus the 23 curated helk_* rules. Verified with a clean docker compose --profile alert down -v && up --build: all 2,332 rules load, and a seeded test event produced a real fired alert end-to-end. |
✅ Done |
| 3 — Analytics | Spark 4.1.2 (Scala 2.13) standalone cluster + GraphFrames 0.12.1 + a rebuilt Jupyter image, gated behind the notebook Compose profile. Verified with a clean docker compose --profile notebook down -v && up --build: worker registers with master, PySpark imports on both driver and executors, GraphFrames' core graph algorithms run correctly on the real distributed cluster, the Postgres-backed Hive metastore provisions and persists, and Jupyter is reachable through Nginx with token auth. |
✅ Done |
| 4 — Lifecycle scripts | Rewrote helk_install.sh/helk_update.sh/helk_remove_containers.sh as thin docker compose wrappers (no more bare-metal OS/Docker provisioning — see the scope note below), fixed the destructive git clean -d -fx and the hardcoded ../.git/refs/heads/master read, dropped the obsolete helk_docker_install.sh/helk_setup_firewall.sh. |
✅ Done |
Confirmed design choice from Phase 1: Elasticsearch runs with authentication on but TLS disabled between containers on the internal Docker network (TLS is terminated only at the Nginx edge). This deviates from MODERNIZATION.md's original recommendation but was confirmed as intentional for this single-node, internal-only deployment — see MODERNIZATION.md §6.
Scope note from Phase 2: converting Sigma's OSSEM field mappings to pySigma
is a full 1:1 port for the ~150 simple field renames, but ~20 fields whose
legacy mapping varied by Windows EventID were deliberately collapsed to a
single dominant value rather than reproduced with full per-EventID fidelity —
see MODERNIZATION.md §6 for
which fields and why. Also, the upstream Sigma repo no longer has a top-level
apt rule category, so only rules/windows/ is converted.
Version correction from Phase 3: MODERNIZATION.md originally targeted Spark 3.5.8; by implementation time the Jupyter base image's newest published tag had moved on to Spark 4.1.2 with no 3.5.x tag left available, so that's what got built and verified instead (GraphFrames 0.12.1 itself didn't change, just which Spark/Scala-targeted artifact of it is current) — see MODERNIZATION.md §6.
Scope note from Phase 3: per an explicit user decision, the 378 Sigma-derived
notebooks were spot-checked rather than individually fixed. All 378 needed
(and got) an image-level dependency fix (the elasticsearch/elasticsearch-dsl
Python packages weren't installed anywhere), but a per-notebook fix — every
one of them calls the Elasticsearch client with no auth credentials, so all
378 currently fail with a 401 — was deliberately left for a future full pass.
See MODERNIZATION.md §6 for
the full spot-check findings and exactly what that future pass would need to
do.
Scope note from Phase 4: per an explicit user decision, the lifecycle
scripts dropped bare-metal Linux provisioning entirely (root checks,
apt/yum, systemd, sysctl tuning, firewalld, htpasswd, installing Docker
itself) rather than porting that provisioning to Compose v2. The three
scripts now assume Docker + the Compose v2 plugin are already present and
are thin wrappers over the single compose.yaml — this was also the only
branch that could actually be exercised here, since this environment can't
run apt/systemd/firewalld. See
MODERNIZATION.md §6 for the
full reasoning and the two named bugs fixed along the way (the destructive
git clean -d -fx and the hardcoded master-branch git-ref read).
One scope decision in MODERNIZATION.md §5 remains open, and it does not gate any phase in the plan above: whether to carry forward Kibana's ~85 legacy saved objects, or treat them as stale 2021-era demo content not worth migrating. (The decisions that gated Phases 3 and 4 — Jupyter path auth, notebook verification depth, and lifecycle script scope — were all resolved during those phases; see above.)
Carried over from the original HELK:
- Provide an open source hunting platform to the community and share the basics of Threat Hunting.
- Expedite the time it takes to deploy a hunt platform.
- Improve the testing and development of hunting use cases in an easier and more affordable way.
- Enable Data Science capabilities while analyzing data via Apache Spark, GraphFrames & Jupyter Notebooks.
Specific to this rebuild:
- Replace every abandoned or unmaintained dependency (dead base images, floating/stale Docker tags, a deleted Sigma rule fork, EOL Python) with currently maintained equivalents, rebuilt from the project's own Dockerfiles instead of pulled from frozen third-party images.
- Preserve HELK's detection/analytics capabilities rather than just bumping version numbers — including deliberately replacing components (e.g. the Sysmon/network correlation currently done in ksqlDB) rather than letting functionality quietly disappear along with the software that implemented it. See MODERNIZATION.md §4 for the specific items being tracked.
- Move onto Kafka's KRaft mode (no ZooKeeper), native Elasticsearch/Kibana
security (free since the Basic license, on by default in 8+), and
actively maintained community successors for anything upstream abandoned
(ElastAlert2,
sigma-cli, mainlineSigmaHQ/sigma).
HELK Rebuilt is an independent, unofficial fork — it is not an official continuation of, or endorsed by, the original project. All credit for the original design, architecture, and detection content goes to:
- Roberto Rodriguez — @Cyb3rWard0g / @THE_HELK — original author and creator of HELK.
- Nate Guagenti — @neu5ron — original committer.
Please refer to the original HELK repository and its documentation site for the project's history and original design rationale.
- MODERNIZATION.md — this fork's inventory, target architecture, open decisions, and phased plan.
- Original HELK: Introduction · Installation
- Welcome to HELK! : Enabling Advanced Analytics Capabilities
- Setting up a Pentesting.. I mean, a Threat Hunting Lab - Part 5
- Apache Spark · Spark Standalone Mode
- GraphFrames · An Integrated API for Mixing Graph and Relational Queries · Graph queries in Spark SQL
- Elastic Products · Elasticsearch Guide
- SigmaHQ/sigma · sigma-cli
- ElastAlert2
HELK Rebuilt remains licensed under the GNU General Public License v3.0, the same license as the original HELK project. See LICENSE.