semantic-recursive-dirscan

This tool now uses ffuf as the scanning backend and keeps the semantic recursion controller on top.

• It does not rely on ffuf -recursion.
• It scans one level at a time.
• It scores the findings from that round.
• It only continues into paths that look structurally valuable.

This fits your workflow better than plain status-based recursion. If /api, /swagger, /graphql, /admin, /v1 or similar paths appear, the tool continues there first instead of blindly descending into everything that returns 200.

Why ffuf here

ffuf is easier to orchestrate when you want custom control logic:

• clean JSON output
• simple request model
• easy rate and thread control
• easier to pair with AI ranking or custom recursion policies

dirsearch is still good for quick out-of-the-box scans, but this project is now built around ffuf.

Files

• semantic_recursive_dirscan.py: main script
• output/summary.json: machine-readable scan summary
• output/README_REPORT.md: readable report
• output/raw/: raw ffuf outputs per round

Requirements

• Python 3.11+
• ffuf in PATH, or pass --ffuf-bin

In your environment, ffuf was installed at:

C:\Users\Administrator\go\bin\ffuf.exe

If that directory is not in PATH, use --ffuf-bin.

Fast start

For normal use, you only need:

• one Burp request txt
• optional custom wordlist txt

Minimal command:

python .\semantic_recursive_dirscan.py `
  -r .\burp-request.txt `
  -w C:\wordlists\content.txt

That is enough to start testing.

If -w is omitted, the tool automatically merges usable .txt dictionaries from dict/ next to the script and uses the merged file for the scan.

Default behavior:

• automatically derives base URL from the request
• inherits method, body, cookie, authorization and useful headers
• uses practical recursion defaults
• enables auto-calibration
• writes results to an output directory next to your request file
• reports likely stale authentication in the output
• starts a local dashboard at http://127.0.0.1:8765/index.html when the port is free

Basic usage

python .\semantic_recursive_dirscan.py `
  -u https://target.example/ `
  -w C:\wordlists\raft-small-directories.txt `
  -o .\output `
  --ffuf-bin C:\Users\Administrator\go\bin\ffuf.exe `
  --max-depth 3 `
  --top-k 6 `
  --min-score 3.0 `
  --min-dir-score 2.2 `
  -e php,txt,bak

Common options

• --include-status: which status codes are kept from ffuf
• --recurse-status: which statuses can trigger deeper scans
• --top-k: maximum children expanded per parent
• --max-depth: maximum recursion depth
• --simple: easiest day-to-day mode
• -r: short form of --raw-request-file
• --raw-request-file: inherit method, body, cookie and headers from a Burp/raw HTTP request
• --run-dirsearch: also launch a separate PowerShell window for dirsearch
• --no-dirsearch: suppress dirsearch even if enabled in config
• --headers: repeatable extra headers
• --cookie: cookie string
• --proxy: proxy such as Burp
• --method: request method
• --data: request body
• --auto-calibration: enable ffuf -ac
• --dirsearch-compat: enable ffuf -D
• --dry-run: print commands without running
• -w can be omitted to auto-use dict/ wordlists

Example for your workflow

If you want to start from the site root, but only continue when paths like /api, /admin, /swagger, /openapi, /graphql, /v1 look promising:

python .\semantic_recursive_dirscan.py `
  -u https://target.example/ `
  -w C:\wordlists\content.txt `
  -o .\output `
  --ffuf-bin C:\Users\Administrator\go\bin\ffuf.exe `
  --max-depth 4 `
  --top-k 5 `
  --min-score 3.4 `
  --min-dir-score 2.4 `
  --include-status 200-399,401,403 `
  --recurse-status 200,204,301,302,307,308,401,403 `
  --exclude-sizes 57,590 `
  --proxy http://127.0.0.1:8080 `
  --auto-calibration

That setup is useful when the target returns many small fake positives, but a few paths reveal structure worth expanding.

Burp raw request support

If you already have a valid request in Burp, you can export or save it and let the tool inherit the session state automatically.

The tool will inherit:

• HTTP method
• request body
• Cookie
• Authorization
• most custom headers such as X-Requested-With, Referer, Origin, Accept, Content-Type

The tool will not blindly forward transport-only headers such as:

• Host
• Content-Length
• Connection
• Transfer-Encoding

Example:

python .\semantic_recursive_dirscan.py `
  -r .\burp-request.txt `
  -w C:\wordlists\content.txt `
  -o .\output `
  --ffuf-bin C:\Users\Administrator\go\bin\ffuf.exe `
  --max-depth 3 `
  --top-k 5 `
  --proxy http://127.0.0.1:8080

If --raw-request-file is used without -u, the tool derives the base URL from the Host header and the request path.

If the report says authentication may be stale, just go back to Burp, grab a fresh authenticated request, overwrite the txt, and run again.

Dirsearch side scan

If you want the tool to also start a separate PowerShell window that runs dirsearch, enable it in config.json:

{
  "dirsearch_enabled": true,
  "dirsearch_new_window": true
}

Or turn it on just for one run:

python .\semantic_recursive_dirscan.py -r .\burp-request.txt -w C:\wordlists\content.txt --run-dirsearch

This gives you two things at once:

• a visible dirsearch window for broad conventional scanning
• the local semantic recursion logic running in the main process
• sidecar status recorded in the final report and summary.json

No heavy refactor was needed for this. dirsearch is treated as a sidecar scan, while the tool's own recursion engine stays unchanged.

The tool now monitors basic sidecar state:

• whether dirsearch was launched
• sidecar PID when available
• whether the sidecar has already exited
• current matched result line count from dirsearch.txt

Local dashboard

When the port is available, the tool starts a local web dashboard:

http://127.0.0.1:8765/index.html

The page reads runtime_state.json from the active output directory and shows:

• current phase
• current URL and queue size
• dirsearch sidecar status
• auth warnings
• top findings
• summary and report file paths

Local GUI launcher

For day-to-day use, start the built-in Tkinter GUI from the same Python file:

python .\semantic_recursive_dirscan.py --gui

The GUI keeps the command-line scanner unchanged but makes common switches explicit:

• 项目选择器：每个工程独立保存在 projects/<工程名>/
• 菜单操作：新建、加载、保存、另存为、打开工程目录
• ? 帮助按钮：查看参数说明
• 字典支持：可选字典目录或单个字典文件；留空时自动合并软件目录 dict/
• 目标输入：每行一个 http(s):// URL 或请求文件路径
• 主界面简洁：高级设置和日志都用弹窗打开
• 启动自动恢复：下次打开自动恢复上次工程
• 目标列表自动回填：工程打开时自动回填到输入框
• 主按钮状态：Start / Pause / Resume
• Stop：在 Windows 上用 taskkill /T /F 关闭 ffuf 子进程树
• 工程设置：保存在 project.json
• 目标输出：保存在 runs/<目标>_<id>/
• 默认并发：5
• 全局并发过高时，自动关闭每目标 Dashboard，避免端口冲突
• 进度说明：界面与报表都会显示运行状态、轮次、队列和已完成数量
• 导出结果：可一键导出中文 JSON、CSV、Markdown 报告、层级说明 MD 和 FreeMind .mm 脑图
• AI说明：默认未启用；可在全局设置弹窗里直接编辑 config.json 中的 AI 配置；启用后只辅助候选路径排序/打分，不直接发包或操作目标

本地日志保存在 projects/<工程名>/gui.log，每个目标目录也有自己的 gui.log。 Start / Continue 会跳过已有成功 summary.json 的目标；失败或待运行目标可继续执行。 界面通过同一条 CLI 扫描链路调用 ffuf，实时状态来自 runtime_state.json，而命令和输出会写入 GUI 日志区和本地日志。

Global proxy

Proxy is now better handled from config.json.

Example HTTP proxy:

{
  "use_proxy": true,
  "proxy_type": "http",
  "proxy_host": "127.0.0.1",
  "proxy_port": 8080
}

Example SOCKS5 proxy:

{
  "use_proxy": true,
  "proxy_type": "socks5",
  "proxy_host": "127.0.0.1",
  "proxy_port": 1080
}

If proxy is enabled there, both ffuf and the optional dirsearch side scan will use it automatically unless you override --proxy on the command line.

AI scoring

AI is optional. The tool works without it.

The recommended way now is to configure AI once in config.json, not on every command.

If you want AI enabled by default, edit config.json like this:

{
  "ffuf_bin": "C:\\Users\\Administrator\\go\\bin\\ffuf.exe",
  "ai_enabled": true,
  "ai_provider": "openai",
  "ai_model": "gpt-4.1-mini",
  "ai_base_url": "",
  "ai_api_key": "sk-...",
  "ai_temperature": 0.0,
  "ai_top_n": 12
}

If you want the model to help rank candidates:

python .\semantic_recursive_dirscan.py `
  -r .\burp-request.txt `
  -w C:\wordlists\api.txt

The model only ranks already discovered paths. It does not replace fuzzing.

Speed and dictionary notes

Is dirsearch faster?

Sometimes, but not in a way I would rely on as a rule.

• dirsearch can feel faster out of the box because its defaults, filtering behavior, and extension handling are convenient.
• ffuf is usually very competitive, and often better once you tune -t, -rate, -ac, filters, and the wordlist.
• In your use case, the backend speed is less important than the recursion strategy. Avoiding useless second-layer scans saves much more time than tool-level micro differences.

Does ffuf ship with wordlists?

Not really in the same way people expect. Treat ffuf as the engine, not the wordlist source.

• ffuf does not come with the rich built-in discovery dictionaries people usually rely on from content scanners or external collections.
• In practice, you usually pair ffuf with your own lists, SecLists, raft, project-specific API lists, or extracted custom candidates.

How different are the dictionaries from dirsearch?

The bigger difference is packaging and defaults, not capability.

• dirsearch is more opinionated as a ready-to-use directory scanner and commonly used with its bundled ecosystem and conventions.
• ffuf is deliberately minimal; you bring the wordlist strategy yourself.
• For API-heavy targets, a focused custom list usually beats either tool's default behavior.

If you care about results quality, the best setup is usually:

• generic discovery list for first-pass surface mapping
• API-specific list for /api, /rest, /graphql, version roots
• custom list generated from JS, docs, swagger, parameter names, and prior findings

About ffuf built-in wordlists

ffuf itself does not ship with a meaningful built-in content discovery dictionary in the installation you are using.

What I verified locally:

• binary: C:\Users\Administrator\go\bin\ffuf.exe
• source module: C:\Users\Administrator\go\pkg\mod\github.com\ffuf\ffuf\v2@v2.1.0

Inside that module there are code files, config examples and images, but no bundled discovery wordlist directory. So in practice:

• ffuf is the fuzzing engine
• your wordlists come from you, SecLists, raft, CAA output, JS extraction, swagger extraction, or custom business dictionaries

Notes

• Use this only on targets you are authorized to test.
• For noisy targets, tune --exclude-sizes, --exclude-regex, and --auto-calibration.
• If you want to fuzz authenticated endpoints, pass cookies and headers from a real session.
• For daily use, the simple behavior is already the default. Just pass -r and -w.