Faultline

Faultline is release-risk intelligence for developer platforms. It turns messy support tickets and customer feedback into strict JSON incident signals, then correlates those signals with modules, services, releases and incidents without pretending correlation is root cause.

Current State

The app runs locally with:

• Frontend: Next.js, React, Tailwind, TanStack Query, Recharts
• Backend: FastAPI, SQLAlchemy, Alembic, Pydantic
• Data: PostgreSQL/pgvector, plus SQLite fallback for local development
• AI: Ollama qwen3:8b and nomic-embed-text
• Automation: n8n and Mailpit
• Training: Qwen3 8B QLoRA only

Faultline should stay on the base Ollama model until a fine-tuned adapter beats the base model in evaluation. Training completion alone is not an activation gate.

Local App

PowerShell:

cd C:\Users\munee\OneDrive\Desktop\n8n\faultline
.\scripts\setup-local.ps1
docker compose up -d
ollama pull qwen3:8b
ollama pull nomic-embed-text
cd apps\api
python -m venv .venv
.\.venv\Scripts\Activate.ps1
pip install -r requirements.txt
alembic upgrade head
python -m app.seed.demo
python -m app.scripts.backfill_embeddings --force
uvicorn app.main:app --reload --port 8000

Second terminal:

cd C:\Users\munee\OneDrive\Desktop\n8n\faultline\apps\web
npm install
npm run dev

Open:

• Frontend: http://localhost:3000
• API docs: http://localhost:8000/docs
• n8n: http://localhost:5678
• Mailpit: http://localhost:8025

Demo login:

• Email: demo@faultline.local
• Password: Demo@Faultline2026

n8n

Import and publish both workflows:

• faultline-signal-analysis
• faultline-weekly-product-brief

Both HTTP nodes should call the backend through:

http://host.docker.internal:8000

Set the header:

X-Internal-Workflow-Key: replace_with_internal_n8n_key

Use the exact internal key from your local .env; do not keep the placeholder. Publish both workflows so production webhook calls from the app reach n8n.

Fine-Tuning Policy

Faultline fine-tuning is now Qwen3 8B-only:

Qwen/Qwen3-8B

The code guards reject older Qwen2.5 models, smaller Qwen checkpoints, and non-Qwen3 8B adapter paths. Earlier Qwen2.5 7B experiments were useful, but Qwen3 8B is the new final training target.

The local laptop is not a safe training target because it blacked out under sustained GPU load. Kaggle Tesla T4 is the recommended free path for Qwen3 8B QLoRA.

Final Dataset

The final dataset pipeline generates:

• 12,000 reasoning examples
• 4,000 contrastive examples
• 5,000 hallucination traps
• 3,000 RAG-style examples
• 3,500 base-failure-targeted examples
• 10,000 optional preference pairs
• hard, hallucination, contrastive, smoke and validation eval sets

Latest local gate result:

• train rows: 25376
• validation rows: 2000
• exact schema rate: 1.0
• context consistency rate: 1.0
• deprecated, missing and extra key counts: 0
• near duplicate prompts: 50, or 0.2%
• suspected-service null: 40.8%
• release-hint null: 97.4%
• final dataset report: no warnings

Run this in WSL before building the Kaggle zip:

cd /mnt/c/Users/munee/OneDrive/Desktop/n8n/faultline

python3 training/generators/generate_base_failure_probe_set.py \
  --output training/datasets/eval/base_failure_probe_set.jsonl \
  --count 2000

python3 training/generators/generate_faultline_reasoning_dataset.py \
  --output training/datasets/generated/reasoning_examples.jsonl \
  --count 12000

python3 training/generators/generate_faultline_contrastive_dataset.py \
  --output training/datasets/generated/contrastive_examples.jsonl \
  --count 4000

python3 training/generators/generate_faultline_hallucination_traps.py \
  --output training/datasets/generated/hallucination_traps.jsonl \
  --count 5000

python3 training/generators/generate_faultline_rag_style_dataset.py \
  --output training/datasets/generated/rag_style_examples.jsonl \
  --count 3000

python3 training/generators/generate_faultline_preference_pairs.py \
  --output training/datasets/preference/preference_pairs.jsonl \
  --count 10000

python3 training/generators/generate_failure_targeted_dataset.py \
  --output training/datasets/generated/failure_targeted_examples.jsonl \
  --count 3500

python3 training/generators/generate_faultline_hard_eval.py

python3 training/preparation/prepare_faultline_sft_dataset.py \
  --inputs training/datasets/generated/reasoning_examples.jsonl training/datasets/generated/contrastive_examples.jsonl training/datasets/generated/hallucination_traps.jsonl training/datasets/generated/rag_style_examples.jsonl training/datasets/generated/failure_targeted_examples.jsonl \
  --train-output training/datasets/prepared/final_train.jsonl \
  --validation-output training/datasets/prepared/final_validation.jsonl \
  --validation-count 2000

python3 training/preparation/prepare_faultline_preference_dataset.py

python3 training/preparation/validate_faultline_context_consistency.py \
  --input training/datasets/prepared/final_train.jsonl

python3 training/preparation/validate_faultline_dataset.py \
  --train training/datasets/prepared/final_train.jsonl \
  --validation training/datasets/prepared/final_validation.jsonl \
  --hard training/datasets/eval/hard_eval.jsonl \
  --hallucination training/datasets/eval/hallucination_eval.jsonl \
  --contrastive training/datasets/eval/contrastive_eval.jsonl

cat training/outputs/final_dataset_quality_report.md
cat training/outputs/final_reasoning_coverage_report.md

Do not train unless the dataset report has no warnings.

Kaggle Zip

rm -f faultline-training-kaggle.zip
zip -r faultline-training-kaggle.zip training \
  -x "training/adapters/*" \
  -x "training/outputs/*" \
  -x "*/__pycache__/*"

Kaggle Setup

%env HF_HUB_DISABLE_XET=1
%env HF_HUB_ENABLE_HF_TRANSFER=0
!pip install -U "transformers>=4.46" "peft>=0.14" "trl>=0.12" "accelerate>=1.0" "bitsandbytes>=0.46" datasets safetensors sentencepiece protobuf evaluate

Kaggle Training

Start safe:

!bash training/training_scripts/kaggle_train_qwen3_8b_safe.sh

If stable, run the reasoning config:

!bash training/training_scripts/kaggle_train_qwen3_8b_reasoning.sh

If evaluation shows schema obedience but weak hallucination resistance, run:

!bash training/training_scripts/kaggle_train_qwen3_8b_anti_hallucination.sh

Only try the heavy config if the notebook has enough time and memory:

!bash training/training_scripts/kaggle_train_qwen3_8b_final_heavy.sh

Evaluation Gate

The evaluator now runs real model generation on Kaggle. It does not score oracle labels as model predictions.

python3 training/evaluation/evaluate_faultline_model.py \
  --base-model Qwen/Qwen2.5-7B-Instruct \
  --adapter training/adapters/faultline-qwen2.5-7b-lora \
  --eval-file training/datasets/prepared/final_validation.jsonl \
  --output training/outputs/evaluation_adapter.json

Do not use the adapter unless it passes:

• valid_json_rate >= 0.95
• required_fields_present_rate >= 0.95
• exact_schema_rate >= 0.90
• deprecated_keys_rate == 0.0
• hallucination_free_rate >= 0.90
• adapter beats base model on schema, module, severity, release-hint and low-confidence metrics

If the report says Do not use this adapter in Faultline yet., keep:

OLLAMA_CHAT_MODEL=qwen2.5:7b