STEM Video QA Studio

Collect STEM experiment/demonstration videos and generate tightly coupled multimodal QA pairs for training vision-language models.

Pipeline

Video Download → Auto Split → QA Generation → Hallucination Check → Difficulty Test

Download — fetches LQ video from YouTube/Bilibili via yt-dlp (VPN required for YouTube), derives HQ+audio locally with ffmpeg
Auto Split — Gemini identifies self-contained narrative segments within each video
QA Generation — Gemini generates 2 challenging multimodal QA pairs per segment
Hallucination Check — verifies every claim in the QA is visually grounded
Difficulty Test — Doubao answers each question; if it gets it right the question is too easy, clip kept if passed

A segment is qualified when at least one of its QA pairs passes difficulty test (Doubao answers incorrectly).

Architecture

Layer	Stack
Backend	FastAPI + SQLAlchemy + SQLite (WAL mode)
Frontend	React + Vite + Ant Design
AI Models	Gemini (generate/check via aihubmix), Doubao (difficulty test via ARK)

Quick Start

Prerequisites

Python 3.11+, Node.js 18+, ffmpeg, yt-dlp
VPN tunnel (tun0) for YouTube access
Gemini API key (aihubmix proxy)
Doubao API key (volcengine ARK)

Backend

cd backend
uv sync
cp .env.example .env  # edit with API keys
uv run uvicorn app.main:app --host 127.0.0.1 --port 8001 --reload

Frontend

cd frontend
npm install
npm run dev

Pipeline

cd backend

# Fresh run — pick N videos from input
uv run python pipeline_runner.py --input ../input/stem_videos.json --count 10000 --max-downloads 7

# Resume from checkpoint
uv run python pipeline_runner.py --resume

Pipeline options:

Flag	Default	Description
`--max-downloads`	7	Concurrent yt-dlp downloads (VPN sweet spot)
`--max-splits`	50	Concurrent Gemini auto-split calls
`--max-generates`	50	Concurrent Gemini QA generation calls
`--max-checks`	90	Concurrent Doubao difficulty checks
`--count`	10	Videos to pick from input (use large number for all)

Pipeline Modes

Mode	Command	When to use
Fresh run	`--input file.json --count N`	First time, or processing a new batch. Picks N random videos from input.
Resume	`--resume`	Pipeline was killed or crashed. Picks up from checkpoint — skips DONE/REJECTED, retries FAILED.
Specific URLs	`--urls url1 url2 ...`	Test run with hand-picked videos.

--resume is safe after any kill — checkpoint saves after EVERY stage. At worst you re-run one in-progress stage.

Troubleshooting

Symptom	Likely Cause	Fix
All downloads timeout	VPN down	`curl --interface tun0 https://www.youtube.com` — if fails, restart VPN
`403 Forbidden` clusters	aihubmix API key expired/quota	Check `/tmp/uv.log` for 403, switch keys in `.env`
`all QA tasks failed` burst	Gemini API transient error	Resume will retry — no action needed
`ghost tasks` messages	Stale checkpoint task IDs	Handled automatically (re-submit fresh tasks)
`LQ missing, re-downloading`	Cleanup deleted files pre-DONE-save	Handled — download endpoint re-downloads if needed
`Video not found` in split	LQ file missing after cleanup	Restart with latest code (re-download logic active)
Disk >80%	Cleanup not keeping up	`du -sh storage/*` to find culprit, manual cleanup if needed
Pipeline appears stuck	Process hung or semaphore deadlock	Check `tail -5 pipeline.log` for activity, `pkill -9` and resume

Quick Health Checks

# Is everything running?
fuser 8001/tcp && echo "backend OK" || echo "backend DOWN"
pgrep -f pipeline_runner && echo "pipeline OK" || echo "pipeline DOWN"

# Last pipeline activity (should be within last 30s)
ls -l --time='+%H:%M:%S' output/pipeline.log | awk '{print "last write: "$7}'

# Recent errors (last 5 min)
tail -500 output/pipeline.log | grep -c "WARNING\|ERROR"

# Disk headroom
df -h / | awk 'NR==2{print "disk: "$5" used, "$4" free"}'

# VPN health
sar -n DEV 1 1 | grep tun0 | awk '{print "VPN: rx="$4"KB/s tx="$5"KB/s"}'

Export

Qualified Segment JSON

cd backend

# Full export
uv run python tools/export_passed.py -o ../output/passed_qa_export.json

# Incremental — only segments NOT in a previous export
uv run python tools/export_passed.py --incremental ../output/previous.json -o ../output/passed_qa_export_20260529_120000.json

# All options
uv run python tools/export_passed.py --help

Flag	Description
`-o`, `--output`	Output JSON path (default: `output/passed_qa_export.json`)
`-i`, `--incremental`	Previous export JSON — export only new segments

Incremental mode matches segments by (source_url, start_time, end_time) — immutable video identity, safe across database resets.

Output format:

{
  "export_time": "2026-05-29T12:00:00",
  "summary": { "total_clips": 92, "total_videos": 88, "total_qas": 184, "total_passed": 93, "pass_rate": "50.5%" },
  "clips": [
    {
      "segment_id": 1234,
      "start_time": 30.0, "end_time": 120.0, "duration": 90.0, "order_index": 1,
      "video": { "id": 567, "title": "...", "source_url": "https://...", "duration": 180.0, "discipline": "物理" },
      "qa_pairs": [
        {
          "qa_id": 890, "question_number": 1,
          "question_en": "...", "answer_en": "...", "explanation_en": "...",
          "question_zh": "...", "answer_zh": "...", "explanation_zh": "...",
          "question_type": "multiple_choice", "dimension": "reasoning",
          "difficulty_status": "passed", "hallucination_status": "passed",
          "video_dependency_status": "dependent", "retry_count": 0, "clip_path": "/path/to/seg_1234.mp4"
        }
      ],
      "passed_count": 1,
      "clip_filename": "seg_1234.mp4"
    }
  ]
}

Clip ZIP

The companion ZIP uses the same timestamp as the JSON export so filenames stay paired:

passed_qa_export_20260529_120000.json   ← metadata
passed_clips_20260529_120000.zip        ← video files

ZIP contains only the seg_{id}.mp4 files referenced in the JSON's clip_filename fields. One clip per qualified segment, shared by both QA pairs.

Project Structure

stem-video-qa-studio/
├── backend/
│   ├── app/
│   │   ├── models.py           # SQLAlchemy models
│   │   ├── database.py         # DB init, migrations, cleanup
│   │   ├── routers/            # FastAPI endpoints
│   │   └── services/
│   │       ├── ai_client.py    # Gemini + Doubao API
│   │       ├── downloader.py   # yt-dlp wrapper + precheck
│   │       ├── clip.py         # ffmpeg video clipping
│   │       ├── storage.py      # File cleanup
│   │       └── qa_pipeline/    # QA generation pipeline
│   ├── tools/
│   │   └── export_passed.py    # Export qualified segments
│   └── pipeline_runner.py      # Async pipeline orchestrator
├── frontend/                   # React annotation UI
├── input/                      # Input URL lists (JSONL)
├── output/                     # Logs, checkpoints, exports (gitignored)
├── storage/                    # Videos, clips, keep_clips (gitignored)
├── AGENTS.md                   # Project spec for AI agents
├── RESCUE.md                   # Operational handbook (mistakes + fixes)
└── README.md

Key Behaviors

Resume-safe: checkpoint saved after every stage; killed pipeline resumes from last completed stage
Cleanup: periodic (every 10 min) — deletes HQ/LQ/audio/clips for DONE videos, moves passed clips to keep_clips/
Precheck: yt-dlp metadata check before download; rejects dead links/oversize/live, timeouts allow download to proceed
Clip naming: seg_{segment_id}.mp4 — one clip per qualified segment, shared by both QA pairs
Rejection output: concise one-line with reason, no traceback for expected failures

Storage

storage/
├── videos/     # LQ video files (downloaded, cleaned up after DONE)
├── audio/      # Extracted audio tracks
├── clips/      # Temporary HDQ clips per QA (cleaned up after checks)
└── keep_clips/ # Passed clips moved here after difficulty test

Disk Cleanup — Zombie Detection

Storage files may become orphaned ("zombies") when the periodic cleanup logic misses them (e.g., due to bugs, disk-full interruptions, or checkpoint/DB drift). Use the following script to cross-reference disk files against the database and identify what can be safely deleted.

Diagnosis Script

cd ~/yjh/entropy/video-qa/stem-video-qa-studio
python3 << 'PYEOF'
import os, sqlite3, hashlib, glob

db = sqlite3.connect('backend/stem_qa.db')
db.row_factory = sqlite3.Row

# Get URL hashes of videos that STILL need their storage files
# (ready status + has QAs that are NOT all terminal)
rows = db.execute("""
    SELECT v.source_url FROM videos v
    WHERE v.status = 'ready'
    AND EXISTS (
        SELECT 1 FROM qa_pairs q JOIN segments s ON q.segment_id = s.id
        WHERE s.video_id = v.id AND q.question_en != ''
          AND q.difficulty_status NOT IN ('passed', 'failed')
    )
""").fetchall()
active_hashes = {hashlib.sha256(r['source_url'].encode()).hexdigest()[:16] for r in rows}
print(f"Active videos (keep): {len(active_hashes)}")

base = 'storage'
total_freed = 0

for dirname in ['videos', 'clips', 'audio']:
    d = os.path.join(base, dirname)
    if not os.path.isdir(d):
        continue
    to_delete = []
    keep = 0
    for f in os.listdir(d):
        fp = os.path.join(d, f)
        if not os.path.isfile(fp):
            continue
        prefix = f.split('_')[0] if '_' in f else f[:16]
        if len(prefix) != 16:
            continue
        if prefix not in active_hashes:
            to_delete.append(fp)
        else:
            keep += 1
    print(f"\n{dirname}/: {keep} to keep, {len(to_delete)} zombies")
    if to_delete:
        # Estimate size
        zombie_size = sum(os.path.getsize(fp) for fp in to_delete)
        print(f"  Zombie size: {zombie_size/1e9:.2f}GB — add --delete to remove")

# Also check keep_clips vs passed QAs
kdir = 'storage/keep_clips'
if os.path.isdir(kdir):
    passed = db.execute("SELECT DISTINCT segment_id FROM qa_pairs WHERE difficulty_status='passed' AND question_en!=''").fetchall()
    passed_ids = {str(r[0]) for r in passed}
    zombies = []
    for f in os.listdir(kdir):
        if f.startswith('seg_') and f.endswith('.mp4'):
            if f[4:-4] not in passed_ids:
                zombies.append(f)
    print(f"\nkeep_clips/: {len(os.listdir(kdir))-len(zombies)} valid, {len(zombies)} zombies")

db.close()
PYEOF

Cleanup

Add --delete flag to actually remove zombie files. Always run diagnosis first without --delete to review. This is also suitable for scheduled cleanup via the Feishu bot — send the script as a message.

Large File Upload via Feishu Drive

Feishu IM has a 20MB file size limit. For larger files (exports, ZIPs), use the Drive multipart upload API. The lark-cli tool handles this automatically for files > 20MB.

Single File Upload (≤ 512MB)

cd /path/to/file/directory
lark-cli drive +upload --file filename.zip

The CLI automatically switches to multipart upload for files > 20MB, using 4MB blocks.

Large File Upload (> 512MB, survives session disconnect)

For files requiring long upload times, use setsid to detach from the bridge process tree (bridge kills child processes when the Claude session ends):

setsid bash -c '
  export PATH="/usr/bin:/home/stu199vc/.npm-global/bin:$PATH"
  export HOME="/home/stu199vc"
  cd /path/to/file/directory
  lark-cli drive +upload --file filename.zip
' > /tmp/upload.log 2>&1 &

Monitor progress:

tail -f /tmp/upload.log

The uploaded file appears in the Drive root folder. Navigate to https://qcn1ywk927rz.feishu.cn/drive/ to manage permissions, or use the file URL printed in the log.

Important: lark-cli uses OS keychain for authentication (~/.lark-cli/lark-channel/config.json). The setsid approach works because the auth is stored independently of the bridge session.

Upload Completion Notification

To auto-notify a group when the upload completes, append to the upload command:

setsid bash -c '
  ... upload command ...
  URL=$(grep -oP "https://[^\"]+" /tmp/upload.log | tail -1)
  if [ -n "$URL" ]; then
    lark-cli im +messages-send --chat-id oc_xxx --text "Upload complete: $URL"
  fi
' > /tmp/upload.log 2>&1 &

Operations Guide (for Claudinux / Feishu bot)

This section is written for remote operation via the Feishu bot Claudinux. Every command below can be sent as a Feishu message and the bot will execute it on the server.

Key Paths

Path	Purpose
`~/yjh/entropy/video-qa/stem-video-qa-studio/`	Project root
`backend/stem_qa.db`	SQLite database (all video/segment/QA data)
`backend/pipeline_runner.py`	Pipeline orchestrator
`backend/tools/export_passed.py`	Export qualified segments to JSON
`backend/app/routers/download.py`	Download API endpoint
`backend/app/routers/qa.py`	QA generation + difficulty test
`backend/app/services/downloader.py`	yt-dlp wrapper + precheck
`backend/app/database.py`	DB migrations, cleanup functions
`output/pipeline.log`	Current pipeline log
`output/.pipeline_checkpoint/`	Sharded checkpoint files (one per URL)
`output/passed_qa_export*.json`	Exported qualified segment metadata
`output/passed_clips_*.zip`	Exported clip video files
`storage/videos/`	Downloaded LQ video files (cleaned up after DONE)
`storage/clips/`	Temporary HDQ clips (cleaned up after checks)
`storage/keep_clips/`	Passed clips (`seg_{id}.mp4`) — do not delete
`input/stem_videos.json`	Original input URL list (6461 videos)
`input/stem_videos_remaining.json`	Filtered list (minus already-passed videos)
`/tmp/uv.log`	Backend server log
`/home/stu199vc/.lark-channel/logs/`	Feishu bridge logs

Kill & Restart

Always use this procedure — never skip steps:

# 1. Kill everything
fuser -k 8001/tcp          # kill backend
pkill -9 -f pipeline_runner # kill pipeline
sleep 2
fuser 8001/tcp              # must print nothing
pgrep -f pipeline_runner    # must print nothing

# 2. Start backend (MUST cd first!)
cd ~/yjh/entropy/video-qa/stem-video-qa-studio/backend
uv run uvicorn app.main:app --host 127.0.0.1 --port 8001 --workers 4 &>/tmp/uv.log &
sleep 6                     # wait for startup — do not reduce!
curl -s http://127.0.0.1:8001/api/health
# Must return: {"status":"ok","version":"0.1.0"}

# 3. Start pipeline
uv run python pipeline_runner.py --resume \
  &> ~/yjh/entropy/video-qa/stem-video-qa-studio/output/pipeline.log &
sleep 5
tail -3 ~/yjh/entropy/video-qa/stem-video-qa-studio/output/pipeline.log

Critical pitfalls:

fuser -k 8001/tcp is the ONLY reliable way to kill the backend. Do NOT use pkill -f uvicorn — the process name is python, not uvicorn.
Always cd into backend/ before starting uvicorn — the import app.main requires CWD to be backend/.
sleep 6 is mandatory — uvicorn with 4 workers takes time to boot. Curl too early and you'll think it failed.
The pipeline uses --resume with checkpoint. Killed at any point, it picks up from the last completed stage.

Export Workflow

Definition: A segment is "qualified" when at least one of its QA pairs has difficulty_status = 'passed' (Doubao/LLM answered incorrectly — the question is hard enough).

cd ~/yjh/entropy/video-qa/stem-video-qa-studio/backend

# Full export (all qualified segments)
uv run python tools/export_passed.py -o ../output/passed_qa_export_$(date +%Y%m%d_%H%M%S).json

# Incremental export (only new segments since last export)
uv run python tools/export_passed.py \
  -i ../output/passed_qa_export_PREVIOUS.json \
  -o ../output/passed_qa_export_$(date +%Y%m%d_%H%M%S).json

# Pack matching clip ZIP (use SAME timestamp as the export JSON!)
# The JSON filename is: passed_qa_export_20260529_120000.json
# So the ZIP gets:   passed_clips_20260529_120000.zip
python3 -c "
import json, os, zipfile
ts = '20260529_120000'  # ← copy from the JSON filename
exp = f'../output/passed_qa_export_{ts}.json'
keep = os.path.expanduser('~/yjh/entropy/video-qa/storage/keep_clips')
with open(exp) as f: data = json.load(f)
files = []
for c in data['clips']:
    cp = c.get('clip_path', '')
    if cp and os.path.exists(cp): files.append(cp)
    else:
        alt = os.path.join(keep, c['clip_filename'])
        if os.path.exists(alt): files.append(alt)
with zipfile.ZipFile(f'../output/passed_clips_{ts}.zip', 'w', zipfile.ZIP_DEFLATED) as zf:
    for f in files: zf.write(f, os.path.basename(f))
print(f'Zipped {len(files)} files, {os.path.getsize(f\"../output/passed_clips_{ts}.zip\")/2**20:.0f}MB')
"

Naming convention: JSON and ZIP always share the same timestamp so filenames stay paired. After export, you can safely delete the corresponding keep_clips/ files that have been exported (but keep unexported ones).

Status Report Checklist

When asked "报告当前情况" or "status report", gather ALL of the following:

1. System Resources

# CPU load
uptime && echo "---" && top -bn1 | head -5

# Disk usage
df -h / && echo "---" && du -sh storage/videos storage/clips storage/audio storage/keep_clips 2>/dev/null

# Network throughput (VPN interface)
sar -n DEV 1 1 | grep tun0
# Columns: rxkB/s = received KB/s, txkB/s = sent KB/s

2. Pipeline Progress (parse from pipeline.log)

Run the diagnostic script to get:

Time elapsed since last restart
Per-stage throughput: Download / Split / QA / Checks — count, avg time, median time, rate per minute, efficiency %
Completion stats: DONE count, DONE/min, rejected count, no_segments count, QA error count
Pipeline water level: how many videos in each stage (download→split→QA→checks→DONE)
ETA: remaining videos / current DONE rate

3. Database State

sqlite3 backend/stem_qa.db "
SELECT 'Videos: ' || COUNT(*) FROM videos
UNION ALL SELECT 'Segments: ' || COUNT(*) FROM segments
UNION ALL SELECT 'QAs: ' || COUNT(*) FROM qa_pairs WHERE question_en != ''
UNION ALL SELECT 'Passed QAs: ' || COUNT(*) FROM qa_pairs WHERE difficulty_status='passed' AND question_en != ''
UNION ALL SELECT 'Qualified Segs: ' || COUNT(DISTINCT s.id) FROM segments s JOIN qa_pairs q ON q.segment_id=s.id WHERE q.difficulty_status='passed';
"

4. Incremental Qualified Segments (since last export)

Compare current qualified segments against the most recent passed_qa_export_*.json:

python3 -c "
import json, sqlite3, glob, os
output = os.path.expanduser('~/yjh/entropy/video-qa/stem-video-qa-studio/output')
exports = sorted(glob.glob(f'{output}/passed_qa_export_*.json'))
if not exports: print('No previous export found')
else:
    latest = exports[-1]
    with open(latest) as f: old = json.load(f)
    old_keys = {(c['video']['source_url'], c['start_time'], c['end_time']) for c in old['clips']}
    db = sqlite3.connect(os.path.expanduser('~/yjh/entropy/video-qa/stem-video-qa-studio/backend/stem_qa.db'))
    db.row_factory = sqlite3.Row
    rows = db.execute(\"\"\"
        SELECT DISTINCT v.source_url, s.start_time, s.end_time, s.id
        FROM segments s JOIN videos v ON v.id=s.video_id
        JOIN qa_pairs q ON q.segment_id=s.id
        WHERE q.difficulty_status='passed' AND q.question_en!=''
    \"\"\").fetchall()
    db.close()
    new = [(r['source_url'], r['start_time'], r['end_time'], r['id']) for r in rows if (r['source_url'], r['start_time'], r['end_time']) not in old_keys]
    print(f'Total qualified: {len(rows)}  Previous export: {len(old_keys)}  NEW: {len(new)}')
    if new: print(f'New segment IDs: {sorted(r[3] for r in new)}')
"

5. Recent Log Anomalies

# Check last 200 lines for warnings/errors
tail -200 output/pipeline.log | grep -E "WARNING|ERROR|ghost|LQ missing|Video not found|403|timed out"
# Count QA failures and split errors in recent window
tail -500 output/pipeline.log | grep -c "all tasks failed"
tail -500 output/pipeline.log | grep -c "split.*task error"

Watch for:

ghost tasks — stale checkpoint task IDs, should be handled by re-submit logic
LQ missing — cleanup deleted files before DONE was saved, transparently handled
403 Forbidden — aihubmix API key issues, cluster indicates key pool problems
all tasks failed — Gemini API errors, transient unless clustered
split.*task error — Gemini auto-split failures, re-fire handles it
precheck timeout — VPN/network issue, now allows download to proceed

Feishu Bot Tips

The bot is Claudinux (App ID: cli_aa932c16d9b8dbb3)
Messages with / prefix are bot commands; everything else goes to Claude
Use /new to start a fresh conversation (context gets long)
The bot uses the deepseek-v4-pro model backend
The bridge is a systemd user daemon — survives SSH disconnect and auto-restarts on crash
To check bridge health: lark-channel-bridge status
Bridge logs: ~/.lark-channel/logs/
The bot's claude subprocess inherits systemd environment variables (not shell rc files)
AGENTS.md — full QA requirements and acceptance criteria

Name		Name	Last commit message	Last commit date
Latest commit History 182 Commits
.playwright-mcp		.playwright-mcp
backend		backend
docs/superpowers		docs/superpowers
frontend		frontend
review-app		review-app
.env.example		.env.example
.gitignore		.gitignore
AGENTS.md		AGENTS.md
README.md		README.md
RESCUE.md		RESCUE.md
cleanup_files_api.py		cleanup_files_api.py
export.py		export.py

Uh oh!

Folders and files

Latest commit

History

Repository files navigation

STEM Video QA Studio

Pipeline

Architecture

Quick Start

Prerequisites

Backend

Frontend

Pipeline

Pipeline Modes

Troubleshooting

Quick Health Checks

Export

Qualified Segment JSON

Clip ZIP

Project Structure

Key Behaviors

Storage

Disk Cleanup — Zombie Detection

Diagnosis Script

Cleanup

Large File Upload via Feishu Drive

Single File Upload (≤ 512MB)

Large File Upload (> 512MB, survives session disconnect)

Upload Completion Notification

Operations Guide (for Claudinux / Feishu bot)

Key Paths

Kill & Restart

Export Workflow

Status Report Checklist

1. System Resources

2. Pipeline Progress (parse from pipeline.log)

3. Database State

4. Incremental Qualified Segments (since last export)

5. Recent Log Anomalies

Feishu Bot Tips

About

Resources

Uh oh!

Stars

Watchers

Forks

Releases

Packages 0

Uh oh!

Contributors

Uh oh!

Languages

Packages