feat(scicode): add SciCode dataset + 0-shot generation task

[jack-scitix-ai]

Type

• feature — new benchmark, task, or capability

Summary

• Add the SciCode research-coding benchmark. SciCodeDataset: 80 problems / 338 dependent sub-steps; upstream jsonl revision-pinned at 69a8cfc, numeric targets raw_ground.h5 mirrored on HF as a plain HTTPS artifact.
• scicode_0shot_gen (SciCodeZeroShotGenTask): sequential per-sub-step generation in the generated-solutions (self-dependency) setting — step i conditions on the model's own code for steps 1..i-1 (upstream's default; gold-conditioning is not implemented, and the test split ships no per-step gold) — with an optional scientist-background axis, scored by the code-evaluator over HTTP.
• Reference: SciCode — https://github.com/scicode-bench/SciCode (pinned at 69a8cfc).
• Vendored upstream helpers under sieval/community/scicode/ (prompt templates, code parsing, HDF5 target readers, in-process scicode.compare.cmp shim), each carrying the upstream Apache-2.0 header — adapted from SciCode @69a8cfc.
• New scicode optional-dependency group (h5py / numpy / scipy): the sieval side reads numeric targets (incl. scipy.sparse matrices) from the h5.
• Validated on Kimi-K2.6 (official hyperparams T=1.0 / top-p=1.0, generated-solutions setting): with-background sub-problem 49.3 vs official SciCode 52.2 (Δ≈2.9, within ~1 SE of a single T=1 run).
• Runtime prerequisite: the code-evaluator vendor PR (adds source="scicode") must be deployed for eval runs; the sieval code + unit tests here are self-contained.

Test Plan

Automated

• Lint/format clean (ruff check && ruff format --check)
• Type check clean (ty check or mypy --strict)
• Unit tests pass (pdm run pytest)

Manual

• Full scripts/check_preflight.py PASS (task/dataset registration, links, dep-coverage, meta-index sync, version). Kimi-K2.6 score check (official hyperparams, generated-solutions, 65-problem test split):

  Model	Setting	Metric	Expected	Actual	Diff
  Kimi-K2.6	with background	sub-problem	52.2	49.3	-2.9(-2.9%)
  Kimi-K2.6	without background	sub-problem	—	37.8	—

Checklist

Required (all PRs)

• PR title follows conventional format (type(scope): description)
• No internal paths, credentials, or personal info in committed files
• AI-generated code has AI-Generated Code - <model> (<provider>) in module docstring
• No new upper-layer dependencies added to core/
• Deleted code verified — no remaining call sites depend on it

If: New or Modified Benchmark

• Reference paper/repo linked in Summary
• Score comparison table included (model, expected, actual, diff)
• Dataset loading tested (sieval dataset download <name> succeeds)
• Task registered in package-level __init__.py

If: community/ Changes

• Upstream diff documented (what differs and why)
• License attribution preserved

If: New Dependency

• Added to correct PDM dependency group
• Justified in Summary (why this package, no lighter alternative?)

[ethan-scitix]

Request changes. Logic is faithful to upstream + the internal GLM-OpenEvals impl (verified: both jsonl checksums, h5 byte-identity via sha256, prompt templates, extract_python_script, and previous_llm_code handling all match). Three things to fix:

🔴 Blocking

• Missing gold-step files. special_step_code() (sieval/community/scicode/prompts.py:102) reads data/{13.6,62.1,76.3}.txt, but .gitignore:156 (data/) excludes them — they're not committed and not on disk. Reproduced: all three raise FileNotFoundError, so problems 13/62/76 crash in infer() and pollute main-problem accuracy. reference_impl.notes claims these are vendored, but they aren't. Suggest inlining them as a .py module (like community/livecodebench/prompts/*.py) to avoid both the gitignore and wheel-packaging ([tool.pdm.build] declares no includes for data files).

🟠 Repro fidelity (blocks matching official numbers)

• Timeout. timeout=300.0 (sieval/tasks/scicode_0shot_gen.py:116) vs the official hardcoded 1800 in upstream test_generated_code.py. 6× shorter → spurious timeouts on slow numerical steps. Bump to 1800 or document why.
• Temperature. Official generation is greedy (temperature=0). Correctly not hardcoded here, but please note in the docstring/reference_impl.notes that reproducing official numbers requires temperature=0 in the model config. (with_background=False default is already correct — matches the official headline mode.)

Nits

• Remove the unused get_function_from_code / extract_function_name exports (sieval/community/scicode/__init__.py) — dead code; they read as "aligning with upstream" but nothing calls them (upstream only uses them on the resume-from-disk path, which this task doesn't take).
• Flesh out reference_impl.notes with the real deviations: h5 targets inlined + remote eval service (vs in-subprocess), special-step verbatim embed, and pipeline-fails-counted-as-unsolved accounting.
• Document the sandbox dependency sympy (injected by _cmp_upstream.py; not in any sieval dep group — must exist in the code-eval service image).
• _cmp_source() (sieval/community/scicode/harness.py:26) re-reads the file per tested step — wrap in functools.lru_cache.

Blocking = the gold files + timeout; the rest can be quick follow-ups.