Skip to content

test(integration): add real-repo SCIP integration tests for all supported languages#468

Open
jafreck wants to merge 18 commits into
mainfrom
test/integration-scip-languages
Open

test(integration): add real-repo SCIP integration tests for all supported languages#468
jafreck wants to merge 18 commits into
mainfrom
test/integration-scip-languages

Conversation

@jafreck

@jafreck jafreck commented Apr 2, 2026

Copy link
Copy Markdown
Owner

Summary

Adds deterministic integration tests that clone real open-source repos, build them, index via SCIP, and query MCP tool handlers — validating the full pipeline for every SCIP-supported language.

What's included

Integration test framework (tests/integration/)

  • harness.ts — shared helpers: prepareRepo (clone → build → index → open DB), tool handler wrappers (lookupSymbol, queryCallees, searchSymbols, findDependents, analyzeStructure, getSnippet, etc.), BuildCommand support, getIndexStats
  • zod.test.ts — TypeScript (18 tests)
  • fastapi.test.ts — Python (21 tests)
  • lore-self.test.ts — TypeScript/Lore (19 tests)
  • scip-languages.test.ts — parametric tests for 10 additional languages (110 tests):
Language Repo Indexer Build step
Java jackson-databind scip-java ./mvnw compile
Go gin scip-go
Rust once_cell rust-analyzer cargo check
C cJSON scip-clang cmake auto-compdb
C++ nlohmann-json scip-clang cmake auto-compdb
C# Humanizer (v2.14.1) scip-dotnet dotnet build
Ruby jekyll scip-ruby
PHP PHPMailer scip-php composer update
Kotlin moshi scip-java — (scip-java wraps Gradle)
Scala playframework scip-java — (scip-java wraps sbt)

SCIP pipeline bug fixes

Found and fixed 5 bugs via the integration tests:

  1. scip-ruby registry — was ['--output', '{output}'], scip-ruby uses bare ['.'] and writes index.scip
  2. scip-php registry — was ['index', '--output', '{output}'], scip-php runs bare [] and writes index.scip
  3. scip-dotnet {project} discovery — was index . which requires a root .sln; added {project} placeholder with auto-discovery of .sln/.csproj files up to one directory deep
  4. Non-zero exit code recoveryloadScipIndexes now checks for index.scip output when the indexer exits non-zero (e.g., scip-ruby exits 100 with warnings but writes valid output)
  5. C# language detectiondetectProjectLanguages now finds .sln/.csproj files by extension in the directory scan, not just by exact filename match at root

CI workflow (.github/workflows/integration.yml)

Runs integration tests on push to main and weekly schedule, with all SCIP indexers installed.

Config changes

  • .gitignore — added .integration-repos/
  • vitest.config.ts — excluded .integration-repos/ from test discovery and tests/integration/harness.ts from coverage

Test results

All tests gated behind INTEGRATION=1. Without it, all 168 tests skip cleanly.

With INTEGRATION=1: 163/168 pass. The 5 remaining failures are external tool limitations:

  • PHP scip-php doesn't emit call refs (only definitions)
  • Kotlin scip-java needs JDK 17 for moshi's records-tests subproject toolchain
  • Scala scip-java needs >120s for full sbt compile from clean checkout (timeout)

…rted languages

- Add tests/integration/ harness that clones pinned repos, builds them,
  indexes via SCIP, and queries tool handlers deterministically
- Cover 12 languages: TypeScript, Python, Java, Go, Rust, C, C++, C#,
  Ruby, PHP, Kotlin, Scala (plus Dart noted as gap)
- Fix scip-ruby registry args (was --output, needs bare . path)
- Fix scip-php registry args (was index --output, runs bare)
- Fix scip-dotnet {project} placeholder for auto .sln/.csproj discovery
- Fix loadScipIndexes to recover index.scip from non-zero exit codes
- Fix detectProjectLanguages to find .sln/.csproj by extension
- Add CI workflow (.github/workflows/integration.yml)
- Update .gitignore and vitest.config.ts for integration repos
@codecov

codecov Bot commented Apr 2, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 92.31%. Comparing base (ad2d825) to head (d2af68f).

Additional details and impacted files
@@            Coverage Diff             @@
##             main     #468      +/-   ##
==========================================
+ Coverage   91.83%   92.31%   +0.47%     
==========================================
  Files          65       65              
  Lines        6160     6193      +33     
  Branches     1715     1723       +8     
==========================================
+ Hits         5657     5717      +60     
+ Misses        503      476      -27     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

jafreck added 17 commits April 1, 2026 20:32
- Java/Scala: increase SCIP timeout to 600s (scip-java wraps build system)
- C: add explicit cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON build step
- Kotlin: install JDK 17+21 (moshi Gradle toolchain needs JDK 17)
- PHP: skip call refs assertion (scip-php only emits definitions)
- Remove all allow-failure flags — every job must pass
…ailure

Go, Rust, C++, C#, Ruby, PHP — required (8 languages, all pass)
Java, C, Kotlin, Scala — allow-failure (need project-specific build toolchains)
…all, remove allow-failure

Root causes:
- scip-java: coursier bootstrap created broken launchers; use coursier launch wrapper instead
- scip-clang: 149MB binary download was slow; cache ~/.lore before install steps
- Auto-installer: fix installViaCoursier to use launch wrapper (cs install needs --contrib)
- All 12 SCIP language jobs are now required (no allow-failure)
…ed indexes

For build-system-dependent languages (Java, Kotlin, Scala, C), run the
SCIP indexer directly in buildCommands and pass the output via scipIndexDir.
This avoids Lore re-invoking the indexer (which re-triggers builds) and
ensures the SCIP index is available regardless of Lore timeout settings.

- Java/Kotlin/Scala: scip-java index --output .scip-indexes/index.scip
- C: cmake + scip-clang --index-output-path=.scip-indexes/index.scip
- buildRepo now adds ~/.lore/bin and ~/go/bin to PATH
Revert the pre-computed index approach (070d898, 7df8a91) which broke
Rust and PHP. Return to letting Lore's pipeline run SCIP indexers
internally. Key fixes:
- Add scipTimeoutMs: 600_000 for Kotlin (scip-java wraps Gradle)
- Use cs (not coursier) in scip-java wrapper for CI compatibility
- Remove unused existsSync import from harness
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant