test(integration): add real-repo SCIP integration tests for all supported languages#468
Open
jafreck wants to merge 18 commits into
Open
test(integration): add real-repo SCIP integration tests for all supported languages#468jafreck wants to merge 18 commits into
jafreck wants to merge 18 commits into
Conversation
…rted languages
- Add tests/integration/ harness that clones pinned repos, builds them,
indexes via SCIP, and queries tool handlers deterministically
- Cover 12 languages: TypeScript, Python, Java, Go, Rust, C, C++, C#,
Ruby, PHP, Kotlin, Scala (plus Dart noted as gap)
- Fix scip-ruby registry args (was --output, needs bare . path)
- Fix scip-php registry args (was index --output, runs bare)
- Fix scip-dotnet {project} placeholder for auto .sln/.csproj discovery
- Fix loadScipIndexes to recover index.scip from non-zero exit codes
- Fix detectProjectLanguages to find .sln/.csproj by extension
- Add CI workflow (.github/workflows/integration.yml)
- Update .gitignore and vitest.config.ts for integration repos
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #468 +/- ##
==========================================
+ Coverage 91.83% 92.31% +0.47%
==========================================
Files 65 65
Lines 6160 6193 +33
Branches 1715 1723 +8
==========================================
+ Hits 5657 5717 +60
+ Misses 503 476 -27 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
…ages as allow-failure
- Java/Scala: increase SCIP timeout to 600s (scip-java wraps build system) - C: add explicit cmake -DCMAKE_EXPORT_COMPILE_COMMANDS=ON build step - Kotlin: install JDK 17+21 (moshi Gradle toolchain needs JDK 17) - PHP: skip call refs assertion (scip-php only emits definitions) - Remove all allow-failure flags — every job must pass
…ailure Go, Rust, C++, C#, Ruby, PHP — required (8 languages, all pass) Java, C, Kotlin, Scala — allow-failure (need project-specific build toolchains)
…all, remove allow-failure Root causes: - scip-java: coursier bootstrap created broken launchers; use coursier launch wrapper instead - scip-clang: 149MB binary download was slow; cache ~/.lore before install steps - Auto-installer: fix installViaCoursier to use launch wrapper (cs install needs --contrib) - All 12 SCIP language jobs are now required (no allow-failure)
…ed indexes For build-system-dependent languages (Java, Kotlin, Scala, C), run the SCIP indexer directly in buildCommands and pass the output via scipIndexDir. This avoids Lore re-invoking the indexer (which re-triggers builds) and ensures the SCIP index is available regardless of Lore timeout settings. - Java/Kotlin/Scala: scip-java index --output .scip-indexes/index.scip - C: cmake + scip-clang --index-output-path=.scip-indexes/index.scip - buildRepo now adds ~/.lore/bin and ~/go/bin to PATH
Revert the pre-computed index approach (070d898, 7df8a91) which broke Rust and PHP. Return to letting Lore's pipeline run SCIP indexers internally. Key fixes: - Add scipTimeoutMs: 600_000 for Kotlin (scip-java wraps Gradle) - Use cs (not coursier) in scip-java wrapper for CI compatibility - Remove unused existsSync import from harness
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds deterministic integration tests that clone real open-source repos, build them, index via SCIP, and query MCP tool handlers — validating the full pipeline for every SCIP-supported language.
What's included
Integration test framework (
tests/integration/)prepareRepo(clone → build → index → open DB), tool handler wrappers (lookupSymbol,queryCallees,searchSymbols,findDependents,analyzeStructure,getSnippet, etc.),BuildCommandsupport,getIndexStats./mvnw compilecargo checkdotnet buildcomposer updateSCIP pipeline bug fixes
Found and fixed 5 bugs via the integration tests:
['--output', '{output}'], scip-ruby uses bare['.']and writesindex.scip['index', '--output', '{output}'], scip-php runs bare[]and writesindex.scip{project}discovery — wasindex .which requires a root.sln; added{project}placeholder with auto-discovery of.sln/.csprojfiles up to one directory deeploadScipIndexesnow checks forindex.scipoutput when the indexer exits non-zero (e.g., scip-ruby exits 100 with warnings but writes valid output)detectProjectLanguagesnow finds.sln/.csprojfiles by extension in the directory scan, not just by exact filename match at rootCI workflow (
.github/workflows/integration.yml)Runs integration tests on
pushtomainand weekly schedule, with all SCIP indexers installed.Config changes
.gitignore— added.integration-repos/vitest.config.ts— excluded.integration-repos/from test discovery andtests/integration/harness.tsfrom coverageTest results
All tests gated behind
INTEGRATION=1. Without it, all 168 tests skip cleanly.With
INTEGRATION=1: 163/168 pass. The 5 remaining failures are external tool limitations:records-testssubproject toolchain