Develop#14
Merged
Merged
Conversation
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com>
- Gate job verifies all CI checks passed on the tagged commit before any build or publish job runs; skipped for workflow_dispatch - Concurrency group prevents parallel release runs for the same ref - publish-homebrew now uses secrets.REPO_TOKEN for cross-repo push to homebrew-tap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Signed-off-by: Matthew Gribben <matthew@hypabolic.com>
This pull request adds significant architectural changes as well as a GitHub Actions workflow for syncing source and test files to a public repository. **Architectural Decisions:** * Establishes a package boundary between provider-neutral structural search contracts (`Hypa.Sdk`) and the Tree-sitter-based engine implementation (`Hypa.CodePatterns`). This separation enables downstream consumers to use structural search results without taking a dependency on native parsing libraries, and allows for future NuGet publication of the engine. * Details the sequencing and rationale for three major 2026 features: MCP Proxy Layer, code grep (structural search/scan/rewrite), and Code Mode Scripting Engine. We plan phased delivery, risk mitigation, and integration points to maximize compound value and minimize technical risk. **Timeouts** * Some commands would hit the hypa internal timeout limit, mainly package manager, so we've introduced pipeline specific limits and an override command to allow the agent to specify a timeout plus more intelligent feedback when there is an error. **Automation and Workflow:** * Introduced a GitHub Actions workflow (`.github/workflows/sync-public.yml`) that automatically syncs the `src` and `tests` directories to the public `Hypabolic/Hypa` repository on pushes to `main`. The workflow includes PR metadata extraction, payload creation, public repo checkout, file replacement, and commit/push logic with descriptive commit messages. Source repo: matt-gribben/Hypa Source SHA: 25f836cb7f71a4f63dc3423d5a971bf57c713083 PR URL: https://github.com/matt-gribben/Hypa/pull/29
workflow action update
This pull request introduces several documentation updates and new documents to clarify workflows, address architectural issues, and plan for future features. The most significant changes include the addition of an ADR for GitHub Copilot Extension integration, detailed documentation for local preview installs, a bug report on `hypa_shell` error handling, and an architecture review highlighting key technical debt and recommendations. There are also minor roadmap clarifications and new usage/setup instructions for Hypa CLI tools. **Key changes:** **1. GitHub Copilot Extension Integration** - Added ADR 0011 describing the design and implementation plan for a thin Node.js ESM adapter to integrate Hypa tools and context with GitHub Copilot CLI sessions, including command rewriting and session lifecycle hooks. The ADR details the rationale, consequences, implementation strategy, and alternatives considered. **2. Local Preview Installs Documentation** - Added a new document and updated the `README.md` to describe the workflow for installing local preview builds of Hypa. This includes commands for installing, restoring, managing, and cleaning preview builds, as well as the effect on update behavior. [[1]](diffhunk://#diff-51a886bc8914e40ea5afcc9459caae38f5fa33b3e6bd25b5fc2e5cbe05f9af5aR1-R38) [[2]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R115-R153) **3. Bug Report: `hypa_shell` Error Handling** - Added a detailed bug report documenting how `hypa_shell` misreports missing executables as working directory errors, including root cause analysis, impact, suggested fixes, and regression test cases. **4. Architecture Review and Technical Debt** - Added an architecture review document identifying several high and medium priority issues, such as dependency rule violations, improper use of application ports, unimplemented ADRs, CLI orchestration bloat, adapter boundary drift, and improper unit test practices. Each issue includes recommendations for resolution. **5. Roadmap and Usage Documentation Updates** - Updated the strategic roadmap to remove week estimates and clarify the sequencing and rationale for technical spikes and MCP proxy layer work. [[1]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L47-R55) [[2]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L77-R77) [[3]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L108-R112) [[4]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690R232-R237) - Added a "Hypa Rules" section to `CLAUDE.md` with best practices for using Hypa CLI wrappers and initial setup instructions.
## Summary
Three features developed across this release cycle, plus post-integration fixes and hardening discovered during live testing.
### MCP Proxy Layer
A full MCP proxy implementation that lets Hypa act as a man-in-the-middle between agents and external MCP servers — forwarding tool calls, compressing responses, and exposing a unified tool search index.
- `McpProxyService` with `DirectMcpDispatcher` for request routing and response compression
- `McpTransportBuilder` for transport creation; `McpClientConnectionFactory` refactored to use it
- MCP import with bearer token authentication and error handling
- `McpServerProbeAdapter` — probes remote servers on `mcp add` to validate connectivity and detect auth requirements; `--no-probe` flag to skip
- Tool search index for fast tool lookup across registered servers
- Credential resolution with secret redaction in test output
### Browser OAuth Onboarding
Automated OAuth flow for MCP servers that require browser-based authentication.
- `OAuth2BrowserConfig` and interactive/non-interactive auth modes in `McpTransportBuilder`
- Browser launch + local callback listener for authorization code exchange
- DCR secret storage, TLS options, state validation, and manual paste fallback for restricted environments
- Auth defaults to `none` on `mcp add`; guided setup triggered on probe auth failure
### Markdown Mode
Structured Markdown indexing and querying via tree-sitter, with a freshness-aware query gate and read hook compression.
- `MarkdownStructureProvider` and `CodePatternExtractor` — extracts sections (heading text, level, anchor, byte spans, plain text) and frontmatter YAML via tree-sitter
- `hypa md <file>` subcommand with `--toc`, `--section`, `--frontmatter`, and `--json` flags
- Git-aware incremental indexer (`IndexIncrementalAsync`): clean tracked files compared by blob OID (no file I/O), dirty/untracked files fall back to mtime+size — only stale files are re-parsed
- `EnsureFreshAsync` gate on `hypa md` — auto-indexes on first use, re-indexes on change, no-op when fresh
- `ReadRedirector` extended to compress large `.md` files to a heading outline; `CLAUDE.md` / `SKILL.md` pass through unchanged
- `hypa code index --full` flag for forced full rebuild
### Post-integration fixes (found during live testing)
- **`IndexFullAsync` OID regression**: full rebuilds stored `git_blob_oid = NULL`, causing the next incremental run to re-index every file. Fixed by calling `GetCleanBlobOidsAsync` in `IndexFullAsync` and persisting the OID alongside mtime.
- **`libtree-sitter-markdown.so` not bundled**: `TreeSitter.DotNet` 1.3.0 doesn't ship the markdown grammar. Fixed by building from source and bundling via `Directory.Build.targets` (RID-aware, copies to output/publish root on all platforms).
- **Provider conflict**: once the native library loaded, both `TreeSitterCodeStructureProvider` and `MarkdownStructureProvider` claimed `CanHandle("markdown")`, causing the generic provider to win and extract C# symbols from markdown files. Fixed with an explicit exclusion in `TreeSitterCodeStructureProvider.CanHandle`.
- **CI hardening**: grammar build script runs on all matrix runners before the .NET build; AOT publish job verifies the symbol export and asserts `markdown: ok` in provider health after indexing.
- **Missing test coverage**: added `MarkdownStructureProviderIntegrationTests` (exercises real tree-sitter native library, asserts sections and provenance) and `CodeStructureProviderRegistryTests` (asserts correct provider selection per language; regression guard ensures no two non-fallback providers claim the same language).
## Test plan
- [ ] `dotnet test tests/Hypa.UnitTests` — 1325 tests, all passing
- [ ] `dotnet test tests/Hypa.GoldenTests` — all passing
- [ ] `hypa mcp add <url>` probes the server and reports auth requirements
- [ ] OAuth browser flow launches, completes exchange, and stores credentials
- [ ] `hypa md README.md --toc` on a fresh clone auto-indexes and returns ToC
- [ ] Second `hypa md README.md --toc` with no file changes is a no-op (no re-parse)
- [ ] `hypa code index` is incremental by default; `--full` re-indexes then subsequent incremental is a no-op
- [ ] `hypa code index --json` shows `markdown: ok` in provider health
- [ ] Large `.md` files intercepted by the read hook produce a heading outline; `CLAUDE.md` passes through unchanged
… and release pipeline - Add native/runtimes/linux-x64/native/libtree-sitter-markdown.so (pre-built) - Add scripts/build-tree-sitter-markdown.sh/.ps1 to build from source on any platform - Add Directory.Build.targets to copy the RID-appropriate grammar to every project output and publish root, matching how TreeSitter.DotNet flattens its native assets - ci.yml: build grammar before dotnet build on all matrix runners; aot-publish job rebuilds with FORCE_BUILD=1 and verifies symbol before and after publish - release.yml: build grammar (FORCE_BUILD=1) in every platform build job before dotnet publish; verify grammar file is present in publish output before packaging
Update src and tests.
* Develop (#4) * added pip and homebrew release workflows * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * added gating to the release workflow * Add CI gate, concurrency control, and REPO_TOKEN for homebrew tap - Gate job verifies all CI checks passed on the tagged commit before any build or publish job runs; skipped for workflow_dispatch - Concurrency group prevents parallel release runs for the same ref - publish-homebrew now uses secrets.REPO_TOKEN for cross-repo push to homebrew-tap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> --------- Signed-off-by: Matthew Gribben <matthew@hypabolic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> * Add release workflows and CI gating plus timeout changes for long running commands (#5) * added pip and homebrew release workflows * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * added gating to the release workflow * Add CI gate, concurrency control, and REPO_TOKEN for homebrew tap - Gate job verifies all CI checks passed on the tagged commit before any build or publish job runs; skipped for workflow_dispatch - Concurrency group prevents parallel release runs for the same ref - publish-homebrew now uses secrets.REPO_TOKEN for cross-repo push to homebrew-tap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Sync public source from PR #29 This pull request adds significant architectural changes as well as a GitHub Actions workflow for syncing source and test files to a public repository. **Architectural Decisions:** * Establishes a package boundary between provider-neutral structural search contracts (`Hypa.Sdk`) and the Tree-sitter-based engine implementation (`Hypa.CodePatterns`). This separation enables downstream consumers to use structural search results without taking a dependency on native parsing libraries, and allows for future NuGet publication of the engine. * Details the sequencing and rationale for three major 2026 features: MCP Proxy Layer, code grep (structural search/scan/rewrite), and Code Mode Scripting Engine. We plan phased delivery, risk mitigation, and integration points to maximize compound value and minimize technical risk. **Timeouts** * Some commands would hit the hypa internal timeout limit, mainly package manager, so we've introduced pipeline specific limits and an override command to allow the agent to specify a timeout plus more intelligent feedback when there is an error. **Automation and Workflow:** * Introduced a GitHub Actions workflow (`.github/workflows/sync-public.yml`) that automatically syncs the `src` and `tests` directories to the public `Hypabolic/Hypa` repository on pushes to `main`. The workflow includes PR metadata extraction, payload creation, public repo checkout, file replacement, and commit/push logic with descriptive commit messages. Source repo: matt-gribben/Hypa Source SHA: 25f836cb7f71a4f63dc3423d5a971bf57c713083 PR URL: Hypabolic/Hypa-Private#29 --------- Signed-off-by: Matthew Gribben <matthew@hypabolic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Develop (#6) * added pip and homebrew release workflows * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * added gating to the release workflow * Add CI gate, concurrency control, and REPO_TOKEN for homebrew tap - Gate job verifies all CI checks passed on the tagged commit before any build or publish job runs; skipped for workflow_dispatch - Concurrency group prevents parallel release runs for the same ref - publish-homebrew now uses secrets.REPO_TOKEN for cross-repo push to homebrew-tap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Sync public source from PR #29 This pull request adds significant architectural changes as well as a GitHub Actions workflow for syncing source and test files to a public repository. **Architectural Decisions:** * Establishes a package boundary between provider-neutral structural search contracts (`Hypa.Sdk`) and the Tree-sitter-based engine implementation (`Hypa.CodePatterns`). This separation enables downstream consumers to use structural search results without taking a dependency on native parsing libraries, and allows for future NuGet publication of the engine. * Details the sequencing and rationale for three major 2026 features: MCP Proxy Layer, code grep (structural search/scan/rewrite), and Code Mode Scripting Engine. We plan phased delivery, risk mitigation, and integration points to maximize compound value and minimize technical risk. **Timeouts** * Some commands would hit the hypa internal timeout limit, mainly package manager, so we've introduced pipeline specific limits and an override command to allow the agent to specify a timeout plus more intelligent feedback when there is an error. **Automation and Workflow:** * Introduced a GitHub Actions workflow (`.github/workflows/sync-public.yml`) that automatically syncs the `src` and `tests` directories to the public `Hypabolic/Hypa` repository on pushes to `main`. The workflow includes PR metadata extraction, payload creation, public repo checkout, file replacement, and commit/push logic with descriptive commit messages. Source repo: matt-gribben/Hypa Source SHA: 25f836cb7f71a4f63dc3423d5a971bf57c713083 PR URL: Hypabolic/Hypa-Private#29 * feat: simplify commit message generation for public repo sync workflow action update --------- Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * Release: MCP Proxy, Browser OAuth, Markdown Mode, and Native Grammar Bundling (#7) * added pip and homebrew release workflows * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * added gating to the release workflow * Add CI gate, concurrency control, and REPO_TOKEN for homebrew tap - Gate job verifies all CI checks passed on the tagged commit before any build or publish job runs; skipped for workflow_dispatch - Concurrency group prevents parallel release runs for the same ref - publish-homebrew now uses secrets.REPO_TOKEN for cross-repo push to homebrew-tap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Sync public source from PR #29 This pull request adds significant architectural changes as well as a GitHub Actions workflow for syncing source and test files to a public repository. **Architectural Decisions:** * Establishes a package boundary between provider-neutral structural search contracts (`Hypa.Sdk`) and the Tree-sitter-based engine implementation (`Hypa.CodePatterns`). This separation enables downstream consumers to use structural search results without taking a dependency on native parsing libraries, and allows for future NuGet publication of the engine. * Details the sequencing and rationale for three major 2026 features: MCP Proxy Layer, code grep (structural search/scan/rewrite), and Code Mode Scripting Engine. We plan phased delivery, risk mitigation, and integration points to maximize compound value and minimize technical risk. **Timeouts** * Some commands would hit the hypa internal timeout limit, mainly package manager, so we've introduced pipeline specific limits and an override command to allow the agent to specify a timeout plus more intelligent feedback when there is an error. **Automation and Workflow:** * Introduced a GitHub Actions workflow (`.github/workflows/sync-public.yml`) that automatically syncs the `src` and `tests` directories to the public `Hypabolic/Hypa` repository on pushes to `main`. The workflow includes PR metadata extraction, payload creation, public repo checkout, file replacement, and commit/push logic with descriptive commit messages. Source repo: matt-gribben/Hypa Source SHA: 25f836cb7f71a4f63dc3423d5a971bf57c713083 PR URL: Hypabolic/Hypa-Private#29 * feat: simplify commit message generation for public repo sync workflow action update * Enhance MCP server configuration, validation, and connection handling This pull request introduces several documentation updates and new documents to clarify workflows, address architectural issues, and plan for future features. The most significant changes include the addition of an ADR for GitHub Copilot Extension integration, detailed documentation for local preview installs, a bug report on `hypa_shell` error handling, and an architecture review highlighting key technical debt and recommendations. There are also minor roadmap clarifications and new usage/setup instructions for Hypa CLI tools. **Key changes:** **1. GitHub Copilot Extension Integration** - Added ADR 0011 describing the design and implementation plan for a thin Node.js ESM adapter to integrate Hypa tools and context with GitHub Copilot CLI sessions, including command rewriting and session lifecycle hooks. The ADR details the rationale, consequences, implementation strategy, and alternatives considered. **2. Local Preview Installs Documentation** - Added a new document and updated the `README.md` to describe the workflow for installing local preview builds of Hypa. This includes commands for installing, restoring, managing, and cleaning preview builds, as well as the effect on update behavior. [[1]](diffhunk://#diff-51a886bc8914e40ea5afcc9459caae38f5fa33b3e6bd25b5fc2e5cbe05f9af5aR1-R38) [[2]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R115-R153) **3. Bug Report: `hypa_shell` Error Handling** - Added a detailed bug report documenting how `hypa_shell` misreports missing executables as working directory errors, including root cause analysis, impact, suggested fixes, and regression test cases. **4. Architecture Review and Technical Debt** - Added an architecture review document identifying several high and medium priority issues, such as dependency rule violations, improper use of application ports, unimplemented ADRs, CLI orchestration bloat, adapter boundary drift, and improper unit test practices. Each issue includes recommendations for resolution. **5. Roadmap and Usage Documentation Updates** - Updated the strategic roadmap to remove week estimates and clarify the sequencing and rationale for technical spikes and MCP proxy layer work. [[1]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L47-R55) [[2]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L77-R77) [[3]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L108-R112) [[4]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690R232-R237) - Added a "Hypa Rules" section to `CLAUDE.md` with best practices for using Hypa CLI wrappers and initial setup instructions. * Release: MCP Proxy, Browser OAuth, and Markdown Mode ## Summary Three features developed across this release cycle, plus post-integration fixes and hardening discovered during live testing. ### MCP Proxy Layer A full MCP proxy implementation that lets Hypa act as a man-in-the-middle between agents and external MCP servers — forwarding tool calls, compressing responses, and exposing a unified tool search index. - `McpProxyService` with `DirectMcpDispatcher` for request routing and response compression - `McpTransportBuilder` for transport creation; `McpClientConnectionFactory` refactored to use it - MCP import with bearer token authentication and error handling - `McpServerProbeAdapter` — probes remote servers on `mcp add` to validate connectivity and detect auth requirements; `--no-probe` flag to skip - Tool search index for fast tool lookup across registered servers - Credential resolution with secret redaction in test output ### Browser OAuth Onboarding Automated OAuth flow for MCP servers that require browser-based authentication. - `OAuth2BrowserConfig` and interactive/non-interactive auth modes in `McpTransportBuilder` - Browser launch + local callback listener for authorization code exchange - DCR secret storage, TLS options, state validation, and manual paste fallback for restricted environments - Auth defaults to `none` on `mcp add`; guided setup triggered on probe auth failure ### Markdown Mode Structured Markdown indexing and querying via tree-sitter, with a freshness-aware query gate and read hook compression. - `MarkdownStructureProvider` and `CodePatternExtractor` — extracts sections (heading text, level, anchor, byte spans, plain text) and frontmatter YAML via tree-sitter - `hypa md <file>` subcommand with `--toc`, `--section`, `--frontmatter`, and `--json` flags - Git-aware incremental indexer (`IndexIncrementalAsync`): clean tracked files compared by blob OID (no file I/O), dirty/untracked files fall back to mtime+size — only stale files are re-parsed - `EnsureFreshAsync` gate on `hypa md` — auto-indexes on first use, re-indexes on change, no-op when fresh - `ReadRedirector` extended to compress large `.md` files to a heading outline; `CLAUDE.md` / `SKILL.md` pass through unchanged - `hypa code index --full` flag for forced full rebuild ### Post-integration fixes (found during live testing) - **`IndexFullAsync` OID regression**: full rebuilds stored `git_blob_oid = NULL`, causing the next incremental run to re-index every file. Fixed by calling `GetCleanBlobOidsAsync` in `IndexFullAsync` and persisting the OID alongside mtime. - **`libtree-sitter-markdown.so` not bundled**: `TreeSitter.DotNet` 1.3.0 doesn't ship the markdown grammar. Fixed by building from source and bundling via `Directory.Build.targets` (RID-aware, copies to output/publish root on all platforms). - **Provider conflict**: once the native library loaded, both `TreeSitterCodeStructureProvider` and `MarkdownStructureProvider` claimed `CanHandle("markdown")`, causing the generic provider to win and extract C# symbols from markdown files. Fixed with an explicit exclusion in `TreeSitterCodeStructureProvider.CanHandle`. - **CI hardening**: grammar build script runs on all matrix runners before the .NET build; AOT publish job verifies the symbol export and asserts `markdown: ok` in provider health after indexing. - **Missing test coverage**: added `MarkdownStructureProviderIntegrationTests` (exercises real tree-sitter native library, asserts sections and provenance) and `CodeStructureProviderRegistryTests` (asserts correct provider selection per language; regression guard ensures no two non-fallback providers claim the same language). ## Test plan - [ ] `dotnet test tests/Hypa.UnitTests` — 1325 tests, all passing - [ ] `dotnet test tests/Hypa.GoldenTests` — all passing - [ ] `hypa mcp add <url>` probes the server and reports auth requirements - [ ] OAuth browser flow launches, completes exchange, and stores credentials - [ ] `hypa md README.md --toc` on a fresh clone auto-indexes and returns ToC - [ ] Second `hypa md README.md --toc` with no file changes is a no-op (no re-parse) - [ ] `hypa code index` is incremental by default; `--full` re-indexes then subsequent incremental is a no-op - [ ] `hypa code index --json` shows `markdown: ok` in provider health - [ ] Large `.md` files intercepted by the read hook produce a heading outline; `CLAUDE.md` passes through unchanged * feat(native): bundle libtree-sitter-markdown grammar and wire into CI and release pipeline - Add native/runtimes/linux-x64/native/libtree-sitter-markdown.so (pre-built) - Add scripts/build-tree-sitter-markdown.sh/.ps1 to build from source on any platform - Add Directory.Build.targets to copy the RID-appropriate grammar to every project output and publish root, matching how TreeSitter.DotNet flattens its native assets - ci.yml: build grammar before dotnet build on all matrix runners; aot-publish job rebuilds with FORCE_BUILD=1 and verifies symbol before and after publish - release.yml: build grammar (FORCE_BUILD=1) in every platform build job before dotnet publish; verify grammar file is present in publish output before packaging --------- Signed-off-by: Matthew Gribben <matthew@hypabolic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * chore: native grammar bundling for v0.1.1 release * added pip and homebrew release workflows * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * added gating to the release workflow * Add CI gate, concurrency control, and REPO_TOKEN for homebrew tap - Gate job verifies all CI checks passed on the tagged commit before any build or publish job runs; skipped for workflow_dispatch - Concurrency group prevents parallel release runs for the same ref - publish-homebrew now uses secrets.REPO_TOKEN for cross-repo push to homebrew-tap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Sync public source from PR #29 This pull request adds significant architectural changes as well as a GitHub Actions workflow for syncing source and test files to a public repository. **Architectural Decisions:** * Establishes a package boundary between provider-neutral structural search contracts (`Hypa.Sdk`) and the Tree-sitter-based engine implementation (`Hypa.CodePatterns`). This separation enables downstream consumers to use structural search results without taking a dependency on native parsing libraries, and allows for future NuGet publication of the engine. * Details the sequencing and rationale for three major 2026 features: MCP Proxy Layer, code grep (structural search/scan/rewrite), and Code Mode Scripting Engine. We plan phased delivery, risk mitigation, and integration points to maximize compound value and minimize technical risk. **Timeouts** * Some commands would hit the hypa internal timeout limit, mainly package manager, so we've introduced pipeline specific limits and an override command to allow the agent to specify a timeout plus more intelligent feedback when there is an error. **Automation and Workflow:** * Introduced a GitHub Actions workflow (`.github/workflows/sync-public.yml`) that automatically syncs the `src` and `tests` directories to the public `Hypabolic/Hypa` repository on pushes to `main`. The workflow includes PR metadata extraction, payload creation, public repo checkout, file replacement, and commit/push logic with descriptive commit messages. Source repo: matt-gribben/Hypa Source SHA: 25f836cb7f71a4f63dc3423d5a971bf57c713083 PR URL: Hypabolic/Hypa-Private#29 * feat: simplify commit message generation for public repo sync workflow action update * Enhance MCP server configuration, validation, and connection handling This pull request introduces several documentation updates and new documents to clarify workflows, address architectural issues, and plan for future features. The most significant changes include the addition of an ADR for GitHub Copilot Extension integration, detailed documentation for local preview installs, a bug report on `hypa_shell` error handling, and an architecture review highlighting key technical debt and recommendations. There are also minor roadmap clarifications and new usage/setup instructions for Hypa CLI tools. **Key changes:** **1. GitHub Copilot Extension Integration** - Added ADR 0011 describing the design and implementation plan for a thin Node.js ESM adapter to integrate Hypa tools and context with GitHub Copilot CLI sessions, including command rewriting and session lifecycle hooks. The ADR details the rationale, consequences, implementation strategy, and alternatives considered. **2. Local Preview Installs Documentation** - Added a new document and updated the `README.md` to describe the workflow for installing local preview builds of Hypa. This includes commands for installing, restoring, managing, and cleaning preview builds, as well as the effect on update behavior. [[1]](diffhunk://#diff-51a886bc8914e40ea5afcc9459caae38f5fa33b3e6bd25b5fc2e5cbe05f9af5aR1-R38) [[2]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R115-R153) **3. Bug Report: `hypa_shell` Error Handling** - Added a detailed bug report documenting how `hypa_shell` misreports missing executables as working directory errors, including root cause analysis, impact, suggested fixes, and regression test cases. **4. Architecture Review and Technical Debt** - Added an architecture review document identifying several high and medium priority issues, such as dependency rule violations, improper use of application ports, unimplemented ADRs, CLI orchestration bloat, adapter boundary drift, and improper unit test practices. Each issue includes recommendations for resolution. **5. Roadmap and Usage Documentation Updates** - Updated the strategic roadmap to remove week estimates and clarify the sequencing and rationale for technical spikes and MCP proxy layer work. [[1]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L47-R55) [[2]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L77-R77) [[3]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L108-R112) [[4]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690R232-R237) - Added a "Hypa Rules" section to `CLAUDE.md` with best practices for using Hypa CLI wrappers and initial setup instructions. * Release: MCP Proxy, Browser OAuth, and Markdown Mode ## Summary Three features developed across this release cycle, plus post-integration fixes and hardening discovered during live testing. ### MCP Proxy Layer A full MCP proxy implementation that lets Hypa act as a man-in-the-middle between agents and external MCP servers — forwarding tool calls, compressing responses, and exposing a unified tool search index. - `McpProxyService` with `DirectMcpDispatcher` for request routing and response compression - `McpTransportBuilder` for transport creation; `McpClientConnectionFactory` refactored to use it - MCP import with bearer token authentication and error handling - `McpServerProbeAdapter` — probes remote servers on `mcp add` to validate connectivity and detect auth requirements; `--no-probe` flag to skip - Tool search index for fast tool lookup across registered servers - Credential resolution with secret redaction in test output ### Browser OAuth Onboarding Automated OAuth flow for MCP servers that require browser-based authentication. - `OAuth2BrowserConfig` and interactive/non-interactive auth modes in `McpTransportBuilder` - Browser launch + local callback listener for authorization code exchange - DCR secret storage, TLS options, state validation, and manual paste fallback for restricted environments - Auth defaults to `none` on `mcp add`; guided setup triggered on probe auth failure ### Markdown Mode Structured Markdown indexing and querying via tree-sitter, with a freshness-aware query gate and read hook compression. - `MarkdownStructureProvider` and `CodePatternExtractor` — extracts sections (heading text, level, anchor, byte spans, plain text) and frontmatter YAML via tree-sitter - `hypa md <file>` subcommand with `--toc`, `--section`, `--frontmatter`, and `--json` flags - Git-aware incremental indexer (`IndexIncrementalAsync`): clean tracked files compared by blob OID (no file I/O), dirty/untracked files fall back to mtime+size — only stale files are re-parsed - `EnsureFreshAsync` gate on `hypa md` — auto-indexes on first use, re-indexes on change, no-op when fresh - `ReadRedirector` extended to compress large `.md` files to a heading outline; `CLAUDE.md` / `SKILL.md` pass through unchanged - `hypa code index --full` flag for forced full rebuild ### Post-integration fixes (found during live testing) - **`IndexFullAsync` OID regression**: full rebuilds stored `git_blob_oid = NULL`, causing the next incremental run to re-index every file. Fixed by calling `GetCleanBlobOidsAsync` in `IndexFullAsync` and persisting the OID alongside mtime. - **`libtree-sitter-markdown.so` not bundled**: `TreeSitter.DotNet` 1.3.0 doesn't ship the markdown grammar. Fixed by building from source and bundling via `Directory.Build.targets` (RID-aware, copies to output/publish root on all platforms). - **Provider conflict**: once the native library loaded, both `TreeSitterCodeStructureProvider` and `MarkdownStructureProvider` claimed `CanHandle("markdown")`, causing the generic provider to win and extract C# symbols from markdown files. Fixed with an explicit exclusion in `TreeSitterCodeStructureProvider.CanHandle`. - **CI hardening**: grammar build script runs on all matrix runners before the .NET build; AOT publish job verifies the symbol export and asserts `markdown: ok` in provider health after indexing. - **Missing test coverage**: added `MarkdownStructureProviderIntegrationTests` (exercises real tree-sitter native library, asserts sections and provenance) and `CodeStructureProviderRegistryTests` (asserts correct provider selection per language; regression guard ensures no two non-fallback providers claim the same language). ## Test plan - [ ] `dotnet test tests/Hypa.UnitTests` — 1325 tests, all passing - [ ] `dotnet test tests/Hypa.GoldenTests` — all passing - [ ] `hypa mcp add <url>` probes the server and reports auth requirements - [ ] OAuth browser flow launches, completes exchange, and stores credentials - [ ] `hypa md README.md --toc` on a fresh clone auto-indexes and returns ToC - [ ] Second `hypa md README.md --toc` with no file changes is a no-op (no re-parse) - [ ] `hypa code index` is incremental by default; `--full` re-indexes then subsequent incremental is a no-op - [ ] `hypa code index --json` shows `markdown: ok` in provider health - [ ] Large `.md` files intercepted by the read hook produce a heading outline; `CLAUDE.md` passes through unchanged * feat(native): bundle libtree-sitter-markdown grammar and wire into CI and release pipeline - Add native/runtimes/linux-x64/native/libtree-sitter-markdown.so (pre-built) - Add scripts/build-tree-sitter-markdown.sh/.ps1 to build from source on any platform - Add Directory.Build.targets to copy the RID-appropriate grammar to every project output and publish root, matching how TreeSitter.DotNet flattens its native assets - ci.yml: build grammar before dotnet build on all matrix runners; aot-publish job rebuilds with FORCE_BUILD=1 and verifies symbol before and after publish - release.yml: build grammar (FORCE_BUILD=1) in every platform build job before dotnet publish; verify grammar file is present in publish output before packaging --------- Signed-off-by: Matthew Gribben <matthew@hypabolic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * feat(instructions): update agent instructions for markdown and incremental indexing * feat(instructions): add hypa_mcp, hypa_read, hypa_search, and MCP server management to agent instructions * fix(packaging): correct license to FSL-1.1-ALv2 and add READMEs for npm and PyPI - npm/hypa and npm/hypa-platform: MIT -> FSL-1.1-ALv2; add README.md - python/pyproject.toml: MIT -> FSL-1.1-ALv2; point readme at README.md file instead of inline text placeholder - python/README.md: new file with install, quick start, and docs link - release.yml: copy README.md into npm staging dirs before publish so it appears on the npmjs.com package pages * Develop (#9) * added pip and homebrew release workflows * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * Potential fix for pull request finding Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Signed-off-by: Matthew Gribben <matthew@hypabolic.com> * added gating to the release workflow * Add CI gate, concurrency control, and REPO_TOKEN for homebrew tap - Gate job verifies all CI checks passed on the tagged commit before any build or publish job runs; skipped for workflow_dispatch - Concurrency group prevents parallel release runs for the same ref - publish-homebrew now uses secrets.REPO_TOKEN for cross-repo push to homebrew-tap Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * Sync public source from PR #29 This pull request adds significant architectural changes as well as a GitHub Actions workflow for syncing source and test files to a public repository. **Architectural Decisions:** * Establishes a package boundary between provider-neutral structural search contracts (`Hypa.Sdk`) and the Tree-sitter-based engine implementation (`Hypa.CodePatterns`). This separation enables downstream consumers to use structural search results without taking a dependency on native parsing libraries, and allows for future NuGet publication of the engine. * Details the sequencing and rationale for three major 2026 features: MCP Proxy Layer, code grep (structural search/scan/rewrite), and Code Mode Scripting Engine. We plan phased delivery, risk mitigation, and integration points to maximize compound value and minimize technical risk. **Timeouts** * Some commands would hit the hypa internal timeout limit, mainly package manager, so we've introduced pipeline specific limits and an override command to allow the agent to specify a timeout plus more intelligent feedback when there is an error. **Automation and Workflow:** * Introduced a GitHub Actions workflow (`.github/workflows/sync-public.yml`) that automatically syncs the `src` and `tests` directories to the public `Hypabolic/Hypa` repository on pushes to `main`. The workflow includes PR metadata extraction, payload creation, public repo checkout, file replacement, and commit/push logic with descriptive commit messages. Source repo: matt-gribben/Hypa Source SHA: 25f836cb7f71a4f63dc3423d5a971bf57c713083 PR URL: Hypabolic/Hypa-Private#29 * feat: simplify commit message generation for public repo sync workflow action update * Enhance MCP server configuration, validation, and connection handling This pull request introduces several documentation updates and new documents to clarify workflows, address architectural issues, and plan for future features. The most significant changes include the addition of an ADR for GitHub Copilot Extension integration, detailed documentation for local preview installs, a bug report on `hypa_shell` error handling, and an architecture review highlighting key technical debt and recommendations. There are also minor roadmap clarifications and new usage/setup instructions for Hypa CLI tools. **Key changes:** **1. GitHub Copilot Extension Integration** - Added ADR 0011 describing the design and implementation plan for a thin Node.js ESM adapter to integrate Hypa tools and context with GitHub Copilot CLI sessions, including command rewriting and session lifecycle hooks. The ADR details the rationale, consequences, implementation strategy, and alternatives considered. **2. Local Preview Installs Documentation** - Added a new document and updated the `README.md` to describe the workflow for installing local preview builds of Hypa. This includes commands for installing, restoring, managing, and cleaning preview builds, as well as the effect on update behavior. [[1]](diffhunk://#diff-51a886bc8914e40ea5afcc9459caae38f5fa33b3e6bd25b5fc2e5cbe05f9af5aR1-R38) [[2]](diffhunk://#diff-b335630551682c19a781afebcf4d07bf978fb1f8ac04c6bf87428ed5106870f5R115-R153) **3. Bug Report: `hypa_shell` Error Handling** - Added a detailed bug report documenting how `hypa_shell` misreports missing executables as working directory errors, including root cause analysis, impact, suggested fixes, and regression test cases. **4. Architecture Review and Technical Debt** - Added an architecture review document identifying several high and medium priority issues, such as dependency rule violations, improper use of application ports, unimplemented ADRs, CLI orchestration bloat, adapter boundary drift, and improper unit test practices. Each issue includes recommendations for resolution. **5. Roadmap and Usage Documentation Updates** - Updated the strategic roadmap to remove week estimates and clarify the sequencing and rationale for technical spikes and MCP proxy layer work. [[1]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L47-R55) [[2]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L77-R77) [[3]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690L108-R112) [[4]](diffhunk://#diff-5e4d86614a4a31c4b4fef6106f6de445905aac468f94d0cbd512f57d825d1690R232-R237) - Added a "Hypa Rules" section to `CLAUDE.md` with best practices for using Hypa CLI wrappers and initial setup instructions. * Release: MCP Proxy, Browser OAuth, and Markdown Mode ## Summary Three features developed across this release cycle, plus post-integration fixes and hardening discovered during live testing. ### MCP Proxy Layer A full MCP proxy implementation that lets Hypa act as a man-in-the-middle between agents and external MCP servers — forwarding tool calls, compressing responses, and exposing a unified tool search index. - `McpProxyService` with `DirectMcpDispatcher` for request routing and response compression - `McpTransportBuilder` for transport creation; `McpClientConnectionFactory` refactored to use it - MCP import with bearer token authentication and error handling - `McpServerProbeAdapter` — probes remote servers on `mcp add` to validate connectivity and detect auth requirements; `--no-probe` flag to skip - Tool search index for fast tool lookup across registered servers - Credential resolution with secret redaction in test output ### Browser OAuth Onboarding Automated OAuth flow for MCP servers that require browser-based authentication. - `OAuth2BrowserConfig` and interactive/non-interactive auth modes in `McpTransportBuilder` - Browser launch + local callback listener for authorization code exchange - DCR secret storage, TLS options, state validation, and manual paste fallback for restricted environments - Auth defaults to `none` on `mcp add`; guided setup triggered on probe auth failure ### Markdown Mode Structured Markdown indexing and querying via tree-sitter, with a freshness-aware query gate and read hook compression. - `MarkdownStructureProvider` and `CodePatternExtractor` — extracts sections (heading text, level, anchor, byte spans, plain text) and frontmatter YAML via tree-sitter - `hypa md <file>` subcommand with `--toc`, `--section`, `--frontmatter`, and `--json` flags - Git-aware incremental indexer (`IndexIncrementalAsync`): clean tracked files compared by blob OID (no file I/O), dirty/untracked files fall back to mtime+size — only stale files are re-parsed - `EnsureFreshAsync` gate on `hypa md` — auto-indexes on first use, re-indexes on change, no-op when fresh - `ReadRedirector` extended to compress large `.md` files to a heading outline; `CLAUDE.md` / `SKILL.md` pass through unchanged - `hypa code index --full` flag for forced full rebuild ### Post-integration fixes (found during live testing) - **`IndexFullAsync` OID regression**: full rebuilds stored `git_blob_oid = NULL`, causing the next incremental run to re-index every file. Fixed by calling `GetCleanBlobOidsAsync` in `IndexFullAsync` and persisting the OID alongside mtime. - **`libtree-sitter-markdown.so` not bundled**: `TreeSitter.DotNet` 1.3.0 doesn't ship the markdown grammar. Fixed by building from source and bundling via `Directory.Build.targets` (RID-aware, copies to output/publish root on all platforms). - **Provider conflict**: once the native library loaded, both `TreeSitterCodeStructureProvider` and `MarkdownStructureProvider` claimed `CanHandle("markdown")`, causing the generic provider to win and extract C# symbols from markdown files. Fixed with an explicit exclusion in `TreeSitterCodeStructureProvider.CanHandle`. - **CI hardening**: grammar build script runs on all matrix runners before the .NET build; AOT publish job verifies the symbol export and asserts `markdown: ok` in provider health after indexing. - **Missing test coverage**: added `MarkdownStructureProviderIntegrationTests` (exercises real tree-sitter native library, asserts sections and provenance) and `CodeStructureProviderRegistryTests` (asserts correct provider selection per language; regression guard ensures no two non-fallback providers claim the same language). ## Test plan - [ ] `dotnet test tests/Hypa.UnitTests` — 1325 tests, all passing - [ ] `dotnet test tests/Hypa.GoldenTests` — all passing - [ ] `hypa mcp add <url>` probes the server and reports auth requirements - [ ] OAuth browser flow launches, completes exchange, and stores credentials - [ ] `hypa md README.md --toc` on a fresh clone auto-indexes and returns ToC - [ ] Second `hypa md README.md --toc` with no file changes is a no-op (no re-parse) - [ ] `hypa code index` is incremental by default; `--full` re-indexes then subsequent incremental is a no-op - [ ] `hypa code index --json` shows `markdown: ok` in provider health - [ ] Large `.md` files intercepted by the read hook produce a heading outline; `CLAUDE.md` passes through unchanged * feat(native): bundle libtree-sitter-markdown grammar and wire into CI and release pipeline - Add native/runtimes/linux-x64/native/libtree-sitter-markdown.so (pre-built) - Add scripts/build-tree-sitter-markdown.sh/.ps1 to build from source on any platform - Add Directory.Build.targets to copy the RID-appropriate grammar to every project output and publish root, matching how TreeSitter.DotNet flattens its native assets - ci.yml: build grammar before dotnet build on all matrix runners; aot-publish job rebuilds with FORCE_BUILD=1 and verifies symbol before and after publish - release.yml: build grammar (FORCE_BUILD=1) in every platform build job before dotnet publish; verify grammar file is present in publish output before packaging * Enhance agent instructions for Markdown and MCP server management Update src and tests. --------- Signed-off-by: Matthew Gribben <matthew@hypabolic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> * Create pi-package-release.yml Signed-off-by: Matthew Gribben <matthew@hypabolic.com> --------- Signed-off-by: Matthew Gribben <matthew@hypabolic.com> Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Implemented Hypa Pi integration MVP and package release path - added Pi rewrite contract documentation - scaffold @hypabolic/pi-hypa Pi package - intercept Pi bash tool calls via hypa rewrite --json - add /hypa diagnostics and CLI-backed hypa_* tools - add Pi adapter for hypa init/uninstall --agent pi - add JSON array install/uninstall operations for Pi package wiring - add TypeScript and .NET tests for Pi integration - update CI, public repo sync, and Pi package publish workflow - document Pi install, configuration, and release publishing setup
Signed-off-by: Matthew Gribben <matthew@hypabolic.com>
Signed-off-by: Matthew Gribben <matthew@hypabolic.com>
Licence and npm package fix
Signed-off-by: Matthew Gribben <matthew@hypabolic.com>
Licence and npm package fix
Signed-off-by: Matthew Gribben <matthew@hypabolic.com>
Update src and tests.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.