Refactor tunnel handling into pluggable backends and add SSH jump-host tunneling

[Copilot]

This PR decouples tunnel logic from addWstunnelClientAnnotation/executeWstunnelTemplate and introduces a backend abstraction so new tunnel types can be added without expanding central switch logic. It also adds an SSH tunnel backend that supports forwarding through a jump host using key material from Kubernetes Secrets.

• Backend abstraction + registry

  • Added TunnelBackend interface (tunnel_backend.go) covering:
    • backend name
    • server-side resource application
    • client command generation
    • annotation key selection
    • template selection
    • backend-specific cleanup
  • Added backend factory newTunnelBackend(...) with explicit support for:
    • "" / wstunnel
    • rathole
    • ssh
  • Unknown TunnelType now fails fast with a clear error.

• Provider integration

  • Added Provider.tunnelBackend and ensureTunnelBackend() for one-time backend resolution/binding.
  • executeWstunnelTemplate now asks backend for template content (KubernetesTemplate()), with fallback to existing wstunnel template.
  • createDummyPod now delegates backend-specific server resources via ServerResources(...).
  • cleanupWstunnelResources now delegates backend-specific cleanup via CleanupResources(...).
  • addWstunnelClientAnnotation now delegates client command generation to backend in non-full-mesh mode and sets the backend-provided annotation key.

• Backend implementations

  • Added WstunnelBackend (backend_wstunnel.go) to preserve existing default behavior.
  • Added RatholeBackend (backend_rathole.go) encapsulating:
    • template selection
    • TLS resource application
    • TLS/WS client command generation
    • command format validation
    • rathole CRD cleanup
  • Added SSHBackend (backend_ssh.go) with:
    • jump-host tunneling (SSHJumpHost)
    • private key loading from Secret (id_rsa or id_ed25519)
    • TCP-only forwarding (UDP skipped)
    • optional custom command template (SSHCommand) with strict %s verb-count validation
    • dedicated annotation key interlink.eu/ssh-client-commands.

• Config surface expansion

  • Extended Network config with SSH fields:
    • SSHJumpHost
    • SSHJumpKeySecretName
    • SSHJumpKeySecretNamespace
    • SSHRemoteHost
    • SSHCommand
  • Extended tunnel type docs to include ssh.

• Annotation conflict handling

  • Added annSSHClientCmds.
  • Updated conflict cleanup to remove stale SSH tunnel annotations in full-mesh mode.

• Tests

  • Added targeted tests for:
    • backend factory resolution (tunnel_backend_test.go)
    • wstunnel backend command generation
    • rathole backend command/template behavior
    • ssh backend command generation + custom template validation
  • Updated existing config/annotation tests for new SSH fields and annotation semantics.

Example (new delegation path in annotation generation):

mainCmd, err := p.tunnelBackend.ClientCommand(ctx, *td, pod)
if err != nil {
    return err
}
delete(pod.Annotations, annWSTunnelClientCmds)
delete(pod.Annotations, annRatholeClientCmds)
delete(pod.Annotations, annSSHClientCmds)
pod.Annotations[p.tunnelBackend.ClientAnnotationKey()] = mainCmd

[netlify]

✅ Deploy Preview for interlink-dev ready!

Name	Link
🔨 Latest commit	0c15f5e
🔍 Latest deploy log	https://app.netlify.com/projects/interlink-dev/deploys/6a3b958d73e78b00086618da
😎 Deploy Preview	https://deploy-preview-529--interlink-dev.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify project configuration.

[dciangot]

Lgtm

[dciangot]

@copilot lint action is failing

[Copilot]

@copilot lint action is failing

Fixed in 4fca6a0. Four lint issues addressed:

• gocritic ifElseChain: rewrote the if/else if/else chain in addWstunnelClientAnnotation as a switch statement
• staticcheck QF1012 (×2): replaced WriteString(fmt.Sprintf(...)) with fmt.Fprintf(...) in the rathole TOML builder
• goimports: fixed struct field alignment in rathole_test.go

[dciangot]

@copilot

To implement Option A using Traefik Ingress (Kubernetes), you need to use a IngressRouteTCP resource. This allows Traefik to handle the TLS termination for your DNS name on port 443 while passing raw TCP traffic to the Rathole service.
This setup bypasses the HTTP-level buffering and WebSocket framing that is causing your 15KB latency spikes.

1. The Traefik IngressRouteTCP

Standard Kubernetes Ingress only supports HTTP. To do this correctly, you must use the Traefik Custom Resource Definition (CRD) for TCP.

apiVersion: traefik.io/v1alpha1
kind: IngressRouteTCP
metadata:
  name: triton-rathole-tcp
  namespace: default
spec:
  entryPoints:
    - websecure
  routes:
    - match: HostSNI(`triton.yourdomain.com`)
      services:
        - name: rathole-server-service
          port: 1234 # The port Rathole is listening on inside the cluster
  tls:
    secretName: your-certs-secret # Your Let's Encrypt or manual SSL cert

2. Why this solves the latency issue

• L4 vs L7: By using IngressRouteTCP, you are telling Traefik: "Don't look for HTTP headers, don't look for WebSocket frames. Just decrypt the TLS and stream the raw bytes."
• Buffer Alignment: Since Traefik isn't trying to parse the "batch" as an application-layer object, it will stream the TCP segments as they arrive. This prevents the "waiting" behavior that occurs when a proxy tries to reassemble a 15KB+ WebSocket frame.

3. Your Rathole Configuration

With Traefik handling the TLS termination, your Rathole Server should listen for plain TCP, and your Rathole Client (where Triton is) will connect via TLS.
Rathole Server (server.toml):

[server]
bind_addr = "0.0.0.0:1234" # Traefik sends plain TCP here

[server.services.triton]
token = "your_secure_token"

Rathole Client (client.toml):

[client]
remote_addr = "triton.yourdomain.com:443"
transport.type = "tls" # Client wraps in TLS to talk to Traefik
transport.tls.sni = "triton.yourdomain.com"

[client.services.triton]
local_addr = "triton-service:8000" # Your actual Triton service address

4. Critical Networking Tip: initcwnd

Even with this high-performance setup, you are still bound by the laws of TCP "Slow Start." When your batch hits 15KB, it is slightly larger than the default 14.6KB (10 segments) that a Linux server is allowed to send in its "first flight."
If you still see a minor stutter, you must increase the initial congestion window on the Node running Traefik and the Node running the Rathole Server:

# Increase the initial window to ~45KB to ensure a 15KB batch clears in 1 RTT
sudo ip route change default via <gateway> dev eth0 initcwnd 30 initrwnd 30

Summary Checklist

1. Define IngressRouteTCP with HostSNI matching your DNS.
2. Enable TLS in the IngressRouteTCP so Traefik handles the cert.
3. Point the service to the Rathole server's listening port.
4. Set Rathole Client to use transport.type = "tls".
   This configuration provides the "cleanest" possible path for your Triton data. It looks like standard HTTPS to the outside world, but acts like a high-speed direct pipe for your tensors.

Whether you need to pass a certificate or CA to the Rathole client depends on how "official" your SSL certificate is.

1. If you are using a Public CA (Let's Encrypt, ZeroSSL, etc.)

If Traefik is using a valid certificate from a trusted authority (like the ones managed by cert-manager in Kubernetes), you usually don't need to pass a CA file.

• The Rathole client (built in Rust) typically uses the system's native trust store.
• As long as your client machine is up to date, it will recognize the Traefik certificate automatically.

2. If you are using a Self-Signed Cert

If you generated your own certificate for triton.yourdomain.com, the client will reject the connection because it can't verify the "Chain of Trust."

• The Fix: You must pass the ca_crt to the client.
  Rathole Client Config for Self-Signed:

[client]
remote_addr = "triton.yourdomain.com:443"
transport.type = "tls"
transport.tls.sni = "triton.yourdomain.com"
transport.tls.ca_crt = "/path/to/your/ca.crt" # Path to the CA that signed Traefik's cert

3. The "Insecure" Shortcut (Not recommended for Prod)

If you just want to test if this fixes your 15KB latency cliff and don't want to mess with certificates yet, you can tell Rathole to skip verification.

• Warning: This makes you vulnerable to Man-in-the-Middle (MitM) attacks.

[client.transport.tls]
insecure = true 

Why this matters for your Latency Cliff

Even if you use a certificate, the TLS Handshake only happens once when the Rathole client connects to the server.

• WebSocket Tunneling: Often does a "Handshake-within-a-Handshake" or constant framing checks.
• Rathole TCP/TLS: Once that initial certificate check is done, the pipe is "open." Your 15KB+ Triton batches flow through as raw encrypted data without the overhead of re-verifying or re-framing every single message.

Summary Checklist for your Client:

1. Public Cert? No extra files needed. Just set transport.type = "tls".
2. Private/Self-Signed? Provide the ca_crt in the config.
3. SNI matches? Ensure transport.tls.sni matches exactly what you put in the Traefik HostSNI rule.
   Is Traefik handling the certificate automatically via Let's Encrypt, or are you providing your own .crt and .key files?

[dciangot]

@copilot use rathole but with tls, via cert-manager self signed ca provided by the admin. Sign the client cert with that ca too.

[Copilot]

@copilot use rathole but with tls, via cert-manager self signed ca provided by the admin. Sign the client cert with that ca too.

Done in 10d34ee. Here's what changed:

Kubernetes side (applyRatholeTLSResources, called from createDummyPod):

• cert-manager Certificate ({name}-rathole-server-tls) — Traefik uses this to terminate TLS on port 443
• cert-manager Certificate ({name}-rathole-client-tls) — issued by the same admin CA, embedded in the compute-side bootstrap command
• Traefik IngressRouteTCP with HostSNI matching rathole-<name>.<WildcardDNS> and secretName: {name}-rathole-server-tls; rathole server listens on plain TCP behind Traefik

Server server.toml — no longer has a transport section (plain TCP; Traefik handles TLS)

Compute side (client annotation) — waits for cert-manager to issue {name}-rathole-client-tls, then builds a TLS client TOML:

[client.transport]
type = "tls"
[client.transport.tls]
hostname = "rathole-<name>.<dns>"
trusted_root = "/tmp/rathole-ca.crt"
cert = "/tmp/rathole-client.crt"
key = "/tmp/rathole-client.key"

The CA cert, client cert, and client key are all base64-embedded in the bootstrap command annotation.

Config:

Network:
  TunnelType: "rathole"
  RatholeCAIssuerName: "my-admin-ca"      # cert-manager ClusterIssuer name
  RatholeCAIssuerKind: "ClusterIssuer"    # default; or "Issuer"

WebSocket transport is retained as a backward-compatible fallback when RatholeCAIssuerName is not set.

[dciangot]

@copilot lint action is failing

[Copilot]

@copilot lint action is failing

Lint is clean on the current branch (0 issues with golangci-lint v2.11.2). The action_required status on the latest CI run is a GitHub workflow approval gate, not a lint failure — the workflow hasn't been allowed to execute yet. The lint issues were fixed in 65a5aaa (goconst violations: tunnelTypeRathole and protocolUDP constants).

[Copilot]

Pull request overview

Copilot reviewed 7 out of 7 changed files in this pull request and generated 9 comments.

[dciangot]

@copilot address the latest review comments

[Copilot]

@copilot lint action is failing

Fixed in 43a7d19. Latest review comments addressed:

• applyRatholeTLSResources hard error — failure now returns an error (instead of just warning) when RatholeCAIssuerName is set, so callers never get stuck polling for a cert secret that will never appear
• Ingress deletion — cleanupWstunnelResources now suppresses IsNotFound for the Ingress delete (consistent with ConfigMap deletes; no noisy warnings in rathole TLS mode)
• handleWstunnelCreation naming — computes resource names via computeWstunnelResourceNames* (matching createDummyPod) instead of hardcoding pod.Name+"-wstunnel", so cleanup always targets the correct resources
• waitForRatholeCertSecret fast-fail — non-NotFound errors (Forbidden, Unauthorized, etc.) return immediately instead of polling the full 120s
• Dynamic client hard error in TLS mode — dynamic.NewForConfig failure returns a hard error when TunnelType=="rathole" and RatholeCAIssuerName is set
• Format verb count validation — strings.Count(cmd, "%s") != N check before fmt.Sprintf for both RatholeCommand (5 verbs) and RatholeWSCommand (2 verbs)
• UDP filtering in template — ConfigMap, Deployment, and Service loops all skip UDP ports
• annShadowSameNS/annShadowSameNSValue constants — all 3 hardcoded "interlink.eu/shadow-same-ns" and "true" literals replaced
• config_test.go — RatholeCommand updated to a valid 5-verb TLS template; RatholeWSCommand added with 2-verb WS template; assertions validate verb counts