Proof of concept for reverse tunnel by edlerd · Pull Request #456 · canonical/microcloud-cluster-manager

edlerd · 2026-03-25T10:13:22Z

Done

added endpoint to open a tunnel from each cluster
proxy endpoint to serve the LXD-UI

Draft because there are open issues

Follow up of #207

webteam-app · 2026-03-25T10:13:26Z

Demo

Jenkins

demos.haus

Copilot

Pull request overview

Proof-of-concept implementation of a reverse tunnel from each remote cluster back to the cluster manager, plus a management-API proxy endpoint intended to serve the LXD-UI through that tunnel. This builds on the earlier spike work in #207 by introducing a shared tunnel store, websocket handling, and some auth/token persistence needed for proxying.

Changes:

Add reverse-tunnel plumbing: websocket endpoint on the cluster-connector, shared tunnel connection store, and HTTP-to-tunnel request/response forwarding.
Add management-API tunnel proxy endpoint and expose tunnel connectivity status via the REST model to the UI.
Add storage for OIDC access tokens (DB + encryption helpers) and a user_secret cookie to support per-user encryption.

Reviewed changes

Copilot reviewed 14 out of 15 changed files in this pull request and generated 16 comments.

Show a summary per file

File	Description
ui/src/types/cluster.d.ts	Adds `tunnel_connected` flag to cluster type for UI display.
ui/src/pages/clusters/ClusterDetail.tsx	Displays tunnel connectivity notification in cluster detail UI.
internal/pkg/types/rest.go	Introduces `TunnelStore`/`Tunnel` structs and tunnel request/response types.
internal/pkg/middleware/request.go	Adjusts request logging to better support websocket upgrades and hijacking.
internal/pkg/database/store/user_access_tokens.go	Adds DB access helpers for storing/retrieving per-user access tokens.
internal/pkg/database/store/remote_cluster_detail.go	Adds `tunnel_member_url` persistence to track which member holds a tunnel.
internal/pkg/database/schema/migrations/00001_db.sql	Adds `tunnel_member_url` column to `remote_cluster_details`.
internal/pkg/database/schema/migrations/00002_db.sql	Adds `user_access_tokens` table for access token storage.
internal/pkg/config/cypher.go	Adds helper functions to create secrets and encrypt/decrypt user values.
internal/pkg/api/models/v1/remote_cluster.go	Exposes `tunnel_connected` in the API model.
internal/pkg/api/api.go	Initializes and wires a shared `TunnelStore` into route config.
internal/app/management-api/core/auth/oidc.go	Adds `user_secret` cookie handling and stores encrypted access token in DB.
internal/app/management-api/api/v1/remote_cluster.go	Adds management-API tunnel proxy endpoint and sets `tunnel_connected`.
internal/app/management-api/api/v1/auth.go	Passes route config into OIDC callback handler.
internal/app/cluster-connector/api/v1/remote_cluster.go	Adds websocket tunnel endpoint and HTTP forwarding endpoints in cluster-connector.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 14 comments.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Copilot

Pull request overview

Copilot reviewed 16 out of 17 changed files in this pull request and generated 10 comments.

Copilot

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 13 comments.

Copilot

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated 4 comments.

omarelkashef

I did an initial review. Very good PR! A few comments below

omarelkashef · 2026-06-24T07:48:41Z

+	MemberStatuses    json.RawMessage `db:"member_statuses"`     // JSON array of member statuses
+	StoragePoolUsages json.RawMessage `db:"storage_pool_usages"` // JSON array of storage pool usages
+	UIURL             string          `db:"ui_url"`              // UI URL
+	TunnelMemberURL   string          `db:"tunnel_member_url"`   // Cluster manager member url that holds the tunnel


The name made me think the variable refers to a member in the cluster but the comment implies a member in cluster manager.

Any more specific terminology I can think of become very bulky, hence I went with this one. Do you have a suggestion?

What do you think of TunnelManagerMemberURL or TunnelHostMemberURL?

omarelkashef · 2026-06-24T08:24:23Z

 			InstanceStatuses:   instanceStatuses,
 			StoragePoolUsages:  storagePoolUsages,
 			UIURL:              e.UIURL,
+			TunnelConnected:    e.TunnelMemberURL != "",


Should this be a more thorough check to make sure the member is reachable?

I would trust the database state here.

I would trust the database state here.

The database state is correct, but it does not guarantee reachability. That would be fine if we do not want the overhead of testing reachability and we will return the error to the caller.

If we want to keep it this way, we can use "established" or "registered" (at least in the UI) instead of "connected" because I also noticed in the UI we inform the user that the tunnel is connected which implies that they can use it.

Signed-off-by: David Edler <david.edler@canonical.com>

edlerd · 2026-06-24T11:24:57Z

Thanks for the review @omarelkashef please have another pass at the changes.

omarelkashef

I did another round of review. Logically, it looks good. Only a few suggestions.

omarelkashef · 2026-06-25T10:38:29Z

+	UserSessions map[string]string
+}
+
+// ClusterManagerTunnelRequest represents the request received through the tunnel.


These comments are confusing. How about "request forwarded over reverse tunnel" here and "response received from to the reverse tunnel" below?

omarelkashef · 2026-06-25T10:41:58Z

 			InstanceStatuses:   instanceStatuses,
 			StoragePoolUsages:  storagePoolUsages,
 			UIURL:              e.UIURL,
+			TunnelConnected:    e.TunnelMemberURL != "",


I would trust the database state here.

The database state is correct, but it does not guarantee reachability. That would be fine if we do not want the overhead of testing reachability and we will return the error to the caller.

omarelkashef · 2026-06-25T10:43:33Z

+type Tunnel struct {
+	Mu           sync.RWMutex
+	WsConn       *websocket.Conn
+	PendingCalls map[string]chan ClusterManagerTunnelResponse


PendingCalls sounds like “pending send” rather than “waiting response”. How about PendingResponses, InFlightResponseCh, ResponseChByRequestID, etc.?

omarelkashef · 2026-06-25T11:00:40Z

+				return
+			}
+
+			tunnel.WsConn = nil


Should we also clear UserSessions and PendingCalls (the whole tunnel entry in TunnelByCluster) here to avoid memory leakage or leave UserSessions for future connection? We can also do the cleanup on remote cluster delete.

omarelkashef · 2026-06-25T11:04:58Z

 			InstanceStatuses:   instanceStatuses,
 			StoragePoolUsages:  storagePoolUsages,
 			UIURL:              e.UIURL,
+			TunnelConnected:    e.TunnelMemberURL != "",


If we want to keep it this way, we can use "established" or "registered" (at least in the UI) instead of "connected" because I also noticed in the UI we inform the user that the tunnel is connected which implies that they can use it.

omarelkashef · 2026-06-25T11:08:44Z

+			tunnel.Mu.Unlock()
+			return response.SmartError(errors.New("Tunnel not connected")).Render(w, r)
+		}
+		err = wsConn.WriteJSON(req)


I am not sure that this part where we send the request here and wait for the channel response coming from remoteClusterWsGet is easily understood on the first try, but I can not think of a better way to do it.

edlerd force-pushed the websocket branch from b9dcf2f to 9580f03 Compare March 25, 2026 12:45

edlerd force-pushed the websocket branch 3 times, most recently from 0bb3c83 to 4792f27 Compare April 29, 2026 15:29

edlerd force-pushed the websocket branch from 4792f27 to f16f100 Compare May 27, 2026 15:13

edlerd requested a review from Copilot May 27, 2026 15:15

Copilot started reviewing on behalf of edlerd May 27, 2026 15:15 View session

Copilot AI reviewed May 27, 2026

View reviewed changes

edlerd force-pushed the websocket branch from f16f100 to 6920b8d Compare May 27, 2026 16:30

edlerd requested a review from Copilot May 27, 2026 16:31

Copilot started reviewing on behalf of edlerd May 27, 2026 16:31 View session

Copilot AI reviewed May 27, 2026

View reviewed changes

edlerd force-pushed the websocket branch 4 times, most recently from 89d075a to 09bb67e Compare May 28, 2026 19:23

edlerd requested a review from Copilot May 28, 2026 19:24

Copilot started reviewing on behalf of edlerd May 28, 2026 19:25 View session

Copilot AI reviewed May 28, 2026

edlerd requested a review from Copilot May 28, 2026 19:52

Copilot started reviewing on behalf of edlerd May 28, 2026 19:52 View session

Copilot AI reviewed May 28, 2026

edlerd requested a review from Copilot May 29, 2026 06:32

Copilot started reviewing on behalf of edlerd May 29, 2026 06:32 View session

Copilot AI reviewed May 29, 2026

View reviewed changes

edlerd force-pushed the websocket branch 2 times, most recently from cd509d6 to fb42e7a Compare May 29, 2026 08:57

edlerd requested a review from Copilot May 29, 2026 08:57

Copilot started reviewing on behalf of edlerd May 29, 2026 08:57 View session

edlerd force-pushed the websocket branch 3 times, most recently from fc7dea7 to 32133a5 Compare June 2, 2026 07:19

edlerd force-pushed the websocket branch 4 times, most recently from 7bde0ff to cde8890 Compare June 16, 2026 09:43

edlerd force-pushed the websocket branch from cde8890 to c0f9535 Compare June 22, 2026 09:05

edlerd changed the title ~~Proof of concept for reverse tunnel [spike]~~ Proof of concept for reverse tunnel Jun 22, 2026

edlerd marked this pull request as ready for review June 22, 2026 09:06

edlerd force-pushed the websocket branch 3 times, most recently from d1c1b43 to 4941268 Compare June 22, 2026 10:18

edlerd requested a review from Copilot June 22, 2026 10:18

Copilot started reviewing on behalf of edlerd June 22, 2026 10:18 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

edlerd force-pushed the websocket branch from 4941268 to 7a0da5c Compare June 22, 2026 11:58

edlerd requested a review from Copilot June 22, 2026 11:59

Copilot started reviewing on behalf of edlerd June 22, 2026 11:59 View session

Copilot AI reviewed Jun 22, 2026

View reviewed changes

Comment thread internal/app/management-api/api/v1/remote_cluster.go

Comment thread internal/app/management-api/api/v1/remote_cluster.go

Comment thread internal/app/cluster-connector/api/v1/remote_cluster.go Outdated

Comment thread internal/app/cluster-connector/api/v1/remote_cluster.go Outdated

edlerd force-pushed the websocket branch 2 times, most recently from 6cec051 to 8845bd0 Compare June 23, 2026 07:54

omarelkashef reviewed Jun 24, 2026

View reviewed changes

edlerd force-pushed the websocket branch from 8845bd0 to b2f978c Compare June 24, 2026 11:10

feat(tunnel) proof of concept for reverse tunnel

a9322d1

Signed-off-by: David Edler <david.edler@canonical.com>

edlerd force-pushed the websocket branch from b2f978c to a9322d1 Compare June 24, 2026 11:24

edlerd requested a review from omarelkashef June 24, 2026 11:24

omarelkashef reviewed Jun 25, 2026

View reviewed changes

Uh oh!

Conversation

edlerd commented Mar 25, 2026

Done

Uh oh!

webteam-app commented Mar 25, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

omarelkashef left a comment

Choose a reason for hiding this comment

omarelkashef Jun 25, 2026 •

edited

Loading