feat: self-heal backplane subscriptions after a cluster reconnect

[d-barker]

Summary

Adds an opt-in mechanism to automatically restore the SignalR backplane's stream subscriptions after the Orleans cluster connection drops and recovers.

Problem

When the Orleans client loses its connection to the cluster and later reconnects (for example, a full cluster recycle such as scaling the silo from one replica to many), the backplane's implicit stream subscriptions are not re-established. The client reconnects to a new cluster generation, but the hub server's in-memory SERVER_STREAM / ALL_STREAM observers are orphaned — so affected servers silently stop receiving backplane messages until the hub server process is restarted.

Change

• OrleansSignalRConnectionMonitor — an IClientConnectionRetryFilter that doubles as a connection-loss signal. On every failed connection attempt it raises ConnectionLost; OrleansHubLifetimeManager subscribes and re-establishes its stream subscriptions as soon as the cluster is reachable again.
• OrleansSignalRConnectionMonitorOptions — configurable MaxRetryAttempts (0 = retry indefinitely, the default), RetryDelay (default 5s), and an optional InnerRetryFilter to compose with an existing client retry/back-off policy.
• AddSignalRBackplaneSelfHealing() — client-builder extension to register the monitor.
• OrleansHubLifetimeManager — wires in the re-subscription path.
• Unit tests covering retry attempts, retry exhaustion, and inner-filter composition.

Client usage

Opt in on the Orleans client (the hub server's client builder), alongside the existing UseSignalR() call:

var client = new ClientBuilder()
    .UseSignalR()
    .AddSignalRBackplaneSelfHealing();

Tune the retry policy via options:

clientBuilder
    .UseSignalR()
    .AddSignalRBackplaneSelfHealing(options =>
    {
        options.RetryDelay = TimeSpan.FromSeconds(2);  // back-off between failed attempts (default: 5s)
        options.MaxRetryAttempts = 0;                  // 0 = retry forever (default); >0 gives up after N
    });

Compose with an application's existing IClientConnectionRetryFilter — the monitor still emits the reconnect signal, but defers the keep-retrying decision (and its back-off) to your filter:

clientBuilder
    .UseSignalR()
    .AddSignalRBackplaneSelfHealing(options =>
    {
        options.InnerRetryFilter = new MyExistingRetryFilter();  // RetryDelay / MaxRetryAttempts are ignored when set
    });

Note: AddSignalRBackplaneSelfHealing() registers the monitor as the client's IClientConnectionRetryFilter. If you already register your own retry filter, pass it via InnerRetryFilter so its policy is preserved rather than replaced.

Notes

• Entirely opt-in — no behaviour change unless AddSignalRBackplaneSelfHealing() is called.
• Independent of fix: tolerate missing/invalidated stream subscription handles in grain activation #169 and fix: harden ClientGrain.OnDisconnect against stream errors and reentrancy #170 (those touch the grain layer; this touches the hub lifetime manager / client builder). No textual overlap.

🤖 Generated with Claude Code