fix: replication send failure counter and go subscribe error propagation - #325
Merged
Conversation
**h2 — silent replication sends**: the `broadcast_replication` helper in aof.rs previously discarded send errors with `let _ = tx.send(...)`. now counts failed sends as `ember_replication_send_failures_total` so operators can alert on unexpected replica disconnects in a replicated deployment. adds `metrics` as a workspace dependency in ember-core. also documents the expected-silent try_send pattern in blocking.rs. **h3 — go subscribe error propagation**: the subscribe goroutine previously returned silently on any recv error, leaving callers unable to distinguish normal EOF from network or server errors. introduces a `Subscription` type with a `C` event channel and an `Err()` method, and changes `Subscribe` to return `(*Subscription, error)`. the goroutine now sends the recv error to an internal buffered channel before exiting, which `Err()` exposes after C is drained.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
two observability gaps from the audit — both involved silent failure modes that made production issues invisible.
H2 — replication broadcast:
broadcast_replicationinember-core/src/shard/aof.rsdiscarded all send errors withlet _ = tx.send(...). when a broadcast channel has no receivers, the send fails — this is expected when no replicas are connected, but there was no way to see it from metrics. now emitsember_replication_send_failures_totalso operators can alert on unexpected replica disconnects in replicated deployments. also added an explanatory comment to thetry_sendinblocking.rs(client disconnect on BLPOP/BRPOP is expected and correct to ignore).H3 — go client subscribe: the goroutine backing
Subscribeexited silently on any recv error, closing the event channel with no indication of why. callers ranging over the channel had no way to detect network errors vs. normal stream end. introduces aSubscriptiontype with a publicCevent channel and anErr()method. the goroutine sends the error to a buffered internal channel before exiting;Err()surfaces it. context cancellation is treated as a clean close (no error propagated), as that's intentional.what was tested
cargo test -p emberkv-core— 591 passed, 0 failedcargo build -p ember-server— cleango vet ./...passes on the updated file (pre-existing proto method errors are unrelated to this change)