fix: auth failure metrics, command memory budget, migration progress - #323
Merged
Conversation
**h1 — auth failure counter**: adds `ember_auth_failures_total{reason}`
prometheus counter. reason is `wrongpass` for bad credentials, `noauth`
for commands sent before authenticating. incremented in `try_auth` and
at the NOAUTH error path in both connection modes.
**m1 — per-command memory budget**: adds `max_command_memory` (128MB
default) to `ConnectionLimits`. `validate_command_sizes` now sums total
bytes across all keys and values in MSET/LPUSH/RPUSH and rejects with
ERR if the combined payload exceeds the budget. guards against DoS via
million-element commands that individually pass per-item limits.
**m3 — migration progress overflow**: replaces `saturating_mul(100) /
total` with f64 division in `Migration::progress()`. the old formula
silently reported 100% when `keys_migrated * 100` overflowed u64.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
summary
three security and reliability fixes identified in the audit.
auth failure visibility (H1): adds
ember_auth_failures_total{reason}to prometheus. previously there was no way to alert on brute-force attempts or misconfigured clients flooding with NOAUTH errors. the counter is labeled by reason (wrongpass,noauth) and is incremented in both sharded and concurrent connection modes.command payload budget (M1):
validate_command_sizeschecked individual key and value sizes but not the total payload of a bulk command. an MSET with 1M × 1-byte key-value pairs would pass all per-item checks while consuming 2MB of memory and CPU just to parse. a newmax_command_memoryfield (128MB default) is added toConnectionLimits. MSET, LPUSH, and RPUSH now accumulate total bytes and return ERR if the budget is exceeded.migration progress arithmetic (M3): the old
(keys_migrated.saturating_mul(100) / total).min(100) as u8would silently jump to 100% whenkeys_migrated * 100overflowed u64 (at ~1.8 × 10^17 keys). fixed by switching to f64 division.note: M2 (incarnation counter saturation) was already addressed —
MAX_INCARNATION_JUMP = 1000is in gossip.rs since a prior PR.what was tested
cargo build -p ember-server -p ember-cluster— cleancargo test -p ember-server -p ember-cluster— 143 passed, 0 faileddesign considerations
max_command_memorydefaults to 128MB with no config flag today — it lives inConnectionLimitsso a follow-up PR can wire it to a TOML field if operators need to tune it.