Warn users about the doubled bucket situation in cluster and probably stop writing to doubled buckets to minimize the effect

[Serpentian]

When rebalancer downloads bucket states it can notice, that something is wrong with the cluster. But all it does is printing the message in logs:

vshard/vshard/storage/init.lua

Lines 2850 to 2852 in 7ccb617

	log.info('Total active bucket count is not equal to total. '..
	'Possibly a boostrap is not finished yet. Expected %d, but '..
	'found %d', M.total_bucket_count, sum)

However, users never care about logs until the real problem happens, but they care a lot about vshard alerts, which are pushed to cartridge/tcm.

I propose to push the alert to vshard.storage.info.alerts, if sum > M.total_bucket_count. It should go away automatically as soon as rebalancer notice, that sum is ok (note, that in case of rebalancer_download_states failure rebalancer starts to wake up every REBALANCER_WORK_INTERVAL (10 seconds), so pretty fast after cluster is fixed). It costs us nothing (one more bool variable in the memory) and will alert the user about possible catastrophe.

Note, that sum < M.total_bucket_count either mean, that we've lost some bucket (e.g. during cluster restore) or bootstrap is not finished, it'd be great, if we could distinguish these two cases, but there's no easy way to do so with the current code: we must set the persistent flag on storages, which indicates, that this storage is bootsrapped, and send that flag during vshard.storage.rebalancer_request_state to rebalancer. Though, I don't like introducing some logic for checking the situation which should never happen, if there're no bugs. So, no missing bucket alert for now.

[Serpentian]

Even after fixing #214 and #576, the doubled bucket situation remains possible:

• due to the misconfiguration (Doubled buckets due to excluding nodes during rebalancing #623).
• due to the invalid cluster backup restore.
• due to smth we still don't know about.

So, alert won't be excessive here

[Serpentian]

Also the doubled bucket can be achieved with backup restoration on the replicaset, from which the buckets were moved to another ones. We can also detect doubled buckets on routers in the discovery service and throw an alert.

[Serpentian]

Disclaimer

How it works now and how can we do better... This are just ideas, the whole thing requires proper RFC and design

---

The doubled-bucket state can be produced by user intervention, for example: enable rebalancing, let a bucket move from replicaset A to replicaset B, then restore A from a backup taken before the move. After that the bucket is ACTIVE on both A (resurrected from the old snapshot) and B (the legitimate new owner).

Current behavior

If a bucket becomes ACTIVE on two replicasets simultaneously (e.g. incorrect backup restoration), the router does not detect or report it. Discovery endlessly reassigns the bucket between the two replicasets, and router.call* keeps routing the bucket's requests to whichever replicaset most recently won a discovery round. This can silently split writes and reads across the two copies, and router.info() shows no alert.

Each storage independently reports the doubled bucket as ACTIVE from vshard.storage.buckets_discovery, so both replicasets claim the bucket in the same discovery round. In discovery_handle_buckets each replicaset's response is applied sequentially and steals the bucket from whoever currently owns it (old_rs ~= replicaset → count + 1 on the claimer, -1 on the old wner, then overwrite route_map). The last-processed claimer wins, so the map is overwritten back and forth every round:

2026-07-06 20:10:56.352 [79952] main/128/vshard.discovery._static_router/vshard.router init.lua:260 I> Updated replicaset(id="ac522f65-aa94-4134-9f64-51ee384f1a54", master=replica(id=storage_2_a, uri=storage@127.0.0.1:3303)) buckets: was 1499, became 1500
2026-07-06 20:10:56.352 [79952] main/128/vshard.discovery._static_router/vshard.router init.lua:265 I> Affected buckets of replicaset(id="cbf06940-0790-498b-948d-042b62cf3d29", master=replica(id=storage_1_a, uri=storage@127.0.0.1:3301)): was 1501, became 1500
2026-07-06 20:10:56.352 [79952] main/128/vshard.discovery._static_router/vshard.router init.lua:260 I> Updated replicaset(id="cbf06940-0790-498b-948d-042b62cf3d29", master=replica(id=storage_1_a, uri=storage@127.0.0.1:3301)) buckets: was 1500, became 1501
2026-07-06 20:10:56.352 [79952] main/128/vshard.discovery._static_router/vshard.router init.lua:265 I> Affected buckets of replicaset(id="ac522f65-aa94-4134-9f64-51ee384f1a54", master=replica(id=storage_2_a, uri=storage@127.0.0.1:3303)): was 1500, became 1499
2026-07-06 20:11:16.397 [79952] main/128/vshard.discovery._static_router/vshard.router init.lua:260 I> Updated replicaset(id="ac522f65-aa94-4134-9f64-51ee384f1a54", master=replica(id=storage_2_a, uri=storage@127.0.0.1:3303)) buckets: was 1499, became 1500
<...>

The total bucket count never changes — it is the same doubled bucket bouncing between the two owners.

What can we do about that

Why the router alone can't prevent it

The router is a cache, not an authority. It can only block a bucket that it has already been told is doubled, so there is always a detection window, and a freshly-started router (or one that has not run verification yet) routes blindly. So router-side blocking is containment + observability (defense-in-depth), not prevention. Real prevention has to happen where the invariant is violated: on the storage.

Maybe we don't need the steps 1 and 2 at all, storages will do only step 3 lazily, though, it's not cheap at all.

Step 1 — Suspect detection (router)

In discovery_handle_bucket, accumulate, per full discovery pass, a map claimed[bucket_id] -> replicaset. If a bucket is claimed by a second, different replicaset within the same pass, add it to router.suspected_buckets. When suspected_buckets becomes non-empty, wake the verification service. Detection is cheap and rides on discovery, which already runs; no extra RPCs here.

Though, the discovery might be disabled, routers won't be able to suspect the doubled buckets.

Step 2 — Verify and trigger recovery via map_callrw (router)

The verification service calls, over all masters, a new storage function through map_callrw (or masters_map_call), e.g. vshard.storage._call('bucket_check_doubled', {bucket_ids}). This is the "trigger recovery on storages" step: the call is not a passive stat. It makes each storage actively reconcile the listed buckets against the rest of the cluster, right now, instead of waiting for the periodic recovery tick.

Step 3 — Storage reconcile: find doubled, make READONLY, alert (storage)

On each storage master, for each requested bucket_id:

1. Read the local _bucket tuple. If it is absent or not ACTIVE/PINNED, return smth, nothing to do here
2. If it is ACTIVE/PINNED locally, run a targeted cluster check by reusing recovery_cluster_bucket_stat to learn who else holds the bucket and with which status and generation.
3. If we find the bucket anywhere else - we either make the bucket READONLY with opts.doubled = true, if generation on the remote is greater, or do that unconditionally (no matter what generation).

The storage alerts all readonly doubled buckets, returns error to router on write. Storage can (and probably should, since the discovery may be disabled) also lazy self-check the ACTIVE buckets in recovery