fix(metastore): disable raft pipeline replication

[aleks-p]

A metastore replica in one of our dev cells stops being replicated to and never recovers. Only restarting the leader clears it. #4892 and #4935 helped other symptoms but not this one, and it still happens with #5307 deployed.

One theory is that it is a deadlock in hashicorp/raft's pipeline replication.

Technical details

The transport sizes both `netPipeline` channels at `MaxRPCsInFlight-2`, and we leave `MaxRPCsInFlight` at the default 2, so both are unbuffered. When a follower answers `Success=false`, `pipelineDecode` returns and closes `finishCh`, leaving `doneCh` with no reader; the SEND loop then races between the closed `finishCh` and a ready `triggerCh`, and picking `triggerCh` issues one more send that parks on `inprogressCh` while `decodeResponses` parks on `doneCh`. `pipeline.Close` is deferred inside `pipelineReplicate`, which is stuck in that select, so `shutdownCh` never closes.

replicate() never returns either, so its deferred close(stopHeartbeat) never runs. That is why the victim looks healthy from the outside: heartbeats keep arriving, so it never times out and forces an election that would have healed things — it just never receives another log entry.

Our own timeoutLogStore (#4892) is one way to produce the triggering Success=false, since raft's appendEntries handler returns without setting Success when StoreLogs fails. A rejected appendEntries at leader election does it too.

The theory was confirmed on local test cluster: the leader had four pipelines but only two pipelineDecode goroutines, with two senders parked in netPipeline.AppendEntries - blocked 4287 and 15948 minutes, one per unready replica. The 4287-minute one matched that replica's idle runFSM and its single log-store write timeout to within a few minutes.

Notes:

• This is a stop-gap; the real fix belongs upstream. fix liveness issue when leader's disk stalls hashicorp/raft#683 will likely improve things but is not a guarantee the problem will go away.
• TestRaftPipelineDeadlock reproduces the interlock; it fails on round 1 without the change. It deliberately asserts only that the interlock never forms, and not that the follower catches up again, because recovering from a stall that lands an abandoned log-store write also needs WIP: fix(metastore): fix follower livelock from abandoned log-store writes #5307 - without it a released follower livelocks on non-monotonic log entries instead. Worth tightening once WIP: fix(metastore): fix follower livelock from abandoned log-store writes #5307 lands.
• Draft because of the point above: this and WIP: fix(metastore): fix follower livelock from abandoned log-store writes #5307 are two halves of the same failure. With both applied the reproduction recovers cleanly in 20/20 rounds; with either alone the replica still stalls permanently.

cc @simonswine