[doc] Add page about latencies

[the-glu]

This PR adds a documentation page about latency, showing how it can affect performance.

Various benchmarks done for performance issues show that a "better" deployment, if possible, will help, and that there are probably non-achievable thresholds of performance in some configurations: resources like CPU and RAM aren't the only factors at play.

It includes tests, done to show how latency impacts performance, either artificially or with real-life deployments.

[mickmis]

LGTM modulo minor comments

[barroco]

Please see comments.

[barroco]

Add a link to the test.

[BenjaminPelletier]

Thanks for doing these experiments -- I think the results are going to be very important to define our system performance boundaries.

[BenjaminPelletier]

Could you help me understand why we would assume no parallelism? Of course latency and bandwidth are effectively different expressions of the same quantity when there is no (or fixed) parallelism, but it seems like that is not a very interesting nor relevant case. With parallelism (which represents real-world conditions), it seems like we could have any combination of {high and low} {latency and bandwidth} (for instance, carrier pigeons with micro SD cards would be extremely high latency but also high bandwidth). I am confused why this section seems to say high latency necessarily leads to lower bandwidth/throughput -- what is the mechanism by which high latency necessarily lowers throughput?

[the-glu]

Yes, there is always some parallelism and it can be increased up to certain limits, but the idea is to measure the impact in a fixed setting.

Also parallelism has it's own limits, it can be increased to a certain point, but not to infinity, and the same, relative graph will occur. (and practically that the Bandwidth-delay product you quoted bellow).

[BenjaminPelletier]

Same question here as above: why does latency limit how much (how much what?) can be realistically moved in a given time window? For instance, geostationary satellites (not Starlink) have massive latency, but also pretty massive bandwidth -- they can move a lot of data in a given time window even though they have very poor latency.

As written, it seems like this paragraph of explanation would also apply to Raftstore -- it is subject to the same latency as CRDB when deployed into the same substrate, yet Raftstore achieves much higher bandwidth. So, it seems like the dominant factor in achievable bandwidth isn't latency.

[the-glu]

Same question here as above: why does latency limit how much (how much what?) can be realistically moved in a given time window? For instance, geostationary satellites (not Starlink) have massive latency, but also pretty massive bandwidth -- they can move a lot of data in a given time window even though they have very poor latency.

Dues to the Bandwidth-delay product quoted bellow. That not the same setup as starlink, we're using single machine, with tcp, somewhat limited buffers, etc. And that also to be seeing in a "fixed settings", even with higher parallelism, you will hit thoses kind of limits.

As written, it seems like this paragraph of explanation would also apply to Raftstore -- it is subject to the same latency as CRDB when deployed into the same substrate, yet Raftstore achieves much higher bandwidth. So, it seems like the dominant factor in achievable bandwidth isn't latency.

That not something we measured, we measured q/s on the client side, not actual bandwidth they achieve on their internal links.

[BenjaminPelletier]

Is this a graph of the BD product using the 65k vanilla TCP limit? If so, is this limit actually relevant to our use case? I.e., is 100 Mbps insufficient to achieve order of 100 QPS? If not, what is the cause of the falloff?

[the-glu]

For the BD: Yes, that basically the point of this section. We should probably link the page.

It's relevant, because depending on data moved, you will start to hit those limits, especially with raft.

In a 9 node setup, with payload side of 35Kb (observed with lot of OVNs), that 220Mbits/s to send in total to others nodes.

[BenjaminPelletier]

The time to respond trend here makes perfect sense to me: when it takes longer to exchange information between internal nodes, of course the overall request will take longer to fulfill. In fact, we can even impute an estimate for the number of node traversals each request takes: 45ms node latency increases request latency by about 2200ms, putting the number of node-edge traversals per request at around 50.

However, I don't understand why there should fundamentally be any decrease in throughput due to latency. It certainly could be that queries interfere with each other and therefore have to wait, so contention + latency = decrease in throughput, but above we write that the queries were chosen not to create congestion issues which I take as intending to rule out this mechanism. There is a BD limit for TCP communication, but it seems like lots of little hops between nodes rather than one big hop should rule this out as a mechanism as well. What is our theory for why throughput on this graph drops with latency?

[the-glu]

It certainly could be that queries interfere with each other and therefore have to wait, so contention + latency = decrease in throughput, but above we write that the queries were chosen not to create congestion issues which I take as intending to rule out this mechanism.

Yes, there are no contention. However, there are always some penalty for sync / data to be transferred around to get results and that have some impact.

There is a BD limit for TCP communication, but it seems like lots of little hops between nodes rather than one big hop should rule this out as a mechanism as well.

Directly in that case, yes, we can probably exclude DB.

What is our theory for why throughput on this graph drops with latency?

On top of previous part (needing to get data), since we're in a "fix number of queries in //", there will always be limit dues to the fixed number of queries that can be processed at the time.

Assuming 9 processes doing queries, if one query individually take 10 time more, you will still get 10 time less q/s globally.

[BenjaminPelletier]

I think we need to be precise here: what does "how fast queries can be processed" mean? Certainly latency controls how fast a query can be processed (and therefore how fast queries can be processed), but I don't see the incompressibility of how fast a batch of queries can be processed due to latency. If we mean the former:

Suggested change

	operation, there is a incompressible limit on how fast queries can be processed.
	operation, there is a incompressible limit on how fast a query can be processed.

...but I don't see how that is very interesting since the thing that is really mysterious here to me is how latency leads to an incompressible limit on bandwidth. And, if latency actually leads to an incompressible limit on bandwidth, what is the mechanism and how to Raftstore overcome it?

[the-glu]

I think we need to be precise here: what does "how fast queries can be processed" mean? Certainly latency controls how fast a query can be processed (and therefore how fast queries can be processed), but I don't see the incompressibility of how fast a batch of queries can be processed due to latency.

Yes, that more for a single query, but also no, because you cannot have infinie-sized batch of queries to have unlimited queries processing. Simple limits would be how much you can keep in memory, how much you receive from clients (and how long do you wait to regroup them), or even number of open connection that will at least be limited to at most 65k.

...but I don't see how that is very interesting since the thing that is really mysterious here to me is how latency leads to an incompressible limit on bandwidth.

Maybe the bandwidth part is a bit confusing: the section above was meant to show 'data transfer limits' and here I don't think 'bandwidth' in term of data / s is reverent.

And, if latency actually leads to an incompressible limit on bandwidth, what is the mechanism and how to Raftstore overcome it?

The main difference is in the number of 'synchronization step'. In a very simplified way, raft only need one round trip exchange per "operation|http query", whereas in cockroachdb do that per "sql query" and they can me multiple ones.

[BenjaminPelletier]

I think this document is intended as a description of a consistent phenomenon/behavior rather than a specific test report, so we should limit the information displayed to that which illustrates the point and is expected to be reasonably reproducible under comparable conditions and procedure. So, for instance, it seems like the QPS should be reported with zero decimals (19 q/s, 19 q/s, 18 q/s) -- otherwise, the reader may be confused looking for the semantic significance of the difference between 19.35 and 19.01.

Also, there seems to be a substantial difference in 95th percentile latency -- is this significant? If so, let's narrate that significance. If not, let's omit it to avoid distracting the reader from the significant information, or else add narration explaining why the difference isn't significant.

[the-glu]

otherwise, the reader may be confused looking for the semantic significance of the difference between 19.35 and 19.01.

Yes ok, I will do a pass of rounding.

Also, there seems to be a substantial difference in 95th percentile latency -- is this significant? If so, let's narrate that significance. If not, let's omit it to avoid distracting the reader from the significant information, or else add narration explaining why the difference isn't significant.

We should probably narrate it, but one important part of the 95th is to check if some queries timeout when it reach 10000ms.

[BenjaminPelletier]

Do we think 2 percent for QPS and 50th percentile latency is a true difference that would persist at approximately that level if we were to repeat the exact same experiment? If the resolving power of this experiment is that fine, why does N3 latency decrease by 6%?

[the-glu]

I think so yes.

For N3 specifically (and in others situation bellow):

N3 is more sensitive to more queries, as it need to synchronize with it peers in the other provider, where the leader (N1) is.

[BenjaminPelletier]

This is confusing: the table shows a substantial drop in throughput, yet the narration is that the impact is pure latency. I don't disagree with the explanation, but I think that means we are not presenting the data as clearly as practical. If QPS and latency are directly linked through an arithmetic relationship, there is certainly no reason to show both. If QPS and latency are sometimes linked through an arithmetic relationship and sometimes not, it seems like we should be able to come up with a different metric that is orthogonal and report that instead of one of the sometimes-linked metrics.

[the-glu]

Yes: there are two latency: the one between the nodes and the ones of request.

It need to be rewritten (specially since the number is the other latency in the same sentence).

[BenjaminPelletier]

Do you know how 2ms can be possible? It seems like a database operation cannot be complete until a majority of the Raft group has adopted it, and the operation cannot have been adopted by any other node in 2ms when communication requires a minimum of 30ms.

[the-glu]

That a good question. N1 is definitely the leaded in that case, and I think the reasonable explanation is that it's doing optimizations / caching, knowing it is the leader and don't wait for adoption?

I can investigate more if you want, I never noticed that strange, but I think that consistent in general with what I saw.

[BenjaminPelletier]

This seems extremely surprising; what mechanism would explain N2 latency increasing to 34x while N3 only increased to 1.5x? It seems like that is the most significant feature of this data and would need to be explained before we could draw any other conclusion -- averaging 34x and 1.5x to get 24x seems to be ignoring whether the 1.5-34x range is reasonable (and therefore whether an average is appropriate).

[the-glu]

I'm wondering if I made a typo when recording the 2200ms (220ms?), because that would feet the *4 ratio between N2 and N3, so I simulated the same setup locally (with modified scripts - having the option at some point would be great ^^').

I'm getting the same kind of results (explainable by the fact that it's on my machine, latencies are simulated, etc..):

N1 at high Q/S (around 17), ~35ms for 50th latency, ~350ms for 95th
N2 is basically not being able to follow, with high oscillations, between 0 and 8 q/s, latencies going into timeouts.
N3 oscillating, but around 12 q/s as well. 55ms for 50th latency, between 500 and 1000ms for 95th

(Note: Graphs are cut to the end of the test - I refreshed them to be sure to have correct value but not directly after - leaving a big empty space)

Network matrix (N2/N3 swapped dues to initialization order but I stay consistent everywhere else):

When I tested (I lost the graph but it was not that bad), N2 was probably on the edge of being still functional, with high but still working latencies.

(That just a feeling because I don't have data, but I also suspect cockroach's range are spread differently, leading to worse performances, especially since now N1 have one RTT of latency)

---

Now for the potential explanation: N2 has 2 time the latency to the leader (N1) than N3. I would think that it's making it less 'capable' of making it transactions passing, as they are longer. N1 mainly have to talk to it self and can process/apply transaction very quickly, whereas N2 has to wait for a reply and have a very higher penalty to retry them. Seeing that was probably just bellow the limit, I think the 2200ms is possible.

[the-glu]

Here is also the diff I used to simulate that: forcedlat.patch

[the-glu]

@BenjaminPelletier Thanks for the review, I tried to give more details/context, let me know if it's more clear; we can then make improvement to the document to make it better.