Performance improvement ideas for v3.8

[serathius]

What would you like to be added?

Want to collect ideas for improvements with large impact. While v3.6 was targeting stability and testing, the v3.7 should be a release that shows that there is still a lot of place for performance improvement in etcd. Goal is to inspire more ambitious work and start working towards it in v3.7. This is not exclusive list , and I'm happy to add ideas:

• RangeStream - Issue Large range queries from api-server could take down etcd due to OOM #12342 Bring the KEP-5116 to etcd. More about it in https://www.youtube.com/watch?v=SdLLOcNZN5E. Based on my initial research we cannot introduce streaming encoders for GRPC, so the only option seems like adding separate streaming range method. PoC in Draft implementation of streaming range request #19766 is very promising. Older attempt Yxjetcd rangestream #12343
• Async raft - Bring async raft improvements in raft: support asynchronous storage writes raft#8 to etcd. Issue Async raft #18027, Draft implementation
• Reduce memory usage of etcd member catchup mechanism - Introduce a mechanism to propagate a slowest member raft index, so other members can compact raft log based on it. Reduce memory usage of etcd member catchup mechanism #17098
• Incremental defrag - do minimal defrag operation part of normal transactions to reduce the need for run full defrag that locks whole database for tens of seconds Incremental etcd defrag bbolt#694
• Rewrite watch to ensure stable memory usage - Watch starvation can cause OOMs #16839 showed how fragile watch is. Like K8s etcd should have a mechanism that drops slow watchers and distributes watch events in a way that maintains a stable memory.
• Rewrite watch to prevent Put&Watch impacting others performance - Slow watchers impact PUT latency #18109 Lower throughput while having more and more watchers #19064
• Async log writing - Would be good to confirm performance impact of logging in etcd, I know sync info logging is issue for K8s. If there is impact we might consider implementing Support asynchronous writing of log file. #17071
• Improve performance for range with limit - One approach would be to rewrite index to store precomputed count within btree Experiments with replacing index implementation of btree with key -> revisions with revision list of immutable trees #18184. Other to provide option to disable count calculation. Optimize RangeRequest Revision count #16510. My preference would be to optimize to avoid overloading API with too many options.

Why is this needed?

Give community ideas to drive more interest in more harder but more impact contributions.

[serathius]

/cc @ahrtr @siyuanfoundation @fuweid

[chaochn47]

Thanks for the list!!

Another topic is the recommended max DB size quota is 8GiB

etcd/server/storage/quota.go

Line 33 in 9d24619

MaxQuotaBytes = int64(8 * 1024 * 1024 * 1024) // 8GB

but blog stated that The read and write operations when etcd stores 100GB of data can be as quickly as storing 2GB.

If so, what's the limiting factor? There might be performance improvement ideas coming from it.

[siyuanfoundation]

Thanks @serathius for the great list!

@chaochn47 One bottleneck I found in the previous investigation of extendible etcd is that the time to build the kvindex is linear with db size. So a newly joint server would take a long time to bootstrap if the size is large. This can be improved if the index can be persisted.

[serathius]

At this moment the suggested etcd size is 8 GB, however it's been known that it is not a limit. Many users (including GKE) run larger etcd sizes, taking the risk of untested configuration.

I have couple of ideas where the bottlenecks are (defrag, exchanging snapshots, bootstrap that @siyuanfoundation), however to fully qualify larged DB sizes we would need to define the etcd scalability tests #16467 and confirm that etcd maintains similar performance under larger size.

[hwdef]

/cc

I am very interested in these features and would like to contribute something to them.
And I agree with increasing the default max DB size.

[silentred]

Have you considered incremental compaction? Instead of scanning all revisions every compaction, an incremental mechanism would save a lot of computing and IO resources. In fact, a forked version with such mechanism in my company has proved it could reduce a lot of impact to online service during compaction. If you are interested, I would like to submit a proposal.

[serathius]

Have you considered incremental compaction?

sure, we would just need to some benchmarks to prove performance improvement.

[xigang]

Thanks @serathius