Skip to content

Support AsyncStorageWrites in new asyncStorage pipeline impl #3

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
nvb wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Support AsyncStorageWrites in new asyncStorage pipeline impl #3

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
7296814
Move HardState from storage.Engine to storage.WAL
nvb Oct 25, 2022
067281c
Support AsyncStorageWrites in new asyncStorage pipeline impl
nvb Oct 25, 2022
17266cb
Avoid a few sources of flakiness and stalls
nvb Oct 26, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 7 additions & 7 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,13 @@ are driven by a `Peer` object, which performs scheduling and orchestrates their
interactions. Each components is defined as an interface so that different
implementations can be experimented with.

| Component | Description | Implementations |
|------------------|----------------------------------------------------------------------------------------|------------------|
| `storage/wal` | A write-ahead log used to store the Raft log | `mem`, `pebble`, `etcdWal` |
| `storage/engine` | A storage engine that Raft entries are applied to | `mem`, `pebble` |
| `storage` | A combined Raft log and storage engine. Can be implemented by a single entity or split | `splitStorage(wal,engine)`, `pebble` |
| `transport` | An interconnect between Raft peers to facilitate message passing | `grpc` |
| `pipeline` | A Raft proposal pipeline loop which handles `raft.Ready` objects | `basic`, `parallelAppender`, `asyncApplier(earlyAck, lazyFollower)` |
| Component | Description | Implementations |
|------------------|----------------------------------------------------------------------------------------|---------------------------------------------------------------------|
| `storage/wal` | A write-ahead log used to store the Raft log | `mem`, `pebble`, `etcdWal` |
| `storage/engine` | A storage engine that Raft entries are applied to | `mem`, `pebble` |
| `storage` | A combined Raft log and storage engine. Can be implemented by a single entity or split | `splitStorage(wal,engine)`, `pebble` |
| `transport` | An interconnect between Raft peers to facilitate message passing | `grpc` |
| `pipeline` | A Raft proposal pipeline loop which handles `raft.Ready` objects | `basic`, `parallelAppender`, `asyncStorage(earlyAck, lazyFollower)` |

The components are configured in `newPeer`. In the future they will all be hooked up to flags.

Expand Down
27 changes: 12 additions & 15 deletions go.mod
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,12 +1,14 @@
module github.com/nvanbenschoten/rafttoy

go 1.18

require (
github.com/cockroachdb/pebble v0.0.0-20221005185728-eec7375f9c44
github.com/gogo/protobuf v1.3.2
github.com/pkg/errors v0.9.1
github.com/rcrowley/go-metrics v0.0.0-20201227073835-cf1acfcdf475
github.com/spf13/pflag v1.0.5
go.etcd.io/etcd/raft/v3 v3.5.5
go.etcd.io/etcd/raft/v3 v3.6.0-alpha.0
go.etcd.io/etcd/server/v3 v3.5.5
go.uber.org/zap v1.23.0
golang.org/x/net v0.0.0-20221004154528-8021a29435af
Expand All @@ -23,6 +25,7 @@ require (
github.com/cockroachdb/errors v1.9.0 // indirect
github.com/cockroachdb/logtags v0.0.0-20211118104740-dabe8e521a4f // indirect
github.com/cockroachdb/redact v1.1.3 // indirect
github.com/coreos/go-semver v0.3.0 // indirect
github.com/getsentry/sentry-go v0.14.0 // indirect
github.com/golang/protobuf v1.5.2 // indirect
github.com/golang/snappy v0.0.4 // indirect
Expand All @@ -35,8 +38,9 @@ require (
github.com/prometheus/common v0.37.0 // indirect
github.com/prometheus/procfs v0.8.0 // indirect
github.com/rogpeppe/go-internal v1.9.0 // indirect
go.etcd.io/etcd/client/pkg/v3 v3.5.5 // indirect
go.etcd.io/etcd/pkg/v3 v3.5.5 // indirect
go.etcd.io/etcd/api/v3 v3.6.0-alpha.0 // indirect
go.etcd.io/etcd/client/pkg/v3 v3.6.0-alpha.0 // indirect
go.etcd.io/etcd/pkg/v3 v3.6.0-alpha.0 // indirect
go.uber.org/atomic v1.10.0 // indirect
go.uber.org/multierr v1.8.0 // indirect
golang.org/x/exp v0.0.0-20221006183845-316c7553db56 // indirect
Expand All @@ -46,19 +50,12 @@ require (
google.golang.org/protobuf v1.28.1 // indirect
)

go 1.18

// At the time of writing (i.e. as of this version below) the `etcd` repo is in the process of properly introducing
// modules, and as part of that uses an unsatisfiable version for this dependency (v3.0.0-00010101000000-000000000000).
// We just force it to the same SHA as the `go.etcd.io/etcd/raft/v3` module (they live in the same VCS root).
//
// While this is necessary, make sure that the require block above does not diverge.
replace go.etcd.io/etcd/api/v3 => go.etcd.io/etcd/api/v3 v3.0.0-20201109164711-01844fd28560
replace go.etcd.io/etcd/api/v3 => ../etcd/api

replace go.etcd.io/etcd/client/v2 => go.etcd.io/etcd/client/v2 v2.0.0-20201109164711-01844fd28560
replace go.etcd.io/etcd/server/v3 => ../etcd/server

replace go.etcd.io/etcd/client/v3 => go.etcd.io/etcd/client/v3 v3.0.0-20201109164711-01844fd28560
replace go.etcd.io/etcd/client/pkg/v3 => ../etcd/client/pkg

replace go.etcd.io/etcd/pkg/v3 => go.etcd.io/etcd/pkg/v3 v3.0.0-20201109164711-01844fd28560
replace go.etcd.io/etcd/pkg/v3 => ../etcd/pkg

replace go.etcd.io/etcd/raft/v3 => go.etcd.io/etcd/raft/v3 v3.0.0-20201109164711-01844fd28560
replace go.etcd.io/etcd/raft/v3 => ../etcd/raft
Loading