fix(broker): widen LIST_GROUPS api versions#142
Merged
novatechflow merged 8 commits intoJun 2, 2026
Conversation
Minimal chart that deploys a single kafscale-broker Pod pointing at external etcd + S3 (MinIO or cloud). Intended for quick smoke tests and blueprint convergence on KIND — NOT a replacement for the full operator-based chart at deploy/helm/kafscale/. Used by scalytics-all-in-one bp-001 Ops Foundation smoke suite: COMP-kafscale-01 (pod Ready) + COMP-kafscale-02 (Kafka TCP reachable). Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Resolves OPS-005 item #2. The Java admin client (kafka-consumer-groups, AdminClient.listConsumerGroups, Schema Registry) negotiates LIST_GROUPS in range [0,4]. Advertising a narrow 5-5 window caused: UnsupportedVersionException: Error listing groups ... The broker does not support LIST_GROUPS with version in range [0,4]. The supported range is [5,5]. The underlying h.coordinator.ListGroups handler is version-agnostic; the encoder handles v0 just as well as v5. The fix is one line — widen the advertised range. Verified: kafka-consumer-groups --bootstrap-server kafscale-broker:9092 --list now exits 0 cleanly (from UnsupportedVersionException pre-fix). The remaining OPS-005 items (INIT_PRODUCER_ID, transactional APIs, Schema Registry NPE on verifySchemaTopic) are substantive broker-engineering work and are not addressed here. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Adds a minimum-viable implementation of the Kafka INIT_PRODUCER_ID API
(API key 22). Allocates a monotonically-increasing producer ID with epoch 0;
does not yet track sequence numbers or deduplicate on replay. Sufficient to
unblock Java AdminClient default producers, franz-go idempotent producers,
and Schema Registry's producer-init probe.
Changes:
- pkg/protocol/api.go: add APIKeyInitProducerID = 22
- cmd/broker/main.go:
* handler gains nextProducerID int64 (atomic allocator)
* dispatch case for *kmsg.InitProducerIDRequest returns pid + epoch=0
* apiVersions: InitProducerID moves from unsupported to {0, 4}
* import sync/atomic
Verified:
- `kafka-console-producer --producer-property enable.idempotence=true`
now succeeds (was: UnsupportedVersionException).
- kaf-mirror (franz-go) replicates primary→standby end-to-end:
PRIMARY offsets = STANDBY offsets, measured lag <1s.
- SCEN-bp002-06_Replication scenario test flipped SKIP → PASS.
Known limitations (production correctness gap, tracked in OPS-005):
- no sequence-number tracking: duplicate-on-retry semantics not enforced
- no epoch management: fencing of stale producers on rebalance not implemented
- PID allocator is process-local, not persisted across broker restart
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Collaborator
|
Tests fail (license), pls fix. |
Collaborator
|
@kamir - please see my last comment. |
novatechflow
changed the title
# Conflicts: # cmd/kafscale-cli/main.go
novatechflow
approved these changes
Jun 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
This PR now contains a single open-source broker compatibility fix:
LIST_GROUPSAPI version range from5..5to0..5Why
Older Kafka admin clients negotiate
LIST_GROUPSin the0..4range. Advertising only5..5causes admin tooling such askafka-consumer-groups --listto fail negotiation even though KafScale already handles the request.Scope
INIT_PRODUCER_IDsupportVerification
go test ./cmd/broker -run TestHandlerApiVersionsUnsupported|TestGenerateApiVersionsAdvertisesListGroupsCompatibility -count=1go test ./pkg/broker -run TestCoordinatorListDescribeGroups -count=1Notes
The earlier
INIT_PRODUCER_IDstub and standalone smoke-test Helm chart were removed from this PR because they are not appropriate for the open-source repo in their current form.