Skip to content

milo-apiserver: support streaming watch-list (sendInitialEvents) for delegating apiservers #671

Description

@scotwells

Problem

Aggregated apiservers that delegate to milo-apiserver (IPAM today; Activity and others on the same pattern) can't use default client behavior. client-go's WatchListClient (now on by default) initializes informers with a streaming initial list (watch=true&sendInitialEvents=true&resourceVersionMatch=NotOlderThan). milo rejects these — e.g. 504 Timeout: Too large resource version: 1280714733, current: 1280614472 — so the delegating apiserver's informers (FlowSchema, PriorityLevelConfiguration, auth configmaps) never sync, /readyz hangs, and that service's API stays offline.

Impact

  • Any service delegating auth/flow-control to milo can silently fail readiness as WatchListClient becomes the client default — taking its API down on an otherwise-routine upgrade.
  • IPAM hit this and is offline until a per-service workaround ships. Each delegating service would otherwise need the same opt-out, and all of them lose the efficiency streaming list is meant to provide.

Ask

milo-apiserver should serve streaming watch-list consistently — honor sendInitialEvents with correct resourceVersion/NotOlderThan semantics and emit the initial-events-end bookmark — so delegating apiservers work with default client settings and need no per-service opt-out.

Context

  • Interim workaround: KUBE_FEATURE_WatchListClient=false on the delegating apiserver (datum-cloud/infra#2871; fix: stop disabling APF so ipam-apiserver readyz passes ipam#38 enabled APF so the informers run).
  • Reproduce: kubectl --kubeconfig <milo> get --raw "/apis/flowcontrol.apiserver.k8s.io/v1/flowschemas?watch=true&sendInitialEvents=true&resourceVersionMatch=NotOlderThan&allowWatchBookmarks=true" returns a 504 ResourceVersionTooLarge instead of the initial list + bookmark.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type
    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions