fix: push-mode informers pick up rotated bearer token without restart#7663
fix: push-mode informers pick up rotated bearer token without restart#7663zhuyulicfc49 wants to merge 1 commit into
Conversation
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: The full list of commands accepted by this bot can be found here. DetailsNeeds approval from an approver in each of these files:Approvers can indicate their approval by writing |
Summary of ChangesHello, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed! This pull request addresses a critical issue where push-mode informers in Karmada would stop functioning after a member cluster's bearer token was rotated, as they held a stale, frozen token from startup. By implementing a token-refreshing transport layer, the system now ensures that long-lived connections periodically re-fetch credentials, allowing for seamless rotation and preventing silent failures in cluster monitoring. Highlights
Using Gemini Code AssistThe full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips. Invoking Gemini You can request assistance from Gemini at any point by creating a comment using either
Customization To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a Limitations & Feedback Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here. Footnotes
|
karmada-bot
added
size/XL
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces a mechanism to handle member cluster token rotation for long-lived clients by implementing a TTL-based tokenRefreshingRoundTripper that periodically re-reads the token from the Karmada Secret. It also adds comprehensive unit and E2E tests to verify this behavior. The reviewer's feedback suggests optimizing the locking mechanism in tokenRefreshingRoundTripper to prevent head-of-line blocking during network I/O, adding defensive nil checks to avoid potential panics, and formatting a function signature on a single line to adhere to the repository style guide.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| type tokenRefreshingRoundTripper struct { | ||
| inner http.RoundTripper | ||
| secretGetter func(namespace, name string) (*corev1.Secret, error) | ||
| secretNS string | ||
| secretName string | ||
|
|
||
| mu sync.RWMutex | ||
| cachedToken string | ||
| cacheExpiry time.Time | ||
| } |
There was a problem hiding this comment.
To prevent head-of-line blocking for all concurrent requests when the token expires, we should avoid holding the read/write lock (t.mu) during the network I/O call to fetch the secret. Adding a separate fetchMu mutex allows us to synchronize the token fetching process independently of token reading.
| type tokenRefreshingRoundTripper struct { | |
| inner http.RoundTripper | |
| secretGetter func(namespace, name string) (*corev1.Secret, error) | |
| secretNS string | |
| secretName string | |
| mu sync.RWMutex | |
| cachedToken string | |
| cacheExpiry time.Time | |
| } | |
| type tokenRefreshingRoundTripper struct { | |
| inner http.RoundTripper | |
| secretGetter func(namespace, name string) (*corev1.Secret, error) | |
| secretNS string | |
| secretName string | |
| mu sync.RWMutex | |
| cachedToken string | |
| cacheExpiry time.Time | |
| fetchMu sync.Mutex | |
| } |
There was a problem hiding this comment.
This is a good point! But I feel secretGetter reads from the controller-runtime in-memory informer cache, which is a map lookup and already fast enough(~100ns). And the write lock is held for that duration only, so concurrent requests are not meaningfully blocked I feel
| func (t *tokenRefreshingRoundTripper) forceRefresh() string { | ||
| t.mu.Lock() | ||
| defer t.mu.Unlock() | ||
|
|
||
| if time.Now().Before(t.cacheExpiry) { | ||
| return t.cachedToken | ||
| } | ||
|
|
||
| secret, err := t.secretGetter(t.secretNS, t.secretName) | ||
| if err != nil { | ||
| klog.Warningf("tokenRefreshingRoundTripper: failed to refresh token from secret %s/%s, keeping last known token: %v", | ||
| t.secretNS, t.secretName, err) | ||
| t.cacheExpiry = time.Now().Add(tokenCacheTTL) | ||
| return t.cachedToken | ||
| } | ||
|
|
||
| if token := string(secret.Data[clusterv1alpha1.SecretTokenKey]); token != "" { | ||
| t.cachedToken = token | ||
| } | ||
| t.cacheExpiry = time.Now().Add(tokenCacheTTL) | ||
| return t.cachedToken | ||
| } |
There was a problem hiding this comment.
Currently, forceRefresh holds the write lock t.mu during the entire network I/O call (t.secretGetter). This blocks all concurrent requests (even those just reading the token) for the duration of the API call.
By using t.fetchMu.TryLock(), we can ensure that only one goroutine performs the network I/O while all other concurrent requests immediately fall back to using the cached token instead of blocking. Additionally, we should add defensive checks to ensure secret and secret.Data are not nil before accessing them to prevent potential nil pointer dereference panics. Also, please add a doc comment to this unexported function to improve maintainability.
// forceRefresh handles the token refresh logic by fetching the secret and updating the cached token.
func (t *tokenRefreshingRoundTripper) forceRefresh() string {
if !t.fetchMu.TryLock() {
t.mu.RLock()
token := t.cachedToken
t.mu.RUnlock()
return token
}
defer t.fetchMu.Unlock()
t.mu.RLock()
if time.Now().Before(t.cacheExpiry) {
token := t.cachedToken
t.mu.RUnlock()
return token
}
t.mu.RUnlock()
secret, err := t.secretGetter(t.secretNS, t.secretName)
t.mu.Lock()
defer t.mu.Unlock()
if err != nil {
klog.Warningf("tokenRefreshingRoundTripper: failed to refresh token from secret %s/%s, keeping last known token: %v",
t.secretNS, t.secretName, err)
t.cacheExpiry = time.Now().Add(tokenCacheTTL)
return t.cachedToken
}
if secret != nil && secret.Data != nil {
if token := string(secret.Data[clusterv1alpha1.SecretTokenKey]); token != "" {
t.cachedToken = token
}
}
t.cacheExpiry = time.Now().Add(tokenCacheTTL)
return t.cachedToken
}References
- Add doc comments to unexported functions that contain key logic to improve maintainability.
There was a problem hiding this comment.
same reason as above I feel
| func NewTokenRefreshingRoundTripperWrapperConstructor( | ||
| secretGetter func(string, string) (*corev1.Secret, error), | ||
| secretNS, secretName, initialToken string, | ||
| ) transport.WrapperFunc { |
There was a problem hiding this comment.
According to the repository style guide, function signatures should preferably be written on a single line, including the parameter list and return types.
func NewTokenRefreshingRoundTripperWrapperConstructor(secretGetter func(string, string) (*corev1.Secret, error), secretNS, secretName, initialToken string) transport.WrapperFunc {References
- Function signatures should preferably be written on a single line, including the parameter list and return types. (link)
There was a problem hiding this comment.
Not sure if I should apply this... the existing NewProxyHeaderRoundTripperWrapperConstructor in the same file uses the same multi-line format. And no linting error with multi-line format..
zhuyulicfc49
force-pushed
the
fix/informer-token-rotation
branch
from
abf0414 to
a8091a4
Compare
There was a problem hiding this comment.
Pull request overview
This PR fixes push-mode member cluster clients (notably informer-backed long-lived watches) holding a stale bearer token after Secret rotation by moving token injection out of rest.Config.BearerToken and into a transport wrapper that periodically re-reads the token from the Secret (TTL-based refresh).
Changes:
- Removed the static
BearerTokenfield fromBuildClusterConfig()and added a token-refreshingWrapTransportlayer. - Introduced
tokenRefreshingRoundTripperwith a 30s cache TTL to refresh the bearer token from the member cluster Secret. - Added unit tests for the round tripper behavior, an integration-style test for
BuildClusterConfig()token rotation, and an e2e test covering informer recovery after rotation.
Reviewed changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| test/e2e/suites/base/token_rotation_test.go | New e2e covering token rotation + forced watch reconnection and validating informer-based collection recovers. |
| pkg/util/round_trippers.go | Adds the TTL-based token-refreshing http.RoundTripper wrapper used to inject updated bearer tokens. |
| pkg/util/round_trippers_token_test.go | Unit tests for token injection, TTL refresh, error handling, chaining, and concurrency dedup. |
| pkg/util/membercluster_client.go | Removes static bearer token config and wires the new token-refreshing wrapper into cluster client config construction. |
| pkg/util/membercluster_client_rotation_test.go | Integration-style test validating a long-lived client created via BuildClusterConfig() picks up a rotated Secret token without rebuild. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| saNamespace, saName := parseSAFromToken(string(secret.Data["token"])) | ||
| klog.Infof("cluster %s uses SA %s/%s on the member", targetCluster, saNamespace, saName) |
There was a problem hiding this comment.
Good catch — fixed both:
- Added
gomega.Expect(saName).ShouldNot(gomega.BeEmpty(), "cluster token must be a parseable SA JWT")right afterparseSAFromTokenso the test fails early with a clear message. - Replaced the literal
"token"withclusterv1alpha1.SecretTokenKeyin both places.
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #7663 +/- ##
==========================================
+ Coverage 42.04% 42.10% +0.05%
==========================================
Files 879 879
Lines 54827 54872 +45
==========================================
+ Hits 23053 23103 +50
+ Misses 30027 30024 -3
+ Partials 1747 1745 -2
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
zhuyulicfc49
force-pushed
the
fix/informer-token-rotation
branch
5 times, most recently
from
910aa4a to
b684d46
Compare
Signed-off-by: zhuyulicfc49 <zyliw49@gmail.com>
zhuyulicfc49
force-pushed
the
fix/informer-token-rotation
branch
from
b684d46 to
0f1a5ea
Compare
karmada-bot
removed
the
do-not-merge/work-in-progress
| tokenRotatePollRate = 5 * time.Millisecond | ||
| ) | ||
|
|
||
| func TestBuildClusterConfig_LongLivedClientPicksUpRotatedToken(t *testing.T) { |
There was a problem hiding this comment.
confirmed it fails without the fix :)
--- FAIL: TestBuildClusterConfig_LongLivedClientPicksUpRotatedToken (0.04s)
membercluster_client_rotation_test.go:112:
Not equal:
expected: "Bearer token-B"
actual : "Bearer token-A"
Messages: the long-lived client must put the ROTATED token on the wire after rotation
membercluster_client_rotation_test.go:116:
Received unexpected error: the server has asked for the client to provide credentials (get nodes foo)
Messages: after rotation the long-lived client must keep working (pick up token-B without a rebuild)
FAIL github.com/karmada-io/karmada/pkg/util 0.409s
| // - Token revocation (instant) instead of waiting for natural token expiry. | ||
| // - Docker restart to force watch reconnection — an open watch survives | ||
| // revocation since the API server only re-validates auth on reconnection. | ||
| var _ = framework.SerialDescribe("push-mode token rotation", func() { |
There was a problem hiding this comment.
- based on the CI logs the time spent of this test is TODO
- confirmed it failed without the fix
E2E Suite
Running Suite: E2E Suite - /home/ubuntu/projects/karmada/test/e2e/suites/base
=============================================================================
Random Seed: 1782267580
Will run 1 of 274 specs
------------------------------
[SynchronizedBeforeSuite]
/home/ubuntu/projects/karmada/test/e2e/suites/base/suite_test.go:142
I0624 02:19:52.538719 182298 cluster.go:332] Use context karmada-host
I0624 02:19:52.540695 182298 cluster.go:332] Use context karmada-apiserver
I0624 02:19:52.571884 182298 cluster.go:198] Got 3 member cluster and all in ready state.
I0624 02:19:52.668807 182298 cluster.go:332] Use context member3
STEP: Check if namespace(karmadatest-clpl4) present on member clusters @ 06/24/26 02:19:52.7
I0624 02:19:52.700444 182298 namespace.go:63] Waiting for namespace present on cluster(member1)
I0624 02:19:57.721375 182298 namespace.go:63] Waiting for namespace present on cluster(member2)
I0624 02:19:57.727812 182298 namespace.go:63] Waiting for namespace present on cluster(member3)
[SynchronizedBeforeSuite] PASSED [5.224 seconds]
------------------------------
[SSS… 273 specs skipped …SSS]
------------------------------
push-mode token rotation informers recover after member cluster token rotates [Serial]
/home/ubuntu/projects/karmada/test/e2e/suites/base/token_rotation_test.go:86
STEP: Creating PropagationPolicy(karmadatest-clpl4/deploy-5gzc4) @ 06/24/26 02:19:57.747
STEP: Creating Deployment(karmadatest-clpl4/deploy-5gzc4) @ 06/24/26 02:19:57.762
STEP: 1. verify workload propagated and status collected (baseline) @ 06/24/26 02:19:57.77
I0624 02:19:57.770902 182298 deployment.go:91] Waiting for deployment(karmadatest-clpl4/deploy-5gzc4) synced on cluster(member1)
E0624 02:19:57.773685 182298 deployment.go:95] Failed to get Deployment(karmadatest-clpl4/deploy-5gzc4) on cluster(member1), err: deployments.apps "deploy-5gzc4" not found
I0624 02:20:07.787185 182298 token_rotation_test.go:92] baseline: deployment karmadatest-clpl4/deploy-5gzc4 present on member1 with ready replicas
I0624 02:20:07.795420 182298 token_rotation_test.go:106] cluster member1 uses SA karmada-cluster/karmada-member1 on the member
STEP: 2. rotate: revoke old token, mint new, write to Secret @ 06/24/26 02:20:07.795
I0624 02:20:07.804461 182298 token_rotation_test.go:120] revoked old tokens by recreating SA karmada-cluster/karmada-member1
I0624 02:20:07.809051 182298 token_rotation_test.go:132] minted new token bound to recreated SA karmada-cluster/karmada-member1
I0624 02:20:07.816946 182298 token_rotation_test.go:137] wrote rotated token to Secret karmada-cluster/member1
I0624 02:20:07.816970 182298 token_rotation_test.go:140] waiting for revocation cache to flush...
STEP: 3. force watch reconnection (docker restart member node) @ 06/24/26 02:20:22.829
I0624 02:20:22.829856 182298 token_rotation_test.go:148] restarting kind container member1-control-plane to force watch reconnection
I0624 02:20:33.416766 182298 token_rotation_test.go:156] container member1-control-plane restarted
I0624 02:20:33.425924 182298 token_rotation_test.go:172] cluster Ready again — watch must now re-establish with the (potentially stale) token
STEP: 4. verify short-lived path still works (cluster stays Ready) @ 06/24/26 02:20:33.425
I0624 02:21:03.426409 182298 token_rotation_test.go:189] short-lived path: cluster stayed Ready throughout
STEP: 5. verify long-lived path recovered (informer sees member changes) @ 06/24/26 02:21:03.426
I0624 02:21:03.438838 182298 token_rotation_test.go:202] scaled deployment to 5 via Karmada control plane
I0624 02:21:03.442966 182298 token_rotation_test.go:212] karmada collected readyReplicas=3 (target 5)
I0624 02:21:08.450480 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:13.458580 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:18.466034 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:23.472915 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:28.477046 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:33.485613 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:38.492077 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:43.498345 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:48.505572 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:53.512496 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:21:58.517130 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:03.520879 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:08.526643 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:13.534435 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:18.547977 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:23.554826 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:28.558937 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:33.562834 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:38.570023 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:43.579846 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:48.587145 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:53.593676 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
I0624 02:22:58.597876 182298 token_rotation_test.go:212] karmada collected readyReplicas=0 (target 5)
[FAILED] in [It] - /home/ubuntu/projects/karmada/test/e2e/suites/base/token_rotation_test.go:215 @ 06/24/26 02:23:03.442
STEP: Removing Deployment(karmadatest-clpl4/deploy-5gzc4) @ 06/24/26 02:23:03.464
STEP: Removing PropagationPolicy(karmadatest-clpl4/deploy-5gzc4) @ 06/24/26 02:23:03.472
• [FAILED] [185.734 seconds]
push-mode token rotation [It] informers recover after member cluster token rotates [Serial]
/home/ubuntu/projects/karmada/test/e2e/suites/base/token_rotation_test.go:86
[FAILED] Timed out after 120.003s.
Karmada must observe the scaled status after rotation (informer recovered)
Expected
<bool>: false
to be true
In [It] at: /home/ubuntu/projects/karmada/test/e2e/suites/base/token_rotation_test.go:215 @ 06/24/26 02:23:03.442
Full Stack Trace
github.com/karmada-io/karmada/test/e2e/suites/base.init.func82.3.5()
/home/ubuntu/projects/karmada/test/e2e/suites/base/token_rotation_test.go:215 +0x3be
github.com/karmada-io/karmada/test/e2e/suites/base.init.func82.3()
/home/ubuntu/projects/karmada/test/e2e/suites/base/token_rotation_test.go:192 +0x6d5
------------------------------
[SSS… 38 specs skipped …SSS]
------------------------------
[SynchronizedAfterSuite]
/home/ubuntu/projects/karmada/test/e2e/suites/base/suite_test.go:203
[SynchronizedAfterSuite] PASSED [0.182 seconds]
------------------------------
Summarizing 1 Failure:
[FAIL] push-mode token rotation [It] informers recover after member cluster token rotates [Serial]
/home/ubuntu/projects/karmada/test/e2e/suites/base/token_rotation_test.go:215
Ran 1 of 274 Specs in 191.152 seconds
FAIL! -- 0 Passed | 1 Failed | 0 Pending | 273 Skipped
--- FAIL: TestE2E (191.17s)
FAIL
Ginkgo ran 1 suite in 3m23.033745281s
Test Suite Failed
exit status 1
GINKGO_EXIT:1
| func NewTokenRefreshingRoundTripperWrapperConstructor( | ||
| secretGetter func(string, string) (*corev1.Secret, error), | ||
| secretNS, secretName, initialToken string, | ||
| ) transport.WrapperFunc { |
There was a problem hiding this comment.
Not sure if I should apply this... the existing NewProxyHeaderRoundTripperWrapperConstructor in the same file uses the same multi-line format. And no linting error with multi-line format..
| type tokenRefreshingRoundTripper struct { | ||
| inner http.RoundTripper | ||
| secretGetter func(namespace, name string) (*corev1.Secret, error) | ||
| secretNS string | ||
| secretName string | ||
|
|
||
| mu sync.RWMutex | ||
| cachedToken string | ||
| cacheExpiry time.Time | ||
| } |
There was a problem hiding this comment.
This is a good point! But I feel secretGetter reads from the controller-runtime in-memory informer cache, which is a map lookup and already fast enough(~100ns). And the write lock is held for that duration only, so concurrent requests are not meaningfully blocked I feel
| func (t *tokenRefreshingRoundTripper) forceRefresh() string { | ||
| t.mu.Lock() | ||
| defer t.mu.Unlock() | ||
|
|
||
| if time.Now().Before(t.cacheExpiry) { | ||
| return t.cachedToken | ||
| } | ||
|
|
||
| secret, err := t.secretGetter(t.secretNS, t.secretName) | ||
| if err != nil { | ||
| klog.Warningf("tokenRefreshingRoundTripper: failed to refresh token from secret %s/%s, keeping last known token: %v", | ||
| t.secretNS, t.secretName, err) | ||
| t.cacheExpiry = time.Now().Add(tokenCacheTTL) | ||
| return t.cachedToken | ||
| } | ||
|
|
||
| if token := string(secret.Data[clusterv1alpha1.SecretTokenKey]); token != "" { | ||
| t.cachedToken = token | ||
| } | ||
| t.cacheExpiry = time.Now().Add(tokenCacheTTL) | ||
| return t.cachedToken | ||
| } |
There was a problem hiding this comment.
same reason as above I feel
| saNamespace, saName := parseSAFromToken(string(secret.Data["token"])) | ||
| klog.Infof("cluster %s uses SA %s/%s on the member", targetCluster, saNamespace, saName) |
There was a problem hiding this comment.
Good catch — fixed both:
- Added
gomega.Expect(saName).ShouldNot(gomega.BeEmpty(), "cluster token must be a parseable SA JWT")right afterparseSAFromTokenso the test fails early with a clear message. - Replaced the literal
"token"withclusterv1alpha1.SecretTokenKeyin both places.
What type of PR is this?
/kind bug
What this PR does / why we need it:
Push-mode informers hold a frozen bearer token from startup. After rotation, long-lived connections (resource watches) silently break while short-lived connections (health checks) keep working — the cluster shows
Ready=Truebut is partially blind.Fix: Remove the static
BearerTokenfromBuildClusterConfig(). Add aWrapTransportthat re-reads the token from the Secret on a 30s TTL — so long-lived clients pick up rotated credentials without a restart.Scope:
BuildClusterConfig()is the single entry point for all push-mode member cluster connections. The fix covers both categories:Backward compatibility: Static-token users see no behavior change — the RoundTripper re-reads the same value every 30s.
Which issue(s) this PR fixes:
Fixes #7562
Special notes for your reviewer:
Testing
Strategy:
I approached this using a shift-left mindset: first design an end-to-end scenario that fails before the fix (RED), then verify the same scenario passes after the fix (GREEN). For this issue, I also wanted the test to validate the suspected root cause by contrasting Karmada's two connection types: before the fix, token expiry should leave short-lived requests working while long-lived informers become stale; after the fix, both should continue working without a restart.
Once the manual RED/GREEN behavior was validated, I converted a similar scenario into a permanent e2e regression test for CI. The automated version uses token revocation and a Docker restart instead of waiting for natural token expiry, reducing the runtime from ~20 minutes to ~65 seconds while preserving the same coverage.
The diagram below shows the test breakdown (T1 = old 10-min token, T2 = new token):
Manual test results
Manual RED (without fix) — informer goes blind after natural token expiry
Environment:
local-up-karmadacluster (kind-based).Helpers:
kcp=karmada-apiserver,kh=karmada-host,km1=member1 direct. The key one:Setup:
Baseline (both paths healthy, before expiry):
Wait for T_short to expire (~10 min):
Watch naturally re-establishes with expired token → sustained Unauthorized:
Cluster still shows Ready (silent failure — health checks use the short-lived path):
Scale on member1 directly, then compare the two views:
Verdict: LIVE=4, COLLECTED=2 → DIVERGED — the informer is frozen on the expired token. Karmada reports
Ready=Truebut is blind to member state.Manual GREEN (with fix) — informer recovers after natural token expiry
Same setup, same natural re-watch, same cluster (rebuilt fresh with fix image). The CM was never restarted after token expiry — recovery is purely via the RoundTripper.
Baseline (before expiry):
T_short expires at 04:03:57 (same ~10 min wait). Then monitor for auth failures:
Zero auth failures across 14.5 minutes post-expiry (vs RED which 401s within ~2 min).
Informer healthy — no sustained errors:
Scale through Karmada and confirm the informer is actively collecting:
Verdict: LIVE=4, COLLECTED=4 → CONVERGED — the RoundTripper supplied T_long on the natural re-watch. The informer recovered without any restart.
Test coverage
BuildClusterConfig→ client → rotated token appears on wireDoes this PR introduce a user-facing change?: