Summary
TestGnsiAuthzRotation intermittently fails in the ADO PR pipeline. Four subtests within the parent function are affected: RotatePolicyEmptyRequest, RotatePolicyEmptyUploadRequest, RotatePolicyWrongJSON, and RotatePolicyNoVersion. All other subtests in TestGnsiAuthzRotation are unaffected.
Symptom
The four subtests call expectPolicyMatch(t, authzTestPolicyFile, authzTestPolicyFileV1) to verify the policy file still contains V1 content after a failed rotation attempt. The assertion fails because the file contains stale or unexpected content.
Root Cause Analysis
The RotateStreamSendError subtest (which runs immediately before the failing tests) triggers a race condition:
- The test sends a valid V2 policy to the server, then immediately closes the gRPC connection.
- The server-side
Rotate handler processes the upload and writes V2 to disk via saveToAuthzFile.
- When the server attempts
stream.Send(resp), it fails due to the closed connection.
- The server enters the error path and calls
copyFile(backup, policyFile) to revert — but this runs asynchronously in the server's goroutine.
- Meanwhile, back in the test loop,
resetAuthzPolicyFile writes V1 to the policy file.
Steps 4 and 5 race: the server-side revert can overwrite the test's V1 reset with the checkpoint backup content, leaving the file in a state that does not match authzTestPolicyFileV1. The four subsequent subtests then fail their expectPolicyMatch assertions.
Affected Tests
- File:
gnmi_server/gnsi_authz_test.go
- Parent:
TestGnsiAuthzRotation
- Subtests:
RotatePolicyEmptyRequest (line 190)RotatePolicyEmptyUploadRequest (line 218)RotatePolicyWrongJSON (line 250)RotatePolicyNoVersion (line 281)
- Other subtests in
TestGnsiAuthzRotation are not affected
Temporary Mitigation
The affected subtests will be temporarily skipped via t.Skip() with a reference to this issue.
Proposed Fix
Either:
- Add synchronization in
RotateStreamSendError to wait for the server-side Rotate handler to fully return before the test loop calls resetAuthzPolicyFile (e.g., poll the authz mutex or add a completion callback).
- Or reorder the test cases so
RotateStreamSendError runs after the error-path subtests that depend on V1 file content.
References
Summary
TestGnsiAuthzRotationintermittently fails in the ADO PR pipeline. Four subtests within the parent function are affected:RotatePolicyEmptyRequest,RotatePolicyEmptyUploadRequest,RotatePolicyWrongJSON, andRotatePolicyNoVersion. All other subtests inTestGnsiAuthzRotationare unaffected.Symptom
The four subtests call
expectPolicyMatch(t, authzTestPolicyFile, authzTestPolicyFileV1)to verify the policy file still contains V1 content after a failed rotation attempt. The assertion fails because the file contains stale or unexpected content.Root Cause Analysis
The
RotateStreamSendErrorsubtest (which runs immediately before the failing tests) triggers a race condition:Rotatehandler processes the upload and writes V2 to disk viasaveToAuthzFile.stream.Send(resp), it fails due to the closed connection.copyFile(backup, policyFile)to revert — but this runs asynchronously in the server's goroutine.resetAuthzPolicyFilewrites V1 to the policy file.Steps 4 and 5 race: the server-side revert can overwrite the test's V1 reset with the checkpoint backup content, leaving the file in a state that does not match
authzTestPolicyFileV1. The four subsequent subtests then fail theirexpectPolicyMatchassertions.Affected Tests
gnmi_server/gnsi_authz_test.goTestGnsiAuthzRotationRotatePolicyEmptyRequest(line 190)RotatePolicyEmptyUploadRequest(line 218)RotatePolicyWrongJSON(line 250)RotatePolicyNoVersion(line 281)TestGnsiAuthzRotationare not affectedTemporary Mitigation
The affected subtests will be temporarily skipped via
t.Skip()with a reference to this issue.Proposed Fix
Either:
RotateStreamSendErrorto wait for the server-sideRotatehandler to fully return before the test loop callsresetAuthzPolicyFile(e.g., poll the authz mutex or add a completion callback).RotateStreamSendErrorruns after the error-path subtests that depend on V1 file content.References