[BUG] "Failed to load API tokens on node start" on second startup

[rursprung]

What is the bug?
this error message is triggered on the second startup of the node:

{"type": "server", "timestamp": "2026-07-09T13:54:27,212+0000", "level": "WARN", "component": "o.o.s.OpenSearchSecurityPlugin", "cluster.name": "test-opensearch", "node.name": "test-opensearch-0", "message": "Failed to load API tokens on node start", "cluster.uuid": "ypjqBTpeTlGM3E3-so_AFg", "node.id": "6r-1M2ewTF2p5ER5hIM_wQ" , 
"stacktrace": ["OpenSearchSecurityException[Received error while reloading API tokens metadata from index]; nested: ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];];",
"at org.opensearch.security.action.apitokens.ApiTokenRepository$1.onFailure(ApiTokenRepository.java:101) ~[?:?]",
"at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-3.7.0.jar:3.7.0]",
"at org.opensearch.action.support.TransportAction$1.onFailure(TransportAction.java:124) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.search.TransportSearchAction.lambda$doExecute$2(TransportSearchAction.java:349) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-3.7.0.jar:3.7.0]",
"at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-3.7.0.jar:3.7.0]",
"at org.opensearch.core.action.ActionListener$1.onFailure(ActionListener.java:90) ~[opensearch-core-3.7.0.jar:3.7.0]",
"at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:84) ~[opensearch-core-3.7.0.jar:3.7.0]",
"at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:140) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.index.query.Rewriteable.rewriteAndFetch(Rewriteable.java:105) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.search.TransportSearchAction.lambda$executeRequest$0(TransportSearchAction.java:508) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-3.7.0.jar:3.7.0]",
"at org.opensearch.search.pipeline.PipelinedRequest.transformRequest(PipelinedRequest.java:73) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.search.TransportSearchAction.executeRequest(TransportSearchAction.java:515) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:352) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.search.TransportSearchAction.doExecute(TransportSearchAction.java:142) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:220) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:313) ~[?:?]",
"at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:186) ~[?:?]",
"at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.support.TransportAction.execute(TransportAction.java:190) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.support.TransportAction.execute(TransportAction.java:109) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.transport.client.node.NodeClient.executeLocally(NodeClient.java:113) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.transport.client.node.NodeClient.doExecute(NodeClient.java:100) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.transport.client.support.AbstractClient.execute(AbstractClient.java:501) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.transport.client.support.AbstractClient.search(AbstractClient.java:632) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.security.action.apitokens.ApiTokenIndexHandler.getTokenMetadatas(ApiTokenIndexHandler.java:104) ~[?:?]",
"at org.opensearch.security.action.apitokens.ApiTokenRepository.reloadApiTokensFromIndex(ApiTokenRepository.java:78) ~[?:?]",
"at org.opensearch.security.OpenSearchSecurityPlugin.onNodeStarted(OpenSearchSecurityPlugin.java:2396) ~[?:?]",
"at org.opensearch.node.Node.lambda$start$2(Node.java:2120) ~[opensearch-3.7.0.jar:3.7.0]",
"at java.base/java.util.ArrayList.forEach(ArrayList.java:1604) ~[?:?]",
"at org.opensearch.node.Node.start(Node.java:2120) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.bootstrap.Bootstrap.start(Bootstrap.java:349) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.bootstrap.Bootstrap.init(Bootstrap.java:423) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.bootstrap.OpenSearch.init(OpenSearch.java:168) [opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.bootstrap.OpenSearch.execute(OpenSearch.java:159) [opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.common.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:110) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.cli.Command.mainWithoutErrorHandling(Command.java:138) ~[opensearch-cli-3.7.0.jar:3.7.0]",
"at org.opensearch.cli.Command.main(Command.java:101) ~[opensearch-cli-3.7.0.jar:3.7.0]",
"at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:125) [opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.bootstrap.OpenSearch.main(OpenSearch.java:91) [opensearch-3.7.0.jar:3.7.0]",
"Caused by: ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]",
"at org.opensearch.cluster.block.ClusterBlocks.globalBlockedException(ClusterBlocks.java:215) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.cluster.block.ClusterBlocks.globalBlockedRaiseException(ClusterBlocks.java:201) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.search.TransportSearchAction.executeSearch(TransportSearchAction.java:1073) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.search.TransportSearchAction.executeLocalSearch(TransportSearchAction.java:903) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.action.search.TransportSearchAction.lambda$buildRewriteListener$0(TransportSearchAction.java:539) ~[opensearch-3.7.0.jar:3.7.0]",
"at org.opensearch.core.action.ActionListener$1.onResponse(ActionListener.java:82) ~[opensearch-core-3.7.0.jar:3.7.0]",
"... 33 more"] }

How can one reproduce the bug?
Steps to reproduce the behavior:

1. create a single-node cluster with the security plugin (i've used the minimal distribution + security plugin)
2. start the node
3. stop the node
4. start the node again and check the logs

What is the expected behavior?
no such warning in the log (it is not actionable & it is not clear what the problem and the impact are)

What is your host/environment?
OpenSearch 3.7.0 (minimal + security plugin + unrelated plugins)

Do you have any screenshots?
n/a

Do you have any additional context?
code triggering the error:

security/src/main/java/org/opensearch/security/OpenSearchSecurityPlugin.java

Lines 2402 to 2409 in 70e2caf

	if (apiTokenRepository != null) {
	apiTokenRepository.reloadApiTokensFromIndex(
	ActionListener.wrap(
	unused -> log.debug("API tokens loaded on node start"),
	e -> log.warn("Failed to load API tokens on node start", e)
	)
	);
	}

due to the ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state not recovered / initialized];]; in the error message i believe that it might be trying to execute this step too early?

[cwperks]

@rursprung ah it may need a retry loop in case cluster manager is not yet discovered. I'll reply back with example of how security index initialization is done. TY for filing this issue.