Data race: vertx-http-client context injection can mutate a request's HeadersMultiMap concurrently with Vert.x HTTP/2 header serialization → NullPointerException in HeadersMultiMap.names()

[hshar89]

Describe the bug

he vertx-http-client-4.0 javaagent instrumentation injects trace-context headers into the outbound request by calling HttpClientRequest.putHeader(...) (via HttpRequestHeaderSetter, a TextMapSetter<HttpClientRequest>). That write lands in the request's io.vertx.core.http.impl.headers.HeadersMultiMap.

HeadersMultiMap is not thread-safe — its insertion-order linked list uses plain (non-volatile) before/after pointer fields, has no lock, and its names() walk (reached from size()) has no e != null guard:

// io/vertx/core/http/impl/headers/HeadersMultiMap.java (vertx-core 4.5.22)
public Set<String> names() {
  Set<String> names = new TreeSet<>(String.CASE_INSENSITIVE_ORDER);
  MapEntry e = head.after;
  while (e != head) {                 // no `e != null` check
    names.add(e.getKey().toString()); // line 401 — observed NPE site
    e = e.after;
  }
  return names;
}

On a pooled HTTP/2 connection pinned to one event loop, when a request is written from a thread other than that connection's event loop (or while another write is in progress), Vert.x re-dispatches header serialization onto the connection's event-loop thread:

// io/vertx/core/http/impl/Http2ClientConnection.java — writeHead()
EventLoop eventLoop = ctx.nettyEventLoop();
synchronized (this) {
  if (shouldQueue(eventLoop)) {                             // !inEventLoop() || writeInProgress > 0
    queueForWrite(eventLoop, () -> writeHeaders(request, buf, end, handler));  // -> eventLoop.execute(...)
    return;
  }
}
// ...
// writeHeaders() line ~597:
if (request.headers != null && request.headers.size() > 0) { ... }  // size() -> names() walk

So the header write performed by the instrumentation and the header serialization walk performed by Vert.x can run on different threads with no happens-before edge between them. When they overlap, the walk can observe a node whose after pointer has not yet become visible (it was null at construction and set non-atomically in addBefore), and dereferences null → NPE.

Note: HttpClientRequestImpl.putHeader is synchronized(this), but Http2ClientConnection.writeHeaders reads the raw HeadersMultiMap without holding that monitor, so the synchronized does not mutually exclude the injecting write against the serialization walk.

Observed stack trace (HTTP/2, prod)

java.lang.NullPointerException: Cannot invoke
  "io.vertx.core.http.impl.headers.HeadersMultiMap$MapEntry.getKey()" because "e" is null
    at io.vertx.core.http.impl.headers.HeadersMultiMap.names(HeadersMultiMap.java:401)
    at io.vertx.core.http.impl.headers.HeadersMultiMap.size(HeadersMultiMap.java:96)
    at io.vertx.core.http.impl.Http2ClientConnection$StreamImpl.writeHeaders(Http2ClientConnection.java:597)
    at io.vertx.core.http.impl.Http2ClientConnection$StreamImpl.lambda$writeHead$0(Http2ClientConnection.java:571)
    at io.vertx.core.http.impl.VertxHttp2Stream.lambda$queueForWrite$8(VertxHttp2Stream.java:249)
    at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:141)  // swallowed here
    ...

Because the exception is thrown inside a Netty event-loop task, it is swallowed by AbstractEventExecutor.safeExecute (logged only as WARN "A task raised an exception"). The writeHeaders task dies before encoder().writeHeaders(...), so the HEADERS frame is never handed to the codec — the write-promise never settles and the request silently hangs until its deadline.

Steps to reproduce

We do not yet have a standalone minimal repro; the failure is load- and timing-dependent. It reproduces in production under this configuration:

1. Vert.x HTTP/2 client with a connection pool (connection pinned to an event loop), driven by gRPC-over-Vert.x (vertx-grpc-client), so calls frequently originate on a thread other than the pooled connection's event loop → shouldQueue(...) is true → serialization is re-dispatched via queueForWrite.
2. Javaagent attached with vertx-http-client instrumentation enabled (default), injecting traceparent on each outbound request.
3. Under concurrency, HeadersMultiMap.names() intermittently throws the NPE above.

Setting -Dotel.instrumentation.vertx-http-client.enabled=false removes the second writer and the NPE/hang stops — which is our evidence that the instrumentation's header write is the concurrent writer. (Application, VCL, and gRPC Metadata headers are unaffected because they are copied onto the map once, on the call thread, before the send is queued.)

Expected behavior

The instrumentation should inject context in a way that cannot race with Vert.x's (possibly cross-thread) serialization of the same non-thread-safe HeadersMultiMap — e.g., ensure the header write and the request send/serialization are ordered with a happens-before edge (inject on the connection's event loop, or before the request is eligible to be written), so a single thread owns the map during serialization.

Actual behavior

The instrumentation's putHeader call and Vert.x's writeHeaders serialization walk race on the same HeadersMultiMap with no happens-before edge, yielding an intermittent NullPointerException in HeadersMultiMap.names(). The exception is swallowed by the Netty event loop and the outbound request hangs until deadline.

Javaagent or library instrumentation version

2.21.0-alpha

Environment

JDK: 17
OS: Linux (container)
Vert.x: 4.5.22 (vertx-core, vertx-grpc-client)

Additional context

• Root cause of the null itself: MapEntry.addBefore sets the new node's own after/before (stores A/B) before rewiring existing nodes to point at it (stores C/D). Single-threaded this is safe; across two threads without a happens-before edge, a reader can observe store C (an existing node now points at the new node) before store A (the new node's after = head), reading the construction-time null for after → NPE. Any concurrent writer on these fields makes the unguarded names() walk reach e == null.
• This may be related in theme to StreamResetException in Vert.x HTTP client when LastHttpContent arrives before HttpResponse in Netty pipeline (HttpClientRequestTracingHandler) #18683 (vertx-http-client instrumentation disrupting Vert.x/Netty write state on pooled-connection reuse), though that report is HTTP/1.1 + StreamResetException, a different symptom.
• Happy to provide additional disassembly, thread dumps, or help construct a minimal repro if useful.

Tip

_{React with 👍 to help prioritize this issue. Please use comments to provide useful context, avoiding +1 or me too, to help us triage it. Learn more here.}