Many different requests with a same traceId, seems a bug?

[lmssy]

I have a server which receive http request, I use auto instrument to log trace and span and save them in elasticsearch.

Today I check my elasticsearch and found many requests this server received have a same traceId, and these requests come from quite a different time, for example today and yesterday.

As you can see below there is 1080949 requests with same traceId.

instrument version:
io.opentelemetry.javaagent.jetty-8.0, 1.2.0-SNAPSHOT

BTW, where can I found the source code of generating this traceId?

[mateuszrzeszutek]

It's in the IdGenerator class.

Are you certain that this traceId wasn't propagated through the request headers? All those SERVER spans all root spans of their respective traces, or do they have parents?

[jkwatson]

could the traces be being initiated by an android client using an old version of otel with a bad IdGenerator implementation?

[lmssy]

Now I update javaagent to 1.4.1 but problem still there, as you can see below one trace "00ac14092616288451054246389d0001" have 30000+ spans:

All those spans have they parentSpanId, but there is not any parentSpanId can be found as a spanId in this trace, this mean spans are isolated to each other with a same traceId.

All spans are SPAN_KIND_SERVER come from an http api provided by my springboot application, clients are outside applications which have no instrument, so my server is the first jump.

So why tihs siutation happens?

BTW, I have tried to search from the source code to find out when a traceId and spanId is created and set into context, but cannot find any code in opentelemetry-java-instrumentation using RandomIdGenerator to do this

[mateuszrzeszutek]

Is it perhaps connected with the Jetty Continuation issue from this thread: #3904?

[lmssy]

Actually I have tested by send many requests concurrently to a server with Jetty Continuation, and the traceId&spanId looks good, no above problem.
So I'm doing further investigation without any clue.

[muneeb-dar]

Was this issue resolved. I am using istio gateway and for kafka requests I am facing simila issue?

[kyouSS]

I ran into this issue as well.
In my case, the problem was that I never closed the scope of span.makeCurrent(). If you are using Span.makeCurrent(), make sure to close the scope by putting that in an try-with-resource statement, or manually close the scope in a finally block.

[Mityushin]

I encountered a similar situation, for a long time I could not find the reason for the "sticking" of trace_id. Your comment helped me find a similar error in the code. Thank you

[Zerrossetto]

I have the same issue using Spring Boot 3.4.5 / Java 21 with embedded Tomcat. We are using the default OTEL library versions shipped with Spring Boot.

What I noticed is that the same traceId is always assiciated to a given thread name.
For example:

• abb11d879fee13e3403a0b6241630e9b sticks to http-nio-8280-exec-1
• 6323cfc83a0a2cc6340d4c79e6df0ae2 sticks to http-nio-8280-exec-2
• fe4774d9f4aa77bc3bbe53b30db84356 sticks to http-nio-8280-exec-3

And so on.

At least on Tomcat threads are re-used, so maybe the value that it has been returned by ThreadLocalRandom.nextLong() is not globally unique but only unique within a given request, because when generated is not generated anymore.
I have too look better at the OTEL code in combination with Spring Boot, the bug could be related to Spring and not to OTEL itself.

[trask]

hi @Zerrossetto! for us to help you with this, you'll need to open a new issue and add a minimal repro and instructions how to run it and reproduce the results you are seeting, thanks

[Zerrossetto]

Hi @trask. I now came to undestood the root cause of what happened on my application.
Basically , instead of just importing

<dependency>
	<groupId>io.opentelemetry.instrumentation</groupId>
	<artifactId>opentelemetry-spring-boot-starter</artifactId>
</dependency>

due to conflicts with previously existing libraries (i.e. prometheus) we cherrypicked

<dependency>
	<groupId>io.opentelemetry</groupId>
	<artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>

this on the surface made everything work, but actually caused the issue with the threads.

[DziubaR]

Hi all! Is there any update here?