[Enhancement] Not seeing spans for code inside goroutines

[rhyek]

Version of orchestrion v1.3.0

Describe what happened: Serialized db queries using sqlx were generating spans. Once moved to a sync.WaitGroup I no longer see them.

Describe what you expected: I expect to see the concurrent sql queries in DataDog.

Steps to reproduce the issue:

Can see spans:

	for _, queue := range currentQueuesWithoutConsumers {
		slog.Info("found queue without consumers", "queue", queue.Name, "messages", queue.Messages)
		id := uuid.New().String()
		db.MustExec(`
			INSERT INTO "InfraRabbitMqQueueWithoutConsumers" (id, vhost, "queueName", messages)
			VALUES ($1, $2, $3, $4)
		`, id, rmq.vhost, queue.Name, queue.Messages)
		latestRowId[queue.Name] = id
	}

No spans:

	var wg sync.WaitGroup
	for _, queue := range currentQueuesWithoutConsumers {
		wg.Add(1)
		go func(q Queue) {
			defer wg.Done()
			logger.Info("found queue without consumers", "queue", q.Name, "messages", q.Messages)
			id := uuid.New().String()
			db.MustExec(`
				INSERT INTO "InfraRabbitMqQueueWithoutConsumers" (id, vhost, "queueName", messages)
				VALUES ($1, $2, $3, $4)
			`, id, rmq.vhost, q.Name, q.Messages)
			latestRowId[q.Name] = id
		}(queue)
	}
	wg.Wait()

In both cases I am seeing the logs (sent directly to DataDog API using https://github.com/samber/slog-datadog)

Additional environment details (Version of Go, Operating System, etc.):
Go 1.23.4
Linux

[RomainMuller]

Hello!

The issue is most likely that your queries are now executing in new goroutines and the trace context isn't propagated in there at all.

The implicit context propagation via goroutine local storage does not currently pass into child goroutines (it could be done, but it's unclear whether this is desirable in all cases or only certain...). This is one case where you'd need to pass a trace-bearing context.Context value from the parent goroutine to the children...

---

If you have a context.Context value available, you might want to try to use db.MustExecContext instead of db.MustExec so that you can pass the current context through to the children goroutines:

	var wg sync.WaitGroup
	for _, queue := range currentQueuesWithoutConsumers {
		wg.Add(1)
		go func(q Queue) {
			defer wg.Done()
			logger.Info("found queue without consumers", "queue", q.Name, "messages", q.Messages)
			id := uuid.New().String()
			db.MustExecContext(ctx, `
				INSERT INTO "InfraRabbitMqQueueWithoutConsumers" (id, vhost, "queueName", messages)
				VALUES ($1, $2, $3, $4)
			`, id, rmq.vhost, q.Name, q.Messages)
			latestRowId[q.Name] = id
		}(queue)
	}
	wg.Wait()

---

If you don't have a context.Context value on hand (or it is not trace-bearing), then you might need to make one; then use it in the children:

// import tracer "gopkg.in/DataDog/dd-trace-go.v1/ddtrace/tracer"

ctx := context.Backrgound()
// With orchestrion, a span may be found in goroutine-local storage. Register
// it in a [context.Context] object so it can be passed around to the children
// goroutines below.
if span, ok := tracer.SpanFromContext(ctx); ok {
	ctx = tracer.ContextWithSpan(ctx, span)
}
var wg sync.WaitGroup
// ... etc ...

Now - this isn't the prettiest, but would probably get you rolling again. As discussed above, while we probably shouldn't be doing this automatically for all goroutines; we probably could automatically do this dance for you (except via goroutine-local storage propagation) by introducing a new declarative fixture, such as supporting a //dd:inherit-trace-context pragma on the go <call> statements... If you like that idea; I can turn this report into a feature request to DataDog/dd-trace-go?

[rhyek]

Thanks for the response. I'm not sure what the best change for this project would be since I've only started using it and in general I'm not a professional go developer, but I will say that automatically propagating the trace/span context to child goroutines and having the possibility to opt-out (not opt-in) would make the most sense to me in particular for a few reasons:

• This is intuitive behavior from an observability point of view. My understanding is DataDog SDKs will automatically propagate trace context across distributed systems without explicit developer intervention. To me child goroutines should behave the same.
• I am most familiar with Node.js dd-trace SDK and in that case, using Node's AsyncLocalStorage, a trace context is known across a complete async call stack within the program similar to how I expected it to work here.
• I believe in most cases it would make more sense to propagate the trace context vs not
• Having to opt-in with something like //dd:inherit-trace-context would feel a bit out of place given my experience with orchestrion so far where everything works out of the box

Would it make sense to automatically enable this and have the option to opt-out with perhaps //dd:reset-trace-context?

[rhyek]

Using db.MustExecContext worked, btw. Thanks!

[RomainMuller]

I guess the issue with doing it automatically is that any workload that uses (explicitly or not) a pool of goroutines as a worker might see incoherent traces... There are many instances of that such as many HTTP router libraries, etc...

I guess we need to carefully evaluate our options there... Thank you so much for spending the time to share your perspective on that front, this is extremely helpful!

[mtoffl01]

Hi @rhyek,

Just a quick heads-up that we’ve reclassified this from a bug to an enhancement, based on the discussion above. We’re actively exploring potential solutions, though we haven’t settled on one just yet.

Thanks for your patience!