[BUG] Sampling rules not respected when using orchestrion

[giou-k]

Version of orchestrion

Latest - 1.6.1

Describe what happened:
I have the following setup:

1. I am creating a docker image for my app via a Dockerfile.
2. Inside the image, I am using serverless-init for my agent, and in there I have set the sampling rules for my service.
3. I am using orchestrion too, in order to build my app.
4. I have run orchestrion pin and pushed my go mod and orchestrion.tool.go changes to the repo.
5. Inside the code I have added //dd:spans and they seem to work fine. I am getting the traces.

I checked my gcp server via dd_trace_startup_logs and the env var is truly getting defined correctly based on my env var DD_TRACE_SAMPLING_RULES='[{"service": "notification-processor", "sample_rate": 0.1}]'.

Describe what you expected:
I would expect that the sampling will be respected but it doesn't. I tried also using sampling value of Zero, but I was getting traces. Looking at ingestion-control in DD I can see that my app has the Automatic label.
When I deleted the orchestrion from my docker image, and used the classic dd tracer client in code, finally the ingestion-control changed to Configured Local and traces tagged with the ingestion reason rule, respecting my sampling rule rate.
I would expect that the same would be happening when I use orchestrion tool.

Steps to reproduce the issue:

here is my Dockerfile, so that you have a better view:

# Build stage
FROM golang:1.24.8-alpine AS builder
RUN apk update && \
    apk upgrade --no-cache && \
    apk add --no-cache ca-certificates && \
    update-ca-certificates && \
    rm -rf /var/cache/apk/*

#RUN go install github.com/DataDog/orchestrion@v1.6.1

WORKDIR /src
COPY . .

#RUN orchestrion go build -o app cmd/main.go
RUN go build -o app cmd/main.go

# Final stage
FROM alpine:latest
# For Security, Update packages to patch known vulnerabilities in the base image.
# Clean up apt cache to keep the image size down.
RUN apk update && \
    apk upgrade --no-cache && \
    apk add --no-cache ca-certificates && \
    update-ca-certificates

# Copy Go binary
COPY --from=builder /src/app /usr/local/bin/app

# Copy Datadog init
COPY --from=datadog/serverless-init:latest-alpine /datadog-init /usr/local/bin/datadog-init

ENV DD_TRACE_DEBUG=true \
DD_TRACE_SAMPLING_RULES='[{"service": "notification-processor", "sample_rate": 0.1}]'

# Set entrypoint and default command
ENTRYPOINT ["/usr/local/bin/datadog-init"]
CMD ["/usr/local/bin/app"]

EXPOSE 8080

Additional environment details (Version of Go, Operating System, etc.):
go version 1.24.8
personla laptop: macOS
docker on amd64 though
Terraform
Gcp cloud run

[mtoffl01]

Hi @giou-k ,

Can you share your code so that we can troubleshoot further? I tested with a very simple app and found that the sampling rules took effect (I'll share the code below).

Some tips for you:

1. Look out for warnings like this one in your logs after starting the app, which would indicate an issue parsing your sampling rules:

2025/11/11 16:55:52 Datadog Tracer v2.3.1 WARN: DIAGNOSTICS Error(s) parsing sampling rules: found errors: 
	error unmarshalling JSON: invalid character 's' looking for beginning of object key string

2. You can also try adding print statements for the DD_TRACE_SAMPLING_RULES env var, to ensure it's taking effect.
3. Are you sure that the service name you're targeting (notification-processor) is the true service name of your spans? I don't see where DD_SERVICE (or similar) is configured.

My test app:
I set env vars in my terminal:

export DD_SERVICE=my-service
export DD_TRACE_SAMPLING_RULES='[{"service": "my-service", "sample_rate": 0.1}]'
export DD_TRACE_DEBUG=true

App code:

package main

import (
	"fmt"
	"os"
	"time"
)

func main() {
	// print out environment variables to ensure they are set
	fmt.Println("DD_SERVICE:", os.Getenv("DD_SERVICE"))
	fmt.Println("DD_TRACE_SAMPLING_RULES:", os.Getenv("DD_TRACE_SAMPLING_RULES"))
	fmt.Println("DD_TRACE_DEBUG:", os.Getenv("DD_TRACE_DEBUG"))
	trace()
}

//dd:span
func trace() {
	time.Sleep(3 * time.Second)
	fmt.Println("Hello, World!")
}

I ran orchestrion go build . and then orchestrion go run ..

Output:
Startup logs contain: "trace_sampling_rules":[{"service":"my-service","sample_rate":0.1}]
I see the env vars took effect:

DD_SERVICE: my-service
DD_TRACE_SAMPLING_RULES: [{"service": "my-service", "sample_rate": 0.1}]
DD_TRACE_DEBUG: true

And that my span was dropped, as expected (_sampling_priority_v1:-1):

2025/11/11 16:58:00 Datadog Tracer v2.3.1 DEBUG: Finished Span: dd.service=my-service dd.trace_id="6913b16500000000636f9271afb519a3" dd.span_id="7165106549146589603" dd.parent_id="0", Operation: trace, Resource: trace, Tags: map[function-name:trace language:go runtime-id:af9a2f59-29ff-41ff-b970-bc9ce4587803], map[_dd.profiling.enabled:0 _dd.rule_psr:0.1 _dd.top_level:1 _dd.trace_span_attribute_schema:0 _sampling_priority_v1:-1 process_id:73178]

[giou-k]

hello @mtoffl01 and thanks for your support.

I think the main issue is, that I am not sure how to figure out if the tracing sampling rule is working or not, but from your message I understand that the way to figure this out is by checking if span was dropped, based on this: (_sampling_priority_v1:-1), correct?

I was looking into the ingestion-control in datadog site GUI, in order to figure this out and I was expecting that the label would be Configured Local which wasn't in my case.

I have to be sure before I push this to production, cause if the sampling is not set, then we will experience many costs 💸

To clarify some things:

I am also getting from the datadog configuration logs that my sampling is set to my desired value, the thing is that when I was testing with "0" sampling rate, I was still getting traces, thus I thought that there might be a "bug".
f.e here are some logs with rate 0.5, so that was not the issue in theory, the qaing process is what made me think of a bug:
"sample_rate":"NaN","sample_rate_limit":"100","trace_sampling_rules":[{"service":"notification-processor","sample_rate":0.5}],"span_sampling_rules":null,"sampling_rules_error":""

the only warning that I am getting is this one and it has nothing to do with our conversation here in my understanding: 2025/11/13 17:28:57 Datadog Tracer v2.3.1 WARN: Remote config startup error: could not subscribe to at least one remote config product: remote config client not started; %!w(<nil>)
this just says that we don't have set up a remote config for DD.

Bottom line, if in the datadog logs I get that the sampling rule is X, then I can be assured that this will be respected, despite what I see in ingestion-control?

[giou-k]

@mtoffl01 I can see that I was getting this kind of logs, with _sampling_priority_v1:-1, so this means that it was working as expected?
f.e:
2025/11/03 15:42:22 Datadog Tracer v1.74.8 DEBUG: Finished Span: dd.service=notification-processor dd.env=development dd.trace_id="744394473555264152" dd.span_id="744394473555264152" dd.parent_id="0", Operation: test-giou, Resource: test-giou, Tags: map[env:development language:go runtime-id:337b8f2e-96b2-4940-b8eb-10736fd19e97], map[_dd.profiling.enabled:0 _dd.rule_psr:0.1 _dd.top_level:1 _dd.trace_span_attribute_schema:0 _sampling_priority_v1:-1 process_id:7]

[mtoffl01]

Hi @giou-k ,

You are right to be confused about how to confirm the sampling rule took effect! In short, it's complicated, and best done using a combination of methods 😄

1. Ensure the Datadog startup logs display your intended rule under "trace_sampling_rules" (which you've done)
2. Optionally put the tracer in debug mode and observe spans being dropped by your rule with _sampling_priority_v1:-1 (which you've done)
3. Check the Ingestion Controls UI, which should tell you the rules is configured and display the expected sampling rate

If #3 cannot be confirmed, then that is concerning. If the issue persists, I'd suggest you open up a Datadog support ticket so that we can have access to your account during the troubleshooting process (I'd like to see what you saw in the Ingestion Controls UI). If you decide to go this route, please link to this github issue inside the ticket content, so our support team has context.

There are some additional caveats to look out for when it comes to sampling, such as making sure you're targeting a root service -- the service name of the first span in the trace -- when specifying a sampling rule.

[giou-k]

@mtoffl01 thanks for your input. We decided to use the tracing client at the end, so we can be sure on what we are doing.

Just to add, on number 3, that what I was getting with orchestrion was the Automatic label.
Now that I've configured it in my tf files via the tracing client, I get Configured Local label, respecting my exact sampling rate number too.

I am more leaning towards that the sampling rate was working with orchestrion, but I didn't want to risk it, since the thing that was missing from the puzzle was the Ingestion Control label.
Also, it was not clear what _sampling_priority_v1:-1 meant before you explain it to me, maybe you could enrich your docs with that regarding sampling rates.