[BUG]: NullReferenceException in GetBasicRequestArgsForWaf

[dougwaldron]

Tracer Version(s)

3.44.0

Operating system and platform

Windows x64

Instrumentation Mode

msi installer

TFM

net10.0

Bug Report

We have an application that gets constant but sporadic NullReferenceException errors that seem to originate from the Datadog .NET Tracer. There are several web applications running on the same web server -- some .NET Framework, some .NET Core -- but only one application, which is running on .NET 10, has been affected. The error can occur for both GET and POST requests.

When the error occurs, the server returns a 500 internal server error. This is typically visible to the user as a page that doesn't load (blank screen). The user is able to reload the page to continue.

Profiling was enabled by setting the following environment variables in web.config:

<environmentVariable name="CORECLR_ENABLE_PROFILING" value="1" />
<environmentVariable name="DD_DBM_PROPAGATION_MODE" value="full" />
<environmentVariable name="DD_LOGS_INJECTION" value="true" />

When I disable profiling by setting CORECLR_ENABLE_PROFILING to 0, the crashes stop occurring, but of course the app stops being profiled for Datadog.

The stack trace can vary, but I think always includes a call to SecurityCoordinator.GetBasicRequestArgsForWaf(). I've copied a couple examples below.

Object reference not set to an instance of an object.
at Datadog.Trace.AppSec.Coordinator.SecurityCoordinator.GetBasicRequestArgsForWaf() in c:\mnt\tracer\src\Datadog.Trace\AppSec\Coordinator\SecurityCoordinator.Core.cs:119
at Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNetCore.BlockingMiddleware+<Invoke>d__5.MoveNext() in c:\mnt\tracer\src\Datadog.Trace\ClrProfiler\AutoInstrumentation\AspNetCore\BlockingMiddleware.cs:93
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1+<ProcessRequestAsync>d__2.MoveNext()

Object reference not set to an instance of an object.
at Microsoft.AspNetCore.Http.DefaultHttpRequest.get_Cookies()
at Datadog.Trace.Util.RequestDataHelper.GetCookies(HttpRequest request) in c:\mnt\tracer\src\Datadog.Trace\Util\RequestDataHelper.cs:106
at Datadog.Trace.AppSec.Coordinator.SecurityCoordinator.ExtractCookiesFromRequest(HttpRequest request) in c:\mnt\tracer\src\Datadog.Trace\AppSec\Coordinator\SecurityCoordinator.cs:178
at Datadog.Trace.AppSec.Coordinator.SecurityCoordinator.GetBasicRequestArgsForWaf() in c:\mnt\tracer\src\Datadog.Trace\AppSec\Coordinator\SecurityCoordinator.Core.cs:118
at Datadog.Trace.AppSec.Coordinator.SecurityCoordinator.Scan(Boolean lastTime) in c:\mnt\tracer\src\Datadog.Trace\AppSec\Coordinator\SecurityCoordinator.cs:45
at Datadog.Trace.ClrProfiler.AutoInstrumentation.AspNetCore.BlockingMiddleware+<Invoke>d__5.MoveNext() in c:\mnt\tracer\src\Datadog.Trace\ClrProfiler\AutoInstrumentation\AspNetCore\BlockingMiddleware.cs:93
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task, ConfigureAwaitOptions options)
at Microsoft.AspNetCore.Server.IIS.Core.IISHttpContextOfT`1+<ProcessRequestAsync>d__2.MoveNext()

Please let me know if I can provide more detail or if I should open a helpdesk request instead.

Reproduction Code

No response

[dougwaldron]

I should add that the same application (along with all the other applications) is also running on a development server with a similar configuration (as close as possible to production), but is not getting the same errors on dev that we're seeing on prod.

[andrewlock]

Hi @dougwaldron, thanks for flagging this, it definitely looks associated with the AppSec module based on those stacks, but it would be useful if you could open a helpdesk ticket anyway, and reference this issue, so we have a path to collect any extra details etc.

In the meantime, you can likely just set DD_APPSEC_ENABLED=0 (instead of disabling Datadog completely, which is what CORECLR_ENABLE_PROFILING=0 does), if you want to still have traces etc while we figure this out.

[dougwaldron]

Thanks @andrewlock! So far, your fix is working. I'll open a helpdesk ticket, and hopefully that will help you find the issue.

[dougwaldron]

See support request # 2905119