Skip to content

tools.md

Latest commit

 

History

History
118 lines (83 loc) · 3.98 KB

tools.md

File metadata and controls

118 lines (83 loc) · 3.98 KB
Raw

Tools to troubleshoot a running Agent

This page attempts to list useful tools and resources to troubleshoot and profile a running Agent.

pprof

The Agent exposes pprof's HTTP server on port 5000 by default. Through the pprof port you can get profiles (CPU, memory, etc) on the go runtime, along with some general information on the state of the runtime.

General documentation: https://golang.org/pkg/net/http/pprof/

In particular/additionally, the following commands can come handy:

  • List all goroutines:
curl http://localhost:5000/debug/pprof/goroutine?debug=2
  • Profile the go heap:
go tool pprof http://localhost:5000/debug/pprof/heap

expvar

The Agent also exposes expvar variables through an HTTP server on port 5000 by default, in JSON format.

General documentation: https://golang.org/pkg/expvar/

Most components of the Agent expose variables (under their respective key). By default expvar also exposes general memory stats from runtime.Memstats (see the runtime.MemStats docs). In particular, the Sys, HeapSys and HeapInuse variables can be interesting.

Using the jq command-line tool, it's rather easy to explore and find relevant variables, for example:

# Find total bytes of memory obtained from the OS by the go runtime
curl -s http://localhost:5000/debug/vars | jq '.memstats.Sys'
# Get names of checks that the collector's check runner has run
curl -s http://localhost:5000/debug/vars | jq '.runner.Checks | keys'

delve

A debugger for Go.

Project page

Example usage:

$ sudo dlv attach `pgrep -f '/opt/datadog-agent/bin/agent/agent run'`
(dlv) help # help on all commands
(dlv) goroutines # list goroutines
(dlv) threads # list threads
(dlv) goroutine <number> # switch to goroutine

gdb

GDB can in some rare cases be useful to troubleshoot the embedded python interpreter. See https://wiki.python.org/moin/DebuggingWithGdb

Example usage (using the legacy pystack macro):

sudo ./gdb --pid <pid>
info threads
thread <number> # switch to thread
pystack # python stacktrace of current thread

To debug a core dump generated with the c_core_dump Agent option, refer to the GDB docker image that includes the Agent symbols.

For simple debugging cases, you can simply use the python-provided pdb to jump into a debugging shell by adding to the python code that's run:

import pdb
pdb.set_trace()

and running the agent in the foreground.

Debugging on Windows

Prior to 7.23, Agent binaries (Datadog Agent, Process Agent, Trace Agent, etc.) on Windows contain symbol information.

Starting from 7.23, Agent binaries on Windows have debugging information stripped. The original files are packed in a file called debug package.

Prerequisite

To debug Agent process, Golang Runtime, Git and Golang Delve must be installed.

Download the matching debug package. If the MSI file is datadog-agent-7.23.0-x86_64.msi, the debug package should be datadog-agent-7.23.0-x86_64.debug.zip.

Live Debugging

Delve debugger on Windows cannot attach to the service process. The corresponding Windows service must be stopped and disabled.

For pre 7.23, start the Agent executable in the interactive session.

For 7.23 or later version, find the file in the debug package. For agent.exe, the file in debug package is under \src\datadog-agent\src\github.com\DataDog\datadog-agent\bin\agent\agent.exe.debug. You might find the same file under \omnibus-ruby\src\cf-root\bin. Use either one is fine. Copy the file to replace the executable file you want to debug, start the agent executable in the interactive session.

Use dlv attach PID to attach to the running process and start debugging.

Non-live Debugging

Use dlv core DUMPFILE EXEFILE to debug against a dump file.

For 7.23 or newer, the EXEFILE is the .debug file in the debug package.