Skip to content

Challenge-07-python.md

Latest commit

 

History

History
118 lines (87 loc) · 8.13 KB

Challenge-07-python.md

File metadata and controls

118 lines (87 loc) · 8.13 KB
Raw

Challenge 07 - Python - Tracing Intelligence: Observability in Agentic AI with Microsoft Agent Framework

< Previous Challenge - Home - Next Challenge >

Introduction

When you build AI solutions, you want to be able to observe the behavior of your services. Observability is the ability to monitor and analyze the internal state of components within a distributed system. It is a key requirement for building enterprise-ready AI solutions.

Concepts

Observability is typically achieved through logging, metrics, and tracing. They are often referred to as the three pillars of observability. You will also hear the term "telemetry" used to describe the data collected by these three pillars. Unlike debugging, observability provides an ongoing overview of the system's health and performance.

Microsoft Agent Framework is designed to be observable from the ground up. It emits logs, metrics, and traces that are compatible with the OpenTelemetry standard, providing comprehensive insights into agent behavior and performance.

Observability Features in Microsoft Agent Framework

  • Logging: Microsoft Agent Framework logs meaningful events and errors from agents, tools, and AI connectors. This includes agent lifecycle events, tool executions, and conversation flows.
  • Metrics: Microsoft Agent Framework emits metrics from agent operations and AI connectors. You can monitor metrics such as agent response times, tool execution duration, token consumption, and conversation success rates.
  • Tracing: Microsoft Agent Framework supports distributed tracing with rich context. You can track activities across different agents, tools, and services, providing end-to-end visibility into complex agent workflows.

Observability Across Agents and Workflows

Microsoft Agent Framework enables observability at multiple levels of your agentic AI applications:

Chat Client Observability:

  • User Interaction Tracing: Capture and analyze user messages, commands, and feedback within the chat client.
  • Conversation Flow Monitoring: Observe the sequence of messages exchanged between users and agents, including context switches and interruptions.
  • Latency and Response Metrics: Measure the time taken for agents to respond to user inputs and overall chat responsiveness.
  • Error and Exception Logging: Track errors, failed responses, and unexpected behaviors in the chat interface.
  • User Experience Insights: Collect telemetry on user engagement, satisfaction, and common issues to improve the chat client experience.

Agent-Level Observability:

  • Individual Agent Monitoring: Track the behavior, performance, and resource consumption of each agent in your system.
  • Agent Lifecycle Events: Monitor agent creation, initialization, activation, and termination.
  • Agent-to-Agent Communication: Observe interactions between multiple agents in collaborative scenarios.
  • Agent State Changes: Track state transitions and context preservation across conversations.
  • Tool Usage Patterns: Monitor which tools agents use most frequently and their success rates.

Workflow-Level Observability:

  • Workflow Orchestration: Track the execution flow across multiple agents and services in complex workflows.
  • Step-by-Step Execution: Monitor each step in multi-step workflows, including decision points and branching logic.
  • Cross-Workflow Dependencies: Observe how different workflows interact and depend on each other.
  • Workflow Performance Metrics: Measure end-to-end workflow execution times, success rates, and bottlenecks.
  • Resource Utilization: Monitor compute, memory, and token consumption across entire workflows.

This comprehensive observability approach allows you to understand not just individual component performance, but also the emergent behavior of your entire agentic AI system.

You can use the following observability tools to monitor and analyze the behavior of your agents and workflows built with Microsoft Agent Framework:

Observability Tools

  • Console Exporter: Although the console is not recommended for production, it provides a simple way to get started with observability during development. Microsoft Agent Framework includes built-in console exporters for quick debugging.
  • Application Insights: Application Insights is part of Azure Monitor, which is a comprehensive solution for collecting, analyzing, and acting on telemetry data from your cloud and on-premises environments. Perfect for production agent monitoring.
  • Azure AI Foundry Tracing UI: Azure AI Foundry provides specialized tracing capabilities for AI applications, offering detailed insights into agent conversations, tool usage, and model interactions.
  • OpenTelemetry Compatible Tools: Since Microsoft Agent Framework uses OpenTelemetry standards, you can integrate with any OpenTelemetry-compatible observability platform like Grafana.

Description

You should incorporate observability into your Microsoft Agent Framework application using a Console exporter and one or more of the following approaches to visualize and analyze the telemetry data:

  1. Console Exporter for development and debugging
  2. Application Insights or AI Foundry Tracing for production monitoring

Use the Agent Framework application created in the previous challenge and add comprehensive observability to track agent behavior, tool executions, and conversation flows.

Talk is cheap. Show me.

Here's how to wire up OpenTelemetry in Agent Framework: https://github.com/microsoft/agent-framework/tree/main/python/samples/02-agents/observability

from agent_framework.observability import configure_otel_providers, enable_instrumentation

# Option 1: environment-driven (recommended)
# export ENABLE_INSTRUMENTATION=true
# export ENABLE_CONSOLE_EXPORTERS=true            # optional
# export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
configure_otel_providers()

# Option 2: programmatic
configure_otel_providers(enable_console_exporters=True)
enable_instrumentation(enable_sensitive_data=False)

As always, if you get stuck, see Coach solution for inspiration.

Observability Pipeline

graph TD
    A["Agent Operations"] -->|Instrument| B["OpenTelemetry Instrumentation"]
    B -->|Collect| C["Spans, Metrics, Logs"]
    C -->|Export| D["OTLP Exporter"]
    D -->|Send| E["Backend"]
    E -->|Supports| F["Jaeger"]
    E -->|Supports| G["Application Insights"]
    E -->|Supports| H["Azure Monitor"]
    E -->|Supports| I["Console / Local Dev"]
Loading

Success Criteria

  • ✅ Ensure that your Agent Framework application is running with observability enabled
  • ✅ See the traces generated with Console exporter
  • ✅ Visualize traces using at least one of the recommended tools (Application Insights, AI Foundry Tracing)
  • ✅ Inspect the telemetry data and observe the sequence of agent operations, tool calls, and AI model interactions
  • ✅ Demonstrate that you can see agent conversation history, tool execution details, and performance metrics
  • ✅ Show how observability helps in debugging and monitoring agent behavior in real-time

Learning Resources

Agents Observability