Add verbose per-step agent execution tracing (#5817)#5820
Open
devin-ai-integration[bot] wants to merge 2 commits into
Open
Add verbose per-step agent execution tracing (#5817)#5820devin-ai-integration[bot] wants to merge 2 commits into
devin-ai-integration[bot] wants to merge 2 commits into
Conversation
- Add crew-level start/finish logs: [crew_name] Starting/Completed - Add per-task start/completion logs: [Agent: X] Starting/Task complete - Add handoff logs between agents: passing to / Received context from - Support custom crew name in logs - Use task.name when available, fallback to description - Add completion/handoff logs for async tasks - Add 7 unit tests covering all new verbose tracing behavior Co-Authored-By: João <joao@crewai.com>
Contributor
Author
|
Prompt hidden (unlisted session) |
Contributor
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughThis PR adds verbose logging to Crew execution: crew start/finish messages, per-task start/completion logs, agent handoff detection via next-agent-role computation, expanded futures metadata for async tasks, and tests validating verbose output. ChangesVerbose Logging for Crew Execution
Sequence Diagram(s)sequenceDiagram
participant Client as Crew.kickoff()
participant Log as _log_* helpers
participant Exec as _execute_tasks / _process_async_tasks
participant Agent as Agent (by role)
Client->>Log: _log_crew_start()
Client->>Exec: execute tasks (verbose, role tracking)
Exec->>Exec: _get_next_agent_role()
Exec->>Agent: run task (agent_role)
Agent-->>Exec: task completes (result)
Exec->>Log: _log_task_completion(role, task, next_agent_role)
Exec->>Log: _log_context_received(role, from_role) when role changed
Exec-->>Client: task results
Client->>Log: _log_crew_finish()
Estimated code review effort🎯 4 (Complex) | ⏱️ ~45 minutes Possibly related issues
Poem
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Comment |
Co-Authored-By: João <joao@crewai.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (2)
lib/crewai/src/crewai/crew.py (2)
1469-1528:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy liftDon't infer handoffs from adjacency alone.
previous_agent_role/_get_next_agent_role()treat neighboring tasks as a real transfer, but async tasks only receivelast_sync_output, and conditional tasks may be skipped entirely. That can producepassing to .../Received context from ...logs for handoffs that never actually happened.
955-973:⚠️ Potential issue | 🟠 Major | 🏗️ Heavy liftMirror the new verbose tracing in
akickoff().This only wires the crew/task completion tracing through the sync path.
Crew.akickoff()still goes through_aexecute_tasks()/_aprocess_async_tasks()without the new crew start/finish or completion/handoff logs, soverbose=Truebehaves differently on the native async API.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/src/crewai/crew.py` around lines 955 - 973, The async kickoff path (akickoff) needs to mirror the synchronous kickoff: call self._log_crew_start() at the start, await the async process/run (use _aexecute_tasks() / _aprocess_async_tasks() or the async equivalents of _run_sequential_process/_run_hierarchical_process), then invoke each after_kickoff_callbacks (await any coroutine callbacks), run self._post_kickoff(result), update self.usage_metrics = self.calculate_usage_metrics(), and finally call self._log_crew_finish(); also ensure any task completion/handoff tracing calls used in the sync flow are invoked in the async flow so verbose=True produces identical trace output.
🧹 Nitpick comments (1)
lib/crewai/tests/test_crew.py (1)
4999-5239: ⚡ Quick winAdd regression coverage for async tracing paths.
These assertions only exercise sync
kickoff()or call helpers directly, so they won't catch false handoffs between consecutive async tasks or the missing completion/handoff logs onakickoff().🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@lib/crewai/tests/test_crew.py` around lines 4999 - 5239, Add async-path tests mirroring the existing verbose sync tests to cover Crew.akickoff(): for cases asserting crew start/finish, per-task start/complete, handoff between agents, no handoff for same agent, crew name/task name logging, patch Task.execute_async with AsyncMock (or patch.object(Task, "execute_async", side_effect=[...]) returning TaskOutput) and run the async flow with pytest.mark.asyncio and await crew.akickoff(), then capture and strip ANSI codes and assert the same log strings (e.g., use Crew.akickoff, Task.execute_async, Crew._log_crew_start/_log_crew_finish, Crew._log_task_start) so false handoffs or missing async logs are detected.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@lib/crewai/src/crewai/crew.py`:
- Around line 955-973: The async kickoff path (akickoff) needs to mirror the
synchronous kickoff: call self._log_crew_start() at the start, await the async
process/run (use _aexecute_tasks() / _aprocess_async_tasks() or the async
equivalents of _run_sequential_process/_run_hierarchical_process), then invoke
each after_kickoff_callbacks (await any coroutine callbacks), run
self._post_kickoff(result), update self.usage_metrics =
self.calculate_usage_metrics(), and finally call self._log_crew_finish(); also
ensure any task completion/handoff tracing calls used in the sync flow are
invoked in the async flow so verbose=True produces identical trace output.
---
Nitpick comments:
In `@lib/crewai/tests/test_crew.py`:
- Around line 4999-5239: Add async-path tests mirroring the existing verbose
sync tests to cover Crew.akickoff(): for cases asserting crew start/finish,
per-task start/complete, handoff between agents, no handoff for same agent, crew
name/task name logging, patch Task.execute_async with AsyncMock (or
patch.object(Task, "execute_async", side_effect=[...]) returning TaskOutput) and
run the async flow with pytest.mark.asyncio and await crew.akickoff(), then
capture and strip ANSI codes and assert the same log strings (e.g., use
Crew.akickoff, Task.execute_async, Crew._log_crew_start/_log_crew_finish,
Crew._log_task_start) so false handoffs or missing async logs are detected.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: d1de7314-a69b-4b36-83a5-88b7c53dc53e
📒 Files selected for processing (2)
lib/crewai/src/crewai/crew.pylib/crewai/tests/test_crew.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the feature requested in #5817: structured per-step verbose logging for crew execution tracing.
When
verbose=Trueis set on a Crew, the following structured logs are now emitted:Changes in
lib/crewai/src/crewai/crew.py:_log_crew_start()/_log_crew_finish()— emit crew-level start/finish logs (usescrew.name, defaults to\"crew\")_log_task_start()— now also emits structured[Agent: X] Starting task: Yverbose log (usestask.nameif set, falls back totask.description)_log_task_completion()— emits[Agent: X] Task complete: Ywith optionalpassing to: Zwhen the next agent differs_log_context_received()— emits[Agent: Y] Received context from Xat handoff points_get_next_agent_role()— helper to look ahead at the next task's agent_execute_tasks()to trackprevious_agent_roleand call the new logging methods_process_async_tasks()to emit completion/handoff logs for async tasks tooagent_roleandnext_agent_roleAll new logs go through the existing
Loggerclass and respect theverboseflag — nothing is printed whenverbose=False.Supersedes #5819 (closed due to repo restructuring).
Review & Testing Checklist for Human
Crew(verbose=True)with a multi-agent sequential workflow and verify structured logs appear in the expected formatCrew(verbose=False)produces no new outputCrewAgentExecutor) still appears alongside the new structured logsNotes
lib/crewai/tests/test_crew.pycovering: crew start/finish logs, task start/completion, handoff logs, no-handoff for same agent, custom crew name, and task name fallbacktest_crew_verbose_outputtest continues to pass unchangedLink to Devin session: https://app.devin.ai/sessions/3893287f119348278ee6998438d33b0f
Summary by CodeRabbit
New Features
Tests