[show, clear]: add 'orchagent tasks' commands to surface per-Executor timing + scheduling latency by venkit-nexthop · Pull Request #4536 · sonic-net/sonic-utilities

venkit-nexthop · 2026-05-11T17:20:20Z

What I did

Add two new CLI commands that surface orchagent's per-Executor execution-time
statistics so operators can answer "where is orchagent spending its time?"
without rebuilding it:

show orchagent tasks — per-Executor run-time and scheduling-latency table.
sonic-clear orchagent tasks — reset the counters in place.

This is the CLI half; the orchagent-side instrumentation that publishes the
reply lives in a companion sonic-swss change. The CLI degrades gracefully
(times out with a non-zero exit code) if the daemon side is not present.

How I did it

Two new click groups (show.orchagent.tasks and clear.orchagent.tasks)
wired into show/main.py and clear/main.py.

The CLI talks to orchagent over two APPL_DB notification channels
(ORCH_TASK_STATS_QUERY / ORCH_TASK_STATS_REPLY) and parses the
14-field pipe-separated reply produced by orchagent:

count | total_run_ns
  | median_run_ns | q1_run_ns | q3_run_ns | max_run_ns
  | high_outliers | low_outliers
  | sched_count   | total_sched_ns
  | median_sched_ns | q1_sched_ns | q3_sched_ns | max_sched_ns

Rendering:

RUN TIME — P² median/q1/q3/max of per-invocation wall-clock duration.
Median is the headline, Q1/Q3 give spread, max exposes the worst tail.
RUNS — number of completed invocations.
OUTLIERS — Tukey 1.5×IQR sum (high + low).
SCHED LATENCY — P² median/q1/q3/max of the gap between when the task
finished and when it was next scheduled. Exposes select-loop starvation.
TOTAL — cumulative <run>/<sched> wall-clock spent inside the task vs
waiting before it.

Rows are sorted by total_run_ns descending; ties break by name ascending.
Empty slots (count = 0) print - for quartet/total cells but keep integer
RUNS / OUTLIERS.

sonic-clear orchagent tasks sends op="clear" on the same query channel
and prints OK on a successful round-trip.

doc/Command-Reference.md is updated with a new Orchagent section
(TOC + body) per the contributing guideline for new show / sonic-clear
subcommands.

How to verify it

tests/show_orchagent_tasks_test.py and tests/clear_orchagent_tasks_test.py
mock swsscommon.NotificationProducer / NotificationConsumer / Select
so the round-trip is exercised end-to-end without Redis. Coverage:

Header presence (TASK / RUN TIME / median/q1/q3/max / RUNS / OUTLIERS /
SCHED LATENCY / TOTAL / run/sched / (in msec))
Sort order (total_run_ns descending, name asc tiebreak)
Quartet formatting (ms, two decimals, no per-cell unit suffix)
Outlier counts visible (high + low summed)
TOTAL column = <run>/<sched> with empty-row dashes
Empty-reply case prints headers only
Timeout and orchagent-error reply paths surface as exit-code-1
Correct op string is sent (show / clear)

12 / 12 pytests pass against the mocked swsscommon round-trip.

Manual: with the companion sonic-swss change loaded, run the commands on
a live switch and observe per-Executor stats / reset.

New command output (if the output of a command-line utility has changed)

admin@sonic:~$ show orchagent tasks
TASK         RUN TIME                          RUNS  OUTLIERS  SCHED LATENCY                    TOTAL
             median/q1/q3/max                                  median/q1/q3/max                 run/sched
             (in msec)                                         (in msec)                        (in msec)
ROUTE_TABLE  1745.53/1391.34/2242.07/3913.36     43         5  1.06/0.41/48.40/1436.01          77619.63/5.04

admin@sonic:~$ sonic-clear orchagent tasks
OK

linux-foundation-easycla · 2026-05-11T17:20:29Z

The committers listed above are authorized under a signed CLA.

✅ login: venkit-nexthop / name: Venkit Kasiviswanathan (7019adc)

Surface per-Executor runtime and scheduling-latency statistics from orchagent without rebuilding. Operators can now answer "where is orchagent spending its time?" from the CLI. New commands: - show orchagent tasks -> per-Executor run-time + sched-latency table - sonic-clear orchagent tasks -> reset the counters in place The CLI talks to orchagent over two APPL_DB notification channels (ORCH_TASK_STATS_QUERY / ORCH_TASK_STATS_REPLY) and parses the 14-field pipe-separated reply. Rows are sorted by total run-time descending; the RUN TIME and SCHED LATENCY columns render P^2 median/q1/q3/max quartets in milliseconds, the OUTLIERS column shows Tukey 1.5x IQR sums, and the TOTAL column shows cumulative run/sched wall-clock. Tests mock swsscommon so the round-trip is exercised without Redis. Coverage: header presence, sort order, quartet formatting, outlier counts, empty/zero-slot rendering, timeout/error paths, and that the correct op string is sent. doc/Command-Reference.md is updated with the new Orchagent section. This is the CLI half; the orchagent-side instrumentation that emits the reply lives in a companion sonic-swss change. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>

venkit-nexthop · 2026-05-29T22:25:38Z

@prsunny @prabhataravind @vivekrnv @deepak-singhal0408
Could you take a look and approve this PR if you think it is good.
This PR implements the CLI required for the show orchagent tasks and sonic-clear orchagent tasks
(described in HLD sonic-net/SONiC#2328)

mssonicbld · 2026-07-11T23:32:58Z

/azp run

azure-pipelines · 2026-07-11T23:33:09Z

Azure Pipelines successfully started running 1 pipeline(s).

Collapse alignment padding in show/orchagent.py assignments and the show_orchagent_tasks_test.py assert statements that tripped flake8's E221 (multiple spaces before operator) and E272 (multiple spaces before keyword) checks in the Pretest Static Analysis stage. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>

mssonicbld · 2026-07-11T23:50:24Z

/azp run

azure-pipelines · 2026-07-11T23:50:38Z

Azure Pipelines successfully started running 1 pipeline(s).

venkit-nexthop · 2026-07-12T15:54:01Z

@deepak-singhal0408 @prsunny
CI has passed. Can you take a look and approve the PR?

deepak-singhal0408

Please take care of multi-asic case as well.

Copilot

Pull request overview

This PR adds new SONiC CLI plumbing to query orchagent for per-Executor timing statistics and to reset those counters, enabling operators to see where orchagent spends time without rebuilding the daemon.

Changes:

Add show orchagent tasks to query orchagent via APPL_DB notifications, parse the reply, and render a timing/latency table.
Add sonic-clear orchagent tasks to request clearing the same counters and report success/failure.
Document the new commands and add unit tests that mock the swsscommon notification round-trip.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.

Show a summary per file

File	Description
tests/show_orchagent_tasks_test.py	Adds unit tests covering rendering, sorting, formatting, and error paths for `show orchagent tasks`.
tests/clear_orchagent_tasks_test.py	Adds unit tests for `sonic-clear orchagent tasks` request/timeout/error behavior.
show/orchagent.py	Implements the `show orchagent tasks` click group/command and notification query/parse/render pipeline.
show/main.py	Wires the new `orchagent` show group into the top-level `show` CLI.
clear/orchagent.py	Implements the `sonic-clear orchagent tasks` click group/command and notification clear request.
clear/main.py	Wires the new `orchagent` clear group into the top-level `sonic-clear` CLI.
doc/Command-Reference.md	Documents the new Orchagent section and both commands (usage + example output).

- Make 'show/clear orchagent tasks' namespace-aware: add -n/--namespace, iterate multi_asic.get_namespace_list(), open APPL_DB per-namespace (4-arg DBConnector), and render an ASIC column when multi-ASIC. Fixes the 10s timeout on multi-ASIC (orchagent runs per-ASIC in asicN namespaces); single-ASIC behavior is unchanged and a fabric supervisor becomes a graceful no-op. - _parse_stats: tolerate a trailing '|' and warn to stderr on skipped rows instead of silently producing an incomplete table. - Fix clear test module docstring; add multi-ASIC + malformed-row unit tests. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>

mssonicbld · 2026-07-15T07:41:09Z

/azp run

azure-pipelines · 2026-07-15T07:41:21Z

Azure Pipelines: Successfully started running 1 pipeline(s).

Bring the branch up to date with upstream master. Resolve the orchagent command registration in clear/main.py to keep two blank lines after the newly-added llr_counters function definition (flake8 E305). Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>

mssonicbld · 2026-07-15T08:18:33Z

/azp run

azure-pipelines · 2026-07-15T08:18:45Z

Azure Pipelines: Successfully started running 1 pipeline(s).

venkit-nexthop · 2026-07-15T13:53:59Z

Please take care of multi-asic case as well.

can you take a look now.

…with -n Enhance the Command-Reference orchagent section to cover multi-ASIC: per-ASIC namespaces, the aggregated ASIC column, -n/--namespace usage, and the supervisor no-op, with single- and multi-ASIC examples for both show and clear. Also make 'show orchagent tasks' render the ASIC column whenever the platform is multi-ASIC (via is_multi_asic()) rather than only when more than one namespace is queried, so selecting a single ASIC with -n keeps the owning ASIC explicit and stays consistent with the clear command's per-namespace labeling. Add a unit test for that case. Signed-off-by: Venkit Kasiviswanathan <venkit@nexthop.ai>

mssonicbld · 2026-07-15T18:38:43Z

/azp run

azure-pipelines · 2026-07-15T18:38:55Z

Azure Pipelines: Successfully started running 1 pipeline(s).

venkit-nexthop force-pushed the venkit/show-orchagent-tasks branch from b0342e7 to 7019adc Compare May 11, 2026 17:34

This was referenced May 12, 2026

[orchagent]: Add ZmqRouteServer for concurrent route updates sonic-net/sonic-swss#4564

Open

Performance Regression in Orchagent due to https://github.com/sonic-net/sonic-swss/pull/3910/changes sonic-net/sonic-buildimage#27098

Open

deepak-singhal0408 assigned venkit-nexthop May 14, 2026

deepak-singhal0408 added this to SONiC Routing Dashboard May 14, 2026

github-project-automation Bot moved this to Todo in SONiC Routing Dashboard May 14, 2026

deepak-singhal0408 mentioned this pull request Jul 10, 2026

[golden_config]: enable SWSS ZMQ on T2 and DRH topologies sonic-net/sonic-mgmt#26043

Open

12 tasks

venkit-nexthop mentioned this pull request Jul 11, 2026

orchagent: add per-Executor task timing instrumentation sonic-net/sonic-swss#4563

Open

Merge branch 'sonic-net:master' into venkit/show-orchagent-tasks

614e0b4

deepak-singhal0408 requested a review from Copilot July 13, 2026 06:20

deepak-singhal0408 reviewed Jul 13, 2026

View reviewed changes

Comment thread show/orchagent.py Outdated

Copilot started reviewing on behalf of deepak-singhal0408 July 13, 2026 06:20 View session

deepak-singhal0408 requested changes Jul 13, 2026

View reviewed changes

Copilot AI reviewed Jul 13, 2026

View reviewed changes

Comment thread show/orchagent.py

Comment thread show/orchagent.py Outdated

Comment thread clear/orchagent.py Outdated

Comment thread tests/clear_orchagent_tasks_test.py

venkit-nexthop requested a review from deepak-singhal0408 July 15, 2026 13:55

deepak-singhal0408 reviewed Jul 15, 2026

View reviewed changes

Comment thread doc/Command-Reference.md

deepak-singhal0408 self-requested a review July 15, 2026 18:00

deepak-singhal0408 approved these changes Jul 21, 2026

View reviewed changes

deepak-singhal0408 moved this from Todo to In Progress in SONiC Routing Dashboard Jul 23, 2026

Uh oh!

Conversation

venkit-nexthop commented May 11, 2026

What I did

How I did it

How to verify it

New command output (if the output of a command-line utility has changed)

Uh oh!

linux-foundation-easycla Bot commented May 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

venkit-nexthop commented May 29, 2026

Uh oh!

mssonicbld commented Jul 11, 2026

Uh oh!

azure-pipelines Bot commented Jul 11, 2026

Uh oh!

mssonicbld commented Jul 11, 2026

Uh oh!

azure-pipelines Bot commented Jul 11, 2026

Uh oh!

venkit-nexthop commented Jul 12, 2026

Uh oh!

Uh oh!

deepak-singhal0408 left a comment

Choose a reason for hiding this comment

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mssonicbld commented Jul 15, 2026

Uh oh!

azure-pipelines Bot commented Jul 15, 2026

Uh oh!

mssonicbld commented Jul 15, 2026

Uh oh!

azure-pipelines Bot commented Jul 15, 2026

Uh oh!

venkit-nexthop commented Jul 15, 2026

Uh oh!

Uh oh!

mssonicbld commented Jul 15, 2026

Uh oh!

azure-pipelines Bot commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

linux-foundation-easycla Bot commented May 11, 2026 •

edited

Loading