Skip to content

Timeout Select errors by up to 1s to avoid callers pinning CPU and filling /var/log#1231

Open
mdhoff-ms wants to merge 4 commits into
masterfrom
dev/matthoffman/throttle-redis-select
Open

Timeout Select errors by up to 1s to avoid callers pinning CPU and filling /var/log#1231
mdhoff-ms wants to merge 4 commits into
masterfrom
dev/matthoffman/throttle-redis-select

Conversation

@mdhoff-ms

@mdhoff-ms mdhoff-ms commented Jul 21, 2026

Copy link
Copy Markdown

I have noticed two instances of filled /var/log on DUTs with many repeated lines:

2026 Jul 21 01:47:00.605758 ERR caclmgrd: :- poll_descriptors: readData error: Unable to read redis reply
2026 Jul 21 01:47:00.605776 ERR healthd: :- poll_descriptors: readData error: Unable to read redis reply
2026 Jul 21 01:47:00.605793 ERR featured: :- poll_descriptors: readData error: Unable to read redis reply

these repeat immediately, leading to /var/log partition being filled very quickly. It seems that the error case of common/select.cpp (e.g. if redis is down) returns immediately, leading to callers who call in loops spinning indefinitely and filling the log partition.

To fix this I propose the following change:
On the Select::ERROR path, we should enforce the timeout requested by the caller (often 1ms, 100ms, etc) up to a cap of 1 second, to avoid pinning the CPU in a hot loop. Additionally the readData error log should be rate limited to once per second, to avoid flooding /var/log/syslog and filling the partition.

Signed-off-by: Matt Hoffman <matthoffman@microsoft.com>
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

@mdhoff-ms
mdhoff-ms requested a review from Copilot July 21, 2026 06:36
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the core swss::Select event loop implementation to throttle repeated Select::ERROR returns, reducing CPU pinning and runaway log growth when underlying selectables (e.g., Redis-backed ones) repeatedly fail to readData().

Changes:

  • Adds a 1-second backoff when Select::select() returns Select::ERROR.
  • Refactors the early-return logic in Select::select() to make return conditions more explicit.
  • Adds <chrono> / <thread> includes to support the new throttling sleep.

Comment thread common/select.cpp Outdated
Comment thread common/select.cpp Outdated
Comment thread common/select.cpp Outdated
Signed-off-by: Matt Hoffman <matthoffman@microsoft.com>
Copilot AI review requested due to automatic review settings July 21, 2026 23:49
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 1 out of 1 changed files in this pull request and generated 1 comment.

Comment thread common/select.cpp Outdated
Copilot AI review requested due to automatic review settings July 22, 2026 06:15
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

Comment thread tests/selectable_priority.cpp
Comment thread tests/selectable_priority.cpp
Signed-off-by: Matt Hoffman <matthoffman@microsoft.com>
@mdhoff-ms
mdhoff-ms force-pushed the dev/matthoffman/throttle-redis-select branch from 111179b to d538530 Compare July 22, 2026 06:20
@mdhoff-ms

Copy link
Copy Markdown
Author

maybe a good idea is to sleep only for the timeout that's passed in, then add a separate backoff (equal to 1s) to the actual logging statement. This would preserve the exact contract the caller is expecting while also mitigating the log spam.

anti-spin timeout capped at 1s. Additional 1s rate limiting for logging.

Signed-off-by: Matt Hoffman <matthoffman@microsoft.com>
Copilot AI review requested due to automatic review settings July 23, 2026 04:58
@mssonicbld

Copy link
Copy Markdown
Collaborator

/azp run

@mdhoff-ms mdhoff-ms changed the title Timeout Select errors by 1s to avoid callers pinning CPU and filling /var/log Timeout Select errors by up to 1s to avoid callers pinning CPU and filling /var/log Jul 23, 2026
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
Successfully started running 1 pipeline(s).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

@mdhoff-ms
mdhoff-ms marked this pull request as ready for review July 23, 2026 23:55
@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
There may be pipelines that require an authorized user to comment /azp run to run.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants