Conversation
*Total -- 2,654.13kb -> 2,214.02kb (16.58%) /web/public/img/github.png -- 13.15kb -> 5.19kb (60.51%) /web/public/img/vodafone.png -- 24.52kb -> 12.65kb (48.41%) /web/public/img/capitalcom.png -- 13.59kb -> 8.12kb (40.23%) /web/public/img/vulture-labs.png -- 61.61kb -> 38.14kb (38.09%) /web/public/blog/banners/enterprise-identity-integration-pilot-protocol.png -- 219.81kb -> 167.40kb (23.84%) /web/public/blog/banners/enterprise-private-networks-roadmap.png -- 289.94kb -> 226.11kb (22.02%) /web/public/blog/banners/zero-dependency-encryption-x25519-aes-gcm.png -- 32.64kb -> 27.19kb (16.7%) /web/public/blog/banners/replace-message-broker-twelve-lines-go.png -- 28.86kb -> 24.08kb (16.57%) /web/public/blog/banners/emergent-trust-networks-agents-choose-peers.png -- 27.48kb -> 22.95kb (16.5%) /web/public/blog/banners/connect-agents-across-aws-gcp-azure-without-vpn.png -- 30.30kb -> 25.34kb (16.36%) /web/public/blog/banners/http-services-over-encrypted-overlay.png -- 28.00kb -> 23.42kb (16.34%) /web/public/blog/banners/why-ai-agents-need-network-stack.png -- 26.67kb -> 22.31kb (16.34%) /web/public/blog/banners/sociology-of-machines-626-agents.png -- 24.84kb -> 20.79kb (16.33%) /web/public/blog/banners/how-pilot-protocol-works.png -- 20.70kb -> 17.36kb (16.13%) /web/public/blog/banners/scaling-openclaw-fleets-thousands-agents.png -- 29.19kb -> 24.49kb (16.11%) /web/public/blog/banners/decentralized-task-marketplace-agents.png -- 27.06kb -> 22.71kb (16.08%) /web/public/blog/banners/lightweight-swarm-communication-drones-robots.png -- 30.41kb -> 25.55kb (15.97%) /web/public/blog/banners/replace-webhooks-with-persistent-agent-tunnels.png -- 28.14kb -> 23.66kb (15.94%) /web/public/blog/banners/nat-traversal-ai-agents-deep-dive.png -- 23.92kb -> 20.11kb (15.91%) /web/public/blog/banners/secure-research-collaboration-share-models-not-data.png -- 29.21kb -> 24.57kb (15.88%) /web/public/blog/banners/pilot-vs-tcp-grpc-nats-comparison.png -- 25.67kb -> 21.60kb (15.86%) /web/public/blog/banners/ietf-internet-draft-pilot-protocol.png -- 73.09kb -> 61.55kb (15.78%) /web/public/blog/banners/peer-to-peer-file-transfer-agents.png -- 23.69kb -> 19.95kb (15.77%) /web/public/blog/banners/polo-score-reputation-without-blockchain.png -- 32.30kb -> 27.23kb (15.68%) /web/public/blog/banners/python-sdk-pilot-protocol.png -- 23.75kb -> 20.03kb (15.68%) /web/public/blog/banners/benchmarking-http-vs-udp-overlay.png -- 29.25kb -> 24.66kb (15.68%) /web/public/blog/banners/secure-ai-agent-communication-zero-trust.png -- 28.46kb -> 24.00kb (15.64%) /web/public/blog/banners/move-beyond-rest-persistent-connections-for-agents.png -- 29.33kb -> 24.75kb (15.62%) /web/public/blog/banners/a2a-agent-cards-over-pilot-tunnels.png -- 28.29kb -> 23.88kb (15.61%) /web/public/blog/banners/clawhub-to-live-network-openclaw-discovery.png -- 28.10kb -> 23.73kb (15.55%) /web/public/blog/banners/hipaa-compliant-agent-communication.png -- 28.62kb -> 24.19kb (15.5%) /web/public/blog/banners/why-autonomous-agents-need-private-discovery.png -- 27.19kb -> 22.98kb (15.49%) /web/public/blog/banners/mcp-plus-pilot-tools-and-network.png -- 27.79kb -> 23.49kb (15.46%) /web/public/blog/banners/build-agent-swarm-self-organizes.png -- 29.63kb -> 25.06kb (15.43%) /web/public/blog/banners/multi-agent-pipelines-openclaw-encrypted-tunnels.png -- 30.30kb -> 25.65kb (15.35%) /web/public/blog/banners/cross-company-agent-collaboration-without-shared-infrastructure.png -- 32.41kb -> 27.44kb (15.34%) /web/public/img/pinterest.png -- 66.67kb -> 56.46kb (15.31%) /web/public/blog/banners/how-626-agents-autonomously-adopted-pilot.png -- 28.48kb -> 24.12kb (15.3%) /web/public/blog/banners/chain-ai-models-across-machines.png -- 27.68kb -> 23.45kb (15.29%) /web/public/blog/banners/claude-agent-teams-over-pilot.png -- 29.14kb -> 24.70kb (15.25%) /web/public/blog/banners/openclaw-task-delegation-polo-reputation.png -- 27.11kb -> 22.99kb (15.2%) /web/public/blog/banners/openclaw-agents-behind-nat-zero-config.png -- 26.10kb -> 22.14kb (15.17%) /web/public/blog/banners/federated-learning-p2p-communication.png -- 25.82kb -> 21.91kb (15.15%) /web/public/blog/banners/run-agent-network-without-cloud-dependency.png -- 27.27kb -> 23.15kb (15.1%) /web/public/blog/banners/openclaw-meets-pilot-agent-networking-one-command.png -- 30.31kb -> 25.74kb (15.08%) /web/public/blog/banners/build-openclaw-agent-self-organizes-pilot.png -- 28.19kb -> 23.95kb (15.04%) /web/public/blog/banners/build-ai-agent-marketplace-discovery-reputation.png -- 30.98kb -> 26.32kb (15.03%) /web/public/blog/banners/how-ai-agents-discover-each-other.png -- 26.74kb -> 22.72kb (15.03%) /web/public/blog/banners/preferential-attachment-ai-networks-trust-graph.png -- 27.42kb -> 23.32kb (14.96%) /web/public/blog/banners/contributing-codebase-tour.png -- 27.64kb -> 23.53kb (14.86%) /web/public/blog/banners/build-multi-agent-network-five-minutes.png -- 23.70kb -> 20.21kb (14.72%) /web/public/blog/banners/connect-ai-agents-behind-nat-without-vpn.png -- 24.63kb -> 21.03kb (14.62%) /web/public/blog/banners/distributed-rag-without-central-knowledge-base.png -- 27.78kb -> 23.72kb (14.62%) /web/public/blog/banners/trust-model-agents-invisible-by-default.png -- 30.90kb -> 26.39kb (14.59%) /web/public/blog/banners/smart-home-without-cloud-local-device-communication.png -- 29.62kb -> 25.30kb (14.59%) /web/public/blog/banners/private-agent-network-company.png -- 27.31kb -> 23.35kb (14.48%) /web/public/blog/banners/distributed-monitoring-without-prometheus.png -- 27.60kb -> 23.64kb (14.36%) /web/public/blog/banners/building-custom-pilot-skills-openclaw.png -- 24.60kb -> 21.07kb (14.35%) /docs/media/pilot-demo.gif -- 492.59kb -> 454.87kb (7.66%) /web/public/img/tencent.svg -- 7.95kb -> 7.60kb (4.3%) Signed-off-by: ImgBotApp <ImgBotHelp@gmail.com>
TeoSlayer
pushed a commit
that referenced
this pull request
Apr 30, 2026
*Total -- 2,654.13kb -> 2,214.02kb (16.58%) /web/public/img/github.png -- 13.15kb -> 5.19kb (60.51%) /web/public/img/vodafone.png -- 24.52kb -> 12.65kb (48.41%) /web/public/img/capitalcom.png -- 13.59kb -> 8.12kb (40.23%) /web/public/img/vulture-labs.png -- 61.61kb -> 38.14kb (38.09%) /web/public/blog/banners/enterprise-identity-integration-pilot-protocol.png -- 219.81kb -> 167.40kb (23.84%) /web/public/blog/banners/enterprise-private-networks-roadmap.png -- 289.94kb -> 226.11kb (22.02%) /web/public/blog/banners/zero-dependency-encryption-x25519-aes-gcm.png -- 32.64kb -> 27.19kb (16.7%) /web/public/blog/banners/replace-message-broker-twelve-lines-go.png -- 28.86kb -> 24.08kb (16.57%) /web/public/blog/banners/emergent-trust-networks-agents-choose-peers.png -- 27.48kb -> 22.95kb (16.5%) /web/public/blog/banners/connect-agents-across-aws-gcp-azure-without-vpn.png -- 30.30kb -> 25.34kb (16.36%) /web/public/blog/banners/http-services-over-encrypted-overlay.png -- 28.00kb -> 23.42kb (16.34%) /web/public/blog/banners/why-ai-agents-need-network-stack.png -- 26.67kb -> 22.31kb (16.34%) /web/public/blog/banners/sociology-of-machines-626-agents.png -- 24.84kb -> 20.79kb (16.33%) /web/public/blog/banners/how-pilot-protocol-works.png -- 20.70kb -> 17.36kb (16.13%) /web/public/blog/banners/scaling-openclaw-fleets-thousands-agents.png -- 29.19kb -> 24.49kb (16.11%) /web/public/blog/banners/decentralized-task-marketplace-agents.png -- 27.06kb -> 22.71kb (16.08%) /web/public/blog/banners/lightweight-swarm-communication-drones-robots.png -- 30.41kb -> 25.55kb (15.97%) /web/public/blog/banners/replace-webhooks-with-persistent-agent-tunnels.png -- 28.14kb -> 23.66kb (15.94%) /web/public/blog/banners/nat-traversal-ai-agents-deep-dive.png -- 23.92kb -> 20.11kb (15.91%) /web/public/blog/banners/secure-research-collaboration-share-models-not-data.png -- 29.21kb -> 24.57kb (15.88%) /web/public/blog/banners/pilot-vs-tcp-grpc-nats-comparison.png -- 25.67kb -> 21.60kb (15.86%) /web/public/blog/banners/ietf-internet-draft-pilot-protocol.png -- 73.09kb -> 61.55kb (15.78%) /web/public/blog/banners/peer-to-peer-file-transfer-agents.png -- 23.69kb -> 19.95kb (15.77%) /web/public/blog/banners/polo-score-reputation-without-blockchain.png -- 32.30kb -> 27.23kb (15.68%) /web/public/blog/banners/python-sdk-pilot-protocol.png -- 23.75kb -> 20.03kb (15.68%) /web/public/blog/banners/benchmarking-http-vs-udp-overlay.png -- 29.25kb -> 24.66kb (15.68%) /web/public/blog/banners/secure-ai-agent-communication-zero-trust.png -- 28.46kb -> 24.00kb (15.64%) /web/public/blog/banners/move-beyond-rest-persistent-connections-for-agents.png -- 29.33kb -> 24.75kb (15.62%) /web/public/blog/banners/a2a-agent-cards-over-pilot-tunnels.png -- 28.29kb -> 23.88kb (15.61%) /web/public/blog/banners/clawhub-to-live-network-openclaw-discovery.png -- 28.10kb -> 23.73kb (15.55%) /web/public/blog/banners/hipaa-compliant-agent-communication.png -- 28.62kb -> 24.19kb (15.5%) /web/public/blog/banners/why-autonomous-agents-need-private-discovery.png -- 27.19kb -> 22.98kb (15.49%) /web/public/blog/banners/mcp-plus-pilot-tools-and-network.png -- 27.79kb -> 23.49kb (15.46%) /web/public/blog/banners/build-agent-swarm-self-organizes.png -- 29.63kb -> 25.06kb (15.43%) /web/public/blog/banners/multi-agent-pipelines-openclaw-encrypted-tunnels.png -- 30.30kb -> 25.65kb (15.35%) /web/public/blog/banners/cross-company-agent-collaboration-without-shared-infrastructure.png -- 32.41kb -> 27.44kb (15.34%) /web/public/img/pinterest.png -- 66.67kb -> 56.46kb (15.31%) /web/public/blog/banners/how-626-agents-autonomously-adopted-pilot.png -- 28.48kb -> 24.12kb (15.3%) /web/public/blog/banners/chain-ai-models-across-machines.png -- 27.68kb -> 23.45kb (15.29%) /web/public/blog/banners/claude-agent-teams-over-pilot.png -- 29.14kb -> 24.70kb (15.25%) /web/public/blog/banners/openclaw-task-delegation-polo-reputation.png -- 27.11kb -> 22.99kb (15.2%) /web/public/blog/banners/openclaw-agents-behind-nat-zero-config.png -- 26.10kb -> 22.14kb (15.17%) /web/public/blog/banners/federated-learning-p2p-communication.png -- 25.82kb -> 21.91kb (15.15%) /web/public/blog/banners/run-agent-network-without-cloud-dependency.png -- 27.27kb -> 23.15kb (15.1%) /web/public/blog/banners/openclaw-meets-pilot-agent-networking-one-command.png -- 30.31kb -> 25.74kb (15.08%) /web/public/blog/banners/build-openclaw-agent-self-organizes-pilot.png -- 28.19kb -> 23.95kb (15.04%) /web/public/blog/banners/build-ai-agent-marketplace-discovery-reputation.png -- 30.98kb -> 26.32kb (15.03%) /web/public/blog/banners/how-ai-agents-discover-each-other.png -- 26.74kb -> 22.72kb (15.03%) /web/public/blog/banners/preferential-attachment-ai-networks-trust-graph.png -- 27.42kb -> 23.32kb (14.96%) /web/public/blog/banners/contributing-codebase-tour.png -- 27.64kb -> 23.53kb (14.86%) /web/public/blog/banners/build-multi-agent-network-five-minutes.png -- 23.70kb -> 20.21kb (14.72%) /web/public/blog/banners/connect-ai-agents-behind-nat-without-vpn.png -- 24.63kb -> 21.03kb (14.62%) /web/public/blog/banners/distributed-rag-without-central-knowledge-base.png -- 27.78kb -> 23.72kb (14.62%) /web/public/blog/banners/trust-model-agents-invisible-by-default.png -- 30.90kb -> 26.39kb (14.59%) /web/public/blog/banners/smart-home-without-cloud-local-device-communication.png -- 29.62kb -> 25.30kb (14.59%) /web/public/blog/banners/private-agent-network-company.png -- 27.31kb -> 23.35kb (14.48%) /web/public/blog/banners/distributed-monitoring-without-prometheus.png -- 27.60kb -> 23.64kb (14.36%) /web/public/blog/banners/building-custom-pilot-skills-openclaw.png -- 24.60kb -> 21.07kb (14.35%) /docs/media/pilot-demo.gif -- 492.59kb -> 454.87kb (7.66%) /web/public/img/tencent.svg -- 7.95kb -> 7.60kb (4.3%) Signed-off-by: ImgBotApp <ImgBotHelp@gmail.com> Co-authored-by: ImgBotApp <ImgBotHelp@gmail.com>
TeoSlayer
pushed a commit
that referenced
this pull request
May 3, 2026
Symptom: a webhook URL that is consistently returning 5xx (or
timing out) gets the FULL retry budget (webhookMaxRetries=3
attempts with exponential backoff) per event, forever. With the
1024-slot queue + 1 s initial backoff, each dead-server attempt
blocks the dispatcher goroutine for ~3 s; ~340 events fill the
queue, then events drop silently. Meanwhile we have made ~1024 × 3
= 3072 HTTP attempts against a known-dead URL — wasting CPU,
outbound bandwidth, and (if the receiver is on the same network)
the receiver's incoming connection budget.
The dispatcher is single-goroutine: post() blocks on each event's
retry loop. A dead URL stalls the queue head-of-line, so HEALTHY
events (e.g., the operator finally fixes the receiver and a
recovery event fires) sit behind dead-URL retries.
Lands in this RED commit:
- consecutiveFailures atomic.Uint32 (currently never updated)
- circuitOpenUntilNano atomic.Int64 (currently always 0)
- circuitSkips atomic.Uint64 (currently never incremented)
- circuitCooldown time.Duration field (default 30s, tests can
override via WithCircuitCooldown)
- WithCircuitCooldown(d) option for tests
- CircuitSkips() accessor on WebhookClient
post() body unchanged — the breaker logic lands in GREEN.
Test (webhook_circuit_breaker_bug_test.go):
- 500-returning httptest.Server
- Fire 20 events with 1ms backoff + 50ms timeout
- assert: server got ~60 POSTs (20 × webhookMaxRetries)
- assert: CircuitSkips=0 (no breaker active)
GREEN flips: server gets ~15 POSTs (5 to open circuit + later events
short-circuited), CircuitSkips counts the rest.
TeoSlayer
pushed a commit
that referenced
this pull request
May 3, 2026
Bug fixed: WebhookClient.post() now opens a circuit breaker after
webhookCircuitOpenThreshold (5) consecutive total failures and
short-circuits every Emit during the cooldown window
(circuitCooldown, default 30s). On the first event after cooldown
elapses, post() probes — success closes the breaker
(consecutiveFailures resets to 0), failure reopens for another
cooldown.
Curbs the dead-URL retry storm:
- Before: 1024 events × 3 attempts = 3072 HTTP attempts to a
dead URL before the dispatcher gives up (queue fills, then
silent drop)
- After: 5 events × 3 attempts = 15 attempts to detect the
breaker should open; subsequent events skip immediately,
accruing CircuitSkips for operator visibility
Implementation (pkg/daemon/webhook.go::post):
// At start: short-circuit if breaker is open.
if openUntil := wc.circuitOpenUntilNano.Load(); openUntil > 0 {
now := time.Now().UnixNano()
if now < openUntil {
wc.circuitSkips.Add(1)
return
}
wc.circuitOpenUntilNano.Store(0) // probe attempt
}
// ... existing retry loop, tracking success/clientErr ...
if success { wc.consecutiveFailures.Store(0); return }
if clientErr { return } // 4xx doesn't trip breaker
failures := wc.consecutiveFailures.Add(1)
if failures >= webhookCircuitOpenThreshold {
wc.circuitOpenUntilNano.Store(time.Now().Add(cooldown).UnixNano())
}
Why 4xx doesn't trip the breaker: 4xx means the URL is reachable
and the receiver is healthy — the issue is the payload, which
waiting won't fix. Tripping the breaker on 4xx would suppress
unrelated future events.
Tests:
- TestWebhookHasNoCircuitBreakerForDeadURL: 20 events to 500-server
→ server gets <= ~18 POSTs (down from 60); CircuitSkips > 0.
- TestWebhookCircuitClosesOnSuccessAfterCooldown: open the breaker
by failing 5+ events, recover the receiver, wait past cooldown,
emit one event → breaker closes (consecutiveFailures=0,
circuitOpenUntilNano=0).
Race-clean across full pkg/daemon (62s). 18 commits ahead of v1.9.0,
11 bug categories closed.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Beep boop. Your images are optimized!
Your image file size has been reduced by 17% 🎉
Details
📝 docs |
repo | 🙋🏾 issues | 🏪 marketplace
~Imgbot - Part of Optimole family