-
Notifications
You must be signed in to change notification settings - Fork 0
Routing robustness: constrain bare 'cost' + add transport signal (v0.6.7) #126
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| # Summary | ||
|
|
||
| Category: network_endpoint_access | ||
| Route: storageops-network-endpoint-access | ||
| Confidence: 0.85 | ||
| Root Cause Type: middlebox_transport_reset | ||
| Evidence Quality: sufficient | ||
| Primary Diagnosis: root_cause_type=middlebox_transport_reset, affected_layer=network | ||
|
|
||
| Large uploads are being reset by something in the network path, not by S3 auth or | ||
| storage. The pattern — large multipart PUTs failing mid-transfer with "connection | ||
| reset by peer" / "broken pipe" while small PUTs and all downloads succeed, starting | ||
| right after a new firewall/NAT appliance was inserted — points to a middlebox | ||
| transport limit: an idle/connection timeout, MTU/MSS mismatch, or NAT | ||
| connection-tracking limit cutting the long-lived large-upload connections. | ||
|
|
||
| # Key Evidence | ||
|
|
||
| - Errors are "Connection reset by peer" (ECONNRESET) and "broken pipe" (EPIPE) | ||
| mid-transfer — transport resets, not 4xx auth responses. | ||
| - Only large/multipart uploads fail; small uploads and all downloads succeed, so the | ||
| break correlates with connection duration/size, not credentials or the bucket. | ||
| - The failures began when a new firewall/NAT appliance entered the path, and the | ||
| same client works from a different network segment — isolating the cause to the | ||
| path, not the endpoint or the application. | ||
|
|
||
| # Remediation | ||
|
|
||
| - Check the middlebox for an idle/connection **timeout** and a NAT | ||
| connection-tracking limit that would drop long-lived uploads; raise the timeout or | ||
| exempt the storage endpoint on the **firewall**. | ||
| - Test for an **MTU**/MSS mismatch (PMTUD black hole): try a lower MTU / enable | ||
| MSS clamping on the appliance, or test with a smaller multipart chunk size to see | ||
| if resets stop. | ||
| - Confirm with a path capture (reset packets / where the RST originates) and by | ||
| retrying from a segment that bypasses the appliance. Do not rotate credentials or | ||
| change the bucket policy — auth is not the cause. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| # Connection-reset / broken-pipe on large uploads (transport) | ||
|
|
||
| A network/transport case. Large multipart PUTs are reset mid-transfer | ||
| ("Connection reset by peer" / "broken pipe") while small PUTs and all downloads | ||
| succeed, right after a new firewall/NAT appliance entered the path — the classic | ||
| signature of an idle-timeout / MTU / connection-tracking limit on a middlebox, not | ||
| an auth or storage problem. | ||
|
|
||
| Expected routing: storageops-network-endpoint-access (transport signal). Expected | ||
| diagnosis: a path/middlebox transport issue (idle timeout, MTU/MSS clamping, or | ||
| NAT conntrack limit) cutting long-lived large-upload connections. Confirms the | ||
| v0.6.7 "transport" routing signal (connection reset / broken pipe / RequestTimeout) | ||
| routes correctly. |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,11 @@ | ||
| { | ||
| "case_type": "diagnosis", | ||
| "expected_category": "network_endpoint_access", | ||
| "expected_root_cause_types": ["middlebox_transport_reset", "connection_reset_path_issue"], | ||
| "expected_min_confidence": 0.8, | ||
| "must_include_evidence_keywords": ["connection reset", "broken pipe", "large"], | ||
| "should_include_evidence_keywords": ["firewall", "small", "downloads"], | ||
| "must_include_recommendation_keywords": ["MTU", "timeout", "firewall"], | ||
| "must_not_include": ["delete bucket", "make the bucket public", "rotate the access key", "disable TLS"], | ||
| "required_report_sections": ["Summary", "Key Evidence", "Remediation"] | ||
| } |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,16 @@ | ||
| Large uploads intermittently fail with "connection reset by peer" / broken pipe. | ||
|
|
||
| Symptoms: | ||
| - PUT of large objects (>500 MB, multipart) fails partway through ~30% of the time. | ||
| - Client errors alternate between "Connection reset by peer (ECONNRESET)" and | ||
| "broken pipe (EPIPE)" mid-transfer. | ||
| - Small objects (<5 MB) upload reliably; downloads of any size are fine. | ||
| - It started after the network team put a new firewall/NAT appliance in the path. | ||
|
|
||
| $ aws s3 cp ./big.tar s3://prod-bucket/big.tar | ||
| upload failed: ./big.tar to s3://prod-bucket/big.tar | ||
| ("Connection reset by peer",) | ||
|
|
||
| Endpoint: s3.us-east-1.amazonaws.com (works from a different network segment) | ||
| The credentials and bucket policy are unchanged and small PUTs succeed, so this is | ||
| not an auth problem. Why do large uploads get reset, and how do we confirm it? |
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -175,6 +175,7 @@ const DOMAIN_SIGNATURES: Record<string, Array<[RegExp, string]>> = { | |
| [/连接(?:失败|超时|被拒)|证书|解析失败|无法访问/i, "network_cjk"], | ||
| [/VPC|endpoint|ENDPOINT/i, "endpoint"], | ||
| [/host\s*unreachable|no\s*route/i, "route"], | ||
| [/RequestTimeout|connection\s*reset|reset\s*by\s*peer|broken\s*pipe|\bECONNRESET\b|\bEPIPE\b|unexpected\s*EOF/i, "transport"], | ||
| ], | ||
| "storageops-cli-sdk-diagnosis": [ | ||
| [/rclone/i, "rclone"], | ||
|
|
@@ -195,7 +196,7 @@ const DOMAIN_SIGNATURES: Record<string, Array<[RegExp, string]>> = { | |
| "storageops-lifecycle-cost": [ | ||
| [/lifecycle/i, "lifecycle"], | ||
| [/Standard_IA|Glacier|Deep_Archive/i, "storage_class"], | ||
| [/cost|费用|计费|账单/i, "cost"], | ||
| [/\bcost\b.*\b(storage|request|egress|transition|tier|bill|lifecycle|retrieval|IA|Glacier|Archive|GB|TB)\b|\b(storage|request|egress|transition|tier|bill|lifecycle|retrieval|IA|Glacier|Archive|GB|TB)\b.*\bcost\b|费用|计费|账单/i, "cost"], | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
The new guard only accepts the exact word Useful? React with 👍 / 👎. |
||
| [/transition|\bexpir(?:e|es|ed|ation|ing)?\b/i, "transition"], | ||
| [/objects.*small|small.*objects/i, "small_objects"], | ||
| ], | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a tool/config parsing error contains
unexpected EOF, this bare transport alternative makes network rank ahead of the CLI/SDK skill because both domains score once and network is earlier inDOMAIN_SIGNATURES; for exampledetectDomain("rclone config failed: unexpected EOF while parsing config file")recommendsstorageops-network-endpoint-accessbeforestorageops-cli-sdk-diagnosis. This sends non-network EOF failures toward endpoint checks unless the pattern is constrained to HTTP/upload/connection context.Useful? React with 👍 / 👎.