Skip to content

S3 ObjectStoreService hangs indefinitely when HTTPS is attempted against a non-TLS endpoint #28577

@keithrigg

Description

@keithrigg

Bug Description

When N8N_EXTERNAL_STORAGE_S3_HOST points to an S3-compatible service that only speaks plain HTTP (e.g. SeaweedFS) and N8N_EXTERNAL_STORAGE_S3_PROTOCOL is not explicitly set, the default https protocol causes the AWS SDK's TLS handshake to hang indefinitely during checkConnection(). Because objectStoreService.init() is awaited during startup, this blocks all subsequent route registration — the editor UI, API endpoints, and readiness probe all become unreachable, even though the process appears healthy (/healthz returns 200 since it is registered before S3 init).

Root cause: The HeadBucketCommand send in ObjectStoreService.checkConnection() has no timeout. When TCP connects successfully but the TLS handshake never completes (the non-TLS server never sends a ServerHello), the call hangs forever. There is also no startup log showing the resolved endpoint, making it difficult to diagnose.

To Reproduce

  1. Deploy n8n with N8N_DEFAULT_BINARY_DATA_MODE=s3 (or just set N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME to trigger S3 init)
  2. Set N8N_EXTERNAL_STORAGE_S3_HOST to a host running a plain HTTP S3-compatible service (e.g. seaweedfs:8333)
  3. Omit N8N_EXTERNAL_STORAGE_S3_PROTOCOL (defaults to https)
  4. Start n8n — it prints "Initializing n8n process" then hangs indefinitely

Minimal local reproduction: Start a silent TCP listener that accepts connections but never responds, then start n8n with N8N_EXTERNAL_STORAGE_S3_HOST=127.0.0.1:9999 and N8N_EXTERNAL_STORAGE_S3_BUCKET_NAME=test. The process hangs after "Registered runner" — it never prints "Version:" or "Editor is now accessible".

Expected behavior

  1. The connection check should time out (e.g. 10 seconds) and surface a clear error naming the resolved endpoint and suggesting which env vars to check
  2. An info-level log at startup should show the resolved S3 endpoint so operators can immediately verify the configuration
  3. The timeout should be configurable via an env var for environments with higher latency

Debug Info

Self-hosted Kubernetes deployment with queue mode. The issue was discovered when switching binary data storage to SeaweedFS S3 (plain HTTP on port 8333). The n8n main pod reached Running 1/1 but every route except /healthz returned 404, and /healthz/readiness returned 503. Diagnosis required reading the n8n source code inside the running container to understand how the S3 endpoint URL is constructed.

Environment

Operating System

Linux (Kubernetes container) / macOS 15.4 for local reproduction

n8n Version

2.17.0 (also confirmed on 2.16.0)

Node.js Version

22.22.0

Database

PostgreSQL

Execution mode

queue

Hosting

self hosted


We have a tested fix ready on our fork (Inovize-Solutions/n8n@fix/s3-connection-timeout) and would like to open a PR. The changes touch packages/core (object-store service) — filing this issue first per CONTRIBUTING.md guidance to contact the team before changes to packages/core.

Metadata

Metadata

Assignees

No one assigned

    Labels

    status:in-linearIssue or PR is now in Linearstatus:team-assignedA team has been assigned the issue or PRteam:catsIssue is with the Cats team

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions