[BUG] Error: Failed to populate remote R2 bucket "<bucket>" for binding "NEXT_INC_CACHE_R2_BUCKET": Failed to write "incremental-cache/<cache>/<key>.cache" to R2 after 15 attempts

[grabmateusz]

Describe the bug

Hi,
since yesterday we can't deploy any tenant, all ISR cache populations fail with:

Error: Failed to populate remote R2 bucket "<bucket>" for binding "NEXT_INC_CACHE_R2_BUCKET": Failed to write "incremental-cache/<cache>/<key>.cache" to R2 after 15 attempts

Steps to reproduce

Build project with ISR support, try to deploy it

Expected behavior

ISR population succeeds

@opennextjs/cloudflare version

1.19.10

Wrangler version

4.77.0

next info output

Operating System:
  Platform: darwin
  Arch: arm64
  Version: Darwin Kernel Version 25.3.0: Wed Jan 28 20:53:15 PST 2026; root:xnu-12377.81.4~5/RELEASE_ARM64_T6000
  Available memory (MB): 32768
  Available CPU cores: 10
Binaries:
  Node: 22.21.1
  npm: 10.9.4
  Yarn: 1.22.21
  pnpm: N/A
Relevant Packages:
  next: 16.2.6 // There is a newer version (16.2.9) available, upgrade recommended! 
  eslint-config-next: N/A
  react: 19.2.6
  react-dom: 19.2.6
  typescript: 5.9.3
Next.js Config:
  output: N/A
⚠ There is a newer version (16.2.9) available, upgrade recommended! 
   Please try the latest canary version (`npm install next@canary`) to confirm the issue still exists before creating a new issue.
   Read more - https://nextjs.org/docs/messages/opening-an-issue

Additional context

No response

[maloyuso]

Hi! +1
Since yesterday we started getting Populating remote R2 incremental cache... errors too

[maloyuso]

Hi again! @grabmateusz were you able to find a solution or a temporal workaround for this issue?

[grabmateusz]

Hi @maloyuso, the answer is "yes and no".
We have put on hold PROD deployments till Cloudflare resolves the issue.
However because we have configurable via env vars types of pages we generate statically during build time, we've reduced number of statically generated pages to minimal value and those builds and deployments pass.
However we're waiting for Cloudflare to resolve the problem, we have P1 support ticket opened since 9:37 AM.

[FredKSchott]

Thanks @grabmateusz, we're chasing this down internally. If you find any other relevant details please share.

[FredKSchott]

One potential workaround that would also help us confirm the root cause:

can you retry with lower OpenNext cache population concurrency: --cacheChunkSize 5 or even down to --cacheChunkSize 1 if needed.

[grabmateusz]

Hi @FredKSchott, reducing --cacheChunkSize 5 made single deployment run in isolation pass, however in the logs of such a job I can still see reattempts to upload:

ERROR Attempt 1 to write "incremental-cache/<bucketId>/<itemId>.fetch" failed with a retryable error: Worker returned a 503 Service Temporarily Unavailable response. Retrying...

So there is for sure a regression on the infrastructure which is not handling same traffic to R2 buckets as it used to.

[james-elicx]

Hi @grabmateusz, I spent quite a bit of time digging into what I can see going on from our side over the past few days.

We managed to reproduce this somewhat reliably under a high concurrency.

From what I can tell though, the 503 is a bit of a red herring. In some internal traces that I've been looking at, I can see connections being made with our network, but those requests time out due to no request body being streamed for 15 seconds.

It appears that some requests are just not sending any data with them.

This doesn't feel entirely unusual though when I think about it - it's likely that a high concurrency is leading to available bandwidth, sockets, or something else, being exhausted on the machine that the requests are coming from, and so some requests then struggle to send bytes and get aborted after nothing was sent for 15 seconds.

This could explain the somewhat sporadic nature, where everything seems to work fine, and then you see a few of the 503 errors, and everything works again for a bit more, and so on.

At the moment, the best thing to do may be to reduce concurrency.

We could potentially also look at changing retry behaviour to halt sending new requests for a brief period to attempt to allow things to stabilise before retrying and resuming upload. Does that sound like something you'd be interested in?

[grabmateusz]

Hi @james-elicx,
You've written:

it's likely that a high concurrency is leading to available bandwidth, sockets, or something else, being exhausted on the machine that the requests are coming from

But you have @maloyuso who gave +1, meaning this is not a change on the machines that upload, this is a change on the infrastructure that receives R2 items.
So I don't believe you have at the moment proper conclusion.
Sure we can reduce the concurrency, but even with reduced concurrency we can see errors are logged, while during last 2,5 months all was working great after delivery of #1159.
I would recommend revisiting changes on Cloudflare side that happened last week, as those are very likely source of the problem.

cheers,
Mateusz

[fredriklaason]

No amount of downgrading the package helps

[grabmateusz]

No amount of downgrading the package helps

On our end mitigation of the issue was to reduce concurrency to 5 as per temporary workaround suggested by @FredKSchott, however it can't be long term approach, as our deployment was much longer and still few tenants failed and we had to restart deployments.

One additional information which might be interesting is one of our APAC R2 buckets is populated extremely slow, we're at:

83%|█████████████████████████████████████        | 16898/20133 [54:20:<10:24:, 5.18it/s]

cc: @james-elicx @maloyuso

[james-elicx]

@grabmateusz are you able to share a more precise time frame for when you first started observing these errors? It would be super helpful for trying to narrow down what might have changed if we could get an idea of the last deployment that didn't see these errors, and the first deployment where you saw these errors in your logs (even if it didnt necessarily fail the entire deployment).

Were any changes made to any tooling or dependencies around the time frame as well, or has that all remained unchanged? Just trying to explore any possible connection

[grabmateusz]

Hi @james-elicx, we had successful deployments when we have successfully deployed dozens of OpenNext tenants with thousands of R2 items each on:

• June 16, 2026, at 9:10 AM GMT+2
• June 16, 2026 at 6:45 PM GMT+2
  Those lasted for ~20-30 minutes and did not report any issue. Second one was the last big successful deployment on our end, it used default concurrency.
  Upload was smooth, example from the logs of one of the biggest tenants:

100%|█████████████████████████████████████████████| 19134/19134 [05:49:<00:00:, 54.80it/s]

We hit the wall on deployment executed at:

• June 18, 2026 at 3:55:55 PM GMT+2
  (There is not a single deployment in between that would use huge ISR upload I can use for reference). During this deployment most of our tenants failed.
  @james-elicx I hope this information is what you look for, sorry I won't be able to narrow it down more on our end.

[james-elicx]

@grabmateusz thank you. Have you made any changes to your wrangler or workerd versions in the past 2 weeks? If so, have you tried switching back to the version prior to that change?

In the issue you wrote v4.77.0, although this is lower than what our peer dependencies have wrangler at for the version of opennext you're using, so I'm wondering if there has been any changes there recently?