Issues with npm datasource updates hitting memory limits on Mend-hosted infrastructure

[jamietanna]

How are you running Renovate?

A Mend.io-hosted app

Which platform you running Renovate on?

GitHub.com

Which version of Renovate are you using?

42.95.2

Please tell us more about your question or problem

We've recently seen an increase in projects using pnpm and yarn hitting memory issues when installing dependencies on a project

• pnpm Renovate stopped working on the open-source software fast-check #40541
• yarn Job fails with kernel-out-of-memory #40352
• yarn: Frequent intermittent kernel out of memory errors (with `npm` datasource) #38880
• pnpm: Renovate started failing with "kernel-out-of-memory" #40940
• pnpm: Increase memory - job fails with kernel-out-of-memory #40935
• pnpm: Renovate-job started to fail with a kernel-out-of-memory Exception #40943
• pnpm: Renovate job fails with kernel-out-of-memory #41020
• pnpm: Increased resources for OSS organization gemini-hlsw #41685

We're in the process of investigating this further, and will share updates in this Discussion.

Related upstream issue: pnpm/pnpm#8441

Logs (if relevant)

No response

[jamietanna]

One attempt at fixing using pnpm configuration: #40541 (reply in thread)

[jamietanna]

#40957 is a suggested change (which we can then use on Mend-hosted)

[ssbarnea]

Is this the magic conversion trick for funnelling more free users into paying customers? -- clearly would not be able to squeeze much from unfunded open source project using it. /sarcasm

Now here is an funny aspect that might have contributed to the OOM issue: the recent new dependency added to the project was renovate npm package itself, which:

• is the only way to install renovate-config-validate (linting config)
• has a huge list of dependencies
• has a huge list of outdated and compromised dependencies
• is updated extremely often (not a problem by itself but if you only need linting config... it becomes one)

I wonder if we can spot a pattern among those affected by this bug.

Considering the number of users reporting this issue very recently, is almost for sure related to increase of memory needs from newer releases. Still, the increase might prove to not be critical and an increase of node memory of 10-20% might address it but sadly I did not see any signs of this being tried on mend.io services.

I seen the need to increase this in the past with tools like webpack.

[jraoult]

Hi @jamietanna, we've been hitting this consistently in our repo for 4 days in a row (therefore blocking any dependency upgrades). Is there an official workaround at the moment?

Happy for you to use our repo if you need a reproduction case.

[jamietanna]

Short answer, no, sorry! We're working on the CLI changes that would allow specifying these environment variables (but a few other things have also popped up needing work on them)

[jamietanna]

A couple of tests I've done so far with (against a few repos who've reported issues):

{
    "toolSettings": {
      "nodeMaxMemory": 1024
    }
  }

seem to be going positively - I'll do a little bit more testing, then we'll have this set across all repos

[Methuselah96]

We had to increase nodeMaxMemory to 1.5GB to fix the out-of-memory errors

[CHC383]

If I understand correctly, toolSettings.nodeMaxMemory is to limit the memory taken by Node, and thus avoid the kernel-out-of-memory issue right?

Unfortunately, setting it to 1024 is not a stable workaround. I still get kernel-out-of-memory from time to time (limit doesn't work or components other than pnpm still take too much memory?). And sometimes even if the run didn't fail with kernel-out-of-memory, pnpm failed with FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory.

Is it really due to pnpm being memory consuming, or is there memory leak somewhere?

Environments:

• Node: v24.14.1
• pnpm: v10.33.0

[Methuselah96]

Never mind, we're still experiencing issues. We've tried setting nodeMaxMemory to 1.0GB, 1.5GB, 2.0GB, and 2.5GB and they all result in a kernel-out-of-memory error.

[jamietanna]

From the debugging I've done, it seems to appear to line up with pnpm / yarn on very large repos - I think if we can get a bug report together for pnpm, they may be able to advise better

[CHC383]

Here is an existing issue on pnpm: pnpm/pnpm#8441

[CHC383]

@jamietanna As mentioned in pnpm/pnpm#8441 (comment), this situation is getting worse on pnpm v11. I wonder if Renovate could expose a way to set PNPM_MAX_WORKERS or PNPM_WORKERS (code). Reducing the number of workers might lower pnpm’s memory usage.

[jamietanna]

Note that I did try that setting a while back (#40541 (reply in thread)) and it didn't seem to change much

If you can share a repository / Job ID for a repository (if you don't want to share repo name publicly) I can set the config for you on your repo, to see if it makes any difference right now

[CHC383]

Here is the job id: d8c69a06-9ac6-4af0-a35d-b965e2825d1b, thanks

[jamietanna]

Actually, re-reviewing Renovate's code, I think we can't actually specify PNPM_MAX_WORKERS as something that goes to the child without explicitly passing it, as it's not marked as basicEnvVars (https://github.com/renovatebot/renovate/blob/HEAD/lib/util/exec/env.ts) or in global:safeEnv to allow setting it in repo config 🤔