[BUG] PHP Memory alloc. error when running with large number of sitemaps

[zegLine]

PHP version

8.3.17

Package version

latest

Application

Docker

Operating system

RHEL 9

Current behavior

When running with a large number of sitemaps (around 300 sitemaps, each with around 5000 URLs ), a PHP memory error occurs:

Parsing sitemaps...

Fatal error: Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) in phar:///cache-warmup.phar/vendor/guzzlehttp/guzzle/src/Handler/CurlHandler.php on line 44)

Expected behavior

No response

Steps to reproduce

No response

Additional context

However, even when running a site with a large number of URLs (around 10000), the error doesn't occur - as long as it is only one sitemap

[eliashaeussler]

Hey @zegLine, thank you for this issue 🙂

Did you read and follow the instructions at https://cache-warmup.dev/faq.html#how-can-i-reduce-memory-consumption-and-cpu-load? If so, I'd need to take a closer look at this area.

[jcvignoli]

I confirm the same is happening to me. Also using php8.3.
No using the command-line but using CacheWarmer class with the following options:

		$crawler_options = new ConcurrentCrawler(
			[
				'concurrency' => $concurrency, 
				'request_options' => [
					'delay' => 500, /
					'timeout' => 1000, 
				],
				'request_headers' => [
					'User-Agent' => 'EliasHaeussler-CacheWarmup/Internal cache warmer ',
				],
			]
		);

$concurrency is 4.