Heap capacity is way too generous than what's actually needed (for Quarkus)

[franz1981]

Since we claimed Quarkus to be very "dense" and memory efficient - i would like this to show up in JVM mode as well.
JVM mode sadly is relying on the usual generational hp and typical batchy/delayed laziness of runtime GCs (serialGC probably is slightly different).

I have verified that Spring and Quarkus, on the tuned profile, in the perf lab, have both some spare heap capacity, by using Flight Recorder
using the default 512 MB and G1:

Quarkus3-jvm

Spring4-jvm

This shows that Quarkus can run quite efficiently young GC pauses, getting back (after GC runs) to ~45MB while Spring ~90MB.

The performance is 13.8K tps for Quarkus and 7.6K tps for Spring.

If we reduce the heap to 128 MB - the numbers clearly change dramatically - given that:

• Spring has a much higher allocation rate
• Spring has less available capacity

I feel bad about it, but - let me show you the numbers first:

• Spring4 jvm drop from 7.6K tps to 734 tps (!!!!!!!!!)
• Quarkus3 jvm drop from 13.8K tps to 10.5K tps

And the GC behaviour, instead...

Quarkus3

Spring4

The statistics helps too, to understand what's going on:

Quarkus3

Spring4

Which shows that Spring is dying due to old-gen pauses while Quarkus is still operating at a decent SLA level.

So, 128m is good and enough w Quarkus, but Spring nope - what should be right to do here?

The purpose of the benchmark is to "stress what the framework does" - at CPU level - but GC is a kind of tax which is known to be paid only if you don't use it well. And currently we give to Quarkus way more capacity than it really needs

[holly-cummins]

I wonder if we could capture some of this scalability-ness in the container/cost calculators, that are still on my todo list. We have to choose a default setting, based on what seems like the best fit for the target scenario. But there is a two-dimensional selection space; if you want to support X transactions per second, you need to allocate Y heap, and thus choose a Z virtual machine. (Or scale horizontally and choose two virtual machines, but that would usually cost more.)

Here's a plot of just the two-dimensional space, with crosses being Quarkus and circles Spring (I didn't bother to place numbers correctly, so I think the scalability curves might have different shapes):

Those curves are interesting, but they're kind of theoretical. But it's a natural mapping from them to a VM size, and then we'd have a plot of cost vs number of transactions, and you can read off the cost for the number of transactions. That's slightly richer (and less arbitrary) than just one cost metric, but without information overload.

[Sanne]

Great experiments @franz1981 , thanks!

And good questions - it's hard to decide what is the fair thing to do. Clearly giving the same amout of memory to both is not fair either.

@holly-cummins yes we need to finally work on the cost aspect - it's definitely the most interesting and possibly, considering such problems, the only way to provide something which approximates a fair, syntethic metric. Consider though that in many cloud providers there rarely is an option for getting exactly the same machine but with a different memory dimension: scaling down the memory implies scaling down the CPU - or - one needs to switch type, which often comes with other architecture / generation changes that complicate the discussion.

Take AWS for example, a popular machine is the "m" series, which starts at:

• 2 vCPUs, 8 GiB Memory (the m5.large)
• 4 vCPUs, 16 GiB Memory (the m5.xlarge)

Another popular type is the "c" series - they follow a similar pattern: doubling the ram doubles the CPUs, and costs scale linearly.

There's a special class of instances which are apparently more interesting:

t3a.micro: 2 vCPUs, 1 GiB ram -> $6.862 (per month, assuming US East region)
t3a.nano: 2 vCPUs, 0.5 GiB ram -> $3.431 (per month, assuming US East region)

What's very interesting of these instances is that while the cost scale linearly and the memory scale linearly, the CPUs seem fixed. Essentially it looks like one could get a 50% discount by reducing memory while not impacting the CPUs available... however there's a trick in the fine print: these are "burstable" machines, they report 2 CPUs but the virtualization will cap execution time, and in practive the larger one has double the credits to consume so it's still linear on both dimensions.

Another problem with this is that the granularity is limited. There's technically no benefit at all in having an app consume only ~300MB vs the other ~400 if the minimum machine class is 512MB. We'll need to show the cost effectiveness of increased density by showcasing multiple containers / JVMs on a larger machine.

So, back to non-burstable machines: there isn't any with less than 2GB of RAM. The most cost effective instance at 2GB is the c6g.medium, offering a single Graviton processor and 2GB or RAM for $24.82 per month. This scales up linearly: a c6g.large would offer two cores, 4GB or RAM, $49.64 per month. (assuming always spot-instances in US East).

And it's a single physical core. I assume we'd want to consider the bare minimum to have two? This brings the minimum level of memory available to 4GB. I guess that's reasonable for Java and for leaving some spare memory to the OS and monitoring instruments.

To show the effective potential cost savings of Quarkus, we need to:

• assume higher density via containers OR a larger, more complex app which actually gets you closer to such limits
• demonstrate that Quarkus performs ~"the same" while halving BOTH: CPU and RAM.