Reload config performance

[Kegelcizer]

Hi,
I've spun up an instance at work, testing it with 30 hots and a quarter of them have like 20 locations.
The more I add the slower the reload gets, right now it's at about 1.5 - 2 minutes and the UI freezes and I think the action is cancelled most of the times at some point.

It runs on a 2 core 4GB machine while using SQLite and all hosts and locations are using docker integration with swarm mode and service mesh both enabled.

Is SQLite the issue and I should switch to PostgreSQL? If not, could a reload button on each host work to avoid reloading stuff that did not change?

Thank you!

[lucasdillmann]

Hi @Kegelcizer.

I recommend switching to PostgreSQL just because is a production deployment. SQLite is intended for initial tests to see if Ignition works for you. Once you decide to use it for real, a more robust solution (PostgreSQL) is recommended to lower the odds of data loss/corruption. If it helps, there's an example compose here.

Regarding the delays, SQLite could be a contributing factor, but it's hard to know for sure without debugging or profiling.

Another thing you can try is to override the memory limit by setting the GOMEMLIMIT environment variable on the Docker container. By default Nginx Ignition uses 128MiB as the limit, which is fine for the target usage scenario (homelabs) but can be too low for 30+ hosts and SQLite (since SQLite uses a portion of the 128MiB for caching, query results and more). Setting GOMEMLIMIT=256MiB or GOMEMLIMIT=512MiB might help.

During a reload hosts are retrieved into memory to build the config files. With 30+ hosts and 20+ locations each, a lot of memory can be used (specially using SQLite for the reasons mentioned above), leading Ignition to spend significant CPU time trying to free memory that it can't release yet because the reload process is still using it. As a result, the reload becomes very slow and CPU usage spikes.

CPU usage is actually a good indicator to know if can be the memory limit or not: if it goes very high during a reload, it's a strong signal that the memory limit is too low.

[Kegelcizer]

I avoided PostgreSQL so far because we already have a beefy Microsoft SQL server and would rather use that instead as a permanent solution, if support is added.
On Monday I'll monitor duration, CPU, memory, and post the results for these combinations:

• reload
• increase memory limit while keeping SQLite
• use default memory limit with PostgreSQL
• both memory increase and PostgreSQL

Thank you for all the hints.

[Kegelcizer]

Made a reload today, same 1.5-2m duration, CPU at 10% or lower, no major spikes, memory did not exceed 80MB.
I concluded there are no CPU or memory issues, thus did not change the GOMEMLIMIT did not migrate to PostgreSQL.

On the VM, I increased the SSD IOPs and throughput (they were a bit low even for non demanding tasks), and total system RAM. I think it was going to swap before, seems like the only explanation for the slow behavior.
Works great now, almost instant!

[lucasdillmann]

Happy to know it's working as expected now, @Kegelcizer. Thanks for the update.