nginx-ignition configuration completely lost overnight, for the second time

[dpaessler]

I don't know, if this is a bug or even related to nginx-ignition.
Woke up today and found that all my servers were unreachable.
Visiting the nginx-ignition Website, I was prompted to create a user, just like first run wizard. After that and logging in, there neither were my servers nor my certificates. This happened once before, I had to recreate everything to get my servers back online.
As this happened this morning shortly before I had to go to work, I didn't investigate anything, just quickly recreated my servers to get them back online.

I use to manage my docker containers with portainer and they automatically get updated with watchtower, if this is relevant.
Is there any way to debug this behaviour? What could have happened?

[lucasdillmann]

Hi @dpaessler.

Ignition stores all data in the database, everything else is ephemeral and will be (re)generated automatically at runtime if needed (including the Nginx conf files).

Since you are experiencing data loss, you are likely having issues with the database. What is your current database configuration, PostgreSQL or the default SQLite?

If you are using SQLite, I strongly recommend migrating to PostgreSQL. SQLite is intended for quick tests to see if Ignition works for you, but for production environments PostgreSQL is the way to go. What might be happening is that Watchtower is recreating the container, and you are losing the SQLite database file at /opt/nginx-ignition/data/nginx-ignition.db because no volume is mapped.

This documentation file has more details on how to connect Ignition to a PostgreSQL server and should help you with the migration.

Another suggestion is to set up automated, scheduled database backups. If this happens again, you will not have to start from scratch.

[dpaessler]

thanks for the quick reply.
I already use PostgreSQL, will have a look at this.

[dpaessler]

I just added a label to the docker-compose.yml file

labels: - "com.centurylinklabs.watchtower.enable=false"

to prevent watchtower from automatically updating the PostgreSQL container.
I will watch, if this resolves the issue.

[lucasdillmann]

Thanks for the update, @dpaessler. Hope it resolves for you.