MariaDB 10.6: can't connect to the database after rebuild from MariaDB 10.4

[benoitchantre]

After a rebuild to switch to MariaDB 10.6 from MaraiDB 10.4, the website can't connect to the database.

When I run lando info, I see that the credentials are correct and that the service is healty.

I got the following error once after a rebuild:
ERROR ==> ERROR 2002 (HY000): Can't connect to local server through socket '/opt/bitnami/mariadb/tmp/mysql.sock

Docker logs

2023-12-18 12:08:43 
2023-12-18 12:08:43 
2023-12-18 12:08:43 
2023-12-18 12:08:43                          STARTING UP                            
2023-12-18 12:08:43 
2023-12-18 12:08:43          ██       █████  ███    ██ ██████   ██████             
2023-12-18 12:08:43          ██      ██   ██ ████   ██ ██   ██ ██    ██            
2023-12-18 12:08:43          ██      ███████ ██ ██  ██ ██   ██ ██    ██            
2023-12-18 12:08:43          ██      ██   ██ ██  ██ ██ ██   ██ ██    ██            
2023-12-18 12:08:43          ███████ ██   ██ ██   ████ ██████   ██████             
2023-12-18 12:08:43 
2023-12-18 12:08:43        The best local development tool in the galaxy!          
2023-12-18 12:08:43 
2023-12-18 12:08:43 
2023-12-18 12:08:43 ===============================================================
2023-12-18 12:08:43 
2023-12-18 12:08:43 
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] InnoDB: Compressed tables use zlib 1.2.11
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] InnoDB: Number of pools: 1
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] mysqld: O_TMPFILE is not supported on /opt/bitnami/mariadb/tmp (disabling future attempts)
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] InnoDB: Using Linux native AIO
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] InnoDB: Initializing buffer pool, total size = 134217728, chunk size = 134217728
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] InnoDB: Completed initialization of buffer pool
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.4.17.
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] InnoDB: Starting shutdown...
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [ERROR] Plugin 'InnoDB' init function returned error.
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] Plugin 'FEEDBACK' is disabled.
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [ERROR] Aborting
2023-12-18 12:08:43 userperms 11:08:43.32 WARN  ==> Only the root user can reset permissions! This is probably ok though...
2023-12-18 12:08:43 loadkeys 11:08:43.43 WARN  ==> Only the root user can load ssh keys! This is probably ok though...
2023-12-18 12:08:43 lando 11:08:43.43 INFO  ==> Lando handing off to: /launch.sh
2023-12-18 12:08:43 lando 11:08:43.44 DEBUG ==> Running command with exec...
2023-12-18 12:08:43 mariadb 11:08:43.51 
2023-12-18 12:08:43 mariadb 11:08:43.52 Welcome to the Bitnami mariadb container
2023-12-18 12:08:43 mariadb 11:08:43.52 Subscribe to project updates by watching https://github.com/bitnami/bitnami-docker-mariadb
2023-12-18 12:08:43 mariadb 11:08:43.53 Submit issues and feature requests at https://github.com/bitnami/bitnami-docker-mariadb/issues
2023-12-18 12:08:43 mariadb 11:08:43.53 
2023-12-18 12:08:43 mariadb 11:08:43.53 INFO  ==> ** Starting MariaDB setup **
2023-12-18 12:08:43 mariadb 11:08:43.63 INFO  ==> Validating settings in MYSQL_*/MARIADB_* env vars
2023-12-18 12:08:43 mariadb 11:08:43.63 WARN  ==> You set the environment variable ALLOW_EMPTY_PASSWORD=yes. For safety reasons, do not use this flag in a production environment.
2023-12-18 12:08:43 mariadb 11:08:43.65 INFO  ==> Initializing mariadb database
2023-12-18 12:08:43 mariadb 11:08:43.66 DEBUG ==> Ensuring expected directories/files exist
2023-12-18 12:08:43 mariadb 11:08:43.72 INFO  ==> Updating 'my.cnf' with custom configuration
2023-12-18 12:08:43 mariadb 11:08:43.74 DEBUG ==> Injecting custom configuration from my_custom.conf
2023-12-18 12:08:43 mariadb 11:08:43.75 INFO  ==> Setting user option
2023-12-18 12:08:43 mariadb 11:08:43.76 DEBUG ==> Setting user to 'wordpress' in mariadb configuration file /opt/bitnami/mariadb/conf/my.cnf
2023-12-18 12:08:43 mariadb 11:08:43.79 INFO  ==> Injecting custom configuration 'my_custom.cnf'
2023-12-18 12:08:43 mariadb 11:08:43.79 INFO  ==> Using persisted data
2023-12-18 12:08:43 mariadb 11:08:43.87 INFO  ==> Running mysql_upgrade
2023-12-18 12:08:43 mariadb 11:08:43.88 INFO  ==> Starting mariadb in background
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [Note] /opt/bitnami/mariadb/sbin/mysqld (server 10.6.5-MariaDB) starting as process 74 ...

I tried the following operations:

1. db-export
2. rebuild
3. db-import

After that, the connection to the database worked again.

Context

• Lando v3.20.8
• Docker v24.0.7
• WordPress recipe with default configuration for MariaDB

[possiBri]

Having this issue today with a second local version of the same Pantheon site (for testing updates in a fully separate copy of the db) after I was able to successfully update my main local version of this same site yesterday. I did get the error yesterday, but after a few restarts (I think? I don't recall exactly what I did since I was trying anything and everything) it somehow worked enough to let me do a fresh lando pull and then everything was good after another lando rebuild. However, this does not now seem to be the case for this second copy of my Pantheon site.

Lando: 3.21-beta.14
Docker: 4.29.0
Drupal 10 through Pantheon recipe

EDIT: I took the nuclear option and did a lando destroy and that seems to have worked to get a lando rebuild to actually re-build and connect to the db service/container. I've also now run lando pull and it completed without issue, and my D10 site shows MariaDB is on v10.6.5.

[reynoldsalec]

Would be nice to isolate whether this problem is coming from an incompatibility between the existing data volume (which was stored assuming MariaDB 10.4) and the new 10.6 service, or if it's some other problem. I did see these two error lines that looked interesting in the original report:

2023-12-18 12:08:44 2023-12-18 11:08:44 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2023-12-18 12:08:44 2023-12-18 11:08:44 0 [ERROR] Aborting

Not sure if they matter but could be a thread to pull on as well.

[reynoldsalec]

...coming back to this, I think this is a standard issue upgrading to Mariadb 10.6, see my comment lando/pantheon#247 (comment) and the linked Mariadb issues there.