Failed to update status: status.roles.reader: Required value, status.roles.writer: Required value

[IngwiePhoenix]

Description

I wanted to add a new DB and user to setup Zitadel in my cluster and was a little perplexed when I ran into this:

{
  "level": "error",
  "ts": "2025-08-22T02:56:40Z",
  "msg": "unable to update status",
  "controller": "postgresqldatabase",
  "controllerKind": "PostgresqlDatabase",
  "controllerGroup": "postgresql.easymile.com",
  "Request.Namespace": "auth-system",
  "Request.Name": "zitadel-db",
  "error": "PostgresqlDatabase.postgresql.easymile.com \"zitadel-db\" is invalid: [status.roles.reader: Required value, status.roles.writer: Required value]",
  "stacktrace": "github.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).manageError\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:898\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).Reconcile\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"
}

I made sure the Helm chart was updated (went from 1.8.0 to 1.8.2) and then let it work for a bit but the error returned. So I restarted the entire node, but the error persists.

Expected Behavior

I expected to create a new database and user and save all the credentials in a secret:

apiVersion: postgresql.easymile.com/v1alpha1
kind: PostgresqlDatabase
metadata:
  name: zitadel-db
  namespace: auth-system
spec:
  engineConfiguration:
    name: default-cluster-instance
    namespace: postgres
  database: zitadel
  masterRole: "zitadel-role"
  dropOnDelete: true
  waitLinkedResourcesDeletion: true
---
apiVersion: postgresql.easymile.com/v1alpha1
kind: PostgresqlUserRole
metadata:
  name: zitadel-db-user
  namespace: auth-system
spec:
  mode: MANAGED
  rolePrefix: "zitadel"
  userPasswordRotationDuration: 720h
  privileges:
    - privilege: OWNER
      database:
        name: zitadel-db
      generatedSecretName: zitadel-db-creds

Actual Behavior

Instead, the above error now keeps showing up during each reconcile.

Environment

• Kubernetes version: 1.33.3
• Project Version/Tag: Chart version 1.8.2, container is thus 3.4.2

Steps to reproduce

I created my cluster like this using CNPG:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: default-cluster
  namespace: postgres
spec:
  instances: 1
  enableSuperuserAccess: true
  storage:
    storageClass: "nfs-bunker"
    size: 10Gi
  walStorage:
    storageClass: "local-path"
    size: 5Gi

...and then bound the operator to it:

apiVersion: postgresql.easymile.com/v1alpha1
kind: PostgresqlEngineConfiguration
metadata:
  name: default-cluster-instance
  namespace: postgres
spec:
  host: default-cluster-rw.postgres.svc.kube.birb.it
  secretName: default-cluster-superuser
  checkInterval: 30s
  waitLinkedResourcesDeletion: true

and have since just been adding users. However, I have had one WAL crash (because the entire node imploded from an update gone wrong) and haven't added any new users or databases in quite a while. I also made sure the CRDs were up to date as well.

Have I missed something perhaps...?

Thank you and kind regards,
Ingwie

[IngwiePhoenix]

Here's some filtered logs:

# kubectl logs -f -n postgres deployment.apps/postgres-operator-postgresql-operator --since=1m | grep zitadel
{"level":"info","ts":"2025-08-22T03:18:45Z","msg":"Reconciling PostgresqlUserRole","controller":"postgresqluserrole","controllerKind":"PostgresqlUserRole","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db-user"}
{"level":"info","ts":"2025-08-22T03:18:45Z","msg":"PostgresqlDatabase not ready, waiting for it","controller":"postgresqluserrole","controllerKind":"PostgresqlUserRole","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitade-db-user"}
{"level":"info","ts":"2025-08-22T03:19:04Z","msg":"Reconciling PostgresqlDatabase","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db"}
{"level":"error","ts":"2025-08-22T03:19:14Z","msg":"issue raised in reconcile","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db","error":"context deadline exceeded","stacktrace":"github.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).manageError\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:883\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).Reconcile\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2025-08-22T03:19:14Z","msg":"unable to update status","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db","error":"PostgresqlDatabase.postgresql.easymile.com \"zitadel-db\" is invalid: [status.roles.reader: Required value, status.roles.writer: Required value]","stacktrace":"github.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).manageError\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:898\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).Reconcile\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2025-08-22T03:19:14Z","msg":"Reconciler error","controller":"postgresqldatabase","controllerGroup":"postgresql.easymile.com","controllerKind":"PostgresqlDatabase","PostgresqlDatabase":{"name":"zitadel-db","namespace":"auth-system"},"namespace":"auth-system","name":"zitadel-db","reconcileID":"d54c3a4d-bc0c-4cc0-9623-e692acc42c56","error":"context deadline exceeded","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2025-08-22T03:19:20Z","msg":"issue raised in reconcile","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db","error":"Internal error occurred: pq: canceling statement due to user request","stacktrace":"github.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).manageError\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:883\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).mainReconcile\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:244\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).Reconcile.func1\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:110"}
{"level":"error","ts":"2025-08-22T03:19:20Z","msg":"unable to update status","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db","error":"client rate limiter Wait returned an error: context deadline exceeded","stacktrace":"github.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).manageError\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:898\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).mainReconcile\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:244\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).Reconcile.func1\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:110"}
{"level":"info","ts":"2025-08-22T03:19:43Z","msg":"Reconciling PostgresqlUserRole","controller":"postgresqluserrole","controllerKind":"PostgresqlUserRole","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db-user"}
{"level":"info","ts":"2025-08-22T03:19:43Z","msg":"PostgresqlDatabase not ready, waiting for it","controller":"postgresqluserrole","controllerKind":"PostgresqlUserRole","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitade-db-user"}
{"level":"info","ts":"2025-08-22T03:20:03Z","msg":"Reconciling PostgresqlDatabase","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db"}
{"level":"error","ts":"2025-08-22T03:20:13Z","msg":"issue raised in reconcile","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db","error":"context deadline exceeded","stacktrace":"github.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).manageError\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:883\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).Reconcile\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2025-08-22T03:20:13Z","msg":"unable to update status","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db","error":"PostgresqlDatabase.postgresql.easymile.com \"zitadel-db\" is invalid: [status.roles.reader: Required value, status.roles.writer: Required value]","stacktrace":"github.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).manageError\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:898\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).Reconcile\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:124\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:118\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:314\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2025-08-22T03:20:13Z","msg":"Reconciler error","controller":"postgresqldatabase","controllerGroup":"postgresql.easymile.com","controllerKind":"PostgresqlDatabase","PostgresqlDatabase":{"name":"zitadel-db","namespace":"auth-system"},"namespace":"auth-system","name":"zitadel-db","reconcileID":"ffb2d281-1ee0-471d-9b85-e3a22069d6f1","error":"context deadline exceeded","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:324\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:265\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/internal/controller/controller.go:226"}
{"level":"error","ts":"2025-08-22T03:20:20Z","msg":"issue raised in reconcile","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db","error":"Internal error occurred: pq: canceling statement due to user request","stacktrace":"github.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).manageError\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:883\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).mainReconcile\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:244\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).Reconcile.func1\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:110"}
{"level":"error","ts":"2025-08-22T03:20:20Z","msg":"unable to update status","controller":"postgresqldatabase","controllerKind":"PostgresqlDatabase","controllerGroup":"postgresql.easymile.com","Request.Namespace":"auth-system","Request.Name":"zitadel-db","error":"client rate limiter Wait returned an error: context deadline exceeded","stacktrace":"github.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).manageError\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:898\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).mainReconcile\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:244\ngithub.com/easymile/postgresql-operator/internal/controller/postgresql.(*PostgresqlDatabaseReconciler).Reconcile.func1\n\t/workspace/internal/controller/postgresql/postgresqldatabase_controller.go:110"}

[IngwiePhoenix]

It... eventually got generated. Is my Postgres that slow?

I see something about a deadline exceeding - probably in a context.Context. Can I adjust the timeout there?

[oxyno-zeta]

Hello @IngwiePhoenix ,

Thanks for the logs and all context !

The issue here is that there is a timeout between operator operations and your PG engine (the deadline exceeding log).
The second issue is a code issue on operator side but that won't help you fixing your main problem :) : It is coming from the fact that timeout have been raised and so no update can be done on resource because client is preventing it.

To increase the timeout on operator with the helm chart, you can use this snippet:

args:
  - --leader-elect
  - --reconcile-timeout=30s # By default it is set to 10s

But I recommend you to check your PG engine installation because 10s should be enough...

Do not hesitate if you have any other question. I will let you close this issue.

Have a nice day !