MAS and Synapse in crashloop due to initContainer permission issue

[oazabir]

Issue Summary

The render-config init containers in ESS (Element Server
Suite) experience a permission denied error when trying
to write configuration files to an emptyDir volume,
causing pods to enter a crash loop.

Environment Details

• Kubernetes Distribution: MicroK8s v1.33.5
• OS: Ubuntu 24.04.3 LTS
• Kernel: 6.8.0-87-generic
• Container Runtime: containerd://1.7.27
• Architecture: amd64
• Helm Chart: matrix-stack-25.12.0
• Matrix Tools Image:
  ghcr.io/element-hq/ess-helm/matrix-tools:0.5.6

Affected Components

1. Synapse: ess-synapse-main-0 StatefulSet
2. Matrix Authentication Service:
   ess-matrix-authentication-service-6644fd87cd-xxxx
   Deployment

Error Details

Error writing to file: open /conf/homeserver.yaml:
permission denied

Root Cause Analysis

1. Security Context Configuration

Both containers have a restrictive security context:
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
readOnlyRootFilesystem: true

2. Volume Configuration

The affected volume is an emptyDir with medium: Memory:
volumes:

• name: rendered-config
  emptyDir:
  medium: Memory

3. Volume Mounts

• render-config init container: Mounts /conf (writable)
• synapse container: Mounts /conf/homeserver.yaml
  (readOnly, subPath)

4. Issue Pattern

The permission issue appears to be related to:

1. The combination of readOnlyRootFilesystem: true with
   in-memory emptyDir
2. Potential timing issue where the volume is not
   properly initialized with the correct permissions
3. The containers running as a non-root user (implicit
   due to dropped ALL capabilities)

Timeline

1. Initial deployment: Dec 6, 23:30:48 UTC (Revision 1)
2. Helm upgrade: Dec 7, 00:39:10 UTC (Revision 2) - This
   might have triggered the issue
3. Issue duration: Approximately 15 hours (until manual
   intervention)

Workaround Applied

Force-deleting and recreating the affected pods resolved
the issue:
kubectl delete pod -n ess ess-synapse-main-0 --force
--grace-period=0
kubectl delete pod -n ess
ess-matrix-authentication-service-xxx --force
--grace-period=0

Potential Solutions to Consider

1. Add explicit user/group to securityContext:
   securityContext:
   runAsUser: 1000
   runAsGroup: 1000
   fsGroup: 1000
2. Add volume permission initialization:
   securityContext:
   fsGroup: 1000
3. Consider using a regular emptyDir instead of memory
   medium for debugging
4. Add init container to set permissions:

• name: set-permissions
  image: busybox
  command: ['sh', '-c', 'chmod 755 /conf']
  volumeMounts:
  • name: rendered-config
    mountPath: /conf

[benbz]

Do you have documentation to suggest that emptyDir of type: memory have permission issues? If anything they should be more resilient to permission problems as there's nothing on disk that could possibly persist between Pod recreations.

Setting readOnlyRootFilesystem: true with emptyDirs at specific paths is the documented approach for making only specific paths read-write. We are not going to remove readOnlyRootFilesystem: true as a default, though you are welcome to configure your values as such

All Pod securityContexts have fsGroup, runAsGroup, runAsUser set. We don't set any of those on individual containers within the Pod as they'll share the same Pod securityContext and so the IDs will be consistent across all containers in the same Pod.

I don't see how adding an initContainer is going to be helpful or practical: if it runs with the same IDs as everything else it is going to run into the same permissions problem when it attempts to chmod. If it runs with privileged IDs then either allowPrivilegeEscalation: true or runAsUser: 0 neither of which is acceptable default change for us, though again you are welcome to configure your values as such.

As per #888 (comment) can you provide the following details:

• Can you post the full logs from one of the init-containers that fails after undoing any and all security context changes and deleting the pods?
  • Can you provide a kubectl get pod -o yaml of one of the failing Pods in this case
• Is SELinux or AppAmor in use on the underlying Ubuntu server?
• Anything interesting been done with quotas or ACLs on the filesystems on the server?