[Monitor] Database connection failures

[islandbitcoin]

[islandbitcoin]

Logs captured during outage on 12/24/2025

Defaulted container "api" out of: api, wait-for-mongodb-migrate (init) {"level":30,"time":1766287783491,"pid":1,"hostname":"api-5ffdf58f94-k4vmc","msg":"Successfully loaded config from /var/yaml/custom.yaml"} {"level":60,"time":1766287817745,"pid":1,"hostname":"api-5ffdf58f94-k4vmc","msg":"error connecting to mongodb"} {"level":50,"time":1766287817745,"pid":1,"hostname":"api-5ffdf58f94-k4vmc","err":{"type":"MongooseServerSelectionError","message":"connec t ECONNREFUSED 10.245.77.32:27017","stack":"MongooseServerSelectionError: connect ECONNREFUSED 10.245.77.32:27017\n at _handleConnecti onErrors (/app/node_modules/mongoose/lib/connection.js:816:11)\n at NativeConnection.openUri (/app/node_modules/mongoose/lib/connectio n.js:791:11)\n at async setupMongoConnection (/app/lib/services/mongodb/index.js:71:9)","reason":{"type":"Unknown","servers":{},"stale ":false,"compatible":true,"heartbeatFrequencyMS":10000,"localThresholdMS":15,"setName":null,"maxElectionId":null,"maxSetVersion":null,"co mmonWireVersion":0,"logicalSessionTimeoutMinutes":null}},"msg":"server error"}

[islandbitcoin]

Used this to temp fix the issue:

kubectl rollout restart deployment -n flash flash-mongodb
kubectl rollout restart deployment -n flash api"

leaving open for monitoring

[brh28]

Current hypothesis is this was caused by a Kubernetes upgrade triggered by DigitalOcean. The upgrade triggered a redeploy of our infrastructure and the Websocket depends on Mongodb