feat: verifier nonce store falls back silently to in-memory when Redis connection fails at startup — multi-replica deployments lose replay protection without any operator alert

[divyanshim27]

🚨 Problem Statement

The README documents: "Verifier nonce replay protection defaults to memory for local/tests and supports Redis for multi-replica deployments. Invalid signatures are rejected before nonce claims, and Redis outages fail closed."

The last clause — "Redis outages fail closed" — is critical. However, the current implementation in verifier/ initializes the Redis nonce store at startup. If VERIFIER_NONCE_STORE=redis but the Redis connection fails during the startup health check, the verifier's behavior is undocumented for this scenario.

If the verifier falls back to in-memory on Redis startup failure, it silently provides zero cross-replica replay protection — a security regression that looks like healthy operation from the outside (HTTP 200s still return, no crash). An operator who configured Redis for multi-replica would have no idea replay protection is broken.

If it fails closed (refuses to start), there is no structured startup error logged with actionable remediation steps — just a generic connection error.

Current Impact

In the docker-compose.yml deployment: if Redis starts after the verifier (common in Docker compose race conditions), the verifier may miss the Redis health check window and silently degrade to in-memory nonce storage for the entire session.

Proposed Fix

In verifier/src/ (Rust/Axum startup code):

// verifier/src/nonce_store.rs or main.rs

async fn init_nonce_store(config: &Config) -> Result<NonceStore, StartupError> {
    match config.nonce_store_type.as_str() {
        "redis" => {
            let client = redis::Client::open(config.redis_url.as_str())
                .map_err(|e| StartupError::NonceStoreInit(format!("Redis URL parse error: {e}")))?;

            // Explicit connection test at startup — fail fast, not silently
            let mut conn = client.get_async_connection().await
                .map_err(|e| StartupError::NonceStoreInit(
                    format!(
                        "Redis connection failed at startup (VERIFIER_NONCE_STORE=redis, REDIS_URL={}): {}. \
                         Set VERIFIER_NONCE_STORE=memory for single-instance development.",
                        config.redis_url, e
                    )
                ))?;

            // Ping to confirm the connection works
            redis::cmd("PING").query_async::<_, String>(&mut conn).await
                .map_err(|e| StartupError::NonceStoreInit(
                    format!("Redis PING failed: {e}. Verifier refusing to start with broken replay protection.")
                ))?;

            tracing::info!("Nonce store: Redis ({})", config.redis_url);
            Ok(NonceStore::Redis(client))
        }
        "memory" => {
            tracing::warn!(
                "Nonce store: in-memory. NOT suitable for multi-replica deployments. \
                 Set VERIFIER_NONCE_STORE=redis for production."
            );
            Ok(NonceStore::Memory(Default::default()))
        }
        other => Err(StartupError::NonceStoreInit(
            format!("Unknown VERIFIER_NONCE_STORE value: '{other}'. Valid: 'redis' | 'memory'")
        ))
    }
}

Also update GET /readyz in the verifier to include nonce store health:

{
  "status": "ready",
  "nonce_store": "redis",
  "nonce_store_healthy": true
}

So operators and the gateway's startup check can detect degraded replay protection before accepting traffic.

Files to Modify

File	Change
verifier/src/ (startup / nonce store init)	Add Redis PING at startup, fail with actionable error if redis store unreachable
verifier/src/ (readyz handler)	Add nonce_store_healthy field to /readyz response
DEPLOY.md	Document that VERIFIER_NONCE_STORE=redis fails hard on startup if Redis is unreachable

Suggested labels: security, backend, verifier, level: intermediate

I would like to work on this. Could you please assign it to me?

[github-actions]

Assigned @divyanshim27 based on the issue request.