After some hours, Cascade no longer serves the zone (NXDOMAIN)

[bortzmeyer]

This is exactly bug #353 but it was closed and I don't have permission to re-open it.

cascade 0.1.0-beta5 (no-git). After one or two days of normal behavior:

% cascade zone status --detailed cours-dns.fr
zone:   cours-dns.fr
policy: ovh-no-ed25519
source: /etc/cascade/zones/cours-dns.fr

review
  loaded: script `/etc/cascade/reviews/review.bash unsigned`
  signed: script `/etc/cascade/reviews/review.bash`

last published
  loaded serial: 2026022000
  signed serial: 2026071503
  timestamp:     2026-07-15T13:33:25.163621923Z
  size:          22 records

status: idle

Published zone available at:
  - 127.0.0.1:8053

DNSSEC keys:
  KSK tagged 49553:
    Reference: file:///var/db/cascade/keys/Kcours-dns.fr.+013+49553.key
    Actively used for signing
  ZSK tagged 36370:
    Reference: file:///var/db/cascade/keys/Kcours-dns.fr.+013+36370.key
    Actively used for signing
  Details:
    key file:///var/db/cascade/keys/Kcours-dns.fr.+013+36370.key expires at 2026-07-22T07:32:44Z
    key file:///var/db/cascade/keys/Kcours-dns.fr.+013+49553.key expires at 2026-09-14T07:32:44Z

But:

% dig @localhost -p 8053 cours-dns.fr SOA    

; <<>> DiG 9.20.23-1~deb13u1-Debian <<>> @localhost -p 8053 cours-dns.fr SOA
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 40316
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 0
;; QUESTION SECTION:
;cours-dns.fr.		IN SOA

;; Query time: 8 msec
;; SERVER: 127.0.0.1#8053(localhost) (UDP)
;; WHEN: Wed Jul 15 21:23:36 CEST 2026
;; MSG SIZE  rcvd: 41

(Same thing for ports 8051 and 8052.)

The last lines of history;

2026-07-15T11:33:15Z -          Keyset command '/home/cascade/.cargo/bin/dnst keyset -c /var/db/cascade/keys/cours-dns.fr.cfg cron' succeeded in 0s
2026-07-15T11:33:15Z -          Started resign
2026-07-15T12:33:20Z -          Keyset command '/home/cascade/.cargo/bin/dnst keyset -c /var/db/cascade/keys/cours-dns.fr.cfg cron' succeeded in 0s
2026-07-15T12:33:20Z -          Started resign
2026-07-15T13:33:25Z -          Keyset command '/home/cascade/.cargo/bin/dnst keyset -c /var/db/cascade/keys/cours-dns.fr.cfg cron' succeeded in 0s
2026-07-15T13:33:25Z -          Started resign
2026-07-15T13:33:25Z 2026071503 Signing succeeded (triggered by a change in signing keys)
2026-07-15T13:33:25Z 2026071503 Signed zone review pending
2026-07-15T13:33:25Z -          Signed zone review approved
2026-07-15T14:33:30Z -          Keyset command '/home/cascade/.cargo/bin/dnst keyset -c /var/db/cascade/keys/cours-dns.fr.cfg cron' succeeded in 0s
2026-07-15T14:33:30Z -          Started resign

% dnst --version
dnst 0.2.0-alpha3

The log says:

Jul 15 18:21:51 ayla cascaded[2932167]: ayla cascaded[2932167]: Returning SERVFAIL for AXFR request from client '127.0.0.1' for zone 'cours-dns.fr': zone is empty
Jul 15 18:22:11 ayla cascaded[2932167]: ayla cascaded[2932167]: Enqueuing a re-sign operation
Jul 15 19:18:37 ayla cascaded[2932167]: ayla cascaded[2932167]: Returning SERVFAIL for AXFR request from client '127.0.0.1' for zone 'cours-dns.fr': zone is empty

The zone is not empty:

% cat /etc/cascade/zones/cours-dns.fr
; -*- zone -*-

$TTL 300
@ IN  SOA ns4.bortzmeyer.org. hostmaster.bortzmeyer.org. (
        2026022000
        7200
        3600
        604800
        300 ) 
...

And did not change when cascade broke.

[bortzmeyer]

Note: when moving from Cascade Alpha to Beta, I did:

rm -f /var/lib/cascade/state.db /var/lib/cascade/zone-state/* /var/db/cascade/keys/*

Before adding the zones.

[bortzmeyer]

Like before, a cascade zone reload solves the problem (temporarily).

[bortzmeyer]

This zone was removed then re-added, but without removing its state (created with Cascade beta, not the old state of the alpha version, see #903). I try again: remove, delete the state, add the zone. We'll see.

[bal-e]

Thanks @bortzmeyer! Does the problem seem to appear when Cascade restarts? The issue might occur when Cascade persists and restores the zone data from disk.

I see the zone was published at 15 July 13:33, but the part of the log file you shared is from 15 July 18:21. Could you share the log file starting from 13:33 (maybe attach as a file if it's very big)? If you have already removed the log file and started over, no worries.

[bortzmeyer]

It now seems to work https://dnsviz.net/d/cours-dns.fr/alofFw/dnssec/ Temporary guess: the problem may be created when removing and re-adding while leaving the state (see #889 )