[subnet] Make validator more resilient

[ibraheem-abe]

Right now a single heartbeat failure can kill the validator, and some paths still call os._exit(1) on transient errors.
We should have some retries on heartbeats so we give a little more time for the platform to come back up.

Things to look into:

• validator heartbeat loop
• validator set-weights loop
• validator evaluation task supervision
• API lifespan background loops

Guidelines:

• do not exit the validator on one missed heartbeat
• retry transient platform/network failures with backoff
• only escalate after repeated failures or clearly fatal conditions

This is a Spike and first we need to understand the side-effects this might have. The os.kill approach did work to keep things in sync but wastes too many resources where its really not needed.