Reload should tell us if restart is required

[SamMousa]

Proposal

Make it so that when we do a nomad reload we get some useful information:

• Verify the config before doing anything, if valid proceed.
• Verify the changes will apply after a reload, if not proceed, but inform us that a restart is required for some of the changes.

Use-cases

Clean approach whenever someone is updating node configurations for nomad. This is mostly a quality of life thing.

Attempted Solutions

N/A

[tgross]

inform us that a restart is required for some of the changes.

The UX of this is not great because unless you're using the /v1/agent/reload API (which hasn't even shipped yet!), the only way to reload is to send a signal to the agent. So the only place to surface "informing" the user is via logs, which is better than nothing I guess? And certainly we could return it to the new reload API I think.

This also might be a useful extension to https://developer.hashicorp.com/nomad/commands/config/validate where we could return a different exit code for "valid but needs restart".

The exact UX needs some design here, but I'll mark this for roadmapping.

(Internal ref https://hashicorp.atlassian.net/browse/NMD-1537)

[SamMousa]

For us the logs is a fine way to service it. Another way would be a dynamic meta attribute (which was recently added iirc) for large fleets this would allow people to avoid those nodes for scheduling etc?

Like a needs-restart label?