Wrap setup_amps in try during relock for two attempts

[tpsatt]

Semi-frequently after a hammer setup_amps fails on uxm_relock, usually due to a key error for the name of one of the amplifiers. It's pretty much always fixed by running uxm_relock again, but ROCs don't know to do this and smurf-tsars can be slow to respond.

This is a very simple solution of wrapping setup_amps in a try. If it fails, it tries again. If the second attempt fails, then it errors out.

[tristpinsm]

Not loving the nested try/except here but ok :)

[dpdutcher]

Yeah I also don't love the nested trys. Maybe include it in a for-loop, like

for attempt in range(max_attempts):
    try:
        <x>
        break
    except Exception as e:
        <log message>
else:
    raise e

Also, could this logic be moved within setup_amps itself, instead of being in uxm_relock?

[tristpinsm]

Actually, sorry I approved too soon. One more request: Could we log the exception when it is caught? Otherwise it gets really hard to figure out what is going on.

[dpdutcher]

Can this logic be moved to be within uxm_setup.setup_amps instead?
Here:

sodetlib/sodetlib/operations/uxm_setup.py

Lines 217 to 222 in 279def4

	amp_biases = S.get_amplifier_biases()
	# For C04, first check drain voltages
	if cc_rev == 'c04':
	for amp in amp_list:
	Vd = amp_biases[f"{amp}_drain_volt"]

That's the cause of the failure and where it should be caught and dealt with, not in this function. uxm_setup.uxm_setup also has a call to setup_amps, and repeating this check there would duplicate code.