Despite the use of dynamodb to synchronize on a lock before submitting jobs, there still is an small opportunity for duplicate job submission:
Scenario 1:
- thread 1 grabs the lock to submit, loses network connectivity, or pauses for a while (e.g. SIGSTOP)
- thread 2 trylocks until the lock held by thread 1 expires, eventually steals it, and submits
- thread 1 resumes and submits thinking it still holds the lock (untested idea: it may be possible to confirm that the lock is (or isn't) still held when the thread attempts to release)
Scenario 2:
- thread 1 grabs the lock, submits job
- user deletes dynamodb table or entry
- thread 2 grabs the lock, doesnt' become aware of the running job of the same name, and submits anew.
Both these scenarios can be mitigated if at the start of the entrypoint script, the program consults the dynamodb table entry corresponding to its job ID, and verifies that the submitter name in the table entry matches that ID in the job's environment variables. If the submitter name in the table and the environment match, the job continues, if not, it means another job was started with the same ID, and the current script aborts.
Despite the use of dynamodb to synchronize on a lock before submitting jobs, there still is an small opportunity for duplicate job submission:
Scenario 1:
Scenario 2:
Both these scenarios can be mitigated if at the start of the entrypoint script, the program consults the dynamodb table entry corresponding to its job ID, and verifies that the submitter name in the table entry matches that ID in the job's environment variables. If the submitter name in the table and the environment match, the job continues, if not, it means another job was started with the same ID, and the current script aborts.