Improve robustness of SENAITE -> Tamanu sync when results are rejected by Tamanu

[rohan-bes]

Requirement

Currently, when Tamanu rejects results from SENAITE, the queue to push results to Tamanu will become blocked. This means that if a single result is considered invalid by Tamanu, the entire integration stops working.

It's also not easy to resolve these issues when they do occur, and require manual database intervention.

Proposal

1. Allow SENAITE to keep sending other results to Tamanu if a result fails. A single failing request should not block the queue
2. Provide an interface to allow users to view the results queue and possibly remove failing requests

[markbes1]

Add a view that allows the user to view tasks that have failed when being sent to Tamanu and add option to cancel them so the system can move on to next task and not block results updating in Tamanu

[kujo4pm]

This is good. I would just like to elaborate on the proposal and get feedback before it is dev ready. This solutions amounts the dead letter queue or quarantine (of the sort once on Tupaia - not sure if it still exists).

It would work like this:
Any failing DiagnosticReport POST would be taken out of the regular task queue (or even flagged in the queue as isInvalid = true and skipped - although over time this might slow things down).
This view would be available to labmanagers.
The view here would be a list of all failed POST attempts ordered by attempt date/time. For each would need to include:

• Link to sample
• Current sample status - has it been successfully sent later.*
• Reported reason for failure: (See: Discussion: Error messages as to why a DiagnosticReport POST has failed #163)
• Action (Delete or Retry)
  • Delete: Remove from quarantine - this can be flagged if compliance forces
  • Retry: Attempt to post this again.

*Note that the same sample can be published multiple times and one can fail while a later attempt can pass - hence we're not quarantining the sample but the attempt to publish it. Another option here might be to delete it from the queue if another report for the same sample is published. @xispa thoughts?

@rohan-bes and @markbes1 are you able to provide feedback or sign off if all good and we can get cracking?

[xispa]

*Note that the same sample can be published multiple times and one can fail while a later attempt can pass - hence we're not quarantining the sample but the attempt to publish it. Another option here might be to delete it from the queue if another report for the same sample is published. @xispa thoughts?

When a sample is verified/published/invalidated/etc., a task for notifying Tamanu about the action is added in a FIFO queue. Each task in the queue has a unique identifier made of <context_uid>-<name>, where, in this particular case of pushing a DiagnosticReport to Tamanu, <context_uid> would be the sample's UID and <name> the name of the task (diagnostic_report). The queue is unique-key aware, meaning that new tasks whose key exists in the queue are automatically declined.

Therefore, if a diagnostic_report task fails, no matter how many tasks of this type for same sample are triggered, none of them will be added to the queue. That said, the diagnostic_report task always rely on current sample information (results report, status, field values, etc.). As a result, if the problem with the notification lives in SENAITE, then the notification process will resume gracefully as soon as the problem is addressed by lab user. Likewise, if the problem lives in Tamanu (e.g. a test for a given LOINC code/keyword is missing) or is because of connectivity issues, the process will resume gracefully as well, as soon as the connectivity is reestablished or Tamanu-side problem is resolved.

This means that in the listing we can only display the current status of the object (the sample in this case) the task (diagnostic_report in this case) is about, not the status it had when the task was added to the queue.

[kujo4pm]

Right - that's all very clear. So there is a 1-to-1 relationship between a given sample and a diagnosticReport task - and the task will fetch the latest Sample data to build the DiagnosticReport.
And the task remains in the queue even if it has failed? I suppose that's the basis for this blockage in the first place. After it has been successfully processed then I'm assuming it is popped from the queue?

[xispa]

And the task remains in the queue even if it has failed? I suppose that's the basis for this blockage in the first place. After it has been successfully processed then I'm assuming it is popped from the queue?

Yes, the task is popped from the queue as soon as it has been successfully processed.