Root Cause

The delay occurred because we failed to process a specific update, which was unable to be sent to the queue that later goes to the external system. This happened due to a connection field being incorrectly updated through the API.

Resolution

As several updates were taking place during the incident, we had to implement a solution in the application to deal with this exception. Essentially, the payload size is now validated before being sent to the queue, which solved the problem during the incident. Other actions have been mapped for further analysis.

Action Plan

We will create a disaster recovery process and resume discussion on alarm systems.

With the applied fix, there will be no more issues. Incident is now solved. We'll send more information on post-mortem in the next 2 business days,

We're monitoring the queue. It is in a very sharp descent, and it'll soon be all good.

Our graphs shows the fix results are coming. We're still monitoring it until it is all good.

The fix has been implemented, and we're now monitoring the outcome of it. The fix was adding an extra validation step. Filters are not working yet because there's still a queue to handle, but it'll soon be all good.

We understood the reason for the issue, and we're now working on the fix.