Write-up published
Resolved
Root Cause
The issue was caused by exceeding the execution limit in integration queues, leading to a loop in automation that triggered numerous webhooks. This resulted in an overload of a specific flow, creating a large processing queue. Consequently, clients experienced delays in their process executions.
Resolution
To resolve the issue, the customer service account was deactivated, which helped alleviate the problem.
Action Plan
To prevent future occurrences, a rate limit will be activated on the system, and the timeout for execution flows will be reduced.
Resolved
This incident has been resolved.
Monitoring
We are continuing to monitor for any further issues.
Monitoring
A fix has been implemented and we are monitoring the results.
Investigating
We are currently investigating this issue.