Root Cause

The issue was caused by a set of bad automation settings within an organization, which led to a loop in the automation process. This resulted in slowness due to high latency in automations.

Resolution

The problematic automations were deleted. A script was executed to clear the processing queue by bypassing the affected cards, ensuring normal processing resumed.

Action Plan

During the incident, the impacted automations were softly deleted, and scheduled automations related to the affected cards were removed to prevent further disruptions.

Ongoing actions include evaluating a more generic solution for handling queued jobs and reviewing critical feature alerts to improve system resilience and prevent similar issues in the future.

This incident has been resolved.

A fix has been implemented and we are monitoring the results.

A fix has been implemented, and the application is expected to recover gradually over time. We are continuing to monitor the platform

The fix was implemented, and the platform is back to normal

We are continuing to work on a fix for this issue.