Root Cause

The system experienced an overload because the implemented scalability strategy of the new model was insufficient to handle the increased load on our new configuration, which resulted in the database being overwhelmed.

Resolution

To solve the problem, a percentage of users were reverted to the application's default configuration and we manually set the scalability strategy temporarily

Action Plan

We have made enhancements to our system to improve its scalability and stability, including but not liited to optimizing the database access mode and adjusting the application scalability. Additionally, we have implemented additional monitoring systems to track performance and ensure stability effectively.

We will continue to internally monitor our infra load to ensure that all users can use the platform, but everything is back to normal

We are monitoring our current infra load to ensure that all users can use the platform. Some webhooks might have faced delays and some users might be having slowness when clicking to open cards/pipes.

We are monitoring our current infra load to ensure that all users can use the platform without latency. Some webhooks might have faced delays.

A fix has been implemented and we are monitoring the results.

We are continuing to investigate this issue.

We are continuing to investigate this issue.

We are continuing to investigate this issue.