Pipefy is down

Incident Report for Pipefy

Postmortem

Root Cause

The system experienced an overload because the implemented scalability strategy of the new model was insufficient to handle the increased load on our new configuration, which resulted in the database being overwhelmed.

Resolution

To solve the problem, a percentage of users were reverted to the application's default configuration and we manually set the scalability strategy temporarily

Action Plan

We have made enhancements to our system to improve its scalability and stability, including but not liited to optimizing the database access mode and adjusting the application scalability. Additionally, we have implemented additional monitoring systems to track performance and ensure stability effectively.

Posted 9 months ago. Jul 11, 2024 - 15:03 UTC

Resolved

We will continue to internally monitor our infra load to ensure that all users can use the platform, but everything is back to normal
Posted 9 months ago. Jul 08, 2024 - 20:52 UTC

Update

We are monitoring our current infra load to ensure that all users can use the platform. Some webhooks might have faced delays and some users might be having slowness when clicking to open cards/pipes.
Posted 9 months ago. Jul 08, 2024 - 19:14 UTC

Update

We are monitoring our current infra load to ensure that all users can use the platform without latency. Some webhooks might have faced delays.
Posted 9 months ago. Jul 08, 2024 - 19:03 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted 9 months ago. Jul 08, 2024 - 18:16 UTC

Update

We are continuing to investigate this issue.
Posted 9 months ago. Jul 08, 2024 - 18:00 UTC

Update

We are continuing to investigate this issue.
Posted 9 months ago. Jul 08, 2024 - 17:47 UTC

Update

We are continuing to investigate this issue.
Posted 9 months ago. Jul 08, 2024 - 17:38 UTC

Investigating

Some customers might face latency and 503 error pages
Posted 9 months ago. Jul 08, 2024 - 17:25 UTC
This incident affected: Application.