Pipefy is down
Incident Report for Pipefy
Postmortem

Root Cause

The migration performed on Sunday involved moving from one cluster to another, where some data types were changed and columns removed. This affected database statistics and execution plans, leading to poor performance.

Resolution

The solution involved increasing the sampling range significantly, which improved understanding and execution plans. Temporary measures included disabling a database feature globally and pre-loading indexes into memory. Once the permanent fix was applied, the feature was re-enabled.

Action Plan

Initial temporary solutions brought the system back online by disabling a database feature and caching indexes to improve queries. Sampling range was increased for better execution plans. Future actions include improving alerts to monitor database health, increasing load tests before migrations

Posted Oct 09, 2024 - 20:21 UTC

Resolved
With the changes implemented so far, the data retrieval process is smoothly running. RCA will be posted on the post mortem. All good now!
Posted Oct 07, 2024 - 19:19 UTC
Update
Optimized data retrieval has been activated on one of our service replicas, and we're monitoring it. Pipefy performance is all good, but we want to make sure everything runs smoothly after this change.
Posted Oct 07, 2024 - 19:00 UTC
Update
Pipefy's performance is all good now. We're now monitoring the performance of our queue management system to prevent unexpected loads.
Posted Oct 07, 2024 - 18:33 UTC
Update
After setting the index of one of our tables in the cache memory, performance is now back to normal. We're monitoring Pipefy's performance for the next minutes.
Posted Oct 07, 2024 - 18:06 UTC
Update
Systems are operational, but there is a delay in the performance, which translates to a slow navigation inside Pipefy. We're spending all the efforts to investigate the cause of this delay.
Posted Oct 07, 2024 - 17:37 UTC
Update
For what we could test, Pipefy is now loading normally, and the navigation inside the app is running smooth. We'll keep monitoring it for a few more moments.
Posted Oct 07, 2024 - 16:57 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Oct 07, 2024 - 16:26 UTC
Update
While latency persists, Pipefy is loading more smoothly. Our engineering team is actively identifying all resources needed to implement an effective solution to resolve the issue.
Posted Oct 07, 2024 - 16:18 UTC
Update
Right now, we are still encountering latency. Our team is dedicated to determinate the issues affecting the platform’s performance and is actively working on a effective solution.
Posted Oct 07, 2024 - 16:00 UTC
Update
At the moment, we continue to experience high latency and error messages while loading Pipefy. Our team is committed to uncovering the factors impacting the platform's performance, and we are focused on implementing a robust solution.
Posted Oct 07, 2024 - 15:45 UTC
Update
We want to assure you that our investigation into the current latency and errors is ongoing. Our engineers are meticulously examining the system to pinpoint the causes and implement necessary fixes.
Posted Oct 07, 2024 - 15:31 UTC
Update
Currently, there is still a high latency and errors showing on screen while loading Pipefy. Our team continues to actively investigating in details the whole platform to find a final solution.
Posted Oct 07, 2024 - 15:15 UTC
Update
Pages are now loading, but taking very long to load and showing errors on screen. Our team is actively investigating the situation to implement a solution as quickly as possible.
Posted Oct 07, 2024 - 14:57 UTC
Update
We are continuing the investigation to identify the root cause for this issue.
Posted Oct 07, 2024 - 14:45 UTC
Update
We are still trying to identify the reason that is causing Pipefy to be down/not loading.
Posted Oct 07, 2024 - 14:28 UTC
Investigating
We are currently investigating this issue.
Posted Oct 07, 2024 - 14:14 UTC
This incident affected: Application.