Slowness and timeout on Pipefy
Incident Report for Pipefy
Postmortem

Summary:
Slowness and timeout caused by an overload on our reader database

Impact:
Major, Pipefy was inaccessible for less than a minute

Detection:
Engineering team noticed the overload on the database through the monitoring tools

Root Cause:
An index in production started to be inefficient for a specific query, which degraded the performance of the core database

Resolution:
A new index was created increasing the efficiency for that query.

Posted Dec 09, 2019 - 20:25 UTC

Resolved
The issue has been solved
Posted Dec 09, 2019 - 15:43 UTC
Monitoring
The fix has been implemented and the Pipefy is now stable. The performance of one of our filters in the database was not good and we improve it and reduce the time spent to execute the query.

We will keep monitoring
Posted Dec 09, 2019 - 15:31 UTC
Identified
We identified where the issue was happening and we are deploying a fix.
Posted Dec 09, 2019 - 15:24 UTC
Investigating
We are currently investigating what is causing the slowness
Posted Dec 09, 2019 - 14:39 UTC
This incident affected: Application.