Slowness and timeout on Pipefy

Incident Report for Pipefy

Postmortem

Summary:
Slowness and timeout caused by an overload on our reader database

Impact:
Major, Pipefy was inaccessible for less than a minute

Detection:
Engineering team noticed the overload on the database through the monitoring tools

Root Cause:
An index in production started to be inefficient for a specific query, which degraded the performance of the core database

Resolution:
A new index was created increasing the efficiency for that query.

Posted Dec 09, 2019 - 20:25 UTC

Resolved

The issue has been solved
Posted Dec 09, 2019 - 15:43 UTC

Monitoring

The fix has been implemented and the Pipefy is now stable. The performance of one of our filters in the database was not good and we improve it and reduce the time spent to execute the query.

We will keep monitoring
Posted Dec 09, 2019 - 15:31 UTC

Identified

We identified where the issue was happening and we are deploying a fix.
Posted Dec 09, 2019 - 15:24 UTC

Investigating

We are currently investigating what is causing the slowness
Posted Dec 09, 2019 - 14:39 UTC
This incident affected: Application.