Inconsistency on Reports
Incident Report for Pipefy
Postmortem

Authors: SRE, Data Engineering and Support Teams

Date: 2020-07025

Status: Resolved 

Summary: 

The performance degradation was caused by an unusually high number of requests generated by a single operation. Those requests triggered automations, which caused exponential growth in our processing queue and ended up causing timeouts in the and affecting the performance of the reports.

Impact: 

Major - The performance degradation on the reports feature happened from Jul 25 15:26 (UTC) to Jul 25 19:11 (UTC).

Root Cause: 

The root cause of what caused the initial overload in our processing queues is under investigation and yet to be determined. 

However, due to our initial findings we strongly believe that the changes that’ll be implemented soon in the application’s non-relational database will be enough to prevent similar incidents from happening again in the future.

Detection and resolution: 

The issue was detected by our internal monitoring system that triggered automated alerts and informed the team.

Action plan: Preventive action items

1. Modifications in our Database configuration in order to increase overall performance. Due date: 08/31/2020

Posted Jul 31, 2020 - 13:41 UTC

Resolved
This incident has been resolved.
Posted Jul 25, 2020 - 19:11 UTC
Monitoring
A fix has been implemented and we are monitoring the results.
Posted Jul 25, 2020 - 18:50 UTC
Update
We are continuing to work on a fix for this issue.
Posted Jul 25, 2020 - 17:30 UTC
Update
The engineering team is still working on a fix for this issue.
Posted Jul 25, 2020 - 16:26 UTC
Identified
Our database for Reports is having a massive volume of requests. Users may face inconsistencies in reports
Posted Jul 25, 2020 - 15:26 UTC
This incident affected: Advanced Reports and Filters.