Filters are not working properly

Incident Report for Pipefy

Postmortem

Root Cause

The issue was caused by a significant update to the system, which led to filters not functioning as expected. This disruption affected the normal operation of the filters, causing inconvenience to users.

Resolution

To resolve the issue, we increased the number of backup systems to ensure smoother operation and prevent similar problems in the future.

Action Plan

We have taken steps to enhance system reliability by increasing backup systems. Additionally, a new monitoring alert has been set up to quickly identify and address any future issues.

Posted Oct 07, 2024 - 19:46 UTC

Resolved

Filters are back to normal now. The actions we took did solve the issue, but the root cause is yet to be found. We'll post more about the root cause in the post-mortem.

Posted Sep 24, 2024 - 15:53 UTC

Monitoring

We manually scaled our VMs to handle the queue, and it is normalized now. We're monitoring the next minutes.

Posted Sep 24, 2024 - 15:47 UTC

Investigating

There was a high load in a queue that manages updates in card attributes (e.g.: title, current phase and labels), and this load led to a delay in the processing of data inside the queue. Even though it is decreasing now, we are investigating the genesis of this event.

Posted Sep 24, 2024 - 15:34 UTC

Update

For more context on the issue: when we apply a filter, matching cards are not being displayed. There are no pop-up error messages, but the expected results aren't show.

Posted Sep 24, 2024 - 15:11 UTC

Identified

When we filter for data inside Pipefy, it might not come as expected. We are looking for the root cause.

Posted Sep 24, 2024 - 15:00 UTC

This incident affected: Features (Filters).