Pipefy's web application is currently facing instabilities.
Incident Report for Pipefy
Postmortem

Authors: SRE, Data Eng, Support team

Date: 2020-04-23

Status: Resolved

Summary: Degradation in web application

Impact: Minor - The performance degradation on the application happened from Apr 23 20:34 (UTC) to Apr 23 20:55 (UTC).

Root Cause:

The identified root cause of this degradation was an unexpected behavior in the structures that control the connections to our databases.

Detection and resolution:

The issue was detected by our internal monitoring system that triggered an alert and informed the team.

After the root cause was identified, the structure was rebooted and the requests were sent and received normally.
Action plan: Preventive action items

Pipefy’s SRE team is already working on an improved structure that will provide a permanent solution and mitigate these issues. We will, however, contact the service that provides us with the structure that failed in order to identify what caused the situation.

Posted May 04, 2020 - 17:37 UTC

Resolved
The instability has been fixed and the performance of the platform has been restored.
As soon as the preventive investigation process is over, we’ll share further details about the causes, implemented fixes and preventive actions to be implemented.
Posted Apr 23, 2020 - 20:55 UTC
Monitoring
The system instability has been fixed and the performance of the platform has been restored.
We are currently monitoring the system to ensure all features are working as expected. As soon as the monitoring process is over we’ll share further details about the causes, investigation and preventive actions to be implemented.
Posted Apr 23, 2020 - 20:45 UTC
Investigating
We are currently investigating the causes of the instability and working towards restoring full access to the platform as soon as possible. Some users may receive error messages/suffer timeout.
Any further details about the system status, investigation and preventive actions to avoid future incidents will be shared as soon as available.
Posted Apr 23, 2020 - 20:34 UTC
This incident affected: Application.