Some users might not be able to log into Pipefy

Incident Report for Pipefy

Postmortem

Root Cause

A technical issue was identified within the cache cluster of the authenticator provider, stemming from previosuly established infrastructure limitations. These constraints hindered the system’s ability to effectively manage cached data, resulting in some users being unable to sign in

Resolution

The system was stabilized by restarting the entire cluster. Once stability was confirmed, further adjustments were made by increasing allocated infra to prevent future issues.

Action Plan

To enhance system reliability, allocated infra limitations were increased. Future steps include enhancing alerts to quickly identify and address potential problems.

Posted Sep 25, 2024 - 20:47 UTC

Resolved

After this rollout, logins are all good now. We'll post more details about this episode in the post-mortem.

Posted Sep 19, 2024 - 20:17 UTC

Monitoring

After a rollout in our infra, the login seems to be more stable. We're now monitoring the next login events.

Posted Sep 19, 2024 - 20:09 UTC

Update

After a rollout in our infra, the login seems to be more stable. We're now monitoring.

Posted Sep 19, 2024 - 20:09 UTC

Identified

Our authentication tool is unstable, and this instability led to login issues. It works every now and then, but we're investigating more details on what could have caused this issue.

Posted Sep 19, 2024 - 20:06 UTC

Update

We are continuing to investigate this issue.

Posted Sep 19, 2024 - 19:53 UTC

Investigating

Users that are already logged in will not be affected, but users that are trying to login now may face difficulties. We're investigating the root cause, and we'll send updates as we progress.

Posted Sep 19, 2024 - 19:52 UTC

This incident affected: Application.