Service Degradation Post-Mortem
Authors: SRE, Security and Support Teams
Date: 2021-04-20
Status: Resolved
Summary:
Due to the unavailability of the third-party authentication service used by Pipefy \(Auth0\), users were unable to log into Pipefy's platform. More information about the causes of the unavailability can be found directly on the vendor's status page.
Users that were already logged in when the incident began were not initially impacted and temporary measures were implemented to prevent the need to re-authenticate while the service was unavailable.
Impact:
Major - The authentication service was unavailable from Apr 20 16:08 \(UTC\) to Apr 21 00:13 \(UTC\)
Root Cause:
The identified root cause of this incident was a major outage in the Auth0 service, affecting mainly the US-1 region.
Detection and resolution:
The issue was detected by our internal monitoring system that identified application failures. The unavailability was also reported by several users.
Due to the fact that the root cause was a major outage in a third-party service, Pipefy's internal teams worked on minimizing the impacts on the users that were already logged in while waiting for the incident to be resolved on Auth0's side.
Action plan: Preventive action items
1\. Implementation of improvements in the systems that monitor the authentication mechanisms. Due date: 05/14/2021