Unexpected internal issues

Incident Report for Pipefy

Postmortem

Post-Mortem: Domain Service Disruption (June 12 - June 13, 2024)
Pipefy experienced a service disruption between June 12, 19:20 UTC, and June 13, 06:26 UTC. This was caused by an issue with our domain name registrar following a Cloudflare outage. Inconsistencies in our internal DNS configuration process and the register's instability led to resolution failures and temporary platform inaccessibility.

Impact:
Users were unable to access the platform due to DNS resolution failures , affecting app.pipefy.com, corporate emails, support chat, status.pipefy.com, api.pipefy.com, and third-party integrations. No customer data was impacted, exposed, or compromised throughout the entire duration of the incident.

Current Status:
The issue has been resolved, and the platform is fully operational.

Lessons Learned & Next Steps: We are committed to continuous improvement.

Key actions include:

Improving our Business Continuity and Disaster Recovery Plans.
Transitioning to a new domain registrar with a better Service Level Agreement (SLA).
Enhancing monitoring, auditing, and internal workflows.
Establishing diverse communication channels.

We are dedicated to providing a secure and uninterrupted experience and appreciate your understanding.

Thank you!

Pipefy

Posted Jun 16, 2025 - 22:21 UTC

Resolved

We would like to confirm that the issue has been officially resolved. The Root Cause Analysis (RCA) will be shared as soon as we have more details.
We sincerely apologize for the inconvenience and for falling short of the level of service you expect from us.

Posted Jun 13, 2025 - 07:25 UTC

Monitoring

The problem has been identified and the fix has been applied by our provider. You may experience some intermittency in the application as the fix may take some time to fully propagate.
We sincerely apologize for the inconvenience and for falling short of the level of service you expect from us.

Posted Jun 13, 2025 - 06:26 UTC

Update

We're still working hard to fix the internal issues that popped up after that recent problem with our third-party provider.

Thanks for hanging in there and for your understanding while we work to get everything back to normal.

Posted Jun 13, 2025 - 00:20 UTC

Identified

We're letting you know that the issue with our third-party provider has been resolved.
Because of this incident, we ran into some internal problems that impacted our services. Our team is actively working to get everything back to normal now.
We appreciate your patience and understanding while we work to fully restore all services.

Posted Jun 12, 2025 - 22:12 UTC

Investigating

We are still waiting for Cloudflare's fix.

Posted Jun 12, 2025 - 20:51 UTC

Update

We are still waiting for Cloudflare's fix.

Posted Jun 12, 2025 - 20:33 UTC

Update

More details about the issue on their status page: https://www.cloudflarestatus.com/incidents/25r9t0vz99rp

Posted Jun 12, 2025 - 19:52 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jun 12, 2025 - 19:47 UTC

Update

We are continuing to work on a fix for this issue.

Posted Jun 12, 2025 - 19:45 UTC

Identified

The issue has been identified and a fix is being implemented.

Posted Jun 12, 2025 - 19:43 UTC

Monitoring

A fix has been implemented and we are monitoring the results.

Posted Jun 12, 2025 - 19:40 UTC

This incident affected: Application.