Third-party provider, Oracle Cloud (OCI) Platform Instability

Incident Report for Pipefy

Postmortem

Summary 

On March 4, 2026, an incident occurred affecting multiple services in Oracle Cloud Infrastructure (OCI). Pipefy identified the disruption at 13:20 UTC, which impacted connectivity and service management operations for resources. As the Pipefy infrastructure relies on OCI, which affects the availability and normal behavior of certain platform functionalities for some customers. Oracle engineers investigated the issue, identified its source, and implemented mitigation actions. Full service functionality was confirmed as restored on March 4, 2026 at 19:40 UTC. Although Pipefy was back to normal at 19:40 UTC, we chose to wait for the provider to resolve the incident, which occurred on March 5, 2026 04:00am UTC . 

Impact 

During the incident window, customers have experienced intermittent latency or connection failures due to an underlying issue within the infrastructure. Because certain Pipefy components rely on OCI networking, services were temporarily degraded. We would like to emphasize that there was no loss or exposure of data. Full functionality was restored once the infrastructure issues were resolved.

Root Cause

The root cause of the incident was identified as an important and massive incident on Oracle's infrastructure that impacted the Pipefy platform. The disruption was caused by an issue affecting multiple Oracle Cloud Infrastructure (OCI) services, particularly involving components related to the Virtual Cloud Network (VCN). This issue affected the ability of services and resources to communicate reliably. 

The incident was triggered by OCI routine maintenance, the issue occurred because the Virtual Cloud Network Control Plane (VCNCP) was unable to propagate network configuration to the network dataplane services due to performance degradation of VCNCP’s internal database. At Pipefy, we identified the disruption at 13:20 UTC, this resulted in an inability to perform lifecycle actions, such as provisioning or scaling compute resources. Following that, OCI service teams were informed and the escalation process started. 

Oracle engineering teams identified the source of the disruption and implemented corrections to stabilize the affected infrastructure and restore normal connectivity and service operations for Pipefy on March 04 at 16:18 UTC, which resulted in full control plane recovery at OCI on March 05 at 04:40 UTC

Preventive Measures Performed

  • To prevent this scenario from happening again, OCI implemented rigorous improvements to its infrastructure, such as migrating VCNCP services to hosts with greater memory capacity. [Done]
  •  It also implemented performance optimizations in the VCNCP internal database, in addition to improvements in service monitoring. [Done]

Next steps

  • Improvements in the monitoring and observability process of the Pipefy platform.

Additional informations

We would like to point out that all information regarding incidents will be available on our support tool, but we reiterate that official communication is done through our status page https://status.pipefy.com.

We will continue to monitor the system closely to ensure stability and prevent future issues, while continuously working to enhance the customer experience.

Posted Mar 13, 2026 - 19:05 UTC

Resolved

This incident has been resolved.
Posted Mar 05, 2026 - 10:46 UTC

Update

We are continuing to monitor for any further issues.
Posted Mar 04, 2026 - 21:23 UTC

Update

We are continuing to monitor for any further issues.
Posted Mar 04, 2026 - 21:14 UTC

Update

We are continuing to monitor for any further issues.
Posted Mar 04, 2026 - 21:13 UTC

Monitoring

A fix has been implemented and we are monitoring the results.
Posted Mar 04, 2026 - 20:54 UTC

Update

Major update: Performance Improvement and Ongoing Monitoring

We would like to share an update regarding the platform instability affecting Pipefy.

The service is currently operating in a more stable state now

We continue to closely monitor the environment together with the third party to ensure stability is maintained and to quickly address any remaining impact.

Current Status:
Most platform functionalities should now be operating normally, while we complete our monitoring and stabilization efforts and waiting for the official update of our third party.


We will continue to monitor the situation and will provide another update once we confirm the platform is fully stable.

Follow the third party updates on:
https://ocistatus.oraclecloud.com/#/incidents/ocid1.oraclecloudincident.oc1.phx.amaaaaaavwew44aahmtvmmxusawis4baxh73alctlnntgb3jtf2lakwtp27a
Posted Mar 04, 2026 - 19:36 UTC

Update

Major update: Performance Improvement and Ongoing Monitoring

We would like to share an update regarding the platform instability affecting Pipefy.

Our engineering team has implemented mitigation measures that have significantly improved platform performance and stability. The service is currently operating in a more stable state compared to earlier during the incident.

We continue to closely monitor the environment together with the third party to ensure stability is maintained and to quickly address any remaining impact.

Current Status:
Most platform functionalities should now be operating normally, although some users may still experience occasional performance fluctuations while we complete our monitoring and stabilization efforts.

Data Integrity:
There is no indication of data loss, exposure, or compromise.

We will continue to monitor the situation and will provide another update once we confirm the platform is fully stable.

Next update in 2 hours.
Posted Mar 04, 2026 - 17:01 UTC

Update

Platform Instability

We are currently experiencing platform instability affecting Pipefy. Our team has identified that the issue is related to a network problem in the Oracle Cloud Infrastructure (OCI) region where our servers are hosted.

Due to this infrastructure issue, we are temporarily unable to allocate enough machines to keep our multi-tenant environment fully stable. As a result, some users may experience slowness or intermittent access to the platform.

What we are doing:
Our engineering team is actively working together with OCI to resolve the issue as quickly as possible. We are dedicating all necessary resources and efforts to restore full platform stability.

Impact:
Some customers may experience degraded performance or intermittent availability while the incident is ongoing. At this moment, there is no indication of data loss or data exposure.

Next Steps:
We will continue to monitor the situation closely and provide updates as progress is made. Once the incident is fully resolved, we will conduct a detailed Root Cause Analysis (RCA) and share the findings and preventive actions with our customers.

We sincerely apologize for the disruption and appreciate your patience and understanding while we work to restore normal service.

Next update: In 2 hours
Posted Mar 04, 2026 - 15:04 UTC

Update

We are continuing to work on a fix for this issue.
Posted Mar 04, 2026 - 14:51 UTC

Update

We identified the queue processing delays has been identified as an ongoing infrastructure issue on the side of our third-party provider, Oracle Cloud (OCI)
Posted Mar 04, 2026 - 14:04 UTC

Update

We identified the queue processing delays has been identified as an ongoing infrastructure issue on the side of our third-party provider, Oracle Cloud (OCI)
Posted Mar 04, 2026 - 14:03 UTC

Identified

Customers may notice that certain requests are taking longer than usual to process.
Posted Mar 04, 2026 - 13:54 UTC

Investigating

We are currently investigating this issue.
Posted Mar 04, 2026 - 13:34 UTC
This incident affected: Application and Features (Dashboards).