We’ve recovered the default Sidekiq queue and it has drained.
The overload was driven by repeated field-condition save mutations (Resolvers::FieldConditions::Save) on a single customer pipe (307245277, Copa Energia). Each condition created enqueued a Cache::PipeInvalidatorWorker job on the default queue, and for this pipe (≈25k cards) each job took ~80–110s, dominating the queue.
As mitigation, we scaled up the Core workers around 5:53pm -03 (America/Buenos_Aires). By ~6:17pm -03 (America/Buenos_Aires), Grafana showed default queue latency dropping back to ~0 and the backlog clearing.
The queue has drained, and high latency is no longer occurring. As aligned with the stakeholders, the incident is being closed.
No components marked as affected
Resolved
We’ve recovered the default Sidekiq queue and it has drained.
The overload was driven by repeated field-condition save mutations (Resolvers::FieldConditions::Save) on a single customer pipe (307245277, Copa Energia). Each condition created enqueued a Cache::PipeInvalidatorWorker job on the default queue, and for this pipe (≈25k cards) each job took ~80–110s, dominating the queue.
As mitigation, we scaled up the Core workers around 5:53pm -03 (America/Buenos_Aires). By ~6:17pm -03 (America/Buenos_Aires), Grafana showed default queue latency dropping back to ~0 and the backlog clearing.
The queue has drained, and high latency is no longer occurring. As aligned with the stakeholders, the incident is being closed.
Investigating
We are investigating the default queue latency above 30s.