Platform Alerts
Incident Report for Anaplan
Postmortem

On November 20, 2024, at 10:50 UTC, we received alerts indicating intermittent hardware performance issues. These issues resulted in CloudWorks integrations running slowly in Anaplan Data Center — Netherlands.

Our initial investigation identified intermittent delayed responses between a network component and the backend infrastructure. In parallel, we detected intermittent network communication delays causing CloudWorks integrations to queue without processing. After a comprehensive assessment, we discovered several hardware resources were unresponsive to network requests. To resolve the issue, we strategically isolated the problematic server from the network. This successfully restored connectivity at 12:18 UTC.

By 12:33 UTC, our CloudWorks service functionality had returned to normal operational status. However, due to earlier hardware performance issues, a backlog of integrations remained pending. We continued monitoring the queued backlog integrations until it was fully cleared at 13:06 UTC, marking the complete restoration of our CloudWorks services.

After a thorough investigation into the network issues, we identified that an incremental data copy process had overwhelmed our network infrastructure. A sub-optimal network path between the network and infrastructure caused network link saturation, resulting in network congestion that directly impacted CloudWorks integrations. In response, we have updated the network path to significantly increase throughput and reduce congestion risks. We are also reassessing our data copy processes to minimize future network performance challenges.

We have also created a priority workstream for CloudWorks to improve the resiliency of the service. We have completed several improvements to the underlying infrastructure to enable vertical and horizontal growth. This has enabled us to increase the volume of requests that CloudWorks can process at any given time. This should also reduce recovery time on issues that impact CloudWorks and the need to increase resources allocation as part of the recovery process. We are closely monitoring the overall health of the service and initial findings are positive.

We apologize for any impact this issue may have had on your business operations. We are continuously strengthening our systems and procedures to ensure we avoid future disruptions to your business and users.

If you have any further questions or concerns, please contact Anaplan Customer Care. Thank you for your patience during this situation and thank you for being an Anaplan customer.

Posted Nov 28, 2024 - 13:59 UTC

Resolved
We have confirmed that the issue is now resolved.

We deeply apologize for any impact this issue may have caused. We appreciate your patience and partnership as we worked through this issue.

We will follow up within 7 business days with a detailed root cause analysis (RCA) that will be shared on our Status Page. If you have any question or concerns, please do not hesitate to contact us at Anaplan Support.
Posted Nov 20, 2024 - 13:06 UTC
Update
General CloudWorks performance has improved and the service now appears to be functioning as expected. Integration back log continues to take time to clear back to normal levels and we are closely monitor activity until it is complete.

We will provide an update in 30 minutes or upon resolution.
Posted Nov 20, 2024 - 12:33 UTC
Update
We are aware of an issue impacting customers’ ability to run CloudWorks integrations in the following region:
Anaplan Data Center - Netherlands

We continue to investigate the service and corresponding network and infrastructure. Currently, we do not yet have a time to resolution. We will continue to provide updates every 30 minutes as we work to resolve this issue as quickly as possible.
Posted Nov 20, 2024 - 12:08 UTC
Investigating
We are currently investigating an issue impacting customers’ ability to run Cloudworks integrations.

We are working to resolve this issue as quickly as possible and will provide updates every 30 minutes or upon resolution.
Posted Nov 20, 2024 - 11:36 UTC
This incident affected: Anaplan Data Center - Netherlands.