Platform Alerts
Incident Report for Anaplan
Postmortem

On November 19, 2024, at 18:00 UTC, we identified that CloudWorks integrations weren't processing as expected. This prevented integrations from completing in Anaplan Data Center — Netherlands.  

Our initial investigation found that a CloudWorks component wasn't operating correctly. We completed a restart of the affected component and integrations started to complete successfully. However, as we were running service health checks, integrations stopped processing once more. After further review, we increased resource allocation to the CloudWorks service. This had a positive outcome, and integrations started to process. We closely monitored the service to ensure that integrations continued to process correctly. Full service was restored at 19:36 UTC.  

We reviewed how this issue occurred. On November 13, we increased resource allocation in response to an incident to enable a backlog on CloudWorks integration to process. Post incident review, we assessed resource utilization for CloudWorks. It was determined that additional resources were no longer required. As such, on November 19, we had reduced the CloudWorks resource allocation back to its previous values. This reduction was committed to the code and issues were observed shortly thereafter. It initially appeared that the service required a restart to apply the resource change. However, the restart wasn't successful and ultimately the resources change had to reverted.  

We have created a priority workstream for CloudWorks to improve the resiliency of the service. We have completed several improvements to the underlying infrastructure to enable vertical and horizontal growth. This has enabled us to increase the volume of requests that CloudWorks can process at any given time. This should also reduce recovery time on issues that impact CloudWorks and the need to increase resources allocation as part of the recovery process. We are closely monitoring the overall health of the service and initial findings are positive. We have also looked at code changes to how resources are allocated for CloudWorks. We have added the steps we need to take to prevent this from happening.  

We apologize for any impact this issue may have had on your business operations. We are continuously strengthening our systems and procedures to ensure we avoid future disruptions to your business and users.  

If you have any further questions or concerns, please contact Anaplan Customer Care. Thank you for your patience during this situation and thank you for being an Anaplan customer.

Posted Nov 26, 2024 - 22:08 UTC

Resolved
We have confirmed that the issue is now resolved.

We deeply apologize for any impact this issue may have caused. We appreciate your patience and partnership as we worked through this issue.

We will follow up within 7 business days with a detailed root cause analysis (RCA) that will be shared on our Status Page. If you have any question or concerns, please do not hesitate to contact us at Anaplan Support.
Posted Nov 19, 2024 - 20:14 UTC
Monitoring
Service has now been restored; you should now be able to resume normal activities.

We will continue to monitor the platform to ensure no additional issues arise. If you have any questions, concerns, or continue to experience issues, please do not hesitate to contact Anaplan Support. We will provide a final update to you when we consider this situation fully resolved.
Posted Nov 19, 2024 - 19:36 UTC
Update
Thank you for your patience as we continue to investigate this issue. We are seeing some integrations complete; however, overall performance is degraded. We are currently engaged in a deep dive of the infrastructure that underpins the service.

Currently, we do not yet have a time to resolution. We will continue to provide updates every 30 minutes as we work to resolve this issue as quickly as possible.
Posted Nov 19, 2024 - 19:13 UTC
Investigating
We are seeing reoccurrence of slowness in CloudWorks integrations.

Currently, we do not yet have a time to resolution. We will continue to provide updates every 30 minutes as we work to resolve this issue as quickly as possible.
Posted Nov 19, 2024 - 18:43 UTC
Monitoring
Service has now been restored; you should now be able to resume normal activities.

We will continue to monitor the platform to ensure no additional issues arise. If you have any questions, concerns, or continue to experience issues, please do not hesitate to contact Anaplan Support. We will provide a final update to you when we consider this situation fully resolved.
Posted Nov 19, 2024 - 18:25 UTC
Investigating
We are currently investigating an issue impacting customers’ ability to run Cloudworks integrations.

We are working to resolve this issue as quickly as possible and will provide updates every 30 minutes or upon resolution.
Posted Nov 19, 2024 - 18:14 UTC
This incident affected: Anaplan Data Center - Netherlands.