On November 19, 2024, at 18:00 UTC, we identified that CloudWorks integrations weren't processing as expected. This prevented integrations from completing in Anaplan Data Center — Netherlands.
Our initial investigation found that a CloudWorks component wasn't operating correctly. We completed a restart of the affected component and integrations started to complete successfully. However, as we were running service health checks, integrations stopped processing once more. After further review, we increased resource allocation to the CloudWorks service. This had a positive outcome, and integrations started to process. We closely monitored the service to ensure that integrations continued to process correctly. Full service was restored at 19:36 UTC.
We reviewed how this issue occurred. On November 13, we increased resource allocation in response to an incident to enable a backlog on CloudWorks integration to process. Post incident review, we assessed resource utilization for CloudWorks. It was determined that additional resources were no longer required. As such, on November 19, we had reduced the CloudWorks resource allocation back to its previous values. This reduction was committed to the code and issues were observed shortly thereafter. It initially appeared that the service required a restart to apply the resource change. However, the restart wasn't successful and ultimately the resources change had to reverted.
We have created a priority workstream for CloudWorks to improve the resiliency of the service. We have completed several improvements to the underlying infrastructure to enable vertical and horizontal growth. This has enabled us to increase the volume of requests that CloudWorks can process at any given time. This should also reduce recovery time on issues that impact CloudWorks and the need to increase resources allocation as part of the recovery process. We are closely monitoring the overall health of the service and initial findings are positive. We have also looked at code changes to how resources are allocated for CloudWorks. We have added the steps we need to take to prevent this from happening.
We apologize for any impact this issue may have had on your business operations. We are continuously strengthening our systems and procedures to ensure we avoid future disruptions to your business and users.
If you have any further questions or concerns, please contact Anaplan Customer Care. Thank you for your patience during this situation and thank you for being an Anaplan customer.