On June 3, 2026, between approximately 20:00 and 23:33 UTC, customers using workspaces in the us5: Cloud – US East region experienced intermittent difficulties opening workspaces and accessing models. During this time, some workspaces appeared stuck on the loading screen, and model requests were not completing.
Root cause
During a planned infrastructure update in the us5: Cloud – US East region, a configuration issue caused the number of active processing components to temporarily exceed intended limits. This placed an unexpected load on our network infrastructure, exhausting available connectivity between our systems and the services they depend on. As a result, model requests could not be completed, and some workspaces appeared unresponsive.
Recovery
Our engineering team identified the issue and began targeted remediation, working to reduce the load on our processing infrastructure and restore available network capacity. As these changes took effect, connectivity gradually restored across our systems. By 23:33 UTC, the issue was fully resolved.
Corrective and preventative actions
We are implementing the following improvements to prevent similar disruptions:
We are revising our infrastructure update procedures to ensure processing capacity increases are completed sequentially. This prevents unintended loads on our network and reduces the risk of similar disruptions.
We are implementing enhanced monitoring and alerting for network capacity and resource utilization across our cloud regions. This means our engineering team can detect and respond to capacity pressure earlier, before it affects customers.
We are increasing the reserve capacity available within our processing infrastructure. This provides a buffer against unexpected load increases and gives our engineering team more time to respond when similar conditions arise.
Closing
We apologize for the impact this issue has had on your operations. We’re committed to the improvements outlined above to prevent similar disruptions. If you have questions or concerns, please contact our Support team.