During an emergency update to our DDoS mitigation systems, an error caused the automated rollback functionality to fail. As a result, the systems entered a continuous boot loop and, due to limited connectivity, were unable to restore the latest version automatically. What would typically be a five-second operation took over 15 minutes as we manually restored the systems.
Although we normally do not perform updates on weekends, a critical bug required an immediate fix to prevent potential issues later today. We appreciate your understanding. The situation has now been resolved, and no further impact is expected over the remainder of the weekend.
Thank you.
Posted Feb 15, 2025 - 17:18 UTC
Monitoring
We are monitoring for further issues but do not expect any further impact.
Posted Feb 15, 2025 - 17:08 UTC
Update
The Amsterdam PoP has been restored.
Posted Feb 15, 2025 - 17:08 UTC
Update
The London, UK PoP has been restored.
Posted Feb 15, 2025 - 17:01 UTC
Identified
The issue has been identified and a fix has been pushed to restore the Ashburn, VA location.
More updates to follow on the other PoPs.
Posted Feb 15, 2025 - 17:00 UTC
Investigating
Hi all,
We are aware of and investigating an issue that is affecting connections to multiple PoPs, including but not limited to: - Ashburn, VA - Amsterdam - London, UK
We'll update as soon as possible. Thank you for your patience.