tag:status.gameserverkings.com,2005:/historyGameserverkings Status - Incident History2024-03-29T05:24:00ZGameserverkingstag:status.gameserverkings.com,2005:Incident/203661732024-04-05T00:00:00Z2024-03-26T21:25:49ZGTT Planned Work - Reston, VA<p><strong>THIS IS A SCHEDULED EVENT Apr <var data-var='date'>5</var>, <var data-var='time'>00:00</var> - <var data-var='time'>04:00</var> UTC</strong></p><p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>21:25</var> UTC</small><br><strong>Scheduled</strong> - GTT is planning a scheduled maintenance during the below timeframe. Service interruption is possible. Maximum service interruption expected is 15 minutes.<br /><br />Start: 2024-04-05 00:00:00 GMT<br /><br />End: 2024-04-05 04:00:00 GMT<br /><br />Backup Start: 2024-04-09 00:00:00 GMT<br /><br />Backup End: 2024-04-09 04:00:00 GMT</p>tag:status.gameserverkings.com,2005:Incident/202302642024-03-26T21:26:07Z2024-03-26T21:26:07ZMass Disconnections in EU and US<p><small>Mar <var data-var='date'>26</var>, <var data-var='time'>21:26</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>11:12</var> UTC</small><br><strong>Monitoring</strong> - We have pushed fixes to all locations and are monitoring proactively to ensure all services fully stabilize and remain stable.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>08:26</var> UTC</small><br><strong>Update</strong> - We have several engineers working on pushing out a full fix ASAP.<br /><br />Updates will follow as we have them for you.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>05:03</var> UTC</small><br><strong>Identified</strong> - The issue has been narrowed down, and we are diligently working on a permanent and full fix. We hope to have a more positive update to share soon.</p><p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>02:44</var> UTC</small><br><strong>Investigating</strong> - We're currently investigating a similar disconnection event as last night. We're already working on a fix, and will update everyone soon. Thanks!</p>tag:status.gameserverkings.com,2005:Incident/202214332024-03-13T00:04:23Z2024-03-13T00:04:23ZMass Disconnections in EU and US<p><small>Mar <var data-var='date'>13</var>, <var data-var='time'>00:04</var> UTC</small><br><strong>Resolved</strong> - This incident is confirmed to be resolved. A post-mortem will follow shortly. We appreciate your patience during this time.</p><p><small>Mar <var data-var='date'>12</var>, <var data-var='time'>05:03</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating this incident. We are aware of some disconnects affecting a large number of users in the last 30 minutes.<br /><br />We will update ASAP.</p>tag:status.gameserverkings.com,2005:Incident/201512142024-03-04T02:43:27Z2024-03-04T02:43:27ZPacket Loss / Disconnects<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>02:43</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 3</var>, <var data-var='time'>18:33</var> UTC</small><br><strong>Investigating</strong> - Good Morning/Afternoon/Evening,<br /><br />We're aware of the packet loss events, and disconnects that are currently happening, and we've been looking into them. We're in the process of implementing some things, and we will have things calmed down ASAP. Thanks for your patience.</p>tag:status.gameserverkings.com,2005:Incident/200225242024-02-19T19:50:23Z2024-02-19T19:50:23ZAshburn outage<p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>19:50</var> UTC</small><br><strong>Resolved</strong> - There was an issue with the dark fiber connection from the network PoP in CoreSite towards our deployment PoP in Evoque. The issue was resolved within a few minutes of appearing and we do not expect further issues.</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>18:08</var> UTC</small><br><strong>Monitoring</strong> - We are monitoring for further issues whilst looking into the root of the issue.</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>17:52</var> UTC</small><br><strong>Identified</strong> - Service should be restored as of this update. We are looking into the root cause of the issue.</p><p><small>Feb <var data-var='date'>19</var>, <var data-var='time'>17:43</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating an outage in the Ashburn location.</p>tag:status.gameserverkings.com,2005:Incident/200053222024-02-17T06:05:13Z2024-02-17T06:05:13ZMoving AMS -> LON traffic to private transport capacity<p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>06:05</var> UTC</small><br><strong>Completed</strong> - Maintenance completed. Enjoy.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>06:04</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>06:04</var> UTC</small><br><strong>Scheduled</strong> - Maintenance completed. Enjoy.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>06:01</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Feb <var data-var='date'>17</var>, <var data-var='time'>05:16</var> UTC</small><br><strong>Scheduled</strong> - Hi everyone,<br /><br />We'll be moving our traffic between london and amsterdam (in both directions) over to private transport capacity rather than the tunneled system we've historically relied on.<br /><br />This should help keep things smooth during the busiest times of the day.<br /><br />No impact is expected while we route traffic over.</p>tag:status.gameserverkings.com,2005:Incident/199274712024-02-07T21:43:33Z2024-02-07T21:43:33ZOutage in DFW/VA<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>21:43</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. We will post a follow up post-mortem in the coming days. We do not expect further issues.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>09:13</var> UTC</small><br><strong>Update</strong> - Services have been stabilized. RFO will follow. <br /><br />After further research, dallas was not having any problems directly. It was traffic flowing from Ashburn to Dallas.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>05:26</var> UTC</small><br><strong>Monitoring</strong> - We have located the issue, and implemented a temporary fix for the issue. As of now, traffic previously routing through Ashburn has been diverted to the next closest pop for your user. We are working on a full scale solution, and will have an update ASAP.<br /><br />For now, things should be stable with slightly increased latency being possible. We will also be sharing a full scale RFO in the next few days.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>04:16</var> UTC</small><br><strong>Investigating</strong> - We are currently investigating the outage in DFW/VA for inbound traffic.</p>tag:status.gameserverkings.com,2005:Incident/199130202024-02-05T18:32:11Z2024-02-05T18:32:11ZDallas connectivity issues<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>18:32</var> UTC</small><br><strong>Resolved</strong> - We believe this incident has been resolved and we've setup additional monitoring for this specific tunnel.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>17:58</var> UTC</small><br><strong>Monitoring</strong> - We have implemented some changes and have not noticed a drop since they have been applied. We are continuing to monitor.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>17:21</var> UTC</small><br><strong>Identified</strong> - The issue has been identified to be affecting mostly European traffic towards Dallas servers. We have found that the tunnel between Amsterdam and Dallas is experiencing problems, and we are diligently working towards a fix.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>14:53</var> UTC</small><br><strong>Investigating</strong> - We have been made aware of some connectivity issues in Dallas and are investigating this issue.</p>tag:status.gameserverkings.com,2005:Incident/198881502024-02-01T23:39:08Z2024-02-01T23:39:08ZPacket loss issues<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>23:39</var> UTC</small><br><strong>Resolved</strong> - Packet loss issue has been identified and resolved, some of our ddos mitigation appliances were having some issues which have now been resolved.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>22:42</var> UTC</small><br><strong>Monitoring</strong> - The issue causing the packet loss has been identified and we've pushed a fix that should clear out the problem over the next 15-20 minutes.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>21:36</var> UTC</small><br><strong>Identified</strong> - We have identified some packet loss issues across multiple PoPs, mainly in Amsterdam but affecting other regions too, and are working on a resolution to this issue as soon as possible. Thank you for your patience.</p>tag:status.gameserverkings.com,2005:Incident/196341482024-01-30T02:17:19Z2024-01-30T02:17:19ZNew Status Alert<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>02:17</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>04:26</var> UTC</small><br><strong>Investigating</strong> - Dear customers,<br /><br />In an attempt to resolve the packet loss issues we've been seeing for the last few days in Dallas, we made some routing changes to address the issues based on the data we have, but unfortunately it caused some heavy BGP turbulence causing a small Disconnect on certain connections in Dallas.<br /><br />We do not expect any further disconnects, and we should hopefully have a fix on our end for most of the packet loss issues you've been seeing. Thanks for your patience.</p>tag:status.gameserverkings.com,2005:Incident/198303072024-01-25T17:05:40Z2024-01-25T17:05:40ZLondon connectivity issues<p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>17:05</var> UTC</small><br><strong>Resolved</strong> - GTT has stated that this incident has been resolved. We do expect any further interruptions. We apologize for the interruptions.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>16:24</var> UTC</small><br><strong>Monitoring</strong> - Services should be restored and operational again. We are monitoring the issue closely.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>16:20</var> UTC</small><br><strong>Identified</strong> - The issue has been identified as a GTT outage in the UK region.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>15:17</var> UTC</small><br><strong>Investigating</strong> - We are aware of issues with London connectivity and are investigating this problem urgently.</p>tag:status.gameserverkings.com,2005:Incident/195064892023-12-22T14:12:21Z2023-12-22T14:12:21ZLondon switch failure<p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>14:12</var> UTC</small><br><strong>Resolved</strong> - Everything has been patched over to the new device. If you are still experiencing any issues please don't hesitate to contact us via your preferred support channel.<br /><br />Anyone affected by this issue should no longer experience these issues. We greatly apologize for the inconvenience and hope everyone has a happy holiday season.</p><p><small>Dec <var data-var='date'>22</var>, <var data-var='time'>13:09</var> UTC</small><br><strong>Identified</strong> - We have acknowledged that a problematic switch in one of the cabinets in London has failed. We have a new device racked and ready and we are beginning to patch everything over. We expect this incident to fully resolved within the hour.</p>tag:status.gameserverkings.com,2005:Incident/194729332023-12-19T05:57:01Z2023-12-19T05:57:01ZNew Status Alert<p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>05:57</var> UTC</small><br><strong>Resolved</strong> - As of around 2 hours ago services have been restored. We've been monitoring since then and we do not expect further issues. We remarking this as resolved.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>02:54</var> UTC</small><br><strong>Monitoring</strong> - Fixes have been implemented and services are stable for around 45 minutes now. We will continue updating as we monitor the situation.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>01:06</var> UTC</small><br><strong>Update</strong> - We are approaching the final stages of our maintenance. We expect to be fully operational and stable soon.<br /><br />Thanks for your patience, and we again apologize for the inconvenience.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>21:58</var> UTC</small><br><strong>Update</strong> - At this current time, I would not inform your clients, players, etc.. that services are stable or online. We are still performing critical maintenance and do not want to confirm services are stable at this current time. As always, we will update here when services are confirmed to be fully restored and stable.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>21:10</var> UTC</small><br><strong>Update</strong> - Services are in the process of being restored. Most clients should now be online, but some are still down and we are working on them. We apologize for the continued downtime.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>20:03</var> UTC</small><br><strong>Identified</strong> - We're still working diligently to bring your services online. We are performing emergency maintenance to the affected networking gear, and working with all hands on deck to quickly and efficiently restore your services. We appreciate your patience, and thank you for your understanding.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>18:43</var> UTC</small><br><strong>Investigating</strong> - We're aware some of you are experiencing an outage in Dallas. This is related to the earlier maintenance which was posted at https://status.gameserverkings.com/. Unfortunately, that task ended and will not allow us to open it back up, but maintenance is still ongoing. The team is on it, and we should have an update and get the affected clients back online shortly. Thanks for the patience.</p>tag:status.gameserverkings.com,2005:Incident/194711942023-12-18T18:05:22Z2023-12-18T18:05:22ZEmergency Maintenance<p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>18:05</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>15:06</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>15:02</var> UTC</small><br><strong>Update</strong> - We will be undergoing scheduled maintenance during this time.</p><p><small>Dec <var data-var='date'>18</var>, <var data-var='time'>15:01</var> UTC</small><br><strong>Scheduled</strong> - Hi everyone,<br /><br />We're performing emergency maintenance on some of our Dallas network equipment.<br /><br />We're expecting to perform the maintenance gracefully, however please be aware there might be brief interruptions/drops to your services. <br /><br />We're working as hard as we can to wrap this up as soon as possible.</p>tag:status.gameserverkings.com,2005:Incident/194020842023-12-11T17:39:05Z2023-12-11T17:39:05ZLondon cabinet switch issue<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>17:39</var> UTC</small><br><strong>Resolved</strong> - Everything is back online and operational. We will be scheduling a period to replace this device entirely within the next couple days. We will provide a timeframe as soon as we have one.<br /><br />Thank you for your patience & understanding.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>16:40</var> UTC</small><br><strong>Monitoring</strong> - Services have been restored. We are currently monitoring and checking for any further impact.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>16:31</var> UTC</small><br><strong>Update</strong> - We have regained access to the device and are restoring it to a usable state.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>15:29</var> UTC</small><br><strong>Identified</strong> - We have confirmed that the top of rack switch in one specific London cabinet is currently offline. This is causing a service interruption for any servers connected to this switch.<br /><br />We have identified the issue and are taking the necessary steps to restore service. Further updates will be posted as soon as they are available.</p>tag:status.gameserverkings.com,2005:Incident/193629112023-12-07T07:03:02Z2023-12-07T07:03:02ZElevated Latency on US-Europe traffic<p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>07:03</var> UTC</small><br><strong>Resolved</strong> - Our carriers seem to have routed around the problem with minimal latency. We're marking this incident as closed.</p><p><small>Dec <var data-var='date'> 7</var>, <var data-var='time'>03:34</var> UTC</small><br><strong>Investigating</strong> - Hi everyone,<br /><br />We're currently experiencing elevated latencies on traffic going between Europe and the U.S.<br />Our NOC is actively engaging with our carriers to figure out what's going on, as well as an ETA on a remedy.</p>tag:status.gameserverkings.com,2005:Incident/191019162023-11-14T01:30:49Z2023-11-14T01:30:49ZConfiguration Change in London<p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>01:30</var> UTC</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Nov <var data-var='date'>14</var>, <var data-var='time'>01:00</var> UTC</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Nov <var data-var='date'>12</var>, <var data-var='time'>22:31</var> UTC</small><br><strong>Scheduled</strong> - We are planning for a scheduled maintenance at 1AM GMT on Tuesday, November 14th. This maintenance will not last more than 30 minutes. It could cause a flap and disconnection of traffic.</p>tag:status.gameserverkings.com,2005:Incident/188330282023-10-18T00:14:18Z2023-10-18T00:14:18ZNew Status Alert<p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>00:14</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. The root cause was a software lockup on our switch which was caused by a bug in the vendor software. We are reporting this failure to the switch vendor. We do not foresee any further issues.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>23:57</var> UTC</small><br><strong>Investigating</strong> - Around 5 minutes ago our internal monitoring detected one of our switches in Dallas was entering into a failed state. We immediately began trying to recover the switch. We have a technician en route, and we will update again shortly. This will not effect everyone, just clients specifically connected to this switch. We will update shortly.</p>tag:status.gameserverkings.com,2005:Incident/187679502023-10-12T17:49:09Z2023-10-12T17:49:09ZNew Status Alert<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>17:49</var> UTC</small><br><strong>Resolved</strong> - We apologize for the inconvenience caused by the incident that took place from 13:20 EST to 13:28 EST. This issue arose following a routine configuration change to our mitigation appliances, which led to unintended side-effects.<br /><br />We detected and resolved the problem promptly:<br />By 13:23, we implemented a solution.<br />By 13:26, connections began to stabilize.<br />By 13:28, we restored full connectivity.<br /><br />Our engineering team is now analyzing this incident in our test environment to prevent any recurrence. We’ve made the necessary corrections and don’t anticipate further disruptions. Users previously experiencing issues with Rust+ connectivity should now find everything in order.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>17:28</var> UTC</small><br><strong>Investigating</strong> - We're aware and investigating.</p>tag:status.gameserverkings.com,2005:Incident/187602542023-10-12T03:01:26Z2023-10-12T03:01:26ZNew Status Alert<p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>03:01</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved. No further impact expected.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>01:55</var> UTC</small><br><strong>Monitoring</strong> - We have implemented a remedy to fix this. We do not believe there to be any further issues, but we are monitoring for such.</p><p><small>Oct <var data-var='date'>12</var>, <var data-var='time'>01:36</var> UTC</small><br><strong>Investigating</strong> - We're aware of the incident in London and are working on it. We'll update ASAP. Thanks for your patience.</p>tag:status.gameserverkings.com,2005:Incident/185658312023-09-21T13:35:59Z2023-09-21T13:35:59ZIncident in DFW<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>13:35</var> UTC</small><br><strong>Resolved</strong> - This was isolated to a single rack cabinet, and services are back online.</p><p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>12:27</var> UTC</small><br><strong>Investigating</strong> - Hi everyone,<br /><br />We're investigating an incident in Dallas, we'll update you once we know more.</p>tag:status.gameserverkings.com,2005:Incident/185670412023-09-21T07:00:00Z2023-09-21T14:48:51ZGTT Outage Caused by Maintenance<p><small>Sep <var data-var='date'>21</var>, <var data-var='time'>07:00</var> UTC</small><br><strong>Resolved</strong> - Early this morning around 03:15 EST / 07:15 UTC, GTT had begun planned maintenance in the Dallas area. This planned maintenance was not supposed to have an effect on our services, but it unfortunately did affect clients traffic in the Dallas area.<br /><br />The total maintenance period was from 07:00 to 11:00 wait an impact period in the first 90 minutes.<br /><br />The total downtime we recorded lasted around 8 minutes from 07:15 UTC to 07:23 UTC. We do not expect there to be any further issues. If there are any more planned maintenances, we will be sure to work with GTT to get proper details of how we will be affected by them.<br /><br />We apologize for the inconvenience.</p>tag:status.gameserverkings.com,2005:Incident/184790382023-09-12T17:24:33Z2023-09-12T17:24:33ZLondon incident<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>17:24</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>11:22</var> UTC</small><br><strong>Update</strong> - Services are stabilizing, no further impact expected.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>11:21</var> UTC</small><br><strong>Investigating</strong> - Hi everyone,<br /><br />We’re aware of repeated impact in London, we’re rolling out a permanent fix as we speak.<br /><br />We’ll update you shortly.</p>tag:status.gameserverkings.com,2005:Incident/184781012023-09-12T09:28:16Z2023-09-12T09:28:16ZConnectivity issues in London<p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>09:28</var> UTC</small><br><strong>Resolved</strong> - Back to normal. Our apologies for the inconvenience.</p><p><small>Sep <var data-var='date'>12</var>, <var data-var='time'>09:22</var> UTC</small><br><strong>Monitoring</strong> - Hi everyone,<br /><br />We just had brief connectivity issues in London, a fix has been implemented and we are monitoring now.</p>tag:status.gameserverkings.com,2005:Incident/182120632023-09-07T16:44:28Z2023-09-07T16:44:28ZIntermittent Packet Loss When Routing Over NTT<p><small>Sep <var data-var='date'> 7</var>, <var data-var='time'>16:44</var> UTC</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>21</var>, <var data-var='time'>16:47</var> UTC</small><br><strong>Identified</strong> - We are currently, and have been aware of the reports about intermittent packet loss throughout the day, and mostly during peak times for players routing over NTT (one of our upstreams).<br /><br />NTT has confirmed the issue to be on their end, and have said it's due to a major subsea cable outage causing this issue in numerous locations. They have said their estimated time for resolution will be on the 25th (Friday). We've been in contact with them daily, and will continue to remain in contact until this is resolved.<br /><br />We'll update this page with any major updates. Thanks for your patience and understanding.</p>