State Datacenter Failure

Published · Jun 23, 2025

affected systems: turn gps

severity: major-outage

resolved: yes

The TSOB data center had a file system failure. DTS DevOps have started an operation to resolve that issue. At the moment there appear to have 28 servers not responding. Once this issue is resolved, they should respond again.

Mon Jun 23 2025 18:20:43 GMT+0000 (Coordinated Universal Time)

DevOps are actively working with two vendor support teams to find a resolution. There is still no resolution ETA.

Mon Jun 23 2025 20:13:04 GMT+0000 (Coordinated Universal Time)

DevOps have identified the issue to a single component of an upgrade. The team is working to roll back. The latest ETA is between 1 to 2 hours.

Mon Jun 23 2025 23:57:24 GMT+0000 (Coordinated Universal Time)

DevOps are in the process of patching firmware on the affected environment. They are hoping this will gradually increase stablility. There is no longer an ETA.