Systems API Database Corrupted
Incident Report for The Fuel Rats
Resolved
SAPI database has been partially reconstructed from recovered data and EDSM dumps, and EDDN updates are being recorded.
Posted Oct 07, 2023 - 21:05 UTC
Update
System API is unavailable due to data corruption, this means mecha star system information is unavailable
Posted Oct 03, 2023 - 21:18 UTC
Update
System API is unavailable due to data corruption, this means mecha star system information is unavailable
Posted Oct 03, 2023 - 21:16 UTC
Monitoring
All systems have been restored to working order, we are monitoring smooth operations.
Posted Oct 03, 2023 - 21:11 UTC
Update
We are continuing to work on a fix for this issue.
Posted Oct 03, 2023 - 20:57 UTC
Update
We are continuing to work on a fix for this issue.
Posted Oct 03, 2023 - 20:48 UTC
Update
We are continuing to work on a fix for this issue.
Posted Oct 03, 2023 - 20:40 UTC
Update
The disk was replaced and services are being restored.
Posted Oct 03, 2023 - 20:35 UTC
Identified
The disk is being replaced by the hosting provider
Posted Oct 03, 2023 - 19:55 UTC
Investigating
A disk is starting to fail on our main server. Services running on the affected disks are being shut down. Other services on the machine might see degraded performance while we wait for a disk replacement. Downtime on the machine and all services will be necessary later to swap the disk. We will update as soon as we have more information.
Posted Oct 03, 2023 - 17:56 UTC
This incident affected: Atlassian (Confluence, JIRA), API (Systems API), IRC (IRC Services, Mechasqueak, Drillsqueak, EU IRC Server), and Web (Rat Kiwi, RodentBin, FuelRats Cloud, Grafana, Queue System).