We’re committed to being open and transparent about our major data loss incident that occurred on February 27, 2025. While mistakes can happen, we wanted to go in-depth on exactly what happened and what we’re doing to avoid issues like this in the future.
The incident:
During a routine “Ghost Server Cleanup” maintenance operation, our script’s job was to find servers that were already terminated and clean up their files. Instead, it mistakenly flagged a large group of active servers for deletion. This resulted in a data loss event.
This was not caused by a breach or any foul play but was due to catastrophic human error.
Who was affected:
A large number of customers had their files either deleted or server access disrupted as the result of locking. Some servers being flagged as “false positives” were only locked, meaning nothing for those groups was lost or overwritten.
Actions taken:
We locked all servers identified as potentially impacted and immediately contacted the affected customers. Then we started a full off-site restoration using remote backups kept at Backblaze, our primary backup storage provider.
Most of the restorations are completed and we’re now handling any lingering issues through support tickets. Our developers have verified that the majority of the servers that were impacted have been recovered and are actively working with support.
What do you need to do:
It’s important to us that your server is recovered properly. Please check your server, and make sure that your files are recognizable.
If you took local backups via a plugin or our backups page, those will be missing. Our pruning tool would have cleared those and our off-site backups don’t include doubled-up backups.
If you notice any discrepancies or missing data, then please contact our support team. Our developers are working on these requests with the highest priority.
How have we ensured this won’t happen again:
We’re tightening QA policies across the board at Apex, including mandatory use of our staging/dev environments, no matter how small or simple a hotfix/change is.
All destructive or disruptive maintenance scripts are now on hold for a thorough safety audit.
Lastly, we’re enforcing a “dry run” phase by default, meaning any scripts that could manipulate the state of live data will require multi-person approval before proceeding.
Special mentions:
We would like to give a massive thank you to Backblaze for their support and prompt response in lifting rate limits which opened the floodgates for our backup restorations. Their service was essential to restoring data quickly and we cannot recommend them enough for their personal and professional backup services.
Start Your Minecraft Server
Get started with your own minecraft server in 5 min and start trying out these great features.
