07-20-2014, 07:37 AM
(This post was last modified: 07-20-2014, 07:39 AM by redstonewarrior.)
As predicted, we were able to recover all data from the build server. (The directories were a bit jumbled from being disassociated from their parents, but who needs 'em.) The harddrive seems to be in decent condition, and we will be testing it soon. In the meantime, we may have build up for tomorrow.
Technical details:
There was an unknown failure point that caused the root file system on build to become available. The journal of the file system was not updated, yet directories were moved around / reallocated / disassociated. We may have lost some variable-data databases, but this does not affect build. As the snapshots directory and current map directory have frequently changing file children, they were affected by the failure. We currently have a heap of unnamed directories sitting in /mnt/pt1/lost+found, and I have managed to locate a backup within three hours of the failure. (I could do better, and may, but I'd call this a success over April.) There are a multitude of other directories, and using guesswork we would be able to reassemble the entire structure. For now, I'm pulling the bare minimum. The mentioned sufficient backup is in ~root/tmp2 mort.
The partition itself seems in okay order, aside from the potential database damage. (Some cleaning.)
It would be possible for me to copy over one of the backups into /servers/ORE and get build running in a minute's time. As build's server is offline while we assessed the situation, we'd have to wait for our host for this. For reasons of "let's not repeat this", it is more likely that we will be hosting on a fresh debian install on a different harddisk tomorrow. (While we thoroughly test the disk, to determine if it was the faultpoint.)
In the meantime, it seems people have been talking about a number of changes for ORE's servers:
* Possible merge of school and build.
* Updating and implementing OREUtilsV2 (
)
* ***Cross-server backups*** (Crontab -e time.)
* Revamping of survival.
You will probably see a slew of polls, discussions, and general chat about these endeavors soon.
We're sorry this happened, and we'll be back online soon!
(Don't worry, we'll keep the temporary map in multiverse :3)
(Forgive me for any inconsistencies, ramblings, and general mistakes in this post. It's rather late here, and in my paranoia, I prolonged the recovery task by hours. I've been at this since early, and could have wrapped it up for tea. Assuming I wasn't sidetracked by reddit.)
Technical details:
There was an unknown failure point that caused the root file system on build to become available. The journal of the file system was not updated, yet directories were moved around / reallocated / disassociated. We may have lost some variable-data databases, but this does not affect build. As the snapshots directory and current map directory have frequently changing file children, they were affected by the failure. We currently have a heap of unnamed directories sitting in /mnt/pt1/lost+found, and I have managed to locate a backup within three hours of the failure. (I could do better, and may, but I'd call this a success over April.) There are a multitude of other directories, and using guesswork we would be able to reassemble the entire structure. For now, I'm pulling the bare minimum. The mentioned sufficient backup is in ~root/tmp2 mort.
The partition itself seems in okay order, aside from the potential database damage. (Some cleaning.)
It would be possible for me to copy over one of the backups into /servers/ORE and get build running in a minute's time. As build's server is offline while we assessed the situation, we'd have to wait for our host for this. For reasons of "let's not repeat this", it is more likely that we will be hosting on a fresh debian install on a different harddisk tomorrow. (While we thoroughly test the disk, to determine if it was the faultpoint.)
In the meantime, it seems people have been talking about a number of changes for ORE's servers:
* Possible merge of school and build.
* Updating and implementing OREUtilsV2 (
)* ***Cross-server backups*** (Crontab -e time.)
* Revamping of survival.
You will probably see a slew of polls, discussions, and general chat about these endeavors soon.
We're sorry this happened, and we'll be back online soon!
(Don't worry, we'll keep the temporary map in multiverse :3)
(Forgive me for any inconsistencies, ramblings, and general mistakes in this post. It's rather late here, and in my paranoia, I prolonged the recovery task by hours. I've been at this since early, and could have wrapped it up for tea. Assuming I wasn't sidetracked by reddit.)