Possible Failiure of Build Server

[Xeomorpher]

We are currently experiencing severe issues with the build server, which may or may not be a catastrophic failure of the hard drive.

We will be keeping everyone as informed as possible about this issue, and request your patience in dealing with this possibly extended downtime.

We honestly have no fucking clue exactly what is wrong with build as of yet. It may be fixed by magic upon a restart, or it may require people in lab coats and a class 10 cleanroom to fix.

[Iceglade]

What is the deal with school?

[PNWMan]

Where is everyone located at the time? Survival is dead (not too surprisingly) and I highly doubt Stym's is the backup location.

[Xeomorpher]

So, I honestly have no clue what has happened to school; I'm not staff anymore. We do have another issue however. Mort will not be back at his home to fix anything for multiple weeks.

A temporary server will be set up asap.

[newomaster]

;_; We have backups of the world on other hard drives... right?

[tyler569]

What is the deal with school?

The school's physical server appears to be offline, survival and school are on the same network, and I can SSH to survival, but school's not responding.

[jxu]

Having a ORE member host has always been an unreliable and laggy solution. If there really isn't backup of the worlds, that is a huge oversight by the staff and I would ask for a big change in the current hosting solution.

[PhysoniumI]

As soon as I fix something I can get a temporary server up. 20/20 up/down US based. Can allot 8+ GB ram on a 3770k (OC'd to 4.5GHz) so lag should not be a problem. I'd need the server files and everything could get up.

EDIT: I'll throw WE into a forced //fast mode so there's less network load to speed up everything for everyone. World Edit would be slightly more work to use, but for a less-laggy experience, it'd be required.

[mort96]

Sadly, I'm not at home atm, nor will I be for some weeks. I just got someone to go over there and reboot the server, to no avail. Sunday, I'll get another guy over there and we'll try to see what can be done. Fingers crossed that we'll be able to save the data and be back online in not too long.

*Note to self: Backups.

[LordDecapo]

Guys, i have a desktop that i have set up to run a temp Server if you would like, my internet has a great up-time as well as good speed. let me know.

[PhysoniumI]

What is "good"? Also, based on Mort's situation of being away, it would have to be a rather extended temporary situation. If there are still copies of the map, (that are relatively recent) a new hosting situation is called for. Being unable to physically interface with a large-scale server like this is unacceptable and causes large amounts of problems in the long-run (people leaving, etc).

[LordDecapo]

the desktop has 8GB RAM, 3.1ghz tri-core (cause the 4th failed quailty tests,,, but it runs great!), and it would be plugged straight into the router, and dedicated to only running this temp server, i also get 50/50mps internet and can temp upgrade if needed.

[jxu]

*Note to self: Backups.

You shouldn't even have to think about making a backup.

[Ntwede]

This is interesting. My perpetual calendar CPU was backed up 4 times, and 3 failed in the past week-ish. My hard drive was overwritten by Ubuntu, my usb drive died, schematics/the server are down, but somehow dropbox survived. Better make more copies soon.

[Cutlassw30]

Well fuck.

[Cutlassw30]

Well what is the current status?

[Nickster258]

I offer my hosting capabilities to you guys.

Pauly's modded server does not take up much at all and it has plenty of room for hosting something such as the build.

Just an offer, you guys can think about it.

Also, I find it necessary to include specs:

Phenom II X6 - 6 Core @ 2.7Ghz w/ 9MB cache.
16GB DDR3 1333Mhz RAM
Reliable internet with good uptime.
Battery backup that will supply power to the server for around 30 minutes in case of a power outage.

[EDevil]

Thank you all for the feedback and the possibilities of setting up a temp. server. However, i think we have enough entries + the already existing list of possible hosts the staff has. The staff team will discuss this further on.

[Neogreenyew]

Yo Mort u nub, get me any error information or any symptoms of failures and get back to me. I'll try to help fix this shit for you. And yes this kind of stuff is my actual job so I'm not just some scrub that thinks he can fix a computer by hitting it with a bat.

[LordDecapo]

Yo Mort u nub, get me any error information or any symptoms of failures and get back to me. I'll try to help fix this shit for you. And yes this kind of stuff is my actual job so I'm not just some scrub that thinks he can fix a computer by hitting it with a bat.

Oh please Neo, we knowz u Iz r skrubstastic lol jk jk <3

[greatgamer34]

Guys lets stay on topic please!

[LordDecapo]

My comment was only one off topic. And it was a 1liner..

[greatgamer34]

It wasnt directed at anyone in specific, please dont make assumptions.

We have plenty of server hosts available. We are working to resolve the issue as fast as possible! :3

[redstonewarrior]

As predicted, we were able to recover all data from the build server. (The directories were a bit jumbled from being disassociated from their parents, but who needs 'em.) The harddrive seems to be in decent condition, and we will be testing it soon. In the meantime, we may have build up for tomorrow.

Technical details:
There was an unknown failure point that caused the root file system on build to become available. The journal of the file system was not updated, yet directories were moved around / reallocated / disassociated. We may have lost some variable-data databases, but this does not affect build. As the snapshots directory and current map directory have frequently changing file children, they were affected by the failure. We currently have a heap of unnamed directories sitting in /mnt/pt1/lost+found, and I have managed to locate a backup within three hours of the failure. (I could do better, and may, but I'd call this a success over April.) There are a multitude of other directories, and using guesswork we would be able to reassemble the entire structure. For now, I'm pulling the bare minimum. The mentioned sufficient backup is in ~root/tmp2 mort.
The partition itself seems in okay order, aside from the potential database damage. (Some cleaning.)
It would be possible for me to copy over one of the backups into /servers/ORE and get build running in a minute's time. As build's server is offline while we assessed the situation, we'd have to wait for our host for this. For reasons of "let's not repeat this", it is more likely that we will be hosting on a fresh debian install on a different harddisk tomorrow. (While we thoroughly test the disk, to determine if it was the faultpoint.)

In the meantime, it seems people have been talking about a number of changes for ORE's servers:
* Possible merge of school and build.
* Updating and implementing OREUtilsV2 ( )
* ***Cross-server backups*** (Crontab -e time.)
* Revamping of survival.
You will probably see a slew of polls, discussions, and general chat about these endeavors soon.
We're sorry this happened, and we'll be back online soon!
(Don't worry, we'll keep the temporary map in multiverse :3)

(Forgive me for any inconsistencies, ramblings, and general mistakes in this post. It's rather late here, and in my paranoia, I prolonged the recovery task by hours. I've been at this since early, and could have wrapped it up for tea. Assuming I wasn't sidetracked by reddit.)

[PhysoniumI]

Great job RSW!

[Guy1234567890]

REEEED <33333

[Nuuppanaani]

Awesome Red! You rock <3