+- Forums - Open Redstone Engineers (https://forum.openredstone.org)
+-- Forum: Announcements (https://forum.openredstone.org/forum-102.html)
+--- Forum: Announcements (https://forum.openredstone.org/forum-7.html)
+--- Thread: Possible Failiure of Build Server (/thread-4027.html)
Possible Failiure of Build Server - Xeomorpher - 07-11-2014
We are currently experiencing severe issues with the build server, which may or may not be a catastrophic failure of the hard drive.
We will be keeping everyone as informed as possible about this issue, and request your patience in dealing with this possibly extended downtime.
We honestly have no fucking clue exactly what is wrong with build as of yet. It may be fixed by magic upon a restart, or it may require people in lab coats and a class 10 cleanroom to fix.
RE: Possible Failiure of Build Server - Iceglade - 07-11-2014
What is the deal with school?
RE: Possible Failiure of Build Server - PNWMan - 07-11-2014
Where is everyone located at the time? Survival is dead (not too surprisingly) and I highly doubt Stym's is the backup location.
RE: Possible Failiure of Build Server - Xeomorpher - 07-11-2014
So, I honestly have no clue what has happened to school; I'm not staff anymore. We do have another issue however. Mort will not be back at his home to fix anything for multiple weeks.
A temporary server will be set up asap.
RE: Possible Failiure of Build Server - newomaster - 07-11-2014
;_; We have backups of the world on other hard drives... right?
RE: Possible Failiure of Build Server - tyler569 - 07-11-2014
(07-11-2014, 07:04 PM)Iceglade Wrote: What is the deal with school?
The school's physical server appears to be offline, survival and school are on the same network, and I can SSH to survival, but school's not responding.
RE: Possible Failiure of Build Server - jxu - 07-11-2014
Having a ORE member host has always been an unreliable and laggy solution. If there really isn't backup of the worlds, that is a huge oversight by the staff and I would ask for a big change in the current hosting solution.
RE: Possible Failiure of Build Server - PhysoniumI - 07-11-2014
As soon as I fix something I can get a temporary server up. 20/20 up/down US based. Can allot 8+ GB ram on a 3770k (OC'd to 4.5GHz) so lag should not be a problem. I'd need the server files and everything could get up.
EDIT: I'll throw WE into a forced //fast mode so there's less network load to speed up everything for everyone. World Edit would be slightly more work to use, but for a less-laggy experience, it'd be required.
RE: Possible Failiure of Build Server - mort96 - 07-11-2014
Sadly, I'm not at home atm, nor will I be for some weeks. I just got someone to go over there and reboot the server, to no avail. Sunday, I'll get another guy over there and we'll try to see what can be done. Fingers crossed that we'll be able to save the data and be back online in not too long.
*Note to self: Backups.
RE: Possible Failiure of Build Server - LordDecapo - 07-12-2014
Guys, i have a desktop that i have set up to run a temp Server if you would like, my internet has a great up-time as well as good speed. let me know.
RE: Possible Failiure of Build Server - PhysoniumI - 07-12-2014
What is "good"? Also, based on Mort's situation of being away, it would have to be a rather extended temporary situation. If there are still copies of the map, (that are relatively recent) a new hosting situation is called for. Being unable to physically interface with a large-scale server like this is unacceptable and causes large amounts of problems in the long-run (people leaving, etc).
RE: Possible Failiure of Build Server - LordDecapo - 07-12-2014
the desktop has 8GB RAM, 3.1ghz tri-core (cause the 4th failed quailty tests,,, but it runs great!), and it would be plugged straight into the router, and dedicated to only running this temp server, i also get 50/50mps internet and can temp upgrade if needed.
RE: Possible Failiure of Build Server - jxu - 07-12-2014
(07-11-2014, 11:33 PM)mort96 Wrote: *Note to self: Backups.
You shouldn't even have to think about making a backup.
RE: Possible Failiure of Build Server - Ntwede - 07-12-2014
This is interesting. My perpetual calendar CPU was backed up 4 times, and 3 failed in the past week-ish. My hard drive was overwritten by Ubuntu, my usb drive died, schematics/the server are down, but somehow dropbox survived. Better make more copies soon.
RE: Possible Failiure of Build Server - Cutlassw30 - 07-12-2014
Well fuck.
RE: Possible Failiure of Build Server - Cutlassw30 - 07-12-2014
Well what is the current status?
RE: Possible Failiure of Build Server - Nickster258 - 07-12-2014
I offer my hosting capabilities to you guys.
Pauly's modded server does not take up much at all and it has plenty of room for hosting something such as the build.
Just an offer, you guys can think about it.
Also, I find it necessary to include specs:
Phenom II X6 - 6 Core @ 2.7Ghz w/ 9MB cache.
16GB DDR3 1333Mhz RAM
Reliable internet with good uptime.
Battery backup that will supply power to the server for around 30 minutes in case of a power outage.
![[Image: 3620689286.png]](http://www.speedtest.net/result/3620689286.png)
RE: Possible Failiure of Build Server - EDevil - 07-12-2014
Thank you all for the feedback and the possibilities of setting up a temp. server. However, i think we have enough entries + the already existing list of possible hosts the staff has. The staff team will discuss this further on.
RE: Possible Failiure of Build Server - Neogreenyew - 07-14-2014
Yo Mort u nub, get me any error information or any symptoms of failures and get back to me. I'll try to help fix this shit for you. And yes this kind of stuff is my actual job so I'm not just some scrub that thinks he can fix a computer by hitting it with a bat.
RE: Possible Failiure of Build Server - LordDecapo - 07-15-2014
(07-14-2014, 06:33 PM)Neogreenyew Wrote: Yo Mort u nub, get me any error information or any symptoms of failures and get back to me. I'll try to help fix this shit for you. And yes this kind of stuff is my actual job so I'm not just some scrub that thinks he can fix a computer by hitting it with a bat.Oh please Neo, we knowz u Iz r skrubstastic lol jk jk <3
RE: Possible Failiure of Build Server - greatgamer34 - 07-15-2014
Guys lets stay on topic please!
RE: Possible Failiure of Build Server - LordDecapo - 07-15-2014
My comment was only one off topic. And it was a 1liner..
RE: Possible Failiure of Build Server - greatgamer34 - 07-15-2014
It wasnt directed at anyone in specific, please dont make assumptions.
We have plenty of server hosts available. We are working to resolve the issue as fast as possible! :3
RE: Possible Failiure of Build Server - redstonewarrior - 07-20-2014
As predicted, we were able to recover all data from the build server. (The directories were a bit jumbled from being disassociated from their parents, but who needs 'em.) The harddrive seems to be in decent condition, and we will be testing it soon. In the meantime, we may have build up for tomorrow.
Technical details:
There was an unknown failure point that caused the root file system on build to become available. The journal of the file system was not updated, yet directories were moved around / reallocated / disassociated. We may have lost some variable-data databases, but this does not affect build. As the snapshots directory and current map directory have frequently changing file children, they were affected by the failure. We currently have a heap of unnamed directories sitting in /mnt/pt1/lost+found, and I have managed to locate a backup within three hours of the failure. (I could do better, and may, but I'd call this a success over April.) There are a multitude of other directories, and using guesswork we would be able to reassemble the entire structure. For now, I'm pulling the bare minimum. The mentioned sufficient backup is in ~root/tmp2 mort.
The partition itself seems in okay order, aside from the potential database damage. (Some cleaning.)
It would be possible for me to copy over one of the backups into /servers/ORE and get build running in a minute's time. As build's server is offline while we assessed the situation, we'd have to wait for our host for this. For reasons of "let's not repeat this", it is more likely that we will be hosting on a fresh debian install on a different harddisk tomorrow. (While we thoroughly test the disk, to determine if it was the faultpoint.)
In the meantime, it seems people have been talking about a number of changes for ORE's servers:
* Possible merge of school and build.
* Updating and implementing OREUtilsV2 (
)* ***Cross-server backups*** (Crontab -e time.)
* Revamping of survival.
You will probably see a slew of polls, discussions, and general chat about these endeavors soon.
We're sorry this happened, and we'll be back online soon!
(Don't worry, we'll keep the temporary map in multiverse :3)
(Forgive me for any inconsistencies, ramblings, and general mistakes in this post. It's rather late here, and in my paranoia, I prolonged the recovery task by hours. I've been at this since early, and could have wrapped it up for tea. Assuming I wasn't sidetracked by reddit.)
RE: Possible Failiure of Build Server - PhysoniumI - 07-20-2014
Great job RSW!
RE: Possible Failiure of Build Server - Guy1234567890 - 07-20-2014
REEEED <33333
RE: Possible Failiure of Build Server - Nuuppanaani - 07-20-2014
Awesome Red! You rock <3