Saturday, 8 June 2013

Downtime Tonight



As many of you probably figured out, we had some server troubles this evening. The reason why it took so long for us to get back up was as follows:

The server guys get notified as soon as the server goes down, so they always know before anyone else. They did a quick check and two of the hard-drives seemed to have failed (computers eh? can't trust 'em.) They got them physically checked and the techies confirmed this. It turns out they both failed simultaneously so the RAID couldn't even protect it from failure.

The techies then replaced the two failed hard-drives with brand spanking new ones. And the server guys re-installed the OS and loaded the latest backup onto them.

I invest heavily into this sort of protection and with the backups in the rare event things like this happen. A lot of sites/games don't. It may have taken some time, but when it involves having to physically replace drives, it can take a while. There might be up to an hour of loss (as the backups are done pretty much hourly.)

Remember to keep an eye on our Facebook Page for updates when the site is down.

I have given everybody a free VIP and VIP+ day. Also a little promo code for everyone to use to make up for the loss and mess ups it would create in regards to the dailies and the slight time jump: junedowntime

Sorry again, and thanks for your patience.
Ash

10 comments:

Anonymous said...

Thanks for the VIP bonuses, and promo! 289510

Boudica Night Tigress said...

Thanks Ash for doing what other games don't, and for getting things taken care of so quickly. Even a technologically handicapped person such as myself knows this was bad. :o Thanks also for the awesome reward! WooT!!!

Darion Lancaster said...

We all owe Ash a giant THANK YOU for having the foresight and awesomeness to invest in backups. It's events like these that justify all the time spent protecting your technology (plus keep us buying into protection to keep not getting screwed over by failed tech). THANKS Ash, for being awesome.

Anonymous said...

Thank You Ash

Lilah said...

First, thank you for the bonuses.
Second, thank you for the explanation as to what went south... I know you've made sincere efforts at upgrading.
Third, I think that it's possible that more than an hour or data was lost... I know I lost BL's and comments made more than an hour before the site went down completely... can't be helped - it is what it is - just sayin', I think the site may have lost a bit more data than you suspect.
Finally, and I'm sure you've already considered this concept, but I'll say it here just in case, have you considered using/paying for a mirror site in addition to the servers that you currently use? The mirror site would keep the same data as your servers, but would kick in the moment your server connections go down for any reason, without loss of data... I'm no network architect, but I know that companies that believe their websites to be mission critical use mirror sites (sometimes multiples) as a means to achieve minimum downtime for the least cost...

Teada Moon said...

That explains why some news that I deleted yesterday has returned and the rant thread that I posted has disappeared! Thanks for the update on the situation Ash.

Anonymous said...

I was asleep. This affects me negatively in no way. Yay for free stuff.

Anonymous said...

Thanks for update and explanation and bonuses to improvise, very grateful.

DarkAsh said...

We do use mirroring techniques :)

Lilah said...

Mirroring drives is good. Mirroring sites (offsite mirroring to an alternate host, in case your entire facility burns to the ground in a worst case scenario) is better. You absolutely use the techniques now - and you do an excellent job with what you have to work with! But data loss and time loss equal potential income loss for you (not to mention all the endless whining!! ;) jk)...

All that said, kudos for all that you do. My suggestion was simply that... food for thought, hopefully something constructive...

Post a comment

Comment this update.

Popular Posts