WTF - Both the primary and RAID drive DIED???

[bren]

EMAIL FROM HOSTROCKET:

"Hi,

We had full server crash on mailbox2 yesterday, where not
only the primary hard drive died, but its redundant RAID
drive as well.

All incoming mail, however, is held in a different server.
Now that mailbox2 has been repaired, it can begin to process
the emails that need to be delivered
from the other server.

Those will eventually arrive at your Inbox over the next few
hours. However, the bad news is that anything that was
stored on those hard drives, which is
apparently your case, are now gone and cannot be recovered..."

I had important emails stored in there!!!

What do they expect me to do now!!

Don't they backup the mail somewhere else at least once a month or something?

I am very angry, but they say there was nothing they could do.

Happen to anyone else?

[Gamerz]

see that is why i would always download the messages onto my computer which would take the off the server and save space.

[lconrad]

I haven't had any since Friday morning.

I requested yesterday morning that if they weren't fixing the problem immediately they set up a forward to one of my working email accounts, but they ignored that request and assured me that it would be fixed.

Today they claim to have processed all the backlog, but I still haven't gotten any email.

They refused to even look at whether there's some extra problem on my account until the *entire* backlog was processed, which took something like 16 hours.

Does anyone have an email account with *good* disaster recovery procedures? I switched to hostrocket from my former webhost because of a problem like this last summer.

[KLH]

This is why I download all my emails to my computer, and if something is important, I forward it to my GMAIL account. It also makes it easier when I have to go dig some email up, I just log into gmail, and its right there. No going back through all my non important emails trying to find it.

[bren]

see that is why i would always download the messages onto my computer which would take the off the server and save space.

The problem wasn't just me... but several other people had accounts through my domain and they don't have the technical ability to download their IMAP messages; they use and prefer the webmail.

I still can't believe both drives died. I bet it was some sort of power supply issue or something. I know its rare but they should do root cause analysis.

Has anyone ever heard of this kind of thing happening on Hotmail, Yahoo or Gmail? Maybe they have a different setup than hostrocket for their mail servers.

[lconrad]

Bren wrote:

Has anyone ever heard of this kind of thing happening on Hotmail, Yahoo or Gmail? Maybe they have a different setup than hostrocket for their mail servers.

I think the big places are better set up for dealing with a server failure. But what concerns me about this one is the cavalier way in which they just spewed out the backlog without setting things up. I personally don't leave mail on their servers if I can get at it, but they started resending the backlog without setting up the forwarding and catchall accounts. Since all my mail goes through catchall accounts, I lost over two days of mail before I realized that I needed to set the catchall.

I did open a trouble ticket when I realized that I wasn't getting mail although they claimed to have fixed the server, but they had a canned response that had nothing to do with my problem, and I couldn't get them past that.

I'm really disappointed in the response to this problem.

[MarkHutch]

Yes, this has happen at other ISP's. Yes, it will happen again. Yes, ALWAYS, download your important email or forward it to another service for safe keeping.

Mark

[HRTimothy]

Yes, this has happen at other ISP's. Yes, it will happen again. Yes, ALWAYS, download your important email or forward it to another service for safe keeping.

Mark

In this case, multiple backups and safeguards failed. We sincerely apologize for any inconvenience this situation caused any of our customers. A team will be reviewing the situation to see if there is anything that could have been handled differently which would have resulted in a better outcome. We will also be putting additional safeguards in place to greatly reduce the remote possibility of this ever reoccurring.

MarkHutch is correct. We strive to make sure that everything works flawlessly. No matter how many safeguards are in place, there is nothing we can possibly do to make all systems 100% problem-free.

[tubby]

Hi all -

I just called the HR support line. The tech support girl began by pretending she knew nothing about it, tried to deflect the problem by asking if I knew how to set up a POP account in Outlook. After insulting my intellegence, she went to "look into my apperent problem", and when she came back on the phone she said, "oh, it seems that there was a bit of a problem with our server on Friday, but they fixed it". Then she preceded to say "oh, i think they put up a note on the website about it". Oh, well, I guess that's okay then...

Well, considering that we lost all our mail, I wouldn't consider it fixed. We had 2 sales guys away all last week. They didn't check their mail since the previous Friday. So any new business messages, client correspondence, etc, are gone. Easy enough to explain to current clients, but we are contected by new people every day. Those new contacts could have super lucrative. Who knows how much $$ this "little problem" is going to cost us.

They also keep saying that this is an isolated problem, and should only affect a few accounts. Well, it affected us, at least 3 of the 12 or so clients we have set up with HR (and maybe a few others), it's affected my personal account, and that's just the people I know...

I asked to be transferred to a sales agent at HR, and she played dumb (of course). She offered me a whole 2 months of free hosting. WOW - what's the value of that? A whole $13?? What a friggin joke!!! Why don't they invest that into a better backup system, or some people who know how to handle important data.

Speaking of handling important data, this is not the first time HR has totally messed me up. Last summer, my site was down. No biggy - I can wait for a bit. But it stayed down for like 2 days. I finally checked inwith HR, and after a bunch of "checking into it", they came out and told me that they DELETED MY ACCOUNT. Just some random mishap. They just deleted it - all my files, databases, mail, settings - gone!! I backup things pretty regularly, but over the course of a few days, a lot of critical (and valuable) data gets accumulated. Then, they told me they wouldn't retrieve my stuff from the backup drive. Why? No real answer, but I suspect they were lazy. Pretty serious allegation? Well, why else not? All they would have to do is put the archive on a spare machine, do an extraction, find my stuff, and dump it back onto another server (once they re-created my account, of course). Instead, they offered me 2 months of hosting... another whole $13 for all my business-critical data.

So, back to the current problem. Although they say the mailserver is working fine again, we're still not really recieving mail. We've had customers calling up saying that they e-mail they are sending is just bouncing back. Sigh.

Anyone totally cheezed??

I think I'm going to post this message to every forum I belong to, and mybe my blog too. I've had it with these guys.

[aov]

Anyone totally cheezed??

Yeah, I am totally pissed of about this. I was never contacted by anyone and told that "we just lost all your email data". The only response was yesterday when someone replied to me that they are aware of the problem and doing something. Now I asked again that when are we going to get our emails back, they told us that everything is lost and they are really sorry.. gee - thanks. No refunds or nothing offered. Way to handle paying customers...

I'm still communicating with the support so at least at the moment I'm still a customer but I have feeling it won't least very long anymore.

[neptune303]

Yeah, I'm totally jaded as well... :redfaced:

I've paid them for a service and they have made a major screw-up; I want to be compensated, not just a lame apology!!!

[npinchot]

This is why I download all my emails to my computer, and if something is important, I forward it to my GMAIL account. It also makes it easier when I have to go dig some email up, I just log into gmail, and its right there. No going back through all my non important emails trying to find it.

Good idea let's not rely on hostrocket to save our important emails, the service we pay for, lets rely on the free service gmail. Yea circumvent the problem, great fix.


Additional comments on the situation as a whole:
If the only backup you have in place is RAID then that's ridiculous. RAID isn't even a backup, it's a hardware failure precaution. You need to have external off site backup in place as well. I could understand having lost a day of email because the backup had not run yet that day but to lose everything is unacceptable. It sounds to me like there is no external backup in place at all. Saying there is no way to have a full proof backup is a cop out. You need to try harder. I've seen plenty of hardware failures where proper backup procedures saved the day.

Stop apologizing with freakin buzz words and start telling us what exactly you're implementing so that it's not going to happen again.