What do you do when things go wrong?

[sirjonathan]

Hey guys!

Alright, I'm looking for input from a lot of the new web host providers out there, and even from you guys who've seen it all .. How do you handle things when they go very wrong with your server and your business is at stake?

Some of you may have NEVER had a problem (lucky bums ) ..but what do you plan to do when something DOES go wrong?

I ask 'cuz I didn't think I'd ever have a problem . Then one day my email server stopped working.. right after a client had just sent out a big newsletter asking for email replies.. ouch. I never expected that and I was in a huge panic as to what to do.

So what happens? I was very fortunate to know people who were able to help me out of it. Otherwise, ugh.. I woulda had NO clue how to go about fixing it.

Communities are an incredible system of support, especially like what we've got going right here.. I can't tell you how many times I've had something solved just by reading posts of people going through similiar experiences .

So how about it? What do you do when things go wrong and it seems beyond your control?

-Jonathan

 

[Exon]

The worst thing we've ever had happen to date is Apache stopped responding. I FLEW home (won't post speeds ) to fix it when I got the call at the store. Between the time I got the call, and got home a tech had already logged in and fixed the problem. Was maybe 10 minutes for the trip.

I was lucky, I knew what needed to be done, but didn't have internet access to do it - but like I said, a datacenter tech had already logged in and fixed the problem when I logged into SSH to fix it.

 

[sirjonathan]

Lol! OUCH.. That wouldn't have been a fun one . Hmmm.. well I know a response that you would NOT want to hear from your web host..

You call them up.. "Apache stopped responding! Help!"

.. they answer..

"Apache? What's that? I didn't know we had that!.."

..hehe . Sounds like you were very well prepared. Score one for the datacenter techs!

-Jonathan

 

[u2mike]

When something goes wrong heres what you must do:

1. Fix it
2. If it was something major you could offer some compensation for the problems.
3. Do everything you can to ensure it doesn't happen again.
4. Explain to your customers and ask for there forgivness, tell them what your doing to ensure this doesn't happen again.

And learn from Exon, you need to have someone keeping an eye on the server. If don't have the employee's look for an outsourced solution to get your server up in case of a hazard. Another good tip, keep uptodate remote backups! If something does go horribly wrong you can always wipe the machine and start over. Another thing is you should know what your doing if your responsible for the server. If you don't know then you need to hire someone who does.

 

[vito]

Redundancy, backups, planning. It is essential to have solid contingency plans in place.

http://www.disaster-recovery-guide.com/ offers some good reading.

Vito

 

[Simon]

When something goes wrong, you thank your lucky stars that your business plan included disaster recovery, in all senses of the word, and that you hired people who knew how to fix thngs.

The best thing I can say, is "don't ever panic". Whilst it may seem like it on the surface, nothing is the end of the world. You calmly fix the issue, or instruct someone to fix it, and pat yourself on the back for doing so. Then, you put whatever system(s) were missing, in place.

Simon

 

[Exon]

Well, the client who called and informed me was a pretty high paying client, and if one person was effected, the rest of 'em were.

Was thinking some more, and come to think of it - the worst thing that has happened thus far (Not per my company, but a LAN group that I use to volunteer time to) just recently lost the mail server. Not like a software/hardware failure - but the datacenter moved the server and PHYSICALLY lost it. Glad I don't colocate there

 

[Moni-Q]

First thing you need to do: Be honest with your customers. The worst thing that could happen to me as a customer is to see my site down and then contact the provider only to be fed BS, or even worse, not get a response at all. If there is a problem, admit it, then truthfully assure me that you are doing all that you can to fix it.

Second thing: Fix it! Right away.

 

[eglim]

Read my story i hope you find some answer of your question.

Some years back I have a Cpanel reseller account. suddenly my sitess stop working (time near 7am). Support was online(on MSN) that time of my

hosting company. I contact him for this. He say let me check. After 2 minuts he told me that server hacked and formated and there is no way

to get data back.
ohhhhhhhh you can understand what happen with me with my heart and right after one hour (when customers start checking mails and checking his

sites) what they do with my body too. It must be police case too. Beacuse many customers have online billing database too.

What I do. First I call my hosting company what they an do for me. But surprised when they say they can't do any thing for me. Even they not

have backup. I can't do any thing with them beacuse they are from another country where I live.

I hire reseller account from another company (I always keep good companies list). I have custom DNS from old company (which hacked). So I ask

new company to set same DNS for me.
I start making new account for customers with same username/passwords and plan which i send them in welcome mails. I also keep record in my

computer of welcome mails.

During that I start calling my customers (before they call me) and told them all what happen with server. Good thing happen that I got

messege from my old hosting company (which one hacked) that they have Database backup which they tack two days before. I get database from

them. I am not able to uplaod database beacuse I not have all sites. So I call customers which use database and ask them I am going to send

database and you please uplaod your site again and with this database. A bad thing happen that many customers not have their sites backup in

their HD. (Beacuse some customers format computer/some have problems with their webmasters so not able to get datat.)
I ask them that I will make site for them when I am free. I know this is a very hard dicission but I want to stay in market.
Many customers sites which I made have backup in my HD. So on other computer I start uploading that sites.
Near 10.30am more then half sites start working.

and after two hours many others sites also start working (my emp. visit customers and get data from them and upload).

NOW I AM LITTLE RELEASE FROM TENSION and take lunch. During that my company support person only got few calls for passwords. I think its

beacuse many of my customers not change their password which send them in welcome mails so they not have problem to again uploding sie.

Its was near 6pm when 96% sites uplaoded. 4% site remaining beacuse some customers not have backup of data and with some customers I not able

to contact beacuse they not come office or not at home. So I think I will contact with them tommorow.

near 30% customer not notice of server failure beacuse I have their data and I upload. They not use their domain email.

Next day I got messeage from my old hosting company that my account active and I can create accouts. I reply them that I not want to stay

with them and ask them for full refund and also for 1000$ extra.
Its beacuse when i signup with them they told me that they do backup on sunday but they not. No reply from them I got in next 12 days.
So I mail them again. Atlast they refund me all money and extra 400$. I also post all story about them on WHT.

that whole month my company make new sites for that customers which not have backup of their sites and lost online data beacuse of my server failur.
I always honest with my customers. So why I am still in this business and my business growing every day. But that company which hacked not in business this time. I just check their site not working. I think they are run away or change their name.

 

[Equinox]

Good post...*hands over reputation points*

 

[Moni-Q]

Read my story i hope you find some answer of your question.

Some years back I have a Cpanel reseller account. suddenly my sitess stop working (time near 7am). Support was online(on MSN) that time of my

hosting company. I contact him for this. He say let me check. After 2 minuts he told me that server hacked and formated and there is no way

to get data back.
ohhhhhhhh you can understand what happen with me with my heart and right after one hour (when customers start checking mails and checking his

sites) what they do with my body too. It must be police case too. Beacuse many customers have online billing database too.

What I do. First I call my hosting company what they an do for me. But surprised when they say they can't do any thing for me. Even they not

have backup. I can't do any thing with them beacuse they are from another country where I live.

I hire reseller account from another company (I always keep good companies list). I have custom DNS from old company (which hacked). So I ask

new company to set same DNS for me.
I start making new account for customers with same username/passwords and plan which i send them in welcome mails. I also keep record in my

computer of welcome mails.

During that I start calling my customers (before they call me) and told them all what happen with server. Good thing happen that I got

messege from my old hosting company (which one hacked) that they have Database backup which they tack two days before. I get database from

them. I am not able to uplaod database beacuse I not have all sites. So I call customers which use database and ask them I am going to send

database and you please uplaod your site again and with this database. A bad thing happen that many customers not have their sites backup in

their HD. (Beacuse some customers format computer/some have problems with their webmasters so not able to get datat.)
I ask them that I will make site for them when I am free. I know this is a very hard dicission but I want to stay in market.
Many customers sites which I made have backup in my HD. So on other computer I start uploading that sites.
Near 10.30am more then half sites start working.

and after two hours many others sites also start working (my emp. visit customers and get data from them and upload).

NOW I AM LITTLE RELEASE FROM TENSION and take lunch. During that my company support person only got few calls for passwords. I think its

beacuse many of my customers not change their password which send them in welcome mails so they not have problem to again uploding sie.

Its was near 6pm when 96% sites uplaoded. 4% site remaining beacuse some customers not have backup of data and with some customers I not able

to contact beacuse they not come office or not at home. So I think I will contact with them tommorow.

near 30% customer not notice of server failure beacuse I have their data and I upload. They not use their domain email.

Next day I got messeage from my old hosting company that my account active and I can create accouts. I reply them that I not want to stay

with them and ask them for full refund and also for 1000$ extra.
Its beacuse when i signup with them they told me that they do backup on sunday but they not. No reply from them I got in next 12 days.
So I mail them again. Atlast they refund me all money and extra 400$. I also post all story about them on WHT.

that whole month my company make new sites for that customers which not have backup of their sites and lost online data beacuse of my server failur.
I always honest with my customers. So why I am still in this business and my business growing every day. But that company which hacked not in business this time. I just check their site not working. I think they are run away or change their name.

Kudos to you. You really handled that situation well.