How to handle an emergency in IT

So the fire is burning high and bright, people are running around asking you when can you fix it every two minutes or so, boss calls wanting to know what the problem is, and someone from remote office just chatted you saying she can't log in.
Fun times.

Basics

1. Stay calm. Sure stay calm, yeah, right, but there's a..., No, just stay calm.
   You are stressed, but if you start panicking, things will go down the drain suprisingly fast. So yeah, do your best to stay calm and composed.



2. Let everyone affected, or if you don't know who's affected let everyone know that there's a problem and you are working to fix it asap.
   Use email, or if you can't use email because just so happens email is down, use chat or whatever, but let users know.
   This will take you 10 minutes to do, but will save you much more time as amount of calls just went down by 50%.



3. Ask your teammember to handle all calls for you. If you're alone in this, ask your co-worker for help. Since you don't have any updates on the issue, or often don't even know what the issue is yet, all calls are basically the same - working on it, sorry for the inconvenience, will update everyone on the progress as soon as i can... Your teammember/coworker/buddy can do that for you, so you can focus at task at hand.



4. When you know anything or have an update, let everyone know, as people (understandably so) grow impatient quickly.

OK, so the fire is still sky high but now you've got some room to manouver and think.

Deal with the issue

1. Make sure you know what is it that you try to fix. Seems like a pretty dumb advice, but often there are multiple issues at play, or what you see at the first glance is a result of something else not working. So umm, yeah, make a list of things to fix.



2. Get to the bottom of the issue. Take your time to make sure you KNOW what's going on.
   What is the cause off all that that mess that you re in?



3. Determine whether you are capable of dealing with your issue on your own or as a team, or whether you need an outside help.
   



4. If you determined that you are the man for the job, create a SOLID path to fixing the problem. Often there are multiple ways of solving an issue, reverse configuration change, restore from backup, rebuild, etc, so make sure you've got things proprly planned starting from an action that would take a least amount of time to implement. It may seem counter productive to waste time here, but the time "wasted" devising a plan of action can save you multiple hours later.
   



5. Implement your fix.



6. Test it, to verify your solution worked.



7. Update everyone. Let your users know they can access IT services now.

Conclusions

The fire's out. Back to regular programming...not so fast.
Why it happened?
Was it a "quick fix" that turned out not so fast and not so "fixing"?
Why wasn't it tested before? Maybe there's not a dev system available to test it on? Why?
Was it a quick configuration change gone wrong?
Why isn't there a policy prventing things like "quick configuration change"?
Was a management pushing for some sort of change to IT systems just because something needed to be done NOW?

One question leads to another, so think about it and write it down. Then you can start making your IT infrastructure and services better and more resilent.

How to prevent emergencies?

You can't prevent it 100%
But by building your infrastructure, services, and preocesses right, you can:

• GREATLY reduce a risk of a system going down, or rather a service being unavailable.
• Minimize downtime.

From our experience, we always start with a site survey. It helps us identify a weak spots in our clients infrastructure.
From the basics - power distribution, wiring audit, network configuration including failover and server configurations.
Through services - VM configuration and redundancycy, backup, management and security.
At the end we are left with a solid roadmap to improve your IT infrastructure and services to minimize risk of emergencies and downtime usually associated with them.

If you are in a pinch and need help to get things going quickly and reliably This email address is being protected from spambots. You need JavaScript enabled to view it..