Problem to Solve — During a “Severity 1 / War Room” situation, how can we reduce the time for the triage process ?

The WAR ROOM

The IT War Room HUH !!!! This may be a blog personal favorite by combining a timeless classic song “WAR” ( https://www.youtube.com/watch?v=dQHUAJTZqF0) and classic IT troubleshooting practice (“WAR Room”). I am thinking that I need to add the HUH !!! to my everyday language.

I consider the IT War Room a timeless classic because we have all been there at one time or another in our careers. Yes, you all know what I am talking about. When you experience a true “Severity 1” service outage that is costing your company …..lost revenue, credibility with customers, or significant down time, etc …. your management assembles the War Room. The War Room consists of all key personnel from the various IT teams, disciplines and silos responsible for — networks, applications, servers, virtualization, security, change management, operations, management, etc. Each team gets thrown into a real (or virtual) conference room where they are tasked to work together to FIX THE PROBLEM ….. HUH !!!!

WAR Room Protocol

Of course, with any tense situation, like a War Room can be, there is certain protocol and expected etiquette.

• Keep the teams well fed and highly caffeinated. Pizza and Coffee are basic food groups.
• Instruct each Subject Matter Expert (SME) from the respective IT silo to go to their tool of choice and …….dig in.
• Review the last 5 to 10 Change Controls in great detail to figure out “what changed” to locate the most likely suspects.
  
• Make sure everyone keeps their heads down, digging into the detailed analysis. Never ever look up as the other War Room participants can smell fear.
• Try to find some shred of evidence to determine the root cause and look like the hero to management.
• Try to deflect all responsibility for the outage from you and your team to someone else

Yes, this is a somewhat satirical viewpoint on the War Room situation. I myself experienced a highly volatile War Room at one of my very largest customers that involved well over 100 contractors, vendors, consultants, network engineers, server administrators, application developers and several weekends of my life. When over 100+ people are involved in a high pressure / business critical situation at an extremely large customer site, trust me it leaves an indelible mark HUH !!

Even with the satire above, I actually believe that the War Room approach can be  extremely effective, but does depend on a few key strategies.

• Enable a Strong Facilitator – Someone typically is in charge of the War Room. Sometimes a VP, sometimes a senior member of IT, sometimes just a well respected technical leader. Bottom line, this person must coordinate, facilitate and drive the War Room team to a problem resolution. More often than not, this person is a “master” of conflict resolution and team building. Let’s face it, put a bunch skilled IT folks from different disciplines into a tiny room, and there will be a conflict or two. Sometimes the conflict can be positive and drive the team toward a resolution. And sometimes it can be negative and hamper the process. The facilitator keeps the triage process progressing towards resolution and sets the tone in the room for the points below.
• Check your Ego at the Door – Every member of the team brings their unique skill set, experience, knowledge and Ego to the War Room. IT folks are no different than any other person on the planet when it comes to Ego. While the most talented, or most respected, or sometimes the loudest person will provide their input on direction, it should be based on probable cause, not on ego. While everyone appreciates someone standing up and displaying leadership, the motive should be geared on how to “solve the problem”.  It should not be geared on “I have been doing this for 20 years”, or “I have 7 Ph.D.’s in IT”, or “I play golf with the CEO”, so you will all listen to me. The quicker that folks check their ego at the door, the quicker the problem gets solved and everyone can go home to their families.  
• Don’t Play the Blame Game – If any of your War Room team members come in with the  mindset to  “deflect all responsibility and try to blame other teams for the problem“, you are in for the long haul in the War Room. If humanly possible, the members of the team should try and have a Top Gun attitude, not the arrogant attitude in the beginning of the movie, but the part where the ship’s commander tells the fighter pilots that they are America’s best, and to “Make us Proud”. The pilots then take off, work together, defeat the enemy, and everyone celebrates the heroic effort. An IT War Room should have the same mantra for the team. You are our best and brightest, make us proud and get to the root cause of this problem ASAP.
• Deploy Reliable Tools – Seems like common sense for companies to have reliable and accurate tools for their IT environments.    However, some companies have not made this a priority and then usually suffer the painful consequences during a serious outage.   You can have the most advanced and highly trained IT staff, but the lack of proper tools will reduce their overall effectiveness.    Those of you that have ever tried to tighten a Phillips screw head with an oversized flathead screwdriver know what I mean.   The right tool applied at the right time makes all the difference in the world, and IT is no different.
• Build the Team – I have seen moments where the egomaniacs had their efforts redirected toward the team concept. If properly fostered by the War Room leader, you will see IT silos break down and people begin to collaborate together to fix the problem–not from a perspective of selfish attitude, but toward a common goal of “let’s fix this thing and get out of here”. I have seen some companies leverage the War Room as an opportunity to strategically move personnel around within IT–not as a punishment, but to better round out the team and organization. For example, if a well respected network architect moves into an application or services team, their perspective on the business will change. They will see the network as a piece to the puzzle, not the puzzle itself. This provides growth opportunities for your team’s potential career path (management possibly) and a useful method for continuing the collaboration movement during the next War Room situation.    HUH !!

Points to Ponder

• Who and what teams are represented in your War Room experience?
• If two different silos or tool sets come up with different data that lead to different actions, how do you resolve conflicts and reconcile to your next steps?
• How many different tool sets are involved?    
• Who is the facilitator responsible for reporting up to Management, the current status, next steps, and root cause?
• What timeless classic lessons have you learned from your War Room experience? 

Until next time ….. HUH !!