Backing Down A Bug

So a bug is found in production.

In spite of your following all the best practices.

You had separate environments for Development, Integration, QA (testing), UAT and Production.

You have been doing versioned releases and following your documented change control procedures.

You had a QA staff that did it's best to test for and find all bugs.

You have three levels of help desk or technical support.

And in spite of this a bug made it all the way out to production and it must be fixed. And the business side of the house is flipping out and the pressure is on. Fix the bug now!!

How you handle this will determine if your company grows and gets a tiny bit better. Or should you fail, you the company will plant the seeds of disruptive behavior that will haunt you for the rest of your tenure.

You must avoid the temptation to storm in and take things over. You need to Back This Bug Down From Production with the care you would back a jumper from a ledge.

First make sure that the help desk or technical support was engaged. If not, engage them. Use this as learning opportunity for the support staff to learn how to mitigate or work around bugs in production. Should you find a work around, you must insist to the suits that it is over. A work around has been found. The work around is in place. Both the bug and issue (the issue is how the bug made it to production) are being researched and will fixed in later release. End of Story. Do not be stampeded on this.

Now chances are good that a work around is not immediately found. In this case, the backing down starts. QA must take a role here. At this point QA has tested the application and their job is to get the bug reproduced in the QA environment. Again, you must resist the desire to jump in and take over. QA has an important role in defining the bug. If QA can reproduce the bug in the QA environment, you should then be able to use this information to reproduce the bug in the Development environment. When you have fixed bug, QA will now be in a position to test the fix. This would not be possible if you had ripped control away from Tech Support or QA.

But this is not the end of the story. There is still the open issue of how the bug survived your processes. Inevitably something is missing in terms of testing or the processes were not followed. Fixing this issue will prevent future bugs.