Sunday, June 1, 2008

How to not do QA.

I write so much about QA you would never know that my primary focus is software development.

There was not one event that formed this unshakable belief in QA. (Just to refresh..Good QA can save you from bad developers but good developers can never make up for no QA.)
But I can recall several first hand experiences that drove home this painfully gained experience.

And looking back, they all have a common theme of:
"How can we do this without QA?"

Here are three approaches that were tried and where I directly felt the pain:
1. Have the customers test it.
2. Have the engineers test their own code.
3. Have the engineers test each others.

The first one, of having the customers test it, was at the insistence of a bright individual who insisted that there was no time. We had to get it to the market now. This was at the same time that we were trying to get a round of formal testing added to the project plan. I did not have a lot of experience at the time and neither did many of my colleagues but we instinctively knew this was a really bad idea. Another bone of contention was getting a cycle to fix anything found in testing together with another round of testing.

What was offered was; we will do a limited beta and fix all the complaints that our users told us about. An obvious difficulty we had with this proposal was that we could not get to our customers machines to see what was wrong or experiment with fixing them.

The compromise was we would have everyone in the company install the product. That is we would use ourselves as 'internal customers' first. Then we would quickly do an external release.
This failed miserably. Not surprising since this was an untested product that was developed against one environment that was deployed through out a company with numerous differences between the machines.The only upside was the failure was so grand that we did not move on to victimize the external customers.

The second and third options of having the engineers test, either their own or each others ended poorly. I was with a company that after a round of layoffs that included nearly all of the QA group, announced that the software developers needed to test their own code. This way we could keep all the products alive for less cost.

At this point you should step back and wonder, if this is a successful practice, why haven't we done it from the start?

Of course this failed miserably. The first release without QA was the second worst one ever. Features were colliding with each other and numerous bugs were added. In hindsight the best thing would have been to back out the release and deputize some developers to be full time QA.

This was not to be. Instead we followed up the second worst release with the absolute worse release that was to be a "bug fix only" release. What went wrong here was all of those bugs that were introduced had not been found. At the same time, our reliable means of verifying features and bug fixes and now most important assuring no regression bugs was gone.

Now I mentioned that the way out might have been to "deputize some developers to be full time QA". This sounds an awful lot like "Have the engineers test each others code."

The key difference is that in the first case they would be "Full Time", in the second case they would be "Part Time". I have experienced the benefit of 'part time' QA. The feeling that the other engineers are just 'phoning it in' is the best you can hope for. In reality what that other engineer wants to do is get this testing over with as quickly as possible so they can get back to the fun of actually doing what they consider real work.

So there you have it. Three ways to not do QA. They do not work, since you did not do QA.

No comments: