User Acceptance Testing, UAT for short, also goes by the name Demo or Demonstration.
If you have a somewhat typical set of environments setup to support software development, they will correspond roughly to the following:
Development (DEV)
Build
Integration (INT)
QA/Test
UAT
PreProduction
Production (PROD)
How these are used also varies slightly from one company to the next but some key concerns are addresses at each level.
Build ensures the code is in Version Control and always builds.
Integration ensures the code under development and third party code, software and legacy systems all still work together.
QA or Test validates the ever increasing feature development and bug fixes.
So it may seem like UAT is overkill. But it is precisely because the previous (lower) environments have checked out and passed on all these details that allows UAT to provide the first chance for the Customer to provide their feedback or approval. The system is in a state where it could be deployed , all of the bugs and issues have been resolved.
But it is precisely because the Customer is providing feedback here that Software Developers will be most insistent on gaining access to the system.
One reason is familiar, they need to fix something. The answer to this is the same as it was on QA, fixes should flow throught the process to keep the version integrity. (Although QA should have already found the bug so if this is a new bug, some investigation should be done to determine why QA missed it.)
Another reason is the Customer is changing their mind. Here Acceptance is called for, not only will the Customer change their wishes, they may never stop. Sometimes they flip flop between several implementations. Sometimes there is more than one customer and they cannot agree. In this case you will need to find a way to make the different options configurable. This will need to be documented in a Use Case and follow the process of every other enhancement.
But the best reason for keeping Developers off of UAT is to see "Can this solution live without the constant care and feeding by Developers?" Even better "Can mere mortal users actually use this system?"
permalink
Showing posts with label UAT. Show all posts
Showing posts with label UAT. Show all posts
Sunday, May 18, 2008
Thursday, May 8, 2008
The QA Enviroment
The QA Enviroment
In Seven Development Environments:
http://andrewboland.blogspot.com/2008/05/seven-software-development-environments.html
I asserted that developers access to higher environments should be restricted. And above the QA, developers should have no access.Developers do not understand this and object. Since the objections are plentiful and compelling, a reasoned response is warranted. Both the objections and the responses vary by the environment. QA, UAT and Prod have similar but differing reasons given for developer access.Thus I will group these by environment and take on the QA ones first.
So the top three compelling reasons for access QA (Quality Assurance, also known as Test) fall into one of:
1. I need access to QA so I can install the program.
2. I need access to QA so I can fix the bugs they find.
3. I need access to QA so I can understand or diagnose the problems QA finds.
What I like about these is they are versions of "I need it to get the job done."
1. I need access to QA so I can install the program.
This is common and reveals a deeper problem or problems with the software development process.Either there is no installer or setup process for the software. Or the installer/setup is broken. Or the software requires prerequisites or extensive configuration.
Taking the last of these first, the need for prerequisites or configuration post install.The reply here is that this means the installer or the setup process is incomplete. It is fairly easy to have the installer program check for prerequisites. These days is also very easy to have the installer program include the prerequisites and install them if they are missing.Likewise, configuration post install means that there is some documentation missing. The QA team should be provided with setup instructions. If it is too complex for QA then it will be too complex for your customers.
2. I need access to QA so I can fix the bugs QA finds.
This sounds noble. And usually there is a follow on objection, "I need to fix this bug so QA can keep on testing."
But this starts a really bad habit. If the developer can change the release being tested by QA, what is it that QA is testing?
Say the version of the release that QA is testing is "1.0.23".
While that version is being tested the developers should be working on "1.0.24".
So what do we call this modified version?And then what is the QA supposed to do? Both during the time the developer is messing around in the QA environment and after?Is QA really testing that "1.0.23" release or is it now something else?
The best outcome is for the developer to spend a brief amount of time to understand the problem and then go work up a solution in the development environment.
3. I need access to QA so I can understand or diagnose the problems QA finds.
Here is where the developer needs to hold back and let QA do it's job. When QA finds a bug it should isolate the bug and provide reproduction steps.If the bug cannot be reproduced it is not a bug. Insisting on reproducible steps will help in diagnosing the problem.
permalink
In Seven Development Environments:
http://andrewboland.blogspot.com/2008/05/seven-software-development-environments.html
I asserted that developers access to higher environments should be restricted. And above the QA, developers should have no access.Developers do not understand this and object. Since the objections are plentiful and compelling, a reasoned response is warranted. Both the objections and the responses vary by the environment. QA, UAT and Prod have similar but differing reasons given for developer access.Thus I will group these by environment and take on the QA ones first.
So the top three compelling reasons for access QA (Quality Assurance, also known as Test) fall into one of:
1. I need access to QA so I can install the program.
2. I need access to QA so I can fix the bugs they find.
3. I need access to QA so I can understand or diagnose the problems QA finds.
What I like about these is they are versions of "I need it to get the job done."
1. I need access to QA so I can install the program.
This is common and reveals a deeper problem or problems with the software development process.Either there is no installer or setup process for the software. Or the installer/setup is broken. Or the software requires prerequisites or extensive configuration.
Taking the last of these first, the need for prerequisites or configuration post install.The reply here is that this means the installer or the setup process is incomplete. It is fairly easy to have the installer program check for prerequisites. These days is also very easy to have the installer program include the prerequisites and install them if they are missing.Likewise, configuration post install means that there is some documentation missing. The QA team should be provided with setup instructions. If it is too complex for QA then it will be too complex for your customers.
2. I need access to QA so I can fix the bugs QA finds.
This sounds noble. And usually there is a follow on objection, "I need to fix this bug so QA can keep on testing."
But this starts a really bad habit. If the developer can change the release being tested by QA, what is it that QA is testing?
Say the version of the release that QA is testing is "1.0.23".
While that version is being tested the developers should be working on "1.0.24".
So what do we call this modified version?And then what is the QA supposed to do? Both during the time the developer is messing around in the QA environment and after?Is QA really testing that "1.0.23" release or is it now something else?
The best outcome is for the developer to spend a brief amount of time to understand the problem and then go work up a solution in the development environment.
3. I need access to QA so I can understand or diagnose the problems QA finds.
Here is where the developer needs to hold back and let QA do it's job. When QA finds a bug it should isolate the bug and provide reproduction steps.If the bug cannot be reproduced it is not a bug. Insisting on reproducible steps will help in diagnosing the problem.
permalink
Saturday, May 3, 2008
12 Basic Software Development Practices
The gap between programming or writing code and developing software is growing.
This paradox is driven by the proliferation of easy to use tools, libraries, frameworks and OSes designed to ease the burden of software development.
Regardless of why, my experience has been that the typical software development shop is not executing on basic fundamental software development practices. A trend of late that I find even more alarming is quite often many of the programmers and managers are not even aware of or do not insist upon implementing them. It reached the point that I have enumerated 12:
12 Basic Software Development Practices
1. Source Control
2. Issue Tracking
3. Bug Tracking
4. Work Item Tracking
5. Environments
6. Integration Builds
7. Automated Builds
8. Unit Testing
9. Regression Testing
10. QA
11. Versioned Releases
12. Setups
So for a while my thinking was I can just enquire about these 12 before taking a new contract or engagement and all will be well. It is not surprising that I was always told "oh yeah we have all of that.." And it is also not surprising that expectations did not match reality. So I formulated follow on questions to determine the depth of practice.
Questions to determine depth of practice
1. Source Control
Where is the code kept?
Central location?
Individual machines?
Which version is authoritative?
2. Issue Tracking(Issues are a catch all term for everything that does not fall into the bug or work item buckets. They include but are not limited to 3rd party software issues, network issues, development machine issues.)
Is there a Central reporting tool / repository for issues?
Is there an established work flow?
3. Bug Tracking
Is there a Central reporting tool / repository for bugs?
Is there a bug workflow which includes states for:
Initial
Approved
Denied – Feature, works as designed
Denied – Not reproducible
Fixed
Verified
Reopened
Closed
4. Work Item Tracking
Is there a Central reporting tool / repository for new feature and enhancements?
Is there a work flow which includes states for:
Initial
Coded
Test
Failed
Accepted
5. Environments
Do separate environments exist for each of?
Development
Integration
Quality Assurance
User Acceptance Testing
Production
Is developer access to environments higher that Development limited?
Is developer access to environments higher than Integration denied?
Does QA or Change Management own the process of moving releases from one environment to the next?
6. Integration Builds
Does the code build from source control?
Does it build on a machine that is not a development machine?
7. Automated Builds (hourly/daily)
Is the code being built a regular basis?
Is this build done by an automated process?
8. Unit Testing
Do developers check in unit tests with their code?
9. Regression Testing
When code is built for release are all unit tests run?
Bug, Feature and QA tests?
10. QA
Is QA done by someone other than developers?
Does QA write automated or unit tests to demonstrate bugs?
Does QA write automated or unit tests to test common test cases for a feature?
Does QA write automated or unit tests for security and other required practices?
11. Versioned Releases
Is there a version number definition?
Are the products tied to the version number with each release?
Is the version number incremented with each release, both internal and external?
Does QA report bugs and test features according to version number?
12. Setups
Does each application have an install program?
If it does not have an install program are there instructions documented for manually installing the application?
After the install does the program run or is more intervention required?
Footnote. Microsoft Team System can provide them all.
permalink
This paradox is driven by the proliferation of easy to use tools, libraries, frameworks and OSes designed to ease the burden of software development.
Regardless of why, my experience has been that the typical software development shop is not executing on basic fundamental software development practices. A trend of late that I find even more alarming is quite often many of the programmers and managers are not even aware of or do not insist upon implementing them. It reached the point that I have enumerated 12:
12 Basic Software Development Practices
1. Source Control
2. Issue Tracking
3. Bug Tracking
4. Work Item Tracking
5. Environments
6. Integration Builds
7. Automated Builds
8. Unit Testing
9. Regression Testing
10. QA
11. Versioned Releases
12. Setups
So for a while my thinking was I can just enquire about these 12 before taking a new contract or engagement and all will be well. It is not surprising that I was always told "oh yeah we have all of that.." And it is also not surprising that expectations did not match reality. So I formulated follow on questions to determine the depth of practice.
Questions to determine depth of practice
1. Source Control
Where is the code kept?
Central location?
Individual machines?
Which version is authoritative?
2. Issue Tracking(Issues are a catch all term for everything that does not fall into the bug or work item buckets. They include but are not limited to 3rd party software issues, network issues, development machine issues.)
Is there a Central reporting tool / repository for issues?
Is there an established work flow?
3. Bug Tracking
Is there a Central reporting tool / repository for bugs?
Is there a bug workflow which includes states for:
Initial
Approved
Denied – Feature, works as designed
Denied – Not reproducible
Fixed
Verified
Reopened
Closed
4. Work Item Tracking
Is there a Central reporting tool / repository for new feature and enhancements?
Is there a work flow which includes states for:
Initial
Coded
Test
Failed
Accepted
5. Environments
Do separate environments exist for each of?
Development
Integration
Quality Assurance
User Acceptance Testing
Production
Is developer access to environments higher that Development limited?
Is developer access to environments higher than Integration denied?
Does QA or Change Management own the process of moving releases from one environment to the next?
6. Integration Builds
Does the code build from source control?
Does it build on a machine that is not a development machine?
7. Automated Builds (hourly/daily)
Is the code being built a regular basis?
Is this build done by an automated process?
8. Unit Testing
Do developers check in unit tests with their code?
9. Regression Testing
When code is built for release are all unit tests run?
Bug, Feature and QA tests?
10. QA
Is QA done by someone other than developers?
Does QA write automated or unit tests to demonstrate bugs?
Does QA write automated or unit tests to test common test cases for a feature?
Does QA write automated or unit tests for security and other required practices?
11. Versioned Releases
Is there a version number definition?
Are the products tied to the version number with each release?
Is the version number incremented with each release, both internal and external?
Does QA report bugs and test features according to version number?
12. Setups
Does each application have an install program?
If it does not have an install program are there instructions documented for manually installing the application?
After the install does the program run or is more intervention required?
Footnote. Microsoft Team System can provide them all.
permalink
Subscribe to:
Posts (Atom)