We had a very minor release of an application last week as the front end application and the back end procedures were operating double standards. The front end was asking for a value greater than or equal “>=” to a certain value and the back end said the rule only applied to values greater than this amount “>”.
It took hours to figure out what was wrong and a minute to fix it. We were looking at the changes that we made in the last release suspecting that a bug must have been introduced at that time. This is because the issue had never occurred before and the application is mature at 10 years old. However, simple boundary value analysis as part of quality assurance testing would have captured the issue. The application was written a long time ago and it went from development to user acceptance testing so it would not have been treated to the full range of checks that the professionals use. It made me appreciate again the value of good quality assurance.
In this short article I just wanted to run through a few of the types of quality assurance tests that take place. Developers and some technical testers run through the code comparing it to use cases in the specification and find and fix any bugs found. Quality assurance generally testing takes place in a non-code environment in preparation for either user acceptance testing or the production release and is not normally based on going through code.
Specification-based quality assurance tests, which is known as black box testing, can usually be written as soon as the specification is ready and sometimes they form part of the specification document. That is of course if the specification is well defined, if it is not, test planning might have to take place as the specification becomes further defined during development or when design decision have been made. A specification usually only defines the behaviour to be achieved rather than how this will be achieved which is the job of designers and developers. Specification-based quality assurance testing is about satisfying the behaviour stated in the specification.
Equivalence partitioning
I was going to start with boundary value analysis as I have already mentioned it in my example above but I started thinking of equivalence partitioning when describing it so I will start here.
The idea behind equivalence partitioning is that many inputs will be of a certain type so the tests for those types can be put together. For example if the system requires a number input then anything that is not a number will fail the test. Likewise, if the input should be an alphanumeric field the the input will either but a number or from letters of the alphabet. Unless specified as such then an alphanumeric will not allow a space and you might want to call the test something else if it does allow, for example, spaces, hyphens and apostrophes such as a name test.
In general, equivalence partitions are broken down into their lowest values so they can be reused. Also, 0 is quite often treated separately because in computer terms, 0 often relates to false or negative. So, if we did have an equivalence partition of 0 and one for the other nine numerals (1-9) then we could have a combined positive number test involving both partitions. If we add to that lower case letters in one partition and uppercase in another then an alphabetic test in English would also have two equivalence partitions. Space could be another and if, so then three. An alphanumeric that allows spaces could be combining the logic from 5 partitions.
Once you have equivalence partitions then you can quickly combine them for all of the acceptable data types and combinations of those types working together for data inputs. A positive number test only has to be tested for 0 and any other positive number to make sure it works with valid inputs. You should also test invalid inputs so anything else entered in this case including a negative number must result in a failure.
Type tests become simpler if you build them up and use combinations of tests in this way and you’ll know that if an equivalence partition is established and tested you can trust it in the future in other places. You don’t have to write as many data type tests if you build it up this way.
You can also apply other logic and not just data types, such as, how many characters, between values, higher, or lower, etc…
For example, lets us say we want to input the temperature in degrees Celsius to determine the state of water. Below at zero or below it will output “solid”, at 100 or above, it will output “gas” and anything in between, “liquid”. We can have a build up equivalence partitions to test for “gas” for numbers at 100 or above where 105C will be a success and 95C and -5C will be a fail. Then another to test for “liquid” for numbers between 0 and 100 excluding both where 95C will be a success and 105C or -5C will be a fail. Finally a third test for “solid” states will show a success for -5C but a fail for 95C and 105C.
Other than input values we can also test output values too in a similar way.
Boundary value analysis
Mistakes quite often happen as in my example at the top of this page around a boundary entry. In that example, the database programmer interpreted the specification as greater than where the front end developer interpreted it as greater than or equal to. A boundary test is to test for success and failure based on the specification around the changes between success and failure.
Boundaries generally include the specified boundary value and one either side of it. So the maximum length of a string input of 10 means a boundary test of 9, 10 and 11. Some might say that 9 would not be required but I always believe that you need to test either side of the boundary just to make sure the logic is correct for exactly that amount and doesn’t fail for nearly that amount.
In the water state determination example of equivalence testing, the boundaries here (using only whole numbers to keep it simple) 0 and 100 so the values that we need to test are -1, 0, 1 and 99, 100, 101.
Decision table testing
Specifications usually contain the business logic which include decisions. Individual decisions might be quite simple but overall when you add in several decisions it can get quite complex. All outcomes need to be tested and the quality assurance tester will resort to a table of test results for which to test outcomes.
A decision table will list out all of the inputs that can occur and their respective expected outcomes.
I think it is best described with an example. Lets say we operate a Indian restaurant that delivers food if delivery is within 10 miles at a cost of £5, it will refuse to deliver if more than 10 miles out. The restaurant will provide free delivery if the food order (excluding delivery charge) is over £10 and will give you free onion bhajis and pomadoms if you spend over £25. Notice that these are both over if you were thinking of boundary testing although the delivery mileage is inclusive of 10 miles out.
The decision table would look like this:
| Order 1 | Order 2 | Order 3 | Order 4 | |
| Input A: Distance in miles | 7 | 7 | 7 | 15 |
| Input B: Order Value | £9.50 | £19.50 | £29.50 | £9.50 |
| Outcome A: Refuse Delivery | N | N | N | Y |
| Outcome B: Charge £5 Delivery | Y | N | N | n/a |
| Outcome C: Extra Sides For Free | N | N | Y | n/a |
I could have answered order 4 that was over 10 miles away with a Y for outcome B and a N for outcome C and gone through order 5 and 6 with 15 miles for input A and similar values for input B as orders 2 and 3. However, I decided that outcome B made outcome B and C irrelevant.
State transition testing
Decision table testing is very useful where combinations of input conditions produce various outcomes. State transition testing is concerned with changes to the current inputs which the affect the outcome or state which was based on previous inputs. Basically the behaviour of the system relies on both the previous state and the current inputs affecting it.
If you can imagine that you are wearing a pedometer whilst you walk to the bottle bank and so it is counting your steps. The state of the display is blank when I have done less than 250 steps. The state remains the same on each step but when I reach 250 steps it gives me a congratulatory message for 5 seconds. When I reach 251, 255 or even 300 steps, the congratulatory message is not repeated. Also, when the time gets to the next hour then the counter is reset and if I then do another 250 steps, it will congratulate me again.
these are the transitions that you would look to test. There is a special way of diagramming a state transition but I will not cover that here. Still test cases can be plotted in a similar way to a decision table.
Here is what we are expecting the screen to show.
| State | Expected |
| At start of each hour | steps = 0 |
| At less than 250 steps in the hour | show steps |
| At 250 steps or more and within 5 seconds of achieving 250 steps | congratulations message |
| At 250 steps or more and after 5 seconds of achieving 250 steps | show steps |
A decision table based on state could be set up as follows.
| State 1 | State 2 | State 3 | State 4 | State 5 | State 6 | State 7 | |
| Input A: Time | 12:00:00 | 12:05:00 | 12:15:11 | 12:15:16 | 12:15:17 | 12:59:59 | 13:00:00 |
| Input B: Steps | 0 | 25 | 250 | 255 | 256 | 844 | 0 |
| Outcome A: Reset Steps to Zero | Y | N | N | N | N | N | Y |
| Outcome B: Show Steps | Y | Y | N | N | Y | Y | Y |
| Outcome C: Show Congratulations | N | N | Y | Y | N | N | N |
Use case testing
Use cases are a way of specifying a piece of functionality as a particular scenario or process flow. Use cases tend to be user driver and quite often are based on a category of user. However, there can be system driven use cases for example in automated processes where the scheduled system task acts as the user. For this reason the user is often referred to as an actor as it will be playing a role and doesn’t have to represent a human being.
An example of a use case could be “Runner adds new destination” or “System sends reminder”.
A use case generally defines a complete process and you can test that from end to end.
You can use all of the tools above but more than that you can use pseudo code to define what the logic implies and then create work flow diagrams and the like.
Use case testing does not have to be black-box and the tester can always look at the code to see if the logic matches their pseudo code or work flows.
There can be such variety within a use case that is is hard to explain here but I might write about use cases in a future analysis article and if so, I will include something about testing too.