Tuesday, June 14, 2016

AUTOMATION 5 - Unit Checks, Part 1: testing randomness

Warning - this example is going to contain some Java samples.  However, I don't want you to be put off that too much if you're not technical.  You should be able to follow the discussion and the outputs even if you're unfamiliar with code.


I love games - in fact so much so my son and I run a wargaming YouTube channel!

Most of the board/wargames we run rely on use dice to determine events - so a dice rolling application seems like something useful to build.

Below is a Java class I've written - don't worry if you can't read it, I'll explain the key features,

  • diceClass (int num) - it's called a constructor which creates a dice object with the number of sides (num) that I request
  • diceClass () - this is a default constructor.  If I don't specify a number, create a 6 sided dice, as they're the most common
  • rollDice() - this creates a random number as if the dice was rolled.  It can be any number between 1 and the number of sides the dice has.
  • returnDiceNumber() - I've used something called encapsulation here.  The number on the dice is private, which means it can't be accessed by anything outside of this code.  Instead I've provided a method which will return the currently rolled value on the dice!



import org.junit.Test; public class diceClass { private int numSides; // number of sides on the dice private int diceNumber; // the random number on the dice public diceClass (int num) { numSides = num; this.rollDice(); } public diceClass () { this(6); } public void rollDice() { diceNumber = 1 + (int)(Math.random() * numSides); } public int returnDiceNumber() { return diceNumber; } }


In layman's terms, I have code which will allow me to create a dice, roll it, find the value.  Pretty simple - but I still want to check it.  Because when I used to program using BASIC on the ZX Spectrum, I remember it was very easy to introduce a rounding error which would mean the first number would be less likely to be represented.

Right now I've not decided what checks to do, or even if it should be GUI or unit check.

Let's actually take a step back - if I was manually testing this what kind of test would I do?  Can I automate that?

Make sure every number comes up test

That seems pretty obvious - let's say the GUI for this looks like this ...


Hopefully it's pretty obvious - every time you hit "roll again" a new number comes up in the box, which currently says "4".

So what's a good test?  Well, I'd press "roll again" several times and expect a different result each time.  Actually random numbers don't work like that ... I'd expect a different number most times.

And I would expect to see the numbers 1-6 appear whilst doing it.

That is a really good first definition of a test.  However you're about to find the difference between a check and a test.  What you've just defined is far too ambiguous to become an automated check.

When we defined automation - it was a script driven set of controls for a system together with the determined outputs.  And that direction is missing.

As a manual tester, I'd run that until I was satisfied.  In an automated check, you have to define "satisfaction" as a hard number.

If I said "if I roll a 6-sided dice 6 times ... I expect to have seen the numbers 1, 2, 3, 4, 5 and 6 at least once" - that check is going to fail a lot.  Checks that fail a lot but don't find a true problem are bad, because they build up a lack of trust in the automation so that when something genuinely problematic occurs "we get fails all the time, but it doesn't matter".  But more on that later.



If I changed it to "if I roll a 6-sided dice 10 times ...  I expect to have seen the numbers 1, 2, 3, 4, 5 and 6 at least once", I expect to see the test fail less. But thanks to randomness, it's still possible to fail.

Let's actually define what that JUnit code would look like for a check with just 6 rolls

import org.junit.Test; import static org.junit.Assert.*; public class testDice { @Test public void everyNumberOccurs() { System.out.println("Confirming every number happens"); diceClass dice1 = new diceClass(); int numSides = 6; int[] countArray = new int[numSides]; int counter, index; int totalRolls = 6; //Roll the dice totalRolls times, count the results for(counter = 0; counter < totalRolls; counter++) { dice1.rollDice(); index = dice1.returnDiceNumber(); countArray[index-1]++; } //Rolling done, show us the numbers for(counter = 0; counter < numSides; counter++) { System.out.println("Assert: Number of " +
(counter+1) + " = " + countArray[counter]);
}

//Now confirm each is greater than 0
for(counter = 0; counter < numSides; counter++)
{
assertTrue("Did not find " +(counter+1)+ " rolled",
countArray[counter]>0);
}
} }

Okay - let's run that.  What's neat about JUnit checks, they run as part of the build process for code, so you get a response pretty quick!

On our first go, you can see in the Eclipse console that not every number has occurred ...


Let's give it another go ... close but no cigar ...


And over under the Eclipse Failure Trace, the assertTrue has generated a failure,


Obviously I'm going to have to sample over a large enough set to reduce the chance of an error.  So I'm going to use as the basis of my check "if I roll a 6-sided dice 100 times ...  I expect to have seen the numbers 1, 2, 3, 4, 5 and 6 at least once".  That's probably excessive - but I've no real clue how many I'd do before I gave up in frustration.  And the automation needs an iron rule.

I'm going to change the value of totalRolls to 100 in the code above.  That means sampling 100 dice rolls.  Let's see what happens now ...



This check runs almost instantaneously.

Now of course I could use my GUI automation here to run this, it'd be okay - but much slower.  I'd need to wait for a press of the button, screen refresh,  read the value from the field, store it.  Generally I find it seems to take about 0.1 seconds for a screen refresh (hey, we get slow internet in New Zealand), unless you have a cut down webpage.  So checking 100 dice rolls could take a few seconds.

According to our iron rule ...


It probably makes sense to use a unit test over a GUI test for such a mass action.  The developer certainly finds out about the problem much simpler - it comes out as part of the build process.




More next time

I'm going to leave it here for today - we're going to look at more example checks we can do, and think a bit deeper of how this compares to GUI tests.  But for today we'll leave it here before you get overwhelmed.


Extension material

You might want to have a play around with this yourself - you might want to install Java Eclipse on your machine.  But don't forget you will also need to install the Java JDK as well from Oracle.


No comments:

Post a Comment