Many of our clients push for faster turnaround and lower costs without any impact on quality. This is a major trend of today and – let’s admit it – we all may be trying to achieve the same even in our households.
Speaking about software testing (and not only localization testing but any type of testing), the traditional approach, if you wanted to be on the safe side, has been to test “everything against everything”.
Such an approach was quite acceptable in the past when most applications were pure stand-alone and with no interaction with third-party software. The new era of software brought in a different way of development, making software dependent on web servers, database engines, web browsers, etc. Furthermore, if you are delivering several language versions, then it gets even more complex (and costly).
Imagine that you would like to test a web-based application and you want to test the following:
Operating system – 5 of them: Windows XP Home, Windows 7 Ultimate, Windows 8 64bit, Ubuntu 12.10, OS X 10.8
Web browsers – 6 of them: Internet Explorer 8 and 10, and the latest versions of Chrome, Safari, Firefox and Opera
Languages – 8 of them: French, Italian, German, Spanish, Chinese Simplified, Chinese Traditional, Japanese, Korean
There are of course some combinations which are not valid (such as those for Internet Explorer versions for OS X, Ubuntu etc.). If we count only those valid options, then we get 192 environments to test on! Honestly – nobody is willing to pay for that and some reduction must be done, but one smart enough to ensure the required quality. Such a reduction is called pairwise testing.
Here is the excellent description of this technique from Wikipedia:
All-pairs testing or pairwise testing is a combinatorial method of software testing that, for each pair of input parameters to a system (typically, a software algorithm), tests all possible discrete combinations of those parameters. Using carefully chosen test vectors, this can be done much faster than an exhaustive search of all combinations of all parameters, by "parallelizing" the tests of parameter pairs.
If the previous paragraph scared you then let me explain: you set up the testing matrix (which in our case consists of an operating system, a Web browser and a language) in a way that each item from any category of the matrix will pair at least once with one of the items in the remaining two categories. This means that not all possible combinations are tested but each option is combined with all options.
Considering the same constraints as for the above case (do not use Internet Explorer 8 for Windows 8, Internet Explorer 10 for XP, Safari not officially available for Ubuntu and no Internet Explorer for OS X and Ubuntu), we have only 53 scenarios on the table – which compared to the 192 scenarios is a serious improvement, right?
Of course you do not have to create the whole matrix manually – there are tools available and I am sure that there’s an app for that as well. We in Moravia have our own tool named CaseCreator which allows you to specify as many inputs as you want – with all the exceptions you need to consider as well.
You can see the data produced by this tool below – it processed the values indicated above in this blog. You can play with that – just pick any item and you will see that it is paired with all the items in other categories.
This testing approach results in a much smaller number of required testing combinations. In this particular scenario, it helps eliminate up to 70% of all combinations (which is not unusual in actual practice), and so achieve a reasonable test coverage without any major impact on the final quality of the tested product.