Unit Testing¶
In the next few chapters, you’re going to write, debug, and optimize a set of utility functions to convert to and from Roman numerals.
13.1 Introduction to Roman numerals¶
13.1.0 Basic Rules in Roman numerals¶
see Case Study: Roman numerals in Section 7.3
13.1.1 Observations¶
The rules for Roman numerals lead to a number of interesting observations:
- There is only one correct way to represent a particular number as Roman numerals.
- The converse is also true: if a string of characters is a valid Roman numeral, it represents only one number (i.e. it can only be read one way).
- There is a limited range of numbers that can be expressed as Roman numerals, specifically
1
through3999
. [1] [2]- There is no way to represent 0 in Roman numerals.
- There is no way to represent negative numbers in Roman numerals.
- There is no way to represent fractions or non-integer numbers in Roman numerals. [3]
13.2 Introduction to Unit Testing¶
- Unit Testing:
basic idea:
you’re going to write a test suite that puts these functions through their paces and makes sure that they behave the way you want them to. (In other words, you’re going to write code that tests code that you haven’t written yet.)
what’s “Unit”?
Since the set of two conversion functions can be written and tested as a unit, separate from any larger program they may become part of later.
Why it’s important?
If you write unit tests, it is important to write them early (perferably before writing the code that they test), and to keep them updated as code and requirements change.
Unit testing is not a replacement for higher-level functional or system testing, but it is important in all phases of development:
- Before writing code, it forces you to detail your requirements in a useful fashion.
- While writing code, it keeps you from over-coding. When all the test cases pass, the function is complete.
- When refactoring code, it assures you that the new version behaves the same way as the old version.
How to do it in Python?
Python has a framework for unit testing, the appropriately-named unittest module. There is another detailed introduction about this framework.
13.3 Introducing romantest.py
¶
Given the rules and observations in Roman numerals, what would you expect out of a set of functions to convert to and from Roman numerals?
roman.py
requirementstoRoman
should return the Roman numeral representation for all integers1
to3999
.toRoman
should fail when given an integer outside the range1
to3999
.toRoman
should fail when given a non-integer number.fromRoman
should take a valid Roman numeral and return the number that it represents.fromRoman
should fail when given an invalid Roman numeral.- If you take a number, convert it to Roman numerals, then convert that back to a number, you
should end up with the number you started with. So
fromRoman(toRoman(n)) == n
for alln
in1 ... 3999
. toRoman
should always return a Roman numeral using uppercase letters.fromRoman
should only accept uppercase Roman numerals (i.e. it should fail when given lowercase input).
Then, the romantest.py
should be a complete test suite to see whether the Roman numeral conversion functions meet
those requirements.
Example 13.1 romantest.py
)
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 | import roman
import unittest
class KnownValues(unittest.TestCase):
knownValues = ( (1, 'I'),
(2, 'II'),
(3, 'III'),
(4, 'IV'),
(5, 'V'),
(6, 'VI'),
(7, 'VII'),
(8, 'VIII'),
(9, 'IX'),
(10, 'X'),
(50, 'L'),
(100, 'C'),
(500, 'D'),
(1000, 'M'),
(31, 'XXXI'),
(148, 'CXLVIII'),
(294, 'CCXCIV'),
(312, 'CCCXII'),
(421, 'CDXXI'),
(528, 'DXXVIII'),
(621, 'DCXXI'),
(782, 'DCCLXXXII'),
(870, 'DCCCLXX'),
(941, 'CMXLI'),
(1043, 'MXLIII'),
(1110, 'MCX'),
(1226, 'MCCXXVI'),
(1301, 'MCCCI'),
(1485, 'MCDLXXXV'),
(1509, 'MDIX'),
(1607, 'MDCVII'),
(1754, 'MDCCLIV'),
(1832, 'MDCCCXXXII'),
(1993, 'MCMXCIII'),
(2074, 'MMLXXIV'),
(2152, 'MMCLII'),
(2212, 'MMCCXII'),
(2343, 'MMCCCXLIII'),
(2499, 'MMCDXCIX'),
(2574, 'MMDLXXIV'),
(2646, 'MMDCXLVI'),
(2723, 'MMDCCXXIII'),
(2892, 'MMDCCCXCII'),
(2975, 'MMCMLXXV'),
(3051, 'MMMLI'),
(3185, 'MMMCLXXXV'),
(3250, 'MMMCCL'),
(3313, 'MMMCCCXIII'),
(3408, 'MMMCDVIII'),
(3501, 'MMMDI'),
(3610, 'MMMDCX'),
(3743, 'MMMDCCXLIII'),
(3844, 'MMMDCCCXLIV'),
(3888, 'MMMDCCCLXXXVIII'),
(3940, 'MMMCMXL'),
(3999, 'MMMCMXCIX'),
(4000, 'MMMM'),
(4500, 'MMMMD'),
(4888, 'MMMMDCCCLXXXVIII'),
(4999, 'MMMMCMXCIX'))
def testToRomanKnownValues(self):
"""toRoman should give known result with known input"""
for integer, numeral in self.knownValues:
result = roman.toRoman(integer)
self.assertEqual(numeral, result)
def testFromRomanKnownValues(self):
"""fromRoman should give known result with known input"""
for integer, numeral in self.knownValues:
result = roman.fromRoman(numeral)
self.assertEqual(integer, result)
class ToRomanBadInput(unittest.TestCase):
def testTooLarge(self):
"""toRoman should fail with large input"""
self.assertRaises(roman.OutOfRangeError, roman.toRoman, 5000)
def testZero(self):
"""toRoman should fail with 0 input"""
self.assertRaises(roman.OutOfRangeError, roman.toRoman, 0)
def testNegative(self):
"""toRoman should fail with negative input"""
self.assertRaises(roman.OutOfRangeError, roman.toRoman, -1)
def testDecimal(self):
"""toRoman should fail with non-integer input"""
self.assertRaises(roman.NotIntegerError, roman.toRoman, 0.5)
class FromRomanBadInput(unittest.TestCase):
def testTooManyRepeatedNumerals(self):
"""fromRoman should fail with too many repeated numerals"""
for s in ('MMMMM', 'DD', 'CCCC', 'LL', 'XXXX', 'VV', 'IIII'):
self.assertRaises(roman.InvalidRomanNumeralError, roman.fromRoman, s)
def testRepeatedPairs(self):
"""fromRoman should fail with repeated pairs of numerals"""
for s in ('CMCM', 'CDCD', 'XCXC', 'XLXL', 'IXIX', 'IVIV'):
self.assertRaises(roman.InvalidRomanNumeralError, roman.fromRoman, s)
def testMalformedAntecedent(self):
"""fromRoman should fail with malformed antecedents"""
for s in ('IIMXCC', 'VX', 'DCM', 'CMM', 'IXIV',
'MCMC', 'XCX', 'IVI', 'LM', 'LD', 'LC'):
self.assertRaises(roman.InvalidRomanNumeralError, roman.fromRoman, s)
def testBlank(self):
"""fromRoman should fail with blank string"""
self.assertRaises(roman.InvalidRomanNumeralError, roman.fromRoman, "")
class SanityCheck(unittest.TestCase):
def testSanity(self):
"""fromRoman(toRoman(n))==n for all n"""
for integer in range(1, 5000):
numeral = roman.toRoman(integer)
result = roman.fromRoman(numeral)
self.assertEqual(integer, result)
class CaseCheck(unittest.TestCase):
def testToRomanCase(self):
"""toRoman should always return uppercase"""
for integer in range(1, 5000):
numeral = roman.toRoman(integer)
self.assertEqual(numeral, numeral.upper())
def testFromRomanCase(self):
"""fromRoman should only accept uppercase input"""
for integer in range(1, 5000):
numeral = roman.toRoman(integer)
roman.fromRoman(numeral.upper())
self.assertRaises(roman.InvalidRomanNumeralError,roman.fromRoman, numeral.lower())
if __name__ == "__main__":
unittest.main()
|
13.4 Testing for success¶
The most fundamental part of unit testing is constructing individual test cases. A test case answers a single question about the code it is testing.
- A test case should be able to:
- run completely by itself, without any human input. Unit testing is about automation.
- determine by itself whether the function it is testing has passed or failed, without a human interpreting the results.
- run in isolation, separate from any other test cases (even if they test the same functions). Each test case is an island.
Example 13.2 testToRomanKnownValues
- Test:
toRoman
should return the Roman numeral representation for all integers1
to3999
.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 | class KnownValues(unittest.TestCase):
knownValues = ( (1, 'I'),
(2, 'II'),
(3, 'III'),
(4, 'IV'),
(5, 'V'),
(6, 'VI'),
(7, 'VII'),
(8, 'VIII'),
(9, 'IX'),
(10, 'X'),
(50, 'L'),
(100, 'C'),
(500, 'D'),
(1000, 'M'),
(31, 'XXXI'),
(148, 'CXLVIII'),
(294, 'CCXCIV'),
(312, 'CCCXII'),
(421, 'CDXXI'),
(528, 'DXXVIII'),
(621, 'DCXXI'),
(782, 'DCCLXXXII'),
(870, 'DCCCLXX'),
(941, 'CMXLI'),
(1043, 'MXLIII'),
(1110, 'MCX'),
(1226, 'MCCXXVI'),
(1301, 'MCCCI'),
(1485, 'MCDLXXXV'),
(1509, 'MDIX'),
(1607, 'MDCVII'),
(1754, 'MDCCLIV'),
(1832, 'MDCCCXXXII'),
(1993, 'MCMXCIII'),
(2074, 'MMLXXIV'),
(2152, 'MMCLII'),
(2212, 'MMCCXII'),
(2343, 'MMCCCXLIII'),
(2499, 'MMCDXCIX'),
(2574, 'MMDLXXIV'),
(2646, 'MMDCXLVI'),
(2723, 'MMDCCXXIII'),
(2892, 'MMDCCCXCII'),
(2975, 'MMCMLXXV'),
(3051, 'MMMLI'),
(3185, 'MMMCLXXXV'),
(3250, 'MMMCCL'),
(3313, 'MMMCCCXIII'),
(3408, 'MMMCDVIII'),
(3501, 'MMMDI'),
(3610, 'MMMDCX'),
(3743, 'MMMDCCXLIII'),
(3844, 'MMMDCCCXLIV'),
(3888, 'MMMDCCCLXXXVIII'),
(3940, 'MMMCMXL'),
(3999, 'MMMCMXCIX'),
(4000, 'MMMM'),
(4500, 'MMMMD'),
(4888, 'MMMMDCCCLXXXVIII'),
(4999, 'MMMMCMXCIX'))
def testToRomanKnownValues(self):
"""toRoman should give known result with known input"""
for integer, numeral in self.knownValues:
result = roman.toRoman(integer)
self.assertEqual(numeral, result)
|
- [1]:
- To write a test case, first subclass the
Testcase
class of theunittest
module. This class provides many useful methods which you can use in your test case to test specific conditions. - [2]:
- This is a list of integer/numeral pairs that I verified manually. It includes the lowest ten numbers, the highest number, every number that translates to a single-character Roman numeral, and a random sampling of other valid numbers. The point of a unit test is not to test every possible input, but to test a representative sample.
- [63]:
- Every individual test is its own method, which must take no parameters and return no value. If the method exits normally without raising an exception, the test is considered passed; if the method raises an exception, the test is considered failed.
- [66]:
- Here you call the actual
toRoman
function. Notice that you have now defined the API for thetoRoman
function: it must take an integer (the number to convert) and return a string (the Roman numeral representation). If the API is different than that, this test is considered failed. - Also notice that you are not trapping any exceptions when you call
toRoman
. This is intentional.toRoman
shouldn’t raise an exception when you call it with valid input, and these input values are all valid. IftoRoman
raises an exception, this test is considered failed.
- Here you call the actual
- [67]:
- To check whether
toRoman
returned the right value. TheTestCase
class provides a method,assertEqual
, to check whether two values are equal. If the result returned fromtoRoman
(result) does not match the known value you were expecting (numeral),assertEqual
will raise an exception and the test will fail. If the two values are equal,assertEqual
will do nothing. If every value returned fromtoRoman
matches the known value you expect,assertEqual
never raises an exception, sotestToRomanKnownValues
eventually exits normally, which meanstoRoman
has passed this test.
13.5 Testing for failure¶
Example 13.3 Testing bad input to toRoman
- Test:
toRoman
should fail when given an integer outside the range1
to3999
.toRoman
should fail when given a non-integer number.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | class ToRomanBadInput(unittest.TestCase):
def testTooLarge(self):
"""toRoman should fail with large input"""
self.assertRaises(roman.OutOfRangeError, roman.toRoman, 5000)
def testZero(self):
"""toRoman should fail with 0 input"""
self.assertRaises(roman.OutOfRangeError, roman.toRoman, 0)
def testNegative(self):
"""toRoman should fail with negative input"""
self.assertRaises(roman.OutOfRangeError, roman.toRoman, -1)
def testDecimal(self):
"""toRoman should fail with non-integer input"""
self.assertRaises(roman.NotIntegerError, roman.toRoman, 0.5)
|
- [4]:
The
TestCase
class of theunittest
provides theassertRaises
method, which takes the following arguments:- the exception you’re expecting
- the function you’re testing
- the arguments you’re passing that function (if the function you’re testing
takes more than one argument, pass them all to
assertRaises
, in order, and it will pass them right along to the function you’re testing.)
Here, instead of calling
toRoman
directly and manually checking that it raises a particular exception (by wrapping it in atry ... except
block),assertRaises
has encapsulated all of that for us. All you do is give it the exception (roman.OutOfRangeError
), the function (toRoman
), andtoRoman
‘s arguments (4000
), andassertRaises
takes care of callingtoRoman
and checking to make sure that it raisesroman.OutOfRangeError
.
Example 13.4 Testing bad input to fromRoman
- Test:
fromRoman
should take a valid Roman numeral and return the number that it represents.fromRoman
should fail when given an invalid Roman numeral.
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 | class FromRomanBadInput(unittest.TestCase):
def testTooManyRepeatedNumerals(self):
"""fromRoman should fail with too many repeated numerals"""
for s in ('MMMMM', 'DD', 'CCCC', 'LL', 'XXXX', 'VV', 'IIII'):
self.assertRaises(roman.InvalidRomanNumeralError, roman.fromRoman, s)
def testRepeatedPairs(self):
"""fromRoman should fail with repeated pairs of numerals"""
for s in ('CMCM', 'CDCD', 'XCXC', 'XLXL', 'IXIX', 'IVIV'):
self.assertRaises(roman.InvalidRomanNumeralError, roman.fromRoman, s)
def testMalformedAntecedent(self):
"""fromRoman should fail with malformed antecedents"""
for s in ('IIMXCC', 'VX', 'DCM', 'CMM', 'IXIV',
'MCMC', 'XCX', 'IVI', 'LM', 'LD', 'LC'):
self.assertRaises(roman.InvalidRomanNumeralError, roman.fromRoman, s)
|
13.6 Testing for Sanity¶
Often, you will find that a unit of code contains a set of reciprocal functions, usually in the form of conversion functions where one converts A to B and the other converts B to A. In these cases, it is useful to create a “sanity check” to make sure that you can convert A to B and back to A without losing precision, incurring rounding errors, or triggering any other sort of bug.
Example 13.5 Testing toRoman against fromRoman
- Test:
- If you take a number, convert it to Roman numerals, then convert that back to a number, you
should end up with the number you started with. So
fromRoman(toRoman(n)) == n
for alln
in1 ... 3999
.
- If you take a number, convert it to Roman numerals, then convert that back to a number, you
should end up with the number you started with. So
1 2 3 4 5 6 7 | class SanityCheck(unittest.TestCase):
def testSanity(self):
"""fromRoman(toRoman(n))==n for all n"""
for integer in range(1, 5000):
numeral = roman.toRoman(integer)
result = roman.fromRoman(numeral)
self.assertEqual(integer, result)
|
Example 13.6 Testing for Case
- Test:
toRoman
should always return a Roman numeral using uppercase letters.fromRoman
should only accept uppercase Roman numerals (i.e. it should fail when given lowercase input).
1 2 3 4 5 6 7 8 9 10 11 12 13 | class CaseCheck(unittest.TestCase):
def testToRomanCase(self):
"""toRoman should always return uppercase"""
for integer in range(1, 5000):
numeral = roman.toRoman(integer)
self.assertEqual(numeral, numeral.upper())
def testFromRomanCase(self):
"""fromRoman should only accept uppercase input"""
for integer in range(1, 5000):
numeral = roman.toRoman(integer)
roman.fromRoman(numeral.upper())
self.assertRaises(roman.InvalidRomanNumeralError,roman.fromRoman, numeral.lower())
|
- [6]:
The most interesting thing about this test case is all the things it doesn’t test. It doesn’t test that the value returned from
toRoman
is right or even consisten; those questions are answered by separate test cases. You have a whole test case just to test for uppercase-ness. You might be tempted to combine this with the sanity check, since both run through the entire range of values and calltoRoman
. But that would violate one of the fundamental rules:- each test case should answer only a single question:
- Imagine that you combined this case check with the sanity check, and then that test case failed. You would need to do further analysis to figure out which part of the test case failed to determine what the problem was. If you need to analyze the results of your unit testing just to figure out what they mean, it’s a sure sign that you’ve mis-designed your test cases.
- [12]:
There’s a similar lesson to be learned here: even though “you know” that
toRoman
always returns uppercase. you are explicitly converting its return value to uppercase here to test thatfromRoman
accepts uppercase input.reason:
toRoman
always returns uppercase is an independent requirement. If you changed that requirement so that, for instance, it always returned lowercase, thetestToRomanCase
test case would need to change, but this test Case would still work. This was another of the fundamental rules: each test case must be able to work in isolation from any of the others. Every test case is an island.
Footnotes
[1] | The Romans did have several ways of expressing larger numbers, for instance by having a bar over a numeral to represent that its normal value should be multiplied by 1000, but you’re not going to deal with that. For the purposes of this chapter, let’s stipulate that Roman numerals go from 1 to 3999. |
[2] | Whether to 3999 or to 4999 is argurable. This simply because whether it is correct to represent 4000 with “MMMM”. If this is not true, then the limit should be 3999. Otherwise, it should be 4999. |
[3] | Actually, there is. See here |