Functional Testing: Problems
Problems and Fragilities in Functional Testing
Functional testing is not always straightforward. In fact its rarely straightforward. Some of the challenges include:
- Fragility due to layout changes
- Timing problems (beware the lure of the voodoo sleep)
- Some UI elements are very hard to test
- System dialogs (that are hard to interact with programatically)
- How do you test printing?
- Bugs in the GUI toolkit
- Spurious, random and impossible failures
Lets look at some of these issues.
Fragility Due to Layout Changes
The following examples (that we used earlier) have a problem:
tabPages = self.mainForm.tabControl.TabPages
They work fine until your application grows and the toolbar is no longer a direct child of the main form but is contained inside another panel.
As your application evolves you will undoubtedly have to refactor your tests alongside, but there are at least two ways we can minimize this.
The first is by encapsulating all access to the toolbar inside a method:
toolBar = self.getToolBar()
When we change the location of the toolbar we only need to update a single method instead of finding all the different places we used it. As a nice side-effect it makes the code easier to read.
A more general method for Windows Forms is to use the fact that we can give every control a 'Name'. We can then have a general 'findControl' method that recursively searches from the top-level control down through the children until it finds the control you have asked for. This is much less susceptible to changes in layout.
Interacting with Controls and Testing Layout
For functional testing we should really only directly interact with user interface elements to investigate their state. In the functional test we wrote to test the 'New Page' dialog we invoked the dialog by calling dialog.AcceptButton.PerformClick - which although convenient violates this rule.
There is a better approach - we can ask the button where it is and send a click event to its centre.
This has the added advantage of testing that the button is actually clickable! If the button is obscured by another control then the click event will never reach the button.
Actually testing layout is very difficult as it is largely an aesthetic judgement (something that computers are still very bad at). At Resolver Systems we have a basic test that checks that none of the elements of our dialogs overlap - but that is about as far as we go with testing layout.
For Windows Forms every control can tell you its location relative to its containing form. They also have a method to turn these relative co-ordinates into absolute screen co-ordinates. Sending mouse move and click events to these locations ensure that your tests aren't dependent on forms being shown at exactly the same location every time.
Unfortunately to send mouse events (left clicks / right clicks / drags etc) we need to use the win32 apis from User32.dll. These are exposed by pywin32, and at Resolver Systems we wrote a simple wrapper layer in C# using P/Invoke to interact with native code.
Other operating systems provide similar hooks for sending user input - and there are testing libraries built around them. See the Alternative Testing Tools section at the end of this article.
The Voodoo Sleep
Timing related issues can be one of the biggest source of problems in functional tests. Depending on system load, the time of year, or what you ate for breakfast can cause the same operation to take different lengths of time to happen every time you run the tests.
Killing resource hungry background applications can help. The first thing our test suite does on starting is kill any processes called 'Thunderbird' or 'Firefox'...
That aside it can still be tempting to add random sleeps to your applications in the hope that an operation will succeed, or some event occur, during that time.
These 'voodoo sleeps' are a recipe for randomly failing tests. A much better pattern is 'wait for condition'. I should have used this to wait for the dialog to appear in the clickNewPageButton method from the previous example.
'wait for condition' basically means polling for a condition, with a timeout, instead of a blind sleep. Lets see how we could have used that for working with the 'New Page' dialog:
taken = 0
while taken < timeout:
taken += pollInterval
self.fail('Failed waiting for condition: ' + message)
button = self.mainForm.toolBar.Items
executor = self.executeAsynchronously(button.PerformClick)
form = Form.ActiveForm
return self.invokeOnGUIThread(lambda: form.Text == 'Name Tab')
self.waitForCondition(condition, 'New page dialog to become visible')
waitForCondition takes a callable that returns a boolean indicating success or failure. It has a configurable timeout and poll interval. If the condition doesn't return True before the timeout then the test fails with an appropriate message.
Some Things Are Hard to Test
Testing isn't easy, and sometimes its damn hard. Rather than giving up (if it aint tested it don't work...) you can apply a little imagination.
We test printing by using a printer driver that outputs an image. We then compare the image to 'one we produced earlier'. If anything changes (so the test starts failing) we create a new image - which of course requires us to check that it really is acceptable first.
We also have some hard to test UI elements. One of these is an error bar that displays a red bar by the side of the scroll bar indicating the position of errors in user code. To test this we screenshot the control and assert that there are some red pixels in the approximate region where the error should be. In fact we use screenshotting in several places where need to test specific colours in the UI.
There is at least one place where we gave up though, and this is working with some of the system dialogs (like the file dialogs). They are immune to our programmatic inspection (being win32 dialogs rather than written in managed code), so selecting files in the same way the user does proved to not be worth the effort. We got round this by making all access to dialogs through a 'dialog manager', and mocking out only the system dialogs when running functional tests.
Ok - so we gave up a bit, but don't forget that it isn't the job of your functional tests to test the user interface toolkit or system you are working with. You want to test your application (the code you wrote) rather than testing other people's code.
There is an exception to this, and that is where you have to workaround bugs in the system or frameworks you are using.
Framework and System Bugs
Of course when you hit a hard to find problem it's easy to declare that it is a bug in the framework you are using rather than a bug in your own code. (You should always assume it is a bug in your own code first - select isn't broken.)
However, running a functional test suite involves starting and stopping the application many times within a single process - starting and stopping the event loop. In Windows Forms, once every few hundred times shutting the event loop down causes a rogue SystemError. We're pretty sure this is a genuine bug rather than our fault (we have spent hours inside a debugger before coming to this conclusion - and to be fair the event loop wasn't intended to be used this way), so we trap the error and ignore it.
SendKeys.SendWait can also be unreliable (failing almost 1% of the time - a problem when you use it hundreds of times in a test run). Fixing this (with a wrapper that checks the input it sends arrives correctly!) took a long time.
Random, Spurious and Impossible Failures
Solving random and spurious failures is probably still the most painful part of maintaining a test framework (shortly followed by working out how to test some things - we spent a whole day once working out how to test the target of shortcut in the start menu placed by our installer).
If you don't fix them you will find yourself habitually ignoring errors and failures in your test framework - and it loses its value.
Sometimes all you can do though is put in extra diagnostics so that if the same test fails again you get a bit more information. One useful technique is saving a screenshot on failure - that way when instant messaging programs pop up message boxes and steal focus from your application you can actually work out what happened...
Alternative Testing Tools
A selection from: http://pycheesecake.org/wiki/PythonTestingToolsTaxonomy#GUITestingTools
- guitest - for pyGTK
- pyGUIUnit - based on unittest for PyQt
- WATSUP - Windows Application Test System Using Python
- winGuiAuto - Low-level library for Windows GUI automation
- dogtail - Created by Redhat. Uses X11 accessibility framework
- ldtp - Also uses the X11 accessability framework
- kiwi ui testing - PyGTK UI Testing framework
Functional testing is definitely hard work, but the costs of not testing applications can be even greater. It's worth the pain.
For buying techie books, science fiction, computer hardware or the latest gadgets: visit The Voidspace Amazon Store.
Last edited Tue Aug 2 00:51:33 2011.