Increasing Automation ROI - Reuse of Automation

Both creating and maintaining an automation framework or a suite of automated tests requires a large investment and buy-in from management.  As with all investments managers are seeking a return on their investment (ROI).  It is often said that the ROI from automation increases after each execution.  Unfortunately this is not always the case; in the case where the product is stable and the test does not alter it's data running the tests multiple times would be unlikely to discover any bugs in the system thus only providing a sense of security.  Without the data of the test or system under test (SUT) changing a test is unlikely to find any new bugs so the question becomes how can we get value from our tests?

Although difficult to achieve randomizing the data of a test can increase it's value.  Imagine a test as trail through a forest, once created with set data each run follows the same trail, not exploring the unknown parts of the forest.  Bugs in this metaphor will often be found in the unexplored areas.  Changes in the data used by the test can expand the coverage of the test and increase it's value.

In a similar fashion when tests are executed in random or the test actions themselves execute in a random order (see model based testing) additional value can be derived from the same tests.

The easiest way though to increase the ROI of your automation suite is to change the SUT.  This could mean running your test against:

• Different operating systems.  Especially valuable now in mobile testing where operating systems are changing more rapidly.
• Different browsers.
• Different devices.
• Different integrated environments/components.

As a manual tester I've done browser upgrades and never has my desire to come into work been dampened to the same extent.  So a side benefit to executing automation against SUT is that it reduces the mindnumbing arduous tasks that need to be completed by manual testers.