A Guide to Pain Free Automated Regression Testing

For automation to work most effectively, it needs to be embedded within the culture of an organisation rather than adopted for a one-off project. Time must be allocated to choosing the correct tool, for training, for experimenting and for learning and adopting what works best. Given the investment required to make the most of test automation, it's rare for a return to be realised on the back of a one-off project. 

Having said that, test automation doesn't suit all projects. It is effective to introduce test automation in large to medium-scale projects involving several sub-systems (e.g., enterprise web applications).  

For small and short-term projects (e.g., simple mobile apps), automation usually doesn't make sense as the time required for automation may well exceed the time taken to manually execute the tests.

2. When to start scripting?

Experience shows that automating regression tests is most effective when it is based on proven manual test cases that have been successfully executed at least once before. Having an existing manual test to work from reduces the risk that the automated test case will not execute, or worse will verify the wrong functionality once completed. 

While admittedly you do not always need to have a tried and tested manual test script to base your automation on, in my experience that is what provides the most efficient, reliable and cost-effective outcome.

3. Which Regression tests should you automate first?

As mentioned above, we can construct regression test scripts based on existing manual regression test cases that have been successfully executed a number of times previously. However, within that test suite, which should you automate first?

There are no hard and fast rules on this, but my thoughts are below:

1. Choose the areas of the system that are most stable and least susceptible to change. Creating automated tests is a significant investment and the last thing that you'd want is to have to keep updating and maintain the tests if you could avoid this.
2. Ideally select the highest risk features. These are the features which would impact you the most if they were to fail in Production and therefore automatically become the best candidates for regression testing and for automation.
3. Automate functionality which requires to be tested with many different variations of data. This could be an interactive web front end, or an inquiry form which a customer must fill in. Once you have written the body of the automated test, it is little additional work to copy this test and insert a different set of data combinations. Automating these types of tests are likely to save you a lot of time for little additional effort.

Environment Shakedown or Smoke Tests

Although perhaps not part of your Test Regression suite, another set of tests that you may want to automate is the Environment Shakedown or Smoke Tests. I've worked on many projects and programmes where the test team were asked to shakedown the test environment daily to ensure that all the functionality and applications were working as intended. This again is an onerous task which ideally should be undertaken before any testing commences that day and as quickly as possible. If the environment is working as designed, then the last thing you'd want to do is to hold up testing unnecessarily. The requirement for regular and speedy execution makes this the perfect candidate for test automation. 

4. Common obstacles to successful Automation

Through experience I can positively say my list of obstacles that may hamper automation progress is not insurmountable and is unlikely to grow exponentially with each and every project. Many of the issues uncovered during test automation would also be a problem in a manual test arena and therefore cannot solely be called out as obstacles to test automation. Inconsistent documentation, a lack of change control and poorly organised tests would be an issue regardless of what approach you took to testing.

However, it is true that the impact of some issues is amplified in an automated testing framework:

Maintenance efforts

An automated regression test suite won't be valid forever. To be efficient and to maintain accuracy, a test suite should reflect all changes introduced to software with the shortest delays possible. Test teams should regularly review automated regression test suites to detect obsolete test cases that no longer fit a project. 

Systems which are best suited for test automation will typically be more mature systems which are prone to frequent small changes rather than large scale changes. There will always be an element of maintenance required for automation tests to reflect ongoing change on the system. However, if the level of change is sizeable and it then takes more effort to update the tests than it would take to re-run them manually, then it's likely that test automation will be abandoned. 

Test Data

Having a stable test data set which can be reused time and again while still providing accurate test results is key to the success of automated testing. Regularly changing the test data in an automated test suite is sometimes more time consuming than if the tests were executed manually. 

A stable and reliable Test Environment

To get the most benefit out of Test Automation you must also have stable and reliable test environment - While this is also true for manual testing, a good manual tester can often skirt around issues in a test environment by selecting and running the tests in areas of the environment he knows are working that day. This is much more difficult and time consuming in an automated testing suite. The goal should be to 'press the start button' on your automated regression suite and watch all the tests run through to completion. Any benefits from executing automating tests in a shorter time frame will quickly be eroded if you must constantly reschedule tests because areas of the environment were not working. With this in mind, it is advisable to avoid tests which require a wait. 

In Summation

Of course, automating tests does not mean we are eliminating the Testers role. There will always be a place for manual testing within a project team, as not every test associated with  feature can be automated and not every project is suitable for automated testing. As a Tester, automation is about making our lives easier, by using it to overcome problems such as time and testing more efficiently to ensure quality is maintained not just within the application being developed but also within the testing process.

If you want to experience the benefits of automated testing but without the pain, contact Agenor Technology and arrange a meeting with one of our Testing Services experts today.