On the Return-on-Investment (ROI) of Test Automation

Posted by KMS Solutions on Mar 15, 2021 3:37:04 PM

Using Self-service Data Analytics tool for your Data Visualization (1)

The idea of Automation Testing of course sounds exciting. What could be better than writing a script only one time, and then re-executing it for up to hundreds, even thousands, of times? 

With Automation Testing, your team can increase test coverage, find bugs earlier, and still be able to use manual testing where it truly delivers values. 

But most of the time, that’s just not enough to get everyone to buy into Test Automation. This is especially true for top management and the CFO who would ask for a convincing ROI analysis for such a major investment like Automation Testing.

ROI Formula for Test Automation

The formula you use to calculate the ROI of Test Automation is just as simple as that of any other kinds of investments: you divide the subtraction of Gain from Investment from Cost of Investment by Cost of Investment and then you multiply the result by 100. The formula looks like this:

ROI= (Gain from Investment - Cost of Investment)Cost of Investment  x 100

In the context of Automation Testing, Gain from Investment refers to:

  • Speed: How faster the Software Testing process and the Software Development process become AFTER you adopt Automation Testing
  • Quality: Whether some or all your product's quality attributes (such as performance, compatibility, or security) are improved AFTER you adopt Automation Testing
  • Cost Savings: How much you are able to save AFTER you adopt Automation Testing

Whereas Cost of Investment is those associated with the tools (Test Automation platforms, Test Management System, or test reporting tool) and resources (man-hours) that go into making Automation Testing work.

Key Input Parameters

To calculate the ROI of Automation Testing, there are some key input parameters you want to put into consideration.

  • The number of Test Cases: The total test cases you create, execute, and manage.
  • Duration of Test Design: the time it takes you to design tests, including both automated and manual tests.
  • Duration of Test Execution and Data Preparation: the time it takes you to execute tests and prepare test data.
  • Analysis and Reporting Effort: How much time and effort you invest into test analysis and test reporting.
  • Total Cost of Labor, Tools, and Infrastructure: the costs associated with hiring new professionals (Automation Testers), purchasing new tools, and setting up new systems for Automation Testing.
  • Level of Reusability and Redundancy: Whether the tests can be re-used for multiple platforms, and how high is the test redundancy.
  • Ratio of Test Stages: how much time and effort that each test stage requires.
  • Number of Releases and Test Cycles: the total quantity of builds, releases, and test cycles that the team creates.

3 Tips for Calculating ROI of Automation Testing

  • Breaking down tasks: always divide the Automation Testing process into different smaller tasks for better execution and reporting. Some major tasks to consider include Planning, Design, Implementation & Maintenance, Execution, Analysis & Reporting
  • Considering costs: also divide the Cost of Investment we discussed earlier into different types for more accurate calculation. These costs include one-time costs (infrastructure, setting up, configuration, installation, etc.), Visible costs (training, collaboration, process), Hidden costs (Training, collaboration, process), and Labor costs (manual vs. automation).
  • Optimizing: Identify areas of Automation Testing that need to be optimized. These areas include the level of redundancy to be reduced, the level of reusability to be improved, the number of instances to be increased, and the number & types of test cases to select.

Example of ROI of Test Automation

In what follows, all the investments, costs, and savings of a real Automation Testing project are presented. 

The table below illustrates the investment of a real Automation Testing project:

Complexity Medium
Test Coverage 73.81%
# of Manual Test Cases 2100
# of Automated Test Cases 1550
Avg. Execution Time of Manual Test Cases (min) 45
Avg. Execution Time of Automation Test Cases (min) 5
Avg. Development Time of Automation Test Cases (min) 15
Avg. Development Time of Manual Test Cases (min) 120
Number of Test Server Instances 12
Nightly Run Yes
Number of Test Cycles 20
License Cost $759
Initial Infrastructure Cost $0

Then we compare that to the efforts going into Manual Testing, and how much the team saved using Automation Testing.

  Manual (hrs) Automation  (hrs) Saving ($)
Initial Setup & Management 0.00 8.00 -200.00
Testing Initiation (Training,...) 0.00 24.00 -600.00
Test Planning 104.00 264.00 -4000.00
Test Design 0.19 3410.00 -85245.16
Test Execution 1395.00 18.30 34417.53
Test Regression 27900.00 150.00 693732.64
Reporting 56.00 205.00 -3725.00

It’s clear that there are areas where test automation costs more than automation testing such as Test Design and Reporting, and those where Manual Testing costs none while Automation Testing incurs new efforts, such as Initial Setup & Management and Testing Initiation.

But in the end, the team has saved up to $610,944.05

Annual Licensing Cost (x4) -$3,036
Current Effort-hours on Manual Testing 29455.19
Project Effort-hours on Automation Testing 4079.19
Efficiency Gain 86%
Cost with Manual Testing $736,379.84
Cost with Automation $125,435.79
Total Savings $610,944.05

The graph and chart below illustrate some interesting facts about Test Automation. In about the first 10 months of starting, Automation Testing costs way more than Manual Testing. But after this period, Automation Testing starts to show its potential, where the efforts and money saved skyrocketing.

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18
Auto. Effort 217 221 226 230 234 239 243 247 252 256 260 265 269 273 278 282 286 291
Manual Effort 25 50 75 100 125 150 175 200 250 250 275 300 325 350 375 400 435 450
Saved Effort -192 -171 -151 -130 -109 -89 -68 -47 -27 -6 15 35 56 77 97 118 139 159

Automation Testing vs Manual Testing

How to Maximize ROI of Automation Testing

There are 3 areas - criteria to consider: Test Coverage, Speed of Test, and Cost Savings

#1 Test Coverage

Key metrics to consider include Quantity and Quality of tests

Quantity:

  • Percentage of test coverage: How much testing that Automation Testing helps to increase
  • Percentage of automatable tests: How many tests to be automated
  • New test scripts growth trend: the number of new test scripts that are created every day

Quality:

  • Code smells
  • Code duplication
  • Size: classes, comments, lines of code

You can analyze product quality using specialized tools such as SonarQube.

Best Practices

  • Define criteria to select test cases to be automated: customer risk (the impacts of the tests to end-users), the value of the test (only automate tests that are valuable, not to automate every kind of test), cost efficiency (there are tests that require days to weeks to write automate scripts, to configure, to schedule, or access servers, that otherwise should be tested manually), history (components or features that are used the most by end-users).
  • Don’t try to automate 100%. Should stay manual or exploratory in some cases: usability and user interface
  • Leverage CI to integrate new automated tests as a part of your testing process
  • Pay more attention to maintaining existing test scripts, don’t just focus on new test cases
  • Integrate code analysis tool to ensure that your code and scripts have high quality
  • Test automation as a part of the Definition of Done (DoD)

#2 Speed of Test

Key metrics to consider include Speed and Stability of Execution, and Test Redundancy and Reusability.

Speed and Stability:

  • Test execution duration: Make sure the time it takes to execute is as short as possible. It makes no sense when manual testing takes you only 30 mins while automation testing takes 60mins
  • Test flakiness or flaky rate
  • Failed test trend
  • Mean time to bug detection
  • Mean time to diagnosis

Test Redundancy and Reusability:

  • Time to develop and execute redundant tests
  • Time to implement reusable tests
  • Coverage cross environments and types of tests

Best Practices

  • Run in parallel and distributed testing. For example, instead of running 1000 test cases in 8 hours, we could split them into different chunks to run them in 30 minutes
  • Split test suites into small chunks in accordance with their purposes (smoke test, regression test, cross-browser test, etc.). 
  • Enable test scripts reusability, stability, and maintainability by adopting code techniques and design patterns
  • Leverage test management tools to search for duplicated tests and test reuses
  • Utilize test reports with AI/ML to support the analysis of test results
  • Consider environment provisioning, to replicate multiple environments to support Automation Testing

#3 Cost Savings

Key Costs include Cost of Lab, Cost of Tool/Framework/Staff, and Cost of Defects and Delays

Cost of Lab:

  • The number of servers, workstations, devices
  • Cost of Lab maintenance

Cost of Tool/Framework/Staff:

  • Cost of number of licenses
  • Cost of building a framework from scratch
  • Cost of maintaining a framework
  • Labor cost: manual vs. automation; the cost of training

Cost of Defects and Delays:

  • Defect leakage
  • Overall build quality
  • Cost of delay (CoD)
  • Leverage CI and environment provisioning to optimize infrastructure cost
  • Thoroughly conduct tool/framework evaluation:  organizational and technical fits
  • Invest in experienced automation testers 
  • Test early, test often and test comprehensively with tools, different levels and with real-world business workflows

If you want to learn more about Software Testing and how it can help you accelerate your Digital Transformation, subscribe to our newsletter.

Subscribe to our Newsletter

Topics: Software Testing, Automation Testing

Our Editorial Mission

With KMS Solutions Blogs, we would like to provide you precious content and offers for the top-notch solutions that work for your business

Recent Posts

Subscribe Here!