Understanding the Return-on-Investment (ROI) of Test Automation
The idea of Automation Testing of course sounds exciting. What could be better than writing a script only once and then re-executing it up to hundreds, even thousands of times?
With Automation Testing, your team can increase test coverage, find bugs earlier, and still be able to use manual testing where it truly delivers values.
But most of the time, that’s just not enough to get everyone to buy into Test Automation. This is especially true for top management and the CFO who would ask for a convincing ROI analysis for such a major investment as Automation Testing.
ROI Formula for Test Automation
The formula you use to calculate the ROI of Test Automation is just as simple as that of any other kinds of investments: you divide the subtraction of Gain from Investment from Cost of Investment by Cost of Investment and then you multiply the result by 100. The formula looks like this:
ROI= (Gain from Investment - Cost of Investment)⁄Cost of Investment x 100
In the context of Automation Testing, Gain from Investment refers to:
- Speed: How faster the Software Testing process and the Software Development process become AFTER you adopt Automation Testing
- Quality: Whether some or all your product's quality attributes (such as performance, compatibility, or security) are improved AFTER you adopt Automation Testing
- Cost Savings: How much you can save AFTER you adopt Automation Testing
Whereas Cost of Investment is those associated with the tools (Test Automation platforms, Test Management System, or test reporting tool) and resources (man-hours) that go into making Automation Testing work.
Key Input Parameters
To calculate the ROI of Automation Testing, there are some key input parameters you want to put into consideration.
- The number of Test Cases: The total test cases you create, execute, and manage.
- Duration of Test Design: the time it takes you to design tests, including both automated and manual tests.
- Duration of Test Execution and Data Preparation: the time it takes you to execute tests and prepare test data.
- Analysis and Reporting Effort: How much time and effort you invest into test analysis and test reporting.
- Total Cost of Labor, Tools, and Infrastructure: the costs associated with hiring new professionals (Automation Testers), purchasing new tools, and setting up new systems for Automation Testing.
- Level of Reusability and Redundancy: Whether the tests can be re-used for multiple platforms, and how high is the test redundancy.
- Ratio of Test Stages: how much time and effort that each test stage requires.
- Number of Releases and Test Cycles: the total quantity of builds, releases, and test cycles that the team creates.
3 Tips for Calculating ROI of Automation Testing
- Breaking down tasks: always divide the Automation Testing process into different smaller tasks for better execution and reporting. Some major tasks to consider include Planning, Design, Implementation & Maintenance, Execution, Analysis & Reporting
- Considering costs: also divide the Cost of Investment we discussed earlier into different types for more accurate calculation. These costs include one-time costs (infrastructure, setting up, configuration, installation, etc.), Visible costs (training, collaboration, process), Hidden costs (Training, collaboration, process), and Labor costs (manual vs. automation).
- Optimizing: Identify areas of Automation Testing that need to be optimized. These areas include the level of redundancy to be reduced, the level of reusability to be improved, the number of instances to be increased, and the number & types of test cases to select.
Example of ROI of Test Automation
All the investments, costs, and savings of a real Automation Testing project are presented in what follows.
The table below illustrates the investment of a real Automation Testing project:
|# of Manual Test Cases||2100|
|# of Automated Test Cases||1550|
|Avg. Execution Time of Manual Test Cases (min)||45|
|Avg. Execution Time of Automation Test Cases (min)||5|
|Avg. Development Time of Automation Test Cases (min)||15|
|Avg. Development Time of Manual Test Cases (min)||120|
|Number of Test Server Instances||12|
|Number of Test Cycles||20|
|Initial Infrastructure Cost||$0|
Then we compare that to the efforts going into Manual Testing, and how much the team saved using Automation Testing.
|Manual (hrs)||Automation (hrs)||Saving ($)|
|Initial Setup & Management||0.00||8.00||-200.00|
|Testing Initiation (Training,...)||0.00||24.00||-600.00|
There are areas where test automation costs more than automation testing such as Test Design and Reporting, and those where Manual Testing costs none while Automation Testing incurs new efforts, such as Initial Setup & Management and Testing Initiation.
But in the end, the team has saved up to $610,944.05
|Annual Licensing Cost (x4)||-$3,036|
|Current Effort-hours on Manual Testing||29455.19|
|Project Effort-hours on Automation Testing||4079.19|
|Cost with Manual Testing||$736,379.84|
|Cost with Automation||$125,435.79|
The graph and chart below illustrate some interesting facts about Test Automation. In about the first 10 months of starting, Automation Testing costs way more than Manual Testing. But after this period, Automation Testing starts to show its potential, where the efforts and money saved skyrocketing.
How to Maximize ROI of Automation Testing
There are 3 areas - criteria to consider: Test Coverage, Speed of Test, and Cost Savings.
#1 Test Coverage
Key metrics to consider include Quantity and Quality of tests
- Percentage of test coverage: How much testing that Automation Testing helps to increase
- Percentage of automatable tests: How many tests to be automated
- New test scripts growth trend: the number of new test scripts that are created every day
- Code smells
- Code duplication
- Size: classes, comments, lines of code
You can analyze product quality using specialized tools such as SonarQube.
- Define criteria to select test cases to be automated: customer risk (the impacts of the tests to end-users), the value of the test (only automate valuable tests, not to automate every kind of test), cost efficiency (some tests require days to weeks to write automate scripts, to configure, to schedule, or access servers, that otherwise should be tested manually), history (components or features that are used the most by end-users).
- Don’t try to automate 100%. Should stay manual or exploratory in some cases: usability and user interface
- Leverage CI to integrate new automated tests as a part of your testing process
- Pay more attention to maintaining existing test scripts, don’t just focus on new test cases
- Integrate code analysis tool to ensure that your code and scripts have high quality
- Test automation as a part of the Definition of Done (DoD)
#2 Speed of Test
Key metrics to consider include Speed and Stability of Execution, and Test Redundancy and Reusability.
Speed and Stability:
- Test execution duration: Make sure the time it takes to execute is as short as possible. It makes no sense when manual testing takes you only 30 mins while automation testing takes 60mins
- Test flakiness or flaky rate
- Failed test trend
- Meantime to bug detection
- Meantime to diagnosis
Test Redundancy and Reusability:
- Time to develop and execute redundant tests
- Time to implement reusable tests
- Coverage cross environments and types of tests
- Run in parallel and distributed testing. For example, instead of running 1000 test cases in 8 hours, we could split them into different chunks to run them in 30 minutes
- Split test suites into small chunks per their purposes (smoke test, regression test, cross-browser test, etc.).
- Enable test scripts reusability, stability, and maintainability by adopting code techniques and design patterns
- Leverage test management tools to search for duplicated tests and test reuses
- Utilize test reports with AI/ML to support the analysis of test results
- Consider environment provisioning, to replicate multiple environments to support Automation Testing
#3 Cost Savings
Key Costs include Cost of Lab, Cost of Tool/Framework/Staff, and Cost of Defects and Delays
Cost of Lab:
- The number of servers, workstations, devices
- Cost of Lab maintenance
Cost of Tool/Framework/Staff:
- Cost of number of licenses
- Cost of building a framework from scratch
- Cost of maintaining a framework
- Labor cost: manual vs. automation; the cost of training
Cost of Defects and Delays:
- Defect leakage
- Overall build quality
- Cost of Delay (CoD)
- Leverage CI and environment provisioning to optimize infrastructure cost
- Thoroughly conduct tool/framework evaluation: organizational and technical fits
- Invest in experienced automation testers
- Test early, test often and test comprehensively with tools, different levels, and real-world business workflows
If you’re looking for world-class testing services, KMS Solutions is the place to contact. With more than 12 years of experience in helping various leading companies in the BFSI sector move to automated testing, increase test coverage, and set up testing flows, we can help your business address the testing challenges effortlessly.