ChatGPT: A Game-Changing Way of Automation Testing

Officially published by OpenAI on 30th November 2022, ChatGPT recorded a surge of over 1 million users only in one week after launching. This technology enables genuine conversational interactions, allowing it to answer and explain concepts plainly, reply to follow-up questions, and acknowledge errors, just like a human. In fact, its capabilities have been applied in various aspects, such as writing an article, composing song lyrics, or even generating lines of code in different programming languages. 

It’s undeniable that ChatGPT is powerful. But do you think this AI technology can transform enterprise software testing? Let’s explore the possibility!

What is ChatGPT?

ChatGPT is a natural language processing chatbot that uses the Generative Pre-Trained Transformer 3 (GPT-3) model to generate human-like text for back-and-forth communication. Since it is trained on a massive amount of text data, the response tone from ChatGPT can be varied based on different scenarios. 

Here’re some of the inherent capabilities of this AI chatbot:

• Understand up to 95 natural languages and analyse words for context.
• Use a conversation model for a chat-like interaction with any user.
• Create creative responses, translate languages, summarise and paraphrase content, and analyse the sentiment of the text, all along with details.
• Remember previous questions and answers, leverage human feedback to follow directions and develop a more coherent conversation.

How Is ChatGPT Related to Automation Testing?

Among its capabilities, ChatGPT can automatically produce correctly formatted and relevant code in response to a simple natural language request. The advantage is that it can generate code in a variety of languages and frameworks and employ multiple built-in packages across those languages. For example, this AI chatbot can write Selenium code in many languages and can even be further queried if any adjustments are necessary.

So, can ChatGPT create code for use in test automation? The answer at this point is, “Yes, but to some extent.” Following are some possible use cases of ChatGPT in enterprise software testing:

• Building automation test cases for different scenarios
• Leveraging CI/CD to create a complex test automation pipeline
• Testing different kinds of enterprise applications and software using multiple microservices
• Delivering easy-to-understand code and clear instructions on how to use the developed code.

ChatGPT can help improve ROI in automation software testing by improving test coverage and increasing testing efficiency while reducing the time and effort required for test case creation and cutting down costs.

How to Improve Test Automation with ChatGPT?

Here are some of the typical testing activities that ChatGPT can perform:

1. Prepare Test Plan for Mobile Automation Testing

Imagine the following scenario: you want to arrange the test automation for a mobile banking app, for which you have created several features. In these cases, you may use ChatGPT to produce a test plan by providing the tool with information about the functional components and how they are utilised.

Prompt: “Prepare a test plan for an automation test to check the functionality of a login form on the mobile banking app. After opening the app, success if the user fills in the right user name, and password and presses the "Login" button."

ChatGPT:

2. Generate UI Automation Script in Different Languages

As mentioned above, ChatGPT's capability includes generating test scripts in various programming languages. You can request ChatGPT to generate a test case to open a specific website and perform an action on that with defined frameworks.

Cypress is a JavaScript-based test automation framework for web applications, so I’ll use ChatGPT to create a test script in this language.

Prompt: “Write Cypress/ Javascript code for opening the URL of the site https://kms-solutions.asia/ and Click on the Link “Blogs”  

ChatGPT: 

The answer from ChatGPT is designed to be straightforward, easy to understand, and flexible, allowing testers to quickly and effortlessly generate automated UI tests. By minimising the need for time-consuming manual testing, ChatGPT frees up engineers to work on higher-level projects while saving the company time and money. 

3. Design Sample Test Cases 

It is possible that high-level test cases may miss critical information that can mislead any QA expert. The AI-driven ChatGPT tool can build test scenarios and enhance its results on the repetition of the same question.

ChatGPT can generate test cases for a variety of requests, including:

• Sample data for a website login form
• Testing ideas for an eCommerce transaction
• Test data for a date picker

Prompt: “Prepare sample test cases to test a login form on mobile, in which has a user name, password, and login button.”

ChatGPT:

Additionally, ChatGPT can provide recommendations and best practices for designing test cases. Since it lists down all the necessary test cases that you need to perform, your testers can have ideas for testing even though they aren't familiar with that functionality.

4. Format Data

ChatGPT can be used to format any information and get it ready for any other format. For instance, you need to format the test plan to input data and track bug simply like Jira or Github; in this scenario, you may leverage ChatGPT by prompting further in the same thread:

Prompt: “Prepare test case table to enter into Jira tickets.”

ChatGPT

5. Classify Positive and Negative Test Scenarios

ChatGPT also can help identify some positive and negative test cases using any possible variants of data input (both valid and invalid).

Prompt: "Write positive test case/ negative test case for mobile login form"

ChatGPT:

Implementing ChatGPT to Low-code Testing 

ChatGPT can be perceived as a powerful tool for writing test cases with no code experience. This AI technology accepts natural language as input, allowing users to write in their natural cadence while still being understood, in contrast to template-based models that often depend on certain language structures or key phrases. 

In essence, using ChatGPT, QA teams can generate test automation scripts, classes, and functions and define both positive and negative test cases for complicated scenarios in automated testing.

Additionally, this AI tool can:

• Support multiple programming languages, libraries, and frameworks like Cucumber, Cypress (based on behaviour-driven development or BDD).  
• Whenever there is any change in the element locator or application structure, it can use the Page Object model to update the code. 
• Separate the code from the test cases, hence streamlining the maintainability of the test script.

As a low-code testing tool, ChatGPT makes it effortless to update and fix code. For instance, QA experts can request ChatGPT to change the previously created code to correct any bugs. Users can leverage ChatGPT to perform precise changes in the code by using its conversational model.

How ChatGPT Can Help in Enterprise Software Testing

• Test Scenario Generation: ChatGPT can assist in generating test scenarios by understanding the requirements, user stories, and business logic of the enterprise software. Testers can interact with ChatGPT, providing relevant information, and it can generate a wide range of test scenarios, including positive and negative test cases, boundary tests, and exceptional cases. This helps in achieving better test coverage.
• Test Data Generation: ChatGPT can generate realistic and diverse test data for enterprise software. By analyzing the data requirements and constraints, it can create relevant test datasets, including valid and invalid inputs, edge cases, and data combinations. This ensures thorough testing of the software's data-handling capabilities.

Limitations of ChatGPT for Automated Testing

Despite tons of potential to become a low-code solution for automation testing, ChatGPT also has some downsides, and it cannot replace the main job of an automtion tester. 

• Limited domain knowledge: ChatGPT has been trained on a vast amount of text data, but its knowledge and expertise in specific domains may be limited. This means that ChatGPT may not be able to provide accurate responses to certain test scenarios or situations that require deep domain knowledge.

• Lack of control over the test environment: ChatGPT is an external tool that runs independently of the application being tested. This means it has limited control over the test environment and may not be able to simulate conditions or scenarios that are necessary for some tests. There is the possibility of factual mistakes or disinformation in the training data ending up in the model.

• Inability to identify deprecated methods: The tool repeatedly used the “find_element_by_*” method to locate elements on the screen, even though that method is no longer functional. You can request ChatGPT to update the code with the correct method, but it needs the ability to recognise the issue.

• Lack of flexibility: As a pre-trained model, which means it may not be able to adapt to changes in the application being tested or accommodate custom requirements for specific testing scenarios. This can limit the flexibility and customizability of automated tests. 

For example, some websites require both the username and password entered on the same page before submitting, but others have you first input the username, then click a button like “Next” and then input the password. ChatGPT cannot flexibly change the test strategies and will need to be trained to perform the test, which can be done simply by the automation tester.

Although ChatGPT has some drawbacks, it is used regularly to reduce stress related to automation testing and create test cases faster.   

Conclusion

ChatGPT is undoubtedly a powerful natural language model with huge potential in automation testing. From conversation to code, it helps simplify the automation testing process, revolutionise overall growth and achieve top quality assurance. Several capabilities from ChatGPT, including advanced language processing, vast knowledge, and others, allow efficient and natural testing experience. 

However, as ChatGPT is in the beta phase so it has various limitations and cannot perform complicated tasks. Thus, this tool is believed to be “a future of automation testing”, but now, it is mainly the QA experts' job. 

Further Reading: ChatGPT and Developers: Friend or Foe?

If you're interested in our Digital Testing content and services, don't forget to leave your information here!