Is AI taking our tester jobs?

Over the years I have immersed myself in various areas of testing and through trial and error I have learned what works and what doesn’t work in terms of testing and quality. As a result, I have been able to speak in the field of testing in recent years at several conventions and seminars of TestNET, Noordertest and the National Test Day, and have provided many workshops for CIMSOLUTIONS customers.

We never stop learning and I learn every day, including the subject of testing with AI. So much is still in motion here, as if the big bang of testing with AI had just happened. The technology is becoming increasingly powerful and more and more appealing applications are appearing. Within CIMSOLUTIONS and with our partners and customers, we are intensively investigating what AI means for testing and how this will influence the playing field.

CIMSOLUTIONS does this internally through the very active test community that shares knowledge and thinks about the possibilities of AI via Special Interest Groups; research into it, its development and testing. CIMSOLUTIONS distinguishes itself in this with the Competence Center AI, in which various disciplines participate and in which all these aspects are addressed.

Future of the testing profession

AI and Machine Learning are developing rapidly, also in the testing field. AI test tools are becoming more advanced and frameworks to develop something yourself are becoming more accessible. This leads to the question of what the role of the tester will be once AI applications have been fully developed within the testing field. In other words: ‘will AI take our job as a tester?’ Together with the Working Group Testing with AI [ad.1], I have drawn up the  white paper “Testing with AI”  . The white paper has made it clear on this point that we do not have to be afraid of the latter.

What is clear is that the testing profession will change with the advent of AI, just as current test automation has done. And this will also have consequences for the role and skills of the tester. CIMSOLUTIONS has an active collaboration with the Testing with AI working group under the auspices of TestNET. This working group contains a large group of professionals who, from different perspectives, strongly advocate the development of testing with AI and, more recently, testing of AI. Below is a short excerpt from the Whitepaper.

How is the testing profession changing?

Dealing with self-learning AI testing tools

While trained AI testing tools will take a lot of work off our hands, we will have to find a solution for the fact that trained models will not behave 100% as expected, unlike programmed testing tools. After all, Machine Learning learns from examples, instead of following fixed rules. This means that the testing tools and our own configuration of them, will also have to be tested. It must be assessed on a case-by-case basis whether the effort of testing the AI ​​testing tool outweighs the benefit that the AI ​​testing tool brings.

Furthermore, test results need to be interpreted differently. A traditionally programmed test tool will crash if it can’t find a certain object. An AI test tool will likely find a workaround to complete the test and tell you that “the test is 87% successful.”

Choose from multiple tools

We asked the tool makers about their vision of the future of testing and tools. We asked them where they think it is going now. The people we spoke to are convinced that the use of testing tools with AI will increase in the near future.

Trends are that tools with AI will be used alongside traditional tools. Tools with AI are currently specialized for one specific task. In the future, it will become more common to use multiple testing tools alongside and with each other, thus having a palette of tools that covers the total range of testing activities.

Another trend is that smaller tool vendors will start offering their services together. For example, combining tools that can explore the application with tools that are good at smart image recognition. And possibly also that a tool like Selenium IDE can be used in combination with one or more AI tools.

The opposite is specialization as you now encounter with tools like test.ai. This tool focuses exclusively on web shops. The goal is to eventually offer a test service for web shops that can work completely independently. There are tools with AI that focus specifically on the app market. For apps it is also important to be able to run quickly and on as many platforms as possible.

Larger test frameworks are also starting to show interest in offering AI. Functionize, Appvance and Eggplant are examples of American test platforms that have been doing this for a while, each in their own way.

Testing becomes more creative

With the advent of tools like Selenium, repetitive testing has been removed from the tester and manual testing has been able to focus more on creative test situations. This will be reinforced as AI testing tools take over even more standard testing from us.

Furthermore, AI will help to write and execute many of the (technical) unit tests, so that the tester can focus more on functional and more complex non-functional testing. The manual, creative testing shifts to verifying the requirements and to delivering value to the organization.

How is the role of the tester changing?

The work of a tester will continue to exist. However, the activities required to provide insight into quality and risks will change. This will also change the necessary knowledge and skills. We see the following developments;

Clearly formulate objectives

This whitepaper shows that an AI can support us in all sorts of ways during the testing process. There is even a possibility that the AI ​​will explore itself. This means that we need to clearly indicate what we actually expect from the application and what we would like to have tested. In other words; more attention to requirements, acceptance criteria and test goals. Because in the most extreme scenario, the AI ​​could explore without any guidance and we would be busy for months assessing the results. A good set of expectations in advance can prevent this.

By the way, this does not mean that we should stop exploring the application ourselves, for example through exploratory testing. It may become even more important; the human tester does a good exploration of the playing field, we leave the large-scale and repeated testing to the AI.

Managing uncertainty in the testing process

Applying self-learning AI (or machine learning) ensures that we do not get absolute ‘good’ or ‘bad’ results. As a tester, we will have to interpret a confidence percentage that the AI ​​returns ourselves. So if the AI ​​finds a test case 87% successful, how do we report this and what follow-up action do we take? This will differ per project; it is human work to determine this and sometimes additional manual testing where necessary.

The test manager will also experience this uncertainty. If most tests pass at least 90% and some tests pass 75%, are we done? This is an interpretation that the test manager has to do and it comes back nicely to the previous theme; make sure there are clear expectations in advance. Moreover, it requires the ability to interpret large amounts of figures. And that brings us to the next point.

Knowledge of statistics

Interpreting large amounts of results and confidence percentages requires the tester and test manager to have some statistical knowledge. Even if ready-made tools claim to be happy to perform the interpretation on your behalf, it is important to have a well-founded opinion about all the figures that an AI returns. Should the tester and test manager then become a part-time data scientist or data engineer? That is still difficult to say, but some basic knowledge will certainly help you on your way.

Better knowledge and understanding of testing tools

There is a very diverse set of tools available. Regardless of whether they are ‘truly AI’, they all have different ways of adding value, in different environments. The trend that is already visible is that a large number of tools are being linked together to achieve the testing objectives. This will include AI-supported tools.

The expectation is that the automated part of the test process will become larger, compared to manual test and test support activities. Knowledge of which tools you can use, in which combination and especially for what purpose, will become even more important.

The testing profession is therefore still expanding. At CIMSOLUTIONS we see that customers are asking for expertise in the field of testing that they do not have in-house. CIMSOLUTIONS facilitates this and works closely with its customers to arrive at a suitable solution for an often very specific problem. But there is also an increasing demand for testing knowledge in the broad sense of the profession. I am currently setting up a general training course in functional testing that will be used specifically to involve the business with limited testing experience in a legacy or DevOps environment much closer to the development of software in order to achieve the required business value.

In other words: there will always be a role for someone who thinks about quality, who asks critical questions and who takes care of the communication about goals and priorities. To quote Jeremias Rößler [ad.2]; the chance that we will ever build software automatically is greater than that we will test it fully automatically. Because precisely when an AI builds an application itself based on a model, it is important that it is not tested according to that same model.

Marek Lof
Project Manager / Technical Consultant

[ad.1] Information about the Testing and AI working group
[ad.2] Dr. Jeremias Rößler – When Will AI Take My Job As a QA Manager