Tricentis and three contributors to the survey project recently published results that provide new insights into global trends in open source testing.
The study focused on the open source tool testing industry, not how or if companies and software developers test their code. This is a problem facing both open source and commercial or proprietary software.
The results revealed that the lack of technical skills is the main obstacle to adopting open source tools, with 30 percent of the organizations surveyed saying they lack the skills to adopt open source tools, according to Kevin Dunne, senior vice president for strategic initiatives at Tricentis.
Cost is the single biggest driver for adopting open source tools, with 39 percent of respondents saying it is the number one advantage. Other factors, such as community support (16 percent) and ease of customization (14 percent), are also very high.
The results show open source test tools that get support as viable options even for larger, security-focused companies. But some respondents still face opposition to open source security and support within their organizations.
“With the current economic climate affecting tool budgets in many organizations, we felt it was essential to further examine developments related to free and open source test tools,” Dunne told LinuxInsider.
Open source testing tools are often used by those identified as testers or quality assurance officers. However, what was surprising was the low level of involvement of developers and engineers within this survey, Dunne noted.
Respondents had an average of nearly nine years of engineering experience. The majority (52 percent) hold quality control roles. Software development engineers comprised 14 percent of respondents. Development engineers (8 percent) and other roles (26 percent) included the rest of the survey participants.
Tricentis supports one of the largest open source test communities in the world.
The survey was conducted in April 2020 and was distributed online through SurveyMonkey. Sponsor companies promoted the survey via mailing lists, social media and test media collaborations, Dunne said.
Results across the board
Tricentis brought together a group of experts in each category or test domain to investigate the results. The panels focused on behavioral development (BDD), functional tests and load / performance tests.
Here are the key results in each area, according to Dunne:
- Functional Tests: 84 percent of organizations still employ dedicated testers to perform functional test automation, despite an alleged tendency to have developers replace dedicated testers to build functional test automation.
- BDD: Only 21 percent of organizations consider themselves experts when it comes to BDD. 51% of organizations are developing their test scenarios in a “Data-when-then” format, but there is still room for growth when it comes to fully automating these tests and organizing that code for reuse.
- Load / Performance Tests: While performance is critical to onboarding and customer retention, only 56% of companies are performing load / performance tests on a regular basis. Adequate time, training, technical skills and test environments remain the main obstacles to more frequent load tests / performance.
Offshore tests still seem popular and thriving, with 61 percent of respondents coming from Asia. Most are located in India and used with large system integrators.
“With COVID-19 allowing everyone to work remotely, the previous idea that agile development would have shifted attention away from offshoring seems not to be the case,” he said.
Non-universal software test
“The Tricentis survey gives us a good insight into the trends affecting open source testing and testing tools. I think it’s a step in a positive direction, particularly with regards to awareness of the problems,” he told LinuxInsider.
The roadblocks highlighted in the results can be reduced to a reluctance to change and a general lack of competence. The question of competence highlights a wider problem with software engineering today, it offered. Our tools are more customizable than they are easy to use.
Trends in industry practices for open source testing fluctuate like pendulums. He sees a greater shift towards the use of open source test tools.
“But when it becomes clear that we need skills and the overheads associated with using these open source tools are better known, I think we will see a return to the adoption of enterprise-level tools,” said Hatch.
Key role of tests
The survey shows a trend towards automation. Multiple companies reported using automated functional tests versus manual testing approaches (50 percent). A surprising result, according to the report, is the low number (31 percent) of respondents who perform unit tests of their code.
Other results in this category include:
- Coded tests (67 percent)
- Exploratory tests (32 percent)
- Load test (23 percent)
- Safety test (14 percent)
Benefits and importance identified
The main reason for using open source test tools instead of commercial or proprietary tools was the cost (39 percent) according to the respondents. The second most common reason for use was a three-way link for community support (16 percent) and integration with tools / workstations (15 percent) and ease of customization (14 percent).
The least popular reasons for using open source test tools were freedom from vendor blocking (7 percent) and scalability (6 percent).
The importance of open source was another strong trend with 92 percent of responding organizations agreeing that having an open source functional test solution was important (34 percent) or very important (58 percent) for them. The remaining 8 percent of respondents are tied to 4 percent each for open source, being important only in relation to certain conditions or not at all important.
The results were similar when respondents rated the importance of an open source solution for performance and / or load testing. Again, 92 percent found it very important (54 percent) or important (38 percent). Only 3 percent thought it important in certain conditions and 5 percent said it wasn’t important.
Only 35 percent of companies specifically interviewed performance engineers by performing performance tests. Instead, quality assurance and others were responsible for functional tests and performance.
- Quality test / warranty – 47 percent
- Site reliability engineering – 3 percent
- Performance Engineering – 22 percent
- Development – 12 percent
- Operations – 8 percent
- None – 7 percent
- Other -1 percent
Roadblocks for testing
Performance and / or load testing with scarce shared resources. Most of the impediments were technical training (39 percent) and time (38 percent).
Roadblocks have increased through test environments (34 percent), test data (23 percent), network or security (20 percent), tool complexity (18 percent) and tool costs (17 percent).
The weather turned out to be a critical block with agile development according to 50 percent of respondents. Technical training and skills counted to 40 percent.
Other impediments to testing in agile development include: test environments (38 percent), test data (31 percent), tool costs (27 percent), tool complexity (25 percent), and network or security (16 percent).
Respondents indicated intensive use of the Given When Then format (51 percent) to increase automation rates and reuse them. The survey also revealed that the use of Given When Then to align teams on requirements and simplify documentation is still lagging behind.
Other category results
Position preference testing is important. Most companies (65%) are still running some form of testing outside their construction pipeline. According to the researchers, the remaining 35 percent will decrease, as the adoption of CI / CD will require that tests be tightly integrated within the pipeline to automate implementation decisions.
The full report can be found Here.