Articles
| Open Access |
DOI: Integrating AI Tools Into the Continuous Testing Process.
Vladyslav Korol , Software Developer Engineer in Test Penn Interactive Miami, USAAbstract
This paper utilizes AI tools to enhance the ongoing test cycle in a DevOps environment, thereby creating Metabase. This data architecture is robust and scalable, supporting a highly responsive release process. The project is vital since the releases have become more frequent; standard automation has already reached its limit, increasing the costs of maintaining scripts and, consequently, resulting in a significantly higher total cost due to the discovery of many more defects. The novelty of this work is grounded in an approach to choosing and applying AI tools through comparative analysis over available commercial and open-source platforms, supported by content analysis of empirical use cases and quantitative assessments from industry reports. Herein, a methodology is presented that consists of architectural solutions for data lake organization, continuous model training scenarios, and ML endpoints integrated into the CI/CD pipeline, which hosts Predictive Test Selection, as well as self-healing and test-case prioritization mechanisms. It narrates the creation of the prompt-engineer position and the connections between QA/ML experts and organizational facets. This paper discusses the application of clear AI measures for risk assessments. The final calls shown here indicate that intelligent automation enables reducing a regression set to a barely necessary size while maintaining 99.9% bug detection and minimizing false alert failures. This, in turn, leads to improvements in MTTR and TCO quality. TestPilot and FlakeFlagger verify Meta’s practices; furthermore, it is anticipated that forecasts will retain a CAGR of 20.9% in the global AI testing market growth. Solution maturity, encompassing both SaaS and on-premises models, offers a flexible choice to regulated industries. Metabase architecture is shown in which raw and processed data are kept separately to ensure timely model retraining as well as to minimize computational costs. This article will be helpful for software architects, QA managers, DevOps teams, and ML engineering specialists involved in building scalable and resilient testing architectures.
Keywords
References
“State of CI/CD Report 2024: The Evolution of Software Delivery Performance,” CD Foundation. https://cd.foundation/state-of-cicd-2024/ (accessed May 01, 2025).
M. S. Bari, A. Sarkar, and S. A. M. Islam, “Ai-augmented Self-healing Automation Frameworks: Revolutionizing Qa Testing with Adaptive and Resilient Automation,” Advanced International Journal of Multidisciplinary Research, vol. 2, no. 6, Dec. 2024, doi: https://doi.org/10.62127/aijmr.2024.v02i06.1118.
Open Text Corporation, “World Quality Report 2024 shows 68% of Organizations Now Utilizing Gen AI to Advance Quality Engineering,” PR Newswire, Oct. 22, 2024. https://www.prnewswire.com/news-releases/world-quality-report-2024-shows-68-of-organizations-now-utilizing-gen-ai-to-advance--quality-engineering-302282709.html (accessed May 03, 2025).
M. Machalica, A. Samylkin, M. Porth, and S. Chandra, “Predictive Test Selection,” arXiv, Jan. 2018, doi: https://doi.org/10.48550/arxiv.1810.05286.
M. Schäfer, S. Nadi, A. Eghbali, and F. Tip, “An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generation,” arXiv, Sep. 2023, doi: https://doi.org/10.48550/arXiv.2302.06527.
A. Alshammari, C. Morris, M. Hilton, and J. Bell, “FlakeFlagger: Predicting Flakiness Without Rerunning Tests,” 2021 IEEE/ACM 43rd International Conference on Software Engineering (ICSE), May 2021, doi: https://doi.org/10.1109/icse43902.2021.00140.
F. Ricca, A. Marchetto, and A. Stocco, “A Multi-Year Grey Literature Review on AI-assisted Test Automation,” arXiv (Cornell University), Aug. 2024, doi: https://doi.org/10.48550/arxiv.2408.06224.
“Gartner Market Guide for AI-Augmented Software-Testing Tools 2024,” Tricentis, Mar. 04, 2024. https://www.tricentis.com/resources/gartner-ai-market-guide (accessed May 07, 2025).
D. Giudice, C. Gardner, A. Cornwall, A. Lozada, and K. Hartig, “The Forrester WaveTM: Continuous Automation Testing Platforms, Q4 2022,” 2022. Accessed: May 10, 2025. [Online]. Available: https://www.tekmarkgroup.com/wp-content/uploads/2023/10/The-Forrester-Wave%E2%84%A2-Continuous-Automation-Testing-Platforms-Q4-2022.pdf
“Applitools Company Overview, Contact Details & Competitors,” LeadIQ, 2024. https://leadiq.com/c/applitools/5a1d9d422300005b008d1c07 (accessed May 14, 2025).
“Mabl company profile,” Tracxn, Aug. 17, 2021. https://tracxn.com/d/companies/mabl/__b9wljjy7_SAZSPVCj34bz9RedSOIojaJFYuWPFn4iBk (accessed May 16, 2025).
“Tricentis Recognized in the February 2024 Gartner® Market Guide for AI-Augmented Software-Testing Tools,” Tricentis, Mar. 06, 2024. https://www.tricentis.com/news/tricentis-2024-gartner-market-guide-for-ai-augmented-software-testing-tools (accessed May 17, 2025).
“2023 Software Testing and Quality Report,” Testrail, 2023. Accessed: May 16, 2025. [Online]. Available: https://content.testrail.com/hubfs/Downloadables/TestRail-2023-Software-Testing-and-Quality-Report.pdf
S. Azzabi, Z. Alfughi, and A. Ouda, “Data Lakes: A Survey of Concepts and Architectures,” Computers, vol. 13, no. 7, p. 183, Jul. 2024, doi: https://doi.org/10.3390/computers13070183.
I. Prapas, B. Derakhshan, A. R. Mahdiraji, and V. Markl, “Continuous Training and Deployment of Deep Learning Models,” Datenbank-Spektrum, vol. 21, no. 3, pp. 203–212, Nov. 2021, doi: https://doi.org/10.1007/s13222-021-00386-8.
M. Sulaiman, M. Farmanbar, S. Kagami, A. N. Belbachir, and C. Rong, “Online deep learning’s role in conquering the challenges of streaming data: a survey,” Knowledge and Information Systems, Feb. 2025, doi: https://doi.org/10.1007/s10115-025-02351-3.
B. Nushi, E. Kamar, and E. Horvitz, “Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failure,” arXiv (Cornell University), Sep. 2018, doi: https://doi.org/10.48550/arxiv.1809.07424.
Datadog Inc., “Datadog’s State of Cloud Costs 2024 Report Finds Spending on GPU Instances Growing 40% as Organizations Experiment with AI,” PR Newswire, Jun. 13, 2024. https://www.prnewswire.com/news-releases/datadogs-state-of-cloud-costs-2024-report-finds-spending-on-gpu-instances-growing-40-as-organizations-experiment-with-ai-302172281.html (accessed May 25, 2025).
“AI-enabled Testing Market Size, Share & COVID-19 Impact Analysis,” Fortune Business Insights, May 12, 2025. https://www.fortunebusinessinsights.com/ai-enabled-testing-market-108825 (accessed May 26, 2025).
Article Statistics
Copyright License
Copyright (c) 2025 Vladyslav Korol
This work is licensed under a Creative Commons Attribution 4.0 International License.
Authors retain the copyright of their manuscripts, and all Open Access articles are disseminated under the terms of the Creative Commons Attribution License 4.0 (CC-BY), which licenses unrestricted use, distribution, and reproduction in any medium, provided that the original work is appropriately cited. The use of general descriptive names, trade names, trademarks, and so forth in this publication, even if not specifically identified, does not imply that these names are not protected by the relevant laws and regulations.