Research Article
Leveraging Conversational AI for Accelerating User-Driven Software Testing
@INPROCEEDINGS{10.1007/978-3-031-81573-7_6, author={Aminata Saban\^{e} and Laura Plein and Tegawend\^{e} F. Bissyand\^{e}}, title={Leveraging Conversational AI for Accelerating User-Driven Software Testing}, proceedings={Towards new e-Infrastructure and e-Services for Developing Countries. 15th International Conference, AFRICOMM 2023, Bobo-Dioulasso, Burkina Faso, November 23--25, 2023, Proceedings, Part II}, proceedings_a={AFRICOMM PART 2}, year={2025}, month={2}, keywords={ChatGPT Debugging Translation Test cases Bug reports}, doi={10.1007/978-3-031-81573-7_6} }- Aminata Sabané
Laura Plein
Tegawendé F. Bissyandé
Year: 2025
Leveraging Conversational AI for Accelerating User-Driven Software Testing
AFRICOMM PART 2
Springer
DOI: 10.1007/978-3-031-81573-7_6
Abstract
This work addresses a research challenge in automating the translation of natural language inputs into programming language specifications. We consider the case of bug reports, which are informally written by users, and that must be specifying into executable test cases for reproducing the bug on the target software. Software bugs are indeed largely reported in natural language by users. Yet, we lack reliable tools to automatically address reported bugs (i.e., enabling their analysis, reproduction, and bug fixing). We therefore build on the recent promises brought by ChatGPT for various tasks, including in software engineering, and establish the following research question:What if Conversational Artificial Intelligence (AI) models could be used to explore the semantics of bug reports as well as to automate their reproduction?We evaluate the capabilities of ChatGPT, a state-of-the-art conversational AI, i.e., chatbot, using the popular Defects4J benchmark with its associated bug reports. The results reveal that ChatGPT can generate executable test cases that could trigger 50% of the bugs reported in natural language. These results are promising not only for the research community, but also for practitioners.
