Consent and Information Form
You are invited to participate in a research study entitled:
When Can Developers Trust LLM Explanations of Code Clones?
Student Researcher:
Sharmin Afroz, MSc Candidate, Department of Computer Science, University of Saskatchewan. Email: s.afroz@usask.ca
Supervisors:
Principal Investigator:
Dr. Banani Roy, Department of Computer Science, University of Saskatchewan. Email: banani.roy@usask.ca
Sub-investigator:
Dr. Chanchal Roy, Department of Computer Science, University of Saskatchewan. Email: chanchal.roy@usask.ca
Purpose and Objective of the Research
Large Language Models (LLMs) are increasingly attached to code-clone detection tools as post-hoc explainers, generating natural-language reasoning to help developers understand why two code fragments are considered clones. While these explanations are often fluent and confident, they can also be wrong in subtle ways, such as referring to identifiers that do not appear in the code or claiming behavioural similarities that the source does not actually support. Whether developers can reliably tell a well-grounded explanation from a fluent-but-ungrounded one, and what makes that judgement easier or harder, is currently untested.
The purpose of this research is to study how developers judge the trustworthiness of LLM-generated explanations of code clones, and to identify the conditions under which that judgement is most likely to fail. Specifically, the study examines whether the readability of the underlying code, that is, how visible the relationship between two fragments is at the surface level, affects a developer's ability to detect unreliable explanations. The objective is to provide empirical evidence on when AI explanations of clones can be safely trusted, and when tools should warn the user, contributing to the design of more trustworthy explainable software engineering tools.
The study targets participants with at least one year of programming experience who are comfortable reading Java code.
Procedure
Participants will complete a single session lasting approximately 45 to 65 minutes.
The session has:
A short briefing and background questionnaire on programming experience and prior use of AI coding tools.
The main task, in which participants read approximately 3 code-clone pairs along with several AI-generated explanations for each. For every pair, participants will judge whether the explanation is reliable, rate their confidence and trust, and indicate which lines of code support the explanation's main claim. At the end of the main task, participants will complete a short workload questionnaire (the NASA Task Load Index).
Participation is voluntary, and participants may decline to answer any question or pause and withdraw from the study at any time without consequence.
Participation Eligibility
To participate in this study, you must:
Be at least 18 years of age
Have at least one year of programming experience (academic, professional, or both)
Be comfortable reading Java code
Funded by
SOAR (Software Analytics Research) and NSERC
Potential Risks
There are no known significant risks anticipated from participating in this study. Participants may experience mild mental fatigue or frustration over the course of the session, since the task involves sustained code reading, explanation reading an