LLMs to Augment Evaluators in Scripting Usability Tasks: An Exploratory Study

Integrating Artificial Intelligence (AI) into many areas is leading to a new scenario in which humans and machines collaborate to reach a shared objective. In this article, we perform an empirical study to explore whether LLMs can support evaluators in planning usability tests, specifically in scripting tasks. To reach this goal, we analyzed how different LLMs, prompt engineering techniques, and temperatures influence this activity. To assess the quality of LLM-generated tasks, an exploratory user study was conducted with 6 participants from academia and industry using two prototypes. Participants were asked to evaluate tasks in terms of four dimensions (i.e., Functionality-Oriented, Utility, Precision, and Completeness) assigning a score from 1 to 3. The results suggest that LLMs may support evaluators, though their oversight and control are necessary to ensure the quality and validity of the tasks. Thus, while concerns of cognitive offloading and deskilling still hold, human control remains necessary.