Asynchronous Remote Usability Tests Using Web-Based Tools Versus Laboratory Usability Tests: An Experimental Study

Remote usability testing is performed by evaluators who are in different physical locations from the participants (synchronous remote testing) and possibly operating at different times (asynchronous remote testing). The tools developed in recent years to support remote tests exploit web technology based on HTML5 and JavaScript ES6 and thus enable previously unexplored scenarios. However, studies providing evidence on the benefits or drawbacks of utilizing recent web-based tools have not yet been reported in the literature. This article sheds some light on the impact of such tools on asynchronous remote usability testing of websites by reporting an experimental study with 100 participants and 15 evaluators to compare real-time laboratory tests with asynchronous remote tests. The study investigates 1) how the metrics results of asynchronous remote usability tests performed through a web-based tool differ from those of usability tests conducted in real-time laboratory settings, and 2) how the experience of participants differs in the two types of tests. The lessons learned in the study are instrumental in informing the design of future tools. Some results of particular interest indicate that the web technology used by the tool for asynchronous remote testing affects task execution times and participants’ satisfaction. Another indication is that slow internet connections must be managed in asynchronous remote testing; slow connections introduce delays when transferring large amounts of collected data, which, together with the lack of human support, make participants of asynchronous remote tests more prone to feel negative emotions.