Van den Haak, M. V., Jong, M. D., & Schellens, P. J. (2003). Retrospective vs. concurrent think-aloud protocols: Testing the usability of an online library catalogue. Behaviour & Information Technology, 22(5), 339-351.
The authors, from the University of Twente, Institute for Behavioural Research in The Netherlands, acknowledge that visual design effectiveness is largely still a misunderstood field and that there is not enough data to fully support one technique over another for all visual design types. They addressed three methods for evaluating visual design effectiveness: video recording of task performance, retrospective and concurrent think-aloud protocols, and surveys on participant experience. The basic principle of think-aloud protocols, as described by the authors, “is that potential users are asked to complete a set of tasks with the artifact tested, and to constantly verbalize their thoughts while working on the tasks.” The main difference between concurrent and retrospective evaluations is that users are asked to verbalize their thoughts while performing the tasks during the test, whereas retrospective think-aloud protocols ask the user to verbalize their thoughts after the tasks have been completed using video recordings of their performance.
One method does not always trump the others, rather each offers its own unique advantages and disadvantages and should be used according to the visual type and information desired. The results of their online library catalogue usability testing showed that significantly more problems were identified by means of observation only through concurrent think-aloud methods. Retrospective think-aloud methods, however, revealed more problems that were not observable. Therefore, the authors note that the concurrent think-aloud method “is a more faithful representative of a strictly task-oriented usability test, while the retrospective think-aloud method is likely to yield a broader gamut of user reactions.”