Evaluating the impact of distractors on user auditory localisation and attention abilities in an immersive virtual reality experience

In Virtual Reality (VR) environments, where visual components hold significant importance, it is essential to also recognise the critical role of audio. Well-designed spatial audio provides users with valuable cues about the location and movement of objects or characters, enriching their sense of presence and engagement in the VR experience. In this context, this thesis presents the results of two studies that analysed participant’s spatial auditory localisation and auditory attention abilities in a VR environment that contained audiovisual distractors. Both are key in the interaction with the surrounding environment as they are fundamental steps in the processing of auditory information by the human auditory system. The inclusion of distractors is particularly important because it reflects real-world conditions, where individuals must continuously filter out irrelevant stimuli to focus on essential auditory information. Furthermore, the design of such VR environments allows us to develop applications that can be used to: (a) optimise multimodal VR experiences; (b) extract participant data continuously throughout the experiment; (c) create applications that can be used to evaluate the participant’s auditory processing ability. Both experimental evaluations analysed the impact of distractors for a listening task in a VR environment.
The first study evaluated spatial audio localisation abilities. For this purpose, 24 spheres were placed surrounding the participant. The task consisted of selecting the sphere that represented the location of the sound source. The experimental protocol was inspired by state-of-the-art procedures for evaluating listening abilities. Listeners experienced three levels of difficulty defined by the number of auditory distractors in the scene and their locations. The analysis revealed that the range of head movements impacts a listener’s performance, with the mean localisation angle error decreasing with larger amounts of head movement. Results from this study provided valuable insights into key psychophysiological metrics, such as pupillary response and questionnaire responses, required to accurately measure levels of cognitive load and performance in this task.
The second study investigated auditory attention in an auditory selective task. Participants were immersed in a virtual classroom and listened to two stories. During the first story, they were asked to identify the presence of distractors in the scene. For the second story, participants were required to identify a keyword that was repeated throughout the story. Participants were split into three groups that differed on the time when distractors were presented in relation to the target stimulus: (a) before the target; (b) at the same time as the target; (c) after the target. Findings from this study revealed significant gender differences for listeners immersed in an environment with competing sounds in recalling a story. Moreover, the time when distractors were presented significantly affected the response time, suggesting that the temporal relationship between distractors and target stimuli plays a crucial role in shaping auditory attention processes.
Finally, this thesis demonstrates the critical role of spatial audio and auditory attention in VR environments. The findings offer insights for designing more effective and engaging VR applications and optimising user experience. Furthermore, this research highlights the potential for VR to not only enhance entertainment and educational experiences but also to serve as a powerful tool for evaluating and training auditory skills.