Developing a Framework for the Perceptually Congruent Sonification of Auditory Line Charts

In an era that correctly prioritises accessibility and inclusivity through universal design, sonification has grown increasingly important. Sonification embraces alternative sensory modalities and fosters multi-sensory engagement, and has the potential to play a significant role in addressing universal-design challenges by encouraging broader accessibility for users with diverse sensory needs and for enabling devices without screen real-estate to present data effectively. For several decades, sonification has been explored within the broader context of data visualisation and auditory display, but as the importance of universal design is increasingly recognised, it has gained renewed prominence as an important tool in data presentation and exploration. However, as sonification is increasingly applied across a wider range of use cases, there is a growing need for well-defined design methodologies and robust psychoacoustic evaluation strategies. These are essential for ensuring consistent and predictable outcomes in sonification applications. Indeed, the 2011 landmark publication ‘The Sonification Handbook’ played a key role in recognising and promoting the fields of auditory perception and Auditory Scene Analysis (ASA) as crucial pillars for shaping future sonification methodologies. However, despite this, it is noteworthy that few publications since have effectively incorporated sychoacoustic principles or ASA as guidelines for designing perceptually congruent auditory displays. This gap highlights the need for further exploration and integration of these disciplines into the field of sonification, as they offer crucial guidance for creating sonification designs that align more comprehensively with human auditory perception. Such approaches would have the potential to enhance the overall user experience as well as strengthen the degree to which sonification becomes a reliable tool in scientific and engineering contexts. To this end, the primary objective of this thesis research is to examine howincorporating the perceptual dimensions of pitch, timbre, loudness and space within the principles of stream segregation impact design strategies for sonifying auditory line charts. The research employs a model-centred approach that utilises ASA-derived stream segregation (SS) to determine how these dimensions are processed perceptually by human listeners within the context of representing data trends. Referred to as the Perceptually Congruent Sonification (PerCS) framework, it not only illustrates the flow of information through perceptual processes, but also explores the multidirectional linkages and interactions between these sonic dimensions using what the author refers to as core information modules. Through these modules, the PerCS model describes the compatibilities, limitations, and constraints inherent in human auditory
perception as a function of the sonic dimensions chosen by designers when depicting data trends using PMSon. Within the scope of this PhD research, the two most commonly-used dimensions in auditory line charts (pitch and timbre) are examined in relation to their outcomes according to PerCS. Although limited by a small sample size, Study 1 examines frequency range choices and this pilot study suggests that a narrow 1 kHz range 400-1400Hz could potentially perform just as well as a more common wider range (65-1480Hz). The study also hints that basic loudness compensation had no effect on performance. It is concluded from this study that further research on frequency range could benefit from more specific test methods with a greater sample size. Study 2 examined timbral characteristics of sonified trends and found that adding up to three harmonics was considered preferable when sonifying a trend, and that when a fourth harmonic was added there was no improvement in listener preference. The results of these evaluations demonstrate the utility of PerCS in providing designers with perceptually-definitive strategies for efficient sonifying line charts where none existed beforehand. By utilising PerCS, it is proposed that sonification designers will be effectively guided through their sonification strategies to generate auditory line charts that more accurately reflect how listeners interpret sonified data streams.