Predicting Quality of Multimedia Experience using Electrocardiogram and Respiration Signals

Quality of experience (QoE) has become a crucial field of study in recent years as multimedia technology continues to grow in popularity. Accurate and reliable methods of evaluating QoE are necessary, but they can be challenging due to their multidimensional nature. QoE, a comprehensive measure of user satisfaction with multimedia services, is influenced by human, system, and contextual dimensions. However, the human factor is often overlooked in QoE assessment studies, which plays a pivotal role in accurately representing end-users perceptions of quality. This article proposes the use of physiological signals to assess the QoE of immersive multimedia applications, which provides insights into users’ emotional and cognitive states. Specifically, we present an implicit evaluation of user QoE in terms of perceived quality, sense of presence, and user content preference using a publicly accessible database. We compare the performances of several machine learning (ML) and deep learning (DL) models in predicting QoE subjective scores from electrocardiograms and respiration signals. The ML models considered are support vector machines, k-nearest neighbor, and random forest algorithms; the DL models considered are bidirectional long-short-term memory (BLSTM), convolutional neural networks (CNN), and a hybrid model of CNN and BLSTM (CNN-BLSTM). We evaluate the performances of these models based on individual and fusion modalities. Additionally, we present a comparison of 2-class and 3-class classifications of QoE scores based on the fusion data. The results demonstrate that the BLSTM network, based on fusion data, has the best performance for classifying QoE subjective scores into two classes, with an F1-score of 87.55% and 81.22% for perceived audio quality and overall quality, respectively.