Music, often played in the background as an accompaniment to everyday activities, is known to be powerful in modulating emotion, changing behaviors, and affecting task performance and engagement. Starting from explorations of music usage among real users, our investigation covers three themes: multimodal analysis of user-music interactions in the lab; remote monitoring of user-music interactions in natural settings; and music recommendations for enhancing learning and wellbeing. Through the series of studies, we aim to broaden the impact of MIR research to related fields such as education, psychology, and cognitive science.

Current and Ongoing Research

Theme 0: Music Usage among Real Users

      • Music Information Behaviors and System Preferences of University Students in Hong Kong
        This paper presents a user study on music usage needs and behaviors of university students in Hong Kong. A survey was completed by 101 participants and supplemental interviews were conducted in order to investigate users’ music information related activities. We found that university students in Hong Kong listened to music frequently and mainly for the purposes of entertainment, singing and playing instruments, and stress reduction. We also examined users’ preferences on music services available in Hong Kong such as YouTube and KKBox, as well as the characteristics liked and disliked by the users.
        [Publications]
        Hu, X., Lee, J. H. & Wong, L. K. Y. (2014, October) Music Information Behaviors and System Preferences of University Students in Hong Kong, In Proceedings of the 15th International Conference on Music Information Retrieval (ISMIR ’14) (pp.579-584). Taipei, Taiwan. [PDF]
      • University students’ use of music for learning and well-being: A qualitative study and design implications
        This study explores music information behaviours for learning and well-being among university students in China. An investigation framework is developed based on theories in multiple related disciplines such as musicology, psychology, and sociology. In-depth interviews were conducted with forty university students in twenty universities, with an interview protocol designed based on the framework. The results reveal how students use music for multiple aspects of life corresponding to learning and major components of well-being, including physical well-being, social relationships, positive emotion, self-esteem, and meaning of life. 
        2_ISMIR_2018_ADiaryStudy_0916_XH

        [Publications]
        Hu, X., Chen, J., & Wang, Y. (2021). University students’ use of music for learning and well-being: A qualitative study and design implications. Information Processing and Management. 58(1),102409. (IF: 7.47) [PDF]
      • Pandemics, music, and collective sentiment: Evidence from the outbreak of covid-19
        As people listen to music daily everywhere in the world, the sentiment of music being listened to can reflect the mood of the listeners and serve as a measure of collective sentiment. However, the exact relationship between real-world events and the sentiment of music being listened to is not clear. Driven by this research gap, we use the unexpected outbreak of COVID-19 as a natural experiment to explore how users’ sentiment of music being listened to evolves before and during the outbreak of the pandemic. We employ causal inference approaches on an extended version of the LFM-1b dataset of listening events shared on Last.fm, to examine the impact of the pandemic on the sentiment of music listened to by users in different countries.
        [Publications]
        Liu, M., Zangerle, E. Hu, X., Melchiorre, A. & Schedl, M. (2020). Pandemics, music, and collective sentiment: Evidence from the outbreak of covid-19. In Proceedings of the 21st International Society for Music Information Retrieval Conference (ISMIR). [PDF]
      • The relations of culture, socio-economics, and friendship on music preferences: A large-scale, cross-country study
        Music listening is an inherently cultural behavior, which may be shaped by music users’ cultural backgrounds and contextual characteristics. Previous literature provides limited empirical evidence of the relationships between possible cross-country differences on a wide range of socio-economic aspects and those in music preferences. To bridge this research gap, and drawing on a large-scale dataset, LFM-1b, this study examines the possible relationship between cross-country differences in artist, album, and genre listening frequencies as well as the cross-country distance in geographical, socio-economic, linguistic, cultural, and friendship connections using the Quadratic Assignment Procedure. 
        [Publications]
        Liu, M. Hu, X.*, & Schedl, M. (2018). The relations of culture, socio-economics, and friendship on music preferences: A large-scale, cross-country study. PLOS One, 13(12), e0208186. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0208186

Theme 1: Multimodal Analysis of User-music Interactions in the Lab

      • Can Background Music Facilitate Learning? Preliminary Results on Reading Comprehension
        It is a common phenomenon for students to listen to background music while studying. However, there are mixed and inconclusive findings in the literature, leaving it unclear whether and in which circumstances background music can facilitate or hinder learning. This is a study investigating the effects of five different types of background audio (four types of instrumental music and one environmental sound) on reading comprehension. A lab experiment was conducted with 33 graduate students, where a series of cognitive, metacognitive, affective variables and physiological signals were collected and analysed.

        [Publications]
        Hu, X., Li, F., & Kong, R. (2019, March). Can Background Music Facilitate Learning?: Preliminary Results on Reading Comprehension. In Proceedings of the 9th International Conference on Learning Analytics & Knowledge (pp. 101-105). ACM. [PDF]
      • Analyzing user interactions with music information retrieval system: An eye tracking approach
        There has been little research considering eye movement as a measure when assessing user interactions with music information retrieval (MIR) systems, whereas many studies have adopted conventional user-centered measures such as user effectiveness and user perception. This study investigates users’ eye movement patterns and measures with two music retrieval tasks and two interface presentation modes. A lab-based user experiment was conducted with 16 participants whose eye movement and mouse click behaviors were recorded through professional eye trackers. 
        6-ISMIR2019_MIR_eye

        [Publications]
        Hu, X., Y. Que, N. Kando, W. Lian (2019). Analyzing user interactions with music information retrieval system: An eye tracking approach. In Proceedings of the 20th International Society for Music Information Retrieval Conference (ISMIR), Delft, The Netherlands, Nov. 4-8, 2019. [PDF]
      • Exploring the Effect of Personalized Background Music on Reading Comprehension
        Regarding the influence of background music on learning, there are inconclusive findings in the literature. Inspired by the gap, we conducted a lab experiment on reading with 100 university students. The participants were tasked to read nine academic passages. In the meantime, those who were randomly allocated to the experiment group listened to their self-provided music in the background during the reading task, while those in the control group did not have background music during reading. During the experiment, participants’ reading logs, self-reported meta-cognition and emotion status were recorded. This paper reports the results of comparing measures on reading performance, meta-cognition and emotion changes between the two groups. In addition, the relationships between participants’ personal traits and their preferred background music types were investigated. 
        [Publications]
        Que, Y., Zheng, Y., Hsiao, J. H., & Hu, X. (2020, August). Exploring the Effect of Personalized Background Music on Reading Comprehension. In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020 (JCDL ’20) (pp. 57-66). [PDF] 
      • Reading with and without background music: An exploration with EEG, eye movement and heart rate
        The effects of background music on learning have been studied in related fields, including psychology and education, but findings are mostly inconclusive. In addition to measurements at the behavioural level, multimodal physiological signals can provide new evidence for exploring the question. This paper presents a pilot study of a reading task for a group of university students whose electroencephalogram (EEG) signals, eye movements, and heart rates were recorded with and without background music. Preliminary results demonstrated the feasibility of integrating multimodal learning analytics to probe the underlying mechanism about the effects of background music on learning. 
        MoBI_Ying

        [Publications]
        Que, Y., Andrea-Penna, G. M.D., Hu, X., Dong, Y., Chiba, A. A., Iversen, J. R. (2021). Reading with and without background music: An exploration with EEG, eye movement and heart rate, In Companion Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK), CrossMMLA Workshop, Online. [PDF]

 Theme 2: Remote Monitoring of User-music Interactions in the Naturalistic Settings 

      • Learning with Background Music: A Field Experiment
        This study aims to probe the underlying mechanism regarding how background music benefits or hinders learning through an experiment in naturalistic setting. 30 participants were recruited to join a field experiment which was conducted in their own study places for one week. During the experiment, participants were asked to conduct learning sessions with music in the background and collect music tracks they deemed suitable for learning using a novel mobile-based music discovery application. A set of participant-related, context-related, and music-related data were collected via a pre-experiment questionnaire, surveys popped up in the music app, and the logging system of the music app.

        [Publications]
        Li, F., Hu, X., & Que, Y. (2020). Learning with Background Music: A Field Experiment. In Proceedings of the 10th International Conference on Learning Analytics & Knowledge (LAK ’20) (pp.224-229). Frankfurt, Germany: ACM. [PDF]
      • Studying with learners’ own music: Preliminary findings on concentration and task load
        Through profiling learners’ music usage in everyday learning settings and depicting their learning experience when studying with a music app powered by a large-scale and real-world music library, this study revealed preliminary observations on how background music impacts learning under varying task load, and manifested intriguing patterns of learners’ music usage and music preferences in various task load conditions. Specifically, we piloted a three–day field experiment in students’ everyday learning environment. During the experiment, participants performed learning tasks with music in the background and completed a set of online surveys before and after each learning session.
        [Publications]
        Li, F., Wang, Z., Ng, J. T. D. & Hu, X. (2021). Studying with learners’ own music: Preliminary findings on concentration and task load, In Proceedings of the 11th International Conference on Learning Analytics and Knowledge (LAK), Online. [PDF]

Theme 3: Music Recommendations for Enhancing Learning and Wellbeing

      • On the relationships between music-induced emotion and physiological signals
        Physiological signals are regarded as related to emotion and thus could potentially be exploited in emotion-aware music discovery. This study explored the possibility of using physiological signals to detect users’ emotion responses to music, with consideration of individual characteristics (personality, music preferences, etc.). A user experiment was conducted with 23 participants who searched for music in a novel MIR system. Users’ listening behaviors and self-reported emotion responses were collected. During music listening, a series of peripheral physiological signals (e.g., heart rate, skin conductance) were recorded. A set of features in the time and frequency-domains were extracted from the physiological signals and analyzed using statistical and machine learning methods. The findings can contribute to further improvement of emotion-aware music recommendation systems exploiting physiological signals as an objective and personalized input.
        ISMIR_2018_poster

        [Publications]
        Hu, X., Li, F., & Ng, J. (2018). On the relationships between music-induced emotion and physiological signals. In Proceedings of the 19th International Society for Music Information Retrieval Conference (ISMIR), Sept. 23 – 27, 2018. [PDF]

      • A multimodal music recommendation system with listeners’ personality and physiological signals
        This preliminary study explored multiple information sources for music recommendation system, including users’ personality traits measured by the Ten-Item Personality Inventory and physiological signals recorded by a wearable wristband. A dataset of 23 participants and 628 song listening records were obtained from a user experiment. Based on the dataset, a machine learning experiment with four regression algorithms was conducted to compare recommendation performances across different combinations of feature sets. Results show that personality features contributed significantly to the improvement of recommender accuracy, while physiological features contributed less. Analysis of top features in the best performing model revealed the importance of some physiological features. Future studies are called for to further investigate multimodal music recommendation system through exploiting user properties and context data. 
        [Publications]
        Liu, R. & Hu, X. (2020). A multimodal music recommendation system with listeners’ personality and physiological signals, In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries. [PDF] Presentation Link: https://dl.acm.org/doi/abs/10.1145/3383583.3398623
      • Detecting music-induced emotion based on acoustic analysis and physiological sensing: A multimodal approach
        The subjectivity of listeners’ emotional responses to music is at the crux of optimizing emotion-aware music recommendation system. To address this challenge, we constructed a new multimodal dataset (“HKU956”) with aligned peripheral physiological signals (i.e., heart rate, skin conductance, blood volume pulse, skin temperature) and self-reported emotion collected from 30 participants, as well as original audio of 956 music pieces listened to by the participants. A comprehensive set of features was extracted from physiological signals using methods in physiological computing. This study then compared performances of three feature sets (i.e., acoustic, physiological, and combined) on the task of classifying music-induced emotion. Moreover, the classifiers were also trained on subgroups of users with different Big-Five personality traits for further customized modeling.  
        [Publications]
        Hu, X.*, Li, F., & Liu, R. (2022). Detecting music-induced emotion based on acoustic analysis and physiological sensing: A multimodal approach. Applied Sciences, 12(18), 9354. doi: 10.3390/app12189354 (IF: 2.84) [PDF]

 

Grant Awards

    • Leveraging Background Music for Learning: An Interdisciplinary Approach, General Research Fund, Research Grants Council, Hong Kong S.A.R.  2019 – 2022
    • Music Recommender Systems Based on Physiological Signals, Young Scholar Science Fund, National Natural Science Foundation of China, 2018 – 2020
    • Regulating Learners’ Emotion Using Music, Seed Fund for Applied Research, University of Hong Kong, 2017 -2019