Sonification of autistic children’s movements: investigation through the lenses of Imitation and Social Motor Synchrony

This thesis focuses on supporting and understanding the interaction of autistic children, with a specific emphasis on two crucial skills related to non-verbal communication: Imitation and Social Motor Synchrony. These skills play a pivotal role in understanding autism. While imitation involves copying another person’s actions, social motor synchrony refers to the coordinated movement patterns that occur between individuals during social interaction, without necessarily imitating specific actions. The aim of this thesis is to utilise motor sonification, which represents body movements through sound, to support and enhance the interactive skills of autistic children with a human facilitator. To accomplish this, I designed an interactive system called OSMoSIS (Observation of Social Motor Synchrony with an Interactive System), which enables motor sonification using the Kinect system. OSMoSIS was utilised to investigate whether and how technology could provide opportunities for autistic children to engage in interactive play with a human facilitator during music therapy sessions. OSMoSIS was also designed to use movement data from the Kinect to provide a measure of social motor synchrony. Three exploratory studies were conducted to evaluate the effects of interventions using the musical interactive system OSMoSIS on children’s communication and interaction. Throughout these studies, Motion Energy Analysis (MEA) was employed to assess social motor synchrony. While MEA is an established method for evaluating social motor synchrony in stationary contexts, its application in full-body interactions presented limitations. Therefore, in the third study, we tested OSMoSIS as a proof of concept (PoC) measurement tool with potential value for future HCI designers working on full-body interaction. In terms of enhancing Social Motor Synchrony and Imitation associated with positive affect, the findings indicated a statistically significant difference when comparing the condition of motor sonification using OSMoSIS with the condition of no sonification in the first two studies. Additionally, in the third study, the effects of motor sonification were compared to the effects of having randomised musical sounds accompanying the activity of autistic children with the human facilitator. Although no statistically significant difference was found in social motor synchrony and frequency of Imitation when comparing motor sonification with random sounds, there was a strong trend towards significance in the frequency of Imitation. Notably, a statistically significant difference was observed in the autistic children’s responses to facilitator prompts, with significantly fewer cases of no response, in the motor sonification condition, suggesting the potential for motor sonification to enhance interaction. Furthermore, OSMoSIS analysis revealed no statistically significant difference in social motor synchrony between conditions, when considering position. However, a statistically significant difference in social motor synchrony was observed when examining specific kinetic energy. We discuss several factors that could potentially account for the disparity in outcomes when comparing MEA and OSMoSIS as evaluation tools. The findings demonstrate the potential validity of OSMoSIS, which may pave the way for new perspectives on social motor synchrony within the context of full-body interaction in HCI. In conclusion, this thesis contributes to the understanding of how motor sonification can enhance social motor synchrony and Imitation, positively impacting the interaction of autistic children with a human facilitator. It also highlights the potential of OSMoSIS as an evaluation tool for studying social motor synchrony from different and more appropriate perspectives within the field of full-body interaction in HCI.