Recognizing the User Social Attitude in Multimodal Interaction in Smart Environments

Ambient Intelligence aims at promoting an effective, natural and personalized interaction with the environment services. In order to provide the most appropriate answer to the user requests, an Ambient Intelligence system should model the user by considering not only the cognitive ingredients of his mental state, but also extra-rational factors such as affect, engagement, attitude, and so on. This paper describes a study aimed at building a multimodal framework for recognizing the social response of users during interaction with embodied agents in the context of ambient intelligence. In particular, we describe how we extended a model for recognizing the social attitude in text-based dialogs by adding two additional knowledge sources: speech and gestures. Results of the study show that these additional knowledge sources may help in improving the recognition of the users' attitude during interaction.