CrowdPulse: A framework for real-time semantic analysis of social streams

The recent huge availability of data coming from mobile phones, social networks and urban sensors leads research scientists to new opportunities and challenges. For example, mining micro-blogs content to unveil latent information about people sentiment and opinions is drawing more and more attention, since it can improve the understanding of complex phenomena and paves the way to the development of new innovative and intelligent services. In this paper we present CrowdPulse, a domain-agnostic framework for text analytics of social streams. The framework extracts textual data from social networks and implements algorithms for semantic processing, sentiment analysis and classification of gathered data. The framework has been deployed in two real-world scenarios in order to identify the most at-risk areas of the Italian territory according to the content posted on social networks and to monitor the recovering state of the social capital of L׳Aquila׳s city after the dreadful earthquake of April 2009, respectively. In both scenarios, the framework showed its effectiveness and confirmed the insight that the combination of technologies specifically designed for Big Data processing with state-of-the-art methodologies for semantic analysis of textual content can provide very interesting findings and permits the analysis of such phenomena in a totally new way.