Supporting Lay People to Define Smart Interactive Experiences. Presentazione al congresso

The increasing proliferation of the Internet of Things (IoT) is promoting the development of smart environments where the behavior of the IoT devices is synchronized with the behavior of people living or acting in the environment [6]. Some devices can also come in the form of tangible interactive objects that people manipulate and bring with them to tackle different tasks [7]. This technological landscape favors Smart Interactive Experiences (SIEs – pronounced see-ehs), i.e., immersive experiences created by orchestrating different devices so that the resulting smart environments can fluidly react to the usage situations induced by the situational needs of the final users. While the number of domains in which SIEs are adopted is growing, critical issues have to be solved to increase their practical impact. The IoT phenomenon has been largely investigated on the technical side, but it is still unclear how lay-people (e.g., non-technical domain experts such as museum curators, caregivers, therapists, educators) can be enabled “to harness the potential power of these large collections of devices to accomplish their tasks” [8]. Some approaches try to facilitate the configuration of single devices [9]; however, it is still hard for lay-people to synchronize the behavior of multiple physical and software resources, installed in the environment or embedded in the tangible objects manipulated by final users. Our research of the last years addresses this gap by proposing a design approach to support non-technical people to define SIEs. The contribution presented at iCities 2019 provided a first introduction to our approach [1]. The main idea is to enable domain experts (in the following also called designers, because they create SIEs) to enrich the resources involved in SIEs with semantic properties that derive from their domain knowledge [2]. The advantage of such user-defined semantics is that the introduced terms can facilitate the definition of Event-Condition-Action (ECA) rules specifying the behavior of IoT systems. The literature recognizes the value of semantic enrichments, e.g., through ontologies, for facilitating the event-driven programming of IoT devices. Some approaches propose to annotate smart devices with ontology concepts to improve the semantics of ECA rules [4]. However, the proposed paradigms have a limited expressiveness with respect to the background and expertise of non-technical people. In addition, there is a scarce or completely missing consideration and systematization of the user-defined semantic; this lack exposes the system to the risk of not covering the actual needs of SIE designers. Our approach tries to alleviate these problems; it goes beyond by promoting user-based semantics as a conceptual tool for the domain experts to conceive SIEs creatively. More specifically, we have been developing a platform that presents to non-technical users a tailoring environment to support the design of a SIE without any coding. IN order to describe the platform, we briefly report a scenario in the Cultural Heritage domain, where Noah and Isabella are guides of an archaeological park hosting an exhibition of the tools adopted by ancient populations. To make the exhibition more engaging, they create a game for pupils visiting the park. Pupils get a collection of tools (tangible interactive objects that are replicas of significant ancient tools), which are smartified through RFID tags. During the game-play, they are asked to identify tools having a specific characteristic, for example, those used in a specific activity of the populations (e.g., cooking, fighting, building). They have to position the tools on one of the 3D models in the park to reproduce some aspects of the ancient activities (e.g., eating/praying moments, typical places for such activities). The game proposes quests such as: “Put on the Messapian kiln all the fighting tools of the same age”. When pupils move the right tool on the target 3D model, this last plays an audio file describing the main characteristics of the tool, and the pupils get points as a reward. Otherwise, the 3D model plays an audio file indicating that the selected tool is wrong. The game continues with pupils answering all the proposed quests. The winner is the group that gets the highest score. To define this game, for each interactive object Noah and Isabella have to define properties (attributes of the object) that express the object meaning and role according to the game dynamics. For example, each tool has descriptive attributes, like “Age” (with values: iron, bronze, Messapian, Roman), “Activity” (with values: cooking, fighting and building), “Material” (with values: gold, pottery, and bronze). Noah and Isabella “freely” define these attributes and their values depending on the goal and the interactivity they want to pursue through the game, without any (syntactic or semantic) constraint on the type of properties to be specified - therefore, we name them custom attributes (CA). The system for CA definition also suggests improvements and enrichments for the CAs initially conceived. After defining CAs, Noah and Isabella use EFESTO-4SIE to visually define the ECA rules controlling the behavior of the smart devices. Assigning attributes to objects has two main advantages when creating ECA rules. First, the language adopted to define the rules is closer to the domain-expert language. For example, the variables used in the rules are exactly the attributes previously defined by the SIE designers (Noah and Isabella). Second, the attributes introduce abstractions that favor generalization. Without CAs, several rules would be defined for every single smart device, like: “IF tool_012.isOn(3DModel_5) THEN 3DModel_5.playsAudioFile(explanation04.mp3)”. This rule would be replicated for each tool and each 3D model, and the designer has to remember the ID of any tool and 3D model. Supposing that the same SIE involves n tools and m 3D models, the first rule covers all the nXm possible rules [3]. Several systems for IoT programming instead force users to create all the combinations [5]. IN ORDER TO define CAs, some prototypes that implement different interaction paradigms have been developed. The first one is a visual paradigm that permits the definition of CAs through the use of graphical widgets called Visual Annotation Containers (VACs) [2]. Three further novel paradigms have been defined with the aim to stimulate creativity during the SIE design process: two are based on Tangible User Interface (TUI) and one on Augmented Reality. These paradigms and the implemented prototypes are presented in [3], also reporting the user studies performed to evaluate them. The prototypes for CAs creation, as well as the visual tool for ECA rule definition, have been framed inside an architecture that fosters an easy integration of further tools for CA and ECA rule creation. In addition, the middleware of this architecture has been enriched through the use of ontologies, to support the management and systematization of user-defined semantics. This is achieved through an algorithm that transparently assists designers in using and extending a domain-specific ontology through their semantic properties. The design approach to support non-technical people to define SIEs, presented in this paper, is one of the cornerstones of the Italian PRIN project “EMPATHY: EMpowering People in deAling with internet of THings ecosYstems”. The goal of this project is to propose new concepts, languages, methods, and tools to support people in creating and tailoring IoT context-dependent interactive applications for their needs. (http://www.empathy-project.eu/).