An open platform for full body interactive sonification exergames

This paper addresses the use of a remote interactive platform to support home-based rehabilitation for children with motor and cognitive impairment. The interaction between user and platform is achieved on customizable full-body interactive serious games (exergames). These exergames perform real-time analysis of multimodal signals to quantify movement qualities and postural attitudes. Interactive sonification of movement is then applied for providing a real-time feedback based on “aesthetic resonance” and engagement of the children. The games also provide log file recordings therapists can use to assess the performance of the children and the effectiveness of the games. The platform allows the customization of the games to address the children's needs. The platform is based on the EyesWeb XMI software, and the games are designed for home usage, based on Kinect for Xbox One and simple sensors including 3-axis accelerometers available in low-cost Android smartphones.


INTRODUCTION
In rehabilitation, recent treatment protocols aim at improving the quality of life of the patient, allowing the reintegration at home through ecological systems monitoring and remote assistance.The STARC project (Sistema di Teleassistenza delle Attività Riabilitative Domiciliari, Progetto 4 PAR FAS 2007/2013, funded by Regione Liguria) aims at designing interactive technology to support the management and control of domiciliary rehabilitation activities, with particular attention to those carried out independently by children with deficits.Recipients of the technology the project develops are children hospitalized for an unexpected acute pathological event or an exacerbation of a chronic situation needing, in many cases, a long period of bed rest (e.g., cancer patients, patients with cystic fibrosis, patients suffering from multiple trauma).Possible users include children whose clinical condition is stabilized, but that still need monitoring and/or show physical deconditioning.These children, despite having completed the most challenging clinical diagnosis and therapeutic interventions, still require rehabilitation.The objective of remote monitoring is to quickly improve clinical conditions and then to maintain good cardiorespiratory conditions.

The rehabilitation program has been designed to:
 Support cognitive rehabilitation (training for attention, memory, executive functions);  Support motor rehabilitation (training for recovery of strength, balance, coordination, endurance);  Support motivation.
Basic assumptions for the developed interactive serious games are the following:  Interaction is based on the automated measure of qualities of full-body movement and postural attitudes;  Interactive sonification of such qualities of movement and postural attitudes is used to provide engaging feedback stimulating a positive rehabilitative response and closed-loop interaction;  Gameplay based on score and mechanism increasingly challenging.
The positive and measurable impact of music on performance of physical exercises and on therapy and rehabilitation is well known in music therapy communities, and is receiving a growing interest by scientific communities [1].Interactive sonification and interactive experience of music can be very effective in stimulating and improving physical and cognitive tasks [2,3].The paper is organized as follows: Section II describes the therapy and rehabilitation games the work focused on; Section III presents a brief description of the platform architecture, including the EyesWeb XMI software and tools; Section IV discusses the design and development of serious game applications in EyesWeb.

II. THERAPY AND REHABILITATION GAMES
The platform is modular and allows adding new interactive games.Each game aims to recover and develop cognitive and motor skills.focuses on developing and recovering general cardiorespiratory endurance and global coordination.

A. Cognitive and Upper Limb Trainer (CULT)
This category of rehabilitation tasks requires the movement of the upper limbs to improve their function and /or to implement a cognitive task.To complete the game successfully, the user is required to reach a moving target before it disappears from the screen.A sound is associated with the position of the target and will deteriorate if the target is not reached.The application allows modulating task difficulty by acting on characteristics of targets (shape, color, dimension, order and speed of presentation), characteristics of background screen areas and audio feedback.In order to make the game more challenging the therapist may decide also to add distractors and double tasks.Figure 1 shows an example of game.It is made up of a fixed background, which represents a natural environment (for example sky and clouds in Figure 1) and of figures of animals (moving targets).The user is expected to choose the animal, which better suits the proposed background, i.e., she has to answer the following simple question: which of the displayed animals lives in that particular environment?To keep music alive and advance in the game, the child needs to reach and grab the animal properly with the movement of his/her hands or, in case this kind of movements is compromised because of a severe disability, with his/her elbows or his/her head.

B. Head Trunk Control Trainer (HTCT)
The platform integrates games to support the development or the recovery of head and trunk postural attitude and balance, and to measure the leaning of the body.
The platform tracks the position of the head, the center of mass of the shoulders and the pelvis, and computes the relative displacement, as shown in Figure 2. The aim of the game is to stimulate the user to keep an appropriate postural attitude by associating such an attitude with a full and high-quality reproduction of her preferred song.the quality of audio deteriorates if postural attitude and balance do not meet therapists' requirements.

C. Coordination Endurance Rehab Trainer (CERT)
This set of games is performed by wearing sensors that capture physical information related to movement.The goal is to complete a series of exergames keeping rhythm and coordination (intra and inter-personal synchronization or alternate movement between limbs, or, in a group, between members) with the aim to improve cardio-respiratory endurance and dynamic sectorial and global coordination.Exergames can be performed in sitting or standing position (on site ambulation combined or not with alternate movements of the limbs).Difficulty can be modulated by changing the complexity of sequences and the requested rhythm of the exergames.

III. PLATFORM ARCHITECTURE
The platform runs on two worksttions (see Figure 3): the first is located at the rehabilitation institute and is operated by the institution's personnel (therapists), the second one is located at the patient's home.
The hospital workstation is used to:  Design sequences of exergames suitable for each individual case, by using the tool described in section IV.A and developed by using MetaEyesWeb (Section III.B). Store personal data, medical records, and therapies.

 Perform analysis and compute statistics on therapy results
The home workstation is intended to:  Perform the therapy prescribed by therapists  Provide a result to the family of the patient  Provide an exhaustive log file for the hospital staff These two workstations communicate over the web and, by exchanging information about the progress of the therapy and about the performance of the patient within and among rehabilitation sessions, allow patients to be monitored daily by the Hospital staff.

A. The EyesWeb Software Platform
The EyesWeb XMI platform 1 has been adopted to manage the development of the building-execution-evaluation process of the rehabilitation exergames.
EyesWeb XMI was conceived for supporting research on multimodal expressive interfaces and interactive systems.Currently it is widely employed for designing and developing real-time dance, music, and multimedia apps.It includes libraries for audio and video processing and for movement analysis.It supports commercial applications.EyesWeb consists of a number of integrated hardware and software modules (blocks) which can be assembled by developers in a visual language to build applications, called patches.Specific patches were developed to implement the proposed riehabilitation platform.These exploit algorithms and modules available in the EyesWeb Gesture Processing Library.The library includes modules for:  Motion Analysis: motion trackers, motion expressive features (e.g., movement energy, contraction, fluidity, and so on). Space Analysis: measures of how a user moves in a (physical or abstract) space.Use of metaphors from physics: hysteresis, abstract potentials, superposition principle, and so on. Trajectory Analysis: processing of 2D (physical or abstract) trajectories (e.g., kinematics, directness). Machine learning techniques: e.g., SVMs, clustering, neural networks, and so on. Time series analysis: analysis of temporal profiles of expressive features.
Moreover, the EyesWeb Social Processing Library (Figure 4) was used for analysis of synchronization.It includes modules   In the proposed platform, MetaEyesWeb is used to develop a collection of FSMs to control real-time motion tracking and to manage each single exergame.MetaEyesWeb also manages a shared folder in the home workstation.It contains an XML configuration file in order to store game parameters.The game is perfomed using these parameters and results are stored in a log file that can be used by therapists to evaluate the progress of the therapy in terms of behavioural features.

C. EyesWeb Mobile
EyesWeb Mobile is an external tool to design and execute user interfaces enabling the user to control EyesWeb patches.This tool hides the implementation details of the patches and allows an easier use of the system.It consists of a designer and a runtime application.The typical way to proceed is, once finished the development of EyesWeb patches, to design a user interface and to bind it to an instance of the EyesWeb kernel.Then the runtime application connects and communicates to the kernel instance by using a network protocol (RCF).
The user interface can be on the same machine or on another device, connected through the network.

B. EyesWeb Tracking System
The tracking system consists of an EyesWeb patch, which integrates data streams from two different kind of sensors: Kinect for Xbox One [4] and 3-axis accelerometers.
In the first prototype, the system extracts and analyses the following features: e. Synchronization between limbs of a single user f.Synchronization between movements of multiple users in a group a) Position, velocity, and acceleration of the limbs To ensure the functioning of the system with children with severe disabilities, a twofold approach was implemented: (i) the EyesWeb patch detects and tracks hands, elbows and head.In such a way, users may benefit from the system according to their ability, i.e., if a user suffers of a disability that affects the movement of the hands, she can perform the games by moving her elbows or her head.(ii) the EyesWeb Tracker module can receive coordinates of body joints by the Microsoft Kinect SDK, or by an algorithm developed to extract the same values from the depth map.So, if Kinect cannot track the user reliably, the output of the algorithm operating on the depth map is used.
a) Quantity of Motion of the whole body An important measure for the exergames is the extent to which a person is moving while performing the game, as defined in [5].Quantity of Motion (QoM) (see also [6]) is a measure of the amount of movement detected by a video-camera.QoM is based on patterns of motions represented as Motion History Images (MHI) computed on a buffer of consecutive silhouettes of the user.At each new frame the brightness of the older silhouettes decreases, whereas the new silhouette is added with maximum brightness.QoM is computed as the weighted sum of the areas of such layered silhouettes.

b) Directness of hands movements
Directness (DI), as defined in [5], is one of the features that are deemed important in the process of measuring movement quality [7].A direct movement is characterized by almost straight trajectories.Directness Index is computed on the trajectory of a body joint in the space as the ratio between the Euclidean distance calculated between the starting and the ending point of the trajectory, and the trajectory's actual length.The Directness Index tends to assume values close to 1 if a movement is direct and values close to 0 otherwise.In the case of three-dimensional trajectories, DI is computed as follows: where xS, yS, zS are the 3D coordinates of the trajectory's starting point, yE, zE are the 3D coordinates of the ending point, and N is the number of 3D points the trajectory consists of.c) Smoothness and Fluidity of hands movements In general, smoothness (SMI) is synonymous for "having small values of high-order derivatives", as defined in [5].Wallbott, in his analysis of qualitative aspects of psychiatric patients' hand movements, noticed that movements judged as smooth are "characterized by large circumference, long wavelength, high mean velocity, but not abrupt changes in velocity or acceleration (standard deviations of velocity and acceleration).Thus, smooth movements seem to be large in terms of space and exhibit even velocity" [8].We therefore adapted Wallbott's statements on the qualitative dimensions of underconstrained arm movements and we computed hands trajectories curvature as a measure of smoothness.Curvature (k) measures the rate at which a tangent vector turns as a trajectory bends.It is defined as the reciprocal of the radius (R) of the curve the trajectory describes: R k 1  (2) A joint's trajectory following the contour of a small circle will bend sharply, and hence will have a higher curvature corresponding to low fluidity; by contrast, a joint trajectory following a straight line will have zero curvature corresponding to high smoothness.Curvature is computed for the trajectory of a single point at time frame i as follows: where ̇ is the velocity of the point at time frame i, and ̈ is its acceleration.The principle of the curvature can be applied to movement velocity and its variation in time to calculate movement fluidity (FI): high curvature of the speed trajectory in time means low fluidity, whereas low curvature means high fluidity.To calculate fluidity, we compute the curvature of the tangential velocity of the desired joint as described in the equation above.

d) Synchronization between limbs or in a group
Starting from accelerations of limbs (of a single user or of a group) the platform computes a phase synchronization index, the Correlation Probability of Recurrence (CPR) exploiting Recurrence Quantification Analysis [9].

C. Exergames Development
Games were implemented as a collection of EyesWeb patches so that patches that are common to multiple games can be easily reused.The software platform receives parameter values from MetaEyesWeb, and customizes the games according to the therapists' prescriptions.This makes the platform highly configurable and expandable.Game patches provide specific graphical feedback if the therapeutic protocol requires it.In addition to this, an audio source is played and modulated according to the quality of the patient's movements.The audio files are chosen from a collection of songs and stories that are part of everyday life of the child.The applied audio modulations are two: the first acts on the volume of the song, the second filters the sound by moving the cutoff frequency of a band-pass filter with fixed bandwidth..These simple modulations were chosen for two reasons:  Patients with severe disabilities can perceive clearly such simple changes in the sound.Moreover, music and stories have, sometimes, different audio characteristics: volume changes and band-pass filtering can be applied on the tracks getting good result in terms of patients' feedback.
 During rehabilitation sessions, therapists have been trying other more complex modulations, but these were not effective.For example, the spatialization of the sound on multiple speakers could not be perceived by patients.Another technique exploits multi-track music recording by progressively adding more and more audio tracks to regenerate the original song or by removing them, until the child gets to listen to a single instrument.This technique was not very effective with patients too.
a) CULT This application implements the motor and cognitive games the medical staff asked for.
The XML configuration file specializes the task of the game to focus the therapy on the specific disease of the patient: through the configuration file, therapists can parameterize velocity, position, dimension of the target, and how the target is displayed on the screen (i.e., the image associated with it).
EyesWeb generates a high-resolution graphic environment, using OpenGL external libraries, in which therapists can manipulate textures.
Figure 9 shows the creation of the target texture.From this information, EyesWeb evaluates whether the posture is correct and generates a log file containing frontal and lateral displacement (in mm) between head, shoulders, and waist and the occupation value frame by frame of a sensitive area therapists defined (see Figure 12).EyesWeb stores in a log file these values and therapists can perform a later evaluation of the game using any software for technical computing or using an evaluation tool integrated in the platform (Figure 11).A first approach for measuring whether the posture is correct consists of evaluating the displacements as in Figure 11.Another approach uses the occupation of the sensitive area.The therapist can determine, acting on sliders, the boundaries of the area in order to induce a gradual return to the correct posture.The EyesWeb patch computes the occupation rate of the body in the area (Figure 12).An audio file is modulated according to such measure.If the occupation rate is high, audio increases its quality, otherwise it is filtered and volume decreases.

c) CERT
This application is a variation and adaptation to the motor rehabilitation context of Sync'n'Move, a former application for active listening to sound and music content Casa Paganini -InfoMus developed in a previous project [3].Patient moves and limbs acceleration is detected and measured by the tri-axial accelerometer sensor.During the task, an index of phase synchronization is extracted in real time from the user's movements: the more users are synchronized (the index is high), the more instrumental tracks are added to the audio output, until the original song is fully reconstructed.If they do not succeed or stop moving (the index is low), the output sound is impoverished by removing instrumental tracks.The EyesWeb application consists of two modules (see Figure 14):  The Data Acquisition module acquires, calibrates, and computes the normalized acceleration sent by the mobile phones the users are moving.
 The Processing and Generation module computes the CPR phase synchronization index [9] and controls the audio output.We think, therefore, that an open platform for interactive exergames can be a tool of great interest, to complement conventional rehabilitation interventions both by improving our abilities to monitor and collect information and by supporting, the recovery process of the child with motor and/or cognitive impairment.Moreover we think that, although this tool can still be improved in the applications that have already been developed and integrated with new ones, its flexibility and modularity can allow its use both in the different stages of the recovery period and in different kinds of impairments.

Figure 1 .
Figure 1.Screenshot of the cognitive upper limb game.

Figure 3 .
Figure 3.The system architecture for the STARC project

Figure 4 .
Figure 4.The EyesWeb Social Signal Processing Library

Figure 7 .
Figure 7.A screenshot of the game configuration tool.It shows the first page in which therapists can choose the game.

Figure 8 .
Figure 8. Another screenshot of the game configuration tool.It shows the second page in which therapists can tune the parameters of a specific game.

1 .
Kinect v.2: a. Position, velocity, and acceleration of the limbs b.Quantity of motion of the whole body c.Directness of the hands movements d.Smoothness and fluidity of the hands movements 2. Accelerometers, Gyro and Magnetic Field sensors:

Figure 9 .
Figure 9.A portion of the CULT EyesWeb patch.It shows the generation of textures and their management.

Figure 10 .
Figure 10.Points obtained from body segmentation algorithm

Figure 14 .
Figure 14.Architecture of CERT games , with the possibility to add cognitive tasks for attention, memory and executive functions.Many of these abilities and/or skills are frequently damaged in children with central nervous system lesions or bedridden in severe diseases. HTCT (Head Trunk Control Trainer) aims to improve control and balance of head and trunk in sitting posture.
At the moment three different games are being developed:  CULT (Cognitive Upper Limb Trainer) is designed to support the recovery of upper limb functions (strength, control of movement, bilateral integration and coordination)