Introducing Neuroberry , a platform for pervasive EEG signaling in the IoT domain

The emergence of inexpensive off-the-shelf wireless EEG devices led researchers to explore novel paradigms in the field of Human Computer Interaction. In fact, the compliance of these devices with the IoT principles towards pervasive EEG signaling in smart home environments enables new models of interaction and a different perspective from traditional affective computing. In this paper, the implementation of wireless EEG (Emotiv EPOC and Mindawave) IoT connectivity of real time raw signals, through IoT hardware devices and through the Raspberry Pi 2, is presented.


INTRODUCTION
The availability of inexpensive wireless electroencephalography (EEG) devices has opened new research perspectives spanning from clinical assessment to human-computer interaction.Wireless headsets are compact and easy to be worn during activities, as the Emotiv EPOC [7] and Neurosky Mindwave [17], and could be deemed as representative devices of the field.Although both of them have not been introduced as medical devices, the comparatively little effort required for installation and use leads to augmented flexibility and mobility over traditional EEG [18].
Emotion recognition [9][19] [25] and cognitive load [4] [22] [23] through wireless EEG devices is a very active area of research, considering that emotions and affective computing has been in the eye of the research community during at least the last two decades [14].Moreover, monitoring player's gaming experience [18] and also while performing serious games [24] has emerged as a very relevant application area.In addition, similar systems have been demonstrated to work well detecting focused thoughts [16] [10] or even controlling smart home devices [21].The maturity level of the research along with the vision for commercial applications based on EEG signals is reflected by the emergence of application stores specifically targeted at exploiting EEG data in different application domains.
On the other hand, the recent convergence of advances in ubiquitous computing, consumer electronics, and wearable sensor technologies have led to the need for standards to handle intercommunication among these devices and applications.Dealing with the physical integration of everyday objects and their unique capabilities on the Internet [20], Internet of Things (IoT) tackles some of these challenges.However, the "device independency" principle of IoT is sacrificed for the development simplicity and specific knowledge requirement, by employing mobile devices as gateways [6][12] rather than developing IoT enabled devices per se.Hence, the device has to be always paired with a smart phone [15].
Aligned with the increased adoption of IoT systems, a large number of IoT compliant hardware and software frameworks have emerged to facilitate developments on the field.Raspberry Pi 2, MinnowBoard Max, Galileo and Arduino are some of the IoT ready hardware devices while Windows 10 IoT Core, WebIOPi, IoTivity, ThingBox and Smart Living are only some of the available frameworks.In addition, there are custom frameworks following the IoT principles that are designed to support commercial game controllers and wireless EEG devices, like the Controller Application Communication (CAC) framework [11].
However, the literature does not exhibit many efforts towards the combination of IoT and the discussed wireless EEG devices.More specifically, the Appmethod IoT framework has been used to connect the Emotiv EPOC as a control source to other IoT devices/gadgets like the Google Glass and the Parrot AR Drone 2.0 1 .However, there is no provision for other wireless EEG devices.As an omen of our work, Sinharay et al. [23] presented their work by assuming the existence of a cognitive load information through commercial low-cost, low resolution, wireless EEG devices connected to the cloud through an Internet of Things (IoT) platform in real-time that could be utilized for different applications.
All this considered, the aim of this paper is to present an IoT approach of the emerged wireless EEG devices though a low cost commercial hardware (Raspberry Pi 2).Although there are implementations where these devices are connected to the Internet through a smart phone acting as a gateway, this work is one of the first implementations towards wireless EEG IoT connectivity of real time raw signals through IoT hardware devices.

MATERIALS AND METHODS 2.1 CAC framework
The CAC cross device-application communication [11] framework facilitates the connectivity and data exchanging among applications and controllers.Based on the standard, real time, Internet communication technologies of Websockets (a Restful web service interface is also available for compatibility with obsolete technologies), the CAC framework encapsulates the raw information from the device, in a structured way and streams it to the subscribed clients.Adopting a lightweight publish/subscribe messaging mechanism, the framework's design principles align with the Internet of Things (IoT) paradigm.A controller/device subscribes to the framework and publishes its information.Hence, every client (application or device) that is subscribed to the same session, receives the corresponding data.Each streamed packet is transmitted along with information regarding the origin device (device type, session ID, mandatory packet transmission) and the current frame (packet's date and time in milliseconds at the source).Given the fact that contemporary controllers are not Internetenabled yet, a processing component / application is in charge of acquiring information from the device and transmit it to the CAC server.
It must be highlighted that any type of modern hardware and software that can access the Internet, including conventional computer applications or embedded software in custom hardware such as smart phones, smart TV sets, tablets, or robotic devices, can subscribe to the CAC framework and get access to the streamed information.

Neuroberry Architecture
The Raspberry PI 2 low cost device (<€50) was selected for the Neuroberry IoT implementation.The Neuroberry system consists of three independent components (c.f. Figure 1): a) the data acquisition of the Mindwave, b) the data acquisition of the Emotiv EPOC and c) the data streaming to the CAC server component.Considering that components need to operate in real time, multiple threads are in charge of specific tasks, also to secure the uninterruptedly functionality of the thread itself by absorbing any hardware or communication error.Additionally, synchronization techniques ensure the highest possible streaming speed (if a packet is currently submitted, the forthcoming packet is skipped to prevent 1 http://blog.appmethod.com/developing-the-internet-of-thingsemotiv-epoch-brain-computer-interface 2 https://github.com/BarkleyUS/mindwave-pythonpacket collision, etc.).As far as the programming language is concerned, Python has been chosen also thanks to the light-weight handling of processes in terms of memory requirements.In addition, the Python script was configured to be executed as a service during startup.Hence, Neuroberry (c.f Figure 2) can be deemed as a black box gateway for the wireless EEG devices without the need for an external monitor screen.

Mindwave and Emotiv Data Acquisition
The Mindwave EEG wireless device with no mobile compatibility was used during the development.Therefore, the Mindwave Data Acquisition component was based on the BarkleyUS/mindwavepython library 2 .Accordingly, Emotiv Data Acquisition component was built upon the ozancaglayan/python-emotiv 3 .Both components connect to the corresponding USB port for initializing and establishing the communication with the corresponding headset.As far as the headset remains connected, every new received packet is transmitted to the CAC framework server by invoking the Websockets component (running in a different thread).Table 1 presents a Mindwave data packet formed according to the CAC framework.When a hardware or communication error takes place (e.g.USB dongle unplugged), the component reinitiates itself and enters the standby mode waiting for a new connection to be established.

WebSockets client (python) and client protocol
Widely available to any programming language, the Websockets client relies on the websocket-client library 4 .The Websockets thread is responsible to keep the thread alive and always connected (when this is possible) to the CAC framework server.On startup as well as in case of disconnection, the component subscribes to the Websockets session by sending the corresponding Session ID.The Websockets component communicates with the other threads by exposing two methods.The SendMessageMindwave method accepts as argument a Mindwave headset object while the SendMessageEpoc method gets an Epoc headset object as argument.The role of these functions is to form a JSON message packet in compliance with the CAC framework and transmit it.Taking into consideration that real time transmission and especially streaming a considerable number of packets per second is not a simple task, the component drops any incoming packet when a transmission is already in progress.This way, high streaming speed is maintained at the cost of dropping some packets.

SCENARIOS
The UNCAP project intention is to deliver a platform based on open industrial standards able to create new care and assistance paradigms.Cognitive or physical exercises, delivered as part of the UNCAP solution, will create incentive mechanisms to help keep users healthy from a physical and cognitive point of view through games.The cognitive exercises will include puzzle solving while the physical exercises will aim at full body standard physical exercises [13].Delivered through different devices and interfaces (the cognitive exercises through PC while the exergames through smart TV or tablet), the games indicated the need for a unified solution of the wireless EEG connectivity and employment.
Following Styliadis et al. work [24] on employing affection in elderly healthcare serious games intervention by applying wireless EEG recordings during exergames, the senior will be asked to utilize the Emotiv EPOC or Mindwave wireless EEG sensor as part of the game.Therefore, the senior's mental status and gaming experience, including focus, engagement, interest, excitement, affinity, relaxation and stress levels, will be monitored and collected during gameplay.

Puzzle game
One of the envisaged scenarios for the application of such technology is in the implementation of puzzle games, where the user is required to compose a puzzle starting from a template image presented on the screen.The game involves and stimulates the physical and cognitive abilities (tiles require manual handling) while being recorded from an ad-hoc acquisition system relying on a regular webcam.The system aims at performing visual matching between the image template and the actual tile positioned on the grid.The system acknowledges the operations through a simple visual interface that turns the corresponding square to green in case the tile is positioned correctly.In order to make the tool more engaging and challenging, puzzles are made of 3D cubes, so as to compose multiple images starting from the same set of cubes.For a complete profiling of the users, additional metrics are collected from the system besides the neurological signals (e.g.cognitive load, attentions, etc.), as the time of execution or the matching score.A sample snapshot of the gaming station is provided for completeness in Figure 3.

HTML5 online games
Another scenario of use is the adoption of the wireless EEG sensors as game controllers in exergames.Therefore, studying emotions, user's mental status, focus and stress levels within the gaming environment and in-game metrics would give a more comprehensive overview of the users and their gaming experience.Moreover, exergame analytics [2] may reveal new knowledge about the role of exergames on the seniors.

DISCUSSION
Our work was motivated from challenges that emerged in the area of IoT and the recent trend for pervasive real-time EEG signaling as well as the need for conducting ecologically valid interventions on active and healthy aging in naturalistic everyday environments [1].It can be inferred that the combination of the main characteristics of IoT, such as Ubiquitous Communication / Connectivity and Pervasive Computing [8] with the wireless EEG interfaces could enrich the Ambient Intelligence approach.More specifically, multiple interfaces (e.g.serious games) could get concurrently access to the wireless EEG devices' output at the same time, regardless of the software and hardware used.Following this approach, the user's mental state, which is in the interest of the research community [14], will open roads in the pervasive Human Computer Interaction.
The work presented in this paper complies with the intuitive interface of the wireless EEG devices, as it is defined by the unobtrusiveness [21].This, in conjunction with the perceived ease of use, which is of primary interest in determining the attitude towards ICT [5], would turn these devices into acceptable means of HCI in daily life.Therefore, scenarios like the ones presented in this paper, which are representative of the recent Active and Healthy Aging smart homes paradigms [3], would be feasible to be carried out at the seniors' homes, provided the capability of embedding modules for Internet connection directly onboard of the EEG.However, such devices should be equipped with native Internet connectivity through standard IoT protocols (MQTT, XMPP, etc).
In the future, the authors' intention is to incorporate a web server into Neuroberry as a remote configuration interface and an internal recording component.Being at the same architectural level with the Websockets, this component will record the measurements in an internal database in case of limited Internet availability.This will give also the opportunity for offline monitoring and subsequent analysis.Finally, the Raspberry PI system image will be available soon for downloading at www.cac-framework.com.

Figure 1
Figure 1 Neuroberry architecture.Each color represents a separate thread.

Figure 2 The
Figure 2 The Neuroberry connected to a monitor through HDMI for demonstration purposes.

Figure 3 .
Figure 3. Overview of the gaming station and the visual interface.