Bioelectric signals of the body: from electronic engineering to artistic performance

This paper aims to present the experience of the use of bioelectric signals, particularly those present in the muscles (EMG: electromyogram), to link an artist with his/her environment by producing sound and video images according to these signals. For this purpose, a device that works like an instrument the performer executes acting on his/her muscles was developed and built. It produces OSC messages, a protocol commonly used by media artists, according to the intensity of the EMG signals the performer produces, thus providing direct communication with the audience. Two presentations were made with the proposed scheme, which allowed to observe the sensations of the artist and the public, as well as their interactions.


Introduction
This paper presents an interdisciplinary research initiative, whose specific purpose is to advance the practical application of electronic human-machine interfaces (HMI) to the human body in performances. In this research, artists, researchers, and engineers from the National University of Tres de Febrero and the Biomedical, Industrial and Scientific Instrumentation Group (GIBIC) of the National University of La Plata participated.
The outcome of the interaction between the technical sensors and signals from human biological functions and anatomical transformations creates highly localized and unique performance practice that binds gestural micromotions to a bioelectric sensing system.
The information gathered from the sensors is translated in real time into visual and acoustic elements through computational algorithms. This allows an almost completely immobile body to communicate through micromovements * Corresponding author. Email: aceriani@gmail.com and have these translated and amplified into visual and acoustic communicative outputs.
We identify the electrical activity of even subtle muscle movements with an electromyograph (EMG) and then encode the digital output from the sensor into visual and acoustic signals. This opens up the possibility of designing a new order of communication from micromovements amplified and translated into sound and video.
The process starts with the concept of body performance, which takes into account the aspects of positive micromobility within the apparent general immobility. The micromovements generate a series of signals, which may then be encoded into a communicative output, which is acoustic or visual.
As we explore how micromotions can be detected by a sensor and generate signals (even micromotions of which we are not consciously aware) we demonstrate what appears to be immobility still may have communicative power as dance and performance through the sensory power of EMG sensing systems. This experience is made possible by joining the disciplines of art and bioengineering.
A.C. Ceriani et al. 2 The intention, from the perspective of artistic research, is to sense the distribution and intensity of electrical activity in the tissue layers by an array of electrodes on the scalp (EEG: electroencephalography) or proximate to a muscle (EMG) or the heart (ECG: electrocardiography), and register the data obtained from this procedure to from a communicative output. Particularly, the distribution of micro movement intensities (EMG) is sensed and processed and then used in an artistic application.

Interactive performance with bioelectric sensing
The initial artistic production will be a dance which will use biomedical instrumentation made available by a local public university.
As Alejandra Ceriani says [1] "Interactive performance: this conjoining of terms comes from the definition of metaformance, by Claudia Giannetti (1997), in which a connection is established between the idea of the construction of a body, the technical metaphor and the notion of the artistic process. The Interactive performances form a new category within the sphere of performing arts and join several branches of interactive art and traditional visual art; since they do away with the physical presence in place of action and simulate the action with an electronic-digital image" (Ceriani, 2018: 13).
To this we will add this new definition: Interactive performance with bioelectric sensing is a species of performance which utilizes the biological signals of the body itself. The performer is connected to wearable neuroprostheses with superficial electrodes which are noninvasive transducers which capture the electric signals produced and propagated along the performer's body, called biopotentials.
Wearable neuroprostheses: assistive devices which, with an installation as simple as wearing a piece of clothing, must be able to interpret the will of their user and act on the physical world accordingly. How is it that these specifications of biopotencial instrumentation influence the measurement of the signals and how do these manifest the will of the participant?
Inside of our skin there is a world of meaningful electrical signals produced by the depolarizations of nerve cells to communicate messages to or through biological tissues such as muscles. These signals coordinate willful actions.
These signals transport information not only about the status of body's health but also about the will of the individual along with the ideas and the desires of the people in which they originate. (Guerrero, 2017: 3) [2].
Capturing and processing the electrical signals produced by the muscles during a muscle contraction is known as electromyography (EMG), that consists essentially in the acquisition and analysis of electrical activity generated in the neuromuscular synaptic junctions through the use of electrodes (on the Surface, by needle, or implanted). The measurements by the EMG render valuable information about the physiology of the subjects of muscular action. Therefore, this system, which allows the acquisition of bioelectric signals, makes visible the internal impulses and processes of the body the performer can control. Thus the performer may begin to control subtle changes in muscle tension in order to generate changes in acoustic and visual output in the performance. Also the performer may control multimedia events jointly, which makes possible a flow of sensory information. We should not only be content to do this experimentally in order to understand the functions of bioelectric synergy along with the dynamic processes to which it gives rise as well as how the signals are organized to coordinate motor functions. Eventually, beginning with what we discover with this project, possible explanations which may identify and schematize the biofeedback or feedback between the measurable motor properties and their interaction with the bioelectric sensing in interactive performance or scenic environments.
The time, which passes in an interactive, performance with bioelectric sensors has considerable complexity. It is, after all, a performance in temporal space, which tends to provoke new relationships gesturing a self-referential perception through empathy, which is experienced by the attendees. This is fundamental for our project since it involves our physical bodies.
In this sense, the performance is experienced as a poetic or metaphorical process, which shares an internal and complete kinetic connection produced by sensory data. At present, most scientific and technological research groups focus their efforts on signal processing and pattern classification. Even so, all seem to agree on the importance of continuing the investigation through a greater interrelationship between designers and users to arrive at a better understanding of how we might more accurately and precisely sense a performer in motion.
It is also important to train the body in the techniques based on biofeedback, this training will allow a performer to reliably generate a signal pattern based on their expressive and creative performance.
Therefore, joint experimentation with the GIBIC / LEICI is an exploration in this interdisciplinary research which combines the practical application of disciplines related to electronic engineering, industrial design, different multimedia programming languages and artistic performance. From this link, two scenic presentations have been made (see Fig. 2 This opens up the possibility of working with artists and art researchers in various technical and scientific laboratories of the state university. Such joint work will not only allow access to professional devices and systems, but also to create and produce artistic works by applying the expertise and knowledge from science and engineering.
Consequently, this use of biomedical signals for interactive performance dance expands the expressive universe to include people with motor disabilities allowing them new dimensions of creative and expressive output through a scenic and technological mediated activity which otherwise would be difficult if not impossible.

The superficial EMG signal as a window to the artist's will.
The surface EMG (sEMG) technique allows non-invasive detection of the electrical activity of the muscles by placing electrodes on the skin, improving the user's comfort and minimizing invasiveness, compared to other EMG techniques that use needles or implantable electrodes [3]. The measured signal is a composition of the electric activity from the nearest muscular tissue, which is closely related to the contraction of the muscles. The brain sends electric impulses (called action potentials) to the muscles through the neuronal circuits, that contracts briefly the muscle fibers. To increase the contraction's strength, and its duration, the brain fires more action potentials repetitively and can start to contract more groups of muscle fibers (motor units). For weak contractions, it can be registered the individual electric spikes (of some microvolts) in the sEMG signal. If a person can control these weak contractions, he or she can produce sEMG signals without any observable movement. As the strength and intensity of contraction increases, the contribution of the different motor units interferes with each other, disturbing the shape of acquired electric pulses, becoming the sEMG signal in a random signal with its amplitude or energy correlated with the contraction level of the subjacent muscles.
In both cases, the sEMG has features that the performer can control at will.

SEMG sensing
The sensing of a sEMG channel requires at least two electrodes, for measuring the differential voltage between them, and a third one as a DC reference. These electrodes are made of particular metallic materials, for instance silver, nickel or steel, and they can have an electrolytic gel or liquid as covering, that enhance the skin electric contact. When this covering is not used the electrode is known as a "dry electrode". These are the standard electrodes in wearable neuroprostheses that requires easy placement and low maintenance but present a poor electric contact with the skin. Also, to avoid interference, it is required the placement of instrumentation electronic (buffers or pre-amplifiers) just over the electrodes. The set composed by the electrodes and the instrumentation electronic is known as "EMG sensor".
In the context of an artistic performance, it is desired to acquire the electric activity generated by one or more muscles related with a particular movement or gesture, therefore the sensors are placed over the skin, near those muscles.
A significant parameter of the EMG sensors is the distance between electrodes, which can range from 5 millimeters to a few centimeters. Bigger inter-electrodes distances result in greater sEMG amplitude but reduce the high frequency content of the signal and deteriorate the rejection of crosstalk (EMG produced by nearby muscles) [4,5].

EMG envelope estimation
In an artistic performance the observation of sEMG signals allows to know not only the artist's intention channeled through gestures, movements and contractions of different degrees of intensity, but also some internal and involuntary processes as his/her hearth rate or muscular tone.
Nevertheless, the raw sEMG is a noisy signal with high bandwidth respect to the information it carries about the contractions. That is, the raw sEMG is "faster" and more "complex" than the movement-related information. For instance, to digitally represent the raw sEMG there are needed about 1000 samples per second (one value every 1 ms) of the signal, while the time of human movements ranges from hundreds of milliseconds to several seconds.
A way to keep the things simple is to process the raw sEMG to extract some simpler and slower contraction-related features or parameters. As previously mentioned, one of these features can be the amplitude or energy of the sEMG (better known as envelope), that is correlated with the degree of strength or contraction of one or more muscles. The envelope can be thought as a slow curve that connect the points of maximum amplitude of the raw signal (see Fig. 3), hiding the complexity of the physiologic signal while keeping the strength information. Its calculation is simple and computationally efficient, and it can be done by many ways. For the case of the Fig. 3 a digital processing scheme was applied to compute the EMG envelope, that consists of the following steps: (i) The sampling of the raw signal at 1000 samples per second (ii) The high-pass filtering of the raw signal to reject movement artifacts and DC drifts (iii) The segmentation of the raw EMG in 20 ms (50 samples) non-overlapped windows (iv) The computation, for each window, of an average of the absolute values of each sample. The value calculated for each window, every 20 ms, is proportional to the EMG envelope. (v) (Optional) The envelope signal computed in (iv) is smoothed by a low pass filter.
Therefore, the described simple algorithm converts a complex raw signal of 1000 samples per second to a smooth envelope signal of 50 samples per second, that preserves the movement-related information.

WIMUMO: a performance-oriented device
To introduce the sEMG signals and other bioelectric signals to the artistic practice, in a reliable and robust way, the development of a new electronic device was carried out, which is a kind of non-invasive neuroprosthesis. This device aims to be a bridge between the performer's internal world (that is observed through biopotential sensors and other auxiliary signals) and the world of the media artists: hardware and software used to create music and audiovisual content, such as Processing (https://processing.org/) and Pure Data (http://puredata.info/).This device must also be portable (small, lightweight and wireless) to allow the freedom of A.C. Ceriani et al. movement of the artist in the stage. Another desired features are to be easy to wear and configure, reducing the dependency of technical assistance.
The developed device was called WIMUMO (Wireless MUlti-MOdal acquisition platform), and is a portable and wearable system (see Fig. 1) composed by the integrated fourchannel biopotential front-end (ADS1299-4) with high resolution Analog-to-Digital Converters (ADCs), a set of EMG sensors, an external 5 V power-bank, and a single board computer (Raspberri PI Zero W) with Linux as its operating system and wireless capabilities (Bluetooth and Wi-Fi). The Wi-Fi technology was chosen for the WIMUMO because its greater spatial range respect the Bluetooth.
The computer reads the raw EMG samples from the ADCs, and processes them to calculate the EMG envelope of each channel. These envelope signals and also -optionally-the raw EMG, are transmitted to other devices in the local network for the generation of audio-visual content, using the OSC (Open Sound Control) protocol, that is a standard protocol widely supported by the hardware and software used by media artists.

Organization of the computer software
Even though the interaction with the acquisition board, the processing and the distribution and display of data, in parallel with the configuration and connection to a network, are performed on the same single board computer, each task operates in an independent software environment with their own technologies [6].
As explained above, for each EMG channel, the samples are grouped to calculate their envelope; for this, a compiled program (written in C language) is responsible for quickly reading each simple of the ADC, processing and transferring internally to other programs the result of the calculated envelope (as well as the original raw signal). The mechanism for communication between processes used were the Linux pipes.
In another process, a Node server (programmed in JS) collects the signals shared by the previous program and displays the web interface that interacts with the user, in this case the performer. Through the different web screens implemented is possible to: • Configure the communication with a Wi-Fi network.
• Configure the sending of OSC packages to other devices connected to the same Wi-Fi network. • Configure different parameters of EMG signal processing • Visualize, in soft real time, signals acquired and processed using semi-interactive graphs.

The IoT paradigm and usability
According the definition of Internet of Things (IoT) as system of interrelated computing devices, mechanical or digital machines or objects that are provided with unique identifiers (UIDs) and the ability to transfer data over a network without requiring human-to-human or human-to-computer interaction, the WIMUMO is a device that fits to that and complains to the IoT paradigm. It allows, after a simple configuration, uninterrupted operation, independent of human interaction, ensuring that the performance is in the hands of the artist and the multimedia team.
The use of a web server accessible from the browser of any mobile phone or computer with Wi-Fi connection allows the performer to carry out the necessary initial configuration in a simple and intuitive way. This is the only way in which the user can configure the WIMUMO, since it does not have its own interface (no buttons or screen).However, this does not undermine the usability of the device, since connecting to a Wi-Fi network and accessing a web page from the browser are common tasks for any average user.
Like other devices that have a connection to a network without its own interface where can be entered the credentials of the Wi-Fi network (printers, routers, Chromecast), being unable to connect to a known network, the WIMUMO configures itself as an access point, creating an own Wi-Fi network and allowing the connection of clients to it. Once connected to that Wi-Fi network, for example from a mobile phone, the performer can enter to the website served by the WIMUMO and configure it. Through it, the name of the Wi-Fi network that will be used for the communication between components is selected from a list and its password is entered. If the connection was successful, WIMUMO is configured as a client of the external network.
Finally, in a different section within the same web page, the IP address and port to which is wanted to send the OSC packages with the sensed values of the biopotentials and their envelopes are indicated, starting the transmission of them.

Results
The inclusion of bioelectric sensors in artistic performances was tested in two public presentations (with an audience) and also in several training sessions. Both public performances were carried out by Alejandra Ceriani. One of them, "Collective Imagination", was held in October 2018 at the Center for Art at the National University of La Plata, Argentina, together with the SPEAK group. This group has developed an open-source software named MOLDEO (www.moldeo.org), to produce multimedia content from different sources such as OSC and MIDI messages, cellular phones and game consoles devices, among others.
The second performance, "Cajografias", was held in November 2018 at the General San Martín Cultural Center, Buenos Aires, Argentina, as part of the event "Electronic November".
For this performance, EMG sensors were placed on the forearm, chest, face, and leg (see Fig. 2). The signals these sensors produced were sent as OSC messages to a computer that generates sounds by using the PureData software (http://puredata.info). Four chords were programmed and their volumes controlled by the amplitude of the sEMG signal envelopes. Simultaneously, a musician produced sound and Bioelectric signals of the body: from electronic engineering to artistic performance noise according to the performer activity, thus composing a collaboration work. A short video of this presentation can be seen in https://youtu.be/p1W4-g8w84A.
In addition to public performances, several tests were carried out. In one of these, two performers with bioelectric sensors installed on their bodies were involved. In these trials, cardiac activity (ECG) that the artist cannot control, but that depends on his emotional state and physical exertion were used. In addition to these signals, which were used as a rhythmic base, EMG sensors were installed on forearms (flexor muscles of the fingers), on the face (major zygomatic muscle) and on the leg (anterior tibial muscle). The envelopes of the EMG signals produced by the artists were transmitted as OSC messages to a computer that generates sounds using the PureData platform. A track recorded during one of these trials can be found at the bandcamp platform.(https://gibic.bandcamp.com/album/ed-1).

Conclusion
An important conclusion of the interdisciplinary work presented in this article is the experimental verification that it is possible the inclusion of biomedical signals in artistic performances, if wearable neuroprostheses are used. This equipment, usually intended for the field of assistive devices for handicapped persons or research purposes [7], usually act as alternative communication channels for the user, allowing him to control devices such as a computer, spellers, or different artifacts from his environment.
During the first stages of the joint work between performers, media artists and engineers, the feasibility of using biopotentials in performances was explored, using general-purpose biopotentials acquisition equipment available at the Laboratory. These devices that were not specifically designed for artistic performances, allowed to verify the feasibility of use of biopotential signals for this purpose but they also show that specifically designed equipment is required, in order to preserve performer's mobility on stage and to provide a friendly communication with software and hardware of media artists.
Based on these preliminar tests the WIMUMO was developed. This new wearable neuroprosthesis for the performance practice has four EMG sensors that capture the muscular activity of the performing artist, processes these signals in real-time and uses the OSC protocol to transmit the acquired and processed signals to the multimedia artist's domain. It also allows connecting another kind of sensors like temperature, humidity, and acceleration, among others.
The WIMUMO device complains to the Internet Of Things (IoT) paradigm, creating its own Wi-Fi network or connecting to an existing one. It can be configured through any web browser without installing custom software. The device is designed taking special care in that its placement, commissioning and set up do not require advanced technical knowledge.