Affordances and Constraints in Interactive Audio / Visual Systems

The recent years have seen a growing interest in crossings between the fields of Human-Computer Interaction (HCI), sound and music computing, and data visualization. This has resulted in the growth of areas such as Digital Musical Instrument (DMI) design, auditory display systems, and movement visualization. This is evidenced by the attention these areas have deserved in specialized conferences such as NIME (New Interfaces for Musical Expression)1, ICAD (International Conference on Auditory Display)2 and MOCO (International Conference on Movement and Computing)3. A recent symposium, SIIDS (Sound, Image and Interaction Doctoral Symposium)4 aimed to focus on cross-disciplinary approaches among these areas. The four papers in this special issue were all authored or co-authored by presenters in the first edition of SIIDS, which took place in October 2018. Although the topics of the four papers are varied in scope, they share common elements, and multiple unifying threads, which we will highlight in this editorial. One key aspect we detect in all four papers is the attention (explicit or implicit) to affordances and constraints, two key concepts in the field of HCI. In this editorial, we will present these concepts, in the context of HCI and their adoption in interactive music technology. We will then present the four papers in light of these concepts.


Introduction
The recent years have seen a growing interest in crossings between the fields of Human-Computer Interaction (HCI), sound and music computing, and data visualization. This has resulted in the growth of areas such as Digital Musical Instrument (DMI) design, auditory display systems, and movement visualization. This is evidenced by the attention these areas have deserved in specialized conferences such as NIME (New Interfaces for Musical Expression) 1 , ICAD (International Conference on Auditory Display) 2 and MOCO (International Conference on Movement and Computing) 3 . A recent symposium, SIIDS (Sound, Image and Interaction Doctoral Symposium) 4 aimed to focus on cross-disciplinary approaches among these areas. The four papers in this special issue were all authored or co-authored by presenters in the first edition of SIIDS, which took place in October 2018.
Although the topics of the four papers are varied in scope, they share common elements, and multiple unifying threads, which we will highlight in this editorial. One key aspect we detect in all four papers is the attention (explicit or implicit) to affordances and constraints, two key concepts in the field of HCI. In this editorial, we will present these concepts, in the context of HCI and their adoption in interactive music technology. We will then present the four papers in light of these concepts.

Affordances and Constraints
The concept of affordance was introduced in the design debate by Don Norman [1], who borrowed the concept from American psychologist James Gibson [2]. Gibson introduced the concept of affordances in 1979 in his book "Ecological Approach to Visual Perception". In Gibson's proposal, an affordance is a relational property that exists between the environment and an agent (either human or animal), consisting of the pairing between the intrinsic properties of an environmental element and the actions that an agent could potentially perform with it. The author proposed the idea of environmental affordance, that is "what it offers the animal, what it provides or furnishes, either for good or ill". In this view an affordance exists per se, as an absolute intrinsic characteristic of an object, independently of the agent's perception.
Developing from the proposal of affordance by Gibson, Norman [3] narrowed the concept to perceived affordances: properties of a given object that a person perceived. Applying the idea of affordance to the computer technology, Norman proposed: "The computer, with its keyboard, display screen, pointing device, and selection buttons (e.g., mouse buttons) affords pointing, touching, looking, and clicking on every pixel of the screen" [1]. Norman's views on affordances have been highly influential in the field of HCI, to the degree that the concept of affordance is usually part of the reasoning toolkit of most user experience designers.

Four Examples of Affordances and Constraints
We will reflect on how the authors of the papers in this Special Issue have dealt with issues of affordances and constraints in interactive audio or visual systems.
Fantechi presents an interactive system augmenting a classical guitar with contact microphones placed on 1

EAI Endorsed Transactions on Creative Technologies
Editorial Nuno N. Correia, Raul Masu its soundboard [4]. In the interaction with this system, "exploration becomes crucial". Fantechi elaborates on this exploration: "The possibility to work with specific gestures and extremely delicate sounds -together with the use of the electronics -enriches the vocabulary of sound material available within the compositional process".
To ground the possibilities of the system, and the explorations it allows, Fantechi uses the concept of affordances, borrowed from Gibson [2]. She discusses that behind the principal affordance of the classic guitar, "the possibility to produce sound by means of plucking its strings", lie further possibilities, accessible after a certain degree of experience. Additionally, there are "layers of embodied practices and idiomatic gestures, present in different repertoires, genres, historical practices and traditions". After becoming more sensitive to these "multiple affordances", the performer then adds her own agency, "often defined as 'expressivity' or 'individual touch'", emanating from diverse ecological factors. The introduction of two piezoelectric microphones (and associated electronics) adds more of these "multiple affordances". For example, the performer can adopt "certain gestures, almost inaudible on an unamplified instrument".
Dal Rì et al. also discuss music performance systems in their paper [5]. But instead of using the augmentation of an existing instrument (a classical guitar) as starting point, they create a new Digital Musical Instrument (DMI). They also use a multilayered approach, where the instrument (DMI) is 'augmented' to a certain extent, but by the genomic data of the performer. The affordances of their system are mainly conveyed through the DMI design, its interface (a sensor glove) and the mappings of genomic data to sound. But the main role of genomic data is to provide constraints to the system: "we propose to use the genome to define limits in the computational engine of a DMI that can then be played through an interface". The authors compare the constraints that the genome exerts over the DMI to the constraints our genomic data represents "in our human existence".
Data is also the main topic for Lenzi et al.'s paper, as their aim is to define design guidelines to create "sonifications that are both efficient and engaging" and "the transition of sonification into a mass medium for the representation of data" [6]. They explore the use of sound for real-time monitoring of anomalous behavior in digital and digital/physical systems, specifically for the sonification of data from a medium-sized water plant and from a medium-sized Internet network. Therefore, data here is not mainly a constraint, as in Del Rì et al., but raw material to create sound with, where "anomalies" in data provide affordances for special "mapping rules" of data to a soundscape. These mappings should lead to trigger a reaction in the operators in case of anomaly: "for example, searching for specific analytical information in the (. . . ) visualization system". In this sense, Lenzi et al. consider sonification as "complementary, and not a substitute, for data visualization". The constraints of data-tosound mappings are technical aspects such as the data streaming frequency, for example: "the constraint of data streaming at an hourly rate made it technically very difficult to use continuous, evolving indexical sounds in a meaningful manner".
If in the case of Lenzi et al.'s research, data sonification is considered a complement to data visualization, in Masu and Correia's paper data visualization (in particular, interaction data) is the key aspect [7]. They aim "to gain an understanding of the audience's perception of live visuals in contemporary dance, and what interaction design elements might be more conductive to audience enjoyment and understanding of the visuals". To that effect, they organized a performance consisting of "four different dance pieces with four choreographies, using four different designs for live visuals". The four different interaction design approaches were: motion capture (with tracking suit), sensors (breath sensor and position), camera (merely for image acquisition) and minimal interaction (small amount of data collected). Each design approach provided its own set of affordances and constraints for data visualization. For example, the authors identify affordances in the camera-based design for "a strong relationship with the actual image of the dancer", while creating a constraint of a "higher perception of duplication and redundancy", which were validated by the audience study.

Conclusion
The papers in this special issue present relevant new perspectives on affordances and constraints. One perspective relates to augmented instruments, and how the augmentation can provide additional layers of affordances in an already multi-layered ecosystem. Another relates to the use of data as both affordance and (mostly) constraint in the design of a Digital Musical Instrument. Anomalies in data also present affordances for data sonification, and for complementing data visualization. Finally, different interaction design approaches to the capture of bodily data can result in different affordances for data visualization. We hope that these papers can provide important insights for designers and researchers interested in developing their own systems related to interactive music technology, data sonification and data visualization -or combinations of these. 2 EAI Endorsed Transactions on Creative Technologies 01 2020 -04 2020 | Volume 7 | Issue 23 | e1