Virtual Reality and Audiovisual Experience in the AudioVirtualizer

INTRODUCTION: Virtual Reality (VR) provides new possibilities for interaction, immersiveness, and audiovisual combination, potentially facilitating novel aesthetic experiences.

OBJECTIVES: In this project we created a VR AudioVirtualizer able to generate graphics in response to any sound input in a visual style similar to a body of drawings by the first author.

METHODS: In order to be able to make the system able to respond to any given musical input we developed a Unity plugin that employs real-time machine listening on low level and medium-level audio features. The VR deployment utilized SteamVR to allow the use of HTC Vive Pro and Oculus Rift headsets.

RESULTS: We presented our system to a small audience at PROTO in Gateshead in September 2019 and observed people’s preferred ways of interacting with the system. Although our system can respond to any sound input, for ease of interaction we chose four previously created audio compositions by the authors of this paper and microphone input as a restricted set of sound input options for the user to explore.

CONCLUSION: We found that people’s previous experience with VR or gaming influenced how much interaction they used in the system. Whilst it was possible to navigate within the scenes and jump to different scenes by selecting a 3D sculpture in the scene, people with no previous VR or gaming experience often preferred to just let the visuals surprise them. They used mainly head movement to change their point of view, whereas people with previous VR or gaming experience quickly learned inherent capabilities of the system. The AudioVirtualizer system is most suited to respond to electronic soundscapes and microphone input.