Gaze-contingent audio-visual substitution for the blind and visually impaired

A lot of effort has been made to advance technologies that improve blind and partially sighted people’s spatial perception. One common approach is to enhance or substitute vision by audition. Most sensory substitution systems, however, have not attached any importance to eye movements. But eye movements play an essential role in mental imagery even in the absence of visual input. Therefore, we propose a system for gaze-contingent auditory substitution of spatial vision. It is intended to be a mobile helper in everyday life of the visually impaired. The prototype we have developed combines eyetracking with depth measuring and sonification techniques. We carried out both a proof-of-concept study in complete darkness and an exploratory EEG study. Our findings indicate that gaze-contingent sensory substitution permits depth perception and leads to intermodal (audio-visual) processing in untrained subjects. Hence, as a result of neuroplasticity, the blind and visually impaired might learn to perceive gaze-dependent sound visually.