Eelke Folmer
Human+ Lab
University of Nevada, Reno

Real-time Sensory Substitution

Video games rely upon being able to see, as visual cues typically indicate what input to provide and when. Players that are blind are unable to perceive these cues and cannot determine what input to provide. Several accessible video games have been developed where visual cues are substituted into compensatory modalities, such as audio or vibrotactile feedback and which allow for users who are blind to play these. To implement sensory substitution you need access to the source code, which for most commercial video games is not possible. This project explore whether alternative cues can be provided without having to modify the game, using a technique we've dubbed real time sensory substitution (RTSS).

How it works

RTSS uses video analysis to detect visual cues in the game and substitutes these with compensatory cues on the fly. We demonstrate RTSS by implementing it for Kinect Sports to make the hurdles game accessible. In the hurdles game, the upcoming hurdle will glow green when it is time to jump (see screenshot below). An external application running on a PC analyzes a video feed of the game and looks for this specific cue in a predefined area. When we detect this cue we provide the player with a haptic cue indicating that it is time to jump. These haptic cues are provided with a Wii remote that is connected to the PC.


The custom glove we developed with a pager engine attached to each finger
The Kinect hurdles game uses a visual cue that indicates when to jump.


Tony Morelli and Eelke Folmer. Real-time Sensory Substitution to Enable Players who are Blind to Play Video games using Whole Body Gestures, Entertainment Computing, 5(1), Pages 83 - 90, 2014.

Tony Morelli, Eelke Folmer. Real-time Sensory Substitution to Enable Players who are Blind to Play Gesture based Videogames, In Proceedings of Foundations of Digital Interactive Games (FDG'11),Pages 147-153, Bordeaux France, June 2011. [29% acceptance rate]