See different for Sensorial Substitution from Vision to Audition
See Differently has been developed by the Neurocomputational and Intelligent Signal Processing (NECOTIS) research group. NECOTIS conducts multidisciplinary research in the area of intelligent systems and machine communications, machine learning and computational neuroscience. Applications are developed around bio-inspired neural network systems information and signal processing. See differently is one of these applications.
Plasticity and sensorial substitution (Prof. Bach y Rita)
Our brain is very plastic and the pioneering work conducted by prof. Bach y Rita and his research team has shown that it is possible to substitute senses. This excellent video summarizes that aspect of the work by prof. Bach-y-Rita.
A substitution system from vision to audition
See Differently is an implementation and a simplification of a sensory substitution system that maps a visual scene into a 2D audible scene. The mapping is done after analysis of the image snapshots taken from the video output of the camera. Sounds are synthesized in such way that location of sound sources is perceived in a virtual acoustical space by exploitation of the Intensity Level Differences (ILD), the Interaural Time differences (ITD) and the spectral characteristics of Head Related Transfer functions (HRTF).
See Differently first performs an analysis of the visual scene and reacts to area of interests inside the visual scene. There is no need to encode the full visual scene into sounds. Only area of interests are encoded into sounds. As a proof of concept, a light version of See Differently has been implemented for use with iPhones/iPods and is freely available on the App Store.
Characteristics of light See Differently
In the actual configuration, light see differently reacts to area in the video inputs with a dense activity of a neural network. The greater the density of textures or gradients, the more active is the neural network. A more elaborated research system is under development. Actually it recognizes specific objects in the visual scene.
How to use light See Differently
As a game, “See Differently” can be used to find basic objects or friends by using sounds without vision (blind man’s buff – “Colin-maillard”).
As a creation tool, “See Differently” can be used to “compose” simple melodies.
“See Differently” can also be a good coach to train people to locate sounds in a 2D auditory sound scene.
“See Differently” can be used to create a relaxing sound immersion environment as natural sounds are used.
Headsets to be connected on the audio jack of the device should be used to perceive the 2D sound effects.
Close your eyes and do not look to the squares on the screen to better feel the effect of the 3D sounds (after being trained).
Some listeners might have difficulties to hear the elevation. This is because the perception of elevation is dependent on the shape of our pinna. For now See Differently assumes that everyone is having the same pinna and is using a standard pinna shape. Those of us which pinna’s shape matches the one that See Differently uses should perceive the elevation. Others might not. Some others might learn to perceive the elevation.
Video illustration of See Differently as a simple musical interface. Move the mouse on the iPod illustration and start the video.