Vibrotactile Captioning of Musical Effects in Audio-Visual Media as an Alternative for Deaf and Hard of Hearing People: An EEG Study

IEEE Access

Maria J. Lucia¹  Tomas Ortiz¹  Ricardo Vergaz²  Alvaro Garcia³  Victor Cerdan⁴  Belen Ruiz⁵  Pablo Revuelta⁵ 

[1] Department of Computer Science, Carlos III University of Madrid, Legan&x00E9;Department of Electronic Technology, Carlos III University of Madrid, Legan&x00E9;Spanish Center for Captioning and Audio Description, Carlos III University of Madrid, Legan&x00E9;s, GDAF-UC3M, Spain;s, Spain;

关键词: Music emotion recognition;    hearing impairment;    vibrotactile;    audio-visual;    captions;    accessibility;   

DOI  :  10.1109/ACCESS.2020.3032229

来源: DOAJ

【 摘 要 】

Standard captioning for the deaf and hard of hearing people cannot transmit the emotional information that music provides in support of the narrative in audio-visual media. We explore an alternative method using vibrotactile stimulation as a possible channel to transmit the emotional information contained in an audio-visual soundtrack and, thus, elicit a greater emotional reaction in hearing-impaired people. To achieve this objective, we applied two one-minute videos that were based on image sequences that were unassociated with dramatic action, maximizing the effect of the music and vibrotactile stimuli. While viewing the video, using EEG we recorded the brain activity of 9 female participants with normal hearing, and 7 female participants with very severe and profound hearing loss. The results show that the same brain areas are activated in participants with normal hearing watching the video with the soundtrack, and in participants with hearing loss watching the same video with a soft and rhythmic vibrotactile stimulation on the palm and fingertips, although in different hemispheres. These brain areas (auditory cortex, superior temporal cortex, medial frontal cortex, inferior frontal gyrus, superior temporal pole and insula) have been consistently reported as areas involved in the emotional perception of music. We conclude that vibrotactile stimuli can generate cortex activation while watching audio-visual media in a similar way to sound. Thus, a further in-depth study of the possibilities of these stimuli can contribute to an alternative subtitling channel for enriching the audiovisual experience of hearing-impaired people.

【 授权许可】

Unknown