Realtime Integration Of Gesture And Speech During Reference Resolution

27th Annual Conference of the Cognitive Science Society

Ellen Campana ([e-mail]) ; Department of Clinical and Social Sciences in Psychology ; University of RochesterRochester ; NY 14627 ; USA ; Department of Brain and Cognitive Sciences ; University of RochesterRochester ; NY 14627 ; USA ; Department of Clinical and Social Sciences in Psychology ; University of RochesterRochester ; NY 14627 ; USA

PID  :  118547

来源: CEUR

【 摘 要 】

There is some disagreement among researchers about the roleof gesture in comprehension; whether it is ignored, processedseparately from speech, used only when speakers are havingdifficulty, or immediately integrated with the content of the co occurring speech. The present experiment provides evidence insupport of immediate integration. In our experimentparticipants watched videos of a woman describing simpleshapes on a display in which the video was surrounded by fourpotential referents: the target, a speech competitor, a gesturecompetitor, and an unrelated foil. The task was to “click on theshape that the speaker was describing”.In half of the videosthe speaker used a natural combination of speech and gesture.In the other half, the speaker’s hands remained in her lap.Reaction time and eyemovement data from this experimentprovide a strong demonstration that as an utterance unfolds,listeners immediately integrate information from naturally co occurring speech and gesture.BackgroundNondeictic manual gestures are ubiquitous in languageproduction. People produce gesture even when theirinterlocuters cannot see them (Alibali, Heath and Myers,2001) and can experience difficulty speaking when theirhands are restricted (Graham and Heywood, 1975).There issome consensus that the production system allows differentaspects of a single message to be expressed simultaneouslyover, and distributed between, linguistic utterances andgestures (Alibali, Kita and Young., 2000) However,researchers disagree about the role of gesture incomprehension; whether it is ignored (Krauss, Dushay, Chenand Rauscher., 1995), processed separately from language(GoldinMeadow and Singer, 2003), used only when speakersare having difficulty (Rauscher, Krauss and Chen, 1996), orimmediately integrated with the content of the cooccurringspeech (McNeill, Cassell, and McCullough, 1994).One factor that may have contributed to the lack ofconsensus is that gesture researchers have tended to focus onanswering the question: is gesture communicative? (seeKendon, 1994 for a review). In doing so, they may haveconfounded two questions that could be consideredindependently: 1) do speakers intend to communicate usinggesture in natural interaction? and 2) do comprehendersbenefit from gesture when it naturally cooccurs with speech? For instance, some results suggesting a lack of intentionalityhave been taken as evidence that listeners’ do not use gestureduring comprehension (Krauss et al., 1994). In the current study, we focus only on comprehenders. Ourgoal is to investigate how and when comprehenders make useof naturally cooccurring gesture and speech (leaving asidefor the moment the question of whether speakers intend forgesture to be communicative). There is a growing body ofevidence that listeners integrate many forms of extra linguistic information when comprehending cooccurringspeech, including emb