When words fail: Collaborative gestures during clarification dialogues

Research on co-speech gestures has primarily focussed on speakers. However, in conversation non-speaking addressees can also gesture. Often this is to provide concurrent feedback such as backchannels but sometimes it involves gestures that relate to the specific content of the speaker's turn. We hypothesise that non-speakers should contribute most actively during clarification sequences i.e. at the moments when mutual-understanding is threatened. We test this hypothesis using a corpus of story-telling dialogues, captured using video and motion capture. The results show that during clarification sequences speaker and non-speaker behaviours tend to merge. Non-speakers in particular move their hands faster and produce more than twice as many content-specific specific gestures in overlap with speaker turns. These results underline the collaborative nature of conversation, the strategic importance of non-verbal resources for sustaining mutual-understanding and the critical role of clarification and repair in successful communication.
Research areas:
Type of Publication:
In Proceedings
Book title:
AAAI Spring Symposium Series: Turn-Taking and Coordination in Human-Machine Interaction
Hits: 2414