Multimodal fusion of non-verbal cues to enhance the understanding of emotional and spatial uncertainties in navigational vocal commands for assistive robots [abstract]

Abstract

The world is experiencing a dramatically rising elderly population. As people age, they may become incapacitated to some degree and need assistance from caregivers in doing their daily activities. Since the gap between the supply and demand of caregivers is widening, it is of great significance to develop devices that can support the elderly in their day-to-day life. Human communication is a pillar in social organization. In domestic environments, humans prefer to interact with each other in multiple modalities. A discussion with two people could be in speech, hand gestures, head gestures and facial cues etc. Speech-hand gesture is a more common multimodal pairing identified in human-human communication. Human-Human verbal instructions have uncertainties. Spatial uncertainties such as direction related uncertainties, distance related uncertainties and contextual emotional uncertainties in these verbal commands can affect human-robot interaction. Therefore, this research focused on enhancing the understanding of spatial and context-emotional uncertainties in vocal navigational commands using hand gesture cues. Intentional hand gestures were used to interpret these uncertainties while unintentional hand gestures were filtered out using verbal-hand gesture timeline features and an ensemble model of LWNB and KNNDW was used. Further, spatial hand gestures such as direction and distance related uncertainties were interpreted using partial gesture distance and direction information extracted from gestures using a fuzzy logic approach. For the context-emotional uncertainties, vocal prosody (tonal modality) and visual prosody (hand gestures) were fused together using an ensemble deep learning approach to enhance the understanding of these uncertainties. These intelligent systems were successful in enhancing the understanding these uncertainties in order to increase the accuracy and human-like capabilities to understand vocal navigational commands.