Multimodal supported chatbot framework for people with aphasia

Abstract

Chatbots have become a trending topic with emerging platforms like ChatGPT, Gemini, and Copilot, for conversation assistance. Current chatbots mainly focus on the general public assuming a natural flow of conversation. However, there is a need for a chatbot that supports people with various communication disabilities. This research fills this gap by offering a novel technique for a chatbot that assists people with Aphasia, a condition characterised by difficulties with language. We propose a multi-modal chatbot that is customised and designed to assist users with communication disabilities in navigating awebsite. Unlike typical chatbots, which rely on one form of communication, our architecture combines multiple modalities to improve comprehension and promote effective communication for people with Aphasia. We focus on gathering multimodal inputs by recognising and combining user intents from diverse sources. The use of Txtai, an all-in-one embeddings database for semantic search improves our chatbot’s capacity to process various inputs efficiently. We leverage specialised models like Whisper for audio transcription and MediaPipe Gesture Recognizer for gesture detection to enhance user interactions. Additionally, Rasa Core integration improves conversational experiences for users. We propose that this new approach will make communication more accessible and inclusive for individuals with Aphasia.