Context aware trilingual conversational service robot for restaurant

Abstract

Conversational AI enables seamless humancomputer interactions across domains like customer service and healthcare. However, current systems face challenges with low-resource languages, real-time latency, and edge-device performance due to computational limitations. Current service robots commonly use obstacle avoidance only, missing auditory context in dynamic environments. This paper presents a domainspecific conversational AI system for restaurant service robots in Sri Lanka, supporting trilingual (English, Sinhala, Tamil) interactions with context-aware order-taking while incorporating voice direction following for enhanced navigation. Our system leverages fine-tuned Whisper Small for automatic speech recognition (ASR) and language identification. For accuracy, we employ a Retrieval-Augmented Generation (RAG) architecture, combining FAISS for efficient retrieval and a fine-tuned LLaMA 3.2 (1B parameters) for generation. Standalone fine-tuning with custom data showed limited domain-query performance, while RAG improved accuracy without compromising edge-device feasibility. Speech output uses SpeechT5-optimized text-tospeech (TTS) models for Tamil and Sinhala. Our system enables scalable multilingual restaurant automation with voice-following. Future work includes edge optimization, TTS refinement, and multimodal enhancements for a better user experience.