Explainable artificial intelligence for speech emotion recognition
Loading...
Date
2025
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
The rapid advancement of Artificial Intelligence (AI) has led to its widespread adoption across multiple domains; however, the inherent opacity of conventional AI models has raised concerns regarding their interpretability and trustworthiness. In the context of Speech Emotion Recognition (SER), this lack of transparency limits understanding of how specific acoustic features influence emotional predictions. This research addresses this challenge by developing an explainable and human-centered framework for SER that prioritizes interpretability over classification accuracy.
A comprehensive feature extraction pipeline was designed to derive eighteen interpretable audio features from raw speech signals, grouped into five functional categories: Acoustic, Temporal, Spectral, Harmonic/Tonal, and Spectral Energy Distribution. Traditional machine learning models Artificial Neural Network (ANN), Support Vector Machine (SVM), Random Forest, XGBoost, and k-Nearest Neighbors (k-NN) were employed to classify seven emotional states: Calm, Happy, Sad, Angry, Fearful, Surprised, and Disgust. Two model-agnostic interpretability techniques, SHapley Additive explanations (SHAP) and Local Interpretable Model-Agnostic Explanations (LIME), were used to quantify the contribution of each feature to model predictions.
A novel Aggregated Feature Importance (AFI) metric was introduced to combine interpretability outcomes by weighting each model’s contribution according to its predictive performance. The analysis revealed noticeable divergence in feature importance rankings when aggregating across individual models, reflecting their inherent learning differences. However, as additional models were incorporated into the ensemble, these variations gradually diminished, showing a clear convergence and increased stability in AFI-based rankings. Key features such as mfcc_mean, mfcc_std, loudness, zero-crossing rate, and etc. consistently emerged as dominant predictors across emotional classes. Statistical validation using independent t-tests confirmed significant differences (p < 0.05) between emotional states, reinforcing the discriminative reliability of these features.
The resulting explainable feature–emotion map provides a transparent interpretation of how specific speech characteristics relate to distinct emotional states, supporting greater trust and usability in emotion-aware AI systems. This work establishes a reproducible framework for feature-level explainability in SER and offers a foundation for future research integrating reverse-engineered and temporally adaptive explainable models.
Description
Keywords
EXPLAINABLE ARTIFICIAL INTELLIGENCE, ARTIFICIAL INTELLIGENCE-Speech Emotion Recognition, MACHINE LEARNING, MACHINE LEARNING-SHapley Additive Explanations (SHAP), EXPLAINABLE ARTIFICIAL INTELLIGENCE-Local Interpretable Model-Agnostic Explanations (LIME), DATA SCIENCE AND ARTIFICIAL INTELLIGENCE-Dissertations, COMPUTER SCIENCE AND ENGINEERING-Dissertations, MSc in Data Science and Artificial Intelligence
Citation
Patabendige, S.S.J, (2025). Explainable artificial intelligence for speech emotion recognition [Master's theses, University of Moratuwa]. Institutional Repository University of Moratuwa. https://dl.lib.uom.lk/handle/123/25909
