Automated user review analysis to facilitate potential mobile application evolution

Abstract

User reviews are crucial for mobile application evolution, but manually analyzing large volumes of feedback is time-consuming and inefficient. This thesis explores advanced techniques for automated user review analysis to facilitate potential mobile application evolution. We introduce a novel Convolutional Neural Network (CNN) based approach for Aspect-Based Sentiment Analysis (ABSA) on app reviews. Our model demonstrates significant improvements over existing baselines, with F1 scores of 0.62, 0.42, and 0.62 for aspect category classification in Productivity, Game, and Social Networking domains respectively. For aspect sentiment classification, we achieve accuracy scores of 0.80, 0.70, and 0.86 in the same domains. We provide empirical evidence on hyperparameter tuning, investigating the effects of batch size, number of epochs, and learning rate on model performance. To enhance our ABSA model, we investigate various word embeddings and data augmentation techniques. We find that Word2Vec embeddings and Round-trip Translation (RTT) augmentation yield the best results, offering insights for future research in this domain. Expanding our exploration of automated review analysis, we evaluate the potential of Large Language Models (LLMs). We provide a comprehensive comparison of state-of-the-art commercial and open-source LLMs in zero-shot and fine-tuned settings. Notably, we demonstrate the feasibility of using commercial LLMs as autonomous annotators, creating a high-quality dataset of 10,000 app reviews while achieving an accuracy of 81.89%. Our research also investigates the impact of various parameters on LLM performance for app review analysis, including training data size, number of epochs, Temperature , and Top_p. We find that fine-tuned open-source models can achieve performance comparable to commercial LLMs, with our best model reaching an F1 score of 0.83416. Our work contributes to the field of mobile application development by advancing automated user review analysis techniques to potentially improve the process of mobile application evolution. This work has the potential to lead to faster and more targeted mobile application improvements, enhancing developers’ ability to respond to user needs effectively..