Classification of mobile application reviews using deep language models

Abstract

User reviews include valuable information for mobile applications such as bug reports, feature requests, and rationale for praising or criticising about the application. Manual analysis of the reviews is costly due to the vast number of reviews received for an application. To reduce this manual effort, the literature mainly focuses on shallow machine learning methods with few studies investigating the deep language models to assign labels to the reviews. This thesis i. defines a new label to distinguish reviews criticising the quality and business strategy of applications, ii. presents a new manually annotated dataset of application reviews of size 2230, and iii. studies the performance of BERT, RoBERTa, DeBERTa, GPT-3 (ada), and GPT-3 (curie) models for review classification. Our results indicate that GPT-3 (curie) significantly outperforms the BERT yet there is no significant difference among the rest considering the F1-score. Additionally, we extend our pipeline by performing topic extraction to identify and capture common themes and topics from the reviews resulting from the classification pipeline. This additional step allows us to gain deeper insights into the prevalent subjects and discussions within the user feedback.