Image comparison based mobile user interface verification framework

Abstract

Due to the highly competitive market, user interface of a mobile application plays a major role in attracting and retaining its user base. In a full stack or web application development, there is usually a user interface (UI)/ user experience (UX) designer or a front-end engineer who implements the front end. On the contrary, in mobile applications, the app developers themselves implement the front end according to the mockups provided by the UI/UX designers. The verification of the user interface of the actual application against the provided mockup happens manually by developers and testers and not by the designers which makes it less accurate and time consuming since their eyes are not trained to identify pixel level visual differences. Until now various researches have been done on automating the verification step of the event flow and underlying functionalities. But verifying the user interface of mobile applications is still left for the human eye. The main objective of this research is to develop a mechanism to get a quantifiable score based on how much the user interface of a mobile application matches with its initial mockups. Two accuracy levels are considered for computing this score; layout and overall. Layout score limits the comparison to the layout of the user interface whereas the overall score compares layout along with color, orientation, etc. For the layout level comparison, three local feature matching algorithms namely, SIFT (Scale-Invariant Feature-Transform), CSIFT (Color SIFT), PCA-SIFT (Principal Component Analysis SIFT) along with a simple bob detection matching algorithm are considered to be experimented with. For the overall level comparison, a pixel level comparison algorithm is used. In parallel a survey is conducted where UI/UX designers would provide a layout and overall score for a set of selected use cases. These scores were compared with the scores from the image comparison algorithms. Based on this, CSIFT is chosen as the underlying algorithm to compute layout comparison scores as it outputs the closest values mimicking designers. For the overall value the pixel based scores ended up being stricter than values given by the designers. In conclusion, the objective of the research is successfully achieved by implementing a framework which will output a score based on the comparison between the mockups and the actual user interface of mobile applications in two accuracy levels; overall and layout only. Overall score based on pixel level matching turned out to be too strict and better suited if the requirement is a strict conformity to the provided user interface. Layout score also has limitations with text intensive applications when the data is dynamically loaded. Both these scores can be used to verify the user interface, but the thresholds and which score to use is dependent on the application and how much deviation the company allows against the provided mockup.