Abstract:
Due to the highly competitive market, user interface of a mobile application plays a major
role in attracting and retaining its user base. In a full stack or web application development,
there is usually a user interface (UI)/ user experience (UX) designer or a front-end engineer
who implements the front end. On the contrary, in mobile applications, the app developers
themselves implement the front end according to the mockups provided by the UI/UX
designers. The verification of the user interface of the actual application against the provided
mockup happens manually by developers and testers and not by the designers which makes it
less accurate and time consuming since their eyes are not trained to identify pixel level
visual differences.
Until now various researches have been done on automating the verification step of the event
flow and underlying functionalities. But verifying the user interface of mobile applications is
still left for the human eye.
The main objective of this research is to develop a mechanism to get a quantifiable score
based on how much the user interface of a mobile application matches with its initial
mockups. Two accuracy levels are considered for computing this score; layout and overall.
Layout score limits the comparison to the layout of the user interface whereas the overall
score compares layout along with color, orientation, etc.
For the layout level comparison, three local feature matching algorithms namely, SIFT
(Scale-Invariant Feature-Transform), CSIFT (Color SIFT), PCA-SIFT (Principal Component
Analysis SIFT) along with a simple bob detection matching algorithm are considered to be
experimented with. For the overall level comparison, a pixel level comparison algorithm is
used.
In parallel a survey is conducted where UI/UX designers would provide a layout and overall
score for a set of selected use cases. These scores were compared with the scores from the
image comparison algorithms. Based on this, CSIFT is chosen as the underlying algorithm to
compute layout comparison scores as it outputs the closest values mimicking designers. For
the overall value the pixel based scores ended up being stricter than values given by the
designers.
In conclusion, the objective of the research is successfully achieved by implementing a
framework which will output a score based on the comparison between the mockups and the
actual user interface of mobile applications in two accuracy levels; overall and layout only.
Overall score based on pixel level matching turned out to be too strict and better suited if the
requirement is a strict conformity to the provided user interface. Layout score also has
limitations with text intensive applications when the data is dynamically loaded. Both these
scores can be used to verify the user interface, but the thresholds and which score to use is
dependent on the application and how much deviation the company allows against the
provided mockup.
