Course Project: Evaluation Methods
Google Maps
vs
SL
Analyzing Efficiency, Purposefulness & Satisfaction
Usability Assessment: Comparing SL Journey Planner with Google Maps
In this study, we compare two major systems, the SL Journey Planner and Google Maps, through a comparative analysis to assess their performance in terms of efficiency, usability, and satisfaction, which collectively determine the overall usability of the systems. This study was conducted to examine how two frequently used systems perform in usability compared to each other. To measure usability, tests are conducted based on time taken, number of clicks and completion of tasks according to a test schedule designed to evaluate the applications' core functionalities.
Study Participants
The study enlisted participants to evaluate both novice and experienced users of two distinct journey planners and preferences in operative system (iOS vs Android). We specifically targeted young adults to ensure that participants' technical proficiency and familiarity with mobile devices had minimal influence on the evaluation. Our focus was on assessing the performance of the systems rather than the participants' individual abilities.
Team Credits
Victor Magnusson - Observer
Filip Norgren - User Test Facilitator, Observer
Aris Emilsson - User Test Facilitator, User Test Organizer
Choosing the Right Approach
We utilized the Comparison Method to evaluate features with similar purposes across two systems. The approach was chosen because the both systems tested are highly popular and offer features that serve the same objectives. However, some users prefer one over the other. Therefore, it is interesting to simultaneously analyze how the systems fulfill their roles as journey planners and then compare their performance.
Test Execution
Background Questionarie
User Testing
System Usability Scale Questionarie
Collected data
Task Accuracy
Percentage of tasks completed includes all participants who successfully completed the task, regardless of whether they needed assistance from the moderator
●
Percentage of Tasks Completed Independently: This category encompasses only those participants who completed the task without any assistance
Task Timings
Mean Time to Complete: This provides a general indication of the the average overall performance of the group
●
Median Time to Complete: Represents the exact midpoint of all participants' completion times
●
Range of Completion Times (High and Low): Highest and lowest completion times for each task
A summary of the results & data analysis
Exploring Efficiency: SL vs Google Maps (GM)
In conclusion, SL significantly outperformed GM across all usability metrics. Both systems demonstrated good usability, with SL achieving a high task completion rate of 98.6% and GM achieving a slightly lower but still commendable rate of 91.4%. SL exhibited superior efficiency, requiring 25% fewer clicks (454) and 31.6% less time (1496s) than GM. The final SUS scores, indicative of user satisfaction, were notably high for SL with a score of 87.25 while GM received a score of 65.7.
Comparing iOS & Android Performance: 19% Slower, 54.7% More Clicks
In the analysis of user groups, both Android and IOS users performed similarly with SL. However, IOS users seemed to face slightly more challenges with GM compared to Android users. IOS users were 19% slower in completing tasks in GM, required 54.7% more clicks, and experienced more failed sessions during GM testing. The reason for this discrepancy is not entirely clear, but one speculation is that SL's minimalist design may be more conducive to IOS users, whereas GM's feature-rich interface may pose challenges for IOS users accustomed to a more minimalist design typical of Apple products.