Course Project: Evaluation Methods

Google Maps

vs

SL

Analyzing Efficiency, Purposefulness & Satisfaction

Usability Assessment: Comparing SL Journey Planner with Google Maps

In this study, we compare two major systems, the SL Journey Planner and Google Maps, through a comparative analysis to assess their performance in terms of efficiency, usability, and satisfaction, which collectively determine the overall usability of the systems. This study was conducted to examine how two frequently used systems perform in usability compared to each other. To measure usability, tests are conducted based on time taken, number of clicks and completion of tasks according to a test schedule designed to evaluate the applications' core functionalities.

Study Participants

The study enlisted participants to evaluate both novice and experienced users of two distinct journey planners and preferences in operative system (iOS vs Android). We specifically targeted young adults to ensure that participants' technical proficiency and familiarity with mobile devices had minimal influence on the evaluation. Our focus was on assessing the performance of the systems rather than the participants' individual abilities.

Team Credits

Victor Magnusson - Observer

Filip Norgren - User Test Facilitator, Observer

Aris Emilsson - User Test Facilitator, User Test Organizer


Choosing the Right Approach

We utilized the Comparison Method to evaluate features with similar purposes across two systems. The approach was chosen because the both systems tested are highly popular and offer features that serve the same objectives. However, some users prefer one over the other. Therefore, it is interesting to simultaneously analyze how the systems fulfill their roles as journey planners and then compare their performance.

Test Execution

Background Questionarie


User Testing


System Usability Scale Questionarie

Collected data

Task Accuracy

Percentage of tasks completed includes all participants who successfully completed the task, regardless of whether they needed assistance from the moderator

Percentage of Tasks Completed Independently: This category encompasses only those participants who completed the task without any assistance

Task Timings

Mean Time to Complete: This provides a general indication of the the average overall performance of the group

Median Time to Complete: Represents the exact midpoint of all participants' completion times

Range of Completion Times (High and Low): Highest and lowest completion times for each task


A summary of the results & data analysis

Exploring Efficiency: SL vs Google Maps (GM)

In conclusion, SL significantly outperformed GM across all usability metrics. Both systems demonstrated good usability, with SL achieving a high task completion rate of 98.6% and GM achieving a slightly lower but still commendable rate of 91.4%. SL exhibited superior efficiency, requiring 25% fewer clicks (454) and 31.6% less time (1496s) than GM. The final SUS scores, indicative of user satisfaction, were notably high for SL with a score of 87.25 while GM received a score of 65.7.

Comparing iOS & Android Performance: 19% Slower, 54.7% More Clicks

In the analysis of user groups, both Android and IOS users performed similarly with SL. However, IOS users seemed to face slightly more challenges with GM compared to Android users. IOS users were 19% slower in completing tasks in GM, required 54.7% more clicks, and experienced more failed sessions during GM testing. The reason for this discrepancy is not entirely clear, but one speculation is that SL's minimalist design may be more conducive to IOS users, whereas GM's feature-rich interface may pose challenges for IOS users accustomed to a more minimalist design typical of Apple products.

KEY TAKEAWAYs

Finding the Right Approach: Tailoring methods to suit the context and objectives of the study for effective usability testing

Planning and Executing User Tests: Thorough planning and meticulous execution of user tests ensure reliable and insightful data collection

Analyzing Data and Drawing Conclusions: Utilizing data-driven analysis techniques enables the generation of meaningful insights and actionable conclusions

Previous
Previous

Exit Tickets