Evaluation of the paper prototype
As you have read in my previous blog posts, I made a (hopefully) better paper prototype to test on users. Here, I will evaluate the results of these tests.
The paper prototype I made now, is mostly in black and white. The reason for this is that colors might distract the user. The prototype is kept simple and quite minimal, but the lay-out is based on that of Foursquare. The NoiseApp (the name will be changed in the future) is actually quite like Foursquare, with the added feature to record sound.
Since the paper prototype cannot support all features of the application, I just wanted to evaluate the clarity of buttons to push and the application flow. These are merely the most important issues that can be evaluated with a paper prototype.
I tested 8 people. Three of them were computer scientists, two of them engineers and three of them had a non-technological background. Also one test user didn’t own a smart phone. This kind of population seemed balanced and enough to highlight all the problems of the application. The problems I discuss here happened to most of the test users.
I made a cardboard casing that resembles a modern Android phone. The back, home and menu button of a typical Android phone are also on there, although it is not necessary to know their function to use the application. The paper screens resemble the screens of the application. Navigating to the screens was done by myself (I was the ‘computer’ behind the application). I made it easy to swap from screen to screen by taping the screens together, so I could just pull at the top to move the screen (see the image).
The test user had to sign a consent form. The consent form made clear what my thesis is about, what I was doing with the user test and with a signature that granted me video recording the session. While performing the test, only hands (and the prototype) were video recorded to ensure anonymity. This way, I could concentrate on operating the prototype.
The test user had to perform three scenarios: a random record, a sound battle and checking the profile. In the full application, it would be possible to do more than three scenarios. I didn’t want to let users take more than 15 minutes, so I chose only three that were the most interesting.
The first two scenarios included the “recording of sound”. I asked the test users to think out loud, so I could understand what they were thinking. It’s a common tool for testing prototypes. I also didn’t give the test users any information about the application beforehand, but only the necessary info (which is the goal of the application). This way I could see whether the application was easy (intuitive) to use.
I also asked the test users to fill in a questionnaire afterwards. The questionnaire was set up so that I could gather data about whether the user had any experience with apps and whether the user found the application easy to use, functional and fun. A part of the questionnaire was based on the SUS (System Usability Scale) questionnaire. This questionnaire is composed of 10 statements on which the user can put a score of 1 (strongly disagree) to 5 (strongly agree) (this is the Likert scale). (see http://www.measuringusability.com/sus.php for more information) The usability can then be calculated to a score of 0 to 100.
The Random Record scenario
1. Main screen
The random record scenario was the most important one, since the recording of sound is the main thing the application can do. Other (not tested) scenarios include the random record scenario too.
When evaluating this scenario, I told the test user this little story: “You are walking outside and you are bored. You have just installed your new NoiseApp application. Now you want to record sound, right where you are. How are you going to do this?”
I then showed them the main screen. Most of the test user knew that they had to push the random record button to get to the next screen.
Most comments were about the record button in the second screen. It was not clear that this was a button, and that it also presented the location where the user was. A better way to do this, might be to separate the location marker (just like in Google Maps) and the record button at the bottom with a microphone.
It was clear to most of the test users to wait at the third screen to pass. Although, one told me that the cancel button seemed more like an error that happened. I might just put the word ‘Cancel’ instead of the button.
The third screen was obvious for everyone.
The Sound Battle scenario
1. Main screen
2.2. Choose player
2.4. At Battle Location
The scenario of the Sound Battle was the most difficult to prototype. On screens 2.3, 2.4 and 2.5, I pasted three Post-Its on each screen to mark the sound locations.
The test users had to start from the last screen of the previous scenario. I told them next story: “So, that were easy points! Although, random sound recording doesn’t gain a lot of points. You can earn more points with competing against someone. I would like you to play a game against a random player.”
All users knew they had to go back to the main menu, but there were different options. By pressing the Android back button or the application back button (arrow at the top), they went to screen 1.2 and then 1. You could also press the house at the top or the application name, but not many people found this option. I might highlight the house more when the application is fully in color.
When they pressed the sound battle button, they got to see screen 2.2. When reminding the test users that they had to play against a random player, they pressed the right button and screen 2.3 appeared.
As said earlier, three sound battle locations were taped on this screen with Post-Its. Altough, most test users didn’t think this was clear. Most of them thought that the post-its resembled other players. This was probably reinforced by using post-its and not icons to represent the locations. I might use the location picker image of Google Maps to make sure that people know it’s a location. Others said that it would be easier if some information was given at the beginning of the game. Because there was no dot in the middle to resemble the players location, many of the test users didn’t know what to do. I will add that in a next prototype. When the test user said that they’d move to one of the three locations, screen 2.4 appeared.
Screens 2.4 and 2.5 had the same comments as in the random record scenario. After recording at the first location, screen 2.3 appeared again, with the post-it of the recorded location marked, so it was clear that location was done. After recording at the three location, screen 2.6 appeared.
The points screen was clear, except most of the test users wanted to know what sound quality and location quality actually means. I might add an information button next to the term that explains it more. Also, ‘location quality’ will be changed into ‘location accuracy’, since the closer the test user records to the given location, the more points. To make sure that users know that location accuracy is important, I might change the color of the location marker. Further away from the location, the marker will be red, but close to the location, the marker turns green. Another criterium might be added to the points that could be earned later. (e.g. is you are the fastest, you gain more points) When the test user pressed ‘Ok’, the badge screen was shown.
The badge screen was clear to everyone.
The Profile scenario
This scenario was started with next story: “Now you want to check how many points you’ve earned.”
Most test users thought they had to go back to the main screen. There they saw that there was no option (of the four main buttons) to check the score. When they looked further, they saw the user icon at the top right. It seems that the top buttons don’t seem that clear to most users. Maybe I might add a button on the main screen (like the four game options) to check the profile. Also, it might have no use checking your profile during your recording.
When they found the user icon, everyone knew they had to press the stats button to show the points earned.
Then I asked the test user to show the badges he/she had earned. Most people knew they had to go back, to press the badges button. I had the comment though, that it might be easier just to present the badges on the same screen as the points. This might be a good idea.
When I asked the test user to see how he/she does in the leaderboard, 7 out of the 8 test users pressed the globe button at the top right. This was unfortunately not the right choice. The globe would present the map with all sound measurements. When I asked them what they’d do if I told them that they can see themselves in the leaderboard between their friends, most of them found their way by pressing the tab button ‘Friends’. The problem with this button was that is was not clearly prototyped. In the digital prototype, it will be more clear that it is a tab.
Since every test user had to fill in the SUS questionnaire, I could generate a general score that expresses the usability of the application. 68% is an average score for the SUS questionnaire.
As you can see in the boxplot, the application scored well on the usability scale. The median is 81.25%, which is way above average. Also, the mean is quite high with 76.25 %. When neglecting the outlier at the bottom, the average rises to 81.07%. Neglecting the outlier makes sense, since the test user had no experience with smart phones or applications whatsoever.
Some extra remarks
- 50% of the test users had an Android phone;
- Average age of the users was 22 y.o.;
- Users that have a smart phone said in general that they could handle smart phones quite well (scored 3.43 on the Likert scale);
- Some users found the application useful.
Since the comments on the prototype were rather small, I will not do a second paper prototype. I will prepare a digital prototype starting from now on. The digital prototype should respond to the comments made. I will already implement the prototype with the Android SDK, to get familiar with the programming language.
Because of time constraints, I will implement the digital prototype iteratively. I will first implement the random record, evaluate that, and then implement another scenario, until I have tested everything.
If you are interested in doing some test on the digital prototype, you can always comment or send a message!