Week 13: Beta Testing

“If you always do what you always do, you will always get what you always get”

Voltaire, Francois-Marie Arouet, French Writer and Philosopher 


For the final beta test, Botify took to the streets once again for some ‘in the wild’ testing.  We tested a total of six users:


1) To assess whether users understand how to navigate through the bundle builder

2) To identify usability issues and navigation errors

3) To understand customers’ views towards the experience


We found a wide degree of usability, linguistic and visual design issues in the final prototype, as listed below.

Usability issues:

  1. Progress bar implementation is an immediate concern. Many users requested to be able to use it, and saw that as a critical feature for them to modify their choices.
  2. Users complained about not being able to see the ‘choose theme packs instead’ option very easily. It seems like the links for the secondary interactions need to be bigger, and were not seen as intuitive.
  3. Not many users found the ‘Search’ field easily. Search function on the channels screen may be more easily found on the main section of the screen, maybe on the right side above channels.
  4. Users did not see or click ‘Show details’ on the footer. One user noted that they would be more comfortable with a cart icon, a convention that they are already used to. Solution listed below:


Visual Design issues:

  1. The drop down for uncollapsing comparisons for internet plans is not clear and intuitive. Not a single user found it. The iconography may need to be changed.
  2. There are two primary CTA’s on the Essentials page – users saw ‘Continue’ instead of ‘Customize channels’       
  3. The secondary CTA on Essentials page is too small, and users did not notice that.
  4. ‘Deals and promotions’ area in the footer section wasn’t attention grabbing. Solution below.


Language issues:

  1. We might need some text on top of the sliders to indicate that they’re meant to be played with. Solution below.
  2. Essentials text needs to be changed to indicate that it is not customizable.

A few quotes from users:

“It’s nice, but some of the things are not that intuitive. Like the compare thing. if you make progress bar clickable, that would be more helpful.”

“This is far and away a better product because it is empowering me and creating a greater level of service”.

“Quicker than my past experience. Goes straight to the bundle I want, and check out, very easy, not complicated.”

All the users we tested seemed welcoming to the overall concept and experience provided by the final prototype of the bundle builder. The issues that they had are mainly regarding usability, language and visual affordances.

A suggested next step for any design team working on this version of the bundle builder would be to extensively test each portion in isolation and refine the details of the interface.


The ‘in the wild’ testing was very fruitful, so much so, we had to do it again. We owed it to ourselves to see how real people would react to our product. One main takeaway from such testing is: one street test is worth ten MDM (in-house) tests, so C12 take note! Their is an irreplaceable richness to the raw reactions of real target customers. Twas fun!



Leave a comment