Week 10: Guerrilla Testing


“However beautiful the strategy, you should occasionally look at the results.”

Winston Churchill, UK Prime Minister, 1940-1945, 1951-1955

After conducting rounds of testing mostly on site at the CDM, Botify took to the streets to find eligible bundle builder customers to gain better insight. The objective was to conduct an observational usability test, with some follow up questions afterwards.

Location:

Nestor’s Market, Vancouver Public Library

Tools:

An InVision prototype with clickable hotspots.

The InVision prototype allowed us to test an on-screen experience as opposed to paper, however, there were limitations. Asking users to select their channels was difficult seeing that the prototype is not fully developed, so only certain elements were interactive. This skewed the test in a way, since customers couldn’t feel the full potential of the experience.

Nonetheless, there was some valuable takeaways from the users interviewed.

Interviewees

Botify interviewed 9 people between 24 and 45. They ranged from our primary target (middle-aged, multi-services customers with families) to younger subjects who are in the market for internet/tv. 5 males and 4 females were tested.

Documentation Method

Since usability was a focus, the team conducted the test in the following format:

  1. One team member recruited suitable users and welcomed them to the testing area, an introduction was made and users were encouraged to experiment (some constraints were mentioned)
  2. Another team member documented results using printouts of the prototype on paper. Using mini post-its, the results were documented in the following way:

Blue: Language/information issues
Neon: Usability issues
Pink: Negative emotions
Orange: Positive emotions

With each user test, we repeated the stickies to create a heat mapped response to each screen combining issues of language, usability, pleasure and displeasure.

Here is a sample page:

Key Takeaways

A key focus was to understand what doesn’t work:

  • Users were visibly confused by the InVision prototype and felt that they could not easily gauge the product in the current format.
  • Users were confused by the fact that they needed to choose their channels, and then make another similar choice in the theme pack selection screen. They still had a problem discerning between bundles, theme packs and individual channels.
  • Some users were not intuitively drawn towards the sliders nor the panel on the left which instructed them to access the sliders.
  • 5 out of the 9 users that we tested felt like the process was clear and intuitive.
  • The other 4 felt that some parts were confusing.
  • Users visibly liked the idea of an a-la-carte process.
  • Users over 40 preferred a simpler process with less options (more web 1.0-esque), younger users enjoyed the options and real-time interactivity
  • There was a lot of animosity towards theme-packs, essentials and any other industry standard that pushes customers to fixed packages. This hate was real…but…there was a lot of love for an a la carte design, something that brightened the spirits of all users.

So what’s next

With 3 weeks to go, a key next step is to get a coded prototype out in the wild and to test it repeatedly, making adjustments as needed and documenting major next steps beyond week 13. It’s crunch time but it’s also fun time, as we see the fruits of our labour start to take formation. What will be pivotal in the remaining weeks is to prioritise and to shift momentum quickly as needed, solving the key issues first. #crunchtime.

Leave a comment