Attention: This is a super important step.
In the previous step (Step 3 – Test) you carefully observed the users’ behaviour and took all the notes possible, and maybe even voice/video recordings. Now, you probably think it’s time to make all the changes to address the comments or suggestions you’ve heard. Not so fast. First, you want to be cautious about what users have said. Relying just on their words, thoughts, or suggestions puts you in a risky position. Users’ behaviour is way more valuable and a much better indicator of what’s working and what’s not working.
There is a great story proving the point. A few years ago, at one of the Walmart’s stores management surveyed their customers for ideas on how to improve their shopping experience. Almost everybody asked for less cluttered aisles and shelves. They said there were too many products on each shelf, too many shelves, and too narrow aisles. The management delivered exactly what their customers had asked for. And I admire their devotion and desire to listen to the most important group of people. However, the reality hit them hard when the sales plummeted after they updated the store’s layout. Shortly after, they reverted the changes, and the sales bounce back to the previous levels. They learned their lesson.
During your testing, you may hear many suggestions on how to improve your product or feature. If your idea resonates with them, they will likely get passionate sharing their thoughts and trying to help you with your new product. It may start with “You know what else would be cool?…” or something like that. And you definitely should make a note of all such ideas because there might be something new and valuable. But don’t jump into making the changes right away. Try to analyze their behaviour instead, and understand the underlying reason why they are asking for this. Use 5-Whys method to dig deeper and get to the core. And after you understand their motivation you can think about the optimal way to address their core need. Sometimes, it can be exactly what they told you about, but most often there is a better way to help them achieve what they are trying to.
Sometimes, your thorough thinking leads to the exact solution that was asked for by your users. That’s totally fine and you may conclude you should have implemented their ask right away. Reality shows that it may be hit or miss, and if you skip analysis phase, the risk of making a mistake is much higher.
Also, don’t forget that you are running a business, and you want to strike the right balance between customer and business needs. So, if you product update cause a negative impact on your bottom line (like with Walmart’s example) you should be ready to act. Such things happen and that’s absolutely fine. The key to success is to test often and iterate quickly. If you are doing an experiment, you may be able to minimize your risks by rolling out your new feature to a limited number of users. Then, observe their behaviour and analytics, and iterate if necessary before launching it to everybody. One of the methods we use for that is A/B testing, and I will be covering this topic in future.
Got a bit sidetracked… Back to analyzing test results =)
Every person has some sort of biases. Usually, you may not even notice this. It could be as simple as an assumption that everybody uses their mobile device with one hand because you do it this way. Humans (including designers, we are also humans, you know =) tend to think their own behaviour applies to everybody or at least majority of their users. This is often incorrect and that’s why you have to get rid of your biases before analyzing the results. One of my favourite quotes – “Design for them, not you” – is a good reminder of this problem. I even have a sticker on my notebook =)
After you identified all the valid problems with your prototype, time to arrange the list, so you can tackle the most important ones. In an ideal world you want to fix every single thing, but in our flawed world with limited time and resources, you want to be strategic about what you spend your precious time on. So, you want to go through a prioritization exercise to group the discovered issues depending on their severity. One of the big questions I ask when prioritizing is “Does it stop users from using your app?” If yes, this is a critical issue that needs to get fixed before the next round of testing (yes, you heard me, it’s an ongoing process and should never stop). It’s hard to give you any solid advice on what issues are a higher priority without seeing the product itself =) So, I can suggest that you play it by ear and you should be able to estimate the severity based on your gut feeling. Also, you can ask me anytime, and I’ll be glad to help you out, free advice just for you =)
Deciding on the list of issues to fix is very important and requires some thinking. But how you are going to fix them will require a bit more creative thinking, and will depend on your circumstances (e.g. type of the prototype, level of fidelity, scope, etc.). In some cases just making a button bigger may solve the issue that users didn’t see it. In other cases, you may have to change the application map to make it better aligned with your users’ mental model. If you have done it right, and have got a good understanding of how your target audience lives and breathes, and what their true needs are, then the overall structure/logic of your product should not need a complete overhaul. But if your testing revealed that they were very confused about what it did, and where to find stuff they were looking for, this requires more time and effort, for sure. But you can do it! I believe in you! =)
A few “red flags” examples that your product needs more work:
– Users found something confusing and didn’t know what to expect from a menu or a button.
– They weren’t sure where to find a piece of information or some functionality.
– They paused looking at the screen.
– They said “interesting”, “oh”, “um”, or any other sign of hesitation.
– They asked where they could find this or that.