Poor Decision Making in User Research

This article was first published on Medium on Aug 8, 2017

UXers must be aware of their own cognitive biases in order to make reasonable, evidence based judgments.

In UX, part of our daily job is to predict the users’ intentions and make decisions based on data and UX research results. But when it comes to judgment and decision making, we are worse than we might think. Tons of psychological research has shown that humans are quite lazy when it comes to weighing all relevant information, instead we apply easy rules of thumb that have worked for us in the past, and ignore many aspects of the problem we are trying to solve. And not only you as UX professionals are vulnerable for oversimplifying your decision making, also your users and test participants are easily tricked into poor judgments, both by their own brains and your question design.

We take mental shortcuts whenever we can.

The human brain is an efficiency beast. It has to be, since it is under constant fire from the outside. We permanently have to react to our environment. Starting from simply navigating through our world without permanently running into objects, to identifying and remembering faces and names, learning and categorising new information, putting ourselves into the position of others, interpreting numerical information, making judgments and decisions all the time — our environment is too complex and too demanding to start long brainstorming sessions for every decision we have to make per day. This is why whenever we can, we take mental shortcuts to come to quick answers. In psychology, we call these shortcuts “Heuristics” as simple rules of thumb that have worked for us in the past.

Let us look at a simple example to illustrate how these rules of thumb work:

We all know these optical illusions in which we have to judge whether the three people differ in size. The illusion is that they do not, but that is pretty hard to accept. The right person just looks larger. Why? Because he is further back in the “3D room”. And from real life, we know that size works in relation to distance. People look smaller the further away they get away from us. All three figures have exactly the same size, but since they differ is their distance from our point of view, we perceive the right person larger as the left person. We simply adapt rules from a well known environment (the real world) to a new scenario (this 2D illustration) which is in fact the most efficient way of solving this problem. But it leads to a wrong judgment.

There are tons of easy heuristics that help us make fast decisions in our everyday lives. And most of the time, these heuristics act unconsciously. We are not aware that we oversimplify a problem, that we reduce complexity, that our judgment is not rational. In UX, we have to make assumptions and estimates every day. And that is why we must be aware of our own cognitive laziness.

The same information can lead to different interpretations.

The Nielsen-Norman group asked more than 1000 UX professionals a hypothetical question: Imagine you are currently working on a search function for your product. Now a usability testing has shown that 4 out of 20 users have not found the search function. Should the search function be redesigned?
51% of the UX professionals answered that “Yes, it should be redesigned to make it easier to find” while 26% said “No. It’s fine as it is.” (23% were not sure).

Now, actually just half of these thousand participants were given the information that 4 out of 20 did not find the search function. The other half was told that 16 out of 20 users did find the search function — which, as you can see, is objectively the same outcome. Still, this group reacted significantly different to this statement: Only 39% said that the search function should be redesigned, and 35% said it is fine as it is (26% unsure).

This effect is called “Framing Bias” — the frame, the story in which you embed an information, influences the way it is perceived. In this case, once the results were framed in a positive statement (“found the search function”) and once it was framed in a negative statement (“did not find the search function”). This frame itself, independently from the actual number, can influence whether you think this message is good or bad, and whether you should do something about it. In the supermarket, “80% fat free” sounds better than “contains 20% fat”. Don’t forget: we take mental shortcuts whenever we can. Watch out for potential framing effects when reporting research results, be aware of the power of your words. Don’t push your audience into assumptions that might not be reasonable based on the data.

In situations of uncertainty, we look for helpful reference points

In order to make sense of certain information we look for reference points (“anchors”) related to the information. After a discount, the new price of a product ($1.29) does not have much power as an isolated information. But in relation to the old price ($1.89) it feels like a great deal.

Even if you leave the new price $1.29 as it is, but manipulate the old price, make it much higher or even lower than $1.29, the new price would perceived completely differently, once as a phenomenal offer and once as a rip off.

If you put your users into situations in which they have to make estimates (“how long have you used our app in the last 30 days”) that are hard to answer, watch out for anchor effects in your question design:

If you offer rating scales with two extremes, these will be perceived as “very low” and “very high” and if you see yourself as a normal person, you will put yourself somewhere into the center between these two extremes. Especially for questions which make it hard to give a realistic estimate, this strategy is helpful, but as you can see, very dangerous for you as researchers. You have the power to manipulate your results by asking your users very hard questions which they could not answer without the help of your predefined scales.

Also do not forgot what you want the user to do when asking them for estimates. Let us assume you ask them for their average food consumption in kcal per day — what you want them to do it walk through their memory and remember what they ate and drink and somehow come up with a reasonable answer. By offering a reference though, you shift the respondent’s focus towards a social comparison with their friends and colleagues. “Am I above or below average here?” will define their answer, not their actual consumption.

We overestimate representativeness of our memory.

Who is more likely to die on the job — A police officer or a wood chopper? How likely is it that your plane will crash or that you will be eaten by a shark? — When it comes to predicting events and estimating probability, we often rely on the first thing that comes to our mind. We all have read stories about police shootings, about plane crash and shark attacks. We have not often heard of loggers getting killed by a tree or a chainsaw. We don’t often read about large numbers of people dying in their cars. Also: The number of people dying in their everyday lives is not often covered in the news.

In fact, 7.3 out of 100,000 police officers die on the job, while 110.9 out of 100,000 wood choppers do. These numbers surprise you? Of course, because they do not reflect the kinds of information you have stored in your memory. Overestimating the representativeness of our memory is called “Availability Heuristic” and you should be aware of it when thinking about prior research results.

When someone asks you for advice based on research results from the past, do not rely on the few details you still remember from that testing. Do not invent anything because you spontaneously remember that one participant who gave a smart or funny answer or who had a hard time finding the solution. Instead, look back into your data and come up with an answer that is valid.

How we feel right now influences our judgments.

We have good days and bad days. And how we currently feel can influence our judgments — even if we are experts in what we do.

A study, released in 2010, looked at 1112 judicial rulings which were all related to prisoners’ requests. Prisoners can for example request to get a tracking device removed, and a judge must either rule in favour or against it. The requests were not ruled in any particular order.

In the study, the authors looked all the time within the day when the ruling occured. It obviously shouldn’t matter whether a ruling happened in the morning or in the evening. But it turns that current mood of the judges did have a large impact on the rulings. The authors state: “We find that the likelihood of a favorable ruling is greater at the very beginning of the work day or after a food break than later in the sequence of cases” (p.2).

When you are about to make an important decision, try to reflect your current mood and eventually postpone it to later. Also in an interview situation: Do not treat the last participant of the day differently than the first.

We can not kill heuristics — but we can become aware of them.

Heuristics work unconsciously. You will not be able to completely eliminate their influence on your daily judgments. The only thing you can do is become aware of them.

As the UX researcher at Onefootball Design I did a 30 minutes talk about this topic as part of the Design Lab Meetup in Berlin. You will hear many more examples there: Click here