There is a saying in our business that if you are not talking to users, you are not doing UX, and I strongly agree with this statement. How else are we going to keep our assumptions about target groups in check? Communicating with users however, can be far from unproblematic, but luckily for us, there is another discipline who also does a lot of research on other human beings, and even though their goals can be different from ours, we can still learn a lot from their mistakes. I am off course thinking of social psychology. These guys have done so many studies that they know a thing or two about how non-verbal communication and peoples mindsets can affect a result, and they even have names for it.
The Screw You Effect
I’m kicking off with this effect, not just because it has the funniest name, but also because although it is relevant, it’s also quite rare. The screw you effect describes how a benevolent respondent can turn into a saboteur if they for some reason decide they don’t like the test leader, or if they would feel like they are being lied to. The being lied to part is probably more common for the social scientist, but it goes to show just how important it is for us to be honest to our respondents on why we have invited them and what we expect of them. Especially since these respondents will do anything to sabotage a session. Once in my early days I had a middle aged man in for a usability test, and I could not for the life of me get him to give me any good answers. His answers were short and un-informative and he was clearly signaling that he was not going to cooperate. I asked myself several times during that session why on earth a person like that would come to a usability test in the first place and I soon realized I wasn’t going to get any decent answers out of him. It was extremely frustrating, and I still don’t know what I did wrong. Maybe I was too young for him to take me seriously.
The Good Subject Effect
Far more common than the screw you effect is the good subject effect. That is when a respondent has made up their mind to do what it takes to make you happy and that means trying to read you, so they can tell you what they think you want to hear. This has nothing to do with deliberate sabotage, they just really, really want to help. At the end of a session it is not unusual to hear them say things like “I don’t know if I’ve been very useful useful, though…” They need some kind of confirmation that they have been useful to you. To negate this effect, it can be helpful to explain to your respondents that they cannot fail in this kind of test and the answers they give are the ones you want to hear, what ever they are. And as always, avoid leading questions. Because this effect is so common, it might also be good to to a little pre-test before the real test, to make the respondents comfortable in speaking out loud and giving both positive and negative feedback. And remember to give them praise; if they say something that you think might be useful for you, acknowledge that! I didn’t when I first started doing my interviews because I wanted to be neutral, but when I did start praising them I could literally feel how my respondents relaxed. It made them feel comfortable and more confident.
Helping the subject
To err is human and it’s only natural that these things happen sometimes, but my take away with this is that we as test leaders are also human. And just as there are examples of how the respondents’ behavior can influence a session, so are there examples of test leaders’ behavior doing the same. The most obvious, I think, is that people don’t always say what they mean. Not out of spite, and not because they don’t have an opinion, but sometimes words don’t come as easy as we would like. Therefore, we as test leaders must learn patience. If a respondent is looking for the right word, try not to help them find it. Consider that they are in an unfamiliar environment and maybe a bit nervous and give them that extra space to help them express themselves. I am currently struggling with this myself and have to bite my tongue more than once during the time of a session; I just want to help! But the word I choose, might not be the word a respondent would choose. Consider how close in meaning the words “hard” and “demanding” are. In some cases they could probably replace one another, but they are still different. Helping your subjects finishing their sentences might be as bad as asking leading questions.
The Street Light Effect
An officer is walking down the street one late night and spots a man further down. The man, who obviously had a beer too many, was looking for something under a streetlight. As the officer came up to him, he asked the man if he could be of assistance. The man lit up and said he was looking for his keys. ‘Is this where you lost them?’ the officer asked, looking down at the empty street. ‘No, I lost them over there’ said the man and pointed to a dark parking lot. ‘But why aren’t you looking over there, then?’ asked the officer, and the man replied ‘but over there I can’t see anything!’
Sometimes the expectation to find a particular answer can obscure the results you are actually getting. Other times some results can be considered less important simply because they don’t fall directly under the light of your focus. The problem with this kind of focus is that a hypotheses is being confirmed or denied, which means that there is an expectation on the outcome. It can seem harmless, but it does affect us as test leaders.
Intons-Peterson illustrated this during the 80’s when she made up a study based on the classic experiment of mentally rotating an object. Subjects were divided into two groups and both groups were given the assignment of solving the problem both with visual help and without. Test leaders were to measure which of the tasks was the quickest to be solved. However – the real study was to also divide the test leaders into two groups and assigning one group of subjects to each group of test leaders. One group of test leaders was told that expectations were that the subjects would be quicker to solve the task with visual help. The other group of test leaders were told that subjects would probably be quicker without the help. Bogus reasons were presented to support these hypotheses. Off course the group of test leaders did not know about each other and were also told not to convey the hypotheses to the subjects in any way. Both groups of test leaders even got identical scripts to follow to make sure the subjects were none-the-wiser. Despite these precautions, there was a strong correlation between what the test leaders believed and the results they got. How did this happen?
When analyzing the videotapes from the experiments it became clear that the test leaders had – unknowingly – given the instructions for the task that supported their hypothesis much more clearly, than the instructions for the task that did not. And the good subjects had picked up on this and done a better job at that task.
Even though our user sessions are different from the experiments of social psychology, I do believe that it is important to keep in mind how easy it is to influence another person. To lean in or sharply inhale just as a respondent is closing in on something critical could be as obvious to them as clapping your hands and exclaiming “you’re getting warmer!”
We are all social beings and far more skilled in non-verbal communication than we often think we are. Consider the UX-guru’s love for filming their respondents, just to get all those subtle signs that could never be expressed in words. As test leaders we must remember that that form of communication also goes the other way. I am in no way trying to scare anyone from talking to their users! Make sure you invite your target group and let the respondents do the talking and you will do fine. However, it does not hurt to be aware of how you read the instructions in the beginning of the session, or what your body language is reveling.
Communicating is something we do all the time, not just when we are speaking.