UX research ≠ good UX research.
Coming from a design background and working in the industry for a couple of years, I’ve seen many occasions where UX research failed to serve its purpose. It’s not uncommon that after a significant amount of time and effort were put into conducting interviews/focus groups/usability testing, the findings turned out to be unsurprising or not in line with what actually happens in the real world.
This is the moment when people, including myself, start questioning about the value of UX research in terms of cost-efficiency and the impact on the final products.
Not until recently when I dig a bit deeper into theories of UX research, did I realized that most of the time it’s not the value of research but the quality of research that results in the misjudgment.
In retrospect, I’m surprised by how easy a bad research can be created, and how few researches were done right. The following are some common pitfalls:
1. Confuse research designs with methods
Research designs are about making the structure of the research (e.g. what kind of data we should collect to answer the research question?); while research methods are about choosing the appropriate ways to collect the data (e.g. survey? interview? A/B testing?).
Although obviously research designs should always be the first step in a UX research, people tend to dive right into selecting research methods, figuring out ways of data collections before knowing what to do with those data. The latter usually produces purely descriptive outcome that tells no story. That’s why it gives the impression that UX research is useless.
2. Invalid inferences
Interpreting data is a subjective work. Although there are no absolute right or wrong arguments, there are valid or invalid inferences. The invalid inferences often involve the following mistakes:
First, confuse correlations with causations — e.g. the fact that the divorce rate and crime rate both increase in 20th century doesn’t mean that divorces cause crimes. There might be the third factor, such as economic environment, influencing both figures.
Second, confuse predictions with causations — e.g. knowing the school that a child attends helps us to predict the his/her academic performance, but it doesn’t mean that schools affect performances. It might be because of family status or conditions that first influences school selection and then leads to the differences in academic results.
From my experience, these pitfalls are prevalent in both the academic and working environment. For example, long ago in a student project, I designed a calendar app on tablet with the attempt to revolutionize the current interaction of the digital calendar. In hindsight, it can not be more obvious that choosing tablet as the device of the calendar was a huge mistake, because people mainly carry their phones around rather than tablets. However, at that time, I argued in the report that “tablet was chosen as the device because research suggested that people use calendars mostly in a sitting position which is same as tablet’s usage scenario”. Apparently, the inference was invalid because the two scenarios (sitting) are coincidental and uncorrelated.
Furthermore, one time our team made an app to introduce a historical district in a city. We made personas and user stories to convince the client (and us) that target users (tourists and locals) would like to use the app to explore that district. We basically just went through the chosen research methods instead of firstly setting up a clear research goal, such as “what do users need to know about the district” or “what does this district mean to the users”. As a result, the research didn’t add much value in framing the right direction, and therefore it was no surprise that the app turned out to be just a nice-looking yet unwanted product.
With all being said, I think we should all care about UX research because it helps to see the bigger picture, making sure to do the right thing at the beginning rather than later on trying to make things right. On top of that, we should invest more in UX research because it is extremely difficult to produce good UX research that requires time, effort, knowledge and experience.
(This article was also published on Medium)