Inside the service design environments, qualitative research is pretty well known and used, it is an integrated tool within the everyday process. But when it comes to different environments, the opinion on which one - between qualitative and quantitative research - is the right one may change. Often we have to face difficult questions that brought us to talk about this topic:
If it is possible to A/B test it, why do we need to use qualitative research?
This question should explain why - on the opposite - we absolutely need to include it into our research.
Since there are other precise and numerable data sources (like user metrics), why should we even bother? The truth is that qualitative research gives back data impossible to get in any other way. It can answer “why” while other methods only answer “what,” “how,” “when,” and “where” (Karina, "Qualitative Research Is Always Biased").
Qualitative Research isn’t considered “real” research within certain types of environment. This aspect has terrible consequences for innovation. Companies are obsessed with the operational effectiveness, begging for quantitative data. But quantitative data does not unveil how or what ways users are making their own decisions. Quantitative data shows standards: how products fit the market, how many products are produced and sold, how fast it is possible to make them. It can tell you the average satisfaction a customer may have, but it cannot reveal any of the detail behind that satisfaction (Sam Ladner, "Qualitative research and innovation"). As far as we know, understanding what is the turning point and why something changed inside the experience of the user is pretty much everything, in order to create meaningful product/service and deliver useful results.
Qualitative research unveils the details: shows how people are using products, why and with which differences. That’s why qualitative research - together with field-based ones like ethnography - is the answer to deliver uncommon and different solutions.
In order to understand why qualitative research is really needed to complete the user research puzzle, here are six reasons for why applying it should be useful in addition to A/B tests.
1_Reach the “why” piece of the puzzle: A/B test results usually tell what the impact of a feature is. What this type of data doesn’t tell is the motivation behind it and why users behaved the way they did. Sometimes the why can be obvious but - most of the time - it isn’t so easy to understand. A/B testing may result negative, ambiguous or flat, but even with positive A/B tests there will be the need to iterate on a feature. Talking to users will help to easily understand where it isn’t working yet.
2_To create hypotheses to be tested: In order to provide meaningful insights, qualitative user research is actually the best tool. With the obtained insights it’s possible to generate hypotheses to test with the A/B method. Obviously A/B test can be the starting point, but qualitative research is going to be faster and more effective. Even during an advanced stage of the project, talking to users is the right tool to adjust the path.
3_A/B test can’t validate features: Not all the features can be A/B tested with meaningful results. Usually, features have so little usage that in order to get an enough consistent sample, the test has to be run forever. Moreover, features have interaction effects that include the ability to react in relation to another user’s action. Even a simple function as “like”, could impact on both people involved in this cross-user situation: A/B test hasn’t the chance to filter it correctly.
4_Unveil new opportunities: In order to expand the reach of a product or service and jump into new customer bases, there is the need to identify those new users’ needs and verify if the solution works for them. This cluster isn’t using the product/service yet, so there isn’t the chance to A/B testing. In this case qualitative research can identify these opportunities talking to prospective customer in order to test prototypes of the new solution.
“A/B testing can yield incremental improvements, but qualitative research is required to jump an entirely new level”
(Jens-Fabian Goetzmann, “Why PMs Need Qualitative Research”).
5_Faster answers: Even if talking to user involves all the existing communication problems, it’s way easier to get a qualitative feedback talking to them instead of actually building an A/B testing. It is much faster and cheaper to build a prototype rather than the whole experience. Moreover, when A/B testing, the hypothesis has to be as specific as possible, making the changes as small as possible. Bigger changes affect multiple aspects of the product/service at once, and there is no chance to be punctual with the quantitative research.
6_Stay humble: Working on a project for a long time gives the feeling of knowing what users need, sometimes having the sensation to know it better than the subjects. This isn’t true. Talking to users on a regular basis provides an indispensable checks and keeps everything on a ground level.
To sum up, qualitative research keeps us grounded and let us build empathy with the end user, with a real human-centered approach. Without qualitative research, the models we create wouldn’t have been characterized by reality and it would be impossible for us to address them to our users’ needs. With qualitative methods, we infuse our models with intuition and hypotheses that can be verified later with quantitative data (Robyn Rap and Vicky Zhang, “Qualitative before Quantitative: How Qualitative Methods Support Better Data Science"). At this point in this context, it’s an accepted truth that qualitative data and quantitative data are able to tell a better and more complete story if they are used together and in a connected way, instead of using each one on its own.