On "Small Data:" The Necessity of Qualitative Research in the Age of Big Data

2013-09-09 17:21:53 +0000

I came across a post on Medium that brilliantly makes the case for qualitative research: the point many of us seem to have forgotten as we deal with “bigger and bigger” data is that there are many things we can’t reliably stick numbers on.

We’re increasingly living in a quantifiable world—if you look around you’ll notice we’re attempting to quantify learning in our schools (No Child Left Behind), our health (see: quantified self), even something as un-quantifiable as love (eHarmony’s “29 Dimensions of Compatibility”). 

At work, we try to quantify our contributions in terms of dollars added to the bottom line (as a side note, how do you even do that if you are, say, a BI report monkey, producing something that isn’t inherently and directly revenue-generating?).

Facebook and LinkedIn attempts to quantify our relationships through social network analysis. Google, likewise, attempts to put a number on the usefulness of a Web document to us using PageRank.

The point is not everything is measurable by numbers. In the Web Analytics space, my sandbox, people try to measure “engagement” (whatever that is), with numbers. I have a hard time with that metric. A number representing how “engaged” someone is with your site? At best, a metric like this is a dull butter knife. It totally fails to capture the nuance of the user experience. It’s a totally lossy signal. Don’t get me wrong—I love and live data and numbers, but there’s tremendous value in actually talking to our users.

The beauty of talking to actual users, instead of trying to imperfectly measuring them, is that you can get inside people’s minds in a way no KPI can. You can directly observe how people interact with your product. You can stop people at any time and ask them why they did something. You can ask them things like, “how can I make my product easier or better to use,” or “were you able to accomplish what you set out to do today with my product?”

This kind of user research gives you super-valuable CONTEXT.

There is one major drawback to qualitative user research. It’s time and resource intensive.

Because it’s time and resource intensive, it is extremely hard to automate. Which is a roundabout way of saying: qualitative user research is expensive.

Because qualitative user research is expensive, we typically operate with small sample sizes.

Because the sample sizes are typically small (even microscopic in this age of Big Data), people have a natural aversion to trusting the findings.

However, this is totally okay. It’s okay because this isn’t your only vector of understanding. Qualitative user research, remember, is a way to add context to your numbers. Qualitative user research gives you insights into user behavior that quantitative research can’t, and vice versa. There’s a symbiotic relationship between the two, and one is much stronger with the other.

Another benefit of the small sample sizes is that it’s very easy to change roadblocks and fix problems as you go along. As soon as you identify a user experience problem you can stop and fix it. The beauty is that you’re not wasting valuable time on things that aren’t working. There’s a name for this—the RITE method—Rapid Iterative Testing and Evaluation. RITE lets qualitative user findings guide quantitative research; done right it minimizes grief and wasted time, and maximizes the return on a usability study.

I’ve been rambling on for a while. Let me sum up the argument like so:

Talking to real users provides context. Without context, your numbers are just numbers: they contain no meaning on their own. Quantitative analysis and qualitative analysis go hand-in-hand if you want to maximize your return on user research.