Beyond Web Analytics: 6 Crucial Qualitative Research Methods You Need in Your Toolbox

This is a crucial question, and one that can’t be answered with web analytics data alone.

That’s where qualitative research methods come in…

In this guide, I’m going to explain: ​

Let’s start with a simple definition of qualitative research first:

And here’s a simple explanation of qualitat​​ive methods from Wikipedia:

That’s a great distinction: when we do qualitative research for our web content, we want to find out why people made the decisions they did (why did they visit the website? Why did they follow the site path that they did?) 

So for our purposes, qualitative research tries to get at the reasons/opinions/motivations of our site visitors by collecting and analyzing non-numerical data (i.e. data that consists of words, not numbers).

I gave part of this answer away in the intro…

Qualitative research is important because if we only used web analytics, we would only understand what is happening on our websites.

We wouldn’t be able to dig deeper and gain insight into why visitors are doing what they’re doing.

But there’s a related point that’s also hugely important: as the U.K.’s Government Digital Service (GDS) says:

As the GDS says, in the context of public sector services, something “works” only if “…people who need it can use it to get the right outcome for them.”

Think about that for a moment...

As web analysts, we’re often asked to produce reports and dashboards with “volumetrics”, i.e. metrics like “pageviews” and “visits” that typically have the biggest numbers.

But in digital performance measurement, size doesn’t matter…

What our metrics should be telling us is what works and (equally important) what doesn’t work.

Quantitative research can do that quite well. By setting up “goals” in Google Analytics (“success events” in Adobe Analytics) we can tell how many people achieved specific outcomes (e.g. downloaded a form; completed an online registration).

But we can’t tell why some people were able to achieve those outcomes, while others were not.

That’s where qualitative research comes in – it can tell us the “why”. And online qualitative research can tell us the “why” quickly, cheaply, and at scale.

Since you’re reading this article, I’ll assume you’re already a convert. So let’s jump right into a proven framework for doing killer online research.

The CO (conversion optimization) agency ConversionXL has developed an evidence-based methodology for generating the insights necessary to increase conversion rates on any website.

You can think of this methodology as a “toolkit” of research methods. Each of these methods gives you deeper insight into the needs and behaviours of your users.

In his article explaining the ResearchXL methodology, the founder of ConversionXL stresses that conversion optimization is not simply a bunch of tactics – it’s a process. As they say:

And Peep hits on a crucial point when he says the most important thing about conversion optimization is “the discovery of what matters…If you figure it out, you know WHAT to optimize, and WHERE.”

Combining this insight with that of the U.K.’s GDS, we can say that:

In the sections below I describe each of the methods in the ResearchXL methodology – what they are, what they’re good for, and how you can use them to improve your digital presence:

(Note that I’m focused in this article on qualitative methods, but the ResearchXL methodology also includes quantitative methods, such as web analytics analysis. I’ll briefly cover those methods, as well.)

“Heuristic analysis” (or “heuristic evaluation”) is a fancy term for the process of reviewing a website (or content grouping, app, etc.) and comparing it against accepted usability principles, to determine what improvements could be made.

To put that in more pedestrian terms: as a web analyst (or usability designer, etc.), you know a good website when you see one. Are the web pages you’re reviewing laid out well? Could a visitor easily accomplish their goal(s)?

The Interaction Design Foundation has a good list of steps for conducting heuristic evaluation:

Technical analysis involves fixing the technical “bugs” that are stopping your visitors from getting the outcomes they want.

Put another way: is there anything wrong with the way your content renders in different browsers or on different devices, and is this preventing your visitors from getting what they want? And does your content load quickly?

Technical analysis consists of these three steps:

Page load speed might sound like a trivial thing, but studies have shown that if your pages take more than three seconds to load, you could be losing nearly half of your visitors. (Think about that!)

Web analytics analysis is all about ensuring your web analytics is set up properly – especially set up to help you understand if your digital presence is contributing to your organization’s objectives.

(Note: if you haven’t set up a performance framework for your digital presence, and you haven’t established KPIs, now is the time to do it!)

Web analytics analysis involves three steps:

Mouse tracking allows us to see where a visitor clicked on, and scrolled down, a web page:

Qualitative surveying is an exploratory research method in which open-ended questions are used to probe a topic (as opposed to quantitative surveys, which typically use close-ended questions, and are meant to provide statistically-significant results for a large population).

Qualitative surveys can provide you with valuable data on:

The key point here is to understand the characteristics of your most successful visitor segments. In other words, what differentiates the people who are getting what they want from your web content, versus the people who are not?

You might think you know the characteristics of your visitors, but do you really know?

 Here are the two major types of online qualitative surveys:

The best uses of on-site surveys are to find out:

You can take that information and use it to remove the obstacles that visitors have told you are getting in their way.

In both the private and public sector, we often have the contact details of clients (at least their email address). By contacting clients who achieved their goal(s) while using your site (e.g. applying for a program), you can learn how and why they were able to successfully use your site.

When formulating the questions to ask clients, remember that you want actionable data. So don’t ask “nice to know” questions – for every survey question you consider asking, put it through this filter:

“What will I do with the data from this question?” 

If you don’t have a good answer, eliminate that survey question.

Here are 4 types of questions to ask in a client survey:

User (aka usability) testing is about task completion. Can a site visitor achieve his or her specified goal efficiently, effectively, and with a high level of satisfaction?

Here’s another definition, by usability guru Steve Krug:

So usability is about a person being able to use something to achieve an outcome or a goal.

The usability group in the U.K.’s Government Digital Service (yes, them again) has a great line related to this:

That’s an interesting position to take: that it’s more important for a website to help all visitors achieve their goal, than it is to provide an efficient or satisfactory experience for some visitors. 

In other words: are we helping every visitor get done what they want to do?

There are a ton of ways you can do user testing. Here’s a great list from Nielsen Norman Group:

Web analytics is an extremely important tool in any web analyst’s toolbox - but when we’re analysing web analytics data, we’re really only able to answer the ‘what’, i.e. ‘what did our visitors do on our site?’

To answer why visitors took the actions they did (or did not) on our site, we need to use qualitative research techniques. Qualitative research is used to gain an understanding of underlying reasons, opinions, and motivations, and involves collecting and analyzing non-numerical data. Online qualitative research has the benefit of being relatively inexpensive, quick, and scaleable.

There are dozens of online qualitative research techniques. In this article, I introduced an excellent framework created by the conversion optimization agency ConversionXL that groups the most important research methods. The framework consists of these categories and methods :