If you’re like most public sector web analysts, you probably agree that web analytics is a highly awesome tool in our toolbox…
But web analytics can’t tell us the whole story.
Because web analytics is great at telling us “what”, i.e. what happened on our website…
But it can’t tell us why – most importantly, why are
people in IT so hard to work with some visitors able to get what they need from our site…while others struggle?
This is a crucial question, and one that can’t be answered with web analytics data alone.
That’s where qualitative research methods come in…
In this guide, I’m going to explain:
Let’s start with a simple definition of qualitative research first:
...is primarily exploratory research. It is used to gain an understanding of underlying reasons, opinions, and motivations …and involves collecting and analyzing non-numerical data.
And here’s a simple explanation of qualitative methods from Wikipedia:
Qualitative methods examine the why and how of decision making.
That’s a great distinction: when we do qualitative research for our web content, we want to find out why people made the decisions they did (why did they visit the website? Why did they follow the site path that they did?)
So for our purposes, qualitative research tries to get at the reasons/opinions/motivations of our site visitors by collecting and analyzing non-numerical data (i.e. data that consists of words, not numbers).
I gave part of this answer away in the intro…
Qualitative research is important because if we only used web analytics, we would only understand what is happening on our websites.
We wouldn’t be able to dig deeper and gain insight into why visitors are doing what they’re doing.
But there’s a related point that’s also hugely important: as the U.K.’s Government Digital Service (GDS) says:
“Find what works, not what’s popular.”
As the GDS says, in the context of public sector services, something “works” only if “…people who need it can use it to get the right outcome for them.”
Think about that for a moment...
As web analysts, we’re often asked to produce reports and dashboards with “volumetrics”, i.e. metrics like “pageviews” and “visits” that typically have the biggest numbers.
But in digital performance measurement, size doesn’t matter…
What our metrics should be telling us is what works and (equally important) what doesn’t work.
Quantitative research can do that quite well. By setting up “goals” in Google Analytics (“success events” in Adobe Analytics) we can tell how many people achieved specific outcomes (e.g. downloaded a form; completed an online registration).
But we can’t tell why some people were able to achieve those outcomes, while others were not.
That’s where qualitative research comes in – it can tell us the “why”. And online qualitative research can tell us the “why” quickly, cheaply, and at scale.
Since you’re reading this article, I’ll assume you’re already a convert. So let’s jump right into a proven framework for doing killer online research.
You can think of this methodology as a “toolkit” of research methods. Each of these methods gives you deeper insight into the needs and behaviours of your users.
In his article explaining the ResearchXL methodology, the founder of ConversionXL stresses that conversion optimization is not simply a bunch of tactics – it’s a process. As they say:
"...amateurs focus on tactics (make the button bigger, write a better headline, give out coupons etc.) while pros have a process they follow.
And Peep hits on a crucial point when he says the most important thing about conversion optimization is “the discovery of what matters…If you figure it out, you know WHAT to optimize, and WHERE.”
Combining this insight with that of the U.K.’s GDS, we can say that:
Qualitative research helps us to discover what matters and why it matters.
In the sections below I describe each of the methods in the ResearchXL methodology – what they are, what they’re good for, and how you can use them to improve your digital presence:
(Note that I’m focused in this article on qualitative methods, but the ResearchXL methodology also includes quantitative methods, such as web analytics analysis. I’ll briefly cover those methods, as well.)
“Heuristic analysis” (or “heuristic evaluation”) is a fancy term for the process of reviewing a website (or content grouping, app, etc.) and comparing it against accepted usability principles, to determine what improvements could be made.
To put that in more pedestrian terms: as a web analyst (or usability designer, etc.), you know a good website when you see one. Are the web pages you’re reviewing laid out well? Could a visitor easily accomplish their goal(s)?
The Interaction Design Foundation has a good list of steps for conducting heuristic evaluation:
ConversionXL assesses each page of a website using these criteria:
Technical analysis involves fixing the technical “bugs” that are stopping your visitors from getting the outcomes they want.
Put another way: is there anything wrong with the way your content renders in different browsers or on different devices, and is this preventing your visitors from getting what they want? And does your content load quickly?
Technical analysis consists of these three steps:
Page load speed might sound like a trivial thing, but studies have shown that if your pages take more than three seconds to load, you could be losing nearly half of your visitors. (Think about that!)
Web analytics analysis is all about ensuring your web analytics is set up properly – especially set up to help you understand if your digital presence is contributing to your organization’s objectives.
(Note: if you haven’t set up a performance framework for your digital presence, and you haven’t established KPIs, now is the time to do it!)
Web analytics analysis involves three steps:
For a detailed tutorial on this step, check out ConversionXL’s Google Analytics Health Check.
(And check out the References & Further Reading section at the end of this article for tutorials on how to set up Google Analytics the right way.)
As I describe in my guide on creating a performance measurement framework for your content, your KPIs should flow from (a) the objectives of your digital presence and (b) your content goals.
In other words: where are people “dropping out” of the funnels/paths?
Once you’ve identified those drop-off points, you can use some of the qualitative techniques outlined in this article to find out why your visitors are dropping out.
Mouse tracking allows us to see where a visitor clicked on, and scrolled down, a web page:
As an example, the image of the SERP (search engine results page) above is a map showing where visitors clicked on the page. The red colour shows where people clicked the most, with fewer clicks as you move away from the red colour into orange, then yellow, then green.
The GIF below shows a user session video:
Qualitative surveying is an exploratory research method in which open-ended questions are used to probe a topic (as opposed to quantitative surveys, which typically use close-ended questions, and are meant to provide statistically-significant results for a large population).
Qualitative surveys can provide you with valuable data on:
The key point here is to understand the characteristics of your most successful visitor segments. In other words, what differentiates the people who are getting what they want from your web content, versus the people who are not?
You might think you know the characteristics of your visitors, but do you really know?
Here are the two major types of online qualitative surveys:
The best uses of on-site surveys are to find out:
You can take that information and use it to remove the obstacles that visitors have told you are getting in their way.
In both the private and public sector, we often have the contact details of clients (at least their email address). By contacting clients who achieved their goal(s) while using your site (e.g. applying for a program), you can learn how and why they were able to successfully use your site.
When formulating the questions to ask clients, remember that you want actionable data. So don’t ask “nice to know” questions – for every survey question you consider asking, put it through this filter:
“What will I do with the data from this question?”
If you don’t have a good answer, eliminate that survey question.
Here are 4 types of questions to ask in a client survey:
User (aka usability) testing is about task completion. Can a site visitor achieve his or her specified goal efficiently, effectively, and with a high level of satisfaction?
Here’s another definition, by usability guru Steve Krug:
Author of Don’t Make Me Think
Usability really just means making sure that something works well: that a person of average (or even below average) ability and experience can use the thing – whether it’s a web site, remote control, or revolving door – for its intended purpose without getting hopelessly frustrated.
So usability is about a person being able to use something to achieve an outcome or a goal.
The usability group in the U.K.’s Government Digital Service (yes, them again) has a great line related to this:
“Effectiveness for all users takes priority over efficiency or satisfaction for some users.”
That’s an interesting position to take: that it’s more important for a website to help all visitors achieve their goal, than it is to provide an efficient or satisfactory experience for some visitors.
In other words: are we helping every visitor get done what they want to do?
There are a ton of ways you can do user testing. Here’s a great list from Nielsen Norman Group:
And here's a quick description of the key methods:
For an excellent (free!) online manual on how to do user research, check out the U.K. Government Digital Service’s user research manual.
Web analytics is an extremely important tool in any web analyst’s toolbox - but when we’re analysing web analytics data, we’re really only able to answer the ‘what’, i.e. ‘what did our visitors do on our site?’
To answer why visitors took the actions they did (or did not) on our site, we need to use qualitative research techniques. Qualitative research is used to gain an understanding of underlying reasons, opinions, and motivations, and involves collecting and analyzing non-numerical data. Online qualitative research has the benefit of being relatively inexpensive, quick, and scaleable.
There are dozens of online qualitative research techniques. In this article, I introduced an excellent framework created by the conversion optimization agency ConversionXL that groups the most important research methods. The framework consists of these categories and methods :
Web Analytics Set-Up:
How to Conduct a Heuristic Evaluation (classic article by Jakob Nielsen - from 1995!)
I've been working in digital marketing for 15 years, with a specialty in web analytics and everything performance measurement. I'm a researcher by avocation and love building frameworks (how nerdy is that!)
Web Analytics 101: What It Is & Why You Should Care
Glossary: X Key Web Analytics Terms Every Analyst Should Know
Learning Resources for Public Sector Web Analysts
Data Storytelling for Web Analysts: How to Educate & Persuade Using Data
Please log in again. The login page will open in a new tab. After logging in you can close it and return to this page.