Researchers love trackers. At the same time, we also hate them. Trackers are designed to help us stalk brand metrics and compare them with those of sister brands and competitors over time, and build real-time dashboards that flag tiny issues before they explode into unresolvable problems. But the more the world changes, the more our trackers stay the same. The questions stay the same, the answers stay the same, and the insights… well, they become impossible to find.

Trackers are inherently problematic

One of the biggest complaints researchers have with trackers is that once they’re written, they can’t be changed.

Ever.

When we inevitably discover a question that is poorly written, no longer relevant, or simply wrong, we can’t touch it or we’ll introduce confounds invalidating the trendline for every subsequent question. Data quality is always top of mind for researchers who care about making valid and reliable generalizations.

Oh the times, they are a’changin

But wait. No matter how much we work to keep questions consistent for the sake of research rigor and validity, everything outside of the questions has changed since day one. Every research supplier constantly improves their techniques and processes over time – without getting our approval. Every research participant changes their demographics, internet providers, and digital devices over time – without getting our approval. Like it or not, third parties change the methodological foundation of our trackers every single day without our approval. They’ve embraced change and it makes no sense except for researchers to embrace change too.

Who’s the boss?

Trackers are inanimate objects we personally create to suit our personal needs. Researchers need data that is valid and reliable. We need data that answers our questions and helps solve our challenges. We need to stop letting questionnaires be the boss of us and start making questionnaires work for us. We need to embrace change.

Choose change-resistant designs

Fortunately, researchers have methodological techniques that are designed to be resistant to change. If we build change into every questionnaire, change will have a vastly smaller impact on our data.

How can we do this?

Randomization! When each person receives answers (or questions) in a different order, it helps prevent confounds related to order. Adding an item to a randomized list greatly reduces its ability to affect subsequent items because everyone sees a different set of subsequent items. Make sure to randomize answer options at every appropriate opportunity. If it also makes sense to randomize the order of some questions, then do that too.

Individual presentation. Potential order effects can be reduced even more by combining randomization with individual presentation. Rather than showing a full list of items so that people can scan through the entire list, show items individually. Since everyone sees a different set of initial items, order effects are different for everyone and therefore greatly minimized over the full sample.

Subsets! If you’re accustomed to breaking long questionnaires into shorter, more manageable chunks for participants, you might already be using question subsets. For example, let’s say Q6 has 20 answer options – perhaps 20 brands or 20 product features. With subsets, each research participant gets only 10 answer options – perhaps three are the same for everyone, and the other 7 answers are randomly assigned. By design, no one sees every answer and your friendly, neighbourhood statistician can easily stitch the full questionnaire of 20 answer options back together. Need to add or remove an answer option? Go right ahead. Since half of people wouldn’t have received that item anyways, you aren’t intruding a serious confound. Even better, everyone benefits from a shorter questionnaire!

Know what questions are carved in cement

Some questions should never change. There are only a few seriously important KPIs that get added to the norms/benchmarks database every time you complete a wave. They probably include:

• Purchase intent
• Recommendation
• Satisfaction
• Trust
• Likeability
• Believability

Identify which items on a questionnaire MUST stay the same. They’re the items that are part of every questionnaire ever written for every product line and SKU. From now on, keep them as close as possible to the beginning of the questionnaire . By ensuring this section always stays the same with no potentially new and leading items before them, we can ensure they won’t be confounded by order effects.

And don’t get caught up in the idea that questions tied to financial incentives can’t be changed. Do you really want to incentivize the wrong KPIs and the wrong behaviors? Absolutely not!

Embrace change

Now here’s the hard part.

Change is good.

Track valid benchmarks: Tracking invalid data serves no purpose. Creating a brand new VALID benchmark serves a great purpose. Once you realize you’ve been tracking invalid data, it’s time to make a change and fix the problem. Similarly, once you realize you’ve missed answer options or used disrespectful language, it’s time to fix the problem.

Watch the world evolve: Change lets us account for our evolving society, culture, technology, and political atmosphere.

• When did you change the sex and/or gender questions on all of your studies to be more respectful and inclusive? If you haven’t done so yet, this PDF from Insights in Color will get you started.
• When did you add Facebook or Instagram as viable channels in addition to door-to-door salespeople, radio, and TV? Have you added TikTok to your list of channels yet? (You’d better!)
• When did you add Madonna to your list of influencers? What about Beyoncé? What about Billy Eilish?

You made those changes and didn’t think twice because it was the right thing to do it.

Plan to measure current issues: Build an entire section into your questionnaire that is all about change. If Section A is your unchangeable KPIs, make Section D completely new every single time. This quarter, it might be all about sustainability. Maybe next quarter it will be innovative packaging and the quarter after that will be all about diversity and equity.

Embrace fun! Change also lets us create questionnaires that are better able to capture the imagination of participants. Social networks and online games are fun because they leverage audio, video, swiping, and dragging. It’s time to change up your questionnaires so they are just as engaging.

.

What’s next?

It’s time for researchers to stop being pushed around by trackers. We know what we’re trying to accomplish and why. We know how change affects data. It’s time for us to be the boss of trackers and make them work for us! Embrace change!

Are you ready to design a useful tracker that generates great quality data using questions that are inherently engaging? Email your project specifications to our research experts using Projects at E2Eresearch dot com!

Learn more from our case studies

• Tracking Customer Satisfaction Across 8 Countries
• Tracking Physician Perceptions of Diabetes Pharmaceutical Representatives

Download information about our services

• Launching Your Tracker with E2E Research  
• Data Quality Processes for Questionnaires 
• Engaging Question Design that Turn Boring into WOW!