Request a quote!

Blog, News &

Case Studies

What is a Census Representative Sample?
By E2E Research | March 29, 2022

The people researchers choose to share their opinions in marketing research can make a huge difference in the quality of answers we receive. That’s why it’s important to understand the research question and who would be best suited to speak with to get the necessary answers.

 

Let’s consider one type of sample that researchers often consider when conducting research – a census representative sample.

 

 

What is a census representative sample?

Decorative imageYou might also hear these referred to as ‘Census Rep’ samples. A census rep sample requires access to census data, something that is typically generated by large-scale government surveys completed by millions of their residents or citizens. In the USA, that’s census.gov and in Canada, that’s Statistics Canada.

 

A census rep sample can be designed to reflect any specific group of people. The key consideration is that the sample of completed questionnaires reflects the larger population on important criteria. The sample could reflect an entire country (e.g., USA, Mexica, Canada), a state or province (e.g., California, Quebec), or a city or town (e.g., Boston, Ottawa). This type of census rep sample is reasonably easy to define.

 

Another type of a census rep sample can be defined by target group behaviors or characteristics. For instance, you might be interested in a census rep sample of people who smoke or who have diabetes. Of course, building these types of census rep samples is far more difficult because government census data tends to be set up to understand basic demographics like age and gender, rather than behaviors like smoking and ailments like diabetes.

 

 

When would I use a census representative sample?

Census rep samples are extremely important for at least couple research objectives.

 

First, when you need to calculate incidence rates for a product or service, you need to first generalize from a representative group of people of your target audience. You need to be able to define your population before you can know what percent of them uses a product or performs a behavior.

 

Second, census rep samples are extremely important for market sizing. Again, you need to generalize from a representative group of your target audience before you can estimate the percent of people who might qualify to use your product or services.

 

 

Why is a census representative sample important?

Decorative imageCreating a census representative sample is extremely important. You could get into trouble if you recruit a sample of research participants who don’t look like actual users.

 

You might gather opinions from too many older people, too many women, too many higher educated people, or too many lower income people. Your final research conclusion might be based on opinions collected from the wrong people and lead to development of the wrong product or product features.

 

 

An example of a census representative sample

Let’s consider an example where we want to determine which flavour of pasta sauce to launch in a new market – California. We’ve got two delicious options – a spicy, jalapeno version and a mild portobello mushroom version.

 

We know people from different cultures and ethnic backgrounds have very different flavor preferences so we need to ensure that the people who participate in our research will accurately reflect the region where we will launch this new pasta sauce.

 

Now, we could recruit and survey a sample of people based on a basic quota that will help make sure we hear from a range of people. It might look like what you see in the first column of the table – even splits among each of the demographic groups with a bit of estimation for ethnic groups. But that’s not actually what California looks like. Instead, let’s build a census rep sample matrix based on real data.

 

Decorative image To start, we need to define a census rep sample of California. First, we find those people in a census dataset. Then, we identify the frequencies for each of the key demographic criteria – what is the gender, age, ethnicity, and Hispanic background (as well as any other important variables) of the people who live in California. Fortunately for us, this data is readily available. On the census.gov website, we learn that in California, 50% of people are female, 6% are Black, and 39% are Hispanic.

 

Now we can recruit a sample of people from California whose final data will match those demographic criteria – 50% female, 6% Black, and 39% Hispanic. You can see just how different those numbers are are from the original basic quotas!

 

In the last two columns, you can see that we’ve even split out the criteria by gender (even better, you can do this based on the census data). This will ensure that one of the age groups isn’t mostly women or one of the Hispanic groups isn’t mostly men. When we nest our criteria within gender, we end up with a nested, census rep sample. Nested demographics are the ideal scenario but they do make fulfilling sample more costly and time-consuming. You’ll have to run a cost-benefit analysis.

 

 

What’s Next?

Are you ready to build a census representative sample for your next incidence rate or market sizing project? Email your project specifications to our research experts using Projects at E2Eresearch dot com. We’d love to help you turn your enigmas into enlightenment!

 

 

 

Sample Conferences

 

 

Learn more from our case studies

 

 

Learn more from our other blog posts
Why the Divide Between Academic and Commercial Market Researchers?
By E2E Research | November 11, 2021

By Satish Pai, MBA (pictured right) and Annie Pettit, PhD

.

Satish Pai

A few months ago, an academic tweeted about the lack of available learnings from market research agencies about predicting market behaviours. Surely, an agency that carried out dozens of econometric marketing mix modelling studies should have discovered and been able to share quantitative patterns and learnings by now.

 

This was a fair request, a valid claim, and a healthy discussion followed.

 

.

To Share or Not To Share?

One perspective was that agencies such as Nielsen BASES, Millward Brown, and Kantar have already shared some of these learnings, though not necessarily in academic journals. Much of their research learnings could be found in their own internal publications, in industry journals, as white papers on association websites, and as part of industry conferences. Seek and ye shall find.

 

The other perspective addressed the lack of sharing as an appreciation for intellectual property, proprietary data, and client privilege and privacy. Almost all research conducted by industry requires significant investments in terms of time and talent, as well as participation from commercial clients who share their privileged data to help test and validate such research products.

 

Given the significant investments required to test and validate learnings, it makes sense to recover the financial investment as early as possible. Doing so requires the protection of intellectual property (IP) to ensure they maintain their unique selling proposition (USP). Pursuing publication in an academic journal could easily prevent this from happening.

 

For industry researchers, the true test and validation of commercial research is not publication in academic journals, but rather success in the market with measurable financial ROI across dozens of products and categories. This external validation is sufficient, particularly since prospective clients can witness research outcomes in the marketplace themselves or follow-up on testimonials from satisfied clients.

 

The academic researchers, however, felt that any research learnings should be subject to the academic process of peer review prior to publication in either academic or industry journals. Peer reviews require multiple experts within the field to scrutinize the methods and processes to ensure quality standards were upheld.  They argued that since agencies generally do not do this, their research isn’t validated and doesn’t pass academic rigour.

 

The conversation concluded as both perspectives had merit.

 

.

Where Should Research Results Be Shared?

There soon followed another Twitter conversation wherein a media agency head wanted to share their agency’s learnings on marketing ROI. They asked for suggestions on how to share those findings in an academic journal.

 

Many academics pitched in with advice. The popular suggestion was that the agency should instead try out industry forums like ARF, ESOMAR, IIEX, and MRS. Those forums would be quick to publish and offered an excellent platform to reach the industry.

 

Despite the original request, there wasn’t much encouragement to publish in academic journals. The agency was warned that the process from submission to publication could take as long as 18 months (though with edits and revisions it could take up to 3 years). Further, journals seen as prestigious have rejection rates as high as 95% meaning that even great quality research may not be published. Along with these warnings to avoid academic publication, several journals that covered the areas of interest were suggested, along with ideas on how to go about submitting.

 

The contrast between these two discussions was striking. The first discussion was grounded in a premise that industry learnings fall short as the commercial vetting process isn’t the same as the academic vetting process. The second discussion was a realistic admission that sharing industry learnings via the academic route would entail several challenges and ultimately might lead to the learnings never being shared.

 

.

Where Should Researchers Publish?

Stepping back, it’s interesting to see the variety of publication sources that marketing and affiliated industries use today. There is social media (e.g., Linkedin, Medium), dedicated media (e.g., Adweek, Admap, Marketing Week) as well as industry publications, dedicated channels, and conferences.

 

Mainstream media also offers considerable opportunity. Long-form columns in diverse publications such as The Guardian, New Yorker, and Financial Times can have greater impact and currency, and some would agree that they offer the best viewpoint on industry developments and learnings.

 

Further, there are a few publications, such as Harvard Business Review, that have bridged the gap and offer opportunity to both industry and academics. If an academic or industry researcher were to publish in such journals, they could be assured that a good number of projects from paying clients would ensue.

 

In the end, however, it would be fair to say that mainstream media dominates by far, and outdoes the limited impact that both academic journals and industry publications have on marketing and related industries.

.

 

Where Do We Go From Here?

These discussions demonstrate an ongoing divide between the working lives of academic and industry researchers.

 

Is there one best vetting process to identify rigorous research studies that generate valid and reliable outcomes? Clearly, there is merit in both peer review and external validation.

 

Is a publication process with high Type 2 error rates (rejects many great papers) preferable to a process with low Type 1 errors (accepts some poor quality papers). Again, there is merit in both being assured that every result has been validated, and preventing the loss of potentially important findings.

 

Academic and industry researchers are on the same team. We strive to better understand the inner workings of consumers and markets but, we do so from within different contexts. Our target audiences are different – corporate end clients or other academics. Our definitions of prestigious career achievements are different – multiple Journal of Marketing publications or multiple ESOMAR awards. The KPIs that determine our raises and promotions are different – publish or perish, or monetize or perish.

 

Let’s continue the discussions and let’s keep the gate open.

 

 

Satish Pai, MBA is a freelance consultant, author, and PhD Candidate who specializes in advertising, branding, strategic management, and insights. He writes the Insights about Insights blog and can be found on LinkedIn and Twitter.

 

Annie Pettit PhD CAIP FCRIC is the Chief Research Officer, North America, at E2E Research, an ISO 27001 certified, ESOMAR corporate member. Annie is a marketing research author, blogger, and regular conference speaker. She can be found on Linkedin and Twitter.

Everything You Need to Know about Conducting Effective Secondary Research
By E2E Research | July 30, 2021

Secondary research is an under-utilized yet fantastic way to better understand your competitors, build business development strategies, understand regional markets, create market entry strategies, and so much more.

 

But what exactly is it? Unlike primary research where you create your own data by launching a questionnaire or focus group, secondary research entails using data previously generated for other purposes. If you wrote any literature reviews in high school, college, or university, chances are you already know all about secondary research. Now that we’re in the business world, secondary research includes finding and analyzing:

 

  • Survey, interview, focus group, mystery shopping, sensory testing, and biometrics research completed by your own company in previous years for other purposes
  • Sales, transactional, and logistics data that was originally collected by your company for the purposes of production and fulfillment
  • Social listening data collected from social networks, online comments, online reviews, blogs, and other user-generated website content.
  • Census research that was conducted by government sources to allocate funding and services throughout the local region or country
  • Academic research conducted at colleges and universities, whether it’s been published as a journal article or stuffed in a file drawer because the professor got interested in something else
  • Research conducted by competitors and presented at conferences or shared in blogs or industry magazines
  • Research conducted by industry associations among their members or their stakeholders
  • Research conducted by third-party groups for the sole purposes of selling for profit to other people (you!)
  • Data collected by internet search engines such as Google Trends

 

Some specific sources of secondary data that are often useful for consumer and market researchers include:

 

  • Acxiom – demographic, home, vehicle, shopping, interests data
  • Arbitron / Nielsen Audio – radio data
  • Comscore – website visits and behaviors, trends, digital/linear/OTT TV viewership
  • Datalogix – online click tracking, consumer lifestyles, demographics, audience data
  • Dunnhumby – customer data via retailer loyalty programs
  • ecommerceDB – traffic of major brands, business trends, revenue by country
  • Epsilon – demographic data
  • Equifax – financial data
  • ExactData – consumer and business names, postal, email addresses
  • IQvia – healthcare and pharmaceutical data, anonymous patient data
  • IRI – purchase, media, social, loyalty data, consumer, shopper, retail data
  • Mintel – new product launches by category
  • Numerator – retail purchases
  • SimilarWeb – websites traffic trends, sources

 

The key point is that someone else already generated or curated data to suit their own purposes and now you are taking advantage of it to make further analyses that suit your purposes.

 

No matter how innovative and ground breaking your research or business problem is, someone has ALWAYS done relevant research prior to you. For example, when the very first academic research examined the validity of online questionnaire data, lots of research had already been conducted to understand the validity of questionnaire data in general. No data exists in a silo!

.

 

Why use secondary research?

Let’s break secondary research into two overly simplified categories: Basic and complex.

 

Basic desk research

This is what you do everyday. It might take a few minutes or a few hours but you regularly:

 

  • Do quick online searches of the top 20 brands in a product category so that your questionnaire doesn’t exclude an important brand
  • Do a quick Twitter search to see the real words people use when they describe a brand so that you can build more informed focus group discussion guides
  • Check your government’s census data to ensure your questionnaire sampling plan is designed to reach a target group that reflects the general population in terms of gender, age, ethnicity, and income
  • Read an online blog post to gain a better understanding of a research methodology you don’t use very often (is this you right now?!)

 

 

Complex desk research

On the other hand, complex desk research might take weeks or months depending on how difficult it is find and analyze the information. For instance, developing a new product and introducing it to a new country with existing competitors would benefit from secondary research to address business problems such as:

 

  • What products already exist, what features do they offer, how are they priced?
  • What should your product cost given the prices and features of competitive brands and consumer characteristics?
  • How big is your market now and how big could it become?
  • How can your business identify the most strategic buyers and markets?
  • What do your suppliers and customers look like and which are the riskiest?
  • Which expansion strategies are effective in different parts of the country or the world?

.

 

Advantages of Secondary Research

There are disadvantages to every research methodology you might consider. For instance, with secondary research,  you won’t always find all the data you need, you’re not always sure about the detailed methodology behind the data collection and reporting, the data will never be complete nor perfect, and you won’t always get the why along with the what. But there are definitely some great advantages that come with triangulating multiple data sources to understand a specific problem.

 

Prove your worth

When launching a new product, it is important to prove to your boss, investors, and other stakeholders that your idea is worth investing in and you understand what is happening in the market. No idea exists in a vacuum and you need to demonstrate that you understand what your potential market looks like and what could go completely wrong (or completely right!) after you launch.

 

Avoid wasting time and money

Upon doing your research, you may discover that someone else has already built the amazingly innovative widget you were planning to build. You now have the opportunity to figure out how to differentiate yourself BEFORE wasting time building the exact same widget.

 

Decrease your margin of error

Why do researchers like large sample sizes? Because the more people you include in a research study, the more you improve your chances of finding the “correct” answer. As just one person, you can only conduct so much research. But, when you invest that time into collecting multiple pieces of research from multiple sources, you will improve your chances of finding the “correct” answer. In the academic world, you could think of this as meta-analysis – when 95 studies prove X and 5 studies prove Y, chances are the X is the “correct” answer.  (There often isn’t one correct answer, just a more comprehensive or well-informed answer.)

.

 

Tips to follow

Like many things, there is an art and a science to finding good quality data and analyzing it well. Here are some helpful tips to ensure you end up with useful secondary research:

 

Define clear research objectives

Just because you aren’t designing a questionnaire or interview doesn’t mean you can get away without well defined research objectives. Build a clear plan with specific research questions. Identify the types of data that could answer those questions – census data? Interview data? Sales data? It’s okay to start the research process with random searching in random places just to get a sense of what you know and don’t know. But, once you’ve used up that allotted discovery time, be specific and detailed about your next steps.

 

Create a framework for discovery

Rather than randomly looking for things, build a framework that will help you plan and organize. For example, “think, feel, do” is a  common framework. As you seek out information, look for data that helps you understand what people 1) think, 2) feel, and 3) do. Other frameworks might specify “buyers, brands, and businesses,” or “finances, logistics, and transactions,” or “design, field, analyze, report.” Focus on addressing the research problem from multiple angles – time frame, geography, target audience, metrics, and products. Whatever framework you build for yourself, it will help ensure that you cover all aspects of the business problem.

 

Seek a range of sources and data types

Look for government data, association data, academic data, newspaper data, and think tank data. Find qualitative and quantitative data, business and personal data, user and non-user data, customer and consumer data. Figure out all the types of places where your data could be and make sure it all gets a chance to be represented.

 

Don’t dismiss old data

Sales numbers, technology, and business processes might change quickly but human behavior changes soooo veeeeery slooooooowly. It’s often reasonable to skip over the technology part of older research – we don’t care about floppy disks, cathode ray tubes, and dot-matrix printers anymore. However, make sure to pay close attention to the human side of things. If people didn’t like something five years, their perceptions and emotions behind those dislikes could very well still be valid.

 

Seek out contradictions

It’s easy to find a set of data that offers conclusions you like and continue on the same path. But, that could lead you down one single path when there are actually multiple paths, all with enlightening and valid outcomes. That’s not to say you should entertain bad, wrong, or unethical ideas just because they are other ideas. Make sure you consciously seek out other ideas and actively reject them for good reasons rather than rejecting them because you didn’t know they existed.

 

Expect and confront bias

Try to identify potential sources of bias– age, gender, ethnicity, disability, sexuality, language, country, political, societal. You might not realize the bias there unless you specifically look for it. Once you’ve found it, then you can decide what to do about it. We’ve learned so much about bias in the last few years. This is your opportunity to stretch your new muscles.

 

Validate everything

Just because something is on the internet doesn’t mean it is good, right, or true. Heck, computers at the library are free to use, WordPress hands out blogs sites for free, and anyone can instantly create “The Authoritative Guide to Three Hour Questionnaires.” Unless that website was endorsed by ESOMAR, the Insights Association, and the Canadian Research Insights Council, I wouldn’t give it a nanosecond of my time.

 

Consider the Source: Who collected the data? Are they reputable? Do respected experts reference them? Do they treat those who disagree with them with respect? Do they point out the drawbacks, faults, and biases of their own research? Who complains about them in social media? Further, just because some data is created for the purposes of selling it for profit to multiple third parties doesn’t mean it’s biased… but it doesn’t inherently mean it’s trustworthy either.

 

Consider the Data: When was it collected? When was it published? How was the data checked for quality? What data might be missing or incomplete? Are key words specifically defined and not left to the imagination? Are the sample sizes appropriate to draw conclusions from? Is the sample reflective of the population it’s supposed to represent?

 

Include everything

When you’ve completed your thorough analysis, incorporate all of the data that led you to your final set of conclusions. This means including valid and trustworthy information that you agree with as well as valid and trustworthy data you disagree with. Share the entire set of information so that other people can come to their own conclusions too. If you’ve laid out your argument well, they should come to similar conclusions as you did.

..

 

What’s Next?

If you’ve got a simple secondary research project ahead of you, enjoy the process and leverage the time and money that someone else put into the research you’re benefiting from.

But, if you’ve got a complex and lengthy project ahead of you, our experienced desk researchers would be happy to help. Email your project specifications to our research experts using Bids at E2Eresearch dot com and we’ll lighten your load.

 

 

Learn from our case studies

 

Learn from our other blog posts

Tips for the First-Time Conjoint Analysis Researcher
By E2E Research | July 16, 2021

Researchers love conjoint analysis. It’s a handy statistical technique that uses survey data to understand which product features consumers value more and less, and which features they might be willing to pay more or less for.

 

It allows you understand how tweaks to combinations of features could increase desirability and, consequently, purchase price and purchase rate. Essentially, it asks, “Would you buy this product configuration if you saw it on the store shelf right now?”

 

Technically, there are numerous ways to present conjoint questions but all of them invite participants to compare two or more things. For example:

 

  • Would you rather buy this in red or yellow?
  • Would you rather pay $5 for a small one or $4 for a large one?
  • Would you rather buy this one or the competitive brand?
  • Would you rather buy this one or keep the one you already own?

 

The comparisons can get extremely complicated as you strive to create scenarios that mirror the complicated options of real life, in-store choices. This is because no two products are have the exact same features. There are always multiple tiny or major things different amongst them including brand, price, color, shape, size, functionality, etc.

 

As you see in the example conjoint question below, participants are being asked to select from among 5 different entertainment bundles, each with a different price and selection of options. Even though this question is nicely laid out, perhaps even nicer than what you might see in a store, it’s not a simple choice!

 

 

example conjoint analysis survey questions

.

 

Quick Conjoint Dictionary

First, let’s cover some quick terminology commonly used with the conjoint method so that the tips we will offer make sense.

 

  • Attribute: A characteristic of a product or service, e.g., size, shape, color, flavor, magnitude, volume, price.
  • Level: A specific measure of the attribute, e.g., red, orange, yellow, green, blue, and violet are levels of the attribute color.
  • Concept: An assembly of attributes and levels that reflect one product, e.g., a large bag of strawberry flavored, red, round candy for $4.99.
  • Set: A collection of concepts presented to a research participant to compare and choose from.
  • Simulator: An interactive, quantitative tool that uses the conjoint survey data to help you review consumer preferences and predict increases or decreases in market share based on potential product features and prices.

.

 

Conjoint Analysis Tips and Tricks

3 to 5: Across all attributes, levels, concepts, and sets, 3 to 5 is a good rule of thumb. With so many possible combinations of attributes, levels, and sets, the ask we’re making of participants could get overwhelmingly complicated and create a lot of cognitive fatigue. That’s why we suggest aiming for no more than 3 to 5 attributes, 3 to 5 levels per attribute, 3 to 5 concepts per set, and 3 to 5 sets. By ensuring that participants enjoy the process and can take the time to review each concept carefully, we can generate much better data quality.

 

Meaningful Levels: Choose attribute levels carefully. Do you really want to test 3 shades of blue or 3 flavors or apple? No. While you could choose price levels of $30, $32, and $34, they aren’t meaningfully different and wouldn’t create a lot of indecision on the store shelf. They wouldn’t create variation within your data. Try to include edge cases – options that are as far apart as you can make them while still being within the realm of possibility.

 

Be frugal with combinations: You already know there are combinations of attributes and levels you would never offer in-store so don’t waste people’s time and cognitive load testing them. Think carefully about which combinations of attributes and levels you would never offer together and exclude them from the test. For example, don’t waste your budget testing the least expensive price and the most expensive feature. Similarly, don’t test the value of adding an extra battery for a version of the product that doesn’t run on batteries.

 

Minimum number of shows: When testing a level, use it in at least 3 concepts for an individual person. Think of it in terms of a ruler – for quantitative metrics (e.g., price, length, volume, weight), you need to see whether the difference between Level 1 and Level 2 is perceived the same as the difference between Level 2 and Level 3.

 

magazinesInclude competitors: The real market includes competitors, often many. People don’t shop for single brands in isolation and neither should they answer your conjoint questions in isolation. Include at least one key competitor in your test, and preferably at least two. Further, if your brand is relatively unknown, you may wish to incorporate a competitor that is also relatively unknown.

 

Include an opt-out: Sometimes when you’re shopping, you discover they don’t have what you’re looking for and you leave the store empty handed. Generating realistic data means we must do the same in our simulated shopping trip – let people select “None of these” and leave without choosing anything. Otherwise, people may be “tricked” into selecting options they would never choose in real life.

 

Easy to read: Remember that conjoint is trying to simulate decisions that would normally happen in-store. Part of the in-store experience is in-store messaging. You’ll rarely see long sentences and paragraphs in the store so avoid them in your conjoint questions too. Use words and phrases that are as close as possible to what someone might see at the store.

 

cookiesUse imagery: We already know that a conjoint task can be cognitively demanding. That’s why imagery helps. Not only does it help people to visualize the product on the shelf amongst it’s competitive brands, it also helps to create a more visually appealing task (mmmmm cookies!). If you can’t provide an image of your product, find other ways to incorporate visuals in the questionnaire.

 

Plan for a hold-back sample: When product development work is extremely sensitive or is associated with life and death decisions, e.g., medical or pharmaceutical research, don’t let your budget determine the validity and rigor of your work. Spend the money to get the sample size you truly need to test each attribute and level with the appropriate rigor. And, build time into the fieldwork and data analysis schedule to permit preliminary analyses and test the model. You might need to tweak attributes, levels, or sets prior to running the full set of fieldwork.

 

Don’t let the statistics think for you: You wouldn’t create an entire marketing strategy based on gender differences just because a statistically significant t-test said 14% of women liked something and only 13% of men liked it. It’s not a meaningful difference. The same thing goes for a conjoint study. Review the model yourself, carefully, regardless of how “statistically significant” it is. Think about the various options suggested by the data. The simulator might reveal that there is a set of attributes and levels that would take over the market but that doesn’t mean you must produce that combination. The human brain is mightier than the spreadsheet!

 

If you’re curious to learn about the different types of conjoint that are available, this video from Sawtooth Software, presented by Aaron Hill, shares details about a few types of conjoint. E2E Research is pleased to offer all of these types to our clients.

 

 

 

What’s Next?

Are you ready to find out what configuration of your products and services consumers would be most keen to purchase? We’d be happy to help you work though the most suitable combinations of attributes and levels and build a conjoint study that meets your unique needs.

 

Please email your project specifications to our research experts using Projects at E2Eresearch dot com.

 

 

Learn more from our case studies

 

Learn more from our other blog posts