Methodology Comparison

Polling accuracy is deeply linked to how pollsters reach voters. Good sampling can make the difference between a poll that tells the right story, and one that is fundamentally flawed.

Yet, for many campaigns and organizations, the differences between polling methodologies can be unclear.

Expert analysis of different methodologies oftentimes only differentiates between phone and online polling. But there are some basic differences even within those two categories that everyone who reads or uses polls should understand.

Polling Methodologies

Change Research

Change Research recruits respondents through a combination of online platforms, including digital ads, to participate in online polls. Change Research’s patent-pending Dynamic Online Sampling Engine delivers a cross-section of respondents that accurately reflects the electorate or population. It doesn’t offer financial incentives to respondents for participation.

Strengths

Change Research is fast, accurate, and affordable.

Change Research’s use of a combination of multiple platforms ensures that strengths and weaknesses are balanced. Change Research’s Dynamic Online Sampling engine allows for a balancing of demographics as the survey fields; additionally, Change Research is able to more easily and effectively reach voters under the age of 50 who rarely answer their phones.

Online Methods

CR_methodology compare website assets_text to web copy 3

Online Ad Invitation

Online Ad Invitation targets respondents through digital ads. These ads reach hundreds of thousands of voters in places they spend their time, allowing for a more representative sample.

Strengths

Affordably survey large sample sizes in a matter of days.

Reach a representative sample even in many small state legislative districts.

Test any sort of visual content ranging from images to videos.

Can field a survey quickly, rather than relying on slow and expensive call centers.

Weaknesses

Due to the targeting methodology, only 20-40% of respondents can be directly matched back to the voter file.

Text to Web

Text-to-web polling is a newer methodology that surveys voters by texting them a link to a survey which they then take online. Polling firms that traditionally fielded live phone polls have increasingly adopted text-to-web polling to supplement their samples.

Strengths

Text-to-web polls can be fielded quickly as compared to phone polling.

The percentage of voters who can be directly matched to the file is high because polls are often fielded by texting voters on a voter file.

Text-to-web can be helpful for reaching respondents who are younger and nonwhite.

Weaknesses

Many people still do not click on links sent from an unknown number.

As cell phone carriers and phone developers take active spam measures, these texts may be marked as spam.

Text-to-web polls must be sent one at a time, which can be a time-consuming and labor-intensive process.

CR_methodology compare website assets_panel copy 2

Online Survey Panels

Online panels are large universes of people that agree to join a panel of survey respondents, sometimes because the panel offers respondents incentives like money or gift cards. Online panels then invite certain participants to respond to certain surveys depending on their pre-identified demographics or other characteristics.

Strengths

You can survey the exact same person over time, which is beneficial for longitudinal studies.

Weaknesses

Most online panels do not have enough participants in small geographies to get a representative sample.

Because respondents are usually compensated, online panels have been found to have higher levels of “bogus” respondents than other methodologies.

Phone Polling Methods

CR_methodology compare website assets_ivr copy 2

Interactive Voice Response (IVR) Phone Polls

Interactive Voice Response (IVR) polls are automated calls where respondents give their answers by pressing corresponding numbers on the phone keyboard.

Strengths

IVR polls are typically less expensive and don’t depend on calling house centers.

Like live calling polls, most IVR polls are fielded by calling voters from a voter file, which means you can match respondents directly to a voter file.

Weaknesses

IVR polls have even lower response rates in many cases than live phone polling.

Voice response systems are prone to errors or issues, seem to encourage more false responses, and cannot legally call cell phones. IVR surveys are also illegal in some states.

The survey experience can be dreadful if many candidates and issues are being tested.

CR_methodology compare website assets_live phone copy 2

Live Phone Polls

In live phone polls, interviewers call a random sample of landlines and cell phones oftentimes usually from a list of voters (the voter file) and ask those who answer survey questions.

Strengths

You can match most survey respondents directly to a voter on voter file.

Weaknesses

The survey industry has been experiencing declining survey response rates for years. Respondents who do answer their phones to unknown numbers are arguably not a representative sample of any electorate.

Polls conducted through live phone polling tend to be expensive, take longer to field, and have a hard time reaching larger sample sizes.

Live phone polling also relies on calling house centers, which are prone to error and often book up during peak campaign season.

Speaking with an individual over the phone may make respondents more likely to answer questions in a “socially desirable” way that does not reflect their true views. This is especially true for culturally sensitive issues like race or abortion.