Through the lens of the U.S. media, when political polls get it right, there is nothing to report. That said, it’s safe to say that after the last two presidential elections, the American people are less inclined to believe the results from political polls. The level and nature of scrutiny in the data is probably at its highest point in modern history.
In the 2016 presidential election, Donald Trump outperformed Hillary Clinton in several states, especially the Midwest and Pennsylvania, with high concentration of white voters without college degrees. However, in 2020, Trump technically lost, but beat national polls by more than three points nationally, and more so in key swing states. Polling was more accurate in the 2022 midterms, but general sentiment of the veracity of polling remains low. Polling, at its core, is about understanding the universe you’re measuring at a given point in time. Seasoned pollsters are spending a lot of time thinking about sampling and data collection methodologies because they need to be talking to the right people in the right proportions.
In our work with clients fielding polls, there is a lot of precision involved to ensure we are indeed speaking with the right people in the right proportions.
Conducting accurate and reliable polling research is no easy feat. It requires methodological rigor, sourcing expertise, ethical considerations, and a deep understanding of statistical principles. In today’s contentious political climate, rigor is the key word that must be applied as the data from polling continues to be scrutinized, whether fairly or unfairly.
Here are some best practices for ensuring the credibility and validity of political polling:
1. Random Sampling
The foundation of any reliable political poll is a representative sample of the population you’re looking to understand. Random sampling ensures every member of the population has an equal chance of being included in the survey, therefore minimizing bias. That’s why it’s critical to find the best partner to ensure the quality of the samples. The pollsters we partner with strive to obtain diverse samples, reflecting the demographic, geographic, psychographic, and socioeconomic diversity of the electorate. Without it, certain segments of the population could be over or underrepresented leading to bias that doesn’t accurately reflect the views of the electorate (or population they’re looking to better understand). It’s incumbent for companies like ours to give our clients as many details as possible on our sourcing to ensure their objectives will be met.
2. Audience targeting
Audience needs will vary depending upon the objective of the political study. When considering audience sourcing, it is best to work with a deeply profiled audience to get more specific with your targeting, if needed. Whether that’s working with political data services who provide granular detail on voters (party affiliation, voting history, etc.) or a panel source, it’s important to understand what is available to create the right sample plan, and that can only happen when your research partner has a deep understanding of your goals and objectives. Asking your audience source(s) what targeting is available is a great step in the planning stages. Sources should be able to consult with you on what is out there in terms of targeting that could help your efforts. For example, some of our clients have started to ask about what kind of behavioral targeting is available to reach a particular audience (i.e., what media respondents are engaged with, social media activity, news consumption, etc.).
3. Transparent Methodology
Polling organizations should be transparent in their methodologies, including sampling techniques, survey questions, and data collection methods. This should go without saying, but headline grabbing “polls” are frequently not the most transparent and would not stand up to scrutiny if their methodology was published. Without a trusted research partner, you risk having your methodologies questioned and rebuked. When those details are available, it provides journalists, consumers – even opponents – with the necessary context to properly frame the results. Transparency builds trust and allows stakeholders to assess the reliability of the findings. Detailed documentation and readily available access to the documentation of the polling process enhances the credibility of the results. Some of our clients go so far as disclosing raw data to allow independent researchers the ability to verify results and conduct their own analysis.
4. Avoiding Leading Questions and Keeping Respondents Engaged
The phrasing of survey questions can significantly influence respondents’ answers. Leading questions that intentionally or inadvertently steer respondents toward a particular response should be avoided at all costs. Questions should be neutral, clear, concise, and free from bias to elicit genuine opinions from participants. At ResponsivMR, we serve as an extra layer of QA for our polling clients to ensure questions aren’t leading. And, like any survey, it is important to look at the survey from the respondents’ POV. To help you achieve optimal respondent experience, ResponsivMR will keep an eye on length, keep questions engaging and relevant, ensure diversity in question type, make sure incentives are in line (or higher) than respondents are used to. An engaging survey further ensures quality data on the backend.
5. Accounting for Margin of Error
No poll is entirely free from error, but with a solid partner, the margin of error can be mitigated and can quantify the level of uncertainty inherent in survey results due to random sampling variation. Larger sample sizes = lower margins of error but needs to be aligned with other considerations. Regardless, pollsters should report the margin of error alongside their findings to provide a context for interpreting the results accurately.
6. Weighting and Adjustments (**beware)
While random sampling aims to achieve a representative sample, certain groups may be underrepresented due to factors such as low response rates or sampling biases. Pollsters may employ weighting techniques to adjust the sample composition and ensure it aligns with known population parameters. However, weighting should proceed with caution to avoid distorting the results. When applying weighting is necessary, selecting the best statisticians is critical. Most of our political polling clients already have these types of professionals in-house. For those who don’t, we provide that level of consultation to ensure the weighting techniques are sound.
7. Nonresponse Bias – Examination, learning and accounting for
Nonresponse bias occurs when certain groups of people are less likely to participate in surveys, leading to an unrepresentative sample. Many attribute nonresponse bias to the inaccuracies of polling leading up to the last two presidential elections. Learning how to limit bias moving forward, and learning from past mistakes, is paramount. Pollsters should assess and mitigate nonresponse bias through techniques such as follow-up surveys or demographic weighting. We can help you understand the reasons behind nonresponse, helping to improve your survey methodologies/design.
8. Ethical Considerations
It is incumbent on pollsters to conduct research ethically, respecting respondents’ privacy and confidentiality. Informed consent must be obtained, and personal information should be handled securely. ResponsivMR ensures the panel you’re sourcing from checks each of these boxes. Polling organizations should adhere to industry standards and guidelines to maintain integrity and credibility.
9. Continuous Monitoring + Evaluation
Political dynamics are constantly evolving, and public opinion can shift rapidly. Pollsters should continuously monitor and evaluate their methodologies to ensure relevance and accuracy. Regular review of sampling procedures, questionnaire design, and data analysis techniques is essential for maintaining the quality of polling research.
Political polling research is a complex and dynamic endeavor that requires careful planning, rigorous methodology, and ethical conduct. Working with a successful and seasoned partner will ensure that best practices, such as: random sampling, transparent methodology and continuous monitoring, are followed, so pollsters can produce reliable and credible data that contributes to a better understanding of public opinion and informs decision-making in the political sphere.
If you would like to learn more about how we can help you, drop us a line below.
Connor Duffey
Vice President, Client Services