OPINION POLLS - IPSOS VIEWS - A continual improvement process
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
OPINION POLLS A continual improvement process February 2022 Henri Wallard IPSOS VIEWS
POLLING IN THE SPOTLIGHT THE “PROBLEM”
Political opinion polls come under great criticism and praise. After the election of Joe Political opinion polls are the public face of the Despite their informational value, internationally,
scrutiny in the run-up to elections, as we Biden as US President in 2020, The Atlantic entire research industry and are an important pre-election polls are also subject to criticism
try to make sense of often changing and published an article entitled “The Polling Crisis is source of information for the media, the public from government bodies who have sometimes
sometimes fragmenting political landscapes. a Catastrophe for American Democracy”.1 Yet, in and decision-makers. So a good understanding of restricted their practice and publication. Our
the wake of the 2019 British General Election, the their contribution is necessary. Our view at Ipsos international professional organisations (ESOMAR
Depending on how close they are to the outcome, BBC noted: “Opinion poll accuracy holds up”. 2
is that polls remain a vital tool for predicting and WAPOR) regularly review the freedom to
opinions of polls themselves can swing between So, in the end, what should we think? election outcomes and, importantly, this is not conduct opinion polls around the world.
only the perspective of polling professionals.
The practice of opinion polls brings with it
Electoral polling has been under a high level many challenges and over time it has faced
THE PURPOSE OF THIS PAPER of scrutiny for decades, not only from the countless questions about reliability. Here too
public but also from authorities and regulators. there has been a significant academic focus,
Our aim here is not only to show that polls execution – this feeds industry learnings and It has also been the subject of considerable with a large number of publications on survey
play a positive role in democracies as they improvements. However, systematic criticism independent academic work. techniques. In recent years, a new academic
deliver honest and independent measurement of polling undermines the value of their field has developed called computational social
of public opinion, but also to demonstrate contribution and risks throwing the baby out In the article “Improving election prediction research, which draws on the contribution of
that these tools are based on strong scientific with the bathwater. internationally” published in the journal Science, social networks and other digital sources. The
and technical bases. They can be challenged Kennedy et al. analysed more than 500 elections, academic publications often revolve around two
by open academic evaluations and are not The purpose of this paper is to offer an concluding that polls “provide a generally points: collection methods (quality of panels,
“black box” types of methods. Of course, overview and a gateway to the considerable accurate representation of likely election interviewing by telephone, via a PC, tablet or
polls must be fairly evaluated without scientific literature on the matter. So, outcomes and help us overcome the many biases smartphone) and adjustment techniques.
complacency after each election to identify ultimately, anyone can form their own opinion associated with human ‘gut feelings’”. 3
possible imperfections in their design and on…opinion polls. Detailed information is available from many
Similarly, in a paper published online in Nature sources – it should be emphasised here that
Human Behaviour, based on the analysis of the quality of the panels and the validation of
1,339 polls over 220 elections in 32 countries the responses obtained is closely monitored.
over a period of more than 70 years, Jennings For example, verification tools are used at many
and Wlezien show that there is no evidence that stages to eliminate suspect respondents (e.g.
poll error has increased over time, and that the those who answer too quickly, or give the same
performance of polls in very recent elections is no score to all the questions) from the responses
exception.4 Declining response rates and growing received.
variation in data collection mode, sampling and
weighting protocols have had little effect on the Of course, no collection is perfect, but the
performance of pre-election polls, at least when information is collected using elaborate
taken together. In fact, over the long term, polling technical platforms and quality control methods.
in the last week of election campaigns has even As for the adjustments, they are based on a
tended to be more accurate, demonstrating the
5
rigorous statistical theory designed to improve
importance of using “fresh” data. the estimators. To find out more on this topic,
2 IPSOS VIEWS | OPINION POLLS OPINION POLLS | IPSOS VIEWS 3
the interested reader can refer to Pascal Of course, errors made by single or multiple FINDING THE WAY FORWARD
Ardilly’s book Les Techniques de Sondages.6 polling organisations can spark debates about
In the end, it is not the method in itself that the reliability or validity of methodologies The task in hand is not to replace polling that This academic work describes well what we
should be criticised when the polls appear to used. So how is it possible for pollsters to get still gets it right in the vast majority of cases have also noticed in our professional experience
have lacked precision, but rather the practical it so wrong? One of the popular but erroneous with an entirely different approach, but to – the combination of sources enables us to
methods of implementation in a particular assumptions is that an opinion poll itself is not adapt these approaches using extreme rigour refine our findings. For instance, we may detect
election. the most appropriate methodology to capture in the implementation and incorporate fresh a certain dynamic in opinion by using social
public opinion. This leads to the search for new, approaches as they are needed. listening, which we can then incorporate into the
Furthermore, on closer inspection a significant modern “miracle methods”. design of surveys.
proportion of criticism of opinion polls can in The discussion that sets pollsters as so-called
fact be attributed to the interpretation of a poll’s When the sole use of these methods (e.g., supporters of “traditional methods” on one side What is important is that the method is based
findings, rather than the method itself. This has social media analysis) show themselves to be and social media or Big Data analysts as keen on solid theoretical ground, and that it is
led to many efforts around the world to facilitate successful in predicting outcomes in a given promoters of “new methods” on the other side implemented with enough care and precision.
their accurate interpretation by the media and election, this gives further fuel to the questioning is an unhelpful categorisation and does not So, while problems and inaccuracies may occur,
the public. of so-called “traditional” methods and the work reflect the reality. We cannot fall into the trap we can’t deny the foundations of the polling
of established polling organisations. of being overly reliant on evidence from isolated methods, and disposing of polling altogether
ESOMAR, the organisation that is the global incidents. Instead, we need to think about would deprive us of a valuable means of
voice for the research data and insights But while these “miracle methods” can be implementing the right method in each context. predicting election outcomes.
community, promotes professional ethical right in isolated elections, more often than not This is why we frequently use social media
standards and has developed a Code of Conduct they fall wide of the mark. The promise that analysis in addition to, and in combination with, After the experience of Brexit and the 2016 US
which all member organisations, including Ipsos, the difficulties we face in measuring voting polls and not as a substitution. elections, Ipsos conducted a thorough review of
must comply with. This illustrates that the intention can be resolved by a new methodology how it carries out polling and made some key
industry in general takes the task of conducting or tool is, frankly, misleading. It means that – On this subject we can refer to the analysis decisions on how we will operate differently in
opinion and electoral polling very seriously. especially in times of uncertainty and disruption made by Matthew Salganik, Professor of light of these learnings. In this paper, we reflect
– the exercise of care, modesty and validation is Sociology at Princeton University, in his book on these recent experiences and consider how
often forgotten. Bit by Bit: Social Research in the Digital Age: the practice of opinion polling is evolving in
“the abundance of big data sources increases – today’s volatile environment.
not decreases – the value of surveys”. 7
The discussion that sets pollsters as so-called
supporters of “traditional methods” on one side and
social media or Big Data analysts as keen promoters
of “new methods” on the other side is an unhelpful
categorisation and does not reflect the reality.
4 IPSOS VIEWS | OPINION POLLS OPINION POLLS | IPSOS VIEWS 5
SOME RECENT HISTORY:
POLLING IN THE REAR-VIEW MIRROR THE CORONAVIRUS EXPERIENCE
The Covid-19 pandemic provides a current information gaps and better understand
Widely considered a year of disruptive political methodological questions posed, brand new
and powerful example of how polling can motivations, for example on take-up of the
changes, 2016 saw the British public vote parties (one of whom topped the poll), low
make a real contribution to telling the coronavirus vaccines.
“Leave” in the EU referendum by a small turnout, and a lot of uncertainty: 32% told us
true story of what is happening on the
margin, followed by the election of Donald they might change their mind even in the very In Britain, the UK government drew on the
ground. Opinion polls have built a nuanced
Trump as President of the United States. In both final days before the poll, much higher than principles of good research practice during
understanding of the crisis, charting
cases, the outcomes were viewed as contrary to we normally see in general elections. All of its Covid-19 Home Testing programme which,
people’s experiences as the weeks became
what the polls had been predicting. this was set against a very volatile political by September 2020, had provided results
months. They have helped governments (and
backdrop. However, Ipsos’ final poll was very based on a representative sample of 600,000
businesses) get closer to how perceptions
Methods such as poll aggregation 8 (which accurate, getting the main story of the night people from across England. This major study
are changing over time, by population
made Nate Silver successful in the 2012 US right, with an average error of under one provided an accurate picture of how many
sub-group and between countries. Public
election) did not prove effective four years later percentage point – the most accurate of all the people had the coronavirus at any one time.9
health agencies have been able to quantify
and contributed to the general wave of “poll final polls released by members of the UK British
bashing” that then followed. Polling Council. This level of accuracy continued
at the subsequent December 2019 General
But, at the beginning of 2017, the accuracy Election.
of what the polls had predicted both for the
Dutch election and for the Presidential election One of the enduring roles of polls remains to
in France when compared to final results led ensure they are telling the story. At the 2021
commentators to switch back to praise of Canadian federal elections, we showed the
opinion polls. This turnaround was fuelled by public and our clients how our research can not
several factors. First, the Dutch and the French only predict what is going to happen but, more
election (first round) were considered difficult importantly, why it was happening.
ones for polls because they featured a wide
offer of political competitors combined with a It is the ability of Ipsos to tell this story of “why”
truly evolutionary climate of opinion. Second, and to provide a deeper understanding of the
the stakes were high in terms of the “risk” of voter numbers that we can be particularly proud
giving power to populist candidates. of, and this is what sets us apart and adds value
to our client work.
The arrival of completely new candidates and
parties to the political landscape represents
a challenge. We saw this in the UK European
elections in 2019. Predicting the results of
this election was a difficult exercise given the
One of the enduring roles of polls remains
to ensure that they are telling the story.
6 IPSOS VIEWS | OPINION POLLS OPINION POLLS | IPSOS VIEWS 7
CHOICES, CHOICES: METHODOLOGY MATTERS THINGS TO WATCH OUT FOR
Knowledge, experience and continual learnings enough to rest upon and replicate what has The potential sources of errors in polls are • The data collection tools used (telephone,
are central to polling performance. Opinion already been done. The market research well-known and have been the subject of online or mobile for instance) and the
surveys in general, and electoral polls in industry must continue to invest in scientific considerable expert discussion and academic impact of using them in combination.
particular, were originally designed on scientific progress and rigorous practice. scrutiny. They tend to relate to a handful of key
grounds and remain this way. But it is not issues such as: • The best way to analyse, weight and
filter the results. For example, polling
• Sampling: a fully representative spread of organisations weight the respondents once
THE POLLSTER’S TOOLKIT different types of voters (and non-voters) the survey is completed to compensate
needs to be interviewed. Special attention for some possible gaps with prior known
The choice of methods is the key question. Ipsos as an independent challenger or “referee” at all needs to be dedicated to the sources and information such as the results of past
uses a variety of techniques precisely because stages of the process. The referee makes sure the authenticity of respondents, making sure elections, or match the level of education
there is not one unique method that can latest learnings are applied by the local team and to eliminate bots or unreliable individuals. in the sample with that of the population
sufficiently answer all marketing and opinion any new lessons are captured and reported back. There has been considerable progress in at large.
research questions. Insights can be gained from the methods used to detect ineligible voters
behavioural economics, neuroscience, machine Through this process, the cross-examination of and in the deployment of countermeasures The 2020 US election provides a specific
learning, Big Data, and social media. These methods lets us apply our international footprint to ensure that these potential threats do not example of how context and election laws add
techniques have become mainstream practice in and accumulated knowledge and expertise from harm the validity of aggregated results. complexity to election polling. In the US the
many of our activities. elections around the world. But this is not to say global Covid-19 pandemic caused multiple
that one size fits all. Quite the contrary, in fact. • Volatility is growing everywhere, creating states to expand vote by mail options to
Each election needs to be taken as a special For example, the Australian system involves an increased need to identify what is minimise potential exposure for voters. At the
case and requires a rethink from A to Z in both compulsory voting with a completely different driving the dynamics of each campaign same time, one candidate and his party began
survey design and execution. This could mean parliamentary system to the United States. and to remain in field until the last possible making claims of fraud in postal voting and
that some categories of voters require special moment. encouraged his supporters to vote in person.
attention and more sampling, that the likely But, through looking at this topic through a This caused the mode people used to vote to
voter model needs adaptation, or that post- strictly international lens, we build a more • The potential impact of non-response rates. become highly correlated with who they planned
survey weighting requires different variables. In rounded understanding of the dynamics involved to vote for. In the US example, vote by mail was
any specific election, there needs to be a special in what we are trying to do. For example, large, • Questionnaire design including the perils decidedly Democratic while in-person voting
focus on where the real “high stakes” are. young or urban populations might require of leading questions or not asking the right was significantly more Republican. This meant
different combinations of techniques; turnout ones. that polls not only had to collect an accurate
Ipsos has moved from a rather localised process to may be quite volatile among certain groups, sample of the population, but also had to
a fully international approach, with the advantage including the so-called “left-behinds”. accurately reflect the distribution of vote by
of giving a greater number of observations of polls method.
and election results than is available in a single Techniques that can work well in some
country. A database of information from 500 countries, such as polling aggregation, don’t The potential sources of errors in polls
elections around the world informs an Ipsos “base
model” that allows us to compute probabilities of
work everywhere. So polling practitioners should
draw on all available tools, including social
are well-known and have been the subject
different parties’ success in elections. For each media, in order to come up with the best of considerable expert discussion and
election, an expert outside the local team acts approach every time. academic scrutiny.
8 IPSOS VIEWS | OPINION POLLS OPINION POLLS | IPSOS VIEWS 9
One central challenge today is to deploy the turnout. While it did drop from the 67% in 2019 What’s more, we often find that rises and Empirically, various models have been developed
right elements of the polling methods to model (down to 62.5%) more Canadians voted than falls in voter turnout are more pronounced to predict the turnout of the elections, derived
voter turnout. Overall levels of turnout are not many anticipated – but Ipsos turnout modelling among particular groups. Pollsters often from answers provided by respondents. In
always stable between one election and another. assisted in predicting an accurate turnout, and find themselves struggling to identify which an increasingly volatile political environment,
For example, the proportion voting at recent the implications on voter intentions. Elsewhere, segments of the population are going to show data must be collected as close as possible to
Canadian federal elections ranged from 58.8% in the UK, just 59.4% voted at the 2001 general up at a given time, in circumstances which election day to minimise the risk of missing last-
in 2008 to 68.3% in 2015. More recently, as election, the lowest since 1918. Fifteen years are often very different to what came before. minute switches in opinion.
the Canadian Federal Election of 2021 was later, 72.2% of the British electorate cast their For example, participation of 18-24 year olds
conducted during the Covid-19 pandemic, vote at the 2016 referendum on membership of in Canadian federal elections rose 17 points Polling is becoming more complicated as vote-
many pundits were predicting the lowest-ever the European Union. between 2011 and 2015. The 2019 election then switching becomes more common. There is a
saw a four-point decline in turnout. great need for well-chosen samples and well-
designed questions that enable us to understand
These are some of the methodological caveats the attitudes and patterns that lie behind voting
that must be continually monitored and adapted intention.
Figure 1: Voter turnout variation on a case-by-case basis to uphold the highest
levels of accuracy. And, as voters become more complicated,
multiple data sources and modes are needed to
reduce coverage error.
Pollsters need to identify carefully which segments of
the population are going to show up at a given time
in circumstances which are often very different to
what came before.
58.8% 68.3% 59.4% 72.2%
2008
Federal election
2015
Federal election
2001
General election
2016
EU Referendum
Source: Ipsos
10 IPSOS VIEWS | OPINION POLLS OPINION POLLS | IPSOS VIEWS 11
“NEW” METHODOLOGIES AND INNOVATIONS
To estimate a national popular vote, you must This enables us to have a better understanding
accurately: of the political situation which guides a better
design of the polls.
1. Poll the total population of eligible voters;
2. Estimate how many are going to show up; But, to pick up on an earlier observation, there
3. Estimate who is going to show up, i.e. the is also the temptation to say that outcomes
demographic and political composition of predicted correctly by social media methods
the voting public. provide proof of validity. This is where claims
are again misleading. The real validation is not
Emerging methods, such as Computational to have been right once, but to have enough
Social Intelligence, can be promising due cases where the validity of your method can be
to the fact that individuals now generate observed. It is certainly a useful tool, but alone it
numeric traces of virtually everything they do. is not enough and more work and testing needs
to be done to establish the right approach.
INNOVATIONS IN POLLING
• Sampling: We’ve found that diversity in • A multi-indicator approach: Social
sampling sources helps manage coverage Intelligence and Analytics tools allow us to
error, as different profiles of voters have detect signals of what is happening on the
a tendency to respond to different data ground. Sometimes it may be about picking
collection methods. Rolling samples and up early signs of change, which may be
longitudinal panels also enable us to better making little noise, at least initially. Other
understand volatility. times there is a more obvious dynamic
at play. Either way, social listening offers
• Behavioural science approach: In a given an invaluable wealth of information about
election context, elements such as the campaign dynamics. Additionally, final
uncertainty surrounding a specific election and population estimates of vote-share (such
the emotions felt about the act of voting are as the proportion of individuals voting
now incorporated into Ipsos’ turnout modelling. Democrat or Republican), can be generated
This approach enables us to better apprehend using multilevel modelling and machine
emotions (e.g., will they ‘regret’ their decisions learning techniques – leveraging individual
about whether to vote and which candidate data from polling and available aggregated
to vote for). Capturing voters’ emotions is data from small geographical areas. Such
something pollsters cannot neglect anymore modelling is now well established as part of
and drawing on perspectives from behavioural the pollster’s toolkit in many countries.
sciences is proving very productive.
12 IPSOS VIEWS | OPINION POLLS OPINION POLLS | IPSOS VIEWS 13CONCLUSION REFERENCES
1. https://www.theatlantic.com/ideas/archive/2020/11/polling-catastrophe/616986/
What causes confusion for many people outside This responsibility extends to taking a lead
the research industry is the sheer proliferation in encouraging good quality media reporting, 2. https://www.bbc.co.uk/news/election-2019-50777965
of polls before an election and the huge variance particularly in today’s era of “Fake News”. However
3. Kennedy, R., Wojcik, S., & Lazer, D. 2017. “Improving election prediction internationally.” Science
in the quality of the polling. These polls (some well-produced and accurate polling may be, it is
(New York, N.Y.), 355(6324), 515–520.
of them simply bogus) can skew forecasts along impossible to control the way it is presented via both
with public sentiment. The result: pollsters get a official media outlets and via the millions of online 4. Jennings, W., Wlezien, C. 2018. “Election polling errors across time and space.” Nature Human
bad rap, and people become even less likely to commentators on social media. Pollsters need to Behaviour 2, 276–283.
talk to professional pollsters. ensure they are always open and transparent about
5. Puleston, J. 2017. “Are we getting worse at political polling?” ESOMAR Congress. November 2017.
their methods, including setting out the limitations in
But, if you want to make some sense of the terms of what the poll is not able to do. 6. Ardilly, P. 2006. Les Techniques de Sondage. Paris. Editions Technip.
state of opinion at any moment in time, you
7. Salganik, M. J. 2019. Bit by Bit: Social Research in the Digital Age. (p.130). United Kingdom:
absolutely need polls. As Prof. Salganik from This paper has been developed very much in this
Princeton University Press.
Princeton University states, the proliferation of spirit and we are pleased to be involved in new
“big data sources increases - not decreases - the initiatives, such as the #HighQualityReporting 8. http://www.aapor.org/Education-Resources/Election-Polling-Resources/Poll-Aggregators.aspx
value of surveys”.6 This has led us to examine campaign recently launched in the UK,
9. https://www.ipsos.com/ipsos-mori/en-uk/latest-Covid-19-home-testing-results-reinforces-need- vigilance
the role that artificial intelligence, social media dedicated to better reporting of opinion polls
listening and alternative approaches can play in and election data in the media.10 10. http://www.mrs.org.uk/article/anytype/improving-reporting-of-surveys-opinion-polls-election-data-
pre-election research, as we search for more in-the-media-joint-mrs-impress-initiative
diverse solutions to assess people’s voting intent,
turnout, and ultimately actual vote. There is a big
responsibility to do this right.
The proliferation of big data sources FURTHER READING
increases the value of surveys. • https://www.political-atlas.com/
• https://onlinelibrary.wiley.com/doi/full/10.1002/rwm3.20589
• https://www.theguardian.com/politics/2019/sep/27/voters-so-promiscuous-the-pollsters-working-
to-predict-next-election
• Darrell Bricker, Ipsos CEO of Public Affairs. “You Have a Right to Know.” 24 November, 2020. A video
series designed for journalists on how public polls are conducted and how to evaluate poll integrity.
https://www.youtube.com/playlist?list=PLS2CHTim4DpG8oVxhTfUWV5sU5O9ZjNfB
• https://www.ipsos.com/en/political-and-public-opinion-polling
• The British Polling Council. 2020. “Opinion Polls: Guidance for Journalists”.
https://www.britishpollingcouncil.org/opinion-polls-guidance-for-journalists/
14 IPSOS VIEWS | OPINION POLLSOPINION POLLS Henri Wallard Deputy CEO and Chairman of Public Affairs, Ipsos The Ipsos Views white papers are produced by the Ipsos Knowledge Center. www.ipsos.com @Ipsos
You can also read
