Modeling Framing in Immigration Discourse on Social Media
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Modeling Framing in Immigration Discourse on Social Media
Julia Mendelsohn Ceren Budak David Jurgens
University of Michigan University of Michigan University of Michigan
juliame@umich.edu cbudak@umich.edu jurgens@umich.edu
Abstract social media content enables us to compare framing
strategies across countries and political ideologies.
The framing of political issues can influence
Furthermore, social media provides unique insights
policy and public opinion. Even though the
into how messages resonate with audiences through
arXiv:2104.06443v1 [cs.CL] 13 Apr 2021
public plays a key role in creating and spread-
ing frames, little is known about how ordinary interactive signals such as retweets and favorites.
people on social media frame political issues. By jointly analyzing the production and reception
By creating a new dataset of immigration- of frames on Twitter, we provide an in-depth analy-
related tweets labeled for multiple framing sis of immigration framing by and on the public.
typologies from political communication the-
ory, we develop supervised models to detect Political communications research has identi-
frames. We demonstrate how users’ ideology fied numerous typologies of frames, such as issue-
and region impact framing choices, and how generic policy, immigration-specific, and narrative.
a message’s framing influences audience re-
Each of these frame types can significantly shape
sponses. We find that the more commonly-
used issue-generic frames obscure important the audience’s perceptions of an issue (Iyengar,
ideological and regional patterns that are only 1991; Chong and Druckman, 2007; Lecheler et al.,
revealed by immigration-specific frames. Fur- 2015), but prior NLP work seeking to detect frames
thermore, frames oriented towards human in- in mass media (e.g. Card et al., 2016; Field et al.,
terests, culture, and politics are associated with 2018; Kwak et al., 2020) has largely been limited
higher user engagement. This large-scale anal- to a single issue-generic policy typology. Multi-
ysis of a complex social and linguistic phe- ple dimensions of framing must be considered in
nomenon contributes to both NLP and social
order to better understand the structure of immi-
science research.
gration discourse and its effect on public opinion
1 Introduction and attitudes. We thus create a novel dataset of
immigration-related tweets containing labels for
Framing selects particular aspects of an issue and each typology to facilitate more nuanced computa-
makes them salient in communicating a message tional analyses of framing.
(Entman, 1993). Framing can impact how people
understand issues, attribute responsibility (Iyengar, This work combines political communication
1991), and endorse possible solutions, thus having theory with NLP to model multiple framing strate-
major implications for public opinion and policy de- gies and analyze how the public on Twitter frames
cisions (Chong and Druckman, 2007). While past immigration. Our contributions are as follows:
work has studied framing by the news media and (1) We create a novel dataset of immigration-
the political elite, little is known about how ordi- related tweets labeled for issue-generic policy,
nary people frame political issues. Yet, framing by immigration-specific, and narrative frames. (2) We
ordinary people can influence others’ perspectives develop and evaluate multiple methods to detect
and may even shape elites’ rhetoric (Russell Neu- each type of frame. (3) We illustrate how a mes-
man et al., 2014). To shed light on this important sage’s framing is influenced by its author’s ideol-
topic, we focus on one issue—immigration—and ogy and country. (4) We show how a message’s
develop a new methodology to computationally framing affects its audience by analyzing favorit-
analyze its framing on Twitter. ing and retweeting behaviors. Finally, our work
Our work highlights unique insights that social highlights the need to consider multiple framing
media data offers. The massive amount of available typologies and their effects.2 Framing in the Media news highlight region and ideology as particularly
important factors. Right-leaning media from con-
Framing serves four functions: (i) defining prob- servative regions are more likely to frame immi-
lems, (ii) diagnosing causes, (ii) making evaluative grants as intruders (van Gorp, 2005), and as threats
judgments, and (iv) suggesting solutions (Entman, to the economy and public safety (Fryberg et al.,
1993). Framing impacts what people notice about 2012). Framing also differs across countries; while
an issue, making it a key mechanism by which a the US press emphasizes public order, discrimina-
text influences its audience. tion, and humanitarian concerns, the French press
Framing Typologies We draw upon distinct ty- more frequently frames immigrants as victims of
pologies of frames that can be applied to the issue global inequality (Benson, 2013).
of immigration: (1) issue-specific, which identify Frame-setting has also been studied in the con-
aspects of a particular issue, or (2) issue-generic, text of immigration. For example, experimental
which appear across a variety of issues and facili- work has shown that frames eliciting angry or en-
tate cross-issue comparison (de Vreese, 2005). thusiastic emotions impact participants’ opinions
Issue-generic frames include policy frames that on immigration (Lecheler et al., 2015). While past
focus on aspects of issues important for policy- work has analyzed linguistic framing in Twitter im-
making, such as economic consequences or fair- migration discourse (e.g., de Saint Laurent et al.,
ness and equality (Boydstun et al., 2013). Other 2020), little is known about how such framing af-
generic frames focus on a text’s narrative; news fects users’ interactive behaviors such as resharing
articles use both episodic frames, which highlight content, which is a key objective of frame setting.
specific events or individuals, and thematic frames,
which place issues within a broader social context. 3 Computational Approaches to Framing
The use of episodic versus thematic frames can
influence the audience’s attitudes. For example, Because many people now generate and consume
episodic frames lead audiences to attribute respon- political content on social media, scholars have
sibility for issues such as poverty to individual citi- increasingly used automated techniques to study
zens while thematic frames lead them to hold the framing on social media.
government responsible (Iyengar, 1991). Large-scale research of framing on Twitter has
Issue-specific frames for immigration focus on commonly focused on unsupervised approaches.
the portrayal of immigrants. Our analysis uses (e.g., Russell Neuman et al., 2014; Meraz and Pa-
Benson (2013)’s set of issue-specific frames, which pacharissi, 2013; de Saint Laurent et al., 2020).
represent immigrants as heroes (cultural diversity, Such approaches, including those focused on hash-
integration, good workers), victims (humanitarian, tag analysis, can reveal interesting framing patterns.
global economy, discrimination), and threats (to For instance, Siapera et al. (2018) shows that frame
jobs, public order, taxpayers, cultural values). usage varies across events. Similarly, topic models
Both issue-specific and generic frames provide have been used to compare “refugee crisis" media
unique insights but present advantages and draw- discourses across the European countries (Heiden-
backs. While issue-specific frames analysis are reich et al., 2019), and to uncover differences in
specific and detailed, they are hard to generalize attitudes towards migrants (Hartnett, 2019). Al-
and replicate across studies, which is a key advan- though lexicon analysis and topic models can pro-
tage for generic frames (de Vreese, 2005). vide insights about immigration discourse, here, we
Framing effects Studies of framing typically adopt a supervised approach to ground our work in
focus on either frame-building or frame- framing research and to enable robust evaluation.
setting (Scheufele, 1999; de Vreese, 2005). We draw inspiration from a growing body of
Frame-building is the process by which external NLP research that uses supervised approaches to
factors, such as a journalist’s ideology or eco- detect issue-generic policy frames in news arti-
nomic pressures, influence what frames are used; cles, a task popularized by the Media Frames
frame-building studies thus treat framing as the Corpus (Card et al., 2015), which contains issue-
dependent variable. Frame-setting studies treat generic policy frame labels for articles across sev-
frames as independent variables that impact how eral issues (Boydstun et al., 2013). Using this
an audience interprets and evaluates issues. corpus, prior work has detected frames with tech-
Prior analyses of frame-building in immigration niques including logistic regression (Card et al.,Frame Type Frame Description
Issue-Generic Economic Financial implications of an issue
Policy Capacity & Resources The availability or lack of time, physical, human, or financial resources
Morality & Ethics Perspectives compelled by religion or secular sense of ethics or social responsibility
Fairness & Equality The (in)equality with which laws, punishments, rewards, resources are distributed
Legality, Constitutionality Court cases and existing laws that regulate policies; constitutional interpretation;
& Jurisdiction legal processes such as seeking asylum or obtaining citizenship; jurisdiction
Crime & Punishment The violation of policies in practice and the consequences of those violations
Security & Defense Any threat to a person, group, or nation and defenses taken to avoid that threat
Health & Safety Health and safety outcomes of a policy issue, discussions of health care
Quality of Life Effects on people’s wealth, mobility, daily routines, community life, happiness, etc.
Cultural Identity Social norms, trends, values, and customs; integration/assimilation efforts
Public Sentiment General social attitudes, protests, polling, interest groups, public passage of laws
Political Factors & Focus on politicians, political parties, governing bodies, political campaigns
Implications and debates; discussions of elections and voting
Policy Prescription &
Discussions of existing or proposed policies and their effectiveness
Evaluation
External Regulation & Relations between nations or states/provinces; agreements between governments;
Reputation perceptions of one nation/state by another
Immigration Victim: Global Economy Immigrants are victims of global poverty, underdevelopment and inequality
Specific Victim: Humanitarian Immigrants experience economic, social, and political suffering and hardships
Victim: War Focus on war and violent conflict as reason for immigration
Victim: Discrimination Immigrants are victims of racism, xenophobia, and religion-based discrimination
Hero: Cultural Diversity Highlights positive aspects of differences that immigrants bring to society
Hero: Integration Immigrants successfully adapt and fit into their host society
Hero: Worker Immigrants contribute to economic prosperity and are an important source of labor
Threat: Jobs Immigrants take nonimmigrants’ jobs or lower their wages
Threat: Public Order Immigrants threaten public safety by being breaking the law or spreading disease
Threat: Fiscal Immigrants abuse social service programs and are a burden on resources
Threat: National Cohesion Immigrants’ cultural differences are a threat to national unity and social harmony
Narrative Episodic Message provides concrete information about on specific people, places, or events
Thematic Message is more abstract, placing stories in broader political and social contexts
Table 1: List of all issue-generic policy (Boydstun et al., 2013), immigration-specific (Benson, 2013; Hovden and
Mjelde, 2019), and narrative (Iyengar, 1991) frames with brief descriptions.
2016), recurrent neural networks (Naderi and Hirst, ied issue-specific frames in news media for issues
2017), lexicon induction (Field et al., 2018), and such as missile defense and gun violence (Morstat-
fine-tuning pretrained language models (Khane- ter et al., 2018; Liu et al., 2019a). We extend issue-
hzar et al., 2019; Kwak et al., 2020). Roy and specific frame analyses to immigration by adopting
Goldwasser (2020) further extracted subcategories an immigration-specific typology developed by po-
of issue-generic policy frames in newspaper cover- litical communication scholars (Benson, 2013).
age using a weakly-supervised approach. Finally, In contrast to prior NLP work focused on tradi-
issue-generic frames have also been computation- tional media or political elites (Johnson et al., 2017;
ally studied in other media, including online fora Field et al., 2018), we highlight the role that social
and politicians’ tweets (Johnson et al., 2017; Hart- media publics play in generating and propagating
mann et al., 2019). We build upon this literature by frames. Furthermore, we provide a new computa-
incorporating additional frame typologies that re- tional model of narrative framing (Iyengar, 1991),
flect important dimensions of media discourse with that together with models for issue-generic policy
real-world consequences (Iyengar, 1991; Gross, and issue-specific frames, provides complementary
2008; Eberl et al., 2018). Beyond detecting frames, views on the framing of immigration. Finally, our
we computationally analyze frame-building and large-scale analysis of frame-setting illustrates the
frame-setting among social media users; though potential for using NLP to understand how a mes-
well-studied in traditional news media, little is sage’s framing shapes its audience behavior.
known about how social media users frame im-
migration or its effects (Eberl et al., 2018). 4 Data
Noting that issue-generic policy frames obscure We first collect a large dataset of immigration-
important linguistic differences, several works stud- related tweets, and then annotate a subset of thisfull dataset for multiple types of frames. influence an issue’s framing. For simplicity, we
Data Collection We extract all English-language treat each tweet as a standalone message and label
tweets in 2018 and 2019 from the Twitter Decahose frames based only on the text (including hashtags).
containing at least one of the following terms: im- Unlike news stories, where frames are clearly
migration, immigrant(s), emigration, emigrant(s), cued, tweets often implicitly allude to frames due
migration, migrant(s), illegal alien(s), illegals, and to character limitations. For example, a tweet ex-
undocumented1 . We focus on content creation and pressing desire to “drive immigrants out" with no
thus exclude retweets from our dataset, though we additional context may suggest a criminal frame,
consider retweeting rates when analyzing the social but criminality is not explicit. To minimize er-
influence of different frames. We further restrict rors, we avoid making assumptions about intended
our dataset to tweets whose authors are identified meaning and interpret all messages literally.
as being located in the United States (US), United Training, development, and test data were anno-
Kingdom (GB), and European Union (EU) by an tated using two procedures after four annotators
existing location inference tool (Compton et al., completed four rounds of training. The dataset
2014). To compare framing across political ide- contains equal numbers of tweets from the EU,
ologies, we obtain ideal point estimates for nearly UK, and US. Training data was singly annotated
two-thirds of US-based users with Barberá (2015)’s and includes 3,600 tweets, while the development
Bayesian Spatial Following model. Our full dataset and test sets each contain 450 tweets (10% of the
contains over 2.66 million tweets, 86.2% of which full dataset) and were consensus-coded by pairs
are from the United States, 10.4% from the United of trained annotators. We opt for this two-tier ap-
Kingdom, and 3.4% from the European Union. proach due to (i) the inherent difficulty of the task2
Data Annotation Tweets are annotated using three and (ii) the need to maximize diversity seen in
frame typologies: (i) issue-generic policy, (ii) training. During annotator training, pilot studies
immigration-specific, and (iii) narrative frames, attained moderate agreement, suggesting that to
where a tweet may use multiple frames simulta- attain high-reliability, consensus coding with ad-
neously. We use Boydstun et al. (2013)’s Policy judication would be needed (Krippendorff, 2013),
Frames Codebook to formulate our initial guide- which comes at a cost of substantially increased
lines to code for policy frames. We use Benson time. Because a large dataset of unique, singly-
(2013)’s immigration-specific frames, but follow coded documents is preferable to a small dataset
Hovden and Mjelde (2019) in including an addi- of documents coded by multiple annotators for text
tional category for framing immigrants as victims classification (Barbera et al., 2021), we decided
of war. Finally, we code for narrative frames using to increase corpus diversity in the training data
definitions from Iyengar (1991). All frames and de- by singly-annotating, at the expense of potentially
scriptions can be found in Table 1, with a complete noisier annotation, and to consensus code all eval-
codebook in Supplementary Materials. Because uation data. On the double annotated data, anno-
annotation guidelines from prior work focus on tators attained Krippendorff’s α=0.45. Additional
elite communications, we first adjusted our code- details are provided in Supplementary Material (§B,
book to address challenges posed by Twitter con- Figures 6 and 7).
tent. Changes were made based on feedback from Results We observe differences across frame ty-
four trained annotators who labeled 360 tweets pologies in coverage rates within the annotated
from 2018, split between the EU, GB, and US. data set. While 84% of tweets are labeled with
Even for humans, identifying frames in tweets at least one issue-generic policy frame and 85%
is a difficult task. Defining the boundaries of what with at least one narrative frame, only 51% are la-
constitutes a message is not trivial. Beyond the beled with at least one issue-specific frame. This
text, frames could be identified in hashtags, images, difference is due to immigration-specific frames
videos, and content from linked pages. Further- being more narrowly-defined, as they require ex-
more, tweets are often replies to other users or part plicit judgment of immigrants as heroes, victims,
of a larger thread. This additional context may or threats. Further details about frame distributions
1 2
We obtained this list by starting with the seed terms im- For example, in identifying just the primary issue-generic
migrants, immigration, and illegal aliens. We then added the frame of a document, the Media Frames corpus attained an
remaining terms by manually inspecting and filtering nearby Krippendorff’s α=∼0.6 (Card et al., 2015, Fig. 4), whereas we
words in pretrained GloVe and Word2Vec vector spaces. ask annotators to identify all frames across three typologies.Random LogReg RoBERTa FT RoBERTa tion discourse on social media in order to capture
0.193 0.296 0.611 0.657 diverse perspectives and arguments.
Table 2: F1 scores on the test set for all models, cal- Table 3 shows several evaluation metrics sepa-
culated as an (unweighted) average over all frames rated by frame type. Precision, recall, and F1 are
and initialization seeds. The fine-tuned (FT) RoBERTa calculated as unweighted averages over all frames
model improvements over all models are significant at belonging to each category. Overall, issue-generic
pError Type Description Example
These instances highlight the challenges of annotation; Interestingly, the criteria to which immigrants would be held would
Plausible interpretation there are convincing arguments that model’s predicted not be met by a large number of the ‘British’ people either.
frames can be appropriate labels. Model erroneously predicted Policy
Inferring frames not Model predicts frames that may capture an author’s intention Stop immigration
explicitly cued in text but without sufficient evidence from the text Model erroneously predicted Threat: Public Order
@EricTrump Eric I have been alive longer than your immigrant
Some frames are directly cued by lexical items
Missing necessary mother in law and you. I paid more in taxes than you did and
(e.g. politicians’ names cue Political frame), but model
contextual knowledge your immigrant mother in law combined...
lacks real-world knowledge required to identify these frames
Model missed Political frame
Many words and phrases do not directly cue frames, but are Lunaria’s figures from 2018 recorded 12 shootings, two murders
Overgeneralizing highly-correlated. The model makes erroneous predictions and 33 physical assaults against migrants in the first two months
highly-correlated features when such features are used in different contexts (e.g. violence since Salvini entered government.
against immigrants, rather than immigrants being violent) Model missed Victim: Humanitarian frame
It’s worse when you have immigrant parents who don’t speak
Coreference resolution is often not possible and annotators avoided
the language cause you have to deal with all the paperwork,
making assumptions to resolve ambiguities. For example, "you"
Pronoun ambiguity be the translator for them whenever they go (...)
can be used to discuss individuals’ experiences (episodic) but its
its tiring but someone has to
impersonal sense can be in broad generalizations (thematic).
Model predicted Episodic but referent is unclear
Table 4: Types of common errors in frame prediction along with brief descriptions and examples.
frame detection to achieve higher performance on tic norms (Papacharissi and De Fatima Oliveira,
conservative tweets due to more linguistic regular- 2008), geographic proximity to immigrant pop-
ities across messages. Indeed, we find that issue- ulations or points of entry (Grimm and And-
generic and issue-specific classifiers achieve higher sager, 2011; Fryberg et al., 2012), and immigrants’
F1 scores on tweets written by conservative authors race/ethnicity (Grimm and Andsager, 2011). At the
compared to liberal authors (Figure 1), even though same time, increased globalization may result in a
there are fewer conservative tweets in the train- uniform transnational immigration discourse (Hel-
ing data (334 conservative vs 385 liberal tweets). bling, 2014). Framing variations across countries
Higher model performance on conservative tweets has implications for government policies and ini-
suggests that, like political and media elites, con- tiatives, particularly in determining what solutions
servatives on social media are more consistent than could be applied internationally or tailored to each
liberals in their linguistic framing of immigration. country (Caviedes, 2015).
Error Analysis We identify classification errors Prior studies on the role of ideology in frame-
by qualitatively analyzing a random sample of 200 building have focused on the newspapers or politi-
tweets that misclassified at least one frame. Table cal movements, showing patterns in frames like
4 shows the most common categories of errors. morality and security by political affiliation in
European immigration discourse (Helbling, 2014;
6 Frame-Building Analysis Hogan and Haltinner, 2015) or in use of economic
In writing about an issue, individuals are known frames by American newspapers (Fryberg et al.,
to select particular frames—a process known as 2012; Abrajano et al., 2017). However, it remains
frame-building—based on numerous factors, such unclear whether these patterns observed for elite
as exposure to politicians’ rhetoric or their own groups can generalize to the effect of individual
identity (Scheufele, 1999). Here, we focus on two people’s political dispositions.
specific identity attributes affecting frame building: Experimental Setup We detect frames for all
(i) political ideology and (ii) country/region. 2.6M immigration-related tweets using the fine-
The political, social, and historical contexts of tuned RoBERTa model with the best-performing
an one’s nation-state can impact how they frame seed on development data. Using this labeled data,
immigration (Helbling, 2014). Immigration has we estimate the effects of region and ideology by
a long history in the USA relative to Europe, fitting separate mixed-effects logistic regression
and former European colonial powers (e.g. the models to predict the presence or absence of each
UK) have longer immigration histories than other frame. We treat region (US, UK, and EU) as a
countries (e.g. Norway) (Thorbjørnsrud, 2015; categorical variable, with US as the reference level.
Eberl et al., 2018). Cross-country variation in Ideology is estimated using the method of Barberá
news framing also arise from differences in im- (2015), which is based on users’ connections to US
migration policies (Helbling, 2014; Lawlor, 2015), political elites; as such, we restrict our analysis of
media systems (Thorbjørnsrud, 2015), journalis- ideology to only tweets from the United States.Frame Type cus on frames with implications for the in-group.
Issue-Specific Issue-Generic Narrative They express concerns about 1) immigrants im-
Liberal Conservative posing a burden on taxpayers and governmental
programs and 2) immigrants being criminals and
Victim: Humanitarian
Hero: Cultural Diversity threats to public safety. We qualitatively observe
Victim: Discrimination
Victim: War three distinct, though unsubstantiated, conservative
Morality & Ethics claims contributing to the latter: (i.) Immigrants
Hero: Worker
Hero: Integration commit violent crimes (Light and Miller, 2018),
Episodic
Fairness & Equality (ii.) Undocumented immigrants illegally vote in
Quality of Life
Cultural Identity US elections (Smith, 2017; Udani and Kimball,
Public Sentiment
Victim: Global Economy 2018), and (iii.) Immigrants are criminals simply
Health & Safety by virtue of being immigrants (Ewing et al., 2015).
Policy Prescription
Economic
Political Factors Figure 2 shows a clear ideological stratification
External Regulation for issue-specific frames: liberals favor hero and
Threat: Jobs
Crime & Punishment victim frames, while conservatives favor threat
Security & Defense
Capacity & Resources frames. This finding is consistent with prior work
Thematic
Threat: National Cohesion on the role perceived threats play in shaping white
Threat: Fiscal
Threat: Public Order American attitudes towards immigration (Brader
et al., 2008), and the disposition of political conser-
0.5 0.0 0.5 vatism to avoid potential threats (Jost et al., 2003).
Coefficient
Second, while all frame categories show ide-
Figure 2: Logistic regression coefficients of political ological bias, issue-specific frames are the most
ideology in predicting each frame. Positive (negative) extreme. Most notably, our analysis shows that fo-
values correspond to more conservative (liberal) ideol- cusing solely on issue-generic policy frames would
ogy. Only frames associated with ideology after Holm-
obscure important patterns. For example, the issue-
Bonferroni correction with p < 0.01 are included.
generic cultural identity frame shows a slight lib-
eral bias; yet, related issue-specific frames diverge:
To account for exogenous events that may im- hero: cultural diversity is very liberal while threat:
pact framing, we include nested random effects for national cohesion is very conservative.
year, month, and date. We further control for user Similarly, the issue-generic economic policy
characteristics (e.g. the author’s follower count, frame is slightly favored by more conservative au-
friends count, verified status and number of prior thors, but the related issue-specific frames threat:
tweets) as well as other tweet characteristics (e.g. jobs and hero: worker reveal ideological divides.
tweet length, if a tweet is a reply, and whether This finding highlights the importance of using mul-
the tweet contains hashtags, URLs, or mentions tiple framing typologies to provide a more nuanced
of other users). We apply Holm-Bonferroni cor- analysis of immigration discourse.
rections on p-values before significance testing to Third, more liberal authors tend to use episodic
account for multiple hypothesis testing. frames, while conservative authors tend to use the-
Ideology Ideology is strongly predictive of fram- matic frames. This difference is consistent with
ing strategies in all three categories, as shown in Somaini (2019)’s finding that a local liberal news-
Figure 2. Our results reveal three broad themes. paper featured more episodic framing in immigra-
First, prior work has argued that liberals and tion coverage, but a comparable conservative news-
conservatives adhere to different moral founda- paper featured more thematic framing. Other ef-
tions, with conservatives being more sensitive to forts that examine the relationship between narra-
in-group/loyalty and authority than liberals, who tive frames and cognitive and emotional responses
are more sensitive to care and fairness (Graham provide some clues for the observed pattern. For
et al., 2009). Our results agree with this argument. instance, Aarøe (2011) shows that thematic frames
Liberals are more likely to frame immigration as are stronger when there are no or weak emotional
a fairness and morality issue, and immigrants as responses; and that the opposite is true for episodic
victims of discrimination and inhumane policies. frames. The divergence of findings could be driven
More conservative authors, on the other hand, fo- by partisans’ differing emotional responses. Ourfindings also highlight important consequences for other factors, interact in intricate ways to shape
opinion formation. Iyengar (1990) shows that how ordinary people frame political issues.
episodic framing diverts attention from societal and Second, cultural identity is more strongly associ-
political party responsibility; our results suggest ated with both the UK and EU than the US. Perhaps
that liberal Twitter users are likely to produce (and, immigrants’ backgrounds are more marked in Eu-
due to partisan self-segregation, consume) social ropean discourse than in US discourse because the
media content with such effects. UK and EU have longer histories of cultural and
ethnic homogeneity (Thorbjørnsrud, 2015). This
Frame Type finding also reflects that Europeans’ attitudes to-
Issue-Specific Issue-Generic Narrative
wards immigration depend on where immigrants
USA EU are from and parallels how European newspapers
Threat: Public Order
Crime & Punishment frame immigration differently depending on mi-
Threat: Fiscal
Political Factors
Threat: Jobs grants’ countries of origin (Eberl et al., 2018).
Security & Defense
Thematic Finally, the bottom of Figure 3 shows that users
Economic
Morality & Ethics from the UK are more likely to invoke labor-related
Policy Prescription
Episodic frames. This prevalence of labor and economic
Hero: Integration
Capacity & Resources frames has also been found in British traditional
Health & Safety
Fairness & Equality
Victim: Humanitarian media (Caviedes, 2015; Lawlor, 2015), and has
Threat: National Cohesion
Hero: Worker been attributed to differences in the labor mar-
Victim: War
Cultural Identity ket. Unlike migrants in the US, Italy, and France,
Victim: Global Economy
External Regulation who often work clandestinely in different economic
USA UK sectors than domestic workers, UK migrants have
Threat: Public Order proper authorization and are thus viewed as com-
Crime & Punishment
Morality & Ethics
Security & Defense petition for British workers because they can work
Victim: Humanitarian
Threat: Fiscal in the same industries (Caviedes, 2015).
Health & Safety
Political Factors
Episodic
Hero: Integration
Threat: National Cohesion 7 Audience Response to Frames
Economic
Quality of Life
Victim: War
Fairness & Equality Chong and Druckman (2007, p. 116) assert that
Public Sentiment
Policy Prescription a “challenge for future work concerns the identifi-
Capacity & Resources
Threat: Jobs
Victim: Discrimination cation of factors that make a frame strong.” Stud-
Cultural Identity
External Regulation ies of frame-setting—i.e., how a message’s fram-
Victim: Global Economy
Hero: Worker ing affects its audience’s emotions, beliefs, and
1 0 1 opinions—have largely been restricted to small-
Coefficient
scale experimental studies because responses to
Figure 3: Effect of author being from the EU (top) or news media framing cannot be directly observed
the UK (bottom) relative to the US. Frames with posi- (Eberl et al., 2018). However, Twitter provides
tive β coefficients are associated with authors from the insight into the frame-setting process via interac-
EU (top) and UK (bottom), and frames with negative tive signals: favorites and retweets. While related,
values are associated with US-based authors. Frames these two actions can have distinct underlying mo-
not significantly associated with region after Holm- tivations: favoriting often indicates positive align-
Bonferroni correction are not included.
ment between the author and the reader; in contrast,
retweeting may also be driven by other motivations,
Region Immigration framing depends heavily on such as the desire to inform or entertain others
one’s geopolitical entity (US, UK, and EU), as (boyd et al., 2010). Different audience interactions
shown in Figure 3. Several notable themes emerge. have been shown to exhibit distinct patterns in polit-
First, many ideologically-extreme frames in the ical communication on Twitter (Minot et al., 2020).
US, including crime & punishment, security & de- Here, we test how a message’s framing impacts
fense, threat: public order, and threat: fiscal are all both the favorites and retweets that it receives.
significantly more likely to be found in US-based Experimental Setup We fit hierarchical linear
tweets relative to the UK and EU. This pattern sug- mixed effects models with favorites and retweets
gests that region and ideology, and likely many (log-transformed) as the dependent variable on USResponse due to their increased emotional appeal to readers
Favorite Retweet (Semetko and Valkenburg, 2000).
Morality & Ethics On the other hand, political factors & impli-
Fairness & Equality
Public Sentiment cations is most highly associated with increased
Cultural Identity retweets. As the political frame emphasizes compe-
Quality of Life
Economic tition and strategy (Boydstun et al., 2013), this re-
Political Factors
Policy Prescription sult mirrors similar links between the “horse-race"
Health & Safety frame in news reports and engagement (Iyengar
Security & Defense
Capacity & Resources et al., 2004); users may prefer amplifying political
Crime & Punishment
External Regulation messages via retweeting to help their side win.
Hero: Integration
Hero: Cultural Diversity Similarly, frames about security and safety (e.g.
Victim: Discrimination crime & punishment, victim: humanitarian) are
Victim: Humanitarian
Victim: Global Economy highly associated with more retweets, but not nec-
Threat: Public Order essarily favorites. While security and safety frames
Threat: Jobs
Threat: National Cohesion may not lead audience members to endorse such
Threat: Fiscal
Thematic messages, perhaps they are more likely to amplify
Episodic these messages due to perceived urgency or the
0.05 0.00 0.05 0.10 desire to persuade others of such concerns.
Change in (log) responses Finally, Figure 4 shows how a message’s narra-
tive framing impacts audience response, even after
Figure 4: Effects of framing on two audience responses: controlling for all other frames. Both episodic and
favorites and retweets. The x-axis shows regression co-
thematic frames are significantly associated with
efficients for the presence of each frame in predicting
the log-scaled number of responses. Along the y-axis increased engagement (retweets), but less strongly
are all issue-generic policy frames (top), immigration- than issue frames. Having a clear narrative is im-
specific frames (middle), and narrative frames (bottom) portant for messages to spread, but the underlying
that are significantly associated with either the number mechanisms driving engagement behaviors may
of favorites or retweets. differ for episodic and thematic frames; prior work
on mainstream media has found that news stories
using episodic frames tend to be more emotion-
tweets with detected author ideology. The presence ally engaging, while thematic frames can be more
of a frame is treated as a binary fixed effect. We persuasive (Iyengar, 1991; Gross, 2008).
control for all temporal, user-level and tweet-level
features as in the prior section, as well as ideology.
8 Conclusion
Results The framing of immigration has a signifi-
cant impact on how users engage with the content Users’ exposure to political information on social
via retweets and favorites (Figure 4). Many issue- media can have immense consequences. By lever-
specific frames have a stronger effect on audience aging multiple theory-informed typologies, our
responses than either of the other typologies. As computational analysis of framing enables us to
recent NLP approaches have adopted issue-generic better understand public discourses surrounding
frames for analysis (e.g., Kwak et al., 2020), the immigration. We furthermore show that framing
strength of issue-specific frames highlights the im- on Twitter affects how audience interactions with
portance of expanding computational analyses be- messages via favoriting and retweeting behaviors.
yond issue-generic frames, as other frames may This work has implications for social media plat-
have larger consequences for public opinion. forms, who may wish to improve users’ experi-
Most frames impact favorites and retweets dif- ences by enabling them to discover content with
ferently, suggesting that the strength of a frame’s a diversity of frames. By exposing users to a
effects is tied to the specific engagement behav- wide range of perspectives, this work can help lay
ior. Cultural frames (e.g. hero: integration) and foundations for more cooperative and effective on-
frames oriented around human interest (e.g. moral- line discussions. All code, data, annotation guide-
ity, victim: discrimination) are particularly associ- lines, and pretrained models are available at https:
ated with more endorsements (favorites), perhaps //github.com/juliamendelsohn/framing.9 Ethical Considerations References
Our analysis of frame-building involves inferring Lene Aarøe. 2011. Investigating Frame Strength: The
Case of Episodic and Thematic Frames. Political
political ideology and regional from users with ex- Communication, 28(2):207–226.
isting tools, so we aggregated this information in
our analysis in order to minimize the risk of ex- Marisa A Abrajano, Zoltan Hajnal, and Hans J.G.
Hassell. 2017. Media Framing and Partisan Iden-
posing potentially sensitive personal data about tity: The Case of Immigration Coverage and White
individuals. Our dataset includes tweet IDs along Macropartisanship.
with frame labels, but no additional social informa-
Pablo Barberá. 2015. Birds of the same feather tweet
tion. However, there are also ethical consequences together: Bayesian ideal point estimation using Twit-
of categorizing people along these social dimen- ter data. Political Analysis, 23(1):76–91.
sions. We acknowledge that reducing people’s so-
Pablo Barbera, Amber E Boydstun, Suzanna Linn,
cial identities to region and ideology obscures the Ryan McMahon, and Jonathan Nagler. 2021. Auto-
wide range of unobservable and non-quantifiable mated text classification of news articles: A practical
predispositions and experiences that may impact guide. Political Analysis, 29(1):19–42.
framing and attitudes towards immigration. Rodney Benson. 2013. Shaping Immigration News: A
We emphasize that our dataset is not fully repre- French-American Comparison. Cambridge Univer-
sentative of all immigration discourse and should sity Press.
not be treated as such. Twitter’s demographics are danah boyd, Scott Golder, and Gilad Lotan. 2010.
not representative of the global population (Mis- Tweet, tweet, retweet: Conversational aspects of
love et al., 2011). Furthermore, our dataset only in- retweeting on Twitter. In 2010 43rd Hawaii Interna-
cludes tweets with authors from particular Western tional Conference on System Sciences, pages 1–10.
IEEE.
countries. All tweets were automatically identified
by Twitter as being written in English, thus ad- Amber E. Boydstun, Justin H Gross, Philip Resnik, and
ditionally imposing standard language ideologies Noah A. Smith. 2013. Identifying Media Frames
and Frame Dynamics Within and Across Policy Is-
on the data that we include (Milroy, 2001). Fur- sues. New Directions in Analyzing Text as Data
thermore, language choice itself can be a socially Workshop, pages 1–13.
and politically meaningful linguistic cue that may
Ted Brader, Nicholas A. Valentino, and Elizabeth
have unique interactions with framing (e.g., Gal, Suhay. 2008. What triggers public opposition to
1978; Shoemark et al., 2017; Stewart et al., 2018; immigration? Anxiety, group cues, and immigra-
Ndubuisi-Obi et al., 2019). tion threat. American Journal of Political Science,
Although we do not focus on abusive language, 52(4):959–978.
our topical content contains frequent instances of Dallas Card, Amber E Boydstun, Justin H Gross, Philip
racism, Islamophobia, antisemitism, and personal Resnik, and Noah A Smith. 2015. The media frames
insults. We caution future researchers about poten- corpus: Annotations of frames across issues. In
53rd Annual Meeting of the Association for Compu-
tially traumatic psychological effects of working tational Linguistics and the 7th International Joint
with this dataset. Conference on Natural Language Processing, vol-
We aim to support immigrants, an often ume 2, pages 438–444.
marginalized group, by shedding light on their rep- Dallas Card, Justin H Gross, Amber E Boydstun, and
resentation on social media. However, there is a Noah A Smith. 2016. Analyzing framing through
risk that malicious agents could exploit our frame- the casts of characters in the news. In Proceedings of
setting findings by disseminating harmful content the 2016 Conference on Empirical Methods in Natu-
ral Language Processing, pages 1410–1420.
packaged in more popular frames.
Alexander Caviedes. 2015. An Emerging ‘European’
10 Acknowledgements News Portrayal of Immigration? Journal of Ethnic
and Migration Studies, 41(6):897–917.
We thank Anoop Kotha, Shiqi Sheng, Guoxin Yin,
Dennis Chong and James N. Druckman. 2007. Fram-
and Hongting Zhu for their contributions to the data ing Theory. Annual Review of Political Science,
annotation effort. We also thank Libby Hemphill 10(1):103–126.
and Stuart Soroka for their valuable comments and
Ryan Compton, David Jurgens, and David Allen. 2014.
feedback. This work was supported in part through Geotagging one hundred million Twitter accounts
funding from the Volkswagen Foundation. with total variation minimization. In 2014 IEEE In-
ternational Conference on Big Data (Big Data).Constance de Saint Laurent, Vlad Glaveanu, and Language Technologies, Volume 1 (Long and Short
Claude Chaudet. 2020. Malevolent Creativity and Papers), pages 1401–1407.
Social Media: Creating Anti-immigration Commu-
nities on Twitter. Creativity Research Journal, Sabina Hartnett. 2019. Willkommenskultur: A compu-
32(1):66–80. tational and socio-linguistic study of modern german
discourse on migrant populations. Transit, 12(1).
Claes H de Vreese. 2005. News framing: Theory and
typology. Information Design Journal, 13(1):51– Tobias Heidenreich, Fabienne Lind, Jakob-Moritz
62. Eberl, and Hajo G Boomgaarden. 2019. Me-
dia Framing Dynamics of the ‘European
Jakob-Moritz Eberl, Christine E. Meltzer, Tobias Hei- Refugee Crisis’: A Comparative Topic Mod-
denreich, Beatrice Herrero, Nora Theorin, Fabienne elling Approach. Journal of Refugee Studies,
Lind, Rosa Berganza, Hajo G. Boomgaarden, Chris- 32(Special_Issue_1):i172–i182.
tian Schemer, and Jesper Strömbäck. 2018. The Eu-
ropean media discourse on immigration and its ef- Marc Helbling. 2014. Framing Immigration in Western
fects: a literature review. Annals of the International Europe. Journal of Ethnic and Migration Studies,
Communication Association, 42(3):207–223. 40(1):21–41.
Robert M. Entman. 1993. Framing: Toward Clarifica-
Nicole Hemmer. 2016. Messengers of the right: Con-
tion of a Fractured Paradigm. Journal of Communi-
servative media and the transformation of American
cation, 43(4):51–58.
politics. University of Pennsylvania Press.
Walter A Ewing, Daniel Martinez, and Rubén G Rum-
baut. 2015. The criminalization of immigration in Jackie Hogan and Kristin Haltinner. 2015. Floods,
the United States. Washington, DC: American Im- Invaders, and Parasites: Immigration Threat Nar-
migration Council Special Report. ratives and Right-Wing Populism in the USA, UK
and Australia. Journal of Intercultural Studies,
Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer 36(5):520–543.
Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Fram-
ing and agenda-setting in Russian news: A compu- Jan Fredrik Hovden and Hilmar Mjelde. 2019. Increas-
tational analysis of intricate political strategies. Pro- ingly Controversial, Cultural, and Political: The
ceedings of the 2018 Conference on Empirical Meth- Immigration Debate in Scandinavian Newspapers
ods in Natural Language Processing, pages 3570– 1970–2016. Javnost, 26(2):138–157.
3580.
Shanto Iyengar. 1990. Framing responsibility for polit-
Stephanie A. Fryberg, Nicole M. Stephens, Rebecca ical issues: The case of poverty. Political Behavior,
Covarrubias, Hazel Rose Markus, Erin D. Carter, 12(1):19–40.
Giselle A. Laiduc, and Ana J. Salido. 2012. How the
Media Frames the Immigration Debate: The Critical Shanto Iyengar. 1991. Is anyone responsible? How
Role of Location and Politics. Analyses of Social television frames political issues. University of
Issues and Public Policy, 12(1):96–112. Chicago Press.
Susan Gal. 1978. Peasant men can’t get wives: Lan- Shanto Iyengar, Helmut Norpoth, and Kyu S Hahn.
guage change and sex roles in a bilingual community. 2004. Consumer demand for election news: The
Language in society, 7(1):1–16. horserace sells. The Journal of Politics, 66(1):157–
175.
Jesse Graham, Jonathan Haidt, and Brian A Nosek.
2009. Liberals and conservatives rely on different Kristen Johnson, Di Jin, and Dan Goldwasser. 2017.
sets of moral foundations. Journal of personality Modeling of political discourse framing on Twitter.
and social psychology, 96(5):1029. In Proceedings of the International AAAI Confer-
Josh Grimm and Julie L. Andsager. 2011. Framing im- ence on Web and Social Media, volume 11.
migration: Geo-ethnic context in california newspa-
pers. Journalism and Mass Communication Quar- John T Jost, Jack Glaser, Arie W Kruglanski, and
terly, 88(4):771–788. Frank J Sulloway. 2003. Political conservatism as
motivated social cognition. Psychological bulletin,
Kimberly Gross. 2008. Framing persuasive ap- 129(3):339.
peals: Episodic and thematic framing, emotional re-
sponse, and policy opinion. Political Psychology, Shima Khanehzar, Andrew Turpin, and Gosia Mikola-
29(2):169–192. jczak. 2019. Predicting Political Frames Across Pol-
icy Issues and Contexts. In Proceedings of the 17th
Mareike Hartmann, Tallulah Jansen, Isabelle Augen- Workshop of the Australasian Language Technology
stein, and Anders Søgaard. 2019. Issue framing in Association, pages 101–106.
online discussion fora. In Proceedings of the 2019
Conference of the North American Chapter of the Klaus Krippendorff. 2013. Content Analysis: An Intro-
Association for Computational Linguistics: Human duction to Its Methodology. Sage Publications.Haewoon Kwak, Jisun An, and Yong-Yeol Ahn. 2020. Innocent Ndubuisi-Obi, Sayan Ghosh, and David Ju-
A systematic media frame analysis of 1.5 million rgens. 2019. Wetin dey with these comments?
New York Times articles from 2000 to 2017. In 12th Modeling sociolinguistic factors affecting code-
ACM Conference on Web Science, pages 305–314. switching behavior in Nigerian online discussions.
In Proceedings of the 57th Annual Meeting of the
Andrea Lawlor. 2015. Local and National Accounts Association for Computational Linguistics, pages
of Immigration Framing in a Cross-national Per- 6204–6214.
spective. Journal of Ethnic and Migration Studies,
41(6):918–941. Zizi Papacharissi and Maria De Fatima Oliveira. 2008.
News frames terrorism: A comparative analysis
Sophie Lecheler, Linda Bos, and Rens Vliegenthart. of frames employed in terrorism coverage in U.S.
2015. The mediating role of emotions: News fram- and U.K. newspapers. International Journal of
ing effects on opinions about immigration. Journal- Press/Politics, 13(1):52–74.
ism & Mass Communication Quarterly, 92(4):812– Shamik Roy and Dan Goldwasser. 2020. Weakly su-
838. pervised learning of nuanced frames for analyzing
polarization in news media. In Proceedings of the
Michael T Light and Ty Miller. 2018. Does undocu- 2020 Conference on Empirical Methods in Natural
mented immigration increase violent crime? Crimi- Language Processing, pages 7698–7716.
nology, 56(2):370–401.
W. Russell Neuman, Lauren Guggenheim, S. Mo Jang,
Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and and Soo Young Bae. 2014. The Dynamics of Public
Derry Tanti Wijaya. 2019a. Detecting frames in Attention: Agenda-Setting Theory Meets Big Data.
news headlines and its application to analyzing news Journal of Communication, 64(2):193–214.
framing trends surrounding US gun violence. In
Proceedings of the 23rd Conference on Computa- Dietram A. Scheufele. 1999. Framing as a the-
tional Natural Language Learning, pages 504–514. ory of media effects. Journal of Communication,
49(1):103–122.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-
dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Holli A Semetko and Patti M Valkenburg. 2000. Fram-
Luke Zettlemoyer, and Veselin Stoyanov. 2019b. ing European politics: A content analysis of press
RoBERTa: A Robustly Optimized BERT Pretrain- and television news. Journal of communication,
ing Approach. arXiv preprint arXiv:1907.11692. 50(2):93–109.
Philippa Shoemark, Debnil Sur, Luke Shrimpton, Iain
Sharon Meraz and Zizi Papacharissi. 2013. Net- Murray, and Sharon Goldwater. 2017. Aye or naw,
worked Gatekeeping and Networked Framing on whit dae ye hink? scottish independence and lin-
#Egypt. The International Journal of Press/Politics, guistic identity on social media. In Proceedings of
18(2):138–166. the 15th Conference of the European Chapter of the
Association for Computational Linguistics, pages
James Milroy. 2001. Language ideologies and the con- 1239–1248.
sequences of standardization. Journal of sociolin-
guistics, 5(4):530–555. Eugenia Siapera, Moses Boudourides, Sergios Lenis,
and Jane Suiter. 2018. Refugees and Network
Joshua R Minot, Michael V Arnold, Thayer Alshaabi, Publics on Twitter: Networked Framing, Affect, and
Christopher M Danforth, and Peter Sheridan Dodds. Capture. Social Media and Society, 4(1).
2020. Ratioing the president: An exploration of pub-
lic engagement with Obama and Trump on Twitter. Robert Courtney Smith. 2017. Dont let the illegals
arXiv preprint arXiv:2006.03526. vote!: The myths of illegal latino voters and voter
fraud in contested local immigrant integration. The
Alan Mislove, Sune Lehmann, Yong-Yeol Ahn, Jukka- Russell Sage Foundation Journal of the Social Sci-
Pekka Onnela, and James Rosenquist. 2011. Under- ences, 3(4):148–175.
standing the demographics of Twitter users. In Pro-
Francesco Somaini. 2019. News stories framed episod-
ceedings of the International AAAI Conference on
ically offer more diversified portrayals of immi-
Web and Social Media, volume 5.
grants. Newspaper Research Journal, 40(2):190–
210.
Fred Morstatter, Liang Wu, Uraz Yavanoglu,
Stephen R. Corman, and Huan Liu. 2018. Iden- Burton Speakman and Marcus Funk. 2020. News, na-
tifying Framing Bias in Online News. ACM tionalism, and hegemony: The formation of consis-
Transactions on Social Computing, 1(2):1–18. tent issue framing throughout the U.S. political right.
Mass Communication and Society, 23(5):656–681.
Nona Naderi and Graeme Hirst. 2017. Classifying
frames at the sentence level in news articles. Interna- Ian Stewart, Yuval Pinter, and Jacob Eisenstein. 2018.
tional Conference Recent Advances in Natural Lan- Si o no, que penses? Catalonian independence and
guage Processing, RANLP, 2017-Septe:536–542. linguistic identity on social media. In Proceedingsof the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Pa- pers), pages 136–141. Kjersti Thorbjørnsrud. 2015. Framing irregular immi- gration in western media. American Behavioral Sci- entist, 59(7):771–782. Adriano Udani and David C Kimball. 2018. Immigrant resentment and voter fraud beliefs in the US elec- torate. American Politics Research, 46(3):402–433. Baldwin van Gorp. 2005. Where is the frame? : Vic- tims and intruders in the Belgian press coverage of the asylum issue. European Journal of Communica- tion, 20(4):484–507.
A Frame distribution in annotated data C Frame detection performance
Figure 5 shows the distribution of frames as a frac- Tables 5-8 and Figures 8-9 provide details about
tion of total tweets in the annotated data. the fine-tuned RoBERTa models’ performance.
Frame Type Frame Type Precision Recall F1-score LRAP
Issue-Generic Issue-Specific Narrative Issue-Generic Policy 0.722 0.727 0.716 0.745
Political Factors Issue-Specific 0.667 0.493 0.550 0.785
Policy Prescription
Cultural Identity Narrative 0.780 0.884 0.825 0.896
Economic
Crime & Punishment
Health & Safety
Fairness & Equality Table 5: Performance by frame type on dev set.
Security & Defense
Morality & Ethics
Legality
External Regulation Issue-Generic Issue-Specific Narrative
Quality of Life
Capacity & Resources Human-Machine 0.443 0.488 0.421
Public Sentiment Human-Human 0.417 0.491 0.458
Threat: Public Order
Victim: Humanitarian
Victim: Discrimination
Threat: Fiscal Table 6: Average Krippendorff α agreement between
Hero: Worker
Threat: National Cohesion human annotators and machine-predicted labels (top
Hero: Cultural Diversity
Hero: Integration row) and between human annotator pairs (bottom row).
Threat: Jobs
Victim: Global Economy Overall, our classifiers had similar agreement with hu-
Victim: War
Episodic man annotators as humans did with one another.
Thematic
0.0 0.1 0.2 0.3 0.4 0.5 0.6
Fraction of tweets Model performance per frame
Figure 5: Distribution of frames in annotated data.
B Inter-annotator agreement plots 0.8
Figures 6 and 7 show inter-annotator agreement 0.6
F1 Score
(Krippendorff’s α) across frame types.
0.4
Agreement With First Author
0.2
Coder 1
RoBERTa FT
0.0 LogReg
0 50 100 150 200 250 300
Coder 2 Support
Annotator
Figure 8: F1 score of logistic regression (1,2-gram
Coder 3 features) and fine-tuned RoBERTa for each frame and
Frame Type frame support in evaluation sets. RoBERTa consis-
Issue-Specific tently outperforms logistic regression, especially for
Coder 4 Issue-Generic
Narrative low-frequency frames.
0.0 0.1 0.2 0.3 0.4 0.5 0.6
Krippendorff Alpha
Figure 6: Inter-annotator agreement between first au- US GB EU
thor and other coders before consensus-coding. 0.8
Agreement With Consensus 0.7
Coder 1 0.6
0.5
F1 Score
Coder 2 0.4
0.3
Annotator
Coder 3 Issue-Specific
Issue-Generic 0.2
Narrative
0.1
Coder 4
0.0
Issue-Generic Issue-Specific Narrative
Coder 5 Frame Type
0.0 0.2 0.4 0.6 0.8
Figure 9: Average F1 scores on combined dev/test set
Krippendorff Alpha separated by region. Models achieve comparable per-
Figure 7: Agreement between each coder and consen- formance for the United States, United Kingdom, and
sus annotations before consensus-coding. European Union, except for slightly lower performance
for issue-specific frames on EU tweets.You can also read
