Modeling Framing in Immigration Discourse on Social Media

Page created by Annette Becker
 
CONTINUE READING
Modeling Framing in Immigration Discourse on Social Media

                                               Julia Mendelsohn                          Ceren Budak                     David Jurgens
                                              University of Michigan                 University of Michigan           University of Michigan
                                             juliame@umich.edu                       cbudak@umich.edu                jurgens@umich.edu

                                                               Abstract                           social media content enables us to compare framing
                                                                                                  strategies across countries and political ideologies.
                                             The framing of political issues can influence
                                                                                                  Furthermore, social media provides unique insights
                                             policy and public opinion. Even though the
                                                                                                  into how messages resonate with audiences through
arXiv:2104.06443v1 [cs.CL] 13 Apr 2021

                                             public plays a key role in creating and spread-
                                             ing frames, little is known about how ordinary       interactive signals such as retweets and favorites.
                                             people on social media frame political issues.       By jointly analyzing the production and reception
                                             By creating a new dataset of immigration-            of frames on Twitter, we provide an in-depth analy-
                                             related tweets labeled for multiple framing          sis of immigration framing by and on the public.
                                             typologies from political communication the-
                                             ory, we develop supervised models to detect             Political communications research has identi-
                                             frames. We demonstrate how users’ ideology           fied numerous typologies of frames, such as issue-
                                             and region impact framing choices, and how           generic policy, immigration-specific, and narrative.
                                             a message’s framing influences audience re-
                                                                                                  Each of these frame types can significantly shape
                                             sponses. We find that the more commonly-
                                             used issue-generic frames obscure important          the audience’s perceptions of an issue (Iyengar,
                                             ideological and regional patterns that are only      1991; Chong and Druckman, 2007; Lecheler et al.,
                                             revealed by immigration-specific frames. Fur-        2015), but prior NLP work seeking to detect frames
                                             thermore, frames oriented towards human in-          in mass media (e.g. Card et al., 2016; Field et al.,
                                             terests, culture, and politics are associated with   2018; Kwak et al., 2020) has largely been limited
                                             higher user engagement. This large-scale anal-       to a single issue-generic policy typology. Multi-
                                             ysis of a complex social and linguistic phe-         ple dimensions of framing must be considered in
                                             nomenon contributes to both NLP and social
                                                                                                  order to better understand the structure of immi-
                                             science research.
                                                                                                  gration discourse and its effect on public opinion
                                         1   Introduction                                         and attitudes. We thus create a novel dataset of
                                                                                                  immigration-related tweets containing labels for
                                         Framing selects particular aspects of an issue and       each typology to facilitate more nuanced computa-
                                         makes them salient in communicating a message            tional analyses of framing.
                                         (Entman, 1993). Framing can impact how people
                                         understand issues, attribute responsibility (Iyengar,       This work combines political communication
                                         1991), and endorse possible solutions, thus having       theory with NLP to model multiple framing strate-
                                         major implications for public opinion and policy de-     gies and analyze how the public on Twitter frames
                                         cisions (Chong and Druckman, 2007). While past           immigration. Our contributions are as follows:
                                         work has studied framing by the news media and           (1) We create a novel dataset of immigration-
                                         the political elite, little is known about how ordi-     related tweets labeled for issue-generic policy,
                                         nary people frame political issues. Yet, framing by      immigration-specific, and narrative frames. (2) We
                                         ordinary people can influence others’ perspectives       develop and evaluate multiple methods to detect
                                         and may even shape elites’ rhetoric (Russell Neu-        each type of frame. (3) We illustrate how a mes-
                                         man et al., 2014). To shed light on this important       sage’s framing is influenced by its author’s ideol-
                                         topic, we focus on one issue—immigration—and             ogy and country. (4) We show how a message’s
                                         develop a new methodology to computationally             framing affects its audience by analyzing favorit-
                                         analyze its framing on Twitter.                          ing and retweeting behaviors. Finally, our work
                                            Our work highlights unique insights that social       highlights the need to consider multiple framing
                                         media data offers. The massive amount of available       typologies and their effects.
2   Framing in the Media                                  news highlight region and ideology as particularly
                                                          important factors. Right-leaning media from con-
Framing serves four functions: (i) defining prob-         servative regions are more likely to frame immi-
lems, (ii) diagnosing causes, (ii) making evaluative      grants as intruders (van Gorp, 2005), and as threats
judgments, and (iv) suggesting solutions (Entman,         to the economy and public safety (Fryberg et al.,
1993). Framing impacts what people notice about           2012). Framing also differs across countries; while
an issue, making it a key mechanism by which a            the US press emphasizes public order, discrimina-
text influences its audience.                             tion, and humanitarian concerns, the French press
Framing Typologies We draw upon distinct ty-              more frequently frames immigrants as victims of
pologies of frames that can be applied to the issue       global inequality (Benson, 2013).
of immigration: (1) issue-specific, which identify           Frame-setting has also been studied in the con-
aspects of a particular issue, or (2) issue-generic,      text of immigration. For example, experimental
which appear across a variety of issues and facili-       work has shown that frames eliciting angry or en-
tate cross-issue comparison (de Vreese, 2005).            thusiastic emotions impact participants’ opinions
   Issue-generic frames include policy frames that        on immigration (Lecheler et al., 2015). While past
focus on aspects of issues important for policy-          work has analyzed linguistic framing in Twitter im-
making, such as economic consequences or fair-            migration discourse (e.g., de Saint Laurent et al.,
ness and equality (Boydstun et al., 2013). Other          2020), little is known about how such framing af-
generic frames focus on a text’s narrative; news          fects users’ interactive behaviors such as resharing
articles use both episodic frames, which highlight        content, which is a key objective of frame setting.
specific events or individuals, and thematic frames,
which place issues within a broader social context.       3   Computational Approaches to Framing
The use of episodic versus thematic frames can
influence the audience’s attitudes. For example,          Because many people now generate and consume
episodic frames lead audiences to attribute respon-       political content on social media, scholars have
sibility for issues such as poverty to individual citi-   increasingly used automated techniques to study
zens while thematic frames lead them to hold the          framing on social media.
government responsible (Iyengar, 1991).                      Large-scale research of framing on Twitter has
   Issue-specific frames for immigration focus on         commonly focused on unsupervised approaches.
the portrayal of immigrants. Our analysis uses            (e.g., Russell Neuman et al., 2014; Meraz and Pa-
Benson (2013)’s set of issue-specific frames, which       pacharissi, 2013; de Saint Laurent et al., 2020).
represent immigrants as heroes (cultural diversity,       Such approaches, including those focused on hash-
integration, good workers), victims (humanitarian,        tag analysis, can reveal interesting framing patterns.
global economy, discrimination), and threats (to          For instance, Siapera et al. (2018) shows that frame
jobs, public order, taxpayers, cultural values).          usage varies across events. Similarly, topic models
   Both issue-specific and generic frames provide         have been used to compare “refugee crisis" media
unique insights but present advantages and draw-          discourses across the European countries (Heiden-
backs. While issue-specific frames analysis are           reich et al., 2019), and to uncover differences in
specific and detailed, they are hard to generalize        attitudes towards migrants (Hartnett, 2019). Al-
and replicate across studies, which is a key advan-       though lexicon analysis and topic models can pro-
tage for generic frames (de Vreese, 2005).                vide insights about immigration discourse, here, we
Framing effects Studies of framing typically              adopt a supervised approach to ground our work in
focus on either frame-building or frame-                  framing research and to enable robust evaluation.
setting (Scheufele, 1999; de Vreese, 2005).                  We draw inspiration from a growing body of
Frame-building is the process by which external           NLP research that uses supervised approaches to
factors, such as a journalist’s ideology or eco-          detect issue-generic policy frames in news arti-
nomic pressures, influence what frames are used;          cles, a task popularized by the Media Frames
frame-building studies thus treat framing as the          Corpus (Card et al., 2015), which contains issue-
dependent variable. Frame-setting studies treat           generic policy frame labels for articles across sev-
frames as independent variables that impact how           eral issues (Boydstun et al., 2013). Using this
an audience interprets and evaluates issues.              corpus, prior work has detected frames with tech-
   Prior analyses of frame-building in immigration        niques including logistic regression (Card et al.,
Frame Type               Frame                                                   Description
Issue-Generic   Economic                      Financial implications of an issue
 Policy         Capacity & Resources          The availability or lack of time, physical, human, or financial resources
                Morality & Ethics             Perspectives compelled by religion or secular sense of ethics or social responsibility
                Fairness & Equality           The (in)equality with which laws, punishments, rewards, resources are distributed
                Legality, Constitutionality   Court cases and existing laws that regulate policies; constitutional interpretation;
                & Jurisdiction                legal processes such as seeking asylum or obtaining citizenship; jurisdiction
                Crime & Punishment            The violation of policies in practice and the consequences of those violations
                Security & Defense            Any threat to a person, group, or nation and defenses taken to avoid that threat
                Health & Safety               Health and safety outcomes of a policy issue, discussions of health care
                Quality of Life               Effects on people’s wealth, mobility, daily routines, community life, happiness, etc.
                Cultural Identity             Social norms, trends, values, and customs; integration/assimilation efforts
                Public Sentiment              General social attitudes, protests, polling, interest groups, public passage of laws
                Political Factors &           Focus on politicians, political parties, governing bodies, political campaigns
                Implications                  and debates; discussions of elections and voting
                Policy Prescription &
                                              Discussions of existing or proposed policies and their effectiveness
                Evaluation
                External Regulation &         Relations between nations or states/provinces; agreements between governments;
                Reputation                    perceptions of one nation/state by another
Immigration     Victim: Global Economy        Immigrants are victims of global poverty, underdevelopment and inequality
 Specific       Victim: Humanitarian          Immigrants experience economic, social, and political suffering and hardships
                Victim: War                   Focus on war and violent conflict as reason for immigration
                Victim: Discrimination        Immigrants are victims of racism, xenophobia, and religion-based discrimination
                Hero: Cultural Diversity      Highlights positive aspects of differences that immigrants bring to society
                Hero: Integration             Immigrants successfully adapt and fit into their host society
                Hero: Worker                  Immigrants contribute to economic prosperity and are an important source of labor
                Threat: Jobs                  Immigrants take nonimmigrants’ jobs or lower their wages
                Threat: Public Order          Immigrants threaten public safety by being breaking the law or spreading disease
                Threat: Fiscal                Immigrants abuse social service programs and are a burden on resources
                Threat: National Cohesion     Immigrants’ cultural differences are a threat to national unity and social harmony
Narrative       Episodic                      Message provides concrete information about on specific people, places, or events
                Thematic                      Message is more abstract, placing stories in broader political and social contexts

Table 1: List of all issue-generic policy (Boydstun et al., 2013), immigration-specific (Benson, 2013; Hovden and
Mjelde, 2019), and narrative (Iyengar, 1991) frames with brief descriptions.

2016), recurrent neural networks (Naderi and Hirst,                ied issue-specific frames in news media for issues
2017), lexicon induction (Field et al., 2018), and                 such as missile defense and gun violence (Morstat-
fine-tuning pretrained language models (Khane-                     ter et al., 2018; Liu et al., 2019a). We extend issue-
hzar et al., 2019; Kwak et al., 2020). Roy and                     specific frame analyses to immigration by adopting
Goldwasser (2020) further extracted subcategories                  an immigration-specific typology developed by po-
of issue-generic policy frames in newspaper cover-                 litical communication scholars (Benson, 2013).
age using a weakly-supervised approach. Finally,                       In contrast to prior NLP work focused on tradi-
issue-generic frames have also been computation-                   tional media or political elites (Johnson et al., 2017;
ally studied in other media, including online fora                 Field et al., 2018), we highlight the role that social
and politicians’ tweets (Johnson et al., 2017; Hart-               media publics play in generating and propagating
mann et al., 2019). We build upon this literature by               frames. Furthermore, we provide a new computa-
incorporating additional frame typologies that re-                 tional model of narrative framing (Iyengar, 1991),
flect important dimensions of media discourse with                 that together with models for issue-generic policy
real-world consequences (Iyengar, 1991; Gross,                     and issue-specific frames, provides complementary
2008; Eberl et al., 2018). Beyond detecting frames,                views on the framing of immigration. Finally, our
we computationally analyze frame-building and                      large-scale analysis of frame-setting illustrates the
frame-setting among social media users; though                     potential for using NLP to understand how a mes-
well-studied in traditional news media, little is                  sage’s framing shapes its audience behavior.
known about how social media users frame im-
migration or its effects (Eberl et al., 2018).                     4    Data
  Noting that issue-generic policy frames obscure                  We first collect a large dataset of immigration-
important linguistic differences, several works stud-              related tweets, and then annotate a subset of this
full dataset for multiple types of frames.                       influence an issue’s framing. For simplicity, we
Data Collection We extract all English-language                  treat each tweet as a standalone message and label
tweets in 2018 and 2019 from the Twitter Decahose                frames based only on the text (including hashtags).
containing at least one of the following terms: im-                 Unlike news stories, where frames are clearly
migration, immigrant(s), emigration, emigrant(s),                cued, tweets often implicitly allude to frames due
migration, migrant(s), illegal alien(s), illegals, and           to character limitations. For example, a tweet ex-
undocumented1 . We focus on content creation and                 pressing desire to “drive immigrants out" with no
thus exclude retweets from our dataset, though we                additional context may suggest a criminal frame,
consider retweeting rates when analyzing the social              but criminality is not explicit. To minimize er-
influence of different frames. We further restrict               rors, we avoid making assumptions about intended
our dataset to tweets whose authors are identified               meaning and interpret all messages literally.
as being located in the United States (US), United                  Training, development, and test data were anno-
Kingdom (GB), and European Union (EU) by an                      tated using two procedures after four annotators
existing location inference tool (Compton et al.,                completed four rounds of training. The dataset
2014). To compare framing across political ide-                  contains equal numbers of tweets from the EU,
ologies, we obtain ideal point estimates for nearly              UK, and US. Training data was singly annotated
two-thirds of US-based users with Barberá (2015)’s               and includes 3,600 tweets, while the development
Bayesian Spatial Following model. Our full dataset               and test sets each contain 450 tweets (10% of the
contains over 2.66 million tweets, 86.2% of which                full dataset) and were consensus-coded by pairs
are from the United States, 10.4% from the United                of trained annotators. We opt for this two-tier ap-
Kingdom, and 3.4% from the European Union.                       proach due to (i) the inherent difficulty of the task2
Data Annotation Tweets are annotated using three                 and (ii) the need to maximize diversity seen in
frame typologies: (i) issue-generic policy, (ii)                 training. During annotator training, pilot studies
immigration-specific, and (iii) narrative frames,                attained moderate agreement, suggesting that to
where a tweet may use multiple frames simulta-                   attain high-reliability, consensus coding with ad-
neously. We use Boydstun et al. (2013)’s Policy                  judication would be needed (Krippendorff, 2013),
Frames Codebook to formulate our initial guide-                  which comes at a cost of substantially increased
lines to code for policy frames. We use Benson                   time. Because a large dataset of unique, singly-
(2013)’s immigration-specific frames, but follow                 coded documents is preferable to a small dataset
Hovden and Mjelde (2019) in including an addi-                   of documents coded by multiple annotators for text
tional category for framing immigrants as victims                classification (Barbera et al., 2021), we decided
of war. Finally, we code for narrative frames using              to increase corpus diversity in the training data
definitions from Iyengar (1991). All frames and de-              by singly-annotating, at the expense of potentially
scriptions can be found in Table 1, with a complete              noisier annotation, and to consensus code all eval-
codebook in Supplementary Materials. Because                     uation data. On the double annotated data, anno-
annotation guidelines from prior work focus on                   tators attained Krippendorff’s α=0.45. Additional
elite communications, we first adjusted our code-                details are provided in Supplementary Material (§B,
book to address challenges posed by Twitter con-                 Figures 6 and 7).
tent. Changes were made based on feedback from                   Results We observe differences across frame ty-
four trained annotators who labeled 360 tweets                   pologies in coverage rates within the annotated
from 2018, split between the EU, GB, and US.                     data set. While 84% of tweets are labeled with
   Even for humans, identifying frames in tweets                 at least one issue-generic policy frame and 85%
is a difficult task. Defining the boundaries of what             with at least one narrative frame, only 51% are la-
constitutes a message is not trivial. Beyond the                 beled with at least one issue-specific frame. This
text, frames could be identified in hashtags, images,            difference is due to immigration-specific frames
videos, and content from linked pages. Further-                  being more narrowly-defined, as they require ex-
more, tweets are often replies to other users or part            plicit judgment of immigrants as heroes, victims,
of a larger thread. This additional context may                  or threats. Further details about frame distributions
   1                                                                2
     We obtained this list by starting with the seed terms im-        For example, in identifying just the primary issue-generic
migrants, immigration, and illegal aliens. We then added the     frame of a document, the Media Frames corpus attained an
remaining terms by manually inspecting and filtering nearby      Krippendorff’s α=∼0.6 (Card et al., 2015, Fig. 4), whereas we
words in pretrained GloVe and Word2Vec vector spaces.            ask annotators to identify all frames across three typologies.
Random        LogReg        RoBERTa         FT RoBERTa         tion discourse on social media in order to capture
     0.193         0.296          0.611            0.657           diverse perspectives and arguments.
Table 2: F1 scores on the test set for all models, cal-               Table 3 shows several evaluation metrics sepa-
culated as an (unweighted) average over all frames                 rated by frame type. Precision, recall, and F1 are
and initialization seeds. The fine-tuned (FT) RoBERTa              calculated as unweighted averages over all frames
model improvements over all models are significant at              belonging to each category. Overall, issue-generic
p
Error Type                   Description                                                           Example
                             These instances highlight the challenges of annotation;               Interestingly, the criteria to which immigrants would be held would
Plausible interpretation     there are convincing arguments that model’s predicted                 not be met by a large number of the ‘British’ people either.
                             frames can be appropriate labels.                                     Model erroneously predicted Policy
Inferring frames not         Model predicts frames that may capture an author’s intention          Stop immigration
explicitly cued in text      but without sufficient evidence from the text                         Model erroneously predicted Threat: Public Order
                                                                                                   @EricTrump Eric I have been alive longer than your immigrant
                             Some frames are directly cued by lexical items
Missing necessary                                                                                  mother in law and you. I paid more in taxes than you did and
                             (e.g. politicians’ names cue Political frame), but model
contextual knowledge                                                                               your immigrant mother in law combined...
                             lacks real-world knowledge required to identify these frames
                                                                                                   Model missed Political frame
                             Many words and phrases do not directly cue frames, but are            Lunaria’s figures from 2018 recorded 12 shootings, two murders
Overgeneralizing             highly-correlated. The model makes erroneous predictions              and 33 physical assaults against migrants in the first two months
highly-correlated features   when such features are used in different contexts (e.g. violence      since Salvini entered government.
                             against immigrants, rather than immigrants being violent)             Model missed Victim: Humanitarian frame
                                                                                                   It’s worse when you have immigrant parents who don’t speak
                             Coreference resolution is often not possible and annotators avoided
                                                                                                   the language cause you have to deal with all the paperwork,
                             making assumptions to resolve ambiguities. For example, "you"
Pronoun ambiguity                                                                                  be the translator for them whenever they go (...)
                             can be used to discuss individuals’ experiences (episodic) but its
                                                                                                   its tiring but someone has to
                             impersonal sense can be in broad generalizations (thematic).
                                                                                                   Model predicted Episodic but referent is unclear

            Table 4: Types of common errors in frame prediction along with brief descriptions and examples.

frame detection to achieve higher performance on                                       tic norms (Papacharissi and De Fatima Oliveira,
conservative tweets due to more linguistic regular-                                    2008), geographic proximity to immigrant pop-
ities across messages. Indeed, we find that issue-                                     ulations or points of entry (Grimm and And-
generic and issue-specific classifiers achieve higher                                  sager, 2011; Fryberg et al., 2012), and immigrants’
F1 scores on tweets written by conservative authors                                    race/ethnicity (Grimm and Andsager, 2011). At the
compared to liberal authors (Figure 1), even though                                    same time, increased globalization may result in a
there are fewer conservative tweets in the train-                                      uniform transnational immigration discourse (Hel-
ing data (334 conservative vs 385 liberal tweets).                                     bling, 2014). Framing variations across countries
Higher model performance on conservative tweets                                        has implications for government policies and ini-
suggests that, like political and media elites, con-                                   tiatives, particularly in determining what solutions
servatives on social media are more consistent than                                    could be applied internationally or tailored to each
liberals in their linguistic framing of immigration.                                   country (Caviedes, 2015).
Error Analysis We identify classification errors                                          Prior studies on the role of ideology in frame-
by qualitatively analyzing a random sample of 200                                      building have focused on the newspapers or politi-
tweets that misclassified at least one frame. Table                                    cal movements, showing patterns in frames like
4 shows the most common categories of errors.                                          morality and security by political affiliation in
                                                                                       European immigration discourse (Helbling, 2014;
6     Frame-Building Analysis                                                          Hogan and Haltinner, 2015) or in use of economic
In writing about an issue, individuals are known                                       frames by American newspapers (Fryberg et al.,
to select particular frames—a process known as                                         2012; Abrajano et al., 2017). However, it remains
frame-building—based on numerous factors, such                                         unclear whether these patterns observed for elite
as exposure to politicians’ rhetoric or their own                                      groups can generalize to the effect of individual
identity (Scheufele, 1999). Here, we focus on two                                      people’s political dispositions.
specific identity attributes affecting frame building:                                 Experimental Setup We detect frames for all
(i) political ideology and (ii) country/region.                                        2.6M immigration-related tweets using the fine-
   The political, social, and historical contexts of                                   tuned RoBERTa model with the best-performing
an one’s nation-state can impact how they frame                                        seed on development data. Using this labeled data,
immigration (Helbling, 2014). Immigration has                                          we estimate the effects of region and ideology by
a long history in the USA relative to Europe,                                          fitting separate mixed-effects logistic regression
and former European colonial powers (e.g. the                                          models to predict the presence or absence of each
UK) have longer immigration histories than other                                       frame. We treat region (US, UK, and EU) as a
countries (e.g. Norway) (Thorbjørnsrud, 2015;                                          categorical variable, with US as the reference level.
Eberl et al., 2018). Cross-country variation in                                        Ideology is estimated using the method of Barberá
news framing also arise from differences in im-                                        (2015), which is based on users’ connections to US
migration policies (Helbling, 2014; Lawlor, 2015),                                     political elites; as such, we restrict our analysis of
media systems (Thorbjørnsrud, 2015), journalis-                                        ideology to only tweets from the United States.
Frame Type                              cus on frames with implications for the in-group.
      Issue-Specific        Issue-Generic            Narrative   They express concerns about 1) immigrants im-
                                 Liberal      Conservative       posing a burden on taxpayers and governmental
                                                                 programs and 2) immigrants being criminals and
       Victim: Humanitarian
    Hero: Cultural Diversity                                     threats to public safety. We qualitatively observe
      Victim: Discrimination
                  Victim: War                                    three distinct, though unsubstantiated, conservative
            Morality & Ethics                                    claims contributing to the latter: (i.) Immigrants
                Hero: Worker
            Hero: Integration                                    commit violent crimes (Light and Miller, 2018),
                     Episodic
         Fairness & Equality                                     (ii.) Undocumented immigrants illegally vote in
               Quality of Life
             Cultural Identity                                   US elections (Smith, 2017; Udani and Kimball,
            Public Sentiment
    Victim: Global Economy                                       2018), and (iii.) Immigrants are criminals simply
             Health & Safety                                     by virtue of being immigrants (Ewing et al., 2015).
          Policy Prescription
                    Economic
             Political Factors                                      Figure 2 shows a clear ideological stratification
         External Regulation                                     for issue-specific frames: liberals favor hero and
                 Threat: Jobs
       Crime & Punishment                                        victim frames, while conservatives favor threat
         Security & Defense
      Capacity & Resources                                       frames. This finding is consistent with prior work
                    Thematic
  Threat: National Cohesion                                      on the role perceived threats play in shaping white
               Threat: Fiscal
        Threat: Public Order                                     American attitudes towards immigration (Brader
                                                                 et al., 2008), and the disposition of political conser-
                                 0.5         0.0         0.5     vatism to avoid potential threats (Jost et al., 2003).
                                           Coefficient
                                                                    Second, while all frame categories show ide-
Figure 2: Logistic regression coefficients of political          ological bias, issue-specific frames are the most
ideology in predicting each frame. Positive (negative)           extreme. Most notably, our analysis shows that fo-
values correspond to more conservative (liberal) ideol-          cusing solely on issue-generic policy frames would
ogy. Only frames associated with ideology after Holm-
                                                                 obscure important patterns. For example, the issue-
Bonferroni correction with p < 0.01 are included.
                                                                 generic cultural identity frame shows a slight lib-
                                                                 eral bias; yet, related issue-specific frames diverge:
   To account for exogenous events that may im-                  hero: cultural diversity is very liberal while threat:
pact framing, we include nested random effects for               national cohesion is very conservative.
year, month, and date. We further control for user                  Similarly, the issue-generic economic policy
characteristics (e.g. the author’s follower count,               frame is slightly favored by more conservative au-
friends count, verified status and number of prior               thors, but the related issue-specific frames threat:
tweets) as well as other tweet characteristics (e.g.             jobs and hero: worker reveal ideological divides.
tweet length, if a tweet is a reply, and whether                 This finding highlights the importance of using mul-
the tweet contains hashtags, URLs, or mentions                   tiple framing typologies to provide a more nuanced
of other users). We apply Holm-Bonferroni cor-                   analysis of immigration discourse.
rections on p-values before significance testing to                 Third, more liberal authors tend to use episodic
account for multiple hypothesis testing.                         frames, while conservative authors tend to use the-
Ideology Ideology is strongly predictive of fram-                matic frames. This difference is consistent with
ing strategies in all three categories, as shown in              Somaini (2019)’s finding that a local liberal news-
Figure 2. Our results reveal three broad themes.                 paper featured more episodic framing in immigra-
   First, prior work has argued that liberals and                tion coverage, but a comparable conservative news-
conservatives adhere to different moral founda-                  paper featured more thematic framing. Other ef-
tions, with conservatives being more sensitive to                forts that examine the relationship between narra-
in-group/loyalty and authority than liberals, who                tive frames and cognitive and emotional responses
are more sensitive to care and fairness (Graham                  provide some clues for the observed pattern. For
et al., 2009). Our results agree with this argument.             instance, Aarøe (2011) shows that thematic frames
Liberals are more likely to frame immigration as                 are stronger when there are no or weak emotional
a fairness and morality issue, and immigrants as                 responses; and that the opposite is true for episodic
victims of discrimination and inhumane policies.                 frames. The divergence of findings could be driven
More conservative authors, on the other hand, fo-                by partisans’ differing emotional responses. Our
findings also highlight important consequences for                     other factors, interact in intricate ways to shape
opinion formation. Iyengar (1990) shows that                           how ordinary people frame political issues.
episodic framing diverts attention from societal and                      Second, cultural identity is more strongly associ-
political party responsibility; our results suggest                    ated with both the UK and EU than the US. Perhaps
that liberal Twitter users are likely to produce (and,                 immigrants’ backgrounds are more marked in Eu-
due to partisan self-segregation, consume) social                      ropean discourse than in US discourse because the
media content with such effects.                                       UK and EU have longer histories of cultural and
                                                                       ethnic homogeneity (Thorbjørnsrud, 2015). This
                               Frame Type                              finding also reflects that Europeans’ attitudes to-
          Issue-Specific          Issue-Generic            Narrative
                                                                       wards immigration depend on where immigrants
                                          USA         EU               are from and parallels how European newspapers
           Threat: Public Order
          Crime & Punishment                                           frame immigration differently depending on mi-
                  Threat: Fiscal
                Political Factors
                    Threat: Jobs                                       grants’ countries of origin (Eberl et al., 2018).
            Security & Defense
                       Thematic                                           Finally, the bottom of Figure 3 shows that users
                       Economic
               Morality & Ethics                                       from the UK are more likely to invoke labor-related
             Policy Prescription
                        Episodic                                       frames. This prevalence of labor and economic
               Hero: Integration
         Capacity & Resources                                          frames has also been found in British traditional
                Health & Safety
            Fairness & Equality
          Victim: Humanitarian                                         media (Caviedes, 2015; Lawlor, 2015), and has
     Threat: National Cohesion
                   Hero: Worker                                        been attributed to differences in the labor mar-
                     Victim: War
                Cultural Identity                                      ket. Unlike migrants in the US, Italy, and France,
       Victim: Global Economy
            External Regulation                                        who often work clandestinely in different economic
                                          USA         UK               sectors than domestic workers, UK migrants have
           Threat: Public Order                                        proper authorization and are thus viewed as com-
          Crime & Punishment
               Morality & Ethics
            Security & Defense                                         petition for British workers because they can work
          Victim: Humanitarian
                  Threat: Fiscal                                       in the same industries (Caviedes, 2015).
                Health & Safety
                Political Factors
                        Episodic
               Hero: Integration
     Threat: National Cohesion                                         7   Audience Response to Frames
                       Economic
                  Quality of Life
                     Victim: War
            Fairness & Equality                                        Chong and Druckman (2007, p. 116) assert that
               Public Sentiment
             Policy Prescription                                       a “challenge for future work concerns the identifi-
         Capacity & Resources
                    Threat: Jobs
         Victim: Discrimination                                        cation of factors that make a frame strong.” Stud-
                Cultural Identity
            External Regulation                                        ies of frame-setting—i.e., how a message’s fram-
       Victim: Global Economy
                   Hero: Worker                                        ing affects its audience’s emotions, beliefs, and
                                      1           0         1          opinions—have largely been restricted to small-
                                           Coefficient
                                                                       scale experimental studies because responses to
Figure 3: Effect of author being from the EU (top) or                  news media framing cannot be directly observed
the UK (bottom) relative to the US. Frames with posi-                  (Eberl et al., 2018). However, Twitter provides
tive β coefficients are associated with authors from the               insight into the frame-setting process via interac-
EU (top) and UK (bottom), and frames with negative                     tive signals: favorites and retweets. While related,
values are associated with US-based authors. Frames                    these two actions can have distinct underlying mo-
not significantly associated with region after Holm-                   tivations: favoriting often indicates positive align-
Bonferroni correction are not included.
                                                                       ment between the author and the reader; in contrast,
                                                                       retweeting may also be driven by other motivations,
Region Immigration framing depends heavily on                          such as the desire to inform or entertain others
one’s geopolitical entity (US, UK, and EU), as                         (boyd et al., 2010). Different audience interactions
shown in Figure 3. Several notable themes emerge.                      have been shown to exhibit distinct patterns in polit-
First, many ideologically-extreme frames in the                        ical communication on Twitter (Minot et al., 2020).
US, including crime & punishment, security & de-                       Here, we test how a message’s framing impacts
fense, threat: public order, and threat: fiscal are all                both the favorites and retweets that it receives.
significantly more likely to be found in US-based                      Experimental Setup We fit hierarchical linear
tweets relative to the UK and EU. This pattern sug-                    mixed effects models with favorites and retweets
gests that region and ideology, and likely many                        (log-transformed) as the dependent variable on US
Response            due to their increased emotional appeal to readers
                                       Favorite      Retweet    (Semetko and Valkenburg, 2000).
           Morality & Ethics                                        On the other hand, political factors & impli-
        Fairness & Equality
           Public Sentiment                                     cations is most highly associated with increased
            Cultural Identity                                   retweets. As the political frame emphasizes compe-
              Quality of Life
                   Economic                                     tition and strategy (Boydstun et al., 2013), this re-
            Political Factors
         Policy Prescription                                    sult mirrors similar links between the “horse-race"
            Health & Safety                                     frame in news reports and engagement (Iyengar
        Security & Defense
     Capacity & Resources                                       et al., 2004); users may prefer amplifying political
      Crime & Punishment
        External Regulation                                     messages via retweeting to help their side win.
           Hero: Integration
   Hero: Cultural Diversity                                         Similarly, frames about security and safety (e.g.
     Victim: Discrimination                                     crime & punishment, victim: humanitarian) are
      Victim: Humanitarian
   Victim: Global Economy                                       highly associated with more retweets, but not nec-
       Threat: Public Order                                     essarily favorites. While security and safety frames
                Threat: Jobs
 Threat: National Cohesion                                      may not lead audience members to endorse such
              Threat: Fiscal
                   Thematic                                     messages, perhaps they are more likely to amplify
                    Episodic                                    these messages due to perceived urgency or the
                                0.05      0.00   0.05    0.10   desire to persuade others of such concerns.
                                Change in (log) responses           Finally, Figure 4 shows how a message’s narra-
                                                                tive framing impacts audience response, even after
Figure 4: Effects of framing on two audience responses:         controlling for all other frames. Both episodic and
favorites and retweets. The x-axis shows regression co-
                                                                thematic frames are significantly associated with
efficients for the presence of each frame in predicting
the log-scaled number of responses. Along the y-axis            increased engagement (retweets), but less strongly
are all issue-generic policy frames (top), immigration-         than issue frames. Having a clear narrative is im-
specific frames (middle), and narrative frames (bottom)         portant for messages to spread, but the underlying
that are significantly associated with either the number        mechanisms driving engagement behaviors may
of favorites or retweets.                                       differ for episodic and thematic frames; prior work
                                                                on mainstream media has found that news stories
                                                                using episodic frames tend to be more emotion-
tweets with detected author ideology. The presence              ally engaging, while thematic frames can be more
of a frame is treated as a binary fixed effect. We              persuasive (Iyengar, 1991; Gross, 2008).
control for all temporal, user-level and tweet-level
features as in the prior section, as well as ideology.
                                                                8   Conclusion
Results The framing of immigration has a signifi-
cant impact on how users engage with the content                Users’ exposure to political information on social
via retweets and favorites (Figure 4). Many issue-              media can have immense consequences. By lever-
specific frames have a stronger effect on audience              aging multiple theory-informed typologies, our
responses than either of the other typologies. As               computational analysis of framing enables us to
recent NLP approaches have adopted issue-generic                better understand public discourses surrounding
frames for analysis (e.g., Kwak et al., 2020), the              immigration. We furthermore show that framing
strength of issue-specific frames highlights the im-            on Twitter affects how audience interactions with
portance of expanding computational analyses be-                messages via favoriting and retweeting behaviors.
yond issue-generic frames, as other frames may                  This work has implications for social media plat-
have larger consequences for public opinion.                    forms, who may wish to improve users’ experi-
   Most frames impact favorites and retweets dif-               ences by enabling them to discover content with
ferently, suggesting that the strength of a frame’s             a diversity of frames. By exposing users to a
effects is tied to the specific engagement behav-               wide range of perspectives, this work can help lay
ior. Cultural frames (e.g. hero: integration) and               foundations for more cooperative and effective on-
frames oriented around human interest (e.g. moral-              line discussions. All code, data, annotation guide-
ity, victim: discrimination) are particularly associ-           lines, and pretrained models are available at https:
ated with more endorsements (favorites), perhaps                //github.com/juliamendelsohn/framing.
9    Ethical Considerations                             References
Our analysis of frame-building involves inferring       Lene Aarøe. 2011. Investigating Frame Strength: The
                                                          Case of Episodic and Thematic Frames. Political
political ideology and regional from users with ex-       Communication, 28(2):207–226.
isting tools, so we aggregated this information in
our analysis in order to minimize the risk of ex-       Marisa A Abrajano, Zoltan Hajnal, and Hans J.G.
                                                         Hassell. 2017. Media Framing and Partisan Iden-
posing potentially sensitive personal data about         tity: The Case of Immigration Coverage and White
individuals. Our dataset includes tweet IDs along        Macropartisanship.
with frame labels, but no additional social informa-
                                                        Pablo Barberá. 2015. Birds of the same feather tweet
tion. However, there are also ethical consequences        together: Bayesian ideal point estimation using Twit-
of categorizing people along these social dimen-          ter data. Political Analysis, 23(1):76–91.
sions. We acknowledge that reducing people’s so-
                                                        Pablo Barbera, Amber E Boydstun, Suzanna Linn,
cial identities to region and ideology obscures the       Ryan McMahon, and Jonathan Nagler. 2021. Auto-
wide range of unobservable and non-quantifiable           mated text classification of news articles: A practical
predispositions and experiences that may impact           guide. Political Analysis, 29(1):19–42.
framing and attitudes towards immigration.              Rodney Benson. 2013. Shaping Immigration News: A
   We emphasize that our dataset is not fully repre-      French-American Comparison. Cambridge Univer-
sentative of all immigration discourse and should         sity Press.
not be treated as such. Twitter’s demographics are      danah boyd, Scott Golder, and Gilad Lotan. 2010.
not representative of the global population (Mis-         Tweet, tweet, retweet: Conversational aspects of
love et al., 2011). Furthermore, our dataset only in-     retweeting on Twitter. In 2010 43rd Hawaii Interna-
cludes tweets with authors from particular Western        tional Conference on System Sciences, pages 1–10.
                                                          IEEE.
countries. All tweets were automatically identified
by Twitter as being written in English, thus ad-        Amber E. Boydstun, Justin H Gross, Philip Resnik, and
ditionally imposing standard language ideologies         Noah A. Smith. 2013. Identifying Media Frames
                                                         and Frame Dynamics Within and Across Policy Is-
on the data that we include (Milroy, 2001). Fur-         sues. New Directions in Analyzing Text as Data
thermore, language choice itself can be a socially       Workshop, pages 1–13.
and politically meaningful linguistic cue that may
                                                        Ted Brader, Nicholas A. Valentino, and Elizabeth
have unique interactions with framing (e.g., Gal,         Suhay. 2008. What triggers public opposition to
1978; Shoemark et al., 2017; Stewart et al., 2018;        immigration? Anxiety, group cues, and immigra-
Ndubuisi-Obi et al., 2019).                               tion threat. American Journal of Political Science,
   Although we do not focus on abusive language,          52(4):959–978.
our topical content contains frequent instances of      Dallas Card, Amber E Boydstun, Justin H Gross, Philip
racism, Islamophobia, antisemitism, and personal          Resnik, and Noah A Smith. 2015. The media frames
insults. We caution future researchers about poten-       corpus: Annotations of frames across issues. In
                                                          53rd Annual Meeting of the Association for Compu-
tially traumatic psychological effects of working         tational Linguistics and the 7th International Joint
with this dataset.                                        Conference on Natural Language Processing, vol-
   We aim to support immigrants, an often                 ume 2, pages 438–444.
marginalized group, by shedding light on their rep-     Dallas Card, Justin H Gross, Amber E Boydstun, and
resentation on social media. However, there is a          Noah A Smith. 2016. Analyzing framing through
risk that malicious agents could exploit our frame-       the casts of characters in the news. In Proceedings of
setting findings by disseminating harmful content         the 2016 Conference on Empirical Methods in Natu-
                                                          ral Language Processing, pages 1410–1420.
packaged in more popular frames.
                                                        Alexander Caviedes. 2015. An Emerging ‘European’
10    Acknowledgements                                    News Portrayal of Immigration? Journal of Ethnic
                                                          and Migration Studies, 41(6):897–917.
We thank Anoop Kotha, Shiqi Sheng, Guoxin Yin,
                                                        Dennis Chong and James N. Druckman. 2007. Fram-
and Hongting Zhu for their contributions to the data      ing Theory. Annual Review of Political Science,
annotation effort. We also thank Libby Hemphill           10(1):103–126.
and Stuart Soroka for their valuable comments and
                                                        Ryan Compton, David Jurgens, and David Allen. 2014.
feedback. This work was supported in part through         Geotagging one hundred million Twitter accounts
funding from the Volkswagen Foundation.                   with total variation minimization. In 2014 IEEE In-
                                                          ternational Conference on Big Data (Big Data).
Constance de Saint Laurent, Vlad Glaveanu, and                  Language Technologies, Volume 1 (Long and Short
  Claude Chaudet. 2020. Malevolent Creativity and               Papers), pages 1401–1407.
  Social Media: Creating Anti-immigration Commu-
  nities on Twitter. Creativity Research Journal,             Sabina Hartnett. 2019. Willkommenskultur: A compu-
  32(1):66–80.                                                  tational and socio-linguistic study of modern german
                                                                discourse on migrant populations. Transit, 12(1).
Claes H de Vreese. 2005. News framing: Theory and
  typology. Information Design Journal, 13(1):51–             Tobias Heidenreich, Fabienne Lind, Jakob-Moritz
  62.                                                           Eberl, and Hajo G Boomgaarden. 2019.       Me-
                                                                dia Framing Dynamics of the ‘European
Jakob-Moritz Eberl, Christine E. Meltzer, Tobias Hei-           Refugee Crisis’: A Comparative Topic Mod-
   denreich, Beatrice Herrero, Nora Theorin, Fabienne           elling Approach.     Journal of Refugee Studies,
   Lind, Rosa Berganza, Hajo G. Boomgaarden, Chris-             32(Special_Issue_1):i172–i182.
   tian Schemer, and Jesper Strömbäck. 2018. The Eu-
   ropean media discourse on immigration and its ef-          Marc Helbling. 2014. Framing Immigration in Western
   fects: a literature review. Annals of the International     Europe. Journal of Ethnic and Migration Studies,
  Communication Association, 42(3):207–223.                    40(1):21–41.
Robert M. Entman. 1993. Framing: Toward Clarifica-
                                                              Nicole Hemmer. 2016. Messengers of the right: Con-
  tion of a Fractured Paradigm. Journal of Communi-
                                                                servative media and the transformation of American
  cation, 43(4):51–58.
                                                                politics. University of Pennsylvania Press.
Walter A Ewing, Daniel Martinez, and Rubén G Rum-
  baut. 2015. The criminalization of immigration in           Jackie Hogan and Kristin Haltinner. 2015. Floods,
  the United States. Washington, DC: American Im-                Invaders, and Parasites: Immigration Threat Nar-
  migration Council Special Report.                              ratives and Right-Wing Populism in the USA, UK
                                                                 and Australia. Journal of Intercultural Studies,
Anjalie Field, Doron Kliger, Shuly Wintner, Jennifer             36(5):520–543.
  Pan, Dan Jurafsky, and Yulia Tsvetkov. 2018. Fram-
  ing and agenda-setting in Russian news: A compu-            Jan Fredrik Hovden and Hilmar Mjelde. 2019. Increas-
  tational analysis of intricate political strategies. Pro-      ingly Controversial, Cultural, and Political: The
  ceedings of the 2018 Conference on Empirical Meth-             Immigration Debate in Scandinavian Newspapers
  ods in Natural Language Processing, pages 3570–               1970–2016. Javnost, 26(2):138–157.
  3580.
                                                              Shanto Iyengar. 1990. Framing responsibility for polit-
Stephanie A. Fryberg, Nicole M. Stephens, Rebecca               ical issues: The case of poverty. Political Behavior,
   Covarrubias, Hazel Rose Markus, Erin D. Carter,              12(1):19–40.
   Giselle A. Laiduc, and Ana J. Salido. 2012. How the
   Media Frames the Immigration Debate: The Critical          Shanto Iyengar. 1991. Is anyone responsible? How
   Role of Location and Politics. Analyses of Social            television frames political issues. University of
   Issues and Public Policy, 12(1):96–112.                      Chicago Press.
Susan Gal. 1978. Peasant men can’t get wives: Lan-            Shanto Iyengar, Helmut Norpoth, and Kyu S Hahn.
  guage change and sex roles in a bilingual community.          2004. Consumer demand for election news: The
  Language in society, 7(1):1–16.                               horserace sells. The Journal of Politics, 66(1):157–
                                                                175.
Jesse Graham, Jonathan Haidt, and Brian A Nosek.
   2009. Liberals and conservatives rely on different         Kristen Johnson, Di Jin, and Dan Goldwasser. 2017.
   sets of moral foundations. Journal of personality            Modeling of political discourse framing on Twitter.
   and social psychology, 96(5):1029.                           In Proceedings of the International AAAI Confer-
Josh Grimm and Julie L. Andsager. 2011. Framing im-             ence on Web and Social Media, volume 11.
   migration: Geo-ethnic context in california newspa-
   pers. Journalism and Mass Communication Quar-              John T Jost, Jack Glaser, Arie W Kruglanski, and
   terly, 88(4):771–788.                                        Frank J Sulloway. 2003. Political conservatism as
                                                                motivated social cognition. Psychological bulletin,
Kimberly Gross. 2008.      Framing persuasive ap-               129(3):339.
  peals: Episodic and thematic framing, emotional re-
  sponse, and policy opinion. Political Psychology,           Shima Khanehzar, Andrew Turpin, and Gosia Mikola-
  29(2):169–192.                                                jczak. 2019. Predicting Political Frames Across Pol-
                                                                icy Issues and Contexts. In Proceedings of the 17th
Mareike Hartmann, Tallulah Jansen, Isabelle Augen-              Workshop of the Australasian Language Technology
 stein, and Anders Søgaard. 2019. Issue framing in              Association, pages 101–106.
 online discussion fora. In Proceedings of the 2019
 Conference of the North American Chapter of the              Klaus Krippendorff. 2013. Content Analysis: An Intro-
 Association for Computational Linguistics: Human               duction to Its Methodology. Sage Publications.
Haewoon Kwak, Jisun An, and Yong-Yeol Ahn. 2020.            Innocent Ndubuisi-Obi, Sayan Ghosh, and David Ju-
  A systematic media frame analysis of 1.5 million             rgens. 2019. Wetin dey with these comments?
  New York Times articles from 2000 to 2017. In 12th           Modeling sociolinguistic factors affecting code-
  ACM Conference on Web Science, pages 305–314.                switching behavior in Nigerian online discussions.
                                                               In Proceedings of the 57th Annual Meeting of the
Andrea Lawlor. 2015. Local and National Accounts              Association for Computational Linguistics, pages
  of Immigration Framing in a Cross-national Per-              6204–6214.
  spective. Journal of Ethnic and Migration Studies,
  41(6):918–941.                                            Zizi Papacharissi and Maria De Fatima Oliveira. 2008.
                                                              News frames terrorism: A comparative analysis
Sophie Lecheler, Linda Bos, and Rens Vliegenthart.            of frames employed in terrorism coverage in U.S.
  2015. The mediating role of emotions: News fram-            and U.K. newspapers. International Journal of
  ing effects on opinions about immigration. Journal-         Press/Politics, 13(1):52–74.
  ism & Mass Communication Quarterly, 92(4):812–            Shamik Roy and Dan Goldwasser. 2020. Weakly su-
  838.                                                        pervised learning of nuanced frames for analyzing
                                                              polarization in news media. In Proceedings of the
Michael T Light and Ty Miller. 2018. Does undocu-             2020 Conference on Empirical Methods in Natural
  mented immigration increase violent crime? Crimi-           Language Processing, pages 7698–7716.
  nology, 56(2):370–401.
                                                            W. Russell Neuman, Lauren Guggenheim, S. Mo Jang,
Siyi Liu, Lei Guo, Kate Mays, Margrit Betke, and              and Soo Young Bae. 2014. The Dynamics of Public
   Derry Tanti Wijaya. 2019a. Detecting frames in             Attention: Agenda-Setting Theory Meets Big Data.
   news headlines and its application to analyzing news       Journal of Communication, 64(2):193–214.
   framing trends surrounding US gun violence. In
  Proceedings of the 23rd Conference on Computa-            Dietram A. Scheufele. 1999.     Framing as a the-
   tional Natural Language Learning, pages 504–514.           ory of media effects. Journal of Communication,
                                                              49(1):103–122.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-
  dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,             Holli A Semetko and Patti M Valkenburg. 2000. Fram-
  Luke Zettlemoyer, and Veselin Stoyanov. 2019b.              ing European politics: A content analysis of press
  RoBERTa: A Robustly Optimized BERT Pretrain-                and television news. Journal of communication,
  ing Approach. arXiv preprint arXiv:1907.11692.              50(2):93–109.
                                                            Philippa Shoemark, Debnil Sur, Luke Shrimpton, Iain
Sharon Meraz and Zizi Papacharissi. 2013. Net-                Murray, and Sharon Goldwater. 2017. Aye or naw,
  worked Gatekeeping and Networked Framing on                 whit dae ye hink? scottish independence and lin-
  #Egypt. The International Journal of Press/Politics,        guistic identity on social media. In Proceedings of
  18(2):138–166.                                              the 15th Conference of the European Chapter of the
                                                              Association for Computational Linguistics, pages
James Milroy. 2001. Language ideologies and the con-          1239–1248.
  sequences of standardization. Journal of sociolin-
  guistics, 5(4):530–555.                                   Eugenia Siapera, Moses Boudourides, Sergios Lenis,
                                                              and Jane Suiter. 2018. Refugees and Network
Joshua R Minot, Michael V Arnold, Thayer Alshaabi,            Publics on Twitter: Networked Framing, Affect, and
   Christopher M Danforth, and Peter Sheridan Dodds.          Capture. Social Media and Society, 4(1).
   2020. Ratioing the president: An exploration of pub-
   lic engagement with Obama and Trump on Twitter.          Robert Courtney Smith. 2017. Dont let the illegals
   arXiv preprint arXiv:2006.03526.                           vote!: The myths of illegal latino voters and voter
                                                              fraud in contested local immigrant integration. The
Alan Mislove, Sune Lehmann, Yong-Yeol Ahn, Jukka-             Russell Sage Foundation Journal of the Social Sci-
  Pekka Onnela, and James Rosenquist. 2011. Under-            ences, 3(4):148–175.
  standing the demographics of Twitter users. In Pro-
                                                            Francesco Somaini. 2019. News stories framed episod-
  ceedings of the International AAAI Conference on
                                                              ically offer more diversified portrayals of immi-
  Web and Social Media, volume 5.
                                                              grants. Newspaper Research Journal, 40(2):190–
                                                              210.
Fred Morstatter, Liang Wu, Uraz Yavanoglu,
  Stephen R. Corman, and Huan Liu. 2018. Iden-              Burton Speakman and Marcus Funk. 2020. News, na-
  tifying Framing Bias in Online News.         ACM            tionalism, and hegemony: The formation of consis-
  Transactions on Social Computing, 1(2):1–18.                tent issue framing throughout the U.S. political right.
                                                              Mass Communication and Society, 23(5):656–681.
Nona Naderi and Graeme Hirst. 2017. Classifying
  frames at the sentence level in news articles. Interna-   Ian Stewart, Yuval Pinter, and Jacob Eisenstein. 2018.
  tional Conference Recent Advances in Natural Lan-            Si o no, que penses? Catalonian independence and
  guage Processing, RANLP, 2017-Septe:536–542.                 linguistic identity on social media. In Proceedings
of the 2018 Conference of the North American Chap-
  ter of the Association for Computational Linguistics:
  Human Language Technologies, Volume 2 (Short Pa-
  pers), pages 136–141.
Kjersti Thorbjørnsrud. 2015. Framing irregular immi-
  gration in western media. American Behavioral Sci-
  entist, 59(7):771–782.
Adriano Udani and David C Kimball. 2018. Immigrant
  resentment and voter fraud beliefs in the US elec-
  torate. American Politics Research, 46(3):402–433.
Baldwin van Gorp. 2005. Where is the frame? : Vic-
  tims and intruders in the Belgian press coverage of
  the asylum issue. European Journal of Communica-
  tion, 20(4):484–507.
A              Frame distribution in annotated data                                          C                Frame detection performance
Figure 5 shows the distribution of frames as a frac-                                         Tables 5-8 and Figures 8-9 provide details about
tion of total tweets in the annotated data.                                                  the fine-tuned RoBERTa models’ performance.

                                             Frame Type                                           Frame Type                      Precision      Recall          F1-score   LRAP
                        Issue-Generic           Issue-Specific             Narrative         Issue-Generic Policy                   0.722        0.727            0.716     0.745
              Political Factors                                                                  Issue-Specific                     0.667        0.493            0.550     0.785
           Policy Prescription
              Cultural Identity                                                                     Narrative                       0.780        0.884            0.825     0.896
                     Economic
        Crime & Punishment
              Health & Safety
          Fairness & Equality                                                                            Table 5: Performance by frame type on dev set.
          Security & Defense
             Morality & Ethics
                       Legality
          External Regulation                                                                                                Issue-Generic           Issue-Specific     Narrative
                Quality of Life
       Capacity & Resources                                                                     Human-Machine                    0.443                   0.488           0.421
             Public Sentiment                                                                   Human-Human                      0.417                   0.491           0.458
         Threat: Public Order
        Victim: Humanitarian
       Victim: Discrimination
                Threat: Fiscal                                                               Table 6: Average Krippendorff α agreement between
                 Hero: Worker
   Threat: National Cohesion                                                                 human annotators and machine-predicted labels (top
     Hero: Cultural Diversity
             Hero: Integration                                                               row) and between human annotator pairs (bottom row).
                  Threat: Jobs
     Victim: Global Economy                                                                  Overall, our classifiers had similar agreement with hu-
                   Victim: War
                      Episodic                                                               man annotators as humans did with one another.
                     Thematic
                                      0.0    0.1     0.2    0.3      0.4      0.5      0.6
                                    Fraction of tweets                                                                        Model performance per frame
       Figure 5: Distribution of frames in annotated data.
B             Inter-annotator agreement plots                                                           0.8

Figures 6 and 7 show inter-annotator agreement                                                          0.6
                                                                                             F1 Score

(Krippendorff’s α) across frame types.
                                                                                                        0.4

                                     Agreement With First Author
                                                                                                        0.2
            Coder 1
                                                                                                                                                                      RoBERTa FT
                                                                                                        0.0                                                           LogReg
                                                                                                              0         50         100         150         200        250       300
            Coder 2                                                                                                                       Support
Annotator

                                                                                             Figure 8: F1 score of logistic regression (1,2-gram
            Coder 3                                                                          features) and fine-tuned RoBERTa for each frame and
                                                                      Frame Type             frame support in evaluation sets. RoBERTa consis-
                                                                        Issue-Specific       tently outperforms logistic regression, especially for
            Coder 4                                                     Issue-Generic
                                                                        Narrative            low-frequency frames.
                  0.0    0.1          0.2       0.3      0.4        0.5        0.6
                                            Krippendorff Alpha
Figure 6: Inter-annotator agreement between first au-                                                                        US           GB              EU
thor and other coders before consensus-coding.                                                          0.8
                    Agreement With Consensus                                                            0.7
            Coder 1                                                                                     0.6
                                                                                                        0.5
                                                                                             F1 Score

            Coder 2                                                                                     0.4
                                                                                                        0.3
Annotator

            Coder 3                                                   Issue-Specific
                                                                      Issue-Generic                     0.2
                                                                      Narrative
                                                                                                        0.1
            Coder 4
                                                                                                        0.0
                                                                                                                  Issue-Generic          Issue-Specific             Narrative
            Coder 5                                                                                                                       Frame Type

                  0.0          0.2              0.4           0.6              0.8
                                                                                             Figure 9: Average F1 scores on combined dev/test set
                                            Krippendorff Alpha                               separated by region. Models achieve comparable per-
Figure 7: Agreement between each coder and consen-                                           formance for the United States, United Kingdom, and
sus annotations before consensus-coding.                                                     European Union, except for slightly lower performance
                                                                                             for issue-specific frames on EU tweets.
You can also read