Stepmothers are mean and academics are pretentious: What do pretrained language models learn about you?
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Stepmothers are mean and academics are pretentious: What do
pretrained language models learn about you?
Rochelle Choenni Ekaterina Shutova Robert van Rooij
University of Amsterdam University of Amsterdam University of Amsterdam
r.m.v.k.choenni@uva.nl e.shutova@uva.nl r.a.m.vanrooij@uva.nl
Abstract Sap et al., 2019). More broadly, biases in the mod-
els can comprise a wide range of harmful behaviors
Warning: this paper contains content that may that may affect different social groups for various
be offensive or upsetting.
reasons (Blodgett et al., 2020).
arXiv:2109.10052v1 [cs.CL] 21 Sep 2021
In this paper, we investigate what types of
In this work, we take a different focus and study
stereotypical information are captured by pre-
trained language models. We present the first stereotypes that emerge within pretrained LMs in-
dataset comprising stereotypical attributes of a stead. While bias is a personal preference that can
range of social groups and propose a method be harmful when the tendency interferes with the
to elicit stereotypes encoded by pretrained ability to be impartial, stereotypes can be defined
language models in an unsupervised fashion. as a preconceived idea that (incorrectly) attributes
Moreover, we link the emergent stereotypes general characteristics to all members of a group.
to their manifestation as basic emotions as a
While the two concepts are closely related i.e.,
means to study their emotional effects in a
more generalized manner. To demonstrate how
stereotypes can evoke new biases or reinforce exist-
our methods can be used to analyze emotion ing ones, stereotypical thinking appears to be a cru-
and stereotype shifts due to linguistic experi- cial part of human cognition that often emerges im-
ence, we use fine-tuning on news sources as plicitly (Hinton, 2017). Hinton (2017) argued that
a case study. Our experiments expose how implicit stereotypical associations are established
attitudes towards different social groups vary through Bayesian principles, where the experience
across models and how quickly emotions and of their prevalence in the world of the perceiver
stereotypes can shift at the fine-tuning stage.
causes the association. Thus, as stereotypical asso-
ciations are not solely reflections of cognitive bias
1 Introduction
but also stem from real data, we suspect that our
Pretraining strategies for large-scale language mod- models, like human individuals, pick up on these
els (LMs) require unsupervised training on large associations. This is particularly true given that
amounts of human generated text data. While their knowledge is largely considered to be a reflec-
highly successful, these methods come at the cost tion of the data they are trained on. Yet, while we
of interpretability as it has become increasingly un- consider stereotypical thinking to be a natural side-
clear what relationships they capture. Yet, as their effect of learning, it is still important to be aware
presence in society increases, so does the impor- of the stereotypes that models encode. Psychology
tance of recognising the role they play in perpetu- studies show that beliefs about social groups are
ating social biases. In this regard, Bolukbasi et al. transmitted and shaped through language (Maass,
(2016) first discovered that contextualized word 1999; Beukeboom and Burgers, 2019). Thus, spe-
representations reflect gender biases captured in the cific lexical choices in downstream applications
training data. What followed was a suite of stud- not only reflect the model’s attitude towards groups
ies that aimed to quantify and mitigate the effect but may also influence the audience’s reaction to it,
of harmful social biases in word (Caliskan et al., thereby inadvertently propagating the stereotypes
2017) and sentence encoders (May et al., 2019). they capture (Park et al., 2020).
Despite these studies, it has remained difficult to Studies focused on measuring stereotypes in pre-
define what constitutes “bias”, with most work fo- trained models have thus far taken supervised ap-
cusing on “gender bias” (Manela et al., 2021; Sun proaches, relying on human knowledge of common
et al., 2019) or “racial bias” (Davidson et al., 2019; stereotypes about (a smaller set of) social groups
(Nadeem et al., 2020; Nangia et al., 2020). This, itchenko and Mohammad, 2018; May et al., 2019;
however, bears a few disadvantages: (1) due to the Tan and Celis, 2019), or relating model predictions
implicit nature of stereotypes, human defined ex- to gender stereotype lexicons (Field and Tsvetkov,
amples can only expose a subset of popular stereo- 2020). To the best of our knowledge, Nadeem
types, but will omit those that human annotators et al. (2020); Nangia et al. (2020) and Manela et al.
are unaware of (e.g. models might encode stereo- (2021) are the first to explicitly study stereotypes in
types that are not as prevalent in the real world); pretrained sentence encoders. While Manela et al.
(2) stereotypes vary considerably across cultures (2021) focus on gender stereotypes using the Wino-
(Dong et al., 2019), meaning that the stereotypes Bias dataset (Zhao et al., 2018), the other works
tested for will heavily depend on the annotator’s propose new crowdsourced datasets (i.e. StereoSet
cultural frame of reference; (3) stereotypes con- and Crowspair) with stereotypes that cover a wide
stantly evolve, making supervised methods diffi- range of social groups. All datasets, however, have
cult to maintain in practice. Therefore, similar to a similar set-up: they contain pairs of sentences
Field and Tsvetkov (2020), we advocate the need of which one is more stereotypical than the other.
for implicit approaches to expose and quantify bias Working in the language modeling framework, they
and stereotypes in pretrained models. evaluated whether the model "prefers" the stereo-
We present the first dataset of stereotypical at- typical sentence over the anti-stereotypical one. In
tributes of a wide range of social groups, com- contrast, we propose a different experimental setup
prising ∼ 2K attributes in total. Furthermore, we and introduce a new dataset that leverages search
propose a stereotype elicitation method that en- engines’ autocomplete suggestions for the acquisi-
ables the retrieval of salient attributes of social tion of explicit stereotypical attributes. Instead of
groups encoded by state-of-the-art LMs in an un- indirectly uncovering stereotypes through compari-
supervised manner. We use this method to test the son, our elicitation method directly retrieves salient
extent to which models encode the human stereo- attributes encoded in the models. Our technique
types captured in our dataset. Moreover, we are the is inspired by Kurita et al. (2019), but while they
first to demonstrate how training data at the fine- measure the LM probability for completing sen-
tuning stage can directly affect stereotypical associ- tences with the pronouns she and he specifically,
ations within the models. In addition, we propose we study the top k salient attributes without posing
a complementary method to study stereotypes in a any restrictions on what these could be. Moreover,
more generalized way through the use of emotion we are the first to include both monolingual and
profiles, and systematically compare the emerging multilingual models in our analysis.
emotion profiles for different social groups across
models. We find that all models vary considerably Stereotype-driven emotions Stereotypes are
in the information they encode, with some models constantly changing and identifying negative ones
being overall more negatively biased while others in particular, is an inherently normative process.
are mostly positive instead. Yet, in contrast to pre- While some stereotypes clearly imply disrespect
vious work, this study is not meant to advocate the (e.g., women are incompetent), others emerge from
need for debiasing. Instead, it is meant to expose excessive competence instead (e.g., Asians are
varying implicit stereotypes that different models good at math). Moreover, stereotypical content
incorporate and to bring awareness to how quickly is heavily influenced by the social pressures of so-
attitudes towards groups change based on contex- ciety at the time. Cuddy et al. (2009) argue that
tual differences in the training data used both at the no stereotype remains stable and predictable from
pretraining and fine-tuning stage. theoretical principles. Hence, many social psychol-
ogists have abandoned the study of stereotype con-
2 Related work tent to focus on systematic principles that general-
Previous work on stereotypes While studies ize across different specific instances of stereotypes
that explicitly focus on stereotypes have remained instead, presumably making them more stable over
limited in NLP, several works on bias touch upon time and place (Cuddy et al., 2009; Mackie et al.,
this topic (Blodgett et al., 2020). This includes, for 2000; Weiner, 1993). Similarly, we explore a more
instance, studying specific phenomena such as the robust approach to uncovering stereotypes in pre-
infamous ‘Angry Black Woman’ stereotype and the trained LMs by studying how stereotypes are more
‘double bind’ (Heilman et al., 2004) theory (Kir- generally manifested as varying emotion profiles in
Search query Category Social group Stereotypical attributes Search engine
Why are black people so race black people fast, athletic, hated, angry, loud Yahoo
Why are british people so race british people polite, pale, tall, reserved, cold DuckDuckGo
Why are californians so lifestyle californians entitled, rich, flaky, backstabby multiple
Why are comedians so profession comedians funny, sad, intelligent, depressed Google
Why are millennials so age millennials fragile, nostalgic, lonely, broke Google
Why is Norway so country Norway healthy, wealthy, happy, rich multiple
Why are conservatives so political conservatives angry, controlling, racist Yahoo
Why are mormons so religion mormons misunderstood, rich, succesful, nice multiple
Table 1: A few (random) examples of stereotypical attributes sampled from the dataset. Note that the number of
attributes vary as we do not retrieve the same number for each social group per search engine.
the models. Previous works show that groups evoke • ‘Why are [TGT] always so [ATTR]?’
different emotional profiles (Cottrell and Neuberg, • ‘Why are all [TGT] so [ATTR]?’
2005; Tapias et al., 2007; Mackie et al., 2000), and For countries we use:
a variety of theories link particular intergroup rela- • ‘Why is [TGT] so [ATTR] ?’
tions to distinct stereotype-driven emotions such as • ‘What makes [TGT] so [ATTR]?’
disgust and anger (Harris and Fiske, 2006, 2009). • ‘How come [TGT] is so [ATTR]?’
• ‘Why is [TGT] always so [ATTR]?’
3 Stereotypes from search engines • ‘Why are all people in [TGT] so [ATTR]?’
Retrieving human stereotypes in an implicit man- where [TGT] are social groups for which we
ner can be useful as people are likely to give more search stereotypes and [ATTR] is the salient at-
politically correct answers when asked for stereo- tribute with which the search engine completes the
types explicitly. Questions we ask to search en- sequence. We tested other (longer and more elab-
gines are often done in the comfort of our own orate) templates but we found that they did not
homes, making them likely to reflect true stereo- produce many autocomplete suggestions. In fact,
types that are out there in the real world (Stephens- we believe that the above queries are so success-
Davidowitz, 2018). When feeding search engines ful precisely because of their simplicity, given that
queries that already imply a stereotype about a spe- people are likely to keep search queries concise.
cific social group, the suggested completions are
Search engines Due to Google’s hate speech fil-
often salient (i.e. stereotypical) attributes of that
tering system the autocompletion feature is dis-
particular group e.g., search query: ‘Why are Asian
abled for frequently targeted groups e.g. black peo-
parents so’ → completion: ‘strict’. As the auto-
ple, Jewish people and members of the LGBTQ+
complete suggestions are based on popular human
community. Thus, we retrieve autocomplete sug-
generated queries, we exploit this property for the
gestions from 3 search engines: Google, Yahoo
acquisition of stereotypical attributes.
and DuckDuckGo. In many cases, identical com-
Social groups We use 382 social groups which pletions were given by multiple search engines.
we divide into 9 categories: age (11), gender (26), We sort these duplicate samples under the category
profession (115), race (99), country (73), religion ‘multiple engines’. We find that most negative (of-
(10), political (9), sexuality (12) and lifestyle (19). fensive) stereotypes are retrieved from Yahoo.
See Appendix C for the full lists of social groups
per category. The social groups include, but are not Pre-processing We clean up the dataset manu-
limited to, the ones used by Nadeem et al. (2020) ally, using the following procedure:
who collected Wikidata relation triples to automati-
cally retrieve target terms. 1. Remove noisy completions that do not result
in a grammatically correct sentence e.g. non
Templates To increase coverage, we design 5 adjectives.
slightly varying templates to retrieve autocomplete 2. Remove specific trend-sensitive references:
suggestions for. For groups pertaining to people e.g. to video games ‘why are asians so good
we use the templates: at league of legends’.
• ‘Why are [TGT] so [ATTR] ?’ 3. Remove neutral statements not indicative of
• ‘What makes [TGT] so [ATTR]?’ stereotypes e.g. ‘why are [TGT] so called’.
• ‘How come [TGT] are so [ATTR]?’ 4. We filter out completions consisting of mul-
tiple words.1 Yet, when possible, the input supports 100 lan- Table 2: Ranking:‘why are
old people so bad with’.
is altered such that only the key term has to guages. We include
be predicted by the model e.g., ‘Why are rus- both versions of a Prior Post
sians so x’, where x = good at playing chess model (i.e. Base and 1. memory 1. memory
2. math 2. alcohol
→ ‘Why are russians so good at x’, x = chess. Large) if available. 3. money 3. technology
Appendix A provides 4. children 4. dates
The final dataset contains ∼2K stereotypes about more details on the
274 social groups. The stereotypes are distributed models.
across categories as follows – profession: 713, race:
412, country: 396, gender: 198, age: 171, lifestyle: Stereotype elicitation method For each sample
123, political: 50, religion: 36. None of the search in our dataset we feed the model the template sen-
engines produce stereotypical autocomplete sug- tence and replace [ATTR] with the [MASK] to-
gestions for members of the LGBTQ+ community. ken. We then retrieve the top k = 200 model
In Table 1 we provide some examples from the predictions for the MASK token, and test how many
dataset. See Appendix B for more details on the of the stereotypes found by the search engines are
data acquisition and search engines. The full code also encoded in the LMs. We adapt the method
and dataset are publicly available.2 from Kurita et al. (2019) to rank the top k returned
model outputs based on their typicality for the re-
4 Correlating human stereotypes with spective social group. We quantify typicality by
salient attributes in pretrained models computing the log probability of the model proba-
bility for the predicted completion corrected for by
To test for human stereotypes, we propose a stereo- the prior probability of the completion e.g.:
type elicitation method that is inspired by cloze
testing, a technique that stems from psycholinguis- Ppost (y = strict|Why are parents so y ?) (1)
tics. Using our method we retrieve salient attributes
from the model in an unsupervised manner and Pprior (y = strict|Why are [MASK] so y ?) (2)
compute recall scores over the stereotypes captured
in our search engine dataset. p = log(Ppost /Pprior ) (3)
Pretrained models We study different types i.e., measuring association between the words by
of pretrained LMs of which 3 are monolingual computing the chance of completing the template
and 2 multilingual: BERT (Devlin et al., 2019) with ‘strict’ given ‘parents’ corrected by the prior
uncased trained on the BooksCorpus dataset chance of ‘strict’ given any other group. Note that
(Zhu et al., 2015) and English Wikipedia; Eq. 3 has been well-established as a measure for
RoBERTa (Liu et al., 2019), the optimized ver- stereotypicality in research from both social psy-
sion of BERT that is in addition trained on chology (McCauley et al., 1980) and economics
data from CommonCrawl News (Nagel, 2016), (Bordalo et al., 2016). After re-ranking by typical-
OpenWebTextCorpus (Gokaslan and Cohen, ity, we evaluate how many of the stereotypes are
2019) and STORIES (Trinh and Le, 2018); BART, correctly retrieved by the model through recall@k
a denoising autoencoder (Lewis et al., 2020) that for each of the 8 target categories.
while using a different architecture and pretrain-
ing strategy from RoBERTa, uses the same train- Results Figure 1 shows the recall@k scores per
ing data. Moreover, we use mBERT, that apart model separated by category, showcasing the abil-
from being trained on Wikipedia in multiple lan- ity to directly retrieve stereotypical attributes of
guages, is identical to BERT. We use the uncased social groups using our elicitation method. While
version that supports 102 languages. Similarly, models capture the human stereotypes to similar
XLM-R is the multilingual variant of RoBERTa extents, results vary when comparing across cat-
(Conneau et al., 2020) that is trained on cleaned egories with most models obtaining the highest
CommonCrawl data (Wenzek et al., 2020) and recall for country stereotypes. Multilingual models
obtain relatively low scores when recalling stereo-
1
Although incompatible with our set-up, we do not remove typical attributes pertaining to age, gender and po-
them from the dataset as they can be valuable in future studies.
2
https://github.com/RochelleChoenni/ litical groups. Yet, XLMR-L is scoring relatively
stereotypes_in_lms high on stereotypical profession and race attributes.
Religion Age Country Gender
0.8 BERT-B 0.8 0.8 0.8
BERT-L
RoBERTa-B
0.6 RoBERTa-L 0.6 0.6 0.6
BART-B
Recall
0.4 BART-L 0.4 0.4 0.4
mBERT
XLMR-B
0.2 XLMR-L 0.2 0.2 0.2
0.0 0 25 50 75 100 125 150 175 200 0.0 0 25 50 75 100 125 150 175 200 0.0 0 25 50 75 100 125 150 175 200 0.0 0 25 50 75 100 125 150 175 200
k k k k
Political Profession Race Lifestyle
0.8 0.8 0.8 0.8
0.6 0.6 0.6 0.6
Recall
0.4 0.4 0.4 0.4
0.2 0.2 0.2 0.2
0.0 0 25 50 75 100 125 150 175 200 0.0 0 25 50 75 100 125 150 175 200 0.0 0 25 50 75 100 125 150 175 200 0.0 0 25 50 75 100 125 150 175 200
k k k k
Figure 1: Recall@k scores for recalling the human-defined stereotypes captured in our dataset using our stereotype
elicitation method on various pretrained LMs.
The suboptimal performance of multilingual mod- the social groups evoke those emotions. To this
els could be explained in different ways. For in- end, we (1) build emotion profiles for social groups
stance, as multilingual models are known to suffer in the models and (2) retrieve stereotypes about the
from negative interference (Wang et al., 2020), their groups that most strongly elicit emotions.
quality on individual languages is lower compared
Model predictions To measure the emotions en-
to monolingual models, due to limited model ca-
coded by the model, we feed the model the 5 stereo-
pacity. This could result in a loss of stereotypical
type eliciting templates for each social group and
information. Alternatively, multilingual models
retrieve the top 200 predictions for the [MASK]
are trained on more culturally diverse data, thus
token (1000 in total). When taking the 1000 salient
conflicting information could counteract within the
attributes retrieved from the 5 templates, we see
model with stereotypes from different languages
that there are many overlapping predictions, hence
dampening each other’s effect. Cultural differences
we are left with only approx. between 300-350
might also be more pronounced when it comes to
unique attributes per social group. This indicates
e.g. age and gender, whilst profession and race
that the returned model predictions are robust with
stereotypes might be established more universally.
regard to the different templates.
5 Quantifying emotion towards different Emotion scoring For each group, we score the
social groups predicted set of stereotypical attributes WT GT us-
ing the NRC emotion lexicon (Mohammad and
To study stereotypes through emotion, we draw Turney, 2013) that contains ∼ 14K English words
inspiration from psychology studies showing that that are manually annotated with Ekman’s eight ba-
stereotypes evoke distinct emotions based on dif- sic emotions (fear, joy, anticipation, trust, surprise,
ferent types of perceived threats (Cottrell and Neu- sadness, anger, and disgust) (Ekman, 1999) and
berg, 2005) or perceived social status and compet- two sentiments (negative and positive). These emo-
itiveness of the targeted group (Fiske, 1998). For tions are considered basic as they are thought to
instance, Cottrell and Neuberg (2005) show that be shaped by natural selection to address survival-
both feminists and African Americans elicit anger, related problems, which is often denoted as a driv-
but while the former group is perceived as a threat ing factor for stereotyping (Cottrell and Neuberg,
to social values, the latter is perceived as a threat to 2005). We use the annotations that consist of a
property instead. Thus, the stereotypes that under- binary value (i.e. 0 or 1) for each of the emotion
lie the emotion are likely different. Whilst strong categories; words can have multiple underlying
emotions are not evidence of stereotypes per se, emotions (e.g. selfish is annotated with ‘negative’,
they do suggest the powerful effects of subtle bi- ‘anger’ and ‘disgust’) or none at all (e.g. vocal
ases captured in the model. Thus, the study into scores 0 on all categories). We find that the cover-
emotion profiles provides us with a good starting age for the salient attributes in the NRC lexicon is
point to identify which stereotypes associated with ≈ 70-75 % per group.
RoBERTa-B BART-B
We score groups by counting the frequencies 60
ersoplends erseceRicocotsopleopleralsualsists
with which the predicted attributes WT GT are asso-
nagpe sbarippGreto s pe pe libesex ion
tepeoor hu st Puer hitelack omorelig
ciated with the emotions and sentiments. For each 50
group, we remove attributes from WT GT that are
w b h
40
not covered in the lexicon. Thus, we do not extract
emotion scores for the exact same number of at- 30
tributes per group (number of unique attributes and 20
coverage in the lexicon vary). Thus, we normalize
scores per group by the number of words for which 10
neg.
anger
fear
pos.
disgust
sad
trust
joy
neg.
anger
fear
pos.
disgust
sad
trust
joy
we are able to retrieve emotion scores (≈ 210-250
per group). The score of an emotion-group pair is
computed as follows: Figure 2: Examples of emotion profiles for a diverse
set of social groups from RoBERTa-B and BART-B.
|WT GT |
X
semo (TGT) = NRCemo (i)/(|WT GT |) (4)
i=w reliable, noble and responsible contributing to trust
for husbands. Interestingly, all multilingual mod-
We then define emotion vectors v̂ ∈ R10 for each els encode vastly more positive attributes for all
group T GT : v̂T GT = [sf ear , sjoy , ssadness , strust social groups (see Apppendix D). We expect that
, ssurprise , santicipation , sdisgust , sanger , snegative , this might be an artefact of the training data, but
, spositive ], which we use as a representation for leave further investigation of this for future work.
the emotion profiles within the model.
Comparison across models We systematically
Analysis Figure 2, provides examples of the emo- compare the emotion profiles elicited by the so-
tion profiles encoded for a diverse set of social cial groups across different models by adapting
groups to demonstrate how these profiles allow us the Representational Similarity Analysis (RSA)
to expose stereotypes. For instance, we see that from Kriegeskorte et al. (2008). We opted for this
in RoBERTa-B religious people and liberals are method as it takes the relative relations between
primarily associated with attributes that underlie groups within the same model into account. This
anger. Towards homosexuals, the same amount of is particularly important as we have seen that some
anger is accompanied by disgust and fear as well. models are overall more negatively or positively
As a result, we can detect distinct salient attributes biased. Yet, when it comes to bias and stereotypi-
that contribute to these emotions e.g.: Christians cality, we are less interested in absolute differences
are intense, misguided and perverse, liberals are across models, but rather in how emotions differ to-
phony, mad and rabid, whilst homosexuals are dirty, wards groups in relation to the other groups. First,
bad, filthy, appalling, gross and indecent. The find- the representational similarity within each model
ing that homosexuals elicit relatively much disgust is defined using a similarity measure to construct a
can be confirmed by studies on humans as well representational similarity matrix (RSM). We de-
(Cottrell and Neuberg, 2005). Similarly, we find fine a similarity vector ŵT GT for a social group
that Greece and Puerto Rico elicit relatively much such that every element ŵij of the vector is deter-
fear and sadness in RoBERTa-B. Whereas Puerto mined by the cosine similarity between v̂i , where
Rico is turbulent, battered, armed, precarious and i = TGT, and the vector v̂j for the j-th group in the
haunted, for Greece we find attributes such as fail- list. The RSM is then defined as the symmetric ma-
ing, crumbling, inefficient, stagnant and paralyzed. trix consisting of all similarity vectors. The result-
Emotion profiles elicited in BART-B differ con- ing matrices are then compared across models by
siderably, showcasing how vastly sentiments vary computing the Spearman correlation (ρ) between
across models. In particular, we see that overall the similarity vectors corresponding to the emotion
the evoked emotion responses are weaker. More- profiles for a group in a model a and b. To express
over, we detect relative differences such as liberals the similarity between the two models we take the
being more negatively associated than homosexu- mean correlation over all social groups in our list.
als, encoding attributes such as cowardly, greedy
and hypocritical. We also find that BART-B en- Results Computing RSA over all categories com-
codes more positive associations e.g., committed, bined, shows us that RoBERTa-B and BART-B ob-N
BERewYorGuard Reut FoxNeBreitb
N
BERewYorGuard Reut FoxNeBreitb 0.7 Reuters 0.7 New Yorker
T-B ker ian ers ws art T-B ker ian ers ws art
0.6 0.6
0 BERT-B
0.5 0.5
NewYorker
0.4 0.4
1
Recall
2 Guardian 0.3 BERT-B 0.3 BERT-B
reuters-100 new_yorker-100
3 Reuters 0.2 reuters-50 0.2 new_yorker-50
4 FoxNews 0.1 reuters-25 0.1 new_yorker-25
reuters-10 new_yorker-10
5 Breitbart 0.0 0 25 50 75 100 125 150 175 200 0.0 0 25 50 75 100 125 150 175 200
k k
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1.0
Gender - Spearman Correlation Age - Spearman Correlation
Figure 4: Effect on recall@k when fine-tuning BERT-B
Figure 3: Correlations in emotion profiles for gender on 10, 25, 50 and 100 % of the data
and age groups across news sources (BERT-B).
Source
tain the highest correlation (ρ = 0.44). While 0.6 New Yorker
Guardian
using different architectures and pretraining strate- Reuters
0.5 FOX news
Breitbart
Spearman Correlation
gies, the models rely on the same training data. Yet,
0.4
we included base and large versions of models in
0.3
our study and find that these models show little to
no correlation (see Appendix E, Fig.10). This is 0.2
surprising, as they are pretrained on the same data 0.1
and tasks as their base versions (but contain more 0.0
10% 25% 50% 100%
model parameters e.g. through additional layers). Proportion of data
This shows how complex the process is in which
Figure 5: Decrease in Spearman correlation (∆ρ) after
associations are established and provides strong
fine-tuning the pretrained models compared to no fine-
evidence that other modelling decisions, apart from tuning (∆ρ = 1) (no correlation left:∆ρ = −1). We
training data, contribute to what models learn about show results for models trained on varying proportions
groups. Thus, carefully controlling training content of the data. Results are averaged over categories and
can not fully eliminate the need to analyze models standard deviations are indicated by error bars.
w.r.t. the stereotypes that they might propagate.
6 Stereotype shifts during fine-tuning methods, including editorial reviews, blind bias sur-
veys, and third party research. Based on these rat-
Many debiasing studies intervene at the data level ings we select the following sources: New Yorker
e.g., by augmenting imbalanced datasets (Manela (far left), The Guardian (left), Reuters (center),
et al., 2021; Webster et al., 2018; Dixon et al., 2018; FOX News (right) and Breitbart (far right). From
Zhao et al., 2018) or reducing annotator bias (Sap each news source we take 4354 articles from the
et al., 2019). These methods are, however, depen- All-The-News4 dataset that contains articles from
dent on the dataset, domain, or task, making new 27 American Publications collected between 2013
mitigation needed when transferring to a new set- and early 2020. We fine-tune the 5 base models5
up (Jin et al., 2020). This raises the question of how on these news sources using the MLM objective
emotion profiles and stereotypes are established for only 1 training epoch with a learning rate of
through language use, and how they might shift 5e-5 and a batch size of 8 using the HuggingFace
due to new linguistic experience at the fine-tuning library (Wolf et al., 2020). We then quantify the
stage. We take U.S. news sources from across the emotion shift after fine-tuning using RSA.
political spectrum as a case study, as media outlets
are known to be biased (Baron, 2006). By revealing Results We find that fine-tuning on news sources
stereotypes learned as an effect of fine-tuning on can directly alter the encoded stereotypes. For in-
a specific source, we can trace the newly learned stance, for k = 25, fine-tuning BERT-B on Reuters
stereotypes back to the respective source. informs the model that Croatia is good at sports
We rely on the political bias categorisation of and Russia is good at hacking, at the same time,
news sources from the AllSides 3 media bias rating associations such as Pakistan is bad at football, Ro-
website. These ratings are retrieved using multiple mania is good at gymnastics and South Africa at
3 4
https://www.allsides.com/media-bias/ Available at: https://tinyurl.com/bx3r3de8
5
media-bias-ratings Training the large models was computationally infeasible.Guardian Reuters FoxNews
s n s s q e s s s s s s
s n s s q e s s s s s s
s n s s q e s s s s s s
60
er e ie an ra pl ic er tie ve er an
er e ie an ra pl ic er tie ve er an
er e ie an ra pl ic er tie ve er an
ag k m lad si I eo em fic ri ati ort sti
ag k m lad si I eo em fic ri ati ort sti
ag k m lad si I eo em fic ri ati ort sti
en c a ay p cad e ofeleberv upp chri
en c a ay p cad e ofeleberv upp chri
en c a ay p cad e ofeleberv upp chri
50
g a lic c ns s
g a lic c ns s
g a lic c ns s
po coump
po coump
po coump
40
tr
tr
tr
30
20
10
neg.
pos.
disgust
anger
fear
sad
trust
joy
neg.
fear
pos.
disgust
anger
sad
trust
joy
neg.
anger
fear
pos.
disgust
sad
trust
joy
te bla
te bla
te bla
Figure 6: A few interesting examples of emotional profiles for a diverse set of social group after fine-tuning
RoBERTa-B for only 1 training epoch on articles from Guardian, Reuters and FOX news respectively.
center bias rating i.e., it does not predictably fa-
vor either end of the political spectrum, we specu-
late that large amounts of more nuanced data helps
transmit fewer stereotypes.
Figure 5 shows the decrease in correlation be-
tween the emotion profiles from pretrained BERT-
B and BERT-B fine-tuned on different propor-
tions of the data. Interestingly, fine-tuning on less
articles does not automatically result in smaller
changes to the models. In fact, in many cases, the
amount of relative change in emotion profiles is
heavily dependent on the social category as indi-
cated by the error bars. This is not unexpected as
news sources might propagate stronger opinions
about specific categories. Moreover, we find that
emotions towards different social categories cannot
always be distinguished by the political bias of the
Figure 7: Stereotypical attribute shifts when fine- source. Figure 3, shows how news sources com-
tuning RoBERTa-B on New Yorker (left) and FOX pare to each other w.r.t. different social categories,
news (right). Removed attributes are red and those
exposing that e.g. Guardian and FOX news show
added green. Attributes that persisted are grey.
lower correlation on gender than on age.
Computing correlation between all pretrained
and fine-tuned models, we find that emotion pro-
rugby are lost. Moreover, from fine-tuning on both files are prone to change irrespective of model or
Breitbart and FOX news the association emerges news source (see Appendix E). In Figure 6, we
that black women are violent, while this is not the showcase the effect of fine-tuning from the model
case when fine-tuning on the other sources. that exhibits the lowest change in correlation, i.e.
In fact, Guardian and Breitbart are the only news RoBERTa-B, to highlight how quickly emotions
sources that result in the encoding of the salient shift. We find that while Reuters results in weaker
attribute racist for White Americans. We find that emotional responses, Guardian elicits stronger neg-
such shifts are already visible after training on as ative emotions than FOX news e.g. towards con-
little as 25% of the original data (∼ 1K articles). servatives and academics. Yet, while both sources
When comparing to human stereotypes, we find result in anger towards similar groups, for FOX
that fine-tuning on Reuters decreases the overall news anger is more often accompanied with fear
recall scores (see Figure 4). Although New Yorker while for Guardian this seems to more strongly
exhibits a similar trend, fine-tuning on the other stems from disgust (e.g. see Christians and Iraq).
sources have little effect on the number of stereo- Lastly, Figure 7 shows specific stereotype shifts
types recalled from the dataset. As Reuters has a found on the top 15 predictions per template. Weillustrate the salient attributes that are removed, dition, our data acquisition process is completely
added and remained constant after fine-tuning. For automated and did not require the help from human
instance, the role of news media in shaping public subjects. While the stereotypes we retrieve stem
opinion about police has received much attention from real humans, the data we collect is publicly
in the wake of the growing polarization over high- available and completely anonymous as the specific
profile incidents (Intravia et al., 2018; Graziano, stereotypical attributes and/or search queries can
2019). We find clear evidence of this polarization not be traced back to individual users.
as fine-tuning on New Yorker results in attributes
such as cold, unreliable, deadly and inept, yet, fine-
tuning on FOX news yields positive associations References
such as polite, loyal, cautious and exceptional. In David P Baron. 2006. Persistent media bias. Journal
addition, we find evidence for other stark contrasts of Public Economics, 90(1-2):1–36.
such as the model picking up on sexist (e.g. women Camiel J Beukeboom and Christian Burgers. 2019.
are not interesting and equal but late, insecure and How stereotypes are shared through language: a re-
entitled) and racist stereotypes (e.g. black peo- view and introduction of the social categories and
ple are not misunderstood and powerful, but bitter, stereotypes communication (scsc) framework. Re-
view of Communication Research, 7:1–37.
rude and stubborn) after fine-tuning on FOX news.
Su Lin Blodgett, Solon Barocas, Hal Daumé III, and
7 Conclusion Hanna Wallach. 2020. Language (technology) is
power: A critical survey of “bias” in nlp. In Proceed-
We present the first dataset containing stereotyp- ings of the 58th Annual Meeting of the Association
ical attributes of a range of social groups. Im- for Computational Linguistics, pages 5454–5476.
portantly, our data acquisition technique enables Tolga Bolukbasi, Kai-Wei Chang, James Zou,
the inexpensive retrieval of similar datasets in the Venkatesh Saligrama, and Adam Kalai. 2016.
future, enabling comparative analysis on stereo- Man is to computer programmer as woman is to
type shifts over time. Additionally, our proposed homemaker? debiasing word embeddings. In
Proceedings of the 30th International Conference
methods could inspire future work on analyzing on Neural Information Processing Systems, pages
the effect of training data content, and simultane- 4356–4364.
ously contribute to the field of social psychology
Pedro Bordalo, Katherine Coffman, Nicola Gennaioli,
by providing a testbed for studies on how stereo- and Andrei Shleifer. 2016. Stereotypes. The Quar-
types emerge from linguistic experience. To this terly Journal of Economics, 131(4):1753–1794.
end, we have shown that our methods can be used
to identify stereotypes evoked during fine-tuning Aylin Caliskan, Joanna J Bryson, and Arvind
Narayanan. 2017. Semantics derived automatically
by taking news sources as a case study. More- from language corpora contain human-like biases.
over, we have exposed how quickly stereotypes Science, 356(6334):183–186.
and emotions shift based on training data content,
Alexis Conneau, Kartikay Khandelwal, Naman Goyal,
and linked stereotypes to their manifestations as Vishrav Chaudhary, Guillaume Wenzek, Francisco
emotions to quantify and compare attitudes towards Guzmán, Edouard Grave, Myle Ott, Luke Zettle-
groups within LMs. We plan to extent our approach moyer, and Veselin Stoyanov. 2020. Unsupervised
to more languages in future work to collect differ- cross-lingual representation learning at scale. In
Proceedings of the 58th Annual Meeting of the As-
ent, more culturally dependent, stereotypes as well.
sociation for Computational Linguistics.
8 Ethical consideration Catherine A Cottrell and Steven L Neuberg. 2005.
Different emotional reactions to different groups:
The examples given in the paper can be considered a sociofunctional threat-based approach to" preju-
offensive but are in no way a reflection of the au- dice". Journal of personality and social psychology,
thors’ own values and beliefs and should not be 88(5):770.
taken as such. Moreover, it is important to note that Amy JC Cuddy, Susan T Fiske, Virginia SY Kwan,
for the fine-tuning experiments only a few interest- Peter Glick, Stéphanie Demoulin, Jacques-Philippe
ing examples were studied and showcased. Hence, Leyens, Michael Harris Bond, Jean-Claude Croizet,
Naomi Ellemers, Ed Sleebos, et al. 2009. Stereo-
more thorough research should be conducted be- type content model across cultures: Towards univer-
fore drawing any hard conclusions about the news sal similarities and some differences. British Jour-
papers and the stereotypes they propagate. In ad- nal of Social Psychology, 48(1):1–33.Thomas Davidson, Debasmita Bhattacharya, and Ing- Jonathan Intravia, Kevin T Wolff, and Alex R Piquero.
mar Weber. 2019. Racial bias in hate speech and 2018. Investigating the effects of media consump-
abusive language detection datasets. In Proceedings tion on attitudes toward police legitimacy. Deviant
of the Third Workshop on Abusive Language Online, Behavior, 39(8):963–980.
pages 25–35.
Xisen Jin, Francesco Barbieri, Brendan Kennedy,
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Aida Mostafazadeh Davani, Leonardo Neves, and
Kristina Toutanova. 2019. BERT: Pre-training of Xiang Ren. 2020. On transferability of bias miti-
Deep Bidirectional Transformers for Language Un- gation effects in language model fine-tuning. arXiv
derstanding. In Proceedings of the 2019 Conference preprint arXiv:2010.12864.
of the North American Chapter of the Association
for Computational Linguistics: Human Language Svetlana Kiritchenko and Saif Mohammad. 2018. Ex-
Technologies, Volume 1 (Long and Short Papers), amining gender and race bias in two hundred sen-
pages 4171–4186. timent analysis systems. In Proceedings of the
Seventh Joint Conference on Lexical and Computa-
Lucas Dixon, John Li, Jeffrey Sorensen, Nithum Thain, tional Semantics, pages 43–53.
and Lucy Vasserman. 2018. Measuring and mitigat-
ing unintended bias in text classification. In Pro- Nikolaus Kriegeskorte, Marieke Mur, and Peter A Ban-
ceedings of the 2018 AAAI/ACM Conference on AI, dettini. 2008. Representational similarity analysis-
Ethics, and Society, pages 67–73. connecting the branches of systems neuroscience.
Frontiers in systems neuroscience, 2:4.
MeiXing Dong, David Jurgens, Carmen Banea, and Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black,
Rada Mihalcea. 2019. Perceptions of social roles and Yulia Tsvetkov. 2019. Measuring bias in contex-
across cultures. In International Conference on So- tualized word representations. In Proceedings of the
cial Informatics, pages 157–172. Springer. First Workshop on Gender Bias in Natural Language
Processing, pages 166–172.
Paul Ekman. 1999. Basic emotions. Handbook of Cog-
nition and Emotion, pages 45–60. Mike Lewis, Yinhan Liu, Naman Goyal, Mar-
jan Ghazvininejad, Abdelrahman Mohamed, Omer
Anjalie Field and Yulia Tsvetkov. 2020. Unsupervised Levy, Veselin Stoyanov, and Luke Zettlemoyer.
discovery of implicit gender bias. In Proceedings of 2020. Bart: Denoising sequence-to-sequence pre-
the 2020 Conference on Empirical Methods in Natu- training for natural language generation, translation,
ral Language Processing (EMNLP), pages 596–608. and comprehension. In Proceedings of the 58th An-
nual Meeting of the Association for Computational
Susan T Fiske. 1998. Stereotyping, prejudice, and dis-
Linguistics, pages 7871–7880.
crimination. The handbook of social psychology,
2(4):357–411. Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man-
dar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Aaron Gokaslan and Vanya Cohen. 2019. Openweb- Luke Zettlemoyer, and Veselin Stoyanov. 2019.
text corpus. https://skylion007.github. Roberta: A robustly optimized bert pretraining ap-
io/OpenWebTextCorpus/. proach. arXiv preprint arXiv:1907.11692.
Lisa M Graziano. 2019. News media and perceptions Anne Maass. 1999. Linguistic intergroup bias: Stereo-
of police: a state-of-the-art-review. Policing: An In- type perpetuation through language. In Advances in
ternational Journal. experimental social psychology, volume 31, pages
79–121. Elsevier.
Lasana T Harris and Susan T Fiske. 2006. Dehu-
manizing the lowest of the low: Neuroimaging re- Diane M Mackie, Thierry Devos, and Eliot R Smith.
sponses to extreme out-groups. Psychological sci- 2000. Intergroup emotions: Explaining offensive ac-
ence, 17(10):847–853. tion tendencies in an intergroup context. Journal of
personality and social psychology, 79(4):602.
Lasana T Harris and Susan T Fiske. 2009. Social neu-
roscience evidence for dehumanised perception. Eu- Daniel de Vassimon Manela, David Errington, Thomas
ropean review of social psychology, 20(1):192–231. Fisher, Boris van Breugel, and Pasquale Minervini.
2021. Stereotype and skew: Quantifying gender
Madeline E Heilman, Aaron S Wallen, Daniella Fuchs, bias in pre-trained and fine-tuned language models.
and Melinda M Tamkins. 2004. Penalties for suc- arXiv preprint arXiv:2101.09688.
cess: reactions to women who succeed at male
gender-typed tasks. Journal of applied psychology, Chandler May, Alex Wang, Shikha Bordia, Samuel
89(3):416. Bowman, and Rachel Rudinger. 2019. On measur-
ing social biases in sentence encoders. In Proceed-
Perry Hinton. 2017. Implicit stereotypes and the pre- ings of the 2019 Conference of the North American
dictive brain: cognition and culture in “biased” per- Chapter of the Association for Computational Lin-
son perception. Palgrave Communications, 3(1):1– guistics: Human Language Technologies, Volume 1
9. (Long and Short Papers), pages 622–628.Clark McCauley, Christopher L Stitt, and Mary Segal. Kellie Webster, Marta Recasens, Vera Axelrod, and Ja-
1980. Stereotyping: From prejudice to prediction. son Baldridge. 2018. Mind the gap: A balanced
Psychological Bulletin, 87(1):195. corpus of gendered ambiguous pronouns. Transac-
tions of the Association for Computational Linguis-
Saif M Mohammad and Peter D Turney. 2013. Nrc tics, 6:605–617.
emotion lexicon. National Research Council,
Canada, 2. Bernard Weiner. 1993. On sin versus sickness: A the-
ory of perceived responsibility and social motivation.
Moin Nadeem, Anna Bethke, and Siva Reddy. American psychologist, 48(9):957.
2020. Stereoset: Measuring stereotypical bias
in pretrained language models. arXiv preprint Guillaume Wenzek, Marie-Anne Lachaux, Alexis Con-
arXiv:2004.09456. neau, Vishrav Chaudhary, Francisco Guzmán, Ar-
mand Joulin, and Édouard Grave. 2020. Ccnet: Ex-
Sebastian Nagel. 2016. Cc-news dataset. tracting high quality monolingual datasets from web
https://commoncrawl.org/2016/10/ crawl data. In Proceedings of The 12th Language
news-dataset-available/. Resources and Evaluation Conference, pages 4003–
4012.
Nikita Nangia, Clara Vania, Rasika Bhalerao, and
Samuel Bowman. 2020. Crows-pairs: A challenge Thomas Wolf, Lysandre Debut, Victor Sanh, Julien
dataset for measuring social biases in masked lan- Chaumond, Clement Delangue, Anthony Moi, Pier-
guage models. In Proceedings of the 2020 Con- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtow-
ference on Empirical Methods in Natural Language icz, Joe Davison, Sam Shleifer, Patrick von Platen,
Processing (EMNLP), pages 1953–1967. Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu,
Teven Le Scao, Sylvain Gugger, Mariama Drame,
Chan Young Park, Xinru Yan, Anjalie Field, and Yu- Quentin Lhoest, and Alexander M. Rush. 2020.
lia Tsvetkov. 2020. Multilingual contextual affec- Transformers: State-of-the-art natural language pro-
tive analysis of lgbt people portrayals in wikipedia. cessing. In Proceedings of the 2020 Conference on
arXiv preprint arXiv:2010.10820. Empirical Methods in Natural Language Processing:
Maarten Sap, Dallas Card, Saadia Gabriel, Yejin Choi, System Demonstrations, pages 38–45, Online. Asso-
and Noah A Smith. 2019. The risk of racial bias in ciation for Computational Linguistics.
hate speech detection. In Proceedings of the 57th Jieyu Zhao, Tianlu Wang, Mark Yatskar, Vicente Or-
annual meeting of the association for computational donez, and Kai-Wei Chang. 2018. Gender bias in
linguistics, pages 1668–1678. coreference resolution: Evaluation and debiasing
Seth Stephens-Davidowitz. 2018. Everybody Lies: methods. In Proceedings of the 2018 Conference of
What the internet can tell us about who we really the North American Chapter of the Association for
are. In Bloomsbury Publishing Plc. Computational Linguistics: Human Language Tech-
nologies, volume 2.
Tony Sun, Andrew Gaut, Shirlyn Tang, Yuxin Huang,
Mai ElSherief, Jieyu Zhao, Diba Mirza, Elizabeth Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhut-
Belding, Kai-Wei Chang, and William Yang Wang. dinov, Raquel Urtasun, Antonio Torralba, and Sanja
2019. Mitigating gender bias in natural language Fidler. 2015. Aligning books and movies: Towards
processing: Literature review. In Proceedings of the story-like visual explanations by watching movies
57th Annual Meeting of the Association for Compu- and reading books. In Proceedings of the IEEE inter-
tational Linguistics, pages 1630–1640. national conference on computer vision, pages 19–
27.
Yi Chern Tan and L Elisa Celis. 2019. Assessing so-
cial and intersectional biases in contextualized word
representations. arXiv preprint arXiv:1911.01485.
Molly Parker Tapias, Jack Glaser, Dacher Keltner, Kris-
ten Vasquez, and Thomas Wickens. 2007. Emotion
and prejudice: Specific emotions toward outgroups.
Group Processes & Intergroup Relations, 10(1):27–
39.
Trieu H Trinh and Quoc V Le. 2018. A simple
method for commonsense reasoning. arXiv preprint
arXiv:1806.02847.
Zirui Wang, Zachary C Lipton, and Yulia Tsvetkov.
2020. On negative interference in multilingual lan-
guage models. In Proceedings of the 2020 Confer-
ence on Empirical Methods in Natural Language
Processing (EMNLP), pages 4438–4450.A Pretrained model details
Model tokenization L dim H params V D task #lgs
BERT-B WordPiece 12 768 12 110M 30K 16GB MLM+NSP 1
BERT-L WordPiece 24 1024 16 336M 30K 16GB MLM+NSP 1
RoBERTa-B BPE 12 768 12 125M 50K 160GB MLM 1
RoBERTa-L BPE 24 1024 16 335M 50K 160GB MLM 1
BART-B BPE 12 768 16 139M 50K 160GB Denoising 1
BART-L BPE 24 1024 16 406M 50K 160GB Denoising 1
mBERT WordPiece 12 768 12 168M 110K - MLM+NSP 102
XLMR-B SentencePiece 12 768 8 270M 250K 2.5TB MLM 100
XLMR-L SentencePiece 24 1024 16 550M 250K 2.5TB MLM 100
Table 3: Summary statistics of the model architectures: tokenization method, number of layers L, hidden state
dimensionality dim, number of attention heads H, number of model parameters params, vocabulary size V ,
training data size D, pretraining tasks, and number of languages used #lgs.
B Data acquisition
For the collection of autocomplete suggestions we rely on the free publicly available API’s from the
respective engines using the following base url’s:
• Google: http://suggestqueries.google.com/complete/search
• Yahoo: http://sugg.search.yahoo.net/sg
• DuckDuckGo: https://duckduckgo.com/ac
All search engine suggestions are automatically generated by an algorithm without human involvement.
These suggestions are supposed to be based on factors like popularity and similarity. We enter the search
queries anonymously such that the resulting suggestions are mainly based on common queries from other
people’s search histories. Unfortunately, however, exact details about the workings of the algorithms are
not publicly available, but an extensive explanation of Google’s search predictions can be found here:
Google’s documentation on autocomplete suggestions. Moreover, Figure B illustrates the contribution of
each search engine to the datasets. We see that while each search engine relies on a different algorithm,
in many cases the engines predict similar stereotypical attributes regardless. Moreover, the dataset was
constructed during the period January-May 2021. However, given that the algorithms behind these engines
are constantly evolving, it is not guaranteed that the same approach will yield identical results in the future.
We will make the dataset and corresponding code available upon publication.
Sample distribution over search engines
1000
800
600
Samples
400
200
0
Google Yahoo duckduckgo multiple
Search engines
Figure 8: Distribution of the contribution of each search engine to the final dataset.C Lists of social groups per category Age (11) "elders", "kids", "millenials", "teens", "boomers", "old people" "old women", "children", "teenagers", "daughters", "sons" Gender (25) "girls, "women", "men", "females", "males", "boys" "boyfriends", "girlfriends", "stepmothers", "stepfathers", "ladies", "gentlemen" "brothers", "sisters", "mothers", "fathers", "grandfathers", ""grandmothers" "wives", "husbands", "brides", "grooms", "schoolgirls", "schoolboys" "mommies", "daddies" Race (99) "Asians", "Americans", "Europeans", "Jews", "Indians", "Russians", "Africans", "Black people" "Mexicans", "Whites" "Blacks", "White people", "Germans", "blondes", "blonde girls", "Arabs" "White Americans", "Black Americans" "Hispanics", "Native Americans", "Black men", "White men" "Asian women", "Asian men" Black women", "the Dutch", "Irish people", "Irish men", "White women" "Indian men", "Indian women", "Chinese men", "Chinese women", "Japanese women", "Japanese men" "Indian parents", "Asian parents", "White parents", "Black parents", "Black fathers", "Latinas", "Latinos" "Latin people", "Brazilian women","Asian kids", "Black kids", "White kids", "African Americans" "Nigerians", "Ethiopians", "Ukrainians", "Sudanese people", "Afghans", "Iraqis", "Hispanic men" "Hispanic women", "Italians", "Italian men", "Italian women", "Somalis", "Iranian people", "Iranians" "Australians", "Australian men", "Australian women", "Aussies", "Ghanaians", "Swedes", "Finns" "Venezuelans", "Moroccans", "Syrians", "Pakistanis", "British people", "French people", "Greeks" "Indonesians", "Vietnamese people", "Romanians", "Ecuadorians", "Norwegians", "Nepalis" , "Scots" "Bengalis", "Polish people", "Taiwanese people", "Albanians", "Colombians", "Egyptians", "Koreans" "Persian people", "Portuguese men", "Portuguese women", "Turkish people", "Austrians" "South Africans", "Dutch people", "Chileans", "Lebanese people" Countries (73) "Ethiopia", "Europe", "Ukraine", "Sudan", "Afghanistan", "Iraq", "Yemen", "Russia", "Italy", "Somalia" "Iran", "Venezuela", "Oman", "Syria", "China", "India", "Britain", "France", "Greece", "Scotland" "Paraguay", "Brazil", "Africa", "Jordan", "Indonesia", "Vietnam", "Pakistan", "Romania, "Ecuador" "Mexico", "Puerto Rico", "Cameroon", "Kenya", "Liberia", "Sierra Leon", "Qatar", "Syria", "Kenya" "Saudi Arabia", "Lebanon", "South Africa", "Korea", "Singapore", "Germany", "Ireland", "Morocco" "Turkey", "Laos", "Bangladesh", "Guatemala", "Ghana", "Nepal", "Albania", "Spain", "Sweden" "Argentina", "Chile", "Taiwan", "Finland", "Australia", "Egypt", "Peru", "Poland", "Columbia", "Bolivia" "Japan", "Norway", "Cape Verde", "Portugal", "Austria", "the Netherlands" "Croatia", "Malta", "Belgium" Profession (115) "students", "politicians" , "doctors", "business men", "librarians", "artists", "professors", "priests" "bosses", "police", "police officers", "soldiers", "scientists", "physicians", "cashiers" "housekeepers" "teachers", "janitors", "models", "actresses", "pilots", "strippers" "brokers", "hairdressers", "bartenders" "diplomats", "receptionists", "realtors", "mathematicians", "barbers", "coaches", "business people" "construction workers", "managers", "accountants", "commanders", "firefighters", "movers", "attorneys" "bakers", "athletes", "dancers", "carpenters", "mechanics", "handymen", "musicians", "detectives" "entrepreneurs", "opera singers", "chiefs", "lawyers", "software developers", "farmers", "writers" "real-estate developers", "butchers", "electricians", "prosecutors", "bankers", "cooks", "plumbers" "football players", "boxers", "chess players", "swimmers", "tennis players", "supervisors", "attendants" "producers", "researchers", "economists", "physicists", "psychologists", "sales people", "assistants" "engineers", "comedians", "painters", "civil servants", "guitarists", "linguists", "laborers", "historians" "chemists", "pensioners", "performing artists", "singers", "secretaries", "auditors", "counselors" "dentists", "analysts", "nurses", "waiters", "authors", "architects", "academics", "directors", "illustrators" "clerks", "photographers", "cleaners", "composers", "pharmacists", "sociologists", "journalists" "guards", "actors", "midwives", "sheriffs", "editors", "designers", "judges", "poets", "maids" Religion (10) "Religious people", "Muslims", "Christians", "Hindus", "atheists", "Buddhists" "Catholics", "Protestants", "Sikhs", "Mormons"
You can also read
