Beyond Anonymity: Network Affordances, Under Deindividuation, Improve Social Media Discussion Quality
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Journal of Computer-Mediated Communication
Beyond Anonymity: Network Affordances, Under
Deindividuation, Improve Social Media Discussion
Quality
1
Kokil Jaidka , Alvin Zhou2, Yphtach Lelkes2, Jana Egelhofer3, & Sophie Lecheler3
1
Department of Communications and New Media, National University of Singapore, Singapore 117416
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
2
Annenberg School for Communication, University of Pennsylvania, Philadelphia, USA
3
Department of Communication, University of Vienna, Vienna, Austria
The online sphere allows people to be personally anonymous while simultaneously being so-
cially identifiable. Twitter users can use a pseudonym but signal allegiance to a political party
in their profile (e.g., #MAGA). We explore the interplay of these two dimensions of anonym-
ity on a custom-built social media platform that allowed us to examine the causal effects of
personal and social anonymity on discussion quality. We find no support for the hypothesis
that personal anonymity breeds incivility or lowers discussion quality in discussions on gun
rights. On the other hand, when personal anonymity is combined with social identifiability
(operationalized as political party visibility), it improves several features linked to discussion
quality, that is, higher rationality and lower incivility. We discuss the mechanisms that might
explain the results and offer recommendations for future experiments about the design of so-
cial media platforms.
Lay Summary
How does our identifiability online affect the quality of our discussions? Social media allows peo-
ple to be anonymous in different ways. Scholars anticipate that when our personal information is
visible, such as their names and photographs, we engage in more civil discussions than when we
are personally anonymous. But when only our social identity information is on display, we feel
the need to conform to a group, and may behave more rationally. Our study tested these expecta-
tions with a custom-built experimental platform. We found support for only one of the claims.
That is, there was no difference in the discussion quality between settings where participants were
personally anonymous or personally identifiable. On the other hand, when only their social iden-
tity information was on display, the quality of the discussions was higher. Our findings can be
explained on the basis of previous research, which suggests that when people are socially identifi-
able as belonging to a group, it increases their need to be consistent with the normative behavior
of the group. Therefore they behave more rationally. We discuss our findings in terms of develop-
ing better instruments of discussion quality on social media platforms.
Corresponding author: Kokil Jaidka; e-mail: jaidka@nus.edu.sg
Editorial Record: First manuscript received on 17 July 2020; Revisions received on 4 September 2021; Accepted by R.
Kelly Garrett on 17 September 2021; Final manuscript received on 28 September 2021
Journal of Computer-Mediated Communication 00 (2021) 1–23 V C The Author(s) 2021. Published by Oxford University Press on behalf 1
of International Communication Association. This is an Open Access article distributed under the terms of the Creative Commons
Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in
any medium, provided the original work is properly cited.Beyond Anonymity K. Jaidka et al.
Keywords: Deliberation, Anonymity, Social Identity, Social Media, Web Experiment, Affordances,
Discussion Quality, Persuasiveness
doi:10.1093/jcmc/zmab019
Social media platforms allow participants to create and share content and belong to and participate in
communities. Many of the world’s political discussions occur on these platforms, which offer ways to
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
understand online political deliberation, public opinion formation, and persuasion.
However, there are still debates about the effectiveness of social media platforms as a rejuvenated
public sphere (e.g., Oz, Zheng, & Chen, 2018; Stroud, Scacco, & Curry, 2015). One critical point of
contention has been focused on the role of anonymity—afforded by many popular social media plat-
forms—in affecting political discussions’ qualities. In anonymous settings, social media users are
reported to be more uncivil than non-anonymous ones (Halpern & Gibbs, 2013; Theocharis, Barberá,
Fazekas, Popa, & Parnet, 2016), and conversations could get more polarized (e.g., Suhay, Bello-Pardo, &
Maurer, 2018; Yamamoto, 2020). While other research has suggested that anonymity could also facili-
tate public deliberation by increasing inclusivity and encouraging participation (Antheunis, Schouten &
Walther, 2020).
Some of these findings have been explained in part by the Social Identity Deindividuation model
(SIDE), which anticipates greater group conformity when discussion participants are privy to social
but not personal identity cues (Postmes, Spears, Sakhel, & de Groot, 2001). The SIDE model, however,
only examines the personal anonymity condition where social identity cues are present. It does not of-
fer a proper understanding of the effects when participants can see others’ personal identities with or
without social identities.
In addition, though many previous studies have examined the impact of anonymity on conversa-
tion health, their methods are limited in two main ways. First, most have conceptualized anonymity
as a single-faceted, binary variable depicting whether individuals can be identified online. Instead, we
contend that anonymity is a multifaceted construct facilitated through the affordances of social media
platforms. Affordances are possibilities for action (Evans, Pearce, Vitak, & Treem, 2017) that, for in-
stance, allow the users of a social media platform to identify or anonymize themselves. Some social
media platforms afford anonymity, that is, the ability to be anonymous on social media platforms.
Individuals can perceive personal anonymity—the lack of personally identifiable information to
others (Christopherson, 2007)—independently from social anonymity, which we define as the absence
of cues (such as those of sexual orientation, race, or partisanship) needed to signal social identities
(Rains, Kenski, Coe, & Harwood, 2017). On social media, one’s list of friends and communities lets
others perceive their social identity through network associations (Fox & McEwan, 2017).
Similarly, political affiliation (Party ID visible/invisible) also acts as a social identity signaled by
profile description or hashtag use.
A second limitation is that the purported effect of anonymity is overwhelmingly based on obser-
vational cross-sectional studies comparing user behavior across media channels. These cross-platform
research designs cannot disentangle the effect of anonymity on political discussions from other affor-
dances or contextual factors. When studies report a platform-specific impact, they are, in essence, es-
timating an ill-defined compound treatment effect (Hernán & VanderWeele, 2011). For instance, a
comparison of the discussion health on Facebook versus Twitter (Oz et al., 2018), Facebook versus
YouTube (Halpern & Gibbs, 2013), or Facebook versus news websites (Rowe, 2015) is a comparison
2 Journal of Computer-Mediated Communication 00 (2021) 1–23K. Jaidka et al. Beyond Anonymity
of the unique combination of many affordances taken together as well as their differences in user
demographics. In contrast, in this study, we examined the effect of different anonymity affordances
on actual and perceived discussion quality in a web experiment. This enabled us to control all other
communication design components while randomly assigning affordance conditions.
We built a custom social media platform for synchronous discussions. Our platform offers a
variable-centered approach (Nass & Mason, 1990), which allowed us to manipulate personal and so-
cial identifiability separately. We anticipated that an interplay of personal and social identifiability
would affect the quality and content of online political discussions. Therefore, we ran a well-powered
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
pre-registered experiment on the platform to test whether the juxtaposition of different anonymity
settings can foster better discussion quality and encourage opinion change.
The topic of discussion was gun control, a hot-button issue in the United States. We measured
discussion quality along the dimensions of constructiveness, justification, and incivility.
Measurements were done by using pre-trained machine learning methods (Jaidka, Zhou, &
Lelkes, 2019), which were validated on a hand-annotated sample.
Our treatments, personal identifiability (absence or presence of personal profile information),
and social identifiability (absence or presence of partisan identity), are discussed in the framework of
affordances outlined by Evans et al. (2017). Beyond simply acting as features of objects or outcomes,
these affordances have the potential to induce change in social media platforms. Changing an afford-
ance can effect a change in platform features, user perceptions, or social experiences (Bucher &
Helmond, 2018; Fox & McEwan, 2017; Zhou, 2021). In our study, we produced the affordance change
by manipulating the feature, which changes the platform’s action possibilities—the propensity of par-
ticipants being personally anonymous or politically visible (Evans et al., 2017).
Our results reveal that social identifiability had the most positive impact on conversations.
Coupled with personal anonymity, social identifiability yielded the most rational discussions. On the
other hand, coupled with social identifiability, personal anonymity delivered the least civil discussions.
Our findings offer a nuanced look at the role of anonymity in political discussions.
They have broad implications on how scholars should conceptualize online anonymity and dis-
cussion health in future studies.
Background
Political conversation is the sine qua non in various models of democracy (Dryzek et al., 2019;
Gutmann & Thompson, 1996). In these models, political decision-making depends on citizens’ ability
to conduct deliberative discussions—that is, rational and civil discussions to arrive at a consensus
(Friess & Eilders, 2015; Steenbergen, Bächtiger, Spörndli, & Steiner, 2003; Stromer-Galley, 2007).
Given the ubiquity of social media platforms and their users, online political discussions are more
likely to occur on social media platforms than anywhere else on the web. Therefore, we consider it ap-
propriate to contextualize our study of online political discussions as one of the many kinds of inter-
personal exchanges on social media platforms. Ledbetter (2021) has called upon academics to
conceptualize social media platforms in ways that engage more closely with “what people do with
social media” (p. 3). This conceptualization allows us to identify any online spaces as social media
platforms if they satisfy the uses and gratifications related to humans and their social and civic rela-
tionships. Online political discussions, therefore, are one such use of social media platforms. A cus-
tom web application that foregrounds issue-focused discussions thus constitutes a social media
platform that fulfills a particular set of uses and gratifications related to social media platforms for its
Journal of Computer-Mediated Communication 00 (2021) 1–23 3Beyond Anonymity K. Jaidka et al.
users. Such a platform emulates recent social media platforms organized around topics, such as
Reddit and Quora, among others. It also offers a way to test the viability of previous work (e.g.,
Vaccari & Valeriani, 2018) in examining informal political talk in synchronous conversations.
Despite some initial optimism, political conversation on social media platforms is far from ideal
and rarely considered “deliberative” or aimed at group consensus. Many people consider social media
platforms to have low discussion quality and heightened incivility (Parker, Horowitz, Igielnik, Oliphant, &
Brown, 2017). Incivility, however, is not the only marker of discussion quality. High-quality discussions
can also be identified from the constructiveness and justification of their arguments (Friess & Eilders,
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
2015; Gastil, 2008). Constructiveness refers to the degree of consensus-building reflected in a discussion
(Steenbergen et al., 2003). Justification refers to evidence given in support of an argument, whether based
on personal feelings or facts and data (Esteve Del Valle, Sijtsma, & Stegeman, 2018).
This study approaches online discussion quality as a consequence of the affordances of social me-
dia platforms. Affordances are loosely conceptualized as the technological features that affect how
something is used (Evans et al., 2017). Affordances can affect the quality, power dynamics, and, most
importantly, the overall health of conversations on social media platforms (Fox & McEwan, 2017;
Freelon, 2015; Jaidka et al., 2019). For example, Twitter’s affordances encourage in-group discussions
with acknowledgments, justifications, and questions (Freelon, 2015).
A body of work has explored the anonymity affordances of social media platforms. Previous stud-
ies have reported that personal identification appears to improve political discussion (Stroud et al.,
2015) while personal anonymity impairs it (Halpern & Gibbs, 2013; Oz et al., 2018).
The literature generally operationalizes anonymity rather bluntly—are people identifiable or not?
However, social media platforms can afford anonymity in different ways and to different degrees
(Scott & Rains, 2020). The absence of personal identity cues in social media profiles does not pre-empty
social anonymity, as even personally anonymous individuals may still indicate their political ideology.
In other words, users can be perceived as more personally anonymous but more socially identifiable.
We argue that the multifaceted nature of online anonymity is comparable with the identity continuum,
where, as per Turner, Oakes, Haslam, & McGarty (1994, p. 3), “personal identity refers to self-categories
that define the individual as a unique person in terms of their differences from other (in-group) persons.
Social identity refers to social categorizations of self and others. These self-categories define the individ-
ual in terms of their shared similarities with members of certain social categories in contrast to other so-
cial categories. Social identity refers to the social categorical self.” As a corollary for anonymity, we are
interested in whether social media users’ perceived personal or social anonymity affects the discussion
health on social media platforms.
Past literature has applied the deindividuation theory (Diener, 1980) and the SIDE model
(Postmes et al., 2001) to understand the effects of personal anonymity in socially identified contexts.
However, these studies do not address the independent and multiplicative effects of personal and so-
cial anonymity. In the following paragraphs, we review the evidence of their effects on discussion
health and motivate our pre-registered hypotheses.
Personal anonymity
Personal anonymity refers to the degree to which social media users are personally identifiable online.
Different social media platforms afford different ranges of personal anonymity. For example, some
platforms, such as Facebook, encourage users to supply their real-world names corresponding to their
national identification cards and link to their phone numbers, which would make them more
4 Journal of Computer-Mediated Communication 00 (2021) 1–23K. Jaidka et al. Beyond Anonymity
identifiable than if they did not. Other platforms, such as YouTube, do not require users to sign up or
provide personal information to use their website.
The effects of personal anonymity on incivility have been examined since the 1980s. Early experi-
ments suggested greater incivility in computer-mediated discussions as compared to face-to-face discus-
sions (Siegel, Dubrovsky, Kiesler, & McGuire, 1986). Many previous studies have indicated that online
personal anonymity affords a lack of accountability and a lack of self-regulation in online discussions
(Chen, 2017; Oz et al., 2018; Theocharis et al., 2016). Furthermore, personal anonymity decreases the
motivation to process information (Lelkes, Krosnick, Marx, Judd, & Park, 2012), which may make the
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
discussion less rational and substantive. Consistent with this hypothesis, political discussion on
Facebook (which prohibits personal anonymity) and Twitter (which does not) have been found to vary
in discussion quality (Oz et al., 2018). Personally anonymous YouTube comments were also more likely
to be uncivil than personally identifiable Facebook comments (Halpern & Gibbs, 2013).
In summary, there is reason to believe that anonymous social media users are less rational and
civil in online discussions than those with profile information. However, these trends do not general-
ize to all examples of personal anonymity or political discussions on social media. For example, the
study by Maia and Rezende (2016) analyzed political comments posted on different social media plat-
forms. They reported no association of incivility with personal anonymity or with other dimensions
of deliberation. Some scholars have also argued that a sense of anonymity could mitigate inequality
and heighten disclosure, leading to honest dialogues that resemble the deliberative ideal (Papacharissi,
2004; Yeshua-Katz & Hård af Segerstad, 2020).
Accordingly, we posed competing hypotheses:
H1a: Personal identifiability would lead to higher discussion quality.
H1b: Personal anonymity would lead to higher discussion quality.
Social anonymity
Following Hayne and Rice (1997), we define social anonymity as the degree to which users’ social
identities are (not) identifiable. Social media users may indicate their social identity through profile
cues such as hashtags, images, emojis, and colors that signal to audiences that they are members of a
specific group and derive a portion of their self-concept from that membership. We focus on political
partisanship as one exemplar of social identities and examine its interplay with personal identity.
Political identities—whether partisan or ideological—can be regarded as social identities (Iyengar,
Sood, & Lelkes, 2012). Therefore, expressing one’s political opinions online is a costly choice involving
the loss of social anonymity. A person may do so by disclosing political group membership in their
profile, for instance, by using hashtags like #theresistance, #MAGA, #bluelivesmatters, but remain
personally anonymous. In general, political partisanship has been found to influence the consensus
formation processes in online settings (Bolsen & Druckman, 2018; Guilbeault, Becker, & Centola,
2018; Wojcieszak & Mutz, 2009). However, the implications of social identification for discussion
quality are not straightforward.
First, we consider the possibility that social identifiability improves consensus formation and the
health of online discussions. When individuals believe that they belong to a social group, it helps
them behave more consistently with their party identities (Bakker, Lelkes, & Malka, 2020; Klar, 2013).
Hence, party cues may prime a ready-made set of arguments (Cho, Ahmed, Keum, Choi, & Lee,
2018). These findings hint that a heterogeneous discussion with visible social identity cues may in-
crease discussion quality. This argument is corroborated by some studies that suggest bearing a social
Journal of Computer-Mediated Communication 00 (2021) 1–23 5Beyond Anonymity K. Jaidka et al.
identity and participating in structured online discussions encourage intergroup contact among peo-
ple with diverse perspectives and thus improve intergroup relations (Kim, Fishkin, & Luskin, 2018).
Furthermore, the SIDE model implies that (personal) anonymity in the presence of social identity
cues can increase group identification and stereotyping, both of which trigger social conformity
(Postmes et al., 2001). A meta-analysis of studies investigating the effect of personal anonymity on
conformity with group norms provides “evidence supporting the SIDE model, such that personal ano-
nymity, coupled with a salient group identity, in online contexts results in adherence to group norms”
(Huang, 2016, p.46). However, a different scenario could emerge and lead to inferior deliberative
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
experiences. Once network information was made salient, negative judgments of out-groups could
lead to intergroup hostility and potentially move political discussions away from the ideal. For in-
stance, Maia and Rezende (2016) found less foul language in homogeneous partisan forums compared
to heterogeneous discussions. Given that the red–blue divide in the U.S. has been encouraging ani-
mosity across party lines in recent years (Iyengar et al., 2012), it is of particular interest to examine
whether the visibility of party identification could prevent the constructive discussion and dialogue
that are necessary for public deliberation.
We, therefore, raised competing hypotheses:
H2a: Social identifiability leads to higher discussion quality.
H2b: Social anonymity leads to higher discussion quality.
Finally, we examined the interplay between the different types of anonymity. Different platform
affordances could interact with each other to affect online political processes, the consequence of
which has been chiefly unattended by existing studies. For example, platforms such as Reddit or
Slashdot forums afford personal anonymity; yet, they do not report comparatively higher incivility
than other platforms such as Facebook (Poor, 2005). On the other hand, affording both personal ano-
nymity and social identifiability, Twitter is notoriously uncivil (Halpern & Gibbs, 2013; Theocharis et
al., 2016).
According to the social identity theory (Tajfel, 1982), identification with groups, as afforded by
social identity cues, naturally produces in-group favoritism and, in some contexts, out-group discrim-
ination, which leads to intergroup hostility and jeopardizes wellness on the population level (Iyengar
et al., 2012; Suhay et al., 2018). Therefore, we anticipated that social identity cues with personal ano-
nymity would decrease discussion quality.
On the other hand, some findings suggest that an apolitical context and prominent social pres-
ence heuristics may help individuals form a favorable first impression without a partisan mindset
(Wojcieszak & Mutz, 2009). Furthermore, chats with personal identity cues (and social anonymity)
could generate relatively positive participation experiences and elicit higher-quality discussions (e.g.,
Baek, Wojcieszak, & Delli Carpini, 2012) as compared to a personally anonymous condition.
Therefore, we pose the following hypotheses:
H3: Social identifiability with personal anonymity leads to lower discussion quality.
H4: Personal identifiability with social anonymity leads to higher discussion quality.
The effect of affordances on perceptions of discussion quality
Next, we examined the impact of social and personal anonymity on how participants perceived the
quality of discussions. Previous work has reported that providing social identity cues can make discus-
sants less satisfied with the quality of online discussions (Lee, 2007).
6 Journal of Computer-Mediated Communication 00 (2021) 1–23K. Jaidka et al. Beyond Anonymity
Additionally, many studies have found that higher levels of incivility induce negative cognition,
triggers aggressive emotions, and heightens perceived media bias among subgroups of study partici-
pants (Rösner, Winter, & Krämer, 2016) and reduces the perceived quality of the discussion (Popan,
Coursey, Acosta, & Kenworthy, 2019). Therefore, as per Hypothesis 3, if social identifiability with per-
sonal anonymity reduces discussion quality, then personal identifiability could improve perceptions
of discussion quality.
H5: Personal identifiability with social anonymity improves participants’ perceptions of dis-
cussion quality.
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
The effect of affordances on opinion change
Social and personal anonymity may also impact information processing and, ultimately, opinions.
Guilbeault et al. (2018) demonstrated that signaling partisanship, even in a subtle and non-intrusive
way, could trigger motivated reasoning, render the benefit of social learning in vain, and leave parti-
san bias retained if not strengthened. Social identity cues may also be why some misinformation cor-
rection strategies could fail, and cross-cutting news exposure could backfire (Bail et al., 2018). These
studies suggest that partisan priming, that is, when participants are reminded of their network affilia-
tions and provided with other people’s partisan information, could polarize participants’ issue stances
and harden their partisan preferences (e.g., Cho et al., 2018).
While social identifiability might restrain opinion change, personal identifiability has been shown
to make political comments more credible and persuasive (Ng & Detenber, 2005).
Furthermore, by “humanizing” political opinions and impeding incivility, personal identity cues
could also make discussants more open-minded towards opposing views (Hwang, Kim, & Kim,
2018). Based on this evidence, we anticipated that personal identity cues might reduce the heightened
polarization and lead to a form of consensus building (c.f., Lee, 2007). Thus, our next hypothesis was:
H6: Personal identifiability with social anonymity increases opinion change.
Previous studies have suggested that the contentiousness of an issue plays a role in determining a
discussion’s quality (Maia & Rezende, 2016; Rowe, 2015). In this experiment, the topic of discussion
was gun rights, given its salience and divisiveness among Americans as a partisan issue (Parker et al.,
2017). We conducted the experiment in 2019 and early 2020 when the United States witnessed the
highest number of mass shootings in recent memory. Throughout our data collection process, several
high-profile mass shootings happened in the country, including the El Paso Walmart shooting and
the Saugus High School shooting (Silverstein, 2020). We anticipated that our participants would be
interested in the topic and find it relevant.
Method
We designed a 2-by-2 between-subject factorial experiment in the form of an online chat room where
participants discuss and exchange opinions on a contentious political issue, in this case, gun rights in
the United States. We pre-registered the experiment on the Open Science Framework at https://osf.io/
dyu3b/. The Institution Review Board of the University of Pennsylvania approved the experimental
protocol.
Our first task was to develop a custom social media platform to implement our treatment condi-
tions. The platform emulates community-based, group-chatting social media platforms, such as
Journal of Computer-Mediated Communication 00 (2021) 1–23 7Beyond Anonymity K. Jaidka et al.
Discord, that are rising in popularity and competing with other personal group-chatting platforms such
as WhatsApp. Discord is a social media platform where users create communities and communicate
around their shared interests. Users can choose whether or not to identify different facets of themselves.
Posts on users’ social feeds are oriented around threaded conversations about various topics. Our plat-
form captures these qualities in a discussion-centered platform for study participants, with both per-
sonal anonymity and social anonymity affordances manipulated by our back-end administrative page.
Data were secured in a firewalled and password-protected public server, separate from users’ sur-
vey responses, and accessible only by the first author and the research assistant.
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
Respondents were randomized to use chat rooms with different social and personal affordances.
Once they cleared the screening, they were asked to personalize their user profiles before partici-
pating in the experiment. While signing up, participants clicked a picture and provided their first
name and the first few letters of their last name. In addition, they identified their partisan leaning and
chose a few hashtags related to partisanship.
All participants entered personal and political information before entering the room. However,
their visibility to other participants was manipulated by our randomly assigned treatment condition.
Having the same sign-up procedure ensured that all participants were primed with the same personal and
social cues. We operationalized personal identifiability by letting participants see others’ photographs and
names. We operationalized social identifiability by letting participants see others’ self-selected hashtags re-
lated to their partisanship. Respondents were briefed that we would delete any identifying information im-
mediately after the experiment. Screenshots of the chat rooms with different conditions are provided in
Supplementary Materials.
Sample and procedure
Recruitment
Figure 1 summ arizes our experimental design together with the participant flow. We recruited a total
of 12,002 participants from Amazon Mechanical Turk who reside in the United States and have approval
rates higher than 80%. Experiments on Mechanical Turk have consistently yielded similar estimates as
experiments run on other platforms (e.g., Coppock, Leeper, & Mullinix, 2018). Our participants completed
a short screening survey on their basic demographics and party identifications. We also collected partici-
pants’ opinions on gun rights, adapting the prompts from previous work (Min, 2007).
Out of 12,002 participants, we re-contacted 10,705 by sampling from the pool of qualified partici-
pants who had not previously attended a chat. Stratified sampling was followed to maintain a balance
in the distribution of gun rights opinions across all chat rooms. We then used the Mechanical Turk
API to re-contact respondents to participate in a 25-minute discussion and fill up a post-discussion
survey. Participants were given a link to the chat room and a time to appear.
Participants were compensated $5 for their participation. Out of 10,705 re-contacted participants,
940 participated in the chat rooms, yielding a response rate in line with other deliberation experi-
ments (Esau, Friess, & Eilders, 2017).
Each discussion was initiated by a pro-gun rights “bot” account and was also attended by one
against-gun rights. These bots ensured a diversity of perspectives and were intended to keep the dis-
cussion lively. Bots were also necessary to ensure that participants had a respondent if they were the
only participant that showed up. Bots’ comments were scripted using excerpts of real arguments from
internet forums on issues of gun control.1 Bots were passive unless activated by a 15-second lag in the
conversation.
8 Journal of Computer-Mediated Communication 00 (2021) 1–23K. Jaidka et al. Beyond Anonymity
Screening and pre-survey N = 12,002
Participant invited N = 10,705
Random assignment to chat rooms
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
Profile Visible Profile Visible Profile Anonymous Profile Anonymous
Party ID Visible Party ID Anonymous Party ID Visible Party ID Anonymous
Each room invites 32 participants
with same opinion distribution
Name: Location:
Age: Party ID: Participant joined N = 940
Gender: Profile Pic: Personalize chat profiles
Online chat 25 minutes
Post-survey and compensation
Figure 1 Experimental design.
Manipulation check
In the post-discussion survey, we conducted a manipulation check. On a 5-point Likert scale, partici-
pants rated “how personally identifiable were the people you interacted with?” (1 ¼ not at all identifi-
able to 5 ¼ extremely identifiable), and “how difficult was it to determine the political ideologies of
others in the chat room?” (1 ¼ not at all difficult to 5 ¼ very difficult).
Dependent measures
Opinion on gun control
A battery of nine survey questions was adapted from Pew Research Center (Parker et al., 2017) to as-
sess participants’ opinions on gun control policies. We asked them how much they support policy
suggestions such as “barring gun purchases by people on the federal no-fly or watch lists” and
“allowing teachers and school officials to carry guns in K-12 schools” on a 5-point Likert scale ranging
from strongly favor to strongly oppose. The same questions were asked both pre- and post-discussion.
Factor analyses revealed two factors, of which we created a summary variable depicting support for gun
rights out of the first factor comprising six items (a ¼ .86). The same questions were asked both
Journal of Computer-Mediated Communication 00 (2021) 1–23 9Beyond Anonymity K. Jaidka et al.
pre-discussion (pre-discussion M ¼ 3.64; SD ¼ 0.79; a ¼ .86) and post-discussion (post-discussion
M ¼ 3.69; SD ¼ 0.79; a ¼ .87).
Perception of discussion quality
We included seven items from the Deliberative Quality Index in the post-experiment survey to mea-
sure the degree to which participants perceived their chat experiences to be deliberative (Nabatchi,
2007). These items asked subjects to evaluate statements such as “the discussions identified shared
values in the community” and “the discussions helped me consider other sides of the issues” on a
7-point Likert scale ranging from “strongly disagree” to “strongly agree” (M ¼ 4.71; SD ¼ 1.18; a ¼ .84).
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
Discussion quality of messages
We followed the operationalization of Discourse Quality offered by Steenbergen et al. (2003), Jaidka
et al. (2019), and Stroud et al. (2015) under the following definitions:
• A constructiveness score deems whether the author of a message appears to contribute to the dis-
cussion by resolving conflicts, fact-checking, or building consensus.
• A justification score denotes whether they offer data, links, facts in a message to support their
argument.
• An incivility score denotes whether a message is abusive constitutes hate speech or indulges in
name-calling.
We used machine learning classifiers based on these definitions that were trained and validated
(average accuracy ¼ 79%) by Jaidka et al. (2019) using Python’s scikitlearn package in previous work.
These classifiers are applied to measure our dependent variables, that is, the discussion quality of mes-
sages participants sent in the chat rooms. Each measure consisted of a prediction on a scale of 0 to 1,
denoting the classifier’s confidence in a positive prediction. Given an input, machine learning algo-
rithms rely on a confidence score to predict the binary class in binary classification problems. They
typically use a confidence value of 0.5 as the default threshold to generate an output of 1.
After obtaining the constructiveness (M ¼ 0.74; SD ¼ 0.15), justification (M ¼ 0.75; SD ¼ 0.16),
and incivility (M ¼ 0.17; SD ¼ 0.09) scores from the classifier, we followed Gastil (2008)’s discussion
of different deliberative components to average constructiveness and justification scores (Pearson’s
r ¼ 0.8, p < 0.001) to create a net rationality score (M ¼ 0.75; SD ¼ 0.15; a ¼ 0.91). Incivility had a
weak negative correlation with the rationality (r ¼ 0.07, p < 0.001).
Annotation task for measurement validation
We validated the discussion quality classifiers against a human-annotated random sample of our
dataset, which comprises 400 chat messages sampled equally from each of our four treatment condi-
tions. Our validation secured an average accuracy of 79.5%.
We crowdsourced the annotations from an online task hosted on Amazon Mechanical Turk.
Workers residing in the United States with at least an 80% approval rate on at least 5000 previous
tasks read the training instructions, viewed over a dozen examples, and annotated chat messages posted
during the experiment, in return for a small compensation per annotation ($0.08 per message). We
adapted the instructions for training and guiding the annotators from the original classification task.2
The classifiers had an average precision rate (number of positive cases correctly predicted positive
out of the total that were predicted positive) of 80% and an average accuracy (all cases correctly
10 Journal of Computer-Mediated Communication 00 (2021) 1–23K. Jaidka et al. Beyond Anonymity
predicted out of the total predictions) of 79.5%, which are both considered to be indicators of good
performance. Further information about the descriptive statistics, annotation procedure, inter-coder
agreement on the human annotation, and the model accuracy are provided in Supplementary
Materials.
Analytical strategies
A total of 11,268 messages were posted in 346 chat rooms by 703 participants, averaging 32 messages
per chat room or an average of 16 messages (median of 14 messages) per participant. Each chat was
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
25 minutes long. An average (and median) of four participants (including two bots) typed messages
in each chat. However, 237 participants did not type any messages and presumably “lurked.”
Analyses of the effect on discussion quality are therefore limited to the 703 participants of the
discussions.3
The average message length was 90 characters (median was 83 characters), and the maximum
message length was 206 characters. Message-level scores were aggregated to the individual level for
subsequent analysis.4
We used regressed our dependent variables (the discussion quality of messages) on two afford-
ance treatments and their interaction term, controlling for user demographics, that is, age (M ¼ 38.34,
SD ¼ 12.09), binarized sex (Female ¼ 54.67%),5 and binarized partisan identity (Democrat ¼
61.45%).6 Bivariate analyses were also conducted for reference. The lme4 package in R was used to fit
linear models with random intercepts, which accounted for the within-chat room non-independence
of observations. Standardized effect sizes were calculated using the effectsize package, based on the
F values obtained from the linear models.
Results
Manipulation check
The experiment had the expected effect on perceptions of personal anonymity and social anonymity.
Subjects who were showed participants’ names and selfies during the chat reported that it was easier
to personally identify others whom they interacted with (Mean for personal identifiability, that is,
Profile Visible condition in Figure 1, M ¼ 2.707; Mean for personal anonymity condition, M ¼ 2.349,
t (933.46) ¼ 4.513, p < .0001). Additionally, subjects who used political hashtags in their chats felt it
was “less difficult to determine the political ideologies of others in the chat room” (Mean for social
identifiability, that is, Party ID visible condition in Figure 1.
M ¼ 1.814; Mean for social anonymity, that is, Party ID anonymous condition in Figure 1.
M ¼ 1.973, t (937.95) ¼ 2.222, p ¼ .027).
Affordances and discussion quality
Consistent with H1b (and not H1a) the relationship between anonymity and lack of rationality was
not significant (b ¼ .003, SE ¼ .005, p ¼ .592) as shown in Column 1 of Table 1. That is, personal
anonymity (Profile Anonymous condition in Figure 1) did not affect rationality in discussion.
Conversely, the findings show support for H1b as personal anonymity led to a significant decrease in
incivility (Model 1 in Column 4, b ¼ .005, SE ¼ .002, p ¼ .035) with a standardized effect size of
0.03 (90% CI: [0.00, 0.08]). When participants felt more anonymous in the discussion, their messages
were 0.5% less likely to be uncivil.
Journal of Computer-Mediated Communication 00 (2021) 1–23 1112
Beyond Anonymity
Table 1 The Impact of Personal Anonymity and Social Identifiability on Rationality, Incivility, and Perceived Discussion Quality
Treatment condition 1 2 3 4 5 6 7 8 9
Rationality Incivility Perceived Discussion Quality
Model 1 Model 2 Model 3 Model 1 Model 2 Model 3 Model 1 Model 2 Model 3
Personal Anonymity .003 (.005) – .012þ(.081) .005* (.002) – .001 (.003) .049 (.099) – .115 (.140)
*
Social Identifiability – .010 (.005) .001 (.007) – .002 (.002 ) .005 (.003) – .153 (.099) .216 (.140)
Personal Anonymity – – .021* (.010) – – .006þ(.005) – – .123 (.200)
Social Identifiability
(N ¼ 703) (N ¼ 940)
*p < .05.
þ
p < .10.
Note. Numbers in bold reflect statistically significant effects (p < 0.05) and marginally significant effects (p < 0.1).
K. Jaidka et al.
Journal of Computer-Mediated Communication 00 (2021) 1–23
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022K. Jaidka et al. Beyond Anonymity
H2a was supported (and not H2b) since social identifiability was related to a significant increase
in message rationality (Model 2 in Column 2, b ¼ .010, SE ¼ .005, p ¼ .041) with a standardized ef-
fect size of 0.01 (90% CI: [0.00, 0.04]). When participants could view each other’s political identities,
the messages they posted were 1.1% more rational than when they could not. There was no significant
effect on the incivility of their messages.
In examining the interaction effect of personal anonymity and social identifiability, our results
contradict H3 and H4. First, we found that personal identifiability moderated the association between
social identifiability and rationality. That is, in Figure 2, we see that when people were personally
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
identifiable, Party ID visibility did not matter. However, when participants are personally anonymous,
messages in the Party ID visible condition were significantly more rational than participants that are
not (Model 3 in Column 3, b ¼ .021, SE ¼ .010, p ¼ .039) with a standardized effect size of 0.02 (90%
CI: [0.00, 0.07]). Second, personal anonymity’s negative effect on incivility was mainly driven by a
subset of chat room participants with social identifiability. That is, as seen in Figure 3, when the Party
ID is visible, messages in the Profile anonymous condition (M ¼ 0.172, SD ¼ 0.027) are significantly
less uncivil than messages in the personally identifiable condition (Profile visible condition in
Figure 1) (M ¼ 0.181, SD ¼ 0.034) (Welch t-test, t (268.75) ¼ 2.139, clustered two-tailed p ¼ .048).
H5 was not supported; that is, the relationship between personal anonymity and perceptions of
deliberation quality was not statistically significant in the main analysis (Model 1 in Column 7, b ¼
.093, SE ¼ .100, p ¼ .620). H6 was partially supported. There was no significant effect of either per-
sonal anonymity or social identifiability taken one at a time. However, their interactive effect had a
statistically significant effect on opinion change in a particular direction, that is, decreased support for
gun rights (Model 3 in Column 3 of Table 2, b ¼ .148, SE ¼ .054, p ¼ .005) with a standardized ef-
fect size of 0.03 (90% CI: [0.00, 0.08]). However, this effect does not generalize to absolute opinion
change (Model 3 in Columns 4–6, b ¼ .048, SE ¼ .040, p ¼ .232).
Discussion
This study examined whether anonymity or social identifiability impair the quality of political discus-
sions in online discussions. Experiments yielded three main findings, which are somewhat inconsis-
tent with the current understanding of anonymity and discussion health.
First, the findings suggest that coupled with personal anonymity, social identifiability induced
greater rationality in our experiment, which supports the expectations of the SIDE model. The SIDE
model predicts that heightened social categorization and identity salience yield greater group confor-
mity (Lea, Spears, & de Groot, 2001). We can speculate that when social media users can identify with
party identities, their participation in online discussions will achieve ideological consistency and
group conformity. In such settings, the increased rationality and civility could be indicative of and
caused by greater social conformity. These findings counter the evidence of inter-group tensions
when partisan identities are salient (Maia & Rezende, 2016; Rains et al., 2017). Our post hoc analysis
suggested that the effects persisted in heterogeneous and democrat-majority rooms, and these results
are provided in the supplementary analysis. Additionally, the findings extend the present understand-
ing of the SIDE model by reporting the interaction effect in the opposite scenario, with personal iden-
tity cues visible and social identity cues invisible. In the absence of social identifiability, personal
identifiability does not lead to greater actual or perceived discussion quality, nor absolute opinion
change.
Journal of Computer-Mediated Communication 00 (2021) 1–23 13Beyond Anonymity K. Jaidka et al.
1
n.s.
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
0.775
Message Rationality
0.750 **
0.725
0
Profile Anonymous Profile Anonymous Profile Visible Profile Visible
Party ID Anonymous Party ID Visible Party ID Anonymous Party ID Visible
Conditions
Figure 2 The impact of personal anonymity and social identifiability on message rationality.
Notes: In conditions with personal anonymity (Profile anonymous), social identifiability (Party ID visible) elicited more ra-
tionality from participants’ discussions on gun rights (Welch t-test, t (268.58) ¼ 2.854, clustered two-tailed p ¼ 0.006).
Second, our findings do not support the expectations for lesser civility in anonymous social media
platforms. Instead, the results suggest that when coupled with social identifiability, personal anonym-
ity can inhibit incivility. This may be because personal anonymity may increase the importance of
other cues (such as linguistic cues or social identity cues; Antheunis et al., 2020; Toma & Hancock,
2010). The language that anonymous people use in a chat discussion, therefore, would be critical in
helping them establish a favorable first impression (Antheunis et al., 2020; Walther, 1992). An alter-
native explanation is that incivility is lower in anonymous discussions due to reduced engagement. In
other words, personal anonymity could be associated with reduced motivation (Lelkes et al., 2012) be-
cause of reduced personal stakes in a deindividuated setting.
14 Journal of Computer-Mediated Communication 00 (2021) 1–23K. Jaidka et al. Beyond Anonymity
1
0.19
n.s.
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
0.18
Message Incivility
**
0.17
0.16
0
Profile Anonymous Profile Anonymous Profile Visible Profile Visible
Party ID Anonymous Party ID Visible Party ID Anonymous Party ID Visible
Conditions
Figure 3 The impact of personal anonymity and social identifiability on message incivility.
Notes: In conditions with social identifiability (Party ID visible), personal anonymity (Profile anonymous) significantly
decreases message incivility (Welch t-test, t(268.75) ¼ 2.139, clustered two-tailed p ¼ .048).
Third, our findings reiterate perspectives that consider incivility an inadequate measure of
discussion quality in measuring deliberation quality. The correlation between rationality and inci-
vility was weakly negative. Empirical results from a recent study have also suggested that discus-
sion health needs to look “beyond incivility” (Rossini, 2020), possibly because incivility and
intolerance occur in different contexts. Intolerance is likely to occur in homogeneous discussions
that engage in othering behavior. On the other hand, incivility is often associated with justified
opinion expression and genuine engagement with policy disagreement. In future work, scholars
should consider examining the effect of affordances on the level of tolerance evinced in online
discussions.
Journal of Computer-Mediated Communication 00 (2021) 1–23 1516
Beyond Anonymity
Table 2 The Impact of Personal Anonymity and Social Identifiability on the Raw and Absolute Change in Support of Gun Rights
1 2 3 4 5 6
Change in Support Absolute Change in Support
Model 1 Model 2 Model 3 Model 1 Model 2 Model 3
Personal Anonymity .018 (.027) – .056(.038) .009 (.020) – .034 (.028)
Social Identifiability – .034 .108** (.037) – .025 (.020) .049þ (.028)
Personal Anonymity – (.027) .148**(.054) – – .048 (.040)
Social Identifiability -
(N ¼ 940)
*p < .05.
þ
p < .10.
K. Jaidka et al.
Journal of Computer-Mediated Communication 00 (2021) 1–23
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022K. Jaidka et al. Beyond Anonymity
While our results contradict some previous empirical findings, it is essential to emphasize that a
theory does not stand or fall based on a single experiment. Our findings raise the possibility that
results from past literature may not generalize to all domains. Or, at a minimum, the affordance of
personal anonymity does not always decrease discussion quality.
Our experimental design subjected us to some unavoidable limitations. For example, we encoun-
tered some attrition. Second, our analysis is based on discussions around a single topic featuring citi-
zens from a country, limiting its external validity to various political contexts. Third, personal and
social anonymity might not be as orthogonal as we operationalized. One’s name or profile picture
may allow other participants to guess their social identities, especially if they appear from an ethnic
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
minority group in the United States. It is not clear whether we can satisfactorily resolve this
limitation.
A methodological contribution of this work has been the use of machine learning methods
and a web experiment. Machine learning methods were used to train linguistic classifiers, vali-
dated on hand-labeled chat data before applying them to measure discussion health. The web ex-
periment was run on a custom-built social media platform, allowing researchers to manipulate
different affordances of a platform and examine their impact on discussion health. In the present
study, we ran a well-powered pre-registered experiment to test whether the juxtaposition of two
affordance—personal and social anonymity—affects the discussion health, the perceived quality
of discussions, and subsequent opinion change. The findings provide insights to fuel the ongoing
debates in academia about online anonymity and discussion health. While the individual effects
are admittedly small, we note that these calculations are at the user level. Thus, we anticipate that
these effects would amplify at the population level where millions of users interact daily, as they
do on all social media platforms.
The design principles underlying our experiment design need to be considered in any discussion
of its generalizability to other social media contexts or even casual online conversations. First, the ex-
periment involved a one-time, prompt-based discussion among a few discussion partners. This expe-
rience is similar to platforms with personal anonymity affordances, where interacting with strangers
may be the norm, such as on community-based platforms (such as Twitter, Reddit, Discord, and so
on). In contrast, some platforms discourage personal anonymity, where conversations are usually
struck between people who are acquainted with each other (Facebook, LinkedIn, and so on). In addi-
tion, social identification is possible through profile bios and network affiliations on some platforms
(e.g., Twitter and Facebook). Therefore, we expect the effects to be more similar if the experiment
were replicated on Discord than on Facebook or LinkedIn.
A second consideration is that the chat discussions occur in a synchronous setting. A previous
study has suggested that synchronous discussions can be more persuasive than asynchronous ones
(Ng & Detenber, 2005), and synchronous discussions may be more appropriate to discuss politics
than asynchronous ones. This would suggest that effect sizes could be weaker in asynchronous
community-based platforms such as Twitter or Reddit than Discord.
A third consideration is about how findings generalize to understanding online political de-
liberation and opinion formation. Scholars have argued that political deliberation develops
through prolonged, deep discussions and consensus formation. We specifically wanted partici-
pants to form an impression of each other based on their context in that very chat room and ab-
sent any preconceived knowledge about each other. Controlling for these factors would be
difficult in a discussion that continues over time. Previous work has reported on the positive
effects of prolonged exposure and repeated conversations on opinion change, as compared to
one-time participation (e.g., Cho et al., 2018). Accordingly, we would expect that the affordance
Journal of Computer-Mediated Communication 00 (2021) 1–23 17Beyond Anonymity K. Jaidka et al.
effects on opinion change would likely become more robust. Spending more and more time with
the group would only strengthen participants’ positive evaluation of it and the need to align their
opinions with the majority.
These considerations notwithstanding, affordances that offer anonymity are a fundamental char-
acteristic across all social media platforms. Thus, the effect sizes may change in the absence of a
moderating situation between participation in an experiment, self-selecting into a goal-directed dis-
cussion, or commenting in an organically emerging debate of a contentious topic.
We recommend that future research could examine the uncanny valley effect when political dis-
Downloaded from https://academic.oup.com/jcmc/article/27/1/zmab019/6427305 by guest on 16 June 2022
cussions on social media platforms involve bots (Ciechanowski, Przegalinska, Magnuski, & Gloor,
2019). A substantial proportion of Twitter users are reportedly bots and anticipated to “exercise a pro-
found impact” on social media platforms since they produce significantly more tweets at a per-
account level than humans (Davis, Varol, Ferrara, Flammini, & Menczer, 2016). In this study, a bot
posted a message to sustain the conversation when discussions stalled for over 15 seconds. However,
our analyses were not statistically powered to detect the effect of interacting only with bots instead of
real people.7
It would also be insightful to examine whether the majority ideology of the room impacted
discussion health and subsequent opinion change. A recent meta-analysis of studies of anonym-
ity in online contexts has suggested a positive relationship between anonymity and conformity
to group norms (Huang, 2016). Many of our chat rooms had a majority of Democrat partici-
pants. They have been reported in the supplementary materials but would warrant further explo-
ration. Finally, future work could re-examine the role of the contexts and cultures in such
experimental setups. Along the same lines, effects may differ across countries, both in how
quickly social norms are violated online and how contentious and divisive online political dis-
cussions are in general.
Conclusion
Much of digital data can be traced, mined, and used to profile individuals, and it is challenging to
be fully anonymous online (Scott & Rains, 2020). However, social media platforms are unique in
allowing users to anonymize certain facets of themselves while making others identifiable. A variable-
centered approach bridges an understanding of social media platforms to the personal and social ano-
nymity they afford through the underlying technological features.
Our study has implications for how social media platforms can build ideal architectures with
an interplay of affordances that encourage better behavior and positive opinion change. For in-
stance, Twitter has begun considering how its design should focus on the overall metrics for
healthy conversations (Metz, 2018), such as allowing shared attention, shared reality, receptivity,
and diversity. The research community has additionally suggested that searchable and network-
visible (Bolsen & Druckman, 2018), or moderated (Matias, 2019) platforms could be ideal for po-
litical discussions. Of course, it would be naive to infer that a platform with all of these communi-
cation possibilities would be ideal for deliberation. Still, it suggests fruitful avenues for
exploration and research.
Acknowledgements
We are grateful to Rich Ling, Kelley Garrett, and the anonymous reviewers for their insightful com-
ments and feedback. We would also like to thank Shubham Jain and Harshit Aneja for their work on
18 Journal of Computer-Mediated Communication 00 (2021) 1–23You can also read
