Insiders and Outsiders in Research on Machine Learning and Society
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Insiders and Outsiders in Research on Machine Learning and Society
Yu Tao1 and Kush R. Varshney2
1
Stevens Institute of Technology
2
IBM Research – T. J. Watson Research Center
ytao@stevens.edu, krvarshn@us.ibm.com
arXiv:2102.02279v1 [cs.CY] 3 Feb 2021
Abstract In this paper, we focus on who is conducting this research
at the intersection of machine learning and society through
A subset of machine learning research intersects with societal the lens of the sociology of science. The theoretical foun-
issues, including fairness, accountability and transparency, as dation for our investigation is the concept of insiders and
well as the use of machine learning for social good. In this
outsiders in the research enterprise (Merton 1972). In the
work, we analyze the scholars contributing to this research at
the intersection of machine learning and society through the social sciences and humanities, researchers are considered
lens of the sociology of science. By analyzing the authorship insiders if they are members of the community being studied
of all machine learning papers posted to arXiv, we show that (and thus have lived experience of that community) and out-
compared to researchers from overrepresented backgrounds siders otherwise. (Formal and natural sciences typically do
(defined by gender and race/ethnicity), researchers from un- not study communities of people, but the societal aspects of
derrepresented backgrounds are more likely to conduct re- research on machine learning and society does.) A different
search at this intersection than other kinds of machine learn- perspective says that members of groups that have been his-
ing research. This state of affairs leads to contention between torically underrepresented in a field of study are outsiders.
two perspectives on insiders and outsiders in the scientific en- These two notions, illustrated in Figure 1 and Figure 2 may
terprise: outsiders being those outside the group being stud-
be at odds. Researchers being insiders from one perspec-
ied, and outsiders being those who have not participated as
researchers in an area historically. This contention manifests tive and at the same time outsiders from the other perspec-
as an epistemic question on the validity of knowledge derived tive raises contention in the production of knowledge, in-
from lived experience in machine learning research, and pre- cluding in the epistemic validity of knowledge arising from
dicts boundary work that we see in a real-world example. lived experience. The social construction of whether scien-
tific knowledge arising from lived experience is valid or in-
valid is an instance of boundary work (Gieryn 1983).
1 Introduction To analyze researchers in machine learning and society
Research on the theory and methods of machine learning has from the theory of insiders and outsiders, first we empiri-
led to the ability of technological systems to grow by leaps cally show that machine learning researchers from underrep-
and bounds in the last decade. With this increasing com- resented backgrounds, compared to researchers from over-
petence, machine learning is increasingly being employed represented backgrounds, are more likely to study the so-
in real-world sociotechnical contexts of high consequence. cietal aspects of machine learning than they are to study
People and machines are now truly starting to become part- aspects of machine learning that are more divorced from
ners in various aspects of life, livelihood, and liberty. society. Recognizing the inadequacy of binary gender cat-
This intersection of machine learning with society has fu- egories, we nevertheless take binary gender as one sensi-
eled a small segment of research effort devoted to it. Two tive attribute. (Women are underrepresented and men are
such efforts include research on (1) fairness, accountability overrepresented.) Recognizing the inadequacy of the social
and transparency of machine learning (FAccT), and (2) ar- constructs of coarse race and ethnicity categories, we also
tificial intelligence (AI) for social good. The first of these take race/ethnicity as another sensitive attribute. (Blacks and
focuses on the imperative ‘do no harm’ or nonmaleficence, Hispanics are underrepresented, and whites and Asians1 are
with a special focus on preventing harms to marginalized overrepresented.) We also examine the intersection of gen-
people and groups caused or exacerbated by the use of ma- der with race/ethnicity.
chine learning in representation and decision making. The Next, we extrapolate beyond what the empirical analysis
second focuses on using machine learning technologies as is able to tell us by critically examining the factors that may
an instrument of beneficence to uplift vulnerable people and have led to the current state. We also predict the character of
groups out of poverty, hunger, ill health, and other societal
1
inequities. Asians may be disadvantaged in certain considerations like ca-
reer mobility in a United States context, but are considered overrep-
Copyright © 2021, by the authors. All rights reserved. resented here in the worldwide machine learning research context.Figure 1: Researchers with lived experience relevant for the
topic of inquiry have traditionally been seen as insiders. We
hypothesize that the topic of machine learning and society is
being conducted at a greater rate by those with lived experi-
ence of marginalization. Figure 3: A hierarchical representation of the different topics
that constitute research on machine learning and society.
8 summarizes and concludes.
2 Research on Machine Learning and Society
As discussed in the introduction, two movements with a so-
cietal focus have arisen alongside the growth of research and
development of machine learning technologies: FAccT and
AI for social good. We briefly summarize these movements
in this section, and also in Figure 3.
Ethical AI, responsible AI, trustworthy machine learn-
ing, and FAccT all refer to the cross-disciplinary theory and
methods for understanding and mitigating the challenges as-
sociated with unwanted discrimination, lack of comprehen-
Figure 2: Researchers from overrepresented groups have sibility, and lack of governance of machine learning systems
traditionally been insiders. Machine learning and society used in applications of consequence to people’s lives such as
is part of a field with underrepresentation of women and employment, finance, and criminal justice (Varshney 2019).
racial/ethnic groups. Broadly speaking, there have been three different kinds of
research in this area (Kind 2020), including (1) philosophi-
cal contributions on ethical principles for AI (Jobin, Ienca,
and Vayena 2019; Whittlestone et al. 2019); (2) technical
the boundary work that may arise in machine learning and contributions on bias metrics and mitigation (Menon and
society. Finally, through a short case study, we confirm that Williamson 2018; Kearns et al. 2019), explainability and in-
there is at least one example in which the theorized epistemic terpretability algorithms (Du, Liu, and Hu 2019; Bhatt et al.
contention has arisen in real life. 2020), and factsheets as transparent reporting mechanisms
The remainder of the paper is organized as follows. Af- (Arnold et al. 2019; Mitchell et al. 2019); and (3) contribu-
ter providing a brief recapitulation of research on machine tions bringing forth a social justice angle by adapting theo-
learning and society in Section 2, we dive into the theory of ries of feminism, decoloniality, and related traditions (Buo-
insiders and outsiders in knowledge production in Section lamwini and Gebru 2018; Mohamed, Png, and Isaac 2020).
3. In Section 4, we discuss the participation of underrep- Research in the first category, ethical principles for AI, tends
resented groups in science and technology with a focus on not to overlap with research on machine learning methods
computer science. Section 5 presents the empirical work; it and algorithms. The second and third categories, technical
is conducted on submissions to arXiv, a preprint server that contributions and social justice perspectives, most certainly
hosts a large fraction of machine learning research papers. do intersect with other machine learning research.
Section 6 analyzes the sociology of knowledge production Algorithmic fairness research has two main branches. The
in the area of machine learning and society using the theory first is concerned with allocation decisions like loan ap-
of insiders and outsiders, and boundary work. We concretize proval, pretrial detention judgement, and hiring (Barocas
this analysis in Section 7 through a brief case study. Section and Selbst 2016). The program of research is to define math-ematical notions of fairness, audit existing systems with re- cation,’ ‘gender equality,’ and twelve others. The form of
spect to those notions, and develop bias mitigation algo- this pairing may be data science competitions, weekend
rithms that optimize for those notions while maintaining fi- volunteer events, longer term volunteer-based consulting
delity to the learning task. The second branch of algorith- projects, fellowship programs, corporate philanthropy, spe-
mic fairness is concerned with representational issues, for cialized non-governmental organizations, innovations units
example in information retrieval, natural language under- within large development organizations, or data scientists
standing, and dialogue systems (Blodgett et al. 2020). Here employed directly by social change organizations. Some
the program of research mainly revolves around defining projects require research and some are more application ori-
the problem itself, since there are many forms of unwanted ented. The ones that require research and whose results are
representational bias ranging from stereotypes encoded into published fall squarely within the intersection of machine
pronouns and occupations, to slurs, offensive language and learning and society.
hate speech, to poorer understanding of dialects and accents
of marginalized groups. In both branches, reasons for ma- 3 Researchers’ Roles in Knowledge
chine learning models to exhibit systematic disadvantage to-
wards marginalized groups include prejudice of human an-
Production: Insider/Outsider Status and
notators who label training data, undersampling of marginal- Boundary Work
ized group members in training data, and subjective biases The insider/outsider discussion in social sciences and hu-
by data scientists in problem specification and data prepara- manities addresses the role of the researcher as an insider
tion. Research in both branches can span the spectrum from (i.e., a member of the community being studied) as opposed
completely formal applied mathematics to wholly social sci- to an outsider in affecting research, approach, relationship
ence with calls for justice, i.e., from the second to the third with participants, and/or findings. The insider doctrine of
kind of FAccT research. Regardless of where on the spec- Merton (1972) highlights the insider’s exclusive access (the
trum it falls, algorithmic fairness research tends to always be strong version) or privileged access (the weaker version) to
considered part of the machine learning and society nexus. knowledge and the outsider’s exclusion from it. Researchers
Explainable and interpretable machine learning, in which are considered as insiders or outsiders based on their as-
the goal is for a person to understand how a machine learn- cribed status (e.g., gender, race, nationality, cultural or re-
ing model makes its decisions, has several methodologies ligious background) or group membership. The strong ver-
appropriate for different contexts and different personas con- sion asserts that the insider and the outsider cannot arrive at
suming the explanations (Hind 2019). One use for explain- the same findings even when they examine the same prob-
ability is to reveal unwanted biases in machine learning lems; the weaker version argues that insider and outsider re-
models, but doing so is not reliable (Dimanov et al. 2020). searchers would focus on different research questions. The
To date, the majority of the research has leaned towards the combined version argues that the researcher needs to be an
formal and mathematical. Calls to ground explainability in insider in order to understand the community and also to
social psychology and cognitive science (Miller 2019) have know what is worth understanding or examining about the
started to bring a greater social science character to the topic. community (Merton 1972).
Nevertheless, many interpretability researchers do not con- However, structurally speaking, it is hard to completely
sider their methodological work to have a societal aspect, distinguish the insider from the outsider because we all oc-
and their papers are not abundant at FAccT-specific venues. cupy a combination of different statuses, including sex, age,
On the other hand, the framing of efforts to increase class, race, occupation, and so on. The insider knowledge
transparency of machine learning lifecycles does incorpo- that is accessible to only individuals who occupy a highly
rate a societal angle. For example in factsheets—a tool and complex set of statuses is limited to a very small group, and
methodology for transparently reporting information about a this way of knowledge production and sharing is not sustain-
machine learning model as it is specified, created, deployed, able. Similarly, social scientists like Karl Marx recognize the
and monitored—the reported information can include the in- value of political, legal, and philosophical theories in eco-
tended use of the model as well as quantitative test results nomics. Another limitation of the insider doctrine is that it
on accuracy, fairness, and other performance indicators. It is takes a static perspective and does not recognize that our sta-
useful to individuals impacted by the machine learning sys- tuses and life experience evolve over time, which shifts our
tem (especially those from marginalized groups) and to reg- status as an insider or an outsider. In the meantime, the out-
ulators charged with ensuring the system behaves according sider, while not being able to completely transcend existing
to laws and societal values. beliefs and social problems, has the advantages of using less
Whereas FAccT is concerned with preventing societal bias in examining social issues and bringing new perspec-
harms, AI for social good takes the opposite track and uses tives to solving issues taken for granted by insiders. The in-
the technology to benefit society, especially those at the teraction of the insiders and outsiders makes intellectual ex-
margins (Chui et al. 2018; Varshney and Mojsilović 2019). change possible, and Merton argues that we could integrate
The working paradigm is to pair data scientists with social both sides in the process of seeking truth.
change organizations to work towards the 17 Sustainable Extending Merton’s and other scholars’ thoughts on the
Development Goals (SDGs) ratified by the member states insider/outsider debate, Griffith (1998) also believes that
of the United Nations in 2015, which include: ‘no poverty,’ the researcher occupies a particular social location, and her
‘zero hunger,’ ‘good health and well-being,’ ‘quality edu- knowledge is situated in particular sets of social relations.However, the insider status is just the beginning but not the 4 Participation in Computer Science: The
end of the research process. Reflecting on her own research Outsider Status
experience in mothering work for schooling, she and her col-
laborator who were both single mothers started as insiders Participation of Women
(mothers). However, they had to cross the social and con- In science, women have been at the “Outer Circle” for a
ceptual boundaries to include only mothers from two-parent long time. Historically, women faced multiple barriers in en-
families (and thus become outsiders in the research process) tering a scientific career, and even those who were able to
as the two-parent family is the ideological norm perceived become a scientist were not allowed into the inner circles
by the schools and society. In other words, researchers are of the emerging scientific community (Zuckerman, Cole,
rarely insiders or outsiders but oftentimes insiders and out- and Bruer 1991). While women’s representation, experi-
siders at the same time, and research is constructed between ence, and advancement in science has increased over time,
the researcher and many Others. many of them continue to face barriers, especially at the
Dwyer and Buckle (2009) argue that both the insider and cultural and structural levels (Zuckerman and Cole 1975;
the outsider statuses have pros and cons, so what is impor- Rosser 2004; Hill, Corbett, and St Rose 2010; National
tant is not the insider or the outsider status but “an ability to Research Council of the National Academies 2010; Ceci
be open, authentic, honest, deeply interested in the experi- et al. 2014). This is especially true in computer science
ence of one’s research participants, and committed to accu- (CS), where, unlike other scientific fields, women’s partic-
rately and adequately representing their experience.” In fact, ipation has been consistently low, with some fluctuations.
researchers occupy the ‘space between.’ Challenging the di- Hayes (2010) records the changes of women’s representa-
chotomy and the static nature of insider versus outsider sta- tion in CS in multiple decades: women represented 11% of
tus, the ‘space between’ recognizes the evolving nature of all CS bachelor’s degree recipients in 1967; this percentage
the researcher’s life experience and knowledge on the re- peaked at 37% in 1984 and then declined to only 20% in
search topic as well as her relationship to participants. 2006. For comparison, women represented 44% of all bach-
elor’s degree recipients in 1966 and 58% in 2006, and other
When there are insiders and outsiders in scientific re- STEM fields also witnessed steady increases in this period.
search, there is also a boundary between them. Specifically, Despite the rapid growth of the computer and mathemati-
the boundary delineates what is considered ‘science’ and cal science workforce, women’s proportion declined from
what is considered ‘non-science’ in a particular subfield. 31% in 1993 to 27% in 2017. However, the silver lining is
Boundary work attempts to shape or disrupt the boundary of that among workers with a doctoral degree in these occupa-
what is considered as valid knowledge (Gieryn 1983, 1999). tions, women’s share increased from 16% in 1993 to 31%
Research reveals two types of boundary work: symbolic and in 2017 (National Academies of Sciences, Engineering, and
social boundaries. Symbolic boundaries are formed when Medicine 2020).
members agree on meaning and definition of the field and Multiple factors that oftentimes reinforce each other con-
obtain a collective identity. Social boundaries enable mem- tribute to women’s low representation relative to men’s in
bers’ access to material and non-material resources (e.g., sta- CS at different life stages. Earlier research reports individ-
tus, legitimacy, and visibility) (Lamont and Molnár 2002; ual factors, such as a lack of early exposure to and experi-
Grodal 2018). ence with computing, women students’ inaccurate percep-
For example, Grodal (2018) details how core communi- tions of their low quantitative abilities, and a lack of self
ties who entered the nanotechnology field early expanded confidence despite their good performance and computer
the boundaries of the field by enlarging the definition of the knowledge level. Other research has also focused on so-
field and associating new members. Peripheral communities, cial, cultural, and structural factors which are much harder to
including service providers, entrepreneurs, and university change. For instance, women’s perceptions of their abilities
scientists, self-claimed membership during the expansion and the field of computing could be affected by the ‘chilly
phase due to newly available material and cultural resources. classroom’ with male students’ unfriendly reactions and pro-
Later on, while some peripheral communities continued to fessors’ lack of attention to them; a lack of role models and
associate themselves to nanotechnology, the core communi- mentoring; stereotypes against women and against the peo-
ties, realizing their collective identity being threatened and ple, work involved, and values of CS; and the perceived mis-
resources being restricted because of the enlarged symbolic match of women’s career orientation to help people and so-
boundaries, contracted boundaries by restricting the defini- ciety and what they think CS could offer. Combating these
tion and policing membership. Also, some peripheral com- barriers could increase women’s representation in CS or
munities, not identifying strongly with the more restrictive lower their attrition from CS (Gürer and Camp 2002; Beyer,
collective identity, self-disassociated and focused on other Rynes, and Haller 2004; Beyer and DeKeuster 2006; Co-
fields of interest. In this process, the insiders or the core hoon 2006; Kim, Fann, and Misa-Escalante 2011; Cheryan,
communities entered the field earlier and had a vested inter- Master, and Meltzoff 2015; Lehman, Sax, and Zimmerman
est to protect, while the outsiders or the peripheral commu- 2016; Cheryan et al. 2017).
nities entered the field later and had a weak association with Policy recommendations and college intervention pro-
the field. The insiders had more power than the outsiders in grams have been made and established to change the cul-
defining the boundaries of the field and making certain types tural and institutional environment in order to recruit and re-
of work and research legitimate. tain more women students and professionals in CS. Some ofthe recommendations were repeatedly made in different time Hill 2018). While minority women scientists of different
periods, reflecting a reluctance of change over time. They in- racial/ethnic groups differ from each other in their career
clude involving women students in research at both the un- experience and outcomes, they all tend to fare less well
dergraduate level and early in their graduate study, actively than comparable white women as well as men of the same
countering stereotypes and misperceptions of CS, and high- racial/ethnic group (Malcom, Hall, and Brown 1976; Mal-
lighting and showing women students the positive social im- com and Malcom 2011; Pearson 1985; Ong et al. 2011; Tao
pact that scientists can make and the diverse group of scien- 2018; Tao and McNeely 2019), revealing the persistent in-
tists making social impacts in their fields (Cuny and Aspray tersectional effect of race and gender.
2002; National Academies of Sciences, Engineering, and
Medicine 2020). Successful college intervention programs Status and Career/Research Focus
in increasing the number and percentage of women CS stu- Broadly speaking, women2 tend to be more engaged than
dents and their sense of belonging all tackled the culture of their male peers in relatively new, interdisciplinary scien-
CS and the institution instead of changing the (women) stu- tific fields (e.g., environmental studies) that are oftentimes
dents. These efforts changed the stereotypes of CS by creat- more contextual and problem-based than traditional fields,
ing introductory CS courses to be inclusive of a diverse stu- may not have existing gender hierarchy, and are not well-
dent body, providing role models and mentoring to women embedded in the structure of academia or knowledge pro-
students, providing research experience, and exposing stu- duction, providing more opportunities for women to build
dents to a wide range of applications of CS in solving so- the discipline (Rhoten and Pfirman 2007). While some
cietal issues (Roberts, Kassianidou, and Irani 2002; Muller women shy away from technical fields because they do
2003; Wright et al. 2019; Frieze and Quesenberry 2019; Na- not see the social engagement of these fields, e.g., (Carter
tional Academies of Sciences, Engineering, and Medicine 2006), those who choose technical fields do so not only
2020). because of the excitement of solving technical problems,
but also the potential of addressing issues concerning them
Participation of Racial and Ethnic Minorities and positively impacting people’s lives, which is consistent
In addition to gender, race/ethnicity also shapes scientists’ with their interpersonal and career orientations (Silbey 2016;
representation and experience in science as well as their out- Bossart and Bharti 2017). Women CS majors choose com-
sider and insider statuses. Among racial/ethnic minorities in puting in the context of what they could do for the world
a United States context, while Asians tend to be overrepre- with computing—they would like to use the computer in the
sented in science, the other groups (blacks, Hispanics, and broader context of education, medicine, music, communi-
American Indians or Alaska natives) are considered as un- cation, healthcare, environmental studies, crime prevention,
derrepresented minorities (URMs) due to their low represen- etc. While they also enjoy exploring the computer, the main
tation in scientific fields, despite their growth over time. For factor reported by men, women are more likely than their
instance, URMs made up 9% of workers in computer sci- male peers to address the broader social context (Fisher,
ence and mathematics occupations in 2003, and this percent- Margolis, and Miller 1997; Carter 2006; Hoffman and Fried-
age increased to 13% in 2017 (Khan, Robbins, and Okrent man 2018). While few women in AI were at the “outer cir-
2020). While their participation increased over time, it was cle” in its initial stage, they were attracted to it when it
still lower than their representation in the general popu- started to develop in the 1980s and 1990s because it was
lation, confirming their persistent “outsider” status in sci- more cognitive than other areas of CS and there were fewer
ence. Similar trends hold in a world context with Asians and existing stereotypes to fight against (Strok 1992). The in-
whites overrepresented compared to black, Hispanic, and in- tersection of machine learning and society makes careers in
digenous people. machine learning meaningful to them (Hoffman and Fried-
Research on race and science finds that racial/ethnic mi- man 2018).
norities, especially URMs, tend to be less likely to pub-
lish their research, receive research grants, get recognition 5 Empirical Study of Knowledge Production
for their work, and get promoted but more likely to work
As discussed in Section 4, women and URMs tend to select
in institutions with less resources and more likely to be
fields in which they perceive they can help people and so-
marginalized in formal and informal scientific communities
ciety. The intersection of machine learning and society pro-
than their white counterparts. Research also reveals some
vides exactly that opportunity to make social impact. There-
improvement in their representation in scientific fields as
fore, we hypothesize that women and URMs are more likely
well as in their career experience and outcomes over time,
to contribute to research in machine learning and society
but the progress is slow relative to the growth of the sci-
rather than machine learning without a direct societal com-
entific workforce (Pearson 1985, 2005; Ginther et al. 2011;
ponent.
Ginther 2018; Tao and McNeely 2019). In the meantime,
We performed the following analysis to test our hypoth-
an increasing number of studies employ intersectionality as
esis. On September 19, 2020, we downloaded the full col-
the research framework that indicates power relations and
social inequalities to examine the double disadvantages that 2
In this part of the paper and later, we discuss women more
minority women scientists suffer from due to both their gen- than Hispanics and blacks, not because of differing experiences,
der and race, e.g., (Malcom, Hall, and Brown 1976; Mal- but because of a dearth of published literature on the analogous
com and Malcom 2011; Collins 2015; Metcalf, Russell, and experience (Spertus 1991).Table 1: Average soft classification score of race/ethnicity
and gender for three categories of authors.
Asian Hispanic Black White Male
no cs.cy 0.370 0.077 0.057 0.497 0.791
both 0.367 0.073 0.055 0.504 0.777
only cs.cy 0.266 0.097 0.071 0.566 0.726
slope -0.0430 0.0077 0.0055 0.0298 -0.0293
p-value 0 0.0062 0.0241Table 2: Average soft classification score of race/ethnicity Table 4: Average soft classification score of gender among
among estimated males for three categories of authors. estimated Asians for three categories of authors.
Asian Hispanic Black White Male
no cs.cy 0.335 0.078 0.059 0.528 no cs.cy 0.738
both 0.343 0.076 0.057 0.525 both 0.732
only cs.cy 0.247 0.097 0.073 0.583 only cs.cy 0.696
slope -0.0345 0.0072 0.0051 0.0223 slope -0.0183
p-valueTable 6: Average soft classification score of gender among learning and society represents an area of AI that is relatively
estimated blacks for three categories of authors. new, interdisciplinary, not well-embedded in the structure
of academia, and without existing hierarchies, and thus one
Male with an opportunity for women and URMs to build, which
no cs.cy 0.825
they are doing.
both 0.871
While our empirical analysis finds that women of differ-
only cs.cy 0.721
ent racial/ethnic groups tend to behave more similarly to
slope -0.0345 each other than to their male counterparts, we also find that
p-value 0
the women’s groups differ from each other, confirming the
intersectionality perspective and that some groups are not
Table 7: Average soft classification score of gender among purely insiders or purely outsiders. We would like to high-
estimated whites for three categories of authors. light Asian women, who are in a unique position in ma-
chine learning (or science as a whole) because Asians are
Male overrepresented but women are underrepresented in science.
no cs.cy 0.822
Asian men tend to behave similarly to their white counter-
both 0.801
only cs.cy 0.739
parts in their career outcomes, but Asian women tend to
behave more like other women’s groups, making the gen-
slope -0.0375
der gap among Asians greater than that among some other
p-value 0
racial/ethnic groups (Tao 2015, 2018; Tao and McNeely
2019). Being insiders in machine learning on the one hand
(Asians) and being outsiders on the other hand (women)
result, it is hard to get into the inner circle of the community. could possibly constrain some of their choices because they
However through the empirical study of Section 5, we know may receive inconsistent expectations and experience multi-
that women and URMs are overrepresented in research on level barriers. In the meantime, the Asian and Asian Amer-
machine learning and society as compared to plain machine ican cultures tend to emphasize technical expertise and the
learning research. instrumental value of education to fight their marginal status
and to achieve upward social mobility in American society
Analysis of the Current State (Xie and Goyette 2003; Min and Jang 2015). The emphasis
Despite being at the center of building the field of machine on technical aspects and some structural barriers they experi-
learning and society research, women’s (and URM’s) experi- ence in their careers may suggest that Asian women pursue
ence in the workplace reflects their overall struggles in soci- machine learning occupations due to the technical and fi-
ety. Similar to women in some other scientific fields, women nancial aspects more than the social impact of such occupa-
computer scientists tend to be more subject to stereotyping, tions that are more likely to be highlighted by other women’s
less likely to be full professors or in senior research and groups.
technical positions, less recognized for their work and paid In addition, the findings reveal complicated issues of
less, more likely to be subject to overt discrimination and power and inequality in the ML community, which reflects
harassment, more likely to face pressures in balancing work societal inequality. Both at the personal level, e.g., in terms
and life, and more likely to be marginalized than their male of exposure to and experience with computing at an early
peers (Strok 1992; Simard and Gilmartin 2010; Rosser 2004; age, and at the cultural and structural levels, e.g., in terms
Tao 2016; Fox and Kline 2016; Khan, Robbins, and Okrent of experiences in computing classes and workplace, statuses
2020). These “outsider” disadvantages provide them with (e.g., as women or racial/ethnic minorities) affect our lived
the insider perspective when conducting algorithmic fair- experience and opportunities to pursue a career in science.
ness and other socially-oriented machine learning research. When entering science, our lived experience could impact
In fact, women (and URM) scientists’ lived experience and our research focus. While women and URMs are not out-
consequent insider status place them in a unique position to siders to ML in the sense of being less technically com-
formulate questions and conduct research at the intersection petent, they are outsiders as historically underrepresented
of machine learning and society. groups that have not been successful in penetrating into the
The finding that women and underrepresented minorities inner circle. As a result, they are not in a position of power
are more likely to work on machine learning and society but are disadvantaged in various ways. In the meantime, be-
research should not be interpreted as that all insiders con- ing insiders to the experience of inequality, they use their
duct only machine learning research without the social as- technical expertise to address and provide solutions to per-
pect and all outsiders conduct only machine learning and sistent social inequality. In this sense, they are empowering
society research. However, this finding is consistent with lit- not only themselves as underrepresented groups but also the
erature that reveals women’ and underrepresented minori- ML community by raising awareness and impact of ML re-
ties’ preference for conducting and applying research in a search with social implications.
broader context—one that goes beyond the technical. As in-
siders of social inequality, they bring their lived experience Epistemic Conflict
and the new perspective into a field where they have been Outsiders’ entrance into the field could be shaped by ex-
outsiders. Now in the late 2010s and early 2020s, machine isting barriers and policed by the insiders. Once outsidersenter a field, they have another challenge of making legiti- it may not happen soon and there may be backlashes.
mate the research that they prefer but somehow diverge from
the mainstream. Although research driven by lived experi- 7 Case Study
ence (including the third category of FAccT research that Let us see if our boundary work predictions from Section
brings in feminist, post-colonial, and other related critical- 6 hold in a specific case study. In June 2020, the soft-
theoretic thoughts) may be celebrated within the intersec- ware for an image super-resolution algorithm (Menon et al.
tion of machine learning and society, it is questioned out- 2020) was posted on GitHub and soon discovered to alter
side of the intersection on epistemic grounds. According the perceived race/ethnicity of individuals whose downsam-
to Haraway (1988), knowledge is situated and embodied in pled face images were presented as input (Johnson 2020a;
specific locations and bodies, and the multidimensional and Kurenkov 2020; Vincent 2020). Examples of input black,
multifaceted views and voices, from both those in power and Hispanic, and Asian face images yielded white-looking re-
those with limited voices, combine to make science. Never- sults. About these results, Facebook machine learning re-
theless, despite scholarship supporting lived experience not searcher Yann LeCun commented on Twitter: “ML systems
being in conflict with scientific objectivity, the common re- are biased when data is biased. This face upsampling system
frain summarized by the feminist and postcolonial episte- makes everyone look white because the network was pre-
mologist Sandra Harding is as follows: “‘Real sciences’ are trained on FlickFaceHQ, which mainly contains white peo-
supposed to be transparent to the world they represent, to ple pics. Train the *exact* same system on a dataset from
be value neutral. They are supposed to add no political, so- Senegal, and everyone will look African.”
cial, or cultural features to the representations of the world In response, Google machine learning researcher Timnit
they produce.” In other words, ways of knowing that do not Gebru pointed to the video of her recently completed tutorial
follow the (Western) scientific method are not seen by prac- (with Emily Denton) Fairness Accountability Transparency
titioners as scientific (Harding 2006), despite scholarly crit- and Ethics in Computer Vision with the comment: “Yann,
icisms to this perspective. The implication in the context of I suggest you watch me and Emily’s tutorial or a number
machine learning and society is that critical-theoretic work of scholars who are experts in this are. You can’t just re-
based on lived experience as the source of knowledge will duce harms to dataset bias. For once listen to us people from
be discounted in mainstream machine learning: insider re- marginalized communities and what we tell you. If not now
search by outsiders is precarious. during worldwide protests not sure when.” She also posted:
“I’m sick of this framing. Tired of it. Many people have tried
Boundary Work Predictions to explain, many scholars. Listen to us. You can’t just reduce
Based on the sociology theory, we may predict two possible harms caused by ML to dataset bias.” A back and forth de-
futures, both involving boundary work. The first is a sever- bate ensued on Twitter with many interlocutors taking sides
ing of the connection between mainstream machine learn- and offering inputs.
ing research and societally-relevant applications and gover- Let us analyze what happened using the insider/outsider
nance, i.e., the expulsion of machine learning and society understanding of research on machine learning and society
from mainstream machine learning. The second is the ex- that we have developed in this paper. LeCun is a white male,
pansion of machine learning research to include knowledge Chief AI Scientist at Facebook, and Turing award winner—
from lived experience, while overcoming tendencies for ex- a person likely without lived experience of marginalization
pulsion and the protection of autonomy that many insiders and a clear insider in mainstream machine learning research.
of machine learning research may have. Gebru is a black female, co-lead of the Ethical Artificial In-
The future that emerges among the two possibilities could telligence Team at Google at the time—a person with lived
depend on what insiders perceive as legitimate, as in the experience of marginalization and thus an insider in algo-
case of nanotechnology. (The insiders hold epistemic au- rithmic fairness research, but an outsider in machine learn-
thority both due to their entrenched status and the power ing overall.
that comes from their identity (race/ethnicity, gender, etc.) Although Gebru et al. (2019) say: “Of particular concern
(Pereira 2012, 2019).) In addition, another factor may shape are recent examples showing that machine learning mod-
the future of machine learning and society research: sus- els can reproduce or amplify unwanted societal biases re-
tainability of the ML field as a thriving site of research to flected in datasets,” which is consistent with LeCun’s argu-
continuously attract the next generation of scholars, includ- ment, Gebru’s comments in the debate point to her holding a
ing women and underrepresented minorities. When the out- stance consistent with Merton’s insider doctrine of lived ex-
siders conduct their machine learning and society research, perience providing privilege (bordering on exclusivity) for
they raise awareness of the broader context of technical is- conducting research on machine learning bias. Additionally,
sues. While machine learning and society research is still her epistemic perspective appears to be that such lived ex-
a small portion of machine learning research overall, it has perience is a valid source of knowledge for “many schol-
been growing. Led by “the outsiders,” this line of inquiry is ars.” The repeated call to listen to scholars is an attempt at
increasingly being addressed and published. Based on this expansion boundary work. On the other hand, LeCun’s per-
trend and considering that the field could benefit from both spective epitomizes a boundary and epistemic authority that
insiders and outsiders’ perspectives, we have reasons to be- leaves lived experience out of machine learning research;
lieve that machine learning and society research will trans- scientifically-derived knowledge is valid knowledge. At the
form and sustain ML knowledge and practice, even though end, some white males in positions of power also joinedthe debate and offered allyship, which may have expanded tension over the boundaries of valid knowledge in machine
the epistemic boundary of machine learning just a little bit. learning and society. Specifically, the epistemic question that
What may have appeared at first glance to be a personal war arises is whether lived experience is a valid source of knowl-
of words was in fact an example of boundary work in prac- edge. Instances of expansion and expulsion boundary work
tice, manifested as contention between two insiders of their are predicted and verified in a case study.
own respective domains. If one takes the normative stance that expansion bound-
After we completed the first draft of this paper in Oc- ary work is preferable to expulsion, then the resolution of
tober 2020, there was further contention involving Timnit the epistemic contention calls for facilitation by researchers
Gebru in December 2020 (Johnson 2020b). She was dis- with ascribed status that is not marginalized and who lean
missed from her research position at Google by Jeff Dean towards including knowledge derived from lived experience
and Megan Kacholia, who are both white and in positions within the boundary of machine learning research.
of power. The dismissal was widely argued in a public man- This paper illuminates a few avenues for future research.
ner. Among others, one of the factors was Gebru’s reluctance One such direction is to dive deeper into the topics of situ-
to remove Google authors from or withdraw the paper “On ated knowledge, feminist epistemology, and boundary work
the Dangers of Stochastic Parrots: Can Language Models to better understand how the field of machine learning and
Be Too Big? a ” (Bender et al. 2021) from the ACM FAccT society may evolve and to understand strategies for directing
Conference. One of the main parts of this paper conducts a that evolution in a beneficial way. Another future direction is
critical analysis of large language models through the lens to study the impact of papers in machine learning and soci-
of decolonizing hegemonic views (Srigley and Sutherland ety produced by teams containing only members with lived
2018), which is a prototypical example of the third, social experience with marginalization, containing only members
justice, angle to FAccT research mentioned in Section 2. without lived experience with marginalization, and contain-
Dean and Kacholia’s criticism of the paper was the inclusion ing both types of researchers, to understand whether work
of the decolonial perspective at the expense of a technical- that bridges the epistemic divide of formal science and crit-
only analysis that would include a discussion of techniques ical theory is more valuable than other pieces of work.
to mitigate representation bias, which correspond to the sec-
ond kind of FAccT research mentioned in Section 2. The first 9 Acknowledgments
author of the paper, Emily Bender, posted on Twitter: “The
The authors thank Delia R. Setola for her assistance and Tina
claim that this kind of scholarship is ‘political’ and ‘non-
M. Kim for her comments.
scientific’ is precisely the kind of gate-keeping move set up
to maintain ‘science’ as the domain of people of privilege
only.” This case is also one of epistemology and further il- References
lustrates how boundary work can engender extreme conflict Arnold, M.; Bellamy, R. K. E.; Hind, M.; Houde, S.; Mehta,
and the tendency for expulsion boundary work. S.; Mojsilović, A.; Nair, R.; Natesan Ramamurthy, K.;
Together, these cases illustrate a little bit of expansion Olteanu, A.; Piorkowski, D.; Tsay, J.; and Varshney, K. R.
boundary work and a healthy dose of expulsion boundary 2019. FactSheets: Increasing Trust in AI Services Through
work, both as predicted by the theory of insiders and out- Supplier’s Declarations of Conformity. IBM Journal of Re-
siders. search and Development 63(4/5): 6.
Barocas, S.; and Selbst, A. D. 2016. Big Data’s Disparate
8 Conclusion Impact. California Law Review 104: 671.
In this paper, we have studied the sociology of researchers Bender, E.; Gebru, T.; McMillan-Major, A.; and Shmitchell,
creating new knowledge in the area of machine learning. We S. 2021. On the Dangers of Stochastic Parrots: Can Lan-
have analyzed the intersection of general machine learning
research with FAccT and AI for social good—collectively guage Models Be Too Big? a . In Proceedings of the ACM
machine learning and society—using Merton’s concepts of Conference on Fairness, Accountability, and Transparency.
insiders and outsiders. Although these concepts are usually Beyer, S.; and DeKeuster, M. 2006. Women in Computer
only applied in studying the social sciences and humanities Science or Management Information Systems Courses: A
rather than the natural sciences and formal sciences, there is Comparative Analysis. In Cohoon, J.; and Aspray, W., eds.,
a clear insider in terms of lived experience in machine learn- Women and Information Technology: Research on Under-
ing and society: a member of a marginalized group. Just as representation. MIT Press.
importantly, these same marginalized groups have low his-
Beyer, S.; Rynes, K.; and Haller, S. 2004. Deterrents to
torical participation and representation in computer science,
Women Taking Computer Science Courses. IEEE Technol-
the parent field of machine learning and are thus outsiders
ogy and Society Magazine 23(1): 21–28.
in a different way. Through an empirical study, we find that
researchers from marginalized groups are overrepresented in Bhatt, U.; Xiang, A.; Sharma, S.; Weller, A.; Taly, A.; Jia,
conducting research on machine learning and society. There- Y.; Ghosh, J.; Puri, R.; Moura, J. M.; and Eckersley, P. 2020.
fore, we have a situation in which the same group takes the Explainable Machine Learning in Deployment. In Proceed-
insider role in terms of lived experience and the outsider role ings of the ACM Conference on Fairness, Accountability,
in terms of historical participation. This situation leads to and Transparency, 648–657.Blodgett, S. L.; Barocas, S.; Daumé, III, H.; and Wallach, Dwyer, S. C.; and Buckle, J. L. 2009. The Space Between:
H. 2020. Language (Technology) is Power: A Critical Sur- On Being an Insider-Outsider in Qualitative Research. In-
vey of “Bias” in NLP. In Proceedings of the 58th Annual ternational Journal of Qualitative Methods 8(1): 54–63.
Meeting of the Association for Computational Linguistics, Fisher, A.; Margolis, J.; and Miller, F. 1997. Undergraduate
5454–5476. Women in Computer Science: Experience, Motivation and
Bossart, J.; and Bharti, N. 2017. Women in Engineering: In- Culture. ACM SIGCSE Bulletin 29(1): 106–110.
sight into Why Some Engineering Departments Have More
Success in Recruiting and Graduating Women. American Fox, M. F.; and Kline, K. 2016. Women Faculty in Comput-
Journal of Engineering Education 8(2): 127–140. ing: A Key Case of Women in Science. In Branch, E. H., ed.,
Pathways, Potholes, and the Persistence of Women in Sci-
Buolamwini, J.; and Gebru, T. 2018. Gender Shades: Inter- ence: Reconsidering the Pipeline, 41–55. Lexington Books.
sectional Accuracy Disparities in Commercial Gender Clas-
sification. In Proceedings of the Conference on Fairness, Frieze, C.; and Quesenberry, J. L. 2019. How Computer
Accountability, and Transparency, 77–91. Science at CMU is Attracting and Retaining Women. Com-
munications of the ACM 62(2): 23–26.
Carter, L. 2006. Why Students With An Apparent Aptitude
for Computer Science Don’t Choose to Major in Computer Gebru, T.; Morgenstern, J.; Vecchione, B.; Vaughan, J. W.;
Science. ACM SIGCSE Bulletin 38(1): 27–31. Wallach, H.; Daumé, III, H.; and Crawford, K. 2019.
Ceci, S. J.; Ginther, D. K.; Kahn, S.; and Williams, W. M. Datasheets for Datasets. arXiv preprint arXiv:1803.09010v4
2014. Women in Academic Science: A Changing Land- .
scape. Psychological Science in the Public Interest 15(3): Gieryn, T. F. 1983. Boundary-Work and the Demarcation
75–141. of Science from Non-Science: Strains and Interests in Pro-
Chang, H.-C. H.; and Fu, F. 2020. Elitism in Mathematics fessional Ideologies of Scientists. American Sociological
and Inequality. arXiv preprint arXiv:2002.07789 . Review 48(6): 781–795.
Chen, J.; Kallus, N.; Mao, X.; Svacha, G.; and Udell, M. Gieryn, T. F. 1999. Cultural Boundaries of Science: Credi-
2019. Fairness under Unawareness: Assessing Disparity bility on the Line. University of Chicago Press.
When Protected Class Is Unobserved. In Proceedings of the Ginther, D. K. 2018. Using Data to Inform the Science
ACM Conference on Fairness, Accountability, and Trans- of Broadening Participation. American Behavioral Scientist
parency, 339–348. 62(5): 612–624.
Cheryan, S.; Master, A.; and Meltzoff, A. N. 2015. Cul-
Ginther, D. K.; Schaffer, W. T.; Schnell, J.; Masimore, B.;
tural Stereotypes as Gatekeepers: Increasing Girls’ Inter-
Liu, F.; Haak, L. L.; and Kington, R. 2011. Race, Ethnicity,
est in Computer Science and Engineering by Diversifying
and NIH Research Awards. Science 333(6045): 1015–1019.
Stereotypes. Frontiers in Psychology 6: 49.
Cheryan, S.; Ziegler, S. A.; Montoya, A. K.; and Jiang, L. Griffith, A. I. 1998. Insider/Outsider: Epistemological Priv-
2017. Why Are Some STEM Fields More Gender Balanced ilege and Mothering Work. Human Studies 21(4): 361–376.
than Others? Psychological Bulletin 143(1): 1. Grodal, S. 2018. Field Expansion and Contraction: How
Chui, M.; Harryson, M.; Manyika, J.; Roberts, R.; Chung, Communities Shape Social and Symbolic Boundaries. Ad-
R.; van Heteren, A.; and Nel, P. 2018. Applying AI for So- ministrative Science Quarterly 63(4): 783–818.
cial Good. Technical report, McKinsey & Company. Gürer, D.; and Camp, T. 2002. An ACM-W Literature Re-
Cohoon, J. M. 2006. Just Get Over It or Just Get On With It: view on Women in Computing. ACM SIGCSE Bulletin
Retaining Women in Undergraduate Computing. In Cohoon, 34(2): 121–127.
J.; and Aspray, W., eds., Women and Information Technol- Haraway, D. 1988. Situated Knowledges: The science Ques-
ogy: Research on Underrepresentation. MIT Press. tion in Feminism and the Privilege of Partial Perspective.
Collins, P. H. 2015. Intersectionality’s Definitional Dilem- Feminist Studies 14(3): 575–599.
mas. Annual Review of Sociology 41: 1–20.
Harding, S. 2006. Science and Social Inequality: Feminist
Cuny, J.; and Aspray, W. 2002. Recruitment and Retention and Postcolonial Issues. University of Illinois Press.
of Women Graduate Students in Computer Science and En-
gineering: Results of a Workshop Organized by the Com- Hart, K. L.; and Perlis, R. H. 2019. Trends in Proportion of
puting Research Association. ACM SIGCSE Bulletin 34(2): Women as Authors of Medical Journal Articles, 2008–2018.
168–174. JAMA Internal Medicine 179(9): 1285–1287.
Dimanov, B.; Bhatt, U.; Jamnik, M.; and Weller, A. 2020. Hayes, C. C. 2010. Computer Science: The Incredible
You Shouldn’t Trust Me: Learning Models Which Conceal Shrinking Woman. In Misa, T. J., ed., Gender Codes: Why
Unfairness From Multiple Explanation Methods. In AAAI Women Are Leaving Computing, 25–49. John Wiley & Sons.
Workshop on Artificial Intelligence Safety, 63–73. Hayfield, N.; and Huxley, C. 2015. Insider and Outsider Per-
Du, M.; Liu, N.; and Hu, X. 2019. Techniques for Inter- spectives: Reflections on Researcher Identities in Research
pretable Machine Learning. Communications of the ACM with Lesbian and Bisexual Women. Qualitative Research in
63(1): 68–77. Psychology 12(2): 91–106.Hill, C.; Corbett, C.; and St Rose, A. 2010. Why So Few? Malcom, S. M.; Hall, P. Q.; and Brown, J. W. 1976. The
Women in Science, Technology, Engineering, and Mathemat- Double Bind: The Price of Being a Minority Woman in Sci-
ics. AAUW. ence. Report of a Conference of Minority Women Scientists,
Hind, M. 2019. Explaining Explainable AI. ACM XRDS Arlie House, Warrenton, Virginia. Technical Report 76-R-3,
Magazine 25(3): 16–19. American Association for the Advancement of Science.
Hoffman, S. F.; and Friedman, H. H. 2018. Machine Learn- Menon, A. K.; and Williamson, R. C. 2018. The Cost of
ing and Meaningful Careers: Increasing the Number of Fairness in Binary Classification. In Proceedings of the Con-
Women in STEM. Journal of Research in Gender Studies ference on Fairness, Accountability and Transparency, 107–
8(1): 11–27. 118.
Menon, S.; Damian, A.; Hu, S.; Ravi, N.; and Rudin, C.
Hofstra, B.; Kulkarni, V. V.; Galvez, S. M.-N.; He, B.; Ju-
2020. PULSE: Self-Supervised Photo Upsampling via La-
rafsky, D.; and McFarland, D. A. 2020. The Diversity–
tent Space Exploration of Generative Models. In Proceed-
Innovation Paradox in Science. Proceedings of the National
ings of the IEEE/CVF Conference on Computer Vision and
Academy of Sciences 117(17): 9284–9291.
Pattern Recognition, 2437–2445.
Jobin, A.; Ienca, M.; and Vayena, E. 2019. The Global Land- Mercer, J. 2007. The Challenges of Insider Research in Ed-
scape of AI Ethics Guidelines. Nature Machine Intelligence ucational Institutions: Wielding a Double-Edged Sword and
1(9): 389–399. Resolving Delicate Dilemmas. Oxford Review of Education
Johnson, K. 2020a. AI Weekly: A Deep Learning Pioneer’s 33(1): 1–17.
Teachable Moment on AI Bias. VentureBeat . Merton, R. K. 1972. Insiders and Outsiders: A Chapter in the
Johnson, K. 2020b. Google AI Ethics Co-Lead Timnit Ge- Sociology of Knowledge. American Journal of Sociology
bru Says She Was Fired over an Email. VentureBeat . 78(1): 9–47.
Kearns, M.; Neel, S.; Roth, A.; and Wu, Z. S. 2019. An Em- Metcalf, H.; Russell, D.; and Hill, C. 2018. Broadening the
pirical Study of Rich Subgroup Fairness for Machine Learn- Science of Broadening Participation in STEM through Criti-
ing. In Proceedings of the ACM Conference on Fairness, cal Mixed Methodologies and Intersectionality Frameworks.
Accountability, and Transparency, 100–109. American Behavioral Scientist 62(5): 580–599.
Kerstetter, K. 2012. Insider, Outsider, or Somewhere Miller, T. 2019. Explanation in Artificial Intelligence: In-
Between: The Impact of Researchers’ Identities on the sights from the Social Sciences. Artificial Intelligence 267:
Community-Based Research Process. Journal of Rural So- 1–38.
cial Sciences 27(2): 7. Min, P. G.; and Jang, S. H. 2015. The Concentration of
Khan, B.; Robbins, C.; and Okrent, A. 2020. The State of Asian Americans in STEM and Health-Care Occupations:
US Science and Engineering 2020. National Science Foun- An Intergenerational Comparison. Ethnic and Racial Stud-
dation, January 15. ies 38(6): 841–859.
Kim, K. A.; Fann, A. J.; and Misa-Escalante, K. O. 2011. Mitchell, M.; Wu, S.; Zaldivar, A.; Barnes, P.; Vasserman,
Engaging Women in Computer Science and Engineering: L.; Hutchinson, B.; Spitzer, E.; Raji, I. D.; and Gebru, T.
Promising Practices for Promoting Gender Equity in Un- 2019. Model Cards for Model Reporting. In Proceedings
dergraduate Research Experiences. ACM Transactions on of the ACM Conference on Fairness, Accountability, and
Computing Education 11(2): 1–19. Transparency, 220–229.
Kind, C. 2020. The Term ‘Ethical AI’ is Finally Starting to Mohamed, S.; Png, M.-T.; and Isaac, W. 2020. Decolonial
Mean Something. VentureBeat . AI: Decolonial Theory as Sociotechnical Foresight in Arti-
ficial Intelligence. Philosophy & Technology 33: 659–684.
Kurenkov, A. 2020. Lessons from the PULSE Model and
Discussion. The Gradient . Muller, C. B. 2003. The Underrepresentation of Women in
Engineering and Related Sciences: Pursuing Two Comple-
Labaree, R. V. 2002. The Risk of ‘Going Observationalist’: mentary Paths to Parity. In Pan-Organizational Summit on
Negotiating the Hidden Dilemmas of Being an Insider Par- the US Science and Engineering Workforce: Meeting Sum-
ticipant Observer. Qualitative Research 2(1): 97–122. mary, 119–126. National Academies Press.
Lamont, M.; and Molnár, V. 2002. The Study of Boundaries National Academies of Sciences, Engineering, and
in the Social Sciences. Annual Review of Sociology 28(1): Medicine. 2020. Promising Practices for Addressing the
167–195. Underrepresentation of Women in Science, Engineering,
Lehman, K. J.; Sax, L. J.; and Zimmerman, H. B. 2016. and Medicine: Opening Doors. National Academies Press.
Women Planning to Major in Computer Science: Who Are National Research Council of the National Academies.
They and What Makes Them Unique? Computer Science 2010. Gender Differences at Critical Transitions in the
Education 26(4): 277–298. Careers of Science, Engineering, and Mathematics Faculty.
Malcom, L. E.; and Malcom, S. M. 2011. The Double Bind: National Academies Press.
The Next Generation. Harvard Educational Review 81(2): Ong, M.; Wright, C.; Espinosa, L.; and Orfield, G. 2011. In-
162–172. side the Double Bind: A Synthesis of Empirical ResearchYou can also read
