ON THE FOUNDATIONS OF DEMOCRATIC EDUCATION
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
CONTINUING A TRADITION OF RESEARCH
ON THE FOUNDATIONS OF DEMOCRATIC EDUCATION:
THE NATIONAL PROFESSIONAL DEVELOPMENT
AND EVALUATION PROJECT
Dennis J. Barr
Director of Evaluation, Facing History and Ourselves
and
Facing History and Ourselves
Copyright 2010 © Facing History and Ourselves National Foundation, Inc.
TABLE OF CONTENTS
OVERVIEW.................................................................................................................................................. 3
EXECUTIVE SUMMARY ........................................................................................................................... 4
THE FACING HISTORY APPROACH..................................................................................................... 7
A TRADITION OF RESEARCH ON THE FOUNDATIONS OF EDUCATING FOR DEMOCRACY..............................8
LAUNCHING THE NEXT GENERATION OF FACING HISTORY EVALUATION RESEARCH.11
THE NATIONAL PROFESSIONAL DEVELOPMENT AND EVALUATION PROJECT ...............13
UNDERTAKING A RANDOMIZED CONTROLLED EXPERIMENT ........................................................................ 15
IMPLEMENTING THE OUTCOME RESEARCH ...................................................................................................... 17
Research Design .......................................................................................................................................... 18
Research Questions .................................................................................................................................... 19
Sample ............................................................................................................................................................. 19
Measures......................................................................................................................................................... 20
Teacher measures................................................................................................................................... 21
Fidelity of implementation.........................................................................................................................23
Student measures............................................................................................................................................23
FINDINGS ................................................................................................................................................................ 25
Overview of Teacher Outcomes ............................................................................................................ 25
Teacher selfefficacy......................................................................................................................................25
Satisfaction with teaching and professional growth .....................................................................26
Fidelity of Implementation of Facing History ................................................................................. 26
Overview of Student Impacts ................................................................................................................. 27
Historical understanding ...........................................................................................................................27
Civic learning ....................................................................................................................................................28
Social and ethical awareness ....................................................................................................................29
EDUCATIONAL IMPLICATIONS..........................................................................................................30
PUTTING THE STUDY IN CONTEXT ..................................................................................................................... 33
ONGOING DATA ANALYSES ON WRITTEN RESPONSES ................................................................................... 34
Teacher Written Responses.................................................................................................................... 34
Student Written Responses .................................................................................................................... 35
CONCLUSION............................................................................................................................................36
REFERENCES............................................................................................................................................38
APPENDIX.................................................................................................................................................45
2
OVERVIEW
Facing History and Ourselves is an international educational organization that provides educators,
students, and the community at large with an evidence-based interdisciplinary model of civic
education that integrates a rigorous investigation of history with critical questions about ethics and
democratic civic engagement. Facing History has nine offices in North America, an international hub
in London, and a network of 29,000 trained educators who reach nearly 1.9 million students each
year. Through major public events such as a traveling multimedia exhibition, speaker series, and
academic conferences in partnership with major universities, Facing History convenes members of
the broader community and reaches audiences beyond the classroom. Facing History’s website
(www.facinghistory.org) and online resources attract more than 700,000 visits from 215 countries and
territories annually.
For more than three decades, Facing History has led dynamic interchanges among scholars and
Facing History staff, focusing on theory, research, and practice related to the profound role that
education can and must play in preparing young people to engage thoughtfully and actively in
democratic societies. This paper highlights some of the nearly 100 studies that have been conducted
on the program, as a context for presenting the latest and most comprehensive evaluation of the
program’s effectiveness: the Facing History and Ourselves National Professional Development and
Evaluation Project (NPDEP). The NPDEP provides evidence, using the most rigorous evaluation
methods, that Facing History promotes teacher self-efficacy, professional satisfaction and growth,
and student academic and civic learning. This is the foundational publication providing details of the
research. 1 A shorter summary is also available. 2 Additional publications focusing on the measures,
design, and findings will follow. 3
This study provides evidence, using the most rigorous evaluation methods,
that Facing History promotes teacher self‐efficacy, professional satisfaction
and growth, and student academic and civic learning.
1 The author is very grateful to the following people for their contributions to the development of this document:
Betty Bardige, Marty Sleeper, Margot Strom, Anna Romer, Terry Tollefson, Marc Skvirsky, Melinda Fine, Ethan
Lowenstein, Robert Selman and Abbey Mann.
2 http://www.facinghistory.org/eval/npdep
3 The first of these is: Selman, R.L., & Barr, D. J. (2009) Can adolescents learn to create ethical relationships for
themselves in the future by reflecting on ethical violations faced by others in the past? In M. Martens et al. (Eds.),
Interpersonal Understanding in Historical Context, (pp. 19‐41). Rotterdam, The Netherlands: Sense Publishers.
3
EXECUTIVE SUMMARY
Since its founding in 1976, Facing History and Ourselves has recognized that teacher effectiveness is
at the heart of educational success for students. A core component of Facing History’s educational
model, therefore, involves providing professional development seminars, workshops, coaching, and
print and online resources for teachers, helping them to promote their students’ academic
engagement and achievement, while also fostering students’ growth as thoughtful participants in
society.
To provide high-quality, timely, and relevant professional development services and resources for
teachers, Facing History equally emphasizes the professional development of its own staff and board
through interchanges with scholars and participation in research. Throughout the organization’s
history, Facing History’s evaluation staff and independent researchers have carried out nearly 100
studies that have yielded a large body of knowledge about the model’s effectiveness, as well as
knowledge about teacher and adolescent development more generally. Independent experts and
review panels have repeatedly validated the program’s effectiveness, based on the findings of
evaluation studies. Facing History was selected for membership in the U.S. Department of
Education’s National Diffusion Network (NDN) from 1980–1996 as “an exemplary program worthy
of national dissemination” (Lieberman, 1993a). Since that time, research on Facing History’s model
has been reviewed and provided the basis for external validation as a promising approach under the
U.S. Department of Education’s Safe, Disciplined, and Drug-Free Schools initiative (D.O.E., 2001),
and as a best practice in the fields of civic education (Fine, 2004), character education (Berkowitz &
Bier, 2005), and Holocaust education (Berman, 2006).
In 2005, Facing History launched an ambitious research project to further its longstanding
involvement in research, theory, and practice related to the foundations of educating for democracy:
the National Professional Development and Evaluation Project (NPDEP). In this paper, we describe
the multifaceted NPDEP, focusing on its core outcome study—a randomized controlled experiment
(hereafter referred to as the “experimental study”)—that investigates the causal impacts of a Facing
History and Ourselves professional development intervention on high school teachers’ sense of
professional efficacy, satisfaction, and growth of their students’ engagement and learning. 4 An
independent research agency, Abt Associates, Inc., carried out the experimental study to ensure the
objectivity of the data collection and analysis. Further, independent university-based researchers,
Robert Selman, Melinda Fine, and Ethan Lowenstein, served as co-investigators to further ensure the
objectivity and validity of the research.
The experimental study involves 134 teachers, and 1,371 of their students in 76 schools across the
United States. High schools that had no previous exposure to Facing History and Ourselves and
were located within one hour of travel time to one of Facing History’s regional offices, and teachers
who had never had Facing History professional development nor had taught the program were
eligible to participate in the study. Sixty-six percent of these schools were “low-performing” using
U.S. Department of Education criteria for math and literacy test scores. Other components of the
NPDEP project experimental study include exploratory research and capacity-building activities for
the organization.
5 Thefocus of this paper is on the hypothesis‐testing component of the experimental study. The exploratory
aspects of the experimental research, and other aspects of the NPDEP that were not within the experimental
design, will be presented in more detail in future papers.
4
FINDINGS The initial results 5 of the NPDEP research on teachers demonstrate that Facing History’s educational model is scalable beyond teachers and schools who actively seek to implement its curricula or methodology. The professional development intervention had a statistically significant and educationally meaningful impact on all aspects of teacher self-efficacy that were measured, as well as on teacher satisfaction and professional growth 6 (Boulay, McCormick, & Kliorys, 2009). Facing History teachers felt more capable than did control group teachers of creating classroom environments and implementing teaching practices to promote students' historical understanding, civic learning, ethical awareness, and character development. In addition, Facing History teachers were more energized and motivated by their professional development experiences than were teachers in the control group and felt a greater sense of accomplishment, engagement, and growth as teachers. 7 No differences were found between Facing History and control teachers in the degree of their emotional exhaustion or depersonalization (disengagement from their work). These findings were sustained longitudinally over two years. Further, the study was replicated (the same outcomes were found) with a second cohort of teachers after their first year of using Facing History resources. With regard to the student research, Facing History students scored higher than control students, on average, for 100% of the academic and civic outcomes measured and demonstrated statistically significant differences on certain scales within each of these categories. 8 Specifically, in the area of historical understanding, Facing History had a statistically significant impact on students’ overall historical understanding. The scores of three sub-scales are used to calculate the overall historical understanding score: skills for interpreting evidence; skills for analyzing what leads people to make ethical choices and skills for thinking critically about cause and effect. In the area of civic learning, Facing History had a statistically significant impact on five civic learning outcomes within three domains: tolerance, efficacy, and opportunities for civic learning. Specifically, positive impacts on civic learning were found for the following variables: civic efficacy, valuing the protection of the civil liberties of people with different political views, awareness of the dangers of prejudice and discrimination, and positive perceptions of their history or English class as offering opportunities to engage with civic matters and as having a safe, inclusive, and respectful climate. These academic and civic findings were replicated in a second year with a new group of students, showing that program effects are sustained in schools over time. 9 No statistically significant differences between Facing History and control students were found on measures of students’ attitudes about ethnic/racial groups, beliefs about the importance of deliberating with others in class about controversial public issues, actual engagement in deliberation of such issues, social awareness, or the parts of the ethical awareness and decision-making measure that have been analyzed thus far. Taken together, the teacher and student findings paint a fuller picture of the impact of Facing History, an educational model that is centered on promoting high-quality teaching and integrating 5 We will continually update this paper as additional results become available. 6P values for group differences on all efficacy outcomes range from .0004 to .0047. The effect sizes range from .49‐.85. The p value for the satisfaction with professional development, expertise, and engagement variable is .0001, and the effect size is 1.00. The p value for the personal accomplishment variable (one aspect of teacher satisfaction) is .0011, and the effect size is .49. 7 The research team assumes that most of the control group teachers received some type of non‐Facing History professional development during the first year of the study. 8 p
academic and civic aims. The study found that Facing History teachers not only felt a greater sense of
efficacy in promoting student academic and civic learning than control teachers but were also
effective in practice as demonstrated by the student outcomes in theoretically aligned areas.
Specifically, the alignment of teacher and student outcomes suggests that Facing History prepares
teachers to address the following critical needs in education:
1. Creating safer and more engaging learning environments
2. Promoting respect for the rights of others whose views differ from one’s own
3. Fostering awareness of the power and danger of prejudice and discrimination
4. Promoting critical thinking about history and contemporary events
5. Increasing students’ belief that they make a difference in society
These initial findings of the NPDEP provide the kind of empirical evidence that policymakers, grant-
making foundations, school administrators, and scholars are hungry for as they seek to identify, and
invest in, “what works.”
A consensus is building among scholars, practitioners, policymakers, and funders committed to
education reform that teacher effectiveness is crucial to educational success for students: a core
operating principle of Facing History and Ourselves since its inception in 1976. Both the United
States government’s American Recovery and Reinvestment Act for education reform and the Gates
Foundation’s educational platform emphasize promoting teacher effectiveness because, as Bill Gates
noted at the November 2008 Forum on Education, “A growing body of evidence tells us that teacher
effectiveness is the single most important factor in student achievement” (Gates, 2008). Previous
research has demonstrated that teachers’ sense of professional efficacy is a critical aspect of their
effectiveness and is associated with student achievement (Ashton, 1984; Woolfolk, Hoy, &
Tschannen-Moran, 1998; Hoy & Davis, 2006) and teacher retention (Yost, 2006). The NPDEP
experimental study provides additional evidence, suggesting a robust connection between teacher
self-efficacy and academic and civic outcomes for students.
While stakeholders are looking for evidence from experimental studies of educational strategies that
aim to impact teacher effectiveness, such studies are relatively scarce. The NPDEP, therefore, makes
a significant contribution by providing a rigorous study of a program’s effectiveness in enhancing
teachers’ capacities to promote student achievement and students’ capacities to participate in society
as thoughtful, caring, and active citizens.
These initial findings of the NPDEP provide the kind of empirical evidence
that policymakers, grant‐making foundations, school administrators, and
scholars are hungry for as they seek to identify, and invest in, “what works.”
6
THE FACING HISTORY AND OURSELVES APPROACH
Facing History’s educational model is based on a sequence of study that moves from individual
identity to responsible civic participation, using historical and literary documents, and the stories of
individuals and groups, to help young people discover the capacity of ordinary people to influence
extraordinary events. Facing History’s content and teaching strategies help students link the lessons
of history to the everyday issues they face in their own lives. By using these methods to study the
actions and decisions of individuals and groups during specific periods in history, students realize
that their choices do matter and that they can be agents of positive change (see Sleeper & Strom,
2006, and Tollefson, Barr, & Strom, 2004, for a more thorough description of Facing History and
Ourselves).
Facing History seminars and workshops introduce educators to Facing History’s pedagogy and the
themes of Facing History’s resources. Facing History’s core resource book, Facing History and
Ourselves: Holocaust and Human Behavior (1994), examines the consequences of hatred by exploring the
events that led to the Holocaust. Facing History is continually developing new resource books to
address a range of historical moments and movements, as well as case studies that illustrate how
citizens can choose to make a positive difference in society.
Facing History believes that, at its best, civic education involves an ongoing constructive tension that
honors dualities in the learning process: engaging students’ minds and their hearts, integrating the
exploration of history and ethics, and seeing both teachers and students as learners. Margot Stern
Strom, the founder and Executive Director of Facing History, underscores the importance of the
moral and civic ends of its model of historical investigation:
We must help students . . . confront not only their own potential for passivity and complicity, but
also their courage and resilience. And we must teach them to value their rights as citizens and take
responsibility for their actions (Strom, 1994, p. xiv).
Facing History believes the adult development of educators to be the key lever of educational
change. To prepare students for thoughtful participation in a democracy, teachers must be capable of
engaging students in a learning process that reflects core aspects of civil society. Teachers are
supported by Facing History to create classroom settings characterized by respectful relationships
and where students deliberate on complex and challenging subject matter. In these classrooms,
sufficient trust develops among students, and between teachers and their students, so that students
can listen to one another, consider one another’s perspectives, connect personally with the content,
take intellectual risks, and learn to form judgments based on a critical analysis of evidence.
Facing History believes the adult development of educators to be the key
lever of educational change.
7
A TRADITION OF RESEARCH ON THE FOUNDATIONS
OF EDUCATING FOR DEMOCRACY
A hallmark of Facing History’s educational model involves creating a dynamic interchange among
theory, research, and practice. The organization has inspired, participated in, and led more than 100
practice-based research studies, generating valuable knowledge for scholars and practitioners, as well
as new tools for assessment, research, and theory building. 10 Facing History regularly brings together
educators and academic scholars with expertise in relevant fields to foster a lively and productive
interchange that enriches both practice and scholarship.
Facing History and Ourselves’ content and methodology is anchored in prominent theories of
adolescent development, teacher professional development, and rigorous research by scholars and
curriculum experts (Strom, 1980; Strom, Sleeper, & Johnson, 1992). For example, the program’s
practices and content are informed by theory and research in the areas of moral development
(Kohlberg, 1976; Gilligan, 1977, 1982), social and ethical development (Selman, 1980, 2003), social
psychology (Staub, 2003), civic learning (Kahne & Westheimer, 2003; Haste, 2005), historical
understanding (Seixas, 1996), intergroup relations (Hawley & Jackson, 1995; Stephan & Vogt, 2004),
and teacher development (Darling-Hammond & Bransford, 2005.)
Facing History has led and inspired research for evaluation and theory-building purposes. The
organization supports an evaluation staff that conducts evaluation studies and works with external
researchers to conduct applied and basic research studies focusing on a wide range of topics related
to how teachers and their students develop their capacities to understand and connect history to their
current social, moral, and civic concerns and commitments (Lieberman, 1981, 1991; Bardige, 1983;
Brabeck, Kenny, Stryker, Tollefson, & Stern Strom, 1994; Fine, 1995; Barr et al., 1998; Schultz, Barr,
& Selman, 2001; Lowenstein, 2003). Many of these studies have provided formative information for
program refinement, while others have yielded extensive evidence of the program’s impact on
teachers, students, and schools.
With respect to teachers, researchers have investigated the influence of the program’s professional
development activities on teachers’ pedagogical and content knowledge, (e.g., Lowenstein, 2003) and
their goals and commitments as teachers. Evaluations have demonstrated the program’s effectiveness
in promoting the growth of key dimensions of teacher effectiveness in integrating an in-depth study
of history and ethics for adolescent students. Teachers who participate in Facing History seminars
report increased confidence, knowledge, and skills necessary for teaching about complex issues raised
by the history of the steps leading to the Holocaust and other examples of genocide, as well as
exploring the connections of the history to complex ethical questions students face (Barr, 2003).
High percentages of teachers indicate that the professional development process revitalizes their
interest in teaching (Barr & Frey, 2004; Romer, 2006a; Romer & Mann, 2008) and causes them to
rethink their methods; they are more likely to utilize questioning, classroom discussion, and content
that relates to important issues of citizenship and individual responsibility today (Lieberman, 1991,
1993b). Further, teachers report that Facing History professional development and classroom
implementation reaffirm their aspirations as teachers and sense of satisfaction in their role. In
longitudinal studies, teachers have consistently reported that these impacts are sustained, even up to
five years after participation in a seminar (Barr, 2002a; Tollefson, 1999; Barr, 2002b; Barr, 2001; Barr
& Frey, 2004).
10For a more detailed summary of past research in the context of Facing History and Ourselves, please see
Evaluation Research Summary, Facing History and Ourselves Evaluation Department,
http://www.facinghistory.org/about/evaluation.
8
With respect to student learning and growth in the context of, and/or as a result of, participation in
Facing History, scholars have explored such topics and outcomes as students’ factual knowledge and
understanding of history (Lieberman, 1981; Glynn, 1982); reflective thinking and pro-social
awareness (Bardige, 1983); moral and psychosocial development (Lieberman, 1981; Brabeck et al.,
1994; Barr et al.,1998; Schultz et al., 2001; Schultz & Barr, 2002; Feigenberg, Steel King, Selman, &
Barr, 2005); empathy and psychological adjustment (Brabeck et al., 1994); violence (Ward, 1988,
Schultz et al., 2001); civic knowledge, skills, and dispositions (Fine, 1991; Schulz & Barr, 2002; );
intergroup relations (Schultz & Barr, 2002); and academic engagement and motivation (Alford &
Romer, 2007; Romer, Barr & Gaber, 2009). Finally, some studies have focused on the classroom
learning environment (Romer & Barr, 2009; Romer, 2007a).
A 2008 review of evaluation studies over the last three decades revealed consistent evidence of
program effectiveness for students in each of four domains: 1) academic engagement and motivation;
2) historical knowledge and understanding; 3) social and moral development and ethical awareness;
and 4) civic knowledge, skills, and values, including intergroup relations.
Within the area of academic engagement, for example, a 2005 study of 9th and 10th graders in Facing
History and comparison classes in a large public school in Florida found that Facing History students
were more likely to report that their classes motivated them to learn (Stecker & Meehan, 2005). Many
other studies (without comparison groups) that ask students to compare Facing History with other
humanities classes they have experienced support the conclusion that Facing History’s professional
support for educators, including its methods, content, and resources, helps teachers to engage their
students’ interest and to motivate students to learn (e.g., Sescher & Barr, 2005; Romer, 2009a).
Students consistently report that Facing History engages them in personally meaningful ways, often
in ways that they had never experienced before and that surprises them.
Students consistently report that Facing History engages them in personally
meaningful ways, often in ways that they had never experienced before and
that surprises them.
A number of studies have documented Facing History’s impact on students’ historical knowledge
and understanding. In one external study, for example, Facing History was compared to three other
approaches to teaching about the Holocaust. Facing History students were found to have gained the
deepest historical knowledge (Glynn, 1982). Many other studies have shown that teachers are
impressed with how Facing History helps their students to learn content and to think more critically
about history and its connections to themselves and to current social and political issues (Barr, 2003;
Romer, 2007b; Romer & Mann, 2009; Romer, 2009b).
Other studies have established evidence of the program’s effectiveness in promoting psychosocial
growth and civic learning. A landmark study funded by the Carnegie Corporation of New York
employed a rigorous, quasi-experimental design comparing a group of Facing History students in
New England to a group of non-Facing History students (Schultz et al., 2001). The findings indicated
positive effects of Facing History on students’ psychosocial growth, including interpersonal
understanding and skills, and reduced racist attitudes. Further, Facing History students reported
decreases in fighting behavior. Aspects of the study were replicated in Tennessee with the same
findings (Schultz & Barr, 2002).
Many evaluations have focused on the impact of Facing History on students’ civic learning. In 2007,
Facing History conducted two studies on the impact of the model using its newest content, Choices in
Little Rock, on civic agency, identity, and engagement among urban middle school students. In both
studies, students agreed most strongly with the following three of the nine statements about their
learning that they were asked to evaluate: Choices in Little Rock increased my capacity to . . . 1) stand
9
up for what I believe, even when others disagree; 2) think critically about issues of racism and
prejudice; 3) get along well with different types of people (Alford & Romer, 2007; Romer, Barr, &
Gaber, 2009).
Beyond impacts on individual teachers and students, research has also shown broader setting-level
effects, including outcomes related to the classroom learning environment and school-level changes
when Facing History is implemented as a model of school reform. With respect to the classroom
learning environment, teachers point to the importance of using Facing History methods and content
to create a democratic ethos in the classroom where students’ voices, deliberation on controversial
issues, and the formation of perspective are valued. Teachers speak about creating a safe atmosphere
that also challenges students to reflect on their assumptions and be open to new ideas. As one urban
teacher put it:
I’m going to ask them what they think, and I’m going to ask them to voice their opinion, and I’m
going to ask that we create a comfortable classroom culture where you are encouraged to say your
opinion, and no one’s going to tell you that you’re wrong. And somebody might not agree with you,
but that’s ok . . . the only way to be successful with the Facing History curriculum is if you have that
culture. (quoted in Romer, 2009a)
A 2006 study of Facing History’s impact on a school-wide turnaround effort at a Chicago middle
school, for example, found that Facing History increased teacher retention, helped increase teachers’
confidence about their positive impact on students, and helped the school recruit qualified teachers,
while also increasing opportunities for student engagement and learning and enhancing the academic
rigor of the curriculum. The school experienced significant increases in student math and reading
scores during the first two years of its partnership with Facing History (Romer, 2007a).
Independent experts have validated the program’s effectiveness based on their review of evaluation
studies. Facing History was initially developed with federal funds designed to improve secondary
education through the teaching of history and ethics (ESEA Title IV, Part C, from 1977 to 1981).
Under the NDN program, federal money helped to support the dissemination of Facing History in
more than 40 states. In 2001, the U.S. Department of Education’s Independent Expert Panel on
Safe, Disciplined, and Drug-Free Schools reviewed evaluation findings and designated Facing History
as a “Promising Program” (D.O.E., 2001). In 2004, an expert in civic education reviewed Facing
History’s program and evaluation research and concluded that the program provides quality civic
education practices in keeping with field principles articulated in the Carnegie Corporation of New
York’s Civic Mission of Schools report. Facing History’s professional development approach was also
found to address a key need in the civic education field, that is, preparing teachers to effectively
foster civic learning in students (Fine, 2004). In 2005, scholars at the Center for Character and
Citizenship conducted a national review of character education programs and their evidence base.
They identified Facing History as a scientifically supported character education program (Berkowitz
& Bier, 2005). Finally, in 2006, based on a review of existing programs and their research evidence,
Facing History was identified as a best practice in Holocaust Education by the Berman Center for
Research and Evaluation in Jewish Education (Berman, 2006).
10LAUNCHING THE NEXT GENERATION OF
FACING HISTORY EVALUATION RESEARCH
In 2004, Facing History reviewed previous research on the program as the basis for a long-term
research agenda. In addition, Facing History reviewed the current landscape of education policy and
research. Though much was already known about the effectiveness of the program, evaluation
methodology and standards of evidence for program effectiveness were evolving. Policymakers,
foundations, and scholars were calling for studies that would provide evidence of causal relationships
between educational interventions and outcomes for students. In addition, developing scholarship in
the fields in which the program was grounded raised new questions about program processes and
outcomes, opening new opportunities for research that could further advance each field’s knowledge
base. Further, over time the organization’s theory of change had been continually refined, as it sought
to maintain an alignment among its practices and resources and accumulating research evidence.
In that context, Facing History embraced the opportunity to ask new research questions to fill gaps
in knowledge that would be of interest to a range of stakeholders. The next generation of research
would be designed to maintain the organization at the forefront of approaches, with a rigorous
evidence base, and to advance the organization’s commitment to initiating dynamic conversations
among theory, research, and practice that feed staff development and contribute new scholarship to
relevant fields.
The evolution of new evaluation methodologies and conceptualizations of teacher and student
growth also brought significant research opportunities. The program’s theory of change, for example,
had always assumed that the key lever of change for student learning and development is teacher
development and effectiveness. Until 2004, however, research in the context of Facing History had
focused on teacher and student outcomes independently of one another. Although more than 30 years
of practical experience had demonstrated, theory had postulated, and other studies had shown strong
links between teacher practices and growth and student outcomes, Facing History research had not
yet provided clear empirical evidence of these connections.
Similarly, while the program’s theory of change posits that in-depth implementation of the program’s
content and methods result in student outcomes within the domains of civic learning, social and
ethical awareness, and historical understanding, these three domains had not been systematically
examined within a single study (see Figure 1). Further, knowledge within each of the relevant fields
of scholarship has generally remained isolated within each discipline. Therefore, while previous
research on Facing History’s model suggested that adolescents, in fact, are motivated by Facing
History’s interdisciplinary, integrative approach to make connections across these domains, little was
known about 1) how effective teachers are at integrating each of these components in their practice
or 2) the processes involved, both interpersonally and intra-psychically, as adolescents integrate their
learning and growth in each domain. The next generation of research would need to be
comprehensive enough to mirror the complexity of the Facing History model in order to yield
knowledge about the model itself, rather than only its constituent parts (Barr, 2007).
Furthermore, the long-term research agenda would entail focusing on filling gaps in knowledge
within each outcome domain. A review of the field of civic learning revealed, for example, that
certain knowledge, skills, and dispositions central to the program’s theory of change, and meaningful
within the field of civic learning, had not received sufficient attention in previous research (Fine,
2004). Facing History sponsored a year-long seminar on civic learning, involving experts in the field,
in order to further refine its long-term research agenda.
11Similarly, within the social and ethical domain, previous studies had a demonstrated impact on
students’ social awareness and skills, core aspects of social development, but had not explored how
students apply such competencies to reflect on ethical decisions they might face in their schools.
Within the domain of historical knowledge and understanding, studies had demonstrated increases in
Facing History students’ factual knowledge and understanding of concepts relevant to the particular
content of the model (Glynn, 1982; Barr, 2003; Alford & Romer, 2007). However, advances in the
field of historical understanding had yielded conceptualizations of core competencies in historical
understanding that had not yet been systematically tested in empirical research on the program.
These include the capacity to recognize plausible explanations for historical questions by analyzing
agency, evidence, and causation in the past.
A similar review of current scholarship—specifically conceptualizations of components of teacher
effectiveness in the humanities, strategies for promoting effectiveness, and the alignment of these
conceptualizations with Facing History’s theory of change—was needed in order to design new
research that would document program outcomes meaningful to the larger academic, practitioner,
and policy communities.
12THE NATIONAL PROFESSIONAL DEVELOPMENT AND
EVALUATION PROJECT
The Richard and Susan Smith Family Foundation has provided support for educational initiatives,
including Facing History, for many years. In 2005, Facing History was in conversation with the
foundation around Facing History’s strategic plan, including its long-term research agenda, to address
the evolving landscape of evaluation and Facing History’s priorities. The conversation was informed
by the review of past research, current standards of evidence in evaluation, and an independent
analysis of the key issues of concern to educational administrators that they believe educational
reforms should address, including increasing the quality of instruction and teacher retention (The
Parthenon Group, 2003).
The Richard and Susan Smith Family Foundation invited Facing History to submit a proposal for
research that would significantly bolster the body of evaluation evidence for Facing History’s model,
make a significant contribution to knowledge about effective civic education practices, and provide
key capacity-building benefits for the organization. The foundation’s priorities aligned with Facing
History’s interest in conducting the next generation of rigorous research that would meet the highest
standards of evidence and would document long-term effects of the program. Dennis Barr, Director
of Evaluation, with input from Facing History management, 11 crafted a proposal that presented a
long-term research agenda for the foundation to review. The foundation generously provided five
years of support for the project.
The project, which came to be called the National Professional Development and Evaluation Project,
has three major phases:
I. Planning, research design, and recruitment of independent researchers to work with Facing
History staff;
II. Implementation of outcome research, including experimental and exploratory research with
longitudinal components; and
III. Continuing longitudinal research, secondary data analysis, and dissemination of findings,
implications for theory and practice, and continuing questions.
In the first phase, Facing History formed the infrastructure for the project and identified the
particular methods and measures that would be used. Facing History had to grapple with a wide
range of questions before launching a study. What design would satisfy the highest standards of
evaluation evidence? How could longitudinal research be integrated within such a design? If an
experimental design were chosen, how could schools and teachers be randomly assigned to
intervention and control groups without dramatically altering Facing History’s traditional
professional development and support model? How could a single study best integrate research on
both teacher and student outcomes? How large a scope should the project have, in terms of the
samples of schools, teachers, and students, in order to identify meaningful effects of the program and
position the research to follow teachers and students longitudinally? Would Facing History have the
capacity to work with that number of newly recruited educators on top of its previous commitments?
And what outcome variables in each of the discrete fields would be most important to examine, both
to advance Facing History’s understanding of its impact and to strengthen knowledge in these fields
(Selman et al., 2007b)?
11Facing History management team participating in the research: Terry Tollefson, Marc Skvirsky, Marty Sleeper,
and Margot Strom. Facing History program staff who managed the recruitment of schools and teachers for the
study included Dan Alba, Fran Colletti, Dunreith Kelly Lowenstein, Peter Nelson, Bonnie Oberman, Molly Schen,
Ted Scott, Rachel Shankman, Fran Sterling, Mark Swaim‐Fox, Corey Todaro, and Jack Weinstein. Facing History
evaluation team members participating in the research: Abbey Mann and Anna Romer.
13Given Facing History’s multifaceted and complex model, what “treatment” would be defined as the
model for the research to test? Facing History has a variety of approaches, including working with
individual teachers, whole departments of educators, and whole schools and districts, as well as
online and face-to-face methods of providing professional development and resources. Which of
these approaches should be the focus of the research? Further, Facing History uses a number of
different historical case studies. What content should the teachers receive? Teachers usually adapt
Facing History to their educational context and use as much of the content and methods as they can
or are allowed to within that context. What would we define as the minimum “dosage” of the
program from which we would expect teachers to be able to have an impact on students in
measurable ways?
Facing History formed a Steering Committee and consulted with field experts regarding evaluation
methodology and measurement 12 to address these questions and make some choices. Facing
History’s Director of Evaluation would serve as the lead investigator, responsible for spearheading
the research and coordinating and integrating the contributions of participating researchers. Facing
History established a three-component research team comprised of its own evaluation staff and
researchers from a research agency and several universities. After a careful review, Facing History
selected Abt Associates, Inc., a renowned international research organization, to participate in the
design of the research and to be fully responsible for the data collection and analysis. This would
ensure the objectivity of the research.
To ensure the relevance, value, and validity of the research for Facing History, and the fields in which
the program and research are grounded, Facing History contracted with university-based researchers
Robert Selman, Melinda Fine, and Ethan Lowenstein to serve as co-investigators. The co-
investigators would participate with Dennis Barr and Abt Associates, Inc., in designing the study,
provide leadership in identifying and/or developing appropriate measurement strategies, 13 and work
together to theoretically align the teacher and student measures. The lead and co-investigators
spearheaded measurement working groups focused on the assessment of teacher growth, 14 civic
learning, 15 social and ethical awareness, 16 and historical understanding. 17 Figure 2 provides a view of
the innovative partnership model involving the three entities: Facing History, the research agency,
and the university-based scholars, with Facing History at the center, directing the overall project and
coordinating and integrating the work of its partners.
12 The Steering Committee included Dennis Barr, Facing History Director of Evaluation; Robert Selman, Harvard
Graduate School of Education; and Terry Tollefson, Facing History Director of Administration. Consultations
were held with: Larry Aber, William Beardslee, Joshua Brown, Chris Dede, Constance Flanagan, Carol Gilligan,
Helen Haste, Stephanie Jones, Joe McDonald, Roger Weissberg, and John Willett.
13 Each of these researchers had experience conducting research on the program that would inform the new
strategies for measurement in the NPDEP.
14 Ethan Lowenstein, Eastern Michigan University
15 Melinda Fine, Fine Consulting and New York University, and Angela Bermudez, formerly a graduate student
research assistant on the project at the Harvard Graduate School of Education and currently a faculty member at
Northeastern University
16 Robert Selman and Dennis Barr
17 Dennis Barr invited Alan Stoskopf, formerly a Facing History staff member and currently a faculty member at
Northeastern University, and Angela Bermudez to provide leadership in developing an historical understanding
assessment tool.
14UNDERTAKING A RANDOMIZED CONTROL EXPERIMENT
Randomized controlled experiments offer the benefit of establishing causal links between an
intervention and its outcomes. Policymakers, foundations, school administrators, and scholars are
calling for this kind of evidence, without which, many argue, we cannot really know “what works.”
These kinds of studies, however, can be very difficult to implement. Further, few educational
interventions have been shown to “work” using these methods.
Further, few educational interventions have been shown to “work” using
these methods.
One set of challenges that such studies present relates to the recruitment of subjects. Experimental
studies generally require large samples, large enough to give the study the statistical “power” needed
to detect program effects. In this study, schools needed to be willing to be randomly assigned to
treatment and control groups, which meant the principals would needed to recruit or assign teachers
to participate in Facing History’s professional development, while also being willing to wait for their
teachers to receive it if their school were assigned to the control group. Would we be able to recruit
enough schools and teachers to participate in the project under these conditions?
Facing History was willing to withhold its intervention from schools for a year, but it was neither
practical nor reasonable to withhold it for longer than that. For many possible personal and
professional reasons, teachers might not be able to wait longer than a year to participate. From an
ethical and strategic point of view, it would not make sense to spark principals’ and teachers’ interest
in Facing History and then withhold the program from some of them for several years. Therefore,
the research teams decided the control group would receive the treatment after one year. This
15decision posed a challenge for the research. Facing History assumes that teachers need time, often
years, to develop the knowledge, skills, and confidence necessary to implement the program deeply
and well. In the first year of using Facing History, for example, teachers must deal for the first time
with such issues as balance (fitting the new methods and content into an already busy and demanding
set of curricular requirements, standards, and testing), pacing (i.e., how to cover all aspects of the
scope and sequence within the time they have allotted), and comfort/confidence (i.e., using new
content and methods that are likely to engage students personally and emotionally). Therefore, we
could not be certain that first-year Facing History teachers would implement the high quality, in-
depth courses that would be necessary to impact students in ways that our measures would be
sensitive enough to capture.
Further, the research design would require recruiting teachers who had never been exposed to Facing
History professional development and resources, who were not seeking such exposure, and who
teach in schools that had not already been exposed to Facing History (i.e., none of the other teachers
had participated in a Facing History professional development workshop or seminar or taught Facing
History). It was necessary to recruit teachers and schools that had not been exposed to Facing
History to rule out the possibility that such previous exposure would have already affected teachers
and students, thus making it impossible to infer whether the intervention being studied within the
experimental study was responsible for such outcomes or not.
The design would require that we recruit teachers who were not already seeking Facing History to
answer questions related to the scalability of the program. More evidence was needed regarding the
impact of the program on a general population of teachers who know very little about it to begin
with and may not be highly motivated at the outset to engage with it. The preponderance of research
on the program has demonstrated its effectiveness for teachers who self-select to participate. These
teachers, however, may be different from a wider population of teachers; at the very least they are
motivated to learn about and participate in this educational approach. To include teachers in the
study who had self-selected to participate in the program might limit the relevance of the results for
those administrators who must decide whether Facing History would be effective for a whole school,
district, or region rather than just individual, self-selected teachers.
Although it would be ideal for the reasons just described to recruit teachers who were not already
seeking Facing History, doing so is significantly more challenging. Could we find sufficient numbers
of schools and teachers in the regions where Facing History has offices that had not already been
exposed to the program and were willing to participate? Would the teachers whom we would recruit
have sufficient motivation to learn the content and methods of the program and implement it fully?
The research team decided that the benefits of what could be learned from this kind of sample
outweighed the challenges that would have to be faced. 18
Another set of challenges had to do with defining the “intervention” and the focal outcomes for the research.
Facing History provides a set of guiding principles, methods, and content but is not highly
prescriptive. Rather, teachers select teaching materials from a range of possibilities and cover the
scope and sequence of the program with varying degrees of depth, with different emphases, and over
varying amounts of time. Often, programs that undergo experimental evaluation are more
prescriptive and structured than Facing History, with a program expert using a specific set of lessons
to teach a discrete skill or piece of knowledge directly to students. It is considerably more challenging
to design an outcome study, much less one with an experimental design, to test the effectiveness of a
18 Facing History program staff in all eight of its regional offices—New England, New York, Memphis/Nashville,
Cleveland, Chicago, Denver, San Francisco Bay Area, and Los Angeles—helped Abt Associates, Inc., to recruit
schools for the study. As the program staff are experts in describing what Facing History entails, and the
researchers are expert in explaining the research, their combined efforts were needed.
16method and content that are flexibly used by teachers. Similarly, it is considerably easier to measure
whether students can demonstrate a discrete skill or learn a piece of factual knowledge than it is to
assess whether students develop awareness, critical thinking, and dispositions within the three
domains of focus in this study—civic learning, social and ethical awareness, and historical
understanding. This would be especially challenging, as we will see, in those areas where no measures
yet existed to capture key outcomes of interest.
Finally, experimental studies of interventions in the real world of schools, rather than in more
controlled conditions, are messy, and the stakes for the program’s developers are high. Many studies
of this type have revealed no effects of interventions, and there was a real possibility that this
research would be no different. It is not surprising, therefore, that few educational programs have
taken the plunge to participate in large-scale, randomized controlled experiments. However,
recognizing the dearth and importance of this type of evidence of effectiveness for policymakers,
scholars, practitioners, and funders, and the fact that Facing History had already demonstrated its
impact using other methods, Facing History committed to carrying out a comprehensive
experimental study with longitudinal components.
IMPLEMENTING THE OUTCOME RESEARCH
During the implementation of the experimental outcome research between 2007 and 2009, Abt
Associates, Inc., conducted the data collection and data analysis, maintaining strict independence
from Facing History and the university-based researchers (see Figure 3). This ensured the objectivity
of the research. Abt Associates, Inc., handled all communication with schools and subjects and
followed strict guidelines for research with human subjects, including obtaining appropriate informed
consent for participation in the research from districts, principals, teachers, parents, and students.
Student and teacher surveys were returned by teachers directly to Abt Associates, Inc., and were
unavailable to Facing History and university-based researchers until Abt Associates, Inc., completed
the analysis of all rating questions and finished their reports on the findings.
17RESEARCH DESIGN
The core of the NPDEP is a randomized controlled trial to confirm hypotheses about Facing
History’s impact on both teachers and students. The theory of change guiding this research, without
listing specific outcomes targeted for teachers and students, is depicted in Figure 4.
As mentioned previously, the experimental design involved the random assignment of schools, and
therefore teachers and students, to treatment and control groups. This design allowed us to identify
causal connections between the Facing History intervention and teacher and student outcomes.
The research team also carried out exploratory research that can provide additional information
about program outcomes and generate new hypotheses for further research. Some of the exploratory
research was carried out within the experimental design, and some was not. The exploratory research
is focusing on 1) analyzing civic learning outcomes from scales that were included in the overall civic
learning instrument but that were not included, a priori, in the hypothesis testing; 2) comparing the
impact of Facing History on students when they are taught by teachers teaching it for the first versus
the second time; and 3) analyses of variation in Facing History’s impact when it is implemented in
different settings (types of schools, region of the country) by different groups of teachers (subject
taught, years of experience) and with different students (gender, grade level).
Longitudinal research tracking the program’s impact on students will take place in Phase III of the
project. The team will track long-term outcomes for students who participated in a full Facing
18You can also read
