Are school boards receptive to information: a field experiment - OSF
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Are school boards receptive to information: a
field experiment ∗
Pre Analysis Plan
Angele Delevoye†
August 28, 2020
Abstract
This project explores the causal effect of providing information to school boards
on their deliberation and decision-making. The lack of resources is often cited as a
key barrier preventing school boards from gathering information and evidence, but we
do not know if school boards would be receptive to information in the counterfactual
world in which they receive it. School boards also provide us with a large diversity
in contexts and characteristics (size, resources, composition and mode of selection of
school board members, partisanship of the county), allowing us to identify heteroge-
neous treatment effects across different institutional settings and contexts. The project
first descriptively explores what the baseline use of information and evidence in school
board settings is, across the different contexts mentioned above and compared to other
institutional settings (congressional floor debates and hearings). The experimental por-
tion of the project consists in measuring the causal effect of receiving a policy memo
on school transportation in a COVID world. 300 school districts were treated, which
means 2000 school board members and 1000 school administrators received the memo.
The timeframe of the project (January 2020 - fall 2020) provides me with pre-and
post- COVID data and allows me to follow 600 school boards as they go through an
extraordinary year. This pre-analysis-plan focuses on the experimental portion of the
project.
∗
IRB protocol ID: 2000028526
†
PhD Student in the Department of Political Science, Yale University. angele.delevoye@yale.edu
11 Introduction and literature
A common wisdom in political science is that evidence and information do not (and some-
times even should not 1 ) play a significant role in policy-making. Politics are about power
and conflict between groups. Politicians’ decisions and preferences are shaped by many pow-
erful incentives (their own preferences, party ID, lobbies, reelection, career advancement).
At best, evidence is very low on that list and is therefore not worth studying. Citizens are
not equipped to deal with the minutia of policy-making, and their ignorance is perfectly
rational. Bureaucrats are generally thought to be better at processing evidence (Workman
2015). I argue that we need to go beyond this common wisdom and more rigorously assess
and understand the exact institutional settings that make evidence processing possible. It
might be that political scientists have been too pessimistic in their assessments. Or they
might have been right - in which case we should be looking at different places and for new in-
stitutional frameworks. Some research already points toward a more subtle and fine-grained
understanding of these questions. For example, we have evidence that bureaucrats also base
their decisions on heuristics and existing values (Majone 1989) and that politicians might
be more open and responsive to evidence than previously thought (Newman, Cherney, &
Head 2016). Some of this evidence is experimental: Doberstein (2017) tested source effects
on local bureaucrats from British Columbia to show that bureaucrats rely on heuristics to
assess evidence, giving academic research more credibility than to think tank or advocacy
organization research, regardless of its content. Hjort, Moreira, Rao, and Santini (2019)
show how mayors in Brazil are willing to pay to get access to impact evaluations and up-
date their beliefs, and that informing them about research on a simple and effective policy
1
Theorists have warned us about the risks of encroachments of technical reason into the public sphere and
improper use of instrumental reasoning to decide issues that properly invoke a greater public good (Goodnight
1982). I do not enter this debate, but I am aware of the risks of relying exclusively on evidence outside of
democratic and political frameworks. I believe that thinking of how evidence is and can be processed within
democratic institutions address these concerns and answers Fung, Russon Gilman, and Shkabatur (2013)’s
call to be more attentive to the institutional dynamics of political systems when we think about the use of
evidence).
2(reminder letters for taxpayers) increases the probability that their municipality implements
the policy by 10 percentage points. Nellis et al. (2019) showed expert policy practitioners 2 in
Washington DC studies of different qualities (meta-analysis of RCTs, one experimental, one
observational) and measured beliefs and predictions on another, unseen study. Their results
make them ’mostly sanguine about the capacity and willingness of policy practitioners to
absorb new evidence, although there are important caveats’. They show that most partici-
pants update their beliefs, but only for a short period of time. A majority of subjects at the
end of the experiment continued to retain beliefs about the efficacy of the intervention, con-
tradicting the evidence presented to them. They also show little improvement on forecasting
accuracy. Finally, research in Citizen Assemblies show evidence that participants’ knowledge
and sophistication increase during deliberative experiences (Abelson et al. n.d.; Carman et
al. 2015,?; Knobloch, Gastil, Reedy, & Cramer Walsh 2013; Luskin, Fishkin, & Jowell 2002).
I am not arguing that citizens are better able to process information and evidence than
bureaucrats. But reality might be more nuanced than our current understanding.
School boards are an ideal setting to explore these questions, for multiple reasons. First
and most importantly, school boards and the policies they make and implement are crucial in
the life of a lot of Americans. Their exacts powers and responsibilities vary across states and
cities, but all of the 15,000 school boards in the country routinely have to make important
budgets, tax, curriculum, or infrastructure decisions. Second, school boards offer a large
variation in the profile of the policy-makers, their responsibilities and the rules under which
they operate, offering ’10,000 laboratories for democracy’ (Berkman 2005). Asen (2015) calls
school board members ’citizens-policy makers’, emphasizing how they constitute a middle
ground between normal citizens and professionalized policy-makers - with large variations
based on the size of the districts and the mode of selection. Significant variations exist in
school districts’ legal status, funding models, responsibilities and sizes. Some districts in
2
Including employees of international organizations, the federal government, think tanks, and various
not-for-profit groups. Many came from the Monitoring and Evaluation teams within their respective orga-
nizations, as the experiment was conducted during an ’Evidence summit’ in DC
3NYC have more than 1 million students enrolled, while some rural districts are responsible
for fewer than 200 students. Large urban districts often have larger boards, reflect a broader
range of interests, are organizationally complex, have central office staff, and often function
like corporations. They tend to be more political (Hess 2002). Small, often rural districts
are more connected to the nuances of community cultures and power structures, lack ad-
ministrative staff (Petersen & Fusarelli 2006). More practically, school board members are
one of the largest group of policy-makers in the US (even the smallest boards have at least
4 members), providing us with a very large subject pool.
Finally, education policy and school boards are closely tied to the Progressivist ideals of
efficiency values, faith in bureaucratically structured and scientifically managed organizations
(Asen 2015). In order to reach these goals, school boards sought to centralize control and
created large, comprehensive school systems and adopted new bureaucratic structures. From
1930 to 1970, the number of districts went from 130,000 to roughly 16,000. The consequences
of consolidation on students and district performance have been widely studied, with mixed
results (Hess & Leal 2005). Two more recent trends carry the legacy of the Progressivist
movement for school boards: the formalization of state and federal accountability systems
and the mayoral take-over of schools. Recent state and federal legislations (No Child Left
Behind 2002, Every Student Succeeds Act 2015) have conditioned funding on school results
and performances, and have developed the evidentiary tools to measure them. Over the past
20 years, mayors have taken over the governance of schools in almost 20 major urban school
systems, including New York, New Haven CT, Chicago, Philadelphia, Baltimore, Hartford
CT, Boston, Providence RI, Cleveland, Indianapolis (Wong & Shen 2013). Progressivist
arguments are put forward by the reformers behind this movement: according to them,
local school boards protect status quo, are captive to unions and special interests, and
are not equipped to deal with evidence, and localism reinforce inequalities. The trade-off
associated with this trend opposes localism and democratic education on one side, technical
skills, evidence based and professionalism on the other. The little evidence that exists on
4the consequences of these mayoral takeovers is mixed. Early descriptive results suggest
positive outcomes on student performance and administrative capacity, but not on resources
or staffing (Wong & Shen 2013). Hoover (2008) finds no differences in student performance
between elected and appointed school board members in Alabama (he uses an IV model on
panel data in which a dummy variable for whether an official is elected is instrumented with
the number of meetings and locations of meetings).
One of the only (to my knowledge) empirical exploration of how information and evidence
are used in school boards settings comes from Asen, Gurke, Conners, Solomon, and Gumm
(2013), who conducted observations (160 school boards meetings) and interviews in 3 school
districts in Wisconsin. They find that superintendents and other administrators reference
research and evidence nearly twice as much as board members. They justify this gap by
the fact that board members do not work full time for the district and lack resources to
access and gather evidence and research. They also finds that advocates whose background
taught them to use research as part of a decision-making process referenced research more
often PhDs, master’s degrees), and that their delivery styles (low, loud voice, commanding
presence, cooperative environment in which people could state their views without risk of
embarrassment, pairing research evidence with examples and testimony) mattered a lot to
how research was received. The authors also introduce an important distinction between two
ways in which evidence was used: general (1 or 2 sentence reference to ’research’, ’studies’,
’literature’, sometimes with an identifying marker, such as a ’University of Texas study’,
and a description of the general finding) or specific (abstracts of single or multiple studies
cited, with some references to the researcher, the population studied, the research design
and conclusions) references.
2 Research question
This field experiment explores the causal effect of providing information to school boards
on the decision making and deliberation in the district. How receptive are school boards
5to new information, and to what extent does the degree to which boards engage with the
information depend on some key characteristics of their districts, such as size, resources,
partisan orientation, composition and mode of selection? The lack of resources and time
is often cited as the reason why school boards do not engage with research and evidence
more 3 . By providing them with this information, I am hoping to shade light on whether
the lack of resources and access is the reason why they do not engage with research - as
opposed to other reasons, such as a lack of confidence in evidence or an unwillingness to
use it. Finally, I will leverage the diversity in school boards’ contexts and compositions to
identify institutional settings that are more conducive to information processing.
3 Experimental sample and treatment assignment
I built a list of 595 eligible districts. A district was deemed eligible if I could find recordings
of its school board meetings published reliably and frequently. I worked with the Youtube
API, complemented by manual searches and checks, in order to build this list. As shown in
Figure 1, the districts in my sample are globally representative of all districts in the country
on most dimensions (financial resources, percentage of white students, pupil-teacher ratio,
expenditures and spending per student), even though the populations they serve tend to be
slightly richer and to vote more Democratic. The districts in my sample also tend to on the
larger side of the distribution. I still catch districts of all sizes (very small districts are defined
as serving less than 271 students, the small ones between 271 and 2420 students), but the
sample distribution is not representative of the population distribution on this dimension.
3
the baseline level of engagement with evidence from schools boards, both in absolute value and compared
to other institutional settings, is unknown. I will look at the question in a different project
6Figure 1: Distribution of key covariates - comparison between sample and pop-
ulation
1.5
2.0
0.10
1.5 1.0
Density
Density
Density
1.0
0.05
0.5
0.5
0.0 0.0 0.00
0.00 0.25 0.50 0.75 1.00 0.00 0.25 0.50 0.75 1.00 0 10 20 30 40
Perc. white students Perc. free lunch eligible students Pupil−Teacher ratio
1.0e−04 1.0e−04
1.5
7.5e−05 7.5e−05
1.0
Density
Density
Density
5.0e−05 5.0e−05
0.5
2.5e−05 2.5e−05
0.0e+00 0.0e+00 0.0
0 10000 20000 30000 40000 0 10000 20000 30000 40000 10.0 10.5 11.0 11.5 12.0
Revenue per student Expenditures per student SES composite score for district
2.5
0.5
0.4 2.0
1.5
Frequence
0.3
Density
0.2 1.0
0.1 0.5
0.0 0.0
Very small Small Medium Large Very Large 0.00 0.25 0.50 0.75 1.00
Size of district 2016 Dem margin in county
All Districts YT Sample
In order to gather covariates on each district, I merged my list of eligible districts
with existing databases of school districts: the National Center for Education Statistics
(NCES)’s database provide variables such as the size, ressources, geographic characteristics
(urban/rural) address, URL of each district. A separate NCES’s dataset provides detailed
financial information on each district. The Stanford Education Data Archive (SEDA) has
7data on the socio-demographics characteristics of each district: percentage of single mom
households, unemployed people, people living under the poverty rate, distribution of incomes.
All variables are broken down by race, and the dataset also has composite variables for the
racial diversity and socio-economic status of each district. Finally, I gathered county-level
partisan data from the MIT Election Data Science Lab (results from the 2016 presidential
election).
I then conducted a block random assignment, using the state of the district as the primary
blocking covariate. In cases in which I had at least 8 districts in my dataset, I used the size
of the district as the second blocking covariate. I used a binary variable for whether the
district’s size (as measured by the total students served) was above or below the state’s
median as the blocking size variable.
Finally, I collected the email addresses of all school board members and staff for all treated
districts. Because districts websites include many email addresses that I did not want (any-
thing from teachers to janitors or contractors), some districts protect their email addresses
from automatic scraping, and no setting or conditions on an email crawler would allow me
to catch the exact email addresses I wanted, I mostly built this email list manually, with the
help of an email crawler when appropriate. For staff, I limited myself to people with a direct
authority on transportation issues (often the superintendent, director of operations and/or di-
rector of transportation).
The units of analysis for this project are the districts. All of the covariates I currently
have are district-wide variables. I will eventually try to gather information on all school
board members in each district: their gender, race, professional backgrounds and mode of
selection (appointed or elected). This work will have to be conducted manually, with the
help of RAs.
83.1 Treatment: email and attached policy memo
For this experiment to work, I needed a policy topic that every school board in the country
was at least interested in, and ideally working on. The COVID crisis provided me with an
opportunity to reach out with a policy I know every school board in the country was working
on. I chose school transportation over other COVID-related issues because the topic was
narrow and well-defined, which made the memo easier to write and will make the outcomes
easier to measure. I wrote the policy memo myself. It summarizes the main research findings
and official recommendations on the question.
I asked the director of ISPS (Alan Gerber) if I could use ISPS or Yale’s official affiliation
on the policy memo and in the email. I was not allowed to make the memo an official Yale
or ISPS document. But he gave me permission to use my (real) ISPS affiliation, as well as
ISPS’s mission to bridge the gap between researchers and policy-makers. I took all logos or
distinctive signs off the memo.
I sent the following email through MailChimp. I sent different emails to school boards
members and staff members. The email contains two links, one to the memo, and one to
a Qualtrics form allowing people to give me feedback and express an interest for additional
memos.
Dear school board member [School Administrator],
My name is Angele Delevoye. I am a researcher at Yale University’s Institution for
Social and Policy Studies (ISPS). You are receiving this email because you are a school
board member [School Administrator with authority over transportation issues].
Many school boards are facing unprecedented challenges as they prepare for the upcom-
ing academic year, with many issues to deal with, significant stakes and little time. As
part of ISPS’s mission to bridge the gap between researchers and policy-makers, we have
9prepared a policy memo on a crucial policy question currently facing school boards: school
transportation in a COVID world. The memo is available at this Google Drive link.
The memo provides a summary of the main research findings and official recommenda-
tions on school transportation amidst COVID. It is not comprehensive and is intended to
complement other available resources. Our goal is to provide school boards with policy-
relevant information that can inform decision-making as end of summer vacation nears.
If you are interested in receiving more memos on different topics, or want to drop us a
comment, you can fill out this quick form. We would welcome any suggestion on how we can
be helpful!
Please do not hesitate to email me with any comments or questions.
Best,
4 Outcomes
I am pre-registering the following outcomes. The basic analysis will be a comparison between
the treatment and control groups on each of these outcomes. I will also look at the following
heterogeneous treatment effects:
• Urban versus rural districts
• Size of district
• Financial resources of district (as measured by expenditures by students)
• Partisanship of county: districts in democratic counties versus districts in republican
counties
• Percentage of women on the school board (or any other relevant biographical informa-
tion, such as percentage of school board members with an advanced degree)
• Elected versus appointed versus mixed school boards
104.1 First set of outcomes: email statistics and qualitative responses
MailChimp provides me with the following statistics: percentage of emails opened (and the
specific email addresses that opened the email), percentage of people who clicked on links
in the email. For each district, I will gather the total number of emails that were sent and
did not bounce back, the number of emails that were opened, and the number of clicks on
the memo and on the form. I will compute separate counts for board members and staff
members.
I ask if people to leave comments on the Qualtrics form, and I invite them to get back to
me through emails. I do not expect enough responses or comments to conduct a quantitative
analysis, but I will gather those responses as qualitative evidence on how my email and memo
were received.
4.2 Second set of outcomes: content of deliberations and policy documents
For the second set of outcomes (content of the deliberations), I will use recordings of meetings
held before I sent the memo as pre-treatment covariates. I will be able to tell a full story of
what was discussed in 2020, before the COVID crisis, during the crisis but before my memo,
and during the crisis but after my memo, in both treatment and control groups.
4.2.1 Find and download recordings of all meetings
While building the experimental sample, I also collected the unique ID of the YouTube
channel that each district uses to post videos of their meeting. I could theoretically use the
YouTube API and packages such as youtube-dl, pytube, or tuber to:
• find a list of all videos posted on each channel in 2020
• only keep the videos that are over 30 minutes in length
• do a manual cleaning based on the name and description of the video (in order to only
keep school board meetings, or at least videos that are related to school policies)
11• download captions when available from youtube (YouTube’s captions are generated
through unspecified machine learning techniques, and are widely perceived to be as
accurate as Google’s speech-to-text or other cutting-edge algorithms)
• download recordings for videos for which captions are not available
• generate transcriptions from these recordings (which might involve paying for the
Google’s speech-to-text algorithm, which is widely recognized as the most performant)
However, the YouTube API has strict quotas limits, and is not reliable: the effective
quotas are often very different from the official quotas, and they are not stable: some code
that works on one day sometimes does not work another day. Moreover, some channels post
many videos on YouTube: some districts post educational material or videos of sports games
for example. This makes the manual cleaning after an API grab very necessary, but also time
consuming. A much more reliable, but time consuming alternative, is to first go through all
the channels manually and gather the URLs of the videos I want to grab: the school board
meetings or other publicly available working sessions. Once I have the videos URLs, I can
gather meta-information on each video (date published, duration, views, title, description)
through XML scraping methods using the Rvest package in R. I can then download captions
and recordings as specified above. This method provides a way around having to use the
YouTube API.
4.2.2 Analyze the recordings
First, I will compare the frequency at which transportation issues are mentioned during
meetings in treated districts as opposed to districts in the control group. In order to compute
how frequently transportation issues are mentioned in a meeting, I will first build a bag of
words that are related to transportation issues: ’transport*, bus*, rout*, driver*, stop*,
*board*, *load*’. I will then compute the percentage of total words mentioned in a hearing
that belong to this transportation dictionary. Ideally, I would also measure the way in
12which the topic is discussed. After having isolated the passages in the transcripts in which
school transportation is discussed, I would run them through a DQI or any other supervised
text analysis method to see whether many arguments, considerations and trade-offs are
mentioned.
Second, I will look for specific mentions of my memo. The process will be similar to
the one above, but I will look for a different set of words: ’Yale, memo, ISPS’ or very
specific words from the memo, such as ’algorithm, ventilation, germicidal, ultraviolet, uv,
irradiation, EPA, CDC, Boston, Rand, summary, recommendations, head start, plexiglass,
barriers, manufacturers, pediatrics, assign, tape, high touch areas, list, cleaners, disinfectors,
safe* products, MIT, Georgia Tech, Atlanta, Denver, Excel, UNC, UCLA, Los Angeles,
sanitized, webinar’. I will conduct a sentiment analysis around those words, to measure
whether the memo is mentioned in a positive or negative way.
Third, I will look for mentions of research or evidence-based words. I will rely on a dic-
tionary I built for another project, which contains the following words: ’evidence, evidence-
based, research*, data, randomiz*, randomis*, doctor, professor, trade-off, hypo*, study, sci-
ence, scienti*, evaluation, rigor*, experiment*, quantitative, methods, analytic*, methodo*,
measur*, results, expert*, model*, indicator*, analys*, impact, studies, statistic*, empiric*,
percent*’.
Fourth, I could conduct similar word searches on other board documents besides the
transcripts of meetings. School boards publish agendas, minutes and policy documents on
their websites. I could scrape all the documents and conduct similar word searches (frequency
of transportation issues in the document, instances of mentions of specific words from the
memo).
134.3 Third set of outcomes: survey of board members in control and treatment
groups
I will send a seemingly unrelated survey through a non-affiliated partner to all the school
board members in my treatment and control groups. I ask questions such as:
• how big of a role are the inputs, recommendations and decisions of the following ac-
tors playing in their district’s decision? (the list of actors will include state and city
governments, federal government, teachers’ unions, parents, other school districts and
the scientific community
• how much time has their school board spent discussing the following topics since the
beginning of the COVID crisis: topics will include meal provision, internet access,
achievement gap, online learning tools, staff and teachers’ safety, cleaning protocols,
athletics, English learners and transportation
• how much help has their school district received from the following sources: state and
local governments, federal government, professional organizations and unions, aca-
demics and researchers, non-profits
• how much help do they wish they had received from the same sources
14References
Abelson, J., Eyles, J., McLeod, C. B., Collins, P., McMullan, C., & Forest, P.-G.
(n.d.). Does deliberation make a difference? results from a citizens panel study of
health goals priority setting. , 66 (1), 95–106. Retrieved 2019-08-22, from http://
www.sciencedirect.com/science/article/pii/S0168851003000484 doi: 10.1016/
S0168-8510(03)00048-4
Asen, R. (2015). Democracy, deliberation, and education (Vol. 13). Penn State Press.
Asen, R., Gurke, D., Conners, P., Solomon, R., & Gumm, E. (2013). Research evidence and
school board deliberations: Lessons from three wisconsin school districts. Educational
Policy, 27 (1), 33–63.
Berkman, M. B. (2005). Ten thousand democracies: Politics and public opinion in america’s
school districts. Georgetown University Press.
Carman, K. L., Mallery, C., Maurer, M., Wang, G., Garfinkel, S., Yang, M., . . . others
(2015). Effectiveness of public deliberation methods for gathering input on issues in
healthcare: results from a randomized trial. Social Science & Medicine, 133 , 11–20.
Doberstein, C. (2017). Whom do bureaucrats believe? a randomized controlled experiment
testing perceptions of credibility of policy research. Policy Studies Journal , 45 (2),
384–405.
Fung, A., Russon Gilman, H., & Shkabatur, J. (2013). Six models for the internet+ politics.
International Studies Review , 15 (1), 30–47.
Goodnight, G. T. (1982). The personal, technical, and public spheres of argument: A
speculative inquiry into the art of public deliberation. The Journal of the American
Forensic Association, 18 (4), 214–227.
Hess, F. M. (2002). School boards at the dawn of the 21st century: Conditions and challenges
of district governance.
Hess, F. M., & Leal, D. L. (2005). School house politics: Expenditures, interests, and com-
15petition in school board elections. Besieged: School boards and the future of education
politics, 249 .
Hjort, J., Moreira, D., Rao, G., & Santini, J. F. (2019). How research affects policy: Exper-
imental evidence from 2,150 brazilian municipalities (Tech. Rep.). National Bureau of
Economic Research.
Hoover, G. A. (2008). Elected versus appointed school district officials: Is there a difference
in student outcomes? Public Finance Review , 36 (5), 635–647.
Knobloch, K. R., Gastil, J., Reedy, J., & Cramer Walsh, K. (2013). Did they deliberate? ap-
plying an evaluative model of democratic deliberation to the oregon citizens’ initiative
review. Journal of Applied Communication Research, 41 (2), 105–125.
Luskin, R. C., Fishkin, J. S., & Jowell, R. (2002). Considered opinions: Deliberative polling
in britain. British Journal of Political Science, 32 (3), 455–487.
Majone, G. (1989). Evidence, argument, and persuasion in the policy process. Yale University
Press.
Nellis, G., Dunning, T., Grossman, G., Humphreys, M., Hyde, S. D., & Reardon, C. (2019).
Cumulative evidence, beliefs, and out-of-sample prediction accuracy: A results dissem-
ination experiment with practitioners (Tech. Rep.). Working paper.
Newman, J., Cherney, A., & Head, B. W. (2016). Do policy makers use academic research?
reexamining the âtwo communitiesâ theory of research utilization. Public Administra-
tion Review , 76 (1), 24–32.
Petersen, G. J., & Fusarelli, L. (2006). The politics of leadership: Superintendents and
school boards in changing times. educational policy in the 21st century: Opportunities,
challenges and solutions. IAP-Information Age Publishing, Inc..
Wong, K. K., & Shen, F. X. (2013). Mayoral governance and student achievement: How
mayor-led districts are improving school and student performance. Center for American
Progress.
Workman, S. (2015). The dynamics of bureaucracy in the us government: How congress and
16federal agencies process information and solve problems. Cambridge University Press.
17You can also read
