Improving Middle School Student Engagement Through Career-Relevant Instruction in the Core Curriculum
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
The Journal of Educational Research, 106:27–38, 2013
Copyright
C Taylor & Francis Group, LLC
ISSN: 0022-0671 print / 1940-0675 online
DOI:10.1080/00220671.2012.658454
Improving Middle School Student
Engagement Through Career-Relevant
Instruction in the Core Curriculum
DENNIS K. ORTHNER PATRICK AKOS
University of North Carolina at Chapel Hill RODERICK A. ROSE
University of North Carolina at Chapel Hill
HINCKLEY JONES-SANPEI
Brigham Young University
to be responsive to the social and learning environment
ABSTRACT. The authors assessed the effect of career-
relevant instruction on school valuing and engagement of of the classroom and the school (Fredricks, Blumenfeld, &
middle school students in a southern U.S. school district. Paris, 2004), making it potentially malleable to pedagogical
Previous research and theory indicate students learn best reforms. Thus, school engagement is an early and predic-
when new knowledge is provided within the context of in- tive indicator of how much attention the student is giving
formation students consider to be of value. The data come to academic work that may be influenced by reforms at the
from a school-based randomized trial of the CareerStart in-
tervention that was introduced in 7 of 14 middle schools, and classroom and school level.
include the initial 3 years of data for 3,493 students. The au- One of the many teaching strategies highlighted in school
thors examined the effect of the CareerStart intervention and reform efforts involves augmenting the relevance of the
student-reported career-relevant instruction on psychosocial curriculum so students can establish a link between the
measures of school engagement and school valuing. After content they are learning and either their environment or
controlling for previous school engagement, demographic, so-
cioeconomic, and academic factors, the analysis confirms that their expectations for their future (Orthner, 2007). In the
students in the treatment schools reported significantly higher present study we evaluated the effectiveness of one such ap-
levels of school valuing than students in the control schools, proach, a teacher-focused, school capacity–building strategy
and students reporting greater career-relevant instruction in- called CareerStart. This was implemented by teachers in
dicated significantly higher levels of school engagement and middle school core subject courses including mathematics,
valuing.
language arts, social studies, and science. The purpose of
Keywords: career orientation, intervention, middle schools, this research was to determine the extent to which the Ca-
student engagement, teacher instruction reerStart treatment and career-relevant instruction (CRI)
influenced middle school students’ psychosocial school
engagement.
F or students to perform well in school, they must
believe that their focus on education will pay divi-
dends for them now or in their future. This focus or
school engagement has important consequences. Research
School Engagement
Presently, significant attention is given to promoting
has confirmed that students who are engaged in their edu- school engagement to improve student attention and in-
cation, consider school as a valuable experience, and want crease student retention. Although many of these efforts
to participate in school activities are more likely to demon- are targeted at the high school level under the rubric of
strate high academic achievement (e.g., Klem & Connell, dropout prevention, outcomes including school failure and
2004; Wang & Holcombe, 2010) and are less likely to drop dropping out of school can be viewed as the culmination
out of school (Finn & Rock, 1997). In particular, recent re- of a long-term process of disengagement that begins before
search has found that middle school performance and school high school (Alexander, Entwisle, & Horsey, 1997), and of-
engagement were critical interim school outcomes and im- ten accelerates during the middle school years. Longitudinal
portant predictors of high school graduation (Blafanz, Fox, research has indicated that many students deemed as being
Bridgeland, & McNaught, 2009). Students who consider
their education as relevant and preparing them to achieve
Address correspondence to Dennis K. Orthner, Schools of Social Work
future goals are also more likely to perform well in school and and Education, University of North Carolina at Chapel Hill, Chapel
graduate (Perry, 2008). Student engagement has been shown Hill, NC 27599-3550, USA. (E-mail: orthner@unc.edu)28 The Journal of Educational Research
vulnerable to leaving school early were already disengaged where I will work when I’m grown up” (p. 228), indicating
from their education before high school; thus, these students a readiness for career exploration.
begin dropping out before innovative high school programs Research has indicated that student career exploration
have an opportunity to make a difference (Orthner, Cook, can have a positive effect on school engagement. Studies
Rose, & Randolph, 2002). Research on school engagement of school-to-work programs offered in high schools have
during the middle school years (i.e., beginning of Grade 6 consistently found that students in these programs exhibit
to the end of Grade 8) has consistently found that students’ higher psychosocial school engagement and lower rates of
behavioral and psychosocial engagement declined precipi- dropout than students not participating in career exploration
tously over this period (Orthner et al., 2010; Woolley & (Castellano, Stringfield, & Stone, 2003; Plank, 2001). In a
Bowen, 2007). This pattern of decline suggests that a high longitudinal study of low-income ninth-grade students, re-
proportion of students arrive in high school with diminished searchers reported that students with higher levels of career
expectations for what their education can provide. There- planning sustained higher levels of school valuing and feel-
fore, it is not surprising that only 75.6% of the nation’s ings of belonging to their school (Kenny, Blustein, Hasse,
youth complete high school within the normal 4-year pe- Jackson, & Perry, 2006). Similarly, Perry, Liu, and Pabian
riod, including those that graduate early (U.S. Department (2010) examined the role of career-planning support during
of Education, 2010b). the middle school and high school years on student school
Student psychosocial engagement and valuing, the types engagement and school grades among low-income students.
of engagement reviewed in this investigation, have demon- Perry and colleagues found that students with greater ex-
strated that they are linked to student academic progress posure to career-relevant planning were significantly more
and school success. Studies using measures of emotional likely to be engaged in their education, and in turn, school
and psychosocial engagement have found positive relation- engagement predicted higher grades.
ships to academic test performance (Borman & Overman, Modern expectancy-value theories of learning are based
2004), student grade point averages (Gonzalez & Padilla, on the observation that learning occurs most effectively
1997), and better student attendance at school (Finn & in the context of what the student believes will help him
Rock, 1997; Klem & Connell, 2004). Further, school en- or her achieve personal goals (Eccles et al., 1993). When
gagement research has indicated that students who regard the information given or the tasks assigned are not clearly
their education as relevant and purposive in preparing them connected to what the student perceives as being of value
to achieve future goals are much more likely to perform well to his or her personal goals, short-term learning may oc-
in school and stay in school to graduate (Perry, 2008). En- cur but the information is less likely to be retained or in-
hanced school engagement not only reduces risks of poor tegrated into the student’s cognitive framework (Wigfield
outcomes, such as living in poverty, receiving public as- & Eccles, 2002). An important corollary of this cognitive
sistance, and having poor health outcomes (Blafanz et al., framework is the work on possible-selves theory (Markus &
2009), but also increases the likelihood of students continu- Nurius, 1986). This motivation theory provides some clues
ing to college or making a successful transition into a job or as to why the career- and job-related relevance of teacher-
career (Fredricks et al., 2004). provided instruction may be an important consideration in
promoting student engagement, especially during the for-
Role of Career Relevance and Instruction mative years of early adolescence (Markus & Nurius, 1986).
The possible selves theory proposes that youth are actively
Although researchers have given little attention to ca- engaged in developing concepts of their future selves that
reer development in elementary and middle schools, some can potentially motivate them to learn and behave in ways
research has shown that tentative college plans and career that are consistent with those projections of future selves
preferences begin emerging in elementary school (Trice & (Oyserman, 2008). Because early adolescents are still form-
King, 1991). Students begin to develop identities during ing these projections, accurate cues to future possible selves
their early adolescence, during which time they are influ- may be quite motivating and serve as a schema around which
enced by early school experiences such as identification with adolescents are able to collect and aggregate new informa-
workers, gender stereotypes, race, class, and social valua- tion that applies to future-oriented self-concepts. Research
tion (Gottfredson, 1981; Trice, Hughes, Odom, Woods, & has supported this assumption and demonstrated that stu-
McClellan, 1995) that can influence later career develop- dent test performance, attendance, and retention were as-
ment. Even though middle school students are not typically sociated with crystallization of the student’s possible self
ready to commit to career choices, these students experience (Oyserman, 2008).
the careers of their parents or community and are expected A further stream of research and theory from the litera-
to make choices in their middle and high school curricula ture on school transition provided support not only for the
that can impact future career and educational opportunities attention to engagement, but also the middle school tim-
(Akos, Lambie, Milsom, & Gilbert, 2007; Akos, Shoffner, & ing of the CareerStart intervention. Research has suggested
Ellis, 2007). Research by Orthner et al. (2010) reported that that when students transitioned from elementary to mid-
80% of the sampled students entering Grade 6 “think about dle school, many early adolescents experienced academicThe Journal of Educational Research 29
declines (Alspaugh, 1998) and increased distress (Ander- of their instruction. For example, CareerStart provides
man, Maehr, & Midgley, 1999). Recommendations from short, high-quality, and easy-to-teach lessons that each core
several authors (Akos & Galassi, 2004; Eccles et al., 1993) teacher in Grades 6–8 can use to illustrate priority con-
have included creating an appropriate person–environment cepts that are part of the state-mandated standard course
fit that supports students’ emerging autonomy and increased of study. CareerStart lessons were prepared by experienced
capacity to think about their futures. CareerStart provides teachers who had been recommended by their curriculum
an opportunity for students to explore career possibilities coordinators. The lessons were then peer reviewed, and then
through structured supports offered in the classroom envi- further reviewed by curriculum specialists prior to publish-
ronment. This may not only help to buffer transition de- ing. Ten example lessons were prepared for each of the four
clines, but suggests an optimal time to impact these devel- core courses in each middle school grade. Each of the 10
opmental pathways. lessons was designed for a 1- or 2-day student experience.
CareerStart lessons and support materials are available on-
CareerStart Intervention line and can be easily accessed via the web at LEARN NC
(www.learnnc.org), an instructional website widely used by
CareerStart is a teacher-focused, schoolwide capacity- teachers in the state. Teachers in the targeted schools were
building strategy that attempts to positively influence the asked to use the CareerStart lessons and to continue de-
educational and workforce trajectory for all students, but es- veloping and incorporating career-relevant illustrations in
pecially those at elevated risk for school failure. The overall other topics they teach.
goal of CareerStart has been to develop, implement, and In addition to the lessons, CareerStart teachers received
evaluate a strategy for future career orientation, designed bimonthly updates through CareerStart Dispatch e-mail
to improve middle school student engagement, academic newsletters, which kept teachers informed of new career con-
performance, and career exploration. If these middle school nections to the core content as well as providing examples
goals are achieved, it is expected that students exposed to of the ways other teachers have augmented class instruc-
CareerStart instruction and support will be better prepared tion to include career relevance. Also, at the beginning of
for high school courses and will stay in school to graduate. the academic year, teachers were provided opportunities for
CareerStart helps teachers in core middle school courses training and coaching on how to best develop career con-
(i.e., mathematics, science, language arts, and social studies) nections in other aspects of their teaching. Lead teachers,
illustrate the value of learning state-required course content curriculum coordinators, and school principals also received
by incorporating career examples drawn from industries rep- updates on lessons learned from CareerStart, which in-
resented in the labor markets in which the schools reside. cluded strategies for promoting CareerStart activities in their
Students in classrooms with operating CareerStart princi- schools.
ples should be able to answer the questions asked most often: The CareerStart strategy also involved parents, caregivers,
“Who uses this information in the real world?” or “When will and other school professionals in the instructional approach.
I ever really use this information when I leave school?” Al- One third of the lessons included a parent-engagement ac-
though middle school philosophy (Bishop & Pflaum, 2005) tivity in which a student is asked to interview or to plan
already recommends relevant curriculum, CareerStart builds an activity with their parents or caregivers. These activities
on these efforts by providing contemporary career illustra- typically engaged students and parents in talking about ca-
tions and encouraging wider use of these teaching methods. reer issues or the types of information and skills that adults
The CareerStart strategy mainstreams career exploration use in their own jobs. In addition, other school personnel are
into the overall curriculum and culture of the middle school. often involved in career exploration activities or as support
Teachers are not asked to teach new core content; they use for CareerStart lessons. School librarians were encouraged to
CareerStart lessons to teach required content with illustra- have U.S. Department of Labor or other career information
tive examples from career fields in which this knowledge is resources available to teachers and students. School coun-
used. For example, students learning to calculate volume in selors were available to teachers and students to help in early
mathematics learn the relevance of that knowledge to heat- career exploration and providing information on high school
ing and air conditioning technicians, equipment operators courses, community college programs, or university oppor-
at a utility, and to applications in manufacturing and de- tunities. School social workers could help students identify
sign processes. Language arts and mathematics content are community resources that can be accessed to obtain career
applied in the exploration of business or office management mentoring experiences or to receive guidance in career op-
activities, the development of business plans, or finance and portunities available to youth and adults in the community,
marketing careers. The jobs that are illustrated in the Ca- especially when parents or caregivers are not able to help
reerStart lessons range from those that require only a high make those links.
school diploma and advanced technical training to careers Although students in middle school may have different
that require college or postgraduate degrees. college or career aspirations, it should be noted that Ca-
CareerStart provides critical tools that help teachers im- reerStart does not track students into particular career voca-
plement their instruction in ways that improve the relevance tions or educational trajectories. CareerStart is a universal30 The Journal of Educational Research
intervention that aims to expand all students’ visions of CRI was conducted as an observational, rather than experi-
the future career and educational opportunities available mental study.
to them. Instead of fostering potential inequity, integrat-
ing workforce examples in the core curriculum allows each Sample
student to engage in career exploration and understand the
relevance of coursework at a fraction of the time and finan- The sample included a 3-year longitudinal cohort of 3,649
cial costs of after-school or other targeted programs. students who began Grade 6 in the 2006–2007 school year.
As part of the evaluation of CareerStart, different lev- The sample was 52% male, 47% Asian or Caucasian, and
els of fidelity to the CareerStart program were noted. Some 53% African American, Hispanic, or Native American. Pre-
teachers in the treatment schools did not use the Career- vious research indicates that Asian students are more similar
Start lessons and some teachers in the control schools used to Caucasian students with respect to risk of dropout and
career examples in their classrooms. Also, some teachers school failure than they are to African American, Hispanic,
transferred between treatment and control schools. There- or Native American students (U.S. Department of Educa-
fore, in addition to the treatment and control indicator, we tion, 2010a). Of the total sample, 56% of students received
used a student-reported measure of CRI in the evaluation. either free or reduced-price lunches at any time during the
study period, 39% came from single-parent homes, and 19%
were academically gifted as designated by the district. Ad-
Method
ministrative data on these students were collected annually,
The evaluation of this intervention centers around the and student survey data were collected during the spring of
questions of whether CareerStart and CRI are likely to in- each of the 3 years of middle school (i.e., Grade 6–8). See
crease the psychosocial school engagement of middle school Table 1 for a sample description with a comparison between
students, as measured by school valuing and school engage- students in treatment and control schools. Although there
ment. were no significant differences between treatment and con-
trol schools on gender, poverty, single-parent status, or on
Research Question 1: Do eighth-grade students after 3 years of the two measures of psychosocial engagement, more students
receiving the CareerStart treatment report higher valuing in the treatment schools were African American or Hispanic
and engagement relative to students in schools receiving and academically gifted. Note that these differences did not
standard instruction? take into account nesting within schools. Analysis of base-
Research Question 2: Do eighth-grade students whose teach- line data taking the multilevel data into account found no
ers provided during the three middle school years more significant differences between the treatment and control
CRI report higher valuing and engagement relative to schools for gender, race/ethnicity, single-parent household
students in schools receiving less CRI? status, academic giftedness, or percentage of students on free
or reduced-price lunch. In addition, our analyses found no
Evaluation Design and Data Sources differences between treatment and control schools with re-
spect to standardized test scores in reading and mathematics
Using a stratified randomization procedure, 14 middle (obtained from elementary school) or psychosocial measures
schools in a single North Carolina school district were ran- (administered at the beginning of Grade 6). None of the
domly assigned to either the treatment (n = 7) or the con- normalized differences (i.e., effect sizes) was larger than 0.10
trol condition (n = 7). Of the 14 schools, six were Eq- (Imbens & Wooldridge, 2009).
uity Plus schools, a district categorization based on Title 1
status. Three Equity Plus schools were randomly assigned Measures
through a random number generation procedure to the treat-
ment condition. Similarly, four of the eight non–Equity Plus Dependent variables: School psychosocial engagement outcome
schools were randomly assigned to the treatment condi- measures. Psychosocial student engagement among eighth-
tion. Thus seven schools in the treatment condition, three grade students was measured using two standardized scales:
Equity Plus schools and four non–Equity Plus schools, re- School Valuing and School Engagement. The distribution
ceived the CareerStart intervention, whereas curricula of the of student responses for these outcome measures were highly
seven schools in the control condition included the standard skewed and not distributed normally because the majority
course of study only. Teachers in the control schools did not of the students reported high valuing and high engagement.
receive the lessons, newsletters, and other supports received Therefore, these variables violated the normal distribution
by teachers in the treatment schools; however, teachers in assumption, raising questions about the appropriateness of
the control schools were not prohibited from using career- linear modeling. Further, linear models assume metric invari-
relevant illustrations in their lessons. Therefore, the second ance. For example, the difference on the scale score between
research question on CRI was included because randomizing 1 and 2 is the same as the difference between 4 and 5, which
schools or students to treatment conditions did not ensure is arguably not the case for the outcomes of interest. There-
treatment exclusivity. The portion of the study regarding fore, a logical threshold was established for each outcomeThe Journal of Educational Research 31
TABLE 1. Descriptive Statistics of Student Variables with Significance Tests
Treatment (n = 1,976) Control (n = 1,655)
Variable n % M SD N % M SD χ2 t
Female (m = 0, f = 1) 926 47.5 786 48.1 0.13
∗∗∗
Minority (Asian–Caucasian = 0, 1,122 57.2 780 47.6 33.20
African American–Hispanic = 1)
Poverty (nFRL = 0, FRL = 1) 1,136 57.5 899 54.3 3.67†
Special education (no = 0, yes = 1) 452 22.9 351 21.2 1.45
∗∗∗
Academically gifted (no = 0, yes = 1) 365 18.5 192 11.6 32.74
Single parent (no = 0, yes = 1) 657 34.0 517 32.1 1.42
Valuing, Grade 6a 3.55 0.90 3.50 0.96 –1.57
School engagement, Grade 6a 4.14 0.63 4.14 0.67 0.05
Note. FRL = receiving free or reduced-price lunch; nFRL = not receiving free or reduced-price lunch.
aAfter 1 year of treatment.
†p < .10. ∗ ∗ ∗ p < .001
measure—above and below the mean response—and logistic Independent variables. Two variables were entered as inde-
regressions were used to test for the associations between pendent measures of interest, with each intended to answer
CareerStart treatment, CRI, and the outcome variables, al- one of the research questions. The first was a school-level
though controlling for student demographic characteristics, variable that identified each school’s status in the random
previously measured outcomes, and school characteristics. assignment to the treatment or control condition. The sec-
The seven-item School Valuing subscale was derived from ond variable was a teaching team-level variable. In core
the Student Identification with School measure (Voelkl, courses, CRI was delivered by teachers working in teams
1996). This subscale reflects a student’s belief that school composed of two to four teachers. The second variable was
is important and provides an opportunity to learn useful a student-reported measure of each teaching team’s compli-
information that he or she will find helpful or useful in ance with the treatment assignment, which was represented
the future. The School Valuing scale includes items such as by a level or dosage of CRI. The level of CRI was calculated
“school is important, school and what I learn there is useful, from survey questions administered to students at the end of
school and what I learn there will be useful in getting a each school year regarding the extent to which they agreed
job, dropping out would be a mistake,” and “school is not with the statement, “the teacher often used career exam-
a waste of time.” The mean of the seven items was used ples from jobs and careers” in the classroom instruction. The
for the Valuing scale. If three or more items were missing same question was asked for each of the four core curricu-
then the scale score was missing. The threshold used for the lum classes: language arts, mathematics, science, and social
binary outcome variable was the mean. Overall, 58% of the studies. The students could choose from response options on
students reported at or above average on the Valuing scale. a 5-point Likert-type scale ranging from 1 (strongly disagree)
For the study sample, the scale had a Cronbach’s alpha value to 5 (strongly agree). The modal response for all students
of .79, and a range of 1–5. was “agree,” with 34%–42% (depending on core subject)
The School Success Profile School Engagement subscale of students either agreeing or strongly agreeing that their
(Bowen, Rose, & Bowen, 2005) measures a student’s positive teachers used career and job examples. The CRI measure in-
anticipation of attending school, using three items that as- cluded in the model was the eighth-grade student–reported
sess the extent of the student’s excitement of being in school CRI aggregated at the teacher team level. In addition, the
and looking forward to learning at school. The School En- student-level deviation from the aggregated teacher team
gagement scale includes items such as “school is fun and measure was included as a control for measurement error.
exciting,” “I look forward to going to school,” and “I look
forward to learning new things.” The mean of the three Covariates. A series of covariates were included in the
items was used for the School Engagement scale. If more models to control for the effects of student preassignment
than one item was missing, then the scale score was missing. and school effects that may have remained after randomiza-
The threshold used for the binary outcome variable was the tion, and to control for effects that, if unaddressed, might
mean. Overall, 58% of the students reported positive antici- have inadvertently confounded the estimation of the treat-
pation at or above average on the School Engagement scale. ment assignment and dosage effects. Student-level covari-
For this study sample, the scale had a Cronbach’s alpha value ates included indicators of minority status, gender, and an
of .80 and a range of 1–5. indicator of free or reduced-price lunch status that was used32 The Journal of Educational Research
as a proxy measure of family socioeconomic status. Other almost no missing individual values left to substantively in-
student-level variables were tested but omitted from the fi- fluence parameter estimates. Given the minimal difference
nal models because sensitivity testing indicated they were between imputation/deletion and complete case analysis, we
not statistically significant and did not substantively alter proceeded with complete case analysis.
the treatment effect; these included indicators of academic
giftedness, special education, and students living with sin- Analysis
gle parents. In addition, the models included measures of Descriptive analysis. Descriptive methods were used to ex-
student deviation from the aggregated teacher team CRI amine the changes over time in school valuing and engage-
dosage measure to control for variation between students in ment as well as their relationship with CRI. The decline in
perception of CRI. middle school psychosocial engagement as well as the corre-
School-level covariates were treated differently in the lation between CRI and psychosocial engagement measures
CareerStart treatment and the CRI models. The treatment were illustrated using descriptive methods.
models included random intercepts for schools. In addition
to these random effects, the CareerStart treatment model in-
cluded two school-level measures. First, an indicator variable Multilevel modeling. The data were multilevel because (a)
of Equity Plus status was included in the model. Schools were the units of random assignment (school) and CRI (teacher
designated as Equity Plus schools by the district based on the team) were not the same as the unit of analysis (student),
percent of students using free or reduced-price lunch and and (b) because of the multistage sampling procedure used.
other need measures. Equity Plus schools were provided ad- The multilevel structure cannot be ignored without threats
ditional resources for smaller classes and additional student to the validity of statistical conclusions in the form of under-
services. Second, an aggregated measure of school poverty estimated standard errors for school- or team-level variables.
(the percentage of students receiving free or reduced-price Further, because the outcomes were binary, nonlinear mod-
lunch) was included in the CareerStart treatment model. For eling was an appropriate choice. Therefore, we used hierar-
the CRI models, school indicator variables were included to chical generalized linear modeling (HGLM; Raudenbush &
control for fixed effects at the school level. Bryk, 2002) to estimate nonlinear models with appropriately
adjusted standard errors. With binary outcomes, the depen-
Missing data. We conducted an analysis of the missing dent variable was assumed to have a binomial distribution,
data to determine the extent of missingness in the data and and a logit function was used to link the probability (Y) of
how to handle the missing values. There were missing values high engagement at the end of Grade 8 to a linear model.
throughout the data, including on student variables such as The pseudo-interclass correlation (ICC) measures recom-
female (number missing = 51), minority (number missing = mended by Snijders and Boskers (1999) for multilevel logis-
31), and single-parent status (number missing = 107). The tic models ranged from 0.02 to 0.07 for the four models. As
dependent variables were missing in only those cases for long as the pseudo-ICC is not zero, then it is sufficient to
which more than half of the items were missing; in cases in have an effect on the standard error and a multilevel model is
which fewer than half of the items were missing, we used a appropriate. The design effect size, which incorporates both
process by which only valid responses were counted (Schafer the ICC and the sample sizes, ranges from 3.45 to 16.17 for
& Graham, 2002). Nevertheless, 21.62% of the sample was the four models (see results tables). A design effect size of 2
missing data for school valuing, and 21.59% was missing data or higher suggests that multilevel modeling should be used
for school engagement. Although student-reported CRI data (Maas & Hox, 2005).
were missing for 15.81% of the sample, we assumed that the The percent of variance explained in the models is calcu-
aggregation of student reports to the teacher level for the lated for Level 2, as there is no error in the models at Level
independent variable in the CRI models minimized the bias 1. The variance, which ranges from −42.1% to 55.6% for
that nonrandom missing values may have caused. the four models is reported in the results tables. The nega-
Prior to estimating the models, we examined the distri- tive percent change for the School Engagement/CareerStart
bution of missingness (i.e., whether the missing values were Treatment model is consistent with the finding that the
missing completely at random [MCAR]) using the Little treatment was not significant in that model.
(1988) test for MCAR. At the individual level, the findings
CareerStart treatment model. The independent variable
of the Little test suggested that the missing values were not
for CareerStart treatment was entered into a two-level
MCAR; therefore, the resulting parameter bias could be re-
(school/student) multilevel model.
duced using a strategy such as multiple imputation (Schafer,
1997). However, after following the recommendation that
Level 1 (student):
imputed values of the dependent variable should be deleted
after imputation to improve efficiency of the parameter esti-
y
mates (von Hippel, 2007), we found that the benefits from ln = β0 j + β1 j femalei j + β2 j minorityi j
1 − y ij
using multiple imputation would be minimal. After list-
wise deletion of the imputed dependent variable, there were + β3 j frli j + β4 j value6i j + β5 j value7i jThe Journal of Educational Research 33
Level 2 (school): models were run using SAS 9.2 PROC GLIMMIX (SAS
Institute, Cary, NC).
β0 j = γ00 + γ01 EquityPlus j + γ02 Sch poverty j
+ γ03 CSTreatment j + u 0 j Results
β1 j = γ10 , β2 j = γ20 , β3 j = γ30 , β4 j = γ40 , β5 j = γ50 Student Engagement During Middle School
In the Level 1 model, minority represents minority stu- Consistent with findings from other studies (Woolley &
dents, including American Indian, African American, and Bowen, 2007), we found that measures of psychosocial en-
Hispanic students; frl represents socioeconomic status (i.e., gagement declined over the middle school years (Table 2).
students receiving free or reduced-cost lunch); value6 repre- For example, the mean score on student-reported engage-
sents the Grade 6 valuing score; and value7 represents the ment declined from 3.53 to 3.22 between the spring of Grade
Grade 7 valuing score. In the Level 2 model, EquityPlus rep- 6 and the spring of Grade 8. Similarly, the mean score of
resents whether the school was assigned EquityPlus status, student-reported valuing declined from 4.14 to 3.95 over
sch poverty represents the percentage of students at the the same period. Because the students had a year of middle
school receiving free or reduced-cost lunch, and CSTreat- school prior to the first engagement measure, this decline
ment represents the school’s status as randomly assigned to includes just 2 years of middle school (in the study location,
either the treatment or the control condition. The combined middle school comprised Grades 6, 7, and 8). A similar anal-
model represents an effort to obtain unbiased estimates of ysis with an earlier cohort that included a pretest measure
the CareerStart treatment effect (γ 03 ). administered before the Grade 6 year found a greater decline
over the full 3 years of middle school.
Table 3 shows the cumulative effect of CRI on psychoso-
CRI model. The independent variable for CRI was en-
cial engagement in Grade 8 based on teacher use of career
tered in a two-level (teacher team, student) multilevel
examples in the four core classes over the 3 years of middle
model. Variation at the school level was controlled us-
school (range 0–12). For example, only 34% of students who
ing school fixed effects (dummy variables for 13 of the 14
reported hearing no career examples in any of their 12 core
schools).
middle school classes also reported high school engagement.
However, 83% of students who reported hearing career ex-
Level 1 (student):
amples in 11 or 12 of their core classes during middle school
also reported high levels of school engagement. In addition,
y although not shown in the table, even after controlling for
ln = β0 j + β1 j femalei j + β2 j minorityi j
1−y ij gender, race, socioeconomic status, single-parent household
status, academic giftedness, and previous engagement mea-
+ β3 j frli j + β4 j value6i j + β5 j value7i j
sures, the increase in school valuing and engagement with
+ β6 j CRIvari j + 13school indicators additional CRI persisted. In other words, we found that stu-
dents whose teachers often provided career examples were
Level 2 (teacher team): more likely to maintain high levels of school valuing and
engagement over time.
β0 j = γ00 + γ01 teamCRI j + u 0 j
β1 j = γ10 , β2 j = γ20 , β3 j = γ30 ,
β4 j = γ40 , β5 j = γ50 , β6 j = γ60
TABLE 2. Psychosocial Engagement During Middle
School Years
In the Level 1 model, CRIvar represents the student de-
viation from the mean teacher team CRI, and teamCRI rep- Fall Spring Spring Spring
resents the mean teacher team CRI. The combined model Grade 6 Grade 6 Grade 7 Grade 8
represents an effort to obtain unbiased estimates of the CRI
Cohort 1
dosage when the coefficient of interest is γ 01 , the effect of School valuing 4.24 4.19 4.10 3.97a
the CRI dosage. School engagement 3.80 3.57 3.42 3.25a
In the next section, the odds ratios of the multilevel lo- Cohort 2
gistic models are reported and discussed for each student School valuing N/A 4.14 4.03 3.95a
engagement outcome. The odds ratios for the CareerStart School engagement N/A 3.53 3.32 3.22a
treatment model compare the odds of the outcome under Note. N/A = not applicable.
the CareerStart treatment to the odds under the control aSignificantly smaller than first measurement either fall or spring of
condition. The odds ratios for the CRI model represent the Grade 6 at p < .001.
odds of the outcome for each unit increase in CRI. Both34 The Journal of Educational Research
Not surprisingly, these data indicated that students who re-
TABLE 3. Percent of Students with High Engagement ported higher levels of school valuing in their Grade 6 and
and Valuing by Student-Reported CRI Grade 7 surveys also reported higher levels of school valuing
in the Grade 8 survey.
Number of core classes student School Although the CareerStart treatment effect was positive
reported hearing career examples School engage-
(Grades 6–8) valuing ment for school engagement, as measured by the School Success
Profile Engagement scale (odds ratio = 1.15), this finding was
0 78% 34% not statistically significant. As was true for school valuing,
1 or 2 85% 48% levels for school engagement were linked to the student’s
3 or 4 87% 44%
ethnicity and their own previous levels of engagement, but
5 or 6 91% 54%
7 or 8 94% 64% no other control factors contributed to the model for en-
9 or 10 98% 67% gagement. It is interesting to note that poverty status, which
11 or 12 97% 83% was measured by either students using free and reduced-price
lunch programs or the school’s percentage of students using
Note. CRI = career-relevant instruction.
these lunch programs did not contribute to either model.
CRI Model
CareerStart Treatment Model
Table 5 presents the results of the multilevel logistic
Table 4 presents the results of the multilevel logistic model model for student-reported CRI. The model found that an
for the CareerStart treatment effect. The model found that additional unit of teacher team CRI was associated with stu-
students in the CareerStart treatment schools were 41% dents being 31% more likely to report above-average levels of
more likely to report above-average levels of school valu- school valuing and 59% more likely to report above-average
ing as compared with students in control schools (odds ratio levels of school engagement than students with lower levels
= 1.41), indicating a significant positive effect for Career- of teacher team CRI. Because school valuing and engage-
Start. In addition, the analysis indicated that boys reported ment levels can be linked to such factors as the students’
significantly lower valuing scores than girls, and Asian and ethnicity, gender, poverty, as well as each student’s own pre-
Caucasian students reported significantly lower valuing than vious levels of engagement, the analysis controlled for these
American Indian, African American, and Hispanic students. factors.
The analysis also controlled for student-reported valuing lev- In summary, boys reported significantly lower school valu-
els measured in the spring of their Grade 6 and Grade 7 years. ing scores than girls, consistent with other research (Voelkl,
TABLE 4. Multilevel Logistic Model Odds Ratios for CareerStart Treatment Effect
Valuing School engagement
Intercept — —
∗∗∗
Student, female (m = 0, f = 1) 1.74 1.16
∗∗∗ ∗∗∗
Student, minority (Asian–Caucasian = 0, 1.93 1.68
African American–Hispanic = 1)
Student, poverty (nFRL = 0, FRL = 1) 0.86 1.10
∗∗∗ ∗∗∗
Student, Grade 6 outcome measure 1.63 1.54
∗∗∗ ∗∗∗
Student, Grade 7 outcome measure 4.24 2.46
∗
School, EquityPlus (0 = not EquityPlus, 0.56 1.07
1 = EquityPlus)
School, percent of students using FRL 1.76 1.31
∗∗
CareerStart treatment 1.41 1.15
Random effect (school) 0.03 0.03
n 2,270.0 2,270.0
−2 log likelihood 10662.8 10500.3
Generalized χ 2 2439.1 2280.6
Design effect size 6.71 16.17
Variance explained 55.6% –42.1%
Note. FRL = receiving free or reduced-price lunch; nFRL = not receiving free or reduced-price lunch.
∗ ∗∗ ∗∗∗
p < .05. p < .01. p < .001.The Journal of Educational Research 35
TABLE 5. Multilevel Logistic Model Odds Ratios for CRI
Valuing School engagement
Intercept — —
∗∗∗
Student, female (m = 0, f = 1) 1.77 1.15
∗∗∗ ∗∗∗
Student, minority (Asian–Caucasian = 0, 1.92 1.73
African American–Hispanic = 1)
Student, poverty (nFRL = 0, FRL = 1) 0.80† 0.99
∗∗∗ ∗∗∗
Student, Grade 6 outcome measure 1.63 1.54
∗∗∗ ∗∗∗
Student, Grade 7 outcome measure 3.99 2.47
∗∗∗ ∗∗∗
Student, variation from Team CRI 1.16 1.16
School indicator variables (13)
∗∗ ∗∗∗
Team, Grade 8 CRI 1.31 1.59
Random effect (teacher team) 0.06 0.09
n 2,222.0 2,221.0
−2 log likelihood 10705.0 10498.0
Generalized χ 2 2432.9 2188.0
Design effect size 3.45 6.68
Variance explained 36.5% 24.3%
Note. CRI = career-relevant instruction; FRL = receiving free or reduced-price lunch; nFRL = not receiving free or reduced-price lunch.
†p < .10. ∗ p < .05. ∗∗ p < .01. ∗ ∗ ∗ p < .001.
1996). Asian and Caucasian students also reported signifi- additional career examples in core classes increased school
cantly lower levels of school valuing and engagement than valuing and school engagement.
did American Indian, African American, and Hispanic stu- These findings are consistent with previous research sug-
dents. These findings are consistent with previous research gesting that student career exploration can have positive ef-
indicating that school psychosocial engagement may have fects on school engagement (Castellano et al., 2003; Kenny
a more significant role in the lives of minority students et al., 2006). The data also support the theoretical propo-
(Wilson, 1996). The analysis controlled for student-reported sitions of possible-selves theory (Markus & Nurius, 1986),
school valuing and engagement levels that were measured which proposes that students will attend to their learning
in the spring of their Grade 6 and Grade 7 years. These data and view school as having more value when class content is
indicated that students who reported higher levels of school provided in the context of information that the student val-
valuing and engagement during Grades 6 and 7 also reported ues or considers relevant for future benefits and choices. The
higher levels of school valuing and engagement in Grade 8. potential contribution of content and course relevancy has
School fixed-effects were included in the model by means of been proposed by the Gates Foundation (Vander Ark, 2005)
indicator variables (not shown). and other education reformers, and the present research sup-
ports this direction of education curricular reform.
Discussion One of the benefits of CareerStart may be its simplicity
in implementation. Because teachers were not asked to add
The findings from this longitudinal study of middle school new substantive content to their already demanding course
students indicated that the students in the schools randomly schedules, the intervention appears to have been readily ac-
selected to implement CareerStart were significantly more cepted by the teachers. In addition, teachers may have been
likely to value their education than students in the control receptive to CareerStart because the lessons were prepared
schools, as measured by a standardized scale that assessed by peer teachers and all the materials needed to carry out the
whether students viewed school as important and providing lessons were available online, which aided teachers in their
useful information and knowledge that will help them in instructional planning. The teacher training and coaching
the future. This finding indicates that the CareerStart pro- was kept as simple as possible, and much of that support
gram is a promising approach to teacher pedagogy and leads was provided by the lead teachers or curriculum coordina-
to an important type of psychosocial engagement among tors in the schools themselves. This use of a school’s existing
students. An additional analysis found that students who infrastructure helped make CareerStart a school-based in-
reported higher levels of CRI from their core teachers over tervention with a broad array of instructional staff engaged
the 3 years of middle school, regardless of whether they were in the effort.
in a treatment or control school, also reported higher levels Although the treatment resulted in a positive effect for
of school valuing and school engagement, indicating that both psychosocial measures of engagement, only our measure36 The Journal of Educational Research
of school valuing (Voelkl, 1996) was significant. The non- given the nonnormal distribution of the responses to the out-
significant findings for the School Success Profile measure come measures, dichotomous outcome variables were used
of school engagement (Bowen et al., 2005) suggests that the in the models.
CareerStart intervention may promote in students a more
positive set of beliefs that participating in school is impor- Implications for Teachers and Schools
tant and will yield longer-term benefits (i.e., valuing) but
not necessarily result in students seeing school in the more One benefit of the CareerStart approach to middle school
immediate sense as being fun and something to look forward instruction is that the program is relatively easy to imple-
to each day (i.e., engaging). Perhaps the career connections ment and support. Many teachers already routinely illustrate
that students are making in the classroom are increasing the their teaching with examples from the students’ environ-
overall relevance of the school experience (Orthner, 2007), ment to help students recognize the relevance of the lesson
a goal of the CareerStart program, but not having as much content. Such teachers understand that contemporary rele-
impact on the day-to-day excitement that students may be vance helps them connect better to their students’ interests
hoping for from their school experiences. and attention. However, the benefits of students recognizing
On a positive note, however, higher levels of CRI, as re- the relevance of content to their futures are not as well un-
ported by students, resulted in significantly higher effects derstood; therefore, teachers may need help in understand-
for both measures of psychosocial engagement. There was ing the value that students place on connecting their middle
a strong and significant association between the number of school instruction to future education and career possibili-
core teachers students reported as providing more frequent ties. The findings reported here indicate that the CareerStart
career examples and higher levels of school valuing and approach and CRI does benefit students’ attention to their
engagement. This confirmed our expectations and may indi- schooling, thus promoting the instructional environment
cate that when students themselves experience career rele- that school teachers and principals want to nourish.
vance in instruction, and when this is repeated in class after CRI can be implemented by simply encouraging teachers
class, and grade after grade, the connection to both short- to meet and share examples of jobs and careers in which
and long-term benefits of being in school are realized. Thus, the content of their courses has application. We have found
CareerStart, as a program, promoted CRI and student school inviting parents and employers into the classrooms to talk
valuing but the more of this instruction that occurred, and about how they use mathematics, science, and language con-
the more teachers who used this pedagogy over the 3 mid- cepts can be helpful to students in realizing the relevance of
dle school years, the more likely that the benefits were seen the content that is explored and taught in their classes. Some
by students in the short term (engagement) and long term middle schools have even offered job fairs where employers
(valuing). are invited to talk about the jobs available in their businesses
or industries and how important the employees’ education is
Limitations to being successful in those jobs. One school had a checklist
for students that asked employers what kinds of reading and
Two primary limitations of this analysis may have diluted writing were required of their employees, as well as the spe-
the treatment effect, even though the reported treatment cific use of mathematics, science, or civic knowledge their
effect was significant for one of the two engagement mea- employees needed to demonstrate. This depth of information
sures. First, the low effective sample size of 14 schools in the and employer feedback can help engage students in an ex-
cluster-randomized evaluation design might have limited the ploration of their future education and career opportunities
treatment effect. Including a larger number of schools would that will also help students connect their present learning to
have increased the robustness of this study. Second, uneven future career thinking. Moreover, as suggested by our find-
treatment fidelity may also have diluted the treatment effect ings, it is this hope—that an investment in education will
due to low implementation in some treatment schools and pay dividends in the future—that serves both to improve
CRI by some teachers in control schools. student engagement and to create the value students place
Other study limitations include that the CareerStart pro- on being in and staying in school.
gram was evaluated in a single district, native to the study
district, and in part developed by teachers in the study dis- Conclusion
trict. In addition, some students moved between treatment
and control schools. To account for this mobility, the anal- The evaluation of teachers’ use of CRI in middle school
ysis used a decision rule that called for all students who classrooms yielded a variety of promising findings. The re-
changed treatment conditions to be assigned to the treat- sults confirm that students who report being exposed to more
ment group. However, this rule may have further diluted career examples as part of their core subject instruction are
the results if, as a group, mobile students have lower school more likely to report higher levels of both school valuing
engagement than other students. Finally, the use of dichoto- and school engagement. Students in CareerStart treatment
mous outcome measures may be considered a limitation in schools also reported significantly higher levels of school
that a certain amount of variance was eliminated. However, valuing than students in the control schools. These findingsThe Journal of Educational Research 37
support the importance of the relevance component of in- Kenny, M. E., Blustein, D. L., Haase, R. A., Jackson, J., & Perry, J.
struction that has been increasingly promoted by the Gates C. (2006). Setting the stage: Career development and the student
engagement process. Journal of Counseling Psychology, 53, 272–279.
Foundation and others as a potential asset for transform- doi:10.1037/0022–0167.53.2.272
ing public education. In this analysis, we confirmed through Klem, A. M., & Connell, J. P. (2004). Relationships matter: Linking teacher
an evidence-based approach that teaching methods or ped- support to student engagement and achievement. Journal of School Health,
74, 262–274. doi:10.1111/j.1746–1561.2004.tb08283.x
agogy can target students’ motivation to learn, offering a Little, R. J. A. (1988). A test of missing completely at random for multivari-
practical approach to education reform that takes advantage ate data with missing values. Journal of the American Statistical Association,
of and augments the capacity of our present instructional en- 83, 1198–1202. doi:10.2307/2290157
Maas, C. J. M., & Hox, J. J. (2005). Sufficient sample sizes for multilevel
vironment in the middle grades. More analyses are planned modeling. Methodology, 1, 86–89. doi:10.1027/1614–2241.1.3.86
to strengthen our understanding of how this instructional Markus, H. R., & Nurius, P. (1986). Possible selves. American Psychologist,
strategy benefits selected student groups. 41, 954–969. doi:10.1037/0003–066X.41.9.954
Orthner, D. K. (2007). Public schools: Building capacity for hope and
opportunity. In J. Edwards, M. Crain, & A. L. Kalleberg (Eds.). Ending
ACKNOWLEDGMENT poverty: How to restore the American dream (pp. 218–229). New York, NY:
New Press.
An earlier version of this study was presented at the American Education Orthner, D. K., Akos, P., Rose, R., Jones-Sanpei, H., Mercado, M., &
Research Association meeting in Denver, Colorado, on May 2, 2010. Woolley, M. (2010). CareerStart: A middle school student engage-
ment and academic achievement program. Children and Schools, 32, 223–
REFERENCES 234.
Orthner, D. K., Cook, P., Rose, R., & Randolph, K. A. (2002). Welfare
Akos, P., & Galassi, J. (2004). Middle and high school transitions as reform, poverty, and children’s performance in school: Challenges for
viewed by students, parents, and teachers. Professional School Counsel- the school community. Children and Schools, 24, 105–121.
ing, 7, 212–221. Oyserman, D. (2008). Possible selves: Identity-based motivation and school
Akos, P., Lambie, G., Milsom, A., & Gilbert, K. (2007). Early adoles- success. In H. Marsh, R. Craven, & D., McInerney (Eds.), Self-processes,
cents’ aspirations and academic tracking: An exploratory investigation. learning and enabling human potential dynamic new approaches (Vol. 3, pp.
Professional School Counseling, 11(1), 57–64. 269–288). Charlotte, NC: Information Age Press.
Akos, P., Shoffner, M., & Ellis, M. (2007). Mathematics placement and the Perry, J. C. (2008). School engagement among urban youth of color: Crite-
transition to middle school. Professional School Counseling, 10, 238–244. rion pattern effects of vocational exploration and racial identity. Journal
Alexander, K. L., Entwisle, D. R., & Horsey, C. S. (1997). From first grade of Career Development, 34, 397–422. doi:10.1177/0894845308316293
forward: Early foundations of high school dropout. Sociology of Education, Perry, J. C., Liu, X., & Pabian, Y. (2010). School engagement as a mediator
70, 87–107. doi:10.2307/2673158 of academic performance among urban youth: The role of career prepara-
Alspaugh, J. W. (1998). Achievement loss associated with the transition to tion, parental career support, and teacher support. Counseling Psychologist,
middle school and high school. The Journal of Educational Research, 92, 38, 269–295. doi:10.1177/0011000009349272
20–25. doi:10.1080/00220679809597572 Plank, S. (2001). Career and technical education in the balance: High school
Anderman, E. M., Maehr, M. L., & Midgley, C. (1999). Declining motiva- persistence, academic achievement and post-secondary destinations. St. Paul,
tion after the transition to middle school: Schools can make a difference. MN: National Research Center for Career and Technical Education.
Journal of Research and Development in Education, 32, 131–147. Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Appli-
Bishop, P. A., & Pflaum, S. W. (2005). Student perceptions of action, cations and data analysis methods (Vol. 1, 2nd ed.). Thousand Oaks, CA:
relevance and pace. Middle School Journal, 36, 4–12. Sage.
Blafanz, R., Fox, J., Bridgeland, J., & McNaught, M. (2009). Grad nation: Schafer, J. L. (1997). Analysis of incomplete multivariate data. London, UK:
A guidebook to help communities’ tackle the dropout crisis. Washington, Chapman and Hall.
DC: America’s Promise Alliance. Retrieved from http://www.everyday- Schafer, J. L., & Graham, J. W. (2002). Missing data: Our view of
democracy.org/en/Resource.169.aspx the state of the art. Psychological Methods, 7, 147–177. doi:10.1037/
Borman, G. D., & Overman, L. T. (2004). Academic resilience in mathe- 1082–989X.7.2.147
matics among poor and minority students. The Elementary School Journal, Snijders, T. A. B.. & Boskers, R. J. (1999). Multilevel analysis: An introduction
104, 177–195. doi:10.1086/499748. to basic and advanced multilevel modeling. Newbury Park, CA: Sage.
Bowen, G. L., Rose, R. A., & Bowen, N. A. (2005). The reliability and validity Trice, A. D., Hughes, M. A., Odom, C., Woods, K., & McClellan, N. C.
of the School Success Profile. Philadelphia, PA: Xlibris Press. (1995). The origins of children’s career aspirations: IV. Testing hypothe-
Castellano, M., Stringfield, S., & Stone, J. R. (2003). Secondary career ses from four theories. Career Development Quarterly, 43, 307–322.
and technical education and comprehensive school reform: Implications Trice, A. D., & King, R. (1991). Stability of kindergarten children’s career
for research and practice. Review of Educational Research, 73, 231–272. aspirations. Psychological Reports, 68, 1378. doi:10.2466/PRO.68.4.1378-
doi:10.3102/00346543073002231 1378
Eccles, J. S., Wigfield, A., Midgley, C., Reuman, D., Mac Iver, D., & U.S. Department of Education. (2010a). Table 6. Status dropout rates
Feldlaufer, J. (1993). Negative effects of traditional middle schools and number and distribution of dropouts of 16 through 24-year-olds, by
on students’ motivation. Elementary School Journal, 93, 553–574. selected background characteristics: October 2008. Trends in High School
doi:10.1086/461740 Dropout and Completion Rates in the United States: 1972–2008. Retrieved
Finn, J. D., & Rock, D. A. (1997). Academic success among students from http://nces.ed.gov/pubs2011/dropout08/tables/table 06.asp
at-risk for school failure. Journal of Applied Psychology, 82, 221–234. U.S. Department of Education. (2010b). Table 110. High school grad-
doi:10.1037/0021–9010.82.2.221 uates, by sex and control of school: Selected years, 1869–70 through
Fredricks, J. A., Blumenfeld, P. C., & Paris, A. H. (2004). School en- 2019–20. Digest of Education Statistics. Retrieved from http://nces.ed.gov/
gagement: Potential of the concept, state of the evidence. Review of programs/digest/d10/tables/dt10 110.asp
Educational Research, 74, 59–109. doi:10.3102/00346543074001059 U.S. Department of Education (2010b). “Table 6. Status dropout rates
Gonzalez, R., & Padilla, A. M. (1997). The academic resilience of Mexican- and number and distribution of dropouts of 16 through 24-year-olds, by
American high school students. Hispanic Journal of Behavioral Sciences, selected background characteristics: October 2008.” Trends in High School
19, 301–317. doi:10.1177/07399863970193004 Dropout and Completion Rates in the United States: 1972–2008. Retrieved
Gottfredson, L. S. (1981). Circumscription and compromise: A develop- from http://nces.ed.gov/pubs2011/dropout08/tables/table 06.asp
mental theory of occupational aspirations. Journal of Counseling Psychol- Vander Ark, T. (2005). Every student a college ready graduate. Seattle, WA:
ogy Monograph, 28, 545–579. doi:10.1037/0022–0167.28.6.545 Bill & Melinda Gates Foundation.
Imbens, G. W., & Wooldridge (2009). Recent developments in the econo- Voelkl, K. E. (1996). Measuring students’ identification with school.
metrics of program evaluation. Journal of Economic Literature, 47, 5–86. Educational and Psychological Measurement, 56, 760–770. doi:10.1177/
doi:10.1257/jel.47.1.5 0013164496056005003You can also read
