CEPEG Policy Brief: Curriculum Counts: Math and Science Textbook Adoptions and Effects
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
CEPEG Policy Brief: Curriculum Counts: Center on Education Policy, Equity and Governance Math and Science Textbook Adoptions and Effects Morgan S. Polikoff Shauna Campbell Shira Korn Rachel White Stephani Wrabel Tenice Hardaway Hovanes Gasparian Cory Koedel May 2020
CEPEG Policy Brief: Curriculum Counts: Math and Science Textbook Adoptions and Effects There has been considerable policy effort to improve math and science teaching and learning in U.S. schools over the last decade. A particularly prominent policy focus has been on the adoption of new content standards [the Common Core State Standards (CCSS) and the Next Generation Science Standards (NGSS)] in many states. The adoption of these new standards reflects the widespread beliefs that the content of instruction matters for student learning (1) and common standards can improve the content and quality of instruction and, through that, educational outcomes (2). But standards alone are unlikely to meaningfully affect instruction if teachers are not supported to understand and implement them. Textbooks and other curriculum materials are key instructional supports that a) are used by nearly all teachers (3), b) serve as a bridge between standards and instruction, affecting what and how teachers teach (4, 5, 6), and c) have been shown to affect student learning (4, 7, 8, 9, 10, 11, 12). Though the textbook market is changing with new open- source and online materials, standard textbooks are still a widely-used and relatively low-cost intervention. The research community is increasingly cognizant of the potential for better textbooks to improve achievement in schools (13). Research on textbooks is hampered by a lack of basic data on textbook adoptions by schools and districts. Most state departments of education keep absolutely no records of which textbook materials are used in which schools and districts. Among the few states we are aware of that do (California, Texas, New Mexico, Louisiana), the data typically require extensive effort to make usable for research. Without access to detailed data on textbook adoptions, it is not possible to understand even the most basic facts about curriculum adoptions, let alone perform rigorous efficacy analyses. We have been funded by the National Science Foundation and several private foundations to collect and analyze statewide textbook adoption data in math and science in the five largest U.S. states—California, Texas, New York, Florida, and Illinois . This work has provided us with the data to study, among other things, the impact of textbooks on elementary student achievement in mathematics (10, 11, 12). In this report, we discuss what we have learned in our efforts to collect and analyze textbook adoption data—to our knowledge the largest such effort in the history of educational research. (Another recent study undertook a similar effort, identifying elementary mathematics textbooks used across six states.) (23) We describe our processes for collecting the data in each state and the challenges we encountered, and report on the most common textbooks used in each grade band in each state and subject. Our work has important implications for implementing the CCSS and NGSS, and more generally, for efforts to improve math and science achievement of students in U.S. schools. COLLECTING TEXTBOOK DATA While the process for collecting and cleaning the textbook data varied across the five states, it was not straightforward in any state. The Texas data are the most readily available. The Texas Education Agency (TEA) provides a publicly available file of all textbook purchases for the current year (14). Previous years’ data can be obtained via a Freedom of Information Act (FOIA) request to the TEA. While the data are comprehensive, their reporting is not standardized. Thus, there is not consistency in naming conventions, even for commonly adopted books. For example, there are over 100 unique spellings in the database for the most commonly adopted science book, Science Fusion. California textbook data are also publicly accessible; however, the reporting is also far from ideal. The state requires each school to report textbook use on annual School Accountability Report Cards (SARCs) as a result of a 2004 civil rights lawsuit (15). SARCs are available in PDF format on the state website for the most recent two years. While they are mostly complete (more than 90 percent of schools provide some textbook data), the data must be manually pulled from each school’s SARC. Furthermore, the lack of a common SARC format or textbook naming convention makes cleaning and analyzing the data difficult. Gathering data for even a single subject took hundreds of hours of work by our team, and often required our discretion in identifying books because of poorly specified titles. Similar to Texas, California’s most common pre-CCSS elementary mathematics book—Scott Foresman Addison Wesley’s enVision Math California—was reported with well over 100 unique spellings in the SARC PDFs. ‒1‒
Although it would be a stretch to say that the data in Texas and California were easily obtained, the situation is much worse in the other three states. Illinois, Florida and New York do not have publicly available sources of textbook data of which we are aware. To collect the data ourselves, we first attempted to survey district leaders through a custom web- survey. This produced a response rate of roughly two percent. We then sent state-specific FOIA letters and emails over multiple rounds to every school district in these states. This produced response rates of 58% to 70% in Florida, 54% to 69% in Illinois, and 29% to 47% in New York, depending on subject and grade span. Moreover, once we finally received responses, converting the textbook information into consistent, usable data was a significant challenge, similar to Texas and California. In addition to the common problem of non-standardized reporting, there were other errors and problems. For example, many schools listed their textbook as the name of the class in which the book was used (e.g., “Biology”, “Chemistry”, or even just “Science”). A conclusion from our data collection efforts is that it is currently extremely difficult to obtain information as basic as which textbooks are being used in which schools in the U.S. WHAT WE HAVE LEARNED TEXTBOOK ADOPTION PATTERNS Based on the data we collected and cleaned, Tables 1 and 2 present descriptive information on the most popular math and science textbooks in public schools in the five focal states for elementary and middle grades as of approximately 2017-18. While this is exceedingly simple information, it is important to recognize that without our data collection efforts in these states, the information would not be available. Table 1 shows several clear patterns. First, there is a good deal of concentration in most subjects, states, and grades. The most concentrated is Florida elementary math, where approximately three-quarters of schools use Go Math. The least concentrated is New York middle school math, where no one book series is used in more than 10% of districts. Second, the most popular books in most states, subjects, and grade levels are books from the “big three” publishers (Pearson, Houghton Mifflin Harcourt, and McGraw Hill). Books from other publishers that achieved modest penetration include Eureka Math, Big Ideas Math, FOSS, and STEMScopes. Third, there is considerable variation in the most popular books across states and grade spans. MyMath, Go Math, and EnVision Math are the most popular in elementary mathe- matics, whereas Go Math, Glencoe Math, and Prentice Hall Math are the most popular in middle school. In elementary science the most popular books are McMillan, Scott Foresman, and Science Fusion, whereas Glencoe, Science Explorer, Science Fusion, and Discovery Education are the most common in middle school science. ‒2‒
TABLE 1 - MOST POPULAR MATHEMATICS TEXTBOOKS IN FIVE PARTNER STATES AT ELEMENTARY AND MIDDLE GRADES Elementary (K-5) Middle (6-8) Percent Percent Title Publisher adopted Title Publisher adopted California* (n=5,133 elementary schools; n=3,826 middle schools) 1 My Math Common Core McGraw-Hill 20.5% GoMath! Common Core Houghton Mifflin Harcourt 16.1% 2 GoMath! Common Core Houghton Mifflin Harcourt 18.1% Middle School Math Series Glencoe/McGraw-Hill 13.4% 3 enVision Math Common Core Pearson 14.3% Core Connections Series CPM 10.0% 4 Eureka Math/Engage NY Eureka Math 10.5% Big Ideas Math Big Ideas 9.8% 5 Math Expressions Common Core Houghton Mifflin Harcourt 9.5% Eureka Math/Engage NY Eureka Math 8.3% Texas (n=1,275 districts with elementary grades; n=1,143 districts with middle grades) 1 GoMath!** Houghton Mifflin Harcourt 43.1% GoMath! Houghton Mifflin Harcourt 41.3% 2 enVision Math 2.0 Pearson 35.5% Texas Math Bundle McGraw-Hill 20.3% 3 Texas Mathematics McGraw-Hill 4.1% digits Pearson 10.5% 4 Saxon Math Houghton Mifflin Harcourt 3.3% Texas STEM bundle McGraw-Hill 5.2% 5 IXL Math IXL Learning 2.3% Glencoe Math McGraw-Hill 4.3% Florida (n=47 districts with elementary grades; 46 districts with middle grades) 1 GoMath! Houghton Mifflin Harcourt 71.4% GoMath! Houghton Mifflin Harcourt 29.4% 2 enVision Math Pearson 8.2% CINCH Math McGraw Hill 11.8% 3 *** Math Connects Macmillan/McGraw Hill 11.8% 4 Glencoe Math Glencoe 8.8% 5 Connected Math Pearson 5.9% Illinois (n=518 districts with elementary grades; n= 509 districts with middle grades) Prentice Hall Mathematics, Course 1, 2, 3 1 My Math Common Core McGraw-Hill 30.8% Common Core Pearson/Prentice Hall 13.9% 2 Everyday Mathematics SRA/McGraw-Hill 19.9% Algebra I Pearson Prentice Hall 9.8% 3 enVision Math Common Core Pearson 15.6% Connected Mathematics Program (CMP) Pearson 9.5% 4 Go Math! Common Core Houghton Mifflin Harcourt 14.5% Glencoe Math Course 1, 2, 3 Common Core Glencoe 8.7% 5 Saxon Math Houghton Mifflin Harcourt 12.3% Glencoe Algebra 1 McGraw-Hill 6.1% New York (n=269 districts with elementary grades; 326 districts with middle grades) 1 enVision Math Common Core Pearson 26.2% Glencoe Math Course 1, 2, 3 Common Core Glencoe 8.2% New York State Department of New York State 2 EngageNY Education 20.0% EngageNY Department of Education 7.8% 3 My Math Common Core McGraw-Hill 14.6% digits, Common Core Pearson 5.6% 4 GoMath! Common Core Houghton Mifflin Harcourt 13.8% Connected Mathematics Program (CMP) Pearson 5.2% 5 Investigations in Numbers, Data and Space Pearson 11.5% Glencoe Pre-Algebra Glencoe/McGraw Hill 4.8% *Note: California reports at school level; all other states at district level **Note: Texas science textbook titles include both national- and state-specific versions ***Note: All other mathematics textbooks in Florida have only been adopted by 1 district
TABLE 2 - MOST POPULAR SCIENCE TEXTBOOKS IN FIVE PARTNER STATES AT ELEMENTARY AND MIDDLE GRADES Elementary (K-5) Middle (6-8) Percent Percent Title Publisher adopted Title Publisher adopted California* (n=5,601 elementary schools; n=3,181 middle schools) 1 Macmillan/McGraw Hill Science Pearson/Scott Foresman 28.2% Glencoe Focus On… Macmillan/McGraw-Hill 26.4% 2 FOSS Delta Education/School Specialty Science 27.2% California Science Explorer- Focus On:... Pearson-Prentice Hall 18.2% Houghton Mifflin California 3 Science Houghton Mifflin Harcourt 21.4% Prentice Hall Science Explorer… Pearson-Prentice Hall 15.4% Holt Science…Earth, Life and/or 4 Scott Foresman Science Pearson/Scott Foresman 19.1% Physical Houghton Mifflin Harcourt 8.5% 5 Harcourt Achieve Science Houghton Mifflin Harcourt 5.2% HMH Focus On… Houghton Mifflin Harcourt 6.2% Texas** (n=1,007 districts with elementary grades; n=967 districts with middle grades) 1 Science Fusion Houghton Mifflin Harcourt 47.5% Science Fusion Houghton Mifflin Harcourt 36.1% 2 STEMScopes Accelerate Learning 33.2% STEMScopes Accelerate Learning 30.2% 3 Interactive Science Pearson/Scott Foresman 22.7% Texas iScience McGraw-Hill 17.6% 4 Motivation Science Mentoring Minds 4.7% Interactive Science Pearson 11.7% 5 Discovery Education Techbook Discovery Education 2.6% Discovery Education Science Discovery Education 4.7% Florida (n=42 districts with elementary grades; 39 districts with middle grades) 1 Science Fusion Houghton Mifflin Harcourt 53.8% Discovery Education Techbook Discovery Education 21.2% National Geographic Science 2 Leveled Readers National Geographic 25.6% Nancy Larson Science Nancy Larson 21.2% 3 Scott Foresman Science Scott Foresman 10.3% Comprehensive Science Pearson/Prentice Hall 12.1% 4 Discovery Education Techbook Discovery Education 5.1% Interactive Earth Science Pearson/Prentice Hall 12.1% 5 Florida Interactive Science Pearson/Scott Foresman 5.1% McGraw Hill Science McGraw Hill 12.1% Illinois (n=416 districts with elementary grades; n=407 districts with middle grades) 1 Scott Foresman Science Scott Foresman 30.1% Science Explorer Series Pearson/Prentice Hall 32.1% 2 McGraw Hill Science McGraw Hill 18.5% Science and Technology Series Holt/HRW 14.5% 3 A Closer Look (Life Science) MacMillan/McGraw Hill 13.0% Glencoe Science McGraw-Hill 10.4% 4 HSP Science Harcourt 12.3% Focus On Series Prentice Hall 7.8% 5 Biology: The Living Environment Barron/Prentice Hall 10.5% Glencoe Physical Science McGraw-Hill 6.6% New York (n=204 districts with elementary grades; 275 districts with middle grades) 1 Scott Foresman Science Scott Foresman 26.2% Science Explorer Series Pearson/Prentice Hall 49.4% 2 McGraw Hill Science McGraw Hill 11.5% Science and Technology Series Holt/HRW 9.1% 3 A Closer Look (Life Science) MacMillan/McGraw Hill 11.5% Glencoe Science [unknown] Glencoe 9.1% 4 Harcourt Science Harcourt 10.0% Glencoe Physical Science Glencoe 8.7% 5 Biology: The Living Environment Barron/Prentice Hall 11.5% Focus On Series Prentice Hall 6.5% *Note: Unless specifically noted “California” in textbook title, all California science textbook title counts include both national- and state-specific versions. Additionally, note that California reports at school level; all other states at district level **Note: Texas science textbook titles include both national- and state-specific versions
TEXTBOOK EFFECTS ON ACHIEVEMENT The textbook data are interesting in a descriptive sense, but they are also useful for a variety of analyses. In work published elsewhere (12), we have linked the California textbook adoption data with school-level achievement outcomes to estimate the effects of elementary mathematics textbook on students’ mathematics achievement. We find that one book—Houghton Mifflin California Math—was more effective than the other three most commonly adopted books in raising third-though-fifth grade mathematics achievement. The effects were educationally meaningful—one tenth to one twentieth of a standard deviation of student achievement. Given the very low marginal cost of choosing one textbook over another (different textbooks are typically similarly priced), the benefit-cost ratio for choosing a more effective textbook is very large. We have also investigated the effects of English language development (ELD) curriculum materials in Texas (16). More broadly it would be of great value to assemble a larger database of curriculum-materials effects. The evaluations we have done, and are currently working on, can be used to assess the relative performance of different textbook options on the whole. This is a useful first step toward improved curriculum adoptions by school districts, which can raise student achievement (4, 7, 8, 9, 10, 11, 12). However, to answer the more fundamental question of why some books produce larger learning gains than others, we need many more efficacy evaluations. A foundational requirement for any efficacy evaluation is data on which curriculum materials are used in which schools. TEXTBOOK ADOPTION AND IMPLEMENTATION PROCESSES We complement our data collection and efficacy analysis work with structured interviews of district leaders aimed at improving our understanding of the process by which curriculum materials are adopted by districts. Among other things, the interviews can help us to identify possible leverage points for getting more effective materials to be more widely used. For instance, in California we selected a stratified random sample of 32 districts (stratifying on student enrollment, achievement levels, and type of textbook adopted) and interviewed them about their textbook adoption practices. A general theme from the interviews is that districts use similar, elaborate processes to choose materials. Teachers are heavily involved in the adoption process in all districts, but district leaders note that teachers lack crucial information about curriculum quality necessary to make their decisions (17). As such, adoption decisions are often driven by the “feel” of the text as opposed to anything more concrete. Textbook evaluation processes are also costly and time- consuming, limiting the amount of truly meaningful analyses that can be done amidst a full teaching schedule. The interviews also reveal that district leaders feel constrained by the poor quality and alignment of curriculum materials to state standards. In particular, district leaders were skeptical of traditional publishers’ claims of textbook alignment (consistent with evidence that books are often not well aligned (18)); furthermore, some expressed skepticism of the state’s efforts to evaluate materials. Districts are especially challenged by the timeline of standards adoption and textbook approval. District leaders report that high-quality textbook options are unavailable in the early years of standards adoption, at a time when teachers need the most help understanding the standards and making instructional changes. POLICY IMPLICATIONS: IMPROVING TEXTBOOK ADOPTIONS Our work suggests several policy implications for both states and school districts. School districts report being chal- lenged with textbook selection in the early years of a new set of standards. One recommendation is that states slow down timelines of standards and assessment rollouts. It may also be an efficient use of state resources to create or disseminate high-quality curriculum materials directly to districts (following the model of EngageNY), as this can reduce expenses for individual districts and lead to better alignment and more rapid curriculum availability for districts. While states have moved away over time from making formal textbook adoption lists with which districts must comply (19), we find that these lists are an important source of information for districts when they make adoption decisions. In California, for instance, though school districts are no longer required to adopt from the state-approved list, we find that the large majority of districts do (4 of the top 5 California elementary math textbooks in Table 1 are on the state list, for instance). Thus, we recommend states continue to put out recommended lists of aligned materials to help inform districts and centralize the process of vetting curriculum materials against state standards. Louisiana provides a model for this kind of leadership (20). ‒5‒
Our interviews in the many small, rural districts in California suggest that these districts are especially challenged with the resource demands associated with making textbook adoption decisions. They rely on intermediary organizations in California known as County Offices of Education to help them review materials, often in collaboration with other small districts in their counties (in some counties, there was even a county-wide adoption). Given the demand for collaborations by districts, states can invest in supporting districts to work together to evaluate and adopt materials. These types of collaborations can reduce redundant evaluative work across similar school districts and may improve decision making. A final policy recommendation is around the data needed for textbook efficacy analyses. Ideally improvements to adoption processes would lead to more reliance on concrete evidence on curriculum quality. However, for researchers to produce this evidence, they require access to data on school and district materials adoptions. We recommend that states 1) routinely collect data on textbook adoptions, purchases, or availability in schools and districts, 2) work to ensure these data are useable (e.g., by implementing standard naming conventions for materials), and 3) make the data and results of such analyses available to district leaders. We need data on which curriculum materials are being used where, and states are by far the best equipped to provide these data. RESEARCH There is a great deal more to know about the adoption and use of curriculum materials in K-12 schools. Broadly, our recommendations for research fall in three main areas: adoption, implementation, and effectiveness. Crosscutting these areas are the two types of materials dominating K-12 classrooms today—traditional textbooks and teacher—or district-selected supplementary materials. While our work in California has helped us understand adoptions there, states with different structures (especially states that do not partially-centralize the evaluation process) may have different practices in schools and districts. Practices may also vary in different subjects, as well as in schools of choice (charters, magnets). As educators are increasingly moving away from traditional textbooks and toward open educational resources, there is also a great deal more to be known about how schools and districts make decisions about the adoption of these non-traditional materials. Lists of textbook names provide useful information, but they do not tell us how these materials are used. Surveys and interviews can be used to gauge curriculum materials implementation in samples ranging from single schools/districts up to state-representative samples. Recent work (3, 21) provides a model for survey or interview protocols; using common questions across studies and sites would allow for comparisons along key dimensions of interest. We know especially little about how teachers choose and implement supplementary curriculum materials, though we know that nearly all teachers now use these materials at least to some degree (21, 22). Finally, the question most of interest to policymakers is undoubtedly which textbooks have the greatest impact on student learning. The quasi-experimental methods we used in the above-mentioned research can certainly be applied in other states, grades, and subjects, and researchers should pursue this. Experiments of curricular efficacy (e.g., 7) can also help us to build an evidence base. Supplementary materials are more challenging to study, but it may be possible to use either experimental designs or big data techniques to understand the effects of individual supplementary materials. CONCLUSION The standard textbook continues to play a prominent role in the large majority of American math and science class- rooms. Despite the ubiquitous use of traditional materials, we know very little about how they are chosen, used, and to what effect. Our work is a first attempt at pushing on these questions in a handful of states; it is our hope that it will encourage other researchers to delve more deeply into these issues of great importance to students’ development of core cognitive skills in the United States. ‒6‒
REFERENCES 1. A.C. Porter, Measuring the content of instruction: Uses in research and practice. Educational Researcher, 31(7), 3-14 (2002). 2. Common Core State Standards. (n.d.). Frequently asked questions: Overview: Why are the Common Core Standards important? Retrieved from http://www.corestandards.org/about-the-standards/frequently-asked-questions/ 3. T.K. Kane, A.M. Owens, W.H. Marinell, D.R.C. Thal, D.O. Staiger, “Teaching higher: Educators’ perspectives on Common Core implementation” (Center for Education Policy Research, Harvard University, 2016). 4. G.N. Cervetti, J.M. Kulikowich, M.A. Bravo, The effects of educative curriculum materials on teachers’ use of instructional strategies for English language learners in science and on student learning. Contemporary Educational Psychology 40, 86-89 (2014). 5. D.J. Freeman, A.C. Porter, Do textbooks dictate the content of mathematics instruction in elementary schools? American Educational Research Journal 26(30), 403-421 (1989). 6. J.T. Remillard, Can curriculum materials support teachers’ learning? Two fourth-grade teachers use of a new mathematics text. The Elementary School Journal 100(4), 331-350 (2000). 7. R. Agodini, B. Harris, An experimental evaluation of four elementary school math curricula. Journal of Research on Educational Effectiveness 3(3), 199-253 (2010). 8. R. Bhatt, C. Koedel, Large-scale evaluations of curricular effectiveness: The case of elementary mathematics in Indiana. Educational Evaluation and Policy Analysis 34(4), 391-412 (20120. 9. G.D. Borman, N.M. Dowling, C. Schneck, A multisite cluster randomized field Trail of Open Court Reading. Educational Evaluation and Policy Analysis 30(4), 389-407 (2008). 10. C. Koedel, M. Polikoff, Big bang for just a few bucks: The impact of math textbooks in California. Economic Studies at Brookings Evidence Speaks Reports 2(5) (2017). 11. P. Morgan, S. Ritter, An experimental study of the effects on Cognitive Tutor® Algebra I on student knowledge and attitude. (Carnegie Learning Inc., Pittsburgh, PA, 2002). 12. C. Koedel, D. Li, M.S. Polikoff, T. Hardaway, S.L. Wrabel, Mathematics curriculum effects on student achievement in California. AERA Open 3(1), 1-22 (2017). 13. M.M. Chingos, G.J. Whitehurst, “Choosing blindly: Instructional materials, teacher effectiveness, and the Common Core” (Brown Center on Education Policy at Brookings, 2012). 14. Texas Education Agency. (n.d.). Instructional Materials Reports. Retrieved from https://tea.texas.gov/Academics/ Instructional_Materials/Instructional_Materials_Allotment/Instructional_Materials_Reports/ 15. Eliezer Williams, et al., vs. State of California, et al. (2000).Superior Court of the State of California, County of San Francisco. 16. Q.T. Le, M.S. Polikoff, The effects of English language development curriculum materials on students’ English proficiency (2019). 17. M.S. Polikoff, S. Campbell, S. Rabovsky, C. Koedel, Q. T. Le, T. Hardaway, H. Gasparian, The formalized processes districts use to evaluate mathematics textbooks. Journal of Curriculum Studies (2020). Advance online publication. doi:10.1080/00220272.2020.1747116. 18. M.S. Polikoff, How well aligned are textbooks to the Common Core Standards in mathematics? American Educational Research Journal 52(6), 1185-1211 (2015). 19. C. Gewertz, States shedding power to adopt class materials. Education Week, 34(21), 1, 10 (2015). 20. Louisiana Department of Education, “Online instructional materials reviews” (www.louisianabelieves.com/ academics/ONLINE-INSTRUCTIONAL-MATERIALS-REVIEWS) 21. V.D. Opfer, J.H. Kaufman, L.E. Thompson, “Implementation of K-12 state standards for mathematics and English language arts and literacy: Findings from the American Teacher Panel” (RAND Corporation, 2017) 22. J.H. Kaufman, J.S. Davis, E.L. Wang, L.E. Thompson, J.D. Pane, K. Pfrommer, M. Harris, “Use of open educational resources in an era of common standards” (RAND Corporation, 2017). 23. D. Blazar, B. Heller, T. J. Kane, M. Polikoff, D. Staiger, S. Carrell, D. Goldhaber, D. Harris, R. Hitch, K. L. Holden, M. Kurlaender, “Learning by the book: Comparing math achievement growth by textbook in six Common Core states” (Center for Education Policy Research, 2019). ‒7‒
The mission of the Center on Education Policy, Equity and Governance is to conduct rigorous, practice- relevant research that advances educational equity for California K-12 students—with an emphasis on greater Los Angeles—and to create partnerships that ensure education policies, practices, and governance structures are guided by evidence and careful analysis. Center on Education Policy, Equity and Governance cepeg.usc.edu @usc_cepeg
You can also read