xml version 1.0 encoding UTF8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchemainstance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20069999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E1100487
0 245
Educational policy analysis archives.
n Vol. 14, no. 14 (June 01, 2006).
260
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c June 01, 2006
505
Examining instruction, achievement, and equity with NAEP mathematics data / Sarah Theule Lubienski.
650
Education
x Research
v Periodicals.
2 710
Arizona State University.
University of South Florida.
1 773
t Education Policy Analysis Archives (EPAA)
4 856
u http://digital.lib.usf.edu/?e11.487
xml version 1.0 encoding UTF8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchemainstance xsi:schemaLocation http:www.loc.govmodsv3mods31.xsd
mods:relatedItem type host
mods:identifier issn 10682341mods:part
mods:detail volume mods:number 14issue 14series Year mods:caption 20062006Month June6Day 11mods:originInfo mods:dateIssued iso8601 20060601
PAGE 1
Readers are free to copy display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for noncommercial purposes only, and no alte ration or transformation is made in the work. More details of this Creative Commons license are available at http://creativecommons.org/licen ses/byncnd/2.5/. All other uses must be approved by the author(s) or EPAA EPAA is published jointly by the Mary Lou Fulton College of Education at Arizona State Universi ty and the College of Educ ation at the University of South Florida. Articles are indexed by H.W. Wilson & Co. Please contribute commentary at http://epaa.info/wordpress/ and send errata notes to Sherman Dorn (epaaeditor@shermandorn.com). EDUCATION POLICY ANALYSIS ARCHIVES A peerreviewed scholarly journal Editor: Sherman Dorn College of Education University of South Florida Volume 14 Numb er 14 June 1, 200 6 ISSN 1068 Examining Instruction, Ac hievement, and Equity with NAEP Mathematics Data 1 Sarah Theule Lubienski University of Illinois at UrbanaChampaign Citation: Lubienski, S. T. (2006). Examining instru ction, achievement, and equity with NAEP mathematics data. Education Policy Analysis Archives, 14 (14). Retrieved [date] from http://epaa.asu.edu/epaa/v14n14/. Abstract The purpose of this article is twofold First, it reports on a study of the distribution of reformori ented instructional practices among Black, White and Hispanic students, and the relationship between those practices and student achievement. The study identified many similarities in instruction across student groups, but there were some differences, such as Black and Hispanic students being assessed with multiplechoice tests significantly more often than were White students. Using hierarchical linear modeling, this study id entified several significant positiveand no negativer elationships betwee n reformoriented practices and 4thgrade student achievement. Specifica lly, teacher emphas is on nonnumber mathematics strands, collabor ative problem solving, and teacher knowledge of the NCTM Standards were positive predictors of achievement. An analysis of interaction effects indicated that the re lationships between various instructional 1 This project was funded through a National Assessment of Education al Progress Secondary Analysis Grant from the National Center for Education Statistics, Institute of Education Sciences. The author would like to thank Eric Camburn, Mack Shelley, Jay Verkuilen, Lateefah IdDeen, Megan Brown and Chris Lubienski for their helpful advice during various stages of this work. Only the author is responsible for the analysis and interpretations presented in this report.
PAGE 2
Education Policy Analysis Archives Vol. 14 No. 14 2 practices and achievement were roughly similar for White, Black and Hispanic students. The second purpose of this article is to make comparisons with another study that used the same NAEP data, bu t drew very different conc lusions about the potential for particular instructional practices to alleviate inequities. A study published in EPAA by Wenglinsky (2004) concluded th at school personnel can eliminate racerelated gaps within their schools by changing their instructional practices. Similarities and differences between these two studies are discussed to illuminate how a researchers framing, methods, an d interpretations can heavily influence a studys conclusions. Ultimately, this articl e argues that the primary conclusion of Wenglinskys study is unwarranted. Keywords: equity, hierarchical linear modeling; ma thematics achievement; mathematics instruction; NAEP. Examinando instruccin, logros, y equidad usando resultados de matemticas de NAEP Resumen El propsito de este artculo es doble. Primero, reportar los resultados de un estudio sobre la distribucin de prcti cas orientadas para producir reformas educativas entre estudiantes negros, blancos e hispnicos y el lazo entre esas prcticas y los logros de los estudiantes. Este estudio identific muchas semejanzas en la instruccin de los grupos de estudi antes, pero tambin algunas diferencias. Por ejemplo los estudiantes negros e hi spnicos son evalua dos con pruebas de opcinmltiple considerablemente ms a menudo que los estu diantes blancos. Usando modelos lineares jerrquicos, este estudio identific varias relaciones significativas positivas y re laciones nonegativas entre prcticas orientadas para producir reformas educativas en los resul tados educativos de estudiantes de 4to. grado. Especficamente, el nfasis de los profesores en la enseanza de aspectos nonumricos en matemticas, la cooperacin en la reso lucin de problemas, y el nivel de conocimiento de los profesor es de los estndares de NCTM fueron predictores positivos del logro educativo. Un anlisis de los efectos de las interacciones indic que lo s lazos entre las diferent es prcticas y los logro educativos fueron bastante similares para los estudiantes blancos, negros e hispnicos. El segundo propsito de este artculo fue hacer comparaciones con otro estudio que utiliz los mismos datos de NAEP, pe ro obtuvo conclusiones muy diferentes acerca del potencial de las prcticas orientadas para pr oducir reformas educativas para aliviar desigualdades. Wenglinsky ( 2004), en un estudio publicado en EPAA concluy que el personal de las escuelas puede eliminar las diferencias educativas relacionadas con aspectos raciales cambiando sus prcticas educacionales. Semejanzas y diferencias entre estos dos estu dios se discuten para iluminar cmo el marco referencial, los mtodo s, y las interpretaciones de un investigador influyen sustantivamente en las conclusiones de un estudio. En ltima instancia, este
PAGE 3
Examining Instruction, Ac hievement, and Equity 3 artculo discute que la conclusin primaria del estudio de Wenglinsky no esta plenamente j ustitificada. Introduction Identifying instructional practices that both boost achievement and promote equity has been of increasing concern among educators, researchers, and policy makers recently. This article reports the results of a study that focuses on the distri bution of instructional practices advocated by the National Council of Teachers of Mathematics (NCT M), and the relationship between those practices and diverse students achievement. Recently a similar study was published in EP AA by Harold Wenglinsky (2004). Wenglinskys study is comparable to this study in many impor tant ways. Both studies utilized the 2000 National Assessment of Educational Progress (NAEP) mathemat ics data. Both studies used hierarchical linear modeling (HLM) to examine relationships between instructional practices and achievement, both overall and for particular subgroups. Additionally both studies identified positive correlations between some reformoriented instructional practices and overall student achievement. Yet, the studies began with different framingsthis with an eye toward NCTMendorsed practices, and Wenglinskys with an eye toward the Bush Admini strations No Child Left Behind (NCLB) act, which requires schools to closely monitor student achievement and reduce ra cerelated achievement gaps. These differences in framing led, in part, to differences in the particular statistical models and methods employed. There was also a difference in the care with which findings were interpreted, including the extent to which causal attributions were made. Ultimately, different conclusions were reached. Specifically, Wenglinsky concluded that by changing instructional practices, any gaps within a given school can be completely eliminated (p 17). In contrast, this studys conclusions are far less definitive and optimistic. This article begins with a report of the current study, including its NCTMbased framing, its methods, and results. The article concludes wi th a comparison of its methods, results and interpretations with those of Wenglinsky, and ultimately raises questions about Wenglinskys conclusions. Finally, issues pertaining to the analys is and reporting of NAEP and other largescale data sets are highlighted. Background Although NAEP mathematics scores have genera lly risen over the past 15 years (Braswell, Daane & Grigg, 2003; Kloosterman & Lester, 2004) there is some debate as to whether the achievement gains occurred because of, or in spite of, reforms promoted in the NCTM Standards Although this crosssectional study cannot offer a defi nitive resolution to this debate, it does offer a birdseye view of the distribution of some reformoriented instructional methods, and their correlations with achievement for various student groups. This study is situated at the intersection of work on reformoriented instruction, mathematics achievement, and equity. Primary aspects of reformoriented mathematics instruction are outlined first, followed by brief discussions of previous work regarding reformoriented instruction and achievement, reformoriented instruction and equity, and equity and mathematics achievement. This is followed by a more specifi c discussion of NAEP data, including a description of NAEP and findings of previous examinations of NAEP data regarding ma thematics achievement, instruction, and equity.
PAGE 4
Education Policy Analysis Archives Vol. 14 No. 14 4 ReformOriented Mathematics Instruction In 1989, NCTM published the Curriculum and Evaluation Standards which, along with additional documents published subsequently (NCT M, 1991, 1995, 2000), called for mathematics instruction to be centered around students reasoning, collaborative problem solving, and mathematical communication (both verbal and written ). NCTM argued that a wider variety of tools (including manipulatives and calculators) and more meaningful forms of assessment should be employed. In addition, NCTM revised curricular goals for grades K to include greater emphasis on measurement, geometry, data analysis, probability, algebra, as well as number concepts. Finally, NCTM called for mathematical power for all studen ts, including those students previously underrepresented in mathematicsbased care ers (NCTM, 1989, 1991, 1995, 2000). ReformOriented Mathematics Inst ruction and Student Achievement The benefits of instruction aligned with the NCTM Standard sor reformoriented instruction as it is termed herehave been the subject of much debate. Evidence from schools that have used new, reformoriented curricula has generally been encouraging, with students outscoring control groups on a variety of measures and in a variety of contexts (e.g., Reys, Reys, Lapan, Holliday & Wasman, 2003; Riordan & Noyce, 2001; Schoenfeld, 2002; Senk & Thompson, 2003). However, some critics of reform have pointed to less encouraging evidence, such as the fact that scores on NAEPs longtermtrend mathemat ics test remained flat during the 1990s, after a period of growth in previous decades (Loveless & Diperna, 2000). One need only make a brief visit to websites such as Mathematicallycorrect.com or NYCHold.com to see that, despite the benefits of reform that some researchers report, much of the public is not convinced of the merits of reformoriented instruction on a broad scale. ReformOriented Mathematics Instruction and Equity Scholars have long argued that lowerSES and minority students have received more than their share of rotebased mathematics instruction (e.g., Anyon, 1981; LadsonBillings, 1997; Means & Knapp, 1991). NCTMs vision of problemcente red instruction for all students challenges the status quo and is intended to correct past inequities (NCTM, 1989, 1991, 1995, 2000). Now that the NCTM reforms are being implemented, scholars have begun to ask whether some students enter the mathematics classroom better positioned than others to learn in the ways envisioned in the Standards (Lubienski, 2000a, 2000b). Hickey, Moore, and Pellegrino (2001) found that reformoriented instruction improved lowand highSES students problem solving skills, but the same instruction increased the SESrelated gap in students performance on the concepts and estimation portion of the Iowa Test of Basic Skills However, still other studies have suggested that reformminded practices are particularly beneficial for lowerSES and minority students (e.g., Boaler, 2002; Ladson Billings, 1997; Newmann & Wehlage, 1995; Schoenfeld, 2002; Silver, Smith, & Nelson, 1995; Stiff, 1990). After analyzing national test score trends, L ee (2002) noted that BlackWhite gaps in achievement decreased du ring the 1970ss emphasis on basic skills, but increased during the 1990s, when emphasis shifted to higherorder thin king. Others have provided additional evidence of this trend (Campbell, Hombo, & Mazzeo, 2000; Jencks & Phillips, 1998). These studies raise the
PAGE 5
Examining Instruction, Ac hievement, and Equity 5 question of whether these patterns are caused by re formoriented instruction, differential access to such instruction, or other confounding variables. 2 Equity and Achievement In recent years, many researchers have struggl ed to understand the underlying causes of persistent inequities in academic achievement, especially racerelated achievement gaps. 3 Clearly, SES differences involving parent education, occupation, income, and educational resources in the home account for much of these gaps (Jencks & Ph illips, 1998; Peng, Wright & Hill, 1995). Several studies of racerelated achievement gaps have al so examined schoolrelated factors, including the roles of teachers, curricula, school funding, student motivation, and student resistance (e.g., Banks, 1988; Cook & Ludwig, 1998; Ferguson, 1998a, 1998b, 1998c; Ogbu, 1995; Steele & Aronson, 1998). Such discussions have tended to focus on the overall academic performance and experiences of minority students, as opposed to an indepth examination of achievement and instructional practices in a particular subject area, such as mathemat ics. This trend was noted by Lee (2002), who concluded his general analysis of patterns in achiev ement data by urging subject matter specialists to further examine inequities in their areas of expertise. This study does not attempt to enter into de bates about the many factors outside of schools that contribute to achievement inequities, but instead focuses on instructional variables over which educators have control. This study focuses specifically on students achievement and learning experiences in mathematics, which is a particularly important subject to consider in relation to equity because it is a key gatekeeper for entry into hi gh status occupations. Researchers in mathematics education have given some attention to racerelated gaps in mathematics achievement, but have rarely examined race and SES simultaneously (Lubienski & Bowen, 2000; Tate, 1997). By exploring the relationship between particular instructional practices and achievement utilizing hierarchical linear models that include both race and SES, this study examines the extent to which racerelated achievement gaps that persist after controlling for SES may be related to differences in students access to particular mathematics instructional practices, as measured by NAEP. The National Assessment of Educational Progress NAEP is the only nationally representative, ongoing assessment of U.S. academic achievement. NAEP measures student performance at 4th, 8th, and 12th grades in mathematics and other subject areas. NAEP also provides surv ey information from students and their teachers regarding mathematical backgrounds, beliefs, and instructional practices. Since 1990, the Main NAEP mathematics assessment has been guided by a framework based on NCTMs Curriculum and Evaluation Stan dards for School Mathematics (1989). Hence, the Main 2 The publication of the 2003 NAEP results prompted even more discussion about these issues, as both NCTM and NCLB were credited by various parties for improvemen ts in scores and decreases in achievement gaps between 2000 and 2003. However, th e 2003 data were not yet av ailable for secondary analysis at the time of this study. 3 For the sake of simplifying the text, the term race is used very loosely to mean race or ethnicity when referring to the NAEP categories of Black, White and Hispanic students. Additionally, to be consistent with NAEP data, the terms Black, White, and Hispanic are used throughout this article.
PAGE 6
Education Policy Analysis Archives Vol. 14 No. 14 6 NAEP assesses students performance on both mu ltiplechoice and constructedresponse items over the five mathematics strands emphasized by NCTM: number/operations, geometry, measurement, data analysis, and algebra/functions. Additionally, some NAEP survey questions administered to students and teachers were designed to identify the extent to which students classroom experiences are aligned with NCTMs vision for mathematics instruction. Previous NAEP Findings on the Distribution of ReformOriented Mathematics Instruction Strutchens and Silver (2000) gave detailed a ttention to racerelated disparities in 1996 NAEP data on mathematics achievement, students beliefs about mathematics, and teachers instructional practices and emphases. They found that Black and Hispanic students were at least as likely as White students to have access to manipu latives, reallife mathematics problems, and student collaboration in their mathematics classrooms. However, according to teacher reports, White eighth graders were more likely than Black or Hispanic students to receive some aspects of reformoriented instruction, such as calculator access, fewer multiplechoice assessments, and a heavy emphasis on reasoning. Students mathematical attitudes and beliefs, although shaped by a variety of factors, are linked to the instruction students receive. Strutchens and Silver (2000) reported that Black and Hispanic students were more likely than White studen ts to agree with the statements, There is only one way to solve a math problem and Learning mathematics is mostly memorizing facts. However, they cautioned that the racerelated diffe rences they reported might be due more to SES than race. More recently, Strutchens, Lubienski, McGr aw and Westbrook (2004) examined the 2000 Main NAEP mathematics data and confirmed the above 1996 findings, with the exception of differential access to teacherreported emphasis on reasoning, for which there were no longer racerelated disparities in the 2000 data. Lubienski and Shelley (2003) extended this work and found that the racerelated gaps persisted even after controllin g for SES. Additionally, in their analysis of 1996 State NAEP data, Swanson and Stevenson (2002) id entified SESrelated differences in instruction, with more affluent schools tending to utilize more re formoriented practices, as measured by a single composite of 16 variables. Overall, previous analyses of NAEP data have indicated some potentially important ways in which White, higherSES students are experiencing more of the fundamental instructional shifts called for by NCTM than less privileged students These differences are reminiscent of those discussed by Means & Knapp (1991), Anyon (198 1) and others who observed poor and minority students receiving more than their share of drillbased, computationfocused instruction. Previous NAEP Findings on ReformOriented Mathematics Instruction, Achievement and Equity The official NAEP report for the 2000 main mathematics assessment highlighted several instructionrelated variables that correlated with achievement. For example 8th graders with unrestricted access to calculators scored significantl y higher than did their peers without such access. Similarly, the report stated that 4th, 8th, and 12th grade students who agreed with the statement, Learning math is mostly memorizing facts, sc ored significantly lower than did students who disagreed with the statement (Braswell, Lutkis Grigg, Santapau, TayLim, & Johnson, 2001). However, given that White, highSES students have been more likely than their less privileged counterparts to have unrestricted calculator access an d to believe that mathematics is more than just fact memorization (Lubienski, 2002; Strutchens et al., 2004), race and SES are likely confounding variables in the correlations noted by Braswell et al. (2001).
PAGE 7
Examining Instruction, Ac hievement, and Equity 7 Hence, the question remains whether reformoriented instructional practices, as reported in NAEP teacher surveys, are positiv e predictors of achievement after controlling for confounding variables. If so, then the differences in instructi onal practices noted above might contribute to raceand SESrelated achievement differences. A prior study by Raudenbush, Fotiu, and Che ong (1998), utilizing 1992 State NAEP data, found that teacherreported emphasis on reasoning in mathematics instruction correlated positively with achievement even after controlling for race an d SES, and that White, highSES students were more likely to have such a teacher. However, dispar ities in students access to teachers emphasizing reasoning were no longer significant in the 2000 NAEP data. Finally, as noted previously, at the same time this study was being conducted, Wenglinsky (2004) used HLM to analyze the 2000 NAEP Math ematics Data, examining whether particular instructional practices related to schools overall ac hievement and the size of their racerelated gaps. He found that teacherreported time on task, use of routine exercises and a geometry emphasis correlated with higher achievement for students, in general, whereas frequent testing, emphasis on facts, and project work correlated negatively with ac hievement. He also concluded that an emphasis on measurement was the most beneficial practice (p. 16) for Black students, while an emphasis on data analysis appeared beneficial for Hispanic students. However, as will be discussed in more detail later, his conclusions require further consideration. The 2000 Main NAEP data included larger sa mples than previous administrations, and also included dozens of teacherreported variables rela ting to reformoriented instruction (many of which were deleted in 2003). An indepth analysis of instruction and achievement using the 2000 data can illuminate relationships among reformoriented instructional measures, student achievement, and equity. Still, given that NAEP data are crosss ectional and not longitudinal, no NAEPbased study can definitively determine which instructional method s are most effective for particular groups of students. Still, the scope and representative nature of NAEP data can lend important evidence to inform current debates and to point towa rd areas in need of further research. Research Questions In the context of the NCTM reform movement and concerns about its impact on mathematics achievement and equity, this study addr esses three questions. First, the study examines the extent to which reformoriented instructional pr actices are reaching all students, regardless of race. Second, the study investigates whether pa rticular reformoriented instructional practices correlate positively or negatively with mathematics achievement, af ter controlling for race, SES, and other potentially confounding variables. Finally the study considers whether reformoriented practices correlate similarly with achievement for diverse student groups, regardless of student race or SES. Taken together, these questions probe whether inequities in access to reformoriented instruction might contribute to achievement gaps, with a particular focus on BlackWhite and HispanicWhite gaps that persist even after c ontrolling for studentand schoollevel SES. Identifying inequities in access to instructional methods that correlate positively or negatively with achievement can shed light on variables potentia lly underlying achievement gaps, enrich our understanding of students experiences with learni ng mathematics, and suggest important areas for further study. While not assuming that instructionrelated variables are the only, or even primary, cause of achievement gaps, it is important to gi ve attention to the area that educators are best positioned to address.
PAGE 8
Education Policy Analysis Archives Vol. 14 No. 14 8 Method Several methodological features of this st udy merit discussion including the NAEP samples used, special challenges of NAEP analyses, th e specific variables utilized, and the analyses conducted. The Samples The 2000 Main NAEP data used in this study were accessed from a restricteduse CDROM. 4 Data regarding the mathematics achievement of a nationally representative sample of 13,511 4th graders who were assessed in late winter/early spring, 2000, were included, as well as data from student background surveys and teacher reports of instructional practices. The unweighted sample of students was 64% White, 17% Hispanic, 13% Blac k, and 6% other groups. The analyses reported here were part of a larger study that gave attention to both 4th and 8th grades, and that examined instructionrelated variables as reported by both students and teachers. 5 For the sake of space limitations and comparability with Wenglinskys stud y, analyses of fourth grade achievement and teacherreported data are the primary focus here. However, additional findings from the larger study are footnoted when particularly relevant. Methodological Challenges of NAEP Data Analyses Several features complicate the analysis of NAEP data. To obtain a representative sample of students, schools are stratified based on urbanicity, minority population, size, and area income, and then schools within each stratum are selected at ra ndom. Finally, students are selected randomly within schools. Deliberate oversampling of certain strata, such as schools with high enrollments of minority students, results in more reliable estimates for the oversampled subgroups, and then student and school weights are used to adjust for both unequal probabilities of selection and nonresponse. To account for the clustered sampling, NAEP data also contains replicate weights for each student and school, which are used in calculating sampling errors using the jackknife repeated replication method. Teacher weights are not assign ed, because NAEP selects samples of students and then surveys their teachers; teacher data are linked to student data and are interpreted at the student level. As a concrete example, NAEP an alyses would not indicate that 80% of teachers reported allowing unrestricted calculator use, but that 80% of students had teachers who reported allowing unrestricted calculator use. To reduce the testtaking burden on individual students, NAEP administers only a subset of items to each student. Hence, individual students achievement is not measured reliably enough to be assigned a single score. Instead, using Item Response Theory (IRT), NAEP estimates a distribution of plausible values for each student s proficiency, based on the students responses to administered items and other student character istics. When analyzing NAEP achievement data, separate analyses are conducted with the five plausibl e values assigned to each student. The five sets of results are then synthesized, following Rubin (1 987) on the analysis of multiplyimputed data. For 4 Researchers who apply for a license from the National Center for Education Statistics may obtain restricteduse data. 5 A detailed (120page) report of the methods and results of the full study was submitted to the National Center of Educ ation Statistics (Lubienski, Camburn & Shelley, 2004). Interested readers may download the report from www.ed.uiuc.edu/naep.
PAGE 9
Examining Instruction, Ac hievement, and Equity 9 more detailed information regarding the structure of NAEP data, see Johnson (1992) and Johnson and Rust (1992). Demographic and InstructionRelated Variables Several studentand schoollevel demographic variables were included in this analysis, along with teacherreported variables pertaining to instructional practices: 6 Student race. Binary Black and Hispanic variab les were created from NAEPs student race/ethnicity variable (taken from student selfreports, or school records when necessary). School race There was no school race variable in the 2000 NAEP mathematics data set. As a proxy, the percentage of White/Asian students in each schools sample was calculated. 7 Student SES. After consideration of the muchdebated meanings of socioeconomic status and social class (e.g., Duberman, 1976; Weis, 19 88), a comprehensive SES variable was created using factor analysis. Six variables were combined to produce a new student SES variable: types of reading material in students homes (newspapers magazines, books, and encyclopedia), computer and Internet access at home, extent to which studie s are discussed at home, and eligibility for school lunch and Title 1 (a federal program for disadvanta ged students). Parent education levels were not reported for 4th graders in 2000 and were therefore not included. The final variable was standardized with a mean of 0 and standard deviation of 1. School SES At each sampled school, an administra tor provided survey data regarding the percentage of students qualifying for Title 1 funds and free/reduced lunch. These two variables were ordinal with rough categories of percentages (e.g., 0, 11, etc.). The final school SES measure was a composite of the studentlevel SES variable aggregated to the school level and the percentage of students eligible for free/reduced lunch and Title 1. 8 Gender. NAEPs Gender variable (coded as boy = 1, else = 0) was included in the analyses because prior research suggests that gender correlates significantly with mathematics achievement (e.g., Fennema, Carpenter, Jacobs, Franke & Levi, 1998; Lubienski, McGraw & Strutchens, 2004). 9 Disability. Given that students with disabilities ten d to score lower than others on NAEP (Foegen, 2004) and that these students could be sub ject to different instructional practices than their peers, a binary student disability variable was used to control for whether students have a nonorthopedic disability (e.g., learning disability, visual impairment, behavioral disorder). 6 Teacher background/certification variables were also considered, including undergraduate major and whether teachers held masters degrees. These variables were ultima tely omitted from the fourthgrade analyses due to a lack of significance. However, in the 8thgrade analyses, secondary mathematics certification was significant. 7 White and Asian students were combined because these groups tend to have higher achievement than other groups. The resulting variable was skewed and somewhat bimodal (revealing school segregation patterns). A natural logarithmic transformation was used to create a somewhat more normally distributed variable, which was then standardized with mean = 0 and standard deviation = 1. Although a more normally distributed variable was desirable for inclusion in the models, the tradeoff in using such a transformation is that the resulting variable is more difficult to interpret. 8 The final composite was standardized with a mean of 0 and standard deviation of 1. For additional details about the creation of this and other demographi c indicators, see Lubienski, Camburn & Shelley, 2004. 9 NAEPs teacherreported data generally does not vary by gender because each teacher survey is linked to all students selected from his/her class. However, th e larger study also included studentreported instructionrelated data, which can vary by gender. For the sake of consis tency, gender was included in all models.
PAGE 10
Education P olicy Analysis Archives Vol. 14 No. 14 10 School sector Public/private school status has been found to correlate with achievement (e.g., Bryk, Lee & Holland, 1993; Lubienski & Lu bienski, 2006) and might also relate to the instructional practices employed. The NAEP variab le schtype was recoded, with Catholic and other private schools = 1 and public schools = 0. Teacherreported instructi onrelated variables. During initial explorations of teacherreported variables that could conceivably be viewed as me asuring some aspect of reformoriented practices, the net was cast widely to include 31 such variable s. Factor analysis was used to create composites of highly correlated instructionrelated variables, thereby reducing the number of predictors included in the HLM analyses and decreasing the danger of fishing for correlations among multiple variables. 10 Several variables did not seem to fit with the others and were excluded because upon further consideration it did not make conceptual or st atistical sense to include them. These variables included the frequency with which students t ook tests, did problems from textbooks, and used computers in mathematics class. Although these variables could be construed to relate to reformoriented instruction, closer inspection of the conten t of the questions combined with their lack of correlation with other reform measures suggested th at these variables were not essential measures of reformoriented instruction. Ultimately, 24 variables remained, with most clustering around six themes: calculator use, facts and skills, collaborative problem solving, nonnumber curricular emphasis, writing about mathematics, and manipulative use. 11 Teacher emphasis on reasoning, use of multiple choice assessments, and teachers knowledge of the NCTM Standards tended to correlate loosely with the other variables, but did not associate strongly with any single factor or with each othe r. These variables were included among the final set of instructionrelated measures, but were treated individually. Six factor analyses were conductedone with each of the 6 clusters of variablesto create a single, standardized factor (with a mean of 0 and standard deviation of 1) representing each theme. In each case, only one factor resulted with an Eigenvalue greater than 1, so that factor was used to represent the cluster. The loadings of each of the original variables on the final resulting factors are listed in Table 1, along with Cronbachs alpha, an indicator of how closely the items correlate with one another. 12 10 The KaiserMeyerOlkin measure of sampling adequa cy (KMO) with all of the variables in a single factor analysis was roughly .8, indicating that factor analysis was appropriate. 11 Variables clustered similarly in the 8thgrade fa ctor analysis, providing some evidence that the NAEP survey items are capturing some meaningful differences in teacher instruction, despite the rather rough response categories used and th e selfreported nature of the data. 12 When designing a survey and developing item clusters, Cronbachs Alphas between .7 and .9 are desirable. However, lower values were considered acceptable for the purposes of this study, in which the goal was not to design a new survey, but to create composit es of existing survey items that capture various aspects of reformoriented instruction. Conceptual connections among items (e.g., whether items refer to calculators or manipulatives) and a desire for consistency in the created composites across 4th and 8th grade were also considered in the development of the composites.
PAGE 11
Examining Instruction, Ac hievement, and Equity 11 Table 1 TeacherReported InstructionRelated FactorsQuestions with Loadings Questions Grouped by Fa ctors Response Options (Alpha) or Loading 1) Calculators (.44) How often do the students in this class use a calculator? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .72 Do you permit students in this class to use calculators for tests? Yes, No .79 Do you permit students in this class unrestricted use of calculators? Yes, No .71 2) Emphasis on Facts & Skills (.62) How much emphasis did you or will you give to learning mathematics facts and concepts? Heavy emphasis, Moderate emphasis, Little/no emphasis .83 How much emphasis did you or will you give to learning skills and procedures needed to solve routine problems? Heavy emphasis, Moderate emphasis, Little/no emphasis .74 How much emphasis did you or w ill you give to number and operations? Heavy emphasis, Moderate emphasis, Little/no emphasis .70 3) Collaborative Problem Solving (.76) How often do the students in th is class discuss solutions to mathematics problems with other students? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .84 How often do the students in this class work and discuss mathematics problems that reflect real life situations? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .74 How often do the students in this class talk to the class about their mathematics work? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .72 How often do the students in this class solve mathematics problems in small groups or with a partner? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .65 How much emphasis did you or will you give to learning how to communicate ideas in ma thematics effectively? Heavy emphasis, Moderate emphasis, Little/no emphasis .62 4) NonNumber Curricular Emphasis (.68) How much emphasis did you or will you give geometry? Heavy emphasis, Moderate emphasis, Little/no emphasis .78 How much emphasis did you or will you give measurement? Heavy emphasis, Moderate emphasis, Little/no emphasis .65
PAGE 12
olicy Analysis Archives Vol. 14 No. 14 12 ons Grouped by Fa ctors Response Options (Alpha) or Loading How much emphasis did you or will you give data analysis, statistics, and probability? Heavy emphasis, Moderate emphasis, Little/no emphasis .76 How much emphasis did you or will you give Algebra and functions (informal introduction of concepts)? Heavy emphasis, Moderate emphasis, Little/no emphasis .69 5) Writing About Mathematics (.69) How often do you use short (e.g., a phrase or sent ence) or long (e.g., several sentences or paragra phs) written responses to assess student progress in mathematics? 1 times a week, 1 times a month, 12 times a year, Never/hardly ever .76 How often do you use individual or group projects or presentations to assess studen t progress in mathematics? 1 times a week, 1 times a month, 12 times a year, Never/hardly ever .70 How often do the students in th is class write a few sentences about how to solve a mathematics problem? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .76 How often do the students in th is class write reports or do mathematics projects? 1 times a week, 1 times a month, 12 times a year, Never/hardly ever .67 6) Manipulatives (.66) How often do the students in th is class work with objects like rulers? Almost every day, 1 times a week, 12 times a month, Never/hardly ever N/A* How often do the students in th is class work with counting blocks or geometric shapes? Almost every day, 1 times a week, 12 times a month, Never/hardly ever N/A* 7) Reasoning How much emphasis did you or w ill you give to developing reasoning and analyt ic ability to solve unique problems? Heavy emphasis, Moderate emphasis, Little/no emphasis N/A* 8) MultipleChoice Assessment How often do you use multiplechoice tests to assess student progress in ma thematics? 1 times a week, 1 times a month, 12 times a year, Never/hardly ever N/A* 9) Knowledge of NCTM Standards How knowledgeable are you about the NCTM Curriculum and Evaluation Standards fo r School Mathematics ? Very knowledgeable, knowledgeable, Somewhat knowledgeable, Little/no knowledge N/A* *Loadings not relevant because variable was not part of a composite of three or more variables. Education PQuesti
PAGE 13
Examining Instruction, Ac hievement, and Equity 13 Given that the goal was to use these variables as predictors of achievement in HLM models, it was preferable for them to be either continuous, normal variables or binary. The single, isolated variables (teacher emphasis on reasoning, multiplechoice assessment use, knowledge of the NCTM Standards ) were ordinal, not continuous. A few of the variables created through a combination of factors were heavily skewed or bimodal. These issue s were addressed by creating binary variables as follows: Calculators (which was a bimodal composite) was reco ded so that above average calculator use = 1, else = 0. Facts and skills was recoded so that heavy emphasis on facts and concepts, skills for routine problems, and number/operations = 0, otherwise 1. Reasoning was recoded so that heavy emphasis = 1, moderate or light emphasis = 0 Multiple choice assessment use was originally an ordinal variable with 4 categories. There seemed to be substantial differences between tea chers using multiple choice assessments weekly, monthly, and annually/never, so two binary vari ables were created: weekly = 1 and less than weekly = 0, and once or twice annually or never = 1, othe rwise 0. This effectively separates the weekly, monthly, and annually/never groups. Knowledge of NCTM Standards originally had four categories: Very knowledgeable, knowledgeable, somewhat knowledgeable, and little/no knowledge. Two binary variables were created: 1 = very knowledgeable about the Standards otherwise = 0; and 1 = little or no knowledge about the Standards otherwise = 0. This set distinguishes between the two extremes and combines the two middle categories. The remaining continuous variables, collaborat ive problem solving, nonnumber curricular emphasis, writing about mathematics, and manipula tives were standardized with a mean of 0 and standard deviation of 1. With the exception of weekly multiplechoice assessment use and little or no knowledge of the NCTM Standards each variable was coded so that a higher number indicated a greater alignment with the NCTM Standards. Data Analysis The initial, descriptive phase of data analys is addressed the first research question: Are reformoriented instructional practices reaching a ll students, regardless of race? HLM models were then developed to answer the second and th ird research questions: Which reformoriented instructional practices correlate positively or ne gatively with mathematics achievement after controlling for confounding variables? Do those correlations vary by student race and/or SES? Phase 1Descriptive analyses of instruction by race. Means of the newly created instructionrelated variables were compared for White, Black and Hispanic students to examine whether differences emerged for the instructional composites created for this study. These comparisons were made using AM Statistical Softwa re, designed by the American Institutes for Research to handle the special weighting and jack knifing needs of complex data sets such as NAEP. Twotailed Ttests were used to determine if means significantly differed between White and Black students and between White and Hispanic students. When interpreting results, issues of multiple comparisons were considered using Bonferroni corrections. Phase 2HLM analyses of in struction and achievement. Because of the nested nature of the data (students and teachers within schools) twolevel HLMs were used to examine whether particular reformbased practices positively or ne gatively predicted achievement while controlling for potentially confounding variables at both the studen t and school level. HLM statistical software was designed specifically to accommodate multilevel datasets, including those with plausible values
PAGE 14
Education Policy Analysis Archives Vol. 14 No. 14 14 (Raudenbush & Bryk, 2002). As HLM computes st atistics related to NAEP achievement, each parameter is estimated for each of the five pl ausible values, and the five estimates are then averaged. 13 In the HLM models, students (level 1) were nested within schools (level 2). 14 The level of classroom or teacher was not included as a sepa rate level because NAEP uses random samples of students and not teachers, and there were no teacher codes in the data to allow for analysis at the teacher level. Given these constraints, and given the studys primary focus on studentlevel disparities in instruction and achievement (as oppose d to schoollevel issues), teacherlevel data were treated as level1 data. In this way, the instru ctional practices linked with students were those they had experienced during that school year. It is im portant to note that, given the lack of a prior achievement measure in NAEP, this study does not examine the change in students achievement during the school year. Therefore, it is very po ssible that relationships between instruction and achievement that appear weak or insignificant in this study could be found to be stronger in longitudinal studies. Because of concerns about collinearity among the 9 teacherreported instructional practice variables, separate HLM analyses were conducted with each of the variables to determine the relationship of each with student achievement. Th e studentand schoollevel demographic variables described above were also included as predictors in the models. Given the focus on general relationships between NCTM practices and achievemen t, as opposed to variation in their slopes by school, slopes were fixed in the HLM models, and continuous predictors were centered around their overall means. Binary predictors were not centered. The changes in coefficients for Black and Hispanic students that occurred after adding each instructional variable to the model were examined in an attempt to gauge the possible impact of each instructional practice on the racerelated achievement gaps that persisted after controlling for SES. This change in coefficients was examined separately with HLM models for each of the 9 instructionrelated variables, and interaction effects were included in the final models to examine whether the coefficient for each instructionrelated variable differed by student race and SES. Finally a larger HLM model was created to examine the change in coefficients and variance when the 9 instructionrelated variables were included simultaneously, yet this model was interpreted cautiously because of collinearity among the 9 instructionrelated predictors. Results To help the reader interpret the results di scussed here, some information about NAEP scores is necessary. NAEP uses a 500point scale on which 4th graders scored an average of 228 in 2000. The fourthgrade HispanicWhite gap was 24 points, and the BlackWhite gap was 31 points. The standard deviation for the 2000 fourthgrade sc ale scores was 31 points. Hence, a difference of 3 points can be considered an effect size of ro ughly 0.1. Note that the size of the BlackWhite 13 A more detailed explanation of the data analysis methods that the HLM program uses is available in Bryk and Raudenbush (1992) an d Raudenbush and Bryk (2002). 14 Due to missing survey data, HLM samples were reduced to include 9,999 students across 611 schools. The demographics for these reduced samples we re very similar to the de mographics of the entire data set, lessening concerns about the results of the analyses being skewed due to missing data.
PAGE 15
Examining Instruction, Ac hievement, and Equity 15 fourthgrade gap was a full standard deviation (ver y large effect size of 1), and the HispanicWhite gap had an effect size of 0.8. 15 Table 2 Means of Grade 4 TeacherReported Instruc tionRelated Factors by Student Race White Black Hispanic Factor Mean S.E. Mean S.E. Mean S.E. Greater than Average Calculator Use = 1 (binary) .35 .03 .37 .04 .27* .03 Not Heavy Emphasis on Facts and Skills = 1 (binary) .22 .02 .23 .02 .22 .02 Collaborative Problem Solving .02 .05 .06 .06 .03 .06 Nonnumber Curricular Emphasis .06 .06 .16* .09 .11* .07 Writing About Math .04 .06 .12 .07 .06 .06 Manipulatives .05 .06 .12* .06 .09 .07 Heavy Emphasis on Reasoning = 1 (binary) .61 .02 .65 .03 .58 .03 Weekly Multiple Choice Assessment Use = 1 (binary) .09 .01 .23*** .03 .22*** .02 Yearly or Never Multiple Choice Assessment Use = 1 (binary) .44 .03 .30*** .03 .32*** .03 Very Knowledg eable About NCTM Standards = 1 (binary) .07 .01 .04 .01 .06 .01 Little/No Knowledge About NCTM Standards = 1 (binary) .35 .03 .36 .02 .41 .03 Note: Means for Black and Hispanic students were compared with means of White students in the significance tests. p < .05; ** p < .01; *** p < .001 Means by Race of InstructionRelated Variables The means and standard errors for the teacherreported instructional composites for White, Black and Hispanic fourth graders are presented in Table 2. 16 There were no significant racerelated differences in teacher emphasis on reasoning an d facts/skills, teacher knowledge of the NCTM Standards collaborative problem solving, and writin g about mathematics. Black and Hispanic 15 Although there have been various methods proposed for calculating effect sizes (Thompson, 2002), this discussion refers to Cohens (1988) d, which is computed by dividing the difference between the means of two samples by the standard deviation of the combined population sample. 16 In the full study, 28 ttests were used to co mpare WhiteBlack and WhiteHispanic means for 14 teacherand studentreported instructio nrelated variables. Hence, it can be considered appropriate to hold p < .05/28 = .002 as the standard for determining statisti cal significance, using the Bonferroni correction for multiple comparisons. However, others might argue for a different clustering of va riables for the Bonferroni correction (e.g., dividing .05 by 2, because two comparisons were made for each variable). Hence, the standard .05, .01, and .001 significance levels are report ed in the tables, leaving read ers to interpret results as they see fit.
PAGE 16
Education Policy Analysis Archives Vol. 14 No. 14 16 students were at least as likely as White students to have access to manipulatives and a nonnumber curricular emphasis. For example, whereas White st udents were 0.05 standard deviations below the mean for use of manipulatives, Black students were 0.12 standard deviations above the mean. Hence, Black students actually appeared to be g etting slightly more access to some reformoriented practices than were their White peers (with means fo r Hispanic students generally in between those for Black and White students). However, consistent wi th previous findings (Strutchens, et al., 2004), Black and Hispanic students were significantly more likely to be assessed with multiple choice tests than were White students. For example, 44% of White students were assessed with multiple choice tests no more than once or twice a year, whereas th is percentage was 30% for Black students and 32% for Hispanic students (because this is a bi nary variable, the means can be interpreted as percentages). 17 HLM Analyses of InstructionRelated Factors and Achievement: The Example of Calculator Access HLM analyses were undertaken to examine th e relationship between particular reformoriented instructional practices and mathematics ac hievement, as measured by NAEP. Because of concerns of multicollinearity, separate models were created for each of the 9 instructional factors. Due to space limitations, the full results of each of the HLM models are not presented here. Instead, full details of the models involving the teacherrep orted calculator composite is presented as an example, and then the main results involvin g the remaining instructionrelated factors are summarized. Table 3 presents the set of models run with the 4thgrade calculator use composite. Recall that the calculator use variable was binary, with 1 = above average and 0 = at or below average. Models 1, 2, and 3 remained constant for all grade 4 HLM analyses regardless of the instructionrelated variable in question. The base model (Mod el 1) shows that the mean achievement across all sampled schools was 230.4 points. It also indica tes that roughly one third of the variance in achievement was between schools (intraclass correlation=.34), and two thirds of the variance was among students within schools. According to mode l 2, the mean for Black students was about 23 points lower than that of their nonHispanic peer s within the same school (White/Asian students were the primary comparison group, with a mean achievement of 235) 18 whereas Hispanic students scored about 17 points lower. The addition of these two studentlevel race variables accounted for almost 40% of the variance between schools, but only about 5% of the variation within schools. In Model 3, we can see that student and school SE S, gender, disability, and school sector all significantly predicted achievement. For example, an increase in one standard deviation in SES was associated with a 7.6 point increase in achievemen t at the student level and 6.1 points at the school level. Similarly, the coefficients reported in model 3 indicate a 3.8 point advantage for males and a 30.3 point disadvantage for students with disabilit ies. Additionally, the private school students in the 17 More frequent multiplechoice te sting for Black and Hispanic students was also found at grade 8. Black and Hispanic eighth graders were also less likely to be given access to calculators by their mathematics teachers. 18 The American Indian/Alaskan Native subgroup comprises only about 2% of students. Because of their small sample size, these studen ts were not denoted by a separate variable, but were included in the general default group of nonBlack, nonHispanic students.
PAGE 17
Examining Instruction, Ac hievement, and Equity 17 sample performed a significant 4.8 points lower than the public school students sampled. 19 The coefficient for school race was insignificant (though it was close to significant, and it was significant when school SES was not included in the model). Mo del 3 also indicates that even after controlling for these other contextual variables, there are still highly significant racerelated gaps within schools of 12.5 (Hispanic) and 17.1 (Black) points. Taken together, the demographic factors in Model 3 explained 70% of the variance between schools, and 16% of the variance within schools. Table 3 HLM Models of NAEP Achievem ent by TeacherReported Calcul ator Use and Achievement, Grade 4 Variable Model 1 Model 2 Model 3 Model 4 Model 5 Level 1 (student & his/her teacher) Intercept 230.4 235.0*** 236.6*** 236.6*** 236.3*** Black 22.8*** 17.1*** 17.1*** 16.8*** Hispanic 17.3*** 12.5*** 12.5*** 12.1*** Student SES 7.6*** 7.6*** 7.1*** Boy 3.8*** 3.8*** 3.8*** Disability 30.3*** 30.3*** 30.4*** Calculator Use 0.1 0.6 Calculator x Black 0.7 Calculator x Hispanic 1.4 Calculator x SES 1.5 Level 2 (school) School SES 6.1*** 6.1*** 6.1*** School Race/Ethnicity 1.0 1.0 1.0 Private School 4.8*** 4.8*** 4.6** Random Effects Variance Component Variance Component Variance Component Variance Component Variance Component Intercept (variance between schools) 275.4 168.4 83.6 83.7 83.5 Level1 (variance within schools) 537.9 510.4 454.2 454.2 454.1 Intraclass Correlation .34 .25 .16 .16 .16 N= 9999 students and 611 schools. p < .05; ** p < .01; *** p < .001 In Model 4, we see that once all of these poten tially confounding variables are controlled, students whose teachers reported giving a higher than average amount of calculator access to students scored an insignificant 0.1 point lower on the NAEP mathematics assessment than did students with teachers reporting less calculator access in their classrooms. 20 Finally, Model 5 controls 19 Further investigation revealed that although overall achievement was substantially higher in private schools than in public schools, this relationship re versed after controlling for SES and other demographic variables. See Lubienski & Lubienski (2006) regarding a followup study. 20 Interestingly, the student reported calculator variable negatively predicted achievement at grade 4. Teacherreported calculator use and achievement were more positively related at grade 8, perhaps because advanced classes utilize calculators more often. Again, see Lubienski, Camburn an d Shelley (2004) for more information.
PAGE 18
Education Policy Analysis Archives Vol. 14 No. 14 18 for all of these factors and also includes studentlevel interaction terms to examine whether the relationship between calculator access and achiev ement differs by student race. None of the interaction terms were significant. The addition of the calculator variables in Models 4 and 5 did not help explain additional variance between or within schools. Summary of HLM Results Each of the other 9 teacherreported instructi onrelated variables were treated in a likewise manner, with the final results condensed in Table 4 (interaction terms are discussed later). Relationship between inst ruction and achievement. The HLM coefficient for each instructionrelated measure is presented in Table 4, each taken from a model equivalent to Model 4 in Table 3. With the exception of weekly mult iple choice assessment use and being not very knowledgeable about the NCTM Standards, for each variable a positive HLM coefficient indicates that a practice aligned with the NCTM Standards positively predicts achievement after controlling for studentand schoollevel demographics an d other potentially confounding variables. Table 4 HLM Coefficients of TeacherRep orted InstructionRelated Fact ors when Predicting Student Mathematics Achievement, After Cont rolling for Demogr aphic Variables Variable Coefficient Calculators 0.1 Deemphasize facts & skills 1.1 Collaborative problem solving 1.1* Nonnumber curricular emphasis 1.6** Writing about mathematics 0.2 Manipulatives 0.6 Reasoning 1.0 Multiple choice assessment use 0.9 (annual use) 0.4 (weekly use) Knowledge of NCTM standards 4.2* (very knowledgeable) 0.13 (not very knowledgeable) p < .05; ** p < .01; *** p < .001 For each of the three significant coefficients found, the direction of the relationship indicated that NCTMbased instruction and knowledge were positively related to achievement. Specifically, collaborative problem solving, teacher knowledge of the NCTM Standard s, and having a nonnumber curricular emphasis were all signif icant, positive predictors of fourthgrade achievement. The results of the larger study were more striking, in that the five teacherreported variables found to significantly predict ac hievement at grade 8 held the same pattern. 21 Reduction of racerelated gaps with teacherreported variables. By comparing the coefficients for Black and Hispanic students before each instructional practice is included in the model (see Table 3, Model 3) and their corresponding coefficien ts after each practice is added (see Model 4), 21 Collaborative problem solving and knowledge of the NCTM Standards were also significant predictors of achievement in grade 8. In addition, calculator use, an emphasis on reasoning, and a deemphasis of facts and skills were also significantly, positively re lated to 8thgrade achievement.
PAGE 19
Examining Instruction, Ac hievement, and Equity 19 one can examine the extent to which disparities in access to particular instructional practices might account for a portion of the achievement gaps. If an instructional practice correlates strongly with achievement, and if that practice is utilized much more with White students than with Black or Hispanic students, then we might see a substantial improvement in the slopes for Black and Hispanic students once we add the instructional va riable to the model. In order words, after controlling for the fact that Black and Hispanic students have less access to such instructional practices, we would see the magnitude of the Black and Hispanic coefficients decrease. However, an examination of the change in racerelated coefficients for each teacherreported instructional variable added revealed that the change was near .1 or less. Even when adding all of the instructionrelated variables together in the same model, the change in the slopes was .2 or less at both 4th and 8th grades, indicating a less than 1% change in the 17 point gaps. 22 Hence, these results indicate that the disparities in reformoriented instruction, as measured in these models by the teacherreported NAEP data, do not help explain much of the racerelated achievement gaps. Yet again, researchers might find a stronger relationship if using more sensitive measures and examining student experiences and growth over several years (Rowan, Correnti, & Miller, 2002). It is worth noting that in the fu ll study, studentreported NCTMaligned beliefs (math is not simply fact memorization and there are multip le ways to solve problems) were strong, positive predictors of achievement at both 4th and 8th gr ades. Such beliefs are formed over years of students experiences learning mathematics. Racerelated gaps slightly but significantly decreased when these and other studentreported factors were included in HLM models. Interaction effects Three interaction effects (Black, Hispanic, and SES) were examined for each of the teacherreported variables. Of these, only one interaction was significant: Nonnumber curricular emphasis had a positive interaction with SES, indicating that a nonnumber curricular emphasis correlated more positively with achiev ement for higherSES students than lowerSES students. Specifically, as shown previously in Table 4, a student of average SE S having a teacher with nonnumber emphasis one standard deviation abov e the mean, scored an average of 1.6 points higher than a student whose teacher reported an average amount of emphasis on nonnumber topics. However, given that the nonnumber X SES coefficient was 1.2, if a student were 1 standard deviation above the mean in terms of SES, that nonnumber curricular emphasis advantage would actually be 1.6 + 1.2 = 2.8 points. If a student were 2 standard deviations below the mean SES, then the coefficient would actually be 1.6 2.4= 0.8 points. 23 22 Only an additional 1% of the overall varian ce in achievement was explained when all of the instructionrelated variables were added to the demographic model. However, multicollinearity among predictors, as well as the fact that these data are not longitudinal, necessitate caution in interpreting these results. 23 There were three signific ant interactions among the student reported variables. Two of these involved SES and might be viewed as following a patt ern consistent with the teacherreported nonnumber emphasis interaction. The coefficients for studentrepo rted calculator use, and studentreported collaborative problem solving were greater for highSES students th an for lowSES students, suggesting that perhaps these aspects of instruction could further the advantages of highSES students. Howeve r, again, given the crosssectional (as opposed to longitudinal) nature of the data, and given the number of interactions tested, these findings should be viewed as merely su ggestive of issues for further study.
PAGE 20
Education Policy Analysis Archives Vol. 14 No. 14 20 Discussion ReformOriented Instruction, Achievement, And Equity This studys descriptive analyses showed rela tively few racerelated inequities in fourthgraders access to instructional practices aligned with the NCTM Standards Black and Hispanic students were actually more likely than White studen ts to have a teacher report a strong nonnumber curricular emphasis and frequent manipulative use. However, consistent with previous findings (Strutchens, et al., 2004), Black and Hispanic studen ts were significantly more likely to be regularly assessed with multiple choice tests than were White students. This studys HLM analyses determined that seve ral reformoriented factors were significantly related to student achievement after controlling for SES, race, disability status, gender, and school sector. Specifically, teachers nonnumber curricula r emphasis, use of collaborative problem solving and knowledge of the NCTM Standards were significant, positive predictors of fourth grade mathematics achievement. Although the primary focus of this article is on the grade 4 results, it is worth noting that at both fourth and eighth grades, in every case wh en a teacherreported, reformoriented instructional factor was significantly related to achievement, the relationship was positive. Additionally, studentreported beliefs aligned with the NCTM Standards were strong, positive predictors of achievement at grades 4 and 8, and such beliefs were more prevalent for White students than for Black and Hispanic students. Given these differences in beliefs, it is very possible that additional racerelated instructional disparities exist that are not ca ptured by the NAEP teacher survey items. Despite the positive relationships between re formoriented instruction and achievement identified in this study, the overall implications for ways to improve equity are less clear. The reductions in the slopes for Black and Hispanic students produced by adding the teacherreported instructional variables to the models were very small. Additionally, some instructional practices that correlated positively with achievement, such as teacherreported nonnumber curricular emphasis and collaborative problem solving, were actually more prevalent for Black and Hispanic students than for White students. Moreover, the few interaction effects that were significant in the full study suggested ways in which NCTMbased practices correlated more positively with achievement for highSES students than for lowSES students. Instead of illuminating possible causes of achievement gaps, these facts seem to only furt her complicate the search for instructionrelated causes. Overall, the NCTMbased instructional practices examined in this study related positively to achievement when they related at all. The consistency of this pattern at grades 4 and 8 (as revealed in the full study) would seem to provide encouraging news for reformers. However, this and other results of the study must be interpreted with care, as is discussed in the next section. Limitations Given the crosssectional nature of NAEP data we cannot be sure whether reformoriented practices actually caused higher achievement, or whether higherachieving students were more likely to receive reformoriented instruction. Either case raises important questions about the reasons for the relationship and its ultimate effects on sustaini ng or furthering achievement disparities. There are several additional cautions to be di scussed. First, NAEP classroom practices data are based on teacher selfreports for that school year only. The accuracy of teachers memory of
PAGE 21
Examining Instruction, Ac hievement, and Equity 21 practices utilized throughout the year and perceived pressure to portray instruction in particular ways could have affected teachers responses. Additionally the threeor fourpoint scales used on many of the teacher survey items were rough and perhaps insensitive to important differences in teacher practices. Many important questions were not aske d that might move beyond surface features of instruction (e.g., manipulative use) to probe at mo re fundamental instructional issues (e.g., the extent to which instruction builds upon and centers around student understanding), as well as to identify larger, structural inequities (e.g., school funding). Still, the fact that when put in a factor analysis, most teacherreported instructionrelated variables clumped with each other in sensible ways, and the fact that several significant relationships b etween student demographics and instructional factors were found, indicate that the NAEP mathematics teacher survey questions are, indeed, measuring some important aspects of variability in instruction. Again, there is no measure of prior achievemen t in NAEP, and so it is students overall achievement, and not growth in achievement, that serves as the outcome variable. This limitation, combined with the limits of the teacherreported data noted above, suggest that this study may be overly conservative in determining the strength of impact that instructional measures can have on both student learning and on achievement gaps. If teacher practices were measured with more sensitive measures over time, and if the data allo wed for examinations of student achievement gains, it is likely that we would see a greater instructional impact on achievement and racerelated gaps than what is indicated here (Rowan, Correnti, & Miller, 2002). On the other hand, standard errors for instructionrelated coefficients were perhaps smaller (less conservative) than they would have been if the clustering of students within classrooms was accounted for in the models (i.e., if teachers could have been treated at a classroom level). Hence, ag ain, the results of this study should be viewed as merely suggestive of relationships that are important to explore further using indepth, longitudinal methods that can take student, teacher/classroom and school levels into consideration. Comparison across Two Studies We now return to the question of how this st udy compares with that of Wenglinsky (2004), who also used HLM with the 2000 NAEP mathem atics data to examine relationships among instruction, achievement and equity. Some findin gs of the two studies were complementary. For example, this study found a positive relationshi p between teachers nonnumber curricular emphasis and achievement. Wenglinsky also obtained positive coefficients for teachers emphasis on geometry, measurement and algebra (although only the coefficient for geometry was significant). There were also consistencies in some factors that did not correlate with achievement in either study, including manipulative use and writing abou t mathematics, as well as teachers college major and degree (which were subsequently deleted from models in this study due to their lack of significance). However, vastly different conclusions were reached about the potential for particular instructional practices to close achievement gaps This study identified only weak, insignificant interactions between particular in structional practices and Black and Hispanic student achievement. Although some significant relationships between instructional practices and overall achievement were found, these relationships did not vary significantly by student race. Additionally, all relationships between instruction and achievement fo und in this study are interpreted with great caution due to the crosssectional nature of NAEP data. In contrast, Wenglinsky concluded from his stud y that a series of instructional practices, when used in concert, can substantially redu ce both the BlackWhite and LatinoWhite achievement gaps (p. 3). Specifically, Wenglinsky asserted that frequent test taking enlarges the BlackWhite gap,
PAGE 22
Education Policy Analysis Archives Vol. 14 No. 14 22 and that an emphasis on measurement helps reduce the gap. (Although not the main point of the concerns raised here, it is worth noting that the actual coefficients Wenglinsky provided appear to indicate the oppositean emphasis on measurement a ppeared to predict a la rger BlackWhite gap, while frequent testing predicted a smaller gap.) He also concluded that an emphasis on data analysis is particularly beneficial for Hispanic students. There are four fundamental differences between the two studies that underlie their divergent conclusions. First, Wenglinsky aggregated teacherlev el data to the school level, whereas this study treated teacherlevel data at the studentlevel. Again, there was no classroom or teacher identification code in the NAEP data, making it nearly impossibl e to treat classroom as a separate level in HLM. Confidentiality concerns might partially underlie NAEPs exclusion of teacher codes, but another reason is that studentsnot teachersare randomly selected within a sc hool. Therefore, NAEP experts recommend connecting teachers with indivi dual student data when making claims about teachers. This was the approach taken in this stud y, compatible with its primary focus on disparities among students classroom experiences and achiev ement. On the other hand, given Wenglinskys primary focus on NCLB and withinschool practices and achievement gaps, his decision to aggregate instructional practices to the school level is certainly conceptually defensible, having the potential to create a stronger measure of the general instructional climate of the school. The interactions between student race and instruction were then trea ted as crosslevel interactions in Wenglinskys study, which fit well with his focus on withinschool gaps (whereas in this study, the interactions were treated at the studentlevel). Overall, the difference in treatment of the teacher data when designing the HLM models (analyzing it at th e student versus the school level) may be one contributor to the differences in the studies findings. However a second and more important difference between the studies is the number of variables included simultaneously in the models. The study reported here utilized factor analysis to reduce roughly 30 instructionrelated variables to nine factors, which were then each examined in separate models. In contrast, in addition to seve ral demographic measures, Wenglinsky included 20 variables pertaining to teaching practices and 3 variables pertaining to teacher background in his model to predict the main intercept, and he also used the 23 teacherrelated variables to predict the withinschool slope (or gap) for both Black an d Hispanic students. Wenglinsky found that the slopes for Black and Hispanic students were signific ant before adding the teacherrelated variables but no longer significant after adding those variables. Wenglinsky then concluded, Thus, by including the 20 instructional practices, the second HLM can explain away the entire withinschool racial gap (p. 16). Wenglinskys full model, then, involved the dete rmination of over 70 coefficients. Fortysix of these were predicting the Black or Hispanic slope, the primary focus of his study. By chance alone we would expect roughly 4 or 5 of those 46 predi ctors to be statistically significant at the p < .1 level (the base level of significance he used), with 2 or 3 of those significant at the p < .05 level. In fact, his model identified only 3 significant predictors of the Black or Hispanic slopes (2 at the .05 level, and 1 at the .1 level). Hence, it is quite possible that these 3 variables are false positives. In fact, it is clear that Wenglinskys full model is problematic, as evidenced by inflated standard errors and the fact that the Hispanic slope went from being 8 (with a standard error of 1) in his base model, to a positive 27 points (with standard error of 22). The slope for Black students went from 16 (standard error of 1) to 9 (standard error of 26). It is worth noting that the BlackWhite gap of 9 was considered eliminated because the gap was not si gnificant, yet the standard error had became so large that even the original BlackWhite gap would not be significant. Again, the huge reversal of the Hispanic slope suggests serious instability in the model, likely caused by the large number of predictors, many of which are collinear (as evidenced by the results of the factor analyses in the study reported here).
PAGE 23
Examining Instruction, Ac hievement, and Equity 23 Hence, while the composites of instructional measures used in the study reported here can be more difficult to interpret than the individual variables included in Wenglinskys models, his full model reveals the danger of including too many variables in an HLM, particularly at the school level, as Wenglinsky, himself notes in a footnote: degrees of freedom are sharply reduced by including so many schoollevel independent variables (p. 16). Other, minor differences between the studies methods could be discussed, including the exact calculations of school race or SES measures, or the fact that some variables, such as disability or time on math were included in one study but not the other. However, the two remaining differences between the studies that merit discussion lie not in their methods of analysis but in their interpretation of the results. First, NAEP data are crosssectionalnot l ongitudinaland therefore not intended for drawing causal conclusions regarding instruction and achievement. Wenglinsky himself notes this in his brief, isolated discussion of the limitations of his study, explaining: This means that nothing is known about the causal direction of the results (pp. 16). And yet causal language and conclusions are prevalent throughout the remainder of the article, beginning with the abstract in which he states that he uses HLM with NAEP data to identify instructional practices that reduce the achievement gap. It finds that, even when taking student background into account, instructional practices can make a substantial difference. Wenglinskys optimistic conclusion that, according to his results, school administrators can succeed at closing the racial achievement gap in their schools, (p. 17) is unwarranted. Again, a correlation between particular practices and achievem ent may not be causal, particularly given that another plausible explanation existsi.e., that higher achieving students might tend to receive different instruction than lowerachieving students. And again, the large number of predictors in Wenglinskys full model should also raise major concerns about drawing conclusions from the particular relationships identified. Second, even if one could conclude from Wenglinskys stud y that particular instructional practices reduced the withinschool BlackWhite an d HispanicWhite gaps to 0, one must interpret this with the understanding that SES was controlled for in the models, and therefore it is the racerelated leftover gap (the part not related to SES) within schools that was reduced in the models. Hence, in practice, there would still be very large withinschool gaps between Black and White students in most schools, as well as between Hispanic and White students, given the strong correlation between race and SES. Additionally, th e focus of NCLB and Wenglinskys study on withinschool gaps ignores raceand SESrelated gaps between schools, which dangerously places responsibility for gap reductions on school personnel and ignores larger societal inequities, such as persistent disparities in community resources that schools alone cannot overcome (Berliner, 2005). Implications for Future Research Uses and Abuses of CrossSectional Data One can understand why researchers utilizing NAEP and other crosssectional data are tempted to overstate rather than understate the conclusions that can be drawn. Soft claims surrounded by a sea of caveats tend to be ignor ed by publishers, the popular media, and policy makers. This dynamic points toward the need for studies indicating no relationship between important variables to be reported along with thos e with more exciting conclusions. Critiques of NAEPs crosssectional nature also raise questions about the usefulness (or lack thereof) of NAEP and other similar largescale data sets (Christen sen & Angel, 2005; Lubienski & Lubienski, 2006).
PAGE 24
Education Policy Analysis Archives Vol. 14 No. 14 24 One might wonder, if causal claims cannot be made from studies of NAEP data, then what good is NAEP? In the study reported here, NAEP data were useful for examining whether reformoriented instructional practices were distributed equally acro ss U.S. students, regardless of race. This is aligned with NAEPs strengthdescribing current achievement and instructionrelated patterns on a large, nationallyrepresentative scale. Analyses of NAEP data can also shed light on which instructional practices do and do not correlate with student achievement while controlling for many potential confounding variables. However, a measure of prior achievement is not available in NAEP, and the causal order of any relationships identified will be unclear. Because of their potential for widespread attention and influence on policy, largescale studies, in particular, must be communicated with care, with proper cautions regarding the limitations of the results emphasized. In the case of the study reported here, wheth er particular NCTMbased practices caused higher achievement, or whether highachieving st udents were more likely to be taught with NCTMbased practices is unclear. Still, the relationships identified raise important questions for further research within classrooms. As Wenglinsky also notes, NAEP analyses cannot replace studies involving indepth classroom observations. Further Research on Equity After multiple reform efforts aimed at changing mathematics instruction and reducing inequities, much work remains. One finding that is clear in both Wenglinskys study and this study is that there are large raceand SESrelated achievem ent gaps, and even after controlling for SES using multiple demographic variables, the unexplained racerelated gap within schools is disturbingly large. 24 NAEP offers one avenue for examining disparit ies in achievement and classroom practices. The patterns identified in this study suggest directions for additional longitudinal and qualitative studies that examine causes of, and ways to address, the patterns identified here. Overall, researchers should continue to examine achievement disparities, considering instructional factors identified in this study, as well as other potential influences not considered here, such as differential access to various resources at both home and school. References Anyon, J. (1981). Social class and school knowledge. Curriculum Inquiry, 11 3. Banks, J. A. (1988). Ethnicity, class, cognitive, and motivational styles : Research and teaching implications. Journal of Negro Education, 57 (4), 452. Berliner, David C. (2005). Our impoverished view of educational reform. Teachers College Record, 108 949. 24 Rothstein (2004) rightly points out that the SES va riables used in studies such as these are limited because they do not measure important historical and contextual factors, such as accumulated family wealth. It is likely that a better SES measure would account fo r a greater proportion of racerelated achievement gaps. Still, the measure used in this study is more compre hensive than SES variables typically utilized in NAEP reports.
PAGE 25
Examining Instruction, Ac hievement, and Equity 25 Boaler, J. (2002). Learning from teaching: Expl oring the relationship between reform curriculum and equity. Journal for Research in Mathematics Education 33 (4), 239. Braswell, J. S., Daane, M. C., & W. S. Grigg (2003) The nation's report card: Mathematics highlights 2003 Washington, DC: U.S. Department of Education. Institute for Education Sciences, National Center for Education Statistics. Braswell, J. S., Lutkis A. D., Grigg, W. S., Santapau, S. L., TayLim, B., & Johnson, M (2001). The nations report card: Mathematics 2000 Washington DC: U.S. Department of Education, National Center for Education Statistics. Bryk, A. S., Lee, V. E., & Holland, P. B. (1993). Catholic schools and the common good Cambridge MA: Harvar d University Press. Bryk, A. S., & Rauden bush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods Newbury Park: Sage Publications. Campbell, J. R., Hombo, C. M., & J. Mazzeo (2000). NAEP 1999 trends in academic progress: Three decades of student performance, NCES 200069 Washington, DC: U.S. Department of Education. Christensen, J. & Angel, L. (2005 ). What NAEP cant tell us about charter school effectiveness. Education Week, 25(14), 35, 37. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. Cook, P. & Ludwig, J. (1998). The burden of acting White. In C. Jencks & M. Phillips (Eds.), The BlackWhite test score gap (pp. 375). Washington, DC: Brookings Institution Press. Cooper, B & Dunne, M. (2000). Assessing childrens math ematical knowledge: So cial class, sex and problemsolving. Buckingham: Open University Press, UK. Duberman, L. (1976). Social inequality: Clas s and caste in America Philadelphia, J.B. Lippincott. Fennema, E., Carpenter, T. P., Jacobs, V. R., Fr anke, M. L., & Levi, L. (1998). A longitudinal study of gender differences in youn g childrens mathematical thinking. Educational Researcher, 27 (5), 6. Ferguson, R. F. (1998a). Teache rs perceptions and expectations and the BlackWhite test score gap. In C. Jencks & M. Phillips (Eds.), The BlackWhite test score gap (pp. 273). Washington, DC: Brookings Institution Press. Ferguson, R. F. (1998b). Comment on Cook & Ludwigs The burden of acting White: Do Black adolescents disparage academic achievement? In C. Jencks & M. Phillips (Eds.), The BlackWhite test score gap (pp. 394). Washington, DC: Brookings Institution Press.
PAGE 26
Education Policy Analysis Archives Vol. 14 No. 14 26 Ferguson, R. F. (1998c). Can schools narrow the BlackWhite test score gap?. In C. Jencks & M. Phillips (Eds.), The BlackWhite test score gap (pp. 318). Washington, DC: Brookings Institution Press. Foegen, A. (2004). Students with disabiliti es and the 2000 NAEP mathematics assessment: achievement levels and sample characteristics Paper presented at the annual meeting of the American Educational Research Association, San Diego. Hickey, D. T., Moore, A. L., & Pellegrino, J. W. (2001). The motivational and academic consequences of elementary mathematics en vironments: Do constructivist innovations and reforms make a difference? American Educational Research Journal, 38 (3), 611. Jencks, C., & Phillips, M. (Eds.) (1998). The BlackWhite test score gap Washington, DC: Brookings Institution Press. Johnson, E. G. (1992). The desi gn of the National Assessment of Educational Progress. Journal of Educational Measurement, 29 (2) 95. Johnson, E. G. & Rust, K. F. (1992). Population inferences and variance estimation for NAEP data. Journal of Educational Statistics 17(2) 175. Kloosterman, P., & Lester, F. K., Jr. (Eds.) (2004). Results and interpretations of the 1990 through 2000 mathematics assessments of the Na tional Assessment of Educational Progress. Reston, VA: National Council of Teachers of Mathematics. LadsonBillings, G. (1997). It doesnt add up: African Amer ican students mathematics achievement. Journal for Research in Mathematics Education, 28 (6), 697. Lee, J. (2002). Racial and ethnic achievement gap trends: Reversing the progress toward equity? Educational Researcher, 31(1), 3. Loveless, T., & Diperna, P. (2000). How well are American studen ts learning? Focus on math achievement (Brown Center Report, Vol. 1, No 1). Washington, DC: Brookings Institution. Lubienski, C., & Lubienski, S. T. (2006). Charter, private, public schools and academic achievement: New evidence from NAEP mathematics data New York: National Center for the Study of Privatization in Education, Teachers College, Columbia University. Retrieved May 29, 2006, from http://www.ncspe.org/readrel.php?set=pub&cat=126 Lubienski, S. T. (2000a). A clash of class cultures? Students experi ences in a discussionintensive seventhgrade mathematics classroom. Elementary School Journal 100 377 403. Lubienski, S. T. (2000b). Problem solving as a means toward mathematics for all: An exploratory look through a class lens. Journal for Research in Mathematics Education 31(4), 45482.
PAGE 27
Examining Instruction, Ac hievement, and Equity 27 Lubienski, S. T. (2002). A closer look at Bl ackWhite mathematics gaps: Intersections of race and SES in NAEP achievement an d instructional practices data, Journal of Negro Education 71 (4), 2697. Lubienski, S. T., & Bowen, A. (2000). Whos counting? A survey of mathematics education research 1982. Journal for Research in Mathematics Education 31 (5), 626. Lubienski, S. T., Camburn, E. & Shelley, M. C. (2004). Reformoriented mathematics instruction, achievement, and equity: Ex aminations of race and SES in 2000 Main NAEP Data Report submitted to the National Center for Education Statistics. Lubienski, S. T. & Lubiensk i, C. (2006). What NAEP can tell us about school achievement. Education Week, 25(26), 28, 30. Lubienski, S. T., McGraw, R., & Strutchens, M. (2004). NAEP findin gs regarding gender: Mathematics achievement, student affect, and learning practices. In P. Kloosterman., & F. K. Lester, Jr. (Eds.) Results and interpretations of the 1990 through 2000 mathematics assessments of the National Assessme nt of Educational Progress (pp. 305). Reston, VA: NCTM. Lubienski, S. T. & Shelley, M. C. (2003). A closer look at U.S. ma thematics instruction and achievement: Examinations of race an d SES in a decade of NAEP data. Paper presented at the American Educational Research Association, Chicago. Means, B. & Knapp, M. S. (1991, December). Cog nitive approaches to teaching advanced skills to educationally disa dvantaged students. Phi Delta Kappan, 73, 282. National Council of Teache rs of Mathematics. (1989). Curriculum and evaluation Standards for school mathematics. Reston, VA: Author. National Council of Teache rs of Mathematics. (1991). Professional standards for teaching mathematics Reston, VA: Author. National Council of Teache rs of Mathematics. (1995). Assessment standards for school mathematics Reston, VA: Author. National Council of Teache rs of Mathematics, (2000). Principles and standards for school mathematics Reston, VA: NCTM. Newmann, F. M., & Wehl age, G. G. (1995). Successful school restructuring : A report to the public and educators. Washington, DC: Office of Educat ional Research and Improvement. Ogbu, J. U. (1995). Understanding cultural diversity and learning. In J. A. Banks & C. A. McGee Banks (Eds.), Handbook of research on multicultural education (pp. 58293). New York: Macmillan. Peng, S. S., Wright, D. & S. T. Hill (1995). Understanding racialethnic differences in secondary school science and mathematics achievement. Washington, DC: U.S. Department of Education, National Center for Education Statistics.
PAGE 28
Education Policy Analysis Archives Vol. 14 No. 14 28 Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousa nd Oaks, CA: Sage. Raudenbush, S.W., Fotiu, R., & Cheong, Y.F. (1 998). Inequality of a ccess to educational opportunity: a national report card for eighth grade math. Educational Evaluation and Policy Analysis, 20 253. Reys, R., Reys, B., Lapan, R., Holliday, G., and Wasman, D. (200 3). Assessing the impact of Standards based middle grades mathematics textbooks on student achievement. Journal for Research in Mathematics Education, 34 (1), 74. Riordan, J. E., and Noyce, P. E. (2001). The impact of tw o standardsbase d mathematics curricula on student achievement in Massachusetts. Journal for Research in Mathematical Education, 32 (4), 368. Rothstein, R. (2004). Class and schools: Using social, economic and educational reform to close the BlackWhite achievement gap Washington, DC: Economic Policy Institute. Rowan, B., Correnti, R. & Miller, R. J. (2002). What largescale survey research tells us about teacher effects on student achi evement: Insights from the Prospects study of Elementary Schools. Teachers College Record, 104 1525. Rubin, D. (1987). Multiple imputation for nonresponse in surveys. New York, NY: Wiley. Schoenfeld (2002). Making mathematics work for al l children: Issues of standards, testing, and equity. Educational Researcher, 31(1), 13. Senk, S. & Thompson, D. (Eds.) (2003). Standardsbased school mat hematics curricula: What are they? What do students learn ? Hillsdale, NJ: Lawrence Earlbaum Associates, Inc. Silver, E., Smith, M. S., & Nelson, B. S. (1995 ). The QUASAR Project: Equity concerns meet mathematics reforms in the mi ddle school. In W. G. Seca da, E. Fennema, & L. B. Adajian (Eds.), New directions in equity in mathematics education (pp. 956). New York: Cambridge University Press. Steele, C. M. & Aronson, J. (1998). Stereotype threat and the te st performance of academically successful African Americans. In C. Jencks & M. Phillips (Eds.), The BlackWhite test score gap (pp. 401). Washington, DC: Brookings Institution Press. Stiff, L. (1990). AfricanAmerican students and the promise of the Curriculum and Evaluation Standards In T. J. Cooney & C. R. Hirsch (Eds.), Teaching and learning mathematics in the 1990s (Yearbook of the National Council of Teachers of Mathematics, pp. 152), Reston, VA: NCTM. Strutchens, M., Lubienski, S. T., McGraw, R. & Westbrook, S. K. (2004). NAEP findings regarding race/ethnicity: Stud ents performance, school ex periences, attitudes/beliefs and family influences. In P. Kloosterman., & F. K. Lester, Jr. (Eds.) Results and interpretations of the 1990 through 2000 mathematics assessments of the National Assessment of Educational Progress (pp. 269). Reston, VA: NCTM.
PAGE 29
Examining Instruction, Ac hievement, and Equity 29 Strutchens, M. E., & Silver, E. A. (2000). NAEP findings rega rding race/ethnicity: Students performance, school ex periences, and attitude s and beliefs. In E. A. Silver & P. A. Kenney (Eds.), Results from the seventh mathematics assessment of the National Assessment of Educational Progress Reston, VA: National Council of Teachers of Mathematics. Swanson, C. B., & Stevens on, D. L. (2002, Spring). Standards based reform in practice: Evidence on state policy and classroom instruction from th e NAEP state assessments. Educational Evaluation and Policy Analysis, 24 (1), 127. Tate, W. F. (1997). Raceethni city, SES, gender, and language proficiency in mathematics achievement: An update. Journal for Research in Mathematics Education 28 652. Thompson, B. (2002). Statistical, practical, and clinical: How many kinds of significance do counselors need to consider? Journal of Counseling and Development, 80 64. Weis, L. (Ed.) (1988). Class, race, & gender in American education Albany: State University of New York Press. Wenglinsky, H. (2004, November 23). Closing the racial achievement gap: The role of reforming instructional practices. Educational Policy Analysis Archives 12 (64). Retrieved December 2, 2004, from http://epaa.asu.edu/epaa/v12n64/.
PAGE 30
Education Policy Analysis Archives Vol. 14 No. 14 30 About the Author Sarah Theule Lubienski University of Illinois at UrbanaChampaign Email: stl@uiuc.edu Sarah Theule Lubienski is an associate professor in the College of Education at the University of Illinois, UrbanaC hampaign. Her scholarship cen ters around intersections of education and equity, with a particular focus on mathematics achievement, instruction, and reform. Her recent NAEPrelated research has been supported by two Secondary Analysis Grants, funded by the National Center of Education Statistics. She is the chairperson of the NAEP Studies Special Interest Group of the Am erican Educational Research Association.
PAGE 31
Examining Instruction, Ac hievement, and Equity 31 EDUCATION POLICY ANALYSIS ARCHIVES http://epaa.asu.edu Editor: Sherman Dorn, University of South Florida Production Assistant: Chris Murre ll, Arizona State University General questions about ap propriateness of topics or particular articles may be addressed to the Editor, Sherman Dorn, epaaeditor@shermandorn.com. Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Robert Bickel Marshall University Gregory Camilli Rutgers University Casey Cobb University of Connecticut Linda DarlingHammond Stanford University Gunapala Edirisooriya Youngstown State University Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Gene V Glass Arizona State University Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Ohio University William Hunter University of Ontario Institute of Technology Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas MauhsPugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Anthony G. Rud Jr. Purdue University Michael Scriven Western Michigan University Terrence G. Wiley Arizona State University John Willinsky University of British Columbia
PAGE 32
Education Policy Analysis Archives Vol. 14 No. 14 32 EDUCATION POLICY ANALYSIS ARCHIVES Englishlanguage Graduate Student Editorial Board Noga Admon New York University Jessica Allen University of Colorado Cheryl Aman University of British Columbia Anne Black University of Connecticut Marisa Cannata Michigan State University Chad d'Entremont Teachers College Columbia University Carol Da Silva Harvard University Tara Donahue Michigan State University Camille Farrington University of Illinois Chicago Chris Frey Indiana University Amy Garrett Dikkers University of Minnesota Misty Ginicola Yale University Jake Gross Indiana University Hee Kyung Hong Loyola University Chicago Jennifer Lloyd University of British Columbia Heather Lord Yale University Shereeza Mohammed Florida Atlantic University Ben Superfine University of Michigan John Weathers University of Pennsylvania Kyo Yamashiro University of California Los Angeles
PAGE 33
Examining Instruction, Ac hievement, and Equity 33 Archivos Analticos de Polticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998003) Roberto Rodrguez Gmez Editorial Board Hugo Aboites Universidad Autnoma MetropolitanaXochimilco Adrin Acosta Universidad de Guadalajara Mxico Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educacin, Chile Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educacin, Argentina Teresa Bracho Centro de Investigacin y Docencia EconmicaCIDE Alejandro Canales Universidad Nacional Autnoma de Mxico Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagoga Popular, Per Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernndez Enguita Universidad de Salamanca. Espaa Gaudncio Frigotto Universidade Estadual do Rio de Janeiro, Brasil Rollin Kent Universidad Autnoma de Puebla. Puebla, Mxico Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California Mara Loreto Egaa Programa Interdisciplinario de Investigacin en Educacin Mariano Narodowski Universidad To rcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Polticas Educativas, Per Vanilda Paiva Universidade Estadual Do Rio De Janeiro, Brasil Miguel Pereira Catedratico Un iversidad de Granada, Espaa Angel Ignacio Prez Gmez Universidad de Mlaga Mnica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de So Paulo Diana Rhoten Social Science Research Council, New York, New York Jos Gimeno Sacristn Universidad de Valencia, Espaa Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, Mxico Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas PublicasUniversidad de Buenos Aires, Argentina Antonio Teodoro Universidade Lusfona Lisboa, Carlos A. Torres UCLA Jurjo Torres Santom Universidad de la Corua, Espaa
