Educational policy analysis archives

Educational policy analysis archives

Material Information

Educational policy analysis archives
Arizona State University
University of South Florida
Place of Publication:
Tempe, Ariz
Tampa, Fla
Arizona State University
University of South Florida.
Publication Date:


Subjects / Keywords:
Education -- Research -- Periodicals ( lcsh )
non-fiction ( marcgt )
serial ( sobekcm )

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
E11-00487 ( USFLDC DOI )
e11.487 ( USFLDC Handle )

Postcard Information



This item has the following downloads:

Full Text
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20069999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00487
0 245
Educational policy analysis archives.
n Vol. 14, no. 14 (June 01, 2006).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c June 01, 2006
Examining instruction, achievement, and equity with NAEP mathematics data / Sarah Theule Lubienski.
x Research
v Periodicals.
2 710
Arizona State University.
University of South Florida.
1 773
t Education Policy Analysis Archives (EPAA)
4 856

xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 14issue 14series Year mods:caption 20062006Month June6Day 11mods:originInfo mods:dateIssued iso8601 2006-06-01


Readers are free to copy display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for noncommercial purposes only, and no alte ration or transformation is made in the work. More details of this Creative Commons license are available at ses/by-nc-nd/2.5/. All other uses must be approved by the author(s) or EPAA EPAA is published jointly by the Mary Lou Fulton College of Education at Arizona State Universi ty and the College of Educ ation at the University of South Florida. Articles are indexed by H.W. Wilson & Co. Please contribute commentary at and send errata notes to Sherman Dorn ( EDUCATION POLICY ANALYSIS ARCHIVES A peer-reviewed scholarly journal Editor: Sherman Dorn College of Education University of South Florida Volume 14 Numb er 14 June 1, 200 6 ISSN 1068 Examining Instruction, Ac hievement, and Equity with NAEP Mathematics Data 1 Sarah Theule Lubienski University of Illinois at Urbana-Champaign Citation: Lubienski, S. T. (2006). Examining instru ction, achievement, and equity with NAEP mathematics data. Education Policy Analysis Archives, 14 (14). Retrieved [date] from Abstract The purpose of this article is two-fold First, it reports on a study of the distribution of reform-ori ented instructional practices among Black, White and Hispanic students, and the relationship between those practices and student achievement. The study identified many similarities in instruction across student groups, but there were some differences, such as Black and Hispanic students being assessed with multiple-choice tests significantly more often than were White students. Using hierarchical linear modeling, this study id entified several significant positiveand no negativer elationships betwee n reform-oriented practices and 4th-grade student achievement. Specifica lly, teacher emphas is on non-number mathematics strands, collabor ative problem solving, and teacher knowledge of the NCTM Standards were positive predictors of achievement. An analysis of interaction effects indicated that the re lationships between various instructional 1 This project was funded through a National Assessment of Education al Progress Secondary Analysis Grant from the National Center for Education Statistics, Institute of Education Sciences. The author would like to thank Eric Camburn, Mack Shelley, Jay Verkuilen, Lateefah Id-Deen, Megan Brown and Chris Lubienski for their helpful advice during various stages of this work. Only the author is responsible for the analysis and interpretations presented in this report.


Education Policy Analysis Archives Vol. 14 No. 14 2 practices and achievement were roughly similar for White, Black and Hispanic students. The second purpose of this article is to make comparisons with another study that used the same NAEP data, bu t drew very different conc lusions about the potential for particular instructional practices to alleviate inequities. A study published in EPAA by Wenglinsky (2004) concluded th at school personnel can eliminate racerelated gaps within their schools by changing their instructional practices. Similarities and differences between these two studies are discussed to illuminate how a researchers framing, methods, an d interpretations can heavily influence a studys conclusions. Ultimately, this articl e argues that the primary conclusion of Wenglinskys study is unwarranted. Keywords: equity, hierarchical linear modeling; ma thematics achievement; mathematics instruction; NAEP. Examinando instruccin, logros, y equidad usando resultados de matemticas de NAEP Resumen El propsito de este artculo es doble. Primero, reportar los resultados de un estudio sobre la distribucin de prcti cas orientadas para producir reformas educativas entre estudiantes negros, blancos e hispnicos y el lazo entre esas prcticas y los logros de los estudiantes. Este estudio identific muchas semejanzas en la instruccin de los grupos de estudi antes, pero tambin algunas diferencias. Por ejemplo los estudiantes negros e hi spnicos son evalua dos con pruebas de opcin-mltiple considerablemente ms a menudo que los estu diantes blancos. Usando modelos lineares jerrquicos, este estudio identific varias relaciones significativas positivas y re laciones no-negativas entre prcticas orientadas para producir reformas educativas en los resul tados educativos de estudiantes de 4to. grado. Especficamente, el nfasis de los profesores en la enseanza de aspectos no-numricos en matemticas, la cooperacin en la reso lucin de problemas, y el nivel de conocimiento de los profesor es de los estndares de NCTM fueron predictores positivos del logro educativo. Un anlisis de los efectos de las interacciones indic que lo s lazos entre las diferent es prcticas y los logro educativos fueron bastante similares para los estudiantes blancos, negros e hispnicos. El segundo propsito de este artculo fue hacer comparaciones con otro estudio que utiliz los mismos datos de NAEP, pe ro obtuvo conclusiones muy diferentes acerca del potencial de las prcticas orientadas para pr oducir reformas educativas para aliviar desigualdades. Wenglinsky ( 2004), en un estudio publicado en EPAA concluy que el personal de las escuelas puede eliminar las diferencias educativas relacionadas con aspectos raciales cambiando sus prcticas educacionales. Semejanzas y diferencias entre estos dos estu dios se discuten para iluminar cmo el marco referencial, los mtodo s, y las interpretaciones de un investigador influyen sustantivamente en las conclusiones de un estudio. En ltima instancia, este


Examining Instruction, Ac hievement, and Equity 3 artculo discute que la conclusin primaria del estudio de Wenglinsky no esta plenamente j ustitificada. Introduction Identifying instructional practices that both boost achievement and promote equity has been of increasing concern among educators, researchers, and policy makers recently. This article reports the results of a study that focuses on the distri bution of instructional practices advocated by the National Council of Teachers of Mathematics (NCT M), and the relationship between those practices and diverse students achievement. Recently a similar study was published in EP AA by Harold Wenglinsky (2004). Wenglinskys study is comparable to this study in many impor tant ways. Both studies utilized the 2000 National Assessment of Educational Progress (NAEP) mathemat ics data. Both studies used hierarchical linear modeling (HLM) to examine relationships between instructional practices and achievement, both overall and for particular subgroups. Additionally both studies identified positive correlations between some reform-oriented instructional practices and overall student achievement. Yet, the studies began with different framingsthis with an eye toward NCTM-endorsed practices, and Wenglinskys with an eye toward the Bush Admini strations No Child Left Behind (NCLB) act, which requires schools to closely monitor student achievement and reduce ra ce-related achievement gaps. These differences in framing led, in part, to differences in the particular statistical models and methods employed. There was also a difference in the care with which findings were interpreted, including the extent to which causal attributions were made. Ultimately, different conclusions were reached. Specifically, Wenglinsky concluded that by changing instructional practices, any gaps within a given school can be completely eliminated (p 17). In contrast, this studys conclusions are far less definitive and optimistic. This article begins with a report of the current study, including its NCTM-based framing, its methods, and results. The article concludes wi th a comparison of its methods, results and interpretations with those of Wenglinsky, and ultimately raises questions about Wenglinskys conclusions. Finally, issues pertaining to the analys is and reporting of NAEP and other large-scale data sets are highlighted. Background Although NAEP mathematics scores have genera lly risen over the past 15 years (Braswell, Daane & Grigg, 2003; Kloosterman & Lester, 2004) there is some debate as to whether the achievement gains occurred because of, or in spite of, reforms promoted in the NCTM Standards Although this cross-sectional study cannot offer a defi nitive resolution to this debate, it does offer a birds-eye view of the distribution of some reform-oriented instructional methods, and their correlations with achievement for various student groups. This study is situated at the intersection of work on reform-oriented instruction, mathematics achievement, and equity. Primary aspects of reform-oriented mathematics instruction are outlined first, followed by brief discussions of previous work regarding reform-oriented instruction and achievement, reform-oriented instruction and equity, and equity and mathematics achievement. This is followed by a more specifi c discussion of NAEP data, including a description of NAEP and findings of previous examinations of NAEP data regarding ma thematics achievement, instruction, and equity.


Education Policy Analysis Archives Vol. 14 No. 14 4 Reform-Oriented Mathematics Instruction In 1989, NCTM published the Curriculum and Evaluation Standards which, along with additional documents published subsequently (NCT M, 1991, 1995, 2000), called for mathematics instruction to be centered around students reasoning, collaborative problem solving, and mathematical communication (both verbal and written ). NCTM argued that a wider variety of tools (including manipulatives and calculators) and more meaningful forms of assessment should be employed. In addition, NCTM revised curricular goals for grades K to include greater emphasis on measurement, geometry, data analysis, probability, algebra, as well as number concepts. Finally, NCTM called for mathematical power for all studen ts, including those students previously underrepresented in mathematics-based care ers (NCTM, 1989, 1991, 1995, 2000). Reform-Oriented Mathematics Inst ruction and Student Achievement The benefits of instruction aligned with the NCTM Standard sor reform-oriented instruction as it is termed herehave been the subject of much debate. Evidence from schools that have used new, reform-oriented curricula has generally been encouraging, with students outscoring control groups on a variety of measures and in a variety of contexts (e.g., Reys, Reys, Lapan, Holliday & Wasman, 2003; Riordan & Noyce, 2001; Schoenfeld, 2002; Senk & Thompson, 2003). However, some critics of reform have pointed to less encouraging evidence, such as the fact that scores on NAEPs long-term-trend mathemat ics test remained flat during the 1990s, after a period of growth in previous decades (Loveless & Diperna, 2000). One need only make a brief visit to websites such as or to see that, despite the benefits of reform that some researchers report, much of the public is not convinced of the merits of reformoriented instruction on a broad scale. Reform-Oriented Mathematics Instruction and Equity Scholars have long argued that lower-SES and minority students have received more than their share of rote-based mathematics instruction (e.g., Anyon, 1981; Ladson-Billings, 1997; Means & Knapp, 1991). NCTMs vision of problem-cente red instruction for all students challenges the status quo and is intended to correct past inequities (NCTM, 1989, 1991, 1995, 2000). Now that the NCTM reforms are being implemented, scholars have begun to ask whether some students enter the mathematics classroom better positioned than others to learn in the ways envisioned in the Standards (Lubienski, 2000a, 2000b). Hickey, Moore, and Pellegrino (2001) found that reform-oriented instruction improved lowand high-SES students problem solving skills, but the same instruction increased the SES-related gap in students performance on the concepts and estimation portion of the Iowa Test of Basic Skills However, still other studies have suggested that reform-minded practices are particularly beneficial for lower-SES and minority students (e.g., Boaler, 2002; Ladson Billings, 1997; Newmann & Wehlage, 1995; Schoenfeld, 2002; Silver, Smith, & Nelson, 1995; Stiff, 1990). After analyzing national test score trends, L ee (2002) noted that Black-White gaps in achievement decreased du ring the 1970ss emphasis on basic skills, but increased during the 1990s, when emphasis shifted to higher-order thin king. Others have provided additional evidence of this trend (Campbell, Hombo, & Mazzeo, 2000; Jencks & Phillips, 1998). These studies raise the


Examining Instruction, Ac hievement, and Equity 5 question of whether these patterns are caused by re form-oriented instruction, differential access to such instruction, or other confounding variables. 2 Equity and Achievement In recent years, many researchers have struggl ed to understand the underlying causes of persistent inequities in academic achievement, especially race-related achievement gaps. 3 Clearly, SES differences involving parent education, occupation, income, and educational resources in the home account for much of these gaps (Jencks & Ph illips, 1998; Peng, Wright & Hill, 1995). Several studies of race-related achievement gaps have al so examined school-related factors, including the roles of teachers, curricula, school funding, student motivation, and student resistance (e.g., Banks, 1988; Cook & Ludwig, 1998; Ferguson, 1998a, 1998b, 1998c; Ogbu, 1995; Steele & Aronson, 1998). Such discussions have tended to focus on the overall academic performance and experiences of minority students, as opposed to an in-depth examination of achievement and instructional practices in a particular subject area, such as mathemat ics. This trend was noted by Lee (2002), who concluded his general analysis of patterns in achiev ement data by urging subject matter specialists to further examine inequities in their areas of expertise. This study does not attempt to enter into de bates about the many factors outside of schools that contribute to achievement inequities, but instead focuses on instructional variables over which educators have control. This study focuses specifically on students achievement and learning experiences in mathematics, which is a particularly important subject to consider in relation to equity because it is a key gatekeeper for entry into hi gh status occupations. Researchers in mathematics education have given some attention to race-related gaps in mathematics achievement, but have rarely examined race and SES simultaneously (Lubienski & Bowen, 2000; Tate, 1997). By exploring the relationship between particular instructional practices and achievement utilizing hierarchical linear models that include both race and SES, this study examines the extent to which race-related achievement gaps that persist after controlling for SES may be related to differences in students access to particular mathematics instructional practices, as measured by NAEP. The National Assessment of Educational Progress NAEP is the only nationally representative, ongoing assessment of U.S. academic achievement. NAEP measures student performance at 4th, 8th, and 12th grades in mathematics and other subject areas. NAEP also provides surv ey information from students and their teachers regarding mathematical backgrounds, beliefs, and instructional practices. Since 1990, the Main NAEP mathematics assessment has been guided by a framework based on NCTMs Curriculum and Evaluation Stan dards for School Mathematics (1989). Hence, the Main 2 The publication of the 2003 NAEP results prompted even more discussion about these issues, as both NCTM and NCLB were credited by various parties for improvemen ts in scores and decreases in achievement gaps between 2000 and 2003. However, th e 2003 data were not yet av ailable for secondary analysis at the time of this study. 3 For the sake of simplifying the text, the term race is used very loosely to mean race or ethnicity when referring to the NAEP categories of Black, White and Hispanic students. Additionally, to be consistent with NAEP data, the terms Black, White, and Hispanic are used throughout this article.


Education Policy Analysis Archives Vol. 14 No. 14 6 NAEP assesses students performance on both mu ltiple-choice and constructed-response items over the five mathematics strands emphasized by NCTM: number/operations, geometry, measurement, data analysis, and algebra/functions. Additionally, some NAEP survey questions administered to students and teachers were designed to identify the extent to which students classroom experiences are aligned with NCTMs vision for mathematics instruction. Previous NAEP Findings on the Distribution of Reform-Oriented Mathematics Instruction Strutchens and Silver (2000) gave detailed a ttention to race-related disparities in 1996 NAEP data on mathematics achievement, students beliefs about mathematics, and teachers instructional practices and emphases. They found that Black and Hispanic students were at least as likely as White students to have access to manipu latives, real-life mathematics problems, and student collaboration in their mathematics classrooms. However, according to teacher reports, White eighth graders were more likely than Black or Hispanic students to receive some aspects of reform-oriented instruction, such as calculator access, fewer multiple-choice assessments, and a heavy emphasis on reasoning. Students mathematical attitudes and beliefs, although shaped by a variety of factors, are linked to the instruction students receive. Strutchens and Silver (2000) reported that Black and Hispanic students were more likely than White studen ts to agree with the statements, There is only one way to solve a math problem and Learning mathematics is mostly memorizing facts. However, they cautioned that the race-related diffe rences they reported might be due more to SES than race. More recently, Strutchens, Lubienski, McGr aw and Westbrook (2004) examined the 2000 Main NAEP mathematics data and confirmed the above 1996 findings, with the exception of differential access to teacher-reported emphasis on reasoning, for which there were no longer racerelated disparities in the 2000 data. Lubienski and Shelley (2003) extended this work and found that the race-related gaps persisted even after controllin g for SES. Additionally, in their analysis of 1996 State NAEP data, Swanson and Stevenson (2002) id entified SES-related differences in instruction, with more affluent schools tending to utilize more re form-oriented practices, as measured by a single composite of 16 variables. Overall, previous analyses of NAEP data have indicated some potentially important ways in which White, higher-SES students are experiencing more of the fundamental instructional shifts called for by NCTM than less privileged students These differences are reminiscent of those discussed by Means & Knapp (1991), Anyon (198 1) and others who observed poor and minority students receiving more than their share of drill-based, computation-focused instruction. Previous NAEP Findings on Reform-Oriented Mathematics Instruction, Achievement and Equity The official NAEP report for the 2000 main mathematics assessment highlighted several instruction-related variables that correlated with achievement. For example 8th graders with unrestricted access to calculators scored significantl y higher than did their peers without such access. Similarly, the report stated that 4th, 8th, and 12th grade students who agreed with the statement, Learning math is mostly memorizing facts, sc ored significantly lower than did students who disagreed with the statement (Braswell, Lutkis Grigg, Santapau, TayLim, & Johnson, 2001). However, given that White, high-SES students have been more likely than their less privileged counterparts to have unrestricted calculator access an d to believe that mathematics is more than just fact memorization (Lubienski, 2002; Strutchens et al., 2004), race and SES are likely confounding variables in the correlations noted by Braswell et al. (2001).


Examining Instruction, Ac hievement, and Equity 7 Hence, the question remains whether reform-oriented instructional practices, as reported in NAEP teacher surveys, are positiv e predictors of achievement after controlling for confounding variables. If so, then the differences in instructi onal practices noted above might contribute to raceand SES-related achievement differences. A prior study by Raudenbush, Fotiu, and Che ong (1998), utilizing 1992 State NAEP data, found that teacher-reported emphasis on reasoning in mathematics instruction correlated positively with achievement even after controlling for race an d SES, and that White, high-SES students were more likely to have such a teacher. However, dispar ities in students access to teachers emphasizing reasoning were no longer significant in the 2000 NAEP data. Finally, as noted previously, at the same time this study was being conducted, Wenglinsky (2004) used HLM to analyze the 2000 NAEP Math ematics Data, examining whether particular instructional practices related to schools overall ac hievement and the size of their race-related gaps. He found that teacher-reported time on task, use of routine exercises and a geometry emphasis correlated with higher achievement for students, in general, whereas frequent testing, emphasis on facts, and project work correlated negatively with ac hievement. He also concluded that an emphasis on measurement was the most beneficial practice (p. 16) for Black students, while an emphasis on data analysis appeared beneficial for Hispanic students. However, as will be discussed in more detail later, his conclusions require further consideration. The 2000 Main NAEP data included larger sa mples than previous administrations, and also included dozens of teacher-reported variables rela ting to reform-oriented instruction (many of which were deleted in 2003). An in-depth analysis of instruction and achievement using the 2000 data can illuminate relationships among reform-oriented instructional measures, student achievement, and equity. Still, given that NAEP data are cross-s ectional and not longitudinal, no NAEP-based study can definitively determine which instructional method s are most effective for particular groups of students. Still, the scope and representative nature of NAEP data can lend important evidence to inform current debates and to point towa rd areas in need of further research. Research Questions In the context of the NCTM reform movement and concerns about its impact on mathematics achievement and equity, this study addr esses three questions. First, the study examines the extent to which reform-oriented instructional pr actices are reaching all students, regardless of race. Second, the study investigates whether pa rticular reform-oriented instructional practices correlate positively or negatively with mathematics achievement, af ter controlling for race, SES, and other potentially confounding variables. Finally the study considers whether reform-oriented practices correlate similarly with achievement for diverse student groups, regardless of student race or SES. Taken together, these questions probe whether inequities in access to reform-oriented instruction might contribute to achievement gaps, with a particular focus on Black-White and Hispanic-White gaps that persist even after c ontrolling for studentand school-level SES. Identifying inequities in access to instructional methods that correlate positively or negatively with achievement can shed light on variables potentia lly underlying achievement gaps, enrich our understanding of students experiences with learni ng mathematics, and suggest important areas for further study. While not assuming that instruction-related variables are the only, or even primary, cause of achievement gaps, it is important to gi ve attention to the area that educators are best positioned to address.


Education Policy Analysis Archives Vol. 14 No. 14 8 Method Several methodological features of this st udy merit discussion including the NAEP samples used, special challenges of NAEP analyses, th e specific variables utilized, and the analyses conducted. The Samples The 2000 Main NAEP data used in this study were accessed from a restricted-use CDROM. 4 Data regarding the mathematics achievement of a nationally representative sample of 13,511 4th graders who were assessed in late winter/early spring, 2000, were included, as well as data from student background surveys and teacher reports of instructional practices. The unweighted sample of students was 64% White, 17% Hispanic, 13% Blac k, and 6% other groups. The analyses reported here were part of a larger study that gave attention to both 4th and 8th grades, and that examined instruction-related variables as reported by both students and teachers. 5 For the sake of space limitations and comparability with Wenglinskys stud y, analyses of fourth grade achievement and teacher-reported data are the primary focus here. However, additional findings from the larger study are footnoted when particularly relevant. Methodological Challenges of NAEP Data Analyses Several features complicate the analysis of NAEP data. To obtain a representative sample of students, schools are stratified based on urbanicity, minority population, size, and area income, and then schools within each stratum are selected at ra ndom. Finally, students are selected randomly within schools. Deliberate oversampling of certain strata, such as schools with high enrollments of minority students, results in more reliable estimates for the oversampled subgroups, and then student and school weights are used to adjust for both unequal probabilities of selection and nonresponse. To account for the clustered sampling, NAEP data also contains replicate weights for each student and school, which are used in calculating sampling errors using the jackknife repeated replication method. Teacher weights are not assign ed, because NAEP selects samples of students and then surveys their teachers; teacher data are linked to student data and are interpreted at the student level. As a concrete example, NAEP an alyses would not indicate that 80% of teachers reported allowing unrestricted calculator use, but that 80% of students had teachers who reported allowing unrestricted calculator use. To reduce the test-taking burden on individual students, NAEP administers only a subset of items to each student. Hence, individual students achievement is not measured reliably enough to be assigned a single score. Instead, using Item Response Theory (IRT), NAEP estimates a distribution of plausible values for each student s proficiency, based on the students responses to administered items and other student character istics. When analyzing NAEP achievement data, separate analyses are conducted with the five plausibl e values assigned to each student. The five sets of results are then synthesized, following Rubin (1 987) on the analysis of multiply-imputed data. For 4 Researchers who apply for a license from the National Center for Education Statistics may obtain restricted-use data. 5 A detailed (120-page) report of the methods and results of the full study was submitted to the National Center of Educ ation Statistics (Lubienski, Camburn & Shelley, 2004). Interested readers may download the report from


Examining Instruction, Ac hievement, and Equity 9 more detailed information regarding the structure of NAEP data, see Johnson (1992) and Johnson and Rust (1992). Demographic and Instruction-Related Variables Several studentand school-level demographic variables were included in this analysis, along with teacher-reported variables pertaining to instructional practices: 6 Student race. Binary Black and Hispanic variab les were created from NAEPs student race/ethnicity variable (taken from student self-reports, or school records when necessary). School race There was no school race variable in the 2000 NAEP mathematics data set. As a proxy, the percentage of White/Asian students in each schools sample was calculated. 7 Student SES. After consideration of the much-debated meanings of socioeconomic status and social class (e.g., Duberman, 1976; Weis, 19 88), a comprehensive SES variable was created using factor analysis. Six variables were combined to produce a new student SES variable: types of reading material in students homes (newspapers magazines, books, and encyclopedia), computer and Internet access at home, extent to which studie s are discussed at home, and eligibility for school lunch and Title 1 (a federal program for disadvanta ged students). Parent education levels were not reported for 4th graders in 2000 and were therefore not included. The final variable was standardized with a mean of 0 and standard deviation of 1. School SES At each sampled school, an administra tor provided survey data regarding the percentage of students qualifying for Title 1 funds and free/reduced lunch. These two variables were ordinal with rough categories of percentages (e.g., 0, 11, etc.). The final school SES measure was a composite of the student-level SES variable aggregated to the school level and the percentage of students eligible for free/reduced lunch and Title 1. 8 Gender. NAEPs Gender variable (coded as boy = 1, else = 0) was included in the analyses because prior research suggests that gender correlates significantly with mathematics achievement (e.g., Fennema, Carpenter, Jacobs, Franke & Levi, 1998; Lubienski, McGraw & Strutchens, 2004). 9 Disability. Given that students with disabilities ten d to score lower than others on NAEP (Foegen, 2004) and that these students could be sub ject to different instructional practices than their peers, a binary student disability variable was used to control for whether students have a nonorthopedic disability (e.g., learning disability, visual impairment, behavioral disorder). 6 Teacher background/certification variables were also considered, including undergraduate major and whether teachers held masters degrees. These variables were ultima tely omitted from the fourth-grade analyses due to a lack of significance. However, in the 8th-grade analyses, secondary mathematics certification was significant. 7 White and Asian students were combined because these groups tend to have higher achievement than other groups. The resulting variable was skewed and somewhat bi-modal (revealing school segregation patterns). A natural logarithmic transformation was used to create a somewhat more normally distributed variable, which was then standardized with mean = 0 and standard deviation = 1. Although a more normally distributed variable was desirable for inclusion in the models, the tradeoff in using such a transformation is that the resulting variable is more difficult to interpret. 8 The final composite was standardized with a mean of 0 and standard deviation of 1. For additional details about the creation of this and other demographi c indicators, see Lubienski, Camburn & Shelley, 2004. 9 NAEPs teacher-reported data generally does not vary by gender because each teacher survey is linked to all students selected from his/her class. However, th e larger study also included student-reported instruction-related data, which can vary by gender. For the sake of consis tency, gender was included in all models.


Education P olicy Analysis Archives Vol. 14 No. 14 10 School sector Public/private school status has been found to correlate with achievement (e.g., Bryk, Lee & Holland, 1993; Lubienski & Lu bienski, 2006) and might also relate to the instructional practices employed. The NAEP variab le schtype was recoded, with Catholic and other private schools = 1 and public schools = 0. Teacher-reported instructi on-related variables. During initial explorations of teacher-reported variables that could conceivably be viewed as me asuring some aspect of reform-oriented practices, the net was cast widely to include 31 such variable s. Factor analysis was used to create composites of highly correlated instruction-related variables, thereby reducing the number of predictors included in the HLM analyses and decreasing the danger of fishing for correlations among multiple variables. 10 Several variables did not seem to fit with the others and were excluded because upon further consideration it did not make conceptual or st atistical sense to include them. These variables included the frequency with which students t ook tests, did problems from textbooks, and used computers in mathematics class. Although these variables could be construed to relate to reformoriented instruction, closer inspection of the conten t of the questions combined with their lack of correlation with other reform measures suggested th at these variables were not essential measures of reform-oriented instruction. Ultimately, 24 variables remained, with most clustering around six themes: calculator use, facts and skills, collaborative problem solving, non-number curricular emphasis, writing about mathematics, and manipulative use. 11 Teacher emphasis on reasoning, use of multiple choice assessments, and teachers knowledge of the NCTM Standards tended to correlate loosely with the other variables, but did not associate strongly with any single factor or with each othe r. These variables were included among the final set of instruction-related measures, but were treated individually. Six factor analyses were conductedone with each of the 6 clusters of variablesto create a single, standardized factor (with a mean of 0 and standard deviation of 1) representing each theme. In each case, only one factor resulted with an Eigenvalue greater than 1, so that factor was used to represent the cluster. The loadings of each of the original variables on the final resulting factors are listed in Table 1, along with Cronbachs alpha, an indicator of how closely the items correlate with one another. 12 10 The Kaiser-Meyer-Olkin measure of sampling adequa cy (KMO) with all of the variables in a single factor analysis was roughly .8, indicating that factor analysis was appropriate. 11 Variables clustered similarly in the 8th-grade fa ctor analysis, providing some evidence that the NAEP survey items are capturing some meaningful differences in teacher instruction, despite the rather rough response categories used and th e self-reported nature of the data. 12 When designing a survey and developing item clusters, Cronbachs Alphas between .7 and .9 are desirable. However, lower values were considered acceptable for the purposes of this study, in which the goal was not to design a new survey, but to create composit es of existing survey items that capture various aspects of reform-oriented instruction. Conceptual connections among items (e.g., whether items refer to calculators or manipulatives) and a desire for consistency in the created composites across 4th and 8th grade were also considered in the development of the composites.


Examining Instruction, Ac hievement, and Equity 11 Table 1 Teacher-Reported Instruction-Related FactorsQuestions with Loadings Questions Grouped by Fa ctors Response Options (Alpha) or Loading 1) Calculators (.44) How often do the students in this class use a calculator? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .72 Do you permit students in this class to use calculators for tests? Yes, No .79 Do you permit students in this class unrestricted use of calculators? Yes, No .71 2) Emphasis on Facts & Skills (.62) How much emphasis did you or will you give to learning mathematics facts and concepts? Heavy emphasis, Moderate emphasis, Little/no emphasis .83 How much emphasis did you or will you give to learning skills and procedures needed to solve routine problems? Heavy emphasis, Moderate emphasis, Little/no emphasis .74 How much emphasis did you or w ill you give to number and operations? Heavy emphasis, Moderate emphasis, Little/no emphasis .70 3) Collaborative Problem Solving (.76) How often do the students in th is class discuss solutions to mathematics problems with other students? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .84 How often do the students in this class work and discuss mathematics problems that reflect real life situations? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .74 How often do the students in this class talk to the class about their mathematics work? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .72 How often do the students in this class solve mathematics problems in small groups or with a partner? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .65 How much emphasis did you or will you give to learning how to communicate ideas in ma thematics effectively? Heavy emphasis, Moderate emphasis, Little/no emphasis .62 4) Non-Number Curricular Emphasis (.68) How much emphasis did you or will you give geometry? Heavy emphasis, Moderate emphasis, Little/no emphasis .78 How much emphasis did you or will you give measurement? Heavy emphasis, Moderate emphasis, Little/no emphasis .65


olicy Analysis Archives Vol. 14 No. 14 12 ons Grouped by Fa ctors Response Options (Alpha) or Loading How much emphasis did you or will you give data analysis, statistics, and probability? Heavy emphasis, Moderate emphasis, Little/no emphasis .76 How much emphasis did you or will you give Algebra and functions (informal introduction of concepts)? Heavy emphasis, Moderate emphasis, Little/no emphasis .69 5) Writing About Mathematics (.69) How often do you use short (e.g., a phrase or sent ence) or long (e.g., several sentences or paragra phs) written responses to assess student progress in mathematics? 1 times a week, 1 times a month, 12 times a year, Never/hardly ever .76 How often do you use individual or group projects or presentations to assess studen t progress in mathematics? 1 times a week, 1 times a month, 12 times a year, Never/hardly ever .70 How often do the students in th is class write a few sentences about how to solve a mathematics problem? Almost every day, 1 times a week, 12 times a month, Never/hardly ever .76 How often do the students in th is class write reports or do mathematics projects? 1 times a week, 1 times a month, 12 times a year, Never/hardly ever .67 6) Manipulatives (.66) How often do the students in th is class work with objects like rulers? Almost every day, 1 times a week, 12 times a month, Never/hardly ever N/A* How often do the students in th is class work with counting blocks or geometric shapes? Almost every day, 1 times a week, 12 times a month, Never/hardly ever N/A* 7) Reasoning How much emphasis did you or w ill you give to developing reasoning and analyt ic ability to solve unique problems? Heavy emphasis, Moderate emphasis, Little/no emphasis N/A* 8) Multiple-Choice Assessment How often do you use multiple-choice tests to assess student progress in ma thematics? 1 times a week, 1 times a month, 12 times a year, Never/hardly ever N/A* 9) Knowledge of NCTM Standards How knowledgeable are you about the NCTM Curriculum and Evaluation Standards fo r School Mathematics ? Very knowledgeable, knowledgeable, Somewhat knowledgeable, Little/no knowledge N/A* *Loadings not relevant because variable was not part of a composite of three or more variables. Education PQuesti


Examining Instruction, Ac hievement, and Equity 13 Given that the goal was to use these variables as predictors of achievement in HLM models, it was preferable for them to be either continuous, normal variables or binary. The single, isolated variables (teacher emphasis on reasoning, multiple-choice assessment use, knowledge of the NCTM Standards ) were ordinal, not continuous. A few of the variables created through a combination of factors were heavily skewed or bimodal. These issue s were addressed by creating binary variables as follows: Calculators (which was a bimodal composite) was reco ded so that above average calculator use = 1, else = 0. Facts and skills was recoded so that heavy emphasis on facts and concepts, skills for routine problems, and number/operations = 0, otherwise 1. Reasoning was recoded so that heavy emphasis = 1, moderate or light emphasis = 0 Multiple choice assessment use was originally an ordinal variable with 4 categories. There seemed to be substantial differences between tea chers using multiple choice assessments weekly, monthly, and annually/never, so two binary vari ables were created: weekly = 1 and less than weekly = 0, and once or twice annually or never = 1, othe rwise 0. This effectively separates the weekly, monthly, and annually/never groups. Knowledge of NCTM Standards originally had four categories: Very knowledgeable, knowledgeable, somewhat knowledgeable, and little/no knowledge. Two binary variables were created: 1 = very knowledgeable about the Standards otherwise = 0; and 1 = little or no knowledge about the Standards otherwise = 0. This set distinguishes between the two extremes and combines the two middle categories. The remaining continuous variables, collaborat ive problem solving, non-number curricular emphasis, writing about mathematics, and manipula tives were standardized with a mean of 0 and standard deviation of 1. With the exception of weekly multiple-choice assessment use and little or no knowledge of the NCTM Standards each variable was coded so that a higher number indicated a greater alignment with the NCTM Standards. Data Analysis The initial, descriptive phase of data analys is addressed the first research question: Are reform-oriented instructional practices reaching a ll students, regardless of race? HLM models were then developed to answer the second and th ird research questions: Which reform-oriented instructional practices correlate positively or ne gatively with mathematics achievement after controlling for confounding variables? Do those correlations vary by student race and/or SES? Phase 1Descriptive analyses of instruction by race. Means of the newly created instruction-related variables were compared for White, Black and Hispanic students to examine whether differences emerged for the instructional composites created for this study. These comparisons were made using AM Statistical Softwa re, designed by the American Institutes for Research to handle the special weighting and jack knifing needs of complex data sets such as NAEP. Two-tailed T-tests were used to determine if means significantly differed between White and Black students and between White and Hispanic students. When interpreting results, issues of multiple comparisons were considered using Bonferroni corrections. Phase 2HLM analyses of in struction and achievement. Because of the nested nature of the data (students and teachers within schools) two-level HLMs were used to examine whether particular reform-based practices positively or ne gatively predicted achievement while controlling for potentially confounding variables at both the studen t and school level. HLM statistical software was designed specifically to accommodate multi-level datasets, including those with plausible values


Education Policy Analysis Archives Vol. 14 No. 14 14 (Raudenbush & Bryk, 2002). As HLM computes st atistics related to NAEP achievement, each parameter is estimated for each of the five pl ausible values, and the five estimates are then averaged. 13 In the HLM models, students (level 1) were nested within schools (level 2). 14 The level of classroom or teacher was not included as a sepa rate level because NAEP uses random samples of students and not teachers, and there were no teacher codes in the data to allow for analysis at the teacher level. Given these constraints, and given the studys primary focus on student-level disparities in instruction and achievement (as oppose d to school-level issues), teacher-level data were treated as level-1 data. In this way, the instru ctional practices linked with students were those they had experienced during that school year. It is im portant to note that, given the lack of a prior achievement measure in NAEP, this study does not examine the change in students achievement during the school year. Therefore, it is very po ssible that relationships between instruction and achievement that appear weak or insignificant in this study could be found to be stronger in longitudinal studies. Because of concerns about collinearity among the 9 teacher-reported instructional practice variables, separate HLM analyses were conducted with each of the variables to determine the relationship of each with student achievement. Th e studentand school-level demographic variables described above were also included as predictors in the models. Given the focus on general relationships between NCTM practices and achievemen t, as opposed to variation in their slopes by school, slopes were fixed in the HLM models, and continuous predictors were centered around their overall means. Binary predictors were not centered. The changes in coefficients for Black and Hispanic students that occurred after adding each instructional variable to the model were examined in an attempt to gauge the possible impact of each instructional practice on the race-related achievement gaps that persisted after controlling for SES. This change in coefficients was examined separately with HLM models for each of the 9 instruction-related variables, and interaction effects were included in the final models to examine whether the coefficient for each instruction-related variable differed by student race and SES. Finally a larger HLM model was created to examine the change in coefficients and variance when the 9 instruction-related variables were included simultaneously, yet this model was interpreted cautiously because of collinearity among the 9 instruction-related predictors. Results To help the reader interpret the results di scussed here, some information about NAEP scores is necessary. NAEP uses a 500-point scale on which 4th graders scored an average of 228 in 2000. The fourth-grade Hispanic-White gap was 24 points, and the Black-White gap was 31 points. The standard deviation for the 2000 fourth-grade sc ale scores was 31 points. Hence, a difference of 3 points can be considered an effect size of ro ughly 0.1. Note that the size of the Black-White 13 A more detailed explanation of the data analysis methods that the HLM program uses is available in Bryk and Raudenbush (1992) an d Raudenbush and Bryk (2002). 14 Due to missing survey data, HLM samples were reduced to include 9,999 students across 611 schools. The demographics for these reduced samples we re very similar to the de mographics of the entire data set, lessening concerns about the results of the analyses being skewed due to missing data.


Examining Instruction, Ac hievement, and Equity 15 fourth-grade gap was a full standard deviation (ver y large effect size of 1), and the Hispanic-White gap had an effect size of 0.8. 15 Table 2 Means of Grade 4 Teacher-Reported Instruc tion-Related Factors by Student Race White Black Hispanic Factor Mean S.E. Mean S.E. Mean S.E. Greater than Average Calculator Use = 1 (binary) .35 .03 .37 .04 .27* .03 Not Heavy Emphasis on Facts and Skills = 1 (binary) .22 .02 .23 .02 .22 .02 Collaborative Problem Solving -.02 .05 .06 .06 -.03 .06 Non-number Curricular Emphasis -.06 .06 .16* .09 .11* .07 Writing About Math -.04 .06 .12 .07 .06 .06 Manipulatives -.05 .06 .12* .06 .09 .07 Heavy Emphasis on Reasoning = 1 (binary) .61 .02 .65 .03 .58 .03 Weekly Multiple Choice Assessment Use = 1 (binary) .09 .01 .23*** .03 .22*** .02 Yearly or Never Multiple Choice Assessment Use = 1 (binary) .44 .03 .30*** .03 .32*** .03 Very Knowledg eable About NCTM Standards = 1 (binary) .07 .01 .04 .01 .06 .01 Little/No Knowledge About NCTM Standards = 1 (binary) .35 .03 .36 .02 .41 .03 Note: Means for Black and Hispanic students were compared with means of White students in the significance tests. p < .05; ** p < .01; *** p < .001 Means by Race of Instruction-Related Variables The means and standard errors for the teacher-reported instructional composites for White, Black and Hispanic fourth graders are presented in Table 2. 16 There were no significant race-related differences in teacher emphasis on reasoning an d facts/skills, teacher knowledge of the NCTM Standards collaborative problem solving, and writin g about mathematics. Black and Hispanic 15 Although there have been various methods proposed for calculating effect sizes (Thompson, 2002), this discussion refers to Cohens (1988) d, which is computed by dividing the difference between the means of two samples by the standard deviation of the combined population sample. 16 In the full study, 28 t-tests were used to co mpare White-Black and White-Hispanic means for 14 teacherand student-reported instructio n-related variables. Hence, it can be considered appropriate to hold p < .05/28 = .002 as the standard for determining statisti cal significance, using the Bonferroni correction for multiple comparisons. However, others might argue for a different clustering of va riables for the Bonferroni correction (e.g., dividing .05 by 2, because two comparisons were made for each variable). Hence, the standard .05, .01, and .001 significance levels are report ed in the tables, leaving read ers to interpret results as they see fit.


Education Policy Analysis Archives Vol. 14 No. 14 16 students were at least as likely as White students to have access to manipulatives and a non-number curricular emphasis. For example, whereas White st udents were 0.05 standard deviations below the mean for use of manipulatives, Black students were 0.12 standard deviations above the mean. Hence, Black students actually appeared to be g etting slightly more access to some reform-oriented practices than were their White peers (with means fo r Hispanic students generally in between those for Black and White students). However, consistent wi th previous findings (Strutchens, et al., 2004), Black and Hispanic students were significantly more likely to be assessed with multiple choice tests than were White students. For example, 44% of White students were assessed with multiple choice tests no more than once or twice a year, whereas th is percentage was 30% for Black students and 32% for Hispanic students (because this is a bi nary variable, the means can be interpreted as percentages). 17 HLM Analyses of Instruction-Related Factors and Achievement: The Example of Calculator Access HLM analyses were undertaken to examine th e relationship between particular reformoriented instructional practices and mathematics ac hievement, as measured by NAEP. Because of concerns of multi-collinearity, separate models were created for each of the 9 instructional factors. Due to space limitations, the full results of each of the HLM models are not presented here. Instead, full details of the models involving the teacher-rep orted calculator composite is presented as an example, and then the main results involvin g the remaining instruction-related factors are summarized. Table 3 presents the set of models run with the 4th-grade calculator use composite. Recall that the calculator use variable was binary, with 1 = above average and 0 = at or below average. Models 1, 2, and 3 remained constant for all grade 4 HLM analyses regardless of the instructionrelated variable in question. The base model (Mod el 1) shows that the mean achievement across all sampled schools was 230.4 points. It also indica tes that roughly one third of the variance in achievement was between schools (intraclass correlation=.34), and two thirds of the variance was among students within schools. According to mode l 2, the mean for Black students was about 23 points lower than that of their non-Hispanic peer s within the same school (White/Asian students were the primary comparison group, with a mean achievement of 235) 18 whereas Hispanic students scored about 17 points lower. The addition of these two student-level race variables accounted for almost 40% of the variance between schools, but only about 5% of the variation within schools. In Model 3, we can see that student and school SE S, gender, disability, and school sector all significantly predicted achievement. For example, an increase in one standard deviation in SES was associated with a 7.6 point increase in achievemen t at the student level and 6.1 points at the school level. Similarly, the coefficients reported in model 3 indicate a 3.8 point advantage for males and a 30.3 point disadvantage for students with disabilit ies. Additionally, the private school students in the 17 More frequent multiple-choice te sting for Black and Hispanic students was also found at grade 8. Black and Hispanic eighth graders were also less likely to be given access to calculators by their mathematics teachers. 18 The American Indian/Alaskan Native subgroup comprises only about 2% of students. Because of their small sample size, these studen ts were not denoted by a separate variable, but were included in the general default group of non-Black, non-Hispanic students.


Examining Instruction, Ac hievement, and Equity 17 sample performed a significant 4.8 points lower than the public school students sampled. 19 The coefficient for school race was insignificant (though it was close to significant, and it was significant when school SES was not included in the model). Mo del 3 also indicates that even after controlling for these other contextual variables, there are still highly significant race-related gaps within schools of 12.5 (Hispanic) and 17.1 (Black) points. Taken together, the demographic factors in Model 3 explained 70% of the variance between schools, and 16% of the variance within schools. Table 3 HLM Models of NAEP Achievem ent by Teacher-Reported Calcul ator Use and Achievement, Grade 4 Variable Model 1 Model 2 Model 3 Model 4 Model 5 Level 1 (student & his/her teacher) Intercept 230.4 235.0*** 236.6*** 236.6*** 236.3*** Black -22.8*** -17.1*** -17.1*** -16.8*** Hispanic -17.3*** -12.5*** -12.5*** -12.1*** Student SES 7.6*** 7.6*** 7.1*** Boy 3.8*** 3.8*** 3.8*** Disability -30.3*** -30.3*** -30.4*** Calculator Use -0.1 -0.6 Calculator x Black -0.7 Calculator x Hispanic -1.4 Calculator x SES 1.5 Level 2 (school) School SES 6.1*** 6.1*** 6.1*** School Race/Ethnicity 1.0 1.0 1.0 Private School -4.8*** -4.8*** -4.6** Random Effects Variance Component Variance Component Variance Component Variance Component Variance Component Intercept (variance between schools) 275.4 168.4 83.6 83.7 83.5 Level-1 (variance within schools) 537.9 510.4 454.2 454.2 454.1 Intraclass Correlation .34 .25 .16 .16 .16 N= 9999 students and 611 schools. p < .05; ** p < .01; *** p < .001 In Model 4, we see that once all of these poten tially confounding variables are controlled, students whose teachers reported giving a higher than average amount of calculator access to students scored an insignificant 0.1 point lower on the NAEP mathematics assessment than did students with teachers reporting less calculator access in their classrooms. 20 Finally, Model 5 controls 19 Further investigation revealed that although overall achievement was substantially higher in private schools than in public schools, this relationship re versed after controlling for SES and other demographic variables. See Lubienski & Lubienski (2006) regarding a follow-up study. 20 Interestingly, the student -reported calculator variable negatively predicted achievement at grade 4. Teacher-reported calculator use and achievement were more positively related at grade 8, perhaps because advanced classes utilize calculators more often. Again, see Lubienski, Camburn an d Shelley (2004) for more information.


Education Policy Analysis Archives Vol. 14 No. 14 18 for all of these factors and also includes student-level interaction terms to examine whether the relationship between calculator access and achiev ement differs by student race. None of the interaction terms were significant. The addition of the calculator variables in Models 4 and 5 did not help explain additional variance between or within schools. Summary of HLM Results Each of the other 9 teacher-reported instructi on-related variables were treated in a likewise manner, with the final results condensed in Table 4 (interaction terms are discussed later). Relationship between inst ruction and achievement. The HLM coefficient for each instruction-related measure is presented in Table 4, each taken from a model equivalent to Model 4 in Table 3. With the exception of weekly mult iple choice assessment use and being not very knowledgeable about the NCTM Standards, for each variable a positive HLM coefficient indicates that a practice aligned with the NCTM Standards positively predicts achievement after controlling for studentand school-level demographics an d other potentially confounding variables. Table 4 HLM Coefficients of Teacher-Rep orted Instruction-Related Fact ors when Predicting Student Mathematics Achievement, After Cont rolling for Demogr aphic Variables Variable Coefficient Calculators -0.1 De-emphasize facts & skills 1.1 Collaborative problem solving 1.1* Non-number curricular emphasis 1.6** Writing about mathematics 0.2 Manipulatives 0.6 Reasoning 1.0 Multiple choice assessment use -0.9 (annual use) -0.4 (weekly use) Knowledge of NCTM standards 4.2* (very knowledgeable) 0.13 (not very knowledgeable) p < .05; ** p < .01; *** p < .001 For each of the three significant coefficients found, the direction of the relationship indicated that NCTM-based instruction and knowledge were positively related to achievement. Specifically, collaborative problem solving, teacher knowledge of the NCTM Standard s, and having a non-number curricular emphasis were all signif icant, positive predictors of fourth-grade achievement. The results of the larger study were more striking, in that the five teacher-reported variables found to significantly predict ac hievement at grade 8 held the same pattern. 21 Reduction of race-related gaps with teacher-reported variables. By comparing the coefficients for Black and Hispanic students before each instructional practice is included in the model (see Table 3, Model 3) and their corresponding coefficien ts after each practice is added (see Model 4), 21 Collaborative problem solving and knowledge of the NCTM Standards were also significant predictors of achievement in grade 8. In addition, calculator use, an emphasis on reasoning, and a deemphasis of facts and skills were also significantly, positively re lated to 8th-grade achievement.


Examining Instruction, Ac hievement, and Equity 19 one can examine the extent to which disparities in access to particular instructional practices might account for a portion of the achievement gaps. If an instructional practice correlates strongly with achievement, and if that practice is utilized much more with White students than with Black or Hispanic students, then we might see a substantial improvement in the slopes for Black and Hispanic students once we add the instructional va riable to the model. In order words, after controlling for the fact that Black and Hispanic students have less access to such instructional practices, we would see the magnitude of the Black and Hispanic coefficients decrease. However, an examination of the change in race-related coefficients for each teacher-reported instructional variable added revealed that the change was near .1 or less. Even when adding all of the instructionrelated variables together in the same model, the change in the slopes was .2 or less at both 4th and 8th grades, indicating a less than 1% change in the 17 point gaps. 22 Hence, these results indicate that the disparities in reform-oriented instruction, as measured in these models by the teacher-reported NAEP data, do not help explain much of the race-related achievement gaps. Yet again, researchers might find a stronger relationship if using more sensitive measures and examining student experiences and growth over several years (Rowan, Correnti, & Miller, 2002). It is worth noting that in the fu ll study, student-reported NCTM-aligned beliefs (math is not simply fact memorization and there are multip le ways to solve problems) were strong, positive predictors of achievement at both 4th and 8th gr ades. Such beliefs are formed over years of students experiences learning mathematics. Race-related gaps slightly but significantly decreased when these and other student-reported factors were included in HLM models. Interaction effects Three interaction effects (Black, Hispanic, and SES) were examined for each of the teacher-reported variables. Of these, only one interaction was significant: Non-number curricular emphasis had a positive interaction with SES, indicating that a non-number curricular emphasis correlated more positively with achiev ement for higher-SES students than lower-SES students. Specifically, as shown previously in Table 4, a student of average SE S having a teacher with non-number emphasis one standard deviation abov e the mean, scored an average of 1.6 points higher than a student whose teacher reported an average amount of emphasis on non-number topics. However, given that the non-number X SES coefficient was 1.2, if a student were 1 standard deviation above the mean in terms of SES, that non-number curricular emphasis advantage would actually be 1.6 + 1.2 = 2.8 points. If a student were 2 standard deviations below the mean SES, then the coefficient would actually be 1.6 2.4= -0.8 points. 23 22 Only an additional 1% of the overall varian ce in achievement was explained when all of the instruction-related variables were added to the demographic model. However, multicollinearity among predictors, as well as the fact that these data are not longitudinal, necessitate caution in interpreting these results. 23 There were three signific ant interactions among the student -reported variables. Two of these involved SES and might be viewed as following a patt ern consistent with the teacher-reported non-number emphasis interaction. The coefficients for student-repo rted calculator use, and student-reported collaborative problem solving were greater for high-SES students th an for low-SES students, suggesting that perhaps these aspects of instruction could further the advantages of high-SES students. Howeve r, again, given the crosssectional (as opposed to longitudinal) nature of the data, and given the number of interactions tested, these findings should be viewed as merely su ggestive of issues for further study.


Education Policy Analysis Archives Vol. 14 No. 14 20 Discussion Reform-Oriented Instruction, Achievement, And Equity This studys descriptive analyses showed rela tively few race-related inequities in fourthgraders access to instructional practices aligned with the NCTM Standards Black and Hispanic students were actually more likely than White studen ts to have a teacher report a strong non-number curricular emphasis and frequent manipulative use. However, consistent with previous findings (Strutchens, et al., 2004), Black and Hispanic studen ts were significantly more likely to be regularly assessed with multiple choice tests than were White students. This studys HLM analyses determined that seve ral reform-oriented factors were significantly related to student achievement after controlling for SES, race, disability status, gender, and school sector. Specifically, teachers non-number curricula r emphasis, use of collaborative problem solving and knowledge of the NCTM Standards were significant, positive predictors of fourth grade mathematics achievement. Although the primary focus of this article is on the grade 4 results, it is worth noting that at both fourth and eighth grades, in every case wh en a teacher-reported, reform-oriented instructional factor was significantly related to achievement, the relationship was positive. Additionally, studentreported beliefs aligned with the NCTM Standards were strong, positive predictors of achievement at grades 4 and 8, and such beliefs were more prevalent for White students than for Black and Hispanic students. Given these differences in beliefs, it is very possible that additional race-related instructional disparities exist that are not ca ptured by the NAEP teacher survey items. Despite the positive relationships between re form-oriented instruction and achievement identified in this study, the overall implications for ways to improve equity are less clear. The reductions in the slopes for Black and Hispanic students produced by adding the teacher-reported instructional variables to the models were very small. Additionally, some instructional practices that correlated positively with achievement, such as teacher-reported non-number curricular emphasis and collaborative problem solving, were actually more prevalent for Black and Hispanic students than for White students. Moreover, the few interaction effects that were significant in the full study suggested ways in which NCTM-based practices correlated more positively with achievement for high-SES students than for low-SES students. Instead of illuminating possible causes of achievement gaps, these facts seem to only furt her complicate the search for instruction-related causes. Overall, the NCTM-based instructional practices examined in this study related positively to achievement when they related at all. The consistency of this pattern at grades 4 and 8 (as revealed in the full study) would seem to provide encouraging news for reformers. However, this and other results of the study must be interpreted with care, as is discussed in the next section. Limitations Given the cross-sectional nature of NAEP data we cannot be sure whether reform-oriented practices actually caused higher achievement, or whether higher-achieving students were more likely to receive reform-oriented instruction. Either case raises important questions about the reasons for the relationship and its ultimate effects on sustaini ng or furthering achievement disparities. There are several additional cautions to be di scussed. First, NAEP classroom practices data are based on teacher self-reports for that school year only. The accuracy of teachers memory of


Examining Instruction, Ac hievement, and Equity 21 practices utilized throughout the year and perceived pressure to portray instruction in particular ways could have affected teachers responses. Additionally the threeor fourpoint scales used on many of the teacher survey items were rough and perhaps insensitive to important differences in teacher practices. Many important questions were not aske d that might move beyond surface features of instruction (e.g., manipulative use) to probe at mo re fundamental instructional issues (e.g., the extent to which instruction builds upon and centers around student understanding), as well as to identify larger, structural inequities (e.g., school funding). Still, the fact that when put in a factor analysis, most teacher-reported instruction-related variables clumped with each other in sensible ways, and the fact that several significant relationships b etween student demographics and instructional factors were found, indicate that the NAEP mathematics teacher survey questions are, indeed, measuring some important aspects of variability in instruction. Again, there is no measure of prior achievemen t in NAEP, and so it is students overall achievement, and not growth in achievement, that serves as the outcome variable. This limitation, combined with the limits of the teacher-reported data noted above, suggest that this study may be overly conservative in determining the strength of impact that instructional measures can have on both student learning and on achievement gaps. If teacher practices were measured with more sensitive measures over time, and if the data allo wed for examinations of student achievement gains, it is likely that we would see a greater instructional impact on achievement and race-related gaps than what is indicated here (Rowan, Correnti, & Miller, 2002). On the other hand, standard errors for instruction-related coefficients were perhaps smaller (less conservative) than they would have been if the clustering of students within classrooms was accounted for in the models (i.e., if teachers could have been treated at a classroom level). Hence, ag ain, the results of this study should be viewed as merely suggestive of relationships that are important to explore further using in-depth, longitudinal methods that can take student, teacher/classroom and school levels into consideration. Comparison across Two Studies We now return to the question of how this st udy compares with that of Wenglinsky (2004), who also used HLM with the 2000 NAEP mathem atics data to examine relationships among instruction, achievement and equity. Some findin gs of the two studies were complementary. For example, this study found a positive relationshi p between teachers non-number curricular emphasis and achievement. Wenglinsky also obtained positive coefficients for teachers emphasis on geometry, measurement and algebra (although only the coefficient for geometry was significant). There were also consistencies in some factors that did not correlate with achievement in either study, including manipulative use and writing abou t mathematics, as well as teachers college major and degree (which were subsequently deleted from models in this study due to their lack of significance). However, vastly different conclusions were reached about the potential for particular instructional practices to close achievement gaps This study identified only weak, insignificant interactions between particular in structional practices and Black and Hispanic student achievement. Although some significant relationships between instructional practices and overall achievement were found, these relationships did not vary significantly by student race. Additionally, all relationships between instruction and achievement fo und in this study are interpreted with great caution due to the cross-sectional nature of NAEP data. In contrast, Wenglinsky concluded from his stud y that a series of instructional practices, when used in concert, can substantially redu ce both the Black-White and Latino-White achievement gaps (p. 3). Specifically, Wenglinsky asserted that frequent test taking enlarges the Black-White gap,


Education Policy Analysis Archives Vol. 14 No. 14 22 and that an emphasis on measurement helps reduce the gap. (Although not the main point of the concerns raised here, it is worth noting that the actual coefficients Wenglinsky provided appear to indicate the oppositean emphasis on measurement a ppeared to predict a la rger Black-White gap, while frequent testing predicted a smaller gap.) He also concluded that an emphasis on data analysis is particularly beneficial for Hispanic students. There are four fundamental differences between the two studies that underlie their divergent conclusions. First, Wenglinsky aggregated teacher-lev el data to the school level, whereas this study treated teacher-level data at the student-level. Again, there was no classroom or teacher identification code in the NAEP data, making it nearly impossibl e to treat classroom as a separate level in HLM. Confidentiality concerns might partially underlie NAEPs exclusion of teacher codes, but another reason is that studentsnot teachersare randomly selected within a sc hool. Therefore, NAEP experts recommend connecting teachers with indivi dual student data when making claims about teachers. This was the approach taken in this stud y, compatible with its primary focus on disparities among students classroom experiences and achiev ement. On the other hand, given Wenglinskys primary focus on NCLB and within-school practices and achievement gaps, his decision to aggregate instructional practices to the school level is certainly conceptually defensible, having the potential to create a stronger measure of the general instructional climate of the school. The interactions between student race and instruction were then trea ted as cross-level interactions in Wenglinskys study, which fit well with his focus on within-school gaps (whereas in this study, the interactions were treated at the student-level). Overall, the difference in treatment of the teacher data when designing the HLM models (analyzing it at th e student versus the school level) may be one contributor to the differences in the studies findings. However a second and more important difference between the studies is the number of variables included simultaneously in the models. The study reported here utilized factor analysis to reduce roughly 30 instruction-related variables to nine factors, which were then each examined in separate models. In contrast, in addition to seve ral demographic measures, Wenglinsky included 20 variables pertaining to teaching practices and 3 variables pertaining to teacher background in his model to predict the main intercept, and he also used the 23 teacher-related variables to predict the within-school slope (or gap) for both Black an d Hispanic students. Wenglinsky found that the slopes for Black and Hispanic students were signific ant before adding the teacher-related variables but no longer significant after adding those variables. Wenglinsky then concluded, Thus, by including the 20 instructional practices, the second HLM can explain away the entire within-school racial gap (p. 16). Wenglinskys full model, then, involved the dete rmination of over 70 coefficients. Forty-six of these were predicting the Black or Hispanic slope, the primary focus of his study. By chance alone we would expect roughly 4 or 5 of those 46 predi ctors to be statistically significant at the p < .1 level (the base level of significance he used), with 2 or 3 of those significant at the p < .05 level. In fact, his model identified only 3 significant predictors of the Black or Hispanic slopes (2 at the .05 level, and 1 at the .1 level). Hence, it is quite possible that these 3 variables are false positives. In fact, it is clear that Wenglinskys full model is problematic, as evidenced by inflated standard errors and the fact that the Hispanic slope went from being -8 (with a standard error of 1) in his base model, to a positive 27 points (with standard error of 22). The slope for Black students went from -16 (standard error of 1) to -9 (standard error of 26). It is worth noting that the Black-White gap of -9 was considered eliminated because the gap was not si gnificant, yet the standard error had became so large that even the original Black-White gap would not be significant. Again, the huge reversal of the Hispanic slope suggests serious instability in the model, likely caused by the large number of predictors, many of which are collinear (as evidenced by the results of the factor analyses in the study reported here).


Examining Instruction, Ac hievement, and Equity 23 Hence, while the composites of instructional measures used in the study reported here can be more difficult to interpret than the individual variables included in Wenglinskys models, his full model reveals the danger of including too many variables in an HLM, particularly at the school level, as Wenglinsky, himself notes in a footnote: degrees of freedom are sharply reduced by including so many school-level independent variables (p. 16). Other, minor differences between the studies methods could be discussed, including the exact calculations of school race or SES measures, or the fact that some variables, such as disability or time on math were included in one study but not the other. However, the two remaining differences between the studies that merit discussion lie not in their methods of analysis but in their interpretation of the results. First, NAEP data are cross-sectionalnot l ongitudinaland therefore not intended for drawing causal conclusions regarding instruction and achievement. Wenglinsky himself notes this in his brief, isolated discussion of the limitations of his study, explaining: This means that nothing is known about the causal direction of the results (pp. 16). And yet causal language and conclusions are prevalent throughout the remainder of the article, beginning with the abstract in which he states that he uses HLM with NAEP data to identify instructional practices that reduce the achievement gap. It finds that, even when taking student background into account, instructional practices can make a substantial difference. Wenglinskys optimistic conclusion that, according to his results, school administrators can succeed at closing the racial achievement gap in their schools, (p. 17) is unwarranted. Again, a correlation between particular practices and achievem ent may not be causal, particularly given that another plausible explanation existsi.e., that higher achieving students might tend to receive different instruction than lower-achieving students. And again, the large number of predictors in Wenglinskys full model should also raise major concerns about drawing conclusions from the particular relationships identified. Second, even if one could conclude from Wenglinskys stud y that particular instructional practices reduced the within-school Black-White an d Hispanic-White gaps to 0, one must interpret this with the understanding that SES was controlled for in the models, and therefore it is the racerelated leftover gap (the part not related to SES) within schools that was reduced in the models. Hence, in practice, there would still be very large within-school gaps between Black and White students in most schools, as well as between Hispanic and White students, given the strong correlation between race and SES. Additionally, th e focus of NCLB and Wenglinskys study on within-school gaps ignores raceand SES-related gaps between schools, which dangerously places responsibility for gap reductions on school personnel and ignores larger societal inequities, such as persistent disparities in community resources that schools alone cannot overcome (Berliner, 2005). Implications for Future Research Uses and Abuses of Cross-Sectional Data One can understand why researchers utilizing NAEP and other cross-sectional data are tempted to overstate rather than understate the conclusions that can be drawn. Soft claims surrounded by a sea of caveats tend to be ignor ed by publishers, the popular media, and policy makers. This dynamic points toward the need for studies indicating no relationship between important variables to be reported along with thos e with more exciting conclusions. Critiques of NAEPs cross-sectional nature also raise questions about the usefulness (or lack thereof) of NAEP and other similar large-scale data sets (Christen sen & Angel, 2005; Lubienski & Lubienski, 2006).


Education Policy Analysis Archives Vol. 14 No. 14 24 One might wonder, if causal claims cannot be made from studies of NAEP data, then what good is NAEP? In the study reported here, NAEP data were useful for examining whether reform-oriented instructional practices were distributed equally acro ss U.S. students, regardless of race. This is aligned with NAEPs strengthdescribing current achievement and instruction-related patterns on a large, nationally-representative scale. Analyses of NAEP data can also shed light on which instructional practices do and do not correlate with student achievement while controlling for many potential confounding variables. However, a measure of prior achievement is not available in NAEP, and the causal order of any relationships identified will be unclear. Because of their potential for widespread attention and influence on policy, large-scale studies, in particular, must be communicated with care, with proper cautions regarding the limitations of the results emphasized. In the case of the study reported here, wheth er particular NCTM-based practices caused higher achievement, or whether high-achieving st udents were more likely to be taught with NCTMbased practices is unclear. Still, the relationships identified raise important questions for further research within classrooms. As Wenglinsky also notes, NAEP analyses cannot replace studies involving in-depth classroom observations. Further Research on Equity After multiple reform efforts aimed at changing mathematics instruction and reducing inequities, much work remains. One finding that is clear in both Wenglinskys study and this study is that there are large raceand SES-related achievem ent gaps, and even after controlling for SES using multiple demographic variables, the unexplained race-related gap within schools is disturbingly large. 24 NAEP offers one avenue for examining disparit ies in achievement and classroom practices. The patterns identified in this study suggest directions for additional longitudinal and qualitative studies that examine causes of, and ways to address, the patterns identified here. Overall, researchers should continue to examine achievement disparities, considering instructional factors identified in this study, as well as other potential influences not considered here, such as differential access to various resources at both home and school. References Anyon, J. (1981). Social class and school knowledge. Curriculum Inquiry, 11 3. Banks, J. A. (1988). Ethnicity, class, cognitive, and motivational styles : Research and teaching implications. Journal of Negro Education, 57 (4), 452. Berliner, David C. (2005). Our impoverished view of educational reform. Teachers College Record, 108 949. 24 Rothstein (2004) rightly points out that the SES va riables used in studies such as these are limited because they do not measure important historical and contextual factors, such as accumulated family wealth. It is likely that a better SES measure would account fo r a greater proportion of race-related achievement gaps. Still, the measure used in this study is more compre hensive than SES variables typically utilized in NAEP reports.


Examining Instruction, Ac hievement, and Equity 25 Boaler, J. (2002). Learning from teaching: Expl oring the relationship between reform curriculum and equity. Journal for Research in Mathematics Education 33 (4), 239. Braswell, J. S., Daane, M. C., & W. S. Grigg (2003) The nation's report card: Mathematics highlights 2003 Washington, DC: U.S. Department of Education. Institute for Education Sciences, National Center for Education Statistics. Braswell, J. S., Lutkis A. D., Grigg, W. S., Santapau, S. L., Tay-Lim, B., & Johnson, M (2001). The nations report card: Mathematics 2000 Washington DC: U.S. Department of Education, National Center for Education Statistics. Bryk, A. S., Lee, V. E., & Holland, P. B. (1993). Catholic schools and the common good Cambridge MA: Harvar d University Press. Bryk, A. S., & Rauden bush, S. W. (1992). Hierarchical linear models: Applications and data analysis methods Newbury Park: Sage Publications. Campbell, J. R., Hombo, C. M., & J. Mazzeo (2000). NAEP 1999 trends in academic progress: Three decades of student performance, NCES 200069 Washington, DC: U.S. Department of Education. Christensen, J. & Angel, L. (2005 ). What NAEP cant tell us about charter school effectiveness. Education Week, 25(14), 35, 37. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Lawrence Erlbaum. Cook, P. & Ludwig, J. (1998). The burden of acting White. In C. Jencks & M. Phillips (Eds.), The Black-White test score gap (pp. 375). Washington, DC: Brookings Institution Press. Cooper, B & Dunne, M. (2000). Assessing childrens math ematical knowledge: So cial class, sex and problem-solving. Buckingham: Open University Press, UK. Duberman, L. (1976). Social inequality: Clas s and caste in America Philadelphia, J.B. Lippincott. Fennema, E., Carpenter, T. P., Jacobs, V. R., Fr anke, M. L., & Levi, L. (1998). A longitudinal study of gender differences in youn g childrens mathematical thinking. Educational Researcher, 27 (5), 6. Ferguson, R. F. (1998a). Teache rs perceptions and expectations and the Black-White test score gap. In C. Jencks & M. Phillips (Eds.), The Black-White test score gap (pp. 273). Washington, DC: Brookings Institution Press. Ferguson, R. F. (1998b). Comment on Cook & Ludwigs The burden of acting White: Do Black adolescents disparage academic achievement? In C. Jencks & M. Phillips (Eds.), The Black-White test score gap (pp. 394). Washington, DC: Brookings Institution Press.


Education Policy Analysis Archives Vol. 14 No. 14 26 Ferguson, R. F. (1998c). Can schools narrow the Black-White test score gap?. In C. Jencks & M. Phillips (Eds.), The Black-White test score gap (pp. 318). Washington, DC: Brookings Institution Press. Foegen, A. (2004). Students with disabiliti es and the 2000 NAEP mathematics assessment: achievement levels and sample characteristics Paper presented at the annual meeting of the American Educational Research Association, San Diego. Hickey, D. T., Moore, A. L., & Pellegrino, J. W. (2001). The motivational and academic consequences of elementary mathematics en vironments: Do constructivist innovations and reforms make a difference? American Educational Research Journal, 38 (3), 611. Jencks, C., & Phillips, M. (Eds.) (1998). The Black-White test score gap Washington, DC: Brookings Institution Press. Johnson, E. G. (1992). The desi gn of the National Assessment of Educational Progress. Journal of Educational Measurement, 29 (2) 95. Johnson, E. G. & Rust, K. F. (1992). Population inferences and variance estimation for NAEP data. Journal of Educational Statistics 17(2) 175. Kloosterman, P., & Lester, F. K., Jr. (Eds.) (2004). Results and interpretations of the 1990 through 2000 mathematics assessments of the Na tional Assessment of Educational Progress. Reston, VA: National Council of Teachers of Mathematics. Ladson-Billings, G. (1997). It doesnt add up: African Amer ican students mathematics achievement. Journal for Research in Mathematics Education, 28 (6), 697. Lee, J. (2002). Racial and ethnic achievement gap trends: Reversing the progress toward equity? Educational Researcher, 31(1), 3. Loveless, T., & Diperna, P. (2000). How well are American studen ts learning? Focus on math achievement (Brown Center Report, Vol. 1, No 1). Washington, DC: Brookings Institution. Lubienski, C., & Lubienski, S. T. (2006). Charter, private, public schools and academic achievement: New evidence from NAEP mathematics data New York: National Center for the Study of Privatization in Education, Teachers College, Columbia University. Retrieved May 29, 2006, from Lubienski, S. T. (2000a). A clash of class cultures? Students experi ences in a discussionintensive seventh-grade mathematics classroom. Elementary School Journal 100 377 403. Lubienski, S. T. (2000b). Problem solving as a means toward mathematics for all: An exploratory look through a class lens. Journal for Research in Mathematics Education 31(4), 45482.


Examining Instruction, Ac hievement, and Equity 27 Lubienski, S. T. (2002). A closer look at Bl ack-White mathematics gaps: Intersections of race and SES in NAEP achievement an d instructional practices data, Journal of Negro Education 71 (4), 2697. Lubienski, S. T., & Bowen, A. (2000). Whos counting? A survey of mathematics education research 1982. Journal for Research in Mathematics Education 31 (5), 626. Lubienski, S. T., Camburn, E. & Shelley, M. C. (2004). Reform-oriented mathematics instruction, achievement, and equity: Ex aminations of race and SES in 2000 Main NAEP Data Report submitted to the National Center for Education Statistics. Lubienski, S. T. & Lubiensk i, C. (2006). What NAEP can tell us about school achievement. Education Week, 25(26), 28, 30. Lubienski, S. T., McGraw, R., & Strutchens, M. (2004). NAEP findin gs regarding gender: Mathematics achievement, student affect, and learning practices. In P. Kloosterman., & F. K. Lester, Jr. (Eds.) Results and interpretations of the 1990 through 2000 mathematics assessments of the National Assessme nt of Educational Progress (pp. 305). Reston, VA: NCTM. Lubienski, S. T. & Shelley, M. C. (2003). A closer look at U.S. ma thematics instruction and achievement: Examinations of race an d SES in a decade of NAEP data. Paper presented at the American Educational Research Association, Chicago. Means, B. & Knapp, M. S. (1991, December). Cog nitive approaches to teaching advanced skills to educationally disa dvantaged students. Phi Delta Kappan, 73, 282. National Council of Teache rs of Mathematics. (1989). Curriculum and evaluation Standards for school mathematics. Reston, VA: Author. National Council of Teache rs of Mathematics. (1991). Professional standards for teaching mathematics Reston, VA: Author. National Council of Teache rs of Mathematics. (1995). Assessment standards for school mathematics Reston, VA: Author. National Council of Teache rs of Mathematics, (2000). Principles and standards for school mathematics Reston, VA: NCTM. Newmann, F. M., & Wehl age, G. G. (1995). Successful school restructuring : A report to the public and educators. Washington, DC: Office of Educat ional Research and Improvement. Ogbu, J. U. (1995). Understanding cultural diversity and learning. In J. A. Banks & C. A. McGee Banks (Eds.), Handbook of research on multicultural education (pp. 58293). New York: Macmillan. Peng, S. S., Wright, D. & S. T. Hill (1995). Understanding racial-ethnic differences in secondary school science and mathematics achievement. Washington, DC: U.S. Department of Education, National Center for Education Statistics.


Education Policy Analysis Archives Vol. 14 No. 14 28 Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data analysis methods (2nd ed.). Thousa nd Oaks, CA: Sage. Raudenbush, S.W., Fotiu, R., & Cheong, Y.F. (1 998). Inequality of a ccess to educational opportunity: a national report card for eighth grade math. Educational Evaluation and Policy Analysis, 20 253. Reys, R., Reys, B., Lapan, R., Holliday, G., and Wasman, D. (200 3). Assessing the impact of Standards -based middle grades mathematics textbooks on student achievement. Journal for Research in Mathematics Education, 34 (1), 74. Riordan, J. E., and Noyce, P. E. (2001). The impact of tw o standards-base d mathematics curricula on student achievement in Massachusetts. Journal for Research in Mathematical Education, 32 (4), 368. Rothstein, R. (2004). Class and schools: Using social, economic and educational reform to close the Black-White achievement gap Washington, DC: Economic Policy Institute. Rowan, B., Correnti, R. & Miller, R. J. (2002). What large-scale survey research tells us about teacher effects on student achi evement: Insights from the Prospects study of Elementary Schools. Teachers College Record, 104 1525. Rubin, D. (1987). Multiple imputation for nonresponse in surveys. New York, NY: Wiley. Schoenfeld (2002). Making mathematics work for al l children: Issues of standards, testing, and equity. Educational Researcher, 31(1), 13. Senk, S. & Thompson, D. (Eds.) (2003). Standards-based school mat hematics curricula: What are they? What do students learn ? Hillsdale, NJ: Lawrence Earlbaum Associates, Inc. Silver, E., Smith, M. S., & Nelson, B. S. (1995 ). The QUASAR Project: Equity concerns meet mathematics reforms in the mi ddle school. In W. G. Seca da, E. Fennema, & L. B. Adajian (Eds.), New directions in equity in mathematics education (pp. 956). New York: Cambridge University Press. Steele, C. M. & Aronson, J. (1998). Stereotype threat and the te st performance of academically successful African Americans. In C. Jencks & M. Phillips (Eds.), The Black-White test score gap (pp. 401). Washington, DC: Brookings Institution Press. Stiff, L. (1990). African-American students and the promise of the Curriculum and Evaluation Standards In T. J. Cooney & C. R. Hirsch (Eds.), Teaching and learning mathematics in the 1990s (Yearbook of the National Council of Teachers of Mathematics, pp. 152), Reston, VA: NCTM. Strutchens, M., Lubienski, S. T., McGraw, R. & Westbrook, S. K. (2004). NAEP findings regarding race/ethnicity: Stud ents performance, school ex periences, attitudes/beliefs and family influences. In P. Kloosterman., & F. K. Lester, Jr. (Eds.) Results and interpretations of the 1990 through 2000 mathematics assessments of the National Assessment of Educational Progress (pp. 269). Reston, VA: NCTM.


Examining Instruction, Ac hievement, and Equity 29 Strutchens, M. E., & Silver, E. A. (2000). NAEP findings rega rding race/ethnicity: Students performance, school ex periences, and attitude s and beliefs. In E. A. Silver & P. A. Kenney (Eds.), Results from the seventh mathematics assessment of the National Assessment of Educational Progress Reston, VA: National Council of Teachers of Mathematics. Swanson, C. B., & Stevens on, D. L. (2002, Spring). Standards -based reform in practice: Evidence on state policy and classroom instruction from th e NAEP state assessments. Educational Evaluation and Policy Analysis, 24 (1), 127. Tate, W. F. (1997). Race-ethni city, SES, gender, and language proficiency in mathematics achievement: An update. Journal for Research in Mathematics Education 28 652. Thompson, B. (2002). Statistical, practical, and clinical: How many kinds of significance do counselors need to consider? Journal of Counseling and Development, 80 64. Weis, L. (Ed.) (1988). Class, race, & gender in American education Albany: State University of New York Press. Wenglinsky, H. (2004, November 23). Closing the racial achievement gap: The role of reforming instructional practices. Educational Policy Analysis Archives 12 (64). Retrieved December 2, 2004, from


Education Policy Analysis Archives Vol. 14 No. 14 30 About the Author Sarah Theule Lubienski University of Illinois at Urbana-Champaign Email: Sarah Theule Lubienski is an associate professor in the College of Education at the University of Illinois, Urbana-C hampaign. Her scholarship cen ters around intersections of education and equity, with a particular focus on mathematics achievement, instruction, and reform. Her recent NAEP-related research has been supported by two Secondary Analysis Grants, funded by the National Center of Education Statistics. She is the chairperson of the NAEP Studies Special Interest Group of the Am erican Educational Research Association.


Examining Instruction, Ac hievement, and Equity 31 EDUCATION POLICY ANALYSIS ARCHIVES Editor: Sherman Dorn, University of South Florida Production Assistant: Chris Murre ll, Arizona State University General questions about ap propriateness of topics or particular articles may be addressed to the Editor, Sherman Dorn, Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Robert Bickel Marshall University Gregory Camilli Rutgers University Casey Cobb University of Connecticut Linda Darling-Hammond Stanford University Gunapala Edirisooriya Youngstown State University Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Gene V Glass Arizona State University Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Ohio University William Hunter University of Ontario Institute of Technology Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Anthony G. Rud Jr. Purdue University Michael Scriven Western Michigan University Terrence G. Wiley Arizona State University John Willinsky University of British Columbia


Education Policy Analysis Archives Vol. 14 No. 14 32 EDUCATION POLICY ANALYSIS ARCHIVES English-language Graduate -Student Editorial Board Noga Admon New York University Jessica Allen University of Colorado Cheryl Aman University of British Columbia Anne Black University of Connecticut Marisa Cannata Michigan State University Chad d'Entremont Teachers College Columbia University Carol Da Silva Harvard University Tara Donahue Michigan State University Camille Farrington University of Illinois Chicago Chris Frey Indiana University Amy Garrett Dikkers University of Minnesota Misty Ginicola Yale University Jake Gross Indiana University Hee Kyung Hong Loyola University Chicago Jennifer Lloyd University of British Columbia Heather Lord Yale University Shereeza Mohammed Florida Atlantic University Ben Superfine University of Michigan John Weathers University of Pennsylvania Kyo Yamashiro University of California Los Angeles


Examining Instruction, Ac hievement, and Equity 33 Archivos Analticos de Polticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998003) Roberto Rodrguez Gmez Editorial Board Hugo Aboites Universidad Autnoma Metropolitana-Xochimilco Adrin Acosta Universidad de Guadalajara Mxico Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educacin, Chile Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educacin, Argentina Teresa Bracho Centro de Investigacin y Docencia Econmica-CIDE Alejandro Canales Universidad Nacional Autnoma de Mxico Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagoga Popular, Per Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernndez Enguita Universidad de Salamanca. Espaa Gaudncio Frigotto Universidade Estadual do Rio de Janeiro, Brasil Rollin Kent Universidad Autnoma de Puebla. Puebla, Mxico Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California Mara Loreto Egaa Programa Interdisciplinario de Investigacin en Educacin Mariano Narodowski Universidad To rcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Polticas Educativas, Per Vanilda Paiva Universidade Estadual Do Rio De Janeiro, Brasil Miguel Pereira Catedratico Un iversidad de Granada, Espaa Angel Ignacio Prez Gmez Universidad de Mlaga Mnica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de So Paulo Diana Rhoten Social Science Research Council, New York, New York Jos Gimeno Sacristn Universidad de Valencia, Espaa Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, Mxico Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas Publicas-Universidad de Buenos Aires, Argentina Antonio Teodoro Universidade Lusfona Lisboa, Carlos A. Torres UCLA Jurjo Torres Santom Universidad de la Corua, Espaa


Download Options

Choose Size
Choose file type
Cite this item close


Cras ut cursus ante, a fringilla nunc. Mauris lorem nunc, cursus sit amet enim ac, vehicula vestibulum mi. Mauris viverra nisl vel enim faucibus porta. Praesent sit amet ornare diam, non finibus nulla.


Cras efficitur magna et sapien varius, luctus ullamcorper dolor convallis. Orci varius natoque penatibus et magnis dis parturient montes, nascetur ridiculus mus. Fusce sit amet justo ut erat laoreet congue sed a ante.


Phasellus ornare in augue eu imperdiet. Donec malesuada sapien ante, at vehicula orci tempor molestie. Proin vitae urna elit. Pellentesque vitae nisi et diam euismod malesuada aliquet non erat.


Nunc fringilla dolor ut dictum placerat. Proin ac neque rutrum, consectetur ligula id, laoreet ligula. Nulla lorem massa, consectetur vitae consequat in, lobortis at dolor. Nunc sed leo odio.