USF Libraries
USF Digital Collections

Educational policy analysis archives

MISSING IMAGE

Material Information

Title:
Educational policy analysis archives
Physical Description:
Serial
Language:
English
Creator:
Arizona State University
University of South Florida
Publisher:
Arizona State University
University of South Florida.
Place of Publication:
Tempe, Ariz
Tampa, Fla
Publication Date:

Subjects

Subjects / Keywords:
Education -- Research -- Periodicals   ( lcsh )
Genre:
non-fiction   ( marcgt )
serial   ( sobekcm )

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
usfldc doi - E11-00458
usfldc handle - e11.458
System ID:
SFS0024511:00457


This item is only available as the following downloads:


Full Text
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20059999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00458
0 245
Educational policy analysis archives.
n Vol. 13, no. 36 (August 31, 2005).
260
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c August 31, 2005
505
What do Klein et al. tell us about test scores in Texas? / Laurence A. Toenjes.
650
Education
x Research
v Periodicals.
2 710
Arizona State University.
University of South Florida.
1 773
t Education Policy Analysis Archives (EPAA)
4 856
u http://digital.lib.usf.edu/?e11.458


xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 13issue 36series Year mods:caption 20052005Month August8Day 3131mods:originInfo mods:dateIssued iso8601 2005-08-31



PAGE 1

EDUCATION POLICY ANALYSIS ARCHIVES A peer-reviewed scholarly journal Editor: Sherman Dorn College of Education University of South Florida Copyright is retained by the first or sole au thor, who grants right of first publication to the Education Policy Analysis Archives EPAA is published jointly by the Colleges of Education at Arizona State University and the University of South Florida. Articles are indexed in the Director y of Open Access Journals (www.doaj.org) and by H.W. Wilson & Co. Volume 13 Number 36 August 30, 20 05 ISSN 1068–2341 What Do Klein et al. Tell Us About Test Scores in Texas?1 Laurence A. Toenjes United States Citation: Toenjes, L. A. (2005, Au gust 30). What do Klein et al tell us about test scores in Texas? Education Policy Analysis Archives, 13 (36). Retrieved [date] from http://epaa.asu.ed u/epaa/v13n36/. Abstract A paper appearing in this journal by Klein, Hamilton, McCaffrey and Stecher (2000) attempted to raise serious questi ons about the validity of the gains in student performance as measured by Texas’ standardized test, the Texas Assessment of Academic Skills (TAAS). Pa rt of their analysis was based on the results of three tests which they adminis tered to 2,000 fifth grade students in 20 Texas schools. Although Klein et al. indica ted that the 20 school s were not selected in a way which would insure th at they were representative of the nearly 3,000 Texas schools that enrolled fifth graders, genera lizations based upon the results for those schools were nonetheless offered. The purpose of this s hort paper is to demonstrate just how unrepresentative the 20 schools used by Klein et al. actually were, and in so doing to cast doub t on certain of their conclusions. 1 The author wishes to thank Prof. Jon Lorence for numerous constructive suggestions. Accepted under the editorship of Sherman Dorn. Send commenta ry to Casey Cobb (casey.cobb@uconn.edu).

PAGE 2

Education Policy Analysis Archives Vol. 13 No. 36 2 This is a reply to the paper “What do Test Scores in Texas Tell Us?”, by Stephen P. Klein, Laura S. Hamilton, Daniel F. McCaffrey, and Brian M. Stecher (2000). Klein et al. (2000) presented numerous findin gs questioning the effectiveness of the Texas accountability system. They specifically challenged the validity of the Texas Assessment of Academic Skills (TAAS) test, required of all eligible students in the Texas public school system. Their conclusions were based on several sets of analys es. First they estimated the association between student test scores and student economic status. Ot her analyses examined the relationships between student performance on the TAAS and three different tests which they administer ed in the spring of 1997 to some 2,000 fifth graders in 20 schools, described only as “a mix of 20 urban and suburban schools in Texas.” Although results based on individual students were as anticipated, aggregate school-level findings indicated that high TAAS performance was largely unrelated to scores on standardized tests not used in the state accountability system. In addition, Klein et al. did not find the expected negative linear association between the percentages of disadvantaged children and test scores commonly reported in the research literature. Although Klein et al. suggested that the discrepancies “could be due to the unique charac teristics of the 20 schools in our study or other factors,” they nonetheless concluded that their findings indicated that observed increases in Texas student academic performance reported by Gris smer et al. (2000) were highly suspect. However, analyses based on all Texas public schools reported in this note reveals that part of the Klein et al. findings were incorrect because the campuses selected for their study were not representative of the population of elementary schools. Proportion of Students Receiving Subsidized Lunches Figure 1. Campus Mean Math Scores Versus Proportion of Disadvantaged Students, Texas Fifth Graders, 1997 TAAS Test Figure 1 (above) shows the relationship between student TAAS test scores and economic disadvantage based on Texas public schools which tested fifth-grade students in mathematics in the spring of 1997. To be consistent with Klein et al ., only 1997 fifth grade TAAS mathematics scores are utilized. The results are shown for non-special education students whose test results were included in the state’s accountability rating system The scatterplot in Figure 1 is based on 225,433

PAGE 3

What Do Klein et al. Tell Us About Test Scores in Texas? 3 students attending 2,825 campuses. A few school s which tested fewer than 20 students were excluded from the analysis in order to obtain re liable estimates of average school performance. The test performance measure, shown on the vertical axis is the simple average for each campus of the number of questions answered correctly. The horizontal axis repres ents the proportion of economically disadvantaged tested students as mea sured by enrollment in the federally-subsidized free or reduced-price lunch programs. Figure 2. Diagram appeared as Figure 5. In Klein, et al. (2000) Although campuses testing fewer than 20 students are omitted, the picture is intended to show the universe from which the twenty campuses used in the study by Klein, et al, were drawn. The corresponding scatterplot based on the 20 campuses analyzed by Klein et al. is reproduced in Figure 2 (above). When the figure from the Klein study is compared with the figure based on all Texas schools, it is apparent how unrepresen tative those 20 schools were with respect to characterizing the relationship between performan ce on the fifth grade math TAAS test and the average proportion of students who were economic ally disadvantaged. Consequently, the Pearson correlation between average math performance an d economic disadvantage based on the 2,825 campuses differs considerably from the correlati on Klein et al. reported. Whereas their correlation was only 0.13, and positive, the correlation from the more comprehensive set of campuses shown is a -0.55. This latter figure has th e expected large negative value, an d is comparable to the correlation coefficients which they found between SES an d the non-TAAS tests wh ich they administered. Those correlations ranged between –0.66 to –0.76. Although there is a degree of heteroscedasticity in the figure based on the more representative se t of schools, due perhaps to ceiling-effect in the low-poverty schools, there is no indication of the u-sh aped association evident in Figure 2, based on the Klein et al. data. In sum, the results derived from the tests Stech er and Klein administered in 1997 to 2,000 students in 20 schools are highly unrepresentative of Texas elementary schools and should not be used to draw conclusions about the relationship be tween poverty and school performance in Texas. The analysis presented here supports the va lidity of the TAAS test. Additional, more comprehensive, school-level analyses examining the association between performance on the TAAS

PAGE 4

Education Policy Analysis Archives Vol. 13 No. 36 4 and other standardized tests are required to ascer tain whether the Klein et al. cross-test findings based on 20 elementary campuses can be generalized. There is no intent here to suggest that the Texas accountability system is immune from criticism. Analyses of NAEP data have also raised questions pertaining to the validity of learning gains reported on Texas. To illustrate, even RAND Corporation researchers differ among themselves in their interpretation of the meaning of NAEP test results for Texas. Whereas Klein et al. (2000) contended NAEP findings showed th e Texas accountability system is ineffective, Grissmer et al. (2000) suggested that the implementa tion of stricter state educational requirements in Texas helped improve student academic learning. The current note does not address this issue of validity (but see Toenjes and Dworkin 2002; Toenje s, Dworkin, Lorence and Hill 2002). My findings are more limited, but clearly demonstrate that the absence of a negative relationship between schoollevel test performance and economic disadvan tage reported by Klein et al., based on 20 unrepresentative schools, does not support their cl aim that the TAAS is an invalid measure of student achievement. References Grissmer, D. W., Flanagan, A., Ka wata, J., Williamson, S. (2000), Improving Student Achievement: What Do State NAEP T est Scores Tell Us, RAND, Santa Monica, Calif. Klein, Stephen P., Hamilton, Laura S., McCaffrey, Daniel F., and Stecher, Brian M (2000) “What do Test Scores in Texas Tell Us?”, Education Policy Analysis Archives 8(49). Retrieved March 22, 2005 from http ://epaa.asu.edu/epaa/v8n49/. Toenjes, L.A. and Dworkin, A.G. (2002). “Are increasing test scores in Texas really a myth, or is Haney's myth a myth?”, Education Policy Analysis Archives 10 (17). Retrieved March 22, 2005 from http://epaa.asu.edu/epaa/v10n17/. Toenjes, L.A., Dworkin, A.G., Lorence, J. Hill, A.N. (2002), “H igh-Stakes Testing, Accountability, and Student Achiev ement in Texas and Houston”, in Bridging the Achievement Gap John E. Chubb and Tom Loveless, editors, Brookings, Washington, D.C. About the Author Laurence A. Toenjes Email: ltoenjes@aol.com Laurence A. Toenjes was Research Associate Profes sor of Sociology and Co-Founder of the Sociology of Education Research Group (S ERG) at the University of Houston. He holds a Ph.D. degree in Economics from Southern Illinois University. R etired three years ago, Toenjes recently complete d a sailing voyage from Texa s to Seattle by way of Ha waii. Interestingly, he noticed the weekly posting of lo cal school testing re sults on the bulletin board at the small Keaukaha Market near the harbor in Hilo during his stay there. There is no escape.

PAGE 5

What Do Klein et al. Tell Us About Test Scores in Texas? 5 EDUCATION POLICY ANALYSIS ARCHIVES http://epaa.asu.edu Editor: Sherman Dorn, University of South Florida Production Assistant: Chris Murre ll, Arizona State University General questions about ap propriateness of topics or particular articles may be addressed to the Editor, Sherman Dorn, epaa-editor@shermandorn.com. Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Casey Cobb University of Connecticut Linda Darling-Hammond Stanford University Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Gene V Glass Arizona State University Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Anthony G. Rud Jr. Purdue University Michael Scriven Western Michigan University Terrence G. Wiley Arizona State University John Willinsky University of British Columbia

PAGE 6

Education Policy Analysis Archives Vol. 13 No. 36 6 EDUCATION POLICY ANALYSIS ARCHIVES English-language Graduate -Student Editorial Board Noga Admon New York University Jessica Allen University of Colorado Cheryl Aman University of British Columbia Anne Black University of Connecticut Marisa Burian-Fitzgerald Michigan State University Chad d'Entremont Teachers College Columbia University Carol Da Silva Harvard University Tara Donahue Michigan State University Camille Farrington University of Illinois Chicago Chris Frey Indiana University Amy Garrett Dikkers University of Minnesota Misty Ginicola Yale University Jake Gross Indiana University Hee Kyung Hong Loyola University Chicago Jennifer Lloyd University of British Columbia Heather Lord Yale University Shereeza Mohammed Florida Atlantic University Ben Superfine University of Michigan John Weathers University of Pennsylvania Kyo Yamashiro University of California Los Angeles

PAGE 7

What Do Klein et al. Tell Us About Test Scores in Texas? 7 Archivos Analticos de Polticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998—2003) Roberto Rodrguez Gmez Editorial Board Hugo Aboites Universidad Autnoma Metropolitana-Xochimilco Adrin Acosta Universidad de Guadalajara Mxico Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educacin, Chile Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educacin, Argentina Teresa Bracho Centro de Investigacin y Docencia Econmica-CIDE Alejandro Canales Universidad Nacional Autnoma de Mxico Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagoga Popular, Per Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernndez Enguita Universidad de Salamanca. Espaa Gaudncio Frigotto Universidade Estadual do Rio de Janeiro, Brasil Rollin Kent Universidad Autnoma de Puebla. Puebla, Mxico Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California Mara Loreto Egaa Programa Interdisciplinario de Investigacin en Educacin Mariano Narodowski Universidad To rcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Polticas Educativas, Per Vanilda Paiva Universidade Estadual Do Rio De Janeiro, Brasil Miguel Pereira Catedratico Un iversidad de Granada, Espaa Angel Ignacio Prez Gmez Universidad de Mlaga Mnica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de So Paulo Diana Rhoten Social Science Research Council, New York, New York Jos Gimeno Sacristn Universidad de Valencia, Espaa Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, Mxico Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas Publicas-Universidad de Buenos Aires, Argentina Antonio Teodoro Universidade Lusfona Lisboa, Carlos A. Torres UCLA Jurjo Torres Santom Universidad de la Corua, Espaa