USF Libraries
USF Digital Collections

Educational policy analysis archives


Material Information

Educational policy analysis archives
Physical Description:
Arizona State University
University of South Florida
Arizona State University
University of South Florida.
Place of Publication:
Tempe, Ariz
Tampa, Fla
Publication Date:


Subjects / Keywords:
Education -- Research -- Periodicals   ( lcsh )
non-fiction   ( marcgt )
serial   ( sobekcm )

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
usfldc doi - E11-00265
usfldc handle - e11.265
System ID:

This item is only available as the following downloads:

Full Text
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 10issue 17series Year mods:caption 20022002Month March3Day 2121mods:originInfo mods:dateIssued iso8601 2002-03-21

xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20029999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00265
0 245
Educational policy analysis archives.
n Vol. 10, no. 17 (March 21, 2002).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c March 21, 2002
Are increasing test scores in Texas really a myth, or is Haney's myth a myth? / Laurence A. Toenjes [and] A. Gary Dworkin.
x Research
v Periodicals.
2 710
Arizona State University.
University of South Florida.
1 773
t Education Policy Analysis Archives (EPAA)
4 856


1 of 11 Education Policy Analysis Archives Volume 10 Number 17March 21, 2002ISSN 1068-2341 A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright 2002, the EDUCATION POLICY ANALYSIS ARCHIVES Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education .Are Increasing Test Scores in Texas Really a Myth, or is Haney's Myth a Myth? Laurence A. Toenjes University of Houston A. Gary Dworkin University of HoustonCitation: Toenjes, L.A. and Dworkin, A.G. (2002, Ma rch 21). Are increasing test scores in Texas really a myth, or is Haney's myth a myth? Education Policy Analysis Archives 10 (17). Retrieved [date] from Abstract Pass rates by Texas tenth-graders on the high schoo l exit exam improved from 52 percent in 1994 to 72 percent in 1998. In h is article "The Myth of the Texas Miracle in Education" ( EPAA August 2000) Professor Walt Haney argued that some part of this increased pass rate was, as he put it, an illusion. Haney contended that the combined effe cts of students dropping out of school prior to taking the 10th gra de TAAS and special education exemptions accounted for much of the incr ease in TAAS pass rates. Relying on the same methodology and data tha t Haney used, we demonstrate that his conclusion is incorrect. None of the 20 percent


2 of 11improvement in the TAAS exit test pass rate between 1994 and 1998 is explained by combined increases in dropout rates or special education exemptions. All may not be right with education in Texas. But n either is it all wrong, as Walter Haney would have everyone believe, judging by his a rticle "The Myth of the Texas Miracle In Education." (Note 1) Haney wastes no space in getting to his main concl usion. In the first paragraph of the introduction he asser ts that "In this article, I review evidence to show that the "miracle" of education reform in T exas is really a myth and illusion." (Note 2) However, he generously invites each reader to arri ve at his or her own decision as to whether he was fair in arriving at h is conclusions. "I leave it to others to judge how fair-minded I have been in recounting thi s version of the Texas miracle." (Note 3) There is no attempt here to deal with all of the is sues raised by Prof. Haney. In fact, only one issue is dealt with, but it is the one that is central to his thesis, namely whether or to what extent increases in dropout rates in Texas wer e caused by the Texas Assessment of Academic Skills (TAAS) exit test and the extent to which any increase in dropouts resulted in an unwarranted increase in the calculat ed pass rate on that test. At a minimum, this is a good example of how two differen t analysts can draw opposite conclusions from the same data.Haney asserted at numerous points in his article, a nd elsewhere, that the TAAS exit test directly resulted in an increase in dropouts, which in turn inflated apparent increases in pass rates on the exit test between the years 1994 to 1998. Typical of statements attributed to Haney include t he following: "I would guess at least half of the apparent increases are a mirage resulti ng from increasing numbers of students being excluded from test results—either because the y dropped out of school or they've been misclassified as special education students. (Note 4) "The Texas miracle in education is a myth," said Walter Haney, a Boston C ollege researcher who studies test statistics. Texas schools, he said, have some of th e nation's highest dropout rates, and the system of accountability that Bush touts helps driv e tens of thousands of students, mostly minorities, to quit school each year—a loss that in turn boosts test scores, he said. (Note 5) The Haney article, as published in the Education Policy Analysis Archives is a distillation of his two-year effort as an expert—an d presumably paid—witness for the Mexican American Legal Defense and Education Fund ( MALDEF) in their suit against the State of Texas. (Note 6) The plaintiffs claimed that the exit test in Texas first administered to students in the spring of their ten th grade, is unfair and discriminates against minority students. The goal of the suit was to prevent the State of Texas from continuing to make passing the exit test a conditio n for high school graduation. Much of Haney's effort was directed towards trying to convi nce Judge Prado that the 20-point increase in the pass rate on the exit test between 1994 and 1998 was due to substantial increases in numbers of students who dropped out be fore even taking the exit test. Haney tried to demonstrate that if students who wou ld most likely fail the exam dropped out of school before they were scheduled to take th e test in the spring of their tenth grade, then the calculated pass rate from the remai ning students would be greater as a


3 of 11consequence. In addition, of course, the alleged in crease in dropouts, especially if they occurred disproportionately among minority students would directly demonstrate the damaging impact upon minority students.Professor Haney went beyond factual arguments and a ttempted to impute motive to administrators and teachers. For example, "These re sults clearly support the hypothesis advanced in my December 1998 report, namely that af ter 1990 schools in Texas have increasingly been retaining students, disproportion ately Black and Hispanic students, in grade nine in order to make their grade 10 TAAS scores look be tter [emphasis added] (Haney, 1998, pp. 17-18). (Note 7) After having convinced himself, at least, that stud ents are intentionally retained so that the exit test pass rate would be higher, he conclud ed: "Hence, it is fair to say that the soaring grade 10 TAAS pass rates are not just an il lusion, but something of a fraud from an educational point of view." (Note 8) While it was important to Haney and his clients to attempt to demonstrate that the very existence of the TAAS exit exam in some way, intent ionally or not, caused an increase in grade nine retention and subsequent increase in dropouts, it was perhaps even more important to their case to demonstrate that these f actors, in turn, were responsible for the dramatic improvement in TAAS exit test pass rates. For if the pass rate increase could be shown to be primarily a result of the increased ret entions and dropouts (and also, perhaps, increases in the use of special education exemptions) then the primary justification for the TAAS-based accountability sys tem itself would be discredited. That is, as long as the state could demonstrate that the academic performance of students who remained in school was improving, then it could be argued that this benefit offset if not outweighed the alleged increase in the number of dr opouts. But if the improved performance, as measured by the pass rate on the ex it test, was an illusion, as Haney asserted, due to the very increases in dropouts and special education exemptions, then the overall impact of the exit test could be argued to be a burden to the state as a whole as well as to the additional students who dropped o ut and thereby failed to earn their high school diplomas.As pointed out by Haney, Judge Prado held that the hypothesis that "schools are retaining students in ninth grade in order to infla te tenth-grade TAAS results was not supported with legally sufficient evidence demonstr ating the link between retention and TAAS (Prado, 2000, p. 27)" (Note 9) In fact, it will be shown below, utilizing data contained in the Haney article itself, that the pos sible impact of increased dropouts and ninth grade retentions actually decreased during the period 1994 to 1998, using Haney's own methodology. Judge Prado was correct.In setting out to quantify the relationship between increases in ninth grade retentions, student dropouts and TAAS pass rates Haney relied u pon the ratio over time of eleventh grade enrollment in a given year to sixth grade enr ollment five years earlier. It is important that the reader be familiar with Haney's own justification for the use of this procedure, and to see how he misused it in reaching his conclusion. Following a discussion of the relationship of grade 9 to grade 8 enrollments, we find the following: At the same time, the analyses of progress for grad e 6 cohorts presented in Section 5.3 revealed that grade 6 to grade 11 progr ession ratios for Whites and minorities varied by not more than 5% during th e 1990s (for Whites,


4 of 11the ratio was consistently between 85% and 89%; and for minorities between 75% and 80%). The reason for focusing here on progress to grade 11 is because the data on enrollments is from the f all whereas TAAS is taken in the spring. But if students progress to gr ade 11, they presumably have taken the exit level version of TAAS in spring of the tenth grade. What this suggests is that the majority of the appa rent 20-point gain in grade 10 TAAS pass rates cannot be attributed to exclusio n of the types just reviewed. Specifically, if rates of progress from g rades 6 to grade 11 have varied by no more than 5% for cohorts of the classe s of the 1990s, this suggests even if we take this as an upper bound, th e extent to which increased retention and dropping out before fall gr ade 11, and add 2% for the increased rate of grade 10 special education cl assification, we still come up with less than half of the apparent 20-point gai n in grade 10 TAAS pass rates between 1993 [sic, 1994] and 1998. (Note 10) After actually looking at the data, Haney was force d to admit that "less than half of the apparent 20-point gain in grade 10 TAAS pass rates" could be accounted for. It is shown below that considerably less than half was accounte d for by the data which he used. To emphasize, if a student is enrolled in the fall of his or her eleventh grade level, presumably the student would have been in the tenth grade the previous spring of the same calendar year, and would therefore have been i n the pool of students who would have taken the tenth grade exit test for the first time. (Note 11) Several issues are glossed over here, such as (a) students repeating grade 11, (b) students beginning public school in Texas as eleventh graders (immigrants or student s previously enrolled outside the public school system), (c) students who may have ta ken the exit test in the spring but who dropped out over the following summer, and (d) changes in the proportion of students exempted from taking the exit TAAS. Regard ing this latter, Haney presented data that suggested that the number of tenth grade students in special education had increased by approximately two percent during the 1 994 to 1998 period. (Note 12) Accepting the simplifying assumptions in Haney's pr ocedure, the results hinge on whether or not the ratio of eleventh graders to six th graders (five years earlier) increased or decreased during the 1994 to 1998 period. Haney referred to these ratios as grade 6 to grade 11 progression ratios. For clarity, let the p rogression ratios for 1994 and 1998 be defined as follows:Progression Ratio(94) = (Grade 11 enrollment in fal l 1994/Grade 6 enrollment in fall 1989)Progression Ratio(98) = (Grade 11 enrollment in fal l 1998/Grade 6 enrollment in fall 1993)If the data show that Progression Ratio(98) is less than Progression Ratio(94), then Haney has made his case. On the assumption that the 1998 ratio was reduced by an increased rate of dropout behavior occurring before the exit tests were taken in the spring of 1998, or an increase in ninth grade reten tions, and assuming that all of the additional students who thereby did not take the ex it test would have failed the test, then the pass rate would have increased by approximately the same rate as the increase in the rate of dropouts and ninth grade retentions, relati ve to the pass rate with no change in these phenomena.


5 of 11Of course, the opposite might also occur, in which case an adjustment to the observed pass rate in the opposite direction should be made. That is, assume the proportion of dropouts decreased, or smaller proportions of stude nts were retained in the ninth grade. This would cause an increase in Progression Ratio(9 8) as compared to Progression Ratio(94), Still assuming, with Haney, that all of the marginal students would fail the exit test, then the pass rate for the set of studen ts in 1998 that would be comparable to the equivalent set in 1994 would be greater than th e observed pass rate for all students tested in 1998. This is because, under such assumpt ions, greater proportions of low performing students would be tested in 1998 then in 1994. In short, if Haney's methodology would call for a negative adjustment to the pass rate if the progression ratio decreased from 1994 to 1998, then applying the same methodology to the opposite outcome, (i.e., an increase in the progression rati os) should require an upwards adjustment to the pass rate actually observed 1998.What really happened? It is difficult to know, if o ne relies only upon Haney's text. In the first of the two paragraphs, there is a reference t o progression ratios during the 1990s without mention of whether they tended upwards or d ownwards, stating only that "they varied by not more than 5% during the 1990s..." In the second paragraph quoted above, he then suggests that this vague 5% variation which occurred sometime "during the 1990s" can be taken as an upper bound of the impact of grade retentions and increased dropouts upon the exit test pass rate increase.One cannot tell from Prof. Haney's own statements w hether the ratio of grade 11 to grade 6 students increased or decreased during the particular period 1994 to 1998, which is the period during which the exit test pass rates increased by 20-points. In proposing to adjust the increase in the passing rates, what happ ened before or after this interval is irrelevant. Hence, reference to a range of variatio n "during the 1990s" is not helpful. Nor is reference to variation in these ratios for White s and minorities taken separately. The 20-point improvement in test pass rates includes al l students taking the exit test, in all subjects. If the grade 11/grade 6 enrollment ratios of Whites increased by 5%, but those for minorities decreased by 5%, they would approxim ately offset one another. If they both decreased by 5%, then the total effect would a lso be 5%. They are not additive. It is necessary to know what happened to the grade 11 to grade 6 enrollment ratios for all students in order that any change be relevant in ad justing changes to pass rates for all students. It is also necessary that they be based o n grade 11 enrollments in 1994 and in 1998.Fortunately, Haney included in an appendix the data necessary to clarify these ambiguities. (Note 13) The enrollment data, by grade level, are available there for the three major ethnic groups from 1989 to 1998. The th ree major ethnic groups included 98 percent of total enrollment in 1989. The major grou p not included was Asian-Americans, for which dropout behavior and TAA S performance is not a major issue. Their omission does not alter the results pr esented below in Figure 1 or in Table 1.


6 of 11 Figure 1. Progression Ratios, 1990–1998. The progression ratios for grade 6 to grade 11 are presented in Figure 1. They are shown separately for African Americans and Hispanics comb ined, for non-Hispanic Whites, and for the totals of all three major ethnic groups The ratios were calculated by dividing the appropriate eleventh grade enrollments by corre sponding sixth grade enrollments five years earlier. The time scale in Figure 1 indi cates the year for which the eleventh grade enrollments correspond. As pointed out by Han ey (see above) the calendar year for which eleventh grade enrollments are recorded in th e fall is also the year in which the same cohort of students would have been first admin istered the exit test the preceding spring. Therefore, a significant change in the prog ression ratio for a given year would be expected to have influenced the percentage of stude nts passing the exit test in the same calendar year. (Note 14) It is interesting to compare the data plotted in Fi gure 1 with Prof. Haney's comments. As he stated (above) the "progression ratios for White s and minorities varied by not more than 5% during the 1990s..."(fn) Indeed, the highes t ratio for minorities was 0.79, the lowest was 0.75 for a range of 0.04 or 4 percentage points. The high for Whites was 0.89 and the low was 0.85, for a range also of 4 percent age points. But it is important to note how these ratios changed over the specific interval of 1994 to 1998, which corresponds to the interval over which the 20-point increase in exit test pass rates occurred. It is not sufficient to merely note the "range". What is impo rtant is whether the ratios increase or decreased over this particular period. Additionally it is the change in the ratio for all ethnic groups combined that is relevant to an attem pt to explain the 20-point improvement in the test pass rate, as this improvem ent reflected the improved performance for all groups combined.So what actually happened to the relevant progressi on ratios? As can be seen in Figure 1, the progression ratio for minorities increased from 0.76 in 1994 to 0.78 in 1998, albeit with a one percent decrease in 1995. For Whites, th e progression ratio also increased,


7 of 11 from 0.86 in 1994 to 0.89 in 1998. For all three ma jor ethnic groups combined, the progression ratio increased from 0.81 in 1994 to 0. 83 in 1998. Therefore, instead of adjusting the improvement in the pass rate from 20points down to 15-points, as implied by Haney, it should be adjusted upwards by 2-points to 22-points due to the increase in the grade 6 to grade 11 progression ratio.Granting a negative adjustment of 2 points due to a n increase in special education exemptions among tenth-graders, the effect upon the improvement in the exit test pass rate attributable to special education exemptions, ninth-grade retentions, and dropout rates net to zero—no impact whatsoever.Total enrollment figures for the three major ethnic groups for the relevant years are shown in Table 1. Once again, the results shown in column 5 of Table 1 are exactly opposite to the assertions by Haney. Instead of a n egative adjustment of 5 points to the 20-point improvement in the exit test passing rate over the 1994 to 1998 period, a positive 2 point adjustment should be made, followi ng the same logic.Table 1 Texas Enrollments for African-American, Hispanic, a nd Non-Hispanic White Students Grade 6 (1989-1993), Grade 11 (1994-1998), and Progression RatiosYear (t-5) Grade 6 EnrollmentsYear (t) Grade 11 EnrollmentsG11/G6 Progression Ratio (1)(2)(3)(4)(5)=(4)/(2) 1989245,8281994199,3790.8111990256,5511995207,1400.8071991269,8391996218,8220.8111992275,7791997226,7940.8221993278,6631998232,4410.834 4-year chg+32,835 +33,062 Data source: Haney (2000), Appendix 7, pages 138-13 9. Notice also that grade 6 enrollment for the three m ajor ethnic groups increased by 32,835 students between 1989 and 1993, while eleven th grade enrollment corresponding to the same cohorts, five years later, grew by 33,0 62 students from 1994 to 1998. The fact that grade 11 enrollments increased more than grade 6 enrollments certainly does not support Haney's claim of increasing dropouts an d/or rates of ninth grade retentions during this time period.Therefore, using Haney's own suggested methodology, and data which he himself provided, none of the improvement in the TAAS exit test pass rate has been shown to be a myth or otherwise fraudulently obtained. Instead of demonstrating that "at least half of the apparent increases are a mirage resulting from increasing numbers of students being


8 of 11excluded from test results", as he had claimed, his data and procedures account for none of the exit test pass rate improvement.Prof. Haney invited readers of his article to form their own judgments as to the fairness of his analysis. We have formed our own judgment as to whether Professor Haney was fair-minded in his use of data and in the conclusio ns he drew from those data regarding the quality and effectiveness of education reform i n Texas over the last decade. Like Professor Haney, we invite readers to arrive at the ir own judgments on this matter.Notes 1 Walter Haney, "The Myth of the Texas Miracle In Edu cation," Education Policy Analysis Archives Volume 8, Number 41, August 19, 2000, available a t ( There are no pag e numbers on the article as posted on the website. The page numbers given below are to a printed version of that document. 2 Haney., p. 7. 3 Haney., p. 8. 4 Debra Viadero, "Testing System in Texas Yet To Get Final Grade", Education Week May 31, 2000, available at( =38taas.h19). 5 John Mintz, 'Texas Miracle' Doubted: An Education 'Miracle,' or Mirage?," The Washington Post, April 21, 2000, p. A01. 6 Haney, p. 7. 7 Haney, p. 46. The reference to Haney, 1998 was to H aney, W. (1998)."Preliminary report on Texas Assessment of Academic Skills Exit Test (TAAS-X)." Chestnut Hill, MA: Boston College Center for the Study of Testing Evaluation and Educational Policy. 8 Haney, p. 54. 9 Haney, p. 46. The reference to Prado, 2000, was to Prado, E. (2000). Order in case of GI Forum Image De Tejas v. Texas Education Western District of Texas (Civil Action No. SA-97-CA-1278-EP. Filed January 7, 2000. (GI Fo rum Image De Tejas V. Texas Education Agency, 87 F. Supp. 667 (W.D. Tex. 2000). 10 Haney, p. 56. 11 The exit exam includes tests in reading, mathematic s, and writing. Each of these may be take more than once. The exit test pass rates be ing considered are based upon the first taking of these tests by (usually) tenth grade stud ents. 12 The increase in tenth grade students which were cla ssified as receiving special


9 of 11 education services is not disputed. However, whethe r or not the increase in special education classifications were appropriate or not i s another matter. 13 Haney, Appendix 7, p. 138-141. 14 To the extent that students dropped out after takin g the exit test in the tenth grade and therefore were not included in the next fall's enro llment counts, the grade6 to grade 11 progression ratio would overstate the likely impact on test scores.About the AuthorsLaurence A. Toenjes is Research Associate Professor of Sociology and C o-Founder of the Sociology of Education Research Group (SERG) at the University of Houston. He holds the Ph.D. degree in Economics from Southern I llinois University. A substantial portion of his research is in the area of school fi nance, and he has developed school finance models for several states. A. Gary Dworkin is Professor of Sociology and Co-Founder of the So ciology of Education Research Group (SERG) at the University o f Houston. He is past chair of the Department of Sociology at the University of Housto n. He holds the Ph.D. in sociology from Northwestern University. The majority of his p ublications is in the areas of the sociology of education and race, ethnic, and gender relations and include nine books and numerous articles and book chapters. He is editor o f a book series entitled "The New Inequalities," published by the State University of New York Press. During the first half of 2001 he served as Senior Research Fellow in Soci ology at the Australian National University in Canberra ACT, Australia.Copyright 2002 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, or reach him at College of Education, Arizona State University, Tempe, AZ 8 5287-2411. The Commentary Editor is Casey D. Cobb: .EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov Thomas F. Green Syracuse University


10 of 11 Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Education Commission of the States William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX) Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton Hugh G. Petrie SUNY Buffalo Richard C. Richardson New York University Anthony G. Rud Jr. Purdue University Dennis Sayers California State University—Stanislaus Jay D. Scribner University of Texas at Austin Michael Scriven Robert E. Stake University of Illinois—UC Robert Stonehill U.S. Department of Education David D. Williams Brigham Young University EPAA Spanish Language Editorial BoardAssociate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico Adrin Acosta (Mxico) Universidad de J. Flix Angulo Rasco (Spain) Universidad de Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho Alejandro Canales (Mxico) Universidad Nacional Autnoma Ursula Casanova (U.S.A.) Arizona State Jos Contreras Domingo Universitat de Barcelona Erwin Epstein (U.S.A.) Loyola University of Josu Gonzlez (U.S.A.) Arizona State


11 of 11 Rollin Kent (Mxico)Departamento de InvestigacinEducativa-DIE/ Mara Beatriz Luce (Brazil)Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.brJavier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mxMarcela Mollis (Argentina)Universidad de Buenos Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mxAngel Ignacio Prez Gmez (Spain)Universidad de Daniel Schugurensky (Argentina-Canad)OISE/UT, Simon Schwartzman (Brazil)Fundao Instituto Brasileiro e Geografiae Estatstica Jurjo Torres Santom (Spain)Universidad de A Carlos Alberto Torres (U.S.A.)University of California, Los