xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20029999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00257
Educational policy analysis archives.
n Vol. 10, no. 9 (January 28, 2002).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c January 28, 2002
Confusing the messenger with the message : a response to Bolon / Victor L. Willson [and] Thomas Kellow.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 10issue 9series Year mods:caption 20022002Month January1Day 2828mods:originInfo mods:dateIssued iso8601 2002-01-28
1 of 4 Education Policy Analysis Archives Volume 10 Number 9January 28, 2002ISSN 1068-2341 A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright 2002, the EDUCATION POLICY ANALYSIS ARCHIVES Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education .Confusing the Messenger with the Message: A Response to Bolon Victor L. Willson Texas A&M University Thomas Kellow University of Houston Citation: Willson, V.L. & Kellow, T. (2002, January 28). Confusing the messenger with the message: A response to Bolon. Education Policy Analysis Archives 10 (9). Retrieved [date] from http://epaa.asu.edu/epaa/v10n9/.Abstract The conclusions by Bolon (2001) based on the relati onship between per capita income and school mean grade 10 mathematics scores in Massachusetts and on instability in year-to-year me an school scores are criticized by us. Our concerns focus on the uninter pretable covariation of economic condition with test performance and the li mitations in interpreting cross-time variability. We agree with Bolon's conclusions but consider the methodology employed inadequate to support them. We suggest alternative requirements and discuss our ow n previous efforts in this area.
2 of 4 In an analysis of the Massachusetts graduation exam ination, Bolon (2001) examined the aggregate grade 10 mathematics test scores for 47 h igh schools and the demographic characteristics of the communities in which they we re situated. From several data analyses, Bolon determined that since the best sing le predictor of mean high school score was community per capita income, "The state is treating scores and ratings as though they were precise educational measures of high significance. A review of thenth-grade mathematics test scores from academic high schools in metropolitan Boston showed that statistically they are not." Further, when removing the variability due to per c apita income, "Large uncertainties in residuals of school-average d scores, after subtracting predictions based on community income, tend to make the scores ineffective for rating performance of schools. Large uncertaint ies in year-to-year score changes tend to make the score changes ineffective for measureing performance trends." While we agree with Bolon's concerns, on the whole, we find little support in the evidence he presents to support them. Our discussio n below details our concerns.Predicting aggregate test scoresOne of the problems with regression analysis is tha t without reasonable theoretical support, all sorts of predictors can be found that produce high correlation. In examining aggregate scores, such as high school test means, i t is no secret that for many decades, as Bolon himself pointed out (Bolon, 2000), achievemen t has been associated with socioeconomic conditions in communities. In earlier eras, when school spending was much more unequal, these differences were more indi cative of opportunity to learn for students. In a judicial climate that has tended to minimize, although not eliminate such disparities, it is much less persuasive, although i t remains an important area for study. The difficulty with using a community aggregate mea sure as a predictor is that it is a surrogate for many other indicators, some of which are absurd at face value but interpretable. Variables such as driver's-license p assing rate or per capita champagne consumption may predict student achievement as well as community per capita income. We can construct meaningful arguments why they migh t. For none is the test invalidated using accepted standards (American Educational Rese arch Association, American Psychological Association, and National Council on Measurement in Education, 1999). In other areas of research such aggregation has pro duced fundamentally misleading conclusions. For example, the literature on intelli gence and income is directly parallel to the discussion here. White (1982) demonstrated the difference between using an aggregate measure of SES (school or community) and individual measure in relating SES to intellectual functioning. Since Bolon used s chool as his unit of analysis, he eliminated proximate measures more appropriate to h is analysis. The school-level variables Bolon eliminated are more appropriate tha n community per capita income on
3 of 4 Copyright 2002 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, email@example.com or reach him at College of Education, Arizona State University, Tempe, AZ 8 5287-2411. The Commentary Editor is Casey D. Cobb: firstname.lastname@example.org .EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov email@example.com Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Education Commission of the States William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX) Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton firstname.lastname@example.org Hugh G. Petrie SUNY Buffalo Richard C. Richardson New York University Anthony G. Rud Jr. Purdue University Dennis Sayers California State UniversityÂ—Stanislaus Jay D. Scribner University of Texas at Austin Michael Scriven email@example.com Robert E. Stake University of IllinoisÂ—UC Robert Stonehill U.S. Department of Education David D. Williams Brigham Young University
4 of 4 EPAA Spanish Language Editorial BoardAssociate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico firstname.lastname@example.org Adrin Acosta (Mxico) Universidad de Guadalajaraadrianacosta@compuserve.com J. Flix Angulo Rasco (Spain) Universidad de Cdizfelix.email@example.com Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho dis1.cide.mx Alejandro Canales (Mxico) Universidad Nacional Autnoma deMxicocanalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State Universitycasanova@asu.edu Jos Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es Erwin Epstein (U.S.A.) Loyola University of ChicagoEepstein@luc.edu Josu Gonzlez (U.S.A.) Arizona State Universityjosue@asu.edu Rollin Kent (Mxico)Departamento de InvestigacinEducativa-DIE/CINVESTAVrkent@gemtel.com.mx firstname.lastname@example.org Mara Beatriz Luce (Brazil)Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.brJavier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mxMarcela Mollis (Argentina)Universidad de Buenos Airesmmollis@filo.uba.ar Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mxAngel Ignacio Prez Gmez (Spain)Universidad de Mlagaaiperez@uma.es Daniel Schugurensky (Argentina-Canad)OISE/UT, Canadadschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil)Fundao Instituto Brasileiro e Geografiae Estatstica email@example.com Jurjo Torres Santom (Spain)Universidad de A Coruajurjo@udc.es Carlos Alberto Torres (U.S.A.)University of California, Los Angelestorres@gseisucla.edu