USF Libraries
USF Digital Collections

Educational policy analysis archives


Material Information

Educational policy analysis archives
Physical Description:
Arizona State University
University of South Florida
Arizona State University
University of South Florida.
Place of Publication:
Tempe, Ariz
Tampa, Fla
Publication Date:


Subjects / Keywords:
Education -- Research -- Periodicals   ( lcsh )
non-fiction   ( marcgt )
serial   ( sobekcm )


General Note:
Also includes EPAA commentaries:Eugene Bartoo, 11 April 1996; John F. Covaleskie, 10 April 1996; Larry S. Bown, 10 April 1996; Andrew Coulson, 10 April 1996; John Stone, 10 April 1996; Dan Cline, 9 April 1996; Andrew Coulson, 8 Apr 1996.

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
usfldc doi - E11-00050
usfldc handle - e11.50
System ID:

This item is only available as the following downloads:

Full Text
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 4issue 1series Year mods:caption 19961996Month January1Day 2323mods:originInfo mods:dateIssued iso8601 1996-01-23

xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c19969999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00050
0 245
Educational policy analysis archives.
n Vol. 4, no. 1 (January 23, 1996).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c January 23, 1996
Also includes EPAA commentaries:Eugene Bartoo, 11 April 1996; John F. Covaleskie, 10 April 1996; Larry S. Bown, 10 April 1996; Andrew Coulson, 10 April 1996; John Stone, 10 April 1996; Dan Cline, 9 April 1996; Andrew Coulson, 8 Apr 1996.
Achievement crisis is real : a review of The manufactured crisis / Lawrence C. Stedman.
x Research
v Periodicals.
2 710
Arizona State University.
University of South Florida.
1 773
t Education Policy Analysis Archives (EPAA)
4 856


1 of 12 Education Policy Analysis Archives Volume 4 Number 1January 23, 1996ISSN 1068-2341A peer-reviewed scholarly electronic journal. Editor: Gene V Glass,Glass@ASU.EDU. College of Educ ation, Arizona State University,Tempe AZ 85287-2411 Copyright 1996, the EDUCATION POLICY ANALYSIS ARCHIVES.Permission is hereby granted to copy any a rticle provided that EDU POLICY ANALYSIS ARCHIVES is credi ted and copies are not sold.The Achievement Crisis is Real: A Review of The Manufactured Crisis Lawrence C. Stedman State University of New Abstract: In a provocative new book, The Manufactured Crisis, David Berliner and Bruce Biddle make four sweeping claims about U.S. achieve ment: there never was a test score decline, today's students are "out-achieving their parents s ubstantially" (p. 33), U.S. students "stack up very well" in international assessments (p. 63), and the general education crisis is a right-wing fabric ation. As a progressive, I'm sympathetic to their concerns but as a scholar who specializes in this material, I find their analysis deeply flawed and m isleading. They mischaracterize the test score decline data, mishandle the international findings, and fail to acknowledge students' continuing low levels of academic achievement.The DeclineAlthough Berliner and Biddle are generally right th at achievement has been stable, they ignored important contradictory evidence and the 1970s decl ine. They claimed "only 'one' test, the SAT" ever suggested a decline (p. 35). This is remarkabl e. High school students' NAEP civics scores, for example, dropped substantially between 1969 and 1976 and have been slipping ever since. Their science scores also fell during the 1970s and have only partly rebounded. Several commercial tests, such as CTBS and STEP, showed dec lines in the 1970s. In the late 1980s, senior high school reading scores declined on the M AT while reading and math scores fell in


2 of 12many grades on the SRA (Linn, Graue, & Sanders, 199 0). In the late 1980s, younger students' NAEP reading and writing performance slipped. (For details, see Stedman & Kaestle, 1991; Stedman, 1993.)They attributed the SAT decline to demographic chan ges in test takers, yet never reviewed the evidence which shows this explains much, but not al l, of the decline. They used "average" SAT scores to claim minority gains, but this masked min ority verbal declines in the late 1970s and late 1980s (Stedman, 1994b). Mexican-American, Puerto-Ri can, and Asian American verbal scores were about the same in the early 1990s as they were in 1976. Berliner and Biddle made sweeping claims about rece nt gains on commercial tests. Their handling of the Linn, Graue and Sanders study demon strates how selective they are with evidence. Their graph omitted Linn, Graue and Sande rs' SRA data which showed declines in many grades. They only graphed the elementary schoo l data, which hid the less impressive high school scores, some of which were declining or stag nating. They never mentioned that Linn, Graue and Sanders pondered, "But the more important question is: Has student 'achievement' improved in recent years?" and concluded that the a nswer was "equivocal" (Linn, Graue & Sanders, 1990, p. 13). Linn, Graue and Sanders dete rmined that recent gains were partly caused by districts' repeated use of the same tests rather than by genuine improvement. The 1980s back-tobasics movements also artificially raised scores by frequent testing and skill-drill approaches (Stedman & Kaestle, 1991).Finally, Berliner and Biddle claimed "virtually all commercial tests would "show that today's students are outachieving their parents substanti ally" (p. 33), yet never presented any evidence to support their claim. They ignored the many revie ws of historical trends on equating studies which refute their claim (Stedman & Kaestle, 1987). The best that can be concluded is that this generation of students "generally" performs about t he same as earlier ones, but the patterns are complicated and there is contradictory evidence.Given changing school populations and societal cond itions, generally stable scores are still a remarkable accomplishment for U.S. schools. This is an important message that the public needs to hear. Nevertheless, the reality is more complica ted than they suggested. Although school critics often exaggerated the extent and ramificati ons of the declines, many did occur (Stedman and Kaestle, 1991). Berliner and Biddle should have admitted that, on several indicators, our students are not performing as well as they once di d.International AssessmentsU.S. performance in the international arena is not as dismal as school critics have asserted, but it certainly is not as glowing as Berliner and Biddle claim. Our students have done well in reading and elementary school science, middling to poor in geography and secondary school science, and last or near-last in mathematics (Stedman, 1994b). Berliner and Biddle offered several arguments to try to explain the weak U.S. performance but, in doing so, they tacitly acknowledged that our international performance often has been poor.Opportunity-to-LearnBerliner and Biddle's opportunity-to-learn argument is a red herring. International researchers pioneered the use of OTL measures and it is already factored into many results. ETS's 1988 international math and science findings, for exampl e, came only from schools in which "more than 75 percent of the students had already had an opportunity to learn the content" (Lapointe et


3 of 12al., 1989, p. 33). Even so, the U.S. did poorly whe ther judged by rankings, proficiency levels, or percentage correct.Berliner and Biddle claimed that our students are a t a disadvantage because we generally delay algebra until 9th or 10th grade. But U.S. students have done poorly in most math areas, not just algebra. In 1988, for example, our 13-year-olds ran ked last in arithmetic and measurement and next-to-last in geometry, data organization, and pr oblem solving (NCES, 1991, p. 395). They also had poor results in 1991 in these areas (NCES, 1992, p. 21). U.S. and Japanese curricula were also more comparab le than claimed. In the Second International Mathematics Study, content coverage was similar in arithmetic, geometry, and statistics, yet U.S. students still scored lower (Stedman, 1994a). In a telling analysis, Baker (1993) found that when one considers "only" the test items that U.S. 8th g raders were taught during the year, they averaged only 40% correct.Westbury StudyThe Westbury study was at the heart of their curric ular claims, but their handling of it revealed they care more about their argument than the eviden ce. First, the study has limited implications because it used data that were over a decade old, d ealt only with one subject--math, and involved a better-thanusual U.S. 8th grade performance. Se cond, they did not even report Westbury's comparisons properly. They took his scores for our most advanced 8th grade math classes--the top 25% comprising algebra and pre-algebra--and com pared them to the "average" Japanese class! No wonder our algebra classes looked good in their comparison. What Westbury actually did was compare our most adv anced 8th grade math classes to the top fifth of Japanese students. Although this was a fai rer approach, it still did not "isolate" the effect s of the curriculum, but confounded them with selecti on effects. U.S. students who study algebra in 8th grade are a select group of 14%, differing f rom other U.S. students in college expectations, math interest, parental support, social class, and academic ethic. Consequently, one cannot tell how much of their performance reflects their algebr a curriculum and how much their background advantages. (Using this comparison directly violate d their own research precept--the Principle of Control, p. 159.)What did Westbury actually find? Our select student s did not do that well. Our pre-algebra classes scored only 56% correct and lagged well beh ind, by a substantial two standard deviations (Westbury, 1992, p. 21). Our algebra classes scored comparably to the Japanese classes, but this was hardly surprising. They were an elite group of only 14% of our classes compared to a less select 20% of the Japanese students. They were judg ed only on the algebra portion of the test, yet they had spent more of their time on algebra (formu las and equations), 61% to 26%, and had covered more of the test problems, 88% to 82%, than the Japanese students (Westbury, 1992, p. 20, p. 21). (So much for claims that curricula were equated!). In two other test areas, geometry and measurement, they even scored below the "averag e" Japanese class (Stedman, 1994a). Finally, our 8th graders were older and had been in school longer--the Japanese students were only 7th graders!Berliner and Biddle ignored Westbury's analysis of U.S. calculus classes, yet this tested the overall quality of our best math programs given to our best students. Our calculus classes fared poorly, however, substantially trailing the "averag e" Japanese class in every tested area (Stedman, 1994a). Given all this, it was misleading for them to claim that "U.S. teachers and schools are [not] deficient compared with those in Japan" (p. 56) and to conclude that "Many,


4 of 12perhaps most, of the studies' results were generate d by differences in curricula" (p. 63).Variability ArgumentBerliner and Biddle tried to explain away poor U.S. international performance by claiming our achievement is "a 'lot' more variable" (p. 58) than other countries, but offered no evidence. In fact, the 1991 IAEP math and science studies showed our variability was similar to that of other nations and less than that of Taiwan and Korea, the leading performers (cf. 10th & 90th percentiles, NCES, 1993b, p. 56; NCES, 1993a, p. 41 5).States-to-Nations ComparisonThey never mentioned that the states-to-nations com parison they cited was designated "experimental" and technically problematic (see cau tion, NCES, 1993b, pp. 54, 94). The international scores were projections from a U.S. s ample that took both the NAEP and IAEP tests. No international student ever took the NAEP test and it is unclear that the IAEP-NAEP relationship would be the same for students in othe r countries. Our states had two important advantages. Our students were older--over half were 14-15 years old whereas the international students were 13-yearolds. Our states' scores cam e from the 1992 NAEP assessment and were higher than what was projected for the U.S. (cf. NC ES, 1993c, p. 83; NCES, 1993b, p. 56). Finding that a few select, typically high-scoring m id-Western states did well in the comparison is not surprising. What is staggering is that our best state scores were only the "average" level in Taiwan and Korea! Berliner and Biddle did not repor t that the same comparison showed that the typical U.S. student was two years behind the avera ge Taiwanese student and scored only around Taiwan's and Korea's 25th percentile (NCES, 1993b, pp. 54, 56). It also showed that only 13-16% of U.S. students reached the proficient leve l, while 35-43% of Taiwanese and Korean students did (Pashley & Phillips, 1993).Social Inequality ArgumentAlthough racism and social inequality have taken a severe toll on many of our students' academic development, this does not explain the poor general performance of U.S. students. The math deficit, for example, is not simply a minority stud ent problem. In 1992, only 30% of "white" U.S. 8th graders demonstrated proficiency in the NAEP ma th assessment; over a quarter did not even make the basic level (NCES, 1993c, pp. 101-102). No r are our problems due to low-achievers. Even our top half have not kept pace internationall y in math and science (Stedman, 1994a). Although U.S. students do not generally fail in int ernational comparisons, it is misleading for Berliner and Biddle to claim that "they stack up ve ry well" (p. 63).Low AchievementThe book's central problem is that Berliner and Bid dle tell only part of the story. Although achievement trends, for the most part, have been st able, academic and general knowledge have been at low levels for decades (Stedman, 1993).In math, NAEP analysts recently concluded that "les s than half (of high school seniors) appeared to have a firm grasp of seventh-grade content" (Mul lis et al., 1991, p. 80). They have trouble even with simple problems involving fractions, deci mals, and percents.


5 of 12Few high school students have done well on NAEP wri ting tests. Only about a third wrote adequate papers and only a small percentage could w rite "elaborated" papers. The one bright spot is their competence in basic grammar and punctuatio n. Our functional illiteracy rate remains around 20-30 %--meaning that millions of adults have trouble with common day-to-day reading tasks (Stedm an & Kaestle, 1987; Kirsch, 1993). Students lack basic knowledge in history and litera ture. In the late 1980s, substantial majorities of our 17-year-olds did not recognize that Upton Si nclair was a muckraker, the Scopes trial dealt with evolution, Jim Crow laws segregated blacks, or the time period of the Civil War. A majority did not recognize classics by Shakespeare, Chaucer, Conrad, and Whitman and were unfamiliar with major women and African-American writers. Thes e were straightforward multiple-choice questions deliberately designed without the usual d istractors. Geographical knowledge also has often been poor. In 1988, Gallup repeated a survey given to adults in 1947 and concluded that "Americans' geogr aphic literacy has gotten worse in the last forty years." They found that, "From outline maps, the average American can identify only four of twelve European countries, less than three of ei ght South American countries, and less than six of ten U.S. states" (Gallup Organization, 1989, p. 162).RejoindersInstead of reviewing and acknowledging this evidenc e, Berliner and Biddle offer several rejoinders why such findings don't matter. They sug gest that the standards for knowledge are unrealistic and are those of classicists, historian s, and test designers. Most people, however, would expect high school seniors to be competent in 7th grade math, literacy, and basic social studies information.Breadth of ExperienceThey argue that U.S. students are focused on a brea dth of experience, but this does not excuse our low achievement. Certainly academic achievement is one of our goals and should be one of our strengths. Nor is it clear that U.S. students h ave a monopoly on breadth or richness of experience. Portraits of Japanese elementary school s clearly show that students are not academic automatons, but are engaged in rich curricular and extracurricular activities--calligraphy, sewing, hands-on math and science activities, group problem-solving, electronics, dance, musical training, play, reading, physical exercise, coopera tive learning, school jobs, etc. (Stevenson & Stigler, 1992).Scaling ProblemsThey rightly argue NAEP scales are flawed, but this does not explain students' poor performance or limited knowledge. Contrary to their assertions, it doesn't require tough questions to generate scale scores or discriminate among U.S. students. T he problems at the highest NAEP levels are actually fairly easy. The 300 level in math, for ex ample, includes simple decimal problems and level 350 has "routine problems involving fractions and percents." This is junior high general math, yet 17-year-olds have trouble with it! In his tory, many 350 level problems required nothing more than simple recognition of basic facts. (For a more detailed look at NAEP findings as well as its scaling problems, see Stedman, 1993.)


6 of 12Many findings of low performance do not come from t raditionally scaled tests. The writing results involve authentic holistic evaluations and thus avoid the scaling problems. The functional illiteracy estimate came from tests of many differe nt designs and was derived through a systematic analysis of individual items not scaled results. The true rate might even be higher because some tests used items that were easier than their real-life counterparts and did not test dropouts, the homeless, prisoners, or non-English s peakers (Stedman and Kaestle, 1987). Low levels of civic literacy and general knowledge were revealed in national surveys as well as standardized tests.Details of Low AchievementCareful reviews of individual items and sets of ite ms have avoided many scaling problems and still indicate students struggle with basic materia l (Carpenter et al, 1988, 1982). Math educators found that students "exhibit serious gaps in their knowledge" and often learn "concepts and skills at a superficial level" They concluded that "studen ts' achievement at all age levels shows major deficiencies" (Carpenter et al, 1988, pp. 40-41). I n 1990, for example, only around half the 17-year-olds could convert a decimal to a fraction, find a number given a percent, estimate a square root, and use the properties of triangles (M ullis et al, 1991, pp. 302-309). 34% could not even find the area of a rectangle, given a diagram and the length of two sides (Mullis et al, 1991, p. 306).Although students' geographical knowledge is better than many have asserted, there still are serious problems (Stedman, 1993). 15-40% of high sc hool students had trouble with basic geographical material. Most could not interpret a g raph showing birth and death rates. Given the Vietnam War, it is unsettling that 63% of our high school seniors could not locate Southeast Asia on a world map. 64% did not know Saudi Arabia's loc ation, although this was before the Persian Gulf War. Half could not answer such simple questio ns as the following: The construction of the Panama Canal shortened the sailing time between New York and [London, Port-au-Prince, Rio de Janeiro, San Fr ancisco] Functional literacy tests have produced some distur bing findings. Twenty percent of the population, for example, had trouble reading and un derstanding dosage information on medicine bottles. Similar percentages had problems with a ho using inspection notice, basic coupons, and price per unit weight. About a third failed at figu ring out train schedules, how much change should come from a purchase, and which subjects had improved on a report card. Student achievement may be even worse than these fi ndings suggest. The NAEP data do not include dropouts who presumably would score lower. To reach a given NAEP level, students only have to answer correctly 65-80% of its problem s. The burden on students is light. Compared to the SATs and achievement tests, which can be hal f-day or all-day affairs, the NAEP tests are short, only 45 minutes. The tests are predominantly multiple choice, recognition-based rather than open-ended, recall which make them easier for students to do well on.Real-World RelevanceBerliner and Biddle argue that findings about low a chievement are irrelevant because the tests did not measure real-world problem solving. This is a curious position given that their claims about stable achievement trends came from these sam e tests! There are several problems with their argument. First, many tests that showed low a chievement did measure the knowledge and skills needed in the real world. The functional lit eracy tests, for example, used real-world tasks


7 of 12with real-world materials. Math tests have involved calculators. graphing, and open-ended items. NAEP reading tests have used poetry, newspaper arti cles, and passages from real literature. Second, in-school and out-of-school tasks, although different in many ways, still involve related abilities. Standardized tests give some indication of real-world problem solving ability. One indication of this is the marked correlation betwee n scores on traditional tests and those on authentic assessment measures (Wang, Haertel, and W alberg, 1993, p. 371). Third, "real-world problem-solving" is not our only educational goal. General knowledge, some of which can be measured successfully via multiplechoice testing, is an important goal in itself. We want informed and knowledgeable citizens. Histor ical knowledge can play a central role in understanding public policy debates.Consider the on-going and highly-charged debates ov er immigration policy and affirmative action. How can we expect students and young adults to make informed appraisals of the arguments when they are ignorant about the history of race relations in this country? NAEP testing in the 1980s showed that the vast majority of high school students did not know what Jim Crow laws were, what the 3/5ths Compromise was, or what the Emancipation Proclamation actually did. Substantial percentages did not know what Plessy v. Ferguson or Brown v. Board of Education were about. They lacked basic information pertaining to the Civil War, one of our nation's epochal events and a key force in shaping race relations. Sizable majorities were unfamiliar with the Missouri Compromise, nullificat ion, the Dred Scott decision, the dates of Civil War, and the dates of Lincoln's term. Such ig norance is not an artifact of an obscure psychometric scaling procedure. Knowledge does matt er. Fourth, there likely will be little comfort in resu lts from more authentic, real-world testing. NAEP is increasingly using performance assessment-the new science test, for example, includes drawing tasks, writing, and open-ended questions. T he new reading assessment has longer passages and is 40% open-ended. Students, however, often do more poorly on the open-ended versions of test items. When their understanding of a subject is probed, surprising gaps and confusions often appear (Bridgeman, 1992; Martinez, 1991; NAEP, 1983, p. 32; Rogers & Stevenson, 1988). Future assessments are likely to produce even more disturbing news about low achievement than we have now.Finally, Berliner and Biddle argue that school crit ics focused more on the imagined economic consequences of low achievement than on the actual achievement evidence. I agree. Soon after the "Nation at Risk" report appeared, I argued that it made too much of a high-skilled, hi-tech future economy as a rationale for reforming educati on (Stedman and Smith, 1983). But the actual evidence is troubling and Berliner and Biddle did n ot examine it. The low levels of achievement are unimpressive results for 12 years of schooling. The tests do measure much of what is being taught in our schools and show we are not succeedin g in our efforts. A complex, democratic society needs a well-read and knowledgeable citizen ry and yet the evidence shows we are not accomplishing this.Teaching Methods and Student Work HabitsOur achievement problems are deep-seated, widesprea d, and long-standing. But this is not the only reason for fundamental and far-reaching school reform. Teaching methods and student work habits also leave much to be desired (Stedman, 1993 ). Although there are a few bright spots, such as the frequent use of demonstrations in science cl asses, the portrait is troubling. NAEP analysts found math instruction


8 of 12"continues to be dominated by teacher explanations, chalkboard presentations, and reliance on textbooks and workbooks. More innovativ e forms of instruction--such as those involving small group activities, laboratory work, and special projects--remain disappointingly rare." (Dossey et al, 1988) History and civics classes are dominated by textboo ks, tests, quizzes, and short-answer questions. It is unusual to find students working in groups or writing long papers. Writing instruction in the schools is also limited and is focused on mechanics Only about a fourth of 8th graders report that their teachers spend more than an hour a week on writing. Interest in science has not been sparked. In 1986, fewer than a fourth of 11th graders reported working on science-related hobbies or talking with friends about science. Only about a third reported going to a science museum or trying to fix something electrical or mechanical. Students do little schoolwork. The data on homework and TV watching are revealing. In 1990, only about a third of our 17year-olds reported sp ending over an hour a day on homework, whereas half reported watching 3 or more hours of T V daily! Reading has been shortchanged. In 1986, over half the 11th graders reported reading o n their own less than once a week; about a fifth reported they never did!One cannot look over this information without a sen se that our schools are not what they should be. Over the past decade, thought-provoking ethnogr aphies and school profiles by Boyer, Fine, Goodlad, Oakes, Sizer, and others have portrayed a school system in crisis. What we're seeing, particularly at the high school level, is that stud ents are often disengaged, teachers' work is often factory-like, and intellectual life is often poor. These accounts were hardly the products of right-wing ideologies (cf. Berliner & Biddle, Chapt er 4, pp. 140-141). Reformers have been busy. They know that the school s are not better than ever, but rather, more than ever, they need to be different than they are. Teachers and other educators who are intimately involved in the life of schools recogniz e there is a serious problem. There are major reform efforts affecting every major aspect of educ ation: curriculum, evaluation, funding, governance, pedagogy, and school organization. Loca l educators are not mere pawns in a conservative political chess game, but have been re sponding actively to real needs and problems.The Scope of ReformFixing the schools is a crucial part of solving our longstanding academic problems. But we also need to create a society that values scholarship an d learning over commercialism and entertainment. This will require a major political and economic transformation. Educators must challenge the vested interests that are more interested in profits than the welfare of communities and civil society. We must fight the economic displacements that disrupt families, produce violence, and undermine students' development. We must take on the media conglomerates that are focused more on selling prod ucts than nurturing our cultural and intellectual life. We must change a system that val ues the bombastic broadsides of radio talk show hosts and political candidates over reasoned a nd civil discourse. To succeed in our most troubled communities, we wil l need to overhaul school financing systems and break down powerful, entrenched bureaucracies. But school reform is no substitute for job creation, income redistribution, and political empo werment. We must make our educational efforts part of a broader social and political agen da, one that promotes full employment,


9 of 12community revitalization, and civic participation.ConclusionIn the 1980s, school critics often exaggerated the size and extent of the test score decline. In spite of enormous changes in society and school populatio ns, U.S. achievement has been remarkably stable for many decades. But it remains inadequate and at low levels. Ignoring this evidence or arguing it is a rightwing fabrication hampers muc h needed school reform. The crisis is real, what is actually being manufactured here is a new m ythology about U.S. student achievement. Interested readers can find an in-depth--and balanc ed-treatment of the achievement evidence in several of my recent articles. I examine the NAEP d ata in the November 1993 Phi Delta Kappan, the Sandia Report and the SAT data in the January-F ebruary 1994 Journal of Educational Research, and the international data in the October 1994 Educational Researcher. References Baker, D. P. (1993). Compared to Japan, the U.S. is a low achiever Really. Educational Researcher, 22(3), 18-20. Berliner, D.C. and Biddle, B.J. (1995). The Manufactured Crisis: Myths, Fraud, and the Atta ck on America's Public Schools Reading, MA: Addison-Wesley. Bridgeman, B. (1922, Fall). A comparison of quantit ative questions in open-ended and multiple-choice formats. Journal of Educational Measurement, 269+. Carpenter, T. P., Corbitt, M. K., Kepner, H. S., Jr ., Lindquist, M. M., & Reys, R. E. (1982). Student performance in algebra: Results from the Na tional Assessment. School Science and Mathematics (October), 514-531. Carpenter, T. P., Lindquist, M. M., Brown, C. A., K ouba, V. L., Silver, E. A., & Swafford, J. O. (1988). Results of the fourth NAEP assessment of ma thematics. Arithmetic Teacher (December), 38-41.Dossey, J. et al. (1988). The Mathematics Report Ca rd. Princeton, New Jersey: Educational Testing Service.Gallup Organization (1989). The Gallup Poll public opinion 1988. Wilmington, Delaware: Scholarly Resources.Kirsch, I. S. (1993). Adult literacy in America. Wa shington, D.C.: National Center for Education Statistics.Lapointe, A., Mead, N., & Phillips, G. (1989). A wo rld of differences. Princeton, New Jersey: Educational Testing Service.Linn, R. L., Graue, M. E., & Sanders, N. M. (1990). Comparing state and district results to national norms: The validity of claims that "everyo ne is above average". Educational Measurement: Issues and Practice (Fall), 5-14. Martinez, M. (1991, Summer). A Comparison of Multip le-Choice and Constructed Figural Response Items. Journal of Educational Measurement, 131-45.


10 of 12Mullis, I. V. S., Dossey, J. A., Owen, E. H., & Phi llips, G. W. (1991). The Sate of Mathematics Achievement. Washington, D.C.: U.S. Department of E ducation. NAEP (1983). The Third National Mathematics Assessm ent. Report No. 13-MA-01. Denver, Colorado: Education Commission of the States.National Center for Education Statistics (1993a). D igest of education statistics 1993. Washington, D.C.: U.S. Department of Education.National Center for Education Statistics (1993b). E ducation in States and Nations. Washington, D.C.: U.S. Department of Education.National Center for Education Statistics (1993c). N AEP 1992 Mathematics Report Card for the Nation and the States. Washington, D.C.: U.S. Depar tment of Education. Pashley, P. J. & Phillips, G. W. (1993). Toward wor ld-class standards: A research study linking international and national assessments. Princeton, New Jersey: Educational Testing Service. Rogers, V. & Stevenson, C. (1988, February). How Do We Know What Kids Are Learning in School? Educational Leadership, 68-75. Stedman, L. C. (1993). The condition of education: Why school reformers are on the right track. Phi Delta Kappan, 75 (3), 215-225. Stedman, L. C. (1994a). Incomplete explanations: Th e case of U.S. performance in the international assessments of education. Educational Researcher, 23(7), 24-32. Stedman, L. C. (1994b). The Sandia Report and U.S. achievement: An assessment. Journal of Educational Research (JanuaryFebruary), 133-146. Stedman, L. C. (1995). The new mythology about the status of U.S. schools. Educational Leadership, 52(5), 80-85. Stedman, L. C. & Kaestle, C. F. (1987). Literacy an d reading performance in the United States, from 1880 to the present. Reading Research Quarterly (Winter), 8-46. Stedman, L. C. & Kaestle, C. F. (1991). The great t est score decline: A closer look. In C. F. Kaestle, H. Damon-Moore, L. C. Stedman, K. Tinsley, & W. V. Trollinger (Eds.), Literacy in the United States (Chapter 4). New Haven: Yale University Press. Stedman, L. C. & Smith, M. S. (1983). Recent reform proposals for American education. Contemporary Education Review, 2(2), 85104. Stevenson, H. & Stigler, J. (1992). The learning gap: Why our schools are failing and w hat we can learn from Japanese and Chinese education. New York: Simon and Schuster. Wang, M, Haertel, G., & Walberg, H. (1993). Toward a knowledge base: Why, how, for whom? Review of Educational Research,Westbury, I. (1992). Comparing American and Japanes e achievement: Is the United States really a low achiever? Educational Researcher, 21(5), 18-24.About the Author


11 of 12 Lawrence C. Stedman Lawrence C. Stedman is Associate Professor of Educa tion at the State University of New York at Binghamton. His Ph.D. is from the Universit y of Wisconsin at Madison in Educational Policy Studies with a minor in Sociology. He has wo rked as a school district policy analyst, secondary school teacher, VISTA volunteer, and educ ational researcher. He has a keen interest in equal opportunity and school reform. His dissertati on and early articles centered on effective schools research and the reform reports of the earl y 1980s. He has helped evaluate ESL, minority achievement, merit pay, and dropout intervention pr ograms. More recently, his research has focused on the gene ral condition of education and its implications for policy-making. He has written arti cles on the test score decline, literacy trends, the international assessments, and the Sandia Repor t. He is currently investigating historical trends in students' and adults' general knowledge. It is the outgrowth of a book he helped author with Carl Kaestle and others on the history of the U.S. reading public ( Literacy in the United States: Readers and Reading Since 1880 Yale University Press, 1991). This new research h as been funded by a SUNY Faculty Research Grant and Fe llowship and by a National Academy of Education Spencer Foundation post-doctoal fellowshi p.Copyright 1996 by the Education Policy Analysis ArchivesEPAA can be accessed either by visiting one of its seve ral archived forms or by subscribing to the LISTSERV known as EPAA at (To sub scribe, send an email letter to whose sole contents are SUB EPAA y our-name.) As articles are published by the Archives they are sent immediately to the EPAA subscribers and simultaneously archived in three forms. Articles are archived on EPAA as individual files under the name of the author a nd the Volume and article number. For example, the article by Stephen Kemmis in Volume 1, Number 1 of the Archives can be retrieved by sending an e-mail letter to LISTSERV@a and making the single line in the letter rea d GET KEMMIS V1N1 F=MAIL. For a table of contents of the entire ARCHIVES, send the following e-mail message to INDEX EPAA F=MAIL, tha t is, send an e-mail letter and make its single line read INDEX EPAA F=MAIL.The World Wide Web address for the Education Policy Analysis Archives is Education Policy Analysis Archives are "gophered" in the directory Campus-Wide Inform ation at the gopher server INFO.ASU.EDU.To receive a publication guide for submitting artic les, see the EPAA World Wide Web site or send an e-mail letter to and include the single l ine GET EPAA PUBGUIDE F=MAIL. It will be sent to you by return e-mail. General questions about ap propriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, Glass@asu.ed u or reach him at College of Education, Arizona Sta te University, Tempe, AZ 85287-2411. (602-965-2692)Editorial Board John Andrew Coulson Alan Davis Mark E.


12 of 12Thomas F. Alison I. Arlen Gullickson Ernest R. Aimee Craig B. Howley u56e3@wvnvm.bitnet William Richard M. Jaeger Benjamin Thomas Dewayne Mary P. Les Susan Bobbitt Anne L. Hugh G. Richard C. Anthony G. Rud Dennis Jay Robert Robert T.


1 of 2 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 11 April 1996Eugene Bartooebartoo@CECASUN.UTC.EDU On April 10, 1996 Andrew Coulson wrote: Like Berliner's 1993 EPAA paper, The Manufactured Crisis is a fanciful and selective romp through the data on public schooling Anyone who is currently relying on or quoting this material will do themsel ves a great favor by reading Larry Stedman's recent detailed critique of their treatme nt of the academic achievement data (also published in the Education Policy Analysis Archives ). OK, I haven't read B & B's book, but read Stedman I, B & B reply, and Stedman II. And Andrew's post. Some impressions: (1) I suspect that many foreign students study more lan guages for longer periods than our students do, but I also think, as has been said by others here, that many foreign students get to use those languages more often and hence the language s tudy takes. There are vast numbers of american students that have studied French for year s and find themselves speechless when confronted with a French waiter. The opportunity an d need to converse in another language is just not there and thus has an effect, I think, on the force of insisting on more rigor and time in learning languages. The argument that it broadens o ne's horizons may have been lost years ago. (2) I am engaged in a project in a local private girls school. I sit in on a precalculus class of junior girls; very bright, quick, high achievers They will do very well later; most will go to good schools; get good jobs; have good lives. This morning I asked them if any of them knew where the Pyrennes were [I read Stedman II last nig ht; I mulled over the data on geography from the NAEP]. Only three girls knew where it was. I'm not sure I knew where it was when I was sixteen. They had geography as seventh graders I imagine that that experience was more like "this is where Africa is". Is this a sobering findi ng? At my university we have one faculty member whose specialty is geography; one! In the teacher ed. program which I helped design some few years ago, we have a geography concentration for social science ed. folks. It has few majors; has had fewer graduates. They have trouble getting the courses; most are taught by adj uncts. For most people in the US, the closest thing they have ever gotten to geography was that y ellow-bordered magazine that is popular at garage sales. I was not surprised at the findings o f the NAEP. I don't know if that is something with which to be concerned. (3) I think Stedman is basically right. We do have poo r achievement. The achievement levels in this country have been poor forever. Most of our fellow citizens do not have a large store of knowledge as measured by the NAEP studies [or others like them]. Our children's achievement patterns are about what their parents' achievement patterns are. Perhaps the school


2 of 2is culpable, but I didn't get the sense that Stedma n took that view. (4) The SAT issue is basically not controversial anymor e. There seems to be general agreement that those data have been stable for twen ty some years; that earlier declines were due to a variety of factors, the largest percentage of variance being SES changes in test takers. And who cares? (5) There has just been another summit on educational matters and the reaffirmations of the need for rigorous standards and assessment were given. Shanker concurred in his latest column, stating that it was time to take control of the content of the curriculum out of local hands. In some areas around here those are fightin' words. And so our dream is to raise ourselves by the boot straps of our children so that, for example, those bright girls would know where the Py rennes were [and I might know how to spell it]? Just some thoughts. Thanks for your indulgence


1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 10 April 1996John F. CovaleskieJCOVALES@NMU.EDU It is good to see Andrew attacking public schools again. If B & B are somewhat selective in their presentat ion of evidence, they are no more than following the lead, if not to the extent, of those who have been unfairly attacking schools for the past two decades. As I understand the core of their argument, it is that there is much evidence that schools are doing a remarkably good job with an increasingl y diverse population. To cite one example, SAT scores have been declining for some time, and t his is pointed to as a failure of schools. However, disagreggated data show that scores are ac tually rising for all subgroups, while the decline is due to the expansion of the test-taking pool among those who score lower, and were previousl;y excluded. The central claims of B & B s eems unrefuted: (1) schools are not perfect, and need to made better, but they are hardly failur es as they are painted, and (2) to make the case that schools are failing requires ignoring a great deal of data, and (3) the media have been complicit with those who, for ideological reasons, oppose public funding of education, quite regardless of its success or failure. Opposition to public education is hardly a matter of evidence; never has been, probably never will be. Support for public education is the same. The issue is what, if anything we owe collectively to our children.


1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 10 April 1996Larry S. Bowenlbowen@OSF1.GMU.EDU In addition to what John Stone has contributed to this thread, I believe it important to recognize that heightened expectations for public e ducation are at work in societal concern over our schools--a phenomenon that likely will persist and guarantee continuing frustration. And with over 2 million teachers in more than 15,000 school systems, criticisms of "the schools" and their teachers will vary enormously in validity. This is of course common sense, but it is oft lack ing when looking for actions to deal with real and severe problems. Attempts to deal with gla ring inequities in funding are too few, even within school districts where a PTA of one school p urchases large numbers of computers and software while another in a poor neighborhood langu ishes. The general effects way of thinking about the process of education is terribly unfortun ate, I believe, with both the layperson and educator unable to develop a differentiated way of thinking and approaching the state of schooling and all its complexities. In the Washington, D.C. area, for example, there i s strong evidence re a number of large districts which are viewed as either "excellent" or "terrible" by the media and the public in general. Outstanding schools in a terrible district are rarely acknowledged (or even known about by most people, I'm persuaded in listening), and in at least one highly-praised district there is mediocrity that fits the mantra of how "schools are failing." Until we are willing to seriously deal with the inequities in public education in those sc hools that are lousy and with teachers of children who are not attaining functional literacy the issue of public education quality will not be adequately addressed. I would suggest that those who are concerned with achieving excellence for all learners in a given school system begin by asking what the ineq uities are and proceed with determining the particular problems of the setting and insist that those problems are the problems for the school division and community as a whole, necessitating a "village" response. But that is hard work, especially for those who have "made it" and believe theirs is theirs.


1 of 2 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 10 April 1996Andrew Coulsonandrewco@IX.NETCOM.COM Dan Cline takes me to task for including a persona l anecdote in my recent post criticizing The Manufactured Crisis In response, I'd just like to say: Good for you! I threw in that personal comment because, in my experience, it's true, and b ecause I was interested in getting a debate going, but it does not play even a minor role in my overall verdict on The Manufactured Crisis. In my post I mentioned Larry Stedman's critique, a nd my own very brief assessment of their treatment of the IQ data and the comparative spending and efficiency data (neither of which Dan mentioned). I have also, independently from Lar ry, examined their achievement data and arguments. When I complete my review of their case I plan to incorporate it into a book I'm writing on school reform. I do not intend to includ e my personal opinions on the educational breadth of European vs. U.S. students unless I can provide some concrete, broad-based data to support them. Berliner and Biddle, however, saw no such need for empirical evidence in making their own claim about American students being more "broad ly educated." They write, without even a token piece of supporting evidence, that: By comparison then, American teenagers probably hav e more nonacademic interests and a wider knowledge base than do students from co untries that stress narrow academic concerns (p52). They do not bother to prove that the emphasis on a cademic achievement enjoyed by other countries is indeed "narrow," nor do they bother to show that foreign students fail to enjoy all the same activities they list for American youth. The p ossibility that "narrow" concerns such as literature, history, geography, civics, might widen people's "knowledge base" also appears not to have occured to them. I trust that readers of The Manufactured Crisis, whether or not they like what B. & B. are saying, will be just as quick to discount their use of unsupported anecdote as Dan was to discount mine. ---------An aside on the "educational breadth" issue: This is such a vague term, and such a, well, "broa d" concept, that I never would have initiated it myself if I were trying to make an int ernational comparison. Still, since we're on the topic, I'll just address a few of Dan's comments on the subject. While I don't disagree with Dan's suggestion that learning second, third, or fourth languages is to some degree related to the child's environment, schooling is c learly a significant factor in Europe and in other regions, such as the Canadian province of Quebec. In Quebec it is eminently practical to become bi/multi-lingual, and schools generally teach whichever languages


2 of 2(French or English or both) are not native to the s tudent. Success seems to depend on how intensively the languages are taught, and how much use the student makes of them (in and out of school). According to the OECD (Education at a Glan ce, 1995), foreign language teaching in 15 European countries takes up an average of 13% of te aching time. This is second only to mathematics, at 16%. It ranges from a low of 9% to a high of 26%. According to the Digest of Education Statistics. (1993), U.S. students spend r oughly 6.5% of their class time on foreign languages--half of the European average, and less t han the lowest rate of the 15 European countries studied by the OECD. Of course it remains to be argued that foreign languages add to one's "breadth" of knowledge, but I think that case can fairly easily be made.


1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 10 April 1996John StoneSTONEJ@EDUSERV.EAST-TENN-ST.EDU On April 9, 1996 Dan Cline wrote: However, I think responses to these works by policy analysts and scholars of whatever pursuasion, even the critics of the school s among them, should perhaps rise above conclusions based just on hunches derived fro m personal encounters. (snip) I am not sure whether Berliner & Biddle were making an empirical knowledge claim or a simple assertion that American students are mo re broadly educated, but in either case, an examination of the claim or assertion is m ore enlightened by empirical evidence, rather than recitation of impressions fro m personal encounters alone the latter can certainly lead to hypotheses to be subje cted to empircal test. I agree that we need to go beyond personal experie nce in assessing this issue, but on this list, I see arguments buttressed by personal experi ence almost daily. Andrew does not need me to defend him but I am confident that he could have be en much more specific and data based about Berliner & Biddle. There is a mountain of credible empirical evidence that points in the opposite direction from Berliner & Biddle. Their claim that the larger public's perception of the schools is the product of a conspiracy or right wing extremism or mere bad publicity is preposterous. I agree with Andrew that Berliner & Biddle are doing the ca use of public education no favor by trying to dismiss its critics and their arguments. I know that the criticisms, the unflattering studi es, and the negative reports in the media are distasteful--especially to people who are worki ng hard to do what they feel is the right thing. However, as one who follows that which is being sai d in the larger public arena, I think we of the education community will either have to move in the direction of that which the public wants or progressively be replaced. People who are as commit ted to public education as Al Shanker have been saying the same thing for years. Professor Ste dman's refutation of Berliner & Biddle may be painful to contemplate but if it moves us to confro nt reality rather than quibbling about its existence, he has, in my opinion, done us a great f avor.


1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 9 April 1996Dan ClineDHC@PAWNEE.ASTATE.EDU Berliner & Biddle have taken some heavy hits for t he Manufactured Crisis and for their response to Stedman's first response to the book. H is second response, "Respecting the evidence..." is apparently being received on this l ist as a balanced and correct refutation of the major claims made by Berliner & Biddle and I suppos e it is -it is certainly a significant and thoughtful contribution on its own. However, I thin k responses to these works by policy analysts and scholars of whatever pursuasion, even the criti cs of the schools among them, should perhaps rise above conclusions based just on hunches derive d from personal encounters. Andrew Coulson, for example, characterizes as "uns upported" and "far fetched" Berliner & Biddle's contention that American students are mo re broadly educated than their foreign counterparts based on his observation that in "pers onal travels" in Europe, "it is common to encounter high school and college students who spea k two, three or even four languages fluently" and are well versed in international events (by wha t measure, not stated). As long as we are on personal experience, I was raised in the Dakotas sp eaking German, English and Russian, as were my mother and her sixteen brother and sisters, out of economic necessity -farmers couln't do business with farmers and businesses from other com munities in the area who spoke other languages unless they were conversant in those lang uages. Since none of them (myself being the exception) attended school beyond the eighth grade, I wonder to what extent being multilingual is a measure of being "broadly educated." I wonder if being multilingual in Europe isn't more a function of close proximity of nations (like New Yo rk and Pennsylvania) and ready access to television broadcasts in many languages rather than a function of curriculum in the schools. I am not sure whether Berliner & Biddle were makin g an empirical knowledge claim or a simple assertion that American students are more br oadly educated, but in either case, an examination of the claim or assertion is more enlig htened by empirical evidence, rather than recitation of impressions from personal encounters alone--the latter can certainly lead to hypotheses to be subjected to empirical test.


1 of 1 Contributed Commentary on Volume 4 Number 1: Stedman The Achievement Crisis is Real: A Review of The Manufactured Crisis 8 Apr 1996Andrew Coulsonandrewco@IX.NETCOM.COM Like Berliner's 1993 EPAA paper, The Manufactured Crisis is a fanciful and selective romp through the data on public schooling. Anyone w ho is currently relying on or quoting this material will do themselves a great favor by readin g Larry Stedman's recent detailed critique of their treatment of the academic achievement data (a lso published in the Education Policy Analysis Archives). For those wondering if only the academic achieveme nt data is mistreated by Berliner and Bracey, the answer appears to be no. My bedtime rea ding over the past week has been a combination of the Manufactured Crisis and two telephone-book sized tomes: The Handbook of Human Intelligence (Edited by Sternberg), and the 1993 issue of the D igest of Education Statistics (Dept. of Ed.). Based on the research compiled in the Handbook, B. & B.'s coverage of the IQ data is very poor. They draw causal conclusions from correlation al data on numerous occations, and are prone to offer unsupported conjecture as fact. They infer from the positive correlation between years of schooling and higher IQs that schooling in creases IQ, but specialists in the field have argued the converse--that students who have an affi nity for academics and schooling and a high tested intelligence tend to choose to stay in schoo l for longer than students who perform poorly. Their figures for per-pupil education spending are outdated (1985) and misleading. They appear to combine private and public school spendin g, without mentioning that private schools spend less per-pupil, thus bringing down the averag e. The research on cost-effectiveness of public schools is given short shrift, and one of th e key meta-analyses, conducted by Childs and Shakeshaft, is not mentioned at all. Furthermore, their belief that American students a re more broadly educated than their foreign counterparts is entirely unsupported, not t o mention far-fetched based on my personal travels in Europe and North America. In Europe it i s common to encounter high-school and college students who speak two, three, or even four languages fluently, and are well versed in current events in the international arena. This (ag ain in my own experience) is much less common in the United States. The Manufactured Crisis, however, is not a total loss. As Larry points out in the introduction to his critique, people have a tendenc y to research their opinions only until they find support. Berliner and Biddle act as devils' advocat es for those, like myself, who are critical of public schooling, pushing us to support every aspec t of our arguments with hard evidence. Because B. & B.'s beliefs are so poorly defended, however, this ends up reinforcing the case for radical reform. Probably not the outcome t hey had in mind.