USF Libraries
USF Digital Collections

Educational policy analysis archives


Material Information

Educational policy analysis archives
Physical Description:
Arizona State University
University of South Florida
Arizona State University
University of South Florida.
Place of Publication:
Tempe, Ariz
Tampa, Fla
Publication Date:


Subjects / Keywords:
Education -- Research -- Periodicals   ( lcsh )
non-fiction   ( marcgt )
serial   ( sobekcm )

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
usfldc doi - E11-00236
usfldc handle - e11.236
System ID:

This item is only available as the following downloads:

Full Text
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20019999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00236
0 245
Educational policy analysis archives.
n Vol. 9, no. 39 (October 03, 2001).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c October 03, 2001
When an "a" is not enough : analyzing the New York State global history and geography exam / S.G. Grant.
x Research
v Periodicals.
2 710
Arizona State University.
University of South Florida.
1 773
t Education Policy Analysis Archives (EPAA)
4 856

xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 9issue 39series Year mods:caption 20012001Month October10Day 33mods:originInfo mods:dateIssued iso8601 2001-10-03


1 of 21 Education Policy Analysis Archives Volume 9 Number 39October 3, 2001ISSN 1068-2341 A peer-reviewed scholarly journal Editor: Gene V Glass, College of Education Arizona State University Copyright 2001, the EDUCATION POLICY ANALYSIS ARCHIVES Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education .When An "A" Is Not Enough: Analyzing the New York State Global History and Geography Exam S. G. Grant University at BuffaloCitation: Grant, S.G. (2001, October 3). An "A" is not enough: Analyzing the New York State Global History and Geography Exam. Education Policy Analysis Archives 9 (39). Retrieved [date] from Week 's report "Quality Counts" judges New York State's curriculum and assessment policy efforts to be an A." Surface-level reviews such as "Quality Counts" tell something abo ut the workings of state policy, but they are more useful as snapshots than as well-developed portraits of curriculum and assessme nt change. In this article, I analyze the new New York State Global Hi story and Geography standards and tests using a set of social studies-s pecific criteria which inquire deeply into the implications for real instr uctional change. From that vantage, I argue that New York's policy effort s, while seemingly well-intentioned and reflective of surface-level ch ange, fail to promote powerful teaching and learning in social studies. T eachers intent on producing ambitious teaching and learning will find little to interfere


2 of 21with their efforts. But as a set of reforms intende d to encourage substantive change, the new global history test fal ls short. By some reports, New York state has made considera ble strides in redesigning its state standards and assessment programs. For exampl e, the authors of Education Week's report, Quality Counts (see ) judge New York's efforts to be an "A." In that report, New York scor ed points for having new content standards in all school subjects and at elementary, middle, and high school levels; for having tests which employ multiple-choice, short an swer, and extended response questions; for requiring passing state assessments for high school graduation; and for using a range of policy tools such as report cards, ratings, financial assistance, and state sanctions to encourage improved test performance. Surface-level reviews such as Quality Counts tell us something about the workings of state policy, but they are more useful as snapshots than as well-developed portraits of curriculum and assessment change. Atte mpts to construct such portraits demand more rigorous criteria than whether a type o f test item appears or not. When such criteria are applied in the context of the new New York state global history exam, it is hard to justify Education Week's lofty grade. In short, an A from Education Week isn't enough. In this article, I do a document analysis of the n ew NYS Global History and Geography standards and tests using a set of social studies-specific criteria which inquire deeply into the implications for real instructional change. From that vantage, I argue that New York's curriculum and assessment efforts, while seemingly well-intentioned and reflective of surface-level change, fail to promote powerful teaching and learning in social studies. Teachers intent on producing ambiti ous teaching and learning will find little to interfere with their efforts. But as a se t of reforms intended to encourage substantive change, new global history test falls s hort.Design of the Study Led by Patricia Avery from the University of Minne sota, several colleagues and I from universities around the U.S. developed a set o f criteria by which to analyze the new state curriculum and assessments emerging in our re spective states. (Note 1) Drawing on the current thinking in our field, especially as it is reflected in national standards documents (e.g., National Center for History in the Schools, 1994; National Council for the Social Studies, 1994) and state-level standards (e.g., New York State Education Department, 1998), we constructed criteria that ask to what extent the new state tests ask students to: demonstrate knowledge of significant concepts and i ssues in history and the social sciences? consider multiple perspectives on issues and events ? manipulate and interpret social science data? engage in higher order thinking about significant s ocial studies concepts and issues? I operationalize these criteria in the sections wh ich follow. Note, however, that these measures really pose two questions. The first inquires about the simple existence of each criteria listed, e.g., is there any evidence to suggest that students much


3 of 21demonstrate knowledge of significant concepts? The second question implies a quality measure, e.g., to what degree must students demonstrate knowledge of significant concepts? The first kind of question is not unlike those asked by Education Week, where the singular appearance or absence of a criteria is deemed important. The second kind of question pushes deeper, asking about the importance or meaningfulness of the measure. Evidence of a measure is interesting, but the exten t to which that measure is meaningful seems ultimately more useful. The prevailing pattern that emerged from my analys is can be termed, "yes, but... ." Yes there is evidence of attention to the subject-spe cific criteria we developed, but inquiries into that evidence suggest that the new g lobal exam comes up far short of a substantive change.Background on New York State Curriculum and Assessm ent In New York state, the belief that tests drive cha nge is alive and well. But while the notion that tests matter is widely held, little empirical evidence supports a robust connection between tests and learning. In fact, Sta ke and Rugg (1991) argue that "in sixty years of vast international research on schoo l testing, the policy of emphasizing test performance in order to improve education has never been validated" (p. xx). If true, it is no surprise to learn that the available research su ggests that the relationship between testing and teachers' practices is complicated at b est (Cimbricz, in review; Cohen & Barnes, 1993; Firestone, Mayrowetz, Fairman, 1998; Grant, in press). Tests matter to teachers (see, for example, Smith, 1991a, 1991b), b ut how teachers interpret and act on the import of new tests is largely uncharted ground That little is known about if and how tests influe nce teaching and learning has yet to inhibit state-level policymakers in New York (an d most other states) from using them. To understand the recent changes in the state asses sment program, however, one needs to consider the long history of state involvement i n curriculum and testing. New York state policymakers draw on a long history of attempts to influence classroom teaching and learning. Administered for o ver 100 years, the Regents testing program tests high school students on standardized, criterion-referenced exams that are tied to state-developed course syllabi in all acade mic subjects. In social studies, students take the Global Studies test at the end of a two-ye ar Global Studies course sequence in ninth and tenth grades; eleventh graders take the U S. History and Government test after completing a course of the same name. State curricu la and tests also exist for elementary and middle school teachers and students.A Mix of Old and New in New York State Standards an d Assessments The most recent changes in the state curriculum an d assessments began in the early 1990s under the previous education commission er, Thomas Sobol. Richard Mills, commissioner since 1994, continued that effort. Int erestingly enough, Mills came to New York intending to decrease the traditional emphasis on standardized testing. The education reform movement Mills led in Vermont resu lted in a state-level assessment program based on student portfolios rather than on tests. Mills abandoned this approach in New York, however. Sensing that the state's draf t curriculum frameworks were being largely ignored, Mills reportedly asked a teacher t o explain. "'You don't get it,' the teacher said, with what Mr. Mills remembers as almo st a sneer. 'If the standards are not on the test, they're not real'" (Hartocollis, 1999, B1). (Note 2) That comment apparently proved key for Mills is no w an unabashed supporter of


4 of 21 standards-based tests as a vehicle for classroom ch ange. The Learning Standards for Social Studies (New York State Education Department, 1998) repres ent the state's latest K-12 curriculum; new tests in grades 5, 8, 10, and 11 are emerging over the next two years. Compared with the previous round of curricular rev isions in the mid-to-late 1980s, the new standards documents represent a mix of old and new. Virtually no change appears in the K-5 grades curricula, which c ontinue to follow an expanding horizons model. There are also no discernible chang es in the seventh and eighth grade U.S. and New York State history sequence, or in the twelfth grade Participation in Government and Economics courses. A modest change i s evident in the eleventh grade U.S. history and government course in that a emphas is on geography surfaces. Major changes seem localized at sixth grade, where the co urse of study expands from Western and Eastern Europe and the Middle East to the entir e Eastern hemisphere, and at ninth and tenth grades, where the emphasis has changed fr om a cultural approach as represented in Global Studies to a chronological, h istory-based study expressed as Global History and Geography. The state-level testing program also reflects a mi x of old and new. Compared to the tests in most other subjects, the new social st udies assessments seem the least changed. Whereas the new mathematics, science, and English-language arts tests make liberal use of open-ended and extended tasks, the s ocial studies exams continue to rely largely on multiple choice questions. Moreover, com pared to the tests in sister subject matters, the multiple choice questions posed on the social studies exams seem directed toward lower levels of understanding. The multiple choice questions notwithstanding, the new state social studies exams do look different from the old ones. The principal change is in the writing portion of the exam. Unlike many minimum competency tests, New Yor k students have always had to write essays on state exams. The new tests are diff erent primarily in the fact that a) students will no longer have a range of essay promp ts to choose from, and b) a new kind of essay question, a document-based question (DBQ), is being introduced on each of the fifth, eighth, tenth, and eleventh grade tests. (No te 3) A DBQ asks students to write an essay synthesizing information from as many as eigh t primary source documents (e.g., short quotes from government documents and famous i ndividuals, political cartoons, poems, charts and graphs). The DBQ from the Global History and Geography exam administered in June, 2000 is as follows: Historical context : Economic systems attempt to meet the needs of the people. Capitalism and communism represent t wo different ways to meet people's economic needs.Task : Using information from the documents and your kno wledge of global history, answer the questions that follow each document in Part A. Your answers to the questions will help you write the Part B essay, in which you will be asked to:Describe how these two economic systems attempt to meet the needs of the peopleEvaluate how successful each system has been at mee ting the economic needs of the people


5 of 21 This task is followed by eight documents, seven qu otations (e.g., R.W. Emerson, Adam Smith, Friedrich Engels) and one political car toon, which present contrasting views of communism and capitalism. One or two main idea questions accompany each document. An example of a document and the attendan t question follows: ...masses of laborers…crowded into factories. Theyare slaves of the machine and the manufacturer.Instead of rising as industry progresses, they sinkdeeper and deeper into poverty… Karl Marx andFriedrich Engels, The Communist Manifesto The attendant main-idea question is: "According to Marx and Engels, what was the effect of the capitalist f actory system?" After responding to short answer questions such as this, students are directed to: write a well-organized essay that includes an intro duction, several paragraphs, and a conclusion use evidence from at least four documents to support your response include additional related information. High school students will also write a second, "th ematic" essay based on a single prompt. The thematic essay from the June, 2000 Glob al History and Geography exam is: Write a well-organized essay that includes an intro duction, several paragraphs addressing the task below, and a conclus ion. Theme : Justice and Human Rights--Through history, the hu man rights of certain groups of people have been violat ed. Efforts have been made to address these violations.Task : Define the term "human rights" Identify two examples of human rights violations th at have occurred in a specific time and place Describe the causes of these human rights violation s For one of the violations identified, discuss one s pecific effort that was made or is being made to deal with the violation. Students are then advised: You may use any example from your study of global h istory. Do not use the United States in your answer. Some sugg estions you might wish to consider include: Christians in the e arly Roman Empire, native peoples in Spain's American colonies untouchables


6 of 21 in India, blacks in South Africa, Jews in Nazi Germ any, Muslims in Bosnia, Kurds in Iraq or Turkey, or Tibetans in China. Each of the two essays is scored by two classroom teachers on a 6 point rubric, from 0-5. On the DBQ above, a score of 0 "fails to address the task or theme, is illegible, or is a blank paper." By contrast, a score of 5: addresses all aspects of the task by accurately ana lyzing and interpreting at least four documents thoroughly describes and evaluates capitalism and c ommunism incorporates information from the documents in the body of the essay and may cite from the document in an appropriate fashion, but do es not copy the entire document incorporates relevant outside information such as t he early British factory system, Stalin's five-year plans, collapse of communist sys tem in the Soviet Union takes into account the point of view of the authors in the description and evaluation of capitalism and communism is a well-developed essay, consistently demonstrati ng a logical and clear plan of organization introduces the theme by establishing a framework th at is beyond a simple restatement of the task or historical context and c oncludes with a summation of the theme Scores between a 5 and a 0 reflect lesser attentio n to each of the points above. For example, under a score of 3, the third point states "incorporates limited or no relevant outside information." Once the tests have been corrected, teachers are d irected to a conversion table on the back cover of their manuals. There, they total a student's multiple choice and short answer scores (total of 61 possible points) and the n look across a series of columns from 0-10, which represent the least to most possible po ints on the two essays. At the cross-section of these two scores is a converted sc ore which ranges from 0-100. In the past, students had to score a 65 in order to pass t he exam. A 65 is still the targeted state score, but districts are allowed to lower the requi red passing score to 55 for the next couple of years. If the new tests themselves are only modestly revi sed, two other changes seem more dramatic. One is that the new fifth and eighth grade tests will mirror the high school exams in form and will produce individual st udent scores. Previously, tests at those levels, termed "Program Evaluation Tests," we re general knowledge exams aimed at helping teachers understand the effectiveness of their content and pedagogical decisions. The shift to Regents-like tests and indi vidual student scores at lower grades seems intended to raise the stakes of these tests b y tying them more directly to the high school Regents exams. The second change concerns th e function of the Regents test. In the past, passing Regents tests in all academic sub jects meant that a student earned a "Regents" diploma, a distinction of some note. Stud ents who desired to could opt to take the less rigorous Regents Competency Exam (RCT) and earn a local diploma. Beginning in 2001, ninth graders will no longer have these op tions. The RCT is being phased out, and all students will have to pass five Regents exa minations (English, mathematics, global history, U.S. history, and science) in order to graduate.


7 of 21"Yes, But... .": Analyzing the NYS Global History E xam By most any measure, NYS policymakers deserve cred it for the curriculum and assessment revisions they have made. They might hav e taken a less ambitious route by leaving the state curriculum and tests largely unch anged or by reverting to a minimal competency exam. Since they did not, Education Week 's grade of A may well be justified. But if the criteria applied are more rig orous and more specific to the subject matter of social studies than those used by Educati on Week, then other interpretations of the new standards and assessments seem valid. Recall that I analyzed the NYS Global History and Geography exam by asking to what extent the new state tests ask students to: demonstrate knowledge of significant concepts and i ssues in history and the social sciences? consider multiple perspectives on issues and events ? manipulate and interpret social science data? engage in higher order thinking about significant s ocial studies concepts and issues? Recall also that I split this question in two. Fir st, I looked for the mere existence of each criteria. Second, I inquired about the qual ity of the evidence for each criteria. My analysis suggests that while evidence for each of t he criteria can be found, in no case is the quality or meaningfulness of that evidence stro ng. In short, the answer to each question is, "yes, but... ."Knowledge of Significant Concepts and Issues To be sure, there is a whole lot of knowledge repr esented on the new global history exam. This claim prompts little surprise, h owever, given the scope of the course title (i.e., Global history and geography"), the 27 single-spaced page s of the state curriculum, and the fact that the course is taught over two school years. A quick review of the curriculum and test suggests apparent attent ion to significant concepts and issues: Geographic influences, religious beliefs, economic systems, political forces, cultural practices, and international relations map across a n array of developed and developing, ancient and modern civilizations. Yet even a surface-level analysis begins to yield some problems. For while the testmakers develop items for a wide range of concep ts and issues, a quick count of the multiple-choice questions offers some troubling pat terns. One pattern is that questions related to western nations (i.e., Europe, including Russia/USSR) dominate the test: Twenty-four questions assess issues relevant to the west, while only 10 questions each are assigned to India/Asia and to the rest of the w orld (Africa, Latin America, Caribbean, South America, and the Middle East). (Note 4) A sec ond pattern is that the numbers of questions related to early civilizations (8 questio ns) and the middle ages (9 questions) are notably subservient to those attached to the mo dern era (31 questions). (Note 5) This latter pattern could be predicted for two rea sons. One is that historians and social scientists know more about modern times than the past, so to see that truism reflected in the apportionment of test questions is no surprise. The second reason is that the state curriculum gives preference to the modern era (18 pages) over early (6 pages) and middle (3 pages) periods. Since the test is rep uted to reflect the state social studies


8 of 21 standards, it makes sense that the ratio of questio ns would reflect the chronological preferences established in the curriculum. The first pattern is harder to understand, however First, the clear preference for western-based questions flies in the face of the na tional movement to be more inclusive of other cultures. While the debate over multicultu ralism has been contentious, it is hard to understand why the testmakers would so clearly p rivilege western history. This action is also hard to understand from a curricular point of view. While New York policymakers' efforts at creating a multicultural c urriculum have been variously praised (Cornbleth & Waugh, 1995) and excoriated (MacDonald 1992), the rhetoric in the social studies standards appears to support a stron g endorsement of a global perspective: This curriculum provides students with the opportu nity to explore what is happening in various regions and civilizations at a given time. In addition, it enables them to investigate issues and themes from multiple perspectives and make global connections and linkages that lead to in-depth unde rstanding. (New York State Education Department, 1998, p. 71) The decision to emphasize questions related to the west is especially difficult to defend when one realizes that within each of the ch ronological units described in the state standards is attention to western and non-western people, events, and issues. For example, the unit entitled, "Global Interactions (1 200-1650)," is divided into four sections, two of which--Early Japanese History and Feudalism and The Rise and Fall of the Mongols and Their Impact on Eurasia--are explic itly non-western. (Note 6) European issues and events do dominate the later units as wo rld and cold wars get heavy play. That said, on the relationship between the west and the rest of the world, the disparity between the state standards and the state test is s tark. The disparities between the nations and eras repre sented and between the state curriculum and exam are interesting, but really do not help us understand whether the concepts and issues portrayed are significant But then what constitutes a significant event turns out to be a pretty thorny issue, both f or historians (see, for example, Carr, 1961) and for students (Barton & Levstik, 1997, 199 8; Grant, 2001; Seixas, 1994, 1997). One might debate the relative merits of questions r elated to Karl Marx v. Ho Chi Minh, but it seems that with few exceptions the test addr esses the big ticket items of a standard account of global history. And that's part of the problem. The disparity in q uestions between western and non-western nations notwithstanding, the real issue related to significance is the type of questions asked rather than the content. In short, test makers aimed at low-level knowledge questions rather than at higher-order thi nking questions. As a case in point, consider this multiple-choice question: The Magna Carta, the Glorious Revolution, and the w ritings of John Locke all contributed to Great Britain's devel opment of: absolute monarchy 1. ethnic rivalries 2. parliamentary democracy 3. imperialist policies 4. Typical of the multiple-choice section, this quest ion reflects an emphasis on generally expected, and clearly western constructs and events. But while the significance


9 of 21 of these elements to global history is undeniable, the question merely asks students to identify and label them. I address this notion of i nsignificant questions about significant events more directly in succeeding sections. For no w then, my analysis suggests that, yes, the new state global exam does demonstrate att ention to important concepts and issues, but does so in a way that may not push stud ents' thinking.Multiple Perspectives The criterion of the extent to which the new state test addresses multiple perspectives is another case of "yes, but... ." Whi le the inclusion of the DBQ indicates a move toward multiple views, that move is less appar ent in the multiple choice section than one might expect. Moreover, the heavy tilt tow ard western themes undercuts the range of perspectives possible. Several multiple-choice questions appear to reflec t diverse perspectives because they give students multiple pieces of information. On closer inspection, however, all but two questions present compatible rather than divers e viewpoints. Typical of this kind of question is the following: Base your answer to question 10 on the statements b elow and on your knowledge of social studies. Statement A : The might of a country consists of gaining surpluses of gold and silver. Statement B : A nation's strength is found in economic independ ence and the maintenance of a favorable balance of trade.Statement C : We need to gain colonies both a sources for raw m aterials and as markets for our manufactured goods.Which economic system is being described by thesestatements? traditional 1. feudal 2. command 3. mercantile 4. Students read three different statements, but each statement is necessarily tied to the others as a vehicle for defining mercantilism. Rather than dealing with multiple perspectives, then, students must only deal with mu ltiple pieces of information. (Note 7) The two multiple choice questions which do ask stu dents to untangle multiple views employ the same stem: Base your answers to questions 46 and 47 on the spe akers' statements below and on your knowledge of social st udies: Speaker A : The gods approached Vishnu, the lord of


10 of 21 creatures, and said: "Indicate to us that one perso n among mortals who alone is worthy of the highest rank... Vishnu reflected, and brought forth a glorious son who bec ame the first king.Speaker B : The traditional African society, whether it had a chief or not, was a society of equals and it conduc ted its business through discussion. Speaker C : Ideally, the best form of government is one where every citizen not only has a voice, but also, at least occasionally, is called on to take actual part. Speaker D : A monarch's authority comes directly from God, and this is how the leadership and power in a socie ty should be determined. 46. Which speakers would support the theory explain ing the power of France's Louis XIV, Spain's Philip II, and Engla nd's Elizabeth I? A and D 1. B and C 2. A and C 3. B and D 4. 47. Which speakers would agree with the idea that s ome form of democracy is the best way to govern a society? A and D 1. B and C 2. A and C 3. B and D 4. One could quibble with the fact that the four stat ements represent only two views of government, but that really is a quibble. The qu estions might have been worded more clearly (especially #46), but the point remains: St udents must be able to sort through differing views of government in order to make sens e of the questions posed. What seems like a similar quibble above rises to t he level of critique in the DBQ. The eight documents divide cleanly into four catego ries: those that support capitalism (an excerpt from an unidentified work by Ralph Wald o Emerson; an excerpt from Adam Smith's Wealth of Nations ), those that support communism (a quote attributed to "Katia," a 16-year-old ninth grader from Moscow in the 1980s ; an excerpt from Friedrich Engels, Principles of Communism ; an excerpt from Harry Schwartz in The New York Times, 1952), those that critique communism (an excerpt fr om, "The Peasant Wars on the Kremlin," by T. P. Whitney; a political cartoon fro m the Providence Journal Bulletin ), and those that critique capitalism (an excerpt from The Communist Manifesto by Marx and Engels). (Note 8) Those who would question the documents selected would rightfully emphasize the clean lines of support for and critique of each system. There is no gray area here, for depending on the source, cap italism and communism are either portrayed as sin or salvation. In the first part of their essays, students are asked merely to describe how each system attempts to meet its citiz ens' needs. Since five of the eight documents provide clear fodder for this task, it ha rdly seems a significant challenge. The


11 of 21 second task, to evaluate how successful each system has been in meeting its citizenry's needs, seems more cognitively provocative. Here, st udents would presumably draw on the documents which critique each system. But notic e what is missing: Students are provided only with partisan critiques. No data appe ar, for example, on how citizen-workers have fared under the respective sys tems. Presumably, students will draw on their knowledge that some countries like the for mer Soviet Union have renounced communism. But without more and better data, and es pecially data that offers direct comparisons, it is difficult to see how students ca n do much with this task. Those who would defend this DBQ might counter that even a weakly constructed DBQ offers a profoundly different task than student s normally undertake on a standardized test. That so much of the testing in s ocial studies relies on multiple-choice questions has long been a sore spot among social st udies educators. Clearly, this DBQ offers a new opportunity for assessing students' kn owledge and skills. Taken together, these points underscore the "yes, but..." argument about the new global exam. Including a DBQ ratchets up the substa nce of the test and begins to promote the notion of multiple perspectives. But th e strength of that claim is undercut by what, with seemingly little effort, could have been a more powerful experience. Substituting documents that presented more nuanced views of capitalism and communism and that presented some comparative data would have gone considerable distance in beefing up a fledgling effort.Manipulating and Interpreting Social Science Data As noted above, the authors of the DBQ could have enhanced the student tasks by including some comparative data. Doing so would hav e contributed greatly to the generally weak way that social science data are han dled on the global exam. The types of questions represented on the new test generally call for definitions of terms (e.g., limited monarchy, totalitarianism, NAF TA) and identification of people, events, and social trends (e.g., Napoleon Bonaparte French Revolution, democracy in Latin America). Few questions probe much below a su rface-level knowledge of global history. And of those questions, a mere handful dea l with social science data. To be sure, there are questions which employ illustrations, pol itical cartoons, and maps. None of these, however, qualifies as data in the sense that students are presented with information that they must manipulate and interpret in order to answer the attendant questions. Of the 50 multiple choice questions, then, only three call upon students to use data. One question presents students with two circle graphs o f the world population by region. The first graph shows the distribution for Europe, Chin a, Latin America, North America, India, and four other areas in mid-1992; the second graph projects the distribution for the same regions in 2025. Two questions follow: Which factor best explains the projected change in China's population by 2025? increased immigration to China 1. religious doctrines discouraging birth control 2. government limits on family size 3. increased agricultural production in China 4. Which conclusion about world population in the next 25 years is


12 of 21 best supported by the information in these charts? Technological improvements will cause a population decline throughout Asia. 1. Developed nations will be home to a majority of the world's population. 2. Efforts to curb population growth in developing nat ions will be successful. 3. Africa may experience problems with overpopulation. 4. A few questions later, students encounter a chart describing Internet usage in countries across the world. Three categories of usa ge along with representative countries are listed. For example, "heavy usage" countries in clude Canada, Norway, and the United States; "medium usage" countries include Chi le, Britain, and Argentina; and "little use" countries include Mexico, Columbia, an d Saudi Arabia. One question follows: Which conclusion about Internet usage can be drawn from this chart? Developing nations have easier access to the Intern et than developed nations do. 1. A high standard of living in a nation is linked to high Internet usage. 2. Internet usage limits international cooperation. 3. Eastern Hemisphere nations use Internet connections more than Western Hemisphere nations do. 4. The final data-based question features a web diagr am of automobile production using straight lines and arrows illustrating the gl obal connections between auto companies and the countries in which they originate For example, Chrysler/USA is connected by a straight line to Renault/France and to Hyundai/South Korea and by an arrow to Mitsubishi/Japan. The distinction between straight lines and arrows is not explained. One question follows: Which conclusion can be drawn about global economic s in the 1990s? Countries became more economically isolated. 1. Higher tariffs reduced trade between nations. 2. France dominated the world automobile industry. 3. Economies of the world were increasingly interdepen dent. 4. These questions meet the ostensible parameters of data-based situations: Students are presented with some data from which they must i nfer trends. That said, there are at


13 of 21 least two problems with these questions. One proble m is that students need not manipulate any of the data in order to answer the a ttendant questions. Students must make an interpretation, but in all cases the "right answers are fairly obvious. The reason the answers are so obvious speaks to a second problem: Not one of the questions demands much in the way of prior social s tudies knowledge. In short, the questions are cast such that only one answer makes common sense. Consider just two examples. First, without knowing anything about Chi na, the question about the projected change in its population can only be reasonably ans wered with response #3 since it is the only answer which explains a declining populace. Th e answer to the question about global economics is just as commonsensical. The onl y response consistent with the web-like diagram is #4 which features the language of "increasingly interdependent." (Note 9) True, students need to know the vocabulary used--population, interdependent, and the like--but these are hardly arcane words use d only in social studies contexts. So while students are asked to make inferences from th e information presented, not only are they low-level inferences at best, but the possible answers are phrased such that the answers are obvious. Once again, then, the surface-level qualifications of the NYS global exam pass muster, but a peek below that surfaces undercuts an y confidence in the A grade assigned by Education Week. As with each of the preceding cr iteria, the new test fails to push students' thinking in substantive directions. The a ppearance of asking students to manipulate and interpret data is not enough.Higher Order Thinking It is probably clear by now that this last criteri a, the extent to which the new test asks students to engage in higher order thinking ab out significant social studies concepts and issues, lies at the heart of my critique. The t est makers can legitimately claim some attention to each of the preceding criteria. On the level of that attention, however, reasonable objections can be lodged. I will not spe culate as to why the exam was constructed in this manner, but that it came so clo se to being a rich experience for students only to fail, is discouraging. Consider two examples of how the exam questions mi ght have been enriched. I argue above that the DBQ is composed entirely on pa rtisan views of capitalism and communism. A small, but significant improvement wou ld be to substitute a graph offering descriptive data on the comparative econom ic productivity and/or the social service conditions of the two nations. Such an addi tion would not only expand the range of documents students consider, but it would also h elp them make a more reasoned response to the portion of the essay prompt that ca lls for them to "evaluate how successful each system has been at meeting the econ omic needs of the people." The multiple choice questions might also have been improved. Consider this example from the 1994 NAEP Geography Assessment (Na tional Assessment of Educational Progress, 1994): Statistical Comparison of Two Countries Country ACountry B Total Population7,193,000123,120,000Urban-Rural Urban 49.0% 76.7%


14 of 21 Rural51.0%23.3%ReligionsRom. Cath.:92.5% Baha'i: 2.6% Other: 4.9% Shinto*: 89.5% Buddhist*: 76.4% Christian: 1.2% Other: 9.3% Life Expectancy atBirth (years) Male Female 50.9 55.4 75.9 82.1 Age Distribution Under 15 15-29 30-44 45-59 60-74 over 74 43.4% 26.4% 15.7% 9.3% 4.4% 0.8% 19.0% 21.6% 22.4% 20.1% 9.2% 7.7% % of the Population over 25 with No Formal Schooling 48.6%0.3% Leading Exports(as % of total exports) Natural Gas: 21.0% Tin: 12.0% Zinc: 5.7% Silver: 5.6% Antimony: 4.0% Coffee: 2.0% Sugar: 1.5% Hides: 1.4% Motor Vehicles:18.4% Machinery: 10.9% Iron and Steel: 5.8% Chemicals: 5.3% Textiles: 2.6% Vessels: 1.5% Radios: 0.8% Televisions: 0.7% (*Some persons practice bothreligions.) Which of the following statements most accurately d escribes Country A? A. It is dependent on raw material exports. B. It probably has a high literacy rate. C. It has a predominantly urban population D. It will experience slow population growth. Which of the following statements most accurately d escribes Country B?


15 of 21 A. It has few medical facilities. B. It is industrialized. C. Its primary imports are manufactured goods. D. Its population is primarily employed in agricult ure. Country B is most likely A. Botswana B. India C. Ireland D. Japan Like the questions on the NYS global exam, these e xamples require understanding of significant social studies termino logy. Unlike those questions, however, these examples push students to do more th an define those terms. The first two questions demand that students evaluate the data in each cell and to draw conclusions across those cells. For example, in the first two q uestions, each of the possible answers directs students toward at a different cell of data To select the best answer, then, a student must evaluate the information across the ch art. The third question also asks students to look across multiple cells, but it adds a twist: Students must compare their assessment of Country B's attributes with their pre vious knowledge of world countries in order to select the best response. These two brief examples point to the possibilitie s missed on the current exam. It seems safe to say that the bulk of the new global e xam aims at low-level knowledge and understanding. The majority of the objective questi ons call for defining terminology, identifying significant people, places, and events, and in the case of the short answer section, describing the main point of a document. S urprisingly, the essays push no harder. The thematic essay asks students to complet e several tasks, but by giving the students numerous examples of human rights cases, i t is difficult to imagine many students struggling. The DBQ seems similarly poised Students must synthesize the views from eight different documents, but there is no nuance in any of them and the clean divisions among them play directly into the t asks to which students are assigned. Taken together, the array of questions on this exam promise much. They do not deliver. There is one more dimension that is worth note und er the general criteria of higher order thinking. The NYS exam presumably scor es high on the Education Week criteria in part because of the "extended response" items or essays. Moreover, the DBQ seems designed to signal a change in the structure of the social studies exams: One might argue that such a question represents a major shift away from traditional testing and toward more authentic assessment of students' histo rical understanding and reasoning. The scoring guide for the test, however, mitigates that claim: In short, students can easily pass the test without a single DBQ point. In fact, students can pass the exam without any essay points at all. A conversion chart on the last page of the teacher guidelines indicates that if students total a minimum of 54 points from the total of 61 possible multiple choice and short answer questions, they pass with a converted score of 65 regardless of whether or not they even attempt the essays. (Note 10) In this light, one can argue that the written portion of the new test has been substa ntially discounted compared to the previous exams. Where the essays once counted for 4 5% of a student's score, they now account for only 29%. Thus students can leave the e ssays blank, answer correctly


16 of 21approximately 72% of the multiple choice and short answer questions, and still pass the exam. Adding the DBQ, then, can be read as a minor revision at best.Implications Since the mid-1990s, state policymakers have intro duced a number of curriculum reforms such as new state standards for social stud ies. Preliminary indications (Grant, Derme, Gradwell, Lauricella, Pullano, & Tzetzo, 200 0) suggest, however, that NYS global teachers view the curriculum and assessment changes as a mixed bag. Some applaud the state's move to a chronological approac h as a more coherently historical mode. Others condemn this move (and some individual teachers and whole departments have rejected it) arguing that it undercuts the pow er of a cultural studies approach. More important than the curricular changes, howeve r, are teachers' concern about the new state tests (Grant, 1997a, 2000). This make s sense for two reasons. First, the curriculum documents produced thus far offer teache rs little assistance in making concrete instructional decisions (Grant, 1997b). Se cond, the messages teachers receive often promote the view that tests are intended to d rive change (Grant, 1996). For example, during sessions introducing the new state social studies standards, one representative from the New York State Education De partment said that new tests will "help grow change in the system." During another se ssion, a different SED representative said, "New assessments will represen t a change in instruction....Kids won't perform well until (teachers') instruction re flects this." And at yet a third meeting, NYS Commissioner Richard Mills added, "Instruction won't change until the tests change." The message that tests matter also surface d during local school and district meetings. A suburban district social studies superv isor, for example, told teachers that "change in content will come if we change the tests ." An urban district supervisor observed, "If we change the assessments, we'll chan ge instruction" (Grant, 1996, p. 271). One might question the focus of test influence--ins truction, curriculum, or the "system" in general--but it is hard to miss the larger point : tests matter. But how the new tests will matter deserves continu ed investigation. Our initial work in this area (Grant, Derme, Gradwell, Lauricel la, Pullano, & Tzetzo, 2000) suggests that teachers' views of the new tests refl ect some ambiance. Most teachers support the use of documents and the DBQ. Yet from what teachers have seen in the test sampler disseminated by the state education departm ent, few see this move as necessitating a fundamental shift either in their o wn pedagogies or as indicating a fundamental shift in the state's emphasis on social studies knowledge as represented in multiple-choice questions. The analysis above, which focuses on the first tes t administered to NYS tenth graders last spring, suggests that teachers have it about right: The new test represents little in the way of fundamental change, and so can be read as demanding little change in classroom practices. True, some teachers report a r atcheting up of anxieties by students, parents, and administrators as test scores become m edia fodder. But responding to test score concerns and responding to the tests at hand may be two very different things.NotesThe impetus for this action was a symposium entitle d, "State Standards-Based Assessments and the Social Studies" held during the annual conference of the National Council for the Social Studies, San Antoni o, Texas in November, 2000. Pat Avery and I were joined by Robin Chandler (Kent ucky), Jean Craven (New 1.


17 of 21Mexico), and Ceola Ross Baber (North Carolina).Thanks to Sandra Cimbricz for bringing this quote t o my attention. 2. Mock test items, called test samplers, are availabl e for the grades 5, 8, 10, and 11 tests (see ). The first administration of the grade 5 test is scheduled for November 2001; th e new grade 8 and 11 tests are scheduled for June 2001. 3. Seven additional questions lump together people, pl aces, and events such that it is difficult to ascribe them to a category. 4. Three additional questions span these time ranges a nd thus are difficult to categorize. 5. The other two segments are: The Resurgence of Renai ssance Europe and Global Trade and Interactions. 6. Alert readers will note that Statement C is the key to the correct answer. Students might consider Statements A and B, but these are ge neral features of most economic systems. 7. The quote from "Katia" might be double-counted as b oth in support of communism and in opposition to capitalism in that, before the bulk of the quote which does the former, she offers this presumed cri tique of capitalism: "Capitalists are rich people who own factories and have lots of money and workers." 8. I recognize that the adverb "increasingly" is probl ematic since no comparative data is presented. Nevertheless, in testmakers' parlance it is clearly the "best answer." 9. Even more startling is the fact that in those many districts that opted to lower the passing score to 55, students need only get 44 of t he possible 61 points to pass. 10.ReferencesBarton, K., & Levstik, L. (1998). "It wasn't a good part of history": National identity and students' explanations of historical significance. Teachers College Record, 99 (3), 478-513.Barton, K. C., & Levstik, L. S. (1997, March 1997). Middle Graders' Explanations of Historical Significance. Paper presented at the American Educational Resear ch Association, Chicago.Carr, E. (1961). What is history? New York: Vintage. Cimbricz, S. (in press). State testing and teachers thinking and practice: A synthesis of research. Educational Policy Analysis Archives. Cohen, D., & Barnes, C. (1993). Pedagogy and policy In D. Cohen, M. McLaughlin, & J. Talbert (Eds.), Teaching for understanding: Challenges for policy a nd practice (pp. 207-239). San Francisco: Jossey-Bass.


18 of 21Cornbleth, C., & Waugh, D. (1995). The great speckled bird: Multicultural politics and education policymaking New York: St. Martin's. Firestone, W., Mayrowetz, D., & Fairman, J. (1998). Performance-based assessment and instructional change: The effects of testing in Mai ne and Maryland. Educational Evaluation and Policy Analysis, 20 (2), 95-113. Grant, S. G.. (1996). Locating authority over conte nt and pedagogy: Cross-current influences on teachers' thinking and practice. Theory and Research in Social Education, 24 (3), 237-272. Grant, S. G.. (1997a). Opportunities lost: Teachers learning about the New York state social studies framework. Theory and Research in Social Education, 25 (3), 259-287. Grant, S. G.. (1997b). A policy at odds with itself : The tension between constructivist and traditional views in the New York state social studies framework. Journal of Curriculum and Supervision, 13 (1), 92-113. Grant, S. G.. (2000). Teachers and tests: Exploring teachers' perceptions of changes in the New York state testing program. Educational Policy Analysis Archives, 8 (14), 1-16. Grant, S. G.. (2001). It's just the facts, or is it ? An exploration of the relationship between teachers' practices and students' understan dings of history. Theory and Research in Social Education 29 (1), 65-108. Grant, S. G.. (in press). An uncertain lever: The i nfluence of state-level testing in New York State on teaching social studies. Teachers College Record Grant, S. G., Derme, A., Gradwell, J., Lauricella, A., Pullano, L., & Tz etzo, K. (in press). Teachers, tests, and tensions: Teachers respond to the New York state gl obal history exam. International Forum for Social Studies Hartocollis, A. (1999,). The man behind the exams: New York's education chief pushes agenda of change New York Times, pp. B1, B10. MacDonald, H. (1992). The Sobol report: Multicultur alism triumphant. The New Criterion, 10 (5), 9-18. National Assessment of Educational Progress. (1994) NAEP Geography Assessment Washington, DC: National Center for Educational Sta tistics. New York State Education Department. (1998). Social studies resource guide Albany, NY: Author.Seixas, P. (1994). Students' understanding of histo rical significance. Theory and Research in Social Education, 22 (3), 281-304. Seixas, P. (1997). Mapping the terrain of historica l significance. Social Education, 61 (1), 22-27. Smith, M. L. (1991a). Put to the test: The effects of external testing on teachers. Educational Researcher, 20 (5), 8-11.


19 of 21 Smith, M. L. (1991b). Meanings of test preparation. American Educational Research Journal, 28 (3), 521-542 Stake, R., & Rugg, D. (1991). Impact on the classro om. In R. E. Stake (Ed.), Advances in program evaluation (Vol. 1, pp. xix-xxii). Greenwich, CT: JAI Press.About the Author S. G. GrantAssociate Professor517 Christopher Baldy HallUniversity at BuffaloBuffalo, New York 14260716-645-2455 x1135 Email: S. G. Grant is an Associate Professor of Social Stu dies Education in the Department of Learning and Instruction at the University at Buffa lo. His research interests lie at the intersection of state curriculum and assessment pol icies and teachers' classroom practices, with a particular emphasis in social stu dies. In addition to publishing papers in both social studies and general education journals, Dr. Grant has published Reforming Reading, Writing, and Mathematics: Teachers' Respon ses and the Prospects for Systemic Reform (1998; Lawrence Erlbaum Associates) and (with Bruc e VanSledright) Constructing a Powerful Approach to Teaching and Le arning in Elementary Social Studies (2001, Houghton Mifflin).Copyright 2001 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, or reach him at College of Education, Arizona State University, Tempe, AZ 8 5287-0211. (602-965-9644). The Commentary Editor is Casey D. C obb: .EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing


20 of 21 Richard Garlikov Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Education Commission of the States William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX) Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton Hugh G. Petrie SUNY Buffalo Richard C. Richardson New York University Anthony G. Rud Jr. Purdue University Dennis Sayers California State University—Stanislaus Jay D. Scribner University of Texas at Austin Michael Scriven Robert E. Stake University of Illinois—UC Robert Stonehill U.S. Department of Education David D. Williams Brigham Young University EPAA Spanish Language Editorial BoardAssociate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico Adrin Acosta (Mxico) Universidad de J. Flix Angulo Rasco (Spain) Universidad de Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho Alejandro Canales (Mxico) Universidad Nacional Autnoma Ursula Casanova (U.S.A.) Arizona State Jos Contreras Domingo Universitat de Barcelona


21 of 21 Erwin Epstein (U.S.A.) Loyola University of Josu Gonzlez (U.S.A.) Arizona State Rollin Kent (Mxico)Departamento de InvestigacinEducativa-DIE/ Mara Beatriz Luce (Brazil)Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.brJavier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mxMarcela Mollis (Argentina)Universidad de Buenos Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mxAngel Ignacio Prez Gmez (Spain)Universidad de Daniel Schugurensky (Argentina-Canad)OISE/UT, Simon Schwartzman (Brazil)Fundao Instituto Brasileiro e Geografiae Estatstica Jurjo Torres Santom (Spain)Universidad de A Carlos Alberto Torres (U.S.A.)University of California, Los