xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20009999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00158
Educational policy analysis archives.
n Vol. 8, no. 14 (February 24, 2000).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c February 24, 2000
Teachers and tests : exploring teachers' perceptions of changes in the New York state testing program / S.G. Grant.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 8issue 14series Year mods:caption 20002000Month February2Day 2424mods:originInfo mods:dateIssued iso8601 2000-02-24
1 of 28 Education Policy Analysis Archives Volume 8 Number 14February 24, 2000ISSN 1068-2341 A peer-reviewed scholarly electronic journal Editor: Gene V Glass, College of Education Arizona State University Copyright 2000, the EDUCATION POLICY ANALYSIS ARCHIVES. Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education Teachers and Tests: Exploring Teachers' Perceptions of Changes in the New York State Testing Program S. G. Grant State University of New York at BuffaloAbstractHow do teachers change their pedagogical practices? While many current initiatives seek to raise educational stand ards and improve student academic performance, there is a curious ga p in national and state reforms. Considerable attention is given to d efining higher expectations for what students will know and be abl e to do, yet little attention is given to how teachers should learn new pedagogical ideas and practices. This exploratory study uses focus gr oup interview data collected over two years to examine how cross-subje ct matter groups of elementary and secondary New York State teachers re spond to one way of learning to change their classroom practices: st ate-level testing. Analysis of the data highlights three issues: the n ature and substance of the tests, the professional development opportuniti es available to teachers, and the rationales for and consequences o f the state exams. Many current initiatives seek to raise educ ational standards and improve student
2 of 28academic performance. Yet, there is a curious gap i n the recent talk about national and state reforms. While much attention focuses on defi ning higher expectations for what students will know and be able to do, little attent ion is given to how teachers should learn new pedagogical ideas and practices. Such pol icies as the federal Goals 2000: Educate America Act and the New York New Compact for Learning focus on the resources, conditions, and practices necessary for all students to learn. None of these efforts, however, seriously addresses how experienc ed teachers will learn the intended innovations. How do teachers change their pedagogical pr actices? Some suggest change comes through new subject matter standards proposed by pr ofessional organizations (National Council for Social Studies, 1994), by national grou ps (National Center for History in the Schools, 1994), or by state education departments ( New York State Education Department, 1996). Others believe teachers change t heir practices in response to organizational restructuring (e.g., smaller classes block scheduling). Still others assert that real change in the classroom lives of teachers and students depends on changes in state-level assessments (Comfort, 1991; Smith & O'D ay, 1991). The assumption in this last case is that testing drives much of what teach ers do, and so curricular and instructional change will occur if and when state t ests change. This last idea is intriguing for, if true, it suggests the potential for big pedagogical changes with a modicum of policy effort: Change the test and one changes teachers' practices. New York state policymakers seem taken w ith this approach, for although they have developed new curriculum standards, it is revision of the state testing program which gets most of the attention (Grant, 1997a). Th e scope of that revision is wide. One piece is the change from program evaluation tests a t the elementary level to high-stakes individual student testing. A second piece is the p hase-out of the less demanding high school Regents Competency Tests and the requirement that all students pass the more demanding Regents tests. A third piece is a change in the content and format of all state tests presumably to reflect the higher expectations expressed in the state's new standards documents. What sense do teachers make of these new st ate tests and how, if at all, do the tests influence their classroom practices? Strange as it seems, there is little empirical evidence to suggest how teachers, especially teachers at dif ferent grade levels, respond to changes in state tests. Assessment is a particularly hot to pic in educational circles today, yet there is surprisingly little research which digs deeply i nto teachers' understandings of the import of standardized tests (Cohen & Barnes, 1993; Grant, in press). Corbett and Wilson's (1991) study of teachers' reactions to a n ew Maryland testing program is well-known as is the on-going work of Mary Lee Smit h and her colleagues in Arizona (Noble & Smith, 1994; Smith, 1991; Smith, Heinecke, & Noble, 1999), but these are few studies in a field that is more prone to study students' responses than teachers'. In this article, I use the data collected t hrough focus group interviews over two years to explore the relationships between teachers and tests. My findings suggest that teachers need to be much more involved in the proce ss of changing state assessments, and that professional development needs to be more attuned to the different needs teachers have.The Study The Teacher Learning and Assessment (TLA) r esearch project (Note 1) is designed to look generally at the intersection of teachers a nd assessments. The research team is a cross-subject matter group of faculty and students (English, mathematics, science, and
3 of 28social studies) who are interested in exploring the relationship between teacher learning and state-level testing. Our study questions includ e: a) In what ways are tests and test results used in classrooms, schools, and the distri cts? b) What do the proposed changes in state-level tests mean for teachers and learners ? c) How are teachers being prepared to respond to the new state assessments? and d) What c hallenges do teachers and administrators anticipate in moving toward new stat e assessments? In each case, we are interested in the extent to which these issues diff er across school subject matters and grade levels.Data Collection In the first year of data collection, we or ganized two focus groups, one composed of 7 elementary school teachers and counselors and one composed of 12 high school teachers. The participants represented a cross-sect ion of urban, suburban, and rural school districts in western New York state, a bread th of teaching experience (2-25 years), and a range of school subjects (language ar ts, mathematics, science, and social studies). Each of the two-hour focus group intervie ws was tape-recorded and transcribed. During the second year of data collection, we again organized separate elementary and secondary focus groups. We debated whether to: a) reconstitute the original groups only; b) develop new groups of teachers separate fr om those involved in the first year's interviews; or c) call together groups that mixed t eachers new to the project with those who had participated during the previous year. We r ejected the first option, fearing that attrition might leave us with groups that were too small. We also rejected the second option, though largely because of timing: We did no t think we could hold four focus groups near the end of the school year. In the end, we decided to constitute mixed groups for two reasons. One reason was that we wanted to e xpand the number of teachers we were talking with; the second reason is that we wer e interested in how the two groups might interact. The secondary focus group consisted of 8 teachers representing mathematics, science, English, and social studies; 5 of the 8 were in the original sample. The elementary focus group consisted of 5 teachers, 3 of whom were in the original sample. (Note 2) The data consist of interview transcripts o f the focus group sessions and post-interview evaluations completed by the partici pants. The focus group interviews followed a semistructured interview protocol (see Appendix). Questions used during the first year asked participants to construct a me taphor to represent their sense of the changes in state-level testing, what the new tests mean for teaching and learning across school subjects, how teachers are being prepared fo r new standards and new assessments, and what challenges teachers believe t hey face. The post-interview questions asked the participants to reflect on the issues raised around the relationship between state-level assessment and classroom practi ce. The interview protocol was largely the same during year two. Changes consisted of replacing the metaphor task with a fill-in-the-blank exercise ("I used to think of t he state assessment as _________, now I [still] think of it as ________________.") and the addition of probes that asked participants if they sensed a change from last year to the present. There were no changes to the postinterview evaluation.Data Analysis
4 of 28 All data were analyzed inductively from an interpretivist stance (Bogdan & Biklen, 1982; LeCompte, Preissle, & Tesch, 1993). That stan ce emphasizes the importance of context, and the multiple ways individuals construc t meaning. All data were also analyzed using a constant comparative method (Bogda n & Biklen, 1982; Glaser, 1978). That method assumes that data collection and analys is are recursive, one informing the other throughout the course of the study. After cod ing the data both within and across grade levels and subject matters, I began seeking p atterns in the informants' responses. The themes which emerged reflect the full data set, but in each case I highlight the implications for social studies. Although this data can be considered largel y exploratory, patterns and themes surfaced as the interview and evaluation data were analyzed related to the research questions. In the analysis of the focus group inter views, I focused on: how teachers make sense of, and make different sense of, the state cu rriculum and assessment documents they encounter; the kinds of learning opportunities they attend, and how, if at all, these reforms and opportunities influence what teachers t hink about and do in their classrooms. Looking across the interviews, I saw pa tterns which help explain the teachers' responses in a social context and the nat ure of their learning in an array of social settings. The three preliminary patterns I s ynthesized from the data and report on in this paper relate to the nature and substance of the tests, the professional development opportunities available to teachers, and the ration ales for and the consequences of the state exams.On Tests and Teaching Standardized tests matter. The professional literature is replete with debates about tests as a means of accountability, as measures of performance, and as levers of change (Corbett & Wilson, 1991; Editors, 1994; Feltovich, Spiro, & Coulson, 1993; Finn, 1995; Fuhrman, Clune, & Elmore, 1988; Koretz, 1988; Ravit ch, 1995; Resnick & Resnick, 1985). These concerns become elevated when situatio ns like CTB/McGraw-Hill's mis-scoring of almost 9000 New York City students' tests occur. In all of the talk about tests, however, one area gets scant regard: What te achers learn from tests, and if and how that knowledge affects their instructional prac tice. Common sense holds that tests drive classroom instruction. Evidence for that opin ion is thin, however. Much research focuses on the relationship between students and te sts (see, for example, Natriello & Pallas, 1998; Stiggins & Conklin, 1992; Wolf, 1998) but relatively few empirical studies explore the relationship between teachers a nd the tests they administer (Corbett & Wilson, 1991; Firestone, Mayrowetz, & Fairman, 19 98; Grant, in press; Noble & Smith, 1994; Smith, 1991). The research that is ava ilable presents a mixed picture at best. Those advocates of tests as a vehicle for d riving educational change tend to cite general positive effects rather than specifics. Som e (Feltovich et al., 1993; Popham, 1998; Shanker, 1995) simply argue that good tests w ill inevitably drive good instruction. Lacking any more specificity, Popham, Cruse, Rankin Sandifer, and Williams (1985) claim that tests measure important learning, and th at good tests results equal good education. Systemic reformers (Fuhrman, 1993; Smith & O'Day, 1991) advocate for testing as part of an overall strategy aimed at fun damental school change. Others (English, 1980; Glatthorn, 1987; Heubert & Hauser, 1999) argue that because standardized tests are a reality in most school dis tricts, they should be used as a fundamental part of curriculum planning. Critics of standardized testing are more di rect in their assessment of the impact of
5 of 28testing on teaching. Madaus (1988) claims, among ot her things, that teachers will teach to the test, that they will adjust their instructio n to follow the form of the questions asked (e.g., multiple-choice, essay), and that tests tran sfer control over the curriculum to whomever controls the test (Note 3). Claims by LeMa hieu (1984) and Koretz (1995) are more tentative, but they too conclude that teachers may tailor their curricula to the content covered on the test. Recent empirical work supports some of these claims. Smith (1991) argues that many teachers respond overtly to test pressures and she offers a typology of eight orientations toward test preparat ion: ordinary curriculum with no special preparation, teaching test-taking skills, e xhortation, teaching content known to be covered by the test, teaching to the test in for mat and content, stress inoculation, practicing test or parallel test items, and cheatin g. Firestone, Mayrowetz, and Fairman (1998) assert that testing programs in Maine and Ma ryland seem to influence teachers' content decisions, although they conclude that such influences are weaker than expected. Corbett and Wilson (1991) argue that testing, espec ially minimum-competency testing, has a pernicious effect on teachers in that it caus es them to narrow their sense of educational purposes and to focus on activities des igned to raise test scores whether or not they think those activities are good for studen ts. They conclude that squeezing teachers in this fashion encourages them to rebel a gainst reform measures good and bad. "Statewide testing programs do control activity at the local level, but the subsequent activity is not reform" (p. 1). Other researchers are less sure that a dire ct relationship exists between standardized testing and teachers' classroom practices. Freeman, Kuhs, Porter, Knappen, Floden, Schmidt, & Schwille (1980), Kellaghan, Madaus, and Airasian (1982), and Salmon-Cox (1981) found little direct impact of standardized t esting on teachers' daily instruction. Firestone, Mayrowetz, and Fairman (1998) claim that while tests may have influenced teachers' decisions about what to teach, there was virtually no influence on their decisions about how to teach. In a crosscase comp arison of two high school teachers' civil rights units (Grant, in press), I found littl e direct influence of testing on either teacher's content or pedagogical decision-making. This brief review suggests two points. Firs t, we need to know more about the relationship between teachers and tests. While the impact of tests on students has been much explored, research that inquires into if and h ow teachers are influenced by standardized tests is lacking. Second, that researc h around teachers and tests fails to show a clear or consistent pattern of influence. Te sts matter, but how and to what extent is unclear.State-Level Curriculum and Assessment in New York S tate State-level influence over curriculum and a ssessment is a well-established tradition in New York State. The Regents test has been admini stered continually for over 100 years. These tests are administered in all academic subjects and are tied to school courses. For example, in social studies, students t ake the Global Studies test at the end of a two-year Global Studies course sequence in nin th and tenth grades; eleventh graders take the U. S. History and Government test after co mpleting a course of the same name. Elementary and middle school teachers also follow a state curriculum in all school subjects and students take state-developed tests.Recent State-Level Curriculum Changes As is the case in most states, educational reform has been steady work since the
6 of 281980s. Begun during the tenure of former Commission er of Education, Thomas Sobol, state-level focus on and activity around school cur riculum hit full stride in the mid-1990s under current Commissioner Richard Mills. Since 1994, working groups of state policym akers, teachers, and administrators have produced new curriculum and learning standards and scope and sequences for all school subjects. Social studies teachers, for examp le, may now consult the Learning Standards for Social Studies (New York State Education Department, 1996) and th e Resource Guide for Social Studies (New York State Education Department, 1998). Compared with the previous round of curricular revi sions in the mid-to-late 1980s, the changes represented in these documents vary from vi rtually no changes in the K-5 grades curricula, which follow an expanding horizon s model, in the seventh and eighth grade U.S. and New York State history, or in the tw elfth grade Participation in Government and Economics courses. Modest changes ar e evident in other curricula, such as the emphasis on geography in the eleventh g rade U.S. history and government course. Major changes seem localized at sixth grade where the course of study expanded from Western and Eastern Europe and the Mi ddle East to the entire Eastern hemisphere, and at ninth and tenth grades, where th e emphasis has changed from a cultural approach as represented in Global Studies to a chronological study as expressed as Global History and Geography.Recent State-Level Assessment Changes The state-level testing program is also cha nging. Although the scope of the changes varies (Note 4) the net effect appears to be a ge neral ratcheting up of the stakes for both teacher and students. State tests of language arts, mathematics, and science have undergone radical transformations which include reducing the number o f multiple-choice items and increasing the number and range of performance task s. For example, new science tests call for students to actually perform experiments. By contrast, the social studies assessments will apparently change little: Multiple -choice questions will still dominate the tests, accounting for 55% of a student's score (Note 5). The major change seems to be in the writing portion of the exam. Unlike many minimum competency tests, New York students have always had to answer essay quest ions on state exams. The new tests are different primarily in the fact that a) student s will no longer have a range of essay prompts to choose from, and b) a new kind of essay question, a document-based question (DBQ), is being introduced on each of the fifth, eighth, tenth, and eleventh grade tests. A DBQ asks students to write an essay synthesizing a number of primary source documents (e.g., short quotes from governmen t documents and famous individuals, political cartoons, poems, charts and graphs) (Note 6). Plans call for students to answer a main idea-type question about each of the documents before writing their essay. High school students will also write a second, "thematic" essay based on a single prompt (Note 7). The inclusion of the DBQ is the primary change in the structure of the social studies exams. One might argue that s uch a question represents a major shift away from traditional testing, but given the scope of the test (and the fact that students can easily pass the test without a single DBQ point), adding a DBQ could be read as a minor revision, or an instance of what Ty ack and Cuban (1995) call "tinkering toward utopia." Three other changes seem more dramatic. One is that the new fifth and eighth grade tests will produce individual student scores. Tests at those levels, termed "Program Evaluation Tests," have aimed at helping teachers u nderstand the effectiveness of their
7 of 28content and pedagogical decisions (Note 8). The shi ft of emphasis to individual students is apparently intended to raise the stakes of these tests and tie them more directly to the high school Regents exams. The function of the Rege nts test is also being fundamentally changed. In the past, passing Regents tests in all academic subjects meant that a student earned a Regents diploma. Students could opt to tak e the less rigorous Regents Competency Exam (RCT) and earn a local diploma. Nin th graders beginning in 2001 will no longer have these options. The RCT will no longer be administered, and all students will have to pass five Regents examination s (English, mathematics, global history, U.S. history, and science) in order to gra duate. Given these changes, state-level tests are no less high-stakes for teachers than they are for students. Since the mid-1990s, state policy makers have introduced a number of curriculum reforms, such as new state standards for social studies, yet it is a concern about the state tests which surfaces most regularly in teachers' talk (Grant, 1997a). This makes sense for two reasons. First, the curriculum documents produced thus far offer teachers little assistance in making concrete instr uctional decisions (Grant, 1997b). Second, the messages teachers receive often promote the view that tests are intended to drive change (Grant, 1996). For example, during ses sions devoted to new state social studies standards, one representative from the New York State Education Department (NYSED) said that new tests will "help grow change in the system." During another session, a different SED representative said, "New assessments will represent a change in instruction....Kids won't perform well until (te achers') instruction reflects this." And at yet a third meeting, NYSED Commissioner Richard Mil ls added, "Instruction won't change until the tests change." The message that te sts matter was echoed during local school and district meetings. A suburban district s ocial studies supervisor, for example, told teachers that "change in content will come if we change the tests." An urban district supervisor observed, "If we change the assessments, we'll change instruction" (p. 271). One might question the focus of test influence--ins truction, curriculum, or the "system" in general--but it is hard to miss the larger point : tests matter.The Prospects and Problems of State-Level Testing I n New York State The tendency of advocates and critics to ca st standardized testing in black and white images is not supported here. My analysis sug gests that teachers see the new NYS tests as a mixed bag. The prospects of tests which more closely mirror and support thoughtful instruction and closer collaboration wit h colleagues are mitigated by the problems of, among other things, uncertainty about the rationale for and consequences of the new tests and the unevenness of the opportuniti es to learn about and respond to changes in the tests. In short, teachers across gra de levels and subject matters express an uneasy combination of hope and fear, anticipation a nd dread. I explore those poles by looking at teachers' perceptions of the new tests i n terms of their nature and substance, the professional development opportunities availabl e, and the rationales and consequences.The Nature and Substance of the New NYS Tests The NYSED is phasing in the new state tests over a period of four years, beginning with the English language arts tests at grade 4 in January, 1999. Consequently most of the teachers interviewed have not seen final versio ns of the tests they will administer. All have, however, received preliminary materials f rom state, district, and professional
8 of 28organization sources and so most assume that they h ave a fair sense of what the new exams will be like. Most believe the tests will be an improvement over past assessments, but questions about the nature and substance arise. Both elementary and secondary teachers expr essed at least modest support for the general direction taken in the new tests. A middle school science teacher suggested simply that the NYSED was "changing what assessment means." An elementary school teacher was more specific. "I think there was a lot of change going on and then they changed the assessment," she said, "I remember givi ng that CTBS (a basic skills test) and teaching a literature-based program, and we wer e all complaining that it wasn't reflective [of our teaching]." Another elementary s chool teacher was more specific: "The new assessments test the same way we teach reading, and where we want kids to be in math." Social studies teachers approved of the mov e to include primary sources within the DBQ. A high school teacher cited the real world rel evance of questions which employ political cartoons. "You give them a cartoon and yo u say, 'Interpret this cartoon,'" she said, "That's interpretation, you know? If you open a paper and you look at a picture in the newspaper and you go, 'What's that mean?' That' s something you would do in real life." A middle school teacher noted she now uses D BQ kinds of questions as a regular part of her instruction: I was working on a social studies test today for gr ade seven where they have to look at a document and think about some stuff li ke, what was the theme about the Revolutionary war, and they've got to wri te notes based on the picture. And it looks-the test is a lesson. It's a lesson in analyzing documents and taking notes from the document so you 're not looking to see if they're right or wrong. You're looking to see ca n they look and think about what's on there. This teacher and most others praised state efforts to bring standardized assessments into closer alignment with the kind of ambitious instruc tion they believe is important, such as analyzing primary sources and understanding that su ch texts can be interpreted in multiple ways. Social studies teachers worry about the continued strong emphasis on multiple-choice questions, but in questions like th e DBQ, they see potential for pushing their students toward richer understandings. But not all teachers held this view. Some f ocused on the continuing heavy presence of generally low-level multiple-choice questions, a rguing that the test has changed little overall. As one middle school teacher explained: From my perspective, the social studies assessment doesn't seem like it's a change at all. Seems like it's kind of repackaged, kind of dressed up a little differently, but not really different and to me, th ere is something broken in [teachers' instruction] and we need to fix it. This new assessment to me isn't fixing it. One might argue about whether teachers' practices a re "broken," but the sentiment that some state tests, like social studies, seem less ch anged than others emerged throughout the focus group sessions. The English language arts and science tests, in particular, were cited as moving away from a heavy reliance on objec tive-style questions and toward questions with more real world and practical applic ations. For example, the English language arts tests asks students to write a range of pieces including technical, literary, and literary analysis essays. The science tests inc lude performance tasks which ask
9 of 28students, for example, to set up a lab experiment. Teachers in these areas had questions about the nature of their respective exams, but the re was a general sense that these exams push in more ambitious directions than the so cial studies tests do. Social studies teachers see the prospective new state assessments as a mix of old and new. While most applaud the presence of primary sources and questions like the DBQ that ask students to analyze and synthesize inf ormation, they wonder if that emphasis won't be undercut by the continuing heavy weight of the multiple-choice section and questions which teachers generally perc eive of as asking for low-level knowledge.Opportunities to Learn About the New State Tests New state tests, like many other educationa l policies, can be viewed as an occasion to learn about the craft of teaching (Cohen & Barne s, 1993; Grant, in press). The focus group teachers nodded in agreement when participant s raised questions such as, "Do I have the skills that I need?" and made assertions s uch as, "We have not been taught the way we're being asked to teach.... And I think that 's really difficult without a lot of staff development to get people to think differently and to teach differently." If the need for professional development wa s widely expressed, the teachers' experiences suggested that they may not be getting all that they want. Studies of professional development activities suggest that wh at session leaders think they are "teaching" and what participating teachers think th ey are "learning" during professional development activities can vary dramatically (Darli ng-Hammond & McLaughlin, 1996; Grant, 1997a; Smylie, 1995). Consequently understan ding what kinds of professional development opportunities teachers had available to them and what sense they made of those opportunities was a major element of the focu s group interviews. Three patterns emerged from analysis of the interview transcripts. One was that all teachers seemed to have had access to a wide range of professional development opportunities both around the new curriculum standa rds and around the new tests. A second pattern was that they found those opportunit ies of uncertain value. Teachers reported that the state, and occasionally district, activities often resulted in incomplete and mixed messages. The frustration many teachers e xpressed about the more formal professional development opportunities was mitigate d, however, by their sense that working more directly with colleagues was a more pr ofitable use of their time. The third pattern, reform by "rumor," began to emerge in the first year of interviews, but was full-blown by the second year. Despite the wide arr ay of professional development opportunities, the teachers clearly felt that there was still much indecision about how tests would ultimately look, how they would be scor ed, and the like. In a context of increasing pressure to respond, but little solid in formation, several teachers reported the sense that rumors were driving much of their respon ses. The professional development opportunities availabl e. Asked to describe the professional development opportunities available to them, the teachers constructed a long and varied list. Some NYSED-led sessions occur red in several venues (e.g., stand-alone sessions, part of district-level in-ser vices, sessions during professional organization conferences) and focused alternately o n the new tests alone or on how the tests reflected the new state curriculum standards. Representatives from local Board of Cooperative Extension Services (BOCES) programs als o led professional development activities as stand-alone and district sessions. So me district-level sessions featured state and BOCES representatives, but others utilized the talents of district personnel, while still others brought in local and national experts. School-level professional development
10 of 28opportunities were also varied in that some called all teachers together, while others asked teachers to meet in grade or department-level activities. The focus group teachers also mentioned state teachers' union sessions, coll ege and university course work, professional literature, informal networks, and col leagues as additional sources of information on tests and testing. The uncertain value of professional development. Of these many sources, teachers were most critical of the state-led sessio ns. Some felt that cuts in the NYSED have left the agency woefully understaffed. Most ot hers, especially the high school teachers, were less generous. An English teacher sa id, "I'm not going to break a sweat trying to reformulate what I do when their people ( NYSED) don't know what they're doing." A social studies teacher was more blunt: "D o they have a clue as to what's going on?" District-level sessions received more mixed reviews. A high school mathematics teacher praised her district's efforts to develop p rofessional development activities that would meet teachers' perceived needs: My district is real supportive. If I say to them we need an inservice on blah, they will say we'll do it. They're wonderful that w ay. It's very teacher driven. Our school district is wonderful as far as them inv olving teachers and listening to the teachers and valuing what the teac hers say. This comment stood largely alone, however, as most other teachers suggested that district-led professional development was lacking i n usefulness. A high school social studies teacher noted: We've had two district wide superintendent's confer ence days and we've talked about [the tests] and gone over some things, but not into the detail that needs to be done to get a good feel for the ty pes of questions and changes. I think in our building many people would still be hard pressed to give an accurate reflection of what the assessment is all about. A middle school science teacher attended a district -sponsored inservice led by a district teacher. She reported that while the session could have been valuable, she left frustrated because the teacher who led the session came from a magnet school where resources are plentiful, whereas she teaches in a resource-starve d neighborhood school. Not all the blame for weak district-sponsored professional deve lopment was laid at the feet of the leaders, however. A secondary social studies teache r panned the district-level sessions she attended, but she assigned much of that respons ibility to her colleagues: We went to the district-wide [in-services]. They (t he inservice leaders) always tried to be very positive, but the overwhelm ing number of teachers who are so negative about this assessment always wi ns out. It basically becomes a complaining session and you really aren't focusing on what the whole meeting was about anyway. The focus group teachers reported that school-, gra de-, and/or department-level professional development activities were generally more useful than state or district efforts. An elementary school teacher, for example, praised the work her grade-level colleagues were doing: We have grade-level meetings. They're very positive you know, even
11 of 28though we all don't want to test, we all feel like we shouldn't have to do it. They're (her colleagues) always very positive, alwa ys very friendly approaching it. Every time we go to a grade level m eeting, [the team leader] always is handing us stacks and stacks of informati on materials. Things that we might need or might be able to use to help the k ids get ready, whether it's for the science or the math or the English [te sts]. There's always something positive going on. A high school mathematics teacher explained that not only has the amount of conversation increased in her department, but that it is becoming increasingly acceptable to say, "I don't know how to do this." She went on to describe how her colleagues, both veteran and novice, were creating a new ethic where by the traditional norms of isolation and "doing your own thing" were fading. Not all teachers are similarly situated, ho wever, and more than any other group, the high school social studies teachers present describ ed their departmental interactions as less than optimal. Several nodded in agreement when an untenured teacher portrayed her colleagues as being obsessed with talk about "how t o beat the test, or change the test, or fight the state, or fix the state or...how is the a dministration wrong, how are we right." Potentially useful discussions of teaching, learnin g, and assessment, she explained, get lost in the mix. If teachers found formal state, district, a nd school-level professional development of uncertain value, all reported instances where in formal networks and relationships had proven valuable. A high school social studies teach er said that, while she appreciated some elements of her district staff development day s, "it is a lot easier to bounce off the ideas with somebody. And I just wrote [a DBQ] a few weeks ago with a colleague. We have now the same planning period so that worked ou t." A high school teacher reported that she and her colleagues have met informally aft er school to consider assessment issues. "There were a handful of us that got togeth er after school on a voluntary basis," she said, ".... It makes my life a lot easier when I talk to other English teachers." In addition to these unstructured activities, several elementary school and high school mathematics teachers described informal networks of educators who meet regularly to discuss a range of issues, including those related to testing. A mathematics teacher described the benefits she has appreciated from her involvement: We have each other (she laughs). We have a network through (a local state university).where there have to be what-about 70 te achers, maybe 100 maybe that-we have meetings four times a year, and so now I don't feel isolated anymore. I mean I can always call [a colle ague in a neighboring district]. I have friends [in another district]. Fr iends just about anywhere. I know what's going on at what school and I can pool resources, and so that helps a lot. The power of such informal relationships is apparent: These teachers sense that they are working with peers who hold similar goals and concerns, who are willing to share ideas and practices, and who offer a sense of belonging. Such relationships, then, have an immediacy and a specificity that seems miss ing from the more formal professional development opportunities teachers typ ically experience. That these teachers have sought out and participated in these relationships is admirable; that they have felt compelled to do so in order to meet their needs is ironic, however, given the seeming wealth of structured opportunities.
12 of 28 Reform by rumor Having informal sources of information and suppor t may help teachers navigate some of the challenges the new st ate tests posed, but they do little to help teachers with the problems of mixed messages a nd unanswered questions. In fact, the more sources of information teachers encounter, the greater the incidence of reform by rumor. Common across teachers of all grade levels and subject matters was a frustration with incomplete and conflicting information about t he new tests. An elementary school teacher noted, "If we just had more information and if we knew what was expected of us and how to do it, possibly.we could do what was exp ected of us." A high school mathematics teacher added: If they're (NYSED) going to give us information, th ey have to give it more structured backing. Not this haphazard changing the rules daily.... Our math department head has said [at an in-service led by a n NYSED representative], "Tell us what you want. We will do it. We will change the way we teach.... But you can't keep changing the me ssages you're giving us." To be sure, state leaders seem to recognize that th ey are sending multiple and, at times, confusing messages. A high school mathematics teach er reported the following experience during a state-sponsored in-service: When we go to state meetings, (the NYSED representa tive) who's in the math ed department always prefaces his remarks with "What I'm going to tell you is true at May 13th at 4 whatever. It's tr ue right now. When I go back to my office, it might not be true." And we ge t to go to a lot of state meetings and everything and find out what's going o n. And we always find out the latest stuff, but then it changes. As this quote suggests, teachers do not necessary b lame the state education representatives, but they are frustrated with the u ncertainty of the situation. A high school social studies teacher's experience summed u p some of the anxiety mixed and multiple messages can induce: I don't know if this geography thing (i.e., that th e state curriculum and test for tenth grade were changed from Global Studies to Global History and Geography) is true or not. But somebody in my depar tment had been in the state conference the week before and said, "I didn' t hear any of this." And then we started frantically calling-I think we call ed the (local state university) Social Studies department, and they wer e calling all over to find if this was true. And I think the final verdict was that, "yes (geography has been added), but geography the way we've always tau ght it, so don't be nervous. They (NYSED) are not asking to name which direction the Danube River flows or anything like that." But, I don't kn ow. It's crazy. This teacher went on to remark, "I see it as just l ots of rumors. It's like every other day we're coming in, 'Did you hear they're cutting out the constructed response? Oh, now the new course is Global History and Geography?'" A cynical interpretation of the above is th at teachers are merely pawns in a game that is being transacted all around them. This view asserts that while changing teachers' practices is the target, teachers' ideas and voices are largely ignored as those above
13 of 28them-state and district-level actors-do the real wo rk of policy change. Teachers, through their professional development opportunities, may l isten in. But as listeners rather than as full participants, they hear only bits and piece s, and rumors rule the day. A more generous interpretation has two elem ents. One is that reforming education is simply hard work, especially when done in midstr eam, or what a policy maker in another state termed, "rebuilding the airplane whil e you're flying it" (Lusi, 1997, p. 91). The second element is that, given the sheer number of teachers and the wide range of circumstances in which they work, policy makers fac e a daunting task in attempting to change pedagogical practices. Whether they should t ry to or not, the parameters of the NYSED operation are intimidating: thousands of teac hers, in thousands of schools, in close to 700 districts, and an agency with little m ore than a handful of employees. Clearly, then, NYSED must rely on the efforts of pr oxies-BOCES educators, professional organizations, district and school-lev el leaders, college and university academics-who may or may not understand and/or supp ort the state agenda. In such a situation, the potential grows for mixed and confus ing messages, and for reform by rumor.The Rationales for and the Consequences of the New NYS Tests The notion of "reform by rumor" functioned as a proxy for a number of comments where focus group teachers talked about feeling lef t out of the conversation about changing state assessments. Teachers across grade l evels and school subjects expressed frustration that, while they are the professionals on whom the tests will have the most impact, their voices are not well reflected in impo rtant discussions about the nature, import, and design of new state tests. As one teach er said, "I really fear that unless there's open communication...this whole thing would be just kind of a charade." Another added, "I just feel that I've been talked at." These teachers remain uncertain about the r ationales for and the consequences of the state assessments, but seek to question rather than condemn. Most said they have attended meetings designed to inform them about the tests, but none said they were satisfied: Their questions either went unaddressed or, if they were addressed, the information they received did not always jive with information circulated previously. While numerous questions arose during the focus gro up interviews, two dominated: questions about the rationales for changing the ass essments and questions about the intended and unintended consequences of the tests. Questioning the rationales for the tests. Whether the NYSED hopes to induce changes in teachers' curriculum decisions, their in structional practices, or both has been unclear for some time (Grant, 1997a). The focus gro up teachers echoed this confusion. They also discussed their uncertainty about whether the state's intention was to change their behavior or the students'. As a middle school social studies teacher said, "Are they (NYSED) doing this to better students' education, o r are they doing it so they can say, 'Look, we changed something.'" On the question of whose behavior NYSED is targeting, teachers expressed considerable frustration. For instance, an elementa ry teacher asked, "Who is it assessing? Is it really assessing the students? Or is it assessing the teachers?" Another elementary teacher echoed this point: "What is the purpose of the state exams? Is it actually to assess the students or to push the teac hers in a direction?" A secondary social studies teacher spoke directly to the issue of whos e life is changing the most as a result of the new state tests:
14 of 28I think it's ironic that the state came out with al l of these decisions in order to improve student learning and to make students be tter students and.I feel like I am doing so much work this year. When I do e ssays, I try to fix things and give them lots of responses and they just-I fee l like I'm doing more work than the kids sometimes.... The last couple we eks it's like "I'm not taking this test! I took this test!" This is you. N ot me. But it seems like the teachers are on the chopping block. And it's just i ronic that it's no longer the student anymore. And it's the kids who are taking t he test. And it seems like the kids are almost less and less responsible.... The last part of the quote above suggests t hat the issue of whether teachers or students are targeted is important, in part, becaus e teachers are unsure where the blame is going to come down should test scores not rise. Many suspect, however, that teachers will take the brunt of the criticism. A high school mathematics teacher said, "They're (local administrators) are going to be pointing the ir finger if your kids don't do well. They're going to be pointing their finger at those teachers and that's unfortunate because they're (the teachers) going to be a scapegoat beca use of it." A secondary English teacher talked about the unfairness of holding the teachers whose students are taking the tests entirely responsible for the outcomes: I think that whole culture needs to change because you are not the sole responsible party for that student's abilities.... If someone did a lousy job last year, then you're getting a group of students without the proper foundation. And is there going to be some kind of m echanism that will address that if you realize that the child did not get proper foundation?...There's no way I solely am responsibl e for that child's [test scores]. I've had students who are functioning very very low and you're asking me to...bring that child further along. Is t hat child going to pass that test? No. So you're going to come to me and say, "W ell, only 55% of your students passed this test. You're lousy!" I'm going to say, "Well, what did you give me?" This quote raises a number of thorny issues, not th e least of which is a seeming deficit view of children. This view implies that students c ome to a teacher with a set of deficiencies, resulting from poor parenting, poor s chooling, and the like, which the teacher must then "correct." The problems with this view are several, but in this case, they serve to amplify the dilemma this teacher face s: She feels the twin burdens of preparing students to take the exam and of being he ld accountable for their performance. Although it seems unfair to make the child the pawn this teacher rightly points out that she alone can not be responsible for test scores. Teacher frustration was also apparent aroun d the question of whether NYSED's intent was to change curriculum, instruction, or bo th. The focus group teachers assumed the tests were meant to induce changes, but they we re unsure what sort of change was expected. A secondary social studies teacher saw the state's aim as primarily directed toward curriculum: But it looks like -the more I hear about it it's as if the state through its tests is controlling what gets taught in the classroom. B y saying that the test is going to be done this way, all of a sudden it's goi ng in and saying well you
15 of 28can't teach this, this, and this when you want to. You have to teach this. You have to teach this. An elementary teacher, by contrast, suspect ed that the state's intention is to influence teachers' instructional practices: Is this a way of making teachers look at their prac tice and alter their teaching techniques because they see a certain topi c being covered on an exam and so they'll say, "Oh, I didn't do that so w ell that time. I guess I have to spend more time on that next year." So if you se e the focus on the exams, then you've got to go back and make sure that you i nclude that type of instruction the next year. And so I think-are the t ests pushing-is the state using the test to push teachers in a certain direct ion with their instruction? While most of the focus groups sensed that the state tests were being used to leverage change of one sort or another, not all did A high school English teacher reported that she had been told, "We've been doing this all along. That this is no big deal...all we have to do is get kids accustomed to the format [of the test]." A secondary science teacher added to this notion, by reciting a familiar teacher expression, that is, "this too shall pass." "In our science department," he said, "they feel because science is the last assessment [to be introduced] that this is all going to blow over." The notion that whatever NYSED introduces is likely to fade in impo rtance over time was not the dominant view among the focus group teachers. But i ts expression should warn state-level reformers that whatever leverage they b elieve tests hold for changing instruction and/or curriculum may be illusory. This is not because teachers do not sense that problems exist: None of the focus group teache rs was willing to suggest that all is right with public education. But several supported the following sentiments of an elementary school teacher who questioned the relian ce on tests as a lever of real instructional change: I understand that certainly there are places in Ame rican education that are in dire need of shaping up somehow....It (the test) ju st seems to me a misdirection of resources. We're spending how much-thousands of dollars on training, on writing these tests or whatever the y're doing to when the real issue is what's happening in the classroom. What ki nd of preparation are teachers getting? What kind of preparation are they getting before they even get a classroom? What kind of thinking is going on here? And are those questions even being asked? Or were they ever asked before this happened? It was just suddenly that we had this massive asses sment. And I don't remember any sort of input from teachers. I don't r emember any state education people coming to us and saying, "What do you think?" Or, "What's going on in your classroom?" It was just th is kind of mandated attempt to reform. And maybe it will work. I mean, I don't know whether it will work or not. But it seems to me there's so muc h more that could be done that hasn't been attempted in terms of helping teachers. To be fair, NYSED officials and the state Board of Regents have proposed a range of reforms that push changes in curriculum and in teac her education. The primacy of the state testing program, however, weighs heavily. The focus group teachers are not opposed to improving teaching and learning, but the y are uncertain about the rationale for standardized tests as a vehicle.
16 of 28 Predicting the consequences of the new tests The idea that the new tests may yield no real consequences for teachers' practices was one of several predictions the focus group teachers made. Most of those predicted consequences were negative, but not all. For example, several teachers in the first yea r focus groups expressed the hope that the tests would mean greater collaboration with the ir colleagues. A high school English teacher summed up the feeling: "If there were more opportunities to get more people together, that would help." While it was far from u nanimous, a number of the year two teachers reported that, in fact, they had found the ir peers receptive to and interested in working together. The overwhelming sentiment, however, was th at the new tests could produce undesirable effects. Those effects grouped loosely around issues of pedagogy, students, and teachers. Two related consequences of tests for pedag ogy arose. One is that, rather than promote more ambitious teaching and learning, the s tate tests may actually push more reductive forms of teaching and learning. The most common expression was that teachers felt increased pressure to tailor one's te aching to the test parameters. As a secondary social studies teacher noted, "You've got people in high places just saying 'teach to the test.'" A middle school English teach er complained that he felt pressure to "teach them (students) test terminology when I coul d be teaching them other things." This teacher went on to describe the kind of suppor t his district provides as little more than practice exercises. "The only thing I've gotte n from my district," he said, "is lots of practices. Every week there's, 'Thank so and so for giving this practice material. Here's another listening practice that you may want to use .' I could have spent my whole year doing practices." The sense that teachers feel pressed to ado pt direct teaching approaches as a means of bolstering short-term test performance was in di rect competition with the sentiments expressed earlier that the new state tests could be viewed as supportive of more ambitious instruction. During the interviews, howev er, no teacher commented on this seeming contradiction. One explanation is that they were simply unaware of its emergence. A more interesting possibility is that t hese teachers can read multiple messages in the tests. Take social studies as an ex ample. Teachers thinking about the multiple-choice questions could reasonably assume t hat a more traditional, direct instruction approach was being encouraged. If those same teachers were thinking instead about the DBQ questions, it seems equally reasonabl e to assume that richer forms of pedagogy were intended. This ambivalence, which has surfaced in a number of places already, underscores the difficulty in understandin g teachers' perceptions of state tests and it suggests that their classroom responses may be more complex and textured than reformers may want or expect. A second potentially negative consequence o f the new tests was an increased emphasis on remediation as a way to deal with low t est scores. The teachers, especially those in the second year interviews, described a wi de array of remedial approaches taken in their schools. Those approaches included additio nal classes designed for students presumably at risk of failing, summer and Saturday test review courses, hiring additional teachers and aides to staff learning labs where stu dents could either come voluntarily or by teacher assignment, and reassigning teachers to classes of students based on their perceived ability to help those students pass the e xam. The teachers offering these examples genera lly seemed supportive of them. The seeming contradiction that ratcheting up remedial e fforts would occur at the same time teachers were being pushed to change their pedagogy went unremarked upon. Again, however, this contradiction may be less apparent th an one might suspect. Empirical
17 of 28evidence is surprisingly thin on the question of wh ich instructional approaches lead directly to high test scores (Cohen & Barnes, 1993; Grant, in press). Consequently, a reasonable response to a new testing situation migh t be both to make changes in "regular" classes and to begin planning for remedia l instruction at the same time. The real danger, however, is that these rem edial opportunities will become little more than drill sessions, a point that was recogniz ed by several teachers. For example, a high school mathematics teacher observed: If the students do not pass, they're going to be re medied with questions that will make them pass. So eventually every student wi ll pass. Doesn't matter the categories, they're going to do component retes ting, so if the student doesn't do well in these three areas, they'll be gr illed in those three areas with a bank of questions, and then the student will have another test from the bank that he was drilled in. So eventually they 'll get it. Such an approach may work for low-level skills, but is of dubious use in areas like social studies where conceptual knowledge is central. As V anSledright & Brophy (1992) observed, "naive but imaginative accounts persisted in some children even after direct instruction designed to change them" (p. 854). With out any definitive research supporting one means of improving test performance over another, drill and practice remediation is as likely to flourish as any other a pproach. A second area of negative consequences anti cipated by the focus group teachers concerned students. An elementary teacher worried g enerally that the net effect of a high profile, highstakes testing program would be a "n ation of test-takers": Something that I've been thinking about more is the effect this has on the children, on the student. What kind of learners is this going to shape? Are we producing a nation of test-takers, and if so, ar e those test-taking techniques or skills what we need to produce life l ong learners that we talked about before? Other teachers expressed more focused concern about the anticipated consequences for urban students. Wiles (1996) argues that test perfo rmance is clearly distributed along socioeconomic lines with upscale, white suburban children consistently outscoring their urban and minority peers. The focus group tea chers, both urbanand suburban-based, recognized the inherent threat that high-stakes testing poses for some children. An elementary school teacher said, "I'm v ery concerned about some of the larger populations in the bigger urban areas. I don 't understand how this is going to positively affect these kids." A high school teache r, commenting on the anticipated testing of special education students, asked, "How do we accommodate the non-standard kids on a standardized test?" No teachers thought their students' scores on the new tests would improve immediately over past test scores. A couple of teac hers did express, however, the hope that their students' scores would increase over tim e. A middle school English teacher said, "I think, naive though it may be, that our ki ds are going to do better ultimately on these exams. Maybe not this year, but ultimately." This hopefulness stood in stark contrast wi th the prevailing view that teachers anticipated problems for their students. Underlying both these sentiments is a harsh truth: These teachers simply do not know how their students will perform on the new tests. Given the general tendency for a correlation between test scores and students' social capital, it is difficult to understand why s uburban teachers would be worried. And
18 of 28yet, analysis of the relative concern expressed by suburban vs. urban teachers suggested that suburban teachers and administrators may be ev en more concerned about potentially low scores than their urban peers. One proxy for th is finding is the observation that the overwhelming number of remedial efforts planned are being developed in suburban schools. As noted above, no teacher feels s/he has a n inside track on what approaches will insure high scores. Left to follow one's hunches, i t is no particular surprise to find concern among all teachers, both suburban and urban But what explains the fact that suburban teachers seem to be more concerned about t heir students' performance than their urban peers? Part of an explanation must cons ider the notion that not all suburban districts are created equal. The suburban teachers in focus group teachers represented first-, second-, and third-ring suburbs. First-ring suburbs tend to include a range of working to middle class students. Second-ring subur bs are more upscale; most students come from middle to upper-middle class homes. Final ly, the third-ring suburbs are rural areas that recently have attracted a large number o f middle and high SES families. With the exception of one or two urban magnet schools, i t is the schools in the secondand third-ring suburbs that consistently rank in the to p quartile according to a highly publicized local business magazine. Top quartile sp ots on this list have real consequences for real estate values, bragging right s, and the like, and so the scramble to move up can be intense. New tests, then, represent a potential threat to schools' past standings. School people in high performing schools want to maintain their positions; educators in middle and low performing schools hope to at least avoid dropping further. The competition for high test scores plays out as a third set of consequences. Here, the focus is on the pressure and uncertainty teache rs feel as they decide if and how to modify their teaching based on their perceptions of the state test. A couple of these pressures have already been described. One is the f eeling of uncertainty teachers have about which approaches will ensure higher scores. A second pressure surfaces as teachers report being made to feel entirely respons ible for their students' results. Putting the point on this feeling is a secondary social stu dies teacher: Just this week I was called down to the office and we were comparing some of the Business First statistics that were out just recently....So accor ding to our administration [if we get low test scores]...pe ople come out to vote and decide they don't want to vote on the budget, there fore the whole community goes down. So, I left the office thinking the weight of this town...is on my shoulders. Whether or not, you know my kids pass. And we had like a 70% last year and we're expected to have at least a 90 if not higher. So, in terms of administration, testing is a pretty big deal. Not all principals apply pressure so directly, but many apparently do. This is more likely to happen in high schools than elementary schools, however. According to several of the focus group elementary school teachers, their princ ipals are more likely to talk about test scores as part of a bigger picture of how students are progressing. These teachers do not necessarily feel any less pressure than their high school peers, but one source of pressure, the school administrator, seems to be les s of a factor. The new elementary school exams are more hi gh-stakes than they used to be; recall that now individual student scores will be reported rather than group scores. The stakes are even higher in the high schools, however, as pa ssing the Regents exams will be necessary in order to graduate. Consequently, it is not hard to understand why high school administrators might be more likely than the ir elementary peers to put pressure
19 of 28on their teachers. Whether that tactic will pay off ultimately or not is hard to predict. But one manifestation of that pressure is to cause teac hers to consider issues that they probably have not had to think about in the past. O ne particularly compelling story came from a high school social studies teacher who said she now wonders about each new student who comes into her classes: I never--it never crossed my mind before that a cer tain kid was going to lower my passing rate or not, and I actually starte d thinking about that this year. And I was so ashamed of myself about that. An d one of the girls I had transferred from a general track. She stayed in my class. I didn't want to just dump her. But she can now take the RCT at the end o f the year. But I had a girl a couple years ago who transferred from anothe r state. She never had Global 9. And I was just happy to work with her and she was going to try it. And if you go to look at an individual kid and say they're not going to do it, it's horrible to think that--to individualize it li ke that. Because I guess every couple kids knocks you down a little bit. And our-I know that our department chairs had our results individualized an d our principal keeps coming into meetings saying, "How can we raise this up? How can we do this better?" This teacher concluded her story with a nervous lau gh, saying, "But I'm glad I have tenure, right?" Yet, having tenure seems little con solation for this thoughtful and dedicated teacher now confronted with the dilemma o f wanting to work with all students, but recognizing that doing so may cause h er teaching to be called into question should her students' scores not measure up. Not all the consequences described were neg ative, however. Several teachers cited greater collaboration with their peers as a key ben efit of the new tests. Elementary teachers and high school mathematics and English te achers were most vocal on this point. "I think we have so much to learn from each other," one elementary teacher said. Another echoed this point, commenting, "We're reall y trying to deal with this [new tests] and trying to work as a faculty to help each other. A high school English teacher noted that information is vital and that colleagues are a n important source, "What's most important to me is being able to communicate with o ther people so I can get some information." A high school mathematics teacher con curred, but pointed out that that the new exams were forcing teachers to rely on each oth er: I think the nature of the testing--it certainly set s the situation up for teachers to talk. Because the types of questions that happen to be asked. They don't have the stockpile of old Regents questions. So [te achers say] "I came up with this. You know, I'm going to use this." We can share, and the nature of the beast is forcing the issue. Social studies teachers reported some positive coll aborations with peers, but they also cited more instances than the other teachers of sit uations where friction had developed. A high school teacher described the tension that ar ose over course assignments: We have attempted to get together and work, but wha t we have found out has been happening is just been a lot of backstab bing and a lot of animosity because there are a couple of teachers wh o just adamantly refuse to teach 10th grade (when the Global exam is admini stered). So the feeling is, well, they can do the ninth grade program. But where is their
20 of 28accountability? Because they just will not do that 10th grade when their kids take the Regents at the end of the year. This teacher's experience points, again, to the variability in the way consequences of the test are playing out. This variation is expl ained, in part, by the development of as many unintended as intended consequences. State-lev el reformers may have hoped, for example, that teachers would see the test as an imp etus for more ambitious instruction, closer collaboration, and the like. And this seems to be occurring. But reformers probably did not predict the more negative conseque nces these teachers are seeing. That these outcomes are unintended is little solace, for they may be just as real to the teachers as the intended outcomes. Actually, these unintende d consequences may ultimately be more important because they seem to receive scant a ttention from state and district-level actors. State and district leaders may be unaware o f these issues, they may be ignoring them, or they may not see them as problems. In any event, it seems interesting that no teacher mentioned that s/he had participated in any explicit conversations about the problems they anticipated. As noted above, teachers did see positive possibilities arising from the new state tests and there was no particula r sense of gloom during the interviews. How teachers will manage the more negat ive consequences is unclear, but the supposition that they will have no effect seems naive.Implications Substantive change is always unsettling. So reform on the scale that New York state is attempting, in all grades and in all schoo l subjects, is bound to generate some frustration, anxiety, and uncertainty. The findings above tell us that while teachers are not adverse to change, they have real concerns abou t the nature of the changes proposed, the professional development opportunities availabl e to learn about these changes, and the rationales for and consequences of the new stat e tests. Given the complexities of teaching and poli cy (Grant, 1998), it is not surprising to learn that teachers see both prospects and problems in the new NYS tests. State-level policymakers in New York, like most of their peers, are attempting reform on a massive level (Lusi, 1997) and are doing so with relatively few levers for change. What this study suggests is that teachers are not passive participa nts and must not be designed around. The dream of teacher-proof curriculum as a means of changing teachers' practices has proven to be a myth (see, for example, Dow, 1991; S chwille, Porter, Belli, Floden, Freeman, & Knappen, 1983). Faith in tests as a mean s of corralling teachers' practices may ultimately prove just as chimerical as long as teachers are left out of the loop. If any of the changes state reformers propose are to stick then these teachers are saying they need to be more actively involved in the formulatio n of those changes. But there is something else. These findings also suggest that th ere are real and important differences in the ways teachers perceive reforms across grade levels. Among other things, this means that reformers can not take a one-sizefitsall stance and that professional development needs to be sensitive to the difference s in the perceived needs of teachers.NotesThe author wishes to acknowledge Bob Stevenson's th oughtful comments on an earlier draft of this article. The TLA study is funded by the Collaborative Resear ch Network, sponsored by 1.
21 of 28the Graduate School of Education at SUNY-Buffalo. T he faculty and students who worked on this study include Suzanne Miller, Ro bert Stevenson, Mark Templin, Meg Callahan, Diana Lawrence-Brown, and Gi na Trzyna. The small number of elementary school teachers was due partly to design and partly to exigencies that prevented the other invit ees from attending on that date. 2. Corbett and Wilson (1991) point out, however, that Madaus's claims are based on limited data: "anecdotes, testimony from public hea rings, historical accounts, and an occasional international study" (p. 26). 3. Revisions of state tests is still in progress so so me of what follows is based on SED reports of changes they expect will occur. 4. The first administrations of new social studies tes ts will begin in the fall of 2000. 5. For example, in the test sampler for the Global His tory and Geography exam (New York State Education Department, 1999), studen ts would be given documents that range from a poem by Lao Tzu; portio ns from Pericles' "Funeral Oration," the English Bill of Rights, the Japanese Constitution, a speech by Benito Mussolini; and a political cartoon about the monarc hy in France during the 1600-1700s. They are then directed to write an essa y in which they "compare and contrast the different viewpoints societies have he ld about the process of governmental decision making and about the role of citizens in the political decision-making process" and to "discuss the advant ages and disadvantages of a political system that is under the absolute control of a single individual or a few individuals, or a political system that is a democr acy" (p. 25). A test sampler in NYS consists of a descrip tion the types of test items, sample questions, a breakdown of the number of ques tions by curriculum standard and topic, rubrics for essay questions, and sample student responses. At present, the only test sampler available is that for tenth grade Global History and Geography. The first administration of that test is scheduled for June 2000. Test samplers for the grades 5 and 8 tests ar e to be available this fall with administration of the grade 5 test scheduled in Nov ember 200 and the grade 8 test in June 2001. The test sampler for the grade 11 tes t is due out in spring 2000 and the new test is scheduled for June 2001. 6. From the Global History test sampler (New York Stat e Education Department, 1999), students are given this theme on belief syst ems: "At various times in global history, members of different religions have acted to bring people together. Members of these same religions have also acted to divide people and have caused conflict." Students are then directed to this task: "Choose two religions from your study of global history and geography. For each rel igion: Describe two basic beliefs of the religion; Explain how members of the religion, at a specific time and place, acted either to unify society or to cause co nflict in society" (p. 29). 7. The PET tests were given at grades 6 and 8. The new tests will be administered at grades 5 and 8. 8.ReferencesBogdan, R., & Biklen, S. (1982). Qualitative research for education: An introduction to theory and methods Boston: Allyn and Bacon. Cohen, D., & Barnes, C. (1993). Pedagogy and policy In D. Cohen, M. McLaughlin, & J. Talbert (Eds.), Teaching for understanding: Challenges for policy a nd practice (pp. 207239). San Francisco: Jossey-Bass.
22 of 28Comfort, K. (1991). A national standing ovation for the new performance testing. In G. Kulm & S. Malcolm (Eds.), Science assessment in the service of reform (pp. 149-161). Washington, DC: American Association for the Advanc ement of Science. Corbett, H. D., & Wilson, B. (1991). Testing, reform, and rebellion Norwood, NJ: Ablex.Darling-Hammond, L., & McLaughlin, M. (1996). Polic ies and support professional development in an era of reform. In M. McLaughlin & I. Oberman (Eds.), Teacher learning: New policies new practices (pp. 202-219). New York: Teachers College Press. Dow, P. (1991). Schoolhouse politics: Lessons from the Sputnik era Cambridge, MA: Harvard University Press.Editors. (1994). Introduction. Symposium: Equity an d educational assessment. Harvard Educational Review, 64 13. English, F. (1980). Improving curriculum management in the schools Washington, DC: Council for Basic Education.Feltovich, P., Spiro, R., & Coulson, R. (1993). Lea rning, teaching, and testing for complex conceptual understanding. In N. Frederikson R. Mislevy, & I. Bejarq (Eds.), Test theory for a new generation of tests (pp. 181-217). Hillsdale, NJ: Lawrence Erlbaum.Finn, C. (1995). Who's afraid of the big, bad test? In D. Ravitch (Ed.), Debating the future of American education: Do we need national s tandards and assessments? (pp. 120-144). Washington, DC: Brookings Institution.Firestone, W., Mayrowetz, D., & Fairman, J. (1998). Performance-based assessment and instructional change: The effects of testing in Mai ne and Maryland. Educational Evaluation and Policy Analysis, 20 (2), 95-113. Freeman, D., Kuhs, T., Porter, A., Knappen, L., Flo den, R., Schmidt, W., & Schwille, J. (1980). The fourth grade mathematics curriculum as inferred from textbooks and tests East Lansing, MI: Institute for Research on Teachin g, Michigan State University. Fuhrman, S. (1993). The politics of coherence. In S Fuhrman (Ed.), Designing coherent education policy: Improving the system (pp. 1-34). San Francisco: Jossey Bass. Fuhrman, S., Clune, W., & Elmore, R. (1988). Resear ch on educational reforms: Lessons on the implementation of policy. Teachers College Record, 90 237-258. Glaser, B. (1978). Theoretical sensitivity: Advances in the methodolog y of grounded theory Mill Valley, CA: Sociology Press. Glatthorn, A. (1987). Curriculum leadership Glenview, IL: Scott, Foresman. Grant, S. G. (1996). Locating authority over conten t and pedagogy: Cross-current influences on teachers' thinking and practice. Theory and Research in Social Education, 24 (3), 237-272.
23 of 28Grant, S. G. (1997a). Opportunities lost: Teachers learning about the New York state social studies framework. Theory and Research in Social Education, 25 (3), 259-287. Grant, S. G. (1997b). A policy at odds with itself: The tension between constructivist and traditional views in the New York state social studies framework. Journal of Curriculum and Supervision, 13 (1), 92-113. Grant, S. G. (1998). Reforming reading, writing, and mathematics: Teache rs' responses and the prospects for systemic reform Mahwah, NJ: Lawrence Erlbaum. Grant, S. G. (in press ). An uncertain lever: The i nfluence of state-level testing in New York on teaching social studies. Teachers College Record Heubert, J., & Hauser, R. (1999). High stakes: Testing for tracking, promotion, and graduation Washington, DC: National Academy Press. Kellaghan, T., Madaus, G., & Airasian, P. (1982). The effects of standardized testing Boston: Kluwer-Nijhoff.Koretz, D. (1988). Arriving in Lake Woebegon: Are s tandardized tests exaggerating achievement and distorting instruction? American Educator, 12 (2), 8-15; 46-52. Koretz, D. (1995). Sometimes a cigar is only a ciga r, and often a test is only a test. In D. Ravitch (Ed.), Debating the future of American education: Do we ne ed national standards and assessments? (pp. 154-166). Washington, DC: Brookings Institutio n. LeCompte, M., Preissle, J., & Tesch, R. (1993). Ethnography and qualitative design in educational research (2nd ed.). New York: Academic Press. LeMahieu, P. (1984). The effects on achievement and instructional content of a program of student monitoring through frequent testing. Educational Evaluation and Policy Analysis, 6 (2), 175-187. Lusi, S. F. (1997). The role of state departments of education in compl ex school reform New York: Teachers College Press.Madaus, G. (1988). The influence of testing on the curriculum. In L. Tanner (Ed.), Critical issues in curriculum: 87th yearbook of the NSSE, Part 1 Chicago: University of Chicago Press.National Center for History in the Schools. (1994). National standards for United States history Los Angeles: Author. National Council for Social Studies. (1994). Expectations of excellence Washington, DC: Author.Natriello, G., & Pallas, A. (1998). The development and impact of high stakes testing. Available on-line: http://www.columbia.edu/~gjn6/his take.html New York State Education Department. (1996). Learning standards for social studies Albany, NY: Author.
24 of 28New York State Education Department. (1998). Social studies resource guide Albany, NY: Author.New York State Education Department. (1999). Global history and geography Regents examination: Test sampler draft Albany, NY: Author. Noble, A., & Smith, M. L. (1994). Old and new belie fs about measurement-driven reform: "Build it and they will come." Educational Policy, 8 (2), 111-136. Popham, J., Cruse, K., Rankin, S., Sandifer, P., & Williams, P. (1985). Measurement-driven instruction: It's on the road. Phi Delta Kappan, 66 (9), 628-634. Popham, W. J. (1998). Farewell, curriculum: Confess ions of an assessment convert. Phi Delta Kappan, 79 380-384. Ravitch, D. (Ed.). (1995). Debating the future of American education: Do we ne ed national standards and assessments? Washington, DC: Brookings Institution. Resnick, D., & Resnick, L. (1985). Standards, curri culum, and performance: A historical and comparative perspective. Educational Researcher, 14 (4), 5-20. Salmon-Cox, L. (1981). Teachers and standardized ac hievement tests: What's really happening? Phi Delta Kappan, 62 631634. Schwille, J., Porter, A., Belli, G., Floden, R., Fr eeman, D., & Knappen, L. (1983). Teachers as policy brokers in the content of elemen tary school mathematics. In L. Shulman & G. Sykes (Eds.), Handbook of teaching and policy (pp. 370-391). New York: Longman.Shanker, A. (1995). The case for high stakes and re al consequences. In D. Ravitch (Ed.), Debating the future of American education: Do we ne ed national standards and assessments? (pp. 145-153). Washington, DC: Brookings Institutio n. Smith, M., & O'Day, J. (1991). Systemic school refo rm. In S. Fuhrman & B. Malen (Eds.), The politics of curriculum and testing (pp. 233-267). New York: Falmer. Smith, M. L. (1991). Meanings of test preparation. American Educational Research Journal, 28 (3), 521-542. Smith, M. L., Heinecke, W., & Noble, A. (1999). Ass essment policy and political spectacle. Teachers College Record, 101 (2), 157-191. Smylie, M. (1995). Teacher learning in the workplac e; Implications for school reform. In T. Guskey & M. Huberman (Eds.), Professional development in education: New paradigms and practices (pp. 92-113). New York: Teachers College Press. Stiggins, R., & Conklin, N. (1992). In teachers' hands: Investigating the practices of classroom assessment Albany, NY: SUNY Press. Tyack, D., & Cuban, L. (1995). Tinkering toward utopia Cambridge: Harvard University Press.
25 of 28Wiles, D. (1996). Networking high performance in New York's secondary education: The Regents curriculum story. Lanham, MD: University Press of America. Wolf, R. (1998). National standards: Do we need the m? Educational Researcher, 27 22-5.About the AuthorS. G. Grant517 Baldy HallState University of New York at BuffaloBuffalo, New York 14260 Email: firstname.lastname@example.org Telephone: 716-645-6493S. G. Grant is an assistant professor of Social Stu dies Education in the Department of Learning and Instruction. He has published papers i n both social studies and general education journals. His most recent journal publica tions have been in Theory and Research in Social Education and the American Educational Research Journal. In the fall of 1998, he published his first book, Reforming Reading, Writing, and Mathematics: Teacher's Responses and the Prospects for Systemic Change An article on the influence of state-level tests on teachers' classroom practic es is forthcoming in Teachers College Record .AppendixFOCUS GROUP PROTOCOLSpring, 1998 Introduction: Why we are here. Guidelines and groun d rules. METAPHORSModerators and participants introduce themselves to group. To get started, introduce yourself to someone next to you and describe an image or metaphor that characterizes your thinking and/or fe elings about the new state assessments.After they have shared in pairs, have them share th eir metaphors with the group. Have participants discuss and elaborate on the meta phors. Lead a discussion of the metaphors. What do they say about our thinking? Com mon features? Significant differences.Direct the discussion toward the next question-what do these assessments mean to you?.
26 of 28 MEANING OF ASSESSMENTSWhat do/will these assessments mean to you? Your sc hool? Your students? Transition to next question-are you prepared to dea l with these implications? BEING PREPAREDHow prepared to deal with these assessments do you feel? How are you being prepared? What are you being prepared for? What opp ortunities do you have to talk about the assessments and related issues?Build on these expressions to move toward a discuss ion of needs. What help do you need?This discussion should lead naturally to talk of ch allenges. CHALLENGESWhat challenges/concerns do you anticipate? How wil l you deal with these challenges/concerns? Who do you expect will help yo u? CLOSUREWhat has this conversation made you think about con cerning teaching and testing (e.g., issue, question, new image) ? Copyright 2000 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, email@example.com or reach him at College of Education, Arizona State University, Tempe, AZ 8 5287-0211. (602-965-9644). The Commentary Editor is Casey D. C obb: firstname.lastname@example.org .EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov email@example.com Thomas F. Green Syracuse University
27 of 28 Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Western Interstate Commission for HigherEducation William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX) Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton firstname.lastname@example.org Hugh G. Petrie SUNY Buffalo Richard C. Richardson New York University Anthony G. Rud Jr. Purdue University Dennis Sayers Ann Leavenworth Centerfor Accelerated Learning Jay D. Scribner University of Texas at Austin Michael Scriven email@example.com Robert E. Stake University of IllinoisÂ—UC Robert Stonehill U.S. Department of Education David D. Williams Brigham Young UniversityEPAA Spanish Language Editorial BoardAssociate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico firstname.lastname@example.org Adrin Acosta (Mxico) Universidad de Guadalajaraadrianacosta@compuserve.com J. Flix Angulo Rasco (Spain) Universidad de Cdizfelix.email@example.com Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho dis1.cide.mx Alejandro Canales (Mxico) Universidad Nacional Autnoma deMxicocanalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State Universitycasanova@asu.edu Jos Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es
28 of 28 Erwin Epstein (U.S.A.) Loyola University of ChicagoEepstein@luc.edu Josu Gonzlez (U.S.A.) Arizona State Universityjosue@asu.edu Rollin Kent (Mxico)Departamento de InvestigacinEducativa-DIE/CINVESTAVrkent@gemtel.com.mx firstname.lastname@example.org Mara Beatriz Luce (Brazil)Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.brJavier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mxMarcela Mollis (Argentina)Universidad de Buenos Airesmmollis@filo.uba.ar Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mxAngel Ignacio Prez Gmez (Spain)Universidad de Mlagaaiperez@uma.es Daniel Schugurensky (Argentina-Canad)OISE/UT, Canadadschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil)Fundao Instituto Brasileiro e Geografiae Estatstica email@example.com Jurjo Torres Santom (Spain)Universidad de A Coruajurjo@udc.es Carlos Alberto Torres (U.S.A.)University of California, Los Angelestorres@gseisucla.edu