USF Libraries
USF Digital Collections

Educational policy analysis archives

MISSING IMAGE

Material Information

Title:
Educational policy analysis archives
Physical Description:
Serial
Language:
English
Creator:
Arizona State University
University of South Florida
Publisher:
Arizona State University
University of South Florida.
Place of Publication:
Tempe, Ariz
Tampa, Fla
Publication Date:

Subjects

Subjects / Keywords:
Education -- Research -- Periodicals   ( lcsh )
Genre:
non-fiction   ( marcgt )
serial   ( sobekcm )

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
usfldc doi - E11-00465
usfldc handle - e11.465
System ID:
SFS0024511:00464


This item is only available as the following downloads:


Full Text
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 13issue 43series Year mods:caption 20052005Month October10Day 2828mods:originInfo mods:dateIssued iso8601 2005-10-28


xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20059999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00465
0 245
Educational policy analysis archives.
n Vol. 13, no. 43 (October 28, 2005).
260
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c October 28, 2005
505
Limiting the unintended consequences of high-stakes testing / Stuart S. Yeh.
650
Education
x Research
v Periodicals.
2 710
Arizona State University.
University of South Florida.
1 773
t Education Policy Analysis Archives (EPAA)
4 856
u http://digital.lib.usf.edu/?e11.465



PAGE 1

Readers are free to copy display, and distribute this article, as long as the work is attributed to the author(s) and Education Policy Analysis Archives, it is distributed for noncommercial purposes only, and no alte ration or transformation is made in the work. More details of this Creative Commons license are available at http:/ /creativecommons.org/licen ses/by-nc-nd/2.5/. All other uses must be approved by the author(s) or EPAA EPAA is published jointly by the Colleges of Education at Arizona State University and the Universi ty of South Florida. Articles are indexed by H.W. Wilson & Co. Accepted under the editorship of Sherman Dorn. Send commentary to Casey Cobb (casey.cobb@uconn.edu) and errata notes to Sh erman Dorn (epaa-editor@shermandorn.com). EDUCATION POLICY ANALYSIS ARCHIVES A peer-reviewed scholarly journal Editor: Sherman Dorn College of Education University of South Florida Volume 13 Number 43 October 28, 2005 ISSN 1068–2341 Limiting the Unintended Conse quences of High-Stakes Testing1 Stuart S. Yeh University of Minnesota Citation: Yeh, S. S. (2005). Limiting the unin tended consequences of high-stakes testing. Education Policy Analysis Archives, 13 (43). Retrieved [date] from http://epaa.asu.ed u/epaa/v13n43/. Abstract Interviews with 61 teachers and administrators in 4 Minnesota school districts suggest that, in their judgment, Minnesota’ s state-mandated test s were well-aligned with curricular priorities and teachers’ instructional goals, emphasizing critical thinking as well as competencies needed to pass the Basic Standards exit exam, and avoiding the type of recall item that would require dr ill and memori zation. This result, in combination with a survey showin g that 85 percent of Minnesota teachers support the exit exam, suggests that Minn esota has been unusua lly successful in designing a high stakes te sting system that has garnered teacher support. The success of Minnesota’s model suggests that unintended narrowing of the curriculum due to high stakes testing may be avoided if pressure on teachers to narrow the curriculum is reduced through well-designed, well-aligned exams. Keywords: testing; assessment; accountabi lity; curriculum; po licy analysis; largescale achievement tests; high stakes. 1 I thank Karen Wrobbel for assistance in conducting the interviews and Karen Seashore and anonymous reviewers for comments on a previous draft of the manuscript.

PAGE 2

Education Policy Analysis Archives Vol. 13 No. 43 2 Introduction The No Child Left Behind (NCLB) Act of 2001 requires annual testing and establishes sanctions for schools that do not maintain “a dequate yearly progress” in improving student achievement. Initial estimates ar e that 80–90 percent of all schools receiving Title I funds will be found “in need of improvement” by the fede ral government and subject to NCLB sanctions (Bracey, 2002). A concern is that this type of high-s takes testing may be fundamentally flawed, that it may narrow the intended curriculum—defined as th e knowledge, skills and habits of thought teachers believe are important for students to learn— causing teachers to “teach to the test” (McNeil, 2000). However, narrowing of the curriculum may not always be undesirable (Smith & O'Day, 1991). Alignment of state standards, tests, curriculum and professional development may be necessary to ensure that students are taught and tested on the content and skills they are expected to know and be able to do. Well-designed tests coul d help teachers to identify and address areas of weakness, improving the quality of instruction as well as providing accountability information. It may be reasonable to narrow the curriculum by focusing teachers’ efforts and emphasizing topics that are valued (Smith & O'Day, 1991). This re asoning has engendered broad support and a decade of bipartisan legislative initiatives culminating in NCLB (Jennings, 2000). Whatever its flaws, NCLB seems unlikely to be repealed (Elmore, 2004). Cu rrently, however, there is an opportunity to modify the legislation and its implementation to reduce unintended negative consequences. The focus of this study is to understand what can be done to avoid unintended, negative consequences due to narrowing of the curriculum in a high-stakes testing environment. In Minnesota, high stakes testing was implem ented in 1998 and diplomas were withheld from students who failed the state Basic Standards Test (BST) beginning in 2000 (sample tests and items available at http://education.state.mn.us/mde/Accountab ility_Programs/Assessment_and_Testing/Assessmen ts/index.html). Surprisingly, however, an overwh elming 85 percent of Minnesota teachers support the State’s exit exam (Draper, 2000). It is impor tant to understand what may contribute to these positive views. It is unlikely that teachers would su pport the exit exam if they believe that it forces them to eliminate valuable chunks of the curriculu m. If teachers in Minnesota do not feel pressure to narrow the curriculum in a way that reduces educational quality, it is important to understand their reasoning. This reasoning is likely to expl ain their choices about what to teach in their classrooms—choices that ultimately determine the extent to which the curriculum may or may not be narrowed. If we understand their reasoning, perhaps we can understand the conditions under which the curriculum may be narrowed—and how ex cessive narrowing can be avoided. The study reported below suggests that the unique design of Minnesota’s testing program may be well-aligned with teachers’ expert judgment about what should be emphasized. This design may contribute to teacher satisfaction with high stakes testing, and may provide lessons about how to implement high stakes testing in a way that minimizes unwanted narrowing of the curriculum. Four Concerns There are four main concerns related to na rrowing of the curriculum and the fear that improved test scores merely reflect teaching to the test, rather than broad gains in learning. One concern is that non-tested subjects, such as art, may be de-emphasized (McNeil & Valenzuela, 2000;

PAGE 3

Limiting the Unintended Conse quences of High-Stakes Testing 3 McNeil, 2000; Smith, 1991; Smith & Rottenberg, 19 91). This might be addressed by requiring schools to provide elective or required instruction in those areas. A second concern is that even within tested sub jects, particular topics or skills not included on the test may be de-emphasized (McNeil & Valenz uela, 2000; McNeil, 2000; Smith, 1991; Smith & Rottenberg, 1991). This might be addressed by ensuri ng that the most important topics and skills are covered by the state-mandated test. A third concern is that testing narrows curricu lum and instruction to memorization of bits of factual knowledge (McNeil & Valenzuela, 2000; McNeil, 2000; Smith, 1991; Smith & Rottenberg, 1991). This may occur if state curriculum standard s and tests aligned to those standards attempt to cover too much factual knowledge. An unanticipate d consequence is that teachers may feel it is necessary to drill students on the universe of factual items that may appear on the test, in the hope that students will correctly answer those items that actually appear. Addressing this issue may require changes in state curriculum standards as well as the state test. A fourth concern is that testing may cause teachers to spend excessive time on test-taking tricks and strategies that have little value outside of the testing situation (McNeil & Valenzuela, 2000; McNeil, 2000; Smith, 1991; Smith & Rottenberg, 1991 ). For example, students may be taught to make sure that bubble sheets are correctly completed This issue might be addressed through tests designed to minimize unwanted trickiness. The first concern might be addressed through policy and the last concern might be addressed through proper test design. The focus of th is paper is the second and third concerns: the construct under-representation that occurs when tests only assess aspects of the intended curriculum that are easy to measure (Baker & Linn, 2004). The question is whether it is indeed possible to design and implement large-scale assessments, suitabl e for state-mandated testing, that are properly aligned with the curriculum in order to reinforce intended learning outcomes (Linn, 2004; Rothman, 2004). A review of state-mandated tests and stat e curriculum standards found that while there is superficial alignment of test con tent with state curriculum standards in many states, a more careful analysis suggests that lack of alignment may indeed be a problem (Rothman, 2004; Rothman, Slattery, Vranek, & Resnick, 2002). Effects of State-Mandated Testing on the Curriculum Two reviews of research found that in some cases—but not others—state-mandated tests narrow the curriculum (Cimbricz, 2002; Mehrens, 1998 ). To a certain extent, variation in findings reflect individual, school and district-level differen ces in experiences, training and attitudes with regard to high-stakes testing (Cimbricz, 2002; Grant, 2000). Some individuals have positive attitudes, feeling that testing can be used to diagnose and ad dress weaknesses, and are well-trained to use test results in this way (Massell, 2001). Others have ne gative attitudes and are not trained or have little experience in using test results for formative pu rposes. A path model suggests that both teacher attitudes and professional development mediate th e effects of testing programs on instructional practices (Pomplun, 1997). However, it seems unlikely that all of the variation in outcomes is due to individual differences. Each state has control over the design and implementation of its state test and it seems likely that those decisions systematica lly influence the degree to which the curriculum is narrowed. Unfortunately, existing research does not provide a strong basis for specifying exactly what factors or conditions are important.

PAGE 4

Education Policy Analysis Archives Vol. 13 No. 43 4 Stake Levels A common supposition is that differences in stake levels explain variation in outcomes. It is hypothesized that low stake levels are associated with little or no narrowing, while high stakes are associated with greater pressure on teachers and mo re narrowing of the curriculum. There is some evidence in support of this hypothesis. During a pe riod of low stakes, when the results of statemandated tests were rarely tied to decisions abou t student promotion or graduation, or threats to reorganize schools, Porter, Floden, Freeman, Sc hmidt, and Schwille (1986) concluded that: “Another myth exposed as being only a half truth is that teachers teach topics that are tested. Little evidence exists to support the supposition that national norm-referenced, standardized tests administered once a year have any important influence on teachers’ content decisions” (p. 11). Kuhs et al. (1985) concluded that Michigan teachers’ topi c selection was not significantly influenced by the state minimum competencies test or the district-used standardized tests. In contrast, several studies of Texas’ high-s takes testing program concluded that preparing students for state-mandated tests narrows the curriculum (Gordon & Reese, 1997; Haney, 2000; Hoffman, Pennington, Assaf, & Pa ris, 1999; McNeil & Valenzuela 2000). Smith and Rottenberg’s (1991) study of two elementary schools in Arizona also concluded that high-stakes testing causes narrowing of the curriculum as teachers align instruction with test content. However, high stake levels do not always result in narrowing of the curriculum. Grant (2001) compared the teaching of two high school teacher s in New York, where passing the state Regents exams entitled students to a prestigious Regents diploma, and passing the easier Regents Competency Test was necessary to obtain a regular diploma. In this high-stakes situation, Grant found little direct influence of testing on conten t or pedagogical decision-making and concluded instead that the teachers’ personal beliefs governed their teaching (also see Grant, 2000). Similarly, a study of Kentucky’s reform efforts concluded that teachers’ responses to these efforts in four exemplary schools were shaped by the participants’ shared vision of curriculum and commitment to children, rather than by high-stakes testing (Wolf, Borko, Elliott, & McIver, 2000). Notably, in both of these cases, it could be argued that while st akes were high, pressure on teachers was low. In New York, students who failed the more demanding Regent s’ exams were not denied a diploma as long as they passed the minimum competency Regents’ Com petency Test. And in the four exemplary Kentucky schools, it appears that strong administ rative leadership deflected pressure from teachers and strong teaching served to raise test scores and allay parent concerns. In other words, effective teaching may have reduced the number of students who failed the state-mandated test and reduced pressure to narrow the curriculum. Performance-Based and Portfolio Assessments If individual differences and differences in stake levels do not adequately explain variation in the impact of state-mandated tests, what about characteristics of the tests themselves? Some researchers are concerned that multiple-choice tests are inherently poorly-aligned with sound teaching practices, causing teachers to “dumb-down” their instruction (McNeil & Valenzuela, 2000; McNeil, 2000; Smith, 1991; Smith & Rottenberg, 1991). In view of this, it has been suggested that performance-based or portfolio assessments are more likely to shape instruction in desirable ways, to encourage teaching that promotes active learning, compared to multiple-choice assessments, and to avoid negative narrowing of the curriculum (B aron & Wolf, 1996; Rothman, 1995). In principle, well-designed performance-based and portfolio as sessments could be aligned with sound teaching

PAGE 5

Limiting the Unintended Conse quences of High-Stakes Testing 5 practices and would provide information about student progress regarding important basic and critical thinking skills that teachers would use to improve instruction. However, evidence regarding this hypothesis is mixed (Mehrens, 1998). A qualitative study of the effects of testing in Maine and Maryland found that performance-based assessments have limited effects on changing basic instructional strate gies when stakes are lo w or moderate (Firestone, Mayrowetz, & Fairman, 1998). In Kentucky, a survey of teachers found that 90 percent agreed that portfolios made it difficult to cover the regular curriculum, and teacher opinion about the effect of portfolios on instruction was evenly split (Koretz, Barron, Mitchell, & Stecher, 1996). Fewer than 45 percent of principals and teachers reported th at KIRIS (Kentucky’s portfolio-based assessment program) provided a better view of school e ffectiveness compared to more conventional, commercial standardized tests. The open-response component was rated as having positive effects on instruction more often than performance events and portfolios, suggesting that it may not be necessary to use costly, time-consuming, and diff icult-to-score forms of assessment in order to achieve instructional improvement. A RAND study of Vermont’s portfolio system concluded that the use of portfolio assessment had substantial positive effects on fourth grade math teachers’ teaching practices, but evidence of validity was unpersuasive and unreliability in test scores precluded most of the intended uses of the scores (Koretz, Stecher, Klein, & McCaffrey, 1994). In Arizona, teacher opinion regarding the consequences of the state’s performance-oriented assessment program was divided (Smith et al., 1997). To the ex tent that it occurs, narrowing of the curriculum may happen independently of the format of the test (Haney & Madaus, 1989; Mehrens, 1998). The research reviewed above suggests that performanceand portfolio-based assessments are unlikely to be panaceas for negative consequences of testing. Research Questions The present study investigated the responses of teachers and administrators in four districts in Minnesota, and addressed the following research questions: 1) What are the views of teachers and administrators in the four Minnesota districts regard ing state-mandated testing? and 2) In what ways does Minnesota’s state-mandated testing program influence teacher and administrator choices about what is taught in their classrooms and schools? Minnesota’s State-Mandated Assessments Minnesota requires all districts within the st ate to administer two types of tests. The Minnesota Comprehensive Assessments (MCA) are high standards tests. At the time of the research study, they were administered at grades 3 (readi ng and math), 5 (reading, math, and writing), 10 (reading and writing) and 11 (math). The MCA tests will also be implemented in grades 4, 6, 7, and 8 beginning in 2005-2006. The reading and math te sts are composed of multiple-choice and short answer questions. The writing test requires a written essay. The MCAs are designed to be aligned with Minnesota Academic Standards, but no information is available about the actual degree of alignment. At the time of the research study, the Minneso ta Basic Standards Test (MBST or BST) was a minimum competency exit exam administered in gr ade 8 (reading and math), 10 (writing), and 12

PAGE 6

Education Policy Analysis Archives Vol. 13 No. 43 6 (for seniors who needed to re-take the MBST).2 Students were required to pass the test in order to graduate from high school. The test was designed to ensure that all graduates possess basic skills in reading, writing and math, and was composed of multiple choice questions and a written essay. Cumulative passing rates on the MBST (after five attempts) for all Minnesota students were high: 99.1% in math and 99.5% in reading/language arts (Gayler, Chudowsky, Hamilton, Kober, & Yeager, 2004). Because the MBST exit exam was not intended to be aligned to Minnesota Academic Standards, no alignment review had been conducted (Gayler et al., 2004). The dual system of using high-standards tests as well as a basic skills exam is a reflection of the desire to test both levels of skills but to ma ke high school graduation contingent on passing a basic skills exam. The MCAs and MBST are representative of the current generation of statemandated tests regarding their format (largely multiple-choice with some open response and essay items), content (reading, math, and writing), and intended use (auditing school performance). At the time of the study, there were no accountability co nsequences or rewards for teachers, schools or districts linked to student performance on the MCAs or exit exam. The state did not provide special programs of support to schools and teachers in the form of professional development aligned to the assessments and standards. Sample tests and items are available at http://education.state.mn.us/mde/Accountab ility_Programs/Assessment_and_Testing/Assessmen ts/index.html. Methods Data reported here come from a qualitative stud y of teacher and administrator responses to state-mandated testing in four Minnesota distri cts during the 2002–2003 school year. Minnesota was selected for two reasons. First, the state testing program is “high stakes”—students must pass the Basic Standards Test, administered beginning in 8th grade, in order to graduate from high school. Therefore, it was possible to investigate the eff ects of a high-stakes testing program on curriculum and instruction. Second, Minnesota has an unusual system where the Minnesota Comprehensive Assessments (MCAs) are challenging “high standards” tests, but the 8th grade exit exam is a minimum competency test. Unlike systems that rely primarily on minimum competency tests, Minnesota’s dual approach may be less sus ceptible to narrowing of the curriculum. Four school districts were selected that wer e representative, in terms of student demographics, size and distance from major urban centers, of four major categories: 1 large urban (40,000+ students, 74% minority, 68% free/reduced lunch), 1 small urban (<2000 students, 62% minority, 37% free/reduced lunch), 1 suburban (1 1,000 students, 28% minority, 8% free/reduced lunch), and 1 semi-rural district (<2000 studen ts, 2% minority, 4% free/reduced lunch). Each district represents the challenges that face districts in the corresponding category. These particular challenges may influence the manner and degree to which the curriculum is narrowed. The large urban district represents the challenges of coordina ting the education of a large, extremely diverse student population, many of whom are at risk of not passing the state-mandated test. The small urban district represents the challenges of meeting the needs of a diverse student population without the resources that are available in a large dist rict—such as a dedicated director of testing. The suburban district represents the challenges of ma intaining a challenging curriculum and meeting the educational needs of students, most of whom are likely to pass the state-mandated test and should not be subjected to remedial instruction designed for students who are at-risk of not passing the 2 The MBST is being replaced by the MCA-II high standards graduation test, starting with the class of 2010.

PAGE 7

Limiting the Unintended Conse quences of High-Stakes Testing 7 tests. The semi-rural district represents the challe nges of meeting student needs with very limited resources. In each district, interviews were obtained with the Superintendent or Assistant Superintendent, Director of Testing, and the prin cipal and four teachers at 1 high school, 1 middle school and 1 elementary school (except in one di strict that only had two schools). Schools were selected that represented the socioeconomic ch aracteristics (ethnicity and percent receiving free/reduced price lunch) of each district. One el ementary school declined to participate after the principal learned that teachers would not be comp ensated for the interviews (a second elementary school was substituted). The researcher has no reas on to believe that this affected the results. Table 1 Distribution of Sample District Type Interviewees Large Urban Small Urban Suburban Rural Total Superintendent 1 1 1 1 4 Director of Testing 1 a 1 a 2 Principal/Assistant Principal 3 2b 3 3 11 Elementary Teacher Regular 3 3 4 4 14 Special Education 1 1 1 0 3 Middle School Teacher a Math 1 1 3c 5 Language Arts 1 1 3c 5 Science 0 1 3c 4 Special Education 2 0 1 3 High School Teacher Math 1 2c 1 1 5 Language Arts 1 2c 1 2 6 Science/social studies 2 0 1 1 4 Special Education 0 1 1 0 2 a Not employed in this context. b Single middle/high school. cTeachers teach multiple subjects. Teachers were stratified by subject (math and reading/language arts) and tested grade, and randomly selected. In addition, science teachers were randomly sampled in order to assess the impact of testing in a non-tested subject. All partic ipants had bachelor’s degrees and approximately one-third also held masters’ degrees (six administ rators held doctorates). Years of experience in the classroom ranged from 1–29 years. Approximatel y 60% of the participants were female, and 40% were male. A total of 61 individuals were interview ed. Table 1 presents the distribution of the research sample. Data Collection Development of the semi-structured interview protocol (Appendix A) was informed by three rounds of previous interviews with approxima tely 100 teachers and principals in three states regarding the effects of state-mandated testing. Each participant was interviewed individually.

PAGE 8

Education Policy Analysis Archives Vol. 13 No. 43 8 Interviews were audio recorded and lasted abou t 50 minutes. The interview protocol included general questions about the impact of testing as well as specific questions about the influence of testing on teaching and learning: “Overall, is your opinion about the impact of the state testing program positive or negative? Does the state tes ting program influence the type of skills and knowledge that you focus on? Are these skills and knowledge important? Is the test designed so you feel a need to drill students on factual material?” Data Analysis In keeping with established methods of qualitative research (Bogdan & Biklen, 1992; Glaser & Strauss, 1967; Miles & Huberman 1994; Strauss, 1987), data analysis followed the three-part modified constant comparative method. Each interviewer asked follow-up questions to make certain that the meanings respondents intended were understood. As interviews were conducted, emergent findings were checked through follow-up questi ons with subsequent interviewees. Interviews were transcribed and coded to identify primary themes. The themes were identified jointly by two raters and cross-checked by a third researcher. Discre pancies were resolved through discussion. Interview excerpts were selected that illustrated views held by the majority of participants. Limitation of the Study A limitation of this study is that it relies on teacher and administrator judgments regarding the effect of Minnesota’s tests on curriculum an d instruction. These judgments may be biased. However, two recent studies suggest that, in some ca ses, self-report data may be a reliable indicator of actual instructional practices. One study co mpared teacher self-reports and independent observations of teachers' enacted curriculum and found strong agreement between the two measures (Porter & Smithson, 2001). A second, national study involving 25,000 teachers found close agreement between teachers’ self-re ported instructional practices and students’ perceptions of those practices (Shim, Felner, Shim, & Noonan, 2001). Results What is the Effect of Testing on Curriculum and Instruction? What are the views of teachers and administrators in the four Minnesota districts regarding state-mandated testing? By an overwhelming two -to-one margin, most interviewees answered that the overall impact of state-mandated testing is pos itive. This margin was consistent across district type (urban, suburban, and rural), and grade level. All administrators answered that the overall impact of testing is positive. The results are cons istent with a random survey of teachers across the state, which found that 85 percent support the Basic Standards exit exam (Draper, 2000). Interviewees who viewed testing positively exhibi ted four patterns when responding to this question. First, teachers felt that testing impr oved the quality of the curriculum, although they initially feared that testing would have a negative impact, causing teachers to focus only on basic skills. According to interviewees, test results force tea chers to acknowledge that many students are below grade level in math and reading, prompting greater collaboration among teachers and

PAGE 9

Limiting the Unintended Conse quences of High-Stakes Testing 9 administrators to improve the quality of the curricu lum and to align the curriculum vertically from grade to grade. For example, a princi pal in a suburban middle school explained: The negative part about it would be if teache rs and staff were simply addressing the basics for the kids—teaching the basi cs, not taking them to the next level because that test has become so important, and that would be their only focus in the classroom. I think there was a lot of in itial talk about that when the testing first started. I don’t see th at happening here in our mi ddle school now that I have that opportunity to be in classrooms… The positive part is that we are looking at education differently and that we’re not working in isolation. Things are on the table, so to speak, and so we can work cooperat ively as a team and make differences in the curriculu m that really enrich the curriculum and enrich the teaching for the kids and the staff. So I see that as the positive side of it. There’s more dialogue, there’s connection between K–12 curriculum, there’s connection between the levels of high school middle school, [and] elementary. Interviewees in the majority grou p asserted that teac hers prepare students for the test by integrating the skills needed to pass the test into the school curriculum throughout the year, rather than through is olated test preparation that narrowl y focused on the types of items expected on the test. A principal in a se mi-rural elementary school explained: I think the test forces us to look at what ’s important… it’s not in a way that the teachers teach to a test, but I think they each in their own way make sure that they bring to the forefront in their instru ction the skills that en able kids to take the test without a high leve l of anxiety. I think they do a great job of that, and they don’t do it a week before or two weeks before; I thin k they do it in a timely fashion so that we can honestly say that we’re not teaching to the test. We’re teaching in a way that children can learn. Second, interviewees in this group felt that testing has made them more accountable and improved the quality of instruction. Teachers believed they were more focused, goal-oriented and reflective about what they needed to teach. Te achers reported more dialogue and communication among themselves. Professional development and sta ffing are said to be more focused on improving student achievement. Teachers report that after-s chool intervention programs have been initiated and focus on the needs of low-achieving children. Fo r example, a high school science teacher in an urban school suggested that testing has impacted instruction through a number of different ways: I think it has been wonderfu l. I think it has raised a number of eyebrows, and it has caused people to step back and ask th emselves, ‘What is it that I’m doing in the classroom that they’re asking me on a test, and why are they asking.’ In other words, it’s causing a lot of self-reflection that teachers often don’t have time to do. [They do] a lot of planning but ofte n not a lot of time reflecting... It’s impacting professional development such that now our profe ssional development is geared more towards the impact on th e classroom, not just on me going out to a meeting of the National Science Teachers Asso ciation meeting in Hawaii. Now I have to relate it to how it’s going to impact the class. So it’s impacted the professional development as well. Obvi ously it has impacte d funding at the district level. We now have specialists in the classrooms or in the buildings. And I think it’s also raised awareness that you have some kids that are special ed and kids that are not, an d those kids that are outside of special ed still need help, and we’re seeing that help is made available to them.

PAGE 10

Education Policy Analysis Archives Vol. 13 No. 43 10 Third, teachers felt that testing improved student attitudes, engagement, and effort by holding students accountable for learning. Teachers noticed that more children are seeking help and participating in after-school programs. An 8th grade math teacher in a suburban middle school explained that her view of testing is positive beca use it causes students to adopt a new attitude toward learning: Overall, positive…[because] it is finally something that the ki ds are accountable for. Up through middle school, you ar e not accountable for anything in Minnesota really. You can fail, fail, fail, fa il, and still go into high school. In high school, you’re accountable. So this is finally something where the kids say, “Oh my gosh, this counts.” So I like that; I like watching the attitude change because for the first time, it counts. Fourth, teachers felt that testing resulted in gr eater efforts to ensure that all children succeed and improved student achievement. For example, a 4th grade teacher in a semi-rural elementary school described a program to identify children with low reading scores and to refer them to a special after-school program: Here in [district name] ou r children coming out of third grade and into fourth grade that have scored low in the readin g area, they are iden tified and are given the opportunity to be in an after school program called “Soar to Success.” I think it’s an 18 week program, ru ns from 3:00 to 4:00. I’ve se en a lot of kids have great success with the program…It’s one of the programs that have come out of the low scores from the state testi ng. [So you view that as a positive?] Yes, very, very much so, very much so a positive. If no thing else, their self-esteem while they’re in that program, they’re ge tting their homework done, they’re learning different ways to attack reading. In What Ways Does Test ing Narrow the Curriculum? In the previous section, interviewees reported that testing led to focusing of the curriculum. Given the concern among educators about the effects of narrowing of the curriculum, the second question addressed in this study is the following: In what ways does Minnesota’s state-mandated testing program influence teacher and administrator choices about what is taught in their classrooms and schools? The themes reported below characteri zed the views of the majority of interviewees who felt, by a two-to-one margin, that state-mandated testing had a positive impact on the curriculum. These themes were consistent across district type (urban, suburban and rural). A focus on basic skills at the 8th grade level As might be expected, interviewees stated that teachers focused on basic skills at the 8th grade level, consistent with the emphasis of the Basic Standards Test on minimum competencies. Teachers also focused on particular topics that were emphasized on the state test. What was unexpected was that teachers in the majority group—across suburban, urban, and rural districts—felt that this focus was appropriate. An 8th grade math teacher in a suburban middle school explained: Absolutely! The first year we took the tes t, we spent that summer rewriting our curriculum to align with [it]. We broke it down; we took the eight strands and we analyzed which strands [our district’s] ei ghth graders did the poorest on. We took those strands and we put them in the front of the year, so that we had them done by February. Ratios, proportions and perc ents were the weakest area, so we did that right before th e test. [Do you feel that these skills and knowledge that you

PAGE 11

Limiting the Unintended Conse quences of High-Stakes Testing 11 teach are important?] Yes, al l of those eight different strands are necessary skills for kids. So it’s not like I’m teaching surfing or some thing totally irrelevant. It is relevant to their daily lives. Although teachers focused on basic skills, they did not feel overly constrained. Both English language arts and math teachers in the majority gr oup viewed the state test as well-aligned with their instructional priorities and believed that they were te aching vital skills that were applicable beyond the test. An 8th grade English teacher in an ur ban secondary school explained: I teach them basic skills that I know will be helpful. . For example, making sure that when you’re reading an article, that you look at so me of the quest ions first so that you’re not just reading random info rmation...They’re good skills to have…I don’t necessarily teach to the test so much as I teach some skills that I think will be beneficial for the test [as well as] in other areas. A math teacher in a semi-r ural high school added: As an adult, and as an ed ucator, yes, I feel [that the skills measured by the Basic Standards Test are] important, because without those skills, most people are not going to be able to make good judgments throughout their lives. With those skills, they have more of an equal opportunity to make good judgments and decisions… [And] I think the grad standa rds in mathematics as a whole have done a lot to improve math education in the state of Minnesota. I think without it, there’d be many school di stricts still doing what they did twenty to thirty years ago, because why change if you don’t have to? Teachers in this group believed that the test co vers basic math skills th at all students should learn. As an 8th grade teacher in an urba n elementary school added: “It’s all part of our curriculum...Those things are supp osed to be basic skills that ev eryone would have. So I really don’t have a problem with the test.” An 8th grade math teacher in an urban secondary school put it succinctly: “The topics on th e test are, I think, topics that students should know.” These views were shared wit hout exception by all of the administrators who were interviewed, across rural, suburban and urban distri cts. Superintendents agreed that the test has focused the curriculum on important skills and knowle dge at every grade level. A superintendent in a semi-rural district said: Definitely in math and reading… it certainly has influenced what we’re looking at, what’s being taught in the curriculu m, and then where we have gaps, we’re purchasing additional curriculum resource s and materials. [Do you think that the skills and knowledge that yo u teach to help kids prep are for the state test, are those the important things that they need to know?] I think I would agree, and most of our teachers wo uld agree, and principals in the area agree. Principals believed that as a result of focusing the curriculum on important basic skills, fewer students are failing to learn those skills. As a principal in a suburban middle school explained: Some of those fluffy extraneous things [have been eliminated from the] curriculum—people just doing what they want to do because it’s fun. So I think that’s been a plus, and I think that we’re focusing on what ne eds to be done for kids to have their skills fo r future lifelong use. Focusi ng on the basics is one of them, and having students be able to be successful and function is a basic. Kids aren’t as easily dropping th rough the cracks as they used to be. So I think that’s important that we have kind of solidifie d some things that we need to address

PAGE 12

Education Policy Analysis Archives Vol. 13 No. 43 12 with students for their suc cess…And it definitely has focused more [attention] on the skills that the kids need. Interviewees in the majority group agreed that testing focused instruction on reading and math—the tested subjects—and felt that this fo cus over nontested subjects was appropriate. As a Principal in an urban middle school explained: Teachers do focus on reading and on ma th, and our students, approximately half of whom are special needs students becaus e they’re either speci al ed or English language learners new to th e English language, they have to learn it. . You have to learn how to read. You can’t learn anything if you don’t. Science teachers agreed with the focus on reading and math, welcomed the opportunity to integrate reading and math into science inst ruction, and asserted that integrated instruction in these areas was desirable. A high school science teacher in an urban school reported: It has [influenced the type of skills and knowledge that I focus on] because now I have to teach math and reading in my science classroom. I cannot just assume that by giving a textbook at the tenth grade reading level that these kids know how to read at the tenth grade reading level. [How do you feel about having to teach reading and math?] I love it! If I had my way, reading, writing, arithmetic would never have come out of anybody’s curriculum. It would be early childhood family education through grade 16. It woul d be in every single classroom, every single day. It would not be compartmentali zed. The thought is totally silly for me that it’s even fragmented; it makes no sense. Teachers in the majority group did not feel that they needed to focus on test-taking tricks and strategies to prepare students for the state test Instead, they asserted that they prepared students by teaching skills that are broadly us eful. For example, a Titl e I Reading Coordinator in an urban middle school said: “It’ s just general reading skills…wh at we focus on is just good reading instruction an d helping the kids with test-taking sk ills, because some kids really need help with that. Good te st-taking skills, ironically enough, are also good reading strategies.” All of the principals in this study asserted that they are opposed to teaching to the test— meaning isolated drills on the types of items expected on the test—and that their teachers understand this. The principals said that they would sanction any teacher caught teaching to the test. As a principal in an urban middle school put it: “W e don’t look at the test and focus on what’s on there, and if I ever find a staff member who does they’re in deep trouble.” This suggests that the principals in this study provided countervailing pressure against teaching to the test. Teachers in the majority group asserted that, in any case, drill is not useful in preparing students for the state tests. For example, a special education teacher in an urban high school said: I find with the math test in particular, the Basic Standards math test, drilling is very little help, because I see the math test is essentially a reading test. My kids have great difficulty compre hending what the question is. Drill to me is helpful when you’re drilling on th e same thing—for instance, addition facts over and over, or multiplication facts ov er and over. Well, that test doesn’t focus on drill at all, because each question is totally differe nt in the expectation or the skill that it requires. Drill to me means repetition and that isn’t one of the things in the math test. Interviewees in this group felt that testing did not interfere with their ability to provide a balanced educational experience for students. Teac hers and administrators felt that they could balance the need to prepare st udents for the test while main taining student interest and

PAGE 13

Limiting the Unintended Conse quences of High-Stakes Testing 13 encouraging students to be self -motivated learners. A curriculum coordinator in an urban middle school explained: We have to focus on the st rands in both reading and ma th. We don’t teach to it in this building directly although I know that there’s a balance, that you have to because it is high stakes. We’d be doing a disservice not to. But I think we try to balance the appropriateness of [instruction ]. We’re not going to teach the whole year from the strands primarily or so lely. [Do you feel that the skills and knowledge in the strands are important?] Yeah, I do. I feel they’re important… I think there’s a way to teach [the strands] to make it intere sting. And I think there’s a way to teach to interest a stud ent to become more of a self-learner. I think some programs take it to the extent that, “We’re going to design everything around the test and we’re go ing to teach to it ,” and I think yo u lose students easily in that type of en vironment. Whereas if you create an environment where you view the strands as what we’re going to focus on, and then create a unit or a project where they’re going to use those sk ills, I think that’s more of a creative way to approach it. A focus on critical thinking in elementary grades Of the teachers who felt (by a two-to-one margin) that testing had a positive impact on the cu rriculum, the comments of the elementary level teachers suggest that the more challenging MCA test administered in the elementary grades prompted a greater focus on critical thinking skills. For example, a 4th grade teacher in a semi-rural elementary school explained that the test was well-a ligned with her goals of teaching reasoning skills: Yes, very much so. [Do you feel that th e test focuses on important skills and knowledge?] Yes. I think it teaches children to look differently at a question, it teaches children how to answ er questions with reasoning. It’s not like a multiple guess or multiple choice; they actually ha ve to explain how they arrived at things, and I think that’s really g ood. I think more of it shou ld be done in classrooms. Elementary level teachers in the majority group be lieved that the influence of the state test on instruction was positive because it involves ch allenging multi-step pr oblems, and teachers are required to prepare students to reason through that type of problem. As a 3rd grade teacher in an urban elementary sc hool explained: I would say the state test is a very challenging test. I th ink that one th ing that my colleagues and I always comment on in third grade is that these are multi-step problems, when I think abou t the math. That’s a very challengin g thing for third graders. You begin to do one step problem s and you feel pretty good. Two steps, okay; beyond that it’s really challeng ing for them. I’m glad that they have exposure to that; it gives us a high mark to shoot for. The emphasis of the state test on complex, higher order thinking in the elementary grades presented a challenge for low-achieving students. A 3rd grade teacher in a semi-rural elementary school described the challenge: My perception is the test is geared more for the higher level thinker. It’s kind of stressful for the lower student. One, it de pends on if they can read well, because whether it’s math or reading, they have to read alone unless they have a special [education] plan. So readin g can enter into it. And then if they’ve got the organizational skills to pr epare themselves—how to se t up problems or methods. And then the actual thought process involved I think it’s difficul t. So I think it’s very difficult for the low en d student. I’m thinking of the ones not in special ed.

PAGE 14

Education Policy Analysis Archives Vol. 13 No. 43 14 However, I have to say with our practice te sting, they don’t se em to have anxiety at the time of testing, wh ich is good. They seem to just accept it and they’re proud of themselves… I think it’s design ed more for the higher level thinkers, not for the average third grade student. It gets complex pretty qui ckly on the test. Teachers felt that the challenging nature of th e state test has raised standards and improved outcomes. A 2nd grade teacher in a semi-rural elementary school described the effect on teaching and learning: We noticed some years ago that our math scores were generally low, so it’s like, “Okay, we’ve got to do something about this.” But we have really upped our standards for all our math. Whoa We have really raised our standards, and so our math scores are coming up. The focus of the test on critical thinking skil ls reportedly has had a po sitive influence on the curriculum in nontested subjects such as science. In both science and math, the curriculum now emphasizes ‘real life’ situations and higher or der thinking. A 5th–6th grade special education teacher in a semi-rural mi ddle school described the ch anges in the curriculum: Our curriculum has changed in science an d math so that we’re teaching the children life skills, real life situations, higher level thin king, math strategies, and science items… It seems to make more sense to them that they’re learning something that they ca n use in the future, rather than to just work on a page of multiplication facts. Need for segmented tests Interviewees who were critical of tes ting suggested a change in the design and implementation of state tests that could address multiple concerns: administering state tests in short segments spread throughout the school year. Segmented testing would allow the number of topics covered to be increased (because the total amount of testing could be increased), reducing the pressure to narrow the curriculum, wit hout forcing students to sit for long periods of time for any single segment. Multiple testing checkpoints throughout the school year could also improve the delivery of diagnostic f eedback to teachers, creating oppor tunities to help students who need assistance. Discussion The results of this study suggest that teache rs and administrators in the four Minnesota districts felt, by a two-to-one margin, that the overall impact of state-mandated testing in Minnesota is positive, consistent with survey results showing that 85 percent of teachers state-wide support the state’s exit exam (Draper, 2000). Within this majo rity group, teachers and administrators felt that the impact of the state tests on curriculum—foc using attention on basic skills at the 8th grade level and critical thinking skills at the el ementary level—was appropriate. Their reasoning explains their choices about what to teach in their classrooms and, thus, the nature of curriculum narrowing in Minnesota due to high stakes testing. Teachers and administrators reasoned that students need to know basic skills in ma th and reading as well as critical thinking skills in order to succeed in the world outside of school. In their judgment, the st ate-mandated tests are well-aligned with key instructional priorities and well-designed to avoid construct underrepresentation. In their view, Minnesota’s state-mand ated tests assess students on skills that students should know. Teachers in the majority group felt th at it is appropriate to emphasis these skills and to de-emphasize less important outcomes. This suggests that a properly aligned, well-designed testing system can avoid excessive narrowing of the curriculum.

PAGE 15

Limiting the Unintended Conse quences of High-Stakes Testing 15 The teachers’ claims that drill-oriented instructi on is not helpful in preparing students for the state-mandated tests suggest that the tests may be consistent with inquiry-based teaching that emphasizes the learner as an active constructor of knowledge. Whether the Minnesota teachers endorsed inquiry methods was not directly addressed in this study, but the evidence from this study does not support the hypothesis that state-mandated testing in Minnesota is inconsistent with an inquiry-oriented approach. Three additional features of Minnesota’s sys tem contributed to proper alignment and sound design. First, Minnesota’s unique testing system en couraged a focus on basic skills as well as critical thinking, maintaining breadth and depth in the curriculum. The high standards MCA exams offset pressure to narrow the curriculum to the basic sk ills tested on the MBST exit exam. In contrast, states that only use minimum competency tests inad vertently risk encouraging teachers to focus only on the basic skills needed to pass those tests. S econd, Minnesota’s decision to select a minimum competency exit standard resulted in a high cu mulative passing rate—over 99 percent—and low pressure on teachers to teach to the test. Thus, teach ers felt that they could prepare students for the exit exam without drilling students on the typ es of items expected on the exam. Third, administrators reportedly discouraged teaching to the test, providing countervailing pressure against any tendency to drill students on narrow test-focused skills. The implication of this study is that well-alig ned, well-designed tests that emphasize basic skills and critical thinking, plus minimum comp etency exit standards that reduce pressure on teachers to teach to the test, and administrators th at discourage teaching to the test, can avoid much of the curriculum narrowing that reduces the overall quality of education received by students in a high stakes testing environment. To the extent that excessive narrowing of the curriculum occurs in other states, the cause may be state-mandated tests that are not well-designed or well-aligned to the intended curriculum. The results of this study highlight the importan ce of aligning the design of state-mandated tests with the intended curriculum, and are consis tent with prior research regarding the importance of alignment (see Fuhrman, 2001). In some instances, this requires changing how tests are designed. In other cases, it requires changing state curri culum standards so that they are not excessively oriented toward factual knowledge and do not inad vertently lead to tests that emphasize recall of such knowledge. The results of this study also suggest the importance of not linking the compensation or job security of administrators to test scores through bonus systems and performance reviews because this may inadvertently undermine the motivation of administrators to provide the type of countervailing pressure noted in this study, possibly creating an environment where teachers feel intense pressur e to raise test scores. It may al so be desirable to explore the feasibility of segmenting state-mandated tests into several shorter tests and spreading the administration throughout the school year. Segmented testing would allow the number of topics covered to be increased, reducing th e pressure to narrow the curriculum. The main finding of this study—that it is possible to avoid excessive curriculum narrowing when stakes are high—may seem surprising if high stakes are equated with high pressure. However, this finding is consistent with prio r research that makes clear that stakes —formal consequences for students and/or schools that are linked to test results—are not synonymous with pressure — communications and informal consequences intended to induce staff to increase test scores (Corbett & Wilson, 1991). In a study of 300 school districts in two states, negative impacts on the curriculum occurred when stakes were high and pressure to raise test scores was high (Corbett & Wilson, 1991, p. 126). Positive impacts on teaching and learning occurred when stakes were high but pressure to raise test scores was low (Corbett & Wilson, 1991). High stakes and low pressure can occur when students must pass a test to graduate from high school but almost all students pass easily. This type of test is not likely to generate concern among staff or parents or pressure to change the curriculum.

PAGE 16

Education Policy Analysis Archives Vol. 13 No. 43 16 Under conditions of high stakes and low pressur e, schools responded in a positive way (Corbett & Wilson, 1991). Educators in these schools accepted th e test as a valid indicator of student learning and used it as the basis for making improvements in instruction while maintaining a balanced curriculum. Schools had adequate time to respond to student weaknesses indicated by test results. Under this condition, teachers implemented strategies for improving learning in a broad, balanced way that led to a rise in test scores without havi ng to teach directly to the test. Teachers were most likely to adopt, adapt or invent more effective inst ructional practices as the best means of improving student learning (Corbett & Wilson, 1991, p. 116). Thus, although there are clearly many factors that influence whether state-mandated tests have positive or negative effects, Corbett and Wilson’s study suggests that, in high-stakes situations, the degree of pressure that teachers feel to improv e student achievement is paramount. The current study suggests that teachers feel less pressure wh en state-mandated tests are well-designed and properly aligned with curricular and instructional priorities, the exit exam focuses on minimum competencies rather than high standards, and admi nistrators provide countervailing pressure against the urge to teach to the test. This framework can be extended to incorporate other factors, such as the dropout rate of minorities and ESL students—these factors may be understood as influencing the pressure that is placed on teachers to improve student achievem ent. A high dropout rate places more pressure on teachers. Similarly, teachers with a higher level of pedagogical content knowledge may feel less pressure to teach to the test, providing a countervailing force against pressures to narrow the curriculum. The Corbett-Wilson theoretical framew ork suggests why the impact of testing may be positive in one school district and negative in anothe r. It suggests, for example, that despite strong consequences for students, testing may have a positive impact on curriculum and instruction if countervailing factors reduce the pressure on teachers Conversely, a lack of countervailing factors may cause teachers to resort to drill and practice. The role of pressure in causing inadvertent narro wing is significant because many states are rushing to implement high standards exit exams that are likely to generate tremendous pressure on teachers to raise test scores (Gayler et al., 2004). The present study suggests that this move may be premature. Minnesota’s system of testing offers an alternative way to raise educational standards without creating excessive pressure on teachers to sa crifice the breadth and depth of the curriculum. State and federal policymakers can increase the probab ility that high stakes testing will have positive, rather than negative, effects on teaching and le arning by drawing upon Minnesota’s model and ensuring that tests are properly aligned and pressure on teachers is reduced. References Baker, E. L., & Linn, R. L. (2004). Validity issue s for accountability systems. In S. H. Fuhrman & R. F. Elmore (Eds.), Redesigning accountability systems for education (pp. 47–72). New York: Teachers College Press. Baron, J. B., & Wolf, D. P. (1996). Performance-based student assessment: Challenges and possibilities Chicago: University of Chicago. Bogdan, R., & Biklen, S. (1992). Qualitative research for education: An introduction to theory and methods Boston: Allyn & Bacon.

PAGE 17

Limiting the Unintended Conse quences of High-Stakes Testing 17 Bracey, G. W. (2002). The 12th Bracey report on the condition of public education. Phi Delta Kappan, 84 (2), 135–150. Cimbricz, S. (2002). State-mandated test ing and teachers' beliefs and practice. Education Policy Analysis Archives, 10 (2), retrieved October 26, 2005, from http://epaa.asu.edu/epaa/v10n2/. Corbett, H. D., & Wilson, B. L. (1991). Testing, reform, and rebellion Norwood, NJ: Ablex. Draper, N. (2000, February 3). Poll finds mo st teachers dislike Profile of Learning; An Education Minnesota survey also found general approval for the state's basic-skills tests Minneapolis Star Tribune, 3B. Elmore, R. F. (2004). The problem of stakes in performa nce-based accountability systems. In S. H. Fuhrman & R. F. Elmore (Eds.), Redesigning accountability systems for education (pp. 274–296). New York: Teachers College Press. Firestone, W. A., Mayrowetz, D., & Fairman, J. (1998). Perf ormance-based assessment and instructional change: The effects of testing in Maine and Maryland. Educational Evaluation and Policy Analysis, 20 (2), 95–113. Fuhrman, S. H. (Ed.). (2001). From the Capitol to the classroom : Standards-based reform in the states. One hundredth yearbook of the National Society for the Study of Education, Part II New York: The University of Chicago Press. Gayler, K., Chudowsky, N., Hamilton, M. Kober, N., & Yeager, M. (2004). State high school exit exams: A maturing reform Washington, DC: Center on Education Policy. Glaser, B. G., & Strauss, A. L. (1967). The discovery of grounded theo ry: Strategies for qualitative research New York: Aldine De Gruyter. Gordon, S. P., & Reese, M. (1997). High stakes te sting: Worth the price? Journal of School Leadership, 7 345–368. Grant, S. G. (2000). Teachers and tests: Exploring teachers' pe rceptions of chan ges in the New York state testing program. Education Policy Analysis Archives, 8 (14), retrieved October 26, 2005, from http://epaa.asu.edu/epaa/v8n14/. Grant, S. G. (2001). An uncert ain lever: Exploring the influenc e of state-level testing in New York State on teaching social studies. Teachers College Record, 103 (3), 398–426. Haney, W. (2000). The myth of the Texas miracle in education. Education Policy Analysis Archives, 8 (41). Haney, W., & Madaus, G. (198 9). Searching for alternatives to standardized tests. Phi Delta Kappan, 70 (9), 683–687.

PAGE 18

Education Policy Analysis Archives Vol. 13 No. 43 18 Hoffman, J., Pennington, J., Assaf, L., & Paris, S. (1999). High stakes te sting in reading and its effects on teachers, teac hing, and students: Toda y in Texas, Tomorrow? The Reading Teacher, 52 482–492. Jennings, J. F. (2000). Why national standards and tests? Politics and the quest for better schools Thousand Oaks, CA: Sage. Koretz, D., Barron, S., Mitchell, K., & Stecher, B. (1996). Perceived effects of the Kentucky Instructional Results Info rmation System (KIRIS) Santa Monica, CA: RAND. Koretz, D., Stecher, B., Klein, S., & McCaffrey, D. (1994). The Vermont portfolio assessment program: Findings and implications. Educational Measurement: Issues and Practice, 13 (3), 5–16. Kuhs, T., Porter, A., Floden, R ., Freeman, D., Schmidt, W., & Schwille, J. (1985). Differences among teachers in their use of curriculum-embedded tests. The Elementary School Journal, 86 (2), 141–153. Linn, R. L. (2004). Accounta bility models. In S. H. Fuhrman & R. F. Elmore (Eds.), Redesigning accountability systems for education (pp. 73–95). New York: Teachers College Press. Massell, D. (2001). The theory and practice of using data to build ca pacity: State and local strategies and thei r effects. In S. H. Fuhrman (Ed.), From the Capitol to the classroom: Standards-based reform in the States. One h undredth Yearbook of th e National Society for the Study of Education (pp. 148–169). New York: The Un iversity of Chicago Press. McNeil, L. M., & Vale nzuela, A. (2000). The harmful impact of the TA AS system of testing in Texas: Beneath the accountability rhetoric Houston: Department of Education and Center for Education, Rice University. McNeil, L. M. (2000). Contradictions of school reform: Educational costs of standardized testing New York: Routledge. Mehrens, W. A. (1998). Consequences of assessment: What is the evidence? Education Policy Analysis Archives, 6 (13), retrieved October 26, 2005, from http://epaa.asu.edu/epaa/v6n13/ Miles, M. B., & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage. Pomplun, M. (1997). State assessment and in structional chan ge: A path model analysis. Applied Measurement in Education, 10 (3), 217–234. Porter, A. C., Floden, R. E., Freeman, D. J., Schmidt, W. H., & Schwille, J. P. (1986). Content determinants (Research Series No. 179). East Lansing, MI: Michigan State University, Institute for Research on Teaching. Porter, A. C., & Smithson, J. L. (2001). Are content standards being implemented in the classroom? A methodology an d some tentative answers. In S. H. Fuhrman (Ed.), From

PAGE 19

Limiting the Unintended Conse quences of High-Stakes Testing 19 the Capitol to the classroom: Standards-based reform in the States. One hundreth yearbook of the National Society fo r the Study of Education (pp. 60–80). Chicago, IL: The University of Chicago Press. Rothman, R. (1995). Measuring up: Standards, a ssessment, and school reform San Francisco: Jossey-Bass. Rothman, R. (2004). Benchmarking and alignment of state standards and assessments. In S. H. Fuhrman & R. F. Elmore (Eds.), Redesigning accountabilit y systems for education (pp. 96– 114). New York: Teachers College Press. Rothman, R., Slattery, J. B., Vranek, J. L., & Re snick, L. B. (2002). Benchmarking and alignment of standards and testing (CSE Technical Report). Los Ange les, CA: National Center for Research on Evaluation, Standards, and Studen t Testing, University of California, Los Angeles. Shim, M. K., Felner, R. D., Shim, E., & Noonan N. (2001, April 10–14). Multi-dimensional assessment of classroom instructional practi ce: A validity stud y of the Classroom Instructional Practice Scale (CIPS). Paper presented at the Annual Meeting of the American Educational Research Association, Seattle, WA. Smith, M., & O'Day, J. A. (199 1). Systemic school reform. In S. Fuhrman & B. Malen (Eds.), The politics of c urriculum and testing (pp. 233–267). Ph iladelphia: Falmer. Smith, M. L. (1991). Put to the test: Th e effects of external testing on teachers. Educational Researcher, 20 (5), 8–11. Smith, M. L., Noble, A., Heinec ke, W., Seck, M., Parish, C., Ca bay, M., Junker, S., Haag, S., Tayler, K., Safran, Y., Penley Y., & Bradshaw, A. (1997). Reforming school s by reforming assessment: Consequences of the Arizona Student Assessment Prog ram (ASAP): Equity and teacher capacity building [CSE Technical Report 425] Los Angeles: National Center for Research on Evaluation, Stan dards, and Student Testing. Smith, M. L., & Rottenberg, C. (1991). Unintended consequences of external testing in elementary schools. Educational Measurement: Issues and Practice, 10 (4), 7–11. Strauss, A. L. (1987). Qualitative analysis for social scientists New York: Cambridge University Press. Wolf, S. A., Borko, H., Elliott, R. L., & Mc Iver, M. C. (2000). "T hat dog won't hunt!": Exemplary school change efforts within the Kentucky reform. American Educational Research Journal, 37 (2), 349–393.

PAGE 20

Education Policy Analysis Archives Vol. 13 No. 43 20 About the Author Stuart S. Yeh University of Minnesota Email: yehxx008@umn.edu The author is currently Assist ant Professor in the Department of Educational Policy and Administration at the Un iversity of Minnesota. His resear ch focuses on improved ways of designing assessment an d accountability systems, and he is writing a book that recommends changes in federal, state, and district level tes ting policies.

PAGE 21

Limiting the Unintended Conse quences of High-Stakes Testing 21 Appendix A Interview Questions 1. Overall, is your opinion about the impact of the state testing program positive or negative? Why? 2. Does the state testing program influence the type of skills and knowledge that you focus on? a. Are these skills and knowledge important? b. Is the test designed so you feel a need to drill students on factual material? 3. How does the state test affect low-functioning students? 4. What are the strengths and weaknesses of the computer-adaptive test, compared to the state-mandated test? 5. Recommendations. a. To guide instruction, would you favor re-d esigning the state tes t to provide more detailed diagnostic information about student strengths and weaknesses? b. Would you favor consolidation of all of the state and district tests into one test? c. Suppose this was done and suppose that you could receive scores within 7 days. Would you favor, at every grade level, giving the test before the school year, as a pretest to diagnose students, and at the en d of the school year, as a post-test to determine how much students improved during the year? d. Would you favor using the test to determine which students should receive afterschool tutoring? e. Should students who fail the end-of-the-year test be required to attend summer school as a condition of promotion to the next grade? f. Would you favor the use of a computer-adapt ive test that adapts the difficulty of the questions to the student’s level? g. Would you favor the use of a test that provided the option of performance assessments for low-functioning students, voc ed students, bilingual students, and special ed students, even if it meant that regular ed teachers might have to share the responsibility of administering and scoring the performance assessments? [Perhaps 2 days of work per teacher]. h. Would you favor state tests that focus on critical thinking rather than recall of facts? i. Until these changes are made, would you favor delaying the requirement that students pass the state test in order to graduate from high school? j. Are there other suggestions that you would make regarding the design or implementation of the state testing program?

PAGE 22

Education Policy Analysis Archives Vol. 13 No. 43 22 EDUCATION POLICY ANALYSIS ARCHIVES http://epaa.asu.edu Editor: Sherman Dorn, University of South Florida Production Assistant: Chris Murre ll, Arizona State University General questions about ap propriateness of topics or particular articles may be addressed to the Editor, Sherman Dorn, epaa-editor@shermandorn.com. Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Casey Cobb University of Connecticut Linda Darling-Hammond Stanford University Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Gene V Glass Arizona State University Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Anthony G. Rud Jr. Purdue University Michael Scriven Western Michigan University Terrence G. Wiley Arizona State University John Willinsky University of British Columbia

PAGE 23

Limiting the Unintended Conse quences of High-Stakes Testing 23 EDUCATION POLICY ANALYSIS ARCHIVES English-language Graduate -Student Editorial Board Noga Admon New York University Jessica Allen University of Colorado Cheryl Aman University of British Columbia Anne Black University of Connecticut Marisa Cannata Michigan State University Chad d'Entremont Teachers College Columbia University Carol Da Silva Harvard University Tara Donahue Michigan State University Camille Farrington University of Illinois Chicago Chris Frey Indiana University Amy Garrett Dikkers University of Minnesota Misty Ginicola Yale University Jake Gross Indiana University Hee Kyung Hong Loyola University Chicago Jennifer Lloyd University of British Columbia Heather Lord Yale University Shereeza Mohammed Florida Atlantic University Ben Superfine University of Michigan John Weathers University of Pennsylvania Kyo Yamashiro University of California Los Angeles

PAGE 24

Education Policy Analysis Archives Vol. 13 No. 43 24 Archivos Analticos de Polticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998—2003) Roberto Rodrguez Gmez Editorial Board Hugo Aboites Universidad Autnoma Metropolitana-Xochimilco Adrin Acosta Universidad de Guadalajara Mxico Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educacin, Chile Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educacin, Argentina Teresa Bracho Centro de Investigacin y Docencia Econmica-CIDE Alejandro Canales Universidad Nacional Autnoma de Mxico Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagoga Popular, Per Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernndez Enguita Universidad de Salamanca. Espaa Gaudncio Frigotto Universidade Estadual do Rio de Janeiro, Brasil Rollin Kent Universidad Autnoma de Puebla. Puebla, Mxico Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California Mara Loreto Egaa Programa Interdisciplinario de Investigacin en Educacin Mariano Narodowski Universidad To rcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Polticas Educativas, Per Vanilda Paiva Universidade Estadual Do Rio De Janeiro, Brasil Miguel Pereira Catedratico Un iversidad de Granada, Espaa Angel Ignacio Prez Gmez Universidad de Mlaga Mnica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de So Paulo Diana Rhoten Social Science Research Council, New York, New York Jos Gimeno Sacristn Universidad de Valencia, Espaa Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, Mxico Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas Publicas-Universidad de Buenos Aires, Argentina Antonio Teodoro Universidade Lusfona Lisboa, Carlos A. Torres UCLA Jurjo Torres Santom Universidad de la Corua, Espaa