xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20039999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00320
Educational policy analysis archives.
n Vol. 11, no. 22 (July 17, 2003).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c July 17, 2003
Local impact of state testing in southwest washington / Linda Mabry, Jayne Poole, Linda Redmond [and] Angelia Schultz.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 11issue 22series Year mods:caption 20032003Month July7Day 1717mods:originInfo mods:dateIssued iso8601 2003-07-17
1 of 35 A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright is retained by the first or sole author, who grants right of first publication to the EDUCATION POLICY ANALYSIS ARCHIVES EPAA is a project of the Education Policy Studies Laboratory. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education Volume 11 Number 22July 17, 2003ISSN 1068-2341Local Impact of State Testing in Southwest Washingt on Linda Mabry Jayne Poole Linda Redmond Angelia Schultz Washington State University VancouverCitation: Mabry, L., Poole, J., Redmond, L., Schult z, A. (July 18, 2003). Local impact of state testin g in southwest Washington. Education Policy Analysis Archives, 11 (21). Retrieved [date] from http://epaa.asu.edu/epaa/v11n22/.AbstractA decade after implementation of a state testing an d accountability mandate, teachers' practices and perspectives regar ding their classroom assessments and their state's assessments of student achievement were documented in a study of 31 teache rs in southwest Washington state. Against a background of national trends and standards of psychometric quality, the data wer e analyzed for teachers' beliefs and practices regarding classroom assessment and also regarding state assessment, commonalities and differences among teachers who taught at grade levels tested by the state and those who did not, teachers' views about the impact of state assessment on their students and their classrooms, and their views about whether state testing promoted educational im provement or reform as intended. Data registered (1) teachers' p references for
2 of 35 multiple measures and their objections to single-sh ot high-stakes testing as insufficiently informative, unlikely to promote valid inferences of student achievement, and often distor tive of curriculum and pedagogy; (2) teachers' objections to the state test as inappropriate for nonproficient speakers of English for students eligible for special services, and for impoverished students; and (3) teachers' preferences for personalized assessments respectful of student circumstances and readiness, rather than st andardized assessments. Teachers' practical wisdom thus appear ed more congruent than the state testing program with measu rement principles regarding (1) multiple methods and (2) v alidation for specific test usage, including usage with disadvant aged subgroups of test-takers. Findings contrasted a distinction of e mphasis: state focus on testing students" as distinct from teachers' focus on "tes ting students ." By 2001-02, standards and standards-based testing w ere being implemented in 49 states to evaluate school and student performance ( Meyer, Orlofsky, Skinner & Spicer, 2002, p. 74), all save Iowa where the state requires district standards (Neuman, 2002) and where it has been reported that virtually all school districts administer the Iowa Test of Basic Skills (ITBS) (Bo nd, Braskamp, van der Ploeg & Roeber, 1996; Mabry & Daytner, 1997). Formal purpos es for standards-based state testing programs typically include statements of in tent to improve student learning. For example, Substitute Senate Bill 5953 (SSB 5953) the origin of Washington state's current testing program, opens with these w ords: If young people are to prosper in our democracy and if our nation is to grow economically, it is imperative that the overal l level of learning achieved by students be significantly increased. To achieve this higher level of learning, the legislature finds that the s tate of Washington needs to develop a performance-based school system. . [T]he state needs to hold schools accountable for their performance b ased on what their students learn. . [I]t will be necessary to set high expectations for all students, to identify what is expected of all stude nts, and to develop a rigorous academic assessment system to determine if these expectations have been achieved. (Washington State Senate, 1992, pp. 1-2) A decade after implementation of this legislation, has Washington's accountability plan had the intended effect? Have the state conten t standards, the Essential Academic Learning Requirements (EALRs), and the sta ndards-based test, the Washington Assessment of Student Learning (WASL), i mproved student learning? With awareness that there have been few empirical s tudies of the effects of the standards movement nationally (Swanson & Stevenson, 2002) and with particular interest in the local context, an interview study o f 31 teachers was undertaken in 2001-02 to discover the impact of reform-oriented, standards-based state testing in southwest Washington, with emphasis on whether it h ad encouraged changes in classroom practices which promoted improved learnin g.Context
3 of 35 National education reformThe implicit theory of action underlying test-drive n accountability systems is that testing will improve student learning through provi sion of accurate data supporting valid interpretations of student achievement, with scores used to identify those who will receive rewards and sanctions, ultimately moti vating improved teaching and learning (Baker, 2002). This theory implies that te achers and students are extrinsically motivated, that test scores and the r ewards and sanctions they trigger are motivating in the manner intended, and that tea chers and students are not working as hard as they could and should (Elmore, 2 002). When the nodes in this chain of logic are examined in sequence (see Figure 1), it becomes clear that threats to any link in the chain can result in test ing that not only does not improve learning but may even be counterproductive. For example, what if test scores do not provide accurate data but if, as has sometimes been charged, the tests are biased agains t racial and ethnic minorities, females, or the poor? What if rewards and sanctions do not motivate teachers to improve teaching but, rather, motivate them to subv ert and distort their practice through teaching to the test or "multiple-choice te aching" (Smith, 1991, p. 10)? While the theory of action suggests the mechanisms and sequencing through which testing can improve teaching and learning, it simul taneously suggests the critical junctures at which testing can undermine teaching a nd learning. In high-stakes testing, theoretical implications ma tter much less than real-life implications. Empirical data indicate that scores d o tend to rise in the years following the implementation of a new test (Linn, 2 000), consistent with the theory of action. Washington state's test data also exhibi ts this trend (see Table 1), although not uniformly. But whether the higher scor es reflect increased student learning is unclear (Haladyna, Nolen & Haas, 1991; Mabry, Aldarondo & Daytner, 1999; Shepard & Smith, 1988; Smith & Rottenberg, 19 91). Are the scores accurate, and are they triggering appropriate consequences th at yield improved teaching and learning?Table 1 Trends in Washington state test scores, 1997-2001: Percentages
4 of 35 of students meeting state standards in reading, mat h, and writing, based on data available online at www.k12.wa.usTest subjects and grades Scores by years 1996-971997-981998-991999-20002000-01 Reading grade 447.955.659.165.866.1grade 7 38.440.841.539.8 grade 10 51.459.862.4 Mathematics grade 421.431.237.341.843.4grade 7 184.108.40.206.4 grade 10 33.035.038.9 Writing grade 442.836.732.639.443.3grade 7 31.337.142.648.5 grade 10 41.131.746.9 Bar graph based on fourth grade reading scores (first row in the table above), rounded to the nearest whole number, to visualize s core increases more clearly Scores 68 67 65.866.166 65 64 63 62 61 60 59.1 59 58 57 55.6 56 55 54 53 52
5 of 35 51 50 4947.9 48 47 46 Years1996-971997-981998-991999-20002000-01 The consequences of state testing in the U.S., wher e the stakes are high and getting higher, indicate widespread acceptanceÂ–at l east implicitlyÂ–of the theory of action. Currently, 43 states require school report cards (including Washington), with two more in development, and 20 of these require th at the report cards be sent home to parents. Twenty states (not including Washi ngton) have the authority to impose serious sanctions on low-performing schools: school closure or reconstitution, student transfers, and loss of fund ing; three more states will be able to do so within two years. Eighteen states (not inc luding Washington) provide rewards to high-performing or improved schools, wit h two more set to do so within two years. Fifteen states use test scores alone, wi th no additional evidence, to evaluate schools (Meyer, Orlofsky, Skinner & Spicer 2002). The difficulty charter schools are experiencing in trying to raise test scores (e.g., Gewertz, 2002) is heightening awareness that raisin g test scores in straightened educational circumstances is not easy. Perhaps beca use of this, test-triggered stakes are increasingly being borne by students who are relatively defenseless (Elmore, 2002). In particular, in seventeen states, a number that will increase by seven in the next two years, adolescents cannot gra duate from high school without passing exit or end-of-course exams (Washington wil l require a graduation test in 2008). An elementary or middle school child's promo tion to the next grade is contingent on test scores in four states (not inclu ding Washington), a number that will double in the next two years. Remediation is r equired for students failing promotion, end-ofcourse, or high school graduatio n exams in seventeen states, most but not all of which provide funds for the rem edial instruction (Meyer, Orlofsky, Skinner & Spicer, 2002).The newly reauthorized Elementary and Secondary Edu cation Act (2001), dubbed "no child left behind" (NCLB) and sometimes derisiv ely called "no child left untested," furthers the trend toward more state tes ting and higher stakes. Stakes include federal Title 1 funding and now, for undera chieving schools, requirements to provide school choice to parents in year 2 of a school's continuing low test scores, tutoring with parental choice as to provide rs in year 3, replacement of curriculum and/or staff in year 4, and reconstituti on in year 5. The basis of these sanctions is state test scores. In Washington, few schools currently meet NCLB standards: only 36 of 1162 elementary schools, 19 o f 554 middle schools, and 13 of 505 high schools ( Oregonian 2002). Superseding the Goals 2000 call for a national syst em of tests in 1994, (Note 1) NCLB requires increased state testing, including st andards-based assessments of reading and math for all students in grades 3-8. In order to receive Title 1 funds, the law requires attainment of proficiency by all s tudentsÂ–including minorities, students with limited proficiency in English, and l ow SES studentsÂ–within twelve
6 of 35 years and proportional annual yearly progress (AYP) in the interim. To discourage states from using easy tests that might distort ach ievement or lower expectations, the law also requires that scores on state tests be confirmed against scores on the National Assessment of Educational Progress (NAEP). (Note 2) The AYP targets are about double the score increase s empirically documented by NAEP over time, which suggest that it might take no t twelve but more than 100 years, by optimistic estimate, to reach the require d 100% proficiency. The AYP targets have been judged especially "unrealistic" f or schools and districts where small enrollments of disadvantaged subgroups of stu dents will result in statistically unstable results (Haertel, 2002). The targets also appear painfully unrealistic for chronically under-resourced urban schools (Lewis, 2 002; Yakimowski, 2002). National policy thus exhibits confidence in a theor y of action that is empirically suspect.State policyThe standards-based, test-driven educational reform initiative mandated by the legislature in SSB 5953 in 1992 lists four purposes for the state of Washington's accountability system: to assess students' academic learning to evaluate instructional practices to select students for remediation to hold schools accountable for student learning (W ashington State Senate, 1992, p. 10). These are very similar to the four purposes for ass essments recently listed by Shepard (2002)Â–diagnosis, monitoring, student selec tion, and program evaluationÂ–with the warning that making a test more valid for one purpose might make it less valid for a competing purpose. Frequen t similar admonitions from the measurement and evaluation communities indicate tha t multiple purposes for a single test are usually problematic, as different p urposes are often in unwitting conflict, undermining achievement of any of the goa ls (e.g., Mabry, 1999). For example, tests used for school accountability have often proved vulnerable to "score pollution" as school personnel administering the tests succumb to pressure to raise scores through a variety of means, some et hically, legally, or statistically questionable (Haladyna, Nolen & Haas, 1991; Haney, 2000; Linn, 2000; Sternberg, 2002).Score increases are not always credible, as evidenc ed by discrepancies between some state NAEP scores and scores on the state test (e.g., Haney, 2000) and by the so-called Lake Wobegon effectÂ–states' insistenc e that more than half of their students were "above average" (Cannell, 1987), a st atistical impossibility. As educators scramble to raise scores to protect their schools, students, and themselves from high-stakes penalties, improved sta te test scores may not necessarily reflect improved student achievement. I nflated scores would obstruct understanding of students' academic learning and wo uld obstruct identification of students needing remedial assistanceÂ–two goals of W ashington state's accountability system.
7 of 35 The Washington Assessment of Student Learning (WASL ) tests literacy and math at grades 4, 7, and 10 and offers multiple-choice a nd constructed-response items, both short and extended writings. Described as a cr iterion-referenced assessment aligned to state standards (Meyer, Orlofsky, Skinne r & Spicer, 2002, p. 75), the WASL is administered in late Spring. Student perfor mance is judged to be "above standard," "meets standard" (the required level of proficiency), "below standard," and "well below standard." In 2001, schools and dis tricts were required to reduce by 25% the number of students not meeting the state 's required standard and to include in public reporting their goals and plans t o do so (online at the state education agency's website, www.k12.wa.us). As note d, in comparison to some states, the stakes associated with the WASL are rel atively low: schools are not threatened with closure or reconstitution; funds ar e not withheld because of low scores; students in grades 4 and 7 are not retained at grade level or compelled into remedial education if they do not meet standards; h igh school students' eligibility for graduation will not be contingent upon WASL sco res until 2008 (Note 3) (Meyer, Orlofsky, Skinner & Spicer, 2002, pp. 74-75).Is Washington's testing program having the intended effect, assuring that "the overall level of learning achieved by students be s ignificantly increased"? State statistics generally suggestimproved achievement (s ee Table 1) but, as of 2001, national statistics indicated that less than a thir d of Washington's fourthor seventhgraders had scored at the "proficient" lev el on NAEP reading, writing, math, or science tests (Orlofsky & Olson, 2001). Of course, it might be that the scores reflect state learning goals but not nationa l learning goals. It might also be that the state's standards-based testing program is improving learning but not yet measurably since, elsewhere, indications have been found that state reforms are resulting in teachers' adoption of classroom practi ces consistent with standards (Swanson & Stevenson, 2002). Local evidence of teac hers' acceptance of Washington's state standards and of positively evol ving classroom practices, if occurring, might suggest gradual improvement which could become measurable in the future.The research reported here investigated the resonan ce between state testing and classroom assessments, whether feedback from the WA SL helps teachers understand their students' achievements and plan mo re effective learning opportunities, whether local classroom practices ar e changing, whether state testing is encouraging the alignment of curriculum to state standards and, if so, whether the alignment is educationally beneficial.MethodThe approach to the study undertaken in Fall 2001 w as qualitative, subscribing to a view of human phenomena as socially constructed (Vy gotsky, 1978) from individuals' perceptions of reality. The research p rocess adhered to interpretive research traditions and methods respectful of emerg ent design, multiple perspectives, and inductive analysis (Denzin, 1989, 1997; Denzin & Lincoln, 1994; Erickson, 1986; Mabry, 2002; Merriam, 1998; Stake, 1978; Wolcott, 1994). Two data collection methods were employed: review of do cuments (Hodder, 1994) related to testing in Washington state and, more im portantly, semistructured interviews (Fontana & Frey, 1994; Rubin & Rubin, 19 95) of practicing teachers in
8 of 35 the local area.After approval of the study by a university Institu tional Review Board and signed consent from each interviewee, graduate students at Washington State University Vancouver (Note 4) interviewed 31 local teachers in Fall 2001. The sa mpling strategy was purposeful rather than representative or randomized, with each graduate student identifying and interviewing two t eachers who taught a subject at a grade level of specific interest to the interview er. (Note 5) This subject selection strategy maximized the sensitivity of the interview ers to each teacher's subject area and grade level.Of the 31 teachers interviewed, 19 taught in high s chools, 5 in middle schools, and 7 in elementary schools (see Table 2). Their teachi ng experience totaled 547 years, with an average of 18 years each and a range of 1-4 0 years. Nineteen interviewees were female and 12 were male. All of the teachers' schools were located in southwest Washington state, and all but one of thes e was a public school. The teachers included 13 who taught subject areas and g rade levels tested by the WASL and 18 who did not. Of the teachers whose stud ents were tested on the WASL, 9 taught in high schools, 2 in middle schools and 2 in elementary schools.Table 2 Teachers interviewed, the subjects and grade levels they taught, and whether these subjects and grades levels were t ested using the WASL (n = 31)LevelSubject/gradeTested subject at this grade level? Teachers (by pseudonym) of this subject at this grade level and years of experience High school n=19, 9 in tested grades English-language arts* YESMs. Apple, 3 years Ms. Brush, 15 yearsMr. Carr, 7 yearsMr. Dustin, 20 yearsMs. Hand, 22 yearsMs. Kroner, 7 yearsMr. Twain, 25 yearsMs. Underwood, 20 years mathematics**YESMr. Alder, 19 years sciencenoMr. Liu (biology), 17 years Mr. Ming (biology), 17 yearsMr. Ochre (biology), 9 years
9 of 35 Ms. Vargas, 20 yearsMs. Walker, 20 yearsMr. Banks, 34 years family and consumer ed noMs. Crane, 30 years Ms. Doe, 14 years foreign languagenoMs. Good, 22 years social studiesnoMr. Inder, 1 year Middle school English-language arts YESMs. Frank, 12 years Ms. Nunn, 5 years n=5, 2 in tested grades historynoMr. Eggle, 25 years social studies and other noMs. Grant, 18 years grade 6no (private school) Ms. Smith, 16 years Elementary school grade 1noMs. Park, 16 years grade 2noMs. Hallo, 13 years n=7, 2 in tested grades grade 3noMs. Jones, 30 years Ms. Quinn, 40 years grade 4YESMs. Roberts, 21 years Mr. Exeter, 8 years grade 5noMr. Felix, 21 years One teacher taught English-language arts and also civics and philosophy. ** This teacher taught math and also P.E.A collaboratively constructed interview protocol (s ee Exhibit 1) guided semi-structured interviewing (Fontana & Frey, 1994; Rubin & Rubin, 1995). Interviews lasted approximately 45 minutes each. In terviewers attempted to capture
10 of 35 as many direct quotations as possible, with some in terviews tape-recorded with permission of the interviewees and others recorded in hand-written notes typed up soon thereafter. For purposes of developing a highquality database with strong internal validity (Campbell & Stanley, 1963) or des criptive validity (Maxwell, 1992), a comprehensive validation strategy (Mabry, 1998) w as used, with each interview written up and presented to the interviewee with a request for review, correction, and elaboration. Exhibit 1 Protocol for semi-structured interviewing of teachers How many years have you been teaching? What grade levels and subject areas have you taught?Has all of your teaching occurred in the state ofWashington? How do you assess your students' achievement? How did you develop your approach to student assessment? Why did you take this approach? Whatinfluenced your thinking? How long did it take todevelop? How has it evolved over time (if it has)? Have you had training in assessment? If so, how much training have you had? How would youdescribe the type of training you have had? Has you r assessment training been related to specific conten t areas? As the state has developed requirements for student learning and for assessing student achievement, hasyour teaching changed? If so, what has changedabout your teaching? Do you consider the changes to be improvements? How do you feel about the WASL (Washington Assessment of Student Learning)? Why? How do you prepare your students for your assessments (if you do)? How do you prepare themfor state assessments (if you do)? If you were to change your classroom assessments, what would you like to do differently? If stateassessments were to change, what type of changewould you favor? Does your school or district require testing (other than state testing)? Are tests part of your school' s or district's graduation requirements for high schoolstudents? Is there anything you would like to add? Thank you very much for your time and information! I will type up my notes from this interview and give them to you. I would very much appreciate it if you would read t he notes and make any corrections to improve accuracy. If th ere is anything you would to add at that time, I hope you will feel
11 of 35 free to make additions then. Again, many thanks Data analysis was emergent in character, with meani ng sought in the data without reference to a priori categories (Denzin & Lincoln, 1994; Erickson, 1986 ; Mabry, 2002; Wolcott, 1994). Analysis involved four phases and two validation efforts. In the first phase, pairs of graduate students analyze d their four interviews for patterns, including commonalities and distinctivene ss across their four subjects. This thematic content analysis (LeCompte & Preissle 1993; Miles & Huberman, 1994) and the resulting preliminary interpretations were written up in eight separate preliminary reports. In the next phase, the first a uthor conducted a similar content analysis across the eight student reports, identify ing 29 themes overall and grouping them in four emergent categories: (1) clas sroom impact, (2) student impact, (3) teacher impact, and (4) teachers' persp ectives (see Table 3).Table 3 Themes emerging from content analysis of teacher in terview data, identified from eight preliminary interview reports and grouped into four categories Interview reports Themes 12345678 A Classroom impact Teachers' approaches to classroom assessment XXXXXXXX Training in assessment for teachers XXXXXXXX Changes in assessments over time X X X Usefulness of the WASL for classroom practice X Impact of state standards/tests on curriculum and instruction (or resistance to impact) XXXXXXXX Preparation in class for the state test XXXXXX Impact of the WASL on classroom assessments X X Impact of the WASL on classroom environment X B Student impact Student accountability based on WASL scores (e.g., graduation, retention) X X X X Equity to students, including X Students whose first language is not English XX Special education students XX
12 of 35 Transfer students X Minority students XX X Students in difficult circumstances (including SES) X Impact of the WASL on students' selfesteem/stress/anxiety XX X X C Teacher impact School/teacher accountability based on WASL XXXXXXX Pressure to perform well on the WASL XXX Impact of the WASL on teacher professionalism X Contrasts of interest to us, including Public and private schools X Tested and non-tested grades/subject areas by the WASL X XXXXX Teacher assessments and state assessments XXXXXXXX D Teachers' perspectives Teacher approval or What teachers like about state testing X XX X Teacher disapproval or What teachers do not like ( or would change) about state testing XX XX X Questioning the constructs tested (or that should be tested) by the WASL XXX XXX Questioning the difficulty level of the WASL XX Scoring concerns X XX Questioning the expense of testing X (Note. An X indicates that data related to the theme (listed b y row) were found in the preliminary report (listed by column) in phase 2 of data analysis. Further review in phase 3 identified additional sources of data on these themes, revisions to these themes, and additional themes.)A third data analysis phase involved micro-review b y the authors of the entire data set for comprehensive identification of all data po ints related to each theme and category. The final phase of analysis was the ident ification, drafting, and formalization of findings. A draft of the resulting manuscript was offered for review
13 of 35 and critique to all 16 interviewers in a second val idation effort. The data and findings were structured for reporting according to the four major thematic categories. The teachers quoted are identi fied by pseudonyms.Classroom impactConsistent with other findings about the impact of standards-based reform (Swanson & Stevenson, 2002), the data made clear th at all of the teachers interviewed were highly aware of the state reform i nitiative and that state policy was definitely felt in local schools and classrooms. Ve teran teachers stood as witnesses to changes ushered in by the reform efforts. For ex ample, a teacher with forty years' experience observed, "Early in my career, th ere was very little emphasis on assessment. This has changed, most recently because of the state Essential Learnings" (Ms. Quinn). She, among others, indicate d that she had seen public school assessments evolve over the years from relia nce on intuitive teacher judgments to formal state standards, the Essential Academic Learning Requirements or EALRs.Response to state initiativeIn response to Washington's state standards and the state test, the WASL, most interviewed teachers had adjusted their classroom p ractice, they reported, some more than others. While most teachers said their in structional styles had not changed, many said the content they taught had alte red, consistent with other research indicating that teachers feel state framew orks are redefining curricula (Shore, 2002). Some teachers reported positively th at new state standards provided explicit objectives which helped focus the ir teaching. For example, one said, "It gives students and teachers a target" (Ms Frank). Another commented, "If you are going to try to do the job right, you shoul d always have [the EALRs] beside you so you can see what you're doing is on target" (Ms. Good). These teachers had approvingly accepted the EALRs as their teaching go als. A high school science teacher suggested that acceptance might sometimes h ave been compelled rather than willing: In general, [the WASL] is a good idea because it ha s forced people to be accountable. Many kids from middle school didn't have the basics, and we had to spend time re-teaching what they shou ld already have known. (Ms. Vargas) Some teachers expressed ambivalence or frustration regarding the superimposing of state goals over their own aims and approaches. For example, a high school teacher expressed hope that aligning her teaching t o the EALRs had "tightened up" her teaching but also said she found it frustrating when this forced elimination of her successful hand-crafted units: "ItÂ’s hard to di tch your pet projectsÂ” (Ms. Good). A third grade teacher who had previously taught the matically said she now taught subject-by-subject, with special attention to state content standards, a change she described not as an improvement but as "a necessity in the constantly changing world of education" (Ms. White).With the state graduation test postponed until 2008 classroom practices were less
14 of 35 affected at the high school level, according to the teachers, and less evident in content areas which were not yet tested by the WASL Even so, many high school teachers indicated strong impact of the test on the ir practices. For example, the only interviewee who baldly admitted to teaching to the state test was a high school teacher who said he had been directed to do so by h is principal because there was Â“a lot at stakeÂ” (Mr. Ochre). Greater impact was ap parent at the elementary level, particularly in fourth grade, a tested grade. At th is grade level, wholesale displacement of the curriculum was noted by some, i ncluding one fourth grade teacher who said, Â“Teachers in [my] building spend from about November to mid-April focused on the WASLÂ” (Ms. Roberts).Classroom assessmentsEvery teacher reported using a variety of assessmen t methods and techniques in the classroom. Often, these featured performance as sessment, their reported practices ranging from observations of student perf ormances to portfolios and projects. Variations among the teachers' assessment ideologies and practices suggested adaptations harmonious with personal styl e, the range surprising some interviewers. For example, Ms. Hand and Ms. Kroner were described by their interviewer as having "vastly different takes on wh at constituted appropriate assessment, yet both were excellent teachers who we re obviously very dedicated to their profession." Even teachers who taught the same subjects and grade levels approached assessment differently (e.g., Mr. Twain and Ms. Underwood, high school English; Ms. Vargas and Ms. Walker, high sch ool science). Suggesting adaptations based on experience and changes in stud ent populations, teachers consistently described continuous efforts across ti me as Â“constantly evolving each year as my class changes and the world around them changes" (Mr. Banks). Using assessment to ensure student success, rather than to identify weaknesses for remediation or penalty, emerged as an important distinction between classroom and state assessments in comments from some teacher s, such as: I want kids to be successful in my classroom. I'm n ot there to fail students. I'm there to teach students. [For] those with low academic abilities, if you put too much emphasis on testing, you would see a high failure rate. (Mr. Liu) Another who offered an earnest rationale for using assessment to improve achievement rather than to punish students said: Over my 17 years of teaching, I've really changed m y approach to student assessment. Initially, I started out really being worried about content. The value of their grade was based more on testingÂ–maybe 80%, 90%Â–less on what they did in the classroom, le ss on behavior. Over the 17 years, I've changed that. Maybe it's no t so important what they're learning but how they're going about doing it, how they're approaching what they're doing in class. I've shift ed my emphasis from content and techniques to behavior and work-related skills. . I want kids to be successful in the classroom. If I based [grades] strictly on content, I'd see too many kids failing. I think today more kids are
15 of 35 coming to my classroom without the tools needed to be successful in terms of learning the same level of content that I expected 17 years ago. So, should they get slapped again because they 're not prepared or not able to do what I expected 17 years ago? I don' t think so. . I think they come with a lot more baggage today tha n they did 17 years agoÂ–a lot more personal issues, parental guidance i ssues. . We're doing more parenting, and that's just as valuable. (Mr. Ming) Mr. Liu and Mr. Ming described their assessments as compassionately tailored to the realities of their students' troubled circumsta nces and consequent skill levels, adaptations not possible with the state's standardi zed test. Manty teachers spoke of efforts to personalize assessment, and many indi cated they wanted to implement even more personalized assessment but were prevente d by serious limitations on a critical resourceÂ–time.Overall, classroom impact data tended to agree with prior research indicating that testing is having so profound an impact in many cla ssrooms that reform is driving curriculum, a positive effect to the extent that th ere may now be "less fluff" but negative where pressure to raise test scores elimin ates flexibility (Horn, 2002) and focuses on scores rather than on students.Assessment training for teachers The teachers' preparedness to meet formal expectations regarding assessment (Washington State Senate, 1992; AFT, NCME & NEA, 1990) appeared to be uneven and inadequate, consistent with wider reports of insufficient teacher training in assessm ent (Hargreaves, Earl & Scmidt, 2002; Stiggins & Conklin, 1992). Most interviewees, although not all, considered their undergraduate assessment training inappropria te for classroom use. For example, two described their pre-service assessment training as Â“minimalÂ” (Ms. Vargas, Ms. Walker). Another said she had had only one assessment class in college, which proved unrelated to her content area and which she found to be Â“uselessÂ” for her own teaching (Ms. Grant).While two teachers reported no assessment training whatever since their initial teacher preparation and many said they had not take n post-graduate assessment courses, others described in-service training in as sessment as a Â“never-ending processÂ” (Ms. Jones) of classes, meetings, seminars and workshops for local educators and administrators. However, opinions abo ut the quality of professional development in assessment were sometimes no more po sitive than those regarding college and university assessment courses one teacher describing assessment training as "a huge inadequacy" (Ms. Nun n). Not only the adequacy but also the appropriateness of the training offere d to teachers emerged as suspect. Recent assessment training by one local di strict, some teachers said, emphasized writing WASL-like questions for implemen tation in their classrooms Â“to get [students] used to that type of assessmentÂ” (Ms Park), rather than understanding of measurement principles. In-service training had preempted rather than promoted teacher-developed assessments, report ed one teacher who said, Â“I never developed my own assessment techniques becaus e I was trained by the school and district in the way that they wanted ass essment doneÂ” (Ms. Apple). Some teachers identified their colleagues as a more important source of
16 of 35 assessment information that pre-service or in-servi ce training. Two teachers referred to assistance they had received from mento r teachers (Mr. Eggle, Ms. Frank), and one of these derided Â“new assessment id eas [as] just old ideas draped in new jargon that confuse and threaten older teach ers" (Mr. Eggle).Student impactThe teachers' awareness of the impact of the state test on their students was abundantly evident in their comments. Some indicate d that they considered student accountability a commendable state goal. One approv ing teacher, for example, said, "The WASL is a good thing to hold kids accoun table" (Ms. Park). A teacher with a less favorable view of the WASL nevertheless implied that more state testing for the purpose of making promotion and retention d ecisions was desirable, with a lament that "we do not have any exams to hold kids accountable for moving on to the next level" (Mr. Ming). Another teacher indicat ed preference for earlier and more frequent imposition of test-based consequences for students in commenting on the inappropriateness of delaying student accoun tability until high school graduation, saying that he considered it "odd" that the WASL "counts" only for tenthgraders (Mr. Carr). Most teachers, however, expressed serious concerns regarding the WASL's impact on students, as the com ments to follow indicate. Effects of the state test on student self-esteemOf the teachers interviewed, those who taught at a tested grade or in a tested subject and those who did not both expressed concer n regarding the impact of the WASL on student self-esteem. The teachers typically described environments in schools and in classrooms as highly charged during testing windows, one teacher referring to "a lot of stress for both the takers a nd the administerers of the test" (Ms. Good). Even a teacher who spoke favorably about the test warned that the WASL: does have too much pressure and overwhelms the stud ents. The scores affect their self-esteem. Nothing is in place [for students who don't meet] the standards. (Ms. Park) Developmental appropriateness Data indicated that few teachers considered the state test too easy relative to the content and exp ectations of their classrooms. One who did said, "[W]hat we expect from students is a lot harder than anything that the WASL tests for" (Ms. Apple). Much more common were concerns that the test was too difficult for some students, to the point of be ing "not developmentally appropriate for fourth graders. I've seen kids cryi ng about the nightmares they've had over this" (Mr. Felix). A fourth grade teacher expressed the greatest degree of concern about the test, saying: It is developmentally too difficult for [fourth gra de] students. They try so hard when they have no chance of passing. [In the w riting portion of the test,] to get a four [the highest score, requires a student to write] better than I could write. This is what they call raising standards. Raising standards means putting it beyond their development al level and hoping they are going to reach for it. We know that doesn' t work. (Ms. Roberts) The difficulty level of the fourth-grade test and p erceptions of its developmental
17 of 35 inappropriateness led some teachers to the conclusi on that the test was unfair. Perceptions of inequity were exacerbated for studen ts eligible for special services, for English language learners, and for students wit h low socioeconomic status. Effects of the state test on diverse and disadvanta ged students Testing special education students Some teachers suggested that, for special education students, test time was ill-spent because the WASL offered them "no chance" to demonstrate their knowledge and skills. Even a teacher who approved the test said, "I feel that special ed students sho uld not have to take the test . Their time could be better spent on more educationa l experiences" (Ms. Park). Another teacher objected: [M]y EMR, which is educable mentally retarded, stud ents have to take the WASL. Learning disabled students have to take t he WASL. . The EMR students are not going to be successful [on the test], yet we put them through 400 minutes of sweat when they could b e having other kinds of experiences. (Ms. Roberts) Few students could be exempted outright from taking the test, teachers said. However, accommodations were available for students classified as eligible for instructional assistance, but only if the accommoda tions provided during testing matched ordinary classroom accommodations. Although this general policy sounded reasonable in the abstract, specific restri ctions on accommodations rendered it useless, according to one teacher who s aid: If a person is learning disabled in writing, and we wanted someone to scribe for themÂ–take dictationÂ–[the student] would have to have that person all year long, every time we had a writing a ssignment. . It can't be just an accommodation for the WASL testing windo w. Individual students [would] be requiring a lot of time from ou r [teaching assistants], which we don't have. (Ms. Roberts) Other accommodations teachers thought might permit documentation of actual achievement were sometimes denied, as described by one teacher recalling a "special needs student [who] could sit down at the computer . [where] she had a way of expressing herself" but was not allowed to u se the computer when taking the WASL (Ms. Crane).These teachers' experiences of testing special educ ation students reflected the views of teachers nationally who have objected to s tate tests as merely providing a new way to show these students they are failures (H orn, 2002). Elsewhere, such perspectives have been brought to bear in legal act ion, for example, in the 1998 class action lawsuit charging the Indiana state tes t with unfairness to special education students.Testing English language learners The teachers described testing practices for English language learners as no better than those f or special education students. A middle school teacher observed: ESL students only get a one-year exemption from the WASL, which is
18 of 35 not nearly enough [time] to [become] familiar with the language, material, and culture to do well on the test. (Ms N unn) A fourth-grade teacher fretted: They test everyone including kids who have only bee n [in the U.S.] for a year and a half, so they're taking a test they cann ot read. . Even though you [might] say, "Oh, they can have assistan ce," the ESL kids [can only] have the problems read to them verbatim. (Ms. Roberts) The students would still have to write answers in E nglish. One teacher who declared that the WASL "doesn't wor k well when used to assess minorities or special ed students" raised a questio n of serious practical consequence: "What do we do with the students who c annot pass [the test] year after year and fail to advance?" (Mr. Ochre).Testing low SES students Teachers indicated that they considered students in straightened economic and personal circumstances in no less need of consideration than special education students and E nglish language learners. One teacher pointedly predicted, "I'm sure the WASL sco res will be best correlated to how much does your mom and dad make economically" ( Mr. Ming). In fact, as this teacher intuited, historically, the strongest corre late with scores has been socioeconomic status. The effect of socioeconomic s tatus on test scores was no small matter to the teachers interviewed. For the t hree-year period 1998-2000, 9-10% of the population in the state of Washington had been considered impoverished (U. S. Census Bureau, 2000). At the ti me of this study, Vancouver, in the Portland metropolitan area, was suffering from Oregon's highest unemployment rate in the U.S. (Preusch, 2001).Some teachers poignantly acknowledged increasing le vels of economic and social disarray in many families and the consequent calami ty in the lives of stricken students. One teacher worried about "how much assis tance and guidance do parents provide and are they abusive or intoxicated (Ms. Grant). No accommodations were available for students sufferin g the effects of these and other detriments to their real academic opportuniti es, and no consideration of such background variables were taken into account in cal culating their individual achievements as scores on the state test.Testing students with diverse learning styles Consistent with the popular theory of multiple intelligences (Gardner, 1983), some teache rs noted an inequity derived from discrepancies between the test (content and fo rmat) when compared to the different kinds of skills, achievements, and knowle dges students might actually possess. Like most standardized achievement testing the WASL emphasized "logicalmathematical" knowledge and skills over m ost other types of achievement. Within this theoretical context, these teachers imp lied that students with strong accomplishments in areas not included on the WASL w ere unfairly judged non-proficient by a state test that measured a rest ricted range of achievement.Teacher impact
19 of 35 Accountability pressuresMost interviewees explicitly recognized that Â“socie ty wants accountabilityÂ” (Ms. Good) and that Â“raising the standard would raise th e credibility of the American public school systemÂ” (Mr. Carr). They were keenly aware of public scrutiny of WASL scores published in local newspapers.Almost unanimously, more pervasively than has been reported nationally (Abrams, 2002), the teachers, even those who described thems elves as relatively unaffected by testing and test pressures, noted societal press ures related to scores and accountability. Fewer than one-fourth of the teache rs interviewed, most of these in untested grades, indicated that the WASL had little impact on them. One said that his plan for avoiding test-related demands was to r etire so that he would be Â“long goneÂ” before it was necessary for him to align his curriculum with the state test (Mr. Liu).No teacher in this study objected to accountability per se but several teachers expressed frustration at being held accountable for test results when student performance depended not only on teaching but also on factors beyond teacher control, factors they listed as including class siz e, student ability, primary language, eligibility for special services, socioeconomic sta tus, transience, family difficulties, and motivation. Â“Your teaching [will] eventually be judged by the kids who blow it off,Â” fumed one teacher (Ms. Good). Another recogni zed teacher vulnerability where Â“students perform badly on the WASL intention ally to make a pointÂ” of their own objections to the test (Mr. Dustin).While several teachers considered the impact of sta te testing meritorious, most expressed concern regarding the appropriateness of the state's prioritization of test scores in reckoning school accountability. This was consistent with other research findings that teachers do not oppose standards or a ccountability, but most disagree with current uses of test scores for school accredi tation (Abrams, 2002; Shore, 2002).Effects on classroom instruction and assessmentSome teachers in this study described classroom eff ects similar to reported trends indicating that state tests "deform curricula" (Sch oenfeld, 2002). The teachers identified such things as how to fill in the bubble s on answer sheets and how to follow prompts as examples of local WASL preparatio n activities which took time away from regular teaching and learning.The extent of curriculum displacement alarmed some teachers, one of whom said, "When we do the WASL, our school is in chaos for th e entire time. I lose a month of teaching. It affects the whole school" (Ms. Doe) Others reported that "test prep" consumed as much as five or six months in a tested grade. One teacher complained that the test overwhelmed classroom inst ruction even in untested grades: I guess I'm one of many teachers who feel there is so much emphasis on the WASL that it has almost become the focus of our teaching. I'm
20 of 35 not real comfortable with that. . [F]or the fou rth graders, the minute they enter fourth grade, they're hearing about the WASL and how they have to do well on the WASL. . But even at thir d grade, I find myself saying to students, "This is the type of question t hat you will have on the WASL when you are in fourth grade." . [W]e're j ust so test-oriented that we've kind of lost sight of what education tru ly is. (Ms. Quinn) Some teachers expressed resistance to reallocating instructional focus and time for test preparation, one saying, Â“I canÂ’t just prepare my students to take the WASL. ItÂ’s not the only thing that should be assessedÂ” (M s. Brush). Another, who complained that the WASL, a standardized test, "doe snÂ’t measure anything that we teach our kids," declared, "we are not willing to c hange because what we do for our kids is what they need" (Ms. Apple).But many fell into line, some with misgivings or un der duress. A high school science teacher, for example, had reluctantly added earth s cience to her curriculum because it was found in the EALRs, although it was outside her specialty (Ms. Walker). Another teacher reported a shift away from thematic instruction and toward a fragmented approach to the curriculum as s he Â“hit subject matter individually while constantly checking and re-check ing the EALRsÂ” (Ms. Jones). One reported her teaching was Â“becoming more canned Â” (Ms. Hallo). Another described herself as physically displaced in her ow n classroom, to some extent, by tutors brought in to ensure her students were prepa red for the WASL (Ms. Doe). The amount of class time devoted to external assess ment was not limited to preparation for and administration of the WASL. Tea chers reported at least eight additional standardized achievement tests, seven de veloped and marketed by big-name commercial testing corporations, in use in their schools or districts. Not only instructional practices, but classroom ass essment practices, too, were increasingly pressed into the WASL mold, data indic ated. For example, one teacher said she had reorganized her students' port folios Â“to match the EALRs.Â” As a member of her district's Â”assessment training tea m," led by an official from the state education agency, she was "learning how to wr ite a practice test similar in format to the WASL" as her district developed "a WA SL-like practice tests for second gradersÂ” (Ms. Park).Teacher perspectivesSlightly less than half of the teachers interviewed expressed approval of the WASL or of some aspects of it. Two praised the test's em phasis on "process," one adding that this emphasis was "good because we're trying t o make a more fair assessment" (Ms. Hand), and the other praising part ial credit given to students who showed workable math procedures even when answers w ere ultimately incorrect (Ms. Doe). The latter also approved the WASL's auth entic eliciting of "the same skills [students] use in real life" (Ms. Doe).Most expressions of approval included qualification s. For example, one teacher said the WASL was "probably a good thing" (Mr. Inde r), another that she felt positive "for the most part" (Ms. Frank), and anoth er that it was a good thing to hold students accountable although "improvements to the test" were needed (Mr. Alder).
21 of 35 One said the WASL "can assess some [students]. I th ink that it cannot assess all" (Ms. Roberts). Content limitations were noted by a teacher who observed that the test content was "not the only thing that should be assessed" (Ms. Brush). One teacher who approved the test distinguished between its quality and its utility: "I like the test. I just don't know how it should be used" (Mr. Carr). The most positive opinions were offered by two teac hers involved in developing either a practice WASL-like test or a rubric to sta ndardize the assessment of student writing. One of the two had been a member o f a district assessment team for four years, "so long it has become a part of me and I have begun to buy into it." Even so, her praise of the WASL was qualified: "The test is still new. The kinks have to be worked out" (Ms. Park). The other, who s aid she had helped develop Six Trait Writing Assessment, indicated that she had ch anged her teaching in response to the state standards, which she considered congru ent with her beliefs and practice, but that she disapproved of the WASL as much too narrow a device" (Ms. Underwood). This small (n=2) positive correlation b etween individuals' involvement with test development and their approval of the WAS L was consistent with findings from a nationwide study: [T]he promotion of greater receptivity towards chan ge at a local level, which might entail teacher knowledge about the refo rm, shaping attitudes toward reform objectives, or providing gr eater "howto" knowledge instrumental for implementing change. . appears to be a likely mechanism through which this policy reform o perates. (Swanson & Stevenson, 2002, p. 15) However, it was unclear whether local data suggeste d that teachers' close scrutiny of the testing system led them to appreciate the WA SL or, alternatively, suggested that involving teachers in development of standards and tests habituated and coopted them.Most teachers took issue with the test, the most vi rulent wording coming from one who called the WASL "stupid" (Mr. Banks) and anothe r saying, "I despise it" because of its counterproductiveness regarding lear ning and its deprofessionalization of teachers (Mr. Twain). Comp laints centered on negative impacts to curriculum, students, classrooms, and sc hools, as previously detailed, articulating questions and concerns about equity an d developmental appropriateness, as noted earlier, and about validi ty, scoring, expense, and volatile state policies and requirements, to be discussed in the next section. Teachers' objections also included lack of useful f eedback in the reporting of test results. "It would be nice for kids to get the test s back and see the mistakes that were made so that they could focus on their weaknes ses," said one of two teachers (Ms. Vargas) who objected to delayed notification o f WASL results. The other estimated the delay as "six months" after test admi nistration, too late for corrective instruction. When results did arrive, she complaine d, there were further obstructions to fulfillment of the state goal that testing help identify remediation needs: As far as I can tell, there has been no interpretat ion of what failing test results mean. . [And] I am not allowed to keep the test results. I am
22 of 35 only allowed to see them for a short time because t hey are locked up. I don't know if that's in every school or just in thi s school. (Ms. Roberts) One teacher objected to the content of a specific i tem, saying he had "lost respect" for the WASL after publicity about a tasteless ques tion that referred obliquely to a notorious trial involving a teacher's alleged seduc tion of a student (Mr. Ochre). Overall, local teachers' concerns about state testi ng closely matched those of their colleagues nationally: fairness, timeliness of feed back, diagnostic value of test result reports, single-shot testing, pacing in clas srooms, the number of tests, extraneous factors that affect scores, and pressure to cover all the standards (Shore, 2002).Validity concernsValidity through multiple measures Although no teacher used technical terms in responding to interview questions, analysis of data from the perspective of traditional psychometrics revealed strong practitio ner understanding of important measurement concepts and principles, particularly r egarding validity. Teachers made clear their intuitive understanding of the inj unction to use multiple measures in order to make valid inferences and decisions reg arding a student's achievement, as specified in the Standards for Educational and Psychological Testing : Standard 11.20. In educational, clinical, and couns eling settings, a test taker's score should not be interpreted in isolatio n ; collateral information that may lead to alternative explanations for the e xaminee's test performance should be considered. (AERA, AEA & NCME 1999, p. 117, emphasis added). Similarly, the standards for educational accountabi lity systems developed by the National Center for Research on Evaluation, Standar ds, and Student Testing (CRESST) and the Consortium for Policy Research in Education (CPRE) prominently and succinctly state, "Decisions about individual students should not be made on the basis of a single test" (Baker, Linn, H erman & Koretz, 2002, p. 3). The American Evaluation Association, in its first publi c policy pronouncement, has counseled against "simplistic application of single tests or test batteries to make high stakes decisions about individuals and groups [which] impede rather than improve student learning" (2002, unpaginated). The National Association for the Education of Young Children (NAEYC) has issued a po sition statement declaring: Decisions that have a major impact on children such as enrollment, retention, or assignment to remedial or special cla sses should be based on multiple sources or information and should never be based on a single test score. (NAEYC, 1988, emphasis added) The teachers interviewed spoke of their own multipl e measures as providing more accurate portrayals of their students abilities tha n the state test could provide. The WASL, said one, was merely "one window into a child for one week. As a teacher, I can tell you about their growth as a studentÂ” (Ms. Hand). All the teachers agreed that frequent and varied methods were needed to und erstand and represent accurately the diverse accomplishments of their stu dents.
23 of 35 Washington state relied essentially on the WASL, (Note 6) although the awareness of the importance of multiple measures was indicate d in such public statements as the following: No single test can tell you everything about a chil d's performance. Looking at information from a variety of tests and assessment tools remains the best way for parents and classroom teac hers to really see how well individual students are learning. (Office of the Superintendent of Public Instruction website, www.k12.wa.us, June 20, 2002) Construct validity Some of the teachers interviewed explicitly chall enged the WASL's construct validity, questioning whether the test did, in fact, test what it purported to testÂ– the construct of student achievement For example, one teacher said, "There is too much confusion about what it is actually trying to measure" (Ms. Nunn). When scores reflect things besides the inten ded construct (i.e., rival constructs ), test results can be misleading, either exaggerat ing achievement or denying due credit.The math problem-solving section of the WASL was pe rceived by some interviewees as troublesome on these grounds, requi ring students to explain their solution procedures. Teachers reported that many st udents who were good at math but weak in writing were unfairly penalized. Said o ne teacher, "Even if they can explain their thinking and they have the answer rig ht, they get marked down because of their writing skills" (Ms. Hallo). Probl ems related to rival constructs were not limited to writing requirements in the math tes t. Teachers suggested several rival constructs actually being measured rather tha n (or in addition to) the intended construct, student achievement, in saying: Rival constructÂ–socioeconomic status of individual students : "I'm sure the WASL scores will be best correlated to how much does your mom and dad make economically. . Socioeconomic stat us is the greatest predictor of student success." (Mr. Ming)Rival constructÂ–personal difficulties : "There are other variables that go into testingÂ–a baby, a job, living in their cars. T hese affect test performance." (Ms. Hand)Rival constructÂ–intelligence : "I firmly believe that WASL performance is not only affected by teaching but also [by] cogniti ve abilities which, to a certain extent, are innate." (Mr. Exeter) WASL scores are used not only as measures of the ac hievements of individual students but "to evaluate instructional practices" and "to hold schools accountable for student learning" (Washington State Senate, 199 2, p. 10). For this reason, the validity of inferences on the construct of school or educational quality emerged as relevant in the analysis. Several teachers' comment s indicated realization that a school's test results might indicate not the qualit y of its educational program delivery but, rather, the characteristics of its st udent body, including student motivation and especially affluence: Rival constructÂ–student motivation : "[Some] students perform badly on
24 of 35 the WASL intentionally to make a point.Â” (Mr. Dusti n) Rival constructÂ–socioeconomic status of school popu lation : "[A school in my district] traditionally has been at the top but, since we've redistricted, they had a huge influx of students from the lower e chelon housing and economic development. That has changed their dynami cs. They didn't do as well as they had hoped [on WASL scores]. . [S]tudents who are socioeconomically deprived don't do as well." (Ms. Roberts) Content validity In describing the test as "much too narrow a devi ce" (Ms. Underwood), one teacher implied that not only const ruct validity but also content validity was at issue, that the content of the test did not sufficiently represent the content of the intended domain (e.g., the English-l anguage arts test did not fully represenent the domain of English-language arts).Instructional validity Relatedly, some teachers indicated that instructi onal validityÂ–the match between what is taught and what is testedÂ–was faulty, one observing that the WASL "doesnÂ’t measure anything t hat we teach our kids" (Ms. Apple). Another teacher complained that test conten t was insufficiently aligned with the curriculum: I like the fact that people are accountable for tea ching certain curriculum, but the assessment part needs work. The re is a lot of mismatch between the curriculum and what the WASL i s testing. (Ms. Walker) Scoring concernsConcerns about the scoring of the state test were a lso raised. Interpreting a student's written explanation requires professional skill, experience, knowledge of child development, and sometimes knowledge of the p articular child, according to one teacher who said: The WASL is graded by people with no idea of knowin g what good communication is for that child. There's a greater possibility for a disconnect that's unfair for the student. (Ms. Unde rwood) The importance of accurate interpretation of text g enerated by children was not limited to tests of reading and English-language ar ts. As noted earlier, there were also concerns that students' math achievements migh t not be fully credited because of the scoring of verbal explanations: "One could be good at math but can't explain their thinking. They would be judged as not passing the test" (Ms. Park).Two teachers complained that some schools were inap propriately penalized because of regulations related to student scores of zero. One reported that the state had required GED students be classified as so phomores and prohibited them from taking the WASL, then had counted the "lack of scores" from these students against her school, the county's GED school, artifi cially lowering the school's results (Ms. Apple).
25 of 35 Test expenseA few teachers expressed concern regarding the cost of testing, one preferring a "standardized test which is cheaper and faster" (Ms Roberts) than Washington's current standards-based (and standardized) test wit h its performance assessment sections. Another hoped "the state isn't wasting mi llions of dollars" (Mr. Alder). Changing state policies and requirementsSome teachers approved the WASL and expected that t esting would always be part of the educational system, but one worried abo ut the diversion of resources to the WASL if it proved merely to be "some fad that w on't be around long" (Mr. Alder). Another expected no more: The WASL is just another one of those things that's going to come, and it, too, shall pass. I haven't changed what I teach or how I teach because, as a conscientious professional, I've look ed at what students should know in terms of biology. (Mr. Ming) In fact, changes to state accountability and testin g policy have been enacted "almost every year" (OSPI website, www.k12.wa.us, J une 20, 2002) since SSB 5953 in 1992 (see Table 4). Frequent changes, creat ing layers of increasing and sometimes conflicting requirements, can be seen acr oss the country as state testing programs have increased during the last dec ade, partly in response to federal requirements regarding Title 1 funding, and in the new federal requirement to test all children in grades 3-8 every year. "Pol icy hysteria" (Stronach & Maclure, 1996) is a term which has been given to frequent, o verlapping policy changes in general (i.e., not necessarily related to testing).Table 4 Summary of state statutes regarding Washington's ed ucation reform initiativeyearbilleffect 1992SSB 5953established the framework for education reform and the Commission on Student Learning (expired 1999), prov iding for the development of the Essential Academic Learn ing Requirements (EALRs) and a new assessment system. 1993ESHB 1209resulting from work by the Governor's Council on Education Reform and Funding (GCERF), established new learnin g goals and Student Learning Improvement Grants (SLIG ) and other programs to help educators help students meet new standards. 1994ESHB 2850established requirements pertaining to character traits and values. 1995SSB 5169made relatively minor changes to prior law.
26 of 35 1997ESB 6072established a timeline for assessment d evelopment. 1997ESHB 2042established a grade 2 reading assessme nt. 1998ESHB 2849required district school boards to est ablish reading improvement goals. Also, a grade 4 NRT was moved to grade 3. Also, the legislature provided funds for profess ional development, instructional materials, and schools w ith reading programs involving volunteer mentors. 1999ESHB 5825made changes to the NRTs and modified the assessment implementation timeline. 1999SSB 5418established the Academic Achievement an d Accountability Commission, established mathematics goals, and crea ted several new assistance programs. 2002ESB 6456authorized the A+ Commission to set per formance improvement goals for all students (e.g., economically-disadvantaged students, limited Englis h proficient students, students with disabilities, an d students from disproportionately underachieving racial and e thnic backgrounds) and to establish high school graduatio n rate goals and dropout reduction goals for grades seven through twelve. Source: Website of the Office of the Superintendent of Public Instruction, State of Washington, www.k12.wa.us, June 20, 2002FindingsVariations among the perspectives of the 31 teacher s interviewed signal continuation of a robust collective struggle to und erstand and improve education. The variations also evidence the kind of diversity and local control which many have considered traditional strengths of American s chooling. The contrasts were so dramatic that two interviewers were "not shocked bu t stymied" in trying to analyze the range of opinion expressed by the four teachers they had interviewedÂ–perceptions of the state test ranging f rom approval to ignorance to objection, perceptions expressed with a range of em otions from candor to arrogance to wariness, perceptions varying as to wh ether the teachers' own assessment practices should follow state mandates o r personal beliefs. Several interviewers expressed surprise that teache rs were not more negative about state testing but, instead, that some had off ered positive comments or described the state test as a tool to help their te aching. Other interviewers were taken aback by teachers' deep distress about the te st and its implications, two interviewers writing, "We feel as overwhelmed as th e teachers." Overall, the teachers in this study, like teachers across the co untry (Shore, 2002), appeared to be adapting and trying to make things work. From th e data they provided, four main findings emerged.(1) The teachers did not fear accountability but oppos ed accountability based on a
27 of 35 single-shot test. Their opposition reflected better understanding of the important principle of multiple measures than was manifested in the state accountability policy. Teachers' intuitive, experiential understan dingÂ–sometimes referred to as "practical wisdom"Â–appeared to be stronger in this regard than the formal understanding of state officials and their testing contractors and consultants who had implemented a test-driven accountability system with heavy reliance on the WASL.(2) The WASL was not appropriate for children who were eligible for special services, who were non-proficient speakers in Engli sh, or who were living in impoverished or marginal situations according to teachers who worked with them day-to-day. Teachers indicated that the state test ensured that these children would not only be left behind but also pressured and puni shed for factors beyond their control. Individual student scores aggregated and r eported as school scores similarly pressured and punished teachers for facto rs beyond their control said some.(3) Teachers repeatedly claimed classroom assessments were more informative but sidelined by the state tests One teacher, for example, referred to the WASL as Â“one measurement done during a short period of time that provides a little glimpse of the students, [whereas] I have them all year so I have a better perspective on them" (Ms. Hand). Teachers already understood the m essage researchers have been trying to share with policymakers, for example : Once-per-year accountability tests can't do the job of day-to-day, week-to-week pupil diagnosis. . What large-scale assessment can't do is document in sufficient detail the what and how of student understandings. (Shepard, 2002) Policy-makers need to support the development of ne w assessments and to avoid reliance on single tests. They should shift resourc es from large-scale assessment to classroom assessment. (Pellegrino, 2002)(4) While some teachers appreciated the focus provided by state standards and testing, other teachers were troubled by the test's replacement of teachers' professional judgment : The WASL goes against everything we know about lear ning and takes assessment out of the hands of educators and puts it into the hands of a corporate organization out for profit. (Mr. Twain)The WASL is robbing me of my professional judgment and replacing learning with inappropriate practices. (Ms. Quinn)If "inappropriate practicesÂ” are the result of stat e testing, teachers should resist. Although some have blamed teachers' insufficient re sistance for the current wave of high-stakes testing (Popham, 2001), some teacher s in this study indicated staunch resistance to Washington's state test withi n their classrooms, their clearest spheres of influence and the location of their prim ary responsibilities. High stakes testing represents a mechanism to ensur e local compliance to policy initiatives typically described as "reform." Effort s to comply were evident in this study. It nevertheless seems unlikely that centrali zed, top-down, state control can
28 of 35 lead to better education, as implied by the term "r eform" (see Fullan, 1991; Sarason, 1990) when it simultaneously deprofessiona lizes teachers by usurping their authority and opportunity to plan and impleme nt educational opportunities for their students. In a postmodern era skeptical of gr and plans and centralized management, it is worth considering whether forcibl y turning teachers into technicians, a return to the previous century's "te chnological perspective" for controlling education (Hargreaves, Earl & Schmidt, 2002) or "technicist approach" for making education efficient (Gillman, 2002), is more likely to re-form education in a detrimental rather than in an improved manner.ConclusionDuring the data analysis phase of this study, Washi ngton state superintendent Terry Bergeson publicly and plaintively remarked th at, as a former school counselor she was not initially an advocate of large-scale te sting, but "we need data" (2001). Four months later, a district administrator from Ka nsas City complained, "We're drowning in data but parched for informationÂ–and th e questions are cosmic" (Wright, 2002). This study suggests that teachers, who face cosmic questions in the microcosms of their classrooms, are a source of information that policy-makers would be well-advised to heed.It is no small matter that more than two-thirds of U.S. teachers consider their state tests not worth the investment (Abrams, 2002) and t hat some teachers are leaving the profession because of test pressure (Gillman, 2 002). Teachers are crucial to educational reform not only for the well-known reas on that top-down mandates succeed only with bottom-up buy-in from implementer s (Fullan, 1991; Sarason, 1990). In addition, teacher perspectives are key to implementing reasonable accountability (Shore, 2002) because it is teachers who bring together understanding of children, their achievements, and how to assess them. Teachers' understanding of assessment, despite deprivation of strong formal training, has been too long underestimated. The data clarified a critical difference of emphasis: teachers' focus on "testing students and state or external focus on testing students."Moreover, understanding teachers' experiences and p erspectives helps to explain research findings regarding "perverse incentives" r elated to state tests, such as teachers' unwillingness to accept or keep positions in low-scoring schools that most need their expertise and energies (Trent, 2002; see also Lankford, Loeb & Wyckoff, 2002). At a time when teacher shortages and high tu rn-over rates are a matter of concern in Washington state, careful consideration is needed in policymaking circles regarding the impact of test-driven account ability on teacher recruitment, retention, and job satisfaction.NotesOnly 19 states ever reached full compliance ( Education Week April 17, 2002, p. 29), and the national system of tests was never developed. 1. Prior to NCLB, NAEP was voluntary for states. 2. The date for making passing the WASL a graduation r equirement has been extended to 2008. 3.
29 of 35 For permission to use their interiview data and for review of a draft of this manuscript, the authors wish to thank Kevin Crouch, Candace Dawson, Patrick Dowell, Daniel Getty, Jeff Herzog, Stephen Klauer, Karissa Lowe, Jennifer Megli, Mark Muckerheide, Mary Nelson, Wayn e Storer, Debra Tidd, and Chad Towe. The authors also thank Marv Alkin of UCLA for review and comments regarding a draft of the article. 4. Since each graduate student interviewed two teacher s, there should have been an even number of teachers in the sample. Howe ver, by chance and without realizing it, two students chose and interv iewed the same teacher. 5. The state also mandated administration of the Iowa Test of Basic Skills (ITBS) in grade 3 reading and math and in sixth gra de reading, language arts, and math; and of the Iowa Test of Educational Devel opment (ITED) in grade 9 reading, language arts, math, and an interest inv entory. (Source: www.k12.wa.us ) 6.ReferencesAbrams, L. (2002, April). Multi-state analysis of t he effects of state-mandated testing programs on teaching and learning: Results of the national surv ey of teachers. Paper presentation to the annual meeting of the American Educational Research Associ ation, New Orleans, LA. American Evaluation Association Task Force on High Stakes Testing. (2002). Position statement on high stakes testing in preK-12 education. Fairhaven, MA: AEA. American Federation of Teachers, the National Counc il on Measurement in Education, and the National Education Association. (1990). Standards for Teacher Competence in Educational Ass essment of Students Washington, D.C.: Authors. Baker, E. L. (2002, April). Validity issues for acc ountability systems. Paper presentation to the annu al meeting of the American Educational Research Associ ation, New Orleans, LA. Baker, E. L., Linn, R. L., Herman, J. L., & Koretz, D. (2002). Standards for educational accountabilit y systems. CRESST Line Winter, 1-4. Bergeson, T. (2001, December 6). Washington reform update and implications for the future. Presentation to the Washington State Assessment Con ference, Seattle, WA. Bond, L. A., Braskamp, D., van der Ploeg, A., & Roe ber, E. (1996). State student assessment programs database, school year 1994-95 Oak Brook, IL: Council of Chief State School Offi cers and the North Central Regional Educational Laboratory. Campbell, D. T. & Stanley, J. C. (1963). Experimental and quasi-experimental designs for res earch Boston: Houghton-Mifflin. Cannell, J. J. (1987). Nationally normed elementary achievement testing in America's public schools: How all 50 states are above the national average. Educational Measurement: Issues and Practice 7 (2), 5-9. Denzin, N. K. (1989). The research act: A theoretical introduction to soc iological methods (3rd ed.). Englewood Cliffs, NJ: Prentice Hall. Denzin, N. K. (1997). Interpretive ethnography: Ethnographic practices fo r the 21st century Thousand Oaks, CA: Sage. Denzin, N. K. & Lincoln, Y. S. (1994). Handbook of qualitative research Thousand Oaks, CA: Sage. Education Week (2002, April 17). 1994 ESEA: The state of state co mpliance. Authors, p.. 29. Elmore, R. (2002, April). Stakes for whom? Paper pr esentation to the annual meeting of the American Educational Research Association, New Orleans, LA.
30 of 35 Erickson, F. (1986). Qualitative methods in researc h on teaching. In M. C. Wittrock (Ed.), Handbook of research on teaching (3rd ed., pp. 119-161). New York: Macmillan. No child left behind (NCLB), reauthorization of the Elementary and Secondary Education Act, Public Law 107-110 (2001). Fontana, A. & Frey, J. H. (1994). Interviewing: The art of science. In Denzin, N. K. & Lincoln, Y. S. (Eds.), Handbook of qualitative research (pp. 361-376). Thousand Oaks, CA: Sage. Fullan, M. (1991). The new meaning of educational change New York: Teachers College Press. Gardner, H. (1983). Frames of mind: The theory of multiple intelligence s New York: Basic Books. Gewertz, C. (2002, April 10). Low-scoring charter s chool to shut down in Chicago. Education Week p. 4. Gillman, C. (2002, April 26). From blooming flowers to marching soldiers: A case study of one kindergarten teacher. Colloquium, Washington State University Vancouver. GOALS 2000: Educate America Act, Public Law 103-227 (1994). Haladyna, T. M., Nolen, S. B., & Haas, N. S. (1991) Raising standardized achievement test scores and the origins of test score pollution. Educational Researcher 20 (5), 27. Haertel, E. H. (2002, April). Technical considerati ons in the use of NAEP to confirm states' achieveme nt gains. Paper presentation to the annual meeting of the American Educational Research Association, New Orleans, LA. Haney, W. (2000). The myth of the Texas miracle in education. Education Policy Analysis Archives 8 (41). ( http://epaa.asu.edu/epaa/v8n41/ ) Hargreaves, A., Earl, L., & Schmidt, M. (2002). Per spectives on alternative assessment reform. American Educational Research Journal 39 (1), 69-95. Hodder, I. (1994). The interpretation of documents and material culture. In Denzin, N. K. & Lincoln, Y S. (Eds.), Handbook of qualitative research (pp. 403-412). Thousand Oaks, CA: Sage. Horn, C. (2002, April). Multi-state analysis of the effects of state-mandated testing programs on teac hing and learning: Results of the fieldwork studies. Pap er presentation to the annual meeting of the American Educational Research Association, New Orle ans, LA. Lankford, H., Loeb, S., & Wyckoff, J. (2002). Teach er sorting and the plight of urban schools: A descriptive analysis. Educational Evaluation and Policy Analysis 24 (1), 37-62. LeCompte, M. D. & Preissle, J. (1993). Ethnography and qualitative design in educational r esearch (2nd ed.). San Diego: Academic Press. Lewis, S. (2002, April). What will be the effects o n assessment and accountability in local school dis tricts of the "no child left behind" legislation? Presenta tion to the annual meeting of the National Council of Measurement in Education, New Orleans, LA. Linn, R. L. (2000). Assessments and accountability. Educational Researcher 29 (2), 4-16. Mabry, L. (2002). In living color: Qualitative meth ods in educational evaluation. In D. Nevo & D. L. Stufflebeam (Eds.), International Handbook of Educational Evaluation Boston: Kluwer-Nijhoff. Mabry, L. (1999). Portfolios plus: A critical guide to alternative as sessments and portfolios Thousand Oaks, CA: Corwin Press. Mabry, L. (1998). Case study methods. In H. J. Walb erg & A. J. Reynolds (Eds.), Evaluation research for educational productivity (pp. 155-170). Greenwich, CT: JAI Press. Mabry, L., Aldarondo, J., & Daytner, K. (1999). Loc al administration of state-mandated performance assessments: Implications for validity. Paper prese ntation to the annual meeting of the American Educational Research Association, Montreal, Canada. Mabry, L. & Daytner, K. G. (March, 1997). Statema ndated performance assessment. Paper
31 of 35 presentation to the annual meeting of the American Educational Research Association, Chicago, IL. Maxwell, J. A. (1992). Understanding and validity i n qualitative research. Harvard Educational Review 62 (3), 279-300. Merriam, S. B. (1998). Qualitative research and case study applications in education San Francisco: JosseyBass. Meyer, L., Orlofsky, G. F., Skinner, R. A., & Spice r, S. (2002). The state of the states. In Quality counts 2002: Building blocks for success (a report on education in the 50 states by the Edi torial Projects in Education). Education Week 21 (17), 68-92. Miles, M. B. & Huberman, A. M. (1994). Qualitative data analysis: An expanded sourcebook (2nd ed.). Thousand Oaks, CA: Sage. National Association for the Education of Young Chi ldren. (1988). NAEYC position statement on standardized testing of young children 3 through 8 years of age, adopted November 1987. Young Children 43 (3), 42-47. Neuman, S. (2002, April). The Bush accountability a nd assessment agenda: New opportunities and challenges. Presentation to the annual meeting of t he National Council on Measurement in Education, New Orleans, LA. Orlofsky, G. F. & Olson, L. (2001). The state of th e states. In Quality counts 2001: A better balance (a report on education in the 50 states by the 2001 Ed itorial Projects in Education). Education Week 20 (17), 86-92, 94-100, 102106. Oregonian (2002, November 29). Nearly all Washington school s fail to hit goals. Portland, OR: Authors, p. B2. Pellegrino, J. (2002, April). Assessment and learni ng: Issues highlighted in the NRC report "Knowing what students know." Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA. Pomplun, M. & Capps, L. (1999). Gender differences for constructed-response mathematics items. Educational and Psychological Measurement 59 (4), 597-614. Popham, W. J. (2001). The truth about testing: An educator's call to acti on Alexandria, VA: Association for Supervision and Curriculum Development. Preusch, M. (2001, December 15). National briefing Northwest: Oregon unemployment rate rises. New York Times p. 14 Rubin, H. J. & Rubin, I. S. (1995). Qualitative interviewing: The art of hearing data Thousand Oaks, CA: Sage. Sarason, S. B. (1990). The predictable failure of educational reform: Can we change course before itÂ’s too late ? San Francisco: Jossey-Bass. Schoenfeld, A. (2002, April). "This is just a test! Paper presented at the annual meeting of the Amer ican Educational Research Association, New Orleans, LA. Shepard, L. A. (2002, April). Building bridges betw een classroom and large-scale assessments. Paper presented at the annual meeting of the American Edu cational Research Association, New Orleans, LA. Shepard, L. A. & Smith, M. L. (1988). Escalating ac ademic demand in kindergarten: Counterproductive policies. Elementary School Journal 89 (2), 135-145. Shore, A. (2002, April). Optimizing the validity an d value in the public debate over testing as a tool in educational reform. Paper presentation to the annua l meeting of the American Educational Research Association, New Orleans, LA. Smith, M. L. (1991). Put to the test: The effects o f external testing on teachers. Educational Researcher 20 (5), 8-11.
32 of 35 Smith, M. L. & Rottenberg, C. (1991). Unintended co nsequences of external testing in elementary schools. Educational Measurement: Issues and Practice 10 (4), 7-11. Stake, R. E. (1978). The case study method in socia l inquiry. Educational Researcher 7 (2): 5-8. Sternberg, R. J. (2002). The "Janus principle" in p sychometric testing: The example of the upcoming SAT-I. The Score newsletter of American Psychological Association Division 5, Evaluation, Measurement, and Statistics, 24 (2), 3Â–5. Stiggins, R. J. & Conklin, N. F. (1992). In teachers' hands: Investigating the practices of classroom assessment Albany, NY: SUNY Press. Stronach, I. & Maclure, M. (1996). Mobilizing meani ng, demobilizing critique? Dilemmas in the deconstruction of educational discourse. In Cultural Studies (vol. 1, pp. 259276). Greenwich, CT: JAI Press. Swanson, C. B. & Stevenson, D. L. (2002). Standards -based reform in practice: Evidence on state policy and classroom instruction from the NAEP state asses sments. Educational Evaluation and Policy Analysis 24 (1), 1-27. Trent, W. (2002, April). The policy implications of federally mandated annual testing. Paper presented at the annual meeting of the American Educational Rese arch Association, New Orleans, LA. U. S. Census Bureau. (2001). Percent of people in p overty by state. Online at http://www.census.gov/prod/2001pubs/p60-214.pdf Vygotsky, L. S. (1978). Mind in society: The development of higher mental p rocess Cambridge, MA: Harvard University Press. Washington State Senate. (1992). Washington Substit ute Senate Bill 5953: Act relating to education. Olympia, WA: Authors. Wolcott, H. F. (1994). Transforming qualitative data: Description, analysi s, and interpretation Thousand Oaks, CA: Sage. Wright, D. D. (2002, April). Who did we miss, and w hy? Factors associated with non-participation of general education students in standardized assessme nts. Paper presented at the annual meeting of the American Educational Research Association, New Orleans, LA. Yakimowski, M. (2002, April). What will be the effe cts on assessment and accountability in local schoo l districts of the "no child left behind" legislation ? Presentation to the annual meeting of the Nationa l Council of Measurement in Education, New Orleans, L A.About the AuthorsLinda Mabry is an associate professor at Washington State Univ ersity Vancouver, where she specializes in assessment of student achi evement, program evaluation, and qualitative research methodoloby, and a member of the boards of the American Evaluation Association and the National Ce nter for the Improvement of Educational Assessment.Jayne Poole is a graduate student in Education at Washington S tate University Vancouver, where she is researching the readingwr iting connection, and a kindergarten teacher of eleven years in Longview, W ashington. Linda Redmond a recent Masters in Education graduate of Washing ton State University Vancouver, has taught in the public scho ols of Washington state for twenty-two years and is currently an elementary mus ic specialist in Longview, Washington.
33 of 35 Angelia Schultz a newly certificated teacher with a BA in English and a graduate student in Education at Washington State University Vancouver, currently works as a substitute teacher. Her interests include the con sequences of high stakes testing and testdriven accountability. The World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu Editor: Gene V Glass, Arizona State UniversityProduction Assistant: Chris Murrell, Arizona State University General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, firstname.lastname@example.org or reach him at College of Education, Arizona State Un iversity, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: email@example.com .EPAA Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Linda Darling-Hammond Stanford University Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman California State UniveristyÂ–LosAngeles Richard Garlikov Birmingham, Alabama Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute ofTechnology Patricia Fey Jarvis Seattle, Washington Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Los Angeles Michele Moses Arizona State University Gary Orfield Harvard University Anthony G. Rud Jr. Purdue University Jay Paredes Scribner University of Missouri
34 of 35 Michael Scriven University of Auckland Lorrie A. Shepard University of Colorado, Boulder Robert E. Stake University of IllinoisÂ—UC Kevin Welner University of Colorado, Boulder Terrence G. Wiley Arizona State University John Willinsky University of British ColumbiaEPAA Spanish Language Editorial BoardAssociate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico firstname.lastname@example.org Adrin Acosta (Mxico) Universidad de Guadalajaraadrianacosta@compuserve.com J. Flix Angulo Rasco (Spain) Universidad de Cdizfelix.email@example.com Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho dis1.cide.mx Alejandro Canales (Mxico) Universidad Nacional Autnoma deMxicocanalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State Universitycasanova@asu.edu Jos Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es Erwin Epstein (U.S.A.) Loyola University of ChicagoEepstein@luc.edu Josu Gonzlez (U.S.A.) Arizona State Universityjosue@asu.edu Rollin Kent (Mxico) Universidad Autnoma de Puebla firstname.lastname@example.org Mara Beatriz Luce (Brazil) Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.br Javier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mx Marcela Mollis (Argentina)Universidad de Buenos Airesmmollis@filo.uba.ar Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mx Angel Ignacio Prez Gmez (Spain)Universidad de Mlagaaiperez@uma.es Daniel Schugurensky (Argentina-Canad) OISE/UT, Canadadschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil) American Institutes forResesarchÂ–Brazil (AIRBrasil) email@example.com Jurjo Torres Santom (Spain) Universidad de A Coruajurjo@udc.es Carlos Alberto Torres (U.S.A.) University of California, Los Angelestorres@gseisucla.edu
35 of 35 EPAA is published by the Education Policy Studies Laboratory, Arizona State University