USF Libraries
USF Digital Collections

Educational policy analysis archives


Material Information

Educational policy analysis archives
Physical Description:
Arizona State University
University of South Florida
Arizona State University
University of South Florida.
Place of Publication:
Tempe, Ariz
Tampa, Fla
Publication Date:


Subjects / Keywords:
Education -- Research -- Periodicals   ( lcsh )
non-fiction   ( marcgt )
serial   ( sobekcm )

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
usfldc doi - E11-00411
usfldc handle - e11.411
System ID:

This item is only available as the following downloads:

Full Text
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20049999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00411
0 245
Educational policy analysis archives.
n Vol. 12, no. 62 (November 09, 2004).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c November 09, 2004
There is another way : the faculty-developed Idaho Comprehensive Literacy Assessment for K-8 Pre-Service Teachers / David Squires, George F. Canney [and] Michael S. Trevisan.
x Research
v Periodicals.
2 710
Arizona State University.
University of South Florida.
1 773
t Education Policy Analysis Archives (EPAA)
4 856

xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 12issue 62series Year mods:caption 20042004Month November11Day 99mods:originInfo mods:dateIssued iso8601 2004-11-09


EDUCATION POLICY ANALYSIS ARCHIVES A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright is retained by the first or sole auth or, who grants right of first publication to the Education Policy Analysis Archives. EPAA is a project of the Ed ucation Policy Studies Laboratory. Articles are indexe d in the Directory of Open Ac cess Journals ( Volume 12 Number 62 November 9, 2004 ISSN 1068-2341 There is Another Way: The Faculty-developed Idaho Comp rehensive Literacy Assessment for K-8 Pre-Service Teachers David Squires Idaho State University George F. Canney University of Idaho Michael S. Trevisan Washington State University Citation: Squires, D., Canney, G. F. & Trevisan, M. S. (2004, November 9). There is another way: The faculty-developed Idaho Comprehensive Literacy Assessment for K-8 pre-service teachers. Education Policy Analysis Archives, 12 (62). Retrieved [date] from Abstract In this era of external teacher testi ng with the intent of ensuring the competence of the teaching force, as well as holding students and institutions accountable for results, the Idaho Comprehensive Literacy Assessment (ICLA) stands in sharp contrast. It represents an alterna tive to external testing of preservice teachers, testing procured from an outside agency unconnected to preservice programs, since it has been developed and is managed by literacy faculty from Idahos major institutions of higher learning. This paper provides a brief history of major events in the field of literacy including teacher testing initiatives and policies, which led to the creation of the ICLA. A description of the ICLA assessment and its construction is provided, along with a report of initial performance. Implications and po licy consequences of this approach are explored.


Squires, Canney & Trevisan: There is another way 2 Regardless of what one thinks of the policy, teacher testing has arrived at the national and State policy scene and will likely remain for the foreseeable future (Ludlow, Shirley, and Rosca, 2002). The No Child Left Behind A ct of 2001 (Public Law 107-110) mandates competent teachers in all classrooms and th at prospective teachers demonstrate this competence before entering the teaching force. States have logically moved to establish teacher tests to determine the competence of pre-service teach ers. In addition, with the reauthorization of Title II of the Higher Education Act in 1998 (Public Law 105-244), teacher education programs must annually re port the performance of pre-service teachers on a variety of state tests. Teacher testing has become a mainstay in teacher education and certification. Youngs, Odden, and Porter (2003) provide a snapshot of state pre-service testing practices for the 2001-2002 academic year, show ing the extent to whic h teacher testing has expanded in a relatively short time. Thirty-seve n of 50 states employ basic skills tests for licensure, 33 states require tests in content know ledge, and 26 states require pedagogy tests. Given the momentum of this policy across states, and national legislation authorizing its implementation, the number of states with teacher testing policies is undoubtedly growing. Despite the momentum for teacher testing the policy is not without justifiable criticism. As an example, Ludlow (2001) iden tified sub-standard psychometric characteristics of teacher tests in Alabama and Massachusetts For Alabama, the consequence of poor test development practice was severe. The court di rected Alabama to pay large compensation packages to individuals who showed that poor test development adversely impacted many test takers, and that in some cases, the test discriminated against minority groups. Teacher testing in Massachusetts received considerable attention. The highly politicized and publicized teacher testing re sulted in firestorm of controversy (Flippo, & Riccards, 2000; Haney, Fowler, & Wheelock, 1999; Ludlow, Shirley, and Rosca, 2002). Apparently, the content of the test is unconnected to good teaching or what teachers actually do (Ludlow, Shirley, & Rosca). In addition, al though the Massachusetts test is lauded as a mastery test, norm-referenced test developm ent procedures were used, undermining the mastery intent of the test (Haney, 1999). The end result is the mis-measurement of teacher competence in Massachusetts. The inflammatory language used by State policy makers to describe those who failed the test and institut ions that performed poorly is disappointing and illustrates the challenges ahead as states continue to impose teacher testing. It is clear that the intent of these teacher tests is to hold students and the system accountable. The accountability approach relies upon an external test, a test developed by an external agency, that requires certain pass scores for students and pass rates for institutions. One real consequence is the possibility of public shaming of teacher education programs that do not meet pass-rate expectations; another is sanctions, which threaten the very existence of the programs. In response, for example, many institutions in Massachusetts conduct a variety of program initiatives in an attempt to boost pass rates (Ludlow, Shirley, & Rosca, 2002). Some of the initiatives have a ctually worked. However, it is unclear whether higher pass rates on tests considered unconnected to teaching, indicate that more competent teachers are entering the teaching force.


Education Policy Analysis Archives Vol. 12 No. 61 3 A Focus on Reading While discussions and debates about the role and place of teacher testing continue, other issues in teacher education have also em erged. One such issue is reading instruction. The focus on literacy occurred as a natural outcome of numerous investigations and reports detailing poor reading achievement in the United States and proposals for intervention. Correspondingly, teacher preparation in readin g has been subject to considerable debate and discussion. A resonant theme from the debate about re ading is that too many students cannot meet basic literacy standards. Numerous reports have called for reform. The first of these reports came in 1983 when the National Commission on Excellence in Education reported to the Secretary of the Department of Education and to the nation that student achievement in reading, math, and science had declined since. In the well known report titled, A Nation At Risk, the National Commission on Excellence in Ed ucation claimed that students in the United States experienced shorter school days, shorter school years, less homework, and lower graduation requirements than did students in many other developed nations. They urged the Nation to set higher educational standards and provide the resources that would assure that the United States would remain globally competitive. In 1990, then President Bush and the Nations governors established six national educational goals for the year 2000. Goals 4 and 6 called for literacy to become a top priority for all students K-12 (Executive Office of the Pr esident, 1990). Under President Clinton, the number of goals was expanded to eight (Allia nce 2000 Project, 1994), including the focus on reading. In 1997 Congress asked the Director of th e National Institute of Child Health and Human Development, in consultation with the Secretary of Education, to form a panel to synthesize research on reading instruction. Th is National Reading Panel (2000) analyzed research in the areas of alphabetics, fluency, comprehension, teacher education and reading instruction, and computer technology and readin g instruction. The report was controversial, further complicating the discussion and debate about reading and how best to achieve reading outcomes. Moreover, serious questions were raised about the trustworthiness of the report itself. Yatvin (2002), a member of the pane l, was one of the panels critics. She relates a disturbing narrative of events that culmina ted in the final report being published without adequate proofing. Camilli, Vargas, and Yurecko (2003) replic ated the National Reading Panels metaanalysis of phonics instruction. After arriving at conclusions different from those in the report, they raised questions regarding Nati onal Reading Panels methodology and the link between the evidence and the Panels conclusions. Recent No Child Left Behind legislation (Public Law 107-110, 2001) reiterated recommendations from previous reports. In additi on, it requires that every teacher be highly qualified in the subjects taught, including readin g. Paraprofessionals would need to earn the equivalent of an Associate Arts Degree. Clos e monitoring of the progress of individual students would become necessary in order to id entify potential at risk learners, intervene instructionally, and measure the effects of instruction on an ongoing basis. The need to interpret these assessments and respond instructionally highlighted the importance of teachers and paraprofessionals having an in depth knowledge of beginning reading instruction and the decoding process.


Squires, Canney & Trevisan: There is another way 4 A second strand in the debate surrounding reading instruction comes from researchers and educators in literacy educat ion. In response to the National Commission on Excellence in Education (1983) report, A Nation at Risk the National Academy of Education Commission on Reading (1985) published Becoming a Nation of Readers This report stated that reading is a complex process and not a series of isolated sub-skills. This report was cautious about the recommendations from the National Commission on Excellence in Education report. Becoming a Nation of Readers reasserted that teachers have a greater impact on literacy learning than any instructional approach, set of materials, or structural change, a recurring theme still advanced today. In 1990, Adams published the seminal work Beginning to Read: Thinking and Learning about Print This extensive review of the research on reading supported the recommendations in Becoming A Nation of Readers (1985). Systematic phonics instruction, while essential, should be accompanied by reading quality childrens liter ature. In addition, as prerequisite skills for phonics, phonemic and phonological awareness are essential to early success in reading. Eight years later, Snow, Burns and Griffin (1998) edited Preventing Reading Difficulties in Young Children which summarized an analysis of 30 yea rs of scientific research on reading instruction and causes of reading failure in young children. Recommendations from this review mirrored those of Adams (1990) by callin g for systematic instruction in reading along with wide reading and writing both in and out of school. The position taken was that such instruction benefits not only students at ri sk in reading, but all students as well. A related body of research has examined the effects of teach er decision-making on reading achievement. Findings have consis tently shown that teachers who make good decisions regarding instruction are more effective than any single instructional strategy deemed best practice (Duffy & Hoffman, 19 99; Pearson, 1997; Ruddell, 1997). Good instructional decisions coupled with good pedago gical strategies are the most effective ways to teach children to read. Duffy and Hoffman ca ll for more flexibility in public educational policies that value teachers decision making. Because of the growing interest in teacher education as a critical component in systemic reform efforts, particularly as it pertains to reading instruction, the International Reading Association established the National Commission on Excellence in Elementary Teacher Preparation for Reading Instruction, commonly known as the IRA Commission (Maloch, Fine, & Flint, 2002). The IRA Commission identified teacher education program features that lead to excellent reading instruction and student reading achievement. These features include: A clearly articulated institutional mission Committed faculty High standards and multiple measures used for selection, monitoring, and support Strong emphasis on current literacy theory and best practices Faculty modeling instructional elements they want to see in their students. Well supervised field experiences The fostering of professional identity Faculty exercising autonomy An interesting first year finding by the IRA Commission was that teachers who had graduated from one of these programs responded to their students needs in flexible, knowledgeable, and strategic ways. In point of fact, these are the types of teachers that research has demonstrated are the most effecti ve (Duffy & Hoffman, 1999; Ruddell, 1997).


Education Policy Analysis Archives Vol. 12 No. 61 5 In sum, numerous reports have documented the perceived and real need for increased student reading achievement and e nhanced teacher preparation. These reports have called for changes in the time allocated to literacy instruction, curriculum development, instructional strategies and materials, allocation of human resources for remedial instruction, and an increased use of both summative and formative assessments. Because there is considerable research that suggests teacher d ecision-making, along with sound instructional approaches, are likely the most important facto rs impacting student reading achievement, teacher preparation programs have reexamin ed the content and pedagogical knowledge emphasized in their literacy courses. There has been a substantial shift in focus, consequently. Phonological, phonic and structural analysis knowledge, accompanied by renewed support for direct, systematic appr oaches for teaching beginning reading, characterize many reading methods courses required of pre-service teachers today. The brief historical review here is not meant to be comprehensive. It is illustrative of the division between policy makers and educators. Policy makers focus on structural change and the use of particular reading strategies to improve reading. Educators cite research showing that teacher decision making, along wi th good instructional strategies, provide the best hope for increased reading achievement. Id aho embraced the latter set of ideas as the State sought to improve reading achievement. Idaho Comprehensive Literacy Initiative With the national focus on literacy intensifyi ng, Idaho, like many other States, began to examine in greater depth the reading perf ormance of its K-8 students. A study of 938 fourth grade students in 41 schools across Id aho found that, depending upon which of six common reading measures were employed, from 21% to a high of 62% of all students tested read below grade level (Canney, 1998; 1999). Of equal concern was the finding that among students qualifying for special assistance (Reading Recovery, Title I, Limited English Proficiency, and Migrant), from 42% to 100% of the students assessed were not reading on a fourth grade level, depending upon the group and the assessment measure. These findings motivated the development of legislation affecting the content and focus of reading instruction K-8. Teams of teachers, administrato rs, university faculty, and interested citizens convened to develop standards for reading instruction and recommendations for best practice (Idahos MOST, 2001). Seeing pre-service education as a key com ponent in systemic State education reform policy, the Idaho Legislature also called for the creation of the Idaho Comprehensive Literacy Assessment (Idaho Statute 33-1207A, 1999), which pre-service elementary and special education teachers would have to pass to certify after August 2002. To recertify, K-8 teachers and principals would have to eith er pass the same assessment or successfully complete a three-credit coursethe Idaho Comprehensive Literacy Course. Efforts to assess teacher knowledge of best practices in reading had already begun in two other States. In California, pre-service tea chers applying for a teaching credential after September 1998 need to pass the Reading Instruction Competence Assessment (RICA), available in a written form or in a video performance form. Four areas are targeted: instruction based assessment; phonological and lin guistic processes; comprehension; oral and written language development. In the wri tten form candidates examine actual student


Squires, Canney & Trevisan: There is another way 6 artifacts to write long-range instructional plans (Carlson, 1999; California Commission on Teacher Credentialing, 1998). Texas began its pre-service teacher-testing program in 1986. The most recent version is a set of assessments known as Texas Examinations of Educator Standards (TExES). The TExES is made up of 33 separate assessmen ts that test a variety of content areas and specialties (State Board of Educator Certificat ion, 2003). There are four English Language Arts/Reading assessments within this set of 33 assessments. While TExES remains under development, each K-12 pre-service teacher must pass two TExES assessments, one on pedagogy, the second in the pre-service teacher s area of expertise (M. Janysek, personal communication, September 11, 2003). Interest in Idaho for measuring pre-se rvice teachers literacy knowledge and performance was informed by the Standards for Professionals in Reading developed by the International Reading Association, Professional Standards and Ethics Committee (1998). The document contains 16 knowledge, performance, and disposition standards for reading professionals. The Standards became an important guide for the committee developing core standards for initial teacher preparation in the language arts. The Standards also influenced the Idaho Comprehensive Literacy Assessment committees efforts to create an assessment that included both a measure of pre-service teachers knowledge of research-based best practices in reading, and their ability to apply this information to classroom teaching situations. The Idaho Comprehensive Liter acy Assessment committee made a deliberate move to develop an assessment that reflects the knowledge and performance recommendations of the IRA Commission and what research in reading instruction has shown to be most effective. The Idaho Comprehensive Literacy Assessment At the direction of the State Board of Ed ucation, a committee of literacy professors, classroom teachers, school administrators, and State Department representatives formed to develop the Idaho Comprehensive Literacy Asse ssment (ICLA). After some deliberation, a committee of literacy professors at various inst itutions in the State worked to create the ICLA, resulting in a pool of items assembled as two forms of the ICLA. Three Statewide pilot tests of these forms were conducted with pre-service elementary education majors between April and December of 2001. Measurement consultants conducted statistical analyses for item and test development. The percent correct for an item was used as an initial means to consider the quality of an item. Since the ICLA is a criteri on-referenced test, the percent correct was used not as means to discriminate among students (a s done with norm-referenced tests), but used instead as an indicator of mastery (Anastasi & Urbina, 1997). Operationally, if more than 50% of the students responding to an item answered incorrectly, the item was examined by the committee to determine if the item was flaw ed, needed revision, or should remain. This process continued systematically after each pilot administration and remains the current practice. The construction of the ICLA was guided by the development of a validity framework (Shepard, 1993). The framework specified the rationale for the assessment, the conceptual basis for the assessment, and logical questions that could be asked of the assessment and process. In addition, the Office of Civil Rights (Berkowitz, Wolkowitz, Fitch, & Kopriva, 2000) developed a set of r ecommendations for the development of high


Education Policy Analysis Archives Vol. 12 No. 61 7 stakes tests to ensure that the test does not unfairly disadvantage students from underrepresented groups. This guide also in formed the construction of the ICLA. Format The Idaho Comprehensive Literacy Assessment is divided into three Standards. Standard I, Language Learning and Literacy Development, addresses emergent literacy, phonological and phonemic awareness, phonics and structural analysis, sight vocabulary, morphemic analysis, and research-based instru ctional practices for developing accurate and automatic decoding. Standard II, Reading Comprehension Research and Best Practices, focuses on fluency, vocabulary development, comprehension instruction, and text genres. Standard III, Literacy Assessment and Inter vention, deals with common assessment procedures, interpretation of assessment results and instructional activities for struggling readers. Each Standard, in turn, is separated into three sections, 2, and 3. Sections 1 and 2 assess basic knowledge of key vocabulary, re ading skills, and instructional strategies and procedures. All items are objectively scored Section 3 contains two open-ended essay questions that assess students application of knowledge; each essay question depicts a classroom scenario. Students are asked to expl ain the purpose of the instructional activity, provide a rationale in light of current research and give two or three alternative activities that address the same instructional objective. A third version of the assessment now exists, and a fourth version is under development. In addition, an on-line versi on of the assessment has been piloted, but no study comparing the paper and pencil and on-line assessment has been conducted. Pass rates appear to be comparable. Due to lack of re sources, refinement of the on-line version has been put on hold. To assist students preparing for the ICLA, an electronic study guide (Squires, Blacklock, Christy, Nelson, and Palmer, 2002 ) was created with content information and sample test items. This guide is available at no charge to students. In addition, each institution schedules test review sessions prior to the administration of the next round of the ICLA. Scoring Rubric and Protocol The ICLA Committee created a four-point rubr ic with which to score the scenarios. The rubric identifies how well a respondent addresses a scenario question, including references to research-based best practices, evidence of understanding and sensitivity to learners differences, and the rationale be hind teachers instructional decisions. Scenario responses are scored by ICLA committee members, all of whom are literacy professors at the several teacher preparation institutions (4 public, 3 private) across the State. Scoring the essay items takes place at a central location. If pre-service teachers respond with unexpected information, the scenario question is reviewed and, if needed, revised for clarity in subsequent administrations. Before any scen ario is scored, the rubric is reviewed and important elements of student responses discusse d. For new members, this procedure serves as an effective induction process.


Squires, Canney & Trevisan: There is another way 8 Two evaluators read each response. If thei r two scores are identical or differ by only one point, the score is averaged between the two readings. If the score of the two readers differs by more than one point, a third eval uator, without knowledge of the previous two scores, also rates the response. Should two ev aluators agree on a score, that is the score assigned. If the three evaluators disagree, the score is averaged. Setting Pass Scores Standard-setting processes were employed to determine the overall pass score for the ICLA. After some deliberation, the ICLA committee decided that each of the three assessment Standards should receive a separate pass score. A measurement consultant guided the standard-setting process by first providing the committee with the advantages and disadvantages of various standard-setting approaches. Given its simplicity and popularity, the committee decided to use the Angoff Method for the objective items; the Benchmark method was used for the scenarios (Cizek, 1996). The process resulted in setting a 70% pass score for each Standard and each form. In addition, the committee considered the relatively heavy influence of the objective items on the pass score. Initially, a student could pass a Standard without obtaining any points on the scenarios. The committee selected a 60-40 pe rcentage split between objective items and scenarios. To accomplish this split, weights were applied to the scenarios in each Standard. Fairness Review A fairness review committee was established to investigate, identify, and correct any item thought to disadvantage students based on race, gender, ethnicity, or disability (Kimtta & Goellner, 2002). The fairness review commi ttee was composed of eight individuals representing people of diverse groups. Representation included the disabled, English as Second Language, African American, Native American, Latin American, lower socioeconomic status, and gender. The fairness review committee was provided with a reading of the review process, a rubric, and a matrix recording form. During the review, members of the committee read each test item and flagged the items that pos ed a problem of bias. Each problem item was brought before the committee for discussion. Th e results of this work were then discussed with the ICLA Committee. Items of concer n were corrected by changes in wording, inserting geographic examples from Idaho, or proper name changes. Administration Procedures The ICLA is administered three times per year, fall, spring, and summer. Pre-service teachers may take one or more Standards during an assessment period, but only one Standard per day, and no Standard more than once during the assessment period. Students failing to pass a Standard may ta ke another form of that Standa rd at the next testing session. Students unable to pass one or more Standards after three attempts may request a special oral examination. Accommodations are available for special needs students.


Education Policy Analysis Archives Vol. 12 No. 61 9 Summary of Initial Assessment Results The following provides a summary of the initial ICLA results for Idaho pre-service teachers by Standard and administration period. Pass rates by attempt are also provided, as well as knowledge (sections 1 and 2 objective items) versus performance (section 3 scenarios) results. Total Pass Rates by Standard Table 1 presents the overall pass rates by St andard for the initial four administration periods, beginning in April 2002. Standard I, Language Learning and Literacy Development, consistently produced the lowest pass rates. Th ere was an overall increase across testing periods in the percentage of pre-service teach ers passing Standard I, with a high of 72 percent passing in December 2002. The number of pre-service teachers passing dropped to 66 percent in April 2003. The pass rates for Standards II and III remained relatively high in comparison to Standard I performance. This pattern was consistent across administrations. Table 1 Total ICLA Pass Rates by Standard, April 2002 April 2003 ________________________________________________________________________ Standard Number Taking Nu mber Passing Percent Passing ________________________________________________________________________ 1 1,457 934 64 2 850 713 84 3 838 712 85 ________________________________________________________________________ Pass Rates by Attempt In April 2002, 186 of the 326 examinees ta king one or more of Standards I-III were doing so for the first time. The other examinees had exposure to one or more of the pilot administrations. April 2002 was also the first fully operational administration of the ICLA. Thus, the results of the first time test takers starting with the first administration, were tracked to determine the number of attempts needed to pass a particular standard. The data provided baseline information to the Idaho teacher preparation institutions. Standard I. Table 2 provides the cumulative pass rates for Standards I, II and III. In April 2002, 80 of 186 pre-service teachers (43 percent) passed Standard I on their first attempt. The following July, 45 of 61 pre-serv ice teachers (24 percent of the 186 pre-service teachers starting in April 2002) passed on their second trial. In December 2002, 16 of 31 preservice teachers taking the Standard for the th ird time passed (9 percent of 186 pre-service teachers starting in April 2002). In April 2003 12 of 14 making a fourth attempt passed (6 percent of 186). Altogether, of the 186 preservice teachers who took Standard I in April 2002 for the first time, 153 passed (82 percent).


Squires, Canney & Trevisan: There is another way 10 Table 2 Cumulative Pass Rates for Candidates taking Standards I, II, and/or III one or more times between April 2002 April 2003 ________________________________________________________________________ Standard Number Taking Number Passing Percent Passing Cumulative Percent _______________________________________________________________________ 1 186 153 82% 43-82% 2 138 136 98% 93-98% 3 144 129 90% 85-90% ________________________________________________________________________ Standard II. Using the same cumulative analysis for Standard II, after three attempts beginning in April 2002, and ending in December 2002, the cumulative pass rate increased from 93% to 98%. Standard III. For Standard III, from the first to the fourth administration, the cumulative pass rate increased from 85% to 90%. Discussion The ICLA has now been operational for approximately two years, after seven administrations of the test. During the begi nning stages of the ICLA development, the committee set a Statewide and institutional goal of an 80% pass rate for each Standard. Some institutions have reached this goal and, while the goal has been met for Standards II and III across the State, students continue to find Standard I to be the most difficult. In addition, students continue to perform better on the k nowledge portion of the assessment (Sections 1 and 2) than on the performance part of the test. These two issuesStandard I difficulty and knowledge versus performance scores, continue to garner a good deal of attention by the committee. Ways to improve the assessment, changes to the curriculum, and revisions to instruction have been topics of ongoing discussion. The authors acknowledge the controversy surrounding teacher testing and the limited research that connects pre-service traini ng and testing with st udent achievement. Our argument, however, is that given mandated testing, there are far more benefits to using the approach discussed in this paper, than could be obtained through the widespread practice of using externally procured tests. We also recognize that there have been reservations about using faculty-controlled assessments A common argument is that the faculty-controlled assessments and scoring process lack objectivity. Facul ty members are viewed as being too close to day-to-day instruction and, as a result, are too vested in the pass rate outcome to remain unbiased. Oversight, therefore, must come from the outside. However, our experience with what has actually occurred in Idaho when higher education faculty members work in a collegial and professional manner has shown that concerns about faculty bias have been unfounded. To the contrary, the benefits of this collaboration have been impressive. Although faculty members set an 80% pass rate, this target has yet to be achieved for all


Education Policy Analysis Archives Vol. 12 No. 61 11 cohorts and standards Statewide. In addition, so me institutions have struggled to meet the 80% target, even after multiple attempts by individual students. Faculty members have consistently shown themselves unafraid to impos e a rigorous examination, even if it means some students may fail or that their respecti ve institutions may not meet the 80% target. Still others may argue that faculty members across institutions who espouse different philosophical orientations about how students learn to read and therefore how reading should be taught, are ill-suited to productive ly work together to create and sustain an assessment of this magnitude. Again, the evidence suggests otherwise. Literacy faculty members from institutions in Idaho that often compete with one another for enrollment, resources, and political support, have chosen to work together amicably. Faculty members have also worked to overcome differences of opinion about best instructional practices. This level of cooperation was accomplished by first agreeing on the outcomes assessed by the ICLA. Once this understanding was reached, faculty members agreed to support the possibility of multiple strategies for obtaining literacy learning outcomes for Idaho preservice teachers. Thus, phonics and whole lang uage approaches remain firmly embedded at the institutional level. The ICLA committee has worked professionally and responsibly to implement operational policies concerning the ICLA. These policies are always developed with forethought and fairness in mind. Policies include when and how the ICLA will be administered, number of possible retakes, acco mmodations, and the like. Recently, the State Board has charged the Colleges of Education Deans Council to oversee the work of the ICLA committee and to develop a budget for the ong oing work of the committee. The Deans Council has remained supportive of the wo rk of the faculty and the role of the ICLA. In large part, faculty members remain in control of the assessment. What, then, does this ICLA experiment offer the teacher education community as well as State and Federal policy makers? Firs t, it is a fundamentally different test development and refinement process than the externally contracted tests now sweeping the country. While the ICLA is in direct response to State legislation focused on improving reading instruction, the process embraced in Idaho is one under the purview of faculty members directly responsible for literacy course s in pre-service programs in the State. Second, it is an assessment approach decidedl y weighted in favor of program improvement rather than common notions of accountability through external testing. Finally, it is a process that exemplifies the recommendati ons put forth by the National Commission on Excellence in Elementary Teacher Preparation for Reading Instruction (Maloch, Fine, & Flint, 2002). The test is definitely high stakes for students and institutions. Students who do not pass the test cannot be credentialed in the St ate of Idaho. Institutions are faced with ensuring that a high percentage of their graduates can demonstrate content and pedagogical knowledge appropriate for well-prepared teach ers (Public Law 107-110, 2001). However, what the State receives for reformulating current notions of accountability is a test with relevance. Rather than external teacher tests with a lack of connection to what teachers should know and do, this Idaho test assesses content and performance outcomes directly related to State and National professional standards for pre-service literacy preparation. Moreover, the ICLA was developed with rigorous measurement procedures that meet industry standards. These measurement pro cedures are consistent with the criterionreferenced nature of the ICLA. The ICLA data indicate that the test provides reliable and valid results that can be used to make sound decisions about teacher competence and refinements to pre-service teacher preparation programs.


Squires, Canney & Trevisan: There is another way 12 A further benefit of this testing process is its contribution to the continuous improvement of pre-service programs. Faculty members develop tests, arrange for their administration, analyze the results by item and task, and discuss results. Since faculty members are intimately involved in all aspects of the assessment process, they act on the results (Weiss, 1998). Based on regular discussions of test results and accompanying input from colleagues across the State, faculty member s make changes in their pre-service course curricula and instruction. Thus, this approach st ands in direct contrast to external testing strategies by external agencies which, to date, have not been shown to effect changes in preservice courses (Cochran-Smith & Dudley-Marling, 2001). Despite the ICLAs initial accomplishments, repeated attempts to obtain a budget from the State for this project have failed. The funding to initiate the project came from one-time State allocations for a related literacy initiative. However, these funds have not been replenished. The funds for the current operation are derived from modest student registration fees, while the faculty members continue to invest enormous time without compensation. The work to print, disseminate, score, and send reports to institutions, is accomplished through short-term time slip employees. Clearly, the infrastructure supporting the ICLA is fragile and its future is in question. We want to underscore the fact that the cost for a testing approach such as the ICLA is likely a fraction of what states typically spend on externally procured teacher tests. By employing a testing process like the ICLA, states will not only have a test with relevance used, in part, to improve pre-service instruction, they will also have a testing process with reduced expenditures on teacher testing, saving scarce state resources. Two additional challenges for the ICLA will need attention. First, much of the policy rhetoric concerning teacher testing is focused on the assessment of performance. A handful of states have explored the implementation of actual performance assessments, although because of the cost and difficulty of establishing adequate reliability and validity, most states have shied away from these assessments (Youngs, Odden, & Porter, 2003). Early deliberations concerning the ICLA stressed the importance of assessment of performance, but given the resources the choice was made to opt for more cost effective and efficient assessment strategies. A consequence is the use of less authentic approaches to assessing performance outcomes, but the push for assessment of performance remains. The committee has outlined a new version of the IC LA that would make it more performanceoriented than the current paper-and-pencil version; development awaits funding. Second, given the high stakes nature of teacher tests, concern for disadvantaged preservice teacher candidates will continue to grow (Young, O dden, & Porter, 2003). The high stakes testing guidelines developed by the Federal government (U. S. Department of Education, 2000) as well as the Standards for Educational and Psychological Testing (American Educational Research Association; American Psychological Association; National Council on Measurement in Education; Joint Committee on Standards for Educational and Psychological Testing, 1999) provide the lega l, professional, and moral imperative for monitoring differential performance across student groups. This monitoring is a critical aspect of the ICLA process. Currently, the number of students in any historically disadvantaged student group is so small in Idaho that statistical analysis has been rendered meaningless. A list of acceptable accommodations for special needs students is now part of the policy governing the ICLA and is clearly outlined in the ICLA Study Guide. In addition, given the relatively small number of students in teacher education programs in general, faculty members know students personally, pa rticularly students from underrepresented


Education Policy Analysis Archives Vol. 12 No. 61 13 groups. Careful oversight for the welfare of these students has been occurring at the program and institutional level. The ICLA Committee will continue to attend to this issue. Summary As external teacher testing continues to wo rk its way into State and Federal reform policy, the ICLA stands apart. By using an assessment developed and maintained by faculty, rather than one developed, procured, and maintained by an external agency, the ICLA represents another way for State and Federal policy makers to think about teacher testing and the benefits and limitations associated with such practices. The ICLA test and process is decidedly focused on rigorously assessing pre-service teacher knowledge and performance in reading, and using the results to revise pre-se rvice instruction. No external teacher test has shown to impact the curriculum and instructi on of pre-service instructional programs. However, in Idaho the effect of pre-service tea cher performance on the ICLA, particularly on Standard I, has directly impacted the literacy course content and delivery at Idaho teacher preparation programs.


Squires, Canney & Trevisan: There is another way 14 References Adams, M. J. (1990). Beginning to read: Thinking and learning about print Cambridge, MA: MIT Press. Alliance 2000 Project of the University of New Mexico (1994). Summary of Goals 2000: Educate America Act Alexandria, VA: National Association of State Directors of Special Education (ERIC Document Reproduction Service No. ED389130). American Educational Research Association; Am erican Psychological Association; National Council on Measurement in Education; Joint Committee on Standards for Educational and Psychological Testing. (1999). Standards for educational and psychological testing. Washington, DC: American Educational Research Association. Anastasi, A., & Urbina, S. (1997). Psychological Testing (7th ed.). Upper Saddle River, NJ: Prentice Hall. Berkowitz, D., Wolkowitz, B., Fitch, R., & Kopriva, R. (2000). The use of tests as part of highstakes decision-making for students: A resourc e guide for educators and policy makers. Washington, DC: U. S. Department of Educ ation, Office for Civil Rights. Retrieved December 23, 2003, from California Commission on Teacher Credentialing. (1998). Reading instruction competence assessment Retrieved December 22, 2003, from Camilli, G., Vargas, S., & Yurecko, M. (2003) Teaching children to read: The fragile link between science and Federal education policy. Education Policy Analysis Archives, 11 (15). Retrieved November 9, 2004, from Canney, G. F. (1998). A study of Idaho fourth grade students reading levels. Portals 6(1), 710. Canney, G. F. (1999). A response to Pehrsson and Denner: Design and implications of the Idaho fourth grade reading study. Portals 6 (2), 13-19. Carlson, R. (1999). Reading instruction competence asse ssment (RICA): Results of the 1998 administrations. Retrieved December 30, 2003, from Cizek, G. J. (1996). Setting passing scores. Educational Measurement: Issues and Practices, 15 (2), 20-31. Cochran-Smith, M., & Dudley-Marling, C. (2001). The flunk heard round the world. Teaching Education, 12 49-63.


Education Policy Analysis Archives Vol. 12 No. 61 15 Duffy, G. G., & Hoffman, J. V. (1999). In pur suit of an illusion: The flawed search for a perfect method. Reading Teacher, 53 10-16. Executive Office of the President. (1990). National goals for education Washington, DC: Executive Office of the President. (E RIC Document Reproduction Service No. ED319143). Flippo, R. F., & Riccards, M. P. (2000). Initia l teacher certification testing in Massachusetts: A case of the tail wagging the dog. Phi Delta Kappan, 82 34-38. Haney, W. (2002). Lake Woebeguaranteed: Misuse of test scores in Massachusetts, Part I. Education Policy Analysis Archives, 10 (24). Retrieved November 9, 2004, from Haney, W., Fowler, C., & Wheelock, A. (1999). Less truth than error? An independent study of the Massachusetts Teacher Test. Educational Policy Analysis Archives 7 (4). Retrieved November 9, 2004, from Idahos MOST. (2001). Idaho maximizing opportunities for students and teachers. Retrieved May 21, 2004, from Idaho Statute 33-1207A. (1999). Title 33, Chapter 12, Teacher Preparation Retrieved March 8, 2004, from International Reading Association, Professional Standards and Ethics Committee. (1998). Standards for reading professionals (Rev. ed.). Newark, DE: International Reading Association. Kmitta, D. & Goellner, L. (2002, July). Fairness review report: Idaho comprehensive literacy assessment Paper presented at the National Evaluation Institute, Boise, ID. Ludlow, L. H. (2001). Teacher test accountability: From Alabama to Massachusetts. Education Policy Analysis Archives 9 (6). Retrieved November 9, 2004, from Ludlow, L H., Shirley, D., & Rosca, C. (200 2). The case that wont go away: Besieged institutions and the Massachusetts teacher tests. Education Policy Analysis Archives, 10 (50). Retrieved November 9, 2004, from Maloch, B., Fine, J., & Flint, A. S. (2002). Trends in teacher certification and literacy. Reading Teacher, 56 348-350. National Academy of Education Commission on Reading. (1985). Becoming a nation of readers: The report of the Commission on Reading Washington, DC: National Institute of Education.


Squires, Canney & Trevisan: There is another way 16 National Commission on Excellence in Education. (1983). A nation at risk: The imperative for educational reform: A report to the nation and the Secretary of Education United States Department of Education Washington, DC: United States Department of Education. National Reading Panel. (2000). Report of the National Reading Panel: Teaching children to read Washington, DC: National Institute of Child Health and Human Development. Retrieved October 20, 2003 from Pearson, P. D. (1997). The firstgrade studies: A personal reflection. Reading Research Quarterly, 32 428-432. Public Law 105-244. (1998). Title II of the Higher Education Act. Retrieved December 30, 2003, from Public Law 107-110. (2001). No Child Left Behind Act of 2001 Retrieved December 30, 2003, from Ruddell, R. B. (1997). Researching the influe ntial literacy teacher: Characteristics, beliefs, strategies, and new research directions. In D. J. Leu, C. K. Kinzer, & K. A. Hinchman (Eds.), The 46th yearbook of the National Reading Conference (pp. 37-53). Chicago: National Reading Conference. Shepard, L. A. (1993). Evaluating test validity. In L. Darling-Hammond & J. A. Banks (Eds.), Review of research in education (pp. 405-450). Washington, DC: American Educational Research Association. Snow, E. C., Burns, M. S., & Griffin, P. (Eds.). (1998). Preventing reading difficulties in young children Washington D. C.: National Academy Press. Squires, D., Blacklock, K., Christy, C., Nels on, C. & Palmer, R. (with Canney, G. F. & Goellner, L.). (2002). The Idaho comprehensive lite racy assessment study guide Retrieved April 6, 2004, from http://ed.i State Board for Educator Certification (2003). Texas examination of educator standards. Available from height=600 U. S. Department of Education, Office for Civil Rights. (2000). The use of tests as part of highstakes decision-making for students: A reso urce guide for educators and policy makers Washington D. C.: Author. Retrieved May 21, 2004, from rchives/testing/index1.html Weiss, C. H. (1998). Have we learned an ything new about the use of evaluation? American Journal of Evaluation, 19 (1), 21-34.


Education Policy Analysis Archives Vol. 12 No. 61 17 Yatvin, J. (2002). Babes in the woods: The wanderings of the National Reading Panel. Phi Delta Kappan, 83, 364-369. Youngs, P., Odden, A., & Porter, A. C. (200 3). State policy related to teacher licensure. Educational Policy, 17 217-236. About the Authors David Squires Idaho State University Campus Box 8059 Pocatello, ID 83209-8059 Phone: (208) 282-4445 Fax: (208) 282-3791 E-mail: David Squires, Ph.D. is currently an Assistant Professor of Literacy Education at Idaho State University. He teaches both graduate and unde rgraduate literacy courses in the College of Education. His research interests center on tea cher education, including the relationship between theoretical beliefs and practices. George Canney University of Idaho College of Education P.O. Box 3082 Moscow, ID 83844-3082 Phone: (208) 885-7712 Fax: (208) 885-0560 E-mail: George F. Canney, Ph.D, is a professor in the Division of Teaching, Learning and Leadership, College of Education, University of Idaho. He teaches undergraduate and graduate literacy courses, including two onlin e graduate literacy courses. His research interests include developmental reading, the ICLA project and literacy assessment in general, and online learning. Mike Trevisan Washington State University P.O. Box 642136 Pullman, WA 99164-2136 phone: 509-335-7063 fax: 509-335-6961 email: Dr. Mike Trevisan is the Director for the Assessment and Evaluation Center at Washington State University. Dr. Trevisan is also an associate professor in the Department of Educational Leadership and Counseling Psychology where he teaches courses in measurement, statistics, and program evaluation.


Squires, Canney & Trevisan: There is another way 18 Education Policy Analysis Archives http:// Editor: Gene V Glass, Ar izona State University Production Assistant: Chris Mu rrell, Arizona State University General questions about ap propriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, glass@ or reach him at College of Education, Ariz ona State University, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: EPAA Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Linda Darling-Hammond Stanford University Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Gary Orfield Harvard University Anthony G. Rud Jr. Purdue University Jay Paredes Scribner University of Missouri Michael Scriven University of Auckland Lorrie A. Shepard University of Colorado, Boulder Robert E. Stake University of IllinoisUC Kevin Welner University of Colorado, Boulder Terrence G. Wiley Arizona State University John Willinsky University of British Columbia


Archivos Analticos de Polticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998003) Roberto Rodrguez Gmez Editorial Board Hugo Aboites Universidad Autnoma Metropolitana-Xochimilco Adrin Acosta Universidad de Guadalajara Mxico Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educacin, Chile Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educacin, Argentina Teresa Bracho Centro de Investigacin y Docencia Econmica-CIDE Alejandro Canales Universidad Nacional Autnoma de Mxico Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagoga Popular, Per Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernndez Enguita Universidad de Salamanca. Espaa Gaudncio Frigotto Universidade Estadual do Rio de Janeiro, Brasil Rollin Kent Universidad Autnoma de Puebla. Puebla, Mxico Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California Mara Loreto Egaa Programa Interdisciplinario de Investigacin en Educacin, Chile Mariano Narodowski Universidad To rcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Polticas Educativas, Per Vanilda Paiva Universidade Estadual do Rio de Janeiro, Brasil Miguel Pereira Catedratico Un iversidad de Granada, Espaa Angel Ignacio Prez Gmez Universidad de Mlaga Mnica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de So Paulo Diana Rhoten Social Science Research Council, New York, New York Jos Gimeno Sacristn Universidad de Valencia, Espaa Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, Mxico Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas Publicas-Universidad de Buenos Aires, Argentina Antonio Teodoro Universidade Lusfona Lisboa, Carlos A. Torres Univ of California, Los Angeles Jurjo Torres Santom Universidad de la Corua, Espaa Lilian do Valle Universidade Estadual do Rio de Janeiro, Brasil