xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20059999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00429
Educational policy analysis archives.
n Vol. 13, no. 7 (January 23, 2005).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c January 23, 2005
Catching up with the Vaughn Express : six years of standards-based teacher evaluation and performance pay / Eileen M. Kellor.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 13issue 7series Year mods:caption 20052005Month January1Day 2323mods:originInfo mods:dateIssued iso8601 2005-01-23
E DUCATION P OLICY A NALYSIS A RCHIVES A peer reviewed scholarly journal Editor: Sherman Dorn College of Education University of South Florida Copyright is retained by the first or sole author, who grants right of first publication to the Education Policy Analysis Archives EPAA is published jointly by the Colleges of Education at Arizona State University and the University of South Florida. Articles are indexed in the Directory of Open Access Journals (www.doaj.org). Volume 13 Number 7 January 23 2005 ISSN 1068 2341 Catching Up With the Vaughn Express: Six Years of Standards Based Teacher Evaluation and Performance Pay Eileen M. Kellor University of Wisconsin Madison Citation: Kellor, E. M. (2 005, January 23). Catching u p w ith the Vaughn e xpress: Six y ears of s tandards b ased t eacher e valuation and performance pay. Education Policy Analysis Archives, 13(7). Retrieved [date] from http://epaa.asu.edu/epaa/v13n7./ Abstract Traditional methods of paying and evaluating teachers in the United States are longstanding, but discussions about changing these systems to support teacher quality and student achievement goals are becoming more common. Efforts to make significant changes to these programs can be difficult and take many years to design, gain approval, and implement; thus, few examples of alternative teacher compensation and evaluation systems exist. Relieved from many of the restrictions and requirements associated with most traditional public school systems, charter schools often are better positioned to implement changes more quickly than a traditional school, yet their experiences can provide useful information for others who are engaging in similar activities. Thus, the experience of a large urban charter school that designed and implemented an innovative teacher compensation system and standards based teacher evaluation system that has been in place for six years offers important lessons in designing, implementing, evaluating and refining th ese systems. The Vaughn Next Century Learning Center has been one of the CPRE UW Teacher Compensation Projects research sites since 1998, when the school first implemented knowledge and skill based pilot program and a school based performance award pr ogram The purpose of this paper is to provide a brief overview of the schools pay
Kellor: Teacher evaluation & performance pay 2 and evaluation programs, their evolution and current status, and to highlight findings from the employee attitude surveys that CPRE UW researchers administered to Vaughn te achers the past five years. Finally, the paper will note key themes that may be of use to others who are considering compensation or evaluation reforms. Although other areas of research and inquiry are logical extensions to the work reported here, this pa per primarily provides a longitudinal review of one schools teacher compensation and evaluation innovation, including design, implementation, and evaluation drawing from multiple information sources at the school and individual level Background The V aughn Next Century Learning Center is a public conversion charter school in the Los Angeles Unified School District. The school became a charter school in 1993 after a majority of the schools teachers petitioned for that status. Each five year charter inc ludes specific outcomes that will be achieved during the charter period and those outcomes inform virtually every decision made at Vaughn. The current principal, Yvonne Chan, has been with the school since 1990 and was transferred there prior to the school s conversion. Chan is widely credited with a dynamic leadership and inclusive style that has contributed to improved student achievement. The schools immediate neighborhood has been somewhat revitalized by the schools creation of a compound that inclu des a Family Center, a one stop shop that provides families with basic needs, such as prenatal care, family literacy, adult education classes, job referrals, and services provided in collaboration with various agencies. The school s current charter inclu des plans to add one additional grade level a year so that the school eventually will serve students pre K through high school; grade seven was added for the 200 3 0 4 school year. Vaughn employs about 75 classroom teachers and 50 paraprofessional and certi ficated support staff. Maximum class size is kept at 20 students (PreK 5) and 25 students in middle school. Every family is expected to be involved in the school community in some way including signing an annual home school compact that the parents will s pend at least 30 hours a school year in activities that support their childrens education. The schools governance structure includes roles for teachers, administrators, and parents, reflecting that responsibility and accountability for the education of V aughns students is shared and that everyone in the school community has a stake. It was from within the context of accountability and desired outcomes that the performance pay and teacher evaluation innovations arose. The development of the knowledge a nd skill based pay program began during the 1997 98 school year. Six main issues and concerns were identified that contributed to the decision to develop a new pay system. They were: 1) pay inequity among teachers, 2) a concern that the single salary sched ule promoted mediocrity, 3) a perceived need to develop a more explicit internal accountability focus to support the external accountability required by the schools charter, 4) a need to retain and incent new teachers, 5) a concern that many classroom tea chers perceived movement into administrative positions as the only way to make more money, and 6) an interest in making improvements to teachers salaries directly related to increased student achievement (Kellor, Milanowski, Odden, & Gallagher, 2001) Implementation of the pilot knowledge and skill based pay system began in 1998 99 for newly hired teachers and for current Vaughn tea chers within their first five years of employment at Vaughn and volunteered to participate. A school based performance award program cover ing all teaching staff also was implemented that year
Education Policy Analysis Archives Vol. 13 No. 7 3 An essential piece of a knowledge and skill based pay system i s a way to measure the knowledge and skills to be compensated, thus, a new evaluation system was implemented concurrent with the first year of the new pay system. The first year it was used only f o r those teachers who were covered by the new pay system, an d was expanded in subsequent years to include all teachers so that all teachers would be held to the same teaching expectations The new system was loosely based on Charlotte Danielsons Framework for Teaching (Danielson, 1996) which defines four broad domains of teaching, with specific attributes defined under each of the domains. Da nielsons framework includes four performance levels, Unsatisfactory (1), Basic (2), Proficient (3), and Distinguished (4), with a description provided at each level f or every performance component. A unique aspect of how Vaughn developed and implemented its new pay program and evaluation system is that th e programs intentionally we re implemented before all the details were resolved. In the view of the principal, they were designing the plane as they were heading down the runway. Both t he staff who were designing the programs and the governance committees that were responsible for approving the programs believed that a quick implementation was necessary to show commitment to the new programs and to get a start on achieving the positive outcomes including improved student achievement that they expected would flow from the new pay and evaluation programs. Thus, important policy and conceptual decisions were made and the programs were implemented as is, with a public commitment to revise and r efine as expe rience was gained. Vaughn has continued to tweak the programs annually and sometimes during the school year, based on internally identified needs and issues and in response to information gleaned from annual employee attitude surveys conducted by Teache r Compensation researchers at the University of Wisconsin Madison branch of the Consortium for Policy R esearch in Education (CPRE UW). It include d questions relating to both the pay program and the evaluation system. Although direct comparisons over time c ant be made because the questions are not identical across all years, some general observations and themes based on the responses will be noted later in this paper. (A copy of the most recent version of the survey is available upon request from the author .) Overview of the Pay and Evaluation Programs This section includes a summary of these programs for each of the past six years. Because the school based performance award program underwent few changes after the second year, that program is summarized f irst in a separate section, followed by an overview of the evolution of the knowledge and skill based pay and evaluation programs School Based Performance Award: A Bridge Between Old and New Although the school based performance award ( SBPA ) program wa s created as an adjunct to the schools knowledge and skill based pay program, the initial compensation design did not include a SBPA program. Early in the design process, senior teachers appeared to view the new pay structure development as a personal th reat o r as implying that Vaughn intended to pay its teachers less. In January 1998 a draft plan was presented to Vaughns Business Committee, which is responsible for teacher evaluation and salary determination. Th e submission of the draft plan brought s enior teachers more directly into the discussion and review of options In May a task force of newer and veteran teachers met to work out more of
Kellor: Teacher evaluation & performance pay 4 the details of the new pay program, and th at group raised the question of why all teachers would not be compen sated for those items included in the draft pay system that were based on specific outcomes at either the school wide, grade level, or classroom level. The program designers then decided to include a SBPA element as part of its new compensation program t o support the school wide goal of improved student achievement and to function as a bridge between those teachers on the traditional salary structure and those in the new pay system. The SBPA also was intended to strengthen the concept of internal accounta bility by linking some form of compensation directly to student achievement for all teachers. The initial SBPA program included two elements of performance: school wide student achievement and school wide redesignation of limited English proficiency stu dents to English language learners (ELL). T hese two performance measures were key parts of the schools charter plan for 1998 2003 and thus were chosen as the SBPAs performance measures. Due to difficulties in establishing a measurement method and goal a nd the introduction of a new statewide accountability program beginning the 1999 00 school year, the redesignation portion was never implemented. For 1998 99, the school used three measures of student achievement: 1) school wide improvement on the state mandated Stanford 9 (SAT9) reading, math and language tests for grades 1 5; 2) school wide performance on the Terra Nova (English and Spanish versions) reading and math tests for grades K 5; and 3) school wide report card grades. To receive the school wi de performance award the school had to post an increase of at least three percentile points over the previous years SAT9 school wide average score and meet at least one of the other two achievement goals: an average school wide score of 37 th percentile or higher on the Terra Nova post test or a school wide average report card grade of C or better. Due to concerns about public pressure and credibility and potential problems in relying on a single measure, the school chose a combination of standardized te st scores and grade level performance as measured by report card grade. Compared to some SBPA programs that use a combination of academic and non academic measures such as attendance, Vaughns plan was relatively straightforward. The school wide student a chievement goals were met for 1998 99 and all certificated staff, including administrators, received $1,500. After the first year, Vaughn continued its SBPA program but modified it in response to a bonus award component of the States new accountability p rogram. A key piece of the Public Schools Accountability Act of 1999 required that the California Department of Education annually calculate an Academic Performance Index ( API ) for every public school in California, including charter schools and to publis h school rankings based on the APIs. The A ct a lso established a minimum five percent annual growth target for each school. Schools that met API growth targets could be eligible for awards under several state programs, with potential individual awards of up to $25,000. Rather than operate a separate program, Vaughn decided to tie its SBPA program directly to the API program, with eligibility for a SBPA dependent on the school meeting its API growth target in a particular year. D ue to changes in the assessmen t tools and the methodology to integrate results from multiple assessment measures the actual calculation of the API varied somewhat from year to year and thus is not included in this paper (California Department of Education, 2003) Vaughns growth ex ceeded the target goal in each of the years for which API data are available. Table 1 shows the API base, growth target, and actual growth from 1999 00 through 200 2 0 3 :
Education Policy Analysis Archives Vol. 13 No. 7 5 Table 1 API base, growth target, and actual growth from 1999 00 through 2002 03 Scho ol year API base Growth target Actual growth 1999 00 443 18 51 2000 01 494 15 85 2001 02 591 10 18 2002 03 619 9 41 Source: CDE, Policy and Evaluation Division, Academic Performance Index Growth Reports For most California schools, t he actual payment of the state funded bonuses was delayed most years by funding complications and difficulties in verifying final student achievement data schools. However, Vaughns teachers received their bonus payments under the Vaughn SBPA program almost as soon as the test data were available, because Vaughns administrators had pledged to pay the bonuses from funds already under the schools control rather than to wait for the state to release the API award funds. Strategically, this allowed Vaughn to maintain teachers trust in the SBPA program and the schools commitment to deliver what it had promised. After three years at $1,500, the bonus award was increased to $2,000 for the 2001 02 school year and has been maintained at that level Several other pay innovations are included in Vaughns pay system. A contingency pay program gives participating teachers a $250 bonus at the end of the school year for each of the identified areas for which the stated goal was achieved ; the contingency plan was revised in 2003 04. (Re fer to Appendix A and B for details.) A gainsharing program was developed, in which participating teachers receive the savings from decreasing the use of substitutes for the respective teacher. The pay program also provides expertise pay for specific roles or responsibilities. In the second year, a new stock options program was created by investing a lump sum received from a new grant. A total of $290,000 was invested on behalf of all certificated staff participating in the performance pay program, with distribution policies still under development, but with the understanding that the Employee Incentive Fund is intend ed to be a long term investment with no immediate payouts. The next section summarizes the performance pay ( knowledge and skill bas ed pay ) plan from its development through the sixth year of program operation. Evolution of the Performance Pay Program The first year of the performance pay program was intended to introduce the new pay system to new and early career teachers, who were perceived to have lesser commitment to or investment in the traditional salary structure than more veteran teachers. By phasing in the new pay system, the school also hoped that veteran teachers would gain trust in the new system as well as confidence that the new system could be a positive change force. Teachers would be evaluate d on the competencies each semester and eligible to receive half of the annual amount for each competency each semester. The competencies included in the performance pay program we re chosen because in addition to directly relating to instructional principles in the schools charter they were viewed as non controversial. The designers also believed that they could be measured by existing assessments or that new assessment tools could easily be developed to measure them.
Kellor: Teacher evaluation & performance pay 6 The base pay rate for a first year teacher was set at $30,000 for an emergency credentialed teacher and $31,000 for a teacher with a California elementary teaching credential. Each year for the first five years a teac her would receive a base increase of $1,000, with an additional $1,000 annually if the teacher had a masters degree. Although th e $36,000 after five years for a credentialed teacher was less than the rate for a five year teacher under Vaughns traditional schedule, $37,867 additional pay increments available under the pilot performance pay system offered the potential of $3,300 more for four competencies related to instruction, and up to $800 more for the four contingency based items. Potentially, a first year teacher would receive a base rate of $31,000 and up to $4,100 in additional pay under the performance pay program. (See Appendix A for a summary of Vaughns performance pay provisions for the first five years.) The plan also offered the prospect of h igher total compensation for existing teachers who were early in their careers at Vaughn. A five year teacher would receive $36,000 in base pay, and up to $4,100 in performance pay, several thousand dollars more than under the traditional salary schedule. Five early career teachers opted into the program the first year The y cited a variety of reasons including the potential for more pay, the equity of a pay system that did not restrict pay opportunities for those who did not have the time and/or money to take advanced coursework needed for salary advancement under the traditional system, and to be a role model by being held accountable for student achievement. Although the original plan was to gradually expand the pilot pay program over several years to i nclude more senior teachers, changes were made effective the second year that would allow more senior teachers to opt in. This change in direction was in large part a response to demands by some veteran teachers that they be able to participate in the pay program after they saw the amount of performance pay that many of the teachers in the pilot program received. These teachers believed that they were already performing at the level that would be required to receive the additional pay and that they could in crease their compensation without significant additional effort or change in their performance. The second force behind the earlier expansion related t o changes at the state level in grant and other funding opportunities that Vaughn could receive for certa in instructional changes, along with state required changes to the English as a Second Language (ESL) instruction that would require all Vaughn staff to be more involved in and responsible for ESL instruction. Thus, effective with the 1999 00 school year, the performance pay program was offered to all Vaughn teachers regardless of their length of service at the school. Participation for new hires continued to be mandatory, and teachers who opted into the new program could not switch back to the traditional salary schedule. T he design team for the expanded pay program was essentially an ad hoc group that included some of the same people who designed the pilot, joined by some veteran teachers who wanted to help make decisions about a revised program that cou ld impact and appeal to more veteran teachers. T he principal was involved in the design process but tried to give the teachers the lead role. She stated that after the fact, this might not have been the best strategy, because it might have put the teacher s in an unreasonably awkward position by requiring them to deal directly with their peers on controversial and sensitive pay issues. The potential availability of state funds earmarked for specific educational reform goals also meant that the pay designers had to consider whether the state funds were related to the schools charter and if so, how they could best be used to support the new pay program This required that the design team deal with school wide issues of curriculum design and delivery, but did give the school access to some additional funds that were used to enhance the pay program. T he design of the initial pilot program was not without challenges and new issues continued to emerge as the designers sought to expand the program. The primary cha llenge was
Education Policy Analysis Archives Vol. 13 No. 7 7 to define pay principles in a way that would not seriously disadvantage veteran teachers, particularly those near the top of the traditional salary structure. Pay plan designers knew that if movement to the new pay system required too large a fi nancial penalty, teachers would be unlikely to take the risk. A second challenge arose as the potential value of the performance pay was considered. For teachers in the pilot program, the potential performance pay available was almost 25 percent of their b ase pay. Th at same dollar amount, however, was only 5 7 percent more than the base pay of a more veteran teacher at or near the top of the salary schedule. Thus, a fundamental question arose: should the potential pay be indexed to the base salary rate, s o that the same potential differential was available to all teachers, or should the performance pay amounts remain a flat dollar amount providing the same amount of compensation for the same demonstrated performance. Estimates of veteran teachers pay unde r the new system suggested that the proposed pay system would net veteran teachers on average the potential for 17 percent more pay than they were receiving under the traditional pay system. While recognizing the veterans concerns that a new pay system should not drastically affect their guaranteed base salary, the design team believed that the new pay system must strike a balance between that concern and the need for a new pay system to foster and be supportive of individual accountability The design ers also believed that for the new pay system to have a motivational effect, the potential payout under the new plan, and the potential cost of not performing well enough to earn at least some of the performance pay, had to be considered. To help address t he tension between the need to decrease base pay rates under the new system and the desire to make the new plan attractive to teachers, the plans designers adopted a policy statement that whatever base rates were decided, monthly off sets would be avail able to help veteran teachers make the transition to the new and likely lower base pay rate. Another challenge to the expansion of the pay program was whether there should be additional or different knowledge and skills identified for experienced teachers. The designers agreed that the initial set of competencies should apply to all teachers, and that additional competencies should be identified that would presumably be applicable to more experienced teachers. The result was the creation of three levels. Le vel 1 was slightly reconfigured for the second year and deemed depth in essentials, those knowledge and skills that all teachers at all levels of experience should be able to demonstrate in their teaching. Level 1 included: literacy, ESL/Sheltered Englis h, technology, special ed ucation inclusion, classroom management, and lesson planning. Level 2 included four additional competencies (mathematics, social studies, science, arts) that represented the breadth of knowledge and skills required of all teachers who, as elementary school teachers essentially function as generalists and are responsible for instruction in all content areas. Level 2 also included English learners support (BCLAD), with three possible strands of expertise. To be eligible for any of the Level 2 performance pay, a teacher must have received an average rating of 3.0 on all Level 1 competencies and have a clear California teaching credential. The latter requirement also was intended to serve as an additional incentive for emergency crede ntialed teachers to obtain a clear credential. Level 3 did not include any additional knowledge or skill areas, but rather required an overall level of teaching performance markedly higher than that required for Level 2 performance pay eligibility. To be e ligible for Level 3 pay, a teacher must have received an average rating of at least 3.5 in all skill areas (Levels 1 and 2). Thus, an experienced teacher with above average teaching performance could earn up to $13,100 in additional pay under the new pay s ystem ($3,500 for Level 1, $5,600 for Level 2, and $4,000 for Level 3). The decision whether to stay with the traditional pay system or opt in to the new system seemed to reveal two disparate views among the veteran teachers. S ome teachers saw the new
Kellor: Teacher evaluation & performance pay 8 pay system as a way to create a more equal playing field that compensated people in a manner that was more directly tied to their actual contributions to Vaughn and its students, rather than on how well they had played the game of taking advanced courses and acquiring degrees. Other teachers who had successfully moved up the traditional salary schedule felt they would be penalized for having played the game by the rules in place at the start and b elieved they should in essence be held harmless in terms of bas e salary rate, and not denied the opportunity to receive performance pay bonuses. Ultimately, the pay program included annual base pay increases up to the 11 th year, and then an increase at year 15. The expanded pay program also modified the criteria for the Masters degree add on, making it available to people who had obtained 30 educational units after attaining a clear California teaching credential but not a Masters. This was intended to help address the concern of some veteran teachers that the addi tional coursework they had taken would be completely disregarded under a new pay system. The beginning of Year 3 (2000 2001) marked completion of a year of experience with the expanded pay program and two years with the programs first cohort of newer teac hers. Some relatively minor changes were made to the amount of performance pay for some competencies to more accurately reflect the relative value of the respective knowledge and skills. In addition, Mathematics was moved from Level 2 to Level 1 and Techno logy was moved from Level 1 to Level 2. These switches were made because after additional consideration, mathematics was viewed as a critical area that should be included at Level 1 to stress its school wide importance and fundamental place in the schools curriculum. Technologys movement to Level 2 was due to ongoing problems with making technology tools regularly available to all teachers to use in their day to day instruction, as well as a reassessment of the relative importance of Technology to the sch ools overall goals. The fourth year (2001 2002) the pay program saw few changes and it was unchanged for the 2002 03 school year Physical Education was added to Level 2, the amount paid for attaining Level 3 was increased, and a fourth level Distingui shed Teacher, was added, which would pay $2,000 to someone after five consecutive years of attaining Level 3 performance. For the fourth year, all but five veteran teachers had opted into the performance pay program. By the end of th at year, three of the v eteran teachers who had remained on the traditional salary system planned to return to Vaughn the following year ; one other planned to retire and one to return to a LAUSD position Ultimately, of the three, one stayed and converted to performance pay, one s witched to performance pay mid year, and one moved out of town. Thus, by the fifth year, the traditional teacher salary schedule became part of Vaughns history and all teachers, current and future, were included in the performance pay system. The sixth y ear (2003 04) saw more significant changes to the performance pay structure. The addition of middle school staff who are subject specific teachers rather than generalists, and the schools goal to have one pay system applicable to all teachers, had made it difficult to use the subject specific competencies of the former system. The school also felt that because most staff were earning the performance pay bonuses in every Level 1 competency area, the pay associated with those competencies should be reallocat ed to base pay, and the performance pay program should focus on more general instructional areas. Thus, although the pay program retained three levels, they became tied to performance on four domains very similar to the original domains in Charlotte Daniel sons Framework for Teaching rather than to content specific competencies. The total amount of performance pay potentially available to a teacher is $13,050; t he revised pay schedule is shown in Appendix B.
Education Policy Analysis Archives Vol. 13 No. 7 9 Evolution of the T eacher E va luation S ystem Te acher evaluation at Vaughn prior to implementing the performance pay system was taken seriously, although it was not tied directly to pay. Teacher evaluation responsibilities were a significant part of two administrators jobs and Vaughn teachers were accu stomed to being observed and evaluated. The development of a new teacher evaluation system at the same time the new pay plan was being implemented meant that there were only two months to communicate and educate teachers about the new evaluation system, co nduct observations and make decisions regarding whether a competency had been earned. The determination as to whether an individual had demonstrated a competency was based on the average of the teachers self evaluation, a peer evaluation, and an administ rators evaluation. The four level rating scale (1, 2, 3, 4) from Danielsons Framework for Teaching was used, but was open to individual interpretation ; there were no rubrics or guidelines to help evaluators determine the rating level Instances where the re were obvious discrepancies between the three ratings were handled on a case by case basis, e.g., if a teacher gave a self assessment that was much higher than the peer and administrators evaluations. The discrepancy was brought to the teachers attenti on and an opportunity given to revise the self assessment, although it was not required that the teacher do so. To receive pay for a competency, the average rating had to be 2.5 or higher. C oncerns emerged about the evaluation system during the pilot year The generic four level rating scale was not adequate to ensure either clear understanding of the expectations for each competency or for consistent rating by all evaluators on performance levels within each competency. The school committed to develop rub rics for each competency area for each level for the second year of the performance pay pilot. The main goal of more detailed rubrics was to provide clearer guidance to both teachers and evaluators at all points of the evaluation process. Beginning with t he 1999 2000 school year, all teachers were evaluated under the new evaluation system, not just those teachers in the performance pay program. This was also intended to encourage more teachers to opt into the performance pay plan because if they were goin g to be evaluated under the same system as those on performance pay, they would probably want to be eligible for the same pay enhancements. By the end of the first semester of the second year, over $107,000 was paid to teachers in the knowledge and skill based pay program, including payments of over $5,000 to several veteran teachers who attained the Level 3 rating and thus received all of Level 1 and Level 2 pay, as well as the Level 3 pay. S ome operational and administrative changes were made to the ev aluation system in the third year Probably the most visible one was the creation of a specific committee, the Performance Assessment Review (PAR) Committee Rather than relying on an ad hoc group of people with no clearly defined set of responsibilities o r relationship to the schools governance structure, t his committee was formally charged with ongoing responsibility to work on the evaluation system, including broad policy issues, as well as the administrative procedures and process. As a result of its work that year, numerous changes were made to the evaluation system, including creating a pre and post conference system, an appeals process for evaluations, methods for evaluating team teachers who do not teach all content areas, and address ing inter ra ter reliability. CPRE UW interviews with teachers near the end of that school year revealed that they clearly viewed the PAR Committee and its responsibilities as important to the overall successful functioning of the evaluation system. The schools quick response to teachers concerns about the evaluation system showed the importance of the evaluation system
Kellor: Teacher evaluation & performance pay 10 to the successful operation of the performance pay program and that the school was serious about creating a fair and trusted evaluation process. A li ngering concern on the part of both teachers and evaluators was the time required to accurately evaluate a teacher. The plan was to have two evaluation windows each semester, for a total of four evaluations from each evaluator each year. For each of the first core subject areas in which a teacher was evaluated, both the peer and administrator were to conduct a longer evaluation of up to 30 minutes of a specific lesson in the core area, supplemented by several shorter walk throughs. During the subject area observation, the evaluator would also look for evidence of the other competencies, e.g., classroom management or lesson planning. Teachers being evaluated for Level 1 competencies thus could expect evaluations in three core areas (Literacy; ESL/Sheltered English; Mathematics) four times a year, by two evaluators. For Level 2 or 3, total observation time in the five core areas (Social Studies and Science, as well as the three previously noted) could be up to about two hours by two observers, four times a ye ar. This became too burdensome for both the teachers and evaluators, and mid way through the second semester, the fourth observation was made optional. Although most teachers waived it, a few requested the fourth set of observations, especially if they tho ught an additional observation would improve their overall evaluation score. Despite the amount of time spent on classroom observations, teachers still had concerns about whether all aspects of their teaching could be evaluated accurately. The introductio n of pre and post conferences was intended to help assuage some of these concerns by providing an opportunity for teachers to bring the evaluators attention to aspects of the rubric that might not be readily observable from a routine classroom observatio n, and for evaluators to ask for evidence of aspects of teaching that were not present in a particular observation session. Feedback after the observation also was ad dressed the third year. Because Vaughn already had a separate mentor system for new teache rs t he evaluation system initially did not include feedback and was intended to be more of a summative than a formative review. I nternal comments and responses to the employee attitude survey after the first two years indicated that teachers and evaluators both wanted to be able to give and receive more feedback as part of the evaluation process and in particular, feedback about areas in which performance could be improved or evidence provided to support items in the rubric. The post conferences introduced the third year offered a partial solution to this problem by providing a specific time where the feedback could be provided. T he evaluation system also was modified to allow evaluators to provide feedback about how to improve ones performance. During th e fourth year (2001 02) w ork began on developing rubrics that would be needed when the middle school expansion began. T he PAR Committee believed that even though only one grade level would be added each beginning with sixth grade, rubrics for all three m iddle school grade levels should be developed so there would be a clear set of expectations for all the grade levels from the outset. T here also was a belief that the teacher and student standards should be developed concurrently to ensure consistency and alignment between the standards and curriculum A set of draft standards for middle school teachers were developed and were intended to be used the following school year when the school added grade 6 The subsequent year, the effort to develop performance rubrics for all grade levels and content areas at the middle school experienced a setback when the sixth grade rubrics proved difficult to administer. An interim solution was devised for the remainder of the school year: continuing faculty were paid on th e basis of their previous years evaluation, and the one new staff member was evaluated based on classroom observations by another sixth grade teacher and an administrator. The entire set of performance rubrics, both middle school and
Education Policy Analysis Archives Vol. 13 No. 7 11 elementary, were put under review and revision by the PAR Committee, with the middle school rubrics being the highest priority. New rubrics applicable to all grade levels were completed for use for the 2003 04 school year and paralleled the changes made to the performance pay program described earlier Teacher Evaluation Scores and Student Achievement: Preliminary Findings Perhaps the most asked question about any education reform is whether it improves student achievement. Clearly, t est scores at Vaughn have risen dramatica lly during the last several years, but the role or influence of the performance pay plan or evaluation system in this improvement cant be presumed. There are inherent difficulties in attempting to measure any particular teachers effect on student achieve ment, but one method that is growing in popularity is value added analysis, wherein pre and post test results are used to identify student achievement growth over a specified period of time. Vaughns standards based teacher evaluation system and the avail ability of several years of test scores for a large number of Vaughns students presented an opportunity for CPRE UW to conduct a value added analysis of student achievement at the classroom level and the relationship of classroom performance, if any, to t eachers evaluation scores. In brief, the study used a sophisticated statistical technique, hierarchical linear modeling, to compare subject area teacher evaluation scores and student subject area standardized test scores for reading, math, and language a rts for thirty four 2 nd 5 th grade teachers and all of their students for whom two years of student achievement were available. Teacher performance was measured by using teachers average score in each evaluation domain across the three observation windows in the 2000 01 school year. A pre and post test model was used, measuring value added achievement on the SAT9 test as administered in California. Of the three subject areas analyzed, a statistically significant correlation was found between the teachers average score on the literacy domain and average gains in classroom level student achievement and positive relationships in the other two subjects. The demonstrated positive relationship between a teachers evaluation score in the subject area domain and s tudent achievement in that subject area is an important finding. (Gallagher, 2004) Vaughn Teachers Reactions t o Pay a nd Evaluation Innovations The introduction of a new pay system or changes to an existing pay system can cause turmoil in an organization, and schools are no different in that regard (Odd en & Kelley, 2002) Because not all of Vaughns staff embraced the idea of a new pay system, the effects of the new pay system on the schools culture and staffs responses towards the new pay system were of interest to both the school and researchers, es pecially since there are relatively few opportunities to examine the effects of compensation changes in the K 12 public school context. At Vaughn, the teachers responses to a new evaluation system were of special significance because of the possible overl ap into their perceptions and acceptance of the new pay system. Changes to an evaluation system also can be controversial, especially so if the evaluation outcome has some influence on pay or pay increases (Kimball, 2002; Odden & Kelley, 2002) The context of Vaughns new pay system was even more interesting because the impetus for change came primarily from staff, making it more of a bottom up change than often is the case when comp ensation or evaluation systems are changed. Similarly, because the new evaluation system evolved out of a need created by the new knowledge and skill pay system, it
Kellor: Teacher evaluation & performance pay 12 could be considered more of an organic change than is typical when evaluation systems cha nge. Vaughns leadership and staff were willing to allow the school to become an ongoing research site for the CPRE UW Teacher Compensation Project, offering a rare opportunity to track the evolution of these innovation s from the first year of implementati on to the present. The schools leadership also use d the results of each years survey as a form of ongoing evaluation, leading to changes and improvements. CPRE UW researchers agreed to conduct an employee attitude survey each year, which was supplemented in most years by on site interviews of representative teachers and administrators. The survey was designed to address themes of interest related to introducing performance pay systems, as well as focusing each year on a question or area of interest to the school. The original themes identified for the survey were: 1) the new process to assess teacher competencies and performance levels; 2) the effect of the pay system on the schools sense of community and overall morale; 3) the alignment of related system s, such as professional development, with the demands of the new pay system, and 4) the extent to which the new pay system was perceived as influencing teachers decisions to work at Vaughn. Overall, the surveys had exceptional response rates, ranging f rom a low of 84 percent to a high of 99 percent. The surveys were distributed to staff at a staff development retreat just before the start of the school year so even though participation in the survey was not mandatory, teachers typically had work time to complete it. The schools culture supported its role as an innovator in education reform, and it is likely that many of the schools teachers shared that belief and thus were more likely to participate in the survey, knowing that the results could help ot hers who were making changes to teacher pay and evaluation systems. After the first and each subsequent survey, the researchers prepared a report for the school. ( Kellor & Milanowski, 2004; Kellor & Milanowski, 2003; Kellor & Milanowski, 2002, September; Milanowski & Gallagher, 2001; Milanowski & Kel lor, 2000; Milanowski & Kellor, 1999) School Based Performance Award Program Th e questions in this section remained the same each year and are relatively easy to summarize and note general trends. Overall, the bonus program generally was accepted quit e well from the outset. A few questions showed dips and rises from year to year, but there were no questions for which a majority did not agree every year, and never any indication that the program should be abandoned. From the first year on, a majority of the respondents agreed that it was fair to hold teachers accountable for student achievement, with 20 percent disagreeing the first year and only 2 7 percent disagreeing in subsequent years. Similarly, a strong majority, 80 percent or more each year, agre ed that it is fair for teachers who increase student achievement to receive a bonus, with the percent agreeing going as high as 98 percent one year. The shift to using the API as the indicator for bonus payment eligibility beginning the second year did no t appear to have any apparent effect on how Vaughns teachers felt about the performance award program. Although respondents agreed with the statement that they would work just as hard to achieve the school wide outcome goals even without the possibility of getting a bonus, a large majority of respondents also agreed each year that a s a way to measure the schools performance, the school wide outcome goals are fair ; they have a clear understanding of the school wide outcome goals ; t he bonus provides a cl ear goal that encourages school staff to work together ; t he possibility of a bonus provides a reason to make the extra effort needed to meet the goals ; t he possibility of a bonus motivates to work toward
Education Policy Analysis Archives Vol. 13 No. 7 13 the goals ; t he amount of bonus money is enough to mo tivate ; and t he school wide bonus should be continued. These responses suggest that even though teachers indicated they would work just as hard to achieve the goals if there was not money associated with it, they seemed to place some value on the prospect of receiving a bonus. Two questions relating to expectancy theory, a motivation theory, were asked the first four years. The first asked the probability or likelihood of the school meeting its outcome goals if the teacher and his/her colleagues worked to gether as hard as they could. The mean rating each year ranged from 78.3 percent to 87 percent, suggesting that there was a relatively high degree of confidence that the goal could be achieved. A similar question was asked of teachers who participated in s chool based performance award programs in Kentucky and Charlotte, North Carolina. Every year, Vaughns teachers were consistently much more positive about this outcome than either the Kentucky respondents (52.7 percent) or Charlotte respondents (62 percent ). (Kelley, Heneman III, & Milanowski, 20 01) The second question asked the probability or likelihood that the teacher would receive the pay bonus if the school met the outcome goals; the mean rating ranged from 81.1 percent to 90 percent. In an ideal world, the response would be 100 percent ev ery teacher believing that the bonus would be received if the goal were met. Nonetheless, the responses from Vaughn teachers suggest a relatively high degree of confidence that the money would be paid, perhaps reflecting the degree of trust the staff in th e schools principal and the financial ability of the school to pay the bonus. In contrast, teachers in Charlotte and Kentucky responded quite differently to a similar question. Although almost three quarters (72.8 percent) of respondents in Charlotte beli eved they would receive the bonus if the goals were met, just over half (54.8 percent) of Kentucky respondents believed they would receive the bonus if the goals were met. There are many possible interpretations for the strikingly different responses in re spondents, including size (school, district, state) and previous experiences of something being promised and not delivered, which was perceived as a factor contributing to Kentucky teachers perception that the probability of receiving the money was only s lightly better than chance. (Kelley et al ., 2001) Performance Pay and Teacher Evaluation Programs Each year the survey included questions about aspects of the performance pay and evaluation programs. The first survey did not address evaluation issues as explicitly as did subsequent surveys. R esponses the first year and feedback from interviews and sources within the school made it clear that there were strong concerns especially about the evaluation process, so specific questions about the evaluation process were added to the survey beginning the second year. Responses to the survey for the first year of the new pay and evaluation programs (199 8 1999) revealed something of a fissure between the general understanding and acceptance of the knowledge and skill pay plan, and how teachers felt abou t how it was being implemented, especially how the knowledge and skills were evaluated. Although a majority of respondents believed they had a clear understanding of the knowledge and skill areas, could explain them to others if asked, believed the knowled ge and skills were important to good teaching at Vaughn, and were motivated to increase their own knowledge and skills in those areas, the positive responses generally did not carry over to how the knowledge and skills were evaluated or what supports were available to teachers to improve their knowledge and skills.
Kellor: Teacher evaluation & performance pay 14 Some striking findings were that less than 30 percent of respondents believed that the knowledge and skill assessment process was valid, that the assessment process was fairly run, and that the way the knowledge and skills were measured was fair. Curiously, although only about 40 percent of the respondents said that the knowledge and skill pay greatly influenced what professional development the person took the previous year, more than 80 percen t said they made a lot of effort the previous year to develop the knowledge and skills that were rewarded by the pay system. The mismatch between the professional development and skill development responses suggest that people may have made individual, per sonal efforts to improve their knowledge and skills, rather than integrating the effort into formalized professional development opportunities. However, almost three quarters (72 percent) of the respondents believed that the pay program should be continued (Kellor & Milanowski, 1999) Most of the open ended responses to the question why people chose not to participate in the knowledge and skill pay system the first year revolved around process fairness issues, including the absence of specific procedures an d policies relating to the evaluation system. Other frequently noted concerns had to do with the applicability and availability of the pay program to veteran teachers and the negative effect the new pay program would have on individuals base pay. These co ncerns, along with the other sentiments expressed through the survey and other feedback mechanisms, were all influential in determining the features of the performance pay program and evaluation system when it was extended to veteran teachers the following year. Survey results for the second year of operation (1999 2000) suggested that staff had become much more comfortable with the knowledge and skill standards. In general, a higher percentage of respondents indicated that they understood and accepted the standards and felt the process was fair. For example, 71 percent (vs. 44 percent) agreed that they knew what to do to increase their knowledge and skills, and 73 percent (vs. 47 percent) agreed that they had a good understanding of how the knowledge and sk ills were measured. The perception of fairness of the evaluation process also improved, with 72 percent (vs. 28 percent) agreeing that the way the knowledge and skills were established was fair, and 74 percent (vs. 28 percent) agreeing that the way the kno wledge and skills were measured was fair. Despite these improvements, less than half (46 percent) agreed that overall, the assessment process was fairly run, and only about two thirds agreed that they got a fair evaluation and that it was accurate. As par t of an effort to examine the assessment process fairness and validity perception, the second year survey included specific questions about the evaluator. In general, respondents agreed that evaluators were knowledgeable and provided positive recognition. They were less likely to agree that the evaluators communicated expectations, gave timely feedback, and provided coaching. Because it is not clear that these areas were explicitly included as part of the evaluation process, in retrospect it is not surprisi ng that they were identified as areas of concern. T he responses to questions about the pay program itself were similar to the previous year. Most respondents felt that the pay amounts were motivating, but only a minority agreed that the pay system influenc ed their professional development choices. Fewer respondents than the previous year (64 percent vs. 83 percent) agreed that they made an effort to develop the knowledge and skills rewarded by the program. Because the program was expanded to include more ex perienced teachers, these responses could reflect an overall higher degree of confidence about ones teaching skills and less of a perception that changes in practice were needed to meet the teaching standards. The expansion of the program to include more veteran teachers did not seem to have a negative effect on how the pay system was perceived overall. The relatively minor changes that were made to the pay and evaluation programs after the second year reflected the generally high level of acceptance of th e pay and evaluation
Education Policy Analysis Archives Vol. 13 No. 7 15 programs by staff. The creation of the Peer Assistance and Review Committee that began to operate during the third year probably was the most obvious response to the ongoing concerns about assessment process fairness and validity, and the responses to the survey after th at year (see below) seemed to suggest that formalizing responsibilities for the oversight of the performance evaluation process paid off. (Milanowski & Kellor, 2000) After the third year (2000 2001), a large majority of respondents indicated that they understood the system, accepted the teaching standards, and found the process and results fair. However, while 76 percent agreed that their own evaluation was fair, only 65 percent felt it was accurate. Some persistent areas of concern continued, with just over half of the respondents agreeing that they had the professional development opportunities needed to develop the knowledge and skills, and 59 percent agreeing that they had the opportunity to develop the skills. Evalua tors were perceived somewhat more favorably than in previous years, with about 80 percent of the respondents agreeing that the evaluators were knowledgeable about good teaching but o nly about 40 percent agreeing that evaluators did a good job of communica ting expectations and gave coaching that was needed to improve ones teaching. Respondents were split almost equally in terms of how they perceived the timeliness of feedback and whether the feedback helped to improve their teaching. Thus, although the res pondents perceptions of the evaluation process seemed to have improved in the aggregate over previous years, there were still lingering areas of concern. T he responses to questions about the pay program were similar to the previous year. Most respondents felt the pay amounts were motivating and fair and for the first time, a majority of respondents indicated that the pay system influenced their professional development choices and that they made real changes to their teaching practice to receive the addi tional pay. A higher proportion of respondents (87 percent vs. 64 percent) also agreed that they made an effort to develop the knowledge and skills rewarded by the pay system. These are noteworthy responses in that they suggest that the pay system was begi nning to have a direct influence on teachers behavior. The responses to questions about the effect of performance pay overall were similar to previous years responses. A large majority of respondents agreed that performance pay had encouraged teachers to focus on improving instruction, improving student achievement, and to help each other develop skills. However, about a quarter agreed that performance pay had reduced the sense of community at the school, had hurt morale and there was more rivalry among t eachers as a result of the performance pay program. About a third of the respondents believed that teachers resented other teachers who rec eived more performance pay. These responses suggested that although a majority of Vaughns staff had a generally posi tive reaction to the performance pay program, there clearly were areas of concern and the pay and evaluation programs were not universally accepted within the school. (Milanowski & Gallagher, 2001) The survey conducted after the fourth year (2001 2002) of the new pay and evaluation programs did not allow d irect comparison of the prior years responses because many of the questions were changed to conform to similar questions being used at other CPRE UW research sites. However, broad themes can be compared, and overall, responses were similar to previous years. T he changes to the teacher evaluation system seemed to have positively affected teachers perceptions of it. About three quarters of the respondents continued to agree that: the processes and procedure s of the evaluation system were fair ; the respondents own evaluation was conducted in a fair way ; the evaluation process was run fairly ; the evaluation scores the respondent received were fair, and the evaluation process helped the respondents improve the ir teaching. However, there was a noticeable decrease from the previous year (77 percent vs. 91.3
Kellor: Teacher evaluation & performance pay 16 percent) in those who agreed that working towards improving ones performance on the scoring matrix would really help teachers teach better. In general, resp ondents reported that they had a good understanding of the scoring matrix and of the evaluation system itself. Almost two thirds of the respondents agreed that professional development opportunities were available to develop the knowledge and skills reward ed by the pay system, and more than three quarters agreed that they had the opportunity to develop the knowledge and skills that she or he needed to do well. This suggested that although professional development opportunities were not always perceived as b eing available, the majority of respondents apparently found ways to develop the necessary knowledge and skills. Respondents attitudes towards the evaluation process generally were favorable although all aspects of the peer evaluator part of the process were perceived less favorably than the administrator parts of the process. The reasons for this disparity were not clear For example, it could be that f eedback and ratings by peers inherently are perceived differently by teachers than feedback and rating s by administrators, but that can not be determined without thoroughly reviewing the evaluation process to identify whether there were real differences between the two categories of evaluators. New teachers appeared to be somewhat more positive about the i nfluence that performance pay has on teachers behaviors and actions They were more likely to agree that performance pay encouraged teachers to focus on improving instruction, that it encouraged teachers to help each other develop their knowledge and skil ls, and that it encouraged teachers to focus on improving student achievement. The effect of knowledge and skill based pay on professional development activities and actual changes to instruction was reported as somewhat less than the previous year. A s the schools staff became more familiar with the teaching expectations and more experienced in teaching in that manner, they might have felt less of a need to change their instruction. Vaughn also tried each year to offer school wide professional development activities that supported development of the desired knowledge and skills, so perhaps staff did not feel as great a need to identify such professional development on an individual basis. Despite the bottom up origins of the new pay system, the perception t hat it reduced the sense of community among teachers continued, with 15 percent (vs. 27% the prior year) of veteran teachers agreeing with that statement. Nonetheless, a strong majority of respondents agreed that the performance pay system should be contin ued, suggesting that the performance pay system is relatively well accepted by the schools teachers. Of special interest to Vaughn administrators the fourth year was whether teachers were concerned about the potentially varying income that they could rece ive each year under the performance pay system. Two questions addressed this concern. Teachers were asked how much more or less stable they think their future income is under Vaughns pay system, compared to Los Angeles Unified School Districts traditiona l salary system. About a quarter of veteran staff felt that their income was somewha t or much less stable, whereas less than ten percent of new staff responded in that manner. At the same time, 43 percent of new teachers and 30 percent of veteran teachers felt that their income was about the same, and about half of both new and veteran teachers felt that Vaughns pay system was somewhat more or much more stable than the Los Angeles districts. (Kellor & Milanowski, 2002) After the fifth year (2002 03), surv ey results relating to pay satisfaction continued to show that the performance pay system seemed to have more influence on new teachers decisions to work at Vaughn than it did for veteran teachers, with 57 percent of new teachers agreeing that the pay sys tem was an important reason why they chose to teach at Vaughn,
Education Policy Analysis Archives Vol. 13 No. 7 17 compared to 25 percent of veteran teachers who agreed with that. All of the new teachers who responded agreed that they have the opportunity to make more at Vaughn than they do elsewhere, compa red to just over half of the veteran teachers who agreed. Finally, while 79 percent of the new teachers agreed that the performance pay system made working at Vaughn more attractive to them, only 49 percent of the veteran teachers agreed with that statemen t. It is hard to determine whether the pay systems appeal to new teachers reflect ed primarily the structural differences between the traditional pay scales and the performance pay system, which has a competitive starting base rate and offers more opportun ities for new teachers to earn additional pay, or if it reflect ed philosophical differences between how new teachers and veteran teachers view their pay opportunities. (Kellor & Milanowski, 2003) As noted previously, significant changes were made to the PA R process and the performance pay program for the 2003 04 school year. Overall, the survey responses showed mostly favorable attitudes towards the new PAR system, and most areas showed an increase in the percentage of favorable responses. One potential are a of concern was in response to the training on the new PAR system. Just over half felt they received enough training, and about a quarter were neutral about the amount of training provided. Given the direct linkage of the evaluation system to the performa nce pay system, it is important to provide adequate training on the PAR system, and the responses suggest the need to improve the training. However, the responses to questions about the fairness of the evaluation process and the clarity of expectations we re generally favorable, and 70% agreed that the new evaluation system was an improvement over the previous one. If the increases in favorable responses are attributable to the changed rubrics, it could suggest that teachers are more comfortable thinking of their teaching in a more general and holistic way, rather than the more specific strategies and concepts that were associated with the content specific rubrics used in previous years. Overall, the evaluation system continued to have a high degree of accep tance, with just over three quarters of respondents indicating that the school has an excellent performance evaluation system. One of the reasons cited for changing the teacher evaluation rubrics from content specific to more general teaching domains was to address concerns that only particular types of teaching were recognized under the previous system. Thus, a new question asked whether changes to the PAR system in 2003 04 made it easier for the respondent to receive the score needed for performance pa y; 60% agreed that it was easier, with 29% neutral. An increase in the number of teachers receiving performance pay could be an indicator of whether the perception of the new PAR being easier than the old one was valid, although the school has not yet co nducted an analysis of the actual pay pay outs under the two systems. Overall, both veteran and new staff had positive attitudes about the performance pay system. The responses of new staff were supportive of the pay program but the extent to which it infl uenced their decision to teach at Vaughn wasnt clear. For example, although 76% of the new teachers agreed or strongly agreed that they would have the opportunity to make more money teaching at Vaughn than . elsewhere, only 34% agreed that the perf ormance pay system is an important reason [they] chose to teach at Vaughn. For veteran teachers, the responses generally were about the same as in previous years, although fewer teachers agreed that performance pay increases rivalry between teachers, and that teachers who get more performance pay are resented by those who dont do as well. The majority of veteran teachers, 73%, agree that the system should be continued. (Kellor & Milanowski, 2004)
Kellor: Teacher evaluation & performance pay 18 Wrapping It Up: Lessons from Vaughns Experience A stan dard phrase in any version of Vaughns history is that it is a conversio n charter school. After six year s of a knowledge and skill based pay program and associated standards based teacher evaluation system, it could rightfully be called a conversion per formance pay school, with all staff now paid on performance pay systems. Although Vaughn is an exceptional school in many ways, its transformation to a performance pay driven organization has not been without some difficulties, some of which the pay plan advocates expected and others which were surprises. One of the reasons the Vaughn staff pursued a new pay system was to use the advantages of flexibility and innovation that being a charter school gave them, so that the lessons they learned from their pay experiment could be shared with others who might be in settings where the ability to take risks and be innovators was less supported. This last section highlights key points that could help make the journey to knowledge and skill based pay or standards bas ed teacher evaluation a bit less arduous for others who are ready to move towards those education reforms. Expect changes, but dont change too much or too often From the outset, Vaughns pay plan developers and designers of the teacher evaluation syste m knew that changes would be necessary, both to refine rough edges as well as to address unforeseen complications or situations. Some years saw more significant changes than other years, but efforts were made to make only those changes that were essential each year. This was done for several reasons, including the ability to accurately project costs for future years, to help teachers gain experience and trust in the new programs, and to be certain that only those changes that truly were necessary were made. Encourage involvement of representative people in designing the system, but recognize the need for a change champion at times. Vaughns governance structure requires teacher, administrator and parent involvement in almost every important decision rela ting to the school. The knowledge and skill pay plan and new evaluation system, although not initially designed by a formally designated group, followed the same inclusive principles. This was important because it ensured that the people most directly affe cted by the new programs the teachers were directly involved in key decisions about the programs. The governance structure included an approval process that ensured that broader concerns of all teachers, and more important, of students, would be considered before the pay and evaluation programs were put into place. The principal played an important role throughout this, providing support, direction and encouragement, along with ensuring that any data or assistance the designers needed was provided. On the f lip side, the principal believed that she might have stepped back too soon, or inappropriately, at a time when the designers were facing controversial and emotional issues. Although she did this to empower the teachers and make it clear they had authority to design the programs, her absence of leadership at a critical time might have made the expansion of the program to veteran staff more difficult. However, it also is possible that the heated discussions that took place when the program was designed and th e true colors that were exhibited at this time could have had a cathartic effect in that peoples positions and concerns were made clear. Others involved in designing a new compensation system should consciously consider the positive and negative implica tions of a leader stepping away from a leadership role at critical points. Even in a collegial setting, expect some fallout from changes for which there is no objective way to determine whats best or fair. According to interviews and anecdotal comm ents made to researchers, the Vaughn staff generally considered themselves to
Education Policy Analysis Archives Vol. 13 No. 7 19 be a collegial group that regularly put students interests ahead of other interests evidenced in part by the fact that the staff who remained at Vaughn when it became a charter school made a conscious choice to stay rather than return to a district position. Yet when it came to bottom line decisions that had to do with individual pay levels, some of that collegiality seemed to disappear and differences began to emerge about how people felt about important things like how individual effort should be recognized and compensated, and if describing the elements of good instruction took away from individual teacher autonomy and diminished the art of teaching. This probably should n ot have been a surprise, especially when changing something as long standing as the single salary schedule, but it seemed to take many people by surprise Over time, Vaughns sense of collegiality and morale seem to be recovering, perhaps aided by the grad ual attrition of those staff who did not agree with the changes. Whether the schools sense of community would have been restored if a significant number of performance pay dissenters had stayed at the school is unknown, but places contemplating changes to their pay systems should consider the possibility that there will be splits among the staff and develop strategies to rebuild the community Communication is essential. Encourage the free flow of information, do what you can to ensure that accurate inf ormation is being presented, and establish and document clear, written policies and procedures so that the rules and expectations are known from the outset. Vaughns pay plan and evaluation system designers did an admirable job of identifying important iss ues in advance and addressing them, either by a specific decision or by establishing policy directions to guide future decisions and make the programs goals clear to teachers. Yet in some important areas the planners knowingly proceeded without specifying details (e.g., implementing the performance pay program at the beginning of the school year and working out the details of the evaluation process during the first several months of school) and in other areas (e.g., creating descriptions of performance lev els in the evaluation system) its effort fell short. In Vaughns environment, the lack of specificity created tension the first year and was a key priority for the second year of the program, but for many schools, the lack of specificity in areas as critic al as these could have been a fatal flaw. The quick response to resolve the problems probably helped give the new programs credibility and to gain teachers trust, but implementing a program without knowing and/or communicating details and policies introdu ces an even greater element of risk to the programs success and is not the preferred method for an effective implementation Transition details from one pay system to another are important; minimizing risk to veterans while meeting the goals of a new pro gram can be a delicate balance to reach. Vaughns approach to implementing a new pay system was to begin with the group of teachers who were likely to have less invested in the current pay system, i.e., new and early career teachers. Although new teachers were required to be part of the new pay system, even early career teachers were given the option to stay under the existing pay plan or to become part of the pilot. When the program was offered to veteran teachers sooner than originally planned, attention to the details of the transition were an important part of the design process, from establishing base rates that were somewhat reflective of the veterans pay under the traditional salary schedule, to setting up a method to essentially keep a teachers mon thly pay close to its previous level on at least a temporary basis so that there would not be a precipitous drop in income before the performance pay was paid out. Another way that Vaughn dealt with transition issues was in structuring the pay system for v eteran teachers so that some vestiges of experience were retained in the salary structure, both in the form of annual increases up to some point and by creating advanced performance levels that realistically would only be attainable by the more experienced teachers. (Vaughns approach was still quite aggressive in that a teacher
Kellor: Teacher evaluation & performance pay 20 could end up with net compensation lower than the previous compensation, but given the overall confidence in teachers ability to meet the performance pay standards the risk of losi ng net pay did not seem as real as it might have in other school settings.) Accommodating veterans interests in the new salary structure in a way that was still compatible with the long term goals of the performance pay program and the evaluation system and identifying transition issues and providing ways to deal with them, very likely made the transition to the new systems more attractive and easier than if the pay system completely ignored the lingering influence of the traditional pay structure. In br ief, Vaughns multi year experience with its knowledge and skill based pay and standards based teacher evaluation system provide useful insights to others who are considering or are in the midst of teacher compensation and evaluation reform projects. Altho ugh the schools status as a charter school makes it difficult to generalize the exact circumstances of its compensation and evaluation reforms, the lessons derived from its experiences the past several years are useful regardless of the context. The evide nce of the predictive ability of its teacher evaluation system on student achievement (that the higher the evaluation score, the higher the learning gains produced in that teachers classroom) is exciting and the fact that the standards based evaluation sy stem was introduced to meet the needs of the knowledge and skill based pay program lends credence to the recommendation to aligning internal systems and resources so they support the same goals. Note This article was prepared for the Consortium for Polic y Research in Education, Wisconsin Center for Education Research, University of Wisconsin Madison. The research reported in this paper was supported by a grant from the U.S. Department of Education, Office of Educational Research and Improvement, National Institute on Educational Governance, Finance, Policymaking and Management, to the Consortium for Policy Research in Education (CPRE) and the Wisconsin Center for Education Research, School of Education, University of Wisconsin Madison (Grant No. OERI R308A 60003). The opinions expressed are those of the authors and do not necessarily reflect the view of the National Institute on Educational Governance, Finance, Policymaking and Management, Office of Educational Research and Improvement, U.S. Department of E ducation, the institutional partners of CPRE, or the Wisconsin Center for Education Research. References California Department of Education. (2003). Academic Performance Index (API) California Department of Education, Policy and Ev aluation Division. Retrieved April, 2003, from the World Wide Web: http://www.cde.ca.gov/psaa/api/fallapi/apiinfo.pdf Danielson, C. (1996). Enhancing Professional Practice: A Framework f or Teaching Alexandria, VA: Association for Supervision and Curriculum Development. Gallagher, H. A. (2004). Vaughn elementary's innovative teacher evaluation system: Are teacher evaluation scores related to growth in student achievement? Peabody Journa l of Education, 79 (4), 79 107.
Education Policy Analysis Archives Vol. 13 No. 7 21 Kelley, C., Heneman III, H., & Milanowski, A. (2001). School Based Performance Awards: Research Findings and Future Directions. Educational Administration Quarterly Kellor, E., & Milanowski, A. (2004). Vaughn Next Century Learning Center Performance Pay Survey School Report. Madison: University of Wisconsin, Wisconsin Center for Education Research, Consortium for Policy Research in Education. Kellor, E., & Milanowski, A. (2003). Vaughn Next Century Learning Center Perform ance Pay Survey School Report Madison: University of Wisconsin, Wisconsin Center for Education Research, Consortium for Policy Research in Education. Kellor, E., & Milanowski, A. (2002, September). Vaughn Next Century Learning Center Performance Pay Sur vey School Report Madison: University of Wisconsin, Wisconsin Center for Education Research, Consortium for Policy Research in Education. Kellor, E. M., Milanowski, A., Odden, A. R., & Gallagher, H. A. (2001). How Vaughn Next Century Learning Center Dev eloped a Knowledge and Skill pay Program (Working paper TC 01 1). Madison, WI: University of Wisconsin, Wisconsin Center for Education Research, Consortium for Policy Research in Education. Kimball, S. M. (2002). Analysis of Feedback, Enabling Conditions and Fairness Perceptions of Teachers in Three School Districts with New Standards based Evaluation Systems (Submitted or Publication to the Journal for Personnel Evaluation in Education.). Madison, WI: University of Wisconsin Madison, Wisconsin Center for Education Research, Consortium for Policy Research in Education. Milanowski, A., & Gallagher, A. (2001). Vaughn Next Century Learning Center Performance Pay Survey School Report. Madison: University of Wisconsin, Wisconsin Center for EducationResearch, Consortium for Policy Research in Education. Milanowski, A., & Kellor, E. (2000). Vaughn Next Century Learning Center Performance Pay Survey School Report. Madison: University of Wisconsin, Wisconsin Center for Education Research, Consortium for Policy R esearch in Education. Milanowski, T., & Kellor, E. (1999). Vaughn Next Century Learning Center Performance Pay Survey: Preliminary Report Madison: University of Wisconsin, Wisconsin Center for Education Research, Consortium for Policy Research in Educa tion. Odden, A., & Kelley, C. (2002). Paying Teachers for What They Know and Do: New and Smarter Compensation Strategies to Improve Schools (2nd ed.) Thousand Oaks, CA: Corwin Press.
Kellor: Teacher evaluation & performance pay 22 About the Author Eileen M. Kellor Associate Researcher Consortiu m for Policy Research In Education University of Wisconsin Madison 1025 West Johnson Street Madison, WI 53706 (608) 263 4260 firstname.lastname@example.org Eileen Kellor is Associate Researcher, Wisconsin Center for Education Research, University of Wisconsin Madison. As part of the Consortium for Policy Research in Educations Teacher Compensation Project, she has provided technical assistance and participated in research activities related to teacher compensation and teacher evaluation innovations.
Education Policy Analysis Archives Vol. 13 No. 7 23 Appendix A: Vaugh n Pay Summary 1998 2003 1998 99 1999 00 2000 01 2001 02 2002 03 Year 1 30,000 31,500 31,500 33,150 33,150 Year 2 31,000 32,500 32,500 34,150 34,150 Year 3 32,000 33,500 33,500 35,150 35,150 Year 4 1 33,000 34,500 34,500 36,150 36,150 Year 5 34,000 36 ,500 36,500 38,150 38,150 Year 6 2 38,000 38,000 41,650 41,650 Year 7 39,500 39,500 43,150 43,150 Year 8 41,000 41,000 44,650 44,650 Year 9 42,500 42,500 46,150 46,150 Year 10 44,000 44,000 47,650 47,650 Year 11+ 45,500 45,500 49,150 49,150 Yea r 15+ 47,500 47,500 51,150 51,150 CA credential 1,000 2,000 2,000 2,000 2,000 MA 3 1,000 2,000 2,000 2,000 2,000 NBPTS 4,000 4,000 4,000 4,000 4,000 Level 1: (depth in essentials; requires average d self peer administrator rating of 2.5 & clear credential) Literacy 1,300 1,300 1,300 1,300 1,300 ESL, Sheltered English 4 1,300 1,300 1,300 1,300 1,300 Technology 400 400 ---Mathematics 1,000 1,000 1,000 Special Ed inclusion 300 300 300 300 300 Classroom mgt 100 100 300 300 Lesso n planning 100 100 300 300 Level 2: (depth +breadth after achieving 3.0 in all Level 1 areas) Mathematics 1,000 ---Social studies 800 800 900 900 Science 800 800 900 900 English learners support (BCLAD) 2,500 2,500 2,500 Strand 1: 2,500 S trand 2: 2,000 Strand 3: 1,000 2,500 Strand 1: 2,500 Strand 2: 2,000 Strand 3: 1,000 Technology -500 500 500 Physical education --500 500 Arts 500 500 500 500 Level 3: If average rubric >3.5 in level 2: 4,000 4,000 4,500 4,500 Level 4: Distinguished teacher After 5 --2,000 2,000 1 From year 4 on, must have an evaluation of average or better to receive annual pay increase 2 To receive Year 6 and above pay levels, must possess a clear California elementary teaching credential 3 Beginning 1999 2000, paid for holding a Master s degree in education, OR for having 30 units after obtaining clear credential 4 Called English language development beginning 1999 2000.
Kellor: Teacher evaluation & performance pay 24 1998 99 1999 00 2000 01 2001 02 2002 03 consecutive years at Level 3 Contingency based Student attendance 25 0 250 250 250 250 Student discipline 25 0 250 250 250 250 Parent partnership 15 0 250 250 250 250 Teaming efforts 15 0 250 25 0 500 500 Outcome based Schoolwide achievement 1,5 00 Schoolwide redesignation 25 0 Grade level achievement 50 0 Individual class achievement 50 0 API (state reform) 1,50 0 1,500 2,000 2,000 Expertise based Committee chair 50 0 500 500 500 500 Clan leader 50 0 500 500 750 750 Faculty chair 50 0 500 500 500 500 Mentor 50 0 500 500 500 Staff dev leader Per ses sio n $25/ hour Parent ed leader Per ses sio n $25/ hour Afterschool 50 0 500 500 $25/hour $25/hour Extended year Ex tr a mo nth pay Per polic y Per policy Per policy Per policy Peer reviewer 500 750 Gainsharing Substitutes 25 0 250 250 250 250 Stock options (minimum) -1,00 0 1,000 1,000 1,000
Education Policy Analysis Archives Vol. 13 No. 7 25 Appendix B: Vaughn Next Century Learning Center Performance Based Pa y Plan Effective 2003 04 School Year Base Salary Base salary, with CA credential Contingency based Year 1 1 33,150 38,900 Student attendance At least 95%: $250 >98%: $1,000 Year 2 34,150 39,900 API (state reform) $2,000 Year 3 35,150 40,900 Ex pertise based Year 4 36,150 41,900 Committee chair $500 Year 5 38,150 43,900 Clan leader $750 Year 6 2 45,400 Faculty chair $500 Year 7 46,900 Peer Reviewer $1,000 Year 8 48,400 Year 9 49,900 Gainsharing Year 10 51,400 Substit utes $250 Year 11+ 52,900 Contingency based Year 15+ 54,900 Student attendance At least 95%: $250 >98%: $1,000 MA in Ed OR 30 units after clear credential $2,000 NBPTS $4,000 Level 1: (depth in essentials; requires averaged self peer administra tor rating of 3.0) Domain 1: Planning and preparation $1,000 Domain 2: Classroom environment $1,775 Domain 3: Instruction $1,775 Domain 4: Professional responsibility $1,000 Total Level 1: $5,550 Level 2: Clear Credential Teachers (depth + breadth a fter achieving 3.0 in all Level 1 areas and average is >3.5) $5,500 Level 3: Distinguished Teacher (after 8 out of 10 consecutive semesters at Level 2) $2,000 1 Teachers with satisfactory evaluation (2.5) will advance yearly 2 To receive Y ear 6 and above pay levels, must possess a clear California elementary teaching credential
Kellor: Teacher evaluation & performance pay 26 Education Policy Analysis Archives http://epaa.asu.edu Editor: Sherman Dorn University of South Florida Production Assistant: Chris Mur rell, Arizona State University General questions about appropriateness of topics or particular articles may be addressed to the Editor, Sherman Dorn, epaa editor@shermando rn .com EPAA Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Linda Darling Hammond Stanford University Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Gene V Glass Arizona State University Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laborator y William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Gary Orfield Harvard University Anthony G. Rud Jr. Purdue University Jay Paredes Scribner University of Missouri Michael Scriven Western Michigan University Lorrie A. Shepard University of Colorado, Boulder Robert E. Stake University of Illinois UC Kevin Welner University of Colorado, Boulder Terrence G. Wiley Arizona State University John Willinsky University of British Columbia
Education Policy Analysis Archives Vol. 13 No. 7 27 Archivos Analticos de Polticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998 2003) Roberto Rodrguez Gmez Editorial Board Hugo Aboites Unive rsidad Autnoma Metropolitana Xochimilco Adrin Acosta Universidad de Guadalajara Mxico Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educacin, Chile Dalila Andrade de Ol iveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educacin, Argentina Teresa Bracho Centro de Investigacin y Docencia Econmica CIDE Alejandro Canales Universidad Nacional Autnoma de Mxico Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagoga Popular, Per Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernndez Enguita Universidad de Salamanca. Espaa Gaudncio Frigotto Universidade Estadual do Rio de Jane iro, Brasil Rollin Kent Universidad Autnoma de Puebla. Puebla, Mxico Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California Mara Loreto Egaa Programa Interdisciplinario de I nvestigacin en Educacin Mariano Narodowski Universidad Torcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Polticas Educativas, Per Vanilda Paiva Universidade Estadual do Rio de Janeiro, Brasil Miguel Pereira Catedratico Universidad de Granada, Espaa Angel Ignacio Prez Gmez Universidad de Mlaga Mnica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de So Paulo Diana Rhoten Social Science Research Council, New York, New York Jos Gimeno Sacristn Universidad de Valencia, Espaa Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, Mxico Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas Publicas Universidad de Buenos Aire s, Argentina Antonio Teodoro Universidade Lusfona Lisboa, Carlos A. Torres UCLA Jurjo Torres Santom Universidad de la Corua, Esp aa