xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20049999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00386
Educational policy analysis archives.
n Vol. 12, no. 37 (August 05, 2004).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c August 05, 2004
Influence of socioeconomic factors on Kentuckys public school accountability system : does poverty impact school effectiveness? / Robert Lyons.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 12issue 37series Year mods:caption 20042004Month August8Day 55mods:originInfo mods:dateIssued iso8601 2004-08-05
EDUCATION POLICY ANALYSIS ARCHIVES A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright is retained by the fi rst or sole author, who grants right of first publication to the Education Policy Analysis Archives EPAA is a project of the Education Policy Studies Laboratory. Articles are indexed in the Directory of Open Access Journals (www.doaj.org). Volume 12 Number 37 August 5, 2004 ISSN 1068-2341 The Influence of Socioe conomic Factors on KentuckyÂ’s Public School Accountability System: Does Poverty Impact School Effectiveness? Robert Lyons Murray State University Citation: Lyons, R. (2004, August 5). Th e influence of socioeconomic factors on KentuckyÂ’s public school accountability system: Does poverty impact school effectiveness? Education Policy Analysis Archives, 12 (37). Retrieved [date] from http://epaa.asu.ed u/epaa/v12n37/. Abstract Under the Commonwealth Accountability Testing System (CATS), KentuckyÂ’s public schools have been assigned individualized baseline and improvement goal indices based upon past school performance in relation to the 2014 statewide index goal of 100. Each schoolÂ’s CATS Accountability Index, a measure of school performance based upon both cognitive and non-cognitive measures, has then been compared to these individualized improvement goals for the purpose of designating schools as Meet Goal Progressing and Assistance Level (Kentucky Department of Education (KDE), 2000). Considered an interim target model, the design of CATS has been intended to negate the biasing effects of socioeconomic factors on school performance on accountability tests through the individua lization of school goals (Ladd. 2001). Results of this study showed that 39.9% to 55.5% of the variance of the CATS indices was shared by school socioeconomic factors. Analysis of this interim target model for the 2000-2002 biennium showed that for elementary and middle schools this model negated the biasing effects of socioeconomic factors, but not for high schools. Moreov er, analysis of the progress of schools toward their Improvement Goals in 2001 showed that both elementary and high schools from higher poverty backgrou nds lagged significantly behind their more affluent peers, indicating inequitable capacity to meet improvement goals
Education Policy Analysis Archives Vol. 12 No. 37 2 between the poorest and most wealthy sc hools. Adaptations to the present accountability systems were suggested for the purpose of providing more accurate information to the public regarding the effectiveness of public schools in Kentucky. The Commonwealth Accountability Testing Sy stem (CATS), KentuckyÂ’s public school accountability system, has been designed to communicate to the pu blic the progress of schools toward KentuckyÂ’s education goals in terms of aggregated student achievement, inclusive of cognitive and non-cognitive measur es. To determine rewards or sanctions, an accountability index has been calculated fo r each school based primarily on assessment results. This accountability index has th en been compared to a school specific improvement goal for each biennial accountability cycle (KDE, 2000). Although the CATS system has been designed so that by 2014 the accountability index of all schools will have achieved the stat ewide goal of 100, or that schools will have been held accountable for their performance, it stands to reason that interim public opinion, interim public policy, or both regard ing KentuckyÂ’s public schools will have been shaped by the Meeting Goal, Progressing, or Assistance Level designations assigned to schools based upon the obtainment or non-obtainment of these biennial improvement goals (KDE, 2000). Because of the impact of school performance designations on public opinion, it has been critical that the accoun tability system communicate accurately to the general public, parents and the school commu nity the relative effectiveness of public schools. Since approximately 90% to 95% of each schoolÂ’s accountability index has been based upon student assessment results (KDE, 2000), and since approximately 52% to 62% of the variance in aggregate school or sc hool district performance on accountability assessments has recently been shown to vary with student socioeconomic factors in both Illinois and Ohio (Lyons, 2001; Sutton & Sode rstrom, 1999; Wilson & Martin, 2001), the question follows as to whether the CATS system has identified Kentucky schools as Meeting Goal that have produced high st udent performance with respect to the socioeconomic background of their students, or simply schools with favorable socioeconomic factors? Moreover, do schools with favorable socioeconomic factors have the same capacity to meet their respective Improvemen t Goals as their higher poverty peers as evidenced by the early attainment of these goal s? Therefore, the purposes of this study were as follows: (1) To determine whether a significant relationship existed between school and community socioeconomic variables and 2000-2002 school-level accountability indices for KentuckyÂ’s public elementary, middle, and high schools. (2) To determine whether the application of individualized improvement goals to classify school performance negates the biasing effects of the socioeconomic factors as evidenced both by KentuckyÂ’s 2000-2002 biennial school performance classifications for public elementary, middle, and high schools. (3) To determine whether socioeconomic factors related to the midpoint attainment of 2000-2002 improvement go als for KentuckyÂ’s public elementary, middle, and high schools.
Influence of Socioeconomic Factors 3Review of Related Literature School accountability systems, which have generally been composed of standards for student performance that articulate statewi de goals, assessment systems that measure student progress toward performance standard s, and a system of rewards and sanctions regarding aggregate school progress toward st ate goals, have been widely viewed as controversial among educators. The ability of standardized measures to accurately capture student achievement, as well as the manner in which these measures have been applied to a system of rewards and sanctions, have become the focal point of the debate concerning school accountability (Hanushek & Raymond, 2001 ; Ladd, 2001; Pearson, Vyas, Sensale, & Kim, 2001). The Public View of School Accountability Although contentious in the educat ion community, the concept of school accountability has been popular with the Americ an people. In a survey of Americans prior to the 2000 election (Business Roundtable, 2000) regarding issues termed Extremely Important to respondents as voters when selecting a Presidential and Congressional candidate, 61% identified Improving Schools a response rate which eclipsed other prominent issues such as Protecting Social Security (56%), Encouraging Traditional Moral Values and Standards (48%), Protecting Patients Healthcare Rights (47%), and Providing Healthcare Coverage (42%). It was interesting to note that pub lic interest in improving education surpassed all other policy issues in the minds of voters during the 2000 election. Additionally, an annual nationwide survey of attitudes toward educat ion conducted in 2001 indicated that when respondents were asked to grade their local public schools on a scale of A to F, 51% assigned a grade of A or B. This approval rating was the highest received on the poll since 1989 (Rose & Gallup, 2001). The results of the aforementioned surveys pointed to a heightened interest in improving public schools amidst an increasi ng level of satisfaction with respondentsÂ’ respective local schools. On its face a mixed message, the results of the surveys taken together communicated a perception on the part of the general public that their local schools have been performing satisfactorily, an d yet might still be improved. Moreover, as evidenced by a mere 23% A or B designati on, respondents showed little confidence in public schools when asked about the quality of public schools nationwide. In short, survey results implied that the majority of the general public was satisfied with their local public schools, but dissatisfied with what they unders tood to be true about public schools in other communities and states (Business Ro undtable, 2000; Rose & Gallup, 2001). Public support for accountability Whether motivated by specific concern over local schools or a general concern over public schools across the country, survey respondents favored the use of systems of testing and accoun tability for public schools. Specifically, respondents were polled as to whether they fa vored President BushÂ’s initiative to hold schools accountable for student performance on standardized tests, with 75% favoring such an initiative. Moreover, although only 31% of respondents indicated that they felt standardized tests were the best way of asse ssing student achievement, 55% supported the increased used of standardized tests for accountability purposes (Rose & Gallup, 2001). Clearly public opinion has supported increa sed school accountability, a trend not unnoticed by federal and state policymakers as a variety of state-level education reform initiatives with accountability provisions ha ve been enacted in recent years (Council of
Education Policy Analysis Archives Vol. 12 No. 37 4 Chief State School Officers (CCSSO), 2000). Wi th the signing into law of Â“No Child Left BehindÂ” in January 2002, the shift toward increased public school accountability has reached the federal-level as well (Cen ter on Education Policy (CEP), 2002). Purpose and Components of Accountability Models Since public opinion regarding the quality of individual schools has been shaped by the specific measures referenced by accoun tability systems (e.g., test scores, student attendance rates, student dropout rates) and reported to the public through school reports cards or other media, the measures used to characterize school performance and the structure of the accountability systems in gene ral have been key issues in the debate over school accountability (Hanushek & Raymond, 20 01; Ladd, 2001; Pearson, et al., 2001). In general, public school accountability models have been constructed around a framework of systemic goals, standards of school performance, a means of school performance measurement, and a system rewa rds and sanctions assigned to schools based upon varying levels of school performance (Hanushek & Raymond, 2001). These accountability systems have sought to leve rage change by opening schools to public scrutiny for the purpose of placing pressure upon schools to take steps to increase student test scores (Gullatt & Ritter, 2000; Ladd, 2001). School progress toward state goals and standards has generally been publicized through detailed reports of school improvement indicators, referred to as school report cards. As of September 2000, all state education agencies had adopted the policy of issuing on an annual basis at least one school accountability indicator report. Forty-six states had adopted as policy th e issuance of accountability indicator reports disaggregated at the school district level, and forty states disaggregated accountability indicators at the school-level (CCSSO, 2000). States issuing a ccountability reports at the school-level in addition to the district-level did so to prevent school districts from hiding poor performing schools within the aggregated results of the district (Ladd, 2001). Goals and standards. Statewide educational goals and performance standards have served both economic and equity purposes. Ec onomically, goals and standards have been aimed at closing the gap between the achievemen t of United States public school students and their international peers. Additionally, goals and standards have been viewed as a mechanism for achieving educational equity by raising the bar for all students, thereby reducing the disparity in achievement betwe en disadvantaged students and their peers (Gratz, 2000). Whether focused on economic impact or equity, statewide goals and standards have been designed to serve as a catalyst for increasing student achievement (Hanushek & Raymond, 2001). It has been noted that statewide goals, which often have been embedded within legislation, have tended to be lofty, ye t overly vague and ambiguous (Hanushek & Raymond, 2001). Moreover, although focused on bringing about increases in student achievement (Hanushek & Raymond, 2001) accountability goals and standards have tended to over-promise and then under-perf orm, often as the result of improper policy implementation (Gratz, 2000). Although impe rfect, with only 22 states receiving a grade of Bor higher in a recent report on state accountability systems, statewide educational goals and standards in all core subject area s have been adopted by 48 states and the District of Columbia, the exceptions being Rhode Island and Iowa (Â“Quality CountsÂ”, 2002). Student Assessment. The identification or development of valid and reliable measures of student performance for accoun tability purposes has generated significant
Influence of Socioeconomic Factors 5controversy (Ladd, 2001). Critics have mainta ined that the multiple-choice format of most norm-referenced standardized tests has been incapable of capturing what students have known and have been able to do (Wiggins as reported in Pearson, et al., 2001). This criticism has led to the development of criteri on-referenced tests aligned to state standards and administered either in place of or in addition to multiple-choice, norm-referenced tests. The format of these criterion-referenced tests has included extended response and short answer questions (Â“Quality CountsÂ”, 2002). As of January 2002, 37 states use crit erion-referenced tests for accountability purposes in English and math at least once at the elementary, middle, and high school levels. Fourteen states have administered both criterion-referenced and norm-referenced tests in English and math in grades three thru eight. The validity and reliability of these tests has been debated, with only 19 states having had their criterion-referenced tests externally aligned and reviewed. Additionally, only two states (Kentucky, Vermont) have incorporated student portfolios into their a ccountability system (Â“Quality CountsÂ”, 2002). Consequences for individual and school performance An underlying assumption of accountability systems has been that in the absence of real consequences for school or individual success or failure, there has been insufficient motivation to focus on the desired outcomes (Hanushek & Raymond, 2002). In 2000-2001, 18 states assigned rewards to schools performing at or above state standard s, with two more states phasing in such policies over the next three years. Sanctions of one type or another were in place in 20 states for schools that were persistently low performing, with 11 states allowing student transfers, 9 allowing closure, 15 providin g for school reconstitution, and 2 states withholding funds from low performing schools. It was noted that in 2000-2001, six states provided for rewards for high-performing school s without the threat of sanctions for lowperforming schools, and seven states provid ed for sanctions to low-performing schools without the incentive of rewards for high-performance (Â“Quality CountsÂ”, 2002) Policies concerning individual accountability in 2000-2001 were more varied than school or district policies. Four states base d decisions regarding grade-level promotion on individual performance on the statewide asse ssment. Seventeen states based decisions regarding high school graduation on statewide exit or end-of-course exams, with six states basing exit exam or end-of-course assessments on tenth grade standards. It was noted that eight states provided for consequences for low-performing schools without holding individual students accountable, and four states provided for consequence for students performing poorly on state assessments without holding low-performing schools accountable (Â“Quality CountsÂ”, 2002). Federal accountability policy The passage of No Child Left Behind (NCLB) in January 2002 has provided for federally mandated consequences for schools failing to meet state specified goals, much as with existing accountability measures in many states. However, federal legislation has called for states to disaggregate student scores on mandatory tests. These desegregations must occur for students grouped in terms of ethnicity, income, disability, limited English proficiency and migr ant student. Schools must meet individual goals regarding the achievement gap that has been shown to exist between such groups (i.e., the gap between poor students and their peers) or face a range of consequences (CEP, 2002). Specifically, schools failing to meet their performance goals for two consecutive years must receive technical assistance from the district, with students having the option to transfer to another school within the district. Schools failing to meet their goals for a third consecutive year will continue to receive techni cal assistance and must allow intra-district
Education Policy Analysis Archives Vol. 12 No. 37 6 school choice, but must also allow eligible stud ents to use their portion of Title I funds to purchase tutoring or other services directly from the district or an outside agency. A fourth consecutive year of failure would continue the previous consequences, but also impose re-staffing or other fundamental changes. After a fifth consecutive year of failure, governance must change (i.e., charter school, privatization of management services, state takeover of operations) (CEP, 2002). The seve rity of the consequences for school failure under NCLB has made it imperative that a ccountability measures and models have been well designed, truly identifying effective and ineffective schools. Accountability System Designs Well-designed and implemented accountability models, characterized by clearly articulated academic standards and tests, have served several functions inclusive of (a) promoting more challenging curricula, (b) fo stering collaboration among teachers within and across schools, and (c) creating more produ ctive dialogue among teachers and parents. Poorly designed and implemented accountabili ty systems have had the effect of impeding, rather than advancing, education reform by creating an overall climate of frustration in schools regarding the measurement of school effectiveness (Gandal & Vranek, 2001). The mandatory sanctions provided for by the NCLB Act highlight the need for fair and consistent accountability systems. When judging the desirability of the desi gn of an accountability system, the following criteria have been considered: (a) The usefulness of the measures used to determine school effectiveness in diagnosing weaknesses in individual schools; (b) The usefulness of the accountability results to parents in making decisions about the education of their children; and (c) The fairness of accountability system to teachers and administrators regarding factors beyond their control (i.e., readiness to learn, home and environmental factors). Moreover, although it has been asserted that it is most appropriate to focus accountability on schools rather than on school districts, or individual students (Ladd, 2001), accountability systems have varied in their approach, some focusing on all three of those levels (Â“Quality CountsÂ”, 2002). Three basic accountability system models have been implemented for either individual schools, school districts or both, each of which has inherent strengths and weaknesses: (a) Accountability systems focusi ng on school-wide averages on test scores through comparison to a cut-off score or to th e scores of other schools; (b) Accountability systems designed to capture the value-added by the school to student learning; and (c) Accountability systems focusing on a target rate of improvement for each individual school (Ladd, 2001). Comparisons of average scores. Accountability systems of this type have used either mandatory cutoff scores on student assessments to determine a schoolÂ’s effectiveness, or have compared schools or school districts to each other, determining effectiveness based upon rank order (Ladd, 2001). For example, the accountability system in Texas has used cut-off scores to determine school effectiveness (Texas Education Association (TEA), 2001). In Texas, schools have been classified as Exemplary, Recognized, Academically Acceptable/Acceptable and Academically Unacceptable/Low Performing depending upon the percentage their students passing the stateÂ’ s accountability tests in reading, writing and mathematics, as well as the percentage of students who dropped out of school. Moreover, when student scores were disaggregated by ethnicity and socioeconomic status, the
Influence of Socioeconomic Factors 7percentage of students within these groups passing the tests and remaining in school must also have met state standards (TEA, 2001). On a philosophical level, this style of accountability system has been consistent with the spirit of standards-based accountability in that all students and schools have been assessed against a uniform standard. However, there have been concerns that unaltered, this style of accountability system has not accounted for differen ces between the schools (i.e., socioeconomic factors) and that schools may have been categorized based as much on the socioeconomic differences as on the quality of the instructional program. Additionally, schools with advantaged populations and ad equate scores have tended to become complacent, whereas schools with disadvantaged populations have struggled to meet seemingly unattainable state standards (Ladd, 2001). Value-added designs It has been noted that accountability systems work best when they measure what a school has added to the learning of children over a given year, and then hold the school accountable (Hanushek & Raymond, 2001). Termed the value-added approach, this style of accountability system has defined school effectiveness in terms of the gains in student achievement, rather than a uniform cutoff score. Through the extensive use of control variables, the valueadded approach statistically accommodates for confounding factors (e.g., socioeconomic facto rs, prior school achievement) and therefore statistically estimates the portion of gain scores on state assessments attributable to teacher effects for a given year. Tennessee has institu ted such a value-added component as part of the stateÂ’s accountability system (Sanders & Horn, 2002). Value-added approaches, although in th eory providing the type of information requisite to promote school improvement an d to communicate most clearly with the public regarding school performance, has m et with resistance for several reasons. In general, there has been little consensus regarding the variables that should be controlled for in the value-added analysis. Additionally, there has been concern regarding potential peer effects that would impact the credibility of the results. On a practical level, the data required to complete a value-added analysis has often been unavailable or cost prohibitive to gather (Ladd, 2001). Interim target designs. Accountability systems that ut ilize interim target goals as shortterm measures toward long-range statewide go als have served to base school effectiveness on school progress toward individualized impr ovement goals rather than either progress toward an absolute cut-score or value-added scores. This approach has allowed schools to be identified as effective provided they meet the improvement target established for a specific time-period. The CATS system in Ke ntucky has been designed in this fashion (Ladd, 2001). Underlying the interim target approach ha s been the assumption that the interim school achievement targets would be reasonably attainable for a school functioning in an acceptable fashion. However, depending upon how the targets have been established, it has been possible that targets have been easier for some schools to meet than others. Schools starting nearer the long-term goal have not had to change their instructional practices significantly as compared to low-pe rforming schools, which typically serving the poorest and most challenging students and w ho have had to make tremendous gains to meet their targets (Ladd, 2001). Impact of Socioeconomic Statu s on Student Assessment Scores Central to any accountability system has been the use of norm-referenced tests, criterion-referenced tests, or both (Ladd, 2001 ; Â“Quality CountsÂ”, 2002). Regardless of the type of test used, empirical studies have demonstrated that school scores on these tests
Education Policy Analysis Archives Vol. 12 No. 37 8 reflect in large part student socioeconomic variables, which are beyond the control of the school, rather than reflecting variables within the control of the school that affect the achievement of students (Lyons, 2001; Su tton & Soderstrom, 1999; Wilson & Martin, 2000). In an analysis of Grade 3 Reading and Mathematics scores from 1994 Illinois Goal Assessment Program (IGAP) assessments, Sutt on and Soderstrom (1999) found that 56% of the variance in mathematics scores and 70% of the variance in reading scores was associated with variables beyond the control of the school, such as the percentage of students at each school participating in the free and reduced lunch program. Analysis of Grade 10 scores for reading and mathematics resulted in variables beyond the control of the school accounting for 74% and 62% of th e variance in student scores, respectively. Similar results have been reported in analyses of school-level and district-level scores in Ohio. In a study of public schools in the Toledo (OH) school district, Wilson and Martin (2001) found that per capita in come was the most dominant predictor of student test scores on the both the Ohio Proficiency Test and the Metropolitan Achievement Test, accounting for 70.56% of the variance in student test scores. In a similar study of school district scores on the Ohio Proficiency Test, Lyons (2001) found that 58.5% of district means varied with school district per capita median income, per pupil property wealth, and percent free/reduced lunch participation. As noted earlier, the impact of demographics on student test scores has made it unclear as to whether school performance on a test has been a result of an effective instructional program, or exceptional student de mographics. The potential for differential socioeconomic status to bias assessment scores has created a fundamental problem with the fairness of the system to the schools sub ject to the accountability system, the parents of the children attending these schools, an d the taxpayers of the state who draw conclusions regarding the quality of the schools they support based upon the results of these accountability systems. The interrelationship of test scores and student demographics has been a focal point for critic s of accountability (Kohn, 2001), but also a consideration for designers of school accountab ility systems. For accountability systems to promote the aforementioned collaboration wi thin school communities, to communicate accurately with parents and the general public and to be fair to schools of all types, a means of either accounting for differences in demographics, should be designed into the system. Accountability in Kentucky In 1992, KentuckyÂ’s accountability system, at that time the Kentucky Instructional Results Information System (KIRIS), was implem ented as part of the Kentucky Education Reform Act (KERA). KIRIS was designed to reflect the degree to which schools were improving the effectiveness of their instru ctional program within the context of the KentuckyÂ’s learning goals, as measured by both cognitive and non-cognitive indicators. A system of rewards and sanctions based upon schoolsÂ’ progress toward improvement goals was implemented as an incentive for schools to progress toward KIRIS goals (KDE, 1998). With KIRIS, each individual school was assigned a KIRIS improvement goal every two-years based upon the difference between ea ch schoolÂ’s current KIRIS accountability index and the 20-year statewide goal of 100. This method of establishing improvement goals was intended to allow interim goals to be individualized for each school, with initially
Influence of Socioeconomic Factors 9low-performing schools having to show larg er biennial gains than schools that were initially high-performing (KDE, 1998). Schools meeting or exceeding their improvement goal were designated as being in Rewards which entitled them to receive monetary rewards from the state. Schools failing to meet their improvement goal, but that scored at or above their baseline, were identified as Maintaining These schools were not eligible for rewards and were required to submit a school improvement plan. Schools whose KIRIS scores dropped below their baseline were designated as being In Decline Depending upon the degree to which the school was In Decline, consequences ranging from a required school improvement plan, the assignment of a highly skilled educator and the option of parents to transfer students to another school were all possible (KDE, 1998). From KIRIS to CATS Highly criticized for several reasons, including the lack of a national norm-referenced test as part of each schoolÂ’s accountability index, KIRIS was replaced by the Commonwealth Accountabili ty Testing System (CATS) in 1998. CATS differed from KIRIS in several ways, but most fundamentally in the addition of the Comprehensive Test of Basic Skills Survey Edition (CTBS-5) in three grade levels (3, 6, 9) to compose five percent of each schoolÂ’s CA TS index. The remaining 90% to 95% of the CATS index for elementary, middle, and high schools was composed of the results of the Kentucky Core Content Test (KCCT), writin g portfolios, and non-cognitive indicators (KDE, 2000). As with KIRIS, CATS stipulated a unique improvement goal for each individual school to reach for each successive biennium. However with CATS, rather than calculating a new improvement goal after each successi ve biennium based upon the discrepancy between current school performance and the state goal at that point in time, the improvement goal for each year from 1998-2014 was determined by extrapolating a line from each schoolÂ’s accountability index at the end of the 1998-2000 biennium to the statewide goal 100 in at the end of the 2012-2014 biennium. A Zone of Fairness was created to accommodate for the standard error of measure for the test each year. Figure 1 illustrates the growth chart format used to communicate the long-term accountability requirements for each school in the State of Kentucky fr om 2000 to 2014. Each biennium schools were identified as Meeting Goals, Progressing, or Assistance based upon whether they scored above, within, or below the Zone of Fairness (KDE, 2000; KDE 2002). School Effectiveness and Demographics As mentioned previously, school effective ness in Kentucky has been defined in terms of progress toward a long-term goal, as gauged by the achievement of incremental improvement goals. By focusing on incrementa l school improvement, rather than absolute aggregate student achievement, CATS had been designed to accommodate for differences in demographics and as well as differences in initial performance th at may have impacted student achievement. The goal has been to make school-based decisions to develop instructional strategies to meet each biennial Improvement Goal, incrementally leading each school to the 20-year absolute goal of 100 regardless of disparate school characteristics, such as socioeconomic status (KDE, 1998).
Education Policy Analysis Archives Vol. 12 No. 37 10 2014 Goal 140 80 100 60 Assistance Line Goal Line 40 Zone of Fairness 2014 2000 2007 C A T S I n d e x Figure 1. Sample CATS Long-term Accountability Model for a school with an initial index of 40 in 2000 (Biennium schools are identified based on whether their average index for that biennium is above the Goal Line ( Meets Goals ), in the Zone of Fairness ( Progressing ), or below the Assistance Line ( In Need of Assistance ).) It has been an underlying assumption of the interim target model in general (Ladd, 2001), and CATS specifically, that the use of interim targets toward a long-term accountability goal has mitigated any concerns regarding the possibility that socioeconomic differences between schools might bias the a ccountability system in against high poverty schools. Therefore, the application of in dividualized improvement goals has been assumed to compensate for demographic differences, providing all schools with an equitable chance of meeting their improvement goals, and ensuring that the improvement goals established for a biennium challenge all schools equally to improve. For elementary, middle, and high schools in the State of Kentucky, the following hypotheses were investigated: (1) There will not exist a significant relationship between the 2000-2002 CATS Accountability Index and school/community de mographic indicators for KentuckyÂ’s public elementary, middle, and high schools. (2) There will not exist a significant rela tionship between the frequency that KentuckyÂ’s public elementary, middle, and high schools were designated as Meeting Goal, Progressing, and Assistance Level fo r the 2000-2002 accountability biennium and their relative socioeconomic status. (3) There will not exist a significant rela tionship between the frequency that KentuckyÂ’s public elementary, middle, and high schools had exceeded their 2000-2002 Improvement Goal by the 2001 midpoint report and their relative socioeconomic status.
Influence of Socioeconomic Factors 11Method Participants To be included in the study, Kentucky elementary, middle, and high schools must have been structured in terms of grade-levels as to contain all three accountability grades for their respective school type. Specifically, schools must have been structured to contain grades three to five for elementary schools, six to eight for middle schools, and nine to twelve for high schools. Of the eligible sc hools, schools were eliminated that (a) did not participate in the federal free/reduced lunch pr ogram, (b) had extreme values in terms of socioeconomic variables, and (c) were identified as an alternative school. Table 1 summarizes the grade level structure of both the population and sample for elementary, middle and high schools. Table 1 Distribution of the Population and Sample in terms of Grade Level Structure Grade Structu re Elementary Middle High EP Â– 4 0 of 23 EP Â– 5 450 of 469 EP Â– 6 165 of 172 EP Â– 7 5 of 5 EP 8 77 of 95 78 of 95 EP Â– 12 1 of 6 1 of 6 1 of 6 4 Â– 5 0 of 3 4 Â– 6 0 of 2 4 Â– 8 0 of 2 5 Â– 6 0 of 3 5 Â– 8 0 of11 5 Â– 12 0 of 2 0 of 2 6 Â– 8 151 of 169 6 Â– 12 1 of 3 2 of 3 7 Â– 8 0 of 30 7 Â– 9 0 of 2 7 Â– 12 0 of 24 22 of 24 8 Â– 12 0 of 1 1 of 1 9 Â– 12 194 of 201 10 12 0 of 1 Variables School and community socioeconomic indica tors and school accountability indices were obtained from the Kentucky Department of Education and the United States Census Bureau for this study. Descriptions of each variable were provided below, with sample summary statistics reported provided in Table 2. CATS Indices. The CATS Accountability Indices for the 2000-2002 biennium and the 2001 midpoint were obtained from the Kentucky Department of Education ( Kentucky
Education Policy Analysis Archives Vol. 12 No. 37 12 Department of EducationÂ—School accountability resu ltsÂ—2002 midpoint; Kentucky Department of EducationÂ—School accountability resultsÂ—2000-2002 biennium ). These accountability indices, which represented the performance of a school on a scale of 0 to 140 with 100 representing proficiency, were calculated for each school based upon both cognitive and non-cognitive measures. Although there wer e subtle differences between the weighting of the cognitive and non-cognitive components for elementary, middle, and high schools, in general the CTBS-5 has been weighted 5%, an d the Kentucky Core Content Test has been weighted 85%-90%. Non-cognitive indicators, such as dropout and retention rates, have been weighted 5%-10% (KDE, 2000). Table 2 Summary Statistics of Va riables for the Sample Elementary Middle High CATS 20002002 Biennial Index (CATS) MEAN = 71.3 S.D. = 10.0 MEAN = 67.0 S.D. = 8.8 MEAN = 66.2 S.D. = 8.3 2002 Free and Reduced Lunch Participation (FRL02) MEAN = 58.8% S.D. = 20.3% MEAN = 55.7% S.D. = 20.4% MEAN = 41.3% S.D. = 17.3% 2001 Free and Reduced Lunch Participation (FRL01) MEAN = 58.1% S.D. = 20.9% MEAN = 55.1% S.D. = 20.9% MEAN = 39.5% S.D. = 17.8% Median Household Income (MHI) MEAN = $32,606 S.D. = $7,515 MEAN = $31,354 S.D. = $7,519 MEAN = $31,846 S.D. = $7,195 Overall Poverty Rate (OPR) MEAN = 16.9% S.D. = 6.4% MEAN = 18.0% S.D. = 6.7% MEAN = 17.1% S.D. = 6.2% Under-17 Poverty Rate (U17P) MEAN = 22.6% S.D. = 21.0% MEAN = 23.7% S.D. = 6.7% MEAN = 23.0% S.D. = 6.4% Related Children 5Â– 17 Poverty Rate (RCP) MEAN = 20.8%% S.D. = 7.8% MEAN = 22.1% S.D. = 8.1% MEAN = 21.2% S.D. = 7.6%
Influence of Socioeconomic Factors 13 School and community socioeconomic status. School socioeconomic status was represented by mean free and reduced lunch participation rate for each elementary, middle, and high school each year during the 2001 and 2002 school years. ( Kentucky Department of EducationÂ—October free and reduced price dataÂ—2001-2002 ). Community socioeconomic status was represented by United States Census inco me and poverty statistics for the county in which each school was located ( United State Census Small area estimates State and county poverty estimates, rates and median household income 1998 ). Specifically, the following indicators were used: (a) Median Household Income, (b) Overall Poverty Rate, (c) the Poverty Rate of Persons Under 17 years of age, and (d) th e Poverty Rate of Related Persons ages 5 to 17. School Classifications. The Kentucky Department of Education reported the accountability classification of each school (Meets Goal, Progressing, Assistance-level) based upon the 2000-2002 biennial CATS index. Additionally, the 2001 CATS midpoint index was used to determine relative progress of schools toward these biennial goals by comparing the 2001 midpoint index with th eir respective 2000-2002 improvement goals and baselines. Schools meeting or exceeding their Improvement Goal were classified as Exceeding Target schools whose index was between their Improvement Goal and Baseline were classified as Below Target and schools scoring below their Baseline were identified as Below Baseline Table 3 and 4 summarized the frequency that schools were classified in each category for both the 2000-2002 biennium and the 2001 midpoint. Table 3 CATS 2000-2002 Accountability Classifications Meets Goal Progressing Assistance Level Elementary 410 (58.7%) 231 (33.1%) 57 (8.2%) Middle 90 (39.0%) 124 (53.7%) 17 (7.4%) High 68 (30.9%) 138 (62.7%) 14 (6.4%) Total 568 (49.4%) 493 (42.9%) 88 (7.7%) Table 4 CATS 2001 Midpoint Progress Exceeding Target Below Target Below Baseline Elementary 341 (48.9%) 209 (29.9%) 148 (21.2%) Middle 70 (30.3%) 113 (48.9%) 48 (20.8%) High 50 (22.7%) 134 (60.9%) 36 (16.4%) Total 461 (40.1%) 456 (39.7%) 232 (20.2%)
Education Policy Analysis Archives Vol. 12 No. 37 14 Procedure Participant elimination. Elementary, middle, and high schools not participating in the federal free and reduced lunch program, that wer e classified as alterna tive schools or that had extreme standard scores ( z < -3 or z > 3) for any single socioeconomic indicator were eliminated from the study. The final sample consisted of 698 of 773 elementary schools, 231 of 350 middle schools, and 220 of 238 hi gh schools. Tables 3 and 4 describe the sample in relation to grade level structure and the variables used in the study, respectively. Regression analysis for the 2000-2002 CATS Index. For each grade level group (elementary, middle, high) a stepwise, multip le linear regression was performed using school and community socioeconomic variables as predictor variables, and the 2000-2002 CATS Accountability Index as the criterion variable. Free and reduced lunch participation rates for 2001 and 2002 were averaged to compensate for the fact that the CATS indices represented two years. Wealth quintiles. The school and community socioeconomic variables used in the study were converted to a categor ical variable to allow for the use of Chi-square analysis. These categories, referred to as Wealth Quintiles were derived from a standardized weighted average of the socioeconomic indicators found to be significant in the elementary, middle and high school multiple-regression analyses. Specifically, raw scores were converted to standard scores for each significant variable. These standard scores were used to create a weighted average, the weights of which refl ected the proportion of the shared variance in the CATS index explained by the addition of each respective variable for each analysis. This weighted average of these standard sc ores was then used to form the Wealth Quintiles, with the poorest schools assigned to the first quintile and the wealthiest to the fifth. Determining wealth neutrality. A 5 x 3 Chi-square was used to determine whether a relationship existed between the socioec onomic level of a school and the schoolÂ’s performance relative to its improvement goal Specifically, the Chi-square was applied to determine whether the Wealth Quintile assigned to a school was related to the accountability classification (Meets Goal, Pr ogressing, Assistance-level) of the school. Determining capacity to meet goals. A similar methodology was applied to determine whether schools from all socioeconomic levels had equal capacity to meet their improvement goals. For these analyses, a 5 x 3 Chi-square analysis was again utilized. However, rather than relating the Wealth Quintile with the biennial accountability results, the Wealth Quintiles were related to mid point CATS progress, as determined earlier (Exceeding Target, Below Target, Below Baseline). Limitations Steps were taken to ensure that schools of each level (elementary, middle, high) were comparable in terms of the grade levels in cluded. However, there were no steps taken to determine what relationship, if any, grade level structures encompassing two or more levels (e.g., p-12, p-8) had with school classificati on. Additionally, approximately one-third of middle schools were excluded from the study due to grade level structure. As a consequence, results of the aforementioned anal yses may not represent middle grades in all schools and districts.
Influence of Socioeconomic Factors 15Results Hypothesis 1 Results of the stepwise, multiple linear regression indicated that a significant relationship did exist between the 2000-2002 Biennial CATS accountability index and school socioeconomic variables, communi ty socioeconomic variables or both for elementary ( R = .634; R2 = .402; p = .000), middle ( R = .677; R2 = .453; p = .000), and high schools ( R = ..737; R2 = .542; p = .000). As a consequence, the hypothesis that there will not exist a significant relationship betwe en the 2000-2002 CATS Accountability Index and school/community demographic indicators fo r KentuckyÂ’s public elementary, middle, and high schools was rejected for all school levels. It was noted that both school and comm unity socioeconomic variables (free and reduced lunch participation, median household income) were significantly related to the accountability index for the elementary and middle school analyses, however only the school socioeconomic variable (free and reduced lunch participation) was significant for the high school analysis. Standardized regr ession equations for elementary school, middle school, and high school levels were listed below: (1) zCATSELEM =( -.752)(zFRL0102) + (-.295)(zMHI) (2) zCATSMID = (-.982)(zFRL0102) + (-.448)(zMHI) (3) zCATSHIGH = (-.737)(zFRL0102) Hypothesis 2 Tables 5, 6 and 7 summarize the observed and expected classifications of elementary, middle and high schools, respectively. Results of the Chi-square analysis indicated that for elementary ( 2 = 11.441, df = 8, p = .178 ) and middle schools ( 2 = 8.238, df = 8, p = .411) there was not a significant relationship between socioeconomic factors and the accountability classification for the 2000-20 02 biennium. However, for high schools ( 2 = 45.251, df = 8, p = .000) there did exist a significant relationship between socioeconomic factors and accountability classification, with over twice as many of the wealthiest schools classified as Meeting Goal as would have b een expected, while a little more that half as many of the poorest schools were classified as Meeting Goal as would have been expected. As a consequence, the hypothesis that there will not exist a significant relationship between the frequency that KentuckyÂ’s public elemen tary, middle, and high schools were designated as Meeting Goal, Progressing, and Assistance Level for the 2000-2002 accountability biennium and their socioeconomic status was accepted for elementary and middle schools, and rejected for high schools.
Education Policy Analysis Archives Vol. 12 No. 37 16 Table 5 Actual and Expected Frequencies of Acco untability Classification for Kentucky Elementary Schools for the 2000-2002 Biennium Wealth Quintiles 2000-2002 CATS 1 2 3 4 5 Met Goal Actual Expected 9 0 8 1 4 7 5 8 2 0 8 9 8 2 0 7 9 8 2 0 7 6 8 1 4 Progressing Actual Expected 2 9 3 7 8 4 2 3 8 1 2 9 3 8 1 4 5 3 8 1 4 5 3 7 8 Assistance Level Actual Expected 2 0 1 9 7 2 3 1 9 9 2 2 1 9 9 1 6 1 9 9 1 8 1 9 7 2 = 11.441; p = .178
Influence of Socioeconomic Factors 17 Table 6 Actual and Expected Frequencies of Acco untability Classification for Kentucky Middle Schools for the 2000-2002 Biennium Wealth Quintiles 2000-2002 CATS 1 2 3 4 5 Met Goal Actual Expected 1 8 0 1 7 9 1 4 0 1 7 9 2 0 0 1 8 3 1 5 0 1 7 9 2 3 0 1 7 9 Progressing Actual Expected 2 6 0 2 4 7 2 6 0 2 4 7 2 3 0 2 5 2 2 7 0 2 4 7 2 2 0 2 4 7 Assistance Level Actual Expected 2 0 3 4 6 0 3 4 4 0 3 5 4 0 3 4 1 0 3 4 2 = 8.238; p = .411
Education Policy Analysis Archives Vol. 12 No. 37 18 Table 7 Actual and Expected Frequencies of Acco untability Classification for Kentucky High Schools for the 2000-20 02 Accountability Biennium Wealth Quintiles 2000-2002 CATS 1 2 3 4 5 Met Goal Actual Expected 3 1 0 1 3 6 1 2 0 1 3 6 8 0 1 3 6 9 0 1 3 6 8 0 1 3 6 Progressing Actual Expected 1 2 0 2 7 6 3 0 0 2 7 6 3 0 0 2 7 6 3 3 0 2 7 6 3 3 0 2 7 6 Assistance Level Actual Expected 1 0 2 8 2 0 2 8 6 0 2 8 2 0 2 8 3 0 2 8 2 = 45.251; p = .000 Hypothesis 3 Tables 8, 9 and 10 summarize the observed and expected frequencies with which elementary, middle and high schools exceeded their biennial improvement goals at the midpoint. Results of the Chi-square analysis indicated that there was not a significant relationship between school and community so cioeconomic factors and classification of schools as Exceeding Target, Below Target or Below Baseline for middle schools ( 2 = 11.630, df = 8, p = .168). However, for elementary ( 2 = 17.806, df = 8, p = .023 ) and
Influence of Socioeconomic Factors 19high schools ( 2 = 39.218, df = 8, p = .000 ) a significant relationship did exist. As a consequence, the hypothesis that there will not exist a significant relationship between the frequency that KentuckyÂ’s public elementary, middle, and high schools had exceeded their 2000-2002 Improvement Goal by the 2001 midpoint report and their socioeconomic status was supported for middle schools, but not supported for elementary and high schools. Table 8 Actual and Expected Freque ncies of Mid-point Improv ement Goal Attainment for Kentucky Elementary Schools in 2001 Wealth Quintiles 2001 Midpoint Results 1 2 3 4 5 Exceeded Target Actual Expected 8 6 0 6 7 9 6 5 0 6 8 4 6 9 0 6 8 4 6 6 0 6 8 4 5 5 0 6 7 9 Below Target Actual Expected 3 0 0 4 1 6 4 0 0 4 1 9 3 9 0 4 1 9 4 6 0 4 1 9 5 4 0 4 1 6 Below Baseline Actual Expected 2 3 0 2 9 5 3 5 0 2 9 7 3 2 0 2 9 7 2 8 0 2 9 7 3 0 0 2 9 5 2 = 17.806; p = .023
Education Policy Analysis Archives Vol. 12 No. 37 20 Table 9 Actual and Expected Freque ncies of Mid-point Improv ement Goal Attainment for Kentucky Middle Schools in 2001. Wealth Quintiles 2001 Midpoint Results 1 2 3 4 5 Exceeded Target Actual Expected 1 9 0 1 3 9 1 4 0 1 3 9 1 5 0 1 4 2 9 0 1 3 9 1 3 0 1 3 9 Below Target Actual Expected 1 9 0 2 2 5 2 5 0 2 2 5 2 2 0 2 3 0 2 9 0 2 2 5 1 8 0 2 2 5 Below Baseline Actual Expected 8 0 9 6 7 0 9 6 1 0 0 9 8 8 0 9 6 1 5 0 9 6 2 = 11.630; p = .168
Influence of Socioeconomic Factors 21 Table 10 Actual and Expected Freque ncies of Mid-point Improv ement Goal Attainment for Kentucky High Schools in 2001 Wealth Quintiles 2001 Midpoint Results 1 2 3 4 5 Exceeded Target Actual Expected 2 4 0 1 0 0 6 0 1 0 0 9 0 1 0 0 6 0 1 0 0 5 0 1 0 0 Below Target Actual Expected 1 5 0 2 6 8 3 0 0 2 6 8 2 3 0 2 6 8 3 3 0 2 6 8 3 3 0 2 6 8 Below Baseline Actual Expected 5 0 7 2 8 0 7 2 1 2 0 7 2 5 0 7 2 6 0 7 2 2 = 39.218; p = .000 Discussion Analyses of the 2000-2002 CATS Accountability Report indicate a strong relationship exists between the socioeconomic status of schools and their achievement on the assessment tests, with shared variance between socioeconomic factors and the accountability assessments ranging from 39.7 % to 60.5% and the shared variance between socioeconomic factors and the CATS accounta bility index ranging from 43.0% to 50.1%. The presence of such a strong and signific ant relationship between factors outside of school and student achievement is not new and is not unexpected. Recent analyses in
Education Policy Analysis Archives Vol. 12 No. 37 22 Ohio and Illinois produced similar results. The passage of the NCLB Act, however, has added a new twist to the implicat ions of such studies. The logic behind public school accountability systems is that the establishment of state goals, standards, and assessments, followed by the systematic dissemination school achievement information relative to state goals and standards is intended to create political leverage for school improvement. This leve rage is enhanced by the administration of rewards and consequences to schools based upon the attainment or non-attainment of their improvement goals, respectively. If a pattern exists whereby differential assessment results follow differential socioeconomic status, it becomes increasingly difficult for the school community, parents, and the general public to discern schools producing high student achievement due to effective instru ctional programs from schools producing high achievement based due to the advantaged stat us of their students in the absence of an effective instructional program. Moreover high poverty schools with exceptional instructional programs quite often are overlooked due to the fact that their test scores are not as high as their affluent peers. The Impact of Poverty of Ke ntuckyÂ’s Accountability Model It is clear that in the absence of any means of Â“leveling the playing fieldÂ”, schools serving high poverty areas are placed in a difficult and arguably unfair position. States using accountability systems that hold all schools to the same goals at all times run the risk of setting the bar so high as to essentially sort schools based upon the nature of their demographics rather than the quality of their instructional program, or to set the bar so low as to be attainable at any time by any school, but to serve little purpose in terms of leveraging school improvement. The use of individualized improvement goal s to establish fair, yet challenging targets for all schools has been the approach used by Kentucky. The results of the analysis indicated that despite the strong and signific ant relationship between the achievement test scores for elementary and middle schools and school socioeconomic status, there was not a significant relationship between the accoun tability classification of these schools and their socioeconomic status. This finding bodes well for the accountability system in general and the instructional programs of schools at the elementary and middle school level in Kentucky, indicating that elemen tary and middle schools at all socioeconomic levels are implementing programs to put them on track for proficiency in 2014. Results of the same analyses for KentuckyÂ’s high schools did not bode as well. A significant relationship existed between soci oeconomic factors and school accountability classification, essentially sorting schools base d upon demographics as much as program effectiveness. Being that the means by whic h each high schoolÂ’s accountability index is calculated is the same as for elementary and middle schools, and the determination of improvement goals is the same as well, the question is raised as to why the accountability system appears to be biased for high schools, but not for elementary or middle schools? Capacity to Meet Improvement Goals An assumption of the interim improvement goal accountability models, such as CATS, is that all schools, regardless of ba ckground, has the capacity to attain their improvement goals provided they operate at a reasonable level of effectiveness. Conceptually, this capacity may include the capacity to implement changes, curricular and otherwise, through the organization, or the ability to engage parents and the public in the school improvement process. Additionally, facto rs such as finance may serve to build this
Influence of Socioeconomic Factors 23capacity. Whatever constitutes this capacity, it is assumed that all schools at all levels are identical in this respect. Should this assumption be true, schools at all levels of poverty should have an equitable chance of meeting th eir school improvement goals at any point in time, in this case the mid-point of the biennium (2001). Results of the Chi-square analyses indi cate that only for middle schools did this assumption hold true. Elementary and high schools from the highest wealth categories were significantly more likely to have alread y met their 2000-2002 improvement goals as of the 2001 midpoint report. More specifically, the most affluent Kentucky high schools had met their biennial school improvement goals at twice the rate expected if the process were wealth neutral, whereas the poorest high schools had met the improvement goal at half the rate expected. Elementary results were significant, although not as pronounced, with poorer schools meeting their goals less freque ntly than expected, and more affluent schools meeting their improvement goals more frequently than expected. Explanations of the impact of socioec onomic status on the CATS Accountability Index, the system of improvement goals use d to classify schools, and the capacity of schools to meet their improvement goals could take several forms. For example, differences in student cohorts taking the tests from year to year, or differences in the availability of qualified staff in higher pov erty areas could both impact the results of student assessments. Regardless of the reas on, data indicates that for the 2000-2002 biennium, socioeconomic status impacted CATS sufficiently to bias the system in favor of school with lower levels of poverty for high schools. Moreover, when the 2001 midpoint results were considered, it became apparent that a relationship existed between the socioeconomic status of elementary and high schools and the early attainment of their improvement goals. This suggests that even with the individualized improvement goals, high poverty elementary schools may soon fall behind their more affluent peers as the improvement goals are raised. The critical nature of accountability under NCLB makes it imperative that states account for the potential biasing effects of poverty on their public school accountability systems. Adaptations in Ohio and Texas. The impact of socioeconomic status on accountability measures has been addressed th rough policy in other states, including Texas and Ohio. The states of Ohio and Texas co mpare similar schools or school districts, in addition to overall ratings against state standard s. That is to say, schools and districts are still accountable for meeting uniform state acco untability requirements, but that relative school and school district achievement is also re ported to the public in a systematic way. Noting that it makes more sense to compare similar school districts, thus allowing parents to answer the essential question Â“ How well is our district performing when compared to school districts with similar ch aracteristics, challenges and resources?Â” the Ohio Department of Education provides for a systematic way of comparing school district performance (ODE, 2002b). Specifically, Ohio communicates differential achievement of socioeconomically and demographically similar school districts to the public by comparing school district achievement with a sample of 20 districts identified as similar based upon poverty, size, socioeconomic status, over all property wealth and school type (urban, suburban, rural). Reports are made availabl e on-line for interested parties to compare the accountability results (testing results, graduati on rates, attendance rates) of these similar school districts (ODE, 2002). Rather than comparing school districts, Texas generates a Comparable Campus Report that clusters 40 schools with comparab le socioeconomic and demographic profiles for the purpose of comparing reading and mathematics. The Texas Learning Index (TLI),
Education Policy Analysis Archives Vol. 12 No. 37 24 a value-added measure of individual student test score improvement, is calculated for comparable students at each school, with these results being aggregated producing a school TLI. Schools with each comparison gr oup are assigned to a quartile based upon their TLI. These quartile rankings, which are only applicable within the comparison group, communicate to parents, teachers and taxp ayers the impact the school had upon student learning a given year relative to schools of comparable socioeconomic and demographic compositions (TEA, 2002). Argument against comparing school scores. Since the implementation of the Kentucky Education Reform Act (KERA) in 1990, it has been asserted that individualized target format of KIRIS, and then CATS, has made it inappropriate to compare schools to each other. Rather, schools were to be compared only to their own improvement goal during the reform effort, at least until 2014 when a ll schools will be held to the same standard. However, the reality is that the general public, inclusive of teachers and principals, look to see how other schools have performed on CATS when gauging performance of schools in the area. Moreover, as each successive biennium passes and schools move closer to being compared to the same standard in 2014, the argument against interschool comparisons becomes less inspiring; 2014 is fast approaching. General Conclusions Conclusions of this study should not be misinterpreted to say that lower accountability standards should be establishe d for high poverty schools. The mantra that Â“All Children Can LearnÂ” applies in that for schools to improve, high standards must be held. The emphasis on closing the achievement gap between disadvantaged students and their peers reinforces the need for high standards. Rather this study illustrates that the absence of a systematic approach of comp aring accountability scores of schools with differential socioeconomic characteristics is un fair to the children, parents, teachers, and principals of high poverty schools. School accountability systems are a politic al means of leveraging improvement through a balance of collaboration for school improvement and consequences for poor performance. Although much attention ha s been given to the appropriateness of the instruments used to measure school performance, possibly the weakest link in the chain of accountability may be the ways in which results are reported and excellence is defined. In light of the implementation of the NCLB A ct, the fact that as of 2002 the CATS accountability system shows potential for bias against high poverty schools, with no provision made to communicate completely to the public the performance of each schools as compared to similar schools is viewed as unjust to all stakeholders. CATS and the NCLB Act. Since the passage of the NCLB Act, negotiations regarding the types of tests and standards to be utilized at the state and local level with the have resulted in some flexibility for states. However, the key federal standard regarding the yearly improvement of schools, adequate ye arly progress (AYP), was not on the table for negotiation (Â“Public agendaÂ”, 2002). As states prepare to respond to federal government guidelines regarding the determina tion of adequate yearly progress for public schools, it is important that they examine th e degree to which existing accountability measures and models have been influenced by external factors such as socioeconomic status. Adapting CATS. Currently, when CATS results are communicated to the public through school report cards or press releases socioeconomic differences between schools are not easily discernable from the minutia of information conveyed. In the absence of a
Influence of Socioeconomic Factors 25systematic way for the public to gauge how a given school did compared to schools statewide, or even in a given geographic area, parents and taxpayers will often make inappropriate comparisons between schools. These comparisons are unfair to teachers and principals as they strive to improve their schools, unfair to parents as they make decisions about their childrenÂ’s education, and unfair to taxpayers as they draw conclusions regarding how well their tax money has been spent. As the CATS accountability system is altered to meet federal guidelines, it is time to implement changes that will ensure that the system recognizes school effectiveness independent of socioeconomic influences. We must embrace the fact that high standards for all children are needed, but that we must not become blind to the impact socioeconomic can have on test scores. The NCLB Act has raised the stakes higher than ever of KentuckyÂ’s public schools. Educators, parents, students, and ta xpayers at large deserve the best possible information relative to the performance of Ke ntuckyÂ’s public schools. Architects of the stateÂ’s accountability system need to build on the strengths of CATS, yet rectify any weaknesses; providing an accountability system that accurately and fairly adjudicates the effectiveness of KentuckyÂ’s public schools. References Business Roundtable (2000, November). What parents, students and teachers think about standards, tests, accountabilityÂ…and more? Retrieved October 26, 2001, from http://www.brtable.org/pdf/477.pdf Center on Education Policy (2002, January). A new federal role in education. Retrieved February 20, 2002, from the Cen ter on Education Policy Web site: http://www.ctredpol.org Council of Chief State School Officers (2000, September). State education accountability reports and indicator reports: Status of reports across the statesÂ—2000. Retrieved January 8, 2002, from the Council of Chief Sta te School Officers Web site: http://www.ccsso.org/pdfs/Accountability Report2000.pdf Gratz, D. B. (2000, May). High standard s for whom? Phi Delta Kappan, 81(9), 681-687. Gullatt, D. E., & Ritter, M. L. (2000). Me asuring local public school effectiveness. Contemporary Education, 71 (4), 29-34. Hanushek, E. A., & Raymond, M. E. (2001). The confusing world of educational accountability. National Tax Journal, 54 (2), 365-385. Kentucky Department of Education (1998, December). Kentucky school and district accountability results: Accountability cycle 3 Retrieved January 8, 2002 from the Kentucky Department of Education Web site: http://www.kde.state.ky.us/oaa/implement/kiris_brief.kirisbp.asp Kentucky Department of Education (2000). Results matter: A decade of difference in Kentucky schools 1990-2000 Frankfort, KY: Kentucky Department of Education. Kentucky Department of Education (2001, Summer). Testing in Kentucky: Keys to understanding accountability. Retrieved January 8, 2002, from the Kentucky Department of Education Web site: http://www.kde.state.ky.us/comm/commrel/cats/tinky2.pdf
Education Policy Analysis Archives Vol. 12 No. 37 26 Kentucky Department of Educ ation (2001, October). Technical considerations for using the October 2, 2001 release of CATS school and dist rict accountability cycle 2002 midpoint results. Retrieved October 5, 2001 from the Kent ucky Department of Education Web site: http://www.kde.state.ky.us/oaa/implement/CATS_RTS_2001/reports/DIRECTI ONS.doc Kentucky Department of EducationÂ—School accountability resultsÂ—2002 midpoint [Data file]. Available from the Kentucky Depa rtment of Education Web site, http://www.kentuckyschools.org Kentucky Department of EducationÂ—School accountability resultsÂ—2000-2002 biennium [Data file]. Available from the Kentucky Department of Education Web site, http://www.kentuckyschools.org Kentucky Department of EducationÂ—October free and reduced price dataÂ—2001-2002 [Data file]. Available from the Kentucky Depa rtment of Education Web site, http://www.kentuckyschools.org Kentucky Department of Education-2001 CTBS/5 results for exiting primary, grades 6, and 9. Available from the Kentucky Depa rtment of Education Web site, http://kentuckyschools.org Kohn, A. (2001). Fighting the tests: A practical guide to rescuing our schools. Phi Delta Kappan, 82 (5), 349-357. Ladd, H. F. (2001). School-based educati onal accountability systems: The promise and pitfalls. National Tax Journal, 54 (2), 385-400. Lyons, R. P. (2001). Supporting adequate outcomes: The impact of differential fiscal support on student achievement. Unpublished doctoral dissertation, The University of Memphis, Memphis, TN. Ohio Department of Education (2002a, January). Local school report card 2002 communications toolkit. Retrieved February 8, 2002, from http://www.ode.state.oh.us/reportcard/toolkit_2002/Default.asp Ohio Department of Education (2002b, January). 2002 Ohio school district rating definitions. Retrieved February 8, 2002, from http://webapp1.ode.state.oh.us/district_rating/rating_definitions.asp Pearson, P. D., Vyas, S., Sensale, L. M., & Ki m, Y. (2001). Maki ng our way through the assessment and accountability maze. Where do we go now? The Clearing House, 74 (4), 175-182. Public agenda: Reality check 2002 (2002, March 6). Education Week Retrieved March 8, 2002, from http://www.edweek.com Quality counts 2002: Building blocks for success. (2002, January 10). Education Week Retrieved January 22, 2002, from http://www.edweek.com/ Rose, L. M., & Gallup, A. M. (2001, September). The 33rd annual Phi Delta Kappa/ Gallup Poll of the publicÂ’s attitudes toward the public schools. Phi Delta Kappan, 83 (1), 41-55. Sanders, W. L., & Horn, S. (n.d.). An overview of the Tennessee Value-Added Assessment System (TVAAS) with answers to frequently asked questions. Retrieved February 22, 2002, from http://www.mdk12.org/practices/ensure/tva/tva_1.htm Sutton, A., & Soderstrom, I. (1999). Pr edicting elementary and secondary school achievement with school-rela ted demographic factors. The Journal of Educational Research, 92 (6), 330-338.
Influence of Socioeconomic Factors 27Texas Education Agency (2001, April). 2001 Accountability Manual. Retrieved February 15, 2002, from the Texas Education Agency Web site: http://www.tea.state.tx.us/perfreport.account/2001/manual United State Census Small area estimates State and county pove rty estimates, rates and median household income 1998 (Excel file). Available at http://cbpa.louisville.edu/ksdc/kpr/povest/pov98.xls Wilson, C. A., & Martin, B. (2000). Race, poverty, and test scores: A model of the determinants of test scores in Toledo. The Negro Educational Review, 51 (1), 23-36. About the Author Robert Lyons Murray State University Assistant Professor Department of Educational Studie s, Leadership, and Counseling Murray State University 3246 Alexander Hall Murray, KY 42071-3818 Email: Robert.email@example.com
Education Policy Analysis Archives Vol. 12 No. 37 28 Education Policy Analysis Archives http:// epaa.asu.edu Editor: Gene V Glass, Arizona State University Production Assistant: Chris Murrell, Arizona State University General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, firstname.lastname@example.org or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: email@example.com. EPAA Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Linda Darling-Hammond Stanford University Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Los Angeles Michele Moses Arizona State University Gary Orfield Harvard University Anthony G. Rud Jr. Purdue University Jay Paredes Scribner University of Missouri Michael Scriven University of Auckland Lorrie A. Shepard University of Colorado, Boulder Robert E. Stake University of IllinoisÂ—UC Kevin Welner University of Colorado, Boulder Terrence G. Wiley Arizona State University John Willinsky University of British Columbia
Influence of Socioeconomic Factors 29EPAA Spanish & Portuguese Language Editorial Board Associate Editors Gustavo E. Fischman Arizona State University & Pablo Gentili Laboratrio de Polticas Pblicas Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998Â—2003) Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico Argentina Alejandra Birgin Ministerio de Educacin Mnica Pini Universidad Nacional de San Martin Mariano Narodowski Universidad Torcuato Di Tella Daniel Suarez Laboratorio de Politicas Publicas-Universidad de Buenos Aires Marcela Mollis (1998Â—2003) Universidad de Buenos Aires Ana Ins Heras Monner Sans Universidad Nacional de Jujuy Jos Luis Bernal Agudo Universidad de Zaragoza Carlos Mora-Ninci Universidad Nacional de Crdoba Brasil Gaudncio Frigotto Professor da Faculdade de Educao e do Pr ograma de Ps-Graduao em Educao da Universidade Federal Fluminense Vanilda Paiva Lilian do Valle Universidade Estadual do Rio de Janeiro Romualdo Portella do Oliveira Universidade de So Paulo Roberto Leher Universidade Estadual do Rio de Janeiro Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte
Education Policy Analysis Archives Vol. 12 No. 37 30 Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Iolanda de Oliveira Faculdade de Educao da Universidade Federal Fluminense Walter Kohan Universidade Estadual do Rio de Janeiro Mara Beatriz Luce (1998Â—2003) Universidad Federal de Rio Grande do Sul-UFRGS Simon Schwartzman (1998Â—2003) American Institutes for Resesarch Canad Daniel Schugurensky Ontario Institute for Studies in Education, University of Toronto Chile Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educacin Mara Loreto Egaa Programa Interdisciplinario de Investigacin en Educacin (PIIE) Colombia Jorge Ossa Universidad de Antioquia Espaa Jos Gimeno Sacristn Catedratico en el Departamento de Didctica y Organizacin Escolar de la Universidad de Valencia Mariano Fernndez Enguita Catedrtico de Sociologa en la Universidad de Salamanca Miguel Pereira Catedratico Universidad de Granada Jurjo Torres Santom Universidad de A Corua Angel Ignacio Prez Gmez Universidad de Mlaga J. Flix Angulo Rasco (1998Â—2003) Universidad de Cdiz
Influence of Socioeconomic Factors 31 Jos Contreras Domingo (1998Â—2003) Universitat de Barcelona Mxico Hugo Aboites Universidad Autnoma Metropolitana-Xochimilco Susan Street Centro de Investigaciones y Estudios Supe riores en Antropologia Social Occidente, Guadalajara Adrin Acosta Universidad de Guadalajara Teresa Bracho Centro de Investigacin y Docencia Econmica-CIDE Alejandro Canales Universidad Nacional Autnoma de Mxico Rollin Kent Universidad Autnoma de Puebla. Puebla Javier Mendoza Rojas (1998Â—2003) Universidad Nacional Autnoma de Mxico Humberto Muoz Garca (1998Â—2003) Universidad Nacional Autnoma de Mxico Per Sigfredo Chiroque Instituto de Pedagoga Popular Grover Pango Coordinador General del Foro Latinoame ricano de Polticas Educativas Portugal Antonio Teodoro Director da Licenciatura de Cincias da Educa o e do Mestrado Universidade Lusfona de Humanidades e Tecnologias, Lisboa USA Pia Lindquist Wong California State University, Sacramento, California Nelly P. Stromquist University of Southern Californi a, Los Angeles, California Diana Rhoten Social Science Research Council, New York, New York
Education Policy Analysis Archives Vol. 12 No. 37 32 Daniel C. Levy University at Albany, SUNY, Albany, New York Ursula Casanova Arizona State University, Tempe, Arizona Erwin Epstein Loyola University, Chicago, Illinois Carlos A. Torres University of California, Los Angeles Josu Gonzlez (1998Â—2003) Arizona State University, Tempe, Arizona