xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20029999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00296
Educational policy analysis archives.
n Vol. 10, no. 48 (December 04, 2002).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c December 04, 2002
K-12 voucher programs and education policy : an exploratory study of policy maker attitudes and opinions / Dan Laitsch.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 10issue 48series Year mods:caption 20022002Month December12Day 44mods:originInfo mods:dateIssued iso8601 2002-12-04
1 of 39 Education Policy Analysis Archives Volume 10 Number 48December 4, 2002ISSN 1068-2341 A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright 2002, the EDUCATION POLICY ANALYSIS ARCHIVES .Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. EPAA is a project of the Education Policy Studies Laboratory. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education .K-12 Voucher Programs and Education Policy: An Exploratory Study of Policy Maker Attitudes and Opinions Dan Laitsch Association for Supervision and Curriculum Developm entCitation: Laitsch, D. (2002, December 4). K-12 vouc her programs and education policy: An exploratory study of policy maker attitudes and opi nions, Education Policy Analysis Archives, 10 (48). Available http://epaa.asu.edu/epaa/v10n48/.Abstract Since the 1983 report, A Nation at Risk the performance of public schools has been increasingly scrutinized, and a va riety of reforms designed to increase student achievement enacted. A mong the reforms discussed, much attention has focused on increasing choice and competition in education. While the effectiveness o f market oriented reforms have been widely debated, little research h as been completed that examines policy maker attitudes toward market reform of education. This study used a researcher designed survey to exa mine policy maker attitudes toward education and education reform in general, as well as the issue of vouchers more specifically. Findings s uggest that policy
2 of 39makers generally accept the market arguments used b y voucher supporters, but are also sympathetic to equity conc erns and funding issues raised by voucher opponents. Additionally, w hile more policy makers responding to this survey supported some typ e of voucher program than opposed vouchers, when viewed in the b roader context of reform options, vouchers did not rate highly.IntroductionOn June 27, 2002, the U. S. Supreme Court ruled tha t the use of vouchers to pay tuition at private K-12 schools in Cleveland, Ohio, did not violate the establishment clause of the U.S. Constitution ( Zelman v. Simmons-Harris 2002). This ruling energized the pro-voucher movement, and resulted in plans by stat e legislators across the country to introduce new voucher legislation (Toppo, 2002). Wh ile there is debate as to how far reaching this decision may ultimately be (some indi viduals have compared the decision to the 1954 Brown v. Topeka Board of Education deci sion), observers generally agree that the policy focus, both legislative and judicia l, regarding the use of vouchers for K-12 tuition, will now turn to the states (Gehring, 2002 ; Toppo, 2002). In fact, less than two months after the Zelman decision, Justice Kevin P. Davey of the Leon Count y Circuit Court, struck down Florida's state-wide voucher pro gram as violating that state's constitutional prohibition on state aid to religiou s institutions ( Holmes v. Bush 2002). The Institute for Justice has also filed law suits in two states (Maine and Washington), seeking to have state-level prohibitions against th e use of public funds at religious schools overturned (Institute for Justice, 2002).As voucher policy is drafted, introduced and debate d, the views and beliefs of state legislators will play a critical role. To help dete rmine just how policy makers react to vouchers and school choice issues, a survey of stat e legislators was conducted in November, 2000. The results of this research are de tailed in the accompanying article. Despite interest from policy makers and advocacy gr oups, little research has been completed that would lend to a clearer understandin g of the effectiveness of arguments used to promote or inhibit voucher plans and how po licy makers respond to such arguments. The study conducted here attempts to ex amine policy maker reaction to these arguments, using the following questions: What role do policy makers feel vouchers play in th e larger context of reform? How do policy makers react to specific policy argum ents about vouchers? How does the educational philosophy of individual p olicy makers relate to their attitude regarding vouchers? How do the demographic traits of policy makers rela te to an their philosophy of education? How do the demographic traits of policy makers rela te to their views on vouchers? This study specifically examined efforts to privati ze education services through the use of vouchers by looking at three states where vouchers were enacted (Ohio, Wisconsin, and Florida), as well as three states where vouchers ha ve not been enacted, but where serious attempts have been made to establish such programs (Michigan, New Mexico, and Pennsylvania). By examining policy maker reaction t o arguments designed to promote or prevent voucher programs in states where vouchers w ere established or under serious
3 of 39consideration, this research sought to help both pr oponents and opponents focus on the concerns of policy makers in education reform.BackgroundThe education market movement is a compilation of a t least three different efforts to expand the role of private providers in public educ ation. These efforts include contracting out support and curricular services (such as food s ervice, student transportation, and curriculum programs); contracting with a private co mpany for management of an entire school or district (such as through Edison schools) ; and privatization of school governance (charter schools, tax-credits, and vouch ers). While the term privatization may encompass all of these movements, as it is used her e, privatization, vouchers, or market reform will generally refer to the privatization of governance through the transfer of public funding from public schools to the private s ector.MethodIt is not until recently that the use of vouchers a s a mechanism for creating a free-market system has been seriously considered; and not until the 1990s that a useful model of a voucher system established. The social and politica l contexts under which voucher programs were established then are relatively recen t. This recent history presented some unique methodological opportunities (the chance to survey actual participants in the policy making process) and difficulties (no conclus ive outcome or evaluative measures, continued doubt as to the constitutionality [at the state level] of such programs). For a detailed description of the methodology used in thi s research, see Appendix I. In general, a quantitative methodology that allowed for wide spread application and comparative analysis, was sought, and consequently, a survey of legislators was chosen as the best option for answering research questions across a wide geographic range and large body of potential data sources. The focal poi nts of this study (Florida, Michigan, New Mexico, Ohio, Pennsylvania, and Wisconsin) were chosen using both critical case sampling and politically important case sampling (M artella, Nelson, & Marchand-Martella, 1999).The survey was constructed using proand antivou cher arguments uncovered during a review of voucher literature. During the validation process, the survey was amended to include questions related to broader issues, includ ing policy maker views on the purpose(s) of education, as well as the potential e ffectiveness of a variety of current reform proposals. Demographic data, including race/ ethnicity, gender, age, and religious affiliation, was gathered to test for any specific response patterns. On October 18, 2000, the survey was mailed to 936 state legislators in F lorida, Michigan, New Mexico, Ohio, Pennsylvania, and Wisconsin. Of the 936 surveys mai led, 89 were returned, for an overall return rate of 9.5% (see Table A1, Appendix 1 for a detailed discussion of the return rate and Appendix 2 for a discussion of methodological i ssues).General Overview of FindingsJust under half of the policy makers surveyed (48%) supported some sort of voucher reform, however, when asked to evaluate vouchers in isolation, and when examined in relationship to other reform options, their support weakened, with vouchers ranking last
4 of 39among eleven other reform options. Policy makers al so viewed the broader concept of school choice more negatively, with three of the fo ur lowest ranked reform options on this survey related to school choice (teacher prepa ration and professional development, early childhood initiatives, phonics based reading programs, and greater use of technology all rated higher as reform strategies).While choice was not favored in comparison to other reforms, policy makers still generally seemed to accept pro-market arguments for voucher programs, as well as statements related to allowing consumers to regain control of educational systems. While respondents seemed to accept at face value statemen ts that competition will improve services, they did not support statements related t o the potentially negative impact of market forces.Even though there was broad agreement on the benefi ts of a pro-market approach, there were concerns expressed relative to resource alloca tion. Respondents did not support the idea that vouchers would help equalize funding, or provide new schools, increased investment, or improved cost controls. In fact, the y felt that voucher programs might result in reduced services to special education stu dents and increased competition for the "best" students. Respondents were also concerned th at private schools would raise tuition and fees, ultimately limiting accesses to the schoo ls by poorer families. One of the most common warnings of voucher opponent s is that voucher programs may result in a separation of students by race. Three i tems were related to this concern were included in the survey. Voucher advocates supported two of the more positively worded items (that vouchers would force schools to focus o n customers seeking specific academic, social, or religious programs, and that s uch programs would result in schools with specific religious affiliations or racial/ethn ic compositions), while voucher opponents were more likely to support the more nega tively worded suggestion that voucher programs would increase segregation by race religion, or income. While some policy makers expressed concerns that vo uchers would result in private schools loosing independence, others worried that t here would be a jump in the number of low quality schools, as entrepreneurs sought acc ess to newly available public funding (this dichotomy between independence and oversight has played out in all of the programs enacted so far, which currently include ei ther no, or very weak, evaluation and oversight components).When examining education more generally, respondent s agreed that the overriding purpose of education is to ensure academic excellen ce in students. Respondents also supported the statement that education should ensur e that students are prepared to meet the needs of businesses and employers, however vouc her advocates were much more likely to want schools to instill strong moral char acter in students, while voucher opponents wanted schools to create good citizens. T here was also a general reluctance to rank social purposes-including the suggestion that education should be used to promote social mobility or diversity-strongly. Women as a g roup tended to rank the creation of good citizens more strongly than men, but that emph asis came at the expense of business. Respondents who were minority group memb ers were slightly more likely to rank social mobility higher than Caucasian responde nts. For both women and minorities, however, the degree of difference between their opi nions and that of the majority was not statistically significant.
5 of 39Item Analysis1Demographic DataThe typical respondent to this survey was a white p rotestant male, however, the diversity represented by the respondents is similar to that o f legislators nationwide and across the study states (see Figures 1 & 2). Approximately 79 % of respondents were male, and 19% female, with 2% not answering (this figure rema ins constant and so will not be repeated). Approximately 80% of respondents were wh ite, with 6% identifying themselves as black, 11% Hispanic, and 1% Native Am erican. The age range of respondents is shown in Figure 3. The large majorit y of respondents were actively religious (87%) and Protestant (65%). 26 percent we re Catholic, and 2% were Jewish. Most respondents had completed a Bachelors degree a nd many had gone on to post-secondary education (BA, 46%; Professional deg ree, 15%; MA, 14%; Ph.D., 7%). A slight majority of respondents were Republican (53% ) while Democrats made up 45% of the responders. A large majority had also served on the education committee (66%). Most respondents attended public K-12 schools (72%), and none attended private non-parochial schools at the K-12 level. Most of th e policy makers also attended a public college (59%), while those who went to a private co lleges and universities were spilt between religious institutions and nonreligious. Wh ile 72% of respondents who had children sent them to public schools, the 28% of re spondents who sent their children to private schools chose parochial schools over non-pa rochial ones (83% to17%). While education service was generally low (ranging from 0 6% in most categories), of those individuals who had worked in education, 27% said t hey had worked as a public school teacher, 22% had volunteered in a public school, an d 9% had volunteered at a private school. Figure 1. Survey/across state demographic compariso n
6 of 39 Figure 2. Survey/across national demographic compar ison Purpose of EducationParticipants were asked to identify what they thoug ht the primary purpose of education is, given six options (instill academic excellence, ens ure students can meet the needs of business, create good citizens, instill moral value s, promote social mobility, promote diversity), as well as an option to write in an opt ion of their own. The majority of legislators responding to this survey clearly felt that the primary purpose of education is to ensure academic excellence in students. 61 perce nt of respondents identified this purpose as their primary choice and 80 percent rank ed this option as first or second. The second choice of respondents was to ensure that stu dents are adequately educated to meet the needs of businesses and employers. 45 percent o f respondents ranked that option as first or second. Very few respondents considered so cial mobility or the promotion of diversity an important function of schooling. Figur e 4 shows how the respondents reacted to each option. Figure 3. Age of respondents
7 of 39 Figure 4. Purpose of Education Success of EducationTo evaluate the perceived need for education reform respondents were asked to rate the effectiveness of schools in addressing the primary purpose of education they identified in the first item. As with the annual polls conducted by Phi Delta Kappa and Gallup, which have consistently shown that the public rates local schools higher than they do schools nationally (Rose & Gallup, 2000), the policy makers responding to this survey also rated local schools more positively. While the overall di fference was relatively small (local schools had an average rating of 2.8 compared to sc hools nationally, at 3.4), it is statistically significant. More respondents gave lo cal schools a one, two or three at each level (the highest ratings) than they did schools n ationally, and more schools nationally were rated with a four, five, or six at each level (the lowest ratings), than were local schools. There was however a strong tendency toward the middle ranges, with 44% of respondents giving local schools a three or four an d 72% rating schools nationally with a three or four.ReformsEven at the local level then, policy makers see sig nificant room for improvement in school performance. While this research focused pri marily on vouchers, market reforms are only a part of the entire reform picture. Resea rch from Tennessee and Wisconsin has shown class size reduction as an effective tool to help raise achievement (Pritchard, 1999), while research by William Sanders (Sanders & Rivers, 1996) has shown teacher effectiveness, and by extension teacher preparation to be the largest influence on student achievement. Policy makers across the country have also been strengthening testing and accountability procedures, and Charter schools are now supported in most states, educating more than 250,000 students (Shokraii-Rees 2000). When policy makers consider voucher programs, it is within this wide v ariety of reform options. To determine the degree of effectiveness policy makers feel thes e reforms may have in improving education, respondents were asked to rate eleven po ssible reforms: class size reduction; site based management; vouchers; open enrollment; s tandards, testing, and
8 of 39accountability; greater use of technology; early ch ildhood initiatives; phonics based reading programs; teacher preparation; increasing t eacher salaries; and charter schools. In addition, respondents were given the opportunity to add some other reform if they desired. Figure 5. Estimated effectiveness of selected refor msÂ—highest to lowest The highest rated option (of the choices offered) w as reforming teacher preparation and professional development; however, the difference b etween teacher preparation and early childhood initiatives was not statistically signifi cant. 77 percent of respondents rated teacher training reforms with a one or a two, and t he item's average rating was 1.90. Early childhood initiatives had the same number of "1" ra tings, but only 67 percent of respondents rated such initiatives a 1 or a 2, resu lting in a 2.02 average (see Figure 5). The reform respondents had the least overall confid ence in was vouchers, followed by charter schools and raising teacher salaries. Despi te their apparent closeness, the difference between vouchers as a reform, and charte r schools or raising teacher salary, is statistically significant. This apparent lack of co nfidence in vouchers as a reform is surprising given that the majority of respondents a ctually supported some type of voucher program (as discussed later in the paper). It is po ssible that respondents may see vouchers as only a limited solution while the other reform o ptions are seen as more available to all students, and so in comparison, vouchers were rated less favorably. As shown in Figures 6 and 7, it is also possible that voucher opponents skewed the average rank downward by choosing the lowest possible options, while voucher supporters spread their ratings more evenly across the scale (there were 39 ratings of f ive or six for vouchers, while the rest of the responses were spilt across ratings 1-4).
9 of 39 Figure 6. Education reform Support for VouchersRespondents were asked to rank order their support for vouchers given options from support for no type of voucher program to support o f an unlimited system (see Figure 8). The response ranked first most often was "no vouche r system", however, when the first ranked responses for all types of voucher programs were combined, more respondents supported vouchers than opposed them (48% to 45%, w ith 7% undecided). Another key finding relates to what options respondents ranked second and third. While voucher opponents were forced to consider what type of vouc her system was least objectionable (and they preferred vouchers for poor students firs t, then students at poor performing schools), voucher advocates had the option to choos e another type of voucher system, or no voucher system at all. This all or nothing appro ach allowed an evaluation of the relative strength of the stated support for voucher s. Two thirds of respondents who ranked vouchers for poor students as their top choi ce chose an unlimited voucher program as their last choice. While this trend was not as strong with supporters of vouchers for families at low achieving schools (25% ), it does suggest that the support of individuals who back narrowly tailored voucher prog rams might not generalize to support to all voucher programs. Not a single respondent ra nked vouchers limited to non-parochial schools as their first choice.
10 of 39 Figure 7. Evaluation of vouchers as a reform Because of the small number of undecided policy mak ers, their responses were generally not analyzed as a group, however five of the six re spondents expressed a tendency to support some type of voucher system in their second place ranking, and so were included in the pro-voucher statistics. The other policy mak er, who expressed a tendency toward no support for vouchers, was included in the anti-v oucher analysis. There is however a moderately strong correlation (r = -0.69) between support for vouchers for impoverished families and vouchers for students in poor performing schools (given the general correlation between high poverty and low performance, this may not be surprising). 89 percent of respondents supportin g vouchers for impoverished students also supported voucher use at low performing school s. The reverse was also true, although slightly less so, with 75% of respondents who supported provision of vouchers to students in poor performing schools, also suppor ting provision of vouchers to impoverished students. Figure 8. Support for voucher programs Pro-voucher Statements
11 of 39Respondents were asked to rank order 15 pro-voucher statements which were organized along three lines: excellence, access and equity, a nd education structures. Within the category of excellence, the highest ranked statemen t of the five options (a statistically significant difference when compared to the second ranked option) was "...voucher programs will make public schools more accountable by allowing consumers the option to take their business elsewhere." 48 percent of re spondents ranked this item first, and its average ranking was 2.04. The next ranked item, wit h an average of 2.67 was "...voucher programs will stimulate innovation, research and de velopment as schools seek better ways to increase achievement." The lowest rated ite m (3.81) was "voucher programs will result in increased service as schools are forced t o offer incentives and enticements to potential students." Both opponents and supporters of voucher programs placed these statements in the same order (on average).With regard to access and equity, the highest ranke d statement (of six options) was "voucher programs will force all schools to offer s trong academic programs, increasing access to quality education for all students." This item's ranking was not statistically different from the second highest ranked statement, "voucher programs will allow funding to follow the student, making it important for public schools to meet the needs of every student." The lowest ranked item, with an ave rage rank of 4.56 and no number one ranking, was "voucher programs will increase fundin g by allowing new investors and entrepreneurs to enter the market." While the top t hree items in average rank were the same for both supporters and opponents of vouchers, voucher opponents gave a lower overall ranking to the statement, "ease the entry o f new schools into the market place, increasing access to education."In the final pro-voucher grouping, education struct ures, respondents strongly felt that voucher programs would "return control to parents" (a statistically significant difference when compared to the second ranked option). Respond ents ranked as last the claim that costs would be reduced as fiscal management practic es are emphasized. While the overall total was the same for the highest and lowest ranke d items, voucher opponents ranked the statement, "reduce bureaucratic oversight since poo r performing schools will be forced by consumers to improve or close" first twice as of ten as they did the "return control to parents" item.Anti-voucher StatementsThe patterns in responses to the anti-voucher state ments are much less defined. For instance, under "Excellence" the statement with the most number one rankings (of six options) was item 47, "voucher programs will force schools to compete for only the best students in order to maximize achievement scores," however, a large number of "six" rankings lowered the mean score of this item to 2.8 9, placing it second by average. The item with the highest mean rank was item 49, "vouch er programs will result in an increased emphasis on standardized tests." Another complicating factor in this analysis is the mode. In this case, item 49 had a mode of "3", while item 47 had a mode of "1", and the third and fourth ranked items both had a mode o f "2". There was also a low degree of variability in the averages. The highest ranked ite ms averaged 2.89 and 2.75 respectively, while the third and fourth ranked items averaged 3. 21 and 3.36, and the fourth and fifth ranked items averaged 4.31 and 4.32 (none of these pairings had rankings that were statistically different). The lowest ranked items w ere item 51, "voucher programs will
12 of 39cause public schools to transfer existing resources to advertising and marketing," and item 50, "result in less spending on research and d evelopment as costs are cut." Voucher supporters and opponents did not rank eithe r the highest or lowest average ranked statement together. Supporters preferred the statement with the overall highest ranking, while opponents preferred item 49. Propone nts rated item 51 lowest while opponents ranked item 50 lowest.The responses to anti-voucher statements in the acc ess and equity section resulted in a more clearly defined top ranked item (that was stat istically difference from the second ranked option), but an even more confused picture o f the other five items. The top ranked statement, vouchers would "reduce services to stude nts traditionally more expensive to educate (special education)," had a mode of two (no item had a mode of one), and an average of 2.71, while the other items all averaged in the three range. With a variance in their average rankings of .08, the other five items were virtually indistinguishable. An examination of their modes did not clarify the rank ings much, with the modes running a three, two fours, a five and a six, with a correlat ion between the average rank and mode occurring at only four and six. The lowest ranked i tem (six mode and sixth by average) was, "voucher programs will result in reduced servi ces for students in rural or low attendance areas." Despite the generally confused p icture of the rankings on this item, the first and last ranked items were the same for both voucher supporters and opponents. While the modes and averages are confused for the n ext section (Structures of education) as well, a greater variability between the averages lends some clarity to the picture (even though the difference between the first and second ranked items was not statistically significant). The item with the highest average, 2. 69, and a mode of 2, was, voucher programs will result in "the diverting of resources from the public sector to the private," while the next ranked item (average of 2.81 and a m ode of 1) was "private schools losing independence as politicians attempt to ensure overs ight of public funding through regulation." The lowest ranked items were again vir tually indistinguishable, and included, voucher programs would result in "a proli feration of low quality schools designed to quickly access public funding with litt le or no oversight," "lower educational quality due to a lack of state and district oversig ht," and "increased segregation by race, religion, or income." The last two statements had t he lowest average rankings at 4.061 and 4.059 respectively.The highest ranked item by voucher supporters and o pponents was, "voucher programs will result in the diverting of resources from the public sector to the private." Proponents, on average, ranked "increased segregation" last, wh ile opponents ranked "a lack of oversight" last.Written CommentsGenerally written comments were in response to opti ons marked "other", and comprised a very low percentage of responses to such items. T he one exception was the first item: the Purpose of Education. Five comments were made r egarding the "other" option on this item, and they were: All of the above are important parts of the whole. 1. Ensure equity among all ability levels. 2.
13 of 39Ensure students are given the tools they need to ma ke positive change and keep learning. 3. Prepare every student for life participation in our economy. 4. Ensure people are adequately educated to lead a goo d family and financial life. 5. Four comments were evaluative of the survey in gene ral. Two respondents commented that they felt the survey was slanted against vouch ers, while one commented that there was clearly a pro-voucher bias. A fourth respondent indicated discomfort with being forced to rank items that he/she did not support.Cross-Sectional AnalysisSupport for VouchersOne of the main purposes of this research was to ex amine the relationship between support for vouchers and the other items on the sur vey. To do this, a correlation coefficient was generated between support for or op position to vouchers in general, and then by type of program, and each item on the surve y. Once an initial relationship was detected, a X2 analysis was conducted to examine relationships wi thin the correlation. A respondent's ranking of the voucher statement "I support the concept of no voucher system" (item 22) was used to identify the level of support for such programs. While the wording of the item suggests opposition to vouchers a ranking of 5 or 6 indicated support for vouchers rather than opposition. This i tem was used rather than the statement of support for an unlimited voucher system (item 26 ) because that allowed for inclusion of support for all types of voucher programs. There was a relatively strong negative correlation between the responses on item 22 and it em 26, support for unlimited voucher programs (-0.74), which would support the use of it em 22 in identifying support for voucher programs.A respondent's ranking of the statements in item 22 also strongly correlated with item 12, evaluation of vouchers as a reform strategy (-0.898 ), and moderately with item 16 (evaluation of early childhood education as a refor m strategy, r = 0.56), as well as item 70, party affiliation (r = -0.54). There was no lin ear relationship between support for vouchers and item 56, decreased access to quality s chools as private schools raised tuition, (r = .000).To further test the relationship between these item s, a X2 test was performed, with H0: item X and item Y are independent; and Ha: item X and Y are not independent. Using a significance level of .05, and combining upper, mid dle, and lower rankings (1,2; 3,4; 5,6) so as to ensure at least 5 expected cell counts, th e null hypothesis was rejected in favor of the alternate hypothesis, that is: item 22 and item 12 are not independent (p = 2.7(10)-14< 0.05). In this test, the two groups compared were voucher supporters and voucher opponents. Upon further examination of the distribu tion of rankings, it turned out that every respondent opposed to vouchers ranked the lik ely effectiveness as a 4, 5, or 6, while voucher supporters generally ranked the refor m highly. While this may not seem surprising, it does buttress the internal validity of the document, as well as the original rejection of a Likert scale for the instrument (whi ch was rejected because it was feared that respondents would tend to the extreme and rank the statements on the high or low
14 of 39ends of the scale).An X2 test could not be performed on item 12 as useful c ategorizations could not be constructed. The correlation may also be the result of a lack of variance in the rankings, with respondents choosing a 1, 2 or 3 for most ever y estimate of the effectiveness of early childhood initiatives. A X2 test was successfully used to examine the relation ship between party affiliation and support for vouchers and once again, the null hypothesis was rejected in favor of the alternate hypothesis ( p = 0.0000056). Figure 9 illustrates the relationship. As could be expected, although there is some crossover appeal, the bulk of support for or opposition to vouchers falls along p arty lines. Figure 9. Party affiliation Stronger correlations became evident when controlli ng for the type of voucher program supported. When looking at individuals who supporte d only voucher programs for impoverished students and students in failing schoo ls, moderate to strong correlations were found with ten items in the survey (items 19, 39, 40, 49, 54, 56, 60, 61, 62, and 84). This group was identified by controlling for suppor t for vouchers but opposition to an unlimited voucher system (item 26). The strongest c orrelation, at 0.94, was between limited support for voucher programs and item 54, t he statement that voucher programs would "result in a reduction of services and opport unities for all students as public schools enact stricter cost controls." Because supp ort for limited voucher plans was identified by looking for a negative evaluation of unlimited voucher programs, this correlation could signify a general lack of support for the statement, even though the correlation is positive. Such an assumption is conf irmed through further examination of the data (70% of respondents supporting limited vou cher programs did not ranked this statement strongly).A strong relationship (-0.81) was also found betwe en support for limited programs and item 40, the statement that voucher programs would ease the entry of new schools into the market place, increasing access to education. I n this case then, the correlation could signify support for the claim that vouchers will in crease access to education. Upon further inspection of the data, however, this is no t necessarily the case. While 50% of the group did rank the statement strongly (1, 2, or 3), the other 50% ranked the statement negatively (4, 5, or 6). When looking at the total evaluations of this statement however, there does seem to be slightly greater support than was evident in the larger population of
15 of 39voucher supporters (where 78% ranked the statement with a 4, 5, or 6). While there was no moderate or strong correlations between support for or opposition to voucher programs and the other survey items not hig hlighted, there were some trends that should be mentioned. In previous writings, women ha ve been identified as opposing voucher programs. While there was only a weak corre lation between gender and voucher support (i.e.., males and females analyzed together ), 63% of female respondents (n = 16) did oppose voucher programs. In addition, while rac e or ethnicity generally did not correlate strongly with support for voucher program s, every African-American respondent supported vouchers (n = 5). Hispanic res pondents were mixed in their support, with 4 opposed to all types, 2 supporting a limited program, and 1 supporting an unlimited program. These n's are especially small a nd should not be generalized to the larger minority population.Evaluation of school effectivenessThere was not a strong correlation between a respon dent's evaluation of the effectiveness of schools and any other item on the survey, althou gh there was a positive correlation between a respondent's view of school success local ly and nationally. Respondent's who ranked local schools poorly (that is, with a 4, 5, or 6), were more likely to support some type of voucher plan than the population as a whole (63% to 48%), while respondents ranking schools strongly were less likely to suppor t voucher programs (69% to 45%). To further examine this relationship, a X2 test was applied comparing the estimated effectiveness of local public schools with support for voucher programs as determined by item 22. Scores were counted as (1,2,3) and (4,5,6) to supply the required expected cell count, and a level of significance of 0.05 chosen. The resulting p value was 0.37, which means there is not sufficient evidence to cause rej ection of the null hypothesis. While it appears that these items might be independent, the X2 test confirms that this is not the case.Support for statements and voucher supportAn X2 analysis was used to examine the relationships bet ween respondent support for vouchers and the policy statements offered in the s urvey. Using a level of significance of .05 and grouping the responses by pairs, the null h ypothesis, that the items are independent, was rejected in eight of the 33 items. The statements, along with their "p" scores, are identified in Table 1.As Table 1 shows, there is a large disparity in the rankings of voucher supporters and voucher opponents on five of the statements. Vouche r advocates generally viewed the statement "Voucher programs will force public schoo ls to focus on the academic, social, or religious demands of specific customer groups" m ore positively than did opponents. A similar statement that was worded more negatively a lso appears in the table: "voucher programs will result in increased segregation of st udents by race, religion, or income." When worded in this manner, voucher proponents rank ed the item negatively, while anti-voucher respondents ranked the item more stron gly. Another item related to the issue of student-body composition and school focus appear s on the list. The statement, "voucher programs will result in a proliferation of schools with specific religious affiliations or racial/ethnic composition," had a h igher average rank with voucher
16 of 39 advocates than with voucher opponents. The other tw o items with a large difference in average rank were: "voucher programs will result in private schools losing independence as politicians attempt to ensure oversight of publi c funding through regulation," and "voucher programs will result in a proliferation of low quality schools designed to quickly access public funding with little or no ove rsight." Voucher advocates ranked the statement regarding loss of independence higher on average that opponents, while voucher opponents were more concerned with a prolif eration of low quality schools. A final note needs to be made regarding interpretat ion of the items in this section. While care was taken to examine relationships between ite ms, and between supporters and opponents of voucher programs, the analysis conduct ed here cannot suggest causation, and can only suggest areas where there appear to be relationships. In addition, the structure of the survey forced policy makers to ran k a series of items. As with any forced ranking system, a low ranking is not necessarily ne gative. Respondents may agree or disagree strongly with all the items, but still ran k them as the survey asks. While this effect is mitigated as the number of responses incr eases it would be unwise to make any absolute statements regarding the impact of the ite ms on this instrument.Table 1 Relationships Between Policy StatementsItem"p" valueData grouping AvgRank Pro-Rank Anti-Rank DifferenceStatement: 430.049By item2.252.382.080.3Voucher programs willreduce bureaucratic oversight since poor performingschools will be forced by consumers to improve orclose. 480.006Pairs3.212.883.7-0.82Voucher programs willforce public schools to focus on the academic, social, orreligious demands of specific customergroups.
17 of 39 520.04Pairs3.363.743.9-0.16Voucher programs willcreate greater pressures to falsely manipulateachievement measures and data reports. 580.019Pairs3.973.844.03-0.19Voucher programs willresult in reduced services for students in rural or lowattendance areas. 590.008Pairs3.072.543.73-1.195Voucher programs willresult in a proliferation of schools with specificreligious affiliations or racial/ethniccomposition. 610.00148Pairs2.812.183.67-1.49Voucher programs willresult in private schools losing independence as politiciansattempt to ensure oversight of public fundingthrough regulation. 630.006Pairs43.94.473.251.22Voucher programs willresult in a proliferation of low quality schoolsdesigned to
18 of 39 quickly access public funding with little or nooversight. 640.0017Pairs4.064.743.391.35Voucher programs willresult in increased segregation ofstudents byrace, religion,or income.Using a Chi-square test, the rankings of the statem ents in the chart above (by legislators grouped according to voucher position) were determined to b e related, using a level of significance of 0.05. Most responses were paired (1,2), (3,4), (5,6) to f ulfill the requirements of Chi-squared tests. Associated average rankings of the entire group, an d by level of support for vouchers, as well as the difference between rankings of voucher advocate s and opponents, is also included.DiscussionBefore reviewing the findings from this research, i t bears repeating that while these conclusions can be used to inform the controversy s urrounding the voucher debate, they cannot be interpreted or generalized to the larger body of legislators. The sample size, though largely composed of legislators in key leade rship positions on education issues, and demographically reflective of legislators natio nally, was small. This is also the first time this survey has been administrated, and no tre nd data are available; thus these data are representative of a single point in time and ca nnot be portrayed as sustainable. Because this effort can be viewed as a pilot, resea rchers interested in replicating this research, or conducting similar legislative surveys may be able to draw useful lessons by examining the survey application and study design.While similar surveys of legislators have shown sli ghtly higher rates of return, surveys of policy makers generally appear to have lower respon se rates than surveys of other populations. A review of dissertations available fr om ProQuest, an on-line repository of dissertations ( http://www.umi.com/hp/Support/DServices/products/d a.htm ), showed survey response rates for legislators ranging from 24% to 61%.(Note 2) Separate from this study, research into why policy makers respond poorly to surveys, as well as techniques that might be used to improve response r ates, could fill a significant gap in the data (see Appendix 2 for a detailed discussion of t he survey returns and how future research might address issues uncovered here).This research sought to answer five basic questions : What role do policy makers feel vouchers play in th e larger context of reform? How do policy makers react to specific policy argum ents about vouchers? How does the educational philosophy of individual p olicy makers relate to their attitude regarding vouchers? How do the demographic traits of policy makers rela te to an their philosophy of education? How do the demographic traits of policy makers rela te to their views on vouchers?
19 of 39An analysis of each of these questions follows.What role do policy makers feel vouchers play in th e larger context of reform?While 48% of policy makers surveyed support some so rt of voucher reform when asked to evaluate vouchers in isolation, when examined in the relationship to other reform options, their support appears much weaker, ranking last among eleven other options. Though apparently counter-intuitive, this is perhap s not an unusual finding. Annual polls sponsored by Phi Delta Kappa show just less than a majority of people they survey support vouchers generally (44% in 2000), however, when asked, "Which would you prefer improving and strengthening the existing p ublic schools or providing vouchers for parents to use in selecting and paying for priv ate and/or church-related schools?" only 27% of respondents chose providing vouchers (Rose & Gallup, 2000). If we look at the concept of choice and deregulatio n as broadly identified in this survey, policy maker support continues to be low. Three of the four lowest ranked reform options on this survey related to school choice (vouchers p laced 11th, charter schools placed 10th, open enrollment placed 8th), while another op tion related to decreased regulation, site based management, placed 7th. The top rated re forms also appear more representative of the concept of change from within Greater support for teacher preparation and professional development, early chi ldhood initiatives, phonics based reading programs, and increased use of technology, suggest that policy makers still feel the "inputs" are important. That is, that change in "what" the schools do, rather than "how" they do it, is still a primary emphasis of po licy makers. How do policy makers react to specific policy argum ents about vouchers? In general policy makers showed a surprising unity of opinion regarding pro-voucher arguments. On average, both opponents and advocates gave 11 of 15 pro-voucher statements the same rank. The items ranked highest were pro-market in nature, suggesting that this has been an area where voucher advocates have successfully stated their positions. In addition, the highest ranked pr o-voucher statements could also be characterized as related to the control of educatio nal decisions, that is, consumers retaining control of educational systems. This is n ot a new concern in educational circles, as one of the primary results of education reform i n the 20th Century was a consolidation and centralization of educational services. While t his consolidation often results in a broader range of educational opportunities, it has also brought on a sense of isolation and loss of power in many communities (Cremin 1964; Rav itch 1983; Tyack, 1996). If pro-voucher support groups are able to capitalize o n this discontent they could gain a significant philosophical boost, moving their argum ents beyond the traditional market advocacy.While there was broad agreement on the pro-market a nd control statements, there was also agreement on the lower ranked statements, whic h were primarily related to resource allocation. Statements concerning equalization of f unding, the opening of new schools, increased investment, and better controlled costs w ere all ranked low. While such low rankings do not necessarily suggest a lack of faith in these effects, they at least suggest that this is an area were voucher opponents could m ake inroads against some traditionally
20 of 39argued pro-voucher positions. In addition, voucher proponents should take seriously concerns related to resource allocation within vouc her systems. Such a strategy was used in Florida, Ohio and Wisconsin, where voucher progr ams also came with increased public school resources or resource guarantees, but was missing in the recently defeated ballot measures in California and Michigan.As stated previously, there was a notable lack of u nity in responses to the anti-voucher statements. Such a lack of consensus suggests that opponents of vouchers may not have been as successful as proponents in focusing policy makers on their concerns. While this lack of unity was true for the respondents in gener al, some broad patterns are still discernible. Both supporters and opponents of vouch er programs exhibited some equity concerns. Statements predicting reduced services to special education students and increased competition for the "best" students were highly ranked by both groups. Respondents were also concerned that private school s would raise tuition and fees, ultimately limiting accesses to the schools by poor er families. As identified earlier, concerns related to resource allocations were highl y rated. These statements were most frequently made by voucher opponents, and such supp ort suggests that the message has been successful in reaching at least some policy ma kers. These concerns should also be noted anew by voucher supporters in designing pilot programs. Such findings confirm advice offered by Coons and Sugarman (1999), sugges ting that advocates seek to establish narrowly tailored programs that take reso urces and equity concerns seriously, and then later attempt to expand the programs.As evidenced in Milwaukee, and to some extent in Fl orida, it is easier to expand a program once it is already in effect. The findings also indicate that advocacy of a format similar to Florida's statewide program might be a s uccessful approach for other states, since support of vouchers for impoverished students (the Milwaukee and Cleveland model) correlated with support of vouchers for stud ents in poor performing schools (the Florida model). Such an approach could then be used to extend voucher programs merely by changing the definition of "poor performing scho ols", whereas changing the definition of poverty would undoubtedly be more difficult.As with the pro-voucher statements, respondents gen erally did not rank very highly statements related to reduced services because of e fforts to cut or control costs. This could signal a possible disconnect in the market mo del. While respondents seem to accept at face value claims that competition will i mprove service, they do not accept statements looking at negative market forces. While this dichotomy is present here, and in the advocacy literature as well, an understandin g of why this disconnect exists could prove useful. For example, are negative market infl uences seen as unlikely because they haven't been generally considered (in which case vo ucher opponents could have a new position to pursue), or is there a belief that such influences are a good thing (forcing bad schools to go out of business), or that they won't affect good schools? One of the most common warnings of voucher opponent s, and a major reason the NAACP has taken an anti-voucher position, is that v oucher programs may result in a separation of students by race. Historically, when schools were desegregated voucher programs were intentionally established to maintain racial separation, most notably in Virginia (although such programs were ruled unconst itutional at the time). Three items in the anti-voucher section were included related to t his concern. On two of the positively worded items (that still imply student separation b y race, religion, or social status), voucher advocates ranked the items strongly. The it ems stated: voucher programs will
21 of 39"force public schools to focus on the academic, soc ial, or religious demands of specific customer groups," and, voucher programs will result in "a proliferation of schools with specific religious affiliations or racial/ethnic co mposition." While voucher opponents were less likely to rate these statements strongly, they did support the statement that such as programs would "increase segregation by race, re ligion, or income." The narrow line between meeting constituent demands and the creation of an illegal or unethical segregation of students, is one requiring close examination. Clearly, supporters feel that addressing constituent needs is important but opponents may have valid concerns that such efforts will result in a segrega ted system. Further definition and clarification of this narrow line, and how constitu ent needs can be met without resulting in a segregated system should be a concern of vouch er advocates. At the same time, this could continue to be an effective angle for voucher opponents to exploit. Because of the low number of undecided respondents (n=6), there has been little mention of their responses pattern on these items. There we re however, two key items on which they responded similarly. As a group, five of the s ix were Democrats, and four of the six respondents expressed concern that vouchers would r esult in private schools loosing independence.This was also a concern expressed by voucher advoca tes. Voucher opponents did not generally share this concern; they did, however, ra nk strongly a statement that vouchers would result in a "proliferation of low quality sch ools designed to quickly access public funding with little or no oversight." This dichotom y is perhaps one of the most difficult to solve in program design, and one which voucher o pponents can continue to highlight. This difference in perspective has played out in al l of the programs enacted so far, which currently include either no, or very weak, evaluati on and oversight components. Proponents typically argue that market forces will perform the oversight function, and that oversight is a "poison pill," believing that p rivate schools will avoid systems that involve even a small amount of government oversight or regulation. The PDK poll cited earlier lends support to the use of oversight as li miting strategy by voucher opponents, as 76% of respondents to that survey thought private v oucher schools should be required to meet the same accountability standards as public sc hools (Rose & Gallup, 2000). How does the educational philosophy of individual p olicy makers relate to their attitude regarding vouchers?In general, respondents agreed that the overriding purpose of education is to ensure academic excellence in students. There was, however a small divergence of opinion when looking at the second and third ranked stateme nts, which within each group were indistinguishable. Both groups supported the statem ent that education should ensure that students are prepared to meet the needs of business es and employers, however voucher advocates were more likely to want schools to insti ll strong moral character in students, while voucher opponents felt schools should work to create good citizens. Voucher advocates were also more uniform in their r esponses to these items. Through their first four rankings, they generally moved thr ough the options in groups (1-6-2-4, or 6-1-4-2), diverging on items three and five. While voucher opponents generally ranked the first item strongest, they quickly diverged, sp litting on items two and six, with no discernible response pattern among the remaining it ems. This suggests that while these
22 of 39groups share a sense of common purpose overall and then initially within the groupings, there is more divergence of opinion for voucher opp onents. While it is a stretch to label either group "liberal" or "conservative", one of th e more common political generalizations over recent years has been that con servatives have generally been more unified in their approach to policy, whereas libera ls have put forth a less unified vision. At the very least, these results suggest that effor ts of advocates to frame vouchers as a "social equity" issue might be effective to a degre e. The general reluctance, however, of either group to rank social purposes strongly sugge sts that the effects would be small (for example, only two voucher supporters, and four vouc her opponents ranked social mobility or promotion of diversity as a primary [1 or 2 ranking] role of education). How do the demographic traits of policy makers rela te to their philosophy of education?It is difficult to determine a relationship between educational philosophy and demographic traits because of the unity of opinion related to the primary purposes of education and because of the small "n" for demograp hic differences. Even so, there were some disparities. Women as a group tended to rank t he creation of good citizens more strongly than men. The emphasis on citizenship came at the expense of business, which had a lower average rank for females than it did fo r males (in effect, women as a group ranked citizenship second, business needs third, wh ile the rankings were the reverse for men). Respondents who were minority group members were slightly more likely to rank social mobility higher than white respondents. The first three rankings, however, were the same for each group (promote academic excellence, m eet business needs, and create good citizens). In either case, for women and minor ities, the degree of difference between their opinions of educational purpose and that of t he majority of respondents was not statistically significant. The same was true when d isaggregating by religion. How do the demographic traits of policy makers rela te to their views on vouchers? Just as with the purpose of education, there is no apparent demographic pattern to those who support or oppose voucher programs. Where there is a difference, because the "n" in each subgroup is so small, the differences are gene rally the result of only one or two respondents, and therefore, not significant. While such a lack of disparity might suggest that support for vouchers is broad based, that, too is a generalization that should nor be made in light of the small number of minority and f emale respondents.Additional ResearchClearly, there is a need for more research into the impact of voucher programs on student learning. Given recent claims by some researchers t hat voucher programs stimulate public school improvement, research into the coroll ary effects of voucher programs on the education system should also be conducted. It i s possible that a pilot voucher experiment could partially answer these questions. Given the failure at Alum Rock, however, along with the tendency of elements on bot h sides to back advocacy research over hard science, success of such a program, to sa y nothing of the possibility of starting one, seems tenuous.While not directly cited here, research into press coverage of voucher issues may also be
23 of 39significant. While pro-voucher research groups like the Heritage Foundation, the Manhattan Institute, and the Project on Education P olicy and Governance have proved skillful in getting their research into news storie s, voucher opponents seem to have been much less successful. An understanding of the inter action between the advocacy organizations, voucher researchers, the press, and public opinion, could all be further areas of inquiryAdditional research into survey methodology, specif ically as it relates to policy maker response rates, could also prove useful. As discuss ed in Appendix 2, many of the techniques used to increase response rates on surve ys of the general public may not apply to policy makers, since courtesies such as personal salutations and signatures are expected, while postage costs are not a concern due to franking privileges. Continued examination of policy maker reactions to the various advocacy arguments identified here is important. It seems likely that the debate will remain a philosophical one, and as such, attempts to sway legislators to o ne side of the issue or the other may ultimately rely on arguments based on emotion, fait h, politics, and beliefs rather than hard data.Final ObservationsAs highlighted earlier, the four major findings of this research are similar to findings of previous surveys oriented to the general public. Fi rst, there is little support for vouchers as a systemic reform strategy. While many legislato rs surveyed supported the concept of vouchers in some form, the large majority opposed t he concept of unlimited voucher programs. Additionally, when offered a variety of r eform options, legislators tended to have more confidence in reforms designed to work fr om within the current system (teacher education, early childhood education, and phonics based curricular reform) rather than reforms designed to fundamentally chang e the form of the current system (such as vouchers, charter schools, and even open e nrollment). Second, despite such dubious support for vouchers w hen compared with other reforms, policy makers responding to this survey generally b elieved in the concept of market reforms and the power of competition to act as a po sitive force in education. At the same time, they did not buy into arguments that the mark et philosophy could hurt education. While this may seem a dichotomy, it may not be. Leg islators in three of the six study states (Florida, Ohio, and Michigan) are term limit ed, meaning that they have most likely only legislated during strong and expanding economi es. It will be interesting to observe whether support for market reforms remains strong d uring times of recession. Third, there continues to be an ill-defined line be tween the positive effects of schools meeting constituent needs, and effectively resultin g in the creation of a system where participants can segregate themselves by race, reli gion or other values. It is difficult to predict the effect of a voucher program on integrat ion because the cities in which vouchers are currently used, Milwaukee and Clevelan d, are already majority minority systems. Research from the effects of charter schoo ls on integration is also mixed, with some studies vulnerable to the same claims of bias as are present in the debate over voucher effectiveness. Various studies (Cobb & Glas s, 1999) indicate a tendency toward segregation, while others (Center for Education Ref orm, 2000c) show charter schools acting as an integrating force.
24 of 39Finally, while party affiliation remains a strong d eterminant of voucher support or opposition, the majority of undecided policy makers were Democrats. While this could signal a place where voucher proponents could make significant inroads, it seems unlikely that in the long term such Democratic supp ort would result in a broadly accessible voucher system. When Democrats did suppo rt vouchers, support was primarily for vouchers directed toward poor families and stud ents in failing schools. While this tendency could result in new programs, it seems unl ikely given the opposition to unlimited programs, that vouchers would expand beyo nd the narrowly tailored plans currently available. Florida's program, however, co uld prove to be the linchpin in expanding voucher programs. The voucher program for special education students is already showing tremendous expansion, and, should t he program emerge from the courts entac, it seems possible that the voucher program f or students in poor performing schools will expand significantly as the state's accountabi lity program becomes fully enacted.NotesTo see the text of each item discussed in this sect ion, see the survey in Appendix 2. 1. The dissertations reviewed involved surveys adminis tered to legislators using a variety of methodologies and across many topics. No surveys were found detailing efforts at cross-state surveying of legislators, so that all rates cited are relevant to policymakers surveyed by a researcher at an institu tion within their states. The review was not exhaustive, and conducted to generat e response rate figures for discussion within the context of this section only. 2.AcknowledgmentThe work presented in this paper was taken from res earch completed for the degree of Ph.D. awarded in the Fall semester, 2001 from Ameri can University, Washington, D.C. The opinions expressed in the article are those of the author and not any institutions and not of any institution with which he is affiliated.ReferencesAmerican Statistical Association. (1998). More about mail surveys ASA Series: What is a survey. Alexandria, VA: Author. Cobb, C. D. & Glass, G. V. (1999). Ethnic segregati on in Arizona charter schools. Education Policy Analysis Archives (7) 1. January 14, 1999. Retrieved October 21, 2002 from http://epaa.asu.edu/epaa/v7n1/ Center for Education Reform (2000). What the research reveals about charter schools. Washington, DC: Author. Retrieved October 23, 2002 from http://www.edreform.com/pubs/charters.htm Coons, J. & Sugarman, S. (1999). Making school choice work for all families San Francisco, CA: Pacific Research Institute. [On-line ] Available: http://www.pacificresearch.org/edu/choice/choice.ht ml (October 14, 1999). Cremin, L. A. (1964). The transformation of the school: Progressivism in American
25 of 39education 1876 1957 New York: Vintage Books. Creswell, J. W. (1994) Research design: Qualitative and quantitative appro aches Sage Publications, Thousand Oaks, CA. Devore, J. & Peck, R. (1997). Statistics: The exploration and analysis of data (3rd ed.). Pacific Grove, CA: Duxbury Press. Fox, R. J., Crask, M. R., & Kim, J. (1988). Mail su rvey response rate: A meta-analysis of selected techniques for inducing response. Public Opinion Quarterly 467 491. Gehring, J. (2002). Voucher battles head to state c apitals. Education Week 21 (42) 1,24,25. Ruth D. Holmes et al., v. John Ellis "Jeb" Bush et al. CV 99-3370 (Leon County Circuit Court, 2002). Retrieved August 13, 2002 from http://22.214.171.124/courtDockets/pdf/CV993370/99-33 70-51.pdf Institute for Justice (2002). Advancing school choice. Retrieved October 21, 2002 from http://www.ij.org/cases/school/ Jones, W. H. (1979). Generalizing mail survey induc ement methods: Population interactions with anonymity and sponsorship. Public Opinion Quarterly pp. 102 111. Majchrzak, A. (1984). Methods for policy research. Applied social researc h methods Series (Vol. 3) Newbury Park, CA: Sage Publications. Martella, R. C., Nelson, R. & Marchand-Martella, N. E. (1999). Research methods: Learning to become a critical research consumer Needham Heights, MA: Allyn & Bacon. Neuman, W. L. (1997). Social Research Methods: Qual itative and Quantitative Approaches. Needham Heights, MA: Allyn and Bacon. p 243. Pritchard, I. (1999). Reducing class size: What do we know? Washington, DC: National Institute of Student Achievement, Curriculum and As sessment: Office of Education Research and Improvement, U. S. Department of Educa tion. Retrieved October 23, 2002 from http://www.ed.gov/pubs/ReducingClass/ Ravitch, D. (1983). The troubled crusade: American education 1945 198 0 NY: Harper Collins. Rose, L. C. & Gallup, A. M. (2000). The 32nd annual Phi Delta Kappa/Gallup poll of the public's attitudes toward the public schools. Bloomington, IN: Phi Delta Kappa International. Sanders, W. L. & Rivers, J. C. (1996). Cumulative and residual effects of teachers on future student academic achievement. Knoxville: University of Tennessee Value-Added Research and Assessment Center. Shokraii-Rees, N (2000). School choice: What's happening in the states. Washington,
26 of 39DC: The Heritage Foundation. Tedin K. L., & Hofstetter, C. R. (1982). The effect of cost and importance factors on the return rate for single and multiple mailings. Public Opinion Quarterly pp. 122 128. Toppo, G. (2002, Aug. 6). Voucher backers put forth bills. Associated Press Tyack, D. B. (1996). The one best system Cambridge, MA: Harvard University Press. Yammarino, F. J., Skinner, S. J., & Childers, T. L. (1991) Understanding mail survey response behavior: A meta-analysis. Public Opinion Quarterly 613 639. Zelman, Superintendent of Public Instruction of Ohi o, et al. v. Simmons-Harris et al. 536 U. S. 00-1751 (2002). Retrieved August 13, 2002 fr om http://www.supremecourtus.gov/opinions/01pdf/00-175 1.pdf .About the AuthorDan LaitschSenior Policy AnalystAssociation for Supervision and Curriculum Developm ent 1703 Beauregard St.Alexandria, VA 22311-1714(703) 575-5628 email@example.com Dan Laitsch is Senior Policy Analyst for the Association for S upervision and Curriculum Development. He was previously Associate Director of the State Issues Clearinghouse at the American Association of Colleg es for Teacher Education, and has taught high school English at a number of alternati ve schools in Fairfax County, including six years with the Enterprise School in V ienna, VA. He has co-taught middle school English classes in Japan as part of the Japa n Exchange Teacher Program. He holds BA in English from Virginia Polytechnic Institute a nd State University, an M.Ed. in Social Foundations of Education from the University of Virginia, and a Ph.D. in Education Policy from American University in Washin gton, DC. This paper is based on the work of his dissertation research. Laitsch's cu rrent interests include teacher quality; research use in, and impact on, education policy; s chool effectiveness; and public school reform issues.Appendix I: MethodologyOverviewIn choosing an approach to take to answer the ident ified research questions, both quantitative and qualitative approaches were examin ed. While the history of voucher programs in Ohio, Florida, and Wisconsin is both im portant, worthy of study, and could have contributed to identifying arguments used effe ctively in those states, the ability to examine policy maker reactions to current issues su rrounding voucher proposals, both in states with such programs, and in states where the programs have been a hotly debated issue, presented a unique research opportunity. For this reason, a quantitative
27 of 39methodology that allowed for wide spread applicatio n and comparative analysis, was sought. A survey of legislators was chosen as the b est methodology for answering the research questions across a wide geographic range a nd large body of potential data sources.Site SelectionIn choosing the states to examine in this study, tw o related techniques were used: critical case sampling, and politically important case sampl ing. Sampling is often done randomly, but can also be collected purposefully; t hat is with the goal of selecting cases rich in information critical to the research questi ons. The focal points of this study (Florida, Michigan, New Mexico, Ohio, Pennsylvania, and Wisconsin) were chosen using both critical case sampling and politically importa nt case sampling. Critical case sampling is used to select cases that are of intere st for a particular reason while politically important cases are selected because either the sit e or the topic (or both) is of political importance (Martella, Nelson & Marchand-Martell, 19 99). Specifically, Ohio and Wisconsin were determined to be critical because th ey are the only two settings with established voucher programs. Florida was chosen be cause it is the first state to enact a state-wide voucher program, while Michigan, New Mex ico, and Pennsylvania, were all considered for inclusion in the sample because of t he prominence of the voucher debate within those states (voters in Michigan recently co nsidered vouchers through a ballot initiative, while the governors in New Mexico and P ennsylvania have strongly advocated for voucher systems in their states).Data SourcesIn this study, the primary methods for collection o f data were through administration of a researcher prepared survey (using a self-administer ed survey) as well as a review of relevant documents. Using these techniques data rel ated to vouchers and education reform was collected from policy makers, while hist orical data was collected to help provide context. These methods for the collection o f data are consistent with survey research, as well as historical research designs an d policy analysis (Majchrzak, 1984; Martella, et al., 1999; Neuman, 1997). An added ben efit of the historical data collection was the ability to partly synthesize the data as th e study evolved, allowing for more focused research as information was processed. Beca use of the ability to synthesize data in this manner, the focus of the survey was expande d to include components designed to evaluate policy maker opinions regarding the purpos e of education as well as the role of vouchers within the larger context of reform.Policy ParticipantsExpert observers active in the voucher debate were contacted both to help establish the policy atmosphere in which vouchers were enacted, a nd to help validate the survey. The people contacted included university scholars; memb ers of national think-tanks, education organizations and associations; and membe rs of unions and other advocacy groups. Individuals were contacted at pro-voucher g roups including the Center for Education Reform and the Institute for Justice; gro ups opposed to voucher programs such as the American Association of School Administrator s, and the National School Boards Association; and neutral groups, including the Nati onal Conference of State Legislatures,
28 of 39the American Association of Colleges for Teacher Ed ucation, and the Center for Education Policy Faculty at American University a lso assisted in construction and validation of the survey.SurveyAs the primary data collection tool for this resear ch, great care was taken in development of the survey instrument, however, some difficultie s arose during the application procedure.Surveys are typically used to identify how people f eel, think, act, or vote (Martella, et al., 1999), which fits well with research designed to ga uge how policy makers react (feel) to specific proand antivoucher statements. The adv antages attributed to survey research include comparative cost effectiveness, the ability to target specific populations, and the tendency of respondents to feel their confidentiali ty is protected (American Statistical Association, 1998). Although associated difficulti es include deciding contact methods, controlling costs, setting deadlines, protecting fa irness, and performing proper follow-up, a survey methodology was determined to be the best way to obtain the data needed to answer the research questions.Items and ContentBecause the debate about vouchers is often one pola rized to extremes, a methodology that would force respondents away from the extremes was developed. The survey was divided into three sections. The first section exam ined school reform issues. The second looked at the voucher statements, and the third sec tion was concerned with demographic data.Section I: Reform issuesThe issue of vouchers cannot easily be separated fr om the concept of education reform and the general role of education. In addition, the perception of the success of our current system in meeting that role needed to be examined. Section I of the survey briefly examined all three of these issues.Because the need for reform is based on the perceiv ed effectiveness of the education system, that is, an effective system would not need reform while an ineffective system may need radical reform, legislators were asked to evaluate the current system's success. Before such an evaluation could take place, however "what" is being evaluated needed to be defined. To accomplish this, legislators were be asked to rank order six possible "purposes" of education, along with the option to a dd a seventh purpose if they desired. Next, respondents were asked to evaluate the degree of effectiveness of the current system, both locally and nationally in addressing t he purpose they previously identified. The next item asked policy makers to indicate their level of support for 11 specific reforms, vouchers included.The final question in this section looked at suppor t for vouchers on the continuum of voucher programs; from an unrestricted voucher syst em, to systems regulated by income or school effectiveness, to no voucher system at al l.
29 of 39Section II: Voucher positionsSection II asked respondents to rank order proand antivoucher statements. The statements were divided along three tension lines f ound within the education system: excellence, access and equity, and structure. These statements were designed to evaluate the type of impact respondents felt vouchers would have on the current education system. Section III: Demographic dataDemographic data that has been previously related t o support or opposition to voucher programs in other studies (race, religion, gender a nd educational experiences) was included. Where possible the items in this section were modeled on other demographic surveys and data sources.DevelopmentThe survey was constructed in four phases. In the i nitial phase, a draft instrument was constructed by reviewing the research and advocacy literature related to voucher programs, as well as literature related to survey c onstruction. This initial survey contained many of the policy arguments included in the final survey (as well as the demographic questions), however the rating scale us ed was a Likert scale. It was in this phase that the primary policy arguments were identi fied and the arrangement of item groupings first noticed.Use of the Likert scale was dropped for the policy arguments during phase two primarily to avoid any response set bias (Neuman, 1997). It w as also felt that due to the extreme positions held by many individuals, the Likert scal e might not elicit the kind of data that would allow for detailed differentiation between it em responses. In other words, people already holding a position would recognize the vari ous proand antivoucher statements and "strongly agree" with all items interpreted as supporting their position, while "strongly disagreeing" with all items interpreted a s being against their beliefs. In such a situation, the middle ground, or swing arguments, w ould be lost. The survey was instead reorganized to encourage differentiation between it ems by asking respondents to rank-order related items. In this way, the stronges t and weakest arguments for each grouping could be analyzed. Since the arguments wer e organized into proand antivoucher sections of the survey, analysis of argumen ts with cross-position appeal could also be made. It was during this phase that similar subgroups of the policy arguments were identified within each position. The subgroups are: access, equity, and educational structures. In other words, the policy arguments co uld be aligned by evaluating the impact of vouchers (as perceived by policy makers) on: access to education; equity of educational opportunity; and their potential impact on current educational structures. Since the arguments were already separated into pro and antivoucher groupings, the result was six final categories, allowing for the e valuation of up to 36 policy arguments (33 were ultimately identified).In the third phase, the scope of the survey was exp anded to include items designed to gauge policy maker reaction to other reform options as well as the purpose and effectiveness of public education. This was done to help provide some context to the data
30 of 39interpretations. While identifying which arguments seem most effective could prove useful in a limited way, placing those conclusions within the broader context of reform and educational philosophy provided a richer unders tanding of the data. A Likert scale was used in this part of the survey, since policy m akers where being asked to guage the degree of effectiveness of the schools generally, a nd various reforms specifically. A rank order response was used for the section related to the purpose of education. Finally, the survey was sent to experts in the fiel d of vouchers and school reform to assist with instrument validation. Despite the positive fe edback from evaluators, difficulties experienced during this phase of development signif icantly effected the process involved in finalizing the survey. While the general instrum ent was completed and disseminated to reviewers in early March of 2000, some reviewers to ok up to five months to review the document. This significant delay in the review proc ess ultimately precluded a formal pilot test.The final step in the typical development of a surv ey is the pilot-test. This survey was not formally pilot-tested for two critical reasons. Fir st, a representative target audience for a pilot test was not readily available. While a legis lative body in a local state could have served as the target population, gaining detailed f eedback from the audience regarding the survey would have been difficult and time consu ming (and such information had already been obtained through the expert reviewers, who were asked to first take the survey themselves and then provide feedback as to t he validity of the survey content, application and structure).A second, and more perplexing difficulty with compl eting a pilot-test arose from the delay caused by the unexpectedly long review proces s. The addresses and electronic contact information for the sample to be surveyed w ere provided by StateNet, an organization that supplies legislative research res ources to private clients. Due to the 2000 elections, this source data was only accurate up until November 8 (election day). Contact with legislators after that deadline would be effected by changes in the sample brought about by the elections (this was the case w ith a second follow-up e-mail sent after the election day). An extensive pilot-test wa s therefore precluded because of the need to apply the survey before the target populati on changed. ApplicationBecause the target audience for the survey was defi ned as legislators in the selected study states, the entire population was available to be s urveyed. While a random, or a stratified sample of legislators was considered, the potential benefits of targeting the entire population were viewed as outweighing the utility o f a smaller sample. The total number of participants surveyed was 936.The survey was applied according to the recommendat ions of the American Statistical Association (1998), which parallel other applicatio n recommendations (Creswell, 1994). A cover letter was included with the survey, as wel l as a pre-addressed stamped return envelope. Follow-up contact was attempted two weeks later, although the follow up procedure differed from the initial contact in that it focused on electronic contact. E-mail messages were sent to each legislator available, to taling 780 (approximately 715 were successfully delivered). The follow-up message aske d legislators to return the hard copy sent to them, or fill in an on-line version of the instrument.
31 of 39Two areas where the application procedure deviated from ASA recommendations include a failure to send a pre-contact notice and failure to offer an incentive for completion of the survey. These guidelines were not followed sinc e legislative offices are generally very careful with the mail they receive, so that while a survey may not have be returned, it was felt that it most likely would be opened and review ed. An incentive was viewed as inappropriate for this audience and for this type o f research. ValidityThere are three types of validity typically used to establish the appropriateness of a measurement device: construct, content, and criteri on. Education activists and professionals were contacted to help establish the face validity of the survey (content validity). While face validity is not the strongest tool for establishing validity, it was one of the only tools available in this instance. Becau se this was the first administration of this device, construct validity could not be establ ished (Martella, et al., 1999; Neuman, 1997). In addition, while individual items were con structed based on other measures of support for vouchers, no similar measure existed fo r the device as a whole (criterion validity).Data AnalysisThe data obtained from the survey was analyzed indi vidually by item and section, as well as summatively across sections and items. In additi on to basic analyses (mean, median, mode, standard deviation, and variance, as appropri ate), a correlational analysis was conducted examining the relationships between demog raphic variables and each section independently, as well as the relationships between support for or opposition to vouchers and policy maker opinion of school purpose and effe ctiveness. A comparison of reactions to policy statements in states with voucher program s and states without such programs, was also conducted.The data analysis did not attempt to establish any causal relationships since survey data cannot generally be used in that manner (Martella, et al., 1999). In addition, because the return rate was relatively small, the available dat a was limited. Therefore, correlational data involving demographic characteristics was not attempted at the state and local levels. A final impact on the complexity of the dat a analysis stems from the nature of the data, which in much of the survey is categorical (q ualitative). While categorical data does not preclude analysis of the data using a multiple regression model, it does significantly expand the predictors used in each model and the mo del complexity (Devore & Peck, 1997).Data CollectionOn October 18, 2000, the survey was mailed to 936 s tate legislators in Florida, Michigan, New Mexico, Ohio, Pennsylvania, and Wisconsin. Poli cy makers were asked to respond by October 27, so that follow-up contact could be m ade before the November 8 elections. A follow-up contact was made via e-mail on November 5 asking policy makers to return the survey or complete an on-line version. Finally, on December 12, the last reminder e-mail was sent notifying legislators that the surv ey would remain on-line until the end of
32 of 39 January, 2001.Survey Completion and Non-completionOf the 936 surveys mailed, 89 were returned, for an overall return rate of 9.5%. Of the items returned, 20 included sections only partially completed, and four were filled out incorrectly and therefore excluded from analysis. W hile the majority of returns were through the mail, 14 surveys were returned on-line, accounting for 16% of all surveys returned.Completion Rate FiguresThe completion rate of viable surveys was 9.21% (se e Table 1). Twenty incomplete surveys were included in the return estimate becaus e they were completed correctly and contained useful data for most sections of the surv ey. Those items that were left blank or were completed incorrectly were not included in any analyses that examined the compromised sections. In all cases included in the estimate, Section 1 was completed correctly, while some incomplete data was present i n Section 2. Demographic data was also completed accurately by all but two respondent s. Items determined as "unusable" were those that were filled out incorrectly through out, or had too little data to be included in any of the analyses (one survey had only a note at the top "I oppose all voucher plans"). Table A1: Survey return ratesSource:SentWebMailTotalRatePartialUnusableUndeliver edRate Florida159110116.9%2037.1%Michigan148113149.5%2228.2%New Mexico1114192320.7%80120.9%Ohio1322575.3%1045.5%Pennsylvania254313166.3%3115.9%Wisconsin1321896.8%2126.2%Unknown1n/a279n/a201n/a Totals9361475899.5%204149.2%1Control labels were removed or the web control numb er not entered correctly. Sampling ErrorBy attempting to survey every legislator in the stu dy states and maintain the focus of any conclusions on those states, it was hoped that the potential for sampling error could be minimized, however, because of the low response rat e, a significant potential for error exists. In addition, while the demographic traits o f respondents are similar to the body of legislative policy makers in each state, some trait s were clearly skewed (for example, two thirds of respondents said that they have served on the committee responsible for education). While these legislators clearly have an interest in education, service on the education committee is not a common characteristic of the broader body of legislators. It can be argued, however, that data gathered from leg islators with experience on the education committee lends to the significance of th e survey results by providing insight
33 of 39into the responses of legislators who are or have b een in key leadership positions on education issues.In addition to the returns, 17 letters or e-mails w ere received from legislative offices in three of the study states. Many of the letters expl ained that the legislator had a blanket policy of not returning surveys because of the larg e number of such contacts they receive. One legislative office offered the opportunity for a face-to-face interview; however, the legislator was not interested in completing a phone interview or mail survey. Similar automatic responses to the e-mail contacts suggest that legislators in general were not disposed to respond to surveys from non-constituent s. Other replies referred to problems with the timing of the survey. One response noted: "It's the day before electiion [sic] day. All I can tell you is that I'm vehemently opposed to vouchers. I'm a Republican" (personal communicatio n, November 7, 2000), while another stated, "Take me off of your list. I have b een term limited out and yesterday was the last" (personal communication, December 14, 200 0). Also complicating survey returns was the electoral controversy in Florida. M ost e-mails sent to the legislators in Florida were met with the following auto-response: "Thank you for contacting me about Florida's recount effort. Because I have received thousands of calls, letters and e-mails it is difficult to respond to each communication indiv idually." One return from a legislator in Pennsylvania manage d to capture the controversy in that state: Why would you presume that every legislator in six states is interested in your research, especially when it requires them to answer several pages of inane questions about a subject which most of them would like to forget . Further an enterprising doctoral student might figu re out that there are a variety of groups on both sides of the issue that a sk candidates to respond to questionnaires during the heat of the campaign whic h might prove far more interesting than some academic exercise" (personal communication, December 14, 2000). Data AnalysisIn evaluating the results of this survey, a brief d iscussion of the scale used is important. Most non-demographic items were scored from 1 6 o n either a Likert scale or as a rank order. In either case, the highest rating an item c ould receive was a "1", while a "6" was generally the lowest rating possible. The mean for each item was calculated and used to help determine the favorableness of each item. Whil e by itself an average of 2.1 wouldn't tell us anything, especially for the rank order ite ms (i.e.., there is no rank of 2.1), it does give us a sense as to how positive an individual it em is seen to be. The variance of each item was also calculated so that a rough picture of the spread of answers within each item could be constructed. Where required for analysis, a .05 level of significance was used.Appendix 2: Survey application and return rateSurvey application and designTo understand the low response rate to this survey, the literature around survey responses
34 of 39was revisited. While the research reviewed supporte d the summary research presented earlier, examination into the theory behind why spe cific techniques appear to improve return rates, and consideration of the impact such theory might have on policymakers suggests some possible explanations for the low ret urn rate on this survey. Because this research was not structured to specifically examine responses rate effects, conclusions reached in this section should be viewed as hypothe tical. One critical gap noted in the research was an exami nation of the response rates of policymakers to surveys. Much of the survey researc h reviewed appears to have been conducted by researchers interested in marketing an d surveys applied to the general public. No research could be found that dealt speci fically with the impact of response rate strategies on policymaker return rates. Consequentl y, in the design of this research it was assumed that strategies successful in increasing th e response rate of the general population would also be successful with policymake rs. However, when looking at the theory behind the specific strategies, weaknesses i n the potential impact of their effect on policymakers become apparent. While this is not to say that the return rate strategies used had no impact, it is possible to see how such impac t might be minimized. Research into survey response rates has generated a number of application strategies that seem to be successful across the board (American St atistical Association, 1998; Creswell, 1994; Fox, Crask, & Kim, 1988; Jones, 1979; Tedin & Hofstetter 1982; Yammarino, Skinner & Childers, 1991). These include the use of first class postage, pre-contacts, follow-up contacts, personalized addresses and salu tations, use of cover letters, signed cover letters, stamped return envelops; guarantees of anonymity; and monetary incentives. The effect of non-application manipulat ions are less clear, but include survey length and complexity, page color, topic importance sponsorship, and type of population. Of these topics, sponsorship, especially sponsorshi p by a university, has been found to effectively increase response rates, as has the imp ortance of the topic. While much of the research examined focused on the effects of the intervention, Tedin and Hofstetter also considered some of the theory b ehind these interventions (1982). They point out that interventions the investigator generally can control relate to survey importance factors and cost factors. In effect, int erventions seen by respondents as emphasizing the importance of their responses (repe ated contact, personal letters, first-class postage, monetary incentives, etc.), an d interventions lowering participation "costs" (time and effort it takes to respond leng th of questionnaire, enclosed and stamped return envelop, etc.), generally increase t he response rates. The response rate interventions used on this survey include the following: inclusion of a cover letter (including a personalized address and salutation and personalized signature), inclusion of a stamped return envelop, use of a sho rt questionnaire (four pages), multiple contacts (in paper and electronic formats), conside ration of topic importance, and a guarantee of anonymity. While such interventions ma y reduce costs and increase the perceived importance of responding for members of t he general public, when examining the way in which legislative offices operate, such considerations may be less likely to have a strong impact.Cost effectsOf the interventions identified above, two can be c ategorized as efforts to reduce
35 of 39respondent costs: inclusion of a stamped return env elop, and use of a short questionnaire. While the impact of a short questionnaire on respon ses rates has been difficult to determine (Fox, Crask, & Kim 1988; Tedin & Hofstett er, 1982), generally the use of first class postage has been shown to increase response r ates; however, the volume of contacts legislators receive requires legislative offices to utilizing procedures which may mitigate this effect. For example, since policymakers receiv e a variety of postal benefits (franking privileges), the monetary cost saving of a stamp an d envelop are negligible. In addition, because legislative offices are organized to respon d to constituent contacts, the cost reduction in the effort to return the survey offere d by inclusion of a stamped envelope may also be reduced.ImportanceThe other four interventionsuse of personalized a ddresses and cover letters, multiple contacts (in paper and electronic formats), conside ration of topic importance, and a guarantee of anonymity-can be categorized as effect ing the respondents perception of survey importance (the perception of respondents th at it is important that the survey be returned). Personalization emphasizes that the indi vidual's opinion is of particular importance, as does the use of multiple contacts. T he topic itself, education reform, was also framed so as to emphasize the importance of th e survey. Finally, the guarantee of anonymity adds to perceptions of importance by reco gnizing the issue as controversial and one in which an open, honest evaluation, free o f constraints (in this case those of political party) is critical. Although each of thes e interventions is validated in survey research, the special circumstances involved in con tacting legislators may reduce their impact on return rates.While it may be unusual for a random sample of the general public to receive a personalized survey (thereby emphasizing its import ance), legislators expect such personalized contact as routine. Contact advice pos ted on at least one of the state web sites confirmed this expectation. Perhaps even more problematic is the expectation that such contacts originate from or impact upon the leg islator's constituents. The automatic responses received to the e-mail contacts highlight ed this point, some specifically stating that the legislator only reviewed in-district e-mai l. This reluctance to work with individuals who the policy maker does not represent and who are not at least partly responsible for his or her election, may have kept response rates low, despite the interventions used.Additionally, because of the large number of contac ts policymakers receive, many offices have constructed pre-formatted replies related to s pecific issues. Although such replies are not directly applicable in this case (other tha n those stating that the policy maker does not respond to any surveys), it suggests that while a policymaker's response is important, individualization (such as required by surveys) is not. In the application of this survey, no steps were taken to associate the survey with the p olicymaker's constituent groups. In fact, neither the researcher, nor the researcher's institution were located within any of the study states.Finally the timing of the survey application may ha ve worked to minimize the attention paid to it. For reasons discussed earlier, the surv ey was administered shortly before the November 2000 elections, and follow-up contact was made during the post-election controversy over election returns in many states, m ost notably Florida. Because of the
36 of 39timing then, the survey was competing for the atten tion of policymakers with both election campaigning and post-election controversie s. As one politician, cited earlier, noted: .... there are a variety of groups on both sides of the issue that ask candidates to respond to questionnaires during the heat of the campaign which might prove far more interesting than some academic exerc ise" (personal communication, December 14, 2000). Two other previously noted comments also highlight the poor timing of the survey: "It's the day before electiion [ sic ] day. All I can tell you is that I'm vehemently o pposed to vouchers. I'm a Republican" (personal communicatio n, November 7, 2000); and, "Take me off of your list. I have been term limited out a nd yesterday was the last" (personal communication, December 14, 2000). The e-mail respo nses from Florida also highlight the lack of immediate importance the survey may hav e had for many legislators ("Thank you for contacting me about Florida's recount effor t. Because I have received thousands of calls, letters and e-mails it is difficult to re spond to each communication individually."). In light of the 2000 election cont roversies, the survey completion rate may have been negatively impacted.Population identification and research instrumentat ion Changes in the methodology used to identify the tar get audience and application procedure used for this instrument might have resul ted in a higher rate of return. For example, a smaller, stratified sample, would have p ermitted more focused efforts to obtain responses. A smaller sample drawn from the s ame states (or even fewer states), could have enabled the use of letter and phone preand postcontacts, and might even have allowed the survey to be administered in an in terview format. Additionally, use of a stratified sample would allow use of a pre-contact strategy designed to elicit a commitment to participation by individual policymak ers. Such a commitment could then allow the researcher to adjust of the actual survey application as a projected response rate is generated. These extensive efforts may then have been seen by policymakers to stress the importance of participation in the survey, incr easing response rates. While such efforts may well have increased the rate of return, manipulations of the sample might also have concealed some findings that would only emerge in a survey of the entire population (for example, the high intere st level of education committee members as evidenced by their high rate of return), and greatly increasing the complexity of survey design, application, and analysis. Despit e such complexities, however, increasing response rates in this case would have s trengthened the research findings. Recommendations for future researchBecause policymakers operate under different expect ations regarding mail contacts than the general public, it is possible that techniques traditionally effective in increasing survey response rates are not as effective with pol icymakers. In addition, because of situational complexities (most notably the applicat ion of the survey during the closing days of a general election), events external to the research design may have lowered response rates. There are, however, additional step s that might be taken to increase response rates of future surveys. These recommendat ions serve to emphasize the
37 of 39 importance of the survey to policymakers. Since sponsored surveys often receive a higher resp onse rate, use of university letterhead on the cover letter may generate a highe r rate of return. It should be noted that the National Council of State Legislatur es was asked to sponsor this survey but due to the controversial nature of the t opic they declined. 1. Because policymakers are more likely to consider th e concerns of their constituents, future survey efforts might benefit f rom the use of proxy administrators located in each state or within inst itutions in each state. By seeking to generate the data from within each legislator's home state, greater attention may be attached to completion of the survey. 2. The survey should be administered just after the en d of each state's legislative session, when legislators do not face as many deman ds on their time. 3. The survey should not be administered during an ele ction year, or, if timing demands such application, the survey should be admi nistered prior to election day and preferably before the summer campaign season. 4. A procedure should be established to allow pre-cont act, as well as extensive post contact. A useful procedure might include a pre-con tact letter of introduction, followed by a pre-contact phone call to attempt to get buy in by the legislator. After application of the initial survey (either in mail, interview, or Internet format), a post contact follow-up should be made either to tha nk the legislator for their cooperation, or to encourage those legislators not responding to complete the survey. 5. Consideration of the survey format could also be va luable. Changing from a mail-administered questionnaire to an interview for mat could greatly increase response rates, since mail surveys are used specifi cally to target a wide audience. Narrowing the sample through a change from question naire format to an interview format be worth the loss of breadth if, by allowin g targeted and frequent pre-contact and follow up, it raises the response r ate. 6. If expense precludes the use of these strategies fo r multi-state or state-wide research, then researchers may want to consider usi ng sampling techniques to narrow the population to be surveyed. Of the sampli ng choices available, stratified sampling could field a list of legislators within e ach strata who might be contacted until sufficient numbers for detailed analysis are generated. Additionally, because the data generated will be categorical in nature, s tratified sampling could ensure that there will be enough legislators in each area of interest identified, so that the appropriate statistical examinations can be complet ed. 7. While these seven recommendations may not guarantee a higher rate of return, it is unlikely, given the low rate of return on this surv ey, that they would negatively effect return rates. It also seems likely that a survey ad ministered under similar circumstances will result in a low rate of return.Copyright 2002 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, firstname.lastname@example.org or reach him at College
38 of 39of Education, Arizona State University, Tempe, AZ 8 5287-2411. The Commentary Editor is Casey D. Cobb: email@example.com .EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov firstname.lastname@example.org Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute ofTechnology Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Education Commission of the States William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX) Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton email@example.com Hugh G. Petrie SUNY Buffalo Richard C. Richardson New York University Anthony G. Rud Jr. Purdue University Dennis Sayers California State UniversityÂ—Stanislaus Jay D. Scribner University of Texas at Austin Michael Scriven firstname.lastname@example.org Robert E. Stake University of IllinoisÂ—UC Robert Stonehill U.S. Department of Education David D. Williams Brigham Young University EPAA Spanish Language Editorial BoardAssociate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico email@example.com
39 of 39 Adrin Acosta (Mxico) Universidad de Guadalajaraadrianacosta@compuserve.com J. Flix Angulo Rasco (Spain) Universidad de Cdizfelix.firstname.lastname@example.org Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho dis1.cide.mx Alejandro Canales (Mxico) Universidad Nacional Autnoma deMxicocanalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State Universitycasanova@asu.edu Jos Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es Erwin Epstein (U.S.A.) Loyola University of ChicagoEepstein@luc.edu Josu Gonzlez (U.S.A.) Arizona State Universityjosue@asu.edu Rollin Kent (Mxico)Departamento de InvestigacinEducativa-DIE/CINVESTAVrkent@gemtel.com.mx email@example.com Mara Beatriz Luce (Brazil)Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.brJavier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mxMarcela Mollis (Argentina)Universidad de Buenos Airesmmollis@filo.uba.ar Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mxAngel Ignacio Prez Gmez (Spain)Universidad de Mlagaaiperez@uma.es Daniel Schugurensky (Argentina-Canad)OISE/UT, Canadadschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil)American Institutes for ResesarchÂ–Brazil(AIRBrasil) firstname.lastname@example.org Jurjo Torres Santom (Spain)Universidad de A Coruajurjo@udc.es Carlos Alberto Torres (U.S.A.)University of California, Los Angelestorres@gseisucla.edu