|USFDC Home||| RSS|
This item is only available as the following downloads:
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 6issue 20series Year mods:caption 19981998Month November11Day 33mods:originInfo mods:dateIssued iso8601 1998-11-03
1 of 38 Education Policy Analysis Archives Volume 6 Number 20November 3, 1998ISSN 1068-2341 A peer-reviewed scholarly electronic journal. Editor: Gene V Glass Glass@ASU.EDU. College of Education Arizona State University,Tempe AZ 85287-2411 Copyright 1998, the EDUCATION POLICY ANALYSIS ARCHIVES.Permission is hereby granted to copy any a rticle provided that EDUCATION POLICY ANALYSIS ARCHIVES is credited and copies are not sold.
2 of 38Critical Evaluation for Education Reform Gisele A. Waters Auburn UniversityAbstract The school reform movement has done little to provide an accurate analysis of the production of inequality or the reproduction of soc ial injustice in the public schools or the larger social order. The ideology that influenc es this movement has often prevented the realization of any notion of an egalitarian ide al, the elimination of inequality, or the improvement of those who are least well-off. I ask educators and evaluators of education reform efforts to reconsider critically their roles in social science research, to reclaim the battleground of public school reform by focusing on the democratic purpose of public schooling, and the institutional problems in educat ional programs and practice that often inhibit action toward this ideal. The first part of this article includes an extensive argument explaining the "why" of critical evaluatio n. The theoretical literature on inquiry in science and social science, the ideology of critical theory, critical social psychology, and Freirean pedagogy are consulted as additional tools for augmenting the practice, policies, and responsibilities of evaluat ors in education. I review three contemporary perspectives of evaluation in order to begin rethinking the purposes and functions that evaluation serves in education. It a lso demonstrates how mainstream and contemporary evaluations can be used to serve a par ticular set of social and political values. The second part of this article begins a pr eliminary journey toward describing the "how" of critical evaluation. Critical evaluators c an fight for social justice by combining the merit criteria of state and federal public educ ation law, and the methods of an adversary oriented evaluation in order to transform educational environments that serve the future potentials of all children. Therefore ed ucation involves the practice of freedom, the means by which men and women deal crit ically and creatively with reality and discover how to participate in the transformati on of their world (Freire, 1985).The Argument for Critical Evaluation of Education R eform Part I: The "Why" of Critical Evaluation Schools are inextricably linked to the co mmunities they serve through social, political, economic, and cultural interests. To bet ter comprehend public education, the socio-cultural, political, and hierarchical relatio nships that transpire within the school as well as within the community must be linked to the broader political and economic issues of society at large (Ogbu, & Matute-Bianchi, in press). To begin to realize the possibility for reforming public education, and to begin fighting for social justice in education, especially for those children who are di sadvantaged, we must first re-examine the historical nature of the problems of education and the communities in which these schools exist (Noll, 1997).Education Reform
3 of 38The Conceptualization of Educational Evaluation as Practical Educational Research Many attempts have been made in recent ye ars to clarify the meaning of evaluation and expose the distinction between evalu ation and other related concepts such as measurement or research. The literature con tains many approaches regarding the conceptualization of evaluation and the determinati on of its countenance in education (Nevo, 1986). According to Nevo (1986), many of the se approaches have been unduly referred to as "models" (for example, the CIPP Mode l, the Discrepancy Model, the Responsive Model, or the Goal-Free Model) in spite of the fact that none of them includes a sufficient degree of complexity and comp leteness that might be suggested by the term "model." For the benefit of those of us wh o lost their way among the various models and approaches, I simply suggest taking a ho listic approach to considering educational evaluation as an extended arm of practi cal educational research. Education is a field like medicine in tha t its name simultaneously refers to a practice and to a field of disciplinary inquiry (Sc riven, 1986). Scriven stated that the paradigm of research in the area of the philosophy of education, to take one example, is surely the paradigm of philosophical research in an y area. But that leaves open the area of research that we normally think of as the domain of scientific research in medicine or education. Traditionally, we have tended to suppose that in this area of medical or educational research the correct model is that of t he related sciences. That is, for example, educational research has modeled itself on social science research. Similarly educational evaluation has modeled itself as an off spring of educational research. In medical research that approach has brought some pro blems because it seems to lead to results that conflict with the practical wisdom of physicians and the economic realities of the patients. The same can be seen in education wit h the refined development of IQ tests, normreferenced testing, and token economies for c lassroom management. Scriven (1986) wrote that the conventiona l "scientific paradigm" way of dealing with these type of problems is not the business of science, they are value issues, and must be sorted out by the citizenry. Instead, he pr oposes a paradigm for practical educational research which subsumes educational eva luation, and which includes ethics, political feasibility, a set of practical alternati ves, and an overall practical significance. Educational research is not, as he is suggesting, t o be defined as all research that in any way involves the concepts related to education, bec ause that's too broad (it includes learning theory), but as research that contributes to the facilitation of education, just as medical research should not be defined as all resea rch that involves concepts related to medicine, since that brings in all physiological re search, but simply as research contributing to health. The research on classroom teaching, educa tional programming, school management, and classroom achievement have mostly b een designed on the "quest for knowledge" model (traditional scientific) rather th an on the "improvement of practice" model. Scriven's main point for educational evaluat ion stresses the acknowledgment that evaluation research in schools can be a far more co mplex business than just a quest for knowledge, a quest for classification, explanation, generalization, causation, and/or prediction. In reviewing some of the theoretical lite rature of inquiry in science and the social sciences, it is hard to avoid the impression that t here is a reluctance to confront the issues of power, democracy, inequality, ethics, pol itics, and pragmatics in educational research, evaluation, and in mainstream social scie nce. Scriven proposed that one cannot reconcile the widespread support for the doctrine o f a value-free social science with the
4 of 38continued, inescapable practice of evaluation by so cial scientists, of the work and worth of students, peers, and selves, except by invoking a kind of phobia which makes them blind to the contradiction between their doctrine a nd their practice. This phobia, Scriven called "value phobia," has blocked us for nearly a century from addressing explicitly the methodology of evaluation and the systematic evalua tion of our own practices in social science research (1986, p.62). With this in mind, I explain how the theoretical literature of inquiry in science and the social sciences can c ontribute to justifying the inclusion of such values as social justice within an expanded fr amework of critical evaluation of education reform.Consideration of Inquiry in Science and the Social Sciences Social Justice and the Distribution of Education Considerations of social justice are appl ied in the distribution of virtually every social good. This is so much the case that, in the eyes of some, social justice simply has to be proclaimed (for example in political programs ) to henceforth characterize the relations between people. In educational policy, ar guments derived from social justice played a role even before World War II and were fou ght over by political parties, teachers' unions, left-wing intellectuals, and "ped agogical entrepreneurs" (Wesselingh, 1997). For them, the phenomenon of unequal particip ation was indeed a social problem, a phenomenon of social injustice. It does not take much effort to see that predominantly economic considerations have prompted the rapid exp ansion of equalopportunities research. Opinions about the just provision of educ ational opportunities combined with economic need, have given the impetus to this resea rch (Wesselingh, 1997). Indeed, the Fall 1998 edition of Educatio nal Evaluation and Policy Analysis includes one of the latest studies completed using Hierarchical Linear Modeling (HLM) on the relationship between students' opportunity t o learn (OTL) and their science achievement. In this study, Jia Wang concludes that content exposure was the most significant predictor of students' written test sco res, and the quality of instructional delivery was the most significant predictor of the hands-on test scores. In support for these types of conclusions, Berliner and Biddle (19 97) clearly argue that opportunity to learn is the most significant predictor of academic achievement. These authors would be content to know that "scientific methods" such as H LM techniques are pushing the analysis of OTL variables at two level of instructi onal processes: the classroom level and the student level. On the cusp of a new millennium, we are s earching for answers not in the homes, economic backgrounds, and individual disadvantages of our students of public education, it would seem that we are finally beginn ing to look at the quality of instruction variables that exist in schooling proce sses instead of "blaming the victim." Can we begin to ask why and how our school systems are failing our children, instead of why and how these children are failing our school s ystems" If schools are to be held accountable for the equitable delivery of education al opportunities and if social justice is to take place within the halls of academic opportun ity, the core of the education performance indicator systems should include school and classroom information. According to Winfield (1993), there are t wo main reasons for obtaining OTL information. First and foremost, teacher and school factors need to be taken into consideration in explaining students' achievement. Teacher and school variables directly and indirectly influence student learning and stude nt performance. Second, the new
5 of 38performance-based assessments make the collection o f OTL information crucial (National Council on Education Standards and Testin g (NCEST), 1992). The performance-based assessments require higher order thinking skills. This may put students from low socio-economic status groups at a disadvantage. Studies have shown that minorities, especially African American and Hi spanics, are more likely to be put into classroom with less learning opportunity even when ability is taken into account (Gross, 1993). If future research on achievement co ntinues to disregard OTL variables, the achievement gap between majority and minority w ill continue to increase and a lack of educational opportunity will continue to expand (Arreaga-Mayer & Greenwood, 1986; Madaus, West, Harmon, Lomax, & Viator, 1992). Education as a good to be distributed gets the char acter of a good that provides access to other goods. The key power of sc hooling is based on the fact that education serves as a criterion for the d istribution of all kinds of other material and immaterial goods. The consequenc e of this development is an instrumentalization of education. It evolves into an outstanding example of an instrument of mobility in a society w here now qualification and rapidly growing demands for qualification creat e the space for moving up and, to a lesser extent, moving down the social ladder. This is at least the idea; the question of how education actually perfor ms or is able to perform its role as a social agency of distribution for var ious social groups is of course not answered (Vervoort, 1975, p. 104). For various groups this question still challenges o ur daily lives as critical evaluators, leaders and researchers of social justice in educat ion. As generally acknowledged since the tradi tional bourgeois ideas of the Enlightenment, the only valid criterion for determi ning who deserves which education is achievement. Achievement as a criterion for selecti on stems from egalitarian principles and is generally accepted in education as a just cr iterion. By now we know that this distribution model has let to serious forms of soci al inequality. The assumption that in schools everybody has equal opportunities to perfor m and thereby has a fair chance to take part in the subsequent competition on the labo r market, has proven to be a misconception (Wesselingh, 1997). Education thus fu nctions as an instrument for the reproduction of social inequality and thereby refle cts the irony of a principle derived from egalitarian Enlightenment philosophy.Social Justice and EducationWalzer's Spheres of Justice published in 1983, can be seen as a reaction to J ohn Rawls's A Theory of Justice published in 1971. Walzer's objective was to prov ide an interpretation of what we contemporary Americans se e as the essence of such concepts as equality and justice. His book makes clear that a discourse on the selection criteria for such an important social good as education is now n eeded more than ever. Reflection on this topic should not be left to politicians and po licy makers for in that case considerations outside the sphere of justice will t end to dominate. Educational scientists, sociologists of education, and educational evaluato rs in particular, should definitely be more concerned with issues of social justice in edu cation. Social justice is one of the most important values that we should hope to secure in critical evaluation studies of educational reform. One of the goals of this article, besides arguing that critical evaluation is needed in
6 of 38order to begin fighting for social justice in educa tion, is to recommend an open and purposeful discourse about social justice in the re form of American public education between the "public intellectuals" (Giroux, 1997, p .263), otherwise known here as the social scientists, educational researchers, evaluat ors, and practitioners, a discourse about social justice in the reform of American public edu cation. The participation in discourse that values a moral imperative and a political comm itment to social justice in the evaluation of education reform is crucial to unders tanding the ideology of a critical evaluator.Reaching Beyond the Incommensurable PerspectivesWhen it comes to dealing with such issues as social justice in education in a way that recognizes its moral complexity and political natur e, the social sciences have "incommensurable perspectives" based in various tra ditions which have had different ideas about the individual and his/her society (Wer tsch, 1998). These views have been updated but often at the cost of further fragmentat ion in the social sciences. The work of the new "public intellectuals" is to translate and connect, the ideologies and contributions of Aristotle's Realism, Plato's Ideal ism, Comtean Positivism, the Vienna Circle of Scholars and their Logical Positivism, Co nstructivism, Postmodernism, Critical Social Theory, and Feminist Theory. The immeasurable challenge of the future is to look through diplomatic eyes without the "terministic screens" (Burke, 1966) of our specializations and disciplines that impair our vision. We could begin to address t he phenomenon of public schooling, its reform, and its evaluation within a politically honest analysis. By refocusing our individual and collective powers into a moral and p olitical analysis, critical evaluation of education reform in the next century can begin to r egain the democratic imperatives or possibilities of public institutions. Exercising th is moral and political "judgement" in evaluation of education reform, as a social respons ibility in public practice, requires instrumental courage and conscience.Analytic PrimacyAlso this article aspires to begin a discourse beyo nd what politicians, educators, and philosophers have debated for centuries, the extent to which education should develop the individual or serve the needs of the state and society. The fact that this debate seems to go on and on with no principled resolution in si ght suggests that deeper issues may be at stake. Namely, it suggests that academic dispute over what has "analytic primacy" (Wertsch, 1998, p.9), the individual or society, ma y reflect an underlying debate, a debate that cannot be resolved through rational arg ument. I am recommending that evaluators of education reform lift the blinders of methodological habit, move beyond their rational arguments, and discover how their ow n morals and politics are partly reflected in their professional decisions. With thi s in mind we live in times of increasing uncertainty as to how to reform public education. P art of the success of education reform will depend on those who have the power to affect s ocial change, who have control over the knowledge base, who judge the worth and merit o f educational programs, and what kinds of morals and politics are profoundly ingrain ed within their minds, spirits, and hearts.Social Justice and Public Practice
7 of 38For the most part, educational research and evaluat ion have remained both moral and political innocents in theory, practice, and policy Part of this political innocence is derived from self reproducing ideologies and scient ific paradigms that have explicitly or implicitly neglected moral and political issues. Th e conception of social justice, as considered here, is not a privilege for some (merit ocractic) but rather a birthright for all (democratic) (Sirotnik, 1990). The value of social justice forms the foundation for working towards the restoration of a moral and poli tical theory in the evaluation of public education reform, as part of a social respon sibility in public practice, and as a part of confronting the moral and political purposes of social inquiry and research. The contributions of Wertsch, (1998), Gir oux (1997), Prilleltensky (1994), Tsoi Hoshmand, (1994), Howard (1985), Kohlberg (1984), R awls (1971), Habermas (1971), and Kuhn (1970) are offered as significant commissi ons to support the reconsideration of our individual and professional decisions in edu cation, by deliberating on our own morals and politics. Reflections and deliberations on our own values, beliefs, passions, and the reasoning for our professional decisions ar e mostly done outside of the confines of our "professional lives." Thus we are left with the interesting and paradoxical conclusion that what "ought" to be the most central in the evaluation of our schooling of American children, the moral and political reasonin g, becomes inevitably peripheral to our public practice (Miller & Safer, 1993). In term s of articulating in-depth moral and political positions related to evaluation in educat ional reform, these considerations and decisions are vital to building and transforming sc hools that are struggling to achieve democratic ideals.Between the Potential and the Present Issues such as equality, democracy, race, gender, class, and poverty are certainly integrated through variable means into the contempo rary scholarship of educational psychology, research, and evaluation. However, thes e issues and their historical, political, moral, and economic meanings are rarely discussed in a comfortable forum naturally or agreeably in the impregnable halls of academia. Therefore, the silent space between the potential in education and the present crisis in public education is successfully and safely insulated decade after deca de. As a result, inquiry and discourse between "public intellectuals" remain fixed in a no npolitical environment without values, beliefs, and passions. This environment wit hin an "ideology of neutrality" became internalized in the consciousness of most re searchers following the establishment of the modern university. The links b etween the political agendas and research were, and often remain, blurred by the leg itimating function of social and educational research. This can be seen in many educ ational evaluation studies that accept the objectives of pedagogical programs and a re organized to "explain" how the objectives were reached.Redefinition of Identity and Purpose No problem is more difficult and complex in the social sciences than that of determining how morals and political values are emb edded within the research methodologies that we employ and the "academic" dec isions that we make (Cronbach & Associates, 1980; Hamnett, Kumar, Porter, & Singh, 1984; Fetterman, 1988; and Sirotnik, 1990; Maruyama & Deno, 1992). That morals and political values should exist in research is no longer denied (Warren, 1963; Fett erman, 1981; Freire, 1985; Apple & Beyer, 1988; Habermas, 1990; Prilleltensky, 1994; G iroux, 1997; Kanpol, 1997;
8 of 38Wertsch, 1998). In terms of educational evaluation, the ideas found in this article, reconfirm the conviction made by Sirotnik (1990) th at the practice of evaluation is part of the political authority structure of society, an d that evaluation as an aid to public decision making entails conceptions of democracy an d social justice, even when these conceptions are not immediately apparent. House (1993) wrote that evaluation receiv es its authority not only from its presumed "scientific method" but also from governme nt endorsement itself. Within the analysis of evaluation in advanced capitalist socie ties, House reviewed how governments face serious problems in governing such a multicultural "amorphous mass of people" (1993, p.vii) and how evaluation is both political and scientific authority applied to practical decision and actions, particul arly public decisions and actions. He went on to explain how governments are capable of m aking decisions based on their own political authority, but that it is easier to g overn based on voluntary acceptance by the populace attained through scientific persuasion particularly when the populace is pluralistic and increasingly non-traditional. In ad dition, House expanded the notion of political and scientific authority by redefining fo rmal evaluation as a new from of cultural authority. Cultural authority can be manif ested in the probability that descriptions of reality and judgements of value wil l prevail as valid, an increasingly difficult accomplishment in societies with disparat e value systems (House, 1993). Current literature in evaluation confirms that eval uation as a social activity is becoming increasingly self-conscious about its own identity and purpose in the larger social order (Cronbach & Associates, 1980; Guba & Lincoln, 1989; Rossi & Freeman, 1993; Patton, 1994; Scriven & Kramer, 1995; Chelimsky & Shadish, 1997).Critical Evaluation Critical evaluation of education reform i nvolves the practice of completing empirical, historical, public and social work by em ploying explicit theories of justice (House, 1976,1980) that require serious commitment, persistence, courage, conscience, and conviction in order to restructure and transfor m education environments. Hence, as a social practice, evaluation involves an inescapable ethic of public and social responsibility that extends well beyond the immedia te clientele by focusing on the democratic purpose of schooling. Social justice in evaluation, then, concerns the manner in which various interests are served. Critical eva luation should serve the interests not only of stakeholders, sponsors, or the reformers, b ut of the larger society and of various groups within society, particularly those most affe cted by the educational programs under review. One of the aims of this article is to reiterate that institutions of higher education must be seen as deeply moral and politica l spaces in which evaluators, indeed intellectuals, assert themselves not merely as prof essional academics but as citizens, whose knowledge and actions presuppose specific vis ions of public life, community, and moral accountability (Giroux, 1997).A Political Theory I propose here that critical evaluation r epresents a kind of political theory that integrates explicitly the value of social justice i nto the practices, policies, and responsibilities of evaluation of educational refor m. Moreover, the political theory of critical evaluation can be defined as the implicit and explicit social and professional ethics of evaluation, and the moral and political c onsequences of these ethics, which could reconstruct and reconsider the power relation s in academia and public education.
9 of 38One of the reasons to begin a journey into a critic al political perspective in educational evaluation is to arrive at an account, a kind of "t ranslation at the crossroads" (Wertsch, 1998, p.7), that would make it possible to link, bu t not reduce, one perspective of "science" to another. Another reason is to begin ad dressing explicitly the methodology of evaluation and systematically evaluate our own p ractices in social science research (Scriven, 1986). The task is to reflect, to discourse, and to collaborate with each other, between and within disciplines, to dialogue about the human con dition, especially the conditions of inequalities that public institutions perpetuate in our democratic society. In order to talk and listen to one another about social justice in e ducation our "knowledge base" and our morals and politics should be integrated into an id eology of hope and sincere cooperation for a better future for children throug h education reform.Overview A characterization of a critical evaluato r will be advanced shortly. The role divisions of academic versus service orientations e xisting in evaluation today are described. The ideology of a critical theory of edu cation, and critical social psychology will then be reviewed in order to consider augmenti ng traditional positivist perspectives of evaluation. Afterwards I give a brief summary of evaluation in general. Three perspectives of evaluation and their purposes are e xplained, in order to illuminate the more traditional positivist approaches in prevalent current evaluation literature and to describe a spectrum of responsibility, purpose, and definition within the discipline of evaluation. The three perspectives on the spectrum are those of accountability, knowledge, and development. Next, the limitations of contemporary and critical evaluation and how these approaches may implicitly serve a particular set of social and political values is forwarded. Integration of critical evaluation into a changing society, Fetterman's silent scientific revolution, the ideas of practicing crit ical evaluation, the neutrality of schools, and change in American schools are also presented. Subsequently this article conceptualizes one important process that an evalua tor must experience in the context of Freirean pedagogy, so that a critical evaluator can begin the special role of critical evaluation in educational reform. The implications of critical social thought for evaluation in educational reform are then proposed. Finally, the second part of this article begins by describing the interdisciplinary methods and procedures of the "how" of critical evaluation, by introducing the integration of American public school law as enhanced by collaborative consultation and the adve rsary-advocate oriented evaluation model.The Critical Evaluator Ernest R. House was the first prominent e valuation theorist to advocate valuing based on principles of social justice (Patton, 1997 ). He has consistently voiced concerns for democratizing decision making in that context, he has analyzed the ways in which evaluation inevitably becomes a political tool in t hat it affects "who gets what." As mentioned earlier, education itself, as well as edu cational evaluation can enhance fair and just distribution of benefits or it can distort such distributions and contribute to inequality. In considering judgements on programs, the social justice evaluator, the critical evaluator, is guided by such principles an d values as equality, fairness, and concern for the common welfare (Sirotnik, 1990).
10 of 38 Kenneth Sirotnik and Jeannie Oakes collab orated in this same endeavor by considering the epistemological connections between critical theory and evaluation. To be specific, they stated that if one accepts the pr oposition that inquiry is never value free and accepts social justice as the ethical starting and ending points for moral argument, then the accumulated body of work done by Freire (1 973), Habermas (1971), and others points the way toward a useful epistemological synt hesis, one that they called critical inquiry, that is evaluative by its very nature (Sir otnik & Oakes, 1990). By no means is critical evaluation a new idea. Regardless, the arg ument for fighting for social justice with critical evaluation of education reform is not a trivial one, but it is an argument that I have extended with much interdisciplinary literat ure and paradigmatic considerations. Michael Quinn Patton (1997) wrote that so cial justice and other similar principles change the role of the evaluator from the tradition al judge of merit or worth to a social change agent. Many evaluators surveyed by Cousins, Donahue, and Bloom (1996) were hostile to or at least ambivalent about whether eva luation, particularly a type of critical evaluation, can or should help bring about social j ustice. Certainly, evaluators undertaking such an approach need to be comfortable with and committed to it, and such an activist agenda must be explicitly recognized, n egotiated with, and formally approved by primary intended users. From Michael Quinn Patto n's utilization focused perspective, using evaluation to mobilize for social action and support social justice are options on the menu of evaluation process uses (1997). In this article, part of the argument is that wherever one places oneself on the spectrum of evaluation responsibility, purpose, and definition; the evaluator can earnestly acknowledge the powerful critical role th at he or she may interpret in placing judgement or giving merit to one of the most profou nd social activities in our lives, that of educating our students and our children. This ro le as a critical evaluator can be found anywhere on the spectrum. As typically happens with most spectrums the outlier situation is pretty rare. A critical evaluator can produce empirically traditional research designs in combination with critical social ideolog y, as long as one maintains a critical stance towards methods, practice, and policy that a ddresses the more difficult questions about the institutional problems in educational pro grams, those of democracy, power, and inequality. Patton (1994) also advocated the us e of "mind shifts back-and forth between paradigms within one evaluation setting." Most of the time, in most environments re presented on the spectrum, "scientific" positivist traditions about knowledge and postmoder n critical social constructions of knowledge are almost bound together, and evaluators must therefore always be prepared to confront them both (Young, 1990). In Ethics, Politics, and International Social Science Research Hamnett, Kumar, Porter, and Singh (1984) compared and described the aforementioned theoretical presuppositions such as that of positivist constructions of knowledge and that of critical theories of knowledg e. A significant point here is that a critical evaluator can utilize the necessary tools and methods within shifting research paradigms and changing concepts of knowledge constr uction, in order to augment practices and policies which are continuously parti cipating in a discourse that values a moral imperative and a political commitment to soci al justice in the evaluation of education reform. This understanding of a moral imp erative and a political commitment in educational evaluation is crucial in establishin g explicitly the ideology of a critical evaluator and in making one's analytical biases cle ar. The following paragraph provides a synops is of Sirotnik's and Oakes' review of the three faces of inquiry and analysis (1990). Mos t educational researchers and evaluators have been schooled in the tradition of t he scientific method and the hypothetico-deductive paradigm borrowed, presumably from the physical sciences. But
11 of 38there are at least two other separate and general o rientations for systematic inquiry having strong philosophical roots and demonstrable utility for the social sciences. The more familiar is the whole class of naturalistic me thodologies. The second major departure from the empirical analytical tradition i s less well known and much more separable, namely, the critique of knowledge. Its r oots are also in the hermeneutical tradition. But as a philosophy of inquiry, it repre sents a significant extension of interpretive inquiry. Inquiry and analysis does not happen in a normative vacuum, as they so eloquently stated. Sirotnik and Oakes (1990) also suspected that "an epistemological trap can be created through assuming necessary and sufficient c onnections between method and the political content of cognitive interests. Conductin g empirical analytic inquiry, for example, does not necessarily imply a hidden agenda of domination. On the other hand, a hidden agenda of domination cannot in principle s urvive an inquiry based on critical theory" (p.45). I agree with these authors that thi s, indeed, points the way out of the trap, a truly practical unification of the three faces of inquiry requires the self correcting epistemological stance that is made to order in a c ritical perspective. At the same time evaluation must consider what kind of orientations are created in practice when these epistemological and empirical stances are postured.Academic Versus Service Organizations One of the most basic role divisions in t he profession today is between academic and service oriented evaluators, a division identif ied by Shadish and Epstein (1987) when they surveyed a stratified sample of the membe rs of the Evaluation Network and the Evaluation Research Society, the two organizati ons now merged as the American Evaluation Association. The authors inquired about a variety of issues related to evaluators' values and practices. They found that r esponses clustered around two contrasting views of evaluation. Academic evaluator s tend to be at universities and emphasize the research purposes of evaluation, trad itional standards of methodological rigor, summative outcome studies, and contributions to social science theory (Patton, 1997). Service evaluators tend to be independent co nsultants or internal evaluators and emphasize serving the stakeholders' needs, program improvement, qualitative methods, and assisting with program decisions (Patton, 1997) According to Shadish and Epstein, The g eneral discrepancy between service oriented and academically oriented evaluators seems warranted on both theoretical and empirical grounds" (1987, p.560). The profession of evaluation remains very much split along, these lines, but with new twists and perhaps deeper antagonisms (Patton, 1997). Patton goes on to explain how the "schism" erupted openly, and perhaps deepened, in the early 1990's, when morality entered into the ev aluation arena much more explicitly, and the American Evaluation Association elected suc cessive presidents who represented two quite divergent perspectives. Yvonna Lincoln (1991), in her 1990 presid ential address, advocated what Patton would call an activist role for evaluators, one tha t goes beyond just being competent applied researchers who employ traditional scientif ic methods to study programs, the academic perspective. She closed her speech by asse rting that "my message is a moral one." The following year, the American Evaluation A ssociation president was Lee Sechrest, who by his own definition represented the traditional, academic view of evaluation. He objected to Lincoln's metaphorical c all for a new generation of evaluators. "I ask myself," Sechrest (1992) mused, "Where in our makeup are the origins of this new creation so unlike us.... I sense a ver y real and large generational gap" (p.2).
12 of 38 From this contemporary discourse in what the role divisions personify in evaluation, one can tell that critical evaluators m ay be characterized as divergent or even marginal in their theoretical and empirical presupp ositions. Here lies the embedded professional challenge of remaining open to plurali stic and cosmopolitan approaches which adapt evaluation practice to new situations, mainly the situation of public education institutions which are failing a growing disproportionate amount of disadvantaged children thereby reproducing social a nd symbolic inequalities. Ultimately, there is no one way to conduct an evaluation. This insight is crucial. The design of a particular evaluation depends on the people involve d and their situation.Ideologies of Critical Theory, and Critical Social Psychology Traditionally social science and social p sychology we are told, is a vocation of scientific method, a devotion to truth that should not be compromised by the researcher's idiosyncracies or other external forces and should not be unduly affected by the social context in which the researcher operates (Hamnett e t al., 1984). In the realm of the natural sciences, statements often appear to be rea ffirming this stance. For instance, in practice there is very little to distinguish Soviet and U.S. nuclear physics. Changes in theoretical presuppositions in one country are rapi dly translated to others. Social science and social psychology, how ever, do not have the canons of theoretical perspective, verification, or even of d ata collection found in natural science (Hamnett et al., 1984). Hamnett and his co-authors state that theoretically, the sociology of knowledge has demonstrated how science (includin g the concepts, methods, and procedures embodied) presupposes historically relat ive values, interests, and ideologies. The taken for granted notion of the methodological neutrality of scientific method has been undermined by theorists of many persuasions in cluding that of critical theorists and critical social psychologists (Wexler, 1983). I agr ee with Wexler when he writes that conventional wisdom and common sense concedes that science is influenced by human values and the political contexts of its expedition This is why the evaluator of education reform cannot posture a neutral, purely objective p oint of view on the object of his research, especially in the reforming of such a soc ial contract as education. The writings of critical theory developed from the Institute for Social Research in Frankfurt. The critical theorists are concerned wit h the role of values and ideology as "part of the conceptual framework which defines wha t it is to have, i.e., scientific knowledge about some phenomenon" (Sabia & Wallulis, 1983). Such a focus raises important questions concerning social science resea rch, ethics, and inevitably the practice of evaluation in education. Critical theor ists state that it would be incorrect to claim that positivist doctrine is responsible for t he unreflexive state of the research ethics and politics debate in social science; the s ocial, historical, economic, and political context of research is of overwhelming importance ( Sabia & Wallulis, 1983). How one views the role of social research its relations to political practice, and how one assesses responsibilities, relationships, a nd appropriate conduct should be explicitly negotiated up front with potential clien ts in terms of one's underlying assumptions and ideological presuppositions. Moreov er, critical research methodology is distinctive from other approaches in that it traces the origin of our concept of validity back to everyday human interaction. This is true, a t least, for the specific brand of critical methodology I advocate, which draws heavil y from Habermas's work on validity (Habermas, 1981, 1987). The later discourse of this critical evaluation perspective, which can be embedded in a positivist scientific me thod, does not assume the posture of rejection or exclusion, but rather will serve to pr ovide an additive component to
13 of 38constructing knowledge and representing it with cri tical and conscious eyes. I repeat what Lewis Carroll's Alice would have said, "things are not what they seem." There is a difference between listening to t he goals of reformers, and listening attentively to the underlying assumptions forwarded by education reform efforts, and consequently holding the reformers responsible for living up to their social ideals and their program mission statements, mainly those miss ion statements that become framed cultural symbols of what a program or a school repr esents. These framed paper certificates, these mission statements, are usually strategically placed in the front office of every public school and meticulously published i n brochures summarizing the goals and objectives that school districts represent to w elcome potential inhabitants of the communities they serve. If we can understand the ce ntral role played by validity claims in normal human communication (symbolic or otherwis e), we will then be able to formulate the special requirements that a critical evaluator conducting formal inquiries into social processes must employ to produce a trus tworthy account. In critical evaluation, the validity claims made by the evaluat or do not differ in nature from validity claims made by all people in everyday contexts. Critical social psychology draws from the critical theory of the Frankfurt School and the theoretical traditions of Marxism (Wexler, 1983). Philip Wexler (1983) augmented and amplified what he perceived as develo ping tendencies in social relations and in social psychological processes. Like Philip Wexler's expression of a need for augmentation, I am asking those who study, practice and use evaluation in education to broaden and amplify their view of the applications and functions of evaluation with an eye to the future. The evaluator could be responsib le for reaching beyond mainstream philosophy and practice in evaluation because the t ransforming of education and the reforming of such a significant social activity req uires an exceptionally conscious human being. Like critical social psychology, a critical evaluator requires a theory which can comprehend and facilitate social change movements. Next I shall give an overview of evaluati on in general, its development, and then review three perspectives of evaluation and their p urposes, in order to illuminate the more traditional positivist perspectives in prevale nt current evaluation literature. These three perspectives again are those of accountabilit y, knowledge, and development. By looking at these three perspectives and their posit ions along a spectrum, I argue that the evaluator must go beyond those delineated perspecti ves in mainstream evaluation theory, policy and practice, in order to take a more critic al posture toward both education and the very process of thinking about education.Evaluation Evaluation as an academic discipline, a pro fession, and a government function has only developed in the past four decades in the Unit ed States and in several other industrially developed nations. In many nations, ho wever, evaluation is in its infancy as a standardized pursuit; and certainly on a global s cale, evaluation is only beginning to enter the scene (Chelimsky & Shadish, 1997). There is comfort in knowing, as previously mentioned, that current literature in ev aluation confirms that evaluation as a social activity is becoming increasingly selfcons cious about its own identity and purpose in the larger social order and is beginning to systematically evaluate its own methodology, utilization, values, and politics (Cro nbach & Associates, 1980; Guba & Lincoln, 1989; Rossi & Freeman, 1993; Patton, 1994; Scriven & Kramer, 1995; Chelimsky & Shadish, 1997; House, 1993; Scriven, 19 91). I would agree with Chelimsky and Shadish (1997) when they propose that evaluators, in whatever field of
14 of 38evaluation they may be, are likely to find themselv es, at least sometimes, at odds with the political actors, systems, and processes in the ir own backyards, that rally against a free flow of information and collaborative action w hich endangers the status quo. Between 1965 and 1990 the methodology, ph ilosophy, and politics of evaluation changed substantially, partly in response to the st ructural transformations in an advanced capitalistic society (House, 1993; Scriven 1991). T he strongest stimulus to the development of evaluation was Lyndon Johnson's Grea t Society legislation, which, though not capable of changing U.S. society as a wh ole, certainly transformed educational and social research. At Senator Robert Kennedy's insistence, the Elementary and Secondary Act in 1965 mandated evaluation of pr ograms for disadvantaged students, and this spread to all social programs an d beyond (McLaughlin, 1975). House (1993) reviewed clearly in lay terms how evaluation moved from monolithic to pluralist conceptions, to multiple methods, multiple measures multiple criteria, multiple perspectives, multiple audiences, and even multiple interests. Methodologically, evaluation moved from a primary emphasis on quantitative methods, in which the standardized achievement test employed in a randomized experimental control group design was most highly r egarded, to a more permissive atmosphere in which qualitative research methods we re acceptable. Mixed data collection methods are advocated now in a spirit of methodological ecumenism (House, 1993). The following three perspectives describe mo re thoroughly the way that evaluation is characterized in contemporary evaluat ion circles.Examples of Purpose and Perspectives in Evaluation (Chelimsky & Shadish, 1997)Below find a review of the definitions and characte rizations that Chelimsky and Shadish write about in Evaluation for the 21st Century They offer an inexhaustible listing of possible purposes for evaluation. These purposes in clude the following: (a) to measure and account for the results of public policy, and p rograms, (b) to determine the efficiency of programs and their component processe s, (c) to gain explanatory insight into social and other public problems, (d) to under stand how organizations learn, (e) to strengthen institutions and improve managerial perf ormance, (f) to increase agency responsiveness to the public, (g) to reform governm ents through the free flow of evaluative information, and (h) to expand results o r efficiency measurement from that of local or national interventions to that of global i nterventions such as reducing poverty and hunger or reversing patterns of environmental d egradation. All of these purposes are, of course, worthwhile and legitimate reasons f or conducting evaluations, but they differ with regard to the questions they address an d the kinds of methods needed to answer these questions. Chelimsky and Shadish propose that these different purposes, along with the questions they seek to answer, seem to fall natural ly into three general perspectives: evaluation for accountability (e.g., the measuremen t of results or efficiency); evaluation for knowledge (e.g., the acquisition of a more profound understanding in some specific area or field); and evaluation for development (e.g., the provision of evaluative help to help strengthen institutions). The methods of these three perspective are not mutu ally exclusive. Though they do represent notable differences on a variety of dimen sions. Each may be needed at
15 of 38particular times or policy points and not others (e .g., evaluation for knowledge may need to precede accountability). Chelimsky and Shadish ( 1997) write that they appear to have considerable explanatory power with regard to the c urrent tension in the evaluation field. (See Table 1 for further details.) Table 1, an adap ted chart from Chelimsky and Shadish's book (1997, p.21), shows the following th ree different perspectives and their respective positions along five dimensions.Table 1 Three perspectives and their positions along five d imensions, adapted from Chelimsky and Shadish (1997, p.21)DIMENSIONS ACCOUNTABILITY PERSPECTIVE KNOWLEDGEPERSPECTIVE DEVELOPMENTALPERSPECTIVE PURPOSE to measure results or value for funds expended:to determine costs, toassess efficiency to generate insightsabout public problems, policies,programs, &processes, to develop new methods and tocritique old ones to strengthen institutions to build agency ororganizational capability in some evaluative area TYPICAL USES policy use, debate and negotiation, agency reform, public use enlightenment use, policy, research and replication, education,knowledge base construction institutional or agency use as part of the evaluativeprocess, public and policyuse EVALUATOR ROLE distant distant or close depending on evaluation design andmethod close, the evaluator is a "critical friend" or may be part of a team ADVOCACY unacceptable unacceptable, but now being debated often inevitable, but correctable through independent outsidereview POSITION UNDER POLICY DEBATE can be strong (depending on leadership) can be strong (if consolidated and dissemination channels exist) uncertain (based on independence and control)The Accountability Perspective From the standpoints of auditors, governm ent sponsors of evaluation studies, donors to international organizations, and many oth ers, evaluation is done to establish accountability. This involves the provision of info rmation to decision makers, whether they are in the public or private sector. Specific cause and effect questions about the results in an accountability perspective might be: What happened to poverty levels among the very poor as a result of development assistance provided" Did an educational intervention or program produce more "effective" le arning for all learners? Has teacher training increased student achievement?
16 of 38 Sometimes, questions about the results fr om an accountability perspective may involve merely documentation of whether or not anyt hing has changed after something new has been tried. Normally, however, the ability to say that something is in fact a "result" hinges on the ability to establish that it came about because of something else. Many methods are used to answer these kinds of acco untability questions including: randomized designs, quasi-experimental designs, mix ed multi-level designs, mixed qualitative/quantitative designs, case studies, pro cess studies, and research synthesis designs.The Knowledge Perspective In the view of many researchers working i ndependently in universities and other evaluators in scientific institutions, evaluation i s done to generate understanding and explanation. Chelimsky and Shadish (1997) stated th at the specific questions may not be especially important to analyze here, given that it is the evaluator who decides what will be asked and answered, and the topic generally foll ows from the researcher's prior work. They explained that the evaluations associated with individual academic researchers, or those of research teams, will be more likely to con tinue in depth cumulative inquiry into particular areas or sectors of research than to be concerned with applying systematic research methods to a variety of sectors, as with a ccountability and developmental evaluations. The larger purpose of the knowledge persp ective is to increase understanding about the factors underlying public problems, about the fit" between these factors and the policy or program solutions proposed, and about the theory and logic (or lack thereof) that lie behind an implemented intervention. "These evaluations may employ any of the methods discussed above, separately or in conjuncti on with each other, but the purpose of knowledge gain leads logically to the use of the st rongest designs as well as the greatest clarity possible in explication and documentation o f methods to facilitate replication or later use in research synthesis and policy formulat ion" (1997, p.14).The Developmental Perspective For government reformers, public managers and others, evaluation is done to improve institutional performance. It serves as a f lexible tool that works: (a) to improve the design of projects, (b) to measure and recommen d changes in organization activities, (c) to develop the indicators and performance targe ts needed to improve institutional effectiveness and responsiveness, (d) to monitor, i n an ongoing way, how projects are being implemented across a number of different site s, and/or (e) to find out how beneficiaries feel about an agency and its programs To some accountability or knowledge perspective evaluators, developmental eva luators may seem more like evaluation "consultants" than evaluators, but those who do developmental work are convinced that building evaluation capability is as important an evaluation function as evaluation itself and that indeed, in some cases, e valuation cannot be done without capacity building. Specific questions asked of evaluators in a developmental perspective might include the following: What is the best research ev idence with respect to formulating a new program or modifying an old one? How can projec ts be structured so that they produce evidence on the value of the intervention b eing tested? What is the most appropriate agenda for the agency? Both process and outcome designs may be used in a developmental perspective, depending on the evaluat ion question posed. In addition to
17 of 38the methods mentioned earlier, the formative method s used in the developmental perspective include the following: monitoring, empo werment evaluation, cluster evaluation, performance measurement, and research s ynthesis of both qualitative and quantitative methods. A developmental evaluator bec omes part of the design team helping to shape what's happening both processes an d outcomes, in an evolving, rapidly changing environment of constant interaction, feedb ack, and change. Using mixed methods and multiple criteria in this perspective a re productions of some of the many current trends in the practice of evaluation.Demonstrating a Particular Set of Social and Politi cal Values Although evaluation has developed as a di scipline, a profession, and as a government function in the past four decades by bui lding on its scientific positivist traditions and by systematically evaluating its own existence in the larger social order, this particle emphasizes continual growth and augme ntation of its practices, policies, and responsibilities through a "conscientization" o f evaluation's socio-political reality. Over the years evaluation has come to be seen as po litical. Michael Quinn Patton, at the National Evaluation Conference in Youngstown State University held in September 1998, summarized 12 recent trends in evaluation. On e of them being the increasing political sophistication and acknowledgment of the role of values and morals in evaluation practice. There can be no doubt, that ev aluation is influenced partly by political forces, and in turn, has political effect s. Whose interests are served and how interests are represented in an evaluation are now very critical concerns in a society with increasing disparate value systems. In the earlier days, it was assumed that the interests of all parties were properly reflected in the traditional outcome measures, but this assumption came to be questioned, and it was recognized that different gr oups might have different interests and might be differentially affected by the educational program and its evaluation (House, 1993). "Stakeholders" (those who had a stake in the program under review) became a common concept, and representing stakeholder views in the evaluation became an accepted practice. The stakeholder concept is based on the p revailing pluralist-elitist-equilibrium theory of democracy, which disclaims any normative judgements and which holds that the current system of competing parties and pressur e groups performs the democratic function of equalizing the diverse and shifting pol itical demands (MacPherson, 1987). It is perceived that describing what others value is t he stance best suited to the political context in which evaluators operate, because decisi on making depends on the values held by relevant policy makers and stakeholders. Pr esumably, these parties will use the findings to make informed decisions. Neither the go vernment nor the evaluator is supposed to intervene to support any particular int erests but rather only to provide information that is valueneutral and interest-neu tral. The interests of various groups somehow dissolve into the values of decision makers and stakeholders. However, one must note that today's profe ssional evaluators sometimes become evaluators by default. We represent an eclectic and diverse combination of various professional, academic, and research areas. Shadish and Epstein (1987) found that 31% of the respondents in their survey described their primary professional identity as that of "evaluator" (p. 560). Others thought of themselves first as a psychologist, sociologist, economist, educator, and so on, with identity of ev aluator secondary. When both Charles Murray (1983,1984) and Michele Fine (1983b, 1988) h ave been successful evaluators representing a particular set of social and politic al values and interests, one has to
18 of 38acknowledge the diverse socio-political reality in which evaluators actually find themselves in practice. In two highly visible stakeholder evaluat ions funded by the federal government, those of Cities-in-Schools and Jesse Jackson's PUSH /Excel program, the evaluations worked against the interests of the program partici pants and the inner-city students which the programs were supposed to serve, thus cal ling into question the justice of these evaluations (House, 1988; Stake, 1986). The r esults of the PUSH/Excel evaluation were used not only to discredit the program but als o to question Jesse Jackson's ability to manage large enterprises during ensuing presidentia l campaigns. In truth, the stakeholder model was never implemented (House, 1988; Stake, 19 86). Charles Murray, the evaluator in both cases, substituted a technocratic model of evaluation and expressed his disdain for the stakeholder concept in his article Stakeholders as Deck Chairs (1983). Although the stakeholder approach seems firmly entr enched, there is disagreement about how to implement it. In reality, stakeholders do no t have equal power to influence and utilize the evaluation, nor do they have equal prot ection from the evaluation. These types of problem in evaluation led into a discussion of misuse of findings. The fact that so much standardized achievement test ing is reported to the public in general and its interpretation left to the media or government officials makes misuse particularly salient (House, 1993). In fact, the pr ofessional standards for evaluation developed by a committee led by Stufflebeam, devote d considerable space to issues of misuse, but the context in which evaluation results are presented does not lend itself to the employment of such standards, even though the s tandards are widely accepted in the evaluation community itself. How misuse of findings and disparate interests can be curtailed is by no means clear. The professional st andards for evaluation developed by the Joint Committee dramatically reflected the ways in which the practice of evaluation had matured in 1981. In 1994, revised standards wer e published following an extensive review spanning several years. While some changes were made in the 30 in dividual standards, the overarching framework of four primary criteria (utility, feasib ility, propriety, and accuracy) remain unchanged. However, the profession of evaluation ha s not yet developed to the point of reflecting a common core of practices and principal s as demonstrated by the original professions, divinity, law, and medicine (House, 19 93). We must pay attention to the fact that certification programs and higher educati on programs in evaluation and evaluation research are a very recent development i n the discipline (Chalimsky & Shadish, 1997). For a deeper understanding of how t he original professions compare with evaluation as a profession, refer to House's ( 1993) book, Professional Evaluation .Limitations of Contemporary Evaluation and a Reflection on American Public School Law There are limitations to contemporary and critical evaluation frameworks. The problem of addressing multiple values and interests and how they should be represented in an "equitable" evaluation can take one directly into the realm of social justice and the recognition of the assumptions, character, and cons equences of conventional forms of educational evaluation and American public school l aw. The problem of evaluation representing a particular set of political and soci al values (i.e., a broadly conservative set) also raises some serious questions about evalu ation in general. Although the socio-political reality of multiple stakeholders an d evaluators who have legitimate values and sometimes conflicting interests is recog nized, how these values and interests are legitimized will become one of the most importa nt challenges for educational
19 of 38evaluation in the future, especially for critical e valuation of education reform. How to synthesize, resolve, and adjudicate all these multi ple multiples in our increasing multicultural and amorphous society remains a formi dable question, as indeed it does for the larger society. One thing we do know is that the socio-po litical reality in evaluation of public programs, both in education and health, often works in favor of higher income groups and against equity despite the stated objectives (B irdsall & Hecht, 1995; Paul, 1991; Fine, 1983). When we look at the political structur es and the broad organization of society, resource allocation and subsequent deliver y of services and programs tend to be skewed in favor of those who have more "voice" (Fin e, 1983; Fine & Weis, 1993). In many cases, powerful stakeholders or groups, which are able to effectively demonstrate their interest in receiving social services and "ef fective" or "successful" social programs, manage to get the lion's share of the resources and the funds. It is no secret that the United States of America is one of the last Western industrialized nations to base their educational financing system on that taxation of la rgely differentiated property values. This financial arrangement alone should illuminate some of the deeper issues at stake in the evaluation of public education environments. American public school law and its case h istory has demonstrated time and again that there are very few instances where citizens ha ve been able to prove that state school finance systems result in revenue disparities which violate the Equal Protection Clause of the Fourteenth Amendment. In 1973, in the case o f San Antonio Independent School District v. Rodriguez, Mr. Justice Powell, delivere d the opinion of the Supreme Court. He said, "to the extent that the Texas system of sc hool financing results in unequal expenditures between children who happen to reside in different districts, we cannot say that such disparities are the product of a system t hat is so irrational as to be invidiously discriminatory...." If disparate allocation of governmental b enefits can be justified on the basis of reasonable classification or the interests involved are not fundamental, then statutes will be regarded as constitutional (Alexander & Alexande r, 1992). The court in the Rodriguez case basically ruled that a state legisla ture can heap benefits on some wealthy school districts and deprive others of fiscal resou rces and not offend the federal Equal Protection Clause. Thus representing the educationa l interests of disenfranchised stakeholders, even within the American public schoo l law domain, can be confounded with many inherently unequal and disparate value sy stems. In other instances, our social service in stitutions, such as education and health, are able to shape the systems to serve their own person al and professional goals at the expense of equitable delivery (Paul, 1991). Problem s created by the limited voice of politically weak or disenfranchised stakeholders ar e exacerbated in educational evaluation, when combined with direct provision of services in virtual public monopolies of the "best teachers," the allocation o f "best practices" in education, and the provision of high quality curriculum and profession al development training which are centralized in higher socio-economic communities. U ltimately, citizens have limited capacity to improve the public education they are p rovided through participating, informing, and making recommendations. This is espe cially true of lower socio-economic community stakeholders which have tr aditionally been limited in their capacity to have their "voice" heard without legal representation (Fine, 1993; Oakes & Guiton, 1995). Historically, when interests have been ig nored and educational procedures have been violated, lower socio-economic communities, mi norities, exceptional populations, and limited English proficient citizenry have had t o turn to the legal system for any kind
20 of 38of adjudication (Paul, 1991; Haring, McCormick, & H aring, 1990; Oakes & Guiton, 1995). Similarly, in terms of fighting for social j ustice in education, evaluation of education reform efforts could benefit from address ing some of the principals in American public school law. This idea will be furth er developed in Part II below. However, for the time being, contemporary evaluatio n which was invented to solve social problems, can be afflicted with many of the problems it was meant to solve. Another limitation of critical evaluation in education reform pertains to its inherent questioning of the institutional character of education. By producing educational criticism and value judgements of insti tutional programs and personnel, in conjunction with the ideologies of critical theory of education, critical social psychology, and Freirean pedagogy, critical evaluators risk cer tain professional isolation from the mainstream. The socio-political reality in which on e can survive as an evaluation professional of education reform becomes integrated into a world with those individuals that agree with your views, particularly those who agree with your views on social justice and in general the democratic purposes of p ublic schooling. As critical evaluators conduct evaluations to address the elimination of i nequality and the improvement of those who are least well off, they will come into c onflict and threaten established authority. Any method of evaluation that claims to be nonobjective and value-laden will be marginalized. Society expects evaluation to be b ased on scientific authority. However, I expect the notion of what is scientific to be substantially redefined. The concepts of objectivity, scientific methodology, an d validity will be recast to accommodate different evaluation approaches (House, 1993).Integration of Critical Evaluation into a Changing Society Evaluation continues to become ever more methodologically diverse. Evaluation in general draws from the theoretical foundations o f many fields and is multi-disciplinary and multi-faceted in nature (Che limsky & Shadish, 1997). It is by now well established that the full array of social scie nce methods belongs in the evaluator's methodological tool kit, including tools from psych ology, statistics, education, sociology, political science, anthropology, and eco nomics (Cronbach & Associates, 1980). When the critical logical and analysis tools given to us by critical theorists and social psychologists are included into an evaluatio n design, the role that evaluators play in judging the worth of educational reform efforts is elaborated. Chelimsky and Shadish (1997) supported the notion that it is often uncomf ortable to stir oneself from familiar cultural, ideological, topical, conceptual, and met hodological niches. However uncomfortable or reactionary one may feel to the content of this article, there is a message: it is that evaluations of educa tional reform efforts in the next century can and probably will be far more powerful and infl uential than they are today. This is because of the ever increasing complexity of social economic, technological, political, and cultural tensions which are questioning the ver y integrity and purpose for public education as a whole (Giroux & Aronowitz, 1991). Th e growing populations with disparate value systems and socio-economic levels a nd the increasing minority populations in this country will demand to particip ate more legitimately in the reformation of their own education. Consequently ev aluation will have to redefine its identity, its purpose and practices. Lee Cronbach, in 1980, advanced the posit ion that the theory of evaluation has to be as much a theory of political interaction as a t heory of how to determine facts or how knowledge is constructed (Cronbach et al., 1980). E ven so, 18 years later, we still do not seem to understand political processes very well, e specially their dynamic nature. This
21 of 38gap in understanding and consciousness is especiall y true for evaluators in the field of education where we are determining "facts" and cons tructing knowledge about educational programs designed to improve teaching a nd learning in the public school domain. We can begin bridging this gap in conscious ness to understand the political nature of evaluation by looking at our own ideologi es as evaluators. Critical thought, indeed, criticism, is essential to enable us to act in ethically and politically just, to say nothing of intellectually honest, ways. Critical th ought entails questioning, reflection and thoughtful interaction with the information and bod y of knowledge at hand. Education then becomes an active and constructive process of continual critical growth (Dewey, 1944). Fundamentally, I am recommending here tha t evaluators, as leaders of educational reform efforts, become more critical and vigilant a bout the questions they are contracted to answer and about the more profound functions of education programs and practices under the rubric of a critical theory of education (Giroux, 1983b; Young, 1990; Apple & Beane, 1995; Apple 1996, Apple & Carlson, in press ). In addition, these same evaluators could integrate the logic of traditional psychology with the logic of critical social psychology to begin the rethinking of educat ion as a social project and a social process. The purpose of this rethinking is to expan d on positivist traditions of considering an incremental perspective on methodolo gical and research design issues in evaluation, into a more open critical ideology of p ractice and policies (Fetterman, 1988; Maruyama & Deno, 1992). These schools of thought, a pproaches, and particular issues should not be eliminated. We should consider these issues together with the notion that evaluation of education places us in a particularly sensitive arena within the confounds of social and human science (Fetterman & Pitman, 19 86; Fetterman, 1988).Silent Scientific Revolution Fetterman (1988) argued that there is a s ilent scientific revolution in evaluation and that educational evaluation is experiencing a c hange in direction. A critical component of this change is a shift in the paradigm s underlying the method and aim of research (Lincoln, 1986). David Fetterman further s uggests that a marked shift is taking place in the professional allegiance of evaluators. This shift in allegiance, he says, is not a simple linear development. As summarized in Fette rman's book (1988), this shift goes beyond perceiving evaluation as a set of chronologi cal transformations that travel from traditional positivist approaches to dominant quali tative forms of evaluation, including ethnography, naturalistic inquiry, generic pragmati c (sociological) inquiry, connoisseurship/criticism, and phenomenography. Rat her he illustrates some significant moments of metamorphosis, revealing the process of shifting allegiance to a circular and interactive paradigmatic perspective. Similarly, I call on evaluators to lift t he blinders of methodological habit, to increase the ideological options and backgrounds av ailable to them, to go beyond any single discipline, and to build on tradition by eng aging the wisdom of critical social thought. This article is simply describing a possib le interplay between the sciences and between the contemporary perspectives in evaluation Whether using the perspectives of account ability, knowledge or development, or any combination thereof, additional questions could be examined as the evaluation/research design is imposed on the school culture and setting (Maruyama & Deno, 1992). Critical evaluators of education refor m could also listen to emerging questions that are integral to the improvement and restructuring of social projects and social processes, by attending to their own conscio usness and motivations (Young,
22 of 381990). Later I will review Paolo Freire's construct ion of conscientization (Hamnett, Kumar, Porter, & Singh, 1984, p.100) to describe th is experience as necessary for critical evaluators of education reform. School and university researchers/evaluat ors who are taking on the challenge of restructuring schools and school systems in urban a reas are involved essentially in the transformation of existing bureaucracies, bureaucra cies that have had the power to control what is taught and how schools are run (Kre tovics & Nussel, 1994). Clearly American education is organized in a bureaucratic f orm. Kretovics and Nussel confirm that at any level, national, state, or local, the t raditional pyramidal, hierarchical arrangement is in effect. Proposals for reform of p ublic schools and their evaluations must consider how the bureaucratic functionaries mi ght respond. Since bureaucracy is "an institutionalized method of organizing social c onduct in the interest of administrative efficiency," the issue of response i s a genuine concern (Kretovics & Nussel, 1994).Practicing Critical Evaluation In the public school domain, genuine conc ern is adequate but more critical thought and action are needed within the world of education al bureaucracy. One way of practicing critical thought and action for critical evaluation would be to negotiate these ideological and theoretical presuppositions up fron t with one's clients and then deliberately confront issues of institutionalized p ower, democracy, and inequality in the educational programs and reform efforts. One can do this by organizing specific research designs and relationships centered around the conce pt of listening to the multiple voices in education and its programs. Fine and Weis (1993) witnessed and wrote about the practices and consequences of silencing in public s chools. I do not think that evaluators are far from becoming partners in these implicit pr actices. Battling the dynamics of power and privilege that nurture, sustain, and legi timate silencing in education is the first purposeful step that a critical evaluator can take to interpret his powerful role as a transformative agent for social change. Creating fl exible, authentic, and reflexive relationships with the stakeholders and with the ex isting bureaucracies during the process of evaluation is a second step that the cri tical evaluator can take towards completing a critical evaluation (Schon, 1983). If innovative and well meaning educationa l programs or educational reform efforts are developed to improve the education of all stude nts in public schools, then the evaluator of these programs has a very special and conscious role in creating opportunities for authentic discourse about these d ifficult issues that go beyond the successes or failures (outcomes) of children within the structural and organizational components of educational practice. The role of the evaluator and the ability to communicate and address the challenging issues such as democracy, power, and inequality to clients in the field of education, es pecially in the future, will be essential to transforming social activity for social change. Mic helle Fine and Lois Weis (1993) included the following quote in their book: It is a false dichotomy which suggests that academi cs and/or intellectuals can only speak to one another, that we cannot hope to speak with the masses. What is true is that we make choices, that we choose voices to hear and voices to silence. If I do not speak in a langu age that can be understood, then there is little chance for dialogue. We must b e ever vigilant. It is important that we know who we are, who we are speak ing to, who we most
23 of 38want to hear us, who we most long to move, motivate and touch with our words (p.2).The Neutrality of Schools (Social Darwinism Revisit ed) Jeannie Oakes (1986) stated that in their general indictment of schools, the authors of the evaluation studies and reform reports do not attach particular importance to the fact that schools fail to serve all students equall y well. Certain topics like institutionalized power, democracy, and inequality are not explicitly addressed because there is a "silenced" understanding of the status q uo in educational practice. Consequently, the evaluators and reformers in the c ommentary made by Jeannie Oakes do not consider as targets of information or unders tanding the school content and processes that limit school achievement for poor an d minority students. Schools, in general, are often seen as essentially neutral, and the reforms are presented as color-blind and affluence blind. Jeannie Oakes (1986, 1995) fur ther argued that current reform efforts do not address the unequal quality of schoo l facilities, programs, materials, counseling, expectations, and instruction. No inter est is shown, for example, in the unequal distribution of competent teachers. Neither do they address school organizational changes likely to equalize access to high quality educational contexts such as desegregation, the elimination of tracking, and the reconceptualizing of vocational education programs. Thus by extracting the logic of critical theory and critical social psychology, I would extend the "meta-evaluation" done by Oakes, i n saying that the evaluators of these reform efforts are additionally hard pressed to face squarely the "silent" demons lurking behind the institutional practices in publi c education. Even though a common issue is made of increasing the skills and knowledg e of teachers, the assumption is that teachers simply need to get better at what they've always done. Also there is an assumption that all the evaluator has to do is to e valuate how the teacher is teaching and whether the outcomes are effective learning. There is little or no mention of the need for teachers to be more knowledgeable about how poverty racism, and limited expectations affect the educational treatment of poor and minori ty children (Levine, 1971; Coleman, 1981; Fine, 1983,1994). Indeed there is no direct m ention and acknowledgment of these issues on any explicit level within the hierarchica l structure and bureaucracy in education (Levine, 1971; Coleman, 1981; Fine, 1983, 1994). Subsequently, mainstream evaluation of th ese reform efforts in teaching practices and educational programs misses a crucial part of t he picture about how schools are functioning for all children. If we as evaluators d o not ask deliberate questions about institutionalized power, democracy, quality of inst ruction, and inequality within the public school domain, during the process of evaluat ion, then we become one more vehicle that perpetuates an already neutral state o f mind about the world of education and its goals for society. While many faults are fo und with schools, unfairness is not one of them. In addition, the omission of these concern s and "silent" demons in evaluation and education reform efforts makes clear the prevai ling conviction that schools, as they are now, are neutral places (Coleman, 1975; Oakes, 1986; Fine, 1994).Change in American Schools Although there is a perception that chang e needs to occur in virtually all American school districts, including those serving the wealt hiest suburbs, the success of the reform movement will be measured ultimately by its impact on our largest most troubled public
24 of 38school environments. For it is in our largest citie s and our most rural districts that the job of the schools is most difficult, given the often o verwhelming social and economic circumstances of students living in desperately imp overished neighborhoods (Oakes & Sirotnik, 1986). These are the neighborhoods most i n need of transformed schools, and it is in these neighborhood schools that the evaluator can choose to undertake his exceptional role of being a vehicle for change and transformation. Jonathan Kozol in Savage Inequalities (1991) took readers inside schools in poor neighborhoods and forced them to see the places imp overished children are compelled to go. Kozol (1991) commented on more than the physica l, economic, and social inequalities among different types of school, those with affluent children, and those with children from poor homes. He addressed the very "et hos" of a school as maintained by the social-class position of the students. Theodore Sizer in Horace's Compromise (1984) also characterized this difference between schools quite modestly: Among schools there was one important difference, w hich followed from a single variable only: the social class of the stude nt body. If the school principally served poor adolescents, its character, if not its structure, varied from sister schools for the more affluent. It got s o I could say with some justification to school principals, tell me about t he income of your student's families and I'll describe to you your school. (p.6 ) Critical educators such as Michael W. Apple, Henry A. Giroux, Paolo Freire, Jeannie Oakes, Gloria Ladson-Billings, and Maxine Greene wo uld probably agree that evaluation and research in impoverished neighborhoo d schools presents the critical evaluator with an exceptional challenge in social r esponsibility. Hence, these impoverished neighborhoods, where educational refor m proponents advocate change, improvement and restructuring of schools, could be the environments that create wonderful opportunities for evaluators to maintain a critical stance toward theory, research, practice, and social policy.Freirean Pedagogy The statement "All men are created equal" is one that resounds throughout American history. The words are found in the Declar ation of Independence and Lincoln's Gettysburg Address; they are also paraphrased and a pplied in numerous settings. For educators and educational evaluators, it has meant that American schools are charged with offering every child equality of educational o pportunity. This concept of equality of educational opportunity is one that has been implic it in most educational practices throughout the period of public education in the ni neteenth and twentieth centuries (Coleman, 1981). However, no white suburb in Americ a would long tolerate the low academic achievement taken for granted in the urban or rural public schools attended largely by AfricanAmericans, Hispanics, and poor children. In big cities all over the United States, minority students by the tens of thousands leave school each year, some as dropouts, some as g raduates, utterly unprepared to participate in and contribute to a democratic socie ty (Oakes & Sirotnik, 1986). They lack the skills that will allow them to obtain gainful e mployment, and they are devoid of the preparation that will lead to success in further ed ucation. Paolo Freire would characterize this lack of skills and preparation as the "inability to act upon and transform one's world" (Hamnett et al., 1984). Consequently h e would say that the democratic society failed to move this person toward the evernew possibilities of a fuller and richer
25 of 38life individually and collectively through the ausp ices of public education (Hamnett et al., 1984). Paolo Freire is most often recognized for his literary and practical works as an educator. His study and conduct in this field have produced radically new philosophical and political insights. His basic assumption is tha t people are seen to be always in the process of developing. He says that the characteris tic of the human species is its repeatedly demonstrated capacity for transcending w hat is merely given, what is purely determined (Hamnett et al., 1984). From Freire's po int of view, education, or any form of activity directed at social change, can never be neutral; it can only be used to dominate or liberate people. Although this dichotom y is limited in my opinion, these extremes serve their purpose in explaining unique i deological commitments to social change, especially as social science researchers an d evaluators in education. I proposed here that evaluation of public educational programs as a form of activity directed at social change, should follow Freire's recommendatio n for conscientization : Conscientization refers to the processes in which m en, not as recipients, but as knowing subjects, achieve a deepening awareness both of the socio-political reality which shapes their lives an d their capacity to transform that reality (Freire, 1970b). This notion conveys the realization that nobody can help or assist others without their participation; otherwise the helper is led only to treat people as objects vulnerable to control and manipulation from outside (Freire, 1973 ). Here we can reflect upon what such a perspective would require in evaluation of p ublic educational programs. Conscientization is at least one experience that cr itical evaluators should pass through in order to become educational leaders and change agen ts for educational reform.Implications for Evaluators of Education Reform Undoubtedly the purposes, methods and fun ctions of evaluation would change if one was to adhere to the philosophical and ideologi cal underpinnings of critical theory, critical social psychology, and Freirean pedagogy. The question remains: would a critical evaluator actually go beyond traditional methodolog ical concerns to design his policy and practice to deliberately address difficult and possibly uncomfortable issues such as institutionalized power, democracy, and inequality in education? Courage, persistence and conviction are presented here as three crucial elements that will consistently be needed for critical evaluation of educational refor m. In addition to these three elements a critical evaluator could benefit from continual ref lection about one's own changing beliefs and landmark experiences. The need for courage, persistence, and co nviction seems fairly obvious but somehow we do not seem to talk about these characte r traits explicitly. Speaking out, in situations that may include numerous political and bureaucratic agendas, all with different viewpoints and axes to grind, and also in sisting on the right to independence in speaking out, takes a strong stomach. Even in the p olitical and cultural environments occurring toward the middle of the spectrum, the no rmal skepticism of the evaluator is unwelcome amongst the pervasive enthusiasm for one program or another. But as we move down the spectrum toward differing ideologies, doubting the conventional wisdom becomes such an offensive tactic as to deconstruct credibility and solid reputations. It also takes fortitude or conviction and strong resistance not to succumb to political or bureaucratic blinders of one sort or a nother. In my experience with the higher
26 of 38echelons of public education evaluation, both as a teacher and as a district based advisor of educational practice, these blinders lure evalua tors into wanting to become political and "institutional" players on the national scene. There is an insidious temptation to avoid ideological and philosophical battles to the promise of glorious career rewards as compensation for obedience. This possible temptatio n is one of the reasons why there needs to be extensive research started in discoveri ng the implicit and explicit social and professional ethics of different types of evaluator s, especially evaluators in education reform. It takes persistence and courage to refuse sponsors the answers they want to hear, and it takes conviction and certainly conscie nce to ask deeper more resounding questions. Goethe said, "Possessions lost, nothing lost. Principles lost, something lost. Courage lost, everything lost" (quoted in "Visions of Public Service," 1986, p.12).A Beginning to the Methods of Critical Evaluation Part II: The "How" of Critical Evaluation National policymakers, educational leader s, "public intellectuals", and children in disadvantaged situations can benefit from critical evaluation, but not in the same ways and not with the same evaluator roles. Neither more nor less activism, in my judgement, is morally superior. Various degrees of activism in volve different ways to practice as an evaluator, often in different arenas. Indeed, how a ctivist to be, involves consideration of an evaluation's purpose, decisions about intended u sers and uses of evaluation, and the evaluator's own values and commitments, all of whic h need to be made explicit. The challenge will be to create appreciation and space for such diversity among both those within and outside the profession who have a single and narrow view of evaluation and its practice. The debate will and should, go on, fo r that is how we discover implications and ramifications of diverse approaches, but I hope and foresee no desire to turn back the clock to a single dominant perspective. By now, there should be no doubt as to th e rationale for making a space for critical evaluation in the reform of public education. Becau se of the complexity of the task of reconceptualizing the evaluation process toward a p rocess that contains an explicit normative social goal, that of social justice, and a process that is designed for purposes of fundamental change, the arguments in this sectio n will only begin to delineate a preliminary path toward a methodology for critical evaluation. However, a more detailed and experienced methodology for critical evaluation would require further conceptual and empirical investigation and time. Essentially t he utilization of American public school law, both state and federal statues, are com bined with the adversary oriented evaluation model in order to propose briefly that t hese statutes can serve as merit criteria for determining the value and worth of educational programs. Critical evaluation will be augmented by commissioning the principles and rules of American public school law as additional references. Lastly, the conclusion elabo rates on the roles and responsibilities of an evaluator in order to highlight the significa nce of our commitment and vision.Adversary Oriented Evaluation (AOE) Adversary Oriented Evaluation is a rubric encompassing a collection of divergent evaluation practices. In its broadest sense, the te rm refers to all evaluations in which there is planned opposition in the points of view o f different evaluators or evaluation teams, and a planned effort to generate opposing po ints of view within an overall
27 of 38evaluation. In 1965, Guba suggested that educationa l evaluation might use aspects of the legal paradigm. I am suggesting not only to use cer tain aspects of the legal paradigm, but also to use the state and federal statutes as merit criteria for determining the worth and value of educational programs, especially those ins tructional programs that serve disadvantaged students. Next, Worthen, Sanders, and Fitzpatrick ( 1997) presented a provocative rationale for such an approach. If trials and hearings were u seful in judging truth claims concerning patents, products, crimes, civil disobed ience, and if human testimony were judged acceptable for determining life or death, as in the judicial system, then might not legal proceedings and public education law be a use ful metaphor for educational program evaluation? Might there be merit in educati onal evaluation "trials," in taking and cross-examining human testimony, and in using t he concept of advocacy to ensure that evaluation fairly examined both sides of issue s? The first self-conscious effort to follow a particular adversary paradigm was made in the early 1970's by Owens. Designed to test the usefulness of a modified judicial model, the evaluation focused on a hypothetical sch ool curriculum and included pretrial conferences, cases presented by the "defense" and prosecution," a hearing officer, a "jury" panel of educators, charges and rebuttals, d irect questioning and redirected questions, and summaries by the prosecution and def ense (Worthen et al., 1997). The reports (Owens, 1973) were intriguing to the commun ity of evaluators and led to further conceptual and empirical work on the adversary appr oach. For further explanation of the development, applications, strengths, and limitatio ns of this kind of approach see Worthen, Sanders, and Fitzpatrick (1997). Several approaches that qualify as advers ary oriented do not employ hearing processes. Kourilsky and Baker (1976) described an adversary model in which two teams prepared, respectively, affirmative and negat ive appraisals of that which was evaluated (the preparation stage); met to present t he views to one another, cross-examining and critiquing one another's conten tions on prespecified criteria (the confrontation stage); and engaged in open-ended dis cussions until reconciliation of views was attained and translated into written reco mmendations in a single report. Levine (1974) proposed that a resident adversary or critic might be assigned to the research project to challenge each bit of informati on collected, searching for other plausible explanations. The Stake and Gjerde (1974) strategy of having two evaluators prepare separate reports summing up opposing positi ons for and against the program is yet another variant of the adversarial approach tha t does not depend on a hearing format. These proposals are all consistent with what Worthe n et al. (1997) also called "critical evaluation." Donmoyer (n.d) proposed a "deliberative" approach to evaluation, which focused on assessing and balancing alternative conceptions of reality and the differing value positions underlying these conceptions. "Because de liberative evaluation is primarily concerned with fostering understanding of alternati ve conception of reality," the evaluator's role is "to foster interaction and faci litate communication among representatives of various stakeholder groups...." (p.9-10). Donmoyer saw different world-views as the cause of underlying disputes, wh ich could be resolved by open presentation of alternative views in some type of f orum. Worthen et al. (1997) reviewed three gene ral approaches to adversary evaluation: (1) adaptations of the legal paradigm and other "tw o-view" adversary hearings, (2) adaptations of quasilegal and other adversary hea rings where more than two opposing views are considered, and (3) use of debate and oth er forensic structures in adversary evaluation. The third type is particularly interest ing for critical evaluation purposes of
28 of 38establishing merit criteria using the public educat ion laws and codes that can serve as partial "anchors" or references for determining the quality of instructional and educational program delivery. The following is a pr actical representation of how the education laws and codes can be used as partial "an chors" or references. For example, if the instructional effecti veness of programs such as bilingual education or special education was to be evaluated at a predominantly Hispanic low socio-economic elementary school in Texas, the crit ical evaluator could turn to the Texas Law School Bulletin (1996) for crucial inform ation on the state's public education laws and codes that applied to the "Educational Pro grams" (Chapter 29, Subchapters A & B). A critical evaluation could include an invest igation of the history of eligibility, assessment, enrollment, and placement into the bili ngual and special education programs as defined in the Texas Law School Bulletin. Simila r to the study completed by Jia Wang (1998), as mentioned previously in this articl e, the evaluation design would also include investigating the quality of instructional delivery, content coverage, content exposure, and content emphasis (opportunity to lear n variables as described by Jia Wang, 1998). In some instances, if the educational dev elopment of certain disadvantaged students, such as their language proficiency and ac ademic achievement or failure were called into question, the evaluation team could rev iew carefully the student's educational history by comparing it to the eligibility criteria assessments, enrollment, and instructional placement education codes as set out by the Law Bulletin. These education codes could be the "anchors", the starting points o r references to further the understanding of current and past campus and distri ct based educational practices that involve high risk decision making. Education code 2 9.056, Enrollments of Students in Program is an example of this kind of "anchor" or r eference: The agency shall establish standardized criteria fo r the identification, assessment, and classification of student of limite d English proficiency eligible for entry into the program or exit from th e program. The student's parent must approve a student's entry into the prog ram, exit from the program, or placement in the program. The school di strict or parent may appeal the decision under Section 29.064 (p. 120). Again, the laws and codes can be used as additional references for the evaluators to place classroom instruction, the school, the progra m, or the school district, in context to legal precedent and required administration. Becaus e a public school is a governmental agency, its conduct is circumscribed by precedents of public administrative law supplemented by those legal and historical traditio ns surrounding educational organization that is state established, yet locally administered. In this setting legal and educational structural issues must be considered th at define the powers to operate, control, and manage the schools (Alexander & Alexan der, 1992). In analyzing the American educational sys tem and comparing it to central state systems of education in foreign countries, one is s truck by the diversity of authority under which the American public schools are governe d. As a federal and not a national system, the government of the United States compris es a union of states united under one central government. The particular form of Amer ican federalism creates a unique educational system, which is governed by laws of fi fty states with component parts amounting to several thousand local school district operating units. Through all of this organizational multiformity, and indeed complexity, runs a legal basis on which the entire system is founded, those generally prescribe d by our constitutional system.
29 of 38 The critical position of education in a d emocratic society is self-evident. Over the years the courts have come to conclude that society is best served by an educational system that teaches "through wide exposure to that robust exchange of ideas which discovers truth out of multitude of tongues. Thus b ecause of the importance of the schools and because this robust exchange of ideas i s vital to the educational process, the perpetuation of that exchange is, at all levels of the educational system, a special concern of the First Amendment" (Alexander & Alexander, 199 2, p.229). No school can function appropriately as a place for the exchange of ideas unless both students and faculty enjoy an atmosphere conducive to debate and scholarly inq uiry. With this in mind, the reform of public e ducation which includes the improvement of educational programs for those children who are least well off, should remain open to alternative views and divergent conceptions of eval uation. Critical evaluation can begin to provide an accurate analysis of the production o f inequality or the reproduction of social injustice in the public schools. The ideolog y of critical evaluation can begin to influence a movement toward the realization of an e galitarian ideal and the elimination of inequality. I have asked educators and evaluator s of education reform efforts to reconsider critically their roles in social science research, to reclaim the battleground of public school reform by focusing on the democratic purpose of public schooling, and the institutional problems in educational programs and practice that often inhibit action toward this ideal.Conclusion Irrespective of the many social, economic technological, cultural, and political problems that face our American communities, the pu blic schools exist for the purpose of educating all children. Teachers are a part of t he never-ending struggle to create conditions in which learning takes place and provid e the best educational opportunities in a given situation. As evaluators rendering judge ment on educational programs, and giving merit or not giving merit to the educational repertoires and learning outcomes of teachers; we also become inextricably linked to the process of either perpetuating an already neutral disconnected reality of education o r critically examining and observing a wide range of crucial issues, structures, and probl ems in contemporary education. As evaluators of education programs and teaching, we c annot ignore that we become a part of the never-ending struggle to make judgment calls about a social activity which creates the conditions or obstacles for social mobility. The central task of the current reform mo vement in education is nothing less than building and transforming schools that are struggli ng to achieve democratic ideals (Fine, 1994). While schools can be described as potentiall y a site of extraordinary democracy, the processes and outcomes of schools deeply reprod uce and promote the very social inequities they are said to equalize (Fine, 1983). This circumstance imposes onto the roles of educational leaders and critical evaluator s a social responsibility, one that demands sincere conscience and deliberate action. E valuators and researchers, who in the past have been content to describe dispassionat ely what schools are doing and how they are functioning, are actually involved in and committed to a collaborative view of knowledge creation. These characters in social chan ge should not struggle to find a voice that sensitively captures both the insider's and ou tsider's view of reality. When characters, such as evaluators of educational refor m, gain the conscience and purposefulness of their critical role, no relations hip is left untouched or unchanged. In conclusion, evaluation is a powerful social forc e that has evolved only
30 of 38recently in advanced capitalistic societies, a new institution that promises to be a major influence over the long term. Its influe nce can be both good and bad. In either case, society before formal evaluati on is not the same as society afterward. Exactly what shape the practice, institution, profession, and discipline will take in the future is impossibl e to predict. What is clear is that its fate will be bound to the government an d the economic structure and determined in part by its own history and tradi tions. Some of its destiny lies within the control of the evaluators themselve s; some does not (House, 1993, p.172). Note. Paper presented at the National Evaluation Co nference, Youngstown State University, Youngstown, Ohio, September, 1998ReferencesAlexander, K., & Alexander, M. D. (1992). American public school law (3rd ed.). St. Paul, MN: West Publishing Company.Apple, M. W. (1996). Cultural power and education New York: Teachers College Press.Apple, M., and Beyer, L. (Eds.) (1988). The curriculum: Problems, politics, and possibilities Albany: State University of New York Press. Apple, M. W. & Beane, J. A. (Eds.) (1995). Democratic education Washington, D.C.: Association for Supervision and Curriculum Developm ent. Apple, M. W. & Carlson, D. (Eds.) (in press). Critical educational theory in unsettling times Boulder, CO: Westview Press. Arreaga-Mayer, C., & Greenwood, C.R. (1986). Enviro nmental variables affecting the school achievement of culturally and linguistically different learners: An instructional perspective. NABE: The Journal for the National Association for Biling ual Education, 10 (2), 113-135. Barth, R. S. (1986). On sheep and goats and school reform. Phi Delta Kappan, 68 (4): 293-296.Berliner, D.C., & Biddle B. J. (1997). The manufactured crisis: Myths, fraud, and the attack on America's public schools White Plains, NY: Longman Publishers USA. Birdsall, N., & Hecht R. (1995). Swimming against the tide: Strategies for improving health. Working Paper No. 305, Washington, D. C.: Inter-Ame rican Development Bank, Office of the Chief Economist.Burke, K. (1966). Language as symbolic action: Essays on life, litera ture, and method Berkeley: University of California Press.Coleman, J. (1975). Racial segregation in the schoo ls: New research with policy implications. Phi Delta Kappan, 57 October 1975, 75-78. Coleman, J. (1981). Quality and equality in America n education: Public and catholic
31 of 38schools. Phi Delta Kappan, 63 159-164. Chelimsky, E., & Shadish, W.R. (Eds.) (1997). Evaluation for the 21st century: A handbook Thousand Oaks, CA: SAGE Publications. Cousins, J. B, Donahue, J. J., & Bloom, G. (1996). Understanding collaborative evaluation: Results from a survey of North American evaluators Unpublished paper submitted for publication, University of Ottawa.Cronbach, L. J., & Associates. (1980). Toward reform of program evaluation San Francisco, CA: Jossey-Bass.Dewey, J. (1944). Democracy and education New York: The Free Press. Donmoyer, R. (n.d.). Evaluation as deliberation: Theoretical and empiric al explorations (Grant No. G 810083). Ohio State University, Colu mbus, National Institute of Education.Fetterman, D. (1981). Blaming the victim: The probl em of evaluation design and federal involvement, and reinforcing world views in educati on. Human Organization, 40 (1), 67-77.Fetterman, D. (Ed.). (1988). Qualitative approaches to evaluation in education: The silent scientific revolution New York, NY: Praeger Publishers. Fetterman, D. & Pitman, M. A. (1986). Educational evaluation: Ethnography in theory, practice, and politics Beverly Hills, CA: SAGE Publications. Fine, M. (1983). Perspectives on inequality: Voices from urban schools. In L. Bickman, ed. Applied Social Psychology Annual IV Beverly Hills: SAGE Publications. Fine, M. (1983b). Dropping out of high school: The ideology of school and work. Journal of Education, 165 259-272. Fine, M. (1988). De-institutionalizing educational inequity. In Council of Chief State School Officers. (Eds.). At risk youth: Policy and Research New York: Harcourt Brace Jovanovich.Fine, M. (Ed.). (1994). Chartering urban school reform: Reflections on publ ic high schools in the midst of change New York: Teachers College Press. Fine, M., & Weis, L. (Eds.). (1993). Beyond silenced voices: Class, race, and gender in United States schools Albany, NY: State University of New York Press. Freire, P. (1970b). Cultural action and conscientiz ation. Harvard Educational Review, 40 (3), 452477. Freire, P. (1973). Education for critical consciousness New York: Seabury. Freire, P. (1985). The politics of education South Hadley, MA: Bergin and Garvey. Giroux, H. A. (1997). Pedagogy and the politics of hope: Theory, culture, and schooling Boulder, CO: Westview Press.
32 of 38Giroux, H. A. (1983). Theory and resistance in education: A pedagogy for the opposition South Hadley, MA: Bergin and Garvey. Giroux, H. A. (1983b). Critical theory and educational practice Geelong: Deakin University Press.Giroux, H. A. (1988). Teachers as intellectuals: Toward a critical pedago gy of learning South Hadley, MA: Bergin and Garvey.Giroux, H. A. & Aronowitz. S. (1991). Postmodern education: Politics, culture, and social criticism Minneapolis, MN: University of Minnesota Press. Gross, S. (1993). Early mathematics performance and achievement: Results of a study within a large suburban school system. Journal of Negro Education, 62 (3), 269-287. Guba, E. G. (1965). Evaluation in field studies Address at evaluation conference sponsored by the Ohio State Department of Education Columbus. Guba, E. G., & Lincoln, Y. S. (1989). Fourth generation evaluation Newbury Park, CA: SAGE Publications.Habermas, J. (1971). Knowledge and human interest Translation. Boston: Beacon Press. Habermas, J. (1981). The theory of communicative action, volume 1: Reaso n and the rationalization of society Boston: Beacon. Habermas, J. (1987). The theory of communicative action, volume 2: Lifew orld and system, a critique of functionalist reason Boston: Beacon. Habermas, J. (1990). Moral consciousness and communicative action Cambridge: MIT Press.Hamnett, M. P., Kumar, K., Porter, D. J., & Singh, A. (1984). Ethics, politics, and international social science research: From critiqu e to praxis East-West Center: University of Hawaii Press.House, E. R. (1976). Justice in evaluation. In G.V. Glass (Ed.), Evaluation studies review annual (Vol. 1, pp.75-100). Beverly Hills, CA: SAGE Publi cations. House, E. R. (1980). Evaluating with validity Beverly Hills, CA: SAGE Publications. House, E. R. (1988). Jesse Jackson and the politics of charisma: The ris e and fall of the PUSH/Excel program Boulder, CO: Westview. House, E. R. (1990). Methodology and justice. In K. A. Sirotnik (Ed.), New directions for program evaluation, Vol.45 (pp. 23-36). San Francisco: Jossey-Bass. House, E. R. (1993). Professional evaluation Newbury Park, CA: SAGE Publications. Howard, G. S. (1985). The role of values in the sci ence of psychology. American Psychologist, 40 255-265.
33 of 38Kanpol, B. (1997). Issues and trends in critical pedagogy Cresskill, NJ: Hampton Press. Kohlberg, L. (1984). Essays on moral development: The psychology of mora l development Vol.2. New York: Harper & Row. Kourilsky, M. & Baker, E. (1976). An experimental c omparison of interaction, advocacy, and adversary evaluation. Center on Evaluation, Development, and Research (CEDR) Quarterly, 9 4-8. Kozol, J. (1991). Savage Inequalities New York: Crown Publishers. Kretovics, J., Farber, K., & Armaline, W. (1991). R eform from the bottom up: Empowering teachers to transform schools. Phi Delta Kappan, 73 (4), 295-299. Kretovics, J., & Nussel, E. J. (1994). Transforming urban education Needham Heights, MA: Allyn and Bacon.Kuhn, T. (1970). The structure of scientific revolutions 2nd Ed. Chicago: University of Chicago Press.Levine, D. U. (1971). Concepts of bureaucracy in ur ban school reform. Phi Delta Kappan, 52 329-333. Levine, M. (1974). Scientific method and the advers ary model. Some preliminary thoughts. American Psychologist, 29 661-677. Lincoln, Y. S. (Ed.). (1986). Organizational theory and inquiry: The paradigm revolution Newbury Park, CA: SAGE Publications. Lincoln, Y. S. (1991). The arts and sciences of pro gram evaluation. Evaluation Practice, 12 (1), 1-7. Madaus, G. F., West, M.M., Harmon, M.C., Lomax, R G ., & Viator, K.A. (1992). The influence of testing on teaching math and science i n grades 4-12 Boston: Boston College Center for the Study of Testing, Evaluation and Education Policy. Maruyama, G. M., & Deno, S. (1992). Research in educational settings Newbury Park, CA: SAGE Publications.McCormick, L., Haring, N. G., & Haring, T.G. (1990) Exceptional children and youth (6th ed.). New York, NY: Macmillan Publishing Compa ny. McLaughlin, M. W. (1975). Evaluation and reform: The elementary and secondary education act of 1965 Cambridge, MA: Ballinger. MacPherson, C. B. (1987). The rise and fall of economic justice Oxford, UK: Oxford University Press.Miller, S. I., and Safer, L. A. (1993). Evidence, e thics, and social policy dilemmas. Education Policy Analysis Archives, 1 (9) (Available online at http://epaa.asu.edu) Murray, C. A. (1983). Stakeholders as deck chairs. In A. Bryk (Ed.), New directions for
34 of 38program evaluation: Vol. 17 Stakeholder-based evaluation (pp.58-61). San Fran cisco: JosseyBass.Murray, C. A. (1984). Losing ground: American social policy, 1950-1980 New York: Basic Books.National Council on Education Standards and Testing (NCEST). (1992). Raising standards for American education: A report to Congr ess, the secretary of education, the national education goals panel, and the American pe ople Washington, D.C.: National Council on Education Standards and Testing.Nevo, D. (1986). The conceptualization of education al evaluation: An analytical review of the literature. In E. R. House (Ed.), New directions in educational evaluation .(pp. 15-29). London, UK: Falmer Press.Noll, J. W. (1997). Taking Sides: Clashing views on controversial educa tion issues (9th ed). Guilford, CT: McGraw-Hill.Oakes, J. (1986). Tracking, inequality, and the rhe toric of reform: Why schools don't change. Journal of Education, 168 (1), 60-79. Oakes, J. & Sirotnik, K. A. (Eds.). (1986). Critical perspectives on the organization and improvement of schooling. Hingham, MA: Kluwer-Nijhoff. Oakes, J., & Guiton, G. (1995). Opportunity to lear n and conceptions of educational equality. Educational Evaluation and Policy Analysis, 17 (3), 323-336. Ogbu, J. U., & Matute-Bianchi, M. E. (in press). Un derstanding socio-cultural factors: Knowledge, identity, and school adjustment. In Cali fornia State Department of Education (Ed.), Socio-cultural factors and minority student achieve ment Sacramento: Author.Owens, T. R. (1973). Educational evaluation by adve rsary proceeding. In E. R. House (Ed.), School evaluation: The politics and process Berkeley, CA: McCutchan. Patton, M. Q. (1994). Developmental evaluation. Evaluation Practice, 15 311-319. Patton, M. Q. (1997). Utilization Focused Evaluation (3rd ed.). Thousand Oaks, CA: SAGE Publications.Paul, S. (1991). Accountability in public services: Exit, voice, and capture Washington, D.C.: The World Bank.Prilleltensky, I. (1994). The morals and politics of psychological discourse and the status quo Albany: State University of New York Press. Rawls, J. (1971). A theory of justice Cambridge, MA: Belknap Press. Rossi, P. H, & Freeman, H. E. (1993). Evaluation: A systemic approach (5th ed.). Newbury Park, CA: SAGE Publications.Sabia, D. R., Wallulis, J. (Eds.). (1983). Changing social science: Critical theory and
35 of 38other critical perspectives Albany, NY: State University of New York Press. Schn, D. A. (1983). The reflexive practitioner: How professionals think in action New York: Basic Books.Scriven, M. (1986). Evaluation as a paradigm for ed ucational research. In E. House (Ed.), New directions in educational evaluation (pp.53-67). Philadelphia, PA: Farmer Press.Scriven, M. (1991). Evaluation thesaurus (4th ed.). Newbury Park, CA: SAGE Publications.Scriven, M., & Kramer, J. (January 1995). Risks, ri ghts, and responsibilities in evaluation. Australasian Journal of Evaluation 15 15-19. Sechrest, L. (1992). Roots: Back to our first gener ations. Evaluation Practice, 13 (1), 1-7.Shadish, W. R. Jr., & Epstein, R. (1987). Patterns of program evaluation practice among members of the evaluation research society and eval uation network. Evaluation Review, 11 555-590. Shapiro, H. S., & Purpel, D. E. (1998). Critical social issues in American education. (2nd Ed.). Mahwah, NJ: Lawrence Erlbaum Associates.Sirotnik, K. A. (Spring, 1990). Evaluation and soci al justice: Issues in public education. In K. A. Sirotnik (Ed.), New directions for program evaluation, Vol.45 San Francisco: Jossey-Bass.Sirotnik, K. A. & Oakes, J. (Spring, 1990). Evaluat ion as Critical Inquiry: School improvement as a case in point. In K. A. Sirotnik ( Ed.), New directions for program evaluation, Vol.45 Evaluation and social justice: Issues in public e ducation. (pp. 37-59). San Francisco: Jossey-Bass.Sizer, T. (1984). Horace's compromise: The dilemma of the American hi gh school. Boston, MA: Houghton Mifflin.Stake, R. E. (1986). Quieting reform: Social science and social action i n an urban youth reform Champaign: University of Illinois Press. Stake, R. E., & Gjerde, C. (1974). An evaluation of TCITY, The Twin City Institute for Talented Youth. In R. H. P. Kraft, L. M. Smith, R. A. Pohland, C. J. Brauner, & C.Gjerde (Eds.), Four evaluation examples: Anthropologist, economic, narrative, and portrayal (AERA Monograph Series on Curriculum Evaluation No 7). Chicago: Rand McNally.Tsoi Hoshmand, L. L. (1994). Orientation to inquiry in a reflective professional psychology Albany: State University of New York Press. Texas Education Agency. (1996). Texas School Law Bulletin Austin, TX: West Publishing Company.
36 of 38Vervoort, C. E. (1975). Onderwijs en maatschappelij ke ongelijkheid als verdelingsprobleem (Education and social inequality as a distribution problem). In P. van der Kley & A. A. Wesselingh (Eds.). Onderwijs en maatschappelijke ongelijkheid. Boekaflevering Mens en Maatschappij, Jrj, 50. Rotte rdam: UPR. Visions of public service. (1986, Fall/Winter). JFK School of Government Bulletin. Warren, R. L. (1963). Social research consultation New York: Russel Sage. Wertsch, J. V. (1998). Mind as action Oxford: Oxford University Press. Wesselingh, A. (1997). Spheres of justice: The case of education. International Studies in Sociology of Education 7(2), 181-194. Wexler, P. (1983). Critical social psychology Boston, MA: Routledge & Kegan Paul. Winfield, L. F. (1993). Investigating test content and curriculum content overlap to assess opportunity to learn. Journal of Negro Education, 62 (3), 288-310. Worthen, B. R., Sanders, J. R., & Fitzpatrick, J. L (1997). Program evaluation: Alternative approaches and practical guidelines (2nd ed.). White Plains, NY: Longman Publishers.Young, R. E. (1990). A critical theory of education: Habermas and our ch ildren's future. New York: Teachers College Press Note: I would like to thank Dr. Patricia Whang and Dr. C ynthia Reed for their advice and support during the writing of this article.About the AuthorGisele A. WatersEducational Foundations Leadership and Technology3084 Haley CenterCollege of EducationAuburn UniversityAuburn, AL 36849Email: firstname.lastname@example.org Education:Student studying for Ph.D. in Educational Psycholog y M.Ed. Special Education/Educational Psychology, 199 6. University of Houston B.A. Economics, 1990. University of Texas at AustinCertification: Texas; Generic Special Education (K12), Bilingual and ESL Education, Elementary Education Working in Texas as a public school teacher for six years in bilingual/ESL
37 of 38 education classrooms, and special education resourc e significantly influenced my critical analysis of the educational process. During that ti me, I held delegate positions of leadership on district and campus advisory teams, s haring the responsibility in the development and implementation of district and camp us level improvement plans. The obstacles encountered to implement research based b est practices carefully positioned my observations and questions about teaching and le arning. Ultimately, my heart lies with teachers and children and the institutional pr essures that affect them. Today my teaching at the undergraduate level reflects an ins tructional approach that frames my passions for issues of social justice, democracy, p ower, voice, and equity in schools and schooling. My methodological interests lie in the c ultivation of a critical social science, a science intended to empower those involved to chang e as well as to understand their world.Copyright 1998 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is http://olam.ed.asu.edu/epaa General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, email@example.com or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. (602-965-26 92). The Book Review Editor is Walter E. Shepherd: firstname.lastname@example.org The Commentary Editor is Casey D. Cobb: email@example.com .EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Andrew Coulson firstname.lastname@example.org Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov email@example.com Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Richard M. Jaeger University of North Carolina--Greensboro Daniel Kalls Ume University Benjamin Levin University of Manitoba
38 of 38 Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Western Interstate Commission for Higher Education William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX) Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton firstname.lastname@example.org Hugh G. Petrie SUNY Buffalo Richard C. Richardson Arizona State University Anthony G. Rud Jr. Purdue University Dennis Sayers Ann Leavenworth Centerfor Accelerated Learning Jay D. Scribner University of Texas at Austin Michael Scriven email@example.com Robert E. Stake University of Illinois--UC Robert Stonehill U.S. Department of Education Robert T. Stout Arizona State University
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c19989999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00111
Educational policy analysis archives.
n Vol. 6, no. 20 (November 03, 1998).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c November 03, 1998
Critical evaluation for education reform / Gisele A. Waters.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)