USF Libraries
USF Digital Collections

Educational policy analysis archives


Material Information

Educational policy analysis archives
Physical Description:
Arizona State University
University of South Florida
Arizona State University
University of South Florida.
Place of Publication:
Tempe, Ariz
Tampa, Fla
Publication Date:


Subjects / Keywords:
Education -- Research -- Periodicals   ( lcsh )
non-fiction   ( marcgt )
serial   ( sobekcm )

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
usfldc doi - E11-00405
usfldc handle - e11.405
System ID:

This item is only available as the following downloads:

Full Text
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 12issue 56series Year mods:caption 20042004Month October10Day 1717mods:originInfo mods:dateIssued iso8601 2004-10-17


EDUCATION POLICY ANALYSIS ARCHIVES A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright is retained by the first or sole author, who grants right of first publication to the Education Policy Analysis Archives EPAA is a project of the Education Policy Studies Laboratory. Articles are indexed in the Directory of Open Access Journals ( Volume 12 Number 56 October 17, 2004 ISSN 1068-2341 Making Research Matter More Ben Levin University of Manitoba Citation: Levin, B. (2004, October 17 ). Making research matter more. Education Policy Analysis Archives, 12 (56). Retrieved [date] from ht tp:// Abstract Interest in strengthening the impact and value of education research has been growing around the world. Here I outline a view of the nature of “impact” and point to instances where research has had a positive impact in education, but always within a larg er social and political framework. A three element “model” of research impact is developed and used as the basis to assess current situations and to suggest steps that could be taken to support a fuller contribution to education and learning from research. Growing Interest in Research Interest in strengthening the impact and value of education research has been growing around the world, among governments and funders as well as scholars. The issue of research impact has also been a prominent theme in Educational Researcher (e.g. Berliner, 2002, Feuer, Towne & Shavelson, 2002). Th is paper outlines a view of the nature of “impact,” examines initiatives in this area in several countries, and su ggests steps that could be taken to support a fuller contribution to education and learning from research. However it should be clear from the outset that the contribution of research is always mediated through broader social and political processes with all their attendant limitations. Nor do I address in this paper some very important re lated questions, such as what issues are most important to study, or who should do research and what training researchers should receive.


Making research matter more 2 The term “research” is itself contested an d can cover quite a wide range of activities, from carefully designed studies by independent, university-based researchers to analysis of data for particular administrative or political purposes to arguments for specific policy positions that may be more or less well ground ed in evidence. Any consensus that might once have existed about what counts as resear ch has vanished in education, with highly contentious arguments about the relative merits of research based on methods from the natural sciences vs modes closer to the humanities. I am not in this paper endorsing or promoting a particular view about what consti tutes “good” research in education; many different forms of research can have an im pact in the senses defined in this paper. The debate about the value of education re search is a long-standing one but appears to have been growing in intensity in recent years. Increasing interest in research is driven by a more educated population and a growing awareness of the need to understand more fully the complex problems confronting us (Homer-Dix on, 2000). Research conjures up images of science and of objectivity. Governments want to claim that their policies are supported by evidence. Increasing requirements for accountability for public spending also put greater emphasis on evidence.1 The media give more mention to research even if the reporting may not always be as careful as might be wished. The phenomenal growth of the internet and its growing use by a wide range of people as a so urce of information is another illustration of this interest. Education, though these larger trends do apply to it, also has some particular characteristics that affect the role that resear ch can play. It is a value-laden activity, inextricably connected to our broadest aspirati ons for society. It embodies a wide range of purposes that are not always mutually cons istent. People may agree on educational goals only at the most general level, with many conflicts not only about goals but about the best means of carrying them out. Many disagreem ents about education research are actually differences over the substance of education policy. Education also has less history of basing policy and practice on research than do some ot her fields. The fact that everyone has gone to school, the relatively low status of teachi ng as an occupation, the recent arrival of education as a field of study in the university, and the lack of a clear disciplinary base, or rather the multiple disciplinary contributions to the field (Lagemann, 2000) are all factors. What is impact? Although research impact is very topical, “There is a surprisingly small literature on the impact of educational research on policy and practice” including an “underdeveloped language in which to explore the topic” (Nat ional Education Research Forum, 2000, 1). In this article I use a variety of terms such as “ use,” “impact” and “value” to stand for the relationship and contribution of research to othe r spheres of life. None of these terms is entirely satisfactory, but we lack a better vocabulary. Impact occurs when research, in any of its multiple forms, makes a difference to subsequent actions that people take or refrain fr om taking. This is a simple statement but a full discussion of its meaning would take this na rrative off course (see Lavis et al. 2002, for a discussion in the context of health research). Re search may be used by people in their public or private lives, or by organizations of all kind s. People may or may not be aware that their ideas and actions have been shaped by research. 1 I am indebted to Jane Kenway, Monash University, Australia for this point.


Education Policy Analysis Archives Vol. 12 No. 56 3 Most importantly, research is only one influence on human action; and, as described more fully later, its impact is always mediated by larger social and political processes. Thus research impact is not always a good thing (Levacic & Glatter, 2001, Willinsky, 2000). Research has been used to support positions th at were later shown to be wrong or, even worse, are now considered morally repugnant, such as the supposed inferiority of some groups of people. In some ways it might be more appropriate to use the more neutral term consequences of research rather than impact.2 Why so much criticism of education research? Reports or position papers bemoaning the lack of value of research in education have appeared in a number of countries over th e years, with a new spate more recently. The complaints are familiar ones – research in educ ation is of poor quality, it does not address the issues people in the field care about, it is equivocal in its findings, it does not provide sufficient guidance to policy or practice, it is not timely. These critiques are being advanced in many countries. In Britain, a very lively debate was provoked a few years ago by a lecture by David Hargreaves (1996) comparing education research and medical research to the detriment of the former, leading to a government commission on the subject (Hillage et al., 1998). France (Prost, 2001) and Australia (McGaw et al., 1992) have had similar public reviews. In the United States, there have also been substantial attacks on the value of education research (Coalition for Evidence-Base d Policy, 2002; Gardner, 2002), with the suggestion that the research is of poor quality and does not affect policy or practice. A website closely linked to the U. S. Department of Education (www. notes: “Our nation’s failure to improve its schools is due in part to insufficient and flawed education research. Even when rigorous research exists, solid evidence rarely makes it into the hands of practitioners, policy-makers and othe rs who need it to guide their decisions.” The criticisms seem stronger than is warranted by the evidence on the impact of research in education. There are many inst ances where research has played an important role in shaping policy and practice. Consider our growing knowledge about the importance of the development of very young children (e. g. Barnett, 1996; Haveman & Wolfe, 1994). In the United States, this evidence has played an important role in creating and sustaining programs such as Head Start and Early Start. In Canada, it was instrumental in a 1999 federal-provincial agreement to expand dramatically public investment in early childhood development. In England, it has also been important in shaping the policy agenda of the government in a variety of ways. Consider a second example: research on school improvement (e.g Fullan, 2001; Hopkins, 2001). The practice of school reform, especially at the local level, increasingly reflects our understanding that meaningful refo rm must build educators’ capacity and pay attention to the whole school context. This is a dramatic change from the top-down, teacher-proof strategies of the 1960s. It wo uld be easy to cite other examples, such as growing attention to working with parents and families or the move of children with disabilities from segregated settings into regu lar schools and programs. Furthermore, studies in the United States (Biddle & Saha, 2002) and in Australia (DETYA 2000) found that very 2 I am indebted to Sally Power, Ca rdiff University, for this point.


Making research matter more 4 large majorities of educators and policy-makers thought that their work was actively informed by research, though largely in a variety of indirect ways. If research in education has in fact had a significant impact, why is there so much criticism of it? One reason is that there ar e many important questions of education policy and practice where research provides little guid ance. Much of education is concerned with producing significant and lasting change in how people think or behave, yet on the whole we do not yet know very much about how to do this, either in schools or in other settings. Policy-makers are often faced with difficult al ternative choices around how to use resources; again, research often has little to say about wh at choices are best. There are good reasons, conceptual and practical, why this is so – to mention two, the issues are often very complex and the total education research effort is compar atively small – but the lack of clear direction is understandably frustrating for users. People also may have strong a priori views about how education ought to be conducted, and may become frustrated when research does not support those views. Much of the debate about research in the United States is more about substantive positions on issues than it is about research methods. A second basis for criticism lies in the frequent assumption that there should be a direct line between research and subsequent poli cy and practice such that research findings point unambiguously to what governments, educ ators or learners should do. This line of thinking appears to lie behind some of the recent direction of the U. S. government in education. Practitioners are also often looking for immediate, clear and unambiguous direction as to what to do. The difficulties with a direct transmissi on view of research impact have been well described elsewhere (e.g. Stone, 1997; Weiss, 19 79). They range from utilitarian concerns to philosophical objections. Knowledge of what to do does not translate directly into policy or practice because these latter are shaped as much or more by social, cultural and political considerations as they are by particular kind s of formal knowledge (Lindblom, 1990; Lavis et al., 2002). Policy makers may face political impediments that make it impossible or undesirable to act. Practitioners may be deep ly enmeshed in practices and beliefs that are highly resistant to change. In health we are learning how hard it is to change physician practices even when evidence seems compelling; in education we are already well aware of how hard it is to change teaching practices on a large scale (Fullan, 2001). On the conceptual side, a considerable body of work has shown how knowledge about human behaviour is in principle different from knowledge of the inanimate world, so that the results of research cannot be assumed to be universal and generalizable as are, say, the laws of chemistry (Bernstein, 1976). This is primarily because people are intentional actors who can and do change their actions as their understanding of the situation changes. (For a wonderful example of this point in the co ntext of research itself, see Orne, 1962). As a result few if any findings about human behavi our will apply under all conditions, nor is it easy, if even possible, to specify the precise conditions under which a given generalization will hold. These complexities are a main reason that mu ch research in social sciences produces probabilities – the chance that something mi ght happen given certain other conditions – rather than certainties (Willinsky, 2000). Howev er assessing probabilities is a tricky business and often leads to conflict over what resear ch results imply for subsequent action (e.g. Powell & Leiss, 1997). Social science know ledge tends to be tentative and contextual, whereas users want certainty.


Education Policy Analysis Archives Vol. 12 No. 56 5 These are real and powerful limitations on the impact of research. They draw our attention to the importance of power relationshi ps, political dynamics, human biases and all the other factors that shape what people do (Dror, 1986). It is not reasonable to think that research will be the primary driver of action ex cept in very rare circumstances. It does not follow, however, that we must despair about the impact of research, as our earlier examples show. These stories of impact provide a better idea of how and when research does make a positive contribution. How impact happens A number of scholars have shown that where research has impact, it occurs over extended periods of time (Weiss, 1979; Willinsky, 2000). As is clear from the earlier examples in education or others such as the dang ers of smoking or the virtues of exercise, it may take decades for research find ings to have real impact. Nor does this impact usually occur through direct contact between a researcher and a decision-maker. Instead, users typically lear n about research through various third party mechanisms as part of larger social and po litical processes. The media, both mass and professional, play a key role in bringing idea s to people’s attention. People often rely on popularizers – people who use research as part of their work in writing and speaking with educators and the public. There is a large industry in education in the area of professional development, much (though certainly not all) of which is informed by current research. Politicians and officials often learn about rese arch from lobbyists and interest groups who use it to advance their political views. A variety of other bodies – think tanks, foundations, professional organizations – are also involved in research dissemination. Another important role can be played by “policy entrepreneur s” (Mintrom, 2000) – people who set out to advance a particular cause and use research as part of that crusade. Of course all these parties have their own reasons for being inter ested in research, which are not necessarily benign. Although third parties use research for a range of reasons, these mechanisms are the main way in which new ideas from research pen etrate existing policy and practice. But these ideas are also filtered by users of through their own interests and beliefs. Each field of human activity – education, health, justice, and so on – has its own set of practices, habits and accepted wisdom. People who work or live in one of these worlds do not tend to see themselves as deficient even though they ma y recognize areas where they do not know how to meet a challenge or achieve a goal. Teachers are immersed in the work of teaching, or bureaucrats in managing government programs work that they feel they understand and know how to do, even if imperfectly. Resear ch affects their practi ce only as they become convinced that the ideas or practices suggested will actually improve their work or their lives in some way (Cordingley, 2000; DETYA 2000). A considerable body of research on change and innovation (e.g. Fullan, 2001; Rogers, 1995) helps us understand the complexities of these processes. Once this dynamic is recognized, several other points follow. The most important is that the use of research is embedded in a set of personal and organizational beliefs and practices that are complicated and often deeply entrenched. These can include personal as well as organizational goals; the standards, policies and culture of the organization or occupation; the practical tasks that confront people every day; and of course personal


Making research matter more 6 predispositions and beliefs, affected as these ar e by people’s biographies, personal as well as occupational. From this perspective we can see why it is so hard to change people’s behaviour and organizational practices. The power of the fo rces of stability is why we have so many examples of changes that seemed completely obv ious being resisted for long periods of time, such as giving fresh fruit and vegetables to sailors on long voyages to prevent scurvy or washing hands in medical care to prevent infecti on, or wearing seatbelts as a way of reducing deaths and injuries in automobile accidents. The same forces apply to individuals. Knowing that seat belts save lives or that smoking is a health hazard has an effect on behaviour but not immediately and not for everyone. In the public arena – the world of governments and most large organizations – yet another set of constraints on change arises from political processes (Levin, 2001). Organizations that are in the public eye – governments, of course, but also many other large organizations – are inevitably sensitive not only to the views of their internal participants but also to larger political currents. One of the rule s of the political world is that what is true is far less important than what most people belie ve to be true. A government may be in a position where it is caught between what it belie ves is the best course of action and what it believes is publicly acceptable. In a world where public acceptance is the key to survival it is easy to predict which interest will dominate. It is only when public beliefs shift that governments will feel required or able to move. In addition to sector structures, the larger social and intellectual context of society is also important. What gets accepted as knowledge is influenced by a larger climate of ideas and conventional wisdom. The growing interest in research and evidence described earlier can also be seen as part of this larger con text. For example, if one accepts the idea of increasing uncertainty and reflectivity in soci ety (e.g. Giddens, 1994), one might see an interest in research as part of a larger search for new sources of truth to compensate for the decline of religious belief or the decline in c onfidence in most of our institutions (Stone, 1997). There could then be a danger that what is presented as scientific gets uncritical acceptance, a possibility that carries its own risks (Willinsky, 2003). These issues are much larger than can be dealt with here, but th eir importance needs to be acknowledged. The point is that the role of research and evidence in any setting will inevitably be mediated by other pressures and demands as well as by existing habits, practices and cultures. Research alone will not overcome strong ly held biases about race or gender. It will not unilaterally convince people to change l ong-standing power relationships. However, with care and over time it can play an impor tant role in changing what people think and what they do. People, policies and practices do ch ange in response to a variety of influences. Recognizing the social and political dimension of research impact and the importance of mediating mechanisms opens new possibilities, for it sug gests that research impact can happen precisely through these same so cial and political forces. Seeing research impact in this longer-term and more diffuse way raises possibilities for building bridges between research (and researchers) and other social processes.


Education Policy Analysis Archives Vol. 12 No. 56 7 A Model of Impact Growing out of the discussion so far, I sugge st that research impact should be seen as having three elements: 1) The context of research production, including what research gets done, who does it, how it is done, what communication activities are undertaken, and so on. Research production is largely located in universities but also takes place in a variety of other organizations. 2) The context of research use, including thos e settings that have an interest in the application of research. This context includes governments, educational organizations of all kinds, teachers of all kinds, and also parents, students and a variety of other community groups with an interest in education. Of interest here are the views, capacities and structures through which such organizations are able – or limited in their ability – to find, understand and use research. 3) The connections and interactions between th e two other contexts, involving all kinds of direct and mediated, face to face, print, electronic and other vehicles, formal and informal. These are the strands of c onnection – some strong, others quite weak – between the two other contexts through which enhanced impact must come over time. While some connections are directl y between researchers and users, most of the connection happens through third party me diation. The connections also run to varying degrees in both directions; that is, research production and mediation are also influenced by contexts of use. These elements are illustrated in Figure 1, which is less a model than an attempt to represent some of the factors and linkages related to research impact. As suggested in the figure, the contexts of use and of mediation ar e larger in size than the research production effort, and the whole enterprise is situated in a larger social context that is itself constantly changing. The diagram should also remind us that the actions of researchers, while important, are only one part of the effort to affect ideas and social practices. The diagram also may overstate the degr ee of separation between the context of production and the context of use. In fact, many people and organizations work in both contexts. Individuals may move back and forth between research posts and educational practice (keeping in mind that most universi ty faculty are also teaching practitioners). Practitioners who are also graduate students pr ovide an important potential bridge (DETYA 2000). Organizations may be involved in research production as well as its use. At the same time, the term “contexts” sugge sts that it is the task and setting more than the person that affects how the work is done. Researchers who move to practice may not themselves be great users of research in their new roles because their new organization does not provide the means or lacks capacity to use the results. The parts of organizations that generate research may find themselves at odds or out of sync with the units that are more directly involved in policy because the research is not of just the right kind at just the right moment.


Making research matter more 8 Elements of Research Impact Time In seeking to improve the impact of research, attention is required to all three elements, and each must be understood on its own terms, not treated as deficient based on criteria that apply somewhere else. The goal cannot be to turn researchers into practitioners or vice versa. In other words, our stance cannot be one that starts with the view that if someone else were more like us, problems would be solved. The State of Research Impact in Education It is risky to summarize developments in this area across countries that differ greatly in many respects. Nonetheless, based on looking at developments in the United States, Canada, and Britain, as well as work by the OECD a few generalizations do seem justifiable. Insufficient research of ten narrowly conceived The education research enterprise around the world is small. The OECD concluded a few years ago that the effort in research in education in all countries was typically well under 1% of education expenditures – a lo w investment relative to other knowledgeContext of research production Academic and applied Funders of research Contexts of research use Policy Politicians Advisers Bureaucracy Practice Leaders Practitioners Clients MediatorsMedia Mass Professional Think tanks Lobbyists Policy entrepreneurs Popularizers Social context Current issues Ways of thinking Popular prejudices Preoccupations Conventional wisdom


Education Policy Analysis Archives Vol. 12 No. 56 9 intensive sectors, especially health (OECD, 2002a). The U. S. has the largest education research effort in the world, but it is still very small in comparison with the size of education as a sector or in comparison with research in health or science (Shavelson & Towne, 2002). Willinsky (2000) noted that the workforce in four -year colleges in the United States is larger than that in steel, auto and textiles combined, yet the research investment in improving the latter is far greater than in the former. The production of research on education in most countries takes place mainly in universities. The largest portion of research is driven by the interests of academics, primarily in faculties of education but also in various other social sciences (such as sociology or economics) and occasionally in other applie d fields such as management or various professions. Processes for linking research ag endas to the needs and interests of practice and policy are often weak. The bulk of the funding for university research comes from the internal resources of universities or from external funding agencies. A small number of research universities tend to dominate the enterprise. Coverage of vario us areas of education is quite uneven. Takenfor-granted ideas about appropriate topics or m ethods may make it difficult for some issues to get any attention or support. Depending on the number and interests of the researchers, some areas get a reasonable amount of attention while others receive very little. In other cases, the interests and work of university re searchers may be shaped to a considerable extent by the ways in which funding is made available, so that a substantial amount of funded research may end up being directed to the issues that funders – typically governments – want to have investigated. Most university researchers in education are concerned with elementary and secondary schools, while early childhood, post-secondary education and adult education get much less attention. Most education research is short-term and small-scale, although increased funding of research networks and centres in several countr ies in the last few years has helped develop better connections among researchers and longer -term approaches. There is a general view that research methods in education have gr adually shifted towards small-scale qualitative work and that there are not enough researchers with good skills in working with large data sets even as the number and quality of such data sets is increasing. University-based research is driven by the university culture and reward system; it is primarily aimed at communication with other scholars and the rewards are related to research grants and peer recognition, alt hough connections with and recognition by the broader education community also play a role Schools of education suffer from low status within the university and in the broader community (Lagemann, 2000; Labaree, 2003). The more influential institutions tend to dominate not only what work gets done, but what work gets noticed. Another component of the research enter prise takes place in governments and nongovernmental agencies. Although most educational practice in the United States and Canada is directed by states, provinces, and local authorities, few of these bodies invest any significant funding in research beyond work done to meet current policy-making needs. Only the larger school districts maintain research operations, primarily in support of operational needs. In all countries, nationa l non-governmental organizations tend to have very modest resources for research. Some research in education is conducted by various third party organizations such as national organizations, interest groups, think ta nks, and, occasionally, other organizations such as the Conference Board or la bour groups. With the increase of interest in training and lifelong learning, a variety of business organiza tions, such as sector councils, have also


Making research matter more 10 involved themselves in some kinds of applied research. In some settings, private companies and consultants are involved in education resear ch, largely when contracted by government. Still, the overall research effort by these bod ies is both small and inconsistent, and is not always well linked to the larger process of knowledge development. In sum, the education research enterprise tends to be modest, narrow and often fragmented, though with pockets of strength. Limited capacity to use research Most of the attention in the literature on research impact focuses on the research production side. However, even the best, most effectively shared research will not matter unless users are willing and able to benefit from it The capacity of users is therefore a vital but largely uninvestigated issue (Lavis et al., 2003). The potential range of people and organizati ons interested in research in education is enormous. Governments are key given their responsibility for the education system. Potential users also include all those – teacher s, students, administrators, policy-makers who are directly involved with early childhood education, schools, college, universities, private providers, adult education and workpla ce learning. Beyond that, almost every group in society – parents, employers, workers and th eir organizations, and the non-profit sector – has an interest in education in one way or another and thus a potential connection with research. The sectors of education vary greatl y in their size and organization. So do user organizations, which range from huge educationa l enterprises or employers to single schools, small businesses, and, of course, individual students and parents. The picture arising from the limited eviden ce available is that although a wide variety of organizations are interested in the results of research, very few organizations have the capacity to be involved actively in research partnerships or to make extensive use of the results. Efforts to increase teacher research or action research run into problems of time and research background among teachers. Ma ny user organizations – for example, schools, adult learning organizations, or individual em ployers – are small and lack any staff with training or skill in research. In the less organi zed sectors, especially adult learning, there are simply fewer organizations of any kind to be involved. Even among more substantial organizations, few appear to have organized pr ocesses for learning about and making use of current research even when it is directly releva nt; rather they often rely on whatever time and effort individual staff members may make or on what happens to cross their desks. The lack of capacity among users means that demand for education research results is low relative to other fields (Shavelson & Towne, 2002). Educators tend to rely on research results as reported by the media or by th ird parties of various kinds. A great deal of third party research or interpretation is driven by other agendas, whether of particular foundations or interest groups, or sometimes of individual “policy entrepreneurs” (Mintrom, 2000) who are promoting their own ideas. The in terplay of ideas through political processes is an important part of democratic governan ce (Lindblom, 1990). However one result of multiple voices is that user organizations do not know what sources to turn to, what sources to trust, or how to identify high quality work, leading to the view that “you can prove anything with research.” Many organizations also have limited or no ability to share research results effectively within their organization. Again, how much sharing of knowledge occurs often depends on the initiative of individual staff members. Organizations are reluctant to get involved in


Education Policy Analysis Archives Vol. 12 No. 56 11 research partnerships because of the demands that may be placed on their resources and the uncertainty of potential benefits. For all user organizations, research is only one part, and often a small part, of their work. So, while interest is high and intentions are good, capacity is quite limited and performance is often weak. Whatever improvements may be made in the production of research, improving the impact of research will also require significant attention to strengthening the capacity of users and their organizations. Inadequate linkages Efforts are being made to connect research with policy and practice, but most are small scale. Researchers are broadening thei r dissemination efforts somewhat – for example by more extensive use of websites. Some newer forms of research, such as action research or teacher research, build in partnerships and a focus on use. Some research funding programs emphasize research impact, leading to a range of strategies such as workshops, publications, briefings, and in some cases staff positions focused largely or wholly on knowledge transfer. However, most disseminati on plans for academic research remain quite limited and traditional, especially if one accepts the view outlined earlier of research as an element of public debate and political practice. A number of national organizations are invo lved in one way or another in supporting research impact. Bodies such as the national membership organizations (e.g., teachers, school boards, administrators) do try to disseminate research in print and especially on the web. The same is true of third parties such as think tanks and interest groups. Some efforts have also been made in other countries to re plicate the success of the ERIC system in the United States as a reasonably comprehensive index of education research; but as the ERIC experience shows, this is a difficult and resource-intensive activity. A variety of other outreach mechanisms have also been tried, such as research programs focused on strategic issues, support for interdisciplinary work, support for dissemination-focused activities such as conferences and publications, and the creation of research networks that involve partners from outside the research community. Particular mention needs to be made of the importance of the internet as a research communication vehicle. Some very interesting efforts are bein g made to use the internet in ways that focus specifically on providing research informati on more effectively and on moving beyond passive web sites to building interaction (e.g. Willinsky, 2000, 2003; Despite these efforts, linkages are relatively weak. The models of dissemination and impact remain quite cautious. Few efforts are based on an articulated theory of how impact occurs with the result that there is still too much reliance on a direct transmission model of impact. Insufficient attention has been paid to the capacity of users and to the critical role of third parties, including the media, in creating research impact. Connections between researchers and potential users remain largely a matter of happenstance, depending often on personal ties that already happen to be in place. Work in health and social policy by Lavis et al. (2002) and in social policy by Landry et al. (2001) show that even applied research organizations employ only a limited range of strategies for increasing the impact of their work.


Making research matter more 12 Noteworthy national developments England The Department for Education and Skills launched an English education research strategy several years ago. The strategy has several main elements. One development has been funding of the EPPI (Evidence-Informed Policy and Practice) Centre at the University of London ( EPPI is involved in doing syntheses of research in selected areas with a strong focus on implications for practice including involvement of users in choosing topics for synthesis, participating in the revi ews, and preparing reports aimed at particular audiences such as teachers or school governors. The government has also committed to doing more evaluations of its policies with the results being made public (e.g. Earl et al., 2003). Research occupies a prominent place on the DfES website ( and in its policy documents. Research centres have been created in key areas such as “the wider benefits of learning” or “economics of education.” The National Education Research Forum (NERF – is a vehicle for broad discussion of research issues and is linked to some other creations, such as the Teacher Research Pane l and the Teaching and Learning Research Program ( funded to promote quality research on teachers and teaching while also increasing the capacity of educators to find and use relevant research. Other activities include a Research Capacity Building Network ( and a program to have teachers and journalists rewri te and summarise “learned journal” articles which are then made available freely through a website. Canada The Social Sciences and Humanities Rese arch Council of Canada (SSHRC) recently launched a new research initiative on “The New Economy” that provides substantial additional research funding for education with a strong emphasis on impact. This initiative comes amidst concerns that Canadian capacity in education research generally is weak. SSHRC is also proposing to transform itself from a granting council to a “knowledge council” focusing not just on supporting rese arch but on supporting its use. In support of this aim the Council has increased funding for dissemination within research projects and has developed a number of vehicles to build research capacity and impact in areas of strategic importance, including funding that is contingent on meaningful partnerships with potential users of research from the outset. Although education in Canada remains a closely-guarded area of provincial responsibili ty, there have been some good examples of federal-provincial-third party collaboration in building the value and impact of education research, such as the Pan-Canadian Educati on Research Agenda ( and the Canadian Education Statistics Council. Human Resources Development Canada, a large federal department, has supported the de velopment of several national longitudinal studies. In 2002 the federal government also announced its intention to create the Canadian Learning Institute to provide a forum for in creased collection, analysis and sharing of evidence and research on learning. United States The Bush administration has adopted a partic ular view of education research with a stress on what is called “science-based” work and an emphasis on using randomised controlled trials (RCT) as the main vehicle in research. The Office for Educational Research


Education Policy Analysis Archives Vol. 12 No. 56 13 and Improvement (OERI) has been renamed, un der the Education Sciences Reform Act of 2002, as the Institute for Educational Science and has advocated a research approach based on randomised controlled trials (Coalition, 2002). The Department of Education is participating actively in the Campbell Colla boration ( and is also creating a clearinghouse of “what works” information ( The government has also recently proposed cutbacks to the ERIC system including the removal of many studies and journals from the index on grounds of low quality. The Department’s stance has been controversial, with many rese archers regarding it as too narrow and as primarily motivated by political considerations. In light of these debates and controversie s, in 2002 the U. S. National Research Council produced a report on “Scientific Rese arch in Education” (Shavelson & Towne, 2002) that tried to provide a balanced approach in assessing quality and increasing impact. The report identified a set of principles that should inform high quality research and laid out ways in which government could support this direction. However the role and nature of research remains a politically controversial issue in the U S to an extent not matched in other countries, and the immediate future of education research remains uncertain. Despite these debates, the U S system has some longstanding practices and organizations devoted to research impact. The ERIC system has for many years, in addition to providing access to research, commissioned re views of research and prepared a variety of print and electronic vehicles for making th ese reviews widely available (although the government now intends to eliminate this latter role). A number of large research centres, such as the Center for Policy Research in Education (CPRE) and the Learning Research and Development Center (LRDC) have been funded by the federal government in key policy areas for many years, and these centres have re search impact as an important part of their mandate. While it is difficult to know just how much impact these efforts have had, they are surely linked to the growing awareness of research among educators (Biddle & Saha, 2002). However many of the centres remain primarily or ganizations of researchers rather than true partnerships with users. National agencies such as the Council of Chief State School Officers, the ASCD, the National Governors Association and others have been involved with efforts to strengthen the impact of research in education. The Spencer Foundation has recently commissioned a series of articles on research for Education Week in an attempt to support a wide and thoughtful debate. AERA has also become in creasingly active in the area of research impact., though the AERA debates on how to move in this direction also illustrate the difficulty of doing so in a way that is accepta ble to a large and diverse membership with quite varied political views. Possibilities for Action Efforts to strengthen the impact and value of research in education need to pay attention to all three contexts production, use, and the linkages between them. We should also see research impact as itself a subject abou t which we need to learn more. In the next sections, some useful but modest actions are outlined under each of these headings. Most of these suggestions have been made before in one place or another, but few recommendations that I have seen begin with a view of the entire process of research impact as interactive. The suggestions are not aimed only at research ers, but also at research funders, users, governments and third party organizations, since researchers alone cannot do what is


Making research matter more 14 needed. They are intended to be practical yet also meaningful, and they could apply to a wide range of kinds of research. Improving research production The goal of the proposals in this section is to increase the attention researchers pay to impact. Researchers alone cannot ensure the e ffective use of research, but they do play an important role. Nor am I arguing that research should only be supported if it has practitioner support or direct implications for practice. For many reasons, not least the unpredictability of what will turn out to be valuable knowledge, an important role must remain for research that is driven purely by researcher interest. However even in interestdriven research not intended for any immediate application there could well be benefits to researchers thinking about who else might be in terested in their work and how that interest could best be encouraged. 1.1 Give greater attention to impact in research granting processes. Build impact into the criteria for evaluating grants, require research ers to include previous impact work as part of their application, and foster careful consideration of impact strategies in peer review committees. 1.2 Provide supports and resources to researcher s to assist research impact. For example, providing a checklist of possible strategies fo r impact might encourage researchers to pay more attention to this element of their work. 1.3 Support universities in building the same ki nd of effort in knowledge mobilization in education as they have in the sciences and engineering. While the processes are not identical, technology transfer offi ces, whose task it is to help researchers with the application of their work and to negotiate relationships between researchers and users, provide a model that could be applied in education also. 1.4 Strengthen the input of potential users in the development and review of research proposals. There are many cases where some early discussion between researchers and users would result in stronger studies as well as enhanced interest by potential users. 1.5 Take steps to extract maximum value from ex isting research. Often researchers get busy with the next project before they have fully ex ploited the potential of the last one. To improve this situation one need is to make data more available to other researchers who might want to use it again. A second step is to encourage researchers to spend more time on exploiting their data instead of immediatel y going on to a new grant and a new study. Finally, strengthening international research links is important to ensure the utility of relevant work wherever it is done. 1.6 Build networks of researchers and users wi th common interests as a way of developing larger-scale programs of resear ch with stronger links to use. Networks can be important in increasing attention to knowledge mobilizati on and in building stronger, more coherent research programs. More work could be done to link researchers with common interests even where they are not part of a formal network with independent funding.


Education Policy Analysis Archives Vol. 12 No. 56 15 Improving use Research will not have impact unless potential users are interested enough to look for it and able to make use of what they find. It is vital to develop the capacity of users to find, understand and use research. However because relatively little is known about how to do this, first steps will require learning more about the limits of the current situation and strategies that might be effective. 2.1 Work with major user organizations in education and beyond to learn more about their capacities and limitations. This kind of analys is could itself be a useful step in generating increased capacity. For example, how do school districts currently find and distribute relevant research? How could their efforts be better supported? 2.2 Work with user organizations that want to be more active in making use of education research. Examples include the creation of user networks, further analysis of effective internet-based knowledge exchange practi ces, learning more about what internal communication practices are most effective, or increasing the number of organizations employing staff with good backgrounds in research. 2.3 Develop better means of exchange of info rmation among user organizations. Examples include shared processes of literature scanni ng, better means of exchanging information on relevant research, or the creation of networks of staff in different organizations who have research impact responsibilities. 2.4 Support placement of researchers and gr aduate students in user organizations. Opportunities for placement of people with good research skills in user organizations, such as internships for graduate students, could help build ongoing capacity in those organizations. Linkages between production and use A number of steps can be taken to broaden and deepen connections between researchers and users. Among the most interesting possibilities are: 3.1 Employ people who are skilled in “translating ” research results into plain language for distribution to non-specialist audiences. Such specialists – people who can write 500 word op-ed pieces or brief summaries for distributi on to policy-makers – could be employed by universities or might be directly sponsored by funders. 3.2 Use media relations experts to build media connections. Despite the importance of the media, few researchers have any experience in attracting media attention or communicating in a way that preserves the essence of thei r message. Again, these resources could be available through universities or through other bodies. 3.3 Make better use of existing venues for eff ective exchange between researchers and users. Face to face contact remains vital in building trust and interest. Almost everyone in


Making research matter more 16 education already participates in various even ts that could have a research component to them yet do not. Sponsoring more opportunities for researchers to participate in user events and for users to attend and participate meanin gfully in research events would help build ongoing links. 3.4 Increase the connection between resear ch and professional development in user organizations. Professional development is a ke y means of bringing research to practice (Cordingley, 2000, Figgiss et al., 2000). Steps could be taken to make it more likely that effective interaction occurs by working with a ssociations or other networks that organize professional development work in various sectors. 3.5 Learn more about effective web interaction around research. Many web-based efforts to disseminate research already exist, includin g those supported by researchers and those supported by user organizations. It is important to understand what practices work most effectively, and especially how it might be possible to move beyond passive information provision to more interactive forms of engagement on the web. 3.6 Reconsider effective print and electroni c vehicles such as books, CDs, journals, magazines, newsletters, and brochures. Print remains a key means of distributing information but publication practices are rapidl y changing largely due to developments in information technology. Often published work take s a long time to appear and is even then not readily accessible to users. New approaches to publication and distribution need investigation. 3.7 Expand literature reviews, research synthese s, and the drawing of “what works” or “what we know” conclusions. Single studies are ra rely powerful impacts on practice, nor, given their limitations, should they be. Users are interested in conclusions drawn from substantial bodies of research. Yet few researchers are involved extensively in this kind of work. More discussion and action are needed on how to develop effective syntheses or to participate in wider synthesis networks. Some of the developments in England mentioned earlier merit wider attention. Learning about research impact Our current situation is that although there is interest, we do not yet know very much about this whole area and therefore need to learn as we go. In many areas it will be as important to study our actions carefully as to take the steps in the first place. It will also be important to organize some selected learning events, web networks and other activities that focus on research impact itself. Connections need to be built with other social policy fields that are engaged in similar effo rts such as health and justice. Conclusion Research is only one part of the way our societies make decisions about policy and practice. Other social and political forces w ill usually be more important and researchers will often feel that their work is ignored or misused. Nonetheless, growing interest in research and its potential contribution to policy and practice in education are heartening


Education Policy Analysis Archives Vol. 12 No. 56 17 developments. While risks exist, and the results are unlikely ever to be as good as researchers might wish, there are steps that coul d be taken by all parties with an interest in education to strengthen the contribution of rese arch to policy and practice in education. Note This article was supported by a research gran t from the Social Sciences and Humanities Research Council of Canada (SSHRC) while I was visiting scholar there in 2002-03. I thank Marc Renaud, President of SSHRC, for the opportunity to think about these issues. Colleagues too numerous to list at SSHRC and in the broader education research community have helped my thinking, and comments by the various reviewers were also very useful. All interpretations and any errors are solely my responsibility. Nothing in this paper should be taken to represent the policy or opinion of SSHRC or any other organization. References Barnett, W. (1996). Lives in the balance: Age-27 benefits-cost analysis of the High/Scope Perry Preschool Program Ypsilanti, MI: High/Scope Press.. Berliner, D. (2002). Educational rese arch: The hardest science of all. Educational Researcher 31 (8), 18-20. Bernstein, R. (1976). The restructuring of social and political theory Philadelphia: University of Pennsylvania Press. Biddle, B. and Saha, L. (2002). The untested accusation: Principals, research Knowledge, and policy making in schools Westport, CT: Ablex. Coalition for Evidence-Based Policy (2002). Bringing evidence-driven progress to education. Report for the U S Department of Education, November, 2002. Department of Education, Training and Youth Affairs (DETYA) (2000) The impact of educational research Canberra: DETYA. Cordingley, P. (2000). Teacher perspectives on the accessibility and usability of research outputs. Paper presented to the British Educational Research Association. Dror, Y. (1986). Policy making under adversity New Brunswick, NJ: Transaction. Earl, L., Watson, N., Levin, B., Leithwood, K. Fullan, M. & Torrance, N. Watching and learning 3: Final report of the OISE/UT evalua tion of the implementation of the National Literacy and Numeracy Strategies. Prepared fo r the Department for Education and Skills, England. Feuer, M., Towne, L. & Shavelson, R. (2002) Scientific culture and educational research. Educational Researcher 31 (8), 4-14.


Making research matter more 18 Figgis, J., Zubrick, A., Butorac, A., and Alde rson, A. (2000) Backtracking practices and policies to research. In The impact of educational research Pp. 279-374. Canberra. Department of Education, Tr aining and Youth Affairs. Fullan, M. (2001). The new meaning of educational change, 3rd edition New York: teachers College Press. Gardner, H. (2002). The quality and qualities of educational research. Education Week, September 4, Vol. 22 (01). Giddens, A. (1994). Beyond left and right Cambridge, UK: Polity Press. Hargreaves, D. (1996) Teaching as a research -based profession: possibilities and prospects. Teacher Training Agency Annual Lecture. London: TTA. Hargreaves, D. (1999) Revitalising educational research. Cambridge Journal of Education, 29 (2) 239-250. Haveman, R. & Wolfe, B. (1994). Succeeding generations: On the effects of investments in children New York: Russell Sage Foundation. Hillage, J., Pearson, R., Anderson, A. and Ta mkin,P. (1998) Excellence in research in schools. London. Department for Education and Employment/Institute of Employment Studies. Homer-Dixon, T. (2000). The ingenuity gap New York: Knopf. Hopkins, D. (2001). School improvement for real London: RoutledgeFalmer. Labaree, D. (2003). The peculiar problems of preparing educational researchers. Educational Researcher 32 (4), 13-22. Lagemann, E. (2000). An elusive science: The troublin g history of education research Chicago: University of Chicago Press. Landry, R., Amara, N. & Lamari, M. (2001) Utilization of social science research knowledge in Canada. Research Policy, 30 (2), 333-349. Lavis, J., Ross S., Hurley, J., Hohenadel, J., Stoddart, G., Woodward, C., & Abelson, J. (2002). Examining the role of health services research in public policy making. Millbank Quarterly 80 (1), 125154. Lavis, J. Roberston, D., Woodside, J. McLeod,C., & Abelson, J. (2003). How can research organizations more effectively transfer research knowledge to decision-makers? Millbank Quarterly, 81 (2), 221-248.


Education Policy Analysis Archives Vol. 12 No. 56 19 Levacic, R. & Glatter, R. (2001). ‘Really g ood ideas?’: Developing evidence-informed policy and practice in educational leadership and management. Educational Management and Administration, 29 (1), 5-25. Levin, B. (2001) Governments and school improvement. International Electronic Journal for Leadership in Learning, 5 (9), May 26. Lindblom, C. (1990). Inquiry and change New Haven: Yale University Press. McGaw, B., Boud, D., Poole, M., Wa rry, and McKenzie, P. (1992) Educational research in Australia: Report of the review panel of research in education Canberra. Australian Government Publishing Service. Mintrom, M. (2000). Policy entrepreneurs and school choice Washington: Georgetown University Press. National Education Research Forum (2000). Re port of the subgroup on Impact of research on policy and practice. London: Department for Education and Skills. OECD (2002a). Educational research and development in England: Examiners’ report. Paris: OECD. CERI/CD(2002)10 OECD (2002b). Knowledge management in education and learning. Report prepared for the Oxford Forum, March. Orne, M. (1962). On the social psychol ogy of the psychological experiment. American Psychologist, 17 776-783. Powell, D. & Leiss, W. (1997). Mad cows and mother’s milk: The pe rils of poor risk communication Montreal: McGill-Queen’s University Press. Prost, A. (2001). Pour un programme stratgi que de recherche en ducation. Report to the ministers de l’ducation nationale et de la R echerche. Report of the working group named by M Prost. Paris: Minis tre de l’ducation nationale. Rogers, E. (1995). Diffusion of innovations New York: Free Press. Shavelson, R. & Towne, L. (eds.) (2002). Scientific research in education Washington: National Academy Press. Stone, D. (1997). Policy paradox New York: Norton. Weiss, C. (1979). The many meanings of research utilization. Public Administration Review 39 (5), 426-431. Willinsky, J. (2003, January 11). Policymak ers' online use of academic research. Education Policy Analysis Archives, 11 (2). Retrieved October 15, 2004 from


Making research matter more 20 Willinsky, J. (2000). If only we knew : Increasing the Public Value of Social-Science Research New York: Routledge. About the Author Ben Levin is Professor in the Faculty of Education at The University of Manitoba. On Dec 6, 2004, he becomes Deputy Minister of Education for the Province of Ontario, a position he will hold while on a leave of absence from a new academic position in the Department of Theory and Policy Studies at OISE/University of Toronto. Email:


Education Policy Analysis Archives http:// Editor: Gene V Glass, Arizona State University Production Assistant: Chris Mu rrell, Arizona State University General questions about appropriateness of topics or particular articles may be addressed to the Editor, Gene V Glass, gl or reach him at College of Education, Arizona State University, Tempe, AZ 85287-2411. The Commentary Editor is Casey D. Cobb: EPAA Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Linda Darling-Hammond Stanford University Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman Arizona State Univeristy Richard Garlikov Birmingham, Alabama Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute of Technology Patricia Fey Jarvis Seattle, Washington Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Berkeley Michele Moses Arizona State University Gary Orfield Harvard University Anthony G. Rud Jr. Purdue University Jay Paredes Scribner University of Missouri Michael Scriven University of Auckland Lorrie A. Shepard University of Colorado, Boulder Robert E. Stake University of Illinois—UC Kevin Welner University of Colorado, Boulder Terrence G. Wiley Arizona State University John Willinsky University of British Columbia


Archivos Analticos de Polticas Educativas Associate Editors Gustavo E. Fischman & Pablo Gentili Arizona State University & Universidade do Estado do Rio de Janeiro Founding Associate Editor for Spanish Language (1998—2003) Roberto Rodrguez Gmez Editorial Board Hugo Aboites Universidad Autnoma Metropolitana-Xochimilco Adrin Acosta Universidad de Guadalajara Mxico Claudio Almonacid Avila Universidad Metropolitana de Ciencias de la Educacin, Chile Dalila Andrade de Oliveira Universidade Federal de Minas Gerais, Belo Horizonte, Brasil Alejandra Birgin Ministerio de Educacin, Argentina Teresa Bracho Centro de Investigacin y Docencia Econmica-CIDE Alejandro Canales Universidad Nacional Autnoma de Mxico Ursula Casanova Arizona State University, Tempe, Arizona Sigfredo Chiroque Instituto de Pedagoga Popular, Per Erwin Epstein Loyola University, Chicago, Illinois Mariano Fernndez Enguita Universidad de Salamanca. Espaa Gaudncio Frigotto Universidade Estadual do Rio de Janeiro, Brasil Rollin Kent Universidad Autnoma de Puebla. Puebla, Mxico Walter Kohan Universidade Estadual do Rio de Janeiro, Brasil Roberto Leher Universidade Estadual do Rio de Janeiro, Brasil Daniel C. Levy University at Albany, SUNY, Albany, New York Nilma Limo Gomes Universidade Federal de Minas Gerais, Belo Horizonte Pia Lindquist Wong California State University, Sacramento, California Mara Loreto Egaa Programa Interdisciplinario de Investigacin en Educacin, Chile Mariano Narodowski Universidad Torcuato Di Tella, Argentina Iolanda de Oliveira Universidade Federal Fluminense, Brasil Grover Pango Foro Latinoamericano de Polticas Educativas, Per Vanilda Paiva Universidade Estadual do Rio de Janeiro, Brasil Miguel Pereira Catedratico Universidad de Granada, Espaa Angel Ignacio Prez Gmez Universidad de Mlaga Mnica Pini Universidad Nacional de San Martin, Argentina Romualdo Portella do Oliveira Universidade de So Paulo Diana Rhoten Social Science Research Council, New York, New York Jos Gimeno Sacristn Universidad de Valencia, Espaa Daniel Schugurensky Ontario Institute for Studies in Education, Canada Susan Street Centro de Investigaciones y Estudios Superiores en Antropologia Social Occidente, Guadalajara, Mxico Nelly P. Stromquist University of Southern California, Los Angeles, California Daniel Suarez Laboratorio de Politicas Publicas-Universidad de Buenos Aires, Argentina Antonio Teodoro Universidade Lusfona Lisboa, Carlos A. Torres University of California, Los Angeles Jurjo Torres Santom Universidad de la Corua, Espaa Lilian do Valle Universidade Estadual do Rio de Janeiro, Brasil

xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20049999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00405
0 245
Educational policy analysis archives.
n Vol. 12, no. 56 (October 17, 2004).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c October 17, 2004
Making research matter more / Ben Levin.
x Research
v Periodicals.
2 710
Arizona State University.
University of South Florida.
1 773
t Education Policy Analysis Archives (EPAA)
4 856