xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20039999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00300
Educational policy analysis archives.
n Vol. 11, no. 2 (January 11, 2003).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c January 11, 2003
Policymakers' online use of academic research / John Willinsky.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 11issue 2series Year mods:caption 20032003Month January1Day 1111mods:originInfo mods:dateIssued iso8601 2003-01-11
1 of 23 Education Policy Analysis Archives Volume 11 Number 2January 11, 2003ISSN 1068-2341 A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright 2003, the EDUCATION POLICY ANALYSIS ARCHIVES .Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. EPAA is a project of the Education Policy Studies Laboratory. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education .Policymakers' Online Use of Academic Research John Willinsky University of British ColumbiaCitation: Willinsky, J. (2003, January 11). Policym akers' online use of academic research, Education Policy Analysis Archives 11 (2). Retrieved [date] from http://epaa.asu.edu/epaa /v11n2/.AbstractIn addressing the question of how new technologies can improve the public quality and presence of academic research, t his article reports on the current online use of research by policymakers. Interviews with a sample of 25 Canadian policymakers at the federal l evel were conducted, looking at the specific role that online research h as begun to play in their work, and what frustrations they face in using this research. The study found widespread use of online research, increasing the consultation of this source in policy analysis and formation. The p rincipal issues remain those of access, indexing and credibility, with pol icymakers restricting themselves in large part to open access sources. St ill, online research is proving a counterforce to policymakers' reliance on a small number of academic consultants as gatekeepers and sources for research. What is
2 of 23needed, it becomes clear, is investigations into w hether innovative well-indexed systems that integrate a range of acad emic and non-academic resources might increase the political impact of research in the social sciences and education.IntroductionTo better understand the complex and diffuse relat ionship that exists between social policy and social science research, I present the r eflections of twenty-nine Canadian policy officials on access to scholarly research af forded by the Internet. While researchers interested in the political impact of t he Internet have tended to focus on "digital democracy" issues of public access to gove rnment information, public consultation and participation, and public privacy and surveillance, a far less dramatic change is also taking place to existing policy-maki ng processes. (Note 1) The knowledge economy of the Internet has significantly increased policymakersÂ’ ability to tap into current, critical, and relevant research without le aving their desk. Although many academic journals remain closed to those without ac cess to a good research library, a growing number of individual papers, journals, and research archives are providing full online access at no charge to users. (Note 2) This new availability has transformed the information environment within which policy develop ment takes place, and it is altering the role of social scientists in democratic process es. Admittedly, the influence of academic research on g overnment policy pales before the impact of the governmentÂ’s own sources of informati on, whether from politicians, bureaucrats, or government professionals. As Harvar dÂ’s Carol Weiss has observed, it takes "an extraordinary concatenation of circumstan ces" for "research to influence policy decisions directly" (1991, p. 44). Weiss has conclu ded, after many years of evaluation work in education, that "governments don't often us e research directly, but research helps people reconsider issues, it helps them think diffe rently, it helps them reconceptualize what the problem is and how prevalent it is, it hel ps them discard some old assumptions, it punctures old myths. It takes time and reconcept ualization before research actually leads to a change in policy" (1998). Still, one fin ds, for example, that a recent U.S. federal education bill calls on school officials wh o seek federal support to secure "scientific evidence" in arriving at program decisi ons, as well as related initiatives to improve federal spending on education research. (No te 3) Similarly, the British governmentÂ’s recent Hillage report calls for educat ion research to play a greater part in policy formation, and to take steps to better prepa re itself to play this role (Hillage 1998). Also, the University of LondonÂ’s Institute f or Education has established a Centre for Evidence-Informed Policy and Practice, which se eks, among other things, to make research more accessible for policymakers and the p ublic, as well as to support a more coordinated, less fragmented, approach to education research. (Note 4) Given the complex and exhaustive knowledge needs of the modern democratic state, academic research has a vital, irreplaceable role t o play in not only informing and evaluating government decisions, programs and polic ies, but also in a broader intellectual sense, whether by challenging assumpti ons, providing a historical perspective, bringing international comparisons to bear, or offering lessons from experimental models and alternative conceptions. (N ote 5) After all, the classic administrative problem, identified by Herbert Simon more than a half-century ago, is the "bounded" rationality of bureaucrats, always limite d in their knowledge of the situation,
3 of 23the issues, and the available alternatives (1945). Although this dilemma led Simon to work on artificial intelligence devices that would supplement these limits, it may be that improved and timely access to relevant research can expand the rationality of the policy process. (Note 6)I would also note, before presenting the policymake rsÂ’ reflections on online sources of social science research, that this focus on researc h-for-policy has been perceived in Great Britain as a threat to academic freedom, as well as in the case of education research, to the professionalism of teachers (Ozga, 2000; Humes and Bryce, 2001). The increased influence of research on government policy also con cerns American political scientists Anne Larson Schneider and Helen Ingram (1997), as t his increase has done little to counter widespread public disillusionment with the U.S. government since the 1960s. In their view, the growing influence of "scientific an d professional perspectives" can further alienate citizens, with the complexity of t heir approaches and exclusionary jargon. This use of science may seem to increase th e rule of reason at the expense of political maneuvering, but its inaccessibility can further shrink the public sphere and public deliberations, as research reduces social is sues to technical ones that can be resolved through expert advice and "best practices" (1997, p. 153). (Note 7) For Schneider and Ingram, science has become a taxsupported "establishment" of limited accountability, which can leave the public wondering why such work needs public support. They identify the National Institut es of Health as an example of scientists securing "overgenerous funding" benefiting "advanta ged populations," in their estimation, as well as scientists and drug companie s (1997, p. 164). (Note 8) They are also concerned with the considerable number of scie ntists employed by governments, as this can reinforce the insular power of the bureauc racy. (Note 9) The obvious point of caution is that policy officials, politicians, and interest groups can selectively represent research studies, if not selectively misinterpret t he results (Barker and Peters, 1993). For Schneider and Ingram, what the policy process is mi ssing is a democratic commitment to "re-energize people and create an educated, enlight ened active citizenship" (1997, p. 7). Their concern harkens back to the democratic divide between John Dewey, who believed in fostering and relying on an increasingly informe d public who have the "the ability to judge the bearing of knowledge supplied by others u pon common concerns" (1926, p. 209), and Walter Lippmann (1922), who held that a c omplex democratic state was best governed through "expert mediation" and "organized intelligence." (Note 10) My own efforts, very much on the side of Dewey, hav e been to explore ways of improving access to social science research for the benefit of both public and policy officials. The Public Knowledge Project, with which I work, seeks to improve not only the scholarly quality of research, but what I would refer to as its "public quality," in the sense of enhancing its public access and intelligib ility. (Note 11) As a foundation for this work, the Project studies the ways in which policym akers, the press, practitioners, parents, and the public, as well as faculty and stu dents, utilize knowledge from online sources. On the basis of this understanding, we are pursuing better designs and structures for organizing, presenting, and integrating researc h with related resources, such as existing policy initiatives (Willinsky and Wolfson, 2001). The study presented here focuses on the current lev els of engagement with academic research among policy officials. It is intended to serve as a baseline for current usage, as well as to identify a number of key issues that sha pe the impact that this knowledge has on the development of policy. My argument is that w e have an opportunity and
4 of 23 responsibility within the social sciences to increa se the public and scholarly value of our work. New online publishing technologies, as they i ncrease access, can do more to inform and expand deliberations among people and po licymakers. This study asks policymakers about how this new medium is assisting or impeding their use of this research. The resulting interviews suggest a number of stepsÂ—including both the labeling and context provided for the researchÂ—that would vastly improve the organization, and thus the value, of policy-relevan t research online for both public and policymakers. The policy officials in this study we re keen to engage with online research, and it appears that the impact on their work would only increase as access and organization are improved. Otherwise, too much of w hat we know about matters of concern to government policy goes unheard and unsee n by those in a position to do something with it, as well as by those who suffer t he consequences of this ignorance.MethodWe interviewed 29 civil servants, policy analysts, research officers, and librarians, drawn from the Canadian federal and provincial government s (Table 1). The government departments and divisions that agreed to participat e in this research project selected areas for us to conduct interviews in which researc h might well come into play. The officials (with only two women among the sample) we re often referred to us by their superiors, as we explained the nature of our work, as people who were involved in the research and information gathering aspects of their government agency. The Policy Research Initiative, for example, had divisions con cerned with Knowledge Integration, as well Social Cohesion and Sustainable Development suggesting a growing interest in the sort of long-term and broad conceptualizations that the social science research community addresses. There were more prosaic, but n onetheless vital, areas included, such as Employment Insurance and Monetary Analysis.Table 1 Number of Policy Official Participants by Division and OrganizationNo.DivisionGovernment Organization 2Policy PanningAgriculture Canada2Strategic PolicyAgriculture Canada3Monetary AnalysisBank of Canada1Knowledge TransferCanadian Health Services Researc h Foundation 1Environment DivisionCanadian International Develop ment Agency 1Global Issues BureauDept. of Foreign Affairs and I ntÂ’l Trade (Canada) 1Policy PlanningDept. of Foreign Affairs and IntÂ’l Trade (Canada) 2Policy Research StatisticsDepartment of Justice (C anada)
5 of 23 1Employment InsuranceHuman Resources Development Ca nada 2Human Resources Partnerships Human Resources Development Canada 4Social Policy DirectorateHuman Resources Developme nt Canada 4LibrariansLegislative Library (Ontario)1Knowledge IntegrationPolicy Research Initiative (C anada) 1North American LinkagesPolicy Research Initiative (Canada) 1Social Cohesion GroupPolicy Research Initiative (C anada) 1Sustainable DevelopmentPolicy Research Initiative (Canada) 29Total The questions posed to the policy officials covered three areas of interest, beginning with their "general regard for scholarship and research, including the role it plays in their work and their means of access to it (Appendix 1). The interview went on to consider their use of "electronically available research," i ncluding search strategies, favorite sources, and challenges and foreseeable changes. It concluded by asking them to consider two designs for organizing research: one w as a prototype site we had previously developed, Policy.ca, and the other a model for pro viding individual studies with a context. (Note 12) Some subjects declined the offer of anonymity in this research, and their names are used in the analysis that follows, which has been grouped by the nature of the government organization, that is, by governm ent department, agency, research initiative, and library.Policy Analysts in Government DepartmentsThe mission of Human Resources Development Canada i s "to enable Canadians to participate fully in the workplace and the communit y." (Note 13) In this department, we spoke with an "employment insurance" policy analyst who described seeking a "diagnositque" from his reading of researchers, sta keholders, trend analyses, and developments in the field, such as a corporation th at had a great day-care policy. He was a regular user of government information sites, suc h as Statistics Canada, but he also found himself often enough facing perplexing Google searches that produced thousands of hits on a topic. Still he admires the speed of i nformation retrieval online, once the wheat has been separated from the vast amounts of c haff: "What we have done is complement the telephone and fax, and what we've go t now is much faster." He also made a most interesting case for hard copy by refer ring to "the cognitive processes necessary for understanding the material" which wer e assisted by the ability to "spread it out on a desktop to compare it." What also has not changed is that his department still tends to work with the "major players" in the resea rch field, using them to ensure the quality of information as well as funnels and filte rs for additional research, which may well limit a policy analystÂ’s reach for fresh persp ectives. Still, he looks forward to when "more sophisticated [research] portals are introduc ed" which will provide access to both overviews and full-text studies. He felt that the u niversities should make an effort to match the Government On-Line (GOL) initiative with a similar approach to research.
6 of 23In the Human Resource Partnerships Directorate of t his same department, we interviewed two senior policy advisors, Silvano Toc chi and Nicholas Wise, who both work with corporations to improve employment opport unities. They, too, rely on an academic network to filter and serve as a conduit f or research. They also use it as a "sounding board" for ideas, suggesting a more inter active policy involvement for researchers, if a rather select, handpicked group. Tocchi and Wise see their own work as immersed in social science research, with their own reports and internal research adhering to academic standards. Given the increasin g availability of relevant information, including research, they find themselv es playing a mediating role with research in their division, or as Wise explains, "s ometimes [there is] a need to Â‘tailorÂ’ information in a manner that is more appealing or b etter understood within the departmentÂ—packaging it in ways that make it meanin gful." They feel it important to browse recent research, without always going after a specific topic, and to that end they subscribe to a number of services that supply a wee kly compendium of abstracts or summaries. They were concerned, finally, that effor ts to improve information systems would provide links to research carried on outside of the universities, whether to private research companies or to similar departments in oth er jurisdictions. With the Social Policy Directorate of Human Resourc es Development Canada, we held a small seminar with four of its members. This group provides both quantitative and qualitative research support for policy development especially as a starting point for new initiatives, that also identifies gaps in what is known. Despite the gains through Internet access, they felt that much relevant unive rsity research is locked away; they want better systems, greater organization of materi als, and more open access to "the real Â‘meatyÂ’ and interesting material" as opposed to ava ilable summaries and overviews. Difficulties with getting at this research only mea nt, they pointed out, that think tank and interest group reports can exert greater influence through their accessibility: "Often you may have only an afternoon to come up with the poin ts you need, so you are going to hit the key groups you know are involved in this area, find the pieces there, and not go any further." Yet they felt that overall the InternetÂ’s increased access to research has added to the divisionÂ’s general level of engagement with res earch: "People are now doing much more of their own research, whereas before there wa s a select few people or they went to the library and asked a librarian to find informati on for them. It is now much easier for an individual to do it themselves."The Department of Justice was the site of an interv iew with two Research Officers working on issues of cyber-crime. They also followe d the traditional pattern of working with a select set of academics in reading their res earch, consulting with them personally, and inviting them to roundtables. The Internet, how ever, had enabled them to expand the group they consulted. Still, they shared concerns w ith others about the overwhelming amount of information on the Web, as well as about the veracity of what is found there: "Although you save time in retrieving the document, you must spend it trying to verify the authenticity of certain material." To help othe r government officials deal with this issue, they publish "Just Research," which provides a guide to valuable websites for Department of Justice employees. They also expresse d a concern that universities do more than just put their work out there: "We need g reater ease of connection to places such as university research sites (possibly through a portal or channel), without the frustration of everyone finding the sites always on their own. Then it will be more like going to a traditional library catalogue, where som eone is keeping these links
7 of 23up-to-date." This interest in a centralized site in cluded gaining access to studies going back some twenty years, to permit a historical pers pective of the very study of justice issues. They were also very articulate about the ne ed for integration among different types of academic literature, from conference paper s to meta-analyses, as well as wondering about copyright issues and access to "res earch data, instruments, and methods" which might be of interest. They were clea rly ready for a far more sophisticated approach to online access to research than is currently available. In the Department of Foreign Affairs and Internatio nal Trade, we spoke with an International Drug Officer responsible for internat ional comparative analyses of drug policies. The Internet now dominates his informatio n seeking activities. He quickly pointed out that he has no shortage of information on drugs, which only meant that ascertaining "good and credible information" was hi s biggest challenge. He wondered if having third-party authenticators or guarantors of quality would not resolve this problem. He certainly appreciated the ability to corroborate studies and crosscheck research claims using the Internet, although it was not always poss ible to do so. The Senior Policy Advisor whom we interviewed from this department al so spoke of his increasing dependence on the Web, with up to 80 percent of his research, he estimated, conducted there. He had a real interest in seeing complete st udies rather than just abstracts, although he found pay-per-view services for full-te xts an economic roadblock. Still, he also reminded us that policies are far more informe d by classified informationÂ—from government and intelligence sourcesÂ—as well as by p ublic resources posted by foundations and other organizations, in part becaus e academics seem to lag behind, in his opinion, in placing information on the web. He felt that research needed a contextÂ—especially with access to other works by an authorÂ—to be judged useful and reliable: "Material and analysis outside of the stu dy itself is useful; and can, perhaps, guide oneÂ’s own interpretation, for example, a lega l decision is much more understandable when the commentaries are reviewed."In Agriculture Canada, we held two sets of intervie ws, the first interview with two policy analysts from the Strategic Policy Branch, which is concerned with farm economics. The two posed an interesting contrast to each other. On e of them relied on a small set of sites, largely government run, such as StatsCan, and worke d with a small circle of researchers: "I usually know the academics that are writing in m y area." The other analyst browsed widely, constantly in search of new developments. H e believed in the "serendipity" factor, using the example of how he had come across an Australian agricultural initiative that has now led to a collaborative effort between the Canada and Australia. Still, for both of them, the Internet had become their main so urce; the one could not remember the last time he had been to the departmental library i n the basement and the other had been led there by online resources. They were interested in how the Internet could provide some historical depth to their review of research, enabling a review of some thirty years of research. In considering the models for improvin g the organization of research that we have been developing, they wondered about gaining a ccess to MasterÂ’s and Ph.D. theses ("which we are seldom aware of") and in being able to establish a context for researchÂ—"Reports on reports (reviews), contra view s, and summaries are often more revealing than the actual studies." They were also interested in "some form of Â‘certification,Â’" as well as knowing who funded the research, as a study "funded 50% by Monsanto does suggest caution and skepticism when r eviewing the results." The other two policy analysts whom we interviewed f rom Agriculture Canada were from
8 of 23Policy Planning. Here we found real concerns with t he failure of research to address the current political issues in agriculture, often forc ing them to rely on the work of other policy analysts, whether here or abroad. They were concerned that the resulting absence of information and data has meant that "future poli cy solutions are more constrained by where we are, where we have been, and our current d omestic situation." They would obviously like to have a greater array of perspecti ves come into play in policy development. As it is, they tend to stay with a sma ll Canadian academic communityÂ—maybe 25-30 professorsÂ—who are conducting relevant research for their area, noting that while they come across others, it requires checking "what else they have published" and the credibility of their institution Â—"If Harvard has hired them...." They were also sensitive to how the use of research in p olicy settings invoked issues of jargon and translationÂ—"translating academic work into a l anguage our executives understand is my job." Yet there are areas were they feel well served by the Web for developing their professional skills: "I use the Internet more for locating management and strategic thinking material, than policy material, i.e., Kenn edy School of Government, Harvard, Stanford, or universities with good management scho ols, I know who the significant management writers are, and then see what they have made available on the site." Among the policy officials working in their four go vernment departments, the Internet has taken hold as a primary information source, off ering ease and speed, while posing challenges of extracting value and establishing cre dibility. There is clearly room for social science research to play a greater part in t he policy process, for expanding the range of ideas considered and opening up the govern mentÂ’s thinking about possibilities that extend beyond the closed traditional circles o f academics.Policy Analysts in Government AgenciesWe worked with two government agencies, the Bank of Canada and the Canadian International Development Agency that are very invo lved in policy issues, if at one remove from the legislative forum of Parliament. At the Bank of Canada, we spoke with three men involved in monetary modeling, applied st udies, and policy research. In each case, they told us, research is what they do. They are economists who both produce reports for the Bank and publish research in academ ic journals. They pointed out that economics is one academic discipline that has a cen tral open-access repository, RePEc (Research Papers in Economics) that provides free a nd centralized access to an enormous number of articles, reports and working pa pers. They see RePEc as a real advantage to their work as researchers, just as the y appreciate having electronic access to complete sets of economic journals online, running back to the turn of the century, through J-STOR. (Note 14) Still, they sometimes fee l frustrated by disciplinary boundaries that separate other relevant areas, such as biology, mathematics, and statistics, from this bounty of economic literature And while they cannot imagine working now without the Internet, they would like t o see greater consolidation of access and indexing, as well as more full-text resources a nd citation linking, as the search for research resources still seems unnecessarily diffic ult. Their opinions speak to the technologyÂ’s ability to foster a greater demand for its powers than it can, at any given moment, meet.Caroline Caza is a Senior Environmental Policy Advi sor, with the Canadian International Development Agency, which is devoted to supporting "sustainable development activities in order to reduce poverty a nd to contribute to a more secure,
9 of 23equitable and prosperous world." (Note 15) Although she does not see herself as a regular consumer of research, CazaÂ’s impression is that academic research does influence the formation of policy, observing that a cademics are hired on occasion to review the literature and present feasible policy p erspectives. She pointed out that within her own agency the press was on to create a "knowle dge-based organization," which would increase its capacity to create effective pol icies. In her own work, Caza prefers reviews of current research which she finds "much m ore useful forms of information," while individual studies can be "too long" and "too narrow to be able to synthesize the diversity of opinion you that need in order to make a credible policy decision." Yet she also felt that what was frustrating about some info rmation rich websites was "the lack of depth and breadth in, for example, Policy.ca" (refe rring to the Public Knowledge Project site). And while senior policy officials are always asking for quality information, they may not be in a position to judge that quality them selves. For her part, she has let her subscriptions to academic journals, acquired during her university days, lapse as they failed to serve in her work: "They didn't allow me to be credible by bringing new scientific information to a forum in a way I could use effectively." The exception was Conversation Biology which Caza continued to receive for its Letters to the Editors, which she felt presented leading-edge issues and an y controversy surrounding them. So while she was turning from print sources, she was s till seeking the ideal information websites which offered authentic and credible infor mation, in what was an all too common theme among policy officials in our sample.Government Research InitiativesWe spoke with representatives from two of the Gover nment of CanadaÂ’s research agencies: the Policy Research Initiative and the Ca nadian Health Services Research Foundation. The Policy Research Initiative was esta blished "to strengthen the federal government's policy research capacity" and sees its elf as "a catalyst for the development of knowledge, people and community ," according to its website (original emphasis). (Note 16) Here, the government is fully engaged in bringing research to bear on areas of Canadian policy. It publishes Isuma: Canadian Journal of Policy Research which features an editorial board made up of academics (a lthough one PRI representative we spoke with felt it was "not as in-depth as a tradit ional academic journal"). The journal offers an open-access electronic version as well as a print run that is distributed "free to a large cross section of CanadaÂ’s policy research com munity." The Policy Research Initiative also publishes Horizons "a snapshot of emerging, cross-cutting research i n the Canadian and international policy environments." It also has a Policy Research Data Group that works with other departments facilitatin g "the development of databases required to carry out research in priority horizont al policy areas." Within the Policy Research Initiative is the Knowle dge Integration Project which is directly involved in exploring new approaches to ma naging and accessing scholarly research, including the use of emerging standards f or document management systems, such as Open Archives Initiative, Government Inform ation Locator Service, and Dublin Core Metadata Initiative. The representative of the Knowledge Integration Project with whom we spoke saw his role within the policy commun ity as developing strategies for the long-term acquisition and storage of "knowledge objects," reflecting a government commitment to using new technologies to increase th e play of research and other sources of information in the policy process. His concern t hat it was often difficult to locate and work with the data sets behind major research studi es suggested the need for developing
10 of 23far more comprehensive scholarly communication syst ems that would enable independent and critical analysis of research, as w ell as greater collaboration and reanalysis of data. As an experienced research read er, he was less than happy with how research was currently made available, both economi cally and structurally: "Until we define or restructure the Â‘ElsevierÂ’ economic busin ess model for publishing, we are going to be stuck with very high transaction costs for finding information.... We need both a common format standard and a common conceptu al standard." The inhibiting factor, in his eyes, is "the transaction cost of se arching on the Internet" which he finds "phenomenal." And while he tended to use materials that were only one or two years old, he also saw a place for keeping a "500-year time fr ame" in addressing major policy issues, pointing to an interest in both historical research, as well as research from earlier periods, as did others in our sample. Finally, this knowledge integration officer expressed a strong interest in a quality indicator or systems of endorsement that went beyond basic peer review, of the sort perhaps, used in bibliometrics to evaluate the "impact factor" of a journal (by how often its arti cles are cited). A second representative of the Policy Research Init iative, this time from the North American Linkages division, was very clear about th e foundation that research provides for policy. She felt that it did fall to the govern ment to strengthen scholarly research in areas of particular policy interest. The government could then draw on these researchers to build an advisory group who could be counted on to keep the government informed and ensure the quality of the resulting policies. I n terms of her reliance on the Internet as a source of research, she pointed out, as did other s, that assessing the quality of the research is as much an issue as accessing it: "You have to exercise a considerable amount of independent judgment about the veracity o f what you are reading." She believes that researchers in her divisions are incr easingly relying on the InternetÂ—where the quality of information is still difficult to as certainÂ—when they should be seeking out the known quantities in print sources. Her worry is that the Internet leaves readers with a misleading sense of having consulted, or at least b rowsed, all that there is to know in a particular field. Otherwise, it could mean that the Internet, at this point at least, may represent a shift in the focus of attention on rese archÂ—to what is readily available onlineÂ—with no real expansion in the consideration of what is known and no guarantee as to the quality of that knowledge.The Policy Research Initiative representative from the Sustainable Development group, Paul Halucha, offered a glimpse of a still very pow erful method of working directly with a small number of researchers to directly address t he issue at hand. He described how he brought together academics to identify eight releva nt trends in research on sustainability which resulted in an "interdisciplinary knowledge s tatement." He saw this process offering a "single window" into the government whic h researchers could use to bring their work to bear on policy. Yet it was exactly th at, a single window, and not an open door. It enabled direct engagement between scholar and policy official, but it was more of a closed process, and needs to be supplemented a t least, by more open processes. It was, to be fair, his way of dealing directly with t he knowledge demands of his work: "Government officials are faced continually with th e daily pressures of producing knowledge." He felt the need for better ways of man aging the deluge, as "information...is piling up...and there comes a point at which one ge ts into declining returns." He also worried about how the Internet does not afford acce ss to older materials, which would diminish "one of the classic public policy values" of "deliberation, rational deliberation." The challenge, then, is to devise systems that do n ot leave one reverting to narrower,
11 of 23traditional processes of consultation because the n ew is simply too much, too quickly. A third representative of the Policy Research Initi ative with whom we spoke was Michael Mackinnon, from the Social Cohesion Team. M ackinnon spoke of how the concept of "social cohesion"Â—which is concerned wit h the degree of peopleÂ’s economic, social and political participation in a societyÂ—was derived from European sociological research, which had been recently popularized on th is continent by Robert PutnamÂ’s work (2001). Mackinnon spoke of the need for better organized websites with taxonomic structures and tagged materials, as well as for res earch written in comprehensible language, with some way of verifying its status. He also pointed to the value of portal sites that served as well organized gateways to a w ide range of resources, commending Policy.ca, as well as Canadian Social Research Link s, which is run as a hobby by Gilles Sguin, a Human Resources Development Canada employ ee. (Note 17) While often satisfied with being able to read research abstract s, Mackinnon also showed an interest in having the opportunity to go deep and review th e actual data sets behind studies, as well as in being able to consult summaries, overvie ws and syntheses. While all of this would be "of significant interest to policy analyst s," he wondered who could afford to produce this level of research support, as an add-o n to existing systems, while what may be needed is a new system in which summaries, porta ls, and taxonomies are all part of the fundamental design of scholarly communication.The Canadian Health Services Research Foundation is a non-profit organization endowed by the federal government to support both b asic and practitioner research on management and policy issues in the area of health services and nursing. (Note 18) It also seeks to bring together decision-makers, polic ymakers, practitioners and researchers to facilitate the dissemination of this research. W e spoke with Michelle Campbell, Assistant Director, Knowledge Transfer Division, wh o works to ensure that the available knowledge gets to the policy researchers, decisionmakers, and administrators in CanadaÂ’s health systems. As part of this job, her g roup does translate research on occasion into common language, on the one hand, and has hired a "knowledge broker," on the other hand, to work with academics. She has learned through this work that while researchers feel they can draw no conclusions when they have only 20 per cent of the answer, policymakers feel that 20 percent of the an swer is about as good as it gets. Given than, policymakers are often frustrated by the rese archersÂ’ reluctance to make recommendations on the basis of the existing eviden ce. Campbell felt that the research abstract should cle arly present the implications of the study, for at that point, the policymaker decides w hether to examine the studyÂ’s conclusion, while only occasionally going back to t he findings and discussion. That said, she felt while it was important for policymakers to have access to complete studies, she also wanted a way of moving up from a study to a sy nthesis or meta-view of the larger research issues, given the range of pertinent resea rch: "The actual breadth of sources is huge...and no one can check them all or find one pl ace where an individual can check them all. Secondarily, the type of research is much more complex to do than traditional clinical intervention research, so that quality is much trickier of an issue." And this time, when the quality assurance issue of information on the Web came up, it was in reference to the "gray literature" which included relevant bu t unpublished reports and commentaries. And while Campbell would like to see more warranted research online, the efforts of her own agency are sometimes curtail ed in this regard. The Canadian Health Services Research Foundation has to be caref ul in its handling of the research it
12 of 23has sponsored so that it did not diminish that rese archÂ’s ultimate "publishibility." This could well result in limited public access to publi cly sponsored research. The research incentive system of publish-or-perish was working a gainst the very mission of the Foundation. There should be a way to provide the tr aditional quality control of peer review while providing wider access to this publicl y sponsored work on improving health care.Among the Librarians To complete the picture of how policy officials wor k with research, we visited the Ontario Legislative Assembly Library and spoke with two librarians and two research officers. Their job, as one of them put it, is "to help members [of the Legislative Assembly] improve the quality of debate." That is, the information they provided came into play after the policies had been drafted and w ere being considered by the Assembly. "WeÂ’re not often asked to help at the inchoate stag es of policy," one of the research officers told us. They saw the information serving as a background to the work of the Assembly, whether in explaining the policy to oppos ition party members or to the press. Using research to inform the discussion of policy, which was not raised by the policy analysts we interviewed, clearly plays a critical r ole in a democratic society for setting the scope and terms of the debate. One of the resea rch officers works extensively with academic research ("I have to keep up with politic al science research," he said. "I need the journals at my fingertips" ) while the other re search officer, who works with legal matters, deals more often with the legislation itse lf, mainly in comparisons of statues across jurisdictions, for example. For the former, greater online access to well-organized bodies of research would be welcome, especially aga inst the time constraints of having to prepare reports quickly that are supported by cu rrent research. And the currency of research by the time it reaches print, he noted, wa s often inadequate. The acquisitions manager of the library pointed out that increasing subscription costs have made it difficult for the library to retain academic serial s, and that online systems were currently being offered as bundled services that required pur chasing access to more than was felt warranted: "I donÂ’t want to buy the whole thing if I need only 10 per cent of it." It suggested that another sort of system was required to otherwise prevent a decline in the consultation of academic sources of information on policy matters.DiscussionThe interviews with these Canadian policy officials and related personnel make it clear that the Internet is now a favored source of inform ation within government. It is used to tap into the research that is consulted as part of the policy process. It is also clear that the research that is most easily accessible, through po rtals and open-access sites, is most often consulted, as policymakers referred to how re adily they were dissuaded from using pay-per-view and subscription services in their pur suit of knowledge. This means that they are tapping into a skewed and somewhat haphaza rd view of the current state of knowledge on a given topic. This could be seen as a further argument for establishing an "open access" economy of scholarly communication th at would make scholarship freely available online. (Note 19)Whatever form online scholarly publishing systems f inally take, nearly everyone we spoke with agreed on the need for a system that war ranted the status or credibility of the research, from clearly marking it as "peer reviewed (with an accompanying explanation
13 of 23of the process) to providing citation statistics, w hich indicated whether others have referred to the paper, and in what context. (Note 2 0) Yet there is more to research online than peer review. The Internet also offers pre-prin ts and working papers referring to research, which has yet to be reviewed, but these a re clearly marked with at least one set featuring a "warning" for "causal readers." (Note 2 1) Among the other suggestions I would draw from the p articipants comments is the need for non-academics to establish a coherent context f or a given study that enables readers (1) to review other studies by the author, as well as related studies both current and those going back decades, (2) to go deep into the body of the study, as far perhaps as the research instruments and raw data set where possibl e, (3) to rise up, above the specific study, to an overview or summary of the larger fiel d of inquiry, (4) to consult related resources, such as relevant court decisions, as one policy maker mentioned, or related policy in other jurisdictions, (5) to access glossa ries or reference materials that further open up the language and ideas to a wider readershi p, and (6) to interact with researchers, to comment on their work and consult d irectly with them, providing the researcher with a sense of that wider audience whic h will, in turn, give shape to how they express their findings.Each of these abilities to work with research is ce rtainly feasible in an online environment. For example, on giving some historical depth to the ability to locate related studies, the J-STOR project has been putting comple te sets of journals, dating back to the nineteenth-century online. (Note 22) For our pa rt, we drawn on these interviews to develop a "research support tool" that would provid e a way for readers to readily move from an individual research study to related resour ces not only in research but in policy, practices, the media and other areas (Figure 1). We are also working on improving the indexing of online research to enable more precise and accurate searching for academic resources. The next phase of the larger study is to test the viability of such a support system with the participants of this study, the maj ority of whom expressed a willingness to take this next step.
14 of 23 Figure 1. Research Support Tool Figure 1 represents an initial design for a tool th at would provide studies in the field of education with a greater context for interpretation judgment, and utilization by policymakers (and the public), based on the suggest ions made in the interviews in this study. A working demo of this tool can be found at http://pkp.ubc.ca/demos/rsttour/index.html As encouraging as this potential for improving poli cy officialsÂ’ greater engagement with research may be, there remains the political cautio n about researchÂ’s influence expressed by Anne Larson Schneider and Helen Ingram, whose wo rk I discussed above: "Where scientists and professionals have the greatest infl uence, however, the consequences are damaging to democratic values" (1997, p. 181). The increasing presence of research, through online technologies may well make citizens feel disempowered and "less capable of self-government" (p. 185) in the face of what Walter Lippmann identified many years ago as the "organized intelligence" nece ssary to run the modern state (1922). (Note 23) The policy officials with whom we spoke d id lend credence to Schneider and IngramÂ’s fear "that scientific and professional net works...have colonized government so that there are no public arenas but only bureaucrat ic maneuvering among privileged specialists" (1997, p. 188). Not only did we speak with government economists, for example, shaping policy, but more than one policy o fficial referred to working with a
15 of 23select group of academics who served as research fi lters and policy consultants. The danger of this limited consultation, outside of pub lic arenas, can be challenged, we believe, by improving the online presence of resear ch as part of knowledgeÂ’s public sphere. Creating open access to research would help policymakers, as well as the public, consult a much broader and more diverse range of in formation. With improved indexing, policymakers and the public would be able to locate innovative and fresh perspectives on specific and pressing issues, as well as ensure tha t they could consult contrary opinions and gain a sense of the range of approaches and opi nions on a topic. What is clear, from a democratic perspective, is th at improving policymakers access to research has to be balanced by similar improvements in public access. This represents the democratic check on researchÂ’s political impact factor, but it also promises to raise the level of public deliberation and the very quali ty of democratic processes. Improving both policy and public access to research entails m any of the same concerns with credibility, overview, integration, and context. In the United States, for example, scienceÂ’s political contribution has been officiall y channeled through advisory bodies, such as the National Academies (with the Royal Soci ety of Canada proposing a similar approach for the Canadian government). The Academie sÂ’ reports are made public and typically try to represent a coherent consensus amo ng experts, with due consideration given to policy implications. As reassuring as such reports can be to the public, Stephen HilgarterÂ’s investigations have revealed the degree to which they are shaped by "stage management and struggles over the enclosure and dis closure of information" (2000, p. 146). These carefully reviewed reports may still mi srepresent the state of dissent among researchers, just as the prestige and authority of these research reports can seem to squelch the deliberative and difficult public proce sses that make for enlightening, if sometimes exasperating, democratic experiences.The democratic impact of research is likely to be a dvanced, given what this sample of policy officials revealed in their interviews, by c reating a far more open, coherent, and integrated access to scholarship in ways that might serve policymakers and the public alike. The greater coherence and integration of sch olarly communication that might be achieved through these new technologies should not be directed at overwhelming policy processes with the dictates of a scientific consens us. Rather, these new systems should be concerned with bringing the rich diversity of va lues and findings of research into public view and play. These systems need to make re search more fully a part of a public sphere, marked by the constant contest of ideas, in the search for greater understanding of what is and can be known.As social science researchers, it will not do, I co ntend, to imagine that our work can stay safely removed from the messy world in which people live and govern. We need to see our work as part of that life. In a few short years the Internet has already increased the presence of research in the policy process. It has to a large extent replaced the traditional print sources, and well before this new communicati on medium has arrived at anything like a sufficiently well organized way of handling the publication of research. It falls to researchers, then, to take hold of the democratic p ossibilities, and to be guided by the clear cautions, in developing new systems for schol arly communication for both public and scholarly use.Acknowledgment
16 of 23I wish to thank Michael J. D. Sutton for his talent ed assistance with the interviews in this article, Anne White for her attentive reading of it the Max Bell Foundation for its support of this work, and the editor of this journa l for his fine editorial assistance.NotesSee Alexander and Pal (1998), Hague and Loader (199 9), Heeks (1999), and Wilhem (2000). The potential influence of this new technology on citizen consultation, for example, has only begun to be exp erimented with, and with mixed results. For our experiments with public and professional consultation on education policy, see the Public Knowledge Policy F orum under prototype websites of the Public Knowledge Project ( http://pkp.ubc.ca ) and Klinger (2001). 1. The American Educational Research Association, for example, lists roughly100 e-journals that offer free, full-text access, inclu ding the well-respected Educational Researcher and Teachers College Record ( http://aera-cr.ed.asu.edu/links.html ). RePEc: Research Papers in Economics provides open access to 102,000 working papers and 56,000 journal articles ( http://repec.org/ ). The University of Pennsylvania provides an excel lent guide to electronic journals in all fields ( http://www.library.upenn.edu/webbin5/resources/ejsp ublic5.cgi ). 2. On the recent new education bill, see Schemo (2002) Also Education policyÂ’s stake in research, at the U.S. federal level, is re flected in the U.S. Education DepartmentÂ’s 2001 fiscal allocation of $185 million for education research and $80 for education statistics. Though it pales befor e the National Institutes of HealthÂ’s $20.3 billion allocation or the Defense De partmentÂ’s $1.4 billion for "basic research," it represents a considerable and stable investment in creating knowledge that will presumably serve education poli cy and practice, and President Bush has proposed a 28 percent increase for educati onal research 2003 (Burd and Southwick, 2002). 3. On the challenges faced in the United States over e vidence-based education, see Loveless et al. (1999), and on its pertinence for education (Author, 2001). 4. See Friedman for a comparison of in-house bureaucra cy, private consultants, and academic sources for policy analysis, with the acad emic "loner" judged to be often out of touch with "shifting sands of policy terrain ," while their advice possesses a "somewhat abstract air" and they are "rarely availa ble" when needed (1987, p. 162). 5. It may also be worth noting, on the other hand, the "social functions of ignorance" among bureaucracies, which Moore and Tumin argued, preserves traditional values and fair competition, as well as privilege a nd stereotypes, giving it a certain value for policy officials over the headlong pursui t of knowledge (1964). 6. Also see Stephen Hilgartner, who has observed that "governments find expert advice to be an indispensable resource for formulat ing and justifying policy and, more subtly, for removing some issues from the poli tical domain by transforming them into technical questions" (2000, p. 146). 7. The National Institute of Child Health and Human De velopment have had considerable influence on education policy, althoug h not without controversy among researchers: see Allington and Woodside-Jiron (1999), with response from Mathes and Torgesen (2000); and see Taylor et. al. (2000) with response from Foorman et al. (2000). 8.
17 of 23Here Schneider and Ingram (1997, p. 154) cite Max W eberÂ’s caution in his essay, "Three Types of Legitimate Domination" (1978, pp. 9 91): "Bureaucracy naturally prefers a poorly informed, and hence powerless, par liament at least insofar as this ignorance is compatible with the bureaucracyÂ’s own interest." 9. Lippmann on the development of democracy in America : "The more enlightened directing minds have called in experts who were tra ined, or trained themselves, to make parts of this Great Society intelligible to t hose who manage it" (1922, p. 234). See Willinsky (2000, pp. 37-42, 100-105) and Aikens (1999). 10. See Willlinsky (2000b) as well as the Public Knowle dge Project ( http://pkp.ubc.ca ). 11. See the Public Knowledge Project website (fn. 10) f or access to the two designs. 12. Human Resources Development Canada ( http://www.hrdc-drhc.gc.ca ). 13. RePEc has some 160,000 items, including 60,000 onli ne ( http://ideas.uqam.ca ); Also see, J-STOR: The Scholarly Journal Archive ( http://www.jstor.org/ ) 14. Canadian International Development Agency ( http://www.acdi-cida.gc.ca/ ). 15. Policy Research Initiative ( http://policyresearch.schoolnet.ca ). 16. Policy.ca ( http://policy.ca ); Canadian Social Research Net ( http://www.canadiansocialresearch.net/ ). 17. Canadian Health Services Research Foundation ( http://www.chsrf.ca/ ). 18. See Public Library of Science ( http://www.publiclibraryofscience.org/ ) and Budapest Open Access Initiative ( http://www.soros.org/openaccess/ ). 19. See the NEC ResearchIndex for an online and open ac cess system for representing the actual context in which a study has been cited by another study ( http://citeseer.nj.nec.com/cs ). 20. See NetPrints: Clinical Medicine and Health Researc h set up by the British Medical Journal and Stanford University Self-Archiv ing Initiative which includes this statement boldly set up at the entrance to the site: "Warning: Articles posted on this site have not yet been accepted for publica tion by a peer reviewed journal. They are presented here mainly for the benefit of f ellow researchers. Casual readers should not act on their findings, and journ alists should be wary of reporting them" ( http://clinmed.netprints.org/ ). 21. J-STOR ( http://www.jstor.org/ ). 22. G. Scott Aikens sees the resolution of the Walter L ippmann and John Dewey struggle between expert and public as lying in the cyberspace potential of public forums and open systems: "Whist Deweyan systems acc ept the need for rich systems of organized intelligence in complex societ ies, these can become richer through the active engagement of experts in open an d free decision-making systems at the local, regional, and global level" ( 1999, p. 192). 23.ReferencesAikens, G. S. (1999). Deweyan systems in the Inform ation Age. In B. N. Hague & B. D. Loader (Eds.), Digital democracy: Discourse and decision making in the Information Age (pp. 179-194). New York: Routledge. Alexander, C. J. & Pal, L. A. (Eds.). (1998). Digital democracy: Policy and politics in the wired world Toronto ON: Oxford University Press. Allington, R. & Woodside-Jiron, H. (1999). The poli tics of literacy teaching: How "research" shaped educational policy. Educational Researcher 28 (8), 4-13.
18 of 23Barker, A. & Peters, B. G. (1993). Introduction: Sc ience politics and government. In A. Barker & B. G. Peters (Eds.), The politics of expert advice: Creating, using and manipulating scientific knowledge for public policy Pittsburgh PA: University of Pittsburgh Press. Burd, S. & Southwick, R. (2002, January 11). Congre ss approves increases of $250 in top Pell Grant and $3-billion for the NIH. Chronicle of Higher Education A32. Dewey, J. (1926). The public and its problems New York: Holt. Foorman, B. R., Fletcher, J. M., Francis, D. J., & Schatschneider, C. (2000). Response: Misrepresentation of research by other researchers. Educational Researcher 29 (6), 27-37. Friedman, J. (1987). Planning in the public domain: From policy to actio n Princeton NJ: Princeton University Press. Hague, B. N. & Loader, B. D., (Eds.). (1999). Digital democracy: Discourse and decision making in the information age London: Routledge. Heek, R. (Ed.). (1999). Reinventing government in the information age: Inte rnational practices in IT enabled public sector reform London: Routledge. Hillage, J., Pearson, R., Anderson, A. & Tamkin, P. (1998). Excellence in research on schools Department for Education and Employment, London. Hillgartner, S. (2000). Science on stage: Expert advice as public drama. Stanford CA: Stanford University Press. Humes, W. & Bryce, T (2001). Scholarship, research, and the evidential basis of policy development in education. British Journal of Educational Studies 49 (3) 329-352. Klinger, S. (2001) "Are they talking yet?" Online Discourse as Politic al Action in an Education Policy Forum Unpublished dissertation, University of British C olumbia. Lippmann, W. (1922). Public opinion New York: Free Press. Loveless, T., Peterson, P., Boruch, R., Cook, T., G ueron, J., Hyatt, H., & Mosteller, F. (1999). Can we make education policy on the basis of eviden ce? What constitutes high quality educational research and how can it be incorporated into policymaking ? A Brookings Press Forum. Washington DC: The Brook ings Institution. Mathes, P. G. & Torgesen, J. K. (2000). A response to Allington and Woodside-Jiron. Educational Researcher 29 (6), 4-14. Moore, W. E. & Tumin, M. M. (1964). Some social fun ctions of ignorance. In B. Rosenberg, I. Gerver, & F. W. Howton (Eds.), Mass society in crisis: Social problems and social pathology New York: Macmillan. >Ozga, J. (2000). Policy research in educational settings Buckingham, UK: Open
19 of 23University Press. Putman, R. D. (2001). Bowling alone: The collapse and revival of American community New York: Simon and Schuster. Schemo, D. J. (2002, January 9). Education bill urg es new emphasis on phonics as method for teaching reading, New York Time s A7. Schneider, A. L. & Ingram, H. (1997). Policy design for democracy Lawrence KA: University Press of Kansas. Simon, H. (1945). Administrative Behavior New York: Free Press. Taylor, B. M., Anderson, R. C., Au, K. H., & Raphae l, T. E. (2000). Discretion in the Translation of Research to Policy: A Case from Begi nning Reading. Educational Researcher 29 (6), 16-26. Weber, M. (1978). Three types of legitimate dominat ion. In C. Roth & C. Wittch (Eds.), Economy and society: An outline of interpretive soc iology Berkeley, CA: University of California Press. Weiss, C. (1998). Interview by Miranda Christou Cambridge MA: Harvard University School of Education. Available at http://www.gse.harvard.edu/~oie/weiss.html Weiss, C. (1991). Policy research: data, ideas, or arguments. In P. Wagner, C. H. Weiss, B. Wittrock & H. Wollman (Eds.), Social sciences and modern states: National experiences and theoretical crossroads (pp. 307-332). Cambridge, UK: Cambridge University Press. Wilhelm, A. G. (2000). Democracy in a digital age: Challenges to political life in cyberspace New York: Routledge. Willinsky. J. (2000a). If only we knew: Increasing the public value of soc ial science research New York: Routledge. Willinsky. J. (2000b). Proposing a Knowledge Exchan ge Model for scholarly publishing. Current Issues in Education 3 (6). Available: http://cie.ed.asu.edu/volume3/number6 /. Willinsky. J. & Wolfson, L. (2001). The indexing of scholarly journals: A tipping point for publishing reform? Journal of Electronic Publishing 7 (2). Available at http://www.press.umich.edu/jep/ .About the AuthorJohn WillinskyDepartment of Language and Literacy EducationUniversity of British ColumbiaVancouver, BC, Canada V6T 1Z4 firstname.lastname@example.org John Willinsky is the Pacific Press Professor of Li teracy and Technology at the
20 of 23University of British Columbia where he directs the Public Knowledge Project. He is the author of a number of books, including Learning to Divide the World which won two outstanding books awards, and If Only They Knew: Increasing the Public Value of t he Social Science Research .AppendixInterview Schedule Used With Policymakers1. Background 1.1 Introduce UBC PKP Project 1.2 Describe your organization, its mission, and your role within it 1.3 What role does your organization play in pol icy making? 2. General regard for scholarly research 2.1 What role does scholarly (university-produce d) research currently play in your policy decisions? 2.2 What is the most frustrating thing you find about accessing scholarly research? 2.2.1 Is there a problem with interpreting the lang uage of the material? 2.2.2 Is there a problem with searching/locating ma terial? 2.2.3 Is there a problem with Internet connection/f irewalls? 2.2.4 Is there a problem with the ability to access /capture/store the information? 2.2.4 Is it a problem with intellectual property ri ghts/copyright? 2.3 What possible change could make that frustra tion go away? 3. Electronically Available Research 3.1 Describe your organization's use of electron ic information sources? 3.1.1 Describe your own personal use 3.1.2 What are your leading sources of online infor mation? 3.2 How has your use/approach changed in recent years? 3.3 How do you foresee such use changing in the coming years? 3.4 What are the promising aspects of increased access and interactivity? 3.5 Have you experienced any important obstacles or issues surrounding: 3.5.1 Comprehensiveness 18.104.22.168 How difficult is it to locate and continue to work with 1 or 2 reliable starting points? 22.214.171.124 Is access to full text versions of research material mandatory/desirable? 126.96.36.199 How much depth/richness must a sit e have in order for you to return to it? How do you determine this? 188.8.131.52 Is detailed, current contact infor mation a requirement for work featured on a site? 184.108.40.206.1 By institution? 220.127.116.11.2 By individual? 18.104.22.168 When do you know you have covered a topic in enough depth to proceed with policy analysis? 22.214.171.124 Is a breakdown by jurisdiction imp ortant? 126.96.36.199.1 Local 188.8.131.52.2 Regional 184.108.40.206.3 Provincial 220.127.116.11.4 Federal 18.104.22.168.5 International?
21 of 23 3.5.2 Accessibility 22.214.171.124 Does source language of material p rovide any constraint? 126.96.36.199 Does the academic language of the research discourage interpretation? 188.8.131.52 Are hardcopy or digital indices us ed to locate relevant research? 184.108.40.206 Is it more common to want material about a "study" rather than the "study" itself? 220.127.116.11 Is a hardcopy mandatory/desirable, or would a digital copy be sufficient? 18.104.22.168 Does the digital format of researc h material inhibit your work? 22.214.171.124.1 Adobe pdf 126.96.36.199.2 MS-Word 188.8.131.52.3 Postscript 184.108.40.206.4 HTML? 220.127.116.11 Do you experience any connectivity constraints, (dial-up vs. HS lines, lack of connection)? 18.104.22.168 How would you rate your on-line se arch skills? 22.214.171.124.1 Novice 126.96.36.199.2 Experienced 188.8.131.52.3 Master? 184.108.40.206 Do you use any current awarenes s services (i.e., email alerts, etc.)? 220.127.116.11 Any intellectual property prob lems encountered around copyright & digital rights? 3.5.3 Currency 18.104.22.168 Is the availability of current mat erial mandatory/desirable? 22.214.171.124 Generally speaking, how current is material you presently use? 126.96.36.199.1 <= 1 month old 188.8.131.52.2 <= 3 months old 184.108.40.206.3 <= 6 months old 220.127.116.11.4 <=1 year old 18.104.22.168.5 > 1 year old 22.214.171.124 How old should historical or ar chival material be in order to make it relevant to your work? 3.5.4 Reliability 126.96.36.199 How do you verify the credibility of both the author and the content of research material? 188.8.131.52 Would direct/immediate contact wit h journal reviewers be important for reliability? 184.108.40.206 Do you believe that research which is available on-line is inherently inferior to hardcopy?4. The Knowledge Exchange Model 4.1 Introduce the Knowledge Exchange Model 4.1.1 Walkthrough Model 4.1.2 Walkthrough http://www.Policy.ca 220.127.116.11 Are there any suggested changes yo u could make for http://www.Policy.ca? 4.2 Given what you know about the Knowledge Cube 4.2.1 Do you think it could help you access scholar ly research? 4.2.2 Why or why not? 4.3 Imagine for a moment that all obstacles to i mplementing such a model had somehow been overcome. 4.3.1 How would the model be different? 4.3.2 What would be its general characteristics?
22 of 23 4.4 Would you be willing to review the final des ign(s) and the resulting prototype(s)?Copyright 2003 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, email@example.com or reach him at College of Education, Arizona State University, Tempe, AZ 8 5287-2411. The Commentary Editor is Casey D. Cobb: firstname.lastname@example.org .EPAA Editorial Board Michael W. Apple University of Wisconsin David C. Berliner Arizona State University Greg Camilli Rutgers University Linda Darling-Hammond Stanford University Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Gustavo E. Fischman California State UniveristyÂ–Los Angeles Richard Garlikov Birmingham, Alabama Thomas F. Green Syracuse University Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute ofTechnology Patricia Fey Jarvis Seattle, Washington Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Les McLean University of Toronto Heinrich Mintrop University of California, Los Angeles Michele Moses Arizona State University Gary Orfield Harvard University Anthony G. Rud Jr. Purdue University Jay Paredes Scribner University of Missouri Michael Scriven University of Auckland Lorrie A. Shepard University of Colorado, Boulder Robert E. Stake University of IllinoisÂ—UC Kevin Welner University of Colorado, Boulder Terrence G. Wiley Arizona State University John Willinsky University of British ColumbiaEPAA Spanish Language Editorial Board
23 of 23 Associate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico email@example.com Adrin Acosta (Mxico) Universidad de Guadalajaraadrianacosta@compuserve.com J. Flix Angulo Rasco (Spain) Universidad de Cdizfelix.firstname.lastname@example.org Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho dis1.cide.mx Alejandro Canales (Mxico) Universidad Nacional Autnoma deMxicocanalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State Universitycasanova@asu.edu Jos Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es Erwin Epstein (U.S.A.) Loyola University of ChicagoEepstein@luc.edu Josu Gonzlez (U.S.A.) Arizona State Universityjosue@asu.edu Rollin Kent (Mxico)Departamento de InvestigacinEducativa-DIE/CINVESTAVrkent@gemtel.com.mx email@example.com Mara Beatriz Luce (Brazil)Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.brJavier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mxMarcela Mollis (Argentina)Universidad de Buenos Airesmmollis@filo.uba.ar Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mxAngel Ignacio Prez Gmez (Spain)Universidad de Mlagaaiperez@uma.es Daniel Schugurensky (Argentina-Canad)OISE/UT, Canadadschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil)American Institutes for ResesarchÂ–Brazil(AIRBrasil) firstname.lastname@example.org Jurjo Torres Santom (Spain)Universidad de A Coruajurjo@udc.es Carlos Alberto Torres (U.S.A.)University of California, Los Angelestorres@gseisucla.edu