xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20019999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00244
Educational policy analysis archives.
n Vol. 9, no. 47 (November 19, 2001).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c November 19, 2001
Knowledge management for educational information systems : what is the state of the field? / Christopher A. Thorn.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 9issue 47series Year mods:caption 20012001Month November11Day 1919mods:originInfo mods:dateIssued iso8601 2001-11-19
1 of 32 Education Policy Analysis Archives Volume 9 Number 47November 19, 2001ISSN 1068-2341 A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright 2001, the EDUCATION POLICY ANALYSIS ARCHIVES Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education .Knowledge Management for Educational Information Sy stems: What Is the State of the Field? Christopher A. Thorn Wisconsin Center for Education ResearchCitation: Thorn, C.A. (2001, November 19). Knowledg e Management for Educational Information Systems: What Is the State of the Field?, Education Policy Analysis Archives 9 (47). Retrieved [date] from http://epaa.asu.edu/epaa/v9n47/.AbstractThis article explores the application of Knowledge Management (KM) techniques to educational information systemsÂ—parti cularly in support of systemic reform efforts. The first section defin es knowledge and its relationship to information and data. There is also a discussion of various goals that might be pursued by organization s using KM techniques. The second section explores some of the fundamental design elements of an educational KM system. These include questions surrounding the unit of analysis, distributed compu ter resources, and organizational characteristics of successful KM eff orts. Section three outlines the benefits that organizations expect to gain by investing in KM. Section four is a case history of the introduct ion of a district-level data system and the parallel efforts to support the aggregation and
2 of 32reporting of high-stakes educational outcomes for 8 th grade students in the Milwaukee Public Schools (MPS) district. Finall y, there are some preliminary conclusions about the capacity of an ur ban district in a complex policy environment to respond to the knowle dge management needs of a decentralized system.IntroductionThis article considers the critical role played by the "management of knowledge" in education, and, specifically, in efforts at educati onal reform. Schools and districts in the United States face mounting pressure to demonstrate the measurable effects of their practices to legislators, parents, the business wor ld, and the public at large. 1 This fact by itself adds to the information management burden pl aced on the educational system. However, other changes, such as the rise in student mobility and ethnic diversity, have increased the complexity of already complex school data systems. This complexity is rooted in the operational requirements of implement ing and assessing instructional interventions and the interactions between racial, ethnic, and socioeconomic characteristics on student learning. The permutatio ns of analytically distinct groups increase knowledge management burdens at an alarmin g rate. As growing diversity has been matched by increasing disparities in educational outcomes between a number of groupsÂ—poor and afflue nt, white and minority, urban and suburbanÂ—a patchwork of uncoordinated programs have been introduced at all education levels by a variety of government entitie s. This ad hoc form of policymaking often results in programs that at best do not reinf orce one another and at worst actively work to undermine each other. For example, the move to base access to Title I resources on economic rather than academic need was an import ant shift in emphasis that allocated resources more equitably. Unfortunately, a number of local programs in many urban school districts relied on the annual testing funded by Title I as an important component in assessing overall system performance a nd often used the data to inform local goals. A more promising response, which has d eveloped over the past decade, has been what is referred to as systemic or standards-based reform Systemic reform, according to U.S. Deputy Secretary of Education Mar shall Smith, is typically based on state-level reforms that implement more rigorous co ntent and performance standards across grades and disciplines. 2 Systemic reform requires that curricular material and assessments be aligned with these standards. Preser vice teacher education and teacher professional development must also support these go als. Finally, funding, technology, physical plant, and human resources must all be all ocated in such a way that each group has equal access to the things it needs in order to succeed. To improve student achievement and to close the equity gap are the ult imate goals of systemic reform efforts, but supporters of systemic reform believe these goals can only be achieved by improving all aspects of the educational system.Major systemic reform initiatives at the national l evel seek to strengthen teacher education, reinvigorate the development of high-str ength curriculum, and promulgate disciplinary standards. Some of these programs oper ate under the auspices of the U.S. Department of Education and the National Science Fo undation. 3 The majority of reforms are state-level initiatives to improve the performance of students in specific subject areas or grade ranges such as elementary re ading or middle school mathematics (Armstrong, 1999). Individual districts also engage in reform initiatives that seek to
3 of 32motivate particular schools or groups of students t o meet individual standard components. Finally, individual schools and classro om teachers work to develop lesson plans to teach concepts based on grade-level standa rds. These trends and the consequent emergence of more data-rich environments raise the need for new technologies and new management techniques for coping with complexity. T hree technical issues in particular appear to be especially crucial.First is the problem of accurately identifying poli cy targets. Identifying the target group and the desired outcomes for a particular reform is necessary if one is to describe, analyze, and locate reform efforts within education al systems. Existing organizational identifiers such as school, classroom, or demograph ic data are often inadequate to isolate for study or evaluation students or teachers who pa rticipated in particular programs or received distinct treatments. The ability to accura tely compare analytically distinct groups is vital if one is to assess the impact of s ystemic reform. Second is the problem of managing data. It is evide nt that successful systemic reform will depend on access to and effective use of large amounts of data. This means that the quantity, timeliness, and level of detail of the da ta needed from decision-support systems will only increase. Proponents of systemic reform p oint to the importance of a process model and evaluation framework for assessing progra ms (Clune, 1998). The process of systemic reform must include actors from all levels and it must include an awareness of resources and barriers confronting actors across ed ucational roles. The focus on process also points to the importance of quality indicators Quality measures might include a detailed analysis of curriculum goals and well-unde rstood, publicly available education standards. High quality instruction demonstrates an alignment between standards, curriculum, and assessment. The difficulty of measu ring differential interventions is compounded by the need to gauge the quality and tho roughness of new, robust curricula and to obtain more detailed analyses of student pro gress. Third is the problem of metrics, i.e., multiple-cho ice versus authentic-performance tasks. Demands for more authentic assessments have led to calls for new metrics of student performance relative to standards, as well as an em phasis on measuring a particular student's attainment of individual proficiencies. 4 The new standards-based approaches have led to the creation of new sorts of testing th at go beyond the typical "fill in the bubble" standardized assessments. The introduction of multifaceted testing regimes (including traditional standardized tests, as well as performanceand portfolio-based assessments) both increases the complexity of the i ndividual instruments and creates new requirements for the underlying data systems th at must both record and provide an analytical environment for the data. The increasing sophistication of assessment practices calls for a parallel development in the a rena of information-management strategies.The problems described above are the technical aspe cts of fundamental research questionsÂ—What to study? How to aggregate the data? And what are the appropriate measures? As our understanding of the educational p rocess deepens, our technical capacity to collect, manage, and analyze data must keep pace. This article seeks to apply insights from the growi ng body of literature on knowledge management (based primarily on research coming out of United States and European business schools) 5 to the specific case of systemic reform in U.S. ed ucation. Knowledge
4 of 32management (KM) strategies can serve as valuable to ols for decision makers at all levels of the educational system. In this paper, I focus o n outlining the distinctive features of knowledge management, identifying the characteristi cs of successful KM efforts and exploring the usefulness of KM for making important educational decisions. I conclude with a discussion of the role of information techno logy (as a component of a KM system) in the implementation of high-stakes accoun tability for 8th grade students in one particular urban district, that of Milwaukee, Wisco nsin.I. Knowledge Management (KM): Definitions and ScopeBefore considering the characteristics of knowledge management, it is important to note the differences between data, information, and knowledge A recent article by Laura Empson, which presents the argument that knowledge is a product that is built from data and information, provides the following definitions : [I]t is perhaps easiest to understand knowledge in terms of what it is not. It is not data and it is not information. Data are obj ective facts, presented without any judgment or context. Data becomes infor mation when it is categorised, analysed, summarized, and placed in co ntext. Information therefore is data endowed with relevanc e and purpose. Information develops into knowledge when it is used to make comparisons, assess consequences, establish connections, and eng age in a dialogue. Knowledge can, therefore, be seen as information th at comes laden with experience, judgment, intuition, and values. (Empso n, 1999) There is a clear progression along the path in whic h value is added to data, as context is combined with it to create information. A further t ransformation occurs when human experience is added to information to make value ju dgments about, and comparisons of, different information. The progression from data to knowledge can be seen both as a temporal process in which data, imported into a system's architecture, aggregates individual facts into summaries and averages that are then presented in a n appropriate context. In an educational setting, this might be a report of stud ent test performance by grade, ethnicity, race, and gender. The addition of deeper contextual information about local school leadership, particular organizational charac teristics, or other less quantifiable factors can be combined with mechanistically genera ted test-score results to describe variance in outcomes that could not be extracted fr om the more traditional reports. It is this application of personal knowledge and of welldesigned models that differentiates information systems from knowledge systems.Knowledge Management as the Use of Data and Informa tion As mentioned above, KM is a follow-up to informatio n management. The bulk of the literature on good information system design focuse s on the technologies and processes used to acquire and manage data. When describing the breadth of approaches to effective knowledge use a number of authors describe a range of system fu nctions from data management to knowledge creation and applicati on. Information management lies somewhere between the two poles of data management and knowledge management. Another way to think about KM is that it is the use or application of information.
5 of 32In districts, the student data system provides the core of such a system. However, this focus on information systems and tools for aggregat ion and on KM as the application of information should not imply some type of computerbased system that is somehow imbued with deep contextual knowledge of the organi zation. This is not just a question of the technologies employed. An important role is played by institutional culture. For example, a district with a collaborative model of i nteraction between schools will typically display a far greater capacity to develop robust analyses of school-level processes and needs than a district lacking such a model. Davenport has identified several important questions that may be helpful in efforts to clarify an organization's approach to the use of knowledge management:Does an organization's culture reward decisions and actions according to how people use and share their knowledge? Or is it content with th e widespread use of intuition and guesswork at the expense of organizing people and p rocesses to apply the best knowledge, experience, and skills to projects and t asks? (Davenport & Davenport, 1999) The Davenports point to the importance of organizat ional culture in enabling or blocking the use of knowledge. Cultures that support knowled ge accumulation and application will be the most effective, efficient organizations Organizational structures and processes provide a window into the value a knowled ge management system will return to any implementer. A willingness to engage in prob lem-solving processes and share information with "outsiders" is an important resour ce for enabling knowledge management efforts.The Objectives of Knowledge ManagementOne of the important questions that an organization evaluating its effectiveness needs to answer is, What is the goal of this organization's knowledge management strategy? Davenport and his colleagues conducted a study of 3 1 KM projects across 24 companies. The authors identified four broad types of objectiv es with different subtypes: Create knowledge repositoriesÂ– a) external knowledg e (competitive intelligence, market data, surveys, etc.), b) structured internal knowledge (reports, marketing materials, techniques and methods), and c) informal internal knowledge (discussion databases of Â‘know how' or Â‘lessons lea rned'). In an educational setting, curriculum aids might be thought of as knowledge repositories. For example, the Milwaukee Public Sch ools Curriculum Design Assistant 6 (CDA) is both a source of documentationÂ—standards, learning goals, etc.Â—and a repository for instructional plans based on this documentation. These lesson plans can be stored in the system and shared with others electronically to provide a knowledge base for a wider audience. 1. Improve knowledge accessthrough a) technical expert referral, b) expert networks used for staffing based on individual competencies, and c) turn-key video conferencing to foster easy access to [geographical ly] distributed experts. Examples of this sort in public school education ar e probably rare, but the Community of Science online database 7 is a data and communication resource that functions well in education research: it links researchers, research intuitions, 2.
6 of 32and funders together. The purpose of the Community of Science online database is to reduce the barriers for those seeking funding fo r research, as well as to reduce the difficulty of funders in locating qualified res earchers. Enhance the knowledge environment Â–a) change organi zational norms and values related to knowledge in order to encourage knowledg e use and knowledge sharing, b) customers may be asked to rate their provider's expertise. This objective focuses on the creation of a technol ogical environment that will contribute to the social transformation of an organ ization. Anderson Consulting used this sort of approach to radically shift norms of information sharing and use among its consulting staff (Graham, Osgood, & Karre n, 1998; Greengard, 1998). In the consulting business, there are traditionally strong norms about keeping personal expertise personalÂ—it represents a large p ortion of individual competitive advantage. Anderson Consulting wanted t o reverse this behavior and reward those who shared information with other cons ultants within the organization. The company began to make participati on in an online, e-mail-based, problem-solving environment mandatory Eventually the bar was raised further and pay and promotion were linked to the number, quality, and immediacy of an individual's responses. This approa ch was draconian, but it was successful in building both a compelling repository of problem-solving information and shifting or overturning a strong no rm against sharing information. This approach models an important tool for accumula ting and diffusing successful educational practices. The call for methods of repl icating successful programs and school initiatives that "beat the odds" could be ad dressed with a system that improves communication within educational systems. 3. Manage knowledge as an assetÂ– a) attempt to measure the contribution of knowledge to bottom line success. (Davenport, DeLon g, & Beers, 1998, pp. 45-48)While this final KM objective sounds the most compe lling, it is also the most difficult to operationalize. Even firms with excell ent data-management practices and sophisticated conceptions of return on investme nt have difficulty assessing the return on intangibles. Learning Landscape 8 is an example of one such effort in education. This system was an outgrowth of the Conn ecticut Academy's NSF-sponsored Statewide Systemic Initiative. The Co nnecticut Academy worked together with the consulting firm, KPMG, to develop a data warehouse environment based on the National Center for Educat ion Statistics' core data elements. The stated purpose of the project was to transform data into knowledge to improve student achievement and teacher quality. This project was abandoned after it became clear that the development costs wo uld exceed what most districts would be able to afford. A new firm has been establ ished, EdExplore, which has the same KM goals but is much more limited in scope and has implementation costs in line with district resources. EdExplore re mains focused on bringing cutting-edge approaches to data warehousing to the evaluation of student and teacher performance. 9 4. These different goals are not mutually exclusive. D avenport states that most projects his team studied were focused on one of these goals, bu t many had features of the other
7 of 32goals interwoven into their projects (Davenport, De Long, & Beers, 1998; Davenport & Prusak, 1998). In addition, the first three objecti ves can be seen as constituting a feedback loop. Repositories must be built. Then, th ese repositories are only useful if users have efficient access to the knowledge contai ned in them. Finally, the use of knowledge in an organization will be enhanced by th e creation of an environment that supports this use. This knowledge-friendly environment will, in turn, demand higher levels of sophistication in the knowledge repositor y, thus closing the loop. The fourth goalÂ—determining the return on investmen t in knowledge managementÂ—is closely related to efforts that attempt to determin e the return on investment from spending on early research and development. Many an alysts have struggled with this vexing issue, which remains only partly addressed. There are so many unquantifiable, human elements in a KM system that it may be very d ifficult to come up with metrics that are generalizable.II. Knowledge Networks and Educational SystemsThere are several problems that must be faced by an y educational system that attempts to create a knowledge management system. The first pro blem is to determine the appropriate level of analysis the system is designe d to support. Another is that of differential access to computing power and the tech nical and analytical skills of the knowledge consumers within the system. In an educat ional setting, users representing students, classroom teachers, principals, and distr ict administration should be involved in the design of a system destined to support impor tant instructional and policy-level decision making. It is only by including users at a ll levels of the knowledge system that designers will have the input necessary to grapple with the problems identified above. The arguments of the authors cited thus far in supp ort of knowledge management systems provide very little insight into the exact analytical approaches one would use in any particular scenario, since so much depends on t he organizational form, sector, or level of analysis. The discussions in the literatur e focus instead on families of tools designed for pattern detection and predictive model ing. It is important to bring these together with the experience of those working in th e specific domain to identify the important dimensions of knowledge in that field, si nce we have defined knowledge as the application of information in context.A crucial feature of educational systems is that th ey are made up of a number of nested systems or organizations. In analytical terms, this can also be described as levels or units of analysis In education, these levels range from the federal level, through the state, district, school, classroom, and student levels. Re porting and analytical needs differ from level to level as do the relevant time scales. For example, there are classroom needs for lesson planning and local testing. At the school le vel, Title 1, free lunch, state-mandated testing, and other mandated programs are focal poin ts of data-management issues. The reporting requirements are as numerous as the fundi ng sources at each level of the organization. Analytical needs differ, but are pres ent at every level of the system. A robust knowledge management system must reflect the information and knowledge management needs of all levels. In particular, data must be gathered at a level of aggregation appropriate to the user with the most f ine-grained analytical needs.
8 of 32The Level-of-Analysis ProblemThe most common focus of school information systems is on the schooland district-level reports produced by a central inform ation technology department. In the case of school district-level knowledge management systems, one might attempt to implement several KM systems as described above by Davenport. One could argue that schools are groups of professionals with both proce ss and content knowledge. One organizational model used to describe reform-based schoolsÂ—"communities of practice"Â—would suggest that they would be more lik ely to focus on enhancing a knowledge environment (Snyder, 2000; Wenger, 1998). Operating under this model, schools would be less likely to accumulate knowledg e for its own sake. They would focus, instead, on sharing knowledge among a group of professionals. This would be consistent with the view that schools are communiti es of learners. A KM system that supports such behavior would be both a repository o f successful practices and a system for conveying positive norms associated with sharin g knowledge. The importance of the level of analysis comes to th e forefront of any effort to describe and apply KM principles. For example, district-leve l functions might include analysis of the quality of the data in the system (this may be in formal terms of validity and/or reliability, as well as of alignment of the metrics in use and the learning or performance standards). District-level analysis might also focu s on the curricula that are in use across the district and assess their relative effectivenes s. These district-level functions call for nested hierarchies of approaches to systems design that support collection of the relevant data and aggregation of these data to the appropria te level of analysis. The identification of distinct levels in an organiz ation with different KM needs is even more vital if one considers the increasing demands technology places on individual units in an organization. Boisot argues that the traditio nal neoclassical economics concept of a linear relationship among the different factors of production fails to adequately explain the returns of knowledge (Boisot, 1998). Boisot poi nts out that as one moves up within an organization, the number of elements that must b e integrated in order to produce a good or service has probably not increased dramatic ally in recent years. However, the pace at which new technologies and needs are introd uced and old technologies become obsolete has increased dramatically (Boisot, 1998). This rapid pace of change in the factors of production places a premium on the abili ty of individuals within an organization to track change and respond to it. Thi s ability is one of the key features of successful KM.Unfortunately, this is an area in which traditional information management systems (such as SASIxpÂ™ or ABACUSxpÂ™ from NCS or eScholar from IBM and Vision Associates) are particularly weak in general. 10 Data warehouse systems are excellent tools for making complex selections of data from ma ny different sources. However, there are no good inferential or predictive models in place within commercial school decision support systems for modeling school or stu dent behavior in real time. This is not because there is not a rich understanding of st udent achievement. Rather, the cost and complexity of real-time KM systems represent a major obstacle to schools or districts taking advantage of this technology. Scho ols have lagged behind industry in replacing paper-based clerical and business functio ns (often referred to as back office functions) with technology-based systems. They ofte n lack the physical infrastructure
9 of 32necessary for communication between numerous geogra phically dispersed buildings. Increasing Access to Computing PowerThe rise of ubiquitous computing which places relatively inexpensive devices for information storage and manipulation on every desk, has made the conversion to technology-based systems possible. This trend is no w finding its way into schools as they move away from using computers for rote learni ng to employ them as information tools. This is not simply a centralization-decentra lization issueÂ—i.e., the spread of technology from the district to the school and clas sroom. These resources can serve additional goals outside of instruction. A personal computer is both an information-gathering and an information-manipulati on tool. The availability of computing at all levels of the educational system c an be a focal point of change. There are certainly budget, security, and managemen t implications in broadening access to sophisticated computing technologies. However, t he need for information transcends district governance issues. Moves toward greater lo cal autonomy and responsibilityÂ—whether in response to school vouche rs, charter initiatives, or other pressuresÂ—mean that there will be an increasing nee d for local analytical capacity. The need for well-informed local decision-making at the classroom level is not unique to any particular organizational form, however, such as bl ock scheduling, or multi-age classrooms. It exists in both flat and hierarchical organizations. The level at which decisions are made may differ ac ross organization types, but there must still be robust data-collection and knowledgedissemination mechanisms at all levels. This is particularly important given the di fferent uses of individual data elements at different levels of aggregation and at different levels of the organization. Even at the school level, one can demonstrate the utility of co mparing groups such as students who are bussed versus not bussed, or students participa ting in a specific after-school program versus students who do not. Traditional aggregation s of data in such instances prove inadequate.Student Data as the Basic Unit of AnalysisThe problems outlined above suggest several possibl e avenues of system design that would lead to more appropriate data structures and produce more useful knowledge in an education knowledge management setting. The focus o n student learning, and the persistent gaps in student achievement described ab ove, should suggest to system designers that knowledge management systems should be student focused. What this means is that individual student data must comprise the basic building block of any knowledge management system in education. This incl udes data about the students themselves (test scores, demographics, attendance, etc.), as well as data about treatments or interventions intended to influence student outc omes. So, for example, professional development efforts aimed at improving the teaching of reading comprehension would need to be tracked, since these should have an impa ct on student learning. This implies that the school or instructor should collect the da ta locally and that the data should be used locally to inform teaching and targeted interv ention programs. This is an important point that is often overlooked. The most basic comm on unit of analysis (sometimes referred to as an atomistic unit) should drive data-collection efforts and pro cesses. If the student is the smallest analytical entity that will be studied, then attributes of that entity
10 of 32should be collected. In schools, this means that da ta should be gathered at the source. If schools and classrooms are where learning is produc ed, then local teachers and administrators need to collect and understand the d ata that will be used to measure that learning.At the same time, however, districts need to integr ate increasing demands for accountability, both fiscal and educational. Increa sing scrutiny at the district level implies an entirely different focus for information system design. In this district accountability model, data are aggregated up to hig her levels, such as the school or program, and are used to justify administrative or managerial decisions that affect large numbers of students and schools. This does not simp ly mean a reliance on central capacity. Knowledge management should be responsive to needs at all levels of the system. The problem is that the events and outcomes being monitored all tend to occur at the lowest levels of aggregationÂ—the student.Even in the most centralized system, instructional outcomes occur at the individual level. However, most district accountability measur es and goals are aggregated to higher levels of the organization. Data on student and tea cher absence, student program participation, teacher professional development act ivities and test scores need to be available at the lowest level for aggregation to an y meaningful unitÂ—classroom, grade, neighborhood, or other coherent grouping. This mean s that only individual-level data will serve. This is important because reform effort s must work on multiple levels. It is also vital that the system be transparent in order to combat efforts to cheat or otherwise tamper with the data. Data collected at the student or classroom level need to be aggregated and fed upwards to higher levels of the educational accountability structure to inform system decisions made at those levels. A system without proper security in place could be subject to manipulation by school-le vel actors. Again, this is not an argument for or against centralization, per se Rather, one must recognize that there are knowledge management decisions that are appropriate and possible at each level of an organization. Data structures and analytical tools should reflect this reality. This divergence between existing data systems and d ata needs points to a gap between new technological capabilities and the policy envir onment that would enable schools to actively collect relevant information and put it to use in a KM system. When the model of change targets student learning, it is essential to focus on identifying those innovations (technical and organizational) that bes t serve schools and classroom teachers. Most existing models of school governance do not provide for data analysis at the classroom level. Often there is also a gap betw een technical capacity and technical feasibility. Most educational systems have very lit tle technical capacity. For example, teachers may not have easy access to the Internet, or district computer staff may not have much experience supporting the analysis of problems relevant at the classroom level. There must be collection and delivery systems in pl ace. There must also be analytical support for complex problems that are not solved by looking at simple bivariate comparisons. What technological capacity does a sch ool system need? That depends on the system's goals.Making instructional decisions in an individual cla ssroom requires a sophisticated understanding of student attributes and of what ind ividual curricular units can provide. The ability to generalize to a larger population is not a relevant concern in this context. At the school level, however, a principal may be ve ry interested in how a new professional development activity is affecting stud ent performance relative to other
11 of 32classes in the same building, or in comparison to o ther schools across the district that did or did not participate in the same training. The to ols and data needed for within-classroom analysis are quite different from those necessary for a cross-school comparison. Differing forms of organizational analy sis call for diverse forms of KM system implementation. One faces similar challenges at any level of aggregation, from the classroom through the district level to the fed eral level. This discussion does not directly address the exist ence or quality of data needed for analysis. The lack of reliable data for school and student evaluation continues to be a vexing problem. Without a solid analytical model an d the appropriate KM infrastructure to collect and aggregate the data, no district of a ny size would be able to implement a robust evaluation system. For example, many distric ts embrace an assessment policy that focuses on individual grade cohorts. This has led t o the construction of information systems and testing plans that do not support longi tudinal analysis of individual students. However, a system that allowed for accurate and fai r evaluation at all levels of the educational system would differ radically from the current model that separates data use and evaluation (occurring at the state and district level) from the people who actually do the teaching and learning (schools, teachers, and s tudents). A fair, consistent knowledge management system would provide an environment for implementing an appropriate stakes model at each level, rather than focusing high-stakes measures on teachers and children.Variations in Organizational Approaches to Knowledg e Management Differing organizational approaches to knowledge ma nagement are one of the most broadly discussed areas in the field. Much of the r ecent literature that seems to be relevant for KM is based on the study of complex co mmercial organizationsÂ—primarily large, multinational, financial, and high technolog y firmsÂ—and comes out of the major business schools in the United States and Europe. T hese studies of KM point to a number of issues that educational technologists mus t address in the design of school district information networks before we can make si gnificant progress in areas that range from measuring individual student performance to th e most demanding evaluations of large districts and state-level educational systems The contributions of research on knowledge management to school and district can be applied most directly to strategic planning efforts. The work of Marchand (1999), Feen y and Feeny (1999), the Schools Interoperability Framework (see Footnote 13), and t he National Center for Education Statistics all provide assistance in applying KM pr inciples to planning and evaluation. Donald Marchand argues that organizational reform e fforts can often best be described by the way in which they operationalize their KM st rategies. He describes four important dimensions of such strategies, as shown in Figure 1 (Marchand, 1999). On the horizontal axis, managing costs and risks are straightforward concepts. On the vertical axis, adding value implies increasing ones return on investment. New realities can be conceived of as both creating new services or goods and creating re lationships with new customers. Points near the center of Figure 1 represent little or no activity along an axis, while points farthest from the center represent best prac tice.
12 of 32 Figure 1. How Information Creates Business Value (M archand, 1999). In this example, Organization A is an organization that sees innovation in information technology as a way to reduce risks and costs. A sc hool district following this strategy might implement a new in-school, computer-adaptive mathematics assessment to reduce outside testing costs and to help minimize instruct ional time lost to testing. Such a system might also help address the problem of mobil e students who may not be included in once-a-year standard assessments. Such an assess ment would be an element of a comprehensive program to reduce the risk of litigat ion associated with differential outcomes between poor and non-poor students by prov iding the opportunity to test adaptively and at smaller time intervals.Organization B, on the other hand, perceives the va lue of improvements in information management as coming from adding value or enhancing existing interactions and creating new information services and products. A s chool following this strategy might invest in workgroup computers and a high-speed Inte rnet connection. This technology would be used to support teaching a chemistry class in a virtual, web-based environment that would allow for the interaction of commercial and university subject-area experts as outside advisors and mentors for student projects.Marchand argues that an effective organization with a well-designed information management plan will have a clear understanding of its place on these dimensions and will plan for its development needs accordingly. It seems clear that an organization with a mature understanding of the importance of informa tion for decision-making might map its strategy onto Marchand's graphic as a circular structure, Organization C. However, the costs involved in moving a complex organization along multiple axes simultaneously would almost certainly be prohibitive. It is import ant to recognize the tradeoffs between adding to and protecting what already exists. A bal anced strategy would reflect this. Mapping one's own organization on this framework ca n provide some important
13 of 32insights. No single effort can conceivably score hi gh in all four dimensions, but a comprehensive knowledge management strategy ensures that, as a whole, investment in appropriate technological innovation addresses all of an organization's major goals. Marchand discusses another dimension not explicitly captured in this graphic. Is a particular knowledge investment required for operat ion, is it essential for competition, or is it likely to bring distinction as a unique actor in the relevant market? Most investment in operational data systems tends to be directed to ward the first two areas, day-to-day function and competition. It is the concept of stan ding out from the crowd that helps to drive truly innovative transformations of KM system s. In his research, Marchand finds that managers "rarely say that significant portions of their investments are focused on applications that give them distinctive competencie s with customers" (Marchand, 1999). The immediacy and relative clarity of an organizati on's current operational needs tend to drive most KM investment strategies. This is no les s true for school districts. Any district will have a working payroll or bookkeeping system. There is no other alternative. These systems must be operational and deliver subst antial value. Investments in long-term payoffs, such as those often encountered in improving educational systems, typically suffer by comparison. A system for tracki ng curriculum development and delivery is typically not characterized by a sense of urgency. The rationale for building such a system is less clear-cut than for the other, more traditional, operational data systems. Operational systems track and manage day-t o-day transactions but are of little use for planning or evaluation purposes. However, a s is the case with basic research in the manufacturing sectors, there needs to be substa ntial investment in a knowledge management infrastructure for any organization to r eap long-term payoffs. The return on investment from knowledge is small in the short ter m but can have a huge impact in the long run.If one applies Marchand's dimensional framework to schools and asks how information creates value in an educational setting, the level of analysis becomes extremely important. For example, one focus of a school distr ict, because it performs the business functions of any large organization system, would b e on managing risks and reducing costs. District administrators would also be intere sted in producing increased learning, but they could only have an impact through indirect effects. One model of district action would be an active structuring of incentives and re sources to create an environment in schools and classrooms that enhances learning. Bure aucratic structures are good at routine tasks and can provide infrastructure at a r easonable cost by taking advantage of economies of scale. Individual teachers, on the oth er hand, are engaged in directly adding value to the educational process and creatin g new realities for their students. This is a case of functional specialization. The problem is that actors at each of the levels need to be cognizant of the knowledge and system dy namics that define the other levels. Even within a district office, there will be differ ences across organizational boundaries. Curriculum support staff might be heavily focused o n "creating new realities" by aligning district curricular resources with new sta ndards or goals. This might include a "training on demand" streamed video system for deli vering professional development when and where it suits individual teachers and coo rdinators. Using technology to overcome the boundaries of time and space can creat e many new opportunities. A legal services team might be focused on reducing risk: fo r example, a district might put all Individualized Educational Plans (IEPs) online with a system for tracking interventions and student performance. This might be one response to concerns that special education
14 of 32students are receiving uneven access to services. T his system could provide both a tool for demonstrating compliance and encouraging school staff to stay on top of student needs by making school-level activities more transp arent. This drastic over-simplification highlights the difficulties one encounters when trying to design a system that meets the needs of actors at multiple l evels in the educational setting. It should be made clear that it is not necessary to build one monolithic knowledge management system. Indeed, it would be extremely di fficult to build a complex system that would be adaptable enough to respond consisten tly to a changing educational policy environment. A resource that has been developed in this regard is the Schools Interoperability Framework, which is an important a cknowledgement of the heterogeneity of data-acquisition and knowledgema nagement systems in use today. 11 Any realistic district information system must be m ade up of a matrix of interlocking systems that serve different functions and differen t user communities. Risk versus Investments in Information TechnologiesSchool districts are not unique in their spotty rel iance on information technologies designed to enable and monitor reform efforts. Many district-level systems were created to comply with externally imposed reporting require ments. Unfortunately, investment in transformative types of information technologyÂ—technologies that will impact the underlying organizational goals and drastically exp and capabilitiesÂ—is inherently risky. In a recent study of large information technology p rojects, David and Leslie Willcocks Feeny found that over 70 percent of these projects went over budget and missed their completion deadlines. They note that . the risks are even greater when, as with many Internet applications, real business innovations are also being looked for. The main reasons for disappointment cited also remain stubbornly familia r: incomplete definition of business requirements, insufficiently detailed t echnical specifications, changing business requirements, and lack of busines s user input during development. (Feeny & Feeny, 1999) Innovation in knowledge management is difficult in a complex organization. In education, we have long experience with the problem of differentiating direct and secondary effects. The expectation is that improvem ents in the educational settingÂ—investments in class size reduction, teache r training, access to computers, etc.Â—will translate into improved learning on the p art of the students. Schools and districts often engage in professional development efforts with the intent of improving student performanceÂ—an indirect effect of improving instruction. However, studies of the effectiveness of such efforts are often hampere d by the difficulty of isolating and measuring the value added by a particular professio nal development program to performance at the individual student level (Kenned y, 1999). Expectations that simple technology initiatives will adequately address such complex problems are wildly optimistic, at best.Unrealistic expectations are as likely to be attach ed to smaller-scale applications of technology as they are to large technology systems. One smaller-scale technological "fix" that is frequently advocated is the availabil ity of computers in classrooms. Proponents argue that computers are valuable tools and resources for both teachers and districts. However, what frequently goes unrecogniz ed is that, in order to live up to the
15 of 32expectations and serve the needs of both teachers a nd district administrators, computers must become an integral part of the classroom envir onment. Technology used as just one more add-on activity will have very little educational impact other than perhaps increasing keyboarding skills. There must also be s ufficient computing resources in the classroom with links to district-level data systems that allow individual teachers to make meaningful queries in real time. If one believes th at frequent measurement is critical for gauging the value added by a particular educational strategy, the ability to record and evaluate data in real time is crucial. District or state level analysis, on the other hand, would be a centralized, top-down approach that woul d have little use for complex data structures and would only rely on school-level tech nology for acquiring data. This distinction in interest between consumers of data a t the different organizational levels also reflects the need for a more global understand ing of the importance of the timeliness of data collection and rapid turn-around for results that are to aid decision making at the classroom level. The tension between district and classroom needs remains a troublesome barrier with respect to both turn-around time and the expected payoff for the time invested. Districts need data t hat are not immediately useful in the classroom (but must be collected there) and classro om teachers routinely assess their students and vary their interventions based on thos e assessments. Finally, there is a serious question about the vali dity of the measures of student learning used at both the school and district level. Teacher s often question the alignment of standardized tests with enacted curricula. There ar e also concerns about the consequential validity of using such tests to make high-stakes decisions about the progress of students and the retention or pay of te achers. By the same token, while a classroom assessment may be valid within that class room, the reliability of such measures is too low to be useful for comparing stud ent progress within a school or across a district. This is a problem that technolog ical innovations can address, but not in purely technical terms. The design of new assessmen t instruments may be enabled by new delivery and recording technologies. The rapid growth of computer-adaptive testing and its immediate scoring and reporting of results represent an enormous change from the typical "fill in the bubble" examinations. Wide access to digital video has made the production of multimedia portfolios of student work a reality in many non-affluent schools. This is an area where the landscape is cha nging rapidly. As distance technologies improve to allow teachers to collabora te with subject-area experts, master teachers, test constructors, and others, it will be come easier to work together on test quality and comparability. The social organization of classroom support structures will need to change as well, but the hope for coordinati on at a higher conceptual level can be realized.III. Expected Benefits from Technological Innovatio nsThere has been a great deal written about new forms of technical infrastructure and about the role that enabling technologies play in k nowledge management. Schools often have very little in the way of technological infras tructure on site. However, the current thinking about successful knowledge management warn s against relying on technology to solve the whole data management problem. 12 Dorothy Leonard-Barton (1995) suggests that the importance of "core technological capabilities" is a myth developed by managers looking for stability in rapidly changing environments. Instead, she provides a list of non-technological characteristics of what she refers to as renewing (or successful) organizations, summarized as:
16 of 32Enthusiasm for knowledge: Knowledge seeking and acc umulation are encouraged and rewarded. A spirit of inquiry drives people. Cu riosity is seen as an important asset. Staying ahead in knowledge: Having the drive to con tinue to learn and expand capabilities. Anticipating customer needs is the fo cusÂ—not responding to them. Tight coupling of complementary skill sets: Tear do wn internal boundaries and operate in teams. There need to be boundary spanner s in each area to make these external connectionsÂ—don't make everyone a generali st. Iteration in activities: you never achieve perfecti on. Iterative improvements are the only constant. Developing core capabilities is more like gardening than building somethingÂ—things need constant care and need to be turned under and replanted from time to time. Higher order learning: Don't just learn from operat ional needs. Listening too hard to current customers (problems, etc.) can blind one to the needs of potential customers in new markets. "For every activity," the manger asks, "what is the potential knowledge-building import of this action? Is it part of a larger pattern to which I should be devoting attention? If not, shoul d it be? If it should not be, why am I doing it?" Leaders who listen and learn: Leaders at all levels need to be knowledgeable about the organization's technologies. Eager learners are the most effective managers. (Leonard-Barton, 1995, pp. 261-266) The model of information networks explored by Leona rd-Barton has implications for many types of organizational culture. The points ou tlined above are entirely familiar to educators: Communities of learners can be described in this way. It is not surprising that the characteristics of successful teachers and teac hing are similar to those necessary for success in other areas of professional interaction. However, it is essential that the importance of the level-of-analysis problem in this area be recognized as well. Information networks that link the different levels of educational innovation will encounter greater challenges to building interest a nd enthusiasm for knowledge if that knowledge is of little use at a particular level in the organizationÂ—especially if it is the level that must shoulder the greatest burden for da ta collection (i.e., the classroom). Likewise, there are difficulties in agreeing on dat a structures and analytical models when purposes differ. Questions focused on the performan ce of a particular third-grade reading program across a range of cohorts passing t hrough that grade will engender datasets that are quite different from those create d to answer questions about the long-term impact of that same program on a single c ohort of students as they advance into higher grades. The data structures, analytical frameworks, and technological infrastructure necessary to answer questions in one area or level may have little relevance in another area or level.Leonard-Barton discussed the importance of organiza tional culture for supporting knowledge management activities. It is also importa nt to consider the involvement of critical users in the creations of a KM system. Dav id and Leslie Willcocks Feeny (Feeny & Feeny, 1999) approach the imperatives for success ful implementation of information
17 of 32technology-based innovation from a different direct ion. They argue that users of a technology must be the focus of any development or change effort, not its target. For example, a centralized reporting system for student attendance might provide detailed output that is designed by a district staffer to sa tisfy a state reporting requirement. Local schools, however, might have multiple additional ne eds that are not served by such a report. This is often the case when technology is d eveloped by a central bureaucracy to serve its own KM needs. A KM system for student ass essment will look very different if a district office designs it for its own use from t he way it would look if it had been designed by groups of classroom teachers to support instruction The Feenys also point out that needs and requiremen ts are not static in a rapidly changing complex system. Traditional approaches to system development are too linear to adequately address the dynamic, multidimensional nature of successful knowledge management systems. Another important characteristi c of successful KM projects is the presence of high-level non-information technology s upporters (e.g., managers not in the technology or computer departments). Since the adop tion of KM technology is largely a social process, it is vital that senior operational managers support the project and show that they value active participation. These caveats seem to be particularly important for projects that attempt to integrate the needs of mul tiple levels within an organization. High-level support that focuses on an overriding ne edÂ—such as improving student test scoresÂ—can be very effective in overcoming traditio nal barriers to cooperation that are often encountered in bureaucratic organizations. Ne eds that can be defined more broadlyÂ—that appeal to organization-wide norms or g oalsÂ—are best articulated from upper levels of management and have the best chance of being widely accepted if they are sponsored by someone with no parochial interest in one system or another. Others have addressed the problem of integrating hu man and technical systems. Karl Eric Svieby has referred to information technology as the primary hygiene factor in KM: "IT is for KM like a bathroom is for a house buyer . essential because without it the house is not even considered by buyers. But the bat hroom is generally not the vital differentiating factor for the buyer" (O'Dell, Gray son, & Essaides, 1998). Technology is important for efficient transfer of vital knowledge but delivers its benefits only as it supports human communication and knowledge construc tion. O'Dell and her colleagues also provide important in sights about some general rules-of-thumb for KM systems. They argue that "the more valuable the knowledge, the less sophisticated the technology that supports it" (O'Dell et al., 1998). For example, large data warehouses and data mining tools typical ly yield low-value knowledge, while a low-tech help desk delivers high-value knowledge. This is the difference between looking at pages of tabular data on the one hand an d statistical analysis and advice from an expert on the other. The expert brings personal experience, context sensitivity, and technical skill and combines it with the data at ha nd to produce integrated knowledge as an output. The important point of this example is t hat the expert interprets informationÂ—data that has been systematized. It is the aggregation of information and expertise that produces knowledge.O'Dell, Grayson, and Essaides also suggest that the low-tech/high-tech split reflects the fact that "tacit knowledge is best shared through p eople; explicit knowledge can be shared through machines. Or, the more tacit the kno wledge, the less high-tech the solution" (O'Dell et al., 1998). District-level inf ormation systems often contain a great
18 of 32deal of explicit knowledge about students and schoo ls. Tacit knowledge is that uncodified knowledge that is based on personal expe rience, absorption of organizational norms, and other factors. Explicit knowledge is inf ormation that has been written down or recorded in an information system. This might se em like a simplistic distinction, but it has important implications for decision-making and the reform process. This does not mean that the accumulation and transmission of taci t knowledge is not possible. Rather, it means that knowledge management systems must hav e imbedded in them some portion of the critical tacit knowledge needed to i nterpret information in the system at hand.For instructional decisions, teachers and school-le vel administrators, for example, often operate on the basis of tacit knowledge about an in dividual student or group of students. These data are much more difficult to aggregate and transfer. The primary problem is not technical. Rather, it is the difficulty of developi ng relevant metrics for a wealth of anecdotal data. Another important example of the im portance of tacit knowledge is the practice of using individuals as the focal point of reform efforts. School districts often use successful principals and other administrators as agents of change. Administrators that have been able to "turn a school around" are s een as a valuable commodity. The literature on KM refers to this process as one of u sing mobile intellectual capital to bring expert skill to bear on a particular local problem (Albert & Bradley, 1997). The value of intellectual capital is often the tacit knowledge a bout how one manages curricular changes or fosters a positive school climate. This process of conveying tacit knowledge about such a complex task is one example of a knowl edge system. The drawback is that tacit data is not easily trans ferred and successes at one location are not easily replicable to another. Some KM system-de signers attempt to imbed the interpretation of experts in the outputs of the sys tem; for example, one might present a bar graph of mean scores on a particular set of ass essments. A more knowledge-rich presentation might include a representation of erro r bands around the mean, or provide a comparison to scores of the same students in a prio r assessment. It is not merely that the information presented should be contextualized. It is important that the contextualization be done in a way that makes a valid comparison and enhances the explanatory power of the measure in question.As the Feenys, Davenport, and others suggest, there are distinctions along the continuum from data to knowledge. What these authors do not p rovide is a detailed understanding of how one applies this continuum to an educational setting. In order to bring about the senior management participation that Feeny refers t o above, it is necessary to establish the payoff of the investment in knowledge managemen t at every level of analysis. If the unit of analysis is the student, then the other que stions are derived from that. The analytical framework should focus on the individual The data structures in this case must be available at the individual level and be su fficiently broad for meaningful diagnostic use. Making the linkages clear between d ifferent levels of the organization and building methods of capturing and using tacit k nowledge are two characteristics that must remain in the forefront of any design effort.Efforts to Reform School Data-Management PracticesMuch of the work that went into this paper was info rmed by nearly eight years of experience working with education assessment and pr ogram data from the Milwaukee
19 of 32Public Schools (MPS). It has become increasingly cl ear that the ability of complex educational agencies to perform timely, in-depth, a nd accurate analysis is severely hampered by data-access problems. Indeed, there are increasing concerns that the district does not have the data it needs to make many import ant decisions or, if the data exists, they resides in a computer system that is difficult to use. In our work with MPS, we hope to take advantage of two major ongoing efforts. The first is the Schools Interoperability Framework (SIF). The SIF "is an industry initiative to develop an open specification for ensuring that K-12 instructional and administrative software applications work together more effectively." 13 The initial area of collaboration will be in the a rea of intra-application communication. The model the SIF group is supporting is an open-system environment This approach recognizes that schools and distric ts will continue to use a mix of information technologies f rom various vendors. The SIF initiative is focused on setting data exchange stan dards that will let the major school management and instructional support packages talk to each other without human mediation. This will help to decrease the transacti on costs of systems with broader functionality and should allow for better aggregati on of data across schools, districts, and states.The founders of the SIF emphasize the need for comp rehensive, consistent data management from a market-driven point of view. They argue that it is impossible to provide sophisticated applications if each individu al school district pursues its own data-management strategy. 14 The challenge the SIF has set for itself is based on the efforts of the business information technology comm unity to move from data-management to knowledge-management systems. Wh ile the issues involved in successful knowledge management are largely absent in the literature on educational administration and assessment, an important and gro wing body of work is emerging from business schools around the world. These works range from thinking about the role of experts in organizational learning (Albert & Bra dley, 1997) to multi-dimensional representations of the lifecycle of knowledge (Bois ot, 1998). The Financial Times recently ran a three-month series reviewing the cur rent thinking in academia about knowledge-management systems. 15 This series does an excellent job of making very complex models of organizational development and im pact assessment accessible to a broad audience and has helped to inform our discuss ions with decision makers in Milwaukee.The second important strand of work comes from the National Center for Education Statistics (NCES, 2000). 16 This effort produced a comprehensive, standards-ba sed model for school data system definitions. Unlike a product created by a particular vendor, district-level, state-level, and federal ed ucation administrators developed this model. Rather than being market-driven, the NCES Fo rum on Educational Statistics focused on the decision-making needs of schooland district-level administrators. This focus on users of data turns the traditional approa ch on its head. Most major school software systemsÂ—such as the offerings for National Computer Systems (NCS)Â—are driven by a lowest-common-denominator approach, whe re the package provides for the minimum needs for the maximum number of possible us ers. The NCES data elements, on the other hand, are developed to a high level of specificity and are intended to be extremely flexible and encompass the widest possibl e use. Both of these efforts point toward the importance o f standardized acquisition and the use
20 of 32of data for day-to-day and long-range decision maki ng. The issue of standardization is particularly important when the focus is on evaluat ion. Increasing demands by outside funders and state and local agencies for data on pr ogram impact continue to raise the analytical burden placed on the district. In tradit ional transactional student data systems, the focus is on managing schedules and tracking att endance and grades. Reporting is designed using a top-down approach that is focused on district, state, and federal reporting requirements. What the NCES proposes is a much more flexible design that would support very fine-grained inquiry from any le vel of the organization.IV. An Example of Mismatched Rationalities: The MPS Case StudySome of the problems one faces in a complex educati on institution can be seen in the following brief case study. The study describes how different organizational levels of a large metropolitan school district responded to the approaching deadline of a high-stakes assessment for its students at the end of eighth gr ade in the spring of 2000. The situational rationality of each major player led to radically different approaches and outcomes as the district struggled to develop an in formation system that would track students on their progress towards proficiency and that would accurately report student outcomes for retention and promotion decisions.The District's Technology Strategic PlanThe practical implications of a robust systemic ana lysis framework are daunting. District officials were not unaware of this problem. In its Technology Strategic Plan, 17 the planning committee outlined specific data needs for teachers and school administrators that are a direct result of district decentralizati on. The following excerpt from the report's Executive Summary outlines the technology needs of the three levels of the organization: Classroom Management in a Decentralized Organizatio n Instructional time can be increased by reducing tea cher time spent on classroom management tasks like attendance and grad e record keeping. A single point of data entry (the teacher) should dis tribute that information across the school. New technology can then make ava ilable that data and integrate all other data relevant to a particular s tudent to assist staff with decision making and the provision of services.MPS has taken steps toward redesigning the student information database maintained at the district level. In addition, a si te-based transaction-oriented database system is required. The two databases toge ther can exchange relevant student information to provide better supp ort. School Management in a Decentralized OrganizationDecentralization has imposed staggering new respons ibilities on school management personnel at the same time that the comp lexity of client needs has increased. School-based technology will help ad dress these challenges. MPS Accountability in a Decentralized Organization
21 of 32Systemic integration of reporting data at both the school and district level is required to tie together school educational plans, school accountability measure reports, district monitoring reports (MPS r eport card), state reports, and federal reports. 18 This portion of the strategic plan was then used to develop a Request for Proposal (RFP) for a new School Management System to enhance and e xtend the existing information system's capabilities. The two major themes of the Technology Strategic Plan and the RFP were "providing data to drive local and distric t decision-making" and assuring that the system "support school innovation by providing a tool that allows schools to implement their own initiatives and educational mod els." 19 These two goals imply an information system that is both a decision-support system that is linked to district and local goals, as well as one that has the capacity t o deliver new data acquisition and reporting capabilities linked to local needs. Eithe r of these goals by itself would have been difficult to achieve. Achieving them simultane ously would take both innovative programming and high-level training for the intende d users in the schools. The RFP laid out global system requirements that ad dressed some of the major shortcomings of the existing system. These requirem ents included an integrated security model, an import-export facility, and a user-friend ly query-and-reporting capability. These prerequisites are important features that the legacy system lacks. The document goes on to elaborate on the current situation (at t hat time) and projected needs in all of the major data subsystems. The distance between the existing capabilities of the system and the projected end points were sometimes quite s ignificant. One of the most positive elements of the RFP was the theme of data-based and data-driven decision making. One of the major considerations to be faced in desi gning a database is to understand the questions that will be asked of the data. Much of t he RFP is focused on improving the timely collection and reporting of student data: at tendance, guidance interventions, discipline, and grades. It is also clear that data collected by the new School Management System (SMS) will be used to evaluate individuals, programs, and processes. The SMS system was purchased from a systems integrator and is being adapted to the needs of the district. The needs of the district require univers al accessÂ—the ability to access a particular set of records from any locationÂ—and rea l-time longitudinal elements that track changes as they occur over time.The shift from a centralized data storage and repor ting system to a responsive, pervasive decision-supported system is a difficult challenge. The client/server topology recommended in the Technology Strategic Plan and required in the RFP provides a division between processing power and data accessib ility that reflects the needs of actors at different levels in the system. The proposed sys tem incorporates the two primary models of client/server system design. First, indiv idual school administrators and teachers will be able to query the central data rep ository from their own computers. Second, the data queried can be downloaded to a loc al computer for further manipulation, or for combining with local data. The central data store might also supply "what if" datasets that allow for the development o f contingency planning based on changes in important systemic variables. 20 Most importantly, the system being developed will allow people at a distance from the central office to become sophisticated consumers of student and system process data.
22 of 32The Case of 8th Grade High StakesThe decision to impose promotion requirements on 8t h graders was made in 1997 as part of a larger change in the district's accountability model. The district was simultaneously engaged in a major effort to develop and implement proficiency testing in middle school both to encourage good teaching practices and to pr ovide a broader range of assessments (in addition to standardized tests) to better under stand and represent student learning. The district was also in the development phase of a district-wide technology strategic planÂ—begun in 1996Â—that had as a central component replacing older transactional systems for the day-to-day management of student re cords and building a data warehouse that would support site-based decision-ma king. One of the important responses to the introduction of a multi-method assessment was the formation of the Middle School Principals Collabora tive. The principals from 12 of the 23 middle schools initially formed this group. The group has since grown to include all middle school principals. One of the central duties of this group (in cooperation with district administrators) has been to work out the d etails of designing, implementing, and evaluating the proficiency standards and assessment structure of the district's middle schools. Over the course of the following two years as the group grew to include all schools, the district and participating schools beg an to negotiate what metrics were to be recorded to demonstrate student proficiency. While several different methods were discussed, they all revolved around weighted averag es of multiple measures. At the district level, the units responsible for im plementing the recently developed Technology Strategic Plan were building a number of new applications. Two of these efforts were of particular interest for school admi nistration. The first is called the Student Management System (SMS). The SMS was to be a new transactional system for managing student information. This would include en rollment, attendance, grades, discipline, program participation, and other elemen ts. SMS was intended to replace a mainframe-based systemÂ—portions of which were over 15 years old. The other important school information system to be developed was a data warehouse for student assessment and other outcome data. This system was intended to provide an analytical resource for studying programs, assessing school ef fectiveness, and generating reports for external accountability.The intention was to make the SMS and the Data Ware house available at all levels in the district. The distribution of the SMS from the dist rict down to the classroom level was designed to accomplish a number of things. First, d ata entry was spread across a wider set of district personnel. Teachers would be able t o take attendance in their classrooms and record assessment data directly. The teachers w ould also be able to check on student program status themselves. The system further allow s teachers to record lesson plans and other data to capture more fine-grained data ab out classroom practices. Planners hoped to be able to integrate much of this data int o the Data Warehouse for later analysis at higher levels of aggregation. This would make it possible to develop a better understanding of such factors as the impact of new curricula and changes in professional development.The Data Warehouse was intended to provide local ac cess to assessment and program participation data extracted from SMS and combined with test data from external
23 of 32vendors, referral data from special populations sup port systems, and other standalone data systems. System designers proposed developing different methods of interacting with the Data Warehouse that would support both dif fering data-use needs and differing technical skill levels of the system's users.During this same period, we were working with staff members of the Office of Research and Assessment to help them develop their support f or databased decision making in schools. Since much of our work is focused on the d istrict level, we felt that it was also important to examine best practices for local data collection and manipulation. To this end, we have been working with Derek Mitchell of CR ESST 21 and the Quality School Portfolio (QSP) 22 to consider the critical elements of good, schoollevel, information-system design.On April 26, 1999, we participated in a district-wi de review of the status of the MPS Student Management System and Data Warehouse projec ts. At this meeting, we also presented the QSP tool as an avenue for forming add itional insights into school-level student data decision making. MPS deputy superinten dents, most department heads, and representatives from the University of Wisconsin-Mi lwaukee and Alverno College attended this meeting. The meeting covered the actu al progress to date of the ongoing design efforts, as well as the pressing data needs of Milwaukee's middle schools. Representatives of the Middle School Principals Col laborative also presented their homegrown approach for tracking student progress. T his initiative was developed as a direct response to the district's inability agree o n a set of proficiency metrics and provide the specifications for a data management system to deliver the needed data in a timely manner. One important outcome of this meeting has b een the growing sense of urgency regarding the delivery of useful analytical data fo r decision making at the school level. The QSP presentation served to provide both a backg round for discussing the needs of site-based decision making and an overview of infor mation-systems planning across the district. The major areas of thrust behind QSP (sch ool action plans, reporting processes, data-based decisions, and accountability) are by no means unique to this software package and reflect the needs of site-based decisio n makers in any field. We discussed how QSP might be used to accelerate the development process of the Data Warehouse by providing a conduit through which site-based man agers could funnel back their own analytical models. We discussed our interest in the research on how schools store, analyze, and retrieve data in support of continuous improvement and other school reform models.There were three significant outcomes of this meeti ng. First, the director of the MPS Department of Technology Services committed his sta ff to doing their best to get all of the middle schools wired and hooked up to the SMS s ystem by Fall1999. He also committed the application development team to build ing and fielding a system for capturing and reporting the middle schools proficie ncy data that would be used to make promotion decisions for students who would be 8th g raders in the 1999-2000 school year as soon as the specifications for this system could be finalized. The second outcome of the meeting was the decision of the Middle School P rincipals Collaborative to continue the development of its own school-based system in t he event that the district would be unable to meet the Spring 2000 deadline. The leader of the effort expressed concern that given the other pressing technology initiatives in progress that it would be difficult for the district to build and deliver a system in such a short span of time would be difficult.
24 of 32Finally, from out point of view, the most important outgrowth of this meeting was a decision by the then director of the Office of Rese arch and Assessment for that office to build its own analytical database. This decision wa s driven by two different concerns. First, there was the recognition that a number of s chools were under pressure to make decisions about preparing students for upcoming hig h-stakes assessments. Neither the existing mainframe system nor the new Data Warehous e being implemented has the capacity to provide the level of flexibility in rep orting needed by schools. Second, there has been a growing realization that meeting day-today, operational data needs and answering questions about accountability require di fferent interface, data-storage, and datamanipulation technologies and do transactiona l or compliance-focused systems. In MPS, this gap between operational and research need s has led district research staff to face the necessity for developing a different data management architecture to support these separate efforts.At this point, there were potentially three differe nt systems that might be in place by the end of the 1999-2000 academic year to track and rep ort on the promotion status of 8th grade students. What was unknown at the time was th at the recently elected (April 1999) school board was about to replace the superintenden t. One month later, a new superintendent was in office; he replaced almost al l department heads (the director of Technology Services retained his position) and one of the two deputy superintendents. In addition, the director of the Office of Research an d Assessment was moved on the organizational chart to report to the director of E ducational Services rather than to one of the deputy superintendents. These changes at the di strict level both halted the plans for the creation of a separate research data system and challenged the leadership of the technology services by replacing many of the experi enced decision makers so that neither group was able to accomplish its goals for supporting the 8th grade graduation decisions. By the end of September 1999, both units formally informed middle school principals that they would not be able to provide a ny direct help in collecting data on student progress towards meeting graduation require ments, nor would they be able to do anything more in reporting on student retention. Al l middle schools were going to be responsible for notifying high schools of the statu s of each student at risk of not passing by the end of the summer school session.Despite the political and technical upheaval occurr ing at the district level, the Middle School Principals Collaborative was continuing to m eet to discuss tracking student performance and reporting student progress towards promotion to teachers and administrators. The middle school principal who had developed his own system for tracking this information formally offered his syst em to the group and agreed to both modify the system to fit several different school o rganization models and to train a small number of people to provide training support in tur n to their counterparts in other middle schools. This effort was designed to use existing h ardware and software and would run on either Windows or Macintosh computer systems. It was also designed with the expectation that it could be combined on a central server and managed by either the Principals Collaborative or some unit at the distri ct level. The system went through several formal reviews and revisions and was used a t the end of the school year to produce electronic files that were sent to the dist rict's Office of Research and Assessment for review. These files were then upload ed to the Data Warehouse for use this fall to generate statistics for the district's accountability report. The most serious dilemmas encountered by participan ts in these development efforts
25 of 32were not technical. The district had not completed its connection of the middle schools to its high-speed network, but the amount of data t hat needed to be communicated was trivial. The barriers all revolved around communica tion. As the end of the school year approached in the spring of 2000, I was asked by th e manager of application development in the Technology Services department t o become a formal member of the Data Warehouse development team. The team is explic itly responsible for making sure the warehouse contains the data necessary to produc e the district's annual accountability report.My involvement was partially based on my knowledge of desktop hardware and software, but the primary reason the team leader wa nted me to participate was to overcome the communication barriers between the Dep artment of Technology Services and the Research and Assessment Office. I was also the only person on the committee who directly represented school-level interests thr ough my involvement with the two middle schools I was helping train to use the QSP t ool described above. It became clear half way through the first meeting that the group h ad not developed a common understanding of the needs of the schools or of the limits of existing data systems in meeting the district's accountability needs. I was also surprised to find that the Data Warehouse team did not include, nor had ever includ ed, a school administrator. The team worked over the next two months to put tog ether a plan for collecting the school-level data and putting it online. We also id entified an alternative method for collecting the results of the summer school assessm ents of students who had not passed by the end of the regular academic year. The mix of technologies included installing the standalone system developed by the Middle School Pr incipals Collaborative in all schools with 8th graders and requiring them to use it, repurposing a dormant mainframe system to collect alternative assessment data on su mmer school students, writing custom programming to aggregate the school data and load i t into the Warehouse, and creating a new report on a discontinued report card system to provide high schools with accurate placement data for students who were or who were no t retained in grade at the end of summer school. While the process was successful in the end, the resources used to respond to this emergency reporting need could have been better spent in other areas if the district's data management system had been in s ynch with the accountability requirements the educational system had placed on s chools and students.Preliminary ConclusionsKnowledge management is such a wide-open area of st udy that it is difficult to understand the implications of these models of know ledge management for an educational setting. One thing seems certain. Schoo l information systems are one of the most difficult to harness because they often lack a ny overall rationality for cooperation and compliance. Differences in data needs and uses across different organizational levels present significant barriers to the collaboration n ecessary for innovation in knowledge management.The case study cited above points to a number of di fferent areas for concern. First, ambitious systemic reform efforts call for radical changes in traditional school information systems. The dimensions of knowledge ma nagement strategy that Marchand maps out in Figure 1 provide a background for the d ifficulties MPS managers encountered when they attempted to make a sweeping overhaul of their information
26 of 32technology infrastructure. Even the most conservati ve deployment estimates of system designers in the original Technology Strategic Plan are more than two years in the past. Developers estimate that it will take another two y ears to complete the wiring and programming necessary to bring all 160 regular scho ols online. Some managers have indicated that the declining political support for the system and the ongoing burden of customization will probably lead to the development of a new system independent of the externally purchased SMS application currently bein g implemented by Milwaukee high schools.The Data Warehouse is also several years behind sch edule. There remain two major stumbling blocks to the development and use of the system. The first hurdle is the difficulty of producing assessment, enrollment, and program participation statistics that match those created by the existing combination of mainframe exports, SAS (statistical software) scripting, and hand manipulation used to produce the district's accountability reports. This problem can be traced back to the inh erent complexity of the analytical puzzle of tracking a highly mobile population of st udents with an archaic information system that relies on a great deal of expert knowle dge on the part of its users. This is simply a high-dimensional analytical problem that c annot be easily moved from one system to another. The other problem is that the ne w data system uses different data element definitions, field layouts, and formats. Th e new system is designed to take into consideration the improvements in methodology and c hanges in how one defines important school metrics such as value-added assess ment and program effectiveness. Any comparison of analytical models between the two methods of producing accountability statistics requires the ability to e ngage in a sophisticated translation among approaches that are vastly different.The other stumbling block the Data Warehouse faces is the method of access that school administrators and teachers will use to extract dat a for local analysis. The tool that was initially adopted is being used by district-level a nalysts in the Department of Technology Services and the Office of Research and Assessment, but was seen as too complicated for the average or casual user. Other solutions tha t rely on Web-based access are either not supported by the some of the desktop technologi es in place, do not meet the security requirements the district must meet to protect stud ent data, or are as complex as the tool the district is currently using.Finally, the lack of strategic planning and strateg ic resource allocation continues to plague development efforts at the district level. I t is possible to raise funds to support the infrastructure in providing high-speed video to sev eral thousand classrooms, but it is difficult to find the resources to train the staff to use video effectively, or to build an evaluation system to track the impact of this techn ology on teaching and learning. It is the mismatch between resources available for high-p rofile technologies and the resources available to measure effective teaching t hat lead to the dilemmas encountered by Milwaukee Public Schools. The leveland unit-of analysis problems alluded to above only exacerbate this mismatch between resourc e availability and needs. When both resources and the external requirements for an nual accountability focus development efforts on the introduction of newer an d newer technologies, school-level needs are bound to be short changed.Notes
27 of 32 1 For an overview of the literature on standards-bas ed reform see, for example, the National Council of Teachers of Mathematics standar ds website at http://www.nctm.org/standards/introducing.htm or Ki rst and Bird (1997) at http://www.wcer.wisc.edu/nise/Publications/Research _Monographs/vol2.pdf 2 This definition is adapted from an address given b y Deputy Secretary Smith at the National Institute for Science Education 1999 Forum 3 See, for example, the NSF Educational System Refor m site at http://www.ehr.nsf.gov/esr/ or the Department of Ed ucation's National Research and Development Centers at http://www.ed.gov/offices/OE RI/ResCtr.html. 4 For more on authentic assessment, see Neumann, Sec ada, & Wehlage (1995). 5 For some of the best examples of this literature s ee the Journal of Knowledge Management at http://www.mcb.co.uk/jkm.htm and reso urces links at the Financial Times Mastering series web site at http://www.ftmastering.com/link s.html 6 http://mpscda.milwaukee.k12.wi.us/html_files/Purpo se_cda.html 7 http://www.cos.com/ 8 http://www.eims.org/ 9 firstname.lastname@example.org 10 Information about NCS can be found at http://k12.n cs.com/. IBM's school and district management site can be found athttp://ind.clearlake.ibm.com/industries/education/s olution/SOLUTIONS_19443.html. For more information about eScholar go to http://ww w.escholar.com/. 11 For a list of organizations involved in developing an interoperability framework between school and district information systems see http://www.schoolsinterop.org/spec/Acknowledgements .htm 12 Others have written more extensively on the comput er-human interface. Two examples of this work are Rouse, 1991; Shneiderman, 1998). 13 The SIF site can be found at http://www.schoolsinterop.org/spec/Acknowledgements .htm 14 The SIF data exchange specification can be found at http://www.sifinfo.org/spec.htm 15 An overview of the entire series on information man agement can be found at http://www.ft.com/mastering/
28 of 32 16 These reports can both be found at the National For um on Education Statistics site at http://nces.ed.gov/forum/publications.html 17 Milwaukee Public Schools' Technology Strategic Plan December 11, 1996 (Rev. 02/01/97) http://whscdp.whs.edu/tsp/plan/ 18 "The Impact of a Client/Server Architecture on Dec ision Support Systems," by M. Whitman and H. Carr, 1994, The Executive's Journal, 10, p. 8-9, 12. 19 MPS. RFP-239. p. 0-3. 20 For more on this, see, for example, M. Whitman & H Carr, Information Strategy Winter 1994, Vol. 10 Issue 2, p. 12. 21 CRESST (The National Center for Research on Evaluat ion, Standards, and Student Testing) is located at UCLA. http://www.cse.ucla.ed u/ 22 More information on the QSP can be found at http:// qsp.cse.ucla.edu/ReferencesAlbert, S., & Bradley, K. (1997). Managing knowledge: Experts, agencies, and organizations Cambridge, UK; New York: Cambridge University Pre ss. Armstrong, A. B., (1999). States go to the head of the class. Business Week, No 3625 (April 19), 44.Boisot, M. (1998). Knowledge assets: Securing competitive advantage in the information economy. Oxford; New York: Oxford University Press. Clune, W. H. (1998). Toward a theory of systemic reform: The case of nin e NSF Statewide Systemic Initiatives. Research Monograph 16. Madison: University of Wisconsin, National Institute for Science Education Davenport, T. H., De Long, D. W., & Beers, M. C. (1 998). Successful knowledge management projects. Sloan Management Review, 39 (2), 43-58. Davenport, T. H., & Prusak, L. (1998). Working knowledge: How organizations manage what they know Boston: Harvard Business School Press. Davenport, T. H., & Davenport, D. M. (1999, March 8 ). Is KM just good information management? The Financial Times Mastering Series: Mastering Inf ormation Management, 2-3. Empson, L. (1999, October 8). The challenge of mana ging knowledge. The Financial Times Mastering Series: Mastering Strategy, 8-10. Feeny, D., & Feeny, L. W. (1999, April 5). Transfor ming IT-based innovation into business payoff. The Financial Times Mastering Series: Mastering Inf ormation Management, 4-6.
29 of 32Graham, W. O., Osgood, D., & Karren, J. (1998, May) A real-life community of practice. Business and Management Practices: Training & Devel opment, 52 (5 ), 34-38. Greengard, S. (1998, October). Will your culture su pport KM? Workforce, 77 (10), 93-94.Kennedy, M. (1999). Form and substance in inservice teacher education (Research Monograph No. 13). Madison: University of Wisconsin National Center for Improving Science Education.Kirst, M., & Bird, R. (1997). The politics of developing and maintaining mathemat ics and science curriculum content standards Madison: University of Wisconsin, National Institute for Science Education.Leonard-Barton, D. (1995). Wellsprings of knowledge: Building and sustaining t he sources of innovation Boston: Harvard Business School Press. Marchand, D. (1999, April 5th). Hard IM choices for senior managers. The Financial Times Mastering Series: Mastering Information Manag ement, 2-4. National Center for Education Statistics, CCSSO, an d the Student Data HandbookWorking Group. (2000). Student data handbook for elementary, secondary, and early childhood education: 2000 Edition. Washington, DC: Author. Newmann, F., Secada, W., & Wehlage, G. (1995). A guide to authentic instruction and assessment: Vision, standards, and scoring Madison: University of Wisconsin, Wisconsin Center for Education Research.O'Dell, C. S., Grayson, C. J., & Essaides, N. (1998 ). If only we knew what we know: The transfer of internal knowledge and best practice. New York: Free Press. Rouse, W. B. (1991). Design for success: A human-centered approach to de signing successful products and systems. New York: Wiley. Shneiderman, B. (1998). Designing the user interface: Strategies for effect ive human-computer interaction (3rd ed.). Reading, MA: Addison Wesley Longmans. Snyder, W. M. (2000, January/February). Communities of practice: The organizational frontier. Harvard Business Review 139. Wenger, E. (1998). Communities of practice: Learning, meaning, and ide ntity Cambridge, UK; New York: Cambridge University Press .About the AuthorChristopher A. ThornWisconsin Center for Education ResearchDirector of Technical Services1025 W. Johnson St., Room 370Madison, WI 53706Tel: 608-263-2709
30 of 32 Fax: 608-265-9300Email: email@example.comChristopher Thorn is a researcher and Director of T echnical Services at the Wisconsin Center for Education Research at the University of Wisconsin-Madison. He is a principal investigator on several projects focused on developing open source tools and collaborative online environments for analysis of v ideo data. He is also engaged in research on information system design for school im provement. Chris is chair-elect of the AERA Technology and Telecommunication Committee Chris completed his Doctor of Sociology (sociology of technology & decision ma king) at the University of Bielefeld, Germany. His earlier work was focused on R&D collab oration between unlikely partners.Copyright 2001 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, firstname.lastname@example.org or reach him at College of Education, Arizona State University, Tempe, AZ 8 5287-0211. (602-965-9644). The Commentary Editor is Casey D. C obb: email@example.com .EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov firstname.lastname@example.org Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Calgary Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Education Commission of the States William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX)
31 of 32 Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton email@example.com Hugh G. Petrie SUNY Buffalo Richard C. Richardson New York University Anthony G. Rud Jr. Purdue University Dennis Sayers California State UniversityÂ—Stanislaus Jay D. Scribner University of Texas at Austin Michael Scriven firstname.lastname@example.org Robert E. Stake University of IllinoisÂ—UC Robert Stonehill U.S. Department of Education David D. Williams Brigham Young University EPAA Spanish Language Editorial BoardAssociate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico email@example.com Adrin Acosta (Mxico) Universidad de Guadalajaraadrianacosta@compuserve.com J. Flix Angulo Rasco (Spain) Universidad de Cdizfelix.firstname.lastname@example.org Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho dis1.cide.mx Alejandro Canales (Mxico) Universidad Nacional Autnoma deMxicocanalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State Universitycasanova@asu.edu Jos Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es Erwin Epstein (U.S.A.) Loyola University of ChicagoEepstein@luc.edu Josu Gonzlez (U.S.A.) Arizona State Universityjosue@asu.edu Rollin Kent (Mxico)Departamento de InvestigacinEducativa-DIE/CINVESTAVrkent@gemtel.com.mx email@example.com Mara Beatriz Luce (Brazil)Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.brJavier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mxMarcela Mollis (Argentina)Universidad de Buenos Airesmmollis@filo.uba.ar Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mxAngel Ignacio Prez Gmez (Spain)Universidad de Mlagaaiperez@uma.es
32 of 32 Daniel Schugurensky (Argentina-Canad)OISE/UT, Canadadschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil)Fundao Instituto Brasileiro e Geografiae Estatstica firstname.lastname@example.org Jurjo Torres Santom (Spain)Universidad de A Coruajurjo@udc.es Carlos Alberto Torres (U.S.A.)University of California, Los Angelestorres@gseisucla.edu