xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam a22 u 4500
controlfield tag 008 c20029999azu 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E11-00294
Educational policy analysis archives.
n Vol. 10, no. 46 (October 20, 2002).
Tempe, Ariz. :
b Arizona State University ;
Tampa, Fla. :
University of South Florida.
c October 20, 2002
Senior school board officials' perceptions of a national achievement assessment program / Marielle Simon [and] Rene Forgette-Giroux.
Arizona State University.
University of South Florida.
t Education Policy Analysis Archives (EPAA)
xml version 1.0 encoding UTF-8 standalone no
mods:mods xmlns:mods http:www.loc.govmodsv3 xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govmodsv3mods-3-1.xsd
mods:relatedItem type host
mods:identifier issn 1068-2341mods:part
mods:detail volume mods:number 10issue 46series Year mods:caption 20022002Month October10Day 2020mods:originInfo mods:dateIssued iso8601 2002-10-20
1 of 16 Education Policy Analysis Archives Volume 10 Number 46October 20, 2002ISSN 1068-2341 A peer-reviewed scholarly journal Editor: Gene V Glass College of Education Arizona State University Copyright 2002, the EDUCATION POLICY ANALYSIS ARCHIVES .Permission is hereby granted to copy any article if EPAA is credited and copies are not sold. EPAA is a project of the Education Policy Studies Laboratory. Articles appearing in EPAA are abstracted in the Current Index to Journals in Education by the ERIC Clearinghouse on Assessment and Evaluation and are permanently archived in Resources in Education .Senior School Board Officials' Perceptions of a National Achievement Assessment Program Marielle Simon Rene Forgette-Giroux University of OttawaCitation: Simon, M. & Forgette-Giroux, R. (2002, Oc tober 20). Senior school board officials' perceptions of a national achievement assessment pr ogram, Education Policy Analysis Archives 10 (46). Retrieved [date] from http://epaa.asu.edu/epa a/v10n46.html.AbstractThe School Achievement Indicators Program (SAIP) has been collecting data across Canada on 13and 16-year-old student a chievement in mathematics, in science, and in reading and writing since 1993. In 1999, it completed its second assessment cycle and was re viewed in Spring 2000. The review design included a survey of offici als from all the school boards/districts that participated in the sc ience assessment program held in 1999. The results of this study sho w that this stakeholder views as the most pressing issue for SA IP to succeed in its mandate, the need for development in four areas: a) Increased teacher
2 of 16and student motivation to participate wholeheartedl y in the program; b) Effective dissemination options; c) Leadership thro ugh innovation in teaching and in assessment practices despite high a ccountability orientation; and d) Cost-effective, yet rigorous me ans of providing both snapshot information and longitudinal means of comp arisons. Although universally appealing, such approaches have yet to be supported by sound educational theory and methodology. Since 1993, the School Achievement Indicators Program (SAIP), under the responsibility of the Council of Ministers of Educa tion, Canada (CMEC), has been collecting data across Canada regarding 13and 16year-old student achievement in Mathematics, in Science, and in Reading and Writing In 1999, it completed its second assessment cycle and was reviewed in Spring 2000. T he review had three objectives: a) To determine the degree to which the CMEC had succe eded in implementing the recommendations adopted from the review of the firs t round of assessments (Crocker, 1997); b) To measure the extent to which SAIPÂ’s obj ectives, set at the beginning of the second cycle, had been attained; and c) To formulat e specific recommendations about the SAIPÂ’s aims, operations, and uses. To achieve t hese objectives, the review design included various data collection approaches, one of which was an investigation of the perceptions of all the school boards across Canada that participated in the 1999 science assessment, toward the national assessment program. The purpose of this paper is to report the results of this survey in order to provi de researchers and policymakers with a better insight into the general interests, particul ar views, and specific needs of one of the most important stakeholders in large-scale educatio nal assessment programs. As a large-scale assessment program, SAIP is simila r to the National Educational Assessment Program (NAEP), conducted in the United States. Both are national, cyclical programs, administered across regions (sta tes or provinces/territories). In both countries, these regions have sole jurisdictional r ights over education. Like NAEP, SAIP is designed to complement existing assessments in e ach province and territory. It is essentially a standards-based program focussing on assessment of content and ability via a mixture of multiple-choice, constructed response, or short hand-on performance tasks. The program also consists of student, teacher, and school administrator questionnaires intended to provide contextual data. Testing usuall y occurs in May and the final report is made available a year later.As mentioned above, SAIP is governed by the Council of ministers of education, Canada. The CMEC Â’s role in the SAIP essentially co mbines that of the United States National Centre for Education Statistics (NCES), re sponsible for NAEPÂ’s operation and technical aspects, and the National Assessment Gove rning Board (NAGB), mandated to select the subject areas to be assessed and their c ontent framework. In Canada, the CMEC also coordinates national participation to oth er large-scale international testing programs such as the Program for International Stud ent Achievement, run by the Organisation for Economic Co-operation and Developm ent (OECD) and the Third International Mathematics and Science Study (TIMSS) governed by the International Association for the Evaluation of Educational Achie vement (IEA). Not all the Canadian jurisdictions participate in the various internatio nal programs. For a more detailed look at the general CMEC operations, SAIPÂ’s sampling tec hniques, and the nature of the results, see for example the most recent SAIP Repor t (CMEC, 2000) or the CMEC WEB site. (Note 1)
3 of 16SAIP was initially established to meet the followin g three objectives: a) To set educational priorities and plan improvements to cur ricula; b) To provide the best education to all young Canadians; and c) To report on certain indicators to the Canadian public. Despite these noble goals, SAIPÂ’s practices and orientations are geared principally towards accountability and overall inst ructional enhancement rather than improving local teaching, learning, and assessment practices. In 1999, all thirteen Canadian jurisdictions, namely the ten provinces an d the three territories, participated in the Science assessment program. A random sample of students was drawn from each of the participating jurisdictions and, within some of these, students were also sampled by linguistic groups (English and French). The next fo ur sections respectively present the framework, the methodology, the results, and a disc ussion of the investigation component of the program review.FrameworkThe theoretical framework for the investigation evo lved first from the examination of three documents: The call for proposals initiated b y the Council of Ministers of Education, Canada, the report on the review of SAIP Â’s first cycle (Crocker, 1997), and the official SAIP memorandum of understanding betwe en the Human Resources and Development, Canada and CMEC, 1999). Second, variou s documents on general program reviews were consulted, such as the Standards for Evaluations of Educational Programs, Projects, and Materials (Joint Committee on Standards for Educational Evaluations, 2nd Ed., 1999). This also included works on: a) The pu rpose of a program evaluation (Wilde & Sockey, 1995), b) The role of p rogram reviews (Chelimsky & Shadish, 1997), c) The use of indicators (Posavac & Carey, 1997; Shavelson, McDonnell & Oakes, 1991a, 1991b), and d) The method ological approaches to program evaluations (Boulmetis & Dutwin, 1999; Popham, 1999 ). Third, the literature review for this study considered other actual systematic evalu ations of large-scale assessment programs, related models, or proposals (Crooks, 199 6; Madaus & Pullin, 2000; Ryan, 2002; Shepard, 1977). Finally, the previous experie nces of this studyÂ’s authors in the field of program evaluation also contributed to sha ping the framework of the review of SAIPÂ’s second cycle (Cousins & Simon, 1993; Macdone ll, A., Forgette-Giroux, R., Schmidt, S., Mougeot, Y, & Levesque, J., 1999). Thi s led to the development of a framework that essentially consisted of five genera l areas of questioning on SAIP: a) Nature and position among other assessment initiati ves, b) Goals and objectives, c) Operations, d) Design, and e) Impact. The nature an d position focussed on the role, function, relationship, differences, similarities, and linkages between SAIP and other large-scale assessment programs. The appropriatenes s of SAIPÂ’s objectives and the identification of possible barriers to attaining th ese objectives concerned the second theme. SAIPÂ’s operations dealt with sampling, stand ard-setting, grading, and reporting procedures currently in place to meet the stated ob jectives, whereas validity, reliability, and questions that focussed specifically on content and format, nature of data collected, and motivational issues, all served to study SAIPÂ’s design qualities. Finally, its impact was examined through questions on significance, per ceptions, values, and overall influence of SAIPÂ’s results on various educational and public settings.Methods
4 of 16The investigation described here represents only on e component of the methodological design for the review and targets only one group of stakeholders: School boards (districts or councils) officials. Interviews with jurisdictional coordinators or directors were also conducted for the review along with in-de pth content analyses of various relevant documents. These various components enable d the triangulation of the data that were ultimately reported in an aggregated format. T he following sections present the subjects, instruments and research design of the su rvey component of the SAIP review. SubjectsAll school boards or districts across Canada with o ne or more schools that participated in the 1999 Science assessment program were contact ed and invited to participate in the investigation. This meant reaching a total of 412 s chool boards from 19 jurisdictions: Ten provinces and three territories, with six of th ese also broken down into two linguistic sub-populations at the time of the asses sment. Only one member of each board, a senior official, was asked to participate. This official could be a Director of Education, a Superintendent of Education, a designa ted SAIP school Official, a local Coordinator or Liaison person, a Board Consultant, or the Principal of a school that was involved in the 1999 SAIP assessment. In some areas the Director of Education is the highest ranking official while in others it is the Superintendent of Education. When both are found within a school board, the Director of Ed ucation usually has priority. InstrumentA written questionnaire was developed based on the frameworkÂ’s five themes. A first draft was submitted for validation to two experts f amiliar with SAIP, one French-speaking and one English-speaking. Their tas k was to determine the relevancy, comprehensiveness, and linguistic equivalence of th e questionnaire. The final version included seven general information questions and si x attitudinal questions with 73 sub-questions. Twelve questions offered multiple op tions while two were open-ended although all questions provided space for additiona l comments. The open-ended question # 13, for example, asked Â“What suggestions would you offer regarding SAIP and its various components in order that your schoo l board or district gives it high priority among all assessment initiatives? Of parti cular interest however were questions 9 and 10. Question 9 asked :Â“To what extent do you agree with the following actual SAIP parameters?Â”, while Question 10 read as: Â“To w hat extent do you agree with the following proposed SAIP parameters?Â” The proposed parameters emerged f rom comments made by the SAIPÂ’s designated jurisdiction al coordinators who had previously participated in the semi-structured inte rviews, from the recommendations that resulted from the review of SAIPÂ’s first cycle, and from their relevance with respect to general school boardsÂ’ interest and needs.ProcedureThe questionnaires were sent to the top administrat or of each of the 412 school boards. The senior officials were instructed to fill out th e questionnaire themselves or to forward it to someone from the school board that had been a ctively involved in the 1999 science assessment. The questionnaires were distributed dur ing the last week of May 2000 with
5 of 16specific written instructions to return the complet ed questionnaires in a self-addressed envelope by June 30, 2000. Two sets of follow-up te lephone calls were made by a superintendent to selected school boards from each of the jurisdictions and their sub-population in order to achieve a maximum rate o f return by all sampled populations. The first was conducted in the second week of June 2000 to approximately half of the boards within each jurisdiction to see whether they had received the questionnaires. The next series of calls were made in August to approxi mately twenty randomly selected boards from each jurisdiction that had a return rat e below 30 % by the end of July.ResultsIn all, 147 questionnaires were completed and retur ned, yielding an overall response rate of 36%. Of the 19 populations, four had all their p articipating schools in a single school board. Two of these four boards returned the comple ted questionnaire. Closer examination of the distribution of responses by jur isdictions reveals that the nine Â“smallerÂ” jurisdictions and minority groups, that i s those that had from two to 12 school boards, yielded response rates varying from 33% to 100 %, with a mean of 59% and median of 54%. The six Â“largerÂ” jurisdictions, name ly those with over 12 school boards (actually between 22 to 62), gave response rates va rying from 27% to 36 %, with a mean of 32 % and a median of 33%. The rate of response f or the 15 jurisdictions with two or more school boards therefore ranged from 27% to 100 %, with an average of 48% and a median of 46%. It can also be reasonably stated tha t respondents were drawn from both small and large school boards and districts, from b oth remote and rural settings, from both central and urban areas, and from both the maj ority and minority linguistic groups. Most questionnaires (80%) were completed by Directo rs of education, Superintendents or Board consultants. Eighty-six percent of respond ents said they were more or less familiar with SAIP or knew it well. Respondents par ticipated mainly in the coordination of the study, the test administration, and the comm unication of results. Although most questions included a four-point rating scale, dicho tomized results (e.g. totally agree and agree versus more or less agree with disagree) are reported here with respect to each of the five themes.NatureThe specific information on this topic was provided mainly in the two open-ended questions. Respondents generally recognize that SAI P provides a valuable index from a national perspective but state that it should disti nguish itself from international and provincial initiatives by: a) Highlighting cross-cu rricular competencies if sampling remains age-based; b) Introducing innovative teachi ng, learning, and assessment approaches that are applicable to the classroom; an d c) Adopting a diagnostic and interpretative approach to contextual and achieveme nt data. Approximately 70 % of respondents also suggest that secondary analyses be performed on the data collected. GoalsAlthough indicating being familiar with SAIP, three quarters of respondents (75 %) also admit that they are somewhat or not very aware of i ts stated goals and objectives. With respect to the appropriateness of the objectives, t hey generally suggest the need for an
6 of 16 objective that would implicate some form of compari son of results. The rank order of the percentage of respondents in agreement with the various types of comparisons of results is presented in Table 1.Table 1 Rank Order of Percentages of Respondents in Agreeme nt with Types of Comparisons of ResultsLongitudinal comparisons from one assessment cycle to next87% With national results (averages)81%With national expectations (standards)65%Between the two age groups (13 and 16)64%With local expectations 50% Among jurisdiction results 44% Table 1 shows support from school board officials f or comparison of data from one assessment cycle to the next and with the national averages. OperationsResults under operations are reported in terms of S AIPÂ’s general administrative, sampling, expectation/standard-setting, scoring, an d reporting procedures. Eighty-nine percent (89 %) of respondents agree with the statem ent that SAIP meets the given time-lines and 67% believe that SAIP is relatively easy to administer. It is interesting to note that only 51 % of the respondents favour the m onth of May as the best time for test administration, 68 % disagree with conducting these in February or March, and 77 % disagree with a Fall administration, thus failing t o achieve a consensus on the best time of year for administrating the various assessment p rograms. Three quarters of respondents favor sampling of 13and 16-year-old s tudents rather than sampling grades. Two thirds agree with comparisons of results with e xpectations or standards and generally approve of the five-point rating approach to scoring. Finally, nearly 70 % of respondents support the idea of disseminating some form of school board, school, or individual-based results for motivational purposes.DesignQuestions on SAIPÂ’s design asked for level of agree ment with the assessmentÂ’s focus on collecting disciplinary and contextual data as well as from theoretical and practical aspects of achievement data gathering techniques wi thin Mathematics and Science. They also looked at motivational issues and perceptions around assessment cycles. Results show that 77 % of respondents agree with the collec tion of contextual data. Most concur with SAIPÂ’s mandate to conduct assessments in Mathe matics, in Science, and in Reading and Writing and over 85 % favor both the th eoretical and practical components of the science and mathematics assessments. However 66 % of the respondents state that SAIP does not sufficiently encourage or motiva te students to give their optimal
7 of 16performance, thus addressing validity and reliabili ty concerns. Finally, nearly 80 % of respondents support the three-year assessment cycle within a discipline. ImpactEighty three percent (83 %) of the respondents beli eve that the program has little or no positive impact on assessment practices in the clas sroom and approximately three quarter of respondents say that SAIP has little pos itive influence on setting priorities in the various disciplines assessed, on curricula, on public perception of the quality of education, on research initiatives, and on teaching practices. Approximately 85 % of respondents, however, would recommend those school boards that have not yet participated to the various SAIP programs to do so.Discussion and ConclusionDespite the overall low response rate obtained in t his study, the results are telling. With one exception, the higher response rates are provid ed by those jurisdictions or minority populations with less than 12 school boards. Althou gh many jurisdictions take part in other large-scale assessment program such as TIMSS and PISA, most respondents indicate general support for the SAIP because it is the only national long-term assessment program in which all thirteen educationa l jurisdictions and respective linguistic sub-populations participate. They also a ppreciate the complementarity of SAIPÂ’s results to those obtained through regional a nd other international assessment programs. Respondents stress, however, that SAIP sh ould be forward looking and be more than a simple indicator system serving account ability purposes. Their responses lead toward interesting suggestions for each of the five themes. These are discussed in the following sections.Nature, role, and positionWith respect to the nature, role, and position of S AIP, the respondents value the nationally representative and continuing aspects of SAIP. Given that participation is voluntary, however, it becomes important that SAIP continues to be attractive and relevant to this stakeholder, particularly to those school boards that have sufficient resources to implement their own assessment program or to participate in most international ones. In that sense, SAIP should cons ider three suggestions offered by the school boards. First, they propose that if SAIP rem ains an age-based program as opposed to grade-based, then it should go beyond the assess ment of disciplinary contents and basic skills in order to focus on socially relevant general competencies such as critical thinking, information management, speaking skills, and on attitudes such as civic values, self-awareness, self-esteem, and student engagement (Jones, 2001). Major organizations, such as the IEA and the OECD, already provide leade rship in the assessment of such competencies via their own large-scale programs. At the onset, this option may appear to be duplicating efforts and resources given that man y jurisdictions also participate in these major international assessment programs but, so far, SAIP has had the advantage of involving all jurisdictions and of being better able to respond to national educational concerns.Second, as mentioned above, the respondents wish to see SAIP as more than an indicator
8 of 16program, one that would offer specific perspectives for adjustment and intervention with respect to setting educational priorities, targetin g curricular improvements, highlighting best teaching strategies, and fostering innovative assessment practices. This push for a shift from an indicator system, i.e., one that prov ides information that can be used to improve education, to a monitoring one, in other wo rds, one that further analyses and interprets key contextual and achievement data in o rder to propose prescriptive feedback, stems largely from those smaller jurisdic tions with scarce resources. The resulting feedback could subsequently translate int o the introduction of innovative teaching, learning, and assessment approaches that are applicable to the classroom. This last statement relates to the respondentsÂ’ third re quest. These three requests contributed to the formulation of two specific recommendations in the review: a) That the concept of achievement indi cator be broaden to include assessment of general competencies as defined by th e pan-Canadian public and b) That SAIP be assigned a diagnostic function with interpr etation of the most obvious links between contextual data and achievement (Forgette-G iroux & Simon, 2000). These recommendations are rather demanding. As with other large-scale studies, SAIP must re-examine it priorities with respect to accountabi lity versus instructional orientations toward educational improvement (Popham, 1999) and e ventually aim at establishing a balance between the two goals. So far SAIP has been mainly oriented toward the need for greater accountability. Moreover, it must rely on sufficiently sound theory or design to efficiently explore any relationships among back ground variables and achievement, to claim causal inferences or to explain why the compa rison of certain groups yields different results (Bechger, van den Wittenboer, Hox & De Glopper, 1999). Goals and objectivesAlthough SAIPÂ’s present objectives are ambitious, u niversal, and aim at continually moving targets, thus ensuring their enduring validi ty, respondents wish to add another objective concerning the specific comparison of res ults from one assessment cycle to the next and with national data. This implies that SAIP should be given the dual goal of providing a snapshot of current achievement levels across jurisdictions and of measuring progress over time. However, in addition to the lac k of proper theory to explain comparative differences in achievement as mentioned above, such an objective can also create tensions such as those experience throughout NAEPÂ’s history, because a single assessment system cannot adequately serve such dive rse purposes, each with its own set of assumptions, processes, and consequences (Linn, 2000). For example, longitudinal programs must find ways to rely on stable and reusa ble instruments while remaining fully aligned with national standards, actual curri culum contents, and evolving theories. In order to meet these challenges, NAEP presently o perates two systems, the Main NAEP for longitudinal measures, and the State NAEP for cross-state comparisons. A recent NAEP review, however, has called for streaml ining and for merging some aspects of the two programs (Pellegrino, Jones & Mitchell, 1999). If the CMEC decided to stress pan-Canadian longitudinal comparisons from one cycl e to the next and comparisons with national averages, then such a decision would have major consequences on the entire programÂ’s structure and development (Bechger et al., 1999). SAIP is currently not designed to meet such goals and does not have the r esources to conduct parallel systems. As with all large-scale studies of achievement, if SAIP decided to opt for the comparative route, then it would have to meet at le ast three conditions to ensure the comparative validity of its results: Construct equi valence, scale equivalence, and
9 of 16measurement equivalence (Bechger, et al., 1999). So far, these equivalencies are unattainable mainly because the theoretical framewo rks underlying the most universally accepted competencies such as reading, problem solv ing, and scientific reasoning are constantly evolving, performance scores across time and groups are interpreted using arbitrary scales, and scale equivalence is greatly hampered by issues such as test translation. Perhaps the only way out at the presen t time is to document and make all related information as comprehensive and transparen t as possible in order to arrive at the most valid interpretations when comparing results.OperationsRespondents generally agree with the existing param eters around SAIPÂ’s administrative, sampling, expectation/standard setting, scoring, an d reporting procedures. Some of their comments, however, stress greater participation of various school board personnel in all stages of SAIPÂ’s assessment programs. Similar state ments are expressed in scholarly papers such as that of Hunter and Gambell (2000), a nd with respect to other large-scale studies such as New ZealandÂ’s National Assessment M onitoring Project (NEMP) (Flockton & Crooks, 1997). They claim greater empow erment and satisfaction by teachers and by other school board members who part icipate in substantial and independent training sessions and in the actual imp lementation of various aspects of the assessment program. Moreover, respondents point tow ard operational changes that consider greater teacher involvement and teacher in put. As a result, the SAIP review offered a recommendation stating that CMEC implemen t, for example, a two-step standard-setting process in which the first series of expectations be formulated principally by teacher representatives.Another operational issue that has not been overtly stated by the respondents but that was raised indirectly through their responses is th e debate over gradeor age-based sampling. Although age-based sampling is seen as a more probabilistic in theory than a grade-based approach, in practice, it is not always respected. Local SAIP coordinators experience difficulty in implementing the recommend ed student sampling process at the school level (Forgette-Giroux & Simon, 2000). This is probably due to the fact that practitioners tend to view a classroom as a unit an d thus see age-based sampling as causing significant disruptions to the class. As a result, student sampling is not done uniformly because of a conscious or unconscious nee d to minimize those perceived disadvantages. Although it is current knowledge tha t, as a result, most schools therefore select students based on their own criteria, unfort unately these practices are not always documented. School sampling is also a problem in ju risdictions with smaller populations because of over-sampling, which means that the same schools participate repeatedly in large-scale programs, again seen by many practition ers as causing significant disturbance.DesignThis section dealt with the format, content, and me asurement qualities of the assessments. Sound decisions about policies and the allocation of resources depend on quality data. Although a significant number of resp ondents support most existing design parameters, they acknowledge the need for greater w illingness by teachers and students to fully engage in the program in order to increase the validity of the data. Despite the respondentsÂ’ understanding that the program is not suitable to offer school-based or
10 of 16individual achievement results, their comments poin t toward the need to publish some type of school-based information, such as thematic reports, that would highlight current exemplary assessment practices. Increased teacher p articipation and support without added stress, time, and effort are also seen as a m eans for attracting teachers to further embrace the process and to encourage their students to provide optimal performances. To that effect, the SAIPÂ’s review thus recommended the exploration of incorporating local assessment practices within the current SAIP design. This give-and-take approach is expected to further empower and meet the needs a nd satisfaction of participating teachers. However, this is easier said than done an d the issue of motivation has been raised in other large-scale studies (Hattie, Jaeger & Bond, 1999; Lane & Stone, 2002; Wilson, 1999). Within the SAIPÂ’s context, external motivation, i.e. students are motivated if teachers are motivated, plays an impor tant role. Offering pizza to participating students, as several jurisdictions do is not enough to entice students to invest wholeheartedly in the assessment. The debate around greater teacher and student involvement leads to several meaningful research qu estions such as: Can such a low-stakes program provide sufficient relevant info rmation to schools to attract teachers and their students to such assessment programs? To what extent would the incorporation of innovative classroom-friendly assessment practic es of socially relevant competencies entice teachers and their students to fully collabo rate? What are some of the most successful practices to motivate students and their teachers to provide optimal performances within age-based sampling programs? Wh at type of feedback would promote greater participation by the teachers and t heir students? SignificanceIn terms of impact, the data indicate that a large proportion of respondents believe that SAIP has little or no influence on various educatio nal contexts at a local level. In that respect, CMEC has not succeeded in implementing som e of the recommendations from the first SAIP review, particularly those addressin g a) better dissemination of results at local level, b) increased SAIP visibility, and c) g reater ownership of SAIPÂ’s objectives by school-board administrators. This implies a defi nite need to develop an awareness-building plan that aims specifically at s chool-based stakeholders. Such an issue can be addressed through various approaches, namely: a) Elaboration of reports for different audiences, b) Publication of clear and pe rtinent frameworks that reflect the most recent theories, research, and practice (e.g., Campbell, Kelley, Mullis, Martin & Sainsbury, 2001; College Board, 1999; CCSSO, 1999a, 1999b; Flockton & Crooks, 1997), c) Release of actual and practice items and tasks, along with their respective scoring guides to serve a innovative practices (e.g ., Robitaille, Beaton & Plomp, 2000), d) Development of a web site that is continually up graded and maintained, e) Support for secondary data analysis studies, and f) Linkage s of SAIP results with those from provincial and international assessment programs. N AEP, TIMSS, and PIRLS have been known for their sustained effort to inform their va rious stakeholders and audiences in a timely fashion and through a variety of reports. NA EP, for example, publishes the following: Report cards, Highlights reports, Instructional rep orts, State reports, Technical reports, Cross-state reports, Trend repor ts, Focussed reports and Service reports and its dissemination process is continually exam ined to improve the usefulness of these various reporting formats (Horkay, 1999). The review of SAIPÂ’s second cycle has led to the formulation of several recommendatio ns to that effect, particularly with respect to the release of sound and clearly articul ated frameworks, exemplary items and
11 of 16scoring guides, and secondary analyses of relations hips among contextual and achievement data.Many of the jurisdictions that participate in SAIP also join other international large-scale assessment programs and most large populations, def ined in terms of number of school boards or districts, implement their own. Eleven Ca nadian jurisdictions took part in the last TIMSS-R assessment and were involved in the PI SA, 2000 program. The survey did not question the practitioners on SAIPÂ’s merits in relation to alternative programs because of the jurisdictionsÂ’ varying degree of inv olvement in these. However, such a question was formally asked to each jurisdictional coordinator or representative. Their answers generally indicated that these programs pro vide valuable complementary data but they do not have a confirmed continuous cyclica l plan and tend to provide a perspective that is more or less detached from the national educational concerns. With its planned long-term assessment cycles, SAIP can c arve out an enviable and enduring place among large-scale evaluations by considering the following three mandates: a) Focussing on cross-curricular competencies that are valued by the Canadian public, b) Adopting a diagnostic function aimed at offering in novative perspectives for adjustment and intervention in educational priorities, in curr icula, and in teaching, assessment and learning strategies, and c) Increasing its usefulne ss toward the smaller size jurisdictions or minority populations.In conclusion, this studyÂ’s objective was to report the voice of one major stakeholder: The school boards. For a variety of reasons, such a s the timing of the investigation, a major overhaul of the educational system, a conside rable employment turnover, and responsibility overload, fewer respondents than ant icipated completed the survey questionnaire. Nevertheless, this voice is a fundam ental one to which any large-scale program should pay particular attention given its f rontline position. In other words, it is the one that provides the raw data. In future revie ws of large-scale assessment programs, this stakeholder should therefore be consulted thro ugh methodologies offering further in-depth and relevant prompting of some of the majo r concerns expressed in this study. This is especially true if the instructional enhanc ement is as much a priority as is educational accountability (Popham, 1999). Such met hodological approaches could perhaps include focus group sessions in which direc tors, superintendents, principals, local coordinators, board consultants, and teachers would share and confront their views regarding those aspects of the assessment programs that have the most impact on them (Haertel, 2002). As this study results show, the mo st pressing issues to be debated would likely include the following: a) Increase teacher a nd student motivation to participate wholeheartedly in the process, b) Develop effective dissemination options, c) Identify ways to ensure that the assessment program can cont inue to provide leadership through innovation in teaching and in assessment practices, and d) Finding cost-effective, yet rigorous means of simultaneously providing snapshot information and longitudinal means of comparisons. Although universally appealin g, such approaches have yet to be supported by sound educational theory and methodolo gy. Finally, despite the fact that a large proportion o f respondents in this study viewed SAIP as having little impact on various educational cont exts and were more or less aware of its objectives, most indicated that they would reco mmend other school boards to participate in SAIP. It appears that for many, SAIP has effectively carved out an important place for itself among large-scale assess ment programs because of its pan-Canadian nature, its capacity to involve all ju risdictions, and its ability to respect
12 of 16many of the technical requirements of such initiati ves (Forgette-Giroux & Simon, 2000). Given the voluntary aspect of this participation an d the importance of gathering valid data, however, SAIP should invest in maintaining it s leadership role by raising its instructional priority, by increasing its visibilit y, and by inspiring local policymakers, administrators, teachers, and students through attr active and meaningful ways.AcknowledgementThe findings reported in this paper are part of a r eview funded by the Council of Ministers of Education, Canada (CMEC) and are publi shed with the written permission of the CMEC. The opinions expressed in this paper h owever do not necessarily reflect the position or policies of the CMEC.Notes 1. http://www.CMEC.ca ReferencesBechger, T.M., van den Wittenboer, G., Hox, J. J., & De Glopper, C. (1999). The validity of comparative educational studies. Educational Measurement: Issues and Practice, 12 (3), 18-26. Boulmetis, J., & Dutwin, P. (1999). The ABCs of evaluation: Timeless techniques for program and project managers. Windsor Ontario: Jossey-Bass Publishers. Campbell, J. R., Kelly, D. L., Mullis, J. V. S., Ma rtin, M. O., & Sainsbury, M. (2001). Framework and specifications for PIRLS assessment 2 001 PIRLS International Study Center, Boston College: Chestnut Hill, MA, USA.Chelimsky, E., & Shadish, W. R (Eds.) (1997). Evaluation for the 21st Century. A handbook Thousand Oaks, CA: SAGE Publications. CMEC (2000). SAIP Science Report 1999 Toronto. College Board (1999). Mathematics framework for the 1996 and 2000 nationa l assessment of educational progress Washington, DC: National Center for Educational Statistics.Council of Chief State School Officers (1999a). Reading framework for the national assessment of educational progress: 1992-2000 Washington, DC: National Center for Educational Statistics. (ERIC Document Reproduction Services ED430209) Council of Chief State School Officers (1999b). Science framework for the 1996 and 2000 national assessment of educational progress Washington, DC: National Center for Educational Statistics. (ERIC Document Reproduction Services ED431618) Cousins, B., & Simon, M. (1993). A review and analysis of the thematic program, Â“Education and work in a changing society" of the S SHRC strategic grants program. Final and Technical reports submitted to Social Sci ences and Humanities Research
13 of 16Council.Crocker, R. K. (1997). Study of the school achievement indicator program Toronto: Council of Ministers of Education, Canada.Crooks, T. (1996). Validity issues in state or national monitoring edu cational outcomes (ERIC Document Reproduction Services ED398285)Crooks, T., & Flockton, L. (1999). The design of New ZealandÂ’s national education monitoring project Paper presented at the annual meeting of the Amer ican Educational Research Association, Montreal, Canada.Flockton, L., & Crooks, T. (1997). Reading & speaking assessment results 1996 Ministry of Education, New Zealand. Dunedin: Educat ional Assessment Research Unit, University of Otago.Forgette-Giroux, R., & Simon, M. (2000). School Achievement Indicators Program Second cycle evaluation Report submitted to the Council of Ministers of E ducation, Canada. Toronto.Hattie, J., Jaeger, R.M., & Bond, L. (1999). Persis tent Methodological Questions in Education Testing. In Asghar Iran-Nejad, & P.D. Pea rson (Eds.), Review of Research in Education, 24 393-446. Haertel, E. H. (2002). Standard setting as a partic ipatory process: Implications for validation of standards-based accountability progra ms. Educational Measurement: Issues and Practice, 21 (1), 16-22. Horkay, N. (Ed.) (1999). The NAEP guide NCES 2000-456. Washington, DC: US Department of Education. National Center for Educat ional Statistics. Human Resources and Development, Canada & Council o f Ministers of Education, Canada. (1999). Memorandum of understanding Unpublished document. Hunter, D., & Gambell, T. (2000). Professionalism, professional development, and teacher participation in scoring of large-scale ass essment. Paper presented at the annual meeting of the Canadian Society for the Study of Ed ucation. Edmonton. Joint Committee on Standards for Educational Evalua tion (1999) Standards for evaluations of educational programs, projects, and materials, 2nd Ed. Toronto: McGraw-Hill-Ryerson.Jones (2001). Assessing achievement versus high-sta kes testing: A crucial contrast. Educational Assessment, 7 (1), 21-28. Lane, S., & Stone, C. A. (2002). Strategies for exa mining the consequences of assessment and accountability programs. Educational Measurement: Issues and Practice, 21 (1), 23-30. Linn, R. L., (2001). The influence of external evaluations on the Nation al Assessment of Educational Progress. CSE Technical Report 548. Los Angeles: Center of t he Study of Evaluation, National Center for the Research on Eva luation, Standards, and Student
14 of 16Testing.Macdonell, A., Forgette-Giroux, R., Schmidt, S., Mo ugeot, Y., & Levesque, J. (1999). Rapport sur lÂ’valuation dÂ’tape du rseau stratgi que en ducation, formation et emploi Conseil de Recherches en Sciences Humaines du Can ada, Ottawa. Madaus, G. F., & Pullin, D. (2000). Questions to ask when evaluating a high-stakes testing program Consortium of Equity in Standards and Testing. Av ailable online: http://wwwstecp.bc.edu/CTESTWEB/documents/CTEST/NCA SPress.pdf Pellegrino, J. W., Jones, L., & Mitchell, K. J. (Ed s.) (1999). Grading the NationÂ’s Report Card: Evaluating NAEP and transforming the a ssessment of educational progress Washington, DC: National Academy Press. Popham, W. J. (1999). Where Large Scale Educational Assessment Is Heading and Why It ShouldnÂ’t. Educational Measurement: Issues and Practice, 18 (3), 18-26. Posavac E. J., & Carey, R. G. (1997). Program evaluation: Methods and case studies (5th Ed.). Upper Saddle River, New Jersey: Prentice-Hal l. Ryan, K. (2002). Assessment validation in the conte xt of high-stakes assessment. Educational Measurement: Issues and Practice, 21 (1), 7-15. Robitaille, D. F., Beaton, A. E., & Plomp, T. (Eds. ) (2000). The impact of TIMSS on the teaching & learning of mathematics & science Vancouver: Pacific Educational Press. Shavelson, R. J., McDonnell, L. M., & Oakes, J. (19 91a). What Are Educational Indicators and Indicator Systems? Practical Assessment, Research and Evaluation, 2 (11). Available online: http://ericae.net/pare/getvn.asp?v=2&n=11 Shavelson, R. J., McDonnell, L. M., & Oakes, J. (19 91b). Steps in Designing an Indicator System. Practical Assessment, Research and Evaluation, 2 (12). Available online: http://ericae.net/pare/getvn.asp?v=2&n=12 Shepard, L. (1977). A checklist for evaluating large-scale assessment p rograms Available online: http://www.wmich.edu/evalctr/pubs/ops/ops09.html Wilde, J., & Sockey, S. (1995). Evaluation handbook New Mexico: EAC West New Mexico Highlands University.Wilson, R. J. (1999). Aspects of validity in largescale programs of student assessment. The Alberta Journal of Educational Research, XLV (4), 333-343.About the AuthorsMarielle Simon is currently Associate Professor at the Faculty of Education, University of Ottawa where she teaches courses in research met hods, assessment, measurement and evaluation. She specializes in classroom and largescale assessment, with particular focus on portfolio assessment and reporting.Email: firstname.lastname@example.org
15 of 16 Rene Forgette-Giroux is Professor with the Faculty of Education, Univer sity of Ottawa. Her studies also focus on classroom and lar ge-scale assessment. She has published on portfolio assessment and grading. She teaches courses in educational research, statistics and assessment.Copyright 2002 by the Education Policy Analysis ArchivesThe World Wide Web address for the Education Policy Analysis Archives is epaa.asu.edu General questions about appropriateness of topics o r particular articles may be addressed to the Editor, Gene V Glass, email@example.com or reach him at College of Education, Arizona State University, Tempe, AZ 8 5287-2411. The Commentary Editor is Casey D. Cobb: firstname.lastname@example.org .EPAA Editorial Board Michael W. Apple University of Wisconsin Greg Camilli Rutgers University John Covaleskie Northern Michigan University Alan Davis University of Colorado, Denver Sherman Dorn University of South Florida Mark E. Fetler California Commission on Teacher Credentialing Richard Garlikov email@example.com Thomas F. Green Syracuse University Alison I. Griffith York University Arlen Gullickson Western Michigan University Ernest R. House University of Colorado Aimee Howley Ohio University Craig B. Howley Appalachia Educational Laboratory William Hunter University of Ontario Institute ofTechnology Daniel Kalls Ume University Benjamin Levin University of Manitoba Thomas Mauhs-Pugh Green Mountain College Dewayne Matthews Education Commission of the States William McInerney Purdue University Mary McKeown-Moak MGT of America (Austin, TX) Les McLean University of Toronto Susan Bobbitt Nolen University of Washington Anne L. Pemberton firstname.lastname@example.org Hugh G. Petrie SUNY Buffalo Richard C. Richardson New York University Anthony G. Rud Jr. Purdue University Dennis Sayers California State UniversityÂ—Stanislaus Jay D. Scribner University of Texas at Austin
16 of 16 Michael Scriven email@example.com Robert E. Stake University of IllinoisÂ—UC Robert Stonehill U.S. Department of Education David D. Williams Brigham Young University EPAA Spanish Language Editorial BoardAssociate Editor for Spanish Language Roberto Rodrguez Gmez Universidad Nacional Autnoma de Mxico firstname.lastname@example.org Adrin Acosta (Mxico) Universidad de Guadalajaraadrianacosta@compuserve.com J. Flix Angulo Rasco (Spain) Universidad de Cdizfelix.email@example.com Teresa Bracho (Mxico) Centro de Investigacin y DocenciaEconmica-CIDEbracho dis1.cide.mx Alejandro Canales (Mxico) Universidad Nacional Autnoma deMxicocanalesa@servidor.unam.mx Ursula Casanova (U.S.A.) Arizona State Universitycasanova@asu.edu Jos Contreras Domingo Universitat de Barcelona Jose.Contreras@doe.d5.ub.es Erwin Epstein (U.S.A.) Loyola University of ChicagoEepstein@luc.edu Josu Gonzlez (U.S.A.) Arizona State Universityjosue@asu.edu Rollin Kent (Mxico)Departamento de InvestigacinEducativa-DIE/CINVESTAVrkent@gemtel.com.mx firstname.lastname@example.org Mara Beatriz Luce (Brazil)Universidad Federal de Rio Grande do Sul-UFRGSlucemb@orion.ufrgs.brJavier Mendoza Rojas (Mxico)Universidad Nacional Autnoma deMxicojaviermr@servidor.unam.mxMarcela Mollis (Argentina)Universidad de Buenos Airesmmollis@filo.uba.ar Humberto Muoz Garca (Mxico) Universidad Nacional Autnoma deMxicohumberto@servidor.unam.mxAngel Ignacio Prez Gmez (Spain)Universidad de Mlagaaiperez@uma.es Daniel Schugurensky (Argentina-Canad)OISE/UT, Canadadschugurensky@oise.utoronto.ca Simon Schwartzman (Brazil)American Institutes for ResesarchÂ–Brazil(AIRBrasil) email@example.com Jurjo Torres Santom (Spain)Universidad de A Coruajurjo@udc.es Carlos Alberto Torres (U.S.A.)University of California, Los Angelestorres@gseisucla.edu