USF Libraries
USF Digital Collections

A program evaluation study of a literacy initiative for students with moderate to severe disabilities

MISSING IMAGE

Material Information

Title:
A program evaluation study of a literacy initiative for students with moderate to severe disabilities
Physical Description:
Book
Language:
English
Creator:
De La Cruz, Carrie F
Publisher:
University of South Florida
Place of Publication:
Tampa, Fla
Publication Date:

Subjects

Subjects / Keywords:
Special education
Significant disabilities
Systems change
Logic model
Qualitative
Dissertations, Academic -- Psychological and Social Foundations -- Doctoral -- USF   ( lcsh )
Genre:
non-fiction   ( marcgt )

Notes

Summary:
ABSTRACT: Recently the National Reading Panel concluded that systematic and direct instruction in phonological awareness, phonics, fluency, vocabulary, and comprehension that is informed by ongoing assessments of student progress results in positive student achievement (NICHHD, 2002). For students with moderate to severe disabilities and students with autism, reading instruction has historically focused on functional sight words. Unfortunately, very little research exists that has examined how the literacy achievement of students with moderate to severe disabilities can be impacted by a more comprehensive, data-driven instructional model. A special education program that serves students with moderate to severe disabilities and students with autism sought to improve reading instruction and literacy outcomes for these students and began the Educational and Life Skills (ELS) Literacy Initiative during the 2005-2006 school year.The purpose of the literacy initiative was to improve teacher skill and confidence in teaching reading, increase the alignment of literacy instruction with the identified best practices, improve the quality of the instructional planning process, and improve student outcomes in the area of literacy. The literacy initiative provided teachers with extensive curricular resources and professional development opportunities in order to achieve the desired outcomes. This study is an evaluation of the ELS Literacy Initiative. More specifically, the goals of this study were to (a) examine how the literacy initiative was being implemented, (b) determine to what extent the anticipated short-term and intermediate outcomes of the initiative were being realized, and (c) determine the next steps in implementation of the literacy initiative.To answer the evaluation questions, a mix of qualitative and quantitative data were collected, including teacher and parent surveys, teacher focus group interviews, and student outcome data. Overall, the outcomes of the ELS Literacy Initiative have been positive, with teachers feeling more confident and supported, instruction being more aligned with best practices, and students having made gains in their literacy skills. However, particular areas of improvement, such as the instructional planning process and curricular resources, should be addressed to meet the needs of students who are nonverbal. A set of recommendations regarding the next steps in the implementation of the ELS Literacy Initiative is included.
Thesis:
Dissertation (Ph.D.)--University of South Florida, 2009.
Bibliography:
Includes bibliographical references.
System Details:
Mode of access: World Wide Web.
System Details:
System requirements: World Wide Web browser and PDF reader.
Statement of Responsibility:
by Carrie F. De La Cruz.
General Note:
Title from PDF of title page.
General Note:
Document formatted into pages; contains 392 pages.
General Note:
Includes vita.

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
aleph - 002029484
oclc - 436923183
usfldc doi - E14-SFE0002907
usfldc handle - e14.2907
System ID:
SFS0027224:00001


This item is only available as the following downloads:


Full Text
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam 2200397Ka 4500
controlfield tag 001 002029484
005 20090917122310.0
007 cr mnu|||uuuuu
008 090917s2009 flu s 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0002907
035
(OCoLC)436923183
040
FHM
c FHM
049
FHMM
090
LB1051 (Online)
1 100
De La Cruz, Carrie F.
2 245
A program evaluation study of a literacy initiative for students with moderate to severe disabilities
h [electronic resource] /
by Carrie F. De La Cruz.
260
[Tampa, Fla] :
b University of South Florida,
2009.
500
Title from PDF of title page.
Document formatted into pages; contains 392 pages.
Includes vita.
502
Dissertation (Ph.D.)--University of South Florida, 2009.
504
Includes bibliographical references.
516
Text (Electronic dissertation) in PDF format.
520
ABSTRACT: Recently the National Reading Panel concluded that systematic and direct instruction in phonological awareness, phonics, fluency, vocabulary, and comprehension that is informed by ongoing assessments of student progress results in positive student achievement (NICHHD, 2002). For students with moderate to severe disabilities and students with autism, reading instruction has historically focused on functional sight words. Unfortunately, very little research exists that has examined how the literacy achievement of students with moderate to severe disabilities can be impacted by a more comprehensive, data-driven instructional model. A special education program that serves students with moderate to severe disabilities and students with autism sought to improve reading instruction and literacy outcomes for these students and began the Educational and Life Skills (ELS) Literacy Initiative during the 2005-2006 school year.The purpose of the literacy initiative was to improve teacher skill and confidence in teaching reading, increase the alignment of literacy instruction with the identified best practices, improve the quality of the instructional planning process, and improve student outcomes in the area of literacy. The literacy initiative provided teachers with extensive curricular resources and professional development opportunities in order to achieve the desired outcomes. This study is an evaluation of the ELS Literacy Initiative. More specifically, the goals of this study were to (a) examine how the literacy initiative was being implemented, (b) determine to what extent the anticipated short-term and intermediate outcomes of the initiative were being realized, and (c) determine the next steps in implementation of the literacy initiative.To answer the evaluation questions, a mix of qualitative and quantitative data were collected, including teacher and parent surveys, teacher focus group interviews, and student outcome data. Overall, the outcomes of the ELS Literacy Initiative have been positive, with teachers feeling more confident and supported, instruction being more aligned with best practices, and students having made gains in their literacy skills. However, particular areas of improvement, such as the instructional planning process and curricular resources, should be addressed to meet the needs of students who are nonverbal. A set of recommendations regarding the next steps in the implementation of the ELS Literacy Initiative is included.
538
Mode of access: World Wide Web.
System requirements: World Wide Web browser and PDF reader.
590
Advisor: Michael J. Curtis, Ph.D.
653
Special education
Significant disabilities
Systems change
Logic model
Qualitative
0 690
Dissertations, Academic
z USF
x Psychological and Social Foundations
Doctoral.
773
t USF Electronic Theses and Dissertations.
4 856
u http://digital.lib.usf.edu/?e14.2907



PAGE 1

A Program Evaluation of a Literacy Initiative for Students W ith Moderate to Severe Disabilities b y Carrie F. De La Cruz A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department o f Psychological and Social Foundations College of Education University of South Florida Major Professor: Michael J. Curtis, Ph.D. Linda M. Raffaele Mendez, Ph.D. Jeffrey D. Kromrey, Ph.D. Deirdre Cobb Roberts, Ph.D. Date of Approval: December 9, 200 8 Keywords: special education significant disabilities, systems change logic model, qualitative nonverbal Copyright 2009 Carrie F. De La Cruz

PAGE 2

Dedication This dissertation is dedicated first and foremost to my husband, Rommel, whose supp ort was unwavering even when my own motivation waned. To Oliver who cheered o mommy go mommy, go during the final weeks of writing To Emerson who arrived late and gave me an extra week to write and who continued to be a ve ry good baby her first two months which allowed me to complete the dissertation To my parents who instilled in me the importance of education and who taught me to believe that I could achieve anything. came to Chicago and gave me the time that I needed after Emerson was born. To m y friends who provided words of encouragement and emotional support when it was needed the most. And finally, t o the students in the EL S program who have taught me never to und erestimate what one can achieve and to their teachers who are steadfastly committed to helping them become all that they can be, including literate adults.

PAGE 3

Acknowledgments I would like to thank the members of my dissertation committee, Dr. Michael Curtis, Dr. Linda Raffael e Mendez, Dr. Jeffrey Kromey, and Dr. Deirdre Cobb Roberts for supporting me through several failed dissertation topics and for sticking with me even when my dissertation approached the 400 page mark. A special thank you to NSSED for allowing me to cond uct the research, to Jennifer Pearson for providing support and guidance as the primary stakeholder in the program evaluation, and to Ellen Hill who is by far the best editor on staff. I would also like to thank the all of the ELS staff members who partic ipated in the study and especially Stefanie Bauer, Amy Cohe n, Kelly Jakymiw, Linda Sever, Dianne Sroka, Kim Swansen and Kristin Swanson whose support was essential to the success of the evaluation.

PAGE 4

i Table of Contents List of Tables vii List of Figures xi Abstract x ii Chapter One : Introduction 1 Chapter Two : Literature Review 7 The Research on Reading 8 Introduction 10 Reading and Literacy Defined 10 Current Legisla tion and the Importance of Quality Reading Instruction 11 Best Pra ctices in Reading Instruction 14 How Reading Develops 14 Conclus ions on Effective Instruction 16 Data Based Decision Making 23 Best Practices in Reading Instruction fo r Students with Moderate to Severe Disabilities 25 Signi ficant Disabilities Defined 26 How Reading Develops 27 The Historical Perspective 30 Conclusi ons on Effective Instruction 33 Phonological Awareness 33 Sight Words 35 Phonics 39 Fluency 45 Comprehension 47 A Balanced Approach 51 Summary 55 Systems Change 57 Systems Change Defined 58 Barrier s to Effective System s Change 60 School Ecology Barriers 61 Human/Cultural Barriers 63 Components of Effective Systems Change 65 Planning for Change 66

PAGE 5

ii Deve loping a Culture of Change 69 Facilitating/Leading Change 72 Staff Development 78 Sust aining Change 82 Conclusions 86 Chapter Three: Method 88 Evaluation Design 88 Procedure 92 Preparation 92 Reasons for the Evaluation 93 S takeholder Identification 93 Evaluator Selection 94 Definition of t he Program t o be Evaluated 95 Program Description 96 Background Information 97 History of the Initiative 100 Scope of the Evaluation 104 Problem Statements 106 Inputs 107 Activities 111 Outcomes 116 Short Term Outcomes 117 Intermediate Outcomes 118 Long Term Outcomes 119 Assumptions 120 Con textual Factors 120 Summary 121 Evaluation Questions 122 Data Collection and Analysis Plan 124 Instruments 126 Teacher Survey 126 Parent Survey 129 Focus Groups 131 Stude nt Literacy Tracking Data 134 L iteracy Benchmarking Data 138 Qualitative Data Analysis 145 Data Triangul ation and Interpretation 152 Reliability and Validity 153 Validity 154 Reli ability 158 Limitations 159 Contextual Limitations 159 Design Limitations 160 Impact Limitations 1 61

PAGE 6

iii D issemination of Information 162 Chapter Four: Results 164 Impleme ntation Evaluation Questions 165 Question 1: How A re the Components of the ELS Literacy Initiative Currently Being Implemented? 165 How is I nstruction Being Delivered? 166 W hat is the Conte nt of Literacy Instruction? 170 Wh o is Providing Instruction? 173 What Resources are Being Used to Provide Liter acy Instruction? 174 What are the Roles of Staff Members in Supporting Literacy Instruction? 179 What Data are Being Collected to Su pport Literacy Instruction? 184 How are Teachers Maki ng Instructional Decisions? 188 Summary 192 Questions 2 and 3: What Factors Serve to Facilitate Implementation of the ELS Literacy Initiative? What Factors Serve as Barriers to the Implementation of the ELS Literacy Initiative? 192 Instructional Materials 193 Staff Related Factors 196 Training a nd Professional Development 199 Student Factors 201 Classroom Environment 202 Parent Factors 203 Summary 205 Outcome Evaluation Questions 207 Question 4 : How are Teachers Using the Resources (Material and People) They Have Been Provided to Help Support Literacy Instruction in Their Classrooms ? 207 Question 5 : How do Teachers Rate Their Feelings of Support, Preparedness, and Confidence as a Result of the Resources That They Have Been Provided Through the Literacy Initiative ? 219 Question 6 : To What Extent is Instruction in the Area of Literacy Align ed With Best Practices and Current Research as Reported by Teachers ? 224 Comprehensiv e Instructional Programming 225 Direct and Systematic Instruction 229 Data to Inform Instructional Pla nning 231 B road Definition of Literacy 233 Incor po rating AAC and Technology 2 35 Inst ructional Individualization 237 Summary 238

PAGE 7

iv Question 7 : To What Extent do Students in the ELS Program Have Access to Appropriate, Research Based Literacy Instruction as Reported by Teachers ? 239 Use of Direct Instruction Programs 243 Use of ELS Literacy Scope and Sequence 245 Use of Ot her Instructional Resources 247 Summary 248 Question 8 : How are Literacy Data Being Used in the Classroom and How do Teachers Rate Those Sources in Ter ms of Helpfulness ? 250 Question 9 : How are the Literacy Benchmark Data Being Utilized ? Are the Data Adequate to Support These Uses? 255 Question 10 : To What Extent do Teachers Believe There is Instructional Continuity for Individual Student s as They Move From One Teacher to the Next? 259 Question 11 : To What Extent do Teachers Believe the Activities of the ELS Literacy Initiative Have Impacted the Inclusion / Integration of Students in ELS Into District Classrooms? 2 63 Questio n 12 : To What Extent do Teachers and Parents Believe the Activities of the ELS Literacy Initiative Have Impacted Generalization of Literacy Skills to the Home? 268 Question 13 : To What Extent Have Student Outcomes in the Area of Literacy Bee n Impacted as a Result of the Literacy Initiative ? 275 Introduction 275 ELS Literacy Tracking Form Data 276 ELS Li teracy Outcomes 2007 2008 279 Literacy Develop ment 2006 2007 to 2007 2008 283 Individual S tudent Literacy Development 289 Summary 291 Survey and Focus Group Data 292 Conclusion 295 Chapter Five : Discussion 296 Purpose of the Study 296 Overview of Method 299 Discussion and Next Steps 301 Short Term Outcomes 302 Instr uctional Resources: Results 302 Instructional Resources: Recommendations 307 Inst ructional Planning: Results 312 Instructiona l Planning: Recommendations 317 Teacher Ski lls and Confidence: Results 322 Teacher Skills and Confidence: Recommendations 323

PAGE 8

v Intermediate Outcomes 3 25 Instruc tional Consistency: Results 325 Instructional C onsistency: Recommendations 327 Integra tion and Inclusion: Results 329 Integration and Inclusion: Recommendations 330 Generalization: Results 331 Gene ralization: Recommendations 332 Long Term Outcomes 3 33 Student Outcomes: Results 333 Studen t Outcomes: Recommendations 335 Post Schoo l Outcomes: Recommendations 337 Mod el Program: Recommendations 338 Sum mary of Recommendations 3 38 Recommendation 1 339 Recommendation 2 339 Recommendation 3 339 Recommendation 4 340 Recommendation 5 340 Recommendation 6 340 Recommendation 7 341 Recommendation 8 341 Recommendation 9 342 Recommendation 10 342 Recommendation 11 342 Recommendation 12 343 Recommendation 13 343 Recommendation 14 344 Recommendation 15 344 Recommendation 16 344 Recommendation 17 345 Recommendation 18 345 Recommendation 19 345 Dissemination of Information 346 Limitations 347 Contextual Limitations 3 47 Design Limitations 3 48 Impact Limitations 3 52 Generalization of Results 353 References 355 Appendices Appendix A: A Logic Model of the ELS Literac y Initiative 366 Appendix B: ELS Literacy Tracking Form 367 Appendix C: Evaluation Questions and Data Sources 368

PAGE 9

vi Appendix D : Teacher Survey 374 Appendix E : Parent Survey 381 Appendix F : Teacher Focus Group Interview Guide 383 Appendix G : Support S taff Focus Group Interview Guide 387 Appendix H : Draft of ELS Literacy Instructional Planning Process 390 About the Author 392

PAGE 10

vii List of Tables Table 1 2007 2008 Participation Rates on ELS Benchmark Assessments by Administration Format and Level 144 Table 2 Teacher Report of the Number of Students for Whom There is a Designated Time in the Schedule for Literacy Instruction Every Day 168 Table 3 Teacher Report of the Number of Students for Whom Literacy Instruction is Provided Throug hout the School Day 169 Table 4 Mean Number of Literacy Skill Areas in Which Students Are Instructed Out of a Possible 8 172 Table 5 Percent of Students Instructed in Each Identified Area of Literacy 172 Table 6 Mean Teacher Ratings of Helpfulness of Material Resources in Implementing Literacy Instruction Where 1 = Not at All Helpful and 5 = Extremely Helpful 175 Table 7 Mean Teacher Ratings of Number of Students for Whom They Use Specific Instructional Resources Where 1 = None of My Students and 5 = All of My Students 175 Table 8 Teacher Report of How Often They Use Data in Developing Student Literacy Instructional Plans 184 Table 9 Teacher Report of How Often They Use Data When Deciding to Make Instructional Changes 185 Table 10 Mean T eacher Ratings of Helpfulness of Data Sources for Planning and Delivering Literacy Instruction Where 1 = Not at All Helpful and 5 = Extremely Helpful 187 Table 11 Teacher Report of the Number of Students for Whom There is a Written Instructional Pla n 191 Table 1 2 Mean Teacher Ratings of Helpfulness of Material Resources in Implementing Literacy Instruction Where 1 = Not at All Helpful and 5 = Extremely Helpful 209

PAGE 11

viii Table 1 3 Mean Teacher Ratings of Number of Students for Whom They Use Specific Instructional Resources Where 1 = None of My Students and 5 = All of My Students 210 Table 1 4 Mean Teacher Ratings of Helpfulness of Data Sources for Planning and Delivering Literacy Instruction Where 1 = Not at All Helpful and 5 = Extremely Helpful 213 Table 15 Mean Teacher Ratings of Helpfulness of People and Professional Development Resources Implementing Literacy Instruction Where 1 = Not at All Helpful and 5 = Extremely Helpful 218 Table 16 Teacher Agreement Wi th the Statement That They ar e Adequately Supported With Sufficient Material Resources for Literacy Instruction 221 Table 17 Teacher Agreement Wi th the Statement That They are Adequately Supported With the Coaching and Professional Development Opportunities Available to Suppo rt Literacy Instruction in Their Classroom 221 Table 18 Teacher Agreement With the Statement That They are Confident Providing Literacy Instruction in Their Classroom 222 Table 19 Teacher Agreement With the Statement That They are Better Prepared t o Provide Literacy Instruction Because of the Supports and Resources That Have Been Provided to Them 223 Table 20 Mean Number of Literacy Skill Areas in Which Students Are Instructed Out of a Possible 8 225 Table 21 Percent of Students Instructed in Each Identified Area of Literacy 226 Table 2 2 Teacher Report of the Number of Students for Whom Literacy Instruction is Provided Throughout the School Day 227 Table 23 Teacher Report of the Number of Students for Whom There is a Designated Time in the Schedule for Li teracy Instruction Every Day 230 Table 24 Teacher Report of the Number of Students for Whom There is a Written Instructional Plan 232

PAGE 12

ix Table 25 Teacher Agreement with the Statement That the Students in Their Classroom Have Access to Appropriate, Research Based Literacy Instruction 242 Table 26 Mean Teacher Ratings of Number of Students for Whom They Use Specific Instructional Resources Where 1 = None of My Students and 5 = All of My Students 243 Table 2 7 Teacher Report o f How Often They Use Data in Developing Student Literacy Instructional Plans 251 Table 28 Teacher Report of How Often They Use Data When Deciding to Make Instructional Changes 251 Table 29 Mean Teacher Ratings of Helpfulness of Data Sources for Pla nning and Delivering Literacy Instruction Where 1 = Not at All Helpful and 5 = Extremely Helpful 254 Table 30 Teacher Report of the Extent to Which There is Instructional Continuity When Students Transition From One Teacher to the Next in the ELS Pr ogram 261 Table 31 Teacher Report of the Extent to Which the Literacy Instruction That has Taken Place in Their Classroom has Improved the Inclusion / Integration of Students Into District Classrooms 264 Table 32 Teacher Report of the Extent to Which the Literacy Skills That They Have Worked on in Their Classroom Have Generalized to the Home Setting 269 Table 33 Parent Report of the Extent to Which the Literacy Skills That Their Child has Gained in the School Setting Have Carried Over to the Home Setting 269 Table 34 Parent Rating of Satisfaction Regarding Communication With Around Literacy Instruction 271 Table 35 Frequency Distributions of Literacy Development: 2007 2008 280 Table 3 6 Frequency Distributions of Lite racy Development for Students With Complete Scores in 2006 2007 and 2007 2008 285 Table 37 The Number of Skill Categories in Which Students Grew One or More Developmental Levels. 290

PAGE 13

x Table 38 The Percent of Students Who Were Rated to have Grown at Least One Developmental Level Within Each Skill Area 291 Table 39 Teacher Agreement with the Statement That Student Literacy Skills Have Improved Because of the Resources in Their Classroom and the Training the Teachers Have Been Provided 292

PAGE 14

xi List of Figures Figure 1. extent of instructional continuity from one teacher to the next in the ELS program. 261 Figure 2. of the degree to which the literacy instruction that has taken place in their classrooms has improved the inclusion/integration of students into district classrooms. 264 Figure 3 The percentage of ELS students falling within each of the developm e ntal levels per skill area. 281 Figure 4 Percentage of students falling within each developmental level for Concepts of Print for 2006 2007 and 2007 2008. 285 Figure 5 Percentage of students falling within each developmental level for Letter Identi fication for 2006 2007 and 2007 2008. 286 Figure 6 Percentage of students falling within each developmental level for Phonological Awareness for 2006 2007 and 2007 2008. 286 Figure 7 Percentage of students falling within each developmental level for Phonics for 2006 2007 and 2007 2008. 287 Figure 8 Percentage of students falling within each developmental level for Spelling and Writing for 2006 2007 and 2007 2008. 287 Figure 9 Percentage of students falling within each developmental level f or Symbol and Word Reading for 2006 2007 and 2007 2008. 288 Figure 10 Percentage of students falling within each developmental level for Vocabulary and Comprehension for 2006 2007 and 2007 2008. 288

PAGE 15

xii A Program Evaluation of a Literacy Initiati ve for Students W ith Moderate to Severe Disabilities Carrie F. De La Cruz ABSTRACT Recently the National Reading Panel concluded that systematic and direct instruction in phonological awareness, phonics, fluency, vocabulary, and comprehension that is in formed by ongoing assessments of student progress results in positive student achievement (NICHHD 2002 ). For students with moderate to severe disabilities and students with autism, reading instruction has historically focused on functional sight words. Unfortunately, very little research exists that has examined how the literacy achievement of students with moderate to severe disabilities can be impacted by a more comprehensive, data driven instructional model. A special education program that serves students with moderate to severe disabilities and students with autism sought to improve reading instruction and literacy outcomes for these students and began the Educational and Life Skills (ELS) Literacy Initiative during the 2005 2006 school year. Th e purpose of the literacy initiative was to improve teacher skill and confidence in teaching reading, increase the alignment of literacy instruction with the identified best practices, improve the quality of the instructional planning process, and improve student o utcomes in the area of literacy The literacy initiative provided teachers with extensive curricular resources and professional development opportunities in order to achieve the desired outcomes.

PAGE 16

xiii This study is an evaluation of the ELS Literac y Initiative. More specifically, the goals of this study were to (a) examine how the literacy initiative was being impleme nted, (b) determine to what extent the anticipated short term and intermediate outcomes of the initiative were being realized, and (c ) determine the next steps in implementation o f the literacy initiative. To answer the evaluation questions, a mix of qualitative and quantitative data were collected, including teacher and parent surveys, teacher focus group interviews, and student outco me data. Overall, the outcomes of the ELS Literacy Initiative have been positive, with teachers feeling more confident and supported, instruction being more aligned with best practices, and students having made gains in their li teracy skills However, particular areas of improvement, such as the instructional planning process and curricular resources, should be addressed to meet the needs of students who are nonverbal. A set of recommendations regarding the next steps in the implementation of the ELS Literacy Initiative is included.

PAGE 17

1 CHAPTER ONE INTRODUCTION The purpose of this evaluation study was to examine the implementation of the Educational and Life Skills (ELS) Literacy Initiative, an initiative within the context of an educational program that serves students with moderate to severe disabilities. The goal of the ELS Literacy Initiative is to apply what is known about best practices in literacy instruction to a unique population that has been largely ignored in the related literature, and to ultimately improve the reading ou tcomes of these students. The ELS program is one of several programs hosted by a special education cooperative located in the northern suburbs of Chicago. The cooperative serves 18 member school districts and supports a total population of approximately 40,000 students. The ELS program is decentralized and has classrooms that are integrated into general education buildings throughout the 18 member districts. The program provides instruction in academic and life skills to students who have moderate to se vere/profound cognitive disabilities, developmental disabilities, multiple disabilities, and autism. Students in the program may have physical disabilities, significant behavioral challenges, medical concerns, and/or expressive and receptive communication difficulties. The program is grounded in a trans disciplinary model, which means that professionals from different disciplines work closely together to provide instructional programming for the students. Each classroom is typically staffed with a full time

PAGE 18

2 teacher, two or more full time teaching assistants, a speech and language pathologist who is in the classroom one and one half days per week, and an intervention specialist who is in the classroom one day per week. Other support personnel who might b e present in the classroom and are part of the educational team include an occupational therapist, a physical therapist, and a nurse. Historically, reading instruction for students with moderate to severe disabilities has emphasized life skills applicati ons and has primarily focused on functional sight word instruction. Prior to systematic efforts to improve literacy instruction, reading instruction had a similar focus in the ELS program. Systematic efforts to improve literacy instruction in the ELS pro gram began during the 2000 2001 school year, when a literacy c ommittee was formed to survey current practices and identify professional development needs. However, the formal work of the ELS Literacy Initiative did not begin until the 2005 2006 school yea r, when the program adopted a comprehensive literacy scope and sequence and identified a core literacy curriculum for the primary and intermediate grade levels. The current study makes use of a program evaluation research design that was selected becaus e the purpose of the study was to collect data that would facilitate decision making (e.g., determining the next steps in implementation) as well as mak ing initial determinations regarding the worth of the program (e.g., the effects on students and teacher s). Prior to beginning the program evaluation, the evaluator worked closely with the program administrator to clarify the reasons for the evaluation and to identify key stakeholders of the initiative and of the evaluation. Key stakeholders were interview ed in

PAGE 19

3 order to gain their input in developing a complete definition of the literacy initiative and to identify the evaluation questions. A description of the ELS Literacy Initiative was put into logic model format (see Appendix A : Logic Model of the ELS Literacy Initiative). A logic model is a visual representation of how a program or intervention strategy is designed to address a specific problem, or set of problems, and how the activities of the program relate to the desired outcomes. In general, the ELS Literacy Initiative was initiated to address lack of teacher training and knowledge in best practices relating to beginning reading instruction for students with significant learning differences and challenges. In addition, the initiative was designe d to correct a lack of research based practices, a lack of curriculum resources for instruction in the area of reading, inconsistencies in instruction between classrooms, failure to use data when making instructional decisions, and disjointed instruction f rom one year to the next. Over the years considerable material, personnel, and financial resources have been dedicated to the support of the ELS Literacy Initiative. The desired outcomes of the initiative can be classified into three categories : short term outcomes, intermediate outcomes, and long term outcomes. The identified short term outcomes include improvements in conditions to support literacy instruction, changes in staff beliefs and skills, and changes in teacher instructional behavior. The d esired intermediate outcomes are those that can be expected to be achieved within two to three years after the formal start of the initiative and they include outcomes such as increased instructional consistency between classrooms and between grades, impro ved student integration into school communities, and increased generalization of reading skills between home and

PAGE 20

4 school. The long term anticipated outcomes are the most important outcomes of the ELS Literacy Initiative and include a) improving student rea ding achievement, b) improving post school outcomes, and c) serving as a model to member districts in the area of literacy instruction. The purposes of the current evaluation study were threefold : 1) to examine how the ELS Literacy Initiative was being implemented, 2) to determine the extent to which the anticipated outcomes of the initiative were being realized, and c) to aid the ELS program administrator in determining the next steps in the implementation of the initiative. The study was designed to a ddress a total of fourteen evaluation questions. The evaluation questions were identified based on input from the stakeholders and were finalized with the program administrator. The three identified goals of the evaluation, along with the fourteen identi fied evaluation questions, formed the direction and foundation of the program evaluation. To best provide answers for the identified questions, the evaluation was multi method in nature and included a mix of qualitative and quantitative data collection and analysis procedures. The methods used included teacher and parent surveys, teacher and from each of these sources were analyzed using the techniques appropri ate for that type of data. Once each of the data sources had been analyzed, the evaluator analyzed all of the information to answer the identified evaluation questions. One of the goals of this study was to support the administrator and stakeholders of the ELS program in identifying the next appropriate steps in the implementation of the ELS Literacy Initiative. Therefore, the degree to which the results can be generalized is

PAGE 21

5 limited. However, the results of the study will have a significant effect on practice within the ELS program and will hopefully result in improved outcomes for students. Of the remaining chapters of this document, Chapter Two consists of two main sections: a review of the literature related to best practices in reading instructi on, and a review of the literature related to systems change. The portion about best practices in reading instruction r eviews the literature pertaining both to students without disabilities and students with moderate to severe impairments. The systems chan ge portion includes both the barriers to effective systems change and the components that facilitate effective systems change. Chapter Three discusses methods for completing this study and first describes the research design and why it was selected. It then describes the several preparatory actions that took place prior to the initiation of the study. A large portion of this chapter is devoted to a description of the ELS Literacy Initiative, including background and historical information as well as the input, activities, and desired outcomes of the initiative. Next it identifies the evaluation questions and outlines the evaluation plan, including data analysis and interpretation procedures. This chapter also discusses limitations of the study, as well as strategies for disseminating information relating to outcomes of the study. Chapters Four and Five describe the evaluation results and provide a discussion of the implication of those results, respectively. The results are organized according to evalu ation questions. Data from different sources are combined and triangulated in order to answer the identified evaluation questions. The final evaluation question (#14: What swered in

PAGE 22

6 Chapter Five. Chapter Five also provides a summary of the program evaluation (including its purpose, method, and results) as well as a discussion of the limitations of the evaluation.

PAGE 23

7 CHAPTER TWO LITERATURE REVIEW The purpose of this lite rature review is to set the stage for the program evaluation study. The purpose of the evaluation study is collect information regarding the implementation of a literacy initiative in a program for students with moderate to severe disabilities and to use t his information to aid the program administrator and other stakeholders in identifying the next steps in the implementation of the initiative. The following review of the literature has two primary components; a review of the research on reading and a r eview of the literature on systems change. The purpose of the first component, a review of the research on reading, is to identify what research suggests is best practice instruction for students without disabilities and for students with moderate to seve re disabilities. The purpose of the literacy initiative (the subject of this evaluation study) is to better align literacy instruction with the research and to ultimately improve the literacy outcomes of the students in the program. The literacy initiat ive necessitates that te achers significantly change the way that does not refer to the actual changes in instruction but to the process that the program and the staff go t hrough in order to make those changes. The second component of this literature review is on systems change and the factors that serve as barriers to change and the factors that serve to facilitate change.

PAGE 24

8 The Research on Reading The importance of readin g and becoming literate cannot be overstated (Snow, Burns, & Griffin, 1998). The authors of What Work Requires of Schools: A SCANS Report for America 2000 (U.S. Department of Labor [USDOL], 1991) identified literacy (reading, writing, speaking, and list ening) as a foundational skill for later success in job performance. The report further emphasized that work success requires not only a basic skill level in this area, but that the demands of the current workplace requires a higher level of skill, with employees having to be able to understand and interpret a diverse set of materials. In the last two decades, a tremendous amount of work has been done to conduct new research and synthesize existing research in order to identify those practices that lea d to the most positive outcomes when it comes to students learning how to read and becoming literate adults. The importance of this work has been reflected in current acts of legislation, which have placed emphasis on the prevention of reading difficultie s through scientifically based instruction (No Child Left Behind [NCLB] Act of 2001, PL 107 110, Individuals with Disabilities Education Improvement Act [IDEA] of 2004, PL 108 446). In addition to scientifically based reading instruction, these pieces of legislation both recommend a tiered model of instruction and the use of data to make important educational decisions. Evidence suggests that when all three of these components are in place (scientifically based instruction delivered in a tiered model of s upports in which instructional decisions are driven by data) that student outcomes improve (Burns, Appleton, & Stehouwer, 2005).

PAGE 25

9 Unfortunately, the work that has been done to identify best practices in reading instruction in the last two decades has not included research on those students with the greatest need: students with moderate to severe cognitive and physical disabilities, and students with autism. Historically, reading instruction for this population of students has placed emphasis on life skil l applications, and has primarily taught students functional sight words. Only recently have experts in the field begun to apply what is known about best practices in reading instruction to students with moderate to severe disabilities (Browder & Spooner, 2006). However, there remains a significant need for research to determine whether of not these best practice strategies are effective for this population (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine, 2006). This current section on The Res earch on Reading will begin with an introduction that will define reading and identify the importance of literacy, and will also discuss the impact of current legislation on reading instruction. Following that will be a review of the research on reading f or typically developing students, including how reading develops, the conclusions that can be drawn from synthesis reports of the literature regarding effective instruction, a description of tiered models of reading support, and an explanation of the use o f data to drive instruction. The research regarding best practices in reading instruction for students with moderate to severe disabilities will then be reviewed, including how reading develops for this population, what the historical perspective on readi ng instruction has looked like, and what the research says about best practices in reading instruction for students with significant disabilities.

PAGE 26

10 Introduction Reading and Literacy Defined posited that the ultimate goal for reading is comprehension of the text that is being read. He suggested that this comprehension depends on two sets of skills. The first set of skills includes those skills that are required to identify the words in the t ext with fluency, such as understanding that words are made up of sounds (phonological awareness), mastering the phoneme to grapheme relationship (phonics) and the automaticity of the application of these skills (fluency). Once the reader has been able to accurately and fluently identify the words in the text, the second set of skills required for comprehending text are the same skills that are needed for verbal comprehension of spoken words, but applied to reading. These skills include having the backgro und knowledge and experiences to know the meaning of the individual words in the text (vocabulary) and the ability to put all of the components of the text together in order to construct meaning of what is being communicated (comprehension). Put in anothe reading is that readers must simultaneously use their word recognition skills to read the text accurately while using their comprehension skills to make meaning of the text that is being read. Some have argued over t he value of teaching literacy to students with moderate to significant disabilities because of the need for this population to attain functional life skills. However, others have identified the importance of literacy as a functional life skill, especially when literacy is defined broadly. Downing (2006) conceptualized a definition of literacy that includes learners of all ability levels, not just those who

PAGE 27

11 eventually learn to read connected text. According to Downing (2006), literacy includes highlights the close relationship that literacy has with expressive and receptive communication -especially for those with significant cognitive disabilities -and how important it is as a life skill. Houston and Torgesen (2004) suggested that reading is a al skill that leads to improved outcomes in all students. Current Legislation and the Importance of Quality Reading Instruction There are two significant pieces of federal legislation that govern instructional practices for students in general and speci al education. The first is the No Child Left Behind (NCLB) Act of 2001 (PL 107 110), formerly known as the Elementary and Secondary Education Act, and the second is the Individuals with Disabilities Education Improvement Act (IDEA) of 2004 (PL 108 446). Both the NCLB and the IDEA prioritize high quality scientifically based reading instruction for all students, including those with the most significant disabilities. NCLB has been one of the greatest legislative influences on education in decades. Three days after taking office, President George Bush announced his plan for educational reform and what he described as the cornerstone of his administration. Signed in January of 2002, NCLB brought high standards, accountability for academic outcomes, an emp law. The primary goals for NCLB are for all students to reach high learning standards,

PAGE 28

12 be taught by highly qualified teachers, attain proficiency in English, be educat ed in environments conducive to learning, and graduate from high school. NCLB places a strong emphasis on the use of research based educational practices, particularly in the area of early reading development. Special funding opportunities were included in the NCLB legislation through the Reading First and Early Reading First initiatives. The Reading First initiative provides grant monies to states that in turn award competitive grants to local communities in order to support scientifically based early literacy and early screening programs, as well as professional development opportunities for education personnel in reading instruction. The Early Reading First initiative offers funding for existing preschool and Head Start programs to support the use of scientifically based practices to promote early reading development. Another important component of the NCLB is accountability for all students. This piece of federal legislation requires that schools demonstrate adequate yearly progress (AYP) for all s tudents, including those with the most significant disabilities. Alternate achievement measures that are aligned with state standards are allowed for the approximate 1% of students who are not able to participate in the state standards test even with acco mmodations. In 1975, special education was mandated nationally for the first time through the Education for All Handicapped Children Act (EHA). Now renamed the Individuals with Disabilities Education Improvement Act (IDEA) of 2004 (PL 108 446), the pu rpose of this legislation is to ensure a free and appropriate education for children with disabilities. Despite being a piece of legislation that governs special education, the IDEA places great

PAGE 29

13 emphasis on early intervention and prevention, high quality scientifically based instruction, and accountability for all students, very similar to NCLB. While pre referral interventions were recommended as part of IDEA 1997, IDEA 2004 places even greater emphasis on early intervention services with the hopes of r educing over identification and unnecessary referrals to special education. The goal is to prevent problems before they need intensive resources and require special education support to be remediated. Significantly, IDEA 2004 allows districts to use up to 15% of their federal special education funds to go toward early intervention including the development, implementation, and maintenance of these intervention services. Specific emphasis is placed on the early grade levels (K 3) and interventions that are scientifically based, including literacy instruction. Accountability is also a significant component of IDEA 2004. Changes in the IDEA 2004 reflect less of an emphasis on compliance with rules and more of an emphasis on student outcomes. Very similar t o accountability guidelines posed by NCLB, the IDEA requires that the achievement of all students be measured against state standards, and that alternate assessments be available for those students who are not able to participate in the state assessments. Furthermore, districts must report the results of these assessments disaggregated by subgroup, including those in special education. In summary, the NCLB emphasizes and promotes scientifically based instructional practices, early intervention and preven tion, early reading and literacy development, teacher training and professional development, and accountability for results in student academic achievement. Importantly, these priorities are in place for all students, even those with the most significant disabilities. IDEA 2004 also prioritizes

PAGE 30

14 early intervention and prevention, scientifically based instruction and intervention, and accountability for the outcomes for all students. The overall message being sent by both of these important pieces of legis lation is that reading and literacy are important, and that schools must be held accountable for the achievement of all students in this area. Best Practices in Reading Instruction Because of the work that has been done by researchers in the last two d ecades, strong conclusions are able to be made regarding how reading develops and the most effective strategies for teaching reading (National Reading Panel, 2000). This section on the Best Practices in Reading Instruction will describe how reading skills develop in most children, and will identify the most effective instructional strategies as described by two important synthesis reports. Two other topics will be included in this section on best practices. The first is the use of a tiered model for conc eptualizing reading instruction and intervention, and the other is the use of a data driven process for making instructional decisions. How Reading Develops Houston, Al Otaiba, and Torgesen (2006) described a common path along which children learn to rea d. This path includes three distinct stages of development: prereading, learning to read, and reading to learn. The prereading stage of development serves as the foundation for later learning. This stage mostly consists of language development (express ive and receptive), which serves as the foundation for later comprehension of written text. This stage also includes children learning about the world around themselves, which fosters vocabulary development and verbal thinking skills. Children at the pre reading stage of development

PAGE 31

15 are beginning to take interest in books and learning how they work (which way to hold the book, turning pages left to right, understanding that the words carry the message, etc.). Some initial knowledge of letters, a beginning awareness of the phonological structure of words, and recognition of environmental print (those words that are recognizable in the context in which they occur, such as ) and very familiar is stage of development. During the learning to read stage, children are learning the skills necessary to identify printed words accurately and fluently. Their awareness of the sounds that make up spoken words (phonological awareness) becomes much more sophisticated, and at the end of this stage they are able to masterfully manipulate phonemes to make new words, children also master the ability to associate sounds wit h letters, and to use these sounds to form words, also known as the alphabetic principle. At the beginning of the learning to read stage, children shift from first using arbitrary features to identify words, to guessing on words based on one or two lette rs in the word, and finally to accurately decoding and identifying words using their knowledge of sound and letter relationships. Through their practice with reading and decoding words, students at this stage of development are able to recognize an increa sing number of words by sight. This is the key to fluent reading, which marks the end of this stage of reading development. Once students are accurate and fluent readers, their attention during the reading process shifts away from decoding words, to com prehending the text that is being read. Students at this stage are actually thinking about what they are reading while they are reading. Once this shift occurs, the reading to learn stage of development begins.

PAGE 32

16 Throughout this stage, students continue t o expand their background knowledge and vocabulary in order to support comprehension. They become more skilled at applying strategies for gaining meaning from written text and expanding their understanding of what they have read. Eventually, students are able to construct their own judgments of text and are able to identify different viewpoints within the text. This stage of reading development does not end and continues to develop through adulthood. Conclusions on Effective Instruction Given the impo rtance of reading as a life skill, it is important to identify scientifically based, effective instructional practices in the area of reading. Two groups were commissioned to review existing research and to identify the practices and strategies that have been empirically found to be effective. The National Academy of Sciences published their synthesis report in 1998 titled Preventing Reading Difficulties in Young Children (Snow, Burns, & Griffin, 1998). Two years later, the National Reading Panel publis hed their meta analysis of the literature on reading instruction, the Report of the National Reading Panel (National Institute of Child Health and Human Development [NICHHD], 2002). Both reports contain important information on teaching children to read, and their results are briefly summarized below. Recognizing the importance of reading to student achievement and to society, the U.S. Department of Education and the U.S. Department of Health and Human Services asked the National Academy of Sciences to s tudy the prevention of reading difficulties in young children. The result of their efforts was the synthesis report, Preventing Reading Difficulties in Young Children (Snow, Burns, & Griffin, 1998). The report summarized the existing research on the proc ess of learning to read, and it reported both risk factors

PAGE 33

17 and predictors of success in learning to read. Recommendations were made regarding effective instructional practices for students who were pre kindergarten through grade three, as well as for futu re research in the area of reading. Throughout the report, several themes emerged including the importance of preventing reading difficulties, providing quality instruction, and having professional development for educational personnel. As part of their report, Snow, Burns, & Griffin (1998) concluded that preventing reading difficulties begins at home before students enter preschool or kindergarten. They recommended that parents provide their young children with literacy rich environments, including boo ks and environmental print, and that they should regularly share literacy experiences with their children. They further recommended that children have opportunities to learn about books and book conventions, to listen to stories and have opportunities to ask questions and discuss those stories, and to engage in other print activities such as scribbling and pretending to write. The priority at this very early age should be with providing rich language and literacy experiences so that children can develop s trong vocabularies and concepts of print (Snow, Burns, & Griffen, 1998). demonstrated that (a) adult child shared book reading that stimulates verbal interaction can enhance langua ge (especially vocabulary) development and knowledge about concepts of print, and (b) activities that direct young children's attention to the sound structure within spoken words (e.g., play with songs and poems that emphasize rhyming, jokes, and games tha t depend on switching sounds within words), and to the relations concluded that preschool experiences should include adult child shared book reading

PAGE 34

18 with opportunit ies for verbal interaction and vocabulary development. Preschool structures of speech and highlight the relationship between speech and print. The report characterized qu ality instruction in kindergarten as focusing on the alphabetic principle, i.e., the understanding that spoken words are made up of sounds, and that written letters can be used to represent those sounds. Additionally, it was recommended that kindergarten classrooms be language and vocabulary rich, and should include instruction in writing and the comprehension of text. In their recommendations for first grade instruction, Snow, Burns, & Griffin (1998) emphasized the importance of teaching letter/sound rel ationships, providing opportunities for reading connected text with fluency, and continuing instruction in writing and reading comprehension. For instruction in the second grade and above, the recommendations were to continue teaching students strategies for sounding out words for the purpose of decoding them, and to provide explicit instruction in comprehension strategies such as summarizing the main idea, predicting events and outcomes, and drawing inferences. Direct and systematic instruction in the a reas of vocabulary, fluency, and writing should also be continued at this level, according to the report. Overall, the recommendations were for comprehensive instructional programming in the area of literacy that teaches skills through direct and explicit instruction. In 2000, the National Reading Panel (NRP) issued a consensus report in response to a congressional mandate to identify the skills and methodologies central to effective reading instruction. The Panel sought to go a step further than the 1998 consensus report, Preventing Reading Difficulties in Young Children (Snow, Burns, & Griffin) by

PAGE 35

19 conducting meta analyses of the research relating to the critical skills identified in the earlier report in order to formulate sound conclusions regarding the most effective strategies for reading instruction. They also conducted a series of public hearings to identify other important areas of study. The Report of the National Reading Panel; Teaching Children to Read: An Evidence based Assessment of the Scie ntific Research Literature on Reading and its Implications for Reading Instruction ( NICHHD 2000) is the most current and comprehensive review of reading research to have been published to date. In their initial search, the committee reviewed more than 100 ,000 studies on reading and, through a careful screening process, included only those that (1) focused directly on reading development in Kindergarten through grade two, (2) were published in a peer refereed journal, (3) utilized an experimental or quasi e xperimental research design, and (4) had an adequate sample size. The Report of the National Reading Panel provides analyses and discussion regarding what they identified as the five necessary components of effective reading instruction: phonemic awarene ss, phonics, fluency, vocabulary, and text comprehension (NICHHD, 2000). Phonemic awareness is the understanding that words are made up of individual sounds. When students have mastered phonemic awareness, they can listen to a spoken word and identify th e individual phonemes (or sounds) in that word, and even manipulate those sounds to change the word. While phonemic awareness is the ability to identify phonemes (the spoken sounds in words), phonics is the understanding of the relationship of the phoneme s to graphemes (the letters and spellings that represent the phonemes) and the ability to apply this understanding to decode unfamiliar words. Fluency, or the ability to read connected text accura tely and smoothly, has a strong

PAGE 36

20 relationship with comprehen sion. Text comprehension is the ability to read with understanding and to communicate with others what has been read. Vocabulary, the fifth component of effective reading instruction, is the ability to understand the meanings of words that are necessary for text comprehension and communication in general. With regards to phonemic awareness, the Panel found that teaching children early to manipulate the phonemes in words significantly improves their ability to learn to read and to spell. The Panel also found that explicit, systematic instruction in phonics is an essential component for any reading program, as it was shown to improve reading and spelling abilities across students of different ages, grade levels, ability levels, and socioeconomic statuses. The Panel defined systematic phonics instruction as the explicit, planned, and sequential teaching and practice of letter sound correspondences and their use in reading and spelling words. With regard to fluency, the Panel identified two major instruct ional approaches, guided repeated oral reading practice and approaches that make a formal effort to increase the amount of independent, or recreational reading. Results of a meta analysis of the research suggested that guided repeated oral reading procedu res were more effective in that they had a consistent and positive impact on word recognition, fluency, and comprehension, as measured by a variety of tests, across a range of grade levels. Several conclusions were offered regarding vocabulary instructio n, most importantly, that direct and indirect instruction be included in any comprehensive reading program. Finally, the Panel concluded that text comprehension can be significantly aided through the use of several reading comprehension strategies, and th at the use of these

PAGE 37

21 strategies leads to increased specific transfer of learning, retention and understanding of new passages, and general improvements in comprehension. In addition to drawing conclusions and making recommendations regarding the five neces sary components of effective reading instruction, the Panel studied other topics such as teacher preparation and the use of technology in reading instruction. Based on the studies that were reviewed, the Panel concluded that both pre service and in servic e teacher training increased knowledge and skills in the area of reading instruction and, in many cases, lead to an increase in student achievement. Therefore, it was recommended that increased emphasis be placed on effective instructional reading practic es at both the pre service and in service levels of teacher education. The Panel also concluded that while computer technology is currently being used to support traditional means of instructing children in reading (e.g., completing worksheets on the comp uter), other uses of computers to improve reading instruction should be investigated. In conclusion, the Report of the National Reading Panel (NICHHD, 2000) has definitively identified effective instructional practices in the area of reading. Additionall y, the Panel concluded that there is enough known about most of the identified reading practices and strategies to justify their immediate implementation in the class Tiered Models of Reading Prevention and Intervention In their report, Preventing Reading Difficulties in Young Children Snow, Burns, and Griffin (1998) recommended a model of prevention and intervention that relied on different levels of support for students with different levels of need. They specifically identified three levels of support, primary prevention, secondary prevention, and tertiary prevention. This conceptualization of a multi tiered system of support was first

PAGE 38

22 introduced in the area of mental health and behavior prevention and intervention (Adelman & Taylor, 1998). This model has since been adopted by those promoting Response to Intervention (RtI) (Batsche, et al., 2005) and has been implemented successfully in a large number of school systems (Burns, Appleton, & Stehouwer, 2005). In a tiered model of support, the intensity o f instruction and intervention is carefully matched to student level of need. The first level of support, also known as the primary prevention level, or Tier 1, is provided to all students, and the goal of this level of support is to effectively meet the needs of as many students as possible and to reduce the number of students who will require additional support (Snow, Burns, & Griffin, 1998). In this way, Tier 1 is intended to be preventative and proactive support (Batsche, et al., 2005). The reading instruction important that this program be scientifically validated and implemented by teachers who are trained to use the program (Batsche, et al., 2005). In any gi ven school system, there are going to be students who need additional support beyond what is provided at Tier 1 in the core instructional program. The second tier of support, or the secondary prevention level, is intended to meet the needs of those studen ts. The goal of this level of support is to remediate the identified concern so that additional support is no longer necessary, and to prevent the problem from becoming a serious, long term deficit (Snow, Burns, & Griffin, 1998). At this level of support supplemental reading instruction is provided in addition to the core reading program. The supplemental instruction is typically delivered in small groups, has a strong research

PAGE 39

23 support for its effectiveness, and is of an intensity that is intended to qu ickly remediate the academic concern (Batsche, et al., 2005). The tertiary prevention level of support, or Tier 3, is the most intensive level of support, and is reserved for those students who have the most intensive instructional needs (Snow, Burns, & G riffin, 1998). The students who are given this level of support are those who did not show adequate progress when given Tier 1 and Tier 2 supports, and who now have an academic deficit that will require an intervention that is going to reduce the signific ance of the identified problem. Batsche, et al., 2005, described Tier 3 interventions as interventions that are longer in duration, delivered in very small groups or in a one to one setting, and provide direct and explicit intensive interventions that hav e been empirically proven to provide results. Data based Decision Making In a three tier model of prevention and intervention, instructional support is provided in direct proportion to student need in a proactive and preventative manner (Batsche, et al., 2005). In order to effectively match student need to level of support, it is important that a systematic data driven process be used to make instructional decisions. The use of a problem solving decision making model that is based on student data is con sidered to be a best practice in delivering effective instruction (IDEA, 2004). Within a problem solving decision making model of instructional delivery, data are used to a) identify students in need of additional support (i.e., universal screening), b) d etermine whether students are making adequate progress over time (i.e., progress monitoring), and c) monitor student response to instructional changes that are made (i.e., improve instruction) (Fuchs & Fuchs, n.d.)

PAGE 40

24 Batsche, et al., (2005) identified an integrated student data collection and assessment system as an essential component of an effective tiered service delivery model. Without data, there is no foundation upon which to make decisions regarding student instructional need. Nine characteristics of assessment procedures that would be effective in supporting a tiered model of service delivery were identified by Batsche, et al. (2005). According to Batsche, et al. (2005, p. 25), effective assessment procedures: 1. directly assess the specific skill s embodied in state and local academic standards; 2. instructional target (e.g., reading comprehension); 3. are sensitive to small increments of growth over time; 4. can be administere d efficiently over short periods; 5. may be administered repeatedly (using multiple forms); 6. are readily summarized in teacher friendly data displays; 7. can be used to make comparisons across students; 8. over time; and 9. have direct relevance to the development of instructional strategies that address the area of need. Curriculum based measurement (CBM) has been identified as an assessment tool that meets the qualities of effective assessment procedures as identified by Batsche, et al. (2005) and can support the educational and instructional decisions that are made within a tiered model of service delivery (Fuchs & Fuchs, n.d.). CBM assessment tools are standardized, brief measures that require students to perform authentic tasks such as

PAGE 41

25 reading aloud for one minute, or producing a writing sample in three minutes. Because these tools measure both accuracy and amount of student behavior performed within a specified time period, they are considered measures of fluency. According to the National Center on Student Progress Monitoring, there are over 200 empirical studies demonstrating the reliability and validity of the CBM family of tools and documenting positive student outcomes that are related to the use o f these tools (Fuchs & Fuchs, n.d.). This discussion, titled Best Practices in Reading Instruction, has reviewed the research on reading for typically developing students, including how reading develops, the conclusions that can be drawn from synthesis re ports of the literature regarding effective instruction, a description of tiered models of reading support, and an explanation of the use of data to drive instruction. The following discussion will examine the research on best practices in reading instruc tion for students with moderate to severe disabilities. In some respects, the best practices in reading instruction for students with moderate to severe disabilities mirror the best practices in reading instruction for typically developing students. In o ther areas, there is a lack of research in the area and/or the research suggests somewhat different practices for this population of students, mainly to accommodate their unique instructional needs. Best Practices in Reading Instruction for Students With Moderate to Severe Disabilities In 2000, the National Reading Panel (NRP) conducted a meta analysis of the research on reading in order to formulate sound conclusions regarding the most effective strategies for reading instruction. While the conclusions d rawn from this report have had a significant impact on legislation and practices in schools, it appears as if this information has not translated to instruction for students with moderate to severe

PAGE 42

26 disabilities (Houston, Al Otaiba, & Torgesen, 2006). The meta analysis conducted by the NRP (2000) excluded most of the research on students with cognitive (IQ) scores below 70. The following discussion identifies the available research on teaching reading to students with moderate to severe disabilities and id entifies best practices in the area. This discussion of the best practices in reading instruction for students with moderate to severe disabilities begins with a definition of significant disabilities then reviews how reading develops for this population of students. The historical perspective on the reading instruction for students with moderate to severe disabilities is also included. Next, the research on effective reading instruction for students with significant disabilities is reviewed in the area s of emergent literacy, augmentative and alternative communication, phonological awareness, sight words, phonics, fluency, vocabulary, comprehension, and a balanced approach to literacy instruction. This discussion concludes with a review of the use of da ta to make instructional decisions when providing reading instruction for students with significant disabilities. Significant Disabilities Defined in reality a very diverse g roup of students, which makes the possibility of defining the population difficult. Browder, Wakeman, Spooner, Ahlgrim Delzell, Algozzine (2006) classified as ha ving moderate to severe mental retardation, who may have additional functional definition of students with significant disabilities, it does not fully capture the diversit y of the students and their needs.

PAGE 43

27 Students identified as having moderate to severe disabilities have needs in multiple areas (Orelove & Sobsey, 1996). In addition to cognitive impairments, these students often have physical and medical needs such as re striction of movement, vision and hearing loss, seizure disorders, diet restrictions, and other medical complications. Students with moderate to severe disabilities often have deficits in receptive and expressive communication. The social and emotional n eeds of these students may include social skill deficits and inappropriate and potentially dangerous behaviors. Despite the numerous limitations that students with significant disabilities face, they can often achieve more than what is expected of them ( Gurry & Larkin, 1999). When expectations are held high and when provided with quality instruction, even students with the most significant disabilities can demonstrate the ability to acquire and use skills, including literacy skills (Al Otaiba & Hosp, 200 4; Kliewer & Biklen, 2001; Kliewer & Landis, 1999). How Reading Develops Gough (1996) presented a simple view of reading in which reading is defined as the simultaneous use of word recognition skills to read text fluently and accurately and of reading co mprehension skills to make meaning of what is being read. According to Downing (2006), the definition of literacy posited by Gough (1996) inherently excludes students with significant disabilities who are unable to access print. Downing (2006) suggested definition includes all communication and encompasses all activities related to learning about and sharing information with others. In this sense, literacy is reading, writing, sp eaking, and listening.

PAGE 44

28 Research suggests that students with moderate to severe can benefit from a comprehensive reading program and make improvements in their reading abilities (Al Otaiba and Hosp, 2004; Hedrick, Katims, and Carr, 1999). These research findings may suggest that for students with moderate to severe disabilities, reading may develop similarly as it does for students without disabilities. If reading does develop similarly, then one might assume that the results of the National Reading Pan el (2000) can be applied to students with moderate to severe disabilities. However, there is some evidence to suggest that reading and literacy does not develop in the same way for students with moderate to severe disabilities as it does for students wi thout disabilities. One factor that may contribute to differences in how reading develops is the life experiences of the two groups beginning early in childhood. The earliest experiences that students have with literacy and language have been found to ha ve a significant impact on their later abilities to learn how to read (Snow, Burns, & Griffin, 1998). These early experiences take place at the pre reading stage of reading development and are sometimes referred to as emergent literacy. For children who are typically developing, emergent literacy begins in the home as parents provide their children with literacy rich environments (access to books and environmental print) and regular opportunities for shared book reading. The parents of these children also provide varied life experiences and talk about these experiences with their children in a way that builds their background knowledge, vocabulary, and expressive and receptive communication skills. These early experiences provide a foundation for later le arning (Houston, Al Otaiba, & Torgesen, 2006).

PAGE 45

29 Students with moderate to severe disabilities do not get the same emergent literacy experiences as students without disabilities (Downing, 2006; Gurry & Larkin, 1999; Koppenhaver, Coleman, Kalman, & Yoder, 19 91). Because of their unique needs, the early childhood experiences of students with moderate to severe disabilities may not have included environments that were rich in language and literacy experiences. In fact, Light and Kelford Smith (1993) found th at preschoolers who used augmentative communication systems, when compared to non disabled peers, were read to less often and had limited access to writing and drawing materials at home. A survey of parents of children with Down syndrome found that these parents spent less time reading to their children with disabilities than the parents of children without disabilities and that their Schuele, 1995). The parents of c hildren with moderate to severe disabilities are more likely to prioritize medical issues and other developmental areas, such as gross and fine motor development (Gurry & Larkin, 1999). As described by if, as a parent, your six yea r old has been diagnosed with mental retardation, a seizure disorder, demonstrates self stimulatory behaviors, lacks eye contact, has never reacted to the books in his room or your attempts to read to him, and sleeps less than six hours a night, the chance s are high that you would consider reading a low web page). Because of the potential differences in how reading develops between typically developing students and students with significant disabilities, Kliewer and Biklen (2001) sugg skills that build upon one another so that higher rungs on the ladder cannot be attained

PAGE 46

30 with out mastery of the lower rungs. According to Kliewer and Biklen, for students with significant disabilities, the attainment of literacy is not so much a ladder as a web of skills, which allows for multiple paths to lead to the same place or outcome (Kliewer & Biklen, 2001). Differences in how reading develops for students with sig nificant disabilities have implications for the literacy instruction for this population. The Historical Perspective While individuals with mental retardation have been around since the beginning of recorded history, the first documented attempts to prov ide systematic instruction in the area of reading to these individuals have been relatively recent (Katims, 2000). Katims (2000) reviewed the historical research on literacy instruction for individuals with mental retardation, dating as far back as possib le. For the purposes of his review, Katims (2000) classified instruction for these individuals into two categories, reductionist interventions and constructivist interventions. Reductionist interventions are those that attempt to teach literacy in a syst ematic, but isolated and decontextualized fashion. These constructivist interventions are those that provide literacy instruction in an integrated and contextuali zed manner. These interventions often combine reading and writing instruction and place a strong emphasis on language experiences. In his summary of the research, Katims (2000) concluded that historical literature on the reading and writing instruction f or individuals with mental retardation over the last two decades has favored a reductionist, decontextualized approach to reading instruction with emphasis on individual skill instruction using drill and practice.

PAGE 47

31 To assess the current state of literacy instruction for students with mental retardation, Katims (2000) conducted a qualitative analysis of the content of contemporary textbooks of both special education and mental retardation. He reviewed the textbooks looking for how the text presented the ac ademic characteristics, assessment procedures, and instructional procedures, related to the literacy (reading, writing, spelling) instruction of students with mental retardation. Based on the level of information the textbook provided in each of the ident ified areas, Katims gave a rating of extensive, useful, limited, or none. The same textbooks were also analyzed for their content related to the literacy instruction for students with learning disabilities. There were eight special education textbooks r eviewed for each of the three indicators (academic characteristics, assessment procedures, and instructional procedures) which resulted in a tota l of 24 possible indicators. Twenty of the 24 a mention of that textbook for its description of instructional procedures for students with mental retardation. In contrast, when a similar analysis was completed of the same textbooks, but for information pertaining to students with learning disabilities, only one of the possible 24 ind of the 24 indicators were rated as mental retardation with the hypothesis that these more specific textbooks would contain more information on literacy instruction for this population. Katims found that slightly more information was provided in these texts on the literacy characteristics, assessment, and instruction for

PAGE 48

32 students wit h mental retardation, but only three of the 10 indicators were classified as Based on his historical review and analysis of contemporary textbooks, Katims (2000) concluded that little e mphasis has been and is currently placed on the literacy instruction of students with mental retardation. Furthermore, the instruction that does happen tends to be decontextualized, drill and practice activities. It is clear that the literacy instructio n of students with significant disabilities has not been a priority in the field and recent reviews of research related to literacy instruction for students with moderate to severe disabilities support this conclusion (Browder, Wakeman, Spooner, Ahlgrim De lzell, & Algozzine, 2006; Connors, 1992; Joseph & Seery, 2004). Even when research is conducted, it does not reflect the emphasis on research for students without disabilities. For example, Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine (2006) r eviewed the existing research on reading instruction for students with moderate to severe disabilities in order to provide a comprehensive synthesis of the research, and determine evidence based practices in this area. The group reported the results of th eir study using the five big ideas of reading posed by the National Reading Panel: phonological awareness, phonics, fluency, comprehension and vocabulary (NRP, 2000). A total of 119 publications that used experimental or quasi experimental designs (includ ing single subject designs) were identified between the years of 1975 and 2003. The majority of these studies were on the acquisition of functional sight words or picture identification, both of which the reviewers classified under the category of vocabul ary. Thirty four percent of the studies

PAGE 49

33 included a measure of comprehension and 28% contained a measure of fluency. Very few (10%) of the studies evaluated phonics or phonemic awareness (4%). Conclusions on Effective Instruction Syntheses reports on ef fective reading instruction (Beginning to Reading: Thinking and Learning about Print [Adams, 1990], Preventing Reading Difficulties in Young Children [Snow, Burns, & Griffin, 1998], Report of the National Reading Panel [NICHHD]) have changed the landscape of reading instruction for the majority of students in schools as evidenced by the current emphasis on scientifically based instruction and on prevention and early systematic intervention (IDEA, 2004 PL 108 446; NCLB, 2001 PL 107 110). The synthesis repor ts identified above have focused on the research and best practices for students without disabilities and students with mild disabilities. Comparatively, very little quality research exists that identifies the best practices in reading instruction for stu dents with moderate to severe disabilities (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine, 2006). The following is a review of the existing research on reading instruction for students with moderate to severe disabilities. This section is divi ded into (a) phonological awareness, (b) sight words, (c) phonics, (d) fluency, (e) comprehension, and (f) a balanced approach to literacy. Within each section, the available research is reviewed, expert option is cited, and where possible, conclusions ar e drawn regarding best practice instruction for students with moderate to severe disabilities. Phonological awareness Phonological awareness is the ability to hear and manipulate the individual sounds, or phonemes, in spoken words (NRP, 2000). Accordin g to the National Reading Panel (2000), phonological awareness is an important

PAGE 50

34 skill in learning to read and phonological awareness training should be included as part of a literacy curriculum. Overall there is little research upon which to draw conclusi ons regarding the best practices in the area of teaching phonological awareness to students with moderate to severe disabilities (Browder, Wakeman, Spooner, Ahlgrim Delzell, and Algozzine, 2006). Houston, Al Otaiba, and Torgesen (2006) reviewed the resear ch on teaching phonological awareness to students with mild disabilities and recommended that instruction in phonological awareness for students with significant disabilities proceed similarly to recommended practices for phonological awareness for all stu dents. Such instruction might include clapping the words in sentences and the syllables in words, playing rhyming games, practice identifying the sounds in words, and manipulating the sounds in words. Houston, Al Otaiba, and Torgesen (2006) note that it is important to consider that for students with moderate to severe disabilities, phonological awareness skills may develop more slowly than they do for their non disabled peers and that these students may require continued practice in phonological awarene ss skills as they progress from the pre reading stage of reading development to the learning to read stage of development. This continued training includes instruction in more advanced phonological awareness skills such as phoneme blending, phoneme substi tution and phoneme deletion. Citing a study conducted by Cossu, Rossini, and Marshall (1993), the authors warned that some students with moderate to severe disabilities may be able to demonstrate higher level reading skills without having mastered the iso lated skills that make up phonological

PAGE 51

35 awareness. This evidence is a reminder that little research has been conducted on phonological awareness in students with moderate to severe disabilities. Sight words. Sight word instruction has historically been t he primary method for teaching reading to students with moderate to severe disabilities (Houston, Al Otaiba, & Torgesen, 2006). Reviews of research have generally found sight word instruction to be effective for teaching students with significant disabil ities to read words in isolation (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine, 2006; Browder & Xin, 1998; Connors, 1992). However, the generalizability and functional utility of sight word instruction has been called into question (Browder, Wa keman, Spooner, Ahlgrim Delzell, & Algozzine, 2006). In order to determine the effectiveness of sight word instruction for students with moderate to severe disabilities, Browder and Xin (1998) conducted a meta analysis of the existing research in this are a. A review of the literature from 1980 to 1997 identified 48 studies that met identified criteria for inclusion in the meta analysis. The authors used the percentage of non overlapping data points (PND) to determine the effectiveness of the intervention This non parametric approach to meta analysis was necessary because 46 of the 48 studies included were of a single subject design. Of the 48 studies, 32 included enough information to be able to calculate PND. These were the studies that were included in the analysis. The data suggested that the sight word interventions were very effective overall, with a median PND of 91. The PND was higher for students identified as having mild disabilities (median PND of 95) than for students identified as having s evere disabilities

PAGE 52

36 (median PND if 89). However, the PND scores suggest a strong treatment effect for sight word interventions for both groups of students. The authors (Browder & Xin, 1998) conducted an analysis of the data and identified which specific intervention variables were most effective in sight word instruction. Included in this analysis was (a) the type of prompting (pre response or post response, time delay or other; errorless learning or not), (b) the number of words included in the training set, (c) the type of reinforcement used (verbal praise or praise with tangibles), (d) the type of error correction (with or without having the student repeat the word correctly), (e) the format for instruction (group or individual), (f) the person deliver ing the instruction (teacher or peer), and (g) whether or not application activities for functional use were included. The authors first analyzed the components that were significant factors for instruction for all students included in the meta analysis. One of the instructional variables that was identified as having a significant impact on the effectiveness of sight word instruction was the opportunity to have students repeat the words after a correction was provided. The second instructional strategy that had a significant impact on effectiveness was the use of post response prompting (i.e., the student is presented a word and identifies it incorrectly and is then given the correct response and possibly asked to repeat the correct word). Next, the au thors analyzed the results specifically for students with moderate to severe disabilities. The use of post response prompting was again identified as having a significant impact on instruction, as was the inclusion of an application activity which encoura ged functional use of the sight words. There was no significant impact for any of the other instructional factors.

PAGE 53

37 Given the abundance of research that supports pre response procedures such as time delay, the positive effect of post response prompting w as a surprising finding for Browder and Xin (1998). When providing instruction using time delay, teachers present sight words to the student while either providing the name of the word simultaneously or after waiting a specified period of time (e.g., thre e seconds). This instructional format is sometimes referred to as errorless learning because students have very few opportunities to make mistakes. This differs from post response feedback in which feedback is provided only after the student has responded which allows students to make many more incorrect responses. According to Browder and Xin (1998 ), there are several reasons that post response prompting was found to have a significant impact on instr uctional outcomes in their meta analysis, and why pr e response prompting was not. First, the authors suggested that, because the research was limited to studies in peer reviewed journals and did not include dissertations and other sources of research, the meta analysis may have been skewed with research th at found post response feedback to be effective. Only studies with positive results are typically published in peer reviewed journals. The alternative hypothesis is that post response prompting is simply more effective than previously thought (Browder & Xin, 1998). Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine (2006) provided a more comprehensive review of the research on reading instruction for students with moderate to severe disabilities in order to determine evidence based practices in sev eral areas. Their review was different from prior reviews of the research of literacy instruction for students with significant disabilities (Browder & Xin, 1998; Conners,

PAGE 54

38 1992) in several ways. First, the authors of this research synthesis used the fram ework for reading instruction posed by the National Reading Panel, and summarized the research using the five big ideas of reading: phonological awareness, phonics, fluency, vocabulary, and comprehension (NRP, 2000). Second, the reviewers separated out re search specific to students with severe disabilities from students with moderate disabilities, as well as the research specific to augmentative and assistive communication (AAC). In their review, Browder, et al. (2006) found strong support for sight word i nstruction. When using the percentage of non overlapping data points (PND) to determine the effect size of single subject designs, the reviewers found that studies related to sight words had an overall PND of 85 percent (41 studies). Few of the group stu across the board. Based on all of the information that was collected, the authors concluded that the only instructional practice with enough support to be deemed an While both Browder and Xin (1998) and Bro wder, et al. (2006) found sight word instruction to be generally effective for students with moderate to severe disabilities, there have been questions raised about both the generalizability and the utility of sight word instruction. According to Browder and Xin (1998), few of the studies of sight word instruction determined whether students were able to generalize the recognition of the target word to real materials or locations (stimulus generalization), and even fewer assess

PAGE 55

39 of the target word (response generalization). This raises the question of educational utility for sight word instruction. Further questioning the utility of sight word instruction is the inherent limitation in reading achievement using a strictly sight w ord instruction approach. Without strategies for approaching unfamiliar or untaught words, students are limited in what they will be able to read across their lifetime. Overall, the research on sight word instruction for students with moderate to severe disabilities is strong and suggests that sight word instruction is an effective reading strategy for this population of students. Sight word instruction should proceed using systematic prompting (either pre or post prompting) and be presented in massed trial format (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine, 2006). Because of concerns with generalization, sight word instruction should include planned and systematic generalization training words in multiple contexts. Finally, the results o f this research synthesis suggest that sight word instruction should be only one component provide students with strategies for identifying words that have not been direct ly taught. Phonics The National Reading Panel identified phonics instruction as an essential component to teaching students how to read (NRP, 2000). From their meta analysis of the existing research, the P anel concluded that direct and systematic ins truction in phonics is a scientifically supported practice for teaching students to learn excluded students with intelligence scores below 70. So the question remains, is direct and systematic phonics instruction an effective teaching strategy for students with

PAGE 56

40 moderate to severe disabilities? There is very little research to determine the answer to this question (Connors, 1992; Joseph & Seery, 2004), but evidence exists t hat this population has the capacity to learn phonetic analysis skills (Bradford, Alberto, Houchins, Shippen, Flores, 2006; Flores, Shippen, Alberto, & Crowe, 2004). Compared to the number of studies that have been conducted on the use of sight word instru ction for students with moderate to severe disabilities, the number of studies that have measured the impact of phonics instruction with this same population is almost non existent. Connors (1992) identified seven studies that measured the phonetic analys is skills of students with moderate mental retardation. Based on his research, Connors (1992) concluded that students with moderate mental retardation can learn phonetic analysis skills. More specifically, both isolated strategies such as stimulus prompt fading only, and more comprehensive instructional programs such as DISTAR, To update the Connors (1992) research, Joseph and Seery (2004) reviewed the literature from 1990 to 2002 order to identify studies that measured the phonics skills of students with mental retardation. They found a similar number of studies from that time period (seven) that measured phonetic analysis as an outcome for students with mild or moder ate mental retardation. Because of the limited numbers of students included in the studies, and because of the design of the studies included in the analysis, the results of the analysis must be interpreted cautiously. However, the studies, taken togethe r, suggested that students with mild to moderate mental retardation can learn phonics skills such as letter sound correspondence and word analysis. Interestingly, none of the seven studies provided direct and systematic phonics instruction to the students as the intervention

PAGE 57

41 (Joseph & Seery, 2004). Some of the interventions identified in the studies included error correction procedures, use of rime and word analogy cues, phonics instruction embedded within the Four Blocks Literacy Program, and computer as sisted instruction. Consequently, Joseph and Seery (2004) were unable to conclude whether students with moderate to severe disabilities benefit from direct and systematic phonics instruction the same way their typically developing peers do. Two studies since the publication of the Joeseph and Seery (2004) review of the literature have determined the impact of direct and systematic phonics instruction on the phonics skills of students with moderate disabilities (Bradford, Alberto, Houchins, Shippen, Flore s, 2006; Flores, Shippen, Alberto, & Crowe, 2004). Both studies examined the effectiveness of a Direct Instruction program, Corrective Reading (Engelmann, et al., 1999) for teaching decoding skills to students with moderate intellectual disabilities. Th e sample in the Flores, Shippen, Alberto, & Crowe (2004) study included six elementary aged students with cognitive ability (IQ) scores ranging from 38 to 52 who were placed in a self contained special education classroom. Prior to beginning the Correctiv e Reading Program literacy instruction in the classroom consisted primarily of functional sight words. The Corrective Reading Program was selected over other reading programs because of its focus on decoding skills only. Some modifications were made to the program in order to meet the needs of this particular group of students. For example, in the standard implementation of the program, the first letter sound taught is / a /. However, this group of students had learned the letter a as a sight word. In o rder to avoid confusion, the letter sound / m / was taught first instead.

PAGE 58

42 The study employed multiple conditions with a specific criterion for phase changes. More specifically, the conditions for the study were (a) identification of the letter sounds / m / and / a /,( b) identification of the letter sounds / s / and / t /, and (c) decoding consonant vowel consonant (CVC) words. The criterion for success before a phase change was made was three consecutive skill probes of 100% accuracy. The independent variable for the study was systematic and explicit instruction in letter/sound correspondence, identifying the individual phonemes in a CVC word (a.k.a., saying the word slowly) and telescoping the CVC word (a.k.a., saying the word fast). The first goal of the s tudy was to teach students letter/sound correspondence. All of the students in the study were able reach the criterion for success for the letter sounds / m / and / a /. There was one student who was not able to reach the criterion for the letter sounds / s / and / t / due to language and articulation difficulties. (This student did not complete the remainder of the intervention.) The number of trials required before students met the criterion decreased as they learned more letter sounds, suggesting that the stu dents generalized the learning of letter/sound correspondences to new letters. With regard to blending and telescoping the CVC words that were made up of the letter sounds that were taught (i.e., sam, mat), the remaining five students all met the criteri on level of performance. However, when given new words that were still made up of the letter sounds that had been taught, but were words that the students had not been directly instructed in (i.e., mas, sat), only one of the students was able to meet the criterion level of performance for telescoping the word. The results of this study suggested that students with very low cognitive scores can be taught to identify letter sounds and to blend letter sounds into real words.

PAGE 59

43 Importantly, the IQ scores of the students did not predict their ability to perform the targeted reading skills. Furthermore, students who were originally excluded from the study because of issues such as behavioral concerns and selective mutism, but who were later included, were able to participate in the program and they, too, demonstrated mastery of most of the targeted skills. Thus, student characteristics should not be the primary determinant of whether or not to attempt to teach the student decoding skills. The Bradford, Alberto Houchins, Shippen, & Flores (2006) study extended the research on the use of a Direct Instruction program to teach decoding skills to students with moderate intellectual disabilities using a population of middle school students. The sample included thre e male students with cognitive ability (IQ) scores ranging from 46 to 55 who were being educated in a self contained special education classroom. The authors of this study utilized the Corrective Reading Program: Decoding A (Engelmann, et al, 1999) as th e decoding intervention. In contrast to the Flores, Shippen, Alberto, & Crowe (2004) study, few modifications were made to the program (because they were not found to be necessary), and the program was implemented all the way through completion of Decodin g A (65 lessons total over a 6 month period). According to the authors of the study, at the end of Decoding A sound correspondence, (b) sounding out words, (c) blending sounds into real words, (d) decoding irregu larly spelled words, (e) reading sentences, and (f) reading short passages Flores, 2006, p. 338). Criterion referenced mastery tests that are given as part of the Corrective Reading Program were used as one indicator of student success in attaining these skills. Accuracy

PAGE 60

44 on these measures, which required students to orally identify letter sounds, and produce the letter sounds through writing, and read a series of word out lou d, ranged from 97 to 99 percent, suggesting that students demonstrated mastery of these skills throughout the program. Measures of oral reading fluency and accuracy rate were also included as an outcome measure. These fluency measures are part of the Cor rective Reading Program criterion for mastery for accuracy but none of them met the criterion for fluency. However, when the Corrective Reading placement test was re admin istered, it was clear that students demonstrated increases in their oral reading fluency rate. Before beginning the program, none of the students could read the connected text on the placement test. Nine weeks after the intervention ended, the oral readi ng fluency rates of the students ranged from 15 words correct per minute to 46 words correct per minute on the placement test passages, which were approximately at the second grade level. Taken together, these results suggested that students can effective ly learn to decode words and read connected text using a systematic and explicit instructional decoding program. had an impact on their functional word reading (Bradford, Alberto, Houchins, Shippen, & Flores, 2006). Prior to beginning the Corrective Reading Program all three students had received sight word based literacy instruction either through the Edmark Reading Program (1992) or using Dolch (1955) high frequency wor ds. Pre and post test data of the reading of unknown functional sight words. This finding emphasizes the importance

PAGE 61

45 of learning to read using decoding strategies and because of the life skill application it offers. Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine (2006) reviewed the research to identify studies that targeted phonics instruction for students with moderate to severe disabilities. As was the ca se in earlier reviews (Conners, 1992; Joseph & Seery, 2004), few studies were found. However, the authors noted that of the few studies that were able to be coded for effect size using PND (three total), the effect sizes were strong (PND of 93%) and in fa vor of phonics instruction for students with moderate to severe disabilities. While additional research is clearly needed in this area (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine, 2006; Joeseph & Seery, 2004), Houston, Al Otaiba, & Torgesen (2006) recommended that students with moderate to severe disabilities should be given an opportunity to learn decoding skills through systematic and direct instruction, and that this instruction should allow for additional opportunities for practice and fo r maintenance activities of the skills that have already been acquired. Fluency Fluency is the ability to automatically and accurately identify words either in isolation or in connected text (NRP, 2000). When students are able to read fluently, they a re able to devote their energies to comprehending the text that they are reading and comprehension is the ultimate goal of reading instruction (NRP, 2000). Research has found that reading instruction that includes repeated reading interventions, or interv entions that target reading fluency, result in improved reading rate, accuracy, and comprehension for elementary students with learning disabilities (Chard, Vaughn, & Tyler, 2002).

PAGE 62

46 In their review of the literature on effective reading instruction for st udents with moderate to severe disabilities, Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine (2006) found that most of the existing research studies that addressed reading fluency measured fluency using error rate instead of counting the words read correctly per minute. In addition, the number of studies that measured fluency was small (20), and very rarely was fluency directly taught in any of these studies. Thus, little is known about the impact of fluency instruction for students with moderate to severe disabilities. Research exists that suggests that Direct Instruction in phonics can improve the fluency rate of students with moderate disabilities. Bradford, Alberto, Houchins, Shippen, & Flores (2006) provided decoding instruction to a group o f three male middle school students, all identified as having moderate mental impairment and all of whom were educated in a self contained special education classroom. The authors used the Corrective Reading Program, Decoding A as the instructional interv ention. The primary ability to read connected text with fluency. The place ment test of the Corrective Reading Program was given to all three participants in the study prior to beginning the program. However, none of the participants was able to complete the placement test because it required the students to read connected text at approximately the second grade level. Nine weeks following the completion of the program, the students were re administered the placement test. All three of the students were able to complete the placement test, and their oral reading fluency rate on the passage ranged from 15 to 46 words correct per minute. However, it is important to note that the students had difficulty meeting the

PAGE 63

47 mastery criterion for fluency when administered the reading passages that were embedded within the Corrective Reading Program There exists a need for researchers to determine how fluency is related to the oral reading of this population of students. Many students with moderate to severe disabilities have processing and articulation difficulties (Orelove & Sobsey, 199 6) that could potentially impact their ability to read fluently at a rate that is commensurate with th eir typically developing peers. Despite these potential differences and barriers, Houston, Al Otaiba, and Torgesen (2006) recommended the inclusion of fl uency practice of both words in isolation and connected text in the literacy instruction of students with moderate to severe disabilities. Comprehension Reading comprehension is defined as the ability to read with understanding and to communicate with others what has been read (NRP, 2000). The ability to successfully read connected text and to comprehend what is read requires several pre skills, including the ability to read connected text. Hasbrouck and Tindal (1992) found students must read basic wo rds fluently at a rate of about 100 words per minute and be able to decode one and two syllable words with accuracy in order to comprehend the text that they are reading and thus enter into the reading to learn stage of reading development. The question i s whether this is the case for students with moderate to severe disabilities. Reading comprehension is also strongly related to language development. Students need to be able to both identify and decode a word and understand the meaning of that word in o rder to make meaning of the text being read (Houston, Al Otaiba, & Torgesen, 2006). Students with moderate to severe disabilities often have deficits in expressive and receptive language, which has the potential to

PAGE 64

48 impact their ability to fully gain meani ng from text. Despite possible deficits in reading fluency and expressive and receptive language, the existing research on reading comprehension with respect to students with moderate to severe disabilities is promising (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine, 2006; Koppenhaver, Spadorcia, & Harrison, 1998). The research on reading comprehension for students with significant disabilities is sparse compared to the research on areas such as sight word recognition (Koppenhaver, Spadorcia, & H arrison, 1998). However, Koppenhaver, Spadorcia, & Harrison (1998) sought to provide a narrative synthesis of the existing research in this area, and draw some initial conclusions regarding what is needed to effectively teach students to read with compreh ension. Because of the small number of studies and lack of replication of results, the conclusions that the authors drew regarding student reading comprehension and effective instruction is tentative. Based on their review of the research, Koppenhaver, Spadorcia, & Harrison (1998) identified three areas that educators could address in order to improve reading comprehension in students with m oderate to severe disabilities: (a) environments and expectations in early childhood, (b) classroom learning oppor tunity, and (c) specific instructional and intervention strategies. The first recommendation is to address the learning opportunities that take place at the early childhood level at home and in schools. As has been described previously, data consistently suggest that children with developmental disabilities have fewer opportunities to be exposed to and interact with books and other forms of literacy before entering kindergarten (Fitzgerald, Roberts, Pierce, & Schuele, 1995; Light & Kelford Smith, 1993). This lack of exposure may be

PAGE 65

49 due to a variety of reasons from parental and educational focus on other areas, such as health and lack of parental and teacher expectations for students with moderate to severe disabilities to read and have literate lives (Gur ry & Larkin, 1999). Because successful comprehension of speech and text is dependent upon background knowledge and vocabulary, it is imperative that students have these early literacy experiences that foster language and vocabulary and help children to le arn more about the world around them (Houston, Al Otiaba, & Torgesen, 2006). According to Koppenhaver, Spadorcia, and Harrison (1998), students continue to have a lack of learning opportunity with regard to reading comprehension when they enter school. A review of the literature conducted by the authors suggested that very little instructional time is devoted to reading for this population of students, and of the reading instructional time that is available, only a small percentage of that time is devote d to reading comprehension (Koppenhaver, Spadorcia, & Harrison, 1998). In their research, the authors found that students had extremely limited opportunities to read texts of a paragraph or longer, but had more opportunities to listen to teachers read tex ts of the same length out loud. Furthermore, most of the reading instructional time was spent doing worksheets and individual word study. When reading comprehension was a distinct component of the classroom instruction, rarely did these lessons conform t o what is known about effective comprehension instruction (Koppenhaver, Spadorcia, & Harrison, 1998). After reviewing the literature on specific instructional and intervention strategies for teaching reading comprehension to students with low incidence di sabilities, Koppenhaver, Spadorcia & Harrison (1998) found little information that would constitute

PAGE 66

50 best practice. While individual studies were able to identify practices that taught children to read with greater comprehension (e.g., questioning techniqu es, class wide peer tutoring, a balanced literacy program), none of the studies included more than three participants. Nor were they replicated. However, the studies did indicate that expecting students with moderate to severe disabilities to be able to demonstrate some level of reading compression is a feasible expectation, and that additional research should be conducted in order to determine the most effective strategies for achieving successful reading comprehension. The impact of a systematic, long term approach to teaching reading comprehension also needs to be examined. In a review of the literature almost ten years later, Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine (2006) identified 23 total studies that taught and/or measured readi ng comprehension, but only half of the studies met the criteria for quality. However, from those 11 studies that were of high quality, the authors were able to draw some conclusions regarding effective practice in teaching reading comprehension to student s with moderate to severe disabilities. The practices that were found to be most effective with this population were the reinforcement of comprehension skills in the context of a functional activity, making the instruction concrete through the use of pictu res, and providing massed trial training with systematic prompting and fading to teach students the correct response to comprehension questions. However, the authors note that all of the studies focused only on the question and answer aspect of reading co mprehension, and that additional research is needed on the effectiveness of teaching additional comprehension strategies (e.g., graphic organizers, summarizing, etc.) to this population of students.

PAGE 67

51 While there may be some initial indication of which ins tructional strategies are effective at improving the reading comprehension skills of students with moderate to severe disabilities (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine, 2006), there exists a need to conduct additional research in order to determine a long term and integrated approach to reading comprehension that will be effective for students with the most significant disabilities. A balanced approach The research presented thus far on effective reading instruction for students with moderate to severe disabilities has addressed effective strategies for individual skills (e.g., sight words, comprehension). The evidence presented suggests that students with moderate to severe disabilities are able to acquire these individual skills wh en appropriate instruction is provided. However, in practice, it is recommended that students not be taught individual skills, but rather be instructed using comprehensive instruction and have exposure to a balanced literacy program (Houston, Al Otaiba, & Torgesen, 2006). The following research suggests that a comprehensive approach to literacy for students with significant needs can be effective at improving a variety of student literacy skills (Al Otaiba & Hosp, 2004; Hedrick, Katims, & Carr, 1999). A l Otaiba and Hosp (2004) sought to determine whether a comprehensive instructional program that contained all five components identified by the National Reading Panel (phonological awareness, phonics, fluency, vocabulary, and comprehension; NRP, 2000) was effective at improving the reading skills of four students with Down syndrome. Two of the students were at the middle school level, and the other two students were in the first grade. The four targeted students all participated in a 10

PAGE 68

52 week reading tutor ing program that was run through the University Reading Clinic. The tutors were all pre service teachers seeking experience with teaching reading. All were provided with extensive training before implementing the intervention and all received ongoing sup ervision during implementation. The interventions were individualized based on pre intervention assessments of identified by the NRP. A systematic and carefully se quenced published program was used to instruct the students in phonological awareness, phonics, and reading high frequency sight words. Vocabulary and comprehension were taught through reading stories and answering questions about the main idea or general story concepts. Fluency was emphasized through a sight word reading game, in which students were rewarded for rapid identification of the words on flash cards. Student outcomes were evaluated on a weekly basis using curriculum based measurement (CBM) pr obes. The skills targeted on intervention skill level. Possibilities included letter sound probes, sight word probes, or probes that required students to read connected text. The effectiveness of the tutor ing program was evaluated using both pre and post standardized measures of reading achievement as well as weekly progress on CBM probes. All students demonstrated some growth in reading skills. However, the skills in which they demonstrated growth varied as did the amount of growth made by each individual. When the Word Identification and Word Attack components of the Woodcock Reading Mastery Test Revised (WRMT R) were re administered, one student received the same score on both administrations. In contrast, all three of the other

PAGE 69

53 students made gains; with the largest gains being 10 standard score points for Word Identification and 36 standard score points for Word Attack. The CBM results suggested that it was a sensitive measure of student growth as all students demonstrated progress on these probes. Of those students who were assessed using letter sound probes, the rate of growth (slope) ranged from .78 to .82. For those students who were assessed using the sight word probes, the rate of growth ranged from .32 to 1.56. Finally, of the students who were given oral reading fluency probes, the rate of growth varied from .49 to .54. As was found in prior studies, student characteristics before intervention were not good predictors of student respo nse to intervention. Hedrick, Katims, and Carr (1999) evaluated the effectiveness of an even more comprehensive and balanced year long literacy program for students with mild to moderate mental retardation using the Four Block model of literacy instructi on as the model program (Cunningham, Cunningham, & Allington, 2002). The Four Block model is a framework for teaching reading and writing that includes four primary components, or blocks: self selected reading block, guided reading block, working with wor d block, and the writing block. Within these instructional blocks, children receive instruction in phonological awareness, phonics, fluency, vocabulary, comprehension, spelling, and writing. The framework is designed for use in general education classroo ms, and the research conducted by Hedrick, Katims, and Carr (1999) was the first time this framework was used with students with mild to moderate disabilities. Nine students participated in the year long study of the impact of the Four Block model on thei r literacy growth. The mean age of the students was 9 years and 8 months, and their cognitive scores ranged from 40 to 76 (excluding the scores of two students

PAGE 70

54 whose scores were unattainable). The students all were educated in a self contained classroom in which instruction was primarily life skill based. The Four Block model was implemented every day, with 45 minutes devoted to each of the four blocks. Some modifications to the model were made based on the needs of the students. For example, prior to t he implementation of the Four Block model in their classroom, most of the students had little experience with self selecting literature and reading independently. Similarly, they had very few opportunities to write and express their thoughts on paper. Con sequently, both of these components of the model were much more structured and supported at the beginning of the school year, but started to approximate more typical implementation toward the end of the school year. The outcome measures used to determine the effectiveness of the intervention included both individually administered formal assessments and informal assessments that were selected to better reflect their day to day progress. On a measure of concepts of print, all students made gains demonstrati ng an increased understanding of the general conventions of written English. When asked to retell the components of a story that was read to them (auditory comprehension), student performance increased markedly, with many of the students scoring 10 out of writing samples were analyzed throughout the school year to determine progress. All but one of the students made gains in invented spelling and/or conventional spelling. This progress can reflect a growth in their phonics skills as well as their writing skills. The Brigance Diagnostic Comprehensive Inventory of Basic Skills was used to assess progress in reading words in isolation. All students made gains on this assessment, and increases ranged from 10 to 4 5 percent from pre test to post test. Finally, an informal

PAGE 71

55 well as to read a passage out loud and answer questions. Students did not show as much growth on this asse ssment tool. Only three students made progress in answering comprehension questions on a passage that had been read aloud. However, a qualitative more phonics skills an d context cues when attempting to identify the words on the post test. Both the Al Otaiba and Hosp (2004) and the Hedrick, Katims, and Carr (1999) studies demonstrated that, when provided a comprehensive literacy program, students with mild to severe dis abilities can demonstrate an ability to acquire reading and writing skills. Both authors concluded that all students with moderate to severe disabilities should be provided with the opportunity to participate in a comprehensive and balanced literacy progr am that targets all five big ideas of reading (Al Otaiba & Hosp, 2004; Hedrick, Katims, & Carr, 1999). Unfortunately, most teachers who are instructing this population of students are not providing comprehensive instruction. Research suggests that readin g instruction for students with moderate to severe disabilities is limited to drill and practice in functional sight words (Katims, 2000). Summary Over the past twenty years, a tremendous amount of work has gone into conducting new research and synthes izing the existing research on effective reading instruction for typically developing students and students with mild disabilities. This research has influenced federal legislation governing general and special education and had a significant impact on ho w schools provide literacy instruction. Comparatively,

PAGE 72

56 very little research has been conducted on effective reading instruction for students with moderate to severe disabilities (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine, 2006). Of the rese arch that has been conducted, most has been in the area of sight word instruction (Browder & Xin, 1998). This finding is in line with current instructional practices for this population of students (Katims, 2000). Research is needed to determine the long term impact of high quality and comprehensive literacy instruction, including phonemic awareness and phonics instruction, on the reaching achievement of students with moderate to severe disabilities (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine 2006). Of the conclusions that can be drawn based on the research that does exist on literacy instruction for students with significant disabilities, some are similar to what research has identified as best practice for students without disabilities and some are different. Of the similarities, research suggests that all students benefit from comprehensive instructional programming including the five areas of instruction identified by the National Reading Panel (phonological awareness, phonics, fluenc y, vocabulary, and comprehension). This instruction should be direct and systematic and make use of data to inform instructional decisions. Ways in which research suggests literacy instruction should be different for students with moderate to severe disa bilities include working from a broader definition of literacy, considering differences in the early life experiences of students with disabilities and thinking about the impact on classroom instruction, honoring differing abilities in expressive and recep tive communication skills, and incorporating the use of AAC when appropriate.

PAGE 73

57 While additional research is needed in the area of reading instruction for students with moderate to severe disabilities, it is safe to say that it is time for teachers to mov e beyond sight word instruction and toward a more comprehensive reading program that is aligned with what is know about effective reading instruction. Unfortunately, this shift is going to require significant change on behalf of the schools and teachers t hat are responsible for the education of these students. Systems change is a long and difficult process but research exists which identifies the strategies that increase the likelihood of a successful systems change effort. The following section, Systems Change will review the literature in the area of systems change and identify those factors that lead to effective systems change. Systems Change There is a large body of research that has identified best practices in literacy instruction, and current le gislation (e.g., Individuals with Disabilities Education Improvement Act, 2004; No Child Left Behind Act, 2002) requires schools to use these research based practices. However, very little research has focused on the application of best practices literacy instruction to students with moderate to severe disabilities, and current practice reflects this (Katims, 2000). Improving literacy instruction for students with moderate to severe disabilities is going to require significant change by school systems and by the teachers who educate this population of students. The fact that the teachers and staff will be going through a systems change process must be addressed if implementation of the research based practices is going to be successful. As stated by Curt

PAGE 74

58 innovative practice and systems change are essential if we are to be successful in bringing The following discussion of systems cha nge will 1) provide a definition of systems change, 2) identify potential barriers to effective systems change, 3) outline the components of effective systems change, and 4) relate the systems change research to increasing the use of research based practic es in the area of literacy for students with moderate to severe disabilities. Systems Change Defined Systems change (also referred to as educational change, educational reform, or school change) does not refer to a particular innovation, but rather to th e process that schools must successfully go through in order to implement an innovation (Hall & Hord, or substance of the change effort, which can be anything from ado pting a new writing program to a total redesign of the entire organizational structure of a system (Hall & Hord, 2005). It is not a particular innovation that will be the topic of discussion in this section; rather, it is the process of systems change for the purpose of implementing any innovation. Researchers who have written about school change have conceptualized this process in several different ways. More specifically, authors have identified variations in the size or magnitude of the change (e.g., systemic vs. small scale, incremental vs. fundamental) as well as differences in the direction of the change (bottom up or top down). Systems change can vary greatly with regard to the magnitude of the innovation or change. Hall and Hord (2005) classified the size of an innovation based on the amount

PAGE 75

59 of effort required for implementation, and on the amount of change required in people and systems for implementation. An example of a small innovation would be the adoption of a new curriculum in a single cla ssroom. An example of a large scale, or systemic change would be a re conceptualization and re organization and of a school scale innovations are typically complex and require significant changes in the roles of sta ff members, take a number of years to implement, and are supported with ongoing professional development and consultation (Hall & Hord, 2005). Cuban (1996) also described differences in the magnitude of systems change and specifically identified two typ es of change, incremental and fundamental. Incremental changes are those that modify or improve the existing framework or structure of the school. Cuban (1996) compared incremental changes to an old car that gets fixed so that it operates better, but the underlying structure and operating system of the car remains the same. Fundamental changes are those that replace or permanently transform the underlying structure or framework of the operating system. This is analogous to buying a whole new car because the old one is beyond repair, or because the owner simply wants a car that functions in significantly different ways. Differences in the magnitude of the innovation have clear implications with regard to the amount of time, resources, and effort required to implement a change (Hall & Hord, 2005). Another way in which change efforts may differ is in the direction of the change (Fullan, 1997). An example of top down change would be change mandated through legislation enacted at the state or federal level. Top down changes could also include decisions that are made at the school district level that are expected to be implemented in

PAGE 76

60 every school and classroom. In contrast, bottom up changes are those that begin at the classroom level, expand through other c lassrooms, and work their way through the system until the practice is implemented throughout the district, or even becomes policy. An example of bottom up change would be a single teacher implementing a new instructional strategy (e.g., cooperative learn ing) in order to improve educational outcomes. When the teacher finds success with this strategy, he or she motivates other teachers to try the new strategy, and before long the new strategy becomes standard practice in the school. Hall and Hord (2005) n ote that history suggests that in isolation, neither bottom up nor top down change has enjoyed a lot of success and instead same vision and are working together to imp lem ent that vision. In a horizontal change effort, all of the players are viewed as being on the same plane, and are thus made partners in the change process. Barriers to Effective Systems Change Schools are in a constant state of change, trying to find new ways to improve outcomes for students (Cuban, 1996). Opinions vary greatly regarding whether schools are effective at implementing change and affecting student outcomes. Those who are of the opinion that schools are ineffective or incapable of imple menting and sustaining innovation cite numerous programs, instructional strategies and theoretical approaches that appear to have come and gone throughout the years (Cuban, 1996). Undoubtedly, there are many educational reform efforts that have not been s uccessful at improving outcomes for students and/or standing the test of time. How can those seeking to

PAGE 77

61 improve student outcomes and to change school functioning avoid becoming just another Researchers agree that careful consideratio n of the school context in which the change is to take place is an essential component of effective change (Curtis, Castillo, & Cohen, 2008; Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). The school context is important because the various components t hat make up the school can serve as either a bridge to successful reform or as a barrier to that reform (Boyd, 1992b). Boyd (1992b) warned that if the barriers are left unrecognized in the process of school change, they can lead to the failure of the chan g e effort. Boyd (1992b) s uggested that barriers to change School Ecology Barriers The school ecology is made of up of the physical, material, inorganic aspects of sc hools (Boyd, 1992b). Boyd described how these aspects of schools could serve as either barriers or bridges to successful school change. More specifically, Boyd suggested that the availability of resources is an important consideration, given that most in novations require significant resources and time to implement. In addition, the factors can come into play, and can potentially become barriers to implementation, as w ell. For example, a school that is inefficiently organized, overcrowded, and unsafe can cause stress among all involved, including administrators, staff, parents and students. These factors also can contribute negatively to staff and student communicatio n and morale. Other ecological factors identified by Boyd were working conditions and the policies that

PAGE 78

62 sense of professionalism and positive attitudes, thus im pacting the facilitation of change in the school. Cooper (1998) identified several school ecology variables that were significantly related to the quality of implementation of the Success for All program, a comprehensive school reform initiative that targ ets schools with a high percentage of at risk students. With data from more than 350 schools, Cooper found that student mobility, student attendance rates, and the racial make up of the student body all served as either facilitators or barriers to impleme ntation of the program. More specifically, Cooper found that schools that were rated as having high levels of implementation of the Success for All program were also likely to have lower student mobility, higher student attendance rates, and a higher perc entage of white students. This study illustrates the impact of ecological factors on the implementation of a specific school reform initiative. Evans (2001) identified several traits related to school ecology as contributing to implementation of an innovation when there is not enough money to support (a) training, (b) consultative support and (c) transition management (i.e., the costs associated with coordinating the change effort). Such costs would be in addition to the cost of innovation related materials, programs, data management systems, etc. Separate from a more pressured the school, the more it may need reform and the less it may be able to undertake

PAGE 79

63 While the inorganic, ecological aspects of systems change are presented here separately from the human/cultural aspects, the two are inextricably intertwined Ecological factors have a significant impact on staff attitudes, beliefs, relati onships, and culture (Boyd, 1992b). Human/Cultural Barriers factors: the attitudes and beliefs of persons both inside the school and in the external environment, the cultural n orms of the school, and the relationships between persons in school reform efforts have failed because they have sought to restructure the school without providing suffic attitudes, beliefs, and inter relationships of school staff. Evans (2001) conceptualized human resistance to change as a normal and inevitable part of school reform. According to Evans, there is a difference between the way change is viewed by those initiating the change and those who are expected to that same change may feel a sense of loss; the change challenges their competence, creates confusion, and causes conflict. Consequentl y, there is a large gap between what the change means for its initiators and what it means for its targets. If this gap is not meaningfully addressed, the change effort will fail (Evans, 2001).

PAGE 80

64 Hall and Hord (2005) also recognized the influence of the individual and how she or he personally experiences; consequently, they developed techniques for assessing According to Hall and Hord, there is a predictable developmental p attern through which individuals move when they are asked to be a part of a change process. Those at the lowest level of concern, Awareness have little concern or involvement with the innovation. The next stage, Informational describes those who are in terested in simply gaining more information about the innovation. After gaining more information, people are likely to start thinking about how the innovation would personally impact them, such as how much added demand it would place on them. Such though ts are associated with the Personal Stage of Concern. People at the next level, Management are starting to implement the innovation, and are most concerned with the logistics of implementation. The final three stages, Consequence Collaboration and Ref ocusing describe those who are fairly skilled and fluent at implementing the innovation, and who are now looking for ways to improve the implementation and to assess how the innovation is impacting student outcomes. When engaging in change, it is importa nt to ascertain and acknowledge where your audience finds itself along the continuum of these stages of concern so that training and coaching can be tailored accordingly (Hall & Hord, 2005). In the study described above by Cooper (1998) relating to ecolo gical factors influencing adoption of the Success for All program in 350 schools, implementation was rated as high quality, medium quality, and low quality. The author found that several school culture variables also were significantly related to the qual ity of the implementation, including a supportive culture for institutional change, the breakdown of

PAGE 81

65 resistance, a commitment to implementing the program, and less concern among teachers for having an increased workload. Schools in which these cultural va riables were absent were found to have a lower quality implementation of the program. While variables related to school ecology and school culture can serve as barriers to the implementation of an innovation, it is important to remember that schools are what learn and adapt under the right conditions. An examination of the literature on systems change reveals several critical variables that are related to effect ive change. A strategic understanding of these variables and their importance to systems change efforts can be helpful in overcoming potential barriers to effective change. Components of Effective Systems Change Systems change in schools does not happe n overnight. It is a complicated and lengthy process that takes place over several years (Hall & Hord, 2005). Due to the complexity of the variables influencing change in schools (e.g., structural, community, historical, emotional, legislative, etc.), an exact recipe for successful change that will have written and studied system s change in order to identify components that will increase the probability for a successful systems change effort. Several themes relative to effective change emerge in the literature, and include: (a) planning for change, (b) developing a culture of cha nge, (c) facilitating/leading change, (d) staff development, and (e) sustaining change.

PAGE 82

66 Planning for Change Systems change, no matter the size or magnitude, requires changes in resources, changes in roles and responsibilities, and changes in belief s ystems. Therefore, the change process is not something that should be taken lightly, and must be planned for appropriately. The importance of planning for change has been consistently emphasized in the literature (Curtis & Stollar, 2002; Grimes & Tilly, 1996; Hall & Hord, 2005). In planning for change, there is no road map to follow. However, there are some important components involved in planning. These include assessing whether the system is ready for change, developing a shared vision, defining the innovation, and gaining buy in and support from key stakeholders. One important consideration in planning for change is whether the system is one that is going to be conducive to change. Boyd (1992a) identified 17 school context elements that facilitate and support school improvement efforts. Boyd and Hord (1994) categorized these elements into four broad categories. More specifically, schools that are ready for change have effectively (1) reduced isolation and fostered a sense of community, (2) increa sed staff capacity through access to resources and shared decision making, (3) provided a caring, productive environment where students, teachers, administrators, and parents work together as allies, and (4) promoted increased quality in instructional prac tice. Assessing the presence or absence of these four basic characteristics in a system prior to the introduction of an innovation can inform the implementation strategies used during the change process. Lehman, Greener, and Simpson (2002) recognized th e importance of assessing a

PAGE 83

67 (ORC) scale. The scale assesses systems in four areas, including motivational readiness (e.g., pressures for change, training needs), resources for change (e.g., works space, staffing), staff attributes, (e.g., adaptability, perceptions of growth), and organizational climate (e.g., clarity of mission and goals, staff cohesiveness, stress). This scale was developed specifically for use in assessi ng readiness for system change in substance abuse treatment facilities. It can, however, serve as an example of one way to identify the presence or absence of system variables that can either facilitate or hinder the change process and to generally assess whether a system is ready for change. Senge (1990) identified five factors, or disciplines, that organizations need to have in order to become a learning organization, or an organization that is continuously improving. One of the disciplines identifi everyone in the system. Shared visions are powerful in organizations because they foster commitment, and they encourage ex perimentation and innovation. It is important to establish a shared vision prior to engaging in systems change because it promotes system thinking, communicates the purpose of the change, and motivates change. Fullen (1997) agreed that it is important t o plan for change. However, he warned against identifying such a vision, or conducting strategic planning too early. According to Fullen, when engaging in systems change in complex and dynamic systems like schools, the creation of a vision requires signi ficant reflective thought and experience. Additionally, it is important that the vision is shared (though it does not have to be shared by all); he emphasized that the development of a shared vision does not happen overnight. Finally, experimentation wit h the innovation is recommended before spending

PAGE 84

68 time with strategic planning. Experience with the innovation will result in a more appropriate and effective plan in the end (Fullen, 1997). Before engaging in the implementation of an innovation, it is i mportant for everyone involved to have a clear understanding of what the innovation is. Hall and Hord (2005) recognized the gap between the creation of an innovation, including what the innovation is intended to look like and accomplish, and its actual im plementation in a particular classroom or school system with its own set of complex variables. In order to increase understanding and reduce the discrepancy between what is designed and what is implemented, Hall and Hord recommended the development of an Innovation Configuration (IC). An IC is a tool that is designed to descriptively communicate what the change is and, just as importantly, what it is not. More specifically, an IC includes all of the components of an innovation, and each component is desc ribed along a continuum from the ideal implementation intended by the developer to an unacceptable implementation. Between the two ends of the continuum are statements describing various levels of implementation that increasingly approximate the ideal. I used both to facilitate implementation of an innovation as well as to evaluate the quality of implementation later in the change process. Curtis, Castillo, and Cohen (2008) identified best practices in planning for change. Several activities were suggested, including beginning by conducting an analysis of the school system to identify its unique characteristics and needs. Other critical practices that should occur during the planning process are gaining the commitment of key personnel by iden tifying leaders and gatekeepers, soliciting the involvement of all stakeholders, and communicating what may be expected during the change process.

PAGE 85

69 Curtis, Castillo, and Cohen also recommend identifying the desired outcomes of the change process, developin g a strategy for implementation, and identifying an evaluation plan to judge the success of the change process. The final step in the planning process recommended by Curtis, Castillo, and Cohen is training the staff in a systematic planning and problem so lving process. Increasing the problem solving capacity of the individuals in the system will increase the capacity of the system to address new problems and challenges as they arise. Developing a Culture of Change When implementing change, the school culture can serve as either a barrier to implementation or a facilitator of implementation of the innovation (Boyd, 1992b; Evans, 2001; Hargreaves, 1997). As described by Fullen (1993), some of the most well intentioned innovations designed to bring about fundamental changes in instruction and practice have failed to bring about even minimal changes because the implementers In order to bring about meaningful change, it also is necessary to change the norms, habits, skills, and beliefs of those in the system (Fullen, 1993). To address potential barriers that can be created by school culture and to create a climate of change, organizations collectively need to bec ome and function as a learning organization. The first is Systems Thinking This is the cornerstone of the learning organization because it

PAGE 86

70 integrates all four of the other disciplines. Systems that are able to practice systems thinking are able to view their organization as a whole, in addition to recognizing its individual and interconnected parts. It is necessary to see an organization from the system perspective because it leads to more effective problem identification and problem solution. The nex t three disciplines, Mental Models Personal Mastery and Building Shared Vision all relate to identifying our individual assumptions and beliefs, developing fostering th e collective vision of the organization (discussed briefly above). Senge describes the last discipline, Team Learning as coming together to share and learn from each other while balancing personal goals with the development of organizational goals. The emphasis of team learning is on interaction, collegiality, and shared decision making. When organizations are able to achieve success in these five disciplines and truly become learning organizations, they are poised to achieve long term success and to pr oactively adapt to the changing environment in which they exist. The development of professional learning communities has been proposed as one way to create a culture of change in school systems. Schools that have been successful in establishing profess ional learning communities have staff members who are motivated to constantly evaluate their practice and search for means of improvement, making change an accepted and welcome part of the organization. Hord (1997) identified five attributes of successful learning communities. The first is shared values and vision When a staff has shared values and vision, all are focused on improving student outcomes and learning, and this is reflected in their individual professional actions and their shared decision making. Collective creativity the second attribute, is attained when staff

PAGE 87

71 members work together to address problems through dialogue, study, and shared decision making, which allows individuals to learn from each other and enables the organization to mo re effectively meet the needs of its students. When leaders of a school are learning and working along side other staff members, they are demonstrating supportive and shared leadership the third attribute. In professional learning communities, the role of the administrator is not to provide solutions, but to facilitate and support the collective learning and application process. Supportive conditions the fourth attribute, are those conditions (both physical and human) that uphold the collaborative prob lem solving and learning processes of the schools. Physical conditions may include schedules that allow shared meeting and planning times, effective communication procedures, and space for groups to meet. The human aspects of supportive conditions may in clude such things as positive staff attitude toward students and parents, a normative context in which professional development is valued and change is viewed as positive, and staff members who are open to feedback. The final attribute of successful learn ing communities is shared personal practice This can also be described as peers helping peers, and requires a culture of mutual respect and trust between colleagues so that open review and feedback can occur. Peer review can lead to improved personal an d professional capacities. Hord (1997) conducted a review of the literature on outcomes associated with professional learning communities and found that the outcomes were impressively strong and positive. Staff reported a reduction in feelings of isolati on, an increased commitment success. The changes led to higher morale, lower staff absenteeism, and increased

PAGE 88

72 satisfaction. Teachers were also found to adapt their inst ruction to student needs and to engage in innovative practices. The implementation of professional learning communities led to an increased commitment for making significant changes, as well as an increased likelihood that staff would engage in fundamenta l systemic change. With regard to student outcomes, schools with professional learning communities reported higher attendance rates, lower dropout rates, greater academic gains, and smaller achievement gaps among subgroups of students. These results sugg est that professional learning communities can be powerful tools for meaningful and lasting change in schools. Facilitating/ Leading Change While there are a number of factors that influence the success of a change initiative, leadership is identified by Hall and Hord (2005) as the most influential. The following discussion will outline differences in the characteristics of change leaders (Evans, 2001; Hall & Hord, 2005; Mendez Morse, 1992), systematic processes that effective leaders utilize when facili tating change (Hall & Hord, 2005; Stollar, Poth, Curtis, & Cohen, 2006), and how facilitating change is most successful when it emphasizes a team process (Hall & Hord, 2005). All styles of leadership are not created equal. Leadership can come in many si zes and forms, and differences in leadership style can lead to different outcomes in change initiatives (Hall & Hord, 2005). In other words, some leaders are more effective at implementing and sustaining change than are others. Many authors have discuss ed the characteristics of leaders who are effective in facilitating change. In his description of the authentic leader Evans (2001) suggested

PAGE 89

73 that the change process begins with trust and confidence in leaders. The key to both trust and confidence, acco rding to Evans, is authenticity. He suggested that successful leaders are not followed because of their knowledge base or because of their approach to the change process; rather, it is because they possess a certain level of integrity and are genuine peop le (i.e., they are authentic). Leaders who are authentic lead with clarity and focus; they know what they want and they work to get it. According to Evans (2001), authentic leadership is the key to successful innovation and change. Drawing from the lit erature on effective schools and effective teacher leadership, Mendez Morse (1992) identified six characteristics of effective change leaders. First, effective leaders have vision -a picture of the future -and they foster a shared vision among their s taff. Second, facilitators of effective change lead with the understanding and belief that the purpose of schools is to educate students, and that all decisions should be made with this in mind. Mendez Morse also found that effective leaders view their s taff as their most important resource, and that they value the professional contributions of their staff. Next, leaders who are successful at brining about change are equally effective as communicators as they are at being good listeners. Finally, effect ive leaders are proactive in anticipating and planning for change, and they take risks to meet the needs of their students and staff members. Based on information from their research and experiences, Hall and Hord (2005) identified three basic change fac ilitator styles: The Initiator The Manager and The Responder The initiator is thoughtful and acts purposefully and strategically. Leaders who fit into this category have a well articulated vision for what they want to achieve and a clear plan for att aining that vision. Initiators have high expectations for their staff and

PAGE 90

74 are also concerned with motivating that staff. They consciously and continuously engage in an improvement process, and question what they and others can do better. The manager is not as vision focused as the initiator but is very effective at running and managing the day to day activities of the school. Managers are not proactive take time t o assess the potential change and make a decision regarding adoption. In terms of the change process, Managers are more likely to take on tasks themselves, and are less likely to delegate. The third type of change facilitator style, the responder, does just what the name versus having a vision for the future. In terms of facilitating change, the responder is most comfortable when others take the lead. They are likely to delay decision making because they want to give everyone in the system a chance to provide input and to express their feelings regarding the decision. Hall and Hord (2005) contend that differences in change facilitator styles lead to different outcomes in the change process. The leadership style that leads to the highest level of use of an innovation is the initiator. The leadership style that leads to the lowest levels of implementation of an innovation is the responder. The authors hypothesized tha good decisions quickly that lead to their success in facilitating change. In summary, leaders who are effective at facilitating change possess certain characteristi cs. They are authentic, have a vision for what they want to achieve, plan accordingly, value their staff, and believe that improved student outcomes is the ultimate

PAGE 91

75 goal in school improvement. However, just possessing these characteristics do not necessa rily lead to successful change outcomes. Successful facilitators of change must also take certain actions and make certain decisions in order to move the change process forward. These actions and decisions make up the process of change. Just as researche rs and authors in the field of systems change have been able to identify the personal characteristics held by successful change facilitators, they have also been able to identify the actions and decisions also referred to as the change process -that ar e made by successful change leaders. Stollar, Poth, Curtis, & Cohen (2006) recommend the use of a collaborative strategic planning (CSP) process when engaging in systems change. The CSP is a five step approach that can be used to address systems level i ssues and is intended to build the problem solving capacity of the school system. The five steps to CSP are as follows: (1) problem identification, (2) problem analysis, (3) goal setting, (4) plan development and implementation, and (5) plan evaluation. CSP is a cyclical process that recycles until the problem originally identified has been remediated. CSP is also a data driven process, which relies on data to identify problems, analyze why they are occurring, develop goals, and monitor progress toward t hose goals. The primary purpose of the CSP is to create and foster a healthy school system that is in a continuous process of evaluation and improvement. Hall and Hord (2005) refer to the actions that leaders take to successfully events that either increase or decrease the potential for the success of change. Based on a

PAGE 92

76 review of the literature and their own personal experience, Hall and Hord (2005) identified six functions that are a necessary part of change interventions. facilitator collaborates wit h the school staff to create a vision of what the change will look like in practice when it is fully implemented, clearly articulates and defines that vision, effectively and continuously reminds staff of the desired outcome, and communicates progress towa rd achieving that vision. The second category of interventions is planning and providing resources. Actions of change facilitators related to this function include developing new policies and guidelines, shifting staff responsibilities, and securing fund s necessary to support the implementation of the innovation. Investing in professional development and learning is the third category of interventions. Change is about new understanding and new ways of doing things; so, supporting learning is an essentia l function of a change facilitator. The first three functions of interventions are actions that change facilitators take before the implementation of an innovation is initiated or during the early stages of implementation. The final three functions all oc cur during the implementation process and reflect sustained support for the intervention. Among these final three, function four is checking on progress toward the attainment of the desired vision. According to Hall and Hord (2005), the change process is not a straight road, but rather a road that is full of bumps and detours. Consequently, progress along the path of change must be carefully monitored. An effective change facilitator also provides continuous assistance to instructional staff in order to support implementation, which is the fifth intervention

PAGE 93

77 function. The final function of an effective change facilitator is creating a context supportive of change. The context of change includes both the physical aspects of the organization (e.g., facil ities, schedules) as well as the people aspects (e.g., norms, beliefs, values). As noted earlier, Hall and Hord (2005) emphasize that change is a process, not an event. It is a process that takes time and energy and often requires a team effort in order 2005, p. 148). In order to meet the demands of a change initiative, the authors recommend that the role of facilitating the change process be shared among two or mo re people, collectively identified as the change facilitator team. Through their research, Hall and Hord (2005) found that, in addition to the principal, who is most often the primary change facilitator, there is a second change facilitator in nearly all change efforts in schools. This person works closely with the school administrator and can hold a variety of positions (e.g., teacher, department chair, school psychologist), but he/she is always someone who has a special interest or skill related to the innovation being implemented. In many cases, this person has been given a special role or assignment to formally act as a change facilitator by providing individual support to teachers, monitoring the implementation process, etc. Hall and Hord also found that there may be third and fourth members of the change facilitator team. For example, an outside consultant might be lending his/her expertise to the change effort, although he/she is not a member of the staff at that school. As identified above with regard to facilitating change, effective change leaders possess certain personal characteristics, engage in a systematic process for facilitating

PAGE 94

78 change, and enlist the assistance of others in the change process. While these leadership characteristics an d behaviors are essential components of an effective change process, other elements are also necessary for success. The next section will identify the importance of staff development in the change process, and will outline the components and characteristi cs of effective professional development activities. Staff Development Staff development is the cornerstone of change and an essential component in the change process (Hall & Hord, 2005). When schools engage in systems change, teachers are asked to thi nk and to act differently. This requires that school staff understand the theories and belief systems behind the innovation and that they acquire the skills necessary to implement the innovation. This new learning requires systematic and comprehensive st aff development. The following discussion will describe the prevailing current model of staff development (Joyce & Showers, 2002), the research that supports the model, and staff development within the larger context of systems change (Little, 1997). Whil e the content of teacher training and staff development varies widely, the methods of effective training do not (Fixsen, Naoom, Blase, Friedman, & Wallace, 2005). The Joyce and Showers model of professional development is based on both research and the ex tensive experience of the authors. The model has been refined over many years, with the current model including four components of effective professional development (Joyce & Showers, 2002). The first component is knowledge and understanding. This compo nent typically takes place at the beginning of the process and focuses on the theory and rationale behind the new practice. Information can be communicated in large or

PAGE 95

79 small group lecture style format and/or in a discussion format. The second component i s the modeling of the new skill(s). This includes an actual demonstration of the skill(s). The demonstration should approximate the classroom or workplace setting where the skill is to be used as closely as possible. The next component, practice of the n ew skill, is an essential piece of effective staff development. Trainees should get their first chance to try out the new skill in a safe environment, with the support of the trainer and their peers. Joyce and Showers (2002) recommended that trainees hav e the opportunity to practice a variety of scenarios that may be encountered in actual practice. This practice component should not be limited to the initial training phase; rather, it should take place over a significant period of time (8 10 weeks), with many opportunities to practice (25 or more trials). Coaching, the fourth component of the Joyce and Showers model of professional development, is defined as the opportunity to collaborate with a peer regarding implementation of the innovation. This may include planning for implementation, creating and collaborating on materials, and developing lessons. Joyce and Showers used to advocate for peer coaching to include systematic feedback on performance during implementation. However, they have since concl uded that feedback can become too evaluative and too supervisory, and that training in effective and supportive feedback requires too much time and resources. Joyce and Showers (2002) have also identified several other considerations when designing effect ive professional development. One of these is how the content of the training is initially selected. More specifically, the authors assert that the topics and skills being taught must be considered useful by the trainees, who must view the new strategies as beneficial to students. In addition, new practices and programs considered for

PAGE 96

80 adoption should be evidence based and well defined prior to initiating training and implementation. Another consideration is that the training should be differentiated bas ed on the intended outcomes and the targeted audience. For example, an intended outcome of simply sharing information might only require the first component of the Joyce and Showers model (i.e., knowledge and information) and might be accomplished in a fe w hours. In contrast, the implementation of a new mathematics curriculum would require all four components of the training model and could take a year or more to attain full implementation. Joyce and Showers (2002) conducted a meta analysis in the area of staff development in order to determine which components of staff development were most likely to impact teacher behavior in the classroom. The authors found that when teachers were simply provided with theory and discussion on new practices, very few of them (5%) were able to demonstrate the new skill during the training session, and none of them (0%) generalized the new skills to the classroom. When theory and discussion were combined with skill modeling only, with no opportunities for practice, the percentage of teachers who were able to demonstrate the skill during the training session was more substantial (20%), but still did not result in the ability to demonstrate the skill in the classroom (0%). When trainees were given opportunities to actuall y practice the skill and receive feedback during training, their ability to demonstrate the skill in the training session increased dramatically (60%), and the percentage who generalized the skill to the classroom increased modestly (5%). When coaching i n the classroom (the fourth component of the Joyce and Showers training model) was added, the percentage that were able to generalize the skill they had learned and to demonstrate that skill in the

PAGE 97

81 classroom increased dramatically, to 95%. It is clear fro m their research that the coaching component of staff development is essential for teachers to change their practices in the classroom. While the Joyce and Showers model of professional development has been found to change teacher behavior in the classroo m, Little (1997) postulated that this model is initiatives and contends that if schools are going to achieve true reform, staff development needs to be completely r e conceptualized. Little describes most current reform initiatives (e.g., increases in student achievement, changes in the nature and use of student assessments, reforms in the social organization of schooling, reductions in achievement gaps, etc.) as bei ng large scale and complex in nature. She argued that this kind of fundamental change cannot take place in isolation and cannot be achieved through small, incremental changes in teacher knowledge and skill as produced by most professional development effo rts. In order to address this mismatch, Little recommended that professional development be reformed to (a) create meaningful social and intellectual engagement with school problems and solutions, so as to increase the professionalism of teaching; (b) take into account the experiences and prior knowledge of teachers so that new ideas can be fully integrated into current practice; (c) incorporate opportunities for teachers to express dissent and challenge prevailing beliefs and practices; (d) explicitly take into account big education; (e) instruct teachers on how to learn and how to engage in consistent inquiry and self improvement rather than teaching isolated skills that become obsolete; and ( f) balance resource allocation between the needs of the individual and the needs of the

PAGE 98

82 the second component of effective change identified above in this literature r eview. Unfortunately, too often even the best efforts at professional development fail to make a real impact on practices in the classroom and on student outcomes or, if an impact is made, the effects of the professional development efforts are only temp orary. The coming and going of innovations in schools has become such a pattern that teachers have new innovation with the hopes that it will be gone before they have to m ake real change. The final component of effective systems change is taking action to sustain the change and the effort to sustain the change begins even before the innovation is implemented. Sustaining Change Efforts to implement new innovations in temporary funding has ended and staff hired or assigned for the specific purpose of supporting the innovation are finished. This is due to projects and innovations being viewed as temporary efforts to bring about chang e, as well as a failure to create an overall climate and infrastructure to support the change long term (Adelman & Taylor, 2003). Adelman and Taylor (2003) describe sustainability as the institutionalization of the innovation, or the innovation becoming an ongoing and smooth component of school becomes the way the school does business and the practice is no longer identified as an innovation. Based on their experience w ith attempts to sustain demonstration projects, Adelman and Taylor (2003) suggested that sustainability must be explicitly planned for

PAGE 99

83 and the actions needed to sustain the change identified. They suggested that those facilitating change should take strid es to increase sustainability even before the project begins, and described a four phase framework for sustainability that includes: (a) creating readiness for change, (b) initial implementation, (c) institutionalization, and (d) ongoing evolution and rene wal. Prior to implementation of an innovation, Adelman and Taylor suggested that several actions be taken that would increase the likelihood of the practice being sustained long term. First, when planning for change, only those innovations that have tr ue value (i.e., focus on improving student performance, have highly valued outcomes, and/or address a problem or area of need) should be considered for implementation. Innovations with little value are not likely to be sustained over time. Second, innova tions being considered for implementation should be connected to the overall vision and mission of the school and a component of the school improvement plan. If the innovation is not connected to the larger school vision and is an island unto itself, it i s less likely to have support for sustainability. Similarly, considerations should be made as to how the innovation can be supported within existing school structures. Temporary staff and funding structures should be used sparingly. Creating a climate and culture for change will increase the likelihood of an innovation being sustained (Adelman & Taylor, 2003). Change is a difficult process because people are asked to think and do things differently. Creating a climate and culture of change increases the buy in and motivation of staff to invest the hard work that is necessary for change to occur. For more information on creating a climate and culture

PAGE 100

84 In addition to carefully considering which innovations to select for implementation, and to creating a climate and culture of change, Adelman and Taylor (2003) suggested that leaders of change identify a long term plan for implementation which includes h ow progress will be determined and formulates structures for continuous improvement. This plan should be reviewed throughout implementation. Once stakeholders are judged to be ready for change, the infrastructure for supporting the innovation has been p ut in place, and a plan has been made for implementation and sustainability, the initial implementation phase can begin. Several actions can be taken during this phase to increase the likelihood of the innovation being sustained (Adelman & Taylor, 2003). First, it is important to systematically phase in changes and to provide sufficient support to staff during this initial phase of the project. Projects often fail when the system becomes overwhelmed by the change. Second, leaders of change and school st aff should enter into the implementation phase with the knowledge and understanding that the change process is going to be complex and that barriers to implementation will arise. These barriers can be successfully overcome by proactively identifying proce sses and procedures for systematically addressing problems and barriers as they arise. Finally, it is important to carefully monitor progress during the implementation phase and to celebrate early successes. Adelman and Taylor (2003) identified four temp orary infrastructure mechanisms that can be used to support the change process during the planning and initial implementation phases of change. These are: (a) a site based steering mechanism to guide and support the pursuit of the vision; (b) a site based change team that has responsibility for coalition

PAGE 101

85 building, implementing the strategic plan, and maintaining daily oversight, problem solving, conflict resolution, and so forth; (c) a change agent (e.g., organization facilitator) who works with the change team and has full time responsibility for the daily tasks involved in creating readiness and the initial implementation of desired changes; and (d) mentors and coaches who model and teach specific elements of new approaches. (p. 13) It is important to not e that these temporary structures are meant to be just that, temporary. The innovation is more likely to be sustained if these separate mechanisms for support are faded out and the innovation is sustained within the pre existing fabric of the system. In stitutionalization of the innovation occurs once the temporary funding and support structures have been removed and the innovation becomes a natural and integrated part of the school system (Adelman & Taylor, 2003). It is important to s formal policies and practices and to ensure that these structures are in support of sustaining the innovation. If the innovation continues to require additional funding for support and sustainability, that funding should be internal to the organization and not dependent on outside monetary sources to sustain the practice. Finally, any structures or mechanisms that were created to support initial implementation of the innovation, such as a steering committee or change team, should be faded out and the re sponsibility for sustaining the innovation given to a pre existing support structure, such as the school improvement team. It is important to explicitly address and plan for the ongoing evolution and renewal of the innovation or practice. Schools are d ynamic systems and, therefore, efforts to sustain a practice or innovation need to be continuously adapted and renewed so

PAGE 102

86 that it continues to meet the needs of the system and be maintained within that system. This can be accomplished by creating a learni ng community (Adelman & Taylor, 2003) and becoming a learning organization (Senge, 1990). Furthermore, the ongoing evolution of an innovation is dependent upon an effective system for monitoring the progress of the innovation. The careful monitoring of p rogress will allow system leaders to identify when the innovation is in trouble and needs to be reviewed and renewed. The structure for promoting the sustainability of an innovation suggested by Adelman & Taylor (2003) is not a simple process and the act ions taken to promote sustainability clearly parallel the other components of effective systems change as identified in this literature review (e.g., planning for change, developing a culture of change, facilitating change, etc.). While structure of this literature review dictates that the components of effective systems change be presented separately from one another, systems change is not a simple process that can be accomplished formulaically. Systems change is a complex process and successful systems change efforts address these components in an integrated and seamless fashion that is tailored to the needs of each specific system. Conclusions This review of the literature has served to identify what research suggests are best practices in reading inst ruction as well as to identify barriers and catalysts to systems change as suggested by the literature in the area. The proposed evaluation study is of a literacy initiative which has strived to implement the identified best practices in reading in the c ontext of a program for students with moderate to severe disabilities. The literacy initiative has required teachers to reconceptualize their literacy instruction and change

PAGE 103

87 their instructional practices. Therefore, it is important to consider how the li terature on systems change relates to the current literacy initiative so that any potential barriers to implementation can be identified and that those leading the initiative can promote factors that support and facilitate implementation. The following chapter, Method, will review the evaluation plan that has been proposed to support the three identified goals of the evaluation study: 1. To examine how the ELS Literacy Initiative currently is being implemented. 2. To determine to what extent the anticipated sh ort term and intermediate outcomes of the initiative are being realized. 3. To determine next steps in implementation of the ELS Literacy Initiative. These goals will form the direction and foundation of the evaluation study.

PAGE 104

88 CHAPTER THREE METHOD Th e pur pose of this program evaluation study was to examine the implementation of a specific literacy initiative (the Educational and Life Skills [ELS ] Literacy Initiative) with in the context of an educational program that serves students with moderate to s evere disabilities, and to use that information to assist the p rogram a dministrator in future decision making The goal of the initiative is to apply best practices in literacy instruction to a unique population that has been largely ignored in the litera ture of the field. Co nsiderable material, personnel, and financial resources as well as professional development efforts, have been dedicated to supporting the initiative There were three broad goals for this study: (1) to examine how the ELS Literacy Initiative was being implemented, (2) to determine to what extent the anticipated short term and intermediate outcomes of the initiative were being realized, and (3) t o determine the next steps in the implementation of the ELS Literacy Initiative This c hapter describes the design and procedures that were used to achieve the identified goals of this study. Evaluation Design Worthen, Sanders, and Fitzpatrick (1997) defined evaluation as the teria to determine an

PAGE 105

89 of formative evaluation and summative evaluati on. The current study may be considered a formative evaluation because the information gathered will be used to improve a program initiative and to determine the next steps in its implementation. While some initial outcomes of the initiative have been ex amined, the goal of the evaluation was not to determine the overall worth of the program based on these outcomes; therefore, this study did not fall into the category of summative evaluation. According to Worthen, Sanders, and Fitzpatrick (1997), program e valuation as a study design has been applied broadly, and many diverse uses of evaluation studies can be found in the field of education (e.g., to judge the quality of a curriculum, to determine the value of a specific program, or to provide evidence of in itiative outcomes for outside funding agencies). Program evaluation was selected as the study design over other methods of inquiry for several reasons. Purely quantitative research designs (e.g., correlational designs or experimental designs) were not de emed appropriate because the purpose of the evaluation was not to validate or confirm relationships between variables and then to generalize that information to the larger population. Rather, this evaluation sought to identify the variables impacting the implementation and outcomes of the l iteracy i nitiative, discover the relationships and themes among those variables, and then use that information to make decisions about and improve upon the program. Other approaches to qualitative research (e.g., case s tudy research and historical research) were also not deemed appropriate because the study was not conducted simply for the purpose of description or to develop theories that explain educational phenomena. Rather, the researcher and the client were interes ted in identifying what was working and should be

PAGE 106

90 maintained in the initiative and, conversely, what was not working and should be changed, as well as to evaluate initial outcomes of the program. Therefore, program evaluation as a study design was identif ied as most appropriate for the current study because the purpose of the study was to collect data that would facilitate decision making (e.g., determine the next steps in implementation) as well as to aid in making initial determinations regarding the wor th of the program (e.g., its effects on students and teachers). There are a number of different approaches to program evaluation, such as objectives oriented evaluation, participant oriented evaluation, expertise oriented evaluation, and consumer oriente d evaluation (Worthen, Sanders, & Fitzpatrick, 1997). For th e current study, the management oriented approach was identified as being most appropriate for meeting the goals of this evaluation study The management oriented approach to evaluation had bee n developed to meet the needs of program decision makers (such as program administrators) who use the information collected to inform specific decisions that must be made. One of the benefits of the management oriented approach is that it can be used at an y time during program implementation before, during, or after to support decision making. Stuffelbeam (1971) developed the CIPP evaluation model, which contributes to the decision making process in program management, to meet the needs of program administ rators who were faced with decisions at different points during implementation. context evaluation, in which information is gathered prior to program selection in order to identify the needs to be addressed by a progra input evaluation, in which information is

PAGE 107

91 gathered to aid decisions regarding which resources are available and what the specific plan for implementation should be. The next step that the CIPP model outlines is the imp process evaluation). Examples of decisions to be made at this stage include whether or not the plan is being implemented as intended, what barriers to implementation have arisen, and what changes might be needed to improve impleme product evaluation, the final stage of implementation. At this point in the process, questions regarding the outcomes of the evaluation are addressed and decisions are made regarding whether or not to continue t he program and what, if any, changes need to be made in order to improve program outcomes. T he management oriented approach offers several advantages over other evaluation approaches (Worthen, Sanders, & Fitzpatrick, 1997). The management oriented approa ch is more rational and orderly than other approaches, moving the evaluation forward with the purpose identified by program management. Additionally, because the purpose of the evaluation is to aid in decision making, the management oriented approach is m uch more focused in its scope than are other approaches. One of the biggest criticisms of past program evaluation efforts has been that the studies did not produce useful information. One of the greatest strengths, then, of the management oriented approa ch is that it directly addresses the question of information utility by identifying the outcome based decisions to be made prior to the beginning of the evaluation process. The management oriented approach does have its limitations, however (Worthen, Sand ers, & Fitzpatrick, 1997). One identified limitation is that the evaluation is

PAGE 108

92 management requests and recommendations. In this sense, the evaluator risks becoming too limited by the evaluation desires of the manager, and thus the evaluator may miss other important evaluation questions and perspectives. This also creates a situation in which the evaluator may be viewed as acting in the best interests of the manager, and not necessaril y in the best interests of other stakeholders associated with the program. Another limitation is that, if followed precisely, the evaluation can quickly become costly and overly complex, especially when several decisions must be made. Finally, the manage ment oriented approach assumes that important questions and decisions to be made can be identified in advance. Because this is not always possible, frequent changes to the original evaluation plan may be necessary for this approach to be successful. Compo nents of other approaches to educational evaluation were also included in this study. For example, because the current evaluator is also directly involved in the implementation of the program being evaluated, the study made use of some ideas that are cent ral to the participant oriented approach to evaluation: namely, the belief that a program evaluation is enhanced by personalizing the evaluation process through involving someone who is directly associated with the implementation of the program. Procedure Preparation S everal actions had to be completed before beginni ng the program evaluation study. F irst, the reasons for initiating the evaluation had to be clarified to ensure that it was being done for appropriate reasons. N ext the stakeholders of the p rogram and of the evaluation had to be identified. T hird, t he person responsible for conduct ing the program

PAGE 109

93 evaluation (e.g., either an internal or external evaluator ) had to be identified. Finally, a com plete definition of the program to be evaluated (e .g., what it does and does not include) was an essential step that had to be taken prior to i nitiating the evaluation. Reasons for the Evalua tion It was important to clarify the reasons for initiating the evaluation to ensure that it was being requeste d for appropriate reasons (e. g. to determine the merit of a program instead of gathering evidence to support a decision that had already been made ). The administrator responsible for the literacy i nitiative was interested in gathering information regardi ng the implementation of the initiative, as well as in gaining a greater understanding of the strengths, weaknesses, and initial outcomes of the program in order to facilitate future decision making. Additionally, t he evaluator was also interested in eval uating the l iteracy i nitiative because of personal investment and involvement with the initiative as well as a genuine desire to identify the next steps in its implementation. The evaluator also initiated the evaluation for the purpose of meet ing the req uirements for dissertation research C larifying the reasons for initiating the evaluation also aided in selecting the most appropriate evaluation approach. Reasons for the selection of the management oriented approach to program evaluation were discussed above in the Evaluation Design Based on the evidence provided, it can be concluded that the reasons for initiating the evaluation were reasonable and ethical. Stakeholder Identification Gall, Gall, and Borg ( 2006 ) define a stakeholder the program being evaluated or who might be affected by or interested in the findings of 684 ). It was important to identify the stakeholders of the program

PAGE 110

94 being evaluat ed from the outset so they could pa rticipate in identifying the reasons for the evaluation, defining the program being evaluated, and assist ing in the identification of evaluation questions. Stakeholders could also assist in the interpretation of results and they should be included when t he results of the evaluation are shared. program a dministrator responsible for the l iteracy i nitiative. Because the evaluation was being conducted primarily to aid in the the greatest influence on how the evaluation proceeded. The evaluator identified a dditional stakeholders of the l iteracy initiative with the assistance of the p rogram a dministrator during initial client in terviews. Several of the stakeholders (i.e., the program administrator, the l iteracy c oach to the program, two teachers, and an intervention specialist in the program) were interviewed in preparation for the evaluation study to assist in defining the init iative and in identifying evaluation questions. Additional stakeholders (e.g., other teachers, parents, etc.) became involved during the data collection and data reporting phases of this study. Evaluator Selection The current program evaluation was condu cted by an internal evaluator. More specifically, the evaluator is currently an employee of the educational program in which the li teracy i nitiative has been implemented and has provided leadership to the l iteracy initiative. Worthen, Sanders, and Fitzpa trick (1997) identified several benefits to the evaluator being a part of the system (as opposed to an external evaluator), including the fact that an internal evaluator would be more knowledgeable about the program and its history, have more familiarity w ith the stakeholders and their interests and concerns, and

PAGE 111

95 have a greater understanding of the organization and its dynamics. An internal evaluator can often begin the evaluation more quickly and has the benefit of being in a position to advocate for the use of the evaluation findings. However, there also could have been benefits to having an external evaluator conduct the study (Worthen, Sanders, & Fitzpatrick, 1997). For example, external evaluators are more likely to be impartial and not have a perso nal bias or agenda, and therefore they may be more credible to outside audiences. They also have an outside perspective, and they may be more honest or even blunt when necessary. Because the primary purpose of the current program evaluation was to aid in decision making and to identify the next steps of implementation (not to evaluate the outcomes of the program), an internal evaluator w as not considered a significant limitation. Definition of the Program to be Evaluated Defining and describing the pr ogram to be evaluated is a fundamental step in the preparation of an evaluation study (Worthen, Sanders, & Fitzpatrick, 1997). The description sets the boundaries of the evaluation (i.e., what it does and does not include) and supports a common understand ing among evaluator, client, and stakeholders. Several sources of information were used to develop the current program description. First, information was compiled, based on initial stakeholder interviews, and during this process specific questions were asked with regard to the program description. Next, historical and current d igital and paper documents pertaining to the literacy i nitiative were reviewed. Finally, the program evaluato r drew upon her personal knowledge of the program. In order t o preve nt personal bias from affecting the program description, the program a dministrator re viewed the program description for accuracy. The next section

PAGE 112

96 provides a comprehensive description of the li teracy i nitiative and includes information related to the back ground and history of the initiative, the scope of the evaluation, the problems that the initiative was designed to address, and the components that define the literacy i nitiative. Program Description First, background information for the Educational and L ife Skills (ELS) Literacy Initiative is provided establishing a context for the i nitiative as well as a historical overview of i nitiative activities. Next, the scope of the evaluation is defined with regard to which school years and grade levels were inc luded in the evaluation. After the scope of the evaluation is identified, the programmat ic issues or concerns that the i nitiative was designed to address (i.e., problem statements) are identified. The next three sections a ddress the inputs, activities, and outcomes of the initiative These sections mirror the primary components of a logic model. A logic model is a visual representation of how a program or intervention strategy is designed to address a specific problem or set of problems, and how the a ctivities of the program link include the training and other components of the program th at take place during implementation. term) and indirect (lon g term) changes in behavior and/ or conditions as a result of the program strategy. The format of the logic model is similar to the CIPP framework of evaluation as defined by Stufflebeam. (1971): C ontext Evaluation, I nput Evaluation, P rocess Evaluation, and P roduct Evaluation.

PAGE 113

97 Other components of a logic model may include the underlyi that serve as the foundation for a program or ser may impact the program and the ability to achieve the identified outcomes. Assumptions and contextual factors are the two final components of th e program description. Background Information The district in which the current evaluation study took place is a special education cooperative located in the north shore suburbs of Chicago. The cooperative serves 19 member districts and supports a total population of approximately 40,000 students. The cooperative hosts s everal instructional programs and provides a variety of services to its member districts. T hree main programs are operated by the cooperative district The Early Childhood Program (ECP) is an early intervention program that serves children from birth th rough age 5 The Educational and Life Skills (E LS) program serves students age 6 through 21 who have moderate to severe cognitive impairments, multiple disabilities and autism. Finally, North Shore Ac ademy (NSA) serves students aged 6 through 21 who hav e moderate to severe emotional and behavioral disabilities. The services that the cooperative district provides to its member districts include among other things, professional development relating to educational best practices, direct coaching and consu ltation relating to academic and behavioral prevention and intervention, occupational and physical therapy, and transition services. The ELS program provides instruction in academic and life skills within integrated school settings to students who have moderate to severe/profound cognitive disabilities, multiple disabilities, developmental disabilities, and autism The program is

PAGE 114

98 grounded within a trans disciplinary model which means that professionals from different disciplines work closely together t o provide instructional programming for students. Each classroom is typically staffed with a full time teacher, two or more full time teaching assistants, a speech and language pathologist who is in the classroom one and a half days per week and an inter vention specialist w ho is in the classroom one day per week. Other support staff members who might be present in the classroom include an occupational thera pist, a physical therapist, and a nurse. Research has found that programs serving a population of students similar to those in the ELS program historically have placed a strong emphasis on, and have devoted most of their instructional time to, the teaching of functional and daily living skills. In contrast, much less emphasis has been placed on acade mic skills, such as reading (Browder, Wakeman, Spooner, Ahlgrim Delzell, & Algozzine, 2006). Furthermore, when teachers have included literacy in their instruction to this population, the focus of that instruction most often involves teaching students fun ctional sight words used for daily living skills (Houston & Torgesen, 2004). A review of historical data and information collected during initial stakeholder interviews suggests that prior to the development of a l i teracy committee during the 2000 2001 sch ool year, literacy instruction in the ELS program mirrored programmatic practices described in the research literature M ost reading instruction and literacy experiences provided by the program were embedded within the context of teaching functional life skills or social skills. When reading was taught directly, the content of that instruction typically consisted of sight words. Some teachers used published curricula as a basis for student sight word instruction (e.g., the Edmark Program). In

PAGE 115

99 addition t o published curricula, sight word instruction was also taught through other instructional formats such as Discrete Trial. ( Discrete Trial is a behaviorally based instructional strategy in which students are drilled in specific skills until they master tho se skills ) Other reading skills that were sometimes taught through the Discrete Trial method included letter names and, less frequently, letter sounds. The research and literature base in the area of reading has grown tremendously in the past five to t en years. Several national ly commissioned committees including the National Reading Panel (NRP) have conducted meta analyses in order to identify scientifically based instructional practices in the area of reading In its report, the NRP concluded that (a) comprehensive reading instruction should address phonemic awareness, phonics, fluency, vocabulary, and text comprehension; (b) reading instruction should begin early (preschool, kindergarten) and, at that level, it should include explicit instruction in phonemic awareness and phonics; and (c) pre and post service teacher training should better prepare teachers to instruct students in the area of reading at all grade levels (NICHHD, 2000). Federal legislation guiding education, particularly the No Chi ld Left Behind ( NCLB) Act (NCLB 2001, PL 107 110) and the Individuals with Disabilities Education Improvement Act ( IDEA of 2 004, PL 108 446), ha s placed specific emphasis on the importance of based and it has rec ognized the importance of instruction reflecting scientifically based practices and early intervention. However, there continues to be a dearth of research regarding best practices in literacy instruction for students with moderate to severe cognitive dis abilities as well as students with autism.

PAGE 116

100 History of t he I nitiative Information about the history of the literacy initiative in the ELS program was gathered from several sources, including information shared by stakeholders during initial interviews, personal written notes provided by stakeholders, and other permanent products and documents generated by the work of the ELS literacy committee over several years. The following is a brief review of the history of the ELS Literacy Initiative based on thi s information. Systematic efforts to improve literacy instruction in the ELS program began during the 2000 2001 school year, when the program administrator created a literacy c ommittee. The committee chair was an intervention specialist, a school psycho logist in the ELS program who had an interest and an educational background in literacy. Under the leadership of the chair, this group began its work by surveying ELS teachers on their current practices and professional development needs. Simultaneously, this group began to review research in order to identify best practices in literacy instruction for students with disabilities. The committee continued its work during the summer and into the next academic year. By the end of the 2001 2002 school year, t he group had achieved two major outcomes. The first was the completion of a literacy pilot study in the ELS program that compared the effectiveness of three reading programs (Wilson Reading, Reading Mastery, and Reading Milestones). The second outcome wa s the creation of a teacher instructional resour ce that identified the skills taught in reading and provided teachers with resources and ideas on how to teach those skills. Efforts to identify a system for tracking student progress in the area of literacy also began during the 2000 2001 school year. The l iteracy committee began its work in

PAGE 117

101 this area by reviewing several available assessment tools in order to evaluate their usefulness for effectively monitoring the literacy skill progress of this populatio n of students. Its work on identifying effective progress monitoring tools continued into the 2001 2002 school year, when it identified assessment tools that would be used to measure the outcomes of the literacy pilot study. The literacy committee select ed s everal tools for this purpose, including an assessment of concepts of print, Curriculum Based Measurement Oral Reading Fluency (CBM ORF), the Dynamic Indicators of Basic Early Literacy Skills (DIBELS), an adapted version of the DIBELS for nonverbal stu dents, and a sight word list. The progress of students participating in the pilot study was assessed three times during the 2001 2002 school year (fall, winter, spring) using one or more of the identified tools. During the 2002 2003 school year, the lite racy progress of all ELS students was assessed in the winter and again in the spring using one or more of the tools used during the pilot study. The work of the ELS Literacy Committee continued its work during the 2002 2003 school year. The goals of th e group included increasing the coordination of literacy instruction across levels, documenting student skill development in literacy, identifying quality literacy materials and programs at the high school level, beginning a literacy mentoring program for new teachers, and expanding and improving representation on the literacy committee itself. Other achievements during the 2002 2003 school year identification and acquis ition of many new literacy instructional materials for all grade levels.

PAGE 118

102 In order to improve instructional planning and increase communication across school years, the committee developed a literacy development tracking form during the 2002 2003 school y ear. Teachers were asked to indicate whether a student had mastered any of the 41 different literacy skills on the list. Data from the 2002 2003 literacy benchmark assessments and from the new literacy tracking forms were compiled in a report titled Educ ational and Life Skills Program Report of Progress in Literacy: 2002 2003 At the end of the 2002 2003 school year, the chair of the literacy committee left the district and did not return for the following school year. When she left, a new l iteracy c om mittee chair was not named to succeed her, mainly because a person with the necessary qualifications had not been identified. However, the committee introduced a c 2004 school year. The primar y purpose of this new position was to provide direct support to teachers in the area of literacy instruction. The program administrator hired a former teacher to fill this new role. The 2003 2004 and 2004 2005 school years were not as productive as the previous three school years had been with regard to making improvements in literacy assessment and instruction in the ELS program. One hypothesized reason for this lack of progress involved the significant changes in leadership that took place within the ELS program. As mentioned, the l iteracy committee chair was no longer with the district and there was no one with as much knowledge and interest in literacy available to take her place. For 2003 2004, a new l iteracy c oach was hired to provide direct sup port to teachers, but the l iteracy c ommittee as a group did not continue. Additionally, the

PAGE 119

103 p rogram a dministrator who had initiated the literacy committee retired at the end of the 2003 2004 school year. During the following year (2004 2005), a new p rog ram a dministrator was appointed and immediately expressed interest in continuing efforts to improve curriculum and instruction. However, she needed time to acclimate herself to the new position before providing significant leadership in this area. The ne w ELS literacy coach continued during the 2004 2005 school year, but she left midway through the year on a maternity leave and was not as available for the remainder of the year. There was an effort to reinitiate the l iteracy c ommittee during the 2004 200 5 school year, with the ELS literacy coach and the current evaluator being named co chairs. However, the current evaluator was not given time in her schedule to provide this leadership, and the l iteracy c oach was on maternity leave. Efforts to reinitiate the ELS literacy benchmark assessment system did come to fruition during the 2004 2005 school year under the leadership of the current evaluator. Some changes were made to this system during that school year for the purpose of yielding more useful and ac curate information. In March of 2005, the current evaluator and another intervention specialist in the Literacy Assessment: Literacy Assessment, Interpretation and C urriculum for Students these two staff members to create a more comprehensive and useful resource on literacy instruction and assessment for ELS teachers than had been available previously. During the following school year (2005 2006), the current evaluator and the intervention specialist, with input from the literacy c oach and the p rogram administrator, created a

PAGE 120

104 comprehensive literacy skill scope and sequence that beg an at the emergent literacy level and was tied to assessment strategies. This important development in the efforts to improve literacy instruction in the ELS program, along with other related activities, may be considered the beginning of the current ELS Literacy Initiative. The activities that were part of the l iteracy i nitiative from the 2005 2006 school year and beyond are described in the remainder of this section. Scope of the Evaluation Historically, efforts to improve ELS literacy instruction can be traced back to the 2000 2001 school year, when the ELS literacy c ommittee was created. However, for the purposes of the current evaluation, the ELS Literacy Initiative is defined as beginning during the 2005 2006 school year. This starting point was identified for several reasons. First, 2005 2006 was the year when the ELS Literacy Scope and Sequence was being developed for adoption by the ELS program This resource identified the developmental steps of learning to read, beginning with pre kindergar ten level skills, and targeted seven areas of instruction (concepts of print, phonological awareness, etc.). Additionally, the Scope and Sequence was tied to assessment materials and to instructional strategies contained within a four volume binder set. It served as a significant development in the l iteracy initiative because this was the first time that teachers were given such a comprehensive set of materials to guide their instruction in the area of reading. The Scope and Sequence and related material s were not ready for adoption until the 2006 2007 school year. The second reason the 2005 2006 school year was selected relates to the identification of a core curriculum. S everal teachers were asked to implement two Direct

PAGE 121

105 Instruction programs ( Language f or Learning and Reading Mastery) and provide feedback on their experiences in considering adoption of the programs as a program wide core reading curriculum during the 2005 2006 school year These programs were selected because of the research base that supported them and because of the prescriptive and systematic nature of the instruction. While many ELS students were not able to participate in this program (e.g., nonverbal and severely impaired students ) the program did show promise for addressing the instructional needs of many students in the program. Consequently, during the following school year, these programs were purchased for all literacy curriculum. Finally 2 005 2006 was the second year that the new program administrator for the ELS program was serving in her position During her first year as program administrator, she devoted most of her time to becoming acclimated to the program. D uring her second year in the position, she was able to provide significant leadership in the area of curriculum and instruction. She made it clear during her second year as a dministrator that she prioritized a nd valued reading instruction. S he actively took strides to ensure th at the program was engaging in best practices in this area. While the ELS Literacy Initiative is intended to improve student outcomes in the area of literacy at all grade levels, the scope of the current evaluation has been limited to grades K 5 (primary and intermediate levels) because resources related to the literacy initiative (especially curricular materials) have been prioritized at this level. More specifically, during the 2006 all primary and intermediate classrooms ; moreover, it was expected that the teachers in

PAGE 122

106 these classrooms would begin to use the curriculum with those students who had the prerequisite skills for participation in the program (e.g., students who were verbal and were abl e to follow directions without support, etc.) While material and human resources also have been allocated to the older grade levels, a core curriculum has not been identified for these grade levels and similar implementation expectations have not been es tablished Limiting the evaluat ion to grades K 5 also increased the feasibility of the evaluation. Problem S tatements During the initial stakeholder interviews, w hen asked the question problem that the literacy initiative was intended t all stakeholders identified the lack of teacher training and knowledge in best practice beginning reading instruction as a problem, including the teachers themselves. The consensus was that teachers typically graduated from college unprepared to teach beginning reading, let alone to teach reading to students with significant learning differences and challenges. Their undergraduate training was either deficient in training in this area, or it was not aligned with current best practice s (e.g., i t was not focused on sight word instruction or literacy imbedded in life skill instruction). The other most commonly cited problems that the l iteracy initiative was designed to correct included a lack of research based practices, a lack of curricular res ources to instruct with in the area of reading, inconsistencies in instruction between classrooms, failure to use data when making instructional decisions, and disjointed instruction for students from one year to the next. All of these problems, in additi on to the primary identified problem of a lack of teacher knowledge and skill in the area of reading

PAGE 123

107 instruction, have been included in a visual representation of the l iteracy initiative as Appendix A ). Inputs The resources th at have been allocated to the literacy i nitiative (the inputs) can be placed into three categories : material, people and financial (see Appendix A ). Most of the material resources have been curricular resources, including the ELS Literacy Scope and Seque nce the identification and purchase of a core language/reading curriculum, and the dissemination of supplemental curricular materials. The ELS Literacy Scope and Sequence is a document that was created to guide teacher instruction in the area of r eading. The document identifies the stages of literacy development ( n ovice, b eginner, e arly to u pper e mergent, and u pper emergent to f luent) in seven different areas of literacy (concepts of print, letter identification, phonemic awareness, phonics, fluency, voc abulary and comprehension, and writing). The Scope and Sequence is tied to assessment strategies and instructional recommendations and it serves as a resource for in struction for all students. The ELS Literacy Scope and Sequence is different from an earl y literacy scope and sequence designed for elementary age general education stud ents because it begins with an emergent literacy level of skills, or those skills that are typically acquired prior to entering kindergarten. Another way in which this resourc e is unique is that many of the assessment strategies and instructional recommendations are adapted to meet the needs of students who are nonverbal and/ or physically impaired. In addition, t improvement over other instructional resources that had been provided to teachers in the past.

PAGE 124

108 Prior to the ELS Literacy Initiative the program lacked a core curriculum in the a nd the content of that programming varied widely within the program. One major change with the ELS Literacy Initiative involved the identification and purchase of a core language and reading program. The selected curriculum was the combination of Languag e fo r Learning and Reading Mastery (two Direct Instruction programs), the first targeting language development and vocabulary and the second targeting phonemic awareness, phonics, and fluency. These Direct Instruction programs were selected because of (a ) the research support for the programs, (b) the structured and predictable nature of the instruction, and (c) the positive results realized from piloting the programs. These programs do require a student to be able to verbally respond on cue ; as a result the percentage of students per classroom who are able to participate in the program varies widely. Regardless, the expectation from the program administration is that these programs will be used with as many studen ts as possible in grades K 5. To achie ve this goal, a set of curriculum materials has been purchased for every pri mary and intermediate classroom Other material curricular resources that have been allocated to the l iteracy initiative include supplemental materials (either as part of the Scop e and Sequence ordered through publishers, or created by teachers), instructional planning tools (e.g., the i nstructional p lanning f orm and the 4 Block planning worksheet), and technology resources (e.g., computer programs targeting literacy skills, techn ology that allows students to access instructional materials, and website subscriptions for instructional materials ).

PAGE 125

109 In addition to curricular resources, material resources allocated to the literacy i nitiative also include assessment resources. The int ervention s pecialist s in the classrooms have all been trained in the use of literacy benchmarking tools as well as curriculum based assessment tools for instructional planning. Most of these tools have also been made available in an adapted version so the tasks can be administered receptively or made more accessible for students with physical disabilities. Additionally, the program has acquired a subscription to AIMS web, which includes a graphing tool for those students whose progress is being monitored w ith standard (non adapted) Curriculum Based Measurement probes. Finally, a form was created to allow teachers to track student literacy developme nt and instruction across grade levels The ELS Literacy Initiative is a program that is rich in human resou rces. S everal people in the program have devoted all or part of their time to supporting teacher instruction and furthering the l iteracy i nitiative For instance the literacy coach has worked part time in the program two day s a week to support teachers in the ELS program. She has a master s degree in reading and she has worked in the program for ten years, first as a teacher and then as a consultant. Some of her responsibilities have include d supporting the implementation of the core curriculum, provi ding ideas and support for the use of supplemental curricular materials, supporting teachers in the use of the ELS Literacy Scope and Sequence and developing literacy plans for students. During the 2005 2006 and 2006 2007 school years, another consultant also provided instructional support to ELS teachers. While the l iteracy c oach has consulted with teachers specifically in the area of literacy instruction, the other consultant supported teachers in a wide range of instructional strategies with only a po rtion of her time being

PAGE 126

110 devoted to literacy instruction. The latter consultant is a former classroom teacher and she worked as a teacher consultant two days a week. The ELS Technology C onsultant served as a nother support person in the classroom. Since the 2003 2004 school year, she has worked in ELS two and a half days a week S ome of her responsibilities include helping teachers to identify appropriate instructional technologies for their students and then training them i n the implementation of those technologies. She has also worked closely with teachers and other staff in the program regarding the acquisition of new technology resources to further support the ELS Literacy Initiative During the 2006 2007 school year, the current program evaluator served as the ELS curriculum and i nstruction consultant one day a week. Due to budget constraints, this time was reduced to approximately one half day per week for the 2007 2008 school year. Some of the responsibilities associated with this role have inc luded lead ing the curriculum c ommittee, support ing the implementation of the literacy initiative, coordinating program wide benchmarking effort s, and continuing to identify best practices in the area of curriculum and instruction for the program. While th e evaluator took on t his role during the 2006 2007 school year, she had served the program unofficially in this capacity for several years. The evaluator has a background in school psychology and she has been with the ELS program for seven years. In addition to the people mentioned above who have time built into their schedules to dev ote to the support of the ELS Literacy Initiative the implementation and support of the initiative is also part of the roles and responsibilities of all ELS staff, parti cularly the classroom teachers and intervention s pecialists. Other human resources

PAGE 127

111 that support the initiative include professional development trainers who are not part of the ELS program ; one of these people provided a day long training to ELS certified staff in the area of literacy during the winter of the 2006 2007 school year. The financial resources involved in the implementation of the ELS Literacy Initiative are closely tied to the above mentioned material and human resources. The financial cost of these resources has a significant impact on the ELS budget. For example, the ELS Program Administrator estimated that during the 2006 2007 school year, curricular materials for the ELS literacy initiative made up approximately 30% of her materials budg et. This estimation is significant, especially when considering the fact that all specialized technology and equipment also comes from the same budget. Additionally, when the full time equivalency (FTE) for all of the people who are directly supporting th e literacy i nitiative is added together, it nearly equals a full time position that is being funded through the ELS program. Activities achieve the desired outcomes (Goldman & Schmalz, 2006). A variety of activities have been designed to achieve the desired results in literacy instruction and student outcomes for the ELS program (see Appendix A ) One of these activities has been a consistent message from the program administr ation on the importance of quality and comprehensive literacy instruction for all students. These types of messages have been overtly communicated through a variety of means including written communication to all staff (e.g., the beginning of the year we lcome letter) as well as through verbal messages delivered at large group gatherings (e.g., new staff trainings and staff

PAGE 128

112 meetings). The message that literacy is a priority for the program also has been communicated through other means, such as through t he purchasing of a core curriculum set for all primary and intermediate classrooms and through all program staff development days devoted to the topic of literacy. The dissemination of curricular materials (i.e., the ELS Literacy Scope and Sequence cor e curriculum materials, supplemental curriculum materials, and technology to support learning in the area of literacy) is another activity associated with the ELS Literacy Initiative This is an important component of the i nitiative, a ppropriate and research cited problems that the initiative was designed to address. Prior to the dissemination of these materials, none of the primary or intermediate level ELS teachers had an in classroom instructional materials were created by teachers to meet the needs of individual students, or they were supplemental instructional programs that were used unsystematical ly. With the dissemination of materials related to the initiative, teachers were provided with a core reading and language program that could be used with all students who were able to participate. T he expectation of the program administration was that a ll primary and intermediate teachers would begin to use the identified core curriculum (Language for Learning and Reading Mastery) during the 2006 2007 school year. In addition to the dissemination of the core curricular materials, other material resource s such as the ELS Literacy Scope and Sequence were also disseminated to support literacy instruction. The Scope and Sequence provided a framework upon which teachers could make instructional decisions regarding which supplemental programs

PAGE 129

113 could be used to instruct in a specific skill area and when these materials could be used, based on a created materials continue to make up a significant portion of classroom instructional materials. However, with access to the Scope and Sequence teachers c an make more informed decisions as to when and how to use these materials. One of the most important components of the ELS Literacy Initiative has been the professional development provided to staff. One of the primary problem statements beginning reading to students with moderate to severe developmental disabilities. Because i t is essential that classroom teachers and staff have an understanding of how reading develops professional development addressing this topic has occurred in several ways such as through training on the contents of the ELS Literacy Scope and Sequence This resource identifies four developmental stages of lit eracy in seven different skill areas and gives teachers information on what a learner looks like at each developmental level, illustrates how to use data to determine if a student falls within that level, and presents ideas for assisting a student in maste r ing the skills for that developmental level and mov ing on to the next level Professional development on the use of this resource has take n place in large groups (staff meetings), small groups (level meetings), and one on one settings (visits to teacher classrooms). In addition to receiving training specifically related to the use of the Scope and Sequence a full day of large group professional development devoted to how reading develops as well as instructional strategies for each of the developmental stages was provided to all certified staff during the 2006 2007 sc hool year. The trainer ha d extensive experience with teaching reading to students with

PAGE 130

114 moderate to severe disabilities and ha d provided professional development in this area for many years. It is important to note that much of the Scope and Sequence was based on prior work by this trainer Consequently, her message was consistent with other information that the certified staff had received and it served to reinforce those messages. In ad dition to having an understanding of how reading develops, teachers must have specific training related to the implementation of curricular materials. This is particularly the case with the core curriculum that has been identified in the ELS program. Mor e specifically, the language and reading programs are both Direct Instruction programs that have a very prescriptive instructional methodology. All teachers are given a full day of training on the implementation of these programs before they are used in t he classroom. Additionally, if someone else in the classroom (e.g., a teaching assistant or a speech and language pathologist ) will be supporting the delivery of the program, they must also attend the training. In order to support follow through and ensu re implementation in tegrity, the literacy coach conducts implementation integrity checks and provides suggestions for on site implementation. In order to help plan literacy instruction, teachers have been given access to different instructional planning to ols. These tools help teachers to develop comprehensive, individualized instructional plans for each student, and teachers have been encouraged to use the tool that best meets their needs. As part of the ELS Literacy Initiative teachers have been given training on how to use these planning tools. These tools have been introduced in large group settings, such as monthly staff meetings ; however, teachers receive more in depth training on an individual basis, such as through consultation with the literacy c oach Furthermore, other certified staff members in the

PAGE 131

115 classroom, namely the i ntervention s pecialist s, have received training on such planning tools and can support their use in the classroom. Another important activity of the l iteracy initiative is the emphasis on the us e of data to inform instruction. The use of data is not a new concept for cert ified staff in the ELS program. Most of the staff members are very knowledgeable when it comes to collecting data to inform progress toward IEP goal s Fu rthermore, one of the primary roles of the intervention s pecialist (typically a school psychologist by training) is to coordinate the collection and use of data in the classroom. Data are used in the ELS program to support literacy instruction in several different ways. First, every student is assessed three times a year using an early literacy or reading CBM tool. Teachers have been encouraged to review these data to ensure that individual students are making progress, and these data also have been use d as a program outcome measure. In addition to benchmarking students three times a year, intervention s pecialist s and some teachers have been trained in the use of CBM progress monitoring tools that can be used to monitor student progress more frequently. To support regular (i.e., weekly or bi weekly) progress monitoring of student progress, teachers and i ntervention s pecialist s have been provided access to an AIMS web progress monitoring account, a web based data management system that tracks and graphs s tudent data. Finally, the i ntervention s pecialist s in the classroom have received training on the administration and use of curriculum based a ssessment tools to analyze student reading and writing. More specifically, the training has consisted of (a) whe n to use the tools, (b) how to use the tools, and (c) how to use the information to support instructional planning.

PAGE 132

116 The final identified activity of the ELS Literacy Initiative is the assessment of literacy development across time and the monit oring of instructional prog ramming across grades. O ne of the primary problems identified by stakeholders that the l iteracy initiative was designed to correct involved disjointed and inconsistent reading instruction across time. For example, when a studen t would move from one teacher to the next, very little information would be communicated regarding the status of that reading development and what prior literacy instruction had involved for that student Consequently, teachers would essentially begin the process of determining appropriate instruction all over again. This resulted in loss of instructional time, inconsistencies in instructional methodology, and gaps In an attempt to correct this problem, a form was developed to be included in students files that identified their status along a developmental continuum in seven different skill last lesson completed (see App endix B: ELS Literacy Tracking Form) The form also The purpose of this l iteracy t racking f orm was to serve as a means of communication from one year to the next. Outcomes The desired outcomes for the ELS Literacy Initiative can be classified into three categories: short term outcomes (which are the direct and immediate result of the initiative activities and should be realized within one to two years of implementation), intermediate outcomes (which sh ould be achieved within two or three years of the initiative), and long term outcomes (changes in behavior or conditions that will be achieved in three or more years).

PAGE 133

117 Short term outcomes The ELS Literacy Initiative formally began during the 2005 2006 s chool year. Consequently, it would be expected that the program would have achieved the identified short term outcomes by this time (see Appendix A ) The identified short term outcomes include improvements in conditions to su pport literacy instruction (i .e. making resources available), changes in staff beliefs and skills (i.e ., improving teacher confidence), and changes in teacher instructional behavi or (i.e ., use of core curriculum creation of literacy plans for all students, use of data to make decisi ons). More specifically, the first identified short term outcome was to increase curricular resources to support research based literacy instruction in the classroom. In other words, it would be expected that every primary and intermediate classroom woul d have copies of the core literacy curriculum and would reflect the use of more research based supplemental instructional resources than it had used prior to the l iteracy i nitiative Furthermore, with the increased amount of resources, and with the profes sional development and individual consultation that teachers have had access to, it would be expected that teachers would feel more confident and supported as well as have improved skills in teaching beginning reading. The next three identified short ter m outcomes all relate to changes in teacher instructional beh aviors. More specifically, it was expected that as a result of ELS Literacy Initiative activities teachers would begin to use the core curriculum and supplemental materials, create individuali zed literacy plans for all students, and increase their use of data in instructional decision making. It is important to note that the ELS Literacy Initiative has not been prescriptive with regard to which supplemental instructional materials are to be us ed (with the exception of the core curriculum), how the

PAGE 134

118 literacy plans are developed, or when and how data should be used to inform instruction. However, the i nitiative has placed particular emphasis on research based instruction ; consequently it is impo rtant that teacher practice s reflect the basic assumptions underlying the i nitiative, such as that a comprehensive literacy plan include at least 5 big ideas relating to reading and include plans to incorporate literacy instruction across the day. Interm ediate outcomes The desired intermediate outcomes of the ELS Literacy Initiative are those that can be expected to be achieved between two to three years after the formal start of the i nitiative. Two of these outcomes would likely be achieved if all of the short term outcomes are achieved. In other words, if teachers were increasing their skill in teaching beginning reading and using the identified core instructional program as well as other i nitiative resources, it would be expected that there would be an increase in instructional consistency between same level classrooms. Additionally, if teachers were using more data to make instructional decisions, creating literacy plans, and tracking data across time, it would be expected that instructional consis tency would increase for students from one grade to the next. In contrast to the first two intermediate outcomes, the last two woul d not naturally be achieved if all of the short term outcomes were achieved, as they would require more direct action. Mor e specifically, one of these intermediate outcomes is an increase in inclusion and integration opportunities for students into their school communit ies as a result of the literacy initiative. Even if student literacy instruction was more aligned with rese arch based practices and students were experiencing improved academic gains i n the area of literacy increased integration and inclusion would require additional action on the

PAGE 135

119 part of the teacher. Similarly, it would be expected that improved articulation of student instructional programs and the use of data to inform instruction would set the stage for improved communication and collaboration between home and school in the area of literacy ; however, this would not occur automatically and would al so requir e additional action to be achieved. Long term outcomes The long term anticipated outcomes are the most important outcomes of the literacy initiative, and those are to ( a) improve student reading achievement, ( b) improve post school outcomes, and ( c) serv e as a model to the member districts in the area of literacy instruction. With regard to student achievement, the identified long term goal of the ELS Literacy Initiative is for students to graduate from the program reading and comprehending at the second grade level. However, even more important than attaining a second grade reading level is the long term desired outcome of improving post school outcomes. Post school outcomes fall into three categories : where and how students live their da ily lives, student employment or other work opportunities and experience s and student leisure and socialization opportunities. Literacy is a skill that has the possibility of improving outcomes in all three of these areas and improving post school outco mes is the ultimate goal of the ELS Literacy Initiative In addition to improving student outcomes, the final desired long term outcome of the literacy initiative is that the ELS program will serve as a model of research based reading instruction for stud ents with disabilities. This is an important outcome because one of the primary functions of the cooperative district is to increase the capacity of its member districts to better meet the needs of their own students.

PAGE 136

120 Assumptions The inputs, activitie s, and desired outcomes of the ELS Literacy Initiative are grounded in several assumptions, or philosophical beliefs. I f the program administration did not hold these beliefs and values, it would not have had a reason to initiate the literacy initiative For example, one of the assumptions of the i nitiative is that all students can benefit from instruction in the area of literacy. If this were not a belief of the pr ogram administration, then the i nitiative would not have the focus on all students that it does. Another assumption of the i nitiative is that when given appropriate and research based instruction, students with moderate to severe disabilities can learn to read. Finally, the program reflects the belief that literacy is a life skill that can im prove post school outcomes and is an important instructional component in the education of students with moderate to severe disabilities. Contextual Factors According to Coffman (1999) the contextual factors of a program are those that can potentially affect the outcomes of the program but may or may not be under your control. A number of contextual factors have the potential to impact the outcomes of the ELS Literacy Initiative either positively or negatively. One significant factor is the relatively high rate of staff turnover. For example, of seven ELS primary classrooms, only two of the teachers in those classrooms return ed for the 2007 2008 school year. The teachers who left the program did so for a variety of reasons (e.g., to stay home with ch ildren full or part time, to accept higher level teaching or administrative positions, or to pursu e other career opportunities). However, regardless the reason, it is obvious that

PAGE 137

121 such high rates of turnover can significantly affect the outcomes of the EL S Literacy Initiative S everal other significant contextual factors are important considerations for the implementation of the literacy initiative. O ne of them involves the student population that the ELS program serves. The program serves students wit h a wide variety of strengths, abilities, and needs which results in a very diverse student population. For example, in one classroom the students may include a student with a utism who is verbal and academically working at that grade level but who has significant behavioral challenges as well as a student who is nonverbal and severely physically and cognitively impaired. Additionally, the nature of the disabilities that these students have can affect their availability for learning on a daily basis (e .g., variations in health or behavior). O ther identified contextual factors for the ELS Literacy Initiative include programmatic funding and budgetary fluctuations, the presence of other program s or district initiatives, levels of parent participation and involvement, and the overall culture s and climate s of both the ELS program and the buildings in which the classrooms are located. Summary Like other school based initiatives, the ELS Literacy Initiative is a complex set of resources and activities that are designed to address several programmatic needs and ultimately improve student outcomes. This program description has identified the background and historical information necessary for understanding the context of the literacy initiative, and it has de scribed ( a) the problems that the i nitiative was designed to address, ( b) the inputs, or resources, that were invested in the i nitiative, ( c) the activities that took place as part of the implementation of the i nitiative, and ( d) the desired

PAGE 138

122 outcomes of th e initiative Additionally, the assumptions that underlie the i nitiative and contextual factors within which it functions were identified. Appendix A presents this program description in the form of a logic model, or a visual representation of the progra m. Evaluation Questions The purpose of this evaluation study was to gather information to aid in future decision making with regard to the ELS Literacy Initiative. Put in the context of the CIPP evaluation model (Stuffelbeam, 1971), the current evaluat ion could be considered both a process evaluation (e.g., How are the components of the ELS Literacy Initiative currently being implemented? What factors serve to facilitate implementation?) and a product evaluation (e.g., To what extent have student outcom es in the area of literacy been impacted as a result of the Literacy Initiative? To what extent do teachers believe there is instructional continuity for individual students as they move from one teacher to the next?). Considering both the process and th e product nature of the evaluation, along with the need to gather information to determine the next steps in the ELS Literacy Initiative, the following general goals were used to guide the evaluation study: 1. Process: To examine how the ELS Literacy Initiati ve was currently being implemented. 2. Product: To determine to what extent the anticipated short term and intermediate outcomes of the initiative were being realized. 3. Next Steps: To determine the next steps in implementation of the ELS Literacy Initiative

PAGE 139

123 Before beginning an evaluation study, it is important to identify and select evaluation questions so as to provide the direction and foundation of the study. The current evaluator utilized a two phase process for identifying the specific questions to be answered by this evaluation study (Worthen, Sanders, & Fitzpatrick, 1997). The first phase of this process was the divergent phase During the divergent phase, the evaluator develops a comprehensive list of potential evaluation questions gathered from mu ltiple sources, including stakeholders (Worthen, Sanders, & Fitzpatrick, 1997). For the current study, potential evaluation questions were drawn primarily from the initial stakeholder interviews, during which interviewees were specifically asked about que stions they would like to see answered by the evaluation. In addition to questions identified by stakeholders, the evaluator also included questions related to the program description/logic model, as well as questions of personal and professional interest According to Worthen, Sanders, and Fitzpatrick (1997), when the point has been reached at which no new questions are being identified, the next step in the process is to organize the questions into categories. During the divergent phase of question gen eration, a total of 93 potential evaluation questions were identified. The evaluator then analyzed the questions to determine the categories into which the questions could be grouped. Interestingly, the questions were grouped into categories that mirrore d the ELS Literacy Initiative Logic Model (see Appendix A ). The next phase of question generation is the convergent phase During this phase, the evaluation questions are prioritized and the final questions are identified (Worthen, Sanders, & Fitzpatrick 1997). To begin this phase of the process, the evaluator rewrote the questions developed during the divergent phase to reduce redundancies and to

PAGE 140

124 increase clarity. This step resulted in a total of 49 questions, about half of the total number of questio ns generated during the divergent phase. Finally, the evaluator met with the ELS program administrator, the client of the current evaluation study, to prioritize a final set of evaluation questions. Several factors were taken into consideration when prio ritizing the final list of evaluation questions. For example, questions that encompassed several other questions were prioritized. At least one question was prioritized per category in order to reflect the components of the logic model. Additionally, qu estions that were identified by multiple sources were prioritized. This prioritization proces s resulted in a final list of 14 evaluation questions ( see Appendix C : Evaluation Questions and Data Sources). The questions on the final list were then grouped under each of the identified goals of the study that were used to guide the current program evaluation. It is important to note that in program evaluation studies, the list of evaluation questions must remain flexible to allow new questions to emerge as n ecessary (Worthen, Sanders, & Fitzpatrick, 1997). Data Collection and Analysis Plan Evaluation studies are typically multi method in nature, including a mix of qualitative and quantitative data collection and analysis procedures and the current study wa s no exception. As has been previously described, the primary purpose of the current program evaluatio n study was to assist the ELS Program Administrator in identifying the next steps in the implementation of the ELS Literacy Initiative. The other goals of the evaluation were both process (i.e. to examine how the ELS Literacy Initiative currently is being implemented ) and product in nature ( i.e., to determine to what extent the anticipated short term and intermediate outcomes of the initiative were being realized )

PAGE 141

125 According to Worthen, Sanders, and Fitzpatrick (1997), process evaluations should ing alert to unanticipated ones, by obtaining specified information for programmed dec isions, by describing the actua l process, and by continually interacting with and observing the activities would be most appropriately achieved th rough qualitative methodologies such as ob serva tion and interviewing. In contrast and measuring outcome criteria, by collecting judgments of outcomes from stakeholders, 99). Therefore, quantitative measures are more appropriate for answering evaluation questions of this nature. The following paragraphs describe both the qualitative and quantitative data collection and analysis procedures that were used to answer the eva luation questions of the current study. The mixed method nature of the current evaluation study had several benefits, including (a) reducing the biases and limitations inherent in any data collection method, (b) allowing for conclusions and answers to eva luation questions to be based on the triangulation of data, and (c) allowing for data collected from one source to complement or enhance the information collected from another. The following data collection and analysis plan describes the instruments that were used in the revaluation, including how the instruments were developed, how they were used to gather data, and how the data were analyzed. The plan also describes the process that was used to analyze and interpret the qualitative data produced by the study, how the data were triangulated in

PAGE 142

126 order to make conclusions and answer the identified evaluation questions, and how the constructs of reliability and validity were addressed in the evaluation study. Instruments The multi method nature of the curre nt program evaluation required several methods for collecting data. The methods and instruments that were developed and used included (a) teacher surveys, (b) parent surveys, (c) focus groups, (d) student literacy tracking data, and (e) literacy benchmark ing data. The following paragraphs describe the purpose of these methods, how they were developed, the data collection process, and how the data were then analyzed. Teacher survey In order to help answer the identified evaluation questions, a survey w as given to primary and intermediate level teachers in the ELS program. The survey was designed to obtain a self report measure of teacher behaviors and opinions surrounding the implementation and outcomes of the ELS Literacy Initiative. Another purpose of the survey was to gain quantitative and qualitative data to triangulate with other data sources, such as the focus group and student outcome data. The survey was an efficient and effective way to get the anonymous opinions of as many teachers as possib le within a short period of time. The teacher survey was created by using the identified evaluation questions to develop the questions for the survey. The questions generated for the survey represented a combination of rating scale items and open ended questions. The survey questions were then transferred to the web based program, Survey Monkey. The online format of the survey was selected because it would allow survey participants to easily complete the survey questions online, using a link from an e mail requesting their participation. A

PAGE 143

127 cover letter was also developed for the survey, describing the purpose of the survey, The first page of the survey served a s informed consent. This pilot version of the teacher survey was reviewed by two members of the Next, the pilot survey was sent to the entire dissertation committee for approval. Once approved, two teachers completed pilot surveys. The first teacher was a former first grade teacher in the ELS program who served as a consultant for the program during the 2007 2008 school year. The second was a current middle school teacher in the ELS program who had been known to place a strong emphasis on literacy in her classroom. The two teachers were asked to comment on ease of completion, and time for completion, and to provide any other information or su ggestions for the survey that they had Pilot survey responses were screened to make sure that the respondents provided the information that the questions were intended to solicit. The teachers who completed the pilot surveys were also briefly interviewe d about their suggestions for changes. This process resulted in s mall changes being made to the teacher survey, including the addition of an NA option for several of the multiple choice questions, small changes in wording for two questions, and the reve rsal of the order of two questions. The final teacher survey consisted of a total of 31 questions and contained a variety of items, including multiple choice items, Likert scale items, and open ended items (see Appendix D ) The items were designed so th at knowledge of an exact definition of the l iteracy i nitiative (what the initiative include s does not include, etc ) was

PAGE 144

128 not necessary in order to answer the survey questions. A response was available for most of the questions and a ll questions were made optional. Based on feedback from the pilot surveys, the onl ine version of the survey took 20 30 minutes to complete. The online version was designed to be flexible so that teachers could start the survey and then later return to co mplete it. The evaluator attended the level meetings of the primary and intermediate teachers in order to personally explain the purpose of the survey, encourage participation, and answer any questions. Following these meetings, an e mail invitation to c omplete the survey was sent to all primary and intermediate teachers in the ELS program (n = 16) participation in the focus groups in order to reduce the possibility that participation in the focus group would influence their responses on the survey. Five days later, a reminder was sent to teachers to complete the survey prior to their participation in the focus groups. Twelve of a possible 16 surveys were completed prior to the original deadline. In an attempt to obtain a higher return rate, the deadline for completion was extended past the dates of the focus group to the end of the school year, with one more reminder being e mailed out during that time. This resulted in the completion of one additional survey, which brought the total return rate to 13 surveys completed (7 primary teachers and 6 intermediate teachers) out of a possible 16, or an 81% return rate. The analysis of the survey data began with a review of each survey in its entirety. The purpose of this review was to identify whether any of the participants were overly negative or overly positive in their responses, whether there were any obvious problems with the completion of the survey, or whether any other noteworthy patterns existed, such

PAGE 145

129 as a participant completing all of the multiple choice items but none of the open ended items. This review revealed one respondent who appeared to provide overly positive responses and another respondent who did not prov ide answers to any of the open ended responses were included in the final ana lysis; none were deemed unusable or invalid. The quantitative items on the survey were all treated as continuous data and were analyzed using means or frequencies. Data were analyzed separately for primary teachers and intermediate teachers and then we re analyzed together. This information has been summarized in table format and is reported in Chapter Four : Results The analysis of the qualitative items on the survey is described in a later section in this Qualitative Data Analysis Parent s urvey A survey was used to gather information from the parents of students in the primary and intermediate grades of the ELS program regarding their perceptions and opinions. More specifically, the survey asked parents their perceptions of the i home, their perspective on the communication between home and school regarding literacy instruction, and their opinions regarding what might be done to improve literacy outco mes for students in the ELS program. The survey was intended to be a simple and questions for the current evaluation study. The parent survey was developed by first identifying which of the program evaluation questions were appropriate to address through the parent survey, and

PAGE 146

130 dissertation committee. Once the survey was approved by the d issertation committee, it was piloted by a parent of a student in the ELS program who also serves in the role of parent mentor for the cooperative district in which the evaluation took place. The feedback from the parents on the pilot survey was all posit ive, and no changes were made. The final survey consisted of six total questions (two rating scale questions and four open ended questions) and was estimated to take approximately five minutes to complete (see Appendix E ) The first page of the survey ex plained the purpose of the survey, identified possible risks and benefits for participation, and served as the informed ensure that all participants shared an underst anding of this term. The online survey service, Survey Monkey, was used to format and collect survey responses. The invitation to complete the online survey was sent via e mail to the parents of children in the ELS program (in grades K 5 ) for whom the m ain office had an e mail address, representing a total of 81 e mails out of a possible 108 families. After the e mail invitation to complete the survey was sent, the evaluator received alert messages indicating that six e mails did not reach their intende d recipients. Parents were given a target completion date of two weeks after they had received the e mail invitation. A reminder was sent to parents to complete the survey one week after the first invitation was sent. Nine parent surveys had been comple ted prior to the reminder. When the survey solicitation period was closed, a total of 26 parents had completed the survey. Given that 81 invitations had been e mailed, 6 of those had been returned to the

PAGE 147

131 evaluator, and 26 participants ultimately complete d the survey, the survey response rate was 34.6%. The analysis of the parent survey data began with an examination of each survey to note any errors, problems, or patterns. The majority of parents responded to all six questions on the survey, even the op en ended questions. No other patterns or problems with the data occurred Frequencies were calculated for the two quantitative items on the parent survey. The analysis of the open ended questions on the survey is described in the section of this chapter Qualitative Data Analysis Focus groups Focus groups were used to gain information from teachers and relevant support personnel to generate qualitative data to triangulate with other data sources, such as the parent and teacher surveys as well as the student literacy data, and to help answer the identified evaluation questions. The focus groups served as an effective way to gain the opinions and comments of as many teachers and support staff as possible within a short period of time. Three foc us groups were conducted, including a primary level teacher group, an intermediate level teacher group, and a support staff group (e.g., teaching assistants, intervention specialists, speech and language pathologists, and instructional consultants). Focu s group interview guides were created to assist the evaluator in facilitating the discussions. One guide was created for use with the primary and intermediate teacher groups (see Appendix F ) and another for the support staff group (see Appendix G ) The i nterview guides contained both guiding questions and probing questions. The guiding questions were designed to stimulate conversation related to broad topics. Probing questions were used to follow up and lead the group into more specific topics that had

PAGE 148

132 not been addressed through the discussion stimulated by the broader guiding questions, and they were only used on an as needed basis. The guiding questions and probing questions had been developed to directly answer one or more of the program evaluation q ues tions The teacher focus group interview guide was pilot tested in a one on one interview format with the same two teachers who had pilot tested the teacher survey. Following the pilot interviews, the teachers were asked to provide specific feedback relating to the interview guide. Several changes were made to the interview guide based questions and express any concerns before beginning the focus group, (b) t he rewording of three questions, (c) the reversal of the order of two of the questions, and (d) the elimination of one question. The current program evaluator served as the facilitator during each of the three focus group meetings. The role of the facil itator was to lead the discussion by posing broad, guiding questions and then following up with the more specific, probing questions when necessary. The facilitator also ensured the participation of all members of each group, moderating the responses of o verly vocal participants and encouraging responses from reluctant participants. In addition to the facilitator, a note taker was also present during two of the focus groups (it was not possible to have a note taker present during the third group, so the f acilitator also served as the note taker during that interview ). While the focus groups were recorded using both a digital voice recorder and a traditional tape recorder, the note the note taker observed the group process and the dynamics among group members and

PAGE 149

133 between the facilitator and group members, including participation rates, nonverbal messages, and the reactions of participants. These notes were used to reflect on the potenti influenced the outcomes. The facilitator made several opening comments to the participants before beginning each of the focus groups. These comments included a reminder of the purpose of the discussion, an assurance of confidentiality on the part of the facilitator and note taker, a request that members of the group not share information outside of the meeting, the identification of possible risks and benefits to the partic ipants, and an opportunity to ask questions and express concerns. The facilitator also provided the participants with the broad definition of literacy adopted by the ELS program so that everyone shared this understanding before beginning the conversation. The participants were also reminded that they could choose not to participate and were asked to sign the informed consent form if they chose to participate. The primary level focus group took place at a location and time that the primary teachers agree d upon. A total of five out of a possible eight primary teachers were present for the focus group interview. Of those who did not attend, two teachers reported having prior commitments and one reported that she had not known she was supposed to attend th regular monthly meeting time and place, and consequently, all eight intermediate teachers were present for the interview. Participants for the support staff focus group were se lected with the assistance of the program administrator, and invitations to participate were sent to a range of different

PAGE 150

134 support staff members associated with the ELS program (teaching assistants, speech and language pathologists, intervention specialists and instructional consultants). Four support services personnel members participated: the reading coach, an intervention specialist, and two speech and language pathologists. Two t eaching a ssistants who had committed to participating were not able to a ttend at the last minute because their classrooms were short of staff that day. The t echnology c onsultant who also had confirmed attendance, had become preoccupied with another task at the time of the meeting and the focus group slipped her mind. lly, a third s peech and l anguage p athologist who was going to attempt to make it to the meeting encountered a last minute conflict as did a second i ntervention s pecialist. The analysis of the focus group data began with the transcription of the recordi ngs into a table. The evaluator listened to each of the recordings a second time and reviewed the transcription to ensure accuracy. During the focus group interviews and during the transcription process, the facilitator/evaluator took additional notes re lating to group dynamics, potential areas of bias, the identification of themes that kept emerging during the interviews, and which questions were omitted during the interview and why The remainder of the data analysis process for each of the focus grou ps is described below Qualitative Data Analysis Student literacy tracking data. Another data source that was used to help answer Literacy Tracking Form. These data were analyzed specifically to answer the evaluation question pertaining to student progress. The information from the literacy tracking forms

PAGE 151

135 was combined with other sources of data to help draw conclusions regarding student outcomes in the are a of literacy. A literacy tracking form is completed annually for each student in the program, following the student from year to year. The form contains several pieces of information, including (a) identifying information, (b) a rating of the studen (literacy beginner, literacy novice, early to upper emergent, or upper emergent to fluent) across seven different literacy skill areas for each school year, (c) identification of the benchmark scores. See Appendix B for a sample of a completed ELS Literacy Tracking Form. The purpose of literacy development tracking is to increase communication among relevant personnel regarding student literacy dev elopment and instruction from one year to the next. Without this type of information, teachers in the ELS program must frequently a new ELS skill level in the various components of literacy (phonemic awareness comprehension, etc.) and in identifying appropriate and effective instruction for that student. T he ELS p rogram a dministrator intended for the l iteracy t racking f orm to be used for evaluating program outcomes beginning in the 2007 2008 school year. ELS teachers were asked to begin using the literacy tracking form for the first time at the end of the 2006 2007 school year. The current evaluator helped to develop the literacy tracking form and was responsible for communicating with teachers about the form for primary teachers or intermediate teachers) to explain the form, why it was to be used,

PAGE 152

136 and how to complete it. The evaluator then directed teachers to the ELS Litera cy Scope and Sequence to assist them in making a determination of which developmental level ions, literacy benchmarking data, IEP goal progress data, permanent products, and any other relevant criteria. If they were not sure how to rate their students developmental level s they were directed to their i ntervention s pecialist who would help them to collect additional data to inform the rating s on the l iteracy t racking f orm s Teachers were then asked to retain cop ies of the form s for themselves that would transition with the student s and to send copies to the district office. Several reminders to complete the l iteracy t racking f orm were sent, and by the end of the school year, t racking f orms had been completed and s ubmitted to the district office from 11 of the 15 primary and intermediate teachers. This resulted in having l iteracy t racking f orm s for 72 out of a possible 101 primary and intermediate students ( a 71.3% return rate). One teacher who had complete d the forms had not rate d all students in all areas, which resulted in a variable total of i ntermediate students for some of the skill area s. The total N per skill area for i ntermediate students varied from 31 to 34. The 2 007 2008 school year was the second year when teachers were asked to complete the ELS literacy tracking f orm. As the c urriculum and i nstruction consultant for the program the current evaluator again attended the level meetings of the teachers to reintroduce the l iteracy t racking f orm and talk with teachers about the purpose of the form ( i.e., communication and outcomes) and expectations for completion ( i.e., one completed form for every student with a copy sent to the district office and a copy kept

PAGE 153

137 for records). Interestingly, many of the teachers reported that they had not receive d the l iteracy t racking f orm for students who were new to their classroom s th at year Given that the primary purpose of the tool is communication from one teacher to the next, this issue was of significant concern. It is possible that communication during the 2006 2007 school year (the first year the form had been used ) had provi ded inadequate information regarding what to do with the form once it was completed. For 2007 2008, teachers had the option to complete the l iteracy t racking f orm to from the previous year the teacher s were given new cop For those students who did not have a completed 2006 2007 literacy tracking for m, the teachers started a new form for the 2007 2008 school year. With several reminders sent to teachers to send the form into the district of fice, 11 out of 16 primary and intermediate teachers completed their forms and sent a copy to the district by th e end of the school year. This resulted in having l iteracy t racking f orms for 76 out of a possible 108 primary and intermediate students ( a 70.4% return rate). The analysis of the l iteracy t racking f orm data was limited to only students in grades K 5 (pr imary and intermediate students) during the 20 06 20 07 and 20 07 20 08 school years. Consequently, the 2006 2007 sample of students was not the same as the 2007 2008 sample because the students who were in the fifth grade during 20 06 20 07 were not included in the 20 07 20 08 sample as they were sixth graders during that year Similarly, a new cohort of k indergarten students was included in the 20 07 20 08 sample. Other differences include students who moved in and out of the program.

PAGE 154

138 The first step in the data analysis process was to calculate return rates (reported above) and to look for missing data. Missing data were caused by teachers who did not turn in their forms and one teacher who did turn in all her forms but failed to rate all of her students in all of the different skill areas in 20 06 20 07. This resulted in vari able totals of students at the i ntermediate level for each of the skill areas in the 20 06 20 07 data. The next step was to separately calculate the frequencies and percentages of students fa lling within each of the developmental levels for the 2006 2007 and 2007 2008 school years. Student growth across time was then examined. In order to have a more pure sam ple with which to examine growth trends, only students in grades K 5 who had complet e l iteracy t racking d ata for both 2006 2007 and 2007 2008 were included in the analysis Fifty students met these criteria. The reason that other data may not have been available for some students for both years include : ( 1) they were a k indergartener in 2007 2008, ( 2) they were a fifth grade student in 2006 2007, ( 3) they enter ed the program in 2007 2008, or ( 4) they moved out of the program for 2007 2008. The final step in the analysis of these data w as to look at individual students growth across tim e To examine individual student growth, the number of skill areas in which students were rated to have growth at least one developmental level from one year to the next was calculated. The results of the analyses of the literacy tracking forms are descr ibed in Chapter Four Literacy benchmarking data. Some form of literacy benchmarking (the assessment of student skills three times a year using a form of curriculum based measurement) has been taking place in the ELS program since the 2002 2003 school y ear, with the exception of the 2003 2004 school year. The tools and data collection

PAGE 155

139 procedures used to collect these data have changed over the years in an attempt to improve the benchmarking system and to meet the needs of this unique population. Howeve r, the benchmark data collection system was consistent from the 2005 2006 school year to the 2007 2008 school year. The purpose of this benchmarking process has been two fold. The primary purpose of benchmarking student literacy skills has been to ensu re that students are making consistent growth in the area of literacy, and to indicate when instructional changes may be necessary. Whether or not teachers and other staff were using these data for this purpose was a separate evaluation question that was addressed through other data collection methods (teacher survey and focus groups). The second purpose of ELS literacy benchmarking has been program evaluation, or to determine how students were achieving overall in the area of literacy. The present progr am evaluation examined the use of the literacy benchmark data for this purpose, in order to help answer the evaluation question regarding overall impact of literacy instruction on student achievement. Because of the nature of the student population in t he ELS program, the literacy benchmarking process had both similarities and differences with the benchmarking tools and processes that are used with more traditional, general education populations. The similarities related to the frequency of data collect ion and the standardized administration of the assessments. Just as in more typical settings, students in the ELS program participated in the benchmarking process three times a year: fall, winter, and spring. However, because of other conflicting activit ies in the ELS program (e.g., participation in the state mandated alternate assessment program), the benchmarking windows (the time frame within which they had to be completed) varied slightly as to when they took place.

PAGE 156

140 These windows have traditionally be en longer in the ELS program (three weeks) than in other settings (two weeks). Just as in more traditional settings, the administration of the benchmarking assessments was standardized. All intervention specialists those responsible for the collection of these data in the ELS program were trained in the administration of the benchmarking tools and provided a standard set of directions for the administration of each of the tools. The differences in the ELS benchmarking process from more traditional bench marking processes include (a) the range of tools that were available, (b) the selection of tools, (c) the characteristics of those tools, and (d) the administration formats available for each tool. First, the tools available were intended to measure a wid er range of literacy skills, namely the pre literacy skills that students typically acquire prior to kindergarten. For example, some students were assessed in picture naming vocabulary and others in concepts of print, both of which involve some of the ear liest measurable literacy skills. This was necessary because many of the ELS students were in lower levels of literacy development. Second, tools were selected for use based on the his or her grade level, as is the case in more traditional benchmarking processes. For example, a student in the fifth grade may have been benchmarked using a tool that measured letter identification because that is what students in the fifth grade were currently learning. For a complete list of available benchmarking tools, see Table 1 The characteristics of the tools themselves also differed from the tools used in a more traditional benchmarking process. The most significant difference is that the t ools used in the ELS program were accuracy based rather than fluency based. Assessment

PAGE 157

141 tools used in the benchmarking process typically come from the curriculum based measurement (CBM) family of tools, and one of the primary characteristics of this famil y of assessments is that they measure student fluency in any given skill area. One of the benefits of measuring fluency over accuracy is that fluency reflects a higher level of mastery of a skill. That is, a student can be accurate in a skill but not hav e mastered it because he or she is not fluent in it. Another benefit is that fluency measures are more sensitive to growth over time. When measuring student progress three times a year, it is necessary that the tool used be sensitive to small amounts of growth. When ELS first began benchmarking student literacy skills, the traditional fluency based CBM tools were used. Experience with these measures suggested that, because of the processing and motoric difficulties of many of the students in the program the fluency based measures did not serve as accurate reflections of their knowledge and mastery of skills. Additionally, because of these complications, students showed very little growth over time. Therefore, during the 2005 2006 school year, the move was made to alter the benchmarking tools to make them accuracy based rather than fluency based measures. The exception to this rule was the Reading CBM assessment tool, for which the only option for administration was fluency based (i.e., counting the num ber of words read correctly per minute). It was not possible to make this assessment tool accuracy based while still maintaining the integrity of the assessment. The final difference between the ELS literacy benchmarking tools and more traditional liter acy benchmarking tools related to the availability of multiple administration formats. Most of the tools could be administered expressively (the traditional format), receptively, or in a format that was significantly altered from

PAGE 158

142 standardized administrati on. This was necessary because of the nature of the ELS student population. For example, many of the students in ELS are nonverbal and would not be able to participate in the assessment if it were not adapted to accommodate nonverbal responses. However, even the nonverbal administration of any given tool was standardized. In most cases, the tool was made nonverbal by simply allowing students to respond from a field of four possible answers. Unfortunately, some students in the ELS program were not able to participate even in an adapted standardized administration of the assessment tools for a variety of reasons; for example, they could only respond from a field of two possible answers, or by using an eye gaze. For these students, an administration forma t was used that was considered significantly altered from the standardized administration. During the 2006 2007 school year, 18% of ELS students received a significantly altered version of the standardized administration of a benchmarking tool. Analysis of the student literacy benchmark data began by organizing the data and calculating participation rates by year, level, and assessment tool. See Table 1 for the 2007 2008 literacy benchmark assessment participation rates. An examination of these data re vealed low numbers of students being assessed with any given assessment tool, even when considering both prim ary and intermediate students and all administration formats combined. For example, when both academic levels were combined and all administration formats were included, the largest number of students being assessed with any given tool was in Letter Sound Identification, with 21 students being assessed using this benchmark assessment tool during the 2007 2008 school year. The assessment tool

PAGE 159

143 with t he next highest participation rate was Picture Naming with 11 students being assessed using this tool in 2007 2008. The low numbers of students assessed within any given assessment tool and administration format posed serious limitations to the ability of the evaluator to use these data to draw conclusions regarding student literacy development in the ELS program across time. Several other problems with this data set also limited the usability of the data to answer questions about student outcomes in th e program. More specifically, because the benchmark measures were accuracy based and not fluency based, the tools were not as sensitive to growth over time as the more traditional versions. In addition, the tools have a ceiling of performance (100% accur acy), whereas fluency measures have no such ceiling, which again limits their sensitivity to growth over time. Finally, because the tools had been altered from their original format, there was no normative information to which student performance on the t ool could be compared. Consequently, it was

PAGE 160

144

PAGE 161

145 Given the limitations of the data set for drawing conclusions regarding the effects of the ELS Literacy Initiative on the literacy outcomes of students in the program, the analysis of these data were limited to the examination of participation rates during the 2007 2008 school year. These data were not used to answer the evaluation question regarding the impact of the E LS Literacy Initiative on student literacy development. However, the information on 2007 2008 participation rates were used to help answer the evaluation question regarding how literacy benchmark data are being utilized in the program and whether the data are adequate to support those uses. The results of this analysis are reported in Chapter Four. Qualitative D ata A nalysi s Several instruments used in the current program evaluation generated data that were considered to be qualitative, including the open ended questions contained in the teacher and parent surveys and the transcripts that resulted from the three focus group interviews. This section describes the analysis procedures employed for those qualitative data. The primary activity in the analysis of qualitative data is the coding of the data. Data codes are the words phrases and numbers assigned to individual bits of data that help to organize the information ( Miles & Huberman, 1994). Data coding typically occurs at two levels. The first level of data coding (Level 1 coding) includes the identifying information about the data, or the when, where, who, and other facts about the data that were coded. The second level of data coding (Level 2 coding) involves the interpretive constructs related to analysis. Level 2 codes describe the analysis of the data, or the data themes.

PAGE 162

146 The analysis of the qualitative data set began with the open ended responses on the teacher and parent surveys. To support the analysis, the data were formatted into a tabl e in which each teacher and parent comment appeared in a separate row, with columns used to code the data and insert comments. The next step in the process was to develop a set of categories that could be used to fully describe and encompass the data set (Level 1 codes). Miles and Huberman (1994) recommend creating a provisional starting list of codes prior to beginning analysis. This provisional list includes descriptive, Level 1 codes that are designed to encompass and identify what is in the data set. The evaluator created a provisional list of Level 1 codes prior to beginning the coding process based on the ELS Literacy Initiative Scope and Sequence and the evaluation questions. This resulted in two broad categories under which Level 1 codes were id entified: Implementation Level 1 codes and Outcome Level 1 codes. Subcategories of Level 1 which is applied to comments that refer to human resources (PEOP), particularly tea ching assistants (TA), and describe their role in the implementation of literacy instruction in the classroom (DES). A total of 47 Level 1 codes were included on the provisional start list. A draft definition was created for each of the Level 1 codes bef ore the coding of the qualitative survey data began. The next step in the process was to try to apply the Level 1 codes to teacher responses to the open ended questions on the teacher survey. If a teacher response contained more than one idea and requi red more than one Level 1 code, or if it was only part of the response that was to be given a Level 1 code, the text of the portion of the response that inspired the code was changed to another color and the text of the

PAGE 163

147 corresponding Level 1 code was also changed to that color so it would be clear which part of the response went with which code. While coding the comments made in the survey the evaluator also added comments e comments regarding decisions that had been made and thoughts about themes that were running through the data, in addition to other personal reflections, while provid ing a context in which to understand a particular teacher response. The process of appl ying the Level 1 codes to the teacher comments on the surveys resulted in seven new Level 1 codes being added to the initial start list of Level 1 codes. This addition accommodate d responses that did not fit the existing code structure and it had been an expected part of the coding process. The next step in the process was to try to use the new, revised list of Level 1 codes to code the focus group data. The focus group transcriptions were formatted similarly to the data in the surveys, which meant that every comment made during the focus group was assigned an individual cell in a table and additional columns were created for the Level 1 codes and reflective remarks. The first group to be given Level 1 codes was the support staff focus group. F our more additional Level 1 codes were added during this process because the existing codes did not adequately describe the data Furthermore, while coding the support staff focus group, the examiner noted that several codes appeared to describe the same data and were not different enough from one another. To address this concern, several of the Level 1 codes that referred to instructional planning and the use of data to make instructional decisions (eight codes total) were collapsed into one set of instructional planning codes (three codes total). Several of the definitions of the Level 1 codes were also modified to

PAGE 164

148 be more accurate and descriptive. The result was the final set of Level 1 codes (56 total) and their corresponding definitions. Before continuing to code new data, the evaluator returned to previously coded data and applied the finalized set of Level 1 codes and definitions. The Level 1 coding process was finalized when the remaining focus group data were given Level 1 codes. In order to support t he next step in the analysis of the qualitative data, all data that were given Level 1 codes were transferred to a spreadsheet that included columns for identifying the data source, the data, the Level 1 code, and a Level 2 code. This formatting allowed t he evaluator to have all of the qualitative data in one place regardless of the data source and facilitated the manipulation of the data when applying Level 2 codes as well as when using the data in answering the evaluation questions. After all of the qu alitative data had been given Level 1 descriptive codes, the next step was to develop and apply Level 2 analysis codes. The purpose of Level 2 codes is to identify the themes that emerge out of the data set and to begin to develop a deeper analysis of the data. An initial list of potential Level 2 codes was developed by of the qualitative data sets. Examples of themes that were identified as reflective remarks includ themes were listed in a separate document (107 total), many of the themes had been listed more than once, or they were listed slightly differently but reflected the same idea, and many of the themes could be grouped under one broader category. The evaluator began collapsing this list by grouping the initial set of themes into broader ca tegories and

PAGE 165

149 possible Level 2 codes. This resulted in the identification of an initial set of 15 Level 2 codes. The next step for the evaluator was to begin to try to apply the Level 2 codes to the qualitative data set. The data set was sorted alphabeti cally from A to Z so that the data would not be listed by data source or Level 1 code, both of which could have biased the Level 2 coding process. After processing 485 pieces of data (there were a total of 609 pieces of qualitative data in the entire set) the evaluator reflected on the application of the Level 2 codes. This reflection process included examining how many pieces of data had been assigned each Level 2 code in order to identify whether a particular code was too broad or too narrow. Addition ally, all of the data that had been given a particular Level 2 code were grouped together to determine if the data appeared to be cohesive in content. The reflection process also included making sure the data that were included within any given Level 2 co de reflected a variety of data sources and Level 1 codes. This was done to ensure that the Level 2 code went beyond the simple description of data and moved toward analysis of the data. Finally, the evaluator examined the data that had not been given a L evel 2 code to determine whether a possible theme, or Level 2 code, had been missed. The Level 2 coding and reflection process resulted in several of the Level 2 codes being combined into one Level 2 code. For example, the initial Level 2 codes of definitions of some of the Level 2 codes were modified to better reflect the group of data that were captured within that Level 2 code.

PAGE 166

150 After applying the Level 2 codes to the 485 pieces of data and making changes to the Level 2 codes and their definitions, the result was a set of 11 Level 2 codes. Furthermore, a total of 246 of 485 pieces of data r eceived a Level 2 code, representing 50.7 % of the data Much of the data that had not receive d a L evel 2 code were simply desc riptive and had been covered within the L evel 1 codes. These often came from the surveys that asked questi is the role of teaching assistants in providing literacy instruction The next step in the qualitative data analysis process was to determine if another analyst would reliably ap ply the Level 2 codes to the same data set. In order to determine this, a stratified random sample of 50 pieces of the qualitative data was taken from the 246 pieces of data to which the evaluator had assigned Level 2 codes. This process consisted of sor ting all of the data by Level 2 categories, assigning each piece of data a number of one through five, rolling a die to pick a random number (which was two) and then pulling all data that were assigned that number for the reliability check. The pieces of data from any given Level 2 category ranged from three to eight. Because about half of the qualitative data set had not been assigned Level 2 codes, it was important in the reliability check to include some data that had not been assigned a Level 2 code. To select these data, the 250 pieces of data that were not assigned Level 2 codes were each given a number of 1 50. The evaluator asked another person to randomly select a number between 1 and 50, and that person selected the number 29. This resulted in 5 additional pieces of data being identified for the reliability check, resulting in the inclusion of a total of 55 pieces of data.

PAGE 167

151 The person who volunteered to perform the reliability check was a former intervention specialist in the ELS program, who currently serves as an autism consultant to the program. She was not involved in any of the data collection efforts for the current program evaluation. She was provided with the 55 pieces of data to be coded and a list of Level 2 codes with corresponding definitions. The target level of reliability was 80% agreement and the first time that she applied the codes resulted in an agreement level of 60%. In order to increase the reliability with which the codes could be applied, the evaluator revised the def initions of several of the Level 2. Additionally, examination of the data suggested that on a few occasions, the evaluator agreed that code that was given by the person doing the reliability check was more appropriate than the one that she had given and t he Level 2 code was changed accordingly. With these changes, the second attempt at applying the Level 2 codes resulted in a 78% agreement level. The qualitative data have been used to aid in answering the identified evaluation questions in several ways. First, the qualitative data were identified by data source. Because the questions on the surveys and the questions posed during the focus groups were specifically designed to solicit answers to the evaluation questions, sorting the data by source and ex amining the responses to these questions as a whole was one way to use the data in answering the evaluation questions. Next, the data were examined according to Level 1 descriptive codes. Sorting the data by Level 1 descriptive codes allowed the evaluato r to examine all of the qualitative data related to a specific topic. For example,

PAGE 168

152 By sorting data according to Level 1 codes, the evaluator was able to see all comments that related to a specific topic, regardless of data source. Finally, data w ere examined by Level 2 analysis codes. The Level 2 codes were important themes that emerged from the data set and could be used to help in answering the identified evaluation questions at a more analytical level than when the data were sorted by data sou rce or Level 1 codes. The results of the qualitative data analysis are reported in Chapter Four Data Triangulation and Interpretation Three main sources of data were included in the current program evaluation: teacher and parent surveys, teacher and support staff focus groups, and student literacy data. The data from each of these sources were analyzed using the techniques appropriate for the type of data as described above in the section titled Instruments Once each of the data sources had been analyzed separately, the next step in the program evaluation process was to combine all of the information to answer the identified evaluation questions. An important component of this analysis was the triangulation of the data. According to Worthen, Fit examining the consistency of results from different sources and methods for measuring improved confidence and increased validity in study outcomes. Also, drawing conclusions based on multiple sources of data reduces the influence of the biases and limitations of any one data source. It is important that consistencies as well as inconsistencies be identified through the tr iangulation process and that the inconsistencies be further explored so differences may be explained (Worthen, Fitzpatrick, and Sanders,

PAGE 169

153 1997). When multiple data sources could be used to answer an identified evaluation question, the evaluator identified similarities and differences in how those data informed the answer to the evaluation question. The triangulation of the data is described according to each evaluation question in Chapter Four It is important to note that the results of the evaluation that are reported in Chapter Four are the interpretations of the data by the current program evaluator. However, when results and interpret findings in isolation. Stakeholde r involvement in this process is considered imperative (Worthen, Fitzpatrick, and Sanders, 1997). Bringing stakeholders together to discuss their interpretations of the data collected through the evaluation to be comprehensive in their scope and are used not only to help interpret the data, but to determine the implications of those interpretations as well. While it was not possible to hold stakeholder meetings prior to the interpretation of the results and the development of the recommendations that are presented in the current evaluation study, the evaluator determine the implicat ions of the results for the initiative, as well as to gain stakeholder perspectives regarding the next steps for the initiative. Reliability and Validity The Joint Committee on Standards for Evaluation (1994) adopted standards related to the evaluation of educational programs. Two of these standards relate to the validity and reliability of the information produced by program evaluations. More gathering

PAGE 170

154 procedures should be ch osen or developed and then implemented so that they will assure gathering procedures should be chosen or dev eloped and then implemented so that they will assure that the information how the concepts of reliability and validity have been addressed in the current program evaluation study. Validity There are two types of validity that are pertinent to the current evaluation study: internal validity and external validity. Internal validity is defined as the 98, p. 201). In other studied? External validity is different from internal validity in that it is concerned with whether research findings can be generalized to other cases. The program evaluation design used in the current study, and the qualitative nature of a large portion of the measures that were used, can be considered strengths when it comes to increasing the internal validity of the study. More specifically, b ecause measured, qualitative research can be considered to be at an advantage when interpreting that reality (Merriam, 1998). Other considerations with regard to internal v alidity include (a) how instruments were developed, (b) how decisions were made throughout the evaluation process, and (c) the extent to which alternative explanations for the results have been explored.

PAGE 171

155 Two instruments were developed to help in answeri ng the evaluation questions in the current study: the teacher and parent surveys and the focus group interview guides. One consideration in the development and use of these instruments was construct validity. Construct validity is defined as the degree t o which the instrument measures the theoretical construct it is intended to measure. Because the instruments were intended to measure the implementation and outcomes of the ELS Literacy Initiative, and because both the surveys and focus group interview gu ides were developed using the logic model of the ELS Literacy Initiative which defines the inputs, activities, and outcomes of the i nitiative, there can be some confidence that the instruments reflect the construct that they were intended to measure. The other consideration in the development of these instruments is content validity, or the extent to which an instrument reflects the specific domain of the content. The other source of information that was used to develop the surveys and focus group interv iew guides was the evaluation questions of the study The evaluation questions for the current program evaluation study were developed through a series of interviews with stakeholders in the l iteracy i nitiative. The instruments were then designed specifi cally to answer these evaluation qu estions. This process increased confidence that the surveys and focus group interview guides ha d strong content validity. It was important for the decision making process that the evaluator made every step of the ev alua tion process transparent and overt Gall, Gall, and Borg (2006) process of e xplicitly, and with great detail, identifying the process used and the decisions made in the collection and analysis of the data so that an external reviewer could follow

PAGE 172

156 the path and have a clear understanding of how the researcher came to the conclusions that he or she did. The evaluator of the current program evaluation maintained a detailed audit trail that contained the date, the audit trail entry, a short description of the audit trail entry, and the evidence that supported the entry. This program e valuation audit trail enhanced the internal validity of the current study. Another way to improve the internal validity of an evaluation study is to ensure that any alternative explanations for the results of the study have been explored. This can b e done through several means, including triangulating data, addressing researcher bias, and implementing member checks. Triangulation refers to comparing the results of ho listic understanding of what is being studied and supports conclusions that better as possible (i.e., quantitative items on surveys, open ended items on surveys, fo cus group data, Level 1 codes, Level 2 codes, and student literacy development data) have been used to answer the identified evaluation questions. When differences existed in the conclusions from the various data sources, those differences were identified and explored. It is also important to explicitly address researcher bias to ensure that the results of the study reflect reality. For the current evaluation study, researcher bias was addressed by having a second observer attend the focus groups to rec ord notes on group dynamics and the behavior of the facilitator, by explicitly identifying the biases held by the evaluator (identified in Chapter Five), and by the evaluator reflecting on and making note of how her views and behavior may have impacted how data were collected and

PAGE 173

157 that she recorded throughout the evaluation process. The fi nal way that the current evaluation study addressed internal validity was through member checks, or taking the data and the conclusions that were drawn back to the stakeholders of the evaluation to obtain their input. Unfortunately, conducting member chec ks was not possible before the data were analyzed and tentative conclusions were drawn, as outlined in Chapters Three and Four However, it has been planned that this part of the evaluation process will occur when the evaluator returns to the program afte r her leave. External validity is different from internal validity in that it is concerned with whether research findings can be generalized to other cases. The conceptualization of external validity relative to program evaluation studies is very differen t when compared to external validity as it relates to quantitative studies. In fact, many qualitative researchers reject the notion of external validity as applicable to evaluation research. Because the primary purpose of the current program evaluation s tudy is to determine the next steps in the implementation of the ELS Literacy Initiative, it is important to note that the data and results generalize to all of the classrooms within the ELS program. In addition, because of the high return rates on the su rveys and the high participation rates in the focus groups, the evaluator was confident that the data that had been collected could be generalized to the primary and intermediate classrooms in the ELS program as a whole.

PAGE 174

158 Reliability The traditional notio n of reliability involves the degree to which research findings can be replicated. This traditional notion of reliability applies to the current evaluation study in two ways. The first is with regard to inter rater reliability and the application of the Level 2 codes. In order to reduce researcher bias and increase the strength of the conclusions that were drawn from the qualitative data, a second rater was asked to apply the Level 2 analysis codes to a stratified random sample of qualitative data. This process ultimately resulted in an agreement level of 78% between the evaluator and a second reviewer The other way in which reliability applies relates to a Teacher s were directed to use the descriptions of each skill developmental level contained in the ELS Literacy Scope and Sequence along with other data, such as the literacy benchmark data and permanent products, to make their judgments of student literacy develo pment; however, it cannot be determined how consistently this was done. Since it is difficult to draw conclusions regarding the reliability of th e data on the literacy tracking form, conclusions were interpreted cautiously and, like all other data sourc es in the current evaluation, the findings were interpreted in the context of other data sources In qualitative inquiry, i t is not so important that the results of a study have the ability to be replicated, as that the conclusions drawn are appropriate b ased up on the data that were collected. In other words, the conclusions must be consistent with the data. This notion of reliability can be enhanced by focus groups through triangulating the data and leaving an audit trail (Merriam, 1998). The triangul ation of data refers to the use of multiple methods to collect and analyze data and it was previously discussed as a means of enhancing interna l validity. An audit trail is the detailed documentation of the

PAGE 175

159 activities of the research including how data were collected, how categories were identified, and how decisions were made. Limitations Several limitations are inherent in the present program evaluation study. The areas in which these limitations may be grouped are: (a) contextual limitations, (b) d esign limitations, and (c) impact limitations. Contextual L imitations Contextual limitations refer to those limitations that are related to the conditions surrounding the initiation and completion of the evaluation study. For example, program evaluati ons are typically initiated by a client who seeks the support of an evaluator to help answer questions about a specific program or initiative. In the case of the current evaluation study, it was not the client (the ELS program administrator) who initiated the evaluation study, but rather, the evaluator. The evaluator had significant involvement with the implementation of the ELS Literacy Initiative and was genuinely interested in helping the client to determine the appropriate next steps in the program. Another potential contextual limitation to the current study was the fact that the evaluator was a staff member of the program that was being evaluated and therefore was internal evaluator are advantages to both internal and external evaluators. External evaluators are more likely to be impartial and provide a fresh, outside perspective. External evaluators also may have more credibility with outside audiences and potentially more experti se in evaluation. Finally, because external evaluators are not a part of the program they are evaluating, they can be more straightforward and honest when necessary. There are also benefits to

PAGE 176

160 evaluators being internal to the program being studied. Inte rnal evaluators have more knowledge of the history of the program as well as more familiarity with the stakeholders and their interests, concerns, and potential hidden agendas. Because of this familiarity, the start up time for the evaluation may be short er than it is when an evaluator is external to a program. Finally, the internal evaluator can help to support the program in using the results of the evaluation to drive practice, helping to implement the recommended changes. According to Gall, Gall, & B by an internal evaluator, especially when the evaluation findings will be used to guide Thus, the fact that the evaluator was internal to the program was n ot considered a significant limitation. Design L imitations One of the limitations that needed to be addressed throughout the data collection and analysis process was that of researcher bias Because the evaluator was a part of the program being evaluate d and was actively involved in leadership of the implementation of the ELS Literacy Initiative, the potential for personal bias to influence the data collection and analysis process was strong. The first step in addressing this limitation was to overtly a cknowledge that the limitation existed and to make active attempts to recognize when personal bias may have influenced the process. It was important for the evaluator to overtly identify her beliefs and theoretical assumptions as well as to note how her v iews might have biased the results of the study. Other strategies that were used to address potential personal bias included actively testing alternative hypotheses for study findings, triangulating the data, reporting the results of the study in great d

PAGE 177

161 collection to the conclusions that were drawn from those data. These strategies for reducing the potential for personal bias also increased the validity of the evaluation findings. Another design limitation was the inability to obtain a direct assessment of student literacy skills over time to inform the evaluation of whether the ELS Literacy Initiative had an impact on student literacy outcomes. The only direct assessment of ELS y skills involved the collection of benchmarking data. However, as of students were assessed in any given literacy benchmarking tool with the same administration form at to draw conclusions regarding student progress as a group over time. The other measure of student literacy development, the ELS student literacy tracking f orm, is not a direct measure of student skill and instead serves as an indication conclusions regarding student literacy development could be drawn. Impact L imitations The primar y purpose of this program evaluation study was to support the administrator and stakeholders of the ELS program in identifying the appropriate next steps in the implementation of the ELS Literacy Initiative. Therefore, the ability to generalize the result s of this study to other programs was limited. However, the methods employed and the results of the data analyses were presented in great detail so that readers might be able to consider potential implications for their own situations. While the results of this study may not have a significant impact on practice in other programs,

PAGE 178

162 the evaluation will have a significant impact on practice within the ELS program. Because the evaluator is internal to the program, she can support the interpretation and use o f the data, as well as outcomes of the evaluation. Unfortunately, one limitation to the evaluation study was that the evaluator was not able to meet with stakeholders of the initiative to review the results of the study and to discuss the implications for the program. Therefore, the results and discussion contained in this report represent the interpretations and opinions of the evaluator. The evaluator does intend to meet with a stakeholder group to discuss the results and implications as soon as possib le. Dissemination of Information How the results of a program evaluation will be shared with the client and with stakeholders of the evaluation is an important consideration. This is even more important for the current program evaluation, given that the primary goal for the evaluation study was to identify the appropriate next steps for the implementation of the ELS Literacy Initiative. The sharing of the results of the evaluation will take several forms. The program administrator will receive a written summary of the methods used for collecting the data and a summary of the outcomes. The outcomes of the evaluation will be shared with additional program stakeholders, most of whom will have been involved with the evaluation from the beginning, through pr esentation format and a brief summary document. Other members of the ELS program outside of this stakeholder group will receive the information in a more abbreviated format. It is ultimately the responsibility of the client (the ELS Program Administrator ) to determine if and how the evaluation results will be shared. The evaluator will work closely with the client to determine

PAGE 179

163 appropriate evaluation audiences and potential strategies for dissemination of the information.

PAGE 180

164 CHAPTER FOUR RESULTS There were three broad goals of current program evaluation study: (1) to examine how the ELS Literacy Initiative was being implemented, (2) to determine to what extent the anticipated short term and intermediate outcomes of the initiative were realized, and (3 ) to determine the next steps in the implementation of the ELS Literacy Initiative. Fourteen evaluation questions had been identified for the current study, and these questions had been designed to meet the three goals of the program evaluation. The firs t 13 of the evaluation questions related specifically to the implementation and outcomes of the ELS Literacy Initiative and are answered in this chapter, which has been organized by eps in the ELS Litera : Discussion. The evaluation questions were developed through a process that involved stakeholders in the ELS program. Several sources of data were used to answer the evaluation questions, in cluding surveys given to teachers and parents; focus groups with primary teachers, intermediate teachers, and support staff; and student lite racy development tracking data. The results of those data collection efforts are described below when they can hel p to answer an identified evaluation question. When possible, the results of several data sources are triangulated to answer the evaluation question. Similarities and differences in the findings are discussed.

PAGE 181

165 Implementation Evaluation Questions T hre e evaluation questions are related to the implementation of the ELS Literacy Initiative. The first question seeks to gain an understanding of what literacy instruction currently looks like in the ELS program, and the information contained within this eval uation question is designed to provide a rich description of current literacy instruction based on data that were collected as part of the program evaluation. The other two implementation related evaluation questions ask what factors appear to facilitate the implementation of the ELS Literacy Initiative (Question 2), and conversely, what factors serve to facilitate the implementation of the ELS Literacy Initiative (Question 3). The results of these two evaluation questions have been combined so as to redu ce overlap and redundancy in reporting. Question 1: How A re the Components of the ELS Literacy Initiative Currently Being Implemented? The ELS Literacy Initiative began during the 2005 2006 school year and was designed to improve literacy instruction in ELS classrooms. Significant material, people, and financial resources have been directed toward the initiative. More specifically, a core curriculum meant to address reading and language was purchased for every primary and intermediate classroom. The EL S L iteracy S cope and Sequence was developed to address the unique needs of students in the ELS program. The S cope and S equence also included assessment and instructional strategies. Other supplementary instructional materials have also been ordered for t he program. In terms of human resources, professional development in the area of literacy has been made available to staff, and several consultants in the program are responsible for supporting teachers in implementation

PAGE 182

166 through coaching and consultation. With all of these supports, it is expected that teachers are providing comprehensive literacy instruction that is aligned with best practices. It is expected that teachers have written literacy plans and use data to develop those instructional plans and determine when changes need to be made. W hat does literacy instruction currently look like, then, in the ELS program? How is instruction being delivered? What is the content of literacy instruction? Who is providing instruction? What resources are bei ng used to provide literacy instruction? What are the roles of staff members in supporting literacy instruction? What data are being collected to support literacy instruction? How are teachers making instructional decisions? The purpose of the current evaluation question is to paint a rich, thick description of current literacy instruction in the ELS program. The following description of literacy instruction in the ELS program was developed using several different data sources including teacher surveys which contained a mix of rating scale and open ended questions as well as focus group interviews with primary teachers, intermediate teachers, and ELS support staff. Clearly differences exist between ELS classrooms with regard to the implementation of literacy instruction. Some of these differences may be attributed to factors such as the student make up in the classroom or the experience level of the teacher. The following description of literacy instruction in the ELS program is based on the combine d data that were collected from teachers, support staff, and parents. Ho w is Instruction Being D elivered ? Descriptions given by ELS teachers and support staff provide some information about what instruction looks like, or how it is being delivered, in E LS classrooms.

PAGE 183

167 Teachers report providing instruction in groups and in one on one settings. Teachers report that they have a designated time for literacy instruction and also provide integrated literacy instruction throughout the school day. Some ELS sta ff described a strong connection between literacy instruction and language instruction in the ELS program and literacy instruction that is likely to be integrated into the school day rather than occurring in isolation. Group instruction varies from class room to classroom, but in general, teachers described group lessons in which all of the students read or look at the same book, and in which the focus of literacy instruction is on concepts of print, vocabulary, and comprehension. Because of the varying n eeds of the students, teachers describe differentiating the group lessons through the use of visuals (e.g., one student may be reading a book that has only words and another student may be reading the same book but with visual supports that go along with t he words) and the use of technology (e.g., use of augmentative and alternative communication [AAC] devices, use of switches that allow students who are nonverbal to share a pre recorded response or make a choice, etc.). One teacher reported focusing her g roup lessons on life skills. Other teachers reported having literacy units that centered on a particular book or theme. One teacher described also having other carry over activities centered around that same book or theme, such as a bingo game that reinf orces the vocabulary in the lesson or a worksheet that the student brings home to complete with his or her parents. During one on one instructional time, students are typically taught discrete skills using direct instruction. Sometimes teachers use publi shed programs as a curricular resource for one on one instruction (e.g., Reading Mastery), and at other times they rely

PAGE 184

168 on teacher created materials. Oftentimes the teacher works with the student on his or her literacy related IEP goals during one on one instructional time. The survey asked t eachers whether they had designated times for literacy instruction in their schedules for none, some, half, many, or all of their students (see Table 2 ). Twelve out of 13 of the respondents specified that they sched ule designated times for literacy instruction every of their students, and the remaining of her students. These data suggest that teachers are scheduling time to provide direct instruction in t he area of literacy. In addition to having a designated time for literacy instruction, it also appears that teachers are providing literacy instruction throughout the school day. Teachers responding to the survey were asked whether literacy instruction (formal and informal) was provided throughout the school day for none, few, half, many, or all of their students. The majority of teachers reported providing instruction throughout the school day for of their students (84.6%), with the other 15.4% r eporting that instruction occurred heir students (see Table 3 ). Primary and intermediate

PAGE 185

169 teachers responded similarly to this question. One teacher described the success a student experienced when the literacy instructio school day and throughout her environment: S o what we ended up doing was just putting. . She had a sight word goal anyway . just putting the sight words around in her environment so she would associate the objects in the environment and she actually just last week read twenty words to me. Two themes emerged through the qualitative data analysis process that help to describe what literacy instruction looks like in the ELS program and how that instruction i s being delivered. The first is the connection between language and literacy. This connection refers to the strong relationship between the language skills of students with moderate to severe disabilities and their literacy instruction. For students who are nonverbal, or who are at the beginner or novice levels of literacy development, examples of instruction that may be considered both language and literacy include practicing expressive or receptive language skills, learning to follow a picture schedule and

PAGE 186

170 identifying picture symbols on their communication devices or in their environment. Teachers described working closely with speech and language pathologists to provide literacy instruction because of its strong connection to language. The other th eme that emerged was the importance of providing comprehensive and integrated literacy instruction. Teachers, parents, and support staff frequently identified the importance of providing literacy instruction that did not occur in isolation but rather was instruction that did not drill students on individual skills, but rather provided instruction, Translated int o classroom practice, comprehensive and integrated instruction would mean that students would be exposed to a variety of different skills in a variety of means in a way that is embedded into what they are already learning in contrast to instruction that s imply focuses on the direct instruction of isolated skills in a one on one setting. What is the Content of Literacy I nstruction ? The content of literacy instruction in ELS classrooms varies, depending primarily on the student and setting. More specifi cally, during group instruction, teachers described focusing on skills such as concepts of print, phonological awareness, vocabulary, comprehension, and general exposure to literacy. In contrast, teachers described one on one instruction as focusing on le tter identification, phonics, sight words, and comprehension. C onsidering both of these practices, teachers appear to be providing comprehensive literacy instruction, or literacy instruction that covers multiple skill areas, for most students.

PAGE 187

171 Not all s tudents in the ELS program appear to be receiving instruction in all sub skills of literacy. In fact, t eachers reported providing instruction in an average of 5.2 areas of literacy per student out of the possible 8 (see Table 4 ). Primary teachers reporte d somewhat more comprehensive programming, providing instruction in an average of 5.7 areas out of 8, and intermediate teachers report ed providing instruction in slightly fewer areas (with an average of 4.6 out of 8 per student), representing less comprehe nsive programming. To gain information about which skill areas students were receiving instruction in, teachers were asked on the survey to identify whether each of the 8 skill levels is part of the regular, systematic literary instruction for each studen t in their classrooms (see Table 5) N otable differences could be seen in how primary and intermediate level teachers spent their instructional time. Primary students were most identification (57.1%). For intermediate students, four skill categories (phonics, sight words, comprehension, and vocabulary) were similar in their frequency of instruc tion (ranging from 68.3% to 73.2%), representing the skills that are most likely to be taught at the in line with what would be expected for the skill level of students in primary and intermediate classrooms.

PAGE 188

172 One theme that emerged through the qualitative data analysis that relates to the content of literacy instruction in the ELS program is that of instructional individualization. Teachers reported individualizing student instruction in a variety of different ways, including making adaptations and modifications to instructional programs, finding instructional strategies that interest and engage a c hild, and making instruction personal by connecting the content to real life events. Teachers also reported making the instruction more meaningful through practices such as selecting words that

PAGE 189

173 ating books about the community that incorporate places where that particular student goes on a regular basis. Literacy instruction in ELS classrooms includes instruction in a variety of literacy skill areas, but the actual content varies depending on th e setting (i.e., group instruction vs. one on one instruction) and on the level of the student (i.e., primary vs. intermediate). However, the majority of students appear to be receiving instruction in a range of literacy skills, suggesting that the instru ction is comprehensive. Furthermore, teachers report that they provide individualized literacy instruction for each student depending on the unique learning needs. Who is Providing I nstruction? As expected, teachers in the ELS program report that they are providing literacy instruction to their students, but they also report that they depend a lot on the assistants in their classrooms to also deliver literacy instruction. Teaching assistants are most likely to deliver instruction on a one on one basis. They often are responsible for implementing the Direct Instruction programs or providing instruction in other discrete skills. Teachers report that they are responsible for providing direction and oversight of the assistants who are deliverin g some of the instruction. One teacher described how she coordinates the instruction that her teaching assistants deliver: I sort of just made a list for all my TAs of like what literacy materials in the classroom each kid should be using. So I did docum ent specifically that so and so was doing this level or was in Red Book 3 or whatever. But, just so that they know that those are materials that are appropriate for them to use. And then we work through them and I just check in on what they are doing.

PAGE 190

174 An other teacher described how she coordinates the instruction that the assistants in her classroom deliver: I generate materials and like I kind of assess where they are and then I present them to the TAs and kind of let them move through them. I give them a lot of kids are doing for their activities. Other people who might provide literacy instruction include the speech and language pathologists and the intervention spec ialists. The speech and language pathologist and intervention specialists often support instruction through integrating and reinforcing literacy concepts and instruction into the social skills, life skills, and language groups that they run. One teacher reported that the speech and language pathologist in her classroom helps to deliver the Language for Learning program. What Resources are Being Used to Provide Literacy I nstruction ? A number of instructional resources are available to teachers in the EL S program, such as teacher created materials, comprehensive published instructional programs, technologies that support instruction, and web based resources. On the survey, teachers were asked to rate the helpfulness of a variety of instructional a nd othe r resources (see Table 6 ). They were also asked to identify the number of students for whom they used various instr uctional resources (see Table 7 ).

PAGE 191

175 As represented in Table 6, t eachers reported that they find teacher created materials the most helpf ul ( M = 4.2 ) and are using teacher created materials with all of their students (see Table 7) Primary teachers rated teacher created materials as being more helpful ( M = 4.4 ) than intermediate teachers ( M = 3.8). During focus group

PAGE 192

176 interviews, teachers commented that they depend on teacher created materials for their group literacy instruction, during which they use literacy units, adapted books, and other supporting materials that they have developed. Using teacher created materials allows them to indi vidualize the instruction to meet the needs of the various members of the group. They also report depending heavily on teacher created materials for use with students who are nonverbal or who are otherwise at the beginner or novice stages of literacy deve lopment. Teachers also rated published curriculum materials as helpful when providing literacy instruction ( M = 4.1 ), but they reported using the published curriculum materials (see Tables 6 and 7, respectively) Two p ublished curriculum programs (Reading Mastery and Language for Learning) have been identified as the core instructional programs for ELS classrooms and were made available to all teachers as part of the ELS Literacy Initiative. Other published programs th at teachers report using in their classrooms include Reading Milestones (a sight word based program), Meville to Weville (a program that emphasizes concepts of print as well as vocabulary/comprehension and comes with materials that have already been adapte d), and Wilson (a program that has been modified by ELS staff from its original version so it can be used with students in the ELS program). Of the published curriculum mentioned (excluding Reading Milestones), teachers reported using Reading Mastery the most ( M = 2.5 M = 1.4 students at the primary level, and primary teachers reported using the pro gram more than teachers at the intermediate level. One teacher commented that while she uses the

PAGE 193

177 published curriculum materials, she recognizes that they are only one piece of a complete Mastery is like one After teacher created materials and published curriculum, the nex t resource that teachers reported finding the most helpful was literacy websites, with teachers giving these websites a mean he lpfulness rating of 3.8 When asked the number of students for whom they use literacy websites to support instruction, teachers reported using literacy M = 3.8 ) and literacy M = 3.6 ). Primary level teachers reported using both of these resources slightly more than intermediate level teachers. The websites for students that teachers mentioned using on the survey and during the focus group interviews included Starfall, Reading A to Z, and Tumblebooks. During the focus group interviews, teachers and support sta ff reported that teachers were likely to use websites for students as independent or practice work. In other words, these websites are not used to provide instruction, but rather to reinforce existing skills, to provide students with alternative literacy experiences, and to allow students to work independently. Websites for teachers that were mentioned included Enchant ed Learning and Reading A to Z. Teachers were asked specifically about the Reading A to Z website on the survey, and they reported that th ey use this resource to M = 3.2 ) of their students. Teachers reported using these websites to access materials that can support instruction, such as printed worksheets or leveled readers.

PAGE 194

178 The resource th at teachers gave the next highest rating of helpfulness was computer software programs ( M = 3.7 ). Similar to how they used websites created for student use, t eachers reported using computer software programs primarily to reinforce existing skills or to al low students time to work independently on the computer. For example, several software programs are essentially books on the computer, and the programs can be adapted for switch users so they can participate through turning the page by hitting the switch. These programs tend to reinforce skills such as concepts of print and reading comprehension, and they provide general exposure to literacy experiences. Other computer software programs that teachers and support staff report using include Edmark (a funct in a wide range of literacy skills), and Intellikeys (a program with an adapted keyboard that teachers report using for writing activities). Teachers also reported accessing general education resources, such as websites for teachers and instructional materials that are used by the general education teachers in their particular schools. For example, one teacher reported using the phonological awareness program that is part of the gene ral education curriculum in her building, and another teacher reported accessing the Guided Reading library in her school for leveled read ers used d uring group instruction. As described by the teacher, And I do adapt a lot of the Guided Reading books, be cause I think the kids like comprehension questions and worksheets for a lot of ours. Our library has a whole leveled reader wall, and so I use that a lot, because I can mak e flashcards or

PAGE 195

179 O ther programs and resources are also used in ELS classrooms. When teachers are looking for new or additional resources, they will often contact the literacy c oach, who will help them to identify appropriate r esources and will either give them a copy of the resource from the ELS literacy loan library or will order the resource for them. What are the Roles of Staff Members in Supporting Literacy I nstruction ? A number of staff members are responsible for supp orting ELS classrooms, including the teacher, teaching assistants, speech and language pathologist, intervention specialist, program supervisor, and various consultants (e.g., literacy consultant, technology consultant). Parents are also considered part o f the classroom team. The following is a description of the role of each of these team members in supporting literacy instruction in the classroom as described by teachers, support staff, and parents on surveys and during focus group interviews. The tea cher is the person who has the primary role of planning and providing literacy instruction in the classroom. The teacher has the responsibility of developing the instructional plans, scheduling the instruction, obtaining or making instructional materials, and carrying through with the instruction. Because teachers are not able to provide all of the instruction themselves, they are responsible for communicating the instructional plans with the teaching assistants and training them on the implementation of those plans. While not always the primary individuals responsible for coordinating data collection, teachers are responsible for at least supporting data collection, including collecting regular progress monitoring data and initiating instructional proble m solving when students are not making adequate progress. Teachers also take the lead in updating IEP goals and annually developing new goals.

PAGE 196

180 Teaching assistants play a large part in delivering literacy instruction in ELS classrooms. F our to six teac hing assistants may be assigned to a classroom with eight students. Because the classroom teacher cannot be with all of the students all of the time, the teaching assistants often assume responsibility for delivering the literacy instruction. They are mo st often responsible for instructing in discrete skills in a one on one setting, which frequently involves the delivery of the Direct Instruction programs. Teaching assistants are responsible for attending workshops on how to correctly deliver programs th at are being used with students in their classrooms. Assistants are also frequently responsible for providing instruction on identified IEP goal as well as for collecting the data on student progress toward these goals. Sometimes assistants support teach ers in The speech and language pathologist is typically assigned to an ELS classroom two and a half days a week, and because of the strong connection between language and literacy, he or she often p lays a large role in supporting literacy instruction. One of the themes that emerged from the qualitative data analysis process was that of a strong language and literacy connection for students in the ELS program. One speech and language pathologist des cribed such a connection, saying, A lot of the materials that are used [by the SLP] just involve literacy since they have a visual support; they are reading. So whatever they are doing with support in a structured activity is involving literacy, whether i following a schedule. For ELS students, literacy is defined broadly and can be considered to include all communication expressive and receptive. The speech and language pathologist spends

PAGE 197

181 the majority of his or her time in the classroom improving student communication and speech and language pathologist does supports literacy development. More specifically, teachers and speech and language pathologists report that they support literacy instruction through providing support with lesson planning, helping teachers make and modify materials for instruction, developing and leading group lessons, modifying materials for inclusion, programming stu dent AAC devices to support instruction, identifying technology supports, adapting books, and identifying IEP goals. The speech and language pathologists also provide direct instruction in literacy skills, such as running literacy groups or delivering the Language for Learning program. The intervention specialists in the ELS program are school psychologists, and they are assigned to a classroom one day a week. The role of the intervention specialist in the classroom can vary, but the following is a rich description of how one intervention specialist described her role in the classroom with regard to supporting literacy instruction: I support literacy with a lot of assessment of students, especially if they are new ool year, to identify instructional targets for the students, and once we identify some of their literacy needs, I help teachers to go materials or strategies for students t down with a teacher and come up with a strategy. And write a script out that they can use, some kind of a visual or something that they can use, with the student. And I help teachers to understand nge something about

PAGE 198

182 their literacy instruction through progress monitoring that we do with the students. area of measure. This particular intervention specialist has a lo t of interest and experience in the area of literacy and is clearly very involved with literacy instruction in her classrooms. However, this is not the case with all intervention specialists, and their level of involvement can vary greatly. One teacher r eported that she would like the intervention specialist who supports her classroom to be more involved in planning and delivering literacy instruction, but that her intervention specialist does not have the background knowledge and skills to be able to do so. All intervention specialists are responsible for supporting the data collection in the classroom, and with regard to literacy, they are responsible for collecting benchmark data on all students three times a year and supporting IEP goal progress monit oring. Some intervention specialists also support the delivery of the Direct Instruction programs in their classrooms. S everal consultants are available to support teachers in the ELS program, including a literacy consultant, technology consultant, and general classroom consultant. On the surveys and during focus group interviews, teachers and support staff described the roles of the literacy and technology consultants. The role of the literacy consultant is primarily to support teachers in identifying and obtaining appropriate instructional resources for their students as well as to support the training and implementation of the published curriculum programs. The literacy consultant will also help teachers with the instructional planning process by he comprehensive instructional plans that are appropriate for each particular student. The

PAGE 199

183 literacy consultant also will observe in classrooms during lessons and provide feedback to the teacher about wh at can be done differently. The technology consultant is available to suggest technology resources, especially in the area of computer software. She also provides trainings on computer software and will give on site support for the use of these programs. Other team members that were mentioned on the survey and during focus group interviews as having a role in supporting literacy instruction included the program supervisor, the occupational therapist, and the parents of the students. The role of the pr ogram supervisor was described as connecting teachers to various resources including available trainings, as well as conducting observations in the classroom. The occupational therapists were described as supporting literacy instruction through teaching a nd reinforcing writing skills as well as adapting any materials that students struggle with manipulating. Finally, parents report supporting literacy at home and working with the school team to identify IEP goals. While the various staff members each t ake a primary role in some aspects of literacy instruction in the classrooms, effective literacy instruction cannot take place without communication and collaboration among staff members. Communication and collaboration was a theme identified through the qualitative data analysis process, and it refers to working together as a team to plan and implement literacy instruction. Teachers reported p ositive outcomes in classrooms in which team members were perceived to be effectively communicating and collabora ting, such as referenced by one support staff

PAGE 200

184 is a team effort that requires communicati on and collaboration among those team members. What Data are Being Collected to Support Literacy I nstruction? Teachers in the ELS program use various sources of data to support the planning and delivery of literacy instruction. With regard to how frequ ently they use data to develop instructional plans for their students, most teachers reported out 8 ). When asked how frequently they use data to help them decide when to make instructional changes, teachers responded similarly, with 53.8% (7 teachers) responding 9 ). Primary and intermediate teachers responded similarly to both questions. The comment of one teacher

PAGE 201

1 85 On the survey and during the focus group interviews, teachers reported using a variety of data sources to help them plan instruction and make decisions regarding student progress. When asked to rate the helpfulness of various sources of data on the teacher survey, all sources of data received an average rating of 3.3 or higher, which ). The o ther s ources of d M = 4.5 ) and IEP Goal Data ( M = 4.4 ). The IEP goal data are progress monitoring data that teachers collect on a frequent basis specifically to inform student progress on their IEP goals. The regular collection of data to inform student p rogress on IEP goals is standard practice in the ELS program, not just for IEP goals related to literacy. It is not clear from s choices that teachers were presented with on the survey. As reported in Table 10, t he lowest rated sources of data in terms of helpfulness M = 3.3) M = 3.6 M = 3.6 ). The l iteracy tracking

PAGE 202

186 (novice, beginner, early to upper emergent, and upper emergent to fluent) in seven and information on their current instructional programming. This form had only been in use for one school year before the teachers were asked to complete the survey and rate its helpfulness. Teacher comments during the focus group interviews suggest that not all teachers completed the form directly after the first year it was in use and that on some occasions, literacy tracking forms were not pass ed on to the next teacher when students transitioned within the ELS program. The mastery tests in the curriculum are the regular assessments that are included in the Direct Instruction programs, which not all teachers are using with their students. Teach ers also report using the placement tests for the Direct Instruction program to help determine appropriate student placement in the program. The literacy benchmark data are curriculum based assessments that are given to all students in the ELS program thr ee times a year. The other two sources of data that teachers were asked to rate the helpfulness of i discrete trial d highest nor the lowest rated in t erms of helpfulness. Discrete t rial data are data that are kept on student progress when students are being instructed in a one on one setting using the discrete t rial instructional method. The only other data source that was mentioned during the focus groups but not included on the t includes sections on reading and writing).

PAGE 203

187 Teachers rely on support staff in their classrooms to collect literacy related data on thei r students. More specifically, teachers reported that the teacher consultants and the literacy consultants had collected data to help develop instructional plans for students in their classrooms. They also depend heavily on intervention specialists to ga ther progress monitoring systems, and to then help to interpret the progress monitoring data once the data have been collected. They also depend on their teaching a ssistants to regularly collect the progress monitoring data. W hile teachers may use the data to help guide their instruction, much of the data collection itself seems to be collected by other staff members in the classroom. Teachers in the ELS program report that they use data to plan instruction and to determine when to make changes to that instruction, and they report finding some data sources (e.g., IEP goal data) more helpful than others (e.g., l iteracy b enchmark data).

PAGE 204

188 Teachers depend heavily on c onsultants, support staff, and assistants in their classrooms to help gather and interpret the data so that they may be used for instructional planning. How are Teachers Making Instructional D ecisions ? The instructional decision making process involves collecting data on student skill development, identifying instructional targets and writing goals, developing instructional plans, monitoring student progress, and modifying instructional plans based on student progress. This is an ongoing, cyclical proce ss that is considered best practice in providing effective instruction. This component of the current evaluation question asks how teachers are making instructional decisions and whether they are using the problem solving process just described. The prev reported on how teachers in the ELS program usually or always use data to develop instructional plans and to determine when to modify those instructional plans. That section also ident ified the data sources that teachers found most helpful in planning Another component to the instructional decision making process includes identifying goals and instructional targets i n other words, determining the in instruction. One source that was made available to teachers through the ELS Literacy Initiative to support this component of instructional planning is the ELS Literacy Scope and Sequence The S cope and Sequence describes the four stages of literacy

PAGE 205

189 development (beginner, novice, early to upper emergent, upper emergent to fluent) for seven literacy skill areas. The purpose of this document is to assist teachers in identifying where their students are in their literacy development in each of the skill areas, and to identify what skills students should be working on next. Some teachers reported using the S cope and S equence in this manner, such as one teacher who stated, This year is the first year where I was looking at goals for next year where they should go next. Cause I was like, okay, they have their letter sounds, they have this. So that was helpful. When teachers were asked on the survey to rate the helpfulness of the ELS Literacy Scope and S equence they provided a mean helpfulness rating of 3.0/5.0, which is ). So me teachers reported that they depend on support staff, such as the intervention specialist, to help them identify appropriate instructional targets. As one teacher stated, ays ask their literacy coaches and with their speech and language pathologists in identifying instructional targets and developing IEP goals. One of the themes that emerged from the qualitative data analysis process that relates to goal writing and indentifying instructional targets was the theme of instruction within a broader perspective. This theme refers to providing and planning instruction with a bigger pictur e in mind, such as having a vision for where a particular student will

PAGE 206

190 be in the future, having high expectations for student success, or making a link to the general education curriculum when planning and providing instruction. One support staff member i llustrated this concept of having a broader perspective when planning and providing instruction when she said, If you have a student who only uses objects now, we want, as their reading goals then, for them to move onto pictures. And as their next reading goal we want them to move on to icons. And we want the icons to be paired with words, and at some point we want those words to be just to be letters and try giving them [teachers] what that whole continuum will look like just so that they have that long term goal, that for this kid, even though they are using items right now, our main goal is for them to be more typical looking readers. Another teacher described how she looks to general education when thinking about planning instruction for her students: not teaching. You know, that I should be teaching. You know, what does a kid in fourth grade, or a kid reading at a second grade level, what is he being exposed to? Teachers in the EL S program use a variety of resources when identifying reading goals for their students and planning their instruction, including the ELS Literacy Scope and Sequence their support staff members in the classroom, and other resources that provide a broader p erspective.

PAGE 207

191 Teachers completing the survey were asked whether they had written i nstructional plans for none, few, half, most, or all of their students. The responses to this question were varied, with half of the respondents indicating that they have a written literacy other half reporting that or all of their students (see Table 11 ). Primary and intermediate teachers displayed a similar pattern of responses. F orms have been made available to teachers to help them to plan literacy instruction, namely the i nstructional p lanning f orm (IPF). O ther planning tools are also available (e.g., a Four Block plan), and teachers in the ELS program are not expected to use a particular written format. Of the teachers who reported using some type of planning form to document student instructional plans, some teachers reported that these forms are helpful especially with regard to communication among staff members. Other teachers reported that using the forms can be tim e consuming and redundant, especially when multiple students have similar plans. One teacher reported not knowing about the instructional p lanning f orm or where to find it. While some teachers in the ELS program

PAGE 208

192 are generating written individualized inst ructional plans for their students in the area of literacy, others are not. Summary In order to provide a rich description of literacy instruction in the ELS program, t hat resources ort literacy evaluation questions identify and describe factors that serve to facilitate implementation of literacy instruction in ELS classrooms and factors that serve as barriers to the implementation of literacy instruction. Questions 2 and 3: What Factors Serve to Facilitate Implementation of the ELS Literacy Initiative? What Factors Serve as Barriers to the Implementation of the ELS Literacy Initiative? The primary purpose of the current evaluation study is to determine the next steps in the implementation of the ELS Literacy Initiative. In order to determine the appropriate next steps, it is important to first identify what is going well with the implementation and what needs to be improved. To help answer the current evaluation question, teachers, support staff, and parents were asked on the surveys and during focus group interviews what factors served to facilitate the implementation of the ELS Literacy Initiativ e and what factors served as barriers to implementation. Factors that they identified as facilitators or barriers to the implementation of the ELS Literacy I nitiative

PAGE 209

193 have been placed into the following categories: instructional materials, staff, training and professional development, students, classroom environment, and parents. Instructional M aterials The qualitative data analysis process resulted in the identification of two themes that relate to instructional materials and how they can either facilit ate or serve as barriers to the implementation of the literacy initiative: access to materials and time. Access to materials emerged as a theme from comments made by teachers, parents, and support staff on the surveys and during focus group interviews. ELS staff and parents identified the general availability of instructional materials and resources in the program as a factor that facilitates the implementation of literacy instruction. The ELS program is resource rich, and teachers clearly appreciate th have a problem getting the resources they need to support instruction. However, the fact that resources are readily available was also identified as a barrier because the program has so many resources to offer that teachers often do resources are purchased for teachers and those resourc when referring to computer website subscriptions, one support staff member commented We buy a lot of subscriptions on websites and then you can actually see the reports of who is using them, how often they are being used disappointing. There are like one or two teachers that fall in love with a sight and

PAGE 210

194 expensive site license. And out of the four teachers we bought it for only two have ever actually logged in. W hile teachers have access to many resources that support literacy instruction, there may end up using everything they have acce ss to. The availability of published curriculum to use for instruction was also identified as a factor that facilitates implementation of literacy instruction. Teachers commented that having a published curriculum is beneficial because it is something t hat teachers can pick up and use instead of having to create materials on a daily basis. Teachers specifically referenced Reading Mastery, Language for Learning, and Meville to Weville as published programs that they appreciate having access to. Teachers also mentioned the benefit of having a published curriculum that is scripted, or that includes verbatim the language that should be used when instructing the students. Teachers mentioned this factor as a benefit because they often have to rely on teachin g assistants to provide some of the literacy instruction, and having a script that assistants can follow facilitates this practice. Unfortunately, the published curriculums are not appropriate for all students, and teachers expressed frustration with the l ack of materials available to use for instruction with students who are nonverbal, severely disabled, or otherwise at the beginner or novice levels of their literacy development. For these students, teachers must rely on teacher created materials. The pr imary barrier that teachers identified regarding the use of teacher created materials was the time it takes to develop the materials. Several teachers expressed frustration with their lack of ability to effectively share teacher

PAGE 211

195 created materials among EL S staff members. Such a system for sharing materials could serve as a way of overcoming the primary barrier to the use of teacher created materials, which is simply the time and effort it takes to develop and individualize such materials to use for instru ction. Another theme that emerged from the qualitative data analysis was lack of time. T eachers and support staff repeatedly mentioned time as a barrier to providing instruction, specifically as it relates to the creation and development of instructiona l materials. Interestingly, teachers expressed the desire for published programs that are for students who are nonverbal or significantly disabled, presumably because of the time it takes for teachers to create the materials that are necess ary for providing literacy instruction to this specific population of students. One teacher suggested that the time and effort it takes to create instructional materials that are individualized for each student may be one factor that contributes to teache r burn out and a high staff turnover rate in the ELS program. The ELS program is rich in resources, and teachers have access to many materials to support literacy instruction. However, some teachers report that they are not aware of all of the resource s that are available, suggest ing that many of the available resources go unused. Teachers appreciate the published curriculums that are available, but they would like to see more ready to use materials available for students who are nonverbal or are at th e beginner and novice stages of literacy development. Lack of t ime to create the necessary materials for instruction acts as a barrier to implementation, and teachers expressed a strong desire to have a means for sharing materials among one another.

PAGE 212

196 Sta ff Related F actors Two themes emerged from the qualitative data analysis that can be considered staff related implementation factors: issues with communication/collaboration and experience/knowledge. Teachers, support staff, and parents consistently ide ntified these two themes as factors that could either support or hinder the implementation of literacy instruction. Communication and collaboration refers to having time to work together as a team to plan and implement literacy instruction, and it was i dentified as a facilitating factor for implementation. P ositive outcomes were reported in classrooms in which it was perceived that team members were effectively communicating and collaborating, ally when you have a team to collaborate as a team as a barrier, which the comment of one teacher illustrated: Yeah, I think if we had more time for people to colla borate with each other. Like, you know with TAs, having more discussion time. If we had more resources that we could have time to talk to them. Or if we had more time with our other literacy. That would probably be the biggest help. M any people collaborate to provide instruction to students in ELS classrooms, including teachers, teaching assistan ts, support staff members (e.g., speech and language of these team members must work together to determine goals for instruction and

PAGE 213

197 appropriate methods of inst ruction. ELS staff and parents clearly recognize the importance of this communication and collaboration, but sometimes they find it difficult logistically to make it happen. Another theme identified through the qualitative data analysis was that of exper ience and knowledge. One of the problems that the ELS Literacy Initiative was intended to address was the lack of teacher training and knowledge in best practice beginning reading instruction for students with moderate to severe disabilities. As one teac to teachers, teaching assistants, support staff, or parents, having knowledge and training in the basics of literacy development and instruction was identified as a facilitating factor for implementing quality literacy instruction in the classroom. Teachers and support staff frequently mentioned teaching assistants and their general l ack of knowledge and experience in how to teach reading as a barrier to the implementation of literacy instruction in the classroom. Teachers depend on t eaching assistants to provide much of the direct instruction in ELS classrooms, but as one teacher com mented, they are the least qualified to do so: are very many TAs who are equipped to do that [teach reading] and who literacy skills are. So I think that really impacts the quality, unfortunately, of the literacy instruction that sometimes they [students] get.

PAGE 214

198 Because the assistants are responsible for providing much of the instruction, it is important that they have a basic understanding of how reading develops as well as best practices in reading instruction. While the program provides regular training for teaching assistants in how to implement the Direct Instruction programs, teachers suggested that the program fall s short of providing training on the basics of literacy and literacy instruction. Teachers suggested that they had difficulty with finding a way to provide this basic level of training to teaching assistants. One teacher suggested that it should be the r esponsibility of the program to provide the assistants with this training before they even start their jobs. Teachers and staff identified other benefits to providing teaching assistants with training in the area of literacy, such as having a shared langu age with instruction. The benefit of having knowledge and experience was also mentioned with regard to ELS teachers. One support staff member suggested that teachers w ith less experience Another support staff member suggested that teachers who have more experience and knowledge are more capable of problem solving when students are not making adequate progress: two and they can really being used? Is it a behavior or a lack of skill?

PAGE 215

199 Support staff also reported that teachers who have received additional training in the area of liter acy appear more confident about their instruction. One barrier to the believe that reading and literacy instruction are important for all students, even those with the most significant disabilities. Based on the comments of ELS support staff, it appears that teachers in the ELS program who have more knowledge and experience with regard to literacy instruction are better able to problem solve, can appropriately focus the ir instruction, are more confident, and are more likely to view literacy instruction as important for all students. The importance of knowledge and experience was also referenced with regard to ELS support staff, such as the speech and language pathologi st and the intervention specialist. Some teachers mentioned that their support staff are very knowledgeable, saying they depend on their support staff for everything from instructional planning support, data collection and support, and help with problem s olving for students who are not making expected growth. However, one teacher mentioned that she would like more support from the intervention specialist who supports her classroom, but that she does not feel that her intervention specialist has the knowle dge base to provide that support. Thus, support staff who have knowledge and experience with regard to reading development and instruction can serve as a facilitating factor for implementation, and support staff a barrier. Training and Professional D evelopment Various sources of professional development are available to teachers and other staff in the ELS program. Workshops and trainings that are specifically designed to meet

PAGE 216

200 the needs of ELS staff, including teaching assistants, are offered through the program. The cooperative district that the ELS program is a part of also hosts a variety of workshops, but these are not necessarily specific to the ELS program. Teachers and staff in the ELS program also atte nd outside workshops and trainings. Teachers and support staff were asked to identify factors related to professional development and training that facilitated the implementation of the ELS Literacy Initiative and factors that served as barriers to implem entation. Having training opportunities available for staff in the ELS program was generally viewed as a facilitating factor for implementing the literacy initiative. However, some trainings were mentioned as being more helpful than others. For example several teachers and support staff mentioned that trainings provided by the ELS program on how to implement Reading Mastery and Language for Learning are helpful. ma terials for their students to use in the classroom the next day, to be helpful. Trainings geared toward a more general education population of students were identified as helpful because they help teachers to gain perspective on what is expected of typica lly developing students at each of the grade levels. One teacher commented that the trainings that are hosted by the ELS program specifically for the ELS staff were spec ific as possible to the students that we are working with. Those are the most With regard to barriers, teachers mentioned that they would like to see more training opportunities for parents, trainings that target a younger population of

PAGE 217

201 stu dents/students who are at the beginner and novice stages of literacy development, and trainings that cover reading comprehension instructional strategies. Teachers also mentioned the desire to have more time to follow through with strategies that they lea Teachers and support staff frequently mentioned t raining for teaching assistants as something that could either serve as a barrier or facilitate literacy instruction in their classrooms. The ELS program provides training for teaching assistants on a regular basis. Mu ch of the training in the area of literacy has focused on how to implement the various published programs that are available for instruction, such as Reading Mastery and Language for Learning. Teachers commented that these trainings are beneficial and hel p facilitate the implementation of these programs in the classroom because more staff are then available to deliver the instruction. However, one barrier that teachers mentioned with regard to teaching assistants attending trainings involved not having en ough staff to send a teaching assistant to training for a full day. One teacher commented that sending out a staff member for the day creates a safety concern in her classroom. Student F actors Teachers, support staff, and parents identified several st udent related factors that could either facilitate or serve as barriers to the implementation of effective literacy instruction in ELS classrooms. Staff identified students who are attentive and show a desire to learn as a facilitating factor, and convers ely, they identified students who have

PAGE 218

202 attention and distractibility issues as a barrier to effective instruction. Staff also identified students who come into their classrooms after having had early exposure to literacy as a facilitating factor. Similar ly, staff identified students who come into the program from a preschool setting without having academic goals as a barrier to literacy instruction. The presence of s tudents who make progress was identified as a facilitating factor because of the motivati ng effects that it has on students and staff alike. In contrast, the presence of students who make very little or no progress, or students who make very slow progress, was identified as a barrier. Teachers often put a lot of work into creating and adapti ng the instruction for these students, and to have them show very little or slow progress in return can be very frustrating. The primary student related factor that was identified as a barrier to effective literacy instruction was having students who are nonverbal and/or severely disabled. The challenge of providing instruction for students who are nonverbal or have other complicated learning needs was a consistent theme that ran throughout the qualitative data analysis. Such students are typically not able to participate in a traditional reading program, such as the core curriculum that was identified through the literacy i nitiative. Teachers consistently referenced the need for a reading program that could be used with students who are nonverbal. Wit hout a curriculum, teachers often must create reading programs from scratch as well as creating or adapting all instructional materials. Classroom E nvironment Teachers were prompted on the survey to identify factors that facilitate literacy instruction and factors that serve as barriers to literacy instruction that were related to the classroom environment. The majority of comments referred to physical structures that

PAGE 219

203 could facilitate instruction, such as bookshelves that allow students to access materi als independently, group tables, classroom dividers, and a place to provide one on one instruction. Having a classroom with a high noise level was identified as a barrier. One teacher identified the classroom schedule as a barrier to providing literacy i nstruction. As she stated, ; you want them all to be there so that they can, you know, be in a group and learn, Parent F actors One theme that emerged through the qualitative data analysis process was the impact that parent involvement can potentially have on clas sroom instruction and student outcomes. The parents of students in the ELS program tend to be well educated and strong advocates for their children, which can be a facilitating factor in the implementation of literacy instruction. For example, the qualit ative data suggest that parents are requesting that teachers use specific reading programs with their students, such as the Reading Mastery program. One parent provided an example of how his involvement affected the classroom instruction: It is my feeli ng that literacy should be a combination of exposure and opportunity for the children in a variety of settings. However, I feel that a structured program should be an integral part as well. This piece [Reading Mastery] was lackin g until we pushed for it in a meeting.

PAGE 220

204 One teacher also described how she felt strongly against using a reading program but did use the program because of a parent request: rather we did it. However, teachers also commented that parent involvement can be a barrier, such as when finding that parents are pushing for programs like Reading Mastery for children who are nonverbal or who otherwise are unable to participate in the program. However, teacher s generally view a parent educating his or herself on reading instruction and advocating for his or her children as a facilitating factor to instruction. facilitating factor b ecause of the potential positive impact on student outcomes. Parents and teachers both commented on the positive effects that parents working with their The results of early targeted supplemental homework in reading and reading comprehension is remarkable. There is no other word . . H e is diagnosed as moderate on the autism spectrum and went from struggling with his alphabet to reading in less than one year. Some t eachers described parents who were repeating some of the Direct Instruction lessons at home, and others described parents who were doing homework with their children or otherwise reinforcing skills at home.

PAGE 221

205 From the comments of parents and teachers, i t is clear that parents are initiating communication and involvement and that this is having an e ffect on what happens in the classroom. One parent described how her efforts resulted in the teacher sending work home: We have develop ed a cooperative relati onship with the teacher where she provides reading and comprehension homework every day, which we religiously have our son work through and complete. All this is at the initiative of us asking and the teacher cooperating. Another parent described how her e fforts resulted in increased communication between home and school: literary goals and achievements. d on the teacher was using the Reading Mastery program. As these examples show, parent involvement can clearly have a strong influence on classroom instruction in t he area of literacy and serve as a facilitating factor for the implementation of the ELS Literacy Initiative. In some cases, teachers viewed parent involvement as a barrier, such as when parents push for programs that are not appropriate for their childre n, but the facilitating and positive factors identified outweighed the potential negative effects. Summary A number of factors that facilitate the implementation of the literacy initiative and serve as barriers to implementation were identified by ELS teachers, support staff, and

PAGE 222

206 parents. Teachers identified a ccess to instructional materials, especially published curriculum, as a facilitating factor, while identifying a lack of such programs to use with nonverbal students as a barrier. They a lso ident ified as a barrier the time and effort it takes to create instructional materials and the lack of an effective way of sharing these teacher created materials among ELS staff members. With regard to staff, communication and collaboration among team members was identified as a facilitating factor, along with having knowledge and experience. Teachers also viewed t he trainings that are offered to ELS staff as a facilitating factor, but they would like to see additional training for TAs in the basics of litera cy, as well as training for parents, trainings that apply to younger students, and trainings in the area of reading comprehension. They also identified s tudent factors and felt that the primary barrier to providing literacy instruction involved trying to meet the needs of students who are nonverbal or who are at the beginner and novice levels of literacy development. Facilitating factors related to the classroom environment primarily consisted of the physical structure of the room, and the classroom sched uling was identified as a barrier, specifically when students from outside the classroom attend inclusion classes, which makes it difficult to provide group instruction. Finally, staff members identified several parent factors, viewing parents who advocat instruction at home as facilitating factors to the implementation of literacy instruction in the classroom.

PAGE 223

207 Outcome Evaluation Questions Evaluation questions 4 through 13 a re considered outcome evaluation questions in that they are designed to determine whether the desired intended short intermediate and long term outcomes of the ELS Literacy Initiative have been attained. Question 4: How are Teachers Using the Resourc es (Material and People) They Have Been Provided to Help Support Literacy Instruction in Their Classrooms? A number of material and human resources have been made available to teachers as part of the ELS Literacy Initiative. With regard to material resour ces, most have been curricular resources, including the development of a literacy scope and sequence, the identification and purchase of a core language/reading curriculum, and the dissemination of supplemental curricular materials. Other material resourc es include assessment and instructional planning resources. The human resources that have been mad e available to teachers include professional development opportunities and various classroom consultants. This evaluation question will aid in determining w hether teachers are using these available resources, how they are using these resources, and the factors that facilitate or serve as barriers to the use of these resources. The ELS Literacy Scope and Sequence was created to guide teacher instruction in th e area of reading. The document identifies the stages of literacy development (novice, beginner, early to upper emergent, and upper emergent to fluent) in seven different areas of literacy (concepts of print, letter identification, phonemic awareness, pho nics, fluency, vocabulary and comprehension, and writing). The Scope and Sequence is tied to assessment strategies and instructional recommendations, and related resources packaged as a series of binders were made available to teachers. The Scope and Seq uence was

PAGE 224

208 developed specifically to address the unique learning needs of students with moderate to severe disabilities, and it was intended to guide teacher instruction in the ELS program Two questions on the teacher survey pertain to the use of the ELS Literacy Scope and Sequence and the corresponding binder series. The first asked teachers to rate the helpfulness of the Scope and Sequence Teachers responding to the survey provided a me an helpfulness rating of 3.0 on a 5 point scale equivalent to a oderately ). Primary teaches appear to find the resource more helpful ( M = 3.3) than intermediate teachers ( M = 2.7). Teachers were also asked abo ut how frequently they use the Scope and Sequence and the corresponding bi nder resources, and they were asked to identify whether they used the resource with none, few, about half, many, or all of their stu dents (see Table 13 ). Teachers provided an average usage rating of 2.5, which suggests that most teachers are using the res difference in the use of the ELS Literacy Scop e and Sequence b inders between the levels, with the primary teachers having an average usage usage rating of 1.8 for this resource. Qualitative data taken from the teacher surveys and focus group interviews suggests that when teachers are using this resource, they are using it to identify instructional goals and targets and not necessarily to identify ideas for instructional activities. As one primary level teac her commented

PAGE 225

209 before. I was always doing the same goals and kind of jumping around and seeing witho ut really knowing, like, exactly what the progression is. So I think that helped. Several teachers commented on why they were not using the ELS Literacy Scope and Sequence and corresponding literacy binders, and they all generally referred to the fact tha t the S cope and Sequence and corresponding binders contain so much information that the y are overwhelming to access. Another teacher reported that she depends on the intervention specialist in her classroom to use and interpret the information in the Scop e and Sequence and the binders and then to pass the information along to her.

PAGE 226

210 One major change in the ELS Literacy Initiative involved the identification and purchase of a core curriculum for language and reading. The chosen curriculum was a combinati on of Language for Learning and Reading Mastery (Direct Instruction programs), the first of which targets language development and vocabulary, and the second of which targets phonemic awareness, phonics, and fluency. The expectation from the administratio n is that these programs will be used with as many students as possible in grades K 5. Whether a student is able to participate in the curriculum depends on his or her level of literacy development and ability to respond to questions orally on cue. A cur riculum set was purchased for every primary and intermediate classroom. When asked on the teacher survey to rate how freque ntly they use d various instructional resources, teachers p rovided a mean rating of 2.4 for both Reading Mastery and Language for Lear ning, which suggests that most teachers are using these programs of their students (see Table 13 ). Data suggest that intermediate teachers are

PAGE 227

211 using the Reading Mastery program ( M = 2.7) with slightly more students than the primary teachers are ( M = 2.1). The ratings for use of the Language for Learning program were more similar between primary and intermediate teachers. Two (of 13) reported using Language The primary reason that teachers cited for not using the Direct Instruction programs in their classrooms is having students who are not able to meet the requirements for participation (i.e., who can not repeat shor t phrases on cue, or who have not yet reached the appropriate level of literacy development ). A large number of students in the ELS program are nonverbal, and the DI programs cannot be used with this population. One teacher cited the scripted nature of t he instructional delivery as another reason for not using the program. Of the teachers who are using Reading Mastery and Language for Learning in their classrooms, they cited several reasons for doing so, including the benefit of having a program that is ready to use and does not require the teacher to make additional materials ; the opportunity to have TAs in their classrooms deliver the instruction, since the DI programs are scripted; and the fact that teachers and TAs have ready access to training in the use of the program s Overall, teachers viewed the availability of the published core curriculum very positively, and a number of teachers expressed the desire to have something similar that could be used with students who are nonverbal. In addition to t he curricular reso urces mentioned above, materials allocated to the literacy i nitiative also include assessment resources. One example is the literacy benchmarking tools. The intervention specialists in the classrooms collect literacy

PAGE 228

212 benchmarking data o n all students in the ELS program three times a year to determine if they are making adequate progress. For more frequent progress monitoring, the program has purchased a subscription to A IMS web, which provides access to a variety of curriculum based asse ssment measures and software for graphing student progress. Finally, a tool was created to track student literacy development and instruction across grade levels within the ELS program. The primary goal of all of these resources is to provide teachers wi th information on student progress to inform instructional decision making. The literacy benchmark data provide information to teachers on student progress with all students three times a year. Ideally these data are to be used by teachers to help pla n their instruction and to assist them in indentifying when instructional changes need to be made. When prompted by the teacher survey to rate the degree of helpfulness of different data sources, teachers rated literacy benchmark data as the second from t he lowest of seven possible data sources (see Table 14) with an average rating of 3.6 out of 5 (with 1 = ). When qualitative data were examined, it was found that of the 56 comments made in reference to in structional planning, only two specifically mentioned the use of the literacy benchmark data. With regard to teachers using the benchmark data to support instructional decision making in thei r classrooms, little data exist to suggest that teachers are usi ng the benchmark data for this purpose. An analysis of why teachers may not be using the literacy b enchmark data

PAGE 229

213 The degree to which teachers are using A IMS web or the extent to which they find this resourc e helpful was not assessed directly. C onsequently, little evidence exists with which to determine the degree to which this resource is being used to support instructional planning. During a focus group interview, one intervention specialist did describe how she uses the AIMSweb account to support literacy instruction : date the graphs and e mail it to my teacher and say, and An examination of the A IMS web account reveals that there are students with progress monitoring data and graphs that are being tracked using the account, which suggests some use by teachers and intervention specialists. It also appears as though the program is being used to monitor student progress in multiple areas such as math and writing in

PAGE 230

214 addition to reading More information is needed to determine how teachers and intervention specialists are utilizing this resource. The final assessment resource associated with the ELS Literacy Initiative is the ELS Literacy Tracki ng Form. This form was developed to help track student literacy development from teacher to teacher and across time. The form contains teacher ratings of student literacy development as well as information benchmarking scores and instructional programming. This form is completed annually by teachers and is supposed to transition with the student. When asked to rate the helpfulness of this resource in planning and delivering literacy instruction, teachers gave rating of 3.3 (see Table 14 ). Primary level teachers rated this resource as more helpful ( M = 3.8) than intermediate level teachers did ( M = 2.6). Data from the focus group interviews suggests that not all teachers are rec eiving this information on their students as the students transition from one classroom to the next. Only one teacher mentioned the literacy tracking f orm in the focus group interviews, and she stated that she had only received the information for one of her students and had found that it was not specific enough to be very useful. The literacy t racking f orm had only been in place for one school year before teachers were asked to rate its helpfulness on the survey. With regard to human resources, teache rs have access to professional development opportunities and various consultants to support the planning and implementation of literacy instruction in their classrooms. Other human resources that are available to classroom teachers include the staff in th eir classrooms (e.g., the intervention specialist, speech and language pathologist, and teaching assistants).

PAGE 231

215 One of the questions on the survey asked teachers to rate the helpfulness of various people and professional development resources in implement ing li teracy instruction (see Table 15 ). As is shown, p rofessional development opportunities that were provided through the cooperative district and through the ELS program were each In M = 4.0 ) of the human and professional development resources. Other professional development may include anything outside of what is offered through the cooperati ve and the ELS program, such as reading conferences that teachers attend. An examination of the qualitative data suggest s that teachers appreciate having professional development opportunities and they generally try to take advantage of them. Some profes sional development opportunities w ere viewed as being more helpful than others. Trainings offered to staff o n how to implement the Direct Instruction programs were viewed positively by teachers and ow teachers to learn a new software program and then create materials using that program that can be used in the classroom the next day. Trainings that provide the general education perspective as well as trainings specific to children with moderate to se vere disabilities were both mentioned by teachers as being helpful. Teachers mentioned that they would like to see more trai ning opportunities for parents, in addition to trainings that target a younger population of students as well as students who are a t the beginner and novice stages of literacy development, and trainings that cover reading comprehension instructional strategies. One barrier to taking advantage of professional development that teachers mentioned was

PAGE 232

216 the issue of having adequate coverag e in their classrooms to keep students safe in their absence, or in the absence of one or more teaching assistants. Several classroom consultants are available to support teachers in planning and implementing literacy instruction in their classrooms. On e of these supports is the in reading and currently works for the program on a part time basis. Her job is to support literacy instruction in ELS classrooms, and she d oes that through a variety of means including providing in classroom consultation, identifying and allocating instructional resources, training staff on the implementation of published programs, and helping staff to problem solve on difficult cases. As re presented in Table 15, t eachers rated the support of the literacy coach as being more helpful when that support is provided on an individual basis in the classroom ( M = 3.9 ) than when the literacy coach attends a group ( M = 3.4 ). Several comments were made on the teacher surveys and during the focus group interviews regarding how staff members are using the support of the literacy coach and how the support could be improved. For example, staff report using the l iteracy coach to answer questions about available materials to use for instruction, and they depend on her a great deal to provide initial training and ongoing feedback regarding the implementation of the Direct Instruction programs. One teacher described how the literacy coach comes to her classroom to observe literacy lessons and provide feedback on what she could be doing better. Another teacher reported that she would likely use the literacy coach more if the coach were available to be in her classroo m more regularly. The literacy coach herself

PAGE 233

217 reported that she would like to be in more classrooms on a more regular basis but that time does not permit that type of regular contact with teachers. Another consultant who is available to assist teachers i n their classrooms is the technology consultant. The role of the technology consultant primarily includes helping teachers to identify appropriate instructional technologies for their students and then training them on the implementation of those technolo gies. She also works to identify new technology resources to further support the ELS Literacy Initiative. On the teacher survey, teachers gave the technology consultant a me an helpfulness rating of 3.8 on a 5 point scale the technology consultant in a similar way to how they use the literacy coach: for identifying and accessing materials (technologies, in this case), receiving training on the use of those materials, and probl em solving for difficult student cases. The only barrier that was identified to using the technology consultant to support literacy instruction in the classroom was time; the technology consultant can be difficult to access because of the number of classro oms she is serving.

PAGE 234

218 A variety of material and human resources have been made available to teachers as part of the ELS Literacy Initiative. The current evaluation question sought to determine whether teachers are using these resources, how these resourc es are being used and the factors that aid in the use of these resources or serve as barriers to their use. Data suggest that teachers find the ELS Literacy Scope and Sequence moderately helpful in planning instruction and that the primary barrier to usi ng the information involves to use core curriculum to be very helpful in providing instruction, but they find that only a few students in their classrooms have the prerequisi te skills to participate in the programs. Literacy benchmark data are being collected on all students three times a year, but teachers rated this resource as only moderately helpful and did not mention it when describing their instructional planning proce sses. The literacy tracking form that teachers only slightly to moderately helpful. However, this resource is still new to teachers and

PAGE 235

219 some are still not receiving the information reliably. With regard to human resources, teachers find a variety of professional development opportunities to be helpful, with definitely find the availa bility of the literacy and technology consultants helpful and use these consultants in a variety of ways. The only barrier to the use of the consultants is time, with teachers and the consultants themselves reporting that they would like the consultants t o spend more time in the classroom. Question 5: How do Teachers Rate Their Feelings of Support, Preparedness, and Confidence as a Result of the Resources That They Have Been Provided Through the Literacy Initiative? During the initial stakeholder interview the lack of teacher training and knowledge in best practice beginning reading instruction as a problem, including the teach ers themselves. The consensus was that teachers typically graduated from college unprepared to teach beginning reading, let alone to teach reading to students with significant learning differences and challenges. Their undergraduate training was either d eficient in training in this area, or it was not aligned with current best practices (e.g., practices that focus on sight word instruction, or literacy imbedded in life skill instruction). Therefore, one of the intended short term outcomes of the ELS Lite racy Initiative was an i ncrease in teacher skill and confidence in teaching beginning reading. In other words, it was expected that as a result of the curriculum materials that teachers were provided, the information and training they received on literacy and the coaching and continuing professional development they received on an

PAGE 236

220 ongoing basis, teachers would rate their feelings of confidence, support, and preparedness positively. The primary data source that was used to inform this evaluation question w as responses to questions on the teacher survey. Five questions on the survey were used to directly inform this evaluation question. When teachers were asked to rate the degree to which they agreed with the statement that they were adequately supported w ith sufficient material resources for literacy instruction in their classrooms (teacher survey question 4), 10 of the 13 teachers (76.9%) agreed that they were adequately supported and an additional 2 teache rs strongly agreed (see Table 16 ). Teachers resp onded similarly when asked to rate the degree to which they agreed with the statement that they were adequately supported through coaching and professional development opportunities (teacher survey question 7), with a total of 12 of the 13 teachers (93.3%) agreeing with the statement and the remaining teacher strongly agreeing with the statement (see Table 17 ). Taken together, these results suggest that the primary and intermediate teachers in the ELS program feel supported when it comes to providing liter acy instruction in their classrooms, both through the materials they have been provided and through the professional development opportunities that have been made available to them. Several comments that teachers provided during the focus group interviews also substantiate the claim that they feel supported, such as this comment from a primary teacher: When I first started, I like walked in my classroom and there were like 800 I

PAGE 237

221 anything, you know? since I started. I like the programs. It is clear that this teacher feels she has benefitted from the professional development and coaching opportunities that she has been provided with.

PAGE 238

222 In addition to reporting positive feelings of support, teachers also reported being relatively confident in providing literacy instruction in their classrooms. As s hown in Table 18, statement and 7.7% (1 teacher) had a neutral reactio n to the statem ent As stated by one sted that because of the training, as with this statement and an additional teach er strongly agreed (see Table 19 ).

PAGE 239

223 Overall, it appears that the primary and intermediate teachers in the ELS program feel supported, confident, and prepared to provide literacy instruction in their classrooms. None of the teachers disagreed or strongly disagreed with statements reflecting their feelings of confidence, support, or preparedness. Comments made during the focus group interviews, such as the following, also provide support for this conclusion: I think that what we have is like, a mazing. What has been created has been above and beyond what a lot of other programs have. Like I know that similar schools works at a school and she is asking me what do I have? Or what has been created? However, not all comments made by teachers were positive toward everything about the literacy initiative, such as comments made by teachers who expressed frustration with not having a curriculum for non verbal students, as discu ssed within other evaluation questions. It is important to note that differences did not appear to exist between how

PAGE 240

224 primary and intermediate teachers rated their feelings of support, confidence, and preparedness. Question 6: To What Extent is Instruction i n the Area of Literacy Aligned W ith Best Practices and Current Research as Reported by Teachers? Over the past twenty years, a tremendous amount of work has gone into conducting new research and synthesizing the existing research on effective reading ins truction for typically developing students and students with mild disabilities. Comparatively, very little research has been conducted on effective reading instruction for students with moderate to severe disabilities. Some of the conclusions that can be drawn, based on the research that does exist on literacy instruction for students with significant disabilities, are similar to what research has identified as best practices for students without disabilities, and some are different. Of the similarities, research suggests that all students benefit from comprehensive instructional programming including the five areas of instruction identified by the National Reading Panel (phonological awareness, phonics, fluency, vocabulary, and comprehension). This inst ruction should be direct and systematic and make use of data to inform instructional decisions. Ways in which research suggests literacy instruction should be different for students with moderate to severe disabilities include working from a broad definit ion of literacy, considering differences in the early life experiences of students with disabilities, honoring differing abilities in expressive and receptive communication skills, and incorporating the use of AAC when appropriate. To answer the current evaluation question, data from the surveys and focus groups have been analyzed to determine the degree to which the following are present in

PAGE 241

225 ELS classrooms: comprehensive literacy programming across the day, direct and systematic literacy instruction, use of data to inform instructional decision making, the adoption of a broad definition of literacy, the incorporation of technology and augmentative and alternative communication (AAC) strategies, and individualized instruction. Comprehensive Instructional P rogramming Comprehensive instructional programming refers to instruction that incorporates a variety of skills and teaches those skills through a variety of means. To determine whether students were receiving instruction across the skill areas targeted by the ELS Literacy Initiative, teachers completing the survey were asked to indicate which of the eight instructional skill areas were a part of their regular, systematic literacy instruction for each child in their classrooms. T eachers reported providing instruction in an average of 5.2 areas of literacy per student out of the possible 8 (see Table 20 ). Primary teachers reported somewhat more comprehensive programming, providing instruction in an average of 5.7 areas out of 8, and intermediate teachers re port ed providing instruction in slightly fewer areas (with an average of 4.6 out of 8 per student), representing less comprehensive programming.

PAGE 242

226 These data were also analyzed to determine in which of the eight skill areas teachers were most likely to s pend their instructional time (see Table 21 ). N otable differences could be seen in how primary and intermediate level teachers spent their instructional time. Primary students were most likely to be instructed in the areas of (89.8%). They were least likely to (57.1%). For intermediate students, there were four skill categories (phonics, sight words, comprehension, and vocabulary) that were similar in t heir frequency of instruction (ranging from 68.3 to 73.2%), representing the skills that are most likely to be taught at the intermediate level. 6%). These findings are in line with what would be expected for the skill level of students in primary and intermediate classrooms. Comprehensive instructional programming in the area of literacy takes place throughout the school day, not just during th e designated instructional time. As one

PAGE 243

227 teacher put s for the instruction to be effective. Teachers responding to the survey were asked whether literacy instruction (formal and in formal) was provided throughout the school day for none, few, half, many, or all of their students. The majority of teachers reported of their students (84.6%), with the other 15.4% reporting that of their students (see Table 22 ). Primary and intermediate teachers responded similarly to this question. The survey data suggest that teachers in the ELS program are providing instruction in a variety of literacy skills across the school day, indicating that they are providing comprehensive and integrated instruction. Qualitative data provide additional support for this conclusion. More specifically, one of the themes that emerged from the qualitative data analysis was the importance of providing comprehensive and integrated instruction. Teachers, parents, and support staff frequently identified the importance of providing literacy instruction that did not occur in isolation, but rather was integrated into

PAGE 244

228 students on individual skills, but rather provided instruction, or at least exposure, in many escribed her literacy instruction in the following way: You would see a balanced literacy program of all the components of Balanced Literacy: writing; working with words ; phonics ; group instruction, you know, guided reading; independent reading, you know lots and lots of printed materials, printed material with visual supports. Another teacher described her perspective on comprehensive and integrated instruction when she emphasized the importance of providing exposure in a variety of skill areas, inclu syllable words and more It is important to note that at least two teachers reported feeling like the ELS program does not adequately emphasize comprehensive and integrated literacy instruction and instead promotes isolated skill instruction. As one teache r stated: ; not talking about what students do learn in a reading program. They le arn about the author, they explore this, they explore the genre, they explore the concepts in literature.

PAGE 245

229 Some teachers expressed their feelings that the Direct Instruction programs tha t were identified as the core instructional programs in ELS classrooms reinforce this message of skill instruction in isolation. One teacher also reported feeling like the program places do not generalize to other settings or link to other areas of literacy development. Literacy instruction in the primary and intermediate classrooms of the ELS program appears to be comprehensive and integrated throughout the school day. Students at the primary level receive instruction in an average of 5.7 of 8 areas of literacy, which is most likely to include instruction in book awareness and letter identification. Students at the intermediate level receive instruction in an average of 4.6 of 8 areas of literacy, which is most likely to include instruction in phonics, sight words, comprehension, and vocabulary. The importance of comprehensive literacy instruction that is integrated throughout the school day was a theme that emerged from the qualitativ e data analysis. However, some teachers believe that the ELS program and the literacy initiative are not promoting instruction that is integrated, feeling that these programs instead place emphasis on discrete skill instruction. Direct and Systematic I n struction Several data sources were examined to determine the extent to which ELS teachers are providing systematic and direct instruction in the area of literacy, including questions taken from the teacher survey and qualitative data collected as part o f the focus group interviews. The surveys included a question that asked teachers whether they had a designated time for literacy instruction in their schedule for none, some, half, many, or all of their students (see Table 23 ). Twelve out of 13 of the r espondents indicated that

PAGE 246

230 of of her students. These data suggest that teachers are schedul ing time to provide direct instruction in the area of literacy. One intended outcome of the ELS Literacy Initiative was an increase in direct and systematic literacy instruction in the ELS program through the identification of a core literacy program. Two Direct Instruction programs, Reading Mastery and Language for Learning, were selected as the core instructional programs for literacy and language. All primary and intermediate classrooms have copies of these programs. The extent to which these progr ams are being used for literacy instruction in the ELS program is reported on question, it suffices to say that teachers report that the DI programs are currently being use Table 26 ). When teachers are not using these programs, it is difficult to determine the extent to which the instruction that is taking place is systematic and direct. However, determining that the majority of teachers d o

PAGE 247

231 have a set time in their schedules to provide literacy instruction for all of their students is an important indicator that direct instruction in literacy is taking place. Data to Inform Instructional P lanning Two of the desired short term outcomes o f the ELS Literacy Initiative were that every student would have a written instructional plan in the area of literacy and that teachers would increase their use of data when engaging in instructional planning. Teachers completing the survey were asked whe ther they had written instructional plans for none, few, half, most, or all of their students. The responses to this question were varied, with half of the respondents indicating that they had a written literacy heir students and the other half reporting that or all of their students (see Table 24 ). Primary and intermediate teachers displayed a similar pattern of responses. It is clear from these data t hat the desired outcome of all students having a written instructional plan has not been obtained, and that some teachers do not have written plans for any of their students. Only two teachers responding to the survey reported that they had written instru

PAGE 248

232 The degree to which teachers in the ELS program are using data to inform their instructional decision making is discus analysis of the data suggests that teach ers in the ELS program usually or always use data when developing their instructional plans and determining when changes ne ed to be made to those plans When asked about which sources of data they find most helpful when making these decisions, teachers re ported that the data resources that have been made available through the ELS Literacy Initiative (i.e., literacy benchmark data, mastery tests in the curriculum, and information on the literacy tracking form) were the three least helpful data sources. The data sources that teachers find most helpful include IEP goal Teachers also rated the helpfulness of two other sources of data: inf ormal teacher observations and discrete t rial data. With regard to answering the current evaluation question, and specifically the use of written instructional plans and data to drive instructional decisions, teachers clearly fall short of having a written literacy plan for all students, and while they appear to be using data to help them make instructional decisions, they are not finding the data sources

PAGE 249

233 that have been made available through the implementation of the ELS Literacy Initiative particularly helpful. Broad Definition of L iteracy Downing (2006) suggested a broad and inclusive de finition of reading and literacy for students with moderate to severe disabilities that includes not only reading and comprehending connected text, but also all communication, encompassing all activities related to learning about and sharing information wi th others. In this sense, literacy means reading, writing, speaking, and listening. One of the themes that emerged through the qualitative data analysis was that of a strong connection between literacy and language. It clearly shows that when students a re learning their letter sounds and answering comprehension questions, they are working on reading and literacy skills. However, for nonverbal students, or students who are at the beginner or pre emergent developmental levels of reading, it is not always so clear when they are working on literacy skills. However, these students are almost always working on their expressive or receptive language skills, learning to follow a picture schedule, or identifying picture symbols that are on their communication de vices or in their environment, all of which are part of literacy instruction for students with moderate to severe disabilities. During the support staff focus group interview, participants described the language/literacy connection and how they are work ing with teachers to strengthen this connection in their instruction in order to improve student outcomes. For example, as one support staff member described, I go in and I try telling people who are saying w ait a second, th ey are following a schedule. You are asking them to match symbols

PAGE 250

234 to words. You are asking for them to show comprehension of that word by matching it to the symbol. So making people more sensitive to the fact that everything that you guys just talked ab out is totally what literacy is. One speech and language pathologist also identified the connection when she described how all of the materials and visual supports that she uses to support language in the classroom build literacy, whether she is using wo rds, pictures, or following a schedule. One teacher during the intermediate teacher focus group interview reinforced this idea when she expressed a desire to work more with the speech and language pathologist: Idealistically I would like to see more of th e speech pathologist involved in literacy. I think that would be great because they go so well hand in hand. You know, I think, umm, just planning together, attacking concepts together . I think would be kinda cool. One area of improvement that wa s mentioned by a support staff member involved strengthening the instructional connection between student language/vocabulary skills and reading comprehension. A number of students in the ELS program are able read connected texts that are at a much higher level than they are able to comprehend with their limited language skills. One support staff member believes this connection is not being adequately emphasized, saying I think we need more professional development in how the language skills support read ing skills. I think that a lot of times teachers look at them as two different comprehension, but 90% of the time they go hand in hand.

PAGE 251

235 Defining literacy broadly and recognizing the connection between language skills and literacy skills is the best practice in literacy instruction for students with moderate to severe disabilities. It appears as if some staff mem bers in the ELS program embrace a broad definition of literacy and recognize the importance of this connection, working to strengthen this connection through collaboration and discussion. However, from the comments made by support staff, it appears as if there is room for improvement in this area and that additional professional development around the language/literacy connection may be needed. Incorporati ng AAC and Te chnology Many students in the ELS program have special needs that require the use of technology for them to fully access instruction, ranging from augmentative and alternative communication (AAC) devices to computer programs with touch screen access or modified keyboards. When considering literacy instruction for this population of studen ts, it is necessary to incorporate those technologies that allow them to access the instruction. Teachers and support staff reported using technology to support instruction in a variety of ways. For example, one teacher described how she uses technology to differentiate her group lessons. By using devices that play pre recorded messages when all students are able to participate in the group lesson by answering questi ons, making choices, etc. A speech and language pathologist described how she incorporates literacy words should be selected for their IEP goal, she reported,

PAGE 252

236 The first thing I do is look at their device. Well, what are you expecting them to say using their device? If you are expecting them to say these things then these that they are goi ng to use in the most contexts . Several teachers mentioned computer software programs that are available to support literacy instruction in the classroom specifically, IntelliKeys and Classroom Suite. IntelliKeys is a keyboard that provides access for students with physical, visual, or cognitive disabilities who have difficulty using a standard keyboard. The keyboard can be adapted for instruction by using overlays that are created by teachers. Classroom Suite is an interactive instructional progra m that was designed to be accessed by any type of learner through alternative keyboard and switch inputs. Teachers generally commented on the benefits of using such programs to support instruction, but they frequently referred to the additional time and t raining it takes to use them effectively in the classroom. As one teacher stated so sitting there and then you walk out and it seems like such a huge thing and I never get to i t. So, one of my suggestions for training in general. . If we basically got trained, and then sometime that same week we got a whole day to just knock out and just do it. Set it all up. Several teachers commented that the new version of Classroom Suit e helps to eliminate some of the preparation time that other technology supports require by having pre made and LS program as another

PAGE 253

237 way to overcome the barriers to using technology. These workshops train the teachers in how to use the specific programs and then allow time during the training to use the program and begin to create materials that are ready to use i n the classroom the next day. Teachers were universally enthusiastic and appreciative of the make and take professional development opportunities. Interestingly, when asked how the ELS program could improve literacy outcomes for students, two parents sp ecifically mentioned technology. As stated by one parent, own communication device, s and assistants need more support to work with assistive technology. More could be done appears that there is roo m for improvement with regard to incorporating and taking advantage of technology to support literacy instruction in ELS classrooms. The technology that is available to support instruction is vast and constantly changing, and constant professional develop ment is required for teachers to be fully aware of what is available and to gain the knowledge and experience to be able to use the available technologies. Instructional I ndividualization One theme that emerged through the qualitative data analysis was that of instructional individualization. Teachers reported individualizing student instruction in a variety of different ways, including making adaptations and modifications to instructional programs, finding instructional strategies that interest and eng age a child, and making

PAGE 254

238 instruction personal by connecting the content to real life events. For example, one teacher who is using the Reading Mastery program with two of her students has found that she has to significantly modify the instructional deliver y of the program to engage her students. For one student, she has incorporated bouncing a ball back and forth to practice the letter sounds, and she allows another child to choose between two books to read after completing a portion of a lesson. While th ese modifications take up a portion of the instructional delivery time, the teacher is getting students engaged, which is essential for students to learn. Teachers also reported making the instruction more meaningful, through practices such as selecting w for sight word lists, or creating books about the community that incorporate places where that particular student goes on a regular basis. Teachers in the ELS program make a great effort to individualize the ins truction for their students, all of which is done to make the instruction more accessible for the students and increase their motivation to learn. Teachers clearly have the best interest of the children in mind and are working hard to provide effective in struction. Of course, all of this individualization requires additional time and effort, which has been identified as a barrier to implementation of the ELS Literacy I Summary The current evaluation question asks to what degree literacy instruction in the ELS program is aligned with best practices. The data suggest that students are receiving comprehensive literacy instruction that takes place throughout the school day. It pr ovides evidence that some students are receiving direct and systematic instruction, particularly

PAGE 255

239 those students who are participating in the identified core curriculum. However, for those students for whom the core curriculum is not appropriate, it is dif ficult to determine the degree to which their instruction is direct and systematic. It appears as if teachers are falling short when it comes to having a written instructional plan for every student, but teachers do report using data to inform their instr uctional decision making. The sources of data that are used for this purpose vary, but teachers mainly depend on data such as IEP goal progress data and informal teacher observations. Data suggest that most teachers and staff in the ELS program have embr aced a broad definition of literacy and recognize the strong connection between language and literacy. Teachers describe using some computer programs and other technolog ies to support literacy instruction, but improvements can be made in this area. Final ly, teachers report individualizing literacy instruction for students based on student need. Based on these data, it appears as though many students in the ELS program are receiving literacy instruction that is aligned with best practices, but that instru ction for some students, especially those who are nonverbal or who are at the beginner and novice stages of literacy development, can be improved. Questio n 7: To What Extent do Students in the ELS Program Have Access to Appropriate, Research Based Literac y Instruction as Reported by Teachers? One of the desired outcomes of the ELS Literacy Initiative is for literacy instruction in the program to be more closely aligned with the research base on effective literacy instruction for students with disabilities. One of the primary ways the initiative has addressed this intended outcome was through the introduction a core literacy instructional program. The other was through the creation of the ELS Literacy Scope and Sequence.

PAGE 256

240 Prior to the ELS Literacy Initiat ive, the ELS program lacked a core curriculum in content of that programming varied widely within the program. One major change with the ELS Literacy Initiative was the identification and purchase of a core language and reading program. The curriculum selected for the initiative included two Direct Instruction programs, Language for Learning and Reading Mastery, the first of which targets language development and vocabul ary, and the second of which targets phonemic awareness, phonics, and fluency. While somewhat limited, researchers studying the use of Direct Instruction programs to teach language and reading to students with low incidence disabilities have found positiv e outcomes (Bracey, Maggs, & Morath, 1975; Bradford, Alberto, Houchins, Shippen, & Flores, 2006; Flores, Shippen, Alberto, & Crowe, 2004; Gregory & Warburton, 1983; Maggs & Morath, 1976). The identified Direct Instruction programs were selected as the pro (a) the research support for the programs, (b) the structured and predictable nature of the instruction within the programs, and (c) the positive results realized from piloting the programs. As part of the literacy initia tive, a curriculum set was purchased for every primary and intermediate classroom. Another curricular resource provided through the initiative to all primary and intermediate teachers was the ELS Literacy Scope and Sequence The ELS Literacy Scope and S equence is a document that was created in order to guide teacher instruction in the area of reading. The document identifies the stages of literacy development (novice, beginner, early to upper emergent, and upper emergent to fluent) in seven different ar eas of literacy (concepts of print, letter identification, phonemic awareness,

PAGE 257

241 phonics, fluency, vocabulary and comprehension, and writing). The ELS Literacy Scope and Sequence discusses assessment strategies and instructional recommendations, and it serv es as an instructional resource for all students. The ELS Literacy Scope and Sequence reflects current research on effective literacy instruction and was specifically designed to address the unique needs of students with moderate to severe disabilities. Along with this document, teachers also received corresponding instructional and assessment resources that came in the form of a binder series. These resources were developed and organized to complement the ELS Literacy Scope and Sequence To answer th e current evaluation question, several data sources were examined, including the t eacher s urvey. Teachers who completed the survey were asked to rate the degree to which they agreed with the statement that the students in their classrooms have access to a ppropriate, research based literacy instruct ion (see Table 25 ). The majority of classrooms have access to appropriate, research based literacy instruction, and the responses was similar between primary and intermediate teachers. Clearly teachers feel that their students have access to research based instruction, which is an impor tant outcome of the ELS Literacy Initiative.

PAGE 258

242 All primary and intermediate teachers in the ELS program have copies of the identified core curriculum as well as the ELS Literacy Scope and Sequence and teachers completing the survey reported that the s tudents in their classrooms have access to appropriate research based instruction. The next important step to answering the current evaluation question was to determine whether the Direct Instruction programs and ELS Literacy Scope and Sequence were being actually being used to instruct students. On the teacher survey, teachers were given a list of instructional resources in their classrooms and were asked to indicate the number of students with whom they were using the materials, with response options ra ll of my ).

PAGE 259

243 Use of Direct Instruction P rograms Of the nine instructional resources that were identified on the survey, Reading Mastery and Language for Learning ranked sixth and seven th res pectively in terms of the mean rating of the number of students with whom teachers were using the program (see Table 26) The mean rating for both curricular resources was 2.4, which suggests that udents. Data suggest that intermediate teachers are using the Reading Mastery program ( M = 2.7) with slightly more students than the primary teachers ( M = 2.1). The ratings for use of the Language for Learning program were more similar between primary an d intermediate teachers. Because the Direct Instruction programs require that student s have a certain level of prerequisite skills for appropriate participation in the program (e.g., can orally repeat small phrases, can respond on cue, etc.), it is not expected that teachers be able to use

PAGE 260

244 these two programs with all, or even half, of the students in their classrooms. However, the expectation is that the programs will be used as a core curriculum and that as many students as possible will be placed into the programs. Qualitative data provide some additional insight as to how teachers are us ing the Direct Instruction programs in their classrooms and possible reasons for why they are using or not using the programs. In their comments, some teachers recognized that the Direct Instruction programs are one component of a comprehensive instructio nal program in the area of literacy and should not be the only instructional resource/strategy used n it comes to teaching teachers who are using Reading Mastery and Language for Learning in their classrooms, they cited several reasons for doing so, including the be nefit of having a program that is ready to use and therefore does not require time to create it; the opportunity to have TAs in their classrooms deliver the inst ruction, since the Direct Instruction programs are scripted; and the fact that teachers and TAs have ready access to training in the use of the program. Teachers also reported on reasons t hat they may not be using the Direct Instruction programs for literacy instruction for some or all of their students. One teacher cited the instructional deliver Instruction, but it takes the creativity out of it. It takes the impromptu modifications that commenting on th e lack of use Direct Instruction programs in some ELS classrooms

PAGE 261

245 suggested that the administrators in the ELS program take more of a stance on the expectation of the use of the programs. One of the most frequently mentio ned factors for not using the Dire ct Instruction programs was the fact that the programs can only be used with verbal students. Teachers across the board are frustrated with the lack of research based instructional resources for students who are non verbal or are not developmentally ready to participate in a structured program. As one teacher stated: No, seriously, because if you only have Direct Instruction for our verbal kids and work. Use of ELS Literacy Scope and Sequence The ELS Literacy Scope and Sequence binders were ranked fifth of the nine instructional resources for frequency of use by teachers and had an avera ge usage rating their students. There was a difference in the use of the ELS Literacy Scope and Sequence b inders between the levels, use of this resource with their students and th e intermediate teachers only having an average rating of 1.8 for this resource. The ELS Literacy Scope and Sequence and corresponding binders had been developed as an instructional resource that could be used to help guide and develop instruction for all students in the ELS classroom. Therefore,

PAGE 262

246 the average teacher rating of 2.5 and the fact that five teachers reported not using the resource at all falls well below the desired level of use for this particular resource. During a focus group interview, on e primary teacher commented on why she has found the use of the ELS Literacy Scope and Sequence helpful in her instructional planning: where they should go next. Cause I wa s like, okay, they have their letter sounds,; kind of referenced it [the ELS Literacy S cope and S equence ] and thinking as far as me to be, all right, they are here; let me move here now. That makes more sense. goals and kind of jumping around and seeing with out really knowing, like, exactly what the progression is. So I think that helped. The comments from this particular teacher describe how the ELS Literacy Scope and Sequence was intended to be used: as an instructional guide for identifying where to go ne identify appropriate instructional strategies once they know what skills to be working on. Several teachers commented on why they were not using the ELS Literacy Sc ope and Sequence and corresponding literacy binders, and they all generally referred to the fact there is so much information contained in the scope and sequence and corresponding binders that the information is overwhelming to access. Another teacher rep orted that she depends on the intervention specialist in her classroom to use and interpret the

PAGE 263

247 information in the ELS Literacy Scope and Sequence and binders and then to pass the information along to her. Use of Other Instructional R esources Other instr uctional resources that teachers were asked to report on regarding how frequently they used them for students in their classrooms included (in order of use) teacher created materials ( M = 5.0), literacy websites for students ( M = 3.8), literacy websites fo r teachers ( M = 3.6), Reading A to Z (a subscription website) materials ( M = 3.2), Wilson (an adapted version of the reading program) ( M = 1.5), and Meville to Weville (an instructional program in the primary classrooms) ( M = 1.4). A follow up question on the survey asked teachers to identify other instructional resources that they used in their classrooms that were not on the list. A few teachers responded to this question, referencing instructional resources such as resources used as part of the general education curriculum, the Reading Milestones program, and other literacy websites for teachers. It appears from the survey data that teachers in the ELS program continue to rely heavily on resources such as teacher created materials and web based instruc tional programs and resources when providing literacy instruction in their classrooms. A support staff person corroborated this finding during a focus group interview when she stated: made materials that are age a ppropriate, have enough repetition, and are at the right level for the student, they make a lot of their own materials that allow the students to participate in different ways.

PAGE 264

248 Teachers consistently referenced the need to use teacher created materials to s upport instruction because of the unique learning needs of the ELS student population. However, teachers also commented on the time and effort that it takes to make these materials. Many teachers complained on the surveys and during the focus group inter views about their lack of ability to share teacher created materials among ELS staff members. Because the ELS program is located in about 15 different school districts, instead of being in a centralized location, sharing electronic materials on a large sc ale basis has been problematic. Teachers view this difficulty as a significant barrier and expressed their desire to do more collaborative sharing of materials among ELS staff members. Summary The current evaluation question asked the extent to which s tudents in the ELS program have access to appropriate, research based literacy instruction. Two specific resources were made available to teachers as part of the ELS Literacy Initiative in order to increase the degree to which instruction in the program w as research based: the Direct Instruction programs Reading Mastery and Language for Learning, and the ELS Literacy Scope and Sequence and corresponding resource binders. Teachers in the program either agreed or strongly agreed that students have access to appropriate, research based instruction, but their reporting of the frequency of use of the two identified resources (DI programs and ELS Literacy Scope and Sequence ) was low compared to programmatic expectations. With regard to the lower than expected u se of the DI programs, ELS staff most commonly cited student characteristics as the primary barrier for not using the program, but they also cited as other possible reasons for non

PAGE 265

249 instructional delivery and a perceived lack of expectatio n from the program administration that the programs would be used. With regard to the ELS Literacy Scope and Sequence teachers cited being overwhelmed by the information as the primary reason for non use. It appears that some students in the ELS progr am have access to and are being instructed using appropriate, research based programs and strategies. More specifically, for the students who are being instructed using the DI programs, and for those students whose teachers are using the ELS Literacy Scop e and Sequence to guide their instruction, there can be some confidence that these students are receiving instruction that is aligned with the research base on effective instruction for students with disabilities. However, many students are not accessing the identified core curriculum, and teachers report low usage of the ELS Scope and Sequence so it appears that for large number of students in the ELS program, the same conclusion cannot be made. This does not mean that their literacy programs are not al igned with current research, but rather, because of the heavy reliance on teacher created materials, and reliance on instructional programs that have little or no research support, it is difficult to determine the extent to which their instruction is resea rch based. The largest group of students for whom this is clearly the case appears to be students who are nonverbal or are otherwise not able to participate in the identified core curriculum. Overall, teachers in the ELS program appear to want to provide the best literacy instruction possible for their students and are doing their best to provide quality, research aligned instruction in this area.

PAGE 266

250 Question 8: How are Literacy Data Being Used in the Classroom and How do Teachers Rate Those Data Sources i n Terms of Helpfulness? The use of a problem solving decision making model based on student data is considered to be a best practice in delivering effective instruction (IDEA, 2004). One of the desired short term outcomes of the ELS Literacy Initiative wa s to increase the use of data in instructional decision making. ELS teachers have access to a number of data sources in their classrooms to help plan instruction and make instructional decisions. Some sources of data have been put into use as part of the literacy initiative, such as the literacy benchmarking data that are collected on all students three times a year. Other sources of data, such as progress on IEP goals and informal teacher observations, have always been available to teachers to support t heir instructional decision making. Stakeholders in the ELS program identified the current evaluation question with the intent of determining which data sources teachers found most helpful and how these data sources were being used to support instructiona l decision making in the classroom. The teacher survey asked primary and intermediate teachers in the ELS program to report on how frequently they used data in developing student literacy instructional plans and how frequently they used data to make cha nges to those instructional plans. With regard to how frequently they used data to develop instructional plans for their students, most teachers reported "usually" (7 teachers) or "always" (5 teachers), with the final respondent reporting that she uses da ta to inform instructional plans "about half the Table 27). When asked how frequently they used data to help them decide when to make instructional changes, teachers responded similarly, with 53.8% (7 teachers) responding "usually" and 46.2% (6 teachers) r

PAGE 267

251 28 ). Primary and intermediate teachers responded similarly to both questions. The comment of one teacher made during a focus group interview reinforces these results: they are using data on a regular basis to inform their instruction. se of data to inform instruction, teachers completing the survey were asked to rate how helpful they found various sources

PAGE 268

252 option fo ). Differences became apparent in the ratings of primary and intermediate teachers, with primary teachers rating the combined sources of data more helpful overall ( M = 4.3 ) than intermediate teachers ( M = 3.5). H owever, their responses showed similar patterns. When the data from primary and intermediate teachers were combined, all sources of data received an average rating of 3.3 or higher, which would be equivalent to a rating of The sources of data that teachers rated as the most M = 4.5 ) and IEP Goal Data ( M = 4.4 ). The IEP goal data are progress monitoring data that teachers collect on a frequent basis specifically to inform student progr ess on their IEP goals. The regular collection of data to inform student progress on IEP goals is standard practice in the ELS program, not just t of that would be different from the other choices that teachers were presented with on the survey. None of the teachers clarified this point when given on the survey However, during the focus group interviews other sources of data were includes sections o n reading and writing). However, little data suggests that this tool is widely used in the ELS program as a data source for helping to plan literacy instruction. Literacy T M = 3.3 M = 3.6 ), and

PAGE 269

253 M = 3.6 ). The literacy tracking form is completed annually are emergent, and upper emergent to fluent) in seven different literacy skill areas. This form ent instructional programming. This form had only been in use for one school year before the teachers were asked to complete the survey and rate its helpfulness. Teacher comments during the focus group interviews suggest that not all teachers completed t he form directly after the first year it was in use and that on some occasions, literacy tracking forms were not passed on to the next teacher when students transitioned within the ELS program. The mastery tests in the curriculum are the regular assessmen ts that are included in the Direct Instruction programs, which not all teachers are using with their students. The literacy benchmark data are curriculum based assessments that are given to all students in the ELS program three times a year. The literacy benchmark data The other two sources of data that teachers were asked to rate the helpfulness of e neither the highest, nor the lowest rated in terms of helpfulness. Discrete trial data are data that are kept on student progress when students are being instructed in a one on one setting using the discrete trial instructional method.

PAGE 270

254 Teachers in the ELS program report either usually or always using data to create student instructional plans and inform decisions as to when and how those instructional plans should be modified. Based on teacher ratings of helpfulness, it appears that the data that t eachers find most helpful in the instructional decision making process are the options on the survey that had been least standardized and research based and which have always been available to teachers in the ELS program, as opposed to other data sources such as literacy benchmark data, mastery tests in the curriculum, and informatio n on the literacy tracking form (the three lowest rated data sources), which have all been implemented as part of the ELS Literacy Initiative. One of the desired short term outcomes of the literacy initiative was to increase the use of data in making inst ructional decisions, which teachers report doing on a regular basis. However, the data sources that

PAGE 271

255 teachers are using to inform their instruction are not those that have been made available through the ELS Literacy Initiative. Question 9: How are the L iteracy Benchmark Data Being Utilized? Are the Data Adequate to Support These Uses? Consistent literacy benchmark data have been collected on all students in the ELS program three times a year since the 2005 2006 school year. The primary purpose of bench marking student literacy skills has been to ensure that students are making consistent growth in the area of literacy and to help to indicate when instructional changes may be necessary. The second potential use of the ELS literacy benchmarking data is pr ogram evaluation, or to determine how students overall are achieving in the area of literacy. The current question was included in the evaluation in order to determine whether the benchmarking process has been able to meet both the needs of teachers, supp orting instructional decision making in the classroom, and the needs of the ELS program administration, for use as an indicator of student outcomes. The collection of benchmark dat a is the responsibility of the intervention specialist in eac h of the ELS c lassrooms. The intervention specialist collects the benchmark data on all students three times a year using adapted versions of curriculum based measurements. The administration and scoring of all of the tools are standardized. Ten different assessment tools are available to use for benchmarking, ranging from tools that measure pre literacy skills (i.e., Concepts of Print) through tools that measure oral readin g fluency (i.e., R CBM). The intervention specialist selects the tool to use for benchmarking has not yet mastered but is expected to make growth in throughout the school year. The

PAGE 272

256 assessment tools have been adapted to serve as accuracy based measures, rather than fluency b ased measures, with the exception of Reading CBM. Furthermore, several One goal of this eval uation question was to determine how teachers were using the literacy benchmark data to inform their instruction in the classroom. Two sources of data were examined to help answer this aspect of the evaluation question: ratings from the teacher survey alo ng with qualitative data collected through the open ended questions on the teacher surveys, and information shared during focus group interviews. When prompted by the survey to rate the degree of helpfulness of different data sources, teachers rated liter acy benchmark data as the second from the lowest of seven possible that of the 56 comments made in reference to instructional planning, only two specifically mentioned the use of the literacy benchmark data. Such low mention of the literacy benchmark da ta is interesting, given that intervention specialist s in the ELS program have be en expected to collect these data on all students in the program three times a year for the past three years. Of the two comments that were made, one came from a teacher who reported that the benchmark data had not really been collected on her students fo r the current year. An examination of the benchmark data suggested that during the 2007 2008 school year, complete literacy benchmark data (data from all three benchmark periods) were collected for 79 of 108 (73.1%) of primary and intermediate students in the ELS program. The same teacher further commented that during the 2005 2006 school

PAGE 273

257 year, when the data were collected, she did not find the data helpful because of the narrow focus of the assessments, which prevented her from using the information to i nform instruction. T he other comment came from an intervention specialist who verbally described the process of how she collected the data and shared it with her teachers in addition to other information about how she generally supports instructional plan ning in the classroom. From the results of the teacher survey and the qualitative data analysis, it appears that teachers in the ELS program may not find the literacy benchmark data helpful in planning and changing their instruction, given their low rating s of helpfulness and their sheer lack of mention of the data source when asked about the data that they use to drive instruction and their instructional planning process. As described in achers reported as being cher ). The first part of the current evaluation question was used to determine how the literacy benchmarking data were being used, and the second was used to determine whether the data were adequate to support these uses. With regard to teachers using the benchmark data to support instructional decision making in their classrooms, little data exists to suggest that teache rs are using the benchmark data for this purpose. It is possible that teachers are not using these data to help them make instructional decisions because the data are not adequate to support those types of decisions. More specifically, as one teacher men tioned, the data collection process is very narrow, with data being collected only on one specific skill area three times a year. If teachers are trying to use these data to create instructional plans, they are likely going to need additional data to

PAGE 274

258 prov planning process. Responses also provided little evidence that teachers are using the literacy benchmark data to determine when changes need to be made in their instruction Again, this may be because the literacy benchmark data do not adequately support this type of decision making. More specifically, the data are only collected three times a year (fall, winter, and spring), and teachers may need more frequent assessment of student progress in order to make determinations about when to change their instruction. Furthermore, because the majority of the literacy benchmark measures were accuracy based and not fluency based, they may not be as sensitive to growth as more trad itional curriculum based measures. The accuracy based nature of the assessments also creates a ceiling of performance for each of the measures. Consequently, this may further limit the utility of the data for making frequent decisions regarding instructi onal strategies in the such as IEP goal progress monitoring data and informal teacher observations, more helpful in planning their instruction and determining when c hanges must be made. The literacy benchmark data can also potentially be used as a programmatic measure of outcomes regarding the success of literacy instruction in the ELS program. To determine whether the literacy benchmark data could be used for this purpose, the current evaluator began by examining participation rates based on administration format in each of the benchmark tools. (See Table 1 for the 2007 2008 literacy benchmark assessment tool, even when considering both primary and intermediate students and all

PAGE 275

259 administration formats combined. The low numbers of students assessed with any given assessment tool and within any administration format preclude the ability to use the data to make conclusions regarding literacy development and outcomes in the ELS program as a whole. Several other problems with this data set also limit the usa bility of the data for answering questions about student outcomes in the program. More specifically, because the benchmark measures are accuracy based and not fluency based, the tools do not have the sensitivity to growth over time that the more tradition al versions have. In addition, the tools have a ceiling of performance (100% accuracy), whereas fluency measures have no such ceiling, which again limits their sensitivity to growth over time. Finally, because the tools have been altered from their origin al format, no normative information exists to which student performance with the tool can be compared. Therefore, it is difficult to concluded that the literacy benchmar king data are not adequate to support the use of the data as a measure of programmatic outcomes in the area of literacy because of the limitations identified above. Question 10: To What Extent do Teachers Believe There is Instructional Continuity for Ind ividual Students as They Move From One Teacher to the Next? An increase in instructional consistency from one classroom to another in the ELS program was one of the desired intermediate outcomes, meaning it would be achieved within two to three years afte r the formal start of the ELS Literacy Initiative. This evaluation question pertains to the question of whether current teachers in the ELS program believe that this outcome has been achieved and whether instructional continuity occurs from one teacher to the next.

PAGE 276

260 On the teacher survey, teachers were asked to rate the extent to which they believed there was instructional continuity when students transitioned from one teacher to seen in the ways in which primary and intermediate teachers responde d to this question (see Table 30 ). The majority of the primary teachers (5 of 7) respo know the extent to which there was instructional continuity, and the other two teachers that four of seven primary teachers were new to the ELS program during the 2007 2008 school year and had not had any experience with transitioning students to another teacher. The primary Rather, the students typically come to their classroom s from an early childhood program that may not have had a literacy program in place. In contrast to the primary teachers, instructional continuity that was present, and the other intermediate teachers responded less positively than the primary teachers who did provide a response. More specifically, Figure 1 for a graph of the differences in responses between the primary and intermediate teachers on this survey question.

PAGE 277

261 Figure 1 Primary (n = 7) and intermediate (n = 6) from one teacher to the next in the ELS program. When given an opportunity to provide an open ended response on the teacher survey, and when asked within the focus groups, several teachers made comments

PAGE 278

262 regarding the instructional continuity of the program. I nterestingly, most of the comments made reference specifically to the use of Direct Instruction programs. A few teachers expressed frustration with how instructional programs such as the Direct Instruction programs failed to carry over from one teacher to the next. The comment from one teacher during a focus group interview illustrates this perception: It really bugs me because I feel like I have put a lot of time into Direct Instruction programs and for certain teachers that my students go to and they do like, I just worked three years on making them successful and they were successful at this program and now they are not going to do it? Like, that makes me really upse t. In reference to improving the instructional continuity, one teacher commented that it is easier to transition a student who is using a Direct Instruction program, and another teacher suggested that her Program Supervisor (the administrator that she repo rts to in the program) has been helpful in making sure that Direct Instruction programs are carried over. A comment by another teacher suggested that the administration should take more ional continuity] more of a requirement at NSSED. . like it has data suggest that while the use of instructional programs for students may have facilitated instructional continuity from one teacher to the next in the ELS program, there is still room for improvement in this area.

PAGE 279

263 Question 11: To What Extent do Teachers Believe the Activities of the ELS Literacy Initiative Have Impacted the Inclusion / Integration of Students in ELS Into District Cla ssrooms? One of the desired outcomes of the ELS Literacy Initiative, as identified by the client of the current evaluation, was the facilitation of inclusion and integration opportunities of students in the ELS program into their school community. In othe r words, it was expected that if student literacy instruction in the ELS program was aligned with research based practices and students were experiencing improved academic gains in the area of literacy, then these factors would facilitate the inclusion and integration process and outcomes. Teachers who completed the survey were asked their opinion of the extent to which the literacy instruction in their classrooms had improved the inclusion/integration of their students into district based classrooms. appeared to differ somewhat on th eir perceptions of the extent that inclusion/integration opportun ities had improved (see Table 31 ). More specifically, primary teachers agreed more overall with the statement that the instruction that had taken place in their classrooms has improved the i ntegration/inclusion of their students into district statement, while one repor intermediate teacher resp ).

PAGE 280

264 Figure 2 Primary (n = 7) and intermediate (n = 6) instruction that has t aken place in their classrooms has improved the inclusion/integration of students into district classrooms.

PAGE 281

265 The responses that teachers provided during the focus group interviews also reinforced the finding that the intermediate teachers feel less positiv ely about the impact of literacy instruction on integration and inclusion when compared to the primary teachers, and it provided some initial explanation as to why this might the case. More specifically, when intermediate teachers commented on the impact on integration and inclusion, several of them expressed the feeling that integration and inclusion at the intermediate level is difficult because of the significant skill difference between the ELS students and their typically developing peers in grades th ree to five. As one teacher d to a gap in skill level as a barrier to inclusion/integration. The gap in student skill development is likely to be greater at the intermediate level than at the primary level, which may help to explain why teachers at the intermediate level are less po sitive about the activities related to the ELS Literacy Initiative that affect integration and inclusion opportunities. One factor that does appear to have a positive impact on the inclusion and integration of ELS students into their school communities is student behavior. During the focus group interviews, several teachers indicated that one of the ways in which the literacy instruction in their classrooms had affected student outcomes was through an ample, when asked about the se behaviors than being able to read the

PAGE 282

266 such as the increased ability to simply sit and listen to a story being read aloud, had allowed her to integrate her students more i nto the intermediate classroom, specifically Another factor that appears to have a positive impact on the inclusion and integration of students into district based classrooms i s a connection between the ELS literacy instruction and the general education literacy curriculum in that building. For example, one teacher described how she gets the Word Wall list from the general education teacher and then teaches the same words in th e ELS classroom so that when the student is integrated into the general education classroom, he or she has already gained familiarity with the words and the integration experience becomes more meaningful. One theme that emerged from the qualitative data a nalysis process was that of instruction that is provided within a broader perspective. This broader perspective can mean having broader perspective was the general e ducation perspective. Teachers reported accessing the general education curriculum, using web based resources that are intended for general education teachers, and attending reading conferences that are intended for general educators all as ways in which they gain the general education perspective. Teachers who seek out the general education perspective may be better prepared to support the integration and inclusion of their students into general education classrooms. During the initial client interview, the program administrator expressed a desire that with the use of Direct Instruction programs, the students in the ELS program could

PAGE 283

267 same Direct Instruction programs. Currently there is little evidence that this is resource classroom for some of their literacy instruction, but this decision had been made ding skill levels than on the program they were being instructed in. Conversely, another teacher who reported using the Direct Instruction programs in her classroom stated that she would like to send her students to receive literacy instruction in the dis currently using a completely different curriculum. Based on these teacher comments, little evidence shows that use of Direct Instruction programs has facilitated cross instruction between the ELS c lassroom and the district based special education programs. In summary, all teachers who responded to the survey rated that iteracy instruction that had taken place in their classrooms. However, teachers at the primary level rated the extent to which integration and inclusion had improved more positively than teachers at the intermediate level. Based on the responses of teach ers during the focus groups, this disparity may be due to the significant skill difference between the ELS students and the general education students at the intermediate level. Two factors that appear to facilitate integration and inclusion opportunities are a connection to the general education curriculum when behaviors, such as being able to sit and listen to a story. There is no evidence at this time that the use of Direc t Instruction programs in the ELS classroom has facilitated the

PAGE 284

268 inclusion of ELS students in district based resource programs for their literacy instruction. Question 12: To What Extent do Teachers and Parents Believe the Activities of the ELS Literacy I nitiative Have Impacted Generalization of Literacy Skills to the Home? One of the desired intermediate outcomes of the ELS Literacy Initiative was that students would not only be able to acquire and demonstrate literacy skills in the classroom, but that th ese skills would transfer over to new settings such as the home or community. Stakeholders in the program identified the current evaluation question in order to determine if this generalization of literacy skills had taken place. Also reported in this se between home and school, how parents are supporting literacy instruction in the home, and the impact of parent participation on literacy instruction and student outcomes. The teacher and parent surveys both asked respondents to rate their opinions on the extent to which literacy skills gained in the classroom had generalized to the home setting. The majority of teachers (46.2%) responded "somewhat" to the statement that the li teracy skills that students had worked on in their classrooms had generalized to the ). Parents wer e more positive in their ratings of the extent to which they had seen skills carry over to the The opinions of the remaining parents were diverse, with 19.2% responding

PAGE 285

269 Parents were asked to explain their ratings of the extent to which skills had carried over to the home setting. An analysis of t his follow up question suggested that parents identified a variety of skill areas in which they had seen improvements in the home setting, such as identifying sight wor ds, sounding out new words, identifying letters and

PAGE 286

27 0 letter sounds, demonstrating increased interest in books and reading, using increased expressive and receptive vocabulary, and demonstrating an increased desire to communicate. Parents who were less favo rable in their rating of the degree to which they were not making as much progress in the area of literacy as they would have liked to have seen. However, one to work with assistive technology. More could be done with technology to help support children with moto Parents of students in the ELS program were also asked to rate their levels of satisfaction with the communication between the home and school regarding literacy Table 34 ). This question was important because while communication betwee n home and school does not guarantee the generalization of skills, good communication can increase the likelihood of generalization. When parents were asked to comment on the s communicate, seeing the schoolwork at home, having a system for communication (e.g., a notebook), maintaining frequent communication, and having a common understanding frustration that a certain program was or was not being used, differences in opinion on

PAGE 287

271 how instruction in literacy should proceed, infrequent communication, and commu nication that occurred only when initiated by the parent. While the majority of parents appear to be satisfied with the current communication between home and school, there appears to be some room for improvement in this area, especially given that 20% of parent commented on the survey: handicaps. The best way to improve literacy is to improve communication through the ELS teacher and SLP and the parents and to constantly update and revise literacy goals for the child. This comment describes the ideal relationship between the parents and the school staff and identifies how communication can aff ect student outcomes. On the survey, parents were asked to comment on how they support literacy instruction at home in order to facilitate the generalization of skills. Parents reported supporting literacy in a variety of ways, such as through reading books with their children

PAGE 288

272 every day, having books around the house that are of interest to their children, their children to explore and read books on their own, and regularly going to the library. Some parents also reported working on specific skills at home, such as sounding out words, reviewing flashcards of sight words, typing words on the computer, reinforcing language and vocabulary, and encouraging the use of t devices. Several parents mentioned reinforcing literacy skills across the day and across settings, such as sounding out words on menus in restaurants or reading signs in the community. One parent even reported delivering the Direct Instruction program that is being used in school on a nightly basis at home in order to give the child an additional dose of that instruction. Several parents also reported that their children were receiving additional tutoring outside of school t hrough private providers who were reinforcing education, valuing literacy instruction and demonstrating their willingness to support this instruction at home. One theme that emerged from the parent surveys and from comments that teachers made on the surveys and in the focus groups involved the issue of homework. The results of early t argeted supplemental homework in reading and reading comprehension is remarkable. There is no other word. When our son came in to demonstrate at our IEP, the administrative staff asked if he really was reading or if it was staged. He is diagnosed as mod erate on the autism spectrum and went from struggling with his alphabet to reading in less than one year.

PAGE 289

273 student outcomes: I think that the parents that do work with th eir kids . I see a big difference. Like with the one student, because now he is reading over last year he could not long way. And in other parents, I have taught them to read; they are reading but support. When asked how they are supporting literacy instruction at home, several parents reported homework as one way that they were supporting literacy instruction. When asked what the ELS program could do to improve the literacy outcomes of students, several parents suggested that more work be sent home to be complete d. It is possible how to support literacy instruction. In contrast, one parent did report wanting less homework to be sent home, suggesting that it is too difficult to complete it with two working parents. During the focus group interviews, some of the teachers reported sending work home on a regular basis while others commented on the difficulty of sending work home. Based on the comments, the teachers who did send work home sent worksheets that were tied directly to the instructional program they were using with a particular student in the classroom, and the teachers who did not send work home viewed homework as requiring additional time and work that they did not h ave in their school day in order to

PAGE 290

274 individualize more materials to send home with students. Based on these comments, it is possible that the use of a program to instruct reading can support the generalization of skills at home by facilitating homework pr eparation. One of the themes that emerged through the qualitative data analysis process was instruction or student outcomes. One example of parent participation is that parents are requesting that teachers use specific reading programs with their students, such as the Reading Mastery program. One parent provided an example of how his involvement affected the classroom instruction: It is my feeling that literacy should be a combination of exposure and opportunity for the children in a variety of settings. However I feel that a structured program should be an integral part as well. This piece [Reading Mastery] was lacking until we pushed for it in a meeting. From the comments of parents and teachers, it is clear that parents are initiating communication and involvement and this is having an affect on what happens in the classroom. One parent described how her efforts resulted in the teacher s ending work home: We have developed a cooperative relationship with the teacher where she provides reading and comprehension homework every day, which we religiously have our son work through and complete. All this is at the initiative of us asking and th e teacher cooperating. Another parent described how her efforts resulted in increased communication between home and school:

PAGE 291

275 literary goals and achievements. One parent even we teacher was using the Reading Mastery program. As these examples show, parent involvement can clea rly have a strong impact on classroom instruction In summary, the majority of teachers in the ELS program report that skills gained generalization and supporting literacy development at home through a variety of means. Communication between home and school can also facilitate the carryover of skills from school to home, and the major of communication. Several parents and teachers brought up homework, and while some parents would like more homework, at least one parent reported wanting less. Some teachers reported sending homework home on a regular basis while others found this to be a challenge. Finally, parent involvement was identified as a theme in the qualitative data analysis and this parent involvement is having an impact on classroom instruction and student outcome s. Question 13: To What Extent Have Student Outcomes in the Area of Literacy Been Impacted as a Result of the Literacy Initiative? Introduction The desired outcomes of the ELS Literacy Initiative range from simply making more appropriate research based curricular materials available to teachers to improving

PAGE 292

276 the consistency of instruction within the program. However, the most important desired outcome of the ELS Literacy Initiative is an increase in student literacy achievement, which is one of the ident ified long term goals of the initiative. The measurement of student achievement in literacy for students in the ELS disabilities (e.g., nonverbal, limitations in moto r functioning, etc.). One possible source of literacy outcome data that has been collected on all students in the ELS program three times a year is the l iteracy b enchmark d chapter, the l iteracy benchmark d ata are not adequate as a measure of programmatic outcomes primarily because of the small number of students that have been assessed using any given benchmarking tool within any given administration format. Instead, several other data sources have been us ed to inform this evaluation question regarding student literacy outcomes in the ELS program, including data taken from the ELS Literacy Tracking Form, the primary and intermediate teacher surveys, and the qualitative data from the focus group interviews a nd open ended items on the teacher and parent surveys. ELS Literacy Tracking Form D ata T eachers in the ELS program were asked to rate the literacy development of all of their students on the ELS Literacy Tracking F orm (see sample form Appendix B ) on an annual basis for the first time at the end of the 2006 2007 school year. The ELS Literacy (beginner, n ovice, e arly to upper emergent, and u pper e mergent to fluent) in seven di fferent skill areas (Concepts of Print, Letter Identification, Phonological Awareness,

PAGE 293

277 Phonics, Spelling and Writing, Symbol and Word Reading, and Vocabulary and Comprehension). The four developmental stages used in the literacy tracking f orm mirror the four developmental stages identified in the ELS Literacy Scope and Sequence which were severe physical and cognitive disabilities (DeCoste, 2005). These stages are particular ly helpful to use with the ELS population of students because of their emphasis on the pre emergent and emergent literacy levels. Below are brief descriptions of the four developmental stages. Literacy beginner is a pre emergent stage of literacy that tak es place between birth and the preschool ages for typically developing children. Children at this level are just beginning to find books pleasurable, understand that books involve interaction between a reader and a listener, and know that books are handle d in certain ways (e.g., holding the book right side up, turning pages from right to left, etc.). Children at this stage believe it is the pictures in the book, not the words that convey the meaning. Students at this level are interested in drawing, mos skill to writing, but they are beginning to approximate representations (people, things, etc.) in their drawings. Students at the literacy novice level are emergent readers and writers, and this developmental stage takes place d uring the preschool ages for typically developing learners. Novice level readers are beginning to learn that the printed words carry the message in books, and they are attending more to those words by doing things like identifying individual letters. Rea ders at this level can identify environmental print (e.g.,

PAGE 294

278 Novice writers intentionally create written symbols (e.g., pictures, scribbled name writing, scribbled message s) that they use to communicate a message. They will attempt to write all or part of their names. They do not yet understand that letters represent sounds in speech (alphabetic principle). Students at the early to upper emergent reader/writer level are now starting to learn the actual skills associated with reading and writing. In typically developing students, this stage takes place between the k indergarten and f irst g rade levels. Early to upper emergent readers are beginning to recognize letters and associate their names with their corresponding sounds. Students writing at this level attend to print and know that print is what carries the message. They are just starting to apply letter/sound relationships in their inventive spellings. They can name and write most letters using their conventional forms. By the end of the upper emergent to fluent reader/writer level, students are reading connected text from conventional books and the emphasis is becoming more on comprehension. They are learning some of the more advanced decoding skills and have a large sight word vocabulary. Upper emergent to fluent writers are using more traditional spelling and writing approaches and the emphasis is on consonant blends, short vowels, digraphs, and phonograms. By the end of this developmental period, students are able to write short paragraphs. This level of development takes place between second and third g rade for typically developing learners. The goal of the ELS program is that students would graduate from th e program at the u pper e mergent to fl uent level of literacy development. Achievement at this level of

PAGE 295

279 development would place students at the second/third gr ade level with regard to their reading and writing. Some students in the program are expected to surpass this level of development in their literacy achievement. Conversely, some students will likely not be able to achieve this level of development even with intensive, research based instruction. Given that each of the four stages of literacy develo pment identified above represents more than one year of development for typically developing learners, it is expected that moving from one developmental level to the next would take more than one year. Consequently, the l iteracy tracking f orm is likely no t intended to be sensitive to annual growth in the area of literacy. Rather, the f orm and its ratings of student skill development are intended to follow students from the time they enter the program to the time they graduate from the program. ELS liter acy o utcomes 2007 2008 Data from the ELS literacy tracking f orm were analyzed in several different ways in order to help describe literacy achievement in the ELS program. First, the literacy tracking data from the 2007 2008 school year were examined to determine how teachers rated the literacy development of primary and intermediate students in the ELS program in order to create a profile of current literacy achievement. Then t he percentage of students who fell within each of the developmental levels ac ross the seven different skill areas was calculated. This analysis was conducted separately for primary and intermediate students and for primary and intermediate students combined ( see Table 35 ). The combined data are vi sually represented in Figure 3

PAGE 296

280

PAGE 297

281 Figure 3 The percentage of ELS students falling within each of the developmental levels per skill area. As a group, ELS students perform differently across the different skill areas. The two areas in which ELS primary and intermediate students ap pear to be rated the strongest are Concepts of Print and Letter Identification. In 2007 2008, 23.7% of students fell within the highest developmental level, u pper e mergent to f luent, for Concepts of Print. In the area of Letter Identification, 34.2% of s tudents fell within the highest developmental level. Concepts of Print and Letter Identification are both considered low level reading skills that students often master early in their literacy development. In contrast, of all of the five remaining skill areas, the skill area with the next highest percentage of students falling within the upper emergent to f luent skill level (after Concepts of Print and Letter Identification) was Symbol and Word Reading (i.e., sight word identification and reading fluency) with 9.2% of students falling in this developmental range.

PAGE 298

282 The b eginner level of skill development is the lowest of the developmental levels. 27.6% of students were rated as falling within the b eginner skill level of Concepts of Print, and 22.4% of st udents f e ll within this developmental level for Letter Identification. Even greater percentages of students were rated as falling within the beginner level of development in the remaining five skill areas. The skill area with the next lowest percentage o f students at the beginner level was Symbol and Word Reading. This skill context. It is possible that ELS students are achieving at higher rates in this skill area when c ompared to other skill areas, such as phonological awareness and phonics, because of the traditional emphasis on sight word instruction and the demonstrated ability of students with moderate to severe disabilities to learn to identify isolated sight words. Four of the seven literacy skill areas all had large percentages of students falling within the beginner level of development and very small percentages of students falling in the u pper e mergent to f luent developmental level These levels included Phonol ogical Awareness, Phonics, Spelling and Writing, and Vocabulary and Comprehension. The data suggest that overall, primary and intermediate students in the ELS program are achieving low levels of development in these literacy areas. For three of the areas Phonological Awareness, Phonics, and Spelling and Writing, there has only recently been strong instructional emphasis on these skills through the ELS Literacy Initiative. The low achievement in the area of Vocabulary and Comprehension is not unexpected, given the significant expressive and receptive communication deficits of students in the ELS program. The data suggest that Vocabulary and Comprehension is the skill area in which pr imary and intermediate students are the least skilled, with 50% of stude nts falling in the

PAGE 299

283 b eginner developmental level and only 2.6% of students falling within the highest developmental category, upper e mergent to f luent. Because it is not known what the profile of literacy development was for students in the ELS program prio r to the ELS Literacy Initiative, it is impossible to conclude from the analysis of 2007 2008 literacy tracking data whether students have made progress as a result of the literacy initiative. However, the profile of literacy development as reflected in t he 2007 2008 data does not suggest that students are achieving at high levels in several of the skill areas that have been targeted by the l iteracy initiative. For example, despite the significant resources and emphasis that have been placed on instructio n in the areas of phonological awareness and phonics as part of the ELS Literacy Initiative, it appears that students in the ELS program continue to demonstrate low levels of mastery of these skills. Literacy development 2006 2007 to 2007 2008 The ELS literacy tracking f orm data were also examined to determine trends in growth across time. However, it is important to note that there were only two years of l iteracy t racking data to examine. As has been mentioned previously, students are not expected to move up one developmental development. Consequently, the ELS literacy tracking f orm data are not designed to be gains that have been made by students as a whole across the two years cannot necessarily be attributed to the instruction that has taken place as part of the ELS Literacy Initiative because other g., maturation and differences in how teachers rate student development) cannot be ruled out.

PAGE 300

284 In order to examine student growth in literacy development from 2006 2007 to 2007 2008, only the data of primary and intermediate students who had complete lite racy development ratings for both years were included in the analysis. Reasons that students may not have had data for both years include: (1) they were kindergarteners in 2007 2008, (2) they were 5th grade students in 2006 2007, (3) they moved into the p rogram in 2007 2008, or (4) they moved out of the program for 2007 2008. This narrowing of the sample resulted in a total of 50 students for the analysis. The data suggest that students who had ELS literacy tracking f orm for both the 2006 2007 and 2007 20 08 school years made growth in their literacy development as a group. The patterns of development from 2006 2007 to 2007 2008 were similar for each of the seven dif ferent skill areas (see Table 36 data depicted visually in Figures 4 10 ). In all seven ar eas, fewer students fell in the b eginner range of development in 2007 2008 than in 2006 2007. The decrease in students at the b eginner level generally resulted in an increase of students at the n ovice and e arly to upper e mergent levels, with some slight v ariations between the skill areas. The percentage of students falling within the highest developmental range (upper e mergent to f luent) remained close to the same from 2006 2007 to 2007 2009. These data suggest that as a group, the students who have rece ived two years of literacy instruction in the ELS program demonstrated growth in their literacy development from 2006 2007 to 2007 2008. These gains are impressive given that the literacy tracking f orm data are not designed to be sensitive to small amount s of growth in student achievement.

PAGE 301

285 Figure 4 Percentag e of students falling within each developmental level for Concepts of Print for 2006 2007 and 2007 2008.

PAGE 302

286 Figure 5 Percentage of students falling within each developmental level for Letter Ide ntification for 2006 2007 and 2007 2008. Figure 6 Percentage of students falling within each developmental level for Phonological Awareness for 2006 2007 and 2007 2008.

PAGE 303

287 Figure 7 Percentage of students falling within each developmental level for Ph onics for 2006 2007 and 2007 2008. Figure 8 Percentage of students falling within each developmental level for Spelling and Writing for 2006 2007 and 2007 2008.

PAGE 304

288 Figure 9 Percentage of students falling within each developmental level for Symbol and Word Reading for 2006 2007 and 2007 2008. Figure 10 Percentage of students falling within each developmental level for Vocabulary and Comprehension for 2006 2007 and 2007 2008.

PAGE 305

289 Individual student literacy development The ELS literacy tracking f orm d ata were analyzed in several different ways to help inform the current evaluation question, including examining the data to gain an understanding of current student literacy achievement in the ELS program (2007 2008) and evaluating the data to determine wh ether ELS primary and intermediate students as a group made gains from 2006 2007 to 2007 2008. The final way in which these data were analyzed was the examination of individual student growth from 2006 2007 to 2007 2008. Only those students who had compl ete literacy tracking data for both years were included in the analysis ( n = 50). Table 37 contains the frequency distribution of individual student gr owth. As identified in the table 28% of primary and intermediate students were not rated to have grow n enough to jump a developmental level in any of the skill areas. The remaining 72% of the ELS students included in the analysis grew at least one developmental level in at least one skill area. The number of areas in which students were rated to have ju mped one or more developmental levels varied. Sixteen percent of the students grew in one skill area and an additional 16% of students grew in two skill areas. Forty percent of the students were rated to have jumped a developmental level in three or more skill areas. In a practical sense, this represents a group of students with whom teachers would likely say they had made a great amount of progress in one school year. T hree students were rated to have grown in six or seven of seven skill areas, represe nting a tremendous amount of skill development in one school year.

PAGE 306

290 An analysis of which skill areas students were most likely to move up one or more categories in suggested a relatively even spread among the skill areas. The skill area in which student s were least likely to make progress was Letter Identification, with only 7.8% of the students making progress in this area (see Table 38 ). However, students in the ELS program also achieved at the highest levels in this skill area (see Table 35), so they may have simply had less room for growth. For the remaining six skill areas, the percentage of students making developmental progress ranged from a low of 13.8% to a high of 18.1%, suggesting that students experienced growth in more than one or two skill areas. It is important to note that students were just as likely to make progress in the areas of Phonological Awareness, Phonics, and Vocabulary and Comprehension, the three areas in which ELS students appear to achieve at the lowest rates.

PAGE 307

291 Summary The ELS literacy tracking f orm data were analyzed in several different ways to help answer the evaluation question about student outcomes in the area of literacy. The 2007 2008 data suggest that overall, students are achieving higher in the skill areas o f Concepts of Print and Letter Identification than in the other five skill areas, with students experiencing the least achievement in the area of Vocabulary and Comprehension. An examination of trends in the data from 2006 2007 to 2007 2008 suggests that students who had complete literacy tracking f orm data for both years made gains across all of the skill areas from the one year to the next, with fewer students falling within the beginner developmental levels and more students falling within the n ovice an d e arly to upper e mergent developmental levels. Finally, an examination of individual student growth data suggests that 72% of students in the ELS program who had l iteracy tracking f orm data for both 2006 2007 and 2007 2008 grew a developmental level or m ore in at least one skill area. Overall, the ELS literacy tracking data suggest that primary and intermediate students in the ELS program are making gains in the area of literacy.

PAGE 308

292 Survey and Focus Group Data Data from the surveys and focus group intervie ws were also examined to determine how student literacy outcomes have been affected by the literacy instruction that had taken place in the classroom. On the survey, teachers were asked to rate the degree to which they agreed with the statement that stude nt literacy skills had improved because of the resources in their classrooms and the training that teachers had been provided with. The majority of teachers either agreed (53.8%) or strongly agreed (38.5%) w ith this statement (see Table 39 ). One teacher statement. Primary and intermediate level teachers responded similarly. These survey data suggest that teachers in the ELS program believe that student outcomes are improving in the area of literacy as a result of their inst ruction and the resources they have been provided with as part of the ELS Literacy Initiative. Qualitative data taken from the teacher and support staff focus group interviews provides additional evidence regarding student outcomes in the area of liter acy. Teachers

PAGE 309

293 and support staff were asked during the focus group interviews to comment on their greatest success stories with regard to student literacy instruction during the 2007 2008 school year, as well as any other positive literacy outcomes they ha ve noted in their classrooms. When responding to the question about positive outcomes, one support staff members were more specific, citing how students learn ed to sound o ut words, gain ed a large sight word vocabulary, made improvements in learning behaviors such as being able to sit and listen to a story, and improved their communication and advocacy skills. S tudents in the ELS program have clearly benefitted in a variety of different ways from the literacy instruction they have received in their classrooms. When teachers were asked to comment on why they believed students had made positive gains in the area of literacy, most of the teachers cited their instruction as t he reason for the positive outcome, such as the individualized nature of the instruction that they provided. This was exemplified by one teacher who commented, I had a student who we had major behavioral issues with this year. Not one, but many. Umm, an d so she was very adverse to sitting down for one on one instruction, which made teaching reading very difficult. Umm, and so what we ended up doing was just putting . s he had a sight word goal anyway . just putting the sight words around in her e nvironment so she would associate the objects in the environment, and she actually just last week read twenty words to me. She had not read one all year. Other teachers cited the use of a structured program as one of the reasons for student success, such as one teacher who stated,

PAGE 310

294 I think just the success that my students with autism in like a reading program that something work and they Other comments that teachers made with regard to why students had experienced success in the area of literacy included having high expectations, having parents who supported literacy at home, a nd students having increased motivation after experiencing success. Interestingly, when teachers were asked about students whom they felt had not made adequate progress, they cited similar reasons for their lack of growth in the area of literacy, such as the time it takes to identify a program that is going to work and to find what motivates the child; expectations that are not too low but also not too high, which can create frustration; and the need for parents and teachers to be on the same page and supp ort one another. Another factor that teachers mentioned was the fact that many students come from the preschool level without academic goals to work on, which teachers reported can affect the entire first half of the school year. Data from the teacher survey and focus group interviews suggest that teachers believe that students are making gains in the area of literacy and that these gains are primarily a result of the instruction the teachers are providing in their classrooms and the resources they have been provided with. However, it is important to note that according to teachers, other factors appear to play a role in student success, such as working collaboratively with parents and having high expectations for student success. The data from the te acher survey and focus group interviews, c ombined with data from the ELS literacy tracking f orm, make a strong case that primary and intermediate

PAGE 311

295 students in the ELS program are making positive gains in the area of literacy. However, it is not possible to attribute these gains directly to the work of the ELS Literacy Initiative, because other reasons for student success (e.g., maturation) cannot be ruled out. It is also important to note that all of the data are teacher reports, including the ELS literacy tracking f orm data, and therefore they are not direct measures of student outcomes. The one direct measure of student literacy outcomes, the ELS literacy benchmark d ata, could not be used to help answer the current evaluation question because of the inher ent limitations of the data set particularly the low numbers of students who were assessed with any given benchmarking tool. Conclusion The results of the program evaluation study have been presented in order of evaluation question. The first three ev evaluation questions, which were designed to meet the first goal of the evaluation study: to examine how the ELS Literacy Initiative was being implemented. The next ten stions, which were designed to meet the second goal of the program evaluation study: to determine the extent to which the anticipated short term and intermediate outcomes were being realized. The final evaluation question Discussion.

PAGE 312

296 CHAPTER FIVE DISCUSSION The following is a discussion of the results of a program evaluation study that examined the implementation and outcomes of the Educational and Life Skills (E LS) Literacy Initiative. First, the purpose of the study and the methods used to evaluate the program are reviewed. Next, a summary of the results and corresponding recommendations are identified for each of the desired outcomes (short term, intermediate and long term) of the ELS Literacy Initiative. The recommendations for next steps in the implementation of the literacy initiative are then summarized. The chapter concludes with information about how the results of the evaluation will be shared, a des cription of the limitations of the study, and a discussion about the generalization of the results. Purpose of the Study A tremendous amount of research has been conducted to identify the instructional practices that lead to students learning how to read a nd becoming literate adults. The Report of the National Reading Panel; Teaching Children to Read: An Evidence Based Assessment of the Scientific Research Literature on Reading and Its Implications for Reading Instruction (NICHHD, 2000) provided a synthes is of the research on reading as well as definitive conclusions regarding effective reading instruction. In the most general of terms, research suggests that effective instruction includes systematic and direct

PAGE 313

297 instruction in phonemic awareness, phonics, fluency, vocabulary, and comprehension beginning in kindergarten. The majority of the research on reading has been conducted with students who are typically developing or have mild disabilities. Very little research exists regarding effective reading ins truction for the students with the greatest need, i.e., students with moderate to severe cognitive and physical disabilities, and students with autism. Historically, reading instruction with this population has focused on functional sight words, a strateg y that research has demonstrated can be effective (Browder & Xin, 1998). Only recently have experts in the field begun to apply what is known about best practices in reading instruction to students with moderate to severe disabilities (Browder & Spooner, 2006). However, there is little research to support whether practices that have been demonstrated to be effective with typically developing children or children who have only mild disabilities are effective for this population. The Educational and Life Skills (ELS) Program is a special education program for students with moderate to severe disabilities, including students with cognitive disabilities, physical handicaps, multiple disabilities, behavioral challenges, and autism. A special education cooper ative on the north shore of Chicago hosts this program, and serves. Literacy instruction in the ELS program has historically emphasized functional sight words. Howev er, systematic efforts to improve literacy instruction for the students in the ELS program began during the 2005 2006 school year with the start of the ELS Literacy Initiative.

PAGE 314

298 The ELS Literacy Initiative was intended to address several problems within t he ELS program, including a lack of teacher training and knowledge of beginning reading instruction, the fact that few research based instructional practices were being used, the deficiency of research based curricular resources to support literacy instruc tion, inconsistent and disjointed instruction between ELS classrooms, and failure to use student data to support instructional decision making. To address these problems, a number of actions were taken including the development of a literacy scope and seq uence to be used to guide instruction in the ELS program, the purchase of a research based core reading and language instructional program, the identification of appropriate literacy assessment tools, and the dissemination of resources to support instructi onal planning. In addition to material resources, the literacy initiative also provided teachers with ongoing professional development opportunities and access to various classroom consultants, including a reading coach and a technology consultant. Th e intended outcomes of the ELS Literacy Initiative can be placed into three categories: short term outcomes, intermediate outcomes, and long term outcomes. The desired short term outcomes of the initiative include increasing teacher skill and confidence i n teaching beginning reading, improving the curricular resources for the program, and improving the instructional planning process to include the use of data and written plans. Increasing instructional consistency between classrooms, improving integration and inclusion opportunities for students, and generalizing student literacy skills outside of the classroom are all desired intermediate outcomes. The intended long term outcomes include improving student literacy skills and post school outcomes as well

PAGE 315

299 regarding providing reading instruction to students with moderate to severe disabilities. The purpose of the current study was to evaluate the ELS Literacy Initiativ e in terms of describing how the initiative was being implemented, measuring the extent to which the desired short term and intermediate outcomes were being realized, and determining the next steps in the implementation of the initiative. Overview of Met hod The current study was conducted as a program evaluation, using a management oriented approach because the purpose of the study was to evaluate the implementation of the initiative and to collect data to facilitate future decision making regarding the n ext steps in its implementation. Program evaluation studies require that several actions be taken before beginning the evaluation, including defining the program to be evaluated and identifying the evaluation questions. In order to accomplish these tasks stakeholders in the ELS Literacy Initiative were identified and interviewed. The primary stakeholder in this evaluation was the ELS program administrator. Because the evaluation was being conducted primarily to help her make decisions about the future of the initiative, the program administrator had the greatest influence on how the evaluation proceeded; additional stakeholders were identified with her input. Of the stakeholders who were identified, several were interviewed (i.e., the program administ rator, the literacy coach, two teachers, and an intervention specialist) prior to beginning the evaluation study in order to assist in defining the initiative and identifying the evaluation questions.

PAGE 316

300 During the initial interviews, stakeholders were ask ed to help define and describe the ELS Literacy Initiative, identifying the problems it was intended to correct, the components that make up the activities and resources that are a part of the initiative, and the desired outcomes of the initiative. Based on the information shared by stakeholders, along with other documents and permanent products, a logic model that defines the ELS Literacy Initiative was developed (see Appendix A ). A logic model acts as a visual representation of how a program or interven tion is designed to address a specific problem, or set of problems, and how the activities of the program link to desired ources, that made the initiative possible, the short term, intermediate, and long initiative was intended to have on behavior and conditions. Other components of the outcomes. Stakeholder s were also asked about the questions that they would like to have answered as part of the evaluation. The stakeholders generated many questions, and through a process of condensing and prioritizing the questions, a final set of 14 evaluation questions wa s identified ( see Appendix C ). These questions were grouped into implementation questions, outcome questions, and next steps. The evaluation study was limited to the primary and intermediate grade levels (K 5) because this constituted the main focus of t he ELS Literacy Initiative at the time of the evaluation.

PAGE 317

301 Several methods were used to gather information in order to answer the identified evaluation questions. Primary and intermediate teachers, as well as parents, were asked to complete surveys that consisted of a combination of rating scale and open ended questions. Focus group interviews were conducted with three different groups: primary teachers, intermediate teachers, and ELS support staff (e.g., intervention specialists, speech and language pat hologists, and program consultants). Data on student reading progress were also examined. The open ended questions on the surveys and the comments made during the focus group interviews generated qualitative data that were analyzed to identify pattern s a nd themes. When possible, data from different sources were triangulated in order to answer the evaluation questions. Discussion and Next Steps This section provides a brief overview of the results of the study and identifies the suggested next steps in the implementation of the ELS Literacy Initiative. In some cases, the suggested next steps have already been discussed with the program administrator and have been acted upon, and in other cases the next steps are solely the recommendations of the evaluat or and have not been discussed with the program administrator. Additional information about how the results of the program evaluation will be shared with stakeholders and how the decisions regarding the next steps in the initiative will be made is include This discussion is organized according to the desired short term, intermediate, and long term outcomes of the literacy initiative. A summary of the results related to each desired outcome is presented, and the implications and suggested next steps are then discussed. Other themes that emerged through the data analysis are also identified, and

PAGE 318

302 the related implications are discussed. This section concludes with a brief summary of the suggeste d next steps in the implementation of the ELS Literacy Initiative. Short Term Outcomes The desired short term outcomes of the ELS Literacy Initiative are those that would be expected to be achieved within one to two years of implementation and they includ e (a) improving the instructional resources for teaching beginning reading, (b) improving the process that teachers use to plan and modify their instruction, and (c) increasing teacher skill and confidence in teaching beginning reading. The following is a discussion of whether these short term outcomes have been achieved and what the implications are for the next steps in the implementation of the literacy initiative. Instructional Resources: Results A variety of curricular resources are available to teachers in the ELS program to support literacy instruction. Some of these resources have been made available through the literacy initiative, such as the ELS Literacy Scope and Sequence and the core reading and language curriculum, and others have been a vailable to teachers prior to the literacy initiative, such as some of the computer software programs and teacher created instructional materials. The results of the program evaluation suggest that, in general, teachers believe that access to the material s that have been provided to them has facilitated the implementation of literacy instruction in their classrooms and that they have adequate support through instructional materials. However, teachers do report finding some resources more helpful than othe rs, and they have identified areas in which additional instructional resources are needed.

PAGE 319

303 One of the curricular resources that was developed as part of the literacy initiative is the ELS Literacy Scope and Sequence This resource describes the developmen tal progression of the various skills associated with literacy (e.g., phonological awareness, reading comprehension) and identifies assessment resources and instructional recommendations for each developmental level The Scope and Sequence was given to te achers along with a series of binders that contained the related assessment and instructional resources for each skill at each developmental level. The Scope and Sequence was designed to provide support to teachers in planning literacy instruction by help ing them identify where students are in their literacy development and what the next steps should be in their instruction. The Scope and Sequence addresses the specific and unique learning needs of students with moderate to severe disabilities and is clos ely aligned with what research says about effective literacy instruction. The results of the evaluation suggest that teachers are not using this resource to and many reported not using it at all. These results are discouraging, given that the Scope and Sequence was intended to serve as an instructional resource that could be used with all students. The teachers and support staff who did report using the resource are appropriately using it to identify instructional targets for their students. Of the teachers who reported not using the resource, comments suggest that while they recognize that the resource contains valuable information, they find the Scope and Sequ ence and binder reported depending on other support staff, namely the intervention specialists in their classrooms, to aid them in using this resource.

PAGE 320

304 Another curricular resource that was made available to teachers as a part of the literacy initiative was access to a published reading and language program to serve as the primary curriculum in the ELS program. The core program selected was a combination of two Direct Instr uction programs: Reading Mastery and Language for Learning. While it was not expected that all students in the ELS program would be able to participate in the core curriculum, one of the desired outcomes of the initiative was that as many students as poss ible would be instructed using one or both of these programs. The results students. Teachers and support staff reported that they appreciated having the programs an d that they liked them because the programs were ready to use and did not require a lot of teacher preparation, and that with proper training, teaching assistants could deliver the programs. Another benefit of use of the Direct Instruction programs is the assurance that when the programs are implemented as intended, students are receiving high quality research based instruction. While one teacher expressed dislike of the Direct Instruction programs, teachers did not generally appear to resist using the pr ograms. The primary reason that teachers reported not using the programs was the characteristics of the students in their classrooms namely, that they were nonverbal and/or not developmentally ready to participate in the programs. Unfortunately, particip ation in these programs is limited to students who are developmentally ready (e.g., who can discriminate between different letters, have some concepts of print, etc.) and who can provide oral responses on cue. Teachers and support staff consistently expre ssed frustration that similar published curriculum programs were not available to use with students who could not participate in

PAGE 321

305 the Direct Instruction programs. More specifically, many teachers reported that if they were to make one improvement to the li teracy initiative, it would be the availability of a published, ready to use, research based curriculum that could be used with students who are nonverbal, significantly impaired, or otherwise at the earliest stages of literacy development. The results o f the evaluation suggest that teachers continue to rely heavily on teacher created materials when providing literacy instruction. All teachers reported that they use teacher students. T eachers specifically reported using teacher created materials during group instruction, in which their lessons typically consist of literacy and life skill units. Teachers find it necessary to create these units themselves because of the individualization that each child requires to participate in the group (e.g., one student needs only written words in the materials, another student needs visuals and words, and another student requires pictures that can be put on a communication device). Teachers also re port having to create instructional materials for students who are not able to fro m the qualitative data analysis, and teachers identified it as a factor that improves student outcomes. Teachers reported individualizing instruction in a variety of ways. community that has digital pictures of where that particular student goes in the community. Making the instruction more meaningful for the student increases his or her motivation to learn and improves literacy outcomes.

PAGE 322

306 The challenge that teachers repor ted with regard to depending on teacher created materials and providing highly individualized instruction is the time that it takes to create and prepare all of the materials. Unfortunately, because of the unique learning needs of the ELS student populati on and the individualized nature of instruction, these are not the types of materials that can simply be purchased and used. However, teachers in the ELS program do create and use materials that are similar between classrooms. Teachers commonly expressed frustration that there is no efficient method for them to share materials with other teachers in the program. If teachers were able to share more of their materials, it might reduce some of the time that teachers have to spend with material preparation. They might not be able to simply print and use something that another teacher has created because the needs of their own students may be slightly different, but at least they would have something to start with. Because ELS classrooms are all located in d from their classrooms, teachers have not had an efficient way for sh aring materials electronically. In addition, the fact that teachers in the ELS program are all loc ated in different buildings and in different districts means that they also have difficulty finding time to effectively collaborate with one another. Teachers currently meet once a month for two hours with other teachers who teach at their grade level. T hey would benefit from having more time to collaborate with one another regarding literacy instruction. In addition to the ELS Literacy Scope and Sequence the core reading and language curriculum, and teacher created materials, teachers use other resour ces to support literacy instruction in their classrooms such as other published programs, computer software programs, websites that have literacy activities for students and

PAGE 323

307 websites that provide teachers with resources. Teachers report finding all of th ese additional resources helpful, but the primary barrier to their use appears to be not knowing what is available. Interestingly, the research on systems change suggests that access to appropriate materials and resources can facilitate the process of cha nge. The ELS program is resource rich and has no barriers related to making resources available to teachers, except where the resources simply do not exist. In fact, the only barrier relating to this issue in the ELS program is that teachers have access to so many instructional materials that they have difficulty keeping track of them. Teachers report depending on people such as the literacy coach and intervention specialists to help them identify and locate appropriate instructional materials and resour ces for their students. Overall, the results of the program evaluation suggest that while teachers find most of the curricular resources they have been provided through the initiative very helpful, they might only be using them with a few of their studen ts (i.e., Reading Mastery and Language for Learning and the ELS Literacy Scope and Sequence ) and they still depend heavily on teacher created materials as well as other curricular resources such as computer programs and websites. However, the program eval uation results suggest that there are areas for improvement when it comes to curricular resources in the ELS program. The following paragraphs describe steps that have already been taken to improve access to curricular resources and discuss other suggesti ons for next steps to be explored by program stakeholders. Instructional Resources: Recommendations One of the steps that has already been taken to improve the curricular resources in the ELS program involves the purchase of an early literacy curriculum that was designed

PAGE 324

308 specifically to meet the needs of students with moderate to severe disabilities, and which can be easily adapted for nonverbal students. The Early Literacy Skill Builder (ELSB; Browder, Gibbs, Ahlgrim Delzell, Courtade, & Lee, 2007) is t he first published program to provide comprehensive literacy instruction, including instruction in phonemic awareness and phonics, for this particular population of students. The authors of the program are part of the UNC Charlotte General Curriculum Proj ects, the same group that has summarized the existing research on literacy instruction for students with moderate to severe disabilities (Browder & Spooner, 2006), and which has also conducted additional research in this area. The ELSB was purchased at th e end of the 2007 2008 school year and delivered to all primary level classrooms at the start of the 2008 2009 school year. The publishing company provided a half day of training for the primary level teachers, in addition to a few teaching assistants, in tervention specialists, and program supervisors, at the beginning of October 2008. Teachers who attended the training provided the initial feedback that they were excited to begin using the program in their classrooms and thought it would help them provid e better instruction to students who were not able to participate in the core curriculum programs. used with students who are nonverbal, significantly impaired, or at the earl iest stages of literacy development. This curriculum is unique in that it (a) begins at the earliest level of literacy development, (b) provides the necessary visual supports, (c) can be adapted so students can respond receptively, and (d) incorporates th e use of a puppet (Moe the frog) to increase student interest and motivation. Most importantly, the ELSB is aligned with what research has identified as best practice instruction (i.e., direct and systematic

PAGE 325

309 instruction) and provides instruction in all im portant skill areas including concepts of print, phonological awareness, and phonics. Furthermore, the ELSB is a multi year program that has built in assessments to monitor student progress. The ELSB is most appropriate for use at the primary level bec ause using a puppet to reinforce instruction is not age appropriate at the intermediate level and higher. Consequently, the ELSB was only provided to primary classrooms, but not to intermediate classrooms. However, one intermediate classroom was provided a copy of the program because a parent advocated for the program to be used with her child. With the exception of that one classroom, the intermediate classrooms as well as the middle and high school classrooms still do not have a program to use for stud ents who are significantly impaired, nonverbal, or otherwise cannot participate in the Direct Instruction programs. Another resource that has been put into place to support literacy instruction is a shared online file storage and management system ( www.box.net ) to which teachers have access. For a monthly fee, the website provides online file storage and file sharing capabilities. This site will provide teachers and staff in the ELS program with a way to effectively a nd efficiently share documents and materials by allowing users to create folders for storing files and upload personal documents and materials into those folders. Teachers report that they can upload materials easily and quickly. In order to encourage te required from each level has been replaced with the requirement that teachers and staff instead spend their time uploading files to share.

PAGE 326

310 While teachers and support staff in the ELS program are just beginning to use the online file sharing site, it shows great promise for overcoming some of the barriers related to curriculum materials that teachers identified during this evaluation. First, it allows teachers a web ba sed solution for file sharing, which means they can access the files from anywhere, effectively removing the location of a classroom as a barrier to sharing materials. The ability to organize the files within the website using folders and descriptors will make the files easily accessible. The site can also be used to make the ELS Literacy Scope and Sequence more accessible and easier to use. More specifically, teachers will be able to access an electronic version of the Scope and Sequence and the relate d assessment and instructional materials will be organized on the site using the same framework as used in the Scope and Sequence first by skill area and then by developmental level. Hopefully, having a clearly organized electronic version of these docum ents will decrease the likelihood that the resources will be viewed as Scope and Sequence and related materials as intended. While the improved file sharing capabilities offered by this website will not fully eliminate the need for teachers to create materials, it will hopefully provide them with more to start with so they will not need to spend as much of their time on preparing materials. The monthly subscription to www.box.net f or online file sharing serves as a temporary solution to the file sharing problems in the ELS program. The cooperative district is in the process of moving to a web based server called Microsoft Office SharePoint 2007. This new web based server will allo w teachers to access information on the server from anywhere, making material storage and sharing possible through the

PAGE 327

311 www.box.net will be easily transferred to the new dis trict server when it becomes operational. SharePoint will also provide teachers with other capabilities beyond simple file sharing, offering an online forum for teacher collaboration and the creation of professional learning communities. Creating a place online where teachers in the ELS program can have discussions about literacy instruction and share ideas and information could further facilitate the implementation of the literacy initiative. Research suggests that professional learning communities faci litate systems change and have been related to gains in student achievement (Hord, 1997). While the adoption of the ELSB curriculum and the subscription to www.box.net go a long way in addressing some of the greatest barriers related to curricular resour ces, these actions do not remove all of the barriers, and additional actions must be taken to improve curricular access and resources in the program. For example, teachers at the intermediate level still do not have access to a published reading curriculu m that can be used with students who are nonverbal, and the search for a quality curriculum to meet this need should continue. Meanwhile, teachers should be encouraged to share instructional materials and collaborate as a community using www.box.net (the temporary solution) and SharePoint (the online server soon to be adopted). Another area of improvement that needs to be addressed is how best to communicate with teachers the availability of various instructional resources in the program as well as when and with whom the resources should be used. The teachers are currently using the reading coach and technology coach to help them identify curricular resources; but, teachers should also be able to access lists of available resources that

PAGE 328

312 include what ski lls these resources address, for what grade levels they are appropriate, and for what types of students each resource is best suited. These types of lists exist to some degree in the program, but teachers clearly do not know where or how to access them. T he final set of recommendations relates to the lack of use of the ELS Literacy Scope and Sequence to support instruction in the program. Teachers are not currently using this resource to help them identify appropriate instructional content and strategies. One action being taken to address this issue involves making the Scope and Sequence and corresponding resources available electronically to teachers through www.box.net. However, it is also recommended that teachers receive additional professional devel opment on the content of this resource, specifically regarding how literacy develops in students with moderate to severe disabilities, what skills should be taught as part of a comprehensive literacy instructional program, how to identify where students ar e in their literacy development, and how to identify appropriate instructional targets. The ELS program has a high rate of teacher turnover, and while significant professional development opportunities have been provided to teachers on these topics in the past, this information must be shared with teachers on an ongoing basis. For teachers who have provided. Instructional P lanning : Results Two of the desired short term outcomes of the ELS Literacy Initiative were to increase the use of data in making instructional decisions and for every student in the ELS program to have an individualized instructional plan in the area of literacy. Both of

PAGE 329

313 these goals relate to t he instructional planning process, which consists of identifying student skill strengths and weaknesses, determining instructional targets, identifying instructional strategies and developing instructional plans, monitoring student progress on an ongoing b asis, and making changes to instructional plans when appropriate. help them make instructional decisions. When asked about which sources of data they find most helpful when making these decisions, teachers reported that they find the data resources that have been made available to them through the ELS Literacy Initiative (i.e., literacy benchmark data, mastery tests in the curriculum, and information on the literacy tracking form) least helpful; other data sources, such as IEP goal data and teacher observations, were identified as most helpful. The literacy benchmark data collected on all students three times a year was intended to serve two purposes. The first was to serve as an indicator of overall programmatic outcomes in the area of literacy. The other purpose was to serve as a source of information that teachers would use to determine whether students were making adequate process in their literacy skills and to signal if instructional changes needed to be made. Unfortunately, teachers do not appear to be using the literacy benchmark data to help them make instructional decisions. One possible reason for this is that data that are collected three times a year do not pr ovide teachers with frequent enough information to make a determination about when they need to change their instruction. Furthermore, because the majority of the literacy benchmark measures are measures of accuracy and not fluency, they may not be as sen sitive to growth as more traditional curriculum based measures, and therefore they may not be adequate to support teacher decision making.

PAGE 330

314 Data from the program evaluation also suggest that teachers simply do not find the data applicable for many of their students, especially when the benchmark data are not related The ELS literacy tracking form is another source of student data that has been made available to teachers as part of the literacy initiative. This form identifies whe re students were rated to fall along a developmental continuum in seven different skill areas, and the lessons they last completed. This form has been completed annuall y for all students, beginning during the 2006 2007 school year. The primary purpose of this form development so they could provide seamless instruction as the student s moved from one teacher to the next. However, teachers did not report using the information on the form to help them plan instruction. In fact, some teachers reported that they had not even received the forms for their new students. This information is clearly not being communicated as intended, and when the information is being communicated, it is not being used. The other data source that teachers have access to through the literacy initiative is the mastery tests that are included as part of the co re literacy programs, Reading Mastery and Language for Learning. These tests are part of the curriculum and are used to determine whether students have mastered the material and whether they are ready to move ahead to the next lesson. Teachers rated this source of data as being more useful than the literacy benchmark data and the data on the literacy tracking form, but not as useful as other sources of data such as teacher observations and IEP progress monitoring

PAGE 331

315 data. This may be because the mastery te sts are only available for students who are participating in the Direct Instruction programs, and as teachers reported, they are only Teachers reported finding other sources of data, such as IEP goal prog ress monitoring data and informal teacher observations, helpful in planning their instruction and determining when changes must be made. IEP goal progress monitoring data are the frequent and ongoing progress monitoring data that are gathered to determine whether students are making progress relative to their specific IEP goals. Typically a data sheet is created based on a specific IEP goal, and the teacher and/or teaching assistants record data by indicating on the data sheet what they observed, such as whether a student was able to provide a correct response to a question. Teachers also reported finding observations helpful in their decision making. How the observations are conducted was not assessed as part of the evaluation; however the observations are likely informal and unstructured and may also involve the examination of student permanent products. While teachers were not asked to directly rate its helpfulness, some t eachers also reported using AIMS web, a web based resource with various assessme nt tools and progress monitoring capabilities. Part of the instructional planning process involves using student data to identify instructional targets. One of the problems that the ELS Literacy Initiative was designed to address was a lack of teacher t raining and knowledge for providing beginning reading instruction to students with significant learning needs. In order to help address this problem, the ELS Literacy Scope and Sequence was developed. One of the intended uses of this resource was to help

PAGE 332

316 programs and to identify appropriate instructional strategies to help them achieve those goals. Some teachers reported using the Scope and Sequence in this manner, but others reported that th intervention specialist, literacy coach, or speech and language pathologist to help them identify instructional targets. The second short term goal of the initiative that relates to instruction al planning is for every student in the ELS program to have a written, individualized literacy instructional plan. The results of the evaluation suggest that half of the teachers in the le the other half has Although some students have written plans, the goal of all students having written plans clearly has not been achieved. Several forms have been made available to teachers that the y could use as templates for their written instructional plans, and teachers can select the format that works best for them. Those who have written plans for their students reported them as being helpful, especially for communicating literacy plans to oth er staff members. Others reported that creating written plans for their students is time consuming and redundant. In some cases, teachers did not even know what tools were available to create the written plans. One of the themes that emerged through th e qualitative data analysis process was the importance of communication and collaboration among ELS staff members in planning and delivering literacy instruction. Teachers and support staff all reported positive outcomes associated with effective communic ation and collaboration among team members. However, one of the barriers associated with effective collaboration that teachers identified involved finding the time to meet and collaborate as a team.

PAGE 333

317 Instructional Planning: R ecommendations The results of the program evaluation suggest that there is room for improvement when it comes to instructional planning in the ELS program. The following paragraphs describe some actions that have already been taken to improve this process as well as additional sug gestions to be considered by the program stakeholders. The collection of literacy benchmark data has been discontinued as of the 2008 2009 school year. Teachers are clearly not using the benchmark data to inform their instructional practices. However, it remains important that teachers have information about how their students are progressing in their literacy development. Therefore, the intervention specialists in the program have been encouraged to work with teachers to develop a more individualized plan for progress monitoring through data collection. The tools that were a part of the benchmark system may or may not be used as part of an individualized plan. It is anticipated that the data will be more useful to teachers if the data collection proc ess is individualized based on student need. Many of the pitfalls associated with the instructional planning process in the ELS program can be addressed through providing teachers with more support from someone who has expertise in the instructional pla nning process and the resources available in the program. More specifically, teachers reported that barriers to the planning process included not knowing how to use available data, not having the time to engage in the process, and being intimidated by the resources that have been created to assist them in the process. The intervention specialists have been identified as a group of individuals who can take on a more supportive role in the instructional planning process. Several

PAGE 334

318 intervention specialists ha ve already taken on this role in their classrooms, and teachers report depending on them heavily and finding their support very helpful. The intervention specialists in the ELS program are in the ideal position to take on more of a supportive role in the instructional planning process in the classroom. These staff members have all been trained as school psychologists and, consequently, have strong backgrounds in data based decision making, consultation, and effective instruction. The intervention special ists are already responsible for the collection and interpretation of student literacy data in the classroom; consequently, the role of providing more support in instructional planning would be a natural fit. The intervention specialists are in the classr oom one day a week, which allows them to provide more frequent support than the other consultants in the program and to follow through with decisions that are made. Because they are in the classroom so often, they are in a better position to observe instr uction in action, identify training needs, and suggest additional resources for support. As part of their increased role in the instructional planning process, the intervention specialists would be responsible for knowing what actions should be taken at d ifferent times during the year as part of the instructional planning process, and would work closely with teachers to accomplish those objectives. The program evaluator met with the intervention specialists in October, 2008 to discuss the need for them to take more of a leadership role in the instructional planning process. The group spent some time describing the instructional planning process for literacy in ELS classrooms and identifying the activities that take place at the beginning, middle, and end o f the school year (see Appendix H ). This exercise was designed to gain group consensus regarding the instructional planning process. The next step involves

PAGE 335

319 identifying the professional development and other supports that the intervention specialists will need in order to be prepared and confident in leading the instructional planning process. Before the intervention specialists formally take on an increased role in supporting teachers with instructional planning, feedback from other stakeholder groups (e .g., program planning, the curriculum committee, and other program consultants) should be solicited. Additional recommendations should also be considered by the program administrator and other stakeholders of the ELS Literacy Initiative. The first is t o continue identifying progress monitoring tools and strategies that meet the unique needs of students in the ELS program. Because many students in the program have processing issues and physical limitations, progress monitoring tools must be measures of accuracy, not fluency. Additionally, the tools must be appropriate for use with students who are nonverbal. Currently, the program makes available a series of progress monitoring tools that have been modified from their original format to meet these requ irements. However, concerns have been raised about the sensitivity and validity of these measures. It is recommended that as part of the literacy initiative, research and investigation into appropriate progress monitoring strategies and practices should continue. Another recommendation is continued professional development for ELS staff in the area of literacy. Even if the intervention specialists in the program begin to take on more of a supportive role in the instructional planning process, staff in the program, especially teachers, are going to need additional professional development regarding how literacy skills develop in students with moderate to severe disabilities, what research says about effective instruction with this population of students, and how to use data to inform

PAGE 336

320 instructional practices. Teachers and other support staff have received professional development on these topics in the past, but the results of the program evaluation suggest that they could benefit from ongoing training in these topics as well as training on how to use the various resources that are available through the literacy initiative, such as the ELS Literacy Scope and Sequence to support instruction It is recommended that the program adopt a standard instructio nal planning process and hold teachers accountable for using that process. The intervention specialists have already started to describe an instructional planning process for literacy in the ELS program and have outlined the related activities that are to be conducted at the beginning, middle, and end of the school year (see Appendix H ). During focus group interviews, teachers suggested that the program administration should hold teachers more accountable for the instruction that is taking place in their classrooms. Standardizing the instructional planning process and holding teachers accountable for using the process would provide clear expectations and increase the likelihood that teachers would engage in a high quality planning process. If a standard process is to be adopted by the program, it will be important to ensure that teachers have the material resources and professional development/coaching support necessary for implementation. The results of the program evaluation suggest that teachers are not using the data on the ELS Literacy Tracking Form as intended. The form was developed to improve and consequently, to assist teachers in their instructional planning. It is recommended that teachers continue to complete the literacy tracking form however, it is apparent that actions must be taken to improve its use. More specifically, teachers need to be held

PAGE 337

321 more accountable for completing the form, sending a copy in to the district office, and keeping a copy with the student records that accompany the student when s/he moves on to the next teacher. More effective ways of sharing student information and data from one student to the next are currently being explored, s uch as sending a CD with the student that contains all of the necessary transition information in an electronic format. It is also possible that student data could be stored and shared more effectively using the new web based server called SharePoint. Fu rthermore, teachers need additional professional development and coaching support on how to use the information on the form. Completing the literacy tracking form and using the information to help in planning instruction is an area in which the interventi on specialists in the classrooms can provide teachers with ongoing support. With regard to having a written instructional plan for every student in the ELS program, some of the recommendations mentioned above apply here as well, such as standardizing the planning process and holding teachers more accountable for using that process. However, if teachers are going to be held accountable for having written literacy plans, it will be important that they be given a planning format that they find beneficial. T eachers currently have access to several forms, but none of these forms are currently used extensively. Furthermore, teachers would benefit from additional professional development and continued support in the classroom regarding how to complete literacy plans. The intervention specialists can take more of a role in the classroom to support this aspect of the planning process.

PAGE 338

322 Teacher Skills and C onfidence : Results Teachers in the program feel supported, prepared, and confident in their literacy ins truction through the material and professional development resources that they have been provided through the ELS Literacy Initiative. Various sources of professional development are available to teachers in the program, such as program sponsored workshop s, workshops hosted by the cooperative district, workshops and conferences hosted outside of the district, and site based coaching and consultation. Teachers report that all of these sources of professional development are helpful to them. With regard to workshops, teachers report that trainings on how to implement the technology staff are particularly helpful. However, they reported that most other training opportunities, such as general education reading conferences, are helpful as well. Teachers and support staff reported that they would like to see additional training opportunities made available for parents specifically on research based reading strategies and how to support literacy at home. They would also like more training for teaching assistants on the basics of literacy development in students with moderate to severe disabilities. In addition, they would like more training for themselves on reading comprehensio n and written expression as well as trainings that are geared toward younger students or students who are at the beginner and novice stages of literacy development. On the other hand, finding the time to attend workshops and having adequate staff to cover development.

PAGE 339

323 T eachers also report that the consultants who provide support in their classrooms are very helpful. Teachers tend to use the reading coach and technology consul tant primarily to help them identify instructional resources, to train staff on those instructional resources, and to help problem solve with them on how best to provide instruction to students with complicated learning needs. Teachers identified that the support from consultants could be improved by having more time with them in their classrooms. Additionally, teachers recommended that the consultants come into their classrooms with a suggested agenda of what needs to be covered, with some room for teach ers to add items to that agenda. Teacher Skills and C onfidence : R ecommendations With regard to professional development, it is recommended that the current workshops offered through the ELS program and the cooperative district continue to be offered to teachers. The cooperative district should also consider providing additional workshops about instructional strategies for students at the earliest levels of literacy development as well as trainings on reading comprehension strategies. Because of the co ntinued need for new and existing staff to be trained, the program committee may want to consider developing a literacy training sequence that would include basic level trainings on literacy development and instructional strategies as well as more advanced trainings for the teachers who already have command of the basics. Such a training sequence could be repeated on a regular basis. Teachers should also be encouraged to attend workshops and reading conferences that are not provided by the cooperative dis trict, because they reported that gaining this outside perspective was helpful in planning their instruction.

PAGE 340

324 Teachers also recommended making additional training opportunities available for teaching assistants and parents. Actions have already been taken to address the need to provide teaching assistants with a basic understanding of how literacy develops and how to support literacy instruction. A workshop covering this content has been developed and is being offered to teaching assistants twice during t he 2008 2009 school year. At the end of the school year, it will be important to examine the feedback provided on this training and determine if the training should be offered again. With regard to training parents, the program may want to consider provi ding this information parents should be explored as well, such as mailing information to parents about how they can support literacy at home. Teachers find the reading coach a nd technology coach very helpful, and it is recommended that these consultants continue to be available to answer teacher questions, to help teachers access instructional and technology resources, and to provide training on these materials as necessary. H owever, teachers reported wanting more frequent and ongoing support and feedback from the consultants in the program. As suggested earlier in this chapter, the intervention specialists in the ELS program could be used to provide this additional coaching a nd support to teachers in the classroom. The intervention specialists are available in each classroom one day a week, and are familiar with and to provide instruct ional support to teachers. While some of the intervention specialists in the program are already prepared to provide this support to teachers, others may need additional training and coaching themselves in areas such as understanding when and

PAGE 341

325 how to use t he Direct Instruction programs, what basic instructional strategies in the area of literacy can be used, and how to use literacy assessments to plan instruction and monitor student progress. It is also important to note that the increased role of the inte rvention specialists in the instructional planning process would not replace the responsibilities of the reading and technology consultants. The consultants should continue to be used to help identify instructional resources, provide training on those res ources, and aid in problem solving on difficult cases. Intermediate Outcomes The desired intermediate outcomes of the ELS Literacy Initiative included (a) increasing instructional consistency between classrooms in the ELS program, (b) integration and inclusion opportunities within their school communities, and (c) improving the home/school connection and the generalization of literacy skills. It was expected that intermediate outcomes would be achieved within two to three years of the formal start of the literacy initiative. The following sections describe the results of the program evaluation and the recommended next steps for each of the intermediate outcomes. Instructional C onsistency : Results One of the problems that the ELS Li teracy Initiative was designed to address was the lack of instructional consistency within the ELS program. Literacy instruction was frequently very different from one classroom to the next, and this inconsistency created disjointed instruction for studen ts who moved from one teacher to another, particularly when students moved from the primary level to the intermediate level. Because of inconsistent instruction and poor communication between classrooms, teachers often

PAGE 342

326 expected that the activities of the ELS Literacy Initiative, such as the use of the core instructional program and the ELS Literacy Scope and Sequence would increase instructional consistency between class rooms. Furthermore, using the ELS literacy tracking form and having individualized literacy plans for every student was intended to instruction. When teachers were aske d to rate the extent to which they believed instructional continuity was maintained within the ELS program, the majority of primary level Instruction programs as the core curriculum was expected to increase instructional consistency within the program. However, while teachers suggested that instructional consistency is easier when the Direct I nstruction programs are carried over from one teacher to the next, they also expressed frustration that the programs sometimes were not continued and students who had made progress in the Direct Instruction programs in their classrooms moved on to classroo ms in which the teacher used another program. development of the ELS Literacy Scope and Sequence was expected to increase instructional continuity for students who were not participating in the Direct Instruction the results of the evaluation suggest that teachers are not using this resource as intended.

PAGE 343

327 The results of the program evalua tion also suggest that teachers are not using the ELS literacy tracking form as intended and, as a result it is not serving to improve communication and instructional consistency. Some teachers in the program reported not having received the tracking for ms for their incoming students, and when they did receive them, they infrequently used the information on the forms to help with planning their literacy instruction. Their failure to reference this communication tool when planning instruction, coupled wit h the fact that very few students in the program have written instructional plans that can be passed along to their next teachers, suggests that literacy instruction in the ELS program continues to be inconsistent and disjointed. Instructional C onsistenc y : R ecommendations Some of the recommendations for addressing instructional consistency have also been made to address improvements related to the desired short term outcomes. For example, one of the recommendations for improving instructional consisten cy is to them through the literacy initiative, such as the Direct Instruction programs and the ELS Literacy Scope and Sequence Because teachers report that it is easie r to transition students who are participating in the Direct Instruction programs, student participation in these programs should be encouraged. However, it is also important to ensure that teachers receiving new students continue using these programs whe n they have been demonstrated to be successful. Currently, the continuation of the Direct Instruction programs with students who transition from one teacher to another is perceived as optional. It is recommended that administrators in the program, such a s the program

PAGE 344

328 supervisors, support the continuation of successful instructional programs across teachers. Similarly, it is recommended that teachers be encouraged to use the ELS Literacy Scope and Sequence The Scope and Sequence serves as an instructio nal blueprint to be followed and can standardize the skills students are taught and the sequence with which those skills are being taught across classrooms. The Scope and Sequence and corresponding resources will be available to teachers electronically th rough the online file storage and sharing system, www.box.net which may help to increase the use of this document. Teachers and other staff in the program should also receive additional professional development on this resource to ensure that all staff m embers understand the resource and how to use it. Communication between teachers can go a long way in increasing instructional consistency; consequently, it is recommended that teachers be held accountable for completing the ELS literacy tracking form and sharing this information with the receiving teachers. The framework of the tracking form mirrors that of the ELS Literacy Scope and Sequence and the two are intended to be used together in the instructional planning process. The tracking form also cont ains information relating to the instructional programs that were being used and the last lesson that a student completed in the program. In addition to holding teachers accountable for the completion of the form, more efficient ways of sharing student da ta and information from one teacher to the next are being explored. It is possible that the new web based server, SharePoint, will facilitate this information sharing process. Because of their role in supporting instructional planning in the classroom, t he i ntervention specialists can also assist

PAGE 345

329 teachers in the completion and communication of the literacy tracking form. In fact, the d to pass student information from one intervention specialist to the next in the ELS program. This will begin during the 2008 2009 to 2009 2010 school year transition. The final recommendation is to also hold teachers accountable for developing written instructional plans for each of their students. Literacy plans typically contain information about specific skills that are being taught, programs or materials being used to teach those skills, when the skills are being worked on, and how student progres s in those skills is being measured. A detailed instructional plan provides a clear picture to receiving teachers of what literacy instruction looks like for students prior to entering their classrooms. The program must ensure that teachers have adequate support for developing individualized plans, which includes providing them with the appropriate templates and professional development on how to use the templates. The reading coach and/or intervention specialist in the classroom can provide on site consu ltation and support to teachers in the development of written instructional plans. Integration and I nclusion : Results It was expected that if student literacy instruction in the ELS program was aligned with research based practices and students were exp eriencing improved academic gains in the area of literacy, then their inclusion and integration opportunities within their school communities would be improved. Despite achievement gains made by students in the program, for the majority of students, the s kill gap between ELS students and their typically developing peers has not been reduced enough to significantly impact inclusion.

PAGE 346

330 However, teachers report that the behavioral gains that have been made by students as a result of their literacy instruction, such as being able to sit and listen to a story, have positively affected their inclusion and integration into the classroom. Overall, the majority of teachers believe that the ELS Literacy Initiative has facilitated the integration and inclusion of ELS One of the desired outcomes of implementing the Direct Instruction programs in the ELS classrooms was the integration and inclusion of students into district based classrooms where the same curriculu m is being used. However, the results of the evaluation suggest that this has not been happening. Another factor that did appear to facilitate integration and inclusion into district based classrooms involved making a connection to the general education curriculum. Teachers reported that when they had a better understanding of what was expected from students in the general education classroom, and when they worked closely with the general education teachers to identify the times during the day when it wo uld be appropriate for an ELS student to be included, students had a more positive integration experience. Integration and I nclusion : R ecommendations One of the themes that emerged from the qualitative data analysis process was that of providing instru ction based on a broader perspective. That broader perspective frequently was said to include general education and knowing the expectations for students in the general education curriculum at any given grade level. This connection appears to improve the inclusion and integration experiences of students and should be fostered in the ELS program. As was recommended with regard to professional development opportunities, teachers in the ELS program should be encouraged to attend

PAGE 347

331 reading and literacy confere nces for general education teachers so they can gain that perspective. It is also recommended that ELS teachers gain a greater understanding of the Illinois Learning Standards as they apply to literacy. Familiarity with the learning standards can facilit ate the link to the general education curriculum. Another theme identified was the importance of communication and collaboration among team members. In order to improve the inclusion and integration opportunities of students in the ELS program, it is r ecommended that the general education teacher be a more active member of the IEP team. ELS teachers should collaborate with general education teachers to identify the most appropriate times for students to be included, and to understand what will be expec ted of a student in the general education classroom during those times so that the necessary skills may be pre taught in order to increase the probability of a positive inclusion experience. Generalization: R esults Students must be able to generaliz e skills learned in the classroom to other settings, such as the home and the community. The generalization of literacy skills was one of the desired intermediate outcomes of the ELS Literacy Initiative. As part of the program evaluation, teachers and pa rents were asked to provide their opinions about the degree to which literacy skills had generalized to the home setting. Teacher estimations estimated that generali Parent involvement was one of the themes identified through the program evaluation. Parents appear to be supporting literacy development at home through a

PAGE 348

332 variety of means, such as completing homework and reading with their child every day. Some teachers reported sending homework home on a regular basis, while others found this to be a challenge. When asked about communication between home and school, the majority of parents were satisfied with the level of communication. The results of the evaluation suggest that parents who advocate on behalf of their children can have a strong impact on classroom instruction and positively affect student outcomes. How ever, teachers reported that when parents do not have sufficient information, they sometimes advocate for inappropriate instructional strategies, which can create tension between that parent and the teacher. Generalization: R ecommendations Given the p ositive effects that parent involvement has on literacy instruction and on student outcomes, one recommendation is to encourage parents to become actively involved in the instructional planning process. Teachers should communicate and collaborate with par ents in developing instructional plans, monitoring student progress, and determining when changes to an instructional plan are needed. In order for parents to be active and meaningful participants in this process, it will be important to provide them with information on how literacy develops, as well as on best practices in literacy instruction for students with moderate to severe disabilities. This information can be shared with parents in a variety of ways, such as through meeting with the ELS parent gr oup, offering a workshop on the topic, or sending information home to parents. The program committee might also consider developing a webpage that provides information to parents in the ELS program.

PAGE 349

333 Parents report supporting literacy instruction at home in a variety of ways, and it is recommended that the ELS program continue to encourage parents to reinforce literacy skills at home. Teachers can encourage this by sending work home to be completed. However, teachers reported that sending work home on a regular basis can be difficult because of the time involved in preparing the materials. However, the online file sharing system that has been adopted by the program may provide teachers with access to materials that can be easily printed and sent home. Another way to facilitate the reinforcement of literacy skills at home involves educating parents on strategies they can use, depending on where their children fall on the developmental continuum. The program could also provide teachers with a list of web based resources that they could use to access materials and activities. Long Term Outcomes The identified long term outcomes are the most important goals of the literacy post area of literacy instruction. The long term outcomes of the literacy initiative would be expected to have been achieved three or more years after the formal sta rt of the initiative. Student O utcomes : R esults An improvement in student literacy achievement is arguably the most important potential outcome of the ELS Literacy Initiative. One of the desired long term outcomes of the initiative is for students to graduate from the ELS program with a demonstrated reading proficiency at a second grade level or higher. In order to determine if progress is being made toward this goal and if students in the ELS program are making gains in the

PAGE 350

334 area of literacy, the data from the ELS literacy tracking form were analyzed in several different ways. Data were first analyzed to gain an understanding of overall literacy development in the program. This analysis suggested that students are achieving the highest levels of deve lopment in the skill areas of concepts of print and letter identification when compared to the other five skill areas. Students are experiencing the least achievement in the areas of spelling/writing and vocabulary/comprehension. One of the themes that emerged from the qualitative data analysis was that of a strong connection between the expressive and receptive language skills of students in the ELS program and their reading comprehension skills. Students in the ELS program characteristically have defi cits in the areas of expressive and receptive communication. Consequently, the low achievement levels in vocabulary and comprehension are expected. Students who have difficulty with auditory comprehension are going to have difficulty with reading compreh ension. One of the factors that was found to facilitate the implementation of the literacy initiative and improve student outcomes was the strong involvement of the speech and language pathologist in literacy instruction. Trends in the data from 2006 20 07 and 2007 2008 suggest that students made gains across all of the skill areas, with fewer students falling within the beginner developmental levels and more falling within the n ovice and e arly to upper e mergent developmental levels. An examination of in dividual student growth data suggests that the majority of students grew one developmental level or more in at least one skill area from 2006 2007 to 2007 2008. Overall, the literacy tracking form data suggest that students in the ELS program are making g ains in the area of literacy. Furthermore, data from the survey and focus group interviews suggest that teachers believe students are

PAGE 351

335 making gains in the area of literacy and that these gains are primarily a result of the instruction they are providing in their classrooms as well as the use of resources that they have been provided. Student O utcomes : R ecommendations The evaluation found a strong connection between language skills and literacy skills in the ELS program. This theme is related to several recommendations, the first of which is to increase the involvement of the speech and language pathologist in the planning and delivery of literacy instruction. This involvement will require that teachers have time to collaborate with their speech and lan guage pathologist. The results of the evaluation suggest that having time to collaborate and communicate with team members facilitated the implementation of literacy instruction and increased student literacy outcomes. Therefore, teachers should not have a designated time to collaborate with only the speech and language pathologists, but with the team as a whole, including the intervention specialists, teaching assistants, and parents, to regularly evaluate student progress and make changes to instruction when necessary. The implementation of a standardized instructional planning process in the ELS program may facilitate communication and collaboration among team members. Having a standardized process that incorporates team collaboration times could incr ease the likelihood that teams will meet to discuss literacy instruction and outcomes in the classroom. Another component to this recommendation involves providing additional training to speech and language pathologists regarding the connection between la nguage and literacy and how they can support literacy instruction in the classroom.

PAGE 352

336 The results of the student data analysis suggested that the second lowest area of literacy achievement in the program is spelling and writing. Teachers expressed the des ire to have more professional development and instructional resources in the area of written expression. If a professional development sequence in the area of literacy is developed, strategies for teaching written expression could be included as one of th e It is important that the ELS program continue to track student literacy outcomes. Currently, data from the literacy tracking form are used to measure and track literacy outcomes. However, it is important to note that the liter acy tracking form data are not a perception of where a student falls along the developmental continuum. One recommendation is to continue to research assessment tools that can (a) provide a direct measure of student literacy outcomes, (b) meet the unique needs of students with moderate to severe disabilities, and (c) serve as a measure of programmatic outcomes. Meanwhile, if the literacy tracking data continue to be used a s a measure of programmatic outcomes, it is recommended that strides be taken to increase the reliability of the data that are collected and to ensure that teacher ratings of student skill development are accurate. One way to do this would be to provide m ore direction to teachers and intervention specialists regarding how the ratings should be made, which and how the results of those assessments can be used to inform teacher ratings. The reliability of the teacher ratings can be increased even more if teachers and intervention specialists are given guidelines regarding how student performance on the various

PAGE 353

337 assessment tools aligns with the ratings of literacy developm ent. The program should commit to evaluating student literacy outcomes on an annual basis and using the information along with other sources of data to determine how literacy instruction in the ELS program can continue to be improved. Post School Outcome s: R ecommendations One of the desired long term goals of the ELS Literacy Initiative is to improve the post school outcomes of graduates of the ELS program. Post school outcomes can be viewed as falling into three categories: where and how students live their daily lives, student employment or other work opportunities and experiences, and student leisure and socialization opportunities. Literacy is a skill that has the possibility of improving outcomes in all three of these areas. It is expected that i f students can graduate from the ELS program having achieved a second grade reading achievement level or higher, their post quality of life. Post school outcomes were not me asured as part of the current program evaluation. It is recommended that the program begin to plan for when and how to regularly collect data regarding the effects of the ELS Literacy Initiative on students once they have graduated. During the 2007 2008 school year, the ELS program began to collect survey data for the first time on the outcomes for graduates of the program for one year. During the 2008 2009 school year, the effort to collect post school outcome data has been expanded to not only include graduates who were one year out of the program, but also students who had been out of the program for five years. It is recommended that the program identify a way to assess the impact of literacy instruction on post school

PAGE 354

338 outcomes using the existing dat a collection system. It may be possible to examine the relationship between the literacy development scores from when the students graduated from the program and their post school outcomes and to then determine what patterns may be associated with student s having higher and lower literacy development at the time of graduation. Model P rogram : R ecommendations The final desired long term outcome of the literacy initiative is that the ELS program serve as a model of research based reading instruction for s tudents with disabilities to the districts that the cooperative serves. This is an important outcome because one of the primary functions of the cooperative is to increase the capacity of its member districts to better meet the needs of the students they serve. In order to be a model for other districts, the ELS program will need to find a way to communicate information to them regarding best practices in reading instruction for students with disabilities and to provide ongoing consultation relating to th ese best practices using a systematic problem solving process, such as the collaborative strategic planning (CSP) process recommended by Stollar, Poth, Curtis, and Cohen (2006). Summary of Recommendations The following is a brief summary of each of the re commendations for improvement and the identified next steps of the ELS Literacy Initiative. While some of these recommendations have already been discussed with the program administrator and other stakeholders in the program and acted upon, others are sol ely the recommendations of the evaluator and will be shared with program stakeholders for feedback and input.

PAGE 355

339 Recommendation 1 The ELS program has purchased and disseminated a research based early literacy curriculum (the Early Literacy Skill Builder, o r ELSB) that was designed specifically to meet the needs of students with moderate to severe disabilities and can be adapted for use with students who are nonverbal for the primary level classrooms. The search should continue for a similar type of program that can be used at the intermediate level. Recommendation 2 The ELS program has purchased a subscription to an online file storage and management system, www.box.net that provides a place where teachers and staff in the ELS program can effectively a nd efficiently share documents and materials. Teachers have been encouraged to upload the instructional materials that they have created to this site. Other literacy resources will also be made available electronically using this site, including the ELS Literacy Scope and Sequence and corresponding assessment and instructional resources. The program should continue to find ways that the online file sharing capabilities of this site can facilitate teacher access to resources related to the literacy initia tive. Recommendation 3 The cooperative district is in the process of moving to a web based server, SharePoint, that has the potential to provide the same online file storage and sharing capabilities as www.box.net The additional capabilities of this server, such as its ability to provide an online forum for teacher collaboration and discussion and its ability to provide easy access to student data, should be explored.

PAGE 356

340 Recommendation 4 Teachers should be provided with clear and easy to access inf ormation about the curricular and technology resources that are available to them, including location of the resources, identification of the skills addressed by the resources, the grade levels for which they are appropriate, the types of students for whom each resource is best suited, and how to access the resources. This information could be made accessible to teachers through the online file storage site www.box.net Recommendation 5 The program and cooperative district should continue to provide pro fessional development in the area of literacy instruction for teachers and ELS staff. Training needs that have been identified for teachers include information on how literacy develops in students with moderate to severe disabilities, what skills should b e taught as part of a comprehensive literacy instruction program, how to identify where students are in their literacy development, and how to identify appropriate instructional targets. Teachers have also requested further training relating to instructio nal strategies for students who are at the earliest levels of literacy development as well as to instructional strategies dealing with reading comprehension and written expression. The program may want to consider developing a professional development str and that offers both basic and advanced training in the area of literacy on a regular basis. Recommendation 6 Teachers should also be encouraged to attend workshops and reading conferences outside of the cooperative district that are geared toward gene ral education. Teachers reported that gaining this outside perspective has been helpful in planning instruction. It

PAGE 357

341 is also recommended that ELS teachers gain a greater understanding of the Illinois Learning Standards as they apply to literacy. Familiar ity with the learning standards can facilitate the linkage of ELS strategies to the general education curriculum. Recommendation 7 In addition to offering supplemental professional development to teachers, it is recommended that the program provide add itional training opportunities for teaching assistants and parents. A workshop for teaching assistants has already been developed that provides information about the basics of literacy development and instruction for student with moderate to severe disabi lities. This training is being offered twice during the 2008 2009 school year. It is also recommended that the ELS program explore ways for providing more information to parents about best practices in literacy instruction and how parents can support lit eracy skills at home. Recommendation 8 The collection of literacy benchmark data has been discontinued as of the 2008 2009 school year. It is recommended that the intervention specialists in the program take a more individualized approach to literacy assessment, identifying for each student which assessments of student skill strengths and weaknesses are necessary for instructional planning and developing a progress monitoring plan that will provide the most helpful information. Furthermore, the progra m should continue to identify progress monitoring tools and strategies that are reliable and valid and that can be used with students who may have processing issues or physical limitations, or who may be nonverbal.

PAGE 358

342 Recommendation 9 The intervention specialists in the program should provide more ongoing and direct support to teachers in the instructional planning process, helping them to interpret student assessments, identify instructional targets, develop individualized instructional plans, locate a nd implement curricular resources, identify the training needs of staff, and determine when changes in instruction need to be made. In this role, the intervention specialists can also facilitate teacher use of the resources that have been made available t o them through the literacy initiative, such as the Direct Instruction programs and the ELS Literacy Scope and Sequence The intervention specialists have already begun to define the instructional planning process and identify their role in supporting tha t process. The intervention specialists should be provided with additional training and support as necessary to prepare them to support literacy instruction in the classroom. Recommendation 10 Teachers find the consultative support provided to be ve ry helpful, and it is recommended that the technology consultant and reading coach continue to provide on site support. Given that this type of support is found to be so helpful to teachers, it is recommended that the program explore options for increasin g coaching support to teachers. One recommendation that has been made is for the intervention specialists to provide this additional instructional support to teachers. Recommendation 11 It is recommended that the ELS program develop and adopt a standa rd instructional planning process and hold teachers accountable for using that process. Appendix H contains an instructional planning process that was described by the

PAGE 359

343 intervention specialists in the program. It is recommended that feedback from other st akeholder groups be solicited with regard to the adoption of such a process. Standardizing the instructional planning process has the potential to provide clear expectations, increase instructional consistency between classrooms, increase the likelihood t hat teachers will engage in a quality instructional planning process, and allow for the program to hold teachers accountable for the activities that are a part of the process. Recommendation 12 In order to improve communication among teachers and incr ease instructional consistency, the program should hold teachers accountable for completing and using the ELS literacy tracking form. To facilitate this objective, the program should continue to explore ways to make the completion and sharing of this docu ment more efficient, such as putting the information on a CD that will be passed from one teacher to another or using the new web based server to house this information. Teachers would also benefit from additional professional development and coaching sup port on how to use the literacy tracking form for instructional planning. Recommendation 13 The ELS program should continue to track student literacy outcomes. This is currently done using information from the ELS literacy tracking form; however the p rogram should continue to research assessment tools that can provide a direct measure and serve as a measure of programmatic outcomes. Meanwhile, it is recommende d that strides be taken to increase the reliability of the data solicited the literacy tracking form.

PAGE 360

344 One way to do this would be to provide more information to teachers and intervention specialists on which assessment tools should be used to measure stud in each of the skill areas and how scores on those assessments align with the four stages of literacy development that are tracked on the form. The program should commit to evaluating student literacy outcomes on an annual basis and usin g the information, along with other sources of data, to determine how literacy instruction in the ELS program can continue to be improved. Recommendation 14 Teachers should be held accountable for having a written instructional plan for each of their st udents. To support teachers in achieving this goal, the program should identify a specific planning template to be used and provide teachers with professional development opportunities and on site coaching in the use of the template. Recommendation 15 The program should encourage communication and collaboration among team members for instructional planning. All team members should be involved in the planning and decision making process for each student, including the ELS teacher, general education tea cher, parent(s), teaching assistants, and other support staff members in the classroom (e.g., the intervention specialist and speech and language pathologist). Having set times during the year when teams review data and make decisions can encourage commun ication and collaboration. Recommendation 16 Teachers should take strides to meaningfully involve parents in the planning of literacy instruction and in the instructional decisions that are made on an ongoing basis.

PAGE 361

345 In order for parents to be active and meaningful participants in this process, it will be important to provide them with information on how literacy develops and best practices in literacy instruction for students with moderate to severe disabilities. This information can be made available t o parents through a variety of methods, such as meetings with the ELS parent group, offering workshops for parents, or sending information home to parents. The program committee may also wish to create a webpage that provides information for parents in th e ELS program. Recommendation 17 The ELS program should take strides to ensure that the speech and language pathologists are an integral part of the planning and delivery of literacy instruction. Other recommendations that have been made, such as iden tifying a time for communication and collaboration among team members and standardizing the instructional planning process, can also help facilitate the involvement of the speech and language pathologists. In addition, it is recommended that the program p rovide additional training to speech and language pathologists about the relationship between language and literacy and how they can support literacy instruction in the classroom. Recommendation 18 The ELS program should begin to plan for when and how d ata will be collected regarding the impact of the ELS Literacy Initiative on post school outcomes. Recommendation 19 To serve as a model for other districts, the ELS program should develop a plan for communicating with districts about best practices in reading instruction for students with

PAGE 362

346 disabilities and how the program will provide ongoing consultation to districts who wish to improve instruction to students in their special education programs. Dissemination of Information The primary goal of this program evaluation study was to assist the ELS program administrator in decision making regarding the ELS Literacy Initiative. Therefore, the results of the evaluation will first and foremost be shared with the program administrator, and who will ultimate ly decide how the results will be shared and used in making decisions about the Initiative. It is recommended that the program administrator involve the key stakeholder groups in interpreting the evaluation results and in identifying the next steps in the implementation of the literacy initiative. Key stakeholder groups in the program include the curriculum committee, which includes teacher representatives from all grade levels, and the program planning committee, which is the decision making body of the ELS program and is made up of representatives of teachers and support staff in the program. It is important to involve these stakeholder groups in the decision making process because many of the recommendations that were made as part of this evaluation st udy have would impact the roles and responsibilities of various staff members in the program. It is recommended that the program administrator take several steps in the dissemination of the evaluation outcomes and results. First, a s part of the program evaluation process, key stakeholders in the program should be asked to provide comments on the validity of the results of the evaluation. Once the stakeholder groups have provided this feedback, the next step would be to review the recommendations and dec ide on actions. A plan of action may include determining the problem to be

PAGE 363

347 addressed, the actions taken to address the problem, the necessary resources, timelines for implementation, desired outcomes of the plan of action, and an intended date to review t he progress of the plan. Because the evaluator currently serves as a consultant on curriculum and instruction to the ELS program, she is in a unique position to support the decision making process and follow through on the identified plans of action. Fur thermore, the evaluator will be available to work closely with the program administrator to review the progress of the ELS Literacy Initiative as well as its impact on student outcomes on an annual basis Limitations The results of this program evaluatio n study must be interpreted within the identified limitations. These limitations can be grouped as (a) contextual limitations, (b) design limitations, and (c) impact limitations. Contextual Limitations Contextual limitations are related to the condition s surrounding the initiation and completion of the evaluation study. For example, the current evaluation was initiated by the ELS program administrator. Program evalu ations are typically initiated by a client who seeks the support of an evaluator to help answer questions about a specific program or initiative. However, the evaluator initiated the evaluation in this case because she had significant involvement with the implementation of the ELS Literacy Initiative and was genuinely interested in helping the client determine the appropriate next steps in the program.

PAGE 364

348 Another contextual limitation is the fact that the evaluator is a staff member of the ELS program. Acco rding to Worthen, Sanders, and Fitzpatrick (1997), there can be some disadvantages to having an internal evaluator, such as not being able to gain an impartial perspective on the evaluation. (Strides taken by the evaluator to avoid researcher bias are dis benefits to having an evaluator who is involved with the program being evaluated, including the fact that the evaluator has a knowledge of the history of the program; familiar ity with the stakeholders and their interests, concerns, and potential hidden agendas; and the ability to support the program by using the results of the evaluation to drive practice, thereby helping to implement the recommended changes. Design Limitation s One of the greatest potential limitations of the current evaluation study was that of researcher bias. The potential for researcher bias influencing the evaluation was strong because the evaluator had personal investment in the program being evaluated, and because studies that use qualitative methodologies are at a greater risk of being influenced by researcher bias. Strides taken to reduce the potential for researcher bias included (a) identifying the beliefs and theoretical assumptions of the evaluato r and making personal reflections regarding the potential effects of these beliefs, (b) seeking reflections on the potential of bias from peers and conducting reliability checks on conclusions drawn from the qualitative data, (c) triangulating the data whe n describing and all decisions made by the evaluator.

PAGE 365

349 The evaluator of the program is a staff member of the ELS program and currently serves as a curriculum an d instruction consultant to the program. It is important to note that she has a strong personal investment with the ELS Literacy Initiative because of her involvement with the development and implementation of the initiative. Because of this personal inv olvement, the potential existed for the evaluator to be more biased toward positive outcomes, or for the evaluator to let personal knowledge and experience influence her perspective regarding the results of the study. In an attempt to limit or at least in crease transparency with regard to this bias, personal reflections were made throughout the process regarding how her personal beliefs and investment could have influenced the process and outcomes. These notes were maintained in several places, as a separ ate column within the raw data in the survey results and focus group transcription, as well as within a set of running n otes that the evaluator kept. A n example of one way in which the personal knowledge and bias of the evaluator influenced the evaluation process occurred during the primary level focus group interview, as the evaluator interjected information into the conversation about a resource that would be made available to teachers the following year to address many of their concerns with regard to c urricular resources. By interjecting this type of information, the evaluator may have changed the course of that conversation and influenced the data that were gathered within that focus group. An examination of the personal reflections that were kept th roughout the process suggest that this was one of the only times when the evaluator believed that bias had clearly entered into the process. It was expected that the status of the evaluator and her involvement with the implementation of the literacy initi ative could influence the

PAGE 366

350 behavior of the teachers during the focus group interviews, but there was no evidence to suggest that this was the case. A second perspective was sought to reduce some of the potential for bias by the evaluator, and this second p erspective was used in two different ways. During the focus group interviews, the evaluator served as the facilitator. In addition, a second observer was asked to attend the focus groups to take notes not only on what was being said, but also on group dy namics and the possible influence of the facilitator on the group. This observer was available for two of the three focus group interviews. Outside of noting a group intervie ws, the second observer did not identify significant potential sources of bias. A second perspective was also sought to determine the reliability of the Level 2 codes that were identified through the qualitative data analysis process. The first reliabili ty check resulted in a 60% pe rcent agreement rate; however, after changes were made to the Level 2 coding definitions, the agreement increased to 78%, increasing the confidence that personal bias did not significantly affect the coding process. There was a potential for researcher bias to enter into the process when the results of the study were being recorded. In order to reduce the potential for bias in this way, as many data sources as possible (i.e., quantitative items on surveys, open ended items on surveys, focus group data, Level 1 codes, Level 2 codes, and student literacy development data) were used to inform the results of each of the evaluation questions. This process is called triangulation, referring to comparing the results of several sourc es understanding of what is being studied and supports conclusions that better reflect

PAGE 367

351 source s, those differences were identified and explored. Furthermore, the results were presented in great detail and with as little interpretation as possible in Chapter Four, so that others reviewing the results would be able to make conclusions and interpreta tions of their own. Interpretation of the data was reserved for the discussion in Chapter Five. With regard to the results and conclusions of the study, best practice in program evaluation suggests that program stakeholders should be involved in the dat a interpretation process so as to increase the validity of the results as well as reduce the potential for researcher bias. This process typically involves presenting the stakeholders with a summary of the results and asking them to draw their own conclus ions and make their own recommendations. Unfortunately, time did not allow for the stakeholder review; however, the evaluator intends for this component of the evaluation process to occur in the near future as part of sharing the results of the study and identifying the next steps in the implementation of the literacy initiative. The potential for researcher bias was also addressed through the completion of an audit trail. An audit trail is a written document that explicitly, and with great detail, desc ribes the process used and the decisions made in the collection and analysis of the data so that an external reviewer can follow this path and gain a clear understanding of how the researcher came to his or her conclusions. The evaluator of the current pr ogram evaluation maintained a detailed audit trail log that contained dates, the audit trail entries, and the evidence sources that supported the entries. Another design limitation involved the inability of the evaluator to obtain a direct assessment of student literacy skills over time in order to answer the question of whether

PAGE 368

352 the ELS Literacy Initiative had an impact on student literacy outcomes. The only direct In sufficient numbers of students were assessed with any given literacy benchmarking tool with the same administration format to draw conclusions regarding student progress as a group over time. The other measure of student literacy development, the ELS lite racy tracking form, does not directly measure student skill and is instead an indicator of literacy tracking forms were analyzed to determine whether the students in the ELS program were making progress as a group. It has been recommended that the ELS development. The results of these analyses can then be used to make more defini tive conclusions regarding student progress and achievement. Impact Limitations The primary purpose of this program evaluation study was to support the program administrator and stakeholders of the ELS program in identifying the appropriate next steps in the implementation of the ELS Literacy Initiative. Therefore, the ability to generalize the results of this study to other programs was limited. However, the methods employed and the results of the data analyses were presented in great detail so that re aders might be able to consider potential implications for their own situations. While the results of this study may not have a significant impact on practice in other programs, the evaluation has the potential for having a significant influence on practi ces within the ELS program. Because the evaluator is internal to the program, she can support the interpretation and use of the data, as well as outcomes of the evaluation. Unfortunately,

PAGE 369

353 one limitation to the evaluation study was that the evaluator was not able to meet with stakeholders of the initiative to review the results of the study and discuss the implications of the program. Therefore, the results and discussion contained in this report represent the interpretations and opinions of the evaluator The evaluator does intend to meet with a stakeholder group to discuss results and implications in the near future. Generalization of Results The program evaluation studied a literacy initiative implemented in a special education program that serves stud ents with moderate to severe disabilities as well as students with autism. The initiative was intended to improve curricular resources in the program, increase teacher knowledge and confidence in teaching beginning reading, and improve student outcomes. The current evaluation study was designed to describe how the literacy initiative was currently being implemented, determine if the desired outcomes were being realized, and identify appropriate next steps in program implementation. The evaluation study was not designed to generalize to other special education programs. However, several lessons from the evaluation can be applied to special education programs that serve a similar population of students. The first lesson is that when provided with direct and systematic instruction that targets all literacy skill areas, including phonological awareness and phonics, students with moderate to severe disabilities can make progress in these skill areas. This finding lends some support to the notion that the re ading research that has been conducted with students who are typically developing may have some application to students with significant disabilities. Another lesson that can be taken from the results of this program evaluation is that the Direct

PAGE 370

354 Instruct ion programs can be of use with students who have significant disabilities, but only if those students are able to meet the basic demands of the program, such as being able to respond verbally on cue. Unfortunately, the majority of students in the ELS pro gram do not meet these criteria. However, these research based programs should be used to the greatest extent possible because they are ready to use, can be implemented by teaching assistants, take the mystery out of teaching reading, and are effective wi th this population of students. The final lesson that can be taken from this evaluation and applied to other programs that serve similar student populations is that the existing tools and strategies for assessing literacy skills have serious limitations a nd cannot be used as intended when monitoring the progress of students with significant disabilities. The ELS program has had to significantly modify available progress monitoring and assessment tools to meet the unique needs of this population of student s, such as by modifying the tools so that student performance is judged on accuracy and not fluency. Overall, the ELS program is making strides in providing effective, research based reading instruction to a population of students that has been largely i gnored in the literature on reading instruction. Other programs that serve students with similar characteristics can learn from the current evaluation study and the work of the ELS Literacy Initiative. However, research on the most effective strategies f or teaching literacy to students with significant disabilities and how best to monitor the reading progress of these students is needed.

PAGE 371

355 References Adelman, H.S., & Taylor, L. (1998). Reframing mental health in schools and expanding school reform Educational Psychologist, 33 (4), 135 152. Adelman, H.S. & Taylor, L. (2003). On sustainability of project innovations as systemic change. Journal of Educational and Psychological Consultation, 14 (1), 1 25. Al Otaiba, S., & Hosp, M.K. (2004). Provi ding effective literacy instruction to students with Down syndrome. TEACHING Exceptional Children, 36 (4), 28 35. Batsche, G., Elliot, J., Graden, J.L., Grimes, J., Kovaleski, J.F., Prasse, D., Reschly, D.J., Schrag, J., Tilly, W.D. (2005). Respons e to intervention: Policy considerations and implementation Alexandria, VA: National Association of State Directors of Special Education. Boyd, V. (1992a). Creating a context for change. Issues about Change 2 (2). Austin, TX: Southwest Educati onal Development Laboratory. Boyd, V. (1992b). School context: Bridge or barrier for change Austin, TX: Southwest Educational Development Laboratory. Boyd, V. & Hord, S.M. (1994). Principles and the new paradigm: Schools as learning communities Paper presented at the Annual Meeting of the American Educational Research Association. New Orleans, LA. Bradford, S., Alberto, P., Houchins, D.E., Shippen, M.E., Flores, M. (2006). Using instruction to teach decoding skills to middle school stude nts with moderate intellectual disabilities. Education and Training in Developmental Disabilities, 41 (4), 333 343.

PAGE 372

356 Browder, D.M., Gibbs, S.L., Ahlgrim Delzell, L., Courtade, G., & Lee, A. (2007). Early Literacy Skills Builder Madison, WI: Attainmen t Company. Browder, D.M., & Spooner, F. (Eds.) (2006). Teaching language arts, math, and science to students with significant cognitive disabilities Baltimore: Brookes Publishing. Browder, D.M., Wakeman, S.Y., Spooner, F., Ahlgrim Delzell, L., & Al gozzine, B. (2006). Research on reading instruction for individuals with significant cognitive disabilities. Council for Exceptional Children, 72 (4), 392 408. Browder, D.M., Xin, Y.P. (1998). A meta analysis and review of sight word research and its implications for teaching functional reading to individuals with moderate and severe disabilities. Journal of Special Education, 32 130 153. Burns, M.K., Appleton, J.J., & Stehouwer, J.D. (2005). Meta analytic review of responsiveness to interve ntion research: Examining field based and research implemented models. Journal of Psychoeducational Assessment, 23 (4), 381 394. Chard, D.J., Vaughn, S., & Tyler, B. (2002). A synthesis of research on effective interventions for building reading f luency with elementary students with learning disabilities. Journal of Learning Disabilities, 35 (5), 386 406. Coffman, J. (1999). Learning from logic models: An example of a family/school partnership program. Harvard Family Research Project, Cambr idge, MA. Conners, F.A. (1992). Reading instruction for students with moderate mental retardation: Review and analysis of research. American Journal on Mental Retardation, 96, 577 597.

PAGE 373

357 Cooper, R. (1998). Socio cultural and within school factors tha t affect the quality of implementation of school wide programs (Report No. 28). Baltimore, MD: Center for Research on the Education of Students Placed at Risk. (ERIC Document Reproduction Service No. ED426173). Cuban, L. (1996). Myths about changing sc hools and the case of special education. Rase: Remedial and Special Education, 17 (2), 75 82. Cunningham, P.M., Cunningham, J.W., & Allington, R.L. (2002). Research on the components of a comprehensive reading and writing instructional program Retr ieved February 2, 2008 from http://www.wfu.edu/education/fourblocks/research.html Curtis, M.J., Castillo, J.M., & Cohen, R.M. (2008). Best practices in system level change (pp. 887 901). In A. Thomas, & J. Grimes (Eds.), Best Practices in School Psycho logy V Washington, DC: National Association of School Psychologists. Curtis, M.J., & Stollar, S. (2002). Best practices in system level change. In A. Thomas, & J. Grimes (Eds.), Best Practices in School Psychology IV (pp. 223 234). Washington, DC: National Association of School Psychologists. DeCoste, D. (2005). Literacy assessment, interpretation, and curriculum for students with cognitive disabilities Professional development training, Arlington Heights, IL. Dolch, E.W. (1955). Methods in reading Champaign, IL: Garrard.

PAGE 374

358 Downing, J.E. (2006). Building literacy for students at the presymbolic and early symbolic levels. In D. Browder & F. Spooner (Eds.), Teaching language arts, math, and science ]to students with significant co gnitive disabilities (pp. 39 61). Baltimore: Brookes Publishing. Edmark reading program: Word recognition (2 nd ed.). (1992). Cambridge, MA: Edmark Corporation. Engelmann, S., L. Meyer, L. Carnine, W. Becker, J. Eisele and G. Johnson. (1999). Co rrective Reading: Decoding strategies Worthington, Ohio: Science Research Associate/McGraw Hill. Evans, R. (2001). The human side of school change: Reform, resistance, and the real life problems of innovation San Francisco, CA: Jossey Bass. Fitzge rald, J., Roberts, J., Pierce, P., & Schuele, M. (1995). Evaluation of home literacy environment: An illustration with preschool children with Down syndrome. Reading and Writing Quarterly, 11 311 334. Fixsen, D.L., Naoom, S.F., Blase, K.A., Friedman R.M., & Wallace, F. (2005). Implementation Research: A Synthesis of the Literature Tampa, FL: University of South Florida, Louis de la Parte Florida Mental Health Institute, The National Implementation Research Network (FMHI Publication #231). Flor es, M.M., Shippen, M.E., Alberto, P., & Crowe, L. (2004). Teaching letter sound correspondence to students with moderate intellectual disabilities. Journal of Direct Instruction, 4 (2), 173 188.

PAGE 375

359 Fuchs, L.S., & Fuchs, D. (n.d.). What is scientifica lly based research on progress monitoring ? National Center on Student Progress Monitoring. Retrieved January 15, 2008 from http://www.studentprogress.org/library/What_is_Scientificall_% 20Based_Research.pdf Fullen, M. (1993). Change forces: Probing the depths of educational reform. Bristol, PA: Falmer Press. Fullen, M. (1997). The complexity of the change process. In M. Fullen (Ed.), The challenge of school change (pp. 32 56) Arlington Heights, IL: IRI/SkyLight Training. Gall, M.D., Borg, W.R., & Gall, J.P. (1996). Educational research: An introduction, 6 th Edition. White Plains, NY: Longman Publishers. Gersten, R., Fuchs, L.S., Compton, D., Coyne, M., Greenwood, C., & Innocenti, M. (2005). Quality indicators for group experimental and quasi experimental research in special education. Exceptional Children, 71 149 165. Goldman, K.D., & Schmalz, K.J. (January 2006). Logic models: The picture worth ten thousand words. Health Promotion Practice 8 12. Gough, P.B. (1996). How child ren learn to read and why they fail. Annals of Dyslexia 46, 3 20. Grimes, J. & Tilly, W.D. (1996). Policy and process: Means to lasting educational change. School Psychology Review, 25 (4), 465 476.

PAGE 376

360 Gurry, S.E., & Larkin, A.S. (1999). Liter acy learning abilities of children with developmental disabilities: What do we know? Currents in Literacy. 2 (1). Retrieved March 3, 2008 from http://www.lesley.edu/academic_ centers/hood/currents/v2n1/gurry_larkin.html Hall, G.E., & Hord, S.M. (200 5). Implementing change: Patterns, principles and potholes, 2 nd Ed. Boston, MA: Allyn and Bacon. Hargreaves, A. (1997). Cultures of teaching and educational change. In M. Fullen (Ed.), The challenge of school change (pp. 57 84) Arlington Heights IL: IRI/SkyLight Training. Hasbrouck, J.E., & Tindal, G. (1992). Curriculum based oral reading fluency norms for students in grades 2 through 5. TEACHING Exceptional Children, 24 (3), 41 44. Hedrick, W.B., Katims, D.S., & Carr, N.J. (1999). Imp lementing a multimethod, multilevel literacy program for students with mental retardation. Focus on Autism and Other Developmental Disabilities, 14 (4), 231 239. Hord, S.M. (1997). Professional learning communities: Communities of continuous inquiry a nd improvement. Austin: Southwest Educational Development Laboratory. Horner, R.H., Carr, E.G., Halle, J., McGee, G., Odom, S., & Wolery, M. (2005). The use of single subject research to identify evidence based practice in special education. Exception al Children, 71 165 180.

PAGE 377

361 Houston, D., Al Otaiba, S., & Torgesen, J.K. (2006). Learning to read: Phonics and fluency. In D. Browder & F. Spooner (Eds.), Teaching language arts, math, and science to students with significant cognitive disabilities (pp. 39 61). Baltimore: Brookes Publishing. Houston, D., Torgesen, J. (2004). Teaching Students with moderate disabilities to read: Insights from research Tallahassee, FL: Florida Department of Education, bureau of Instructional Support and Comm unity Services. Retrieved March 3, 2008 from http://www.cpt.fsu.edu/ESE/pdf/ESE_Read.pdf Individuals with Disabilities Education Improvement Act of 2004, PL 108 446, 20 U.S.C. §§ 1400 et. seq The Joint Committee on Standards for Educational Evaluation. (1994). The program evaluation standards (2nd ed.) Thousand Oaks, CA: Sage. Joseph, L.M., & Seery, M.E. (2004). Where is the phonics? A review of the literature on the use of phonetic analy sis with students with mental retardation. Remedial and Special Education, 25 (2), 55 94. Joyce, B., & Showers, B. (2002). Student achievement through staff development, 3 rd Edition Alexandria, VA: The Association for Supervision and Curriculum De velopment. Katims, D.S. (2000). Literacy instruction for people with mental retardation: Historical highlights and contemporary analysis. Education and Training in Mental Retardation and Developmental Disabilities, 35 (1), 3 15.

PAGE 378

362 Kliewer, C., & Bikl synthesis of the literate lives of students with severe disabilities. Journal of the Association for Persons with Severe Handicaps (JASH), 26 (1), 1 12. Kliewer, C., & Landis, D. (1 999). Individualizing literacy instruction for young children with moderate to severe disabilities. Exceptional Children, 66 (1), 85 100. Koppenhaver, D., Spadorcia, S., & Harrison, M. (1998). Improving reading comprehension for children with disab ilities: A review of research (Final Report, Section II). Washington, DC: Special Education Programs ED/OSERS. Koppenhaver, D., Coleman, P.P., Kalman, S.L., & Yoder, D.E. (1991). The implications of emergent literacy research for children with develo pmental disabilities. American Journal of Speech Language Pathology, 1 38 44. Lehman, W.E.K., Greener, J.M., & Simpson, D.D. (2002). Assessing organizational readiness for change. Journal of Substance Abuse Treatment, 22 (4), 197 209. Light, J., & Kelford Smith, A. (1993). The home literacy experiences of preschoolers who use augmentative communication systems and their nondisabled peers. Augmentative and Alternative Communication, 9 10 25. opment in a climate of educational reform. In M. Fullan (Ed.), The Challenge of School Change (pp. 57 84). Arlington Heights IL: Skylight Professional Development. Mendez, Morse, S. (1992). Leadership characteristics that facilitate school change A ustin, TX: Southwest Educational Development Laboratory. Merriam, S.B. (1998). Qualitative research and case study applications in education. San Francisco, CA: Jossey Bass Publishers.

PAGE 379

363 National Institute of Child Health and Human Development. (2000 ). Report of the National Reading Panel. Teaching children to read: an evidence based assessment of the scientific research literature on reading and its implications for reading instruction: Reports of the subgroups (NIH Publication No. 00 4754). Washing ton, DC: U.S. Government Printing Office. No Child Left Behind Act of 2001, PL 107 110, 115 Stat. 1425, 20 U.S.C. §§ 6301 et. seq. Orelove, F.P., & Sobsey, D. (1996). Educating children with multiple disabilities: A transdiciplinary approach (3 rd Edit ion). Baltimore: Brookes Publishing. Senge, P. (1990). The fifth discipline New York, Doubleday. Snow, C.E., Burns, M.S., & Griffin, P. (1998). Preventing reading difficulties in young children Washington, DC: National Academy Press. Retrieve d November 15, 2007 from http://www.nap.edu/books/030906418X/html/index.html Stollar, S.A., Poth, R.L., Curtis, M.J., & Cohen, R.M. (2006). Collaborative strategic planning as illustr ation of the principles of systems change. School Psychology Review 35(2), 181 197. Stufflebeam, D.L. (1971). The relevance of the CIPP evaluation model for educational accountability. Journal of Research and Development in Education, 5, 19 25. S undberg, Mark L. and James. W. Partington. The Assessment of Basic Language and Learning Skills (The ABLLS): An assessment, curriculum guide, and skills tracking system for children with autism or other developmental disabilities. Pleasant Hill, CA: Behav ioral Analysts, Inc., 1998.

PAGE 380

364 Lonigan, C.J., Burgess, S.R., & Anthony, J.L. (2000). Development of emergent literacy and early reading skills in preschool children: Evidence from a latent variable longitudinal study. Developmental Psychology, 36 (5), 596 613. U.S. Department of Labor. (1991). What work requires of schools: A SCANS report for America 2000 Washington, DC: Department of Labor. (ERIC Document Reproduction Service No. ED332054) Worthen, B.R., Sanders, J.R., & Fitzpatrick, J.L. (1997). Program evaluation: Alternative approaches and practical guidelines, 2 nd Edition. White Plaines, NY: Longman Publishers.

PAGE 381

365 A PPENDICES

PAGE 382

366 Appendix A: A Logic Model of the ELS Literacy Initiative

PAGE 383

367 Appendix B : ELS Li teracy Tracking Form

PAGE 384

368 Appendix C : Evaluation Questions and Data Sources Evaluation Question Data Source(s) PROCESS EVALUATION: IMPLEMENTATION 1. How are the components of the ELS Literacy Initiative currently being implemented? (e.g., What are the roles of various team members? How is instruction being delivered? How are resources being utilized?) Teacher Survey Question 2: Please rate the helpfulness of the following resources in implementing literacy instruction in your classroom Teacher Survey Qu estion 3: to why. Teacher Survey Question 5: Please rate the helpfulness of the following people and professional development resourc es in implementing literacy instruction in your classroom: Teacher Survey Question 6: that resource could have been more helpful. T eacher Survey Question 10: General comments on literacy resources and supports. Teacher Survey Question 11: Please indicate the number of students for whom you use the following literacy instructional resources. Teacher Survey Question 12: Other in structional materials that I use on a regular basis include. Teacher Survey Question 15: Please briefly describe the role, if any, of each of the following individuals in planning, implementing, and evaluating literacy instruction in your classroom. Teacher Survey Question 22: Please indicate how helpful the following sources of literacy data are for planning and delivering literacy instruction in your classroom. Teacher Survey Question 23: How often do you use data in developing student literac y instructional plans? Teacher Survey Question 24: How often do you use data in deciding when to make instructional changes? Teacher Focus Group Interview Guide Question 1: What does literacy instruction look like in your classroom? In other wor ds, if I were to come into your classroom to observe, what would I see? Teacher Focus Group Interview Guide Question 2: Tell me how you are using material resources in your classroom to support literacy instruction. Teacher Focus Group Intervie w Guide Question 3: Tell me about how you are using coaching support and professional development opportunities to support literacy instruction in your classroom. Teacher Focus Group Interview Guide Question 6: Tell me how you go about planning st udent instruction. Support Staff Focus Group Interview Guide Question 1: Tell me about your role in supporting literacy instruction. Support Staff Focus Group Interview Guide Question 2: Tell me how you are using material resources to support li teracy instruction in your classroom. Support Staff Focus Group Interview Guide Question 3: Tell me about how you are using coaching support and professional development opportunities to support literacy instruction in your classroom. Support Staf f Focus Group Interview Guide Question 6: Tell me how you are using literacy data to support instruction. Qualitative Data Analysis: Level 1 and Level 2 codes

PAGE 385

369 Appendix C : (Continued) Evaluation Question Data Source(s) 2. What factors serve to facilit ate implementation of the ELS Literacy Initiative? Teacher Survey Question 14: How, if at all, do the factors below FACILITATE the implementation of comprehensive and systematic literacy instruction in your classroom? (For example, one factor might be ha ving access to a published curriculum.) Teacher Focus Group Interview Guide Question 1e: Tell me about your greatest success story this past year of a student who made the most growth in reading. What factors were most important in creating this succ ess story? Teacher Focus Group Interview Guide Question 5: What factors in your classroom have facilitated the implementation of literacy instruction (students, staff, classroom environment, materials, training, other)? Support Staff Focus Group Int erview Guide Question 4: What factors in your classroom have facilitated the implementation of literacy instruction (students, staff, classroom environment, materials, training, other)? Qualitative Data Analysis: Level 1 and Level 2 codes 3. What factor s serve as barriers to the implementation of the ELS Literacy Initiative? Teacher Survey Question 13: How, if at all, do the factors below serve as BARRIERS to the implementation of comprehensive and systematic literacy instruction in your classroom? (Fo r example, one barrier might be lack of time to adequately plan instruction.) Teacher Focus Group Interview Guide Question 1f: Tell me about your greatest challenge this past year the student who did not make as much growth as you would like. Why do you think this was the case? Teacher Focus Group Interview Guide Question 4: What factors in your classroom have served as barriers to the implementation of literacy instruction (students, staff, classroom environment, materials, training, other)? S upport Staff Focus Group Interview Guide Question 3: What factors in your classroom have served as barriers to the implementation of literacy instruction (students, staff, classroom environment, materials, training, other)? Qualitative Data Analysis: Level 1 and Level 2 codes

PAGE 386

370 Appendix C : (Continued) Evaluation Question Data Source(s) PRODUCT EVALUATION: SHORT TERM AND INTERMEDIATE OUTCOMES 4. How are teachers using the resources (material and people) they have been provided to help support literacy instruction in their classrooms? Teacher Survey Question 2: Please rate the helpfulness of the following resources in implementing literacy instruction in your classroom: Teacher Survey Question 3: all H to why: Teacher Survey Question 5: Please rate the helpfulness of the following people and professional development resources in implementing literacy instruction in your classroom: Teacher Survey Question 6: that resource could have been more helpful: Teacher Survey Question 10: General comments on literacy res ources and supports: Teacher Survey Question 11: Please indicate the number of students for whom you use the following literacy instructional resources: Teacher Survey Question 12: Other instructional materials that I use on a regular basis include : Teacher Survey Question 15: Please briefly describe the role, if any, of each of the following individuals in planning, implementing, and evaluating literacy instruction in your classroom: Teacher Focus Group Interview Guide Question 1: What do es literacy instruction look like in your classroom? In other words, if I were to come into your classroom to observe, what would I see? Teacher Focus Group Interview Guide Question 2: Tell me how you are using material resources in your classroom t o support literacy instruction. Teacher Focus Group Interview Guide Question 3: Tell me about how you are using coaching support and professional development opportunities to support literacy instruction in your classroom. Teacher Focus Group Int erview Guide Question 6: Tell me how you go about planning student instruction. Support Staff Focus Group Interview Guide Question 1: Tell me about your role in supporting literacy instruction. Support Staff Focus Group Interview Guide Question 2: Tell me how you are using material resources to support literacy instruction in your classroom. Support Staff Focus Group Interview Guide Question 3: Tell me about how you are using coaching support and professional development opportunities to su pport literacy instruction in your classroom. Support Staff Focus Group Interview Guide Question 6: Tell me how you are using literacy data to support instruction. Qualitative Data Analysis: Level 1 and Level 2 codes

PAGE 387

371 Appendix C : (Continued) Eva luation Question Data Source(s) 5. How do teachers rate their feelings of support, preparedness, and confidence as a result of the resources that they have been provided through the literacy initiative? Teacher Survey Question 4: Overall I am adequately su pported with sufficient material resources for literacy instruction in my classroom. Teacher Survey Question 7: Overall, I am adequately supported through the coaching and professional development opportunities available for literacy instruction in my classroom. Teacher Survey Question 8: I am confident in providing literacy instruction in my classroom. Teacher Survey Question 9: I am better prepared to provide literacy instruction because of the support and resources provided to me. Teacher Survey Question 10: General comments on literacy resources and supports: Qualitative Data Analysis: Level 1 and Level 2 codes 6. To what extent is instruction in the area of literacy aligned with best practices and current research (e.g., comprehensive programming, direct instruction, data to inform instruction, literacy across the day, etc.) as reported by teachers? Teacher Survey Question 11: Please indicate the number of students for whom you use the following literacy instructional resources: Te acher Survey Question 12: Other instructional materials that I use on a regular basis include: Teacher Survey Question 16: For each student in your classroom, indicate whether each of the following is part of their regular, systematic literacy instru ction: Teacher Survey Question 17: Literacy instruction (formal and informal) is provided throughout the school day for: Teacher Survey Question 18: There is a designated time for literacy instruction in the schedule every day for: Teacher Surve y Question 19: I have a written literacy instructional plan for: Teacher Survey Question 23: How often do you use data in developing student literacy instructional plans? Teacher Survey Question 24: How often do you use data in deciding when to make instructional changes? Teacher Focus Group Interview Guide Question 6: Tell me how you go about planning student instruction. Qualitative Data Analysis: Level 1 and Level 2 codes 7. To what extent do students in the ELS program have access to ap propriate, research based literacy instruction as reported by teachers? Teacher Survey Question 11: Please indicate the number of students for whom you use the following literacy instructional resources: Teacher Survey Question 12: Other instruction al materials that I use on a regular basis include: Teacher Survey Question 16: For each student in your classroom, indicate whether each of the following is part of their regular, systematic literacy instruction: Teacher Survey Question 20: The students in my classroom have access to appropriate, research based literacy instruction. Qualitative Data Analysis: Level 1 and Level 2 codes

PAGE 388

372 Appendix C : (Continued) Evaluation Question Data Source(s) 8. How are literacy data being used in the classr oom and how do teachers rate those data sources in terms of helpfulness? Teacher Survey Question 22: Please indicate how helpful the following sources of literacy data are for planning and delivering literacy instruction in your classroom: Teacher S urvey Question 23: How often do you use data in developing student literacy instructional plans? Teacher Survey Question 24: How often do you use data in deciding when to make instructional changes? Teacher Focus Group Interview Guide Question 6: Tell me how you go about planning student instruction. Teacher Focus Group Interview Guide Question 7: Tell me how you are using literacy data in your classroom. Support Staff Focus Group Interview Guide Question 6: Tell me how you are using li teracy data to support instruction. Qualitative Data Analysis: Level 1 and Level 2 codes 9. How are the literacy benchmark data being utilized? Are the data adequate to support these uses? Teacher Survey Question 22: Please indicate how helpful the fol lowing sources of literacy data are for planning and delivering literacy instruction in your classroom: Teacher Focus Group Interview Guide Question 7: Tell me how you are using literacy data in your classroom. Literacy Benchmark Data Analysis acr oss multiple years Qualitative Data Analysis: Level 1 and Level 2 codes 10. To what extent do teachers believe there is instructional continuity for individual students as they move from one teacher to the next? Teacher Survey Question 26: To what extent is there instructional continuity when students transition from one teacher to the next in the ELS program? Teacher Focus Group Interview Guide Question 8: What have been some of the positive outcomes related to literacy instruction in your classroom ? Qualitative Data Analysis: Level 1 and Level 2 codes 11. To what extent do teachers believe the activities of the ELS Literacy Initiative have impacted the inclusion / integration of students in ELS into district classrooms? Teacher Survey Question 27: The literacy instruction that has taken place in my classroom has improved the inclusion / integration of students into district classrooms. Teacher Focus Group Interview Guide Question 8: What have been some of the positive outcomes related to lite racy instruction in your classroom? Qualitative Data Analysis: Level 1 and Level 2 codes 12. To what extent do teachers and parents believe the activities of the ELS Literacy Initiative have impacted generalization of literacy skills to the home? Teache r Survey Question 28: The literacy skills that students have worked on in my classroom have generalized to the home setting. Parent Survey Question 1: To what extent have you seen literacy skills gained by your child in the school setting carry over to the home setting? Parent Survey Question 2: Please explain your answer to question 1. Parent Survey Question 3: How satisfied are you with communication Parent Survey Ques tion 4: Please comment on communication between Parent Survey Question 5: Please describe how literacy is reinforced and supported with your child at home. Teacher Focus Group Interview Guide Question 8: What have been some of the positive outcomes related to literacy instruction in your classroom? Qualitative Data Analy sis: Level 1 and Level 2 codes

PAGE 389

373 Appendix C : (Continued) Evaluation Question Data Source(s) 13. To what extent have student outcomes in the area of literacy been impacted as a result of the Literacy Initiative? Teacher Survey Question 29: Student literacy skills have improved because of the resources in my classroom and the training that I have been provided. Teac her Focus Group Interview Guide Question 8: What have been some of the positive outcomes related to literacy instruction in your classroom? Support Staff Focus Group Interview Guide Question 7: What have been some of the positive outcomes related to literacy instruction in your classroom? Literacy Benchmark Data Analysis across multiple years Literacy Development Tracking Data Analysis across two years Qualitative Data Analysis: Level 1 and Level 2 codes NEXT STEPS 14. What should be the next steps in the Literacy Initiative? Teacher Survey Question 31: Please comment on how you think literacy instruction and outcomes can be improved for students in the ELS program: Parent Survey Question 6: Based on your experiences, how can the ELS program improve literacy outcomes for students? Teacher Focus Group Interview Guide Question 9: What would help you improve literacy outcomes in your classroom right now? Teacher Focus Group Interview Guide Question 10: If you had unlimited reso urces and were in charge of the program, how would you change things to improve literacy outcomes for students in the ELS program? Support Staff Focus Group Interview Guide Question 8: What would help you improve literacy outcomes in your classroom right now? Support Staff Focus Group Interview Guide Question 9: If you had unlimited resources and were in charge of the program, how would you change things to improve literacy outcomes for students in the ELS program? Qualitative Data Analysis: Level 1 and Level 2 codes

PAGE 390

374 Appendix D : Teacher Survey

PAGE 391

375 Appendix D : (Continued)

PAGE 392

376 Appendix D : (Continued)

PAGE 393

377 Appendix D : (Continued)

PAGE 394

378 Appendix D : (Continued)

PAGE 395

379 Appendix D : (Continued)

PAGE 396

380 Appendix D : (Continued)

PAGE 397

381 Appendix E : Parent Survey

PAGE 398

382 Appendix E : (Continued)

PAGE 399

383 Appendi x F : Teacher Focus Group Interview Guide Teachers present : Location : Date : Start time and End Time : Describe the Room : represent broad open ended questions that are designed to stimulate conversation related to specific topics. Probing questions (indented and identified with letters under each general question) will be used to follow up if the group has not discussed or brought up the topic on their own. The probing questions will be used on an as needed basis. Opening comments o Remind the group of the purpose of the discussion. o Inform the group that the discussion is only for those persons present and that specific comments will n ot be identified with individuals and will not be shared in that way with anyone outside the group. o Remind the group why it is important to be open and honest and to try to make everyone comfortable within the context of the confidential conversation. o Info rm the members that participation in the discussion is voluntary and that any individual can choose to withdraw and leave the group at any time. However, since no comments will be identified in terms of the originator, no comments will be deleted from the recording and transcription should the originator choose to discontinue participation. o Ask if anyone has any questions o Ask members to sign the informed consent form if they have not done so already. Only persons who have signed an informed consent form may participate. o Provide a definition of literacy in the ELS program: For the purposes of this survey, the term "literacy" is defined broadly. The following are examples of literacy skills that are targeted in the ELS program: interest in / understanding of books awareness of the sounds that make up words letter knowledge letter / sound correspondence knowledge sight word identification picture vocabulary oral vocabulary written vocabulary expressive and receptive communicati on reading words, sentences, and paragraphs

PAGE 400

384 Appendix F : (Continued) Students may demonstrate these skills expressively or receptively or they may use assistive technology to demonstrate these literacy skills. 1) What does literacy instruction look lik e in your classroom? In other words, if I were to come into your classroom to observe, what would I see? a. Describe when literacy instruction is provided in a group format and what it looks like. b. Describe when literacy instruction is provided individuall y and what that looks like. c. Tell me about the role of teaching assistants in supporting literacy instruction in your classroom. d. Tell me about the role of other support staff in supporting literacy instruction in your classroom. e. Tell me about your greatest success story this past year of a student who made the most growth in reading. What factors were most important in creating this success story? f. Tell me about your greatest challenge this past year the student who did not make as much growth as you w ould like. Why do you think this was the case? 2) Tell me how you are using material resources in your classroom to support literacy instruction. a. How are you using the published curriculum materials that are in your classroom? b. How are you using Reading Mastery and Language for Learning (the identified core curriculum)? c. How are you using teacher created materials? d. How are you using the ELS Literacy scope and sequence and resource binders? e. How are you using computer software programs? f. How are you using literacy websites? g. How are you using the instructional planning tools that you have been provided? h. What other material resources are you using to support literacy instruction? i. currently have?

PAGE 401

385 Appendix F : (Continued) 3) Tell me about how you are using coaching support and professional development opportunities to support literacy instruction in your classroom. a. How are you using the Literacy Coach? If not, why? b. How are you using the te chnology coach? If not, why? c. How are you using other people resources in the program? d. In what types of professional development activities have you participated in? What has been the most helpful? e. What types of professional development activities are your staff accessing? What has been the most helpful to them? 4) What factors in your classroom have served as barriers to the implementation of literacy instruction (students, staff, classroom environment, materials, training, other)? 5) What factors in yo ur classroom have facilitated the implementation of literacy instruction (students, staff, classroom environment, materials, training, other)? 6) Tell me how you go about planning student instruction. a. What information do you use when creating student instru ctional plans? b. What types of skills are you targeting through instruction? c. d. receiving teachers)? e. How do you infuse litera cy instruction and practice into other activities in the 7) Tell me how you are using literacy data in your classroom. a. What are your sources of data? b. How are you using the benchmark data that are collected three times a year by the In tervention Specialist? c. How are you monitoring student literacy progress? d. How are you using student literacy data in relation to student IEP goals? e. How do you know when to change instruction? 8) What have been some of the positive outcomes related to lite racy instruction in your classroom? a. Tell me about the home / school relationship when it comes to literacy instruction. b. Tell me how literacy instruction in your classroom has had an impact on c. What have the student related outcomes been? d. consistency of literacy instruction.

PAGE 402

386 Appendix F : (Continued) 9) What would help you improve literacy outcomes in your classroom right now? 10) If you had unlimited resources and were in charge of the program, how would you change things to improve literacy outcomes for students in the ELS program? 11) What else would you like to share with me about literacy instruction?

PAGE 403

387 Appendix G : Support Staff Focus Group Interview Guide Support Staff P resent : Location : Date : Start time and End Time : Describe the Room : represent broad, open ended questions that are designed to stimulate conversation related to specific topics. Probing questions (indented and identified with letters under each general question) will be used to follow up if the group has not discussed or brought up the topic on their own. The probing questions will be us ed on an as needed basis. Opening comments o Remind the group of the purpose of the discussion. o Inform the group that the discussion is only for those persons present and that specific comments will not be identified with individuals and will not be sha red in that way with anyone outside the group. o Remind the group why it is important to be honest and to try to make everyone comfortable within the context of the confidential conversation. o Ask members to sign the informed consent form if they have not don e so already. Only persons who have signed an informed consent form may participate. o Inform the members that participation in the discussion is voluntary and that any individual can choose to withdraw and leave the group at any time. However, since no comm ents will be identified in terms of the originator, no comments will be deleted from the recording and transcription should the originator choose to discontinue participation. o Provide a definition of literacy in the ELS program. o Tell group that I am going to be asking some of the same questions that I asked the teacher focus group. Ensure them that it is OK not to have a response to a particular question. o Provide a definition of literacy in the ELS program: For the purposes of this survey, the term "lit eracy" is defined broadly. The following are examples of literacy skills that are targeted in the ELS program: interest in / understanding of books awareness of the sounds that make up words letter knowledge letter / sound correspondence kno wledge sight word identification picture vocabulary oral vocabulary written vocabulary expressive and receptive communication reading words, sentences, and paragraphs

PAGE 404

388 Appendix G : (Continued) Students may demonstrate these skills e xpressively or receptively or they may use assistive technology to demonstrate these literacy skills. 12) Tell me about your role in supporting literacy instruction. a. In other words, if I were to come into the classroom to observe, what would I see if you we re supporting literacy instruction? What actually happens as part of literacy instruction? b. What are you responsible for when it comes to literacy instruction in your classroom? c. What are you not responsible for when it comes to literacy instruction in your classroom? d. Tell me about communication between you and the classroom teacher. e. Tell me about the role of other support staff in supporting literacy instruction. 13) Tell me how you are using material resources to support literacy instruction in your classroo m. a. How are published curriculum materials being used? b. How are Reading Mastery and Language for Learning (the identified core curriculum) being used? c. How are teacher created materials being used? d. How are ELS Literacy scope and sequence and resource bin ders being used? e. How are computer software programs being used? f. How are literacy websites being used? g. What other material resources are being used to support literacy instruction? h. curr ently have? 14) Tell me about how you are using coaching support and professional development opportunities to support literacy instruction in your classroom. a. In what types of professional development activities have you participated during the past year? b. What has been the most helpful? 15) What factors in your classroom have facilitated the implementation of literacy instruction (students, staff, classroom environment, materials, training, other)? 16) What factors in your classroom have served as barriers to the implementation of literacy instruction (students, staff, classroom environment, materials, training, other)?

PAGE 405

389 Appendix G : (Continued) 17) Tell me how you are using literacy data to support instruction. a. What are your sources of data? b. How are you using th e benchmark data that are collected three times a year? c. How are you monitoring student literacy progress? d. How are you using student literacy data in relation to IEP goals? e. How do you know when to change instruction? 18) What have been some of the positive outcomes related to literacy instruction in your classroom? a. Tell me about the impact on students and their skills. b. Tell me about differences between your students and the outcomes that you have seen from the literacy instruction that you have provided. D o some students benefit significantly more than others? Why or why not? 19) What would help you improve literacy outcomes in your classroom right now? 20) If you had unlimited resources and were in charge of the program, how would you change things to improve l iteracy outcomes for students in the ELS program? 21) What else would you like to share with me about literacy instruction?

PAGE 406

390 Appendix H : Draft of ELS Literacy Instructional Planning Process Beginning of the Year: Aug / Sept Resources Identify student cu rrent skill level using existing resources and collecting additional data when necessary. Current IEP Assessments in Literacy Binders Benchmark Assessment Binders AIMSweb Informal assessments / observations Develop com prehensive instructional plan Literacy Binders / Scope and Sequence Instructional planning form Direct Instruction placement tests ELS Instructional Consultants Identify progress monitoring strategy (what, when, who). This has to be done for the IEP goal and may be done in other skill areas also. Current IEP Assessments in Literacy Binders Benchmark Assessment Binders AIMSweb Implement instructional plans Obtain / create materia ls Put into schedule Identify training needs Periodically review progress on implementation Direct Instruction and other Literacy Programs Shared materials on Box.net Literacy Binders / Scope and Sequence NSSED professional development Direct Instruc tion implementation coaching Implement progress monitoring plan Obtain materials when necessary Train staff when necessary Create graph for data AIMSweb Excel Other graphing tools Middle of the Year: Oct Mar Resources Regularly update progress monitoring data on chart / graph. AIMSweb Excel Other graphing tools Periodically review data to determine whether students are making adequate progress. Make instructional changes when appropriate. Note changes in instruction on Instructional Plan or any changes in the goal. Literacy Binders / Scope and Sequence ELS Instructional Consultants Periodically check to ensure Instructional Plans are being carried through with integrity. Are we doing what we said we would do? Direct Instruction integrity checks Direct Instruction implementation coaching

PAGE 407

391 Appendix H : (Continued) End of the Year: Apr Jun Resources Re Evaluations Revi ew existing data Update of student skill level / skill analysis through CBE and survey level assessments Identify current instructional plan and direction of future instruction Identify opportunities for generalization Assessments in Literacy Binders Benchmark Assessment Binders AIMSweb Current Instructional Plan Literacy Binders / Scope and Sequence Write new goals Identify current level of performance across areas Identify where to go next Work with team to write goals Current progress monito ring data Literacy Binders / Scope and Sequence Current Instructional Plan Assessments in Literacy Binders Benchmark Assessment Binders AIMSweb Update Literacy Tracking Forms Identify current skill developmental level Current literacy program Curre nt progress monitoring data Literacy Binder / Scope and Sequence Current progress monitoring data Info on current Instructional Plan

PAGE 408

392 About the Author in P sycholog y from Illinois Wesleyan University where she graduated with honors and earned the distinction of Outstanding Senior. In the fall of 1998, Mrs. De La Cruz was awarded a graduate f ellowship to attend the doctoral program in S chool P sychology at the Univers ity of South Florida in Tampa. While in Tampa, Mrs. De La Cruz focused her studies and research on advocacy and best practices for at risk populations. Mrs. De La Cruz c urrently works at the Northern Suburban Special Education District (NSSED) in Highland Park Illinois At NSSED, Mrs. De La Cruz coordinates services to member districts and is responsible for the design and management of the professional development opportunities Mrs. De La Cruz also consults with area school districts on effective i nstruction, data based decision making, and the problem solving process.