xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam Ka
controlfield tag 001 001670411
007 cr mnu|||uuuuu
008 051129s2005 flu sbm s000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0001287
The effect of direct instruction math curriculum on higher-order problem solving
h [electronic resource] /
by Pamela Christofori.
[Tampa, Fla.] :
b University of South Florida,
Thesis (M.A.)--University of South Florida, 2005.
Includes bibliographical references.
Text (Electronic thesis) in PDF format.
System requirements: World Wide Web browser and PDF reader.
Mode of access: World Wide Web.
Title from PDF of title page.
Document formatted into pages; contains 47 pages.
ABSTRACT: Previous research has examined the effectiveness of Direct Instruction Curriculum over the past thirty years in a variety of areas including rate of learning, effectiveness on different types of learners, and comparisons to other types of instruction. This study attempted to determine the effects of the use of a direct instruction math curriculum on higher-order problem solving. Two groups of 3 5 students each participated. The procedures included administering the Kauffman Achievement test to determine current grade level in math and reading. The Saxon Math Second Grade Curriculum was used to instruct the participants. The effects on higher-order problem solving with the Corrective Math Curriculum were assessed on two different dependent measures: solution of word problems consisting of both addition and subtraction operations, and performance of the students within the curriculum. Results were assessed using the delayed multiple baseline design.
Adviser: Dr. Jennifer L. Austin.
x Applied Behavior Analysis
t USF Electronic Theses and Dissertations.
The Effect of Direct Instruction Math Curriculum on Hig her-Order Problem Solving by Pamela Christofori A thesis submitted in partial fulfillment of the requirements for the degree of Master of Arts in Applied Behavior Analysis College of Graduate School University of South Florida Major Professor: Jennifer L. Austin, Ph.D. Patricia Barbetta, Ph.D. Kelli McCormick Brown, Ph.D. Date of Approval: July 15, 2005 Keywords: scripted lessons, computation, generalization, mast ery Copyright 2005, Pamela Christofori
Acknowledgments I would like to thank Dr. Jennifer Austin and Dr. Patri cia Barbetta for their expertise and guidance. They each traveled great distances and spent their valuable time mentoring me through this process.
Table of Contents List of Tables ii List of Figures iii Abstract iv Chapter One 1 Introduction Chapter Two Participants and Setting 13 Institutional Review Board Procedures 14 Dependant Variables and Data Collection 15 Interobserver Agreement 16 Procedures 16 Baseline 16 Direct Instruction Lessons 17 Procedural Integrity 19 Chapter Three Results 21 Chapter Four Discussion 27 References 34 Appendices 36 Appendix A: Parental Informed Consent 37 Appendix B: Reinforcer Survey 41 Appendix C: Procedural Integrity Data Sheet 42 i
List of Tables Table 1 Kaufman Test Of Achievement results 14 for participants ii
List of Figures Figure 1 Number of word problems correct across 23 baseline and treatment. Figure 2 Scores by group on mastery tests 25 expressed as percent correct. iii
The Effect of Direct Instruction Math Curriculum on Hig her-Order Problem Solving Pamela Christofori ABSTRACT Previous research has examined the effectiveness of Direct Instruction Curriculum over the past thirty years in a variety of area s including rate of learning, effectiveness on different types of learners, an d comparisons to other types of instruction. This study attempted to determine th e effects of the use of a direct instruction math curriculum on higher-order probl em solving. Two groups of 3 Â– 5 students each participated. The procedures i ncluded administering the Kauffman Achievement test to determine current grade le vel in math and reading. The Saxon Math Second Grade Curriculum was use d to instruct the participants. The effects on higher-order problem so lving with the Corrective Math Curriculum were assessed on two different dependent measures: solution of word problems consisting of both addition and subtract ion operations, and performance of the students within the curriculum. Re sults were assessed using the delayed multiple baseline design. iv
1 Chapter One Introduction This is a time of crisis in American education. Critics of current educational practices and outcomes are abundant, and thei r concerns do not appear to be unfounded. For example, the 1996 Mathematics Report Card reports that 75% of the nationÂ’s 8 th graders do not take algebra by the end of 8 th grade, and only 21 percent score at or above the profic ient level (National Center for Education Statistics, 2001, p.1). Additionally, American 8 th graders scored below the international average among 41 countries in the Third International Mathematics and Science Study (TIMSS,1999). With regard to reading, the 1994 National Assessment of Educational Progress ( NAEP) Reading Report Card found that 41% of 4 th graders could not read at the basic level and only 28% performed at or above the proficient level (NCES,2003) AmericaÂ’s apparent failure to produce quality educationa l outcomes for all children has led to both state and national initiative s to reform educational practices. In 1994, the United States Congress passed the Goals 2000: Educate America Act This legislation was passed to improve learning and t eaching by providing a national framework for educational reform to promote research, and support systemic changes needed to provide equitable educat ional opportunities. Within the legislation were several lofty educational goals that were to be met by the year 2000. These included a standard that all ch ildren would start school
2 ready to learn, that the high school graduation rate would increase to at least 90 percent, and that the United States would be the fir st in the world in mathematics and science achievement. The National Assessment of Educational Progress (NAEP) rel eased a progress report on those goals in 2000, and the results were less than impressive. One of the indicators used to measure readin ess to learn was the percentage of parents that regularly read to their 3 Â– 5 year olds. Results showed only a 3% increase on this variable. Moreover, there h ave been virtually no increases in high school graduation rates over the last thi rteen years, and the United States scored lower than 49% of the nations that participated in the 1999 International Mathematics and Science Study. The most recent legislation for school reform is the No Child Left Behind Act of 2001 (NCLB). This law expands the federal govern mentÂ’s role in KÂ–12 education by making federal aid conditional on those scho ols meeting academic standards and abiding by policies set by the federal go vernment. The four basic points in NCLB are: 1) Accountability for results through statewide progress goals and annual testing. 2) Emphasis on doing what works based on scientific research. 3) Expanded parental options by allowing par ents the opportunity to move their child to a better performing school in the l ocal district. 4) Expanded local control and flexibility in the use of funds to devo te more attention to student needs.
4 In a report published in January of 2004, the Center o n Education Policy found that 26% of the nationÂ’s public schools had failed to make adequate yearly progress (AYP). Schools are considered to have failed i n making AYP if they do not raise achievement scores in every subgroup of students in every grade for two or more consecutive years, or if they fail to improve graduation rates or ensure that 95% of students in each subgroup take the required tests. These data suggest that NCLB has not fulfilled its promise to improve educational outcomes for AmericaÂ’s children. Clearly, the stakes of educational reform are high. Th e effects of education and schooling on the development of an indiv idualÂ’s abilities have implications that reach across the life span. A report from the Carnegie Forum on Education and the Economy (1986) describes some of the potential repercussions of school failure: If our standard of living is to be maintained, if the growth of a permanent underclass is to be averted, if democracy is to function ef fectively into the next century, our schools must graduate the vast majority o f their students with achievement levels long thought possible for only th e privileged few. The American mass education system, designed in the early pa rt of the century for a mass-production economy, will not succeed unless it not only raises, but redefines the essential standards of excellence and strives to make quality and equality compatible with each other. (p.3) Despite the gravity of the situation, educational refo rm efforts have stemmed largely from public opinion, theory, and Â“common senseÂ”, as oppose d to sound empirical evidence of effective courses of action. An approach to educational reform that is very popular today and began in the 1980Â’s is whole school reform, which involves high profile education reformers and
5 organizations developing comprehensive models of curriculu m and instruction that encompass the entire school system. Traub (1999) examined ten of these school wide models with regard to such dimensions as student achievemen t, staff development and support, graduation rates, and attendance. The programs reviewed were chosen because they were either in fairly wide use, or they repr esented a significant body of thought in education. Traub noted that each model is based on a theory, and all of these theories cannot be equally true. The differences in the models ar e broad and deep, although there seems to be at least one shared assumption across most of the models; problems in education lie in classroom practice. Ther efore, reform must focus on changing not only what is taught, but how it is taught (Traub, 1999). Of the programs reviewed, the one that best illustrated this principal w as Direct Instruction (DI). Interestingly, DI was also one of only two programs that showed strong empirical evidence of effectiveness (Traub, 1999). Direct Instruction is an outgrowth of the work Siegfried Englemann and Carl BereiterÂ’s work with disadvantaged children (Bereit er & Englemann, 1966). Their work was based on the assumption that disadvantaged children can Â“catch upÂ” with their more academically competent peers if they are provided with effective and efficient instruction. According to Bereite r and Engelmann, the only way to close the gap between these two groups of children is by teaching at a faster than average rate. To accomplish this, DI curricul a focus on the goal of teaching more in less time. This involves using teaching procedures that
6 maximize the time students spend in instruction and arran ging materials to teach a Â“general case.Â” A general case strategy is one that us es the smallest possible number of examples to produce the largest possible amoun t of learning. One example of a general case in DI is through teaching 40 sounds and blending skills it gives the student a generalized decoding skill th at is relevant to over onehalf of the most common words in English (Gersten & Magg s, 1983). DI teaching procedures are distinguished from more tradi tional strategies by their focus on structure and explicit instructions for teachers The most noticeable departure from traditional instruction is t he use of scripted presentations that tell the teacher what to say and do f or each task. The examples and sequences used within the scripts have been pr e-tested and empirically established as effective. Without explicit di rections, teachers may use language the student doesnÂ’t understand or that distracts attention from the example (Binder & Watkins, 1990). Scripts also provide teachers with information about how to handle student errors. With in a DI context, errors are viewed as a means to help the teacher understand the a reas that are problematic for students. Different types of errors and the proper way to correct them is specified in the DI Curriculum. DI lessons are generally taught in small groups of 5 Â– 10 students, which provides for more adult direction and feedback. The use of frequent unison responding generates higher rates of student responding than most traditional teaching methods, which rely heavily on hand-raising as a means for generating
7 student participation in the lesson (Heward, 1994). I ncreased response opportunities also help decrease inattention during a lesson when one student at a time answers (Binder & Watkins, 1990). In a typical D I lesson, the teacher uses signals to cue students when to respond. These signals are used as both a prompt and an evaluation tool. By having the students r espond in unison, the teacher can determine whether or not each student is ma stering the particular skill they are instructing. Another key feature of DI is rapid pacing of instruction In addition to allowing more information to be covered within a lesso n, brisk pacing also helps to maintain student attention to the task, which may increase learnin g and decrease behavior management problems (Binder & Watkin s, 1990). Gradually, DI instruction moves from teacher-guided to m ore studentguided. This process, called mediated scaffolding (Kameenu i & Carnine, 1998), involves teaching students problem-solving strategies, fading assistance, and introducing more complex contexts to help students distin guish essential from nonessential details (Becker & Carnine, 1981). The goal of the process is to foster independence and higher-order thinking (Kozloff & Bessellieu, 2000). As mentioned previously, DI curricula have been tested ri gorously in empirical studies and in field trials (Kozloff & Besselli eu, 2000). This characteristic clearly differentiates DI from most instructi onal approaches, and also makes it unequivocally consistent with the mandates se t forth by NCLB. The largest study conducted to show the superiority of DI to other teaching
8 methods was entitled Project Follow Through (Adams & En gelmann, 1996). This national study compared the performance of children in over 20 different instructional models that represented the range of curr ent educational practice at the time. The results indicated that the Direct Instruct ion model was clearly the most effective of all the programs on measures of basic ski lls achievement, cognitive skills, and self-concept. Despite clear dat a confirming DIÂ’s effectiveness, the release of the studyÂ’s results generated a great deal of controversy with educational circles, presumably because the p rinciples of DI failed to fit with the predominant views of educatio nal theory and practice. EducationÂ’s reliance on theory as opposed to data may he lp explain the lack of implementation of DI programs after the projectÂ’s resul ts were made available to the public (Cooms, 1998). The research base for DI is not founded solely on the re sults of Project Follow Through. Research continues to be conducted to val idate the positive learning outcomes associated with DI teaching. Adams and Engelmann (1996) conducted a meta-analysis of DI programs that included Corrective Mathematics, DISTAR Arithmetic I and II and Connecting Math Concepts The studies included were required to have means and standard devi ation groups, the use of suitable comparison groups, and random selection of parti cipants to groups. In a sample polling of means conducted by the authors, 87% of the studies favored DI programs. A summary of the statistical analysis of math resu lts showed an effect size of 1.11 in favor of DI math programs in 33 of the 37 studies included.
9 Przychodzin, Marchand-Martella, Martella, and Azim (2004 ) conducted a review of DI mathematics studies that clearly demonstrate the superiority of DI methods in teaching math skills, especially with children w ho have history of failure with regard to arithmetic. For example, Parsons, Marchand-Martella, Waldron-Soler, Martella, and Lignugaris/Kraft (2004) studied the use of Corrective Mathematics delivered by peer tutors in a secondary general education class for students struggling in math. Ten s tudents were assigned to the learner group based on referrals by a school counsel or. All of those students had failed the lowest level math available at the schoo l. Nine other students were recruited by the Corrective Mathematics teacher to serve as peer tutors. All students were preand post tested using the Calculation and Applied Problems subtest of the Woodcock Johnson-Revised: Test of Achievement (WJ-R). After 60 instructional days, the authors found that both learn ers and peer tutors experienced posttest gains in one or both areas of the W J-R. Another study, conducted by Snider and Crawford (1996) e xamined 46 fourth graders who were randomly assigned to two gener al education classrooms. One teacher used Connecting Math Concepts (CMC), Level D, a DI program, whereas the other teacher used Invitation to Mathematics (SF) by Scott Foresman CMC students scored higher that the SF students on the Computation subtest of the National Achievement Test. Additionally, CMC students scored significantly higher on both the multipl ication facts test and on curriculum-based measures based on CMC and SF
10 Finally, Tarver and Jung (1995) compared CMC to a program that combined Math Their Way (MTW) and Cognitively Guided Instruction (CGI). One hundred nineteen students entering first grade were assi gned to five classrooms. One experimental classroom used CMC while four control classrooms used MTW / CGI. Data were collected on student learning gains d uring both first and second grade. At the end of second grade, CMC students scored higher than the control group on all post measures of the Comprehensive T est of Basic Skills Â– Mathematics as well as on the experimenter-constructed ma th attitudes survey. Tarver and Jung noted positive effects for both low a nd high performing students. Although there has been a great deal of current rese arch conducted to validate the educational benefits of DI, criticisms of the program are still common in the educational community. One such criticism centers on the notion that scripted presentations and predetermined lessons stifle th e teacherÂ’s creativity. Adams and Engelmann (1996) challenged this criticism by st ating that the most important measure of teacher creativity is how well the teacher succeeds at teaching and accelerating student performance and teachin g students things they typically have trouble learning. The creative poten tial of students is limited by their current knowledge. The first job of the teacher then, is to teach basic skills and knowledge. If the teacher is not achieving attai nable instructional goals, the student cannot benefit from any attempt at creativi ty by the teacher (Adams and Engelmann).
11 Another common criticism is that direct Instruction igno res individual differences among students, presumably because the program approaches teaching all students in the same manner. However, the measure of whether a program recognizes individual differences is simply to eva luate if the program accommodates students of varying abilities and styles. Â“If st udents learn the content on the projected time schedule, their performan ce is a clear declaration that the program accommodates the full range of individual differencesÂ” ( Adams and Engelmann,1996, p.37 ). Still another criticism is that direct Instruction progra ms are appropriate for low performers only. If this statement were true, low performers would perform in a generically different manner than high performers ( Adams and Engelmann, 1996). An example of this might be low performers le arning from manipulation, while high performers did not. However, in working wit h students of different abilities the only differences that occurred were that hi gh performers require less repetition, less review, fewer examples, and often less reinforcement than lower performers (Adams and Engelmann). The greatest challeng e to this myth is that research has shown that DI programs have accelerated lower performers beyond higher performers in regular education classrooms (Robinso n and Hesse, 1981; Tarver and Jung, 1995; Vitale and Romance, 1992).One of the most firmly held beliefs by many educators is that DI is only appropriate for teaching basic skills and impedes the development of higher order problem-so lving skills. Adams & Engelmann (1996) discuss this issue by pointing out that DI programs attempt to
12 introduce models that permit generalizable learning o f core skills. DI teaching units are successively more complicated and less structured, so students are learning how to learn as they master the content. Furt her, Brody and Good (1992) suggest that the structured learning presented i n DI may make independent problem solving an easier pursuit for stude nts because they have a better understanding of how to organize rules, facts, a nd operations. The present study is designed to examine the criticism th at skills taught within a DI curriculum preclude the development of high er order problem solving in the absence of direct teaching of those skills. Specifica lly, the study sought to determine whether students taught basic addition and su btraction skills using DI are able to generalize those skills to solve more advanc ed mathematics problems requiring the same skill set.
13 Chapter Two Method Participants and Setting Two groups of 2 nd graders from five regular education classes participated in the study. Group 1 included four girls (Josie, Mona Marci, and Mary), and one boy (Mark), aged 8 to 9. Group 2 included three boys (Gean, Joe, and Ed) and two girls (Edie and Karlie) aged 8 to 9. Classrooms selected from which to draw students were those i n which the teachers expressed an interest in participation after be ing given a brief explanation of the study. The students selected were i dentified by their teachers as low to average performers in the 2 nd grade math curriculum as assessed by a research assistant with the Kauffman Test of Educational Achievement (K-TEA). Those students who scored in the low to average range of recommended accuracy levels for addition and subtraction were selected fo r participation in the study. Because it was necessary for students to read word pr oblems as part of the studyÂ’s procedures, the participantsÂ’ reading levels we re also assessed using the K-TEA. Only students reading at the end of the 1 st grade proficiency levels were selected to participate. K-TEA scores for each par ticipant are shown in Table 1. All experimental sessions were conducted in a resource classroom at White City Elementary School in Ft. Pierce, Florida.
14 Table 1 Kaufman Test of Achievement Results for Participants Group Student Math Grade Level Reading Grade 1 Marci 1.7 4.4 Josie 1.5 3.7 Mona 1.9 3.9 DeDe 1.0 3.5 Mary 1.9 3.1 2 Gean 2.2 3.1 Joe 2.5 3.1 Ed 2.8 5.2 Mark 2.4 5.7 Karlie 2.2 3.3 Institutional Review Board Procedures The Institutional Review Board at the University of S outh Florida and the St. Lucie County School Board approved all procedures pr ior to data collection. The primary investigator met with the teachers of the stu dents chosen for the study to review the informed consent letter and to answer any questions. Students selected as participants were given assent forms a nd their parents were given informed consent forms prior to data collectio n (Appendix A). A letter outlining the study accompanied both the assent and con sent forms (Appendix
15 B). Two phone calls were made to the parents of each p articipant. The first call was to explain the study and their childÂ’s participation in the study. The second call occurred several days later to ask if there were any que stion or concerns prior to them making a decision about consent. All fo rms were sent home and collected by each teacher, and subsequently were given to the researcher. Child assent and parent consent were obtained for all the chil dren who participated in the study prior to the start of data collection. Dependent Variables and Data Collection The primary dependent variable in this study was the solut ion of word problems consisting of both addition and subtraction oper ations involving the concepts of money, temperature, and measurement. Each stu dentÂ‘s performance was measured throughout the study with multi ple probes of the word problems. The probes consisted of short tests containing 10 word problems randomly selected from two web-based banks of wor d problems (www.EdHelper.com and www.MathStories.com), which were created for teachers from which to draw curriculum. All word problem s on these sites were leveled by grade. Only word problems developed for 2 nd grade were used in the study. Those problems selected for inclusion in the study assessed the basic arithmetic skills taught as part of the DI (Saxon Math) curriculum, but did not include problems or scenarios directly taught or described to students during the lessons.
16 Tests were scored by trained research assistants using the an swer key provided by the web site. Each test item was scored accordi ng to the answer key, and subsequently calculated as a percentage (total numb er of problems completed correctly divided by total number of problems). The second dependent measure was taken from the mastery te sts included within the Saxon Math curriculum. There wer e two in-program mastery tests in each unit. The mastery test began with lesson 25 and appeared approximately every 5 lessons. Mastery tests assessed the master y of the concepts taught in the previous unit. Mastery tests were scored by the primary researcher and another trained research assistant. Each studentÂ’s final answers to the problems were scored as correct or incorrect and scor es were presented as percent of problems completed correctly. Interobserver Agreement Both mastery tests, spaced across the course of the study, w ere scored for interobserver agreement (IOA) with the research a ssistant. The IOA score for probes was 100%. The mean IOA score for mastery tests w as 95% (range, 90% to 100%). The IOA calculation used was for the perce ntage of agreement for permanent products (i.e., the number of agreement s divided by the total of agreements and disagreements multiplied by 100%). Procedures Baseline. Prior to beginning the DI lessons, each participant was pretested on a series of probes that consisted of tests con taining 10 word
17 problems randomly selected from the bank of word problem s. To obtain stable baseline responding, seven pretests were administered to six of the participants, eight pre-tests to two of the participants, and nine pretests were administered to three of the participants. Additionally, all particip ants took a standard DI placement test to determine the appropriate starting p oint within the curriculum for use during the intervention phase. All placement scores for students on Group 1 indicated that they should start in the same uni t. Placement test scores for three participants in Group 2 indicated they should start in the same unit. The other two participants placed at the end of the previ ous unit. Those two students were given the last lesson in the previous unit to compl ete independently. Both scored 100% so all participants in second group started o n the same lesson. Direct Instruction Lessons The DI curriculum used for the study was Saxon Math. This series focuses on teaching strategies for learning and retaining facts, understanding place value, solving comput ational problems, discriminating among various types of story problems, and a ccurately translating story problems into numerical statements. The lessons used in this study focused on basic math skills, learning and retaining facts, understanding place value, solving computational problems, and defining mat h vocabulary. Lessons were delivered by a trained research assistant who was enrolled in the special education teacher preparation program a t a local university. A daily lesson with the group of participants occurred Mo nday through Friday, with each lesson lasting 35 55 minutes. This session length w as slightly longer than
18 the time recommended in the Saxon Math TeacherÂ’s Manua l. The session time decreased to the recommended time of 25 Â– 45 minutes as the teacher became more fluent with the format of the curriculum. Each lesson in Saxon Math is divided into tasks and includes four components: the Meet ing, the Lesson, Class Practice, and Written Practice. A daily lesson is structur ed as follows: 1. The Meeting and the Lesson : These were teacher-directed activities. Teacher presented exercises through use of the script, list ened to student responses, and corrected errors immediately. 2. Class Practice: The Student Workbook contained sample skills that had just been taught in the program and that were cri tical prerequisites for learning the upcoming skills. StudentÂ’s completed these during the lesson. 3. Written Practice : In most lessons, students did a series of exercises on their own. Those exercises reviewed students on previou sly taught skills. A total of 14 lessons were completed during the inter vention phase of the study. Each participant had to score a minimum of 90% on the written assessment in order to move ahead to the next module. If more than four students scored under the minimum, the instructor conducted extra sessions outside of the daily meeting to bring those participa nts score to the minimum 90%. When five or more of the participants scored belo w 90% the instructor conducted extra sessions with the entire group of partici pants to bring their scores up to the minimum 90%. This occurred only once duri ng the study with Group 2. The teacher re-taught the entire lesson and brought those students
19 above the 90% required competency. Records were maintain ed for all participants with regard to test scores for all attempts, as well as the number of tests and sessions required to meet the mastery requiremen t. A daily meeting occurred with the primary researcher and the research assist ant after each lesson to score the daily written practices and determine the lesson to be taught the next day. Participants met in a student resource room at the schoo l site each day at the same time. The session was conducted without a break. At the end of the session, participants were rewarded with their choice of an edible (Appendix C) if they had participated in the lesson by answering individ ual questions, choral responding, and completing the written practice. Procedural Integrity Each lesson within the Saxon Math curriculum is scripted a nd sequenced in the same order. To determine if the lessons were be ing delivered as prescribed, the researcher developed a checklist (Appendix D) with all the tasks in each lesson in the correct sequence. Two additional t rained research assistants conducted the observations and completed the proce dural integrity checklist for 30% of the lessons. A procedural integrity sco re for each lesson was derived by dividing the number of steps completed co rrectly by the total number of steps required to implement the lesson. The mean integrity score was 98.75 % (range, 90% to100%). One hundred percent of the procedural integrity observations were scored for IOA using the same calculatio n used for dependent
20 measures. The mean score for the procedural integrity be tween the two observers was 98% (range, 90% to 100%).
21 Chapter Three Results A multiple probe design across participants was used to analyze the effectiveness of using a DI math curriculum on studentsÂ’ ab ilities to solve higherorder word problems. Figure 1 shows the number of corr ect word problems for each student across baseline and treatment conditions. The total possible score for each probe session was 10. Overall, results show the direct instruction curriculum was effective in increasing the mathematical problem sol ving skills of all children involved in the study. With the exception of one data p oint for Karlie, all treatment probe scores during treatment were above baseline levels f or every participant. Josie obtained a mean baseline score of 1.6 (range, 0 to 4). During treatment, scores were high (mean = 9.3) and more stable (range 8 to 10). Mark obtained a mean baseline score of 2.9 (range, 1 to 5). During tr eatment, scores improved substantially (mean = 9.8) and variability decreased (rang e, 9 to 10). For Mona, the baseline mean was 1.9 (range, 0 to 4). Treatment yiel ded a mean score of 8.4 and reduced variability (range, 7 to 9). Marci obtained a mean baseline score of .71 (range 0 to 3). Substantial increases were observed duri ng treatment (mean = 8), though there was increased variability (range, 5 to 10) and a downward trend across sessions. For Mary, the mean baseline score was .75 (r ange, 0 to 3). The mean score during treatment was 6.2, though data were variable (range, 4 to 8). Joe obtained a mean baseline score of 4.4 (range, 2 to 8). During treatment, scores improved substantially (mean = 9.6) and remained sta ble across sessions (range, 9 to 10). For DeDe, the mean baseline score was .63 (range, 0 to 3).
22 During treatment, the mean score increased to 8.3 (rang e, 7 to 10), although a downward trend was observed across sessions. Edie obtained a mean baseline score of 3.4, though a great deal of variability was obse rved across baseline sessions (range, 0 to 7). During treatment, scores increa sed to a mean of 9.4 and variability decreased (range, 8 to 10). For Karlie, the mean score across baseline was 2. Though initially variable, baseline data stabil ized across the later sessions (range, 1 to 5). During treatment, the mean increase d to 7.7 across an upward trend (range, 5 to 10). Gean obtained a mean baselin e score of 2.1 and demonstrated a good deal of variability across sessions (ran ge, 0 to 6). During treatment, the mean score increased to 9.3 and remaine d stable across sessions (range, 9 to 10).
23 0 2 4 6 8 10 1234567891011121314 Mona 0 2 4 6 8 10 1234567891011121314 Mark 0 2 4 6 8 10 1234567891011121314 Mary 0 2 4 6 8 10 1234567891011121314 Joe 0 2 4 6 8 10 1234567891011121314 DeDe 0 2 4 6 8 10 1234567891011121314 Edie 0 2 4 6 8 10 1234567891011121314 Karlie 0 2 4 6 8 10 1234567891011121314 Gean 0 2 4 6 8 10 1234567891011121314 Josie Baseline Treatment 0 2 4 6 8 10 1234567891011121314 Marci # Of Word Problems Correct Sessions Figure 1. Number of word problems correct across baseline and treatment.
24 Figure 2 shows the participantsÂ’ scores on the Saxon Mat h mastery tests. Performance across the tests was variable within Group One (i.e., test one, M = 73%, range, 67% 100%; test two, M = 76%, range, 50 % 100%; test three, M = 91.5%, range, 83% 100%). With regard to specific e rrors, four of the five participants appeared to have difficulty identifying even numbers in the first test. In the second mastery test, several children had problems wr iting the number sentences. In the third mastery test, two participants (Ma rci and Mary) missed several of the addition facts. However, all participan ts except Marci improved their scores from the first test to the third test (DeDe, who was absent for the third test, showed improvements from the first to second t est). Group Two did not complete the final unit prior to sch ool ending, so the third mastery test was not administered to this group. Within and across the two mastery tests given, a good deal of variability was observed (i.e., test one, M = 75%, range, 67% 83%; test two, M = 87%, range, 67% 100%). Two of the students (Melanie and Gean) improved their scores from t he first to second test. However, JoeÂ’s performance remained stable and Joe did sl ightly worse on the second test. Only one test was administered to Karlie, on which she scored 100%. With regard to specific errors, several students i n Group Two also had problems identifying even numbers, although overall the y scored higher than the children in Group One.
25 0 20 40 60 80 100Josie Mona Marci Mary DeDe Melanie Joe Ed Karlie GeanGroup 1 Group 2Percent Correct Mastery 1 Mastery 2 Mastery 3 Figure 2. Percentage correct scores on the mastery te sts for each participant by group.
26 Chapter Four Discussion The present study was designed to examine whether studen ts taught basic math skills using a DI curriculum would able to gene ralize learned skills to solve more advanced mathematics problems requiring the sam e skill set. The results of the present study suggest that the use of the Sa xon Math DI curriculum led to generalization of skills to higher-order proble m solving, without any specific instruction to the students on the more advanced problems. Word problems were used as the primary measure of the participantsÂ’ abi lities to use the skills in a novel way. The number of word problems correct increa sed from baseline to treatment for every child who participated in the stud y, although some children showed more dramatic changes than others. One of the most common criticisms of DI is that it impede s the development of higher-order problem solving skills throu gh the use of too much teacher-directed drill and practice (Adams & Engelmann,1 996). The results of this study, however, do not support these claims. Instead they indicate that the mastery of basic skills did lead to increased ability to sol ve more complicated problems (i.e., word problems) for which the students h ad no prior training or experience. Six of the participants (Josie, Mark, Joe, Gean, Mona, and Ediedemonstrated immediate improvement in problem solvi ng skills and maintained the gains across time. The other four partic ipants (Mary, Karlie DeDe, and Marci) also showed improvements over baseline, altho ugh their data
27 revealed either slower rates of acquisition or more var iable levels of improvement. Closer examination of the data revealed that some of the differences in performance could be attributed to specific skill defici ts. For example, Marci showed initial improvements in her performance that even tually diminished over time. Inspection of MarciÂ’s work showed she had difficulty writing number sentences, which is important to the solution of a word problem. The research assistant also reported problems with compliance and atten ding to instruction, which could have negatively affected her performance, esp ecially as lessons became more complex. Mary showed the least improvement o f all the children in the study. Inspection of her work indicated she had diffi culty writing numbers and required more repetitions to master a skill. Mary was also absent for 4 lessons, which probably affected her rate of acquisition due to limited exposure to material and fewer opportunities to practice. DeDe showed impro vement from baseline to treatment, but had a decreasing trend in her treatme nt data. DeDe became frustrated easily and would refuse to repeat a task when she made an error. These behaviors probably adversely affected her scores, espec ially as lessons progressed and tasks became more difficult. Despite some performance deficits for several of the chil dren, it is important to reiterate that all childrenÂ’s scores improve d during the DI lessons and that almost all treatment data points fell above t he baseline range. These findings suggest that the DI curriculum was more effectiv e than the childrenÂ’s
28 regular mathematics curriculum in promoting the applicat ion of math skills to novel problems. The students that participated in the study were from five different 2 nd grade classrooms, where they received regular math instr uction from a variety of teachers. None of the participants improved their performance on the word problem probes prior to the introduction of DI i nstruction. Therefore, one can not reasonably argue that changes in the classroom envir onment accounted for improvements in the childrenÂ’s math performance. Fu rther, regular math instruction was suspended once students began DI lessons, whi ch increases the robustness of treatment effects. Another important finding of the study is more closely re lated to the independent variable than the dependent variables. N amely, this study showed that the DI teacher could learn how to use the curriculu m quickly and obtain good results, despite being inexperienced both with DI and teaching in general. Although the teacher was a student in a university teacher -preparation program, she had relatively little experience as the primary instru ctor for a group of children. This finding may be particularly relevant for principals and teachers. The large number of instructional requirements, coupled with teacher shortages and a large percentage of teachers teaching Â“out of fie ld,Â” make an effective, easy-to-master curriculum an incredibly valuable tool. Anot her benefit to school districts might be that paraprofessionals, tutors, and vo lunteers could be easily trained to use the curriculum effectively and increase the number of instructional staff available to students. It is also worth noting th at the teacher reported liking
29 the DI curriculum and found it user-friendly. During the daily meetings with the teacher after she had taught the lesson for the day, sh e stated that the lessons were easy to follow and she enjoyed using the curriculum. This study also showed that the DI curriculum could be eff ective even when threats to treatment integrity were present. Alt hough overall treatment integrity scores were high, the teacher did experience som e problems with implementing the curriculum. The research assistants who conducted procedural integrity checks noted that the instructor did not consiste ntly using the error correction procedures in the early lessons. The problem w as corrected by conducting practice sessions with the primary researcher and the teacher, but it is important to note that students still made impressive gains even when the error correction procedure was used sporadically. Another probl em was that the teacher did not consistently require mastery before goin g to the next lesson. When questioned, the teacher stated that the participa nts objected many times when she asked them to repeat a lesson or a specific task. Du e to her limited experience working with students, she was not sure how to gain compliance in this type of situation. The primary investigator discussed several methods to reinforce compliance during instruction. The teacher init ially reported success with the procedures, but later reported the behavior r eturned and occurred sporadically throughout the instruction. Despite encouraging results, the current study is not with out some limitations. One concern that might be raised is wheth er the primary dependent
30 variable (word problem probes) was a valid measure of hi gher-order processing. One of the most widely accepted definitions of higher-or der problem solving is in BloomÂ’s Taxonomy (Bloom, 1984). The second highest orde r of categorization in the taxonomy is synthesis, which is partially defined as gen eralizing from given facts. Mathematical facts are given in word problems that must be interpreted and generalized to solve for the answer. Therefore, on e could argue convincingly that the dependent measure used in this study was, in fact, an example of a higher-order skill. However, future rese arch is needed to more clearly identify and define what constitutes higher-order processing. In the current study, measures of face validity by math experts and teachers regarding whether the word problems used in the study were a type of higher-order task would have been beneficial. Despite this oversight, th e results of the current study show, at a minimum, that the use of DI curriculum r esulted in generalization to a novel type of math task. Future rese archers should explore the extent of this generalization by testing other type s of mathematics tasks concurrent with DI instruction of basic skills. The current study had participants placed in two groups o f 5 students each. It could be argued that the results were due to the amount of attention the teacher was able to give to students in a small group s etting. Additional research is needed to determine if DI curriculum would be as eff ective during whole group instruction with a large class of students.
31 Another notable variable that may have affected the re sults involved the timing of the study. Data collection occurred during the last month of school and the final word problem probe was administered the la st full day of school. The participants were involved with many end-of-the-year a ctivities and this may have competed with the motivation of some of the students to attend to math instruction (i.e., those that showed downward performan ce trends or relatively lower scores for the last 1-2 lessons). Although all stud ents showed improvements, one wonders if performance increases could ha ve been greater for some of the students had the DI lessons been conducte d earlier in the school year. It is clear that future research on the effects of DI Math curriculum on higher-order problem solving is needed. Currently, th e educational communityÂ’s belief that scripted curriculum stifles teachersÂ’ abiliti es to teach at the Â“concept levelÂ”, and subsequently stifles studentsÂ’ abilities to rea ch that level, has adversely affected the dissemination and widespread use of one of the most effective curriculums developed to date (Adams & Engelmann 1996). Research can begin to change the perceptions of educators by continu ing to investigate a variety of skills that are commonly thought of as higher-o rder tasks and evaluating DIÂ’s effectiveness on teaching those tasks. Educat ors are practicing in a time where accountability is high. Many teachers are sea rching for strategies that deliver faster, better results. Continued research a imed at demonstrating the effectiveness of DI to teach and promote the generalizati on of a range of
32 academic skills would benefit both teachers and the student s that depend upon them.
33 References Adams, G.L., & Engelmann, S. (1996). Research on Direct Instruction: 25 years beyond DISTAR. Seattle, WA: Educational Achievement Systems. Becker, W., & Carnine, D. W. (1981). Direct Instruction: A behavior theory model for comprehensive educational intervention with the disad vantaged In S.W. Bijou & R. Ruiz (Eds.), Behavior modi fication: Contributions to education (pp.145-210). Hillsdale, NJ: Lawr ence Erlbaum associates. Bereiter, C. & Engelmann, S. (1966). Teaching disadvantaged children in the preschool Engelwood, Cliffs, NJ: Prentice-Hall. Binder, C., & Watkins, C. L. (1990). Precision Teaching and Direct Instruction: Measurably superior instructional technology i n schools Performance Improvement Quarterly, 3(4), 74-96. Brophy, J., & Good, T. (1992). Looking in classrooms Boston: Addison-Wesley. Carnegie Forum on Education and the Economy. (1986). A Nation Prepared: Teachers for the 21 st Century. Report of the Task Force on Teaching as a Profession. New York: Carnegie. Center On Education Policy (2004). From The Capital To The Classroom: Year 2 Of The No Child Left Behind Act. Retrieved August 25 th 2004, from http://www.cep-dc.org/pubs/nclby2/ Ed Helper.com. TeacherÂ’s web-based support for grades K Â– 6 in Mathematics, Reading, Language Arts, Science. http://www.edhelper.com/ Heward, W. L., (1994) Three Â“low-techÂ” strategies for increasing the frequency of active student response during group instru ction. In R. Gardner III, D. M. Sainato, J.O. Cooper, T.E. Heron, W.L. He ward, J. Eschleman, & T. A. Grossi (Eds.), Behavior Analysis in education: Focus on measurably superior instruction (pp. 283-320). Monterey, CA: Brooks/Cole. H.R. Goals 2000: Educate America Act. (1994). Archived Information Retrieved August 23, 2003 from http://www.ed.gov/legislation/GOALS2000/ TheAct/sec102.html Kameenui, E. J., & Carnine, D. W. (1998). Effective teaching strategies that accommodate diverse learners Upper Saddle River, NJ: Merrill Kozloff, M. A., & Bessellieu, F. B. (2000, April). Direct Instruction Is
34 Developmentally Appropriate. Retrieved August 25 th 2003, from http://curry.edschool.virginia.edu/sped/projects/ose/paper s/MK_DI_DAP.pdf MathStories.com. House of Math Word Problems for Child ren. Teacher support for Mathematics for grades K Â– 6. Word probl ems leveled according to grade. http://www.mathstories.com/ National Center for Education Statistics. (2001, June 21 st ). Voluntary National Tests: 4 th Grade Reading 8 th Grade Mathematics Retrieved August 25 th 2003 Institute of Education Sciences, U.S. Dept. of Educati on Online Access: htpp://www.ed.gov/nationaltests/index.html Parsons, J. L., Marchand-Martella, N. E., Waldron-Soler K, Martella, R. C., & Lignugaris/Kraft, B. (2004). Effects of a High School-Based PeerDelivered Corrective Mathematics Program. Journal of Direct Instruction 4(1), Winter pp. 95-103 Przychodzin, A. M., Marchand-Martella, N. E., Martella, R. C., & Azim, D. (2004). Direct Instruction mathematics Programs: An Ove rview and Research Summary. Journal of Direct Instruction. 4(1), Winter pp.53-84 Snider, V. E., & Crawford, D. B. (1996) Action Resea rch: Implementing Connecting Math Concepts. Effective School Practices 15(2) Spring pp. 17-26 Tarver, S. & Jung, J. S. (1995) A Comparison of mathe matics Achievement and Mathematics Attitudes of First and Second Gr aders Instructed with either A Discovery-Learning Mathematics Curriculum or a Direct Instruction Curriculum. Effective School Practices 14(1) Winter pp. 49-57 Traub, J., (1999). Better By Design? A ConsumerÂ’s Guide to Schoolwide Reform Retrieved September 21 st 2003, from Thomas B. Fordham Foundation. http://edexcellence.net/library/bbd/better_by_design.htm l U.S Department of Education. Introduction: No Child Left Behind. Retrieved February 28 th 2004, from http://www.ed.gov/nclb/overview/intro/in dex.html
36 Appendix A Parental Informed Consent Social and Behavioral Sciences University of South Florida Information for Parents Who are being asked to allow their child to take pa rt in a research study The following information is being presented to help you decide whether or not you want to allow your child to be a part of a minimal risk research study. Please read this carefully. If you do not understand anything, ask the person in charge of the study. Title of research study: The Effect Of Direct Instruction Math Curriculum On Higher-Order Problem Solving Person conducting the study: Pamela Christofori Where the study will be done: White City Elementary School Your child is being asked to participate because there is a need to find effective and efficient classroom curriculum for students. Many of o ur students are performing below their potential because we are not using the most effective teaching strategies available in education. Your childÂ’s t eacher has identified your child as one who might benefit from participating in this study. General Information about the Research Study The purpose of this research study is to assess the effects o f Direct Instruction Curriculum on the skill of higher-order problem solving in math. Direct instruction is a scripted, sequential teaching method used to teach academic content. The procedures involve your child participating in a group w ith 3-5 other children. The group will be instructed using the Saxon Math Direct I nstruction Curriculum. Plan of Study Two groups of 3-5 students will come to a resource room at different times during the day, at the school, 5 days a week for 25Â– 45 minutes. During that time he/she will receive instruction in math using the Saxon Math Direct Instruction Curriculum. This is a research-based program that has bee n shown to be very effective in the teaching of math skills. Your child wil l be given a pretest of word
37 problems three times before the instruction begins and six times during the study. Your child will also be assessed every 10 lessons completed in the curriculum using a written te st that is part of the curriculum. Data will be collected on the performance of your child on each of these assessments. Your child will be observed by an independent research assistant for every 3 out of 10 lessons conducted. The observations are done to insu re that the instructor is conducting the lessons according to the curriculums directi ons throughout the study. Participation in the study will require your child to spend 25Â–45 minutes out of their classroom engaged in this math instruction. Your child will be given the choice of a drink, a snack, or a sticker at the end of each lesson. Please tell us of any allergies or restrictio ns you have for your child regarding food and drink. Payment for Participation Your child will not be paid to participate in this st udy. Benefits of Taking Part in this Research Study A potential benefit to having your child in the study might be increased performance in their grade level math and in their pr oblem solving abilities. An overall benefit of this study could be the increased u se of effective teaching methods so that students can reach their fullest potentia l. Risks of Being a Part of this Research Study : There are no known risks to your child for participation in this study, and you may w ithdraw at any time. Confidentiality of Your ChildÂ’s Records You and your childÂ’s privacy and research records will be ke pt confidential to the full extent required by law. Authorized research person nel, employees of the Department of Health and Human Services, and the USF I nstitutional Review Board may inspect the records from this research project. The results of this study may be published. However, the data obtained from your child will be combined with data from other child ren in the publication. The published results will not include your childÂ’s name or any other information that would personally identify your child in any way.
38 Volunteering to Take Part in this Research Study Your decision to allow your child to participate in t his research study is completely voluntary. You are free to allow your child t o participate in this research study or to withdraw him/her at any time. If yo u choose not to allow your child to participate or if you remove your child fro m the study, it will in no way affect your childÂ’s grade or their student status. Questions and Contacts If you have any questions about this research study, cont act: Pamela Christofori: 772-529-3029 Dr. Jennifer Austin: 813-494-4577 Ms. Angie Difruscio: 772-468-0480 If you have questions about your rights as a person who i s taking part in a research study, you may contact the Division of Research Co mpliance of the University of South Florida at (813) 974-5638. Consent for Child to Take Part in this Research Stu dy I freely give my consent to let my child take part in this study. I understand that this is research. I have received a copy of t his consent form. ________________________ ________________________ ______ Signature of Parent Printed Name of Parent Date of child taking part in study Investigator Statement I have carefully explained to the subject the nature of the above protocol. I hereby certify that to the best of my knowledge the subject signing this consent form understands the nature, demands, risks, and benefits involved in participating in this study. ________________________ ________________________ ______ Signature of Investigator Printed Name of Investigator Date or authorized research investigator designated by the Principal Investigator
39 ChildÂ’s Assent To Participate in Study Plan of the study Â– You will be using a different book to learn your ma th. ItÂ’s called Saxon Math. You will be in a group of 3-5 classmates and go to the resource room with an instructor to have math class. Class will be from Monday to Friday at the same time your current math is scheduled, about 25 Â– 45 minut es each day. About once or twice a week you will be given 10 word problems t o solve. At the end of every 10 lessons there is a mastery test to see what you lear ned. ChildÂ’s Assent Statement Pamela Christofori has explained to me this research stud y called The Effect Of Direct Instruction math Curriculum On Higher-Order Prob lem Solving. I agree to take part in this study. ________________________ ________________________ ______ Signature of Child Printed Name of Child Date taking part in study ________________________ ________________________ ______ Signature of person Printed Name of person Date obtaining consent obtaining consent
40 Appendix B REINFORCER SURVEY Students Name: ___________________________ Date: _________ Completed By:____________________________ At the end of each lesson, after you have completed al l your work, you will be able to choose one of these items each day. Please answer the following questions so we will have stuff to earn that you really like. What is you favorite thing to eat for a snack ? ________ ______________________ What is your favorite thing to drink ? _______________ ______ Put a check mark next to the items you would like to ea rn in math class: ____ Pokemon ___ Barbie ___ Sponge Bob ____ Dora The Explorer Stickers Stickers Stickers Stickers ____ Apple Juice ___ Grape Juice ___ Orange S oda ____ Yahoo Soda ___ Peanuts ___ Potatoes Chips ____ Frito s Chips ____ Cheese Crackers ___ Tootsie Roll ___ Snickers ____ Plain M&MÂ’s ____ ReeseÂ’s Pieces
41 Appendix C Procedural Integrity Data Sheet Observers Name :______________________ Date:__________ Participants: Check the group you are observing: Group 1___ Gr oup 2 ___ # of Lesson Observed : _______ Correct Sequence for presentation of the DI Lesson : A checkmark indicates correct implementation of ste p. Preparation for Daily Lesson: ______ 1. Teacher has prepared for the daily lesson by reading it through, identifying any new formats and consulted the Presentation Book. Observer will ask these six questions of teacher prior to s tudents arriving. What is this format teaching?____________________ __________________________________ How is it structured ? __________________________ __________________________________ Does the format specify that any steps are to be repeated ?______________________________ Where are individual turns specified ?___________ ____________________________________ What kinds of mistakes are the students likely to make ?________________________________ What correction procedures should be used ?______ ___________________________________ _____ 2. Instructional area is prepared before stu dents arrive: Student Books on table in front of as signed seat, ex tra sharpened pencils, scrap paper. _____ 3. Stand at door of classroom to receive stu dents. Greet with a smile and direct to assigned se at. Implementation of Daily Lesson: _____1. Format Â– Followed format of lesson closely. _____ 2. Signals Same signal throughout lesson. _____ 3. Signals Â– All students responding together when signal is given at right time. _____ 4. Watching Â– Pays close attention to students responses and r esponds accordingly. _____ 5. Watching Talking to students while standing in front of group. _____ 6. Watching Â– Walking among students when they are writing or t eacher is checking work. _____ 7. Corrections Corrects every error properly according to type of error and procedure required. ______ 8. Correction s Are delivered to student in a positive tone. ______ 9. Feedback Â– Students are reinforced as a group for participa tion and/or correct answers. _____10. Feedback Students are reinforced individually for partic ipation and/or correct answers. _____11. Pacing Moving through lesson as fast as possible with out forcing the students to make mistakes. Start Time:________ / End Time:__ _______