xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam Ka
controlfield tag 001 001787908
007 cr mnu|||uuuuu
008 070514s2005 flu sbm 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0001431
Zabel, Sally A.
Metacognitive strategies in a web-enhanced environment :
b the effects on achievement in problem-solving for engineering undergraduates
h [electronic resource] /
by Sally A. Zabel.
[Tampa, Fla] :
University of South Florida,
ABSTRACT: This study focused on the effects of using metacognitive strategy cuing integrated into problem-solving activities in a web-based learning environment. Purposes of the study were to investigate: (a) differences in posttest achievement between students who received metacognitive strategies embedded as cues in engineering problem sets and students who did not receive the treatment; (b) differences in perceptions of problem-solving skills between students who received metacognitive strategies embedded as cues in engineering problem set and students who did not receive the treatment; (c) differences in thermodynamics knowledge; (d) problem-solving steps students reported using across problem sets; (e) characteristics of sampled students, and (f) students' perceptions of web-based problem sets.The sample consisted of 81 students enrolled in an undergraduate thermodynamics course. In-class lectures were scheduled twice weekly, and web-based problem sets were assigned as homewo rk. Two groups, the treatment group using embedded metacognitive cuing and the control group not using the embedded metacognitive cuing, practiced with problem-solving activities over a fifteen-week-semester. Two-thirds through the semester, comprehensive posttest achievement scores were compared between groups. Analyses showed no significant differences between groups when metacognitive strategies were incorporated into web-based problem sets. An instrument was developed and validated to measure students' perceptions of their abilities to plan, monitor, and evaluate problems. Pre- and post testing of students' self-reported perceptions were measured. The results indicated no significant differences between groups. When differences in thermodynamics knowledge and skills between students were measured, pretest to posttest results showed equal improvement for both groups, contradicting the hypothesis those students in the treatment group would improve in skills and knowledge more tha n the control group. A frequency analysis revealed differences in the amount of times students' reported using engineering problem-solving steps while working through exercises. Most frequently chosen was Step Two List Variables (91 %) and Step Seven Solved Equations (91%).. The least chosen response was Step Four -- Made/stated Assumptions which was selected only three percent of the time. Implications from this investigation, along with previous research, facilitate definition of boundary conditions when employing metacognitive cuing in web-based learning.
Dissertation (Ph.D.)--University of South Florida, 2005.
Includes bibliographical references.
Text (Electronic dissertation) in PDF format.
System requirements: World Wide Web browser and PDF reader.
Mode of access: World Wide Web.
Title from PDF of title page.
Document formatted into pages; contains 141 pages.
Adviser: Ann Barron, Ed.D.
x Secondary Education
t USF Electronic Theses and Dissertations.
Metacognitive Strategies in a Web-Enhan ced Environment: The Effects on Achievement in Problem-Solving for Engineering Undergraduates by Sally A. Zabel A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Secondary Education College of Education University of South Florida Major Professor: Ann Barron, Ed.D. Frank Breit, Ph.D. Lou Carey, Ph.D. William Young, Ed.D. Date of Approval: December 8, 2005 Keywords: Metacognition, instructiona l strategies, embedded cuing, reflective assessment, web-based instru ction, self-regulated learning Copyright 2006, Sally A. Zabel
Dedication To my loved ones, family, friends, and colleagues.
i Table of Contents List of Tables iv List of Figures v Abstract vi Chapter One Introduction and Background 1 Introduction 1 Statement of Problem 1 Purpose of Study 3 Research Questions 5 Significance of Study 6 Definition of Terms 7 Limitations and Delimitations of the Study 9 Chapter Summary 9 Organization of Remaining Chapters 10 Chapter Two Literature Review 12 Overview 12 Introduction 12 Theoretical Basis of Study 13 Information Processing Theory 13 Problem-Solving 19 Social Cognitive Theory 23 Self-Regulated Learning (SRL) 24 Self-Monitoring 29 Metacognitive Awareness 33 Metacognition and Performance 34 Metacognitive Cues 37 Instrumentation 42 Chapter Summary 45 Chapter Three Research Methods 47 Participants 47 Ethical Considerations 48 Instructional Procedures 48 Both Groups 48 Control Group 49 Experimental Group Inst ruction and Materials 50
ii Instruments 54 Pretest 54 Posttest 55 Attitudes 57 Research Design 58 Establishing Comparable Groups 59 Post Treatment Data Analysis 60 Qualitative Questions 62 Matching Posttest for Pretest 62 Reflections on Procedures Used 62 Biographical Questionnaire 62 Attitude Survey 63 Chapter Summary 63 Chapter Four Results 64 Introduction 64 Participants 64 Establishing Comparable Groups 65 Achievement Pretest 65 Attitude Pretest 66 Post treatment Data Analysis 67 Question 1 67 Question 2 70 Question 3 72 Question 4 74 Question 5 75 Question 6 80 Chapter Summary 81 Chapter Five Discussion 83 Introduction 83 Discussion of Results 83 Question 1 83 Question 2 87 Question 3 90 Question 4 91 Question 5 97 Question 6 99 Limitations 100 Conclusions and Recommendations for Further Study 101 Implications for Practice 102 References 104 Appendices 119 Appendix A: Engineer ing Problem-Solving 120
iii Appendix B: How Do You Solve Problems (HDYSP) Inventory 129 Appendix C: Participant Survey 132 Appendix D: Pre/Posttest of Thermodynamics Concepts 134 Appendix E: Grading Rubric 136 Appendix F: Web-Based Problem Solving Tutorial Survey 137 Appendix G: Web-Based Homewo rk Survey 04.26.05 Written Comments 139 About the Author End Page
iv List of Tables Table 1 Metacognitive Checklist 18 Table 2 Metacognitive Functions Classifi ed According to the Process Phases 26 Table 3 Phase Structure and SubProcesses of Self-Regulation 28 Table 4 Group Assignment Ranking 59 Table 5 Data Collection Before, Du ring and After Treatment Phases 61 Table 6 Thermodynamics Knowledge Pretest Results 65 Table 7 Attitude Pretest Results 67 Table 8 Intrarater-Interrater Reliability Correlation Matrix 69 Table 9 Achievement Composite Score 70 Table 10 Attitude Pretest and Posttest Results 71 Table 11 Analysis of Covariance for Attitude Performance Scores 72 Table 12 Thermodynamics Knowledge Pretest and Posttest Results 73 Table 13 Analysis of Covariance fo r Pretest and Posttest Achievement Performance Scores 74 Table 14 Frequency Distribution for Metacognitive Reflection Responses 75 Table 15 Group Comparison by Biographical Dimension 77 Table 16 Means for Survey Respon ses of Students Using Web-Based Problem-Solving Tutorials 81
v List of Figures Figure 1 The Basic Model of Learning and Memory 16 Figure 2 Components of Metacognition 20 Figure 3 Triadic Form of Self-Regulation 31 Figure 4 Screen Shot of Instruct ional Frame for Problem Sets 50 Figure 5 Screen Shot of the Engi neering Problem-Solving Model 51 Figure 6 Screen Shot of Instructio nal Frame for Problem Sets with Embedded Cuing 52 Figure 7 Screen Shot of Metacognitive Cuing Within the Â“HelpÂ” Feature 53 Figure 8 Screen Shot of Metacognitive Cue Reflection 54
vi Metacognitive Strategies in a Web-Enhanced Environment: The Effects on Achievement in Problem-Solving for Engineering Undergraduates Sally A. Zabel ABSTRACT This study focused on the effects of usi ng metacognitive strategy cuing integrated into problem-solving activities in a web-base d learning environment. Purposes of the study were to investigate: (a) differences in posttest achievement between students who received metacognitive strategies embedded as cues in engineering problem sets and students who did not receive the treatment; (b ) differences in perceptions of problemsolving skills between students who received metacognitive strategies embedded as cues in engineering problem set and students who did not receive the treatment; (c) differences in thermodynamics knowledge; (d) problem-sol ving steps students reported using across problem sets; (e) characteristic s of sampled students, and (f) studentsÂ’ perceptions of web-based problem sets. The sample consisted of 81 students enrolled in an undergraduate thermodynamics course. In-class lectures we re scheduled twice w eekly, and web-based problem sets were assigned as homewor k. Two groups, the treatment group using embedded metacognitive cuing and the control group not using the embedded metacognitive cuing, practiced with probl em-solving activities over a fifteen-weeksemester.
vii Two-thirds through the semester, comp rehensive posttest achievement scores were compared between groups. Analyses showed no significant differences between groups when metacognitive strategies were in corporated into web-based problem sets. An instrument was developed and valida ted to measure student sÂ’ perceptions of their abilities to plan, monitor, and evaluate problems. Preand post testing of studentsÂ’ self-reported perceptions were measured. The results in dicated no significant differences between groups. When differences in thermodynamics knowledge and skills between students were measured, pretest to posttest results showed equal improvement for both groups, contradicting the hypothesis those students in the treatment group would improve in skills and knowledge more than the control group. A frequency analysis revealed differe nces in the amount of times studentsÂ’ reported using engineering problem-solving step s while working through exercises. Most frequently chosen was Step Two List Variables (91 %) and Step Seven Solved Equations (91%).. The least chosen response was S tep Four Â– Made/stated Assumptions which was selected only three percent of the time. Implications from this investigation, along with previous research, facilitate definition of boundary conditions when em ploying metacognitive cuing in web-based learning.
1 Chapter One Introduction and Background Introduction Statement of problem. The ability to solve problems is an essential life skill, especially for students entering the field of engineering. Core courses that emphasize problem-solving in undergraduate engineering education are difficult for most students and impossible for some. In order for student s to become better problem solvers, they need to possess a general understanding of pr oblem-solving and, in particular, they need to understand their own intellectual ab ilities (Davidson & Sternberg, 1998). Livingston (1997) suggests faculty ca n help students understanding problemsolving and cognitive goal setting by having th em use metacognitive strategies to control their cognitive activities. The term metacognitive is defined as the learnerÂ’s ability to be conscious of and manage oneÂ’s own lear ning processes (Peters, 2000). Innumerable opportunities exist to improve studentsÂ’ metacognitive proficiency through guided classroom instructional practices, and student s should have the opportu nity to use their newly acquired skills to improve perfor mance (Flavell, 1987; Gourgey, 2001; Schraw, 2001). Recently, researchers have investigated the use of computer-mediated programs as a means to encourage learnersÂ’ thinki ng and reflection on inst ructional content, resulting in positive support of further study (Lin & Lehman, 1999; Watson & Allen, 2002; White & Frederiksen, 1998). Consequentl y, the present study ex plored the effects
2 of using metacognitive stra tegies when problem-solvi ng in a web-based learning environment. Over the last decade, technology has tremendously changed the dissemination of education. Â“Emerging technologies are le ading to the development of many new opportunities to guide and enhan ce learning that were unimagi ned even a few years agoÂ” according to Bransford, Brown, & Cocking (1999, p.4). The Internet, and specifically web-based instruction, has become widely ac cepted at many educational institutions in the United States and Canada. According to The Sloan Consortium Report (2003), over 90% of all public post-seconda ry schools offer at least one fully online or blended learning course. Web-based instruction offers options not available in the traditional classroom. Some advantages of how web-based instruc tion shapes the learning process include the ability to: (a) pres ent students with immediate feedback, (b) expose learners to real-world data, (c) provide deeper learning experiences, (d ) facilitate critical thinking skills, (e) allow learner reflection before responding, and (f) grant equal treatment to learners (Horton, 2000). Technology and media expand the ability of a studentÂ’s perception, listening, manual dexterity, and speech (Ryder & Wilson, 1996). Technology makes possible some types of learning activities (e.g., discovery learning) and supports others (e.g., cooperative learning) that otherwise would be more difficult or impossible to achieve without technology (Smith, 2002). Â“Hypermedia helps to focus a studentÂ’s attention on relationships rather than discrete facts, which assists learners in building accurate mental repres entationsÂ” stated by Jacques, Nonnecke, McKerlie, & Preece (1993, p. 225). As a student becomes more proficie nt in the use of
3 technology, less attention is need ed to focus on mundane activ ities and affords more time to contemplate higher order thinking. To that end, some researchers propose that hypermedia supports higher order thinking, such as calculating the suitability of information (Dede, 1987). The application of metacognitive strategies in learning can be addressed through development of an onlin e environment of homework problems in which students receive immediate feedback and are guided, when needed, through the process of problem-solving. Integration of metacognitive strategies into web-based homework activities provides a unique mechanism to steer students towards the development of problem-solving skill s necessary in engineering. Purpose of Study. This research studied the effect of using metacognitive strategies when problem-solving in a web-ba sed learning environment. Problem-solving is an integral requirement of the learning process for engineers. Traditionally, the learning situation created by the teacher is a predetermined curriculum outlining course objectives and activities. In this type of environment, assignments, homework and testing are all evaluated by the instruct or. In many cases, students request more opportunities from faculty to work through probl em solutions than time permits in any given class period. Although homework problems have customar ily been assigned from the textbook, there are several reasons why this strategy im pedes the learning process. The primary obstacle is that undergraduate core courses have sizeable enrollments making it difficult for instructors to interact on an individual basis with students. Sometimes students who encounter difficulty solving a problem cannot proceed without assistance from the instructor or teaching assistant and often stud ents work together in groups to support each
4 other; however, this can lead to an uneven distribution of effort within the group. Proper assessment of contributions to homework assign ments is difficult since often more than one student has taken part in the work. This makes it challenging for the instructor to grade an individualÂ’s work appr opriately. Without a structure or a set of strategies in which to frame problems, students are left to devise their own methods of problemsolving (Buck, 2004). When immediate feedback is absent, students are unaware if they have solved the problem co rrectly and, by the time they do receive feedback, new material has been covered. In courses with large enrollments, it is more difficult for the faculty to address an individua l studentÂ’s concerns through on e-to-one interactions, thus making it necessary for students to take more responsibility for their own learning. One way to assist independent learni ng is to teach students to us e metacognitive strategies so they can plan and monitor their own performanc e and decide whether it is appropriate to use a specific strategy at a partic ular time (Ashman & Conway, 2002). While there is a significant amount of research in the l iterature regarding metacognition and its benefits, there is a sparse amount of literature discussing metacognitive prompts embedded in web-ba sed instruction on l earning outcomes and student self-perception of problem-solving ab ility. Because there has been a heavy emphasis on metacognition in classroom settings as it relates to learning over the past thirty years, it seems a natural evolution to investigate metacogn ition within web-based education. Therefore, this study evaluated the effect of implementing student selfevaluation in the learning environment th rough metacognitive reflection. Specifically, this study examined whether there was a di fference in student performance, when reflective-assessment was in troduced, and if change in undergraduate studentsÂ’
5 perceptions of self-efficacy in problem-solv ing were affected through a self-reflection intervention. Research Questions The study design utilized mixed-method research. According to Johnson and Turner, 2003) Mixed methods are used to obtain corroboration of findings, minimize alternative explanations for conclusions, and to elucidate divergent aspects of the research. Â“Methods should be mixed so that they have complementary strengths and nonoverlapping weaknessesÂ” (316). The focus of th is research centered around the effect of using metacognitive strategies on problem-solvi ng in an on-line learning environment. Differing types of web-based lessons in an undergraduate engin eering course were compared. All of the students solved probl em sets for a thermodynamics course. Two types of instruction were administered : (1) web-based homework problem sets with embedded metacognitive strategy cuing and self-reflection, and (2) web-based homework problem sets without the embedded metacognitive stra tegy cuing and self-reflection. This research concentrated on the learning outcomes and perceptions of students who used metacognitive strategies included within problem-solving instruction and practice. The overall research question was: What is the effect on studentsÂ’ problemsolving ability when direct instruction on embedded cues for using specific metacognitive strategies is included in web-based instructio n? Two outcomes were an ticipated from this research: (1) improved performance outcome s on a posttest measure of comprehension and (2) increased student self-perception of problem-solving practices. Six questions were considered: 1. Was there a difference in posttest achie vement between students who received
6 direct instruction using metacognitive st rategies and embedded cues in their thermodynamics problem sets and student s who did not receive instruction in metacognitive strategies information and cuing? 2. Was there a difference in perceptions of their thermodynamics problem-solving abilities between students who received di rect instruction in using metacognitive strategies and embedded cues in their problem sets and students who did not receive instruction on metacognitive strategies information and cuing? 3. Were there differences in thermodyna mics knowledge between students who received direct instructi on using metacognitive strategies and embedded cues in their thermodynamics problem sets and st udents who did not receive instruction in metacognitive strategies information and cuing? 4. Which of the problem-solving steps did students report using across the problem sets? 5. What are the characteristics of the students in the sample? 6. What were the participantsÂ’ percepti ons of the web-based problem sets? Significance of Study Colleges of engineering expe rience a significant rate of attrition during the first two years of study (R utz et al., 2003), and as many as 25% of students enrolled in thermodynamics classes at this university do not complete the necessary requirements for a pa ssing grade in the course (J oseph, 2004). They either drop out of the program or subsequently retake th e course in another semester. The ability to comprehend higher-order thinking when probl em-solving is essential for success in thermodynamics courses and engineering in general. The current evaluation criteria outlined by the Accreditation Board for Engineering and Technology (ABET, 2003) states program graduates must demonstrate co mpetency to identify, formulate, and solve engineering problems. Common practice in courses such as ther modynamics is for the teacher to prepare and deliver the lectures, assign and grade homework assignments, and evaluate students
7 through objective testing procedures, in short, a teacher-centered approach. Limitations placed upon adult learners strongly conflicts with their inherent need to be self-directing and can lead to disagreement, indifference or estrangement (deLeon, 1996). Rather than control the act of learning, the teacher can improve the likelihood of certain behaviors by encouraging and supporting the student in various activities (Gagne, 1985). Nielson (2004) found that students with high levels of self-efficacy were more prone to be cognitively engaged when tryi ng to learn the material th an students with low selfefficacy. It is hoped the findings from this study would add to the body of literature on integration of metacognitive stra tegies into the curriculum. Outcomes from this research could lead to a change in the nature of student-teacher interactions in the learning environment, where students assume more re sponsibility for their own learning (a power shift) through the evaluative pr ocess. This is turn may l ead to more use of web-based support materials to facilitate stude ntsÂ’ skills for lifelong learning. Definition of Terms The following terms are defined to assist with the understanding within the context of this study. 1. Andragogy: applying the process of learning to adults (Knowles, 1968). 2. Construct validity: the extent inferences can be made from th eoretical constructs to operationalizations within a study (Trochim, 2005). 3. Embedded cues: using written or verbal prompts to stimulate thinking. 4. Executive control: regulating the processes that take place during learning. 5. Human agency: having the capability of being conscious of and control over oneÂ’s own actions. 6. Instructional fading: deliberately dimi nishing the amount of instructor support by reallocating more and mo re control to the learner (Wilson, Jonassen, & Cole, 1993).
8 7. Metacognition: managing subordina te thought processe s by higher order thought processes (Broadbent 1977). Further, according to Hacker (1998), fundamental to the construct of metac ognition is the idea of thinking about oneÂ’s own thoughts. The process can be what one knows (i.e. metacognitive knowledge), what one is performing (i.e., metacognitive skill), or what is oneÂ’s existing cognitive affective state (i .e., metacognitive experience). 8. Metacognitive strategies: are the highe r level thought processe s used to control or modify lower level (or cogniti ve) thought processes (Hacker, 1998). 9. Metacognitive cuing: promoting thi nking about oneÂ’s thi nking processes through written or verbal prompting (Condor, 2001). 10. Problem-Solving: attaining a desire d outcome through the application of knowledge. 11. Reflective assessment: employing an ac tivity in which students reflect upon their own learning inquiry. 12. Scaffolding: providing support (models, cues, prompts, hints, or partial solutions) to students to bridge the ga p between what students can do on their own and what they can do with guida nce from othersÂ” (Hartman, 2001, p. 167). Additionally, it Â“is a form of coaching or tutoring which helps learners accomplish tasks that they cannot do without assistance, therefore aiding in the construction of expertise in the task s, engendering autonomous performance aptitudeÂ” (McNeill, 2002, p. 3). 13. Self-efficacy: the concept that people have the ability to obtain desired results through self-motivated acts (Bandura, 2001; Onwuegbuzie, 2001). 14. Self-regulated learning (SRL): the self -directive method of learning is defined as a series of instructional activitie s focused on needs assessment, procuring learning resources, employing learning activ ities and the evaluation of learning (Hiemstra, 1998). 15. Web-based instruction: any intentional us e of web technologies in order to aid in the educational pr ocess (Horton, 2000). Limitations and Delimita tions of the Study Potentially, the internal validity of th e findings may be threatened due to the experimental design which lacks random a ssignment through differential selection of participants (Wiersma, 1995). Pa rticipants in this study were chosen from a pre-existing
9 group of students who registered for a cour se section offered in the engineering undergraduate curriculum. Because the group was intact, it was important to compare ability levels of the students when checki ng for group equivalence. Consequently, the ability to generalize the results to the larg er population could be impaired by selection bias (Wiersma, 1995). A second threat to internal validity results from mortality or the loss of participants over the course of the experiment. Taking in to account the drop/add period which occurs during the first week of cla sses, the study began af ter this period was completed. Although this threat is likely to occur during the remainde r of the semester, a robust sample size has been chosen to account for further loss of participants. Other threats under consideration included the amount of time participantÂ’s in the experimental group spent on the treatment (i.e., reading th e material and reflecting upon strategic choices) and the possibility of the particip ants from the two groups discussing the differences in the instructional format. Again, because of insufficient sample randomness, it is imperative to take into account external validity threats when attempting to interp ret results. Ecological validity is threatened due to demogr aphic constraints (i.e. limited geographic region) while the population validity is restricted by selection of only engineering stude nts as participants. There is insufficient evidence to disregard th e potential threat of temporal validity. Chapter Summary Problem-solving is a necessary skill for engineers. The core courses, which define the engineering curriculum, are cha llenging to most students and require the ability to problem solve effectively. A priori ty in engineering education is to provide
10 students with skills and competencies that permit them to progress easily into professional life (H adjileontidou, 2004). When students actively think about the processes involved in problem-solving, they are utilizing metacognitive strategies. Since these strategies have been shown to be important to the student for se lf-regulation of learning activities, they should play an essential role in instructional activities. According to Flavell (1987), metacognitive experiences play a significant role in daily cognitive lives. Through the incorporation of metacognitive strategies in to problem-solving homewor k, students will have the opportunity to become more aware of the stra tegies and have the opportunity to practice with them. Organization of Remaining Chapters. The remaining chapters include a comprehensive review of the literature in Chapter Two, the experimental design and analyses in Chapter Three, followed by the re sults in Chapter Four and conclusions in Chapter Five. Several theoretical frameworks (i.e. Information Processing Theory, Social Cognitive Theory, and Self-Regulated Learning) form the basis of discussion relating to the research questions and subsequent hypothe ses. Following the evaluation of literature in Chapter Two, Chapter Three is a presentation of the research design used in the study. A mixed-methodology approach to the research was employed because the nature of the study investigated both quantitative and quali tative dimensions. Participant selection, ethical considerations, instruments, procedur es, variables, resear ch design, and data analyses are discussed followed by a brief summary of the methods section. Chapter Four is a discussion of the study results, ending with conclusions, implications, and recommendations for further study in Chapter Five.
12 Chapter Two Literature Review Overview Chapter Two is a review of literature focusing on the role metacognition plays in a web-based, problem-solving environment as it relates to studentsÂ’ academic performance and perceived problem-solving ab ility. Beginning with an introduction to the theoretical framework of the study thr ough a discussion on Information Processing Theory, Social Cognitive Theory, and Self -Regulated Learning Theory, a further exploration delves into the research on metacognitive awareness and performance, problem-solving, and self-monitoring. Included in the chapter is a discussion of the research surrounding metacognitive strategies in instruction and its association to the research questions. This chapter will wrap up with a summary of instruments designed to measure metacognitive awareness. Introduction Leading educators consider, Â“the really important, central point of education is to teach people to think, to use their rational powers, to become better problem solversÂ” (Gagne, 1989, p. 458). The demonstration of successful problem-solving is indicated when the learner is able to change from one strategy to another, to choose or discard a strategy, or to quickly deliberate upon a probl em solution (Gagne, 1989). Such activities employ metacognitive knowledge, engaging learners in higher order thinking skills which have a significant part in cognition and problem-solving (Bransford, Sherwood, Vye, &
13 Reiser, 1986; Jonassen, 2004). Â“Metacognition is especially important because it affects acquisition, comprehension, retention and appli cation of what is learned, in addition to affective learning, efficiency, critical th inking and problem-solvingÂ” (Hartman, 2001, preface). Students, who take part in metac ognitive activities such as self-evaluation, monitoring, and revising, e nhance their learning (Gourge y, 2001; King, 1991; Lin, 2001; Schoenfeld, 1985). Recent research has found positive relati onships between learning outcomes and a studentÂ’s use of effective learning strate gies (Covington, 2000; Zimmerman, 1989). An examination of literature for this study re vealed conditions under which metacognitive interventions have particular success. In the studies, (Everson & Tobias, 1998; Hong, McGee, & Howard, 2001; Kapa, 2001; Schoenf eld, 1985), results point to increased performance among both high and lower achie ving students. Lower achieving students showed the most dramatic improvements, ther eby closing the gap with their more capable peers. Condor (2001) and Kramarski & Zeichner (2001) found implementation of metacognitive strategies into learning situ ations increased awareness of metacognition demonstrating significant differences for th e treatment groups. Wa tson & Allen (2002), however, discovered no measurable differences in posttest achievement scores citing a possible interaction effect. Theoretical Basis of Study Information Processing Theory. It is not enough for students to know what to do; they must also be aware of how and when to do it. The heuristics a learner uses to internalize habitual behaviors of self-mon itoring and self-guiding activities are called cognitive strategies (Bullmaster & Alcoc k, 2003; Gagne & Drisco ll, 1988; Rosenshine,
14 1997). Simply phrased, cognitive strategies are the basic methods to guide students in Â“attending, learning, remembering and thinki ngÂ” (Gagne & Driscoll, 1988, p. 55). Information-processing theory was deve loped through the work of those in the field of cognitive psychology. It is concerned with the explanation of how information is managed by the human brain. Changes to c ognition occur, as attention, memory and metacognitive functions mature The theory as it applie s to human thinking has been compared metaphorically to the flow of da ta input and output w ithin a computer. (Flavell, 1985; McCown & Roop, 1992). The f undamental aspect of this theory is focused on how cognitive information is coded, stored and retrieved. Learning occurs when neural impulses of taste, touch, sm ell, hearing and sight (stimuli) from the environment enter the nervous system through a sensory register. The learner perceives the incoming information selectively, coding it in a conceptual form kept for a brief time in short-term memory. The information is then transformed into meaningful representations for long-term stor age. The ability to then retr ieve information from either long or short-term memory is evidence learning has taken place (Gagne, 1989). There is a limited capacity w ithin the brain to process stimuli, and it can easily be overloaded when the Â“processing demand Â… [exceeds] its processing capacityÂ”. (Flavell, 1985, p. 76). In order to deal with incoming in formation, four elements of attention are required. The first is the abil ity to control the length of tim e for concentration. This is often referred to as Â“attention spanÂ”. Secondly, a person must be able to appropriately associate task demands. For example, when two variables of a probl em are presented, the learner is capable of forming relationships. The third aspect require s proficiency to plan attention, having the means to choose what is important for the task at hand. And finally,
15 the monitoring of attention is to know when and how to modulate concentration. Initially, the limited capacity can impede th e speed at which processing takes place, however as strategies are de veloped, memory capabilities are liberated and can be extended (Flavell, Miller & Mi ller, 1993). Strategies ex ercised to sort retrieved information stored in memory are, accord ing to Brown (1987), e ssential to intelligent problem-solving. Memory plays a key role in Information-Processing Theory. Input of data is transferred among three structures within me mory. Short-term memory, also known as Â“working memoryÂ” is where information is pr ocessed or worked-on as it resides in a personÂ’s consciousness. How the space is used becomes important due to a threshold of how much information can be managed at a gi ven time. Two types of space exist within memory; 1) operating space and, 2) storage space The former is where operations are performed and the latter is where supplementar y information is kept. Inexperience with problems requires the use of much of this space to undertake the operations. Through practice and biological maturation, automaticity, or the relief of conscious effort, leads to more efficient problem-solving. The information-processing model of learning and memory is a fundamental structure for a number of cognitive learni ng theories (Gagne & Driscoll, 1988). The model, as shown in Figure 1, is a represen tation of how a stimulus moves from the environment at large, through sensory percep tion and is transformed into information stored in short and long-term memory for la ter recall. Transformation actions, known as
16 learning processes, occur when the informati on is retrieved to elicit responses such as speech and movement. Figure 1 The Basic Model of Learning and Memory From Essentials of Learning by R.M. Gagne and M.P. Driscoll, 1988, p.13. The manner in which learning occurs is significantly affected by the executive control and expectancies structures. When f aced with a learning situation, an expectancy about the outcome of the learning is anticipat ed which influences how the information is coded into memory. Similarly, the learner has the ability to c ontrol the coding and retrieval processes. The activ ation and modification of info rmation flow directed through these processes are referred to as cognitive strategies (Gagne & Driscoll, 1988). They make possible the ability to perform higher order operations (e.g. reading comprehension, writing or mathematical problem-solving) and to exert executive control.
17 Executive control is more general in the re gulation of the proce sses that take place during learning. Thinking about oneÂ’s own thinking is a simplified definition of metacognition. Flavell (1987) used this term to mean the learnerÂ’s ability to be conscious of and manage their own learning processes. Metacognitive processes are central to planning, problem-solving, and evaluation of a studentÂ’s own learning. When knowledge is used to meet a goal through strategic planning it is said to be a metacognitive process. For example, a student prepares to solve a homework problem, first by determining the complexity of the task, then recognizing wh at they do or do not know about the problem, planning approximately how long the task will take, checking successfulness as they work, applying all relevant re sources, and finally reviewing their conclusions to verify the answer is reasonable given the problem (Hartman, 2001). Flavell (1979, 1981, & 1987) furthered his definition of metacognition by distinguishing between its two aspects; metacognitive knowledge and metacognitive experiences or regulation. Met acognitive knowledge is comprised of three dimensions: 1) the student; 2) the task; and 3) the strategy. In order for a student to become a better problem solver, he/she needs to possess Â“ knowledge about problem-solving, in general, and about their own mental processes, in particularÂ” (Davidson & Sternberg, 1998). A student may, for example, use a metacognitive knowledge process when planning how to proceed with a reading assignment: What do I (person variable) know about this topic (task variable), so I will be able to understand both the content and vocabulary in the passage (strategy variable)? (Livingston, 1997).
18 Metacognitive experiences ar e concerned with the sequent ial processes of cognitive activities and the attainment of a cognitiv e goal for example, understanding a problem (Brown, 1987; Livingston, 1997). Table 1 (Ferre r, 2001) is an example of what occurs when a student activates me tacognitive strategies compared to when they do not. Students who are actively engaging in thinki ng about the process of problem-solving are using metacognitive strategies. Table 1 Metacognitive Checklist After Reading an Assignment Active Metacognitive Strategies In active Metacognitive Strategies Reflect on what was read Stop reading and thinking Summarize major ideas Do nothing extra Seek additional information from F eel satisfied that reading is enough outside sources Feel success is a result of ef fort Attribute success to luck Note From Metacognition by Ferrer, 2001, p.2. Â“Research as well as personal experience ha ve demonstrated that students who use metacognitive strategies, notably identifyi ng goals, self-monitoring, self-questioning, reasoned choice of behaviors, and self-assessment, are more academically successful than students who do not use these strategies Â” (Gourgey, 2001, p. 30). There are several reasons why an emphasis should be placed on the teaching of metacognitive skills: 1. Long term, students need to learn gene ral skills of planning and how they apply to a wide variety of tasks a nd domains, rather than learning a specific skill or task.
19 2. Effective cognitive performance de pends upon the ability to utilize metacognition. 3. Students generally are not in the hab it of questioning themselves, rather, they blindly follow instructions. 4. Students who are deficient in metacognitive skills are unaware of the specifics in performing a task. 5. Metacognitive skills are important for students to: estimate task difficulty, monitor their understanding of the task, plan ahead, oversee their performance (knowing when the have reached mastery of a topic), apply all germane information, and avoid incorrect conclusions or representations. (Hartman, 2001, Wagner & Sternberg, 1984). The use of general strategies and metacogni tive knowledge as well as domain specific knowledge has been linked significantly to thinking and problem-solving (Bransford et al., 1986). Problem-Solving. The dissimilarity between be ing a good or a poor problem solver is often in the learnerÂ’s ability to think about oneÂ’s pr oblem-solving activities (Gardner, 1991; Schraw, 2001). Â“Poor problem so lvers lack spontaneity and flexibility in both pre-planning and monitoringÂ” (Brown, 1987, p. 86). Research in problem-solving for mathematics assumes those considered e xperts initiate a thr ee stage process of metacognitive and cognitive activity when wo rking on problems. Figure 2, graphically depicts the preactive (or planning) phase, th e interactive (or monitoring) phase and the post active (or evaluating phase). Each phase is an interconnected process integrating into the learning activity.
20 Figure 2. Components of Metacognition From Mathematics teaching as pr oblem-solving: A framework fo r studying teacher metacognition underlying instructional practice in mathematics, by Artz, A.F. & Armour-Thomas, E. (2001), p. 130. In H.J. Hartman (Ed.), Metacognition in Learning and Instruction, Theory, Research and Practice, p. 130. Boston, MA: Kluwer. Higher order cognitive skills are required to solve problems effectively. Novices are not generally able to rec ognize the distinction of problem types, so they must rely on general strategies for problem-solving, wh ich do not provide st rong strategies for problem solutions (Jonassen, nd). Because experts have the ability to recognize similarities in solving partic ular types of problems and spe nd more time in the planning stage, they tend to be better problem solv ers (Brown, 1987; Davids on & Sternberg, 1998; Sweller, 1988). According to Everson and Tobias (1998): Learning in complex domains such as science and engineering, or making
21 diagnosis in medicine or ot her fields, often requires that students bring substantial amounts of prior learning to bear in order to understa nd and acquire new knowledge or solve problems. Some prio r learning may be recalled imperfectly, or may never have been completely ma stered during initial acquisition. Students who can accurately distinguish between what they know and do not know should be at an advantage while working in such domains, since they are more likely to review and try to relearn imperfectly ma stered materials needed for particular tasks, compared with those who are less accurate in estimating their own knowledge (p. 76). Metacognitive awareness allows the lear ner to discern and select a suitable strategy to solve the problem. When a probl em solver chooses to use a correct strategy from one related problem to another, it sign ifies metacognitive ability and demonstrates he/she has the ability to know how and when to use it (Jackson & Butterfield, 1986). Proficiency in three areas seems to be present in productive thinking and problemsolving: Â“intellectual skill (concept a nd rules), verbal knowledge, and cognitive strategiesÂ” (Gagne, 1989, p. 464). While cognitive strategies f acilitate the construction of knowledge, metacognition aids Â“science lear ners to develop and use effective and efficient strategies for acquiring, understan ding, applying and retaining extensive and difficult concepts and skillsÂ” (Hartman, 2001, p. 198). Research on expert versus novice behavior reports, experts have the ability to set clear goals, comprehend concepts and the relationships of concepts, keep track of their understanding, and make decisions on whether their actions are leading to wards defined goals (Gourgey, 2001).
22 The following two study descriptions illustrate metacognitive strategies are more efficiently used by experts than novices and especially when employed in complex problem-solving situations. Schoenfeld (1985) conducted a study which investigated the relationship between a studentÂ’s proficiency at problem-solving and their perceptions of the problem-solving process. The research illustrated when metacognitive skills were employed by experts more efficient perfor mance resulted. The findings supported the three hypotheses within the st udy. First, novices perceive problem-solving differently than experts whereby they look only at surf ace characteristics which could result in incorrect conclusions. Secondly, experts perc eive problem-solving with an eye for deep structure, which allows them to recognize, categorize and select e fficient solutions to problems, thereby eliminating protracted expe riences. And finally, as students became more adept at problem-solvi ng, their perceptions of the pr ocess and their performances became more expert-like. In two related studies by Hong, McGee, and Howard (2001), 9th-grade students and 6-8th-grade students participated in re search to look at four mental components (cognition, metacognition, non-cognitive variable s and justification skills) deemed important for successful problemsolving. Bo th studies were investigated separately over a 4-week period. The first study us ed an open-ended response format for presentation of both the well-structured a nd ill-structured problems. The second study used a multiple-choice format for similar well-structure and ill-structured problems. Students were given an inventory measur ing both knowledge of metacognition and regulation of cognition. The researchers conc luded Â“regulation of cognition was strong predictor in solving only ope n-ended ill-structured problems. The results suggest that
23 problems have to be complicated enough to challenge students to use regulation of cognition for researching successful solution. In other words, students may not need to use regulation of cognition if the problem l acks conceptual and structural complexity, even though they have those skillsÂ” (p. 4). Â“Metacognitive knowledge may also compensate for low ability or lack of relevant prior knowledge (Sch raw, 2001, p. 7). SwansonÂ’s (1990) investigation found high levels of metacognitive knowledge about problem-solving compensated for lower aptitudes in children from grades 4 and 5. Two pre-tests of aptitudes were administered to participants with diverse academic aptit udes. A metacognitive questionnaire was first administered followed by problem-solving task s. Significant results from this study indicated problem-solving performance is pos itively influenced by high-metacognitive ability regardless of aptitude. Additionall y, high aptitude is only important when metacognitive ability is low. Social Cognitive Theory. The facility to have the power over the conditions of oneÂ’s existence is the Â“quintessenceÂ” of being human (Bandura, 2001). He believed people possess a self system which allows a degree of contro l over their thoughts, feelings and actions; and facilitates the pe rceiving, regulating a nd evaluating of oneÂ’s own behavior within an external environm ental context (Marzano, 1998; Pajares, 1996). A role of the self system is to self-regula te both control and ag ency. The concept of agency is the ability to be aware of and in command of oneÂ’s own actions. Three types of agency are distinguished with in this theory: direct pe rsonal agency, proxy agency, and collective agency.
24 Direct personal agency refers to pe opleÂ’s ability to im agine innovative ideas intentionally while consideri ng unique ways to implement them. The concept of proxy agency suggests authorizing someone to act on oneÂ’s behalf to gain preferred effects, while collective agency is based on collaborati ve efforts for desire d outcomes (Bandura, 2001). Self-efficacy, a core feature of agenc y, is a concept that individuals have the capacity to produce desired effects from self -motivated activities to meet goals and expected outcomes. (Bandura, 2001; Onwuegbuzie, 2001). Self-Regulated Learning (SRL). Many adults learn best when they feel empowered, autonomous, goal-orientated and responsible for their own learning (Cranton, 1994; Hiemstra, 1998; Knowles, 1980; Lieb, 1991; Mezirow, 1997). According to Merriam & Caffarella (2001), adults are adept at managing the many facets of their lives and are able to take responsibility for, or at the least take part in the process of, planning their own learning. An approach to directing and planning oneÂ’s own learning can be found in the research on self-regulated learning (SRL) theory. The self-directive method of learning is defined as a process of instruction based upon such activities as needs assessment, procuring learning resour ces, employing learning activities and the evaluation of learning (Hiemstra, 1998). Intentionality and forethought, self-reactiv eness, and self-reflectiveness, core elements of human agency are underlying cons tructs of Social Cognitive Theory which allow people a role in self-development, se lf-renewal and adaptation over time (Bandura, 2001). The concept of intenti onality within this theor y, describes humans actively participating within their environments through mindful decisions when faced with changes rather than passively reacting to them. Manifest ation of agency describes
25 forethought and self-reactiven ess, allowing people to move beyond immediate boundaries while molding and controlling their present st ate to a preferred future state (Bandura, 2001). Additionally, self-reflectiveness is the be lief one has to maintain a certain amount of influence over managing themselves a nd environmental events surrounding them (Bandura, 1997; Pajares and Schunk, 2001). Self-efficacy is a fundamental tenant of human agency. Bandura (2001) states the construct of efficacy is key to self-regul ating motivation through setting challenging goals and outcome expectancies. Through metacognitive activities of self-efficacy, decisions are made on Â“what challenges to undertake, how much effort to expend in the endeavor, how long to persevereÂ” (Bandur a, 2001; Strauser, Ketz, & Keim, 2002; Wongsri, Cantwell, & Archer, 2002). Co mponents of self-efficacy include: 1) expectations about the future; 2) influence th e way we behave in sp ecific situations; 3) beliefs about yourself Â– contro l personal agency; 4) mastery experiences Â– self-efficacy beliefs; and 5) evaluate experiences thr ough self-reflection (Ha nnibal & Gymnasium, 2003). An essential skill to successful learning in the sciences is problem-solving and requires students who demonstrate metacogniti ve proficiency by generating questions to achieve solutions (McLoughlin & Hollingworth, 2001). Table 2 represents the phases of cognitive tasks in problem-solving and the a ssociated metacognitive regulation of such tasks.
26 Table 2 Metacognitive Functions Classified According to the Process Phases Solving-phase The metacognitive function a) Problem identification Collecting data, coding and remembering b) Problem representation Analogy, infe rence, imaginativeness, selective comparison and combination c) Planning how to solve Integrat ion, conceptualization, heuristic choosing and formulating d) Planning performance Controll ing and monitoring performance components of algorithmic mathematical knowledge and appropriate rules e) Evaluation Adjusting and contradicting a few possible solutions or suggesting alternative solution methods Note. From Â“A metacognitive support during the pro cess of problem-solving in a computerized environmentÂ” by E. Kappa, 2001, Educational Studies in Mathematics, 47(3), p. 318. Results from research in the literature demonstrate that learners competent in metacognitive self-assessment, those cognizant of their abiliti es, are more intentional and outperform those who are unaware (River s, 2001; Schraw & Dennison, 1994; Swanson, 1990). Cognitive strategies are not specific pro cedures to be followed rather; the learner develops internal processes to enable comp rehension. Wren (2004) lists five reasons why higher order cognition is of importance: Â“(a) enables students to grapple with intellectually sophisticated chal lenges, (b) enables students to integrate multiple ideas and facts, (c) enables students to undertake difficu lt problems, (d) enables students to find
27 effective and creative solutions to dilemmas, and (e) reduces the burden on memory and attention to detailÂ” (p. 1). Information processing theory explains the manner in which executive control (metacognitive) processes ma nage and regulate cognition, providing an explanation as to why some students are better than others at learning and remembering (Woolfolk, 1998 in Hartman, 2001, p. 33). Because metacognitive strategies facilitate the evaluation and application of knowledge to nove l situations, Â“metacognition is critical to cognitive effectivene ssÂ” (Gourgey, 2001, p.18). Efficacy can be impacted through self-regul ated practices as well as a studentÂ’s ability and past experiences. Students w ho actively engaged in their learning while applying control over goal se tting and goal attainment are re ported to be self-regulated learners (Schunk, 1989). When students expe rience acceptable results as they monitor progress towards learning goals, they are enco uraged to further improve their skills (Brown, 1999; Schunk, 1989). Three important regulatory skills appear in the literature regarding control over oneÂ’s cognitive activities: planning, monito ring, and evaluating (Jacobs & Paris, 1987; NCREL, 1995; Schraw, 2001). Pl anning refers to the selecti on of appropriate strategies and allocation of resources that affect perf ormance. An example would be apportioning time or attention selectively before beginning a task. Monitoring is defined as a personÂ’s awareness of comprehension and task performan ce. An example is the ability to engage in periodic self-testing while learning. Evaluation entail s judging the outcomes and efficiency of oneÂ’s learning. The re-exami nation of oneÂ’s goals and conclusions is a typical example (Schraw, 2001). Table 3 is an illustration of the phase structure and subprocesses of self-regulation.
28 Table 3 Phase Structure and Sub-Processes of Self-Regulation Cyclical Self-Regulatory Phases Forethought Performance Self-Reflection Task Analysis Self-Control Self-Judgment Goal Setting Self Instruction Self Evaluation Strategic Planning Imagery Causal Attribution Attention Focusing Task Strategies Self-Motivation Beliefs Self Observation Self-Reaction Self-Efficacy Self Monitoring Self Satisfaction/Affect Outcome Expectations Adaptive-Defensive Goal Orientation Intrinsic Interest Note From Enhancing self-monitoring du ring self-regulated learning of speech (p. 209), by D. Ellis and B.J. Zimmerman, 2001, In H.J. Hartman (Ed.), Me tacognition in Learning and Instruction, Theory, Research and Practice. Boston, MA: Kluwer. Self-regulated behavior, like metacognitive strategies, can improve studentsÂ’ success and ability to transfer learned skills to new situations th rough familiarity of problem meanings and strategic examination (Gourgey, 2001). Research by Wolters & Pint rich (2001), studied the di fference of motivation, selfregulated learning and classroom performan ce across subject matter areas and gender. The investigation concluded a relationship ex isted between knowledge of strategies to self-reported strategy use. Findings of the study revealed that, as a group, both genders reported similar use of regulatory strategi es in the varying subject areas, although
29 cognitive strategies were employed more often in social studies. Females responded with more favorable usage of cognitive strategies ac ross subjects than thei r male counterparts. Regarding self-efficacy, student s who indicated confidence in comprehension of course materials were more apt to also indicate they employed cognitive and self-regulatory strategies. Everson and Tobias (1998) conducted two se parate studies in order to develop an evaluation instrument of studentsÂ’ knowledge monitoring ability (KMA). The measure compared the differences between studentsÂ’ perception of their ow n knowledge within a specific domain and studentsÂ’ actual know ledge based upon outcomes of an objective performance measure. Summarizing the resu lts of both studies, the conclusion sustained the KMAÂ’s validity related to metacognitive knowledge monitoring and predictive ability in assessment. The findings reveal that while all students demonstrat ed an increase in vocabulary knowledge from preto post te sting, the more capable students showed greater increase in monitoring ability. Self-Monitoring. If the responsibility for learning lies with th e student, so too, should the responsibility for critical evaluation (Baron, 2003; Mitrovic, 2001). According to Baron (2003) from the University of Las Vegas Teaching and Learning Center, there are several reasons for using student self-a ssessments: (a) it promot es an attitude of inquiry in that students ha ve an active relationship to the material; (b) it provides opportunities for students to demonstrate relatio nships between course material and other experiences such as work, travel, and read ing; (c) it promotes consideration of the meaning and relevance of the material lear ned and tasks accomplis hed; (d) it empowers students to add their voices to the feedback they receive from their teachers; (e) it teaches
30 students to engage in a self-directed process; (f) it encourages reflect ive learning; (g) it provides students with an opportunity to co mbine quantitative and qualitative assessment of their learning; (h) it creates a shift in th e dynamics of teacher-st udent relationship (i.e. a power shift); and (i) it helps develop skills for life-long learning. Engaging students in self-directed activi ties also implies the involvement of the instructor. There are a number of benef its and challenges fo r both the learner and teacher. From the student perspective, they take control of and re sponsibility for their own learning with critical in tention, while the evaluation process is obvious for both parties. However, because students lack routine involvement in analyzing their own work with a critical eye, they may lack the confidence to do so. It may also be difficult for students to discuss academic problems or issues and they may be uncomfortable communicating their successes. For faculty, they will have to prepare differently by becoming familiar with the self-directed evaluative processes, defining their roles regarding feedback, and organizing thei r curricula accordingly (Baron, 2003). In order to integrate self-monitoring into the curriculum, the evaluative procedure must initiate with the learner. Zimmerm an (1989) illustrated the process of selfregulation in a triadic feedback loop in which self-monito ring of information is regularly processed among person, behavior a nd environment (Figure 3).
31 Figure 3 Triadic Form of Self-Regulation From Â“Enhancing self-monitoring during self-regulated learning of speech,Â” by D. Ellis and B.J. Zimmerman, 2001, In H.J. Hartman (Ed.), Metacognitio n in Learning and Instruction, Theory, Research and Practice, p. 207. Boston, MA: Kluwer. During the process of behavioral self-re gulation, self-monitori ng activities correct performance while monitoring environmental co nditions. Covert self-regulation refers to revising cognitive and affective states, as n ecessary. Self-monitoring of these conditions directly affects desired outcomes of learni ng strategies (Ellis & Zimmerman, 2001; Paris & Winograd, 1990). The growing body of eviden ce demonstrates that students, who take responsibility for their own learning, devel op deeper and more permanent knowledge and skills (Brockett & Hiemstra, 1991). In a study by Ellis (1994), 80 undergraduate students enrolled in a remediated speech class participated in an investigation of self-monitoring training. Initially students were requested to use a standard pronuncia tion of a word while reading a story aloud. Later, they listened to their voice recordings to evaluate if, indeed, they had used the
32 standard pronunciation. This data was used as a baseline measure of self-evaluative accuracy. Participants subsequently were assign ed to one of four experimental groups or the control group. Assignment included: Di scrimination with Self-monitoring training, Discrimination Only training, Self-monitori ng Only training, and Practice only (no training in either self-mon itoring or discrimination). After training, students we re given three times to pr actice sentences and then report responses of self-efficacy to the que stion Â“how sure they were that they could say the wordÂ” exercising the standard pronunciatio n. At the end of post testing, students reported if they felt they Â“hadÂ” used the standard pronunciation. Students were then tested on other words containing similar sounds to check for near and far transfer. Results clearly indicated the Discri mination plus Self-monitoring group ( M = 35.88, SD = 5.71) performed substantially better on the posttest scores at the p <.01 level. The Discrimination Only group ( M =14.63, SD =17.20), the Self-monitoring Only ( M = 13.06, SD = 16.28), the Practice Only ( M = 1.38, SD = 2.75) and the control group with no treatment ( M = .98, SD = 1.18) indicated significantly lower results. The researcher reported students traine d in only self-monitoring rated themselves lower on self-efficacy and self-evaluation th an the control and practice groups while students who received training on both di scrimination and self-monitoring ranked themselves much higher in these areas. Th ree important conclusions resulted from this study: 1) self-monitoring improved learning ou tcomes, 2) this group of college students lacked adequate self-regulatory skills to improve on their own, and 3) practice alone, by students, is not enough; intervention is needed to teach self-monitoring techniques. The
33 effect on students from this intervention was increased metacognitive awareness and enhanced self-efficacy. Metacognitive Awareness. Â“A strategy is defined as a conscious, deliberate use of a specific method, whereas a skill is define d as a refined strategy which is used selectively, automatically, and unconscious ly as neededÂ” (Hartman, 2001, p. 33). GarnerÂ’s (1990) theory of settings states se veral contextual factor s affect strategy use; lack of knowledge about th e relationship between stra tegy use and task demands, classroom settings that do not value the effort ful application of stra tegies, and learnerÂ’s who use Â“primitiveÂ” routines and demonstr ate inadequate cognitive monitoring (in Hartman, 2001). Metacognition functions to direct the cognitive processes such as thinking and remembering that take pl ace during learning (McCown & Roop, 1992). Two distinct constructs of metacognition ha ve been generally r ecognized; knowledge of cognition and regulation of cognition (B rown, 1987; Davidson & Sternberg, 1998; Flavell, 1987; Schraw, 2001). Research has sh own learnersÂ’ who have the ability to plan, monitor and evaluate their ow n learning are more strategic and perform better (Garner & Alexander, 1989; Hartman, 2001; Schraw, 1998, 2001). According to the North Central Re gional Educational Laboratory (1995), metacognition includes three fundamental components: 1. Developing a plan of action: Consider what prior knowledge has been brought to current task, what are the first steps to be taken, and how much time will it take. 2. Maintaining/monitoring the plan: Is the pl an on task? What information is needed to proceed? What other resources can be us ed to continue? Is the pace too fast, too slow, or adequate? 3. Evaluating the plan: Did the outcomes align with the plan of action? What went
34 well and what could have been undertaken differently? Could this plan be applied to other situations? (p. 1). Applying metacognitive strategies is particular ly critical to learning because it affects the acquisition, understanding, retention, and re levance of knowledge while impacting critical thinking, problem-solvi ng and learning effectiveness. Â“Reflective thinking is the essence of metacognitionÂ” (Hartman, 2001, p. xi). White and Frederiksen (1998) analyzed an instructional approach based on scientific inquiry deve loped to engage students, specifi cally targeting those of diverse backgrounds. The curriculum consisted of tw o dimensions: (a) a metacognitive model of research and (b) a metacognitive reflective process. Students in middle school physics classes were specifically instru cted on how to reflect and critique their own and other studentsÂ’ analyses while lear ning to build complex models of force and motion. The results from this research are consistent with other research findings (Case, Gunstone, & Lewis, 2001; King, 1991; Mevarech & Kramar ski, 2003; Schoenfeld, 1985) which found lower achieving students gained more benefits from the implementation of metacognition into the curriculum, actually closing the gap in performance with the higher achieving students. However, overall both high achie vers and low achievers showed increased improvement. Metacognition and Performance. Â“Metacognition is essential to successful learning because it enables individuals to better manage their c ognitive skills and to determine weaknesses that can be correct ed by constructing new cognitive skillsÂ” (Schraw, 2001, p. 13). Regarding academic success, metacognitions Â“are the kinds of knowledge and strategies that successful people tend to figure out for themselves and that
35 some people must be taughtÂ” (Hartman, 2001, p.33) Critically important, but frequently overlooked in learning, is that many times st udents possess the necessary knowledge and skills to tackle complex issues, but often do not make use of them. In other words, according to Hartman and Sternberg (1983) students may have Â“declarative and procedural knowledge, but not the contextual or conditional knowledge needed for application and transferÂ” (as cited in Hartman, 2001, p. 34). According to Hartman (2001), research re ports the following about metacognition: 1. High achieving students (HAS ) learn and remember more than others (Woolfolk, 1998) and are more metacognitive than low achieving students (LAS) (Sternberg, 1985). 2. (HAS) have been found to possess more metacognitive awareness and engage in more self-regulatory behavior than low achieving students. 3. Metacognition has been found to be an im portant characteristic of expertise. (Meichenbaum & Biemille r, 1998; Sternberg, 2001). 4. Demonstrated to be essential to lear ning: general strategic, metacognitive knowledge and strategies, and domain-speci fic knowledge have been shown to have important roles in thinking and pr oblem-solving (Bransford et al., 1986). Unless the student is able to employ self-regu lation, metacognition al one is not adequate for academic success. According to Case, Gunstone, and Le wis (2001) Â“enhanced and appropriate metacognitive abilities will onl y be achieved by means of an integrative perspective on metacognition, in which metacognitive traini ng is recognized to be intimately bound up in issues of content and cont extÂ” (p. 315). In a study by Mevarech and Kramarski (2003), an assimilated approach was used to ex amine differences among students who were informed of metacognitive training (MT) and those who used worked out examples
36 (WE). The research occurred over two academic years following a group of eighth grade students through ninth grade. A pretest was administered to all part icipants, followed by random assignment to either the MT or WE groups. Learning mate rials designed explicitly for each treatment were used to study the unit, ending with an immediate posttest of the material. The following year, all ninth grad e classrooms underwent the de layed posttest examination. Additionally, within the coope rative setting, each group was videotaped to observe and later analyze problem-solving behaviors. Re sults concluded significantly higher for the MT students than the WE students on th e immediate posttest. Dimensions of mathematical reasoning (verbal explanations algebraic representa tions and algebraic solutions) analysis confirmed statistically significant results for both lower and higher achievers in the treatment condition. King (1991) reported similar results for students who were trained in asking and answering metacognitive questions. They dem onstrated the ability to express conceptual understanding better and gave more explana tions to peers when presented with novel problems, than those who were not traine d. In a related study by Schoenfeld (1985), more than 100 hours of videotapes were revi ewed by researchers depicting the behaviors of high school and college student sÂ’ attempts at solving prob lems. He found more than 60% of studentsÂ’ problem-solving abilities were hampered by the tendency to jump into the problem quickly without fi rst attempting to ask questions and plan a solving strategy. Zhang and RiCharde (1998) performed a longitudinal study of outcomes on the dimensions of academic achievement and metacognitive development. The study followed university students ( N = 300) at a public institution from freshman year up to
37 graduation. The authors concluded metacogn itive development is fostered by academic achievement, indicating students with Â“good academic standing possess a stronger ability to reason, think, and make decisi ons about personal and social issues than their peers and that ability is central to metacognitive a nd intellectual developmentÂ” (Zhang & RiCharde, 1998, p. 15). A comparison of means throughout th e fours years of the study revealed the top 10% of the participants scored si gnificantly higher on l ogical reasoning and probability estimate than the middle 40% or the bottom 50%. Also the top 10% obtained significant differences on problem-solving approach than the bottom 50%. The study found the middle 40% of students scored si gnificantly higher on the measure of metacognition than the lower 50%. Their result s are in agreement with previous research by Flavell (1985). Interestingl y, this research found engine ering majors exhibited an increased level of metacognitive development over liberal arts students as measured by logical reasoning, probability estimate, and problem-solving approach, however, science students outperformed both engineering a nd liberal arts majors on problem-solving approach during the four-year period. Metacognitive Cues The maintenance and monitoring aspect of metacognitive regulation has been the focus of research on metacognitive prompts. Strategies are embedded as indicators in the instructional event to stimulate the learnerÂ’s conscious control over their own learning. There are many instructi onal methods that can be used to facilitate or strengthen a studentÂ’s use of cognitive strategies. Some of these teaching approaches include (a) procedural promptshaving the student pos e questions of who, what, where, when and how; (b) model res ponses for students; (c ) thinking aloud to summarize, thinking ahead or cl arifying difficult concepts; (d) guide student practice; and
38 (e) provide feedback and corrections (Lloyd, Kameanui, & Chard, 1997). The web-based learning environment in this study employed more than one instructional method. Specifically, guided st udent practice ha s been designed into the web-based instruction using modeled res ponses accessed through the Â“helpÂ” button. During the lesson, feedback and corrections were used during and following the problemsolving activity for the experimental group. Prior research has concluded feedback during the learning process is favored over learning without feedback (McDaniel & Fi sher, 1991; Zellermayer, Salomon, Globerson, & Givon, 1991). Feedback has been found to be an essential component of success in student-centered environments (McCown, Driscoll & Roop, 1996). In the study by Kramarski and Zeichner (2001), students who were exposed to two differing kinds of feedback in a computerized environment re sulted in significantly higher achievement outcomes for students receiving th e metacognitive feedback (M F) than students receiving result feedback (RF). A group of 186 eleventh grade students from eight classes in four schools were randomly assigned to either th e control or experimental condition. MF consisted of metacognitive questions (e.g. Â“wha t is this problem/ta sk all about?Â”) acting as cues for mathematical reasoning, whereas RF provided cues relevant only to the final answers (e.g. Â“check it once moreÂ” and Â“very good!Â”). StudentsÂ’ se lf-regulated learning was performed in a computer laboratory setti ng with a teacher present only for technical problems, not intervention with the learning se quence. Indications were reported from the researchers on the importance of metacogniti ve feedback embedded in a computerized learning environment on achievement and math ematical reasoning skills. Analytical results of the two research questions (achievement and mathematical explanations)
39 concluded: 1) significantly high er performance of the MF gr oup on the total scores of all measures: general term formul a, rule of recursion and ve rbal problems and 2) richer mathematical reasoning by the MF group usi ng verbal arguments(30.2%) more often than the RF group (20%). Similarly, the MF stude nts demonstrated frequency of use (63.5%) employing a combination of algebraic rules a nd verbal arguments more than RF students (31.6%). Watson and Allen (2002) studied the use of embedded metacognitive prompts or cues in a computer-based tutorial for 5thgrade students studying science concepts. The instructional sequence invol ved a 20-30 minute lesson, an announcement of a quiz followed by the actual examination. Two quizzes were administered over the course of the study. Both the control and experimental groups received the same instruction, the same quiz announcement, and the same quiz, in that order. However, the experimental group had access to metacognitive prompts directly after the quiz announcement. The prompts asked questions such as Â“Are you r eady? If you think you need to review, you can use the [navigational] b ack button to go back nowÂ”. Results indicated a significant differe nce in the two groups of students to accurately predict their own posttest performan ce. Nevertheless, there was no significant overall effect when measuring posttest compre hension of the combined score for the two embedded quizzes. The researchers were su rprised by these results and upon further examination tentatively concluded a more complex interaction of gender differences influenced the overall effect on compre hension. Posttest outcomes found an improvement among the female students while th ere was a decrease in posttest results for the male students.
40 Metacognitive prompts provide a mean s of coaching students to think about cognitive processes involved in problem-solvi ng, skills that are essential when transferred to other kinds of problems (K apa, 1999b). KapaÂ’s (2001) st udy considered appropriate timing for intervention of metacognitive reinforcement during problem-solving. The study explored the effect of metacognitive suppo rt introduced at differing intervals of the problem-solving process in a computerized le arning environment. Eighth-grade students were randomly assigned to groups for one of four different interv ention implementation phases: (a) during the solution process and after the completion of the problem-solving process, (b) during the problem-solving process, (c) at the end of the solution process, and (d) no metacognitive supports. The trea tment occurred over the course of a twomonth period while complete data gathering transpired over the entire academic year. Pretest scores exhibited no significant differences between the groups; however, a significant difference existed between student s with high or low prior knowledge. The three treatment groups of students with low previous knowledge were able to reduce the difference in problem-solving abilities be tween themselves and students with high previous knowledge in problem-solving. A study by Condor (2001) analyzed the diffe rences in metacognitive ability when comparing two groups using different com puter environments and the relationship between problem-solving ability and metacogni tive ability when solving statistical word problems. The study consisted of 120 comm unity-college students enrolled in a beginning-level statistics course. Students were randomly assigned to one of four groups (two groups for each of the two course sect ions): metacognitively-cued, computer-tool (MCCT) and metacognitively-cued, compute r-coached (MCCC). Performance was
41 measured on the outcomes of an instruct or-developed objective examination and responses to written metacognitive cues. Each of the two sections of the course was taught by separate instructors, one being the researcher, however, the same curri culum, guidelines and time schedules were adhered to by both teachers. The study lasted twelve weeks, the entire summer semester. Treatment began in week four and three uni t exams were included in this phase for analysis. Students from both groups (MCCT & MCCC) were directed to complete written metacognitive cues sheets in class whil e attempting to solve the word problems. The MCCT group had access to their textbook, class notes and the instructor whenever they required assistance. The comput er was utilized strictly as a tool to manipulate the problem-solving activities. The MCCC group also had access to their textbooks, class notes, and instru ctor although in this case in structors merely guided the student to where they could find explanations on how to solve the problem. This group also had unlimited access to a computer program on CD-ROM which allowed further discovery to problem solutions through in-depth explanati ons and examples, acting as a computer coach. Both groups were also administered an instrument to report how successful they felt their performance was on the in-class examinations. Measurement of studentsÂ’ test scores were compared to th e self-report measure checking for discrepancies of studentsÂ’ perceived ability to actual performance. The study resulted in slight differences of problem-solving ability between the two groups and a small to medium corre lation between metacognition and problemsolving. There was, however, a significant difference in academic performance between the two groups in their metacognitive awareness. By the fifth exam, the MCCC group
42 was outperforming the MCCT group. The author attributes this to a slow, but steady progression of the treatment effect. Severa l limitations to the study could impact the findings. First, a small sample size was used due to convenience sampling of the naturally occurring class sizes. Second, one of the instructors was also the researcher which may explain the differences in group scores. Additionally, homogeneity of the students may have negatively inhibited a stronger correlation between metacognitive ability and problem-solving abilities. Instrumentation In development of this studyÂ’s instrument, a literature search was initiated for inventories evaluating me tacognitive awareness. Over the years, a number of instruments have been desi gned for domain-general measurement of metacognition. The following discussion reviews t hose tools as they relate to the current research on the metacognitive strategies of planning, monitoring, and evaluation. Armour-Thomas and Haynes (1988) devel oped an instrument to evaluate a studentÂ’s metacognitive awareness in problem -solving called the Student Thinking About Problem-solving Scale (STAPSS). Their aim was to create a measurement Â“used to diagnose inefficiencies in metacognitive pro cessing and help to improve problem-solving skillsÂ” (p. 92). At the time, inventories to judge lower level cognitive abilities such as learning and study strategies ex isted but none to gauge high order thinking processes. In crafting the STAPSS instrument, it ems were generated based on problemsolving processes identified by Sternberg (1986). Three learning and cognition experts reviewed for content validity, narrowing the poo l to thirty-seven statements. Once piloted and revised, the STAPSS was administered to 172 students representing three high schools. The participants were categorized into groups based on achievement: (a) below
43 average (> 65), (b) average (66-84), and (c ) above average (85-100) established through report card grades for all subj ects and SAT scores. The ab ility of this inventory to classify the participants according to achie vement level indicated Â“modest predictive abilityÂ” (p. 92) with 58 percent accuracy. Fortunato, Hecht, Tittle, and Alvarez in 1991 set about to re-focus studentsÂ’ attention away from problem solutions to th e strategic cognitive act ivities necessary to solve problems. They developed an inst rument, called Â“How Do I Solve Problems (HISP)Â” to measure the way students worked a problem and the strategies a student might use. A sample of 165 seventh graders from twenty-three classes was asked to work an atypical coin problem and to fill out one of three questionnaires distributed randomly among the classes. The questionnaires consiste d of twenty-one statements divided into four sections: (a) planning, (b) monitoring, (c) evaluation, and (d) ways in which the problem was worked out. A three-point scale indicated student responses of Â“yes, no, or maybeÂ”. Interestingly, when students were as ked to solve routine problems, they were less cognitively aware of the strategies they used. The researchers felt the results were useful for creating classroom activities base d on the questionnaire responses. Forty percent of students indicated Â“yesÂ” to the statem ent, Â“I tried to remember if I had worked a problem like this beforeÂ”, however, another forty-two percent indicated Â“noÂ” to the use of this strategy. The findings from this st udy were also supported in the research by Hong et al. (2001), indicating problems should be challenging for students in order for metacognitive activities to be useful. Schraw and Dennison (1994) constructe d the Metacognitive Awareness Inventory (MAI) to facilitate the measurement of me tacognition without leng thy and cumbersome
44 interviews. Organized into eight subcompone nts and grouped into two larger categories of knowledge and regulation of cognition, the tool is directed at adult learners. Undergraduates at a Mi dwestern university ( N = 197) took part in experiment one of the study. Out of the original 120 items, a fifty-tw o item self-report inventory resulted in the two broad categories and six subcategor ies. Experiment two consisted of 110 participants, for the purpose of validati ng the instrument on three measures: (a) metacognitive knowledge, (b) test performan ce, and (c) metacognitive regulation. The conclusions provided support for both constructs of the metacognitive model; knowledge and regulation. The results confir med a high internal consistency, whereas the internal consistency on the factor s measuring multiple subcomponents of metacognition was marginal. Statistical significance was achieved between the relationship of knowledge and regulation. When the MAI was compared to perfor mance, significant relationships existed between pre-test self-assessm ent and monitoring ability, as well as pre-test assessment and test performance. However, the resu lts of monitoring accuracy and the MAI or monitoring accuracy and pre-test assessment di d not withstand testing for significance. Hong et al. (2001) followed the work of Fortunato et al. (1991) to develop an instrument which also measured the gene ral domain of metacognition. The study was conducted in two phases, the first measured variables of current techniques through a self-report. The second phase collected data with two existing inventories of metacognition and problem-solving. Initial ite m were sorted using reliability analysis, and factor analysis on the remaining items. Th eir research resulted in five constructs of
45 metacognition related to problem-solving: (a ) knowledge of cogniti on, (b) objectivity, (c) problem representation, (d) subtask monitoring, and (e) evaluation. The goal of the researchers was to devel op an instrument targeted at 12-18 yearolds within a classroom environment. A 32-item inventory was created to self-report metacognition. Participants ( N = 829) from across the United States tested the revised inventory resulting in a reliabl e instrument. The inventory (Appendix C) created for this research is modeled upon Â“How Do You Solv e ProblemsÂ” questionnaire by Hong et al. (2001) however; it is modified for domain specifi city and to meet the course objectives of the thermodynamics class in engineering. Wh ile the instrument has been customized to this course, the structure is general in na ture following previous research on factors considered important to metacognitive awar eness; planning, monito ring and evaluation. Further discussion of the instrument will follo w in Chapter Three, the Methods section. Chapter Summary This chapter began with a discussion of how Information Processing Theory set the underlying context to define and understa nd cognition with furthe r investigation of Social Cognitive Theory hypothesizing about control over oneÂ’s actions through intentional behaviors and st rategies (Bandura, 2001). A co mprehensive review of the literature supported the development of me tacognitive awareness as a process for students to develop problem-solving strate gies. Through the monitoring of their own progress towards previously outlined goals and completing the development of selfdirected activities, the responsib ility of learning is placed with the student. The chapter concluded with a retrospective look at seve ral instruments to measure metacognitive
46 activity which, have been developed for th e domain general constructs of planning, monitoring, and evaluation.
47 Chapter Three Research Methods As suggested by Livingston (1997), f aculty can help students understand problem-solving and cognitive goal setting by having them use metacognitive strategies to control their cognitive abilities. More res earch is needed in th e area of metacognitive cuing embedded in web-based instruction on learning outcomes and student selfperception of problem-solving ability. This research study investig ated the relationship between metacognitive strategies interventi on in a web-based instructional model with student achievement and perception of problem-solving. Through metacognitive prompting provided within the lessons, student s were asked to refl ect upon the processes used in solving problems and to rate their problem-solving ability. The chapter provides an overview of the research procedures, in cluding information about the participants, instructional methods, the instrume nts, and the research design. Participants The study took place at a Research I university in the southeastern United States. Approximately 40,000 enrolled students co mprise a diverse population of ethnic backgrounds represented by 11.1 percent African American, 9.8 percent Hispanic, 5.3 percent Asian, and 0.4 percent American India n. Students participati ng in this study were selected from an undergraduate engineering core course in thermodynamics which is prerequisite to subsequent cour ses in the engineering curriculum All students enrolled in
48 the course are engineering majors from vary ing engineering disciplines including civil, chemical, computer science, el ectrical, industrial, and mechan ical within the college. Participants were selected using a non-probability strategy of convenience sampling. Convenience sampling as descri bed by Tashakkori and Teddlie (1998) is a sampling selection technique undertaken due to accessibility of par ticipants such as a class of students. The thermodynamics course typically has 140 st udents enrolled; thus, the minimum criteria for adequate sample size will likely be met (Cohen, 1988). Participants were selected at end of the fi rst week of classes when the drop/add period was completed to minimize attrition. Ethical Considerations Use of this course and the students en rolled was approved by the Chair of the Chemical Engineering Department and the course instructor of record. The application to conduct research involving human participants was submitted to the Institutional Review Board (IRB) at this university and has b een approved following submission of the research proposal. Although quantitative data was used for this study, only aggregated data was reported in order to maintain the c onfidentiality and privacy of the participants. All data was kept in the locked office of the researcher. Risks to individuals were minimized, and students were not exposed to any undue discomfort or deception during or following the investigati on. Any provisions needed to comply with cultural or language barriers, physical or mental impair ments or other unforeseen factors were handled on an individual basis. Instructional Procedures Both Groups. The thermodynamics course met twice each week for a total of
49 three hours classroom lecture. The course format combined in-class lectures and out-ofclass homework problems. Mixing web-based instruction and classroom techniques can take advantage of the complementary strengths of each (Horton, 2000). The interactive web-based tutorial of homework problem sets was developed to provide individualized, immediate feedback through built-in assessments. Features of the tool include: (a) easily acce ssible interface through the web by students, (b) time-limited exercises not to exceed one hour in lengt h, (c) immediate feedback to facilitate knowledge of results and motivation, (d) onl ine help function to guide the student towards a correct response, (e ) links to additional course material for supplemental information, (f) creation of unique problems se ts to provide indivi dualized part icipation, and (g) integration with othe r tools such as Matlab to assist in solving complex engineering problems (Buck, 2004) Students from both the experimental and control groups who required more guidance on a partic ular problem or section of a problem accessed additional information through the aid of the Â“helpÂ” button which displayed a pop-up window. Control group. The control group followed the usua l course instructional format of encountering the problems within the we bsite, solving them, and submitting them for grading without any direct instruction on metacognitive strategies, cuing, or evaluation/reflection on the ut ility of metacognitive strate gies for problem-solving in engineering. Figure 4 is an example of an instructional screen a student in the control group encountered during one of the web-based problems sets.
50 Figure 4. Screen Shot of Instructional Frame for Problem Sets From Thermodynamics course website by West, J. (2004), http://thermodynamics.eng.usf.edu/index.html. Experimental Group Inst ruction and Materials. The metacognitive instruction for the treatment group had three elements: direct instruction, cuing, and reflection. Students in the experimental group began by re viewing a problem-solving model called Â“Engineering Problem-solvingÂ” (Appendix A) developed by Joseph (2004) to guide a student through the solution pro cess for engineering problems. Eight separate stages: (1) abstract abstraction, (2) list variables, (3) identify basis for calculation, (4) list assumptions, (5) list references, (6) develop model equations, (7) solve, and (8) interpret solution were defined to illustrate the manne r in which problems are solved (see Figure 5).
51 Figure 5 Screen Shot of the Engineering Problem-Solving Model From Engineering Problem-solving, by Joseph, B. (2004). Next, metacognitive cuing was integrated within each instructional screen designed for the web-based problems as reminders of how to solve engineering problems. At the end of the problem set, students were required to reflect upon the usefulness of metacognitive strategies for solving the co mpleted problems by indicating which of the eight stages were used during the problem-solvi ng process. In order to accomplish this, a parallel website was constructed for the e xperimental group containing the same problem sets; however, the second website introduced and highlighted metacognitive prompting within the instructi onal lessons. In this second website, metacognitive cues of the engineering problem-solving procedures were in serted within each instructional frame as
52 a prompting reminder to students while they practiced solving thermodynamics problems (see Figure 6). Figure 6 Screen Shot of Instructional Frame for Problem Sets with Embedded Cuing From Thermodynamics course website by West, J. (2004), http://thermodynamics.eng.usf.edu/index.html. An online help function for the experiment al group assisted in the scaffolding of studentsÂ’ learning through the use of met acognitive strategies embedded within the instructional framework. The term scaffoldi ng in used to describe the models, cues, prompts, hints or partial solutions that provi de links guiding students from what they can do by themselves toward what they can do w ith assistance from others (Hartman, 2001). Scaffolding is a particularly effective teaching method for improving higher level cognitive strategies (Rosenshin e & Meister, 1994). Prompts w ithin the Â“helpÂ” screen are
53 included in the early problem sets (1 & 2) a nd withdrawn from the later sets (instructional fading) as students became familiar with the eight step model of problem-solving (see Figure 7). Figure 7 Screen Shot of Metacognitive Cuing Within the Â“HelpÂ” Feature From Thermodynamics course website by West, J. (2004), http://thermodynamics.eng.usf.edu/index.html. At the end of a problem set, students in the metacognitive cuing group reflected on which of the eight steps they used duri ng the problem-solving process, checked the ones they used from a list provided, and submitted their responses electronically. An instructional screen containing the met acognitive reflection response section was presented to the student at the conclusion of the problem set (Figure 8). Both the cues, for evaluation, and the problem solutions, for r ecording of grades, were collected. 8 Steps to Engineering Problem-solving: 1. Abstract problem 2. List variables 3. State basis of calculations 4. Make/state assumptions 5. List references 6. Develop model equations 7. Solve equations 8. Interpret solutions, make conclusions. Help File For all engineering problems, both sides of the equation must have the same units, i.e. the equation is required to be dimensionally homogeneous. If the units are not the same, the equation is invalid. Thus, dimensional homogeneity may be used to determine if the equation is wrong. However, simply because dimensional homogeneity is achieved does not assure the correctness of the equation.
54 Figure 8. Screen Shot of Metacognitive Cue Reflection From Thermodynamics course website by West, J. (2004), http://thermodynamics.eng.usf.edu/index.html. Instruments Pretest. Multiple achievement instruments were used to explore the effect of metacognitive instruction on studentsÂ’ achie vement including a matched pretest and posttest as well as a comprehensive posttest fo r examining group differences in depth. In order to gather a baseline of studentsÂ’ prio r knowledge, a test (not a true pre-test) of thermodynamics concepts and skills was given at the beginning of the semester before the intervention implementation followed by a postte st at the end of the semester. Scores from this test were used to create matche d samples in pre-requisite knowledge and skills in thermodynamics.
55 The pretest consisted of seventeen qu estions developed studentsÂ’ knowledge and skills in: (1) the genera l understanding of the concepts of energy, how to measure it and the different forms of expression of energy in nature, (2) the concep t of conservation of energy which says that energy cannot be created or destroyed, but can be converted from one form to the other, and (3) the appro ach to analyzing engineering problems and application of basic knowle dge to answer simple engineering questions. Test items of the pretest consisted of objective multiple-choice and short answer formats (Appendix D). Possible responses were valued at one point each for a potential total score of 27. To inaugurate grading cons istency, all exams were scored by the same teaching assistant. The exam, given by pencil and paper, had been used and refined over multiple semesters. The content validity of this instrument was established through a survey of two professors with expert knowledge in chemical engineering. Content validity, as defined by Wiersma (2000, p. 300), is Â“the process of establishing the representative ness of the items with resp ect to the domain of skills, tasks, knowledge, and Â… whatever is being measuredÂ”. Posttest. A more comprehensive posttest of studentsÂ’ knowledge and skills in thermodynamics was used to compare the groups for statistical differences in their problem-solving skills. The major instructi onal units covered incl ude; the control volume analysis using energy, the second law of thermodynamics, and the use of entropy. For this measure, a composite posttest score was created by combining studentsÂ’ scores on exams three, four, and five administer ed in the latter two-thirds of the class. Aggregated scores were used, rather than the thermodynamics concepts pre-test and the twin posttest. The composite posttest inst rument measured specific course outcomes,
56 avoided administration bias of a test-retest procedure, and co ntrolled for history effects where participantsÂ’ responses may be infl uenced through new experiences and learning opportunities (Davis & Smith, 2005). Content va lidity, as explained by Gay and Airasian (2003), is the degree to which a test measures the represen tative content of a specific subject area when determined by experts in the field. The cont ent validity of the achievement tests was reviewed by two tenured professors from the Chemical Engineering Department who teach the thermodynamics course. They judged the items contained in the tests were congruent w ith the thirteen course outcomes of exit knowledge and skills required for students comp leting the course. Both the treatment and control groups were given the same exams. Wiersema (2000) suggests using several methods to establish consistency among the graders. Three procedures were used in this study to control for grader consistency: 1) use of a grading rubric (see Appendix E) created to restrict Â“driftÂ” during the exam assessments; 2) training of the teaching assist ants (TA) on the proper use of the grading rubric; and 3) random assignment of exams from both the control and experimental group to each TA. To standardize evaluation pro cedures, intrarater and interrater reliability measures were used. Each grader formed clusters of his or her exams consisting of strong, mid-range, and weak exams, and then three exams were randomly chosen from each TAÂ’s clusters for a total of nine exam inations. One week late r, each TA re-graded the nine randomly selected exams to verify in trarater reliability. Interrater reliability will be checked by having each TA re-grade the othe rÂ’s tests. Because the tests consist of
57 interval rating scales, re liability was assessed usi ng PearsonÂ’s product-moment correlation coefficient. Attitudes. The second hypothesis related to whether significant differences existed between the groups in their percepti ons of their problem-solving abilities in thermodynamics. A separate self-report instrument, How Do You Solve Problems? (HDYSP) (Zabel, 2004), was administered at the beginning and end of the course (Appendix B). The HDYSP was developed by modifying a metacognitive inventory, titled Inventory of Metacognitive Self-R egulation (IMSR) authored by Hong et al. (2001). The original instrument collected re sponses applied to ge neral problem-solving focusing on five independent factors (a) knowledge of cognition, (b) objectivity, (c) problem representation, (d) subtask monitoring, and (e) evaluation. The revised instrument, based on expert review, reflected specific problem solving processes used in engineering. Three self-regulatory constructs (planning, monitoring, and evaluating) were identified as important as metacognitive strategies related to problem-solving. Each section, defi ned separately for admi nistrative clarity, contained statements to rate self-report responses for a total of 32 items. The HDYSP was divided into three distinct dimensions of metacognition: (1) planning, (2) monitoring, and (3) evaluating. Part 1 Planning contained nine items pertai ning to the selection of appropriate strategies and alloca tion of resources that affect performance. Part 2 Â– There were eighteen items related to defining oneÂ’s awareness of comprehension and task performance in the Monitoring section. Part 3 Â– The Evaluating section was made up of five items concerning the appraisal of produc ts and efficiency of oneÂ’s learning.
58 The responses were formatted using a freque ncy base and a Likert -type scale ranging from 1= rarely to 4= almost always. The possible scores ranged from 32 to 128. The construct validity of the instrument was reviewed using two content experts in problem-solving strategies within the Chemical Engi neering Department and two professors in the College of Education knowledgeable in survey construction and instructional technology. Coefficient of relia bility was calculated using CronbachÂ’s alpha because it Â“provides a convenient way to es timate the lower bound of the coefficient of precision for a test by using item-response data obtained from a single administration of that testÂ” (Crocker & Algina, 1986, p. 122). Research Design The analysis of data for this st udy involved a mixed methodology framework congruent with the Tashakkori and TeddlieÂ’s (1 998) definition for mixed methods studies which Â“are those that combine the qualitat ive and quantitative approaches into the research methodology of a single study or mu ltiphase study (p.17).Â” According to Onwuegbuzie and Teddlie (2003), the reason for th is type of data analyses is two-fold; representation and legitimation. The former is to cull sufficient information from the data while the latter is conducted w ith a concern for validity. Th e design approach, as defined by the principles of mixed methods, combined both quantitative and qualitative research strategies in a simultaneous approach (Quan + qual). Morse (2003) explains the use of more than one technique presents a Â“more complete picture of human behavior and experienceÂ” (p. 189). In order to obtain a power level of .8 at the .05 level of significance for the analysis, thirty-one participants were needed for both the control and the treatment
59 groups. Based upon enrollment records from previous classes, 140 students were expected to participate in this study. Establishing Comparable Groups At the beginning of the semester, students completed an achievement pretest of thermodyna mics concepts and skills and an attitude questionnaire to measure their perceptions of their problem-solving ability. Students were ranked according to their pretest achievement scores and then assigned in order to one of two treatments (e.g. highest score = trea tment 1; next highest = treatment 2; next down = treatment 2; next down = treatment 1, and so forth down the spiral pattern). After group assignment, a t test was run on pretest scores to verify the initial comparability on achievement for the two groups Table 4, below, is an example of ranking assignment based upon group matching according to their prior achievement. Table 4 Group Assignment Ranking Pretest Exam Score Group Assignment 100 Group 1 99 Group 2 98 Group 2 97 Group 1 96 Group 1 95 Group 2 94 Group 2 93 Group 1 92 Group 1 91 Group 2 90 Group 2 89 Group 1 88 Group 1 87 Group 2 Continued Â…
60 Post Treatment Data Analysis. Materials and procedures in this section are organized using the hypotheses for the study. Two research hypotheses and four qualitative questions were analyzed in this study: 1. Was there a difference of posttest ach ievement between students who received direct instruction using metacognitive st rategies and embedded cues in their thermodynamics problem sets and student s who did not receive instruction in metacognitive strategies information and cuing? 2. Was there a difference in perceptions of their thermodynamics problem-solving abilities between students who received di rect instruction in using metacognitive strategies and embedded cues in their probl em sets and students who did not receive instruction on metacognitive stra tegies information and cuing? 3. What were the differences in thermodynamics knowledge between students who received direct inst ruction using metacognitive strategi es and embedded cues in their thermodynamics problem sets and student who did not received instruction in metacognitive strategies information and cuing? 4. Which ones of the problem-solving step s did students report using across the problem sets? 5. What were the characteristics of the students in the sample? 6. What were the participantsÂ’ percep tions of the web-based problem sets? All data collection was c onducted by this researcher. Table 5 represents the schedule of instrument administration fo r pre-treatment, treatment and posttreatment phases of the experiment.
61 Table 5 Data Collection Before, Duri ng and After Treatment Phases Pre-Treatment Treatment Phase Post-Treatment Phase Phase Phase Week 1 Informed Consent Biographical Questionnaire Pretest: Thermodynamics concepts/skills Pre-survey: HDYSP Spiral Group Assignment Week 7 Performance Analysis -Exam 3 Week 10 Performance Analysis -Exam 4 Week 13 Performance Analysis -Exam 5 Week 15 Composite Performance Analysis-Exams 3, 4, 5 Posttest: Thermo Concepts Post survey: HDYSP Both groups will receive the same problem sets, however only the treatment group will receive direct instruct ion in using metacognitive st rategies and cuing to solve problems. A t-test design was chosen to co mpare the composite pos ttest scores and the posttest of student attitudes towards problem-solving between the control and treatment groups. In order to analyze any observed differences in problem-solving skills, a .05
62 level was used to establish statistical signi ficance. Correlation anal yses by group and all were performed to report, (a) perception of problem-solving and achievement grade, and (b) preand postattitudes. Qualitative Questions Matching Posttest for Pretest. What are the differences in skills among students as measured by this test? The pretest of achievement, described previously, will be administered again as a posttest at the end of the semester in orde r to graph studentsÂ’ growth in the particular skills included on that test. This posttest was used in comparing groups statistically since it l acks adequate comprehensiveness, i.e., few items, and is potentially compromised by the same items being experienced by students on the prior administration. Descriptive statistics depict reported results of eac h groupÂ’s achievement scores which are graphed and subsequently discussed. Reflections on Procedures Used Which ones of the problem-solving steps do students report using across the problem sets? At the end of each problem set within the instruction, students were asked to reflect on the steps they used in the problem-solving process and report their process when submitti ng their score of the exercises. An item analysis of this data using descriptiv e statistics was repor ted through graphical representation. Biographical Questionnaire. What are the characteristics of the students in the sample? Demographic information was collected through a participan t survey (Appendix C) given in the beginning of the semester. Students were asked to indicate age, race, major, status as a student (part-time vs. full-time), year of study, current GPA, and
63 residency declaration. Responses were used to portray an accurate description of the student sample. Attitude Survey. What are the participantsÂ’ percep tions of the web-based problem sets? A survey, developed by the academic department, was administered on paper to participants for their subjective impressions when using the web-based problem-solving tutorials during the last week of classes. The survey consis ted of thirty-four statements (see Appendix G) regarding the web-based problem-solving tutorial. Five of the statements specifically related to familiarity and use of problem-solving strategies. Descriptive statistics were used to report means for each of the five statements. Chapter Summary The chapter presented the approach to this study. Methodology of the research discussed herein included the nature in which the study was conducted (mixed-method design), the data collected vi a various instruments (preand posttest of thermodynamics concepts and skills, HDYSP, composite perfor mance analysis) and interpretation of data through t-tests, correlations, and descriptive st atistics. The research questions have been addressed within the context of the e xperimental procedures for the study.
64 Chapter Four Results Introduction Several research questions were posed that were associ ated with dimensions of achievement and attitude, metacognitive refl ection, characteristics of the participant sample, and their relationship to performance as measured by learning assessments. This chapter is organized to present the analyses of data and findings relative to each research question. Participants. The potential sample consiste d of 113 students enrolled in thermodynamics, a core course in the engine ering undergraduate curriculum. However, 32 participants were eliminated based upon their decision not to take part in the research. This left an effective sample size of 81 tota l, with assignments to the experimental group ( n = 39) and the comparison group ( n = 42). Overall, the pop ulation was 65 percent male, 51 percent white (non-Hispanic), and 64 percent indicated that English was their native language. The age of the participants ranged from 19 to 50, with a mean of 29 years. For more details related to the dem ographics of the control and treatment groups, see Question 5 on page 76. As stated in Chapter One, typically 25 percent student drop-out rates are reported in thermodynamics courses in any given semest er at this college. By comparison, the enrollments for this class declined by 39 percent throughout the semester.
65 Establishing Comparable Groups Achievement Pretest. A pretest was used to determine whether differences existed in thermodynamics background knowledge between the control and experimental groups at the outset of the study. An achievement pr etest of thermodynamics concepts and skills measured studentsÂ’ ability. When a t test was performed on pr etest scores to confirm initial comparability on achievement between the two groups (see Tabl e 6), no significant differences were observed ( p =.28), using a .05 alpha level. Table 6 Thermodynamics Knowledge Pretest Results ControlTreatment Mean 11.7112.59 Standard Deviation 3.803.49 Sample Variance 14.4512.20 Kurtosis 0.19-0.54 Skewness -0.340.16 Range 18.0014.00 Observations 4239 Hypothesized Mean Difference 0 Df 79 t Stat -1.08 P(T<=t) two-tail 0.28 t Critical two-tail 1.99
66 Attitude Pretest. Attitudes were assessed using a su rvey instrument titled Â“How Do You Solve ProblemsÂ” (HDYSP), which was administered to both groups as a pretest. Obtained reliability coefficien ts for each section, planning ( =.53), monitoring ( =.83) and evaluating ( =.72) were computed using Cronbach alphas. Each dimension of the HDYSP varied in the number of survey items, therefore a Spearman-Brown Prophecy coefficient analysis was run for equalizati on comparison. This test predicts how the reliability coefficients would compare if the number of items in each dimension were equivalent. Predicted r values, when equated to items size of the monitoring section, resulted in modified coefficients for planning ( r =.69) and evaluating ( r =.90). Testing for group equivalence of attit ude revealed a significant diffe rence between means of the two groups ( t (79) = -2.19, p =.03), with a higher mean for the treatment group ( M = 78.98, SD = 28.02) than the control group ( M = 63.55, SD = 34.67).
67 Table 7 Attitude Pretest Results ControlTreatment Mean 63.5578.98 Standard Deviation 34.6728.02 Sample Variance 1201.86785.24 Kurtosis -0.513.34 Skewness -0.96-1.81 Range 106119 Observations 4239 Hypothesized Mean Difference 0 df 79 t Stat -2.19 P(T<=t) two-tail 0.03 t Critical two-tail 1.99 Post Treatment Data Analysis Further analyses are discussed by addre ssing each question separately. An alpha level of .05 was used for all statistical analyses. Question 1. Was there a difference in achievement between students who received direct inst ruction using metacognitive strategi es and embedded cues in their thermodynamics problem sets and student s who did not receive instruction in metacognitive strategies information and cuing?
68 A composite of the scores for exams 3, 4, and 5 were used to measure student achievement. These exams were chosen because they were given mid-way through the semester, allowing the treatment time to take effect. These exams were graded by two graduate assistants assigned to the course Therefore, prior to the analysis of group differences for the achievement scores, both interrater and intrarater reliability were examined. StudentsÂ’ test papers were ra ndomly assigned to each grader, forming two groups. Graders were unaware of (blind to) wh ether the studentsÂ’ papers were from the experimental or control group. After each of the three exam administrations, the graders selected from their exams one strong, one mid-range, and one weak exam for a total of nine exams per grader. The selected exams were then given to the other grader for scoring. One week later the graders re-graded their own exams. Pearson correlation analyses of the original scores and re -graded scores indicated strong positive relationships for intrarater ( r = 1.00) and interrater ( r = .99) reliability as shown in the following table.
69 Table 8 Intrarater-Interrater Reliability Correlation Matrix Original Grader A Re-grade Grader A Re-grade Grader B Original Grader B Re-grade Grader B Re-grade Grader A Exam 3 Strong 100 100 100 100 100 100 Midrange 67 66 66 75 75 73 Weak 39 39 42 39 39 39 Exam 4 Strong 100 100 100 100 100 100 Midrange 63 62 62 81 81 81 Weak 30 30 30 64 64 66 Exam 5 Strong 100 100 100 100 100 100 Midrange 82 82 84 70 70 67 Weak 67 67 67 32 32 34 Correlation 1 0.99 1 0.99 The achievement score used in the analys is consisted of a composite score from the three comprehensive exams. When the data was analyzed using a t test, no significant differences (see Table 9) existed between the control and treatment groups, suggesting
70 that one group cannot report doing better than the other when me tacognitive cuing is embedded within the problem sets ( p = .96). Table 9 Achievement Composite Score Control Treatment Mean 191.10190.47 Standard Deviation 57.9161.68 Variance 3353.083804.12 Kurtosis 0.180.20 Skewness 0.05-0.55 Range 264266 Observations 4239 Hypothesized Mean Difference 0 df 79 t Stat 0.05 P(T<=t) two-tail 0.96 t Critical two-tail 1.99 Question 2. Was there a difference in percep tions of their thermodynamics problem-solving abilities between students w ho received direct in struction in using metacognitive strategies and embedded cues in their problem sets and students who did not receive instruction on metacognitive strate gies information and cuing? A pretest, discussed earlier, was used to determine initial differences in the groups when measuring attitudes towards problem-solving. There wa s a significant difference between means for
71 the control group ( M = 63.55) and for the treatment group ( M = 78.97) when a t test was performed on the data (refer to Table 7). Pretest to posttest responses were analyzed separately for each group. Results from the control group indicated no significant difference in attitudes ( t (82) = -1.65, p = .10), when measured using a two-tail t test. Similarly, a comparison of the treatment gr oupÂ’s posttest results on attitude towards problem-solving abilities also indicated (T able 10) no significant difference from the pretest data ( t (76) =.60, p =.55). Table 10 Attitude Pretest and Posttest Results Control Treatment PretestPosttestPretestPosttest Mean 63.5575.9378.9774.51 Standard Deviation 34.6733.9328.0236.82 Sample Variance 1201.861151.58785.241355.94 Kurtosis -0.511.283.340.56 Skewness -0.96-1.49-1.81-1.41 Range 106120119116 Observations 42423939 Hypothesized Mean Difference 00 df 8276 t Stat -1.650.60 P(T<=t) two-tail 0.100.55 t Critical two-tail 1.99 1.99
72 Because of the significant differences in the pretest attitudes, an ANCOVA was conducted to examine the posttest data. After re moving the effect of th e pretest covariate, no significant differences were shown between the two groups ( p =.75). Means and standard deviations for the cont rol and treatment groups were ( M = 75.93, SD = 33.93 and M = 74.51, SD = 36.82), respectively (refer to Table 11). Table 11 Analysis of Covariance for At titude Performance Scores Effect SS Df MS F p Intercept 72505.40 1 72505.40 58.06 0.00 Pretest 131.61 1 131.61 0.11 0.75 Error 98649.45 79 1248.73 Question 3. Were there differences in thermodynamics knowledge between students who received direct instruction using metacognitive strategies and embedded cues in their thermodynamics problem sets and students who did not receive instruction in metacognitive strategies information and cuing? The pretest and subsequent posttest of thermodynamics skills and concepts were used to descriptively compare the groups. As was assumed, both groups significantly increa sed pretest to posttest. The evidence, (see Table 12), demonstrates a significant difference within groups comparison of the preand posttest administ rations using a paired t test for the control group ( t (75) = -4.18, p < .001) and the treatment group ( t (71) = -3.64, p < .001).
73 Table 12 Thermodynamics Knowledge Pretest and Posttest Results Control Treatment PretestPosttestPretestPosttest Mean 11.7115.5112.5915.65 Standard Deviation 3.804.163.493.68 Sample Variance 14.4517.3212.2013.51 Kurtosis 0.190.35-0.54-1.02 Skewness -0.34-0.810.160.21 Range 18181413 Observations 42353934 Hypothesized Mean Difference 00 df 7571 t Stat -4.18-3.64 P(T<=t) two-tail 0.000.00 t Critical two-tail 1.991.99 An ANCOVA was conducted to examine the posttest data after removing for the effect of pretest performance. The resu lts, reported in Table 13, demonstrates no significant difference between the control and treatment groups ( p = .70)
74 Table 13 Analysis of Covariance for Pretest and Posttest Achievement Performance Scores Effect SS Df MS F p Intercept 433.97 1 433.97 35.53 0.00 Pretest 228.46 1 228.46 18.71 0.00 Group 1.93 1 1.93 0.16 0.70 Error 806.05 66 12.21 Question 4. Which of the problem-solving step s did students report using across the problem sets? At the c onclusion of the six problem sets, participants in the experimental group were asked to reflect upon which, if any, of the eight steps to engineering problem-solving they used dur ing the solution phase of the problem exercises. The frequency distribution for the metacognitive reflection responses are shown in Table 14. For any one of the eight categories, there were a total of 234 possible responses. The two most freque ntly reported responses were Step Two List Variables (91 %) and Step Seven Solved Equations (91%). The least chosen response was S tep Four Â– Made/stated Assumptions which was selected only three percent of the time. Another item, selected less than 50 percent of the time, was Step Five Â– Listed References (30%).
75 Table 14 Frequency Distribution for Metacognitive Reflection Responses Categories Response Frequencies Possible # of Responses Frequency Percentages 1. Abstract Problem 165 234 70 % 2. List Variables 214 234 91 % 3. State Basis of Calculations 144 234 62 % 4. Made/stated Assumptions 6 234 3 % 5. Listed References 72 234 30 % 6. Developed Model Equations 167 234 71 % 7. Solved Equations 214 234 91 % 8. Interpret Solutions 143 234 61 % Total 1125 1872 Question 5. What are the characteristics of th e students in the sample? Gibbons (2004) provides descriptive information on the engineering stude nt population in the United States. Responses targ eted at obtaining the same in formation as Gibbons, reveals a near identical pattern, where composition wa s mostly white (non-Hispanic), male, and in their twenties. More than 90 percent of the participants listed Florida as their primary residence, closely split be tween the control (45%) and the treatment (47%) groups. Slightly more recorded Junior as their Year of Study in th e control group (33%) than in the treatment group (27%) and entered the unive rsity as Freshman w ith 27 percent and 31 percent, respectively. All academic departments in this college were represented with the exception of the Computer Science and Engineering department. Mechanical Engineering was listed more often than the other declared majors as indicated by 25
76 percent of the control group and 19 percen t of the treatment group. Following behind were the Civil and Environmental (control = 8%, treatment = 14%) and the Chemical Engineering departments (control = 13%, treatm ent = 8%). The Electrical, Industrial, and General Engineering responses were le ss than 10 percent for both the control and treatments groups as seen in Table 15. Of the participants who completed the survey, 23 percent control and 30 percen t treatment answered they were taking 13 to 18 Credit Hours this semester while 22 percent of the control group and 19 percent of the treatment group were taking 9 to 12 Credit Hours this seme ster. At the far ends of the response scale, 2 percent of only the treatment group responded to taking 0 to 3 Credit Hours or More Than 18 Credit Hours this semester. More of the treatment group answered to working Part-time (22%) or Not Working (20 %) outside of class th an the control group who answered working Part-time (20%) or Not Working (17%) outside of class. However, the control group re ported a larger percentage working Full-time (11%) than treatment group (9%). The highest self-re ported GPA in the 3.5 to 4.0 range was 20 percent by the treatment group and 14 per cent by the control group. Both groups indicated that three pe rcent of them were in the lowe st GPA range of 2.0 to 2.4. When asked How Many Times Have You Take a Th ermodynamics Course?, the control group answered First Time (44%), Second Time (3%) and Other (2%) while the treatment group answered First Time (42%), Second Ti me (6%) and Other as (3%) The following data were self-reported responses by partic ipants to the biogra phical questionnaire.
77 Table 15 Group Comparison by Biographical Dimension Control Treatment Dimension TotalFrequencyFrequency % Frequency Frequency % Gender Male Female 64 25 6 39% 9% 28 5 44% 8% Ethnicity African American Asian/Pacific Islander American Indian Hispanic White/Non-Hispanic Other 64 2 2 0 2 22 3 3% 3% 0 3% 34% 5% 0 6 0 4 19 4 0 9% 0 6% 30% 6% Native Language English Spanish Other 64 26 1 4 41% 2% 6% 26 2 5 41% 3% 8% Residency Florida Out of State Out of Country 64 29 1 1 45% 2% 2% 30 0 3 47% 0 5%
78 Year of Study Freshman Sophomore Junior Senior 64 0 8 21 2 0 13% 33% 3% 0 11 17 5 0 17% 27% 8% When Did You Enter the University? Freshman Transfer From a Community College Transfer From Another University 64 17 9 5 27% 14% 8% 20 10 3 31% 16% 5% Declared Major Chemical Engineering Civil/Environmental Engineering Computer Science & Engineering Electrical Engineering Industrial & Management Engineering Mechanical Engineering 64 8 5 0 0 2 16 0 0 13% 8% 0 0 3% 25% 0 0 5 9 0 1 3 12 2 1 8% 14% 0 2% 5% 19% 3% 2%
79 Engineering Undecided # of Credit Hours Taken This Semester 0 Â– 3 4 Â– 8 9 Â– 12 13 Â– 18 More than 18 64 0 2 14 15 0 0 3% 22% 23% 0 1 0 12 19 1 2% 0 19% 30% 2% # of Hours Working Outside of Class Part-time Full-time I do not work at this time 64 13 7 11 20% 11% 17% 14 6 13 22% 9% 20% Current GPA 2.0 Â– 2.4 2.5 Â– 2.9 3.0 Â– 3.4 3.5 4.0 64 2 9 11 9 3% 14% 17% 14% 2 6 12 13 3% 9% 19% 20% Times Taken a Thermodynamics Class? 64
80 First Time Second Time Other 28 2 1 44% 3% 2% 27 4 2 42% 6% 3% Note Number of blank responses to questionnaire: Control = 12, Treatment = 5. Question 6. What were the participantsÂ’ per ceptions of the web-based problem sets? Data were collected at the end of the study to analyze participantsÂ’ attitudes of the web-based problem sets. There were 61 pa rticipants who completed the survey, a 74 percent response rate. Five-point Likert-t ype rating scales were used with answers including: 1 (strongly disagree) to 3 (neutral) to 5 (strongly ag ree). Five of the statements from the survey, numbers 29 through 33, focu sed on perceptions of web-based problemsolving strategies, therefore only the statemen ts directly related to this study will be discussed. Statement 29. I am familiar with the genera l problem-solving stra tegies. Both control and treatment means ( M = 4.07) were the same, indicating the groups equally agreed to their level of general familiarity with problem-solving strategies. Statement 30. I used general problem-solving strate gies when doing these problems. The mean for the control groupÂ’s responses were M = 3.9 while the mean for the treatment groupÂ’s responses were slightly higher at M = 4.0. Both groups appeared to agree with the statement. Statement 31. I have not had a formal introducti on to engineering problem-solving Means for the control and treatment groups, ( M = 2.5, M = 2.63) respectively, were in the mid-range between neutral and disagree responses. Statement 32. I think an introduction to general problem-sol ving strategies would have helped me. While the mean for the control groupÂ’s responses was M = 3.0 indicating a neutral response the mean for the treatment group was slightly lower at M = 2.67. Statement 33. I am learning problem-solvi ng strategies through example, but it would be helpful to formalize it. Means for the groupsÂ’ responses regarding whether formalizing the problem-solving process would be useful were M = 3.33 for the control group and M = 3.27 for the treatment group.
81 The entire instrument Survey of Students Using Web-Based Problem-Solving Tutorials is available in Appendix G. The following table (16) summarizes the results from the survey. Table 16 Means for Survey Responses of Students Using Web-Based Problem-Solving Tutorials Statement Control Treatment 29. I am familiar with the general problem-solving strategies 4.07 4.07 30. I used general problem-solving strategies when doing these problems 3.9 4 31. I have not had a formal introduction to engineering problem-solving 2.5 2.63 32 I think an introduction to general problem-solving strategies would have helped me 3 2.67 33. I am learning problem-solving strategies through example, but it would be helpful to formalize it 3.33 3.27 Chapter Summary The research study investigated the e ffect of metacognitive cuing on problemsolving ability. No significant differences were found in achievement or attitude of problem-solving abilities between the two groups. Several r easons are offered for these results: 1. Other studies have found st atistically significant resu lts in achievement when longer experimentation was used. St atistically significant resulted when participants were followed for more than one semester, two academic years in one study while the other study longitudinally looked at students from freshman through graduation. The current study o ccurred over one academic semester. 2. No differences in perception of problem-s olving abilities were discerned because students perceived their problem-solvi ng abilities as highly developed. 3. The same questions were used to co llect preand posttest data for both administrations of the thermodynamics know ledge instrument. Changes in scores
82 can occur simply because the test has been repeated. Practice effect is a possible threat to internal validity. 4. Participants were asked to record upo n which, if any, of the eight steps to engineering problem-solving they used while working the problem sets. Qualitative reflection Â– why participants made certain choices Â– was not captured resulting in only frequenc y data for the responses. An ancillary finding resulted from developm ent of an effective grading rubric. In the analyses, strong relationships were found fo r intraand interrate r reliabilities of the instrument. In previous semesters of the thermodynamics course, a structured grading method did not exist resulting in differences in the scoring of exams. Informal feedback from the graders was very positive regarding the use of rubrics in future offerings of the course.
83 Chapter Five Discussion Introduction This research study investigated the e ffect of using metacognitive strategies through instructional cuing on problem-solving abilities and percepti ons of abilities in web-based learning. The chapter is a disc ussion summarizing the significance of the study results. Included herein are limitations wi thin the study, conclusions and recommendations for further study, and implications for practice. Discussion of Results Question 1. The purpose of the first research question was to determine if differences existed -as indi cated by achievement on a composite exam score -between students who received direct instruction in us ing metacognitive strate gies with web-based embedded cuing and those students who did not receive the instructi on and cuing. From the analysis of this question, statistically significant differe nces in achievement from the cuing of metacognitive strate gies were not attained ( p = .96). These findings are not consistent with results from previous resear ch. In several previous studies, use of metacognitive strategies practiced over time positively influenced problem-solving abilities (Ellis, 1984; Kramarski & Zeichner, 2001; Swanson, 1990). Results from the following three studie s lend support to slow, yet significant, cognitive maturation of metacognitive stra tegy use. Mevarech and Kramarski (2003) reported significant differences between st udents receiving metac ognitive training and
84 students practicing worked-out examples wit hout the metacognitive training. In their study, the participants were followed over two academic years; whereas participation in the present study occurred over one academic semester. After pretest administration to bot h groups, the Mevarech and Kramarski participants were randomly assigned to ei ther the metacognitively-trained (MT) or worked-out examples (WE) in cooperative groups. Students in both groups had been using cooperative learning since the seventh grade. The WE group was given worked-out examples and then practice problems. St udents had the opportunity to explain the material with the members of the group. The MT group was not given the worked-out examples, but was instructed to use a me tacognitive questioning procedure while they solved the practice problems. At the conclu sion of the instructiona l unit, students took an immediate posttest of the material. The next academic year the same students, as ninth-graders, were given a delayed posttest examination. Results were significantly higher for the MT students than the WE students on both the immediate and delayed posttests results. Dimensions of mathemati cal reasoning (verbal explanations, algebraic representations and algebraic solutions) analys is confirmed statistically significant results for both lower and higher achievers in the treatment condition. Two other studies found similar results as the Mevarech and Kramarski study of metacognitive knowledge developing over time. In the first of the two studies, Zhang and RiCharde (1998) conducte d a longitudinal investigati on tracking metacognitive and intellectual development of undergraduates fr om their freshman year to graduation. Students were measured three times: as incoming freshman, at completion of the sophomore year, and prior to graduation. The researchers concluded metacognitive
85 development: 1) has an irregular pattern duri ng the undergraduate year s; 2) fluctuates by academic discipline with engineering student s outperforming liberal arts majors and science majors outperforming both engineer ing and liberal arts majors on problemsolving ability; 3) is influenced by academic achievement; 4) plays a role in studentsÂ’ increased confidence as their perspective ch anges from absolute thinking to a broader understanding of events ; 5) differs with personality type. The Zhang and RiCharde study differed from the current study in: sample size over three times as many participants; gender only male students were included; academic training Â– three different disciplines were included; and le ngth of the study a f our-year longitudinal investigation. In the third study, Condor (2001) recommended extendi ng the time of treatment exposure based upon his research results. He analyzed the differences in metacognitive ability when comparing two groups using di fferent computer environments and the relationship between problemsolving ability and metacognitive ability when solving statistical word problems. The study c onsisted of 120 commun ity-college students enrolled in a beginning-level statistics c ourse. Students were randomly assigned to one of four groups (two groups for each of the two course sections): metacognitively-cued, computer-tool (MCCT) and metacognitively-cued, computer-coached (MCCC). Performance was measured on the outcomes of an instructo r-developed objective examination and responses to written metacognitive cues. Each of the two sections of the course was taught by separate instructors, one being the researcher, however, the same curri culum, guidelines and time schedules were adhered to by both teachers. The study lasted twelve weeks, the entire summer semester.
86 Treatment began in week four and three uni t exams were included in this phase for analysis. Students from both groups (MCCT & MCCC) were directed to complete written metacognitive cues sheets in class whil e attempting to solve the word problems. The MCCT group had access to their textbook, class notes and the instructor whenever they required assistance. The comput er was utilized strictly as a tool to manipulate the problem-solving activities. The MCCC group also had access to their textbooks, class notes, and instru ctor although in this case in structors merely guided the student to where they could find explanations on how to solve the problem. This group also had unlimited access to a computer program on CD-ROM which allowed further discovery to problem solutions through in-depth explanati ons and examples, acting as a computer coach. Both groups were also administered an instrument to report how successful they felt their performance was on the in-class examinations. Measurement of studentsÂ’ test scores were compared to th e self-report measure checking for discrepancies of studentsÂ’ perceived ability to actual performance. The study resulted in slight differences of problem-solving ability between the two groups and a small to medium corre lation between metacognition and problemsolving. There was, however, a significant difference in academic performance between the two groups in their metacognitive awareness. By the fifth exam, the MCCC group was outperforming the MCCT group. The author attributes this to a slow, but steady progression of the treatment effect. CondorÂ’s study was similar to the current research in a number of ways. First, convenience sampling of naturally occurring cl asses was used for both studies. Second, the sample size was limited by enrollments within the course. Third, the length of the
87 experiment spanned one semester. CondorÂ’s research differed from the present study because he used more than one instructor fo r the course, two types of computer programs were used (MCCT and MCCC), and reflect ion from the metacognitive cuing was completed on paper rather than with in the computer instruction. An ancillary finding, resulting from Question 1, was the effective development of an effective grading rubric for the thermodynami cs course. In previ ous semesters of this course, a structured method of grading wa s not implemented and differences in the scoring of tests were experienced among the graders. The TAÂ’s, act ing as subject matter experts, developed the rubrics for exams 3, 4 an d 5. An analysis for intraand interrater reliability coefficients found strong intrarater ( r = 1.00) and interrater ( r = .99) relationships for the instruments. Informal feedback from the graders was extremely positive regarding the use of grading rubrics. Their perception of the grading consistency resulted in personal confidence when scoring student exams. This confidence extended their ability to reduce scoring discrepancies when using the obj ective evaluation methods. The graders expressed a desire to use th e grading rubric model in fu ture engineering courses. Question 2. The intent of the second research question was to investigate if differences existed in the attitudes towards thermodynami cs problem-solving abilities between the group who received metacognitive strategies instruction and cuing and the group who did not receive the treatment. Result s did not indicate a change in perceptions of the participantsÂ’ problem-solving ability from pretest to pos ttest on the attitude instrument. No significant di fferences were shown on pretes t to post test data between-
88 groups ( p = 0.75). When measured within-groups, there were no sign ificant differences from preto post test results (control, p = .10 and treatment, p = .55). Problems get increasingly more difficult in thermodynamics as new concepts are introduced. Hong et al. (2001) found problems n eed to have conceptual and structural complexity in order for students to engage in regulation of cognition. Problem-solving studies have demonstrated experts more efficiently use metacognitive strategies than novices in complex problem-solving situa tions (Brown, 1987; Davidson & Sternberg, 1998; Sweller, 1988). Schoenfeld (1985) stat es novices perceive problem-solving differently than experts and that more adeptn ess at problem-solving leads to more expertlike behavior. A similarity in self-percepti on of expert-like beha vior towards problemsolving ability could explain no signifi cant differences between the groups. In a 2005 study by Hutchinson, Follman and Bodner, students were given a survey to Â“identify the factors related to st udentsÂ’ self-efficacy beliefs during their first engineering course. The survey was admini stered to freshmen engineering students (n=1387) mid-way through the semester to enrolled in a course titled Engineering Problem-Solving and Computer Tools (ENGR 106). The open-ended survey asked student to lis t the factors Â“affecting their confidence in their ability to succeed in the courseÂ” ( p. 6). Eight factors emerged as indicated by students responding to the survey. In the orde r of influence on self-efficacy according to the survey responses: 1. Understanding / Mastery of materials 2. Drive / Motivation 3. Teaming
89 4. Computing Abilities 5. Help 6. Doing Assignments 7. Problem Solving Abilities 8. Enjoyment, Interest, and Satisfaction Understanding or learning the course cont ent was the most important factor listed by students as an influence of their conf idence to succeed in the course ENGR 106. Problem-solving was rated near the bottom of the 8 items, listed in seventh-place. The majority of the responses from males indicate d an increase in their self-efficacy beliefs was due to their perception as successful pr oblem-solvers. On the contrary, the women did not respond with as much confidence to wards problem-solving abilities leading to their success. Three-quarters of the women sampled were positively influenced by their problem-solving abilities and the remaining wo men perceived them harmful to success. The present study sample c onsisted of 39 percent males and 9 percent females in the control group and 44 percent males and 8 pe rcent females in the treatment group. As the majority of the sample was males, it is possible there were no significant results between the groupsÂ’ perception of problem-solving ability because they considered themselves already adept at problem-solving skill s. Â“Again, studentsÂ’ efficacy beliefs are being shaped by whether or not they feel they have mastered the ability to use problemsolving techniques effectivelyÂ” (p. 9). Another consideration was the instru ment measuring perception of problemsolving ability. The HDYSP survey result s were averages of answers taken from studentsÂ’ attitudes toward their own probl em solution skills. Measurement of the
90 instrumentÂ’s reliability fell within an acceptable range for monitoring and evaluatin g dimensions ( r = .83, r = .90), respectively. However, modification of the reliability coefficient for the planning dimension resulted in a lower predicted value ( r = .69). The planning dimension of the instrument should be re-examined to improve the reliability of this section as a coefficient of .80 or above is considered accepta ble in most social science applications. It is suggested in th e future that research tease apart actual strategies rather than perceptions of problem-solving ability. Question 3. Differences in thermodynamics knowledge and skills between students were measured in the beginning of the semester prior to the start of the experiment and repeated again at the end of the semester after the conclusion of the intervention. It was anticipated significan t differences might exist when comparing posttest to pretest within-subjects comparison of means. There was sufficient evidence to conclude improvement of thermodynamic sÂ’ skills and concepts for both groups throughout the semester. Pretest to posttest results showed equal improvement for both groups, contradicting the hypothe sis those students in the treatment group would improve in skills and knowledge more than the control group. Another possible reason there were no si gnificant differences in thermodynamics knowledge between groups because of the data collection instrument. Test items of the pretest consisted of objective multiple-choice and short answer formats. Seventeen questions were developed by content experts in the Chemical Engineering Department to assess universally studentsÂ’ knowledge and ski lls in: (1) the general understanding of the concepts of energy, how to measure it and the different forms of expression of energy in nature, (2) the concept of conservation of energy which says that energy cannot be
91 created or destroyed, but can be converted from one form to the other, and (3) the approach to analyzing engi neering problems and applica tion of basic knowledge to answer simple engineering questions. The data in the current study was measured using an instrument with the same test items for both administrations. Practice effect is a definite threat to internal validity when testing participants mo re than once (Davis & Smith, 2005). Changes in scores can occur simply because you have done nothing othe r than repeating the test. Future studies should look at an instrument designed to measure thermodynamics concepts equally, however, varying the test items. Question 4. The fourth question examined which of the eight step s to engineering problem-solving students used across the proble m sets. A frequency analysis revealed differences in the amount of times studentsÂ’ reported using the various engineering problem-solving steps while working through the exercises. In this study, students responded most frequently to using Step Two List Variables (91 %) and Step Seven Solved Equations (91%). The least chosen response was S tep Four Â– Made/stated Assumptions which was selected only three percent of the time. Another item, selected less than 50 percent of the time, was Step Five Â– Listed References (30%). Taking a closer look at the response frequencies, Step One Â– Abstract the Problem was selected 70 percent of the time. This it em has two parts Â– first, to understand fully what is being asked in a part icular problem and secondly to draw an engineering sketch to depict the problem graphically. Cons idering the 70 percent response frequency, students may have completed the first part of the step, skipping the second more involved practice of drawing the sketch. Step Two List Variables had one of the highest
92 frequencies (91%) reported by students. In Step Two students are asked to list all variables and unknowns related to the problem. This is a ne cessary phase to solving the problem sets. Step Three Â– State the Basi s for Your Calculations and Step Eight Â– Interpret Solution were reported 62 percent and 61 per cent of the time. Because many of the problems include the basis for calculati on, students may not have deemed this step essential to completing the problems. Step Eight suggests validation of the solution using common sense. Students are encouraged to use their intuition in deciding if their answer is reasonable. It is possible students auto matically (unconsciously) completed this step, as just above half of th e respondents indicated they interpreted the solution The least chosen responses, selected less than 50 percent of the time, were S tep Four Â– Made/stated Assumptions (3%) and Step Five Â– List Your References (30%). It was recommended in Step Four to make assumptions about the problem including justification for the answer. Step Five proposed reporting all sources of information and data used in the problem solutions. Using Steps Four and Five in the web-based program to complete the problem sets were not re quired, which may be why very few students reported using them. Nearly three qu arters of the students reported using Step Six Â– Develop Model Equations (71%) which advised writing dow n problem variables, using algebraic symbols, and stating how each equati on was obtained. It is possible students used only one or more parts of this ste p, therefore they did not report it in the metacognitive reflections. Nearly all the respondents indicated they used Step Seven Solved Equations (91%). The assumption is this step should be used 100 percent of the time, however, some students may not have completed all the problem sets which reduced the frequency for Step Seven Universally for all the step s, it is possible students
93 made arbitrary choices of the Eight-steps to Engineering Problem-solving during the reflection phase since they were not require d have to explain why they chose a certain response. Research from Hong et al. (2001) indicated justificati on skills are an important predictor for open-ended problem-solving scores They concluded Â“in order to promote studentsÂ’ problem-solving skills, educators mu st develop teaching and learning strategies that use different cognitive components. Specific educational goals and the problems adapted for their instruction must in turn be designed to build specific cognitive skillsÂ” (p. 4). In two related studies, one group included ninth-grade students and the second group included sixth to eighth-gr ade students, participated in research to look at four mental components (cognition, metacognition, non-cognitive variables and justification skills) deemed important for su ccessful problemsolving. Both studies were investigated separately over a 4-week period. The first study used an open-ended response format for presentation of both the well-structured a nd ill-structured problems. The second study used a multiple-choice format for similar well-structure and ill-structured problems. Hong et al. found justification sk ills were statistically signifi cant as predictors in solving open-ended problems and concluded student s who were able to provide logical arguments would be able to successfully solve those problems. In another study by Condor (2001), st udents were asked to fill out a Metacognitive Cue Worksheet while they worked through statistical word problems following a computer-based lesson. Th e students responded to a series of metacognitively-structured questions by writing on the cue worksheets how they worked through the problem solutions. Students recorded written reflections on paper. An
94 example of the type of questions asked included: What exactly are you doing? (Can you describe it precisely?) Slight differences in proble m-solving for the treatment group were found when metacognitive cuing was introduced: and there was a positive correlation between metacognitive awareness and problem-solving. A third study by King (1991) found that st udents in the fifth-grade who were trained in asking and answering metacognitive questions demonstrat ed the ability to express conceptual understand ing better and gave more explanations to peers when presented with novel problems, than those w ho were not trained. During a three-week period, forty-six students were assigned to one of three treatment groups: guided questioning, unguided questioning, and control. Eleven general questions were divided into three dimensions (planning, monitori ng, and evaluating). Examples include questions such as Â“What is the problem?Â”, Â“A re we using our plan or strategy?Â”, and Â“What worked?Â”. Only the two questioning groups were instructed on how to use questioning during problem-solving. These sa me two groups were given cards with the eleven questions printed on them to refer to during the problem-solving exercises. Additional results from the study indicated st udents trained in the questioning procedures were more successful than the non-trained stude nts in a paper and pencil test of problemsolving abilities and in solving novel computer problems. The metacognitive reflection section of the learning sequence for the Eight-Steps to Engineering Problem-Solving model required students to check off which of the eight steps they used during the problem-solving process. The web-based instructional program did not include an area for short answer responses al lowing students the opportunity to articulate why they chose one response over another. According to Gupta
95 (1992), Â“Reflection on the feedback from expe riences and subsequent abstraction of the results of the reflection into oneÂ’s co gnitive structures fosters metacognitive developmentÂ” (as cited in Zhang, RiCharde a nd Stephen, 1998). Further development of the web-based program could include the ab ility to capture short-answer data for analysis. A prerequisite prior to treatme nt was for students to read the Eight-Steps to Engineering Problem-Solving. It is possible students failed to complete this assignment. The web-based program was not designed to tr ack if students spent time on the reading section or how much time was spent on th e reading section. Tracking if students completed the reading could be a future vers ion of the web-based problem set tutorial. In the current study, all students receive d the same debriefing of the problem solutions in-class. Students using the webbased problem sets did not receive feedback specific to the metacognitive strategies they employed while problem-solving. Development of metacognitive awareness requir es practice and reflective contemplation (Kramarski & Zeichner, 2001) One way to help students learn the c ognitive skills of planning, monitoring and evaluating is to ha ve the instructor model the behavior. Additionally, the web-based in structional program should provide examples of how to effectively use the strategies. Student s worked independently on problem-solving without formalized discussion of the met acognitive process. Any group work among students was done informally in an ad hoc natu re, as it was not part of the study design. Therefore, there may have been no difference between the groups in achievement because students did not debrief their probl em-solving activities either verbally or through written reflection.
96 Although the metacognitive reflections app eared at the end of each problem set, responses could have been completed by student s at a later time, rath er than immediately after the problem sets. Future programmi ng should address requiring a response from students before the program allows continuation to the next lesson. This study was concentrated on achieve ment outcomes, however, establishing a relationship between the exercises and the engineering problem-solving model could further extend this study. It may be possi ble to understand why students responded in a certain way by adding an interview component to the study. The metacognitive reflection was used as an intermission to allow student contemplation of the steps used while so lving problems. Encouraging successive approximation is needed, through feedback, as a student engages in the problem-solving process. Further study is needed on the e ffectiveness of a computer tutoring program prompting students as to wh ich strategy is appropriate while students practice problemsolving. Rather than generalizing the problem -solving steps on each instructional screen, incorporation of how and when a student s hould implement the steps. According to Campione (1987), instruction of metacogniti ve skills concurrently with the domainspecific skills they are to manage appears more effective than teaching each type of skill independently. This was not a part of this research, however, it could be used to extend future versions of the web-based problem sets.
97 Question 5. The biographical questionnaire was completed by study participants to collect demographical information. Overall, national statistics gathered on undergraduate engineering students were refl ected. Composition of the group indicated mostly white, non-Hispanic, males in their ear ly twenties who declared English as their native language. The majority had entered the university as freshmen as First Time in College (FTIC) students, rather than transfers either from community colleges or other institutions. Geographically, th e participants were mainly Florida residents, with two percent of the participants declaring out-of-state residency and seve n percent declaring out-of-country residency. Responses from the questionnaire confir med the homogeneity of this group. The majority of students reported th eir academic standing (GPA) from 3.0 to 4.0 and their year of study as Juniors This would imply highly motivated, capable students, with academic success the engineering academic cu rriculum. Zhang and RiCharde (1998) concluded metacognitive development is fo stered by academic achievement, indicating students with Â“good academic standing possess a stronger ability to reason, think, and make decisions about personal and intellectual developmentÂ” (p. 15). As discussed previously in Question Two, the Attitudes section, Hutchinson et al. (2005) found Understanding / Mastery of Materials as having the most influence on studentsÂ’ self-efficacy beliefs in their abil ity to succeed in a course. Problem-solving abilities ranked seventh out of eight factors identified by st udents as influential towards their self-confidence in course completi on. Men responding to the Hutchinson et al. survey clearly showed an increase in th eir self-efficacy beliefs was due to their perception as successful problem-solvers. The women responding the same survey, on
98 the other hand, did not indicated as much confidence towards problem-solving abilities leading to their success. Most of the women sampled believed they were positively influenced by their problem-solving abil ities and about one quarter of the group perceived problem-solving abilities harmful to their success. The perception of problem-solving abilitie s might have limited variability among engineering students due to pred isposition as they chose this pa rticular discipline in part because of their problem-solving skills. It is possible because of their similarities in abilities, they already have well-develope d problem-solving skills and any treatment would have minimal effect. Â“ Regarding se lf-efficacy, students who indicated confidence in comprehension of course materials were more apt to also indicate they employed cognitive and self-regula tory strategiesÂ” (Wolte rs & Pintrich, 2001, p. 28). Generalization from such a homogenous group to other populations or settings may have different results. Future resear chers should consider a more diverse sample such as multiple institutions, the same group of students in different courses, and/or examining student differences longitudinally. This thermodynamics class had an unus ually large percentage of course noncompleters. Swanson (1990) found problem-sol ving performance is positively influenced by high-metacognitive ability regardless of ap titude. A comparison of high, mid, and low achieving studentsÂ’ use of metacognitive stra tegies when utilizing the engineering problem-solving model warrants additional study. Further research could investigate the potential impact of using metacogni tive strategies on drop-out rates.
99 Question 6. Participants were asked to self-re port their percep tions of the webbased problem sets. The Survey of Students on Web-based Problem-Solving Tutorials was developed by the academic department offering the thermodynamics course and added a priori to the research. Questions rega rding perceptions of problem-solving knowledge and ability were vague and open to individual interpretation. The results for the problem-solving statements of this survey are ambiguous and can lead to misinterpretation. It is recommended the wo rding of the statements in the survey is changed in future research. More specific statements about metac ognitive strategies could be used as the statements may have been too general for st udents to understand the implication of them. For example, the statement (#29) I am familiar with the general problem-solving strategies resulted in both groups agreeing with the statement, as reflected by the mean ( M = 4.07) of their responses. The statement (#30), I used general problem-solving strategies when doing these problems, indicated a slightly higher mean ( M = 4.0) for the treatment group than the control groupÂ’s mean (3.9). It was expected the treatment groupÂ’s mean to be significantly different from the control groupÂ’s mean as a result of the intervention. It is possibl e the treatment group was referr ing to their general knowledge of problem-solving skill level, based upon the prior question, and did not interpret this question to mean metacognitive strategi es. Responses to the statement (#31) I have not had a formal introduction to engineering problem-solving showed answers, for both groups, in the mid-range between either neutral and disagree with this statement. As students read that particular statement, it is unclear whethe r it they took the meaning as their experience with formal instruction in problem-solving skills before this course or
100 during this course. When reviewi ng the data for the statement (#32), I think an introduction to general problem-solving strategies would have helped me, means for the control ( M = 3.0) and treatment ( M = 2.67) groups were similar, with the mean for the treatment group slightly lower. Since both groups indicated a neutral response it appeared the experimental group received little to no effect from the treatment. Again, it is possible this statement was interpreted by the students as problem-solving skills they received prior to this course. Results for statement #33, I am learning problem-solving strategies through example, but it w ould be helpful to formalize it, revealed a higher mean for the control group ( M = 3.33) than for the treatment group ( M = 3.27). This statement does not clarify whether the problem -solving strategies th rough example were from in-class lectures, from homework probl ems set or, in the case of the treatment group, from metacognitive cuing. Limitations This study was conducted on an intact gr oup of students from one section of thermodynamics in the spring semester of 2005. When interpreting the findings from this study, the following limitations should be considered: 1. Population validity Â– The selected gr oup of participants were students majoring in engineering. No other acad emic disciplines were included in the study. Generalization to other populati ons should take into account the selection bias of this sample. 2. Attrition Â– As reported in the results, this class section had an atypical number of participants w ithdraw from the course throughout the semester. Anticipating experimental mortality, a robust sample size was chosen reducing the impact of the 39% particip ant loss. The end result maintained comparable sizes between the control group (n = 42) and the experimental group (n = 39).
101 3. Random Sampling Â– ABBA Matching was used for group assignment. Choosing this method suggests the gr oup means are not identical, however they are as alike as you can expect without random assignment. When measured, prior to treatment, group comparability was established. 4. Social Threats Â– The possibility of students discussing differences in instructional format existed. The same section of the thermodynamics course was divided into the two groups (treat ment and control). Using separate sections of the same course could minimize this effect. 5. Diffusion of treatment effect: If studen ts chose to work together informally, their responses may have been the cons ensus of a group rath er than their own answers. Students worked on their homework assignments in a natural setting without observation, not in a controlled environment. Conclusions and Recommendations for Further Study The purpose of this research was to investigate the effect of metacognitive cuing on problem-solving ability. No significant di fferences were found in achievement or perception of problem-solving abilities betw een the two groups. Several reasons are offered for these results. First, the length of the study needs to be increased to allow time for maturation of the treatment condition. Other studies have found a significant difference in achievement when longer expe rimentation was used. Second, studentsÂ’ perceived their problem-solving abilities to be highly develope d. Future research should tease apart actual strategies rather than the studentsÂ’ perceptions. Third, the instrument used to collect data of st udentsÂ’ thermodynamics knowledge before and after treatment was suspect to practice effect. The same test items were used for both administrations. Fourth, when studentsÂ’ were asked to reflect upon the Eight-steps to Engineering Problem-Solving they used, justification of why th ey used certain steps was absent. Further study is needed linking metacognitive strategies with the problem-solving steps. Fifth, the composition of this sample dem ographically was homogeneous and therefore
102 results may not be generalized to other popul ations. And finally, to benefit from the results of studentsÂ’ perceptions of the web-ba sed problem sets tutori al, changes need to be made to the instrument, specifically, cl arifying the statements relating to problemsolving. Research of web-based learning environm ents is just beginning in the area of embedded metacognitive cuing. There is st ill much to be learned about the role metacognitive strategies play on problem-solv ing ability when embedded cuing is used in a web-enhanced environment. Implications for Practice The findings from this in vestigation, along with previ ous research, facilitate the definition of boundary conditions when em ploying metacognitive cuing in web-based learning. Translating the research advances practical application of the work. Recommendations can be made to incorpor ate the outcomes of th is study into the classroom. Modeling the utility of metacognitive stra tegies should become part of classroom lectures to supplement the web-based ho mework problems. The format of the thermodynamics course combined a didactic approach of in-class lecture with homework assignments practiced outside of class. Pr oblem set solutions were reviewed at the following class meetings. In the futu re, a four-step method is suggested: (1) introduction and demonstration of metacognitive strategies during the classroom lectures; (2) practice of the problem sets by student s through the homework assignments; (3) reinforcement of the strategies by the instruct or during the solution review sessions in the following class period; and (4) reflection of the process by the stude nt after the review
103 sessions. Â“When new, and particularly di fficult, skills are being taught, it may be necessary to include self-regulation traini ng, even for subject who, on other occasions, have been known to engage those skills them selvesÂ” (Day as cited in Campione, 1987, p. 134). A tutorial should be included to the web-based instructional program before students begin using the on-line problem sets. Because the Eight-Steps to Engineering Problem-Solving model is such an important component in skill development, it should be required reading prior to using the on-lin e homework problems. The tutorial would allow students the opportunity to practice before beginning the graded on-line instruction.
104 References Accreditation Board for Engi neering and Technology (2003). Evaluation criteria: 200405 engineering criteria Retrieved July 15, 2004, from http://www.abet.org Armour-Thomas, E. & Haynes, N.M. (1988, Se ptember). Assessment of metacognition in problem-solving. Journal of Instructional Psychology, 15 87-93. Artz, A.F. & Armour-Thomas, E. (2001). Ma thematics teaching as problem-solving: A framework for studying teacher metacognition underlying instructional practice in mathematics. In H.J. Hartman (Ed.), Metacognition in Learning and Instruction, Theory, Research and Practice ( pp.127-171). Boston, MA: Kluwer. Ashman, A.F. & Conway, R.N.F. (2002). An introduction to cognitive education: Theory and applications London: Routledge. Bandura, A. (1997). Self-efficacy: The exercise of control New York, NY: Freeman. Bandura, A. (2001). Social cognitive th eory: An agentic perspective. Annual Review of Psychology, 52, 1-26. Baron, L. (2003). Student self-assessment: Conceptual frameworks Retrieved September 10, 2003, from the University of Nevada Las Vegas, Teaching and Learning Center: http://www.unlv.edu/centers/tlc/articles/A ssessmentEvaluation/tlc_studentselfasse ssmentconceptualframeworks.html
105 Bransford, J. D., Sherwood, R. S., Vye, N. J., & Rieser, J. (1986). Teaching thinking and problem-solving: Research foundations. American Psychologist, 41 1078-1089. Bransford, J.D., Brown, A.L., & Cocking, R.R. (Eds.) (1999). How people learn: brain, mind, experience, and school Retrieved June 15, 2004, from http://www.netlibrary.com/urlap i.asp?action=summary&v=1&bookid=6633 Broadbent, D.E. (1977). Levels, hier archies, and locus of control. Quarterly Journal of Experimental Psychology, 29 181-201. Brockett, R. G., & Hiemstra, R. (1991). Self-direction in adult le arning: Perspectives on theory, research, and practice New York, NY: Routledge. Brown, A.L. (1987). Metacogni tion, executive control, self -regulation, and other more mysterious mechanisms. In F.E.Weinert, & R.H. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 65-116). Hillsdale, NJ: Erlbaum. Brown, K.M. (1999). Social Cognitive Theory Retrieved December 20, 2004 from http://www.med.usf.edu/~kmbrown/Soc ial_Cognitive_Theory_Overview.htm Buck, S.M. (2004). Evaluation of a hybrid web-based tutorial in a classroom setting Unpublished honors thesis, University of South Florida, Tampa. Bullmaster, M., & Alcock, H. (2003, May). Cognitive processing, effective teaching, and cognitive strategies: Research foundations for Kaplan K12 Advantage Campione, J. (1987). Metacognitive components of instructional res earch with problem learners. In F.E. Weinert, & R. H. Kluwe (Eds.), Metacognition, motivation, and understanding (pp. 117-137). Hillsdale, NJ: Erlbaum.
106 Case, J., Gunstone, R., & Lewis, A. (2001). StudentsÂ’ Metacognitive Development in an Innovative Second Year Chem ical Engineering Course. Research in Science Education, 31 313-335. Cohen, J. (1988). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ: Erlbaum. Condor, J.A. (2001). Effects of computer coaching in metacognitively-cued elementary statistics instruction Unpublished doctoral dissertation, University of South Florida, Tampa. Covington, M. (2000). Goal theory, motivation, and school achievement: An integrative review. Annual Review of Psychology, 51 171-200. Cranton, P. (1994). Understanding and promoting transform ative learning: A guide for educators of adults San Francisco, CA: Jossey-Bass. Crocker, L. & Algina, J. (1986). Introduction to classical & modern test theory Holt, Rinehart and Winston, Inc., USA. Davidson, J.E. & Sternberg, R. J. (1998). Smart problem-solving: how metacognition helps. In D.J. Hacker, J. Dunlosky, J., & A.C. Graesser (Eds.), Metacognition in educational theory and practice (pp. 47-68). Mahwah, NJ: Erlbaum. Davis, S. F. & Smith, R. A. (2005). An introduction to statisti cs and research methods: becoming a psychological detective Upper Saddle River, NJ: Pearson-Prentice all. Dede, C.J. (1987). Empowering environments, hypermedia, and microworlds. The Computing Teacher, 15 20-24.
107 deLeon, L. (1996). About learning contracts, Conditions of learning & principles of teaching Retrieved September 6, 2003, from http://carbon.cudenver.edu/~lde leon/ pad5220/learning_contracts /about_contracts/conditns.html Ellis, D. (1994). Effects of self-monitoring and disc rimination training on pronunciation change by nonstandard speakers of English Ann Arbor, MI: UMI Dissertation Services. Ellis, D. & Zimmerman, B. J. (2001). Enha ncing self-monitoring during self-regulated learning of speech. In H.J. Hartman (Ed.), Metacognition in learning and instruction, theory, research and practice (pp.205-228). Boston, MA: Kluwer. Everson, H.T. & Tobias, S. (1998). The ability to estimate knowledge and performance in college: A metacognitive analysis. Instructional Science 26 65-79. Ferrar, M. (2001). Metacognition Retrieved August 3, 2004, from http://www.sbcc.edu/learningresources/index.php?sec=1523 Flavell, J. H. (1979). Metacognition and cogni tive monitoring: A new area of cognitivedevelopmental inquiry. American Psychologist, 34, 906-911. Flavell, J.H. (1981). Cognitive monitoring. In W. Dickson (Ed.), ChildrenÂ’s oral communication skills New York: Academic Press. Flavell, J.H., (1985). Cognitive development (2nd ed.). Englewood Cliffs, NJ: PrenticeHall. Flavell, J.H., (1987). Speculations about th e nature and development of metacognition. In F.E. Weinert & R.H. Kluwe, R.H. (Eds.), Metacognition, motivation, and understanding (pp. 21-29). Hillsdale, NJ: Erlbaum.
108 Flavell, J.H., Miller, P.H., & Miller, S.A. (1993). Cognitive development (3rd ed.). Englewood Cliffs, NJ: Prentice-Hall. Fortunato, I., Hecht, D., Titt le, C.K., & Alvarez, L. (1991) Metacognition and problemsolving. Arithmetic Teacher 38 (4), 38-40. Gagne, R.M. (1985). The conditions of learning, and theory of instruction (4th ed.). New York: CBS College Publishing. Gagne, R.M. (1989). Studies of learning: 50 years of research Tallahassee, FL: Learning Systems Institute. Gagne, R.M., & Driscoll, M.P. (1988). Essentials of Learning. New Jersey: Prentice Hall. Gardner, H. (1991). The unschooled mind New York: Harper & Row. Garner, R. (1990). When children and adults do not use learning strategies: Toward a theory of settings. Review of Educational Research, 60 (4), 517-529. Garner, R., & Alexander, P.A. (1989). Metacognition: answered and unanswered questions. Educational Psychologist 24 143-58. Gay, L. R. & Airasian, P. (2003). Educational research: Competencies for analysis and applications (7th ed.) .Upper Saddle River, NJ: Merrill/Prentice. Gibbons, M. T. (2004). The year in numbers Retrieved September 18, 2005, from http://asee.org Gourgey, A. F. (2001). Metacognition in basic skills instruction. In H.J. Hartman (Ed.), Metacognition in learning and instruc tion, theory, research and practice (pp.17-32). Boston, MA: Kluwer.
109 Hacker, D.J. (1998). Definitions and empirical foundations. In D.J. Hacker, J. Dunlosky, & A.C. Graesser (Eds.), Metacognition in educat ional theory and practice (pp.1-23). Mahwah, NJ: Erlbaum. Hadjileontiadou, S.J. (2004). Using metacognition to enhan ce web-based collaboration: A case study in engineering education Retrieved December 13, 2004, from http://skyblue.csd.auth.gr/~bci1/ Panhellenic/470h adjileontiadou.pdf Hannibal, J. & Gymnasium, N. (2003, February). Developing self-efficacy in students through study skills Presentation at the Danish/Swe dish IB Teachers Conference: Retrieved December 1, 2004 from http://www.norreg.dk/ib/studyskills030228.htm Hartman, H.J. (Ed.) (2001). Metacognition in learning and instruction: Theory, research and practice Boston, MA: Kluwer. Hiemstra, R. (1998). Self-advocacy and self-directed lear ning: A potential confluence for enhanced personal empowerment Retrieved September 15, 2003, from http://home.twcny.rr.com/hiemstra/advocacy.html Hong, N.S., McGee, S., & Howard, B. C. (2001, April). Essential components for solving various problems in multimedia learning environments Paper presented at the annual meeting of the American Educatio nal Research Associ ation, Seattle, WA. Horton, W. (2000). Designing web-based training: How to teach anyone, anything, anywhere, anytime New York, NY: John Wiley & Sons, Inc.
110 Howard, B.C., McGee, S., Shia, R., Hong, N.S. (2001). Computer-based science inquiry: How components of metacognitive se lf-regulation affect problem-solving Paper presented at the annual meeting of the American Educational Research Association, Seattle, WA. Hutchison, M.A., Follman, D.K., & Bodner, G.M. (2005). Shaping the self-efficacy beliefs of first-year engi neering students: What is the role we play? Paper presented at the annual conference and e xposition of the American Society for Engineering Education, Portland, OR. Jackson, N.E., & Butterfield, E.C. (1986). A conception of giftedness designed to promote research. In R.J. Ster nberg & J.E. Davidson (Eds.), Conceptions of giftedness (pp. 151-181). New York: Cambridge University Press. Jacobs, J.E., & Paris, S.G. (1987). ChildrenÂ’ s metacognition about reading: Issues in definition, measurement, and instruction. Educational Psychologist 22 255-278. Jacques, R., Nonnecke, B., McKerlie, D., & Preece, J. (1993). Current designs in HyperCard: What can we learn? Journal of Educational Multimedia and Hypermedia, 2 219-238. Johnson, B., & Turner, L.A. (2003). Data co llection strategies in mixed methods research. In A. Tashakko rri & C. Teddlie (Eds.), Handbook of mixed methods in social and behavioral research (pp. 297-319). Thousdand Oaks, CA: Sage. Jonassen, D.H. (n.d.). Toward a meta-theory of problem-solving Retrieved July 2, 2004, from: http://www.coe.missouri.edu/~jonassen/problems.htm Joseph, B. (October 2004). Engineering problem-solving Personal communication.
111 Kapa, E. (1999b). The effects of metacognitive feedbac k on solving mathematical word problems with the computer Unpublished doctoral disser tation, Department of Education, Bar-Ilan University, Israel. Kapa, E. (2001). A metacognitive support dur ing the process of problem-solving in a computerized environment. Educational Studies in Mathematics, 47 (3), 317-336. King, A. (1991). Effects of training in st rategic questioning on childrenÂ’s problemsolving performance. Journal of Educational Psychology 83 307-317. Knowles, M.S. (1968). Andragogy, not pedagogy. Adult leadership 16 (10), 350-352, 386. Knowles, M. S. (1980). The modern practice of adult education: Andragogy vs. Pedagogy New York, NY: Association Press. Kramarski, B. & Zeichner, O. (2001). Using technology to enhance mathematical reasoning: Effects of feedback and self-regulation learning. Educational Media International. Retrieved September 10, 2004, from http://www.tandf.co.uk/journals Lieb, S. (1991). Principles of adult learning. VISION Arizona Department of Health Services. Retrieved September 22, 2003, from http://www.hcc.hawaii.edu/intranet/ committees/FacDevCom/guide bk/teachtip/adults-2.htm Lin, X. (2001). Designing metacognitive activities. ETR&D 49 (2), 23-40. Lin, X. & Lehman, J.D. (1999). Supporting lear ning of variable control in a computerbased biology environment: effects of pr ompting college students to reflect on their own thinking. Journal of Research in Science Teaching 36 (7), 837-858.
112 Livingston, J.A. (1997). Metacognition: An overview Retrieved August 3, 2004, from http://www.gse.buffalo.edu/ fas/shuell/cep564/Metacog.htm Lloyd, J.W., Kameanui, E.J., & Chard, D. (Eds.), (1997). Advances in research on instruction: Issues in educa tion, students with disabilities Mahwah, NJ: Lawrence Erlbaum. Marzano, R.J. (1998). Models of standards implemen tation: Implications for the classroom Retrieved September 12, 2004, from http://www.mcrel.org/products/standards/models.asp McCown, R., Driscoll, M., & Roop, R. (1996). Educational psychology: A learningcentered approach to classroom practice (2nd ed.). Needham Heights, MA: Allyn & Bacon. McCown, R.,& Roop, R. (1992). Educational psychology and classroom practice: A partnership Boston, MA: Allyn & Bacon. McDaniel, M.A., & Fisher, R.P. (1991). Tests and tests feedback as learning sources. Contemporary Educational Psychology 16 192-201. McLoughlin, C. & Hollingworth, R. (2001). Deve loping science studentsÂ’ metacognitive problem-solving skills online. Australian Journal of Educational Technology 17 (1), 50-63. McNeill, J.R. (2002). Metacognitive scaffolding of information search Retrieved July 31, 2004, from http://www.jeffmcneill.com/research/msis.doc Meichenbaum, D., & Biemiller, A. (1998). Nurturing independent learners: Helping students take charge of their learning Cambridge, MA: Brookline Books.
113 Merriam, S. B., & Caffarella, R.S. (2001). New directions for adult and continuing education. San Francisco, CA: Jossey-Bass. Mevarech, Z.R., & Kramarski, B. (2003). Th e effects of metacognitive training versus worked-out examples on stude ntsÂ’ mathematical reasoning. British Journal of Educational Psychology 73 449-471. Mezirow, J. (1997). Transformative learning: Theory to practice. New directions for adult and continuing education, 74 5-12. Mitrovic, A. (2001). Investigating students' self-assessment skills Retrieved September 15, 2003, from http://www.umich.edu/~pals/manuals.html Morse, J.M. (2003). Principles of mixed methods and multi-method research design. In A. Tashakkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 189-208). Thousand Oaks, CA: Sage. Nielson, S.G. (2004). Strategies and self-e fficacy beliefs in inst rumental and vocal individual practice: A study of st udents in higher music education. Psychology of Music 32 (4), 418-431. North Central Regional Edu cational Laboratory, (1995). Strategic teaching and reading project guidebook. Retrieved September 10, 2003, from http://www.ncrel.org/sdrs/areas/i ssues/students/learning/lr1metn.htm Onwuegbuzie, A. J. (2001). Role of study skills in graduate-level educational research courses. The Journal of Educational Research 94 (4), 238-246.
114 Onwuegbuzie, A.J., & Teddlie, C. (2003). A framework for analyzing data in mixed methods research. In A. Tash akkori & C. Teddlie (Eds.), Handbook of mixed methods in social & behavioral research (pp. 351-383). Thousand Oaks, CA: Sage. Pajares, F. (1996). Self-efficacy beliefs in academic settings. Review of Educational Research, 66 (4), 543-578. Pajares, F., & D. H. Schunk (2001). Self-bel iefs and school success: Self-efficacy, selfconcept, and school achievement. In R. Riding & S. Rayner (Eds.), Perception (pp.239-266). London: Ablex Publishing. Paris, S.G., & Winograd, P. (1990). How me tacognition can promote academic learning and instruction. In B. F. Jones & L. Idol (Eds.), Dimensions of thinking and cognitive instruction (pp.15-51). Hillsdale, NJ: Lawrence Erlbaum Associates. Peters, M. (2000). Does constructivist epis temology have a place in nurse education? Journal of Nursing Education, 39 (4), 166-170. Pintrich, P.R., & deGroot, E.V. (1990). Motivational and self-regulated learning components of classroom academic performance. Journal of Educational Psychology, 82 33-40. Rivers, W. (2001). Autonomy at all costs: an ethnography of metacognitive selfassessment and self-management among experienced language learners. Modern Language Journal, 85 (2), 279-290. Rosenshine, B. (1997). Advances in Research on Instruction. In J. W. Lloyd, E.J. Kameanui and D. Chard (Eds.) Issues in educating students with disabilities (pp. 197-221). Mahwah, N J. Lawrence Erlbaum.
115 Rosenshine, B., & Meister, C. (1994). Reci procal teaching: A re view of research. Review of Educational Resear ch, 64 (4), 479-530. Rutz, E., Eckart, R., Wade, J.E., Maltbie, C., Rafter, C., & Elkins, V. (April 2003). Student performance and acceptance of instructional technology: Comparing technology-enhanced and traditional instruction for a course in statistics. Journal of Engineering Education 92 (2), 133-140. Ryder, M. & Wilson, B. (1996). Affordances and Constraints of the internet for learning and instruction. Paper presented at the A ssociation for Educational Communications Technology join t session, Indianapolis, IN Schoenfeld, A.H. (1985). Mathematical problem-solving Orlando, FL: Academic Press. Schraw, G. (1998, 2001). Promoting general met acognitive awareness. In H.J. Hacker (Ed.), Metacognition in learning and instruc tion, theory, research and practice (pp.3-16). Boston, MA: Kluwer. Schraw, G. & Dennison, R.S. (1994). Assessing metacognitive awareness. Contemporary Educational Psychology 19 460-475. Schunk, D. (1989). Self-efficacy and cognitive achievement: implications for students with learning problems. Journal of Learning Disabilities 22 14-22. Smith, D.A. (July 2002). How people learn Â… mathematics Retrieved June 10, 2004, from http://www.math.duke.edu/~das/essays/hplm/hplm.pdf Strauser, D. R., Ketz, K., & Keim, J. ( 2002). The relationship between self-efficacy, locus of control and work personality. Journal of Rehabilitation 68 (1), 20-26. Sternberg, R.J. (1985). Beyond IQ: A triarchic theory of human intelligence New York: Cambridge University Press.
116 Sternberg, R. J. (1986). Critical thinking: Its nature, measurement, and improvement New Haven, CT: Yale University. (ERI C Document Reproduction Service No. ED 272 882). Sternberg, R.J. (2001). Metacognition, abiliti es, and developing expertise: What makes an expert student? In H.J. Hartman (Ed.), Metacognition in learning and instruction, theory, research and practice (pp.247-260). Boston, MA: Kluwer. Swanson, L.H. (1990). Influence of metac ognitive knowledge and aptitude on problemsolving. Journal of Educational Psychology, 82 306-314. Sweller, J. (1988). Learning and transfer in subject-matter learni ng: A problem-solving model. International Journal of Educational Research 11 607-622. Tashakkori, A. & Teddlie, C. (1998). Mixed methodology: Combining qualitative and quantitative approaches. (Applied social research methods, No. 46). Thousand Oaks, CA: Sage Publications. The Sloan Consortium. (2003). Sizing the opportunity: The quality and extent of online education in the United States, 2002 and 2003 Retrieved August 24, 2004, from http://www.sloan-c.org/resources/sizing_opportunity.pdf Trochim, William M.K. (2005). Research methods knowledge base Retrieved September 1, 2005, from http://www.socialresearchmethods.net/kb/ Wagner, R., & Sternberg, R.J. (1984). Altern ative conceptions of intelligence and their implications for education. Review of Educational Research 54 (2), 179-223. Watson, J.B. & Allen, B.S. (2002). The effect of metacogni tive prompts on student navigation, comprehension, and metacogniti ve awareness in a multimedia science tutorial Paper presented at the annual AE RA conference, New Orleans, LA.
117 West, J. (2004). Thermodynamics Retrieved October 10, 2004, from http://thermodynamics.eng.usf.edu/index.html White, B.Y. & Frederiksen, J.R. (1998). I nquiry, modeling, and metacognition: making science accessible to all students. Cognition and Instruction 16 (1), 3-118. Wiersma, W. (1995). Research methods in education: An introduction Boston, MA: Simon & Schuster. Wiersma, W. (2000). Research methods in education: An introduction (7th ed). Boston, MA:Allyn and Bacon. Wilson, B. G., Jonassen, D. H., & Cole, P. ( 1993). Cognitive approaches to instructional design. In G. M. Piskurich (Ed.), The ASTD handbook of inst ructional technology (pp. 21.1-21.22). New York: McGraw-Hill. Wolters, A., & Pintrich, P.R. (2001). Contextu al differences in student motivation and self-regulated learning in mathematics, E nglish and social studies classrooms. In H.J. Hartman (Ed.), Metacognition in learning and in struction, theory, research and practice (pp.103-124). Boston, MA: Kluwer. Woolfolk, A. (1998). Educational Psychology Boston: Allyn and Bacon. Wongsri, N., Cantwell, R.H, & Archer, J. (2002, December). The validation of measures of self-efficacy, motivation and self-r egulated learning among Thai tertiary students Paper presented at the annual mee ting of the Australian Association for Research in Education, Brisbane, Australia. Wren, C. (n.d.). Higher order cognition Retrieved August 6, 2004, from http://condor.depaul.edu/~cour ses/other/outmn444/ch7think.htm Zabel, S.A. (August, 2004). How do you solve problems? Personal communication.
118 Zellermayer, M., Salomon, G., Globerson, T ., & Givon, H. (1991). Enhancing writingrelated metacognition through a computerized writing partner. American Educational Research Journal 28 373-391. Zhang, Z. & RiCharde, R.S. (1998, April 13-17). Assessing college studentsÂ’ development: a repeated-measures analysis using a mixed model Paper presented at the Annual Meeting of the American Educational Research Association, San Diego, CA. Zimmerman, B.J. (1989). Models of Se lf-Regulated Learning and Academic Achievement. In B.J. Zimmerman & D.H. Schunk (Eds.). Self-Regulated Learning and Academic Achievement: Theory, Research, and Practice ( pp. 1-26 ). New York, NY: Springer-Verlag.
120 Appendix A: Engineering Problem-Solving By Joseph Babu, PhD and Professor 2004 Good problem-solving ability is an essentia l skill for all engineers. There is no universally accepted methodology for solving engineering problems. Problem-solving skills are attained through practice. The problem sets assigned as homework in your engineering courses are the mechanisms for ma stering the concepts learned in the lecture and in the reading material. It is important that you spend time tryi ng to do the problems on your own before seeking help, because the pr ocess of thinking about and strategizing a solution procedure is extremely important in assimilating the concepts. The computer-based problem sets designe d for this course are intended to build up your problem-solving skills. They are not different from chapter end problems in a good text book, except that we have tried to give you guidance and feedback during the problem-solving process so you know you are on th e right track. Hint and help files are provided to guide you towards a solution. In case you are unable to ge t the right answer, you can take a look at how the solution was a rrived at. Where possible we have provided references to text readings and examples that are relevant to the problem at hand. Even though there are no general pr ocedures for problem-solving, several techniques can help. Resist the urge to write formulas and substitute numbers into them. This usually leads to errors and mistakes be sides making it harder for some one else to check your solution. Some suggested steps are gi ven below. At first these steps may seem superfluous and unnecessary, but being systemat ic is the hallmark of a good engineer. It
121 Appendix A: (Continued) will also help you develop good written commun ication skills which are essential to function as an effective engineer. Not all problems will require you to go through all the steps, so use your judgment and intuition. Step 1. Abstract the Problem Remind yourself that you can do the probl em with the information given. While some problems appear to be difficult at first, rest assured that after reading it a few times, you will be able to tackle it. Write an abstra ct of the problem statement listing all of the information given and defining some variab les and constants along the way. Ask the question: what exactly is being asked in th is particular problem? Do not repeat the problem statement, rather try to restat e it in terms of how you interpreted it One good way to abstract the problem statemen t is to draw an engineering sketch This means if the problem is about a comp ressor you would draw a sket ch and supply appropriate information. Use engineering symbols wh ere possible as shown in your reading assignments. You are defining and planning. Include the specified cons traints. If it is useful, write down given information at appropriate locations in the sketch. Drawing a sketch, even if it is given in the problem statement, allows you to concentrate and focus your attention on the prob lem. This is good practice even in tests. When you are drawing a sketch you are tran slating from a verbal to a graphical interpretation of the problem statement. The sketch may be of the equipment or of the events taking place as verbally described in the problem. Use engineer ing paper if it helps you to draw better sketches. This paper has vertical and horizontal ruling that allows better sketches.
122 Appendix A: (Continued) Step 2. Make a List of Variables List all the variables/unknow ns associated with the pr oblem. Make a list of known quantities. Pay close attention to dimensiona l units. Each number (variable) must be accompanied by units. Do not assume that you have to use all the data given in the problem statement to solve the problem. On the other hand, data given may not be sufficient and in that case you may need to ma ke certain assumptions as stated below. Step 3. State the Basis for Your Calculations In many problems, the statement may incl ude a base flow rate or volume or production capacity. If it is give n in the problem, restate it. If it is necessary to assume one (e.g. you can do the problem assuming 100 kg of feed) do so and state your basis clearly. Step 4. Make and State Your Assumptions Most problems require assumptions to arrive at an answer. Sometimes these assumptions are given in the problem statem ent. Sometimes you may need to make them yourself. If you do, you must state it, and if possible, justify why you made that assumption. Any assumptions you make in a rriving at your mathematical model should be clearly stated. Do not oversimplify the prob lem because in that case your answer may not apply. Step 5. List Your References Often, you need to get additional data to solve the problem. The source of all data and information used in your solution, except that contained in the problem statement, should be referenced. References must contain enough information so that your
123 Appendix A: (Continued) supervisor could easily look up your referenced data. Step 6. Develop Model Equations Write down all the governing equations us ing the algebraic symbols to represent the unknowns. Remember the acronym: KISS ( Keep it Simple and Solvable). If necessary make more assumptions to simplify the problem. You may need to add more variables to define the model. Check for cons istency of units used, eg. Both sides of an equation must have the same set of units. All terms being added or subtracted must have the same units. Terms within exponentials or logarithms must be dimensionless. Each equation must be preceded by a line stating what it is or how it is obtained. A ll variables should be clearly identified and defined with appropria te units. The most common mistake is using inconsistent sets of units. Try to use conventional symbols where possible (e.g. x i for liquid mole fractions, L for liquid flow rates, etc.). Graphical correlations would be included here if they are required to solve the problem. You will need one equation for every unknown variable in the problem. If you do not have enough equations, you may need to think about other possible relationships among the variables you have overlooked. If you have more equations than variables, some of the equations may be redundant in na ture. Go back and check your assumptions and model if there is an inconsistency. La ter on, you will be introduced formal methods of analyzing a model to determine if there are enough degrees of fr eedom to solve the problem (called degrees of freedom analysis).
124 Appendix A: (Continued) Step 7. Solve the Equations Equations may be algebraic in nature or they may be in the form of differential equations if there is a time or distance variable involved in the problem. There is powerful software available for solving such equations. TKSolver is great for solving collections of linear and nonlinear equations Mathcad, Matlab and Maple are great for differential equation systems. Simulink is great for ordinary differential equation models. Femlab is great for solving pa rtial differential equations. Maple and Mathematica are great for solving symbo lic equations. You should get familiar with these tools in the course of your engineer ing education. Use the right tool for the problem. Choose a method of solving the mathematical model for the unknowns and execute the solution. You should label each equation when you write it down. Round off answers to reasonable significan t digits because calculator s and computers report 7 or more digits and typically you do not have that kind of accuracy in the data or the model used. Remember the principle Â“GIGO: Garb age in; Garbage OutÂ” as it applies to computer based solutions. Your answer is only as good as what you put in. This is why the next step is important. Step 8. Interpret the Sol ution and Make Conclusions Look at the solution to the equation and interpret the numbers in light of the problem statement. Check if it makes sense physically. For example, if the answer is a fluid velocity, then compare against normally e xpected velocities in the problem context. If it is a temperature, then check if it is very high or very low. Use your intuition and
125 Appendix A: (Continued) common sense to validate your answer. If th e answer is unreasonable, then may be you need to check your assumptions and/or model equations. Using computers and calculators you can avoid errors in arithmetic. Engineering design problems are a different breed. These are typically stated with minimal information and often have multip le answers or solutions. The problem statements are often vague. Eg. Design a bridge to cross the river. You as the engineer must choose the location, the length, width, ma terial of construction, structure etc., and taking into account the requirements of the bridge and constraints imposed by social, economic, safety and political considerations. By the time you graduate, you are expected to pick up the necessary skills to tackle such open ended problems. Example of Engineering Problem-solving Problem Statement: A ball is through into th e air with a velocity of 1.0 m/sec. at a 45 angle. Calculate the time it takes before it hi ts the ground. Assume that the ball is initially 1m above the ground. (This is actually a problem from Physics, but quite similar to engineering problems) Step1. Problem Abstraction First draw a figure depicting the scenario outlined in the problem.
126 Appendix A: (Continued) Needed: time to hit the ground Given: Initial velocity and direction a nd initial location. Abstraction: Ball goes up and is pulled down by gravity and follows a Â“parabolicÂ” path ( from empirical observation of thrown objects) Vertical velocity will decrease, become zero and then become negative. Motion in vertical and horizontal directions can be decoupled. Step 2. List of Variables We will need vertical and horizontal velocities, vx and vy Initial Velocities are known: vxo = (1m/sec) cos 45 = 2 1 m/s vyo = (1m/sec) sin 45 = 2 1 m/s dy = vertical distance traveled by the ball, m t = time, sec 45 1 meter
127 Appendix A: (Continued) Step 3. Basis for Calculations None required for this problem Step 4. Assumptions Ground is flat Air resistance can be neglected Acceleration due to gravity is 9.8 m/sec2 Step 5. List References Physics Text, p.135 Step 6. Develop Model Equations The vertical motion is governed by the equati on (from the definition of acceleration). We define a new variable, y (meters) to indi cate the vertical dist ance from the ground. The variable g is used to indicate the accelera tion due to gravity (assumed constant, another assumption). Acceleration = g dt y d 2 2 Initial position (t=0) y = 1.0 m; Initial ve rtical velocity = sec /2 1m Step 7. Solve Integrating: yov gt dt dy
128 Appendix A: (Continued) (Applying initial velocity) = 2 1 gt Integrating Again: oy t gt y 2 1 2 12 (Applying Initial Position) 1 2 1 2 12 t gt We want to know when the ball hits the ground (y = 0). 1 02 1 2 2 1 t gt Substituting for 28 9s mg sec 385 0 sec 529 0 or t Negative answer is not acceptable. Hence time to hit the ground = + 0.529 sec. Step 8. Interpret Solution The answer seems reasonable from an intuitive point of view. The ball does not travel far before it hits the ground. You can get a physical feel for the answer by throwing the ball yourself and seeing how long it takes to hit the ground.
129 Appendix B: How Do You Solve Problems (HDYSP) Inventory How Do You Solve Problems? In order to provide the best instruction possible, we need to understand how engineering students approach solving problems in this course. Help us obtain this understanding by reading the following sentences and choosing the answer that BEST describes your approach to solving problems. To help, you might think about a typical problem that you have encountered in your other math or engineering courses. There is no right or wrong answers so simply identify the frequency with which you personally use each of the following problem-solving procedures. Please tell what you actually do rather than what you think you should do. This questionnaire is confidentia l and will not be graded. Your responses will be summarized with those of the total group, and your individual answer s will not be used or shared with your instructor or fellow students. There are three sections to this questionnaire: 1. Planning Â– what do you do before you begin to solve a problem? 2. Monitoring Â– what do you do while you are solving a problem? 3. Evaluating Â– what do you do after you finish working on a problem? Choose ONE of the following responses for each statement: 1 = Rarely 2 = Sometimes 3 = F requently 4 = Almost always Part 1 Â– Planning is defined as (1) selecting appr opriate strategies and (2) allocating resources that affect performance. An example wo uld be allocating time or attent ion selectively before beginning a task. 1. When considering a problem, I ask myself what exactly is being asked. 2. To be sure I understand the problem, I read it more than once. 3. Instead of repeating the problem statement, I try to restate it in terms of how I interpret it. 4. I remind myself that I can do the problem with the information given. 5. When I prepare to solve a problem, I write an abstract of the problem statement listing all given information. 6. I define the variables and constants when approaching a problem. 7. I use engineering symbols when I am abstracting a problem. Continued
130 Appendix B: (Continued) 1 = Rarely 2 = Sometimes 3 = Frequently 4 = Almost always Planning continued. 8. Depending on the problem, I draw an engineering sketch. 9. When drawing an engineering sketch, I use engineering paper for a more accurate picture. Part 2 Â– Monitoring is defined as a personÂ’s aw areness of comprehension and task performance. An example is the ability to engage in periodic self-testing while learning. 10. When assumptions are not provided in the problem, I state my own. 11. When necessary, I make further assumptions to simplify the problem. 12. I write down the known quantities of the problem. 13. I list all the variables/unknowns associated with the problem. 14. If needed, I gather additional data to solve the problem. 15. When using additional data not stat ed in the problem statement, I clearly reference it. 16. If the base flow rate or volume or production capacity is not given, then I assume one. 17. When thinking about the basis for my calculations, I restate the base flow rate or volume or production capacity. 18. When developing model equations, I write down all governing equations using algebraic symbols to represent the unknowns. 19. I justify my assumptions in arriving at my mathematical model by clearly stating them. 20. If needed, I use more variables to define the model. 21. I label each equation when I write it down. 22. When developing model equations, I use conventional symbols where possible. 23. I precede each equation with a line stating what it is or how it is obtained (e.g. L for liquid flow rates). 24. For every unknown variable in the problem, I have one equation. 25. When solving for unknowns in a mathematical model, I am able to choose an appropriate method to execute the solution.
131 Appendix B: (Continued) 1 = Rarely 2 = Sometimes 3 = Frequently 4 = Almost always 26. I check for consistency of units (e.g. both sides of an equation must have the same set of units). Part 3 Â– Evaluating is defined as appraising the products and effi ciency of oneÂ’s learning. An example is re-evaluating oneÂ’s goals and conclusions. 28. When checking the solution against the equation, I interpret the numbers in light of the problem statement. 29. I check to make sure the solution makes sense physically (e.g. if the answer is a fluid velocity, then compare against normally expected velocities in the problem context). 30. To validate my answer, I use my common sense/intuition. 31. If my answer seems unreasonable, I re-check my assumptions and model equations. 32. I check the efficiency of my solutions and Â“rethinkÂ” my errors and Â“false starts.Â” Thank you for your time in taking this questionnaire
132 Appendix C: Participant Survey USF Â– University of South Florida Participant Survey This survey will be used to collect anony mous demographic information of students enrolled in Thermodynamics (EGN 3343) during the current academic semester (fall 2004). According to the Institutional Revi ew Board (IRB), the information you provide will remain confidential and used fo r the express purpose of this study. Directions: For each question, select the most appropria te response to reflect the information indicated below. 1. My year of study is: o Freshman o Junior o Sophomore o Senior 2. When did you enter USF? As a: o Freshman o Transfer from a community college o Transfer from another university 3. My declared major is: 4. The number of credit hours I am taking this semester are: o 0-3 credit hours o 4-8 credit hours o 9-12 credit hours o 13-18 credit hours o More than 18 credit hours 5. Outside of this course, the number of hours I am working are considered: o Part-time o Full-time o I do not work at this time
133 Appendix C: (Continued) 6. My current GPA is: o 4.0-3.5 o 3.4-3.0 o 2.9-2.5 o 2.4-2.0 7. How many times have you taken a Thermodynamics class? o First time o Second time o Other 8. My gender is: o Male o Female 9. My year of birth is: 10. My ethnic background is: o African American o Asian/Pacific Islander o American Indian o Hispanic o White/Non-Hispanic o Other 11. My native language is: o English o Spanish o Other 12. For residency purposes, I am considered a: o Florida resident o Out of state resident o Out of country resident Thank you for your time in completing this survey.
134 Appendix D: Pre/Posttest of Thermodynamics Concepts EGN 3343. Thermodynamics. Pretest/Posttest of Concepts. Which of the following units can be used to measure pressure? Check all that apply. a. lbf b. lbf/in2 c. mm Hg c. Pascal d. Bar e. atm Which of the following units can be used to measure specific volume of a fluid? a. cubic feet/lbm b. liter/gm c. gm/cc d. cc/gm e. gm/liter The atomic weight of Oxygen is 16. The atom ic weight of hydrogen is 1. What is the mass in lbs of a lbmole of water? a. 16 b. 17 c. 18 d. 19 e.20 4. 1 gm of water has a volume of 1 cm3. What is the mass of 1 ft3 of water? (1 lbm= 454 g; 1 ft=12 in; 1 in = 2.54 cm) 5. What is the volume occupied by 1 kg of oxygen at 25 C and 1.1 atm? Assume it is an ideal gas and the universal gas cons tant is = 82.5 lit er.atm/(kmol.K) Air is compressed from 1 atm to 2 atm in a co mpletely insulated cylinder. This process is a. an adiabatic process b. an isothermal process c. an adiabatic and isothermal process d. none of the above 6. A pump is used to deliver water to a reacto r. If the pump is to be modeled as a system how would you characterize it ? Check all that apply. a. an open system b. a closed syst em c. both open and closed system d. neither open nor closed system. 7. Grape juice is fermented in a tank to make wine. How will you characterize this process? Check all that apply. a. steady state system b. unsteady state sy stem c. transient process e. continuous process f. batch process 8. If you are stirring a cup of coffee are you transferring any energy to the coffee? 9. A box containing 10 red marbles is separa ted by a thin wall from a box above it containing 10 blue marbles. The wall is removed and the marbles mix with each other. Is this a re versible process?
135 Appendix D: (Continued) 10. A water pump is run by an electric moto r that uses 100 watts of power. If the pump efficiency is 60% how much energy is c onverted into thermal energy in the pump? 11. A fluid is flowing at a stea dy rate through a constant diam eter pipe. There is friction between the wall and the flowing fluid. Will th e velocity be the same at the inlet and the outlet of the pipe? Will the pressure in the pipe be th e same at the inlet and the exit? 12. Can thermal energy be transferred from a room at 75C to a room at 85 C? 13. If a person living in a well insulated apartment (heat neith er enters nor leaves through the walls or windows) leaves her fan r unning will the temperature of the room: a. remain constant b. increase c. decrease 14. A gas is in a cylinder is expanding in volume while the pressure remains constant. Does it do any work? 15. Can a gas expand in volume while the pressure remains constant? 16. Does it take the same amount of energy to raise the temperature of 1 g of water by 1 deg C when it is near room temper ature versus when it is near 80C? 17. The conservation of energy principle applie d to a system says that the total energy entering a system at steady state must equal the total energy leaving the system if there are no chemical reactions taking place inside the system. Consider a mixing tank which is being heated by a 1000 watt (1 watt = 1 joule/s). Water enters and leaves the system at a constant rate. Thermal energy losses from th e system totals 300 calories/s. (1 cal = 4.2 joules). Will the temperature of the fluid leaving the system be greater or less than the entering temperature?
136 Appendix E: Grading Rubric ACTIVITY Excellent Very Good Good Poor Incomplete Engineering Problem-solving 100 pts. (Weight) 99-80 pts. (Weight) 79-69 pts. (Weight) >60 pts. (Weight) 0 pts. (Weight) 1. Problem Abstraction 2. List Variables 3. Basis for Calculation 4. Assumptions 5. List References 6. Develop Model Equations 7. Solve 8. Interpret Solution
137 Appendix F: Web-Based Probl em-Solving Tutorial Survey Survey of Students Using Web-Based Problem-Solving Tutorials Check the answer which best reflects your experience with web-based problem sets Please note the following Conventions: 5. Strongly Agree 4. Agree 3. Neutral 2. Disagree 1. Strongly Disagree Strongly Agree Agree Neutral Disagree Strongly Disagree Mean Do not complete survey if you did not do web-based problems 1. Web-based problems sets are too easy and do not force me to think as well as paper-based problem sets 5 4 3 2 1 1.74 2. I prefer Web-based problems se ts over paper based assignments 5 4 3 2 1 2.34 3. I learned better from paper-based problems sets than web-based assignments 5 4 3 2 1 3.19 4. The help features of web-based problem sets, allowed me to tackle the problem better 5 4 3 2 1 2.97 5. Web-based assignments were difficult to access 5 4 3 2 1 2.38 6. My performance in the course suffered because of the web-based assignments 5 4 3 2 1 2.58 7. Web-based assignments forced me to review my course material 5 4 3 2 1 3.18 8. I liked the fact that I got immediate feedback on whether I did the problem correctly 5 4 3 2 1 2.59 9. I found web-based assignments frustrating because it did not let me see the entire problem at once 5 4 3 2 1 3.19 10. I think I got a fair score in the web-based problem sets 5 4 3 2 1 2.22 11. The problem set was broken down into too many questions 5 4 3 2 1 3.51 12. I prefer to work alone on my homework 5 4 3 2 1 2.31 13. The web-based problem sets took much longer for me to complete 5 4 3 2 1 2.45 14. The web-based problems sets prepared me better for the assessment quizzes and mid-term tests 5 4 3 2 1 2.34 15. The web-based problem sets were frustrating to use 5 4 3 2 1 2.89 16. The web-based problem sets motivated me better 5 4 3 2 1 2.43 17. In the future I would prefer to use paper based homework problems 5 4 3 2 1 3.05 18. I had problems with the technology in using the web based problems, and couldnÂ’t use them effectively 5 4 3 2 1 1.81 19. My performance in the course impr oved because of web-based problem sets 5 4 3 2 1 2.05 20. I did not like the grading system used in the web-based problem sets 5 4 3 2 1 2.65 21. I felt isolated when working the web-based problem sets 5 4 3 2 1 2.04 22. The web-based problem sets offered too much guidance 5 4 3 2 1 2.01 23. I found the web-based problem sets too easy, did not offer enough of a challenge to me 5 4 3 2 1 2.86 24. I learn better when I collaborate with fellow students to solve homework problems 5 4 3 2 1 2.96 25. I chose to solve more problem exam ples than were provided in the web-based problem sets as additional practice 5 4 3 2 1 1.86 26. I would prefer to take the exams and quizzes on the web rather than with paper and pencil 5 4 3 2 1 2.92 27. I would prefer more guidance through the help and hint buttons than was provided to me 5 4 3 2 1 1.80 28. It is too easy to cheat on web-bas ed problems and hence it should not be used 5 4 3 2 1 3.88
138 Appendix F: (Continued) Survey of students on Web-Based Problem-Solving Tutorials Â– page two Check the answer which best reflects your experience with web-based problem sets Please note the following Conventions: 5. Strongly Agree 4. Agree 3. Neutral 2. Disagree 1. Strongly Disagree Strongly Agree Agree Neutral Disagree Strongly Disagree Mean 29. I am familiar with the general problem-solving strategies 5 4 3 2 1 3.78 30. I used general problem-solving st rategies when doing these problems 5 4 3 2 1 1.49 31. I have not had a formal intro duction to engineeri ng problem-solving 5 4 3 2 1 2.32 32. I think an introduction to general pr oblem-solving strategi es would have helped me 5 4 3 2 1 3.08 33. I am learning problem-solving stra tegies through example, but it would be helpful to formalize it 5 4 3 2 1 2.49 34. Circle the % that best describes the amount of time you worked with others on your homework 0 25 50 75 100 Responses to question # 34 28 14 14 4 1 Comments: # of respondents = 74 5 did not complete survey 8 did not attempt survey 13 comments ( see attached Word document)
139 Appendix G: Web-based Homework Survey 04.26.05 Written Comments 1. The only problem with the web-based probl ems are that of the solutions are wrong because they used the wrong information. Also, sometimes the Â“writeÂ” solutions would not go through. When l ooking at the solutions, if they were wrong, showed the wrong way to solve the problem. 2. I didnÂ’t like the online homework. Ma ny solutions were wrong, or at least different from Dr. Smith. I didnÂ’t like it that we werenÂ’t given full solutions so we could study for a test. 3. Quizzes should be online and given more time, they bring down the grades of the students. 4. The homework has a lot of bugs that need to be worked out. 5. On your 3rd attempt of missing a problem most of the time the hints would just blatantly give you the answer. This s hould not happen, too easy then. Maybe a better hint not the answer. Or make th e examples given have #Â’s further form what the get values will give! 6. The web-based homework assignments were ve ry useful. I was much more likely to do the homework, especially if I wa s already using my compy [computer]. That fact alone probably helped bring up my final grade almost a whole letter. Thank you web-based homework and your noble deeds. Thou art a modern day Lancelot Â… here to steal my woman. 7. The online homework is a pain. It does no good. All the things I learned were from going to class everyday. We would be better served to work problems on paper and then turn in th e paper. I often wonder how engineers learned before without online homework. They used pencil and paper! 8. The web-base design would work if all bugs were worked out of the system (i.e. when entering a value you are told it is incorrect, but when the solution pops up, it is the same value that you entered.) Th ese program errors caused students to not know if they were doing the problem correctly. Also, I believe that paper would facilitate learning more effectively. Ju st look @ the grade differences between the morning and afternoon sections. 9. The web homework as more of a problem th an a help. Often times the site would not work or the images need to solve the problem wouldnÂ’t load. The web h.w. was also a nuisance when the answers would contradict what the professor had solved in class or even posted on Blackboa rd. The same problem with 3 different answers is very misleading in a class. The h.w. Â“toolÂ” just complicated the course.
140 Appendix G: (Continued) 10. The online problems would be ok if the pe rson who did the solutions got their act together. Time and again, the program would tell me that I was wrong when I wasnÂ’t. It was hard to build confid ence with the material. The piss-poor quality in the solutions reinforced my belief that the online problem sets are intended to make life easier for the LAZY TAÂ’s and professor than to enhance the learning experience (maybe overworked would have been more appropriate and fair Â… point is, the format of this class was frustrating ). After sitting in on a few afternoon classes, I feel cheated from what could have been an excellent learning experience from what seems to be a very knowing Professo r with exceptional teaching potential. 11. In answer to Question 10 of the survey, Â“I think I got a fair score in the web-based problem setsÂ” Â– DonÂ’t Know! Never got any kind of confirmation on whether or not Jen received my submissions. The we b problems are plagued with mistakes that left me extremely frustrated and wondering if I know what I was doing. I was told by a TA that I Â“should know when Jen mistakenly used wrong valuesÂ”. How would I? When I am supposed to be just learning this material. The previous comments are not coming from someon e with a low grade in this class. I will leave with a strong A but it was after a semester of self-teaching. I think the web problems could be a wonderful tool if they are fixed. I should not have to spend hours figuring out if I know what I am doing or if the problems I am working on have yet again wrong values or a wrong answer. 12. Someone should verify the problems solutions to ensure they are correct before they are posted. The online problems did have a few bugs, but overall I liked the format. 13. The online problems did have a few bugs, but overall I liked the format.
141 About the Author Sally A. Zabel received a B.A. in Interd isciplinary Studies in 1996 and a M.S. in Curriculum and Instruction (In structional Technology) from the University of South Florida in 1997. She has been working fo r the continuing education and professional development office in the College of Engineering at USF since 1998. Recently, the responsibilities of her department have b een expanded to include marketing and public relations for the college. She is cu rrently the Director of the program. Ms. Zabel is an active board member of the Continuing Professional Development Division of the American Society for Engineer ing Education (ASEE). She also serves on USFÂ’s Engineering Executive Committee, and is a participating memb er of the Statewide Systems Operating Committee (SSOC) Â– an oversight body for engineering distance education in the State of Florida.