|USFDC Home | USF Electronic Theses and Dissertations||| RSS|
This item is only available as the following downloads:
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam 2200385Ka 4500
controlfield tag 001 002029099
007 cr bnu|||uuuuu
008 090916s2009 flu s 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0002885
Granger, Benjamin P.
Moderating effect of training content complexity on the relationship between training media and training outcomes
h [electronic resource] /
by Benjamin P. Granger.
[Tampa, Fla] :
b University of South Florida,
Title from PDF of title page.
Document formatted into pages; contains 91 pages.
Thesis (M.A.)--University of South Florida, 2009.
Includes bibliographical references.
Text (Electronic thesis) in PDF format.
ABSTRACT: Web-based training (WBT) and classroom instruction (CI) constitute two training media that are commonly employed by organizations. Although the effectiveness of one medium relative to the other depends on a number of factors (e.g., Sitzmann, Kraiger, Stewart, & Wisher, 2006) this study aims to address several methodological issues common in the extant media research and investigate the moderating role of training content complexity on the relationship between media and important training outcomes. Utilizing a 2x2 experimental design, one hundred forty-two undergraduate students were randomly assigned to one of four training courses. Each training course involved a PowerPoint 2007 training tutorial in which trainees were presented with information about certain PowerPoint functions. The CI training courses included three instructors who presented course material to trainees in a predetermined time frame while the WBT courses gave trainees substantial control over their allocation of time during the training course. Results suggest that trainees in the CI courses spent substantially more time on course-related activities than those in the WBT courses, which led to less knowledge acquisition when trainees in the WBT course were presented with relatively complex training material. These findings suggest that although learner control is generally considered a positive aspect of WBT (e.g., Kinzie & Sullivan, 1989) it can lead to less time-on-task and ultimately less learning and less effective transfer when the training content is complex in nature. Theoretical and practical implications of these findings are also discussed.
Mode of access: World Wide Web.
System requirements: World Wide Web browser and PDF reader.
Advisor: Edward L. Levine, Ph.D.
Transfer of training
t USF Electronic Theses and Dissertations.
Moderating Effect of Training Content Complexity on the Relationship Between Training Media and Training Outcomes by Benjamin P. Granger A thesis submitted in partial fulfillment of the requirements for the degree of Master of Arts Department of Psychology College of Arts and Sciences University of South Florida Major Professor: Edward L. Levine, Ph.D. Jennifer Bosson, Ph.D. Michael Coovert, Ph.D. Date of Approval: December 16, 2008 Keywords: web-based training, e-learning, classroom instruction, learning, transfer of training, time-on-task Copyright 2009, Benjamin P. Granger
Dedication This Masters thesis is dedicated to my loving parents Ed and Liz Granger, whose love and support made this possible and to Lindsey who has sacrificed so much for me throughout this entire process.
Acknowledgments I would like to give much thanks to my major professor, Dr. Edward Levine, for his invaluable guidance and wisdom throughout this project. I would also like to recognize the following individuals for their contributions to the thesis: Dr. Michael Coovert Dr. Jennifer Bosson Meng U. Taing Kyle W. Groff Tena Nock Vera Polyakova
i Table of Contents List of Tables . ..v List of Figures . ...vi Abstract ..vii Introduction .. 1 Advantages and Disadvantages of WBT and Technologically Mediated Training Programs ..3 Advantages ...3 Disadvantages ...4 Blended Learning .5 Which Training Medium is More Effective? ...6 Affective Outcomes .7 Learning Outcomes .8 Moderators 10 Procedural Knowledge .. 12 Key Confounding Variables ...13 Instructional Method ..14 Learner Preference .14 Trainee-Trainee Communication ...15 Trainee-Instructor Communication 16
ii Attributes of the Instructor .17 Current Study and Hypotheses ...18 Time-on-task ..19 Hypothesis 1 ...20 Cognitive Learning 20 Hypothesis 2a .21 Transfer of Training ...21 Hypothesis 2b .22 Hypothesis 3a .23 Hypothesis 3b .23 Affective R eactions 23 Research Question 1 ...23 Research Question 2 ...24 Study Design ..24 Manipulating the Complexity of the Learning Task ..25 Method ...27 Participants .27 Control Variables ... 27 Familiarity with PowerPoint ..27 Comfort with Computers ...28 Ratings of Instructor Effectiveness 28 Dependent Measures ..28
iii Time-on-task ..28 Declarative and Procedural Knowledge .28 Affective Reactions to Training .30 Transfer of Training ...30 Manipulation Checks .31 Perceived Learner Control .31 Subjective Training Content Complexity ..31 Perceived Exam Difficulty .32 Training Conditions ...32 Training Modules and Practice Sessions ...32 CI Conditions .33 WBT Conditions 34 Easy vs. Complex Conditions 35 Procedure ...36 Rater Training 38 Pilot Testing ...39 Results 40 Manipulation Check ..43 Hypothesis Tests 43 Research Questions 52 Supplementary Analyses 53 Discussion ..57
iv Summary of Findings .58 Implications 61 Limitations .62 Future Research ..65 Conclusions 65 References ..66 Appendices .73 Appendix A: PowerPoint Familiarity Scale ..74 Appendix B. Instructor Effectiveness Rating Scale ...75 Appendix C: Personal Time Log 76 Appendix D: Post Course Examination .77 Appendix E: Affective Reactions to Training Course ...82 Appendix F: Transfer Exercise Instructions .. 83 Appendix G: Perceived Learner Control Scale ..84 Appendix H: Perceived Training Content Complexity Scale 85 Appendix I: Perceived Exam Difficulty Scale. ..86 Appendix J: Practice Instructions for the Easy Condition .87 Appendix K: Practice Instructions for the Complex Condition .88 Appendix L: General Instructions to Trainees in the CI Condition ...89 Appendix M: General Instructions to Trainees in the WBT Condition .90 Appendix N: Transfer Rater Guide 91
v List of Tables Table 1 Means, Standard Deviations, and Intercorrelations among the Focal Variables . . 41 Table 2 ANOVA Results for Time-on-task . .44 Table 3 ANOVA Results for Overall Cognitive Learning . ...46 Table 4 ANOVA Results for Declarative Knowledge ..47 Table 5 ANOVA Results for Procedural Knowledge .. .48 Table 6 ANOVA Results for Transfer of Training Effectiveness. 49 Table 7 ANOVA Results for Trainee Affective Reactions ...53 Table 8 Mean Ratings for Instructors in the CI Conditions. .54
vi List of Figures Figure 1 Interaction between Learner Control and Training Content Complexity for Total Exam Percentage. ..46 Figure 2 Interaction between Learner Control and Training Content Complexity for Declarat ive Percentage .. .47 Figure 3 Interaction between Learner Control and Training Content Complexity for Procedural Percentage 48 Figure 4 Interaction between Learner Control and Training Content Complexity for Transf er Effectiveness 50
vii Moderating Effect of Training Content Complexity on the Relationship Between Training Media and Training Outcomes Benjamin P. Granger ABSTRACT Web-based training (WBT) and classroom instruction (CI) constitute two training media that are commonly employed by organizations. Although the effectiveness of one medium relative to the other depends on a number of factors (e.g., Sitzmann, Kraiger, Stewart, & Wisher, 2006) this study aims to address several methodological issues common in the extant media research and investigate the moderating role of training content complexity on the relationship between media and important training outcomes. Utilizing a 2x2 experimental design, one hundred forty-two undergraduate students were randomly assigned to one of four training courses. Each training course involved a PowerPoint 2007 training tutorial in which trainees were presented with information about certain PowerPoint functions. The CI training courses included three instructors who presented course material to trainees in a predetermined time frame while the WBT courses gave trainees substantial control over their allocation of time during the training course. Results suggest that trainees in the CI courses spent substantially more time on course-related activities than those in the WBT courses, which led to less knowledge acquisition when trainees in the WBT course were presented with relatively complex training material. These findings suggest that although learner control is generally
viii considered a positive aspect of WBT (e.g., Kinzie & Sullivan, 1989) it can lead to less time-on-task and ultimately less learning and less effective transfer when the training content is complex in nature. Theoretical and practical implications of these findings are also discussed.
1 Introduction For organizations in the United States, employee training and development is a multi-billion dollar investment (Noe, 2008). The method by which organizations present training material to trainees varies greatly. Millions of workers in the United States conduct work with the use of electronic tools away from the traditional workplace (Hill, Ferris, & Martinson, 2003), and this includes employee training. Because of the dramatic increase in the use of electronic learning media in the last few decades, it is becoming increasingly important for organizations to build a knowledge base regarding the advantages and disadvantages of web and computer-based learning and to compare them with other training media given its long standing role in training instruction. Classroom instruction (CI) refers to training environments in which a trainer presents material to a group of trainees in real time. One of the most prominent characteristics of CI is that it involves face-to-face contact between the trainer and trainees and is perhaps the gold standard against which other media are compared. Web-based training (WBT) or instruction (WBI), on the other hand, refers to training that involves the use of the internet and computers to deliver training material that can be shared, updated, and distributed (Rosenberg, 2001). WBT is part of a broader category of training methods, commonly referred to as computer-based training (CBT) or e-learning. Although much research has been directed at attempting to find out which training medium is more effective (e.g. Arbough, 2000; Gist, Schwoerer, & Rosen, 1989; Kulik &
2 Kulik, 1991; Sankaran, Sankaran, & Bui, 2000;) the current study will expand upon the research in this area by examining the potential moderating effect of training content complexity on the relationship between training media and relevant outcomes (i.e. affective reactions, cognitive learning, transfer of training). Another important issue that has not been properly addressed by the literature is which of the potentially many variables differentiating CI and WBT accounts for the media effects. In other words, although training media are operationalizations of many variables, past research has failed to identify and isolate them in an attempt to investigate which of the variables, individually or in combination, account for the media effects. This trend in the research has led to findings that are difficult to interpret. In fact, it is still unclear whether the media themselves account for the effects often attributed to them. Thus a secondary purpose of this study is to attempt to catalogue and control for the variables that may potentially be confounded with training media, thus providing a true experimental test of the differences between WBT and CI. Finally, the current study will place a special focus on the training of procedural knowledge skills, which has received little attention in the training literature. The thesis begins with a review of the relevant literature regarding the relative effectiveness of WBT and CI. Next, I review the literature on the common criterion variables and moderators identified in past research. I then attempt to catalogue several variables that have been confounded with media in past research and provide a rationale for why each should be controlled in media research. The development of hypotheses and research questions as well as the presentation of a detailed description of the research
3 methodology and a pilot study to assess portions of the methodology follows. A description of the results of the study follows, along with a discussion of the main findings, limitations of the study, and implications for theory and practice. Advantages and Disadvantages of WBT and Technologically Mediated Training Programs Advantages WBT and other technologically-mediated training systems are becoming increasingly popular in organizations, the military, and in educational settings (Salas & Cannon-Bowers, 2001). Because of the pervasiveness of technologicallymediated training methods, it is clear that they must provide significant advantages to the organizations implementing them. For example, WBT has the advantage of being highly flexible (Bently, 1998; Cascio & Aguinis, 2005; Kraiger, 2003; Long & Smith, 2004), adaptive to individual trainees (Drucker, 2000; Kraiger, 2003; Long & Smith, 2004), convenient for trainees (Welsh et. al., 2003), and potentially cost effective (Alexander, 2001; DeRouin, Fritzsche, & Salas, 2004; Kraiger, 2003; Welsh et. al., 2003). In addition to these advantages, it has also been suggested that WBT involves a reduction in total training time (Kulik & Kulik, 1991; Welsh, et. al., 2003), such that trainees may cover with the same amount of material in less time as compared to CI. Additionally, WBT can provide consistent training material worldwide (Bell & Kozlowski, 2002; Long & Smith, 2004; Welsh, et. al., 2003) assuming that equivalent translations are available. This is particularly advantageous for organizations operating globally. It has also been suggested that WBT may actually reduce information overload (Welsh, el. al., 2003), such that trainees can access material at spaced intervals. Pratt
4 (2002) further pointed out that WBT offers a wide range of features that may be beneficial to trainees (e.g. online bulletin boards, email, live chat rooms, etc.). Furthermore, many organizational contexts do not lend themselves to certain types of training programs (e.g. on-the-job training in hazardous environments) due to the potential hazards of a mistake or accident. Certain technologically-mediated training media allow training to occur without the risk of accidents and lost productivity. Thus, another advantage of technologically-mediated training is that it can create a safe training environment (Bell & Kozlowski, 2002). Finally, in WBT, the learner is the focus of the training (Galagan, 2000; Owston, 1997), such that web-based learners are no longer on the receiving end of a massive flow of information. Learners are more active participants in the learning process. Similar to this contention, Lee and Lee (2008) point out that elearning is characterized as a self-directed learning (SDL) method, which is supported by the constructivist educational philosophy. The constructivist philosophy focuses on how the learner builds an understanding of the world through exploration and interaction with the environment (Rovai, 2004). From this perspective, a learners active participation in training is clearly seen as advantageous. Disadvantages Although WBT has several significant advantages to organizations utilizing the technology and to the individual learners themselves, there are several disadvantages to the use of WBT technology. Interestingly, WBT has been shown to have high drop-out rates (Frankola, 2001; Parker, 1999). This is clearly a major problem, but a less noticeable yet related issue associated with WBT is that trainees do not always utilize their increased control over the learning environment to their advantage
5 (Brown, 2001; Steinberg, 1989). That is, although it is often assumed that increased learner control is beneficial to trainees (e.g. Kinzie & Sullivan, 1989) it can also lead to negative outcomes (Bell & Kozlowski, 2002; DeRouin, et. al., 2004). For example, Brown (2001) found that some trainees did not ut ilize the practice time offered to them and consequently reduced their learning of the material. Other important disadvantages of WBT include organizations not fully understanding the technology being used for training (Long & Smith, 2004), organizations failing to plan appropriately for training and failing to develop a clear strategy and goal for training (Long & Smith, 2004). It is important to note, however, that these disadvantages can be problematic for any type of training medium, not just WBT. Additionally, Long and Smith (2004) identified several other disadvantages such as; trainees needing access to computers to participate in WBT, a possible lack of collaboration between trainees, and a lack of guidance from trainers. The last issue is again echoed in the findings of Brown (2001) in that there is evidence that without some guidance, trainees may not effectively utilize training time. Additionally, there is evidence that non-traditional (Holderness, 1998) and older users (Enoch & Soker, 2006) may be at a significant disadvantage when training is computer mediated. Although the use of technologically-mediated training appears to address some of the problems associated with traditional classroom training, the nature of the actual hardware required for the training to occur may lend itself to a number of unique problems (e.g. computer malfunctions, difficulties with internet access or software, etc.) (Owston, 1997). Finally, although one of the advantages of WBT is that it can potentially be cost effective, the
6 development and implementation of WBT technology can be very expensive initially (Cascio & Aguinis, 2005; Hall, 1997; Noe, 2008). Overall, it is important for organizations to take each of these advantages and disadvantages into account when deciding whether or not to use WBT technologies or other technologically-mediated training programs. Blended Learning In order to benefit from the advantages of face-to-face instruction and technologically mediated training, many organizations are utilizing blended learning programs. As is implied in the name, blended learning involves online learning as well as face-to-face instruction (Noe, 2008). In accordance with media richness theory, which contends that greater learning will result when material is presented through numerous media (Daft & Lengel, 1986), blended learning should be an effective method of training employees. Meta-analytic support has been found for this contention as blended learning has been found to be more effective than traditional CI (e.g. Paul, 2001; Sitzmann, et. al. 2006). Although this hybrid training medium is gaining in popularity, it w ill not be addressed in the current study due to the possibility that it will interfere with the secondary purpose of this study: to identify and control for variables that have been confounded with media in past research. Which training medium is more effective? With knowledge of the advantages and disadvantages of WBT and the continuing use of CI in training contexts, two simple questions can be posed: Is WBT more effective than CI or is CI more effective than WBT? Although much research has attempted to
7 answer these seemingly simple questions, parsimonious answers are currently unavailable. One issue making simple answers to these questions difficult is how exactly to define effectiveness. Past research has primarily focused on two training outcomes, the affective reactions of trainees and the extent of their learning. Although much of the empirical work has identified affective reactions and learning as distinct outcomes, learning outcomes are known to be multidimensional. In other words, learning includes changes in behaviors and skills, but also includes affective changes as well (Kraiger, Ford, & Salas, 1993). Thus, learning can be affective, behavioral, and cognitive. However, the focus here is on trainee reactions to the actual training course, not necessarily affective changes that will occur due to training. Thus, affective reactions and learning will be treated as distinct outcomes. Another issue making an answer to the questions of which medium is more effective difficult is the fact that there is empirical support for the use of both training media and much of the literature is in disagreement about the effectiveness of WBT relative to CI. Finally, as was suggested by Clark (1983), research comparing training media has confounded media with a number of other variables. Thus, it is difficult to conclude that training media are responsible for the differences in outcomes based on the available research. The following sections summarize these last two issues in turn. Affective Outcomes One criterion that is commonly measured is trainees affective reactions to training. Reactions can range from reactions to the training medium itself or some aspect of the material being presented. Some of the research comparing WBT to CI shows that trainees prefer WBT to CI (e.g. Sankaran, et. al., 2000). In fact,
8 early meta-analytic findings suggest that students prefer courses that are based on computer instruction (Kulik & Kulik, 1991). Contrary to these findings, however, other research has found that trainees prefer CI to WBT or online training methods (e.g. Rovai & Barnum, 2003). In their study, Rovai and Barnum (2003) further investigated this finding by asking participants to give reasons for their preference in training media. They concluded that participants felt that positive aspects of the instructor in CI would inspire learning and communicating online was tedious and considered ineffective by some trainees. Gist, et al. (1989) found that participants were significantly more satisfied with a behavior modeling training medium as opposed to a computer tutorial training medium. In this case, the tutorial method is more closely analogous to WBT. The behavior modeling method is analogous to blended learning, because it involved face-to-face contact between the instructor and the learners as well as the use of computers. Interestingly, the most recent meta-analytic results suggest that across numerous studies, there is no difference between trainee satisfaction for WBT and CI (e.g. Sitzmann et. al., 2006). However, this same meta-analysis found that trainees were 6% more satisfied with CI than with blended learning. Because of the differences in the findings, one can posit that there are intervening variables involved in these relationships (i.e. moderators and mediators). Learning Outcomes. Much research has already been conducted to address the question of whether WBT is more effective for facilitating learning than CI and as suggested above, the results vary greatly. A large body of research addressing this issue has found no significant difference between the effectiveness of internet based training
9 courses and classroom based training courses. For example, in a meta-analysis, Zhao, Lei, Lai and Tan (2005) found that there was no significant difference between distance learning and face-to-face learning on measures of effectiveness (including measures of learning). In addition, Arbaugh (2000) found no significant difference in post-test scores between an internet based college course section and a classroom based college course section. In their meta-analysis comparing the media, Sitzmann et al. (2006) found that the WBT and CI training media were equally effective for training procedural knowledge. It is important to note, however, that the number of studies included in this meta-analysis that focused on the training of procedural knowledge was relatively small. Overall, these meta-analytic findings lend support to Clarks (1983) contention, that the training medium is not critical for learning and thus, by itself, has no effect on learning. In contrast to this body of research finding no significant difference between online and classroom training media regarding trainee learning, other studies have concluded that WBT or other technologically-based training media are more effective than CI. Maki and Maki (2002) found that college student participants learned more and performed better in online versions of a class compared to those in traditional classroom sections of the course. Research by Kulik and Kulik (1991) supports this finding, such that students performed better on final examinations when academic courses were taught via computer as opposed to in-class lecture. Additionally, Sitzmann et al. (2006) found that WBT was 6% more effective for training declarative knowledge than was CI across a number of studies. On the other hand, some research has resulted in findings suggesting that CI is more effective than WBT (e.g. Gist et. al., 1998; Mottarella, Fritzsche, &
10 Parrish, 2004). For example, Gist et al. (1989) found that when compared to a computer tutorial training method, a behavioral modeling training method that involved trainees watching a model perform tasks was more effective at enhancing learning. Although the literature in this area provides mixed results regarding the effect of training media on trainees affective reactions and learning, both Sitzmann et al. (2006) and Zhoa, et al. (2005) recognized that the studies used in their meta-analyses differed considerably. Therefore, instead of simply attempting to answer the question: Which training medium is more effective, perhaps a more appropriate question is, under what circumstances is it better to utilize one training medium as opposed the other. Moderators. Indeed, research suggests that some settings and task characteristics are better suited to WBT whereas others are better suited to CI (Sitzmann et al., 2006; Welsh et al., 2003; Zhao et al., 2005). Sitzmann et al. (2006) identified several moderators of the relationship between training media (WBT and CI) and learning of declarative knowledge. For example, they found that experimental design was a significant moderator of this relationship, such that CI was 10% more effective than WBT when experimental designs were employed whereas WBT was more effective when quasi-experimental designs were used. Length of training was also found to moderate the relationship such that as the length of training increased, web-based trainees learned more declarative knowledge than classroom trainees. In addition, learner control was found to be a moderator, such that WBT was more effective when more learner control was afforded. Yet another moderator identified by Sitzmann et al. (2006) was instructional method. When instructional methods were the same, WBT and CI were
11 found to be equally effective, but when they differed (i.e. the methods used to convey training material in the CI and WBT conditions differed), WBT was more effective. Another moderator is practice, such that WBT is more effective when it involves practice regardless of whether or not CI involves practice. However, WBT was found to be less effective when it did not involve practice and CI did or did not. Additionally, in another meta-analysis, Zhao et al. (2005) identified several other moderators of the media-outcome relationship. It is important to note however, that Zhao et al. (2005) were primarily interested in the differences between distance education and face-to-face learning environments, and not WBT and CI specifically. However, because WBT is commonly used in distance education and CI is a face-to-face training method, it seems important to include such findings. One interesting moderator was the year of publication such that studies published prior to 1998 reported no significant difference between the effectiveness of distance education and face-to-face learning while those published after 1998 typically found distance education to be more effective. Machtmes and Asher (2000) attribute such a finding to the possibility that newer technologies are able to convey learning material more effectively. Another important moderator was instructor involvement or the extent to which the instructor of a distance education course is involved in the training course. When instructor involvement was low, face-to-face training was more effective. On the other hand, when instructor involvement in the distance education course was higher, distance education was more effective than face-toface methods. Another interesting moderator identified by Zhao et al. (2005) was the education level of the learner. Distance education was found to be more effective for
12 learners with high school diplomas. However, there was no significant difference for learners with a college degree. Finally, content area was also identified as moderator, such that distance education was more effective for learners in the business, computer science and medical science content areas, whereas there was no difference for learners in the social and hard sciences. Knowledge of these moderators can aid practitioners in deciding which medium is favorable for different situations. Another potentially important characteristic that has not been empirically investigated with regard to the relationship between training media and relevant outcomes is training content complexity. Although Welsh et al. (2003) did find that some practitioners do worry about the complexity or depth of the material being trained online, the authors did not cite or find any empirical evidence supporting this concern. The current study will investigate training content complexity as a potential moderator of the media-outcome relations. Procedural Knowledge. Much of the media research has focused exclusively on the training of declarative knowledge, which is formally defined as knowledge of facts and principles as well as the relationships among relevant elements (Kraiger et al., 1993). Thus, as is pointed out by Sitzmann et al. (2006), there has been relatively little attention placed on the differences between WBT and CI for training procedural knowledge. Unlike memory of facts and principles, procedural knowledge refers to information about how a task or action should be performed (Kraiger et al., 1993). In fact, due to the dearth of research comparing WBT to CI for training procedural skills, the moderators presented in Sitzmann et al.s (2006) meta-analysis are only relevant to the training of declarative
13 knowledge. Not only will this study advance the literature by exploring an additional moderator (i.e. training content complexity) of the media-outcome relations, it will explore this issue in the context of both declarative and procedural knowledge. Key Confounding Variables Even though it is clear that web-based and classroom training media differ in substantial ways, much of the research comparing the media has failed to identify and isolate the specific variables by which they typically differ. Thus when the media are compared, it is unclear which of the variables account for differences or the absence of differences in outcomes. As such, it is not very useful to claim that one training medium is superior or equivalent to another when the differences between the media themselves are potentially numerous. Moderators of the media-outcome relationship have been identified (Sitzmann et al., 2006; Zhoa, et al., 2005), but again, research has failed to quantitatively identify the variables that account for these interactions. In some cases, it has simply been assumed that the media themselves are responsible for such findings. Over two decades ago, Clark (1983, 1984) suggested that the research comparing training media was fraught with problems including the fact that many studies confounded media with instructional method and failed to utilize experimental designs. Although Sitzmann et al.s (2006) results support Clarks arguments by finding that training media do not differ in effectiveness when instructional methods were the same; the media research has still not identified which of the numerous variables account for the effects attributed to the media. It is likely that the inconsistent findings present in the media literature are due to this problem. In other words, past research has treated training
14 media as an independent variable and thus failed to control for important confounds. CI and WBT represent operationalizations of potentially many variables and in order to determine if the media account for the effects found in the literature, potential confounds must first be identified and controlled for. Instructional Method. As suggested by Clark (1983), much of the early research comparing training media confounded media with instructional method. Instructional methods refer to the methods within a course that convey the content to be trained (Sitzmann et al., 2006). Examples include lectures, class assignments, group discussion sessions, etc. To provide a concrete example, Maki and Maki (2002) had participants choose either a web-based or classroom version of an introductory psychology course which differed considerably with regard to the instructional methods used. The classroom group was presented with material via lecture and students often engaged in class discussions. Students in the classroom group were also frequently provided with demonstrations. In the web-based group, students were required to engage in a variety of very different activities, such as online quizzes and the completion of outlines. Thus, even though the differences in outcomes between the groups can be misattributed to the media alone, it is unclear whether the instructional methods or the media were responsible for the findings. Learner Preference. As evidenced by the results of Sitzmann et al.s (2006) metaanalysis, trainee preferences may also account for media differences. That is, research design was found to moderate the relationship between training media and training outcomes, such that CI was 10% more effective than WBT when participants were
15 randomly assigned to conditions, but WBT was more effective when participants selfselected into conditions. This issue is echoed by Clark (1983) who suggested that lack of random assignment is a pervasive problem in the early media research. Because many studies comparing media employ college student samples, quasi-experiments are conducted on participants who have already self-selected into college courses (e.g. Arbough, 2000; Maki & Maki, 2002; Sankaran et al., 2000). In these examples, participants presumably chose the media that they felt most comfortable with. Thus, as Clark (1983) suggests, studies comparing the two media can simply utilize random assignment to control for learner preference. Trainee-Trainee Communication. One common difference between WBT and CI is that they often employ different communication tools. In CI courses, learners are usually able to interact with one another face-to-face. This is usually not the case in a WBT or distance education course. When training is delivered via the web, there are usually communication tools imbedded within the training course (Pratt, 2002). Examples include online message or bulletin boards as well as email and instant messaging. These tools allow learners to communicate with each other throughout the course, since they are often unable to communicate face-to-face or verbally. However, Gilbert, Morton, and Rowley (2007) found that there was some concern about the amount and quality of communication between students in online courses. In fact, in their study, some students reported that they would have preferred more student-student interaction. Additionally, in a study conducted by Rovai and Barnum (2003), a sub-set of participants were asked to indicate why they preferred the classroom medium over the web-based medium. The
16 most common response was that the participants felt that the online communication made available was tedious and inefficient (Rovai & Barnum, 2003, pp. 69). This evidence suggests that if the tools for communication between trainees are different for the media, then it may again be unclear which variable (communication tools or media) is accounting for group differences. Trainee-Instructor Communication. Similar to trainee-trainee communication, the tools made available for trainee-instructor interaction often differ between CI and WBT courses. In a classroom setting, the instructor is often readily available for face-to-face interaction whereas this is usually not possible in WBT or distance courses. The findings of Gilbert et al., (2007) and Rovai and Barnum (2003) also suggest that the tools made available for trainee-instructor communication may influence trainees affective reactions to training. However, it seems plausible that the differences in the tools may also affect learning and performance outcomes. For instance, in a classroom environment, trainees can verbally communicate with the instructor to clarify issues and ask questions. In a WBT course, however, concerns about the course or questions about the course material may be conveyed via email or discussion board. In the latter case, feedback may be delayed and even ambiguous. Hara and Kling (2001) found that students enrolled in online distance education courses often reported feeling confused and frustrated about the lack of clear and prompt feedback due to the communication tools provided. Furthermore, Hara and Kling (2001) found that students perceived emails and other communication with the instructor to be ambiguous at times.
17 Similarly, communication researchers have focused some attention on what is known as immediacy. This refers to verbal and non-verbal communication behaviors that can improve the psychological and social connection between trainers and trainees (Brown, Rietz, & Sugrue, 2005). Such behaviors include making eye contact, using humor and examples, addressing students by name or as individuals, etc. (Gorham, 1988). In fact, meta-analytic findings suggest that instructor immediacy is highly related to affective learning (Allen, Witt, and Wheeless, 2006). Clearly, CI environments are more conducive to fostering immediacy than most WBT environments. Thus, aspects of trainee-instructor communication should be controlled for in media studies, but since one of the primary characteristics of CI is the presence of an instructor, it is impossible to fully control for immediacy. However, if multiple instructors are used to t each a classroom course, participants reactions to training are less likely to be dependent on the positive or negative communication style of a single instructor. Attributes of the Instructor. As suggested above, yet another potential problem with media studies is that the attributes of the instructor in a classroom course may affect the way in which trainees respond to training. This was suggested by the results of Rovai and Barnums (2003) qualitative investigation of why trainees preferred a classroom course to a web-based course. Training outcomes may be partially dependent on the skill and/or charisma of the course instructor. If, for example, a classroom course is taught by an exceptional instructor, participants may more effectively attend to the material and thus react favorably to training. The opposite may be expected for a training course taught by an instructor who is less skilled and/or charismatic. Support for this was found
18 by Sitzman et al., (2008) who found that instructor style was strongly related to trainee reactions. Therefore, it seems necessary to control for attributes of the training instructors when comparing training media. This can be accomplished by having multiple instructors involved in the training. In summary, five major confounds present in much of the media research have been identified. They include instructional method, learner preference, trainee-trainee communication, trainee-instructor communication, and attributes of the instructor. Although there are many other potential confounds (e.g. feedback, practice opportunities, course content), the variables listed above have at least been identified by past research but have rarely been properly addressed. The current study will make an attempt to control for these variables in an effort to conduct a true comparison of the media. Current Study and Hypotheses The current study will compare CI and WBT in the context of a PowerPoint 2007 training course. One purpose of this study is to investigate training content complexity as a potential moderator of the relationship between training media and relevant outcomes. Additionally, this study has identified several key variables that have been confounded with media in much of the past research. An attempt was made to control for these potential confounds. Finally, this study adds to the scant research comparing the effectiveness of CI relative to WBT for training procedural knowledge. Since much of the past research has treated training media as an independent variable and neglected important confounds, it is necessary to establish exactly what is meant by training media. In this study, the key difference between CI and WBT will be
19 the presence or absence of an instructor/lecturer. More importantly, the presence or absence of an instructor will constitute a manipulation of learner control, such that trainees in a classroom environment have less control over their learning than those in a web-based environment in which case no instructor is present (Sitzmann, et al., 2008). Thus, this study addresses whether task complexity moderates the relationship between learner control, as operationalized by web-based or classroom delivery media, and training outcomes. Since learner control is the primary variable of interest in this study, WBT and CI will also be referred to as high learner control and low learner control conditions respectively. Moreover, the mechanism by which learner control is proposed to account for outcome differences is time-on-task, such that less time-on-task is expected to lead to unfavorable training outcomes. Time on Task. Because one of primary purposes of this study is to conduct a true test of media differences, the only difference between the two conditions (i.e. CI and WBT) will be the presence of an instructor in a classroom setting or the lack of an instructor in a web-based setting. Again, the presence or absence of an instructor is an operationalization of learner control, such that trainees in a WBT condition will have more control over the time they spend on the course-related activities than trainees in a CI condition. Oftentimes, classroom training settings include built in exercises and practice opportunities (e.g. Arbough, 2000) that are to be conducted within a predetermined time frame. Instructors in classroom settings typically guide learners through their learning experience (Sitzmann, et al., 2008). This is typically not the case in asynchronous web-based training environments, since the goal is for learners to have
20 more control over their learning (Sitzmann, et al., 2006). It would therefore be much easier for trainees in a WBT environment to skip (Brown & Ford, 2002) or speed through training material. This difference in control is therefore likely to lead to differences in the amount of time trainees spend on course-related activities. This is consistent with the earlier research suggesting that learner control may lead to less time-on-task (Brown, 2001). Based on these findings, the following hypothesis is presented. Hypothesis 1: Participants in the CI conditions will spend more time-on-task than participants in the WBT conditions. Cognitive Learning. Learning outcomes represent the second level of Kirkpatricks (1976) training criteria and constitute an important dependent variable in this study. Although learning is often used interchangeably with cognitive outcomes; learning can refer to affective, behavioral, and cognitive changes (Kraiger, et al., 1993). The first hypothesis predicts that trainees taking a CI course will spend more time-ontask than trainees in a WBT course. The primary independent variable here is learner control which, again, is operationalized as the presence or absence of an instructor. Although not synonymous, instructor presence may produce similar effects as performance monitoring, such that some degree of monitoring has been found to lead to increased employee performance (e.g. Komaki, 1986; Komaki, Zlotnick, & Jensen, 1986). Additionally, given the known disadvantages of learner control (e.g. Bell & Kozlowski, 2002; Brown, 2001) characteristic of WBT and other technologically-based training programs, it may become increasingly difficult for trainees to utilize learner control when in complex training environments (Bell & Kozlowski, 2002). In other
21 words, learner control has been shown to lead to less time-on-task (Brown, 2001; Freitag & Sullivan, 1995) and decreased time-on-task may be especially detrimental to trainees presented with complex material. However, when relatively easy training content is presented, learner control and time-on-task may not have the same effects. This leads to the following hypothesis. Hypothesis 2a: Training content complexity will moderate the relationship between learner control and cognitive le arning outcomes such that increased learner control in WBT will lead to less learning relative to the CI condition when complex training material is presented but not when relatively easy training material is presented. Transfer of Training. Although transfer of training is not as common a criterion in media research as trainee satisfaction and learning and performance (Sugrue & Rivera, 2005), it represents one of the ultimate goals of organizational training programs (Noe, 2008). Transfer of training refers to a learners ability to successfully apply information learned from training to other contexts and maintain those behaviors (Broad & Newstrom, 1992; Noe, 2008). It must be noted, however, that there is no conclusive definition of transfer of training to date (Barnett & Ceci, 2002). Despite this fact, the notion of transfer of training remains an important issue in both educational and organizational settings. In the work context, organizational training programs are presumably more closely related to the contexts in which learning should be applied as opposed to other settings (Barnett & Ceci, 2002). Therefore for the purposes of this study, it may be appropriate to measure what is commonly referred to as near transfer. This term refers to the application of learned abilities to work situations that are very similar or identical to the training context (Noe, 2008). In other settings (e.g. educational)
22 far transfer is more heavily focused on as opposed to near transfer (Barnett & Ceci, 2002). In its most basic form, far transfer refers to the application of learned abilities in training to contexts that are not the same as the training context (Noe, 2005). However, as Barnett and Ceci (2002) point out, it may not be very useful to categorize transfer as either being near or far. Rather, they suggest categorizing certain individual aspects of the content and context being trained as being on a continuum of near to far. For the purposes of this study, and in accordance with presumed organizational training goals for transfer of training, most of the individual aspects of the content and context when measuring transfer should be considered near as opposed to far. Because the operationalization of transfer in the study will involve the application of skills learned throughout the PowerPoint 2007 course, I expect similar results for the effects of learner control and training content complexity on transfer as for learning of declarative and procedural knowledge. Hypothesis 2b: Training content complexity will moderate the relationship between learner control and transfer of training outcomes such that increased learner control in WBT will lead to less effective transfer relative to the CI condition when complex training material is presented but not when relatively easy training material is presented. Finally, as is suggested above, differences between the CI and WBT regarding trainee learning and transfer are only expected when trainees are presented with relatively complex training material. Additionally, the mechanism by which these differences are expected to occur is time-on-task, such that decreased learner control in the WBT condition will lead to less time-on-task which will consequently lead to less learning and
23 less effective transfers relative to the CI conditions. Thus, the following hypothesis is presented. Hypothesis 3: Time-on-task will mediate the relationship between learner control and (a) cognitive learning outcomes and (b) transfer of training. Affective Reactions. It is reasonable to speculate that time-on-task will lead to feelings of mastery or satisfaction in learners. It may be that when presented with relatively easy training material, participants possess feelings of mastery regardless of the medium which would lead to no difference between the affective reactions of participants in the CI and WBT conditions. In a complex training situation, participants in the CI condition may feel more satisfied because of the increased time-on-task than those in the WBT condition due to the presence of the instructor. Another complicating factor is learner control which may lead to higher satisfaction but lower levels of learning and transfer. In light of these factors, it is not surprising that the research linking affective reactions and learning and performance has been equivocal. For example, numerous models of training effectiveness posit that affective reactions and learning and performance should be positively related (Kirkpatrick, 1996; Mathieu, Tannenbaum, & Salas, 1992), whereas meta-analytic findings fail to support these relationships (e.g., Alliger, Tannenbaum, Bennet, Traver, & Shotland, 1997). The most recent meta-analysis in this area found that affective reactions do predict learning and performance, albeit weakly (Sitzmann, et. al., 2008). Due to the mixed results in this area, the following research questions are posed: R esearch Question 1: Will there be a difference between WBT and CI in affective reactions?
24 Research Question 2: Will training content complexity moderate the relationship between learner control and affective reactions? Study Design In order to test these hypotheses, an experimental design was utilized. The experiment involved a 2x2 design in which there was a WBT-complex condition, WBTeasy condition, CI-complex condition, and CI-easy condition. The two independent variables in this study are learner control (WBT vs. CI) and training content complexity (easy vs. complex). Again, WBT and CI represent different levels of learner control since the presence of an instructor in the CI conditions will afford less control to participants than the WBT conditions. The four dependent variables of interest are time-on-task, cognitive learning, transfer of training, and affective reactions to the training course. These dependent variables represent three of Kirkpatricks (1976) four criteria for assessing the effectiveness of training. These include trainee reactions (i.e. are trainees satisfied with training?), learning of the training material (i.e. do trainees learn what is being taught?), demonstration of the behaviors taught (i.e. can trainees engage in the specific behaviors being trained, how easily do trainees perform the behaviors, and what is their capacity to perform in other contexts?), and global measures such as productivity (i.e. has training increased organization-wide productivity?). Because the current study will not involve any type of organization that lends itself to global measurements, only three of the four of Kirkpatricks (1976) criterion variables will be included. The actual training course included the presentation of numerous concepts and operations common to PowerPoint 2007. Because the sample consisted of college students enrolled in psychology courses, a PowerPoint 2007 training course was thought
25 to have been considered relevant to participants, possibly increasing motivation to learn and transfer learning to other contexts (Noe, 2008). In support of the importance of this notion, DeRouin, et al. (2004) stressed the importance of making training matter to the learners in a paper outlining specific guidelines for the development and use of learnercontrolled training. Because the PowerPoint 2007 training program involved both the presentation of factual information as well as instructions on how to perform numerous operations, both declarative and procedural knowledge must be acquired to successfully perform the tasks being trained. As stated above, utilizing both declarative and procedural knowledge is of particular importance to the media literature due to the relative dearth of information available on the effectiveness of WBT and CI for training procedural knowledge (Sitzmann et. al., 2006). In fact, Welsh et al. (2003) reported that many practitioners questioned the appropriateness of utilizing WBT technology for training procedural skills, for which procedural knowledge is critical. As such, this study presents an opportunity to address the issue empirically. Manipulating the Complexity of the Learning Task Of primary importance to this study is the way in which the learning tasks complexity is manipulated. Since content complexity is the potential moderator of interest, it is important to make sure that the easy training content actually represents an easier task to trainees than the complex training content. According to Wood (1986), there are several components that make up complexity; component complexity, coordinative complexity, and dynamic complexity. Taken together, manipulation of these proposed complexity components basically
26 involves increasing the number of distinct bits of information that must be processed by the learner, increasing the complexity of the features of the training content, such as timing, frequency, intensity, and location, and finally, introducing the requirement of learner adaptation. Although this description is somewhat convoluted in its presentation, a more parsimonious, yet similar description of how to effectively manipulate the complexity of training content comes from cognitive load theory. According to this perspective, a task and similarly, a learning task, can be made more difficult simply by increasing the number of elements that must be attended to (Van Merrienboer & Ayres, 2005). By increasing the number of elements one must attend to, learners are faced with increased interconnectivity of the elements. In order to learn the material and successfully accomplish a complex learning objective, learners must process many elements and their relationships in working memory (Van Merrienboer, Kester, & Paas, 2006). Although the above discussion refers to aspects of objective learning task complexity, other researchers have stressed the importance of subjective learning task difficulty, which refers to a learners perception of the difficulty of the task (Campbell, 1988; Mangos & Steele-Johnson, 2001). In fact, Campbell (1988) posited that objective task difficulty and subjective task difficulty are actually two distinct constructs. As Campbell (1988) pointed out, different tasks can be perceived differently by different individuals. In a training context, it is quite possible that some trainees perceive the task being trained as being more or less difficult than other trainees. Therefore, for the purposes of the current study, subjective judgments of the training courses complexity were measured.
27 Method Participants Participants consisted of 142 undergraduate students from a large university in the southeast. Students signed up for the study through an online university experiment recruitment website. Participants received extra credit for their participation in the study. Upon signing up for the experiment, individuals were randomly assigned to one of the four conditions: CI-complex (n=38), CI-easy (n=35), WBT-complex (n=34) and WBTeasy (n=35). The demographics of the sample are as follows: The sample consisted of 82.4% females; the average age was 20.54 years ( SD = 4.07); their races were reported as being either Caucasian (55.6%), Hispanic, (16.2%), African American (14.1%), or Asian (7%); and their levels in college were reported as being either a freshman (31%), sophomore (26.8%), junior (18.3%), senior (20.4%) or other (2.8%). Control Variables Familiarity with PowerPoint Prior to the training course, trainees were asked to provide some indication of how familiar they were with PowerPoint. This12 item scale (Appendix A) required participants to rate their familiarity with several specific operations common to PowerPoint on a scale from 1 (Extremely Unfamiliar) to 5 (Extremely Familiar). Example items include Opening a blank PowerPoint presentation and Including Footers into a PowerPoint presentation. Trainees responses to the 12
28 items were averaged to obtain a single PowerPoint familiarity score for each trainee. The measure showed good internal consistency ( = .92). Comfort with computers. Trainees were asked to indicate the extent to which they were comfortable with using computers using a single item. They rated how comfortable they were with computers on a scale from 1 (extremely uncomfortable) to 4 (extremely comfortable). Ratings of Instructor Effectiveness. Following the training course, participants in the CI conditions were asked to indicate their overall reactions toward each of three instructors of the course (Appendix B). Participants were asked to rate each instructor individually on a 0-10 scale in which low numbers represent poor reactions and high numbers represent positive reactions. Dependent Measures Time-on-task. Time-on-task was measured by providing all participants with a personal time log (Appendix C). The time log included start and end times for each of the six distinct portions of the training course. That is, participants were asked to log in their start and end times for each of three distinct training modules as well as three accompanying practice sessions. Trainee total time-on-task was computed by adding the number of minutes each trainee spent on all six portions of the training course. Thus, time-on-task was operationalized as the number of minutes each trainee engaged in course-related activities. Declarative and Procedural Knowledge Upon completion of the training course, participants completed a 20 question multiple choice exam (Appendix D). Each question
29 included 4 possible options. 10 of the questions on the final exam measured acquisition of declarative knowledge by requiring trainees to have an understanding of the different definitions and concepts associated with PowerPoint 2007 (e.g., Which of the following options best describes the purpose of the Ribbon within PowerPoint?). The additional 10 questions measured procedural knowledge acquisition by requiring participants to have an understanding of the steps required for the successful completion of certain tasks common to PowerPoint (e.g., Which of the following is the correct sequence for using the Ribbon to insert pictures into your slideshow?). The final exams for all conditions were identical. However, it was expected that trainees in the complex conditions, who received training in engaging in more advanced PowerPoint 2007 functions, would perform better than trainees in the easy conditions. Prior to taking the multiple choice exam, participants in the easy conditions were informed that the material they covered in the training course may not be sufficient for the successful completion of all questions on the test. Trainees in the easy courses were told to simply do their best. Since this represents a potential confound, cognitive learning outcomes were operationalized as the percentages of relevant questions answered correctly. For instance, the complex courses prepared trainees for all 20 questions in the examination. Thus, their total percentages were calculated by summing the number of correct answers and dividing this number by 20. However, only 11 of the 20 questions were relevant for trainees in the easy conditions. Thus, their total percentages were calculated by summing the number of correct answers to the 11 questions and dividing that number by 11. Similarly, trainee declarative and procedural knowledge acquisition were operationalized as the percentages of relevant
30 questions answered correctly. Overall, the 20 question multiple choice exam showed reasonable internal consistency ( = .70). Affective Reactions to Training Trainees affective reactions to the training course were measured using a newly created 4 item scale (Appendix E). Example items include: Overall, I feel satisfied with the training course and I would like to learn under similar conditions in the future. Trainees were instructed to provide responses ranging from 1 (strongly disagree) to 5 (strongly agree). Trainees responses were averaged to create a single affective reaction score for each participant, where high scores reflect satisfied trainees. This measure showed good internal consistency ( = .86). Transfer of Training In order to measure the effectiveness with which trainees transferred their learning to an actual task, participants were asked to create a short, 3 slide, PowerPoint 2007 presentation from scratch. A very limited number of parameters were given to trainees (as may be the case in a real educational or organizational setting) and trainees were instructed to use the skills that they learned in the training course to successfully complete the task (See Appendix F for the transfer exercise instructions). The effectiveness of the individual PowerPoint presentations were assessed independently by 3 trained research assistants. Specifically, raters were instructed to provide a single score ranging from 0 (Creator of presentation did an extremely poor job of reaching the ultimate goal and shows that he or she does NOT understand any of the skills taught in the training course) to 10 (Creator of presentation did a superior job of reaching the ultimate goal and shows that he or she completely understands and can utilize ALL skills taught in the training course). Again, higher scores represent effective
31 transfer of training. Spearman-Brown reliability was estimated for the ratings (.91) suggesting that there was a high degree inter-rater agreement. B ecause the inte r-rater agreement was good, the three ratings were averaged, yielding a single transfer effectiveness score for each trainee. Manipulation Checks Perceived Learner Control. Following the training courses, participants indicated the extent to which they perceived that they had control over their allocation of time and learning throughout the training course on a 4 item, 5 point Likert scale (Appendix G). Example items include: Overall, I felt that I was in control of the time I spent learning the material in the training course and I should have had more control over the time I spent on the training course
32 Perceived Exam Difficulty Immediately following the post-course examinations, participants completed a 3 item, 5 point Likert format, scale measuring the extent to which they felt that the examination was difficult (Appendix I). Example items include: Overall, I felt that this examination was very difficult and I feel that this examination was relatively easy
33 CI Conditions Both CI training sessions (easy and complex) took place in computer labs made available by the psychology department. Each trainee was provided with his/her own computer and work station in order to follow along with the training course. The two CI training courses took place in the same computer lab, but at different time slots, since the courses differed in complexity. The pace of the classroom conditions were determined during pilot testing. The predetermined pacing for the CI-complex condition is as follows: 6 minutes for training session 1, 6 minutes for practice 1, 9 minutes for training session 2, 5 minutes for practice session 2, 7 minutes for training session 3 and 5 minutes for practice session 3. The total time predetermined for the CIcomplex condition was 38 minutes. The predetermined pacing for the CI-easy condition is as follows: 4 minutes for training session 1, 4 minutes for practice 1, 7 minutes for training session 2, 5 minutes for practice session 2, 8 minutes for training session 3 and 4 minutes for practice session 3. The total time predetermined for the CI-easy condition was 32 minutes. Prior to beginning the training course, an instructor stressed the importance of trainees following along with the instructor throughout the course and informed trainees that they would be monitored during practice sessions to ensure that they were working on the practice exercises for the full time allotted. In order to control for communication with other trainees and with the instructor, trainees were instructed not to communicate with other trainees or the instructor. To control for aspects of a single instructor, three different instructors led one module each. That is, both CI conditions had a total of three different instructors. The instructors were the same for all CI training sessions.
34 WBT Conditions Trainees in both WBT conditions (easy and complex) completed the training course in the same physical environment as the CI conditions, except that there was no instructor present during the actual training. Each trainee completed the course in a computer lab with several others working on the same course. Like the CI conditions, prior to beginning the training course, participants were instructed not to communicate or work with other individuals in the room. All trainees were provided with computers and personal workspaces. Web-based trainees were given more control over their own learning due to the absence of any instructor. Moreover, were also explicitly instructed to allocate their time as they saw fit. However, trainees in the WBT conditions were given recommendations for how long they should spend on the training course. These recommendations mirrored the predetermined time frames that were followed by the instructors in the CI conditions. In addition to the material presented to the learners visually through PowerPoint, the training modules for the WBT conditions included audio instructions. The audio was created by having three research assistants read from previously created scripts. The same scripts were used by the instructors in the CI conditions. During the WBT conditions, a single research assistant was available for communication with the trainees just prior to training in order to brief trainees. As mentioned above, like participants in the classroom conditions, participants in the WBT conditions were strongly urged not to communicate with other participants. A second research assistant remained in the room during the training course to aid participants with any technical issues (e.g., computer malfunctions).
35 Easy vs. Complex Conditions In both the WBT and CI versions of the training course, trainees were required to learn a number of steps necessary when creating and using PowerPoint 2007. The complex conditions (web and classroom) required trainees to learn exactly the same material. As described above, the scripts used by the instructors in the CI-complex condition and the readers for the WBT-complex audio were identical. Moreover, the visual material available to trainees in the two complex conditions was also identical. Thus, the instructional method, which has been found to be an influential variable in determining the relative effectives of one training medium over the other (Sitzmann et al., 2006), was held constant for both conditions. Relative to the easy conditions, the complex training conditions required trainees to learn operations in PowerPoint 2007 that are substantially more advanced and required a more sophisticated understanding of PowerPoint. For instance, in addition to learning how to create new slide shows, trainees in the complex condition were required to learn about custom animations (including pictures and icons), SmartArt (PowerPoint operation that allows presenters to better organize material visually), as well as several other advanced operations common to PowerPoint 2007. The easy conditions (WBT-easy and CI-easy) did not require learners to master tasks non-central to creating a basic PowerPoint 2007 presentation. The tasks presented to participants in the easy conditions included selecting slide designs and colors, inputting text into the body of the slides, including titles, navigating through multiple slides, etc. These basic tasks were also covered in the complex conditions, but substantially less time was spent on them in order for the course to move ahead to the
36 presentation of more advanced operations. Thus, participants in the complex conditions were presented with more distinct bits of information and were required process the information in a relatively short period of time. Procedure Upon signing up for the study on the online experiment recruitment system, participants were randomly assigned to one of the four PowerPoint 2007 training courses described above. Participants were then designated a study time. Between 8 and 20 trainees participated during a single study time. Upon entering the computer lab and just prior to the actual training course, participants were given packets that included all study materials. The packets included the personal time log, practice instructions, all study measures mentioned above, and a unique identification number. The identification number was later used to match participants with their responses on the various measures. Once given the packet, participants were first asked to read and sign an informed consent form. Participants were then asked to read a short paragraph which included the general instructions (See Appendices L and M for the general instructions given to trainees in the CI and WBT conditions respectively) for participation in the course and complete the comfort with computers measure as well as the familiarity with PowerPoint measures. The experimenter then stressed to participants the importance of refraining from communication with others during the entire training course. Participants were instructed that they could only communicate with a technical assistant who was made available throughout the duration of the course to help with computer and equipment issues. Once all participants in the room were finished completing the preliminary measures and
37 indicated that they understood all directions, they were instructed to begin the actual training course. At this point, instructors in the CI conditions began the training course. In the WBT conditions, the experimenter left the room and only the technical assistant remained. Immediately following the completion of the entire training session, participants completed a short survey which included measures of perceived difficulty of the training course, perceived learner control, liking for each instructor (CI conditions only), and overall affective reactions to the training course. Next, trainees completed the 20 item multiple choice examination, measuring their learning of the concepts and procedures presented throughout the course. Trainees were instructed to close out the actual training course while completing the examination. They were also explicitly told to treat it as a real test. Immediately following the examination, trainees completed a measure of the perceived difficulty of the examination. Finally, trainees created a PowerPoint presentation given a set of predetermined parameters. The packet that was given to trainees included instructions for this step. Each participant was instructed to include their personal identification number on the first slide of their PowerPoint presentation. Once participants completed their PowerPoint presentations, they were instructed to save the slideshow they had just created on the desktops of their individual computers so that the technical assistant would be able to access each presentation at a later time. Upon completion of the PowerPoint presentation, each participant turned in his/her training packet and left the room.
38 Rater Training As mentioned above, participants in the study were asked to create PowerPoint presentations from scratch utilizing the skills taught in the training course. Three independent raters assigned scores ranging from 0 10 to each presentation. In order to ensure that the raters were familiar with material trained in the actual course prior to rating, each was put through two 30 minute frame-of-reference training sessions. Each rater was first asked to go through the easy condition course. They were then provided with hard copies of the instructions that were given to trainees during the course (See Appendix F) and were provided with written descriptions of what several of the rating values should indicate (Appendix N). For example, they were informed that a rating of 0 should be assigned to a presentation if it indicated that the c reator of presentation did an extremely poor job of reaching the ultimate goal and shows that he or she does NOT understand any of the skills taught in the training course. After each rater had completed the easy training course and had reviewed and indicated that they understood all instructions, they independently rated the PowerPoint presentations created by trainees in both easy conditions. Once the raters were finished rating all easy presentations, the same steps were followed for the complex conditions. Raters went through the complex course, familiarized themselves with the skills that trainees should have been familiar with following training, and rated the PowerPoint presentations created by participants in both complex conditions.
39 Pilot Testing In order to ensure that the independent variables were successfully manipulated, the training course was pilot tested on a total of 61 undergraduate participants who were not included in the final sample. The results of the pilot study indicated that the learner control and training content complexity manipulations were effective and no substantial changes were made to the procedures or implementation of the training course.
40 Results Means, standard deviations and correlations among the focal variables are presented in Table 1. The descriptive statistics reported in Table 1 suggest that in general, trainees tended to score quite well on the post-course exam ( M = 86.04, SD = 13.33). Similar means and standard deviations were found for declarative knowledge ( M = 85.70, SD = 14.94) and procedural knowledge ( M = 86.36, SD = 16.63). These findings suggest that overall, the post-course exam was relatively easy for trainees and that the distribution of scores may not be normal. Indeed, the distribution of total exam percentage scores was negatively skewed (skewness = -1.23, standard error = .20). This is likely due to a ceiling effect for scores on the post-course exam. Transfer effectiveness scores also tended to be high ( M = 7.00, SD = 2.12) and like the cognitive learning outcomes, this distribution of transfer effectiveness scores was negatively skewed (skewness = -1.11, standard error = .22). Moreover, on average, trainees in all conditions tended to be relatively satisfied with the training course ( M = 3.34, SD = .90) and again the distribution of affective reactions was negatively skewed (skewness = -1.17, standard error = .20). To test for significant departure from normality in the outcome distributions, 95% confidence intervals were computed around the skewness statistics for each outcome. None of the confidence intervals contained 0, suggesting that the cognitive learning, transfer effectiveness, and affective reaction distributions all departed significantly from normality.
41 Variable 123456789101112 1. Learner Control( -) 2. Task Complexity-.03( -) 3. Computer Comfort-.01.14( -) 4. PowerPoint Familiarity.06.01.24**(.92) 5. Time-on-task -.86**.20* .02-.14( -) 6. Overall Cognitive Learning-.42**-.19*.12.10.36**(.70) 7. Declarative knowledge-.40**-.18*-.01-.02.37**.80**(.52) 8. Procedural Knowledge-.32**-.15.20*.16.25**.88**.41**(.54) 9. Transfer Effectiveness-.04-.38**-.05.05-.02.45**.39**.37**(.91) 10. Affective Reactions-.07.02.12.21*.06.11.06.12-.04(.86) 11. Perceived Control.70**-.03.03.16-.63**-.18*-.18*.12-.03.25**(.89) 12. Perceived Complexity-.04.22**-.21*-.37**.18*-.30**-.23**-.27**-.29**-.42**-.26**(.80) Mean1.491.513.343.0026.6586.0485.7086.367.003.883.641.79 SD.50.50.801.0010.4313.3314.9416.632.12.901.26.78 Note: For learner Control, low learner control was coded as 1 and high learner control was coded as 2. For Task Complexity, eas y was coded as 1 and Table 1 Means, SDs, and Intercorrelations Among the Focal Variables complex was coded as 2. Reliability estimates are reported on the diagonals. p < .05 ** p < .01
42 Interestingly, although PowerPoint familiarity and comfort with computers were measured in order to control for their potential effects on the outcome variables, their relations with the outcomes variables were quite weak overall. Its conceivable that individuals who are highly familiar with PowerPoint prior to the training course, would tend to perform well on learning tasks measuring knowledge of PowerPoint. However, PowerPoint familiarity was not significantly related to overall cognitive learning ( r = .10, n.s. ), declarative knowledge ( r = -.02, n.s. ), procedural knowledge ( r = .16, n.s. ), or transfer effectiveness ( r = .05, n.s. ). Trainees who reported being more familiar with PowerPoint prior to training did, however, tend to have more positive affective reactions to the training course ( r = .21, p < .05). General computer comfort showed a similar pattern of weak relationships with the outcome variables, at it was correlated significantly with only procedural knowledge ( r = .20, p < .05). Based on these preliminary results, it seemed unnecessary to control for these variables in the primary analyses. Additionally, several other important results are shown in Table 1, including the non-significant relations between affective reactions and overall cognitive learning ( r = .11, n.s. ). Affective reactions were also found to have a non-significant correlation with transfer effectiveness ( r = -.04, n.s. ). However, as would be expected, all cognitive learning outcomes were found to relate significantly with transfer effectiveness, such that transfer effectiveness was significantly correlated with measures of overall cognitive learning ( r = .45, p < .01), declarative knowledge ( r = .39, p < .01) and procedural knowledge ( r = .37, p < .01).
43 Manipulation Check In order to ensure that learner control was effectively manipulated, an independent samples t-test was conducted. The learner control manipulation was effective, t (140) = -11.60, p < .001, such that trainees in the WBT conditions ( M = 4.53, SD = .69) perceived significantly more control over their learning than those in the CI conditions ( M = 2.77, SD = 1.06). Additionally, an independent samples t-test was conducted to ensure that trainees in the complex conditions perceived the course to be more complex than those in the easy conditions. This was confirmed, t (139) = -2.53, p < .05, such that trainees in the complex conditions ( M = 1.95, SD = .78) did perceive the course to be more complex than trainees in the easy conditions ( M = 1.62, SD = .74). Although participants were randomly assigned to one of four conditions, it was necessary to ensure that there were no systematic differences between the groups on potentially important demographic variables. ANOVAs were conducted to determine if there were any differences between the conditions on the demographic variables measured. There were no significant differences among the four conditions regarding the proportion of males and females F (3, 137) = .67, p > .05, mean age F (3, 137) = .81, p > .05, race F (3, 137) = .075, p > .05, or level in college F (3, 137) = .43, p > .05. Moreover, there were no significant differences between the conditions regarding trainees average level of comfort with computer usage F (3, 135) = .99, p > .05, or self-reported familiarity with PowerPoint F (3, 135) = 1.69, p > .05. Hypothesis Tests To test hypothesis 1, that trainees in the high learner control condition (WBT) would spend significantly less time on course related activities than those in the low
44 learner control condition (CI), a factorial ANOVA was conducted. As shown in Table 2, hypothesis 1 was supported as learner control had a significant main effect on time-ontask, F (1, 138) = 489.44, p < .001, = .78. Overall, trainees in the low learner control condition ( M = 35.5, SD = 3.67) spent twice as many minutes on average as those in the high learner control condition ( M = 17.56, SD = 6.54). As expected, trainees in the complex condition ( M = 28.67, SD = 11.43) spent slightly more time on average than those in the easy conditions ( M = 24.52, SD = 8.85), F (1, 138) = 20.65, p < .001, = .13. Interestingly, there was also a significant interaction between learner control and training content complexity, F (1, 138) = 15.48, p < .001, = .10, such that there was no significant difference between average time-on-task for the high learner control easy ( M = 17.31, SD = 6.71) and complex ( M = 17.80, SD = 6.46) conditions, whereas trainees in the low learner control-complex condition ( M = 38.68, SD =.99) spend significantly more time-on-task than trainees in the low learner control-easy condition ( M 31.94, SD = 1.79). Overall, support was found for hypothesis 1, meaning that trainees given a high level of control over their learning spent substantially less time-on-task than trainees given little control. Sum of Squares df Mean Square F p Intercept99059.43199059.434419.39.000 Learner Control11172.44111172.44498.44.000 Complexity462.961462.9620.65.000 Learner Control* Complexity346.891346.8915.48.000 Error 3093.241383093.24 Total116229.0014222.42 Table 2 ANOVA Results for Time-on-task Several of the following analyses involve ANOVA which include each of the training outcome variables used as the dependent variables and since it was shown that
45 these distributions deviated significantly from normality, it seemed important to address this issue prior to the primary analyses. Moreover, when ANOVA was conducted, I also conducted Levenes test for equality of variances. In most cases the results suggested significantly non-equal variances among the groups. The latter and former issues represent violations of the normality and equality of variance assumptions of ANOVA respectively. However, since the groups sizes were relatively equal and ANOVA is robust to moderate violations of the equality of variance assumption (Box, 1954) and the normality assumption (Hays, 1994) there seemed to be little need for concern about these violations. In order to test hypothesis 2a, that training content complexity moderates the relationship between learner control and cognitive learning outcomes, several factorial ANOVAs were conducted. As a first test of hypothesis 2a, I utilized overall cognitive learning as the dependent variable of interest. In other words, I first tested whether training content complexity moderates the relationship between learner control and cognitive learning. As shown in Table 3, the results of this analysis confirmed that training content complexity did in fact moderate the relationship between learner control and overall cognitive learning, F (1, 138) = 12.21, p < .01, = .08. More importantly, as shown in Figure 1, the moderation was in the expected direction, such that there was no significant difference between the high learner control-easy ( M = 86.49, SD = 10.89) and low learner control-easy conditions ( M = 89.7, SD = 12.24) on the percentage of questions answered correctly whereas trainees in the low learner control-complex condition ( M = 92.11, SD = 8.1) acquired significantly more knowledge than those in the high learner control-complex condition ( M = 75, SD = 15).
46 Sum of Squares df Mean Square F p Intercept104.721104.728033.57.000 Learner Control.441.4433.60.000 Complexity.111.118.24.005 Learner Control* Complexity.161.1612.21.001 Error 1.79138.013 Total107.63142 Table 3 ANOVA Results for Overall Cognitive Learning Note: Dependent variable is Total Exam Percentage Figure 1. Interaction between Learner Control and Training Content Complexity for Total Exam Percentage Because the final examination consisted of questions measuring declarative and procedural knowledge, it seemed useful to find out if the above mentioned findings would hold for the two types of cognitive learning outcomes. First, a factorial ANOVA was conducted to find out if training content complexity moderates the relationship between learner control and the acquisition of declarative knowledge. As shown in Table 4, the results of this analysis mirrored those of the previous analysis, F (1, 138) = 11.61, p < .01, = .08, such that there was no significant difference in mean levels of declarative knowledge acquisition between trainees in the low learner control-easy ( M = 90.59, SD = 14.13) and high learner control-easy ( M = 86.29, SD = 13.52) conditions, 50 60 70 80 90 100 EasyComplex WBT CI
47 whereas trainees in the low learner control-complex condition ( M = 92.37, SD = 7.86) acquired significantly more declarative knowledge than those in the high learner controlcomplex condition ( M = 73.14, SD = 15.68). A graph of the interaction is presented in Figure 2. Sum of Squares df Mean Square F p Intercept103.861103.866112.95.000 Learner Control.491.4928.87.000 Complexity.111.116.73.010 Learner Control* Complexity.191.1911.61.001 Error 2.35138.02 Total107.45142 Table 4 ANOVA Results for Declarative Knowledg e Note: Dependent variable is Total Declarative Percentage Figure 2. Interaction between Learner Control and Training Content Complexity for Declarative Percentage As a final test of hypothesis 2a, I examined whether training content complexity moderates the relationship between learner control and procedural knowledge acquisition. 50 60 70 80 90 100 EasyComplex WBT CI
48 Again, as shown in Table 5, hypothesis 2a was supported, F (1, 138) = 5.27, p < .05, = .04. Trainees in the low learner control-easy ( M = 89.71, SD = 16.58) and high learner control-easy ( M = 86.67, SD = 15.55) conditions did not differ significantly whereas trainees in the low learner control-complex condition ( M = 91.84, SD = 11.36) acquired significantly more procedural knowledge than those in the high learner control-complex condition ( M = 76.47, SD = 18.89). A graph of this interaction is presented in Figure 3. Overall, substantial support for hypothesis 2a was found. Sum of Squares df Mean Square F p Intercept105.521105.524431.66.000 Learner Control.391.3916.29.000 Complexity.101.104.16.043 Learner Control* Complexit y .131.135.27.023 Error 3.29138.02 Total109.81142 Table 5 ANOVA Results for Procedural Knowledge Note: Dependent variable is Total Procedural Percentage Figure 3. Interaction between Learner Control and Training Content Complexity for Procedural Percentage 50 60 70 80 90 100 EasyComplex WBT CI
49 In order to test hypothesis 2b, that training content complexity will moderate the relationship between learner control and transfer of training effectiveness, a factorial ANOVA was conducted. Similar to cognitive learning outcomes and as shown in Table 6, the results suggest that task complexity did moderate the relationship between learner control and transfer effectiveness, F (1, 119) = 5.29, p < .05, = .04. Moreover, the moderation was in the expected direction such that there was no significant difference between the high learner control-easy ( M = 8.06 SD = 1.21 ) and low learner controleasy conditions ( M = 7.59, SD = 2.10) on the effectiveness with which they transferred their learning to a task, whereas trainees in the low learner control-complex condition ( M = 6.68, SD = 1.81 ) transferred their training significantly better than those in the high learner control-complex condition ( M = 5.54, SD = 2.53 ). A graph of this interaction is presented in Figure 4. Overall, hypothesis 2 was fully supported as training content complexity was indeed found to moderate the relationship between learner control and cognitive learning outcomes as well as transfer of training. As expected, it was consistently found that learner control was detrimental to learning and transfer when trainees were presented with relatively complex training material, but learner control had no such effect when training material was relatively easy. Sum of Squares df Mean Square F p Intercept5878.515878.51568.94.000 Learner Control3.318.104.22.168 Complexity85.52185.5223.62.000 Learner Control* Complexity19.80119.805.29.023 Error 445.871193.75 Total6583.69123 Table 6 ANOVA Results for Transfer of Trainin g Effectivenes s Note: Dependent Variable is Average Transfer Effectiveness Score
50 Figure 4. Interaction between Learner Control and Training Content Complexity for Transfer of Training Hypothesis 3a predicted that time-on-task would mediate the relationship between learner control and cognitive learning outcomes. According to Baron and Kenny (1986), several relationships must be established in order to demonstrate mediation. First, there must be a relationship between the predictor (learner control) and criterion variables (cognitive learning outcomes). As shown in Table 1, there were significant bivariate correlations between learner control and all three cognitive learning outcomes (total exam percentage, procedural percentage, and declarative percentage). Second, the predictor (learner control) must be related to the proposed mediator (time-on-task). The bivariate correlation, as shown in Table 1, between learner control and time-on-task was significant. Third, there must be a relationship between the proposed mediator (time-ontask) and the criterion variables (cognitive learning outcomes). Again, in Table 1, the bivariate relationships between time-on-task and all cognitive learning outcomes are 0 2 4 6 8 10 EasyComplex WBT CI
51 significant. Finally, in order to show mediation, the effect of the predictor should be suppressed (partial mediation) or become non-significant (full mediation) once the proposed mediator is included into the regression equation. Mediated regression (Baron & Kenny, 1986) was used to test this final requirement. Each cognitive learning outcome was regressed on the predictor and mediator individually. In the first regression, overall cognitive learning was included as the criterion. Learner control ( = -.42, p < .001) was included in step 1 of the regression and the proposed mediator, time-on-task, was included added in step 2. After entering time-ontask into the regression equation, the relationship between learner control and total exam percentage remained unchanged ( = -.44, p < .01). Thus, time-on-task does not appear to mediate the relationship between learner control and total knowledge acquisition. Hypothesis 3a was tested again, using declarative knowledge acquisition as the criterion variable. Once time-on-task was added to the regression equation in step 2 of the regression, the relationship between learner control and declarative knowledge acquisition ( = -.313, p < .05) dropped, but remained significant, again suggesting that time-on-task did not fully mediate the relationship between learner control and declarative knowledge acquisition. Finally, when procedural knowledge acquisition was included as the criterion variable, time-on-task was not found to mediate the hypothesized relationship. The relationship between learner control ( = -.41, p < .05) and procedural knowledge acquisition was not suppressed when time-on-task was added into the regression equation in step 2. These results suggest that time-on-task does not mediate the relation between learner control and procedural knowledge acquisition.
52 According to hypothesis 3b, time-on-task was expected to mediate the relationship between learner control and transfer of training. However, as shown in Table 1, neither the first nor second conditions of mediation (see Baron & Kenny, 1986) were met as average transfer scores were not significantly correlated with learner control or total time-on-task. Overall, hypothesis 3 was not supported. Research Questions Although perhaps not as critical as learning and transfer criteria, affective reactions are nonetheless important to consider in a training context. Since differences, if any, between WBT and CI in trainee affective reactions can have substantial practical implications, I posed two research questions regarding this training criterion. Specifically, my research questions dealt with the difference between trainees reactions to the WBT and CI courses as well as the potential moderating role of training content complexity on the relationship between learner control and affective reactions. To investigate these questions, a factorial ANOVA was conducted. As shown in Table 7, not only were there no significant differences between the high (WBT) and low (CI) learner control groups in affective reactions, F (1, 138) = .72, p > .05, but there was no significant interaction between training content complexity and learner control when trainee affective reactions was the dependent variable, F (1, 138) = .012, p > .05. Overall, it appears that trainees did not prefer one medium over the other.
53 Sum of Squares df Mean Square F p Intercept2128.6312128.632595.47.000 Learner Control.522.214.171.124 Complexity.051.05.06.80 Learner Control* Complexity.011.01.01.91 Error 113.18138.82 Total2248.00142 Table 7 ANOVA Results for Trainee Affective Reactions Supplementary Analyses One possible explanation for the findings that CI was more effective than WBT in a complex training environment is that positive aspects of the instructors in CI could have influenced the favorable outcomes. If this were indeed the case, then it seems likely that the same pattern of results would be found for the WBT and CI easy conditions. That is, since the same instructors taught both CI courses (easy and complex), the same positive attributes should have influenced trainees reactions to the training course. Since the results consistently showed no significant differences between the easy conditions, it is possible that the trainers were perceived as having more positive attributes in the complex condition as opposed to the easy conditions. In order to rule out this possibility, a MANOVA was conducted comparing the easy and complex CI conditions on their ratings of the three instructors. Results showed a non-significant Wilks Lambda # = .89, F (3, 60) = 2.31, p = .09, suggesting that as a set, ratings of the instructors did not differ between the easy and complex CI conditions. In fact, as shown in Table 8, the mean ratings for the three instructors tended to be more favorable in the easy CI condition. Therefore, it seems unlikely that aspects of the instructors are responsible for the results favoring CI in complex training environment.
54 Instructor MeanSDMeanSD Instructor 126.96.36.1991.75 Instructor 28.032.017.531.85 Instructor 39.121.438.131.66 CI-Easy CI-Complex Table 8 Mean Ratings for Instructors in CI conditions Although little support for hypothesis 3 was found, prior findings consistently showed that there were no significant differences between learners in the easy (WBTeasy vs. CI-easy) conditions. This suggests that learner control may not be as important a factor for predicting cognitive learning or transfer effectiveness when training material is relatively easy. The first analysis was concerned with the cognitive learning. More specifically, overall cognitive learning was included as the criterion. To conduct the mediation analysis, I first selected only the complex conditions. I then confirmed that there were significant bivariate correlations between the predictor (learner controlcomplex conditions) and the criterion (total exam percentage), r = -.59, p < .001, the predictor and the proposed mediator (time-on-task), r = -.92, p < .001, and the proposed mediator and the criterion, r = .59, p < .001. I then regressed total exam percentage onto the learner control-complex variable in step 1 and added time-on-task into the regression equation in step 2. The standardized regression coefficient for the learner controlcomplex variable became non-significant in step 2 ( = -.26, p = .28), suggesting that trainees in the high learner control-complex condition learned significantly less than those in the low learner control-complex condition because they spent substantially less time on course related activities. I followed the same steps to find out if time-on-task also mediates the relation between learner control and transfer effectiveness when only the complex conditions
55 were considered. The bivariate relations between the predictor (learner control-complex conditions) and the criterion (transfer effectiveness), r = -.26, p < .05 and the predictor and the proposed mediator (time-on-task), r = -.92, p < .001 were both significant. However, the relation between the proposed mediator and the criterion, r = .25, p =.05, was not statistically significant at the .05 level, possibly suggesting that there may not be sufficient power to detect a significant correlation. Even though this relation was not found to be statistically significant, it is still possible that the variables correlate in the population. Thus, I conducted mediated regression. I first regressed transfer effectiveness onto the learner control-complex variable in step 1 and added time-on-task into the regression equation in step 2. The standardized regression coefficient for the learner control-complex variable became non-significant in step 2 ( = -.21, p = .52). However, since the magnitude of the change in the Beta weight was relatively small, I conducted a Sobel test which failed to detect significant mediation (Sobel test statistic = 0.18, p = .86). Overall, it appears that although time-on-task fully mediated the relation between learner control and cognitive learning in the complex conditions, it did not mediate the relation between learner control and transfer effectiveness. Finally, it is important to gain an understanding of the reasons trainees given high levels of learner control tend to spend less time on course-related activities. Although this issue was not directly addressed in this study, there was evidence that trainees tend to misgauge the extent to which they performed poorly on the final exam. As mentioned before, trainees in the easy conditions were not prepared for 9 of the 20 questions present on the post-course exam and thus should consider the final exam to be substantially more difficult than those in the complex conditions. Moreover, it was expected that trainees in
56 the complex conditions would have higher raw scores on the exam than those in the easy conditions. To test the former prediction, an independent samples t-test was conducted to find out whether trainees in the easy conditions perceived the final exam to be more difficult than those in the complex conditions. This post hoc hypothesis was not supported, t (140) = 1.94, p = .054. However, it should be noted that the difference in the perceived difficulty of the exam was in the expected direction, such that trainees in the easy conditions ( M = 2.64, SD = .96) perceived the exam to be only slightly more difficult than those in the complex conditions ( M = 2.32, SD = 1.05). Interestingly, however, it was found that although trainees in the easy condition did not perceive the exam to be significantly more difficult than those in the complex conditions, those in the easy condition ( M = 14.91, SD = 2.41) did tend to score significantly worse (raw score wise) than trainees in the complex conditions ( M = 16.74, SD = 3.01) on the final exam, t (140) = -3.97, p < .001. These disparate findings suggest that trainees in the easy conditions tended to misgauge the extent to which the exam was difficult. Thus, although they scored significantly worse on the final exam, they did not perceive the test to be more difficult.
57 Discussion Employee training is one of the most commonly employed human resource functions (Scaduto, Lindsay, & Chiaburu, 2008), and organizations have numerous options when deciding on a training delivery medium. However, it is important for organizations to build a knowledge base regarding the factors that influence the effectiveness of one medium relative to others. In this study, I outlined and discussed two popular training media: WBT and CI. Although much research has attempted to test the relative effectiveness of one medium over the other, several issues have made substantive conclusions ambiguous. More recent work (e.g., Sitzmann et al., 2006) has identified moderators of the relationship between training media and important training outcomes (i.e., cognitive learning, affective reactions) and has suggested that the relative effectiveness of one medium over the other may depend on a number of factors. Moreover, much of the media research has treated training media as an independent variable and thus confounded it with several other potentially important variables. This study adds to the extant media research in several important ways. First, this study made an attempt to isolate learner control as the key variable of interest. Second, it investigated training content complexity as a moderator of the media-outcome relations, which is a potentially important factor for organizations to consider when deciding on a training medium. Finally, the current study adds to the overall training literature by investigating
58 the relative effectiveness of two commonly used training media on the acquisition of procedural knowledge. Summary of Findings Overall, the results of this study suggest that although learner control is often touted as an advantage of WBT (e.g. Kinzie & Sullivan, 1989), increased learner control can lead to less time-on-task and less learning. Interestingly, several researchers have noted that WBT reduces total training time (Kulik & Kulik, 1991; Welsh, et. al., 2003) such that trainees in WBT can be presented with the same amount of material in substantially less time than those in CI. Although the results of this study do suggest that increased learner control leads to less training time, this difference is not necessarily beneficial. In fact, increased learner control appears to be detrimental to cognitive learning when training material is relatively complex in nature. However, when training material was relatively easy, learner control was not found to be detrimental to learning. Again, this study adds to the extant literature on the effectiveness of training media on the acquisition of procedural knowledge. Like declarative knowledge, trainees who were given substantial control over their learning and were presented with relatively complex material acquired substantially less procedural knowledge than those who were not afforded much control over their learning. This finding suggests that learner control may indeed have a detrimental effect on the acquisition of procedural knowledge when the content of training is relatively complex. It also addresses Welsh et al.s (2003) concern that WBT may not be appropriate for training procedural knowledge. Although the results of this study do not warrant this conclusion, they do suggest that such concerns
59 are well founded and that in complex training environments, WBT may not be as effective as CI for training procedural knowledge. In addition to cognitive learning outcomes, the findings of this study suggest that learner control tends to have a detrimental effect on transfer of training when training material is relatively complex in nature. As expected, trainees who were given control over their learning and were presented with a relatively complex training course did not transfer their knowledge and skills as effectively as those who were afforded less learner control. Again, this pattern was not found for trainees presented with relatively easy training material. An additional purpose of this study was to investigate the mechanisms by which learner control leads to decreased cognitive learning outcomes. Although it was originally predicted that the total amount of time trainees spend on course-related activities would mediate the relation between learner control and cognitive learning outcomes, mixed support was found for this hypothesis. Although time-on-task was not found to mediate the relation between learner control and cognitive learning, time-on-task was found to mediate this relation when only the complex conditions were considered. This implies that although time-on-task does not appear to be an important factor when training material is relatively easy in nature, it is highly important when training material is relatively complex. Overall, trainees presented with relatively complex training material and are given high levels control over their learning, tend to spend less time-on-task and learn substantially less than those given less control over their learning. Additionally, it was originally predicted that time-on-task would mediate the relation between learner control and transfer effectiveness such that trainees given more
60 control over their learning would spend less time-on-task and thus transfer their training less effectively. Overall, transfer effectiveness was found to be unrelated to both learner control and time-on-task. When only the complex conditions were considered, transfer effectiveness was found to relate to learner control and time-on-task but time-on-task did not mediate the relation between learner control and transfer effectiveness. This is an intriguing finding since it appears that learner control has an indirect effect on cognitive learning via time-on-task, while this was not found for transfer of training effectiveness. This suggests that different mechanisms drive the relations among learner control and cognitive learning and transfer of training. Finally, several supplementary analyses were conducted to find out if trainees tend to misgauge the extent to which the final examination was difficult. Although trainees taking the easy course were not expected to be fully prepared for many of the questions on the final exam, they did not report perceiving the exam as very difficult. Trainees in the easy course performed substantially worse on the exam than trainees taking the complex course, but they still failed to recognize that they were not performing at a high level. This is an interesting finding because it leads one to speculate about the possibility that trainees tend to overestimate the extent to which they understand the material being trained. It seems possible that one of the reasons trainees spend less time on course-related activities when given control over their learning is because they overestimate the extent to which they understand the material being presented. This phenomenon has been previously identified, as some empirical work has found that learners judgments of learning (JOL) are sometimes inflated (e.g., Koriat, Sheffer, & Maayan, 2002) especially with little practice. With increased practice, however, learners
61 tend to become more accurate in their JOL and may even underestimate their learning (Koriat et al., 2002). This issue has implications for both practice and theory as it implies that without some guidance, trainees may spend less time-on-task and misjudge how much time they need to allocate to learning and practicing in order to master material being trained. Implications Not only do the findings of this study add to the media literature by investigating the role of training content complexity on training effectiveness, but it makes several important practical contributions. For example, practitioners may consider utilizing classroom training media or blended learning media in which trainees are given some guidance, when the content of training is relatively complex in nature. On the other hand, if the content of training is relatively simple and does require trainees to process large amounts of information in relatively short periods of time, then training time may be reduced without a substantial drop in learning and performance by employing WBT technologies. It is also possible that different forms of trainee guidance can be included into technologically-mediated training programs when involving face-to-face instructors is not viable. For instance, like Brown (2001), my findings suggest that without some guidance, learners may tend to spend less time on training tasks and consequently learn less in complex training environments. This does not necessarily imply that WBT technologies are ineffective for training complex material. If WBT technologies include some type of monitoring or learner guidance, unfavorable outcomes may be averted, since time-on-task
62 was identified as a key variable when considering the relation between learner control and cognitive learning outcomes. Additionally, although numerous models of training effectiveness suggest that affective reactions and learning and performance should be related (Kirkpatrick, 1996; Mathieu, et al., 1992), the findings of this study, like those of Sitzmann et al. (2008) suggest that this relationship is weak at best. Specifically, the relations among trainee affective reactions and cognitive learning outcomes and transfer effectiveness were weak in magnitude and non-significant. Thus, it appears that the happy trainee is not necessarily the learned trainee. This is important for organizations because affective reactions are by far the most commonly measured outcome variable (Sugrue & Rivera, 2005). When evaluating the effectiveness of a training program, many organizations only employ measures of trainee affective reactions. The findings of this study suggest that this practice may lead to misinformed conclusions regarding the effectiveness of a training program. Even if trainees respond favorably to a training program, it is still uncertain whether they actually learned the material and will transfer what they have learned to the workplace. Additionally, in accord with the recent meta-analytic findings (see Sitzmann et. al., 2006), the results of this study suggest that there are no differences between trainees affective reactions to WBT and CI. Therefore, it appears that trainees dont necessarily prefer one medium over the other. Limitations There are several important limitations of this study. Firstly, the use of an all college student sample reduces the generalizability of the findings. Despite this possibility, it has been well argued (e.g., Greenberg, 1987) and empirically demonstrated
63 (e.g., Locke, 1986) that student samples are not necessarily more problematic than other nonrepresentative adult samples. A more important limitation is that it is not altogether clear whether the participants in the study were fully motivated to learn the material taught or perform to the best of their abilities. However, training media effectiveness is not only an important issue in organizational research; it is also an important issue to be addressed in educational settings. In addition, none of the participants scored near chance on the final examination (25% correct). This suggests that participants in the study did not simply answer exam questions randomly, thus providing some indication that trainees were at least somewhat motivated to perform well. A second limitation of this study is that it is still somewhat unclear how to practically define training content complexity. For example, although the current study found that decreased learner control led to more favorable outcomes in a relatively complex training environment, it is somewhat unclear how a complex training course could be identified in an organization. In other words, this study compared two training tasks that differed in the degree to which the material being presented was complex in nature, but it is unclear at which point, the complex training course should be considered complex. For example, is a training course that includes the presentation of 10 distinct bits of information in 30 minutes considered a complex training task? If so, relative to what? Or relative to whom? Additionally, it may be argued that the media effects found in this study were due to the Hawthorne effect, such that the favorable outcomes of CI in the complex conditions were merely due to the presence of instructors. However, if this were indeed the case, it seems likely that same results would have been found for the easy condition.
64 Since it was consistently shown that the easy conditions did not differ on the training outcome variables, this argument does not seem viable. Moreover, as mentioned before, a technical assistant was present in during the WBT courses and thus could have been just as influential in affecting trainees performance as the actual instructors. A fourth limitation is the way in which transfer of training was operationalized and measured. Transfer of training is typically defined as the ability of trainees to perform and maintain skills learned in training to other contexts (Broad & Newstrom, 1992; Noe, 2008). In this study, participants were asked to perform a single task shortly after completing the training course and a post-course examination. Thus, it is unknown whether trainees would maintain those behaviors and transfer their learning to other tasks and/or contexts in the future. Strong substantive conclusions regarding the effectiveness of the training media on the maintenance of learned behaviors are not warranted based on the data here. But, if short term transfer is impaired, it seems very likely the long term transfer would also suffer. Finally, it is unknown whether the time-on-task measure accurately measured the amount of time trainees spent on training related activities. This issue is more problematic for trainees in the CI or low learner control condition since they were prompted when to fill in specific times. It is possible that some trainees were not paying attention or were not engaged in training related activities during these periods of time. Thus, for some trainees, the time-on-task measure may have been an inaccurate measure of time spent on training related activities.
65 Future Research Since it is possible that the findings of this study are esoteric to the content being trained (i.e., a computer software package), it would be useful to extend this research to other types of training content. For example, would these findings be replicated for the training of motor tasks (e.g., assembling a gun) or safety procedures? Additionally, would these results be replicated when the goal of training is affective changes in trainees (e.g., diversity or sensitivity training)? It is also important to determine whether trainees motivation levels play an additional moderating role in the relationships investigated here. For instance, if employees are highly motivated to learn training material (e.g., results of training have implications for promotion), then the current findings may not hold. For instance, if trainees are highly motivated to learn (Noe, 1986) or highly motivated to transfer their learning to the workplace (Noe & Schmiitt, 1986), those given high levels of learner control may potentially spent more time on course related tasks than those who are given less control over their learning. But, if trainee motivation to learn and perform well is low or moderate, the results may mirror those found in this study. That is, trainees who are given more control over their learning may choose to skip or speed through material (less time-on-task) while those given less control over their learning may go along with the instructor. Conclusions This study adds to the media research by attempting to deconstruct the media into the key variable of interest (i.e., learner control), examining the effectiveness of one medium over the other for the acquisition of procedural knowledge, and investigating the
66 role of training content complexity on the media-outcome relations. Overall the findings suggest that in complex training environments, learner control may have a detrimental effect on learning and transfer. Moreover, time-on-task was identified as a mechanism by which learner control leads to unfavorable learning outcomes in complex training environments. Organizations and practitioners now have a substantial body of research to at their disposal to make informed decisions about when and where certain training media are more effective than others.
67 References Alexander, S. (2001). E-learning developments and experiences. Education & Training, 43 240-248. Allen, M. Witt, P.L., & Wheeless, L.P. (2006). The role of teacher immediacy as a motivational factor in student learning: Using meta-analysis to test a causal model. Communication Education, 55 21-31. Alliger, G.M., Tannenbaum, S.I., Bennet, W., Jr., Traver, H., & Shotland, A. (1997). A meta-analysis of the relations among training criteria. Personnel Psychology, 50 341-358. Arbaugh, J.B. (2000). Virtual classroom versus physical classroom: An exploratory study of class discussion patterns and student learning in an asynchronous internetbased MBA course. Journal of Management Education, 24 (2), 213-233. Barnett, S.M. & Ceci, S.J. (2002). When and where do we apply what we learn? A taxonomy for far transfer. Psychological Bulletin, 128 612-637. Baron, R. M., & Kenny, D. A. (1986). The moderator-mediator variable distinction in social psychological research: Conceptual, strategic, and statistical considerations. Journal of Personality and Social Psychology, 51 1173-1182. Bell, B.S., & Kozlowski, S.W.J. (2002). Adaptive guidance: Enhancing self-regulation, Knowledge, and performance in technology-based training. Personnel Psychology, 55 267-306. Bently, T. (1998). Learning beyond the cl assroom: Education for a changing world London: Routledge. Bonk, C.J., Kim, K.J., & Zeng, T. (2005). Future directions of blended learning in higher education and workplace learning settings. In Bonk C.J., Graham, C.R. (Eds.) Handbook of blended learning: Gl obal perspective, local designs (pp. 550-568). San Francisco: Pfeiffer. Box, G. E. P. (1954). Some theorems on quadratic forms applied in the study of analysis of variance problems. I. Effect of inequality of variance in the one-way classification. Annals of Mathematical Statistics 25 290-302.
68 Broad, M.L. & Newstrom, J.W. (1992). Transfer of Training Reading, MA: AddisonWesley. Brown, K.G. (2001). Using computers to deliver training: Which employees learn and why? Personnel Psychology, 54 271-296. Brown, K.G., Rietz, T.A., & Sugrue, B. (2005). The effects of video conferencing, class size, and learner motivation on training outcomes. Performance Improvement Quarterly, 18 58-82. Campbell, D.J. (1988). Task Complexity: A review and Analysis. Academy of Management Review, 13 40-52. Cascio, W.F. & Aguinis, H. (2005). Applied Psychology in Human Resource Management (6th ed.) Upper Saddle River: Prentice Hall. Clark, R.E. (1983). Reconsidering research on learning from media. Review of Educational Research, 53 445-460. Clark, R.E. (1984). Media will never influence learning. Educational Technology Research and Development, 42 21-29. Chen, G., Casper, W.J., Cortina, J.M. (2001). The roles of self-efficacy and task complexity in the relationships among cognitive ability, conscientiousness, and work-related Performance: A meta-analytic examination. Human Performance, 14(3) 209-230. Daft, R.L., & Lengel, R.H. (1986). Organizational information requirements, media richness, and structural design. Management Science, 32 554-571. DeRouin, R.E., Fritzsche, B.A., Salas, E. (2004). Optimizing E-learning: Research based guidelines for learner-controlled training. Human Resource Management, 43 147-162. Drucker, P. (2000). The Long View. Training and Development, 54 (12), 27. Enoch, Y. & Soker, Z. (2006). Age, gender, ethnicity and the digital divide: University students use of web-based instruction. Open Learning, 21 (2), 99-110. Ford, K.J. & Kraiger, K. (1995). The application of cognitive constructs and principles to the instructional systems model of training: Implications for needs assessment, design, and transfer. In Cooper, C.L. & Robertson, I.T. (Eds.), International Review of Industrial and Organizational Psychology (pp. 1-48). New York: Wiley.
69 Frankola, K. (2001). Why online learners drop out. Workforce, October, 53-60. Freitag, E.T., & Sullivan, H.J. (1995). Matching learner preference to amount of instruction: An alternative form of learner control. Educational Technology, Research and Development, 43 5-14. Galagan, P.A. (2000). E-learning revolution. Training and Development, 54 (12), 25-30. Gist, M.E. (1989). The influence of training method on self-efficacy and idea generation among managers. Personnel Psychology, 42 787-805. Gist, M.E., Schwoerer, C. & Rosen, B. (1989). Effects of alternative training methods on self-efficacy and performance in computer software training. Journal of Applied Psychology, 74 (6), 884-891. Goldstein, I.L. (1993). Training in organizations: Needs assessment, development, and evaluation (3rd ed.) Monterey, CA: Brooks/Cole. Gorham, J. (1988). The relationship between verbal teacher immediacy and student learning. Communication Education, 37 40-53. Greenberg, J. (1987). The college sophomore as guinea pig: Setting the record strait. Academy of Management Review, 12, 157-159. Hall, B. (1997). Web-based training cookbook New York: Wiley. Hays, W.L. (1994). Statistics ( 5th ed .). Belmont, CA: Wadsworth Group/Thompson Learning. Hill, J.E., Ferris, M., & Martinson, V. (2003). Does it matter where you work? A comparison of how three work venues (traditional office, virtual office, and home office) influence aspects of work and personal/family life. Journal of Vocational Behavior, 63 220-241. Kirkpatrick, D.L. (1996). Evaluation. In R.L. Craig (Ed.). The ASTD training and development handbook (4th ed., pp. 294-312). New York: McGraw-Hill. Kirkpatrick, D.L. (1976). Evaluation of training. In R.L. Craig (Ed.), Training and development handbook (pp. 18.1-18.27). New York: McGraw-Hill. Kinzie, M.B., & Sullivan, H.J. (1989). Continuing motivation, learner control, and CAI. Educational Technology Research and Development, 37 5-14.
70 Koriat, A., Sheffer, L., & Maayan, H. (2002). Comparing objective and subjective learning curves: Judgments of learning exhibit increased underconfidence with practice. Journal of Experimental Psychology: General, 131 (2), 147-162. Kraiger, K., Ford, J.K., Salas, E. (1993). Application of cognitive, skill-based, and affective theories of learning outcomes to new methods of training evaluation. Journal of Applied Psychology, 78 311-328. Kulik, C-L.C., & Kulik, J.A. (1991). Effectiveness of computer-based instruction: An updated analysis. Computers in Human Behavior, 7 75-94. Lee, J-K., & Lee, W-K. (2008). The relationship of e-learners self-regulatory efficacy and perception of e-learning environmental quality. Computers in Human Behavior, 24 32-47. Locke, E.A. (1986). Generalizing from laboratory to field: Ecological validity or abstraction of essential elements. In E. A. Locke (Ed.), Generalizing from laboratory to field settings (pp. 3-9). Lexington, MA: Lexington Books. Locke, E.A., & Latham, G.P. (1990). A theory of goal setting and task performance Englewood Cliffs, New Jersey: Prentice-Hall Long, L.K., Smith, R.D. (2004). The role of Web-based distance learning in HR development. Journal of Management Development, 23 270-284. Machtmes, K., & Asher, J.W. (2000). A meta-analysis of the effectiveness of telecourses in distance education. American Journal of Distance Education, 14 (1), 27-46. Maki, W.S., & Maki, R.H. (2002). Multimedia comprehension skill predicts differential Outcomes of web-based and lecture courses. Journal of Experimental Psychology: Applied, 8 (2), 85-98. Mangos, P.M., & Steele-Johnson, D. (2001). The role of subjective task complexity in goal orientation, self-efficacy, and performance relations. Human Performance, 14(2) 169-186. Mathieu, J.E., Tannenbaum, S.I., & Salas, E. (1992). Influences of individual and situational characteristics on measures of training effectiveness. Academy of Management Journal, 35 828-847. Maurer, T.J., & Andrews, K.D. (2000). Traditional, likert, and simplified measures of self-efficacy. Educational and Psychological Measurement, 60 (6), 965-973. Middleton, A.J. (1997). How effective is distance education? International Journal of Instructional Media, 24 (2), 133-137.
71 Mottarella, K., Fritzsche, B., & Parrish, T. (2004). Who learns more? Achievement scores following web-based versus classroom instruction in psychology courses. Psychology Learning & Teaching, 4 (1), 51-54. Noe, R.A. (1986). Trainees attributes and attitudes: Neglected influences on training effectiveness. Academy of Management Review, 11 737-749. Noe, R.A., & Schmitt, N. (1986). The influence of trainee attitudes on training effectiveness: Test of a model. Personnel Psychology, 39 (3), 497-523. Owston, R.D. (1997). The world wide web: A technology to enhance teaching and learning. Educational Research, 26 (2), 27-33. Parker, A. (1999). A study of variables that predict dropout from distance education. International Journal of Educational Technology, 1 (2), 1-12. Paul, D.S. (2001). A meta-analytic review of the factors that influence the effectiveness of web-based training within the context of distance learning. Unpublished doctoral dissertation, Texas A&M University, College Station. Pratt, J. (2002). E-learning: Is it right for your organization? Home Health Care Management & Practice, 14 (6), 471-474. Rosenberg, M.J. (2001). E-learning: Strategies for delivering knowledge in the digital age. New York: McGraw Hill. Ross, S.M., & Rakow, E.A. (1981). Learner control versus program control as adaptive strategies for selection of instructional support on math rules. Journal of Educational Psychology, 73 745-753. Rovai, A.P. (2004). A constructivist approach to online college learning. The Internet and Higher Education, 7 79-93. Rovai, A.P., & Barnum, K.T. (2003). On-line course effectiveness: An analysis of student Interactions and perceptions of learning. Journal of Distance Education, 18 (1), 57-73. Salas, E., Cannon-Bowers, J.A. (2001). The Science of Training: A decade of progress. Annual Review of Psychology, 52 471-499. Sankaran, S.R., Sankaran, D., & Bui, T.X. (2000). Effect of student attitude to course format on learning performance: An empirical study in web vs. lecture instruction. Journal of Instructional Psychology, 27 (1), 66-73.
72 Scadto, A., Lindsay, D., & Chiaburu, D. S. (2008). Leader influences on training effectiveness: Motivation and outcome expectations processes. International Journal of Training and Development,12 (3), 158-169. Sizmann, T., Brown, K.G., Casper, W.J., Ely, K., & Zimmerman, R.D. (2008). A review and meta-analysis of the nomological network of trainee reactions. Journal of Applied Psychology, 93 (2), 280-295. Sitzmann, T., Kraiger, K., Stewart, D., Wisher, R. (2006). The comparative effectiveness of web-based and classroom instruction: A meta-analysis. Personnel Psychology, 59 623-664. Sobel, M. E. (1982). Asymptotic intervals for indirect effects in structural equations models. In S. Leinhart (Ed.), Sociological methodology 1982 (pp.290-312). San Francisco: Jossey-Bass. Steinberg, E.R. (1989). Cognition and learner control: A literature review, 1977-1988. Journal Of Computer Based Instruction, 16 117-121. Steinberg, E.R., Baskin, A.B., & Matthews, T.D. (1985). Computer-presented organizational/memory aids as instruction for solving Picofomi problems. Journal of Computer-Based Instruction, 12 44-49. Sugrue, B., & Rivera, R.J. (2005). State of the industry : ASTDs annual review of trends in workplace learning and performance Alexandria, VA: ASTD. Throndike, E.L. (1913). Educational Psychology: Vol. 2. The Psychology of Learning New York: Teachers College Press. Van Merrienboer, J.J.G, & Ayres, P. (2005). Research on cognitive load theory and its design implications for e-learning. Educational Technology Research and Development, 53 (3), 5-13. Van Merrienboer, J.J.G, Kester, L., & Paas, F. (2006). Teaching complex rather than Simple tasks: Balancing intrinsic and germane load to enhance transfer of learning. Journal of Applied Cognitive Psychology, 20 343-352. Welsch, E.T., Wanberg, C.R., Brown, K.G., Simmering, M.J. (2003). E-learning: Emerging uses, empirical results and future directions. International Journal of Training and Development, 7(4) 245-258. Zhao, Y., Lei, J., Lai, B.Y.C., & Tan, H.S. (2005). What makes the difference? A practical analysis of research on the effectiveness of distance education. Teachers College Record, 107 (8), 1836-1884.
74 Appendix A: PowerPoint Familiarity Scale 1 2 3 4 5 Extremely Unfamiliar Somewhat Unfamiliar Neutral Somewhat Familiar Extremely Familiar 1. Opening a blank PowerPoint presentation............................................... 1 2 3 4 5 2. Creating multiple slides within a PowerPoint presentation 1 2 3 4 5 3. Inserting text into a PowerPoint Presentation . . 1 2 3 4 5 4. Choosing different visual layouts for a presentation ... 1 2 3 4 5 5. Choosing different color schemes for a presentation ...... 1 2 3 4 5 6. Inserting pictures and clip art into a presentation................................... 1 2 3 4 5 7. Identifying and using the Ribbon within PowerPoint . 1 2 3 4 5 8. Inserting slide transitions within a slideshow .. 1 2 3 4 5 9. Using and manipulating SmartArt . 1 2 3 4 5 10. Inserting Footers into a PowerPoint presentation .. .. 1 2 3 4 5 11. Including Action buttons into a PowerPoint presentation .. 1 2 3 4 5 12. Saving a PowerPoint presentation onto a computer .. 1 2 3 4 5
75 Appendix B. Liking For Instructor Rating Scale Liking for Instructors Instructions: Please indicate your overallreactions to each of the three instructors that presented the material i n the course by circling one number for each instructor. Low numbers indicate a poor rating of an instructor while high numbers indicate a good rating of an instructor. For example, 0 is the lowest rating for an instructor while 10 is the highest rating for an instructor. First Instructor: 0 1 2 3 4 5 6 7 8 9 10 Very ineffective Average Very effecti ve Second Instructor: 0 1 2 3 4 5 6 7 8 9 10 Very ineffective Average Very effectiv e Third Instructor: 0 1 2 3 4 5 6 7 8 9 10 Very ineffective Average Very effecti ve
76 Appendix C: Personal Time Log Instructions: Please use this sheet throughout the entire training course. There are 6 separate portions of the training course and there are 12 separate spaces below. Please keep track of your START and END times for each separate section of the course. For example, if you start the first training section (Training Session 1) at 2:15pm, enter this time into the blank labeled Start time under the Training Session 1 heading. If you finish Training Session 1 at 2:30pm and then start on Practice session 1, enter this time in both the End time blank for Training Session 1 and the start time for Practice Session 1. A sample log entry is presented below. Please make sure you fill in each blank accurately! It is very important to this study! Sample Log Training Session 1 Start time ____2:15pm________ End time ____2:30pm________ Practice 1 Start time ____2:30pm________ End time ____2:36pm________ ________________________________________________________________________ Personal Time Log Training Session 1 Start time __________________ End time __________________ Practice 1 Start time __________________ End time __________________ Training Session 2 Start time __________________ End time __________________ Practice 2 Start time __________________ End time __________________ Training Session 3 Start time __________________ End time __________________ Practice 3 Start time __________________ End time __________________ This Top Portion is only a Sample!
77 Appendix D: Post Course Examination Instructions: To the best of your ability, please select the best answer to each of the following questions. Please note that there may be questions on this exam that were not covered in the training course. Your performance on the examination will have no bearing on the number of extra credit point you receive. 1). Which of the following includes the three major areas on any PowerPoint page? a). Slide plane, Text box and Title space b). Slide plane, Notes, and Plane slide view c). Slide plane, Start menu, and Blank presentation d). Notes, Home tab, and presenter notations 2). Which of the following is the correct sequence for moving a text box around a PowerPoint page? a). Left click inside the text box and use the arrow keys to move the box b). Left click on the edge of the text box and drag it to its next location c). Left click anywhere on the PowerPoint page and drag your cursor across the screen d). Right click on the text box that you want to move and follow the instructions provided by PowerPoint 3). Which of the following options best describes the purpose of the Ribbon within PowerPoint. a). The Ribbon is PowerPoints text box creation center b). The Ribbon is PowerPoints new slide creator c). The Ribbon is PowerPoints Command Center d). The Ribbon is PowerPoints Slide plane view organizer 4). Which of the following is a common tab located on the Ribbon? a). Slide Organization tab b). Slide Plane view tab c). Home tab d). Advanced Functions tab
78 Appendix D (Continued): Post Course Examination 5). Which of the following represents the easiest way to include text into a PowerPoint page? a). Left click inside a text box and type in the desired text b). Right click outside of a text box and type in the desired text c). Left click on the edge of a text box and drag your cursor d). Select a slide from the slide tab and drag it into a text box 6). Which of the following represents the easiest way to navigate through many slides in a slideshow? a). Access the Notes area of a PowerPoint page and simply scroll through slides b). Access the Slides Tab on the left hand side of the PowerPoint page and scroll through slides c). Access the View tab on the ribbon and change view to normal d). Access the Home slide and navigate through your slide with the Enter Key 7). Which of the following include the correct steps you would take when selecting a Theme for your slideshow? a). Select the Design tab on the Ribbon and left click on a theme you like b). Select the Home tab on the Ribbon and select New Slide from the dropdown menu c). Select a them of your choice in the Plane slide view d). Access the Format tab on the Ribbon and select the Slide view tab from the dropdown menu 8). Why would you want to include slide transitions into your PowerPoint presentation? a). They allow you to easily navigate through multiple slides b). They allow you to easily access the Design tab on the Ribbon c). They allow you to move from slide to slide smoothly and make a presentation flow more smoothly d). They allow you to insert pictures into your slideshow 9). Which of the following is the correct sequence for using the Ribbon to insert pictures into your slideshow? a). Access the Home tab on the Ribbon, move your cursor over the insert option of your choice and left click on the insert option b). Access the View tab on the Ribbon and left click on the slide view tab c). Access the Insert tab on the Ribbon, move cursor over the insert option of your choice, and left click on the insert option d). Its not possible to insert pictures using the Ribbon
79 Appendix D (Continued): Post Course Examination 10). Why is it useful to include pictures into a slideshow? a). Pictures can help keep the audience interested and can complement the text you are presenting b). Pictures can overload your slides and take away from the point you are trying to make c). Pictures allow you to navigate through multiple slides quickly d). Pictures are never useful to a slideshow 11). What is the primary difference between custom animation and slide transitions? a). Custom animations make movements from slide to slide smooth, but slide transitions do not b). Custom animations can be applied to individuals lines of text or objects but slide transitions are usually applied to all slides in the slideshow c). Custom animations are only available under the Home tab, but slide transitions are accessible under all tabs on the Ribbon. d). Custom animations are always applied to every slide of the slideshow, unlike slide transitions. 12). What is the primary function of the Slide Master in PowerPoint 2007? a). It allows you to access every tab on the Ribbon quickly b). It allows you to insert text only into your PowerPoint presentation c). It allows you to include text or any icons into every slide of your presentation d). It allows you to use SmartArt for inserting graphics into your presentation 13). Which of the following is the correct sequence for accessing the Slide Master? a). Select the View tab on the Ribbon and choose the Slide Master option b). Select the Home tab on the Ribbon, select view from the dropdown menu and choose the Slide Master option c). Right click on the slide plane, select view and choose Slide Master from the dropdown menu. d). Select the Applications tab on the Ribbon and choose the Slide Master option. 14). When would you be less likely to use SmartArt in your PowerPoint presentation? a). SmartArt graphics would help enhance the information you are trying to present b). SmartArt graphics would add to the visual appeal of your presentation c). SmartArt graphics would help your audience better understand complex information d). SmartArt graphics would possibly distract your audience from the main point
80 Appendix D (Continued): Post Course Examination 15). Which of the following is the easiest way to access SmartArt graphics? a). Access the Home tab on the Ribbon, choose the view options and select the SmartArt option b). Create a new slide and select the green arrow out of the six possible icons shown in the middle of the slide c). Create a new slide and select the charts options out of the six possible icons shown in the middle of the slide d). Access the Home tab on the Ribbon and simply select applications which then accessed SmartArt 16). Which of the following represent the correct steps for inserting sounds into your presentation? a). Select the Insert tab on the Ribbon and then click the arrow next to the Sound option b). Select the Home tab on the Ribbon, choose the Insert option and select the Sound option c). Select the Insert tab on the Ribbon and select the multimedia option under the Sound dropdown menu d). Select the View tab and left click on the Applications menu 17). Which of the following is not a possible option when including sounds into a PowerPoint presentation? a). Can make sounds within slides start automatically b). Can insert sounds from both CDs and microphones c). Can choose the sounds option by selecting the Home tab in the Ribbon d). Can choose sounds by accessing the Insert tab on the Ribbon 18). Which of the following would not be a common use for a footer within a PowerPoint presentation? a). Including the date of the presentation on all slides b). Including an organization or company name on all slides c). Including the sounds options on the bottom of all slides d). Including the name of the presenter on all slides 19). What is the correct way to insert a footer into your slideshow? a). Select the Insert tab on the Ribbon and select the Header and Footer option b). Right click on a new PowerPoint slides and select the Header and Footer option c). Select the Home tab on the Ribbon, choose the Insert option and select Footers from the dropdown menu d). Create a new slide and select the green arrow from the six possible icons shown in the middle of the slide
81 Appendix D (Continued): Post Course Examination 20). What are the proper steps for saving a PowerPoint presentation? a). Access the View tab on the Ribbon and select the save icon b). Access the circular window icon at the top left hand corner of the PowerPoint screen and select the Save As option from the dropdown menu c). Move cursor to the circular window icon which is located within the Home tab and select the Save option d). Choose the desktop option from the Home tab on the Ribbon and select Save As
82 Appendix E: Affective Reactions to Training Course Instructions : Below, please provide indicate the extent to which you agree with each of the following statement. Low numbers indicate that you disagree with the statement and high numbers indicate that you agree with the statement. 1 2 3 4 5 Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree 1. Overall, I feel satisfied with the training course............................................... 1 2 3 4 5 2. I would like to learn under similar conditions in the future . 1 2 3 4 5 3. I would recommend this training course to friends 1 2 3 4 5 4. I feel that the way in which the training was presented was sufficient for my learnin g of the material ... 1 2 3 4 5
83 Appendix F: Transfer Exercise Instructions Instructions: Please follow the guidelines below in order to create a new PowerPoint presentation. Plea s e note that you are not allowed to communicate with anyone else in the room. Please follow the guidelines to the best of your ability. Your performance on this exercise will have no bearing on the number of extra credit points you receive. Guidelines 1). You will create a new slideshow from scratch in PowerPoint 2007. 2). The content of your PowerPoint presentation will be how you study for a college course (For example, you can include information in your slides that describes to a viewer how you go a bout studying for a course in college). 3). The slideshow must be exactly 3 slides long. 4). The first slide must include your identification number. This can be placed anywhere on the slide (This is the number that is on all materials you have received thus far). 5). Your slideshow will be rated on the extent to which you create your slideshow using the operations taught in the training course. These are the only guidelines you must follow. You can include any specific information you like into your presentation. Once you are done creating the slideshow, please save it on the desktop Save it under the name PowerPoint exercise
84 Appendix G: Perceive Learner Control Scale Instructions : Below, please provide indicate the extent to which you agree with each of the following statement. Low numbers indicate that you disagree with the statement and high numbers indicate that you agree with the statement. 1 2 3 4 5 Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree 1. Overall, I felt that I was in control of the time I spent learning the material in the training course................................................................................................ 1 2 3 4 5 2. I felt that someone other than me was in control of the time I spent learning the material in the training course .. . 1 2 3 4 5 3. I felt that I had a lot of control over the time I spent practicing the material 1 2 3 4 5 4. I should have had more control over the time I spent on the training course 1 2 3 4 5
85 Appendix H: Perceived Training Content Complexity Scale Instructions : Below, please provide indicate the extent to which you agree with each of the following statement. Low numbers indicate that you disagree with the statement and high numbers indicate that you agree with the statement. 1 2 3 4 5 Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree 1. Overall I thought that the training course was difficult................................... 1 2 3 4 5 2. The training course was not very complex .. . 1 2 3 4 5 3. I had no trouble following along with the material because of its simplicity 1 2 3 4 5 4. I would require much more time than was offered in order to properly understand the material in the training course ... 1 2 3 4 5 5. The large amount of information presented in the course made it difficult for me to learn ... ............. 1 2 3 4 5
86 Appendix I: Perceived Exam Difficulty Scale Instructions: Below, please provide indicate the extent to which you agree with each of the following statement. Low numbers indicate that you disagree with the statement and high numbers indicate that you agree with the statement. 1 2 3 4 5 Strongly Disagree Somewhat Disagree Neutral Somewhat Agree Strongly Agree 1. Overall, I felt that this examination was very difficult..................................... 1 2 3 4 5 2. I feel that this examination was relatively easy . 1 2 3 4 5 3. I had no trouble successfully answering the questions on this examination 1 2 3 4 5
87 Appendix J: Practice Instructions for the Easy Condition Practice Session 1 1). Exit the PowerPoint 2007 training course by left clicking anywhere on the screen and selecting end show from the list. Minimize the Course, but DO NOT CLOSE IT! 2). Open up a blank PowerPoint page 3). Identify the slide plane, slides tab and the notes areas of the page 4). Practice moving text boxes around the slide plane 5). When you are comfortable with these actions, minimize your new presentation and go back to the PowerPoint training course. Select the From current slide option or the From Beginning option and move to the overview page to access training session 2. The options are under the Slide Show tab on the Ribbon. Practice Session 2 1). Exit the training course the same way you did for Practice session 1. 2). Select the same PowerPoint presentation that you worked on in practice session 1 3). Find the Ribbon and investigate the different command group tabs on the Ribbon 4). Add a new slide into your presentation. Try adding a new slide and choose a layout for the slide. 5). Try adding text into the empty text boxes (try the term airplane or jet). 6). Now that you have multiple slides, move back and forth through them using the slides tab. 7). When you are comfortable with these actions, minimize your new presentation and go back to the PowerPoint training course. Select the From current slide option or the From Beginning option and move to the overview page to access training session 3. Practice Session 3 1). Exit the training course the same way you did in Practice sessions 1 and 2. 2). Experiment with different slide themes. Try several different slide themes and see how it changes your new slideshow each time 3). Experiment with different slide transitions. Try multiple transitions 4). Try inserting a piece of clip art into your presentation (any picture you like) 5). Try saving your new slideshow to the Desktop Remember to use the Save as option not just the save option. 6). Exit the training course and practice presentation and access the Post Course survey packet (this was give n to you prior to the study). Dont forget to log in your End time for practice session 3!
88 Appendix K: Practice Instructions for the Complex Condition Practice Session 1 1). Exit the PowerPoint 2007 training course by left clicking anywhere on the screen and selecting end show from the list. Minimize the course, but DO NOT CLOSE IT! 2). Open up a blank PowerPoint presentation 3). Identify the slide plane, slides tab and the notes areas of the page. Experiment with the different command tabs located on the Ribbon 4). Practice moving text boxes around slide plane and inserting text into them 5). Add a new slide into your presentation and practice using the slides tab to navigate through your slides 6). When you are comfortable with these actions, minimize your new presentation and go back to the PowerPoint training course. Select the From current slide option or the From Beginning option and move to the overview page to access training session 2. These options are under the Slide Show tab on the Ribbon. Practice Session 2 1). Exit the training course and select the same PowerPoint presentation that you worked on in practice session 1. 2). Experiment with different slide themes. Try several different ones and see they change your new slideshow each time 3). Experiment with different slide transitions. Try multiple transitions. 4). Try inserting a piece of clip art into your presentation (try the terms airplane or jet) 5). Access the Master slide and experiment with it 6). When you are comfortable with these actions, minimize your new presentation and go back to the PowerPoint training course. Select the From current slide option or the From Beginning option and move to the overview page to access training session 3. Practice Session 3 1). Exit the training course the same way you did in Practice sessions 1 and 2. 2). Access SmartArt and experiment with different options. 3). Add footers into your new presentation (try different placements and content) 4). Try saving your new slideshow to the Desktop Remember to use the Save as option not just the save option. 5). Exit the training course and practice presentation and access the Post Course survey packet (this was give n to you prior to the study). Dont forget to log in your End time for practice session 3!
89 Appendix L: General Instructions to Trainees in the CI condition General Instructions (Classroom instructions) In the following training packet you will find a series of surveys and a personal time log. You will be prompted by the instructor when to log times into your personal time log. Please follow all directions given to you by the training course instructors. Please follow along with the instructor of the course and do not move ahead or fall behind the course instructor. Throughout the course and practice sections, the instructor will monitor you to ensure that you are spending the time set-aside for each section. Since we are attempting to get an accurate indication of how much time people are spending on certain sections of the course p lease record times in your personal time log as accurately as possible. Also, it is VERY IMPORTANT that you do not communicate with anyone else in the room during the experiment. In other words, please do not attempt to speak to the course instructors or any other participant in the room during the course If you are having trouble with your computer, you may speak with the lab technician who can only help you with computer malfunctions or problems. It is very important to this study that there be no communication between p articipants, so I greatly appreciate your cooperation on this matter.
90 Appendix M: General Instructions to Trainees in the WBT condition General Instructions (Web-based instructions) In the following training packet you will find a series of surveys, examinations and a personal time log. You will be prompted by the narrators on the computer when to log times into your personal time log. Please note that there are no time minimums for which you much stay in one section of the training course. It is up to you how much time to actually spend on certain sections of the course. You will not be scored on the amount of time you spend on each section and it will in no way affect the number of extra credit points you receive for your participation. Moreover, you will not be monitored in any way throughout this course. However, you have been given recommendations for how much time you may spend on each section of the course. Since we are attempting to get an accurate indication of how much time people are spending on certain sections of the course so please record times in your personal time log as accurately as possible. It is VERY IMPORTANT that you do not communicate with anyone else in the room while you are participating in the experiment. In other words, please do not attempt to speak any other participant in the room during the course If you are having trouble with your computer, you may speak with the lab technician who can only help you with computer malfunctions or problems. It is very important to this study that there be no communication between participants, so I greatly appreciate your cooperation on this matter
91 Appendix N: Transfer Rater Guide Please rate each PowerPoint presentation on a scale from 0 10 0 1 2 3 4 5 6 7 8 9 10 Descriptions of Several Scores 0 = Creator of presentation did an extremely poor job of reaching the ultimate goal and shows that he or she does NOT understand any of the skills taught in the training course 5 = Creator of presentation did a fair job of reaching the ultimate goal and shows that he or she understands and can utilize SOME of the skills taught in the training course 10 = Creator of presentation did a superior job of reaching the ultimate goal and shows that he or she completely understands and can utilize ALL skills taught in the training course