|USFDC Home | USF Electronic Theses and Dissertations||| RSS|
This item is only available as the following downloads:
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam 2200373Ka 4500
controlfield tag 001 001790453
007 cr mnu uuuuu
008 070619s2006 flu sbm 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0001501
Burke, Jennifer L.
b an investigation of the effects of Remote Shared Visual Presence on team process and team performance in urban search and rescue teams
h [electronic resource] /
by Jennifer L. Burke.
[Tampa, Fla] :
University of South Florida,
This field study presents mobile rescue robots as a way of augmenting communication in distributed teams through a remote shared visual presence (RSVP) consisting of the robot's view. It examines the effects of RSVP on team mental models, team processes, and team performance in collocated and distributed Urban Search & Rescue (US&R) technical search teams, and tests two models of team performance.Participants (n=50) were US&R task force personnel drawn from high-fidelity training exercises held in California (2004) and New Jersey (2005). Data were collected from the 25 dyadic teams as they performed a 2 x 2 repeated measures search task entailing robot-assisted search in a confined space rubble pile. Team communication was analyzed using the Robot-Assisted Search and Rescue coding scheme (RASAR-CS). Team mental models were measured through a team-constructed map of the search process. Ratings of team processes (communication, support, leadership, and situation awareness) were made by onsite observers, and team performance was measured by number of victims (mannequins) found. Multilevel regression analyses were used to predict team mental models, team process, and team performance based upon use of RSVP (RSVP or no-RSVP) and location of team members (distributed or collocated). Results indicated that the use of RSVP technology predicted team performance (=-1.322, p = 0.05), but not team mental models or team process. Location predicted team mental models (=-0.425, p = 0.05), but not as expected.Distributed teams had richer team mental models as measured by map ratings. No significant differences emerged between collocated and distributed teams in team process or team performance. Findings suggest RSVP may enhance team performance in US&R search tasks. However, results are complicated by differences detected between sites. Support was found for both models of team performance, but neither model was found sufficient to describe the data. Further research is suggested in the use of RSVP technology, the exploration of team mental models, and refinement of a modified model of team performance in extreme environments.
Dissertation (Ph.D.)--University of South Florida, 2006.
Includes bibliographical references.
Text (Electronic dissertation) in PDF format.
System requirements: World Wide Web browser and PDF reader.
Mode of access: World Wide Web.
Title from PDF of title page.
Document formatted into pages; contains 131 pages.
Adviser: Michael D. Coovert, Ph.D.
Shared mental models.
Field research methods.
t USF Electronic Theses and Dissertations.
RSVP: An Investigation of the Effects of Remote Shared Visual Presence on Team Process and Performance in Urban Search & Rescue Teams by Jennifer L. Burke A dissertation submitted in partial fulfillment of the requirements for the degree of Doctor of Philosophy Department of Psychology College of Arts and Sciences University of South Florida Major Professor: Michael D. Coovert, Ph.D. Walter C. Borman, Ph.D. Michael T. Brannick, Ph.D. Mark S. Goldman, Ph.D. Robin R. Murphy, Ph.D. Date of Approval: April 10, 2006 Keywords: human-robot interaction, re scue technology, shared mental models, communication analysis, field research methods Copyright 2006, Jennifer L. Burke
Dedication For my family: my mother Jane, who always found something good to say, and told me she was proud of me; my sister s Jill and Krista, who patiently listened to my tales of woe when things were not going well; my father Garfield, who inspired me to go for it in the first place; sweet daughter Sara, who dealt gr acefully with a busy student-mom (I cant cook dinner, I have homework!); and for my husband Michael, fellow traveler through life, whose love and support kept me going through the adventures of the past five yearsas it has throughout our lives together.
Acknowledgements There are many people who played a part in this study coming to fruition, from funding agents that supported th e research to fellow student s who helped with the nitty gritty details. Whatever the role, I extend my heartfelt gratitude for your help. Thanks go to DARPA and NSF for their support; SSRRC staff members Sam Stover, Tim Slusser, and Colleen Cleveland; CS E staff members Valerie Mirra and John Giannoni; USF students Rahul Agarwal, Laura Barnes, Jeff Craighead, Thomas Fincannon, Matt Long, David Richards, Ashley Gra y, Jamie Griffiths, Brian Day, and Rachel Sessions; CMU students Jason Geist, Mike Scherwin, E lie Shammas, and Anthony Kolb; and many others, including Bob Dolci at NASA-Ames, Ji m Bastan at NJTF-1, Howie Choset from CMU, Jean Scholtz from NIST, Mark Micire and Erika Rogers (w ho bought me the little red book in which to begin the writing process). Special thanks go to my committee members: Wally Borman, who gave me good counsel on rater training, and lots of positiv e reinforcement; Mike Brannick, who always made time to answer my (many) questions on methodological issues; Mark Goldman, who gave me my first research job, and whose own research sets the standard by which I measure; and finally, to my mentors and advisors, Mike Coovert and Robin Murphy. Mikes guidance, support, and encouragemen t inspired me, and his quiet questions challenged me to think and stretch to come up with new ideas. I can only hope to be like Mike in my career. Lastly, my thanks go to Robin Murphy, who made it possible for me to go places and do things I never dreamed possible. The day I visited her lab was a turning point in my career, opening the door to a future in a brand new fieldhumanrobot interaction. I look forward to our future collaboration!
i Table of Contents List of Tables................................................................................................................. ....iii List of Figures....................................................................................................................iv Abstract....................................................................................................................... .........v Chapter 1: Introduction....................................................................................................... 1 Chapter 2: Related Work....................................................................................................8 Teams.......................................................................................................................8 Distributed Teams......................................................................................10 Distributed Teams in US&R......................................................................11 Description of Technical Search Team Operations.......................11 Communication and Coordination Challenges in Distributed Teams....................................................................................................14 Shared Mental Models...................................................................16 Team Communication and Shared Mental Models.......................20 Shared Visual Presence..................................................................22 Robots........................................................................................................25 What is a Robot?............................................................................25 Human-Robot Interaction..............................................................27 Human-Robot Team Performance.................................................28 Situation Awareness and Team Processes.....................................30 Chapter 3: Mobile Robots as Shared Visual Presence: Approach....................................32 Hypotheses.............................................................................................................36 Chapter 4: Method............................................................................................................3 8 Setting, Participants, and Apparatus......................................................................38 Setting........................................................................................................38 Participants.................................................................................................39 Apparatus...................................................................................................40 Design....................................................................................................................41 Location (Collocated vs. Distributed Teams)............................................41 Remote Shared Visual Presence (RSVP or no-RSVP)..............................41 Measures................................................................................................................42 Team Performance Measure......................................................................43 Team Process Measures.............................................................................44 Shared Mental Model Measures................................................................45
ii RASAR-CS................................................................................................46 Procedure...............................................................................................................51 Data Collection..........................................................................................51 Data Analysis.............................................................................................54 Data Editing and Preparation.........................................................54 Data Coding and Rating.................................................................54 Statistical Analyses : Multilevel Regression...................................56 Chapter 5: Results............................................................................................................. .64 Descriptive Analyses.............................................................................................65 Demographics............................................................................................65 RASAR-CS Analyses................................................................................66 Descriptive Analyses for Study Variables.................................................72 Multilevel Regression Analyses............................................................................76 Team Performance Measure......................................................................76 Supplemental Regression Analyses...............................................78 Shared Mental Model Measure..................................................................80 Supplemental Regression Analyses...............................................81 Team Process Measure..............................................................................83 Supplemental Regression Analyses...............................................84 Summary of Hypotheses and Findings..................................................................85 Chapter 6: Discussion........................................................................................................94 RSVP and the Model.............................................................................................96 Location and Site Effects.......................................................................................98 Theoretical and Practical Implications.................................................................104 Theoretical Implications......................................................................................104 Practical Implications...........................................................................................107 Limitations...........................................................................................................110 Parting Thoughts and Future Directions..............................................................111 References........................................................................................................................114 Appendices.......................................................................................................................122 Appendix A: Script for Search Task Scenario....................................................123 Appendix B: Demographi c Survey Questionnaire.............................................125 Appendix C: Complete Correla tion Table for Study Variables..........................127 About the Author...................................................................................................End Page
iii List of Tables Table 1 Distribution of teams across experimental conditions.....................................42 Table 2 Team process dimensions................................................................................45 Table 3 Robot-Assisted Search and Rescue Coding Scheme (RASAR-CS)................48 Table 4 Raw agreement, K n and K for the four RASAR-CS categories......................56 Table 5 Means/proportions, standard deviations, and percentages for demographic survey data.................................................................................66 Table 6 RASAR-CS statement fr equencies and percentages........................................68 Table 7 RASAR-CS form x content comp arisons of statement frequencies and proportions for speaker-recipient dyads...........................................................70 Table 8 RASAR-CS statement proportions by location and RSVP conditions............71 Table 9 Means, standard deviations, a nd correlations for study variables of interest (N = 50)..............................................................................................73 Table 10 Multilevel regression analysis of location and RSVP effects on team performance.....................................................................................................77 Table 11 Supplemental multilevel regression analyses for team performance...............78 Table 12 Multilevel regression analysis of location and RSVP effects on shared mental model....................................................................................................81 Table 13 Supplemental regression an alyses for shared mental model............................82 Table 14 Multilevel regression analysis of location and RSVP effects on team process..............................................................................................................84 Table 15 Supplemental regression analyses for team process........................................85 Table 16 Study hypotheses a nd evidence of support......................................................86
iv List of Figures Figure 1. Organizational Structure of US&R Ta sk Force (FEMA,1992).......................13 Figure 2. Klimoski & Mohammeds Fr amework for Explaining the Role of Team Mental Models in Team Performance...................................................19 Figure 3. Model of Team Performance in Robot-Assisted Technical Team Search...............................................................................................................34 Figure 4. Inuktun Micro-VGTV Robot System..............................................................40 Figure 5. A Mannequin Hand Hidden in the Search Space Serves as a Visual Cue...................................................................................................................44 Figure 6. SA Indicators in the RASAR-CS.....................................................................49 Figure 7. A 3-Person Team of Researchers Manned Each Task Scenario Site...............53 Figure 8. Baseline and Ordercode M odel Estimates of Performance.............................63 Figure 9. Mean Performance Scores Plotted by Location and Use of RSVP.................99 Figure 10. Bar Graphs Showing Mean Pe rformance Scores at NASA-Ames and NJTF-1 Sites According to Location (Distributed or Collocated) and Use of RSVP..................................................................................................102
v RSVP: An Investigation of the Effects of Remote Shared Visual Presence on Team Process and Team Performance in Urban Search & Rescue Operations Jennifer L. Burke ABSTRACT This field study presents mobile re scue robots as a way of augmenting communication in distributed teams through a remote shared visual presence (RSVP) consisting of the robots view. It examines the effects of RSVP on team mental models, team processes, and team performance in collocated and distributed Urban Search & Rescue (US&R) technical search teams, a nd tests two models of team performance. Participants (n=50) were US&R task force personnel drawn from high-fidelity training exercises held in California (2004) and New Jersey (2005). Da ta were collected from the 25 dyadic teams as they performed a 2 x 2 repe ated measures search task entailing robotassisted search in a confined space rubble pile. Team communication was analyzed using the Robot-Assisted Search and Rescue c oding scheme (RASAR-CS). Team mental models were measured through a team-constructed map of the search process. Ratings of team processes (communication, support, leader ship, and situation aw areness) were made by onsite observers, and team performan ce was measured by number of victims (mannequins) found. Multilevel regression analys es were used to predict team mental models, team process, and team perform ance based upon use of RSVP (RSVP or noRSVP) and location of team members (distribut ed or collocated). Results indicated that the use of RSVP technology predicted team performance ( = -1.322, p = 0.05), but not
vi team mental models or team process. Location predicted team mental models ( = -0.425, p = 0.05), but not as expected. Distributed te ams had richer team mental models as measured by map ratings. No significant di fferences emerged between collocated and distributed teams in team process or team performance. Findings suggest RSVP may enhance team performance in US&R search tasks. However, results are complicated by differences detected between sites. S upport was found for both models of team performance, but neither model was found su fficient to describe the data. Further research is suggested in the use of RSVP technology, the explorat ion of team mental models, and refinement of a modified model of team performance in extreme environments.
1 Chapter 1 Introduction Distributed team performance is becomi ng one of the most popular and critical research areas in industrial-organizational psychology. The perfect storm combination of workforce globalization, the proliferation of teams as an organizational structure, and the onslaught of increasingly complex technology has made the concept of distributed teams a necessary reality without a clear unde rstanding of what it takes to make them work. The communication and coordination challenges posed by distributed teaming are readily evident as a hindrance to effec tive team performance (Olson & Olson, 2003). These challenges are sometimes exacerbated rath er than alleviated by the insertion of new technology designed to address them. This study, which explores the effects of one type of technology (robots) on distributed team performance, is motivated largely in the interests of making robots a hel p, not a hindrance, in distribute d team tasks of the future. This study presents mobile rescue robots as a way of augmenting communication in distributed teams through a shared remote visual presen ce consisting of the robots view. The research has a theore tical focus, testing portions of an existing model of team performance (Klimoski & Mohammed, 1994), a nd probing further to examine questions as to how certain constructs within the m odel are related. It moves from theoretical constructs to a quasi-experimental investigation that will tease apart some of the issues pertaining to the relationship s between team mental models, team processes and team performance. The question is addressed with in the purview of the Federal Emergency
2 Management System, which became part of th e U.S. Department of Homeland Security in March, 2003. FEMA's continuing mission with in the new department is to lead the effort to prepare the nation for all hazards and effectively manage federal response and recovery efforts following any national incide nt. Emergency/disaster management is a system composed of many (distributed) ad hoc teams; one of these is Urban Search and Rescue (US&R). Urban search and rescue has been posed by the DARPA/NSF study on humanrobot interaction (Burke, Murphy, Rogers, Lumelsky, & Scholtz, 2004) as an exemplar domain for human-robot interaction (HRI). US&R involves the rescue of victims from the collapse of a man-made structure. The e nvironment can be characterized as a pile of steel, concrete, dust, and other rubble and debris. The areas are perceptually disorienting; they no longer look like recogni zable structures due to the collapse, it is dark, and everything is covered in gray dust from concrete or sheet ro ck. Robot assisted search and rescue in this field domain requires that small shoe-box si zed physically situated robots operate under these unstructured, outdoor e nvironmental conditions in real-time to visually search areas that are either to o narrow for safe human or canine entry or generally unsafe for human exploration The robots are short, provi ding a viewpoint from less than one foot off the ground. This exac erbates any keyhole effects (Woods, Tittle, Feil, & Roesler, 2004). These domain and agen t characteristics pr esent many challenges that distinguish US&R from other human-r obot interaction settings, e.g. manufacturing, entertainment and office-oriented applications. The relationship between humans and robots in US&R is different than manufacturing, office, or even security appl ications of robots. Robots must physically
3 team with people to perform any activity. B ecause of their small size and the mobility challenges imposed by the US&R environment, robots must be carried in backpacks to the voids targeted to be searched. Humans must interpret the video, audio, and thermal imaging data provided from the robots and fuse it with other data sources (e.g., building plans) and knowledge (e.g., time of day) in order to identify vict ims and structural anomalies as well as conduct and coordinate la rge-scale rescue efforts. The information extracted from the robots search must be abstracted and propagated up a hierarchy of decision makers as well as distributed late rally among search specia lists. Therefore, the human-robot team must cooperatively transfor m data into information and levels of knowledge. This means human-robot interacti on in US&R must c onsider distributed information transfer and cooperation. Though the use of robots as remote sh ared visual presence is presented specifically within the domain of US&R, the potential applications have far-reaching implications for propagation throughout the emergency management system structure, encompassing all of the federal response a nd recovery efforts following any national incident. It is believed that using the remo te presence provided by the robot as the basis for establishing mutual knowledge among dist ributed teams can increase communication efficiency in distributed teams by build ing shared awareness, or common ground. Communication efficiency has b een shown to reduce workload and can result in better performance (MacMillan, Entin, & Serfaty, 2004). Moreover, the use of mobile robots as a visual resource allows for assessment of the situation while avoiding information overload, and can assist in re source allocation by incident command. It can also provide reliability/redundancy in communication s upport to existing communication channels,
4 and help with accountability of resources and personnelall critical issues in firefighting and emergency response (Jiang, Hong, Takayama, & Landay, 2004). This study hypothesizes that the shared visual presence provided by the robots eye view can serve as common gro und for the distributed team onsite, and for others removed from the site. Robot-assisted techni cal search presently requires a 2:1 human-torobot ratio (Burke & Murphy, 2004a). The robot operator bears the brunt of the cognitive load in teleoperating the robot and searchi ng the remote environment; the tether-handler can provide some physical assistance thr ough manuevering the tether but mostly relies on communication with the operator to particip ate in the cognitive aspects of the task. The tether-handler can share the cognitive load by assisting with the search, but lacks the same point-of-view as the operator, since she is typically several meters away and cannot observe the visual image provided by the robo t. The robots view offers a medium for building a shared mental model of the remote search space, allowing for feedthrough of awareness information by positions, orientat ion and movement of both the robot and other artifacts in the visual envi ronment (Gutwin & Greenberg, 2004). Shared visual presence in the remote environment can enhance team communication and coordination, serving as a source for conversational grounding (common ground), and facilitating the use of gaze awarene ss, deictic references and targeted communication, as well as providing a feedback source (v isual evidence of understanding). Operators and tether-handlers currently utilize verb al communication to create team mental models of the search environment, and to support mutual knowledge of both the task and their respective role s and actions in performing the task. By providing the tether-han dler with the same remote visual presence experienced by the
5 robot operator via an auxiliary monitor, mo re contextual information can be conveyed using targeted communication and deictic refere nces toward artifacts in the environment. The robot operator can convey gaze awarene ss by camera manipulation or changes in the robots configuration. Feedback between th e two team members may be enriched by providing the visual channel as a conduit for confirmatory communication on an implicit level. For example, the tether-handler may sugge st that the robot operator take a closer look at a particular artifact in the remote environment. When the robot operator focuses on the intended object, the tether-handler has visual confirmation of the operators understanding. In addition, using the visual im age provided by the robot to build mutual knowledge may allow team member s to anticipate each others informational needs, and provide needed information without being as ked. This simple switch from explicit to implicit coordination in teams can increas e communication efficiency, and has been linked with effective team performance in high stress situations (Mor ris, Rouse, & Zee, 1987). Therefore it is expected th at the use of mobile robots as a shared visual presence in remote environments may lead to more effective distributed team performance in robot-assisted technical search tasks. Moreover, as wi reless communica tion technology improves, this shared visual presence does not have to be limited to the distributed team onsite. A team member looking over the shoulde r of the robot operator from a site well-removed from the Hot Zone can assist in the search task, offering a fresh pair of eyes that are not subject to the physical and cognitive stressors present onsite. As mentioned earlier, this study is a theoretical piece, presenting a contextand task-specific model of team performance that tests relationships between shared (team)
6 mental models, team process, and team performance as outlined in Klimoski and Mohammeds (1994) framework explaining the role of team mental models in team performance. This model draws on Kraiger and Wenzels (1997) proposed framework for mental models as well, which situates shar ed mental models in a nomological net of antecedents and consequent effects (incl uding team process and performance.) Unlike Klimoski and Mohammeds model, however, the Kraiger and Wenzel framework makes no attempt to explain how shared mental models influen ce team process and performance. This proposal tests a portion of Klimoski and Mohammeds theoretical model, that shared mental models do contribute to team performance directly and indi rectly through greater team capacity and more effective team proce sses. A further theoretical contribution is made by exploring more specifically how th ese team mental models are created, and whether particular team pro cesses (communication, backup/s upport, leadership, situation awareness) are affected by thei r quality. In this study, the remo te shared visual presence provided by the robot is posited as a team re source that increases the teams capacity (readiness) and serves as the common gr ound from which team mental models are constructed. Through communicatio n analysis and other measures, I intend to trace the formation of the team mental model, exam ine its influence on team processes and investigate the effects, if any, on team performance. To understand the research presented in this study, there are some terms and background information you need to know. Th e following chapter will give you some background on teams and distributed teams, and on the communication and coordination challenges that confront distri buted teams. The concepts of shared awareness and shared mental models are discussed, as is research on shared visual space. Next, a short review
7 of the research on robots and human-robot interaction is recounted, focusing on the studies that relate to the s earch and rescue domain. Armed with this information, you can then wade into the approach outlined in Ch apter 3, which explains the proposed model of performance in robot-assisted technical se arch teams, and enumerates specific hypotheses. This is followed by the method section (Chapter 4), which details the specifics of the field study, and the analyses used to examine the research questions. Results are presented in Chapter 5, and ar e followed by discussion and conclusions in Chapter 6.
8 Chapter 2 Related Work This chapter reviews important characteristics of teams and distributed teams, explains some of the communication and coordination challenges that confront distributed teams, and presents relevant re search on human-robot in teraction. It begins with a brief discussion of the characteristics that define teams and how we look at team processes and performance. Next, the comm unication and coordination challenges that confront distributed teams are enumerated. Th e concepts of shared awareness and shared mental models are introduced here, as is research on shared visual space. Finally, a short review of the research on robots and huma n-robot interaction is recounted, focusing on the studies that relate to th e search and rescue domain. Teams Teams are an important organizing structur e in the workplace, a fact reflected in the burgeoning literature and re search on team performance, processes, and training. The increasing size, complexity and globalization of organizations have fostered the use of teams to speed development and delivery of pr oducts and services in a timely and costeffective manner. Moreover, the technology-dr iven changes in work settings often necessitate coordinated efforts between team members (Coovert & Foster Thompson, 2001). Teams exist as they perf orm cyclically over context and time, interacting among themselves and others (Ilgen, Hollenbeck, Johnson, & Jundt, 2005), and have an identity as a work group within a defined organizati onal function (e.g., a t echnical search team
9 within a USAR Task Force unit) (West, Bo rrill, & Unsworth, 1998). There are many definitions of a work team, most of whic h share the notions of interdependence and shared goals. For the purposes of this study, a work team is defined as two or more people with different tasks who work together adaptively to achieve specified and shared goals (Brannick & Prince, 1997). More than thir ty years of research on the factors which contribute to team performance has yielded a plethora of models and theories of team functioning. Theoretical models of team work typically focus on the inputs, processes and outputs that characterize team effectiven ess (Hackman, 1987; McGrath, 1984; Salas, Dickinson, Converse, & Tannenbaum, 1992), t hough recent models have included more complex representations of teams to include temporal influences (Marks, Mathieu, & Zaccaro, 2001), mediational effects (Ilgen et al., 2005), and multilevel constructs (DeShon, Kozlowski, Schmidt, Milner, & Wi echmann, 2004). Key to all of these models and theories is the assumption that various team variables and pro cesses contribute to successful team functioning. Va riables on the environmental, organizational, team and individual levels can influence the proce sses seen as integral to effective team performance: e.g., communication, coordina tion, situation awareness, leadership, adaptability, and support/backup behavior. In th is study, the input variab le of interest is the team level psycho logical construct of shared mental models This construct is discussed in detail later. Distributed Teams Distributed or virtual teams are those whose members are mediated by time, distance or technology (Driskel l, Radtke, & Salas, 2003). Distributed teams are usually project or task-focused groups. The team memb ership may be stable (e.g., an established
10 sales team) or change on a regular basis (e.g., in project teams). Members may come from the same organization, or from many different organizations. They can be colocated and work in the same physical spac e, but usually are thought of as working interdependently in remote, geographically separated workspaces. Moreover, they may work at different times, i.e. asynchronous wo rk-cycles. Other factors can influence team processes in distributed teams, such as wh ether the teams are assembled for a single, time-limited project or on a long term basis, whether team members know each other and have worked with each other before, and whether they expect to have any interaction/shared work in the future. Research on distributed teams has c onsistently reported special challenges in communication and coordination which negative ly affect team performance (Hinds & Bailey, 2003; Thompson & Coovert, 2003). The difficulties in establishing shared context across distributed teams can lead to incr eased conflict and conf usion between team members, and less satisfaction with team processes and outcomes. These difficulties are discussed in greater detail later in this chapter. All of the characteristics of distribute d teams described above are present in the US&R organizational structure, whic h consists of teams of teams. Distributed Teams in US&R Distributed teams in US&R are both project-and task-foc used, in that they are created specifically to perform certain tasks in response to disaster incidents, and are mobilized as part of a larger emergency mana gement effort, or project. Responders must go through a rigorous training a nd certification process to be eligible to serve on a regional or national US&R Task Force, and are often members of local fire rescue
11 departments. Members of the same Task Force may have worked together before; however, in most responses that require ac tivation of US&R functions, teams are drawn from all over the country, so it is very lik ely that rescue workers will be working alongside others from different teams for the single response incident. Teams work in 12hour cycles, and though there is some overlap, they must coordinate their efforts with others as they enter or leave the Hot Zone (immediate disaster area). In the Hot Zone, team members operate in deconstructed, unfamiliar environments and rely heavily on radio communication as they work. The physical environment is dangerous and workers are required to wear heavy personal protec tive equipment, which exacerbates cognitive fatigue by making even the simplest tasks (br eathing, walking) effortful. The work itself is highly stressful and time pressured, with se rious (life-threatening) consequences for error. Technical search (Figure 1) is one of the four US&R functions: search technical support medical and rescue or extrication. These four operations represent subspecialties within the task force. Technica l search teams are the particular type of distributed team examined in this study. Description of technica l search team operations While no two disasters are managed precisely the same way, US&R techni cal search operations often begin with a manual reconnaissance of the area of damage, called the hot zone Victims on the surface or easily removed from light rubble are extracted immedi ately as encountered. After reconnaissance, the command staff determines wh at the safest strategy is to effectively search the hot zone for survivors within the rubble. In areas that are deemed safe for humans to investigate, canine teams may be sent forward. In most cases, technical search teams wait until called for. When a dog has indicated signs of a survivor in an area,
12 technical search specialists are summoned ont o the pile. The command staff attempts to minimize the number of people in the hot zone so technical search teams wait at the forward station of the hot zone perimeter until called over the radio or assigned an area to search. Technical search specialists may carry a fiber-optic boroscope, thermal imager, or a video camera mounted on a wand for a visual inspection of the rubble, depending on the verbal description of the void or the sp ecific request of a part icular device by the leader. If a survivor is found, the search t eam and command staff brings in the medical and rescue teams, who call on members of the technical support team as needed. Before leaving the void, the technical search team ma rks the exterior of the void with symbols indicating that it has been searched, the structural condition, and presence of survivors/remains.
Figure 1. Organizational structure of USAR Task Force (FEMA, 1992). The visual inspection of a void is most often done with a boroscope or a camera on a wand. These technologies generally cannot penetrate more than 12 feet into a void, whereas robots are well-suited for voids longer than 20 feet. Regardless of tool, the search activity takes on the order of 3-30 minutes, and a technical search team may spend most of a 12-hour shift waiting, and then work furiously for a few minutes. The command staff may periodically evacuate the hot zone and cease all operations so that 13
14 dds ical search task ith as rdination of ac tivities pose significant challenges to can affect team performance. Communication in distributed s c al e m technical search specialists can apply sensitiv e acoustic listening devices. This also a to the cognitive stress. The field data collected in this study used the robots for a visual technical search task, where robots served as cameras on wheels. The visual techn consists of four activitie s in order of importance: search for signs of victims report of findings to the team or task force leader, not e any relevant structur al information that might impact the further in vestigation of the void, and estimate the volume that has been searched and map it rela tive to the rubble pile In this case, the technical search teams operated a robot instead of a boroscope or thermal imager. Technical search, along w the other primary tasks of victim rescue and extraction, medical care and patient transfer, requires close coordination of efforts with bot h co-located and remote team members well as prompt and accurate information tr ansfer to incident command, local medical authorities, and others involved in the re sponse. The communication and coordination challenges faced by distributed US&R team s are by no means unique, but are certainly exacerbated by the extreme environment and other stressors. Communication and Coordination Challenges in Distributed Teams Communication and coo distributed teams which teaman suffer from the loss of informati on gleaned through back channels such as physical gestures, body language, an d interaction with artifacts in the environment. AI or computer-mediated technologies that do not support transmission of contextu information are impoverished and provide less visibility and feedback, both of which ar needed for establishing and maintaining mutual knowledge i.e. knowledge that tea
15 high d, re derstood as in tended (Clark & Ma rshall, 1981). Three rimary pposes s members share and know they share (Kra uss & Fussell, 1990). Computer-mediated communications impact on mutual knowledge is likely to be greater for tasks where individual team members possess a large quantity of unique information, and where contextual information between remote sites differs. This problem is attenuated by requirements for complexity, workload and in terdependence, and can lead to confusion among team members and errors in performance (Cramton, 2001). Coordination of activities, whic h requires shared awareness, or common groun is difficult when team members are distribu ted, and often requires more confirmatory communication, since many of the back channe l types of awareness mentioned above a unavailable. Successful collaboration among dist ributed team member s requires situation awareness ongoing awareness of what each pe rson is doing, status of task, and the environment (Endsley, 1995) and conversationa l grounding working with each other to ensure messages are being un p sources for common ground are comm on group membership (which presu a set of common knowledge), lingui stic co-presence (h earing the same verbalizations), and physical co-presence (inhabiting the same physical setting). Physical co-presence provides multiple resources for building common ground, most prominently visual copresence. Shared awareness and conversa tional grounding are integral components in creating shared mental models. Shared Mental Models The concept of shared mental models has been invoked to explain team dynamic and performance for many years (Mathieu, Heffner, Goodwin, Salas, & Cannon-Bowers, 2000). Shared, or team mental models re present efforts to simplify events or
16 rdinate l oup Wellens e ls ntal and responsibilities to make them more tractable, and are an em ergent characteristic of the team. Shared mental models serve an or ienting and coordinati ng function in team processes. They contain organized knowledge and may hold a variety of content, e.g., representations of task s, situations, response patterns, or working relationships (Klimoski & Mohammed, 1994). Shared mental models are thought to improve team performance in several ways. First, they enable team me mbers to form accurate explanations and expectations for a task. Second, shared mental models allow team members to coo actions and adapt behavior to task demands Lastly, they can facilitate information processing (Kraiger & Wenzel, 1997). Examples of applications of the shared menta model construct include Orasanus shared situation models (1990) and Wellens gr situation awareness model (1993). Orasanus shared situation models among air crew members included shared understanding of a problem, and team member roles described group situation awareness among di stributed decision making teams as th sharing of a common perspective regarding cu rrent environmental events, their meaning and projected future status. Shared mental models can be described in terms of their focus or content. Cannon-Bowers, Salas and Converse (1993) listed four types of t eam mental models: equipment/technology, task, team interaction, and team. Equipment/technology mode include knowledge of operating procedures, equipment functions, limitations, and likely failures. Task models content includes task procedures and stra tegies, environme constraints, likely contingenc ies and scenarios. Team inter action models reflect role responsibilities, information sources, interaction pattern s and communication channels. Lastly, team models contain knowledge of team members knowledges, skills, abilities
17 ut ent of measures and their interrelationships to ade quately assess a) how team members perceive, process or react to external stimuli, b) how they organize or structure task-related knowledge, c) common attitudes or effect for task-relevant behavior, and d) shared expectations of behavior. In the proposed framework, antecedents of shared mental models are classified as environmental, organizational, team or individual. These determinants affect the development of shared mental models in terms of knowledge, behaviors, and attitudes. Shared mental models, in turn, may affect both team effectiveness and team performance. The au thors note the reciprocal nature of the relationships, showing that these outcomes can have an influence on all of the preceding factors in the framework. In their seminal article, Klimoski and Mohammed (1994) use the term team c 003), representing efforts to simplify events or responsibilities to make them more tractable. Team mental m odels are thought to affect decisio and other characteristics. Shared mental models can further be characterized in terms of knowledge type (e.g., declarative, procedural, structural), leve l of specificity (abstract, concrete), or function. They can provide in formation about what an element is, how it works, or why it is needed. Moreover, they may contain not only types of knowledge, b also behaviors and attitudes (Kraiger & Wenzel, 1997). This makes the measurem shared mental models a complicated undertakin g. Indeed, Kraiger a nd Wenzel argue for a construct-oriented approach, i.e., identifyi ng a nomological net of related concepts, mental model to define the sh ared mental model construct as an emergent characteristi of the team, similar to Cooke et al. (2 n-making, team dynamics and performance, and to enhance the quality of teamwork skills and team effectiveness. Training, team composition and life cycle,
18 tal ong with leadership) influen 2). communication patterns and cohesion are listed as determinants of team mental models. They acknowledge that content and form of team mental models can vary, and depend largely on the function (mission/task/subtask) of the team and its members, as well as the situational context. Further, team mental models reflect internalized beliefs, assumptions and perceptions, and exist to the extent they are apprehended by the team members at some level of awareness. Klimoski & Mohammed placed the existence of team menmodels in a framework of team performance as a factor (al cing performance directly and indirectly through its effect on team capacity, i.e. a teams latent potential for demonstrating effective process and performance (Figure Figure 2. Klimoski & Mohammeds framework for explaining the role of team mental models in team performance (1994).
19 bers potential for performance, 2) le to the t eam. A teams capacity (4), or read ing g nd effectiveness. Suppor s iving to Figure 2 walks you through the framework, be ginning with the f actors thought to determine team capacity: 1) the individual team mem team composition and size, and 3) resources availab iness, has an impact on team proce ss (7) and performance (8) contingent on the availability of some orienting factor that enables the team to harness that capacity, to put it to work; team mental models and leadership (5 and 6) are 2 such factors. The orienting and coordinating aspects of team mental models can create smooth functioning teams through shared cognition. Interestingly, the authors imply leadership can serve as a guiding factor when that shared cognition is not possible: absent the availability of TMMs a lead er can serve to guide the team, serv such executive functions as assigning in formation gathering activities among team members, doing information integration and interpretation, adjudicating disagreements and/or directing individual team members into action (p.431.) It may be that RSVP technology can assume a leadership role (or help team members do so) by performing some of thes e executive functions (Coovert & Burke, 2005). RSVP technology may benefit ot her team processe s (communication, support/backup behaviors, situation awareness) in similar fashion. For example, givin team members access to RSVP may increase communication clarity a t/backup behaviors may occur more frequen tly due to quicker de tection of error and better monitoring capabilities. Team s ituation awareness may be enhanced by g all team members access to previously restrict ed sources of information, enabling them make projections as to each others inform ation needs before being asked. The model utilized in this study posits RS VP as a team resource that is used to help create a team
20 s and ce, f n in 2) est work am comm work ld specifically w ith human police SWAT teams). y in this article recorded real robot-user interaction as it occurred context and situation, work process and dom ain-specific information were needed to situation model on-the-fly. Therefore it serves as a test of the theo retical construct relationships outlined in Klimoski and Mohammeds (1994) model of team performan specifically looking at the infl uence of technology as a reso urce enabling the creation o richer shared mental models among team members. Team Communication and Shared Mental Models Research on team communication and shared mental models in police SWAT teams and other domains offers similar findi ngs regarding the role of communicatio building shared awareness. Jones and Hinds qualitative analysis (Jones & Hinds, 200 of police SWAT teams (an extreme team doma in similar to US&R) is the clos conceptually to the goals of this study. Jones and Hinds expl ored the importance of te communication in the development of a shar ed mental model (which they termed on ground), and noted the implications for SWAT team performance. They observed police SWAT teams in training exercises, and identified leader roles in establishing common ground and c oordinating distributed team member actions as factors transferable to system design for coordinati ng distributed robots. Jones and Hinds studied distributed SWAT teams (people) to mo del a team of distribut ed robots that cou work together in similar fash ion (but not In contrast, the field stud between team members and a si ngle robot to inform the development of coordinated human-robot systems within the or ganizational struct ure of US&R. In a study of military command and cont rol exercises, (Sonnenwald & Pierce, 2000) found that frequent communications between team members about the work
21 of ember comm unication, (Orasanu, 1990) found tion) e effects of fatigue. 6 t maintain shared situation awareness in dynami c, constraint-bound cont exts. In a study the cognitive functions of cockpit crew m that captains of high performing crews explic itly stated more plans, provided more explanations, and made more predictions, wh ich were articulated for the whole crew This enabled crew members to contribute rele vant information or strategies from their specialized perspectives, and to interpret requests and commands unambiguously. (Foushee, Lauber, Baetge, & Acomb, 1986) studied the effects of fatigue on crew coordination and performance, and suggested that team processe s (e.g., communica contributed to the development of shared mental models in crews. They found that superior performance was associated with more task-related communications among crew members, specifically more comma nds, suggestions, statements of intent, exchanges of information, and acknowledgeme nts. They also found that crews with mental models based on shared experiences we re able to overcome th Mathieu et al. (2000) noted that team communication mediated the relationship between mental model convergence and team effectiveness in a la boratory study using 5 undergraduate dyads who "flew" a series of missions on a personal-computer-based fligh combat simulation. This study uses the findi ngs regarding the criticality of shared awareness in team-based, dynamic work domains as a justification for exploring team communications in the US&R domain. Shared Visual Presence Prior research investigating the effect of shared visual presence in collaborative physical tasks shows that the availability of a shared visual workspace can positively impact performance and team process in two ways: through creating awareness of the
22 ons, subtasks within the co llaborative process of ed l f und to task state (i.e., situation awareness) a nd through providing an e fficient resource for conversational grounding (Gergle, Kraut, & Fussell, 2004b; Kraut, Fussell, & Siegel, 2003; Kraut, Gergle, & Fussell, 2002). Research investigating the role of visual informati on in collabor ative physical tasks decomposed the visual information available when people share physical copresence into 4 categories: participants heads and faces, participants bodies and acti task objects, and work environment/context (K raut et al., 2003). Each type of visual information offers certain benefits in various maintaining situation awareness and grounding conversation. Video conferencing systems usually focus on the participants heads and upper bodies, which affords limited benefit in attaining situation awarene ss and common ground. Research on workspace oriented video systems that provide input on task objects and work environment/context suggests it is likely to be more useful in supporting SA and conversational grounding ((Fussell, Setlock, & Kraut, 2003; Fussell, Setlock, & Parker, 2003). Investigations comparing different shar ed fields-of-view revealed that a sceneoriented view of the workspace was more conducive to performance than headmount video. In a study examining distributed partic ipants performing a collaborative physica task (Kraut et al., 2003) the use of head-mounted video did not seem to aid performance in terms of speed or accuracy. It did change the nature of communications between the participants, allowing for use of deictic re ferences to task objects. However, the limitations of the field-of-vision provided by head-mounted video led to more queries designed to establish a shared field of view. In a follow-up study comparing the effects o head-mounted video to workspace-oriented vi deo, the scene oriented video was fo
23 such on h rs cond that a) be superior in terms of task completion, co mmunication efficiency and user ratings of work quality, suggesting that providing remote helpers with a wide-angle, static view of the workspace was most valuable (Fussell, Setlock, & Kraut, 2003). The researchers noted the difficulties in gaining the advantages of static cameras in mobile settings as emergency telemedicine or remote repair Finally, in a relate d gaze study using eye tracking technology during a co llaborative physical task (F ussell, Setlock, & Parker, 2003), results indicated that the remote helper s gaze was most directed at task (task objects, pieces/tools, and workers hands), i.e. targets relevant to gathering information about steps to be completed and task status. Additional studies conducted by Gergle, Kr aut, & Fussell suggest that a comm visual referent facilitates th e development of shared mental models of what each other knows or assumes of the situation, which is critical for team members to coordinate activities necessary to accomplish team goals. Th ey reported that pairs with shared visual space in a collaborative puzzle task perfor med more quickly and accurately, and were more likely to use deictic references, and less lik ely to explicitly verify their actions wit speech; that is, they relied on observed actio ns to provide the necessary communicative and coordinative cues (Gergle, Kraut, & Fussell, 2004a; Gergle et al., 2004b). They examined a collaborative puzzle task manipul ating the extent to which team membe viewed the same work area a remote assist ant had access to the same view, a 3-se delayed view or no view. They also manipulat ed various features of the visual space, such as proportion of shared field of view, sp atial perspective, and color drift. Dyads had a simultaneous shared view performed a bout 1/3 faster than did the dyads in the delayed view or no view condition. Results sh owed that having a shared visual space
24 hen the information is visually complex and n vocabulary for describing the world is pants. Se quential analysis of conversat ional discourse between team membe rlier, they ha d a shared visual space (though these results vary de of d the remote om ies ffect helps team members understand the current st ate of the task, and enables them to coordinate activities and communicate efficiently and b) is even more important w o simple available to partici rs revealed that they used the visual information in two ways: 1) as a more efficient, less ambiguous source of confirma tion (thus team members are less likely to verify w/speech) and 2) for coordination cues, e. g. team members can de tect errors ea and remedy them before their actions become nested and more difficult to untwine. Overall, these studies show that dyads working together on a collaborative task were 30% faster on average when pending on the features of the shared visual space), and that the availability shared visual space supports communication by creating a shared aw areness of the task state and by serving as an efficient res ource for conversational grounding. In each of these studies, one of the partic ipants is operating in the task environment and the other is assisting remotely in the pro cess. In robot-assisted search, both the robot operator an tether-handler are removed from the task en vironmentthe robot serves as the presence in the environment for both of them The robot operator te leoperates the robot using the Operator Control Unit (OCU), whic h provides audio-visual information fr the robots camera as the robot moves through the remote environment. The tetherhandler can manipulate the r obot grossly through manueve ring the tether, and can sometimes physically see the robo t when it is first inserted in to the void, but mostly rel on communication with the operator to particip ate in the task. My interest is in how providing both team members with a common vi ew of the remote work space may a
25 team pr it al es not environ s e their e ocesses and performance in robot-assisted team tasks. This work extends the exploration of shared visual presence into a real world domain application. Moreover, provides a rare realtime obser vation of human-robot intera ction in a work setting. Robots What is a Robot? The term robot came from Karl Capeks 1921 play R.U. R. (Rossums Univers Robots). It was used to describe a race of menial workers, artificial humans created from a vat of biological parts to serve as slave labor for real humans. Science fiction books and movies transformed robots into mech anical creatures, a nd propitiated their menial stance by portraying them as factual-minded automatons that mimicked human qualities without understanding. In reality, an intelligent robot is a m echanical creature which can function autonomously and interact with its world (Murphy, 2000). Intelligen ce implies it do perform in a mindless fashion, while autonom y means it can adapt to changes in the ment (or itself) and conti nue to reach its goal. It is im portant to note that a robot goals are generated by a human, not by the robo t itself. This is a critical distinction pointed out by Clancey (2004) that restricts one from re ferring to a robot as a collaborative team member. Br ooks (2002) defines two principl es that distinguish robots from computers: situatedness and embodiment. Robots are situated in that they are embedded in the world, and interact with th e world through sensors which influenc behavior. They are embodied in that sense of having a phys ical body that experiences th world in part through the influence of the world on that body. Like computers, robots
26 re n the worlds of entertainment, work and everyday life. nally been used for th e three Ds: dull, dangerous or dirty g, e ut arch Proj ects Agency (DARPA). Mobile robots have manitari an concerns, and are the primary focus in search n g ator has supervision). Others have built upon the noti on of shared control, where the robot does have evolved from research laboratories and military/industrial applications, and a rapidly gaining a presence i Robots have traditio work. Industrial robots have been devel oped for economic reasons in manufacturin agriculture and service industries, to incr ease productivity and re duce inefficient human resource allocation, particularly in hard-t o-staff menial labor positions. Because th original goal was precision and repeatability for use in mass production, little effort was put into machine intelligence or human fact ors considerations. As the space program evolved, the need for artificial intelligence, i.e. robots capable of learning, planning, reasoning and problem-solving, spurred resear ch sponsored not only through NASA, b also by the Defense Advanced Rese developed more from safety and hu nuclear, space exploration, military and rescue applications. This study is directed toward human-robot interaction with mobile robots. While the pervading noti on in past re has been the substitution of robots for peopl e, the current trend is toward robots as assistive technology, i.e. designed to comple ment humans rather than replace them. The current state of the art in mobile robot s is situated autonomy (the robot acts on its own using information from its sensor s), though teleoperation is more common i practice. Teleoperation is when a human opera tor controls a robot from a distance usin sensors and a display. (This differs from re mote-control operation, where the oper visual contact with the robot). Some app lications have moved to semi-autonomous control, where the robot is given an instru ction or task to do on its own (but under
27 ore obotic by cognitive functionality but rather according to the domain into and with a the tric of Yanco, the dirty work and the human does that whic h requires finesse. Cert ainly there are m autonomous applications in the commercial sector (Hondas Asimo, Sonys Aibo r dog), but systemic problems have slowed th e rate of development in military and governmental application. Human-Robot Interaction Human-robot interaction is a relativel y new field. Studies in human-robot interaction are often categorized not functionality of the r obots in question: industrial, professional service, or personal service robots (Thrun, 2004). Urban sear ch and rescue (US&R) robots fall the professional service category, along with th ose in the medical field, the military space applications (e.g., Ambrose et al., 2000; Endo, MacKenzie, & Arkin, 2004; Pineau, Montemerlo, Pollack, Roy, & Thrun, 2003), wh ere robots are intended to work human to meet the humans goals. Human-Robot Team Performance Studies concentrating on improving ta sk performance in human-robot teams appear to be universally r obot-centric, testing either th e robots abilities or some component of the robot as a f actor influencing performance, and ignoring the role of human. This provides no insight into the la rger human-robot team. Professional service robots assist people in attaini ng their professional goals; ther efore, a logical me human-robot team performance is whether th e teams goal is achieved. However, the question of how to measure human-robot inte raction in terms of team performance is largely ignored, with the notable excepti ons of Bruemmer et al. (2005); Marble, Bruemmer, & Few (2003); Nourbakhsh et al., (2005); Scholtz, Young, Drury, &
28 or -focused, this study relies on methodologies taken from the psychology commu ke, d environ s and the (2004); Yanco, Drury, & Scholtz, (2004). Since these methods are gene rally usabilityevaluation nity and encapsulated in the RASARCS scheme described in Chapter 3. To date, no one has attempted to exam ine human-robot team performance from the human side, i.e., through investigation of team processes. Situation awareness and team processes have been identified as important elements needed for effective communication and coordination of activities in robot-assisted US&R operations (Bur Murphy, Coovert, & Riddle, 2004). The annual Robocup Rescue Competition ha s been used to compare human-robot teams performance in a rescue-oriented domain (Scholtz et al., 2004; Yanco et al., 2004). However, the physical setting and conditions are quite different from those experienced at a disaster site, and the robots used ar e not fieldable, i.e., they are designe for short competition rounds in the NIST te stbed rather than for a true disaster ment (Murphy, Blitch, & Casper, 2002). Moreover, the people on these teams are robot developers rather than rescue profe ssionals, and have neither the training nor the skills of the intended end-users. Nourbakhs h et al. (2005) have created a simulated US&R environment in which humans, agents and robots operate in high-fidelity gamegenerated simulations of the NIST US&R test arenas, allowing them to test various configurations of heterogeneous agents a nd robots in actual Robocup competitions as well. Marble, Bruemmer and associates at the Idaho National Engineering and Environmental Laboratory have measured hum an-robot team performance in search task conducted in laboratory settings and in field locations, varyin g the levels of control autonomy (Bruemmer et al., 2005; Marble et al., 2003). These experiments, by far
29 l end-user s in ecologically valid disaster response t red phy, ies, a most advanced in terms of experimental design and analysis, are gauged toward evaluation of robot systems characteristics, and participants are typically high schoo students. Field studies conducted with true settings offer a more realistic look at human -robot team performance in terms of curren capabilities. Four field studi es conducted by the Center fo r Robot-Assisted Search and Rescue (CRASAR) prior to this study recorded real robot-user interaction as it occur between team members and a si ngle robot to inform the development of coordinated human-robot systems within the organizati onal structure of US &R (Burke & Murphy, 2004b; Burke, Murphy, Coovert et al., 2004; Casper & Murphy, 2002; Casper & Mur 2003). In each of these, situation awaren ess (Endsley, 1988) is a key construct for understanding (and improving) human-robot interaction. In the two most recent stud team processes, particularly communication between team members, have emerged as critical to the development of situation awareness in r obot-assisted team tasks. The first CRASAR field study was an ethnogr aphic study of Flor ida Task Force 3 members using robots to search for a vict im (Casper & Murphy, 2002), while the second was an analysis of data co llected during the us e of rescue robots at the World Trade Center disaster (Casper & Mu rphy, 2003). The Florida Task Fo rce 3 study suggested that two operators are needed to interpret multiple sensor data while navigating due to the simultaneous nature of activities described as part of the technical search task (searching for victims and structural inspection). Casper and Murphys (2003) anal ysis of video dat collected during the World Trade Center disast er response found that operators lack of awareness regarding the state and situate dness of the robot in the rubble impacted
30 h process. In Burke et al. (2004), team members created shared mental models (mutual knowledge) of the search space through talking about the environment, the robots situatedness in that environment, and search strategy. Operators who reported to team members about the en vironment being searched by the robot and search strategy were rated as having better situation awaren ess (Burke & Murphy, 2004b). If the tether-handler or another distributed team member had access to the same robots eye view, the effort required to establish mutual knowledge, or common ground, would be far less. This is important because analyses comparing the performance of operators performance of human-robot teams. Operators also had difficulty linking current information obtained from the robot to ex isting knowledge or e xperience. Both the Florida Task Force and World Trade Center human-robot interaction studies reveal difficulties in operator teleproprioception and telekinesthesis, consistent with the problems described in Sheridan (1992). Situation Awareness and Team Processes. In the two most recent field studies (Burke & Murphy, 2004b; Burke, Murphy, Coovert et al., 2004) conducted with 33 teams (robot operator-tether handler), communication analyses revealed that 50 60 % of operator communication during a technical search task was related to building and maintaining situation awareness. One of the challenges presented was the fact that the robot operator was cognitively overloaded; he couldnt drive and look at same time. Because the tether-handler did not share th e same viewpoint, the operator alone had to interpret the robots eye view, and used talking with th e tether handler (who might sometimes have an external view of the r obot) to build common ground so that the tetherhandler could assist in the s earc
31 rated as having good or poor situat ion awareness in a victim search scenario revealed that those with high SA were 9 times as likely to locate the victim as those with low SA. This chapter has covered a great und. Teams, distributed teams, and distributed teams tion and e) rse a teams capacity, enabling th e creation of richer shared mental models ng tive deal of gro in US&R have been described, as have the communica coordination challenges in distributed teams. Th e concepts of situati on awareness, shared awareness, common ground, and shared mental models (all variants on a similar them are introduced, and the research on shared visual presence as a conduit for these constructs is reported in de tail. Finally, robots, human-r obot interaction, and relevant studies on human-robot interaction in US&R are presented and discussed. These dive topics all play a part in the current study, which is described more fully in the coming chapters. The current study is a theoretical piece, testing portions of Klimoski and Mohammeds model of team performance (1994). RSVP is presented as a team resource that increases amo team members. These richer mental mo dels can, in turn, foster more effec team processes and enhance team performa nce. The approach for testing the model (which includes the creation of a more cont extand task-specific modela submodel, if you please) is now explained.
32 ses and el present ting a Chapter 3 Mobile Robots as Shared Visual Presence: Approach This chapter outlines the proposed mode l of team performance for human-robot teams in technical search, and presents sp ecific hypotheses. The research questions addressed in this study are as follows: Does using the robot as a remote shared visual presence affect robot-assisted team performa nce? Team process? Can this remote shared visual presence facilitate the performan ce of human-robot teams where human team members are distributed? The model proposed in this study posits that use of a remote shared visual presence (RSVP) by team memb ers will lead to creation of richer team mental models. These richer team mental mode ls will in turn enhance team proces performance. As described in Chapter 2, there are well-established theoretical models circumscribing the role of shared mental models in team performance. Kraiger & Wenz (1997) place shared mental models within a nomological net of determinants and outcomes, along with measurement implicati ons. Klimoski & Mohammed (1994) a framework for explaining the role of team mental models in team performance (Figure 2). These theoretical frameworks are used to situate this study with in the body of exis research. Indeed, the proposed model and study se rve as a test of some of the theoretical constructs and relationships posited in these frameworks. The model presented in this study is more specific in two ways. First, it is limited to the US&R environment, and particular task within that environment. This is importa nt because the dynamic, high
33 a situationl., ight on some of the cloudier issues relate d to team mental models: how do these shared mental models r, adership, situation awareness) and performance? Figure 3 shows the conceptual model of team mental model formation for robotassisted technical search. It is a synthesis of the results of previous field studies and is consistent with Kraiger & Wenzels (1997) framework for mental models, but is contextually task-specific, and focuses on team communications as central to the development of the team situation model. The model is explained following the diagram, bottom upward. The human-robot team consis ts of two operators and one robot. The operators are the robot operator and the tether-handler The robot operator is the person who directly operates the robot. The tether-handl er handles the robot te ther or safety line. The input to the robot operator is data (video) from the robot processed through the Operator Control Unit (OCU). The OCU serves as the user interface Both the robot stress, high consequence nature of this task in this environment adds cognitive load to seemingly low-complexity task. The data colle cted is deliberately c ontextand specific to inform analysis of this technologyaltered task. Second, this model posits that team communication can be used as a measure of the shared mental model (Cooke et a 2003; Kiekel, Cooke, Foltz, & Shope, 2001; Or asanu, 1990; Orasanu, 1995)in that the quality of the shared model is related to the presence of various indicators in team communication: that they talk more about goa l-related aspects of the task (environment, robot situatedness, search strategy, informati on synthesis), engage in more planning and reporting, and anticipate each ot hers information needs and provide information without being asked (communication efficiency). Thus, this model attempts to shed l actually contribute to team processe s (communication, backup/support behavio le
34 k f the void space (and at some ps this study), the tether-handler is given a separate monitor that display operator and the tether-handler can view the OCU; the robot operator generally looks constantly at the OCU, and the tether-handler typically only looks at the OCU intermittently as responsibilities permit (often he is located several meters away from the robot operator.) This means that the tether-handler has a different perspective on tasprogress and the overall situation since he can see the exterior o oints what the robot is doing), and can feel the robots movements through the tether (e.g., whether it is moving forward and drawing more line). To explore the effectof creating a remote shared visual presence between the two team members (one of the two dependent variables in s the same robots view which appears in the robot operators OCU. Figure 3. Model of team performance in robot-assisted technical team search.
35 fte n the case with US&R teams) these sh of the problem goes fu at fusion takes place via communication between the human ls (Fiore & 1990; Sonnenwald & Pierce, 2000). The formati on of a team situation model is an or awareness of a given process explicit shared mental situatio ith technic here 50-60 % of operator communications was Each team member has a situation mental model representing his or her current understanding of the task, system, and team member roles. Situation models extend beyond the more static mental models of th e system, task and team to represent the dynamic, present state of the system. The model posits a team ( shared) situation model as a fusion of the two individual situation m odels into a common ground (Jones & Hinds, 2002). The team situation model concept fo llows Orasanu (1990), who suggested that teams faced with novel situations or emergencies (as is o must also develop shared situation models for the specific problem. Important parts of ared situation models according to Orasanu include shared understanding goals, information cues, strategies a nd member roles. The model in this study rther and assumes th team members, which is consistent with re search on team processes and mental mode Schooler, 2004; Foushee et al., 1986; Kanki, Lozito, & Foushee, 1989; Orasanu, emergent process aptly described in (Fiore & Schooler, 2004) only as one articulates ones underst anding does it truly become known to oneself and others. Thus, the act of making knowledge facilitates the developm ent of not only ones own mental model but also a model. (p.143). In other words, communication plays a c onsistent role in both the operators n models and the team situation model. This is supported by previous work w al search operators and teams, w
36 2004b; backup perform While the quality of the user interface will have some impact on individual SA, this study is restricted to the formation of the team situati on model through team communication, and its influence on team pro cess and performance. This restriction permits the study scope to be tractable. Hypotheses Hypotheses are presented rela ted to each of the two e xperimental conditionsthe presence of RSVP technology, and whether the teams are collocated or distributed. If the use of RSVP has the hypothesized effect, then differences will emerge between teams that have access to the technology and teams that do not. Based upon what is known about distributed team performance compared to that of collocated teams, the collocated teams will outperform the distributed teams. If using RSVP makes a difference, however, it may close the gap between collocated and di stributed teams in te rms of team process and performance. Specific hypotheses are listed below: H1: Teams having access to RSVP technology will generate richer, more accurate team situation models than teams not having access to the technology, as measured by search maps generated by the team, and frequencies of SA indicators in the RASAR-CS. related to building and maintaining SA (Burke & Murphy, 2004a; Burke & Murphy, Burke, Murphy, Coovert et al., 2004). In the model, the team situation mode l affects team processes (communication, behaviors, leadership/initiative, and team SA), which in turn affect team ance.
37 H2: Teams having access to RSVP will exhib it more effective team processes, as measured by observer ratings and by frequenc ies of team process indicators in the RASAR-CS. H3: These (RSVP) teams will accordingly perf orm better in the search task scenarios, uted teams in the search task sce utcome scores. VP technology, then th e differences between tilize as measured by performance outcome scores. H4: Collocated teams will generate richer, more accurate team situation models than distributed teams, as measured by se arch maps generated by the team, and frequencies of SA indicators in the RASAR-CS. H5: Collocated teams will exhibit more e ffective team processes than distributed teams, as measured by observer ratings and by frequencies of team process indicators in the RASAR-CS. H6: Collocated teams will accordingly perfor m better than distrib narios, as measur ed by performance o H7: If there is a main effect for use of RS collocated and distributed teams will be si gnificantly less in the teams that u RSVP than in those that do not.
38 R t it nd d-multilevel regression. Setting, Participants, and Apparatus Setting h Chapter 4 Method This chapter describes the details of the study. Due to the cost and rarity of US& field exercises, the data used is archival. The pr oblem with archival data analysis is tha is constrained, in that you cant always as k more questions. However, this data was collected with a goal in mind, and therefore is presented as a quasi-experimental field study with great attention and e ffort directed toward addressi ng threats to the validity a reliability of the results. The following secti ons discuss the participants, apparatus, and setting, the quasi-experimental design, measures used, and pr ocedures followed. The final section describes the type of anal ysis use Data collection took place in two diffe rent locations during US&R training exercises. The first exercise was con ducted in May, 2004 at NASA-Ames Research Center in Menlo Park, CA. NASA-Ames is ho me to one of the most advanced Urban Search and Rescue teams in the country. NAS A's Disaster Assistance and Rescue Team (DART) was organized almost tw enty years ago in the agency's attempts to comply wit disaster preparedness regulations for fede ral facilities. DART comprises nearly 140 personnel from the Research Cent er who have been trained a nd certified in a variety of emergency response and recovery systems and techniques. Most its members are also a part of California US&R Task Force 3. Th e training exercise that took place was a
39 rt vited to r Station at urst in Toms River, New Jersey and has leased property from the Federal Govern s of s) chosen through convenience sampling of the intended end-user population (US&R task force personnel) via their participation in sc heduled training exercises c onducted at NASAAmes, Moffett Field, CA, and Lakehurst NAVAI R, Lakehurst, NJ. Study participants at the NASAAmes site included 13 responders from other parts of the country who came to participate in the training exercise with DART members. All study participants at the New Jersey site were members of NJTF-1. Participant demographics were co llected, incl uding age, gender, and relevant experience. The majority were males (90%) between the ages of 3554 (76%). Participants received no payment for their participation; it was viewed as a training accustomed to putting in long Technology Meets Responder event in which various new technologies were introduced and used by the responders during the course of the training exercise. As pa of the event, approximately 30 responders from all over the United States were in attend and participate with DART members in the exercise. The second exercise was held in February, 2005 at the Lakehurst Naval Air Warfare Center in Toms River, NJ. New Jersey Task Force-1 (NJTF-1), the states ow n Urban Search & Rescue Task Force, is based there. The team consists of career and volu nteer fire, police, and EMS personnel from all 21 counties in New Jersey. The task fo rce is quartered at the Naval Ai Navy/Lakeh ment at Navy Lakehurst to support the logistical and training operati onal need the team. This training exercise was a sim ilar Technology Meets Responder event, with participants drawn solely from NJTF-1. Participants Participants were 62 men and women (31 2-person team experience, and US&R task force members are
40 hours of unpaid training just to attain/mair certification as US&R teams. The particip rticipants not being able to complete both runs. Power analysis for this study showedequipped with a color CCD camera on a tilt unit and two-way audio through a set of ers on the robot and Operator Control Unit. The operator is given basic c intain the ants knew that we were collecting data for research purposes, but they did not have specific information about what we were studying other than human-robot interaction. The final sample used in this analysis was reduced to n=50 (25 teams), due tosome pa that to attain power = .80 in a single-group repeated measures design at alpha = .05, given an estimate of r = .80 as the average correlation for the repeated measures, a total of 15 teams are required to detect a medium size effect (Stevens, 2002). Apparatus The robot systems used in the study were Inuktun Micro Variable Geometry Tracked Vehicle (VGTV) robots. Each robot system consists of a small, tracked platform microphones and speak ontrol capability: traversal, power, camera tilt, focus, illumination, and height change for the polymorphic robot (Figure 4.) Figure 4. Inuktun Micro-VGTV robot system.
41 le sence (RSVP), described elow: Location (collocated vs. distributed teams). The robot operator and tether-handler y-si di se visually in the other. Due to the constrainara ting walls or corners on the pile), for the distributed artice told to face away from each other (see Appendix A Task o Instrucpeci) Ters in th e distributed condition were allowed to talk, but not use gestures, ey e contact, or other beha viors associated with working side-by-side. era connected to the OCU; in the other task, only the robot operator has access nd remote as poss s Design This is a 2 x 2 repeated measures desi gn, partially crossed, counterbalanced for location and remote shared visual presence. E ach team participated in 2 of the 4 possib conditions. The 2 IVs are locati on and remote shared visual pre b work side-b de in one con tion, and are parated setting ts (no sep condition p ipants wer Scenari tions for s fic wording eam memb Remote Shared Visual Presence (RSVP or no RSVP). In one condition, the robot operator and tether-handler have access to the same visual image (robots view) via a second DV cam to the visual image. In order to determine the effects of the independent variables (location a shared visual presence) on the various depe ndent variables (shared mental models, team process, performance), efforts were made to limit or control any ex traneous or nuisance variables that might influence the results. For example, testing conditions were as nearly ible the same for each team. A sta ndardized protocol for running each team through the experimental task was developed and followed. The instructions were given the same way each time; runtimes and observations were consistent and accurate in term
42 Table 1 Distribution of teams acro ss experimental conditions Collocated Totals of detailed documentation. Experimenters were friendly, but minimized conversation with participants during the tasks, and a voided discussing the experiment, extraneous information about the robots or the research, etc. The number of teams nested in each combination of experimental cond itions is shown in Table 1. Condition Distributed RSVP 15 teams 14 teams 29 teams No RSVP 10 teams 11 teams 21 teams Totals 25 teams 25 teams 50 teams Measures Measures used in this study include an initial demographic survey questionnaire administered to individual participants, and dependent variable instruments assessed at the team level. Survey questionnaires were designed to collect demographic data from participants regarding thei r background prior to the ta sk. Participants provided information about age, gender, years of e xperience in firefighting, US&R, mil self reported skill with various technologies (Appendix B). Age and years of experien itary, and ce were codified by range, and technology skill was rated on a 1 (low)-5 (h igh) Likert scale. e, am processes, and shared mental models. Team performance is measured as an outcome ch task scenario. Team processes are measured through The dependent variables in this study fall into three categories: team performanc te score on a confined space sear
43 onsite o earch ert et Team Performance Measure Four visual cues were placed in each of the search spaces. These target objects were mannequin pieces (upper/lower arm, lo wer leg, hand, foot, nose/mouth portion of a face) and indicators of possible human presen ce (clothing pieces, webbing strips used by firefighters.) These objects were placed in va rying degrees of visibi litysome were lying in plain view in the rubble, others were partially buried under the rubble (Figure 5). Teams scored 3 points for each target object located during a run (0-12 possible points). Teams were not penalized for errors in identi fication (e.g., if they identified a mannequin piece as part of an arm when it was actually a lower leg), and received an extra .5 point for location of significan t target cues that were not hidde n in the searchspace as part of the task. Examples: finding 2 pennies, a pen. These are important because teams were instructed to note anything that might indicate human presence in the rubble. Identification of non-human related debris (b ig rocks, wood, rebar) was not scored. bserver ratings and codi ng frequencies obtained from the Robot-Assisted S and Rescue communication coding scheme (RASAR-CS) (Burke, Murphy, Coov al., 2004). Shared mental models are assesse d using a spatial map construction measure and coding frequencies from the RASA R-CS. The measures and RASAR-CS communication coding scheme used to assess each of these variables are described below.
F igure 5. A mannequin hand hidden in the search space serves as a visual cue (as seen through the robot operators OCU). Team Process Measures Team process variables are communication effectiveness, support/backup, adaptation of the team performance dimensions used in the TADMUS (Tactical Decision in the early 90s (Smith-Jentsch, Johnston, & Payne, ngs on the Team Process dimensions as shown in Table 2. leadership, and team situation awareness. These variables are measured using an Making Under Stress) program 1998). These are operationally defined as rati 44
45 Table 2 Team process dimensions Team Process Dimension Facet Communication Clarity of communications Degree of non-task related communications Monitor each other for overload and needed Overall support effectiveness Leadership Clear agreement on priorities Overall communication effectiveness Support Prompt correction of team errors assistance Guidance and feedback for decision making/problem solving Overall team leadership areness Used all ava ilable sources of information Passed information to right person without being asked Overall level of team SA Situation Aw Provided situation updates Team process ratings were made by an ons ite observer during each run, utilizing a 1-4 point Likert scale (low-hi gh). In addition to the separate dimension ratings, a global team process rating is formed from the mean of the four dimension ratings to be used as a dependent variable in part of the analysis pr ocess. Other indicators of team process are drawn from the RASAR-CS th rough analysis of communication dyad, form, content and function. Shared Mental Model Measures The shared (team) situation model is m easured in two ways: through pertinent statement frequencies/proportions (SA indi cators) rendered through RASAR-CS coding, and through a spatial map create d by the team onsite duri ng the exercise showing the
46 Likert n ss: fficient labeling of details, a nd notations regarding start point and path aveled. RASAR-CS Team interactions are classified usin g the Robot-Assisted Search and Rescue Coding Scheme (RASAR-CS). The RASARCS (Table 3) draws on the FAAs Controller-to-Controller Communi cation and Coordination Taxonomy (C4T) (Peterson, Bailey, & Willems, 2001), which uses verbal information to assess team member interaction from communication exchanges in an air traffic control environment. Like the C4T, the RASAR-CS is domain-specific, cap turing not only the how and what of US&R human-robot team member situation awareness. RASAR-CS addresses the goals of ca pturing team process and situation awareness by coding each statement on four categories: 1) conversational dyad: speakerrecipient, 2) form: grammati cal structure of the communica tion, 3) content: topic of the communication, and 4) function: intent of the communication. Team processes are examined using the dyad, form, function and content categories to results of their search. Each team produced a hand-drawn map of the search as part of the task scenario. These maps are scored on sim ilarity/accuracy/coverage using a 1-4 scale (1=Low, 4=High.) Maps were scored by a subject matter expert familiar with the task scenario and with US&R search protoc ols. The SME was provided with a schematic of the search space containing ground truth as to the placement of the mannequi pieces in the voidspace. The following points we re considered in the rating proce indications of time, scale or compass directi on, use of 2D and 3D representation, level of detail and su tr s, but also the who, as well as observable indicators of team
47 e which team members are interacting and what they are communicating about. nt and function dicators of the fn awother elements as indicators of the highest level of situation awareness, planning, rojection and prounction ere ge Q-sort tech l. (2004). determin Team situation awareness is explored using elements of the conte categories. For exam ple, when statements are coded for content, certain elements serve as in irst two levels of situatio areness, perception and comprehension; p blem-solving (Figure 6). Elements in the content and f categories w nerated using a nique (Sachs, 2000), as reported in Burke et a Figure 6. SA rs in the RASAR-CS indicato
48 Table 3 Robot-Assisted Search and Rescue Coding Scheme (RASAR-CS) Category Elements Definitions Sender/Recipient 1. Operator-Tether Handler Operator: individual teleoperating the robot 2. Tether Handler-Operator Tether Handler: individual manipulating the tet and assisting operator with robot specialist 5. Tether Handler-Researcher 6. Researcher-Tether Handler 7. Operator-Other Other -individual interacting with the operator w is not a tether handler or researcher 9. Tether Handler-Other 10. Other-Tether Handler Statement Form 1. Question Request for information 3. Answer Response to a question or an instruction not a question, instruction or answer Content 1. Environment Characteristics, conditions or events in the search environment 3. Robot situatedness Robots location and spatial orientation in the environment; position 4. Information synthesis Connections between current observation and pri observations or knowledge 6. Navigation Direction of movement or route her 3. OperatorResearcher Researcher: in dividual acting as scientist or robot 4. Researcher-Operator ho 8. Other-Operator 2. Instruction Direction for task or activity 4. Comment General statement, initiated or responsive, that is 2. Robot state Robot functions, parts, errors, capabilities, etc. or 5. Victim Pertaining to a victim or possible victim 7. Search Strategy Search task plans, procedures or decisions 2. Plan Projecting future goals or steps to goals cise rming a previous statement or observation 6. Convey uncertainty Expressing doubt, disorientation, or loss of 7. Provide information Sharing information other than that described in ing members 8. Off task Unrelated or extraneous subject Function 1. Report Sharing observations about the robot, environment or victim 3. Seek information Asking for information from someone 4. Clarify Making a previous statement or observation more pre 5. Confirm Affi confidence in a state or observation report either in response to a question, or offer unsolicited information 8. Correct error Correcting errors made by self or team
49 nvolve ns between individuals. Three dyad codes classify statements made by the operato er d in the nd be a g scribes the topic of co mmunication, and consists of eight elements: 1) statements related to robot arts, erro rs, or capabilities (robot tate ), 2 Speaker-recipient dyad codes were developed based upon the anticipated roles/individuals present in a US&R envir onment (Table 3). The primary dyads i the operator and tether-handler (the pers on manipulating the robots tether during teleoperation), operator/tether-handler and researcher, or operator/tether-handler and another person not involved in the scenario. Ten dyads were constructed to describe conversatio r to another person: operator-tether handler, operator-researcher, or operatorother Similarly, three codes classify statements made by the tether-h andler to others: tether-handler operator, tether-handler -researche r, tether-handler -other The remaining four classify statements received by the operator or te ther-handler from another person: researcher-operator, other-operator researchertether-handler, oth tether-handler In this study there were only 55 st atements (approximately 0.5%) that included the category element other; therefor e these statements were not include analysis. Verbalizations betw een individuals which did not include the team members were not coded (e.g., communi cations between members of the research team). The form category describes the grammatical structure of the communication, a contains the elements: question, instruction, comment or answer (A statement can whole sentence, or a meaningful phrase or sentence fragment.) Statements not matchin these categories are classified as undetermined. The content category de functions, p s ) statements surrounding the robots location, spatial orientation in the environment, or position ( robot situatedness ), 3) statements describing characteristics,
50 tions and prior observations or knowledge hesis ), 5) statements concerning the victim ( victim ), 6) indicators of directio ans, ask and is om recise or offering unsolicited information ( provide information ), and 8) correcting errors m the data collection process is explained. This is followed by a detaile d description of the conditions or events in the search environment ( environment ), 4) statements reflecting associations between current observa ( information synt n of movement or route ( navigation ), 7) statements reflecting search task pl procedures or decisions ( search strategy), and finally 8) statemen ts unrelated to the t ( off task ). The function category is used to classify the purpose of the communication comprised of eight elements: 1) sharing obs ervations about the r obot or environment ( report ), 2) projecting future goa ls or steps to goals ( plan ), 3) asking for information fr someone ( seek information ), 4) making a previous statement or observation more p ( clarify ), 5) affirming a previous statement or observation ( confirm ), 6) expressing doubt, disorientation, or loss of conf idence in a state or observation ( voice uncertainty ), 7) sharing information other than that described in report either in response to a question, ade by self or other team members (correct error) The function elements of report and provide information merit explanation, as they appear very similar. Report involves perception and comprehension of th e robot state, robot situatedness, the environment, information synthesis, or th e victim. Any other information shared by a team member, in answer to a question or on his or her own, is classified as provide information (e.g., navigation). Procedure This section describes the procedures used in data collection and analysis. First,
51 ating, and Data C e. report to one of the two search exercis ed a ot f e e data analysis process, incl uding data editing and prepara tion, data coding and r application of sta tistical procedures. ollection Prior to performing the search task exercises, participants completed a pretraining survey and received 1.5 hours basi c robot awareness tr aining. This training included an opportunity to observe and ope rate the robots in open, visible spac Participants were organized into 2-man teams for the search task exercises. Some participants chose their own teammate, but mo st were assigned by the researcher based on where they were sitting during the traini ng. Each team participated in 2 20-minute confined space search task scenarios ove r a 2day period. Teams were assigned a treatment condition by the experimenter and a time to e locations on the pile. (Other traini ng activities were going on during the data collection process, so participants were involved in other events throughout the 2-day period.) Human-robot teams (2 people: 1 robot) were videotaped as they perform technical search task to capture how the robo t operator and tether-h andler used the rob to search for signs of victims. The method of data collection was a modified version o the procedure used in (Burke, Murphy, Coove rt et al., 2004). A team of 3 researchers (Experimenter, Videographer, Team Process Ra ter) used 2 Sony digital videocameras to record the teams as they performed each of the 2 search task scenarios (Figure 7). On camera was attached to the robots Operator Control Unit to record the view through the robots camera. This camera was also used by the tether-handler as the RSVP in 2 of th
52 experimental conditions. A second cam era was held by the Videographer to simultaneously capture a view of th e operator and the tether-handler. The Experimenter explained the task scenario to each team, answered any questions about robot capabil ities/operation (but NOT about search strategy, navigation, victim location or other mission related inform ation) and made sure that conditions were administered correctly according to th e data condition assignment sheet. The Experimenter also recorded critical even ts (such as finding a mannequin piece) during each run. The Videographer filmed each team as they ran the scenario. The Onsite Rater observed and rated each team using the team process rating measure described above, and collected the maps generated by each team All three experimenters monitored each others tasks to make sure controls were not violated. In the first run, the participants self-organ ized to use the robot, i.e. they decided r ndix A for the specific instructions). During participants switched roles, and again were given standardized instruct ng id not complete the feedback surveys onsite. These participants receive ed a 4 who would fill each role, and were given standa rdized instructions to search the void fo victims or signs of human presence (see Appe the second run, ions regardi ng the search task. Upon completi on of the second task scenario, participants returned to the classroom a nd completed a post-training feedback survey containing individual ratings of effectiveness, usefulness, eas e of use, satisfaction, alo with self ratings on the team process variables described in the Measures section. In a few cases, participants d d a followup letter and survey to co mplete and return by mail. Complete survey data was obtained for each team member incl uded in this study. The 50 runs yield total of 16 hours, 40 minutes of videotape for analysis.
53 Figure 7. A 3-person team of researchers manned each task scenario site. (From left to right: Videographer g explain prepares to film a team; tether-handler looks downward into voidspace; Onsite Rater readies her ratinsheet; robot operator is obscured behind Onsite Rater; Experimenter oversees preparations; onlookersobserve prior to the next run.) Data Analysis Data analysis provides a way of organizing the data for the study. This section s the three steps of the data analysis process: data editing and preparation; data coding and rating; and application of statistical procedures. The section on statistical analysis includes a detailed description of the multilevel regression modeling technique applied to the data. Data editing and preparation. The data analysis process began with data editing and preparation, which time synchronized the robots camera videotape with the matching operator videotape to produce a side-by-side video recording of the robot and
54 gs l ed de scriptions of the disaster rill and data collection pro cedures, and then reviewed de finitions for all the codes. al examples selected from other data sets (video recordings) were reviewed, and enhance reliability. The m en t at in A fixed number of statements was es tablished (9,954 statem ents across the 25 teams) to be coded. Raters coded each statement across four categories: dyad (speakerrecipient pair), form (grammatical structure of the communication), content (topic) and function (intent of the communication). Ten of the 50 team observations (20%) were coded by all four raters for reliability (n = 1,486 statements). Both raw agreement indices and int 4. t, it the operator manipulating the robot. The di scourse between team members during the search task scenario was transcribed to produce a fixed number of meaningful statements for coding. Editing and transcription took approximately 6 man-months. These recordin and transcriptions were then used to code statements made by team members with the Observer Video-Pro (Noldus, Trienes, Hendr iksen, Jansen, & Jansen, 2000) behaviora analysis software. Data coding. Data coding is the next step in the process: raters were trained in using the coding system, a fixed number of st atements to be coded was decided upon, and the actual coding required approximately 280 man-hours. Coders we re three graduate students (drawn from the psychology, business, and computer science departments) and one undergraduate psychology student who were trained by the author to code the videotapes. During 10 hours of code r training, raters review d Behavior coding guidelines were devel oped to reduce ambiguity and to ajority of the training c tered on coding s ements together and reach g consensus. errater reliability estimates are reporte d for the four coding categories in Table While Cohens Kappa (1960) is the most frequen tly used coefficient of rater agreemen
55 hich ter ices and/or Brennan and Predigers Kappan (1981) ng the tent ate to is not so much a measure of raw agreement as it is of improvement in agreement beyond chance. Cohens Kappa is based upon a main effects model of rater independence, w only takes into account the differences in fre quencies with which raters used different rating categories. Brennan and Predigers Kappa n is a variant of K based upon the null model, which solves the problem created when raters use categorie s at different rates (maximum agreement is reduced when this oc curs). The most current research on ra analysis suggests reporting raw agreement i nd in addition to the more traditional Cohens K in the interests of best capturi magnitude of agreement (Von Eye & Mun, 2005). The lower estimates for the con and function categories of the RASAR-CS refl ect the difficulty in orthogonal coding of dimensions with multiple elements. However, ratings in all categories exhibit moder excellent agreement (Flei ss, 1981; Landis & Koch, 1977). Table 4 Raw agreement, K n, and K for the four RASAR-CS categories RASAR-CS Category Raw Agreement Brennan & Predigers K n Cohens K Dyad .84 .80 .71 Form .71 .61 .56 Content .52 .46 .43 Function .44 .42 .36 Statistical Analysis: Multilevel Regression Finally, statistica l procedures are applied to the data. In additi on to standard descriptive a nd correlational analyses, this
56 teams across levels of experimental conditions, and the nature of the data, which were co s le here in schools; in cross-cultural studies, where individuals are nested e study uses multilevel regression analyses to ex amine the effects of location and RSVP on team mental models, team processes and t eam performance. Multilevel analysis is appropriate for this data for two reasons: the complex nesting structure of repeated trials within llected at both individual and team levels. To clarify how multilevel regression i applied here, a brief overview of multilevel modeling is presented, followed by an example using variables in the current study. Multilevel modeling is a type of statistical analysis designed to work with hierarchical or cross-classified data. Psychological res earch often asks questions involving the relationship between indivi duals and groups, based upon the assumption that individual persons are influenced by the properties of the groups to which they belong. Similarly, groups are thought to be influenced or shaped by the individuals that comprise the group. This creates a hierarchical system with individuals and groups at different levels, and variables describing each at these levels. Observations from individuals examined within a group c ontext often violate the assumption of independence made in traditional statistical models. Multilevel analysis draws samp data from each level of a hierarchical populati on, and uses variables at higher levels to adjust the regression of lower level depe ndent variables on lowe r level explanatory variables (Hox, 2002). Examples of multilevel re search can be found in education, w students might be nested with within units by nationality or ethnicity; and in business, where individuals are nested within departments in organizations. Mu ltilevel analysis can be applied to less obvious data structures, e.g. re peated measures studies, wh ere events/observations ar
57 nt nts, or perimental conditions, and sites. ysis ple vel also al levels, to examine the direct effects me nested within individuals; or meta-analyses, where individuals are ne sted within differe studies altogether. In this st udy, observations are nested within pairs of participa teams. These observations/teams are in turn nested within ex Multilevel analysis offers statistical a nd conceptual advantages over traditional statistical analysis. Traditional statistical te sts tend to aggregate lower level variables data into a smaller number of hierarchical units or conversely, to disaggregate the higher level unit into data on a lower level. In the first case, information is lost and the anal loses power. In the latter, the smaller number of grouping units is expanded into multi values for a larger quantity of individual units, which are then treated as independent information from that population of units. Th is can result in mi sleadingly significant outcomes. Conceptually, analyzing hierarchical or nested data all at one level makes it easy to commit the ecological fallacy of assumi ng inferences made at the group le hold at the individual level (Robinson, 1950); or to incorrec tly form inferences about higher level constructs based upon lower level data, known as the atomistic fallacy (Diez Roux, 2000). The ultimate goal of multilevel an alysis is to answer questions about relationships between variables at different hierarchic of individual and group level explanatory variables on an outcome of interest, and to see if group level variables moderate in dividual level relati onships by testing for interactions between levels. A multilevel regression model (aka random co efficient model, hierarchical linear model, variance component model) assumes a hi erarchical data set, with a single outco variable measured at the lowest level, and explanatory variables at all levels. In this
58 ata oint also entered at this level: Location (1=Distributed, 2=Collocated) rly, the four team process dimensions rcept, slope and error (residual) term. In Equation 1, the intercept 0j is the overall average performance by a team on a run (occasion). The slope 1xij is the regression coefficient study, multilevel regression is used to genera te separate models for performance, map score, and a team process composite m easure. The following example uses the performance outcome to illustrate how the models are built. This study used data from 25 teams with 2 observations (occasions) per 2-person team. So that we can look at both individual and team level data across observations, d were arranged so that occasions are nested within participants, giving a sample of 100 cases for analysis. Thus outcome variables are measured at the lowest hierarchical level. Outcome variables were performance score (0-12 points), map score (1-4 pts.), and global team process rating (1-4). Explanator y variables at level 1 include firefighting experience, US&R experience, and technology experience, each measured on a 5-p Likert scale (low/high). Becau se each team completed runs in two different conditions, experimental IVs are and RSVP (1=RSVP, 2=no RSVP). Simila (Communication Effectiveness, Support, Leader ship, and SA) are entered at level 1, each measured on a 4-point Likert scale. Explanator y variables entered at level 2 include site (1=NASA-Ames, 2 = NJTF-1), and order of conditions (1-11 listing various orders in which teams went through experimental conditions). To analyze the data, separate regression equations are created for each occasion to predict the outcome variable y by the explanatory variable x y ij = 0j + 1 x ij + e ij (Equation 1) The equations form is that of a traditional regressi on equation, consisting of an inte
59 r l 2 s n m quantities with means = 0 and a normal distribution, so that varianc el is r, it is consi (slope) for the explanatory variable x The intercept and slope make up the fixed part of the equation. The error e ij is the part of performance not predicted by the fixed regression part of the relationship, where i is the occasion and j is the team. The error term has a mean of 0 and a variance to be estimated. In a multilevel analysis, the level 2 groups, in this case teams, are regarded as a random sample from a population of teams, thereby leading to the expression of the intercept 0j as 0 + u 0j With multiple teams, the erro term u 0j is the deviation of the j th teams intercept from the ove rall value, and is a leve residual which is considered to be the same for all occasions in group j The residual i now partitioned into a level 1 component e ij and a level 2 component, u 0j corresponding to each level in the hierarchy. The variance between level 2 groups is 2 u and the variance between occasions within a given level 1 group is 2 e The regression model ca thus be expressed as y ij = 0j + 1 x ij + u 0j + e ij (Equation 2) where u j and e ij are rando es 2 u and 2 e can be estimated. These variances are assumed to be uncorrelated since they are at different levels, and are known as the random parameters of the model. The intercept and slope (fixed parameters) can also be estimated. This multilevel mod sometimes called a variance components model. The model can be expanded to include multiple explanatory variables at any level ( both categorical and continuous); howeve dered prudent to begin with an intercept-only model and build the model by adding theoretically important variables. The model is estimated using the iterative
60 ntil n del fit is evaluated using the deviance statistic, defined as 2*log ( ributed gure s the ordercode variab le. This is a dummy code to assess whether the order in which teams participated in experimental conditions had any effects on the outcome. Looking at the upper model in Figure 8, the intercept 0j (4.350/0.312 = 13.94) and level 1 error variance eij (8.615/1.723 = 5) are both significant, the level 2 error variance u0j is not (0.546/1.298 = 0.42), and the baseline model deviance is 505.121. In generalized least-s quares method (IGLS) 1 where the model is computed repeatedly u a specified level of convergen ce is reached (that is, the m odel is no longer changing from iteration to iteration.) As seen in the exam ple that follows, the model provides standard errors for the intercepts, regression coefficien ts and variance estimates. These are used i assessing the significance of coefficients using the Wald statistic, which is the ratio of a coefficient to its standard error (Wald, 1943). It is tested as a Z-value compared to a standard normal distribution at a given p-level ( p =.05 is the level of significance used throughout this study). Mo Likelihood) where Likelihood is the va lue of the function at convergence and log is the natural logarithm. Many models can fit a data set, but a smaller deviance typically corresponds to a better fit of the data. When models are nested, the change in deviance can be tested for significance using a chi-squa re test, where the difference is dist as a chi-square with degrees of freedom corresponding to the ch ange in number of parameters of the model. In this model, performance is assumed to be normally distributed, and is notated as y ~ N (XB, ) where XB is the fixed part of the model and is the random part. Fi 8 shows the baseline, inter cept-only model, followed by th e more general model that include 1 IGLS is the default procedure used in MLwiN 2.0 (Rasbash, Steele, Browne, & Prosser, 2005), the statistical software used to estimate model parameters in this study.
61 the lower model depicted in Figure 8, the e xplanatory variable o rdercode has been added to the equation. It is not significant (-0.012/0.101= 0.12) signaling that the order of experimental condition does not predict performance. Note that though the coefficients for the intercept and level 2 error variance u0j vary slightly, their significance level has not changed. The model deviance does not appear to have significantly changed ( 2 = 0.13, ns .) Therefore ordercode does not appear to contribute to the ove rall prediction of performance in this study. However, it is incl uded in all analyses to control for possible effects on other variables. Multilevel regression allows the testing of variables at both individual and team vels within the same analysis. Is it mo re complicated than traditional regression? n be. The results in terms of the fixed regression estimates are very close to those obtained through traditional multiple regression, and offer a more conservative estimate of the standard errors of these parameters. Though the effect size estimates may be similar, multilevel modeling can reveal differences in variance ac ross units of analysis at different levels, making it a useful tool in situations such as this study, where th ere are not only data nested within teams, but explanatory variab les present at both the individual and team level. le Depending on the number of parameters and inte ractions tested, it ca
Figure 8. Baseline and ordercode model estimates of performance. 62
63 nd major it dings from this study are clearer when presented according to the question (performance, shared mental model, team process), so the finding a lot of significant results from supplemental analyses to be Chapter 5 Results This chapter presents descriptive analyses for participant demographics a study variables, and findings from the communication analysis conducted using the RASAR-CS, followed by the results for the multilevel regression analyses of team performance, shared mental models, and team process. Typically results are presented according to hypotheses, beginning with Hypothesis 1, etc. However, upon reflection seems that fin dependent variable in s are reported in that fashion. However, all findings are clearly linked to the hypotheses in question, so you will have no trouble connecting them. In addition, a summary chart displaying all hypotheses and evidence of support can be found at the close of this chapter. As you read, you will see looking at variables other than location and RSVP, many of whic h are not directly tied the hypotheses. These are not fishing expeditions, but are desi gned to explore the relationships between various constructs in the model described in Figure 3. All will made clear in the end. Findings are re ported as statistica lly significant at p < 0.05 unless noted otherwise.
64 Descriptive Analyses arizes the results from demographic survey questionnaire given ants. The majority were male 90%) between the ag es of 35-54 (76%). ting, US&R, and chnology exi ence were included in the muvel nalyses as potial moderators of performe in human-robot teams. very few participnts reported having any miperience (lthan 20%), a posnat va riable for the purposes of this analysis. self-ratings wre av eraged across questions 8-11, which can be found in Analyses section repor findings froh e communication analysis conducted using S. The hyptheses stated the en d of Chapter 3 include results from the R-CS as a seconda metric of support. In the interests of clarity, the findings n ar the es in conjunction with tultilevel analyses that flow. encies anpercentages m the RASAR-CS are reported in Table 6. am member exchanges, and tether handler s initiating 46%. Over half of all coded stateme Demographics Table 5 summ the to particip s ( Firefigh te per ltile regression a te n anc Because a lit ary ex ess this was dr opped as s ible expla ory Technology e Appendix B. RASAR-CS This ts m t the RASAR-C o at RASA ry reported in this sectio e linked to hypothes h e m regressi on ol Statemen t frequ d fro Communications were balanced between team members, with operato rs initiating 47% of te nts were comments (53%), with th e remainder divided equally among the other categories (answer, question, instruction).
65 Means/ Table 5 proportions, standard deviations, and percentages for demographic survey data Survey Question M ( SD ) Total % NASA-Ames % NJTF-1 % Age (5 groups) 3.44 (0.95) 1. under 25 4 4 5 3. 35-44 40 39 40 4. 45-54 36 32 41 5. 55 and up 12 14 9 Gender -1. Male 90 93 86 2. Female 10 7 14 Firefighting Experience 3.42 (1.81) 1. 0-3 years 30 40 18 2. 4-7 years 4 0 9 3. 8-11 years 4 7 0 5. over 15 years 52 46 59 1. 0-3 years 36 40 32 3. 8-11 years 16 21 2. 25-34 8 11 5 4. 12-15 years 10 7 14 US&R Experience 2.16 (1.21) 2. 4-7 years 34 14 60 8 4. 12-15 years 6 11 0 over 1 Military Experience -55 years 8 14 0 1. Yes 18 25 14 2. No 82 75 86 Technology Experience 2.9 a (0.88) 1. Very Little 8 10 4 2. Some 18 18 18 3. Moderate 52 43 64 4. Above Average 20 25 14 5. Expert 2 4 0 Note Total N = 50. NASA-Ames n = 28, NJTF-1 n = 22. a A technology experience composite rating was created based upon respondents self ratings of experience In terms of content, teams communicated most often about the robots state (30%) and navigation (22%), with the environmen t, search strategy, and robot situatedness categories each claiming another 10% of th e total. In the function category, team communications were focused on reporti ng, planning, and confirming (29%, 23%, a with remote-controlled vehicles, video games, video cameras, technical search technology, and robots. nd
66 pectively). munication between team members, s x content co ere generat operatortether and tether-operator dyads (Table 7) The upper portion of the table categorizes operator-tether utterances by form and content: for example, operators made 8 fied as answers to teth dlers. This re presents 17% of all operator-tether statements in terms of form ose 807 answers,re classified as being about the environment. Looking acro table, you can sestatements about the environment made up 15% of all opera er statements in of content (n=704 statements). Proportions within the re easily calculaividing the srequency by the appropriate cat half of the table crances made by tether-handleerators. and tether-handlers made r kinds of statem verall in terms of content. The one content category that sta nds out is navigation. Operator statements regarding navigation totaled 18% of all st ats (n = 852), and t-handler sabout navigation totaled 29% (N = 1,333). This means that tether-handlers talked to the operators 60% more often abo igation than the ors talked to them a ting when you consi t the operator s are the ones doing the navigating. Turning to the form category, t s another striking difference between t tether-handlers gave over 4s as many instru ctions to the operators (N lear that the majority of those (54%) were instructions about navigati on (N = 651). Referring back to Table 6, instructions made up 15% of all statements coded in the study (N = 1,506). 18%, res To gain some insight into the patterns of com peaker-recipient dyad x form mp arisons w ed for the 07statements classi er-han Of th 100 we ss th e e that to r-teth terms table a ted by d tatement f e gory total. The lower lassifies utte rs to op Operators simila ents o temen ether tatements ut nav perato bout it. This is interes d er tha h ere i eam members: time = 1,203) as operators gave to them (N = 253). L ooking within the table, it is c
67 gory Frequency Percent Table 6 RASAR-CS statement frequencies and percentages Coding Dimension/Cate Dyad Oper ator-Tether 4629 47% er-Operator cher-Operat esearch 9 earcher 8 1% er-Tether 2 54 T eth 4543 253 46% Resear o r 3% Operator-R s er 20 17 2% Tether-Re Research 14 1% 99 100% Form Comment 52 51 4 r 83 6 53% Question 161 16% Answe 1 5 16% Instruction 150 15% 9954 100% Content Robot state Navi 3001 30% gation 2203 22% ent 1237 strategy tedness 2 7 7% 6 n synthe 54 Environm 12% Search 1013 11% Robot situa 98 73 10% Victim Off task 60 6% Informatio sis 175 2% 99 100% Function Report 289 3 5 tio 4% 333 3% Correct error 132 1% 9954 29% Plan 2 25 23% Confirm 1818 18% Provide informa n 1053 11% Seek information 1040 11% Clarify 430 Convey uncertainty 100% Total number of statements = 9,954 for 50 occasions. Categori es are ordered by frequency. M = 199, SD = 69.
68 speaker-recipient dyads Table 7 RASAR-CS form x content comparisons of statement frequencies and proportions for Operator-Tether Content tegory tal Proportion of total Environment 100 447 147 704 0.15 Dyad Form Content Answer Comment Instruction Question ca to 10 Information Synthesis 19 65 1 0 5 tion 148 527 83 94 852 0.18 20 83 1 20 124 0.03 ss 66 2 18 93 449 0.10 276 8 100 203 1412 0.31 106 3 32 59 512 0.11 72 3 8 74 481 0.10 807 2 253 700 4629 1.00 1 9 0.02 Naviga Off tas k Robot Situatedne 72 Robot State 33 Search 15 Victim 27 Form category total 869 Proportion of total 0.17 0 0.05 0.15 1.00 perator Answer Cent Instr Questio Conten categor total Proportion of total nvironment 95 302 13 118 528 0.12 .62 Tether-O Dyad Form Content omm uction n t y E Informati on 10 54 5 6 75 0.02 139 4 651 139 1333 0.29 9 7 1 23 103 0.02 uatedness 88 3 24 59 524 0.12 191 4 390 225 1290 0.28 62 1 111 92 438 0.10 1 8 69 252 0.06 tegory total 624 1 1203731 4543 1.00 Synthesis Navigation 04 Off task 0 Robot Sit 53 Robot State 84 Search 73 Victim 30 45 Form ca 985 P roport ion of total 0.14 0 0.26 0.16 1.00 .44 Total oper ator statements N = 4,629. Tota l tether-handlatements N = 4,543. tatements, 80% were made b er-handlers to operatohe tetherinsert the ronto th e accessible voidspace and manipulate the tether teleoperates thet through the void. While it has be en shown that the er st Of th ose s y te th rs. T handlers role is to bot i as the operator robo
69 backse e at driver phenomenon is common in human-robot teams (Burke, Murphy, Coovert et al., 2004), this is an unusually sk ewed ratio. Could it be due to one of th experimental conditions in the study? Table 8 RASAR-CS statement proportions by location and RSVP conditions Location RSVP Coding Dimension/Category Distributed Collocated RSVP No RSVP Dyad Operator-Tether 0.47 0.47 0.44 0.50 Tether-Operator 0.44 0.45 0.48 0.40 Operator-Researcher 0.03 0.03 0.03 0.03 Researcher-Operator 0.03 0.02 0.02 0.03 Tether-Researcher 0.02 0.02 0.02 0.02 Researcher-Tether 0.02 0.01 0.01 0.02 Form Answer 0.15 0.16 0.15 0.17 0.53 0.51 0.56 Instruction 0.14 0.16 0.18 0.10 Question Environment 0.13 0.11 0.12 0.12 Navigation 0.21 0.23 0.23 0.21 Robot situatedness 0.12 0.09 0.09 0.12 Search 0.09 0.10 0.10 0.09 Comment 0.53 0.18 0.16 0.17 0.17 Content Information synthesis 0.01 0.01 0.02 0.01 Off task 0.08 0.06 0.06 0.07 Robot state 0.30 0.31 0.31 0.29 Victim 0.06 0.09 0.07 0.08 Function Clarify 0.04 0.04 0.04 0.05 Confirm 0.18 0.18 0.18 0.19 Convey uncertainty 0.03 0.03 0.03 0.04 Correct Error 0.01 0.01 0.01 0.01 Plan 0.21 0.22 0.24 0.18 Provide information 0.10 0.10 0.10 0.10 Report 0.30 0.31 0.29 0.32 Seek information 0.12 0.11 0.11 0.11
70 ntal ted teams. Collocated teams gave sligh tly more instructions, and talked less about robot situatedness. Looking at the RS performnce (r = 0.36, p = 0.01). Situation awareness was related to map score ( r = 0.33, Taking a quick look at the RASAR-CS proportions according to experime conditions in Table 8, there were few notable differences between distributed and colloca VP teams and no-RSVP teams, there are several differences that stand out. Tether-handlers ta lked to operators more often when they shared the robot view (48% vs. 40%), and gave more instructions (18% vs. 10%). Like the collocated teams, RSVP teams talked less about robot situatedness (9% vs. 12%). They did slightly less reporting ( 29% vs. 32%), and engaged in more planning compared to no-RSVP teams (24% vs. 18%). These observations cannot be taken at facevalue since the two conditions were crossed and nested within teams. However, they serve as interesting indicators of relationships to look for in later analyses. Descriptive Analyses for Study Variables Means, standard deviations, and correla tions between team performance scores, map scores, team process dimension ratings and selected RASAR-CS categories are reported in Table 9. The incorporation of a ll 26 category element variables used in the RASAR-CS makes for an ungainly correlation tabl e, so only those elements that played a role in the reported results ar e included. The complete correlat ion matrix is available in Appendix C. The overall mean performance score was rather low ( M = 4.35, SD = 3.06), with scores ranging from 0-12.5. Mean map score was 2.54 ( SD = 0.96), with scores ranging from 1-4. Team process means hovered around 3 on a 4-point Li kert scale. There was no correlation between map score (shared mental model measure) and performance score. Of the four team process dime nsions, only Communication was related to a
71 p = 0.02), underscoring the importance of SA in creating shared mental models. Correlations between team process dimensions were all significant, ranging fr1 0.64. llocated teams gave slightly more in structions, and talk ed less about robot situatedness. Looking at the RSVP teams and no-RSVP teams, there are several differences that stand out. Tether-handlers ta lked to operators more often when they shared the robot view (48% vs. 40%), and gave monstrsv the collocated teams, RSVP teams talked less about robot situat edness (9% vs. 12%). They did slightly less reporting (29% vs. 32 ore planning compared to no-RSVP teams (24% vs. 18%). Tho tn t since the two conditions were crossed n we ra interesting indicators of relati onships to look for in later analyses. In the RASAR-CS cate tt le dimons ( r = 0.29-0.4e i wr significant ( r = 0.27, p = 0.06). The tether-operator dyad wnegly associated wi SA vi t apphed sig r i tethr eer tings on these team process dimonAhrotif inc s wlso ted oor a evidenced by the negative relationship ( r = -0.29). Statements a bout navigation were not assoedth, t wai cnege rensh ip score ( ) crae e wersitiv rel t s om .03 to Co re i uct ion (18 % s. 1 0% ). Lik e %), and engaged in m bse ese rva ion s ca no t be ta k en a fa ceva lue and est ed ith in t am s. H ow eve th ey se rve s gor ensi ( r = roac er-h ensi ciat 8, p ies, t he op era tortethe r dya d w as s ig nif ican ly rela e d to a l fo ur t am pr oc ess 7), and th e r lati on shi p w th map sco re as ma gin all y as he ative aders th -0 .4 2, p < 0. 01 ), a nd a ne gati e r ela tio nsh p w ith Le hip dim en sio n nif ica nc e a s w el l ( r = 0.2 6, p = 0.0 7). Mo e c omm un icat ons fr om the an dle to th op ato r are associated with poor ra s. hig er p por on o stru tion as a rela to p SA, s wi .01 SA In but ont here st, s as arch signif statem ant nts ativ e po latio ely i p w ated th ma o map r = 0.3 < 0 core
72 Table 9 for study variables of interest (N=50)SD 2 3 6 10 12 13 14 15 Means, standard deviations, and correlations Variable M 1 4 5 7 8 9 11 1. 3.06 Performance 4.35 1 2. MapScore 0.96 0.138 1 0.339 3. Communicati 0.77 0.359* 0.101 0.010 0.485 4. 3.02 0.71 -0.041 0.779 0.444 5. 3.08 0.64 0. 043 0.152 0.769 0.291 6. Situation 3.08 0.63 0.054 0.707 0.021 ther 0.47 0. 13 0.049 0.268 0.059 8. Tether-Operator 0.45 0. 13 0.025 0.217 9. Instruction 0.15 0.11 0.230 -0 0.131 10. Navigation 0.22 0.10 -0.045 0.007 11. Robot Situatedness 0.11 0.05 -0.257 0. 113 0.435 0.30 0.09 -0.110 0.043 0.769 13. 0.02 1 310* 1 0.642*1 0 0.394* 0.465*0.479** 0.001 0 0.351* 0.287* 0.382* 0.467*1 0.013 0.043 0.001 -0.189 -0.257 0.417** 0.900** 0.195 0.188 0.003 0 -0.098 -0.186 0.671** 0.733*1 0.681 0.497 0.042 0 -0.023 0.089 -0.044 -0.229 0.471** 0.528*0.544** 0.872 0.538 0.764 0.11 0 0 0.028 0.112 0.165 -0.355* 0.449** -0.296* 1 0.847 0.439 0.011 0.001 0.037 0.061 -0.004 0 -0.303* -0.052 0.75 0.676 0. 361 0.839 0.976 1 0.718 024 0.103 0.022 -0.191 0.229 0.477 0.878 0. 2.54 on 3.18 Support 0.111 0. 0.029 Leadership 0.353* 0.012 Awareness 0.327* * 1 0.005 7. Operator-Te 0.734 0. 006 -0.178 -0.186 1 0.861 0. 071 .217 0.06 -0.289* 0.108 0. 195 0 0.379* 1 0.754 0.001 0 -0.052 0.072 0.719 1 0.252 12. Robot State -0.002 -0.046 -0.132 -0 .03 1 0.447 0.99 0.033 Search 0.10 0.05 0.209 0.429** -0.052 0.093 0.0.173 0.064 -0.24 0.348* 1 0.145 00.72 0. 52 0.87 661 0.093 0.185 0.013 14. Report 0.30 0.11 -0.018 -0.108 0.281* 0.001 0.107 0.102 0.221 -0.27 0.414** -0.151 0.254 0.239 0.550** 1 0.900 0.454 0.048 0. 992 0.459 0.479 0.124 0.058 0.003 0.297 0.075 0.094 0 15. Plan 0.21 0.11 0.308* 0.004 0.141 -0.028 -0.116 -0.072 -0 .346* 0.494** 0.810** 0.375** 0.500** -0.198 0.366** 0.644** 1 0.030 0.981 0.327 0.847 0.423 0.621 0.014 0 0 0.007 0 0.168 0.009 0 *p = 0.05, **p = 0.01.
73 arch strategy than navigation were able to creat t least, as measured by the robot state a nd robot situatedness were unrelated to the outcom being less nts in rior r is ( r = 0.43, p < 0.01), suggesting that teams that talk ed more about se e a richer shared mental model (a map score). The categories of e variables, though the negative re lationship between robot situatedness and performance was nearly significant ( r = -0.26, p = 0.07). However, they were each negatively associated with navigation (r = -0.30 for both). This is partly due to their in the same coding dimension, but statements in one content categ ory do not preclude statements in another. It may be that teams that talked more about navigation talked about the robot in general, focusing on direc tion instead. The proporti on of stateme the report category was positively related to Communication ratings ( r = 0.28). Teams that had a higher proportion of statements reporting what they were seeing in the search space were rated has having more effec tive communication. Finally, the function category plan was associated with performance (r = 0.31), signaling that teams that engaged in more planning were more successful in locating victims. Some interesting findings have emerged fr om these analyses. Some confirm p research findings: 1) planning and effective communications are related to performance; and 2) search strategy and situation awarene ss are important in creating a shared mental model of the search. Other findings are qui te novel: something is going on with the tether-handler giving all those instructi ons about navigation! Having communications from the tether-handler detract from team SA is unexpecte d, since prior research has shown that more frequent communication be tween the operator and tether-handle associated with better SA and more eff ective performance (Burke & Murphy, 2004a). However, before speculating as to why this oc curred, there are further analyses to report
74 ough ubsequent equations to control for order effects in other conditions. Thus, Model B e baseline model for Models C-G. There were no effects for individual cient htncp ignificance. No e location (Model D) on performance. This m odel contradicts Hypothesis 6, which stated that collocas would perform distributed teams. More positive results were found in the next moodel te hypothesized diffein perfoe between teams with and withut RSV. T ative regression cot for R = -1.322/ SE 0.611) signifies a positive rel n RSVP rforman (recling that the RSVP variable is coded model is significantly reduced as well ( 2 = 4.52). The next step in the analysis was to test for site effects, i.e. to look for differences s accordhere t weollecSA-Am, NJ-1Unfortunately, I fo: Model Fs drop in Deviance and the significant Beta or the site variab 2.103/ S 0.5 left ukable e ce offe performance by site. (This is unfotunate in the sense that the variance rformance was effecficantl other than the independent variable s in question.) However, after re-entering which may shed some light on the findings from the RASAR-CS. Multilevel Regression Analyses Team Performance Measure Table 10 displays model parameters for the regression of location and RSVP on team performance. No effects were found for the order in which teams ran thr experimental conditions, as seen in Model B. I chose to keep the Order variable in s serves as th experience of any type (firefighting, US&R, or technology), though the regression coeffi for firefig ing experie e ap roach ed s ffects were found for ted team bette r than del. M E sts for rences rmanc o P he neg efficien SVP ( ations hip betwee and pe ce al 1=RSVP, 2 = no RSVP). The deviance in the between the team ing to w he data re c ted (NA es = 1 TF = 2 ). und them weight f le ( = E = 73) nmista vid en dif rences in r in pe ted signi y by something
75 the RSV e he the results Regression Coefficients P variable into the equation along with order and si te and using Model F as th baseline for comparison, the regression coeffi cient for RSVP remained significant and t model deviance was significantly reduced ( 2 = 5.67). Despite the fact that there were site effects, the use of RSVP does have a positive effect on perfor mance. Therefore reported in Table 10 provide support for Hypothesis 3 (RSVP teams better performance), but not for H ypothesis 6 (collocated teams better performance). Table 10 Multilevel regression analysis of location and RSVP effects on team performance M Deviance Deviance df Variables SE odel A. 505.12 --Null 5 -0.012 0.101 C500.98 4.14 3 rder hting Experience Experience echnology Experience 0.003 0.322 -0.245 0.159 0.100 0.170 0.256 0.345 D5 ion 3 8 0 E. 500.59 4.52* 1 51 2* 5 1 F. 492.62 12.49* 1 3 07* 3 3 G 490.15 2.47 1 rder ion -0.081 6* 0.096 7 1 H4 8* 4 8 6 B. 05.11 0.01 1 Order O Firefig US&R T 02.69 2.42 1 Order Locat -0.03 0.984 0.10 0.61 Order RSVP -0.0 -1.32 0.10 0.61 Order Site -0.03 -2.1 0.09 0.57 O Site Locat 2.12 0.935 0.56 0.59 86.95 5.67* 1 Order Site RSVP 0.034 2.16 -1.416* 0.09 0.55 0.58 p < 0.05.
76 d process ratings to predict team performance. Though not ere entered d l Supplemental regression analyses. In addition to testing the main hypotheses relating to location and RSVP, I examined th e relationships between team process an performance by using the team specifically stated as a hypot hesis, this relationship is both part of Klimoski & Mohammeds team performance framework (Fi gure 2) and part of the model of robotassisted team performance (Figure 3) proposed in this study. Sin ce no particular team process dimension was theorized to be more im portant than another, all four w simultaneously into the regression equation. Ta ble 11 lists the regression coefficients an standard errors for the four team process dimensions, along with comparisons of mode fit. Table 11 Supplemental multilevel regression analyses for team performance Regression Coefficients Model Deviance Deviance df Variables SE A 505.12 -Null Order -0.033 0.093 Site Communication Support Leadership 1.860* 1.605* -0.827 0.849 0.546 0.340 0.533 0.559 Site Plan Robot situatedness Robot state 2.231* 7.123* -20.802* -5.198 0.489 2.848 5.698 2.798 B. 492.62 12.50* 2 Site -2.107* 0.573 C. 478.43 14.19* 4 Order Situation Awareness 0.077 -0.826 0.103 0.530 D. 457.78 34.85* 4 Order Tether-operator -0.154 -5.065* 0.084 2.174 p < 0.05.
77 sured tgun ASARy, ficant bles bot e, and tether-operator. The first thr ee are theoretically important accordi tion tion e s f performance. Talking more bout the robot (state or situ atedness in the environment), however, seems to detract from Communication appeared to have signif icant influence on performance. After controlling for order and site, deviance in the model was significantly reduced ( 2 = 14.19). This naturally leads to the question of whether team communication as mea by the RASAR-CS would show any predictive power for team performance. With 26 of the 30 possible coding categories used in this study, it made no sense to take the sho approach of trying all category elements as predictors. To narrow the field of possibilities, I first l ooked to the elements classified as indicators of SA in the R CS: environment, information synthesis, robo t state and situatedness, search strateg plan and report. I then reviewed zero correlation tables to see if ther e were any signi relationships between these RASAR elements (or any others) and primary study varia (both dependent and independent). I chose to lo ok at the following variables: plan, ro situatedness, robot stat ng to prior research, and were signifi cantly related to one or more major study variables. The tether-operator proportion expresses that part of the teams communica that is initiated by the teth er handler speaking to the r obot operator. This proportion displayed some interesting rela tionships with several study va riables (e.g., team situa awareness), and was chosen for that reason. When entered simultaneously into th baseline model controlling for order and site, three of the four RASAR proportion variables were significant pr edictors of perfor mance (Table 11). Model deviance wa significantly reduced from 492.62 to 457.78 ( 2 = 34.84). As noted in earlier studies, planning in human-robot teams is a positive predictor o a
78 rmance, as does more frequent communication from the te ther handler to the ier. S Mental ea Results from the multilevel regression analysis investigatingfects of location and RSVP on shared mtal ml presented in Tabl e 12. There were no significant effects for previous experience a order effects in thl mode computions. The regression coefficient fo e significan site entered into the regression (Model D). The order in which team experimental conditions did influence thei r ores, which sugg team constructed better maps in some conditions than in others. It may be that there was simplypractict-teew be r n the second runatter w combination of location and RSVP conio ere in. Site waactor redicting map score. No support was found for Hypothesis 1 (RSVP teams better shared mental models). Teams with RSVP we re no better at constructing ma ps of their search than teams without it, as evidenced by the lack of change in model deviance when RSVP was entered into the regression equation (using Model D for comparison). However, the location of team members played a role in predicting teams mental map scores: distributed teams drew better maps of their search than collocated teams ( = 0.425 /SE = 0.174). This is contradictory to the outcome predicted in Hypothesis 4 (Collocated teams better shared mental models). This model was a significantly better fit ( 2 = 5.33) and interestingly, the order effects disappeared. team perfo robot operator. These findings are similar to those reported from the RASAR-CS earl hared Model M sure the ef en ode s are nd no e initia l ta r order becam nt whe was s went through map sc ests that s a e effec ams dr tte maps o no m hat dit ns they w s not a f in p
79 ental model nts Table 12 Multilevel regression analysis of locati on and RSVP effects on shared m Regression Coefficie Model ce ce s Devian Devian df Variable SE 3 -A 270.8 -Null 14 1 r 64 3 C. 267.09 0.04 3 r perienc Experience y Experienc 63 10 03 008 3 7 6 6 D. 266.81 0.33 1 Order -0.065* 16 0.033 2 E. 266.46 0.67 1 71* 152 4 4 261.80 5.33* 1 Location -0.425* 5 0.174 B 267. 3.69 Orde -0.0 0.03 Orde Firefighting Ex e 0.0 US&R Technolog e 0. -0.0 0.0 0.03 0.05 0.08 0.11 Site 0.1 0.20 Order RSVP -0.0 0. 0.03 0.18 F. Order -0.042 0.03 N ote. p < 0.05. e Supplemental regression analyses Following the procedure for analyzing team performance, I tested for effects of team pr ocesses on shared mental models. Again, this was not a hypothesis specified for this study, but an exploratory effort to look for possibl relationships between the team process rati ngs and map scores. The four team process ratings were entered simultaneously into the regression equation. Table 13 shows the regression coefficients and standard errors for the four team process dimensions along with model fit comparisons. After controlling for order effects, ratings of team situation awareness were predictive of the team map scores ( = 0.53/ SE = 0.173), and model deviance was significantly reduced ( 2 = 9.57), suggesting that team s with better SA were more likely to share a mental model of their search process.
80 egression analyses for shared mental model Regression Coefficients Table 13 Supplemental multilevel r Model Deviance Deviance df Variables SE A 270.83 --Null B. 267.14 3.69 1 Order -0.064 0.033 C. 257.56 9.57* 4 Order Support Leadership Situation Awareness -0.059* -0.100 0.051 0.530* 0.030 0.168 0.187 0.173 D. 238.86 28.28* 5 Search Robot situatedness Instruction Tether-operator 7.059* 1.441 -1.274 0.732 1.633 1.830 1.172 0.91 Communication -0.072 0.112 Order Navigation -0.042 -2.129* 0.509 0.990 9 p < 0.05. ch, regression coefficients for robot situatedness, instruction, and teth er-operator were not Five proportion categories from the RA SAR-CS were chosen to examine the influence of communication elements on the creat ion of a shared mental model. Sear navigation, and robot situatedne ss were chosen based on theory and findings from prior research. Instruction and tether-operator pr oportions were chosen because they were inversely related to team SA ratings in this study. The propor tion of communication devoted to search strategy was predictiv e of the teams shared mental model ( = 7.059/ SE = 1.633), in that more talk about search strategy is associated with higher map scores. Navigation emerged as a negative predictor ( = -2.129/ SE = 0.990), in that less talk proportionately about navigation is associated with higher map scores. The
81 t. The change in model deviance ( 2 = 28.28) is significant, and the order effects e The four team process dimension ratings were averaged to create a mean team process rating for the purposes of this anal ysis. Table 14 presents the model parameters aomns of f the l location and RSV s on teocess. No effects were found for experience, orfo and mcoren s wed for ef the experimental conditions as well. R ion coefficients forn ( = -0.053/ SE = 0.109) and RSVP ( = 0.150/ SE = 0.105) were non-significant and none of the models ted p a bet of td n the null model. Therefore, no support was found for Hypothesis 2 (RSVP tea more effective teamsses) n ypothesis 5 (Collocated teams more effective team processes). Supplemental regression analyses. For the final set of analyses, I chose the RASAR-CS categories of plan, report, and te ther-operator to see if these patterns of communication predicted mean team process ratings. Plan and report are RASAR team process indicators that have proven important in prior rese arch, and the tether-operator proportion has drawn attention in earlier anal yses. All three vari ables significantly predicted mean team process ratings (Table 15). Plan ( = 2.058/SE = 0.587) and report ( = 1.661/ SE = 0.533) were both positive predictors of team process. Teams that focused more of their communications on these two pur poses functioned more effectively overall. Higher proportions of tether-operator comm unication were negatively associated with significan once again disappeared. Team P rocess M asure nd c pariso it for ana ysis of P effect am pr order of condition or site, as were in previous analyses f pe rmance ap s U fortunatel y, no effect re foun ither o e gress locatio se t rovided ter fit he a ta th a ms proce or for H
82 effects on team process R Table 14 Multilevel regression analysis of lo cation and RSVP egression Coefficients Model D De Vale SE eviance viance df riab s A. 152.8-Nu 9 ll B 150.5 39 Order -0 0. C150.10.04 Order Firefightnce USR Ee Technolo -0 0. -0 -0 0.01 0. 0. 0. 0.24 1 Order Location -024 -0.053 0.019 0.109 E. 148.47 2.03 1 Order RSVP -0.032 0.150 0.018 0.105 F. 146.87 3.64 0.017 0.102 0 2. 1 .027 01 8 2 3 ing Experie & xperie nc gy Expe rience .0 28 011 .022 .005 8 029 044 059 D. 150.26 .0 1 Order Site -0.029 -0.196 Note. p < 0.05. team process (( = -2.042/ SE = 0.397), confirming the negative relationships observed in Table 9 with the team process dimensions of Leadership and SA. The model deviance was greatly reduced ( 2 = 29.42), providing a much better fit of the data.
83 process Regre nts Table 15 Supplemental regression analyses for team ssion Coefficie Model Deviance Deviance df Variables SE A. 152.89 -Null 150.50 2.39 1 Order Order Plan B. -0.027 0.018 C. 121.08 29.42* 3 Report Tether-operator -0.022 2.058* 1.661* -2.042* 0.015 0.587 0.533 0.397 Note. p < 0.05. inding RASAR-CS communication analyses, and multile ing to the outcome measure in questio elation to the hypotheses listed in Chapter 3. Table 16 provides a quick review of the hypothe ses and evidence of suppor o not directly address the study hypoth eam performa nce pictured in Figure 3. r revi ewing the hypotheses. Summary of Hypotheses and F Results from descriptive analyses, s vel regression analyses have been reported accord n. Now it is time to pull these results together in r t. Many of the significant findings in the analyses d eses, but rather the propos ed model of t These findings are summarized in relation to the model afte
84 Table 1 6 Study hypotheses and evidence of support Supported? Hypothesis Regression RASARCS Analyses Analyses H1: Teams having access to RSVP technology will generate richer, more accurate team situation models than teams not having access to the technology, as measured by search maps generated by the team, and frequencies of SA indicators in the RASAR-CS. No No H2: Teams having access to RSVP will exhibit more ratings and by frequencies of team process No Partial search task scenarios, as measured by performance team situation models than distributed teams, as and frequencies of SA indicators in the RASARprocesses than distribute d teams, as measured by observer ratings, and by frequencies of team process indicators in the R effective team processes, as measured by observer indicators in the RASAR-CS. support H3: RSVP teams will accordingly perform better in the outcome scores. Yes --H4: Collocated teams will gene rate richer, more accurate measured by search maps generated by the team, CS. No No H5: Collocated teams will exhibit more effective team ASAR-CS. No No H6: Co llocated teams will accordingly perform better than distributed teams in the search task scenarios, as measured by performance outcome scores. No --
85 ble s with s by l feedback he was receiving than thinking about the task and his 15), H1: The first hypothesis stated that teams with RSVP would generate richer, more accurate shared mental models, as evidenced by map scores and frequencies of RASARCS indicators of SA. In multilevel regressi on analyses summarized in Table 12, RSVP was not a significant predictor of map scores Supplemental regression analyses (Ta 13) revealed that the RASAR-CS categorie s search and navigation did predict map scores, as did ratings on the team process di mension SA. More statements about search strategy were positively associated with map scor es. This is consistent with prior research linking more frequent communications about search with better SA (Burke & Murphy, 2004b). However, RSVP teams did not talk appreciably more about search than team with it. Navigation statements were negativ ely related to the shared mental model measure, i.e., better map scores came from teams that talked less about navigation. Comparisons of RASAR-CS proportions be tween RSVP and No RSVP conditions revealed that tether-handlers in the RSVP condition initiated more communications the operator, many of which we re instructions about naviga tion. It may be that having RSVP detracted from the creation of a shar ed mental model between team member creating an attentional tunneling effect: when the tether-handler had RSVP, he spent more time reacting to the visua role in accomplishing it. H2: The second hypothesis predicted t eams with RSVP would exhibit more effective team processes, as measured by team process ratings and indicators in the RASAR-CS. There were no effects on team pr ocess mean scores from RSVP in the regression analyses summarized in Table 14. Su pplemental analyses revealed that the RASAR categories plan and report were positive predictors of team process (Table
86 o that ing, m to summa s a ors were and that tether-handler communications were negatively associated. In the RASAR-CS comparisons between RSVP and No RSVP condi tions, there were notable differences in these categories. First, there was more involve ment of the tether-handler in RSVP teams because he could see what was occurring in the search space. RSVP teams did less reporting than teams without RSVP, likely for the same reasonit was unnecessary t report some observations about the search environment since both members could see what was happening. As mentioned earlier, more instructions were given, suggesting the problem-holder role was shared between team members. Though this seemed to play a negative role in creating the team mental model, it is not n ecessarily a bad thing statement function comparisons for operators and tether-handlers revealed that 18% of operator statements and 31% of tether-handler statements were classified as planning. RSVP teams had 25% more planning statements than did No RSVP teams, and plann as noted above, predicts more effective t eam processes. Taken together, these four observations (increased tether -handler involvement, less re porting, more instructions, more planning) suggest that team pr ocesses in RSVP teams are at least different whether they are more effective is arguable, but the increase in planning would see support their being beneficial. H3: The hypothesis that RSVP teams would perform better in the search task as measured by performance scores was supported in the multilevel regression analysis rized in Table 10. Even after accounting for site effects that emerged in previou models, RSVP positively predicted performance ( = -1.416/ SE =0.586) and produced significantly better fit to the data ( 2 = 5.67, df = 1). Though RASA R-CS indicat not included in the hypothesis, supplemental regression analyses revealed that the
87 would ms. Other and the h environment has been associated with better SA in past studies why it ; into play rbut this may be due to th e level of the measure used. S f e 12 were actually in th e opposite direction: proportion of search statements was a positiv e predictor of perfor mance (Table 11). In light of the larger proportion of planning statements attributed to RSVP teams, this seem to strengthen the case for the effectiven ess of team processes in those tea significant predictors of perf ormance in the RASAR-CS were robot situatedness proportion of statements initiated by the teth er-handler. Both of these were negative predictors: that is, fewer statements about robot situatedness and less communication from the tether-handler were associated w ith better performance. Talking about the robots situatedness in the sear c and better SA is typica lly linked with better perfor manceit is not clear seemed to detract from performance in th is study. The mixed benefits from increased involvement of the tether-handler have alrea dy been discussed. It is easy to blame poor team performance on the tethe r-handlers hijacki ng of the operators navigator role however, this is not a new phenomenon, and it ma y be that other factors come here (perhaps those dreaded site effects) The team process dimension Communication also predicted performance. This is not su rprising, as the importance of effective communication is a point well understood in team performance research. What is surprising is that SA was not a predicto A is defined somewhat differently as a team measure, relying more on the sharing of important information about the environmen t rather than just being aware of it. H4: The fourth hypothesis stated that co llocated teams would generate richer, more accurate shared mental models, as ev idenced by map scores and frequencies o RASAR-CS indicators of SA. This hypot hesis was not supported. Results from regression analyses summarized in Tabl
88 distribu effects ent, ltilevel regression analyses. was to iabe differen m of ask ted teams had higher map scores ( = -0.425/ SE =0.174). There were order present in earlier models, but these becam e insignificant when the location IV was entered into the model. RASAR-CS anal yses did not support this hypothesis: comparisons between collocated and distributed teams revealed no appreciable differences in statement proportions categorized as SA indicators (search, environm information synthesis, plan, and report). Ho wever, collocated teams did talk less about robot situatedness, a RASAR category that was negatively related to performance in mu H5: Collocated teams were hypothesized to exhibit more effective team processes, as measured by team process ra tings and RASAR-CS i ndicators. Location not a significant predicto r of team process ratings. This is surprising since it is generally accepted both in research and the workplace that collocated teams function more effectively than distributed teams. It is possible that the distributed condition manipulation was weak, or that the mean team process ratings lacked enough variance show differences. In the RASAR-CS analyses by location, there were no apprecl ces in the pattern of communicati on between team members, the for statements, in content, or in function. H6: The hypothesis that collocated teams w ould perform better in the search t as measured by performance scores was not supported in the multilevel regression analysis summarized in Table 10. Location di d not predict performance. There were no differences between collocated and distri buted teams in the RASAR-CS categories associated with performance. Notably, there were no differences in planning or reporting as observed in the RSVP comparisons.
89 dy iables d mental models, team process, and team performance. The e er ng to the rch atings of al ies bode well Six of the 7 hypotheses relating to RS VP and location from Chapter 3 are presented in Table 16, along with indicators of whether support was found in the stu analyses. These hypotheses were tests for main effects of the two independent var (RSVP and location) on share seventh hypothesis, which deals with the potential interactive effects of RSVP on performance by location, is not directly testable in light of the complexities posed by th crossed and nested data, and the observed site effects. It is discussed, however, in Chapt 6. Looking at Table 16, it seems that the inve stigation of the effects of RSVP and location on shared mental models, team process, and team performance yielded very little: RSVP predicted performa nce, and nothing else; locati on predicted shared mental models, but not in the way expected. Yet the results are full of significant findings that tell much more than what appears in the tabl e, and a review of thes e findings is merited. Results of the supplemental analyses are pr esented in relation to the three dependent variables of map score, mean team pro cess ratings, and performance outcome. RSVP did not predict better shared mental models, but the RASAR communication categories search and naviga tion, and the team process ratings for situation awareness did So we do know something about whats contributi development of the shared mental model betw een team members. More talk about sea contributes to good SA. This is a point made in earlier studies comp aring SA r operators (Burke & Murphy, 2004a), and it is reified here. Getting caught up in the details of navigation, on the other hand, is not conducive to forming rich shared ment models of the search process. Moreover, the RASAR-CSs predic tive abilit
90 ssisted s: be tter SA (Burke & Mu rphy, 2004a). In this study its clea communication patterns between team member s are critical in team functioning, and since the model of team perfor mance in robot-assisted search in Chapter 3 theorizes that the shared mental model is formed through team communication, we can make a link between the creation of shared mental models and effective team process. Finally, these analyses showed that team process ratings for Communication predicted performance scores, thus esta blishing a link between effective team communication and better team performance. The RASAR-CS regression analyses pinpointed some of the specific categories and patterns of comm unication that were associated with performance: plan, robot situatedness, a nd tether-operator for its abilities to cap ture the process of creating shared mental models of robot-a technical search. It is also evident that the relationship between team processes and shared mental models is a reciprocal one, in that the team process ratings for SA predicted better map scores. More effective team processes were pr edicted from three RASAR-CS categorie plan, report, and tether-opera tor communications. Again, past research has shown that operators who focused on goal-directed co mmunication (planning the search, reporting on what is seen in the environment) had ir that these communication functions influe nce team processes overall. The role of the tether-handler in the search process was revealed to have much import on effective team functioning, as more communi cations from the teth er-handler to the operator seemed to have a negative effect. A large part of these communications were about navigation, which were deleterious to the development of a shared mental model between the two team members. In all, th e significance of these findings show that
91 communications. The import of the plan a nd tether-operator categories has been discussed at length, but the impact of robot situatedness has not. In past research, operators who talked about th e robots location and spatial orientation in the search environment (position) performed bette esults were puzzling. Robot s a yses e del of r. Here, the r situatedness was not a significant predictor of the shared mental model, yet there a strong negative relationship with performance ( = -20.802/ SE = 5.698). Too, the RSVP teams talked less about the robots position, as did collocated teams. Clearly, this is a topic to be explored in future research. In this chapter, results from descri ptive analyses, communication anal conducted using the RASAR-CS, and multile vel regression analyses of team performance, shared mental models, and team process have been presented. Results hav been linked with 6 of the 7 study hypotheses, and with the team performance mo robot-assisted technical search (Figure 3). What does it all mean? And yes, what about that 7 th hypothesis? That, dear reader, is the t opic for discussion in the next chapter
92 gs from the study are discussed, focusing on the mmunication in distributed US&R teams through a shared remote visual presenc Chapter 6 Discussion In this chapter, the findin hypothesized interaction between the two main study variable s as a way of interpreting the results. The main effect found for RSVP on performance (but not shared mental models or team process) is discussed in re lation to the model of robot-assisted technical search team performance that was introdu ced in chapter 3. The effects found for the influence of location on the development of a shared (team) situation model (and subsequent lack of effect on team proce ss and performance) are considered, and linked with the site effects noted in the multilevel regression analyses of performance. Theoretical and practical applications of these results are broached, as are the limiting factors that bound the findings. The chapter closes with some parting thoughts and conclusions about the future of RSVP in human-robot teams. This research began by proposing the use of mobile rescue robots as a way of augmenting co e consisting of the robots view. I hypothesized that by helping team members build a shared mental model, the use of mobile robots as a shared visual presence in remote environments might lead to more eff ective distributed team performance in robotassisted technical search teams. This led to two main research que stions: 1) does using the robot as remote shared visual presence a ffect team process and performance; and 2) can RSVP facilitate performance in distribut ed human-robot teams? These two research
93 effect for use of RS VP technology, then the differences between collocated teams and distributed team ess in the teams that utilize RSVP t nce do) SVP no d the st, eam questions culminated in 7 hypotheses, 6 of wh ich have been discussed in the previous chapter. The last hypothesis states: H7: If there is a main s will be si gnificantly l han in those that do not. Results from the first 6 hypotheses show that there was an effect for using the robot as RSVP, i.e., it did seem to help performance. However, the process did not unfold as predicted in the model and hypotheses deli neated in Chapter 3. There was no evide that RSVP contributed to the shared mental model held by team members, and conflicting support for its influence on team processes. As far as the location of team members goes, I anticipated that collocated teams would pe rform better all around (a s they typically and hypothesized in H7 that if RSVP had an effect, then the distribu ted teams with R might do a little better than the distributed te ams without it (not as well as collocated teams, but better). What happened instead was this: the collocated teams didnt do better, after all. In fact, the distributed teams had be tter shared mental models, and there were differences at all between the collocated and di stributed teams in terms of team process or performance. So, RSVP worked, but not as predicted, and it is unclear whether it helpe distributed teams catch up to the collocated ones in terms of perf ormance, because the collocated teams didnt perform better in the fi rst place. Two main questions arise. Fir if RSVP didnt help the team s form better shared mental models, or have better t processes, then how did it influence perfor mance? Second, whats up with the collocated teams not performing better than the distributed teams? To answer the first question, look
94 e een to the theoretical model in Chapter 3. As fo r the second question, I believe this is where the site effects come into play. Lets look at each of these in turn, and perhaps some light will be shed on H7. RSVP and the Model To review the model of team performa nce in robot-assisted technical search (Figure 3), RSVP was posited to augmen t communication between team members by giving the tether-handler the same robot data from the remote search environment as the robot operator. By having the same visual referent, the team members would form a richer shared mental model of the search en vironment and process as they performed th search. This shared situation model, form ed through enhanced communications betw the team members, would in turn positivel y affect team processes (Communication Effectiveness, Support/Backup Behavior, Lead ership/Initiative, and Team Situation Awareness), thus leading to bett er team performance. Let us look at the relationships (signified by lines/arrows) among the constr ucts in the model and examine where the findings apply. First, the centr al dotted-line box that holds the shared (team) situation model is crossed by the bi-d irectional line (communications ) between the robot operator and tether-handler, representing that the shared mental model is formed through communications between team members. This is paralleled in the results by the predictive relationship between the RASAR-CS categories search and navigation and the map score which measured the team mental model. The arrow connecting the shared (team) situation model box to the team pro cess box appears in the results of the supplemental analyses of team process, wher e the RASAR-CS categories of plan, report and tether-operator predicted the mean team process scores. (An arrow going back from
95 re the we (via and d s. Is the es. VP and s to ituation model through the robot operator and tether-handlers own mental models of the sear ch process er, the unshared data receive the of team process to the shared situation model can be added to illustrate the reciprocal natu of the team process/shared mental model relationship obs erved in the supplemental analyses for the shared mental model. R ecall that the team process ratings for SA predicted the map scores.) Next, the arrow from team process to team performance in model is mirrored in the results by the fact that team process ratings for Communication predicted performance scores. Through the findings of the supplemental analyses, have traced the process of shared mental model formation through communication the RASAR-CS) and established a reciprocal link between the shared mental model team process; lastly, we have shown that the team process of communication is linke with team performance (thereby validati ng a portion of Klomoski & Mohammeds model). What has not been identified is how RSVP contributed to that proces model described in Figure 3 de ficient? I think not, but it does not completely match what was measured in the study. I assumed that th e effects of RSVP would be captured in the communications between team members, and to some degree, they were; differences between RSVP teams and no-RSVP teams we re observed in the RASAR-CS analys However, the hypothesis stated in H1 does not account for the path between RS the shared situation model (.teams havi ng access to RSVP technology will generate richer, more accurate team situation models. In the actual model, RSVP contribute the shared (team) s and environment. Moreov d by each of the team members (e.g., the OCU interface for the robot operator, and the part of the search space visible to th e tether-handler from where he inserted robot into the void) is not accounted for. It ma y be that combining these different kinds
96 re are e ntal model created together might clarify exactly what it is in RSVP that helps th e team perform more effectively. Too, the model is somewhat deficient in that it does not take into account the other factors which may c ontribute: the individual, team, environmental and organizational antecedents listed in Kraiger and Wenzels framework for shared mental models (1997), or the resources available and other factors feeding into team capacity in Klimoski and Mohammeds model of team performance (1994). What this model did do is illuminate the processes that go on in robot-assisted technical search teams, and demonstrate the value of RSVP as a team resource. To understand its (RSVP) contribution toward team performance, howev er, the model must be expanded to include ithin the model. Location and Site Effects Turning to the location question: why di d the collocated teams not outperform the distributed teams, and what do the site eff ects have to do with it? To answer these questions, comparisons of performance by lo cation and RSVP condition must be made input with the data from the robot acting as RSVP contributes to each team members individual situation model in a unique way. While it is possible to make some inferences about each individuals mental model by looking at what he or she talked about, the obviously some internal cognitions that ar e not voiced by team members. So, while analyzing communications between team memb ers can help trace the process of shared mental model formation, it cannot completely capture the formation of each team members individual model of the situa tion. Taking measures of team members individual mental models and comparing th em with the team m other constructs, and refined to explain the relationships between existing constructs w
97 all n the independent and this is not a representation of a statistically significant interaction. In Figure 9, the solid line represents the distributed teams and the dotted line, the collocated teams. The mean performance scores for collocated teams across RSVP conditions are very similar (for RSVP, M = 4.64, SD = 3.42; for no-RSVP, M = 4.86, SD across sites. Before looking at these comparisons, though, we need to revisit H7. Recthat this hypothesis assumed that collocated teams would outperform distributed teams, and predicted that if RSVP had an effect on performance, the differences between collocated and distributed team performance would be smaller in the RSVP conditiothan in the no-RSVP condition. To establish a start point for the discussion, the mean performance scores for the four combinations of experimental conditions are presented in Figure 9. It is important to note that these are not independent groups, and the lines infigure do not represent a statistically significant interaction. Figure 9. Mean performance scores plotted by location and use of RSVP. Scores are not
98 M = 2.30, SD = 1.99). The wide variance for all four with the oy had RSVP. The collocated teams with no-RSVP performed slightly better than the distributed teams without RSVP, but not by much. Looking next at NJTF-1 (lower graph), this time the results followed the pattern predicted in the location hypotheses: collocated teams had better performance than distributed teams rega rdless of whether they had RSVP or not. The results for the RSVP hypotheses, however, are contradictory: RSVP helped teams in the distributed condition, but in the colloca ted condition, they actually performed better without it. At both sites, something else seem s to have influenced the performance of the = 3.98). The means for the distributed teams, in contrast, are markedly different (for RSVP, M = 5.07, SD = 2.04; for no-RSVP, mean scores underscores the nature of the datathese are repeated measures teams having scores in more than one c ondition. If these were independent groups, however, the visual impact of the interaction is obvious: distributed teams with RSVP performed as well as collocated teams. The f act that there was no main effect for location points to something else making this effect occur: and so we must address the likely culprit, site effects. Figure 10 presents gra phs of mean performance scores according to location (distributed or collocated) and use of RSVP at the two sites, NASA-Ames in California, and NJTF-1 in New Jersey. Agai n, all teams completed runs in 2 of the 4 conditions, so these data are dependent; they are used here to tease apart the nature of differences between sites. Looking first at NASA-Ames (upper gra ph), the results followed the pattern predicted in RSVP hypotheses: RSVP team s had better performance scores than n RSVP teams in both conditions. The results for the location hypotheses, however, were not as expected: the distribut ed teams outperformed the collocated teams when the
99 teams. There are two possibili ties that come to mind based on what is known about the two sites: experience, and team cohesion. Experience (firefighting, US&R, and t echnology) was included in the study analyses because of its potential effect on study outcomes of interest. In the multilevel regression analyses for shared mental models, team process, and team performance, however, experience did not prove to be a significant predicto r. However, comparisons of firefighting experience between the two sites (Table 5) reveal that while both NASAAmes and NJTF-1 have a significant percenta ge of highly experienced participants with 15+ years of experience (46% and 59%, respec tively), at NASA-Ames there were also a significant number of participants with ve ry little time on the job. There, 40% of the participants had 0-3 years of firefighting e xperience; at NJTF-1, that percentage was much s NJTF-1 id so much better than the no-RSVP teams at NASA-Ames: it could be that those less experienced participants were more receptive to using a new type of search tool, as they did not have a backlog of prior, more traditional search experiences to overcome. maller (18%). It may be that having fe wer participants with less experience at made a difference. This could also be part of the reason the RSVP teams d
100 Figure 10. Bar graphs showing mean performance scores at NASA-Ames and NJTF-1 sites according to location (distributed or collocated) and use of RSVP. Means are dependent.
101 The second possible explanation is differe nces in team cohesion across sites. Team cohesion is the degree to which team members are attracted to their team and desire to remain in it. Components of team cohesion include inte rpersonal attraction, group pride, and task commitment (Driskel l et al., 2003). The NASA-Ames teams had a mix of DART responders that worked togeth er regularly onsite and task force members from other teams across the country who came to participate in the tr aining exercise. Of the 14 teams, only 5 consisted of two DART responders; the rest were paired with visiting responders from other units. The NJTF-1 teams, in contrast, were all from the same Task Force and had many years of experi ence working with each other. This could explain to some degree why the collocat ed teams there performed best in the collocated/no-RSVP condition: it most closely resembled their normal pattern of work. There are certainly other factors that ma y have contributed to these patterns of performance, ranging from environmental conditions to individual differences in technology acceptance. The observations regard ing experience and team cohesion seem to be the most defensible, as they are supported by the demographic details in the study. Do they offer any support for the existence of the location x RSVP interaction predicted in H7? That is a matter of speculation. It s eems that RSVP can help distributed teams be more like collocated teams in technical search in terms of perf ormancebut it cannot replace the human factors of individual experience and t eam strength (cohesion) that comes from team members knowing and workin g with each other over time. In any case, it is safe to say that there was most definitely a site x location x RSVP interaction.
102 odel ssisted e s he ect the Theoretical and Practical Implications Theoretical Implications One of the proposed theoretical contributions of this study was to test a portion of Klimoski & Mohammeds model, and to extend it to show how the shared mental m is formed through communication, as posited in the more specific model of robot-a team performance described in Chapter 3. The findings of the supplemental analyses support the relationship between team pro cesses and performance, and validate the reciprocal link between the shared mental model and team process, thereby providing support for that portion of the Klimoski & Mohammed framework. As a further theoretical contribution, using this studys model of robot-a ssisted team performance, th process of shared mental model formati on, and its influence on team process and performance was traced through communicati on via the RASAR-CS, providing support for the concept of using communication as a measure of the mental model that emerge through the interaction between team members. Neither model truly expresses all of t constructs and interrelations hips that characterize t eam performance in extreme environments such as US&R. Klimoski and Mohammeds model does not adequately capture the influence of various constructs on team mental models, and the model of robot-assisted team performance neglects th e broader influences that aff individuals and the team in incident response. As a t hought exercise, how would one model these influences? To begin with, Klimoski and Mohammed say that team mental models reflect team processes (which I agree with). In f act, I would go so far as to say that the development of team mental models is a team process. They also say that team mental
103 it team processes which include the develop ds to nt uld nd mmunity as well as those brought to bear by the organization, am and individuals. The intera ction of these three broad f actors (environment, time, and models are a force with which to harness a te ams capacity, or readiness. In this we are also in agreement. However, the authors dont discuss team capacity other than to say is the teams latent potential for effective pro cess and performance. I think that this (team capacity) is a key element in how well teams can form shared mental models, and deserves to be looked at as an antecedent of the ment of shared mental models. Moreover I think the concept of capacity nee be expanded to levels above and below that of the team. Potential influences on team capacity could include the teams current work-life and past work history, cohesion, group tensions, and relationships with other groups both within and outside its pare organization. Other important influences that need to be acknowledged and defined are individual capacity and organizational capacit y. Individual capacity, or readiness, wo include not only levels of trai ning and experience, but other variables that could impact performance in extreme environments, such as personal morale, emotional state, and cognitive readiness (Wood, Lugg, Hysong, & Harm, 1999). Organizational capacity might include leadership, command structur e, resource allocati on, and both intera intra-organizational coordination and coopera tion. All three of thes e capacity levels (individual, team, and organiza tion) are impacted by the environment, time, and available resources. Environmental considerati ons include the weather, extent of damage/disruption caused by the incident, a nd current level of danger/continued risk Temporal factors include the time elapsed since the incident occurred, time since mobilization of response, etc. The resources available include considering those existing in the environment and co te
104 e capacity (both perc eived and real) of the individual, team, and organiz hat ared l resources) determines th ation. These factors in turn all play a role in the development of team mental models, and other team processes that ultimatel y affect team performance. Note that this multilevel model could be further expanded to include constructs of individual and organizational effectiveness and performance. Using a map constructed by both team me mbers as a measure of the shared mental model is another contribution that has implications for future research in shared mental models. The fact that it showed c onvergence with elements in the RASAR-CS shows that were getting at the same underlyi ng construct. Maps are a particularly valuable measure in this domain, more illustrati ve of the type of shared mental model t is needed and formed during the task. Still, it is a static representation of a dynamic process. Maybe there is a wa y to capture the development of the shared mental model over time by looking at how th e map is created at differe nt points during the search processwhen are details of spatial dimens ion added, how does the search path form, and do temporal elements play a role in the mental model as captured on the map constructed by the team member s? Too, it behooves us to inve stigate the influence of the individual mental models held by team members; perhaps, as suggested earlier, by having the two team members construct their maps se parately, and compare them with the sh version prepared after the f act. Looking to see what elements are merged, what gets dropped or corrected, and observing how team members come to agreement over the fina correct version of the map would offer an in triguing window into the process of shared mental model formation.
105 The practical implications of this study are numerous. The fact that RSVP helped US&R he iving ay hink about naviga d be he t Practical Implications technical search teams perform more effectively is encour aging, even though we havent quite figured out how it helps. Ther e are some troubling aspects, however, that need to be taken into consideration. By giving the tether-handler the shared view from t robot in the search environment, we inadvert ently distracted him from his role by g him a window into the operators task of na vigation. The visual at traction of RSVP m have an attentional tunneling effect, where team members react to what they are seeing (in terms of navigation) instead of focusing on the primary task of helping search for victims. Clear roles and task goals need to be identified for team members working in human-robot teams that optimize their streng ths and available resources. Moreover, we need to train them to function effectively in these roles. The operator has to t tion and search simultaneously, and its th e search that suffersthis is the task the tether-handler can help take on because he has the luxury of not having to drivean therefore as the passenger can pay far more attention to whats being seen in the environment. In the future, when advances in wireless technology make the tether itself unnecessary, this role could b ecome completely cognitive and mission-oriented. While it is feasible (and in many ways desirable) that this role could be fulfilled by someone outside the Hotzone, in reality US&R responde rs work in teamsno one is going to out there alone, so it makes sense to utilize that fact. It takes a NASA team to handle t Mars Rover, a military team to operate the Pr edator UAV, and the same is true here: i takes a US&R team to fully use a rescue robot. What is needed is training at the team
106 team creased. s rsonnel. Once a victim has been located, the rescue and extrication process levelhow to coordinate, what information to share, and what roles and tasks to focus on. In terms of design, what can be done to make RSVP better? What roles can the robot take on as it gains more autonomy? If we can make navigation autonomous, members can concentrate on the real task at hand (locating victims). The key question then becomes, what are the salient elemen ts of RSVP that need to be emphasized? Indicators of the robots state and situate dness relative to the search environment are clearly of interest. If team members had a way of visua lly annotating their search (perhaps with a tablet-pc type of interface), and sharing that with other team members, the power of RSVP as a builder of mutual knowledge would be dramatically in Finally, having the ability to play back earl ier portions of their search for comparison would help team members synthesize information obtained from the robot. The use of RSVP has applications not onl y in other functions with US&R teams, but beyond US&R as well. Within the US &R system, RSVP has potential uses for coordination of actions across operations technical search, structur al evaluation, rescue and victim extrication, medical reachback a nd victim management, safety monitoring, logistics and resource management. Many of these operations could be expanded to include offsite pe can take from 4-10 hours. During the extrication process, a robot providing shared visual presence can be used as a communication channel not only for medical personnel, but also for a counselor or family member at a remote location to provide comfort/reassurance/encouragement to the victim during the prolonged extrication period. Safety monitoring is another poten tial domain application. US&R operations
107 e team during work process and the ongoing condi oid, allowing an objective viewpo r nse. cue eeds o is require the presence of a safety officer ons ite during rescue operations. If a robot goes into a remote work area (deep into a void or confined space) with a rescue team, it can provide the rescue squad leader outside the void space with a visual image of th tion of the v int (that is, not in the void) from which to monitor the work process and look fo signs of fatigue, physical danger, or unnoti ced errors. This simultaneously opens new opportunities for logistics and resource mana gement during an emergency respo Having the robot with a rescue team in a remo te, confined work space provides the res squad leader or incident commander with a more efficient communication channel than radio, as the robots view can provide common ground for more implicit team coordination. For example, the leader may conduct an onsite assessment of the teams progress, estimated completion time of the ta sk, and the teams ongoi ng logistical n by observing the activities via the shared visu al presence of the r obot. In summary, using mobile robots as RSVP in US&R environments can serve as a conduit for the transfer of information horizontally (to other team member s involved in common tasks), vertically t higher/lower levels (squad leaders, incident command, victims) and through diffusion to specialists (structural engin eers, medical personnel) and others (family members, task force liaisons). The search and safety monitoring capabil ities offered by RSVP are a natural fit for the military and homeland security domains, and the recent advances in telemedicine bode well for extending its usage into that area as well. Remote assisted care-giving not too far away, as evidenced by a recent ar ticle in Newsweek where two sons rescued their elderly mother from her apartment wh ere she had suffered a stroke. They kept a
108 t pan. has its difficulties in trying to conduct quasi-experimental studies in field setting t of e es that e windy, ere not told that it was an assessment tool. It may be necessary in future studies t e measure an e video-camera in her home for communicating via the internet, and noticed she was no walking around the apartment. Imagine being able to help her remotely through a caregiver robot. Creepy? Perhaps, but alrea dy underway in research labs in Ja Limitations Before closing, certain limitations shoul d be acknowledged when considering these findings. A wise man once told me never to apologize for field research-it place, its value is immeasurable, and not meeting the strict requirements of the experimentalist model is what gives it power in terms of external validity. That being said, there are always s. Creating a distributed condition when pa rticipants had to stay within earsho each other was difficult: though pa rticipants faced away from each other and were told to pretend theres a wall between you, this manipulation may not have been strong enough to find differences between distributed and collocated teams (though there wer differences in map scores that suggest it wa s effective). The map measure used to ass the team mental model of the search has some vulnerabilities of its own. It is likely some teams had better drawing ability than others, which may have led to differences in ratings for reasons other than the quality of th e shared mental model. It is also possibl that some teams took more care in constr ucting their maps than others, for various reasons: the circumstances in which they ha d to draw were not convenient (cold, on top of a rubble pile), and they w o train part icipants on how to respond to th getting them to tell the story of their search process on pape r. However, I do see it as appropriate measure in this domain, and thi nk it shows promise as a team-level measur
109 ed by llyof shared mental models. Other measures, su ch as ranking lists of relevant factors in order of importance, do not lend themselves easily to this applicati on. Insertion of a new technology changes the list of relevant factors, and creates new factors. The team process ratings used in this study were based limited to 4 certain dimensions (communication effectiveness, s upport, leadership, and situation awareness). There are many other ratings categories that can be explored. For example, team orientation, monitoring, and fee dback are other dimensions of team process propos Dickinson & McIntyre (1997) th at might provide insight in to how RSVPs influence on performance occurs. Finally, the RASAR-CS is a complex communication coding scheme. Most coding analyses classify statements into 4 or 5 categories at best; the RASAR-CS has 4 dimensions with a total of 30 possible coding categories. There are many statements that fall into more than one category, particularly in terms of content and function. This is reflected in the lower estimates of rater ag reement for these dimensions. The rate of agreement is acceptable for the purpose of the study, and the findings point to the RASAR-CS as a valuable techni que for getting at the details which give insight into the results. However, other techniques can a nd should be explored. For example, coding categories from the RASAR-CS could serve as the basis for construc tion of behaviora anchored rating scales or frequency check lists, offering another avenue into the observation of team functioning. Parting Thoughts and Future Directions This study was motivated by the desire to see whether using mobile robots as a shared visual presence in remote environments could help distributed teams in US&R
110 getting theres an obvious gain for the first part; if it worked with team embe nment be t what if it breaks down? We need to know what the salient aspects of RSVP are so that we can improve robot RSVP designand we need to know what the shortcomings and pitfalls of using RSVP are so that we can deal with them, either through design, or more likely, through training teams how to use it most effectively. In a training program for resc ue robot teams developed and piloted by the Center for Robot-Assisted Search and Rescue (Burke, Murphy, & Ka lyadin, 2005), initial measures of participant r eaction and learning yielded encouraging results. Following Kirkpatricks (1994) model of training evaluation, future developments in training should include criteria for behavior (perform ance on the job) and results (impact on organizational objectives). Other future directions for this re search include continuing the search for the elusive shared mental model. Results from this study suggest investigation of the influence of individual mental mode ls on the development of a shared mental perform more effectively. The reasoning be hind this was twofold. Effective performance in this domain consists of two main elemen ts: successfully locating victims and them out of a dangerous environment, and doi ng it without getting hurt. If RSVP could help teams locate victims, mrs in different loca tions, then people safely away from the dangerous enviro could help search for victimspeople with fresh minds, clear eyes and no cognitive fatigue from being onsite for 48 hours stra ight. The import of this cannot be overemphasized. US&R teams operate in extreme environments and are under tremendous emotional strain in addition to the physical and cognitive stress. The findings from the study seem to point to RSVP as an idea wort h pursuingbut further research needs to performed to figure out how it works. Its an alogous to driving a car: you dont have to know how it works to drive one, bu
111 odel. In addition, the use of maps as a meas ure of shared mental models merits further xploration. Finally, I have introduced a multilevel conceptual model of team performance in extreme environments t the egg of an idea. This model needs huge potential as a way of allowing unexpe e chance to use it. Robots with RSVP took the US&R search teams into an environment they co fer an in hysical environments without being physically presentth rough the contributions of their ne m e hat is merely to be refined and tested. Because of their mobility, robots offe r humans a presence in remote environmen ts. As with any new technology, new and cted uses and applications will emerge as the technology ma tures and people hav a uldnt have reached to search before In like fashion, robots with RSVP of entry into a whole new world of teamingwhere team memb ers can work together p w team member, the robot.
112 References Ambro er, s, IEEE [see also IEEE Expert], 15 (4), 57-63. Brannic rement. & C. Prince (Eds.), Team Performance Assessment and Measurement: Theory, Methods and Applications (pp. 3-16). Mahwah, NJ: Brenna uses, and alternatives. Educational and Psychological Measurement, 41 687-699. Brooks : -504. h rant (No. CRASAR TR2005-14). Tampa, FL: Center for Robot-Assisted Search and Rescue. Burke, than One. Paper presented at the IEEE RO-MAN 13th International Workshop on Robot and Human Interactive Burke, ion Awareness and Task Performance in Robot-Assisted Technical Search : Bujold Goes to Bridgeport (No. CRASARBurke, ddle, D. L. (2004). Moonlight in Miami: A field study of human -robot interaction in the context of an urban search n, se, R. O., Aldridge, H., Askew, R. S ., Burridge, R. R., Bluethmann, W., Diftl M., et al. (2000). Robonaut: NASA's space humanoid. Intelligent System k, M. T., & Prince, C. (1997). An ove rview of team performance measu In M. T. Brannick, E. Salas Erlbaum. n, R. L., & Prediger, D. J. (1981). Coefficient kappa: Some uses, mis R. (2002). Flesh and Machines: How Robots Will Change Us New York Pantheon. Bruemmer, D. J., Few, D. A., Boring, R. L., Marble, J. L., Walton, M. C., & Nielsen, C. W. (2005). Shared understanding for collaborative control. Systems, Man and Cybernetics, Part A, IEEE Transactions on, 35(4), 494 Burke, J. L., Murphy, R., & Kalyadin, D. (2005). Annual Report for the Florida Hig Tech Corridor Workforce Enhancement G J. L., & Murphy, R. R. (2004a, Sept. 20-22, 2004). Human-Robot Interaction in USAR Technical Search: Two Heads are Better Communication, Kurashiki, Okayama, JAPAN. J. L., & Murphy, R. R. (2004b). Situat TR2004-23). Tampa, FL: Center for R obot-Assisted Search and Rescue. J. L., Murphy, R. R., Coovert, M. D., & Ri and rescue disaster resp onse training exercise. Human-Computer Interactio 19(1-2), 85-116.
113 s, 34(2), 103-112. up asper, J., & Murphy, R. (2002). Workflow Study on Human-R obot Interaction in USAR. d Cybernetics Part B-Cybernetics, 33(3), 367-385. l ing ambridge: Cambridge University Press. 37-46. J. (2003). Coovert, M. D., & Burke, J. L. (2005). Leadership and decision making. In Y. AmichaiHamburger (Ed.), The social net: Human behavior in cyberspace (pp. 219-246). oovert, M. D., & Foster Thompson, L. (2001). Computer Supported Cooperative Work: Cramton, C. D. (2001). The mutual knowle dge problem and its consequences for dispersed collaboration. Organization Science, 12 (3), 346. Burke, J. L., Murphy, R. R., Rogers, E., Lumelsky, V. J., & Scholtz, J. (2004). Final report for the DARPA/NSF interdisciplin ary study on human-robot interaction. Ieee Transactions on Systems Man and Cybernetics Part C-Applications and Review Cannon-Bowers, J. A., Salas, E., & Converse, S. A. (1993). Shared mental models in expert team decision making. In N. J. Castellan (Ed.), Individual and gro decision making: Current issues (pp. 221-246). Hillsdale, NJ: Erlbaum. C Paper presented at the IEEE Intern ational Conferen ce on Robotics and Automation (ICRA 2002). Casper, J., & Murphy, R. R. (2003). Human-r obot interactions during the robot-assisted urban search and rescue respons e at the World Trade Center. Ieee Transactions on Systems Man an Clancey, W. J. (2004). Roles for agent assist ants in field scien ce: understanding persona projects and collaboration. Systems, Man and Cybernetics, Part C, IEEE Transactions on, 34(2), 125-137. Clark, H. H., & Marshall, C. E. (1981). Defin ite reference and mutu al knowledge. In A K. Joshi, B. L. Webber & I. A. Sag (Eds.), Elements of discourse understand (pp. 10-63). C Cohen, J. A. (1960). Coefficient of agreement for nominal scales. Educational and Psychological Measurement, 20 Cooke, N. J., Kiekel, P. A., Salas, E., St out, R., Bowers, C., & Cannon-Bowers, Measuring team knowledge: A window to the cognitive underpinnings of team performance. Group Dynamics, 7(3), 179-199. New York: Oxford University Press. C Issues and implications for worke rs, organizations, and human resource management Thousand Oaks, CA: Sage.
114 Mahwah, NJ: Erlbaum. Diez-R w gical s, ndsley, M. R. (1988). Design and evaluation for situat ion awareness enhancement. Teamwork and the development of shared problem models. In E. Salas & S. M. l Foushee, H. C., Lauber, J. K., Baetge, M. M., & Acomb, D. B. (1986). Crew factors in flight operations: III. The operational significance of exposure to short-haul air DeShon, R. P., Kozlowski, S. W. J., Schmid t, A. M., Milner, K. R., & Wiechmann, D. (2004). A multiple-goal,multilevel model of feedback effects on the regulation of individual and team performance. Journal of Applied Psychology, 89 (6), 10351056. Dickinson, T. L., & McIntyre, R. M. (1997) A conceptual framework for teamwork measurement. In M. T. Brannick, C. W. Prince & E. Salas (Eds.), Team Performance Assessment and Measurement: Theory, methods and applications oux, A. V. (2000). Multilevel Anal ysis in Public Health Research. Annual Revie of Public Health, 21 (1), 171-192. Driskell, J. E., Radtke, P. H., & Salas, E. (2003). Virtual teams: E ffects of technolo mediation on team performance. Group Dynamics-Theory Research and Practice 7(4), 297-323. Endo, Y., MacKenzie, D. C., & Arkin, R. C. (2004). Usability evaluation of high-level user assistance for robot mission specification. Systems, Man and Cybernetic Part C, IEEE Transactions on, 34 (2), 168-180. E Paper presented at the Human Fact ors Society 32nd Annual Meeting, Santa Monica, CA. Endsley, M. R. (1995). Toward a theory of situation awareness in dynamic systems. Human Factors, 37(1), 32-64. Fiore, S. M., & Schooler, J. W. (2004). Process mapping and shared cognition: Fiore (Eds.), Team Cognition: Understanding the factors that drive process and performance (pp. 133-152). Washington, D.C.: American Psychologica Association. Fleiss, J. L. (1981). Statistical methods for rates and proportions (2nd ed.). New York: Wiley. transport operations (No. NASA Technical Memo No. 88322). Mountain View, CA: NASA.
115 ussell, S. R., Setlock, L. D., & Kraut, R. E. (2003, April 5-10, 2003). Effects of head), ssell, S. R., Setlock, L. D., & Parker, E. M. (2003, April 5-10, 2003). Where do helpers mputer-Human Interac tion (CHI 2003), Ft. Lauderdale, FL. l n Computer Supported Cooperative Work, Chicago, IL. Gergle, al of utwin, C., & Greenberg, S. (2004). The impor tance of awareness for team cognition in on: ackman, J. R. (1987). The design of wo rk teams. In J. W. Lorsch (Ed.), Handbook of inds, P. J., & Bailey, D. E. (2003). Out of si ght, out of sync: Understanding conflict in Hox, J. Erlbaum. gen, D. R., Hollenbeck, J. R., Johnson, M., & Jundt, D. (2005). Teams in organizations: Kanki, B. G., Lozito, S., & Foushee, H. C. (1989). Communicati on indices of crew coordination. Aviation, Space, & Environmental Medicine, 60 (1), 56-60. F mounted and scene-oriented video system s on remote collaboration on physical tasks. Paper presented at the Conference on Computer Human Interaction (CHI Ft. Lauderdale, Florida. Fu look? Gaze targets during collaborative physical tasks. Paper presented at the Conference on Co Gergle, D., Kraut, R. E., & Fussell, S. R. (2004a). Action as language in a shared visua space. Paper presented at the Conferen ce o D., Kraut, R. E., & Fussell, S. R. (2004b). Language effi ciency and visual technology: Minimizing collaborative effort with visual information. Journ Language and Social Psychology, 23 (4), 491-517. G distributed collaboration. In E. Salas & S. M. Fiore (Eds.), Team Cogniti Understanding the factors that drive process and performance (pp. 177-202). Washington, D.C.: American Psychological Association. H Organizational Behavior (pp. 315-342). Englewood Cliffs, NJ: Prentice-Hall. H distributed teams. Organization Science, 14(6), 615-632. (2002). Multilevel Analysis: Techniques and Applications Mahwah, NJ: Il From input-process-output models to IMOI models. Annual Review of Psychology, 56(1), 517-543. Jiang, X., Hong, J. I., Takayama, L. A., & Landay, J. A. (2004). Ubiquitous computing for firefighters: Field studies and protot ypes of large displays for incident command. Paper presented at the Conferen ce on Computer-Human Interaction 2004, Vienna, Austria. Jones, H., & Hinds, P. (2002). Extreme work groups: Using SWAT teams as a model for coordinating distributed robots. Paper presented at the CSCW 2002 Conference on Computer Supported Coope rative Work, New York.
116 Kiekel, m cognition through an alysis of communication da ta. In M. J. Smith, G. Salvendy, D. Harris & R. J. Koubek (Eds.), Usability evaluation and interface Kirkpa rograms: The four levels San Francisco: Berrett-Koehler. Klimoski, R., & Mohammed, S. (1994). Team mental model: Construct or metaphor? Journal of Management, 20 (2), 403-437. Kraiger Brannick, E. Salas & C. Prince (Eds.), Team performance assessment and h, NJ: Erlbaum. Krauss nicative effectiveness. In J. Galegher & R. E. Kraut (Eds.), Intellectual teamwork: Social Kraut, ation as a conversational resource in collaborative physical tasks. Human-Computer Interaction, 18 13-49. Kraut, formation in shared visual spa ces: Informing the deve lopment of virtual co-presence. Paper presented at the Conf erence on Computer Supported andis, J. R., & Koch, G. G. (1977). The measurement of observer agreement for MacMillan, J., Entin, E. E., & Serfaty, D. (2004). Communication overhead: The hidden cost of team cognition. In E. Salas & S. M. Fiore (Eds.), Team Cognition: arble, J. L., Bruemmer, D. J., & Few, D. A. (2003). Lessons learned from usability tests P. A., Cooke, N. J., Foltz, P., & Shope, S. M. (2001). Au tomating measurement of tea design (pp. 1382-1386). Mahwah, NJ: La wrence Erlbaum Associates trick, D. L. (1994). Evaluating training p K., & Wenzel, L. (1997). Conceptual development and empirical evaluation of measures of shared mental models as i ndicators of team effectiveness. In M. measurement: Theories, methods and applications (pp. 63-84). Mahwa R. M., & Fussell, S. R. (199 0). Mutual knowledge and commu and technological foundations of cooperative work. (pp. 111-145): Lawrence Erlbaum Associates, Inc. R. E., Fussell, S. R., & Siegel, J. (2003). Visual inform R. E., Gergle, D., & Fussell, S. R. (2002, November 16-20, 2002). The use of visual in Cooperative Work 2002, New Orleans, LA. L categorical data. Biometrics, 33 (1), 159-174. Understanding the Factors That Dr ive Process and Performance (pp. 61-82). Washington, D.C.: American Psychological Association. M with a collaborative cognitive workspace for human-robot teams. Paper presented at the IEEE International Conference on Systems, Man and Cybernetics.
117 Journal of Applied Psychology, 85 (2), 273-283. McGra rmance Englewood Cliffs, NJ: Prentice Hall. Morris, N. M., Rouse, W. B., & Zee, T. A. (1987). Adaptive aiding for human-computer control: An enhanced task environment fo r the study of human problem solving in Murph Noldus, L., Trienes, R., Hendriksen, A., Jans en, H., & Jansen, R. (2000). The Observer Video-Pro: New software for the colle ction, management, and presentation of search mputers, 32 197-206. lson, G. M., & Olson, J. S. (2003). Distance matters. Human Computer Interaction, 15 Orasan ). Princeton, NJ: Princet on University, Cognitive Sciences Laboratory. Orasan reness through communication. Paper presented at the International C onference on Experimental Analysis and Peterso 001). Controller-to-controller communication and coordination taxonomy (C T) (No. DOT/FAA/AM-01/19). Washington, D.C.: Marks, M. A., Mathieu, J. E., & Zaccaro, S. J. (2001). A temporally based framework and taxonomy of team processes. Academy of Management Review, 26 (3), 356376. Mathieu, J. E., Heffner, T. S., Goodwin, G. F., Salas, E., & Cannon-Bowers, J. A. (2000) The influence of shared mental mode ls on team process and performance. th, J. E. (1984). Groups: Interaction and Perfo complex situations (No. TR-87-011). Norcross, GA: Search Technology, Inc. y, R. (2000). Introduction to AI Robotics Cambridge, MA: MIT Press. Murphy, R. R., Blitch, J. G., & Casper, J. L. (2002). RoboCup/AAAI Urban Search and Rescue events: Reality and competition. AI Magazine 37-42. time-structured data from videot apes and digital media files. Behavior Re Methods, Instruments, and Co Nourbakhsh, I. R., Sycara, K., Koes, M., Yong, M., Lewis, M., & Burion, S. (2005). Human-robot teaming for search and rescue. Pervasive Computing, IEEE, 4 (1), 72-79. O 139-179. u, J. (1990). Shared mental models and crew decision making (No. Technical Report No. 46 u, J. M. (1995). Evaluating team situation aw a Measurement of Situation Awareness, Daytona Beach, FL. n, L., Bailey, L., & Willems, B. (24 Department of Transportation, Federal Aviation Administration, Office of Aerospace Medicine.
118 tic nges and results. Robotics and Autonomous Systems, 42 (3-4), 271-281. Rasbas .0 re for Multilevel Modelling, University of Bristol. achs, J. (2000). Using a small sample Q sort to identify item groups. Psychological Salas, E tanding of team performance and training. In R. W. Swezey & E. Salas (Eds.), Teams: Their training and performance. (pp. 3-29): Ablex Publishing. Scholtz awareness in search and rescue. Paper presented at the Robotics and Automation, 2004. Proceedings. ICRA 04. 2004 IEEE International Conference Sherida control Cambridge, MA: MIT Press. Smith-J d J. A. Cannon-Bowers & E. Salas (Eds.), Making decisions under stress (pp. 61-88). Washington, D.C.: American Psychological Association. Sonnenwald, D., & Pierce, L. (2000). Info rmation behavior in dynamic group work contexts: Interwoven situational awareness, dense social networks and contested collaboration in command and control. Information Processing and Management, 36, 461-479. Stevens, J. P. (2002). Applied Multivariate Statistics for the Social Sciences (4th ed.). Mahwah, NJ: Lawrence Erlbaum Associates. Thompson, L. F., & Coovert, M. D. (2003). T eamwork online: The effects of computer conferencing on perceived confusion, satisfaction and postdiscussion accuracy. Group Dynamics, 7(2), 135-151. Thrun, S. (2004). Toward a framew ork for human-robot interaction. Human-Computer Interaction, 19 (1-2), 9-24. Pineau, J., Montemerlo, M., Pollack, M., R oy, N., & Thrun, S. (2003). Towards robo assistants in nursing homes: Challe h, J., Steele, F., Browne W., & Prosser, B. (2005). A User's Gude to MLwiN 2 United Kingdom: Cent Robinson, W. S. (1950). Ecological correlations and the behavior of individuals. American Sociological Review, 15 351-357. S Reports, 86, 1287-1294. ., Dickinson, T. L., Converse, S. A., & Tannenbaum, S. I. (1992). Toward an unders J., Young, J., Drury, J. L., & Yanco, H. A. (2004). Evaluation of human-robot interaction on Robotics and Automation. n, T. (1992). Telerobotics, automation, and human supervisory entsch, K. A., Johnston, J. H., & Payne, S. C. (1998). Measuring team-relate expertise in complex environments. In
119 on Eye, A., & Mun, E. Y. (2005). Analyzing Rater Agreemen t: Manifest Variable Methods. Mahwah, NJ: Erlbaum. ald, A. (1943). Tests of sta tistical hypotheses concerning se veral parameters when the number of observations is large. Transactions of the Am erican Mathematical Society (54), 426-482. ellens, A. R. (1993). Group situation awaren ess and distributed d ecision making: From military to civilian applications. In N. J. Castellan (Ed.), Individual and group decision making: Current issues (pp. 267-291). Hillsdale, NJ: Erlbaum. est, M., Borrill, C., & Unsworth, K. (1998). T eam effectiveness in organizations. In C. L. Cooper & I. T. Robertson (Eds.), International Review of Industrial and Organizational Psychology (Vol. 13, pp. 1-48). Chichester, UK: John Wiley & Sons. ood, J., Lugg, D. J., Hysong, S. J., & Harm D. L. (1999). Psychological Changes In Hundred-Day Remote Antarctic Field Groups. Environment and Behavior, 31 (3), 299-337. oods, D. D., Tittle, J., Feil, M., & Ro esler, A. (2004). Envisioning human-robot coordination in future operation actions on Systems Man and Cybernetics Part C-Applications and Reviews, 34 (2), 210-218. Yanco, H. A., Drury, J. L., & Scholtz, J. ( 2004). Beyond usability evaluation: Analysis of human-robot interaction at a major robotics competition. Human-Computer Interaction, 19 (1-2), 117-149. V W W W W W s. Ieee Trans
121 Appendix A Task Scenario Instructions sk the team if this is their first run or second, follow script 1 or 2 accordingly. cript for Search Task Scenario #1 ear ch this void using the robot, looking for victims or signs of ictims (clothing, etc.). We want you use the LOVR technique you learned in the ,SV : You will both have access to the robots eye view, but you cannot look at each you. ts eye view. voidspace you search. You can do it any way you antas you search, or when you finish. We ar e recording this on video, so please speak ell s the task and comple tes the map, thank them for participating and ll them where to go next (either th e next station or back to the BoO). A S Begin by introducing yourself and the others on your research team. Verbally confirm that each participant has signed both informed consent and video consent forms. Ask the team to decide who will act as robot operator and tether-handler on this run, explaining that they will exchange roles for the second s cenario. Ask if they would like to review the Operator Control Unit before continuing. Then say: Your teams task is to s v awareness training as you work, so talk to your team mate as you search. Depending on the CONDITION, say: R otherpretend there is a wall between R, Not: You cannot look at each other-pretend there is a wall between you. C, SV: You will both have access to the robo C, Not: ---(nothing extra to say) Wed like you to draw a map of the w clearly and loud enough for us to hear. Youll ha ve 15 minutes to search this void. Ill t you when youve been in 10 minutes, and when 15 minutes have passed, Ill ask you to being the robot back to the start point. If you have questi ons about how to operate the robot I can answer those for you, but I cant an swer any questions about the search task or help you with that. Do you have any questions before we start? When the team finishe te
122 rio #2 yourself and the others on your research team. Verbally confirm at each participant has signed both informed consent and video consent forms. Ask the team robot operato r and tether-handler on the first run, explaining that they wilroles for the second scenario. Al so ask them whether they both had the SV in the first run, and if they were told not to l ook at each other. Be sure to not to administer the samtions on this run! (You s hould have this information already, but doublecheck). Ask if they would like to revi ew the Operator Control Unit before ontinuing. Then say: our teams task is to sear ch this void using the robot, looking for victims or signs of vict e the LOVR technique you learned in the awareness training as you work, so talk to your team mate as you search. ,SV : You will both have access to the robots eye view, but you cannot look at each otherpretend there is a wall between you. R, Not: ot look at each other-pretend there is a wall between you. C, S You will both have access to the robots eye view. C, Not: extra to say) ed like you to draw a map of the voidspace you search. You can do it any way you speak learly and loud enough for us to hear. Youll ha ve 15 minutes to search this void. Ill tell you when youve been in 10 minutes, and when 15 minutes have passed, Ill ask you to being the robot back to the start point. If you have questi ons about how to operate the robot I can answer those for you, but I cant an swer any questions about the search task or help you with that. hen the team finishes the task and comple tes the map, thank them for participating and ey). Appendix A (Continued) Script for Search Task Scena Begin by introducing th who acted as l exchange e condi c Y ims (clothing, etc.). We want you us Depending on the CONDITION, say: R You cann V: ---(nothing W wantas you search, or when you finish. We ar e recording this on video, so please c W tell them where to go next (back to th e BoO to complete the posttest surv
123 Appendix B: Dem lease answer the following questions. All information is confidential, and will be used 1. W e? 4. 45-54 5. 55 and up Please indicate your gender: How many years experience do you have on the job as a firefighter? 1. 0-3 years 3. 8-11 years 5. over 15 years 4. How many years experience do you have on the job as a USAR team member? 1. 0-3 years 2. 4-7 years 3. 8-11 years 5. over 15 years 5. Dave any military experience? (If answer is no, go to #8.) 2. No so, what branch of the military? 1. Air Force 3. Marines 4. Navy ographi c Survey Questionnaire P for research purposes only. hat is your ag 1. Under 25 2. 25-34 3. 35-44 2 1. M 2. F 3 2. 4-7 years 4. 12-15 years 4. 12-15 years o you h 1. Yes 6. If 2. Army
124 Appendix B (Continued) 7. How many years of military experience? 1. 0-3 years 4-7 years 3. 8-11 years 4. 12-15 years 5. over 15 years In the nex sk the scale below: 1 2 3 4 5 very little som 8. Remote-controlled cars, planes, boats 9. Handheld or joy-stick type vid eras r Have yover operated a robot before? 1. 2. No If pl experience? If so, please describe. 2. t section, p lease rate your d egr ee of ill w it h t he fol low in g it em s a cc ord ing to e m oderate above average expert eo gam es 10. Video cam 11. Search-cam 12. 13. 14. Do you have any other relevant s o r o the tech ni cal re scu e e qu ipm en t u e Yes so, eas e d es crib e (wh at kin d of robot, where, when, why, etc.)
Appendix C Complete Co rrelation Table for Study Variables M2 3 4 5 6 7 8 9 10 1 13 Variable SD 1 1 12 14 15 16 1. Performan ce 4.35 3.06 1 438 0.339 2.4303 -0. 0.0040.9 730.1 -0.2 0.7920.2 0.04 8590.1 0.13 0. 0.0100.4 0.34 0. 16250.1 0.06 0. 0.8620.4 0.64 0. 11.1 -0.1 0. 0.0.0. 3.0240.1 -0.0 0. 0.0. 2.86 00.1-0.20. ** 04* 0.0.130 04 3.18640.0.* *6*.196 0.0. .172 3.08430.0. **2*6000.7 0.0. 0 2.99860.-0.00. 6*. 1 0.0. 37. 80.-0.0. **46*5970.20. 0.0800 0.0.0 0 3.10600.52-0 48 0. 0.20 0530 0. 2. MapScore 2.5 0.96 0.1 1 3. Com1 1.20 0.4 ** 011 1 41 4. Com2 2.9 1.26 -0.0 8 57 87* 1 76 4 5. ComTotal 3.1 0.77 0.3 01 5 195 1 85 9 175 6. Supp1 3. 0.58 0.0 16 7 249 0.264 1 22 4 082 0.064 7. Supp2 2.4 1.05 -0.1 4 0 71 47 426** -0.005 0.416* 1 430 0. 234 31 002 0.974 0.003 8. SuppTotal 0.71 -0.0 1 11 58 317* 0.310* 0.641* 0.630 1 779 0. 444 691 0. 025 0.029 0 0 9. Ldr1 1.14 -0.1 0 73 15 640** 0.297* 0.279* 0.500 0.4 1 492 0. 229 4 0.036 0.05 0 0.0 10. Ldr2 0.66 0.0 198 157 0. 007 0.377* 0.373* 0.245 0.57 0 1 658 0. 168 275 0. 964 0.007 0.008 0.086 0 0 11. LdrTotal 0.64 0.0 152 034 0. 248 0.353* 0.305* 0.481 0.64 0. ** 36** 1 769 0. 291 816 0. 082 0.012 0.031 0 0 0 12. SA1 0.77 0.0 221 5 366** 0.347* 0.23 0.156 0.29 0 333* 0. 324* 0. 362** 554 0. 123 729 0. 009 0.014 0.109 0.278 0.0 0 018 0. 022 0. 01 13. SA2 2.86 1.11 -0.10. 4 200 0. 179 214 245 6 756** 0.162 .26 0.383*0.006 0.7190 0.50 0.0 ** 58 07 447**01 0.297*.036 1 14. SA3 0.80 0.20. 068 0. 287* 0.043 0 0** .153 88 0.128.377 0.325*0.021 0.2360.1 0.10.3 0. 0. 082 0.569 0. 158 0.273 0. 133 58 0.249.082 062668 1 p=0.05, ** p=0.01 125
Appendix C (continued) Variable SD 1 2 4 5 7 8 9 M 3 6 10 11 12 13 14 15 16 15. SATotal 3.08 0.0.0.320.220.100.30.22.214.171.124.50.4.46260.5 63 054 7* 4 6 94** 25** 6 465** 330* 08** 79** 0 6** 0. 7 94** 1 0.0.0.0.0.0.0.000.000.060 0.-0-0.10.17-0.-0.2-0.3-0.0. -0-0.-0.1-0.1-0.4-0.-0.06 0.0.0.0.0.0.0.262 0.0.0.0.0.67her 0.0.0.260.010.020.30.00.20.0.0.30.30.38220.2.460.0 0.0.0.0.0.60.0.0.0.0.0.0.000.her-Operator 0.-0-0.10.390.0.0-0.2-0.-0.-0-0.-0.-0.20.1-0.0.060. 0.0.0.0.0.0.0.0.0.0.0.0.660her-Tether 0.-0-0.0-0.0-0.00.3-0.1-0.-0.-0-0.-0.2-0.1-0.1-0.0.080. 0.0.5126.96.36.199.0.0.070.0.0.560.perator 0.0.-0.1-0.10.060.10.0-0.-0.-0-0.-0.2-0.20.1-0.-0.-0. 0.0.0.0.0.0.0.0.0.070.0.000.esearcher 0.-0-0.1-0.00.1-0.3-0.1-0.-0.-0-0.-0.2-0.10.0-0.0.120. 0.0.0.0.0.0.0.0.0.460.0.400.0.-00.07-0.0-0.00.3-0.1-0.-0.-0-0.-0.00.19-0.00.0-0.0.1 0.0.0.0.0.0.0.0.188.8.131.52.910. 0. 0.0.-0.0.000.0.0.0.0.170.-0. 0.0.0.0.0.10.30.0.0.210.0.040. 0.0.-0.-0.00.120.00.1-0.-0. 0.-0.0.-0.2-0.-0.0.-0. 0.0.0.0.0.0.0.497 0.0.0.100.0.040.n 0.-0-0.00.23-0.0.2-0.-0.-0.-0-0.-0.-0.1-0.10.-0.080.4 0.510.0.0.00.0.0.0.0.240.240.420.60.560. 0.-00.20-0.0.100.1-0.-0.0.0.00.0.15110.183 0.17-0. 0.0.0.0.0.0.0.0.0.280.0.204 0.230.mation Synthesis 0.-00.22-0.20.04-0.10.00.10.0.0.0-0.00.080.130.00.08-0. 0.0.0.0.0.0.0.0.0.530.0.570.22 0.0 -045 -0.379** 0.-0.-0.0.0.0.-0098 -0.-0.-0.-0.00. -0.229 -0.216 0.754 0.007 0.05 0.524 0.872 0.173 0.371 0.538 0.5 0.973 0.764 0.106 0.837 0.658 0.11 0.132 707 0. 021 117 462 0. 005 002 068 001 019 1 1 16. Operator-Researcher 0.02 03 .025 11 6 348* 3 38* 351* 162 .303* 132 33 94 39** 037 1 1 864 441 221 0. 013 108 016 012 033 36 359 0. 176 0. 001 799 2 17. Operator-Tet 0.47 13 049 8 7 1 51* 7 11 287* 215 11* 82** 0** 0. 8 57 0 7** 16 734 059 907 0. 884 013 3 141 043 134 028 006 0. 006 0. 112 072 1 91 18. Researc 0.03 02 .005 16 5** 002 84 3 264 188 .22 258 208 99* 12 079 3 543* 975 421 005 0. 988 564 108 064 19 124 07 148 0. 035 0. 439 587 4 19. Researc 0.01 02 .094 87 78 46 37* 47 188 142 .081 184 58 13 38 057 3 067 515 588 753 017 308 0. 192 326 0. 574 201 0. 436 338 696 0. 6 643 20. Tether-O 0.45 13 025 78 06 5 86 97 067 189 .095 19 57 55 07 221 417** 333* 861 217 465 653 195 503 0. 646 188 0. 511 186 0. 071 4 461 123 0. 3 018 21. Tether-R 0.02 02 .279* 28 09 09 70** 98 106 127 .13 174 25 06 6 066 1 124 049 377 953 453 008 169 0. 464 379 0. 369 226 0. 116 2 679 649 0. 1 391 22. Answer 0.16 06 .261 2 01 73 25* 42 078 206 .109 056 94 7 3 99 015 7 067 621 993 613 021 326 0. 589 152 0. 45 516 838 494 0. 9 238 23. Comment 0.53 13 036 195 071 9 229 125 0. 149 255 0. 081 228 0. 276 9 0. 199 099 0. 287* 106 805 175 626 953 1 389 0. 074 0. 577 112 0. 052 5 165 495 0. 3 464 24. Instruction 0.15 11 230 217 3 9 6 06 125 098 036 124 186 35 154 207 289* 18 108 131 837 374 681 462 0. 387 803 392 0. 195 1 287 149 0. 2 212 25. Questio 0.17 05 .244 95 2 186 64 329* 003 146 .157 209 169 67 16 058 3 25** 087 0. 105 195 0. 064 2 984 312 277 144 6 2 9 5 002 26. Environment 0.12 07 .080 9 290* 7 23 017 0. 013 055 261 25 046 4 0. 8 199 580 144 041 461 393 909 0. 926 703 0. 067 863 0. 752 6 413 8 165 27. Infor 0.01 02 .135 8 4 9 33 41 32 102 064 56 14 9 2 35 2 05 350 111 093 734 358 778 0. 361 479 0. 658 702 0. 922 7 362 811 0. 3 729 28. Navigation 0. 1 .0 279* 092 023 196 129 089 005 044 232 3 064 p=0.05, ** p=0.01 126
Appendix C (continued) Variable M SD 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 29. Offtask 0.07 .04 -8 1 -0 -0.3-0.1-0.07 -0.033 .284* 0.332* -0.327-0.1-0-0 043 355* 0 0.20 -0.23 0.03 .213 06* 84 -0 69 .16 .075 -0. 0. 0.148 0.106 0.838 0.137 031 202 0.631 0.822 0.0.0.021 0.1 .266 0.603 0.0. Robot Situatedness 0.0.05 -0.257 0.113 0.049 0.025 .052 -058 -0.3 0 -0.089 0.058 0.0.2 .067 0.03 0.112 0.355* 0.0.435 0.735 0.866 719 688 0.358 1 0.0.847 0.2 .644 0.836 0.0.1. Robot State 0.0.09 -0.110 0.043 -0.179 0.149 .002 -225 0.-0.0.071 0.001 0.-093 .118 0.440** -0.132 0.092 0.7690.213 0.301 99 116 0.879 0.0.0.0.676 0.8 .413 0.001 0.2. Search 0.0.05 0.429*-0.152 0.087 -.052 173 0.092 0.093 0.0.0.024 0.6* .002 0.126 0.0.145 0.002 0.291 0.55 0.72 231 0.527 0.52 0.0.0.87 0.03 991 0.384 0.0.33. Victim 0.07 0.06 053 0.266 -0.182 0.238 035 -0.216 -0.083 -0.0.0.0.18 .253 0.229 0.0. 0 0.0620.207 096 811 0.132 0.568 0.0.0.407 0. .076 0.11 0.23 34. Clarify 0.0.036 0.043 -0.11 -0.063 -.334* -138 -0.5 -0.156 -0.-0.-0.113 -011 .048 0.034 -0.0.143 68 9 0.664018 338 0.511 0.28 0.0.435 0.4 .741 0.816 0.0. 0.63 7 03 .172 -048 -0.2 -0.091 -0.0.0.035 -065 .041 0.068 -0.04 8 4 .234 742 0.722 0.0.0.0.81 0.3 .776 0.64 0.certainty 0.03 7 41 65 --.078 .0.1060.053 0.125 0.205 0.27 0.041 0.081 0.21 0.381* 3 52 0591 0.753 0.465 0.717 0.386 0.154 0.79 .777 0.576 0.0. rror 0.0.01 53 -00.23 41 0.116 0.0.155 -0046 .159 -0.033 0.001 6 0.055 0.092 0.0.0.282 0.4 .27 0.822 0.ation 0.06 6 0.15 0.-0.0.001 0.6 .202 0.006 0.0.033 7 7 0.0.0.0.994 0.3 .16 0.964 0.0.0.11 08 -0.0.000.04 0.0.107 0.5 .002 0.095 0.0. 4 7 0.0.0.459 0.3 .991 0.511 0. 0. 8 7 5* -0.1-0.0220.214 1-004 .005 0 -0.106 0. 1 5 0.0.0.0.448 0.5 .974 1 0.0.0.11 -0.0.0680440.116 02 .131 0.091 -0.0. 0.981 0.709 0.637.7590.423 0.1 .365 0.529 0. 0. 0. 046 018 24 0 766 0. 011 3 11 -0 0. 13 028 17 0 072 0. 0. 538 0. 691 23 0 439 011 3 30 -0 0. 022 046 061 .1 0 0.447 0. 0. 75 622 993 17 0 361 0. 527 3 10 0.209 0 0. 121 101 30 0 173 238 0. 0. 401 486 0. 229 096 0.541* -0. 0. 121 062 12 -0 155 173 0.00 0.717 0. 0. 402 671 21 0 281 0. 04 -0.21 0 0. 09 086 017 .1 -0 08 0.131 0.7 0.44 0. 0. 553 0. 905 44 0 582 321 35. Confirm 18 0.0 -0.20 0.0 20.85 -0.2 0.034 0.813 -0 0. 05 063 036 .0 0 075 0. 0.1 5-0.20 0.158 0 0. 53 662 805 65 0 605 0. 783 36. Convey Un 3 0.0 0.1 0. 00.6 0.0240.87 0.204 0.155 0 0 046 1 0.149 0.3 .. 3 0 143 006 37. CorrectE 1 0 -0.26 08 0.2 -0.1 2683 -0.02 6 0. 16 13 7 0.2 144 0 148 0. 0.06 0.07 0.3 0.858 0.268 0.369 423 318 75 0 306 0. 996 38. Provide Inform 10 0. 0.068 0.23 -0.09 0.171 -0.232 0.001 238 0. 104 188 17 0 012 0.639 0.10 0.50 0.235 0.105 0.997 096 3 471 0. 191 22 0 933 823 39. Report 30 0. -0.018 -0.1 0.128 -0.174 0.281* 0.008 095 1 09 05 0 102 055 0.900 0.45 0.37 0.227 0.048 0.954 511 0. 992 0. 781 535 70 0 479 0. 704 40. Seek Information 11 0.05 -0.16 -0.21 0.28 -0.112 -0.147 0.207 08 37 -0.1 -0. 1 .2 -0 249 0.244 0.13 0.04 0.44 0.308 0.15 455 796 401 0. 136 15 0 463 082 41. Plan 21 0. 0.308* 0.004 -0.054 0.145 0.141 0.157 079 028 0. 0. .0 -0 072 272 0.030 0.315 0.327 0.276 585 0. 847 0. 0 89 0 621 0. 056 p=0.05, ** p=0.01 127
Appendix C (continued) Vari ables 17 24 25 26 27 28 29 30 35 18 19 21 22 23 31 32 33 34 17. Operator-Tether 1 perator 1 2 her-Tether 22*5 2 Operator 0*-0.290*51 0. 21. Tether-Researcher 07*09 855 22. Answer 041 .061 .01 0.066 80.64 3. Comment 8**.24-0.1 -0.5 0.0 4. Instruction 71* .227-0.0 -0. -01 1120.220 22290*2210.4380.39-0. -0 1 88 0.0.091 0..0.0.0370.0.0. tion Synthesis 74 .218 049 -0.00.3-0-1 91290.0. 1* 0.0202-0.-00.* 01.3330.0 5 29. Offtask 67 306*5440.6600.03-0.-0217 1 0 0.0.077 18. Researcher-O 0.00 1 0.99 19. Researc -0.4 -0.03 1 0.00 0. 811 20. Tether-0.900 0.1041 0. 294 -0.3 0. 0. ** 1 0. 03 0. 536 0 -0. -0 -0 1 0.77 0. 672 0. 943 7 2 0.60 0. 11 -0 7 84 11** 1 0 0. 446 0. 084 2 2 -0.60 *-00. 0. 0. 176 222 4 1755 .669** 25. Question 782 0.** -0.0 0. 0. 3** 408** .241 0. 0. 041 0. 124 001 0. 005 0. 003 26. Environment 0. 296* 205 -0 057 159 017 0. 243 0.258 0.059 1 0. 0. 153 694 269 0. 907 0. 089 07 0.682 27. Informa -0.0 -0 0. 01 00* .089 0.061 -0.03 0.078 0.60 0. 0. 737 996 0. 034 0. 538 675 0.838 0.589 28. Navigation -0.47 -0.14 8 0. 8 073 .440** 544** 0.01 -0.451* -0.03 1 0.0-0.0 00. 0. 0. 848 ** 845 0.** 612 0.3 001 021 0.9470.455* 0.001-0.253 0.83-0.146 -0.113 0. 646 0. 031 0 821 0. 885 13 0.001 0. 0.311 0.436 p=0.05, ** p=0.01 128
129 Appendix C (Continued) Variables 17 18 19 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 30. Robot Situatedness 0.165 0.282* 0.14 0.345* 0.077 0.279 -0.449** 0.2 -0.227 -0.199 -0.296* 0.467** 1 0.252 0.047 0.333 0.014 0.597 0.05 0.001 0.163 0.113 0.167 0.037 0.001 31. Robot State -0.03 0.278 -0.1 -0.099 -0.133 0.084 0 -0.019 -0.199 -0.074 -0.303* -0.117 -0.05 1 0.839 0.05 0.488 0.494 0.359 0.562 1 0.898 0.167 0.608 0.033 0.418 0.718 32. Search 0.103 -0.368** 0.075 -0.053 0.094 -0.023 0.064 -0.181 0.233 0.333* -0.24 -0.173 -0.19 -0.348* 1 0.477 0.009 0.606 0.717 0.518 0.873 0.661 0.208 0.104 0.018 0.093 0.231 0.185 0.013 33. Victim 0.341* 0.01 -0.288* -0.378** 0.082 0.178 -0.16 -0.208 0.007 -0.251 -0.194 -0.183 -0.06 -0.271 -0.036 1 0.015 0.947 0.043 0.007 0.571 0.216 0.268 0.147 0.963 0.079 0.178 0.203 0.661 0.057 0.804 34. Clarify 0.016 -0.025 0.285* 0.167 0.465** -0.246 -0.03 0.118 0.105 0.199 -0.143 0.136 0.069 -0.088 0.167 -0.13 1 0.913 0.864 0.045 0.248 0.001 0.085 0.837 0.415 0.466 0.165 0.323 0.345 0.632 0.545 0.246 0.364 35. Confirm 0.126 -0.043 -0.139 -0.155 0.096 0.295* -0.313* -0.173 0.404** 0.088 -0.307* -0.071 0.235 -0.15 0.11 -0.02 0.301* 1 0.384 0.766 0.336 0.284 0.506 0.038 0.027 0.23 0.004 0.543 0.03 0.625 0.1 0.3 0.445 0.896 0.033 36. Convey Uncertainty 0.133 0.096 -0.103 -0.176 0.533** -0.158 -0.247 0.249 -0.035 0.247 -0.227 0.049 0.169 -0.038 0.113 0.135 0.330* 0.2 0.356 0.509 0.477 0.221 0 0.273 0.084 0.081 0.811 0.083 0.113 0.736 0.242 0.792 0.433 0.349 0.019 0.164 37. Correct Error 0.142 -0.044 0.028 0.08 0.376** -0.133 -0.128 0.134 -0.046 0.408** -0.093 0.128 -0.08 -0.017 0.322* -0.17 0.347* -0.026 0.324 0.762 0.845 0.581 0.007 0.358 0.374 0.354 0.753 0.003 0.52 0.377 0.582 0.908 0.023 0.232 0.014 0.858 38. Provide Information 0.018 0.05 -0.05 -0.006 0.057 0.167 -0.154 -0.149 -0.242 0.376** -0.087 0.053 0.014 0.028 0.360* -0.07 -0.13 -0.068 0.9 0.732 0.73 0.969 0.696 0.247 0.286 0.303 0.091 0.007 0.548 0.716 0.925 0.849 0.01 0.627 0.367 0.64 39. Report 0.221 0.261 -0.109 0.037 -0.261 0.460** -0.414** 0.049 0.076 -0.449** -0.151 0.082 0.254 0.239 -0.550** 0.188 -0.448** -0.252 0.124 0.067 0.452 0.799 0.067 0.001 0.003 0.734 0.601 0.001 0.297 0.57 0.075 0.094 0 0.19 0.001 0.078 40. Seek Information -0.008 0.374** 0.18 0.469** -0.056 -0.138 -0.176 0.756** -0.259 -0.273 0.166 0.442** 0.17 0.163 -0.391** -0.21 -0.277 -0.389** 0.958 0.007 0.211 0.001 0.699 0.34 0.222 0 0.07 0.055 0.248 0.001 0.239 0.259 0.005 0.15 0.052 0.005 41. Plan -0.346* -0.414** 0.068 -0.18 -0.077 -0.523** 0.810** -0.313* -0.072 0.157 0.375** -0.329* -0.500** -0.198 0.366** -0.02 0.114 -0.185 0.014 0.003 0.638 0.21 0.597 0 0 0.027 0.619 0.275 0.007 0.02 0 0.168 0.009 0.877 0.432 0.199 p=0. 05, ** p=0.01
130 Appendix C (continued) Variables 36 37 38 39 40 41 36. Convey Uncertainty 1 37. Correct Error 0.545** 1 0 38. Provide Information 0.073 0.196 1 0.614 0.172 39. Report -0.339* -0.362** -0.262 1 0.016 0.01 0.066 40. Seek Information -0.219 -0.172 -0.075 0.354* 1 0.127 0.231 0.605 0.012 41. Plan -0.076 0.015 -0.196 -0.644** -0.407** 1 0.598 0.919 0.173 0 0.003 p=0.05, ** p=0.01
131 About the Author Jennifer Burke received a Bachelors Degree in Business from Florida State University in 1982, an M.A. in Counseling from the University of North Florida in 1990, and a M.A. in Industrial-Organizational Psychology from the Un iversity of South Florida in 2003. She taught mathematics in Florida public schools for 14 years. Since entering the Ph.D. program at the Univ ersity of South Florida, Jenny has taught courses in psychology and conducted training for technologists and first responders through the Center for Robot-Assisted Search and Rescue. She has co-authored two journal articles, a book chapter, and numerous technical reports, a nd has presented her research at national and international conferences. Ms. Burke is a member of the Association for Computing Machinery (ACM), the Human Factors and Ergonomics Society (HFES), the So ciety for Industrial and Organizational Psychology (SIOP), and the American Psychological Society (APS).