USF Libraries
USF Digital Collections

An inconsistency-based approach for sensing assessment in unknown environments

MISSING IMAGE

Material Information

Title:
An inconsistency-based approach for sensing assessment in unknown environments
Physical Description:
Book
Language:
English
Creator:
Gage, Jennifer Diane
Publisher:
University of South Florida
Place of Publication:
Tampa, Fla
Publication Date:

Subjects

Subjects / Keywords:
Uncertainty handling
Robotics
Dempster-Shafer
Sensing metrics
Sensor suitability
Dissertations, Academic -- Computer Science and Engineering -- Doctoral -- USF   ( lcsh )
Genre:
non-fiction   ( marcgt )

Notes

Summary:
ABSTRACT: While exploring an unknown environment, an intelligent agent has only its sensors to guide its actions. Each sensor's ability to provide accurate information depends on the environment's characteristics. If the agent does not know these characteristics, how can it determine which sensors to rely on? This problem is exacerbated by sensing anomalies: cases where sensor(s) are working but the readings lead to an incorrect interpretation of the environment, e.g. laser sensors cannot detect glass. This work addresses the following research question: Can an inconsistency-based sensing accuracy indicator, which relies solely on fused sensor readings, be used to detect and characterize sensing anomalies in unknown environments? A novel inconsistency-based approach was investigated for sensing anomaly detection and characterization by a mobile robot using range sensing for mapping.Based on the hypothesis that sensing anomalies manifest as inconsistent sensor readings, the approach employed Dempster-Shafer theory and six metrics from the evidential literature to measure the magnitude of inconsistency. These were applied directly to fused sensor data with a threshold, forming an indicator, used to distinguish minor noise from anomalous readings. Experiments with real sensor data from four indoor and two outdoor environments showed that three of the six evidential inconsistency metrics can partially address the issue of noticing sensing anomalies in unknown environments. Polaroid sonar sensors, SICK laser range finders, and a Canesta range camera were used. Despite extensive training in known environments, the indicators could not reliably detect sensing anomalies, i.e. distinguish them from ordinary noise. However, sensing accuracy could be estimated (correlations with sensor error exceeded 0.8) and regions with suspect readings could be isolated.Trained indicators failed to rank sensors, but improved map quality by resetting suspect regions (up to 57.65%) or guiding sensor selection (up to 75.86%). This work contributes to the robotics and uncertainty in artificial intelligence communities by establishing the use of evidential metrics for adapting a single sensor or identifying the most accurate sensor to optimize the sensing accuracy in unknown environments. Future applications could enable intelligent systems to switch information sources to optimize mission performance and identify the reliability of sources for different environments.
Thesis:
Dissertation (Ph.D.)--University of South Florida, 2009.
Bibliography:
Includes bibliographical references.
System Details:
Mode of access: World Wide Web.
System Details:
System requirements: World Wide Web browser and PDF reader.
Statement of Responsibility:
by Jennifer Diane Gage.
General Note:
Title from PDF of title page.
General Note:
Document formatted into pages; contains 253 pages.
General Note:
Includes vita.

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
aleph - 002063191
oclc - 555793412
usfldc doi - E14-SFE0002989
usfldc handle - e14.2989
System ID:
SFS0027306:00001


This item is only available as the following downloads:


Full Text
xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam 2200409Ka 4500
controlfield tag 001 002063191
005 20100316091318.0
007 cr mnu|||uuuuu
008 100316s2009 flu s 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0002989
035
(OCoLC)555793412
040
FHM
c FHM
049
FHMM
090
TK7885 (Online)
1 100
Gage, Jennifer Diane.
3 245
An inconsistency-based approach for sensing assessment in unknown environments
h [electronic resource] /
by Jennifer Diane Gage.
260
[Tampa, Fla] :
b University of South Florida,
2009.
500
Title from PDF of title page.
Document formatted into pages; contains 253 pages.
Includes vita.
502
Dissertation (Ph.D.)--University of South Florida, 2009.
504
Includes bibliographical references.
516
Text (Electronic dissertation) in PDF format.
520
ABSTRACT: While exploring an unknown environment, an intelligent agent has only its sensors to guide its actions. Each sensor's ability to provide accurate information depends on the environment's characteristics. If the agent does not know these characteristics, how can it determine which sensors to rely on? This problem is exacerbated by sensing anomalies: cases where sensor(s) are working but the readings lead to an incorrect interpretation of the environment, e.g. laser sensors cannot detect glass. This work addresses the following research question: Can an inconsistency-based sensing accuracy indicator, which relies solely on fused sensor readings, be used to detect and characterize sensing anomalies in unknown environments? A novel inconsistency-based approach was investigated for sensing anomaly detection and characterization by a mobile robot using range sensing for mapping.Based on the hypothesis that sensing anomalies manifest as inconsistent sensor readings, the approach employed Dempster-Shafer theory and six metrics from the evidential literature to measure the magnitude of inconsistency. These were applied directly to fused sensor data with a threshold, forming an indicator, used to distinguish minor noise from anomalous readings. Experiments with real sensor data from four indoor and two outdoor environments showed that three of the six evidential inconsistency metrics can partially address the issue of noticing sensing anomalies in unknown environments. Polaroid sonar sensors, SICK laser range finders, and a Canesta range camera were used. Despite extensive training in known environments, the indicators could not reliably detect sensing anomalies, i.e. distinguish them from ordinary noise. However, sensing accuracy could be estimated (correlations with sensor error exceeded 0.8) and regions with suspect readings could be isolated.Trained indicators failed to rank sensors, but improved map quality by resetting suspect regions (up to 57.65%) or guiding sensor selection (up to 75.86%). This work contributes to the robotics and uncertainty in artificial intelligence communities by establishing the use of evidential metrics for adapting a single sensor or identifying the most accurate sensor to optimize the sensing accuracy in unknown environments. Future applications could enable intelligent systems to switch information sources to optimize mission performance and identify the reliability of sources for different environments.
538
Mode of access: World Wide Web.
System requirements: World Wide Web browser and PDF reader.
590
Co-advisor: Robin R. Murphy, Ph.D.
Co-advisor: Lawrence Hall, Ph.D.
653
Uncertainty handling
Robotics
Dempster-Shafer
Sensing metrics
Sensor suitability
0 690
Dissertations, Academic
z USF
x Computer Science and Engineering
Doctoral.
773
t USF Electronic Theses and Dissertations.
4 856
u http://digital.lib.usf.edu/?e14.2989



PAGE 1

AnInconsistency-basedApproachforSensingAssessmentinUnknownEnvironments by JenniferDianeGage Adissertationsubmittedinpartialfulfllment oftherequirementsforthedegreeof DoctorofPhilosophy DepartmentofComputerScienceandEngineering CollegeofEngineering UniversityofSouthFlorida Co-MajorProfessor:RobinR.Murphy,Ph.D. Co-MajorProfessor:LawrenceHall,Ph.D. GitaSukthankar,Ph.D. RahulTripathi,Ph.D. ZenonMedina-Cetina,Ph.D. DateofApproval: June18,2009 Keywords:uncertaintyhandling,robotics,Dempster-Shafer,sensingmetrics,sensor suitability c r Copyright2009,JenniferDianeGage

PAGE 2

DedicationTomyparentsDianaandMartin,husbandAaron,andsonTristanfortheirlovingpatienceandsupport.

PAGE 3

AcknowledgmentsEarlierstagesofthisresearchwerefundedbyONRGrantN00773-99PI543,DOEGrantDE-FA02-01ER45904,andSAIC.TheauthorwouldliketothankDr.RobinMurphyforherguidanceandsupportthroughoutthedevelopmentofthisworkandDr.LarryHallforhisinsightfulsugges-tionsforimprovingthisdocument.TheauthorwouldalsoliketothankDr.GitaSukthankarforherhelpfulsuggestions,includingtheideaofusingaboosting-stylelearningapproachtoautomat-icallyndsolutionsfornewlogicalsensors.ThanksalsotoDrs.ZenonMedina-CetinaandRahulTripathiforservingonmycommittee.TheauthorwouldliketothankDr.ClintKellyforhisen-couragementintheearlystagesofthisresearch.ThankstotheotherstudentsinDr.Murphy'sAIRoboticslabwhoneverhesitatedtohelpandprovidedanabundanceofusefulfeedbackalongtheway.Itwasapleasuretosharethisjourneywithyou.

PAGE 4

NotetoReaderTheoriginalofthisdocumentcontainscolorthatisnecessaryforunderstandingthedata.TheoriginaldissertationisonlewiththeUSFlibraryinTampa,Florida.

PAGE 5

TableofContentsListofTablesvListofFiguresviiAbstractixChapter1Introduction11.1ResearchQuestion11.2Denitions21.3Motivation:SensingAnomalies31.3.1CommonnessofSensingAnomaliesforExteroceptiveSensors41.3.2SensingAnomalyDetectionandCharacterization41.4Contributions61.5OrganizationofDissertation7Chapter2RelatedLiterature92.1SourceorSensingAssessment142.1.1EstimatingSensingAccuracy152.1.2EstimatingSourceReliability172.1.3RankSourcesbyReliability302.1.4Discussion312.2SensorFaultDetectionandIdentication332.2.1BriefHistoryofFDI342.2.2Inconsistency-basedFDI362.2.3Consensus-basedFDI372.2.4LearnedModelsUninformed402.2.5QualitativeandStochasticModels442.2.6LearnedModelsInformed482.2.7AnalyticalModels492.2.8Discussion522.3IsolatingPoorlySensedRegions532.4AdaptiveFusion552.5InconsistencyMetrics622.5.1Background642.5.2Conict-BasedMetrics652.5.3DegreeofInconsistencyMetrics662.5.4CombinedMetrics68i

PAGE 6

2.6Conclusions69Chapter3Approach723.1IdentifyingSensorSuitabilityinUnknownEnvironments733.2TheoreticalApproach773.2.1Consistency783.2.2SourcesofInconsistency783.3Inconsistency-basedSensingAccuracyAssessment793.3.1DetectingandCharacterizingSensingAnomalies793.3.2InconsistencyMetrics803.3.3Complexity823.4TargetApplication:RangeSensorAnomalies833.4.12DMapsfromRangeReadings833.4.2ImplementationofIndicators853.5Summary85Chapter4FeasibilityStudy874.1Testbeds884.2DataCollectionandRepresentation894.3AnalysisMethodsandMetrics914.3.1SensingAccuracyIndicators914.3.2ObjectiveSensingAccuracyAssessment924.3.3QuantifyingIndicatorPerformance934.4Results944.5Conclusions100Chapter5FurtherExperiments1025.1Testbeds1035.2DataCollectionandRepresentation1075.3TrainingPhase1105.3.1SensingAccuracyIndicators1115.3.2GroundTruthSensingAccuracyAssessment1125.3.3StatisticalAnalysis1145.3.4DetectionofSensingAnomalies1155.3.4.1Method1155.3.4.2Results1165.3.5EstimationofSensingAccuracy1185.3.5.1Method1195.3.5.2Results1195.3.6IsolationofPoorlySensedRegions1205.3.6.1Method1225.3.6.2Results1225.3.7Summary1245.4VericationPhase1255.4.1GroundTruthSensingAccuracyAssessment1265.4.2StatisticalAnalysis127ii

PAGE 7

5.4.3DetectionofSensingAnomalies1275.4.3.1Method1275.4.3.2Results1285.4.4EstimationofSensingAccuracy1295.4.4.1Method1295.4.4.2Results1305.4.5IsolationofPoorlySensedRegions1305.4.5.1Method1315.4.5.2Results1315.4.6Followup:SourcesofErrorandInconsistency1335.4.6.1Method1335.4.6.2Results1345.4.7Summary1365.5UtilityPhase1375.5.1GroundTruthSensingAccuracyAssessment1385.5.2StatisticalAnalysis1385.5.3ApplyingDetectiontoSwitchSensors1395.5.3.1Method1395.5.3.2Results1405.5.4ApplyingIsolationtoResetSuspectRegions1435.5.4.1Method1435.5.4.2Results1445.5.5ApplyingEstimationtoRankSensorsByRelativeAccuracy1465.5.5.1Method1465.5.5.2Results1475.5.6Summary1505.6Conclusions151Chapter6Discussion1546.1ImplicationsoftheExperimentalResults1556.2Contributions1586.2.1SensingAssessmentinUnknownEnvironmentsRelyingSolelyonReadingsfromaSingleSensor1586.2.2ValidationofEvidentialInconsistencyMetricsforQuantifyingInconsis-tencyinSensorReadings1606.2.3FrameworkforIdentifyingSensorSuitabilityinUnknownEnviron-ments1606.3LimitationsofExperiments1616.4Recommendations:TowardsAdaptationtoSensingAnomalies1636.5Summary165Chapter7SummaryandFutureWork1677.1Summary1687.1.1Experiments1687.1.1.1Training1717.1.1.2Verication171iii

PAGE 8

7.1.1.3UtilityPhase1737.1.2Contributions1747.2FutureWork174References178Appendices196AppendixADetailedResults:FeasibilityStudy197AppendixBDataCollection:Experiments202AppendixCDetailedResults:Experiments209AbouttheAuthorEndPageiv

PAGE 9

ListofTablesTable1Examplesofcommonbuildingmaterialswhichproduceanomaliesinrangesensorsandcameras.5Table2Categorizationofrelatedworkaddressingsensingproblemsingeneralbytaskcolumnsandmethodrows.13Table3Relatedworkonsourceorsensingassessmentbymethod.16Table4RelatedworkonsensorfaultdetectionandidenticationFDIbymethod.35Table5Relatedworkonisolatingpoorlysensedregionsbymethod.54Table6Relatedworkonadaptivefusionbymethod.56Table7Metricsformeasuringinconsistencyinquantitativeuncertaintymodels.63Table8Thetasksassignedtoeachsub-moduleinsensingassessment.75Table9Characteristicsofthethreeindoorhallwaysusedfordatacollection.88Table10Theparametersusedinthesensormodels.90Table11Theparametersusedtocreatesensingaccuracyindicatorsfrommethodsforquantifyinginconsistencyinevidentialmodels.92Table12Thetestbedsandrobotsusedfordatacollectionineachanalysisphaseoftheexperiments.104Table13CharacteristicsoftheATRV-Jrtrainingtestbeds.105Table14CharacteristicsoftheATRV-Jrvericationtestbeds.106Table15CharacteristicsoftheNomad200testbeds.107Table16Theparametersusedinthesensormodels.110Table17Theparametersusedtocreatesensingaccuracyindicatorsfrommethodstoquantifyinconsistencyinevidentialmodels.112Table18Trainedperformancefordetectionforeachinconsistencymethod.116Table19Trainedperformanceforestimationforeachinconsistencymethod.120Table20Trainedperformanceforisolationforeachinconsistencymethod.123Table21Detectionperformanceforthetrainedindicatorsinthetrainingandverica-tiontestbeds.128v

PAGE 10

Table22EstimationperformancefortheTBMCONFLICT,0.18indicatorinthetrain-ingandvericationtestbeds.130Table23IsolationperformancefortheTBMCONFLICT,0.38indicatorbrokendownbytestbed.132Table24Detectionaccuracyforthetrainedindicatorsbrokendownbytestbed.135Table25StatisticallysignicantchangesintheErrorscore.143Table26ImprovementsinsensingaccuracyachievedbyusingtheTBMCONFLICT,0.38indicatortoresetsuspectcellsbrokendownbytestbed.145Table27ComparisonofthegroundtruthandtheTBMCONFLICT,0.18indicator'srankingofthesensors,brokendownbytestbed.148Table28ComparisonofmeanErrorscoreswhenthebest,worst,ortheTBMCON-FLICT,0.18indicator'stoprankedsensorisusedbrokendownbytestbed.149Table29Comparisonofsensingaccuracyestimationresultsfromthefeasibilitystudyandtrainingphaseoftheexperiments.155Table30Comparisonofsensinganomalydetectionresultsfromthefeasibilitystudyandtrainingphaseoftheexperiments.156Table31Groupsoftargetsensorsexaminedi.e.trained,veried,orappliedintheexperiments.169Table32Thetestbedsusedandgroupsoftargetsensorsexaminedineachanalysisphaseoftheexperiments.170Table33Rawresultsforthefeasibilitystudy.197Table34Randomlygeneratedstartingpositionsmetersandobstaclepositionsme-tersforeachtestbed.202Table35Rawresultsforestimationanddetectionforall184indicators.209Table36Rawresultsforestimationanddetectionforlaserreadingsonly.214Table37Rawresultsforestimationanddetectionforrangecamerareadingsonly.219Table38Rawresultsforisolationforall184indicators.225Table39Rawresultsforisolationforlaserreadingsonly.233Table40Rawresultsforisolationforrangecamerareadingsonly.241Table41Rawresultsforisolationbrokendownbytestbed.249Table42Rawresultsfortheanalysisofthesourcesoferrorandinconsistency.251Table43Rawresultsfortheanalysisofthesourcesoferrorandinconsistencyforlaserreadingsonly.252Table44Rawresultsfortheanalysisofthesourcesoferrorandinconsistencyforrangecamerareadingsonly.253vi

PAGE 11

ListofFiguresFigure1Asourcereliabilityestimationapproachformulti-agentsystemsfromBar-berandKim,2001.19Figure2AsensorfaultdetectionanddiagnosisapproachforSoika,1997a.39Figure3Anovelframeworkofasystemforunsupervisedidenticationofthesuit-abilityofsensorsinunknownenvironments.74Figure4AchangeheuristicforstaticoccupancygridmapsfromGambino,Ulivi,andVendittelli,1997.82Figure5GraphicaldepictionoftheconemodelMurphy,2000derivedfromthesonarparameters.84Figure6Thethreeindoorhallwaysusedfordatacollection.89Figure7TheNomad200robotusedfordatacollection.90Figure8ProcedureusedtocalculatetheErrormetric.93Figure9Estimationresultsforeachinconsistencymethod.96Figure10Detectionresultsforeachinconsistencymethod.97Figure11Changeincorrelationcoefcientr,numberoffalsepositiveandfalsenega-tiveexamplesforANXIETYasthethresholdvaries.98Figure12Falsenegativerateforeachinconsistencymethod.99Figure13Falsepositiverateforeachinconsistencymethod.100Figure14ATRV-Jrtrainingtestbeds.105Figure15ATRV-Jrvericationtestbeds.105Figure16TheNomad200testbeds.107Figure17Therobotsandsensorsusedintheexperiments.108Figure18ProcedureusedtocalculatetheErrormetric.113Figure19Varianceindetectionaccuracyforthethresholdvaluestested.117Figure20ExampleofaCanestarangecamerasensinganomalythatpreventedthein-consistencymethodsfromachievinglowfalse-negativerates.118Figure21Varianceinestimationperformanceforthethresholdvaluestested.121Figure22Procedureusedtoclassifyeachcellasaccurateorerroneous.123vii

PAGE 12

Figure23Varianceinisolationperformanceforthethresholdvaluestested.124Figure24Procedureusedtoclassifyeachcellasaccurateorerroneous.132Figure25MeanandvarianceintheisolationperformanceoftheTBMCONFLICT,0.38indicatorbrokendownbytestbed.132Figure26Detectionaccuracyblackbarscomparedtotherelativecontributionofsensinganomaliesred,poseestimationerrorsyellow,andsensornoisegreenbrokendownbytestbed.135Figure27Resultsfromtheswitchscenario.142Figure28MeanandvarianceinErrorscoresforthebaselinedashedandisolatingsolidscenariosbrokendownbytestbed.145Figure29Exampleofisolation-basedadaptationtoasensinganomaly.146Figure30Exampleofranking-basedadaptationtoasensinganomaly.149Figure31MeanandstandarddeviationofErrorscoreswhenthebest,worst,ortheTBMCONFLICT,0.18indicator'stoprankedsensorisusedbrokendownbytestbed.150Figure32Picturestakenduringdatacollection.203Figure33SchematicsforeachATRV-Jrtestbed.204Figure34AutomaticallygeneratedoccupancygridsforeachATRV-Jrtestbed.205Figure35Expectederrorsourcemapsforlaser.206Figure36Expectederrorsourcemapsfortherangecamerainnormalmode.207Figure37Expectederrorsourcemapsfortherangecamerainmolemode.208viii

PAGE 13

AnInconsistency-basedApproachforSensingAssessmentinUnknownEnvironmentsJenniferDianeGageABSTRACTWhileexploringanunknownenvironment,anintelligentagenthasonlyitssensorstoguideitsactions.Eachsensor'sabilitytoprovideaccurateinformationdependsontheenvironment'schar-acteristics.Iftheagentdoesnotknowthesecharacteristics,howcanitdeterminewhichsensorstorelyon?Thisproblemisexacerbatedbysensinganomalies:caseswheresensorsareworkingbutthereadingsleadtoanincorrectinterpretationoftheenvironment,e.g.lasersensorscannotdetectglass.Thisworkaddressesthefollowingresearchquestion:Cananinconsistency-basedsensingaccuracyindicator,whichreliessolelyonfusedsensorreadings,beusedtodetectandcharacter-izesensinganomaliesinunknownenvironments?Anovelinconsistency-basedapproachwasinvestigatedforsensinganomalydetectionandcharacterizationbyamobilerobotusingrangesensingformapping.Basedonthehypothesisthatsensinganomaliesmanifestasinconsistentsensorreadings,theapproachemployedDempster-Shafertheoryandsixmetricsfromtheevidentialliteraturetomeasurethemagnitudeofinconsis-tency.Thesewereapplieddirectlytofusedsensordatawithathreshold,forminganindicator,usedtodistinguishminornoisefromanomalousreadings.Experimentswithrealsensordatafromfourindoorandtwooutdoorenvironmentsshowedthatthreeofthesixevidentialinconsistencymetricscanpartiallyaddresstheissueofnoticingsensinganomaliesinunknownenvironments.Polaroidsonarsensors,SICKlaserrangenders,andaCanestarangecamerawereused.Despiteextensivetraininginknownenvironments,theindicatorscouldnotreliablydetectsensinganomalies,i.e.distinguishthemfromordinarynoise.However,sensingaccuracycouldbeestimatedcorrelationswithsensorerrorexceeded0.8andregionswithsuspectreadingscouldbeisolated.Trainedindicatorsfailedtoranksensors,butim-ix

PAGE 14

provedmapqualitybyresettingsuspectregionsupto57.65%orguidingsensorselectionupto75.86%.Thisworkcontributestotheroboticsanduncertaintyinarticialintelligencecommunitiesbyestablishingtheuseofevidentialmetricsforadaptingasinglesensororidentifyingthemostaccuratesensortooptimizethesensingaccuracyinunknownenvironments.Futureapplicationscouldenableintelligentsystemstoswitchinformationsourcestooptimizemissionperformanceandidentifythereliabilityofsourcesfordifferentenvironments.x

PAGE 15

Chapter1IntroductionWhileexploringanunknownenvironment,anintelligentagenthasonlyitssensorstoguideitsactions.Eachsensor'sabilitytoprovideaccurateinformationdependsontheenvironment'scharacteristics.Assumingthattheagentdoesnotknowthesecharacteristics,howcanitdeterminewhichsensorstorelyon?Thisproblemisexacerbatedbysensinganomalies:caseswherephysicalsensorsareworkingwithinthemanufacturer'sspecicationsbutthereadingswouldleadtoanincorrectinterpretationoftheenvironment.Thesecasesonlyexistwhenasensorinteractswiththeenvironmentandoftenaffectmultiplesensors,forexamplemanylaserrangendersseethroughglasswindowswhichalsoscattersonarsignals.1.1ResearchQuestionTheresearchquestionthisworkaddressesisasfollows:Cananinconsistency-basedsensingaccuracyindicator,whichreliessolelyonfusedsensorreadings,beusedtodetectandcharacter-izesensinganomaliesinunknownenvironments?Morespecically,cananinconsistency-basedsensingaccuracyindicatorenableamobilerobottoautonomously:Estimatetheaccuracyofasensororsetofsensorsinthecurrentsensingcontext,Isolateregionswithinanunknownenvironmentwheresensinganomaliesoccur,Detectwhenasensinganomalyhasoccurredwithoutrelyingonaprioriinformationabouttherobot'senvironment?Thisworkisfocusedondetectingandcharacterizingsensinganomaliesrelyingsolelyonfusedreadingsfromasinglepossiblyaffectedsensor.1

PAGE 16

Autonomousmobilerobotsinparticulararedependentonaccuratesensingtoensuresafeandeffectiveoperationinunknownenvironmentsprevalentindomainslikespaceexploration,searchandrescue,andmilitaryoperationswhererobotsarebecomingincreasinglycommon.Todatemobilerobotsusedinthesedomainshaveenjoyedlittletonoautonomy.AccordingtoBrooks,thisislargelyduetocriticallimitationsinperceptualcapabilitiesLopesetal.,2001onwhicharobot'ssuccesslargelydependsMurphy,2000.Problemslikesensinganomalieswhichcanoc-curinunstructuredenvironmentsNourbakhshetal.,2005onlyexacerbatethisshortcoming.Bydevelopingasolutiontotheproblemofdetectingandcharacterizingsensinganomaliesthisworkpavesthewayforthedevelopmentofmobilerobotsthatcanautonomouslyrecoverfromprevi-ouslyundetectablesensingproblemsandadaptasthesensingsituationchanges,greatlyincreasingtheutilityofsuchrobotsinthesedomains.Thiscapabilitywouldallowrobotsinthesechalleng-ingenvironmentstoknowwhentheyarehavingtroublesensing,andneedtoreexamineoravoidareasthataredifculttosense.Theremainderofthischapterisorganizedasfollows.FormaldenitionsofthekeytermsusedinthisdissertationareprovidedinSection1.2.Section1.3discussestheprevalenceofsensinganomaliesforexteroceptivesensorswhichmeasureimportantcharacteristicsoftheenvironmentandthechallengesassociatedwithdetectingsuchanomaliesinthecontextofexistingsensorfaultdetectionandidenticationFDIapproachesandtheapproachproposedhere.Section1.4pro-videsabriefoverviewofthecontributionsofthisworkandSection1.5discussestheorganizationfortherestofthisdissertation.1.2DenitionsThissectionestablishesdenitionsforthekeytermsusedinthisdissertationrelatedtosensingassessmente.g.qualityversusaccuracyandsensinganomalies.Thisworkisfocusedonassess-ingtheaccuracyandstatusnormaloranomalousofsensinginunknownenvironments.HereafterasensorreferstoanyphysicaldevicethatmeasuressomeattributeoftheworldMurphy,2000.Ingeneral,sensingreferstoasetofsensorsandalgorithmsthatenableanautonomousagenttoperceiveitsenvironment.Inbehavioralroboticstermssensingreferstoasetofactiveperceptualschema.Thisdissertationdenesanunknownenvironmentinthestrictsense,inthatnoaprioriknowledgeisavailableandnogeneralcharacteristicse.g.smoothwallsandninetydegreecor-2

PAGE 17

nerscanbeassumed.Sensinganomaliesarecasesinwhichphysicalsensorsareworkingwithinthemanufacturer'sspecicationsbutthereadingswouldleadtoanincorrectinterpretationoftheenvironmentCarlsonandMurphy,2005.ForexamplesseeSection1.3.1.Zilbersteinhasdenedsensingqualityastheobjectivequalityoftheinformationpro-vided.Healsoidentiedthreefactorsthatdeterminesensingquality:Howaccuratelytheinformationreectsthetruestateoftheworld,Howoftenupdatedsensinginformationbecomesavailableascomparedtothefrequencyandextentofchangeintheworld,Theeasewithwhichtheagentcaninterprettheinformationproducedandexploititeffec-tively.Theresearchquestionisfocusedonassessmentoftherstofthethreefactorsreferredtohereassensingaccuracy.ArelatedmetricdescribingthetrustworthinessofanindividualsensororothersourceofinformationissensorreliabilitywhichistypicallyformulatedastheprobabilitythatasensorsourcewillprovideaccurateinformationseeSection2.1.2.1.3Motivation:SensingAnomaliesThissectiondescribestheprevalenceofsensinganomaliesforexteroceptivesensorsandthechallengesassociatedwithdetectingandcharacterizingthesecasesinthecontextofexistingsen-sorFDIapproacheswhichmotivatestheneedforanewapproach.Section1.3.1providesabriefoverviewofhowsensinganomaliesoccurforactiveandpassiveexteroceptivesensorswhichmea-sureimportantcharacteristicsoftheenvironmente.g.rangesensorsandlistsexamplesofcom-monbuildingmaterialsorenvironmentalconditionswhichcauseanomaliesforthesesensors.Section1.3.2illustratesthatsensinganomaliesareparticularlydifculttodetectandcharacter-izebecausetheyoftenaffectmultiplesensorsanddonotexistuntilthesensorinteractswiththeenvironment.TheimplicationsofthesecharacteristicsarediscussedforexistingsourcereliabilityandsensorFDIapproachesandtheapproachpresentedinChapter3.3

PAGE 18

1.3.1CommonnessofSensingAnomaliesforExteroceptiveSensorsExteroceptivesensors,likerangesensorsandcameras,whichmeasurecharacteristicsoftheenvironmentthatgenerallyvaryintimeandspacearemostsusceptibletosensinganomalieswhichoccurfrequentlyinunstructuredenvironments.Activerangesensors,e.g.sonartransducers,IRproximitysensors,laserrangenders,andstructuredlightcomputervisionsystemswillprovideinaccuratereadingsinthepresenceofmaterialsthattransmitorabsorbtoomuchoftheiractivesignal.Inaddition,anomaliesoccurwhenthesignalreectsawayfromthesensor.Passivesens-ingsystemslikecomputervisionsystemsusingvisiblelightandinfrarednightvisioncamerasalsotendtoencounteranomalieswithtransmissiveandabsorptivematerialsoralackofsufcientambientsignal.ExamplesofcommonbuildingmaterialswhichhavetheseeffectsoncommonlyusedrangesensorsandcamerasarelistedinTable1.Mirrorsarealsocommonbuildingmaterialswhichpassivecomputervisionsystemsdonotperceiveassurfaces.Thevisionsystemwillinsteadprovideincorrectinformationregardingthereectedscenee.g.distanceandlocationtomirroredobstacles.AdditionalsensinganomaliesforrangesensorsandCCDcamerasarefoundinunstructuredoutdoorenvironments.Duetotheuseofanarrowbeam,laserrangendersdetectsmallparticlesinfog,smoke,dust,orsnowandsmallobjectse.g.leavesinsparsefoliagewhichleadstode-tectionofspurioussurfacesAngelopoulouandJr.,1999.Otherexamplesarestandingwateroricewhichreectnear-infrarede.g.laserrangendersandinfrarede.g.nightvisioncamerassignalspoorlyBaldridgeetal.,2009.Standingwatersometimesbehaveslikeamirrorforvisi-blelightsensors.Layersofcoolandhotaircandistortvisiblelightleadingtoeffectscommonlyreferredtoasmirages.Avoidingthesematerialsandenvironmentalconditionsseverelylimitstherangeofoperatingenvironmentsforautonomousmobilerobots,thusasolutionisneededtoenablesuchsituatedsystemstoadapttothepresenceofsensinganomaliesinrealtime.1.3.2SensingAnomalyDetectionandCharacterizationSensinganomaliesareparticularlydifculttodetectandcharacterizeinunknownenviron-mentsbecausetheanomalyislikelytoaffectmultiplesensorsanddoesnotexistuntilthesen-sorinteractswiththeenvironment.Unlikehardwareandcalibrationfailures,theenvironmentalconditionsthatcausesensinganomaliesarenotonlylikely,butguaranteed,toaffectallidentical4

PAGE 19

Table1.Examplesofcommonbuildingmaterialswhichproduceanomaliesinrangesensorsandcameras. Material SensingTechnology Affect Soundproongpanels activeacoustic absorbsignal Paintedsheet-rockwalls activeacoustic reectsignalaway texture-basedvision insufcientfeatures Glass visiblelight transmitsignal activenear-IR transmitsignal activeacoustic reectsignalaway Darksurfaces visiblelight absorbsignal activenear-IR absorbsignal sensorsinasensingsystem.Inaddition,sensorswhichusedissimilarsensingtechnologymaysuf-ferfromthesameanomaly.Glasssurfacesforexamplearedifcultfornear-IRandvision-basedrangesensorsduetotransparencyandsonarduetospecularacousticreectiontosense.ExistingapproachesforsensorFDIareunsuitedtodetectsensinganomaliesastheyarede-signedtohandlecaseswheresensorsaremalfunctioningorincorrectlycalibrated.TraditionalFDItechniquesseeforexampleDeardenetal.,2004;Monteriuetal.,2007brelyonmodelsofnormaland/orfaultyplantsensorbehaviortodetectanddiagnosefaults.Sensinganomaliesaredependentonenvironmentalcharacteristicswhichcannotbemodeledifthatenvironmentisun-known.Learning-basedsensorFDItechniquesseeforexampleChenandSaif,2007;Guetal.,2001areunsuitablesinceunknownenvironmentsarelikelytoinduceanomaliesnotfoundinthetrainingdata.Consensus-basedsensororsourcereliabilityestimationseeforexampleBarberandFullam,2003andsensorFDItechniquesseeforexampleBacon,Ostroff,andJoshi,2001;Soika,1997arelyonredundancytoassesssourcesordetectandidentifyfaultsbytrackingincon-sistenciesbetweenreadingsfromasinglesourceandtheconsensusofallavailablesources.Thesewouldtheoreticallydetectanddiagnosesensinganomalies,butonlyincaseswheretheconsensuswasaccuratewhichcannotbeassumedinthepresenceofsuchanomalieswhichmayaffectmulti-plesensors.Inthisworkacomputationallyefcientinconsistency-basedapproachfordetectingandchar-acterizingsensinganomaliesisproposedwhichreliesonDempster-Shaferformulationsofevi-dence,asopposedtoprobabilisticmethods.TheDempster-Shaferapproachwasselectedduetothehypothesisthatsensinganomalieswouldmanifestasinconsistencyinthesensorreadings.Dempster-Shaferformulationsde-couplethebeliefincontradictoryhypothesese.g.beliefinA5

PAGE 20

and:Aarenotrequiredtosumtooneenablingthecreationofmetricstomeasurethemagnitudeofinconsistencyindependentlyfrominformativeness.Inconsistencymetricsfromtheuncertaintyliteraturewereapplieddirectlytofusedsensorreadingswiththresholdsusedtodistinguishor-dinarynoisefromanomalies.Thisapproachisunique,andparticularlysuitedforproblemslikesensinganomalies,inthatitcandetectproblemsevenifthereisonlyonesensororallthesensorsinactiveuseareaffectedbytheanomaly.Inadditionitisefcientwithlineartimecomplexityonthenumberofbeliefmassesprovidedthattheevidentialsensormodelconsidersasmallsetofhypothesesandeasytoimplement,requiringonlytheimplementationofaclosedforminconsis-tencymetricandasystemtoapplythemetrictothefusedreadingsandrespondtotheresults.1.4ContributionsThisdissertationcontributestotheroboticsanduncertaintyinarticialintelligencecommu-nitiesbyestablishingafoundationfortheuseofevidentialmetricsforadaptingasinglesensororidentifyingthemostaccuratesensorinasuitetooptimizetheaccuracyofsensinginunknownen-vironments.Thisworkpavesthewayforthedevelopmentofintelligentsystemsthatcanswitchinformationsourcestoensurethebestmissionperformance,identifytherelativecontributionandreliabilityofsourcesfordifferentenvironments,andreasonaboutwhichsourcestouseunderwhatcircumstances.Sensingassessmentinunknownenvironmentsrelyingsolelyonreadingsfromasinglesen-sor.Thisworkpresentstherstknowngeneralapproachcapableofestimatingsensingaccu-racyandisolatingpoorlysensedregionsinunknownenvironmentswhenonlyreadingsfromasinglesensorareavailable.Existingapproachesforestimatingtheaccuracyofpeersinsen-sornetworksormulti-agentsystemsdependoncomparisonswithtrustedlocalreadingse.g.Momani,Challa,andAlhmouz,2008oranaccurateconsensusofmultiplesourcese.g.Ganeriwal,Balzano,andSrivastava,2008.Anexistinginconsistency-basedapproachisolatespoorlysensedregionsbutitsapplicabilityislimitedto3DpointclouddataRomanandSingh,2006.Theapproachpresentedinthisdissertationislessconstrained.Itdoesnotrequireanaccurateconsensusandisapplicabletoanymodeloffusedsensordata,providedthatasuit-ableinconsistencymetriccanbeformulatedi.e.onethatcandistinguishinconsistencyfromlackofinformation.ExperimentsseeChapter5providestatisticallysignicantevidence6

PAGE 21

thatindicatorsbasedontheconictbeliefmassfromSmets'transferablebeliefmodelTBMcanestimatetheaccuracyoflaserandCanestarangecamerasensorscorrelationswithtrueerrorabove0.8inunknownenvironmentsandisolatepoorlysensedregionsrelyingsolelyonfusedreadingsfromasinglesensor.Validationofevidentialinconsistencymetricsforquantifyinginconsistencyinsensorreadings.Thisworkcontributestotheuncertaintyinarticialintelligencecommunitybyexploringtheapplicationofevidentialinconsistencymetricstodetectandcharacterizesensinganomaliesinfusedsensorreadings.Itproducedtherstknownexperimentalstudyofevidentialmetricsformeasuringsensorinconsistencytoprovidefeedbackforsensinginunknownenvironments.Frameworkforidentifyingsensorsuitabilityinunknownenvironments.Thisdissertationalsoprovidesasystemarchitectureforconceptualizingthelargerproblemofidentifyingthesuit-abilityofsensorsinunknownenvironments.Section3.1introducestheframeworkwhichcombinestheworkproposedherewithexistingapproachesfromtheroboticsandmulti-agentsystemsliterature:sourcereliabilityestimation,traditionalsensorFDI,sensormanagement,andsimultaneouslocalizationandmappingSLAM.Thecompletesystemwouldenableamobilerobottoeffectivelymanageitssensingaccuracywhileidentifyingwhichsensorsper-formbestinanygivenregionofanaprioriunknownenvironment.1.5OrganizationofDissertationTheremainderofthisdissertationisorganizedasfollows.Chapter2reviewsrelatedworkad-dressingsensingproblemsingeneral,withaparticularfocusonapproachesthatcanbeusedinunknownenvironments.Thefollowingtopicsaresurveyed:sourcee.g.asensororagentassess-ment,sensorFDI,isolationofpoorlysensedregions,andadaptivesensorfusion.ForreferenceChapter2alsoincludesasurveyofinconsistencymetricsfromtheuncertaintyliterature.Chap-ter3providesadetaileddescriptionoftheinconsistency-basedapproachfordetectingandchar-acterizingsensinganomaliesinunknownenvironments.Chapter4describesafeasibilitystudytoexplorethemeritsofthisapproachusingrealsensordatacollectedbyaNomad200mobilerobotequippedwitharingofsonarsensorsandaSICKPLSlaserrangender.Thestudyfoundthatindicatorscouldbetrainedinknownenvironmentstodetectsensinganomalieswithbetterthan7

PAGE 22

95%accuracyorestimatesensingaccuracycorrelationswithtrueerrorabove0.85.Chapter5describesexperimentsdesignedtovalidatetheeffectivenessoftheapproachforunknownenviron-mentsusingSICKLMSlaserandCanestarangecamerareadingscollectedbyaniRobotATRV-Jrmobilerobotinunclutteredstaticindoorandoutdoorenvironments.Theresultsshowthattheinconsistency-basedmethodinvestigatedinthisdissertationprovidesapartialsolutiontotheprob-lemofnoticingsensinganomalies,specically:Theindicatorscouldnotreliablydetectsensinganomalies,i.e.distinguishordinarysensorerrorfromsensinganomalies.Trainedindicatorsdidnotaccuratelydetectanomaliesinnewtestbeds.71%to74.71%accuracyrecordedacrossthetestbeds.Sensingaccuracycouldbeestimatedinnewenvironmentsusingmethodsandthresholdsde-terminedduringtraining.AtrainedindicatorbasedontheconictbeliefmassfromSmets'TBMestimatedsensingaccuracyabove0.8correlationwithtruesensorerrorforthelaserandCanestarangecamerawhenusedindividuallyinnewenvironments.Regionswithsuspectsensorreadingscouldbeisolatedinnewenvironments.Atrainedin-dicatorbasedonSmets'conictisolatedpoorlysensedregionswithoverlapbetweenerro-neousandsuspectregionsfrom0.49to0.62ascomparedto0.01overlapwhenregionsarerandomlyclassiedinthenewenvironmentsforthelaserandCanestarangecamera.Chapter6examinestheresultsfromthefeasibilitystudyandexperimentstoidentifytheoveralllessonslearnedfromthiswork.Theresultsindicatethatfutureapplicationswithnewsensorswillneedtoperformtrainingrunsinknownenvironmentstodeterminewhichmethodworksbestforthatsensorandtonddistinctthresholdstoestimatesensingaccuracyandisolatepoorlysensedregions.Detaileddiscussionsofthecontributionsofthisdissertation,thelimitationsoftheexper-iments,andrecommendationsforadditionalworktowardenablingonlineadaptationtosensinganomaliesarealsoprovided.Chapter7providesasummaryofthisdissertationandaexploresopportunitiesforfutureworkonsensinganomalydiagnosis,autonomousidenticationofsensorsuitabilityinunknownenvironments,andatheoreticalfoundationtodescribehowsensinganoma-liesmanifestbasedoncharacteristicsofasensingsystem.8

PAGE 23

Chapter2RelatedLiteratureThischapterwillshowthatthisistherstknownstudytodevelopageneralapproachtoad-dresstheproblemofsensinganomalies.Sensinganomaliesoccurwhenphysicalsensorsareworkingwithinthemanufacturer'sspecicationsbutthereadingswouldleadtoanincorrectin-terpretationoftheenvironment.Forexample,rangefromstereosystemscannotdetectmirroredsurfacesbutinsteadprovideinaccurateinformationaboutthemirroredscenee.g.locationofanddistancetomirroredobstacles.Beforeasituatedagentcanbetrustedtooperateinthepresenceofsensinganomaliessolutionsareneededtoestimatesensingaccuracy,detecttheseanomalies,andisolatetheirenvironmentalsourceswithoutrelyingonaprioriinformationabouttherobot'senvironment.Thesesolutionscanbeusedtobuildmoreadvancedcapabilitiesliketheabilitytoranksensorsaccordingtotheirrelativesensingaccuracyanddiagnose,i.e.determinewhichsen-sorsareaffectedbyanomalies.Thischaptersurveysrelatedworktodevelopsimilarsolutionsforsensingproblemsingeneral,specically:Sourceorsensingassessment.Thesestudies,discussedinSection2.1,evaluatesensorsorothersourcesofinformationtoidentifyoratleastreducetheinuenceofsensorsthatareaffectedbysensingproblems.Thissurveyisparticularlyfocusedonassessmentmethodsthatdonotrelyonaprioriinformation.SensorfaultdetectionandidenticationFDI.Thesestudies,discussedinSection2.2,aredesignedtodetectanddiagnosesensorfaultsand/orcalibrationproblems.Isolatepoorlysensedregions.Thesestudies,discussedinSection2.3,isolatepoorlysensedregionsinanaprioriunknownenvironment.Adaptivesensorfusion.Thesestudies,discussedinSection2.4,attempttocopewithsensingproblemsbyadjustingfusionparameterstodiscountinformationfromaffectedsensors.9

PAGE 24

OnlyonespecializedsolutionMorales,Takeuchi,andTsubouchi,2008fordetectionofGPSsensinganomalieshasbeenfoundintheliteraturewhichappliesastandardoutlierdetectiontesttorejectGPSreadingsthatdivergetoofarfromaKalmanlter-basedestimateofamobilerobot'sposition,anapproachwhichimplicitlytruststherobot'sotherposesensorswheelencodersandIMU.Thesolutionpresentedinthisdissertationismoregeneral.Itcanbeusedwithasinglesen-sorandonlyassumesthatsensinganomalieswillmanifestasinconsistentreadingsandappliesinconsistencymetricsfromtheevidentialliteraturedirectlytofusedsensorreadings.Section2.5describestheinconsistencymetricsthemselves.Theseareincludedforreferenceandtoprovidethereaderwithanunderstandingofthegeneralcontextinwhichthesemetricsweredeveloped.Inthischapterrelatedstudiesaddressingsensingproblemsingeneralarecategorizedandeval-uatedaccordingtothemethodusedtoassesssensorsorothersourcesofinformation,performsensorFDI,isolatepoorlysensedregions,ormakeadjustmentsinadaptivesensorfusion.Sens-inganomaliesdonotexistuntilthesensorinteractswiththeenvironmentandoftenaffectmultiplesensorsseeChapter1.Solutionsfordetectingandrespondingtotheseanomaliesinunknownen-vironmentsmustnotrelyonaprioriinformationorassumptionsinvalidatedbyanomalies.Giventheserestrictionstheperformanceofasolutionforothersensingproblemswhichareofteniso-latede.g.hardwarefailureoracompromisedsensornodeandexistregardlessofthesensor'ssurroundingsisconsideredirrelevant.Insteadthissurveyfocusesonthemethodemployedbythestudytoassesssensorsanddeterminetheirstatus.Themethodsfoundintheliterature,indescend-ingorderbysuitabilitytoaddresssensinganomaliesinunknownenvironments,areasfollows:Inconsistency-based.Thesemeasureinconsistencywithinasetofsensorreadingsorbetweenreadingsfromdistinctsensors.Theyonlyrequirethatthetruestateoftheworldisconsistent.Consensus-based.Thesecomparereadingsfromasinglesensortothefusedconsensusofmultiplesensors.Theyrequireanaccurateconsensus.Consensus-basedpair-wise.Theseperformpair-wisecomparisonstodeterminewhichsen-sorsagreewithmoreoftheirpeers.Theyrequireamajorityofaccuratesensors.Learnedmodelsuninformed.Theselearntherelationshipbetweensensorreadingsandthestatusofthesensorswithoutrelyingonaprioriinformation.Theymaynotdetectanomaliesthatarenotencounteredinthetrainingsetandarevulnerableduringthetrainingperiod.10

PAGE 25

Inconsistency-basedtrustedsource.Thesemeasureinconsistencybetweensensorreadingsandatrustedsourcee.g.locallysensedvalueswhenevaluatingpeers.Theyarelimitedtoenvironmentswherethetrustedsourceisnotaffectedbyanomalies.Qualitativemodels.Theserelyonasetoflogicalstatementse.g.causalmodeltodescribethebehaviorofasystemanditssensorsundernormalandfaultyconditions.Thesemodelsareunlikelytocapturesensinganomalieswhichdependonanunknownenvironment.Stochasticmodels.Theserelyononeormoreprobabilisticmodelstodescribethebehaviorofasystemanditssensorsundernormalandfaultyconditions.Thesemodelsareunlikelytocapturesensinganomalieswhichdependonanunknownenvironment.Learnedmodelsinformed.Theselearntherelationshipbetweensensorreadingsandthesta-tusofthesensorswithinaframeworke.g.neuralnetworkdesignedtocaptureknownsys-temcharacteristicsand/orsensorproblems.Theymaynotdetectanomaliesthatarenoten-counteredinthetrainingsetandwillbeunabletolearnaboutanomaliesthatinvalidatethedesigner'sassumptions.Analyticalmodels.Theserelyononeormoreanalyticalmodelstodescribethebehaviorofasystemanditssensorsundernormalandfaultyconditions.Thesemodelsareveryunlikelytocapturesensinganomalieswhichdependonanunknownenvironment.Measuredsensorcharacteristics.TheseperformassessmentbymeasuringcharacteristicslikequalityofserviceQoSorresponsivenessthathavenothingtodowiththeaccuracyoftheinformationprovided.Thesecannotbeusedtodetectorcharacterizesensinganomalies.Knownsensorcharacteristics.Thesedonotevaluatesensorsbasedontheactualaccuracyoftheirreadingsbutestimatetheexpectedaccuracybasedonaprioriknowncharacteristicse.g.classicationaccuracy,thereforetheycannotadaptastheenvironmentchanges.Thislimitationmakesthemunsuitableforhandlingsensinganomaliesinunknownenvironments.Unspecied.Thesedevelopgenericapproachesortheoriesforsensingassessmentwhichcanuseanyoftheabovemethods.Table2groupsrelatedstudiesfromtheliterature,whichaddresssensingproblemsingeneral,bytaskandmethodasabovetoevaluatetheirapplicabilitytotheproblemofsensinganomalies11

PAGE 26

andtoprovideanorganizationaltoolforsurveyingthesestudies.ThisworkisincludedinTable2forreference.Thesourceorsensingassessmenttaskisbrokendownintothreesubtasks:estimat-ingsensingaccuracywhichevaluatesasetofsensorse.g.sensornetworkasawhole,estimat-ingsourcereliabilitywhichestimatesthereliabilitye.g.probabilityofcooperationoraccuracyofagivensource,andrankingsourcesbyreliabilitywhichenablesanintelligentagenttoselectthemostreliablesources.ThesensorFDItaskisdecomposedintodetectionanddiagnosisofsensingproblems.Ifmultiplestudiesusethesamemethodtoaddressthesametask,Table2pro-videsarepresentativeexampleandthetotalnumberofstudiesinbrackets.AfulllistofstudieswhichaddresseachtaskaregiveninTables3through6providedatthebeginningofSections2.1through2.4respectively.Table2alsoliststhenumberofstudiesthataddresseachtaskbottomrowanduseeachmethodright-mostcolumnwithduplicateentriese.g.studieswhichusemul-tiplemethodstoaddressthesametaskcountedonceforeachcolumnandrow.Table2showsthatthemostpopularsolutionsforsensingassessment:learned,stochastic,oranalyticalmodel-basedsensorFDIandestimationofsourcereliabilitybymeasuringresponsive-nessi.e.reliabilityasprobabilityofcooperation,arenotsuitableforaddressingsensinganoma-liesandthatlittleattentionhasbeengiventotheproblemsofestimatingsensingaccuracy,rankingsourcesbyreliability,andisolatingpoorlysensedregionsintheenvironmentwithoutrelyingonaprioriinformation.Recenttrendsinestimatingsourcereliabilityhaveshownincreasedinterestinapproacheswhichdonotrelyonaprioriinformationinboththemulti-agentsystemsandsen-sornetworkingeldswithpopularapproachesbeginningtoemergeforexampleGaneriwalandSrivastava,2004.Thesearelargelyfocusedontheproblemofhowtointegratedirectexperienceandreputationinformationi.e.directexperienceofpeerswhichmayormaynotbeaccuratelyre-portedtodeterminethereliabilityofapeer,leavingtheproblemofevaluatingdirectexperiencetotheuserorrelyingoncharacteristicssuchasresponsivenesswhicharetrivialtomeasure.Thosestudieswhichdoevaluatedirectexperiencetendtochooseconsensus-basedapproachesontheas-sumptionthatproblemse.g.compromisedpeersorsensingfaultswilloccurinisolation,makingtheprobabilityofaninaccurateconsensusforlargescalemulti-agentsystemsandsensornetworksverysmall.Unfortunatelythisisnotthecasefortheenvironmentalcausesofsensinganomalieswhichwillaffectallidenticalsensorswithinrange.Asimilarexampleformulti-agentsystemswouldbeacomputervirusorwormwhichspreadsquickly,providingfalseinformationfrom12

PAGE 27

Table2.Categorizationofrelatedworkaddressingsensingproblemsingeneralbytask(columns)andmethod(rows). Sourceorsensingassessment(Sec.2.1) SensorFDI(Sec.2.2) Method Estimate sensing accuracy Estimate sourcereliability Ranksources byreliability Detectsensingproblems Diagnose sensing problems Isolatepoorly sensedregions (Sec.2.3) Adaptive fusion (Sec.2.4) # Inconsistency-based Thiswork Thiswork Thiswork Afgani (2008a,b) Thiswork Roman(2006) Ayrulu (2002) 5 Consensus-based Barber(2003) [4] Shayer (2002) Soika (1997a,b) Soika (1997a,b) Yu(2006) [3] 10 Consensus-based (pair-wise) Ganeriwal (2008)[2] Pon(2005) Bacon (2001)[2] Bacon (2001)[2] Delmotte (1998) 6 Learnedmodels (uninformed) Yu(2003) Christensen (2008)[11] Christensen (2008)[11] 12 Inconsistency-based (trustedsource) Momani (2008)[2] Bank(2002) Bank(2002) Baltzakis (2003) Morales (2008)[2] 6 Qualitativemodels Williams (1996)[2] Williams (1996)[2] 2 Stochasticmodels Dearden (2004)[11] Dearden (2004)[11] 11 Learnedmodels (informed) Plagemann (2007)[6] Plagemann (2007)[6] 6 Analyticalmodels Ryutov(2007) Monteriu (2007)[9] Monteriu (2007)[9] 10 Measuredsensor characteristics Hongjun (2008) Ganeriwal (2004)[13] 14 Knownsensorcharacteristics Wang (1999) Sivaraman (2007) Kobayashi (1999) 3 Unspecied Wang(2007) 1 Total 3 23 3 43 40 3 8 13

PAGE 28

alargenumberofinfectedagents.ContinuingtrendsinsensorFDIaretofocusoncreatingevermoreexpressiveandexibleanalyticalorstochasticmodelstoimproveperformanceortoincreasethebreadthofdetectableanddiagnosablesensingproblems,butthistrendisunlikelytoextendtoincludesensinganomalieswhicharedifculttomodelduetodependenceonanunknownen-vironment.RecenttrendsinsensorFDIhaveshownanincreasedinterestinpureuninformedlearningapproacheswhicharebettersuitedtohandlesensinganomaliesthanthemorepopularmodel-basedapproachesbutarestilllimitedtodetectinganddiagnosinganomaliesthataresimi-lartothoseencounteredduringtraining.Thisworkisamongveryfewstudiesthatapplythemostsuitablemethodinconsistency-basedforsensinganomaliesandisoneoffewtoaddresstheprob-lemsofestimatingsensingaccuracy,rankingsourcesbyreliability,andisolatingpoorlysensedregionsintheenvironment.Section2.5presentsmetricsdevelopedtomeasureinconsistencyinasystemthatmodelsun-certaintyquantitatively,i.e.probability,possibility,orevidentialDempster-Shafertheories.SinceDempster-Shafertheoryisnotasprevalentasprobabilitytheoryorfuzzysettheory,thissectionopenswithabriefreviewofDempster-ShafertheoryinSection2.5.1.Therearetwoformsofin-consistencythesemetricsattempttomeasure:theamountofevidencesupportingcontradictoryhypothesesandthedegreeofinconsistencywithinamodeltheseoftenreducetoentropyorbe-tweentwomodels.Themetricswillbeevaluatedbasedonthreecriteria:applicabilitywhichde-scribestherangeofuncertaintymodelswithinthedomainofthemetric,complexitywhichde-termineshowefcientlythemetriccanbeappliedtofusedsensorreadings,andexposurewhichdistinguishesprovenmetricsfromthosethathaveonlybeenappliedtotheoreticalexamples.2.1SourceorSensingAssessmentThissectionpresentsapproachesforevaluatingsensorsorothersourcesofinformationtoidentifyoratleastreducetheinuenceofsensorsthatareaffectedbysensingproblems,withaparticularfocusonassessmentmethodsthatdonotrelyonaprioriinformation.Inthissectionthreeindividualsensingassessmenttasksareconsidered:estimatingsensingaccuracywhichas-sessessensingsystemsasawhole,estimatingsourcereliabilitywhichestimatestheprobabilitythatasourceisreliablei.e.cooperativeoraccuratedependingonthemethodused,andrankingsourcesbyreliability.Sincenoneofthesestudiesaddressessensinganomalies,theseapproaches14

PAGE 29

arecategorizedandevaluatedaccordingtothemethodusedtoperformtheassessmentasopposedtoexperimentalresults.Indecreasingorderofsuitabilityforsensinganomaliesthesemethodsareasfollows:inconsistency-based,consensus-based,consensus-basedpair-wise,learnedmodelsuninformed,inconsistency-basedusingatrustedsource,analyticalmodels,measuredsensorcharacteristicse.g.qualityofserviceorQoS,andknownsensorcharacteristicse.g.classi-cationaccuracy.Recenttrendsinthemulti-agentsystemsandsensornetworksliteraturehaveshownanincreasedinterestinestimatingsourcereliabilitybutmostoftheseofthe23studiesfoundarefocusedonmeasuringsourcecooperativenessbyrelyingonmeasuredcharacteristicssuchasQoSoruserassessmentsofoutcomes.Thosethatconsidersourceaccuracytypicallyem-ployconsensus-basedmethods,assumingthatinlargescalesystemsanaccurateconsensuswillalwaysbeavailable.Thisassumptionisinvalidfortheenvironmentalcausesofsensinganomalieswhichoftenaffectmultiplesensors.Thisworkaddressesthetwoothertasks:estimatingsensingaccuracyandrankingsourcesbyreliabilitywhichhavereceivedlittleattentionintheliterature.Existingapproachesforestimatingsensingaccuracyeitherrelyonirrelevantcharacteristicse.g.packetdroppingstatisticsoraprioriknowncharacteristicsi.e.classicationaccuracyforas-sessment.Consensus-basedapproacheshavebeenappliedtotheproblemofrankingsourcesbyreliabilitybutagaintheserelyonanaccuratemajorityofsourcestoproduceanaccurateranking.ThissectionisorganizedaccordingtothecategorizationofsensingassessmentstudiesgiveninTable3whichliststhestudiesrstbrokendownbytask,thensortedaccordingtotheprimarymethodusedinthestudyforassessment.Section2.1.1reviewsexistingapproachesforestimat-ingsensingaccuracy.Section2.1.2discussesapproachesthatestimatesourcereliability.Sec-tion2.1.3isfocusedonapproachesforrankingsourcesbyreliability.Thesesectionspresentthestudiesinorderbymethodfromthemosttotheleastrelevantfortheproblemofsensinganoma-liesinunknownenvironments.StudieswhichcombinemethodsarelistedinmultiplerowsinTa-ble3butappearinthissectiononlyonce,accordingtotheprimarymethodemployedi.e.wherethestudyisnotlistedinitalics.2.1.1EstimatingSensingAccuracyTwostudiesfoundintheliterature,likethisstudy,estimatetheaccuracyofasensingsystemasawhole.Unlikethiswork,neitherisappropriateforestimatingsensingaccuracyinthepres-15

PAGE 30

Table3.Relatedworkonsourceorsensingassessmentbymethod.Studieswhichcombinemeth-odsarelistedinmultiplerowswiththeprimarymethodinnormalprintandsupplementalmethodslistedinitalics. Method Studies EstimatesensingaccuracySection2.1.1 Inconsistency-based Thiswork MeasuredSensorCharacteristics Hongjun KnownSensorCharacteristics Wang EstimatesourcereliabilitySection2.1.2 Consensus-based BarberandKim,2003BarberandFullamRyutov Consensus-basedpair-wise DragoniGaneriwal LearnedModelsuninformed Yu Inconsistency-basedtrustedsource HurMomani AnalyticalModels Ryutov MeasuredSensorCharacteristics Yu,2003SabaterGaneriwalDong-HuynhSunTeacyReeceStefanWangChenKhos-ravifarLetia KnownSensorCharacteristics Sivaraman Unspecied WangandSingh RanksourcesbyreliabilitySection2.1.3 Inconsistency-based Thiswork Consensus-based Shayer Consensus-basedpair-wise Pon enceofsensinganomaliesduetorelianceonmeasuredcharacteristicsi.e.cooperationthathavenothingtodowithaccuracyHongjun,Zhiping,andXiaona,2008ortheexclusiveuseofapri-oriknowncharacteristicsi.e.classicationaccuracyforassessment.InHongjun,Zhiping,andXiaona,2008theproblemofuncooperativenodesisaddressedwhileWangandShen,1999isconcernedwithambiguityindecision-makingsystems.Hongjun,Zhiping,andXiaonapresentanapproachforestimatingthereliabilityofawirelesssensornetworkbasedonpropagationofmeasuredcharacteristicsdescribingtherelia-bilityofindividualnodesdenedastheprobabilityofcooperationoncommunicationtaskssuchasforwardingpackets.TheapproachusesShannon'sentropyHxtoevaluateawirelesssen-sornetworkasawholebyapplyingitdirectlytotheprobabilitythatnodeAwillcooperatee.g.forwardpacketsorprovidereadingsforaggregationwithnodeBforallpossiblepairsofnodesinthenetwork.Correctmethodsforpropagatingtrustinthenetworkareexplored.Experiments16

PAGE 31

wereperformedwithsimulatednodesandrandomlygeneratedpvalueswithupto10%maliciousnodes.Theresultsshowthatwhenmaliciousnodesconsistentlylieaboutthetrustworthinessortheirneighborstheuntrustworthinessofthenetworkincreases,relativetothemagnitudeofthelie.WangandShenpresentatheoreticalapproachtoevaluatetheprobabilityofacor-rectclassicationformultiplehypothesisdecision-makingsensingsystemsbasedontheaprioriknownprobabilityofaccurateclassicationforeachsensor.Sensormodelstaketheformofcon-ditionaldensityfunctionswhichstatetheprobabilityofacorrectdecisionifoneofnhypothesesareassumed.Theprobabilitiesofmultiplehypothesesaremappedintoadecisionspacebasedonthesensormodels.Threesensingqualitymetricsarederivedfromthisspace.Thetotalprobabil-ityofacorrectdecisionmeasurestheareainthedecisionspacewhereanobservationcanonlybemappedtothecorrecthypothesis.Similarlytheprobabilityofanincorrectdecisioncorrespondstoregionswhereobservationsaremappedtoanincorrectorsetofincorrecthypotheses.Theremain-ingareaofthedecisionspaceistheprobabilityofanuncertaindecision.Noexperimentalresultswerereported.2.1.2EstimatingSourceReliabilityThissectiondiscussesstudieswhichestimatetheprobabilitythatasourcesensororagentwillbereliablei.e.cooperativeoraccuratewithoutrelyingonaprioriinformationalthoughthemostpopularapproachesseeforexampleGaneriwalandSrivastava,2004relyonmeasuredchar-acteristicslikequalityofserviceQoSwhichareunrelatedtoaccuracy,oruserassessmentsofoutcomesseeforexampleTRAVOS,Teacyetal.,2006.Thesestudiesarefocusedonwhatiscalledsoftsecurityformulti-agentsystemsandsensornetworkswheremaliciousorfaultysourcesi.e.sensorsbecomeisolatedaspeerslearnnottotrustthem.ConsensusbasedapproachesBar-berandFullam,2003;BarberandKim,2001,2003;DragoniandGiorgini,2003;Ganeriwal,Balzano,andSrivastava,2008;RyutovandNeuman,2007relyonanaccurateconsensusand/ormajorityofsourcestoensurefaithfulassessmentsofpeers.Forexample,DragoniandGiorgini,2003showedthatthemeasuredreliabilitywasinverted,i.e.reliableagentswerelabeledasunre-liableandviceversa,whenaminorityofagentswereaccurate.Anaccuratemajorityorconsensuscannotbeassumedinanunknownenvironmentinthepresenceofsensinganomaliesaffectinganunknownpercentageofsensors.Theremainingstudiesdonotexplicitlyconsidersourceaccu-17

PAGE 32

racybutaremoreinterestedinestimatingreliabilityintermsofcooperativenessandresponsive-ness,relyingonmeasuredcharacteristicssuchasQoSorpacketdroppingratesGaneriwalandSrivastava,2004;LetiaandSlavescu,2008;Schmidtetal.,2007;Sunetal.,2006;YuandSingh,2002,2003aoronuserassessmentsofoutcomesDong-Huynh,Jennings,andShadbolt,2006;Khosravifaretal.,2008;Reeceetal.,2007;SabaterandSierra,2002;Teacyetal.,2006;Wangetal.,2007.Sincethesemeasuredcharacteristicshavenothingtodowiththeaccuracyoftheinfor-mationprovidedtheseapproachescannotbeusedtoestimatesensorreliabilityinthepresenceofsensinganomalies.InWangandSingh,2007ageneraltheoreticalframeworkformanipulatingtrustvaluesiscreatedwithoutregardforsourceassessmentmethods.Sourcereliabilityisanimportantconsiderationinthegeneraleldoffusionwhichincludes,forexample,beliefrevision,databaseandknowledgebaseupdating,combiningknowledgebases,andrequirementsengineeringAppriouetal.,2001.Inmanyapplicationsoffusionhigh-levelobservations,preferences,orrulesarecombinedandthegroundtruthcannotbedeterminede.g.requirementsengineeringDick,Hull,andJackson,2004.Asaresult,thoseapproachesthatas-sesssourcereliabilityintermsofaccuracytypicallyuseaconsensus-basedapproachwhichdoesnotrelyonaprioriinformation.Themeasuredreliabilityisusedtodeterminetheinuenceofin-formationfromthatsourceinfutureupdates,toguidesourceselection,andprovidesearchmetricsforinformationgathering.Barberandhercolleaguesuseprobabilisticmodelsandconsensus-basedapproachestomea-surethereliabilityofsourcesinanefforttoisolatemaliciousorincompetentagentswithinagroupofcooperatingsoftwareagents.Figure1showsthisauthor'sinterpretationofthesystemdescribedinBarberandKim,2001formaintaininganagent'sknowledgebasewhileprovidingsoftsecu-rityisolatingunreliableagents.IntheexampledepictedinFigure1theagent'suserhasmadeaninquiryregardingthelogicalstatementq.InresponsetheagentqueriesitstrustedsourcesotheragentsinthesystemanditsknowledgebaseKB.ItthenusesBayes'theoremtocombineallstoredKBandnewmessagestocalculateanewprobabilityforq,takingintoaccounteachsource'sreportedcondenceinqandtheirreliabilitytheprobabilitythattheywillprovidecorrectinformation,orPSi.Theagent'sworkingknowledgebaseKisderivedasthemaxi-malconsistentsetoflogicalstatementsbasedontheupdatedprobabilitiesinKB.Bayes'theoryisagainusedtoupdateeachsource'sreliabilitybasedonagreementwithK.Ifagivensource's18

PAGE 33

Figure1.Asourcereliabilityestimationapproachformulti-agentsystemsfromBarberandKim,2001.messageagreeswithKe.g.thesourceassertsqandP0q=1:0thenitsreliabilitywillin-crease.Ifthesourcedisagreese.g.assertsqandP0q=0:0thenitsreliabilitywilldecrease.TheamountofchangeinPSidependsonthesource'scondence,wherelowercondenceresultsinsmallerchanges.InBarberandKim,2003thisapproachwasextendedbyallowinganagenttodetermineagivensource'sreliabilitythroughbothdirectinteractionasinBarberandKim,2001andbyreputationwheretheagentqueriesitspeersfortheiropinionofthesource.Simulatedtargettrackingexperimentswereperformedtocomparethetwoapproachesformain-tainingsourcereliability.Thedirectinteractionapproachwasshowntobelesssusceptibletonoisebuttookmoretimetosettleonthecorrectvaluesforeachsource.InBarberandPark,2004thisapproachisusedtodevelopsearchstrategiesforndingbettersourceswhentheoverallsourcereliabilitydrops.BarberandFullamextendthisbeliefrevisionsystemfurthertohandlediscreteandcontinuousvaluesbyusingaGaussiandistributiontomodelagents'condenceininformation.Herethereliabilityofanagentforasingleupdateismeasuredastheintersectionbetweenthesourceagent'scondencedistributionandthecondencedistributionforthefusedresult.Theagents'reliabilityisassumedtobestaticovertimesoasimpleaverageisusedtode-19

PAGE 34

scribeanagent'sreliabilityoverninteractions.Aseriesoffourexperimentswithasimulatedtar-gettrackingmulti-agentsystemvariedthepercentageofreliableagents,thenumberofagentsinthesystem,themeanerror,andstandarddeviationerrorforunreliableagentsrespectively.Theresultsshowedstatisticallysignicantimprovementsinknowledgebaseaccuracyascomparedtothesamethebeliefrevisionapproachwithoutreliabilitymodeling,withtheexceptionofcaseswheretwo-thirdsof9ormoreofthesourceswereunreliable.Recentworkbythisgrouphasfocusedonthetrade-offbetweenexperienceandreputationbasedtrustmodeling.InFullamandBarber,2007Q-learningisappliedtoalinearcombinationparameterwhichdeterminestherela-tiveinuenceofexperienceandreputationmodelsonadecisionwhetherornottointeract.Feed-backforlearningisprovidedbythepayoffforeachtransactioncompleted.Experimentalresultsshowedthatthelearningapproachperformedbetterthanrelyingonexperienceorreputationex-clusively.Q-learningwasalsousedinAhnetal.,2008;Ahn,DeAngelis,andBarber,2008todeterminetheoptimalcombinationofalargergroupofagentattributesincludingreliability,qual-ity,availability,andtimelinessforteamselection.TheapproachforsourceassessmentdevelopedbyBarberandhercolleaguesreliesonanaccurateconsensustofaithfullyassessthetrustworthi-nessofotheragents.Suchaconsensuscannotbeassumedforadistributedsensingsystemusedinanunknownenvironmentinthepresenceofsensinganomaliesaffectinganunknownpercentageofpeers.RyutovandNeumanpresentahybridconsensus-basedapproachtondcompromisedorfaultysensornodeswhichreliesonanalyticalmodelstoextendthesetofcomparablesensors,enablingdetectionandisolationofsensingproblemsevenwhenanentiregroupofidenticalnodesarecompromised.Theapproachisdesignedforremotesensornetworkmonitoringwherethenet-workisusedtoprotectandmanageanindustrialprocesswithknownphysicalcharacteristics.Asuspicionlevelforeachsensorisperiodicallyincreasedordecreasedaccordingtoitsbehavior,suspiciousornormalrespectively,andusedtoreducetheinuenceofcompromisedorfaultysen-sorsonthestatusreportedbythemonitoringsystem.Forevent-reportingsensorsallsensorsre-portingthesameattributee.g.leakdetectionsensorsinstalledincloseproximityareplacedintotwogroupsaccordingtowhetherornottheyreportedtheevent.Whetherornottheeventoccurredisdeterminedbythevoteofthegroupwiththelowestcumulativesuspicionlevel.Agreementwiththisconsensusresultisusedtoincreaseordecreaseeachsensor'ssuspicionlevel.Forsensorsthat20

PAGE 35

reportcontinuousvaluese.g.owthroughapipeaweightedaverageusingthesuspicionlevelasweightsisusedtodeterminetheconsensusvalue.Thedifferencebetweentheconsensusandanindividualsensor'sreadingisusedtodeterminewhetheritsbehaviorissuspiciousornormal.Physicalrelationshipsbetweensensorsthatmeasuredistinctaspectsoftheindustrialprocessaregivenasasetoftrainingexamples.Theseexamplesareusedtolearnadecisiontreewhichde-termineswhichgroupsofsensorsallmeasuringthesamepropertytotrustbasedonwhichrela-tionshipsdoanddonotholde.g.ifowsensorsateachendofapipereportthesameordifferentvalues.Ifthedecisiontreecannotprovideanunambiguousansweronwhichgrouptotrust,thenfurtherstepsaretaken.Firstthemeansuspicionlevelforeachgroupiscalculated,andiftheyaresufcientlydifferent,thenthegroupwithhighersuspectlevelsisassumedtobelying.Ifthesus-picionlevelsaresimilarthenthegroupreportinglowerprobabilityvaluesusingaGaussiandistri-butionareassumedtobelying.Byleveragingknowledgeoftherelationshipsbetweenattributesthesensorswereinstalledtomonitor,thisapproachprovidesaninformed,consensus-basedeval-uationofnotjustindividualsensorsbutwholegroupsofsensorswhichcouldbecompromisedbyanattacker.Thesystemwasstillindevelopmentandnoexperimentalresultswerereported.DragoniandGiorginipresentabeliefrevisionsystemsimilarinstructuretoBarberandKim,2001wherethereliabilityofanagentisdeterminedbypair-wisecomparisonsoflogicalstatementsmadebyallagentstodeterminethelikelihoodthatasourcebelongstoaconsistentsetofagents.InDragoniandGiorgini,2003logicalstatementsareexchangedamongagentsandanevidentialknowledgebaseKBismaintainedusingDempster-Shafertheory,wherethesource'sreliabilitydeterminesthebeliefassociatedwithnewevidencefromlogicalstatementsassertedbythatsource.UnlikeBarber'sapproachwhichisentirelynumericuntiltheworkingknowledgeKisderived,thereisacleanseparationoflogicalandnumericproceduresandastrongerem-phasisonlogicalbeliefrevision.TheevidentialKBisusedtoenablenon-monotonicreasoning.Thereliabilityofagivensourceismodeledprobabilisticallyasthelikelihoodthatthesourcebe-longstoasetoflogicallyconsistentsourcesallthestatementsprovidedbysourcesinthissetarecollectivelyconsistent,withoutregardtothenumericbeliefassignedwitheachstatement.Anex-tensivesetofexperimentalresultsexaminingarchitecturalcentralizedversusdecentralizedissuesandcommunicationstrategieswerepresentedbasedonsimulationsofveagentssuppliedwitharbitrarylogicalstatements.Theagentreliabilityresultswereonlyaccuratewhenamajorityof21

PAGE 36

agentswereinfactreliable.Whentheaveragetruereliabilityofthesimulatedagentsfellbelow0.5themeasuredreliabilitywasinverted,i.e.reliableagentswerelabeledasunreliableandviceversa.Theonlyexperimentsinwhichthiseffectdidnotoccurinvolvedtheuseofanoracle,aper-fectlytrustedagentthatonlyreportedcorrectinformation.Ganeriwal,Balzano,andSrivastavacontinuedevelopmentofareputationmiddlewarefordistributedsensornetworkswhichusesapair-wiseconsensusapproachbasedonagenericdis-tancemetrictoevaluatenodereliabilityinthepresenceofcommonsensorfailures.InGaneriwal,Balzano,andSrivastava,2008thepopularapproachinGaneriwalandSrivastavaisex-tendedbyaddingagenericdistance-basedwatchdogmechanismtoevaluatereal-valuedRsensorreadingsandallowingcontinuousasopposedtojustbinaryinputstothereputationsystem.ThelatterisdonebygeneralizingfromthedistributiontoaDirichletProcessandusingtheoutputfromthewatchdogmechanismover[0;1]toupdateparametersand.Ganeriwal,Balzano,andSrivastavaalsoproposeintegratingpeerreputationsbyleveragingtheapproachproposedinJsangandIsmail,2002toapplyDempster-Shaferdiscountingintheprobabilisticframework.Thisresultsinstraightforward,closedformequationsforcombininglocalandpeerreputationswiththelatterweightedbytheirrespectivereputations.Experimentswithrealandsimulatedtem-peratureandhumiditysensorsdidnotincludepeerreputationintegration.Theresultsshowedac-curateidenticationawidevarietyofcommonsensorfailuresamonggroupsofsensorsmeasuringthesamevalue.TheapproachespresentedinHuretal.,2005;Momani,Challa,andAlhmouz,2008estimatethereliabilityofpeernodesinasensornetworkbycomparingthevaluesprovidedbythenodeswithlocallysensedvalues,implicitlytrustinglocalsensingaccuracywhichisnotasafeassump-tioninthepresenceofsensinganomalieswhichmayaffectasensornodeandallitsneighbors.Momani,Challa,andAlhmouzillustratetheimportanceofconsideringbothcommunica-tionanddatareliabilityinwirelesssensornetworks.TheyapplytheapproachfromGaneriwal,Balzano,andSrivastava,2008totheproblemofcommunicationreliabilitymodelinganddeveloptheirownreliabilitymetricfordataintegrityusingaGaussiandistributiontocapturetheerrorincontinuousreadings.Togeneratethereliabilitymetriceachnodetrustsitsownvalueandevalu-atesneighboringnodesaccordingtothedifferencebetweentheirreadingsanditsown,generatingameanandvariancetodeneaGaussiandistribution.Anodewillalsoqueryitsneighborsfor22

PAGE 37

assessmentGaussiandistributionofthetargetnodeandallofthesevaluesarecombinedusingtraditionalstatisticalmethodsintoanewGaussiandistribution.Theoverallreliabilitymetricmod-elstheaccuracyofeachnodebasedonbothmeanerrorandvariancevalues.Simulationresultsshowhownodeswithgoodcommunicationbehaviorcanhavepoordatareliabilityandviceversaandtheresponseofthecommunicationsanddatareliabilitymetricsinthesecases.Theproblemofcombiningthesetwometricsintoacommonreliabilitymetricisleftforfuturework.Huretal.presentanapproachforimplementingaresilientsensornetworkbasedonmeasuringthetrustworthinessofneighboringnodes.Anode'strustworthiness)]TJ/F15 10.909 Tf 8.484 0 Td[(1;1isbasedonaweightedcombinationoftheratioofinconsistenttoconsistentmeasurements,theratioofsensingeventmissesversushits,anditsbatterylevel.Todetermineifareadingfromaneighboringnodeisin-consistent,theevaluatornodecalculatesanacceptablerangeofvaluesbasedonitsownsensinginformation.Anyreadingsthatfalloutsideofthatrangeareconsideredinconsistent.Thenodesarearrangedonaregulargridwhereeachcellisassumedtocontainmultiplenodes.Themosttrustednodewithineachcellactsasanaggregator,whichcomputesaweightedaverageofthesen-sorreadingsfromeachnodeinthecellusingtheirrespectivetrustworthinessasweightsandre-portsthevaluetothenetworksink.Thesinktakesthemedianofthevaluesreportedbyeachofthegridcells'aggregatorsasthenalvalue.Experimentswith300simulatedtemperaturesensingnodescomparedtheapproachpresentedinHuretal.,2005withsimpleaveragingandmedianaggregationmethods.Theresultsshowthattrust-basedaggregationismoreresilienttoattacksorfailuresofupto25nodes.YuandSingh,2003ausemeasuredcharacteristicsi.e.cooperativenesstomodelthereliabilityofsources,withuninformedlearningQ-learningusedtoadjustreliabilityvaluestomatchactualoutcomesofinteractionswithagents.InYuandSingh,2002reliabilityrepresentstheprobabilitythatanagentwillprovideagivenqualityofserviceQoSandismodeledusingDempster-Shaferbeliefmasses.PredeterminedthresholdsareusedtomapQoSvalueselicitedfromusersintobeliefmassesforthehypotheses:fcooperative;uncooperative;unknowng.Be-liefmassesarecombinedusingDempster'sruleofcombination.Anagent'sreliabilityisbasedonbothdirectinteractionandbyreputation.Experimentswithsimulationsofmulti-agentcommu-nitiesor100agentsshowedefcientisolationofuncooperativeagentsandstableadaptationtochangingQoSinlargercommunitiesagentswithupto20%uncooperativeagents.InYu23

PAGE 38

andSingh,2003athisapproachisexpandedtohandletheproblemofdeceptiveagentsagentsthatlieaboutthereliabilityoftheirpeers.Learnedweightsover[0,1]areappliedtotheinforma-tiongatheredfromeachagent.Q-learningbasedonthepredictedQoSandtheQoSfromdirectinteractionwithanagentisappliedtoupdatetheweights.Experimentalresultsshowedaccuratedetectionofdeceptiveagents.GaneriwalandSrivastavapresentapopulardesignofareputationmiddlewarefordis-tributedsensornetworkswhichhelpstoidentifyandisolatemaliciousorfaultynodesbasedonmeasuredcharacteristicsi.e.uncooperativebehavior.ThestructureofthesystemissimilartothatinYuandSingh,2002wherebothlocalreputationbuiltfromdirectinteractionandglobalreputationbuiltfrombothdirectandindirectinteractionaremaintained.Bothsystemsalsousethereputationofanodetoweighttheimpactofthatnode'srecommendationregardinganothernode,i.e.reputationinformationfromacompletelyuntrustednodehasnoimpactontheglobalreputation.GaneriwalandSrivastavausethedistributiontomodelandupdatethereputationofneighboringnodes.UsingaBayesianapproachthistwo-parameterdistributioncanbeupdatedsimplybyaddingincomingvaluese.g.numberofcooperativeversusuncooperativeinteractionstotheexistingvalues,enablingenhancementslikeanagingfactorwhichgraduallydiscountstheimpactofearlierinteractions.Largeportionsofthesystemwerestillindevelopmentwhenthepaperwaswrittenincludingthewatchdogmodulewhichassessesneighboringnodesbutsomepreliminarysimulationresultswereincluded.Theseshowedgoodresiliencytocommonattacksfoundinexistinge-commercesystemse.g.teamingupwithuserstounderratecompetingnodes,ortoinateone'sownreputationanddemonstratedisolationoffaultyormaliciousnodesinsmallnetworksto6nodes.Chenetal.presentasystemsimilartoGaneriwalandSrivas-tava,2004fordetectingpacket-droppinganddenial-of-serviceattacksfromcompromisednodesinwirelesssensornetworks.Usingthesamemetricsfornodeevaluationandthedistributiontomodelreputation,theymodifytheoutcometoexplicitlyquantifyuncertaintyintheevaluation.Ex-perimentalresultsfromsimulatednetworksshowedthatthisenhancementleadstoquickeradapta-tiontoatrustednodethatbecomesmalicious.SabaterandSierrapresentasystemforassessinganagent'strustworthinessknownastheREGRETsystemwhichreliesontheusertodeterminetheoutcomeofinteractions.Thegoalofthesystemistoapplyatypelabelwhichdescribesinhumantermstheexpectedbehaviorof24

PAGE 39

anagent,e.g.honest,swindler,ortardydeliverer.Inthisdistributedapproach,eachagentusesallavailableinformationincluding:directexperience,reputationdirectexperienceofotheragents,associationexperiencewithagentsheavilylinkedwiththetarget,roleinanorganization,andon-tologicalcombinationsofthesetodetermineanotheragent'stype.Trustworthinessaccordingtodirectexperienceismeasuredbytheoutcomeofinteractionsalongagivensetofattributesprice,qualityofservice,etc..Thereliabilityofthistrustworthinessvalueandalltofollowisdepen-dentonthenumberofexamplesdirectinteractionswiththetargetandthevarianceoftheout-comes.Thereputation-basedtrustworthinessisbuiltbyqueryingtheopinionofasetofrepresen-tativeagentswithinagivensocialnetworkandusingtheagent'strustinthewitnessestoweighttheiropinioninthenalreputation-basedvalue.Thetrustworthinessofawitnessisdenedei-therbytheoutcomeofdirectinteraction,orifthatoutcomeisunreliable,fuzzyrulesonthesocialrolesoftheagent,witness,andtargetareusedtodeterminetrustworthinesse.g.ifthewitnessandtargetarecompetitorsthenthewitness'sopinionisexpectedtobebiasedlow.Trustworthinessbasedonassociationisjustalinearcombinationofthetrustworthinessofthetarget'sneighborsagentssimilartothetarget.Therole-basedtrustworthinessisahard-codedheuristicbasedontheagent'sopinionofindividualsineachroleandisgivenlowreliability.Allthesetrustworthinessvaluesarecombinedaccordingtoapredeterminedontologywithweightsassociatedwitheachlinkthatrelatecharacteristicswithtrustworthinessattributese.g.honest,swindler,defaulter,etc..Asufcientlyrichagentenvironmenttosupportthissystemwasstillindevelopmentthereforenoexperimentalresultsweregiven.Jenningsandhiscolleagueshavedevelopedseveralsophisticatedsolutionstotheproblemofestimatingtrustinmulti-agentsystemswhichrelyonuserfeedbacktodeterminetheoutcomeofinteractions.Dong-Huynh,Jennings,andShadboltpresenttheFIREsystemforcom-prehensiveassessmentsofagenttrustworthinessinmulti-agentsystems.Theapproachusesfoursourcesofinformation:directinteractionexperience,witnessesreputation,role-basedassign-ment,andwhatiscalledcertiedinformationprovidedbythetrusteeagent.TheapproachfromSabaterandSierra,2002isusedtomodeltrustfromexperienceandthereferralstructuredevel-opedinYuandSingh,2003bisusedtondwitnessesinanetworkofagents.Domain-specicrulesdeterminetherole-basedtrustvalue.Certiedinformationisprovidedbythetrusteeagentincludingreferralsfromotheragents.Theoveralltrustvalueforthetrusteeagentiscalculatedas25

PAGE 40

aweightedaverageofthetrustvaluesfromeachsourcewheretheweightsareafunctionofthereliabilitybasedonrecencyandconsistencyandrelativeimportanceofthetrustvaluegivenbythatsource.ExtensiveexperimentswithsimulatedagentswhichexhibitbothstaticanddynamicmisbehaviorshowsignicantimprovementsinperformanceascomparedtoSPORAS,asophis-ticatedcentralizedagenttrustsystem.Teacyetal.presenttheTRAVOSsystemforassess-ingagenttrustworthinessinmulti-agentsystems.Theapproachusestwosourcesofinformation:directinteractionexperienceandreputationbasedonpeers'directinteraction.Eachinteractionisclassiedbytheuseroranothermoduleassuccessfulorunsuccessful.Thetrustee'sreliabilityTAismodeledastheprobabilityofasuccessfulinteractionbasedonpastexperienceandismod-eledusingthebetadistributionwithparameters,equaltothenumberofsuccessfulandunsuccessfulinteractionsrespectively.Anagent'scondenceinagivenreliabilityvalueismod-eledastheproportionofthecalculateddistributionthatliesbetweenTA)]TJ/F18 10.909 Tf 11.486 0 Td[(andTA+,forauser-denedmarginoferror.Foraslongasanagent'scondenceinitsestimateofthetrustee'sreliabilityistoolow,itwillqueryotheragentsinthesystemtogathermoreinformationaboutthetrusteeandanewdistributionDristhenbuiltbysummingthereportedinteractionoutcomesweighedbytheagents'expectedaccuracy.Experimentswereperformedwith101simu-latedagentswhichprovidedeitheraccurate,noisy,orveryinaccurateopinions.TheresultsshowedthatTRAVOSachievedlowererrorratesascomparedtoasimilarreputationsystemproposedbyJsangandIsmail,especiallyasthenumberofinteractionsincreased.Reeceetal.extendtheseapproachestoincludereal-valuedRevaluationsofinteractionoutcomesandhet-erogeneouscombinationsofevaluationsforbundledoutcomes.ByusingtheDirichletProcess,realvaluedevaluationsofinteractionoutcomescanbecombinedinthesamefashioni.e.summa-tionasfusingmultiplebinaryoutcomesusingthedistribution.Inadditiontothisenhancement,theapproachutilizesestablishedmethodsfromtheeldofdatafusiontocombineevaluationsre-gardingsomeorallserviceswithinabundledcontracte.g.video,audio,anddataservices.Ex-perimentalresultsusingcontractoutcomesrandomlygeneratedfromajointdistributionoftwoservicesvalidatestheimprovedaccuracyoftheapproachascomparedtomodelingeachserviceseparatelyasanindependentDirichletProcess.Theseapproachesleaveassessmentofaninter-actiontotheuseroranothermodulemakingthemunsuitedforunsupervisedhandlingofsensinganomaliesinunknownenvironments.26

PAGE 41

Khosravifaretal.presentatrustmanagementsystemforautonomousagentswhichreliesonuserclassicationoftheoutcomeofinteractionswithotheragents.Thesystemtakesthefollowingintoaccount:directinteraction,reputation,volumeofinteraction,andrecencyofinformation.Theoutcomeofagiveninteractionisclassiedbyanothermoduleintoadiscretesetofpossibleoutcomes,asubsetofwhichisconsideredpositive.Trustbasedondirectinterac-tionismeasuredastheratioofinteractionswithpositiveoutcomestothetotalnumberofinter-actions,weightedbytheimportanceassignedtoeachinteraction.Onlythislocalinformationisexchangedbetweenagents.Reputationisgatheredfromtrustedagents,andifaninsufcientnumberofthesereplybecausetheydonotwanttoorbecausetheyhavenoinformationthenthetargetisaskedforreferencestootheragentsthatcanrecommendthem.Trustaccordingtorepu-tationiscalculatedastheaveragegatheredreputationvalueweightedbythetrustvalueassignedtotheagentthatprovidedthereputationvalue.Anoveralltrustvaluecanbecalculatedbyalsoweightingthisvaluebythenumberofinteractionsthereferringagenthashadwiththetargetandhowrecentlythereputationvaluewasprovided.Thetrustworthinessofreferringagentsisalsoad-justedaftermoredirectinteractionwiththetargetbyadjustingtheirrespectivetrustvaluestomin-imizethedifferencebetweenthepredictedcombinedtrustvalueandtheactualtrustvalueobtainedthroughdirectinteraction.Aproof-of-conceptimplementationoftheapproachwasimplementedinJadex,theJava-basedagentsimulationframework.Noexperimentalresultswereprovided.Schmidtetal.presentauser-tunablefuzzyframeworkforacombinedexperienceandreputation-baseddistributedtrustsystemfore-commercebasedonthemeasuredoutcomeofcom-pletedtransactions.InthissystemthetrustassignedtoagivenagentisdeterminedbasedonbothpastexperienceandrecommendationsfromotheragentsRecommendingAgents.ThelatterisconditionedontheirassignedAgentCredibilityAC.Therelativeinuenceofthesetwosourcesonanagent'soverallopinionofthetrusteeagentisoneofmanyuser-tunableparameters.Iftheagentdecidestocompletethetransactionwiththetrusteeagent,thentheoutcomeoftheinterac-tioniscalculatedbasedonuser-denedcriteriaandtheagent'sopinionofthetrusteeisupdated.ThenewtrustvalueT0forthetrusteeagentisthencomparedtothevaluesT1:::nprovidedbytheRecommendingAgentsRA1:::n.TheACivalueforeachagentRAiisincreasedifTiiswithinauser-deneddistancefromT0anddecreasedotherwise.TheextentofthechangeinACidependsonthevarianceoverallrecommendationsreceived.Thatis,ifthetrustee'sreliability27

PAGE 42

changesfrequentlyovertimethentheresultsofasingleinteractionneitherincreasenordecreasetherecommendingagents'credibilitybyalargeamount.Illustrativeexamplesareprovidedshow-ingthestepstakenbyauser'sagentwhendealingwithanewagentandonethathasanestablishedreputationinthesystem.LetiaandSlavescupresentasysteminspiredbytheARTtestbedcompetitionsseeFullametal.,2005whereabootstrappingapproachisusedtolearnagentcompetencybasedoninaccuraciesinreportedcompetenciesversustheoutcomeofactualtransactions.Thegoalofthestudyistoreducetheimpactofisolateddishonestbehaviore.g.tit-for-tatretaliationonthepartofanotherwisecompetentagentontheestimatedcompetence.AgentreliabilityismodeledasaGaussiandistributionwithvariance2astheincompetence.Atraditionalprobabilisticsam-plingapproachforestimating2withfewdatapointsi.e.bootstrappingisused.Examplesshowslightincreasesin2valuesfortransactionswhereanagentisuncooperative.Thisleadstoveryhigh2fortrulybadagents,verylow2forcompetentandcooperativeagents,andonlyslightlyhigher2foragentsthatareuncooperativeonlyonrareoccasions.AgentswhichusetheproposedapproachintheARTtestbedwerestillindevelopment.Sunetal.proposeaprobability-theorybaseddistributedreputationsystemforadhocnetworkswhichmeasuresthefrequencyofuncooperativebehaviore.g.packetdroppingtodeter-minenodetrustworthiness.Theapproachutilizesasingletrustmetricfordistributedevaluationofnodesinadhocnetworksthatcapturestwofeatures:trustversusdistrust,anduncertainty.Usingentropy,themetricmapsagivenprobabilityover[0:0;1:0]whichrepresentsanarbitraryevalua-tionofthetargetnodetoatrustvalueover[)]TJ/F15 10.909 Tf 8.485 0 Td[(1:0;1:0]suchthathightrustp=1:0mapsto1:0,highdistrustp=0:0to)]TJ/F15 10.909 Tf 8.485 0 Td[(1:0,andhighuncertaintyp=0:5to0:0.Aprecisedenitionoftrustandaxiomstodeneitsbehaviorforadhocnetworksareestablished.Methodsforpropagat-ingeitherthetrustmetricortheunderlyingprobabilitythroughthenetworkarepresentedwhichweightarecommendationbasedonthesource'strustworthinessforrecommendationsasepa-ratelymaintainedtrustvalueandadheretothegivenaxioms.Concreteproceduresforevaluatingnodesbasedonpackettransmissionratesandminimizingoverheadwhenpropagatinginforma-tioninthenetworkarepresented.Experimentswithasimulatedadhocnetworkwith100nodesshowthatmaliciousnodesnodesthatdroppacketscanbedistinguishedfromgoodnodes.Theresultsalsoshowimprovedthroughputoftheadhocnetworkwhenmaliciousnodesaredetected28

PAGE 43

andavoidedbyneighboringnodes.NotethattheapproachinSunetal.,2006,whiledesignedforanadhocnetwork,isgenericandcouldeasilybeadaptedforuseinsensornetworks.Wangetal.provideaframeworkfortrackingthetrustworthinessofanagentovertimeusingdirectexperiencewhichisevaluatedbytheuserandrecommendationsfrompeersweightedbytheirtrustworthiness.Theapproachusestwotrustmetrics:Tsub2[0;1]whichisbasedsolelyondirectexperience,andTobj2[0;1]whichisbasedsolelyonrecommendationsfrompeers.AthirdmetriccalledtrustentropyTE2[0;1],denedasalinearcombinationofTsubandTobj,isusedtotrackdisagreementbetweenthesetwosources.Allthreemetricsareupdatedovertimebyalinearcombinationofthenewandcurrentvaluee.g.TE=TEn+1+)]TJ/F18 10.909 Tf 11.282 0 Td[(TEn.Theevaluationofagiveninteractionpositiveornegativeislefttotheuseroranothermodule.Anagent'sperformanceTsubinagiventimeframeiscalculatedasthepercentageofpositiveinter-actionsoverallinteractionsinthattimeframe.Therelativeinuenceofdirectexperienceversusrecommendationsonanagent'sdecisiontointeractwiththetrusteeisauser-denedparameter.PreliminaryresultsshowedthattheTEmetricdropssteeplywhenanodemisbehavesthenin-creaseswhenthenode'sbehaviorimproves.SivaramanandChangprovideanefcienttheoreticalsolutiontotheproblemofevalu-atingtheclassicationperformanceforasinglesensororsetoffusedsensorsbasedonprobabilitytheoryandtheaprioriknownclassicationaccuracyofthesensors.Theapproachrequiresapri-oriknowledgeoftheprobabilityofoccurrenceforeachtargettypetobeclassiedandthelocalconfusionmatrixLCMforeachsensor.TheLCMindicatestheprobabilitythatthesensorwillclassifyatargetase2ftarget typesggiventhatitobservesf2ftarget typesgforallpos-siblecombinationsoffe;fg.AglobalclassicationmatrixGCMisderivedfromeachsensor'sLCM.TheGCMdeterminestheprobabilityofaccurateclassicationforeachtargetafterkinde-pendentobservationsofthattarget.Itisshownthatthismatrixwillconvergetotheidentitymatrixaskincreasesonlyifnotworowsinthesensor'sLCMareidenticali.e.thesensorcandistinguishbetweenanytwogiventargets.MetricsarealsoderivedtodeterminetherateofconvergencefortheGCMandtheminimumnumberofobservationsrequiredtoachieveagivenclassicationac-curacy.Thesemetricsrevealthatthemostinuentialattributeisnottheaccuracyofdetectionofasingletarget,buttheabilitytodistinguishdifferenttargettypes.Furtheranalysisshowedthattheclassicationmetricforfusedsensorsissimplythesumofeachindividualsensor'sconver-29

PAGE 44

gencemetric.ExamplesusingarticialinputsshowedthattheproposedapproachprovidedbetterpredictionsofclassicationaccuracyandismorecomputationallyefcientthanMonteCarlosim-ulationswhicharethecurrentstate-of-the-art.WangandSinghfocusonarigorousunderstandingoftherelationshipbetweenevi-denceandcertaintyinreputationnetworksandleavetheproblemofevaluatingnodesunspecied.Towardthisend,aprobabilisticframeworksimilartoTeacyetal.,2006orGaneriwalandSri-vastava,2004isdenedtocaptureevidenceanduncertaintyforagivensetofinteractionswheretheoutcomeisbinaryeithersuccessorfailure.Thisframeworkprovidesthemathematicalfoun-dationsrequiredtodiscountevidenceduetosourceunreliabilityortheageofevidenceinacon-sistentfashionbyprovidingfunctionsandalgorithmstotransferevidencetouncertaintyspaceandviceversa.Nospecicapproachesforperformingthesecorrectionswereproposed.2.1.3RankSourcesbyReliabilityTwopapersfoundintheliteraturerankinformationsourcesbycomparingtheinformationfromeachsourcewiththatofitspeersPonandCardenas,2005ortheconsensusvalueShayeretal.,2002.Bothapproachesaredesignedtofunctionintheabsenceofaprioriinformationbutassumethatamajorityofsourceswillbeaccurate.Thiscannotbeassumedwhenasensingsystemisusedinanunknownenvironmentinthepresenceofanunknownnumberofsensinganomalies.ExperimentalresultsseeChapter5haveshownthattheapproachdescribedinChapter3canfaithfullyranksensorsaccordingtotheirrelativeaccuracywithoutrelyingonthisassumption.InShayeretal.,2002aconsensus-basedmethodforrankingperceptualschemasourcesofprocessedsensordatainroboticsispresentedbasedonapreviouslydevelopedadaptivesen-sorfusiontechnique.Eachperceptualschemaproducesabinaryoccupancygrid1.Afuzzy-logicapproachisusedtocombinethesebinarygridsintoaglobalfusedgridalsobinary,takingintoaccountthepastperformanceofeachperceptualschema.Theperformanceismeasuredbycom-paringtheperceptualschema'sgridwiththeglobalfusedgridusingfourmetrics:thetrueposi-tive,truenegative,falsepositive,andfalsenegativerates.Toranktheperceptualschema,apairofschemaisselectedatrandomandxedtruepositiveratesof0.25unreliableand0.8reliablearearbitrarilyassignedtoeach.Thetwogridsarefusedandthetruepositiverateoftheunreli1Amapwhichrecordsthestatusoccupiedoremptyforeachcellinaregulargrid.Elfes,198930

PAGE 45

ablesensorisusedastheresultofthecomparisonbetweenthepair.Thiswasrepeatedforeachpair,andtheresultsweresortedtoprovidetheranking.Experimentswitharticialdatafromthreesimulatedperceptualschemawitherrorrangingfrom0%showedaccurateranking83.33%ofthetime.Thetestederrorratewasnotallowedtoexceed50%becausepreliminaryexperimentalassessmentsoftherankingprocedurealsoinShayeretal.,2002showedsymmetricresultsaboutthe50%errorratee.g.90%and10%errorratesproducedthesameresults.PonandCardenasdevelopedaconsensus-basedapproachforrankingdatasources.Eachsourcedi2DisrankedaccordingtoanovelaccuracymetricgiveninEquation.2whichisaweightedaverageofitshistoricalaccuracyAt)]TJ/F16 7.97 Tf 6.587 0 Td[(1iandacohesionfunctionci;twhichmea-suresitsagreementordisagreementwithaccurateorinaccuratepeers.Ahistoricalweightfunctionhtover[0;1]controlsthecontributionofprioraccuracyestimates.Ati=htAt)]TJ/F16 7.97 Tf 6.586 0 Td[(1i+)]TJ/F18 10.909 Tf 10.909 0 Td[(htci;t.1whereci;t=fi;t+1)]TJ/F18 10.909 Tf 10.909 0 Td[(fi;t jDj)]TJ/F15 10.909 Tf 16.363 0 Td[(1Xdj2Dfdigai;j;tcj;t.2Thecohesionfunctioncontainsanagreementfunctionai;j;twhichmeasurestheextenttowhichvaluesfromsourcesdianddjagreee.g.Euclideandistanceandadampeningfunctionftwhichistheprobabilitythatasourceisaccurate,independentofagreement.Inexperimentsthedampeningfunctionftsimplyreturnedaconstantvalue.Sincetheagreementisweightedbycj;t,thisfunctionmustbesolvedasasystemofequationswithjDjunknowns.Simulationexperimentsof100datasourceswithrandomlygenerateddataexaminedvaryingagreementfunc-tionsanddampeningfunctionvalues.Euclideandistanceperformedbestwith90%accuracyinrankingthetop10datasources.2.1.4DiscussionThissectionsurveyedapproachesforestimatingsensingaccuracy,estimatingsourcereliabil-ity,andrankingsourcesbyreliabilitywiththegoalofreducingtheimpactofmaliciousorfaultyagentsorsensornodesontheoverallaccuracyofthesystemandfoundthattheseemployeither31

PAGE 46

consensus-basedmethodsorrelyoncharacteristicswhichdonotprovideinformationregardingtheaccuracyofsensinginagivenenvironment.Thelatterareunsuitedforassessmentofsensinginthepresenceofsensinganomalies.Consensus-basedapproaches,whichcomparereadingsfromasensorwithafusedconsensusortothatofpeers,couldtheoreticallybeusedtomeasuretheim-pactofsensinganomaliesonsensingaccuracyorsourcereliabilitybutwouldrequireanaccurateconsensus.Sincetheenvironmentalconditionswhichleadtosensinganomaliesoftenaffectmul-tiplesensors,theseconsensus-basedapproacheswouldrequireanintelligentagenttoactivelyuseawidevarietyofsensorsatalltimestoensurethatsensinganomaliesareaccuratelycharacterized.Thisrequirementisunattainablefordomainswhereenergy,space,and/orweightcapacityareinlimitedsupplye.g.spaceroboticsorwirelesssensornetworks.Inadditionsensorsareexpensiveandsimultaneouslyusingmanysensorstaxesoftenlimitedcomputationalresources,increasesthecomplexityofanintelligentsystem,andincreasesthefrequencyofsensorfaults.Notethatnoneoftheapproachesdiscussedinthissectionweredesignedtoaddresstheproblemofsensinganomalies.Section2.1.1discussedtwostudiesforestimatingtheaccuracyofasensingsystemasawholewhichrelyonmeasuredcharacteristicsi.e.cooperationthathavenothingtodowithaccuracyHongjun,Zhiping,andXiaona,2008ortheexclusiveuseofaprioriknowncharacteristicsi.e.classicationaccuracyforassessment.InHongjun,Zhiping,andXiaona,2008theproblemofuncooperativenodesisaddressedwhileWangandShen,1999isconcernedwithambiguityindecision-makingsystems.Section2.1.2surveyedapproachesthatlearnthereliabilityofotheragentsorsensorsindis-tributedsystemsbasedsolelyonpriorinteractionswiththatagentandtheopinionofanagent'speersalsobasedoninteractionsinanefforttoisolatemaliciousorfaultysourcesfromtherestofthesystem.Bothconsensus-basedmethodsandmethodsthatrelyinmeasuredcharacteristicse.g.QoSthathavenothingtodowithaccuracyhavebeenappliedtothisproblem.Whilecon-sensusbasedapproachestonotrelyinaprioriinformationtheydorequireanaccurateconsensustoensurethecorrectnessoftheestimatedreliability.Suchaconsensuscannotbeassumedwhenasensingsystemisusedinanunknownenvironmentinthepresenceofanunknownnumberofsensinganomalies.Notethatundersuchconditionsincreasingthenumberofidenticalorsimi-larsensorsdoesnoteliminatethepossibilityofaninaccurateconsensussincetheenvironmental32

PAGE 47

conditionsthatcausesensinganomalieswillaffectallidenticalsensorsinthesamefashion.TheapproachdescribedinRyutovandNeuman,2007usesahybridconsensus-basedandanalyticalmodel-basedapproachwhichreliesonknownphysicalrelationshipsbetweenthevaluesreportedbydistinctsetsofsensorstodetectcompromisedsetsofnodes.Whiletheconsensus-basedaspectoftheapproachisproblematic,themodel-basedaspectispromisingforhandlingsensinganoma-liesinmulti-sensorsystemswheretherelationshipsbetweensensorreadingscanbereliablymod-eled.Apossibleextensionoftheapproachpresentedinthisdissertationwouldbetoleveragesuchrelationshipsinasimilarfashiontomeasureinconsistencieswithinamorediversesetofsensorreadings,enablingimproveddetectionandcharacterizationofsensinganomalies.Thesestudiesdifferfromthisworkinthattheyaredesignedtoevaluatesourcesonanindividualbasiswhereasthelatterisdesignedtoevaluateasensingsystemasawhole.Thisdistinctionsuggeststhatthetwoapproachesmaybecomplementary,i.e.ifusedtogethertheywouldprovideasensingman-agementsystemwithmoreinformationregardingtheperformanceofitsavailablesensorsinanunknownenvironmentascomparedtousingeachinisolation.Section3.1describesaproposedsystemforidentifyingsensorsuitabilityinunknownenvironmentswhichutilizesbothinthisfash-ion.Section2.1.3consideredtwoconsensus-basedapproachesfoundintheliteraturethatrankinformationsourcesPonandCardenas,2005;Shayeretal.,2002.BothapproachesassumethatamajorityofsourceswillbeaccuratewhereasexperimentalresultsseeChapter5haveshownthattheapproachdescribedinChapter3canaccuratelyranksensorsaccordingtotheirrelativeaccuracywithoutrelyingonthisassumption.2.2SensorFaultDetectionandIdenticationThissectionpresentsrelatedworkondetectinganddiagnosingsensorproblemsingeneral.Sincenoneofthesestudiesaddressessensinganomalies,theseapproachesarecategorizedandevaluatedaccordingtothemethodusedtoperformsensorFDIasopposedtoexperimentalre-sults.Thesemethodsareasfollowsindecreasingorderofsuitabilityforsensinganomalies:inconsistency-based,consensus-based,consensus-basedpair-wise,learnedmodelswhichdonotrelyonapriorisysteminformationuninformed,inconsistency-basedusingatrustedsource,qualitativeand/orstochasticmodels,learnedmodelswhicharedesignedwithatargetsystemin33

PAGE 48

mindinformed,andanalyticalmodels.ThemostcommonapproachesforsensorFDIuseapri-orimodelseitheranalyticalorstochastictomodelnormaland/orfaultysystembehavior.Unlikethetraditionalsensorfaultstheseapproachesaredesignedtoaddress,sensinganomaliesdonotexistindependentlyofanunknownenvironmentwhichmakesthemnearlyimpossibletomodel.Whilestochasticandanalyticalmethodsremainpopularintheliterature,recenttrendsinsensorFDIhaveshownasincreasedinterestinpureuninformedlearningapproaches.Theseapproachescouldtheoreticallybeusedtolearnaboutsensinganomaliesinunknownenvironmentsbuttheirabilitytodetectanddiagnosesuchanomaliesislimitedbasedonthepresenceorabsenceofsimilaranomaliesinthetrainingdataandthelearningapproach'sabilitytogeneralizetoanewenvironment.Consensus-basedapproacheswhichrelysolelyoncomparisonsbetweenreadingsfromdistinctsensorsoracomparisonofthosereadingstoafusedconsensusarerareinthesensorFDIliteratureonlythreewerefound.Thesearemoresuitedtoaddresstheproblemofsensinganomaliesthanlearning-basedapproachesbutstillrequireanaccuratemajorityofsensorswhichcannotbeassumedinthepresenceofsensinganomalieswhichoftenaffectmultiplesensors.Theinconsistency-basedapproachdescribedinAfgani,Sinanovic,andHaas,2008a,bissimilartotheapproachpresentedinChapter3inthatitalsodetectsanomaliesbyapplyingathresholdtoamet-ricderivedfromtherecordeddata,buttheirspecializedapproachislimitedtosignalswhichshowhighperiodicityundernormalconditions.ThissectionisorganizedaccordingtothecategorizationofsensorFDIstudiesgiveninTa-ble4whichliststhestudiesaccordingtotheprimarymethodusedforFDI.Section2.2.1providesabriefhistoryoffaultdetectionandidenticationtoestablishthecontextinwhichthesensorFDIapproachespresentedinthissectionweredeveloped.Sections2.2.2through2.2.7reviewexistingapproachesforsensorFDIinorderbymethodfromthemosttotheleastrelevantfortheprob-lemofsensinganomaliesinunknownenvironments.StudieswhichcombinemethodsarelistedinmultiplerowsinTable4butappearinthissectiononlyonce,accordingtotheprimarymethodemployedi.e.wherethestudyisnotlistedinitalics.2.2.1BriefHistoryofFDIForthirtyyearsFDIsystemshavebeenstudiedanddeveloped.Fromthelate1970suntiltheearly1990smostFDIsystemswereexpertsystemsdevelopedtoemulateandassisthumantechni-34

PAGE 49

Table4.RelatedworkonsensorfaultdetectionandidenticationFDIbymethod.Studieswhichcombinemethodsarelistedinmultiplerowswiththeprimarymethodinnormalprintandsupple-mentalmethodslistedinitalics. Method Studies Inconsistency-based Thiswork,Afgania,b Consensus-based Soikaa,b Consensus-basedpair-wise BaconBank LearnedModelsuninformed LiuFravoliniGuCanhamBongardKoushanfarLarkeyChenHerediaPaivaChristensen Inconsistency-basedtrustedsource Bank QualitativeModels WilliamsNarasimhan StochasticModels RoumeliotisGoelWashingtonHashimoto,2007DeardenNarasimhanKobayashiDuanWeiPlagemann LearnedModelsinformed DeukerGoelRanganathanKawa-bataPlagemannDu AnalyticalModels VosUmVinsonneauSchwallBlakeMonteriuPertewRothenhagenGao ciansorphysicians.AwellknownexampleisMYCINShortliffeetal.,1975,anexpertsystemwhichrecommendedtherapyforpartiallyknownorunknownbacterialinfectionsandperformedbetterthansomemedicaldoctors.Srinivaswrotewhatappearstobetherstworkonfaulthandlingforrobotsusingaheuristicplanningbasedapproachforndinghypotheses,testingthosehypotheses,andrecoveringfromthefault.Inthemid-to-late1980smodel-baseddiagnosisMBDexpertsystemsbegantoappearwhichprovidedageneralmeansofperformingfaultdetectionandidenticationFDIusingrenedmethodsforhypothesisgenerationandtestranking,leav-ingtherestofthesystemdetectionandtestexecutioninblackboxes.DeKleerandWilliamstooktheleadinthisareawiththeirGeneralDiagnosticEngineGDEdeKleerandWilliams,1987andSherlockdeKleerandKurien,2003;deKleerandRaiman,1993,1995;deKleerandWilliams,1989systems.ThroughcontinuousrenementtheyandotherresearcherslikeCon-soleandDresslerandReiterbroughtMBDtoalevelofmaturitywhichallowedcommerciallyviablesystemstobeproducedby1999ConsoleandDressler,1999.AstheMBDapproachwasappliedtoreal-worldproblems,existingtoolsfromcontroltheoryorarticialintel-ligencewereexploitedtollintheblackboxes.Controltheoryresearchersappliedmathemati-35

PAGE 50

calsystemsanalysis,generatingmodelsandresidualsConsoleandDressler,1999;deKleerandKurien,2003.Articialintelligenceresearchersusedtrendanalysis,stochasticorlearnedmod-eling,andothergeneralfeatureextractingtechniquesChantleretal.,1998.In1996WilliamsdivergedfromtheMBDparadigmtocreateLivingstoneWilliamsandNayak,1996,theFDIandgeneralstatetrackingmodulefortheDeepSpaceOnespacecraft.AsadirectdescendantofGDE/Sherlock,LivingstoneservedasabridgebetweenthematureMBDapproachandmorere-centarticialintelligenceapproachestoFDI.Thesetendtofocusmoreonstatetrackingdeter-miningwhichofasetofknownstatestheagentwasin,withfaultdetectionandidenticationasalogicalside-effectofmodelingandtrackingfaultstatesseeforexampleBiswas'workJietal.,2003;Lerneretal.,2000;NarasimhamandBiswas,2007;Narasimhametal.,2000.Ingen-eral,FDItechniquesinbothareascontroltheoreticandarticialintelligencehavebranchedoffandcreatedtheirownapproachestoidentication,relyinglessonMBD.Somerecentdata-drivenapproachesseethesurveyinPettersson,2005havethrownoutsystemmodelsaltogetherandfo-cusedontherelationshipbetweenrawinputandthepresenceoffaults.2.2.2Inconsistency-basedFDIAfgani,Sinanovic,andHaasa,bpresentaspecializedinconsistency-basedapproachfordetectinganomaliesinrealtimeforwirelesssignalswhichislimitedtosignalswhichshowhighlyperiodicbehavior.Theapproachusesashort-termFouriertransformSTFTtoconvertthesignaltoaspectrogram.Twoslidingwindowsofanempiricallydeterminedsizeseparatedbythesignalperiodareappliedtothespectrogram.AhistogramisusedtomodeldatawithineachwindowandKullback-LeiblerKLdivergenceisusedtomeasurethedifferencebetweenthetwohistograms.Ananomalyisdetectedwhenthedivergenceexceedsathreshold.Exampleswithrealwirelesssig-nalssubjecttoanomalouseventse.g.short-terminterferenceandpowerlevelchangeshowedaveryhighsignaltonoiseratiofortheKLdivergencemetric.Theresultsalsoshowedthat,byre-ducingtheseparationbetweenthetwoslidingwindows,theapproachcouldbeusedtolearnthesignalperiodwhichistheonlyaprioriinformationtheapproachrequires.Liketheapproachde-scribedinChapter3,Afgani,Sinanovic,andHaasa,bdetectanomaliesfromasinglesourcebyapplyingathresholdtoametricderivedfromtherecordeddatawithoutrelyingonapriorimodelslearnedorotherwisewhichdescribetheexpectedvalues.Thelatterismorelimitedin36

PAGE 51

itsapplicationsincetheunderlyingsignalmustshowverysimilarperiodicbehavior,whereastheformerisapplicabletoanysignalwhereanomaliesproduceinconsistencies.2.2.3Consensus-basedFDIThissectionexploresconsensus-basedapproachestosensorFDI.Mostconsensus-basedsys-temsaresimplevotingschemesbuiltforhomogeneoussensorsystemsseeforexampleAhlstrometal.,2002;KrishnamachariandIyengar,2004;LeeandXu,2001.ThreeapproachesforFDIofsituatedagentshavebeenfoundthataremoresophisticatedandcanbeusedwithheteroge-neoussensorsystemsand/orfuseddata.BothSoika,1997a,bandBacon,Ostroff,andJoshi,2001usepurelyconsensus-basedapproachestodetectsensorfaultsormis-calibrationinaringofsonarsensorsandredundantinertialnavigationsensorsrespectively.InBank,2002acombinedconsensus-basedandinconsistency-basedmethodwasdevelopedinwhichthelatterreliedonalaserrangenderasatrustedsource.OfthesetheprobabilisticapproachdevelopedinSoika,1997aisthemostsophisticatedandevenmentionedahypotheticalsolutiontoisolatepoorlysensedregions,butthiswasneverdevelopednortested.Theseapproachesaredesignedtofunctionintheabsenceofaprioriinformationbutassumethatamajorityofsensorswillbeaccurate.Thisassumptionisunlikelytoholdforasensingsysteminanunknownenvironmentinthepresenceofsensinganomalieswhicharelikelytoaffectmultiplesensors.InSoika,1997baprobabilisticconsensus-basedsensorFDIapproachispresentedwhichan-alyzestheconsistencyofredundantsensorreadings.Aprobabilisticsensormodelisusedtotrans-laterawsensorreadingsintostatementsaboutthestatusofagivenvariableforexamplewhetherapointinspaceisoccupiedorempty.Statementsfrommultiplesensorsarecombinedbyaninde-pendentopinionpool.ThereliabilityofasensorisdeterminedbycomparingastatementfromthesensorandthefusedstatementfromallotheravailablesensorsaccordingtoEquation.3PslmjA=KXkm=1KXk=1PslmjakmMm;akfMgPakmMmPakfMg.3whereslmdescribesthestatusl2fworking;defectivegofthesensorm,Aisasetofsensorreadings,k2Kisasetofpossiblevaluesforaprobabilisticvariable,akmMmisthestatement37

PAGE 52

aboutkfromsensorm,andakfMgisthesameforthesetofothersensors.TheprobabilityofeachstatementPakmMmandPakfMgaredenedbythesensormodelandthefusedvaluerespectively.Thereliabilityofthesensorovermultiplereadingsisagaincombinedbyanindependentopinionpool.Thisapproachwastestedusingsonarreadingscombinedinaproba-bilisticoccupancygrid2.InthiscasePslmjakmMm;akfMgwasreducedtoanindicatorfunctionwhichevaluatedto1whenthesensoragreedwiththeothersand0otherwise.InSoika,1997athegeneralapproachpresentedinSoika,1997bwasfurtherrenedforusewithoccupancygrids.Figure2showshowtheapproachdetectedanddiagnosedsensingproblemswithoutrelyingonaprioriinformation.Foreachcellwithintheoccupancygrid,theprobabilityofoccupancyiscalculatedaccordingtoEquations.4and.5assumingthatsensorjisworkingOKorisnotworkingKOrespectively.3Thesevalueswereusedtodeterminetheprobabil-ityofconsistencyKiforcelligiventhatsensorjisworkingEquation.6andnotworkingEquation.7.PokSj=PCijfMng=occupied.4PkoSj=PCijfMk;k6=jg=occupied;Mj=empty.5PKijSj=PokSj+)]TJ/F18 10.909 Tf 10.909 0 Td[()]TJ/F18 10.909 Tf 10.909 0 Td[(PokSj.6PKij:Sj=)]TJ/F18 10.909 Tf 10.909 0 Td[(PkoSj+)]TJ/F18 10.909 Tf 10.909 0 Td[(PkoSj.7whereMj=occupied>0:5.8Mj=empty<0:5.9Atunableparameterwasincorporatedtocontroltheinuenceofaparticularcell'sconsis-tencyonagivensensor'sevaluation.FinallyBayes'ruleisusedtodeterminetheprobabilitythatthesensorisworking.Thisprobabilityiscomparedtoathresholdtodeterminethestatusofthesensoraccordingtocelli,consistentorinconsistent.Thethresholdisexpectedtobeapplication-dependent.Grid-wideevaluationofthestatusofagivensensorwasexplicitlydenedastheratio 2Amapwhichrecordstheprobabilityofoccupancyforeachcellinaregulargrid.3Fusionofmultiplesensorreadingse.g.PCijfMng=occupiedwasstillpresumablyperformedusinganindependentopinionpool.38

PAGE 53

Figure2.AsensorfaultdetectionanddiagnosisapproachforSoika,1997a.ofinconsistentcellstoconsistentcellsoverallevaluatedcells.Thestudyrecommendedevaluationofagridcelljustbeforedeletionwhenitwastoofarawayfromtherobottotonarobot-centricmap.Soika,1997aalsoproposedacell-by-cellreliabilitymeasuree.g.forisolationofsensinganomaliesdenedas1)]TJ/F18 10.909 Tf 10.945 0 Td[(HxwhereHisShannon'sentropy.Thismeasurewasonlymentionedonceinthepaperasanargumentinfavoroftheoverallapproach,andwasapparentlyneverin-tendedforimplementationortesting.Experimentswitharingof24sonarsensorsonacustommobilerobotshowedthatthesystemcoulddistinguishbetweenworkinganddefectivesonarsen-sorsandcouldbeusedforre-calibration.Bankpresentsacombinedconsensusandinconsistency-basedapproachforsensorFDIforaringof24wideanglesonarsensorsonaNomadXR4000whichusesaSICKlaserrangenderasatrustedsourceofinformationtodeterminewhatthesonarreadingsshouldhavebeen.Theactualpathsofsonarsignalsandtheirintersectionswithsonarsensorsarecalculatedusingamodeloftheenvironmentbuildfromthelaser'sreadings.Onevectorofcondencevaluesforeachsensoriscreatedbasedonagreementwiththemodel.Asecondcondencevectoriscal-culatedbasedonagreementbetweenneighboringsensorswithoverlappingsensedregionsconeshapedduetolowangularresolution.Thesetwovectorsaresummedtogenerateanoverallcon-dencemeasurementforeachindividualsonarsensor.Anumericexamplewithasingleinjectedfaultisgiven.Thecondencevalueforthefaultysensoriszero,enablingtrivialdetectionand39

PAGE 54

diagnosis.Notethattheapproachislimitedtoassessmentofrangesensorsandassumestheavail-abilityofaperfectsensorSICKlaserwhichlimitsitsapplicabilitytoenvironmentswhichdonotcontainlaseranomaliese.g.noglassandnofogorsmoke.InBacon,Ostroff,andJoshi,2001acontrollerformannedaircraftwithbuiltinconsensus-basedaccelerometerandgyrosensorFDIispresented.TheFDIsystemassumestripleredundancyoftheactualphysicalsensors,andacustomdifferencemetrice.g.Equation.10givesthemet-ricforsensor1whererisasensorreadingisusedtodetectandidentifyfaults.d1=jr1)]TJ/F18 10.909 Tf 10.909 0 Td[(r2jjr1)]TJ/F18 10.909 Tf 10.909 0 Td[(r3j.10Toreducetheimpactofnominalnoise,afadingmemoryaccumulatorisusedtoaveragethismet-ricovertimeandathresholdisappliedfordetection.AsimulatedtaillessaircraftmodelfromLockheedMartinandarobustnesstestercalledRASCLEwereusedtoevaluatethenewcon-troller.Theresultsshowedaccuratedetection,identication,andadaptationtomultipleaccelerom-eterandgyrofailureswithnotabledelaysinthelatter.Notethattheapproachisspecializedforsensingsystemswithtripleredundancyandthatitassumesthattwoofthethreesensorsineachgroupareaccurate.Theenvironmentalconditionsthatleadtosensinganomalieswillaffectallidenticalsensors,thereforetheapproachisunsuitedfordetectionofsensinganomaliesevenwithinthislimitedsetofsensingsystems.2.2.4LearnedModelsUninformedThissectionisfocusedonsensorFDIapproacheswhichuselearningapproachesthatdonotrelyonaprioriinformation,butinsteaduserawsensorinputinalearningsystemtodiscovertherelationshipbetweeninputsandthepresenceoffaults.BasedonthebreakdownofFDIworkpre-sentedinPettersson,2005theseapproachesfallintothedata-drivencategoryinwhichsystemmonitoringisperformedbasedonrawinputalone,notrelyingonsystemmodels.SystemslikeCanham,Jackson,andTyrrell,2003;ChenandSaif,2007;Guetal.,2001;Herediaetal.,2008;Larkey,Bettencourt,andHagberg,2006whichuseunsupervisedlearningcouldhypotheticallybeusedtolearnaboutsensinganomaliesinanunknownenvironmentbutwouldbeunlikelytode-tectanomaliesthatarenotencounteredinthetrainingset.Inadditionthesystemtobemonitoredwouldbevulnerabletosensinganomaliesduringthetrainingperiod.40

PAGE 55

Guetal.presentadata-drivenapproachforaroboticprostheticeyewhichusesprin-cipalcomponentanalysisPCAtocapturetheessentialrelationshipswithinthetargetsystemforsensorFDI.AsystemmodelisbuiltbyapplyingPCAtosensorreadingscollectedduringfault-freeoperation.Theresultingfeaturesareusedtopredictsensorreadingsforonlinemonitoringwiththestandardsquaredpredictionerrorstatisticusedtodetectafault.Afaultisolationsolutionisdevelopedbycreatinganidealisolationstructurematrixwhichdescribesthetargetrelation-shipbetweenresidualsandsensorstatus,thensolvingfororestimatingthetransformbetweenthelearnedPCAmodelandtheisolationstructure.Thisapproachisappliedtoanarrayofseveninfraredsensorsusedtotrackeyemovementinaroboticprostheticeyesystem.Experimentalre-sultswithsimulatedfaultsshowedthatsoftandhardfailuresoneachsensorwerecorrectlyde-tectedanddiagnosed.Twopaperspresentsensorfaultdetectionanderrorcorrectionapproachesfordistributedsen-sornetworksinwhichrecordeddataareusedtobuildprobabilisticmodelsofthedataovertimeconsecutivereadingsonthesamenodeandspacebetweenreadingsonneighboringnodes.Bothstudiesarefocusedonenvironmentalmonitoringapplicationswheremanyinexpensiveex-teroceptivesensorsareusedtomeasureattributessuchastemperatureandhumiditywhichshowlowtemporalandspacialvariance.InLarkey,Bettencourt,andHagbergprobabilitydistri-butionsarederivedtodescribethearithmeticdifferencebetweenthecurrentandpriorreadingorbetweenthelocalreadingandthoseofneighbors.Fordifferenceswithknownunderlyingdistri-butionse.g.Gaussiantheparametersareupdatedasnewreadingsarecollected,withtheinu-enceofolderreadingsdiminishingovertime.Anapproachfornon-parametricestimationofanunknowndistributionisalsodescribed.Thesedistributionsareusedtodetectsensingfaultslowprobabilityandtoestimatethevalueofmissingreadings.Experimentalresultswithrealsensordatadeployedinanoutdoorenvironmentshowedverygoodestimationoftherealsensorreadingsforasinglenode.Articialexampleswithsimulatednoiseshowedperfectdetectionofinjectedsensingfaults.KoushanfarandPotkonjakusesemi-MarkovchainmodelstodetectsensingfaultsandcorrecterrorsforanetworkofMICA2sensormotes.Theapproachuseslabeledsen-sordatatoderivenon-parametricdensityfunctionsdescribingtheprobabilitythatamote'sread-ingwillbecorrect,faulty,ormissingi.e.thediscretestatesofthesemi-Markovchain.Foreachstate,twonon-parametricdensityfunctionsareusedtodescribetheprobabilityofremainingin41

PAGE 56

thatstateandtransitioningtoanotherstaterespectively.Thesedensityfunctionsarebuiltfromtherecordeddatabycreatinghistogramsofconsecutiveupdatesspentinthesamestateandtransitionfrequenciesrespectivelyandapplyingsmoothingtechniquestoimprovethestatisticalaccuracyoftheresultingdistribution.KoushanfarandPotkonjakdidnotincludeexperimentalresults.Canham,Jackson,andTyrrellpresentasysteminspiredbyimmunesystemsfoundinnature.Theapproachconsistsofasetofdetectorswhichareautomaticallygeneratedduringafaultfreetrainingperiod.Duringtrainingeachdetectormonitorsanassignedmulti-dimensionalvaluewithinthesystemandtracksitsminimumandmaximumvaluesalongeachdimensiontodeterminetherangeofnormalvalues.Iftherangeexceedsapre-determinederrorthresholdthedetectorissplitintomultipledetectorsthenretrained.Onlinemonitoringisperformedsimplybycheckingifthemeasuredvalueslieoutsidethelearnedranges.Theapproachwasappliedwithoutmodicationtotwodistinctroboticsystems:aKheperarobotwithafuzzymotioncontrollerandaRASCALrobotwiththecontrollersuppliedbyBAEsystems.TheFDIsystemsweretrainedusingdatacollectedoneachrobotthentestedbysupplyingarticialvaluesoverthefullrangeofthetargetvalue.Asexpected,thesystemdetectedaproblemwhenthevaluesexceededthepre-determinederrorthreshold.ThreeapproachesusedneuralnetworkstolearnthenormalbehaviorofasystemforsensorFDI.InLiu,Shen,andHu,1999recurrentneuralnetworksareusedtolearnthetemporalevolu-tionofasensor'svalueovertimeandspatialrelationshipsbetweenreadingsfrommultiplesen-sorspropertiesofrawsensordataforFDI.Twonetworksareusedforeachsensor,onewhichexaminesthepastreadingsovertime,andanotherwhichcomparesthereadingagainstthemea-surementsfromalltheothersensors.Thenetworksaretrainedduringfault-freeoperationanddivergencefromtheestimatesprovidedbythenetworkstriggersfaultdetectionandaccommo-dationreplacingthesensorreadingswiththeestimate.Ageneralmathematicalairplanemodelisusedtodemonstrateaccommodationoftwohardsensorfailures,aspikeandaconstantbias,bothofwhichweredetectedandaccommodated.InFravolinietal.,2001radialbasisfunctionRBFneuralnetworksareusedastrainedadaptiveestimatorsofnormalbehaviorforaUAV.Thenetworkswerebuiltaspartofthelearningprocessbasedontheminimalresourceallocationap-proach,sotheirdesignwasbasedentirelyonthetrainingdata.Thenetworkwastrainedoff-lineusingpurelysimulateddata,thenonlineusingadetailedaircraftmodelprovidedinMat-42

PAGE 57

lab.Onceasensorfailurewasdetected,theestimateprovidedbythenetworkreplacedthesensorreadingsinthesimulatedcontroller.Examplesdemonstratedthatthesystemcouldprovideac-curate,stableresidualsforsixtypesofsimulatedfaultsinthepitchratesensorincludingminorand/ortransientproblems.InChristensenetal.atime-delayneuralnetworkisusedtode-tectbothsensorandeffectorfailuresforans-bot.Traditionalback-propagationwasusedtotrainthenetworkwhiletherobotcarriedoutoneofthreetasks:ndperimeter,followtheleader,andphysicallyconnecttoanotherrobot.Trainingdataincludedbothnormaloperationandoperationwithsoftware-injectedfaults.Inexperimentsthestructureoftheneuralnetworkremainedconstantwhiletheinputgroupdistancewhichcontrolsthespanoftimeaccessibletothenetworkandthethresholdontheoutputnodewerevaried.Theresultsshowedaccuratefaultdetectionwithinsec-ondsmedianof2.0secondsofthefaultoccurrence.Threepapersuseadata-drivenapproachtolearnobserversestimatesofthesystem'soutputforsensorFDI.Allthreeapproachesuseestimationerrorasfeedbackforlearningalinearinput-outputmodelofthetargetsystem.ChenandSaifuseanadaptiveapproach,improvinges-timatesofparametersforanunobservablelinearsystembasedonthedifferencebetweentheesti-matedandactualoutputi.e.theresidualwhilethesystemisinoperation.Theapproachdecom-posesthesystemintopobservers,oneforeachsensorinthesystem.Unliketraditionalobserver-baseddiagnosticsystems,thesystem'sstateisnotmodeled.Insteadtransferfunctionsareusedtoestimatecurrentoutputsignalsbasedsolelyonpriorinputandoutputsignals.Parametersforthetransferfunctionsareadjustedovertimebasedontheaccuracyofpriorestimates.Faultsaredetectedandisolatedwhentheresidualvalueforagivensensorexceedsanexperimentallydeter-minedthreshold.ExperimentalresultsusingasimulatedghterjetshowfastandaccurateFDIforsinglesensorscaleanddriftfaults.Sincetheobserversareadaptive,theresidualspeakwhenthefaultoccursandthenadjusttomodelthefaultystate.Herediaetal.usedrealightdatatolearnanAutoRegressiveeXogenousARXmodelforasmallautonomoushelicopter.Thismodelwasthendecomposedtoprovidedistinctobserversforeachactuatorandsensorinthesys-tem.Faultdetectionandisolationoccurredwhenagivencomponent'sresidualexceededanexper-imentallydeterminedthreshold.Extensiveexperimentsusingrealightdatawithinjectedfaultsshowedfastdetectionandisolationofallactuatorandmostsensorfaults.Thesystemhadtroublewithminorsensordriftfaultswhichwereoftendetectedlateormissedaltogether.Paiva,Galvao,43

PAGE 58

andYoneyamausedatacollectedundernormalconditionstotrainabankofwavelet-basedmodelswithxedfrequenciesofthetargetlinearsystemtoserveasoutputobservers.Experi-mentalresultswithasimulatedservo-mechanismandBoeing747aircraftcomparedtheproposedapproachtoatraditionalobserverusingaprecisemodelofthetargetsystem.Theresultsshowclearerlargeramplitudeintheresidualbutdelayeddetectionascomparedthetraditionalob-server.InBongardandLipson,2004anevolutionaryroboticsapproachtofaultidenticationandrecoveryforaleggedrobotispresented.Inthisapproachafaultisdetectedbynotingadisparitybetweentheexpecteddisplacementoftherobotandtheactualdisplacementduringoperation.Di-agnosisisperformedbyrecordingtheinputandactualdisplacementoftherobotforseveraltimesteps,thenre-evolvingthecontrollerofinetomatchthisnewinput-outputpair.Theresultingof-inecontrollerimplicitlycontainsanewmodeloftherobot'sdynamicswhichistransferredbacktothephysicalrobottoenableanotherevolutionprocessaimedatfaultrecovery.Experimentswereperformedusingsimulatedfour-andsix-leggedrobots.Theresultsshowedimprovedcon-trollerperformancefollowingnon-catastrophicfailurese.g.sensorfailureandreasonableaccom-modationofcatastrophicfailuresincludingafailurethatcouldnotbemodeledbytheevolutionarycontrolleri.e.afaultwithinthecontrolleritself.2.2.5QualitativeandStochasticModelsThissectionfocusesonsensorFDIapproachesthatusequalitativeorstochasticmodelsorcombinationsthereofofthesystem'sbehaviorundernormalandfaultyconditionstodetectanddiagnosetraditionalsensingproblems.TheseFDIsystemsattempttodeterminethesystem'sstateatalltimes,whereknownfaultsaretreatedasasetofpossiblestates.Theserequireexplicitmod-elingoffaultstateswhichmakesthemunsuitedforhandlingsensinganomaliesinunknownenvi-ronments.AnearlyexampleofthisapproachwhichusedqualitativemodelingwasLivingstonedescribedinWilliamsandNayak,1996whichservedastheFDIsystemfortheDeepSpaceOnespace-craft.LivingstonewasamodiedapplicationofSherlockbuiltspecicallytoworkwiththere-activeplanningandschedulingsystemsdevelopedforDeepSpaceOne.Twomoduleswerede-velopedtoworkwithanaprioritemporalpropositionalmodelthatdescribedallknownnormal44

PAGE 59

andfaultystatespossibleinthespacecraftsystemandtheprobabilityoftransitionsbetweenthosestates.Themodeidenticationmodulestudiedthemostprobabletransitionsfromthelastknownstateandthecurrentsensordatatodeterminethemode.Themoderecoverymodulecreatedaplanforreturningthesystemtothecorrectmodeusingtheleastcostlysetofsteps.InWilliamsandNayak,1996experimentswithasimpliedmodeloftheCassinispacecraftdevelopedbyJPLandastandardcircuitsimulatorshowedthatthesystemcouldefcientlyontheorderofsecondsidentifyfaultsandrecongurethesystemtorecover.AcommonapproachforFDIofcomplexdynamicsystemsishybridcontinuousanddiscretetrackingofthesystem'sstateseeBiswas'workJietal.,2003;Lerneretal.,2000;NarasimhamandBiswas,2007;Narasimhametal.,2000.4Intheseapproachescontinuoustrackersarecom-paredtorawsensorreadings.Onceasignicantdivergenceisdetected,modelsofthediscretestatesareexaminedtodeterminewhichstatethesystemhastransitionedinto.Thecontinuoustrackersforthatstatereplacetheoldonesandtheprocesscontinues.Faultsaremodeledasdis-cretestates.AnexampleofsuchasystemcanbefoundinWashington,2000whereaprelimi-naryattempttocreateafaultdetectionsystemforroversispresentedusingapartiallyobservableMarkovdecisionprocessPOMDPmodeldiscretestatemodelingandKalmanlterscontinu-oussystemmodeling.Thesystemwastestedusingtelemetryfromaprototyperoversystemop-eratingoveruneventerrainwithonebrokenwheel.Washingtonshowedthatthesystemac-curatelytrackedtherover'sstatewiththeexceptionofsometransitoryfalsepositives.Thebrokenwheelwascorrectlydetectedanddiagnosed.Anothercommonapproachformobilerobotsismultiple-modeladaptiveestimationMMAEwhichissimilartohybridtrackingtechniques.InMMAEmultiplemodelsanalogoustodiscretemodesareactiveatalltimes,inthattheyareassessingtheirlikelihoodbycomparingtherawsen-sordataandtheirpredictedvalues.Atanypointthemostlikelycandidatemodelormodelscanbedetermined.Roumeliotis,Sukhatme,andBekeybextendedtheapproachintroducedinRoumeliotis,Sukhatme,andBekey,1998atodevelopasensorFDIsystemforwheeledmobilerobotsusingabankoffourKalmanltersbasedonkinematicmodelstomodelthenominalstate,asoftsensorfailure,andtwohardsensorfailuresrespectively.Thelters'outputareusedtocalculatetheconditionalprobabilitythatthesystemisineachofthefourmodeledstates.Re4ItshouldbenotedthatBiswasisaleaderinstatetrackingFDIwhenoneconsiderscomplexdynamicsystemsingeneral.45

PAGE 60

sultsfromaPioneer1robotshowedcorrectdetectionanddiagnosisofallmodeledfaultswithadelayedresponseforaslightvariationsmallermagnitudeonthesoftfailure.ThisapproachwasextendedfurtherinGoeletal.,2000byusinganeuralnetworktoidentifythesystemstatebasedonresidualsfromeightKalmanltersmodelingsevenfaultsincludingsensorandactuatorfaultsandthenominalstate.ExperimentswithdatafromaPioneerATrobotincludingtestcaseswheretherobotwasmovingfasterascomparedtotrainingcasesandsimulatedfaultsshowedthatthesystemcorrectlydetectedanddiagnosed5allfaultswithinthreeseconds.Hashimotoetal.buildontheapproachfromRoumeliotis,Sukhatme,andBekey,1998bbyexplicitlymodelingprobabilistictransitionsbetweenmodes.InHashimotoetal.,2001abankof16ltersisusedtomodelallpossiblecombinationsoffourhardsensorfaultswithinthedrivesystemincludingthenominalcase.HeuristicsappliedtotheoutputoftheKalmanltersareusedtodeterminethestateoftherobot'ssensorsandadaptthenavigationsystemtoprovidefault-tolerantcontrol.Asetoffourscenariosincludingnofailures,aseriesoftwoindependentfailures,twosimultaneousfailures,andaseriesofthreeindependentfailuresleadingtocatastrophicfailurerespectivelywereusedtotesttheFDIsystem.Theresultsshowedthatthesystemcorrectlydetected,diagnosed,andadaptedtothesensorfailuresonacustombuiltthreewheeledrobot.Hashimoto,Watanabe,andTakahashithenexpandedtheirapproachtodetectscalesensorfaultsforacustom-builtjoystick-drivenwheelchair.Byutilizingamodel-freepoint-matchingalgorithmforlaserrangescans,thetruevelocitiestranslationandangularofthewheelchaircanbeestimatedwhichen-ablesdetectionofscalefaultsininternalsensors.Detectionofsensorfaultsinthelaserrangenderisleftforfuturework.KobayashiandSimonpresentahybridKalmanltermodelforanaircraftenginewhichcombinesoutputfromanon-linearmodelofthesystemandacollec-tionofpiecewiselinearmodelsusedtodescribethesystem'sstate.AsdiscussedinKobayashiandSimon,2006,thesamenon-linearmodelcaninformanynumberofKalmanlterswhosepiecewiselinearmodelsaretunedtodetectdistinctfaultconditions.ManyresearchgroupshavereplacedKalmanlterswithparticleltersforMMAEfores-timationofstatevariablesandFDIscenariosthataredifculttomodel,includingfaultswhichdependonrobot-environmentinteraction,e.g.collisionwithanunseenobjectorbecominghigh-centeredonroughterrain.Particlelteringcanbeusedtomodelanydistributionatanarbitrary 5Thesystemcouldnotdistinguishbetweensingleandmultipleattiresonthesameside,otherwiseallfaultswerecorrectlydiagnosed.46

PAGE 61

levelofabstraction.Eachparticlerepresentsapossiblestateofthesystemincludingadiscretemodeandvaluesforallcontinuoussystemparameters.Deardenetal.presentparticlel-teringapproacheswherethetraditionalparticlelteringtechniqueisaugmentedbytwomodi-cations:incorporatingariskvaluetoensurethathigh-riskstatesfaultsaretrackedevenwhentheyarenotlikely,andadaptivevariableresolutionmodeling.InadditiontheGaussianparticlel-terisintroducedasamoreefcientmodel.Eachimprovementtothetraditionalparticlelteringtechniquewastestedindividuallyonrealandsimulatedroverplatformsandusedordersofmagni-tudetimesfewerparticlestoprovideaccuratetrackingofsystemstatus.InNarasimhan,Dearden,andBenazera,2004DeardencollaboratedwithNarasimhanandBenazeratointegratethisparticlelteringtechniquewithLivingstone3seeNarasimhan,Brownston,andBurrows,2004forimprovedidentication.Duan,Cai,andYuintegrateparticlelteringwithfuzzylogicbyconstrainingthesamplingspaceoftheparticlestoafuzzysubsetbasedondomainknowl-edge.Anexamplewithawheeledmobilerobotshowsimprovedaccuracycomparedtotraditionalparticlelteringinwheelencoderandgyroscopediagnosiswithminimalcomputationalover-head.Wei,Huang,andChenevaluatetworecentlydevelopedapproachesforimportancesamplingchoosingthenextsetofparticlesbasedonthecurrentstate:mixtureKalmanlteringMKFandthestochastic-MalgorithmSMA,foruseinsensorFDIforasimulatedaircraft.TheresultsshowedthatbothapproacheseffectivelytrackedthestateofthesimulatedsystembutSMAwasmoreaccurateandefcientintermsofcomputationtimethanMKF.Plagemann,Fox,andBurgardusealearnedGaussianProcessmodelforimportancesamplingtoestimateboththefailurestateandfailureparametersforaPioneer3DXmobilerobot.TheGaussianProcessmodelwastrainedinsimulationthenusedonlinetoperformreal-timecollisiondetectionwithob-staclesthelaserrangendercouldnotdetect.Exampleresultsshowaccuratedetectionoftwocol-lisioneventswhichwereundetectedbyatraditionalparticlelterapproachandimprovedaccuracyintheestimatedtrajectoryoftherobot.NotethattheexperimentalresultsfromPlagemann,Fox,andBurgard,2007showadaptationtoacollisioncausedbyasensinganomalyundetectedobsta-clebutnoeffortwasmadetoimproveperception.InsteadthestatesresultingfromtheanomalycollisionwereconsideredsafeandwerelearnedaprioriduringthetrainingoftheGaussianPro-cessmodel.Othersensinganomaliese.g.missingaglassdoorcouldresultinanunknownandprobablyunsafestate.47

PAGE 62

2.2.6LearnedModelsInformedThissectiondescribessensorFDIapproachesthatlearntherelationshipbetweensensorread-ingsandthestatusofthesensorswithinaninformedframeworke.g.neuralnetworkdesignedtocaptureknownsystemcharacteristicsand/orsensorproblems.Theseapproachesareoftenmoreexiblethanpureanalyticalmodelswhichusedeterministicmathematicalmodelsofthetargetsystembutarestilllimitedbythedesigner'sassumptions.Aswithuninformedlearningmethodsthesearealsounlikelytodetectanomaliesthatarenotencounteredinthetrainingset.Fourap-proacheswerefoundintheliteraturewhichusethismethod.Thesearedescribedinorderfromtheleasttothemostinformedintermsofthestructureimposedbythesystemdesignersapproaches.DuandJinappliesprincipalcomponentanalysisPCAtorecordedsensorreadingstocapturetheessentialrelationshipsofthetargetsystemforsensorFDIforaheating,ventilationandairconditioningHVACsystem.Duetothecomplexityofthetargetsystem,apurelydata-drivenapproachwasnotused.InsteadknowledgeoftheunderlyingphysicsofHVACsystemswasusedtopartitioninputvariablesintothreegroupsknowntointeract,resultinginthreePCA-basedes-timatesofthesystemstate.FaultswereisolatedbycomparingtheanglebetweenthePCAmodelandagivenmeasurementwiththesameangleforallknownfaults.Expertruleswereusedtoim-provethenalisolationresultsbasedontheoutputofthethreePCA-basedmodels.ExperimentswithasimulatedHVACsystemshowedaccuratedetectionandisolationontheorderofminutesofconstantbiasesinthesensorreadings.Twoapproachesuseneuralnetworkstoreplacethetraditionalsystemmodel.Ranganathan,Patel,andSathyamurthypresentanFDIsystemtodetectasetofknownfaultconditionsforanunmannedunderwatervehicleUUV.AnarticialneuralnetworkANNisdesignedbasedondetailedknowledgeofninenon-catastrophicfaultsexhibitedbytheUUVandtrainedtodetectandidentifythosefaults.Afuzzyrule-basedexpertsystemFESreceivestheoutputoftheANN,suggestsrecoveryprocedures,anddeterminestheworkingstatusofthesystem.TheentiresystemwasimplementedonasingleVLSIchip.SimulatedtestdatawascorrectlyclassiedbytheANN94%ofthetimewhiletheFESprovidedcorrectinformation95%ofthetime.InDeuker,Perrier,andAmy,1998aneuro-symbolichybridFDIsystemispresentedforunmannedunderwatervehi-clesUUV.Expertknowledgeismodeledasrulesatthesymboliclevelthatarecompiledintoaneuralnetworkusingmathematicalmodelsofthesystem'snormalbehavior.Oncetheneuralnet-48

PAGE 63

workistrained,itisusedtoprocessrawsensordataontheUUVforfaultdetectionanddiagnosis.Thesystemwastestedinsimulationandtwoexampleactuatorfaultswereusedtodemonstratethesystem'sresponsivenessandexibility.Therstmodeledfaultwasefcientlywithinasec-onddetectedandcorrectlydiagnosed.Thesecondunmodeledfaultwasdetectedbutincorrectlydiagnosedsinceitpresentedsymptomssimilartorstexamplefault.Anewrulewasaddedtotheknowledge-basetohandlethismiss-classiedexampleandtheneuralnetworkwasmodiedandre-trained.AlocalizedapproachwaspresentedinKawabataetal.,2003whereFDIsystemsincludingaddeddiagnosticsensorsweredevelopedforeachsubsysteminthedrivesystemonaZEN-450omnidirectionalrobot.Thenormalvarianceofeachsensorvalueinthefault-freestateisrecordedandusedtondanappropriatethresholdforonlinemonitoring.Comparisonofvaluesamongre-dundantsensorsisusedtoisolatethefault,e.g.whetherthefaultisinthehardwareorasensor.Detailedknowledgeofeachsubsystemandofdiagnosticsensorreadingsareusedtodeterminetheimpactofafaultandtoselectanappropriatemethodforaccommodation.Experimentswereper-formedontherealrobotwithtworealhardwarefaultsdisablingtranslationalongx,thenalongyintroducedinthedrivesystemduringoperation.Theresultsofthoseexperimentsshowedcorrectdiagnosisandaccommodationofthefaults.2.2.7AnalyticalModelsThissectionpresentscontroltheoreticapproachestotheproblemofsensorFDIwhichemployanalyticalmodelsofthetargetsystemtodetectanddiagnosetraditionalsensingproblems.Theseapproachestypicallyuseasinglemodelofthenominalstateofthesystemcoupledwiththresholdstodetectfailures.Traditionallysomeformofinferenceisthenusedtoidentifythefailurebasedonthefeaturesoftheresiduals,i.e.thedifferencebetweenthepredictedandactualsystemoutput.Whenrecoveryisautomateditisusuallycarriedoutbyanadaptivecontroller.6Theseareunsuitedforhandlingsensinganomaliesinunknownenvironmentsduetotheneedforapriorifaultmodelsand/ordetailedmodelsofthesystemundernormalconditionswhichrestrictstheseapproachestoFDIofknownfaultsincontrolledenvironments. 6ForabriefperiodthemostactivegroupinthisareawastheRobotsforHazardousEnvironmentsGroupatRiceUniversityledbyCavallaroandWalkerseeforexampleDixonetal.,2000;McIntyreetal.,2004;Visinsky,Cavallaro,andWalker,1995.49

PAGE 64

SixpapersusepurelymathematicalapproachesforsensorFDI.UmandLumelskycou-plemotorcontrolsignalswithakinematicmodelofasixdegree-of-freedomindustrialrobotarmtopredictthereadingderivativeschangeovertimeforanetworkofinfraredsensorsinstalledonthemanipulator.Sensorswhosereadingsdeviate,withinanexperimentallydeterminedlimit,fromtheestimatedderivativewereconsideredfaulty.Onlineexperimentsfoundthatextremedeviationsweredetectedduringnormaloperationbutdetectionofminorerrorsrequiredthepresenceofaatcalibrationobstaclevisibletoallsensorsinthenetwork.Vinsonneauetal.usenon-linearmodelstopredictairowandmanifoldpressureinaJaguarengineatvaryingatmosphericpres-suresandambienttemperatures.Usingthesemodelsasabaselinefordevelopingobservers,resid-ualsforthreeimportantfailuresaredeveloped:afaultyairowsensor,afaultymanifoldpressuresensor,andanairleakinthemanifold.Asimplelogictablewasusedtodeterminewhichfaulthadoccurred.Experimentsperformedwithasimulationoftheenginewithperiodicallyinjectedfaultsshowedthattheresidualscouldbeusedtodetectanddiagnosethethreemodeledfaults.InVosandMotazed,1996anapproachforfaulttolerantcontrolwaspresentedforunmannedaerialvehiclesUAVswithanalyticalredundancies.Propercontrolofsuchsystemsislinearparame-terdependentLPD,ordependentonparameterse.g.altitudeorspeedwhichvaryovertime.TheapproachtakesadvantageofpreviouslydevelopedmodelsforUAVswhichbehavelikelineartimeinvariantLTIsystemse.g.cruisingataconstantaltitudeandspeed.ThepaperutilizesapreviouslydevelopedapproachforreducingLPDsystemstoLTIsystemsusingcoordinatetrans-formsandafeedbackcontrollaw.VosandMotazedapplyexistingLTImodels,theninvertcoordinatetransformstoderiveobserversformonitoringanLPDsystem.SimulationsusingrealtelemetryfromaightwithaverticalgyrofailureshowtimelyfaultdetectionandrecoveryinacasewheretheactualUAVlostcontrolandcrashed.Pertew,Marquez,andZhaopresentanapproachforsensorfaultdetection,diagnosis,andmagnitudeestimationfornon-linearsystemsthatareLipschitzcontinuousi.e.thereexistsaconstantC,wherethechangeinsystemstateoroutputneverexceedsC.Theapproachusesasinglelinearmodelthatisdynamicallyupdatedbasedonstateandmeasurementestimationerrors.Mathematicalproofsareprovidedtoshowthattheapproachcanbeusedtoperfectlydetect,diagnose,andestimatethemagnitudeofsens-ingfaults.NoexperimentalresultswereprovidedinPertew,Marquez,andZhao,2007.Gao,Breikin,andWangpresentanapproachforsensorFDIsystemswitharbitrarystatechange50

PAGE 65

andinputdelays.Theapproachimprovesonthetraditionallinearobservermodeltoexplicitlymodeldelays.Twoobserversarederived:oneforthestateestimateandoneforanaprioriun-knownsensorfaultwhichmaybeunbounded.Bothareusedtocreateastablefeedbackmodelforfault-tolerantcontrolofthetargetsysteminthepresenceofsensorfaults.Asimulatedmodelofthreecascadedreactorsshowsaccuratesensorfaultdetectionandaccommodationofstuck-at-constantandunboundederrors.Rothenhagen,Thomsen,andFuchsusealinearmodelofadoublyfedinductiongeneratorDFIGtocreateanobserverforavoltagesensorusedasinputinthegenerator'scontrolsystem.ExperimentswithasimulatedDFIGcontrollershowaccuratefaultdetectionforhardsensorfaults.Theresultsalsoshowedthattheobservercouldbeusedtoestimatethecorrectvoltagereadinginthecaseofsensorfailure.Threepapersrelyonstandardmathematicalmodelsassystemobserversandincorporateprob-abilisticorfuzzymodelsformorerobustdetectionanddiagnosis.SchwallandGerdescombinedthetraditionalcontroltheoreticapproachwithadynamicBayesiannetworkforFDIofthedrivingsystemofacar.Residualsgeneratedfrommathematicalmodelswererecordedduringfault-freedrivinginvaryingconditionsandaGaussiandistributionwasusedtomodeltheirbe-havior.Thesemodelswerealsousedtoestimatetheresidualdistributioninthepresenceofoneormorefaultsbyincreasingthevarianceten-fold.AdynamicBayesiannetworkwasbuilttoestimatetheprobabilityofeachpossiblesystemfaultbymodelingtherelationshipsbetweentheresidualsandfaults.Oneormorefaultsaredetectedwhentheirprobabilityexceedsapredeterminedthresh-old.ThesystemwastestedusingdatafromaMercedesE320withminortransitoryfaultsinjectedofine.SchwallandGerdes,2002showedcorrectifdelayeddetectionanddiagnosisoftwomi-nortransitoryfaults.Monteriuetal.busestructuralanalysisofnavigationsensorsGPS,encoders,andIMUonanATRV-JrtogenerateresidualsforsensorFDIbasedonanalyticalredun-danciesinthenavigationsystem.Twomethodswereproposedtodetectandisolatehardsensorfaultsbasedontheresiduals.Therstapproach,referredtoasadhoc,dividedtheresidualscalcu-latedovertimeintodistincttime-slicesandusedsimplethresholdingoftheresidualvaluebasedonitsrangeintheprevioustime-slice.Thesecondproposedapproachusedparticlelterstoesti-matetheprobabilityofafaultgiventheresidualvalue.ExperimentalresultspresentedinMon-teriuetal.,2007ashowedthatbothapproachesforresidualevaluationproducedequallyaccurateresults.Singleandmultiplefaultscenarioswereproducedbyinjectingadditiveerrorsinthesensor51

PAGE 66

readings.Allfaultsweredetectedwithinonetimestepandallsinglefaultswerecorrectlydiag-nosed.BlakeandBrownpresentanapproachforsimultaneoussensorandactuatorfaultFDIfornon-linearsystems.Thenon-linearsystemisdecomposedintofuzzysetsoflinearmodels.Traditionallinearcontrolmodelsareconsideredwhichprovidesystemstateandoutputestima-tionsbasedonpriorinputandoutput.Inthesemodelstheadditiveeffectsofsimultaneoussensorandactuatorfaultsaremadeexplicit,enablingthecreationofdetectionanddiagnosisobserversforbothtypesoffaults.Theglobaldetectionanddiagnosisobserversareobtainedfromaweightedsumoftheoutputofeachlinearmodel,wheretheweightsarethemodels'membershipinthecur-rentglobalfuzzystatee.g.foranairplanethesemightbetake-off,cruising,andlanding.Twonumericalexamplesshowaccuratedetectionanddiagnosisofsimultaneousadditivesensorandactuatorfaults.2.2.8DiscussionThestudiesexaminedinthissectionhaveproposedawidevarietyofapproachesforsensorFDIfortraditionalsensorfaultswhich,unlikesensinganomalies,existregardlessoftheoperat-ingenvironmentofthesensorandtendtobeisolatedonlyaffectonesensoratatime.Forthesetraditionalsensingproblemsaprioriknowledgecanbemodeledorlearnedtofacilitatedetectionanddiagnosisoffaults,butsuchknowledgeisoflittleuseforsensinganomalieswhichdependoninteractionwithanaprioriunknownenvironment.ApproacheswhichrelyonanalyticalmodelsofthesystemareparticularlyunsuitedforFDIincomplexunknownenvironmentsforexterocep-tivesensorse.g.laserrangendersandcameraswhichmeasureaspectsoftheenvironment.ThedifcultlyofexplicitlymodelingsuchenvironmentsisreectedinthefactthatthesesystemsaredesignedformanipulatorsUmandLumelsky,1999whichoperateincontrolledenvironments,UAVsVosandMotazed,1996whichoperateintherelativelyunclutteredaerialdomain,orin-ternalsensorsVinsonneauetal.,2002whicharelargelyunaffectedbytheenvironment.Ap-proachesthatusequalitativeand/orstochasticmodelsorlearningmethodsthatrelyonaprioriinformationaboutfaultstatesDeuker,Perrier,andAmy,1998;Ranganathan,Patel,andSathya-murthy,2001canbeexpectedtodetecttraditionalsensorfaultsinunknownenvironments,buttheyareunlikelytodetectproblemslikesensinganomalies.Forlearning-basedapproachesthatdonotrelyonaprioriinformationtheabilitytodetectproblemslikesensinganomaliesishighly52

PAGE 67

dependentonthepresenceofsimilaranomaliesinthetrainingsetandhowwellthelearnedsystemgeneralizestoanovelenvironment.SystemslikeCanham,Jackson,andTyrrell,2003;ChenandSaif,2007;Guetal.,2001;Herediaetal.,2008;Larkey,Bettencourt,andHagberg,2006whichuseunsupervisedlearningcouldtheoreticallyprovidetherobotwiththeabilitytolearnaboutsens-inganomaliesonlineinanunexploredenvironment,butwouldleavetherobotvulnerabletosuchfaultsduringthelearningperiod.Consensus-basedapproacheslikeSoika,1997adonotrelyonmodelsorlearningandcouldtheoreticallybeusedtodetectanddiagnosesensinganomaliesbuttheserequireanaccurateconsensus.Sincetheenvironmentalconditionswhichleadtosens-inganomaliesoftenaffectmultiplesensors,theseapproacheswouldrequireanintelligentagenttoactivelyuseawidevarietyofsensorsatalltimestoensurethatsensinganomaliesareaccu-ratelycharacterized.Theinconsistency-basedapproachdescribedinAfgani,Sinanovic,andHaas,2008a,bissimilartotheapproachpresentedinChapter3inthatitalsodetectsanomaliesbyap-plyingathresholdtoametricderivedfromtherecordeddata.TheapproachdescribedinAfgani,Sinanovic,andHaas,2008a,bwasdevelopedtodetectanomaliesinperiodicwirelesssignalsandislimitedtosignalsi.e.sourceswhichshowhighperiodicityundernormalconditions.Forthemostcommonlyusedsensorsformobilerobotse.g.wheelencoders,GPS,laserrangenders,sonar,camera-basedvisionsystemstheattributestheyaredesignedtomeasurearenotperiodic,thereforehighperiodicitycannotbeassumed.NotethatwiththeexceptionofAfgani,Sinanovic,andHaas,2008a,bwhichwasdevelopedtodetectanomaliesinwirelesssignalsallofthestud-iesdescribedinthissectionarefocusedontheproblemoftraditionalsensorfaultssuchasdriftandmis-calibration.Noneoftheapproachesdiscussedinthissectionweredesignedtoaddresstheproblemofsensinganomalies.2.3IsolatingPoorlySensedRegionsTwostudiesfoundintheliteraturepresentspecializedapproachesforisolatingpoorlysensedregionsseeTable5bymeasuringinconsistencywithinasetofsensorreadingsRomanandSingh,2006ordisagreementwithatrustedsourcestereopairofcamerasBaltzakis,Argy-ros,andTrahanias,2003.Notethatneitheroftheseapproacheswerespecicallydesignedtoaddresstheproblemofsensinganomalies.InRomanandSingh,2006aspecializeddistancemetricisappliedtoconsecutive3Dscansmodeledaspointcloudstoisolateinconsistencies.53

PAGE 68

Table5.Relatedworkonisolatingpoorlysensedregionsbymethod. Method Studies Inconsistency-based Thiswork,Roman Inconsistency-basedtrustedsource Baltzakis Sincetheapproachdoesnotrequireaprioriinformationitcouldbeappliedtosensinganomalies,butwouldbelimitedtonon-dynamicdatamodeledaspointclouds.InBaltzakis,Argyros,andTrahanias,2003theapproachforcomparinglaserandstereorangecamerareadingsrequiresanindoor,unclutteredenvironmentandthestereopairofcameraswasassumedtobeaccurate.Theseassumptionswouldseverelyrestricttheoperatingenvironmentsofasensinganomalycharacteri-zationsystem.TheapproachdescribedinChapter3ismoregeneral;itcanbeusedwhereverthetruestateisinternallyconsistentandevidentialmodelsareusedtoaccumulateinformationfromsources.RomanandSinghpresentaspecializedmetricformeasuringinconsistencyinpartiallyoverlappingscansfroma3Dsonarsensorwhichdoesnotrelyonaprioriinformationregardingtheenvironmentorthesensorbutisonlyapplicabletonon-dynamicdatamodeledaspointclouds.Thegoalofthestudyistomeasurethereliabilityofpointclouddatathroughoutthemapandpro-videavisualguideforend-users.Asimpledistancemeasurementi.e.thicknesswasusedtode-scribeinconsistency.Thepointcloudwasdividedintobinsandarandompointwasselectedfromeachscanthatcontributedmeasurementstoagivenbin.Foreachpointselected,thenearestpointineveryotherscanwhichprovidedreadingstothecurrentandsurroundingbinswasdetermined.Themaximumdistancetothenearestpointoneachscanservedastheinconsistencyorthicknessmeasure.TwoexamplesusingrealscanstakenindeepwaterwereusedtocomparetheapproachtoPCAbasedonvariance.Theresultsshowedimprovedisolationofinconsistentregionswithinthemapandinsensitivitytobinsizevariance.Anexampleusingthenewmetrictoimproveregis-trationofanewscantoaglobalmapwasalsoprovided.Baltzakis,Argyros,andTrahaniaspresentanapproachforisolatingpoorlysensedre-gionsina3Dmodelbuiltfrom2Dlaserrangescansbasedondisagreementwitha3Drangeimagefromastereopairofcameraswhichwasassumedtobeaccurate.Anindoor,unclutteredenviron-mentwasassumedwherelaserreadingsinterpretedsimplyaspointscorrespondtopointsonwallsand/orcorners.Usingthisassumptionaloneathreedimensionalmodeloftheworldisbuilt54

PAGE 69

usingthelaserdata.Thestereopairofcamerasisusedtovalidatethemodel.Thisisdonebycom-paringthelaser-builtmodelwithonebuiltfromthestereopair.Thenormalizedcross-correlationmetricacommonlyuseddifferencemetricincomputervisionbasedonthesquareddifferencesbetweentheintensitiesoftwoimagesisusedtomeasurethegoodness-of-tbetweenthetwomodels.Asthestereomodelisassumedtobeaccurate,thevalueofthismetricisusedtodeter-minehowwellthelaserscannerisperforming.Thestereopairisthenusedtoprovideadditionalmeasurementsoftheareaswherethelaserdatawasdeterminedtobeinaccurate.Usingafusedoccupancygridgeneratedbythecombinationoflaserandstereorangedatawhereneededobsta-cleavoidanceandpathplanningweretestedonaniRobot-B21rinanindoorhallway.Examplesshowedimprovedaccuracyoverusinglaserdataalone,butnoquantitativeexperimentalresultswereprovided.2.4AdaptiveFusionThissectionisfocusedonapproachesseeTable6foradaptivesensorfusionorbeliefrevi-sionwhichadjusttheimpactofagivensource'sinformationbasedontheestimatedoraprioriknownsourcereliability.OneofthesestudiesMorales,Takeuchi,andTsubouchi,2008presentsaspecializedapproachfordetectingsensinganomaliesinGPSreadings.Sincenoneoftheotherstudiesaddressedsensinganomalies,theseapproachesarecategorizedandevaluatedaccordingtothemethodusedtoadjustfusionparametersasopposedtoexperimentalresults.Indecreas-ingorderofsuitabilityforsensinganomaliesthesemethodsareasfollows:inconsistency-based,consensus-based,consensus-basedpair-wise,inconsistency-basedusingatrustedsource,andaprioriknownsensorcharacteristics.Theapproachespresentedinthissectionwithoneexcep-tion,Kobayashi,Avai,andFukuda,1999useeitherinconsistency-basedorconsensus-basedmethods.Consensus-basedapproachesreducetheimpactofunreliablesensorsbyusingeitherpair-wisecomparisonsDelmotte,Dubois,andBorne,1996oracomparisonofinformationfromasinglesensorandthefusedresultofmultiplesensorstoadjustfusionparameters.Thesestud-iesmaketheassumptionthatamajorityofsensorswillbeaccuratewhichmaynotbetrueforasensingsystemusedinanunknownenvironmentinthepresenceofsensinganomalies.InAyruluandBarshan,2002anadaptivefusionapproachwasdevelopedtoimprovefeatureclassicatione.g.cornersversuswallsforasetofsonarsensorswhereanadhocinconsistencymetric,applied55

PAGE 70

Table6.Relatedworkonadaptivefusionbymethod. Method Studies Inconsistency-based Ayrulu Consensus-based Parra-LoeraYuCohen Consensus-basedpair-wise Delmotte Inconsistency-basedtrustedsource LiMorales Knownsensorcharacteristics Kobayashi toDempster-Shaferbeliefmasses,servesasaweighttoreducetheimpactoflessreliablevotesontheoverallfusedclassication.BothLiandPham,2003andMorales,Takeuchi,andTsub-ouchi,2008useinconsistency-basedmethodswithtrustedsources.LiandPhamdevelopsatheoreticalframeworkforbeliefrevisionwherelocallyheldbeliefsarealwaystrusted.Morales,Takeuchi,andTsubouchidetectanomaliesinGPSdatabyapplyingastandardoutliertestforKalmanltermodelsi.e.thenormalizedinnovationsquaredtesttomeasuretheinconsistencybetweenGPScoordinatesandtheestimatedpositionbasedonencoderandIMUreadings.InAyruluandBarshan,2002anadhocinconsistency-basedapproachforadaptivesensorfusionispresentedtoimprovepatternrecognitionforrobotsinunknownindoorenvironments.InthisstudybothmajorityvotingandDempster-Shaferbasedfusionwereusedforrecognitionofthreebasicindoorwallpatternsplane,corner,andacutecornerinanemptyroom.Fivemetricsweredevelopedtomeasurethetrustworthinessofareading.Fouroftheseusedbasicpropertiese.g.sensorrangeandangularresolutionsimilartotheinformationencodedintraditionalsonarmodels.Thefthmetricmeasuredthedifferencebetweenthebeliefmassesassignedtoeachsen-sor'srstandsecondchoice.Thislastmetricperformedbestintermsofimprovedclassicationaccuracy.EachmetricwasusedasaweightinboththemajorityvotingandDempster-Shaferap-proaches,reducingtheimpactoflessreliablehypotheses.Theexperimentsused7Panasonicsonarsensorsinvaryinglocationsinvedifferentrooms.TheresultsshowedslightimprovementsinaccuracyforDempster-Shaferlessthan4%withamaximumof90%classicationaccuracy.TheresultsformajorityvotingshowedthatincorporationofthemetricsimprovedtheaccuracysothatitconsistentlyoutperformedDempster-Shaferupto95%accuracyateachlocationineachroom.Consideringonlyuseofthefthmetric,theadaptivesensorfusionapproachadjuststheinuenceofareadinginafusedresultwithoutrelyingonaprioriinformationregardingtheenvironmentorthesensors.Theapproachdevelopedforthisdissertationalsoappliesmetricsdi-56

PAGE 71

rectlytobeliefmassestodetectandcharacterizesensinganomalies.Inthisworkthemetricsusedaretheoreticallygroundedinconsistencymetricsfromtheuncertaintyliterature.TheexperimentalresultsprovidedinAyruluandBarshan,2002showthatanadhocambiguitymetric,whenap-plieddirectlytobeliefmasses,canbeusedtoimproveclassicationaccuracy.ThisresultsupportsthehypothesisthattheapproachpresentedChapter3,ifusedinanadaptivefusionsystem,couldprovidesimilarimprovements.YuandSycarapresentaconsensus-basedapproachforadaptivesensorfusionfordis-tributedtargetrecognitionforunmannedaerialvehiclesUAVs.TheapproachissimilartopriorworkonestimatingsourcereliabilityseeYuandSingh,2003a.InYuandSycara,2006amo-bilesensore.g.UAVwillidentifyatargetbyreportingitslocationandasetofpotentialvehicletypeswithacondencelevelassignedtoeach.Ifthehighestcondencelevelinthesetdoesnotmeetapre-determinedthresholdseeEq..11,thenthesensorpropagatesthisinformationtoitsneighbors.WhenaUAVreceivesinformationfromapeeritwilluseavariantofDempster-Shafertheorytofusethisinformationwithothermessagesaboutthesametarget,thenrecalculatethecon-dencelevel.Messagepassingstopswhenthethresholdisreached.CTk=max8A2mA+m.11AsinYuandSingh,2003areliabilityweightsover[0;1]areassignedtoeachsensorandusedtoreducetheinuenceofunreliablesensorsduringfusion.InYuandSycara,2006theweightsareupdatedbasedonacustomconictmeasurecomparingtheinformationfromagivensensormiXwiththatofthefusedresultm0X:conflict=m0fAg)]TJ/F18 10.909 Tf 10.909 0 Td[(mifAg)]TJ/F18 10.909 Tf 10.909 0 Td[(mi.12wherem0fAgismax8A2m0fAg.Theseweightsarethenusedinamodiedcombinationrule:m00iA=wimiA PLi=1wi.13m00i=wimi PLi=1wi+1)]TJ/F18 10.909 Tf 25.872 7.381 Td[(wi PLi=1wi.1457

PAGE 72

whereAandA6=,tocreatethefusedresult.Thisapproachwastestedusingvesimu-latedautomatictargetrecognitionsystemsbasedonunmannedaerialvehiclesUAVswithsyn-theticapertureradarSAR.Anexampleshowedcorrectidenticationofthetargetvehiclewith0.7condencewithvereportedsightings.Thereliabilityestimationresultsfromthisexampleweremoredubious.Althoughonlyonesensormis-classiedthetarget,thereliabilityofallsen-sorsdecreasedto0.4or0.5outof1.0injusttwoupdates.ThecondencemetricproposedinYuandSycara,2006isinterestinginthat,liketheapproachproposedhere,itattemptstoquantifythereliabilityofinformationprovidedbysensorsasawhole.Unfortunatelythismetricisawed,forexampleitmapstotalignorancem=1:0tocompletecondenceCTk=1:0,andthereforecannotbeusedtocharacterizesensinganomaliesinunknownenvironments.InParra-Loera,Thompson,andSalvi,1991apreviouslydevelopedsensorfusionsystemformulti-targettrackingisextendedbyincorporatingaconsensus-basedsensorreliabilitymetric.AsinYuandSycara,2006,eachsensorreportsthelocationandclassicationofatargetwitharawcondencefactorrcf.ThereadingsfrommultiplesensorsareclusteredandfusedusingDempster-Shafertheory.Adomain-dependentcharacteristicequationisusedtomeasurethedif-ferencedareaunderthecurvebetweeneachsensorreportandthefusedresult.Thisvalueisusedtoupdatethesensor'scondencecorrectionfactorccf,seeEquation.15.Theccfwillcontroltheinuenceofthesensor'sreportsinthenextroundofclusteringandfusion.ccf[k+1]=1)]TJ/F24 10.909 Tf 12.105 17.243 Td[(Pni[k]j=1d[k]rcf[k]ccf[k)]TJ/F15 10.909 Tf 10.909 0 Td[(1] ni[k].15Theapproachwastestedusingvetargettrackingsensors7.Theresultsshowedlittledeviationintheccf'softhesensors,reectingonlyminordifferencesbetweenthesensorreadingsandthefusedresult.CohenandEdandescribeaconsensus-basedsystemforonlineautonomousselectionoffusionalgorithmsandlogicalsensors8withoutrelyingonaprioriknowledgeoftheenviron-ment.Inthissystemeachlogicalsensorprovidedabinary0forempty,1foroccupiedrobot-centricoccupancygridmap.BasedontheresultsofpriorworkseeCohen,2006twoalgorithms 7AdetaileddescriptionoftheexperimentalsystemwasnotprovidedinParra-Loera,Thompson,andSalvi,1991.Whetherthesensorswererealorsimulatedisunknown.8AcomputationalprocessoperatingonoutputfromphysicalsensorsorotherlogicalsensorsadaptedfromHender-sonandShilcrat,1995.58

PAGE 73

wereemployedforsensorfusionofthesemaps:simplemajorityvotingdenotedMOSTandacomplexadaptivefuzzylogicfusionproceduredenotedAFLwhichusessensorperformancemeasurestomorerobustlydeterminethestatusofeachcell.Theresultingfusedmapiscomparedwitheachsensor'smap,resultinginatruepositiverate,truenegativerate,falsepositiverate,andfalsenegativeratewhichareusedasthesensorperformancemeasuresinthenextroundoffusion.Alinearcombinationofthesemetricswasusedasanoverallassessmentofeachlogicalsensor'sperformance.Iftheoverallassessmentofthesensorwassufcientlylowwithanempiricallyde-terminedthresholdorthesensorreturnedallemptyoralloccupiedmaps,itwasautomaticallydisabled.ThesystemautomaticallyswitchedbetweentheMOSTandtheAFLalgorithms,usingthemorecomplexAFLalgorithmonlywhenneeded.Thissystemwastestedinaseriesofexper-imentswithrealsonarrangesensorsandcameras.Multiplealgorithmswereusedtointerpretthereadings,resultinginvelogicalsensorstwosonar-basedandthreecamera-based.Atleastonelogicalsensorwasmanuallysettoreturnallemptyineachexperiment.Thetestbedincludedrealandfalseobstaclestoconfusethecamerathelatterorwhichwererandomlymovedthroughouttheexperiments.TheresultsprovidedstaticallysignicantevidencethattheadaptivesystemandAFLprovidedsuperiormapsascomparedtosimplelogicalfusionofthesensorreadings,withtheadaptivesystemwhichusedMOSTthemajorityofthetimeperformingbest.Therewereseveralcasesinwhichtheadaptivesystemfailedduetoaninaccurateconsensus.Delmotte,Dubois,andBorneproposeameansofincorporatingconsensus-based,expert-based,andcontext-basedreliabilityinpossibilisticfusionofsensordata.Consensus-basedsourcereliabilitytintforsensoriistheaverageagreementoverallsensorsreportingonthesamevariableasshowninEquation.16.AgreementistheintersectionrijgiveninEquation.17oftwopossibilitydistributions:ixprovidedbythesensorbeingevaluatedandjxanothersensorreportingpossiblevaluesforx.Thetotalnumberofsensorsreportingpossiblevaluesforxisn.tint;i=Pj6=irij n)]TJ/F15 10.909 Tf 10.909 0 Td[(1.16rij=maxxminix;jx.1759

PAGE 74

Expertswereassumedtoprovidefuzzynumbersdescribingthereliabilityofeachsensorandsincetheexperts'reliabilitycouldnotbedeterminedafuzzyunionovertheiranswerswasused.Context-basedreliabilitywasperformedbyallowingtheenvironmenttoserveasoneoftheex-pertsandbymodifyingtheinternalreliabilityinproportiontothereliabilityassignedbytheenvi-ronment.Thatis,Equation.16becomesEquation.189wheresiisthereliabilityassignedtosensoribytheenvironment.tint;i=siPj6=irij n)]TJ/F15 10.909 Tf 10.909 0 Td[(1.18InDuboisetal.,1997theapproachisdevelopedintoamulti-modelcontrollerforarobotwhereactionselectionistreatedseamlesslywithsensorfusioninapossibilisticframeworkandeachmodelrepresentscharacteristicse.g.robustnessorefciencyofthecontroller.Theresultingcontrollerispresumedtoprovidesmoothercontrolandallowtherobottoadaptitsmodelson-linebasedoncontrollerperformancebutnoexperimentalresultswereprovidedtosupporttheseclaims.InLiandPham,2003aninconsistency-basedapproachforbeliefrevisionispresentedwherelocallygatheredinformationisalwaystrusted.AgeneralmathematicalframeworkispresentedforbeliefrevisionwhereinformationisrepresentedasanintervalstructureandprobabilisticagentreliabilityisusedtoderivethedegreeofbeliefaDempster-Shaferbeliefmassinthatinforma-tion.Ifanagentsuppliesaconictmessage,wheretheintersectionofthemessageandlocalob-servationsisempty,thesourceagent'sreliabilityistransferredtoitsneighbors.Anagent'sneigh-borsaredenedasagentswithinthesameequivalenceclassestablishedaprioriwhosemes-sagesintersect.Forexampleanagentisgivenanimageofanobjectbutcanonlyobservethatitisred.OtheragentsinthesameclassA1,A2,andA3supplymessagestheobjectisanapple,theobjectisagreenapple,andtheobjectisastrawberryrespectively.SincethemessagefromA2conictswiththelocalobservationbutnotwiththemessagefromA1,thereliabilityofA2isaddedtothatofA1thensettozero,resultinginalargerbeliefthattheobjectisanappleinsteadofastrawberry.Inthismannerconictinginformationisneitherdiscardedasintraditionalbeliefrevisionnorallowedtocorruptanagent'sbeliefwithinconsistentinformation.Insteadconicting 9Inthepaper,thisequationwasgivenwithoutthedenominatorn)]TJ/F44 8.966 Tf 9.561 0 Td[(1whichmakeslittlesenseandispresumedtobeatypographicalerror.60

PAGE 75

informationisusedtobolsterbeliefincorroboratingmessagesthatdonotconict.Neithercon-creteexamplesnorexperimentsweredescribedinLiandPham,2003.Morales,Takeuchi,andTsubouchipresentasensorfault-tolerantfusionsystemforout-doorposeestimationusinganIMU,wheelencoders,andGPSwhichmeasurestheinconsistencybetweennewGPSreadingsandotherposesensorsi.e.wheelencodersandanIMUtodeterminewhentoexcludeGPSreadingsi.e.detectionofGPSsensinganomalies.AnextendedKalmanlterEKFisusedtofusetheaveragevelocityreportedbytheencodersandtheyawvelocityre-portedbytheIMUtoproduceanestimateoftherobot'sposition.TheGPSisusedtoperiodicallycorrecttheestimatedposition,thusreducingtheeffectofaccumulatederrorsfromtheencoderandIMU.SinceGPSreadingsarenotalwaysreliableatwo-stageprocesswasusedtorejectpoten-tiallyinaccuratereadings.IntherststageaheuristicwasappliedtothediagnosticdataprovidedbytheGPS,e.g.rejectreadingswithlowreportedprecisionortoofewsatellites.ThenormalizedinnovationsquaredNIStestwasappliedtoallreadingsthatpassedtherststage.Thistestcon-sidersthedifferencebetweenthecurrentestimateandnewdatacalledtheinnovationaswellasthelter'scondenceinthecurrentestimate,i.e.thehigherthecondencethesmallerthein-novationmustbetopass.TheNISmetrichasachi-squareddistributionsodatarejectioncanbeperformedinthesamemannerasthetraditionalchi-squaredtestforoutliers.ExperimentswereperformedbyteleoperatingaYamabicomobilerobotequippedwithaCrossbowIMU,tworotaryencoders,andaGPSusingaStarFireDifferentialServicealongapathover300meterslongwithvaryingtreecover.ResultsshowedthattheEKFprovidedaccuratelocalizationwithin3manduncertaintyestimation,withthelargestdriftcorrespondingtothethickesttreecoverwheremostoftheGPSreadingswererejected.Kobayashi,Avai,andFukudapresentasystemforadaptivesensorfusionbasedontheaprioriknownreliabilityasensorwhenitprovidesreadingsforasubsetofitsrangetoarecurrentneuralnetworklocalizationsystem.Therangeofasensorisdividedintointervalsandanindi-vidualrangereadingistreatedasahitwithinitsassignedinterval.Foreachreading,apossi-bilisticreliabilitymetricisdenedbasedonapriorirangeerrorestimateswhichmayvaryoverasensor'srangeandthedistancefromthereadingtothecenterthereading'sassignedinterval.Thismetricisusedtoselectsensorswithmorereliablereadingstoensurehighaccuracywithintherangeofallsensors.Arecurrentneuralnetworkacceptsinputintheformofhitswithin61

PAGE 76

rangeintervalsandthecalculatedreliabilityfortheselectedsensorreadings.Botharepropagatedthroughtheneuralnetworktoderiveanoverallreliabilityofthenetwork'soutput.Theapproachwastestedinsimulationwithandwithoutthesensorselectionfeaturewithnearlyidenticalresults.2.5InconsistencyMetricsThissectionpresentsmetricsdevelopedtomeasureinconsistencyinasystemthatmodelsun-certaintyquantitatively,i.e.usingprobability,possibility,orevidentialDempster-Shafertheo-ries.Thesemetricsattempttomeasuretwoformsofinconsistency.Section2.5.2describesmetricswhichmeasuretheamountofevidencesupportingcontradictoryhypotheses,alsoreferredtoasconictmetrics.Section2.5.3describesmetricsthatmeasurethedegreeofinconsistencywithinamodeltheseoftenreducetoentropyorbetweentwomodels.Thedifferencebetweenthesetwotypesofinconsistencyisintuitivelyunclear,indeedLiuaarguesthatbothareneededtondandquantifytrueinconsistencyseeSection2.5.4.SinceDempster-Shafertheoryisnotaspreva-lentasprobabilitytheoryorfuzzysettheory,thissectionopenswithabriefreviewofDempster-ShafertheoryinSection2.5.1.Themetricsdescribedinthissectionwillbeevaluatedbasedonthreecriteria:applicability,complexity,andexposure.Applicability.Towhatuncertaintymodelscanthemetricbeapplied?Complexity.Howdifcultisthemetrictocalculate?Exposure.Hasthemetricbeenusedtosolveavarietyofsimilarproblems,orhaveitsapplica-tionsbeeneitherpurelytheoreticalorunrelated?Theapproachproposedinthisdissertationfordetectingandcharacterizingsensinganomaliesinunknownenvironmentsisapplicabletoabroadfamilyofproblemsincludingbutnotlimitedtosourceassessmentandassociationinbeliefrevision.Asaresultawidelyapplicableinconsistencymetricispreferred.Applicationsofintelligentagentsoftendealwithlargevolumesofinforma-tione.g.webcrawlers,tighttimeconstraintse.g.real-timesystems,orbothe.g.mappingformobilerobots,thusthecomplexityofthemetricisaconcern.Finallyametricwhichhasbeenproventhroughrepeatedapplicationstorealworldproblemsisconsideredmorelikelytosucceed62

PAGE 77

Table7.Metricsformeasuringinconsistencyinquantitativeuncertaintymodels. Metric Applicability Complexity Exposure Conict-BasedMetricsSection2.5.2 Con Dempster-Shafer Priorwork TBMm Smet'sTBM Association,Detectingchange,Priorwork IncK Fuzzyknowledgebases n Unrelated DegreeofInconsistencyMetricsSection2.5.3 AntD Fuzzysets n Unrelated TC[;m] Evidentialmodels n2 Theoretical AMm Evidentialmodels log2n Theoretical AbellanandGomez Incompatiblecredalsets n2 Theoretical CombinedMetricsSection2.5.4 Liua Evidentialmodels,Fuzzysets n Theoretical ascomparedtoanewlyproposedanduntestedmetric.Inordertomaintainthefocusofthischap-ter,theexposureofametricwillonlyconsiderapplicationswithinthescopeofthisliteraturere-view.Table7evaluatestheinconsistencymetricsdescribedinthissectionbasedontheirapplicabil-ity,complexity,andexposure.InthistableEvidentialmodelsreferstoallvariantsofDempster-Shafertheory.Complexityisdenedintermsofn=j2janddoesnotincludethestepsrequiredtocalculateanupdatedbeliefmassorfuzzyset.Thistableshowsthatthemostwidelyapplicableinconsistencymetrics:TC[;m],AMm,and,arenon-trivialtocalcu-lateandhaveonlybeenvalidatedfortheoreticalexamples.Shafer'sConandSmets'TBMmaretrivialtocalculateandhavebeenusedforseveralapplicationsrelevanttothisworkincludingpriorworkbytheauthor,buttheyhavemorelimitedapplicability.InthisworksensorreadingsarefusedusingDempster-ShaferorSmets'TBM.SincefuzzysetsaretheoreticallyasubsetofSmet'sTBMseeSection2.5.1theonlymetricsconsideredirrelevantforthisstudyareIncKandAbellanandGomez.Theremainingmetrics,saveforAM,10wereexaminedaspoten-tialsensingaccuracyindicatorsinexperimentsdescribedinChapters4and5. 10ThismetricpresentedinJousselmeetal.,2006isrelevantbutwasnotdiscoveredbytheauthoruntilaftertheexperimentswerecomplete.Futureworkwillconsiderthismetricasapossiblesolution.63

PAGE 78

2.5.1BackgroundThissectionincludesabriefreviewofDempster-ShafertheoryandSmets'variantofthisthe-orycalledthetransferablebeliefmodelTBM.Dempster-Shafertheoryoperatesonsubsetsofaframeofdiscernment,,whichcontainsnmutuallyexclusivepossibleoutcomesforanexperi-ment.Abasicbeliefassignmentmisamappingfromsubsetsofto[0;1]whichdescribesthemeasureofbeliefthatiscommittedexactlyShafer,1976toeachsubset.Twobasicbeliefas-signmentsarecombinedtoproduceanewbasicbeliefassignmentusingDempster'sruleofcombi-nationaccordingtoEquation.19denedinShafer,1976.mC=PAiBj=C;C6=m1Aim2Bj 1)]TJ/F18 10.909 Tf 10.909 0 Td[(k.19wherek=XAiBj=m1Aim2Bj.20whereC22,A1;:::;An22arethefocalelementsofm1wherem1A>0,B1;:::;Bn22arethefocalelementsofm2,andistheemptyset.TheTBMisavariantofDempster-Shafertheorywheretheemptysetistreatedasavalidhypothesis.BasicbelieffunctionsarecombinedaccordingtoEquation.21denedinSmets,1990a.mC=XAiBj=Cm1Aim2Bj.21IntheTBMtheconictbeliefmassisalwaysmaintainedatthecredalleveli.e.belieflevelasevidenceisgatheredandcombined.WhenadecisionneedstobemadeitisnormalizedoutviathepignistictransformationdenotedBetP,showninEquation.22wherejAjisthenumberofele-mentsofinA.Thistransformationmapsbelieffunctionsmontopignisticprobabilityfunctions,whichcanbetreatedastraditionalprobabilityfunctionsSmets,1990b.Possibilitytheory,whichcanbeformulatedasaspecialcaseoftheTBMDuboisandPrade,1999,isthemathematicalframeworkonwhichfuzzysettheoryisbuilt.InSmets,2005theTBMwasextendedtohandlerealvaluedframesofdiscernment.Usingthisextension,AreguiandDenoeuxshowedthat64

PAGE 79

theTBMcantheoreticallybeusedforfaultornoveltydetectionwheredatafromtheplantsys-tem,machine,oragentbeingmonitoredcanbediscreteorcontinuous.BetPA=XBjABj jBjmB 1)]TJ/F18 10.909 Tf 10.909 0 Td[(m8A.222.5.2Conict-BasedMetricsThreemetricsfromtheliteraturemeasuretheweightofevidencesupportingcontradictoryhypotheses:Shafer'sweightofconictmetricShafer,1976orCon,Smets'conictmassas-signmentSmets,1990a,andthedegreeofinconsistencyBouchon-Meunieretal.,1999metricinpossibilitytheory.WhilemanystudiesintheliteraturehavetriedtonduniversallyapplicablemethodsforredistributingthemassassignedtoconictinDempster-ShafertheoryseeSmets,2007,onlyShaferandSmetshavestressedtheimportanceofmeasuringtheamountanddevel-opingsystemstointelligentlydeterminethesourceofthecontradictoryevidence.SimilarlymanystudieshavechiseledawayattheproblemofndingandeliminatinginconsistencyinfuzzyrulesetsseeforexampleCasillasandMartinez,2007;Ciliz,2005andpossibilisticknowledgebasesHunterandKonieczny,2005,butnostudieswerefoundthatusethedegreeofinconsistencymet-ricasatoolforinconsistencymanagement.InShafer,1976theweightofconictmetricwasproposedtoaccumulatetheweightofcon-tradictoryevidenceasbelieffunctionsarecombined.ThisisdenotedConBel1;:::BelnseeEquation.23andtakesvalues[0;1;ask!0:0,Con!0:0,andask!1:0,Con!1.Itisadditive,whichmeansthattheconictfrommorethantwobelieffunctionscanbeattainedeasily.Theapplicationofthismetricfordetectingsensor-relatedproblemswasrstproposedinMurphy,1998andexploredfurtherbytheauthorinCarlsonetal.,2005andCarlsonandMurphy,2005.ConBel1;Bel2=log1 1)]TJ/F18 10.909 Tf 10.909 0 Td[(k.23Smetsmakesthecasefortheformulationandhandlingofconictbysimplytreatingtheemptysetlikeanyothersubsetofandusingadomain-specicexpertsystemtorespondbyreducingoreliminatingtheconictinanintelligentfashion.InSmets,2007adetailedanal-ysisofconictmanagementinevidencetheoryandageneraldesignforadomain-specicexpert65

PAGE 80

systemtohandleconictareprovided.Smets'conictbeliefmassisusedtosolvetheassociationproblem11inAyounandSmets,2001andSchubert,1993andtodetectchangeinastaticallymodeledenvironmentinGambino,Ulivi,andVendittelli,1997.Althoughmostapplicationsofpossibilitytheoryandfuzzysetsrequireconsistenti.e.nor-malizedevidenceandrulesets,thedegreeofinconsistencymetricIncKcanbeusedtoenablereasoningwithinconsistentknowledgebasesDuboisandPrade,1999.12Wheninconsistencyex-istsinapossibilisticknowledgebase,thebeliefmassassignedtobecomeslessthanunityandtheknowledgebaseanditscorrespondingfuzzysetsaredescribedasunnormalized.ThedegreeofinconsistencyisdenedasshowninEquation.24whereKisaknowledgebaseandKisapossibilitydistributionovertheelementsofwhichdescribesthelikelihoodthatthetruevalueisthatelementBouchon-Meunieretal.,1999.Thismetricalsodescribesthelikelihoodforallinconsistentelementsinaknowledgebase,thusinDuboisandPrade,1999itisusedtoallowasystemtoreasonwithaninconsistentknowledgebasebyignoringthosestatementswithsupportvaluesatorbelowIncK.IncK=1)]TJ/F18 10.909 Tf 10.909 0 Td[(max!2K!.242.5.3DegreeofInconsistencyMetricsFourmetricsfoundintheliteraturemeasurethedegreeofinconsistencywithinJousselmeetal.,2006;Pal,1999;Yager,1982andbetweenAbellanandGomez,2006uncertaintymodels.Yager'smetricofinconsistencyforagivenfuzzysetiscalledanxietywhichistheantithesisofhiscommonlyusedspecificitymetric.Pal'sinconsistencymetricisdenedforDempster-ShaferbelieffunctionsandreducestoentropyintheBayesiancase.Jousselmeetal.deneanambiguitymeasuretocapturebothconictandnon-specicityforDempster-Shaferbelieffunctions.AbellanandGomezdevelopedaninconsistencymetricforcredalsetswhichareasupersetofprobabilistic,evidential,andpossibilisticfuzzysets.Yagerproposedameasureofanxietyregardingdecisionsmadeonfuzzysets.ThismetricAntDisgiveninEquation.25wheremaxisthelargestmembershipgradeinthede11Thegeneralproblemininformationfusionofcorrectlyassociatinginformationwithoneofseveralobjectsitcoulddescribe.Appriouetal.,200112EffortstowardreasoningwithinconsistentlogicalsymbolicdatabaseshaveproducedmetricsmeasuringtheinconsistencyofthosedatabasesseeHunterandKonieczny,2005thatareoutsidethescopeofthisliteraturereview.66

PAGE 81

cisionfunctionD,andjDjisthenumberofalternativeswhosesupportisatleastYagerandKikuchi,2004.ForclaritythetwochoiceformulationforthismetricisgiveninEquation.26wherenisthedegreeofsupportforagivenchoice.YagerandKikuchiapplythismetricformodelinganxietyaboutwhentostartworkingonatasktoensurethatagivendeadlineismet.WhileYager'sanxietymetrichasreceivedlittleattention,tranquility1)]TJ/F18 10.909 Tf 11.43 0 Td[(AntDfrequentlyreferredtoasspecicitywastherstofmanysimilarmetricsfrequentlyusedtomeasuretheuse-fulnessoftheinformationinafuzzysetGarmendia,2005.ExperimentsdiscussedinChapter4showthatthismetriccanbeusedtodetectsensinganomaliesandestimatesensingaccuracyinrealsonarandlaserdata.AntD=1)]TJ/F24 10.909 Tf 10.909 14.849 Td[(Zmax01 jDjd.25AntD=1)]TJ/F18 10.909 Tf 10.909 0 Td[(max1;2+1 2min1;2.26PalproposedatotalconictmetricforanormalizedDempster-Shaferbelieffunctionwhichmeasurestheuncertaintythatcomesfromconictorinconsistency.ThismetricisdesignedtodescribethedissimilaritybetweentwofocalelementsasthemetricdistancebetweenthemasshowninEquation.27.Thetotalconictforabelieffunctionmistheaverageovereachfo-calelementoftheaveragedistancebetweenthatelementandtheotherfocalelementsasshowninEquation.28.ThisfunctionreducestoavariantofquadraticentropyspecicallyVajda'squadraticentropyintheBayesiancasePal,1999.Realworldapplicationsofthismetrichavenotbeenfoundintheliterature.A;B=1)]TJ 12.104 7.38 Td[(jABj jA[Bj.27TC[;m]=XA2mAXB2mBA;B.28Jousselmeetal.presentanambiguitymeasureAMwhichattemptstocapturetwotypesofambiguitythatcoexistinDempster-Shafertheory:conictandnon-specicitytwoormorealternativesleftunspecied.TheresultingmeasureappliesSmets'BetPfunctiontoabe-67

PAGE 82

liefmassfunctionmforallsingleelementsa2thenusesaformulationsimilartoShannon'sentropytocombinetheresultsseeEquation.29.ItwaspresumedthatAMsatisedallcondi-tionsnecessarytobedeemedatrueambiguitymetric.MonteCarlosimulationswereperformedtoshowthatthismetricwaslesscomplexandmoresensitivetochangesinevidenceascomparedtoexistingmetricswhichdomeetallthecriteria.UnfortunatelyKlirandLewisfoundaprob-lemwhichnegatedtheclaimthatAMisatruemetric.Realworldapplicationsofthismetrichavenotbeenfoundintheliterature.AMm=X8a2BetPmalog2BetPma.29AbellanandGomezpresentmeasuresforcomparisonofcredalsets,orsetsofproba-bilitydistributionsforthevaluesofrandomvariables,whichareasupersetofprobabilistic,evi-dential,andpossibilisticfuzzysets.Aninconsistencymetricisdevelopedforcredalsetswhichdescribesthedistancebetweenincompatiblecredalsetswhichhavenoprobabilitydistributionsincommon.Importantmathematicalcharacteristicsofthismetricareproposedandaformulawhichsatisesthesecharacteristicsisgivenastheminimumdistancebetweenanypairofelementsfromthetwocredalsets.Thepropertiesofthisnewmetricwerevalidatedwitharticialnumericexam-ples.Itisunclearhowusefulthismetricwouldbeforsolvingrealworldproblems.2.5.4CombinedMetricsLiuapresentsaninconsistencymetricfordeterminingwhenitisappropriatetousetraditionalDempster-Shafertheorytocombineevidence.Throughalongseriesofexamplesthepaperdemonstratesthattwobelieffunctionsareonlyintuitivelyconictingwhenboththemassassignedtotheemptysetbeforenormalizationandthedifferencebetweenthetwobelieffunc-tionsisaboveagiventhreshold.ThelattermetricisdenedasthemaximumdifferencebetweentheSmets'pignisticprobabilityforeachhypothesisseeEquation.30.diffBetP=max8a2jBetPm1a)]TJ/F18 10.909 Tf 10.909 0 Td[(BetPm2aj.30Usingthesetwometrics,thepaperproposedaschemefordeterminingwhenandwhennottouseDempster-Shafer'scombinationoperator.Theirapproachfortrueconictdetectionconsistedof68

PAGE 83

applyinganapplication-dependentthresholdtothetwovalues.InLiu,2006bthismetricandthatofJousselme,Grenier,andBosseisappliedtofuzzysetsi.e.possibilitytheoryandinLiu,2007thisresultwasdevelopedintoanapproachforautomaticallyselectingtherightmergingoperatorbasedonthedegreeofinconsistencybetweenthemodelstobemerged.Realworldapplicationsofthismetrichavenotbeenfoundintheliterature.2.6ConclusionsThischapterhasestablishedthatthisistherstknownstudytodevelopagenericapproachaddressingtheproblemofsensinganomalies.ExistingsensororsourceassessmentapproachesseeSection2.1arefocusedonimprovingoverallaccuracyinsystemswherepeerscanbeun-cooperativeordeceptive.SensorFDIapproachesseeSection2.2aredesignedfortraditionalsensorfaultssuchasdrift,faultyhardware,ormis-calibration.TwoapproachesfromtheliteratureseeSection2.3isolatepoorlysensedregionswithoutrelyingonaprioriinformationbutthesearenotfocusedontheproblemofsensinganomalies,butinsteaddevelopspecializedsolutionstohighlightinconsistenciesin3DpointclouddataRomanandSingh,2006orndpoorlymodeledregionsina3Dmodelbuiltfrom2DlaserrangescansBaltzakis,Argyros,andTrahanias,2003.ApproachesforadaptivesensorfusiondiscussedinSection2.4aredesignedtoadaptsensingtoanunknownenvironment.OneofthesestudiesMorales,Takeuchi,andTsubouchi,2008devel-opedaspecializedsolutionfortheproblemofdetectingGPSsensinganomalies.ThatstudywasfocusedonrobustposeestimationinoutdoorenvironmentswhereGPSreadingscanbecomein-accurateunderdensetreecover.AstandardoutlierdetectionnormalizedinnovationsquaredtestwasusedtorejectGPSreadingsthatdivergedtoofarfromaKalmanlter-basedestimateofamo-bilerobot'sposition.Themostcommonlyused,andthereforemostmature,methodstoaddresssensingproblemsingeneralarenotsuitabletoaddresstheproblemofsensinganomaliesinunknownenvironments.Theseuseirrelevantcharacteristicse.g.responsivenessorqualityofserviceorrelyonquali-tative,stochastic,oranalyticalapriorimodelsofthetargetsystemundernormaland/orfaultystates.Nearlytwo-thirdsoutof27ofthesensororsourceassessmentstudiessurveyedinthischapterrelyoneitheruserassessmentsseeforexampleTRAVOS,Teacyetal.,2006,measuredcharacteristicswhichhavenothingtodowithaccuracyseeforexampleGaneriwalandSrivastava,69

PAGE 84

2004,oraprioriknowncharacteristicse.g.classicationaccuracywhichdonotprovideinfor-mationregardingtheaccuracyofsensinginagivenenvironment.Theseareunsuitedforassessingsensorsorsourcesaffectedbysensinganomalies.Exactlytwothirdsoutof42ofthesensorFDIapproachesincludedinthissurveyrelyonapriorimodelsofthetargetsystem.Sincesens-inganomaliesdonotexistuntilthesensorinteractswiththeenvironmentthesecannotbemodeledunlesstheenvironmentalconditionsareknown.Thischapterhasalsoestablishedthatthisistherstknownapproachtoestimatesensingaccu-racyinunknownenvironmentsusingdatafromasinglesensor.Consensus-basedapproacheslikeBarberandFullam,2003;Soika,1997acanbeusedtoestimatesourcereliabilityinunknownen-vironmentsbuttheserequireanaccurateconsensusofsensorsoragents.Sincetheenvironmentalconditionswhichleadtosensinganomaliesoftenaffectmultiplesensors,theseconsensus-basedapproacheswouldrequireanintelligentagenttoactivelyuseawidevarietyofsensorsatalltimestoensurethatsensinganomaliesareaccuratelycharacterized.Thisrequirementisunattainablefordomainswhereenergy,space,and/orweightcapacityareinlimitedsupplye.g.spaceroboticsorwirelesssensornetworks.Inadditionsensorsareexpensiveandsimultaneouslyusingmanysensorstaxesoftenlimitedcomputationalresources,increasesthecomplexityofanintelligentsystem,andincreasesthefrequencyofsensorfaults.Thisworkrepresentstherstknowngeneralapproachfordetectinganomaliesandisolatingpoorlysensedregionsinunknownenvironmentsusingdatafromasinglesensor.Inconsistency-basedsolutionscanperformthesetasksusingdatafromasinglesourcebutthosefoundinthelit-eratureprovidehighlyspecializedapproacheswithlimitedapplicability.ForexampletheapproachdescribedinAfgani,Sinanovic,andHaas,2008a,btodetectanomaliesinwirelesssignalsislim-itedtosignalsi.e.sourceswhichshowhighperiodicityundernormalconditions.Similarly,Ro-manandSinghdevelopaspecializeddistancemetrictoisolateinconsistenciesinconsec-utive3Dscanswhichislimitedtonon-dynamicdatamodeledaspointclouds.Theinconsistency-basedapproachdescribedinChapter3isgeneral.Ifprovidedwithasuitableinconsistencymetric,itcouldalsobeappliedforanomalydetectioninwirelesssignalsortoisolateinconsistenciesin3Dpointclouds.Thisworkistherstknownapproachtoapplyevidentialinconsistencymetricstomeasuresensorinconsistencyforasinglesensorasopposedtoexistingmethodswhichrequiremultiple70

PAGE 85

sensorstobeactive.Theseareusedinthisworktoquantifyinconsistenciesin2Dmodelsofcon-sistentenvironmentsfordetectionandcharacterizationofsensinganomalies.ThisrepresentstherstknownrealworldapplicationofthesemetricstoprovidefeedbackforsensinginunknownenvironmentswiththeexceptionofpriorworkbytheauthordescribedinCarlsonandMurphy,2005,2006;Carlsonetal.,2005althoughsimilarapproachesutilizingthesemetricshavebeenappliedtoresolveassociationissuesAyounandSmets,2001ortodetectoutdatedinformationinastaticmapGambino,Ulivi,andVendittelli,1997.71

PAGE 86

Chapter3ApproachThischapterpresentsanapproachtodevelopinconsistency-basedsensingaccuracyindica-torsfordetectingandcharacterizingsensinganomaliesinunknownenvironmentsrelyingsolelyonfusedsensorreadings.Thisworkismotivatedbysensinganomalies,caseswherethephysi-calsensorsareworkingwithinthemanufacturer'sspecicationsbutthereadingswouldleadtoanincorrectinterpretationoftheenvironment.Forexampleicepoorlyreectsinfraredi.e.heatandnear-infraredsignalsBaldridgeetal.,2009,makingitdifculttodetectwithnear-IRbasedlaserrangenders.Basedontheassumptionthatsensinganomalieswillmanifestasinconsistentreadings,thisapproachreliesonDempster-Shaferformulationsofevidenceandappliesevidentialinconsistencymetricsfromtheliteraturetofusedsensorreadings.Thresholdsareappliedtodistin-guishordinarynoisefromanomalies.Inconsistency-basedsensingaccuracyindicatorsareformedfrommetricsi.e.methodsassociatedwithaxedthresholdvalue.Indicatorsareusedtodetectsensinganomaliesandcharacterizesensingbyestimatingtheoverallsensingaccuracy,andisolat-ingpoorlysensedregions.Thisapproachisappliedtomodelsbuiltfromrealsensordatacollectedbymobilerobots,specicallyatraditional2Dmapoccupancygridofastaticenvironmentbuiltfromrangesensorreadings.ThisapproachlaysafoundationfortheuseofevidentialmetricstoimprovetheaccuracyofsensinginunknownenvironmentsasillustratedinSection3.1.Thissectionpresentsaframework,orsystemarchitecture,foraddressingthislargerproblembyspecifyingtheroleofsensinganom-alydetectionandcharacterizationinenablingamobilerobottoeffectivelymanageitssensingaccuracywhileidentifyingthesuitabilityofeachsensorinanygivenregionofanunknownen-vironment.Theframeworkcombinesthisapproachwiththefollowingexistingtechniques:sensormanagemente.g.A.GageandMurphy,2004,simultaneouslocalizationandmappingSLAM,seeThrun,2003b,sourcereliabilityestimationseeSection2.1.2,andsensorfaultdetectionandidenticationFDI,seeSection2.2.72

PAGE 87

Theremainderofthischapterisorganizedasfollows.Section3.2presentsthetheoreticalap-proachwhichassumesthetruestateoftheworldisconsistentandidentiessourcesofinconsis-tencyinasensor-basedworldmodelas:sensingerror,sensinganomalies,inaccurateapriorimod-els,ortheuseofaawedinternalrepresentation.Section3.3describeshowinconsistencymetricsfromtheliteratureareemployedtoestimatesensingaccuracy,detectsensinganomalies,andiso-latetheenvironmentalsourcesofsensinganomalies.Section3.4illustrateshowthisapproachisappliedtotheproblemofsensinganomaliesforrangesensorsbyapplyinginconsistencymetricsdirectlyto2Dmodelsbuiltfromrangesensorreadings.3.1IdentifyingSensorSuitabilityinUnknownEnvironmentsThissectionillustratestheroleofsensinganomalydetectionandcharacterizationandclariesitsrelationshipwithrelatedproblemsbydevelopinganovelframeworktosolvealargerproblem:unsupervisedidenticationofthesuitabilityofsensorsinunknownenvironments.Figure3showstheframeworkwhichcombinestheapproachdescribedherewithexistingtechniquesdevelopedforroboticsthesensormanagement,sensorFDI,andSLAMmodulesandmulti-agentsystemsthesensorreliabilitysub-module.Itelucidatesthecomplementaryrelationshipsbetweenthisap-proachandrelatedworkonsensingassessment:sourcereliabilityestimationseeSection2.1.2andsensorFDIseeSection2.2,whileidentifyingnewavenuesforfutureresearchthesensinganomalydiagnosisandsensingcontexttrackingmodules.Figure3includesallthemajormod-ulesandspeciestheowofdatabetweenthem.Thisframeworkenablesasensingmanagementsystemtomaximizesensingaccuracywhileexploringanunknownenvironmentbyprovidingthefollowinginformation:Thepresenceofsensinganomaliesorsensorfaults;Anup-to-dateestimateofeachsensor'sreliability;Thestatusofeachactivesensor,namelyworking,affectedbyanomalies,orfaulty;Thesensingcontext,denedastheenvironmentalcontextoftherobotintermsofsensorper-formance.73

PAGE 88

Figure3.Anovelframeworkofasystemforunsupervisedidenticationofthesuitabilityofsen-sorsinunknownenvironments.Modulesdevelopedinthisdissertationarehighlighted.74

PAGE 89

Table8.Thetasksassignedtoeachsub-moduleinsensingassessment.yindicatesmodulesdevel-opedinthisdissertation. Sub-module TaskDescription Sensingaccuracyestimationy Estimatetheaccuracyofsensingasawhole Sensorreliability Evaluatethetrustworthinessofeachsensorindividually Sensinganomalydetectiony Detectanysensinganomaliesinthecurrentsensingcontext Sensinganomalyisolationy Generateamapofregionswithintheenvironmentwheresensingispoor Sensinganomalydiagnosis Determinewhichsensorsareaffectedbysensinganomalies Theapproachpresentedinthischapterprovidestherstknowngeneralsolutionforthehigh-lightedsub-moduleswithinthesensingassessmentmoduleseeTable8:sensingaccuracyes-timation,sensinganomalydetection,andsensinganomalyisolation.Thegeneralapproachforeachsub-moduleisdiscussedindetailinSections3.2and3.3withapplication-specicdetailspro-videdinSection3.4.Thesensingaccuracyestimationmoduleusesaninconsistency-basedsens-ingaccuracyindicatortrainedinknownenvironmentstoestimatesensingaccuracyfortheactivesensors.Thisestimateisusedbythetrainedindicatorinthesensinganomalydetectionmoduletodeterminewhenasensinganomalyhasoccurredandtoalertthesensormanager.Thesensinganomalyisolationmodulealsousesatrainedindicatortoisolateregionswithintheenvironmentwherethesensorsareprovidinginaccurateinformationandreportsitsndingstothesensingcontexttrackingmodule.Thesensinganomalydiagnosismoduledetermineswhichsensorsareaffectedbydetectedsensinganomaliesandreportsitsndingstoboththesensormanagerandthesensingcontexttrackingmodules.Asolutionforthissub-moduleisanaturalextensionofthisdissertationbutisleftforfuturework.Thesensorreliabilitysub-moduleevaluateseachsensorin-dividuallybyutilizingasourcereliabilityestimationapproachseeSection2.1.2.Theresultingestimatesareprovidedtothesensormanager.Asolutionforthissub-moduleisnotproposedinthisdissertationsinceapproacheswhichdonotrelyonaprioriinformationhavealreadybeende-velopedseeforexampleBarberandFullam,2003;Soika,1997a.Thesensingassessmentmod-uleasawholeprovidesinformationaboutsensinganomaliesandestimatesofsensorreliabilityinthecurrentsensingcontexttothesensormanagerandthesensingcontexttrackingmodules.AtraditionalsensorFDImoduleisusedtodetectandidentifysensorfaultsforthesensormanager.SensorFDIsystemsseeSection2.2aredesignedtodetectsensorfaultse.g.physicaldamage,faultywiring,ormiscalibrationanddeterminewhichsensorsareaffected.Overthirty75

PAGE 90

yearsofworkdevelopingFDIsystemsseeSection2.2haveproducedverysophisticatedsolu-tionsforcommonlyusedsensors,butmostFDIapproachesrelyonapriorimodelsorlearning.Thismakesthemunsuitedtohandleproblemslikesensinganomalieswhichdependoninteractionwithanunknownenvironment.Thesensinganomalydiagnosismoduleplaysasimilarroleintheframeworkbydeterminingwhichsensorsareprovidinginconsistentreadings,butitcannotdeter-mineifthecausewasanenvironment-dependentanomalyorafault.Bothmodulesarerequiredtodeterminethetruestatusoftheactivesensors.TheSLAMThrun,2003bmoduleusesreadingsfromtheactivesensorstocreateanaccuratemapandanestimateoftherobot'sposition.InthisframeworktheSLAMmoduleservesasare-sourceforthesensingcontexttrackingmodule,enablingittomapthelocationsofsensingcontexttransitionsanddetermineiftherobothasre-enteredaparticularcontext.SLAMapproachesaredesignedtosolveachicken-and-eggproblem:mappinganenvironmentisstraightforwardwhentherobotknowswhereitislocalization,andlocalizationisgreatlysimpliedwithanaccuratemap.SLAMtechniquessolvebothproblemssimultaneously,enablingamobilerobottoaccuratelymapanaprioriunknownenvironmentseeThrun,2003b.ThisproblemhasreceivedagreatdealofattentionsincetheintroductionoftherstsolutioninThrun,Fox,andBurgard,1998.UnlikeprobabilisticsolutionstotheSLAMproblemThrun,2003b,theapproachdescribedhereusesevidentialmodels.AsdiscussedinSection3.2,thisapproachisbuiltontheassumptionthatthetruestateoftheworldisconsistent,sosensingproblemscanbedetectedandcharacterizedbymeasuringinconsistencyasopposedtouncertainty.Evidentialmodelsde-couplethebeliefincontradictoryhypothesese.g.beliefinAand:Aarenotrequiredtosumtooneenablingthedevelopmentofmetricstomeasureinconsistencyindependentlyfrominformativeness.ThesensingcontexttrackingmoduledeterminesthecurrentsensingcontextbydetectingtransitionsinsensorperformanceandcombiningthatinformationwiththemapprovidedbytheSLAMmodule.Readingsfromtheactivesensorsandfeedbackfromthesensingassessmentmod-uleareusedtodetecttransitions.Forexamplethismodulewouldreportatransitiontothesensormanagerwhentherobotmovesfromacleartoafoggyareadegradedaccuracyforopticalsen-sorsordrivesoutofaglass-walledhallwayimprovedlaserperformance.Sensingcontexttrack-ingappearstobeanewavenueforfutureresearchasithasnotbeenspecicallyaddressedintheliterature.HypotheticalsolutionscouldbeassimpleasapplyinglabelstoaSLAM-basedmapor76

PAGE 91

ascomplexasreproducinghippocampalfunctionstondacommonrepresentationforcontextual,behavioral,andsensoryinformationinamodelforspatialnavigationseeforexampleBarakovaandLourens,2004,2005.Thesensormanagerselectssensorstoactivateordeactivatetoensuretherobothasthesens-inginformationrequiredtocompleteitsmission.ForexamplethesensormanagerinthesensorfusioneffectsSFXarchitectureisaspecializedagenttaskedwithallocatingsensorstoactivebe-haviors1inamannerthatoptimizesthechancesoftaskcompletionMurphy,2000.Themostre-centlydevelopedversionofthissensormanagerA.GageandMurphy,2004utilizesameasureofsensorsuitabilitye.g.acombinationofsensorreliabilityandabehavior-speciedsuitabilitymet-rictooptimizesensingperformance.Byutilizingtheinformationprovidedbytheothermodulesthesensormanagercanselectthemostappropriatesetofsensorstousebytrackingthestatusandidentifyingthesuitabilityofeachsensorineachsensingcontextastherobotexploresanaprioriunknownenvironment.3.2TheoreticalApproachThissectiondevelopsthetheoryfordetectingandcharacterizingsensinganomaliesinun-knownenvironmentsbyassumingthatthetruestateoftheworldisconsistentandidentifyingsourcesofinconsistencyinmodelsbuiltfromsensorreadings.Thistheoryprovidessolutionsforthesensingaccuracyestimation,sensinganomalydetection,andsensinganomalyisolationsub-modulesinthenovelframeworkforidentifyingsensorsuitabilitygiveninSection3.1.Ifanappropriatemodelonethatcanrepresentthetrueworldstateisusedtointerpretthesensordata,sensinganomaliescanbedistinguishedfrominconsistenciesgeneratedbyordinaryerror.ThisistherstknowntheorytoexplicitlyaddresstoproblemofsensinganomaliesseeChapter2al-thoughsimilartheorieshavebeenappliedtotheproblemsofdataassociationAyounandSmets,2001andndingoutdatedinformationinstaticmapsofadynamicenvironmentGambino,Ulivi,andVendittelli,1997. 1Inroboticsabehaviorisamappingofsensorreadingstoapatternofeffectoractions.Murphy,200077

PAGE 92

3.2.1ConsistencyTheconsistencyassumptiontheassumptionthatthetruestateoftheworldisconsistentistakentobetrueinallenvironmentsandthereforecanbeutilizedevenifnothingmoreisknown.Forexampleapointinspacecannotbebothoccupiedandempty,norcanamobilerobotbeintwoplacesinthesameinstant.IncommonlanguagesomethingisconsistentifitisfreefromvariationorcontradictionMerriam-WebsterInc.,2008.Inlogicamodelisconsistentifandonlyifthereisnopredicatesuchthatbothand:canbederivedfromthatmodelEbbing-haus,Flum,andThomas,1994.Forthepurposesofthiswork,consistencyisinterpretedasfreeofinconsistencyasdenedbymetricsfromtheevidentialliteratureseeSection3.3.2.Forex-ample,ShaferandSmetsainterpretconsistencyasfreeofconict,,denedastheintersectionofmutuallyexclusivehypothesese.g.occupied:occupied=.3.2.2SourcesofInconsistencyGiventhisconsistencyassumption,ifinconsistencyappearsinasensor-basedworldmodelitmustbecausedbyoneormoreofthefollowing:Sensingerror.Errorisintroducedatmanypointsintheprocessofgatheringandinterpretingsensorreadings,includingnoise,thesensorhardwaresamplingerror,interpretationquanti-zationerrorinthesensormodel,andregistrationwithinalargermodellocalizationerroronamobilerobot.Sensinganomalies.Inappropriatesensorsmaybesporadicallyinaccurateleadingtoincon-sistentreadings.Forexample,lessthan8%oftheenergyfromanearinfraredNIRlaserisreectedbyglass2,thusanNIRnmrangesensorcannotreliablydetectglass.Inaccurateaprioridata.Accurateobservationswillbeinconsistentwithsuchdata.Theuseofaawedinternalrepresentation.Ifthemodelitselfisincapableofcorrectlycap-turingthetruestateoftheworld,thenevenperfectlyaccurateobservationscanappeartobeinconsistent. 2BasedonBaldridgeetal.,2009andinformalreectancemeasurementsthereferencematerialsrequiredforabsolutemeasurementswerenotavailableofglasssurfacesintheexperimentaltestbedstakenwithaspectrometer.78

PAGE 93

Thisformulationofconsistencyallowsinconsistencytobeusedtodetectavarietyofsensingandmodelingproblems,providedthatoneoftheabovecausesisassumedtogeneratesignicantlymoreinconsistencythantherest.Thisworkexaminesthecaseofamobilerobotexploringstaticunknownenvironments.Inthiscasetraditionalmodelscancapturethetruestateoftheworlde.g.astaticenvironmentcanbeaccuratelymodeledasanoccupancygridandaprioriinformationisnotavailable.Theremainingsourcesofconictarisefromsensingerrorandsensinganomalies,whichthisapproachdistinguishesbyapplyingathresholdtomeasuredinconsistencyinfusedsen-sorreadings.3.3Inconsistency-basedSensingAccuracyAssessmentTheapproachtakeninthisdissertationtocreatesensingaccuracyindicatorsfordetectingandcharacterizingsensinganomaliesinunknownenvironmentsistoapplyinconsistencymet-ricsfromtheevidentialliteraturetoDempster-Shafermodelsderivedfromsensorreadings.Theapproachestimatestheinaccuracyofsensing,withoutrelyingonaprioriinformation,bymea-suringthedegreeofinconsistencyinamodelofaconsistentworld.Indicatorsfordetectingandisolatingsensinganomaliesaredevelopedbyapplyingthresholdstothismeasurement.Atrainedsensingaccuracyindicatorcanbeusedinnewenvironmentsifitsabilitytoestimatesensingaccu-racy,detectsensinganomalies,orisolatetheenvironmentalsourcesofsensinganomaliesismain-tainedacrossdistinctenvironments.AminimalamountofcomputationaloverheadinrequiredseeSection3.3.3,makingthisapproachsuitableforonline,real-timeapplications.Notethatthisap-proachreliesontheassumptionthatsensinganomalieswillmanifestasinconsistentsensorread-ings.3.3.1DetectingandCharacterizingSensingAnomaliesTheobjectiveofdevelopingasensingaccuracyindicatorbasedoninconsistencymetricsistoenableamobilerobottodetectandmeasureimportantattributesofsensinganomalies.Inthisworkanestimateoftheinaccuracyofasensororsetofsensorsinthecurrentsensingcontexttherobot'simmediateenvironment,calledaconictscore,isderiveddirectlyfromthevaluesgivenbyaninconsistencymetric.Sensinganomaliesaredetectedbyapplyingathresholdtotheconictscore.Theconictscorecanalsobeusedtorankasetofsensorsaccordingtotheirrel-79

PAGE 94

ativeaccuracybyassessingeachsensorindividually,thensortingthemindescendingorderac-cordingtotheirscore.Forenvironmentmodels,regionswithintheenvironmentwheresensingisinaccurateareisolatedbyapplyingthresholdstoindividualbeliefmassese.g.cellsinaregu-largridwithinthelargermodel.Indicatorsaretrainedtoperformthesetasksusinggroundtruthinformationsuchastheactualsensingerrorandthepresenceofsensinganomaliesinknownen-vironmentstoselectthemethodi.e.metricandthresholdthatproducethemostaccurateresults.Indicatorscanbetrainedforoneormoresensorsbyapplyingthisprocesstofusedsensorread-ingsforthetargetsensors.Atrainedsensingaccuracyindicatorisonlyapplicableinunknownenvironmentsifitsabilitytocharacterizesensingismaintainedacrossdistinctenvironments.3.3.2InconsistencyMetricsInthisworksixmethodsforquantifyinginconsistencyinevidentialmodelsfromtheeviden-tialandroboticsliteratureareconsideredaspossiblesolutionstotheproblemofdevelopingsens-ingaccuracyindicatorstoestimatesensingaccuracy,detectsensinganomalies,andisolateenvi-ronmentalsourcesofsensinganomaliesinunknownenvironments.InaframeworkforidentifyingsensorsuitabilityseeSection3.1thesewouldbeusedtoimplementthesensingaccuracyesti-mationsub-module.Fiveofthesearemetricsdevelopedtosolvethegeneralproblemofquanti-tativelymeasuringtheconictorinconsistencyofuncertaininformation.Thesixth,GAMBINO,isactuallyaspecializedandcomparativelyadhocheuristic,buttheresultsofpriorworkCarlsonandMurphy,2005,2006showedthatithadpotentialasasolution.ANXIETY.Yagerdenedameasureofanxietyregardingdecisionsmadeonfuzzysets.ThetwochoiceformulationforthismetricisgiveninEquation3.1whereNisthenumberofchoicesandchoicenisthedegreeofsupportforeachchoice.AntDecision=1)]TJ/F18 10.909 Tf 10.909 0 Td[(Maxchoice1;choice2+1 NMinchoice1;choice2.1CON.ShaferdenedtheConmetrictomeasuretheweightofevidenceinsupportofcontradictions.ThismetricisgiveninEquation3.2whereA1;:::;An22theframeofdiscernmentarethefocalelementsofm1wherem1A>0,B1;:::;Bn22arethefocal80

PAGE 95

elementsofm2,andistheemptyset.EarlierworkfoundastrongcorrelationbetweenthismetricandoverallsonarsensorerrorCarlsonetal.,2005.ConBel1;Bel2=log1 1)]TJ/F18 10.909 Tf 10.909 0 Td[(k.2wherek=XAiBj=m1Aim2Bj.3INCONSISTENCY.PaldescribedatotalconictmetrictomeasuretheinconsistencywithinanormalizedDempster-ShaferbeliefmassseeEquations3.4and3.5,whichreducestoavariantofentropyintheBayesiancase.TC[;m]=XA2mAXB2mBA;B.4A;B=1)]TJ 12.104 7.38 Td[(jABj jA[Bj.5TBMCONFLICT.AyounandSmetsdevelopedamethodtosolvethedataassociationproblemusingtheconictbeliefmassmfromSmets'transferablebeliefmodelTBM,Smets,1990awithathresholddenedasEquation3.6wherecisthelevelofinternalconictexpectedfromaworkingsensorandnisthenumberofsensors.1)]TJ/F15 10.909 Tf 10.909 0 Td[()]TJ/F18 10.909 Tf 10.909 0 Td[(cn.6LIU'SCONFLICT.LiuadenedametricforDempster-ShafertheorytodeterminewhentwobeliefmassesareconictingbasedonbothTBMCONFLICTandthedifferencebe-tweenthepignisticprobabilitiesBetPasdenedinSmets,1990bfortwobeliefmasses:<;diffBetP>seeEquation3.7.diffBetPm1;m2=maxA2jBetPm1A)]TJ/F18 10.909 Tf 10.909 0 Td[(BetPm2Aj.781

PAGE 96

ifNEW_OCCUPIED>error//error=0.1andOLD_EMPTY>0.5andNEW_CONFLICT-OLD_CONFLICT>0.1assert``occupiedcellisnowempty''ifNEW_EMPTY>error//error=0.1andOLD_OCCUPIED>0.5andNEW_CONFLICT-OLD_CONFLICT>0.1assert``emptycellisnowoccupied'' Figure4.AchangeheuristicforstaticoccupancygridmapsfromGambino,Ulivi,andVendit-telli,1997.GAMBINO.Gambino,Ulivi,andVendittellidevelopedaheuristictodetectenviron-mentalchangesinastaticoccupancygridmapbasedpartlyonTBMCONFLICT.IfanewreadingprovidedevidencethatapreviouslyoccupiedcellwasnowemptyorviceversaseeFigure4fordetails,thenthecell'sbeliefwasresettoenableadaptationtoachangingenvi-ronment.InearlierworkCarlsonandMurphy,2005,2006thisheuristicshowedpotentialforestimatingsensingaccuracyanddetectingsensinganomalies.3.3.3ComplexityThisapproachprovidesalineartimesolutiononthenumberofbeliefmassesminthetar-getmodelwhichenablesanintelligentagenttoadaptinrealtimeasthesensingsituationchanges,providedthatthesetofvalidhypothesesusedintheevidentialsensingmodelissmalli.e.jjm.Toassessthesensingsituationthevalueofaninconsistencymetriciscalculatedforeachbeliefmasswithinthetargetmodel.Calculatingthevalueofaninconsistencymetricassum-ingthatthebeliefmasshasalreadybeencalculatedvariesfromconstantCONandTBMCONFLICTtolinearnANXIETY,LIU'SCONFLICT,andGAMBINOtoquadraticn2INCONSISTENCYonthesizeofthepowersetoftheframeofdiscernmentthesetofvalidhypotheses.TheoveralltimecomplexityofthisapproachisshowninEquation3.8fm;n2Omgn.8wheremisthenumberofbeliefmassesinthetargetmodel,n=j2j,andgnrepresentsthecomplexityoftheinconsistencymetricused.Toreducetheimpactofthegnfactor,thevalue82

PAGE 97

oftheinconsistencymetriccanbecalculatedandstoredonlywhenabeliefmassisupdated.Thisimprovementcanreducetherunningtimeifasmallportionofthetargetmodelisupdated,butalsorequiresmstoragespace.3.4TargetApplication:RangeSensorAnomaliesThissectiondescribeshowthisapproachwasappliedinthefeasibilitystudyandexperimentsseeChapters4and5todetectandcharacterizesensinganomaliesinrangesensorse.g.sonar,laserrangenders.TraditionaloccupancygridmapsarecreatedfromrawsensorreadingsusingtheapproachoutlinedinMurphy,2000andevidentialtheory.Inconsistency-basedsensingac-curacyindicatorsareapplieddirectlytothebeliefmassesmaintainedineachcelltoestimatetheinaccuracyofthereadings,detectsensinganomalies,andisolateenvironmentalsourcesofsens-inganomalies.3.4.12DMapsfromRangeReadingsThisworkfollowstheapproachdescribedinMurphy,2000forbuilding2Dmapsfromsen-sorreadingsusingthecombinationoperatorfromDempster-ShafertheoryorSmets'TBMde-pendingontheinconsistencymetricbeingevaluated.InthisstudyatraditionaloccupancygridrepresentationCohenandEdan,2008;Elfes,1989;Miura,Negishi,andShirai,2006wasused.Thisdividesatwodimensionalspaceintoasetofequallysizedcells,eachlabeledoccupiedoremptywithsomelevelofcertainty.3Sensormodelsareusedtoconvertrangereadings,d,intobeliefmasses,m,distributedoveraconeseeFigure5.TheconehasamaximumradiusofRthemaximumrangeofthesensorandahalf-angle.Theconeisdividedintotworegionsbasedontheactualrangereadingd:RE-GIONIlieswherethedistancerfromthesourcesensoriswithinerrorofd,andREGIONIIliesbetweenthesourceandREGIONI.ForthefeasibilitystudyseeChapter4withlaserandsonarreadingstheequationsforcalculatingthebeliefmassvariedlinearlywiththedistancefromthesensorasgiveninEquations.9.11forREGIONIwhereMaxoccupied=0:98,andforREGIONIIinEquations.12.14whereistheanglebetweenthereadinglocationandthe 3ProbabilisticThrun,2003a,possibilisticLopez-Sanchez,deMantaras,andSierra,1997,andevidentialmodelsMurphy,2000havebeenusedwithsuchgridsinmobilerobotmapping.83

PAGE 98

Figure5.GraphicaldepictionoftheconemodelMurphy,2000derivedfromthesonarparame-ters.Beliefattributedtoemptyisdepictedasnegative,occupiedaspositive.source.Additionalsensorsusedinthein-depthexperimentsseeChapter5hadsmallerarangeresolutionandmaximumrange.Thetraditionalmodelwasthereforemodiedtousealogarith-micdrop-offasthedistancefromthesensorincreased.ThiswasdonebyreplacingR)]TJ/F19 7.97 Tf 6.587 0 Td[(r RinEqua-tions.9and.13withthevalueof1:0uptoonemeterfromthesensor,thenR)]TJ/F19 7.97 Tf 6.586 0 Td[(lnr Rbeyond1:0meters.ThespecicparametersusedtobuildthesensormodelsforthefeasibilitystudyandexperimentsaregiveninSections4.2and5.2respectively.moccupied=)]TJ/F19 7.97 Tf 6.195 -4.541 Td[(R)]TJ/F19 7.97 Tf 6.586 0 Td[(r R+)]TJ/F19 7.97 Tf 6.587 0 Td[( 2Maxoccupied.9mempty=0.10m=1)]TJ/F18 10.909 Tf 10.909 0 Td[(moccupied.11moccupied=0.12mempty=)]TJ/F19 7.97 Tf 6.195 -4.541 Td[(R)]TJ/F19 7.97 Tf 6.586 0 Td[(r R+)]TJ/F19 7.97 Tf 6.586 0 Td[( 2.13m=1)]TJ/F18 10.909 Tf 10.909 0 Td[(mempty.1484

PAGE 99

3.4.2ImplementationofIndicatorsDevelopinginconsistency-basedsensingaccuracyindicatorsforfusedrangereadingsinanoccupancygridmaprequirescalculationofanoverallconictscoreandtheuseoftwothresholdsoneforindividualcellsandoneforentiremapstoclassifythedatainthecellandsensingsit-uationrespectivelyasnormalorsuspect.Thesemeasuresareusedtoevaluateallactivesensorssensorscurrentlyinuseinthecurrentsensingcontexttherobot'simmediateenvironment.Eachcellinanoccupancygridmaintainsitsownbeliefregardingthestateoccupiedoremptyofitsassignedregionwithintheenvironment,thusmethodsi.e.inconsistencymetricscanbeapplieddirectlytoeachindividualcell.Thecell-levelthresholdisappliedtothevalueoftheinconsistencymetrictodistinguishordinarynoisefromanomalousreadings.Thisthresholdiscriticalforisolat-ingregionsintheenvironmentthataredifcultfortheactivesensorstosense,i.e.classifyingeachcellasnormalorsuspect.Theconictscoreservesasanestimateoftheaccuracyofthesensorsandisafunctione.g.sumoftheinconsistencymetricvaluesforeachgridcellinthemap.Themap-levelthresholdisusedtodetermineifasensinganomalyispresent.InthefeasibilitystudyseeChapter4themap-levelthresholdwasappliedtotheconictscoreandwasdenedas75%ofthecell-levelthresholdmultipliedbythenumberofsuspectcells.IntheremainingexperimentsChapter5amoregeneralapproachwasused.Thesensingsituationwasdeterminedbycalcu-latingthepercentageofupdatedcellsthatwereclassiedassuspect.Amap-levelthresholdwasappliedtothispercentagetoclassifythesensingsituation.Sincetheframeofdiscernmentissmalltwohypothesestheoverallcomplexityofthissolutionislinearonthenumberofcellsinthegrid.3.5SummaryThischapterhaspresentedtherstknowngeneralapproachtoaddresstheproblemofdetect-ingandcharacterizingsensinganomaliesinunknownenvironments,relyingonlyonaccumulatedsensorreadings.Toputthisprobleminperspective,aframeworkwasintroducedforsolvingthelargerproblemofidentifyingthesuitabilityofsensorswhileexploringanunknownenvironment.Theframeworkcombinestheapproachdescribedherewithexistingsolutionsfromtheroboticsandmulti-agentsystemsliterature.Itclariestherelationshipbetweentheapproachproposedinthischapterandrelatedworkonsensingassessment:sourcereliabilityestimationseeSec-tion2.1.2andsensorFDIseeSection2.2,whilerevealingavenuesforfuturework.Thesein-85

PAGE 100

cludethedevelopmentofanapproachfordeterminingwhichsensorsareaffectedbysensinganomaliesdiagnosis,trackingtherobot'scurrentsensingcontextdenedastheenvironmentalcontextintermsofsensorperformance,andimplementationandtestingofthecompleteidentica-tionsystem.Theapproachassumesallenvironmentsareconsistentandappliesevidentialinconsistencymetricstofusedsensorreadingstodetectandcharacterizesensinganomalies.Theapproachre-liesonDempster-Shaferformulationsofevidencewhichwereselectedduetothehypothesisthatsensinganomalieswouldmanifestasinconsistencyinthesensorreadings.Theseformulationsde-couplethebeliefincontradictoryhypothesese.g.beliefinAand:Aarenotrequiredtosumtooneenablingthecreationofmetricstomeasurethemagnitudeofinconsistencyindependentlyfrominformativeness.Withintheevidentialliterature,sixinconsistencymetricswereidentiedaspossiblecandidatestodirectlyevaluatefusedsensordata:Yager'sANXIETY,Shafer'sCON,Smets'transferablebeliefmodelTBMCONFLICT,Pal'sINCONSISTENCYmetric,LIU'SCON-FLICTmetric,andGAMBINO'schangeheuristic.Eachmetricdependedonathresholdtodistin-guishminornoisefromanomalousreadings.Therefore,eachofthesixmetricsandanassociatedthresholdmethod,thresholdformedwhatthisthesisdenedasaninconsistency-basedsensingaccuracyindicator.Notethattheapproachisgeneralandcanbeappliedtodetectandcharacterizeanomaliesinawiderangeoffusedsensormodelsprovidedthatsuitableinconsistencymetricscanbeformulatedi.e.thesemustbeabletodistinguishinconsistencyfromalackofinformation.Aconcreteapplicationofthisapproachisdevelopedforrangesensore.g.sonarorlaserrangenderanomalieswherereadingsarefusedovertimetobuilda2Devidentialmapi.e.occu-pancygridmapofastaticenvironment.Rangesensorreadingsarecombinedusingacone-shapedmodelandevidentialtheorytobuilda2DmapusingtheapproachoutlinedinMurphy,2000.In-consistencymetricsareappliedtothebeliefmaintainedforeachrectangularregioncellinthe2Dmap.Thecalculatedvaluesareusedtoderiveanoverallconictscoreforthemap.Athresh-oldisappliedatthecellleveltolteroutordinarynoiseandtolabelindividualcellsasnormalorsuspect.Anotherthresholdisappliedatthemapleveltodetectsensinganomalies.Theapproachprovidesalineartimesolutiononthesizeoftheoccupancygridtotheproblemofdetectingandcharacterizingsensinganomalies,enablingrealtimeadaptationasthesensingsituationchanges.86

PAGE 101

Chapter4FeasibilityStudyThischapterdescribesafeasibilitystudytoexplorethemeritsoftheevidentialinconsistency-basedapproachintroducedinthisdissertationbytrainingsensingaccuracyindicatorstodetectandcharacterizesensinganomaliesrelyingsolelyonfusedsensorreadingsforuseinknownen-vironments.Sensinganomaliesarecasesinwhichthephysicalsensorsareworkingwithinthemanufacturer'sspecicationsbutthereadingswouldleadtoanincorrectinterpretationoftheenvironment.Forexample,sonar-basedrangesensorshavedifcultydetectingsmoothsurfaces,especiallythoseathighincidenceanglestothedetectorastheacousticsignaltendstoreectaway.Theapproachreliesonevidentialinconsistencymetricsfromtheuncertaintyliteraturetodirectlyevaluatefusedsensorreadingswiththresholdsappliedtodistinguishordinarynoisefromanom-alies.Inconsistency-basedsensingaccuracyindicatorsareformedbyassociatingametrici.e.methodwithaspecicthresholdvaluemethod,value.Thischapterprovidesstatisticallysigni-cantevidencethattheapproachcanbeusedtodetectandcharacterizesensinganomaliesinknownenvironmentsbyidentifyingindicatorstodetectsensinganomalieswithupto99.67%accuracyandestimatesensingaccuracywithcorrelationsofupto0.87inrealsonarandlaserreadingscol-lectedbyaNomad200inunclutteredindoorhallways.Thefeasibilitystudyaddressedthefollowingexperimentalhypothesis:inconsistency-basedsensingaccuracyindicatorscanbetrainedtoestimatesensingaccuracyanddetectsensinganom-aliesinknownenvironments.Totestthishypothesis,sensordatawascollectedonaNomad200robotwitharingof16PolaroidsonarsensorsandoneSICKPLSlaserrangesensorin45totalrunsinthreeunclutteredindoortestbeds.8to2.5meterswideand10meterslongwithknowngroundtruth.Section4.1describestheexperimentaltestbeds,indoorhallwaysknowntoinducesensinganomaliesduetotransparencyglassinthenear-IRrangeandspecularreectionsmoothsurfacesintheacousticrange.Section4.2describeshowrealsensorreadingsfromthesonarand87

PAGE 102

Table9.Characteristicsofthethreeindoorhallwaysusedfordatacollection.ThewidthWandlengthLarereportedinmeters.Smoothreferstopaintedsheet-rockwalls. Hallway Sonar Laser W L Walls narrow poor good 1.8 11.2 Smooth wide poor good 2.5 14.2 Smooth window poor poor 2.0 27 Smoothandlargewindows laserrangesensorswerecollectedinthetestbedenvironmentsandregisteredtoastatic2DmapusingthemethodoutlinedinMurphy,2000andevidentialtheoryforofineevaluation.900oc-cupancygridsbuiltfromeithersonarorlaserreadingswereevaluatedinthecourseoftheexperi-ments.Section4.3describestheexperimentalmethodusedtoaddressthishypothesisbyquantify-ingtheabilityof184inconsistency-basedsensingaccuracyindicatorstodetectsensinganomaliesandestimatesensingaccuracy.ThemethodusedanovelquantitativemapqualitymetricseeSec-tion4.3.2tomeasurethetrueaccuracyofthesensorsandempiricallydeterminedthresholdswereappliedtothisvaluetoclassifythesensingsituationasnormaloranomalous.Standardstatisticalanalyseswereusedtoevaluatethesignicanceoftheresults.Section4.4describestheresultsofthefeasibilitystudywhichindicatedthatthreeofthesixmethodsdidappeartobeuseful:ANXI-ETY,GAMBINO,andINCONSISTENCY.TheANXIETYandGAMBINOmethodscouldbetrainedi.e.byselectingtherightthresholdtodetectsensinganomaliesineithersonarorlaserreadingswith95%accuracy.TheINCONSISTENCYandANXIETYmethodscouldbetrainedtoestimatesensingaccuracyforthesesensors,withstatisticallysignicantcorrelationsabove0.85.Thestudyalsosuggestedthattherewasnospecicindicatorcapableofdetectingandcharacterizingsensinganomalies,thatthethresholdforamethodwouldhavetobeidentiedbytraining.InSection4.5thischapterconcludesthattheevidentialinconsistency-basedapproachpresentedinthisdisserta-tionshowspromisefordetectingandcharacterizingsensinganomalieswhichshouldbeexaminedmorethroughly,aswillbecoveredinChapter5.4.1TestbedsThreeunclutteredhallwayswereselectedfordatacollectionwhichprovidedastraightpathtotraversereducingrobotposeestimationproblemsandwereknowntoinducesensinganoma-liesforthelaserrangenderduetothepresenceoftransparentmaterialglassandthesonarduetoacousticspecularreectioninthepresenceofsmoothsurfacesglassandpaintedsheet-rock88

PAGE 103

aThenarrowhallway bThewidehallway cThewindowhallwayFigure6.Thethreeindoorhallwaysusedfordatacollection.walls.Table9describesthecharacteristicsofthetestbedspicturedinFigure6intermsofsonarandlasersensorperformance,dimensions,andthebuildingmaterialspresentinthetestbeds.Notethatsonarsensingaccuracyisassumedtobepoorinallthreetestbedsandthatneithersensorissuitableforthewindowtestbed.4.2DataCollectionandRepresentationRealsensorreadingsfromaringof16sonarsensorsandalaserrangendercollectedinun-clutteredindoorenvironmentswereregisteredtoastatic2Dmapforofinetrainingofindicatorsfordetectionandcharacterizationofsensinganomalies.Thissectiondescribesboththedatacol-lectionprocessusedtogatherthesensorreadingsandthemethodsusedtofusethosereadingsintoevidential2Dmapsoccupancygridmapsoftherobot'ssurroundings.Anofineanalysissystemcomparedthesemapstogroundtruthmapsgeneratedfrommanualmeasurementsofthetestbedenvironments.ANomadicTechnologiesNomad200robotseeFigure7wasusedtosimultaneouslycollectsonarandlaserreadingsintheexperimentaltestbeds.Therobotwasteleoperateddownthecenterofthehallwayforadistanceofsixmeterswhilereadingsfromtherobot'ssingleringof16Po-laroidsonarsensors,andaSICKPLSlasersensormountedjustabovethem,werecollectedforofineanalysis.Thebest45runs,intermsofconsistentsamplingratesandtheabsenceoferrorsinthedatacollectionprocess,wereusedinthisanalysis.89

PAGE 104

Figure7.TheNomad200robotusedfordatacollection.Table10.Theparametersusedinthesensormodels.distheactualrangereadingdistance. Sensor R Maxocc Rangeerror sonar 6.477m 12.5 0.98 1%ofd laser 27.5m 0.5 0.98 1%ofd Anofineanalysissystemusedsensormodelstoalternatelyregisterthesonarorthelaserreadingstoa28metersquareoccupancygridmapElfes,1989,withacellresolutionof10cmaboutfourinchesinboththexandydirections.Themapdividedatwodimensionalspaceintoasetofequallysizedcells,eachlabeledoccupiedoremptywithsomelevelofcertainty.ThisstudyusedtheapproachoutlinedinMurphy,2000andevidentialtheorytobuildanoccupancygridfromrangereadingsusingacone-shapedmodelseeSection3.4fordetails.ThespecicparametersusedtobuildthesonarandlasermodelsaregiveninTable10whereRisthesensor'srangeandistheangularresolution.Rangeerrorspeciestheexpectedrangeresolutionofthesensor.ThesonarparameterswerederivedfromspecicationssuppliedbyNo-madicsmanualsfortheNomad200'ssensingsystem.FollowingArbuckle,Howard,andMataric,thewidthofthelaserconewasnarrowedtoonedegreetoreecttheangularaccuracyoftheSICKPLSlaserrangender.ThemaximumrangeoftheSICKPLSsensorwassuppliedbythesensor'smanual.90

PAGE 105

4.3AnalysisMethodsandMetricsThissectiondescribesthe184sensingaccuracyindicatorsexaminedintheseexperimentsandthemethodsandmetricsusedtodeterminewhichifanyoftheseindicatorscanbeusedtode-tectandcharacterizesensinganomaliesrelyingsolelyonfusedsensorreadingsinknownenviron-ments.Section4.3.1describeshowtheindicatorswerederivedfromsixmethodsforquantifyinginconsistencyinevidentialmodels:veinconsistencymetricsfromtheuncertaintyliteratureandonechangeheuristicfromtheroboticsliteratureseeSection3.3.2.Section4.3.2describeshowaquantitativemapqualitymetricErrorwasusedtoquantifythetrueaccuracyofagivensensorbymeasuringthearithmeticdifferencebetweenoccupancygridsbuiltfromsensorreadingsandgroundtruthgrids.Section4.3.3describesthemethodsusedtoquantifythesensingaccuracyin-dicators'abilitytoestimatesensingaccuracyanddetectsensinganomaliesrelyingsolelyonfusedsensorreadingsusinglinearcorrelationanalysisandclassicationstatisticsrespectively.4.3.1SensingAccuracyIndicatorsAtotalof184sensingaccuracyindicatorsbasedonsixmethodsforquantifyinginconsis-tencyinevidentialmodelsseeSection3.3.2:veinconsistencymetricsfromtheuncertaintyliteratureandonechangeheuristicfromtheroboticsliteraturewereevaluatedfortheirabilitytodetectsensinganomaliesandestimatesensingaccuracyforsonarandlasersensorsinknownen-vironments.Inthisworkaninconsistency-basedsensingaccuracyindicatorisamethodcoupledwithathresholdmethod,threshold.ForthosemethodswithniteupperboundsANXIETY,IN-CONSISTENCY,TBMCONFLICT,andLIU'SCONFLICT,thethresholdvaluestobetestedwereevenlydistributedthroughoutthemethod'srange.Forthosemethodswithinnitetheoreticalup-perboundsCONandGAMBINOthemaximumvaluewasdeterminedexperimentallyintheinitialexploratorystudydescribedinCarlsonandMurphy,2005.Thethresholdvaluesforthesemeth-odswereevenlydistributedthroughouttheresultingexperimentalrange.AsdescribedinSection3.4,implementationofthesemethodsforuseasasensingaccuracyindicatoronanoccupancygridmaprequirescalculationofaconictscoreandtheuseoftwothresholdstoclassifythecellandsensingsituationrespectivelyasnormalorsuspect.Forthissetofexperiments,asummationofthevaluesforallsuspectcellscellswhosevaluesexceededthe91

PAGE 106

Table11.Theparametersusedtocreatesensingaccuracyindicatorsfrommethodsforquantifyinginconsistencyinevidentialmodels. Method ThresholdsT jTj Cellvalue ANXIETYYager,1982 f0.1,0.14,...,0.9g 21 Metricvalue INCONSISTENCYPal,1999 f0.05,0.07,...,0.45g 21 Metricvalue CONShafer,1976 f0.25,0.5,...,5.0g 20 Metricvalue TBMCONFLICTSmets,1990a f0.1,0.14,...,0.9g 21 Metricvalue LIU'SCONFLICTLiu,2006a 81 minm;diffBetP m f0.1,0.2,...,0.9g diffBetP f0.1,0.2,...,0.9g GAMBINOGambino,Ulivi,andVendittelli,1997 f0.5,1.0,...,10.0g 20 Numberofchangesdetected Total 184 cell-levelthresholdwasusedastheconictscore.Themap-levelthresholdwassetto75%1ofthecell-levelthresholdmultipliedbythenumberofsuspectcells.Table11describesthemethodforcalculatingthevalueforeachcell,therangeofthresholds,andthenumberofthresholdvaluestestedintheseexperiments.NotethatLiu'sconictmethodusestwopairedvalues.Foracelltobedeemedsuspectbothvalueshadtoexceedtheirrespectivethresholds.4.3.2ObjectiveSensingAccuracyAssessmentTheexperimentsdescribedinthischapterusedaquantitativemapqualitymetric,Error,asameasureoftheactualsensingaccuracyandappliedxedthresholdstothatvaluetodeterminethetruestatusofthesensorsnormalversusaffectedbyanomalies.Thismetricmeasuredthediffer-encebetweentheoccupancygridgeneratedfromthesensorreadingsandmanuallycreatedgroundtruthmapsoneforeachtestbed,asdenedinFigure8,wheregrid ox;yandgrid ex;ygivetheoccupancyandemptyvaluesfromthesensor'soccupancygrid,andtruth ox;yandtruth ex;yarethesamevaluesfromthegroundtruthgrid,respectively.Athresholdof0.5wasusedtolterouterrorsofinsufcientsizetoaffecttherobot'sbehavior.LowerErrorscoresin-dicateabettermatchwiththegroundtruth.EmpiricallydeterminedthresholdsofError210forDempster-ShaferandError410forSmets'TBMgridswereusedtoautomaticallyclassify 1ThisratiowasselectedbasedonpriorworkCarlsonandMurphy,2006withGAMBINOwhichproducedaccuratedetectionresultsusingacell-levelthresholdof2.0,andamap-levelthresholdof1.5appliedtotheaverageoverallsuspectcells.92

PAGE 107

Error=0foreachcellx,yiftruthx,y!=UNKNOWNandgridx,y!=UNKNOWNif|grid_ox,y-truth_ox,y|>0.5occupied_error=|grid_ox,y-truth_ox,y|elseoccupied_error=0.0if|grid_ex,y-truth_ex,y|>0.5empty_error=|grid_ex,y-truth_ex,y|elseempty_error=0.0IncreaseErrorbyMAXoccupied_error,empty_error Figure8.ProcedureusedtocalculatetheErrormetric.thestatusofthesensorsforofineanalysis.Thesevalueswereselectedbasedonapost-hocex-aminationoftheErrorscoreswherethetruesensingstatusnormaloranomalouswasknownapriori.4.3.3QuantifyingIndicatorPerformanceToaddressthehypothesisthatinconsistency-basedsensingaccuracyindicatorscanbetrainedtoestimatesensingaccuracyanddetectsensinganomaliesanofineanalysissystemexaminedoccupancygridmapsbuiltfromeithersonarorlaserreadingsandusedlinearcorrelationanalysisandclassicationstatisticstocomparethegroundtruthevaluationwiththatofeachindicator.FortheestimationcomponentPearson'slinearcorrelationanalysiswasusedtodetermineifthecon-ictscorefromasensingaccuracyindicatorestimatedthegroundtruthErrorscore.Bothscoreswerecalculatedeveryhalfmeterofrobottravel.ForthedetectioncomponentthesensingaccuracyindicatorwasconsultedandathresholdwasappliedtotheErrorscoretodeterminethestatusoftheactivesensoreveryhalfmeterofrobottravelandcommonclassicationstatisticsweregener-atedtodeterminethedetectionaccuracyoftheindicator.TheresultsofthisanalysisareprovidedinSection4.4.Intheofineanalysis,sonarandlaserreadingswereappliedtooccupancygridswhichwerecheckedforsensinganomaliesandevaluatedforsensingaccuracyeveryhalfmeterusinganindi-cator'sconictscoreandtheErrorscore.Foreachrunreadingswereappliedtotheoccupancy93

PAGE 108

griduntiltherobothadtraveledonemeter,atwhichpointthegridwasevaluated.Thegridwasreevaluatedeverysubsequenthalfmeterproducing10samplesforthesonarandlaser,foratotalof20samplesperrun.Thisprocesswasrepeatedforeachofthe45runsinthethreetestbeds,re-sultinginanErrorscore,anindicator'sconictscore,andpairedclassicationsforeachof900samplesin90timeseriesrecordedforuseinpost-hocanalysis.Linearcorrelationanalysisandclassicationstatisticswereusedtoquantifyeachindicator'sabilitytoestimateErrorandcorrectlydetectsensinganomaliesrespectively.Theabilitytoes-timatetheoverallmapquality,i.e.theextenttowhicharobot'scurrentsensingisinappropriate,wasmeasuredusingPearson'scorrelationcoefcientrDowdy,Weardon,andChilko,2004andthecorrespondingprobability.2Inthiscasethetestwasusedtodetermineiftheconictscore,as-signedbytheindicatortobeevaluated,variedlinearlywiththegrid'sErrorscore.Theabilitytocorrectlydetectsensinganomalieswasmeasuredusingcommonclassicationstatisticsincludingthepercentageofcorrectlyclassiedexamples,thefalsepositiverate,andthefalsenegativerate.Thefalsepositiveratewascalculatedbydividingthenumberoffalsepositiveexamplesbytheto-talnumberofnegativeexamples.Thefalsenegativeratewascalculatedbydividingthenumberoffalsenegativeexamplesbythetotalnumberofpositiveexamples.4.4ResultsTheexperimentalresultspresentedinthissectionsupportthehypothesisthatinconsistency-basedsensingaccuracyindicatorscanbetrainedtodetectsensinganomaliesfourindicatorsob-tainedbetterthan95%detectionaccuracyandestimatesensingaccuracyindicatorsachievedstatisticallysignicantcorrelationcoefcientsof0.8orbetterrelyingsolelyonfusedsonarorlaserreadingsinknownenvironments.TheANXIETYmethodperformedbestoverall,achiev-ing99.67%detectionaccuracyandgoodestimation,i.e.linearcorrelationstrainedcorrelationof0.8538withtheErrorscore.TheGAMBINOmethodalsoshowedpotentialfordetectionbyachieving96.33%accuracy.TheINCONSISTENCYmethodperformedbestintermsofestimatingsensingaccuracy.8735correlation.NotethattheresultspresentedhereareasynopsisofthecompletesetofresultsgiveninAppendixA. 2Sincethedataconsistoftimeseries,thenumberofsamplesusedtocalculatetheprobabilityfromrwascorrectedforautocorrelation.94

PAGE 109

Theresultsforestimatingsensingaccuracyrelyingsolelyonfusedsensorreadingsarepre-sentedinFigure9.TheresultsaregivenintermsofPearson'scorrelationanalysiswhichcom-paringeachindicator'sconictscorewiththequantitativemapqualitymetric,orErrorscore.Figure9ausesaboxandwhiskerstylegraphtoshowtheminimumbottomofline,maximumtopofline,mediantriangle,25thpercentilebottomofbox,and75thpercentiletopofboxcorrelationcoefcientrforeachmethod.Figure9bgivesthetrainedcorrelationcoefcientrforeachmethodandthethresholdvalueusedtoachievethisresult.Atwo-tailedt-testatap-valueof0:01i.e.lessthana1%chanceofnorelationshipbetweentheconictscoreandtheErrorscorewasusedforstatisticalsignicance,andpassingcoefcients3aremarkedwithy.Theesti-mationcomponentisdifferentfromdetectionbecauseitestimatestheoverallerrorinsensorread-ingswhichmaybecausedbysensornoise,discretizationintheoccupancygridmap,etc.withoutdistinguishingthesefromsensinganomalies.Figure9showstheINCONSISTENCYandANXIETYmethodsestimatedsensingaccuracywell.Eachachievedtrainedcorrelationcoefcientsabove0:85.MostindicatorsbasedonANXIETYtendedtoperformbetterthanthosebasedonINCONSISTENCY.Detailedexaminationofthere-sultsrevealsaslightdropinrwitheachincreaseinthethresholdusedwithANXIETY,suggestingthatthismethodwouldbeeasiertotrain.ForINCONSISTENCY,correlationsremainedhighabove0.83untilthethresholdreached0:25,atwhichpointthecorrelationdroppedto0.22andremainedthere.ThedetectionaccuracyresultsforeachindicatorarepresentedinFigure10whichshowthattheANXIETYandGAMBINOmethodsdemonstratedveryaccuratedetectionofsensinganomalieswithatraineddetectionaccuracypercentageofcorrectclassicationsoutof900examplesex-ceeding96%.Figure10bgivesthetraineddetectionaccuracyforeachmethodandthethresholdvalueusedtoachievethisresult.Figure11showsthatthechangeindetectionaccuracyforANXIETYislargelymonotonic,fromalargenumberoffalsepositivestoasmallnumberoffalsenegatives,asthethresholdvalueisincreasedwhichsuggeststhatthismethodiseasytotrain.Formostoftheselectedthresholdvalues.1.54,theanxietyindicatorsweretoosensitiveandlabeledanyamountofinconsis-tencyasindicativeofasensinganomaly.Asthethresholdincreased.58.78,theindicators 3NotethatbothautocorrelationcorrectionandBonferronicorrectionfortheuseofmultiplethresholdvalueswereappliedintheprocessofdeterminingstatisticalsignicance.95

PAGE 110

aVarianceovertestedthresholdvalues. Method Thresholds r INCONSISTENCY 0.05 0.8735y ANXIETY 0.1 0.8538y GAMBINO 2.0 0.7061y TBMCONFLICT 0.9 0.6379y LIU'SCONFLICT 0.9/0.1 0.6274y CON 0.25 0.3598 bTrainedperformance.Figure9.Estimationresultsforeachinconsistencymethod.Theboxandwhiskercharttopshowsthevarianceincorrelationroverthetestedthresholdvalues.Eachmethod'strainedper-formanceandtheassociatedthresholdsareshowninthetablebottom,whereyisusedtomarkstatisticallysignicantresultsatap-valueof0.01.begancorrectlyclassifyingthenegativecases.Asthethresholdincreasedfurtherasmallnum-berlessthan1%ofexamplesoffalsenegativesbegantoappear.IncomparisontheGAMBINOmethodonlyshowsgooddetectionperformanceatathresholdof2.5or3.0.Thresholdsbelow2.5labeledanyamountofinconsistencyasindicativeofasensinganomalyandthoseabove3.0weretoospecicresultinginover25%oftheexamplesmisclassiedasnegative.Figure12givesthethefalsenegativerateFNR,i.e.missrate,resultsforeachindicatorwhichshowsthatonlyindicatorsbasedonGAMBINOandLIU'SCONFLICTtendedtobetoospe-cic,i.e.generatedalargenumberoffalsenegatives.Figure12bgivesthetrainedFNRforeachmethodandthethresholdvalueusedtoachievethisresult,orallforthosemethodswhoseFNR96

PAGE 111

aVarianceovertestedthresholdvalues. Method Thresholds %correct ANXIETY 0.86 99.67% GAMBINO 3.0 96.33% LIU'SCONFLICT 0.4/0.1 70.11% INCONSISTENCY 0.45 69.56% CON All 68.89% TBMCONFLICT All 68.56% bTrainedperformance.Figure10.Detectionresultsforeachinconsistencymethod.Theboxandwhiskercharttopshowsthevarianceinaccuracyoverthetestedthresholdvalues.Eachmethod'strainedaccuracyandtheassociatedthresholdsareshowninthetablebottom.97

PAGE 112

Figure11.Changeincorrelationcoefcientr,numberoffalsepositiveandfalsenegativeexam-plesforANXIETYasthethresholdvaries.didnotvary.Withasufcientlylowthresholdallinconsistencymethodswilllabelanyamountofinconsistencyassuspect,asaresulttheywillnevermissasensinganomaly.Duetodiscretiza-tionwithinthesensoranalogtodigitalandagainasthereadingisconvertedintobeliefmassesoveraregulargridbythesensingmodel,evenaperfectlyaccuratesensorwillgeneratesomein-consistentbeliefmasses.Thereforeanyinconsistencymethodcanproducea0.0%FNR,butonlyatruesensingaccuracyindicatorcanachievebothalowFNRandalowfalsepositiverate.ThefalsepositiverateFPR,i.e.falsealarmrate,resultsforeachindicatorarepresentedinFigure13andshowthattheCON,INCONSISTENCY,andTBMCONFLICTmethodsweretoosen-sitive,i.e.theywouldlabelanyamountofinconsistencyassuspect.Figure13bgivesthetrainedFPRforeachmethodandthethresholdvalueusedtoachievethisresult,orallforthosemeth-odswhoseFPRdidnotvary.CloseexaminationalsorevealsthatLIU'SCONFLICTdidnotclas-sifywell,i.e.thosethresholdsthatproduceda0.0%ornear0.0%falsepositiverateweretoospecic,generatingamissrateabove94%.OnlyindicatorsbasedonGAMBINOshowedgoodfalsealarmrates,butinmostofthesecasesallbuttwothresholdvaluesitwasalsotoospe-cic.IndicatorsbasedonANXIETYshowedthesmoothesttransitionfromoverlysensitivetohighdetectionaccuracy,withperformanceagainmonotonicallyincreasingasthethresholdincreased.98

PAGE 113

aVarianceovertestedthresholdvalues. Method MaxThresholds Falsenegativerate ANXIETY 0.86 0.48% CON All 0.0% GAMBINO 3.0 5.19% INCONSISTENCY 0.45 0.0% LIU'SCONFLICT 0.4/0.1 0.0% TBMCONFLICT All 0.0% bTrainedperformance.Figure12.Falsenegativerateforeachinconsistencymethod.Theboxandwhiskercharttopshowsthevarianceinperformanceoverthetestedthresholdvalues.Eachmethod'strainedperfor-manceandtheassociatedthresholdareshowninthetablebottom.99

PAGE 114

aVarianceovertestedthresholdvalues. Method MinThresholds Falsepositiverate ANXIETY 0.86 0.0% GAMBINO 3.0 0.35% LIU'SCONFLICT 0.4/0.1 95.05% INCONSISTENCY 0.45 97.86% CON All 100.0% TBMCONFLICT All 100.0% bTrainedperformance.Figure13.Falsepositiverateforeachinconsistencymethod.Theboxandwhiskercharttopshowsthevarianceinperformanceoverthetestedthresholdvalues.Eachmethod'strainedperfor-manceandtheassociatedthresholdsareshowninthetablebottom.4.5ConclusionsThischapterdescribedafeasibilitystudytodetermineiftheevidentialinconsistency-basedapproachfordetectingandcharacterizingsensinganomaliescouldbeusedatall.Tothisend,sensordatawascollectedonaNomad200robotwitharingof16PolaroidsonarsensorsandoneSICKPLSlaserrangesensorin45totalrunsinthreeunclutteredindoorhallways.Thesetestbedswereselectedbasedonthefollowingprinciples:testbedsshouldrepresentenvironmentalcondi-tionsknowntoinducesensinganomaliesandprovideastraightpathfortherobottotraversetopreventposeestimationerrorsfromtherobot'sencoders.Readingsfromeitherthesonarorlasersensorswereregisteredtoastatic2DmapoccupancygridusingthemethodoutlinedinMur-100

PAGE 115

phy,2000andevidentialtheoryforofineevaluation.Thetrueaccuracyofthesensorsbasedonacomparisonwithmanuallybuiltgroundtruthoccupancygridsandtheindicators'evaluationofthesensingsituationweredeterminedeveryhalfmeterofrobottravel,resultingin900occupancygridsbuiltfromeithersonarorlaserreadingsevaluatedintheexperiments.184inconsistency-basedsensingaccuracyindicators,denedasaninconsistencyquanticationmethodforevidentialmodelscoupledwithanassociatedthresholdmethod,threshold,wereexaminedfortheirabilitytodetectsensinganomaliesbythepercentageofcorrectlyclassiedoccupancygridsandes-timatesensingaccuracyusingPearson'scorrelationanalysistocompareanindicator'sconictscorewiththetruesensingerror.Theresultsofthefeasibilitystudypresentedinthischaptershowthatinconsistency-basedsensingaccuracyindicatorscanbetrainedtoestimatesensingaccuracyanddetectsensinganom-aliesrelyingsolelyonfusedsensorreadings.Yager'sANXIETYandPal'sINCONSISTENCYmeth-odsachievedtrainedcorrelationsof0.8orbetterwithtrueerrorinsonarorlaserreadings.ANX-IETYandGAMBINO'sheuristicachievedbetterthan95%detectionaccuracyoutof900sampledoccupancygridsbuiltfromeithersonarorlaserreadings.TheANXIETYmethodperformedbestoverall,achieving99.67%detectionaccuracyandgoodestimation,i.e.linearcorrelationstrainedcorrelationof0.8538withtheErrorscore.Inadditionitsperformancevariedmonotonicallywiththethresholdvalueindicatingthatthismethodwouldbeeasytotrain.Theresultsalsoshowthatnosingleindicatorcanbeusedtodetectandcharacterizesensinganomalies.Estimatingsensingaccuracyanddetectingsensinganomaliesi.e.distinguishingthesefromordinarynoiserequiretheuseofdistinctthresholdvaluesatthecellleveltolteroutordi-narynoise.Thresholdvaluesforestimationtendedtobelowerascomparedtodetection,indicat-ingthatthelatterrequiresahigherlevelofltering.OnlytheANXIETYmethodperformedwellforbothdetectionandestimationforfusedsonarorlaserreadings.Notethatallindicatorstrainedintheseexperimentswereusablewiththesonarorlasersensorsinterchangeablysuggestingthat,althoughdistinctindicatorsmaybeneededfordetectionandestimation,asingleindicatorforeachofthesecomponentscouldprovidefeedbackforseveralsensorsinasuite.101

PAGE 116

Chapter5FurtherExperimentsTheexperimentsdescribedinthischapterexpandonthefeasibilitystudypresentedinChap-ter4.Anewrobot,anATRV-JrmobilerobotequippedwithaSICKLMSandCanestarangecam-era,wasemployedtobroadenthesetofexaminedsensorsandtestbedenvironmentsoneaddi-tionalindoorandtwooutdoor,providinganopportunitytoexploregeneralizability.Thedatacol-lectionprocedurewasenhancedtoprovidemorerealisticscenariosfordetection.Heretheoccur-renceofsensinganomalieswasvariedwithineachrun,asopposedtothefeasibilitystudywherethesameanomalieswerepresentthroughoutagivenexperimentaltestbed.Inaddition,theseex-perimentsaddressimportantaspectsofdetectingandcharacterizingsensinganomaliesinun-knownenvironmentsthatwerenotexploredinthefeasibilitystudy,namelyisolationofpoorlysensedregionsandtestingtheapplicabilityoftrainedindicatorsinnewenvironments.Theexperimentswereconductedinthreephases.InthetrainingphaseindicatorstodetectandcharacterizesensinganomalieswereidentiedforrangesensorsinstalledontheATRV-Jrtodetectsensinganomalies,estimatesensingaccuracy,andisolatepoorlysensedregions.Thevericationphaseevaluatedtheperformanceoftheidentiedindicatorsinthreetestbedsnotincludedinthetrainingphase:twounclutteredindoortestbeds.0and2.5meterswideand10meterslongandoneunclutteredoutdoortestbed.2meterswideand10meterslong.Theresultsrevealedthattheindicatorstrainedfordetectionwereunreliablewithaccuracyrangingfrom28.71%to92.23%acrosstestbeds,butthattheestimationandisolationindicatorscouldbeusedinthenewenviron-mentswithlittledegradationinperformance.Theutilityphaseexploredapplicationsoftheseindi-catorstoimprovesensingaccuracyorranksensorsaccordingtotheirrelativeaccuracy.Theresultsshowedthatsimplestrategieswhichemployedtrainedindicatorstodetectsensinganomaliesandisolatepoorlysensedregionsproducedstatisticallysignicantimprovementsinsensingaccuracy.Thetrainedindicatorfortheestimationcomponentdidnotrankwell.Itsabilitytoselectthemostaccuratesensorwasnotreliablealthoughitcoulddeterminetheleastaccuratesensor.102

PAGE 117

Theremainderofthischapterdescribestheexperimentsconductedusing2Dstaticmapsoc-cupancygridmapsbuiltfromrealsensorreadingscollectedbymobilerobotsinunclutteredstaticenvironmentsandisorganizedasfollows.Section5.1describestheexperimentaltestbeds.Fourunclutteredindoorhallwaysandtwooutdoorsidewalkswereselected.Thesewereknowntoin-duceanomaliesforthesensorsconsideredinthisworkduetotransparencyglassandabsorbancedarksurfacesinthenear-IRrangeorspecularreectionsmoothsurfacesintheacousticrangeundervaryingenvironmentalconditionse.g.temperatureandambientlighting.Section5.2de-scribeshowrealsensorreadingsfromaringof16sonarsensors,twoSICKlasersensors,andaCanestarangecamerawerecollectedinthetestbedenvironmentsandregisteredtoastatic2DmapusingthemethodoutlinedinMurphy,2000andevidentialtheoryforofineevaluation.Sec-tion5.3presentstheresultsofthetrainingphasewheremethodsforquantifyinginconsistencyinevidentialmodelsweretrainedtodetectandcharacterizesensinganomaliesforaSICKLMSlaserandaCanestarangecamera.Thisrequiredtheselectionofindicatorsfromthesetof184inconsistency-basedsensingaccuracyindicatorsseeSection5.3.1examinedinthisworkwhichperformedbest.ThemethodsusedanovelquantitativemapqualitymetricseeSection5.3.2tomeasurethetrueaccuracyofthesensorsandstandardstatisticalanalysesseeSection5.3.3toevaluatethesignicanceoftheresults.Section5.4presentstheresultsofthevericationphasewheretheselectedindicatorswereevaluatedfortheirperformanceinnewenvironments.Sec-tion5.5presentsresultsfromtheutilityphasewhichexploresapplicationsoftheseindicatorstodeterminetheirusefulness.Section5.6concludesthattheapproachpresentedinthisdissertationcannotdistinguishordinarynoisefromsensinganomalies,butdoesprovidetherstknowngen-eralapproachforsensingassessmentrelyingsolelyonfusedsensorreadingsfromasinglesensor.5.1TestbedsToensurethattheexperimentsprovidedanaccurateassessmentoftheapproach'sabilitytodetectandcharacterizesensinganomaliesinunknownenvironments,sixunclutteredindoorandoutdoortestbedswereselectedfordatacollection.TheselectionoftestbedsforeachphaseseeTable12followedtheseprinciples:eachsetshouldrepresentenvironmentalconditionsknowntoproduceanomaliesforthetargetsensors,thetestbedsshouldprovidelongstraightpathsfortherobottotraversetomaximizethelengthofthedatacollectionrunswhileminimizingposeesti-103

PAGE 118

Table12.Thetestbedsandrobotsusedfordatacollectionineachanalysisphaseoftheexperi-ments. Phase Training Verication Utility Switching Resetting Ranking Sensors SuspectData IndoorTestbeds Narrow Nomad Cubicle ATRV-Jr Nomad ATRV-Jr ATRV-Jr Bridge ATRV-Jr Nomad ATRV-Jr ATRV-Jr Lab ATRV-Jr ATRV-Jr ATRV-Jr OutdoorTestbeds Walkway ATRV-Jr ATRV-Jr ATRV-Jr Sidewalk ATRV-Jr ATRV-Jr ATRV-Jr TotalRuns 16 30 45 46 46 mationerror,andthesetoftestbedsshouldexposethesensorstovaryingenvironmentalcondi-tionswhichaffectsensingaccuracy,e.g.ambienttemperatureaffectstheaccuracyofnear-infraredbasedsensors.Tworobotsequippedwithdistinctsetsofrangesensorswereusedtocollectsen-sorreadingsinthesetestbeds:aniRobotATRV-JrequippedwithaSICKLMSlaserandaCanestarangecameraandaNomad200equippedwitharingof16PolaroidsonarsensorsandaSICKPLSlaserrangenderusedonlyintheutilityphase.ThetestbedsutilizedbytheATRV-Jrweredividedintotwogroups,trainingandvericationusedinthetrainingandvericationphasesrespectively.Thereadingscollectedforthesephaseswereopportunisticallyreusedintheutilityphase.ThetrainingtestbedsareshowninFigure14anddescribedinTable13.Sensorreadingsgatheredinthesetestbedswereusedinthetrainingphasetoidentifyindicatorstodetectsensinganomalies,estimatesensingaccuracy,andisolatepoorlysensedregionsfortheSICKLMSlaserandCanestarangecamerasensors.TheseindicatorswereappliedtothereadingsgatheredinthevericationtestbedspicturedinFigure15anddescribedinTable14toevaluatetheirperformancewhenappliedtosimilarbutdistinctenvironments.FormoredetailedinformationonthesetestbedsincludingschematicsseeAppendixB.Thesetestbedswereknowntoinducesensinganomaliesforthenear-IRbasedrangesensorsinstalledontheATRV-Jrduetothepresenceoftransparentglassandabsorbentdarksurfaces.ThelaserandCanestarangecamerabothusetime-of-ightofnear-infrarednear-IRlighttomea-suredistancestonearbysurfaces.Thelab,bridge,andsidewalktestbedscontainedoor-to-ceilingwindowsthatbothsensorshavedifcultydetectingassuchsurfacesarelargelytransparentless104

PAGE 119

aLab bWalkwayFigure14.ATRV-Jrtrainingtestbeds.Table13.CharacteristicsoftheATRV-Jrtrainingtestbeds.Darkreferstothepresenceofdarkobjects.DescriptionslabeledLandRarealongtheleftorrightsideoftherobotrespectively. Anomalies Testbed Runs Type Laser Canesta Description Laba 4 Indoor Glass Hallwaywithpaintedsheet-rockwalls,woodendoorsoneitherside,onewidewindoweddoorL Labb 4 Glass Dark Asabovewithdarkobstacles Walkwaya 4 Outdoor WalkwaywithabrickwallLandwidelyspacedconcretepillarsR Walkwayb 4 Dark Asabovewithdarkobstacles Total 16 aBridge bCubicle cSidewalkFigure15.ATRV-Jrvericationtestbeds.105

PAGE 120

Table14.CharacteristicsoftheATRV-Jrvericationtestbeds.Darkreferstothepresenceofdarkobjects.DescriptionslabeledLandRarealongtheleftorrightsideoftherobotrespectively. Anomalies Testbed Runs Type Laser Canesta Description Bridgea 5 Indoor Glass Hallwaywithpaintedsheet-rockwallsandwoodendoorsL,largeoor-to-ceilingwindowsR Bridgeb 5 Glass Dark Asabovewithdarkobstacles Cubicle 10 Indoor Dark Hallwaywithsoundproongfoamcoveredinadarkfabricalongonewall Sidewalk 10 Outdoor Glass SidewalkalongabrickwallandanarrowwindoweddoorRwithanopeneldL Total 30 than8%oftheenergyisreected1inthenear-IRrange.TheCanestarangecameracancorrectlymeasuredistancestoglasssurfaces,butonlyiftheexposuretimeforthecameraisincreased.Inthismode,denotedmolemode,surfaceswithgoodreectancee.g.paintedsheet-rockwallsbe-comeoverexposedwhichthesensorinterpretsasaveryclosesurface.Thecubicletestbedcontainsdarksurfaceswhichtendtoabsorbnear-IRandarethereforeinvisibletotheCanestarangecam-eraunlesstheexposuretimeisincreased.Interestinglythelaserdoesnothavethesamedifcultlywiththedarkobstaclesused,eitherbecauseitusesawavelengththatisdeeperintheIRrange,orbecauseitusesahigherpower,focusedbeam.Byintroducingdarkobstaclesinhalfoftherunsinthelab,bridge,andwalkwaytestbeds,theseexperimentscovercaseswheretherearenosensinganomalies,onlylaseranomalies,onlyCanestarangecameraanomalies,andwherebothsensorsencounteredanomaliesinasinglerun.ThetestbedsutilizedbytheNomad,picturedinFigure16anddescribedinTable15,induceanomaliesfortheacousticandnear-IRbasedrangesensorsinstalledonthismobilerobotduetothepresenceofsmoothpaintedsheetrockwallsandglassandtransparentglasssurfacesre-spectively.TwoATRV-JrtestbedscubicleandbridgewerereusedbytheNomadfordatacol-lectionwiththecaveatthatdarksurfaceswerenotplacedinthesetestbedsastheywerefortheATRV-Jrruns.Thepresenceofsensinganomaliesforthetargetsensorsdidnotvaryoverthecourseofarunorbetweenrunswithinthesametestbed,i.e.thesonarreadingswerealwaysscat1BasedonBaldridgeetal.,2009andinformalreectancemeasurementsthereferencematerialsrequiredforabsolutemeasurementswerenotavailableofglasssurfacesinthelabandbridgetestbedstakenwithaspectrometer.106

PAGE 121

aNarrow bWideCubicle cWindowBridgeFigure16.TheNomad200testbeds.Table15.CharacteristicsoftheNomad200testbeds.ThewidthWandlengthLarereportedinmeters.Smoothreferstopaintedsheet-rockwalls. Hallway Sonar Laser W L Walls Narrow poor good 1.8 11.2 Smooth WideCubicle poor good 2.5 14.2 Smooth WindowBridge poor poor 2.0 27 Smoothandlargewindows teredbythesurroundingsurfacesand/orthelaserwasalwaysadjacenttoaglasssurfaceinthewindowtestbed.Thesetestbedswereonlyusedintheutilityphasetoexploretheapplicationofdetectionofsensinganomaliestoswitchsensorstoimprovetheoverallsensingaccuracy.5.2DataCollectionandRepresentationRealsensorreadingsfromrangesensors:sonar,SICKlaserrangenders,andaCanestarangecamera,wereregisteredtoastatic2Dmapforofineevaluation.Thissectiondescribesboththedatacollectionprocessusedtogatherthesensorreadingsandthemethodsusedtofusethoseread-ingsintoevidential2Dmapsoccupancygridmapoftherobot'ssurroundings.Anofineanal-ysissystemcomparedthesemapstogroundtruthmapsgeneratedfrommanualmeasurementsofthetestbedenvironments.Tworobotsequippedwithdistinctsetsofrangesensorswereusedtocollectsensorreadings:aniRobotATRV-JrequippedwithaSICKLMSlaserandaCanestarangecameraandaNomad200equippedwitharingof16PolaroidsonarsensorsandaSICKPLSlaserrangenderusedonlyintheutilityphase.Andetaileddescriptionofthedatacollectionprocess107

PAGE 122

aAniRobotATRV-Jrrobot bClose-upofsensorsinthebridgetestbed cNomad200robotFigure17.Therobotsandsensorsusedintheexperiments.fortheATRV-Jrrobot,schematicsforthetestbedsusedbythisrobot,andtheassociatedgroundtruthmapscanbefoundinAppendixB.TworangesensorsmountedonanATRV-Jrmobilerobotwereusedfordatacollectionforthetrainingandvericationphases:aSICKLMSlaserrangenderandaCanestarangecameraseeFigure17b.TheSICKLMSlasersensorisanear-IRbasedtime-of-ightrangesensorwitha180eldofviewandanangularresolutionofeither0:5or1:0usertunableparameter.Therangeresolutioncanbesetto0.1or1.0cmforamaximumrangeof8.2and82metersrespec-tively.TheCanestarangecameraisalsoanear-IRbasedtime-of-ightrangesensorwitha6464pixelCMOSdetectorandadiagonaleldofviewof30.Therangeresolutionofthiscameravariesbasedonusertunableparameterse.g.exposuretimeandtheenvironmentalconditionswhichdeterminethesignaltonoiseratio.Themaximumrangevariesfrom1.44to11.5metersbasedonthemodulationfrequencyselectedbytheuser.Thiswassetto52MHzforamaximumrangeof2.88meters.ThesesensorsweremountedonthefrontoftheATRV-Jrwhichisaskidsteeringindoor/outdoormobilerobotplatformproducedbyiRobotCorporationseeFigure17a.TheATRV-Jrcollectedsensorreadingsforatotalof46runsinveexperimentaltestbedsseeSection5.1withvaryingstartinganddarkobstaclepositionstopreventbiasintheoutcomeoftheexperiments.ReadingsfromthelaserandCanestarangecamerawerecollectedwhiletheATRV-Jrdrovedownthecenterofatestbedforadistanceofsixmeters.Thisprocedurewasconductedeighttimesineachtrainingtestbedandtentimesineachvericationtestbedforatotalof46runs.108

PAGE 123

Forallodd-numberedrunsinthreeofthetestbedslab,bridge,andwalkwayadarkobstaclesty-rofoamcoveredinblackfeltwasplacedalongonewallinapositiondeterminedrandomlybysoftware.Thestartingpositionoftherobotwithinthetestbedwasalsodeterminedrandomlyforeachrunbysoftware.TheNomad200collectedsonarandSICKPLSreadingsforatotalof45runsinthreeex-perimentaltestbedsseeSection5.1.Readingsfromthelaserandsonarsensorswerecollectedwhiletherobotdrovedownthecenterofatestbedforadistanceofsixmeters.Thisprocedurewasconductedtwentytimesineachtestbedandthebest45runsintermsofalackoferrorsinthedatacollectedwereusedintheutilityphaseoftheexperiments.Thesamestartingpositionoftherobotwasusedineachrunwithinagiventestbed.AnofineanalysissystemusedsensormodelstoregisterrangereadingstoanoccupancygridmapElfes,1989,withacellresolutionof10cmabout4inchesinboththexandydirections.Themapdividesatwodimensionalspaceintoequallysizedcells,eachlabeledoccupiedoremptywithsomelevelofcertainty.TheapproachoutlinedinMurphy,2000wasusedtobuildanoccu-pancygridfromrangereadingsseeSection3.4fordetailsusingevidentialcone-shapedmodels.ThespecicparametersusedtobuildthesensormodelsaregiveninTable16whereRisthesensor'srange,theangularresolution,anddistheactualrangereadingdistance.Rangeer-rorspeciestheexpectedrangeresolutionofthesensor.ThesonarparameterswerederivedfromspecicationssuppliedbyNomadicsmanualsfortheNomad200'ssensingsystem.FollowingAr-buckle,Howard,andMataric,thewidthofthelaserconewasnarrowedtoonedegreetoreecttheangularaccuracyoftheSICKlaserrangenders.ThemaximumrangeoftheSICKPLSsensorwassuppliedbythesensor'smanual.TheSICKLMSwasconguredformaximumaccuracy.1cmresolutionwhichdeterminesthesensor'ssmallestcongurablemaximumrangeof8.0meters.TheremainingparametersofthismodelarethesameforbothSICKlasers,butwithalogarithmicincreaseintherangeerrorfortheSICKLMStoaccountforthesmallerrangeres-olution.TheCanestarangecamerawasusedintwodistinctmodes:normalfordetectingtypicalsurfaceswithshortermsexposuretimesandmolemodefordetectingtransparentglassorabsorbingdarksurfaceswithlongermsexposuretimes.TheparametersfortheCanestarangecameramodelwerederivedfromthephysicalsensingmodeloutlinedinGokturk,Yalcin,andBarnji,2004tomatchknowncamerasettingse.g.modulationfrequencyandexposuretime109

PAGE 124

Table16.Theparametersusedinthesensormodels.Thesearebasedonexperimentationandmanufacturerspecications.CanestareferstotheCanestarangecamera. Sensor R Maxocc Rangeerror SICKLMS 8.0m 0.5 0.98 1%oflndor0.0whered<1:0 Canestanormal 2.88m 0.5 0.98 2.5%ofd Canestamole 2.88m 0.5 0.98 8.0%ofd sonar 6.477m 12.5 0.98 1%ofd SICKPLS 27.5m 0.5 0.98 1%ofd toobservederrorintheexperimentaltestbeds.ThepercentagesusedtocalculatetherangeerrorinTable16providedareasonableestimatewithin0.5cmoftheresultingphysicalmodelsuptoadistanceof10.0meters.5.3TrainingPhaseTheobjectiveofthisphaseoftheexperimentswastoidentifytheindicatorsfortheSICKLMSandaCanestarangecameraontheATRV-Jrforuseinthevericationandutilityphases.Towardthisendreadingsfromthesetwosensorswerecollectedinthetrainingtestbeds:labandwalkwayandappliedto2Doccupancygridsforofinetraining.Section5.3.1describeshowthe184sensingaccuracyindicatorsexaminedintheseexperimentswerederivedfromsixmethodsforquantifyinginconsistencyinevidentialmodelsfromtheevidentialliterature:Yager'sANXIETY,Shafer'sCON,Smets'transferablebeliefmodelTBMCONFLICT,Pal'sINCONSISTENCYmetric,LIU'SCONFLICTmetric,andGAMBINO'schangeheuristic.Section5.3.2describeshowaquan-titativemapqualitymetricErrorandagroundtruthevaluatingsystemwereutilizedtodeter-minethetruestatusofagivensensorbycomparingoccupancygridsbuiltfromsensorreadingstogroundtruthgrids.Section5.3.3describesmodicationstoPearson'slinearcorrelationtestusedtoevaluatetheindicators'abilitytoestimatesensingaccuracytocompensatefortime-dependencebetweensampleswithinandbetweendatacollectionruns.ThisphaseidentiedtrainedindicatorsbasedontheTBMCONFLICTandLIU'SCONFLICTmethodsforfurthertestinginthevericationphasebasedontheirabilitytodetectsensinganom-alies,estimatesensingaccuracy,andisolatepoorlysensedregionsfortheSICKLMSandCanestarangecamera.Section5.3.4presentedtheapproachusedtotrainmethodsfordetectionbyexam-iningthepercentageofcorrectlyclassiedoccupancygridsi.e.anomalousornormalandthe110

PAGE 125

results.Basedonthoseresultsthefollowingtrainedindicatorswereselectedforuseintheveri-cationphase:LIU'SCONFLICT,0.4and0.2forusewiththeSICKLMSonly,TBMCONFLICT,0.86forusewiththeCanestarangecamera,andLIU'SCONFLICT,0.9and0.1forusewithei-thersensor.Section5.3.5presentedtheapproachfortrainingmethodsforestimationbymeasur-ingcorrelationoftheindicators'conictscorewiththetruesensorerrorandtheresults.BasedonthoseresultstheTBMCONFLICT,0.18indicatorwasselectedbasedonitsabilitytoesti-matesensingaccuracyforthelaserandCanestarangecameraindividuallyandforeithersensorinterchangeably.Section5.3.6describeshowmethodsweretrainedforisolationbyexaminingthedegreeofoverlapbetweenthetrulyerroneouscellsandthoselabeledsuspecti.e.inconsistencyexceededtheassignedthresholdbytheindicatorandtheresults.BasedonthoseresultstheTBMCONFLICT,0.38indicatorwasselectedbasedonitsabilitytoisolatepoorlysensedregionsforthelaserandCanestarangecameraindividuallyandforeithersensorinterchangeably.NotethattheresultspresentedhereareasynopsisofthecompletesetofresultsgiveninAppendixC.5.3.1SensingAccuracyIndicatorsAtotalof184sensingaccuracyindicatorsbasedonsixmethodsforquantifyinginconsistencyinevidentialmodelsseeSection3.3.2:veinconsistencymetricsfromtheuncertaintylitera-tureandonechangeheuristicfromtheroboticsliteraturewereevaluatedfortheirabilitytodetectsensinganomaliesandestimatesensingaccuracyforsonarandlasersensorsinunknownenvi-ronments.Inthisworkaninconsistency-basedsensingaccuracyindicatorisamethodcoupledwithathresholdmethod,threshold.ForthosemethodswithniteupperboundsANXIETY,IN-CONSISTENCY,TBMCONFLICT,andLIU'SCONFLICT,thethresholdvaluestobetestedwereevenlydistributedthroughoutthemethod'srange.Forthosemethodswithinnitetheoreticalup-perboundsCONandGAMBINOthemaximumvaluewasdeterminedexperimentallyintheinitialexploratorystudydescribedinCarlsonandMurphy,2005.Thethresholdvaluesforthesemeth-odswereevenlydistributedthroughouttheresultingexperimentalrange.AsdescribedinSection3.4,implementationofthesemethodsforuseasasensingaccuracyindicatoronanoccupancygridmaprequirescalculationofaconictscoreandtheuseoftwothresholdstoclassifythecellandsensingsituationrespectivelyasnormalorsuspect.Asumma-tionofthevaluesforallsuspectcellscellswhosevaluesexceededthecell-levelthresholdwas111

PAGE 126

Table17.Theparametersusedtocreatesensingaccuracyindicatorsfrommethodstoquantifyinconsistencyinevidentialmodels. Method ThresholdsT jTj Cellvalue ANXIETYYager,1982 f0.1,0.14,...,0.9g 21 Metricvalue INCONSISTENCYPal,1999 f0.05,0.07,...,0.45g 21 Metricvalue CONShafer,1976 f0.25,0.5,...,5.0g 20 Metricvalue TBMCONFLICTSmets,1990a f0.1,0.14,...,0.9g 21 Metricvalue LIU'SCONFLICTLiu,2006a 81 minm;diffBetP m f0.1,0.2,...,0.9g diffBetP f0.1,0.2,...,0.9g GAMBINOGambino,Ulivi,andVendittelli,1997 f0.5,1.0,...,10.0g 20 Numberofchangesdetected Total 184 usedastheconictscore.Thesensingsituationwasdeterminedbyexaminingthepercentageofupdatedcellsthatwereclassiedassuspect.Toclassifythesensingsituation,theindicatorsap-pliedthesamethresholdvalue%tothispercentageasthegroundtruthevaluatingsystemseeSection5.3.2.Table17describesthemethodforcalculatingthevalueforeachcell,therangeofthresholds,andthenumberofthresholdvaluestestedintheseexperiments.NotethatLiu'scon-ictmetricusestwopairedvalues.Foracelltobedeemedsuspectbothvalueshadtoexceedtheirrespectivethresholds.5.3.2GroundTruthSensingAccuracyAssessmentTheexperimentsdescribedinthischapterusedaquantitativemapqualitymetric,Error,asameasureoftheactualsensingaccuracyandagroundtruthevaluatingsystemtodeterminethetruestatusofthesensorsnormalversusaffectedbyanomalies.Theseprovidedanobjectiveassess-mentofasensor'saccuracytoserveasthegroundtruth.TheErrormetricmeasuresthedifferencebetweentheoccupancygridgeneratedfromthesensorreadingsandagroundtruthoccupancygrid.Thegroundtruthgridsareautomaticallygen-eratedfrommanuallymeasuredmapsseeAppendixBforschematicsandgroundtruthgridsforeachtestbed.ThealgorithmforcalculatingtheErrorscoreisgiveninFigure18,wheregrid ox;yandgrid ex;ygivetheoccupancyandemptyvaluesfromthesensor'soccupancygrid,andtruth ox;yandtruth ex;yarethesamevaluesfromthegroundtruthgrid,respec-112

PAGE 127

Error=0foreachcellx,yiftruthx,y!=UNKNOWNandgridx,y!=UNKNOWNif|grid_ox,y-truth_ox,y|>0.5occupied_error=|grid_ox,y-truth_ox,y|elseoccupied_error=0.0if|grid_ex,y-truth_ex,y|>0.5empty_error=|grid_ex,y-truth_ex,y|elseempty_error=0.0IncreaseErrorbyMAXoccupied_error,empty_error Figure18.ProcedureusedtocalculatetheErrormetric.tively.Athresholdof0.5isusedtolterouterrorsofinsufcientsizetoaffecttherobot'sbehav-ior.LowerErrorscoresindicateabettermatchwiththegroundtruth.Thegroundtruthevaluatingsystemwasdevelopedtodeterminethetruestatusofthesensorsbasedontheaccuracyoftheirreadings.Theautomatedsystemwasbasedonasimplesimulationtodeterminewhatagivenreadingshouldhavebeendistancetothenearestsurfacealongtheread-ing'svectorversusitsactualvalue.Areadingwasdeemedaccurateifitsvaluewaswithin10cmthewidthofagridcellofthecorrectvalue.Foreachcellintheoccupancygridwhichcontainedasurfaceoccupiedcellsthepercentageofinaccuratereadingswascalculated.Ifatleast50%ofthereadingswereinaccurate,thecellwaslabeledasinaccuratelysensed.Thepercentageofin-accuratelysensedcellsofthosethatshouldhavebeenhitbythereadingswasusedtoassessthesensingsituation.Forthisstudyitwasassumedthatthetruestatusofthesensorsdependedontheerrortoleranceassociatedwiththeirassignedtaske.g.surfacemodelingversusrobotnaviga-tion,thereforeaclassicationsystemthatreliesonanintuitiveandgeneralthresholdpercentageoferrorisappropriate.Basedonempiricalevaluationoftheautomatedsystem'sclassicationac-curacyfordetectingsensinganomalies,apercentageof14%wasselectedtoserveasthethresholdforclassicationofthesensingsituation.113

PAGE 128

5.3.3StatisticalAnalysisTheanalysesemployedPearson'slinearcorrelationtestwithmodicationsassuggestedinDawdyandMatalas,1964tocompensatefortime-dependencebetweensamplesoftheconictscoreandErrorscorewithinandbetweendatacollectionruns.Pearson'slinearcorrelationanal-ysiswasusedtodetermineifanindicator'sconictscoreestimatedtheErrorscore.Bothscorestendedtoincreaseasnewreadingswereappliedtoanoccupancygrid,resultinginserialcorrela-tionbetweensampleswithinarun.Samplestakenatthesamedistancebetweenrunsinagiventestbedalsotendedtoproducesimilarvalues.Tocompensateforthesedependencies,samplesizeswereadjusteddownaccordingtothedegreeofserialcorrelationwithineachsetofscores.TocompensateforserialcorrelationwithinaruntheconictscoreandErrorscoresweretreatedassamplesfromarst-orderMarkovprocesswherethenextvaluedependsonlyonthecurrentvaluewiththeadditionofrandomnoise.ForpairedsamplesetsSandS0ofsizenacor-rectedsamplesizen0canbecalculatedasfollowsfromDawdyandMatalas,1964:n0=n1)]TJ/F18 10.909 Tf 10.909 0 Td[(r1r01 1+r1r01.1wherer1andr01aretherstorderserialcorrelationcoefcientsseeEq..2wherek=1forSandS0respectively.rk=1 N)]TJ/F19 7.97 Tf 6.586 0 Td[(kPN)]TJ/F19 7.97 Tf 6.587 0 Td[(ki=1xixi+k)]TJ/F16 7.97 Tf 24.746 4.295 Td[(1 N)]TJ/F19 7.97 Tf 6.587 0 Td[(k2PN)]TJ/F19 7.97 Tf 6.586 0 Td[(ki=1xiPN)]TJ/F19 7.97 Tf 6.587 0 Td[(ki=1xi+k q 1 N)]TJ/F19 7.97 Tf 6.587 0 Td[(kPN)]TJ/F19 7.97 Tf 6.586 0 Td[(ki=1x2i)]TJ/F16 7.97 Tf 24.745 4.295 Td[(1 N)]TJ/F19 7.97 Tf 6.586 0 Td[(k2PN)]TJ/F19 7.97 Tf 6.586 0 Td[(ki=1xi2q 1 N)]TJ/F19 7.97 Tf 6.586 0 Td[(kPN)]TJ/F19 7.97 Tf 6.587 0 Td[(ki=1x2i+k)]TJ/F16 7.97 Tf 24.745 4.295 Td[(1 N)]TJ/F19 7.97 Tf 6.587 0 Td[(k2PN)]TJ/F19 7.97 Tf 6.586 0 Td[(ki=1xi+k2.2Betweenrunsserialcorrelationsweretreatedascomingfromamovingaverageprocesswheredependencebetweenvaluesdoesnotextendthroughoutthetimeseries,butonlywithinagivenmnumberofsamples.Heremisusedtogroupallrunsperformedinthesametestbedandkissettothenumberofsamplestakenperrun.ForpairedsamplesetsTandT0ofsizenacorrectedsamplesizen0canbecalculatedasfollowsfromDawdyandMatalas,1964:n0=n 1+2rkr0k.3114

PAGE 129

whererkandr0karethekthorderserialcorrelationcoefcientsseeEquation.4formvaluesfromTandT0respectively.rk=Pmi=0xixi+k Pmi=0x2i.4Thetwocorrectionsareappliedserially.Thecorrectedsamplesizen0fromEquation.1calculatedacrossallrunsinagiventestbedistreatedastheuncorrectedsamplesizeninEqua-tion.3whichprovidesthenalcorrectedsamplesizen0.5.3.4DetectionofSensingAnomaliesTheresultsdiscussedinthissectionprovideevidencethattheevidentialinconsistency-basedapproachpresentedinthisdissertationcandetectsensinganomaliesnineindicatorsachievedbet-terthan80%classicationaccuracyforCanestarangecameraandlasersensorsinknownen-vironments.Inthisphasetheofineanalysissystemwasusedtotestall184indicatorswithreallaserandCanestarangecamerareadingsgatheredinthetwotrainingtestbeds,labandwalkway.Theresultsshowingeneralthatperformancecanbeimprovedifeachsensorispermitteditsownsensingaccuracyindicator.Forexampledetectionaccuracyforlaserreadingsreached92.98%fortwoindicatorsbasedonLIU'SCONFLICTinthetrainingtestbedswhilethepeakclassicationaccuracyforeithersensorwas84.37%.ThedetectionresultsshowedthatLIU'SCONFLICTper-formedbestwithmoderateaccuracybetterthan70%achievedbyCON,TBMCONFLICT,andGAMBINO.Theseresultsleadtotheselectionofthefollowingtrainedindicatorsforuseinthever-icationphaseseeSection5.4.3:LIU'SCONFLICT,0.4and0.2forusewiththeSICKLMSonly,TBMCONFLICT,0.86forusewiththeCanestarangecamera,andLIU'SCONFLICT,0.9and0.1forusewitheithersensor.5.3.4.1MethodTotraininconsistency-basedsensingaccuracyindicatorstodetectsensinganomaliesanof-ineanalysissystemexaminedoccupancygridmapsbuiltfromeitherlaserorCanestarangecam-erareadingsandusedclassicationstatisticstocomparethegroundtruthevaluationwiththatofeachindicator.Theindicatorsweretrainedeitheronreadingsfromasinglesensorforexclusiveusewiththatsensororonreadingsfromapairofsensorssothesensorscouldbeusedinter-115

PAGE 130

Table18.Trainedperformancefordetectionforeachinconsistencymethod.Tsreferstothethresholdvaluesusedwitheachmethod. %Correct Method LaserorCanesta Ts Laser Ts Canesta Ts LIU'SCONFLICT 84.37% 0.9,0.1 92.98% 0.5,0.2 82.50% 0.5,0.1 CON 79.42% 4.0 78.66% 5.0 83.40% 1.75 TBMCONFLICT 78.30% 0.9 75.20% 0.9 83.39% 0.86 GAMBINO 76.23% 0.5 72.34% 0.5 82.24% 0.5 INCONSISTENCY 39.50% 0.05 49.06% 0.23 24.81% 0.05 ANXIETY 39.48% 0.1 49.06% 0.34 24.75% 0.1 changeablywithoutswitchingindicators.Theindicatorthatachievedthebestdetectionaccuracy,i.e.thepercentageofcorrectlyclassiedoccupancygrids,forthetargetsensorswasselectedasthetrainedindicator.Intheofineanalysis,laserandCanestarangecamerareadingswereappliedtooccupancygridswhichwerecheckedforsensinganomaliesaftereachsensorscanwasappliedtothegrid.Thisprocesswasrepeatedforeachofthe16runsinthelabandwalkwaytestbedsre-sultingin241,680pairedclassicationsrecordedforthisphaseoftheexperiments.Theoveralldetectionaccuracywasdenedasthepercentageofcorrectlyclassiedexamples,thefalseposi-tiverate,andthefalsenegativerate.Thefalsepositiveratewascalculatedbydividingthenumberoffalsepositiveexamplesbythetotalnumberofnegativeexamples.Thefalsenegativeratewascalculatedbydividingthenumberoffalsenegativeexamplesbythetotalnumberofpositiveex-amples.5.3.4.2ResultsThedetectionaccuracyresultsforeachmethodpresentedinTable18andFigure19showthatLIU'SCONFLICTmethodperformedbestoverall.Theresultsaredividedintothreegroups:laserorcanestaresultsaregeneratedbysamplesfromalloccupancygrids,laserfromgridsbuiltfromlaserreadingsonly,andcanestafromCanestarangecameraderivedgridsonly.Figure19usesaboxandwhiskerstylegraphtoshowtheminimumbottomofline,maximumtopofline,me-diantriangle,25thpercentilebottomofbox,and75thpercentiletopofboxdetectionaccu-racyforeachmethod.Table18givesthetrainedaccuracyforeachmethodandthethresholdvalueusedtoachievethisresult.LIU'SCONFLICTdemonstratedaccuratedetectionforlaserreadingswithapercentageofcorrectclassicationsoutof241,680examplesexceeding92%.116

PAGE 131

aLaserorCanesta bLaser cCanestaFigure19.Varianceindetectionaccuracyforthethresholdvaluestested.Resultsaregiveninde-scendingorderbytrainedaccuracy%correctbrokendownbymethod.117

PAGE 132

aGroundtruthoccupancygrid. bAssensedbytheCanestarangecamerainmolemode.Figure20.ExampleofaCanestarangecamerasensinganomalythatpreventedtheinconsistencymethodsfromachievinglowfalse-negativerates.Suchananomalyisalreadydetectablewithcur-renttechnology.Noneoftheindicatorsachieved85%detectionaccuracywhentrainedforusewitheithersen-sorduetoanapparentlimitationonaccuratedetectionfortheCanestarangecamera.Atmost83.40%ofoccupancygridsbuiltfromitsreadingswerecorrectlyclassied.Adetailedlookattheresultsshowsthatthefalse-negativerateneverfellbelow18.4%innearlyoneoutofeveryvecasesananomalywasmissed.ThisislikelyduetothefactthattheCanestarangecamerainmolemodewasoverexposedformostscansinthetrainingtestbeds,sometimesleadingtoconsistentbutinaccuratereadingsatthesensor'slocationseeFigure20.AnanalogoussensinganomalyincommonCCDcamerasisinsufcientlighting.Suchanomaliesarealreadydetectiblewithcurrenttechnology,thereforetheyarenotthefocusofthiswork.5.3.5EstimationofSensingAccuracyTheresultsdiscussedinthissectionprovidestatisticallysignicantevidencethattheeviden-tialinconsistency-basedapproachpresentedinthisdissertationcanestimatesensingaccuracyindicatorsachievedstatisticallysignicantcorrelationcoefcientsof0.95orbetterforCanestarangecameraandlasersensorsinknownenvironments.Inthisphasetheofineanalysissystemwasusedtotestall184indicatorswithreallaserandCanestarangecamerareadingsgatheredinthetwotrainingtestbeds,labandwalkway.MethodsthatrelyonSmets'transferablebeliefmodelTBMCONFLICT,LIU'SCONFLICT,andGAMBINOperformedwell,achievingcorrelationsof0.84orbetter,foreithersensorandforeachsensoronanindividualbasis.Duetoobservedstabil-ityofTBMCONFLICTintermsofnearpeakperformanceacrossalltestedthresholdvalues,thismethodwasselectedforfurthertestinginthevericationphaseseeSection5.4.4.Morespeci-callytheTBMCONFLICT,0.18indicatorwasselectedforvericationforusewitheithersensorinterchangeablyandforeachsensorindividually.118

PAGE 133

5.3.5.1MethodTotraininconsistency-basedsensingaccuracyindicatorstoestimatesensingaccuracy,i.e.theextenttowhicharobot'scurrentsensingisinappropriate,anofineanalysissystemexaminedoccupancygridmapsbuiltfromeitherlaserorCanestarangecamerareadingsandusedlinearcor-relationanalysistocomparethegroundtruthErrorwiththeconictscoreforeachindicator.Theindicatorsweretrainedeitheronreadingsfromasinglesensorforexclusiveusewiththatsen-sororonreadingsfromapairofsensorssothesensorscouldbeusedinterchangeablywithoutswitchingindicators.Theindicatorthatachievedthebestsensingaccuracyestimationresults,i.e.correlationwiththeErrorscore,forthetargetsensorswasselectedasthetrainedindicator.Intheofineanalysis,laserandCanestarangecamerareadingswereappliedtooccupancygridswhichwereevaluatedforsensingaccuracyeveryhalfmeterusingtheErrorscoreandthecon-ictscore.Foreachrunreadingswereappliedtotheoccupancygriduntiltherobothadtraveledonemeter,atwhichpointthegridwasevaluated.Thegridwasreevaluatedeverysubsequenthalfmeterproducing11samplesforthelaserandeachCanestarangecameramode,foratotalof33samplesperrun.Thisprocesswasrepeatedforeachofthe16runsinthelabandwalkwayenvi-ronments,resultinginanErrorscoreandanindicator'sconictscoreforeachof528samplesin48timeseriesrecordedforuseinpost-hocanalysis.TheabilitytoestimatesensingaccuracywasmeasuredusingPearson'scorrelationcoefcientrDowdy,Weardon,andChilko,2004andthecorrespondingprobabilitywithmodicationsseeSection5.3.3.Inthiscasethetestwasusedtodetermineiftheconictscore,assignedbytheindicatortobeevaluated,variedlinearlywiththegrid'sErrorscore.5.3.5.2ResultsThesensingaccuracyestimationresultsforeachmethodarepresentedinTable19andFig-ure21.TheresultsarereportedintermsofPearson'slinearcorrelationcoefcientrwhichcom-pareseachindicator'sconictscorewiththeErrorscoretodeterminethemostpromisingindi-catorsforestimatingsensingaccuracy.Theresultsaredividedintothreegroups:laserorcanestaresultsaregeneratedbysamplesfromalloccupancygrids,laserresultsfromoccupancygridsbuiltfromlaserreadingsonly,andcanestafromCanestarangecameraderivedgridsonly.Table19givesthetrainedcorrelationcoefcientrforeachmethodandthethresholdvalueusedtoachieve119

PAGE 134

Table19.Trainedperformanceforestimationforeachinconsistencymethod.Tsreferstothethresholdvaluesusedwitheachmethodandydenotesstatisticallysignicantcorrelationp=0:01. Correlationr Method LaserorCanesta Ts Laser Ts Canesta Ts TBMCONFLICT 0.9674y 0.7 0.9866y 0.26 0.8399y 0.1 GAMBINO 0.9695y 0.5 0.9828y 0.5 0.8412y 0.5 LIU'SCONFLICT 0.9603y 0.1,0.1 0.9560y 0.1,0.1 0.8477y 0.1,0.1 INCONSISTENCY 0.7193y 0.23 0.8034y 0.23 0.5536 0.25 ANXIETY 0.7048y 0.46 0.8177y 0.46 0.4490 0.62 CON 0.5551y 0.25 0.6442 0.25 0.0511 0.25 thisresult.Figure21usesaboxandwhiskerstylegraphtoshowtheminimumbottomofline,maximumtopofline,mediantriangle,25thpercentilebottomofbox,and75thpercentiletopofboxcorrelationcoefcientrforeachmethod.Atwo-tailedtestatap-valueof0:01i.e.lessthana1%chanceofnorelationshipbetweentheconictscoreandtheErrorscorewasusedforstatisticalsignicance,andpassingcoefcients2aremarkedwithy.Table19showsthatthetrainedindicatorsbasedonTBMCONFLICT,GAMBINO,andLIU'SCONFLICTperformedverywellonthistask,achievingcorrelationcoefcientsabove0:83forCanestarangecamerareadings,andabove0:95otherwise.Figure21showsthatalltestedindica-torsbasedonTBMCONFLICTandLIU'SCONFLICTperformedwellonthistask.ForGAMBINO,correlationsremainhighabove0.9untilthethresholdreaches2:5,atwhichpointthecorrelationdropsto0.0andremainsthere.5.3.6IsolationofPoorlySensedRegionsTheresultsdiscussedinthissectionprovideevidencethattheevidentialinconsistency-basedapproachpresentedinthisdissertationcanisolatepoorlysensedregionsindicatorsachievedOverlapscoresatorabove0.5fortheCanestarangecameraandlasersensorsinknownenviron-ments.Inthisphasetheofineanalysissystemwasusedtotestall184indicatorswithreallaserandCanestarangecamerareadingsgatheredinthetwotrainingtestbeds,labandwalkway.TheresultsshowthatthebestperformingindicatorsarederivedfromSmets'transferablebeliefmodel,specicallyTBMCONFLICT,LIU'SCONFLICT,andGAMBINO.TheTBMCONFLICTmethod 2NotethatbothsamplesizecorrectionseeSection5.3.3andBonferronicorrectionfortheuseofmultiplethresh-oldvalueswereappliedintheprocessofdeterminingstatisticalsignicance.120

PAGE 135

aLaserorCanesta bLaser cCanestaFigure21.Varianceinestimationperformanceforthethresholdvaluestested.Resultsaregivenindescendingorderbytrainedcorrelationrperformance,brokendownbymethod.121

PAGE 136

performedbestwithconsistentlygoodperformanceacrossalltestedthresholdvalues.BestontheseresultstheTBMCONFLICT,0.38indicatorwasthereforeselectedforfurthertestinginthevericationtestbedsforusewitheithersensorinterchangeablyandforeachsensorindividually.5.3.6.1MethodTotraininconsistency-basedsensingaccuracyindicatorstoisolatepoorlysensedregionsanofineanalysissystemexaminedoccupancygridmapsbuiltfromeitherlaserorCanestarangecamerareadingsandusedacustomOverlapmetrictodeterminehowmanytrulyerroneouscellswerelabeledsuspect.Intheofineanalysis,laserandCanestarangecamerareadingswereap-pliedtooccupancygridsandeveryhalfmeterofrobottravelanOverlapscorewascalculatedseeEquation.5andstored.Foreachrunreadingswereappliedtotheoccupancygriduntiltherobothadtraveledonemeter,atwhichpointthegridwasevaluated.Thegridwasreevalu-atedeverysubsequenthalfmeterproducing11samplesforthelaserandeachCanestarangecam-eramode,foratotalof33samplesperrun.Thisprocesswasrepeatedforeachofthe16runsinthelabandwalkwaytestbeds,resultinganOverlapscoreforeachof528samplesin48timese-riesrecordedforuseinpost-hocanalysis.TocalculatethevalueoftheOverlapmetric,theofineanalysissystemrequestedbinarymapsfromtheindicatorandthegroundtruthevaluatingsystemwhichindicatedsuspectanderro-neouscellsrespectively.ThismetricwasinspiredfromPal'sINCONSISTENCYmetricPal,1999andisdenedasfollows:Overlap=jSuspectErroneousj jSuspect[Erroneousj.5whereSuspectreferstothesetofcellswithintheoccupancygridlabeledassuspectbyasensingaccuracyindicatorandErroneousreferstothesetoferroneouscellsaccordingtotheprocedureoutlinedinFigure22.5.3.6.2ResultsTheresultsforisolatingpoorlysensedregionsforeachmethodarepresentedinTable20andFigure23whichshowthatTBMCONFLICTperformedwell.Itsabilitytoisolatepoorlysensedregionswasmaintainedoverawiderrangeofthresholdvaluesascomparedtotheothermethods.122

PAGE 137

foreachcellx,yerroneousx,y=falseiftruthx,y!=UNKNOWNandgridx,y!=UNKNOWNif|grid_ox,y-truth_ox,y|>0.5occupied_wrong=trueelseoccupied_wrong=falseif|grid_ex,y-truth_ex,y|>0.5empty_wrong=trueelseempty_wrong=falseerroneousx,y=occupied_wrongORempty_wrong Figure22.Procedureusedtoclassifyeachcellasaccurateorerroneous.Table20.Trainedperformanceforisolationforeachinconsistencymethod.Tsreferstothethresholdvaluesusedwitheachmethod. MeanOverlap Method LaserorCanesta Ts Laser Ts Canesta Ts TBMCONFLICT 0.61 0.38 0.69 0.38 0.57 0.38 GAMBINO 0.61 0.5 0.68 0.5 0.57 0.5 LIU'SCONFLICT 0.51 0.3,0.1 0.62 0.5,0.1 0.46 0.3,0.1 CON 0.15 5.0 0.15 5.0 0.14 3.5 ANXIETY 0.03 0.86 0.05 0.1 0.02 0.1 INCONSISTENCY 0.03 0.45 0.05 0.05 0.02 0.05 Figure23showstherangeandvarianceoftheOverlapmetric.Aboxandwhiskerstylegraphisusedtoshowtheminimumbottomofline,maximumtopofline,25thpercentilebottomofbox,and75thpercentiletopofboxvalueforeachmethod.TheredlineinFigure23showstheOverlapscorewhencellsarerandomlylabeledasnormalorsuspectasareferenceforcompar-ison.Table19showsthetrainedperformanceforeachmethodintermsofthemeanvalueoftheOverlapmetricaveragedoverthe528samplesavailableinthelabandwalkwaytestbeds.ItshowsthatthebestperformingmethodsfortheisolationcomponentarebasedonSmets'transferablebeliefmodeli.e.TBMCONFLICT,LIU'SCONFLICT,andGAMBINO.CONshowedalimitedca-pacityforisolationwithOverlapscoresof0.14.15.TheTheremainingindicatorsperformedpoorlywithlittletonooverlapbetweentheirsuspectanderroneouscells.123

PAGE 138

Figure23.Varianceinisolationperformanceforthethresholdvaluestested.ResultsaregivenindescendingorderbytrainedperformanceOverlapforeachmethod.TheredlineindicatesthemeanOverlapscorewhencellsarerandomlylabeled.5.3.7SummaryTheresultspresentedinthissectionwereusedtoselectindicatorstodetectsensinganoma-lies,estimatesensingaccuracy,andisolatepoorlysensedregionsforusewiththeSICKLMSandCanestarangecamerasensorsinstalledonanATRV-Jrinthevericationphase.Thiswasdonetodetermineifanindicatortrainedinknownenvironmentscanbeusedtodetectandcharacterizesensinganomaliesinnewunexploredenvironments.Basedontheexperimentalresultsexamin-ing184inconsistency-basedsensingaccuracyindicatorsderivedfrommethodsforquantifyinginconsistencyinevidentialmodelsfoundintheevidentialliterature,veindicatorsbasedontwomethodswereselectedforthevericationphase.Threeoftheseweretrainedfordetection:LIU'SCONFLICT,0.4and0.2forusewiththeSICKLMS,TBMCONFLICT,0.86forusewiththeCanestarangecamera,andLIU'SCONFLICT,0.9and0.1forusewitheithersensor.TheothertwoweretrainedforestimationTBMCONFLICT,0.18andisolationTBMCONFLICT,0.38.Section5.3.4providesevidencethattrainedindicatorscandetectsensinganomaliesninein-dicatorsachievedbetterthan80%classicationaccuracyfortheCanestarangecameraandlasersensorsinknownenvironments.Performancecanbeimprovedifeachsensorispermitteditsownindicator.Forexampledetectionaccuracyforlaserreadingsreached92.98%whilethebestac-curacyforeithersensorwas84.37%.LIU'SCONFLICTperformedbestwithmoderateaccuracybetterthan70%achievedbyCON,TBMCONFLICT,andGAMBINO.124

PAGE 139

Section5.3.5providesstatisticallysignicantevidencethattrainedindicatorscanestimatesensingaccuracyindicatorsachievedcorrelationcoefcientsof0.95orbetterforthesesen-sorsinknownenvironments.MethodsthatrelyonSmets'transferablebeliefmodelTBMCON-FLICT,LIU'SCONFLICT,andGAMBINOperformedwell,achievingcorrelationsof0.84orbetter,foreithersensorandforeachsensoronanindividualbasis.TBMCONFLICTperformedbest,pro-vidingnearpeakperformanceacrossalltestedthresholdvalues.Section5.3.6showsthattrainedindicatorscanalsoisolatepoorlysensedregionsindi-catorsachievedOverlapscoresatorabove0.5forthesesensorsinknownenvironments.TheresultsshowthatthebestperformingindicatorswereagainderivedfromSmets'transferablebe-liefmodel,specicallyTBMCONFLICT,LIU'SCONFLICT,andGAMBINO.TheTBMCONFLICTmethodperformedbestwithconsistentlygoodperformanceacrossalltestedthresholdvalues.Overallthetrainingresultsshowthatdifferentthresholdsarerequiredtolterordinarynoiseforthethreecomponentsofdetectingandcharacterizingsensinganomaliesconsideredinthiswork.AsinthefeasibilitystudyseeChapter4,thetrainingresultsshowthataccuratedetec-tionrequireshigherthresholdvaluesthanestimation.Thresholdsforisolationtendtofallbetweenthese.5.4VericationPhaseTheobjectiveofthisphasewastodetermineiftheevidentialinconsistency-basedapproachfordetectingandcharacterizingsensinganomaliespresentedinthisdissertationcouldberelieduponinnewunexploredenvironments.TowardthisendreadingsfromtheSICKLMSandCanestarangecameraontheATRV-Jrwerecollectedinthevericationtestbeds:bridge,cubicle,andside-walkandappliedto2Doccupancygridsforofineevaluation.ThisphaseusedthesameapproachforgroundtruthsensingaccuracyassessmentandstatisticalanalysisasthetrainingphasebriefreviewsareprovidedinSections5.4.1and5.4.2respectively.ItagainemploysaquantitativemapqualitymetricErrorandagroundtruthevaluatingsystemtodeterminethetruestatusofagivensensorbycomparingoccupancygridsbuiltfromsensorreadingstogroundtruthgrids.Pearson'slinearcorrelationtestwasagainusedtoevaluatetheindicators'abilitytoestimatesens-ingaccuracy,withmodicationstocompensatefortime-dependencebetweensampleswithinandbetweendatacollectionruns.Basedontheresultsfromthetrainingphasethefollowingindica-125

PAGE 140

torswereusedtodetectsensinganomalies:LIU'SCONFLICT,0.4and0.2forusewiththeSICKLMSonly,TBMCONFLICT,0.86forusewiththeCanestarangecamera,andLIU'SCONFLICT,0.9and0.1forusewitheithersensor.TheTBMCONFLICT,0.18indicatorwasusedtoestimatesensingaccuracyandtheTBMCONFLICT,0.38indicatorwasusedtoisolatepoorlysensedre-gions.Thisphaseexaminedtheperformanceofthetrainedindicatorsandshowedthattheycouldestimatesensingaccuracyandisolatepoorlysensedregionsinnewenvironmentsbuttheabilitytodetectsensinganomaliesdidnottransfer,with28.71%to74.71%accuracyinthevericationtestbeds.Section5.4.3presentedthemethodusedtoevaluatetheperformanceoftheindicatorsselectedfordetectionandshowedthattheyperformedpoorly.Thissuggeststhattheapproachpre-sentedinthisdissertationcannotbereliedupontodistinguishsensinganomaliesfromordinarynoiseinunknownenvironments.Sections5.4.4and5.4.5describethemethodusedtoevaluatetheperformanceofthetrainedindicatorsforestimatingsensingaccuracyandisolatingpoorlysensedregionsinthevericationtestbedsrespectively.Theresultsshowedthatbothmaintainedtheirper-formanceonthesetasksfortheSICKLMS,Canestarangecamera,andforbothusedinterchange-ablyinthenewenvironments.Afollow-upanalysispresentedinSection5.4.6wasconductedtodeterminewhydetectionfailedinsomeenvironmentswhenestimationandisolationtransferredwithlittledegradationinperformance.Thisanalysisexaminedthecontributionsofsensornoise,poseestimationerror,andsensinganomaliestothesensorerrorandoverallinconsistencyasmea-suredbythetrainedindicators.Itconcludedthatthefailureofdetectionwasduetosubstantialposeestimationerrors,sensornoise,andsensinganomaliesi.e.overexposurethatproducedcon-sistentbuterroneousreadingsconsistentnearzerorangereadings.5.4.1GroundTruthSensingAccuracyAssessmentThisphaseusedthesameapproachforgroundtruthsensingaccuracyassessmentasthetrain-ingphaseseeSection5.3.2.Toreview,theexperimentsusedaquantitativemapqualitymetric,Error,asameasureoftheactualsensingaccuracywhichmeasuresthedifferencebetweentheoccupancygridgeneratedfromthesensorreadingsandagroundtruthoccupancygrid.Thesens-ingsituationwasclassiedusinganautomatedsystemtodeterminethetruestatusofthesensorsnormalversusaffectedbyanomalies.Thissystemperformedasimplesimulationtodetermine126

PAGE 141

thepercentageofcellscontainingsurfacesi.e.occupiedcellsinthegroundtruthmapsthatwereaccuratelysensedandappliedanempiricallydeterminedthresholdof14%tothatpercentagetoclassifythesensingsituationasnormaloranomalous.5.4.2StatisticalAnalysisThisphaseusedthesameapproachforstatisticalanalysisasthetrainingphaseasdescribedinSection5.3.3.Toreview,theanalysisemployedPearson'slinearcorrelationtestwithmodica-tionsassuggestedinDawdyandMatalas,1964tocompensatefortime-dependencebetweensamplesoftheconictscoreandErrorscorewithinandbetweendatacollectionruns.ThesemodicationsreducedthesamplesizeaccordingtothemagnitudeofserialcorrelationwithintherecordedsetsofconictscoresandErrorscores.5.4.3DetectionofSensingAnomaliesTheresultsdiscussedinthissectiondonotsupportthehypothesisthatthetrainedindicatorscandistinguishsensinganomaliesfromordinarynoise,i.e.detectsensinganomalies,innewen-vironments.InthisphasetheofineanalysissystemwasusedtotestthetrainedindicatorswithreallaserandCanestarangecamerareadingsgatheredinthethreevericationtestbeds:bridge,cubicle,andsidewalk.Theseindicatorswere:LIU'SCONFLICT,0.4and0.2forusewiththeSICKLMSonly,TBMCONFLICT,0.86forusewiththeCanestarangecamera,andLIU'SCONFLICT,0.9and0.1forusewitheithersensor.Detectionresultsinthevericationtestbedsarepoor.46%forlaserto70.51%fortheCanestarangecamera.5.4.3.1MethodTodetermineifinconsistency-basedsensingaccuracyindicatorstrainedtodetectsensinganomaliesinknownenvironmentscanbeusedinnewenvironmentsanofineanalysissystemexaminedoccupancygridmapsbuiltfromeitherlaserorCanestarangecamerareadingsfromthevericationtestbedsandusedclassicationstatisticstocomparethegroundtruthevaluationwiththatofeachindicator.Intheofineanalysis,laserandCanestarangecamerareadingswereap-pliedtooccupancygridswhichwerecheckedforsensinganomaliesaftereachsensorscanwasappliedtothegrid.Thisprocesswasrepeatedforeachofthe30runsinthebridge,cubicle,and127

PAGE 142

Table21.Detectionperformanceforthetrainedindicatorsinthetrainingandvericationtestbeds. %Correct LaserorCanesta Laser Canesta Training 84.37% 92.98% 83.39% Verication 55.69% 40.46% 70.51% sidewalktestbedsresultingin485,364pairedclassicationsrecordedforthisphaseoftheexper-iments.Theoveralldetectionaccuracywasdenedasthepercentageofcorrectlyclassiedex-amples,thefalsepositiverate,andthefalsenegativerate.Thefalsepositiveratewascalculatedbydividingthenumberoffalsepositiveexamplesbythetotalnumberofnegativeexamples.Thefalsenegativeratewascalculatedbydividingthenumberoffalsenegativeexamplesbythetotalnumberofpositiveexamples.5.4.3.2ResultsTheresultsforthetrainedindicatorsonthedetectiontask,giveninTable21,showpooraccu-racyatbest70.51%inthevericationtestbeds.Table21liststhedetectionaccuracyforthelaserreadingsusingLIU'SCONFLICT,0.4and0.2,CanestarangecamerareadingsusingTBMCON-FLICT,0.86,andforeithersensorusingLIU'SCONFLICT,0.9and0.1inthetrainingversusthevericationtestbeds.Theseresultsshowpoorperformance,especiallyforlaserreadings.46%accuracy,inthevericationtestbeds.Poordetectionaccuracyofthemostpromisingindicatorsinthevericationtestbedsledtomultipleeffortstondalternativesolutionsandnallyadetailedanalysisofthesourcesofer-rorandinconsistencyineachofthetestbedsseeSection5.4.6.Firstmoreoftheindicatorsthatclassiedwellatorabove80%accuracyinthetrainingphaseweretested.TheheuristicthatwassuccessfulindetectingsensinganomaliesinpriorexperimentsseeSection3.4wasalsotested.Newindicatorsweretrainedfordetectionusingthisheuristicbuttheyachievedatbestonly80.08%forlaserreadings,66.57%forCanestarangecamerareadings,and71.85%foreitherinthetrainingtestbedsfollowedbyevenpoorerdetectionperformanceinthevericationtestbeds.ThedetectionresultsforboththegenericapproachusedinthisphaseandtheadhocapproachusedinthefeasibilitystudycanbefoundinAppendixC.Anin-depthanalysisofthesourcesoferrorandinconsistencyineachofthetestbedsseeSection5.4.6ledtotheconclusionthatthe128

PAGE 143

presenceofhighlevelsofposeestimationerrorsandsensornoiseinsometestbedsviolatedtheassumptionrequiredfordetectionthatinconsistenciesfromsensinganomalieswoulddominateothersources.5.4.4EstimationofSensingAccuracyTheresultsdiscussedinthissectionprovidestatisticallysignicantevidencethatsensingac-curacycouldbeestimatedforunknownenvironmentsusingmethodsandthresholdsdeterminedduringtraining.Theestimationcomponentisdifferentfromdetectionbecauseitestimatestheoverallerrorinsensorreadingswhichmaybecausedbysensornoise,discretizationintheoccu-pancygridmap,etc.withoutdistinguishingthesefromsensinganomalies.TheTBMCONFLICT,0.18indicatorachievedstatisticallysignicantcorrelationswiththeErrorscoreinexcessof0.8fortheSICKLMSandCanestarangecamerareadingsinboththetrainingandvericationphasesoftheexperimentswithatmosta0.1556dropincorrelation.TheseresultswereachievedrelyingsolelyonreadingsfromasinglesensorSICKLMSortheCanestarangecamera.Thismeansthatarobotenteringanunknownenvironmentwouldhaveameansofestimatingthetrustworthinessofsensedreadings,andsomeindicationwhensensingqualityisdiminished.5.4.4.1MethodTodetermineifinconsistency-basedsensingaccuracyindicatorstrainedtoestimatesensingaccuracyinknownenvironmentscanbeusedinnewenvironmentsanofineanalysissystemex-aminedoccupancygridmapsbuiltfromeitherlaserorCanestarangecamerareadingsfromthevericationtestbedsandusedlinearcorrelationanalysistocomparethegroundtruthErrorwiththeconictscore.Intheofineanalysis,laserandCanestarangecamerareadingswereappliedtooccupancygridswhichwereevaluatedforsensingaccuracyeveryhalfmeterusingtheErrorscoreandtheconictscore.Foreachrunreadingswereappliedtotheoccupancygriduntiltherobothadtraveledonemeter,atwhichpointthegridwasevaluated.Thegridwasreevaluatedeverysubsequenthalfmeterproducing11samplesforthelaserandeachCanestarangecam-eramode,foratotalof33samplesperrun.Thisprocesswasrepeatedforeachofthe30runsinthebridge,cubicle,andsidewalktestbeds,resultinginanErrorscoreandanindicator'sconictscoreforeachof990samplesin90timeseriesrecordedforuseinpost-hocanalysis.Theabil-129

PAGE 144

Table22.EstimationperformancefortheTBMCONFLICT,0.18indicatorinthetrainingandvericationtestbeds.ydenotesastatisticallysignicantcorrelationp=0:01. Correlationr LaserorCanesta Laser Canesta Explore 0.9673y 0.9866y 0.8399y Evaluate 0.9179y 0.8310y 0.8598y itytoestimatesensingaccuracywasmeasuredusingPearson'scorrelationcoefcientrDowdy,Weardon,andChilko,2004andthecorrespondingprobabilitywithmodicationsseeSec-tion5.3.3.Inthiscasethetestwasusedtodetermineiftheconictscore,assignedbytheindica-tortobeevaluated,variedlinearlywiththegrid'sErrorscore.5.4.4.2ResultsTheresultssupportthehypothesisthatasensingaccuracyindicatorchosenforagivensens-ingsuitebasedonexperimentsinknownenvironmentscanbeusedtoestimatesensingaccuracyinnewunexploredenvironments.Table22showstheperformanceoftheTBMCONFLICT,0.18in-dicatorinthetrainingversusthevericationtestbeds.Theresultsshowstatisticallysignicantcor-relationsof0.83orbetterforeithersensorandforeachsensorindividually,withatmosta0.1556dropinrforlaserreadingsinthevericationtestbeds.5.4.5IsolationofPoorlySensedRegionsTheresultsdiscussedinthissectionprovideevidencethataninconsistency-basedsensingac-curacyindicatortrainedinknownenvironmentscanbeusedtoisolatepoorlysensedregionsinnewenvironments.Theisolationcomponentisdifferentfromdetectionbecauseitdeterminestheregionsintheenvironmentwherethereislikelytobesensorerrorwithoutdistinguishingerrorduetonoisefromthatofsensinganomalies.InthisphasetheofineanalysissystemwasusedtotesttheTBMCONFLICT,0.38indicatorwithreallaserandCanestarangecamerareadingsgath-eredinthethreevericationtestbeds:bridge,cubicle,andsidewalk.Theresultsshowsimilarorbetterperformanceontheisolationtaskbetweenthetrainingandvericationtestbeds,withanav-erageoverlapwitherroneouscellsof0.61inthetrainingtestbedsascomparedtoanoverlapof0.01whencellsarerandomlylabeledandanoverlapof0.49to0.62inthevericationtestbeds.TheseresultswereachievedrelyingsolelyonreadingsfromasinglesensorSICKLMSorthe130

PAGE 145

Canestarangecamera.Thismeansthatarobotoperatinginanunknownenvironmentcouldiden-tifypoorlysensedregionsenablingonlineadaptatione.g.avoidingorre-sensingthoseareas.5.4.5.1MethodTodetermineifinconsistency-basedsensingaccuracyindicatorstrainedtoisolatepoorlysensedregionsinknownenvironmentscanbeusedinnewenvironmentsanofineanalysissys-temexaminedoccupancygridmapsbuiltfromeitherlaserorCanestarangecamerareadingsfromthevericationtestbedsandusedacustomOverlapmetrictodeterminehowmanytrulyerro-neouscellswerelabeledsuspect.Intheofineanalysis,laserandCanestarangecamerareadingswereappliedtooccupancygridsandeveryhalfmeterofrobottravelanOverlapscorewascal-culatedseeEquation.6andstored.Foreachrunreadingswereappliedtotheoccupancygriduntiltherobothadtraveledonemeter,atwhichpointthegridwasevaluated.Thegridwasreeval-uatedeverysubsequenthalfmeterproducing11samplesforthelaserandeachCanestarangecam-eramode,foratotalof33samplesperrun.Thisprocesswasrepeatedforeachofthe30runsinthebridge,cubicle,andsidewalktestbeds,resultinganOverlapscoreforeachof990samplesin90timeseriesrecordedforuseinpost-hocanalysis.TocalculatethevalueoftheOverlapmetric,theofineanalysissystemrequestedbinarymapsfromtheindicatorandthegroundtruthevaluatingsystemwhichindicatedsuspectanderro-neouscellsrespectively.ThismetricwasinspiredfromPal'sINCONSISTENCYmetricPal,1999andisdenedasfollows:Overlap=jSuspectErroneousj jSuspect[Erroneousj.6whereSuspectreferstothesetofcellswithintheoccupancygridlabeledassuspectbyasensingaccuracyindicatorandErroneousreferstothesetoferroneouscellsaccordingtotheprocedureoutlinedinFigure24.5.4.5.2ResultsTheresultssupportthehypothesisthatasensingaccuracyindicatorchosenforagivensens-ingsuitebasedonexperimentsinknownenvironmentscanbeusedtoisolatesensinganomaliesinnewunexploredenvironments.Table23andFigure25showtheperformanceoftheTBM131

PAGE 146

foreachcellx,yerroneousx,y=falseiftruthx,y!=UNKNOWNandgridx,y!=UNKNOWNif|grid_ox,y-truth_ox,y|>0.5occupied_wrong=trueelseoccupied_wrong=falseif|grid_ex,y-truth_ex,y|>0.5empty_wrong=trueelseempty_wrong=falseerroneousx,y=occupied_wrongORempty_wrong Figure24.Procedureusedtoclassifyeachcellasaccurateorerroneous.Table23.IsolationperformancefortheTBMCONFLICT,0.38indicatorbrokendownbytestbed. MeanOverlap Testbed LaserorCanesta Laser Canesta Lab 0.59 0.58 0.59 Walkway 0.63 0.80 0.54 Bridge 0.62 0.61 0.62 Cubicle 0.59 0.73 0.52 Sidewalk 0.49 0.40 0.53 Figure25.MeanandvarianceintheisolationperformanceoftheTBMCONFLICT,0.38indica-torbrokendownbytestbed.Exploratorytestbedsaretotheleftandvericationtestbedsaretotheright.132

PAGE 147

CONFLICT,0.38indicatorbrokendownbytestbedforboththetraininglabandwalkwayandvericationbridge,cubicle,andsidewalktestbeds.Table23showsthemeanOverlapscoresforeachtestbedbrokendownintolaserorcanestaforresultsgeneratedfromalloccupancygrids,laserforgridsbuiltfromlaserreadingsonly,andcanestaforgridsderivedfromCanestarangecamerareadingsonly.Figure25givesthemeanandstandarddeviationerrorbarsOverlapscoresforeachtestbed.Redlinesinthisguredepictthemeanperformancewheneachcellisran-domlylabeledassuspectornormal.TheresultsfromthevericationtestbedsshowedthesameorbetterperformanceofTBMCONFLICT,0.38withOverlapscoresof0.40.73forthelaser,0.52.53fortheCanestarangecamera,0.49.62forbothsensorsinterchangeably.TheOverlapscoresshowconsiderablybetterperformanceascomparedtoarandomuninformedclassier,evenwhenthebroadvarianceinperformancewithinallbutthelabtestbedistakenintoaccount.5.4.6Followup:SourcesofErrorandInconsistencyThefollow-upstudydescribedinthissectionwasdesignedtoaddressthehypothesisthatpoorperformanceofthetrainedindicatorsfordetectionofsensinganomaliesinthisphasewasduetounexpectedlyhighlevelsofsensornoiseandposeestimationi.e.localizationerrors.Annotatedmapsweregeneratedwhichprovidethemostlikelysourcesoferrororinconsistencyobservedinagivencellandusedtoquantifythepercentageandmagnitudeoferrororinconsistencygener-atedbyeachsource.Thecontributionofsensornoise,localizationerrors,andsensinganomaliestotheErrorandconictscoreswasquantiedandcomparedtothedetectionaccuracyresultsforeachofthetrainingandvericationtestbeds.Theresultspartiallysupportthehypothesisthatthepoorperformanceinsomeenvironmentsofthetrainedindicatorsfordetectionofsensinganoma-lieswasduetoviolationoftheassumptionthatsensinganomaliescausemoreinconsistencythanothersources.Theresultsshowthatthefailureofdetectionforthelaserwasduetosubstantiallo-calizationerrors,andfortheCanestarangecamerawasduetosensornoiseandsensinganomaliesoverexposurethatproducedconsistentbuterroneousreadingsconsistentnearzeroreadings.5.4.6.1MethodThecontributionofsensornoise,localizationerrors,andsensinganomaliestotheErrorandconictscoreswasquantiedandcomparedtothedetectionaccuracyresultsforeachofthetrain-133

PAGE 148

ingandvericationtestbedstodeterminewhydetectionofsensinganomaliesfailedinsomeenvi-ronmentsandnotinothers.AnnotatedmapsseeAppendixBwereautomaticallygeneratedforeachtestbedwhichprovide,foreachcell,themostlikelysourceoferrororinconsistencyobservedinthatcellforthecurrentactivesensor.Forexample,cellsonornearanordinarywallweregiventhelabellocalizationforthelasertomeanlocalizationorquantizationerrorswerethemostlikelysourceoferrorsorinconsistencywhereascellsonornearglasspanesweregiventhelabelsens-inganomaly.Theseannotatedmapswereusedtorecordthenumberoferroneousandthenumberofsuspectcellsaccordingtothemostpromisingindicatorsfordetectioncausedbyeachofthefollowing:sensinganomalies,sensornoise,andlocalizationerroraftereachsensorscanwasreg-isteredtothemap.Thepercentageoferroneousandsuspectcellscausedbyeachsourcewasalsorecordedforpost-hocanalysis.Thisprocesswasrepeatedforall46experimentalrunsresultinginmagnitudeandpercentagesamplesforeachsourcefrom727,044occupancygrids.5.4.6.2ResultsTheresultspresentedinthissectionpartiallysupportthehypothesisthatthepoorperformanceoftheapproachpresentedinthisdissertationfordetectionofsensinganomaliesinsomeenviron-mentswasduetoviolationoftheassumptionthatsensinganomaliescausemoreinconsistencythanotherpotentialsources.Todeterminewhydetectionfailed,annotatedmapsforeachenviron-mentwereexaminedtoquantifythemagnitudeoferrorandinconsistencycausedbylocalizationerror,sensornoise,andsensinganomalies.Theresultsshowthatthefailureofdetectionforlaserreadingswasduetosubstantiallocalizationerrors,andforCanestarangecamerareadingswasduetosensornoiseandsensinganomaliesthatproducedconsistentbuterroneousreadings.Ta-ble24andFigure26showtheresultsofthisanalysisincludingdetectionaccuracyandtherelativecontributionofsensinganomalies,sensornoise,andlocalizationerrorforoccupancygridsbuiltfromlaserreadingsonly,Canestarangecamerareadingsonly,andfromeithersensor'sreadings,brokendownbytestbed.Table24givesthedetectionaccuracyforthetrainedindicatorsseeSec-tion5.3.4,namelyLIU'SCONFLICT,0.4and0.2forlaserreadings,TBMCONFLICT,0.86forCanestarangecamerareadings,andLIU'SCONFLICT,0.9and0.1forusewitheithersen-sor.Figure26showsthesesameaccuracyresultssuperimposedasthickblacklinesonbargraphswhichdepicttherelativecontributionofsensinganomaliesred,localizationerrorsyellow,and134

PAGE 149

Table24.Detectionaccuracyforthetrainedindicatorsbrokendownbytestbed. Sensors Lab Walkway Bridge Cubicle Sidewalk Overall Laser 90.17% 96.15% 50.68% 21.71% 50.27% 58.69% Canesta 93.22% 70.00% 62.80% 75.48% 73.92% 74.54% LaserorCanesta 92.23% 75.22% 74.71% 28.71% 64.28% 65.23% aLaserErrorscore bLaserconictscore cCanestaErrorscore dCanestaconictscoreFigure26.Detectionaccuracyblackbarscomparedtotherelativecontributionofsensinganom-aliesred,poseestimationerrorsyellow,andsensornoisegreenbrokendownbytestbed.sensornoisegreentothenumberoferroneousleftcolumnandsuspectrightcolumncells.ThetoprowofgraphsshowstheresultsforoccupancygridsbuiltfromlaserreadingsonlyandthebottomrowfromCanestarangecamerareadings.TheresultspresentedinTable24andFigure26showthatthefailureofdetectionforlaserreadingswasduetosubstantiallocalizationerrors.Intheabsenceofsensinganomaliesseeresultsforwalkwayversuscubicle,themagnitudeoflocalizationandquantizationerrorsdeterminedtheaccuracyoftheLIU'SCONFLICT,0.4and0.2indicator.Otherwisetheratioofsuspectcellsduetolocalizationerrorandthoseduetosensinganomaliesappearstobethedecidingfactor.Thisratiowas1.8inthelabtestbedwheredetectionaccuracywas90.17%,and2.3.7inthebridgeandsidewalktestbedswhereaccuracydroppedto50%.Duetothelargeeldofview180and135

PAGE 150

range.0metersofthelaser,minorrotationalerrorsintherobot'slocalizationcausedsubstantialinconsistencyintheoccupancygridcellsonornearwallsaheadoftherobot.TheresultsshowthatfailureofdetectionforCanestarangecamerareadingswasduetosensornoiseandsensinganomaliesoverexposurethatproducedconsistentbuterroneousreadingscon-sistentnearzeroreadings.FortheCanestarangecamera,localizationerrorwaslessofaprob-lemsincethenarroweldofview30ofthesensorwasdirectedatawalllessthan2.0metersaway.Theresultsforthissensorshowaccuratedetection.22%whenthepercentageoferro-neouscellsduetosensinganomalieswasabove60%.99%forlabandmoderatedetectionper-formance.80.54%otherwise.AnotherexplanationforreducedaccuracyfortheCanestarangecameraistheinclusionofsensinganomaliesforthesensorinmolemodewheretheread-ingswereconsistentbutwrong.ToseethisresultcomparetheheightofsensinganomaliesredinthewalkwaytestbedresultsforErrorandtheconictscore.Anotherlikelysourceisinconsisten-ciesintroducedbysensornoisethatdidnotresultinerroneouscells.ToseethisresultcomparetheheightofnoisegreeninthebridgetestbedresultsforErrorandtheconictscore.5.4.7SummaryThissectionhasshownthatevidentialinconsistency-basedmethodscanpartiallyaddresstheissueofnoticingsensinganomaliesinunknownenvironments.Section5.4.3showsthattheindi-catorstrainedfordetectionperformedpoorlywithdetectionaccuracyof28.71%to74.71%inthevericationtestbeds,indicatinganinabilitytodistinguishordinarynoisefromsensinganomalies.Theestimationandisolationindicators,ontheotherhand,performedwell.Section5.4.4showedthattheTBMCONFLICT,0.18indicatorachievedstatisticallysignicantcorrelationswithtruesensingerrorabove0.8inboththetrainingandvericationtestbeds.Inadditionthissingleindi-catorshowedtheabilitytoestimateerrorinSICKLMSreadings,Canestarangecamerareadings,andwhenbothsensorswereusedinterchangeably.ThesamewastrueofTBMCONFLICT,0.38forisolationofpoorlysensedregionswithapeakoverlapbetweenerroneousandsuspectcellsof0.8forlaserreadings.Theseresultsindicatethatasinglemethodcanbeusedforbothtasks,butthatdifferentthresholdvaluesarerequiredtolteroutminorinconsistencyintheoverallassess-mentforsensingaccuracyversusdeterminingifaspeciccellwaspoorlysensed.Afollow-upstudywasconductedtoaddressthehypothesisthatpoorperformanceofthetrainedindicatorsfor136

PAGE 151

detectionofsensinganomaliesinthisphasewasduetounexpectedlyhighlevelsofsensornoiseandposeestimationi.e.localizationerrors.Theresultsshowthatthefailureofdetectionforlaserreadingswasduetosubstantialposeestimationerrors,andforCanestarangecamerareadingswasduetosensornoiseandsensinganomaliesthatproducedconsistentbuterroneousreadings.5.5UtilityPhaseThisphaseoftheexperimentsexploresapplicationsoftheknowledgegainedfromtrainedsensingaccuracyindicators.Fordetectionanapplicationtoguidesensorselectioni.e.switchsensorswasexplored.Thisanalysisusedrealsonarandlaserdatacollectedinindoorenviron-mentsbyaNomad200tosimulateswitchingsensorsforofineevaluation.Thesamequantita-tivemapqualitymetricErrorusedinthetrainingphasewasusedtodeterminethetruesens-ingaccuracy.TheadhocclassicationapproachfromthefeasibilitystudywasusedfordetectionSection5.5.1.Section5.5.2describesmodicationstoStudent'st-testusedtodetermineifim-provementsinsensingaccuracyaresignicanttocompensatefortime-dependencebetweensam-pleswithinandbetweendatacollectionruns.Forisolationtheimprovementsinsensingaccuracyachievablebysimplyresettingsuspectregionsweredetermined.Forestimationatrainedindicatorwasappliedtotheproblemofrankingsensorsbytheirrelativeaccuracywithoutrelyingonapri-oriinformation.ForthelattertwocomponentstheindicatorswereappliedtotheSICKLMSandCanestarangecamerareadingscollectedinallveoftheATRV-Jr'stestbeds.TheresultsfortheapplicationsofthedetectioncomponentinSection5.5.3andisolationcom-ponentinSection5.5.4showedthatstatisticallysignicantimprovementsinmapqualitywereachievedusingtwosimplestrategiesforimprovingsensingaccuracy.Bothmethodsdidwell.Switchingtoamoreaccuratesensorwhenananomalywasdetectedshowedimprovementsofupto75.86%byswitchingfromthesonartotheSICKPLSwhentheANXIETY,0.86indica-tordetectedasensinganomaly.Inonecase,thesameindicatorproduceda40.52%improvementinsensingaccuracycomparedtoblindlytrustingthemostsophisticatedsensor.Resettingsuspectregionsofanoccupancygridproduceda57.65%improvementinmapqualitybyusingtheTBMCONFLICT,0.38indicatortoresetsuspectregionsoftheoccupancygrid.Theapplicationofatrainedestimationindicator,namelyTBMCONFLICT,0.18,totheprob-lemofrankingsensorsbytheirrelativeaccuracydidnotworkwell.Inspiteofitsabilitytoes-137

PAGE 152

timatesensingaccuracy,theindicatorwasunabletoreliablydeterminethemostaccuraterangesensorinthesuite,althoughitcouldidentifytheleastaccuratesensor.TheleastaccuratesensorasmeasuredbytheerrorintheoccupancygridmapwasrankedrstbyTBMCONFLICT,0.18atmostvetimesoutof110classications.Acomparisonoftheoverallaccuracywhenthemostaccuratesensorwasusedversustheindicator'stoppickrevealedlittledifferenceinerror,showingthattheindicatoroftenselectedasensorthatwasnearlyasaccurateasthemostaccuratesensor.5.5.1GroundTruthSensingAccuracyAssessmentTheexperimentsdescribedinthissectionusedthesamequantitativemapqualitymetricasthepriorphases,namelyError,butreliedontheadhocapproachtoclassifythesensingsituationusedinthefeasibilitystudyseeChapter4.Toreview,ErrorquantiedthetruesensingaccuracybymeasuringthedifferencebetweentheoccupancygridgeneratedfromthesensorreadingsandgroundtruthoccupancygridsseeSection5.3.2.FixedthresholdsofError210forDempster-ShaferandError410forSmets'TBMgridswereusedtoautomaticallyclassifythestatusofthesensorsforofineanalysis.Thesevalueswereselectedbasedonapost-hocexaminationoftheErrorscoresfortheNomaddatacollectionrunswherethetruesensingstatusnormaloranomalouswasknownapriori.ThisclassicationapproachwasonlyappliedtotheNomaddatacollectionrunsinSection5.5.3,whereasimplestrategytoimprovesensingaccuracybyswitchingsensorswhenananomalywasdetectedwasexplored.5.5.2StatisticalAnalysisTheanalysesemployedStudent'st-test,withmodicationssuggestedinDawdyandMatalas,1964tocompensatefortime-dependencewithinsamplesofthepairedErrorscoresexaminedinthisphaseoftheexperiments.Student'st-testwasusedtodetermineifindicator-drivenadap-tationstosensingproblemsproducedstatisticallydifferentErrorscores.Errorscorestendedtoincreaseasnewreadingswereappliedtoanoccupancygrid,resultingintime-dependencebetweensamplestakenineachrun.Forthesamereasonsamplestakenatthesamedistancebetweenrunswithinagiventestbedtendedtoproducesimilarvalues.Tocompensateforthesedependencies,samplesizeswereadjusteddownbyapplyingthesameapproachtondcorrectedsamplesizesfor138

PAGE 153

Pearson'slinearcorrelationanalysisinthetrainingandvericationphasesoftheexperiments.FordetailsseeSection5.3.3.5.5.3ApplyingDetectiontoSwitchSensorsThissectiondescribesastudywhichappliedtrainedindicatorstoimprovesensingaccuracybyswitchingsensorswhenananomalywasdetected.InthiscasetwoindicatorsANXIETY,0.86andGAMBINO,3.0wereidentiedinthefeasibilitystudyseeChapter4fordetectionofsens-inganomaliesforsonarandlaserreadingscollectedbyaNomad200robot.Toexplorethetrainedindicators'potentialsimulationswereperformedinwhichthesensingmanagerswitchedsensorsfromthesonartothelaserorviceversawhenananomalywasdetected.TheErrorscorewasusedtocomparethesensingaccuracyinthisscenarioiscomparedtoabaselinescenarioinwhichdetectionwasdisabled.TheresultsshowthatthewarningsredbytheANXIETY,0.86indicatorproducedstatisti-callysignicantimprovementsof73.6%narrowtestbedand75.8%widetestbedwhenasens-inganomalywaspresentandamoresuitablesensori.e.thelaserwasavailable.TheANXIETY,0.86indicatoralsoproduceda40.52%improvementoverthenavestrategyinwhichthemostso-phisticatedsensorlaserisalwaysused.TheGAMBINO,3.0indicatordidnotperformaswellduetorelativelatencyindetectionandatendencytoremultiplewarnings,inducingoscillationsbetweensensors.5.5.3.1MethodToquantifytheimprovementsinsensingaccuracyachievablebyrespondingtodetectedsens-inganomaliesaresponsescenariowassimulatedinwhichadetectedanomalycausedtheofineanalysissystemtoswitchsensors,fromthesonartothelaserorviceversa.TheErrorscorefromthisscenariowascomparedtoabaselinescenarioinwhichdetectionwasdisabled.Student'st-testwasappliedtodetermineifanydifferencesintheErrorscoreswerestatisticallysignicant.Sensinganomaliesweredetectedwhenanoccupancygrid'sconictscoreexceededtheindica-tor'smap-levelthresholdseeSection3.4.Notethattheindicatorsimplementedforthisstudydidnotusethegenericapproachfordetectionofsensinganomaliesusedinthetrainingorvericationphasesoftheexperiments.InsteadtheseusedtheadhocmethoddevelopedinpriorworkCarl-139

PAGE 154

sonandMurphy,2006andusedinthefeasibilitystudydescribedinChapter4.Toreview,themap-levelthresholdwasapplieddirectlytotheconictscoreandwassetto75%oftheindicator'sassignedthresholdmultipliedbythenumberofsuspectcells.Onceawarningwasred,themap-levelthresholdwouldincreasefromtheautomaticallydeterminedvaluetotheconictscorewhichtriggeredthewarning.Thishelpedtoreducetheimpactoffalsepositivesbyenablingtheindica-tortoswitchbacktoasuspectsensorandpreventingoscillationcausedbyringawarningatthesamevalue.Intheofineanalysis,sonarandlaserreadingswereappliedtooccupancygridswhichwereevaluatedforsensingaccuracyeveryhalfmeterusingtheErrorscore.Foreachrunreadingswereappliedtotheoccupancygriduntiltherobothadtraveledonemeter,atwhichpointthegridwasevaluated.Thegridwasreevaluatedeverysubsequenthalfmeterproducing10samplesperrun.Eachrunwasperformedinabaselinescenariowithdetectiondisabledandthescenariowithdetectionenabled;thatis,eithertheANXIETY,0.86ortheGAMBINO,3.0indicatorcheckedtheoccupancygridaftereveryreadingwasregisteredandredawarningifananomalywasde-tected.ThisprocesswasperformedusingthesonarthenthelaserastheinitialsensorresultingintwopairedErrorscoresforatotalof20pairedsamplesperrun.Thisprocesswasrepeatedforeachofthe45runsinthethreetestbedsutilizedbytheNomad200,resultinginpairedErrorscoresforeachof900samplesin90timeseriesrecordedforuseinpost-hocanalysis.Improve-mentinsensingaccuracywascalculatedasshowninEquation.7tocomparetheswitchingsce-narioresultswiththoseofthebaselinescenario.Student'st-testwasappliedtodetermineifthoseimprovementswerestatisticallysignicant.Improvement=baselineError)]TJ/F18 10.909 Tf 10.909 0 Td[(switchingError baselineError.75.5.3.2ResultsTheresultspresentedinthissectionsupportthehypothesisthatasensingaccuracyindicatorthatcandetectsensinganomaliescanbeusedtoimprovesensingaccuracybyshowingthattheANXIETY,0.86indicatorproducedstatisticallysignicantimprovementsof73.68%and75.86%onaveragesimplybyswitchingsensors,butonlywhenamoresuitablesensorwasavailable.TheresultsalsoshowedthatbothindicatorsredfewANXIETY,0.86tonoGAMBINO,3.0140

PAGE 155

falsealarmswhenasuitablesensorwasalreadyinuse.Bothindicatorsredrepeatedwarningswhenanappropriatesensorwasnotavailable,resultinginallbutonecaseindecreasedperfor-manceascomparedtothebaselinescenarioasthesystemwasforcedtooscillatebetweenthetwosensors.TheANXIETY,0.86indicator'sresponsesinthesecasesweremorestable,resultingina40.52%increaseinaccuracyascomparedtoblindlyusingthelasersensoralone.TheresultsofthisexperimentaregiveninFigures27af.OntheleftseeFigures27a,27c,and27earetheresultsfromexperimentswherethesonarwastheinitialsensorthusforthebaselinescenarioonlythesonarsensorwasused.OntherightseeFigures27b,27d,and27faretheresultsfromthesameexperimentswiththelaserastheinitialsensor.TherstrowofchartsseeFigures27aand27bshowthepercentofimprovementfromthebaselinemeanErrorscoretotheswitchscenariomeanforboththeANXIETY,0.86andtheGAMBINO,3.0indicators.ThesecondrowofchartsseeFigures27cand27dshowdetailedresultsfortheANXIETY,0.86indicator,includingthemeanErrorscoreforthebaselineandswitchsce-nariosasbarswiththestandarddeviationassociatedwitheachmeanvalueaswhiskers.ThelastrowseeFigures27eand27fgivesthesamedetailedresultsfortheGAMBINO,3.0indicator.Figures27afshowmapqualityimprovementsof73.68%and75.86%onaverageinthenarrowandwidetestbedsrespectivelywhentheANXIETY,0.86indicatorwasusedtoswitchfromanunsuitablesensortoanappropriateandmoreaccuratesensor.Anexaminationoftheoc-cupancygridsforthisscenariorevealedthattheANXIETY,0.86indicator'ssensitivitygreatlycontributedtotheseimprovementsascomparedtotheGAMBINO,3.0indicator,whichallowedseveralerroneoussonarreadingstoberegisteredbeforeringawarning.Thelaserresultsshowthatbothindicatorsreactedcorrectlyinthenarrowandwidetestbedswherethelaserwastheap-propriatesensor.OnlytheANXIETY,0.86indicatorredfalsealarmsinthewidetestbed.Inthesecasestheperformanceofthesonarinitiatedanimmediateswitchbacktothelaserresultinginlittleadditionalerror.02%.Theresultsfromthewindowtestbedweremoremixedbecauseneithersensorwassuitedforthatenvironment.Thesonarperformedslightlybetterthanthelaser,becausethelaserrarelydetectedtheglasswindows.Whenthesonarwastheinitialsensor,bothin-dicatorsperformedworsethanbaselinebyoscillatingbetweenthesonarandlasersensors.Whenthelaserwasusedrst,respondingtotheANXIETY,0.86indicator'swarningsimprovedthesensingaccuracyby40.52%ascomparedtothebaselinecasewhereonlythelaserwasused.141

PAGE 156

aSonarimprovement. bLaserimprovement. cAnxietySonarbaselinecomparison. dAnxietyLaserbaselinecomparison. eGambinoSonarbaselinecomparison. fGambinoLaserbaselinecomparison.Figure27.Resultsfromtheswitchscenario.142

PAGE 157

Table25.StatisticallysignicantchangesintheErrorscore. Scenario Initialsensor Indicator Testbed %Improvement Switch sonar anxiety narrow 73.68% wide 75.86% Table25liststhestatisticallysignicantchangesinErrorscoreascomparedtothebase-linescenariobasedontheresultsofat-testusingap-valueof0:01lessthana1%chancethattheErrorscorescomefromthesamedistribution.3TheANXIETY,0.86indicatorproducedstatisticallysignicantimprovementsinthenarrowandwidetestbedswheretheinitialunsuitablesensorsonarcouldbereplacedwithanappropriatemoreaccuratesensorlaser.5.5.4ApplyingIsolationtoResetSuspectRegionsThissectionexploresanapplicationofisolatingpoorlysensedregionstoimprovesensingac-curacybyresettingthoseregions,enablingnewerandpossiblymoreaccuratedatatobeused.Inthisanalysisanindicatortrainedtoisolatepoorlysensedregions,specicallyTBMCONFLICT,0.38,wasusedtoresetsuspectcells.TheErrorscorewasusedtomeasurethesensingaccuracywhensuspectdatawasremovedascomparedtoabaselinescenarioinwhichisolationwasdis-abled.TheresultsshowthatusingTBMCONFLICT,0.38toresetsuspectcellsproducedsensingaccuracyimprovementsofupto57.65%5.5.4.1MethodToquantifytheimprovementinsensingaccuracyachievablebyresettingsuspectcells,Errorscoresfromtwoscenarioswerecompared:rstanisolatingscenarioinwhichsuspectcellswereresetaftereachsensorscanwasaddedtotheoccupancygrid,andsecondabaselinescenarioinwhichresettingwasdisabled.Student'st-testwasusedtodetermineifresettingsuspectcellspro-videdstatisticallysignicantimprovementsinsensingaccuracy.Intheofineanalysis,laserandCanestarangecamerareadingswereappliedtooccupancygridswhichwereevaluatedforsensingaccuracyeveryhalfmeterusingtheErrorscoreinabase-linenocorrectionandisolatingscenariowheresuspectcellswereidentiedandresetaftereachsensorscanwasappliedtothegrid.Foreachrunreadingswereappliedtotheoccupancygridun3Notethatthesamplesizewasadjustedforautocorrelationforthecalculationofthetstatisticandprobability.143

PAGE 158

tiltherobothadtraveledonemeter,atwhichpointthegridwasevaluated.Thegridwasreevalu-atedeverysubsequenthalfmeterproducing11samplesforthelaserandeachCanestarangecam-eramode,foratotalof33samplesperrunforthebaselinescenario.Therunwasthenrepeatedwherethesensingaccuracyindicatorwaspermittedtoresetthebeliefforanycellwhosevalueex-ceededitsassignedthresholdi.e.suspectcellsaftereachscanwasaddedtothegrid,resultingin33samplesperrunfortheisolatingscenario.ThisprocesswasrepeatedinallvetestbedsutilizedbytheATRV-Jrfordatacollection,resultinginpairedErrorscoresforeachof1,518samplesin138timeseriesrecordedforuseinpost-hocanalysis.Thepercentofimprovementfromthebase-linescorewasusedtoevaluatetheresultsandisdenedasfollows:Percent Improvement=baseline)]TJ/F18 10.909 Tf 10.91 0 Td[(isolating baseline100%.8Student'st-testwithmodicationsseeSection5.5.2wasusedtodetermineiftheErrorscoreshowedstatisticallysignicantimprovementswhenthebeliefinsuspectcellswasregularlyreset.5.5.4.2ResultsTheresultsfromthisexperimentshowthatsensingaccuracyimprovementsofupto57.65%canbeachievedsimplybyresettingcellslabeledassuspectbyTBMCONFLICT,0.38whichiso-latedpoorlysensedregionswellinthetrainingandvericationphases.Table23andFigure28showtheimprovementsinsensingaccuracybrokendownbytestbed.Table23liststhenumberofsamplesnincludedintheanalysis,themeanErrorscoresfortheisolatingandbaselinesce-narios,andthepercentofimprovementforeachtestbed.Figure28showsthemeanandstandarddeviationerrorbarsoftheErrorscoreforeachtestbedforthebaselinedashedandiso-latingsolidscenarios.Improvementsinsensingaccuracyrangedfrom40.22%to57.65%.DuetosmallsamplesizesandvarianceintheErrorscoreswithineachtestbedseeFigure28,onlysidewalkshowedastatisticallysignicantimprovement.Visualinspectionofoccupancygridsinthebaselineandisolatingscenariosrevealedaninter-estingresult:bothTBMCONFLICT,0.38andCON,5.0isolatedaglasssurfaceusuallymissedbythelaser.OncethisresultwasfoundfortheTBMCONFLICT,0.38indicatorwhichusesSmets'transferablebeliefmodel,theanalogousmethodforDempster-Shafermodelsi.e.CONwasexaminedasabasisofcomparison.Bothmethodsisolatedtheglasssurface.Intheisolating144

PAGE 159

Table26.ImprovementsinsensingaccuracyachievedbyusingtheTBMCONFLICT,0.38in-dicatortoresetsuspectcellsbrokendownbytestbed.ydenotesastatisticallysignicantchangep=0:05. MeanError Testbed n Isolating Baseline %Improvement Lab 264 92.87 219.31 57.65% Walkway 264 55.97 122.83 54.44% Bridge 330 74.53 166.26 55.17% Cubicle 330 70.44 157.42 55.25% Sidewalk 330 58.76 98.30 40.22%y Figure28.MeanandvarianceinErrorscoresforthebaselinedashedandisolatingsolidsce-nariosbrokendownbytestbed.145

PAGE 160

aBaselinelasergridforCON. bCON-basedimprovement. cBaselinelasergridforTBMCONFLICT. dTBMCONFLICT-basedimprovement.Figure29.Exampleofisolation-basedadaptationtoasensinganomaly.Twolargeglassdoorstotheleftoftherobottopofimageinlabwerethesource.Occupiedcellsaredepictedasred,emptyasgreen,unknownasblue,andconictingasblack.scenariothisenabledtherobottodetecttheglassbyresettingallbutthelastreadingregisteredtothosecellsasshowninFigure29.5.5.5ApplyingEstimationtoRankSensorsByRelativeAccuracyThissectionexploresanapplicationofestimatingsensingaccuracytorankasetofalternativesensorsbytheirrelativeaccuracywithoutrelyingonaprioriinformationabouttheenvironment.Inthisanalysisanindicatortrainedtoestimatesensingaccuracy,specicallyTBMCONFLICT,0.18,wasused.Theresultsshowthattheindicatorcannotfaithfullychoosethemostaccuratesensorbutcanidentifytheleastaccuratesensor.Toquantifytheaccuracyofranking,theErrorscoresfromthreescenarioswerecompared:bestinwhichthemostaccuratesensoraccordingtotheErrorscorewasused,selectedwheretheleastinconsistentsensoraccordingtoTBMCON-FLICTwasused,andworstinwhichtheleastaccuratesensoraccordingtoErrorwasused.Theresultsshowedatmosta24.59%dropinaccuracybetweenthebestandselectedscenariosandsta-tisticallysignicantimprovementsofupto82.46%intheselectedscenarioascomparedtoworst.5.5.5.1MethodTodetermineifaninconsistency-basedsensingaccuracyindicatortrainedtoestimatesens-ingaccuracycanalsobeusedtorankasetofsensorsbasedontheirrelativeaccuracy,theErrorscoreandconictscorefortheindicatorwereusedtoranksensors.Therankingswerecompareddirectlytodeterminehowoftentheindicatorandgroundtruthagreedonthetoprankedsensor,theworstrankedsensor,etc.Therankingswerealsocomparedthroughsimulationsinwhichtheac-146

PAGE 161

tivesensorwasautomaticallyswitchedeveryhalfmetertooneofthefollowing:thebestsensoraccordingtothegroundtruth,theworstsensoraccordingtothegroundtruth,andtheselectedtoprankedsensoraccordingtothesensingaccuracyindicator.Torankthesensorsaccordingtotheirtrueaccuracyandthatestimatedbyasensingaccuracyindicator,laserandCanestarangecamerareadingswereappliedtooccupancygridswhichwereevaluatedforsensingaccuracyeveryhalfmeterusingtheErrorscoreandtheconictscore.Foreachrunreadingswereappliedtotheoccupancygriduntiltherobothadtraveledonemeter,atwhichpointthegridwasevaluated.Thegridwasreevaluatedeverysubsequenthalfmeterproduc-ing11samplesforthelaserandeachCanestarangecameramode,foratotalof33samplesperrun.Thegroundtruthrankingwasdeterminedbysortinglaser,theCanestarangecamerainnor-malmode,andtheCanestainmolemodeaccordingtotheirrespectiveErrorscores,inascendingorder.Thesensingaccuracyindicator'srankingwasperformedinthesamemannerexceptthatthesensorsweresortedaccordingtotheirconictscore.Thisprocesswasrepeatedforall46runsintheexploratoryandvericationtestbeds,resultingin506storedrankings.Oncetherankingsweregenerated,theofineanalysissystemwasusedagaintoquantifytheutilityoftheindicator'srankingwhenthesensorstoberankedmaybesimilarintermsoftheirac-curacy.Thestoredrankingswereusedtoautomaticallyswitchsensorsattheevaluatedlocationstosimulate:thebest-casescenarioinwhichthemostaccuratesensorwasused,theworst-casescenarioinwhichtheleastaccuratesensorwasused,andtheselectedscenarioinwhichthetoprankedsensoraccordingtotheindicatorwasused.Student'st-testwithmodicationsseeSec-tion5.3.3wasusedtodetermineiftheErrorscoreshowedstatisticallysignicantdifferencesbetweenthebestandselectedscenariosandtheworstandselectedscenarios.5.5.5.2ResultsTheresultspresentedinthissectionshowthattheTBMCONFLICT,0.18indicator,inspiteofitsabilitytoestimate,didnotrankwell.Theindicatorrarely.55%orlessforoveronehun-dredclassicationsselectedtheleastaccuratesensorandforthreeoutofveofthetestbedsse-lectedthebestsensorwithabove80%accuracy.Toquantifytheaccuracyofranking,theErrorscoresfromthreescenarioswerecompared:bestinwhichthemostaccuratesensoraccordingtotheErrorscorewasused,selectedwheretheleastinconsistentsensoraccordingtoTBMCON-147

PAGE 162

Table27.ComparisonofthegroundtruthandtheTBMCONFLICT,0.18indicator'srankingofthesensors,brokendownbytestbed. Testbed n AllCorrect BestChosen WorstChosen Lab 88 97.73% 97.73% 0.00% Walkway 88 18.18% 18.18% 2.27% Bridge 110 18.18% 18.18% 0.00% Cubicle 110 100% 100% 0.00% Sidewalk 110 54.54% 80.91% 4.55% Overall 506 57.71% 63.44% 1.38% FLICTwasused,andworstinwhichtheleastaccuratesensoraccordingtoErrorwasused.TheresultsshowedlittlechangebetweenErrorscoresinthebestandselectedscenariosandstatisti-callysignicantimprovementsintheselectedscenarioascomparedtoworst.Table27showstheresultsfromcomparingthegroundtruthandTBMCONFLICT,0.18rankingdirectlywhichshowsthattheindicator'sabilitytorankisunreliable.TheresultsaregivenintermsofthepercentageofrankingsinwhichbothwereincompleteagreementAllCorrect,thepercentageinwhichtheyagreedonthetoprankedsensorBestChosen,andthepercentageinwhichtheindicatorplacedtheleastaccuratesensorinthetoppositionWorstChosen.There-sultsshowthatTBMCONFLICTrarelychosetheleastaccuratesensoratmost5of110cases.TheAllCorrectandBestChosenresultsvariedbytestbed.Theindicatorachievedveryaccuraterankingabove97%correctrankingsinthelabandcubicletestbeds,moderatelyaccurateranking.91%agreementonthetoprankedsensorinthesidewalktestbed,andpoorrankingaccuracyinthewalkwayandbridgetestbeds.AdetailedexaminationoftherankingsinthewalkwayandbridgetestbedsshowedthattheindicatorfavoredtheCanestarangecamerainmolemodeovernormalmodewhentheErrorscoredidtheopposite.InthewalkwaytestbedthisislikelyduetothefactthatCanestarangecam-eramolesensinganomalieswerecausedbyoverexposure,sometimesresultinginconsistentbuterroneousreadingsatthesensor'slocationseeFigure20.InthebridgetestbedtheCanestarangecamerainmolemodedidabetterjobofdetectingthelargeglasswindows,butduetosensornoisegeneratedalargerErrorscoreseeanexampleinFigure30.Table28andFigure31showtheresultswhentherankingsareusedtoautomaticallyswitchsensorsevery0.5meterinthebest,selected,andworstscenarios,brokendownbytestbedandoverall.Table28showsmeanErrorscoresfortheeachscenarioandthepercentofchangein148

PAGE 163

aBridgetestbed. bGroundtruth. cCanestarangecamerainnormalmode. dCanestarangecamerainmolemode.Figure30.Exampleofranking-basedadaptationtoasensinganomaly.Largeglasswindowstotherightoftherobotbottomofimageinbridgewerethesource.Occupiedcellsaredepictedasred,emptyasgreen,unknownasblue,andconictingasblack.Table28.ComparisonofmeanErrorscoreswhenthebest,worst,ortheTBMCONFLICT,0.18indicator'stoprankedsensorisusedbrokendownbytestbed.Statisticallysignicantchangesp=0:05aremarkedwithy. Errorscore %Change Testbed Best Selected Worst Best-Selected Worst-Selected Lab 99.97 107.70 397.71 -7.73% 72.92%y Walkway 70.62 80.12 198.39 -13.44% 59.62%y Bridge 76.71 95.56 323.13 -24.59%y 70.42%y Cubicle 54.02 54.02 307.99 0.00% 82.46%y Sidewalk 68.24 70.87 132.36 -3.86% 46.46%y Overall 72.92 80.59 269.64 -10.51%y 70.11%y Errorbetweenthebestandselectedscenariosbest)]TJ/F41 7.97 Tf 6.587 0 Td[(selected best100%andbetweentheworstandselectedscenariosworst)]TJ/F41 7.97 Tf 6.587 0 Td[(selected worst100%.Atwo-tailedt-testatap-valueof0:05i.e.lessthana5%chancethattheErrorscorescomefromthesamedistributionwasusedforstatisticalsigni-cance,andpassingchanges4aremarkedwithy.Figure31showsthemeanandstandarddeviationerrorbarsErrorscoresforthebestdashed,selectedsolid,andworstvariablesizeddashesscenarios.TheresultspresentedinTable28andFigure31showthatthedecreaseinaccuracywhentheestimationindicatordoesnotselectthemostaccuratesensorisminimal,especiallyascomparedto 4NotethatsamplesizecorrectionseeSection5.3.3wasappliedintheprocessofdeterminingstatisticalsigni-cance.149

PAGE 164

Figure31.MeanandstandarddeviationofErrorscoreswhenthebest,worst,ortheTBMCONFLICT,0.18indicator'stoprankedsensorisusedbrokendownbytestbed.theworst-casescenarioinwhichtheleastaccuratesensorisalwaysselected.Theresultsfromthebestandselectedscenariosarestatisticallythesameinallbutthebridgetestbed.InthistestbedtheindicatordidselectthecorrectsensorastheCanestarangecamerainmolemodewastheonlysensoravailablethatcoulddetectthelargeglasswindowsinthistestbed,butduetosensornoisetheErrorscoreformolemodewasslightlyabovethatoftheCanestarangecamerainnormalmode.IncontrasttheErrorscoresfromtheworstandselectedscenariosarestatisticallydistinctinalltestbedenvironments.5.5.6SummaryInthissection,applicationsoftheevidentialinconsistency-basedapproachfordetectingandcharacterizingsensinganomaliesinunknownenvironmentswereexploredusingtrainedindicatorsfromthefeasibilitystudyandthetrainingphaseoftheexperiments.Theresultsshowedthatsim-plescenariosforrespondingtofeedbackfromtraineddetectionandisolationindicatorsproducedstatisticallysignicantimprovementsinsensingaccuracy.Theapplicationofatrainedindicatorforestimatingsensingaccuracy,inspiteofitsabilitytoestimatethetrueerrorofsensinginthevericationphase,couldnotreliablyranksensorsaccordingtotheirrelativeaccuracy.Section5.5.3exploredtheapplicationoftraineddetectionindicators,namelyANXIETY,0.86andGAMBINO,3.0,fromthefeasibilitystudyseeChapter4andshowsthatstatistically150

PAGE 165

signicantimprovementsof73.68%and75.86%onaveragecanbeachievedsimplybyswitch-ingsensorswhenananomalyisdetected.Whenanappropriatesensorwasnotavailablei.e.thewindowtestbedtheresultswerelargelynegativeastheindicatorsredrepeatedwarnings,forc-ingthesystemtooscillatebetweenthetwosensors.TheresultsrevealthattheANXIETY,0.86indicator'sresponsesweremorestable,enoughsothatthesystemwasabletogeneratea40.52%increaseinaccuracyascomparedtoblindlyusingthelasersensoralone.Section5.5.4showedthatsensingaccuracyimprovementsofupto57.65%canbeachievedbyresettingregionsconsideredtobesuspectbyanindicatortrainedtoisolatepoorlysensedregions.TheTBMCONFLICT,0.38indicatorprovidedconsistentsensingaccuracyimprovementsacrossallvetestbedenvironmentsinwhichlaserandCanestarangecamerareadingswerecollectedbyanATRV-Jrmobilerobot.Visualinspectionofthemodiedoccupancygridmapsrevealedthatthissimplestrategyenabledthelasersensortodetectglasssurfacesthatweremissedinthebase-linescenariowherenocellswerereset.Notethatthisstrategyisnotalwaysappropriateasnewreadingsmaybelessaccuratethanpriorones,soresettingsuspectcellsmaynotalwaysproducemoreaccurateresults.Section5.5.5exploredtheuseofatrainedestimationindicator,namelyTBMCONFLICT,0.18,torankthesensorswithinasensingsuitebytheirrelativeaccuracy.Unfortunately,thisin-dicatorwasunabletoreliablydeterminethemostaccuraterangesensor,butitcouldidentifytheleastaccuratesensoroutofthree.Theleastaccuratesensorasmeasuredbytheerrorintheoccu-pancygridmapwasrankedrstbyTBMCONFLICT,0.18atmostvetimesoutof110classi-cations.Acomparisonoftheoverallaccuracywhenthemostaccuratesensorwasusedversustheindicator'stoppickrevealedlittledifferenceinerror,showingthattheindicatoroftenselectedasensorthatwasnearlyasaccurateasthemostaccuratesensor.5.6ConclusionsThischapterhasdescribedexperimentsusingrealrangesensorreadingscollectedbymobilerobotsinunclutteredstaticenvironmentstodetermineiftheevidentialinconsistency-basedap-proachpresentedinthisdissertationcandetectandcharacterizesensinganomaliesinunknownenvironments.SixmethodsforquantifyinginconsistencyinevidentialmodelswereexaminedseeSection3.3.2:Yager'sANXIETY,Shafer'sCON,Smets'transferablebeliefmodelTBMCON-151

PAGE 166

FLICT,Pal'sINCONSISTENCYmetric,LIU'SCONFLICTmetric,andGAMBINO'schangeheuristic.Indicatorswereformedbyassociatingamethodwithaspecicthresholdvaluemethod,valueusedtodistinguishordinarynoisefromsensinganomalies.Theseindicatorswereevaluatedfortheirabilitytodetectsensinganomalies,estimatesensingaccuracy,andisolatepoorlysensedre-gionsbyapplyingtrainedindicatorsforuseinnewenvironments.Oneunclutteredindoorhallwayandoneoutdoorsidewalkservedastrainingtestbedsandsensordatacollectedintheseenviron-mentswereusedtondthemostpromisinginconsistency-basedsensingaccuracyindicatorsin-consistencymethodcoupledwithathresholdtodistinguishnoisefromanomaliesforestimatingsensingaccuracy,detectingsensinganomalies,andisolatingtheenvironmentalsourcesofsens-inganomalies.Todetermineifasensingaccuracyindicatorselectedbasedonitsperformanceinaknownenvironmentcouldbeusedinanewenvironment,sensordatacollectedinthreeadditionaltestbedstwoindoorandoneoutdoorwereusedtoverifytheperformanceofthemostpromis-ingindicatorsforeachtask.TheresultsprovideevidencethattheapproachdescribedinChapter3can:Estimatesensingaccuracyinunknownenvironments.TBMCONFLICT,0.18providedac-curateestimatesofaquantitativemapqualitymetricwithstatisticallysignicantcorrelationsabove0.9inboththetrainingandvericationtestbeds.Isolateenvironmentalsourcesofsensinganomaliesinunknownenvironments.TBMCON-FLICT,0.38showedtheabilitytoisolatepoorlysensedregionsinboththetrainingandver-icationtestbedswithoverlapbetweenerroneouscellsandthoseconsideredsuspectbytheindicatorabove0.6ascomparedtoanoverlapof0.01whencellsarerandomlyclassied.Enablestatisticallysignicantimprovementsinsensingaccuracy.Statisticallysignicantimprovementsofupto75.86%wereachievedbyswitchingfromthesonartotheSICKPLSwhentheANXIETY,0.86indicatortrainedinthefeasibilitystudyseeChapter4detectedasensinganomaly.Inadditionimprovementsofupto57.65%wereachievedbyusingthetrainedTBMCONFLICT,0.38indicatortoresetsuspectcells.Theresultsdonotsupportthehypothesisthattheapproachcan:152

PAGE 167

Ranksensorsaccordingtotheirrelativeaccuracyinunknownenvironments.TBMCON-FLICT,0.18failedtoreliablyranksensorsbytheirrelativeaccuracyinspiteofitsabilitytoestimatesensingaccuracy.Detectsensinganomaliesinsomeenvironments.Performanceofthemostpromisingindi-catorsbasedonLIU'SCONFLICTandTBMCONFLICTonthedetectiontaskvariedwidelyacrosstestbedsfrom28.71%to92.23%accuracy.Afollow-upexperimentrevealedthatfail-ureofdetectionforlaserreadingswasduetosubstantiallocalizationerrorsandforCanestarangecamerareadingswasduetosensornoiseandthepresenceofsensinganomaliesthatpro-ducedconsistentbuterroneousreadings.Basedontheseresults,thisdissertationprovidestherstknowngeneralapproachforsensingas-sessmentrelyingsolelyonfusedsensorreadingsfromasinglesensorseeChapter2.Chapter6willprovidefurtherdiscussiontoputtheseresultsintocontext.153

PAGE 168

Chapter6DiscussionThischapterprovidesadetaileddiscussionoftheexperimentspresentedinChapters4and5andthecontributionsoftheapproachfordetectingandcharacterizingsensinganomaliesinun-knownenvironmentspresentedinthisdissertation.Itistherstknownapproachexplicitlyde-signedtodetectandcharacterizesensinganomaliesandtherstknowngeneralapproachcapableofestimatingsensingaccuracyandisolatingpoorlysensedregionswhenonlyreadingsfromasinglesensorareavailable.Sensinganomaliesarecasesinwhichphysicalsensorsareworkingwithinthemanufacturer'sspecicationsbutthereadingswouldleadtoanincorrectinterpreta-tionoftheenvironment.Forexamplesomenear-IRrangesensorscannotdetectdarksurfacesthatabsorbinthenear-IRrange.Thisworkprovidesasituatedintelligentagentwiththeabilityto:Estimatetheaccuracyofasensorinthecurrentsensingcontext,Isolatepoorlysensedregionswithinanunknownenvironmentrelyingsolelyonfusedsensorreadingsandtheassumptionthatthetruestateoftheworldiscon-sistent.Thisapproachwillimproveasituatedintelligentagent'sabilitytomanagetheaccuracyoftheinformationitgathersintheunknownandenablethedevelopmentofincreasinglyautonomousrobotsforapplicationslikespaceexploration,searchandrescue,andmilitaryoperationswhereaprioriinformationissparse.Thischapterisorganizedasfollows.Section6.1comparestheexperimentalresultspresentedinChapters4and5anddiscussesimportantdifferencesintheexperimentalmethodsusedineachsetofexperiments.Section6.2discussesthecontributionsofthisdissertationtotheeldsofro-boticsanduncertaintyinarticialintelligencecommunities.Section6.3discussesofthelimita-tionsoftheexperiments.Section6.4providesrecommendationsforadditionalworktowarden-ablingintelligentsituatedagentstoadapttosensinganomaliesinrealtime.FinallySection6.5providesasummaryofthischapter.154

PAGE 169

Table29.Comparisonofsensingaccuracyestimationresultsfromthefeasibilitystudyandtrain-ingphaseoftheexperiments.TheperformanceisgivenasPearson'slinearcorrelationcoefcientr.CanestareferstotheCanestarangecamera. FeasibilityStudy Experiments SICKPLSand SICKLMSand SICKLMS Canesta Method Sonar Canesta TBMCONFLICT 0.6379 0.9674 0.9866 0.8399 GAMBINO 0.7061 0.9695 0.9828 0.8412 LIU'SCONFLICT 0.6274 0.9603 0.9560 0.8477 INCONSISTENCY 0.8735 0.7193 0.8034 0.5536 ANXIETY 0.8538 0.7048 0.8177 0.4490 CON 0.3598 0.5551 0.6442 0.0511 6.1ImplicationsoftheExperimentalResultsConsideringboththefeasibilitystudyfromChapter4andthein-depthexperimentsdescribedinChapter5theresultsshowthattheevidentialinconsistency-basedapproachdevelopedforthisdissertationcanpartiallyaddresstheissueofnoticingsensinganomaliesinunknownenviron-ments.Theresultsalsoindicatethatdifferentsensorswillrequiredifferentmethods,thusfutureapplicationswithnewsensorswillneedtoperformtrainingrunsinknownenvironmentstondtherightindicatorsforasensor.Table29providesasummaryoftheresultsforestimationofsensingaccuracyforbothsetsofexperiments.TheresultsarethecoefcientsrfromPearson'slinearcorrelationanalysisforthetrainedindicators,oneforeachofthesixmethodsforquantifyinginconsistencyinevidentialmodelsfromtheevidentialliteratureexaminedinthiswork.Theanalysismeasuredhowwelleachmethodleft-mostcolumnestimatesaquantitativemapqualitymetricwhichmeasuresthedifferencebetweensensor-basedandgroundtruthoccupancygrids.Theresultsarebrokendownintofoursetsofsensorsforwhichindicatorsweretrained:sonarandSICKPLSsensors,SICKLMSandaCanestarangecamera,SICKLMSonly,andtherangecameraonly.Indicatorstrainedforapairofsensorscouldbeusedwitheithersensorinterchangeably.Theirperformancewhenappliedtothefusedresultofbothsensorswasnotmeasured.Table29showsthatdifferentmetricsestimatewellfordifferentsensors.Forthesonar,SICKPLS,andSICKLMS,metricsrelatedtoprobabilisticentropy,namelyYager'sANXIETYYager,1982andPal'sINCONSISTENCYPal,1999,providedaccurateestimatesofsensingaccuracy155

PAGE 170

Table30.Comparisonofsensinganomalydetectionresultsfromthefeasibilitystudyandtrain-ingphaseoftheexperiments.Theperformanceisgivenasthepercentageofcorrectlyclassiedoccupancygridsoutof900forthefeasibilitystudyand241,680fortheexperiments. FeasibilityStudy Experiments SICKPLSand SICKLMSand SICKLMS Canesta Method Sonar Canesta ANXIETY 99.67% 39.48% 49.06% 24.75% GAMBINO 96.33% 76.23% 72.34% 82.24% LIU'SCONFLICT 70.11% 84.37% 92.98% 82.50% CON 68.89% 79.42% 78.66% 83.40% TBMCONFLICT 68.56% 78.30% 75.20% 83.39% INCONSISTENCY 69.56% 39.50% 49.06% 24.81% withstatisticallysignicantcorrelationsabove0.8whereasthesemetricsperformedpoorlyforCanestarangecamerareadingsachievingatbest0.5536correlation.Inthetrainingphase,met-ricswhichrelyonevidentialconictfromSmets'transferablebeliefmodel,namelyTBMCON-FLICT,LIU'SCONFLICT,andGAMBINO,alsoestimatedsensingaccuracywithstatisticallysig-nicantcorrelationsabove0.95fortheSICKLMSandabove0.8fortheCanestarangecamera.Thesemetricsprovidedweakerestimatesofsensingaccuracystatisticallysignicantcorrelationsfrom0.6to0.7forthesonarandSICKPLSwhenconsideredtogether.Table30providesasummaryoftheresultsfordetectionofsensinganomaliesforbothsetsofexperiments.Theresultsarereportedasthepercentageofcorrectlyclassiedoccupancygridsforthetrainedindicatorsforeachofthesixmethodsforquantifyinginconsistencyinevidentialmod-elsconsideredinthiswork.TheresultsareagainbrokendownintofoursetsofsensorssonarandSICKPLSsensors,SICKLMSandCanestarangecamera,SICKLMSonly,andCanestarangecameraonlyforwhichindicatorsweretrained.Table30providesfurthersupportforthehypothesisthatdifferentmetricsareneededfordif-ferentsensorsbutthedetectionresultsweremoredisparateascomparedtoestimation.Aswiththeestimationresults,ANXIETYandINCONSISTENCYperformedpoorly.75%accuracyfortheCanestarangecamerabutallconict-basedmetricsdetectedsensinganomaliesequallywell%accuracyforthissensor.InthefeasibilitystudybothANXIETYandGAMBINOachievedabove96%accuracyfordetectionofsensinganomaliesforsonarandSICKPLSwhereasANXIETYperformedpoorly.06%accuracyandGAMBINOperformedonlymoderatelywell.34%accuracyfortheSICKLMSinthetrainingphase.IntheseexperimentsonlyLIU'S156

PAGE 171

CONFLICTshowedtheabilitytoaccuratelydetectsensinganomalieswith92.98%accuracyfortheSICKLMSinthetrainingtestbedsbutitsaccuracydroppedto40.46%forthesamesensorinthevericationtestbeds,aresultthatwasperhapsforeshadowedbyitsperformanceatbest70.11%accuracywith95%ofnegativeexamplesclassiedasanomaliesinthefeasibilitystudy.AkeydifferencebetweenthefeasibilitystudyandtheexpandedexperimentsdiscussedinChapter5ismorerealisticscenariosforsensinganomalydetectionconsideredinthelatter.Inthefeasibilitystudysensinganomalieswerepresentatthebeginningofeachrunanddidnotchangeoverthecourseoftherun.Theseconditionsaresimplerthanrealworldsituationswheresensinganomaliescanappearanddisappearanywherewithinanenvironment.Thein-depthexperimentsweredesignedtoensurethatzerototwosensinganomaliesappearedanddisappearedatrandompointswithineachrunbyvaryingthedistancebetweenthestartingpointaccordingtosoftware-generatedrandomvaluesandanyxedsourcesofsensinganomaliese.g.glasssurfacesandplacingmobilesourcesofsensinganomaliesdarkfelt-coveredobstaclesatrandomdistancesagainaccordingtosoftware-generatedrandomvaluesfromthestartingpoint.Anotherkeydifferencewhichinuencedthedetectionresultswastheimplementationofagenericapproachforclassifyingthesensingsituationemployedinthetrainingandvericationphasesoftheexperiments.InthefeasibilitystudyanadhocmethoddevelopedinpriorworkCarlsonandMurphy,2005wasusedtodeterminethethresholdappliedtotheconictscoretodetectsensinganomalies,specically75%ofthecell-levelthresholdmultipliedbythenumberofsuspectcells.Whilethismethodworkedwellupto99.67%accuracyfortestbedswhereananomalywasalwayspresent,itwasobservedthatananomalywouldbedetectedassoonasasin-glecellwithintheoccupancygridexceededthecell-levelthreshold.Asaresultagenericmethodwasdevelopedforthetrainingandvericationphasesoftheexperimentstobeusedbythesensingaccuracyindicatorsandthegroundtruthclassicationsystem.Thismethodassumedthaterrortol-eranceistaskdependentandthereforeappliedauser-denedthresholdbasedonanintuitivecon-ceptpercentoferrortotheratiooferroneousorsuspectcellstothetotalnumberofsensedcellswithintheoccupancygrid.Fortheexperimentsthisuser-denedthresholdwassetto14%basedontheempiricalobservationthattheautomatedsystemusedtodeterminewhensensinganomalieswerepresentseeSection5.3.2producedthemostaccurateresultsatthatthreshold.Whentheexperimentalresultsindicatedthatthisgenericmethodwasunreliableinunknownenvironments157

PAGE 172

withaccuracyrangingfrom28.71%to92.23%acrosstestbedstheoriginaladhocmethodwastried,buttheresultsprovedtobejustaspoor.INCONSISTENCYachievedthebesttrainedaccuracyof80.08%fortheSICKLMS,whichfellto58.94%inthevericationtestbeds.Overalltheresultsindicatethattheevidentialinconsistency-basedapproachdevelopedforthisworkcannotdistinguishordinarynoisefromsensinganomaliesbutitcanprovideimportantinfor-mationregardingthetrustworthinessofinformationgatheredinunknownenvironments.Sensinganomaliesaremostprevalentforexteroceptivesensorswhichprovidesituatedagentswithimpor-tantinformationregardingitssurroundings.Anassessmentofthereliabilityofinformationpro-videdbythesesensorswhichdoesnotfailinthepresenceofsensinganomaliesisessentialforthedevelopedofautonomoussystemsthatcanoperaterobustlyintheunknown.6.2ContributionsThisdissertationcontributestotheroboticsanduncertaintyinarticialintelligencecommu-nitiesbyestablishingafoundationfortheuseofevidentialmetricsforadaptingasinglesensororidentifyingthemostaccuratesensorinasuitetooptimizetheaccuracyofsensinginunknownen-vironments.Thissectiondiscussesthreespeciccontributionsofthiswork:ageneralapproachforsensingassessmentinunknownenvironmentsrelyingsolelyonreadingsfromasinglesensor,validationofevidentialinconsistencymetricsforquantifyinginconsistencyinsensorreadings,andpresentationofasystemarchitectureforautonomousidenticationofsensorsuitabilityinun-knownenvironments.Thesecontributionspavethewayforthedevelopmentofintelligentsystemsthatcanswitchinformationsourcestoensurethebestmissionperformance,identifytherelativecontributionandreliabilityofsourcesfordifferentenvironments,andreasonaboutwhichsourcestouseunderwhatcircumstances.6.2.1SensingAssessmentinUnknownEnvironmentsRelyingSolelyonReadingsfromaSingleSensorThisworkpresentstherstknowngeneralapproachcapableofestimatingsensingaccuracyandisolatingpoorlysensedregionsinunknownenvironmentswhenonlyreadingsfromasin-glesensorareavailable.Bysupplyingfeedbackforsensinginunknownenvironmentswithoutrequiringaccesstoalargenumberofsourcesi.e.sensorstoensureanaccurateconsensus,this158

PAGE 173

approachprovidesasituatedintelligentagentwithimportantinformationregardingthetrustwor-thinessofsenseddatathatisnototherwiseavailable.Anintelligentagentthatcanestimatesensingaccuracycanadaptivelyselectsensorstoensurethebestmissionperformance,identifytherelativecontributionandreliabilityofsensorsfordifferentenvironments,andreasonaboutwhichsensorstouseunderwhatcircumstances.Ifpoorlysensedregionsareisolatedwithinanunexploredenvi-ronment,applicationscouldautonomouslygeneratemapsindicatingregionswheresensorsareorarenotreliable,allocateadditionalsensingresourcestotroublesomeareas,orsimplyavoidtheseareastoensuresafenavigation.Theapproachpresentedinthisdissertationislessconstrainedthanexistingapproachesforsensingassessmentinunknownenvironments.Theseeitherrelyonmultiplesourcesorarelimitedintheirapplicability.Existingsolutionsforestimatingtheaccuracyofpeersinsensornetworksormulti-agentsystemsdependoncomparisonswithtrustedlocalreadingse.g.Momani,Challa,andAlhmouz,2008oranaccurateconsensusofmultiplesourcese.g.Ganeriwal,Balzano,andSrivastava,2008.Sincetheenvironmentalconditionswhichleadtosensinganomaliesoftenaffectmultiplesensors,theseapproacheswouldrequireanintelligentagenttoactivelyuseawidevarietyofsensorsatalltimestoensureanaccurateconsensus.Thisrequirementisunattainableformobileroboticsorsensornetworkapplicationswhereenergy,space,and/orweightcapacityareinlim-itedsupply.Inadditionsensorsareexpensiveandsimultaneouslyusingmanysensorstaxesoftenlimitedcomputationalresources,increasesthecomplexityofanintelligentsystem,andincreasesthefrequencyofsensorfaults.Anexistinginconsistency-basedapproachisolatespoorlysensedregionsbutitsapplicabilityislimitedto3DpointclouddataRomanandSingh,2006.Theap-proachpresentedinthisdissertationismoregeneral.Itdoesnotrequireanaccurateconsensusandisapplicabletoanymodeloffusedsensordata,providedthatasuitableinconsistencymetriccanbeformulatedi.e.onethatcandistinguishinconsistencyfromlackofinformation.ExperimentsseeChapter5providestatisticallysignicantevidencethatindicatorsbasedonTBMCONFLICTcanestimatetheaccuracyoflaserandCanestarangecamerasensorsandisolatepoorlysensedregionsinunknownenvironmentsrelyingsolelyonfusedreadingsfromasinglesensor.Resultsfromthevericationphaseoftheexperimentsshowedthatanindicatortrainedtoprovideaccurateestimatesofthetruesensingerrorinknownenvironments,namelyTBMCON-FLICT,0.18,maintainedstableestimationcapabilitiescorrelationswithtrueerrorabove0.8in159

PAGE 174

newenvironmentswhichsupportsthehypothesisthatthisapproachcanbeusedtoestimatesens-ingaccuracyinunknownenvironments.SimilarlytheTBMCONFLICT,0.38indicatorselectedforitsabilitytoisolateerroneousreadingsduringthetrainingphaseperformedjustaswellinthevericationphase.Overlapbetweenerroneousregionsandthoselabeledsuspectrangedfrom4.9and6.2acrossthenewtestbedsversusanoverlapof0.01whenregionsarerandomlyclassied.6.2.2ValidationofEvidentialInconsistencyMetricsforQuantifyingInconsistencyinSen-sorReadingsThisworkcontributestotheuncertaintyinarticialintelligencecommunitybyexploringtheapplicationofevidentialinconsistencymetricstodetectandcharacterizesensinganomaliesinfusedsensorreadings.Thisworkistherstknownapproachtoapplyevidentialinconsistencymetricstomeasuresensorinconsistencyforasinglesensorasopposedtoexistingmethodswhichrequiremultiplesensorstobeactive.Theseareusedinthisworktoquantifyinconsistenciesin2Dmodelsofconsistentenvironmentsfordetectionandcharacterizationofsensinganomalies.ThisrepresentstherstknownrealworldapplicationofthesemetricstoprovidefeedbackforsensinginunknownenvironmentswiththeexceptionofpriorworkbytheauthordescribedinCarlsonandMurphy,2005,2006;Carlsonetal.,2005althoughsimilarapproachesutilizingthesemet-ricshavebeenappliedtoresolveassociationissuesAyounandSmets,2001ortodetectoutdatedinformationinastaticmapGambino,Ulivi,andVendittelli,1997.6.2.3FrameworkforIdentifyingSensorSuitabilityinUnknownEnvironmentsThisdissertationalsoprovidesaframeworkforsolvingthelargerproblemofidentifyingthesuitabilityofsensorsinunknownenvironmentswhichclariesthecomplementaryrelationshipsbetweenthisworkandrelatedworkonsensingassessment:sourcereliabilityestimationseeSec-tion2.1.2andsensorFDIseeSection2.2,whileidentifyingnewavenuesforfutureresearchsensorcontexttracking.Theframeworkcombinestheworkproposedherewithexistingap-proachesfromtheroboticsandmulti-agentsystemsliterature.Learninghowsensorsperformindifferentareaswithintheenvironmentwillenableamobilerobottoforexample:160

PAGE 175

Moreeffectivelysensetheenvironmentonsubsequentpasses,Passtheinformationalongtootherrobotsinamulti-robotteam,Provideuserswithacondencemapdescribingitscertaintyregardingsensinginanygivenregionoftheenvironment,Providedesignerswithfeedbackonitssensingsuite.Section3.1introducedtheframeworkwhichenablesasensormanagertoselectthemostap-propriatesetofsensorstousebytrackingthestatusandidentifyingthesuitabilityofeachsensorineachsensingcontext,ortheenvironmentalcontextoftherobotintermsofsensorperformance,astherobotexploresanaprioriunknownenvironment.ThesensingassessmentmoduleusestheapproachpresentedinChapter3andasourcereliabilityestimationapproachseeSection2.1.2toprovideboththesensormanagerandthesensingcontexttrackingmoduleswithinformationregardingthepresenceofsensinganomalies,whichsensorsareaffectedbyanomalies,andanestimateofeachsensor'sreliability.AtraditionalsensorFDIseeSection2.2moduleprovidesthesensormanagerwithinformationonthestatusi.e.workingorfaultyoftheactivesensors.AsimultaneouslocalizationandmappingorSLAMThrun,2003bmoduleisusedtobuildanac-curatemapofthepreviouslyunexploredenvironment.Thismapservesasaguideforthesensingcontexttrackingmodulewhichprovidesthesensormanagerwiththesensingcontextbydetectingtransitionsinsensorperformance.Finally,asensormanagerMurphy,2000allocatessensorstobehaviors1tooptimizethechancesofmissioncompletion.6.3LimitationsofExperimentsTheexperimentsinChapters4and5collectivelyusedrealsensorreadingsfromfourrangesensorscollectedinsevenunclutteredindoorandoutdoorenvironments.Theseprovidestatisti-callysignicantevidencetosupportthehypothesisthattheapproachpresentedinthisdissertationcanbeusedforsensingassessmentinunknownenvironments.However,therearelimitationstotheexperimentsthatmeritdiscussion.Therstlimitationwastheuseofanoccupancygridrepresentationwhichrestrictedtheex-perimentstostaticunclutteredenvironments.Tomaintainfocusofthisworkontheproblemof 1Inroboticsabehaviorisamappingofsensorreadingstoapatternofeffectoractions.Murphy,2000161

PAGE 176

sensinganomalies,acommonlyusedoccupancygridrepresentationseerecentexamplesinCo-henandEdan,2008;Miura,Negishi,andShirai,2006wasselectedtocreate2DmapsfromrangesensorreadingsasoutlinedinMurphy,2000.Anoccupancygridisaregulargridmadeupofequallysizedcells,eachofwhichmaintainsavaluee.g.probabilityorbeliefmassindicatingwhetherthecellisoccupiedorempty.Thisrepresentationisrestrictedtostaticenvironmentsandisonlyaccurateforunclutteredspacesthatcanbeapproximatedbyxed-sizedcells.Intheexper-imentsseeChapter5thisinexiblerepresentationledtohigherthanexpectedbaselineinconsis-tencyduetominorinaccuraciesintherecordedpositionsandorientationsforthesensorslocal-izationerrorandminorvariancesordinarynoiseinthesensorreadings.Sincetheseexperimentswereconducted,moreexiblerepresentationmodelsforrangesensorreadingshaveappearedintheliteraturee.g.O'Callaghan,Ramos,andDurrant-Whyte,2009whichwouldbelesslikelytoinducesuchspuriousinconsistencies.Thesenewrepresentationmodelsareprobabilisticwhereastheinconsistencymetricsusedinthisworkarerestrictedtoevidentialmodelssocorrespondingrepresentationsusingevidentialtheoryoranalogousinconsistencymetricsforprobabilisticmodelswouldhavetobedeveloped.Thesecondlimitationwastheambiguitybetweenisolatingtheenvironmentalsourcesofsens-inganomaliesversusisolatingpoorlysensedregions;asaresultitismoreaccuratetostatebasedontheexperimentalresultsthatTBMCONFLICTcanisolatepoorlysensedregionsingeneral,notthesourcesofsensinganomaliesinisolation.Toquantifytheinconsistency-basedsensingaccu-racyindicators'abilitytoperformtheisolationtasktwosetsofcellswithinoccupancygridsbuiltfromsensorreadingswerecompared:thesetoferroneouscellsbasedonacomparisonwiththegroundtruthoccupancygrid,andthesetofsuspectcellswhoseinconsistencyexceededtheindica-tor'sassignedthreshold.Sincethesensorreadingswerecollectedinrealasopposedtosimulatedtestbedsandwereregisteredtostatic2Dmaps,anysensinganomaliesencounteredwereneverthesolesourceofinaccuraciesinthesensor-basedoccupancygrids.AsnotedinChapter3,errorisintroducedatmanypointsintheprocessofgatheringandinterpretingsensorreadings,includ-ingnoise,thesensorhardwaresamplingerror,interpretationquantizationerrorinthesensormodel,andregistrationwithinthelargermodellocalizationerroronamobilerobot.Allofthesefactorsinuencedwhichcellsfellintotheerroneousset.Duetotheseinuencesitwouldbemoreaccuratetostatethaterroneouscellsrepresentpoorlysensedregionsintheenvironment,notthe162

PAGE 177

environmentalsourcesofsensinganomaliesinisolation.AsaresultstatingforexamplethatTBMCONFLICTcanisolatetheenvironmentalsourcesofsensinganomaliesisnotasaccurateasstatingthatthismetric,whencoupledwithapre-determinedthreshold,canisolatepoorlysensedregionsinunknownenvironments.Thethirdlimitationwastheuseofalimitedsetofrangesensorsrepresentingonlypartofatypicalsensorsuiteforamobilerobot.Theexperimentsfocusedonsensorsthataremostsuscep-tibletosensinganomalies,namelyexteroceptivesensorsthatmeasurevaryingcharacteristicsoftheenvironment,andprovidereadingsthatareeasytointerpretforanoccupancygridrepresen-tationtomaintainfocusontheproblemofsensinganomalies.Realsensorreadingsfromthreedistincttypesofrangesensorswereused:sonar,laserrangender,andarangecamera.Whilethesonarandlaserrangenderarecommonsensorsintheroboticsdomain,theyrepresentonlypartofatypicalsensorsuitewhichoftenincludesoneormoreCCDcamerasandposesensorssuchaswheelorjointencoders,compasses,globalpositioningsystemGPSreceivers,and/orinertialmeasurementunitsIMUs.Mostsensingsuitesemployanalyticallyredundantposesensorswhichuseverydifferentsensingtechnologye.g.bothGPSandencodersthusexistingsensorFDIap-proachesseeSection2.2canbeusedtomakethesemorerobusttoanomalies.TheCanestarangecamerahascapabilitiesanalogoustocomputervisionsystemsbuiltfromCCDcamerasliketheabilitytoprovide3Dscansoropticalowinformationbutforsimplicitywasusedinthesamefashionasthelasersensorsbysamplingonlythemiddlerowofpixels.Theapplicationofthisap-proachtoCCDcamerasandthewealthofapplicationsthesesupportisapromisingavenueforfutureworkonthistopic.6.4Recommendations:TowardsAdaptationtoSensingAnomaliesBasedontheexperimentalresults,thisworkprovidesonlyapartialsolutiontoaddresstheproblemofsensinganomaliesinunknownenvironments.Moreworkisneededbeforeintelli-gentsituatedagentscanrelyonthisapproachtoenableadaptationtosuchanomaliesinrealtime,specically:Findingasolutionforrobustdetectionofsensinganomalies.Experimentalresultsshowedthattheapproach'sabilitytodetectsensinganomaliesistoosensitivetosensingerrorsforusewithoccupancygridbasedsensingsystems.Theseareparticularlysusceptibletoinconsistencybe-163

PAGE 178

causeminorsensorerrors,likeordinarynoiseandinaccuratepositioningofthesensorintheenvironmentlocalizationerror,producehighlevelsofinconsistencyalongoccupied/emptybordersduetotheuseofxed-sizedgridcells.Whilemethodsexisttoreducelocalizationer-rorsinunknownenvironmentsseeThrun,2003btheseassumethatsensingisaccurateoratleastconsistentandmaynotconvergetosufcientlyaccuratelocalizationinthepresenceofsensinganomalies.Combinationsofmethodsforquantifyinginconsistencymayprovidemorerobustresults.ThesecouldbelearnedbyboostingFreundandSchapire,1996withsens-ingaccuracyindicatorstrainedfordetectioni.e.usingthebestperformingthresholdforeachmethodservingasweakclassiers.Allpotentialsolutionsshouldbetestedinnewenviron-mentstovalidatetheirapplicability.Validationofthisapproachforrealtimeadaptationonamobilerobot.Theapproachpre-sentedinChapter3providesanefcientlineartimeandspacecomplexitysolutiontotheproblemofsensinganomaliesinunknownenvironments.Todatethisapproachhasonlybeenvalidatedinofineexperimentsusingrecordedsensordata.Onceasolutionfordetectionisfound,developmentofanonlinerealtimeversionwouldrequireonlytheimplementationofoneormoreclosedforminconsistencymetricsandasystemtoperiodicallyapplythesetothefusedreadingsandrespondtotheresults.Developinganapproachforndingthemostpromisingindicatorsforanewsensorsuite.Sincetheresultssuggestthatdifferentsensingsystemswillrequiredifferentsensingaccu-racyindicators,applyingthisapproachtoanewsensorsuitewillinvolvendingtherightin-dicators.BoostingFreundandSchapire,1996mayserveasastraightforwardmethodforndingthemostpromisingindicatorsfordetectionofsensinganomalies.Similarapproachesforestimationofsensingaccuracyandisolationneedtobedeveloped.Determiningifanalogoussolutionscanbedevelopedformoreprevalentprobabilisticmod-els.Probabilisticapproachesarefarmorecommonintherobotics,multi-agentsystems,andsensornetworksliteratureseeChapter2thanevidentialmodelsbutatpresenttheapproachislimitedtothelatter.CobbandShenoy,2006haveshownthatthesetwoquantitativetheoriesofuncertaintyareequallyexpressiveandthattransformsbetweenthemcanbedevel-oped.Thesetransformsshouldbestudiedindetailtodetermineiftheevidentialmethodsfor164

PAGE 179

quantifyinginconsistencyemployedinthisapproachhaveanalogousmethodsinprobability.Ifsuchmethodsexist,theircomputationalcomplexityshouldbeexaminedtodetermineiftheycanbeappliedinrealtime.Implementingtheapproachasitappearsinthisdissertationrequiresthecollectionofsensorreadingsinknownenvironmentstotrainthemethodi.e.ndtheinconsistencymetricandthresh-oldthatworksbestforaparticularlogicalsensori.e.physicalsensoranditsinterpretationsys-tem.Theselectionofenvironmentsfordatacollectionshouldfollowtheseprinciples:eachsetshouldrepresentenvironmentalconditionsknowntoproduceanomaliesforthetargetsensors,thetestbedsshouldprovidelongstraightpathsfortherobottotraversetominimizeposeestimationerror,andthesetoftestbedsshouldexposethesensorstovaryingenvironmentalconditionswhichaffectsensingaccuracy,e.g.ambienttemperatureaffectstheaccuracyofnear-infraredbasedsen-sors.Readingsfromafewoftheenvironmentsshouldbeexcludedfromthetrainingsetsotheycanbeusedtoverifytheeffectivenessofthetrainedsolutions.6.5SummaryThischapterhasdiscussedtheexperimentalresultspresentedinChapters4and5,thecon-tributionsofthisdissertation,thelimitationsoftheexperiments,andrecommendednextstepsto-wardenablingintelligentsituatedagentstoadapttosensinganomalies.Section6.1comparedtheexperimentalresultsfromthefeasibilitystudyChapter4andtrainingphaseChapter5whichsupportthehypothesisthatdifferentsensorsrequiredifferentindicatorse.g.Yager'sANXIETYestimatessensingaccuracyforsonarandlaserbutnotCanestarangecamerareadingsfordetect-ingandcharacterizingsensinganomalies.Thissectionalsodiscussedthekeydifferencesbetweenthefeasibilitystudyandtheremainingphasesoftheexperimentsasfollows:xedversusvaryingpresenceofsensinganomaliesandtheuseofagenericclassicationapproachfordetectingsens-inganomalies.Threecontributionswereidentiedanddiscussed:Sensingassessmentinunknownenvironmentsrelyingsolelyonreadingsfromasinglesensor,Validationofevidentialinconsistencymetricsforquantifyinginconsistencyinsensorreadings,Providingaframeworkforidentifyingsensorsuitabilityinunknownenvironments.165

PAGE 180

Section6.3describedthelimitationsoftheexperimentalapproachincluding:theuseofaregularevidentialgridoccupancygridrepresentation,ambiguityinthemethodbetweenisolatingtheen-vironmentalsourcesofsensinganomaliesversusisolatingpoorlysensedregions,andtheuseofalimitedsetofrangesensors.Section6.4describesthestepsrequiredtoapplytheapproachpre-sentedinthisdissertationtoanewsensingsuiteandprovidedfourdirectionsforadditionalworkneededtoenablerealtimeadaptationtosensinganomalies:ndingasolutionforrobustdetec-tionofsensinganomalies,validationofthisapproachforrealtimeadaptationonamobilerobot,developingapproachesforndingthemostpromisingindicatorsforanewsensorsuite,anddeter-miningifanalogoussolutionscanbedevelopedformoreprevalentprobabilisticmodels.166

PAGE 181

Chapter7SummaryandFutureWorkThisdissertationaddressesthefollowingresearchquestion:Cananinconsistency-basedsens-ingaccuracyindicator,whichreliessolelyonfusedsensorreadings,beusedtodetectandcharac-terizesensinganomaliesinunknownenvironments?Morespecically,cananinconsistency-basedsensingaccuracyindicatorenableamobilerobottoautonomously:DetectwhenasensinganomalyhasoccurredEstimatetheaccuracyofasensororsetofsensorsinthecurrentsensingcontext,Isolateregionswithinanunknownenvironmentwheresensinganomaliesoccur,withoutrelyingonaprioriinformationabouttherobot'senvironment?Sensinganomaliesarecaseswherephysicalsensorsareworkingwithinthemanufacturer'sspecicationsbuttheread-ingswouldleadtoanincorrectinterpretationoftheenvironment.Thesecasesonlyexistwhenasensorinteractswiththeenvironmentandoftenaffectmultiplesensors,forexamplemanylaserrangendersseethroughglasswindowswhichalsoscattersonarsignals.Thisworkisfocusedondetectingandcharacterizingsensinganomaliesrelyingsolelyonfusedreadingsfromasinglepossiblyaffectedsensor.Thischapterisorganizedasfollows.Section7.1providesanoverallsummaryofthisdisserta-tionfocusingonthenovelevidentialinconsistency-basedapproachforsensinganomalydetectionandcharacterizationdevelopedforthiswork,theresultsofexperimentswithrealrangesensorreadingscollectedbymobilerobotsindoorandoutdoorenvironments,andtheoverallcontribu-tions.Section7.2presentsdirectionsforfutureworktoenablesituatedintelligentsystemstode-tectandadapttosensinganomalieswhileoptimizingtheaccuracyofsensinginunknownenviron-ments.167

PAGE 182

7.1SummaryThisdissertationinvestigatesanovelinconsistency-basedapproachforsensinganomalydetec-tionandcharacterizationbyamobilerobotusingrangesensingformappingindoorandoutdoorenvironments.Theoverallgoalwastondamethodandthresholdtodeterminewhenasinglesensorinanunknownenvironmentisnotreturningconsistentinformationandwhatregionsofspacewereproducingtheanomalousresultssothattheareacouldbeavoided,re-sensed,oranal-ternativesensorinstantiated.Thedissertationalsoinvestigatedtwosimplestrategiesforimprovingsensingaccuracy:switchingsensorswhenananomalyisdetected,anderasingtheaccumulatedsensordataforasuspectregiontoallownewer,presumablybetter,datatobeused.Finallytheutil-ityofestimatingsensingaccuracywasassessedbyapplyingatrainedsolutionforthiscomponenttotheproblemofdeterminingtherelativerankingofsensorsbyaccuracyinthepresenceofsens-inganomalies.Thegeneralapproach,presentedinChapter3,reliesonDempster-Shaferformulationsofev-idence,adifferentapproachthanprobabilisticmethods.TheDempster-Shaferapproachwasse-lectedduetothehypothesisthatsensinganomalieswouldmanifestasinconsistencyinthesensorreadings.Dempster-Shaferformulationsde-couplethebeliefincontradictoryhypothesese.g.be-liefinAand:Aarenotrequiredtosumtooneenablingthecreationofmetricstomeasurethemagnitudeofinconsistencyindependentlyfrominformativeness.Withintheevidentialliterature,sixinconsistencymetricswereidentiedaspossiblecandidatestodirectlyevaluatefusedsensordata:Yager'sANXIETY,Shafer'sCON,Smets'transferablebeliefmodelTBMCONFLICT,Pal'sINCONSISTENCYmetric,LIU'SCONFLICTmetric,andGAMBINO'schangeheuristic.Howevereachmetricdependedonathresholdtodistinguishminornoisefromanomalousreadings.There-fore,eachofthesixmetricsandanassociatedthresholdmethod,thresholdformedwhatthisthe-sisdenedasaninconsistency-basedsensingaccuracyindicator.7.1.1ExperimentsTheinvestigationofthegeneralapproachincludedafeasibilitystudyandhadthreephasesallusingrealsensordataasindicatedinTable32:training,verication,andanexaminationoftheutilityoftheapproach.Indicatorswereeithertrainedforusewithonesensorortwosensorsinter-changeablyinthetrainingphase,veriedfortheirpredictivecapabilitiesinnewenvironments168

PAGE 183

Table31.Groupsoftargetsensorsexaminedi.e.trained,veried,orappliedintheexperiments.Foreachgroupacommondatasetwasusedfortraining. Group TargetSensors Notes Nomad SonarorSICKPLS ForeitherrangesensorinstalledontheNomad200robot ATRV-Jr CanestarangecameraorSICKLMS ForeitherrangesensorinstalledontheATRV-Jrrobot Canestarangecamera FortheCanestasensoronly SICKLMS Forthelasersensoronly inthevericationphase,orappliedtodeterminetheutilityoftheapproachintheutilityphase.Theutilityphasewasperformedopportunisticallyusingthesensorreadingscollectedfortheothertwophases.Table31liststhetargetsensorsforwhichindicatorsweretrained,veried,orappliedintheexperiments.Theindicatorsweretrainedeitheronreadingsfromasinglesensorforex-clusiveusewiththatsensororonreadingsfromapairofsensorssothesensorscouldbeusedinterchangeablywithoutswitchingindicators.Thetargetsensorsaregroupedbasedontheuseofacommondataseti.e.asetofdatacollectionrunsintoNomadfortargetsensorsinstalledonaNomad200andATRV-JrforthoseinstalledonaniRobotATRV-Jrrobot.Aringof16Polaroidsonarsensorsexaminedinthefeasibilitystudywasnotusedinthevericationphaseinordertofocusthisworkontheproblemofdetectingandcharacterizingsensinganomaliesinreadingsfromasinglesensor.TheSICKPLSlasersensorusedinthefeasibilitystudywasnotavailablefortheadditionaldatacollectionrunswiththeATRV-JrandwasreplacedwithaSICKLMS.TheCanestarangecamerawasselectedforitsabilitytodetectglasssurfaces.Thissensorwasconguredtocollectsensorreadingsintwodistinctmodes:normalfordetectingtypicalsurfaces,andmolewheretheexposuretimewasincreasedtodetectpoorreectingsurfacese.g.glassordarksur-faces.Atotaloffourindoorandtwooutdoorunclutteredtestbedswereusedintheexperiments.Theselectionoftestbedsforeachphasefollowedtheseprinciples:eachsetshouldrepresenten-vironmentalconditionsknowntoproduceanomaliesforthetargetsensors,thetestbedsshouldprovidelongstraightpathsfortherobottotraversetomaximizethelengthofthedatacollectionrunswhileminimizingposeestimationerror,andthesetoftestbedsshouldexposethesensorstovaryingenvironmentalconditionswhichaffectsensingaccuracywhilenotproducingsensinganomalies,e.g.ambienttemperatureaffectstheaccuracyofnear-infraredbasedsensors.169

PAGE 184

Table32.Thetestbedsusedandgroupsoftargetsensorsexaminedineachanalysisphaseoftheexperiments.Feasibilityisusedtoindicatetrainingrunsperformedforthefeasibilitystudy. Phase Training Verication Utility Feasibility Ranking Improvement IndoorTestbeds Narrow Nomad Nomad Cubicle Nomad Nomad ATRV-Jr ATRV-Jr ATRV-Jr Bridge Nomad Nomad glass ATRV-Jr ATRV-Jr ATRV-Jr Lab ATRV-Jr ATRV-Jr ATRV-Jr glass OutdoorTestbeds Walkway ATRV-Jr ATRV-Jr ATRV-Jr Sidewalk ATRV-Jr ATRV-Jr ATRV-Jr glass TotalRuns 45 16 30 46 91 TheobjectiveofthefeasibilitystudydescribedinChapter4wastodetermineiftheevidentialinconsistency-basedapproachcouldworkatall.Todeterminethis,sensordatawascollectedontheNomadrobotwitharingof16PolaroidsonarsensorsandoneSICKPLSlaserrangesensorin45totalrunsinthreeunclutteredindoortestbeds.8to2.5meterswideand10meterslongwithknowngroundtruth.184indicatorswereappliedtothedataandexaminedfordetectionbythepercentageofcorrectlyclassiedoccupancygridsi.e.anomalousornormalandforestima-tionbycorrelationwiththetruesensorerror.Thefeasibilitystudyindicatedthatthreeofthesixmethodsdidappeartobeuseful:ANXIETY,GAMBINO,andINCONSISTENCY.TheANXIETYandGAMBINOmethodscouldbetrainedi.e.byselectingtherightthresholdtodetectsensinganom-aliesineithersonarorlaserreadingswith95%accuracy.TheINCONSISTENCYandANXIETYmethodscouldbetrainedtoestimatesensingaccuracyforthesesensors,withstatisticallysigni-cantcorrelationsabove0.85.Thestudyalsosuggestedthattherewasnospecicindicatorcapableofdetectingandcharacterizingsensinganomalies,thatthethresholdforamethodwouldhavetobeidentiedbytraining.170

PAGE 185

7.1.1.1TrainingTheobjectiveofthetrainingphase,asindicatedinChapter5,wastoidentifytheindicatorsfortherangesensors,SICKLMSandaCanestarangecameraontheATRV-Jr,asthatrobotwouldbeusedforthevericationandutilityphases.TheadditionalsensorsfromtheATRV-Jrprovidedawidercomparisonandopportunitytoexploregeneralizability.AsindicatedinTable32read-ingsfromthesesensorswerecollectedinoneunclutteredindoortestbed.8meterswideand10meterslongandoneunclutteredoutdoortestbed.7meterswideand10meterslongforthetrainingphase.184indicatorswereagainappliedtothedataandexaminedforthedetectionandestimationcomponentsinadditiontoisolationbythedegreeofoverlapbetweenthetrulyerro-neouscellsandthoselabeledsuspecti.e.inconsistencyexceededtheassignedthresholdbytheindicator.Intotal,countingthedefactotrainingrunsduringthefeasibilitystudywiththeNomadandtheadditionalrunsforSICKLMSandaCanestarangecamera,61runswereconductedinvetestbedstotrainindicatorstodetectandcharacterizesensinganomalies.7.1.1.2VericationTheobjectiveofthevericationphasedescribedinChapter5wastodeterminetheperfor-manceofthegeneralapproach;thatis,iftheindicatorscouldmeetthethreecomponentsofthere-searchquestion:detect,estimate,andisolate.ThisobjectivewastestedusingthetrainedindicatorsfortheATRV-JrsensorsSICKLMSandCanestarangecamerainthreetestbedsnotincludedinthetrainingphaseasshowninTable32:twounclutteredindoortestbeds.0and2.5meterswideand10meterslongandoneunclutteredoutdoortestbed.2meterswideand10meterslong.Intermsoftherstcomponent,detection,theexperimentsshowedthatnoneoftheindicatorscoulddistinguishordinarynoisefromsensinganomalies,thoughTBMCONFLICTorANXIETYcoulddetermineifasinglesensorwasaccurateoverall.ANXIETYachieveddetectionaccuracyof99.67%forsonarandSICKPLSanomaliesinthefeasibilitystudywhensuchanomalieswereal-wayspresentthroughoutagiventestbed,e.g.thesonarreadingswerealwaysscatteredbythesur-roundingsurfaces.Whentheappearanceoftheseanomalieswererandomlyvariedforthetrainingandvericationphasesrepresentingmorerealisticscenarios,thedetectionaccuracyhadanun-acceptablywidevariationrangingfrom75.22%to92.23%inthetrainingtestbedsand28.71%to74.71%inthevericationtestbeds.Thiswouldpreventitfrombeingusefulforactualoperation.171

PAGE 186

However,forthesecondcomponenttheresultsshowedthatsensingaccuracycouldbeesti-matedforunknownenvironmentsusingmethodsandthresholdsdeterminedduringtraining.Theestimationcomponentisdifferentfromdetectionbecauseitestimatestheoverallerrorinsensorreadingswhichmaybecausedbysensornoise,discretizationintheoccupancygridmap,etc.withoutdistinguishingthesefromsensinganomalies.TheTBMCONFLICT,0.18indicatorper-formedbest,achievingstatisticallysignicantcorrelationswithoverallsensorerrorinexcessof0.8fortheSICKLMSandCanestarangecamerareadingsinboththetrainingandvericationphasesoftheexperiments.Thismeansthatarobotenteringanunknownenvironmentwouldhaveanindicationwhensensingqualityisdiminished.Theresultsforthethirdcomponentwerealsogood;theyshowthatregionswithsuspectsen-sorreadingscouldbeisolated.Theisolationcomponentisdifferentfromdetectionbecauseitdeterminestheregionsintheenvironmentwherethereislikelytobesensorerrorwithoutdistin-guishingerrorduetonoisefromthatofsensinganomalies.TheTBMCONFLICT,0.38indicatorisolatedpoorlysensedregions,achievinganaverageoverlapwitherroneouscellsof0.61inthetrainingphaseascomparedtoanoverlapof0.01whencellsarerandomlylabeledandanover-lapof0.49to0.62inthevericationphaserelyingsolelyonreadingsfromasinglesensorSICKLMSortheCanestarangecamera.Thismeansthatarobotoperatinginanunknownenvironmentwouldknowwhatregionsweresuspectandcouldrespondappropriatelye.g.,avoidthoseareas,re-sense,etc..Returningtotheresearchquestion,theresultsshowthatevidentialinconsistency-basedmeth-odscanpartiallyaddresstheissueofnoticingsensinganomaliesinunknownenvironments.AsdiscussedinChapter6,theresultsindicatethatfutureapplicationswithnewsensorswillneedtoperformtrainingrunsinknownenvironmentstodeterminewhichmethodworksbestforthatsen-sorandtonddistinctthresholdstoestimatesensingaccuracyandisolatepoorlysensedregions.TheTBMCONFLICTmethodworkedbestforestimationandisolationforlaserandCanestarangecamerasensors,buttheresultsshowedthatdifferentthresholdswereneeded,0.18tolteroutmi-norinconsistencyintheoverallassessmentofsensingaccuracybut0.38todetermineifaspeciccellwaspoorlysensed.Thetrainingphaseresultsalsoindicatethatdifferentsensorsrequiredif-ferentmethods.Forexample,ANXIETYestimatedwellforsonarachievingastatisticallysigni-cantcorrelationof0.854butpoorlyforCanestarangereadingsatbest0.449correlation.172

PAGE 187

7.1.1.3UtilityPhaseTheutilityphasedescribedinChapter5hadtwoobjectives.Therstwastoexplorewhethertheknowledgeofsensingaccuracycouldbeusedtoselectthemostaccuratesensor,i.e.rank-ingthesensors.Thesecondwastodetermineifknowledgeofsensinganomaliesorthelocationofsuspectregionscouldbeusedtoimprovesensingquality.Theseobjectivesweretestedusingtrainedindicatorsforthesonar,bothSICKLasersensors,andtheCanestarangecamerainallsixtestbedswherereadingsforthesesensorswerecollectedasshowninTable32:fourunclutteredindoortestbeds.8to2.5meterswideand10meterslongandtwounclutteredoutdoortestbeds.2and2.7meterswideand10meterslong.Intheutilityphasethetrainedindicatorforestimationdidnotperformwellforranking.Inspiteofitsabilitytoestimatesensingaccuracy,theTBMCONFLICT,0.18indicatorwasunabletoreliablydeterminethemostaccuraterangesensorinthesuite,althoughitcouldidentifytheleastaccuratesensor.TheleastaccuratesensorasmeasuredbytheerrorintheoccupancygridmapwasrankedrstbyTBMCONFLICT,0.18atmostvetimesoutof110classications.Acomparisonoftheoverallaccuracywhenthemostaccuratesensorwasusedversustheindicator'stoppickrevealedlittledifferenceinerror,showingthattheindicatoroftenselectedasensorthatwasnearlyasaccurateasthemostaccuratesensor.Theresultsfromtheutilityphasealsoshowedthatstatisticallysignicantimprovementsinmapqualitywereachievedusingtwosimplestrategiesforimprovingsensingaccuracy.Bothmethodsdidwell.Switchingtoamoreaccuratesensorwhenananomalywasdetectedworkedwell.Improvementsofupto75.86%wereachievedbyswitchingfromthesonartotheSICKPLSwhentheANXIETY,0.86indicatordetectedasensinganomaly.Inthetestbedenviron-mentwhichcontainedglassi.e.thelaserencounteredananomalythesameindicatorproduceda40.52%improvementinsensingaccuracycomparedtoblindlytrustingthemostsophisticatedsen-sor.Resettingsuspectregionsofanoccupancygridsawanoticeablebutsmallerimprovement.A56.50%improvementinmapqualitywasobtainedbyusingTBMCONFLICT,0.38toresetsus-pectregionsoftheoccupancygrid,thoughavisualinspectionofthebeforeandaftermapsshowsamorepronounced,qualitativeimprovement.ThisqualitativeimprovementwasnotreectedinthequantitativeresultsduetolargevariancesintheCanestarangecamerareadingsinwhichglasssurfacesweredetectedinthecorrectcellsbutalsothoseimmediatelyadjacentcausingerrors.173

PAGE 188

Notethatnewreadingsarenotalwaysmoreaccuratethanpriorones,soresettingsuspectcellsmaynotalwaysproducemoreaccurateresults.7.1.2ContributionsAsdiscussedinChapter6,thisworkcontributestoboththeroboticsanduncertaintyinarti-cialintelligencecommunitiesbysettingthefoundationfortheuseofevidentialmetricsforadapt-ingasinglesensororidentifyingthemostaccuratesensorinasuitetooptimizetheaccuracyofsensinginunknownenvironments.Thedissertationalsoprovidedaframework,orsystemarchi-tecture,foraddressingtheseproblemsbyspecifyingthefunctionsofsensing,fusion,andmappingandwheresensinganomalydetectionandcharacterizationwouldt.Itproducedtherstexper-imentalstudyofevidentialmetricsforsensorinconsistencyforasinglesensorversusexistingmethodswhichrequiremultiplesensorstobeactive.Itfoundthatsuchmetricscannotreliablydistinguishbetweensensingnoiseandatrueanomalyacrosstestbeds,butcanidentifythatanoma-loussensingisoccurringandregionsofsuspectsensingeitherfromnoiseorfromanomaliesinlineartimeonthesizeoftheoccupancygrid.However,determininganomaloussensingandwhereitwasoccurringrequiredifferentthresholdvalueswhichaddstothecomputationalcom-plexitybyasmallconstantfactoraseachcellwillbeevaluatedtwice.Theexperimentsshowedthatsensorsinasuitecannotbereliablyrankedintermsofsuitablyforanunknownenvironment,butthatleastaccuratesensorscanbeidentiedonceamethodandthresholdarefoundforestimat-ingsensingaccuracyheretheTBMCONFLICT,0.18indicatorwasusedforranking.Futureapplicationscouldenableintelligentsystemstoswitchinformationsourcestoensurethebestmis-sionperformance,identifythereliabilityofsourcesfordifferentenvironments,andreasonaboutwhichsourcestouseunderwhatcircumstances.Itistherstknowngeneralapproach,eitherevi-dentialorprobabilistic,capableofestimatingsensingaccuracyandisolatingpoorlysensedregionswhenonlyreadingsfromasinglesensorareavailable.7.2FutureWorkThisdissertationrepresentstherstknownworktoexplicitlyaddresstheproblemofsensinganomaliesinunknownenvironmentsandassuchitopensmanyavenuesforfutureworkafewofwhichwillbediscussedinthissection.Straightforwardextensionsofthisworkincludeapplying174

PAGE 189

inconsistency-basedsensingaccuracyindicatorstofusedsensorreadingsandbroadeningtheexist-ingapproachtosolvethesensinganomalydiagnosisproblem.Duetoadvancesincomputervisionandrobotics,inexpensiveCCDcamerasarebecomingmorecommoninsensorsuitesandautono-mousoperationinnaturalisticanddynamicenvironmentsisbecomingtenable.Studiesareneededtoestablishtheapplicabilityofinconsistency-basedsensingaccuracyindicatorsforthesesensorsandenvironments.Inaddition,futureresearchcouldprovideautonomousidenticationofsensorsuitabilityinunknownenvironmentsbydevelopingasolutionbasedontheframeworkpresentedinSection3.1.Finally,atheoreticalfoundationtoaddresstheproblemofsensinganomaliesisneededtoguidethedevelopmentofsolutionsforspecicsensorsuites.Thissectionwilldiscusstheseopportunitiesforfutureresearchinorderfromthemoststraightforwardtothosewhichmaytakeyearstocomplete.Immediateopportunitiesforfutureworkare:verifyingtheapplicabilityofevidentialinconsis-tencymethodsforassessmentofthefusedresultsofmultiplesensors,anddevelopinganapproachfordiagnosisofsensinganomalies.Wheninconsistency-basedsensingaccuracyindicatorsareap-pliedtofusedresultsfrommultiplesensorsitbecomesnon-trivialtodeterminewhichsensorsareaffectedbyadetectedanomaly,thusanapproachisneededtodiagnosesensinganomalies.Thiscouldbeassimpleasreplayingahistoryofpastsensorreadingsusingdifferentsubsetsofsensorstodeterminewhichcombinationsproduceanomalies.OncetherecommendationsfornextstepsgiveninChapter6arecomplete,themostpress-ingareaforfutureworkisvalidationofthisapproachforCCDcamerasandabroaderrangeofenvironments.Theexperimentsfocusedonsensorswhichareparticularlysusceptibletosens-inganomalies,namelyexteroceptivesensorswhichmeasurecharacteristicsoftheenvironmentthattendtovaryintimeandspace.Realsensorreadingswereusedfromthreetypesofrangesensors:sonar,laserrangender,andaCanestarangecamerawhichfallintothisclass.AnothercommonlyusedsensorwhichfallsintothisclassistheCCDcameraandtheapplicationofthisapproachtoanomaliesforthissensorandthewealthofapplicationsitsupportsisapromisingdi-rectionforfutureworkasadvancesincomputervisionmakethisinexpensivesensormoreusefulforroboticsapplications.Moreworkisneededtovalidatetheeffectivenessofthisapproachinnaturalisticanddynamicenvironmentswhichrequiretheuseofmoreexiblerepresentationmod-else.g.O'Callaghan,Ramos,andDurrant-Whyte,2009.Thesenewrepresentationmodelsare175

PAGE 190

probability-basedsoinconsistencymetricsapplicabletotheprobabilisticdomainoranalogousevi-dentialmodelsmustrstbedeveloped.Developingasolutionforautonomousidenticationofsensorsuitabilityinunknownenviron-mentswouldenableasensingmanagementsystemtomaximizeitssensingaccuracywhileexplor-inganunknownenvironmentandprovideusersorplanningsystemse.g.pathplannerswithacondencemapdescribingitscertaintyregardingsensinginanygivenregionoftheenvironment.TheframeworkpresentedinSection3.1outlinesasolutionforthisproblemwhichcombinesthisworkwithexistingsolutionsfromtheroboticsandmulti-agentsystemsliteratureandrequiresso-lutionsfortheproblemsofsensingcontexttrackingandsensinganomalydiagnosisasabovetocompletethesensorsuitabilityidenticationsystem.Sensingcontexttrackingdeterminestheen-vironmentalcontextoftherobotintermsofsensorperformanceandappearstobeanewavenueforfutureresearchasithasnotbeenspecicallyaddressedintheliterature.Hypotheticalsolu-tionscouldbeassimpleasapplyinglabelstoa2Dmaporascomplexasreproducinghippocam-palfunctionstondacommonrepresentationforcontextual,behavioral,andsensoryinformationinamodelforspatialnavigationseeforexampleBarakovaandLourens,2004,2005.Finallyatheoreticalfoundationtodescribehowsensinganomaliesmanifestbasedonchar-acteristicsofasensingsystemisneededtoguidethedevelopmentofsolutionsbasedonthisap-proachforspecicsensorsuites.Theexperimentalresultssuggestthatdifferentsensingsystemsi.e.logicalsensorsHendersonandShilcrat,1995willrequiredifferentsensingaccuracyindica-torse.g.Smets'conictbeliefmassestimatessensinganomalieswellforlaserandCanestarangecamerabutnotforsonarreadings.Thedevelopmentofatheoreticalfoundationforthisapproachwhichexplainshowdifferencesinsensingtechnologyorinterpretationsystemsusedtofusesen-sorreadingsaffecthowsensinganomaliesmanifestinthefusedmodelisneeded.Thiswouldhelpguideandeventuallyautomatetheselectionofeffectivesensingaccuracyindicatorsforsensingsuites.Assuchsuitesbecomemoreexibleandadaptive,automatedselectionwillbecomein-creasinglyimportant.Suchafoundationcouldalsobeusedtoguidethesearchformorerobustsolutionsforsensinganomalydetection.Toconclude,thisdissertationintroducedanovelinconsistency-basedapproachfordetectingandcharacterizingsensinganomaliesthatcanbeusedtoprovidefeedbackforsensinginunknownenvironmentsbutcannotdistinguishsensinganomaliesfromordinarynoise.Itcontributestherst176

PAGE 191

knowngeneralapproachcapableofestimatingsensingaccuracyandisolatingpoorlysensedre-gionsusingonlyfusedreadingsfromasinglesensor.Whiletheexperimentswererestrictedtosensinganomalydetectionandcharacterizationbyamobilerobotusingrangesensingformap-pingindoorandoutdoorenvironments,theapproachisgeneral.FutureapplicationscouldenableintelligentsystemstoautonomouslyresolveassociationissuesAppriouetal.,2001,switchin-formationsourcestoensurethebestmissionperformance,identifytherelativecontributionandreliabilityofsourcesfordifferentenvironments,andreasonaboutwhichsourcestouseunderwhatcircumstances.177

PAGE 192

ReferencesAbellan,J.,andGomez,M.2006.Measuresofdivergenceoncredalsets.FuzzySetsandSystems157:1514.Afgani,M.;Sinanovic,S.;andHaas,H.2008a.AnomalydetectionusingtheKullback-Leiblerdivergencemetric.InProceedingsoftheInternationalSymposiumonAppliedSciencesonBiomedicalandCommunicationTechnologies,1.IEEEPress.Afgani,M.;Sinanovic,S.;andHaas,H.2008b.Informationtheoreticapproachtosignalfeaturedetectionforcognitiveradio.InProceedingsoftheIEEEGlobalTelecommunicationsConfer-ence,1.IEEEPress.Ahlstrom,K.;Torin,J.;Fersan,K.;andNobrant,P.2002.Redundancymanagementindistributedightcontrolsystems:experienceandsimulations.InProceedingsoftheDigitalAvionicsSys-temsConferenceDASC,volume2,13C3C3.Ahn,J.;Sui,X.;DeAngelis,D.;andBarber,K.S.2008.Identifyingbenecialteammatesusingmulti-dimensionaltrust.InProceedingsoftheInternationalJointConferenceonAutonomousAgentsandMultiagentSystems,1469.Ahn,J.;DeAngelis,D.;andBarber,K.S.2008.Teammateselectionusingmulti-dimensionaltrustandattitudemodels.InProceedingsoftheInternationalJointConferenceonAutonomousAgentsandMultiagentSystems,volume5396ofInternationalWorkshoponTrustinAgentSoci-eties,1.Springer.Angelopoulou,E.,andJr.,J.R.W.1999.Laserscannertechnology.TechnicalReportMS-CIS-99-16,DefenseAdvancedResearchProjectAgency.TechnicalreportsubmittedaspartofTMR.178

PAGE 193

Appriou,A.;Ayoun,A.;Benferhat,S.;Besnard,P.;Cholvy,L.;Cooke,R.;Cuppens,F.;Dubois,D.;Fargier,H.;Grabisch,M.;Kruse,R.;Lang,J.;Moral,S.;Prade,H.;Safotti,A.;Smets,P.;andSossai,C.2001.Fusion:Generalconceptsandcharacteristics.InternationalJournalofIntelligentSystems16:1107.Arbuckle,D.;Howard,A.;andMataric,M.2002.Temporaloccupancygrids:Amethodforclas-sifyingthespatio-temporalpropertiesoftheenvironment.InProceedingsoftheIEEE/RSJIn-ternationalConferenceonIntelligentRobotsandSystemsIROS,volume1,409.Aregui,A.,andDenoeux,T.2006.Noveltydetectioninthebelieffunctionsframework.InPro-ceedingsoftheInformationProcessingandManagementofUncertaintyinKnowledge-BasedSystems,volumeI,412.Ayoun,A.,andSmets,P.2001.Dataassociationinmulti-targetdetectionusingthetransferablebeliefmodel.InternationalJournalofIntelligentSystems16:1167.Ayrulu,B.,andBarshan,B.2002.Reliabilitymeasureassignmenttosonarforrobusttargetdiffer-entiation.PatternRecognition35:1403.Bacon,B.J.;Ostroff,A.J.;andJoshi,S.M.2001.RecongurableNDIcontrollerusinginertialsensorfailuredetectionandisolation.IEEETransactionsonAerospaceandElectronicSystems37:1373.Baldridge,A.;Hook,S.;Grove,C.;andRivera,G.2009.TheASTERspectrallibraryversion2.0.RemoteSensingofEnvironment113:711.Baltzakis,H.;Argyros,A.;andTrahanias,P.2003.Fusionoflaserandvisualdataforrobotmo-tionplanningandcollisionavoidance.MachineVisionandApplications15:92.Bank,D.2002.Anerrordetectionmodelforultrasonicsensorevaluationonautonomousmobilesystems.InProceedingsoftheIEEEInternationalWorkshoponRobotandHumanInteractiveCommunication,288.Barakova,E.I.,andLourens,T.2004.Noveltygatedepisodicmemoryformationforrobotexplo-ration.InProceedingsoftheIEEEInternationalConferenceonIntelligentSystems,volume1,116.IEEE.179

PAGE 194

Barakova,E.I.,andLourens,T.2005.Eventbasedself-supervisedtemporalintegrationformulti-modalsensordata.JournalofIntegrativeNeuroscience4:265.Barber,K.S.,andFullam,K.2003.Applyingreputationmodelstocontinuousbeliefrevision.InProceedingsoftheWorkshoponDeception,Fraud,andTrustinAgentSocietiesatAutonomousAgentsAAMAS2003.Barber,K.S.,andKim,J.2001.Beliefrevisionprocessbasedontrust:Agentsevaluatingreputa-tionofinformationsources.InProceedingsoftheWorkshoponDeception,Fraud,andTrustinAgentSocieties,volume2246/2001ofLectureNotesinComputerScience,73.Barber,K.S.,andKim,J.2003.Softsecurity:Isolatingunreliableagentsfromsociety.InPro-ceedingsoftheInternationalJointConferenceonAutonomousAgentsandMultiagentSystems,volume2631/2003ofLectureNotesinComputerScience,181.Barber,K.,andPark,J.2004.Robustpartnerselectionschemeforinformationqualityassurancedespiteuncertaintyinopenmulti-agentsystems.InProceedingsoftheInternationalConferenceonInformationTechnology:CodingandComputingITCC,volume1,430.Blake,D.,andBrown,M.2007.Simultaneous,multiplicativeactuatorandsensorfaultestimationusingfuzzyobservers.InProceedingsoftheIEEEInternationalFuzzySystemsConference,1.IEEEPress.Bongard,J.C.,andLipson,H.2004.Automateddamagediagnosisandrecoveryforremoterobot-ics.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,35453550.Bouchon-Meunier,B.;Dubois,D.;Godo,L.;andPrade,H.1999.FuzzySetsinApproximateReasoningandInformationSystems.Norwell,MA,USA:KluwerAcademicPublishers.chapterFuzzysetsandpossibilitytheoryinapproximateandplausiblereasoning,15.Canham,R.;Jackson,A.;andTyrrell,A.2003.Roboterrordetectionusinganarticialimmunesystem.InProceedingsoftheNASA/DoDConferenceonEvolvableHardware,199.180

PAGE 195

Carlson,J.,andMurphy,R.R.2005.UseofDempster-Shaferconictmetrictodetectinterpreta-tioninconsistency.InProceedingsoftheConferenceonUncertaintyinArticialIntelligence,94.AUAIPress.Carlson,J.,andMurphy,R.R.2006.UseofDempster-Shaferconictmetrictoadaptsensorallocationtounknownenvironments.InProceedingsoftheInternationalFloridaArticialIntelligenceResearchSocietyConferenceFLAIRS,866.Carlson,J.;Murphy,R.;Christopher,S.;andCasper,J.2005.Conictmetricasameasureofsens-ingquality.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,2032.IEEE.Casillas,J.,andMartinez,P.2007.Consistent,completeandcompactgenerationofDNF-typefuzzyrulesbyapittsburgh-stylegeneticalgorithm.InProceedingsoftheIEEEInternationalFuzzySystemsConference,1.IEEE.Chantler,M.;Coghill,G.;Shen,Q.;andLeitch,R.1998.Selectingtoolsandtechniquesformodelbaseddiagnosis.ArticialIntelligenceinEngineering12:81.Chen,W.,andSaif,M.2007.Adaptivesensorfaultdetectionandisolationinuncertainsystems.InProceedingsoftheAmericanControlConference,3240.IEEEPress.Chen,H.;Wu,H.;Hu,J.;andGao,C.2008.Agent-basedtrustmanagementmodelforwirelesssensornetworks.InProceedingsoftheInternationalConferenceonMultimediaandUbiquitousEngineering,150.IEEEPress.Christensen,A.L.;O'Grady,R.;Birattari,M.;andDorigo,M.2008.Faultdetectioninautono-mousrobotsbasedonfaultinjectionandlearning.AutonomousRobots24:49.Ciliz,M.K.2005.Rulebasereductionforknowledge-basedfuzzycontrollerswithapplicationtoavacuumcleaner.ExpertSystemswithApplications28:175.Cobb,B.R.,andShenoy,P.P.2003.Acomparisonofbayesianandbelieffunctionreasoning.InformationSystemsFrontiers5:345.181

PAGE 196

Cobb,B.R.,andShenoy,P.P.2006.Ontheplausibilitytransformationmethodfortranslatingbelieffunctionmodelstoprobabilitymodels.InternationalJournalofApproximateReasoning41:314.Cohen,O.,andEdan,Y.2008.Asensorfusionframeworkforonlinesensorandalgorithmselec-tion.RoboticsandAutonomousSystems56:762.Cohen,O.2006.Statisticalevaluationmethodforcomparinggridmapbasedsensorfusionalgo-rithms.InternationalJournalofRoboticsResearch25:117.Console,L.,andDressler,O.1999.Model-baseddiagnosisintherealworld:Lessonslearnedandchallengesremaining.InProceedingsoftheSixteenthInternationalJointConferenceonArticialIntelligence,1393.Dawdy,D.R.,andMatalas,N.1964.HandbookofAppliedHydrology:ACompendiumofWater-ResourcesTechnology.McGraw-HillBookCompany.chapterStatisticalandProbabilityAnaly-sisofHydrologicData,PartIII.AnalysisofVariance,Covariance,andTimeSeries,8.deKleer,J.,andKurien,J.2003.Fundamentalsofmodel-baseddiagnosis.InProceedingsoftheSymposiumonFaultDetection,SupervisionandSafetyforTechnicalProcesses,25.deKleer,J.,andRaiman,O.1993.Howtodiagnosewithverylittleinformation.InProceedingsoftheInternationalWorkshoponPrinciplesofDiagnosis,160.deKleer,J.,andRaiman,O.1995.Tradingoffthecostsofinferencevs.probingindiagnosis.InProceedingsoftheFourteenthInternationalJointConferenceonArticialIntelligence,17361741.deKleer,J.,andWilliams,B.C.1987.Diagnosingmultiplefaults.ArticialIntelligence32:97130.deKleer,J.,andWilliams,B.C.1989.Diagnosiswithbehavioralmodes.InProceedingsoftheInternationalJointConferenceonArticialIntelligence,1324.Dearden,R.;Willeke,T.;Simmons,R.;Verma,V.;Hutter,F.;andThrun,S.2004.Real-timefaultdetectionandsituationalawarenessforrovers:Reportonthemarstechnologyprogramtask.InProceedingsoftheIEEEAerospaceConference,volume2,826.IEEE.182

PAGE 197

Delmotte,F.;Dubois,L.;andBorne,P.1996.Context-dependenttrustindatafusionwithinthepossibilitytheory.InProceedingsoftheIEEEInternationalConferenceonSystems,Man,andCybernetics,volume1,538.IEEE.Deuker,B.;Perrier,M.;andAmy,B.1998.Fault-diagnosisofsubsearobotsusingneuro-symbolichybridsystems.InProceedingsoftheIEEEOceanicEngineeringSocietyConference,830.Dick,J.;Hull,E.;andJackson,K.2004.Requirementsengineering.Springer.Dixon,W.;Walker,I.;Dawson,D.;andHartranft,J.2000.Faultdetectionforrobotmanipulatorswithparametricuncertainty:aprediction-error-basedapproach.IEEETransactionsonRoboticsandAutomation16:689.Dong-Huynh,T.;Jennings,N.;andShadbolt,N.2006.FIRE:Anintegratedtrustandreputationmodelforopenmulti-agentsystems.AutonomousAgentsandMulti-AgentSystems13:119.Dowdy,S.;Weardon,S.;andChilko,D.2004.StatisticsforResearch.WileySeriesinProbabilityandStatistics.Wiley-Interscience,3rdedition.Dragoni,A.,andGiorgini,P.2003.Distributedbeliefrevision.AutonomousAgentsandMulti-AgentSystems6:115.Du,Z.,andJin,X.2007.Detectionanddiagnosisofsensorfaultinhvacsystems.EnergyConver-sionandManagement48:693.Duan,Z.;Cai,Z.;andYu,J.2006.Fuzzyadaptiveparticlelteralgorithmformobilerobotfaultdiagnosis.InProceedingsoftheInternationalConferenceonNeuralInformationPro-cessing,PartIII,volume4234ofLectureNotesinComputerScience,711.SpringerBerlin/Heidelberg.Dubois,D.,andPrade,H.1999.FuzzyLogicinArticialIntelligence,volume1566ofLectureNotesinComputerScience.SpringerBerlin/Heidelberg.chapterThePlaceofFuzzyLogicinAI,9.Dubois,L.;Fukuda,T.;Delmotte,F.;andBorne,P.1997.Approximatereasoningforthecontrolofarobotinanuncertainenvironment:Amulti-modelapproach.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,volume1,823.IEEE.183

PAGE 198

Ebbinghaus,H.-D.;Flum,J.;andThomas,W.1994.MathematicalLogic.Springer,2ndedition.Elfes,A.1989.Usingoccupancygridsformobilerobotperceptionandnavigation.IEEECom-puter46.Fravolini,M.;Campa,G.;Napolitano,K.;andSong,Y.2001.Minimalresourceallocatingnet-worksforaircraftSFDIA.InProceedingsoftheIEEE/ASMEInternationalConferenceonAd-vancedIntelligentMechatronics,volume2,1251.Freund,Y.,andSchapire,R.E.1996.Experimentswithanewboostingalgorithm.InProceedingsoftheInternationalConferenceonMachineLearning,148.Fullam,K.K.,andBarber,K.S.2007.Dynamicallylearningsourcesoftrustinformation:Ex-periencevs.reputation.InProceedingsoftheInternationalJointConferenceonAutonomousAgentsandMultiagentSystems,1.ACM.Fullam,K.K.;Klos,T.B.;Muller,G.;Sabater,J.;Schlosser,A.;Topol,Z.;Barber,K.S.;Rosen-schein,J.S.;Vercouter,L.;andVoss,M.2005.AspecicationoftheagentreputationandtrustARTtestbed:experimentationandcompetitionfortrustinagentsocieties.InProceedingsoftheInternationalJointConferenceonAutonomousAgentsandMultiagentSystems,512.NewYork,NY,USA:ACMPress.Gage,A.,andMurphy,R.R.2004.Sensorschedulinginmobilerobotsusingincompleteinforma-tionviamin-conictwithhappiness.IEEETransactionsonSystems,ManandCybernetics,PartB34:454.Gambino,F.;Ulivi,G.;andVendittelli,M.1997.Thetransferablebeliefmodelinultrasonicmapbuilding.InProceedingsof6thIEEEInternationalConferenceonFuzzySystems,volume1,601.Ganeriwal,S.,andSrivastava,M.B.2004.Reputation-basedframeworkforhighintegritysensornetworks.InProceedingsoftheACMWorkshoponSecurityofAdHocandSensorNetworks,66.NewYork,NY,USA:ACMPress.Ganeriwal,S.;Balzano,L.K.;andSrivastava,M.B.2008.Reputation-basedframeworkforhighintegritysensornetworks.ACMTransactionsonSensorNetworks4:1.184

PAGE 199

Gao,Z.;Breikin,T.;andWang,H.2008.Reliableobserver-basedcontrolagainstsensorfailuresforsystemswithtimedelaysinbothstateandinput.IEEETransactionsofSystems,ManandCybernetics,PartA38:1018.Garmendia,L.2005.IntelligentDataMining,volume5ofStudiesinComputationalIntelligence.SpringerBerlin/Heidelberg.chapterTheEvolutionoftheConceptofFuzzyMeasure,185.Goel,P.;Dedeoglu,G.;Roumeliotis,S.;andSukhatme,G.2000.Faultdetectionandidenticationinamobilerobotusingmultiplemodelestimationandneuralnetwork.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,volume3,2302.Gokturk,S.B.;Yalcin,H.;andBarnji,C.2004.Atime-of-ightdepthsensor-systemdescrip-tion,issuesandsolutions.InProceedingsoftheConferenceonComputerVisionandPatternRecognitionWorkshop,volume3,35.IEEEComputerSocietyPress.Gu,J.;Meng,M.;Cook,A.;Faulkner,M.;andLiu,P.2001.Multiplesensordatafusioninroboticprostheticeyesystem.InProceedingsoftheIEEEInternationalSymposiumonComputationalIntelligenceinRoboticsandAutomation,65.Hashimoto,M.;Kawashima,H.;Nakagami,T.;andOba,F.2001.Sensorfaultdetectionandidenticationindead-reckoningsystemofmobilerobot:Interactingmultiplemodelapproach.InProceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,volume3,1321.Hashimoto,M.;Watanabe,S.;andTakahashi,K.2007.Sensorfaultdetectionandisolationforapoweredwheelchair.InProceedingsoftheIEEE/ASMEInternationalConferenceonAdvancedIntelligentMechatronics,1.IEEEPress.Henderson,T.C.,andShilcrat,E.1995.MultisensorIntegrationandFusionforIntelligentMa-chinesandSystems.AblexPublishingCorp.chapterLogicalSensorSystems,81.Heredia,G.;Ollero,A.;Bejar,M.;andMahtani,R.2008.Sensorandactuatorfaultdetectioninsmallautonomoushelicopters.Mechatronics18:90.185

PAGE 200

Hongjun,D.;Zhiping,J.;andXiaona,D.2008.Anentropy-basedtrustmodelingandevaluationforwirelesssensornetworks.InProceedingsoftheInternationalConferenceonEmbeddedSoftwareandSystems,27.IEEEPress.Hunter,A.,andKonieczny,S.2005.InconsistencyTolerance,volume3300ofLectureNotesinComputerScience.SpringerBerlin/Heidelberg.chapterApproachestoMeasuringInconsistentInformation,191.Hur,J.;Lee,Y.;Hong,S.-M.;andYoon,H.2005.Trustmanagementforresilientwirelesssensornetworks.InProceedingsoftheInternationalConferenceonInformationSecurityandCryptol-ogy,volume3935ofLectureNotesinComputerScience,56.SpringerBerlin/Heidelberg.Ji,M.;Zhang,Z.;Biswas,G.;andSarkar,N.2003.Hybridfaultadaptivecontrolofawheeledmobilerobot.IEEE/ASMETransactionsonMechatronics8:226.Jsang,A.,andIsmail,R.2002.Thebetareputationsystem.InProceedingsoftheBledConfer-enceonElectronicCommerce,1.Jousselme,A.-L.;Liu,C.;Grenier,D.;andBosse,E.2006.Measuringambiguityintheevidencetheory.IEEETransactionsofSystems,ManandCybernetics,PartA36:890.Jousselme,A.-L.;Grenier,D.;andBosse,E.2001.Anewdistancebetweentwobodiesofevi-dence.InformationFusion2:91.Kawabata,K.;Okina,S.;Fujii,T.;andAsama,H.2003.Asystemforself-diagnosisofanautono-mousmobilerobotusinginternalstatesensorysystem:faultdetectionandcopingwithinternalcondition.AdvancedRobotics17:925.Khosravifar,B.;Bentahar,J.;Gomrokchi,M.;andAlam,R.2008.Anapproachtocomprehensivetrustmanagementinmulti-agentsystemswithcredibility.InProceedingsoftheInternationalConferenceonResearchChallengesinInformationScience,53.IEEEPress.Klir,G.J.,andLewis,H.W.2008.Remarksonmeasuringambiguityinevidencetheory.IEEETransactionsonSystems,ManandCybernetics,PartA38:995.186

PAGE 201

Kobayashi,T.,andSimon,D.L.2006.Hybridkalmanlterapproachforaircraftenginein-ightdiagnostics:Sensorfaultdetectioncase.TechnicalReport2006-214418,NationalAeronauticsandSpaceAdministrationNASA.Kobayashi,F.;Avai,F.;andFukuda,T.1999.Sensorselectionbyreliabilitybasedonpossibilitymeasure.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,volume4,2614.IEEE.Koushanfar,F.,andPotkonjak,M.2005.Markovchain-basedmodelsformissingandfaultydatainMICA2sensormotes.InProceedingsoftheIEEESensorsConference,576.IEEEPress.Krishnamachari,B.,andIyengar,S.2004.DistributedBayesianalgorithmsforfault-toleranteventregiondetectioninwirelesssensornetworks.IEEETransactionsonComputers53:241.Larkey,L.;Bettencourt,L.;andHagberg,A.2006.In-situdataqualityassuranceforenvironmen-talapplicationsofwirelesssensornetworks.TechnicalReportLA-UR-06-1117,LosAlamosNationalLaboratory.Lee,C.F.,andXu,Y.P.2001.Amulti-sensorbasedtemperaturemeasuringsystemwithself-diagnosis.InProceedingsofIEEERegion10InternationalConferenceonElectricalandElec-tronicTechnology.TENCON2001,903.Lerner,U.;Parr,R.;Koller,D.;andBiswas,G.2000.Bayesianfaultdetectionanddiagnosisindy-namicsystems.InProceedingsoftheSeventeenthNationalConferenceonArticialIntelligenceAAAI-2000,531.Letia,I.,andSlavescu,R.2008.Towardsamethodforbootstrappingagenttrust.InProceedingsoftheIEEEInternationalConferenceonAutomation,QualityandTesting,Robotics,volume3,369.IEEEPress.Li,Y.,andPham,B.L.2003.Beliefgeneralizationforintelligentagents.AsianJournalofInfor-mationTechnology2:203.Liu,Y.;Shen,Y.;andHu,H.1999.Anewmethodforsensorfaultdetection,isolationandac-commodation.InProceedingsofthe16thIEEEInstrumentationandMeasurementTechnologyConference,volume1,488.187

PAGE 202

Liu,W.2006a.Analyzingthedegreeofconictamongbelieffunctions.ArticialIntelligence170:909.Liu,W.2006b.Measuringconictbetweenpossibilisticuncertaininformationthroughbelieffunctiontheory.InProceedingsoftheInternationalConferenceonKnowledgeScience,En-gineering,andManagement,volume4092ofLectureNotesinComputerScience,265.SpringerBerlin/Heidelberg.Liu,W.2007.Conictanalysisandmergingoperatorsselectioninpossibilitytheory.InPro-ceedingsoftheEuropeanConferenceonSymbolicandQuantitativeApproachestoReason-ingwithUncertainty,volume4724ofLectureNotesinComputerScience,816.SpringerBerlin/Heidelberg.Lopes,L.;Connell,J.;Dario,P.;Murphy,R.R.;Bonasso,P.;Nebel,B.;Nilsson,N.;andBrooks,R.2001.Sentienceinrobots:Applicationsandchallenges.IEEEIntelligentSystems16:6669.Lopez-Sanchez,M.;deMantaras,R.L.;andSierra,C.1997.Incrementalmapgenerationbylowcostrobotsbasedonpossibility/necessitygrids.InProceedingsoftheConferenceonUncer-taintyinArticialIntelligence,351.McIntyre,M.;Dixon,W.;Dawson,D.;andWalker,I.2004.Faultdetectionandidenticationforrobotmanipulators.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,4981.Merriam-WebsterInc.2008.consistent-denitionfromthemerriam-websteronlinedictionary.Miura,J.;Negishi,Y.;andShirai,Y.2006.Adaptiverobotspeedcontrolbyconsideringmapandmotionuncertainty.RoboticsandAutonomousSystems54:110.Momani,M.;Challa,S.;andAlhmouz,R.2008.Canwetrusttrustednodesinwirelesssensornetworks?InProceedingsoftheInternationalConferenceonComputerandCommunicationsEngineering,1227.IEEEPress.188

PAGE 203

Monteriu,A.;Asthan,P.;Valavanis,K.;andLonghi,S.2007a.Model-basedsensorfaultdetectionandisolationsystemforunmannedgroundvehicles:Experimentalvalidationpartii.InPro-ceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,2744.IEEEPress.Monteriu,A.;Asthan,P.;Valavanis,K.;andLonghi,S.2007b.Model-basedsensorfaultdetectionandisolationsystemforunmannedgroundvehicles:Theoreticalaspectsparti.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,2736.IEEEPress.Morales,Y.;Takeuchi,E.;andTsubouchi,T.2008.Vehiclelocalizationinoutdoorwoodlandenvironmentswithsensorfaultdetection.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,449.IEEEPress.Murphy,R.1998.Dempster-Shafertheoryforsensorfusioninautonomousmobilerobots.IEEETransactionsonRoboticsandAutomation14:197.Murphy,R.R.2000.IntroductiontoAIRobotics.MITPress.Narasimham,S.,andBiswas,G.2007.Model-baseddiagnosisofhybridsystems.IEEETransac-tionsonSystems,ManandCybernetics,PartA37:348.Narasimham,S.;Zhao,F.;Biswas,G.;andHung,E.2000.Faultisolationinhybridsystemscom-biningmodelbaseddiagnosisandsignalprocessing.InProceedingsoftheSymposiumonFaultDetection,SupervisionandSafetyforTechnicalProcesses,512.Narasimhan,S.;Brownston,L.;andBurrows,D.2004.Explanationconstraintprogrammingformodel-baseddiagnosisofengineeredsystems.InProceedingsoftheIEEEAerospaceConfer-ence,volume5,3495.IEEE.Narasimhan,S.;Dearden,R.;andBenazera,E.2004.Combiningparticleltersandconsistency-basedapproachesformonitoringanddiagnosisofstochastichybridsystems.InProceedingsoftheInternationalWorkshoponPrinciplesofDiagnosis.Nourbakhsh,I.;Sycara,K.;Koes,M.;Yong,M.;Lewis,M.;andBurion,S.2005.Human-robotteamingforsearchandrescue.IEEEPervasiveComputing4:72.189

PAGE 204

O'Callaghan,S.;Ramos,F.;andDurrant-Whyte,H.2009.ContextualoccupancymapsusingGaussianprocesses.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomationtoappear.Paiva,H.;Galvao,R.;andYoneyama,T.2008.Awaveletband-limitinglterapproachforfaultdetectionindynamicsystems.IEEETransactionsonSystems,ManandCybernetics,PartA38:680.Pal,N.R.1999.Onquanticationofdifferentfacetsofuncertainty.FuzzySetsandSystems107:81.Parra-Loera,R.;Thompson,W.E.;andSalvi,A.P.1991.Adaptiveselectionofsensorsbasedonindividualperformancesinamultisensorenvironment.InProceedingsoftheSPIEDataStructuresandTargetClassication,volume1470,30.SPIE.Pertew,A.;Marquez,H.;andZhao,Q.2007.LMI-basedsensorfaultdiagnosisfornonlinearLipschitzsystems.Automatica43:1464.Pettersson,O.2005.Executionmonitoringinrobotics:Asurvey.RoboticsandAutonomousSystems53:73.Plagemann,C.;Fox,D.;andBurgard,W.2007.Efcientfailuredetectiononmobilerobotsus-ingparticlelterswithGaussianprocessproposals.InProceedingsoftheInternationalJointConferenceonArticialIntelligence,2185.AAAIPress.Pon,R.K.,andCardenas,A.F.2005.Dataqualityinference.InProceedingsoftheInternationalWorkshoponInformationQualityinInformationSystems,105.ACMPress.Ranganathan,N.;Patel,M.;andSathyamurthy,R.2001.Anintelligentsystemforfailuredetectionandcontrolinanautonomousunderwatervehicle.IEEETransactionsonSystems,ManandCybernetics,PartA31:762.Reece,S.;Roberts,S.;Rogers,A.;andJennings,N.R.2007.Amulti-dimensionaltrustmodelforheterogeneouscontractobservations.InProceedingsoftheAAAIConferenceonArticialIntelligence,128.AAAIPress.190

PAGE 205

Reiter,R.1987.Atheoryofdiagnosisfromrstprinciples.ArticialIntelligenceJournal32:5795.Roman,C.,andSingh,H.2006.Consistencybasederrorevaluationfordeepseabathymetricmap-pingwithroboticvehicles.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,3568.IEEE.Rothenhagen,K.;Thomsen,S.;andFuchs,F.2007.Voltagesensorfaultdetectionandrecong-urationforadoublyfedinductiongenerator.InProceedingsoftheIEEEInternationalSym-posiumonDiagnosticsforElectricMachines,PowerElectronicsandDrives,377.IEEEPress.Roumeliotis,S.;Sukhatme,G.;andBekey,G.1998a.Faultdetectionandidenticationinamobilerobotusingmultiple-modelestimation.InProceedingsofIEEEInternationalConferenceonRoboticsandAutomation,volume3,2223.Roumeliotis,S.;Sukhatme,G.;andBekey,G.1998b.Sensorfaultdetectionandidenticationinamobilerobot.InProceedingsofIEEEInternationalConferenceonIntelligentRobotsandSystems,volume3,13.Ryutov,T.,andNeuman,C.2007.Trustbasedapproachforimprovingdatareliabilityinindustrialsensornetworks.InEtalle,S.,andMarsh,S.,eds.,ProceedingsoftheJointiTrustandPSTCon-ferencesonPrivacy,TrustManagementandSecurity,volume238ofInternationalFederationforInformationProcessingIFIP,349.Boston:Springer.Sabater,J.,andSierra,C.2002.Reputationandsocialnetworkanalysisinmulti-agentsystems.InProceedingsoftheInternationalJointConferenceonAutonomousAgentsandMultiagentSystems,475.Schmidt,S.;Steele,R.;Dillion,T.S.;andChang,E.2007.Fuzzytrustevaluationandcredibilitydevelopmentinmulti-agentsystems.AppliedSoftComputing7:492.Schubert,J.1993.Onnonspecicevidence.InternationalJournalofIntelligentSystems8:711725.191

PAGE 206

Schwall,M.,andGerdes,J.2002.Aprobabilisticapproachtoresidualprocessingforvehiclefaultdetection.InProceedingsofthe2002AmericanControlConference,volume3,2552.Shafer,G.1976.AMathematicalTheoryofEvidence.PrincetonUniversityPress.Shayer,G.;Cohen,O.;Edan,Y.;andKorach,E.2002.Anadaptivefuzzy-logicprocedureforrankinglogicalsensoryperformance.InProceedingsoftheIEEEConferenceonSensors,vol-ume2,1317.Shortliffe,E.;Davis,R.;Axline,S.;Buchanan,B.;Green,C.;andCohen,S.1975.Computer-basedconsultationsinclinicaltherapeutics:ExplanationandruleacquisitioncapabilitiesoftheMYCINsystem.ComputersandBiomedicalResearch8:303.Sivaraman,E.,andChang,K.-C.2007.Performanceevaluationofmulti-sensorclassicationsystems.IEEETransactionsonAerospaceandElectronicSystems43:1265.Smets,P.1990a.Thecombinationofevidenceinthetransferablebeliefmodel.IEEETransactionsonPatternAnalysisandMachineIntelligence12:447.Smets,P.1990b.Constructingthepignisticprobabilityfunctioninacontextofuncertainty.InProceedingsoftheFifthAnnualConferenceonUncertaintyinArticialIntelligence,29.North-HollandPublishingCo.Smets,P.2005.Belieffunctionsonrealnumbers.InternationalJournalofApproximateReason-ing40:181.Smets,P.2007.Analyzingthecombinationofconictingbelieffunctions.InformationFusion8:387.Soika,M.1997a.Gridbasedfaultdetectionandcalibrationofsensorsonmobilerobots.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,2589.Soika,M.1997b.Asensorfailuredetectionframeworkforautonomousmobilerobots.InPro-ceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,17351740.192

PAGE 207

Srinivas,S.1977.ErrorRecoveryinRobotSystems.Ph.D.Dissertation,CaliforniaInstituteofTechnology.ComputerScience.Sun,Y.L.;Yu,W.;Han,Z.;andLiu,K.2006.Informationtheoreticframeworkoftrustmodel-ingandevaluationforadhocnetworks.IEEEJournalonSelectedAreasinCommunications24:305.Teacy,W.;Patel,J.;Jennings,N.R.;andLuck,M.2006.Travos:Trustandreputationinthecon-textofinaccurateinformationsources.AutonomousAgentsandMulti-AgentSystems12:183198.Thrun,S.;Fox,D.;andBurgard,W.1998.Aprobabilisticapproachtoconcurrentmappingandlocalizationformobilerobots.AutonomousRobots5:253.Thrun,S.2003a.Learningoccupancygridswithforwardsensormodels.AutonomousRobots15:111.Thrun,S.2003b.ExploringArticialIntelligenceintheNewMillennium.SanFrancisco,CA,USA:MorganKaufmannPublishersInc.chapterRoboticMapping:ASurvey,1.Um,D.,andLumelsky,V.1999.Faulttoleranceviaanalyticredundancyforamodularizedsen-sitiveskin.InProceedingsoftheIEEE/RSJInternationalConferenceonIntelligentRobotsandSystems,volume2,1191.Vinsonneau,J.;Shields,D.;King,P.;andBurnham,K.2002.ImprovedSIenginemodellingtech-niqueswithapplicationtofaultdetection.InProceedingsofthe2002InternationalConferenceonControlApplications,volume2,719.Visinsky,M.;Cavallaro,J.;andWalker,I.1995.Adynamicfaulttoleranceframeworkforremoterobots.IEEETransactionsonRoboticsandAutomation11:477.Vos,D.,andMotazed,B.1996.TheapplicationoffaulttolerantcontrolstoUAVs.InProceedingsoftheSPIE-TheInternationalSocietyforOpticalEngineering,69.Wang,X.,andShen,H.C.1999.Multiplehypothesistestingmethodfordecisionmaking.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,2090.193

PAGE 208

Wang,Y.,andSingh,M.P.2007.Formaltrustmodelformultiagentsystems.InProceedingsoftheInternationalJointConferenceonArticialIntelligence,1551.AAAIPress.Wang,S.;Wang,J.;Dang,C.;andLiu,A.2007.Atrustevaluationmethodofmobileagentsys-tem.InProceedingsoftheInternationalConferenceonWirelessCommunications,NetworkingandMobileComputing,21.IEEEPress.Washington,R.2000.On-boardreal-timestateandfaultidenticationforrovers.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutomation,1175.Wei,T.;Huang,Y.;andChen,P.2006.Particlelteringforadaptivesensorfaultdetectionandidentication.InProceedingsoftheIEEEInternationalConferenceonRoboticsandAutoma-tion,3807.IEEEPress.Williams,B.C.,andNayak,P.P.1996.Amodel-basedapproachtoreactiveself-conguringsystems.InProceedingsoftheNationalConferenceonArticialIntelligence,971.AAAIPress.Yager,R.R.,andKikuchi,S.2004.Ontheroleofanxietyindecisionsunderpossibilisticuncer-tainty.IEEETransactionsonSystems,ManandCybernetics,PartB34:1224.Yager,R.R.1982.Measuringtranquilityandanxietyindecisionmaking:Anapplicationoffuzzysets.InternationalJournalofGeneralSystems8:139.Yu,B.,andSingh,M.P.2002.Anevidentialmodelofdistributedreputationmanagement.InPro-ceedingsoftheInternationalJointConferenceonAutonomousAgentsandMultiagentSystems,294.ACMPress.Yu,B.,andSingh,M.P.2003a.Detectingdeceptioninreputationmanagement.InProceedingsoftheInternationalJointConferenceonAutonomousAgentsandMultiagentSystems,73.ACMPress.Yu,B.,andSingh,M.P.2003b.Searchingsocialnetworks.InProceedingsoftheInternationalJointConferenceonAutonomousAgentsandMultiagentSystems,65.Yu,B.,andSycara,K.2006.Learningthequalityofsensordataindistributeddecisionfusion.InProceedingsoftheInternationalConferenceonInformationFusion,1.194

PAGE 209

Zilberstein,S.1996.Resource-boundedsensingandplanninginautonomoussystems.Autono-mousRobots3:31.195

PAGE 210

Appendices196

PAGE 211

AppendixADetailedResults:FeasibilityStudyThisappendixcontainstherawresultsforthefeasibilitystudydescribedinChapter4.ThesearegiveninthetablebelowwhereT1referstothethresholdusedatthecell-levelwithANXIETY,CON,GAMBINO,INCONSISTENCY,andTBMCONFLICT.ForLIUSCONFLICTT1referstothecell-levelthresholdappliedtom(!)andT2referstothecell-levelthresholdappliedtodiffBetP.ThistablealsoincludesPearsonscorrelationcoefcientr,theassociatedprobabilityp(calculatedusingautocorrelationcorrection),thepercentageofcorrectlyclassiedexamples(%Correct),thefalsepositiverate(FP-Rate),andfalse-negativerate(FN-Rate).Table33:Rawresultsforthefeasibilitystudy.MethodT1T2rp%CorrectFP-RateFN-Rate ANXIETY0.10.85386.22E-1568.89%100.00%0.00%ANXIETY0.140.84926.66E-1568.89%100.00%0.00%ANXIETY0.180.84471.27E-1468.89%100.00%0.00%ANXIETY0.220.84001.31E-1468.89%100.00%0.00%ANXIETY0.260.83522.53E-1468.89%100.00%0.00%ANXIETY0.30.82982.89E-1468.89%100.00%0.00%ANXIETY0.340.82386.39E-1468.89%100.00%0.00%ANXIETY0.380.81827.37E-1468.89%100.00%0.00%ANXIETY0.420.81301.42E-1368.89%100.00%0.00%ANXIETY0.460.80801.52E-1368.89%100.00%0.00%ANXIETY0.50.80183.21E-1368.89%100.00%0.00%ANXIETY0.540.79437.67E-1368.89%100.00%0.00%ANXIETY0.580.78161.89E-1269.00%99.64%0.00%ANXIETY0.620.76081.58E-1169.67%97.50%0.00%ANXIETY0.660.73322.89E-1071.44%91.79%0.00%ANXIETY0.70.69575.24E-0977.78%71.43%0.00%ANXIETY0.740.64073.12E-0783.33%53.57%0.00%ANXIETY0.780.56791.70E-0588.56%36.79%0.00%ANXIETY0.820.47766.88E-0493.56%19.64%0.48%ANXIETY0.860.37861.23E-0299.67%0.00%0.48%ANXIETY0.90.29166.10E-0299.11%0.00%1.29%CON0.250.35981.93E-0268.89%100.00%0.00%CON0.50.35931.95E-0268.89%100.00%0.00%CON0.750.35821.83E-0268.89%100.00%0.00%CON10.35711.87E-0268.89%100.00%0.00%CON1.250.35621.91E-0268.89%100.00%0.00%CON1.50.35521.94E-0268.89%100.00%0.00%CON1.750.35411.84E-0268.89%100.00%0.00% Continuedonnextpage 197

PAGE 212

AppendixAContinuedTable33ContinuedMethodT1T2rp%CorrectFP-RateFN-Rate CON20.35271.89E-0268.89%100.00%0.00%CON2.250.35121.94E-0268.89%100.00%0.00%CON2.50.34951.86E-0268.89%100.00%0.00%CON2.750.34851.90E-0268.89%100.00%0.00%CON30.34721.95E-0268.89%100.00%0.00%CON3.250.34571.86E-0268.89%100.00%0.00%CON3.50.34441.91E-0268.89%100.00%0.00%CON3.750.34271.97E-0268.89%100.00%0.00%CON40.34081.91E-0268.89%100.00%0.00%CON4.250.33911.97E-0268.89%100.00%0.00%CON4.50.33762.03E-0268.89%100.00%0.00%CON4.750.33591.96E-0268.89%100.00%0.00%CON50.33382.04E-0268.89%100.00%0.00%GAMBINO0.50.64505.07E-0568.56%100.00%0.00%GAMBINO10.64505.07E-0568.56%100.00%0.00%GAMBINO1.50.70615.12E-0768.56%100.00%0.00%GAMBINO20.70615.12E-0768.56%100.00%0.00%GAMBINO2.50.68341.29E-0996.33%0.35%5.19%GAMBINO30.68341.29E-0996.33%0.35%5.19%GAMBINO3.50.42652.32E-1173.56%0.00%38.57%GAMBINO40.42652.32E-1173.56%0.00%38.57%GAMBINO4.50.06721.41E-0139.56%0.00%88.17%GAMBINO50.06721.41E-0139.56%0.00%88.17%GAMBINO5.50.00001.00E+0031.44%0.00%100.00%GAMBINO60.00001.00E+0031.44%0.00%100.00%GAMBINO6.50.00001.00E+0031.44%0.00%100.00%GAMBINO70.00001.00E+0031.44%0.00%100.00%GAMBINO7.50.00001.00E+0031.44%0.00%100.00%GAMBINO80.00001.00E+0031.44%0.00%100.00%GAMBINO8.50.00001.00E+0031.44%0.00%100.00%GAMBINO90.00001.00E+0031.44%0.00%100.00%GAMBINO9.50.00001.00E+0031.44%0.00%100.00%GAMBINO100.00001.00E+0031.44%0.00%100.00%INCONSISTENCY0.050.87354.44E-1668.89%100.00%0.00%INCONSISTENCY0.070.87078.88E-1668.89%100.00%0.00%INCONSISTENCY0.090.86748.88E-1668.89%100.00%0.00%INCONSISTENCY0.110.86361.33E-1568.89%100.00%0.00%INCONSISTENCY0.130.85981.33E-1568.89%100.00%0.00%INCONSISTENCY0.150.85562.66E-1568.89%100.00%0.00%INCONSISTENCY0.170.85085.33E-1568.89%100.00%0.00%INCONSISTENCY0.190.84555.77E-1568.89%100.00%0.00%INCONSISTENCY0.210.83951.40E-1468.89%100.00%0.00%INCONSISTENCY0.230.83005.11E-1468.89%100.00%0.00% Continuedonnextpage 198

PAGE 213

AppendixAContinuedTable33ContinuedMethodT1T2rp%CorrectFP-RateFN-Rate INCONSISTENCY0.250.22201.10E-0168.89%100.00%0.00%INCONSISTENCY0.270.22241.06E-0168.89%100.00%0.00%INCONSISTENCY0.290.22271.06E-0168.89%100.00%0.00%INCONSISTENCY0.310.22331.05E-0168.89%100.00%0.00%INCONSISTENCY0.330.22529.83E-0268.89%100.00%0.00%INCONSISTENCY0.350.22669.63E-0269.00%99.64%0.00%INCONSISTENCY0.370.22749.50E-0269.00%99.64%0.00%INCONSISTENCY0.390.22788.84E-0269.11%99.29%0.00%INCONSISTENCY0.410.22528.64E-0269.11%99.29%0.00%INCONSISTENCY0.430.22497.89E-0269.11%99.29%0.00%INCONSISTENCY0.450.23566.09E-0269.56%97.86%0.00%LIU'SCONFLICT0.10.10.60541.49E-0468.56%100.00%0.00%LIU'SCONFLICT0.20.10.60661.43E-0468.56%100.00%0.00%LIU'SCONFLICT0.30.10.60691.42E-0468.56%100.00%0.00%LIU'SCONFLICT0.40.10.60771.38E-0470.11%95.05%0.00%LIU'SCONFLICT0.50.10.60961.01E-0463.11%69.61%21.88%LIU'SCONFLICT0.60.10.61089.74E-0538.44%19.08%81.04%LIU'SCONFLICT0.70.10.61478.53E-0534.67%1.77%94.49%LIU'SCONFLICT0.80.10.62106.88E-0533.00%0.00%97.73%LIU'SCONFLICT0.90.10.62744.21E-0531.89%0.00%99.35%LIU'SCONFLICT0.10.20.59841.47E-0468.56%100.00%0.00%LIU'SCONFLICT0.20.20.59921.43E-0468.56%100.00%0.00%LIU'SCONFLICT0.30.20.59921.43E-0468.56%100.00%0.00%LIU'SCONFLICT0.40.20.59981.40E-0468.56%100.00%0.00%LIU'SCONFLICT0.50.20.60171.32E-0468.67%99.65%0.00%LIU'SCONFLICT0.60.20.60291.27E-0453.33%90.81%26.42%LIU'SCONFLICT0.70.20.60678.76E-0529.11%67.14%72.61%LIU'SCONFLICT0.80.20.61336.98E-0532.33%25.09%87.20%LIU'SCONFLICT0.90.20.61995.54E-0533.56%4.24%94.98%LIU'SCONFLICT0.10.30.58761.64E-0468.56%100.00%0.00%LIU'SCONFLICT0.20.30.58841.60E-0468.56%100.00%0.00%LIU'SCONFLICT0.30.30.58861.59E-0468.56%100.00%0.00%LIU'SCONFLICT0.40.30.58961.54E-0468.56%100.00%0.00%LIU'SCONFLICT0.50.30.59261.40E-0468.56%100.00%0.00%LIU'SCONFLICT0.60.30.59461.04E-0467.56%100.00%1.46%LIU'SCONFLICT0.70.30.59998.71E-0560.44%100.00%11.83%LIU'SCONFLICT0.80.30.60855.06E-0547.67%99.29%30.79%LIU'SCONFLICT0.90.30.61743.64E-0528.89%95.41%59.97%LIU'SCONFLICT0.10.40.55861.80E-0468.56%100.00%0.00%LIU'SCONFLICT0.20.40.55921.76E-0468.56%100.00%0.00%LIU'SCONFLICT0.30.40.56041.69E-0468.56%100.00%0.00%LIU'SCONFLICT0.40.40.56191.61E-0468.56%100.00%0.00%LIU'SCONFLICT0.50.40.56431.49E-0468.56%100.00%0.00% Continuedonnextpage 199

PAGE 214

AppendixAContinuedTable33ContinuedMethodT1T2rp%CorrectFP-RateFN-Rate LIU'SCONFLICT0.60.40.56651.39E-0468.56%100.00%0.00%LIU'SCONFLICT0.70.40.56981.01E-0468.56%100.00%0.00%LIU'SCONFLICT0.80.40.57727.78E-0568.56%100.00%0.00%LIU'SCONFLICT0.90.40.58464.80E-0567.11%99.29%2.43%LIU'SCONFLICT0.10.50.54651.81E-0468.56%100.00%0.00%LIU'SCONFLICT0.20.50.54661.80E-0468.56%100.00%0.00%LIU'SCONFLICT0.30.50.54751.75E-0468.56%100.00%0.00%LIU'SCONFLICT0.40.50.54961.64E-0468.56%100.00%0.00%LIU'SCONFLICT0.50.50.55241.49E-0468.56%100.00%0.00%LIU'SCONFLICT0.60.50.55461.38E-0468.56%100.00%0.00%LIU'SCONFLICT0.70.50.55791.01E-0468.56%100.00%0.00%LIU'SCONFLICT0.80.50.56577.71E-0568.56%100.00%0.00%LIU'SCONFLICT0.90.50.57344.73E-0568.56%100.00%0.00%LIU'SCONFLICT0.10.60.55411.16E-0468.56%100.00%0.00%LIU'SCONFLICT0.20.60.55411.16E-0468.56%100.00%0.00%LIU'SCONFLICT0.30.60.55441.15E-0468.56%100.00%0.00%LIU'SCONFLICT0.40.60.55611.08E-0468.56%100.00%0.00%LIU'SCONFLICT0.50.60.55889.84E-0568.56%100.00%0.00%LIU'SCONFLICT0.60.60.56069.23E-0568.56%100.00%0.00%LIU'SCONFLICT0.70.60.56326.88E-0568.56%100.00%0.00%LIU'SCONFLICT0.80.60.57115.16E-0568.56%100.00%0.00%LIU'SCONFLICT0.90.60.57843.16E-0568.56%100.00%0.00%LIU'SCONFLICT0.10.70.54591.05E-0468.56%100.00%0.00%LIU'SCONFLICT0.20.70.54591.05E-0468.56%100.00%0.00%LIU'SCONFLICT0.30.70.54591.05E-0468.56%100.00%0.00%LIU'SCONFLICT0.40.70.54591.05E-0468.56%100.00%0.00%LIU'SCONFLICT0.50.70.54799.80E-0568.56%100.00%0.00%LIU'SCONFLICT0.60.70.54949.27E-0568.56%100.00%0.00%LIU'SCONFLICT0.70.70.55098.80E-0568.56%100.00%0.00%LIU'SCONFLICT0.80.70.55805.61E-0568.56%100.00%0.00%LIU'SCONFLICT0.90.70.56373.69E-0568.56%100.00%0.00%LIU'SCONFLICT0.10.80.53869.41E-0568.56%100.00%0.00%LIU'SCONFLICT0.20.80.53869.41E-0568.56%100.00%0.00%LIU'SCONFLICT0.30.80.53869.41E-0568.56%100.00%0.00%LIU'SCONFLICT0.40.80.53869.41E-0568.56%100.00%0.00%LIU'SCONFLICT0.50.80.53869.41E-0568.56%100.00%0.00%LIU'SCONFLICT0.60.80.53869.41E-0568.56%100.00%0.00%LIU'SCONFLICT0.70.80.53979.04E-0568.56%100.00%0.00%LIU'SCONFLICT0.80.80.54397.78E-0568.56%100.00%0.00%LIU'SCONFLICT0.90.80.54685.80E-0568.56%100.00%0.00%LIU'SCONFLICT0.10.90.43817.33E-0468.89%98.94%0.00%LIU'SCONFLICT0.20.90.43817.33E-0468.89%98.94%0.00%LIU'SCONFLICT0.30.90.43817.33E-0468.89%98.94%0.00% Continuedonnextpage 200

PAGE 215

AppendixAContinuedTable33ContinuedMethodT1T2rp%CorrectFP-RateFN-Rate LIU'SCONFLICT0.40.90.43817.33E-0468.89%98.94%0.00%LIU'SCONFLICT0.50.90.43817.33E-0468.89%98.94%0.00%LIU'SCONFLICT0.60.90.43817.33E-0468.89%98.94%0.00%LIU'SCONFLICT0.70.90.43817.33E-0468.89%98.94%0.00%LIU'SCONFLICT0.80.90.43817.33E-0468.89%98.94%0.00%LIU'SCONFLICT0.90.90.43051.04E-0368.89%98.94%0.00%TBMCONFLICT0.10.62281.09E-0468.56%100.00%0.00%TBMCONFLICT0.140.62301.08E-0468.56%100.00%0.00%TBMCONFLICT0.180.62321.07E-0468.56%100.00%0.00%TBMCONFLICT0.220.62361.06E-0468.56%100.00%0.00%TBMCONFLICT0.260.62381.05E-0468.56%100.00%0.00%TBMCONFLICT0.30.62391.05E-0468.56%100.00%0.00%TBMCONFLICT0.340.62411.04E-0468.56%100.00%0.00%TBMCONFLICT0.380.62471.02E-0468.56%100.00%0.00%TBMCONFLICT0.420.62491.01E-0468.56%100.00%0.00%TBMCONFLICT0.460.62559.94E-0568.56%100.00%0.00%TBMCONFLICT0.50.62589.82E-0568.56%100.00%0.00%TBMCONFLICT0.540.62649.63E-0568.56%100.00%0.00%TBMCONFLICT0.580.62659.59E-0568.56%100.00%0.00%TBMCONFLICT0.620.62699.48E-0568.56%100.00%0.00%TBMCONFLICT0.660.62819.09E-0568.56%100.00%0.00%TBMCONFLICT0.70.62958.69E-0568.56%100.00%0.00%TBMCONFLICT0.740.63138.16E-0568.56%100.00%0.00%TBMCONFLICT0.780.63287.76E-0568.56%100.00%0.00%TBMCONFLICT0.820.63437.37E-0568.56%100.00%0.00%TBMCONFLICT0.860.63676.80E-0568.56%100.00%0.00%TBMCONFLICT0.90.63796.51E-0568.56%100.00%0.00% 201

PAGE 216

AppendixBDataCollection:ExperimentsThisappendixcontainsdetailedinformationonthetestbedsanddatacollectionprocessfortheadditionalexperimentsdescribedinChapter5.Table34givestherandomlygeneratedstart-ingpositionfortherobotandtherandomlygeneratedpositionofthedarkobstacleplacedinodd-numberedrunsinthebridge,lab,andwalkwaytestbeds(seeforexampleFigure32(a)).Thestart-ingpositionisrelativetoazeropositionselectedforeaseofmeasurementwithineachtestbed,alwaysfourmetersfromaglasssurfaceinthosetestbedswithglass(bridge,sidewalk,andlab).Theobstaclepositionsarerelativetothestartingpositionfortheirrespectiveruns.Table34.Randomlygeneratedstartingpositions(meters)andobstaclepositions(meters)foreachtestbed.denotestestbedsusedintheexploratoryphaseoftheexperiments. Testbed Startingpositions Obstaclepositions Lab 2.7,2.3,1.0,1.9,0.5,0.8,0.4,2.8 0.0,0.1,0.2,0.3 Walkway 0.1,0.6,0.6,1.2,1.4,2.0,2.4,2.5 0.2,0.1,0.0,0.0 Bridge 1.6,1.2,0.0,0.4,3.2,2.2,0.7,3.5,0.3,2.9 0.15,0.15,0.05,0.25,0.33 Cubicle 1.1,1.2,1.3,2.1,2.4,2.7,3.0,3.1,3.4,3.7 Sidewalk 0.0,0.4,0.8,0.9,1.5,1.6,1.8,2.2,2.3,3.8 FordatacollectionaSICKLMSlaserrangescannerandCanestarangecamerawereinstalledonanATRV-Jr(seeFigure32(b))whichwasteleoperateddownthecenterofthetestbedtoadis-tanceof6.0metersfromthestartingposition.Toreducelocalizationerrorsduetoodometryer-ror,aducialwasmountedontherobotwhichenabledapositioningsystem(usinganotherSICKLMS)placed7.5metersfromthestartingpointtodeterminethelocationoftherobotasthesensorreadingswerecollected.TheducialconsistedofcardboardtapedtotwoPVCpipesmountedatthefrontoftherobot(seeFigure32(c)).Thepositioningsystemfoundthepointsassociatedwiththeducialandperformedlinearregressiononthosepointstodeterminetherobotslocationandorientationwithinthetestbed.BoththesensorreadingsandlocalizationinformationwerestoredwithtimestampsandthecomputersinvolvedusedNTPtomaintainagreementbetweentheirre-spectiveclocks.Thisenabledanofinesystem(usinglinearinterpolation)tocalculateapositionandorientationforeachsensorreading.Theresultswerecomparedtotheschematicsforeachtestbedandwerefoundtobesuperiortoodometry(whichshowederrorsofameterormore)withnearzeroerroralongthex-axis(directiontherobotisdriving)andminorerrorsalong202

PAGE 217

AppendixBContinued aWalkwaytestbedwithadarkobstacle bClose-upofsensorsinthebridgetestbed cLaser-basedpositioningsysteminthecubicletestbedFigure32.Picturestakenduringdatacollection.they-axiscenteringwithinthetestbedandorientation.Duetotherange.0metersandeldofview180oftherobot'slaser,theseminororientationandy-axiserrorscouldpreventaccu-ratereadingsfromliningupproperlyonthegroundtruthoccupancygridsusedintheexperimentsdescribedinChapter5.Anautomatedsystemwasthereforeimplementedtocorrecttheseminorerrorsusingalocal-beamsearchtoadjusttheorientationandy-axisvaluestominimizethetotalerrorinthelaserreadingsforeachscan.ThecorrectedpositionsforthelaserscanswereappliedtotheCanestarangecamerareadingsusinglinearinterpolationbasedontheirrespectivetimestamps.Thefollowingguresshowgraphicalrepresentationsofthevetestbedsusedinthedatacol-lectionandanalysisprocess.Figures33athrough33eshowschematicsfortheveselectedtestbeds.Ineachtestbedtherobottraveledforsixmetersfromitsstartingpositiontowardtherightendoftheschematic.Figures34athrough34eshowthegroundtruthoccupancygridsusedinexperimentsdescribedinChapter5todeterminetheaccuracyofthegridsbuiltfromlaserandCanestarangecamerasensorreadings.Figures35athrough35egraphicallyshowtheexpectedsourceoferrororinconsistencyforlaserreadingstakeninthattestbed.Thesemapswereusedtodeterminewhydetectionofsensinganomaliesfailedinsomeofthetestbedenvironmentsbyanalyzinghowmuchoftheerrororinconsistencywasduetolocalizationorquantizationerrors,sensornoise,orsensinganomalies.ThesamemapsfortheCanestarangecamerainnormalandmolemodeareshowninFigures36athrough36eandFigures37athrough37erespectively.203

PAGE 218

AppendixBContinued aLab bWalkway cBridge dCubicle eSidewalkFigure33.SchematicsforeachATRV-Jrtestbed.Basedonmanualmeasurementsperformedbytheauthor.204

PAGE 219

AppendixBContinued aLab bWalkway cBridge dCubicle eSidewalkFigure34.AutomaticallygeneratedoccupancygridsforeachATRV-Jrtestbed.red=occupied,green=empty,andblue=unknown.205

PAGE 220

AppendixBContinued aLab bWalkway cBridge dCubicle eSidewalkFigure35.Expectederrorsourcemapsforlaser.red=sensinganomalies,yellow=localizationerrors,green=sensornoise,blue=unknown.206

PAGE 221

AppendixBContinued aLab bWalkway cBridge dCubicle eSidewalkFigure36.Expectederrorsourcemapsfortherangecamerainnormalmode.red=sensinganomalies,yellow=localizationerrors,green=sensornoise,blue=unknown.207

PAGE 222

AppendixBContinued aLab bWalkway cBridge dCubicle eSidewalkFigure37.Expectederrorsourcemapsfortherangecamerainmolemode.red=sensinganoma-lies,yellow=localizationerrors,green=sensornoise,blue=unknown.208

PAGE 223

AppendixCDetailedResults:ExperimentsThisappendixcontainstherawresultsfortheexperimentsdescribedinChapter5.Therstthreetablesgivetherawresultsontheestimationanddetectioncomponentsforall184indicatorsforbothlaserandCanestarangecamera,justlaser,andjustCanestarangecamerareadingsre-spectively.InthesetablesT1andT2refertotherstandsecond(onlyusedbyLIUSCONFLICT)thresholdvaluesassignedtotheindicatortobetested.Pearsonslinearcorrelationcoefcientisgivenasr.Thenumberofsamplesnusedinthecorrelationanalysisinthetrainingphasewiththeexploratorytestbedsis528,176,and352forboth,laseronly,andCanestaonlyresultsetsrespec-tively.Vericationresultsfromtheevaluationtestbedsareincludedatthebottomofthetableandnfortheseresultsare990,330,and660forboth,laseronly,andCanestaonlyresultsetsrespec-tively.n!givesthecorrectedsamplesizeusedtondthep-valueisgivenasp.Correctreferstothepercentageofcorrectlyclassiedoccupancygridsregardingthepresenceofsensinganomalies.FP-RateandFN-Ratearethefalsepositiveandfalsenegativeratesforsensinganomalies.Table35:Rawresultsforestimationanddetectionforall184indicators.PalreferstoPalsIN-CONSISTENCY,LiutoLIUSCONFLICT,andTBMtoTBMCONFLICT.n=528GenericClassicationAdHocMethodT1T2n!rpCorrectFP-RateFN-RateCorrect ANXIETY0.1540.66903.22E-0839.48%0.06%99.69%56.64%ANXIETY0.14540.66893.22E-0839.46%0.05%99.73%56.45%ANXIETY0.18550.67231.90E-0839.46%0.05%99.74%56.67%ANXIETY0.22560.67181.44E-0839.45%0.05%99.75%57.02%ANXIETY0.26560.67351.29E-0839.45%0.04%99.75%57.18%ANXIETY0.3570.67717.31E-0939.45%0.03%99.77%56.16%ANXIETY0.34570.68274.94E-0939.44%0.00%99.79%56.06%ANXIETY0.38590.68971.53E-0939.44%0.00%99.80%55.39%ANXIETY0.42590.69461.04E-0939.44%0.00%99.80%56.13%ANXIETY0.46590.70484.64E-1039.39%0.00%99.89%55.06%ANXIETY0.5600.68611.45E-0939.32%0.00%99.99%51.98%ANXIETY0.54630.67919.50E-1039.32%0.00%100.00%52.66%ANXIETY0.58670.66309.82E-1039.32%0.00%100.00%52.97%ANXIETY0.62930.62043.31E-1139.32%0.00%100.00%58.75%ANXIETY0.66950.53662.09E-0839.32%0.00%100.00%56.38%ANXIETY0.7940.48268.38E-0739.32%0.00%100.00%50.41%ANXIETY0.74910.35944.67E-0439.32%0.00%100.00%43.89% Continuedonnextpage 209

PAGE 224

AppendixCContinuedTable35ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect ANXIETY0.785280.00001.00E+0039.32%0.00%100.00%42.66%ANXIETY0.825280.00001.00E+0039.32%0.00%100.00%42.44%ANXIETY0.865280.00001.00E+0039.32%0.00%100.00%39.33%ANXIETY0.95280.00001.00E+0039.32%0.00%100.00%39.33%CON0.25390.55512.45E-0475.61%46.39%10.13%56.36%CON0.5390.55512.45E-0475.60%46.36%10.17%56.36%CON0.75390.55512.45E-0476.17%44.80%10.24%56.36%CON1390.55512.45E-0476.37%43.93%10.47%56.36%CON1.25390.55512.45E-0476.71%42.67%10.74%56.35%CON1.5390.55512.45E-0477.21%41.19%10.86%56.35%CON1.75390.55512.45E-0477.45%40.44%10.96%56.35%CON2390.55512.45E-0477.83%39.16%11.16%56.35%CON2.25390.55512.45E-0478.31%37.33%11.56%56.35%CON2.5390.55512.45E-0478.58%36.45%11.68%56.35%CON2.75390.55512.45E-0478.56%36.05%11.97%56.35%CON3390.55502.45E-0478.91%35.01%12.08%56.35%CON3.25390.55502.45E-0478.93%34.68%12.25%56.38%CON3.5390.55502.45E-0479.14%34.01%12.34%56.38%CON3.75390.55502.45E-0479.29%33.25%12.60%56.38%CON4390.55502.45E-0479.42%32.40%12.92%56.38%CON4.25390.55502.46E-0479.38%32.09%13.19%56.38%CON4.5390.55502.46E-0478.83%32.05%14.13%56.40%CON4.75390.55502.46E-0478.61%31.48%14.86%56.39%CON5390.55492.46E-0478.59%31.00%15.19%56.39%GAMBINO0.5350.96950.00E+0076.23%44.47%10.35%60.68%GAMBINO1350.96950.00E+0076.23%44.47%10.35%56.36%GAMBINO1.5340.90044.19E-1339.32%0.00%100.00%56.36%GAMBINO2340.90044.19E-1339.32%0.00%100.00%55.35%GAMBINO2.55280.00001.00E+0039.32%0.00%100.00%55.35%GAMBINO35280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO3.55280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO45280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO4.55280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO55280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO5.55280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO65280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO6.55280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO75280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO7.55280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO85280.00001.00E+0039.32%0.00%100.00%39.32% Continuedonnextpage 210

PAGE 225

AppendixCContinuedTable35ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect GAMBINO8.55280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO95280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO9.55280.00001.00E+0039.32%0.00%100.00%39.32%GAMBINO105280.00001.00E+0039.32%0.00%100.00%39.32%Pal0.05530.68052.07E-0839.50%0.10%99.63%56.62%Pal0.07540.68111.45E-0839.50%0.08%99.66%56.59%Pal0.09550.68141.03E-0839.48%0.06%99.69%56.63%Pal0.11550.68209.88E-0939.47%0.05%99.71%56.63%Pal0.13560.68744.89E-0939.46%0.05%99.73%56.60%Pal0.15570.68673.68E-0939.46%0.05%99.73%56.64%Pal0.17580.68792.44E-0939.46%0.05%99.74%56.76%Pal0.19590.69141.34E-0939.45%0.04%99.75%57.12%Pal0.21600.70393.53E-1039.45%0.03%99.76%56.25%Pal0.23620.71934.52E-1139.45%0.00%99.78%56.26%Pal0.25780.63783.39E-1039.32%0.00%99.99%71.85%Pal0.27780.63593.96E-1039.32%0.00%99.99%71.73%Pal0.29780.64202.38E-1039.32%0.00%99.99%71.19%Pal0.31770.64123.34E-1039.32%0.00%99.99%70.53%Pal0.33780.64561.75E-1039.32%0.00%99.99%69.49%Pal0.35770.63425.92E-1039.32%0.00%99.99%68.33%Pal0.37800.64141.46E-1039.32%0.00%100.00%70.86%Pal0.39810.63442.03E-1039.32%0.00%100.00%70.71%Pal0.41820.61041.14E-0939.32%0.00%100.00%67.87%Pal0.43840.59652.14E-0939.32%0.00%100.00%67.26%Pal0.45890.52841.03E-0739.32%0.00%100.00%65.46%Liu0.10.1350.96030.00E+0081.63%27.09%12.73%56.35%Liu0.20.1350.96030.00E+0081.63%27.09%12.73%56.33%Liu0.30.1350.96030.00E+0081.63%27.09%12.73%56.35%Liu0.40.1350.96020.00E+0081.87%26.46%12.73%56.67%Liu0.50.1350.95990.00E+0082.48%24.79%12.81%56.43%Liu0.60.1350.95960.00E+0082.54%24.08%13.17%53.29%Liu0.70.1350.95920.00E+0082.58%23.31%13.60%45.98%Liu0.80.1350.95820.00E+0083.75%19.71%14.01%35.89%Liu0.90.1350.95750.00E+0084.37%16.24%15.23%33.83%Liu0.10.2350.95670.00E+0079.69%3.29%31.34%56.34%Liu0.20.2350.95670.00E+0079.69%3.29%31.34%56.34%Liu0.30.2350.95660.00E+0079.68%3.29%31.36%56.41%Liu0.40.2350.95650.00E+0079.70%3.19%31.40%56.30%Liu0.50.2350.95620.00E+0079.68%2.89%31.62%56.37%Liu0.60.2350.95590.00E+0079.39%2.88%32.10%56.27% Continuedonnextpage 211

PAGE 226

AppendixCContinuedTable35ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect Liu0.70.2350.95550.00E+0078.80%2.72%33.17%55.64%Liu0.80.2350.95470.00E+0078.34%2.50%34.07%54.61%Liu0.90.2350.95400.00E+0077.84%2.25%35.07%51.58%Liu0.10.3350.95120.00E+0067.89%0.86%52.36%56.40%Liu0.20.3350.95120.00E+0067.89%0.86%52.36%56.40%Liu0.30.3350.95120.00E+0067.83%0.86%52.45%56.40%Liu0.40.3350.95110.00E+0067.74%0.86%52.61%56.40%Liu0.50.3350.95080.00E+0067.61%0.83%52.84%56.41%Liu0.60.3350.95060.00E+0067.31%0.83%53.34%56.43%Liu0.70.3350.95030.00E+0067.04%0.81%53.79%56.42%Liu0.80.3350.94970.00E+0066.65%0.76%54.47%56.41%Liu0.90.3350.94910.00E+0066.08%0.76%55.41%56.41%Liu0.10.4350.95080.00E+0067.58%0.81%52.90%56.41%Liu0.20.4350.95080.00E+0067.58%0.81%52.90%56.41%Liu0.30.4350.95080.00E+0067.52%0.81%52.99%56.41%Liu0.40.4350.95070.00E+0067.44%0.81%53.13%56.41%Liu0.50.4350.95050.00E+0067.31%0.78%53.36%56.41%Liu0.60.4350.95020.00E+0067.02%0.77%53.86%56.43%Liu0.70.4350.95000.00E+0066.71%0.77%54.37%56.42%Liu0.80.4350.94940.00E+0066.17%0.74%55.28%56.41%Liu0.90.4350.94880.00E+0065.60%0.73%56.22%56.41%Liu0.10.5350.95070.00E+0067.35%0.81%53.29%56.41%Liu0.20.5350.95070.00E+0067.35%0.81%53.29%56.41%Liu0.30.5350.95070.00E+0067.35%0.81%53.29%56.41%Liu0.40.5350.95060.00E+0067.26%0.80%53.43%56.41%Liu0.50.5350.95030.00E+0067.14%0.77%53.65%56.41%Liu0.60.5350.95010.00E+0066.73%0.77%54.34%56.42%Liu0.70.5350.94980.00E+0066.43%0.76%54.84%56.42%Liu0.80.5350.94920.00E+0065.88%0.73%55.75%56.42%Liu0.90.5350.94870.00E+0065.27%0.73%56.75%56.41%Liu0.10.6350.95040.00E+0067.16%0.80%53.59%56.42%Liu0.20.6350.95040.00E+0067.16%0.80%53.59%56.42%Liu0.30.6350.95040.00E+0067.16%0.80%53.59%56.42%Liu0.40.6350.95040.00E+0067.14%0.80%53.63%56.42%Liu0.50.6350.95010.00E+0067.01%0.77%53.86%56.42%Liu0.60.6350.94990.00E+0066.57%0.76%54.60%56.43%Liu0.70.6350.94960.00E+0066.12%0.76%55.34%56.43%Liu0.80.6350.94900.00E+0065.63%0.73%56.17%56.42%Liu0.90.6350.94840.00E+0065.05%0.72%57.12%56.42%Liu0.10.7350.94970.00E+0066.71%0.72%54.39%56.43% Continuedonnextpage 212

PAGE 227

AppendixCContinuedTable35ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect Liu0.20.7350.94970.00E+0066.71%0.72%54.39%56.43%Liu0.30.7350.94970.00E+0066.71%0.72%54.39%56.43%Liu0.40.7350.94970.00E+0066.71%0.72%54.39%56.43%Liu0.50.7350.94960.00E+0066.59%0.70%54.61%56.43%Liu0.60.7350.94930.00E+0066.07%0.69%55.47%56.44%Liu0.70.7350.94910.00E+0065.64%0.69%56.17%56.43%Liu0.80.7350.94850.00E+0065.21%0.67%56.90%56.42%Liu0.90.7350.94800.00E+0064.66%0.66%57.82%56.42%Liu0.10.8350.95360.00E+0049.17%0.01%83.75%56.30%Liu0.20.8350.95360.00E+0049.17%0.01%83.75%56.30%Liu0.30.8350.95360.00E+0049.17%0.01%83.75%56.30%Liu0.40.8350.95360.00E+0049.17%0.01%83.75%56.30%Liu0.50.8350.95360.00E+0049.17%0.01%83.75%56.30%Liu0.60.8350.95360.00E+0049.17%0.01%83.75%56.30%Liu0.70.8350.95340.00E+0048.74%0.01%84.47%56.29%Liu0.80.8350.95290.00E+0048.39%0.00%85.04%56.27%Liu0.90.8350.95260.00E+0048.09%0.00%85.55%56.28%Liu0.10.9340.95360.00E+0039.32%0.00%100.00%56.20%Liu0.20.9340.95360.00E+0039.32%0.00%100.00%56.20%Liu0.30.9340.95360.00E+0039.32%0.00%100.00%56.20%Liu0.40.9340.95360.00E+0039.32%0.00%100.00%56.20%Liu0.50.9340.95360.00E+0039.32%0.00%100.00%56.20%Liu0.60.9340.95360.00E+0039.32%0.00%100.00%56.20%Liu0.70.9340.95360.00E+0039.32%0.00%100.00%56.20%Liu0.80.9340.95360.00E+0039.32%0.00%100.00%56.20%Liu0.90.9340.95330.00E+0039.32%0.00%100.00%56.20%TBM0.1350.96730.00E+0075.61%46.39%10.13%56.36%TBM0.14350.96730.00E+0075.61%46.39%10.13%56.36%TBM0.18350.96730.00E+0075.61%46.39%10.13%56.36%TBM0.22350.96730.00E+0075.61%46.39%10.13%56.36%TBM0.26350.96730.00E+0075.61%46.39%10.13%56.36%TBM0.3350.96730.00E+0075.60%46.39%10.15%56.36%TBM0.34350.96730.00E+0075.60%46.39%10.15%56.36%TBM0.38350.96730.00E+0075.61%46.36%10.15%56.36%TBM0.42350.96730.00E+0075.75%45.90%10.22%56.36%TBM0.46350.96720.00E+0075.85%45.63%10.24%56.36%TBM0.5350.96710.00E+0076.07%45.07%10.24%56.36%TBM0.54350.96710.00E+0076.17%44.79%10.24%56.37%TBM0.58350.96720.00E+0076.23%44.39%10.40%56.37%TBM0.62350.96730.00E+0076.38%43.93%10.46%56.36% Continuedonnextpage 213

PAGE 228

AppendixCContinuedTable35ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect TBM0.66350.96730.00E+0076.49%43.47%10.57%56.35%TBM0.7350.96740.00E+0076.65%42.91%10.68%56.35%TBM0.74350.96730.00E+0076.87%42.15%10.81%56.35%TBM0.78350.96680.00E+0077.21%41.19%10.86%56.35%TBM0.82350.96680.00E+0077.39%40.59%10.95%56.35%TBM0.86350.96680.00E+0077.83%39.17%11.16%56.35%TBM0.9350.96650.00E+0078.30%37.25%11.63%56.35%Evaluationtestbeds Liu0.90.1790.91160.00E+0055.70%75.05%21.71%TBM0.18840.91790.00E+0061.16%80.61%8.15%Pal0.251220.26373.34E-0357.91% Table36:Rawresultsforestimationanddetectionforlaserreadingsonly.PalreferstoPal'sIN-CONSISTENCY,LiutoLIU'SCONFLICT,andTBMtoTBMCONFLICT.n=176GenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect ANXIETY0.1240.74093.46E-0549.04%0.08%99.97%50.95%ANXIETY0.14230.73985.46E-0549.04%0.06%99.97%51.00%ANXIETY0.18240.74622.83E-0549.04%0.06%99.97%51.02%ANXIETY0.22250.75201.46E-0549.04%0.06%99.97%51.05%ANXIETY0.26250.75111.52E-0549.05%0.05%99.97%51.08%ANXIETY0.3240.75871.73E-0549.05%0.03%99.99%51.16%ANXIETY0.34270.76782.94E-0649.06%0.01%100.00%51.34%ANXIETY0.38290.78823.84E-0749.06%0.00%100.00%51.44%ANXIETY0.42310.79817.55E-0849.06%0.00%100.00%51.48%ANXIETY0.46320.81771.11E-0849.06%0.00%100.00%51.51%ANXIETY0.5320.78321.16E-0749.06%0.00%100.00%51.63%ANXIETY0.54300.75141.70E-0649.06%0.00%100.00%55.01%ANXIETY0.58290.71261.45E-0549.06%0.00%100.00%56.80%ANXIETY0.62380.68202.42E-0649.06%0.00%100.00%68.06%ANXIETY0.66340.58342.93E-0449.06%0.00%100.00%66.16%ANXIETY0.7280.54292.83E-0349.06%0.00%100.00%59.39%ANXIETY0.74200.36131.18E-0149.06%0.00%100.00%55.78%ANXIETY0.781760.00001.00E+0049.06%0.00%100.00%54.57%ANXIETY0.821760.00001.00E+0049.06%0.00%100.00%54.20%ANXIETY0.861760.00001.00E+0049.06%0.00%100.00%49.07%ANXIETY0.91760.00001.00E+0049.06%0.00%100.00%49.07% Continuedonnextpage 214

PAGE 229

AppendixCContinuedTable36ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect CON0.25230.64429.09E-0471.49%55.89%2.14%50.94%CON0.5230.64429.09E-0471.47%55.85%2.21%50.94%CON0.75230.64429.09E-0472.13%54.37%2.36%50.94%CON1230.64429.09E-0472.13%54.32%2.40%50.94%CON1.25230.64419.09E-0472.54%53.37%2.51%50.94%CON1.5230.64419.10E-0473.35%51.61%2.60%50.94%CON1.75230.64419.10E-0473.58%51.10%2.65%50.94%CON2230.64419.10E-0474.23%49.75%2.68%50.94%CON2.25230.64419.10E-0475.15%47.84%2.71%50.94%CON2.5230.64419.10E-0475.67%46.75%2.73%50.95%CON2.75230.64419.10E-0475.77%46.50%2.79%50.94%CON3230.64419.11E-0476.37%45.22%2.84%50.94%CON3.25230.64419.11E-0476.48%44.93%2.90%50.96%CON3.5230.64419.11E-0476.87%44.07%2.96%50.96%CON3.75230.64409.12E-0477.32%43.08%3.03%50.96%CON4230.64409.12E-0477.85%41.96%3.07%50.96%CON4.25230.64409.12E-0478.01%41.57%3.14%50.96%CON4.5230.64409.12E-0478.02%41.53%3.16%50.96%CON4.75230.64409.13E-0478.37%40.80%3.18%50.96%CON5230.64409.13E-0478.66%40.16%3.21%50.96%GAMBINO0.5130.98282.04E-0972.34%53.70%2.57%50.94%GAMBINO1130.98282.04E-0972.34%53.70%2.57%50.94%GAMBINO1.5120.92701.45E-0549.06%0.00%100.00%50.96%GAMBINO2120.92701.45E-0549.06%0.00%100.00%50.96%GAMBINO2.51760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO31760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO3.51760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO41760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO4.51760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO51760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO5.51760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO61760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO6.51760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO71760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO7.51760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO81760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO8.51760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO91760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO9.51760.00001.00E+0049.06%0.00%100.00%49.06%GAMBINO101760.00001.00E+0049.06%0.00%100.00%49.06% Continuedonnextpage 215

PAGE 230

AppendixCContinuedTable36ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect Pal0.05230.74624.34E-0549.03%0.12%99.94%50.94%Pal0.07230.74584.41E-0549.04%0.10%99.94%50.94%Pal0.09230.74804.06E-0549.04%0.08%99.96%50.94%Pal0.11230.74834.03E-0549.04%0.07%99.97%50.94%Pal0.13230.75343.32E-0549.04%0.06%99.97%50.96%Pal0.15240.75551.97E-0549.04%0.06%99.97%50.98%Pal0.17230.76302.29E-0549.04%0.06%99.97%51.00%Pal0.19240.75931.69E-0549.05%0.06%99.97%51.01%Pal0.21240.77807.64E-0649.05%0.04%99.98%51.09%Pal0.23280.80342.65E-0749.06%0.01%100.00%51.18%Pal0.25370.80272.28E-0949.06%0.00%100.00%75.91%Pal0.27370.80232.37E-0949.06%0.00%100.00%76.06%Pal0.29350.79581.10E-0849.06%0.00%100.00%75.23%Pal0.31350.79081.58E-0849.06%0.00%100.00%75.34%Pal0.33360.79258.43E-0949.06%0.00%100.00%75.47%Pal0.35360.79001.01E-0849.06%0.00%100.00%75.53%Pal0.37520.79232.63E-1249.06%0.00%100.00%80.08%Pal0.39540.79021.20E-1249.06%0.00%100.00%79.87%Pal0.41510.75122.16E-1049.06%0.00%100.00%75.64%Pal0.43510.73091.14E-0949.06%0.00%100.00%75.47%Pal0.45580.72181.63E-1049.06%0.00%100.00%75.69%Liu0.10.1150.95602.69E-0881.10%34.84%3.54%50.94%Liu0.20.1150.95602.69E-0881.10%34.84%3.54%50.94%Liu0.30.1150.95602.71E-0881.10%34.84%3.55%50.94%Liu0.40.1150.95572.81E-0881.50%34.02%3.55%50.94%Liu0.50.1150.95483.20E-0882.47%31.97%3.63%50.92%Liu0.60.1150.95453.34E-0882.87%31.06%3.71%50.84%Liu0.70.1150.95393.61E-0883.21%30.14%3.94%48.55%Liu0.80.1150.95254.40E-0885.39%25.51%4.11%35.12%Liu0.90.1150.95184.85E-0887.39%21.04%4.49%36.55%Liu0.10.2150.94966.37E-0892.96%4.17%9.80%50.94%Liu0.20.2150.94966.37E-0892.96%4.17%9.80%50.94%Liu0.30.2150.94956.45E-0892.95%4.17%9.82%50.94%Liu0.40.2150.94926.73E-0892.98%4.03%9.89%50.94%Liu0.50.2150.94837.51E-0892.99%3.64%10.26%50.94%Liu0.60.2150.94817.74E-0892.89%3.63%10.45%50.94%Liu0.70.2150.94758.26E-0892.83%3.44%10.77%50.93%Liu0.80.2150.94619.80E-0892.70%3.18%11.27%50.89%Liu0.90.2150.94541.07E-0792.70%2.84%11.60%50.03%Liu0.10.3160.93718.88E-0884.19%1.01%30.07%50.94% Continuedonnextpage 216

PAGE 231

AppendixCContinuedTable36ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect Liu0.20.3160.93718.88E-0884.19%1.01%30.07%50.94%Liu0.30.3160.93698.98E-0884.09%1.01%30.25%50.94%Liu0.40.3160.93669.28E-0883.95%1.01%30.54%50.94%Liu0.50.3160.93619.89E-0883.79%0.97%30.89%50.94%Liu0.60.3160.93619.83E-0883.60%0.97%31.27%50.94%Liu0.70.3160.93561.04E-0783.48%0.95%31.52%50.94%Liu0.80.3160.93481.13E-0783.38%0.92%31.73%50.94%Liu0.90.3160.93421.20E-0783.17%0.92%32.15%50.94%Liu0.10.4160.93649.58E-0883.93%0.95%30.64%50.94%Liu0.20.4160.93649.58E-0883.93%0.95%30.64%50.94%Liu0.30.4160.93629.69E-0883.84%0.95%30.82%50.94%Liu0.40.4160.93591.00E-0783.70%0.94%31.09%50.94%Liu0.50.4160.93531.07E-0783.53%0.90%31.47%50.94%Liu0.60.4160.93541.06E-0783.34%0.90%31.84%50.94%Liu0.70.4160.93491.11E-0783.20%0.89%32.12%50.94%Liu0.80.4160.93421.21E-0783.07%0.89%32.37%50.94%Liu0.90.4160.93361.28E-0782.82%0.88%32.88%50.94%Liu0.10.5160.93639.63E-0883.70%0.94%31.09%50.94%Liu0.20.5160.93639.63E-0883.70%0.94%31.09%50.94%Liu0.30.5160.93629.73E-0883.70%0.94%31.09%50.94%Liu0.40.5160.93591.01E-0783.57%0.93%31.35%50.94%Liu0.50.5160.93531.07E-0783.39%0.90%31.74%50.94%Liu0.60.5160.93531.07E-0783.19%0.89%32.14%50.94%Liu0.70.5160.93491.11E-0783.07%0.88%32.39%50.94%Liu0.80.5160.93421.20E-0782.92%0.88%32.67%50.94%Liu0.90.5160.93361.28E-0782.68%0.88%33.16%50.94%Liu0.10.6160.93581.01E-0783.53%0.93%31.44%50.94%Liu0.20.6160.93581.01E-0783.53%0.93%31.44%50.94%Liu0.30.6160.93581.01E-0783.53%0.93%31.44%50.94%Liu0.40.6160.93571.03E-0783.49%0.93%31.52%50.94%Liu0.50.6160.93501.10E-0783.31%0.88%31.91%50.94%Liu0.60.6160.93511.09E-0783.10%0.88%32.33%50.94%Liu0.70.6160.93481.13E-0782.95%0.88%32.62%50.94%Liu0.80.6160.93401.22E-0782.83%0.88%32.86%50.94%Liu0.90.6160.93341.30E-0782.56%0.87%33.40%50.94%Liu0.10.7160.93411.21E-0783.00%0.82%32.58%50.94%Liu0.20.7160.93411.21E-0783.00%0.82%32.58%50.94%Liu0.30.7160.93411.21E-0783.00%0.82%32.58%50.94%Liu0.40.7160.93411.21E-0783.00%0.82%32.58%50.94%Liu0.50.7160.93391.23E-0782.82%0.80%32.96%50.94% Continuedonnextpage 217

PAGE 232

AppendixCContinuedTable36ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect Liu0.60.7160.93401.22E-0782.58%0.80%33.43%50.94%Liu0.70.7160.93381.26E-0782.52%0.80%33.55%50.94%Liu0.80.7160.93311.35E-0782.40%0.80%33.79%50.94%Liu0.90.7160.93251.43E-0782.21%0.78%34.18%50.94%Liu0.10.8160.93996.48E-0860.58%0.00%77.39%50.94%Liu0.20.8160.93996.48E-0860.58%0.00%77.39%50.94%Liu0.30.8160.93996.48E-0860.58%0.00%77.39%50.94%Liu0.40.8160.93996.48E-0860.58%0.00%77.39%50.94%Liu0.50.8160.93996.48E-0860.58%0.00%77.39%50.94%Liu0.60.8160.93996.48E-0860.58%0.00%77.39%50.94%Liu0.70.8160.93956.75E-0860.42%0.00%77.70%50.94%Liu0.80.8160.93897.25E-0860.23%0.00%78.07%50.94%Liu0.90.8160.93857.54E-0860.22%0.00%78.09%50.94%Liu0.10.9170.94926.23E-0949.06%0.00%100.00%50.94%Liu0.20.9170.94926.23E-0949.06%0.00%100.00%50.94%Liu0.30.9170.94926.23E-0949.06%0.00%100.00%50.94%Liu0.40.9170.94926.23E-0949.06%0.00%100.00%50.94%Liu0.50.9170.94926.23E-0949.06%0.00%100.00%50.94%Liu0.60.9170.94926.23E-0949.06%0.00%100.00%50.94%Liu0.70.9170.94926.23E-0949.06%0.00%100.00%50.94%Liu0.80.9170.94926.23E-0949.06%0.00%100.00%50.94%Liu0.90.9170.94906.41E-0949.06%0.00%100.00%50.95%TBM0.1140.98667.94E-1171.49%55.89%2.14%50.94%TBM0.14140.98667.94E-1171.49%55.89%2.14%50.94%TBM0.18140.98667.94E-1171.49%55.89%2.14%50.94%TBM0.22140.98667.94E-1171.49%55.89%2.14%50.94%TBM0.26140.98677.94E-1171.49%55.89%2.14%50.94%TBM0.3140.98677.94E-1171.46%55.89%2.19%50.94%TBM0.34140.98668.01E-1171.46%55.89%2.19%50.94%TBM0.38140.98668.29E-1171.48%55.86%2.19%50.94%TBM0.42140.98668.06E-1171.71%55.25%2.32%50.94%TBM0.46140.98658.54E-1171.83%54.98%2.35%50.94%TBM0.5140.98648.98E-1172.06%54.52%2.35%50.94%TBM0.54140.98629.69E-1172.13%54.36%2.36%50.94%TBM0.58140.98639.15E-1172.12%54.34%2.39%50.94%TBM0.62140.98619.93E-1172.13%54.32%2.40%50.94%TBM0.66140.98619.97E-1172.32%53.85%2.47%50.94%TBM0.7140.98629.70E-1172.39%53.68%2.50%50.94%TBM0.74140.98601.07E-1072.81%52.73%2.58%50.94%TBM0.78140.98551.29E-1073.36%51.60%2.60%50.94% Continuedonnextpage 218

PAGE 233

AppendixCContinuedTable36ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect TBM0.82140.98541.37E-1073.59%51.10%2.64%50.94%TBM0.86140.98521.49E-1074.22%49.76%2.68%50.94%TBM0.9140.98501.62E-1075.20%47.74%2.71%50.94% Evaluationtestbeds Liu0.40.2220.71222.00E-0440.46%39.07%81.27%TBM0.18260.83101.47E-0751.07%95.78%6.16%Pal0.37560.05546.85E-0158.94% Table37:Rawresultsforestimationanddetectionforrangecamerareadingsonly.PalreferstoPal'sINCONSISTENCY,LiutoLIU'SCONFLICT,andTBMtoTBMCONFLICT.n=352GenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect ANXIETY0.1400.41417.90E-0324.75%0.00%99.41%65.40%ANXIETY0.14410.41646.77E-0324.69%0.00%99.48%64.87%ANXIETY0.18420.41626.12E-0324.68%0.00%99.49%65.38%ANXIETY0.22420.40378.02E-0324.67%0.00%99.51%66.22%ANXIETY0.26440.41245.41E-0324.66%0.00%99.52%66.57%ANXIETY0.3450.41524.57E-0324.64%0.00%99.54%63.87%ANXIETY0.34430.42005.05E-0324.62%0.00%99.57%63.35%ANXIETY0.38450.41005.15E-0324.60%0.00%99.59%61.48%ANXIETY0.42470.41383.84E-0324.60%0.00%99.60%63.30%ANXIETY0.46490.42252.49E-0324.47%0.00%99.77%60.53%ANXIETY0.5500.41143.00E-0324.31%0.00%99.99%52.52%ANXIETY0.54540.44607.25E-0424.30%0.00%99.99%49.03%ANXIETY0.58540.44567.33E-0424.30%0.00%99.99%47.05%ANXIETY0.62550.44905.85E-0424.30%0.00%99.99%44.40%ANXIETY0.66590.38842.37E-0324.30%0.00%99.99%41.31%ANXIETY0.7720.29281.25E-0224.30%0.00%100.00%36.57%ANXIETY0.743520.00001.00E+0024.30%0.00%100.00%25.56%ANXIETY0.783520.00001.00E+0024.30%0.00%100.00%24.31%ANXIETY0.823520.00001.00E+0024.30%0.00%100.00%24.31%ANXIETY0.863520.00001.00E+0024.30%0.00%100.00%24.31%ANXIETY0.93520.00001.00E+0024.30%0.00%100.00%24.31%CON0.25410.05117.51E-0181.97%16.81%18.42%64.71%CON0.5410.05117.51E-0181.97%16.81%18.42%64.71%CON0.75410.05107.51E-0182.40%15.03%18.42%64.73%CON1410.05107.52E-0182.92%11.60%18.84%64.71% Continuedonnextpage 219

PAGE 234

AppendixCContinuedTable37ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect CON1.25410.05097.52E-0183.14%9.35%19.28%64.70%CON1.5410.05077.53E-0183.16%8.78%19.43%64.70%CON1.75410.05067.54E-0183.40%7.29%19.58%64.70%CON2410.05057.54E-0183.39%6.22%19.94%64.69%CON2.25410.05047.54E-0183.17%4.64%20.74%64.68%CON2.5410.05037.55E-0183.07%4.38%20.96%64.68%CON2.75410.05037.55E-0182.86%3.53%21.50%64.68%CON3410.05027.55E-0182.82%3.22%21.66%64.69%CON3.25410.05007.56E-0182.71%2.77%21.95%64.74%CON3.5410.04997.57E-0182.63%2.70%22.08%64.75%CON3.75410.04987.57E-0182.31%2.65%22.52%64.75%CON4410.04957.58E-0181.85%2.61%23.14%64.74%CON4.25410.04947.59E-0181.50%2.58%23.62%64.73%CON4.5410.04907.61E-0180.07%2.55%25.51%64.79%CON4.75410.04867.63E-0178.98%2.49%26.97%64.77%CON5410.04837.64E-0178.48%2.48%27.63%64.76%GAMBINO0.5550.84128.88E-1682.24%15.74%18.42%64.71%GAMBINO1550.84128.88E-1682.24%15.74%18.42%64.71%GAMBINO1.5560.57783.12E-0624.30%0.00%100.00%62.14%GAMBINO2560.57783.12E-0624.30%0.00%100.00%62.14%GAMBINO2.53520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO33520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO3.53520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO43520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO4.53520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO53520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO5.53520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO63520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO6.53520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO73520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO7.53520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO83520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO8.53520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO93520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO9.53520.00001.00E+0024.30%0.00%100.00%24.30%GAMBINO103520.00001.00E+0024.30%0.00%100.00%24.30%Pal0.05390.45173.89E-0324.81%0.03%99.32%65.38%Pal0.07390.45453.65E-0324.78%0.00%99.36%65.30%Pal0.09390.45303.78E-0324.75%0.00%99.41%65.41%Pal0.11400.45533.16E-0324.72%0.00%99.44%65.41% Continuedonnextpage 220

PAGE 235

AppendixCContinuedTable37ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect Pal0.13410.46522.18E-0324.69%0.00%99.48%65.30%Pal0.15410.46102.42E-0324.69%0.00%99.48%65.37%Pal0.17410.45422.85E-0324.68%0.00%99.50%65.66%Pal0.19430.47261.37E-0324.66%0.00%99.52%66.54%Pal0.21440.48408.71E-0424.66%0.00%99.53%64.22%Pal0.23430.49697.00E-0424.64%0.00%99.55%64.10%Pal0.25490.55363.69E-0524.31%0.00%99.98%65.60%Pal0.27490.55074.14E-0524.31%0.00%99.98%65.05%Pal0.29490.55313.76E-0524.31%0.00%99.98%64.95%Pal0.31490.54315.53E-0524.31%0.00%99.98%63.11%Pal0.33490.54515.13E-0524.31%0.00%99.98%60.27%Pal0.35490.53846.60E-0524.31%0.00%99.99%57.23%Pal0.37490.53696.96E-0524.30%0.00%99.99%56.63%Pal0.39490.52949.17E-0524.30%0.00%99.99%56.58%Pal0.41490.53427.70E-0524.30%0.00%99.99%55.89%Pal0.43520.52586.25E-0524.30%0.00%99.99%54.61%Pal0.45550.45195.34E-0424.30%0.00%99.99%49.68%Liu0.10.1560.84772.22E-1682.44%2.94%22.25%64.68%Liu0.20.1560.84772.22E-1682.44%2.94%22.25%65.65%Liu0.30.1560.84772.22E-1682.44%2.94%22.25%64.69%Liu0.40.1560.84772.22E-1682.43%2.94%22.26%65.52%Liu0.50.1570.84710.00E+0082.50%2.46%22.33%64.92%Liu0.60.1570.84630.00E+0082.03%2.34%22.98%57.06%Liu0.70.1570.84510.00E+0081.63%2.04%23.62%42.03%Liu0.80.1560.84264.44E-1681.23%1.66%24.27%37.06%Liu0.90.1570.84074.44E-1679.72%1.30%26.37%29.63%Liu0.10.2570.84420.00E+0059.22%0.55%53.70%64.67%Liu0.20.2570.84420.00E+0059.22%0.55%53.70%64.67%Liu0.30.2570.84420.00E+0059.22%0.55%53.70%64.83%Liu0.40.2570.84420.00E+0059.21%0.55%53.70%64.57%Liu0.50.2570.84362.22E-1659.15%0.55%53.78%64.74%Liu0.60.2570.84282.22E-1658.58%0.52%54.55%64.50%Liu0.70.2570.84182.22E-1657.17%0.50%56.41%62.90%Liu0.80.2570.83974.44E-1656.20%0.40%57.73%60.35%Liu0.90.2570.83794.44E-1654.92%0.40%59.42%53.96%Liu0.10.3580.83674.44E-1642.76%0.40%75.48%64.83%Liu0.20.3580.83674.44E-1642.76%0.40%75.48%64.83%Liu0.30.3580.83674.44E-1642.76%0.40%75.48%64.83%Liu0.40.3580.83664.44E-1642.74%0.40%75.51%64.83%Liu0.50.3580.83634.44E-1642.67%0.40%75.61%64.84% Continuedonnextpage 221

PAGE 236

AppendixCContinuedTable37ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect Liu0.60.3580.83504.44E-1642.20%0.38%76.23%64.88%Liu0.70.3580.83454.44E-1641.70%0.38%76.89%64.87%Liu0.80.3580.83284.44E-1640.84%0.26%78.06%64.85%Liu0.90.3580.83106.66E-1639.73%0.26%79.53%64.83%Liu0.10.4580.83604.44E-1642.37%0.40%75.99%64.84%Liu0.20.4580.83604.44E-1642.37%0.40%75.99%64.84%Liu0.30.4580.83604.44E-1642.37%0.40%75.99%64.84%Liu0.40.4580.83594.44E-1642.36%0.40%76.01%64.84%Liu0.50.4580.83564.44E-1642.31%0.40%76.07%64.85%Liu0.60.4580.83434.44E-1641.85%0.38%76.69%64.88%Liu0.70.4580.83384.44E-1641.28%0.38%77.44%64.87%Liu0.80.4580.83214.44E-1640.10%0.26%79.05%64.85%Liu0.90.4580.83038.88E-1639.05%0.26%80.43%64.84%Liu0.10.5580.83544.44E-1642.13%0.40%76.32%64.85%Liu0.20.5580.83544.44E-1642.13%0.40%76.32%64.85%Liu0.30.5580.83544.44E-1642.13%0.40%76.32%64.85%Liu0.40.5580.83534.44E-1642.12%0.40%76.33%64.85%Liu0.50.5580.83494.44E-1642.08%0.40%76.38%64.85%Liu0.60.5580.83364.44E-1641.34%0.38%77.37%64.88%Liu0.70.5580.83314.44E-1640.77%0.38%78.12%64.87%Liu0.80.5580.83146.66E-1639.61%0.26%79.69%64.86%Liu0.90.5580.82968.88E-1638.44%0.26%81.23%64.85%Liu0.10.6580.83434.44E-1641.94%0.40%76.57%64.87%Liu0.20.6580.83434.44E-1641.94%0.40%76.57%64.87%Liu0.30.6580.83434.44E-1641.94%0.40%76.57%64.87%Liu0.40.6580.83434.44E-1641.94%0.40%76.57%64.87%Liu0.50.6580.83404.44E-1641.89%0.40%76.63%64.87%Liu0.60.6580.83264.44E-1641.08%0.38%77.71%64.90%Liu0.70.6580.83204.44E-1640.18%0.38%78.90%64.89%Liu0.80.6580.83038.88E-1639.10%0.26%80.36%64.86%Liu0.90.6580.82858.88E-1638.07%0.26%81.72%64.86%Liu0.10.7580.83364.44E-1641.60%0.40%77.02%64.88%Liu0.20.7580.83364.44E-1641.60%0.40%77.02%64.88%Liu0.30.7580.83364.44E-1641.60%0.40%77.02%64.88%Liu0.40.7580.83364.44E-1641.60%0.40%77.02%64.88%Liu0.50.7580.83334.44E-1641.57%0.40%77.06%64.88%Liu0.60.7580.83194.44E-1640.61%0.38%78.33%64.91%Liu0.70.7580.83136.66E-1639.62%0.38%79.63%64.90%Liu0.80.7580.82968.88E-1638.72%0.26%80.87%64.87%Liu0.90.7580.82791.11E-1537.60%0.26%82.34%64.86% Continuedonnextpage 222

PAGE 237

AppendixCContinuedTable37ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect Liu0.10.8580.82421.78E-1531.60%0.03%90.35%64.55%Liu0.20.8580.82421.78E-1531.60%0.03%90.35%64.55%Liu0.30.8580.82421.78E-1531.60%0.03%90.35%64.55%Liu0.40.8580.82421.78E-1531.60%0.03%90.35%64.55%Liu0.50.8580.82421.78E-1531.60%0.03%90.35%64.55%Liu0.60.8580.82421.78E-1531.60%0.03%90.35%64.55%Liu0.70.8570.82373.55E-1530.73%0.03%91.49%64.53%Liu0.80.8570.82284.00E-1530.14%0.00%92.28%64.49%Liu0.90.8570.82194.66E-1529.37%0.00%93.30%64.50%Liu0.10.9570.79791.07E-1324.30%0.00%100.00%64.31%Liu0.20.9570.79791.07E-1324.30%0.00%100.00%64.31%Liu0.30.9570.79791.07E-1324.30%0.00%100.00%64.31%Liu0.40.9570.79791.07E-1324.30%0.00%100.00%64.31%Liu0.50.9570.79791.07E-1324.30%0.00%100.00%64.31%Liu0.60.9570.79791.07E-1324.30%0.00%100.00%64.31%Liu0.70.9570.79791.07E-1324.30%0.00%100.00%64.31%Liu0.80.9570.79791.07E-1324.30%0.00%100.00%64.31%Liu0.90.9570.79751.14E-1324.30%0.00%100.00%64.29%TBM0.1550.83991.11E-1581.97%16.81%18.42%64.71%TBM0.14550.83991.11E-1581.97%16.81%18.42%64.71%TBM0.18550.83991.11E-1581.97%16.81%18.42%64.71%TBM0.22550.83991.11E-1581.97%16.81%18.42%64.71%TBM0.26550.83991.11E-1581.97%16.81%18.42%64.71%TBM0.3550.83991.11E-1581.97%16.81%18.42%64.71%TBM0.34550.83991.11E-1581.97%16.81%18.42%64.71%TBM0.38550.83991.11E-1581.97%16.81%18.42%64.71%TBM0.42550.83991.11E-1581.97%16.81%18.42%64.71%TBM0.46550.83981.11E-1582.04%16.53%18.42%64.71%TBM0.5550.83941.33E-1582.25%15.65%18.42%64.71%TBM0.54550.83921.33E-1582.40%15.01%18.43%64.73%TBM0.58550.83911.33E-1582.58%13.42%18.71%64.73%TBM0.62550.83881.33E-1582.93%11.60%18.83%64.73%TBM0.66550.83791.33E-1582.93%11.13%18.98%64.70%TBM0.7550.83751.55E-1583.20%9.40%19.17%64.70%TBM0.74550.83681.78E-1583.12%9.21%19.34%64.70%TBM0.78550.83432.66E-1583.16%8.78%19.43%64.70%TBM0.82550.83283.11E-1583.27%7.86%19.58%64.70%TBM0.86550.83223.55E-1583.39%6.22%19.94%64.69%TBM0.9550.82915.33E-1583.09%4.57%20.88%64.68% Continuedonnextpage 223

PAGE 238

AppendixCContinuedTable37ContinuedGenericClassicationAdHocMethodT1T2n0rpCorrectFP-RateFN-RateCorrect Evaluationtestbeds TBM0.86690.84150.00E+0070.51%49.04%18.83%TBM0.181010.85980.00E+0074.50%53.45%10.28%ANXIETY0.26870.64999.68E-1264.27% Thenextfourtablesgiverawresultsfortheisolationcomponentforall184indicators.ThersttwogiverawresultsusinglaserorCanestarangecamerareadings.Thenexttwopairoftablescontainresultsforjustlaser,andjustCanestarangecamerareadingsrespectively.InthesetablesT1andT2refertotherstandsecondonlyusedbyLIU'SCONFLICTthresholdvaluesassignedtotheindicatortobetested,treferstoStudent'ststatistic.Thenumberofsamplesnis528and990respectivelyinthetrainingandvericationtestbeds.n0givesthecorrectedsamplesizeusedtondthep-value.ErrorscoresaresummarizedbymeanandstandarddeviationacrossallsamplesforthebaselineandisolatingscenariosseeSection5.5.4.ThesetablesalsoprovidethemeanandstandarddeviationfortherecordedOverlapscores.224

PAGE 239

AppendixCContinuedTable38:Rawresultsforisolationforall184indicators.n=528BaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean ANXIETY0.161.9848.0985.3676.77-37.72%47-1.23382.20E-010.030.03ANXIETY0.1461.9848.0974.7961.27-20.67%46-1.31461.92E-010.020.03ANXIETY0.1861.9848.0972.5358.34-17.02%46-1.33701.85E-010.020.03ANXIETY0.2261.9848.0970.9756.62-14.50%46-1.33321.86E-010.020.03ANXIETY0.2661.9848.0969.6355.07-12.35%46-1.34671.81E-010.020.03ANXIETY0.361.9848.0968.3353.57-10.24%46-1.37021.74E-010.020.03ANXIETY0.3461.9848.0967.1852.33-8.38%46-1.38181.70E-010.020.03ANXIETY0.3861.9848.0966.0451.31-6.54%46-1.35101.80E-010.020.03ANXIETY0.4261.9848.0965.1450.53-5.10%46-1.38581.69E-010.020.03ANXIETY0.4661.9848.0964.4850.04-4.03%46-1.38591.69E-010.020.03ANXIETY0.561.9848.0963.5449.37-2.52%46-1.35641.78E-010.020.03ANXIETY0.5461.9848.0962.9248.87-1.52%46-1.28662.02E-010.010.02ANXIETY0.5861.9848.0962.4548.53-0.76%46-1.25062.14E-010.010.02ANXIETY0.6261.9848.0962.2148.34-0.38%46-1.06742.89E-010.000.01ANXIETY0.6661.9848.0962.1648.28-0.29%46-0.92323.58E-010.000.01ANXIETY0.761.9848.0962.1148.24-0.20%46-0.81234.19E-010.000.01ANXIETY0.7461.9848.0962.0248.14-0.07%46-0.57855.64E-010.000.00ANXIETY0.7861.9848.0962.0048.11-0.03%46-0.34797.29E-010.000.00ANXIETY0.8261.9848.0961.9948.10-0.02%46-0.37357.10E-010.000.00ANXIETY0.8661.9848.0961.9848.090.00%460.00001.00E+000.000.00ANXIETY0.961.9848.0961.9848.090.00%460.00001.00E+000.000.00CON0.2561.9848.0974.4849.74-20.16%49-1.68449.54E-020.140.09 Continuedonnextpage 225

PAGE 240

AppendixCContinuedTable38ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean CON0.561.9848.0974.3049.61-19.88%49-1.67269.77E-020.140.09CON0.7561.9848.0973.0149.02-17.79%49-1.53621.28E-010.140.09CON161.9848.0971.1448.31-14.78%50-1.32501.88E-010.140.09CON1.2561.9848.0969.8047.67-12.62%50-1.14522.55E-010.140.09CON1.561.9848.0968.8747.30-11.12%50-1.00683.17E-010.140.09CON1.7561.9848.0967.9746.89-9.67%50-0.86603.89E-010.140.09CON261.9848.0967.4946.80-8.90%50-0.79284.30E-010.140.09CON2.2561.9848.0967.1846.57-8.40%50-0.75024.55E-010.140.09CON2.561.9848.0966.7246.29-7.64%50-0.68914.92E-010.140.09CON2.7561.9848.0966.2345.90-6.86%50-0.61025.43E-010.150.09CON361.9848.0965.9445.99-6.39%50-0.56575.73E-010.150.09CON3.2561.9848.0965.5045.72-5.68%50-0.49446.22E-010.150.09CON3.561.9848.0965.0745.32-4.99%50-0.43426.65E-010.150.09CON3.7561.9848.0964.8345.42-4.60%50-0.40626.85E-010.150.09CON461.9848.0964.6645.50-4.32%50-0.37977.05E-010.150.09CON4.2561.9848.0964.0845.45-3.39%50-0.30487.61E-010.150.09CON4.561.9848.0963.4245.24-2.32%50-0.21028.34E-010.140.09CON4.7561.9848.0963.3745.28-2.25%50-0.20538.38E-010.140.08CON561.9848.0962.8745.09-1.44%50-0.13398.94E-010.140.08GAMBINO0.5171.07122.60106.1783.3237.94%412.41761.79E-020.610.24GAMBINO1171.07122.60106.1783.3237.94%412.41761.79E-020.610.24GAMBINO1.5171.07122.60169.87121.090.70%401.03973.02E-010.080.10GAMBINO2171.07122.60169.87121.090.70%401.03973.02E-010.080.10GAMBINO2.5171.07122.60171.07122.600.00%400.00001.00E+000.000.00 Continuedonnextpage 226

PAGE 241

AppendixCContinuedTable38ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean GAMBINO3171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO3.5171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO4171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO4.5171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO5171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO5.5171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO6171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO6.5171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO7171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO7.5171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO8171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO8.5171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO9171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO9.5171.07122.60171.07122.600.00%400.00001.00E+000.000.00GAMBINO10171.07122.60171.07122.600.00%400.00001.00E+000.000.00INCONSISTENCY0.0561.9848.0994.5086.30-52.47%46-1.26542.09E-010.030.03INCONSISTENCY0.0761.9848.0990.4382.76-45.90%46-1.22042.25E-010.030.03INCONSISTENCY0.0961.9848.0985.3876.84-37.75%47-1.23162.21E-010.030.03INCONSISTENCY0.1161.9848.0976.8864.65-24.05%46-1.26592.09E-010.030.03INCONSISTENCY0.1361.9848.0973.8359.92-19.12%46-1.33721.85E-010.030.03INCONSISTENCY0.1561.9848.0972.3858.20-16.78%46-1.34241.83E-010.020.03INCONSISTENCY0.1761.9848.0971.1856.93-14.85%46-1.32961.87E-010.020.03INCONSISTENCY0.1961.9848.0970.0055.37-12.94%46-1.37601.72E-010.020.03INCONSISTENCY0.2161.9848.0968.4553.67-10.44%46-1.39781.66E-010.020.03 Continuedonnextpage 227

PAGE 242

AppendixCContinuedTable38ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean INCONSISTENCY0.2361.9848.0966.8451.91-7.83%46-1.42961.56E-010.020.03INCONSISTENCY0.2561.9848.0962.7348.70-1.21%46-1.21602.27E-010.010.03INCONSISTENCY0.2761.9848.0962.6748.65-1.12%46-1.19012.37E-010.010.03INCONSISTENCY0.2961.9848.0962.6648.63-1.09%46-1.18822.38E-010.010.03INCONSISTENCY0.3161.9848.0962.6348.62-1.04%46-1.15472.51E-010.010.03INCONSISTENCY0.3361.9848.0962.5548.61-0.92%46-1.07842.84E-010.010.02INCONSISTENCY0.3561.9848.0962.5248.52-0.87%46-1.06112.91E-010.010.02INCONSISTENCY0.3761.9848.0962.4948.49-0.82%46-1.07752.84E-010.010.02INCONSISTENCY0.3961.9848.0962.4848.48-0.81%46-1.13722.58E-010.010.02INCONSISTENCY0.4161.9848.0962.4748.47-0.79%46-1.14232.56E-010.010.02INCONSISTENCY0.4361.9848.0962.4448.45-0.74%46-1.15022.53E-010.010.02INCONSISTENCY0.4561.9848.0962.3248.38-0.55%46-1.00883.16E-010.010.02LIU'SCONFLICT0.10.1171.07122.6075.1449.9256.07%472.15263.40E-020.510.21LIU'SCONFLICT0.20.1171.07122.6075.1649.9456.06%472.15313.39E-020.510.21LIU'SCONFLICT0.30.1171.07122.6075.1349.9056.08%472.15273.40E-020.510.21LIU'SCONFLICT0.40.1171.07122.6075.2149.9856.03%472.15433.38E-020.510.21LIU'SCONFLICT0.50.1171.07122.6078.5751.9354.07%462.13763.53E-020.510.21LIU'SCONFLICT0.60.1171.07122.6084.0655.0550.86%462.15873.35E-020.500.20LIU'SCONFLICT0.70.1171.07122.6089.8357.7947.49%462.13763.53E-020.500.20LIU'SCONFLICT0.80.1171.07122.6097.3663.1743.09%452.11773.70E-020.490.20LIU'SCONFLICT0.90.1171.07122.60107.8571.4736.96%442.10023.86E-020.480.20LIU'SCONFLICT0.10.2171.07122.6078.3750.1954.19%482.11953.67E-020.350.15LIU'SCONFLICT0.20.2171.07122.6078.3850.2054.18%482.12013.66E-020.350.15LIU'SCONFLICT0.30.2171.07122.6078.3750.1854.19%482.11983.67E-020.350.15 Continuedonnextpage 228

PAGE 243

AppendixCContinuedTable38ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean LIU'SCONFLICT0.40.2171.07122.6078.5150.2854.11%482.12063.66E-020.350.15LIU'SCONFLICT0.50.2171.07122.6081.5051.8852.36%472.10263.82E-020.350.15LIU'SCONFLICT0.60.2171.07122.6086.4554.5249.47%472.11833.68E-020.350.15LIU'SCONFLICT0.70.2171.07122.6091.6457.3946.43%472.11053.75E-020.340.15LIU'SCONFLICT0.80.2171.07122.6098.4562.1142.45%462.07964.04E-020.340.15LIU'SCONFLICT0.90.2171.07122.60107.6269.0237.09%452.04224.41E-020.340.15LIU'SCONFLICT0.10.3171.07122.60112.9468.6433.98%481.89926.06E-020.260.14LIU'SCONFLICT0.20.3171.07122.60113.0068.7433.95%481.90106.04E-020.260.14LIU'SCONFLICT0.30.3171.07122.60113.2168.9833.82%481.90475.99E-020.260.14LIU'SCONFLICT0.40.3171.07122.60113.6569.4833.57%481.91035.91E-020.260.14LIU'SCONFLICT0.50.3171.07122.60115.7671.6732.33%471.89896.07E-020.260.14LIU'SCONFLICT0.60.3171.07122.60119.1774.7730.34%461.88676.24E-020.260.14LIU'SCONFLICT0.70.3171.07122.60122.5877.8828.35%451.85926.63E-020.260.14LIU'SCONFLICT0.80.3171.07122.60126.4981.2626.06%441.83906.94E-020.260.14LIU'SCONFLICT0.90.3171.07122.60131.5685.8523.10%431.82617.14E-020.260.14LIU'SCONFLICT0.10.4171.07122.60113.4768.9633.67%481.89336.14E-020.260.14LIU'SCONFLICT0.20.4171.07122.60113.5469.0833.63%481.89586.11E-020.260.14LIU'SCONFLICT0.30.4171.07122.60113.9369.5633.40%481.90236.02E-020.260.14LIU'SCONFLICT0.40.4171.07122.60114.4870.0533.08%481.90675.96E-020.260.14LIU'SCONFLICT0.50.4171.07122.60116.6472.2831.82%471.89406.14E-020.260.14LIU'SCONFLICT0.60.4171.07122.60120.0675.5629.82%461.88776.23E-020.260.14LIU'SCONFLICT0.70.4171.07122.60123.3578.6327.90%451.86086.61E-020.260.14LIU'SCONFLICT0.80.4171.07122.60127.1381.8225.69%441.83596.98E-020.260.14LIU'SCONFLICT0.90.4171.07122.60132.1386.4422.76%431.82877.10E-020.260.14 Continuedonnextpage 229

PAGE 244

AppendixCContinuedTable38ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean LIU'SCONFLICT0.10.5171.07122.60114.5769.7333.03%481.88806.21E-020.260.14LIU'SCONFLICT0.20.5171.07122.60114.5769.7333.03%481.88806.21E-020.260.14LIU'SCONFLICT0.30.5171.07122.60114.8770.0732.85%481.89106.17E-020.260.14LIU'SCONFLICT0.40.5171.07122.60115.6170.8232.42%471.85916.62E-020.260.14LIU'SCONFLICT0.50.5171.07122.60117.8873.3031.10%461.84986.76E-020.260.14LIU'SCONFLICT0.60.5171.07122.60120.9676.2929.29%461.88116.32E-020.260.14LIU'SCONFLICT0.70.5171.07122.60124.1579.3027.43%451.85296.72E-020.260.14LIU'SCONFLICT0.80.5171.07122.60127.9182.6525.23%441.83307.03E-020.260.14LIU'SCONFLICT0.90.5171.07122.60132.6386.7022.47%431.80837.41E-020.260.14LIU'SCONFLICT0.10.6171.07122.60116.1470.9132.11%481.87746.36E-020.260.14LIU'SCONFLICT0.20.6171.07122.60116.1470.9132.11%481.87746.36E-020.260.14LIU'SCONFLICT0.30.6171.07122.60116.1470.9132.11%481.87746.36E-020.260.14LIU'SCONFLICT0.40.6171.07122.60116.6371.5331.82%481.88586.24E-020.260.14LIU'SCONFLICT0.50.6171.07122.60119.0774.1030.40%461.83606.97E-020.260.14LIU'SCONFLICT0.60.6171.07122.60122.1877.2428.58%461.87056.47E-020.260.14LIU'SCONFLICT0.70.6171.07122.60125.2980.1126.76%451.83876.93E-020.260.14LIU'SCONFLICT0.80.6171.07122.60128.8583.0524.68%441.80557.45E-020.260.14LIU'SCONFLICT0.90.6171.07122.60133.4287.0622.01%431.78627.77E-020.260.14LIU'SCONFLICT0.10.7171.07122.60118.7673.5330.58%471.85096.74E-020.260.14LIU'SCONFLICT0.20.7171.07122.60118.7673.5330.58%471.85096.74E-020.260.14LIU'SCONFLICT0.30.7171.07122.60118.7673.5330.58%471.85096.74E-020.260.14LIU'SCONFLICT0.40.7171.07122.60118.7673.5330.58%471.85096.74E-020.260.14LIU'SCONFLICT0.50.7171.07122.60120.3975.1629.62%461.83177.03E-020.260.14LIU'SCONFLICT0.60.7171.07122.60123.6078.2727.75%461.86416.56E-020.260.14 Continuedonnextpage 230

PAGE 245

AppendixCContinuedTable38ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean LIU'SCONFLICT0.70.7171.07122.60126.7781.3825.90%451.83626.97E-020.260.14LIU'SCONFLICT0.80.7171.07122.60130.1184.2923.94%441.80607.44E-020.250.14LIU'SCONFLICT0.90.7171.07122.60134.2787.8421.51%431.78597.77E-020.250.14LIU'SCONFLICT0.10.8171.07122.60134.4887.4321.39%431.72098.90E-020.210.11LIU'SCONFLICT0.20.8171.07122.60134.4887.4321.39%431.72098.90E-020.210.11LIU'SCONFLICT0.30.8171.07122.60134.4887.4321.39%431.72098.90E-020.210.11LIU'SCONFLICT0.40.8171.07122.60134.4887.4321.39%431.72098.90E-020.210.11LIU'SCONFLICT0.50.8171.07122.60134.4887.4321.39%431.72098.90E-020.210.11LIU'SCONFLICT0.60.8171.07122.60134.4887.4321.39%431.72098.90E-020.210.11LIU'SCONFLICT0.70.8171.07122.60137.4190.3819.68%431.73858.58E-020.210.11LIU'SCONFLICT0.80.8171.07122.60139.9992.6118.17%421.69049.47E-020.210.11LIU'SCONFLICT0.90.8171.07122.60142.8495.4316.50%421.71778.96E-020.210.11LIU'SCONFLICT0.10.9171.07122.60153.04103.3410.54%411.48371.42E-010.150.08LIU'SCONFLICT0.20.9171.07122.60153.04103.3410.54%411.48371.42E-010.150.08LIU'SCONFLICT0.30.9171.07122.60153.04103.3410.54%411.48371.42E-010.150.08LIU'SCONFLICT0.40.9171.07122.60153.04103.3410.54%411.48371.42E-010.150.08LIU'SCONFLICT0.50.9171.07122.60153.04103.3410.54%411.48371.42E-010.150.08LIU'SCONFLICT0.60.9171.07122.60153.04103.3410.54%411.48371.42E-010.150.08LIU'SCONFLICT0.70.9171.07122.60153.04103.3410.54%411.48371.42E-010.150.08LIU'SCONFLICT0.80.9171.07122.60153.04103.3410.54%411.48371.42E-010.150.08LIU'SCONFLICT0.90.9171.07122.60154.61104.769.62%411.46661.46E-010.150.08TBMCONFLICT0.1171.07122.6074.4649.7256.48%472.16893.27E-020.610.25TBMCONFLICT0.14171.07122.6074.4649.7256.48%472.16893.27E-020.610.25TBMCONFLICT0.18171.07122.6074.4749.7356.47%472.16943.26E-020.610.25 Continuedonnextpage 231

PAGE 246

AppendixCContinuedTable38ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean TBMCONFLICT0.22171.07122.6074.4849.7456.46%472.16953.26E-020.610.25TBMCONFLICT0.26171.07122.6074.4649.7156.48%472.16943.26E-020.610.25TBMCONFLICT0.3171.07122.6074.4349.7056.49%472.16913.27E-020.610.25TBMCONFLICT0.34171.07122.6074.4449.7256.49%472.16953.26E-020.610.25TBMCONFLICT0.38171.07122.6074.4249.7356.50%472.17013.26E-020.610.25TBMCONFLICT0.42171.07122.6074.8250.0956.27%462.12803.61E-020.610.25TBMCONFLICT0.46171.07122.6075.9750.9355.59%462.14353.48E-020.610.25TBMCONFLICT0.5171.07122.6078.1451.9154.32%462.15413.39E-020.610.25TBMCONFLICT0.54171.07122.6080.9353.7352.69%462.17353.24E-020.610.25TBMCONFLICT0.58171.07122.6082.7755.1351.62%462.18943.12E-020.600.24TBMCONFLICT0.62171.07122.6085.1956.2550.20%462.19203.10E-020.600.24TBMCONFLICT0.66171.07122.6087.8257.6248.67%462.18403.16E-020.600.24TBMCONFLICT0.7171.07122.6090.1258.8147.32%462.17533.22E-020.590.24TBMCONFLICT0.74171.07122.6092.6460.3945.85%462.17983.19E-020.590.23TBMCONFLICT0.78171.07122.6096.0162.5843.88%452.13593.55E-020.580.23TBMCONFLICT0.82171.07122.6099.3265.7341.94%452.16193.33E-020.580.23TBMCONFLICT0.86171.07122.60103.3968.8139.56%442.11733.71E-020.580.23TBMCONFLICT0.9171.07122.60108.6772.7336.48%442.13753.54E-020.570.23 232

PAGE 247

AppendixCContinuedTable39:Rawresultsforisolationforlaserreadingsonly.n=176BaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean ANXIETY0.187.0355.07152.4887.94-75.21%13-1.47571.53E-010.050.04ANXIETY0.1487.0355.07120.9769.85-39.00%11-1.28662.13E-010.040.03ANXIETY0.1887.0355.07114.6866.28-31.78%11-1.30042.08E-010.040.03ANXIETY0.2287.0355.07110.5564.27-27.03%10-1.16432.59E-010.040.03ANXIETY0.2687.0355.07107.0662.31-23.01%10-1.19662.47E-010.040.03ANXIETY0.387.0355.07103.4060.54-18.82%10-1.19732.47E-010.040.03ANXIETY0.3487.0355.07100.5258.95-15.50%10-1.25622.25E-010.040.03ANXIETY0.3887.0355.0797.6357.89-12.18%9-1.08812.93E-010.040.03ANXIETY0.4287.0355.0795.1257.15-9.30%9-1.09602.89E-010.040.03ANXIETY0.4687.0355.0793.3056.80-7.20%9-1.03533.16E-010.040.03ANXIETY0.587.0355.0790.9156.27-4.46%9-0.99123.36E-010.040.03ANXIETY0.5487.0355.0789.2455.87-2.54%9-0.79924.36E-010.020.02ANXIETY0.5887.0355.0787.9655.60-1.07%9-0.66185.18E-010.020.02ANXIETY0.6287.0355.0787.4155.43-0.44%9-0.43386.70E-010.000.01ANXIETY0.6687.0355.0787.3155.35-0.32%9-0.34447.35E-010.000.01ANXIETY0.787.0355.0787.2455.31-0.24%9-0.32207.52E-010.000.01ANXIETY0.7487.0355.0787.1455.16-0.13%9-0.32807.47E-010.000.00ANXIETY0.7887.0355.0787.0855.09-0.06%8-0.17738.62E-010.000.00ANXIETY0.8287.0355.0787.0655.08-0.03%8-0.19128.51E-010.000.00ANXIETY0.8687.0355.0787.0355.070.00%80.00001.00E+000.000.00ANXIETY0.987.0355.0787.0355.070.00%80.00001.00E+000.000.00CON0.2587.0355.0799.6451.85-14.49%19-0.76624.49E-010.150.05 Continuedonnextpage 233

PAGE 248

AppendixCContinuedTable39ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean CON0.587.0355.0799.1451.70-13.92%19-0.74354.62E-010.150.05CON0.7587.0355.0796.0851.08-10.41%19-0.57665.68E-010.150.05CON187.0355.0792.9750.28-6.83%19-0.38217.05E-010.150.05CON1.2587.0355.0790.3049.89-3.75%19-0.21408.32E-010.150.05CON1.587.0355.0788.3049.52-1.46%19-0.08389.34E-010.150.05CON1.7587.0355.0786.5048.960.60%200.03649.71E-010.150.05CON287.0355.0785.2649.182.03%190.11789.07E-010.150.05CON2.2587.0355.0784.6849.082.70%200.16508.70E-010.150.05CON2.587.0355.0783.8648.613.64%200.22728.22E-010.150.05CON2.7587.0355.0782.8848.154.77%200.29177.72E-010.150.05CON387.0355.0782.2348.175.52%200.34127.35E-010.150.06CON3.2587.0355.0781.1548.116.76%220.45296.53E-010.150.06CON3.587.0355.0780.3147.567.71%210.49436.24E-010.150.06CON3.7587.0355.0780.2547.597.79%200.48096.33E-010.150.06CON487.0355.0780.0147.608.06%210.52476.03E-010.150.06CON4.2587.0355.0779.7847.658.33%210.54865.86E-010.150.06CON4.587.0355.0779.0147.669.22%200.57165.71E-010.150.06CON4.7587.0355.0778.9147.959.33%210.61695.41E-010.150.06CON587.0355.0778.6248.059.66%210.64515.23E-010.150.06GAMBINO0.5297.92116.68190.5981.1836.03%112.17354.19E-020.680.12GAMBINO1297.92116.68190.5981.1836.03%112.17354.19E-020.680.12GAMBINO1.5297.92116.68294.65115.351.10%131.39061.77E-010.220.04GAMBINO2297.92116.68294.65115.351.10%131.39061.77E-010.220.04GAMBINO2.5297.92116.68297.92116.680.00%130.00001.00E+000.000.00 Continuedonnextpage 234

PAGE 249

AppendixCContinuedTable39ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean GAMBINO3297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO3.5297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO4297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO4.5297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO5297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO5.5297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO6297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO6.5297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO7297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO7.5297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO8297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO8.5297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO9297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO9.5297.92116.68297.92116.680.00%130.00001.00E+000.000.00GAMBINO10297.92116.68297.92116.680.00%130.00001.00E+000.000.00INCONSISTENCY0.0587.0355.07179.1991.43-105.90%14-1.90806.75E-020.050.04INCONSISTENCY0.0787.0355.07167.6291.76-92.61%14-1.70889.94E-020.050.04INCONSISTENCY0.0987.0355.07152.6188.07-75.35%13-1.47451.53E-010.050.04INCONSISTENCY0.1187.0355.07127.1074.67-46.05%12-1.34291.93E-010.050.04INCONSISTENCY0.1387.0355.07118.1768.16-35.78%11-1.30892.05E-010.040.03INCONSISTENCY0.1587.0355.07114.0766.32-31.08%10-1.15642.63E-010.040.03INCONSISTENCY0.1787.0355.07110.9964.86-27.54%10-1.13792.70E-010.040.03INCONSISTENCY0.1987.0355.07107.6862.74-23.73%10-1.20072.45E-010.040.03INCONSISTENCY0.2187.0355.07103.5060.69-18.92%10-1.20602.43E-010.040.03 Continuedonnextpage 235

PAGE 250

AppendixCContinuedTable39ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean INCONSISTENCY0.2387.0355.0799.1758.54-13.95%10-1.22072.38E-010.040.03INCONSISTENCY0.2587.0355.0787.5955.83-0.65%8-0.28057.83E-010.010.01INCONSISTENCY0.2787.0355.0787.5455.77-0.58%8-0.25888.00E-010.010.01INCONSISTENCY0.2987.0355.0787.5255.75-0.57%9-0.29607.71E-010.010.01INCONSISTENCY0.3187.0355.0787.5055.73-0.55%8-0.24838.08E-010.010.01INCONSISTENCY0.3387.0355.0787.5155.74-0.56%8-0.25588.02E-010.010.01INCONSISTENCY0.3587.0355.0787.4055.55-0.42%8-0.24528.10E-010.010.01INCONSISTENCY0.3787.0355.0787.4355.52-0.47%8-0.28507.80E-010.010.01INCONSISTENCY0.3987.0355.0787.4555.50-0.49%8-0.32767.48E-010.010.01INCONSISTENCY0.4187.0355.0787.4355.47-0.46%8-0.33587.42E-010.000.01INCONSISTENCY0.4387.0355.0787.4155.46-0.44%9-0.37857.10E-010.000.01INCONSISTENCY0.4587.0355.0787.3255.41-0.33%9-0.34587.34E-010.000.01LIU'SCONFLICT0.10.1297.92116.6899.6551.9566.55%203.67727.26E-040.610.13LIU'SCONFLICT0.20.1297.92116.6899.7151.9966.53%203.67927.22E-040.610.13LIU'SCONFLICT0.30.1297.92116.6899.6151.9066.57%203.67867.23E-040.610.13LIU'SCONFLICT0.40.1297.92116.6899.8652.0366.48%203.68247.15E-040.610.13LIU'SCONFLICT0.50.1297.92116.68109.4252.9963.27%193.45091.44E-030.620.13LIU'SCONFLICT0.60.1297.92116.68121.8655.4959.10%183.27582.43E-030.610.13LIU'SCONFLICT0.70.1297.92116.68132.8057.8755.42%183.23512.71E-030.610.13LIU'SCONFLICT0.80.1297.92116.68150.4561.8949.50%152.63371.36E-020.610.13LIU'SCONFLICT0.90.1297.92116.68173.2869.3941.84%132.25733.34E-020.600.13LIU'SCONFLICT0.10.2297.92116.68101.1252.7766.06%203.65567.73E-040.480.11LIU'SCONFLICT0.20.2297.92116.68101.1652.8066.04%203.65807.68E-040.480.11LIU'SCONFLICT0.30.2297.92116.68101.1252.7566.06%203.65667.71E-040.480.11 Continuedonnextpage 236

PAGE 251

AppendixCContinuedTable39ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean LIU'SCONFLICT0.40.2297.92116.68101.5352.8765.92%203.65127.83E-040.480.11LIU'SCONFLICT0.50.2297.92116.68110.1253.6263.04%193.43261.52E-030.480.11LIU'SCONFLICT0.60.2297.92116.68121.8155.1959.11%193.40441.64E-030.480.11LIU'SCONFLICT0.70.2297.92116.68132.4957.8055.53%173.01185.04E-030.480.11LIU'SCONFLICT0.80.2297.92116.68148.2761.3450.23%152.60131.47E-020.480.11LIU'SCONFLICT0.90.2297.92116.68168.7266.6343.37%142.34242.71E-020.480.11LIU'SCONFLICT0.10.3297.92116.68164.2472.5044.87%162.88057.26E-030.400.10LIU'SCONFLICT0.20.3297.92116.68164.4272.6544.81%162.88677.15E-030.400.10LIU'SCONFLICT0.30.3297.92116.68165.0772.8844.59%162.89057.09E-030.400.10LIU'SCONFLICT0.40.3297.92116.68166.3473.3844.17%162.89177.06E-030.400.10LIU'SCONFLICT0.50.3297.92116.68172.5675.1442.08%152.71211.13E-020.400.10LIU'SCONFLICT0.60.3297.92116.68181.7077.1639.01%152.70031.16E-020.400.10LIU'SCONFLICT0.70.3297.92116.68190.1379.7536.18%142.49601.92E-020.400.10LIU'SCONFLICT0.80.3297.92116.68200.2881.2132.77%142.43012.23E-020.400.10LIU'SCONFLICT0.90.3297.92116.68213.4383.2728.36%132.14784.20E-020.400.10LIU'SCONFLICT0.10.4297.92116.68165.3772.5244.49%162.85377.76E-030.400.10LIU'SCONFLICT0.20.4297.92116.68165.6072.7244.41%162.86217.60E-030.400.10LIU'SCONFLICT0.30.4297.92116.68166.7773.2344.02%162.86857.48E-030.400.10LIU'SCONFLICT0.40.4297.92116.68168.3873.4443.48%162.85197.79E-030.400.10LIU'SCONFLICT0.50.4297.92116.68174.7475.1541.35%152.66131.27E-020.400.10LIU'SCONFLICT0.60.4297.92116.68183.9077.6238.27%152.67861.22E-020.400.10LIU'SCONFLICT0.70.4297.92116.68191.9480.4335.57%142.49281.94E-020.400.10LIU'SCONFLICT0.80.4297.92116.68201.7181.6332.29%132.23263.52E-020.400.10LIU'SCONFLICT0.90.4297.92116.68214.6383.9027.96%132.16134.09E-020.400.10 Continuedonnextpage 237

PAGE 252

AppendixCContinuedTable39ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean LIU'SCONFLICT0.10.5297.92116.68167.4773.0543.79%162.87127.43E-030.400.10LIU'SCONFLICT0.20.5297.92116.68167.4773.0543.79%162.87127.43E-030.400.10LIU'SCONFLICT0.30.5297.92116.68168.3873.3343.48%162.86357.57E-030.400.10LIU'SCONFLICT0.40.5297.92116.68170.5673.8942.75%162.85357.76E-030.400.10LIU'SCONFLICT0.50.5297.92116.68177.1976.0340.52%152.68041.22E-020.400.10LIU'SCONFLICT0.60.5297.92116.68185.5978.2137.70%162.86487.55E-030.400.10LIU'SCONFLICT0.70.5297.92116.68193.3181.0935.11%142.50011.91E-020.400.10LIU'SCONFLICT0.80.5297.92116.68202.9882.8031.87%142.45122.13E-020.400.10LIU'SCONFLICT0.90.5297.92116.68215.2183.9627.76%132.13024.36E-020.400.10LIU'SCONFLICT0.10.6297.92116.68170.7473.7142.69%162.84447.94E-030.400.10LIU'SCONFLICT0.20.6297.92116.68170.7473.7142.69%162.84447.94E-030.400.10LIU'SCONFLICT0.30.6297.92116.68170.7473.7142.69%162.84447.94E-030.400.10LIU'SCONFLICT0.40.6297.92116.68172.1974.4242.20%162.85847.67E-030.400.10LIU'SCONFLICT0.50.6297.92116.68179.3076.4739.82%152.66971.25E-020.400.10LIU'SCONFLICT0.60.6297.92116.68187.7179.1436.99%152.69361.18E-020.400.10LIU'SCONFLICT0.70.6297.92116.68195.0981.8734.51%142.51501.84E-020.400.10LIU'SCONFLICT0.80.6297.92116.68204.2582.7931.44%142.41102.33E-020.400.10LIU'SCONFLICT0.90.6297.92116.68215.9684.3927.51%132.14454.23E-020.400.10LIU'SCONFLICT0.10.7297.92116.68177.2076.3140.52%152.69041.19E-020.400.10LIU'SCONFLICT0.20.7297.92116.68177.2076.3140.52%152.69041.19E-020.400.10LIU'SCONFLICT0.30.7297.92116.68177.2076.3140.52%152.69041.19E-020.400.10LIU'SCONFLICT0.40.7297.92116.68177.2076.3140.52%152.69041.19E-020.400.10LIU'SCONFLICT0.50.7297.92116.68181.9777.3838.92%142.50111.90E-020.400.10LIU'SCONFLICT0.60.7297.92116.68190.7279.6935.98%142.49541.93E-020.400.10 Continuedonnextpage 238

PAGE 253

AppendixCContinuedTable39ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean LIU'SCONFLICT0.70.7297.92116.68198.3382.8833.43%142.51391.85E-020.400.10LIU'SCONFLICT0.80.7297.92116.68206.9284.1430.54%132.24673.41E-020.400.10LIU'SCONFLICT0.90.7297.92116.68217.4585.4327.01%132.18923.85E-020.400.10LIU'SCONFLICT0.10.8297.92116.68215.8385.5727.55%132.13364.33E-020.330.08LIU'SCONFLICT0.20.8297.92116.68215.8385.5727.55%132.13364.33E-020.330.08LIU'SCONFLICT0.30.8297.92116.68215.8385.5727.55%132.13364.33E-020.330.08LIU'SCONFLICT0.40.8297.92116.68215.8385.5727.55%132.13364.33E-020.330.08LIU'SCONFLICT0.50.8297.92116.68215.8385.5727.55%132.13364.33E-020.330.08LIU'SCONFLICT0.60.8297.92116.68215.8385.5727.55%132.13364.33E-020.330.08LIU'SCONFLICT0.70.8297.92116.68223.1588.0725.10%132.11154.53E-020.330.08LIU'SCONFLICT0.80.8297.92116.68229.2189.3823.06%132.04425.21E-020.330.08LIU'SCONFLICT0.90.8297.92116.68236.4091.2920.65%121.86477.56E-020.330.08LIU'SCONFLICT0.10.9297.92116.68255.8798.1314.11%121.69541.04E-010.230.05LIU'SCONFLICT0.20.9297.92116.68255.8798.1314.11%121.69541.04E-010.230.05LIU'SCONFLICT0.30.9297.92116.68255.8798.1314.11%121.69541.04E-010.230.05LIU'SCONFLICT0.40.9297.92116.68255.8798.1314.11%121.69541.04E-010.230.05LIU'SCONFLICT0.50.9297.92116.68255.8798.1314.11%121.69541.04E-010.230.05LIU'SCONFLICT0.60.9297.92116.68255.8798.1314.11%121.69541.04E-010.230.05LIU'SCONFLICT0.70.9297.92116.68255.8798.1314.11%121.69541.04E-010.230.05LIU'SCONFLICT0.80.9297.92116.68255.8798.1314.11%121.69541.04E-010.230.05LIU'SCONFLICT0.90.9297.92116.68259.9198.6012.76%121.58711.27E-010.230.05TBMCONFLICT0.1297.92116.6899.5751.8266.58%203.67517.31E-040.690.12TBMCONFLICT0.14297.92116.6899.5751.8266.58%203.67517.31E-040.690.12TBMCONFLICT0.18297.92116.6899.6251.8466.56%203.67707.27E-040.690.12 Continuedonnextpage 239

PAGE 254

AppendixCContinuedTable39ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean TBMCONFLICT0.22297.92116.6899.6451.8566.55%203.67737.26E-040.690.12TBMCONFLICT0.26297.92116.6899.5851.8166.57%203.67827.24E-040.690.12TBMCONFLICT0.3297.92116.6899.5351.7666.59%203.67647.28E-040.690.12TBMCONFLICT0.34297.92116.6899.5451.8366.59%203.67897.23E-040.690.12TBMCONFLICT0.38297.92116.6899.4851.8866.61%203.68437.12E-040.690.12TBMCONFLICT0.42297.92116.68100.6652.3066.21%203.68227.16E-040.690.12TBMCONFLICT0.46297.92116.68104.1652.9265.04%193.49171.29E-030.690.12TBMCONFLICT0.5297.92116.68110.0852.7463.05%193.42991.53E-030.690.12TBMCONFLICT0.54297.92116.68116.7654.5360.81%183.26372.51E-030.690.12TBMCONFLICT0.58297.92116.68121.0256.0059.38%183.30312.26E-030.690.12TBMCONFLICT0.62297.92116.68125.9256.7057.73%183.31882.16E-030.690.12TBMCONFLICT0.66297.92116.68131.0957.8956.00%183.28762.35E-030.690.12TBMCONFLICT0.7297.92116.68135.2959.3054.59%173.09594.06E-030.690.12TBMCONFLICT0.74297.92116.68140.6660.6852.78%173.10024.02E-030.690.12TBMCONFLICT0.78297.92116.68148.8361.1950.04%152.63841.34E-020.680.12TBMCONFLICT0.82297.92116.68156.4464.9247.49%142.49371.93E-020.680.12TBMCONFLICT0.86297.92116.68165.1767.5944.56%142.45932.09E-020.680.12TBMCONFLICT0.9297.92116.68176.6670.4740.70%132.28053.18E-020.680.12 240

PAGE 255

AppendixCContinuedTable40:Rawresultsforisolationforrangecamerareadingsonly.n=352BaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean ANXIETY0.149.4638.5851.8040.05-4.74%35-1.72268.95E-020.020.02ANXIETY0.1449.4638.5851.7040.00-4.54%35-1.67229.91E-020.010.02ANXIETY0.1849.4638.5851.4539.79-4.03%35-1.63881.06E-010.010.02ANXIETY0.2249.4638.5851.1739.69-3.47%35-1.51961.33E-010.010.02ANXIETY0.2649.4638.5850.9239.55-2.96%35-1.44631.53E-010.010.02ANXIETY0.349.4638.5850.7939.44-2.69%35-1.45631.50E-010.010.02ANXIETY0.3449.4638.5850.5039.28-2.12%35-1.34751.82E-010.010.03ANXIETY0.3849.4638.5850.2439.13-1.58%35-1.20142.34E-010.010.03ANXIETY0.4249.4638.5850.1539.11-1.40%35-1.17262.45E-010.010.03ANXIETY0.4649.4638.5850.0739.07-1.24%35-1.15272.53E-010.010.03ANXIETY0.549.4638.5849.8638.95-0.81%35-0.94533.48E-010.010.02ANXIETY0.5449.4638.5849.7738.84-0.63%35-0.91273.65E-010.010.02ANXIETY0.5849.4638.5849.6938.80-0.48%35-0.85513.96E-010.000.01ANXIETY0.6249.4638.5849.6138.72-0.32%35-0.90373.69E-010.000.01ANXIETY0.6649.4638.5849.5838.69-0.26%35-0.84284.02E-010.000.01ANXIETY0.749.4638.5849.5438.64-0.17%35-0.69214.91E-010.000.01ANXIETY0.7449.4638.5849.4638.59-0.01%35-0.21848.28E-010.000.00ANXIETY0.7849.4638.5849.4638.580.00%350.00001.00E+000.000.00ANXIETY0.8249.4638.5849.4638.580.00%350.00001.00E+000.000.00ANXIETY0.8649.4638.5849.4638.580.00%350.00001.00E+000.000.00ANXIETY0.949.4638.5849.4638.580.00%350.00001.00E+000.000.00CON0.2549.4638.5861.9043.57-25.16%37-2.36502.07E-020.140.10 Continuedonnextpage 241

PAGE 256

AppendixCContinuedTable40ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean CON0.549.4638.5861.8843.57-25.13%37-2.36232.09E-020.140.10CON0.7549.4638.5861.4743.66-24.28%37-2.31862.33E-020.140.11CON149.4638.5860.2243.42-21.77%37-2.21133.02E-020.140.10CON1.2549.4638.5859.5543.07-20.41%37-2.16383.38E-020.140.10CON1.549.4638.5859.1643.04-19.61%37-2.12713.68E-020.140.10CON1.7549.4638.5858.7142.99-18.71%37-2.06694.23E-020.140.10CON249.4638.5858.6142.96-18.51%37-2.02484.66E-020.140.10CON2.2549.4638.5858.4442.72-18.16%37-2.04504.45E-020.140.10CON2.549.4638.5858.1542.64-17.57%37-2.00074.92E-020.140.10CON2.7549.4638.5857.9142.42-17.09%37-2.01004.82E-020.140.10CON349.4638.5857.8042.64-16.87%36-1.88476.36E-020.140.10CON3.2549.4638.5857.6842.43-16.62%36-1.92645.81E-020.140.10CON3.549.4638.5857.4542.21-16.16%36-1.94445.59E-020.140.10CON3.7549.4638.5857.1242.30-15.49%36-1.89326.25E-020.140.10CON449.4638.5856.9842.44-15.21%36-1.79817.65E-020.140.10CON4.2549.4638.5856.2342.25-13.70%36-1.71279.12E-020.140.10CON4.549.4638.5855.6341.92-12.48%36-1.70349.29E-020.140.10CON4.7549.4638.5855.6141.84-12.44%36-1.72198.95E-020.140.09CON549.4638.5855.0041.41-11.20%36-1.63461.07E-010.140.09GAMBINO0.5107.6560.6563.9642.1540.58%503.26611.50E-030.570.28GAMBINO1107.6560.6563.9642.1540.58%503.26611.50E-030.570.28GAMBINO1.5107.6560.65107.4860.580.15%591.05852.92E-010.010.02GAMBINO2107.6560.65107.4860.580.15%591.05852.92E-010.010.02GAMBINO2.5107.6560.65107.6560.650.00%590.00001.00E+000.000.00 Continuedonnextpage 242

PAGE 257

AppendixCContinuedTable40ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean GAMBINO3107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO3.5107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO4107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO4.5107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO5107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO5.5107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO6107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO6.5107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO7107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO7.5107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO8107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO8.5107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO9107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO9.5107.6560.65107.6560.650.00%590.00001.00E+000.000.00GAMBINO10107.6560.65107.6560.650.00%590.00001.00E+000.000.00INCONSISTENCY0.0549.4638.5852.1540.24-5.45%35-1.67989.76E-020.020.02INCONSISTENCY0.0749.4638.5851.8440.03-4.81%35-1.72478.91E-020.020.02INCONSISTENCY0.0949.4638.5851.7639.96-4.66%35-1.77018.12E-020.020.02INCONSISTENCY0.1149.4638.5851.7739.99-4.69%35-1.74438.56E-020.020.02INCONSISTENCY0.1349.4638.5851.6639.94-4.45%35-1.70239.33E-020.020.02INCONSISTENCY0.1549.4638.5851.5339.82-4.20%35-1.67289.90E-020.020.02INCONSISTENCY0.1749.4638.5851.2839.71-3.68%35-1.52931.31E-010.010.02INCONSISTENCY0.1949.4638.5851.1639.66-3.44%35-1.50501.37E-010.010.03INCONSISTENCY0.2149.4638.5850.9339.56-2.98%35-1.44371.53E-010.010.03 Continuedonnextpage 243

PAGE 258

AppendixCContinuedTable40ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean INCONSISTENCY0.2349.4638.5850.6739.38-2.45%35-1.36391.77E-010.010.03INCONSISTENCY0.2549.4638.5850.3039.26-1.70%35-1.23782.20E-010.010.03INCONSISTENCY0.2749.4638.5850.2439.22-1.59%35-1.24052.19E-010.010.03INCONSISTENCY0.2949.4638.5850.2239.20-1.55%35-1.23292.22E-010.010.03INCONSISTENCY0.3149.4638.5850.1939.18-1.48%35-1.20972.31E-010.010.03INCONSISTENCY0.3349.4638.5850.0739.11-1.25%35-1.09022.79E-010.010.03INCONSISTENCY0.3549.4638.5850.0839.12-1.26%35-1.07682.85E-010.010.03INCONSISTENCY0.3749.4638.5850.0239.06-1.14%35-1.04053.02E-010.010.03INCONSISTENCY0.3949.4638.5850.0039.04-1.10%35-1.06422.91E-010.010.03INCONSISTENCY0.4149.4638.5850.0039.05-1.09%35-1.06872.89E-010.010.03INCONSISTENCY0.4349.4638.5849.9539.01-1.00%35-1.07442.86E-010.010.02INCONSISTENCY0.4549.4638.5849.8238.90-0.73%35-0.92743.57E-010.010.02LIU'SCONFLICT0.10.1107.6560.6562.8944.0941.58%493.33451.22E-030.460.22LIU'SCONFLICT0.20.1107.6560.6562.8944.0941.58%493.33451.22E-030.460.22LIU'SCONFLICT0.30.1107.6560.6562.8944.0941.58%493.33471.21E-030.460.22LIU'SCONFLICT0.40.1107.6560.6562.8944.0941.58%493.33501.21E-030.460.22LIU'SCONFLICT0.50.1107.6560.6563.1543.9741.34%493.31941.28E-030.460.22LIU'SCONFLICT0.60.1107.6560.6565.1744.0539.46%503.38771.02E-030.450.21LIU'SCONFLICT0.70.1107.6560.6568.3544.2336.50%513.42938.81E-040.440.21LIU'SCONFLICT0.80.1107.6560.6570.8244.2834.21%513.37301.06E-030.430.20LIU'SCONFLICT0.90.1107.6560.6575.1345.2730.21%523.39549.77E-040.420.20LIU'SCONFLICT0.10.2107.6560.6567.0044.7737.76%503.38521.02E-030.280.12LIU'SCONFLICT0.20.2107.6560.6567.0044.7737.76%503.38521.02E-030.280.12LIU'SCONFLICT0.30.2107.6560.6566.9944.7737.77%503.38541.02E-030.280.12 Continuedonnextpage 244

PAGE 259

AppendixCContinuedTable40ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean LIU'SCONFLICT0.40.2107.6560.6567.0044.7737.76%503.38591.02E-030.280.12LIU'SCONFLICT0.50.2107.6560.6567.1944.6337.58%503.37191.07E-030.280.12LIU'SCONFLICT0.60.2107.6560.6568.7744.7636.12%513.44288.42E-040.280.12LIU'SCONFLICT0.70.2107.6560.6571.2144.9833.85%513.41749.16E-040.280.12LIU'SCONFLICT0.80.2107.6560.6573.5445.2731.68%523.42798.78E-040.270.12LIU'SCONFLICT0.90.2107.6560.6577.0646.1628.41%533.44618.22E-040.270.12LIU'SCONFLICT0.10.3107.6560.6587.2949.7318.91%563.76402.70E-040.190.09LIU'SCONFLICT0.20.3107.6560.6587.2949.7318.91%563.76402.70E-040.190.09LIU'SCONFLICT0.30.3107.6560.6587.2849.7318.92%563.76422.70E-040.190.09LIU'SCONFLICT0.40.3107.6560.6587.3049.7518.90%563.76662.68E-040.190.09LIU'SCONFLICT0.50.3107.6560.6587.3649.7018.84%563.74912.85E-040.190.09LIU'SCONFLICT0.60.3107.6560.6587.9049.8418.35%563.74342.91E-040.190.09LIU'SCONFLICT0.70.3107.6560.6588.8050.0417.51%563.71163.25E-040.190.09LIU'SCONFLICT0.80.3107.6560.6589.5950.2916.78%563.65393.97E-040.190.09LIU'SCONFLICT0.90.3107.6560.6590.6250.6615.82%563.56425.41E-040.190.09LIU'SCONFLICT0.10.4107.6560.6587.5149.9018.70%563.77122.63E-040.190.09LIU'SCONFLICT0.20.4107.6560.6587.5149.9018.70%563.77122.63E-040.190.09LIU'SCONFLICT0.30.4107.6560.6587.5149.9118.71%563.77142.63E-040.190.09LIU'SCONFLICT0.40.4107.6560.6587.5349.9218.69%563.77382.61E-040.190.09LIU'SCONFLICT0.50.4107.6560.6587.5949.8918.63%563.75722.77E-040.190.09LIU'SCONFLICT0.60.4107.6560.6588.1350.0418.13%563.75102.83E-040.190.09LIU'SCONFLICT0.70.4107.6560.6589.0550.1917.27%563.70993.27E-040.190.09LIU'SCONFLICT0.80.4107.6560.6589.8450.4616.55%563.65503.96E-040.190.09LIU'SCONFLICT0.90.4107.6560.6590.8850.8815.58%563.56475.40E-040.190.09 Continuedonnextpage 245

PAGE 260

AppendixCContinuedTable40ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean LIU'SCONFLICT0.10.5107.6560.6588.1150.3518.15%563.78512.51E-040.190.09LIU'SCONFLICT0.20.5107.6560.6588.1150.3518.15%563.78512.51E-040.190.09LIU'SCONFLICT0.30.5107.6560.6588.1150.3618.15%563.78532.50E-040.190.09LIU'SCONFLICT0.40.5107.6560.6588.1350.3718.13%563.78772.48E-040.190.09LIU'SCONFLICT0.50.5107.6560.6588.2250.3818.05%563.76522.69E-040.190.09LIU'SCONFLICT0.60.5107.6560.6588.6450.4617.66%563.77052.64E-040.190.09LIU'SCONFLICT0.70.5107.6560.6589.5850.6516.79%563.73393.00E-040.190.09LIU'SCONFLICT0.80.5107.6560.6590.3851.0016.04%563.67303.72E-040.190.09LIU'SCONFLICT0.90.5107.6560.6591.3451.4115.15%563.58725.00E-040.190.09LIU'SCONFLICT0.10.6107.6560.6588.8350.9717.48%563.83482.10E-040.190.09LIU'SCONFLICT0.20.6107.6560.6588.8350.9717.48%563.83482.10E-040.190.09LIU'SCONFLICT0.30.6107.6560.6588.8350.9717.48%563.83482.10E-040.190.09LIU'SCONFLICT0.40.6107.6560.6588.8550.9917.46%563.83772.08E-040.190.09LIU'SCONFLICT0.50.6107.6560.6588.9650.9917.36%563.81122.28E-040.190.09LIU'SCONFLICT0.60.6107.6560.6589.4151.0316.94%563.81302.27E-040.190.09LIU'SCONFLICT0.70.6107.6560.6590.3851.2816.04%563.77232.62E-040.190.09LIU'SCONFLICT0.80.6107.6560.6591.1651.5915.32%563.70973.27E-040.190.09LIU'SCONFLICT0.90.6107.6560.6592.1552.0114.40%573.68443.55E-040.190.09LIU'SCONFLICT0.10.7107.6560.6589.5451.4316.82%563.83682.08E-040.190.09LIU'SCONFLICT0.20.7107.6560.6589.5451.4316.82%563.83682.08E-040.190.09LIU'SCONFLICT0.30.7107.6560.6589.5451.4316.82%563.83682.08E-040.190.09LIU'SCONFLICT0.40.7107.6560.6589.5451.4316.82%563.83682.08E-040.190.09LIU'SCONFLICT0.50.7107.6560.6589.6051.4216.76%563.82002.21E-040.190.09LIU'SCONFLICT0.60.7107.6560.6590.0351.4116.36%563.82392.18E-040.190.09 Continuedonnextpage 246

PAGE 261

AppendixCContinuedTable40ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean LIU'SCONFLICT0.70.7107.6560.6590.9951.6515.48%563.77182.63E-040.190.09LIU'SCONFLICT0.80.7107.6560.6591.7051.9614.81%563.72173.14E-040.180.09LIU'SCONFLICT0.90.7107.6560.6592.6952.3813.90%573.69223.45E-040.180.09LIU'SCONFLICT0.10.8107.6560.6593.8153.3712.86%573.95331.35E-040.160.08LIU'SCONFLICT0.20.8107.6560.6593.8153.3712.86%573.95331.35E-040.160.08LIU'SCONFLICT0.30.8107.6560.6593.8153.3712.86%573.95331.35E-040.160.08LIU'SCONFLICT0.40.8107.6560.6593.8153.3712.86%573.95331.35E-040.160.08LIU'SCONFLICT0.50.8107.6560.6593.8153.3712.86%573.95331.35E-040.160.08LIU'SCONFLICT0.60.8107.6560.6593.8153.3712.86%573.95331.35E-040.160.08LIU'SCONFLICT0.70.8107.6560.6594.5453.5712.18%573.93941.42E-040.160.08LIU'SCONFLICT0.80.8107.6560.6595.3853.9111.39%573.87081.83E-040.160.08LIU'SCONFLICT0.90.8107.6560.6596.0654.1610.76%573.85391.94E-040.160.08LIU'SCONFLICT0.10.9107.6560.65101.6257.275.60%583.67143.69E-040.100.06LIU'SCONFLICT0.20.9107.6560.65101.6257.275.60%583.67143.69E-040.100.06LIU'SCONFLICT0.30.9107.6560.65101.6257.275.60%583.67143.69E-040.100.06LIU'SCONFLICT0.40.9107.6560.65101.6257.275.60%583.67143.69E-040.100.06LIU'SCONFLICT0.50.9107.6560.65101.6257.275.60%583.67143.69E-040.100.06LIU'SCONFLICT0.60.9107.6560.65101.6257.275.60%583.67143.69E-040.100.06LIU'SCONFLICT0.70.9107.6560.65101.6257.275.60%583.67143.69E-040.100.06LIU'SCONFLICT0.80.9107.6560.65101.6257.275.60%583.67143.69E-040.100.06LIU'SCONFLICT0.90.9107.6560.65101.9657.385.28%593.64084.07E-040.110.06TBMCONFLICT0.1107.6560.6561.9043.5742.50%493.30311.34E-030.570.28TBMCONFLICT0.14107.6560.6561.9043.5742.50%493.30311.34E-030.570.28TBMCONFLICT0.18107.6560.6561.9043.5742.50%493.30311.34E-030.570.28 Continuedonnextpage 247

PAGE 262

AppendixCContinuedTable40ContinuedBaselineIsolatingOverlapMethodT1T2MeanMean%Improvementn0tpMean TBMCONFLICT0.22107.6560.6561.9043.5742.50%493.30311.34E-030.570.28TBMCONFLICT0.26107.6560.6561.9043.5742.50%493.30311.34E-030.570.28TBMCONFLICT0.3107.6560.6561.8943.5742.51%493.30351.34E-030.570.28TBMCONFLICT0.34107.6560.6561.8943.5842.51%493.30361.34E-030.570.28TBMCONFLICT0.38107.6560.6561.8943.5742.51%493.30351.34E-030.570.28TBMCONFLICT0.42107.6560.6561.8943.6042.50%493.30201.35E-030.570.28TBMCONFLICT0.46107.6560.6561.8843.6042.52%493.30161.35E-030.570.28TBMCONFLICT0.5107.6560.6562.1743.5042.25%493.29061.40E-030.570.28TBMCONFLICT0.54107.6560.6563.0243.4441.46%503.36641.09E-030.560.28TBMCONFLICT0.58107.6560.6563.6443.5840.88%503.37071.07E-030.560.28TBMCONFLICT0.62107.6560.6564.8343.6039.78%503.36161.11E-030.550.27TBMCONFLICT0.66107.6560.6566.1843.6538.52%503.34641.16E-030.550.27TBMCONFLICT0.7107.6560.6567.5343.6437.27%513.39959.71E-040.540.26TBMCONFLICT0.74107.6560.6568.6343.6536.25%513.38081.03E-030.540.26TBMCONFLICT0.78107.6560.6569.5943.7435.35%513.35241.13E-030.530.26TBMCONFLICT0.82107.6560.6570.7643.9634.26%513.33771.19E-030.530.26TBMCONFLICT0.86107.6560.6572.5144.2732.65%523.39019.95E-040.520.25TBMCONFLICT0.9107.6560.6574.6844.6030.63%523.35111.13E-030.520.25 248

PAGE 263

AppendixCContinuedTable41:Rawresultsforisolationbrokendownbytestbed.BaselineIsolatingOverlapMethodTTestbedMeanMeannn0tpMean LaserorCanesta TBMCONFLICT0.38Lab219.31146.5092.8752.72264221.60301.16E-010.590.09TBMCONFLICT0.38Walkway122.8363.1455.9738.63264241.61521.13E-010.630.34TBMCONFLICT0.38Bridge166.26120.8774.5347.97330211.22082.29E-010.620.16TBMCONFLICT0.38Cubicle157.42114.3870.4440.91330271.53111.32E-010.590.19TBMCONFLICT0.38Sidewalk98.3044.7658.7624.37330423.16542.17E-030.490.21RandomLab80.2151.29658.88395.0426422-2.16473.61E-020.020.01RandomWalkway43.7536.56613.95462.2026421-1.50911.39E-010.010.01RandomBridge57.6447.57680.45390.5433024-2.30562.57E-020.010.01RandomCubicle48.1827.58776.51439.0333035-3.26611.71E-030.010.01RandomSidewalk50.8923.39282.67112.7733043-5.34347.68E-070.010.01 LaserOnly TBMCONFLICT0.38Lab397.7079.92141.8035.838882.94391.07E-020.580.02TBMCONFLICT0.38Walkway198.1329.1957.1722.5588125.18103.40E-050.800.06TBMCONFLICT0.38Bridge323.1346.56128.4735.2711092.40052.89E-020.610.07TBMCONFLICT0.38Cubicle307.9936.24111.8726.86110103.84961.17E-030.730.03TBMCONFLICT0.38Sidewalk126.7839.2367.1117.3311071.11622.86E-010.400.19RandomLab135.8831.081155.14204.658810-5.21825.80E-050.030.01RandomWalkway38.1817.491150.77253.698813-5.73506.56E-060.010.00RandomBridge119.4224.031162.91208.7011016-7.32823.67E-080.020.00RandomCubicle68.9518.861311.53264.8911015-6.34497.27E-070.010.00RandomSidewalk66.4314.94364.5072.4811013-4.81446.66E-050.020.00 Continuedonnextpage 249

PAGE 264

AppendixCContinuedTable41ContinuedBaselineIsolatingOverlapMethodTTestbedMeanMeannn0tpMean CanestaOnly TBMCONFLICT0.38Lab130.1171.2268.4141.64176273.34981.51E-030.590.11TBMCONFLICT0.38Walkway85.1935.9455.3644.59176221.82237.55E-020.540.38TBMCONFLICT0.38Bridge87.8248.2747.5725.42220394.01181.40E-040.620.18TBMCONFLICT0.38Cubicle82.1343.8149.7229.35220273.32711.62E-030.520.19TBMCONFLICT0.38Sidewalk84.0640.4054.5926.27220353.55706.89E-040.530.20RandomLab52.3733.68411.57170.1317630-5.22502.47E-060.010.01RandomWalkway46.5442.82345.67268.8117616-1.18252.46E-010.010.01RandomBridge26.7515.39438.01174.8122043-7.16892.71E-100.010.00RandomCubicle37.8025.31509.94200.3322037-6.20113.16E-080.010.01RandomSidewalk43.1123.00241.44107.1122037-5.01443.69E-060.010.01 250

PAGE 265

AppendixCContinuedThenextthreetablesgivetherawresultsfortheanalysisofthesourcesoferrorandinconsis-tencyforboththelaserandCanestarangecamera,onlylaser,andonlyCanestareadings,brokendownbytestbedandoverall.Forreferencethesetablescontainthepercentageofcorrectclassi-cationsandthetotalnumberofclassicationsn.Thetablesgivethemeanandstandarddeviationofthepercentageoferroneouscells,percentageofsuspectcells,thenumberofer-roneouscells,andthenumberofinconsistentcellsforsensinganomaliesAnomaly,localizationandquantizationerrorsLocalization,andsensornoiseNoiseforeachclassicationperformed.Table42.Rawresultsfortheanalysisofthesourcesoferrorandinconsistency.TestbedLabWalkwayBridgeCubicleSidewalkOverall n129,960111,720169,352166,708149,304727,044%Correct92.23%75.22%74.71%28.71%64.28%65.23% %ofErroneousCells Anomaly40.05%17.03%37.32%22.81%34.23%30.73%Anomaly26.01%34.62%26.86%34.31%31.52%31.92%Localization48.03%48.58%48.65%72.94%53.71%55.14%Localization29.67%27.32%31.89%39.60%33.41%34.59%Noise11.89%34.17%13.95%4.25%11.64%13.99%Noise13.07%19.65%19.89%8.49%15.11%18.21% %ofInconsistentCells Anomaly43.65%6.94%36.96%22.19%37.48%30.26%Anomaly22.84%17.85%29.06%31.38%27.84%29.59%Localization44.51%41.56%48.63%70.80%43.15%50.77%Localization28.24%26.21%34.61%40.52%31.59%35.12%Noise11.65%42.11%13.39%6.92%16.44%16.63%Noise18.49%22.12%22.55%15.36%21.53%23.07% #ofErroneousCells Anomaly80.6113.8753.3218.6129.3739.26Anomaly41.0833.8431.2436.4330.3042.03Localization134.5678.22116.77173.6052.66113.89Localization108.4857.56101.34138.7540.51107.78Noise37.1550.8826.416.1512.0824.50Noise44.5432.1934.979.1216.1233.41 #ofInconsistentCells Anomaly115.963.1768.6520.3849.4752.04Anomaly71.408.3064.5634.3032.8261.46Localization158.8662.68119.92190.9870.55124.24Localization138.1447.87106.07160.1864.43124.63Noise35.1163.6625.2910.9321.1528.80Noise49.9047.5543.6821.7026.8642.13 251

PAGE 266

AppendixCContinuedTable43.Rawresultsfortheanalysisofthesourcesoferrorandinconsistencyforlaserreadingsonly.TestbedLabWalkwayBridgeCubicleSidewalkOverall n75,12071,47295,47296,38084,512422,956Percentcorrect90.17%96.15%50.68%21.71%50.27%58.69% %ofErroneousCells Anomaly24.89%0.00%23.43%0.00%15.69%12.85%Anomaly3.35%0.00%10.07%0.00%2.79%12.02%Localization63.66%58.59%67.88%98.58%76.35%74.25%Localization8.99%16.78%8.08%1.86%10.09%17.44%Noise11.44%41.41%8.69%1.42%7.95%12.90%Noise8.95%16.78%8.34%1.86%11.59%16.79% %ofInconsistentCells Anomaly33.30%0.00%22.85%0.00%24.24%15.91%Anomaly6.58%0.00%12.67%0.00%9.70%15.54%Localization60.79%44.12%72.17%99.08%64.40%69.99%Localization6.33%19.33%13.38%1.56%12.94%21.75%Noise5.92%55.88%4.98%0.92%11.36%14.10%Noise5.71%19.33%4.52%1.56%12.45%21.72% #ofErroneousCells Anomaly89.570.0066.170.0017.4134.32Anomaly25.020.0023.950.004.9039.26Localization223.57111.60200.28287.7884.44186.22Localization37.5641.5745.9245.2720.1784.97Noise49.4071.6829.934.6912.2331.16Noise46.8017.7631.286.6618.6136.21 #ofInconsistentCells Anomaly124.360.0063.060.0040.6244.44Anomaly46.710.0047.780.0021.8654.75Localization222.1765.46170.45267.5997.34169.42Localization72.9928.0045.7768.7142.6992.45Noise28.5284.1513.403.2119.1426.87Noise34.7833.2212.995.7426.3436.35 252

PAGE 267

AppendixCContinuedTable44.Rawresultsfortheanalysisofthesourcesoferrorandinconsistencyforrangecamerareadingsonly.TestbedLabWalkwayBridgeCubicleSidewalkOverall n54,84040,24873,88070,32864,792304,088Percentcorrect93.22%70.00%62.80%75.48%73.92%74.54% %ofErroneousCells Anomaly60.82%47.27%55.26%54.06%58.42%55.60%Anomaly29.01%43.56%30.86%33.19%35.31%34.22%Localization26.62%30.79%23.80%37.79%24.18%28.55%Localization34.39%32.81%33.94%39.69%29.97%34.95%Noise12.51%21.32%20.76%8.12%16.45%15.50%Noise17.16%17.70%27.11%11.84%17.60%19.93% %ofInconsistentCells Anomaly56.11%19.33%55.62%52.45%55.17%50.07%Anomaly29.81%25.94%33.93%26.90%33.81%32.95%Localization24.53%28.08%19.46%31.52%15.01%23.35%Localization33.50%32.09%32.08%35.38%26.46%32.63%Noise19.17%26.55%23.08%15.84%23.27%21.20%Noise25.43%19.94%31.18%21.44%27.99%26.30% #ofErroneousCells Anomaly68.3338.4936.7144.1044.9746.12Anomaly53.6747.2331.7444.9740.6744.71Localization12.6318.948.8517.1311.2113.28Localization15.3425.2012.4521.7314.6718.24Noise20.3613.9421.878.1411.8815.24Noise34.8213.5538.7611.3912.1326.44 #ofInconsistentCells Anomaly75.5710.4863.2058.6857.2756.14Anomaly67.7714.3879.9543.9146.5660.61Localization15.9417.139.1921.409.1914.28Localization19.2522.8113.6527.2214.9120.61Noise40.5915.8434.8926.5821.0428.52Noise68.4914.6467.9437.0324.8850.15 253

PAGE 268

AbouttheAuthorJenniferGageisaPh.D.candidateinComputerScienceandEngineeringattheUniversityofSouthFlorida.SheholdsaMastersandaBSinComputerSciencefromtheUniversityofSouthFlorida.Ms.Gage'sresearchinterestsincludemobilerobots,sensing,articialintelligence,faulttolerance,andautonomiccomputing.ShehastaughtacourseinarticialintelligenceattheUni-versityofSouthFlorida.