USF Libraries
USF Digital Collections

Rem :

MISSING IMAGE

Material Information

Title:
Rem : relational entropy-based measure of saliency
Physical Description:
Book
Language:
English
Creator:
Duncan, Kester
Publisher:
University of South Florida
Place of Publication:
Tampa, Fla
Publication Date:

Subjects

Subjects / Keywords:
Bottom-Up
Renyi Entropy
Relational Histograms
Scale-Variation
Perceptual Grouping
Dissertations, Academic -- Computer Science & Engineering -- Masters -- USF   ( lcsh )
Genre:
non-fiction   ( marcgt )

Notes

Abstract:
ABSTRACT: The incredible ability of human beings to quickly detect the prominent or salient regions in an image is often taken for granted. To be able to reproduce this intelligent ability in computer vision systems remains quite a challenge. This ability is of paramount importance to perception and image understanding since it accelerates the image analysis process, thereby allowing higher vision processes such as recognition to have a focus of attention. In addition to this, human eye fixation points occurring during the early stages of visual processing, often correspond to the loci of salient image regions. These regions provide us with assistance in determining the interesting parts of an image and they also lend support to our ability to discriminate between different objects in a scene. Salient regions attract our immediate attention without requiring an exhaustive scan of a scene. In essence, saliency can be defined as the quality of an image region that enables it to stand out in relation to its neighbors. Saliency is often approached in either one of two ways. The bottom-up saliency approach refers to mechanisms which are image-driven and independent of the knowledge in an image, whereas the top-down saliency approach refers to mechanisms which are task-oriented and make use of the prior knowledge about a scene. In this thesis, we present a bottom-up measure of saliency based on the relationships exhibited among image features. The perceived structure in an image is determined more by the relationships among features rather than the individual feature attributes. From this standpoint, we aim to capture the organization within an image by employing relational distributions derived from distance and gradient direction relationships exhibited between image primitives. The R\'enyi entropy of the relational distribution tends to be lower if saliency is exhibited for some image region in the local pixel neighborhood over which the distribution is defined. This notion forms the foundation of our measure. Correspondingly, results of our measure are presented in the form of a saliency map, highlighting salient image regions. We show results on a variety of real images from various datasets. We evaluate the performance of our measure in relation to a dominant saliency model and obtain comparable results. We also investigate the biological plausibility of our method by comparing our results to those captured by human fixation maps. In an effort to derive meaningful information from an image, we investigate the significance of scale relative to our saliency measure, and attempt to determine optimal scales for image analysis. In addition to this, we extend a perceptual grouping framework by using our measure as an optimization criterion for determining the organizational strength of edge groupings. As a result, the use of ground truth images is circumvented.
Thesis:
Thesis (MSCS)--University of South Florida, 2010.
Bibliography:
Includes bibliographical references.
System Details:
Mode of access: World Wide Web.
System Details:
System requirements: World Wide Web browser and PDF reader.
Statement of Responsibility:
by Kester Duncan.
General Note:
Title from PDF of title page.
General Note:
Document formatted into pages; contains X pages.

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
usfldc doi - E14-SFE0004539
usfldc handle - e14.4539
System ID:
SFS0027854:00001


This item is only available as the following downloads:


Full Text

PAGE 1

REM:RelationalEntropy-BasedMeasureofSaliency by KesterDuncan Athesissubmittedinpartialfulllment oftherequirementsforthedegreeof MasterofScienceinComputerScience DepartmentofComputerScienceandEngineering CollegeofEngineering UniversityofSouthFlorida MajorProfessor:SudeepSarkar,Ph.D. DmitryGoldgof,Ph.D. RangacharKasturi,Ph.D. DateofApproval: May7,2010 Keywords:Bottom-Up,RenyiEntropy,RelationalHistograms,Scale-Variation, PerceptualGrouping Copyright c 2010,KesterDuncan

PAGE 2

DEDICATION Tomymother,andtherestofmyfamily,fortheirlovingencouragementovertheyears.

PAGE 3

ACKNOWLEDGEMENTS FirstandforemostIwouldliketothankmyHeavenlyFatherthroughJesusChrist forbeingmyprovider,mystrength,myrock,mypeace,myshield,mygreatreward,and mylife.Iwouldalsoliketoacknowledgemyadvisor,Dr.Sarkar,whohasbeenpatient withmeduringthisjourney.Ithankhimforhisimmeasurablesupportandthewealth ofknowledgeandwisdomheprovided.Manythanksgoouttomyfamilyforproviding themoralsupportnecessaryforthisventure.ThanksalsotoMichelle,Sharrine,Lasceeka, Gabe,Darnell,Rianna,Sean,Liselle,andcountlessothersformakinglifeasastudent worthliving.Additionally,IwouldliketoexpressmygratitudetoTammyAvraham, NabilOuerhani,andRomanvonWartburgfortheirvaluableassistance.

PAGE 4

TABLEOFCONTENTS LISTOFTABLESiii LISTOFFIGURESiv ABSTRACT vi CHAPTER1INTRODUCTION1 1.1WhatisSaliency?1 1.2ContributionsofthisWork4 1.3LayoutofThesis5 CHAPTER2RELATEDWORK6 2.1Bottom-UpSaliencyApproaches6 2.2Top-DownSaliencyApproaches7 2.3IntegratedSaliencyApproach8 2.4RelationalHistograms8 CHAPTER3REM:RELATIONALENTROPY-BASEDMEASURE10 3.1RelationalDistributions10 3.2Pixel-BasedFeatures11 3.3Sampling14 3.4Entropy14 3.5SaliencyMeasure16 3.6R.E.M.SaliencyMap18 3.7ScaleSpace19 3.8R.E.M.asanOptimizationCriterion19 3.9LearningAutomata21 CHAPTER4RESULTSANDANALYSES23 4.1Datasets23 4.2GeneralPerformanceofSaliencyMeasure24 4.2.1EvaluationwithRegardstoHighlightingPedestrians24 4.2.2EvaluationwithRegardstoHighlightingTracSigns27 4.3ComparisonwithHumanSaliencyMaps29 4.3.1SubjectiveComparison30 4.3.2ObjectiveComparison32 4.4ComparisonwithaDominantSaliencyModel32 4.5ScaleVariation36 4.5.1EvaluationataSpeciedPixelLocation37 i

PAGE 5

4.5.2EvaluationoftheSaliencyMapatDierentScales42 4.6ComparisonofShannonandRenyiEntropies44 4.7EvaluatingHistogramBinSize45 4.8EvaluationoftheExtensiontoaPerceptualGroupingFramework46 CHAPTER5CONCLUSIONANDFUTUREWORK49 REFERENCES50 ii

PAGE 6

LISTOFTABLES Table4.1Correlationcoecients between HumanFixationMaps [1]andREM saliencymaps.32 Table4.2Correlationcoecients of REM saliencymapsand iLab saliencymaps withhumansaliencymaps[2].34 Table4.3Correlationcoecients between REM and iLab saliencymapswith theirrespective HumanFixationMaps [1].34 Table4.4Comparisonof G valuesusingRenyiandShannonentropyimages weretakenfromthePASCALchallengedataset[3].44 iii

PAGE 7

LISTOFFIGURES Figure1.1Thelledsquareisimmediatelyperceivedbysimplyglancingatthe image.2 Figure3.1Pixel-basedbinaryrelationaldistribution.12 Figure3.2Imagesandtheirrelationaldistributionshistograms.13 Figure3.3Localpixelneighborhoodstomeasurethesaliencyofacentralpixel imagebestviewedincolor.17 Figure3.4Animageanditssaliencymap.18 Figure3.5Systemblockdiagramofthealteredperceptualgroupingframework.20 Figure4.1 REM saliency'sperformancewithregardstohighlightingpedestrians.26 Figure4.2 REM saliency'sperformancewithregardstohighlightingtracsigns.28 Figure4.3Originaltestimagesusedforthehumanxationmapcomparisons.29 Figure4.4Comparingsaliencyresultsand humansaliencymaps .31 Figure4.5Originaltestimagesusedforcomparisonwiththestateoftheart.33 Figure4.6Comparing REM saliencyresultswith humansaliencymaps and iLab .33 Figure4.7Comparingsaliencyresultsand humansaliencymaps .35 Figure4.8R.E.M.vs.iLab-graphofthecorrelationcoecientsfor120images fromtheBruceandTsotsosdataset.36 Figure4.9`Baby'imageevaluatedatthepixellocation244 ; 180.37 Figure4.10PlotoftheentropyvaluesfortheimageinFigure4.9withconvergence occurringataneighborhoodsizeofx165.38 Figure4.11`Bedroom'imageevaluatedatthepixellocation ; 56.38 Figure4.12PlotoftheentropyvaluesfortheimageinFigure4.11withconvergence occurringataneighborhoodsizeofx157.39 Figure4.13`Livingroom'imageevaluatedatthepixellocation ; 170.39 iv

PAGE 8

Figure4.14PlotoftheentropyvaluesfortheimageinFigure4.13withconvergence occurringataneighborhoodsizeofx101.40 Figure4.15`Oce'imageevaluatedatthepixellocation ; 71.40 Figure4.16PlotoftheentropyvaluesfortheimageinFigure4.15withconvergence occurringataneighborhoodsizeofx135.41 Figure4.17Scalespaceevaluationof helmets image.42 Figure4.18Scalespaceevaluationof band image.43 Figure4.19Scalespaceevaluationof beer bottles image.43 Figure4.20Eectsofvaryingthenumberofhistogrambinsonthesaliencymap.45 Figure4.21Perceptualgroupingextensionresults.47 v

PAGE 9

REM:RelationalEntropy-BasedMeasureofSaliency KesterDuncan ABSTRACT Theincredibleabilityofhumanbeingstoquicklydetecttheprominentorsalient regionsinanimageisoftentakenforgranted.Tobeabletoreproducethisintelligent abilityincomputervisionsystemsremainsquiteachallenge.Thisabilityisofparamount importancetoperceptionandimageunderstandingsinceitacceleratestheimageanalysis process,therebyallowinghighervisionprocessessuchasrecognitiontohaveafocusof attention.Inadditiontothis,humaneyexationpointsoccurringduringtheearlystages ofvisualprocessing,oftencorrespondtothelociofsalientimageregions.Theseregions provideuswithassistanceindeterminingtheinterestingpartsofanimageandtheyalso lendsupporttoourabilitytodiscriminatebetweendierentobjectsinascene.Salient regionsattractourimmediateattentionwithoutrequiringanexhaustivescanofascene. Inessence,saliencycanbedenedasthequalityofanimageregionthatenablesitto standoutinrelationtoitsneighbors. Saliencyisoftenapproachedineitheroneoftwoways.Thebottom-upsaliencyapproachreferstomechanismswhichareimage-drivenandindependentoftheknowledgein animage,whereasthetop-downsaliencyapproachreferstomechanismswhicharetaskorientedandmakeuseofthepriorknowledgeaboutascene.Inthisthesis,wepresenta bottom-upmeasureofsaliencybasedontherelationshipsexhibitedamongimagefeatures. Theperceivedstructureinanimageisdeterminedmorebytherelationshipsamongfeatures ratherthantheindividualfeatureattributes.Fromthisstandpoint,weaimtocapturethe organizationwithinanimagebyemployingrelationaldistributionsderivedfromdistance vi

PAGE 10

andgradientdirectionrelationshipsexhibitedbetweenimageprimitives.TheRenyientropyoftherelationaldistributiontendstobelowerifsaliencyisexhibitedforsomeimage regioninthelocalpixelneighborhoodoverwhichthedistributionisdened.Thisnotion formsthefoundationofourmeasure. Correspondingly,resultsofourmeasurearepresentedintheformofasaliencymap, highlightingsalientimageregions.Weshowresultsonavarietyofrealimagesfromvarious datasets.Weevaluatetheperformanceofourmeasureinrelationtoadominantsaliency modelandobtaincomparableresults.Wealsoinvestigatethebiologicalplausibilityof ourmethodbycomparingourresultstothosecapturedbyhumanxationmaps.Inan eorttoderivemeaningfulinformationfromanimage,weinvestigatethesignicanceof scalerelativetooursaliencymeasure,andattempttodetermineoptimalscalesforimage analysis.Inadditiontothis,weextendaperceptualgroupingframeworkbyusingour measureasanoptimizationcriterionfordeterminingtheorganizationalstrengthofedge groupings.Asaresult,theuseofgroundtruthimagesiscircumvented. vii

PAGE 11

CHAPTER1 INTRODUCTION 1.1WhatisSaliency? Certainstructuresorregionsinasceneoftenattractourimmediateattentionwithout requiringanexhaustivescanofthesceneitself.Thewaytheseregionsarecapturedbythe HumanVisualSystemHVSwithouttheneedforfocusedattentionisoftendescribedas pre-attentive processingwhichwassuggestedbyNeisserin[4]astherstofthetwostagesof humanvisualprocessing.Neissersuggestedthathumanvisualprocessingwasdividedinto the pre-attentive stage,andthe attentive stage.Thepre-attentivestageconsistsofparallel processesthatoperateconcurrentlyonlargeregionsofthevisualeld,formingstructures towhichattentioncanbedirected.Theattentivestageconsistsofafocusedprocessing eortappliedtoarestrictedregionofthevisualeld.Atthisstage,relationshipsbetween imagefeaturesarefoundandgroupingoccurs.Thesetofvisualpropertiesofascenethat areprocessedpre-attentivelyislimited.Furthermore,anythingperceivedwithinthepreattentivetimeframewhichistypically200millisecondsincorporatesonlytheinformation availablefromasinglecursoryglimpse[5]. Intermediateandhigherlevelvisualprocessesonlyutilizeapredenedproportionof theavailablesensoryinformationbeforefurtherprocessing.Thisisdonemostlikelyto reducethecomplexityofsceneanalysis[6].ThisisillustratedinFigure1.1.Bysimply glancingattheimage,thelledsquareisimmediatelyperceived.Thetargetsquarehas thevisualpropertylled"thattheemptydistractorsquaresdonot.Anobservercan perceiveataglancewhethertheobject,inthiscasethelledsquare,ispresentorabsent. Moreover,thehumanbrainandthevisionsystemworkintandemtoidentifysuchrelevant regions[7].Aninstantaneousvisualarousaloccursintheearlystagesofhumanvisual 1

PAGE 12

Figure1.1.Thelledsquareisimmediatelyperceivedbysimplyglancingattheimage. processing[8]asaresultofthesepre-attentivelydistinctivepartsofascene,anditisthis ideathatisoftenreferredtoas saliency Formanyyears,visionenthusiastshavebeeninvestigatinghowthehumanvisualsystem analyzedimageswhicheventuallyleadtothepre-attentivemodelbeingincorporatedinto manycomputervisionalgorithms.Furthermore,thetermsaliencycameintotheforefront afterthevastamountofpsychology-basedworkonselectivevisualattention[9].With regardstocomputervision,saliencycanbedenedasthequalityofanimagefeaturethat allowsittostandoutinrelationtoitsneighboringfeatures.Thesefeaturesarealmost unique,therebymakingitpossibletodiscriminatebetweenobjectsinascene.Theycan alsobeconsideredastheoutliersofthehomogeneousregionaftersegmentationdueto thefactthatthegoalofsegmentationistogroupareasinanimagethatsatisfysome homogeneitypredicate[8].Itmustbenotedhowever,thatsalientregionsinanimagemay notnecessarilybelongtoanobjectofinterest. Additionally,saliencycanoftenprovidethefoundationforavisualattentionmechanism wherebytheneedforcomputationalresourcesissignicantlyreduced[9].Saliencyfeatures areofparamountimportancewhenvisualrecognitionmustbeperformedincluttered scenes.Consequently,theselectionofacommensuratesetofsalientfeaturesformstherst stepinmanycomputervisionalgorithms.Salientfeatures,points,orregions,facilitate 2

PAGE 13

objectrecognition,perceptualorganization,segmentation,andgure-groundseparation becausetheypermitimmediateconcentrationonobjectsofinterestinanimage. Variousdenitionsofsaliencyhavebeenproposedoverthelastfewdecadesandit hasbeenextensivelystudiedinthecomputervisionliterature[10,11,12].Saliencyhas beendescribedasvisualattention,focusofattention,andtheimpetusfortheselectionof xationpoints[13].Saliencyresearchhasitsoriginsinattemptingtoexplainperceptual phenomena,therebyfocusingonbiologicallyplausiblemechanismsforndinglongand smoothcurves[12].Mostsaliencyresearchhowever,hasbeenfocusedontheextraction of interestpoints fromanimagethatexpressedsomestrongmathematicalproperty[11]. Asubstantialamountofworkhasbeenamassedwiththegoalofmodelingmechanisms ofperceptualorganizationsuchascontoursaliencyandmoregeneralGestaltphenomena. Interestpointdetectorshavebeenquitesuccessfulinrecognitionandobjecttrackingapplicationsthereforedemonstratingtheapplicabilityofsaliencymechanisms.Alargebody ofexistingsaliencymechanismshavealsobeeninspiredbytheknownpropertiesofpsychophysicsandthephysiologyofpre-attentivevision.Therehasalsobeenresearchthat concentratedoncomputingthesalientgroupingsoflow-levelfeatures[14]. Thenotionofsaliencyhowever,hasbeenusedimplicitlyinanumberofcomputervision algorithms.Salientregiondetectionhasbeenusedtoextractdescriptionsthatwerethen usedtosolvevisionproblemsrequiringcorrespondence[8].Additionally,theideaofusing edgedetectorstoextractobjectdescriptionsembodiestheideathattheedgesaremore `salient'incomparisontootherpartsoftheimage.Furthermore,oneofthecentraltasks ofperceptualorganizationistodetectsalientstructures.Saliencyhasalsobeenusedfor thedetectionofspecicvisualattributessuchascorners,edges,andcontours.Inmore recentliterature,saliencyhasbeendenedasimagecomplexity[11].Saliencymethods incorporatingthisdenitionoerstheadvantageofgreaterexibilitybecausetheycan detectanyofthelow-levelattributes{corners,contours,andedges. Consequently,mostsaliencymechanismsareapproachedintwoways: bottom-up saliency and top-down saliency.Bottom-upapproachesareanalogoustorapid,imageorstimulusdrivenmechanismsinpre-attentivevisionandaretoagreatextentindependentofthe 3

PAGE 14

knowledgeofthecontentinanimage.Theyarefasttoimplement,requiringnoprior knowledgeofthescene.However,top-downapproachesaregoal-orientedandmakeuseof priorknowledgeaboutthesceneorthecontexttoidentifysalientregions[7].Theyare task-dependenttherebydemandingamorethoroughunderstandingofthecontextofthe image,resultinginhighcomputationalcosts.Top-downmechanismscanbeenvisionedas weakclassiersthatfocusattentionontheregionsofthevisualeldwhicharerelevant. Thesetwoapproachesareanalogoustopre-attentiveandattentivevisionrespectively.Integrationofthesetwoapproacheshasbeendeemedcrucialforrobotnavigation,visual surveillance,andrealisticvisualsearchesandtheyhavebeenstudiedintheliterature[15]. Saliencymechanismsutilizingthisapproachfallintothecategoryknownas integrated approaches. Thereisalsothenotionoflocalsaliencyandstructuralglobalsaliencyasspeciedin [14].Localsaliencytakesplacewhenanimageprimitivebecomesdiscerniblebyhavinga distinguishinglocalpropertysuchascolor,contrast,ororientationthatenablesitto`popout'inrelationtoitsneighbors.Structuralsaliencyoccurswhenthestructureisperceived inamoreglobalmanner[14]wherebypartsofthestructurearenotsalientinisolation.In thisthesis,wepresentameasureofsaliencybasedontherelationsbetweenlow-levelimage features.Ournotionofsaliencyreliesonthedistributionofrelativegradientdirections andtheeuclideandistancesofedgepixels. 1.2ContributionsofthisWork Thesaliencymeasurepresentedinthisworkcanbeintegratedinapplicationsthat performobjecttracking,objectrecognition,andvisualattentionasapreprocessingstep thatcanimproverobustness.Themeasureisalsogenerictherebymoreexiblethan saliencymeasuresthataretiedtospecicvisualfeatures.Itisnotdrivenbyrecognition goalsanditisindependentofcontext,whichisachievedbyfocusingonlow-levelvision.Our measurehighlightstheregionsofanimagewhichpresentsomeformofspatialdiscontinuity orcontrast,andthisiscapturedbythegeometricrelationaldistribution.Itcanalsobe 4

PAGE 15

usedtoextractdescriptions,whichcouldthenbeusedtosolvevisionproblemsinvolving matching.Ourmeasureissensitivetospatialdiscontinuitiesandthiskindofarchitecture iscapableofdetectinglocationswhichstandoutrelativetotheirneighborhood,andthis isageneralcomputationalprincipleoftheretina[15]. 1.3LayoutofThesis Thisthesisislaidoutasfollows.Inchapter2,wesurveythestateofthecommunity intheeldofsaliency,saliencydetection,visualattention,andinterestpointdetection, andpriorworkwithregardstosaliency.Wealsosurveysomeworkdoneusingrelational distributionsandentropy.Inchapter3,wepresentourrelationalentropy-basedmeasure forsaliencyrstbydescribingrelationaldistributionsandtheirapplicability,thenwemove ontothedescriptionofentropyandthedierentmeasuresofentropyused.Thecoreofour measureispresentedhere.Wealsodescribetheextensionthatwemadetoaperceptual groupingframeworkwherebyourmeasureisusedasanoptimizationcriterion.Inchapter 4,wepresentourresultsonvariousimages,includingsaliencymapsandanalyses.We compareourndingstothoseofadominantsaliencymodel.Additionally,wepresentsome resultsfromtheevaluationofourmeasureasanoptimizationcriterionforselectingstrong groupsinaperceptualgroupingframework.Inchapter5,wesummarizeourndingsand discusstheimplicationsofourworkonsaliencyresearchandalsolookatsomepotential directionsforfuturework. 5

PAGE 16

CHAPTER2 RELATEDWORK Therehasbeenextensiveworkonsaliencyoverthelastfewdecades.Avastmajority oftheworkhasbeenamassedinbottom-upapproachestosaliencyratherthantop-down. Amorerecenttrendexistswherebytheintegrationofthesetwoapproachesarebeing explored. 2.1Bottom-UpSaliencyApproaches In[14],Sha'ashuaandUllmanpresentedasaliencymeasurebasedoncurvatureand curvaturevariation.Thestructurestheirmeasureemphasizedwerealsosalientinhuman perception,oftencorrespondingtoobjectsofinterestintheimage.Theauthorssuggested whatmadestructuressalientandproposedamechanismfordetectingsalientlocations usingalocallyconnectednetwork.Motivatedbytheworkdonein[14],Berengoltsand Lindenbaum[12]presentedasaliencymeasurebasedonprobabilisticcues,estimatedlength distributionsandtheexpectedlengthofcurves.Theyshowedthatprobabilisticsaliency hadtheabilitytoopenpathwaysfordierentrealizationsofsalienciesbasedondierent cues,therebyallowingothersourcesofinformationtobeused.Theiraimwastoseparate agurefromitsbackground. KadirandBradyintroducedamultiscalealgorithmforsalientregionselectionand appliedittomatchingproblemssuchastracking,objectrecognition,andimageretrieval[8]. Theirtechniquedeterminedsalientregionsasthoseexhibitingunpredictablecharacteristics simultaneouslyinsomefeature-spaceandoverscale.Theyinvestigatedtheuseofentropy measurestoidentifyregionsofsaliencywithinabroadclassofimagesandimagesequences. Theyusedthelocalintensityasthedescriptorforsaliency.Theyintroducedanovel 6

PAGE 17

algorithmwhichcreatedahierarchyofsalientregionsthatoperatedacrossfeaturescale andspace.Inthisthesis,wealsoinvestigatetheimplicationsofscaleandsaliencyand alsoadopttheargumentpresentedin[8]thatscaleisintimatelyrelatedtotheissueof determiningsaliencyandextractingrelevantdescriptions."Weexplorethisinourmeasure todeterminethescaleneighhborhoodatwhichapixelremainssalient.HareandLewis useKadirandBrady'sscalesaliencymethodforimagematchingandfeature-basedtracking [9]. AvrahamandLindenbaumproposedanovelstochasticmodeltoestimatesaliencyin [16].Theyutilizeacoarsepre-attentivegroupingprocesstoextractuniformregions.These regionswerethenusedasinitialcandidatesforattentionsaliency.Theiresaliency" mechanismdeterminesifanimagepartisofinterestwiththegoalofndingsmallimage regionswheresalientobjectsarepresent. 2.2Top-DownSaliencyApproaches Whenvisualrecognitionmustbeperformedinclutteredscenes,saliencymechanisms areofparamountimportance.Gao,Han,andVasconceloscouplesaliencytotherecognitiongoalin[11].Theyarguedthatthesaliencyjudgmentsbecomesignicantlymore adaptive,onlyhighlightingimageareaswhichwererelevanttorecognition.Theauthors equatesaliencytodiscriminationtherebydeviatingfromexistingmodels.Theyreferred totheoptimalsalientfeaturesasthosethatweremaximallyinformativeofthepresence orabsenceofthetargetclassinaeldofview[11].Additionally,GaoandVasconcelos denediscriminantsaliencyasthefeatureswhoseresponsebestdistinguishesanobjectto berecognizedfromthesetofallothersthatmaybeofpossibleinterest[17].Thisconcept hasbeenappliedtothedesignofobjectrecognitionsystemswhichhavebeenshownto performwell. Gopalakrishnan,Hu,andRajanpresentedasalientregiondetectionframeworkbased onthecolorandorientationdistributioninimages[7].Thisframeworkconsistedofa colorsaliencyframeworkandanorientationframework.Thecolorsaliencyframework 7

PAGE 18

detectedsalientregionsbasedonthespatialdistributionofthecomponentcolorsinthe imagespace.Theorientationframeworkdetectedsalientregionsintheimagesbasedon theglobalandlocalbehaviorofdierentorientationsintheimage,therebymakinguseof theimagecontext.Theyalsoproposedanorientationhistogramasthelocaldescriptorand determinedhowdierentitsentropywasfromthelocalneighborhood,leadingtothenotion oforientationentropycontrast.Ourmeasurepicksoutrelevantpartsofascenebyusing arelationalhistogramofthegradientorientationsanddistancesasthelocaldescriptor. 2.3IntegratedSaliencyApproach IttiandNavalpakkamintegratebothbottom-upandtop-downapproachesofsaliency foranovelapproachin[15].Theyarguedthattheintegrationofbottom-upandtop-down saliencymeasureswasessentialforrobotnavigation,visualsurveillance,andanyrealistic visualsearch.Theirmethoddecomposedthevisualinputintoasetoftopographicfeature maps.Subsequently,dierentspatiallocationscompetedforsaliencywithineachmapsuch thatonlylocationswhichstoodoutlocallyrelativetotheirsurroundingswouldpersist.The bottom-upcomponentwasresponsibleforcomputingthesaliencyoflocationsindierent featuremapswhereasthetop-downcomponentusedstatisticalknowledgeofthetarget objecttotunethebottom-upmaps.Theyelaboratedonthebiologicalmotivationforthe methodstheyutilized. 2.4RelationalHistograms Theideaofusingrelationalhistogramsisnotnew.Theyhavebeenshowntobe quiteeectivefordatabaseindexingandobjectrecognition.Huetetal.usedrelational histogramsin[18]forshapeindexingandtheywereshowntobeaveryecientwayof indexingimagesintolargeimagedatabases.Therelationaldistributionscreatedbygated pairwiseattributehistogramswereusedtorecallcomplexline-patternstherebypermitting rapidsimultaneoushistogramcomparisons.Theuseofthedirectedrelativeangleandthe directedrelativepositionpermittedtheencodingofstructuralinformation.Inthisthesis, 8

PAGE 19

weusetheedgegradientangleandthedistancebetweenpixelsasthepairwiseattributes toconstructourrelationaldistribution. Similarly,VegaandSarkarusedrelationalhistogramsformotion-basedrecognitionof humans[19].Theyusedanormalizedhistogramoftheobservedinter-featurerelationsto showthatitwaspossibletorecognizeindividualsfromtheirjoggingandrunninggaits, andnotjustfromtheirwalkinggait.Moreover,AshbrookandFisher[20]usedpairwise geometrichistogramstorepresentandclassifyarbitrary2 1 2 and3 )]TJ/F15 10.9091 Tf 8.485 0 Td [(dimensionalsurface shapes.Withthisrepresentation,theywereabletondcorrespondencesbetweendierent objectsreliablyandeciently.Osadaetal.alsousedsampleddistributionsforshape-based retrievalin[21].Thackeretal.demonstratedin[22]thatpairwisegeometrichistograms werecompletewithregardstodescribingline-basedapproximationstoarbitrarycurves. Altogether,relationalhistogramshaveprovedtobeusefulforgeometricdescriptionsand itisonthispremisethatwebuildoursaliencymeasure. 9

PAGE 20

CHAPTER3 REM:RELATIONALENTROPY-BASEDMEASURE Oursaliencymeasureisformulatedontheentropyofgeometricrelationaldistributions. Thesetopicsaredescribedinfurtherdetailinthesubsequentsections. 3.1RelationalDistributions Weadoptthenotionspeciedin[19]thatthestructureperceivedinanimageis determinedmorebytherelationshipsamongimagefeaturesratherthanbytheindividual featureattributes."Weutilizeamechanismtocapturethisstructure.Imagestructures canberepresentedbyprobabilityfunctions.Inourcase,theseprobabilityfunctionsare referredtoasrelationaldistributions.Werepresenttheserelationaldistributionsusing relationalhistogramsgeometrichistograms.Theconceptofrelationalhistogramsisnot anoveloneandtheyhavebeenusedextensively.Theywereusedfordatabaseindexing [18],motion-basedrecognitionofhumans[19],shapeanalysis[21],andobjectrecognition [20].Wedenerelationaldistributionsindenitions3.1.1and3.1.2following[19]. Denition3.1.1 Let: F = f f 1 ;:::;f N g representthesetof N featuresinanimage. F k representarandomk-tupleoffeatures,and Therelationshipamongthesek-tuplefeaturesbedenotedbyR k Therefore,pairwiseorbinaryrelationshipsbetweenfeaturesarerepresentedby R 2 .Loworderspatialdependenciesarecapturedbysmallvaluesof k whereashigher-orderdependenciesarecapturedbylargervaluesof k 10

PAGE 21

Denition3.1.2 Lettherelationships R k becharacterizedbyasetof M attributes A k = f A k 1 ;:::;A kM g .Hence,imagestructurescanberepresentedbyjointprobabilityfunctions: P A k = a k ,alsodenotedby P a k 1 ;:::;a kM or P a k ,where a ki isthevaluetakenbythe relationalattribute A ki Theseresultantprobabilitiesarereferredtoas RelationalDistributions .Thesedistributionscanbeinterpretedas:Givenanimage,ifyoupick k -tuplesoffeaturesin ourcase,two,whatistheprobabilitythatitwillexhibittherelationalattributes a k or P A k = a k ?Werepresenttheserelationaldistributionsinanormalizedhistogram.The histogrambinsizecanvary,however,weutilized10x10or100-binhistogramsformost ofourexperiments. 3.2Pixel-BasedFeatures Theconceptofrelationaldistributionsisillustratedbyconsideringthepixelproperties asfeatures.Eachpixel, f i ,isassociatedwiththegradientdirection, i ,estimatedusinga CannyEdgedetector.Tocapturesomestructurebetweentwopixels,weusethedierence betweengradientangles i )]TJ/F20 10.9091 Tf 10.288 0 Td [( j andtheeuclideandistance d i )]TJ/F20 10.9091 Tf 10.288 0 Td [(d j betweenthemasthe attributes, f A 21 ;A 22 g ,of R 2 .Theseattributesareidealbecausetheyareinvariantwith respecttoimageplanerotationandtranslation.Figure3.1depictsthecomputedattributes. Inadditiontotheseattributes,wealsoutilizethegradientmagnitudedierencesbetween pixelsasweights w i forhistogrambinvoting.TheimageinFigure3.1cillustratesthe relationaldistributionthatisformedfortheprobabilityfunction P d; .Wecanseefrom thisgurethatthedistributionismulti-modal.Italsoexhibitssomerepeatedstructure fortherespectiveimage.Foramorecomplexormoreclutteredimage,thisdistribution tendstobemoreuniform,whereasforsimplerimages,thepeaksintheillustrationare moreapparent.Gradientdirectionestimationisperformedonanimageandtherelational distributionisbuilt.Figure3.2presentsavarietyofimagesandtheirrelationaldistribution histogramimages.Theoriginalimagesareontheleftandtherelationaldistributionsfor theimageareontheright.Theverticalaxisrepresentsthegradientanglerangesfrom 11

PAGE 22

aOriginalimage bPixelAttributes cRelationalDistribution P d; Figure3.1.Pixel-basedbinaryrelationaldistribution. 0atthetop-leftto atthebottom-leftandthehorizontalaxisrepresentsthedistance rangesfrom0atthebottomlefttothemaximumdistancepossibleintheimageatthe bottom-right. 12

PAGE 23

Figure3.2.Imagesandtheirrelationaldistributionshistograms. 13

PAGE 24

ForeachimageinFigure3.2,therelationalhistogramwascreatedbasedontheestimatedgradientangles.Thenumberofbinsusedforthesehistogramswas2061x51. Thehorizontalaxisfromlefttorightrepresentsthedistancebetweenpixelswitharange from0tothemaximumpossibledistancefortheimage,whereastheverticalaxisfrom toptobottomrepresentstheedgegradientanglewitharangefrom0to 3.3Sampling Relationaldistributionscanbecomputednaivelybyperforminganexhaustiveenumerationofallfeaturepairs.Thisiscomputationallyexpensivewithatimecomplexity of O n 2 ,where n isthenumberofpixelsinalocalpixelneighborhood.Basedonwork donein[23],wealsofoundthatusingasampling-basedmethodtoestimatetherelational distributionoersanecientalternative.Foreachpixel i ,wesample m pairsofpixels fromtheneighborhoodof i L i tocreatetherelationaldistribution,where m isdirectly proportionalto L i .Wethenrepeatthesamplinguntiltheentropyofthedistributionconvergeschangeintheentropyissmall,inourcase0 : 001.Thecomplexityoftherelational distributioncomputationisreducedto O km ,where k isthenumberofiterationsfor whichtherelationaldistributionisupdated. 3.4Entropy Let P = p 1 ;p 2 ;:::;p n beadiscreteprobabilitydistribution.Theamountofuncertainty ordisorderorrandomnessofthedistribution P isreferredtoastheentropyof P and itismeasuredbythequantity H [ P ]= H p 1 ;p 2 ;:::;p n [24].Inourcase,theprobability distributionis P d; ,whichiscapturedbytherelationalhistogram.Entropydesignates theextenttowhichthefeaturescharacterizedbytherelationalhistogramareuniformly distributed[13].Entropyisdenedbythecommonformby: H P = )]TJ/F21 7.9701 Tf 14.929 12.69 Td [(n X i =1 p x i log 2 p x i : .1 14

PAGE 25

Equation3.1isuniversallyknownas Shannon'sEntropy .Thisformofentropyhas somespecialproperties[25]: Itiscontinuous,sothatsmallamountsofprobabilitychangesonlyresultinsmall amountsofentropychanges. Itissymmetricwherebythemeasureisunchangediftheoutcomes p i arereordered. Itismaximalwhenallthepossibleeventsareequallyprobabletheentropyvalue wouldbethehighestinthiscaseH p 1 ;p 2 ;:::;p n H 1 n ; 1 n ;:::; 1 n .Similarly, Itisadditivesuchthattheamountofentropyisindependentofhowaprocessis dividedintoparts. Itisinvariantwithregardstoaddingorremovinganeventwithzeroprobability. WealsoutilizeanotherformofentropywhichisageneralizationoftheShannonentropy in3.1.Itisdenedasfollows: H P = 1 1 )]TJ/F20 10.9091 Tf 10.909 0 Td [( log 2 n X i =1 p x i ; .2 Equation3.2isknownas Renyi'sEntropy oforder where 0.Increasingvaluesof produceaRenyientropythatisdevisedbyfavoringthehigherprobabilityevents.The probabilityeventsareconsideredmoreequallyforlowervaluesof .When =1,weget theShannonentropy.Weranourexperimentswithan valueof2,whichisdenedas: H 2 P = )]TJ/F20 10.9091 Tf 8.485 0 Td [(log 2 n X i =1 p x i 2 .3 Equation3.3issometimesreferredtoas Collisionentropy .Acomparisonofthethesetwo entropymeasuresispresentedinalatersection.Furthermore,weutilizeanextraterm l )]TJ/F15 10.9091 Tf 11.61 0 Td [(1 log 2 e 2 N shownbyAbein[26]tobethetheexpecteddivergencebetweenanite probabilitydistribution Q on f 1 ; 2 ;:::;l g anditsempiricaloneobtainedfromthesample ofsize N drawnfrom Q ".Thistermisaddedtotheentropyvalue H [ P ]inaneort 15

PAGE 26

tocomputetheexpecteddivergencebetweentheestimatedprobabilitiesandtheactual underlyingprobability. Webelievethatndingtheentropyoftherelationaldistribution P d; isagood indicatorofthepop-out"structuresinanimage.Wedeneoursaliencymeasureinfull inthesubsequentsection. 3.5SaliencyMeasure Wedenesaliencyasthequalityofanimagefeaturethatenablesittostandoutor pop-out"relativetoitsneighbors.Wequantifythisqualitywithanentropicmeasure basedontherelationaldistributionsoflocalpixelneighborhoods.Therefore,let P d; beourrelationaldistributionbasedonthepixelattributesspeciedinsection3.2.Thus, thesaliencyfunction isdenedas: =1 )]TJ/F25 10.9091 Tf 10.909 0 Td [(H i [ P d; ].4 where i iseither1or2forShannon'sentropyorRenyi'sentropyrespectively. Wecalculateourmeasurebothglobally G andlocally L G isnotnecessarily ameasureofsaliency.Itmeasuresthedisordercomplexityexhibitedintheimage. L however,isameasureofpixelsaliencywithregardstosome M ,inwhich M isa k + 1 k +1neighborhoodofapixelwith k> 1.Highervaluesofindicatehighersaliency andviceversa.Additionally,isnormalized.Duetothefactthat G isaglobalmeasure, itismoreproblematicfromacomputationalpointofview.Wecomparepixelpairsto determinethepairwisegeometricrelationshipsbetweentheminaneorttobuildthe relationaldistribution.Inordertoascertain G ,weapproachedtheproblemintwoways: usebruteforcewith n 2 comparisonstogatheraneestimateof G where n isthetotal numberofpixelsorcomparetheedgepixelsofanedgemaptogetacoarseestimateof G .Wefoundempiricallythatthelatterapproachwasanorderofmagnitudefasterthan theformerandlesscomplex.Furthermore,thevaluederivedonlydieredfromthene estimatebyapproximately )]TJ/F15 10.9091 Tf 8.484 0 Td [(0 : 01inmostcases.Wealsonoticedthattheinecientbrute 16

PAGE 27

forceapproachwasanupperboundforthecoarseestimate.Withthisinmind,thelatter approachwaspreferred. Tocalculate L ,whichassignstoeachrespectivepixellocationasaliencyvalue,we consideredthepairwisecomparisonsofpixelsintheneighborhood M i ofacentralpixel f i ,where i istheindexofthepixel.ExamplesofneighborhoodsusedareshowninFigure 3.3.Inthisgure,thecentralpixelisdepictedasayellowdot.Thisprocedureisalso Figure3.3.Localpixelneighborhoodstomeasurethesaliencyofacentralpixelimage bestviewedincolor. administeredtocalculate G with M beingtheentirepixelsetoftheedgemap.The majordierenceisthateachpointisnotgivenasaliencyvalue.Thevaluethatisretrieved isameasureof`organization'.Theeuclideandistance d andthegradientangledierence betweenapixelpairisusedtoconstructthedistributionasdescribedinsection3.2. 17

PAGE 28

3.6R.E.M.SaliencyMap Weconstructasaliencymapwhichisarepresentationofanimageemphasizingsalient locationsasdenedbyourmeasure L .Ouralgorithmtakesagrayscaleimageasinput andproducesagrayscalemaphighlightingsalientstructuresbyassigninganestimated prioritytoeverypixellocation.Saliencymapsareproducedforimagesforvaryinglocal neighborhoodscales.Thebrighterareasofthesaliencymapsignifythemoresalientareas oftheimagesandviceversaasdepictedinFigure3.4.ThismapmaythenbeintensitynormalizedandconvolvedwithaGaussiansmoothinglterforcomparisonwiththeresults ofothersaliencymethodsmeansaliencymap.Wehereafterrefertoanyofoursaliency mapsasthe REM map.InFigure3.4,thevalueof G was=0.171,thesizeofthelocal pixelneighborhoodusedwas11x11with25histogrambins.ThemapwasthenintensitynormalizedandsmoothedwithaGaussiansmoothinglter.Inthiscase,a30x30mask wasusedwith =5.Ifyousimplyglanceattheoriginalimage,someofthestructures aOriginalImage b REM Saliencymap Figure3.4.Animageanditssaliencymap. thatmay popout toyouduringthatbriefattentionspanarerepresentedinthesaliency map.Thiswouldbeexploredfurtherinsubsequentsections. 18

PAGE 29

3.7ScaleSpace Objectsintheworldappearindierentwaysdependingonthescaleofobservation andthisfacthasimportantimplicationsiftheyaretobedescribed.Multi-scalerepresentationsarenecessarytocompletelyrepresentandprocessimages[27].Acharacteristic propertyofstructuresinimagesisthattheymayonlybemeaningfuloverdeniteranges ofscale.Forinstance,amapoftheUnitedStateswouldcontainthelargestcities,towns, andsomeinterstatehighways,whereasacitymapchangesthelevelofabstractionsubstantiallytoincludestreetsandbuildingsetc.Incomputervision,theprimaryfocusison derivingsignicantandmeaningfulinformationfromimagesdeterminingthatsomething ismeaningful"however,iscontextspecic.Consequently,inthisthesis,weexplorethe signicanceofscalerelativetooursaliencymeasureforderivingmeaningfulinformation fromanimageandattempttoselecttheoptimalscalesfortheiranalysis[28].Representationsofscale-spacewouldenableustoanalyzeanimagepointofinterestatdierent scales,yettheydonotindicateatwhichscalesubsequentprocessingmustbeperformed. Aspreviouslynoted,oursaliencymapemphasizessalientlocationsinanimagefora speciedscale.Weprocessanimageatdierentlocalpixelneighborhoodscalesforsquare neighborhoodscalessatisfying2 k +1dimensions,where k =1 ;:::; 5. 3.8R.E.M.asanOptimizationCriterion Inthisthesis,weexploredextendingaperceptualgroupingframeworktodetermineif ourrelationalentropy-basedmeasurewasusefulasanoptimizationcriterionforselecting structurallyorganizededgegroupings.WeutilizedtheperceptualgroupingframeworkdescribedbySoundararajanin[29].Thegoalofthisframeworkwastogrouplow-levelimage featureswhichwerelikelytobelongtoasingleobjectusinggraphspectralpartitioning. Thisunderlyingprocessitselfiscommonlyreferredtoasperceptualorganization. Perceptualorganizationcanbesuccinctlydenedastheabilitytodetect salient structuresandgroupingsinanimagebasedonsomeformoforganizationexhibitedamong them.Theabilitytogroupsalientstructuresisoneofthefundamentalissuesinvisionand 19

PAGE 30

isofparamountimportancetothedesignofvisionsystems.Thegroupingsthatariseasa resultofperceptualorganizationcanbethoughtofasbeingusedtoinitiatetheprocedures usedforobjectrecognitionandotherhigherlevelprocessessinceitsignicantlyreduces thecomplexityandsearchspaceformodelcomparison[30].ThegroupinginthisframeworkisdonebasedonsalientrelationshipsbetweenGestaltprinciples,namelyparallelism, continuity,similarity,symmetry,commonregion,andclosure[31,32]. Theframeworkcastsgroupingparametersasprobabilitieswhicharelearnedfromaset oftrainingimagesofobjectsintheirnaturalcontextstheobjectsofinterestaremanually outlined[29].However,wealterthisfunctionalitybyeliminatingtheuseofgroundtruth images,andincorporateourrelationalentropy-basedmeasureasanoptimizationcriterion. Anoverviewofthisframeworkasalteredinthisthesis,isdepictedinFigure3.5.Constant Figure3.5.Systemblockdiagramofthealteredperceptualgroupingframework. curvatureedgesegmentsformthelow-levelimagefeaturesthatareusedasinputtothe groupingalgorithm.Theoutputconsistsofsalientgroupsoftheseedgesegments.The featuregroupingalgorithmconsistsofscenestructuregraphspecicationandspectral 20

PAGE 31

partitioningofthescenestructuregraph.Aweightedrelationalgraphcapturesthesalient relationshipsamongtheedgesegments. Quanticationoftherelativeimportanceofthesalientrelationshipshasnotfullybeen exploitedinthecomputervisioncommunity.Consequently,withthisgroupingframework, theimportanceofeachrelationshipisparameterizedandislearnedusinganN-player stochasticautomatagameframework[33].Theprobabilitiesthatformthefoundation oftherelationalgraph,alongwiththeotheralgorithmparametersarelearnedbythe automata.Alearningautomatonisanalgorithmthatadaptivelychoosesfromasetof possibleactionsonarandomenvironmentsoastomaximizetheexpectedfeedback[29]." Theenvironmentinourcaseisthegroupingalgorithmalongwiththeimageset.In responsetoachosenaction,theenvironmentproducesarandomlydeterminedoutput whichisusedbythelearningautomatatodecideonthenextaction.Theaimistoselect theactionthatproducesthemaximumexpected formoredetailsonhowthelearning automataworks,thereaderisreferredto[29,33]. 3.9LearningAutomata Thelearningautomatateamdecidesonthecontributionsfortheparametersusedto quantifytherelationships,whichisbasedonthefeedbackthatitreceivesfromtheenvironment.Thisfeedbackmeasurecapturestheperformanceofthegroupingalgorithmonan image.Thiswasoriginallycalculatedbycomparingtheoutputofthegroupingalgorithm withmanuallyoutlinedtrainingimagesgroundtruth.Wealteredthisfeedbackmeasure inaneorttocircumventtheuseofgroundtruthimagesfordeterminingperformance. Therationalebehindthisnewfeedbackmeasuregoesasfollows: Denition3.9.1 Let G denotethesetofgroupsfoundbytheperceptualgroupingalgorithm from[29]andlet N G denotethenumberofgroupsfound.Let bethenewperformance feedbackmeasuredenedasfollows: = 1 N G X g 2G 1 )]TJ/F20 10.9091 Tf 10.909 0 Td [(H [ P d; g ] ; .5 21

PAGE 32

where P d; g istherelationaldistributionformedbythepairwisecomparisonsofthe pixelsof g .Thismeasurerangesfromrangesfromzerotooneandlargervaluesindicate strongergroupingsandviceversa.Othermetricsmightbedesirable,butthisonesuces fortheillustrationoftheessentialideas. 22

PAGE 33

CHAPTER4 RESULTSANDANALYSES Wepresentanalysesandevaluationsofoursaliencymeasureinthissection.Weinvestigateitsperformanceforawidevarietyofimagesandcompareourresultswiththestate oftheart.Wealsopresentresultsforourextensionofaperceptualgroupingframework asdescribedinsection3.8.Inthesubsequentsectionwedetailtheimagedatabasesthat wereusedinthisthesis. 4.1Datasets Awidevarietyofrealimagesfromvariouspubliclyavailabledatasetsweregathered. Thesewereusedinthedesignandevaluationofoursaliencymeasure.Representative sampleimageswerechosenfromeachsetandusedinthisthesisinaneorttoillustrate thegenericandbiologicallyplausiblenatureofourmeasure.Alloftheimagesusedwere originallycolorimages.Theywereconvertedtograyscaleforuseinthiswork. PASCAL: Saliencymechanismsareoftenprecursorstoobjectrecognition.Therefore,webelievedthatitwouldbettingtoevaluateourmeasuresonimagesfromanobject recognitionchallengedataset.Thischallengesetcanbeutilizedforthedevelopment andtestingofrecognitionalgorithms.ThePASCAL 1 dataset[34]oersawidearray ofrealisticimagesforthispurpose.Thisisoneroutewedecidedtoembarkon. Bruce: ImagesutilizedbyBruceandTsotsosfortheirstudyin[1]werealsoused.Wechose 1 PASCALisanacronymforpatternanalysis,statisticalmodeling,andcomputationallearning. 23

PAGE 34

theseimagesduetotheavailabilityoftheirhumanxationmaps,whichfacilitates theevaluationandcomparisonofourresultswiththestateoftheart. iLab: Imagesfromtheubiquitous iLab 2 oftheUniversityofSouthernCaliforniawerealso usedinaneorttoevaluatetheperformanceofoursaliencymeasurewiththeground truthdatathatwasavailable[35]. StreetScenes: TheMIT StreetScenes [36]databasecontainsimagesofurbanscenesfrommanycategories.Weusedthissimplytodetermineifourmeasurehighlightedthepedestrians inascenetosomedegree. 4.2GeneralPerformanceofSaliencyMeasure Inthissection,wepresentavarietyofrealimagesandtheirsaliencymapresults. Aspreviouslynoted,ourmeasureisapurebottom-up,task-independentapproachto saliencydetection.Thereisnoknowledgeaboutthecontextofthescenethatisusedto determinesaliency.Salientregionsaresimplythoseregionswhichstandoutrelativeto theirneighborhood.Sinceweutilizegradientinformation,namelygradientdirectionand magnitude,boundariesofsalientregionsareemphasizedratherthantheirinterior.This isduetothefactthatwithinthesalientregionifcomposedofmanysalientpixels,there maybenothingthatstandsoutlocally,henceuniformity.Thiscanbeobservedinthe subsequentsections. 4.2.1EvaluationwithRegardstoHighlightingPedestrians Weusedoursaliencymeasuretodetermineifpedestrianswouldbeemphasizedas salient insomeimagestakenfromthe StreetScenes database[36].Wemustnotehowever thatpedestriansarenotessentiallysalient.Withourmeasure,ifpedestriansstandout 2 iLabisthenamegiventotheresearchlaboratoryheadedbyProfessorL.IttioftheUniversityof SouthernCalifornia 24

PAGE 35

relativetotheirsurroundings,theywouldberecognizedassalient.Theoriginalimages wereconvertedtograyscaleforevaluationandtherelationaldistributionswerecalculated forneighborhoodsof9x9pixels. 25

PAGE 36

aImageevaluated bMarkedpedestrians cREMsaliencymap Figure4.1. REM saliency'sperformancewithregardstohighlightingpedestrians. 26

PAGE 37

4.2.2EvaluationwithRegardstoHighlightingTracSigns Analogoustohighlightingpedestrians,weusedourmeasuretodetermineiftrac signswouldbeemphasizedinimagestakenfromthe iLab imagedataset[35].Similarlyto thehighlightingofpedestriansinsection4.2.1,tracsignsareonlysalientiftheystand outrelativetotheirsurroundings.Tracsignsshouldbesalientiftheyaretoperform theirfunctioneectively.Theoriginalimageswereconvertedtograyscaleforevaluation. Followingtheprocedureindicatedintheprevioussection,andtherelationaldistributions werecalculatedforneighborhoodsof9x9pixels. 27

PAGE 38

aImageevaluated bMarkedtracsigns cREMsaliencymap Figure4.2. REM saliency'sperformancewithregardstohighlightingtracsigns. 28

PAGE 39

4.3ComparisonwithHumanSaliencyMaps Themaingoalofthisthesiswastodevelopapurebottom-upsaliencymechanism basedonrelationshipsbetweenimagefeaturesandwehighlightthoseimageregionswhich standoutrelativetosomelocalneighborhood.Weanalyzetheperformanceofourproposed methodinrelationtothatobtainedbythehumanvisualattentionmechanism.Weexecuted thisbycomparingoursaliencymapswithempiricalhumanxationmapsorxation densitymapsfromworkdonebyBruceandTsotsosin[1].Thehumansaliencymaps werecapturedbyrecordinghumaneyexationsoveranimagewhichwasdisplayedto testsubjectsforalimitedamountoftime.Eachxationpointintheseimageswerethen convolvedwithaGaussian.Formoredetailsonthisprocedure,thereaderisreferredto [1]. Wefollowtheevaluationmethodsof[2]byusingasubjectiveandobjectivecomparison ofthesaliencymaps.Thesubjectivecomparisonprovidesanapproximateevaluationof thecorrelationbetweenthehumanxationmapandtheREMsaliencymap.Conversely, thecorrelationcoecientbetweenthemapsprovidesamoreobjectivecomparison.This measurehasbeenusedasde-factostandardforcomparingsaliencymaps.Thetestimages usedinthisevaluationareshowninFigure4.3. image1 image2 image3 image4 image5 image6 image7 image8 Figure4.3.Originaltestimagesusedforthehumanxationmapcomparisons. 29

PAGE 40

4.3.1SubjectiveComparison Thismethodgivesusanestimatedideaaboutthecorrelationbetweenthehuman xationandsaliencymapsvisually.Weassignadierentcolorchanneltoeachmap-blue foroursaliencymap,andgreenforthehumanxationmap.Theredchannelisgivena valueof0.Withthiscomparisonimage,anobservercanseewherethemapscorrelateand wheretheydonot.Blackregionsoftheimageindicatetheabsenceofsaliencyforboth maps,whereasbrightregionsindicatesaliencyonbothmaps.Similarly,uncorrelatedparts areeitherblue,whichsigniesREMsaliencybutnohumanxation,orgreen,signifying humanxationbutnosaliencyasdeterminedbyourmethod.Thecomparisonisshown inFigure4.4.Thehumanxationmapswerebasedonthecolorversionsoftheimagesin columna. 30

PAGE 41

aImage evaluated bHuman saliencymap cREMsaliency map dComparison map Figure4.4.Comparingsaliencyresultsand humansaliencymaps 31

PAGE 42

4.3.2ObjectiveComparison ThecorrelationcoecientservesastheobjectivecomparisonbetweenthehumanxationmapandourREMsaliencymap.Thecorrelationcoecient iscalculatedasfollows: = P x [ M h x )]TJ/F20 10.9091 Tf 10.909 0 Td [( h M s x )]TJ/F20 10.9091 Tf 10.909 0 Td [( s ] 2 p P x M h x )]TJ/F20 10.9091 Tf 10.909 0 Td [( h 2 P x M s x )]TJ/F20 10.9091 Tf 10.909 0 Td [( s 2 .1 where M h x isthehumanxationmap, M s x istheREMsaliencymap, h isthe meanintensityofthehumanxationmap M c x ,and s isthemeanintensityofour map M s x .IfthereisnorelationshipbetweenthehumanxationmapandtheREM saliencymap,thecorrelationcoecientis0orverylow.Asthestrengthoftherelationship betweenthehumansaliencymapandREMsaliencymapincreases,sodoesthecorrelation coecient.Aperfectrelationshipgivesacoecientof1.0.Thus,thehigherthecorrelation coecientthebetter.Thevaluesaredisplayedintable4.1.Theimagesarereferencedas image1"toimage8"withregardstohowtheyarelistedinFigure4.3. Table4.1.Correlationcoecients between HumanFixationMaps [1]andREMsaliency maps. image1 image2 image3 image4 image5 image6 image7 image8 0.581 0.607 0.490 0.454 0.397 0.535 0.367 0.188 4.4ComparisonwithaDominantSaliencyModel Followingtheapproachproposedin[2]andadoptedin[16]andsomeofthetestimages whichwereused,wecomparedtheresultsofouralgorithmtothatoftheresultsproduced byiLab's[35,37]availablein[38]saliencyalgorithminrelationtotherespectivehuman saliencymaps.iLab'ssaliencymechanismisconsideredthedominantsaliencymodelin thestateoftheart.Wemustnotethatourmethodonlyusesgrayscaleimages,whereas theiLabmakesuseofthecolorinformation.Theoriginalimageswhichwereusedfor thiscomparisonaredisplayedinFigure4.5.TheREMsaliencymapswereproducedby evaluatingalocalpixelneighborhoodsizeof11x11.Theywerethendownsampledsothat theycouldbecomparedtoiLab'ssaliencymaps.Thecorrelationcoecientsforboth 32

PAGE 43

mechanismsareshownintable4.2.Thecorrelationsbetweenthehumanmapsandthe REM mapsarehigherthanthecorrelationswith iLab saliencyintwooutofthefourimages shown. road coke swissalps forest Figure4.5.Originaltestimagesusedforcomparisonwiththestateoftheart. aImage evaluated bHuman saliencymap c iLab saliency map. d REM saliency map Figure4.6.Comparing REM saliencyresultswith humansaliencymaps and iLab 33

PAGE 44

Table4.2.Correlationcoecients of REM saliencymapsand iLab saliencymapswith humansaliencymaps[2]. coke road swissalps forest REM 0.216 0.280 0.621 0.573 iLab 0.400 0.362 0.523 0.436 Inaddition,wecomparedourresultstoiLab'swithregardsto120humanxationmaps fromtheBruceandTsotsosdatasetsomeexamplesareshowninFigure4.4.Someof theresultsareshowninFigure4.7andtable4.3. Table4.3.Correlationcoecients between REM and iLab saliencymapswiththeir respective HumanFixationMaps [1]. image1 image2 image3 image4 image5 image6 image7 image8 REM 0.581 0.607 0.490 0.454 0.397 0.535 0.367 0.188 iLab 0.470 0.357 0.265 0.369 0.441 0.433 0.106 0.255 Theoverallperformanceforthese120imagesisdisplayedinFigure4.8.In63.3%of theimages,thecorrelationcoecientsbetweentheR.E.M.saliencymapsandthehuman xationmapswerehigherthaniLab's. 34

PAGE 45

aImage evaluated bHuman saliencymap c REM saliency map d iLab saliency map Figure4.7.Comparingsaliencyresultsand humansaliencymaps 35

PAGE 46

Figure4.8.R.E.M.vs.iLab-graphofthecorrelationcoecientsfor120imagesfromthe BruceandTsotsosdataset. 4.5ScaleVariation Werstproceededbyevaluatingoursaliencymeasureforvariouslocalpixelneighborhooddimensionsforaspeciedpixellocation.Thepixellocationswerechosenmanually andtheneighborhooddimensionsextendedfromx3toeither50%oftheimagesize, ortheneighborhoodatwhichconvergencewasachieved.Convergenceisachievedinthis casewhenthereisnosignicantchangeintheentropyofthepixelfortwentyiterations and H p d; > 0 : 001.Keepinmindthatoursaliencymeasureis1 )]TJ/F20 10.9091 Tf 11.011 0 Td [(H [ P d; ],where P d; representstherelationaldistributionofalocalpixelneighborhood.Thisisdepicted inFigures4.9to4.16. 36

PAGE 47

4.5.1EvaluationataSpeciedPixelLocation Inthissection,weaimtodeterminetheoptimalscaleatwhichthesaliencyofapixel canbeanalyzed. Figure4.9.`Baby'imageevaluatedatthepixellocation ; 180. WeseefromFigures4.10,4.12,4.14,and4.16thattheentropyislowerforsmaller neighborhoodsofapixelnotethatoursaliencymeasureissimply1 )]TJ/F20 10.9091 Tf 11.928 0 Td [(entropy .We candeducethatourmeasuredependsstronglyuponthescaleatwhichitismeasured. Thequestionposediswhichneighborhoodsizeisapproximatelyoptimalforourentropic measure?Fromtheseimages,wecanseethatafterascale of15,theentropyvalues convergetoapproximately0.90.Amoreexhaustiveapproachmustbetakentodetermine theexactscaletoderivemeaningfulinformation,whichcanvaryfromimagetoimageand regiontoregion.Consequently,weinvestigatethisbyexaminingthesaliencyvaluesfor eachpixelatdierentscales.Ifanimagepointissalientuptoascale ,thismeansthat 37

PAGE 48

Figure4.10.PlotoftheentropyvaluesfortheimageinFigure4.9withconvergenceoccurringataneighborhoodsizeofx165. Figure4.11.`Bedroom'imageevaluatedatthepixellocation ; 56. itssaliencyvalueremainsrelativelyunchangeduptothisscale.Thus,itpersists.This ideaisinvestigatedinthesubsequentsection. 38

PAGE 49

Figure4.12.PlotoftheentropyvaluesfortheimageinFigure4.11withconvergence occurringataneighborhoodsizeofx157. Figure4.13.`Livingroom'imageevaluatedatthepixellocation ; 170. 39

PAGE 50

Figure4.14.PlotoftheentropyvaluesfortheimageinFigure4.13withconvergence occurringataneighborhoodsizeofx101. Figure4.15.`Oce'imageevaluatedatthepixellocation ; 71. 40

PAGE 51

Figure4.16.PlotoftheentropyvaluesfortheimageinFigure4.15withconvergence occurringataneighborhoodsizeofx135. 41

PAGE 52

4.5.2EvaluationoftheSaliencyMapatDierentScales Inthissection,weevaluatethechangesthatoccurtoasaliencymapoveranarrowrange ofscales.Thescalerange isasfollows: k +1x k +1,where k = f 1 ;:::; 5 g ; = n wouldalwaysreferto n x n .Thesaliencymapswereintensity-normalizedandsmoothed withaGaussiansmoothinglter.Fromtheseimages,wecansubjectivelyconcludethat aTestImage b =3 c =5 d =9 e =17 f =33 Figure4.17.Scalespaceevaluationof helmets image. theborderoftheceilinglightsandthehelmetreectionsarethemostsalientoverthese narrowrangeofscales.Allotherimageregionsfadetothe`background'.Fortheseimages, wecanconcludethatthebordersofthechairsarethemostsalientoverthesenarrowrange ofscales.Theheadsofthebandmemberspersistuptothe =17scale,buttheyarenot soapparentat =33.WecanseefromalltheimagesinFigures4.17-4.19thatthemost salientimageregionspersistthroughtothelargestscale.Lesssalientregionsfadeasthe scaleincreases. 42

PAGE 53

aTestImage b =3 c =5 d =9 e =17 f =33 Figure4.18.Scalespaceevaluationof band image. aTestImage b =3 c =5 d =9 e =17 f =33 Figure4.19.Scalespaceevaluationof beer bottles image. 43

PAGE 54

4.6ComparisonofShannonandRenyiEntropies TheentropymeasurewasformulatedwithboththeShannonandRenyientropies.A comparisonofvaluesforourglobalmeasure G isshownintable4.4.Wecanseefromthe Table4.4.Comparisonof G valuesusingRenyiandShannonentropyimagesweretaken fromthePASCALchallengedataset[3]. Image G usingRenyi'sentropy G usingShannon'sentropy 0.245 0.182 0.161 0.123 0.167 0.120 0.157 0.112 0.156 0.117 tablethat G hashighervalueswhentheRenyientropyisincorporatedandlowervalues whentheShannonentropyisused.Duetothefactthatisinverselyproportionaltothe entropyvalue,thisprovidesevidencethattheShannonentropyisanupperboundtothe Renyi,asiscommonlyknown. 44

PAGE 55

4.7EvaluatingHistogramBinSize Inthissection,weevaluatetheeectofvaryingthehistogrambinsizeonthesaliency mapcomposedof L valuesaswellthevalueof G foranimage.ThelocalpixelneighborhoodsizeusedinFigure4.20tomeasurethesaliencywasx9witharandomsampling percentageof25%.WecanseefromtheFigurethatthecontrastofthesaliencymapis aImageevaluated b25bins, G =0 : 130 c100bins, G =0 : 102 d900bins, G =0 : 072 e3600bins, G =0 : 060 f8100bins, G =0 : 055 Figure4.20.Eectsofvaryingthenumberofhistogrambinsonthesaliencymap. alteredwithanincreaseinthenumberofhistogrambins.Theareasoftheimagewhich werehighlightedassalientpersistedinallthemaps,whereastheareaswhichwerenot highlightedassalientdidnot.Thereisasomewhat`smoothing'eectthatoccurredasa 45

PAGE 56

result.The L valuesofnon-salientregionsleveledoutasindicatedbythehomogenous intensitycharacteristics. 4.8EvaluationoftheExtensiontoaPerceptualGroupingFramework Inthissectionwepresentresultsoftheextensionoftheperceptualgroupingframework of[29].Figure4.21showssomesampleresultsonavarietyofrealimages.Therstcolumn startingfromtheleftdisplaysthegray-levelinputimages.Thesecondcolumndisplays theedgemapofthefeaturesthatmustbegrouped.Thethirdcolumndisplaysthedierent groupsretrievedusingtheoriginalframework,andthefourthcolumndisplaysthegroups retrievedusingourmeasureasanoptimizationcriterion 46

PAGE 57

aOriginalimagebEdgemapcOriginalresultdOurresult Figure4.21.Perceptualgroupingextensionresults. 47

PAGE 58

ThepreliminaryresultsinFigure4.21areencouraging.Withouttheuseofground truthimages,wewereabletoproducesimilarresultstothatoftheoriginalframework whichutilizedgroundtruth.Nosupervisedlearningusinggroundtruthdatawasdonein ourcase.Webelievethatwithsomerenement,ourmeasurecanbeusedasastandalone metricorasametricintandemwithgroundtruthinformationforselectingperceptual edgegroupings. 48

PAGE 59

CHAPTER5 CONCLUSIONANDFUTUREWORK Themaingoalofthisthesiswastodevelopapurebottom-upsaliencymechanism basedonrelationshipsexhibitedbetweenimagefeatures.Wehighlightedthoseimageregionswhichstoodoutrelativetosomelocalpixelneighborhood.Weadoptedabottom-up saliencyapproachduetoitsgenericnatureandexibility.Ourmeasureisnottiedtospecicvisualfeatures.Wevalidatedoursaliencymeasureusingavarietyofimagedatasets. Wedemonstratedhowourresultscoincidewithhumanxationsandalsopresentresults thatarecomparabletoadominantsaliencymodel.Theseresultsareencouraging.Consequently,webelievethatourmeasuremaybeusedasthefoundationofafocusofattention mechanism.Asanextension,ourmeasuremayeliminatetheneedforgroundtruthina perceptualgroupingframeworkwithsomeadjustmentstothewayimagestructuresare captured.Nosupervisedlearningusinggroundtruthdatawasdonewithourextensionto theframework. Forfuturework,weaimtousericherrepresentationstocapturemoreofthelow-level structureinanimage.Wealsoseektoincorporatemoreprobabilisticprinciplesinour measuretomakeitmorerobust.Wealsoaimtoexploreourmeasurewithregardsto videosequencesanddepthestimation.Inaneorttoreducethedimensionalityandmemoryusageinherentwithrelationalhistograms,weaimtoincorporatethekernelentropy componentanalysisintroducedin[39]toestimatetheRenyientropyoflocalneighborhoods.Anotherfuturegoalistoimplementanintegratedsaliencyapproach,incorporating bothtop-downandbottom-upsaliencyapproachestoaidinvisualattentionandobject recognition. 49

PAGE 60

REFERENCES [1]NeilD.B.BruceandJohnK.Tsotsos.Saliency,Attention,andVisualSearch:An InformationTheoreticApproach. JournalofVision ,9:1{24,2009. [2]NabilOuerhani,RomanvonHartburg,HeinzHugli,andReneMuri.EmpiricalValidationoftheSaliency-basedModelofVisualAttention. ElectronicLettersonComputer VisionandImageAnalysis ,3:13{24,2004. [3]M.Everingham,L.VanGool,C.K.I.Williams,J.Winn,andA.Zisserman.The PASCALVisualObjectClassesChallenge2009VOC2009.http://www.pascalnetwork.org/challenges/VOC/voc2009/workshop/index.html. [4]UlricNeisser. CognitivePsychology .Springer-Verlag,NewYork,1edition,1967. [5]ChristopherG.Healey,KelloggS.Booth,andJamesT.Enns.High-speedVisual EstimationUsingPeattentiveProcessing. ACMTransactionsComputer-HumanInteraction ,3:107{135,1996. [6]L.Itti,C.Koch,andE.Niebur.AModelofSaliency-basedVisualAttentionforRapid SceneAnalysis. IEEETransactionsonPatternAnalysisandMachineIntelligence 20:1254{1259,1998. [7]V.Gopalakrishnan,YiqunHu,andD.Rajan.SalientRegionDetectionbyModeling DistributionsofColorandOrientation. IEEETransactionsonMultimedia ,11:892{ 905,2009. [8]TimorKadirandMichaelBrady.Saliency,Scale,andImageDescription. International JournalofComputerVision ,45:83{105,2001. [9]JonathonS.HareandPaulH.Lewis.ScaleSaliency:ApplicationsinVisualMatching, TrackingandView-BasedObjectRecognition.In DistributedMultimediaSystems 2003/VisualInformationSystems2003 ,pages436{440,2003. [10]DashanGaoandNunoVasconcelos.DiscriminantSaliencyforVisualRecognition fromClutteredScenes.In NeuralInformationProcessingSystems ,pages481{488, June2009. [11]DashanGao,SunhyoungHan,andNunoVasconcelos.DiscriminantSaliency,The DetectionofSuspiciousCoincidences,andApplicationstoVisualRecognition. IEEE TransactionsonPatternAnalysisandMachineIntelligence ,31:989{1005,2009. [12]AlexanderBerengoltsandMichaelLindenbaum.OntheDistributionofSaliency. IEEE TransactionsonPatternAnalysisandMachineIntelligence ,28:1973{1987,2006. 50

PAGE 61

[13]NeilBruceandJohnTsotsos.SaliencyBasedonInformationMaximization.In AdvancesinNeuralInformationProcessingSystems18 ,pages155{162,2006. [14]AmnonSha'ashuaandShimonUllman.StructuralSaliency:TheDetectionofGlobally SalientStructuresUsingaLocallyConnectedNetwork.In InternationalConference onComputerVision ,pages321{327,December1988. [15]V.NavalpakkamandL.Itti.AnIntegratedModelofTop-DownandBottom-Up AttentionforOptimizingDetectionSpeed.In IEEEComputerSocietyConferenceon ComputerVisionandPatternRecognition ,pages2049{2056,2006. [16]TamarAvrahamandMichaelLindenbaum.EsaliencyExtendedSaliency:MeaningfulAttentionUsingStochasticImageModeling. IEEETransactionsonPattern AnalysisandMachineIntelligence ,32:693{708,2010. [17]DashanGaoandN.Vasconcelos.Bottom-upSaliencyisaDiscriminantProcess.In IEEE11thInternationalConferenceonComputerVision ,pages1{6,October2007. [18]BenoitHuetandEdwinR.Hancock.RelationalHistogramsforShapeIndexing. IEEE TransactionsonPatternAnalysisandMachineIntelligence ,21:1363{1370,1999. [19]IsidroRobledoVegaandSudeepSarkar.StatisticalMotionModelBasedonthe ChangeofFeatureRelationships:HumanGait-BasedRecognition. IEEETransactions onPatternAnalysisandMachineIntelligence ,25:83{105,2003. [20]A.P.Ashbrook,RobertB.Fisher,CraigRobertson,andNaoufelWerghi.Finding SurfaceCorrespondanceforObjectRecognitionandRegistrationUsingPairwiseGeometricHistograms.In Proceedingsofthe5thEuropeanConferenceonComputer Vision-VolumeII ,pages674{686,1998. [21]RobertOsada,ThomasFunkhouser,BernardChazelle,andDavidDobkin.Shape Distributions. ACMTransactionsonGraphics ,21:807{832,2002. [22]NAThacker,PARiocreux,andRBYates.AssessingtheCompletenessPropertiesof PairwiseGeometricHistograms. ImageandVisionComputing ,13:423{429,1995. [23]S.Nayak,S.Sarkar,andB.Loeding.Distribution-BasedDimensionalityReduction AppliedtoArticulatedMotionRecognition. IEEETransactionsonPatternAnalysis andMachineIntelligence ,31:795{810,2009. [24]AlfredRenyi.OnMeasuresofEntropyandInformation.In ProceedingsoftheBerkeley SymposiumonMathematicalStatisticsandProbability ,pages547{561,1961. [25]RobertM.Grayr. EntropyandInformationTheory .Appleton-Century-Crofts,New York,1edition,1990. [26]SyuujiAbe.ExpectedRelativeEntropybetweenaFiniteDistributionanditsEmpiricalDistribution. SUTJournalofMathematics ,32:149{156,1996. [27]TonyLindeberg.Scale-spaceTheory:ABasicToolforAnalyzingStructuresatDifferentScales. JournalofAppliedStatistics ,28:225{270,1994. 51

PAGE 62

[28]TimorKadirandMichaelBrady.ScaleSaliency:ANovelApproachtoSalientFeature andScaleSelection.In InternationalConferenceonVisualInformationEngineering pages25{28,November2001. [29]SudeepSarkarandPadmanabhanSoundararajan.SupervisedLearningofLargePerceptualOrganization:GraphSpectralPartitioningandLearningAutomata. IEEE TransactionsonPatternAnalysisandMachineIntelligence ,22:504{525,2000. [30]JamesD.McCaerty. HumanandMachineVision-ComputingPerceptualOrganization .EllisHorwoodLimited,NewYork,1edition,1990. [31]DavidG.Lowe. PerceptualOrganizationandVisualRecognition .KluwerAcademic Publishers,Massachusetts,1edition,1985. [32]SudeepSarkar.AnIntroductiontoPerceptualOrganization.In InternationalConferenceonIntegrationofKnowledgeIntensiveMulti-AgentSystems ,pages330{335, 2003. [33]M.A.L.ThathacharandP.S.Sastry.LearningOptimalDiscriminantFunctions throughaCooperativeGameofAutomata. IEEETransactionsonSystems,Man andCybernetics ,17:73{85,1987. [34]J.Ponce,T.L.Berg,M.Everingham,D.A.Forsyth,M.Herbert,S.Lazebnik, M.Marszalek,C.Schmid,B.C.Russell,A.Torralba,C.K.I.Williams,J.Zhang,and A.Zisserman.DatasetIssuesinObjectRecognition. LectureNotesinComputer Science ,4170/2006:29{48,2006. [35]LaurentItti.iLabBottom-UpVisualAttentionHomePage-ExampleImages. http://ilab.usc.edu/bu/images/index.html. [36]StanleyMichaelBileschi. StreetScenes:TowardsSceneUnderstandinginStillImages PhDthesis,MassachusettsInstituteofTechnology,2006. [37]DirkWaltherandChristofKoch.ModelingAttentiontoSalientProto-Objects. Neural Networks ,19:1395{1407,2006. [38]LaurentItti.SaliencyToolbox2.2.http://www.saliencytoolbox.net. [39]R.Jenssen.KernelEntropyComponentAnalysis. IEEETransactionsonPattern AnalysisandMachineIntelligence ,32:847{860,2010. 52


xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam 22 Ka 4500
controlfield tag 007 cr-bnu---uuuuu
008 s2010 flu s 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0004539
035
(OCoLC)
040
FHM
c FHM
049
FHMM
090
XX9999 (Online)
1 100
Duncan, Kester.
0 245
Rem :
b relational entropy-based measure of saliency
h [electronic resource] /
by Kester Duncan.
260
[Tampa, Fla] :
University of South Florida,
2010.
500
Title from PDF of title page.
Document formatted into pages; contains X pages.
502
Thesis (MSCS)--University of South Florida, 2010.
504
Includes bibliographical references.
516
Text (Electronic thesis) in PDF format.
538
Mode of access: World Wide Web.
System requirements: World Wide Web browser and PDF reader.
3 520
ABSTRACT: The incredible ability of human beings to quickly detect the prominent or salient regions in an image is often taken for granted. To be able to reproduce this intelligent ability in computer vision systems remains quite a challenge. This ability is of paramount importance to perception and image understanding since it accelerates the image analysis process, thereby allowing higher vision processes such as recognition to have a focus of attention. In addition to this, human eye fixation points occurring during the early stages of visual processing, often correspond to the loci of salient image regions. These regions provide us with assistance in determining the interesting parts of an image and they also lend support to our ability to discriminate between different objects in a scene. Salient regions attract our immediate attention without requiring an exhaustive scan of a scene. In essence, saliency can be defined as the quality of an image region that enables it to stand out in relation to its neighbors. Saliency is often approached in either one of two ways. The bottom-up saliency approach refers to mechanisms which are image-driven and independent of the knowledge in an image, whereas the top-down saliency approach refers to mechanisms which are task-oriented and make use of the prior knowledge about a scene. In this thesis, we present a bottom-up measure of saliency based on the relationships exhibited among image features. The perceived structure in an image is determined more by the relationships among features rather than the individual feature attributes. From this standpoint, we aim to capture the organization within an image by employing relational distributions derived from distance and gradient direction relationships exhibited between image primitives. The R\'enyi entropy of the relational distribution tends to be lower if saliency is exhibited for some image region in the local pixel neighborhood over which the distribution is defined. This notion forms the foundation of our measure. Correspondingly, results of our measure are presented in the form of a saliency map, highlighting salient image regions. We show results on a variety of real images from various datasets. We evaluate the performance of our measure in relation to a dominant saliency model and obtain comparable results. We also investigate the biological plausibility of our method by comparing our results to those captured by human fixation maps. In an effort to derive meaningful information from an image, we investigate the significance of scale relative to our saliency measure, and attempt to determine optimal scales for image analysis. In addition to this, we extend a perceptual grouping framework by using our measure as an optimization criterion for determining the organizational strength of edge groupings. As a result, the use of ground truth images is circumvented.
590
Advisor: Sudeep Sarkar, Ph.D.
653
Bottom-Up
Renyi Entropy
Relational Histograms
Scale-Variation
Perceptual Grouping
690
Dissertations, Academic
z USF
x Computer Science & Engineering
Masters.
773
t USF Electronic Theses and Dissertations.
4 856
u http://digital.lib.usf.edu/?e14.4539