USF Libraries
USF Digital Collections

Background subtraction using ensembles of classifiers with an extended feature set

MISSING IMAGE

Material Information

Title:
Background subtraction using ensembles of classifiers with an extended feature set
Physical Description:
Book
Language:
English
Creator:
Klare, Brendan F
Publisher:
University of South Florida
Place of Publication:
Tampa, Fla
Publication Date:

Subjects

Subjects / Keywords:
Tracking
Classification
Segmentation
Fusion
Illumination invariant
Dissertations, Academic -- Computer Science -- Masters -- USF   ( lcsh )
Genre:
non-fiction   ( marcgt )

Notes

Abstract:
ABSTRACT: The limitations of foreground segmentation in difficult environments using standard color space features often result in poor performance during autonomous tracking. This work presents a new approach for classification of foreground and background pixels in image sequences by employing an ensemble of classifiers, each operating on a different feature type such as the three RGB features, gradient magnitude and orientation features, and eight Haar features. These thirteen features are used in an ensemble classifier where each classifier operates on a single image feature. Each classifier implements a Mixture of Gaussians-based unsupervised background classification algorithm. The non-thresholded, classification decision score of each classifier are fused together by taking the average of their outputs and creating one single hypothesis. The results of using the ensemble classifier on three separate and distinct data sets are compared to using only RGB features through ROC graphs. The extended feature vector outperforms the RGB features on all three data sets, and shows a large scale improvement on two of the three data sets. The two data sets with the greatest improvements are both outdoor data sets with global illumination changes and the other has many local illumination changes. When using the entire feature set, to operate at a 90% true positive rate, the per pixel, false alarm rate is reduced five times in one data set and six times in the other data set.
Thesis:
Thesis (M.S.C.S.)--University of South Florida, 2008.
Bibliography:
Includes bibliographical references.
System Details:
Mode of access: World Wide Web.
System Details:
System requirements: World Wide Web browser and PDF reader.
Statement of Responsibility:
by Brendan F. Klare.
General Note:
Title from PDF of title page.
General Note:
Document formatted into pages; contains 60 pages.

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
aleph - 002021440
oclc - 428440170
usfldc doi - E14-SFE0002573
usfldc handle - e14.2573
System ID:
SFS0026890:00001


This item is only available as the following downloads:


Full Text

PAGE 1

BackgroundSubtractionUsingEnsemblesofClassierswithanExtendedFeatureSet by BrendanF.Klare Athesissubmittedinpartialfulllment oftherequirementsforthedegreeof MasterofScienceinComputerScience DepartmentofComputerScienceandEngineering CollegeofEngineering UniversityofSouthFlorida MajorProfessor:SudeepSarkar,Ph.D. LawrenceO.Hall,Ph.D. DmitryB.Goldgof,Ph.D. DateofApproval: June30,2008 Keywords:tracking,classication,segmentation,fusion,illuminationinvariant c Copyright2008,BrendanF.Klare

PAGE 2

DEDICATION MythesisisdedicatedtoFredKlare,ChristinaHindman,andthe75thRangerRegiment. Thesehavebeenthelargestinuencesandoeredthemostsupportinmylife.WithoutthemI wouldnotbewhereIamtoday. RLTW!

PAGE 3

ACKNOWLEDGEMENTS Iwouldliketothankeveryonewhohashelpedmecompletethisthesis.Particularly,Iwould liketothankDr.SudeepSarkarforhisinvaluabletutoringandguidancethroughoutmythesis. ThereweremanyoccasionswhenIenteredhisocewithconsiderablefrustrationanddoubt, andleftwithoptimismandfocus. IwanttothanktheDepartmentofComputerScienceandEngineeringforprovidingmewith thefellowshipthathelpedmakethisthesispossible,aswellasthemanyfacultywhotaughtme theknowledgethatenabledthiswork. Finally,IwouldliketothanktheUniversityofSouthFlorida.AfterInishedmytourinthe USArmy,theUniversityofSouthFloridaclearlyshowedthemostinterestinmeasaprospective student,andIbelievebothpartieshavebenetedgreatlyfromthis.

PAGE 4

TABLEOFCONTENTS LISTOFTABLES ii LISTOFFIGURESiii ABSTRACT v CHAPTER1INTRODUCTION1 CHAPTER2RELATEDWORKS7 2.1EarlyWorkinBackgroundModeling7 2.2ModernTechniquesinBackgroundClassication11 2.3IlluminationConsiderations15 2.4EnsembleMethods18 CHAPTER3FEATURES20 3.1GradientFeatures20 3.2HaarFeatures21 3.3AlternateColorSpaces23 CHAPTER4MULTIPLECLASSIFIERALGORITHMS25 4.1ChangestoMixtureofGaussiansAlgorithm25 4.1.1VariableNumberofDistributions25 4.1.2Training27 4.2ClassierFusion28 CHAPTER5RESULTS30 5.1Methodology30 5.2DataSets31 5.3ParameterSelection33 5.4Results36 5.4.1OTCBVSDataSet36 5.4.2PETS2001DataSet37 5.4.3PETS2006DataSet41 5.4.4FeaturePerformance46 CHAPTER6CONCLUSION52 6.1Summary52 6.2FutureWork55 6.3FinalThoughts56 REFERENCES 57 i

PAGE 5

LISTOFTABLES Table3.1Featuresusedinclassierensemble21 Table5.1Descriptionofdatasetsused32 Table5.2Framesusedfortraining34 Table5.3Framesusedfortesting34 Table5.4Parametersusedforeachdataset34 Table6.1Falsepositivesata90%truepositiverate52 ii

PAGE 6

LISTOFFIGURES Figure1.1Surveillancetrackingsystemowchart1 Figure1.2BackgroundclassicationusingonlyRGBfeatures3 Figure1.3Backgroundclassicationusingensembleclassier5 Figure3.1SetofHaarfeaturesbeingused22 Figure3.2IllustrationofequationtocomputeHaarvalue22 Figure3.3Eectsofvaryingilluminationondierentfeatures24 Figure4.1Highlevelfewofensemblealgorithm26 Figure5.1SampleframesfromOTCBVSdataset32 Figure5.2SampleframesfromPETS2001dataset33 Figure5.3SampleframesfromPETS2006dataset33 Figure5.4Exampleofthresholdingaclassierhypothesis35 Figure5.5OverallresultsontheOTCBVSdataset38 Figure5.6ResultsforeachindividualfeatureontheOTCBVSdataset38 Figure5.7FramebyframefalsepositiveresultsonOTCBVSdataset39 Figure5.8RGBclassicationimageofanOTCBVSframe39 Figure5.9EnsembleclassicationimageofanOTCBVSframe40 Figure5.10OverallresultsonthePETS2001dataset42 Figure5.11ResultsforeachindividualfeatureonthePETS2001dataset42 Figure5.12FramebyframefalsepositiveresultsonPETS2001dataset43 Figure5.13RGBclassicationimageofaPETS2001frame44 Figure5.14EnsembleclassicationimageofaPETS2001frame45 Figure5.15OverallresultsonthePETS2006dataset46 Figure5.16ResultsforeachindividualfeatureonthePETS2006dataset47 iii

PAGE 7

Figure5.17FramebyframefalsepositiveresultsonPETS2006dataset47 Figure5.18Weakhypothesesfusedintoasinglestronghypothesis49 Figure5.19Eectofobjectsizeongradientmagnitude50 Figure5.20Classicationusingensembleandgradientmagnitude51 Figure6.1Falsepositiveresultsat90%truepositiverate53 Figure6.2OTCBVSROCgraphsummary53 Figure6.3PETS2001ROCgraphsummary54 iv

PAGE 8

BackgroundSubtractionUsingEnsemblesofClassierswithanExtendedFeature Set BrendanF.Klare ABSTRACT Thelimitationsofforegroundsegmentationindicultenvironmentsusingstandardcolor spacefeaturesoftenresultinpoorperformanceduringautonomoustracking.Thisworkpresents anewapproachforclassicationofforegroundandbackgroundpixelsinimagesequencesby employinganensembleofclassiers,eachoperatingonadierentfeaturetypesuchasthethree RGBfeatures,gradientmagnitudeandorientationfeatures,andeightHaarfeatures.These thirteenfeaturesareusedinanensembleclassierwhereeachclassieroperatesonasingle imagefeature.EachclassierimplementsaMixtureofGaussians-basedunsupervisedbackground classicationalgorithm.Thenon-thresholded,classicationdecisionscoreofeachclassierare fusedtogetherbytakingtheaverageoftheiroutputsandcreatingonesinglehypothesis.The resultsofusingtheensembleclassieronthreeseparateanddistinctdatasetsarecomparedto usingonlyRGBfeaturesthroughROCgraphs.Theextendedfeaturevectoroutperformsthe RGBfeaturesonallthreedatasets,andshowsalargescaleimprovementontwoofthethree datasets.Thetwodatasetswiththegreatestimprovementsarebothoutdoordatasetswith globalilluminationchangesandtheotherhasmanylocalilluminationchanges.Whenusingthe entirefeatureset,tooperateata90%truepositiverate,theperpixel,falsealarmrateisreduced vetimesinonedatasetandsixtimesintheotherdataset. v

PAGE 9

CHAPTER1 INTRODUCTION Thereisarapidincreaseinthenumberofcamerasconstantlyrunningandmonitoringdaily life.Thisgrowthisatrendthatisnotpredictedtoslowanytimesoon.Manyofthesecameras areusedforsurveillancepurposes.Typicallytherehavebeentwoapproachestoprocessingthis data.Therstistohavehumanobserversmonitoraremotelocationinreal-timeinorderto detectunwantedevents.Theotheristorecordthevideosforforensicpurposesintheeventan unwantedeventoccurs.Increasingcomputationalcapabilitiesarenowallowingathirdusefor thisdata:automated,real-timeeventdetection. Automatedeventdetectionisabroaddescriptionofsomeofthemanyapplicationsofmodern surveillancesystems.Thesesystemsmaybeusedtoautonomouslyidentifyashoplifterinaction, locateaspecicperson,monitorhighwaytrac,buildabehaviormodelofascene,orreportany anomalouscondition.AgeneralframeworkforsurveillancetrackingsystemsisshowninFigure 1.1,wheretherstprocessesrepresentlowleveltasksandlaterprocessesrepresenthighlevel tasks.Thisworkwillfocusonthelowleveltaskofforegroundsegmentation,whichishighlighted. Foregroundsegmentationistheprocessofdecidingwhichpixelsinagivenframebelongto scene'sforegroundandwhichpixelsbelongtoascene'sbackground.Thisisaformofbinary Figure1.1.Surveillancetrackingsystemowchart 1

PAGE 10

classicationbecauseapixeleitherdoesordoesnotbelongtothebackground.Thisprocessis synonymouslyreferredtoasbackgroundclassicationandbackgroundsubtraction. Simplebackgroundsubtractionthresholdsastaticmodelimagefromacurrentframeand classiestheresultingpixelsasforeground.Sophisticatedbackgroundsubtractioninvolvesthe unsupervisedlearningofabackgroundmodelbasedonpreviousimagehistory.Thislearningcan occuratdierentrates,causingtheadaptationtooccuratdierentrates.Thelearningrateis generallyafunctionoftheframerateanddomainknowledge. Backgroundclassicationisacriticalstepinsurveillancetracking.Atrackingsystemcanonly beasreliableastheinformationitisprovided,andiftheforegroundsegmentationisperformed poorlythenthehighleveltrackerwillbeprocessingerroneousinformation. Manyissuesplaguebackgroundclassiers.Alearningratemustwalkthenelineofadapting toobjectsthatbecomeamemberofthebackgroundsuchasaparkedcar,andrecognizingaslow movingobjectsuchasacarparkedatatraclight.Theexampleofhandlingbothaparked carandawaitingcarmaybemitigatedbymaintainingalessadaptiverateandrelyingonthe highleveltrackertodistinguishbetweenthetwo.Othersituationsmaynotbeignoredaseasily. Backgroundclassicationalgorithmsaregenerallyresponsibleforhandlingdynamicbackgrounds. Dynamicbackgroundsrepresentregionsinanimagesequencethatoccasionallyundergochange, butnotasaresultofanyinterestingforegroundobject.Examplesoftheseincludeswayingtrees, ripplingwater,andotherregionsthatoccasionallyexhibitmultimodalproperties. Arguablythemostdicultdynamicbackgroundeventtoprocessisvaryingilluminations. Becausethebrightnessofapointonasurfaceisdirectlyrelatedtotheilluminationitreceives [1],achangeinilluminationwillcausetheintensityofapixeltovaryaswell.WithRGBand grayintensitylevels,thisintensitychangewillappeartobeaforegroundobjectwheninfactit isthesamebackgroundobjectonlyunderdierentillumination. Illuminationchangeshappeneitherlocallyorglobally.Aglobalilluminationchangeresults ineverypixelintheimageundergoingthesamechangeinillumination.Thisiscommonin indoorsceneswhenalightisswitched,orduringduskanddawninoutdoorscenes.Becausethe entiresceneisundergoingthechange,globalilluminationchangesareofteneasytodetect.Local illuminationchangesoccurwhenonlyaspecicregionofanimageundergoesanillumination change.Thiscommonlyoccurswithshadowsandwithdirectedlightsources,suchasaashlight. 2

PAGE 11

aSimpleFrame bSimpleFrameClassication cDicultFrame dDicultFrameClassication Figure1.2.BackgroundclassicationusingonlyRGBfeatures Localilluminationchangesaremorediculttohandlebecauseiftheregionisrecognizedas foregroundthenitislikelytobeprocessedbyahighleveltracker.Thisisbecauseitsshapecan besimilartootherrealworldobjects. Illuminationchangesmayalsobegradualorsharp.Gradualilluminationchangescanbe handledbyadaptivebackgroundmodelingalgorithmswithfewissues.Sharpilluminationchanges willgenerallycauseaclassiertofailforaperiodoftimeuntilitisabletoadapttothechange. Ifthesharpchangesoccuratahighenoughfrequencythenmultimodalmodelingalgorithmsmay beabletoovercomethisoccurrence. Figure1.2showsclassicationonaframewithlowdynamicproperties,aswellasaframe fromthesamescenethathasundergoneanilluminationchange.Afterbackgroundsubtraction, manyfalsepositivepixelsarepresentintheframewiththevaryingillumination.Whilethe performanceinthesimpleframeisacceptable,itisnotforthedicultframe. 3

PAGE 12

Dicultieswithvaryingilluminationconditionsinbackgroundclassicationisoneofthe motivatingissuesforthiswork.Mostbackgroundclassicationalgorithmseitherignorethe problem,orapplyaspecicheuristictoaparticulardomain.Itistheintentofthisworkto examinewhethertrainingclassiersondierentimagefeatureswillallowameta-classiertobe morerobusttodicultscenariossuchasilluminationchangesanddynamicbackgrounds,while stillhandlingthesimplebackgroundclassicationscenarios.Heightenedperformanceisdesired ingeneralclassicationaswell. Inthisworkanovelsolutiontobackgroundclassicationispresentedwheremultiplebackgroundclassiersareused.TheseclassiersoperateonfeaturesthatincludethestandardRGB intensities,gradientorientation,gradientmagnitude,andeightseparateHaarfeatures.Theresultsgeneratedsupportthefactthatthisisasuperiormethodtobackgroundclassicationif computationdemandsarenotconsidered.Usingtheapproachdescribedthroughouttherestof thepaperoersapromisingnewdirectioninbackgroundclassicationindicultenvironments. InFigure1.3,classicationusingthemultipleclassiersalgorithmpresentedinthispaper onthesametwoframesthatwereclassiedusingRGBfeaturesinFigure1.2isshown.Inthe dicultframethatisundergoinganilluminationchangeitisseenthatbackgroundsubtraction yieldsmuchbetterclassicationresults.Performanceinbothframesareacceptable,whichwas notthecaseusingonlyRGBfeatures. Theremainderofthethesisisorganizedasfollows:InChapter2relatedworkstothispaper arediscussed.Section2.1discussestheearlyapproachestobackgroundclassication,Section 2.2discussesrecentmethodsofbackgroundclassication,Section2.3explainsworkthatfocuses onilluminationissuesinbackgroundmodeling,andSection2.4discussesensemblemethods.In Chapter3thefeaturesthattheseparateclassiersusearedescribed.Section3.1overviews theuseofgradientfeatures,Section3.2describestheHaarfeatures,andSection3.3discusses theuseofalternatecolorspacesinbackgroundclassication.Chapter4detailsthealgorithm usedforbackgroundsubtractioninthispaper.Section4.1describesthechangesmadetothe MixtureofGaussiansalgorithm,andSection4.2discussesthemethodusedforfusingtheensemble ofclassiers.Chapter5containstheresultsfromusingtheensemblealgorithm.Section5.1 describesthemethodologyforgeneratingthecomparativeresults,Section5.2discussesthedata setsused,andinSection5.3theparameterspaceofthealgorithmisdiscussed.InSection5.4 4

PAGE 13

aSimpleFrame bSimpleFrameClassication cDicultFrame dDicultFrameClassication Figure1.3.Backgroundclassicationusingensembleclassier 5

PAGE 14

comprehensiveresultsusingROCgraphsareprovided.Chapter6concludesthiswork,where Section6.1containsasummaryofthework,Section6.2discussesfuturework,andSection6.3 providesnalthoughts. 6

PAGE 15

CHAPTER2 RELATEDWORKS 2.1EarlyWorkinBackgroundModeling Becauseofthecomputationaldemandsthatdynamicbackgroundmodelingincurs,progressin theeldhasmainlybeenoverthepastdecade,parallelingtheexplosionoffastercomputers.Prior tothecurrentrobustmodels,simplebackgroundestimationwasperformedwherethebackground wasassumedtobestatic.IntheseschemesanalgorithmsimilartotheoneinEquation2.1was used,where B p istheestimatedbackgroundpixelvalueatpixel p n isthenumberofframes usedtobuildthebackgroundmodel,and I t p isthevalueofpixel p attime t B p = n X i =0 I t p n .1 Forafutureframe t ,pixel p isthenpredictedtobeforegroundFGorbackgroundBG basedonEquation2.2,where isthethresholdtypicallysetaround50. if j I t p )]TJ/F20 10.9091 Tf 10.909 0 Td [(B p j then FG else BG .2 Thelargestadvantageofthistechniqueisthatitisfastandhasverylowmemoryrequirements becausetheimagesmaybedeletedaftertheaverageiscalculated.Themajorproblemwithsuch analgorithmisthatitdoesnotadapttoitsenvironment,anditdoesnothavetheabilityto detectdynamicbackgroundregions.Thefailuretoadaptiseliminatedifarunningaverageis takeninstead.Therunningaverageusesalearningrate where0 << 1,andupdatesthe backgroundmodelwithanewframe I new usingEquation2.3: B new p = I new p + )]TJ/F20 10.9091 Tf 10.909 0 Td [( B old p .3 7

PAGE 16

UpdatingthebackgroundimageviaEquation2.3oersasolutiontonothavinganadaptive background.Howeveramajorproblemstillexistsbecausethealgorithmstillreliesheavilyon Apoorselectionof willeitherresultinmanyfalsepositivesorfalsenegatives.Onesolution thateliminatestheusageoftheratherarbitrarythreshold isusingaGaussiandistributionto modeleachpixelinsteadofmerelythemeanorrunningaverage,asisdemonstratedin[2,3]. BecauseapixelismodeledasaGaussiandistribution,foregrounddetectionisbasedonaframe's currentpixelvalueagainstthevariance. if j I t p )]TJ/F20 10.9091 Tf 10.909 0 Td [(B p j k then FG else BG .4 InEquation2.4,now = k ,where k issomeconstant,typicallyaround2.Becausethe thresholdisdirectlyrelatedtothevariancetheequationisabletoadapttoitsenvironment. In[4]oneofthemostinuentialpapersinbackgroundmodeling,Stauer etal. rstproposed modelingthebackgroundasacombinationofmultipleGaussiandistributions.Thisalgorithm isreferredtotheMixtureofGaussiansalgorithm.Eachpixel p ismodeledwithagroupof K Gaussiandistributionsforeachoftheredandgreencolorcomponentsof p Itisassumedthat thebluecolorcomponentisignoredduetoitspoorreceptioninhumanvision,where K isa heuristicvaluegenerallysetbetweenthevaluesof3and5.Thealgorithmisrstinitialized, whereaseriesofimageframesareusedtotraineachpixelbyclusteringthepixel'sobserved trainingvaluesinto K setsusingsimpleK-meansclustering[5].Foreachset k 2 K ,themean k andvariance 2 k arecomputedtoparameterizethecorrespondingGaussiandistribution. Becausetherearemultipledistributionsforasinglepixel,eachdistribution k isinitiallyweighted suchthat w k = jj k jj jj K jj .Thereisafutureconstraintthat K X k =1 w k =1 which,holdsinitiallyaswell. Whenanewimageisprocessedforsegmentation,apixelisconsideredtomatchaparticular distributionifthepixel'svalueiswithin2.5standarddeviations,where2.5isaheuristicthat maychangebasedonaparticulardomain.Sowith K Gaussiandistributions,andapixelhistory forsomepixel p = I t x;y attime t being f X 1 :::X t g ,theprobabilityofobservingthepixel X t is: 8

PAGE 17

P X t = K X k =1 w t k X t ; k;t ; k;t .5 X t ;; = 1 n 2 j j 1 2 e )]TJ/F19 5.9776 Tf 7.782 3.258 Td [(1 2 X t )]TJ/F21 7.9701 Tf 6.587 0 Td [( t T )]TJ/F19 5.9776 Tf 5.756 0 Td [(1 X t )]TJ/F21 7.9701 Tf 6.586 0 Td [( t .6 where istheGaussiandensityfunctionshowninEquation2.6. Asincomingimagesareprocessedeachdistributionmatchedisupdatedbasedonthepixel's value,alearningrate ,andtheprobabilitythenewpixelbelongstothedistribution,asseenin Equations2.7,2.8,and2.9. t = )]TJ/F20 10.9091 Tf 10.909 0 Td [( t )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 + X t .7 2 t = )]TJ/F20 10.9091 Tf 10.909 0 Td [( 2 t )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 + X t )]TJ/F20 10.9091 Tf 10.909 0 Td [( t T X t )]TJ/F20 10.9091 Tf 10.909 0 Td [( t .8 = X t j k ; k .9 Themeanandvarianceofunmatcheddistributionsremainthesame.Theweightsofevery distributionareupdatedbasedonEquation2.10,where M t k is1ifdistribution k ismatched attime t ,and0otherwise. w t k = )]TJ/F20 10.9091 Tf 10.909 0 Td [( w t )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 k + M t k .10 Inordertodetermineifthepixelisamemberofthebackgroundorforeground,thesumof eachmatcheddistribution'sweight, ,iscalculated.Ifthisvalueisgreaterthanthethreshold T thenthepixelisclassiedasamemberofthebackground,otherwiseitisclassiedasforeground. Thevaluefor T istypicallyaround0 : 2. Thenoveltyoftheapproachin[4]atthetimeitwaspresentedgreatlyaectedthefuture progressofbackgroundmodeling.Thisisbecausebyusingamixtureofdistributions,thealgorithmcanhandledynamicbackgroundeventsaswellasgradualilluminationchanges.Previous algorithmsfailedtomakethisguarantee. TheparameterspaceusedinMixtureofGaussiansbackgroundmodelingischaracterized andevaluatedbyAtev etal. in[6].Theeectsofsetting and ,usingvariouscovariance 9

PAGE 18

representations,andvariouscolorspaceswereexploredinordertounderstandtheeectsof alteringtheparameters. In[7],Gordon etal. usedaMixtureofGaussiansalongwithdepthinformationfromastereo visionsystem.Addingthedepthcomponentdemonstratedmoreeectiveresultsinregionswith manyforegroundobjects,buttheuseofthisalgorithmiscontingentontheimplementationof astereovisionsurveillancesystem.Similarly,in[8],depthinformationgeneratedfromastereo visionsystemisusedwithaMixtureofGaussiansmodelbyHarville.InthismethodHarville et al. useYUVcolorspaceinordertomakethealgorithmmoreilluminationinvariant.Alsothe learningratesforeachpixelaredynamicinordertoallowpixelstoadaptatdierentratesbased ontheuniquecharacteristicsthatthepixelobserves. FirstproposedbyKarmann etal. in[9]andlaterbyRidder etal. in[10],usingaKalmanlter [11]tomodelthebackgroundisanotherpopularmethodofperformingbackgroundclassication. AKalmanlterisarecursiveestimatorthatmakesapredicationonafuturestateofavariable basedonpreviousstateinformationandnoiseestimation.Whenusedinbackgroundestimation eachimagepixelismodeledwithaKalmanlter.AkeyadvatageofaKalmanlterisitsability tohandleslowilluminationchanges.Becauseitrecursivelyupdatesitselfandaccountsfornoisein theestimations,slowilluminationchangesareseamlesslyincorporatedintothelter.Thefailings inaKalmanlterapproachtoFG/BGsegmentationisitsabilitytohandlesharpillumination changes.Theproblemisthatinordertohandlesharpilluminationchangesonemustincrease thegainonthelter,becauseincreasingthegainallowsforamorerapidupdate.Accordingto [10],whenthegainisincreaseditisnotpossibletostopforegroundobjectsfrombeingrapidly adaptedtobythelterandbecomingmodeledasbackground.SotheKalmanlteralonehas noabilitytodistinguishbetweensharpbackgroundchangesandforegroundobjects. Basedontheequationsusedin[9]and[10],theprocedureforusingaKalmanlterin backgroundmodelingisasfollows.Apixel p isclassiedasforegroundif j I p )]TJ/F15 10.9091 Tf 10.159 0 Td [(^ s t p j > ,where isthethresholdand^ s t istheKalmanlterpredictionattime t .^ s t isgeneratedusingEquations 2.11,2.12,and2.13.Both and arelearningrates,where < inordertoupdatethelter lesswhenaforegroundoutlierispresent. 10

PAGE 19

^ s t p =^ s t )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 + K p;t I t p )]TJ/F15 10.9091 Tf 11.345 0 Td [(^ s t )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 .11 K p;t = m t )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 p + )]TJ/F20 10.9091 Tf 10.909 0 Td [(m t )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 p .12 if I i p = FG then m i p =1else m i p =0.13 Whenwehaveasmallnumberofimagesforbackgroundmodeling,theuseofaframedifferencingalgorithmiscommon[12].Inframedierencing,thedierencebetweentwoadjacent framesisusedtodeterminethepresenceofforeground,asseeninEquation2.14.Again,theadvantageofthisalgorithmisthatitisabletooperateonalimitedamountofdata.Shortcomings lieinthefactthatthesizeofthedetectedforegroundregionswillbetoolarge,largeholesmay existinsideofobjects,anddynamicbackgrounddetectionisdicult. if j I t p )]TJ/F20 10.9091 Tf 10.909 0 Td [(I t )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 p j then FG else BG .14 Toyama etal. describetheWallowerframeworkin[13],whichperformsbackgroundsubtractioninthreeseparatephases:atthepixellevel,regionlevel,andframelevel.Thepixellevels usesaWienerltertopredictfutureintensityvalueofthepixel.Ifthisdiersbeyondathresholdthenthepixelisconsideredamemberoftheforeground.Thisapproachissimilartousing aKalmanlterforbackgroundclassication.Theregionlevelincorporatesspatialinformation fromneighboringpixelsintoeachpixel'sclassication.Theframelevelisusedtoincorporatea multi-modalpropertyintotheframework.Multiplepixelmodelsareusedtohandlethedierent modeseachpixelhasobserved.ThesemodesaredenedusingK-meansclusteringoveratraining period. 2.2ModernTechniquesinBackgroundClassication In[14],Zhang etal. buildontheworkdonebyStauerin[4],andtheworkbyHouandHan in[15].TheyusethesameK-meansclusteringofadaptiveGaussianstomodelthebackground. TheirtechniquevariesinthattheinitializationoftheGuassiansisperformedbasedonthe assumptionthatapixelsbackgroundvaluewillalwaysbemorefrequentthenapixelsforeground value.UsingthisassumptiontheGaussiansarebuiltaroundaninitializationimageset.As 11

PAGE 20

eachframeisprocessedapixel'svalueisusedtoupdateitsMixtureofGaussiansbasedonits similaritytoeachGaussian. In[16],Shimada etal. proposeanimprovementtotheMixtureofGaussiansframework, wherethenumberofdistributionsmayincreaseordecreasethroughouttracking.Thealgorithm ishighlysimilartotheoneinbyStauerandGrimsonin[4],wherethekeydierenceisthe additionofstepsthatdecideonwhetherornottoaddorremoveaGaussian.Anewdistribution isaddedifnocurrentdistributionmatchesthecurrentpixelvalue.Adistributionisremoved whenitsweightdecreasesbelowanheuristicthreshold.Alsotwodistributionswillbecombined ifthedierenceoftheirmeansarebelowacertainthreshold.Oneofthemostsignicantbenets demonstratedbythistechniqueisimprovingtheruntime.Adirectrelationwasshownbetween thecomputationtimeandthenumberofGaussians.Bymodulatingthenumberofdistributions anoptimalnumberwillalwaysbeused,ensuringaminimalamountofcomputation. AspecializedKalmanlterisusedbyGao etal. in[17]tomodelthebackground.The backgroundismodeledbasedonsmallregionsinsteadofapixelbasedmodel.Thisisbasedon theassumptionthatilluminationchangesandnoiseareidenticaltopixelswithinaregion.The parametersoftheKalmanlterarepredictedbyarecursiveleastsquareRLSlter.Theuse ofaRLSlterallowsforproperparameterizationoftheKalmanlterinvariousillumination conditions. Messelodi etal. proposeaKalmanlterthatisabletohandlesharp,globalillumination changesin[18].Thisisdetectedbyperformingaratiocomparisonbetweeneachpixeltothe backgroundmodelsvalue.Foreachpixel p ,thevalue I t p =K p where K p istheKalman predictedvalueatpixel p isplacedinahistogram.Ifthepeakofthehistogramdoesnotlie near1thenasharpilluminationchangeisconsideredtohaveoccurred.Thisapproachwas demonstratedtohavehighlysatisfactoryresultsinrecognizingthesharpilluminationchanges. Themajordrawbackofthistechniqueisthatitdoesnotsuccessfulhandledynamicbackground regions,suchasmovingleaves. In[19],WangandSuterproposeabackgroundmodelusingconsensusmethods,whichessentiallystacksseparatebackgroundmodelsthateachusealargelydisjointsetofparameters.In thepapertheseparametersaretheseparatecolorcomponentsinRGB,thoughanyotherpixel propertycouldbeusedinstead.Athreephaseprocedureisusedtogeneratethebackground 12

PAGE 21

model.Initially,for K frames,anadjacentframedierencealgorithmisusedtosegmentthe foregroundandbackgroundpixels.InthenextphaseaframeworkcalledSACONSampleConsensusisusedtodetermineifanypixelsthatwereinitiallyconsideredforegroundactuallybelong tothebackground.TheinputtoSACONisthe K images,atimeoutmapTOMwhichcontains consecutiveforegrounddetectionamountsforeachpixel,andthelocationsofpixelslabeledas foreground.UsingthisdataSACONoutputsareducedamountofforegroundpixels.Inthe nalphaseholesinsideforegroundregionsareinvestigated.Thesolemethodofupdatingthe backgroundmodelafterthesethreephasesisviatheTOM.WhileaTOMworkswellfornewly insertedbackgroundobjects,ithasbeenproventobepoortoadjusttogradualchanges.This backgroundmodelalsodoesnotdevelopastatisticaldistributionfortheperceivedbackground, insteadtheonlydatamaintainedregardingthebackgroundisdatafromtheinitialsetofimages. In[20],AvidanperformsFG/BGsegmentationusinganensembleofbackgroundclassiers. Thisworkhasastrongresemblancetotheworkproposedforthispaperinthefactthatit seekstocombinetheoutputofmultipleclassiersintoonesinglehypothesis.Theensemble ofclassiersstrictlyusesweakclassiers,andcombinestheoutputsusingtheAdaBoost[21] ensemblealgorithm.WhilewewillrefertoourtaskofFG/BGsegmentationasbackground classication,in[20]theterm classication isusedinafarmoreliteralsense.Thisisbecause classiersareexplicitlytrainedonlabeleddata.Thislabeleddataisintheformoftheinitial locationofaforegroundobject,whichmustbemanuallypassedin.Usingthisinitialobjectand aseriesofimages,thealgorithmgenerates k weakclassiers.Atthenextframe i eachclassier isusedtogenerateacondencemapofwheretheforegroundobjectislocated.Togetherwitha meanshiftalgorithm,thelocationoftheforegroundobjectinframe i isusedtoretrainanew setof k classierswhichwillbeusedinframe i +1.Thisprocessrepeatsitselfforallimages. ThetechniqueusedbyAvidanin[20]istheonlyworkfoundthatexplicitlyusesanensemble ofbackgroundalgorithmsforFG/BGsegmentation.Itisafarmorelimitedapproachtotheone proposedinthispaperbecauseitisreliantonamanualinitializationofaforegroundobject,and itonlyconsidersforegroundandbackgrounddistinctionbasedonthisforegroundobject.This meansonlyoneobjectmayberecognizedasforegroundatatime.Thisapproachhasserious advantageswhenattemptingtotrackaspecicforegroundtarget,thoughitlimitedtothistype oftracking. 13

PAGE 22

Segmentingimagesthroughtheconceptofimagelayersisapopularapproachtobinary segmentation,wherethelayersrepresentdierentplanesorobjectgroupsinanimageTypically manylayerswillrepresentboththeforegroundandbackground.Aforegroundlayerisgenerally consideredalayerthathasspatiotemporalchange. Criminisi etal. performbackgroundclassicationusingimagelayeringin[22].Aprobabilistic approachisusedwherethemotion,color,andcontrastarecombinedwiththespatialandtemporal priors.Thisuseofmotioncuesisgenerallyalimitingfeatureduetothecomputationaldemands, howeverthealgorithmpresentedisabletoruninreal-timebecausetheactualpixelvelocitiesare nevercomputed.Insteadabinarymotionvalueisassignedtorepresentthepresenceorabsence ofmotion. In[23],Patwardhan etal. presentanautomatedmethodofpixellayering.Thealgorithmhasa longinitializationstepfollowedbyasimplerdetectionprocess. T framesareusedininitialization. Therstframeissegmentedusingthefollowingprocedure.Themaximumofthehistogramof thegraylevelpixelintensities, h max isfound,andallpixelswhosegraylevelsarewithin h max areaddedtotheinitiallayer,where isderivedfromtheimagecovariancematrix.Asampling oftheabout10-20%oftheimageisusedtoperformaKernelDensityEstimationKDE,which generatesaprobabilitydensityfunctionfortheinitiallayerbasedonapixel'sfeaturevectorin thiscaseRGBfeaturesaremappedtothespacepresentedin[24],whichisdiscussedfurtherin Section2.3..BasedontheKDEprobability,theimagepixelsarereassignedtothelayer,which iscalledtherenementstep.ThesamplingandKDEcontinuesuntiltheinitiallayerstabilizes. Thepreviouslyextractedlayerissuppressedfromtheinitialhistogramandthisprocessoflayer extractioncontinuesuntilaKullback-Leiblerdivergencestatesthemostrecentextractedlayer wasnotmeaningful.Therestoftheimagesareprocessedusingthepreviouslayersandstarting attherenementstep. Oncetheimagesetislayeredfromtheinitialization,futureframesareabletobeprocessed. Inordertoincorporatespatialinformation,a w w M windowisusedforeachpixel,where M isthetheamountofpreviousframesbeingconsidered,and w istheamountofpixelsinthe x and y directionconsideredforspatialinformation.Foreachlayerthatexistsinthiswindow,the KDEisusedtogeneratetheprobabilitythatpixel p belongstothelayer,aswellasthelayerof 14

PAGE 23

outlierswhichcomprisestheforeground.Dependingonthelayerclassication,thepixelisthen classiedasforegroundorbackground. Opticow[25]hasbeenusedforimagesegmentationaswell.In[26],Zucchelli etal. cluster motioneldsinordertorecognizeimageplanes.Whiletheseplanesarenotappliedtoforeground segmentation,thestepishighlyintuitivegiventheplanesrecovered.Theuseofmotioninimage segmentationhasbeengainingtractionrecentlybecauseofnewhardwareimplementationsthat areabletocalculateowinreal-time.Thisrepresentsanexcitingavenueforfutureresearch usingthisadditionalfeature. In[24],motionisuseddirectlyforbackgroundsubtraction.KernelDensityEstimation[27]is usedtomodelthedistributionofthebackgroundusingvedimensionalfeaturevectors.Based onthisdistributionclassicationisperformedbasedonthresholdedprobabilitiesofinstances belongingtothebackgrounddistribution.Thefeaturevectorsconsistofthreedimensionsfrom colorspaceintensities,andtwofromopticowmeasurementswhichconsistsoftheowmeasurementsandtheiruncertainties.Thisalgorithmperformedextremelywellwhencomparedto MixtureofGaussianmodels. 2.3IlluminationConsiderations Manyapproacheshavebeenusedforilluminationinvariantbackgroundclassication.One reasonforpoorFG/BGsegmentationinvaryingilluminationisbecausemosttrackingsystems relyoncolorfromRGBcolorspace,whichishighlyvarianttoilluminationchanges.Because grayscalecolorspaceisalineartransformationfromRGBthesameproblemexists.Acommon solutionisthenon-linearmappingofRGBcolortoanothercolorspacethatislessillumination invariant. Manycolorconversionsthatclaimtobeilluminationinvarianthavebeenproposed.Theoritically,thehuecomponentofHSIcolorspaceandthelumacommponentYofYCbCrcolor spaceareilluminationinvariant,howeverinpracticethisisnottypicallythecaseseeSection 3.3.In[24]MittalandParagiosuseacolormappingof RGB rgI where I = R + G + B = 3, r =3 R=I ,and g =3 G=I duetoclaimedilluminationinvarianceundercertainconditions.Experimentationperformedinthisworkusingthiscolorspacefailedtoobservesuchillumination invariantconditions,thoughthisisnottosaytheydonotexist.In[28],GeversandSmeulders 15

PAGE 24

presentthecolorspaceinEquation2.15asanotherilluminationinvariantcolorspace.Afurther discussionoftheobservedfailuresusingthesealternatecolorspacescanbefoundinSection3.3. l 1 = R )]TJ/F20 10.9091 Tf 10.909 0 Td [(G 2 R )]TJ/F20 10.9091 Tf 10.909 0 Td [(G 2 + R )]TJ/F20 10.9091 Tf 10.909 0 Td [(B 2 + G )]TJ/F20 10.9091 Tf 10.91 0 Td [(B 2 l 2 = R )]TJ/F20 10.9091 Tf 10.909 0 Td [(B 2 R )]TJ/F20 10.9091 Tf 10.909 0 Td [(G 2 + R )]TJ/F20 10.9091 Tf 10.909 0 Td [(B 2 + G )]TJ/F20 10.9091 Tf 10.91 0 Td [(B 2 .15 l 3 = G )]TJ/F20 10.9091 Tf 10.909 0 Td [(B 2 R )]TJ/F20 10.9091 Tf 10.909 0 Td [(G 2 + R )]TJ/F20 10.9091 Tf 10.909 0 Td [(B 2 + G )]TJ/F20 10.9091 Tf 10.91 0 Td [(B 2 Illuminationchangesoftencauseerrorsinbackgroundsegmentationbecausewhenilluminationchangesaresharpthepixelintesityvaluewillvaryconsiderably.Whenthesechangesare notglobalandisolatedintime,carefulcolorspaceanalysisisoftenusedtopreventtheseilluminationchangesbeingimproperlyclassied.In[29],Horprasert etal. performsegmentationinto fourseparatepixelclasses:normalbackground,shadedbackground,highlightedbackground,and foreground.Thisisaccomplishedbystatisticallymodelingeachpixelbasedonitschromacity Equation2.18andbrightness Equation2.17,where,overanobservedinitializationperiod, R p isthemeanvaluefortheredcolorchannelatpixel p G p isthevarianceovertheperiod forthegreencolorchannel,and I B p isthecurrentvaluepixel p p =argmax I R p )]TJ/F21 7.9701 Tf 6.587 0 Td [( p R p R p 2 + I G p )]TJ/F21 7.9701 Tf 6.586 0 Td [( p G p G p 2 + I B p )]TJ/F21 7.9701 Tf 6.586 0 Td [( p B p B p 2 .16 = I R p R p 2 R p + I G p G p 2 G p + I B p B p 2 B p h R p R p i + h G p G p i + h B p B p i .17 p = r I R p )]TJ/F21 7.9701 Tf 6.587 0 Td [( p R p R p 2 + I G p )]TJ/F21 7.9701 Tf 6.586 0 Td [( p G p G p 2 + I B p )]TJ/F21 7.9701 Tf 6.587 0 Td [( p B p B p 2 .18 Thismodelisbestunderstoodbyconsideringpixelcolorvaluesasavectorinathreedimensionalspace,wherethered,greenandbluecolorcomponentsarethedierentdimensions.For anyintensityvalue I p ,changingonlythebrightnessofthatcolorwillresultinanewintensity I 0 p ,where I 0 p = I p .Inotherwords,thenewvector I 0 p isthesameunderlyingcolor as I p ,onlyitsintensityhaschangedbasedontheradianceofthepixelundertheillumination condition.Ifthechromacitycolorofthepixelhaschangedto I 00 p ,however,then represents 16

PAGE 25

theshortestEuclideandistancefromthevalue I 00 p tothelinethatrepresentsthevector I p Thismeansthechromacitydistanceisbasedonlythecolorofthepixelandnottheillumination. Whenanewpixelisprocessedbythealgorithmfrom[29],thedistance andscale are computedbasedontheexpectedvalueofthepixelfromtheinitializationperiod.When surpassesathresholdthepixelisclassiedasforegroundbasedonthefactthatthecolorhas signicantlychanged.Otherwisethepixelislabeledasnormalbackground,shadedbackground orilluminatedbackgroundbasedon .Thisalgorithmdoesnotupdatetheexpectedvalues generatedoverthetrainingperiod.Thischangecouldeasilybeappliedwherethemeanand varianceofthepixelsareupdatedusinglearningratessimilartothemethodsin[4].Themethod presentedalsoperformsanautomaticthresholdselectionbasedonhistogramanalysis,making themethodnon-parametric. In[30],XuandEllisuseMixtureofGaussianbackgroundmodelingbymappingtheRGBcolor componentsintoamoreilluminationinvariantcolorspace.DenotingaRGBpixelas p RGB =

,amappingisperformedsuchthat p RGB p rgb ,where p rgb =

and p i = p I = q p 2 R + p 2 G + p 2 B for i f r;g;b g and I f R;G;B g .Usingthiscolorspaceresultedina claimedhigherperformancewhenclassifyingbackgroundpixelsinimagesequenceswithvaried illumination. Onepromisingtechniqueforbackgroundmodelingwithilluminationinvarianceisusingimage gradientinformationasfeaturesforbackgroundclassication.Oneoftheearliestapplications ofusinggradientfeaturesforbackgroundsubtractionwasbyJabri etal. in[31].Inthiswork resultsareshownusingtrackingonindoorimagesequences.Thebackgroundissimplymodeled usingthemeanandvariancethefeaturessetRGBandgradientmagnitude. In[32],Javed etal. useavedimensionalfeaturesetcontainingRGB,gradientmagnitudeand gradientdirectionforbackgroundsubtraction.TheRGBfeaturesareprocessedusingaMixture ofGaussiansalgorithm,andthehypothesisisthenaugmentedwithinformationtakenfromthe imagegradients.Resultsareshowntoimproveinanindoorsequencewithvariedillumination. 17

PAGE 26

2.4EnsembleMethods Ensemblealgorithmsusetheresultsofseveralseparatealgorithmsinconjunction.Originally proposedbyDasarathyandSheelain[33],theconceptofensemblesystemsisthatbyusing multiplemodelsinsteadofonethereisagreaterchanceinamorerobustmodel. Theearliestpublisheduseofensemblesystemsinclassicationandmachinelearningwasby DasarathyandSheelain[33],wherethemeritofusingacompositeofclassierswasdiscussed. Becausethesearchspaceofaparticularproblemwastoolarge,itwaspartitionedsothatseparate classierscouldoperateoneachspace.Theresultsofeachclassierwasthenmergedtogetherto createasingleresult. In[34],Schapireprovesthatweaklearnersareequivalenttostronglearnersbyintroducing boosting.Weaklearnersareconsideredalgorithmsthatpredictbetter,butonlyslightlybetter, thanrandomguessing.Stronglearnersareconsideredpolynomialtimealgorithmsthatachieve lowpredictionerroronallavailableclasses.Thispaperrepresentedsomeoftheearliestresearch onensemblelearningandprovedthatanensembleofweaklearnerscanbejustapowerfulasa stronglearner. Ingeneral,therearetwoseparatetypesofensemblelearners:onesthatcombinemodelsfrom thesamesetofdata,suchasbootstrapaggregationbagging[35],boosting[34]orAdaBoost [21],andthosethatbuildseparateclassiersusingeitherdierentalgorithmsordisjointfeature sets.Thisworkwillbeconcernedwiththelaterofthetwobecausethegoalistooverlapthe biasesofeachfeature.TherstuseofcombininguniqueclassierswasbyWolpertin[36],and wasreferredtoasStackedGeneralization,thoughnowcommonlyreferredtoasstacking.Thefull implementationofastackingalgorithmsinvolvesmanyseparatetrainingperiodsusinglabeled trainingdataandcross-validation.Thisentireprocedurewouldnotbereasonablyappliedto thecombinationofbackgroundclassiers,butthegeneralconceptishighlyapplicable.Another ensemblemethodforcombininguniquealgorithmsiscalledamixtureofexperts[37].Amixture ofexpertsissimilartostackingexcepttheoutputsofeachclassierarethenusedasinputsto anothersophisticatedalgorithm,suchasaneuralnetwork. Anensembleclassierconsistsoftwokeysteps:generatingtheoutputfrommanyseparate classiers,andcombiningthatoutputintoonesingleoutput.Wementionedthatthemixtureof expertsalgorithmoftenusesneuralnetworkstocombineclassierresults.Therearemanymore 18

PAGE 27

methodscommonlyused,mostofwhichareconsiderablylesssophisticated.Onepopularmethod iscalledmajorityvoting[38].Inmajorityvotingeachclassierisgivenonevotetowardsthenal outputoftheensemble,andtheoutputtypicallywillbethemajorityofallvotes.Majorityvoting issimpleandintuitive.Itsmainlimitationisthatitisnotabletoovercometheoutputofpossible noisyclassierswithintheensemble.Thislimitationisovercomewithaweightedmajorityvoting [38],wherethenameadequatelyimpliestotechnique.Byaddingweightstoindividualclassiers basedontheirperformance,poorlyperformingclassierscanberestrictedtoalimitedinput. Thistechniquerequiresfeedbackfortheperformanceoftheensemble'salgorithms,whichisoften generatedusingvalidationdata.AfutheranaylsisofclassierfusionwillbeprovidedinSection 4.2. In[39],SiebelandMaybankusedafusionmethodinobjecttracking,howevertheframe workedpresentedwastargetedtowardsunderstandinghighersemanticsinimagesequencesthan justbackgroundclassication.Instead,hypothesesweregeneratedforthepresenceofpeopleusing shapedetection,previouslydetectedregions,andforegroundinformation.Forthisreasonthere workdierssignicantlyfromtheworkinthisproject.Theirworkusesaxed,andcomplex, fusionstrategythatisnotconsideredapplicabletoourtask. In[40],Kittler etal. combinedierentclassiersforclassicationwithmultipleclasses. In[41],Kunchevaevaluatesbinaryclassicationusingclassierfusion.Thesepaperscompare resultsofthefusionrulessuchasminrule,maxrule,averagerule,productrule,medianrule,and majorityvoting.In[42],Ho etal. usemultipleclassierstorankthelikelihoodforeachclass. Theserankingsarethenreducedandre-rankedbyafusionfunctionthatgeneratesanalclass. Ifbinaryclassicationwereperformedthenthemethodissimilartomajorityvoting. In[43],RossandJaindiscussmanydierentfusiontechniquesforbiometricclassiers.Becausebiometricclassicationtypicallyinvolvesavastnumberofclasses,asinglenoisyfeature measurementoraclassierbiasmaycauseincorrectresults.Methodsoffusingbothmultiple featuressetsandmultipleclassierdecisionsarepresented.Themostsuccesswasfoundusing thesum/averagerule,whichagreeswiththeresultsin[40]. 19

PAGE 28

CHAPTER3 FEATURES Standardbitmapsstoreathreedimensionalvectoroffeaturesperpixel:red,green,andblue intensityvalues.Inthisworkthenumberoffeaturesperpixelswillbeincreasedthroughnonlineartransformationsinordertoprovideseparatefeaturebiasesofthescenesbeingobserved. AsummaryoftheimagefeaturesthatwillbeusedinthisworkcanbefoundinTable3.1. Usingthesefeaturesgivesatotalof13featuresperpixel.Eachofthesefeatureswillbeusedby aseparateclassierthatonlyhasknowledgeofitsrespectivefeature. 3.1GradientFeatures Gradientfeaturesareusedtodetectedgesandpeaksoverintensitychangesinimages.Forthe backgroundclassiers,eachpixelintheimageframewillbecharacterizedbythenon-thresholded valuesfromthemagnitudeandorientationvaluesoftheCannyedgedetector[44].Usingthese twofeatureswilloeradvantagesanddisadvantagesnotfoundusingonlyRGBfeatures.Amajor advantageisfoundundervaryingillumination.InFigures3.3gand3.3h,acomparisonofthe gradientmagnitudeofthesamescenewithdierentilluminationconditionsshowsthatgradient magnituderemainslargelyinvarianttotheilluminationchange.Thispropertywillallowour classiertobemorerobusttovaryingilluminationconditions. Onedisadvantageofusingthegradientmagnitudeisthatforegroundobjectswithhomogeneousintensitieswillnotappeartochangefortheclassierintheinnerareasoftheobject.This factleadstoaimportantpointaboutthefeaturesbeingused.Individually,thesefeaturesdonot oerasignicantenoughrepresentationofthesceneforaclassiertomakeaccuratepredictions. Instead,thecombinationeachoftheselessertrackingfeaturesareassumedtoprovideahigher representationoftheimagespacewhenusedinanensemble. 20

PAGE 29

Table3.1.Featuresusedinclassierensemble Feature Source Spatial MinValue ab MaxValue RedIntensity CCD/Bitmap No 0 255 GreenIntensity CCD/Bitmap No 0 255 BlueIntensity CCD/Bitmap No 0 255 GradientMagnitude CannyEdgeDetector Yes 0 255 GradientOrientation CannyEdgeDetector Yes 0 HaarFeature1 HaarWavelet Yes )]TJ/F21 7.9701 Tf 9.681 4.295 Td [(w 2 h 255 w 2 h 255 HaarFeature2 HaarWavelet Yes )]TJ/F20 10.9091 Tf 8.485 0 Td [(w h 2 255 w h 2 255 HaarFeature3 HaarWavelet Yes )]TJ/F21 7.9701 Tf 9.681 4.295 Td [(w 3 h 255 2 w 3 h 255 HaarFeature4 HaarWavelet Yes )]TJ/F21 7.9701 Tf 9.681 4.295 Td [(w 2 h 255 w 2 h 255 HaarFeature5 HaarWavelet Yes )]TJ/F20 10.9091 Tf 8.485 0 Td [(w h 3 255 w 2 h 3 255 HaarFeature6 HaarWavelet Yes )]TJ/F20 10.9091 Tf 8.485 0 Td [(w h 2 255 w h 2 255 HaarFeature7 HaarWavelet Yes )]TJ/F21 7.9701 Tf 9.681 4.296 Td [(w 3 h 3 255 8 w 3 h 3 255 HaarFeature8 HaarWavelet Yes )]TJ/F15 10.9091 Tf 8.485 0 Td [(2 w 2 h 2 255 2 w 2 h 2 255 a w isthewidthoftheHaarwindow b h istheheightoftheHaarwindow. 3.2HaarFeatures Haarwavletsprovideaneasilycomputablesetoffeaturesthatrepresentthedierencebetween imageintensitiesoveraregion.TheemergenceofHaarfeaturesinpatternrecognitionbegan withPapageorgiou etal. in[45],[46].OneofthemostprominentworkstouseHaarfeatures wasbyViolaandJonesin[47],inwhichaframeworkforreal-timefaceandobjectdetectionis demonstrated.Thisremainsthestandardincurrentfacerecognitiontechnology. Haarfeaturesareparticularlydesirablebecausetheyarefasttocompute.Thekeycostin usingHaarfeaturesisgeneratingtheintegralimage.Thisrequiresanentirepassoftheimage usingthealgorithminEquation3.1,whereIN p x;y isthevalueoftheintegralimageatpixel p x;y .Oncetheintegralimageiscomputed,thevalueofaHaarrectanglemaybecomputedusing onlyfourreferencestotheintegralimage. IN p x;y = X x 0 p x ;y 0 p y I p x 0 ;y 0 .1 The8Haarfeaturesusedintheworkarederivedfrom[48]and[45].TheseHaarfeaturesare showninFigure3.1.WhencomputingtheHaarfeatures,theblackregionsrepresentnegative values,andthewhiteregionsrepresentpositiveregions.Theintegralimageiseasilycomputed usingtheequationfrom[48],whichisshowninEquation3.2.Oncetheimageiscomputed, 21

PAGE 30

Figure3.1.SetofHaarfeaturesbeingused Figure3.2.IllustrationofequationtocomputeHaarvalue thevalueofaboxiscomputedusingEquation3.3,whichisillustratedinFigure3.2.Inthe illustration,thepointsrepresentdierentvaluesoftheintegralimage.AHaarBoxisanyofthe blackandwhiteregionswithintheHaarfeaturesshowninFigure3.1. IN p x;y =IN p x;y )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 +IN p x )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 ;y )]TJ/F15 10.9091 Tf 10.909 0 Td [(IN p x )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 ;y )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 )]TJ/F20 10.9091 Tf 10.91 0 Td [(I p x;y .2 HaarBox=IN x;y +IN x +width ;y +height )]TJ/F15 10.9091 Tf 10.909 0 Td [(IN x;y +height+IN x +width ;y .3 AnexampleofhowthevalueforaparticularHaarfeatureiscomputedwillnowbeshown. SupposethatHaarFeature1istobecomputedatpoint x =20 ;y =20,usinga12x12sized window.Firstthevaluesoftheboxesthatdenethefeaturearecomputed.AsseeninFigure 3.1,oneboxcoversthelefthalfofthewindowandtheotherboxcoverstherighthalf.Forbox onthelefthalftheboundariesoftheboxare:left=14,right=20,top=14,bottom=26.For theboxontherighthalftheboundariesoftheboxare:left=20,right=26,top=14,bottom =26.UsingtheintegralimagefromEquation3.2,thevaluefortheleftbox V l iscomputed usingEquation3.3: V l =IN ; 26+IN ; 14 )]TJ/F15 10.9091 Tf 10.478 0 Td [(IN ; 26 )]TJ/F15 10.9091 Tf 10.478 0 Td [(IN ; 14.Thevalueoftheright 22

PAGE 31

box V r is: V r =IN ; 26+IN ; 14 )]TJ/F15 10.9091 Tf 10.901 0 Td [(IN ; 26 )]TJ/F15 10.9091 Tf 10.901 0 Td [(IN ; 14.SothevalueofHaarFeature 1atpoint x =20 ;y =20willbe V l )]TJ/F20 10.9091 Tf 10.909 0 Td [(V r 3.3AlternateColorSpaces Itiscommonintrackingalgorithmsforalternatecolorspacestobeused,generallydueto theirclaimedilluminationinvariance.Thisworkdoesnotplantousealternatecolorspaces, suchasthosementionedinSection2.3,primarilybecauseilluminationinvariancehasnotbeen observedusingalternatecolorspaces. In[28],anevaluationofvariouscolorspacesisperformedtodeterminewhichareinvariant toilluminationchanges.Inthispaperitisclaimedthathueisinvarianttoilluminationchanges, howevertheimagesinFigures3.3cand3.3dshowthatthisisnotnecessarilythecase.Because hueisrepresentedasanangle : 2 [0 ; 360,eachpixel p inthetwohueimagesinFigures3.3c and3.3dmaps to p accordingtoEquation3.4.Ofcourse,inthisrepresentationvaluesfor =0and =355willhavetwohighlydierentpixelvalues,whenthecircumferaldistance isverysmall.However,forthepurposesofthisvisualizationthatinaccuracydoesnomatter becausethechangeinhuefromtheilluminationchangeisclearlynotbecauseofthis.Figures 3.3eand3.3fshowthattheluminancecomponentofYCbCralsoexhibitssignicantchanges undervaryingillumination. p =255 360 .4 23

PAGE 32

aOriginalImage,noshadow bOriginalImage,shadow cHueImage,noshadow dHueImage,shadow eLumafromYCbCr,noshadow fLumafromYCbCr,shadow gGradientMagnitude,noshadow hGradientMagnitude,shadow Figure3.3.Eectsofvaryingilluminationondierentfeatures 24

PAGE 33

CHAPTER4 MULTIPLECLASSIFIERALGORITHMS InthisworkaseriesofMixtureofGaussianclassiersareusedforeachpixel.Eachclassier willprocessonedatafeatureandgenerateahypothesisfortheframebasedonthatfeatures value.Thehypothesesfromeachclassierarethenfusedintoasinglehypothesisforthatpixel. AdiagramofthisalgorithmmaybeseeninFigure4.1. InthisworksomechangestotheoriginalMixtureofGaussiansareproposed,whichare discussedinSection4.1.InSection4.2adescriptionofthepossibleclassierfusiontechniques ispresented.Usingthisframeworkwillyieldanunsupervisedensemblelearningalgorithmfor classicationofforegroundpixelsinimagesequences. 4.1ChangestoMixtureofGaussiansAlgorithm ThedetailsoftheMixtureofGaussiansalgorithmwerediscussedinChapter2.Theclassiers usedinthisworkadheretoStauer'soriginalMixtureofGaussiansalgorithm,withtheexception ofafewchanges.Thesechangesareproposedtooermoregeneralitytothealgorithmaswell asfastertrainingtimes. 4.1.1VariableNumberofDistributions TherstchangepresentedishavingavariablenumberofGaussiandistributionsperpixel, whichissimilartothemethodusedin[16].In[4]StauerandGrimsonusedaxednumber ofGaussiansperpixel.Thisnumber, K ,isgenerallybetween3and5.Whenanewinstance occursthatisnotmatchedbyanyGaussian,thedistributionwiththelowestweightwasreplaced byanewdistributioncenteredonthenewinstance.Thisapproachgenerallyworksbecauseas longas K isgreaterthanorequaltothenumberofdistributionseachpixelactuallyobserves overaperiodoftimetherewillbeenoughdistributionstomatchtheprocesses.Ifthenumber 25

PAGE 34

Figure4.1.Highlevelfewofensemblealgorithm 26

PAGE 35

ofdistributionsobservedexceeds K thenabehaviorsimilartotrashinginoperatingsystemswill occur. Inthisworkeachpixelbeginswithonedistribution.Whenannewvalueoccursthatdoes notamatchadistribution,anewGaussianiscreatedandaddedtothatpixel.Periodicallythe weightofthedistributionsateachpixelarechecked,anddistributionsareremovediftheirweight islessthanthelearningrate Usingavariablenumberofdistributionsperpixelallowseachpixeltocharacterizethemultimodalityofitsobservedregionbasedonobservationsandnotapredenedheuristicvalue.The computationaloverheadisminimalwheneachpixelstoresitsdistributionsinalinkedlist. 4.1.2Training InorderinitializetheGaussiandistributionsusedintheMixtureofGaussiansalgorithm aseriesoftrainingframesareusedtoprovideaninitialcharacterizationofthescene.This generallyrequiresclusteringtheobservedvaluesfromeachclusterinto K distributions.Unlessa convergenceofclustersoccursrapidly,clusteringiscomputationallyexpensive-whichmeansthe modelmaynotbetrainedinreal-time.Afurtherproblemtothestandardinitializationisthat ifapixel'sobservedtrainingvaluesarefromaregionthatremainedstaticthroughthetraining framesi.e.onlyoneclusteractuallyexists,thatpixel'svalueswillstillbepartitionedinto K clusters. ThetrainingmethodfortheMixtureofGaussiansalgorithminthisworkisamoresimplied approachthatisenabledbythefactthatthenumberofGaussiandistributionsusedisvariable. Usingasmallernumberoftrainingframeslessthan25,themeanofeachofthesepixelsare calculatedandasingledistributioniscreatedforthatpixelusingthemeancalculatedandaxed variance,typicallyaround10 2 axedvarianceisusedbecauseofregionsthatmayhavechanges intheirtrainingdata,causinganabnormallyhighvariance.Forstaticregionsthisapproach ismorerobustintermsofinitialaccuracyandspeed.Forregionsthatobservechangeoverthe trainingframesthespeedofthisapproachoutweighstheinitiallackofaccuracy. OnereasonthisapproachworksisthatMixtureofGaussiansisanadaptivealgorithm.Initiallysettingonlyonedistributionforapixeldoesnothavealongtermimpactbecauseasother modespresentthemselvestothatpixelnewdistributionswillbeadded.Becausethemeanand 27

PAGE 36

varianceareabletoadapt,aninitialmeanandvariancethatareinaccuratewillbecomecorrected afterashortperiodofframes. 4.2ClassierFusion Aseachframeisprocessed,andeachclassiermakesitspredicationastowhethereachpixel isamemberofthebackgroundorforeground,thosepredicationsmustbemergedintoasingle prediction.Becausethisalgorithmisanunsupervisedlearner,i.e.thereisnolabeledtraining dataoruserfeedback,traditionalensemblealgorithmssuchasbaggingandboostingarenot possible.Insteadclassierfusionapproacheswillbeused. Threemainfusiontechniqueswillbeconsideredinthiswork,bothtakenfrom[40],[41].These arethemaxrule,theaveragerule,andthemajorityvoterule.In[41]itwasshownthatforbinary classicationthemaxruleclassiercombinationisthesameastheminrule,andthatmajority voteisthesameasmedianrule,whichiswhytheyarenotbeingdirectlyconsidered. If K classiersexist,thenbecause P FG =1 )]TJ/F20 10.9091 Tf 9.343 0 Td [(P BG forthetwoclassesthemetadecision rulesarefoundinEquation4.1forthemaxrule,Equation4.2fortheaveragerule,andEquation 4.3forthemajorityvoiterule.Intheequations, C i isthe i thclassierintheensemble,and P BG j C i isthebackgroundprobabilitypredictedbyclassier i .InEquation4.3,theoutputof F i x is0if P x j C i <: 5,and1if P x j C i > = : 5,where x 2 BG,FG. argmax x 2f BG,FG g argmax i P x j C i .1 argmax x 2f BG,FG g 1 K K X i =0 P x j C i # .2 argmax x 2f BG,FG g K X i =0 F i x # .3 Itisimportanttonotethattheuseofmaxrulecombinationandmedianrulecombinationare possiblebecausetheclassiersbeingusedareabletogenerateprobabilitiesofclassmembership. Thisisbecausetheirdecisionsarebothbasedonthresholdingfunctionaloutputswhichrepresent adegreeofmembership.Contrarily,ifaclassieronlygeneratedabinarydecisionthenmajority votewouldberequired. 28

PAGE 37

Inboth[40]and[41],theaveragerulewasshowntobethemosteectivefusiontechnique. Basicobservationsyieldnokeydierencesbetweenthethreewhenfusingtheclassiersinthis work.Becauseofthistheaveragerulewasusedforclassierfusioninthiswork. 29

PAGE 38

CHAPTER5 RESULTS 5.1Methodology Inordertoevaluatetheresultsoftheproposedmeta-classier,acomprehensivemethodof comparingtheoutputsfromtheclassiersmustbeused.Eachclassieroutputsaperpixel hypothesisindicatingwhetherthatpixelisamemberoftheforegroundorsomebackground process.Ahighleveltrackingsystemwouldreceivethesepixels,clusterthemintoblobs,and processthemforahigherlevelsemantic. Classiersinthisworkwillbeevaluatedbasedontheirratiooftruepositivetofalsepositive classication.UsingthesemeasurementsallowsforareceiveroperatingcharacteristicsROC graph.ROCgraphsarehighlyexpressive,andprovideavisualizablecomparisonofmultiple classiers'performances.ROCgraphsaremoreidealwithclassiersthatgenerateaclassprobabilityordegreeofmembership[49],whichtheMixtureofGaussiansdoesbyvaryingthe thresholdparameter.AnotherbenetofusingROCgraphsisthattheresultsareinvariantto thedistributionsofclasses[49],whichisnotthecaseusingmetricssuchasprecisionandrecall [50]. Inonemethodofevaluatingtheclassiertruepositiveresultswouldbebasedonitsperpixel accuracy,howeversomeproblemsexistusingthismethod.Onemajorproblemwithaperpixel evaluationcriteriaisthatthetruepositiveresultsmaybemisleading.Consider,forexample,a scenewithtwoforegroundobjects x and y thateachoccupythesamenumberofimagepixels.If ClassierAcorrectlyclassiesallthepixelsofobject x asforegroundandincorrectlyclassiesall thepixelsofobject y asbackground,thenitwillhave50%truepositiverateforthatframe.If ClassierBcorrectlyclassieshalfofthepixelsinbothobjectsthenittoowillhavea50%true positiverate.However,astrongargumentcouldbemadethatClassierBismoresuccessful. Thisisduetothefactthathalfofanobject'spixelsbeingclassiedasforegroundshouldbe enoughforthehigherleveltrackingalgorithmtorecognizebothoftheseobjects.Theoutput 30

PAGE 39

fromClassierAwillcertainlynotbesucientinrecognizingbothobjects.Inadditiontothis drawback,generatingperpixelgroundtruthinformationishighlytediousandpronetoerror. Anothermethodofgeneratingtruepositiveresultsisbasedonthenumberofobjectscorrectly found,whichisthemethodthatwillbeusedinthiswork.Usingthismethodthegroundtruth willconsistofboundingboxlocationsoftheforegroundobjects.Becausetheboundingboxfor eachforegroundobjectwillalsocontainbackgroundpixels,whetherornotanobjectisfoundwill bebasedonthepercentageofpixelsclassiedasforegroundwithinthebox. Therearefewchoicesforrepresentationsoffalsepositives.Thesepixelscouldbegroupedinto objectsandthenumberoffalseobjectsfoundwouldthenbereported,howeverthismetricfails toincorporatethesizeorshapeoftheobjectsfound.Instead,asimplecountofthenumberof falsepositivepixelswillbeused.Ofcourse,thiswillonlyincludepixelsoutsideofthebounding box,groundtruthregionslabeledasforeground.TheROCgraphspresentedwilllisttheamount offalsepositivesinarangeof0to1.Thiswillbetheaveragenumberoffalsepositivespixels fromeachframedividedbythetotalareaoftheframe.Ofcoursethispreventseverhavingthe falsepositiveratioequal1whenforegroundobjectsarepresentbecausenotallpixelscanbe labeledasfalsepositive.Thealternativeisrepresentthenumberthefalsepositivesinaframeas thenumberoffalsepositivepixelsdividedbytheamountofpossiblefalsepositivepixelswhich wouldbethetotalimageareaminustheareaoccupiedbyforegroundregions.Theoreticallythis istheappropriatemetric,howeverishasthepotentialtofalselycharacterizetheperformance. Thereasonforthisisthatisthatthefalsepositivemeasurewillbeafunctionoftheforeground size.Ifoneframehasalownumberofforegroundregionsthentheasmallnumberoffalsepositive pixelswillappearequallyasnegativeasaframewithalargenumberofforegroundregionsand manymorefalsepositivepixels.Clearlythefalsepositiveperformanceinthelatercaseisworse andshouldbereectedwhenanalyzingthetotalresultsoverafeatureset. 5.2DataSets Threepubliclyavailabledatasetsareusedinthisworkforevaluation.Twoofthedatasetsare takenfromtheInternationalWorkshoponPerformanceEvaluationofTrackingandSurveillance PETS.Oneisdataset2fromthe2001conference[51],andtheotherisfromdatasetS3,subset 31

PAGE 40

Table5.1.Descriptionofdatasetsused DataSet Setting Diculty Resolution FrameRate Illumination OTCBVS[52] Outdoor Hard 320x240 Low HighlyDynamic PETS2001[51] Outdoor Medium 768x576 High SlightlyDynamic PETS2006[53] Indoor Easy 720x576 High Static Figure5.1.SampleframesfromOTCBVSdataset 3,fromthe2006conference.Thethirddatasetusedisdataset03fromtheOTCBVSdataset [52].ThesesetseachrepresentseparateimagedomainsasseeninTable5.1. TheOTCBVSdatasetoersthemostdicultlyduetothevaryingilluminationcausedfrom cloudcover.TheexamplesthatwereshowninFigure3.3weretakenfromthisdataset.The sharpilluminationchangescausedfromrollingcloudcovercausesextremevariationsintheRGB intensitiesanddonotmatchthebackgrounddistributionsintheMixtureofGaussiansmodel. MoresampleframesfromtheOTCBVSdatamaybeenseeninFigure5.1. AswillbementionedinSection5.3,eachdatasetissplitintotwoseparatesets:onefor trainingtogeneratetheoptimalparameters,andonefortesting.ThesplitofthePETS2001 datasetswassuchthatinthetrainingsetnomajorilluminationvariationswerepresentand inthetestingsetagradual,globalilluminationchangeoccurred.Thisnon-stratiedsplitis expectedwhenperformingsequentialsplitsofdatasets,butitcausesasuboptimalperformance ofallclassierstestedonthetestset.Onesolutionwouldbetogeneratethesplitofthetesting andtrainingsetsbyputtingeveryotherimageinonesetbecausethiswouldreectthesame illuminationconditionineachset.Thiswasnotdone,however,becausehavingunpredictable conditionsinanimagesetdemonstratesagreaterreectionoftherealworlddicultiestracking systemsface. 32

PAGE 41

Figure5.2.SampleframesfromPETS2001dataset Figure5.3.SampleframesfromPETS2006dataset OtherthantheglobalilluminationchangeattheendofthePETS2001datasetitisnota terriblydicultdataset.Therearetreeswhichswaylightlyfromwindandsomeforeground objectsthatareofarinthedistance.Figure5.2containssampleimagesfromthePETS2001 dataset. ThePETS2006datasetistheeasiestdatasetofthethree.Itisinanindoorenvironmentand thecameraisincloseproximitytotheforegroundobjects.Thelargestdicultlyisthereectance oftheoorwhichcausesminorreectionsfromtheforegroundobjects.Sampleframesfromthe PETS2006datasetmaybefoundinFigure5.3. 5.3ParameterSelection Eachofthethreedatasetswereseparatedintotwodichotomic,sequentialsets,whereone wasusedfortraining/parameterexploration,andtheotherwasusedforevaluation.Tables5.2 and5.3showtheprecisesplitsthatwereusedforeachdataset.Threeparametersexistinthe 33

PAGE 42

Table5.2.Framesusedfortraining DataSet TrainingFrames Count OTCBVS Subset1b:"img 00000.bmp"-"img 02106.bmp" 1054 PETS2001 SubsetCamera1:"0001.jpg"-"2989.jpg" 2989 PETS2006 "S7-T6-B.00000.jpg"-"S7-T6-B.01700.jpg" 1701 Table5.3.Framesusedfortesting DataSet TestingFrames Count OTCBVS Subset2b:img 00000.bmp-img 01200.bmp 601 PETS2001 SubsetCamera2:"0001.jpg"-"2989.jpg" 2989 PETS2006 "S7-T6-B.01701.jpg"-"S7-T6-B.03400.jpg" 1700 MixtureofGaussiansalgorithm.Therstparameteristhelearningrate oftheclassier.The optimalityofthisparameterislargelydependentontheframerateoftheimagesequence.The nextparameteristhescalefactorofthestandarddeviationformatchinganincomingvalueto anexistingdistribution,whichwewillrefertoas k .In[4], k wassetto2 : 5. Thenalparameteristhethreshold ,whichistheparameterthatdetermineswhatthetotal weightofthematchedGaussiandistributionsatapixelmustbeforanobjecttobebackground. ThisparameterisvariedinordertogenerateaROCgraphsoitsoptimalitywasnotexplored. Anexampleoftheeectsofvaryingthisparameteronaclassier'shypothesisoveranimage isshowninFigure5.4.TherawimageinFigure5.4bshowstheclassierpredictionspriorto thresholding.Thesumofmatchedweightsforeachdistributionatapixelismultipliedby255in ordertomaparangeof[0 ; 1]to[0 ; 255],wherethecloserto0blackapixelisthemorelikely itisbackgroundandthecloserto255whiteapixelisthemorelikelythatpixelisforeground. ROCgraphsweregeneratedforadiscretesamplingof and k ,where 2f : 001 ;: 0025 ;: 005 ;: 01 ;: 02 ;: 04 g and k 2f 1 : 5 ; 2 : 5 ; 3 : 5 ; 4 : 5 g .Thisresultedin24distinctparametercombinationstestedforeachdatasetusingthetraditionalRGBfeatureset.Table5.4lists theparametersselectedforeachdatasetbasedontheseevaluations. Table5.4.Parametersusedforeachdataset DataSet k OTCBVS[52] : 0025 2 : 5 PETS2001[51] : 001 1 : 5 PETS2006[53] : 001 1 : 5 34

PAGE 43

aOriginalImage bNon-ThresholdedClassierHypothesisImage cThreshold=130 dThreshold=170 eThreshold=210 fThreshold=250 Figure5.4.Exampleofthresholdingaclassierhypothesis 35

PAGE 44

Thegroundtruthforthetestsetsweretakenataboutevery30framesintheset,starting about800to1000framesintotheset.Thisoeredarepresentationoftheentiresetwhile providedenoughtimefortheclassiertostabilizeafterinitialization. 5.4Results Theoverallresultsforusingthepresentedfeaturesetsareextremelyencouraging.Theuseof theextendedfeaturesetoutperformedthebaselineRGBclassierallthreedatasets.Themost signicantgainswereobservedintheOTCBVSdatasetandthePETS2001dataset.Eachof theseareoutdoordatasetsandpresentdierentilluminationissues.PerformanceinthePETS 2006datasetwasalsoslightlyimproved,butbecausethedatasetitselfisnotdicultonly limitedgainswerepossible. WhenevaluatingtheresultsofensembleclassiersintheROCgraphs,veseparatefeature combinationswillbecompared.Therstsetoffeaturescomparedistheperformanceusingonly theRGBfeatures.Thisisthebaselineperformanceandfailuretoimproveuponclassication usingonlyRGBfeaturesimpliesanoverallfailureofthefeatureset.Thesecondsetoffeatures beingcomparedistheuseofeveryfeaturementionedinthispaper,whichwillbereferredtoas theextendedfeatureset.Thisis13featuresforRGB,2forgradient,and8forHaar.The nexttwofeaturesetscomparedareonlytheHaarfeaturesandonlythegradientfeatures.Finally afeaturesetwithonlytheRGBandgradientfeaturesiscompared. 5.4.1OTCBVSDataSet InFigure5.10theROCgraphoftheperformanceforeachfeaturesetontheOTCBVSdata setisshown.TheperformanceofclassicationusingonlytheRGBfeaturesisclearlytheworst. Usingallofthefeaturesappearstoperformthebestunlessaliberalacceptanceoffalsepositives isused,inwhichcaseitonlyperformsslightlyworsethanthenusingonlytheHaarfeatures. TheOTCBVSdatasetisthemostdicultdatasetbecauseofthetroublethecloudcover causes.Dicultiesinclassicationonthisdatasetwaslargelyresponsiblefortheconceptof usinganextendedfeatureset.Theoverwhelminglyhigherperformanceoftheextendedfeature setcomparedtotheRGBfeaturesetisviewedamajorsuccess. 36

PAGE 45

InFigure5.6,anROCgraphoftheperformanceofclassiersusingonlyindividualfeatures isshown.Theresultsshowthattheindividualfeaturesthatmakeuptheextendedfeaturesetdo notperformwellindividually.ThisisespeciallythecasewiththeHaarfeatureswhichgenerate ahighnumberoffalsepositiveswhenworkingindependently.Theseresultsareexpectedfrom theindividualclassiersthatmakeupanensemble.Thatis,asuccessfulensembleclassieris generallythefusionofmanyweakclassiers[34].Thoughthegradientmagnitudeoutperforms thebaselineRGB,thiswillbeshowntonotbeconsistentthroughoutallthedatasets.Further anaylsisoftheindividualfeaturesperformancewillbeshownfortheotherdatasetsanddiscussed inSection5.4.4. Resultsofthefalsealarmrateata90%truepositiverateforeachframetestedareshownin Figure5.7.Theperformanceofusingallfeaturesishighlyconsistent,whilewhenusingtheRGB featuresresultsmoresporadic,andworseateveryframe.Consistencyinclassicationisahighly desirablefeature,anditisclearlyexhibitedherewhenusingtheextendedfeatureset. SamplebitmapsoftheclassicationofaframeintheOTCBVSdatasetatvariousthresholds areshownforRGBclassicationinFigure5.8andclassicationusingtheentirefeaturesetin Figure5.9.Thedramaticeectsofthecloudcoverisapparentinboth,butitissignicantly diminishedusingtheextendedfeatureset.IntheRGBclassicationusingamorediscriminating thresholderodestheforegroundobjectsbeforeerodingtheerroneousclassicationofthecloud cover,asclearlyvisibleinFigure5.8.Observingthebitmapsfromtheextendedfeaturesetshow thatwhileoptimalperformanceisstillnotachieved,thegainsareclearandsignicantwhen comparedtousingtheRGBfeatureset. 5.4.2PETS2001DataSet TheresultsforthePETS2001datasetareshowninFigure5.10.Itisquiteclearthatthe baselineRGBfeaturesetwaseasilybeatenbyallothercombinations.Onceagainusingall13 featuresgeneratedtheoptimalperformancethroughoutmostoftheROCgraph.Dependingon theacceptedtoleranceforfalsepositives,usingonlytheHaarfeaturesandonlythegradient featuresoersolidperformanceaswell. ThedisparitybetweentheperformanceofthetraditionalRGBfeatureset'sperformanceand theextendedfeatureset'sperformanceareclearandsignicantwhenclassifyingonthePETS 37

PAGE 46

Figure5.5.OverallresultsontheOTCBVSdataset Figure5.6.ResultsforeachindividualfeatureontheOTCBVSdataset 38

PAGE 47

Figure5.7.FramebyframefalsepositiveresultsonOTCBVSdataset aOriginalImage bThreshold=220 cThreshold=230 dThreshold=240 Figure5.8.RGBclassicationimageofanOTCBVSframe 39

PAGE 48

aOriginalImage bAllFeatures,Threshold=220 cAllFeatures,Threshold=230 dAllFeatures,Threshold=240 Figure5.9.EnsembleclassicationimageofanOTCBVSframe 40

PAGE 49

2001dataset.ThesamedramaticimprovementwasshownaswellfortheOTCBVSdataset. Thiscanleadtoageneralizationthatusingtheextendedfeaturesetonoutdoorimageryis optimaltousingonlyRGBfeatures. TheresultsoftheindividualfeaturesonthePETS2001datasetareshowninFigure5.10. OnceagaintheindividualfeaturesdonotperformaswellastheRGBfeatureseteventhough theensembleofthesefeaturesgreatlyoutperformstheRGBonlyclassier.Aswasalsothecase intheOTCBVSdataset,thegradientmagnitudefeaturedoesperformbetterthantheRGB feature.ThereasonforthiswillbediscussedinSection5.4.4. TheindividualframeperformanceisshowninFigure5.12.Onceagain,usingtheentire featuresetresultsinanextremelyconsistentfalsealarmrate.Thisisstillnotthecaseforthe RGBfeatureclassication.Itexhibitsresultscomparabletotheextendedfeaturesetinitially, butthisperformancedeterioratessignicantlyattheendofthedataset.Theendofthedata setiswhentheglobalilluminationchangeoccurs. Asmentionedpreviously,onecharacteristicofthePETS2001datasetisthatthereisa gradual,globalilluminationchangetowardstheendofthesequence.Thisilluminationchange causesproblemsusingonlyRGBfeaturesbutnotwhenusingtheextendedfeatureset.An exampleofthisisshowninFigures5.13and5.14,wheretheregionsintheyellowboxindicate theforegroundobjects,whicharetwopeoplewalkingtogether. InFigure5.13,theRGBclassicationofthesampleframefromwhentheilluminationchange isoccurringshowsthatthereectivenessofthebuildingiscausingmanypixelsofthebuildingto beclassiedaforegroundwhenamoreliberalthresholdisapplied.Asthatthresholdisraised, though,theactualforegroundpixelsarelost.Thisshowsthatanoptimalthresholddoesnotexist thatcandiscriminatethefalsepositivesfromthetruepositives.InFigure5.14thesamesampleis presentedbutclassicationisnowdonewiththeentirefeatureset.Withaliberalthresholdfalse positivesexistaswell,howeverasthisthresholdisraisedthefalsepositiveregionsaresuppressed andtheactualforegroundobjectsarestillmaintained.Clearlythisisansignicantimprovement. 5.4.3PETS2006DataSet Figure5.15containstheROCgraphfortheperformanceofthefeaturesetsonthePETS 2006dataset.ItisnotthecasethattheRGBfeatureperformedtheworstinthisset.Infactthe 41

PAGE 50

Figure5.10.OverallresultsonthePETS2001dataset Figure5.11.ResultsforeachindividualfeatureonthePETS2001dataset 42

PAGE 51

Figure5.12.FramebyframefalsepositiveresultsonPETS2001dataset performanceoftheRGBfeaturesetwasquitecompetitivewiththeextendedfeatureset.Itisstill thecase,though,thattheextendedfeaturesetprovidedtheoptimalperformancewithrespect totheotherfeaturesetcombinations.ThepoorperformancewhenusingonlytheHaarfeatures andonlythegradientfeaturesarequitenotableaswell.Despitethedisappointingperformance ofthesefeaturestheystillmanagetoaugmenttheRGBfeaturesandmaketheextendedfeature settheoptimalclassier. InFigure5.16,theresultsoftheindividualfeaturesclassicationonthePETS2006dataset areshown.TheRGBclearlyoutperformsallotherfeatures,includingthegradientmagnitude. Thisnalizesthetrendseennowinallthreedatasetsthattheindividualfeaturesdonotoer anyconsistentimprovementovertheRGBfeatureset,whileusingallofthesefeaturesinan ensembleclassieroutperformstheRGBfeatureseverytime. Figure5.17containstheperformanceatindividualframesforthePETS2006dataset.The extendedfeaturesetappearslessconsistentinthedataset.Thismayberathermisleading, however,becausethefalsealarmrateissolowthattheinconsistenciesbecomeexaggerated. ThePETS2006datasetoeredthelowestperformancegainswhencomparingtheextended featuresettheRGBfeatures.Theprimaryreasonforthisisthatthisdatasetisnotaterribly 43

PAGE 52

aOriginalImage bThreshold=220 cThreshold=230 dThreshold=240 Figure5.13.RGBclassicationimageofaPETS2001frame 44

PAGE 53

aOriginalImage bAllFeatures,Threshold=220 cAllFeatures,Threshold=230 dAllFeatures,Threshold=240 Figure5.14.EnsembleclassicationimageofaPETS2001frame 45

PAGE 54

Figure5.15.OverallresultsonthePETS2006dataset dicultdataset.Becauseitisanindoorenvironmentandtheforegroundobjectsareinclose proximitytothecamerathereislittleroomforimprovementfromthestandardRGBfeatures. DespitethisaslightimprovementwasstillseenwhenincorporatinggradientfeaturesandHaar features.Inrealworlduseitwouldbeunlikelytousetheseadditionalfeatures,however,dueto theincreasedcomputationalloadtheyincurandthemarginalgainstheyoer. 5.4.4FeaturePerformance Theperformanceofclassiersonlyusingasinglefeaturefromtheextendedfeaturesetwas poor,asobservedinFigures5.6,5.11,and5.16.Whentheseweakclassiersarecombinedinto anensembleofclassiers,themeta-classicationoerssubstantialresults.Thisadherestothe theoryofensemblelearningpresentedbySchapirein[34]. AnexampleofhowaseriesofweakclassierhypothesesareabletoresultinastronghypothesisinbackgroundclassicationisshowninFigure5.18,wherethe8hypothesesfromeach individualHaarfeatureareshownaswellasthefusionofthoseclassiers.TheHaarclassier clearlyhasatendencytooverclassifypixelsareforeground,resultinginmanyfalsepositives. ThisisevidencedintheROCgraphswheretheclassiersusingindividualHaarfeatureswillboth maximizethetruepositivesandfalsepositives.Fortunatelythereisalowoverlapbetweenthese 46

PAGE 55

Figure5.16.ResultsforeachindividualfeatureonthePETS2006dataset Figure5.17.FramebyframefalsepositiveresultsonPETS2006dataset 47

PAGE 56

featuresfalsepositivesandahighoverlapintheirtruepositives,resultinginstrongbackground classication. ItwasobservedinthePETS2001datasetandtheOTCBVSdatasetthattheperformance ofclassicationusingonlythegradientmagnitudefeaturewasmoresuccessfulthanusingthe baselineRGB.SimilarresultswerenotobservedinthePETS2006dataset.Itisbelievedthat thereasonthisoccuredisthattheforegroundobjectsinthePETS2001andOTCBVSdata setsaremuchfartherfromthecamerathantheyareinthePETS2006dataset.Thisresults inlowerresolutionsfortheforegroundobjects,whichinturnimplieslesshomogeneousregions overthespaceoccupiedbytheforegroundobject.Thenalimplicationisthatedgefeatures aremoreprominentonthelowresolutionobjects.ThismaybeobservedinFigure5.19,where anobjectfromthePETS2006andOTCBVSdatasetareshownaswellastheircorresponding gradientmagnitudeimage.Inthelowresolutionobjectithasdistinctedgefeaturesoveritsentire silhouette,whileinthehighresolutionobjectthesefeaturesareonlyprominentontheboundaries oftheobjectandnotinthehomogeneousregions.Aconclusionfromthisobservationisthat theuseofgradientmagnitudeinbackgroundclassicationofsmallerobjects,orobjectsarefar distances,issuperiortotheRGBfeatures. DuetotheheightenedperformanceofthegradientmagnitudeinthePETS2001andOTCBVS datasets,acomparisonoftheperformanceagainsttheextendedfeaturesetwasperformed.The ROCcomparisonscanbeseeninFigure5.20.IntheOTCBVSdataset,theuseofallfeatures resultsinfarmoreaccurateclassicationthanonlyusingthegradientmagnitude.Theextended featuresetalsooutperformsthegradientmagnitudeinthePETS2001dataset,thoughthere islesssignicantofanimprovement.Becauseoftheconsistentoutperformanceoftheextended featuresetoverusingonlythegradientmagnitude,itrecomendedtouseallthefeaturesformore accurateclassication.Ifcomputationaldemandsarealimitationinasystemobservinglow resolutionobjectsthenusingonlythegradientmagnitudeisanadequatealternative. 48

PAGE 57

aOriginalImage bHaar1Classication cHaar2Classication dHaar3Classication eHaar4Classication fHaar5Classication gHaar6Classication hHaar7Classication iHaar8Classication jAllHaarClassicationsCombined Figure5.18.Weakhypothesesfusedintoasinglestronghypothesis 49

PAGE 58

aPETS2006OriginalImage bPETS2006MagnitudeImage cOTCBVSOriginalImagex Zoom dOTCBVSMagnitudeImagex Zoom Figure5.19.Eectofobjectsizeongradientmagnitude 50

PAGE 59

Figure5.20.Classicationusingensembleandgradientmagnitude 51

PAGE 60

CHAPTER6 CONCLUSION 6.1Summary Traditionalimagefeaturesinbackgroundclassicationareinsucientinmanyenvironments. Particularlyoutdoorenvironments,environmentswithvaryingillumination,andsceneswherethe foregroundobjectsareeithersmallorafardistancefromthecamera.Withouttheavailability ofadditionalspectraldatasuchasinfraredorthermal,alternatefeaturesetswillneedtobe derivedfromthestandardRGBfeatures.Usingafeaturesetthatincludesgradientmagnitude, gradientorientation,andeightseparateHaarfeaturesforclassicationyieldedsignicantlysuperiorclassicationperformanceonmoredicultdatasets,andstillmarginallyoutperformedin asimplerdataset. Asummaryoftheperformancesintermsoffalsepositiveratesonallthreedatasetsare showninFigure6.1NOTE:InFigure6.1,falsepositiveperformanceintheOTCBVSdataset scaleddownbyafactorof6.75.Eachperformanceisbasedonthelowestfalsepositiverate thatgeneratedatleasta90%truepositiveclassicationrate.Table6.1showsthesameresults, aswellastheratioofimprovementseenusingtheextendedfeatureset.Itcanbeseenthat falsepositivesaremorethan5timeshigherintheOTCBVSdataset,andmorethan6times higherinthePETS2001datasetwhenusingRGBfeaturesinsteadoftheextendedfeatureset presented.Thesetwodatasetsarebothoutdoordatasetswithdynamicilluminationevents.In Figure6.2andFigure6.3ROCgraphsareshownthatcompareRGBclassicationstrictlytothe classicationusingtheextendedfeaturesetoneachofthesetwodatasets. Table6.1.Falsepositivesata90%truepositiverate DataSet RGBFP AllFeaturesFP FPRatio OTCBVS 0.1046 0.0204 5.17:1 PETS2001 0.0094 0.0014 6.71:1 PETS2006 0.0027 0.0019 1.42:1 52

PAGE 61

Figure6.1.Falsepositiveresultsat90%truepositiverate Figure6.2.OTCBVSROCgraphsummary 53

PAGE 62

Figure6.3.PETS2001ROCgraphsummary ComparisonsusingROCgraphsclearlyshowedtheadvantageofusingadditionalfeaturesfor backgroundclassicationinimagesequences.Thealgorithmusedwasanensemblealgorithm. Itfusedtheresultofmultipleclassierhypothesesintoasinglehypothesis.Eachindividual classieroperatedononlyoneofthefeaturesinthefeatureset,yieldingadecisionindependent ofallotherfeaturesassumingthefeaturesthemselvesareindependent,whichstrictlyspeaking theyarenot. Variousfeaturesetcombinationswereexploredandtheoverallresultwasthatusingallofthe featuresresultedinthehighestperformance.Individuallymostfeaturesperformedpoorly.This adheredtothepredominanttheoryinensembleclassiersthatthefusionofmanyweakclassiers willacttogetherasasinglestrongclassier.Whenthesefeatureswereallfusedtogether,the classicationataxedtruepositiveratealwaysyieldedlessfalsepositivesthanusingonlyRGB features. Onesinglefeaturethatdidperformwellindividuallyontwoofthreedatasetswasthegradient magnitude.Eachofthetwodatasetsitexcelledinwereoutdoorandtheforegroundobjectswere atafardistancefromthecamera.Theperformancedidnotsurpassusingallofthefeatures,but itdidsurpassusingtheRGBfeatures.Thisdrawstheconclusionthatforalowcostandmore 54

PAGE 63

ecientclassierwhenbeingrestrictedtolowresolutionimagesoftheobjectsbeingtracked, usinggradientmagnitudeasasinglefeaturewilloersatisfactoryperformancegainsoverRGB features. 6.2FutureWork Theoptimisticresultsgeneratedopensthedoorformanyavenuesoffutureresearchinusing ensemblesofbackgroundclassiers. WhilesignicantimprovementswereshownusingHaarfeatures,aswellasgradientfeatures, incorporatingadditionalimageprocessingfeaturesmayallowforafurtherimprovement.One featurethatwouldbeintuitivetoincorporateistexture.PerpixeltexturefeaturesusingGabor ltersasdescribedin[54],[55]wouldoerasimilar,butdistinctlydierent,representationof apixelthanHaarfeatures.AdditionalfeaturesthatcouldbeincludedaretheLaplacianof Gaussian,non-thresholdedSIFTfeatures[56],depth,andmotionfeatures. Anotherdirectionisthefusionofdierentclassieralgorithms.Forexample,insteadofusing aseriesofMixtureofGaussianclassiers,anensembleofMixtureofGaussian,KalmanFilter, andotherclassierscouldbeused.Thisapproachwastherstdirectionthisworkresearched. Themajorstoppingpointwasthelackofdistinctclassiersforbackgroundsubtraction.Insteadoffusingstrongclassieralgorithms,usingaseriesofweakalgorithmsshouldoersimilar improvementstousinganextendedfeatureset. Anextendedfeaturesetcouldalsobeusedwithouthavingmultipleclassiers.Becausethe MixtureofGaussiansalgorithmnormallyoperatesonafeaturevectorwherethethreefeatures areRGB,usingthesamefeaturesinthispaperasasinglevectorforoneclassiershouldproduce uniqueresults.Itmaynotbethecasethatsignicantlydierentresultsareyieldedwiththis approachunlessafullcovariancematrixiscomputed,whichisgenerallynotthecase.Instead itisgenerallythecasethatonlythediagonalofthecovariancematrixiscalculatedinorderto reducecomputationaldemands.Anotherconsiderationwhenusingavectoroffeaturesinstead ofanensembleofclassiersisthatifasingledimensionofnewinstancevectordoesnotfall within2 : 5 ofitscorrespondingdimension,andeveryothervectordimensiondoes,thenthe instancedoesnotmatchthedistribution.Withalargerfeaturesetitisoftenthecasethatnot everyfeaturematchesitscorrespondingdimensionforitsfeatureinthemodel.Thisapproachis 55

PAGE 64

alsodierentthanusinganunanimousvoteschemeforclassierfusionintheensembleclassier presentedinthiswork.Thisisbecauseifonlyafewfeaturesmatchthenewinstancetheclassiers willupdatethemselvesbasedonamatchintheensemblealgorithm,whileforthesinglefeature vectoralgorithmtheupdatewouldnotbebasedonamatchunlesseverydimensionwasmatched wherematchingisbeingwith2 : 5 Evaluationoftheensemblealgorithm'sperformancecouldbeperformedusingtheVACEevaluationframeworkin[57].Thisframeworkoersamorecomprehensiveanalysisofthealgorithms robustness,aswellasacomparisonagainsttheperformanceofothertrackingalgorithms. Real-timeimplementationofthissystemwouldrequireasystemwithmultipleprocessing coresandalargesizedmainmemory,butotherwiseitsrealizationisbelievedtobereasonable. Enhancedengineeringofthealgorithmstoexploittheinherentparallelismoftheensembleshould allowforthehandlingofrelativelyhighframeratesdependingontheimageresolution. 6.3FinalThoughts Itisbelievedthatfurtherresearchneedstobeconductedinordertofurtherunderstandthe gainsthatwereobservedinthiswork.Therewasaclearlydemonstratedadvantagetousing additionalfeaturesinbackgroundclassication,eventhoughmanyofthesefeaturesarevery poorwhenusedindividually.Throughgreaterinvestigationamoreclearlydenedsetofoptimal ensemblefeatureswouldallowforamoregeneralbackgroundclassicationalgorithmthatcould havekeyimpactsindefense,robotics,andsurveillanceapplications. 56

PAGE 65

REFERENCES [1]R.Jain,R.Kasturi,andB.Schunck. MachineVision .McGrawHill,1995. [2]C.Wren,A.Azarbayejani,T.Darrell,andA.Pentland.Pnder:Real-timetrackingofthe humanbody. IEEETrans.PatternAnal.Mach.Intell. ,19:780{785,1997. [3]D.Koller,J.Weber,andJ.Malik.Robustmultiplecartrackingwithocclusionreasoning. In ECCV ,pages189{196,1994. [4]C.StauerandW.Grimson.Adaptivebackgroundmixturemodelsforreal-timetracking. In IEEEConferenceonComputerVisionandPatternRecognition ,volume2,1999. [5]J.B.MacQueen.Somemethodsforclassicationandanalysisofmultivariateobservations. In Proceedingsof5thBerkeleySymposiumonMathematicalStatisticsandProbability ,pages 281{297,1967. [6]S.Atev,O.Masoud,andN.Papanikolopoulos.Practicalmixturesofgaussianswithbrightnessmonitoring.In The7thInternationalIEEEConferenceonIntelligentTransportation Systems ,2004. [7]G.Gordon,T.Darrell,M.Harville,andJ.Woodll.Backgroundestimationandremoval basedonrangeandcolor.In ImageProcessing,2001.Proceedings.2001InternationalConferenceon ,volume2,pages395{398,2001. [8]M.Harville,G.Gordon,andJ.Woodll.Adaptivevideobackgroundmodelingusingcolor anddepth.In ConferenceonImageProcessing ,pages90{93,2001. [9]K.Karmann,A.Brandt,andR.Gerl.Movingobjectsegmentationbasedonadaptive referenceimages.In EuropeanSignalProcessingConf ,pages951{954,1990. [10]C.Ridder,O.Munkelt,andH.Kirchner.Adaptivebbackgroundestimationandforeground detectionusingkalmanltering.In InProc.ICAM ,pages193{199,1995. [11]R.E.Kalman.Anewapproachtolinearlteringandpredictionproblems. JournalofBasic Engineering ,82:3545,1960. [12]M.LeungandY.Yang.Humanbodymotionsegmentationinacomplexscene. Pattern Recognition ,20:55{64,1987. [13]K.Toyama,J.Krumm,B.Brumitt,andB.Meyers.Wallower:Principlesandpracticeof backgroundmaintenance.In ICCV ,pages255{261,1999. [14]Y.Zhang,Z.Liang,Z.Hou,H.Wang,andM.Tan.Anadaptivemixturegaussianbackground modelwithonlinebackgroundreconstructionandadjustableforegroundmergencetimefor motionsegmentation.In IEEEInternationalConferenceonIndustrialTechnology ,volume 14-17,pages23{27,2005. 57

PAGE 66

[15]Z.HouandC.Han.Abackgroundreconstructionalgorithmbasedonpixelintensityclassicationinremotevideosurveillancesystem.In ProceedingsoftheSeventhInternational ConferenceonInformationFusion ,pages754{759,2004. [16]A.Shimada,D.Arita,andR.Taniguchi.Dynamiccontrolofadaptivemixture-of-gaussians backgroundmodel.In ProceedingsoftheIEEEInternationalConferenceonVideoandSignal BasedSurveillance ,page5,2006. [17]DGao,JZhou,andLXin.Anovelalgorithmofadaptivebackgroundestimation.In InternationalConferenceonImageProcessing ,2001. [18]S.Messelodi,C.Modena,N.Segata,andM.Zanin.Akalmanlterbasedbackground updatingalgorithmrobusttosharpilluminationchanges.In ICIAP2005,13thInternational ConferenceonImageAnalysisandProcessing ,pages163{170,2005. [19]H.WangandD.Suter.Backgroundsubtractionbasedonarobustconsensusmethod.In ICPR'06:Proceedingsofthe18thInternationalConferenceonPatternRecognition ,pages 223{226,Washington,DC,USA,2006.IEEEComputerSociety. [20]S.Avidan.Ensembletracking. IEEETransactionsonPatternAnalysisandMachineIntelligence ,29:261{271,2007. [21]Y.FreundandR.E.Schapire.Decision-theoreticgeneralizationofon-linelearningandan applicationtoboosting. JournalofComputerandSystemSciences ,55:119139,1997. [22]A.Criminisi,G.Cross,A.Blake,andV.Kolmogorov.Bilayersegmentationoflivevideo. CVPR ,1:53{60,2006. [23]K.Patwardhan,G.Sapiro,andV.Morellas.Robustforegroundsegmentationusingpixel layers. PatternAnalysisandMachineIntelligence,IEEETransactionson ,4:746{751,2008. [24]A.MittalandN.Paragios.Motion-basedbackgroundsubtractionusingadaptivekernel densityestimation. ComputerVisionandPatternRecognition ,2:302{309,2004. [25]B.K.P.HornandB.G.Schunck.Determiningopticalow. ArticialIntelligence ,17:185203, 1981. [26]M.Zucchelli,J.Santos-Victor,andH.Christensen.Multipleplanesegmentationusingoptical ow.In BritishMachineVisionConference ,2002. [27]B.W.Silverman. DensityEstimationforStatisticsandDataAnalysis .ChapmanandHall, 1986. [28]TGeversandASmeulders.Color-basedobjectrecognition. PatternRecognition ,32:453{ 464,1999. [29]T.Horprasert,D.Harwood,andL.S.Davis.Astatisticalapproachforreal-timerobust backgroundsubtractionandshadowdetection.In ICCV ,1999. [30]M.XuandT.Ellis.Illumination-invariantmotiondetectionusingcolourmixturemodels. In BritishMachineVisionConference ,2001. [31]S.Jabri,Z.Duric,H.Wechsler,andA.Rosenfeld.Detectionandlocationofpeopleinvideo imagesusingadaptivefusionofcolorandedgeinformation.In InternationalConferenceon PatternRecognition ,volume4,pages627{630,2000. 58

PAGE 67

[32]O.Javed,K.Shaque,andM.Shah.Ahierarchicalapproachtorobustbackgroundsubtractionusingcolorandgradientinformation.In MotionandVideoComputing,Proceedingsof Workshopon ,pages22{27,2002. [33]B.V.DasarathyandB.V.Sheela.Compositeclassiersystemdesign:Conceptsandmethodology. ProceedingsoftheIEEE ,67:708{713,1979. [34]R.Schapire.Thestrengthofweaklearnability. MachineLearning ,5:197{227,1990. [35]L.Breiman.Baggingpredictors. MachineLearning ,24:123140,1996. [36]D.H.Wolpert.Stackedgeneralization. NeuralNetworks ,5:241{259,1992. [37]R.A.Jacobs,M.I.Jordan,S.J.Nowlan,andG.E.Hinton.Adaptivemixturesoflocalexperts. NeuralComputation ,3:7987,1991. [38]R.Polikar.Ensemblebasedsystemsindecisionmaking. IEEECircuitsandSystemsMagazine ,6:21{45,2006. [39]N.SiebelandS.Maybank.Fusionofmultipletrackingalgorithmsforrobustpeopletracking. In Proceedingsofthe7thEuropeanConferenceonComputerVision ,volumeIV,pages373{ 387,May2002. [40]J.Kittler,M.Hatef,R.Duin,andJ.Matas.Oncombiningclassiers. PatternAnalysisand MachineIntelligence,IEEETransactionson ,20:226{269,1998. [41]L.Kuncheva.Atheoreticalstudyonsixclassierstrategies. PatternAnalysisandMachine Intelligence,IEEETransactionson ,24:281{286,2002. [42]T.Ho,J.Hull,andS.Srihari.Decisioncombinationinmultipleclassiersystems. Pattern AnalysisandMachineIntelligence,IEEETransactionson ,16:66{75,1994. [43]A.RossandA.K.Jain.Informationfusioninbiometrics. PatternRecognitionLetters 24:2115{2125,2003. [44]J.Canny.Acomputationalapproachtoedgedetection. IEEETransactionsonPattern AnalysisandMachineIntelligence ,8:679{714,1986. [45]C.Papageorgiou,M.Oren,,andT.Poggio.Ageneralframeworkforobjectdetection.In InInternationalConferenceonComputerVision ,1998. [46]M.Oren,C.Papageorgiou,P.Sinha,E.Osuna,andT.Poggio.Pedestriandetectionusing wavelettemplates.In Proc.ComputerVisionandPatternRecognition ,pages193{199,1997. [47]P.ViolaandM.Jones.Rapidobjectdetectionusingaboostedcascadeofsimplefeatures. In ProceedingsIEEEConferenceonComputerVisionandPatternRecognition ,2001. [48]R.LienhartandJ.Maydt.Anextendedsetofhaar-likefeaturesforrapidobjectdetection. In InternationalConferenceonImageProcessing ,2002. [49]T.Fawcett.Anintroductiontorocanalysis. PatternRecognitionLetters ,27:861{874,2006. [50]J.Makhoul,F.Kubala,R.Schwartz,andR.Weischedel.Performancemeasuresforinformationextraction.In ProceedingsofDARPABroadcastNewsWorkshop ,1999. [51]PETS2001Dataset.ftp://ftp.pets.rdg.ac.uk/pub/PETS2001.Dataset2. 59

PAGE 68

[52]OTCBVSBenchmarkDatasetCollection.http://newton.ex.ac.uk/tex/pack/bibtex/btxdoc/ node6.html.Dataset03. [53]PETS2006Dataset.http://www.cvg.rdg.ac.uk/PETS2006/data.html.DatasetS3. [54]A.K.JainandF.Farrokhnia.Unsupervisedtexturesegmentationusinggaborlters.In Systems,ManandCybernetics,IEEEInternationalConferenceon ,1990. [55]B.ManjunathandW.Ma.Texturefeaturesforbrowsingandretrievalofimagedata. Pattern AnalysisandMachineIntelligence ,18:837{842,1996. [56]D.Lowe.Distinctiveimagefeaturesfromscale-invariantkeypoints. InternationalJournalof ComputerVision ,60:91{110,2004. [57]V.Manohar,M.Boonstra,V.Korzhova,P.Soundararajan,D.Goldgof,R.Kasturi, S.Prasad,H.Raju,R.Bowers,andJ.Garofolo.Petsvs.vaceevaluationprograms:A comparativestudy.In ProceedingsoftheNinthIEEEInternationalWorkshoponPerformanceEvaluationofTrackingandSurveillance ,2006. 60


xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam Ka
controlfield tag 001 002021440
003 fts
005 20090729113821.0
006 m||||e|||d||||||||
007 cr mnu|||uuuuu
008 090729s2008 flu s 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0002573
035
(OCoLC)428440170
040
FHM
c FHM
049
FHMM
090
QC76 (Online)
1 100
Klare, Brendan F.
0 245
Background subtraction using ensembles of classifiers with an extended feature set
h [electronic resource] /
by Brendan F. Klare.
260
[Tampa, Fla] :
b University of South Florida,
2008.
500
Title from PDF of title page.
Document formatted into pages; contains 60 pages.
502
Thesis (M.S.C.S.)--University of South Florida, 2008.
504
Includes bibliographical references.
516
Text (Electronic thesis) in PDF format.
3 520
ABSTRACT: The limitations of foreground segmentation in difficult environments using standard color space features often result in poor performance during autonomous tracking. This work presents a new approach for classification of foreground and background pixels in image sequences by employing an ensemble of classifiers, each operating on a different feature type such as the three RGB features, gradient magnitude and orientation features, and eight Haar features. These thirteen features are used in an ensemble classifier where each classifier operates on a single image feature. Each classifier implements a Mixture of Gaussians-based unsupervised background classification algorithm. The non-thresholded, classification decision score of each classifier are fused together by taking the average of their outputs and creating one single hypothesis. The results of using the ensemble classifier on three separate and distinct data sets are compared to using only RGB features through ROC graphs. The extended feature vector outperforms the RGB features on all three data sets, and shows a large scale improvement on two of the three data sets. The two data sets with the greatest improvements are both outdoor data sets with global illumination changes and the other has many local illumination changes. When using the entire feature set, to operate at a 90% true positive rate, the per pixel, false alarm rate is reduced five times in one data set and six times in the other data set.
538
Mode of access: World Wide Web.
System requirements: World Wide Web browser and PDF reader.
590
Advisor: Sudeep Sarkar, Ph.D.
653
Tracking
Classification
Segmentation
Fusion
Illumination invariant
690
Dissertations, Academic
z USF
x Computer Science
Masters.
773
t USF Electronic Theses and Dissertations.
4 856
u http://digital.lib.usf.edu/?e14.2573