USF Libraries
USF Digital Collections

Shape and pose recovery of novel objects using three images from a monocular camera in an eye-in-hand configuration

MISSING IMAGE

Material Information

Title:
Shape and pose recovery of novel objects using three images from a monocular camera in an eye-in-hand configuration
Physical Description:
Book
Language:
English
Creator:
Colbert, Steven
Publisher:
University of South Florida
Place of Publication:
Tampa, Fla
Publication Date:

Subjects

Subjects / Keywords:
Shape Reconstruction
Shape from Silhouettes
Object Classification
Robot Vision
Machine Vision
Dissertations, Academic -- Mechanical Engineering -- Masters -- USF   ( lcsh )
Genre:
non-fiction   ( marcgt )

Notes

Abstract:
ABSTRACT: Knowing the shape and pose of objects of interest is critical information when planning robotic grasping and manipulation maneuvers. The ability to recover this information from objects for which the system has no prior knowledge is a valuable behavior for an autonomous or semi-autonomous robot. This work develops and presents an algorithm for the shape and pose recovery of unknown objects using no a priori information. Using a monocular camera in an eye-in-hand configuration, three images of the object of interest are captured from three disparate viewing directions. Machine vision techniques are employed to process these images into silhouettes. The silhouettes are used to generate an approximation of the surface of the object in the form of a three dimensional point cloud. The accuracy of this approximation is improved by fitting an eleven parameter geometric shape to the points such that the fitted shape ignores disturbances from noise and perspective projection effects. The parametrized shape represents the model of the unknown object and can be utilized for planning robot grasping maneuvers or other object classification tasks. This work is implemented and tested in simulation and hardware. A simulator is developed to test the algorithm for various three dimensional shapes and any possible imaging positions. Several shapes and viewing configurations are tested and the accuracy of the recoveries are reported and analyzed. After thorough testing of the algorithm in simulation, it is implemented on a six axis industrial manipulator and tested on a range of real world objects: both geometric and amorphous. It is shown that the accuracy of the hardware implementation performs exceedingly well and approaches the accuracy of the simulator, despite the additional sources of error and uncertainty present.
Thesis:
Thesis (M.S.M.E.)--University of South Florida, 2010.
Bibliography:
Includes bibliographical references.
System Details:
Mode of access: World Wide Web.
System Details:
System requirements: World Wide Web browser and PDF reader.
Statement of Responsibility:
by Steven Colbert.
General Note:
Title from PDF of title page.
General Note:
Document formatted into pages; contains X pages.

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
usfldc doi - E14-SFE0003425
usfldc handle - e14.3425
System ID:
SFS0027740:00001


This item is only available as the following downloads:


Full Text

PAGE 1

ShapeandPoseRecoveryofNovelObjectsUsingThreeImagesfromaMonocularCamerainan Eye-in-HandConguration by StevenC.Colbert Athesissubmittedinpartialfulllment oftherequirementsforthedegreeof MasterofScienceinMechanicalEngineering DepartmentofMechanicalEngineering CollegeofEngineering UniversityofSouthFlorida MajorProfessor:RajivDubey,Ph.D. RedwanAlqasemi,Ph.D. SudeepSarkar,Ph.D. DateofApproval: April6,2010 Keywords:ShapeReconstruction,ShapefromSilhouettes,ObjectClassication,RobotVision, MachineVision c Copyright2010,StevenC.Colbert

PAGE 2

TableofContents ListofTables..........................................iii ListofFigures..........................................iv Abstract.............................................vi Chapter1Introduction....................................1 1.1Motivation.......................................1 1.2Objectives.......................................2 1.3Outline........................................3 Chapter2LiteratureReview.................................4 2.1ModelBasedRecovery................................4 2.2RuntimeReconstruction...............................6 Chapter3FundamentalBackground............................9 3.1CoordinateRepresentations..............................9 3.1.1HomogeneousCoordinates..........................9 3.1.1.1TwoDimensions..........................10 3.1.1.2ThreeDimensions.........................11 3.1.2CoordinateTransformations.........................11 3.1.33Dto2DProjection.............................15 3.1.42Dto3DProjection.............................17 3.2MachineVision....................................18 3.2.1CameraModels................................19 3.2.1.1IdealCameraModel........................19 3.2.1.2CCDCameraModel........................21 3.2.2DigitalImageRepresentation........................22 3.2.3Neighborhoods................................25 3.2.4ConnectedComponents...........................26 3.2.5EdgeDetection................................27 3.2.6FloodFill...................................28 3.2.7BinaryImageFeatures............................30 3.2.8ANoteonImageSegmentation.......................31 3.3Superquadrics.....................................32 3.3.1Overview...................................32 3.3.2DerivationandRepresentation........................32 3.3.3UsefulProperties...............................38 i

PAGE 3

Chapter4ReconstructionAlgorithm............................39 4.1ImageCapture.....................................39 4.1.1CameraCalibration..............................40 4.1.2ImageUndistortion..............................41 4.2SegmentationandSilhouetteGeneration.......................42 4.2.1SegmentingSimulatedShapes........................43 4.2.2ColorBasedSegmentationofRealObjects.................44 4.3SurfaceApproximation................................50 4.3.1ShapefromSilhouettesOverview......................50 4.3.2ConstructionoftheBoundingSphere....................51 4.3.3Approximatingthe3DSurface........................56 4.3.4PerspectiveProjectionError.........................59 4.4GeometricShapeFitting...............................60 4.4.1ClassicalSuperquadricCostFunction....................63 4.4.2ErrorRejectingCostFunction........................64 4.4.3InitialParameterEstimation.........................65 4.5ExampleReconstruction...............................68 Chapter5Simulation....................................71 5.1CoreAlgorithm....................................71 5.2GraphicalSimulator..................................72 5.2.1Capabilities..................................72 5.2.2Limitations..................................73 5.3SimulationTrialsandResults.............................73 Chapter6HardwareImplementation............................77 6.1HardwareComponentsandSetup..........................77 6.1.1Robot.....................................77 6.1.2Camera....................................79 6.1.3Network....................................81 6.1.3.1ExternalKUKAControllerandtheOPCServer.........81 6.1.3.2WirelessCameraandObjectReconstruction...........82 6.2ViewingPositions...................................83 6.3TestObjects......................................83 6.4ExperimentalTrialsandResults...........................84 6.4.1BatteryBox..................................85 6.4.2CupStack...................................86 6.4.3YarnBall...................................86 6.4.4CardinalStatue................................87 6.5Limitations......................................88 Chapter7ConclusionandFutureWork...........................91 7.1FutureWork......................................91 References............................................93 ii

PAGE 4

ListofTables Table1SimulationResults..................................75 Table2ExperimentalResults................................89 iii

PAGE 5

ListofFigures Figure1Apointin R 3 .....................................10 Figure2Twocoordinateframesseparatedbypuretranslation................13 Figure3Twocoordinateframesseparatedbypurerotation..................14 Figure4Twocoordinateframesseparatedbyageneraltranslationandrotation.......16 Figure52Dto1Dprojection.................................17 Figure6Theidealcameramodel...............................20 Figure7Anexampleofradialdistortion...........................22 Figure8Anrepresentativeexampleofadigitalimage....................24 Figure9AcolorRGBimage.................................25 Figure10Asmoothingoperationusinganeighborhood...................26 Figure11A4-neighborhoodand8-neighborhood......................27 Figure12Connectedcomponentlabeling...........................28 Figure13Cannyedgedetection................................29 Figure14Anexampleofaoodlloperation........................29 Figure15Abinaryimageexample..............................30 Figure16Asampleofsuperquadricshapes..........................33 Figure17Threeorthogonalviewingdirections........................40 Figure18Anundistortedimage................................42 Figure19Asilhouettegenerationalgorithm.........................43 Figure20Imagesofsimulatedobjectsandtheircorrespondingsilhouettes.........44 Figure21Animageofanobjectofinterestascapturedbytherobot.............45 Figure22HSVcolorspace..................................46 Figure23Hueplanethresholding...............................47 Figure24Hueandsaturationthresholding..........................48 Figure25Seedpointdetermination..............................48 Figure26Floodllingredpixels...............................49 Figure27Holelling.....................................50 iv

PAGE 6

Figure28Centroidlocalization................................53 Figure29Thegeometricinterpretationofthesilhouetteradius r max .............54 Figure30Thegeometryforndingtheradiusoftheboundingsphere............55 Figure31Analgorithmforevenlyspacedpointsonasphere.................57 Figure32Asphereofpointsgeneratedaroundasimulatedobject..............58 Figure33Thegeometryofpoint x i new ............................59 Figure34Theresultsofsurfaceapproximation........................60 Figure35A2Dillustrationofperspectiveprojectionerror..................61 Figure36Anillustrationofthevisualhullconcept......................61 Figure37Anexampleofextremeperspectiveprojectionerror................62 Figure38Overestimationinthepresenceofperspectiveprojectionerror..........66 Figure39Theeffectsofthemodiedcostfunction......................66 Figure40Thereconstructionprocessasastepbystepsimulation..............70 Figure41Ascreenshotofthegraphicalsimulator......................72 Figure42Thereconstructionofsimulatedobjects......................76 Figure43Robotsetupwithcamera..............................78 Figure44Selfcenteringrobotendeffector..........................79 Figure45Thecameravoltageregulatorschematic......................80 Figure46Thecameramountedinthegripper.........................80 Figure47Networkandcommunicationlayout........................82 Figure48Threeimagescapturedbytherobotduringatestrun...............83 Figure49Thefourreal-worldtestobjects...........................84 Figure50Thereconstructionofthebatterybox........................86 Figure51Thereconstructionofthestackoftwocups....................87 Figure52Thereconstructionoftheyarnball.........................87 Figure53Thereconstructionofthecardinalstatue......................88 v

PAGE 7

ShapeandPoseRecoveryofNovelObjectsUsingThreeImagesfromaMonocularCamera inanEye-in-HandConguration StevenC.Colbert ABSTRACT Knowingtheshapeandposeofobjectsofinterestiscriticalinformationwhenplanningrobotic graspingandmanipulationmaneuvers.Theabilitytorecoverthisinformationfromobjectsfor whichthesystemhasnopriorknowledgeisavaluablebehaviorforanautonomousorsemiautonomousrobot.Thisworkdevelopsandpresentsanalgorithmfortheshapeandposerecovery ofunknownobjectsusingno apriori information.Usingamonocularcamerainaneye-in-hand conguration,threeimagesoftheobjectofinterestarecapturedfromthreedisparateviewingdirections.Machinevisiontechniquesareemployedtoprocesstheseimagesintosilhouettes.The silhouettesareusedtogenerateanapproximationofthesurfaceoftheobjectintheformofathree dimensionalpointcloud.Theaccuracyofthisapproximationisimprovedbyttinganelevenparametergeometricshapetothepointssuchthatthettedshapeignoresdisturbancesfromnoiseand perspectiveprojectioneffects.Theparametrizedshaperepresentsthemodeloftheunknownobject andcanbeutilizedforplanningrobotgraspingmaneuversorotherobjectclassicationtasks. Thisworkisimplementedandtestedinsimulationandhardware.Asimulatorisdevelopedto testthealgorithmforvariousthreedimensionalshapesandanypossibleimagingpositions.Several shapesandviewingcongurationsaretestedandtheaccuracyoftherecoveriesarereportedand analyzed.Afterthoroughtestingofthealgorithminsimulation,itisimplementedonasixaxisindustrialmanipulatorandtestedonarangeofrealworldobjects:bothgeometricandamorphous.Itis shownthattheaccuracyofthehardwareimplementationperformsexceedinglywellandapproaches theaccuracyofthesimulator,despitetheadditionalsourcesoferroranduncertaintypresent. vi

PAGE 8

Chapter1 Introduction 1.1Motivation Autonomousroboticmanipulatorsareubiquitousinmanyareasofindustry,andtherolethesemachinesplayinautomatedassemblylinesandproductionandpackagingfacilitiesaroundtheworldis wellknown.Theserobots,whilegreatinnumber,operateinrelativelyrestrictedenvironments;the tasktobecompletediswelldenedandthetypicalrobotwillexecutethesamemotionrepeatedly andtirelessly. Thereisanotherclassofrobots,however,whichseektooperateinunstructuredenvironments; thetaskstobecompletedaretypicallynotknownbeforehand,andtheenvironmentwithinwhich therobotoperatesisoftenunknown.Thesetypesofrobotsaretypicallydesignedtobeserviceor assistanceoriented.Incontrasttotheirindustrialcounterparts,theunstructuredenvironmentplaces aheavyrequirementontheserobotstocollectandinterpretdatafromtheirsurroundingssuchthat theycanintelligentlyandsuccessfullycompleteatask.Oneofthemostfundamentaloftasksfor anassistiverobotistograspandpickuporotherwisemanipulateaparticularobjectofinterest.It isthisactionthatthisworkseekstofacilitate. Inorderforarobottomanipulateanobjectofinterest,severalpiecesofinformationmustbe knownaboutthatobjectbeforetheactioncancommence.Chiey,theobjects'geometrymustbe fullydenedtoasufcientapproximationrelativetotherobot.Thatistosay,thelocation,orientation,shapeandsizeoftheobjectmustbeknowninordertoplantheappropriateroboticmotions. Givenanunstructuredenvironment,thispredicatestherobothavingtheabilitytointerrogateits surroundingsandrecoverthenecessaryinformationoftheobjecttobemanipulated. Therearetwoapproachestogatheringthisinformation:thosethatusepriorknowledgeofthe objects,andthosethatdonot.Undertherstapproach,therobotisprogrammedinadvancewith detailedknowledgeofseveralobjectsitislikelytoencounterduringoperation.Thisknowledgemay 1

PAGE 9

takeseveralformsdependingonthesensorstherobotemploys:camera,sonar,laserrangenders, andothers.Intheend,therobotisprovidedwithadatabaseofobjectsitiscapableofmanipulating apriori .Whentherobotthenencountersanewobjectinservice,itattemptstomatchthefeaturesof thisnewobjecttooneofitsdatabasemodels.Whenitndsthebestmatch,itbehavesaccordingly. Whilethismethodofmodelbasedoperationhasprovenaccurate,itcarrieswithitthefundamental limitationofbeingunabletohandleobjectswhichitcan'tidentifyorwhichdonottwithinthe descriptionofoneitspre-denedmodels.Inahouseholdsitutation,whereanassistiverobotwill betaskedwithmanipulatingaplethoraofvariousobjects,thislimitationbecomesasignicant hindrance. Thesecondapproachtorecoveringtherequiredinformationoftheobjectistoreconstructthe geometryoftheobjectatruntimebasedonthesensorydataavailabletotherobot:usuallyinthe formofimagesorrangedata.Stereoimagingisextremelypopularforthispurpose,however,it hasthedrawbackofyieldinginaccurateresultswhentheobjectofinteresthaslittletonotexture;a casethatisoftenfoundedwithhouseholdobjects.Anotherpopularmethodreconstructstheobjects' geometryfromseveralimagesoftheobjectcapturedfromvariousdisparatelocations.Thismethod, termed`shapefromsilhouettes`hastheadvantageoffunctioningeffectivelyforobjectswithor withouttexture,butwiththedrawbackofrequiringanexcessivelylargeamountofimagesofthe object,oftenrenderingtheprocedureimpracticalforunstructuredenvironments. Thisworkseekstomakeprogressintheruntimereconstructionofthegeometryofunknown objectsbydevelopingapracticalshapefromsilhouettesalgorithmthatcanachieveasufciently accuratereconstructionwithaminimalnumberofimages. 1.2Objectives Theprincipleobjectiveofthisworkistodevelopandimplementanalgorithmthatiscapableof reconstructing,toasufcientdegreeofaccuracy,thethreedimensionalshape,pose,andorientation ofanovelobjectusingamonocularcameramountedontheend-effectorofaroboticmanipulator forthepurposesofgraspandmanipulationplanning.Thespecicgoalsofthealgorithmarelisted asfollows: Reconstructtheshape,pose,andorientationofanovelobjecttoasufcientdegreeofaccuracy thatpermitstheplanningandexecutionofgraspingandmanipulationmaneuvers. 2

PAGE 10

Calculatetheseparametersusingno apriori informationoftheobject,withtheexceptionthat agivenobjectisidentiedastheobjectofinterest. Requireonlyaminimalnumberofimagesforreconstruction;signicantlyfewerthanthestatus quo. Operateefciently,suchthatcalculationtimeoftheobjectparametersisnegligiblecompared tothetimerequiredfortherobottocapturetheimages. 1.3Outline Therestofthisworkisdividedintosixchapters.Chapter2providesashortliteraturereviewconcerningthestateoftheartofobjectreconstructionandcoversthemethodsthataremodelbasedas wellasruntimebased.Chapter3providestherelevantmathematicalandscienticbackgroundfor understandingthealgorithmdevelopment.Topicssuchascoordinatetransformations,cameramodels,machinevisiontechniques,andsuperquadricsarecoveredinthischapter.Chapter4presents thedevelopmentofthereconstructionalgorithm.Chapter5detailsthesoftwaresimulationenvironmentaswellastheresultsoftestingthedevelopedalgorithminsimulation.Chapter6concernsthe developmentofthehardwaresystemaswellastheresultsoftestingthealgorithmonthehardware. Chapter7providesaclosingdiscussionaswelltheplanforfutureworkanddevelopmentofthe algorithm. 3

PAGE 11

Chapter2 LiteratureReview Therehasbeenmuchrecentresearchinapplyingvisualsensorydatatothecontrolofarobotic manipulatorinunstructuredenvironmentsfortheexpressedpurposeofgraspingorotherwisemanipulatingobjects.Muchoftheworkmakesusesofsomeformofpredenedobjectmodels,which limitsthepracticalityintermsofbroadadoptionandadaptabilitytoneworunknownenvironments. Therehasalsobeenresearchanddevelopmentsinshapeandobjectreconstructionthatisnotmodel orpriorknowledgebased,thoughtheamountofresearchandprogressinthisareaiscomparatively small. Astheresearchpresentedinthisthesismakesuseofdevelopmentsfromboththemodeland non-modelbasedareasofresearch,recentdevelopmentsinbothoftheseareasarepresentedinthis chapter. 2.1ModelBasedRecovery Kim,Lovelett,andBehal[11]havedevelopedawheelchairmountedroboticarmthatcangrasp andmanipulatetexturedobjectsassociatedwithActivitiesofDailyLivingADLs.Thesystemuses anarrowbaselinestereocameraandalargedatabaseoftemplateimages.Oncetheuseridenties theobjectofinterestonthevideoscreen,thesystemutilizesthestereocameratoperformasparse stereoreconstructionbasedonSIFTfeatures[9].Thisroughapproximationisusedtoconvergeon theobjecttoacloseproximity,atwhichpointnewSIFTfeaturesaregeneratedandmatchedtoa templateinthedatabase.Thenalgraspingisexecutedviavisualservoingtoorientthemanipulator intheposedictatedbythetemplate.Theauthorsovercomesomeoftheprominentchallengesin planninggraspingmaneuversinunstructuredenvironmentsbyseparatingthemotionintogrossand nesegments.Thisallowsthemtoutilizetheconvenienceofastereocamerafromadistance,while relegatingtheprecisepositioningtoamorerobustmethod.However,therelianceonatemplateto 4

PAGE 12

nalizetherobotpositioninglimitsthesystemtointeractingonlywiththoseobjectsdenedinits database. Efndi,Jarvis,andSuter[39]developedamethodfortablemountedSCARArobottoidentify andgrasppreviouslydenedobjectsusingastereocamera.Thesceneisobservedbytherobot mountedcameraandadisparitymapisgenerated.Basedonthedisparitymap,theobjectsinthe scenearesegmentfromthebackground.Thesegmentedobjectsarematchedagainstimagesina databaseusingSIFTfeatures.Whenamatchisfound,theshapeoftheobjectisapproximatedby calculatingtheboundingboxthatencompassesthe3Dpointsthatweregeneratedfromthedisparity map.Theobjectisthengraspedusinganoverheadmaneuver.Whilethisworkhasaminimal relianceonpreviouslydenedmodels,itisrelegatedtoapproximatingtheshapeofanyobjectbya simpleboundingbox.Thoughasimpleboxmodelmayprovesufcientforacertainclassofobjects, itwillfailtocapturethenatureofmanycommonhouseholdobjectssuchascupsandotherrounded shapesandwilllikelycauselimitationsforanythingotherthanoverheadgraspingmaneuvers. Schlemmer,Biegelbauer,andVincze[31]combinetheshapeandappearanceofobjectsina systemdesignedforeventualusebyassistiverobotsinahouseholdsetting.Duringthetraining phase,thesystemisshownanobjectthatislatertobefoundinascene,andusingalaserrange nder,the3Drangedataisparametrizedbyabesttsuperquadric.Inthedetectionphase,thescene isscannedbythelaserrangender,andtherangedataanalysedtodetectanyshapesthatwere learnedinthetrainingphase.Ifamatchisfound,theobjectcanbegraspedbyvisualservoingusing SIFTfeaturepointsprovidedbyamonocularcamera.Thoughthesystemwasimplementedonlyon atestbench,andthepracticalityofusinga3Dlaserscannerinahouseholdsettingisdebatable,the workisfundamentalinthatitshowedthattheshapeofmosthouseholdobjectscanbeeffectively modeledbyasimpleandcompactsuperquadricrepresentation. Kragic,Bjrkman,Christensen,andEklundh[12]developedasystemthatusesmultiplesetsof stereocamerastoallowamanipulatortooperateinanunstructuredenvironmentusinga detectapproach-grasp paradigm.Thestereocamerasaremountedinbotheye-in-handandstationarycongurationsandservetosegmentthesceneandisolateforegroundobjectsforthedetectionphase. Variouskeypointrecognitionschemesareemployedusingthemonocularcues,andadatabaseof predenedobjectsissearchedforamatch.Ifamatchisfound,2.5Dvisualservoingisusedforthe approachandgraspphases.Ifa3DCADmodelisavailablefortheobject,thenthefullposeinfor5

PAGE 13

mationoftheobjectisestimatedandusedtoincreasethegraspingaccuracy.Theauthorsdescribe ascenariowheretheirsystemwouldbecapableofmanipulatinganunknownobjectprovidedthat certainassumptionsaremadeabouttheobject:namely,thatthedepthoftheobjectisnotsignicant.Thatis,theauthorsrecognizetheinabilityofstereocameratoreconstructthefull3Dgeometry oftheobjectfromasingleposition,andthatincaseswheretheobjectisunknown,imagesfrom multiplevantagepointswouldberequired. LiefhebberandSijs[16]proposedavisionbasedcontrolsystemforapopular,commercially availablewheelchairmountedroboticarm.Intheirsystem,imagesgatheredfromamonocular cameraareusedtocontroltheorientationoftherobotendeffectorwithrespecttotheobjectof interest.Theuserisresponsibleforcontrollingtheposition.SIFTfeaturesareextractedfrom theimageandmatchedtoanobjectinadatabaseandafeedbackvisualservoingalgorithmis employedtomaintainapredenedorientation.Inadditiontothedrawbackofrelyingonadatabase ofpredenedobjects,theuseofamonocularcamerainthiscaserequiresthatthe3Dmodelof eachobjectinthedatabasebedenedintermsofthe3DlocationoftheSIFTfeaturesrelativeto someobjectcoordinatesystem.Thisintroducesaratherlengthyandcomplicatedtrainingphase foranyobjectthatshouldbeincludedinthedatabase.Furthermore,SIFTfeaturesarenotoriously computationallyexpensivetoextractfromagivenimage. 2.2RuntimeReconstruction InaseriesofworksbyYamazakietal.[27,28,26],theauthorsdevelopedamobilerobotthatis capableofreconstructingtheshapeandposeofanovelobjectusingalmostno apriori knowledgeof theobject.Thereconstructionwasachievedbyhavingtherobotdrivecompletelyaroundtheobject andcapturingacontinuoussequenceofimagesasittravelled.Theseimageswereregisteredwith eachotherusingStructureFromMotionSFMtechniquesanddisparitymapswerecreated.From thedisparitymaps,adense3Dreconstructionwasperformedtoyieldthegeometryoftheobject. Finally,astatisticalanalysisofthedenserangedatawasperformedtodeterminetheidealgrasping pose.Anadvantageofthetechniquetheauthorsemployedisthatthepositionoftherobotneednot beknowntoahighdegreeofaccuracy.Thatis,odometryinformationfromtherobotisnotusedfor thereconstruction,andthecameraparametersaredetermineddirectlyfromthesequenceofimages. Thiseliminatesmanyissuesthatcanarisefromrobotinaccuracies.Thedownsidestothisapproach 6

PAGE 14

however,arethatitrequirestherobottohave360degreeaccessaroundtheperipheryoftheobject anditnecessitatesanexceedinglylargenumberofimageswhichresultsinhighercomputational costs.Further,sincetheresultinggeometryinformationissimplyacollectionof3Dpoints,any structuralinformationmustbeextractedusingstatisticalmethods,introducingmorecomputational requirements.Analdrawback,whichwasexhibitedinthelaterwork[26],isthatthelocationof theobjectneedstobeidentiedtothesysteminadvance. Yoshikawa,Koeda,andFujimoto[40]usedanomnidirectionalcameraandasoftngeredgripper onaroboticmanipulatortorecognizeandgraspunknownshapes.Theauthorstakeseveralimages fromaroundtheperipheryoftheobjectand,usingasimpliedversionofthevolumeintersection technique,calculatethe2Dvisualhulloftheobjectonahorizontalplane.Theyalsodevelopa simplegraspcriteriontodeterminetheoptimalgrapingpositionbasedonthe2Dvisualhull.The authorschosetosimplifytheshapereconstructiontoatwodimensionalproblemcitingtherealtime computationalconsiderationofreconstructingthefullthreedimensionalgeometry.This,ofcourse, relegatesthesystemtoonlybeingabletograspobjectsthatlieonapredenedhorizontalplane. Furthermore,sincethereconstructionislimitedtoa2Dplane,thesystemisunabletodistinguish objectsthathavevariationalongtheverticaldirection. Nguyenet.al.[20]developedamobilemanipulator,namedEl-E,fortheexplicitpurposeof retrievingunmodeled,everydayobjectsfromatsurfacesforpeoplewithmotordisabilities.The robotcanretrieveobjectsthatareidentiedbytheuserthroughthemeansofalaserpointer.An omnidirectionalcameradetectsthelaserpointerandinturn,pointsastereocamerarigatthepoint. The3Dlocationofthepointisdeterminedfromthestereoimagesandtherobotthendrivestoa speciedthresholddistancefromtheobject,usinga2Dlaserscanneronthebottomofthemobile basefornavigation.Onceattheobjectl,the2Dlaserscannerisused,incombinationwithae,to detecttheheightoftheatsurface.Theendeffectoroftherobotisthenorientedperpendicular totheatsurfaceandpositionedabovethelocationindicatedbytheuser'slaserpointer.Usinga monoculareye-in-handcamera,theobjectisvisuallydandtheend-effectororientedperpendicular totheaxisofleastinertia.Finally,theend-effectordescendsupontheobjectuntilinfraredsensorsin thegripperindicatethepresenceofanobjectoranimminentcollision.Theauthorsshowedhighly accurateresultsusingthissystemwithareportedoveralltaskaccuracyof86%.Itisworthnoting thattheshapeoftheobjectisneveractuallyfullyrecoveredwiththissystem.Itinsteadreliesonthe 7

PAGE 15

infraredsensorstodeterminethepresenceofanobjectwithinthegripper.Butgiventhereported accuracy,perhapstheonlydrawbackofthissystemisthelargevarietyofsensorsrequiredforits operation. Saxena,Driemeyer,andNg[3]approachedtheproblemofgraspingnovelobjectsslightlydifferently.Thesystemtheydevelopeddoesnotrequire,nortriestobuild,a3Dmodeloftheobject. Instead,thesystemdetermineslocationsina2Dimagewhichlikelyrepresentavalidgraspinglocationontheobject.Itthenndsthissamelocationinmultipleviewsandtriangulatesthelocation ofthegraspingposition.Thesystemistrainedtondvalidgraspinglocationsbybeingshownan initiallargetrainingsetofrenderedimagesofobjectswiththeidealgraspinglocationsidentied. Then,usingvariouscomputervisiontechniques,buildsaclassiertolateridentifysuchappropriateareasonnovelobjects.Theauthorsreportedhighaccuracy,intermsofgraspsuccessrate,for objectsrangingfromastaplertoanoddshapedpower-horn.Eventhoughthesystemrequiresbeingtrainedinadvance,theimagesinthetrainingsetarecompletelyunrelatedtothenovelobjects therobotislatercapableofgrasping.Thus,itisconsideredthatthissystemrequiresno apriori informationofthenovelobjectsitwishestomanipulate.However,trainingtheclassiertakesa considerableamountoftimethoughitisonlyrequiredonceandtheaccuracyofthesystemwas saidtoimproveasthenumberandqualityofthetrainingimagesimprove. 8

PAGE 16

Chapter3 FundamentalBackground 3.1CoordinateRepresentations Acornerstoneoftheunderlyingmaththatenables3Dreconstructionfrommultiple2Dimagesis therepresentationandinterpretationofpointsandlinesinspaceandhowtheyaretransformedfrom onecoordinatesystemtoanother.Considerthepoint P asdepictedinFigure1inthespace R 3 .In theCartesiancoordinateframe f A g ,thepointcanbedenedbythevector P = r 1 =x A ; y A ; z A T Likewise,thesamepointcanberepresentedinthecoordinateframe f B g : P = r 2 =x B ; y B ; z B T Clearly,both r 1 and r 2 describethesamepoint P inspace,sotheremustbeanoperationor mapping thatcanbeappliedtothedescriptionof P inframe f A g toyield P asdescribedinframe f B g and viceversa. Thissectionpresentsthenecessarymathematicalconstructsthatfacilitatesuchmappingsfrom onecoordinatespacetoanother.Inthegeneralcase,thesemappingsarenotlimitedtopurely Euclideansystemsbutratheranytwosystemsthatarerelatedbyageneralprojectivetransformation. However,suchgeneralcasesarenotdirectlyrelevanttothisworkandarenotpresented.Rather,this sectionfocusesoncoordinatesystemsthatdifferbyasimilaritytransformation.Thatis,aEuclidean transformationcombinedwithanisotropicscaling.Forafulltreatmentofprojectivegeometryin thegeneralcase,thereaderisdirectedto[34]. 3.1.1HomogeneousCoordinates Ahomogeneouscoordinateisanalternativerepresentationofapointisspacewhichisobtained byaddinganadditionalelementtothecoordinatetuple.Thatis,a2-elementvectorbecomes3elements,a3-elementvectorbecomes4-elements,andsoon.Anytwon-elementvectorsaresaid tobelongtothesameequivalenceclass,providedtheydifferbyacommonmultiple[34].Thatis,if a 1 ;a 2 ;:::;a n = k b 1 ;b 2 ;:::;b n thenboth a and b arehomogeneousrepresentationsofthevector 9

PAGE 17

Figure1. :Apointin R 3 .Thepoint P canbedenedintwoseparatecoordinateframes f A g and f B g c = c 1 ;c 2 ;:::;c n )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 ; 1= a 1 a n ; a 2 a n ;:::; a n a n = b 1 b n ; b 2 b n ;:::; b n b n .Thisrepresentationisextremely usefulwhenmappingcoordinatesfromonesystemtoanotherandisusedextensivelyinthiswork. Thusthederivationandproofofthehomogeneousrepresentationfortwoandthreedimensionsis giveninthefollowingsections. 3.1.1.1TwoDimensions Ina2-Dplaneinthespace R 2 ,apoint P canberepresentedasatwoelementvector P =x ; y T Furthermore,theimplicitequationofalineonthatplaneisgivenby ax + by + c =0 .1 whereanypoint P thatsatisestheequationliesontheline.Thus,wecanrepresentaline L on thetwodimensionalplaneasathreeelementvector L =a ; b ; c T .Now,wecanrewriteEquation 3.1asaninnerproductoftwovectors x;y; 1 a;b;c T =0 andthus x;y; 1 L =0 .But,since thepoint P liesontheline L ifandonlyifitsatisesEquation3.1,then x;y; 1 T mustalsobean equivalentandvalidrepresentationforthepoint P Ifanon-zeroconstant k isintroducedintoEquation3.1,thenweseethattheequivalentequation k x;y; 1 L =0 willstillholdprovidedthatpoint P liesontheline L .Thusingeneral,thethree 10

PAGE 18

elementvector kx;ky;k T isavalidrepresentationforthepoint P = x;y T foranynon-zero valueof k .Thisthree-elementformisthehomogeneousrepresentationofthe2-Dpoint,andisa many-to-onerepresentation.Givenahomogeneousrepresentationofa2-Dpoint X = x;y;w T onedeterminesthecoordinatesofthatpointin R 2 as P = x w ; y w T .Conversely,givenapoint P = x;y T in R 2 ,oneeasilyconstructsahomogeneousrepresentationas X = x;y; 1 T 3.1.1.2ThreeDimensions Homogeneouscoordinatesinthreedimensionsaredevelopedinmuchthesamefashionastwo dimensions.Inthespace R 3 ,atwodimensionalplaneisdenedbytheimplicitequation 1 x + 2 y + 3 z + 4 =0 .2 andthusaplanecanberepresentedasafour-elementvector = 1 ; 2 ; 3 ; 4 T ,andanypoint P = x;y;z T thatsatisesEquation3.2liesontheplane .Likethe2-Dcase,wecanrepresent Equation3.2astheinnerproductoftwovectors x;y;z; 1 1 ; 2 ; 3 ; 4 T = x;y;z; 1 =0 .It isreadilyseenthat x;y;z; 1 T mustalsobeavalidrepresentationfor P .Introducinganon-zero constant k intoEquation3.2yieldstheequivalentequation k x;y;z; 1 =0 ,whichholdsprovided thepointliesontheplane.Thusingeneral, kx;ky;kz;k T isavalidrepresentationofthepoint P andisthehomogeneousrepresentationofapointin R 3 Giventhehomogeneouscoordinatesofapoint X = x;y;z;w T ,onedeterminesthecoordinates ofthepointin R 3 as P = x w ; y w ; z w T .Givenapoint P = x;y;z T in R 3 ,ahomogeneous representationcanbecreatedas X = x;y;z; 1 T 3.1.2CoordinateTransformations Anoteonconventions: Inwhatfollows,weadoptthenotationconventionpresentedin[24]to representthevariousentitiesinvolvedincoordinatetransformationrepresentation.Briey: Braces fg indicatedaCartesianframeofframeofreference. f A g representsCartesianframe seeFigure1. Aleftsuperscriptindicatestheframeinwhichthequantityisdened. A r isthevector r dened inframe f A g 11

PAGE 19

Thehat^indicatesaunitvector. ^ x isaunitvectorinthe x direction. Arightsubscriptindicatesthattheentitybelongstoacertainframe. ^ x A istheunitvectorinthe x directionoftheframe f A g Aleftsubscriptandsuperscriptindicatesthequantityisdenedrelativebetweentwoframes. A B R isthequantity R andisdescribedinframe f B g relativetoframe f A g Vectors,withtheexceptionofunitvectors,areboldfacedandconsideredascolumnvectors. Thus,thefollowingareallequivalent P = P 1 ;P 2 ;P 3 T = 2 6 6 6 4 P 1 P 2 P 3 3 7 7 7 5 Matricesarerepresentedasboldfaced. Givenapoint P inspace R 3 denedinsomecoordinatesystem f B g ,wewishtobeabletoexpress thispointintermsofanyotherarbitrarycoordinatesystem f A g seeFigure1.Fornow,weassume thatcoordinateframes f A g and f B g arerelatedbyaEuclideantransformation T .Thatis,theyare relatedbyanarbitrarytranslationandrotation,butthescaleremainsconstantinalldirections.Then, thepoint P expressedinthecoordinateframe f B g canbedenedas B P = B A T A P .Inorderto ndavalidtransformationfor B A T ,webreakdownthetransformationintoitsindividualtranslation androtationcomponentsandthenshowhowthetwocomponentscanbecombinedintothesingle transformation. Figure2showstwocoordinateframes f A g and f B g thatareseparatedbyapuretranslation.Ifwe imaginethatbothcoordinateframesexistinthesameencompassingglobalframe,thentheorigins ofeachcoordinatecanberepresentedbythevectors P A ORG and P B ORG andthetranslationof f B g withrespectto f A g isthevector A B t = P B ORG )]TJ/F43 10.9091 Tf 10.843 0 Td [(P A ORG .Thepoint B p isthepoint p expressedin thecoordinatesofframe f B g ,andfromvectoradditionweseethat A p = P B ORG )]TJ/F43 10.9091 Tf 10.957 0 Td [(P A ORG + B p So,wecanrepresentthetranslationportionoftheEuclideantransformationas A p = A B t + B p .3 12

PAGE 20

Figure2. :Twocoordinateframesseparatedbypuretranslation. Nowconsiderthecasewheretheoriginoftwocoordinateframesarecoincident,butthereexists anarbitraryrotationaboutsomeaxisbetweentheframes.Figure3showsthisscenarioforthe twocoordinateframes f A g and f B g .Now,theprojectionofonevectorontoanothervectoris accomplishedthroughdotproductofthetwovectors.Sowecanrepresenttheunitvectorsofthe frame f B g expressedintermsoftheunitvectorsofframe f A g accordingtotheequation h A ^ x B A ^ y B A ^ z B i = 2 6 6 6 4 ^ x B ^ x A ^ y B ^ x A ^ z B ^ x A ^ x B ^ y A ^ y B ^ y A ^ z B ^ y A ^ x B ^ z A ^ y B ^ z A ^ z B ^ z A 3 7 7 7 5 = 2 6 6 6 4 B ^ x T A B ^ y T A B ^ z T A 3 7 7 7 5 .4 Now,ifgivenapoint B p andonewantstondtherepresentationofthepoint A p ,wejustproject thepoint B p usingadotproductinthesamemannerasEquation3.4, A p x = B ^ x A B p A p y = B ^ y A B p A p z = B ^ z A B p .5 Ifweletthe3x3matrixbetermedarotationmatrix R ,thenitcanbeseenthattransforming B p into A p canbeeasilyaccomplishedwiththematrixproduct 13

PAGE 21

Figure3. :Twocoordinateframesseparatedbypurerotation. A p = A B R B p .6 where A B R isdenedusingEquation3.4 A B R = h A ^ x B A ^ y B A ^ z B i = 2 6 6 6 4 r 11 r 12 r 13 r 21 r 22 r 23 r 31 r 32 r 33 3 7 7 7 5 .7 andthusEquation3.6representsthetransformationequationfortwoframesseparatedbyapure rotation. Wenowconsiderthegeneralcasewheretwoframes f A g and f B g areseparatedbybothageneraltranslationandorientation.Figure4illustratestheconcept.Sincethetwocomponentsofthe transformationareseparable,theycanbecombinedthroughtheprincipleofsuperposition.Think ofthisascreatinganewtemporarycoordinateframewiththeoriginat P B ORG butwithanorientationthatisalignedwith f A g .Now,inordertotransform B p into A p ,werstrotate B p intothis newtemporaryframe.Butsincethetemporaryframehasthesameorientationas f A g thiscanbe accomplishedwiththerotationmatrix A B R .Thelaststepistotranslatetheresultinto f A g viathe translationvector A B t andsothefullequationbecomes A p = A B R B p + A B t .8 14

PAGE 22

WewishtoexpressEquation3.8intheform A p = A B T B p andthiscanaccomplishedbyrearrangingtheequationasamatrixproductas A p = A B R j A B t 000 j 1 B p = 2 6 6 6 6 6 6 4 r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z 0001 3 7 7 7 7 7 7 5 2 6 6 6 6 6 6 4 p x p y p z 1 3 7 7 7 7 7 7 5 .9 where A B T = 2 6 6 6 6 6 6 4 r 11 r 12 r 13 t x r 21 r 22 r 23 t y r 31 r 32 r 33 t z 0001 3 7 7 7 7 7 7 5 .10 InthecourseofrearrangingEquation3.8intoEquation3.9,wehaveconvertedthepoints A p and B p intotheirhomogeneousrepresentation.Byusingthisrepresentation,weareabletorepresent theentiretransformationrotationandtranslationasasinglematrixmultiplication.Forthisreason, A B T iscalleda homogeneoustransformationmatrix .Further,bydeningthefourthrowof A B T as wehave, A B T isnowa4x4matrixwhichcanbeinvertedinasimplemanner.Theinverseofa homogeneoustransformationmatrixcanbeviewedasaninversemapping.Thatis, B p = B A T A p = )]TJ/F21 7.9701 Tf 5 -4.332 Td [(A B T )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 A p .11 3.1.33Dto2DProjection Inordertomoreclearlyrepresent3Dto2Dprojection,westartwiththesimplercaseof2Dto 1Dprojection.Figure5showsa2Dplanewithorigin ; 0 andpoint x;y onthatplane.Further, thereisline L intheplanelocatedadistance d fromtheoriginandorientedparallelto y axis.We wishtondtheprojectionofthepoint x;y ontotheline L whichweindicatedaspoint P .From thegure,itcanbeseenthatpoint P isequalto x 0 andcanbecalculatedviasimilartriangles 15

PAGE 23

Figure4. :Twocoordinateframesseparatedbyageneraltranslationandrotation. y x = d x 0 x 0 = dx y .12 Now,wecanexpressEquation3.12inthehomogeneousformbyconvertingpoint P intoageneral homogeneousrepresentationandallowingtherighthandsideoftheequationtobecomeamatrix multiplication. P = 2 4 kp k 3 5 = 2 4 d 0 01 3 5 2 4 x y 3 5 .13 RecallfromSection3.1.1thatinordertoretrievethecanonicalrepresentationofapointinaspace R n ,simplydividethroughbythelastelementinthevector.Thuswehave P = 2 4 kp k 3 5 = 2 4 dx y 3 5 P = 2 4 p 1 3 5 = 2 4 dx y 1 3 5 P = dx y = x 0 .14 whichisinagreementwithEquation3.12. 16

PAGE 24

Figure5. :2Dto1Dprojection. Thecaseof3Dto2Dprojectionisanaturalextensionofthe2Dto1Dcasejustdiscussedand theresultisnowpresentedwithoutderivation.Givenapoint x;y;z T inthespace R 3 anda2D planelocatedadistance d fromtheoriginasmeasuredalongthe z axis,then,providedthatthe planeisperpendiculartothe z axisandtheoriginoftheplaneisthepoint ; 0 ;d T theprojection of x;y;z T intotheplaneatpoint P isgivenas P = 2 6 6 6 4 wP x wP y w 3 7 7 7 5 = 2 6 6 6 4 d 00 0 d 0 001 3 7 7 7 5 2 6 6 6 4 x y z 3 7 7 7 5 .15 Equation3.15isthehomogeneousprojectionofa3Dcoordinateontoa2Dplane.Ifwemultiply thepoint x;y;z T byanon-zeroconstant k ,weseethattheequationwillstillbevalidasthe constant k willbeabsorbedinto w .Thus,Equation3.15isamany-to-onemappinginthatanypoint lyingalongthelinedenedbytheoriginandthepoint x;y;z T willprojecttothesamepointon theplane. 3.1.42Dto3DProjection Asitwasshownthata3Dto2Dprojectionisamany-to-onemapping,thereisnogeneralway fromasingle2Dprojectiontorecovertheparticular3Dpointthatcausedtheprojection.Infact, thenumberofpossiblepointsthatcangenerateaunique2Dprojectionisinnite.However,thereis 17

PAGE 25

somethingalloftheseinnitepointshaveincommon:theyallliealongthesameline.Thiscanbe shownbyinvertingequation3.15asfollows 2 6 6 6 4 1 d 00 0 1 d 0 001 3 7 7 7 5 2 6 6 6 4 wP x wP y w 3 7 7 7 5 = 2 6 6 6 4 1 d 00 0 1 d 0 001 3 7 7 7 5 2 6 6 6 4 d 00 0 d 0 001 3 7 7 7 5 2 6 6 6 4 x y z 3 7 7 7 5 2 6 6 6 4 wP x d wP y d w 3 7 7 7 5 = 2 6 6 6 4 x y z 3 7 7 7 5 .16 butsince w cantakeonanyvalueandstillbeavalidhomogeneousrepresentationof P ,theleft handsideoftheequationdoesnotrepresenttheoriginalpoint x;y;z T ,butratheravectorfrom theoriginthatpassesthroughthatpoint,i.e.alineinthespace R 3 .Henceweneedtoaugmentthe righthandsideofEquation3.16withanon-zeroconstant k .Theappropriateformis 2 6 6 6 4 wP x d wP y d w 3 7 7 7 5 = k 2 6 6 6 4 x y z 3 7 7 7 5 .17 Thus,giventhelocationoftheplaneinspace,andapointonthatplane,weareabletodetermine thelinethatpassesthroughthatpointintheplaneandtheoriginofthespace,anoperationcalled backprojection.Backprojectionisafundamentaltoolforthereconstructionof3Dobjectsfrom multiple2DimagesandisusedextensivelyinSection4.3. 3.2MachineVision Accordingto[36],thegoalofmachinevisionistoextractusefulinformationaboutascenebased onitstwodimensionalprojections.Animageisa2Dprojectionofascene,andassuchdoesnot containanydirectlyavailablethreedimensionalinformationofthescene.Theeldofmachine visionisconcernedwiththemethodsandalgorithmsthatenabletherecoveryofinformationfrom 2Dprojections.Thisinformationcanbecombinedwiththetheoryofprojectivegeometrycovered intheprevioussectionstoenablethe3Dreconstructionofscenes. 18

PAGE 26

Broadly,theentiregoalofthisthesisistoextractthreedimensionalinformationaboutobjectsin ascenefrommultiple2Dimagesofthatscene.Inthatrespect,theentireworkfallsundertheguise ofmachinevision.Theaimofthissectionhowever,isnottoprovideanexhaustivebackground onmachinevision.Rather,itpresentsonlyafewofthekeyconceptsandmethodsfrommachine visionthatareusedaspartoftheoverallalgorithm.Theseconceptsaredescribedinabroadfashion withpictorialexamples;noalgorithmsfortheirimplementationaregiven.Theinterestedreaderis directedtooneofmanyreferencetextswhichderivethealgorithmsindetail:[36,30,35].Allof themachinevisionalgorithmsdescribedinthefollowingsectionsarefreelyavailableinmanyOpen SourcecomputervisionsoftwarelibrariessuchastheOpenCVlibrary[18]. 3.2.1CameraModels Therststepinmachinevisionisunderstandingthemathematicsbywhichacameraconvertsa threedimensionalsceneintoatwodimensionalimage.Thoughthereareawidevarietyofcamera models,thisworkfocusesonthemodelswhicharerelatedtocommonCCDtypedigitalcameras. Therstcameramodeldescribedistheidealcameramodel.Thesecondmodelisanextensionof therstandallowsforthenon-idealdistortionsandothereffectsthatarisewithCCDcameras. 3.2.1.1IdealCameraModel TheidealcameramodelisillustratedinFigure6.Inthisgure,thecenterofprojectionor focal point ofthecameraisrepresentedbythepoint O .Thecoordinatesystemofthecameraisoriented sothatthecameraislookingdowntheZaxis;anaxisalsoreferredtoasthe principlecameraray Theprinciplecamerarayisperpendiculartoandintersectstheimagingplaneatapoint O 0 .The imagingplaneislocatedadistance f fromthefocalpoint;adistancetermedthe focallength ofthe camera. Animageofapoint P inspaceisimagedtothepoint P 0 atthelocationofintersectionofimage planeatthelineconnectingpoints P and O .Thispositioncanbecalculatedanalyticallythrough theuseofsimilartriangles.Byinspectionofthegureitisreadilyseenthat 19

PAGE 27

Figure6. :Theidealcameramodel.Thepositionofthepoint p 0 intheimageplanecanbedeterminedfromthepositionofthepoint p andthefocallengthofthecamera f bymeansofsimilar triangles. P 0 = )]TJ/F20 10.9091 Tf 5 -8.836 Td [(x 0 ;y 0 T .18 x 0 = f x z .19 y 0 = f y z .20 orusinghomogeneousmatrixnotation P 0 = 2 6 6 6 4 f 00 0 f 0 001 3 7 7 7 5 2 6 6 6 4 x y z 3 7 7 7 5 = MP .21 The3x3matrix M isthecamera intrinsicsmatrix andforthisidealcameraisdenedbythe singleparameter f .Forthistypeofcamera,allobjectsinasceneprojectontotheimageplanein focusandthesizeoftheimagedobjectisrelativeonlytoitsdistancefromthecameraandcamera's focallength.TheastutereaderwillnoticethatEquation3.21isequivalenttoEquation3.15. 20

PAGE 28

3.2.1.2CCDCameraModel WhenconsideringimagesthatareformedbytheactionofaCCDcamera,theidealcameramodel oftheprevioussectionmustbeextendedtoaccountforsomespecicdeviationsanddistortions. Therstdeviationfromtheidealmodelcomesfromtherepresentationoftheoriginoftheimage plane.Intheidealmodel,theoriginofimageplaneistheintersectionoftheprinciplecameraray withtheimageplane,butinaCCDimagetheoriginistakenastheupperleftcorneroftheimaging sensor.Thus,themodelmustaccountforanoffsetinthe x and y direction.Thisisaccomplished withtheconstants c x and c y whichbecomepartofthecameraintrinsicsmatrix M = 2 6 6 6 4 f 0 c x 0 fc y 001 3 7 7 7 5 .22 TheseconddeviationofaCCDcamerafromtheidealmodelisthatofdifferentfocallengths inthe x and y directions.Thisconditioncanarise,forexample,withcheaporpoorqualityCCD sensorswherethepixelelementsarerectangularratherthansquare.Allowingforthismodication, thenalformofthecameraintrinsicsmatrixforatypicalCCDcamerabecomes M = 2 6 6 6 4 f x 0 c x 0 f y c y 001 3 7 7 7 5 .23 wheretheunitsof f x ;f y arepixelsperunitdistanceandtheunitsof c x ;c y arepixels. WhiletheintrinsicsmatrixofEquation3.23takesintoaccountnon-squarepixelsandtheoffset oftheimage,itisassumingthelensofthecameraisperfect.Inreality,mostlensesarenotperfect andintroducesomeamountofdistortion.Theprimarytypeofdistortioniscalledradialorbarrel distortion,andtheeffectcanbeseeninFigure7.Fromthegure,itisclearthatlinesthatshould bestraightarewarpedintocurves.Theamountofwarpingincreaseswithdistancefromtheimage center.Theothermaintypeofdistortionthatcanbeintroducediscalledtangentialdistortionand ariseswhenthecamera'simagingsensorisnotperfectlyparalleltothecameralens[17].Though therearemanyotherformsofdistortionpossibleinacamera,theradialandtangentialarethemost commonandhavethemostsignicanteffect[17,34]. 21

PAGE 29

Figure7. :Anexampleofradialdistortion.Noticehowlinesthatshouldbestraightbecomemore distortedthefartherawaytheyarefromtheimagecenter. Itisbeyondthescopeofthisworktoderivetheparametersfortheradialandtangentialdistortion, buttheyarepresentedhere,withoutderivation,asgivenin[17].Letthepoint x d ;y d T betheimage locationofthepoint x w ;y w ;z w T afterprojectionbyacamerawithradialandtangentialdistortion. Then,theundistortedpositionofthepoint x p ;y p T isgivenas x p = )]TJ/F15 10.9091 Tf 5 -8.837 Td [(1+ k 1 r 2 + k 2 r 4 + k 3 r 6 x d + )]TJ/F15 10.9091 Tf 5 -8.837 Td [(2 p 1 x d y d + p 2 r 2 +2 x 2 d .24 y p = )]TJ/F15 10.9091 Tf 5 -8.836 Td [(1+ k 1 r 2 + k 2 r 4 + k 3 r 6 y d + )]TJ/F15 10.9091 Tf 5 -8.836 Td [(2 p 2 x d y d + p 1 r 2 +2 y 2 d .25 Theparameters k 1 ;k 2 ;k 3 representtheradialdistortionandtheparameters p 1 ;p 2 representthe tangentialdistortion.Thevalue r isthedistanceofthepoint x d ;y d T fromtheimagecenter.The point x p ;y p T istheundistortedimagepointasifthepoint x w ;y w ;z w T wereprojectedbya distortion-freecamerausingtheintrinsicsmatrix M ofEquation3.23. 3.2.2DigitalImageRepresentation Beforediscussinganyofthealgorithmsofmachinevision,itiscriticaltounderstandhowanimageisrepresentedinacomputer.Whenhumans see animage,theyimmediatelyrecognizefeatures suchas:color,shapes,foreground,background,faces,etc.Theydonotseeapieceofbrouspaper withmulticoloredinktediouslyandpreciselyapplied,nordotheyseeanextremelydensearrayof 22

PAGE 30

liquidcrystalelementsthatdisplayonlyoneofthreecolors.Forhumans,theprocessofconvertingthisrawsensorydataintowhatisperceivedasanimageissoeffortless,weoftenoverlookthe enormouscomplexityofthevisualprocess. Machineshaveamuchmoredifculttimerecognizingfeaturesinanimage.Toamachine, animagelikeallothermachinedataispurelyasequenceofnumbers.Atthelowestlevel,the sequenceiscomposedofpurely1'sand0's,otherwiseknownasbinarydata.Atonelevelof abstractionhigher,animagecanberepresentedbysequenceofintegervaluesthatrepresentthe intensityoftheimageatacertainlocations.Otherrepresentationsthatarenotintegervaluedare alsousedtorepresentintensityvalues,however,theintegerrepresentationisthemostcommonly usedandunlessotherwisenoted,iswhatisusedthroughoutthiswork. Sincethemachinerepresentationofanimageisasequenceofdiscretenumbers,or digits ,the machinerepresentationisoftentermeda digitalimage .Considerthesmallrepresentativedigital imageshowninFigure8.Thisimagehassevenrowsandsevencolumnsforatotalof49pixels.It isalsoseenthattheimageiscomposedofvedifferentintensitylevelsrangingfromblacktowhite. Inatypicaldigitalimage,eachpixelisstoredinthecomputer'smemoryasanunsignedinteger representedby8bits.Theminimumvaluethisintegercanhaveis0binarycode00000000and themaximumis255binarycode11111111foratotalof256possibleintensitylevels.Ingeneral, avalueof0correspondstopureblackandavalueof255correspondstopurewhite;othervalues arevariousshadesofgray.InthecaseofFigure8,thearrayofintensitylevelsmaylooksomething like Image = 2 6 6 6 6 6 6 6 6 6 6 6 6 6 6 6 4 255255200200200255255 255200150150150200255 200150808080150200 20015080080150200 200150808080150200 255200150150150200255 255255200200200255255 3 7 7 7 7 7 7 7 7 7 7 7 7 7 7 7 5 .26 Thediscretenatureofthedigitalimagerepresentationpresentssomeimmediateshortcomings. Inparticular,twoquantitiessufferlossesduringthediscretizationprocess:intensityandshape.As previouslymentioned,thetypical8-bitunsignedintegerrepresentationofapixelisonlycapableof 23

PAGE 31

Figure8. :Anrepresentativeexampleofadigitalimage. representing256discretelevelsofintensity.Therefore,ifthedifferenceinlightintensitydetected fromanobjectinscenebyneighboringpixelsisbelowsomethreshold,thenthetwopixelswillbe digitizedtothesamevalue,eventhoughtheremayhavebeenanintensitydifferencepresent.In thecaseofshaperepresentation,thesquarenatureofpixelsrendersthedigitalimageincapableof perfectlyrepresentingacurvedshape.ThisiseasilyseeninFigure8,whichrepresentsthebest possibleapproximationofacirclewithina7x7grid.Withsuchacoarseresolution,itisclearlynot anaccuraterepresentation.Thislimitationcanbesomewhatovercomebyincreasingtheresolution ofthedigitalimage.A1280x1280pixelimage,forexample,willbebetterabletoapproximatea circlethana7x7image.Inpractice,suchhighresolutionimagesareoftenusedinmachinevision applications. Digitalimagesarenotlimitedtorepresentingonlygrayscalevalues.Colorimagerepresentation isachievedinmanydifferentways,themostpopularofwhichmaybetheRGBcolormodel.RGB standsforRed,Green,andBlueandindicatesthatacolorimagerepresentedwiththismodelis composedofvaryingamountsofthecolorsred,green,andblue.Typically,eachpixelinanRGB imageisrepresentedby24bits.Therst8bitsaretheamountofredinthepixel,thenext8 bitstheamountofgreen,andthenext8bitstheamountofblue.Conceptually,onecanthink ofacolorRGBimageasthreegrayscaleimagesstackedoneontopoftheother,withaspecial interpretationassignedtoeachlayer.TheRGBmodelforcolorrepresentationisoneofthemost popularrepresentationfordigitalimagesbecausethisisthenativeformatusedbymostdisplay 24

PAGE 32

Figure9. :AcolorRGBimage.Lefttoright:thefullcolorimage,theredplane,thegreenplane, theblueplane.Fullimagefrom:http://beautifulwork.les.wordpress.com/2009/06/crayons.jpg devices.Thatis,anLCD/LEDscreenismadeupofmanypixels,eachofwhichhasared,green, andbluecomponent.Theintensityatwhicheachofthesecomponentsisoperateddeterminesthe perceivedcolorofthepixel.Figure9showsacolor24-bitRGBimageandeachofthethreecolor planesthatcompriseit. Throughoutthiswork,grayscaleimagesarerepresentedas2Darraysof8-bitintegersandcolor imagesarerepresentedas3Darraysof8-bitintegerswherethe3rddimensionindicatesthecolor planei.e.R,G,andBrespectively.Withthebasicoverviewofhowanimageisrepresentedinthe computer,wearenowinpositiontodiscusssomeoftheimportantmachinevisionalgorithmsfrom ahighlevelperspective. 3.2.3Neighborhoods Weareseldominterestedinthevalueofasinglepixelinanimageinandofitself.Indeed,sucha singularquantityyieldslittleinformationofvalue.Instead,weareconcernedwiththevaluesofthe pixelscontainedwithinadenedarea.Itisonlywhenanalysingthevaluesofpixelsacrossanarea thatanyusefulinformationcanbeobtained.Inmachinevision,thisareahasaspecialnamecalleda neighborhood .Neighborhoodscanbeanyshapeorsizeandareusedtospecifytheareaofinterest surroundingaspeciedanchorpixel.Inmanyalgorithmsthepixelsdenedbyaneighborhoodare usedtocomputeavaluethatisrepresentativeofsomefeature:averages,differences,etc.Andmore oftenthannot,thisvalueiscalculatedateverypixelintheimage.Thatis,theneighborhoodisslid acrosstheimageallowingeachpixeltobecometheanchorpixel.Theresultofthisoperationis anewimagewhereeachpixelinthisimageistheresultoftheneighborhoodoperationaboutthe 25

PAGE 33

Figure10. :Asmoothingoperationusinganeighborhood.Fromlefttoright:theoriginalsharp image,theneighborhoodmaskdenedforthisoperationwiththeanchorpixeldarkened,theresult oftheaveragingoperationoverthedenedneighborhood. respectivepixelintheoriginalimage. Considerasimpleimagesmoothingoperation.Ainitialimageisgiven,andwewishtoreducethe sharpnessoftheimagethroughsomeformofsmoothingoperation.Inaiveimplementationconsists ofusinga5x5neighborhoodandaveragingallthevaluesintheneighborhoodateachanchorpoint. Figure10givesanexampleofthisoperation.Itshowstheoriginalimage,the5x5neighborhood withtheanchorpixelshowndarkened,andtheresultofusingthatneighborhoodalongwithan averagingoperation.Inshort,theneighborhoodwasslidacrosstheimagetoeverypixel,foreach pixelintheoriginalimagethatwasalignedwiththeanchorpixel,thecorrespondingpixelinthe resultsimagewassettotheaveragevalueofallpixelscontainedintheneighborhood.Theeffectis toproduceasmoothedimage. Perhapsthetwomostpopularneighborhoodsinmachinevisionalgorithmsarethe4-neighborhood and8-neighborhood,whichastheirnameimplies,containthe4or8closestpixelstotheanchor pixel.The4-neighborhoodand8-neighborhoodareshowninFigure11.Ingeneral,aneighborhood canbeanysize.Forexample,a24-neighborhoodwouldcontainthe24nearestpixelstotheanchor pixel.ThisisexactlytheneighborhoodusedinFigure10. 3.2.4ConnectedComponents Inabinaryimage,itisoftennecessarytoidentifyindependentregionsinanimage.Thisprocess iscalled connectedcomponentlabeling [30,36]anditsusesincludecountingthenumberofitemsin 26

PAGE 34

Figure11. :A4-neighborhoodand8-neighborhood. animageandndingholeswithinanitem.Analgorithmfortheimplementationofsuchaprocedure isbeyondthescopeofthistextandtheinterestedreaderisreferredtonearlyanytextonMachine Vision.Inshort,aconnectedcomponentsalgorithmscanstheimageandassignsauniquenumber toeachfullyseparatedregion.Everyforegroundpixelthatisan8-neighborwithanotherpixelwill havethesamenumberasthatpixel.ThisprocessisexplainedpictoriallyinFigure12.Inthegure, imageaisabinaryimageshowingthreeseparateregions.Imagebisthelabeledimagewhich showstheuniquenumberassignedtoeachpixeloftheseparatecomponent.Bothimageshavehad theirrangeexpandedtoshowdetail. Afterlabelingtheconnectedcomponentsinanimage,countingthenumberofitemsisassimple asndingthevalueofthelargestlabel.Onecanndholesinanitembyrstinvertingtheimage andthenapplyingtheconnectedcomponentsalgorithm.Ifthereareholesinanyitems,theywill showupasseparatecomponentsandtypicallyhavemuchsmallerareathanthelargestcomponent whichistheactualbackground. 3.2.5EdgeDetection Edgesareonesetoffeaturesinanimagethatarerelevanttoestimatingthestructureandpropertiesofanobjectinascene[36].Edgesarecharacterizedassignicantlocalintensitychanges inanimageandoftenrepresenttheboundarybetweenobjectsorregions.Ifweconsidertheleft imageinFigure10,weseethatthesceneiscomposedofmanydifferentobjects:awoman,ahat,a mirror,etc.Further,eachoneofthesecomponentscanbethoughtasbeingmadeofupsub-regions: thewomanhasafaceandarm,thehathasdecorations,andsoon.Manyoftheseindependent 27

PAGE 35

Figure12. :Connectedcomponentlabeling.Left:abinaryimageconsistingofthreeseparate components.Right:thelabeledimagewhichidentiesauniquenumberforeachcomponent. regionshaveanintensityleveldifferentthantheirneighboringregions.Therefore,anoperationthat candetectthesechangesinintensitycandetecttheboundariesoftheregions.Thisisexactlythe functionofanedgedetector. Thesimplestedgedetectorcansimplyidentifypixelsinanimagewhoseneighborhoodpixels differbyacertainthresholdvalue.Whilethisapproachworks,itwilloftenleadtomanyspurious orunimportantedges.Inpractice,edgedetectionalgorithmsusuallyemployavarietyofcriteria whendeterminingwhetherornotapixelbelongstoanedge.Someofthesecriteriamightbeedge strength,zerocrossingofthesecondderivative,edgevalueofneighboringpixels,andsoon.A practicaledgedetectionalgorithmthathasseenwidespreadadoptionistheCannyedgedetector.It usesavarietyofoperationsaboveandbeyondbasicintensitydifferences,butyieldsstrikinglygood results.Thedetailsoftheimplementationarelefttooneofthepreviouslycitedcomputervisionor digitalimageprocessingtexts,oreventheoriginalCannypaper[23].TheresultsofCannyedge detectionappliedtotheleftimageinFigure10isshowninFigure13. 3.2.6FloodFill Aoodllalgorithmis,conceptually,theoppositeofanedgedetector.Insteadofrepresenting regionsinanimagebytheirboundariesidentiedbylocalintensitychanges,regionsarerepresentedbylocalintensityuniformityornearuniformity.Thealgorithmisfedaseedpointinthe imagethencheckstheneighborsofthatpoint,iftheneighborsdonotdifferbyacertainthreshold, thentheyareassumedtobeequivalentandassignedthesamevalue.Thealgorithmthenshifts 28

PAGE 36

Figure13. :Cannyedgedetection.TheresultsofapplyingtheCannyedgedetectortotheleft imageinFigure10.Showninvertedforclarity. Figure14. :Anexampleofaoodlloperation.Left:originalimage.Right:aoodedregion. overtothenewneighborandrepeatstheprocess.Theprocesscontinuesuntiltherearenomore neighboringpixelsthatmeetcriteria.Theresultisasinglecontinuousareaofuniformvaluethat representssomeregionintheimage.Thepaintbuckettoolinmanyimagemanipulationprograms isanexampleofaoodllalgorithm.Figure14givesanexampleofaoodllalgorithm.Apoint ontheyellowpepperswasgivenasaseedpointalongwithanappropriatethreshold,theresulting lledareaisindicatedinblueinthesecondimage.Noticethatevenwithanadvancedoodllalgorithm,variousholesandimperfectionsremaininthelledarea.Inpractice,severalmachinevision algorithmsarecombinedinasequencewhichyieldsthedesiredresultsforaparticularapplication. 29

PAGE 37

Figure15. :Abinaryimageexample.ThebinaryrepresentationoftheoodedregioninFigure 14b.Dynamicrangeexpandedtoshowdetail. 3.2.7BinaryImageFeatures Theultimategoalofidentifyingindependentregionsinanimageissothattheregionscanlater beprocessedforcharacteristicinformation.Oftentimes,thisinformationisintheformoflocation, shape,andorientationinformation.Suchfeaturescanbecalculateddirectlyfromwhatiscalled thebinaryimagerepresentation.Thatis,oncetheregionhasbeenidentiedthroughsomeprocess edgedetection,oodll,etc.thenitisseparatedfromtherestoftheimageandeverypixelinthe regionisgivenavalueof1andallotherpixelsintheimagearegivenavalueofzero.Ifwetake Figure14basanexampleandconsidertheoodedblueregionasthetarget,thenthebinaryimage ofthatregionisshowninFigure15,whichhashadthedynamicrangeexpandedtoshowdetail. Oncethebinaryimageofaregionhasbeencreated,calculatingtheregion'sarea,centroid,and orientationarerelativelytrivialmatters.Ifweletabinaryimageofsize n m bedenotedby B [ i;j ] where n isthenumberofrowsand m thenumberofcolumns,thenthearea A andthe x; y location ofthecentroidcanbegivenas A = n )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 X i =0 m )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 X j =0 B [ i;j ] .27 x = P n )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 i =0 P m )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 j =0 jB [ i;j ] A y = )]TJ/F26 10.9091 Tf 10.303 8.182 Td [(P n )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 i =0 P m )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 j =0 iB [ i;j ] A .28 Notethattheaboveequationsassumetheoriginoftheimagetobeintheupperleftcornerandthat thepositive y directionisup.Theorientationofabinaryimageismoreinvolved,andthederivation 30

PAGE 38

isnotpresentedhere.Rather,theresultfrom[36]ispresentedandtheinterestedreadershould consultamachinevisiontextforthederivation.Theorientation ofabinaryimageisgivenby x 0 = x )]TJ/F15 10.9091 Tf 11.602 0 Td [( x y 0 = y )]TJ/F15 10.9091 Tf 11.658 0 Td [( y a = n )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 X i =0 m )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 X j =0 x 0 ij 2 B [ i;j ] b =2 n )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 X i =0 m )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 X j =0 x 0 ij y 0 ij B [ i;j ] c = n )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 X i =0 m )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 X j =0 y 0 ij 2 B [ i;j ] tan = b a )]TJ/F20 10.9091 Tf 10.909 0 Td [(c .29 ThereareclearlytwovalidsolutionstoEquation3.29bynatureoftheperiodicityofthe tan function.Thetwopossibleresultsrepresenttheaxisgreatestandleastinertia,andtheyarealways separatedby90degrees.Inthecasewhere a = c and b 6 =0 ,theorientationisundened.Thiswill happenwhentheshapehasnoclearlydenedmajororminoraxis,suchasacircle. 3.2.8ANoteonImageSegmentation Theactofpartitioninganimageintopiecesthatrepresentindependentregionsiscalled segmentation .Segmentationiscurrentlystillaveryactiveeldofresearchandisconsideredbysometo beoneofthemoredifcultproblemsintheeldofmachinevision.Ingeneral,thereisnotasingle segmentationalgorithmthatworkforeveryimageandsuccessfulalgorithmsareusuallytunedto theirspecicapplication. Itisnotthepurposeofthisworktosolvethesegmentationproblem.Aswillbeshowninalater section,someassumptionsmustbemadeaboutthenatureoftheobjectsofinterestinordertomake thetaskofsegmentationtractable.Withtheaidoftheseassumptions,themachinevisiontechniques justdescribedcanbeemployedtoyieldreasonablyreliablesegmentations. 31

PAGE 39

3.3Superquadrics Thepurposeofthissectionistogiveabriefintroductiontogeometricmodellingtoolusedinthis work:thesuperquadric.Asupercialintroductiontothenatureofsuperquadricshapesisfollowed bytheirformalderivationandmathematicalrepresentationaswellassomeusefulpropertiesthat canbedirectlyderivedfromthesuperquadricparameters. 3.3.1Overview Superquadricsareafamilyofthreedimensionalgeometricshapesthatincludessuperellipsoids, supertoroids,andsuperhyperboloidsofoneandtwosheets[1].Incommonparlance,theterm superquadricisoftenusedtodescribeasuperellipsoidand,fortheremainderofthisthesis,this conventionisadoptedandthetwotermsareusedinterchangeably. Asaparametricshape,asuperquadricisaveryconvenientmodelingtoolthatiscapableof representingalargevarietyofregularshapesinasimple,closedformexpression.Figure16shows severalexamplesofsuperquadricsofvariousparametervalues.Thegureisintendedtogivea broadperspectiveoftheshapescapableofbeingmodeledbysuperquadrics.Fromthegure,itcan beseenthatmostregularconvexshapescanbefaithfullyrepresentedbyasuperquadric. Allsuperquadricsshareafundamentalfeature;theyallexhibitthreemutuallyorthogonalplanes ofsymmetry.Therefore,theycannotperfectlymodelageneralconvexorotherwiseamorphous shapewithoutextendingthemodelthroughlocaldeformations;afacetwedonotentertaininthis work.However,manyhouseholdobjectsarewellrepresentedquitewellbysuperquadrics,afact shownin[31].Andtotheextentthatthegoalofthisworkistoreconstructtheshapeofsuch householdobjects,superquadricsarewellsuitedtothetask. 3.3.2DerivationandRepresentation Itwasshownin[5]thatthesphericalproductofapairoftwodimensionalcurvesresultsina threedimensionalsurface.Mathematically,ifwehavethetwodimensionalcurves 32

PAGE 40

Figure16. :Asampleofsuperquadricsshapes.Theshapesinthisgurewerecreatedwithvarious parametrizations.Noticethewideandcontinuousvariationofpossibleshapes. 33

PAGE 41

h = 2 4 h 1 h 2 3 5 ;! 0 ! 1 m = 2 4 m 1 m 2 3 5 ; 0 1 thenthesphericalproductofthetwocurves h m isthethreedimensionalsurface x !; x !; = 2 6 6 6 4 m 1 h 1 m 1 h 2 m 2 3 7 7 7 5 ; 0 ! 1 0 1 .30 Asuperquadriciscreatedbythesphericalproductoftwosuperellipses.Theimplicitequationof asuperellipsein x;y hastheform x a 2 + y b 2 =1 whichcanbeparametrizedasfollows s = 2 4 a cos b sin 3 5 ; )]TJ/F20 10.9091 Tf 8.485 0 Td [( .31 Itwasnotedin[1]thatexponentiationwith representsa signedpowerfunction andshouldbe interpretedas cos = sign cos j cos j .Theauthorsalsopointoutthatfailuretotreatthefunctionasasignedfunctionisacommonoversightinmanytexts,butisacrucialfactwhenrendering superquadricsinsoftware.Thefollowingderivationwaspresentedin[1]. Giventwosuperellipses s 1 ; s 2 ,thesuperellipsoid r ;! canbeobtainedusingthesphericalproductfromEquation3.30 r ;! = s 1 s 2 = 2 4 cos 1 a 3 sin 1 3 5 2 4 a 1 cos 2 a 3 sin 2 3 5 = = 2 6 6 6 4 a 1 cos 1 cos 2 a 2 cos 1 sin 2 a 3 sin 1 3 7 7 7 5 ; )]TJ/F21 7.9701 Tf 9.68 4.296 Td [( 2 2 )]TJ/F20 10.9091 Tf 8.484 0 Td [( .32 34

PAGE 42

Equation3.32istheexplicitequationofasuperellipsoid.Animplicitequationcanbederivedfrom theexplicitformusingtheidentity cos 2 +sin 2 =1 .Equation3.32canberewrittenintheform x a 1 2 =cos 2 1 cos 2 2 .33 y a 2 2 =cos 2 1 sin 2 2 .34 z a 3 2 =sin 2 1 .35 RaisingEquations3.33and3.34by 1 2 andaddingthemtogetherresultsin x a 1 2 2 + y a 2 2 2 =cos 2 1 2 .36 Now,weraiseEquation3.35by 1 1 andEquation3.36by 2 1 andaddthetworesultstogethertoyield theimplicitequationforasuperellipsoid x a 1 2 2 + y a 2 2 2 2 1 + z a 3 2 1 =1 .37 Thus,anypoint x;y;z thatsatisesEquation3.37liesonthesurfaceofthesuperellipsoid.Ifwe considerthelefthandsideofEquation3.37asafunction F x;y;z = x a 1 2 2 + y a 2 2 2 2 1 + z a 3 2 1 .38 thenforapoint x;y;z T ,ifEquation3.38evaluatestolessthan1,thepointliesinsidethesuperquadric.Ifitevaluatestogreaterthan1,thepointliesoutsidethesuperqaudric.Forthisreason, Equation3.38istermedthe inside-outside function. PointsthatsatisfyEquation3.37willbeexpressedinthesuperquadriccenteredcoordinatesystem.Itisdesirabletobeabletorepresentasuperquadricingeneralpositionandorientation.This canbeaccomplishedthroughtheuseofahomogeneoustransformationmatrixseeSection3.1.2. Wedeneahomogeneoustransformationmatrix 35

PAGE 43

W Q T = 2 6 6 6 6 6 6 4 n x o x a x p x n y o y a y p y n z o z a z p z 0001 3 7 7 7 7 7 7 5 .39 whichisthetransformationofthesuperquadricwithrespecttotheworld.However,whatneedisto beabletoexpressworldcoordinatesinthesuperquadriccenteredcoordinateframe.Todothat,we invert W Q T toyieldthetransformationoftheworldwithrespecttothesuperquadric Q W T = )]TJ/F21 7.9701 Tf 5 -4.333 Td [(W Q T )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 = 2 6 6 6 6 6 6 4 n x n y n z )]TJ/F15 10.9091 Tf 8.485 0 Td [( p x n x + p y n y + p z n z o x o y o z )]TJ/F15 10.9091 Tf 8.485 0 Td [( p x o x + p y o y + p z o z a x a y a z )]TJ/F15 10.9091 Tf 8.485 0 Td [( p x a x + p y a y + p z a z 0001 3 7 7 7 7 7 7 5 .40 Sonow,givenasetofhomogeneousworldcoordinates x w ;y w ;z w ; 1 T ,wecancalculatethesuperquadriccoordinatesaccordingtotherelation 2 6 6 6 6 6 6 4 x q y q z q 1 3 7 7 7 7 7 7 5 = Q W T 2 6 6 6 6 6 6 4 x w y w z w 1 3 7 7 7 7 7 7 5 .41 ApplyingEquation3.41toEquation3.37yieldstheinside-outsidefunctionforasuperquadricin generalpositionandorientation F x w ;y w ;z w = n x x w + n y y w + n z z w )]TJ/F20 10.9091 Tf 10.909 0 Td [(p x n x )]TJ/F20 10.9091 Tf 10.909 0 Td [(p y n y )]TJ/F20 10.9091 Tf 10.91 0 Td [(p z n z a 1 2 2 + + o x x w + o y y w + o z z w )]TJ/F20 10.9091 Tf 10.909 0 Td [(p x o x )]TJ/F20 10.9091 Tf 10.91 0 Td [(p y o y )]TJ/F20 10.9091 Tf 10.909 0 Td [(p z o z a 2 2 2 2 1 + + a x x w + a y y w + a z z w )]TJ/F20 10.9091 Tf 10.909 0 Td [(p x a x )]TJ/F20 10.9091 Tf 10.909 0 Td [(p y a y )]TJ/F20 10.9091 Tf 10.909 0 Td [(p z a z a 3 1 1 .42 Aswillbeshowninlatersections,Equation3.42formsthebasisfortheminimizationfunctionthat willbeusedintherecoveryprocess. 36

PAGE 44

Atrstglance,itmayappearthatEquation3.42contains17independentparameters.Thisisnot actuallythecaseas9oftheparametersarerelated.IfwelookattheportionofEquation3.40that representstherotationmatrix Q W R = 2 6 6 6 4 n x n y n z o x o y o z a z a y a z 3 7 7 7 5 .43 andrecallfromSection3.1.2thatthecolumnsofthismatrixrepresenttheunitvectorsofframe f Q g expressedinthecoordinatesofframe f W g ,weseethat W Q R isorthonormalandtherefore hasatotalofsixconstraints.Wecanrepresentthisrotationmatrixasthecombinationofthree sequentialrotationsaboutthecoordinateaxesbythreeindependentamounts.ThisistheEuler Anglerepresentationofarotation.Forthepurposesofsuperquadricrepresentation,weusethe ZYZEulerAngleconventionwhichrotatesframe f W g rstabout ^ Z W byangle ,thenaboutthe new ^ Y w byangle ,andnallyaboutthenew ^ Z w byangle .Thisoperationiscapturedinthe followingequation Q W R = R ^ Z R ^ Y R ^ Z = = 2 6 6 6 4 cos )]TJ/F15 10.9091 Tf 10.303 0 Td [(sin 0 sin cos 0 001 3 7 7 7 5 2 6 6 6 4 cos 0sin 010 )]TJ/F15 10.9091 Tf 10.303 0 Td [(sin 0cos 3 7 7 7 5 2 6 6 6 4 cos )]TJ/F15 10.9091 Tf 10.303 0 Td [(sin 0 sin cos 0 001 3 7 7 7 5 = = 2 6 6 6 4 cos cos cos )]TJ/F15 10.9091 Tf 10.909 0 Td [(sin sin )]TJ/F15 10.9091 Tf 10.303 0 Td [(cos cos sin )]TJ/F15 10.9091 Tf 10.909 0 Td [(sin cos cos sin sin coscos+cos sin )]TJ/F15 10.9091 Tf 10.303 0 Td [(sin cossin+cos cos sin sin )]TJ/F15 10.9091 Tf 10.303 0 Td [(sin cos sin sin cos 3 7 7 7 5 .44 Equation3.44canbesubstituteddirectlyintoEquation3.40.Further,weseethattheparameters n x ;n y ;n z ;o x ;o y ;o z ;a x ;a y ;a z canbedenedintermsof ;; andthereforeEquation3.42 actuallyonlyhas11independentparameters a 1 ;a 2 ;a 3 ; 1 ; 2 ;;;;p x ;p y ;p z .Thephysical meaningsoftheseparametersarelistedbelow. a 1 ;a 2 ;a 3 arethedimensionsofthesuperquadricinthe x y ,and z directions,respectively,as measuredfromthecentroidofthesuperquadric. 37

PAGE 45

1 ; 2 aretheshapeexponentialswhichdeterminetheshapeofthesuperquadricindependent ofitssize.SeeFigure16. ;; aretheZYZEulerAngleswhichdeterminetheorientationofthesuperquadricwith respecttotheworldframe. p x ;p y ;p z arethe x;y;z worldcoordinatesofthecentroidofthesuperquadric.i.e.theorigin ofthesuperquadric-centeredcoordinatesystem. 3.3.3UsefulProperties The11parametersofthesuperquadriccanbeusedtodirectlycomputemanygeometricproperties oftheshapeaboveandbeyondwhatisalreadyplainlydenedbytheparameters.Someofthe propertiesthatareeasilycalculatedinclude Theradialdistancebetweenapointandthesurface Surfacearea Volume Momentsofinertia Thederivationforeachonethesepropertiesispresentedin[1].Inthiswork,weusethevolumeof thesuperquadricasaterminanaccuracymetricandthustheexpressionforvolumeisgivenbelow withoutderivation. Thevolumeofasuperquadricintermsofits11parametersgivenas V =2 a 1 a 2 a 3 1 2 B 1 2 +1 ; 1 B 2 2 ; 2 2 .45 where B isthebetafunctionandisdenedintermsofgammafunctions B x;y = \050 x \050 y \050 x + y .46 \050 z =2 1 0 t z )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 e )]TJ/F21 7.9701 Tf 6.586 0 Td [(t dt .47 38

PAGE 46

Chapter4 ReconstructionAlgorithm Thereconstructionalgorithmcanbelogicallypresentedasasequenceofmacroscopicoperations, eachofwhichcarriestheirownsetofchallenges.Bytreatingeachoftheoperationsasitsown independenttask,thechallengesofeachoperationcanbedealtwithefcientlyandindependentof therestofthealgorithm.Theresultisanalgorithmthatisconceptuallyeasiertounderstandand easiertomaintain. Thealgorithmisbrokendownintothefollowinghighleveltasks:imagecapture,segmentation andsilhouettegeneration,surfaceapproximation,andshapetting.Eachofthesetasksdependson theresultsoftheprevioustaskinordertoproperlyfunction.Thischapterpresentstheimplementationofeachofthesetasksintheorderinwhichtheyareexecuted. 4.1ImageCapture Thealgorithmproposedbythisworkrequiresthreeimagesoftheobjectofinterest.Theseimages mustbecapturedfromthreedisparateviewinglocations.Intheory,theimagescouldbecaptured fromanywherearoundtheobject.However,mostobjectsarenotoatinginspace,andaccessto theentireperipheryoftheobjectwillnotalwaysbeavailable.Therefore,theimagesarecaptured fromthreemutuallyorthogonalpositions:twofrominfrontoftheobject,andonefromoverhead. Forthesimulationenvironmenttobepresentedinalaterchaptersuchaviewingconditionposes noproblems.However,aroboticmanipulatorwilloftenhaveasingularcongurationorkinematic limitationwhentryingtoachievethreeorthogonalpositions.Insuchacasetheviewingpositions areonlyapproximatelyorthogonal.Aswillbeshownintheresultschapters,thisrelaxationdoesnot haveasignicantimpactonaccuracy.AnillustrationoftheviewingpositionsisshowninFigure 17. 39

PAGE 47

Figure17. :Threeorthogonalviewingdirections.Thisconstraintisrelaxedinthepresenceof manipulatorkinematiclimitations. Aftercapturingtheimages,theymustbecorrectedforthedistortionsintroducedbycameraas describedinSection3.2.1.2.Thetopicsofcameracalibrationandimageundistortionarepresented inthenextsections. 4.1.1CameraCalibration AsdetailedinSection3.2.1.2,thereare9internalparametersofthecamerathatneedtobefound: thefocallengths f x ;f y theimagesensoroffsets c x ;c x ,andtheradialandtangentialdistortion coefcients k 1 ;k 2 ;k 3 ;p 1 ;p 2 .Itisbeyondthescopeofthisworktoderiveorimplementsucha calibrationroutine.Rather,weusethecameracalibrationfunctionavailableintheOpenCVlibrary [18].Thisfunctionusesasequenceofimagesofaknownpatternachessboardandemploys variousregressiontechniquestocalculatethe9internalcameraparameters.Afullmathematical derivationofthiscalibrationfunctionisgivenin[17]. Inadditiontothe9internalcameraparameters,theexternalor extrinsic cameraparametersare alsorequired.Thesearethe6parametersrotation,3translationthatmakeupthetransformationofthecamerawithrespecttotheworldbaseframe.Thistransformationisrequiredinorder toprojectpointsinworldspaceintotheimage.Ifwelettheextrinsiccameraparametersbedenotedbythetransformationmatrix C W T ,theintrinsiccameramatrixdenotedby M ,andapointin homogeneousworldcoordinatesbe W P ,thentheprojectionof W P intotheimageisgivenas 40

PAGE 48

P 0 = M C W T W P .1 wherethedivisionofthehomogeneouscoordinatebyitslastelementisimplicit.TheOpenCV libraryprovidesafunctionforcalculatingthematrix C W T .ItusesasetofknownpointcorrespondencesDpointsandtheircorresponding2Dimagelocationsalongwiththecameraintrinsicsand distortionparameters,andemploysleastsquaresminimizationtocomputethesolution. Ingeneral,computing C W T isimpracticalsincethevalueofthismatrixchangesasthecamera moves.Insteadwecomputetherelativetransformationofthecamerawithrespecttotherobotend effector.Sincethistransformationisconstantnomatterwherethecameraislocated, C W T canthen becomputedatanypointbyknowledgeoftherobotendeffectorpositionandorientationalone.If welet W R T representthetransformationoftherobotendeffectorwithrespecttotheworld,thenthe transformationofthecamerawithrespecttotherobotendeffectorcanbecomputedas R C T = )]TJ/F21 7.9701 Tf 5 -4.332 Td [(C W T W R T )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 .2 Once R C T isknown, C W T canbecomputedatanylocationaccordingtotheformula C W T = )]TJ/F21 7.9701 Tf 5 -4.332 Td [(W R T R C T )]TJ/F18 7.9701 Tf 6.587 0 Td [(1 .3 Sincethevalue W R T isknowntohighdegreeofaccuracyforawellcalibratedrobot,Equation 4.3isusedtoaccuratelydeterminethelocationandorientationofthecamerawheneachimageis captured. 4.1.2ImageUndistortion Equation4.1isvalidforcameraswithnodistortions.Butaspreviouslymentioned,mostcameras willhavedistortionsandcalculatingthosedistortionsispartofthecameracalibrationprocess.So, ingeneral,theactofacameraofasetofworldpointswillresultinadistortedimagewhichmust becorrectedifEquation4.1istohold.ThevalidityofEquation4.1iscrucialforimagebased computationssuchasmeasuringdistancesintheimagespaceorbackprojectingraysintothree dimensionalspace.Equations3.24and3.25provideaformulaforcorrectingadistortedimage intoitsundistortedform.Further,theOpenCVlibrary[18]providesafunctionforautomating 41

PAGE 49

Figure18. :Anundistortedimage.TheimageinFigure7aftercorrectingforradialandtangential distortions. thisprocess.Aftercalibratingthecameratodetermineitsintrinsicanddistortionparameters,this functionisappliedtotheimageinFigure7toyieldtheundistortedforminFigure18.Noticeinthe gurethatthelinesthatshouldbestraightarenowstraightasopposedtohighlycurvedasinthe distortedimage. 4.2SegmentationandSilhouetteGeneration Oncetheimageshavebeencapturedand,ifnecessary,pre-processedintoanundistortedform, thenextstepofthealgorithmistoconverttheseimagesintosilhouetteimagesoftheobjectof interest.AsilhouetteimageisnothingbutabinaryimageseeSection3.2.7withtheobjectof interestrepresentedastheforeground,andeverythingelseasbackground.Inordertoaccomplish this,theobjectofinterestmustbesegmentedfromtherestoftheimage.Aswasmentionedin Section3.2.8,themethodsusedtoaccomplishsegmentationarehighlydependentonthenatureof theimageandtheproblem,andthereisnotasingletechniquethatworksforallsituation. Sincetheaimofthisworkisnottosolvegeneralproblemofsegmentation,theobjectsofinterest usedinevaluationofthealgorithmwerechosensuchthattheyfacilitatedthetaskofsegmentation. Thatis,allobjectsusedinthisworkareuniformincolorandalsoacolorthatisdissimilarfrom thebackground.Inthecaseofsimulatedtesting,wherethenatureofthebackgroundcanalsobe 42

PAGE 50

def generate_silhouetteimage: #imageisanNxMx3array #representinganRGBimage rows=image.shape[0] cols=image.shape[1] silhouette=zerosrows,cols for i in rangerows: for j in rangecols: sum=image[i,j,0]+ image[i,j,1]+ image[i,j,2] if sum!=0: silhouette[i,j]=1 return silhouette Figure19. :Asilhouettegenerationalgorithm.Thisalgorithm,writteninPython,willcalculatethe silhouttefromtheimageofasimulatedobject. controlled,thecoloroftheobjectisirrelevantprovideditstilldiffersfromthebackground.For testingconductedintherealworld,thetestobjectsareallredincolor,andthusthetaskbecomes oneofsegmentingaredobjectfromthebackground.Thesegmentationalgorithmwasimplemented differentlyforbothcases,withthelatterbeingmuchmorecomplex.Eachofthesegmentation algorithmsisnowpresentedinturn. 4.2.1SegmentingSimulatedShapes Fortestingthealgorithminsimulation,thebackgroundofthesimulatorissettoblack.Thatis, everypixelinthebackgroundisrepresentedbytheRGBpixeltuple ; 0 ; 0 .Sincetheobjectof interestistheonlyobjectpresentinthesimulatedscene,andgiventhefactthattheobjectisnot blackincolor,anynon-zeropixelrepresentsanobjectpixel.Thus,thesilhouetteimageforthe objectofinterestcanbecalculatedwiththeverysimplealgorithmpresentedinFigure19.Some exampleimagesofsimulatedobjects,andtheircorrespondingsilhouettesareshowninFigure20. 43

PAGE 51

Figure20. :Imagesofsimulatedobjectsandtheircorrespondingsilhouettes.Thesilhouetteshave hadtheirdynamicrangeexpandedtoshowdetail. 4.2.2ColorBasedSegmentationofRealObjects Incontrasttotheeasewithwhichthesilhouettecanbecalculatedfromtheimageofasimulated object,theprocesstodosowitharealimageismuchmoreinvolved.Ratherthanattempttoexplain theprocesswithpseudo-code,wewalkthroughtheprocessstep-by-step.Noclaimismadethatthe followingsegmentationalgorithmistheoptimalapproachtosegmentingourparticulartestobjects fromthebackground.Again,theaimisnottosolvetheproblemofsegmentation,butratherhave functionalsegmentationtotheextentnecessarytotestandvalidatethereconstructionalgorithm.To thatend,thesegmentationmethodpresentedbelowsuitedourneeds. ConsidertheimageinFigure21,whichisascaled-downversionofanimagethatwascaptured bytherobotduringthereconstructionprocess.Theredboxnearthecenteroftheimageistheobject ofinterest,andistheobjectwewishtosegmentfromthebackground.Thebackgroundhowever,is relativelybusyandcontainsothersourcesofredcolor.Eventhoughothersourcesofredaresmall, itisnotenoughtosimplyassumethatallredbelongstotheobject.Instead,wewanttolocateand isolatethelargestpatchofredintheimage. Inordertolocatethelargestareaofredintheimage,wemusthaveamethodtodeterminewhich pixelsintheimagerepresentredpixels.Itwouldbeconvenientifthecolorofapixelcouldbe expressedasasingularvalueratherthantheRGBtripletthathasbeenpreviouslydescribed.Itturns 44

PAGE 52

Figure21. :Animageofanobjectofinterestascapturedbytherobot. outthatthisisinfactpossiblebyconvertingtheimageintoadifferentcolor-space.TheHSVcolor spacerepresentsthecomponentsofapixelasHue,Saturation,andValue,ratherthancomponents ofred,green,andblue.IntheHSVcolormodel,thehueplanecontainsthecolorinformation, thesaturationplanecontainstheamountofcolorpresent,andthevalueplanecontainstheoverall intensitylevelofthepixel.ForthemathinvolvedinRGBtoHSVcolorspaceconversion,thereader isdirectedtoatextonimageprocessingsuchas[35].Thehue,saturation,andvalueplanesofthe imageinFigure21areshowninFigure22 TherangeofpossiblevaluesfortheHueplaneis0-360whichrepresentsthenumberofdegrees inacircle.Thatis,Hueisacircularquantitywith0and360representingthesamecolor:pure red.Since360istoolargeavaluetorepresentwith8bits,therangeisusuallyscaledto0-180at thecostofasmallamountofresolution.Nowthatthecolorisrepresentedbyasingularvalue,we canconstructathresholdthatboundsthecolorwewanttokeep,andusethisthresholdtodetermine whichpixelsareredpixels.Figure23showstheresultsofthresholdingthehueplanesothered pixelsaredenedasthosepixelswithahuelessthan20orgreaterthan160rememberthathueis circularandwrapsat180.Itisclearlyseeninthegurethatthethresholdingcorrectlyidenties allthepixelsbelongingtotheredbox,however,italsoidentiesotherpixelsintheimagethatare redincolor,butareotherwiseunsaturated.Thatis,theyarenota`strong`red. Theshortcomingsofthresholdingthehueplanecanbeovercomebyaddingtheadditionalcriteria thatnotonlymustthepixelberedincolor,butitmustalsobe`strong`inthecolor.Thiscanbe 45

PAGE 53

Figure22. :HSVcolorspace.Toptobottom:thehue,saturation,andvalueplanesoftheimagein Figure21. 46

PAGE 54

Figure23. :Hueplanethresholding.ThepixelsidentiedasredpixelbythresholdingtheHue plane. accomplishedbythresholdingthesaturationplanefromFigure22andcombiningitusinganAND operationwiththeresultsofthehuethresholding.TheresultofthisoperationisshowninFigure 24usingathresholdvalueof150.Thus,anyredpixelassopreviouslydenedwithasaturation valuegreaterthan150isconsideredapixelofinterest. Figure24isveryclosetoidentifyingonlythosepixelswhichbelongtotheredbox.However, itisstillnotperfect,andtherearemanyextraneouspixelsthataresmallsourcesofredelsewhere intheimage.However,thebulkofthepixelsbelongtotheobjectofinterest,andnowwecan usethebinaryimagepropertiesdiscussedinSection3.2.7tocalculatethecentroidofthisimage. Sincethevastmajorityofthepixelbelongtotheobjectofinterest,thecentroidoftheimagewillbe somewhereonthatobject.Thislocationcanthenbeusedasaseedlocationforaoodllalgorithm. ThecalculatedcentroidfortheimageinFigure24isidentiedinFigure25bytheredcircle. Uptothispoint,theentiregoalwastondapixelthatisassuredtobelongtotheobjectof interest.EventhoughtheimageinFigure24isclosetobeingthesilhouetteoftheobjectand indeeditcouldbecleanedupquitenicely,itwillnotalwaysbethecasethatthesegmentation willbesoaccurateatthisstage.Instead,onceapixelisknowntobelongtotheobjectofinterest, weuseaoodllalgorithmseeSection3.2.6thatisdesignedtondareasofsimilarcolor.The oodllalgorithmchosenisthecvFloodFillalgorithmavailableintheOpenCVlibrary.Weusethe computedcentroidastheseedvalueandsetthethresholdsothataneighboringpixelisconsidered 47

PAGE 55

Figure24. :Hueandsaturationthresholding.TheresultsofcombininghueandsaturationthresholdingwithanANDoperation. Figure25. :Seedpointdetermination.ThecalculatedcentroidoftheimageinFigure24isidentied bytheredcircle.Thisistheseedpoint. 48

PAGE 56

Figure26. :Floodllingredpixels.Theresultsofusingaoodlloperationontheredpixelsof imageinFigure21usingtheseedpointfromFigure25. partofthesameobjectprovideditsintensitydoesnotchangebymorethan 45 inanyoftheRGB channels.TheresultofthisoperationisshowninFigure26.Itisclearlyseenthattheoodll operationhascompletelyisolatedtheobjectofinterestfromtherestofthescene.However,there arestillsomeinteriorpixelsthathavenotbeencorrectlyidentied.Thisisaddressedinthenal stepofsegmentation. Thegoalnow,istoensurethatallthepixelswithinthebodyoftheobjectarecompletelylled. Therstreactionmightbetousemorphologicaloperationstollthesmallholes.However,thereis noknowinginadvancehowsmalltheholeswillbeandthereforeselectionofapropermorphological kernelisdifcult,andmorphologicalholellingdoesnotguaranteetopreservetheimagesize.A betteralternativeistouseconnectedcomponentlabelingasdiscussedinSection3.2.4.Oncethe imagehasbeenlabeled,anycomponentthatisnotthebackgroundcanbeassumedtobeaholeinthe silhouetteandthusmerged.Thisnalstepinthesegmentationoperationyieldsausefulsilhouette andisshowninFigure27. Thissegmentationalgorithmisnotfool-proof.Forexample,iftheobjectisholloworhasahole atthelocationofthecomputedcentroid,theoodllalgorithmwillfailbecausetheseedpointwill beinvalid.Thus,thissegmentationalgorithmassumesallobjectsarewithoutanyholes.Seethe noteonsegmentationinSection3.2.8. 49

PAGE 57

Figure27. :Holelling.Thenalgeneratedsilhouetteafterusingconnectedcomponentstollthe holesintheimageinFigure26. 4.3SurfaceApproximation Thesilhouettesgeneratedfromthe2Dimagescapturedbytherobotareusedtocalculatearough approximationofthe3Dsurfaceoftheobject.Thisportionofthereconstructionalgorithmfalls withinaclassofalgorithmscalled,appropriatelyenough, shapefromsilhouettes .Thissection explainsindetailtheportionofthereconstructionalgorithmthatgivesaninitialapproximationto thesurfaceoftheobject. 4.3.1ShapefromSilhouettesOverview Shapefromsilhouettesistheprocessofreconstructingthe3Dshapeofanobjectionofinterest bysegmentingtheobjectfromthebackgroundofeach2Dimageandreprojectingthevisualcones. Theunionoftheseconesgivesanapproximationofthe3Dobjectsurface.Astheofthenumberof imagesapproachesinnity,thereconstructed3Dsurfaceistermedthevisualhull[2].Thevisual hullisthetheoreticallybestpossible3Dreconstructionofanobjectusingonlytwodimensional silhouettes.Itcanbeshownthat,duetothenatureofsilhouettes,anyshapefromsilhouettesalgorithmisincapableofreconstructingsurfaceconcavities.Thisdoesnotrepresentalargeproblemfor thereconstructionalgorithmpresentedinthisworkhowever,becausethechosenmodelingtoolthe superquadricalsocannotrepresentsurfaceconcavitiesintheformtowhichwelimitit. 50

PAGE 58

Shapefromsilhouetteshasbeenwellstudied[8,2,21,25,37].Intheseimplementations,calculatingtheintersectionofthevisualconeswasaccomplishedthroughvoxelcoloringusinganoctree representation.Withthesemethods,theshapeoftheobjectisconvergeduponinaniterativeprocess.Initially,theobjectisassumedtobecontainedwithinaboundingvolume,andthisvolumeis successivelysplitandreneduntilthedesiredaccuracyisachieved.Thisrequiresmultiplepasses ofthealgorithmacrosseachimage.Further,thenatureofthealgorithmsaresuchthattheymust spendtimetestingvoxelsthatresideintheinterioroftheobject,ratherthanonlydealingwiththe surfaceoftheobject. Recently,Lippielloetal.[41]introducedanewshapefromsilhouettesalgorithmthatreconstructs onlythesurfaceoftheobject.Thisalgorithmfunctionsbyconstructingasphereofpointsaround theobjectandthenshrinkingthissphereuntilthepointsintersecttheobject.Thisapproachhas severaladvantagesoverthevoxelcoloringmethodsandtheiroctreerepresentations.Theoctree datastructureisrathercomplexandimplementingoneefcientlyisnotatrivialtask.Indeed, therehasbeenmuchresearchintheefcientimplementationofthisdatastructure[37,21].In contrast,thedatastructureintheLippielloalgorithmisasimplearrayofpoints;muchmoretractable andefcient.Furthermore,sincethisapproachreconstructsonlythesurfaceoftheobject,itis computationallymoreefcientthanthevoxelcoloringmethods. Whilethealgorithmpresentedin[41]isattractivecomparedtothehistoricalimplementations, wefoundthattherewasroomforimprovement.Thisalgorithm,justlikeitshistoricalcounterparts, hasaniterationrequirement.Intheauthors'originalimplementation,eachpointisshrunkalong itsradiusiteratively,byadynamicamount,untilthatpointintersectsallthesilhouetteimages. Wefoundthisiterationsteptobeunnecessary.Usingthepropertiesofprojectivegeometry,the amountthepointmustbeshrunkcanbedeterminedanalyticallyinasinglestep.Bymakingthis modication,theresultingshapefromsilhouettesalgorithmcanbeappliedinasinglepass,without iteration,resultinginadrasticperformanceimprovementoverotherapproaches.Eachstepinthe shapefromsilhouettesalgorithmisnowpresentedindetail. 4.3.2ConstructionoftheBoundingSphere Therststepinconstructingasphereofpointsthatboundstheobjectisdeterminingwhereto placethecentroidofthesphere.Logically,thebestlocationtoplacethecentroidofthesphereis 51

PAGE 59

atthecentroidoftheobject.Thiscanbecalculatedbyreprojectingalinefromthecameracenter ofeachimagethroughthecentroidofthesilhouetteanddeterminethepointwhichminimizesthe sumofsquareddistancestoeachline.ThegeometricinterpretationofthisisshowninFigure28. Inthisgurethethreecameracentersareshownatpoints C 1 ;C 2 ;C 3 alongwiththeassociated imageplane.Eachimageplaneshowstheimagedsilhouetteoftheunknownobjectinthecenter. Alineisprojectedthrougheachcameracenterthatpassesthroughtherespectivelocationofthe imagedsilhouettecentroid.Itisseenthattheselineshaveanapproximateintersectionthatcorrespondsroughlytothe3Dlocationofthecentroidoftheunknownobject.Ingeneral,thethreelines willnotintersectatacommonpoint,andaspreviouslymentioned,thepoint P isfoundthrough minimization. Alineinthreedimensionalspacecanbespeciedbytwopoints,sinceweknowthelocationof thecameracenter,thetaskbecomesoneofndinganotherpointonthelinethatpassesthroughthe cameracenterandtheimagedsilhouettecentroid.Thiscanbeaccomplishedbyinvertingthecamera matrixandrepresentingthecentroidpixelasahomogeneouscoordinate.Givenacameramatrixthat projectspointsinthecameracoordinatesontotheimagingplane I C T seeSection3.2.1.2theinverseofthatmatrix C I T willprojecthomogeneouspixelcoordinatestocameracenteredcoordinates. Note,thattheresultingcameracenteredcoordinateisonlyguaranteedtolieonthelinedenedby thecameracenterandthepixelontheimageplane.Thisissufcientinformationhowever,for deningtheline.Thus,thesecondpointontheline,asdenedingeneralworldcoordinatescanbe foundviathetransformation 2 6 6 6 6 6 6 4 x w y w z w 1 3 7 7 7 7 7 7 5 = W C T C I T 2 6 6 6 6 6 6 4 p x p y 1 1 3 7 7 7 7 7 7 5 .4 wherethematrix C I T hasbeenimplicitlyaugmentedbyanidentityrowandcolumnsotheresulting sizeis4x4. Wenowhavetwopointsdeningaline,wewilllabelthem x 1 and x 2 ,andapointwewishto nd,denedas x 0 ,whichistheapproximatecentroidoftheobject.Theperpendicularsquared distance d 2 betweenthepointandthelineisgivenas[42] 52

PAGE 60

Figure28. :Centroidlocalization.Thegeometricinterpretationofndingthe3Dobjectcentroid frommultiplesilhouettes. d 2 = j x 2 )]TJ/F43 10.9091 Tf 10.909 0 Td [(x 1 x 1 )]TJ/F43 10.9091 Tf 10.909 0 Td [(x 0 j 2 j x 2 )]TJ/F43 10.9091 Tf 10.909 0 Td [(x 1 j 2 .5 Thenforthreelines,thefunctionwewishtominimizeis S x 0 ;y 0 ;z 0 = 3 X i =1 d 2 i .6 Thevalue x 0 ;y 0 ;z 0 thatminimizesthefunction S canbefoundwithoneofthemanyfreely availablenumericalminimizationsoftwareroutines.Inthiswork,weuseanimplementationof Powell'smethod[32],availablein[15],tondthevectorthatminimizes S Oncethe3Dcentroidoftheobjecthasbeencalculated,theonlyremainingpieceofinformation neededinordertoconstructedthesphereistheradius.Thatis,weneedtoknowhowlargetomake thesphereinorderthatitcompletelyboundstheobject,andthusweareinterestedinndingthe pointontheobjectthatisfarthestawayfromitscentroid.Providedthattheradiusofthesphereis greaterthanthedistancefromthispointtothecentroid,thenthespherewillboundtheobject. Therststepincalculatingthesphereradiusisdeterminingwhichsilhouetteimagehasthelargest silhouetteradius.Thesilhouetteradiusisdenedasthefarthestdistancefromthesilhouettecentroid totheedgeofthesilhouette.Wecallthispoint r max .Figure29givesthegeometricinterpretation 53

PAGE 61

Figure29. :Thegeometricinterpretationofthesilhouetteradius r max ofthequantity.Thesilhouetteradiusiscalculatedforeachsilhouette,andsilhouettewithlargest r max isselectedforfurtherprocessing. Oncethelargest r max isfound,theradiusofthespherecanbedeterminedthroughsimplegeometry,whichisdrawnoutinFigure30.Inthegure, p 1 isthecameracenteroftheimagecontaining thelargest r max p 2 isthecentroidlocationfoundinthepreviousstep, p 3 isapointonthelinethat isthebackprojectionofthepoint r max throughthecameracenter p 1 andcanbecalculatedusing Equation4.4,and p 4 isthepointwewishtodetermine.Itisseeninthegurethatthepoint p 4 will alwaysyieldtheradiusofaspherethatencompasstheviewableportionoftheobject.However, thereisnoguaranteethatthespherewillcompletelyboundtheobjectinactual3Dspace.Rather, theonlyguaranteeisthatthespherewillcompletelyboundtheobject asviewedfromtheimages Butgiventhatnothingabouttheobject'sshapeisknownexceptforwhatisprovidedintheimages, thisissufcientinformation.Thepoint p 4 iscalculatedinthefollowingmanner: Letapoint p thatliesonthelinedenedbythepoints p 3 and p 1 bebeparametrizedas p = p 3 + p 3 )]TJ/F43 10.9091 Tf 10.909 0 Td [(p 1 t .7 andlettheplanethatcontainsthepoint p 2 andisnormaltothevector p 1 )]TJ/F43 10.9091 Tf 10.909 0 Td [(p 2 bedenedas p 1 )]TJ/F43 10.9091 Tf 10.909 0 Td [(p 2 p )]TJ/F43 10.9091 Tf 10.909 0 Td [(p 2 =0 .8 54

PAGE 62

Figure30. :Thegeometryforndingtheradiusoftheboundingsphere. ThepointthatsatisesbothEquations4.7and4.8isthepointofintersectionbetweenthelineand theplane.Thispointcanbefoundbycombiningbothequationsandsolvingfortheparameter t t = )]TJ/F15 10.9091 Tf 8.485 0 Td [( p 1 )]TJ/F43 10.9091 Tf 10.909 0 Td [(p 2 p 3 )]TJ/F43 10.9091 Tf 10.909 0 Td [(p 2 p 1 )]TJ/F43 10.9091 Tf 10.909 0 Td [(p 2 p 3 )]TJ/F43 10.9091 Tf 10.909 0 Td [(p 1 .9 Equation4.9canbesubstitutedintoEquation4.7toyieldthepoint p .InspectionofFigure30shows thatthepoint p 4 isequalto p .Thusthescalarradiusofthesphere r iscalculatedas r = j p )]TJ/F43 10.9091 Tf 10.909 0 Td [(p 2 j .10 Inpracticetheradius r willbeincreasedbyaconstantfactorafactorof1.5isusedinthiswork toensurethatthesphereabsolutelyencompassestheobject.Thisisan`insurancepolicy`against errorcausingcornercasessuchaspoint x beingcoincidentwithpoint p 4 .Insuchacase,any quantizationerrorcouldcausetheradiustobetoosmall,leadingtoreconstructionerror.Sinceno accuracyissacricedbyincreasingtheradiusofthespherebythisfactor,thereisnoargument againstsuchanoperation. Itisnotedthatingeneral,oneisunabletosolveforthepoint x asshowninFigure30.Sincethat particularpointcanbelocatedanywherealongtheline p 3 + p 3 )]TJ/F43 10.9091 Tf 10.135 0 Td [(p 1 t providedtheshapehasthe appropriategeometry,solvingfor x wouldrequireanadditionalconstraintwhichisnotavailable. 55

PAGE 63

Oncethecentroidandradiusofthespherehavebeendetermined,allthatremainsistogeneratea setofpointsthatareevenlydistributedacrossthesurfaceofthesphere.Asitturnsout,aperfectly evendistributionofpointsonthesurfaceofasphereisonlypossibleforacertainrestrictednumber ofcasestheplatonicsolids[13].Therefore,wemustsettleforanapproximatelyevendistribution. Thesearchforthebestapproximationisapuzzlingmathematicalproblemandisstillactivelyresearched.Forthepurposesofthiswork,adistributionthatisvisually'even'issufcient.Weuse analgorithmbasedontheGoldenRatio[4]toplacethepointsonthesurfaceofthesphere.The algorithmdividesthespherehorizontallyinto n equalsegments.Thenforeachsegementincrement thelongitudebythegoldenratio andplaceapointatthatlocationonthesurface.Thisalgorithm isdetailedinFigure31.Figure32showsasimulatedobjectandthesphereofpointsthatwere generatedtoencompassitusingtheproceduredetailedinthissection. 4.3.3Approximatingthe3DSurface Oncethesphereofpointsisgenerated,thenextstepistomodifythepositionofeachpointsuch thattheresultingsetofpointsapproximatesthe3Dsurfaceoftheobject.Thisisaccomplished throughthefollowingprocedure: 1.Letthecenterofthecamerabe c 0 2.Letthecenterofthespherebe x 0 3.Let x i beanypointinthesphereotherthan x 0 4.Let x i new betheupdatedpositionofpoint x i 5.Lettheprojectionofthecenterofthesphereintotheimagebe x 0 0 6.Then,foreachpoint x i : aProject x i intothesilhouetteimagetoget x 0 i bIf x 0 i doesnotintersectthesilhouette: i.Findthepixelpoint p 0 thatliesontheedgeofthesilhouettealongthelinesegment x 0 i x 0 0 ii.Reproject p 0 into R 3 togetthepoint p 56

PAGE 64

def sphere_pointsn=1000: #Constructsnevenlydistributed #pointsontheunitsphere. x=arrayn y=arrayn z=arrayn #thegoldenratio phi=1+sqrt5/2 #aunitspherehasdiameter2 dz=2.0/floatn #azimuthincrement az_incr=2 pi/phi for i in rangen: z[i]=dz i )]TJ/F52 10.9091 Tf 14.624 0 Td [(1+dz/2 azimuth=i az_incr #theradiusofthecircularcross #sectionofthesphereatpointz r=sqrt1 )]TJ/F52 10.9091 Tf 14.929 0 Td [(z z x[i]=r cosazimuth y[i]=r sinazimuth points=x,y,z return points Figure31. :Analgorithmforevenlyspacedpointsonasphere.ThisalgorithmiswritteninPython. 57

PAGE 65

Figure32. :Asphereofpointsgeneratedaroundasimulatedobject. iii.Lettheline c 0 p be L 1 iv.Lettheline x 0 x i be L 2 v.Let x i new bethepointofintersectionoflines L 1 and L 2 7.Repeatsteps2-6foreachsilhouetteimage. Thegeometricinterpretationofstep6bisshowninFigure33.Theresultofapplyingthisoperation isasetofpointsthatapproximatesthesurfaceoftheobjectandisshowninFigure34,wherethe procedurewasappliedtothesceneinFigure32. Step6bintheprocedureisworthdiscussionasitrepresentstheimprovementthisworkhasmade tothisalgorithmovertheoriginalversionpresentedbytheauthorsin[41].Thisportionoftheprocedureanalyticallydetermineswheretoplacethepoint x i alongthelineconnecting x i and x 0 such that x 0 i liesontheedgeofthesilhouette.In[41],ratherthantreateachpointindividually,theauthors shrinktheentireradiusofthesphereatonce,forall x i ,byanamountthatisdynamicallydetermined basedonapoint x 0 j thatliesclosestto,butdoesnotintersect,atleastoneofthesilhouettes.This processisrepeateduntil x 0 j intersectseverysilhouette.Whenthishappens,point x j isremoved fromcomputationandtheprocessisrepeatedforallremaining x i .Theauthorsstatethestepis variablebecause,whenapointisbackprojectednearthesilhouettecontours,thestepisreduced toreachabetterapproximationoftheobjectmodel.Step6bshowsthatsuchanapproximationis unnecessarybecausethepositionofthepointcanbecalculatedexactly,inasinglestep. 58

PAGE 66

Figure33. :Thegeometryofpoint x i new .Thepointistheintersectionoflines L 1 and L 2 .Theline L 2 isdenedbyknownpoints x i and x 0 .Theline L 1 isdenedbypoint c 0 ,whichisthecamera center,andpoint p ,whichisthereprojectionoftheimagepoint p 0 into R 3 Itisnotedthatintheperfectlytheoreticalcase,thelines L 1 and L 2 willhaveanintersection. However,sincethepoint p 0 isnotsub-pixelaccurate,thelineswilltypicallynotintersect.Instead, onendsthepointofnearestintersectionofthetwolines.Thisturnsouttobethemidpointofthe linesegmentthatistheperpendiculardistancebetweenthetwolines,andthereforehasaclosed formsolution. 4.3.4PerspectiveProjectionError TheresultsoftheprocedureinSection4.3.3will,ingeneral,onlyyieldaroughapproximation oftheobjectssurface.Thisisdueinlargeparttotheerrorintroducedbyperspectiveprojection. ConsiderFigure35,whichillustratestheconceptintwodimensions.Inthegure,theraythat passesthroughthecameracenter C representstheboundaryoftheobject'ssilhouette.Itisthis boundarytowhichthepointsintheboundingsphereorcircleforthe2Dcaseareshrunk.The blackpointsonthecirclerepresenttheoriginallocationofthepointsonthesphere,andthered pointsarethelocationsofthepointsaftertheyhavebeenshrunktothesilhouetteboundary.Asseen inthegure,notallofthepointswillactuallyshrinktotheobject'ssurface.Rather,theyshrink tothe perspectiveprojection oftheoutermostsurface.Now,asthenumberofimagesoftheobject approachesinnity,theapproximationwillconvergetotheconvexvisualhull[2].Thisconceptis 59

PAGE 67

Figure34. :Theresultsofsurfaceapproximation.Thisimageshowstheresultsofapplyingthe procedureofSection4.3.3tothisimageinFigure32. showninFigure36.However,whenthenumberofimagesissmall,likethethreeimagesusedin thisthesis,theapproximationofthesurfaceremainsfairlyrough,asseeninFigure34.Anextreme exampleisshowninFigure37.Sincecapturingalargenumberofimagesinnotfeasiblefromboth akinematicandcomputationalstandpoint,amethodtorenetheapproximationgeneratedfroma smallnumberaviewsisneeded.Thenextstepinthealgorithmaccomplishesthis. 4.4GeometricShapeFitting AsmentionedattheendofSection4.3.4,itistypicallynotpossibleorotherwiseimpracticalto capturealargeenoughnumberofimagessuchthatthesurfaceapproximationofSection4.3.3is sufcientlyaccurate.Thisisduetoboththekinematiclimitationsofthemanipulatorandcomputationalconsiderations.Amanipulatorwillnot,ingeneral,havecompleteaccessaroundtheentire peripheryoftheobject,andevenifitdid,processingalargenumberofimagesintroducesasignicantcomputationalburden;the134imagesusedforthereconstructionin[26]tookinexcess of100secondstoprocess.Therefore,ameansofimprovingtheapproximationofsurfacefroma smallnumberofimageswasdeveloped.Aswillbeshownintheresults,thismethodprovidesa sufcientlyaccurateapproximationtotheshapeofobjecttoallowforgraspingandmanipulation planning. 60

PAGE 68

Figure35. :A2Dillustrationofperspectiveprojectionerror.Giventheraythroughpoint C that istheboundaryoftheimagedsilhouetteoftheobject,theblackpointsontheencompassingcircle willonlybeshrunktothelocationsindicatedbytheredpoints.Thisresultsinanapproximation oftheobjectssurface.Theaccuracyoftheapproximationwillincreaseasthenumberofimages increases. Figure36. :Anillustrationofthevisualhullconcept.Asthenumberofimagesoftheobjectin Figure35becomeslarge,theapproximationconvergestotheconvexboundaryoftheobject. 61

PAGE 69

Figure37. :Anexampleofextremeperspectiveprojectionerror. Theapproximationofthesurfaceoftheobjectisimprovedbyttingasuperquadrictotheset pointsthataretheresultoftheprocedureinSection4.3.3.Themotivationforusingsuperquadrics arethreefold: 1.Theparametrizednatureofasuperqaudric,asdetailedinSection3.3,naturallylendsitselfto graspplanning.Indeed,the11parametersofthesuperquadriccanbedirectlyusedtodetermine theposition,shape,size,andorientationoftheobject. 2.Superquadrics,asshownin[31],arecapableofaccuratelymodelingmanyeverydayhousehold object. 3.Thestructurednatureofthesuperquadrichastheeffectofignoringlocalizednoiseduetoquantizationandsegmentationerror. Thefollowingsectionsdetailtheprocedurebywhichasuperquadricisttothepoints.First,the classicalsuperquadriccostfunctionispresentedfollowedbyamodiedversionwhichprovides theaddedbenetofforcingthesuperquadrictoeffectivelyignoreperspectiveprojectionerrorslike thoseseeninFigure37. 62

PAGE 70

4.4.1ClassicalSuperquadricCostFunction Thegoalistondthesuperquadricthatbesttsthesetofpointswhichistheoutputofthe pointsurfacereconstructiondetailedinSection4.3.3.Thestandardmethodofsolvingsuchan optimizationproblemistoformulateacostfunctionoftheparametersofinterestandthenuse numericalmethodstondthevaluesoftheparametersthatminimizesthevalueofthefunction.For superquadrics,theinside-outsidefunction,asdevelopedinSection3.3,providesanidealbasisfor acostfunction.Ifwelettheinside-outsidefunction,Equation3.42,bedenotedby F ,thenanaive costfunctioncouldbe min n X i =1 F )]TJ/F15 10.9091 Tf 10.909 0 Td [(1 2 .11 where = f 1 ; 2 ;:::; 11 g = f a 1 ;a 2 ;a 3 ; 1 ; 2 ;;;;p x ;p y ;p z g .12 Sincetheinside-outsidefunctionevaluatesto 1 ifapointliesonthesurfaceofthesuperquadric, itwouldappearthatEquation4.11willbeataminimumforasuperquadricthatisbestttothe points.Howeverinpractice,Equation4.11willnotyieldauniquesolutionincertainsidecases. Anexampleiswhenthepointsdonotformaclosedboundary.Insuchacase,thesuperquadric canextendinthedirectionoftheboundaryopeningwithoutchangingthevalueoftheminimization function.TwomodicationstoEquation4.11aresuggestedin[1]toyieldarobustandefcient costfunction.Therstmodicationistomultiplytheerrortermby p 1 2 3 whichhastheeffect ofrecoveringthesmallestsuperquadrictotthesetofpoints,thusalleviatingtheproblemwiththe aforementionedcornercases.Thesecondmodicationistoraisetheinside-outsidefunction F to thepowerof 1 ,whichaccordingto[1],makestheerrormetricindependentoftheshapeofthe superquadricthatisregulatedby 1 andtherebypromotesfasterconvergence,whilenotaffecting theshapeofthesuperquadric.Thus,thefunctiontominimize,accordingto[1]is min n X i =1 p 1 2 3 F 1 )]TJ/F15 10.9091 Tf 10.909 0 Td [(1 2 .13 Equation4.13isanon-linearequationwith11independentvariablesandmustbesolvedviaiterativenumericalmethods.Thereareseveralalgorithmssuitableforthesolutionofsuchanequation. 63

PAGE 71

Theauthorsof[1]use,forexample,aLevenberg-Marquardtmethod,howeverthisworkmakesuse oftheL-BFGS-Bnon-linearconstrainedoptimizationroutine[7]whichisavailableasaFORTRAN implementationintheScipylibrary[15].Thisroutineallowsforspecifyinglimitsonanynumberof theindependentvariableswhichallowsforcontrollingtheallowableshapesofthesuperquadricand promotingrobustconvergence.Thefollowingconstraintsareimposedontheindependentvariables a 1 ;a 2 ;a 3 > 0 : 0 -Thesuperquadricmusthavepositivesizeinalldirections. 0 : 1 < 1 ; 2 < 2 : 0 -Thesuperquadricisconstrainedtoeverydayconvexshapeswhilealso avoidingsingularconditions.Thisconstraintpromotesrobustconvergence. ;;;p x ;p y ;p z 1 -Noconstraints.Thesuperquadriccanhaveanypositionand anyorientation. 4.4.2ErrorRejectingCostFunction ThecostfunctiongiveninEquation4.13issuitablewhenthepointstowhichthesuperquadricis tareagoodapproximationtotheobject.Whenperspectiveprojectionerrorispresenthowever,as inFigure37,thecostfunctionwillcausethesuperquadrictooverestimatethesizeoftheobjectasit triestoadjustfortheoutlyingpoints.ThiseffectisshownFigure38.Inthatgure,asuperquadric isttothepointsinFigure37andisshownastheopaqueyellowsurface.Whileatrstglance itmayappearthatthesuperquadricaccuratelymodelsthecube,infactthesuperqaudrichas25% morevolumethanthecube. Toincreasetheaccuracyinsuchcases,theerrormetricismodiedwithanadditionaltermto forcethesuperquadrictobesttthepointswithoutextendingbeyondtheboundarydenedbythe points.Thatis,sincetheabsolutebestapproximationthepointscanprovidewheninnityimages areavailableistheconvexvisualhull,itisknownforcertainthatnopartoftheobjectliesoutside theboundsofthepoints.Wethereforerestrictthesuperquadrictobecontainedwithinthesetof pointsbyheavilypenalizingpointsthatevaluatetolessthanoneintheinside-outsidefunction.The modiedcostfunctionisgivenas 64

PAGE 72

min w n X i =1 p 1 2 3 F 1 )]TJ/F15 10.9091 Tf 10.909 0 Td [(1 2 + )]TJ/F20 10.9091 Tf 10.909 0 Td [(w n X i =1 p 1 2 3 F 1 )]TJ/F15 10.9091 Tf 10.909 0 Td [(1 2 2 F 1 < 1 .14 Thisworkusesavalueof w =0 : 2 ,effectivelyplacinganeightypercentweightontheerror forpointsthatliewithintherecoveredsuperquadric.Thismodicationsignicantlyreducesthe effectofperspectiveprojectionerrorandallowsextremelyaccuraterecoveryofregularobjects fromthreeorthogonalviews.Thisvaluewasdeterminedempirically,viatrialanderror,toprovide agoodbalancebetweenstabilityandaccuracyimprovement.Theresultofapplyingthisconstraint isshowninFigure39.Inthisgure,theopaqueyellowsurfaceistherecoveredsuperquadricfor thepointsinFigure37thistimecalculatedwithEquation4.14.Theoriginalobjectisshownasa wireframeduetothefactthatboundariesofthesuperquadricliedirectlyontheboundaryofthe cubeandtheycannotbothbevisualizedsimultaneously.Asopposedtothereconstructionusingthe classicalcostfunctionFigure38,thisreconstructionoverestimatesthevolumebyonly7%;nearly a20%improvementathardlyanyadditionalcomputationalcost. ItisnotedthatinbothFigures38and39,thecornersofthesuperquadricarenotsharp.This isadirectconsequenceoflimitingthelowerboundsoftheparameters 1 ; 2 to0.1duringthe optimizationprocess.Shouldthoseparametersbeallowedtoreach0.0,thenthesuperquadricwould havesharpcornersaswell,butthesolutionwouldbeunstableandperhapsnotconvergeduetothe singularity.Instead,weacceptthetradeoffofasmalllossinaccuracyforincreasedrobustness.In practice,knowingthatthevaluesof 1 ; 2 tobe0.1allowsonetoassumetheobjectisprismatic, andtheimportantparametersbecomethesizes a 1 ;a 2 ;a 3 .Ofthetworeconstructions,theoneusing Equation4.14providedthemostaccurateresultsfortheseparameters. 4.4.3InitialParameterEstimation ThenumericaloptimizationmethodusedtosolveEquation4.14requiresaninitialguessfor themodelparametersinordertobegintheminimizationprocess.Thisinitialguessshouldbefairly accurateasacompletelyerroneousguesswillleadtonon-convergenceandfalseresults.Thissection describeshowtocomputetheparametersforinitialguess.Themethodspresentedherefollowthose whicharepresentedin[1]. 65

PAGE 73

Figure38. :Overestimationinthepresenceofperspectiveprojectionerror. Figure39. :Theeffectsofthemodiedcostfunction.Theoriginalobjectisshownasawireframe. ComparetheaccuracyoftherecoveredsuperquadrictothatinFigure38. 66

PAGE 74

Theinitialvaluesof 1 ; 2 arealways1,whichmeansthesuperquadricalwaysstartstheiteration processasanellipsoid.Theinitialpositionofthesuperquadricisestimatedasthecentroidofthe cloudofpoints.Thatis, p x 0 = x = 1 n n X i =1 x w i .15 p y 0 = y = 1 n n X i =1 y w i .16 p z 0 = z = 1 n n X i =1 z w i .17 Theinitialorientationofthesuperquadricisdeterminedviathematrixofmomentsaboutthecentroid.Thismatrixisgivenas M C = 1 n n X i =1 2 6 6 6 4 y i )]TJETq1 0 0 1 214.127 449.146 cm[]0 d 0 J 0.436 w 0 0 m 5.74 0 l SQBT/F20 10.9091 Tf 214.127 442.922 Td [(y 2 + z i )]TJETq1 0 0 1 268.205 449.146 cm[]0 d 0 J 0.436 w 0 0 m 5.553 0 l SQBT/F20 10.9091 Tf 268.205 442.922 Td [(z 2 )]TJ/F15 10.9091 Tf 8.485 0 Td [( y i )]TJ/F15 10.9091 Tf 11.657 0 Td [( y x i )]TJ/F15 10.9091 Tf 11.602 0 Td [( x )]TJ/F15 10.9091 Tf 8.485 0 Td [( z i )]TJ/F15 10.9091 Tf 11.564 0 Td [( z x i )]TJ/F15 10.9091 Tf 11.602 0 Td [( x )]TJ/F15 10.9091 Tf 8.484 0 Td [( x i )]TJ/F15 10.9091 Tf 11.602 0 Td [( x y i )]TJ/F15 10.9091 Tf 11.658 0 Td [( y x i )]TJETq1 0 0 1 319.888 428.823 cm[]0 d 0 J 0.436 w 0 0 m 6.235 0 l SQBT/F20 10.9091 Tf 319.888 422.598 Td [(x 2 + z i )]TJETq1 0 0 1 374.461 428.823 cm[]0 d 0 J 0.436 w 0 0 m 5.553 0 l SQBT/F20 10.9091 Tf 374.461 422.598 Td [(z 2 )]TJ/F15 10.9091 Tf 8.485 0 Td [( z i )]TJ/F15 10.9091 Tf 11.565 0 Td [( z y i )]TJ/F15 10.9091 Tf 11.658 0 Td [( y )]TJ/F15 10.9091 Tf 8.485 0 Td [( x i )]TJ/F15 10.9091 Tf 11.603 0 Td [( x z i )]TJ/F15 10.9091 Tf 11.564 0 Td [( z )]TJ/F15 10.9091 Tf 8.485 0 Td [( y i )]TJ/F15 10.9091 Tf 11.658 0 Td [( y z i )]TJ/F15 10.9091 Tf 11.564 0 Td [( z x i )]TJETq1 0 0 1 426.143 408.499 cm[]0 d 0 J 0.436 w 0 0 m 6.235 0 l SQBT/F20 10.9091 Tf 426.143 402.275 Td [(x 2 + y i )]TJ/F20 10.9091 Tf 10.909 0 Td [(y 2 3 7 7 7 5 .18 Itisdesiredtondtherotationmatrixthatdiagonalizesthematrix M C ;thisistherotationmatrix thatwillalignwiththesuperquadriccenteredcoordinatesystem.Thusweseekarotationmatrix T R suchthat D = T )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 R M C T R .Itiswellknownthattheeigenvectorsofamatrix, Q ,havethis diagonalizingproperty[10]: D = Q )]TJ/F18 7.9701 Tf 6.586 0 Td [(1 M C Q ,andthus T R = Q .Thatis,therotationmatrixis composedoftheeigenvectorsofmatrix M C T R = eig M C .19 Theorderoftheeigenvectorsisimportant.ItisdesiredtoinsurethattheZ-axisisaligned withtheaxisofleastinertiaforelongatedobjects,andtheaxisofgreatestinertiaforatobjects;a conceptuallynaturalassignment.Thiscanbeaccomplishedbyinvestigationofthemagnitudesof theeigenvalues.Fortheorderedsetofeigenvalues k 1
PAGE 75

Therotationmatrix T R representsEquation3.44andtheparameters ;; canbedirectlyextractedfromthatmatrix. =arctan T R [2 ; 3] sin T R [1 ; 3] .20 =arctan T R [2 ; 3] T R [1 ; 3] .21 =arctan )]TJ/F43 10.9091 Tf 9.681 7.38 Td [(T R [3 ; 2] T R [3 ; 1] .22 Itisnotedthatinsoftwareapplication,theseequationsshouldbecalculatedviatheatan2function. Thenalparameters a 1 ;a 2 ;a 3 areinitiallydeterminedbythesizeoftheboundingboxthatis alignedwiththeobjectcenteredcoordinatesystem.Programmatically,thisismosteasilydeterminedbyusingEquation3.43torotatethepointssothattheyareviewedfromthesuperquadric centeredsystemvia x q = Q W Rx w .23 Thelimitsoftheboundingboxarethen a 1 =max j x q i )]TJ/F15 10.9091 Tf 11.602 0 Td [( x j ;i =1 ::n .24 a 2 =max j y q i )]TJ/F15 10.9091 Tf 11.658 0 Td [( y j ;i =1 ::n .25 a 3 =max j z q i )]TJ/F15 10.9091 Tf 11.564 0 Td [( z j ;i =1 ::n .26 4.5ExampleReconstruction Theprecedingsectionsofthischapterhavegivendetaileddescriptionsandderivationsonevery stepoftheproposedreconstructionalgorithm.Thissectiongivesanillustratedexampleofeachstep ofthealgorithmstartingwiththeimagecaptureandendingwiththenalreconstruction. Figure40showsthestepbystepreconstructionofasimulatedprismaticobject.Imageashows theoriginalobject.Imagesb-dshowthethreesilhouettesgeneratedfromthethreecapturedimagesfollowingtheprocedureinSection4.2.1.Inthiscase,theimaginglocationswereallmutually orthogonalandalignedwiththeobject.Imageeshowsthesphereofpointsthatwasconstructed 68

PAGE 76

aroundtheoriginalobjectusingtheinformationderivedfromthesilhouettesasexplainedinSection 4.3.2.Imagesf-hshowthepointcloudafterallthepointshavebeenshrunksothattheirreprojectionsintersectorlieontheboundaryofthesilhouettesbyusingtheprocedureinSection4.3.3. Imagesi-jshowthesuperquadricthatminimizesEquation4.14forthisparticularsetofpoints. ThisnalstepwasdescribedinSection4.4. 69

PAGE 77

Figure40. :Thereconstructionprocessasastepbystepsimulation.aTheoriginalshape.b-d Thegeneratedsilhouettes.eTheencompassingsphereofpoints.f-hThepointcloudafterthe pointshavebeenshrunktothesilhouetteboundaries.Errorduetoperspectiveprojectionisclearly seen.i-jThesuperquadricthatwasttothepointcloud.Originalshapeshownasawireframe. Noticetheabilityofthesuperquadrictoignoretheperspectiveprojectionerror. 70

PAGE 78

Chapter5 Simulation ThereconstructionalgorithmdevelopedandpresentedinChapter4wasimplementedandtested insimulationbeforebeingimplementedandtestedinhardware.Thischapterdescribesthesoftware simulationenvironmentthatwasdevelopedforthepurposesoftestingthealgorithmaswellasthe resultsoftestingthealgorithmonseveraldifferentsimulatedshapes. 5.1CoreAlgorithm ThealgorithmofChapter4wasbrokenapartintotwoencapsulatedconceptssoastofacilitate softwarereuse:1imagecaptureandprocessingand2objectreconstruction.Theimagecaptureandprocessingishighlydependentonthescenebeingobservedandthecameras/simulation environmentsinvolvedandthereforemustbeimplementedonacasebycasebasis.Theobjectreconstructionportion,however,ishardwareagnostic.Providedthatthesilhouetteimagesandcamera informationintrinsicsandextrinsicsareavailable,objectreconstructioncanproceed.Byseparatingtheproceduresintoseparateentities,theobjectreconstructionroutinescanbeusedbyboththe simulationenvironmentandthehardwareimplementation. Theobjectreconstructionroutinesconsistofthefollowing: Calculationoftheboundingsphereofpoints-Section4.3.2 Thesurfaceapproximationoftheobject-Section4.3.3 Thettingofthesuperquadrictothepointcloud-Section4.4 TheseroutinesareimplementedinPython[19],makinguseoftheNumPyandScipylibraries[15] fornumericalalgorithmsandarraytypedatastructures.PortionsthatareparticularlycomputationallyintensivearewritteninCython[38]forenhancedperformance. 71

PAGE 79

Figure41. :Ascreenshotofthegraphicalsimulator. 5.2GraphicalSimulator Agraphicalsimulatorwasbuiltontopofthecorealgorithmforthepurposesoftestingthealgorithmonawidevarietyofshapesandcameraviewingpositions.Thesimulatorwasalsowrittenin PythonusingtheTraitsgraphicaltoolkit[14].The3Drenderingwasachievedwiththehelpofthe Mayavi3DVisualizationlibrary[33].AscreenshotofthesimulatorinactionisshowninFigure 41.Inthegure,thesimulatorisshowingtheoriginalobjectasawireframe,theoriginalsphere ofpoints,thepointsafterbeingshrunkentothesurface,andtherecoveredsuperquadric.Theyare shownsimultaneouslyforillustrationpurposesonly. 5.2.1Capabilities Thesimulatorcontainstwomainentities:thesimulatedobject,andthesimulatedcamera.Both oftheseobjectsarehighlycongurable.Thesimulatedobjectcantakeonawidevarietyofregular shapesfromprismstoellipsoids,anditsgeometrycanbemodiedinheight,width,anddepth,and itscentroidcanbeplacedinanylocationinspace.Thecameracanbemovedtoanylocationin spacebyspecifyingapointoffocusthepointthecameraislookingat,thedistancethecamerais fromthispoint,andtheazimuthandelevationangleswhichorientthecamerawithrespecttothe pointoffocus. 72

PAGE 80

Inordertosimulateareconstruction,theusercreatesanobjectofinterestandthenspecies thethreelocationsofthecameratocapturetheimages.Theuserthenclicksthe`simulate`buttonandthereconstructionprocessbegins.Whenthereconstructionhascompleted,therecovered superquadricisrenderedintothesimulationenvironmentandtheoriginalobjectconvertedtoawireframe.Thisallowsforquickvisualinspectionoftheaccuracyofthereconstruction.Additionally, theelevenparametersoftherecoveredsuperquadricaredisplayedinadataboxontheleftsideof thesimulatorwindow.Thisallowsfordirectcomparisonoftherecoveredparametersversusthe groundtruth. 5.2.2Limitations Thesimulatorcurrentlyhastwomainlimitations:1Itcannotsimulatearbitraryobjects.Itis limitedtoprismaticandellipsoidalobjects.2Itcannotsimulatemultipleobjectsimultaneously. Thoughtheselimitationsexist,theywerenotalimitationintermsoftestingthealgorithmdeveloped inthiswork.Thealgorithm,initscurrentform,isunabletodealwitheitherofthesecases.Thatis, itisnotdesignedtoreconstructnon-prismaticornon-ellipsoidalobjectsthoughitwillbeshown laterthatitstillgivesusefulresultsforthesecases,anditcannothandlemultipleobjectsofinterest inthesameimage. 5.3SimulationTrialsandResults Thealgorithmwastestedusingthesimulatorforavarietyofobjectsthatrepresenttherangeof shapeslikelytobeencounteredinadomesticsetting.Namely,thealgorithmwastestedonprisms, cylinders,cubes,andspheres,allofvarioussizes.Further,eachoftheshapeswastestedwitha varietyofcameraviewingcongurations.Thatis,theorientationofthethreeorthogonalviewswith respecttotheobjectwasvaried,aswellasthedistancefromthecameratotheobject.Itwasfound thatthealgorithmwasrobusttothesevariations,andconsistentlyyieldedaccurateresultswhenthe cameralocationswerekeptwithinsoundreason. Ratherthanpresenttheresultsofeverytestedvariation,arepresentativesubsetoftheresults ispresentedinTable1.Thetablegivesthenumericalresultsforthereconstructionofasimulated prism,cylinder,cube,andsphere.Foreachcase,thevaluesoftheelevenreconstructedsuperquadric parametersaregivenalongwiththegroundtruthfortheshape.Inaddition,atwelfthvalueisgiven, 73

PAGE 81

v f ,whichstandsforthevolumefraction.Itisdenedasthevolumeoftherecoveredsuperquadric dividedbythevolumeofthegroundtruthshape.Thevolumefractionwasusedin[25]asaconvenientsingularvaluemeasureoftheaccuracyofthereconstruction.Eachofthesereconstructionsis depictedinFigure42withtheoriginalshapeshownasawireframeandthereconstructionoverlayed asanopaquesurface. Theresultsinthetableclearlyindicatethatthealgorithmiscapableofrecoveringtheshapeof theoriginalobjecttohighdegreeofaccuracy.Mostparametersdeviatefromgroundtruthbyonly afewpercent.Careshouldbetakenwheninterpretingthevaluesfor a 1 ;a 2 ;;; inthetable. Sincetheonlyorientationenforcedduringrecoveryistheorientationoftheobject'sZ-axis,the valuesoftheseveparameterscanbeinterchangedfororientationvaluesthatdifferbymultiples of = 2 .AnotherwaytothinkofthisisgivenacubewiththeY-axispointingup,swappingthe XandZaxesstillresultsinacube.Theseorientationambiguitiesbecomeevenmorepronounced withthecylinderandsphere,wheretheorientationofthelatteriswhollyirrelevant.Finally,we giveareminderthethevaluesof 1 ; 2 areboundedwithintherange [0 : 1 ; 2 : 0] forthepurposes ofstabilityoftheminimizationroutine.Therefore,eventhoughthegroundtruthofanobjectmay dictatethat 1 =0 : 0 ,thebestthattheminimizationroutineisallowedtoreportis 1 =0 : 1 .This situationoccursfrequentlyforshapeswithsharpcornerssuchastheprismandcube. 74

PAGE 82

Table1 :SimulationResults Shape Prism Cylinder Cube Sphere Truth Reco Truth Reco Truth Reco Truth Reco a 1 0.4 0.422 1.0 0.987 0.5 0.515 1.0 0.96 a 2 0.5 0.518 1.0 0.993 0.5 0.515 1.0 0.96 a 3 1.25 1.265 1.5 1.544 0.5 0.513 1.0 0.968 1 0.0 0.1 0.0 0.186 0.0 0.1 1.0 0.793 2 0.0 0.172 1.0 0.724 0.0 0.1 1.0 0.781 1.571 -1.571 1.571 -1.575 0.0 -0.558 0.0 0.49 1.571 1.571 1.571 1.573 0.0 3.13 0.0 -0.005 -1.571 -1.571 0.0 0.013 0.0 -0.558 0.0 -0.188 p x 0.0 -0.009 0.0 -0.01 0.0 -0.008 0.0 -0.012 p y 0.0 0.007 0.0 0.0 0.0 0.008 0.0 0.005 p z 0.0 0.003 0.0 0.007 0.0 -0.001 0.0 0.004 v f 1.087 1.088 1.077 1.092 75

PAGE 83

Figure42. :Thereconstructionofsimulatedobjects.Clockwisefromupperleft:aprism,acylinder, asphere,andacube.ThenumericalresultsforthesereconstructionsaregiveninTable1. 76

PAGE 84

Chapter6 HardwareImplementation 6.1HardwareComponentsandSetup Theimplementationhardwareconsistsofthreemainentities:theroboticmanipulatorwhich performstherequiredmotions,thecameratocapturetheimages,andthenetworkwhichconsistsof thevariouscomputersresponsibleforcontrollingtherobot,thecamera,andperformingtheactual objectreconstructioncomputations. Itisdesiredtohavethesevarioussystemsinterconnectedinthemostdecoupledandhardware/operatingsystemagnosticmannerinordertofacilitatesoftwarereuseonandwithotherplatforms, robots,andcameras.Thus,portabilitywasachiefgoalbehindthesystemdesign.Thefollowing sectionsdescribeeachsubsystemcomponentindetail. 6.1.1Robot TheroboticarmusedinthisworkisaKUKAKR6/2,manufacturedbyKUKARoboticsGmbH [29].Itisasixaxis,lowpayload,industrialmanipulatorwithhighaccuracyandarepeatabilityof < 0 : 1 mm.Itssmallersizethoughstilltoolargeforuseonamobileplatformandlargeworkspace makesitwellsuitedforlaboratoryuseandawiderangeofexperiments.Therobotsetup,including thecameradescribedinSection6.1.2isshowninFigure43. TheKUKAcontrolsoftwareprovidesaproprietaryuserinterfaceenvironmentdevelopedinWindowsXPEmbedded,whichinturnrunsatoptherealtimeVxWorksoperatingsystem.Theuser interfaceprovidesaprogramminginterfacetotherobotutilizingtheproprietaryKUKARobotLanguageKRLaswellasanOPCOLEforProcessControlserverthatallowsforconnectionsfrom outsidecomputersandthereadingandwritingofOLEsystemvariables.AsKRLdoesnotprovide facilitiesforcommunicatingwithoutsideprocessesorcomputers,theOPCserverconnectionwas 77

PAGE 85

Figure43. :Robotsetupwithcamera. usedinconjunctionwithasimpleKRLprogramtoexportcontroltoanoutsidemachine.Thedetails ofthisaredelayeduntilSection6.1.3. TheendeffectormountedontheKUKArobotisageneric,two-ngered,pneumaticallyactuated gripper.Itisdesignedinsuchafashionthatmanydifferenttoolscanbeusedwiththerobotand interchangedon-line.Thegripperacceptsanytoolwithamountthatconsistsofa5cmx5cm block,withhemispherescutintoeitherside.Thesehemispheresmatewiththeircounterpartinthe gripperandthustheentireassemblyisself-centering.Withthisdesign,therobotcanautonomously pickupatool,performanoperation,andthenchangetools,allwithnointerferencebyahuman operator.ThegrippercanbeseeninFigure44. 78

PAGE 86

Figure44. :Selfcenteringrobotendeffector. 6.1.2Camera ThecamerausedforimageacquisitionisanAxis207MWwirelessnetworkcamera.Itisrelativelyinexpensiveandhasmegapixelresolution.Themainbenecialfeatureofthecameraisthat itcontainsabuiltinHTTPwebserverwithsupportforacquiringimagesviaCGIrequests.This meansthatthecameracanbeusedbyanyprogramminglanguagewithlibrariessupportingHTTP andCGIconnections.Needlesstosay,thelistofqualifyinglanguagesisextensive. Inordertotransformthecameraintoacompletelywirelesscomponent,awirelesspowersupply wasdeveloped.Namely,acustomvoltageregulatorwasdesignedandfabricatedtoregulatethe voltageofabatterypackdowntotherequired5Vforthecamera.Theregulatorwilloperatewith anyDCvoltagefrom7-25V,allowinginteroperationwithawidevarietyofbatterypacks.A schematicofthedevelopedvoltageregulatorisshowninFigure45. Acameramountwasdesignedforusewiththegripper.Itprovidesforthemountingofthe camera,batterypackandvoltageregulator,androutesthecablesinternallytoavoidpossibleentanglementwiththerobot.Giventheselfcenteringdesignofthegripper,thistypeofmounthasthe advantagethattheextrinsicparametersofthecameradonotchangewhenthecameraismounted anddismounted.Thus,therobotisfreetopickupandputdownthecameraanynumberoftimes withoutrequiringrecalibrationofthecameraextrinsics.Thecamera,mount,andvoltageregulator canbeseengraspedbytherobotinFigure46. 79

PAGE 87

Figure45. :Thecameravoltageregulatorschematic. Figure46. :Thecameramountedinthegripper. 80

PAGE 88

6.1.3Network Inordertoachieveourgoalofportability,thenetworkwasdesignedarounddistributedcomponentsthatusefreeandopensourcestandardsforinterprocesscommunication.Eachcomponent inthenetworkiscapableofoperatingindependentlyonitsownmachinefromanywherethathas accesstothecentralswitch.Inthecaseofourexperiments,thecentralswitchisalocal802.11 Wi-FirouterprovidingWi-LANaccesstothelocalcomputersinthelaboratory.Inournetwork setup,therearefourdistributedcomponentsthatshareinformationacrosstheLAN: 1.TheKUKArobotcomputerrunningKRLprogramsandtheOPCserver 2.TheAxis207MWwirelessnetworkcamera 3.Theobjectreconstructionsoftware 4.TheexternalKUKAcontrolsoftware Thelogicalarrangementofthesecomponents,theirinterconnection,andthecommunicationprotocolsusedareillustratedinFigure47andareexplainedindetailinthefollowingsections. 6.1.3.1ExternalKUKAControllerandtheOPCServer Aspreviouslymentioned,theKUKArobotsoftwareprovidesanOPCserverthatcanbeusedto readandwritesystemvariablesatruntime.WhileOPCitselfisanopenstandard,usingitremotely requiresextensiveDCOMcongurationwhichisbothtediousanderrorprone,aswellaslimiting inthatitrequirestheclientmachinetorunaMicrosoftWindowsoperatingsystem.TheOpenOPC project[6]providesasolutiontothisproblem.BuiltonPython[19],OpenOPCprovidesaplatform agnosticmethodofmakingremoteOPCrequests.Itrunsaserviceonthehostmachineinthiscase WindowsXPembeddedwhichrespondstorequestsfromtheclientmachine.Thehostservicethen proxiestheOPCrequesttothenowlocalOPCserver,thusbypassingallDCOMrelatedissues. ThenetworkcommunicationusesTCP/IPtotransmitserializedPythonobjects[22]. AsimpleprogramwaswrittenintheKRLlanguageandrunsontheKUKArobotcomputerin parallelwiththeOPCserver.Thisprogramsitsinanidleloopmonitoringthesystemvariablesuntil acommandvariablechangestoTrue.Atthispoint,theprogrambreaksoutoftheloopandmoves 81

PAGE 89

Figure47. :Networkandcommunicationlayout. therobottoapositiondictatedbyothersystemvariableswhicharealsosetbytheclientmachine. Atthecompletionofthemotion,theprogramre-enterstheidleloopandtheprocessrepeats. TheexternalKUKAcontrollertheclientrunsonaseparatemachineundertheUbuntuLinux operatingsystem.ThismachinemakesaconnectiontotheOpenOPCservicerunningontheKUKA computerandmakestheappropriaterequeststoreadandwritethesystemvariables.Inthismanner, thisexternalmachineisabletospecifyadesiredrobotposition,eitherabsoluteorrelative,andthen, bysettingthecommandvariabletoTrue,forcestherobottoexecutethemotion.Thismachinealso actsasthemaincontrollogic,synchronizingtherobotmotionwiththeimagecapturingandobject reconstruction. 6.1.3.2WirelessCameraandObjectReconstruction ThewirelesscamerapresentsitselfonthenetworkasanHTTPserverwhereimagescanbe obtainedbymakingCGIrequests.TheimageisreceivedintheformofrawJPEGdatawhichmust beconvertedtoRGBrasterformatforthepurposesofimageprocessing.Sothatthedataneednot traversethenetworktwice,theconnectiontothecameraismadefromtheobjectreconstruction programandimagesarecapturedandconverteduponrequestbythemaincontrolprogram. Theconnectionbetweenthemaincontrollerandobjectreconstructionprogramsutilizesthe XML-RPCprotocol.Aformofremoteprocedurecall,XML-RPCcompletelyabstractsthenetwork connectionfromtheprogramminginterface.Thisallowscallingmethodsonthehostmachineasif 82

PAGE 90

Figure48. :Threeimagescapturedbytherobotduringatestrun.Thenatureofthedisparate viewinglocationscanbeinferredfromtheseimages. theywerelocaltotheclientsoftware,drasticallysimplifyingtheinterchangeofcomplexdataand operations.XML-RPCisanopenstandardandavailableonallmajorplatformsandformanydifferentprogramminglanguages.OurimplementationutilizestheXML-RPCimplementationavailable inthePythonstandardlibrary. 6.2ViewingPositions Therobotisprogrammedtoobservethescenefromthreelocations.Duetokinematicconstraints oftherobot,theselocationsarenotmutuallyorthogonal,buttheyapproachsuchacondition.The threeimagescapturedbytherobotduringoneofthetestrunsareshowninFigure48.Fromthese images,onecanseethenatureofthedisparateviewinglocations;thefrontalviewsarenotperfectly horizontalnoristheoverheadviewperfectlyvertical.Wenotethatduringreconstruction,therobot isnotinformedofthelocationoftheobjectonthetable.Rather,itismerelyassumedthattheobject isvisibleinallthreeimagesofthescene;thelocationoftheobjectinthesceneisdeterminedas partofthereconstructionthe p x ;p y ;p z parametersofthesuperquadric. 6.3TestObjects Thealgorithmwastestedonfourdifferentobjects:aprismaticbatterybox,anelongatedcylinder composedoftwostackedcups,aballofyarn,andasmallcardinalstatue.Thesefourobjectsare showninFigure49.Therstthreeobjectsrepresenttherangeofgeometricshapesfrequently encounteredindomesticsettings:prismatic,cylindrical,andellipsoidal.Itwasexpectedthatthe algorithmwouldachieveaccuratereconstructionsfortheseshapes.Thelastobjectisamorphous 83

PAGE 91

Figure49. :Thefourreal-worldtestobjects.Clockwisefromupperleft:aprismaticbatterybox,a stackofcups,acardinalstatue,andaballofyarn. andwasincludedtotesttherobustnessofthealgorithmwhenpresentedwithdatathatisincapable ofbeingaccuratelydescribedbythemodel.Inallcases,thetestobjectsareredincolortoeasethe taskofsegmentationandsubsequentlyreliablesilhouettegeneration.Again,itisnottheaimofthis worktosolvethebroadermachinevisionproblemofsegmentation. 6.4ExperimentalTrialsandResults ThissectionsdiscussesthereconstructionresultsofeachofthetestobjectsmentionedinSection 6.3.Eachofthecaseswiththeexceptionofthecardinalisaccompaniedbyarenderedgurewhich showsthegroundtruthoverlayedbythecalculatedreconstruction.Thegroundtruthisshownas wireframeandthereconstructionasanopaquesurface.Theaccuracyisdiscussedfromaqualitative perspectiveintheframeofwhetherornotthereconstructedshapecouldbeusedtoplanagrasping 84

PAGE 92

maneuver.Thenumericalresults,presentedinthesamefashionandwiththesamemeaningasthe simulatedreconstructionsinTable1,aregiveninTable2. Wheninterpretingtheaccuracyoftheresults,itmustbekeptinmindthatthereareseveralsources oferrorthatarecompoundedintotheseresultswhicharenotpresentinthesimulation: Imprecisecameracalibration:intrinsicsandextrinsics Robotkinematicuncertainty Imperfectsegmentation Groundtruthmeasurementuncertainty Thelastbulletisnoteworthy.Sincetheobjectisplacedrandomlyintherobot'sworkspacethe onlypracticalwayofmeasuringthegroundtruthpositionandorientationistouseameasuring deviceattachedtotheendeffectoroftherobot.Thoughmoreaccuratethanattemptingtomanually measurefromtherobotbase,theerroriscompoundedbybothmachineinaccuracyandhumanerror. Itmustbepointedout,thatdespiteallofthesesourcesoferror,theaccuracyofmostreconstructionsiswithinacouplemillimetersofgroundtruth.Comparethiswiththeresultsin[27],where areconstructionwithover200imagesresultedinanerrorof10millimeters.Furthermore,thealgorithmrequiresonly0.3secondsonaveragetoperformthecompletereconstruction.Thisisa signicanttimesavingscomparedtothe100secondsrequiredforthereconstructionin[26]. 6.4.1BatteryBox Thereconstructionofthebatterybox,showninFigure50,wasoverallthemostaccurateofall thereconstructions.Itisclearlyseenthatthemodelcorrectlycapturestheheight,width,depth, andshapeofthebatteryboxwithonlyaslightdeviationinpositionandorientation.Thenumerical valuesoftheresultsinTable2conrmthis.Thoughthisreconstructionhasthelargestdeviation fromunityforthevolumefraction,thereisnoquestionthattheresultantmodelcanbeusedasa modelforgraspplanning.Furthermore,theaccuracyoftheshaperepresentationopensthedoorfor otherpossibilitiessuchastaskinferencebasedonshapeand/orappearance. 85

PAGE 93

Figure50. :Thereconstructionofthebatterybox.Groundtruthisshownasawireframe. 6.4.2CupStack Thereconstructionofthestackofcups,whichwouldbeaccuratelyapproximatedasacylinder, didnotachievehighaccuracyinallparameters.Namely,theshapeparameters 1 ; 2 wereinaccurate withrespecttogroundtruth.ShownifFigure51,itisseenthatthereconstructedshapeisbordering onprismaticratherthancylindrical.Thisisabyproductthatstemsfromthenatureofperspective projectionshadowsandcanbeeliminatedbyeithermoreviews,oraviewperfectlyinlinewiththe majoraxis.Therestofthereconstructionparametersheight,width,depth,positionhowever,are allaccurate,withonlytheorientationdeviatingslightly.Sincethegroundtruthshapeiscylindrical andvertical,theonlyorientationparameterofsignicanceis ,andinthiscasethevaluesof 0 and areequivalent.Thus,weseethattheorientationhasonlyaslightamountoferror.Itisnoted againthatthiserrorstemsfromacombinationofthemanycompoundederrorsourcesmentionedin thebeginningofthissection. Despitethenon-cylindricalshapeoftheobject,itisbelievedthattheoverallsizeandposition arestillaccurateenoughtoattemptagraspingmaneuverbasedonthemodelparameters.Arobot designedtooperateinadomesticsettingshouldhavenoproblemwiththemarginoferrorpresent inthisreconstruction. 6.4.3YarnBall Theyarnbarnreconstruction,Figure52,isnearlyasaccurateasthebatterybox.Thereisslight deviationintheorientationsimilartothetwopreviouscases.Forthiscase,theorientationparameter isinsignicantsincetheshapehassymetryaboutthataxis,and 0 and areequivalentvalues 86

PAGE 94

Figure51. :Thereconstructionofthestackoftwocups.Groundtruthisshownasawireframe. Figure52. :Thereconstructionoftheyarnball.Groundtruthisshownasawireframe. for .Theyarnballwasthelargestofallobjectstestedat150mminlengthand100mmindiameter. Andthoughsuchanobjectislikelytoolargetobegraspedbymostdomesticsizedmanipulators, theaccuracyissufcienttoplanthemaneuverprovidedthemanipulatorhassufcientcapacity.It isnotedthatthesizeparameters a 1 ;a 2 ;a 3 canbedirectlyusedasacriteriatodetermineifanobject iswithincapabilitylimitsofthemanipulator. 6.4.4CardinalStatue Thegurineofthecardinalwasincludedtotesthowthealgorithmperformswhenprovidedwith datathatdoesnottwellwiththereconstructionmodelandassumptions.Thistestcaseisshown inFigure53.Sinceitwouldbedifculttomodelthegroundtruthasawireframe,theresultsof thesurfaceapproximationphaseofthealgorithmareusedinstead.Fromthegure,itisclearthat therewouldbenowaytoinferfromtheboxshapewhichisthenalreconstructionthattheoriginal objectwasabird.However,itisinterestingtonotethatthereconstructionisveryclosetowhata 87

PAGE 95

Figure53. :Thereconstructionofthecardinalstatue.aSideview.bTopview.cRearview. Thepointsaretheresultsofthesurfaceapproximationphase.Theopaquesurfaceisthetted superquadric.Aperspectiveprojectionshadowisclearlyevidentinthebottomrightcornerofthe pointcloudinc. humanwouldlikelyprovideifaskedtoselectaboundingboxthatbestdescribestheobject.That is,thereconstructedshapedoesanexcellentjobofcapturingthebulkformofthestatuedespitethe factthatthedataisillformedwithrespecttothemodelingassumptions.Itisnotastretchofthe imaginationtothinkthatagraspcouldbeaccuratelyplannedforthisobjectusingthereconstructed shape. Thisexampleshowsthat,evenwhentheobjectdoesnottakeaformthatcanbeaccuratelymodeledbyasinglesuperquadric,thealgorithmstillgeneratesusefulresults. 6.5Limitations Thissectiondetailsthemainlimitationsencounteredduringtheimplementationandtestingof thealgorithmontherobotichardware. Thesimulationenvironmentdevelopedinthisworkdemonstratedexcellentreconstructionswhen threeorthogonalviewsoftheobjectwereavailablewiththeobjectcenteredineachview.Inactual implementation,itwasfoundthatitisdifcultfortherobottoreachthreemutuallyorthogonal positionswithoutexceedingitsjointlimitsorreachingsingularpositions,thusthepositionsreached wereonlyapproximatelyorthogonalandtheywerenot,ingeneral,centeredontheobject. Thereishowever,nohardrequirementfromthealgorithmthattheobjectbecenteredintheframe. Indeed,itisdesignedtofunctionprovidedonlythattheobjectiscompletelyvisibleintheframe, whereverthatmaybe.However,whentheobjectislocatedatanobliquelocationfromthecamera, 88

PAGE 96

Table2 :ExperimentalResults Shape BatteryBox CupStack YarnBall CardinalStatue Truth Reco Truth Reco Truth Reco Truth Reco a 1 30 32.9 34 41.0 50 57.1 25 1 24.0 a 2 15 16.9 34 37.8 50 51.5 25 1 18.4 a 3 52.5 51.6 60 61.2 75 74.4 30 1 29.5 1 0.1 0.2 0.1 0.3 0.7 0.6 0.1 2 0.1 0.2 1.0 1.4 1.0 1.1 0.4 0.0 3.12 0.0 -0.3 -0.17 3.07 -9.35 1.57 1.56 0.0 3.10 1.53 1.52 -0.82 0.0 0.10 0.0 -2.60 0.0 0.86 6.01 p x 880 878.4 898 893.8 898 893.9 898 892.0 p y -924 -924.6 -915 -917.5 -915 -912.7 -915 -908.9 p z 865 864.9 892 894.8 855 854.1 862 867.3 v f 1.18 1.13 1.14 1 Approximationbasedontheboundingbox thatwouldencompassthebulkofmass. Thevaluehasnomeaninginthecontextofthisshape. 89

PAGE 97

largeperspectiveprojectionshadowsareintroduced,andthistendstostretchthereconstructedshape inthedirectionoftheshadow.Thiserrorcanbehandledinoneoftwoways.First,ifthecentroid ofthesilhouetteisdeterminedtobetoofarfromtheimagecenter,therobotcanberepositionedand anewimagecaptured.Second,thedirectionandamountofskewcanbedeterminedbyanalysisof thesetofpointsandtheeffectcompensatedforbyplacingconstraintsontheappropriatevariables beforeexecutingtheminimizationroutine. Imagesegmentationiswidelyregardedasoneofthemoredifcultproblemsincomputervision. Uptothispoint,wehavedevelopedoursystemwithoutregardtotheprocessofsegmentingthe objectofinterestfromthebackground.i.e.wehaveassumedaperfectsilhouettetobeavailable. Inasimulatedenvironment,perfectsegmentationistrivialtoachieve,asfullcontrolovertheenvironmentisavailable.Thesituationbecomeexceedinglydire,however,inarealenvironmentwith anunstructuredbackground,uorescentlighting,andotherconfoundingeffects.Ourtestingenvironmentconsistedsolelyofredobjectswithverylittletonoreectance.Thisdrasticallysimplied thetaskofsegmentation,thoughtheresultswerestillnotperfect,andthethresholdshadtobemanuallytunedforvariouslightingconditionsorvariationsincolorshade.Thesesimplicationsand manualthresholdtuningareobviouslyunacceptableinthecontextofachievingouroriginalgoalof requiringnopriorinformationabouttheobject. 90

PAGE 98

Chapter7 ConclusionandFutureWork Thisworkhasdevelopedanalgorithmcapableofrecoveringthethreedimensiongeometryofa novelobjectusingonlythreecameraviews.Itwasshownthatbycapturingthethreeimagesof thenovelobjecttakenfromdisparatelocations,thealgorithmisabletocalculateaparametrized modelofthatobjectwithsufcientaccuracytoallowfortheplanningofroboticgraspingandmanipulationmaneuvers.Incontrasttoothereffortsintheliterature,theproposedalgorithmrequires fewerimages,signicantlylesscomputationtime,andyieldsanoverallhigherreconstructionaccuracy.Furthermore,theparametersofthereconstructedmodelcanbedirectlyusedforgraspand manipulationplanning.Nofurtheranalysisoftheshapeortimeconsumingstatisticalmethodsare necessary. Thealgorithmwasimplementedinbothsimulationandhardware.Inbothenvironments,the algorithmyieldingexceedinglyaccuratereconstructions.Despiteadditionalsourcesoferrorand uncertaintyinthehardwareenvironment,theaccuracyofthealgorithmdecreasedonlymarginally. Furthermore,whenpresentedwithdatathatwereunabletobemodeledaccuratelyaccordingthe modelingparadigm,thealgorithmstillgeneratedusefulandlogicalresults.Withrespecttothe resultspresentedhere,thealgorithmhasmettheobjectivesofSection1.2anditisbelievedthatthere ismerittofurtherinvestigationandresearchintothisproposedmethodofnovelobjectrecognition. 7.1FutureWork Futureplansincludeintegratingagraspingalgorithmbasedonthereconstructedsuperquadricparametersandtestingtheaccuracyofthegrasponavarietyofhouseholdobjects;specicallythose involvedwithADLs.Thealgorithmwillalsobetestedtoobservethebehaviorwhentheviewing locationsbecomelessandlessdisparate.Itisplannedtoinvestigatewhatcanbedonetoincrease theaccuracytoanacceptablelevelwhensuchaconditionarises,suchasincorporatingtheappear91

PAGE 99

ancedatathatisdiscardedbyusingonlysilhouettes.Thatis,anattemptwillbemadetoincorporate structurethatcanbeinferredfromtherasterimageswiththestructureofthesuperquadrictoimprovetheaccuracyandoverallrobustnessofthereconstruction.Itisalsoplannedtoinvestigate incorporatingothersensoryinformationtoaugmenttheabilitiesoftheopticalreconstructionby providingdepthinformationthatcannotberecoveredduetoprojectionshadows.Workinthisarea hasalreadybegunwiththeuseofasinglepointlaserrangender.Wealsoplantoinvestigaterobust regressiontechniquesthatwilltmultiplesuperquadricstothepointcloud,therebyallowingusto reconstructcomplexobjectsasasequenceofsuperquadrics.Finally,sinceouralgorithmdepends onhighqualitysilhouettesinordertoachieveaccurateresults,havingfacilitiesforrobustsegmentationisahighpriority.Wewillinvestigatetheadoptionofsegmentationalgorithmsthatcan,toa largeextent,automaticallyadapttovariouslightingconditionsandobjectfaades. 92

PAGE 100

References [1]A.Jaklic,A.Leonardis,andF.Solina. SegmentationandRecoveryofSuperquadrics ,volume20of ComputationalImagingandVision .KluwerAcademicPublishers,2000. [2]A.Laurentini.TheVisualHullConceptforSilhouette-BasedImageUnderstanding. TransactionsofPatternAnalysisandMachineIntelligence ,16,February1994. [3]AshutoshSaxena,JustinDriemeyer,andAndrewY.Ng.RoboticGraspingofNovelObjects usingVision. TheInternationalJournalofRoboticsResearch ,27:157,2008. [4]AuthorUnknown.EvenlyDistributedPointsonSphere. http://www.cgafaq.info/wiki/Evenly_distributed_points_on_sphere,3April2010. [5]A.H.Barr.Superquadricsandangle-preservingtransformations. IEEEComputerGraphics andApplications ,1:11,1981. [6]BarryBarnreiter.TheOpenOPCProject.http://openopc.sourceforge.net/,16December2009. [7]C.Zhu,R.H.Byrd,andJ.Nocedal.L-BFGS-B:Algorithm778:L-BFGS-B,FORTRANRoutinesforLargeScaleBoundConstrainedOptimization. ACMTransactionsonMathematical Software ,23:550,1997. [8]CharlesR.Dyer. VolumetricSceneReconstructionFromMultipleViews ,pages469. FoundationsofImageUnderstanding.Kluwer,Boston,2001. [9]D.G.Lowe.ObjectRecognitionfromLocalScale-InvariantFeatures. Proceedingsofthe InternationalConferenceonComputerVision ,2:1150,1999. [10]D.G.ZillandM.R.Cullen. AdvancedEngineeringMathematics,ThirdEdition .Jonesand BartlettPublishers,2006. 93

PAGE 101

[11]D.Kim,R.Lovelett,andA.Behal.Eye-in-HandStereoVisualServoingofanAssistiveRobot ArminUnstructuredEnvironments. InternationalConferenceonRoboticsandAutomation pages2326,May2009. [12]DanicaKragic,MartenBjorkman,HenrikI.Christensen,andJan-OlofEklundh.Visionfor roboticobjectmanipulationindomesticsettings. RoboticsandAutonomousSystems ,52:85 100,2005. [13]DaveRusin.TopicsonSphereDistributions.http://www.math.niu.edu/rusin/knownmath/95/sphere.faq,3April2010. [14]EnthoughtInc.TheTraitsFrameworkforValidationandEvent-DrivenProgrammingin Python.http://code.enthought.com/projects/traits/,2001. [15]EricJones,TravisOliphant,PearuPeterson,etal.Scipy:Opensourcescientictoolsfor Python,2001. [16]FreekLiefhebberandJorisSijs.Vision-basedcontroloftheManususingSIFT. International ConferenceonRehabilitationRobotics ,June2007. [17]G.BradskiandA.Kaehler. LearningOpenCV:ComputerVisionwiththeOpenCVLibrary O'ReillyMedia,2008. [18]GaryBradski,TrevorDarrell,IntelCorporation,etal.OpenCVComputerVisionLibrary. http://opencv.willowgarage.com/wiki/. [19]GuidovanRossumetal.ThePythonProgrammingLanguage.http://www.python.org,2009. [20]HaiNguyen,CresselAnderson,AlexanderTrevor,etal.El-E:AnAssistiveRobotthatFetches ObjectsfromFlatSurfaces. HRIWorkshoponRoboticHelpers:UserInteractionInterfaces andCompanionsinAssistiveandTherapyRobots ,2008. [21]HiroshiNoborio,ShozoFukuda,andSuguruArimoto.ConstructionoftheOctreeApproximatingThree-DimensionalObjectsbyUsingMultipleViews. IEEETransactionsonPattern AnalysisandMachineIntelligence ,10,November1988. [22]I.deJong.PythonRemoteObjects.http://pyro.sourceforge.net/,2010. 94

PAGE 102

[23]J.Canny.AComputationalApproachtoEdgeDetection. IEEETransactionsonPattern AnalysisandMachineIntelligence ,8:679,1986. [24]JohnJ.Craig. IntroductiontoRobotics:MechanicsandControl-ThirdEdition .Pearson PrenticeHall,2005. [25]K.ShanmukhandArunPujari.VolumeIntersectionwithOptimalSetofDirection. Pattern RecognitionLetters ,12:165,1991. [26]K.Yamazaki,M.Tomono,andT.Tsubouchi. PickingupanUnkownObjectthroughAutonomousModelingandGraspPlanningbyaMobileManipulator ,volume42/2008of STAR SpringerBerlin/Heidelberg,2008. [27]KimitoshiYamazaki,MasahiroTomono,TakashiTsubouchi,andShin'ichiYuta.3-DObject ModellingbyaCameraEquippedonaMobileRobot. InternationalConferenceonRobotics andAutomation ,April2004. [28]KimitoshiYamazaki,MasahiroTomono,TakashiTsubouchi,andShin'ichiYuta.AGrasp PlanningforPickingupanUnknownObjectforaMobileManipulator. InternationalConferenceonRoboticsandAutomation ,2006. [29]KUKARoboticsGmbH.KR6/2.http://www.kukarobotics.com/en/products/industrial_robots/low/kr6_2/. [30]LindaShapiroandGeorgeStockman. ComputerVision .PrenticeHall,February2001. [31]M.J.Schlemmer,G.Biegelbauer,andM.Vincze.RethinkingRobotVision-CombiningShape andAppearance. InternationalJournalofAdvancedRoboticSystems ,4:259,2007. [32]PowellMJD.Anefcientmethodforndingthemethodofafunctionofseveralvariables withoutcalculatingderivatives. ComputerJournal ,7:152,1964. [33]PrabhuRamachandranandGalVaroquaux.TheMayavidatavisualizer. http://code.enthought.com/projects/mayavi,2005. [34]R.HartleyandA.Zisserman. MultipleViewGeometryinComputerVision .CambridgeUniversityPress,secondedition,2003. 95

PAGE 103

[35]RafaelC.GonzalezandRichardE.Woods. DigitalImageProcessing-ThirdEdition .Pearson PrenticeHall,2008. [36]RameshJain,RangacharKasturi,andBrianG.Schunck. MachineVision .McGraw-Hill,Inc., 1995. [37]RichardSzeliski.RapidOctreeConstructionfromImageSequences. CVGIP:ImageUnderstanding ,58,July1993. [38]StefanBehnel,RobertBradshaw,DagSverreSeljebotn,GregEwing,etal.Cython:CExtensionsforPython.http://www.cython.org,2009. [39]SutonoEffendi,RayJarvis,andDavidSuter.RobotManipulationGraspingofRecognized ObjectsforAssistiveTechnologySupportUsingStereoVision. AustralasianConferenceon RoboticsandAutomation ,2008. [40]TsuneoYoshikawa,MasanaoKoeda,andHiroshiFujimoto. ExperimentalRobotics ,volume 54/2009of SpringerTractsinAdvancedRobotics ,chapterShapeRecognitionandOptimal GraspingofUnkownObjectbySoft-FingeredRoboticHandswithCamera,pages537. SpringerBerlin/Heidelberg,2009. [41]V.LippielloandF.Ruggiero.SurfaceModelReconstructionof3DObjectsFromMultiple Views. InternationalConferenceonRoboticsandAutomation ,pages2400,May2009. [42]EricW.Weinstein.Point-LineDistance-Dimensional.FromMathWorldAWolframWeb Resource.http://mathworld.wolfram.com/Point-LineDistance3-Dimensional.html,2010. 96


xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam 22 Ka 4500
controlfield tag 007 cr-bnu---uuuuu
008 s2010 flu s 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0003425
035
(OCoLC)
040
FHM
c FHM
049
FHMM
090
XX9999 (Online)
1 100
Colbert, Steven.
0 245
Shape and pose recovery of novel objects using three images from a monocular camera in an eye-in-hand configuration
h [electronic resource] /
by Steven Colbert.
260
[Tampa, Fla] :
b University of South Florida,
2010.
500
Title from PDF of title page.
Document formatted into pages; contains X pages.
502
Thesis (M.S.M.E.)--University of South Florida, 2010.
504
Includes bibliographical references.
516
Text (Electronic thesis) in PDF format.
538
Mode of access: World Wide Web.
System requirements: World Wide Web browser and PDF reader.
3 520
ABSTRACT: Knowing the shape and pose of objects of interest is critical information when planning robotic grasping and manipulation maneuvers. The ability to recover this information from objects for which the system has no prior knowledge is a valuable behavior for an autonomous or semi-autonomous robot. This work develops and presents an algorithm for the shape and pose recovery of unknown objects using no a priori information. Using a monocular camera in an eye-in-hand configuration, three images of the object of interest are captured from three disparate viewing directions. Machine vision techniques are employed to process these images into silhouettes. The silhouettes are used to generate an approximation of the surface of the object in the form of a three dimensional point cloud. The accuracy of this approximation is improved by fitting an eleven parameter geometric shape to the points such that the fitted shape ignores disturbances from noise and perspective projection effects. The parametrized shape represents the model of the unknown object and can be utilized for planning robot grasping maneuvers or other object classification tasks. This work is implemented and tested in simulation and hardware. A simulator is developed to test the algorithm for various three dimensional shapes and any possible imaging positions. Several shapes and viewing configurations are tested and the accuracy of the recoveries are reported and analyzed. After thorough testing of the algorithm in simulation, it is implemented on a six axis industrial manipulator and tested on a range of real world objects: both geometric and amorphous. It is shown that the accuracy of the hardware implementation performs exceedingly well and approaches the accuracy of the simulator, despite the additional sources of error and uncertainty present.
590
Advisor: Rajiv V. Dubey, Ph.D.
653
Shape Reconstruction
Shape from Silhouettes
Object Classification
Robot Vision
Machine Vision
690
Dissertations, Academic
z USF
x Mechanical Engineering
Masters.
773
t USF Electronic Theses and Dissertations.
4 856
u http://digital.lib.usf.edu/?e14.3425