USF Libraries
USF Digital Collections

Generation capacity expansion in restructured energy markets

MISSING IMAGE

Material Information

Title:
Generation capacity expansion in restructured energy markets
Physical Description:
Book
Language:
English
Creator:
Nanduri, Vishnuteja
Publisher:
University of South Florida
Place of Publication:
Tampa, Fla
Publication Date:

Subjects

Subjects / Keywords:
Deregulated electricity markets
Generation expansion planning
Matrix games
Reinforcement learning
Conditional value-at-risk
Dissertations, Academic -- Industrial and Management Systems Engineering -- Doctoral -- USF   ( lcsh )
Genre:
non-fiction   ( marcgt )

Notes

Summary:
ABSTRACT: With a significant number of states in the U.S. and countries around the world trading electricity in restructured markets, a sizeable proportion of capacity expansion in the future will have to take place in market-based environments. However, since a majority of the literature on capacity expansion is focused on regulated market structures, there is a critical need for comprehensive capacity expansion models targeting restructured markets. In this research, we develop a two-level game-theoretic model, and a novel solution algorithm that incorporates risk due to volatilities in profit (via CVaR), to obtain multi-period, multi-player capacity expansion plans. To solve the matrix games that arise in the generation expansion planning (GEP) model, we first develop a novel value function approximation based reinforcement learning (RL) algorithm.Currently there exist only mathematical programming based solution approaches for two player games and the N-player extensions in literature still have several unresolved computational issues. Therefore, there is a critical void in literature for finding solutions of N-player matrix games. The RL-based approach we develop in this research presents itself as a viable computational alternative. The solution approach for matrix games will also serve a much broader purpose of being able to solve a larger class of problems known as stochastic games. This RL-based algorithm is used in our two-tier game-theoretic approach for obtaining generation expansion strategies. Our unique contributions to the GEP literature include the explicit consideration of risk due to volatilities in profit and individual risk preference of generators. We also consider transmission constraints, multi-year planning horizon, and multiple generation technologies.The applicability of the twotier model is demonstrated using a sample power network from PowerWorld software. A detailed analysis of the model is performed, which examines the results with respect to the nature of Nash equilibrium solutions obtained, nodal prices, factors affecting nodal prices, potential for market power, and variations in risk preferences of investors. Future research directions include the incorporation of comprehensive cap-and-trade and renewable portfolio standards components in the GEP model.
Thesis:
Dissertation (Ph.D.)--University of South Florida, 2009.
Bibliography:
Includes bibliographical references.
System Details:
Mode of access: World Wide Web.
System Details:
System requirements: World Wide Web browser and PDF reader.
Statement of Responsibility:
by Vishnuteja Nanduri.
General Note:
Title from PDF of title page.
General Note:
Document formatted into pages; contains 89 pages.
General Note:
Includes vita.

Record Information

Source Institution:
University of South Florida Library
Holding Location:
University of South Florida
Rights Management:
All applicable rights reserved by the source institution and holding location.
Resource Identifier:
aleph - 002063947
oclc - 558721573
usfldc doi - E14-SFE0003031
usfldc handle - e14.3031
System ID:
SFS0027348:00001


This item is only available as the following downloads:


Full Text

PAGE 1

GenerationCapacityExpansioninRestructuredEnergyMarketsbyVishnutejaNanduriAdissertationsubmittedinpartialfulllmentoftherequirementsforthedegreeofDoctorofPhilosophyDepartmentofIndustrialandManagementSystemsEngineeringCollegeofEngineeringUniversityofSouthFloridaMajorProfessor:TapasK.Das,Ph.D.JoseL.Zayas-Castro,Ph.D.AlexSavachkin,Ph.D.KandethodyRamachandran,Ph.D.RalphE.Fehr,III,Ph.D.AudunBotterud,Ph.D.DateofApproval:May12,2009Keywords:DeregulatedElectricityMarkets,GenerationExpansionPlanning,MatrixGames,ReinforcementLearning,andConditionalValue-at-RiskcCopyright2009,VishnutejaNanduri

PAGE 2

DEDICATIONToAmma,Daddy,Anna,Vinnu,andChaitu.

PAGE 3

TABLEOFCONTENTSLISTOFTABLESiiiLISTOFFIGURESivABSTRACTvPREFACEviiCHAPTER1INTRODUCTION11.1GenerationExpansionPlanning21.1.1RegulatedSettingsv/sRestructuredSettings21.2ResearchObjectives31.3ResearchContributions41.4DissertationOutline5CHAPTER2LITERATUREREVIEW72.1GenerationCapacityExpansionPlanning72.2NashEquilibriaofMultiplayerGames9CHAPTER3BASICCONCEPTS133.1BasicConceptsofGameTheory133.1.1ZeroSumandNon-ZeroSumGames133.1.2PureandMixedStrategy143.2EquilibriuminPowerMarketGames143.3SolutionStrategies173.3.1OptimizationofIndividualBiddingStrategies173.3.2ApproachesSeekingEquilibriumStrategies213.3.2.1LCP223.3.2.2EPEC243.3.2.3RLBasedApproach253.4BriefOverviewofReinforcementLearning26CHAPTER4GENERATIONEXPANSIONPLANNINGMODEL284.1Two-TierMatrixGameModelforGEP284.1.1TopTier:InvestmentGame30i

PAGE 4

4.1.1.1RiskConstrainedProtCalculationModel314.1.2BottomTier:SupplyFunctionGame324.1.2.1OptimalPowerFlowModel34CHAPTER5REINFORCEMENTLEARNINGBASEDSOLUTIONALGO-RITHMFORMULTIPLAYERMATRIXGAMES375.1MatrixGames375.1.1EquivalentMatrixGamesforDiscountedRewardStochas-ticGames395.1.2EquivalentMatrixGamesforAverageRewardStochasticGames425.2FindingNEofMatrixGames445.2.1AValueIterationAlgorithmforn-PlayerMatrixGames45CHAPTER6EMPIRICALANALYSISANDPRACTICALAPPLICATION476.1NumericalEvaluationoftheLearningAlgorithm476.1.1MatrixGameswithKnownEquilibria476.1.2APowerMarketMatrixGame516.2SomeRemarks54CHAPTER7SOLUTIONFRAMEWORKFORTWO-TIERGEPMODEL567.1SolutionAlgorithmfortheTwo-TierGEPModel56CHAPTER8NUMERICALANALYSIS608.1NumericalExperimentationandAnalysis608.1.1ComputationalResults638.1.1.1MixedStrategiesandMultipleEquilibria678.1.1.2GeneratorProtsandConsumerSurpluses678.1.2NodalPriceSensitivityAnalysis708.1.3ImpactofRiskPreferenceonGEP73CHAPTER9CONCLUDINGREMARKS759.1AdvancesMadebythisResearchinGEP759.2PracticalApplicationsofMatrixGames77CHAPTER10FUTURERESEARCHDIRECTIONS7810.1Cap-and-TradeProgramsandRenewablePortfolioStandards7910.2JointModelforGEPandEmissionsControl8010.3FurtherPolicyAnalysisandPlanningApplications81REFERENCES82ABOUTTHEAUTHOREndPageii

PAGE 5

LISTOFTABLESTable2.1ImportantModelingAttributesfromGEPLiterature9Table3.1SomeImportantModelingAttributesfromBiddingStrategyLit-erature19Table6.1SampleMatrixGameswithPureStrategyNashEquilibria48Table6.2PureStrategyNashEquilibriumResults48Table6.3MixedStrategyEquilibriumResults50Table6.4ResultsfromtheStudyof4-BusPowerNetwork54Table7.1StepsforCalculatingCVaR59Table8.14-yearDemandProjections61Table8.2SupplyFunctionParametersofGenerators61Table8.3LineFailureScenariosusedinComputingCVaR62Table8.4DemandVariationsusedinComputingCVaR63Table8.5GEPDecisionsforDemandsfromTable8.1=0:164Table8.6PriceandQuantityAllocationsfortheFourYearPlanningHori-zon67Table8.7GeneratorPayoMatricesandGameSolutionsfortheFourYearPlanningHorizon=0:168Table8.8LineCapacityandConsumerDemandSlopeLevels71Table8.9FStatisticValuesin2-FactorDesign71Table8.10LinearRegressionModelCoecients72iii

PAGE 6

LISTOFFIGURESFigure4.1SchematicoftheTwo-TierGEPModelforaTwoGeneratorSce-nario29Figure6.14-BusPowerNetwork52Figure7.1SchematicforTwo-TierGEPModelSolutionAlgorithm58Figure8.1FiveBusElectricPowerNetwork62Figure8.2ProtsandConsumerSurplusesinStrategicBiddingandPerfectCompetition69Figure8.3ProtsandExpansionPlansversusRiskPreferences74iv

PAGE 7

GENERATIONCAPACITYEXPANSIONINRESTRUCTUREDENERGYMARKETSVishnutejaNanduriABSTRACTWithasignicantnumberofstatesintheU.S.andcountriesaroundtheworldtradingelectricityinrestructuredmarkets,asizeableproportionofcapacityexpan-sioninthefuturewillhavetotakeplaceinmarket-basedenvironments.However,sinceamajorityoftheliteratureoncapacityexpansionisfocusedonregulatedmarketstructures,thereisacriticalneedforcomprehensivecapacityexpansionmodelstar-getingrestructuredmarkets.Inthisresearch,wedevelopatwo-levelgame-theoreticmodel,andanovelsolutionalgorithmthatincorporatesriskduetovolatilitiesinprotviaCVaR,toobtainmulti-period,multi-playercapacityexpansionplans.TosolvethematrixgamesthatariseinthegenerationexpansionplanningGEPmodel,werstdevelopanovelvaluefunctionapproximationbasedreinforcementlearningRLalgorithm.CurrentlythereexistonlymathematicalprogrammingbasedsolutionapproachesfortwoplayergamesandtheN-playerextensionsinliteraturestillhaveseveralunresolvedcomputationalissues.Therefore,thereisacriticalvoidinliteratureforndingsolutionsofN-playermatrixgames.TheRL-basedapproachwedevelopinthisresearchpresentsitselfasaviablecomputationalalternative.Thesolutionapproachformatrixgameswillalsoserveamuchbroaderpurposeofbeingabletosolvealargerclassofproblemsknownasstochasticgames.v

PAGE 8

ThisRL-basedalgorithmisusedinourtwo-tiergame-theoreticapproachforob-taininggenerationexpansionstrategies.OuruniquecontributionstotheGEPlitera-tureincludetheexplicitconsiderationofriskduetovolatilitiesinprotandindividualriskpreferenceofgenerators.Wealsoconsidertransmissionconstraints,multi-yearplanninghorizon,andmultiplegenerationtechnologies.Theapplicabilityofthetwo-tiermodelisdemonstratedusingasamplepowernetworkfromPowerWorldsoftware.Adetailedanalysisofthemodelisperformed,whichexaminestheresultswithre-specttothenatureofNashequilibriumsolutionsobtained,nodalprices,factorsaectingnodalprices,potentialformarketpower,andvariationsinriskpreferencesofinvestors.Futureresearchdirectionsincludetheincorporationofcomprehensivecap-and-tradeandrenewableportfoliostandardscomponentsintheGEPmodel.vi

PAGE 9

PREFACEFirstandforemost,Iwouldliketothankmyfamilyfortheirconstantsupport,love,andencouragementthroughoutmylife.ThankyouforstandingbymewhileIsuccessfullycompletedthislong,arduous,andfruitfuljourney.Iconsidermyselfextremelyblessedtohavesuchanamazingfamilythatissosupportiveandencour-aging,everyminuteofeveryday.MomandDad,Iamherebecauseofyourblessings,theysustainme.AnnaandVinnu,Icannotthankyouenoughforallthenon-stopencouragementandloveandforandkeepingmyspiritsupbeat.AdebtofgratitudeisowedtoDr.TapasDas,myGuru,mentor,friend,andfamily,withoutwhomthisworkwouldnotbepossible.Heisatreasurechestofknowledge,andaconstantsourceofinspiration.Hishardwork,persistentworkethic,anddriveforexcellence,havebeenatruesourceofmotivation.Iwillfondlyrememberourthoughtfuldiscussionsaboutresearch,daysandweeksofpaperandproposalwriting,andourmanydigressionsintoothertopicsoflife,family,andbeyond,overthepast6years.Ilookforwardtoalifetimeoffriendship,researchcollaborations,morejointpublications,andlastbutnotleast,Iwilllooktohimforcontinuedinspiration.ThetimeIspentatUniversityofSouthFlorida,andspeciallytheDepartmentofIndustrial&ManagementSystemsEngineeringIMSE,willremainsomeofthemostpreciousmomentsofmylife.Ithasbeenmyhomeawayfromhomeandmyfamilyawayfromfamily.IwanttothankDr.JoseZayas-Castro,theChairofIMSEforeverythinghehasdoneforme.Hisleadershipinthedepartmenthasbeenagreatsourceofinspiration.Histakeonseveralissuessuchasresearch,education,vii

PAGE 10

camaraderie,andhissteadfastfocusonthewellbeingofstudentsandfacultyaretremendous.IhavelearnedagreatdealfromhimandIlookforwardtoourcontinuedfriendshipandcollaborations.IMSEwouldnotbethesamewithoutthededicatedworkofMs.GloriaLatterandMs.JackieStephens.Thankyouforeverythingfromthebottomofmyheart.IamprofoundlythankfultoDr.DasandDr.Okogbaaforgivingmetheoppor-tunitytoserveastheProjectManageroftheNSFfundedGK-12program-STARSNSFGrantDGE0638709.TheGK-12programandcontinuedinteractionswithDr.DasandDr.OkogbaaandthemanagerialteamofWilkistarOtienoandDianaPrietohavehelpedmypersonalandprofessionalgrowthtremendously.IwishtothankDr.DasfortheresearchassistantshipviaNSFECCSgrant#0400268andtheIMSEdepartmentfortheteachingassistantships,withoutwhichIwouldhaveneverbeenabletocometoUSFandfulllthisdream.MysincereandheartfeltthankstoDr.Ramachandranforhisguidanceonseveralaspectsofthereinforcementlearningalgorithm.IwillcontinuetousehisclassnotesforthestochasticprocessescoursesthatIplantoteachinthefuture.IwouldalsoliketothankDr.RalphFehr,Dr.AudunBotterud,andDr.AlexSavachkin,forgivingmesomevaluablesuggestionsthathaveimprovedthequalityofthedissertation.Thanksareduetomylab-matesandmyclosestfriendsduringmydoctoralstudiesatUSF:WilkistarOtieno,PatricioRocha,andDianaPrieto.Iwillforeverrememberourlengthyimpromptudiscussionsinthegraduatelababoutalltopicsunderthesunandallthefunpot-lucksandget-togethersateachothershomes,allofwhichhavemademygraduateeducationunforgettable.ThankstoalltheINFORMSstudentchaptercommitteemembers,withwhomIhaveserved.IhavelearnedagreatdealfrommyconstantinteractionswithalltheSTARSGK-12Fellows.Itwasawonderfulexperience.ThankstothosestudentsthatworkedbeforewithDr.Das:Dr.Rajeshviii

PAGE 11

GanesanandDr.AbhijitGosavi;bothofwhomhaveknowinglyandunknowinglyservedasaninspirationtome.Lastandbynomeanstheleast,Iamluckytohavemybestfriendasalifepart-nerverysoon-Chaitu.Iamblessedtohaveherasoneofthemostknowledgeablecolleagues,labmate,runningpartner,guidinglight,andtherockofmylife.Herunas-sumingattitude,calm,soothingdemeanorhaveallowedmetoforgeaheadsmoothly.Despiteherchallengingscheduleandnon-stopwork,shehasalwaysfoundtimetoreadthroughseveraldraftsofmypublications,dissertation,debugandhelpwithmycodes,andsitthroughinnumerablepracticeresearchpresentations.Herbeliefandcondenceinmearethefuelformyenergy.Herpatienceandcomposureastoundmeandsustainme.Iamtrulyluckytohavemether11yearsago,andhavecompletedbothmyundergraduateandgraduateeducationalongwithher.Shecontinuestoamazemeeverydaywithherquitegrace,innerstrength,andaheartofgold.Thankyou.ix

PAGE 12

CHAPTER1INTRODUCTIONMotivatedbythesuccessofderegulationinindustriessuchastelecommunications,airlines,andtransportation,theelectricpowerindustryrestructuringwasintroducedinmanypartsoftheU.S.aswellasinmanycountriesaroundtheworld.Electric-itymarketrestructuringhasspurredasignicantamountofresearchtomodelandsubsequentlyimproveourunderstandingofhowvarioussegmentsofthemarketper-formandinteractwithoneother.Duetotheinteractionsofpolitical,socioeconomic,andtechnologicalforces,thederegulatedelectricpowerindustriesbothintheUnitedStatesandabroadhaveundergonemanystructuraltransformations.Thoughsigni-cantdierencesexistintheworkingofmarketsaroundtheworld,thecommongoalsofrestructuringarethereductionofpricesfortheend-user,usheringintechnologicalinnovation,andincreaseofsocialwelfare.DespitesomemajorinitialsetbacksinCalifornia,successfulderegulatedmarketslikePennsylvania-NewJersey-MarylandPJMinterconnection,NewYorkIndepen-dentSystemOperatorNYISO,ElectricReliabilityCouncilofTexasERCOT,andseveralmarketsaroundtheworld,havereinvigoratedthepolicymakers.Currently,overafourthofthestatesacrosstheU.S.,andseveralcountriesaroundtheworld,notablyUK,Nordiccountries,andAustraliatradeelectricityinaderegulateden-vironment.Severalinsightfulmonographs[1,2,3]thatdealwithpowersystemeconomicsandoperationofrestructuredmarketsexistintheliterature.Recently,surveypaperswerepresentedtothepowermarketliteraturebyVentosaetal.[4],1

PAGE 13

Dayetal.[5],BoucherandSmeers[6],andNanduriandDas[7].Thesurveyin[4]consistsofanexcellentoverviewofrecentmarketmodelingtrends,and[5]and[6]dis-cussmarketequilibriumformulationsrespectively.However,theabovemonographsandsurveypapersdonotshedlightonthemodelsolutionapproaches,andcertainimportantissueslikeelectricityauctionsandsolutionapproachesusedtoobtainop-timalbiddingstrategiesandNashequilibria.Thesetopicsarediscussedindetailin[7].1.1GenerationExpansionPlanningAccordingtotheNationalEnergyPolicyNEPdevelopedin2001andtheAnnualEnergyOutlook2007,energydemandintheU.S.isslatedtoincreasesharplyoverthenexttwodecades[8,9].ItisstatedintheNEPthattheUnitedStateswillneedabout393,000MWofnewgeneratingcapacityby2020tomeetthisgrowingdemand.WithaboutfteenStatesintheU.S.currentlytradingelectricityinrestructuredmarkets,asignicantproportionoftheaforementionedcapacityexpansionwillhavetotakeplaceinamarketbasedenvironment.Currentliteratureisrichwithresearchexaminingcapacityexpansionundertheregulatedmarketparadigm.However,thereisacriticalneedfordevelopingcomprehensivecapacityexpansionmodelsinrestructuredmarketsettings.Thisresearchaimstoaddressthisneed.1.1.1RegulatedSettingsv/sRestructuredSettingsGEPintraditionalsettingsisformulatedasaleastcostoptimizationproblemthatminimizesproductionandcapitalcosts.GEPinrestructuredsettings,ontheotherhand,needstobemodeledasanon-cooperativeprotmaximizationproblem.ThisisbecauseGEPinrestructuredsettingshasmultiplecompetingdecisionmakers,asopposedtoasingledecisionmakerintraditionalsettings.Therefore,theinvestment2

PAGE 14

decisionsmadebyageneratorinrestructuredmarketsaectnotonlyhis/herprotsbuttheothergenerators'protaswell,andhencetheneedtomodelitasanon-cooperativegame.GenerationexpansionplanningGEPinarestructuredmarketisthechallengeofdeterminingwhichtype,where,andatwhattimeperiodsnewgenerationcapacitiesarelikelytobeinstalledbythecompetinggeneratorsinresponseto:expecteddemandgrowth,changesinnetworkconditions,andmarketdesignincentives.Thisresearchaddressestheabovechallengebydevelopingacomprehensivematrixgamemodelthatsubsumeselectricpowermarketfeatureslikemultiplecompetinggenerators,amulti-yearplanninghorizon,transmissionconstraints,anddemandstochasticity.Themodelalsoexplicitlyconsidersriskduetovolatilitiesinprotusingaconditionalvalue-at-riskmeasureaswellasusingindividualgeneratorriskpreferences.Themodelhasatwo-tiermatrixgameconstructthatiterativelybuildsmulti-year,multi-playerexpansionstrategiesforthecompetinggenerators.Theexpansionstrategiesfromthemodelareobtainedusingareinforcementlearningbasedvaluefunctionapproximationalgorithmforsolvingmatrixgames,whichwepresentinChapter5see[10].1.2ResearchObjectivesTheobjectivesofthisresearcharethefollowing.Eachofthesebroadresearchobjectivesareaddressedinvariouschaptersofthedissertation.1.Developacomprehensivematrixgamemodelthataddressesthechallengeofgenerationcapacityexpansioninrestructuredelectricpowermarkets2.Developasolutionalgorithmtosolvethematrixgamesembeddedinthetwo-tiermodel3

PAGE 15

3.Performadetailedempiricalanalysisofthematrixgamesolutionalgorithm4.Formulateanoverallsolutionframework,whichusesthematrixgamesolutionalgorithm,tosolvetheGEPproblem5.DemonstratetheapplicabilityoftheGEPmodelusingsamplepowernetworks1.3ResearchContributionsThisresearchmakessomesignicantcontributionsintheadvancementofthestate-of-the-artbothingenerationcapacityexpansionplanningaswellasinsolu-tionapproachestomultiplayermatrixgames.Ournoveltwo-tiermatrixgamemodelforgenerationexpansionplanninginrestructuredpowermarketsettingsistherstofitskind.Thetwo-tiermodelconsidersinvestmentcompetitionattheuppertierandtheembeddedsupplyfunctioncompetitionatthelowertier.Theuseofarein-forcementlearningalgorithm,aspresentedinChapter5andin[10],showspromiseinsolvingmatrixgamesofrelativelyhigherdimensionality.Thecontributionsalsoincludetheincorporationofgeneratorriskpreferencesandameasureofconditionalvalue-at-riskCVaR,whichmakestheinvestmentdecisionsmorerobust.Themodelanditssolutionmethodologyaredemonstratedonasamplenetworkwithvebuses,seventransmissionlines,threegenerators,andfourloads.Thesimultaneousconsid-erationofseveralimportantelementsinexpansionplanning,suchas,transmissionconstraints,risk,demandvariations,multi-periodplanninghorizon,andmultiplegen-erationtechnologiesisnotfoundintheexistingliterature.ThenovelvaluefunctionapproximationbasedreinforcementlearningalgorithmforobtainingNEofn-playermatrixgamesisasignicantcontributiontothelitera-ture.ExtensivenumericalexperimentationispresentedinChapter5,whichdemon-stratestheabilityofthelearningalgorithmtoobtainNashequilibrium.Thissection4

PAGE 16

includessixteenmatrixgameswithuptofourplayersandsixtyfouractionsforeachplayer,followedbyanexampleofarestructuredpowernetworkwithcompetinggen-erators.ThenumericalresultsindicatethatthelearningbasedapproachpresentedinthisresearchholdssignicantpromiseinitsabilitytoobtainNEforlargen-playermatrixgames.Toourknowledge,thealgorithmistherstofitskindthatharnessesthepowerofstochasticvalueapproximationmethodthathasbeensuccessfullyusedinsolvinglargescaleMarkovandsemi-Markovdecisionprocessproblemswithsingledecisionmakers[11,12,13].AformalproofestablishingtheconvergenceofthealgorithmtoNashequilibriumsolutionsisnotfullydevelopedyet,andiscurrentlybeinginvestigated.However,asdiscussedinChapter6,theempiricalevidenceclearlyindicatesthealgorithms'abilitytoconvergetoNEsolutions.1.4DissertationOutlineAbriefoverviewofpowermarketequilibriaandgenerationexpansionplanningliteraturecanbefoundinChapter2.SomefundamentalconceptsofgametheoryandsolutionapproachestogametheoryproblemsasfoundinpowermarketliteraturearepresentedinChapter3.Thecomprehensivetwo-tiermatrixgamemodeladdressingGEPinrestructuredsettingsisdevelopedinChapter4.Tosolvetheseembeddedmatrixgames,wedevelopavaluefunctionapproximationbasedreinforcementlearn-ingalgorithm,whichispresentedindetailinChapter5.AnempiricalanalysisoftheperformanceoftheRLalgorithmispresentedinChapter6.Thesolutionframeworkusedtosolvethetwo-tierGEPmodelispresentedinChapter7.Chapter8consistsofademonstrationoftheapplicabilityofthemodelviaasampleproblem.Thechaptercontainsdetailedexaminationofmodelresultsastothenatureofexpansionplans,generatorpayos,andnodalprices,forgivendemandgrowth.Aregressionmodelisdevelopedtoidentifythefactorsaectingnodalpricespost-expansion.Thischapter5

PAGE 17

alsoexaminesconsumersurplusesunderstrategicbidding,andchoiceofgeneratorexpansionplansundervaryingriskpreferences.ConcludingremarksbasedonthisworkarepresentedinChapter9andsomefutureresearchdirectionssuchastheinclu-sionofcap-and-tradeprogramsandrenewableportfoliostandardsforCO2emissioncontrolarereservedforChapter10.6

PAGE 18

CHAPTER2LITERATUREREVIEW2.1GenerationCapacityExpansionPlanningUntilthelatenineties,asignicantnumberofpapersappearedintheliteratureexaminingthegenerationexpansionplanningGEPprocessinregulatedelectricitymarkets.Someofthekeycontributionsare[14,15,16,17,18,19].AnexcellentreviewpaperbyZhuandChow[20]discussesbothmathematicalprogrammingbasedandheuristicbasedtechniquesusedtosolveGEPproblemsinregulatedsettings.MostofthepaperslistedaboveformulatedtheGEPproblemwiththeobjectiveofminimiz-ingproductionandcapitalcosts.Ontheotherhand,GEPinrestructuredmarketsneedstobemodeledasanoncooperativegame,wherethegeneratorscompetetomax-imizetheirprots.Inbothcases,however,theconstraintshavetoincludecapacity,transmission,energybalance,investment,andsystemreliability.WhilecommercialsoftwarelikeWienAutomaticSystemPlanningWASP[21]andElectricGenerationExpansionAnalysisSystemEGEAS[22]existtoaddressGEPinregulatedmar-kets,GEPresearchinderegulatedmarketsisstillinitsearlystages.Hence,thereisaneedandroomformoreresearchtofullyexploreandunderstandtheGEPprobleminthecurrentcompetitiveenvironment.ThecurrentcompetitiveenvironmenthasintroducedsomerathernewchallengesinthealreadycomplexGEPproblem.Firstofwhichisthemodelingofcompetitivebehaviorofgenerators.Second,thegener-atorsinvestinginthemarkethavetoconsiderindividualrisksduetovolatilitiesin7

PAGE 19

prots.Third,emissionsandenvironmentalregulationsaresettobecomecriticallyimportantinthecomingyears,duetowhich,comprehensivemodelsincorporatingcap-and-trade/carbontradingmechanismswillbecomenecessary.Inthenextfewparagraphs,wefocusourattentiononsomeoftherecentresearchcontributionsaddressingGEPinrestructuredmarkets.Thepapersthatwehavechosentoreviewhere,helptohighlightthesimilaritiesthatourmodelshareswiththeliterature,aswellasthedistinctions.Chuangetal.[23]presentedoneoftheinitialGEPmodelsinarestructuredsetting.TheymodelGEPasaCournotgamebymakingthefollowingmainassumptions:generatorscompeteonlyinquantities,newentriesdonotoccurinthemiddleofthegame,andallgeneratorsmakeinvest-mentdecisionssimultaneously.Whilewealsomakethelattertwoassumptions,weusesupplyfunctioncompetitioninsteadofCournotcompetitiontomoreaccuratelyrepresentpowermarketbidding.Chuangetal.computethepriceandquantityal-locationsofgeneratorsusingtheCaliforniaISO/PowerExchangePXsystem,thatbuysandsellsenergythroughauctions.Finally,thesolutionoftheGEPCournotgameisobtainedusingasimpleiterativesearchprocedure.Weuseanoptimalpowerowformulationtoobtainprice-quantityallocationsandthencomputethegeneratorprots.Thereafter,avaluefunctionapproximationbasedlearningalgorithmisusedtondthesolutionofthegame.MurphyandSmeers[24]presentthreedierentGEPmodels.Therstmodel,whichconsidersperfectcompetition,isdevelopedtoserveasabaselinecaseforcom-parisonagainsttheothertwomodels.ThesecondmodelisanopenloopCournotmodelwhereinvestmentdecisionsandpowerdispatchoccursimultaneously.Thethirdmodelisa2-stageequilibriumproblemwithinvestmentsinstage1andpowerdispatchinstage2.This2-stagemodelisanextensionoftheMPECtypeproblems,whichwhilerealistic,areoftenextremelydiculttosolveandarefraughtwithconver-8

PAGE 20

Table2.1.ImportantModelingAttributesfromGEPLiterature Authors Model Risk Emissions TransmissionConstraints SystemRelia-bility MultiyearHorizon DemandVaria-tions Technology Chuangetal.[23] CournotGame No No No Yes No No Yes MurphyandSmeers[24] CournotGame No No No No No No Yes Kaymazetal.[25] CournotGame No No Yes Yes No No No JirutitijaroenandSingh[26] Optimization No No Yes Yes No No No Kimetal.[27] CournotGame No No No Yes Yes No Yes Ngetal.[28] CournotGame No No Yes Yes No No Yes EhrenmannandSmeers[29] CournotGame No No No No Yes No No EhrenmannandSmeers[29] StackelbergGame No No No No Yes No No Botterudetal.[30] optimization Yes No No No No No No gencerelatedchallenges[25].Inthisresearchweadoptthestrategyofsimultaneousinvestmentandpowerdispatch,similartothesecondmodelin[24].Oneoftheimportantfeaturesmissingfromthemodelsin[23]and[24]istheconsiderationoftransmissionconstraints.Kaymazetal.[25]includetransmissionconstraintsandextendHobbs'sLCPformulation[31]forpowermarketsbyincorpo-ratingGEP-relateddecisionvariablesintheobjectivefunction.Theirmodelresults,asexpected,showthattransmissionconstraintsaectthecapacityexpansionde-cisions.Theyalsoshowthattransmissionconstraintsadverselyaecttheconsumerbenets,whichoftentendtoreduceduetohighelectricitypricesandcongestionrents.Inthisresearchwealsoconsidertransmissionconstraints.Forthesakeofbrevity,wesummarizeotherrelevantGEPliteratureandtheirkeymodelingattributesinTable2.1.2.2NashEquilibriaofMultiplayerGamesGlobalizationhasplayedasignicantroleoverthelastdecadeintransformingthemarketplaceintoonewheremostgoodsandservicesaretransactedthroughmulti-partycompetition.Consequently,thestudyofgametheoreticconceptsandthede-9

PAGE 21

velopmentofeectivemethodsforsolvingmultiplayergameshavegainedincreasingattentionintheresearchliterature.Gamesoccurintwoprimaryforms:matrixgamesandstochasticgames.Ann-playermatrixgameischaracterizedbyndierentre-wardmatricesoneforeachplayerandasetofactioncombinationscharacterizingtheequilibriaNash-equilibria,inparticular.Nash[32]denedequilibriumtobeanactioncombinationfromwhichnosingleplayercouldunilaterallydeviatetoincreaseprot.Stochasticgamesarecomprisedofniteorinnitehorizonstochasticprocesseswithnitestatesandstatetransitionprobabilitystructure,inwhichtheplayersseekequilibriumactionsforeverystatesoastomaximizetheirrewardsfromtheoverallgame.Therefore,stochasticgamesareconstruedassequenceofmatrixgamesoneforeachstateconnectedwithtransitionprobabilities.Furtherclassicationofgamesarisesfromthenatureofrewardstructure:zerosumgamesandnonzerogeneralsumgames.Rewardsofstochasticgamesareclassiedasdiscountedreward,averagereward,andtotalreward.Thoughthefundamentalsofgametheoryarefairlywellestablished[32],thecomputationaldicultiesassociatedwithndingNashequilibriahaveconstrainedthescopeoftheresearchliteraturelargelytothestudyofbimatrixgameswithlim-itedactionchoices.Evenintheabsenceofsucienttoolstoappropriatelyanalyzestochasticormatrixgames,amajorityofthemarketplaceshaveevolvedtoincor-poratetransactionsthroughcompetition.Therefore,toensurehealthygrowthofthecurrentcompetitionbasedeconomy,itisimperativetodevelopcomputationallyfeasibletoolstosolvelargescalestochasticandmatrixgames.Inrecentyears,re-searchershavebeenabletocharacterizeequivalentmatrixgamesforbothdiscountedrewardandaveragerewardstochasticgames[33,34,35,36].Theyalsoharnessedtheadvancesinreinforcementlearningbasedtechniquestoconstructtheseequivalentmatrixgames[33,34].However,obtainingtheNashequilibriumfortheseequivalent10

PAGE 22

matrixgameshasremainedanopenresearchissue,whichisoneofthefociofthisresearch.Asdiscussedin[37],theappropriatemethodofcomputingNashequilibriaofamatrixgamedependsonwhetheritisrequiredtondoneorallequilibriumpoints,thenumberofplayersinthegame,andtheimportanceofthevalueoftheNashequilibrium.Nocomputationallyviablemethodaddressingalloftheaboveisavail-ableinexistingliterature.Nashequilibriaofn-playermatrixgamescanbeobtainedbysolvinganonlinearcomplementarityproblemNCP,whichfora2-playermatrixgamebecomesalinearcomplementarityproblemLCP[37].LemkeandHowson[38]developedanecientalgorithmforobtainingNashequilibriaforbimatrixgamesbysolvingtheassociatedLCP.TheiralgorithmwasextendedforndingNashequi-libriaofn-personmatrixgamesin[39]and[40].However,thesealgorithmsstillhaveunresolvedcomputationalchallenges.Mathiesen[41]proposedamethodofsolvingNCPforn-playermatrixgamesthroughasequenceofLCPapproximations.Asurveyby[42]summarizestheseandotherdevelopmentsonthistopic.Itmaybenotedthatthesemethodsarenotguaranteedtoobtainglobalconvergenceandoftendependonthechoiceofthestartingpoint.Toourknowledge,theonlyopenlyavailablesoftwarethatattemptstosolvemultiplayermatrixgamesisGAMBIT[43].However,asobservedbyLeeandBaldick[44],thissoftwaretakesanunusuallylongcomputationtimeasthenumberofplayersandtheiractionchoicesincrease.Gametheoreticmodelshavebeenstudiedextensivelyinexaminingmarketcom-petitionintheenergyandtransmissionsegmentsofrestructuredpowermarketsasinPennsylvania-Jersey-Maryland,NewYork,NewEngland,andTexas.Thesegamesarecharacterizedbymultidimensionalbidvectorswithcontinuousparameters.Uponsuitablediscretizationofthesebidvectors,manyofthesegamescanbeformulatedasmatrixgames.Thedegreeofdiscretizationdictatesboththecomputationalburden11

PAGE 23

andtheprobabilityofidentifyingtheNashequilibria.Almostalloftheliteraturestudyingpowermarketgamesisdevotedtooptimizationbasedapproaches,suchasmathematicalprogramming[45,46,47],co-evolutionaryprogramming[48],andexhaustivesearch[49].Eveninalimitednumberofstudies,wheresuchgamesareformulatedasmatrixgames,numericalexamplesareconvertedtobimatrixgamesandaresolvedusinglinearprogrammingandLCPapproaches[44,50,51].MathematicalprogrammingapproachtondingNEofmatrixgameshastwopri-maryvariants:mathematicalprogramwithequilibriumconstraintsMPEC,[52],andequilibriumproblemwithequilibriumconstraintsEPEC,[53].MPECisageneralizationofbilevelprogramming,whichinturnisaspecialcaseofhierarchi-calmathematicalprogrammingwithtwoormorelevelsofoptimization.MPECsresembleStackelbergleader-followergames,whichformaspecialcaseoftheNashgame.InaNashgameeachplayerpossessesthesameamountofinformationaboutcompetingplayers,whereas,inStackelbergtypegames,aleadercananticipatethereactionsoftheotherplayers,andthuspossessesmoreinformationinthegame.TheleaderinaStackelberggamechoosesastrategyfromhis/herstrategyset,andthefollowerschoosearesponsebasedontheleadersactions[52],whileinaNashgameallplayerschooseactionssimultaneously.WhenmultipleplayersfaceoptimizationproblemsintheformofMPECs,EPECmodelshavebeenusedtosimultaneouslyndtheequilibriaoftheMPECs[47,53,54,55].MPEC,LCP,andEPECproblemsarediscussedbrieyinthenextchapteralongwithsomegametheoryfundamentals.12

PAGE 24

CHAPTER3BASICCONCEPTSTheobjectiveofthischapteristocoversomefundamentalsofgametheory,rein-forcementlearning,anddiscussequilibriumstrategiesfromtheperspectiveofpowermarketoperations.Wepresentadetailedreviewofsomepapersthatdevelopmethod-ologiestoobtainpowermarketequilibria.3.1BasicConceptsofGameTheoryGametheoryexaminesthebehaviorofrationalplayersininteractionwithotherrationalplayers.Playersareconsideredtoberationaliftheymaximizetheirobjectivefunctionsgiventheirbeliefsabouttheenvironment.Inagametheoreticsetting,play-ersactinanenvironmentwhereotherplayers'decisionsinuencetheirpayos.Theconceptofstrategyasacompleteplanofactionprovidesanapproachformodelingbe-haviorthattakesinformationalaswellasdynamiccharacteristicsoftheenvironmentintoaccount.3.1.1ZeroSumandNon-ZeroSumGamesGamescanbeclassiedbasedonpayostructureaszerosumgamesandnon-zerosumgames.Atwoplayerzero-sumgameisagameinstrategicformsuchthatp1s1;s2+p2s1;s2=0;8s12S1;s22S2.113

PAGE 25

wherep1,p2arethepayofunctionsoftwoplayersandS1andS2arethepurestrategysetsofthetwoplayers.Fromtheabovedenition,itisseenthatzerosumgamesarestrictlycompetitivewhichmeansthatwhatoneplayergainstheotherloses.Innon-zerosumgamessomeoutcomesaremorefavorabletosomeplayersthanothers.Someoutcomesmayevenyieldapositivepayoandothersanegativepayoforeveryplayer.Thisintroducesacertaincommoninterestamongplayerstoattainsuchmorefavorableoutcomeseveniftheyarenotthemostfavorableoutcomesforeveryone.Suchgamesarenon-strictlycompetitivesincetheyhavebothcompetitiveandcooperativeelements.3.1.2PureandMixedStrategyTheconceptofstrategyisfundamentaltogame-theoreticanalysisasitprovidesacompleteplantotheplayerforhowtoplaythegame.Whenplayersplayeachstrategywithprobabilityone,thentheplayersaresaidtohaveapurestrategy.Amixedstrategysimplymeansthattheplayersrandomlychooseapurestrategy.Thusamixedstrategyisaprobabilitydistributiononthesetofpurestrategies.Thesetofmixedstrategiesalwaysincludesallpurestrategiesbecauseapurestrategycanbeconsideredasaspecialcaseofamixedstrategyinwhichtherespectivepurestrategyisplayedwithprobabilityoneandanyotherpurestrategywithprobabilityzero.3.2EquilibriuminPowerMarketGamesParticipantsoftheenergymarketattempttomaximizetheirbenetsbyseekingoptimalbiddingstrategies.Agenericversionofthebiddingstrategyformulationprobleminapowernetworkcanbegivenasfollows.LetBdenotethesetofbusesinthenetwork,andBsBdenotethesubsetofsupplybusesnodes.Letthenumberofgeneratorsatasupplybusi2BsbedenotedbyNi,andMdenotethenumber14

PAGE 26

ofloadsinthenetwork.LetGi=f1;2;;NigandL=f1;2;;Mgdenotethesetofgeneratorsatasupplybusiandthesetofloadsinthenetworkrespectively.LetN=PNi,andG=[Gi.Tokeeptheexpositionsimple,weconsideronlygeneratorsidebiddinginthemarket.LetthestateofthenetworkattimetXtbethevectorofrealizedloadsdemandsqstandpricespst.Hence,Xt=fqst;pstg,whereqst=q1t;q2t;;qjBjtandqstdenotestherealizedhourlyloadquantityvectoratthesthbus,s2B.Also,pst=p1t;p2t;;pjBjt,wherepstrepresentstherealizedhourlypricevectoratbuss2B.LetthebiddecisionvectoratthetthtimebegivenbyDt=fDlt:l2Gg,whereDltisthedecisionvectorofgeneratorlandDlt2fDlgandDldenotesthesetofallbidparametersvectorsforgeneratorl.Thesebidparametersdependonthena-tureofbids,forexample,polynomialfunctionsandpiecewiselinearfunctions,anddeterminetheoerpricescorrespondingtothegenerationquantities.Thebiddingprocessinvolvesselectionofbidparametersbythegenerators,whoseektomaximizetheirindividualprotsfortheforecastedstateofthenetworkXt.Theprotscor-respondingtoasetofbidssubmittedatanytimetbythegeneratorsareobtainedbysolvingtheoptimalpowerowOPFmodel.Theprotmaximizationproblemforgeneratorj,ascommonlypresentedintheliterature,canbestatedasabi-levelproblemasfollows.ChooseDjt;soastomaximizeprotgfjt;Pjt;subjecttochoiceofotherbiddersDlt:l2GnjandtheOPFProblemanditsconstraints.Where,fjtandPjtarethenodalclearingpricecostofpowergenerationandquantityallocationforgeneratorjasdeterminedbytheOPFmodel,whichisprovidednext.OPFmodelsareformulatedeithertomaximizesocialwelfareortominimizethetotalcostofmeetingthepowerdemandofanetwork.TheOPFmodelsimultaneouslysatisesseveralsystemrelatedconstraintssuchasdemandandsupplyconstraints,voltageconstraints,thermallimitconstraints,andtheconstraintsofpowerow.Sev-15

PAGE 27

eralpaperspresentedtotheliteratureutilizeaDCversionoftheOPFmodeltocurtailthecomputationalcomplexityinvolvedinsolvinganAC-OPFmodel.Weadoptasimilarapproachinourwork.However,forthesakeofcompleteness,weprovidebelowagenericmathematicalformulationofthecostminimizationversionoftheAC-OPFmodel.Letfjtdenotethecostofactivepowergenerationbysupplierjatadecisionepoch.Also,letPjtandQjtdenotetheactiveandthereactivepowergenerationquantitiesrespectively.minXj2BsfjtPjt.2subjectto:Xj2BsPjt)]TJ/F20 11.955 Tf 11.955 0 Td[(l)]TJ/F20 11.955 Tf 11.956 0 Td[(lV;=0;.3Xj2BsQjt)]TJ/F15 11.955 Tf 11.879 3.155 Td[(~l)]TJ/F15 11.955 Tf 11.879 3.155 Td[(~lV;=0;.4Sy;zSmaxy;z8y6=z2fBg.5VminwVwVmaxw;8w2fBg;B=fsetofbusesg:.6PjminPjtPjmax;8j2fBsg.7QjminQjtQjmax;8j2fBsg.8Constraint3.3intheOPFmodelensuresthatalltheactivedemandlandtheactivetransmissionlosseslV;aremetbythegeneratorsselectedfordispatchatanygiventimeactivepowerbalanceequation.Theconstraint3.4ensuresthatallthereactivedemand~landthereactivetransmissionlosses~lV;aremetbygeneratorsselectedfordispatchreactivepowerbalanceequation.ThetermSy;zinequation3.5denotestheowlimitforthepowertransmittedfromBusytoBusz.Constraint3.5ensuresthatthemaximumowlimitconstraintsinbothdirectionsarenotviolated.Theconstraint3.6isusedtomaintainthevoltagelimitsforeachBus.16

PAGE 28

Constraints3.7and3.8areusedtomaintainactiveandreactivepowergenerationlimits.3.3SolutionStrategiesNotethatthebi-levelbiddingstrategyproblemispresentedabovefromtheper-spectiveofprotmaximizationofgeneratorj.But,therequirementoftheknowledgeofbidchoicesoftheotherplayers,asstatedintheconstraintset,makesthebi-levelproblemunsolvableinaderegulatedmarket,wherebidchoicesarenotknownapriori.Thus,theoptimalgeneratorbidsshouldbederivedfromtheNashequilibriumstrate-giesofthegame.However,nonavailabilityofcomputationallyviabletoolstosolveforNashequilibriaofmultiplayergameshadmotivatedresearcherstolookforalter-nativeapproachestoobtainoptimalbiddingstrategies.Forthepurposeofexaminingtheexistingliterature,weclassifythesecontributionsintotwomajorcategories:ap-proachesthatoptimizeindividualstrategiesforgivenstrategiesofotherplayers,andapproachesthatseekequilibriumstrategies.3.3.1OptimizationofIndividualBiddingStrategiesSeveraldierentoptimizationapproacheshavebeenusedforthistaskincludinggeneticalgorithms[56],[57],[58],evolutionaryprogramming[59],MonteCarlosimulation[60],dynamicprogramming[61],[62],andmathematicalprogramwithequilibriumconstraints[52,46].Inwhatfollows,wereviewthekeycontributionsandlimitationsoftheabovepapers.Theworkpresentedin[56]oersageneticalgorithmGAapproachtooptimizingprotsofindividualgeneratorshavingmultiplegeneratingunits.Solutionofindivid-ualgeneratorprotsareobtainedbyassumingthatthebidsofotherplayersareknownintheformofprobabilitydistributionfunctions.GAisusedasameansto17

PAGE 29

navigatethroughthelargeactionsspacesDjoftheindividualgeneratorsj2Gwhileconsideringrandomizedbiddingbehavioroftheotherplayers.Thesolutionsthusobtaineddonothaveanyequilibriumproperties,sinceinanoncooperativebiddingenvironment,norationalgeneratorcanbeexpectedtobehaverandomlyguidedbyaprobabilitydensityfunction.Asaresult,theexpectedgeneratorprotscalculatedbythealgorithmareunlikelytobeeverrealized.Attaviriyanapapetal.[59]presentanevolutionaryprogrammingapproachtondingbiddingparametersthatmaximizeindividualgeneratorsprots.Theauthorsattempttoobtainoptimalbiddingstrategiesofasupplierwhoownsmultiplegen-eratingunits.TheclearingpriceftjisobtainedusingaPX-typemarketsettlementsimplematchingofsupplyanddemandcurvesfor24hoursoftheday.TheroleofEPinthispaperistosimplysearchthroughthedecisionspaceforprotablebids.DuetolackofconsiderationofOPFandtransmissionconstraintsliketheEquations3.3-3.8theuseofsuchmodelsinrealpowermarketsisineective.WenandDaviduseaMonteCarloMCsimulationmethodtoobtainoptimalgeneratorbiddingstrategiesin[60].In[60],theauthorsconsiderrivalsbidsDtl:l2fGnjgtobeavailableintheformofprobabilitydensityfunctionsandsubsequentlyuseMCsimulationtoobtainrandomsamplesfromthesebidpdf's.Thesesamplesarethenconsideredtobexedintheoverallgeneratorbiddingstrategyproblem.Then,anelementarysearchtechniqueknownasgoldensectionmethodusedinndingtheprotmaximizingbid.However,itmayberemarkedherethattheassumptionofprobabilisticestimationofrivalsbidsaectstheabilityofthisapproachtoattaintrueoptimality.RajaramanandAlvarado[61]presentadeterministicnesteddynamicprogram-mingDPapproachofndingoptimalbiddingstrategiesformultiperiodpowermarketproblems.DP-basedapproachesaresuitableforsmallscaleproblemswhere18

PAGE 30

Table3.1.SomeImportantModelingAttributesfromBiddingStrategyLiterature SolutionMethodology OverallProblemStructure #ofbuses MarketClearing TypeofBids Geneticalgorithms[56] Twoleveloptimization 9-bus DC-OPF linearsupplyfunctions Geneticalgorithms[57] Traditionaloptimization 24-Bus PX-type linearsupplyfunctions MPEC[46] Bi-leveloptimization 30-Bus DC-OPF linearsupplyfunctions Evolutionarypro-gramming[59] Traditionaloptimization 10-Bus PX-type linearsupplyfunctions MonteCarlosimu-lation[60] Stochasticoptimization 6-Bus DC-OPF linearsupplyfunctions Dynamicprogram-ming[62] Twoleveloptimization 5-Bus PX-type stepfunctionbidcurve decisionsfromoneperiodaectthedecisionsandprotsinsubsequentperiodsdayaheadauctionmarkets.Theauthorsin[61]presentseveralcaseswithconsiderationofhydroandthermalgeneratorsaswellascaseswithpricemakingandpricetakinggenerators.However,theirstudydoesnotconsidermultiplecompetinggeneratorsortransmissionconstraints.Also,theauthorsassumethatthetransitionprobabilitymatricesTPMsarereadilyavailable.However,itiswellknownthatevenforprob-lemsofrelativelysmallsizes,determinationofTPMsbecomesalmostimpossible.Asaresultofsuchcomputationalandmodelinglimitations,theapproachpresentedin[61]cannotbeappliedtolargetransmissionconstrainednetworkshavingmultiplecompetinggenerators.Nevertheless,theDPmodelmayserveasaguidancetoolforindividualgeneratorsindeterminingprotablebiddingstrategies,forverysmallnetworkswithlimitedstatespaces.Hobbsetal.[46],presentamathematicalprogramwithequilibriumconstraintsMPECapproachtondingoptimalbiddingstrategiesofgeneratorsinapowernetwork.Theauthorsassumethatwhilemakingtheirownbidallgeneratorshavecompleteinformationaboutrivalplayers'bids.Abileveloptimizationmodelisformu-lated,whereagenerator'sprotmaximizationproblemattherstlevelissubjectedtotheOPFconstraintsatthesecondlevel.AspartoftheMPECprocedure,the19

PAGE 31

OPFconstraintsarethenreplacedwithequivalentKKTconditionsresultinginalinearcomplementarityproblemframeworkLCP.This2-levelproblem,knownasMPEC,hasamaximizationproblemintherstlevelandequilibriumconstraintsinthesecondlevel.Suchproblemstructureshavebeengainingsignicantattentionlatelyduetotheirwidespreadapplicabilityinavarietyofeldssuchaschemicalengineering,transportationscience,andpowersystemeconomics.Forthisreason,wechosetopresentagenericformulationofanMPECproblembasedon[63].Maxx;y;zx;y;zSubjectto:0Fx;y;z?x0;Gx;y;z=0;z2S;x;y;z2<;.9wherezrepresentsrstlevelvariablesandxandyrepresentsecondlevelvariables,whichmustsatisfyanLCPwithxedvaluesofzfromtherstlevel.Ingeneral,0x?y0isreadasx0,y0,andxy=0.Inthepowermarketcontext,therstlevelvariablesaregeneratorbidssimilartoDltwhichserveasxedparametersinthesecondlevelOPFproblem.TheaboveMPECproblemisanon-convexopti-mizationproblem,whichhastobesolvedusingspecialsolutionalgorithmssuchasthepenaltyinteriorpointPIPmethod.DetailsofthePIPalgorithmcanbefoundin[46].Table3.1presentssomeimportantattributesofbiddingstrategyformulationproblemsavailableinliterature.20

PAGE 32

3.3.2ApproachesSeekingEquilibriumStrategiesInacompetitivepowernetworkwithmultipleparticipants,NashequilibriaNEisthatcombinationofstrategiesfromwhichnomarketparticipanthastheincentivetounilaterallydeviate.Thiscanbemathematicallystatedas:gxj;x)]TJ/F21 7.97 Tf 6.587 0 Td[(jgxj;x)]TJ/F21 7.97 Tf 6.586 0 Td[(j8j.10where,xjistheoptimalbidofaparticipantj,andx)]TJ/F21 7.97 Tf 6.587 0 Td[(jaretheoptimalbidsforallotherparticipants.Asalludedtoearlier,duetononavailabilityofcomputation-allyviableapproachestondNEstrategies,manyresearchershaveapproachedtheproblemfromtwodierentviewpoints:1individualgenerators'protmaximizationperspectivediscussedearlier,and2methodologiesthatsolveforequilibriaofNashgamesbymakingassumptionsaboutthecompetitivebiddingbehaviorofgeneratorsexplainednext.SomeoftheseassumptionsareNash-Cournot,Nash-Bertrand,andNash-supplyfunction,whereallplayersbidsimultaneously.Theseassumptionsareexplainednextfollowedbyadetaileddiscussionoftheequilibriumseekingmethod-ologies.1.Nash-CournotCompetition:UndertheCournotassumptionthegeneratorscompeteonlywithquantities.Eachgeneratorassumesthattheopponentsquantityisxedandthenmakeshis/herownquantitydecision.ThenthegameissolvedforaNash-Cournotequilibrium,wherenogeneratorgainsbyunilaterallydeviatingfromhis/herbidquantity.2.Nash-BertrandCompetition:UndertheBertrandassumptionthegeneratorscompetewithprices.Eachgeneratorassumesthattheopponentspriceisxed21

PAGE 33

andthenmakeshis/herownpricebid.TheNEobtainedundersuchcompetitionistermedasBertrand-Nashequilibrium.3.Nash-SupplyFunctionCompetition:Supplyfunctionsareprice-quantitycurvessubmittedbygeneratorstotheISO.SupplyfunctioncompetitionisoftenarguedtorepresenttheworkingofISO-typepowermarketsmorecloselythanCournotandBertrandtypecompetitions.TheresultingequilibriaareknownasNash-supplyfunctionequilibria.4.StackelbergCompetition:UnlikeintheabovethreeNashgames,incertainoligopolisticsituations,itisassumedthatoneoftheplayershasmoreinfor-mationthantherest.Suchanassumptionleadstotheso-calledStackelberggame.InaStackelberggame,aleader"makesadecisionrst,andthenthefollowers"maketheirdecisionknowingtheleader'sdecision.Suchcompeti-tionhasbeenshowntobeusefulinmodelingoligopolisticmarketswithalargedominatingrmandafewsmallercompetingrms.Eventhoughtheaboveassumptionshavebeenextensivelyusedinbiddingstrat-egyliterature,itmaybenotedthatthepremiseofcompleteinformationaboutrivalsbidsbeforemakingone'sownbiddingdecisionisnotrepresentativeofnon-cooperativepowermarketgames.IntheremainderofthissectionwebrieydiscusssomeapproachestondNEbiddingstrategiesofpowermarketgames:linearcom-plementarityProblemsLCP,equilibriumproblemwithequilibriumconstraints,andreinforcementlearningRLbasedapproach.3.3.2.1LCPAgeneralformulationforlinearcomplementarityproblemsLCPfrom[64]isgivenhere.Theobjectiveistondvariableswandzwherew=w1;;wnT;z=22

PAGE 34

z1;;znTsatisfyw)]TJ/F20 11.955 Tf 12.674 0 Td[(mz=q,andw0;z0andwizi=08i.Hobbs[31]usessuchaframeworktoidentifymarketequilibriainaPOOLCOsetting.Hede-nesmarketequilibriumasthosesetofprices,supply,demand,andlineowsthatsimultaneouslysatisfyeachmarketparticipantsrstorderconditionsformaximizingprotwhilematchingnetworkdemandandsupply.TheLCPframeworkfrom[31]ispresentedhereforexposition.Foraconstrainedoptimizationproblem,astheonegivenbelow,MaxFx;y;Subjectto:Gx;y=0;Hx;y0;x0;.11theKKTconditionscanbewrittenasfollows:x:@F=@x)]TJ/F20 11.955 Tf 11.955 0 Td[(@G=@x)]TJ/F20 11.955 Tf 11.955 0 Td[(@H=@x0;x0;x@F=@x)]TJ/F20 11.955 Tf 11.955 0 Td[(@G=@x)]TJ/F20 11.955 Tf 11.955 0 Td[(@H=@x=0;y:@F=@y)]TJ/F20 11.955 Tf 11.956 0 Td[(@G=@y)]TJ/F20 11.955 Tf 11.955 0 Td[(@H=@y0;:Gx;y=0;:Hx;y0;0;andHx;y=0:.1223

PAGE 35

Theequationsassociatedwiththenon-negativevariablesareknownascomplemen-tarityconditions,andandarethedualvariablespertainingtotheconstraintsGandH[31].HobbsdevelopssuchKKTconditionsandcombinesthemwiththemar-ketclearingconditions.TherstorderKKToptimalityconditionstogetherwiththemarketclearingconditionsformtheLCP.AnequivalentquadraticprogramcanthenbewrittenfortheLCPandsolvedusingstandardsolversavailableinGAMSsoftware.Anotherpaperwhichdiscussespowermarketgames,[65],utilizesthewellestablishedLemke-HowsonalgorithmofsolvingLCPs.In[65],theLCPisformulatedfromabimatrixpowermarketgame.Itmaybenotedthat,whileLCP'shavebeenshownboththeoreticallyandcomputationallytoobtainNEof2-playergames,nonlinearcomplementarityproblemNCPframeworkshaveonlybeentheoreticallypresentedtosolvegameswithmorethantwoplayers.Theproposedapproachesofsolvingmultiplayergames,suchas[39,40],stillhaveunresolvedcomputationalchallenges.3.3.2.2EPECTheMPECoptimizationapproachpresentedearliercanbeextendedtoagametheoreticsettingwithmultiplecompetingplayers,knownasequilibriumproblemwithequilibriumconstraintsEPEC.InEPEC,eachplayerissolvinganMPECproblemsubjecttoasetofcommonOPFconstraints.WeadoptthesamenotationusedintheMPECproblemdiscussedearlier.LetallKplayershavetherstleveldecisionvariableszk,k=1K.TheEPECproblemcannowbestatedasfollows[63].zksolvesMaxx;y;zkkx;y;zk;z)]TJ/F21 7.97 Tf 6.587 0 Td[(kSubjectto:0Fx;y;zk;z)]TJ/F21 7.97 Tf 6.587 0 Td[(k?x0;Gx;y;zk;z)]TJ/F21 7.97 Tf 6.586 0 Td[(k=0;24

PAGE 36

zk2Sk;andx;y;z2<:.13Thevariablesz)]TJ/F21 7.97 Tf 6.587 0 Td[(krepresentoptimalandxedvaluesofopponents.Accordingto[66],therearetwogeneralmethodstosolvetheEPECproblem:obtaintheoptimalityconditionsKKTsforalltheMPECproblemsandsolvethemtogetherasacom-plementarityproblem,oriterativelysolveeachoftheMPECsusingstandardMPECalgorithmslikePIPuntiltheequilibriumsolutionoftheEPECgameisobtained.TheEPECproblemisextremelycomplicatedandmoreoverdoesnotguaranteeanNEsolution.Ifasolutiondoesexit,itiscalledasubgameperfectNashequilibrium.SomegoodapplicationsofEPECmodelshavebeenpresentedin[53,47,54,55].3.3.2.3RLBasedApproachValuefunctionapproximationbasedReinforcementLearningRLapproach,whichwedevelopinthisresearch,tondingNEdierssignicantlyfromthemathematicalprogrammingapproacheslikeEPEC,NCP,andLCP.Unlikeinthemathematicalprogrammingapproaches,whereoneassumescompleteknowledgeofrivalsbids,inourapproach,allplayerscompetesimultaneouslywithoutknowledgeofotherplay-ersactions.Suchaframework,webelieve,representsthetruenoncooperativegameamongstpowermarketparticipants.InChapter5,weusethewellestablishedvalueapproximationmechanismwhichwaspreviouslysuccessfullyemployedinsolvinglargescale,Markovandsemi-Markovdecisionprocessproblemswithasingleplayer,[67],todevelopareinforcementlearningbasedalgorithmthatsolvesforNEofmultiplayernoncooperativegames.Wenextpresentsomebasicsofthereinforcementlearningapproach.25

PAGE 37

3.4BriefOverviewofReinforcementLearningThetheoryofRLisfoundedontwoimportantprinciples:Bellman'sequationandthetheoryofstochasticapproximation[68,69].Anylearningmodelcontainsfourbasicelements:1.Systemenvironmentsimulationmodel2.Learningagentsmarketparticipants3.Setofactionsforeachagentactionspaces4.SystemresponseparticipantrewardsConsiderasystemwiththreecompetingmarketparticipants.Atadecisionmakingepochwhenthesystemisinstates,thethreelearningagentsthatmimicthemarketparticipantsselectanactionvectora=a1;a2;a3A.Theseactionsandthesystemenvironmentmodelcollectivelyleadthesystemtothenextdecisionmakingstatesays0.Asaconsequenceoftheactionvectoraandtheresultingstatetransitionfromstos0,theagentsgettheirrewardsr1s;a;s0,r2s;a;s0,andr3s;a;s0fromthesystemenvironment.Usingtheserewards,thelearningagentsupdatetheirknowledgebaseR-values,alsocalledreinforcementvalueforthemostrecentstate-actioncombinationencountereds,a.TheupdatingoftheR-valuesiscarriedoutslowlyusingasmallvalueforthelearningrate.Thiscompletesalearningstep.AtthistimetheagentsselecttheirnextactionsbasedontheR-valuesforthecurrentstates0andthecorrespondingactionchoices.ThepolicyofselectinganactionbasedontheR-valuesisoftenviolatedbyadoptingarandomchoice,whichisknownasexploration,sincethisallowstheagentstoexploreotherpossibilities.Theprobabilityoftakinganexploratoryactioniscalledtheexplorationrate.Bothlearningandexplorationratesaredecayedduringtheiterativelearningprocess.This26

PAGE 38

processrepeatsandtheagentperformancescontinuetoimproveuntiltheprocessconvergestotheoptimalsolution.ForadetaileddescriptionofRL,itsapplications,andrecentadvances,thereadersarereferredtothetextsbyGosavi[70],andSuttonandBarto[71].Inthenextchapter,wepresentatwo-tiermatrixgame-theoreticmodeltoobtaingenerationexpansionplansforcompetinggenerators.27

PAGE 39

CHAPTER4GENERATIONEXPANSIONPLANNINGMODEL4.1Two-TierMatrixGameModelforGEPThegenerationexpansionplanningmodelthatweproposeinthisresearchcon-sistsoftwotiers,asshowninFigure4.1.Thetoptierofthemodelrepresentstheinvestmentcompetitionamongstgenerators.Thiscompetitivedecisionmakingsce-narioismodeledasamatrixgameandishenceforthreferredtoasinvestmentgame.Thebottomtier,ontheotherhand,representsthecompetitionamongstgeneratorstosupplyelectricityintothenetwork.Thisscenarioisalsomodeledasamatrixgameandisreferredtoassupplyfunctiongame.ItiscalledasupplyfunctionmatrixgameduetothefactthatthegeneratorsareassumedtocompetewithsupplyfunctionsCournotorBertrandcompetitionscanbeusedaswell.Eachstrategycombinationoftheinvestmentgamerepresentsapossiblegenerationcapacityexpansionalterna-tive.Therefore,foreachsuchalternative,thereexistsacorrespondingsupplyfunctiongame,whichwhensolvedallowstheexaminationoftheprotabilityofeachexpan-sionalternative.Wenextpresenttheinvestmentmatrixgamemodeltoptierandsupplyfunctionmatrixgamemodelbottomtierindetail.Wealsoexplainhowthepayomatricesofthesematrixgamesarecalculated,andhowthetwotiersinteractwitheachotherinordertoresultinamulti-year,multi-playergenerationexpansionstrategy.28

PAGE 40

Figure4.1.SchematicoftheTwo-TierGEPModelforaTwoGeneratorScenario29

PAGE 41

4.1.1TopTier:InvestmentGameTheinvestmentmatrixgameisdenedbyatuple.Theelementsofthetupleareasfollows.1.Ndenotesthenumberofgenerators.2.Akdenotesthesetofexpansionalternativesavailabletogeneratork.3.Rk:A1...AN!Risthepayofunctionforgeneratork,whereanelementrka1;:::;aNofRkistheriskconstrainedpayoexplainedlaterofgeneratork.Rkforallk,canbewrittenintheformofN-dimensionalmatricesrepresentingtheinvestmentmatrixgameasfollowsRk=hrka1;a2;;aNi:jA1j;:::;jANja1=1;:::;aN=1.1Thegeneratorsselectexpansionalternativesfromthesetofavailablechoiceswiththegoalofmaximizingtheirpayoswhichdependonallothergenerators'selections.TheconceptofNashequilibriumisusedtodescribeastrategyasbeingthemostrationalbehaviorbythegeneratorsactingtomaximizetheirpayos.So,fortheinvestmentmatrixgame,apurestrategyNashequilibriumisacollectionofexpansionalternativesa=a1;;aN,forwhichrkak;a)]TJ/F21 7.97 Tf 6.587 0 Td[(krkak;a)]TJ/F21 7.97 Tf 6.587 0 Td[(k;8ak2Ak,andk=1;2;;N,whereakindicatestheselectionofanon-Nashequilibriumalternativebythekthgeneratoranda)]TJ/F21 7.97 Tf 6.586 0 Td[(kindicatestheNashequilibriumchoiceofalltheothergenerators.WedevelopedariskconstrainedprotcalculationmodelforobtainingthepayomatricesRk.Thismodelispresentednext.30

PAGE 42

4.1.1.1RiskConstrainedProtCalculationModelInelectricpowermarkets,theamountofrevenueearnedbyagenerator,dependsontheinteractionofseveralfactorssuchasstrategicbiddingbehaviorofthecompet-inggenerators,transmissionconstraints,systemcontingencieslinefailures,generatoroutages,fuelpricevolatilities,anddemandvariations.Forinstance,inatransmis-sionconstrainedpowernetwork,thegeneratorsmaybeabletouselocationinthenetworktotheiradvantagetobidstrategicallyandmakehigherprots.Ontheotherhand,unforseenoutagesandfuelpricevolatilitiesmayadverselyaecttheprots.ConditionalvalueatriskCVaR,alsoknownasexpectedshortfallES,isariskmet-ricthatcanbeusedbygeneratorstocapturesuchvariabilitiesforprotcalculations.Asnotedintheliterature[72,73],CVaRisgainingpopularityinthenanceandinsuranceindustriesasaviableriskmetric.BeforewediscussCVaR,itisimportanttodescribehowitisanextensionofthetraditionallyusedmetriccalledvalue-at-riskVaR.Inadditiontoourwork,toourknowledge,theonlyotherpaperinopenGEPliteraturethatusesCVaRtoaidGEPinvestmentdecisionsis[74].ThefollowingdiscussionaboutVaRandCVaRisbasedon[75].LetZbetherandomvariablewhichindicatesthereturnonaninvestment.Let=A%2;1representapercentageofworstcasescenariosofthereturnontheinvestment.ThentheVaRwithrespecttothezquantileoftheofworstcasescenariosisgivenas,VaRZ=)]TJ/F15 11.955 Tf 11.291 0 Td[(supfzjP[Zz]g:.2However,VaRhastwofundamentaldeciencies:aitisthethresholdoflossesintheworstcasescenarios,anddoesnotprovideanyinformationaboutthoselossesthatmaybesignicantlygreaterthanVaR,andbitdoesnotsatisfytheproperty31

PAGE 43

ofsubadditivity,whichstatesthattheglobalriskofaportfolioisalwayslessthanorequaltothesumoftherisksoftheindividualassetsformoredetailssee[72,73].TheseweaknessesofVaRmotivatedresearcherstodevelopanew,yet,relatedmetriccalledconditionalvalue-at-riskorexpectedshortfall,ES.In[75],CVaRhasbeenshowntoaddressboththeaboveweaknessesofVaR.WhileVaRistheminimumoftheA%worstcasescenarioslosses,CVaRistheaverageoftheA%worstcasescenarios.InotherwordsCVaRistheexpectedvalueoflossesgiventhatthelossesaregreaterthanVaR.In[75]anestimatorforthismeasurewasdenedasfollows:CVaRnZ=)]TJ/F1 9.963 Tf 10.494 16.058 Td[(Pwi=1Zi:n w;.3where,Zi:naretheorderstatisticsofthereturnoninvestmentrandomvariable,w=bnc=max[mjmn;m2N],nisthetotalnumberofscenarios,and2;1isaprobabilityvalue.Weusetheequilibriumprotfromthebottomtiersupplyfunctionmatrixgametocalculatetheriskconstrainedprotforgeneratoriasfollows.i=i)]TJ/F20 11.955 Tf 11.955 0 Td[(iCVaR;.4whereiistheequilibriumprotforgeneratorifromthesupplyfunctiongame.Thetermidenotestheriskpreferenceofgeneratorivaryingbetween0and1,andCVaRisasdescribedinEquation4.3.4.1.2BottomTier:SupplyFunctionGameThesupplyfunctiongameisdenotedbythetuple:.Theelementsofthetupleareasfollows.32

PAGE 44

1.Ndenotesthenumberofgenerators.2.~Akdenotesthesetofsupplyfunctionbidchoicesavailabletogeneratork.3.~Rk:~A1:::~AN!Risthepayofunctionforgeneratork,whereanelement~rkb1;:::;bNof~Rkistheprotofgeneratorkwhenthegeneratorschoosesupplyfunctionbidsb1throughbN.~Rkforallk,canbewrittenintheformofN-dimensionalmatricesrepresentingthesupplyfunctionmatrixgameasfollows~Rk=h~rkb1;b2;;bNi:j~A1j;:::;j~ANjb1=1;:::;bN=1.5Thegeneratorsselectbidsfromthesetofavailablesupplyfunctionbidchoiceswiththegoalofmaximizingtheirpayoswhichdependonallothergenerators'bids.ThepurestrategyNashequilibriumforthesupplyfunctiongameisdenedasthatbidchoiceproleb=b1;;bN,forwhich~rkbk;b)]TJ/F21 7.97 Tf 6.586 0 Td[(k~rkbk;b)]TJ/F21 7.97 Tf 6.587 0 Td[(k;8bk2~Ak,andk=1;2;;N.Thegeneratorprots~rkb1;b2;;bNconstitutingthesupplyfunctiongamearecalculatedasfollows[76].~rkb1;b2;;bN=1=2[pi)]TJ/F20 11.955 Tf 11.955 0 Td[(xi+pi)]TJ/F15 11.955 Tf 11.955 0 Td[(xi+yiqi]qi;.6where,piandqiaretheoptimalpriceandquantityallocationsforbidchoicesb1,b2,,bN.TheseoptimalpriceandquantityallocationsareobtainedbysolvingalinearizedDC-OPFmodel,whichispresentednext.NotethatiinEquation4.4istheequilibriumprotofthesupplyfunctiongame,obtainedasi=1=2[pi)]TJ/F20 11.955 Tf 11.955 0 Td[(xi+pi)]TJ/F15 11.955 Tf 11.955 0 Td[(xi+yiqi]qi;.733

PAGE 45

wherexi,yiarecostfunctionparameters,andpi,qiaretheequilibriumpriceandquantityallocationsofgeneratori.4.1.2.1OptimalPowerFlowModelTheoptimalpowerowmodelusedinthisresearchisadoptedfrom[76].TheindependentsystemoperatorISOreceivessupplyanddemandfunctionsfromthemarketparticipantsandthensolvesasocialwelfaremaximizationproblem.TheOPFcomputesoptimalpriceandquantityallocationsateachbusofthenetwork,whilesatisfyingsystemsecurityandtransmissionrelatedconstraints.TheDC-OPFmodelusedhereisrathersimpleandallowsforeasiereconomicinterpretationsthannonlinearAC-OPFmodels.WeassumethatgeneratorssubmitlinearsupplyfunctionstotheISO.Thesupplyfunctionshavethefollowinggeneralform:pi=xi+yiqi;8i2G;.8where,Gisthesetofgenerators,pi$/MWHandqiMWHarethepriceandquantityrespectively,andxi,yiaretheinterceptandslopeofthelinearsupplyfunction.WeassumethatconsumerssubmitdecreasinglineardemandfunctionstotheISO.Thedemandfunctionshavethefollowinggeneralform:pj=xj)]TJ/F20 11.955 Tf 11.955 0 Td[(yjdj;8j2C;.9where,Cisthesetofconsumers,pj$/MWHanddjMWHarethepriceandquantityrespectively,andxj,yjaretheinterceptandslopeofthedemandfunction.34

PAGE 46

Asstatedearlier,thisDC-OPFformulationisadoptedfromBerryetal.[76].Bh[ph]isthetotalbenettotheconsumersandCh[Ph]isthetotalcosttothegener-atorssee[76]fordetails.QhisthetotalamountofpowersuppliedbyallgeneratorsandDhisthetotalamountofpowerdemandedbyallconsumers,atbush.Rhkisthereactanceonthepathfrombushtok,thkisthepowerowingfrombushtok,qi[ph]isthepowersuppliedbysupplieriatthepriceph,anddj[ph]isthequantityofpowerdemandedatpriceph.Assumingthatsupplyanddemandbidssubmittedbythegeneratorsandconsumersarelinear,thisbecomesanoptimizationproblemwithaquadraticobjectivefunctionsubjecttolinearconstraints4.11-4.16.Constraints4.13and4.14helptosatisfyKirchho'scurrentandvoltagelawsrespectively,whileconstraints4.15and4.16areusedtosatisfytransmissionlimits.MaxTW[P]=XhBh[ph])]TJ/F1 9.963 Tf 11.955 9.963 Td[(XnCh[ph].10Subjecttoconstraints:Qh)]TJ/F1 9.963 Tf 16.199 9.962 Td[(Xi2ihqi[ph]=08nodesh.11Dh)]TJ/F1 9.963 Tf 17.2 9.962 Td[(Xj2jhdj[ph]=08nodesh.12Qh)]TJ/F20 11.955 Tf 11.955 0 Td[(Dh)]TJ/F1 9.963 Tf 17.938 9.963 Td[(Xk2khthk)]TJ/F20 11.955 Tf 11.955 0 Td[(tkh=08nodesh.13Xhk2AvRhkthk)]TJ/F20 11.955 Tf 11.955 0 Td[(tkh=08voltageloopsv.14thkThk8arcshk.15thk08arcshk.16ThepayosforeachgeneratorcalculatedfromthesolutionoftheOPFmodelareusedtopopulatetheN-dimensionalpayomatricesforthesupplyfunctiongame.Then,thereinforcementlearningalgorithm,wedevelopinChapter5see[10],isusedtoobtaintheequilibriumbidsandcorrespondingpriceandquantityallocations.35

PAGE 47

Theseallocationsareusedtocomputetheriskconstrainedprots,which,asexplainedbefore,formtheN-dimensionalpayomatricesoftheinvestmentgame.Then,thereinforcementlearningalgorithmChapter5isusedtoobtaintheequilibriumex-pansionplanforagivenyearforallgenerators.Thisprocessisrepeatedoneyearatatimetoobtainthemulti-year,multi-player,generationexpansionstrategy.Inthenextchapterweshowhowthematrixgamessuchasthoseencounteredinbothtiersarethefundamentalbuildingblocksofamuchlargerclassofproblemsknownasstochasticgames.Wethendevelopavaluefunctionapproximationbasedlearningalgorithmtosolvethesematrixgames.Later,thesolutionsobtainedbythealgorithmarebenchmarkedagainstthoseobtainedbyacommercialmatrixgamesolver.36

PAGE 48

CHAPTER5REINFORCEMENTLEARNINGBASEDSOLUTIONALGORITHMFORMULTIPLAYERMATRIXGAMES5.1MatrixGamesAmatrixgamecanbedenedbyatuple.Theelementsofthetupleareasfollows.1.ndenotesthenumberofplayers.2.Akdenotesthesetofactionsavailabletoplayerk.3.rk:A1:::An!Risthepayofunctionforplayerk,whereanelementrka1;:::;anisthepayotoplayerkwhentheplayerschooseactionsa=a1;;an.~Rkforallk,canbewrittenasann-dimensionalmatrixasfollows~Rk=hrka1;a2;;ania1=jA1j;:::;an=jAnja1=1;:::;an=1:.1Theplayersselectactionsfromthesetofavailableactionswiththegoalofmaxi-mizingtheirpayoswhichdependsonalltheplayers'actions.TheconceptofNashequilibriumisusedtodescribethestrategyasbeingthemostrationalbehaviorbytheplayersactingtomaximizetheirpayos.Soforamatrixgame,apurestrategyNashequilibriumisanactionprolea=a1;;an,forwhichrkak;a)]TJ/F21 7.97 Tf 6.586 0 Td[(krkak;a)]TJ/F21 7.97 Tf 6.586 0 Td[(k;8ak2Ak,andk=1;2;;n.TheequilibriumvaluesdenotedbyVal[]forplayer37

PAGE 49

kwithpayomatrices~RkisobtainedasVal[~Rk]=rka1;;an.TheappealingfeatureoftheNashequilibriumisthatanyunilateraldeviationfromitbyanyplayerisnotworthwhile.AmixedstrategyNashequilibriumformatrixgamesisavector1;;n,forwhichwecanwritejA1jXa1=1:::jAnjXan=1kak)]TJ/F21 7.97 Tf 6.587 0 Td[(ka)]TJ/F21 7.97 Tf 6.586 0 Td[(krkak;a)]TJ/F21 7.97 Tf 6.587 0 Td[(kjA1jXa1=1:::jAnjXan=1kak)]TJ/F21 7.97 Tf 6.586 0 Td[(ka)]TJ/F21 7.97 Tf 6.587 0 Td[(krkak;a)]TJ/F21 7.97 Tf 6.587 0 Td[(k;.2where)]TJ/F21 7.97 Tf 6.587 0 Td[(ka)]TJ/F21 7.97 Tf 6.587 0 Td[(k=1a1k)]TJ/F18 7.97 Tf 6.586 0 Td[(1ak)]TJ/F18 7.97 Tf 6.586 0 Td[(1:k+1ak+1nan.AmatrixgamemaynothaveapurestrategyNashequilibrium,butitalwayshasamixedstrategyNashequilibrium[32].ThereexistmethodsforsolvingNashequilibriumofnitenonzero-summatrixgames[37,40,43].Sinceinmatrixgames,therearenotransitionprobabilityfunctions,matrixgamesarestatic.Alsomatrixgamescanbeviewedasrecursivestochasticgameswithasinglestate.Ontheotherhand,stochasticgamescanbeviewedasextensionsofmatrixgamesfromasinglestatetoamulti-stateenvironment.Ageneralsumstochasticgamehasequivalentmatrixgames.Therefore,oncetheequivalentmatrixgamesareestablished,solutionofastochasticgamereducestosolvingthesetofmatrixgamesoneforeachstate.Hence,matrixgamesplayaverycriticalroleforsolvingthisbroadclassofproblems.TheintentofthefollowingsectionistoprovideabriefoverviewofthemainresultsfromtherecentliteratureconcerningtheexistenceofequivalentmatrixgamesforbothdiscountedrewardDRandaveragerewardARstochasticgames.38

PAGE 50

5.1.1EquivalentMatrixGamesforDiscountedRewardStochasticGamesAstochasticgamecanbedenedbyatuple,whichdiersfrommatrixgamesbyhavingthefollowingadditionalelements:1.S:anitesetofstatessoftheenvironment,and2.P:thesetoftransitionprobabilitymatrices,whereps0js;aisthetransitionprobabilityofreachingstates0asaresultofajointactionabyallofthenplayers.Inastochasticgame,thetransitionprobabilitiesandtherewardfunctionsdependonthechoicesmadebyallagents.Thus,fromtheperspectiveofanagent,thegameenvironmentisnonstationaryduringitsevolutionphase.However,forirreduciblestochasticgames,optimalstrategiesconstitutestationarypoliciesandhenceitissuf-cienttoconsideronlythestationarystrategies[36].Wedeneksasthemixedstrategyatstatesforagenti,whichistheprobabilitydistributionoveravailableactionset,Aks,ofplayerk.Thusks=fks;a:a2Aksg,whereks;ade-notestheprobabilityofplayerkchoosingactionainstates,andPa2Aksks;a=1.Then=1;:::;ndenotesajointmixedstrategy,alsocalledapolicy.Apureac-tiona2Aks;acanbetreatedasamixedstrategykforwhichka=1.LetthecardinalityofAksbedenotedbymks.Underpolicy,thetransitionprobabilitycanbegivenasps0js;=m1sXa1=1mnsXan=1ps0js;a1;:::;anns;an1s;a1:.339

PAGE 51

Theimmediateexpectedrewardofplayerkinducedbyamixedstrategyinastatesisgivenbyrks;=m1sXa1=1:::mnsXan=1rks;a1;:::;anns;an:::1s;a1:.4ThentheoveralldiscountedvalueofapolicytoplayerkstartinginstatescanbegivenasVks;=1Xt=0tEsrkt=1Xt=0tXs02Spts0js;rks0;;.5wherept:denotesanelementofthetthpowerofthetransitionprobabilitymatrixP.Thediscountedrewardgivenin5.5canberewrittenincomponentnotationintermsofexpectedimmediaterewardandtheexpecteddiscountedvalueofthenextstateasfollowsVks;=rks;+Xs02Sps0js;Vks0;;.6fromwhichthedenitionofNashequilibriumcanbegivenasrks;+Xs02Sps0js;Vks0;rks;)]TJ/F21 7.97 Tf 6.586 0 Td[(k;k+Xs02Sps0js;)]TJ/F21 7.97 Tf 6.586 0 Td[(k;kVks0;)]TJ/F25 7.97 Tf 6.586 0 Td[(k;k:.7DirectlysolvingforNashequilibriumusingtheinequality5.7isdicult,evenwhentherewardfunctionsandtransitionprobabilitiesareavailable.FilarandVrieze[36]combinedthetheoriesofdiscountedMarkovdecisionprocessesandMatrixgamestodevelopanauxiliarybi-matrixgamefortwoplayerdiscountedstochastic40

PAGE 52

games.Theabovetechniqueisextendedin[34]ton-playergamesforconstructingn-dimensionalequivalentauxiliarymatricesQk:forallplayersk=1;:::;n.TheelementsoftheQk:matricesarepayosforallpossiblepureactionsetsa,whichtakeintoaccountboththeimmediaterewardandthefutureopportunities.Fors2S,thematrixwithsizem1sm2s:::mnsforthekthplayeris:Qks=24rks;a1;:::;an+Xs02Sps0js;a1;:::;anVks0;35a1=m1s;:::;an=mnsa1=1;:::;an=1.8whereVks0;istheequilibriumvalueforthestochasticgamestartingatstates0forplayerk.Notethatthisauxiliarymatrix,Qk:capturestheinformationfromthematrixgameresultingfromthepurestrategiesaswellastheequilibriumpayoofthestochasticgame.Thisenablestheestablishmentoftheconnectionbetweenthematrixgamesanddiscountedrewardstochasticgamesasgivenbythefollowingresultof[34].Inthefollowing,Theorem1,items1and2areequivalent.1.isanequilibriumpointinthediscountedrewardstochasticgamewithequi-libriumpayosV1;:::;Vn.2.Foreachs2S,thestrategysconstitutesanequilibriumpointinthestaticn-dimensionalmatrixgameQ1s;:::;QnswithequilibriumpayosVal[Q1s;];;Val[Qns;].TheentryofQkscorrespondingtoac-tionsa=a1;;anisgivenbyQks;a=ris;a+Ps02Sps0js;aVis0;,fori=1;:::;n,wherea2nQi=1Ais.Wenotethat,theentriesinthismatrixgame5.8havesimilarstructuretotheBellman'soptimalityequationfordiscountedMDP.WellknownalgorithmstosolveBellman'sdiscountedoptimalityequationarevalueiterationandpolicyiteration.41

PAGE 53

Anextensionofthevalueiterationandredenitionofthevalueoperatortosolvestochasticgameswaspresentedin[77].ThereexistlearningalgorithmsthatattempttolearntheentriesoftheQkmatrices.Thematricesareupdatedduringeachstageandareexpectedtoconvergetotheiroptimalforms.MinmaxQ-learningalgorithmfordiscountedzero-sumgamesispresentedin[78].ANashQ-learningfordiscountedgeneral-sumgamesispresentedin[79].BothMinmaxQ-learningandNash-Qlearningalgorithmsareextensionsofthemodel-freereinforcementQ-learning[80,71].Asummaryoftheavailablestochasticgamealgorithmscanbefoundin[81].OneassumptionthatisinherentintheaboveliteratureisthatoncetheequivalentmatricesQkareconstructed,theycanbesolvedusingexistingmethods.However,theexistingmethodsforobtainingNEvalueVal[Qks;]ofn-playern>2matrixgamesarefraughtwithcomputationalandconvergencerelatedchallenges[39,40].DevelopmentofacomputationallyviablemethodofndingtheNEvalueofamatrixgameValQkts;tisstillanopenchallengeandisaddressedinthisresearch.5.1.2EquivalentMatrixGamesforAverageRewardStochasticGamesLetVkdenotethegainequilibriumvalue,andhkdenotethebiasequilib-riumvalueofanaveragerewardstochasticgame.TheaboveequilibriumvaluescanbedenedasVks;=limsupT!11 TT)]TJ/F25 7.97 Tf 6.587 0 Td[(1Xt=0pts0js;rks0;:.9andhks;=limT!1EsT)]TJ/F18 7.97 Tf 6.587 0 Td[(1Xt=0[rkt)]TJ/F20 11.955 Tf 11.956 0 Td[(gk];.10wheregkislong-runexpectedaverage-reward,whichcanbegivenbygk=limsupT!1E1 TT)]TJ/F18 7.97 Tf 6.586 0 Td[(1Xt=0rkt:.1142

PAGE 54

Similartothediscountedgames,forn-playeraveragerewardgames,itisshownin[33]thatn-dimensionalequivalentauxiliarymatricesRk:forallplayersk=1;:::;ncanbeconstructed.Theelementsofthesematricesarepayosforallpossiblepureactionsetsa,whichtakeintoaccountboththeimmediaterewardandthefutureopportunities.Fors2Sthematrixwithsizem1sm2s::mnsforthekthplayercanbegivenbyRks=24rks;a1;:::;an)]TJ/F20 11.955 Tf 11.955 0 Td[(Vk+Xs02Sps0js;a1;:::;anhks0;35a1=m1s;:::;an=mnsa1=1;:::;an=1.12Thefollowingtheoremestablishestheconnectionbetweenaveragerewardirreduciblestochasticgamesandtheaveragerewardmatrixgames[33].Inthefollowing,Theorem2,items1and2areequivalent.1.isanequilibriumpointintheaveragerewardirreduciblestochasticgamewithbiasequilibriumvaluehkandgainequilibriumvalueVkfork=1;2;;n.2.Foreachxeds2S,thestrategysetsconstitutesanequilibriumpointinthestaticn-dimensionalequivalentmatrixgameR1s;;Rnswithbiasequilibriumvaluehks;andgainequilibriumvalueVal[Rks;]fork=1;;n.Sofar,wehavedenedmatrixgamesandpresentedasummaryoftheavailableresultsfrom[34]and[33].Theseresultsshowthatforbothdiscountedandaveragerewardstochasticgames,thereexistequivalentmatrixgames,thesolutionsofwhichprovidetheequilibriumstrategiesandvalues.Clearly,computationallyfeasibleso-lutionmethodologiesformatrixgamesplayafundamentalroleinsolvingalarge43

PAGE 55

classofstochasticgames.Inwhatfollows,wepresentanewalgorithmthatusesareinforcementlearningapproachtosolvematrixgames.5.2FindingNEofMatrixGamesInthissectionwepresentanewapproachtoobtainNashequilibriumofn-playermatrixgames.LetRkadenotetherewardmatrixofthekthplayerofwhichrka1;;anarethematrixelements.DenethevalueofanactionaktoplayerkasVal[Rkak]=Xfa1;;annakgpa)]TJ/F21 7.97 Tf 6.586 0 Td[(k;akrka1;;ak;;an;.13wherepa)]TJ/F21 7.97 Tf 6.586 0 Td[(k;akdenotestheprobabilityofchoiceofanactioncombinationa)]TJ/F21 7.97 Tf 6.587 0 Td[(kbyalltheplayerswhileplayerkchoseactionak.IndecisionmakingproblemswithasingleplayerMDPsandSMDPs,thereexistoptimalvaluesforeachstate-actionpair,whichdeterminetheoptimalactionineachstate[68].Drawingananalogy,formatrixgamesthathavemultipleplayersandasinglestate,weconjecturethatthereexistoptimalvaluesforallactionsoftheplayersthatcanyieldpureandmixedNEstrategies.However,theprobabilitiespa)]TJ/F21 7.97 Tf 6.586 0 Td[(k;akneededtocomputethesevaluesareimpossibletoobtainforreallifeproblemswithoutpriorknowledgeofplayers'behavior.Therefore,weemployalearningapproachtoestimatethevaluesoftheactionsasfollows.Werewrite.13asVal[Rkt+1ak]=)]TJ/F20 11.955 Tf 11.955 0 Td[(t[Rktak]+thrka1;;ak;;ani:.14Thealgorithmpresentedbelowutilizesthevaluelearningscheme.14toderivepureandmixedNEstrategiesforn-playermatrixgames.44

PAGE 56

5.2.1AValueIterationAlgorithmforn-PlayerMatrixGamesWeassumethatthegamehasn-playersandeachplayerkhasasetofAkactionchoices.Hence,ndierentrewardmatricesofsizejA1jjA2jjAnjareavailable.1.Eliminaterowsandcolumnsofthematricesassociatedwiththedominatedstrategies.Adominatedstrategyisonethatwillneverbeadoptedbyarationalplayerirrespectiveofthechoicesofotherplayers.Astrategya2Akforplayerkissaidtobedominatedifrk;a;a)]TJ/F21 7.97 Tf 6.586 0 Td[(krk;a;a)]TJ/F21 7.97 Tf 6.587 0 Td[(k,wherea2Aknaanda)]TJ/F21 7.97 Tf 6.586 0 Td[(kdenotestheactionsofallotherplayers.2.Letiterationcountt=0.InitializetheR-valuesforallplayerandactioncombinationsRk;atoanidenticalsmallpositivevaluesay,0.001.Alsoinitializethelearningparameter0,explorationparameter0,andparameters,neededtoobtainsuitabledecayratesoflearningandexploration.LetMaxstepsdenotethemaximumiterationcount.3.IftMaxsteps,continuelearningoftheR-valuesthroughthefollowingsteps.aGreedyactionselectionforpurestrategyNashequilibrium:Eachplayerk,withprobability)]TJ/F20 11.955 Tf 9.312 0 Td[(t,choosesagreedyactionforwhichRkaRk;a.Atieisbrokenarbitrarily.Withprobabilityt,theplayerchoosesanex-ploratoryactionfromtheremainingelementsofAkexcludingthegreedyaction,whereeachexploratoryactionischosenwithequalprobability.ProbabilisticactionselectionformixedstrategyNashequilibrium:Com-putetheprobabilitiesfortheactionchoicesusingtheratioofR-valuesatiterationtasfollows.Foreachplayerk,theprobabilityofchoosingtheactiona2AkisgivenbyRk;a Pb2AkRk;b.45

PAGE 57

bR-ValueUpdating:UpdatethespecicR-valuesforeachplayerkcorre-spondingtothechosenactionausingthelearningschemegivenbelow.Rt+1k;a)]TJ/F20 11.955 Tf 11.955 0 Td[(tRtk;a+trk;a;.15whereadenotestheactioncombinationchosenbyplayers.cSettt+1.dUpdatethelearningparameterstandexplorationparametertfollowingthedecayschemegivenbyDarkenetal.in[82]:t=0 1+u;whereu=t2 +t!;.16where0denotestheinitialvalueofalearning/explorationrate,andisalargevaluee.g.,106chosentoobtainasuitabledecayrateforthelearning/explorationparameters.Explorationrategenerallyhasalargestartingvaluee.g.,0.8andaquickerdecay,whereaslearningratehasasmallstartingvaluee.g.,0.01andveryslowdecayrate.Exactchoiceofthesevaluesdependsontheapplication[11,12].eIft
PAGE 58

CHAPTER6EMPIRICALANALYSISANDPRACTICALAPPLICATION6.1NumericalEvaluationoftheLearningAlgorithmInthischapterwepresentresultsfromanextensivecomparativenumericalstudyconductedwithanobjectiveofestablishingtheabilityoftheRLalgorithmtoobtainNashequilibriumforn-playermatrixgames.Forthispurpose,sixteenmatrixgameexampleswithknownNashequilibriaweresolvedbyusingbothanopenlyavailablesoftwareGAMBITandtheRLalgorithm.Todemonstratethepracticalapplicabil-ityoftheRLalgorithm,wealsosolvedamatrixgamethatmodelsstrategicbiddinginarestructuredelectricpowermarket.6.1.1MatrixGameswithKnownEquilibriaMatrixgamesthatwerestudiedconsistedofuptofourplayersandsixtyfourdierentactionchoices.TenoutofthesesixteenexampleshavepurestrategyNashequilibria,whichweresolvedusingthevariantoftheRLalgorithmthatseeksapurestrategy.TheremainingsixgamesweresolvedusingthemixedstrategyversionoftheRLalgorithm.Table6.1summarizesthematrixgamesspecifyingthenumberofplayersandtheiravailableactionchoices.SomeoftheseproblemsareadoptedfromGAMBITlibraryofmatrixgames,forwhichthelenamesusedinGAMBITareusedasidentiers.TheNashequilibriumsolutionsobtainedbybothGAMBITandRLalgorithmare47

PAGE 59

Table6.1.SampleMatrixGameswithPureStrategyNashEquilibria Table6.2.PureStrategyNashEquilibriumResults 48

PAGE 60

summarizedinTable6.2.Thefollowingobservationscanbemadefromtheresults.Foralltengames,theRLalgorithmfoundaNashequilibriumwhichcoincidedwithaGAMBITsolution.ItmaybenotedthatGAMBITobtainedmultiplepurestrategyNEforsixoutofthetengames.ForeachofthesegamesexceptinGame#7,RLalgorithmchosetheequilibriumwiththehighestplayerrewards.Thoughaformalmathematicalproofwillberequiredtosupportthisobservation,webelievethat,sincetheRLalgorithmlearnsthevaluesfortheactionsandchoosesactionsbasedonthesevalues,thesolutiontendstoconvergetotheNEwiththehighestplayerrewards.Table6.2alsopresentstheconvergencetimeoftheRLalgorithmwhichwasrunfor10,000iterationsforallthegamesonacomputerwitha1.6GHzPentiumMprocessor.However,anaccurateassessmentoftheconvergencetimewillrequirefurtheroptimizationofthelearningparametersofthealgorithm,whichcouldbeproblemdependent.Forexample,manyofthegamesthatarepresentedinthetableconvergedmuchsoonerthan10,000iterations.Hence,theconvergencetimespresentedhereareintendedonlytoprovideageneralideaofthecomputationaleortsrequiredbythealgorithm.Table6.3presentsthecomparisonofmixedstrategiesobtainedbyGAMBITandtheRLalgorithmforsixmatrixgames.ThoughGAMBITfoundmultiplemixedNEformostoftheseproblems,forfairnessofcomparison,onlythoseNEwithmaximumplayerrewardsobtainedbyGAMBITarepresentedinthetable.Asevidentfromthetable,thoughthemixedstrategiesobtainedbytheRLalgorithmaredierentfromtheNEobtainedbyGAMBIT,playerrewardsfromtheRLalgorithminalmostallofthegamesarecomparable.ItcanalsobeseenfromthetablethatevenwhenthemixedstrategyversionoftheRLalgorithmisimplemented,ityieldsapurestrategyifoneexists,asinGames4and5.ItmaybenotedthatforGames4and5,GAMBITalsondsthepurestrategies.However,inthistablewepresentonlymixed49

PAGE 61

Table6.3.MixedStrategyEquilibriumResults strategyresultsobtainedbybothGAMBITandtheRLalgorithm.InGame6,wherethetwoplayershave64actionseach,themixedstrategiesforbothplayershavelargesupportsetsandthuscouldnotbepresentedinthetable.Therefore,wechosetopresentonlytheplayerrewardsasmeansforcomparison.Inthenextsubsection,we50

PAGE 62

presentamatrixgameexamplefromareallifemarketplacethatissettledthroughmultipartycompetitiononaperiodicbasis.6.1.2APowerMarketMatrixGameInrestructuredelectricpowermarkets,likeinPJMPennsylvania-Jersey-Mary-land,NewYork,NewEngland,andTexas,poweristradedinlongtermbilateralmarket,dayaheadmarket,andspotmarket.Thegeneratorsandretailerscompeteinthemarketbystrategicallybiddingforpriceandquantityofpowertradedinordertomaximizeprots.Themarketissettledbyanindependentsystemoperator,whomatchesthesupplyanddemandandsatisesthenetworkconstraintswhilemaxi-mizingsocialwelfaretotalbenetminustotalcost.Thissettlementyieldspriceandquantityallocationsatallthenetworknodes.Thegeneratorsstrategizetoraisetheirpricesabovethemarginalbasecosts,whiletheretailers'strategiesareaimedatmaintainingpricesclosetothemarginalcosts.Theabilityofthegeneratorstomaintainpricesabovethemarginalcostsforasustainedperiodoftimeisdenedasmarketpower.Amarketissaidtobecompetitivewhenthepricesareatornearthemarginalcosts,whichisoneoftheprimaryobjectivesofarestructuredelectricitymarketdesign.Adayaheadpowermarketcanbemodeledasarepeatedn-playermatrixgame,ofwhichtherewardmatricescanbeconstructedusingtheproducersurplusforgeneratorsandconsumersurplusforretailers.WeconsiderafourbustwogeneratorsandtworetailerspowernetworkasshowninFigure6.1,whichwasstudiedin[76].ThesupplyfunctionbidsofthegeneratorsatnodesAandBandthedemandfunctionsoftheretailersatnodesCandDareasfollows:pS1=a1+m1q1,pS2=a2+m2q2,pD1=100)]TJ/F15 11.955 Tf 12.149 0 Td[(0:52d1,pD2=100)]TJ/F15 11.955 Tf 12.149 0 Td[(0:65d2,whereq1andq2arethequantitiesinmegawatt-hour,MWhproducedbygeneratorsS1andS2respectively,andd1andd2arethequantitiesdemandedbytheretailersD151

PAGE 63

andD2respectively.Thesupplyfunctionhastwostrategicbidparametersinterceptain$/MWhandslopemthatthegeneratorsmanipulatetomaximizetheirprots.Demandsidebiddingbytheretailersisnotconsideredandhencethedemandfunctionparametersaremaintainedconstantattheirbasevalues.Asin[76],thereactancesareconsideredtobethesameonalllines. Figure6.1.4-BusPowerNetworkIn[76],theeectsofstrategicbiddingarestudiedbyimposingtransmissioncon-straintsonlinesACandBDoneatatimeresultinginnetworkcongestion.Nashequilibriaforbothslope-onlyandintercept-onlybiddingscenariosforeachofthetransmissionconstrainedcasesACandBDareseparatelyexamined.Berryetal.[76]usedaniterativealgorithmtoobtainNEoftheabovegame.ThealgorithminvolvessolvingtheISO'sproblemforaseriesofbidoptionsofagenerator,whileholdingthebidsoftheothergeneratorconstant.Thebidoptionthatproducesmaximumprotisthenxed,andthesameprocedureisrepeatedfortheothergenerator.Thisprocessisrepeateduntilneithergeneratorhasanalternativebidtofurtherimproveprot.Thematrixgameapproachdevelopedinthisresearchdiers52

PAGE 64

fromtheaboveapproachinthatallgeneratorsselectactionssimultaneouslywithoutanyknowledgeoftheothersactions.Inordertoapplythelearningalgorithm,asarststep,therewardmatricesforthegeneratorsareconstructed.Toaccomplishthis,thefeasiblerangeofthebidparametersaresuitablydiscretizedwhichdictatethesizeoftherewardmatrices,andtherewardsforeachcombinationofthegeneratorsbidsarecalculated.Itmaybenotedthatgeneratorrewardisafunctionofthenodalpricesandquantities,whichareobtainedbysolvingasocialwelfaremaximizationproblem.Detailsofthemathematicalformulationcanbefoundin[76].Thefeasiblerangesofslopeandinterceptparametersarediscretizedto250valuesgivingmatrixsizesof250250.Inparticular,theslopeparameterrangedfrom0.35to2.85forS1and0.45to2.95forS2,bothinstepsof0.01.TheinterceptbidparameterforbothgeneratorsS1andS2rangedfrom10$/MWhto260$/MWhwithasteplengthof1unit.ThesolutionofthesocialwelfareproblemandcalculationofthegeneratorrewardsforalltheabovebidcombinationsareaccomplishedusingGAMSsoftware.Theresultsfrom[76]andthosefromthelearningalgorithmarepresentedinTable6.4.Itcanbeseenfromthetablethatthelearningalgorithmobtainsbetterorcomparableprotsforbothgeneratorsinallcases.Wealsoextendthenumericalexperimentationbyallowinggeneratorstobidforbothslopeandintercepttogether,insteadofbiddingforoneparameteratatimeasin[76].Thebidparametersinthisexperimentarediscretizedasfollows.Theslopeisvariedintwentyvestepsof0.1forbothgeneratorsrangingfrom0.35to2.85forS1and0.45to2.95forS2.Theinterceptisvariedintwentyvestepsof3rangingfrom10$/MWhto85$/MWh.Hence,eachgeneratorhas2525=625actionchoicesandtheresultingrewardmatricesareofsize625625.TheRLalgorithmisrunfor500,000iterations,whichtook770secondsonacomputerwitha2GHz53

PAGE 65

PentiumIVprocessor.AsshowninTable6.4,intheAC-congestioncase,biddinginbothslopesandinterceptsleadtosimilarprotsasinthecasesofoneparameteratatimebidding.Whereas,inthecaseofBD-congestion,theprotsobtainedbytheplayersthroughjointbiddingismuchhigherthanbiddingoneparameteratatime.Table6.4.ResultsfromtheStudyof4-BusPowerNetwork 6.2SomeRemarksThoughtheinterneterahasprovidedthetechnologicalinfrastructurenecessarytoinvigoratemarketcompetition,lackofcommensurateadvancementsincomputa-tionalalgorithmstosolvemultiplayergameshasbeenalimitingfactorinexaminingthemarketbehavior.Meteoricriseincomputingpowerviateraandpetascalecom-putingmadepossiblebyecientharnessingofclustercomputinghascreatedanopportunitytobreakthroughperceivedcomputationalbarriersofstatespaceexplo-54

PAGE 66

sion.ThisresearchpresentsanewcomputationalapproachtondNashequilibriumofmultiplayermatrixgames.Theapproachisfoundedonthevaluefunctionlearningstrategythatisbeingsuccessfullyusedinsolvinglargescaledecisionmakingprob-lemsmodeledasMarkovandsemi-Markovdecisionprocesses.Inthewakeofrecentstudiesthatlinkalargeclassofstochasticgamestomatrixgames[34,33],oursolutionapproachstandstoimpactabroadrangeofdecisionmakingproblems.ThecomparativenumericalresultspresentedforalargenumberofmatrixgameshelptodemonstratethevalidityofourconjectureinChapter5,Section5.2onvaluefunctionguidedNEdetermination.Thoughonemightthinkthatgamesgen-erallyinvolvealargernumberofplayersthanwhatisconsideredintheexampleproblems,inreallife,applicationsofmatrixgamestendtohavealimitednumberofplayers.Thisoligopolisticstructureofmostcontemporarymarketsnaturallyoccursduetoextensivemarketsegmentation.Someexamplesofsucholigopolisticmarketsincluderetailsales,homeandautoinsurance,mortgagelending,serviceindustrieslikeairlines,hotels,andentertainments.InChapter5,wedevelopedasolutionalgorithmtosolvemulti-playermatrixgamesandinthecurrentchapterwebenchmarkedthesolutionsobtainedfromtheRLalgorithmwiththoseobtainedfromGAMBIT.Inthefollowingchapter,wepresentadetailedsolutionframeworkfortheoveralltwo-tiermodeltoobtainmultiyear,mul-tiplayerGEPstrategies.ThealgorithmutilizestheRLbasedsolutionalgorithm,de-velopedhere,tosolvethematrixgamesembeddedwithinthetwo-tierGEPmodel.55

PAGE 67

CHAPTER7SOLUTIONFRAMEWORKFORTWO-TIERGEPMODEL7.1SolutionAlgorithmfortheTwo-TierGEPModelThefollowingstepbystepalgorithmisusedtosolvethetwo-tiermatrixgamemodelforgenerationexpansionplanning.AschematicrepresentationofthealgorithmispresentedinFigure7.1.1.Atthestartofeveryyear,potentialinvestorsgeneratorsassessthefuturede-mandprojections,protsfrompreviousyears,networkconditions,andmarketdesignincentivestodevelopasetoffeasiblegenerationexpansioninvestmentalternativesBox1.2.Letai:i=1;;Ndenotethenumberofinvestmentalternativesavailabletogeneratori.Then,theinvestmentmatrixgameAisanN-dimensionalmatrixofsizea1a2aNBox2.3.ForeachelementofmatrixgameA,thereisacorrespondingsupplyfunctionSFmatrixgameofsizeQNi=1bi,wherebidenotesthenumberofsupplyfunctionbidsofgeneratoriBox3.4.ProtsforeachelementoftheSFgames~rkb1;b2;;bNareobtainedaftersolvingthecorrespondingDC-OPFBox4.SeeEquation4.6.5.OncetheprotsforeachelementoftheSFgamesareobtained,avalueapprox-imationbasedreinforcementlearningalgorithmChapter5isusedtondthe56

PAGE 68

equilibriumprotsiforthegeneratorsBox5.SeeEquation4.7fortheformulausedtocomputei.6.Subsequently,theseequilibriumprotsandrespectiveequilibriumbidsareuti-lizedtocomputetheriskconstrainedprotsRCP,iviaaconditionalvalue-at-riskmeasureBox6.SeeTable7.1forthestepsinvolvedincomputingtheCVaRandseeEquation4.4fortheformulatocomputei.7.TheseriskconstrainedprotvaluesconstitutethepayomatricesforinvestmentmatrixgameA.Finally,thereinforcementlearningalgorithm,developedinChapter5,isusedonmatrixgameAtoobtaintheequilibriumsolution.ThissolutionistheriskconstrainedgenerationexpansionstrategyfortheyearunderconsiderationBox7.8.ThisprocedureSteps1-7isrepeatedoneyearatatime,untilthegenerationexpansionstrategyfortheentireplanninghorizonisobtainedforeachgenerator.57

PAGE 69

Figure7.1.SchematicforTwo-TierGEPModelSolutionAlgorithm58

PAGE 70

Table7.1.StepsforCalculatingCVaR Step1 Denecontingencyscenariosj=1;;nwheren2Nisthetotalnumberofscenarios. Step2 Usetheequilibriumprice-quantitybidsfromeverySFgameandsolvetheDC-OPFproblemforallcontingencyscenarios. Step3 Obtainprotsijforeachcontingencyscenariojandgenera-tori. Step4 ^ijiscomputedasthedierencebetweentheequilibriumprof-itsoftheSFgameandtheprotsijfromcontingencyscenarios. Forexample,iftheequilibriumprotfromtheSFgameis$100andtheprotmadeduetoacontingencyscenariois$70,the^ijis-$30. Step5 Using^ijvaluesfromallthescenarios,computetheorderstatistics^i:n;;;;^in:n;8i. Step6 Forapre-denedvalueof2;1,calculateCVaRn,usingEquation4.3. Step7 Dependingupontheriskpreferenceofeachgeneratori,com-putetheriskconstrainedprotusingEquation4.4. Step8 Repeatsteps1-7forallQNi=1aiSFgames. 59

PAGE 71

CHAPTER8NUMERICALANALYSIS8.1NumericalExperimentationandAnalysisTodemonstratehowthetwo-tiermodelworks,wechosea5-busnetworkfromPowerworldsoftwarepackage[83].The5-busnetworkshowninFigure8.1,consistsoffourloadsandseventransmissionlinesandiscurrentlyservedbythreegenerators.Therearetwomainreasonsforchoosingthisparticularnetwork.First,webelievethatthenumberofbusesandgeneratorsareadequatetodemonstratetheapplicabilityofthetwo-tiermodel.Second,therelativelysmallsizeofthenetworkallowsfordetailednumericalexperimentationandanalysis.Wenextpresenttheimportantfeaturesofthesampleproblem.Table8.1showsdemandcurveparametersforafouryearplanninghorizon.Theinterceptparameterincolumn2isconsideredthesameforallfouryears,whiletheslopeparameterisreducedeachyearindicatingagrowthindemand.Consistentwithindustrystandardsandenergyliterature,wemakeappropriateassumptionsaboutmarginalsupplyfunctionsforcoal,naturalgas,nuclear,andpetroleumredplants,asshowninTable8.2.Tokeeptheproblemexpositionsimple,weassumethatGenerator1locatedatBus1andGenerator2locatedatBus4competeagainsteachother,whileGenerator3locatedatBus2actsastheprice-taker.Inotherwords,Generators1and2submitstrategicbidsaimedatmaximizingindividualprots,whileGenerator60

PAGE 72

Table8.1.4-yearDemandProjections Table8.2.SupplyFunctionParametersofGenerators 3simplyacceptsthepricesetbythemarket.Inordertomaintainthedimensionalityofthesamplenetwork,weassumethatgeneratorsdonotbidstrategicallyfornewlybuiltplants,i.e.,newgeneratingplants,ifandwhenaddedatBuses3and5,actasprice-takers.Whilebidding,generators1and2holdtheirinterceptsconstantattheirbasevaluesasshowninTable8.2andbidstrategicallyonlywithrespecttoslopes.Weallowgeneratorstobidinincrementsof0.1fromtheirbasevaluesupto10steps,whichmeansthatthesupplyfunctionmatrixgamehasasizeof1010.Itmaybenotedthatthereinforcementlearningbasedsolutionalgorithmiscapableofhandlingmuchlargergamessee[10].Thatis,wecouldallowgeneratorstobidstrategicallyinbothslopeandintercept.Limitingthesupplyfunctiongameto1010isdoneonlyfortheeaseofexposition.61

PAGE 73

Figure8.1.FiveBusElectricPowerNetworkTable8.3.LineFailureScenariosusedinComputingCVaR TodemonstratecomputationofCVaR,wesubjectedthesamplenetworktosigni-cantvariabilitythroughhundreddierentdemandandlinefailurescenarios.Columns2and4ofTable8.3showthelinesthatareassumedtofail.Thetendemandvari-abilityscenariosareshowninTable8.4.UsingtheprocedureshowninTable7.1,wecomputetheCVaRandsubsequentlyuseittoobtaintheriskconstrainedprot.62

PAGE 74

Table8.4.DemandVariationsusedinComputingCVaR 8.1.1ComputationalResultsTheobjectivesofthenumericalexperimentationarethree-fold:todemonstratetheabilityofthemodeltoobtainmulti-year,multi-player,generationexpansionplans,topresentastatisticalanalysisoftheimpactofdemandvariationsandtransmissionconstraintsonthenodalpricesinthenetworkbeforeandafterexpansion,andtodemonstratehowriskpreferencesofgeneratorsaectthechoiceofexpansionplans.Table8.5presentsresultsoftheGEPmodelforafouryearplanninghorizonforthedemandprojectionscenariodepictedinTable8.1.Generatorsareassumedtobehighlyriskprone=0:1duringthefouryearperiod.AspresentedintheTable8.5Year1,topsegment,thecurrentplantsinthenetworkare:a50MWnaturalgasplantatBus1ownedbyGenerator1,a50MWnaturalgasplantatBus4ownedbyGenerator2,anda100MWnuclearplantatBus2ownedbyGenerator3.Asshowninthetable,Generator1hasthefollowinginvestmentalternatives:donothingorpostponeexpansion,expandcapacityoftheexistingnaturalgasplantatBus1from50MWto100MW,orbuilda50MWcoalplantatBus3.Similarly,Generator63

PAGE 75

Table8.5.GEPDecisionsforDemandsfromTable8.1=0:1 64

PAGE 76

2hasthefollowinginvestmentalternatives:donothing,expandcapacityofexistingnaturalgasplantatBus4from50MWto150MW,orbuildapetroleumredplantatBus5withcapacityof50MW.Theinvestmentalternativeschosenherearesomewhatarbitrary.Inreality,fea-sibleexpansionplanscanbedevelopedbyconsideringoneyearofoperationatatime.OperatingprotsforpotentialexpansionactionscanthenbecalculatedfrompreviousyearsdispatchandLMPresults.Investmentcostscanbeannualizedbasedonarisk-adjusteddiscountrateandsubtractedfromtheoperatingprots.Thisgivesarankingoftechnologiesfromwhichaselectionofpossibleexpansionactionscouldbemade.Anexpansionactioncanconsistofasingleplantoraportfolioofnewplantsatdierentlocationsinthegrid.Auser-denedlimitcouldbeimposedonthenumberofpossibleactionsforeachgeneratortoreducethedimensionalityoftheproblem.Notethatwedonotconsidertransmissioninvestmentsaspotentialexpan-sionactionsinthemodel.However,investmentsinnewtransmissioncanbespeciedasanexogenousinputtothemodel.Futureretirementsofexistinggeneratingplantsandtransmissionlinescanalsobeincludedasauser-denedinput.However,wedonotperformaninvestmentanalysisinthiswork.Ourgoalhereistodemonstratehowmulti-year,multi-playergenerationexpansionplanscanbeobtainedbyasetofcompetinggeneratorswithgiveninvestmentalternatives.Inthenumericalexample,thethreeinvestmentalternativesforbothGenerators1and2giverisetoaninvestmentmatrixgamewithnine3elements,eachofwhichisapotentialexpansionalternative.Eachyear,alltheexpansionalternativesareanalyzedassupplyfunctionmatrixgamesandprotsarecalculatedusingOPFandCVaRmodels.Thereafter,theequilibriumexpansionplansareobtainedusingthereinforcementlearningalgorithm.Thisprocedureisrepeatedforeachyearoftheplanninghorizon.65

PAGE 77

Theequilibriumexpansionplansobtainedbythereinforcementlearningalgorithmforyear1are:Generator1buildsacoalplantatBus3andGenerator2buildsthepetroleumredplantatbus5.Thisexpansionplanisshowninagrayshadeinyear1oftheTable8.5,indicatingthatitisapurestrategyNashequilibriumsolution.Theequilibriumsolutionofthetwo-tiermodelfromyear1isassumedtobepartoftheexistingnetworkforthesubsequentyears.Thenetworkcongurationforyear2isshowninTable8.5.Itshowsboththecurrentplantsinthenetworkaswellastheequilibriumplanschoseninyear1.TheexpansionalternativesconsideredforGenerator1foryear2are:donothing,expandthenaturalgasplantatBus1to100MW,andexpandthenewlybuiltcoalplantfurtherupto100MW.Generator2considersthefollowinginvestmentalternatives:donothing,expandnaturalgasplantatBus4to100MW,orexpandthepetroleumredplantatBus5to100MW.Thetwo-tiermodelissolvedagainforyear2givinganequilibriumexpansionplanforbothgenerators.Theequilibriumexpansionplanforyear2is:Generator1expandsthecoalplantatBus3andGenerator2expandsthenaturalgasplantatBus4to150MW.ThisisshowninagrayshadeintheTable.Similarly,theequilibriumexpansionstrategiesaresuccessivelycomputedforyears3and4.Wepresentthenodalpricesp1throughp5andquantityallocationsq1throughq5obtainedfromthetwo-tiermodelforthefouryearplanninghorizoninTable8.6.ThepayomatricesfortheinvestmentgameforallthefouryearsofexpansionareshowninTable8.7,wherethetwoelementsineachcellrepresentthepayosforgenerators1and2respectively.Eachelementofthepayomatrixisthesolutionofasupplyfunctionmatrixgame.Thepayomatricesforthesupplyfunctionmatrixgamesarenotshownhereforthesakeofbrevity.Itmaybenotedfromthepay-omatricesthatthereinforcementlearningalgorithmndsthepurestrategyNash66

PAGE 78

Table8.6.PriceandQuantityAllocationsfortheFourYearPlanningHorizon equilibriumNEforeachyearoftheexpansion.NEisdenedasthatcombinationofstrategiesfromwhichnoplayerwillgainbyunilaterallydeviating.ThesepurestrategyNEsolutionsarehighlightedinagrayshadeineachofthefouryearsinthetable.8.1.1.1MixedStrategiesandMultipleEquilibriaItiswellknownthatmatrixgamesmaynotalwayshaveapurestrategyNEbutwillalwayshaveamixedstrategyNE.However,forproblemssuchasGEP,amixedstrategysolutionisimpracticalfromanapplicationstandpoint.Therefore,whenthereisnopurestrategyNEsolution,thegeneratorsshouldconsiderothergoodout-of-equilibrium"[84]purestrategiesgeneratedbyourRLalgorithm.Insomeothercases,amatrixgamemayhavemultiplepurestrategyNE.Inthesecases,asshowninChapter5[10],ourvaluebasedreinforcementlearningalgorithmidentiestheNEwiththebestvalue.8.1.1.2GeneratorProtsandConsumerSurplusesGeneratorscanexaminetheirprotsunderperfectcompetitionwheregeneratorsbidatmarginalcostsandunderimperfectcompetitionstrategicbidding,togauge67

PAGE 79

Table8.7.GeneratorPayoMatricesandGameSolutionsfortheFourYearPlanningHorizon=0:1 68

PAGE 80

Figure8.2.ProtsandConsumerSurplusesinStrategicBiddingandPerfectCom-petitiontheamountofadditionalprottheycanmake.TheFigure8.2presentsaplotofgen-eratorprotsprimaryY-axisandconsumersurplussecondaryY-axisagainsttheplanninghorizon.Itcanbeseenfromthegraphthatthetotalprotsmadebybothgeneratorsunderperfectcompetitionarelesserthanthoseunderstrategicbidding.Thedierenceinprotgrewlargerwithincreaseindemand.Thisisexpected,sincehigherdemandprovidesmoreopportunitiesforstrategicbiddingbygenerators.Thegraphalsoshowsconsumersurplusalongthe4-yearplanninghorizon.Byobservingthechangesinconsumersurplus,generatorscanassesshowmuchoftheoverallsur-plusestheyareabletotransfertothemselves.However,generatorsneedtobewaryofbiddingtoohighinthemarket,sincehigherbidsmayleadtohigherprots,resultinginerodingconsumersurpluses,whichmayinviteapotentialregulatoryintervention.69

PAGE 81

8.1.2NodalPriceSensitivityAnalysisTheobjectiveofthefollowingstatisticalanalysisistoassesstheeectthatlinecapacityandtheslopeofthedemandcurvehaveonthepriceateachbuspost-expansion.Linecapacitywasdeemedanimportantfactortoincludeinthisanalysisduetoitsdirectimpactontransmissioncongestionand,asaresult,ontheprice.Theslopeoftheconsumerdemandcurveisincludedintheanalysisbecauseitspriceelasticityisexpectedtohaveanimpactonthetypeofexpansionplanchosenbyagenerator.Understandingtheinuencethatlinecapacityanddemandexertoverpostex-pansionpricesisbenecialforallstakeholdersintheelectricitymarket.Itenablesgeneratorstoforecasthowexpansionplanswillimpactnodalprices.Likewise,theISOcanassesstheimpactoflinecapacityrestrictionsanddemandvariationsonthenodalpricesafterapotentialexpansiondecision.Finally,theconsumerscanalsoben-etbyexaminingwhattypeofdemandcurvevariationscanhelpthemtopossiblyattainlowerpost-expansionprices.Forthesakeofsimplicityandtoaidthevisualizationofpotentialeects,linecapacitywasvariedamongthreelevelsandonlyconsideredinlines1)]TJ/F15 11.955 Tf 11.432 0 Td[(3;3)]TJ/F15 11.955 Tf 11.432 0 Td[(4;4)]TJ/F15 11.955 Tf 11.432 0 Td[(5seeFigure8.1.Therationalebehindselectingtheselinesliesinthefactthatnodes1,3,4,and5areconsideredforpotentialexpansions.Thedemandslopeisvariedamong5valuesatnodes2,3,4,and5thesamedemandslopeisusedatallnodesforeachrun.Thisyieldsatwofactormixedfactorialexperimentwiththefactorsatthreeandvelevelsrespectively.Table8.8presentsthelevelsfordemandandlinecapacity.Afterobservingthepricesbeforeandafterexpansionforfteenrunslevelsoflinecapacity5levelsofslopeofthedemandcurvethedierenceinnodalpricesforeachbuswascapturedastheresponsevariable.Ananalysisofvariance70

PAGE 82

Table8.8.LineCapacityandConsumerDemandSlopeLevels Table8.9.FStatisticValuesin2-FactorDesign ANOVAforthefactorialdesignwascarriedoutwhoseresultsarepresentedinTable8.9.NotethatinTable8.9thosevaluesaccompaniedbywerefoundtobesignicantatthelevelof0.05,whereasthosewithweresignicantat0.1.BasedonresultsfromTable8.9,demanddoesnotappeartoplayasignicantroleinbus1andneitherdoeslinecapacityinbus2.Demandandlinecapacityappeartohaveasignicanteectonthepricedierentialobservedinbus3,whereastheinteractionbetweenthemwastheonlysignicantfactoratbus5.Inaccordancewiththeresultsfromthefactorialdesign,regressionmodelswerettedtomeasuretheeectofeachfactoronthepricedierentialP=nodalpriceafterexpansion-nodalpricebeforeexpansionateachbus.Table8.10,presentsthecoecientsobtainedforamodelwiththegeneralform,y=0+1xdem+2xlinecap+3xdemxlinecap:.1SincelinecapacitywastheonlysignicantfactoraectingPatbus1seeTable8.9,themodelatbus1wasttedconsideringonlylinecapacity.Thenegativesign71

PAGE 83

Table8.10.LinearRegressionModelCoecients inthelinecapacitycoecientimpliesthatthehigherthelinecapacitythehigherthereductioninpriceaftertheexpansionplansareimplemented.Thisresulttswithintheframeworkofbasicmicroeconomictheory.Reducedlinecapacitycausescongestion,generatinglessresourceavailabilityandthereby,increasedprices.Iflinecapacityisreducedto13.5MW54%offullcapacitythepricedierentialiszero.Smallervaluesoflinecapacitywillincreasethepricepost-expansionatthatbus.Onlytheslopeofthedemandwasconsideredintheregressionmodelforbus2.Theresultingregressioncoecientwaspositiveimplyingthatthesteeperthedemandcurvethehigherthereductionofpriceatthebus.Flatdemandcurvesi.e.valuesfortheslopelessthan0.26willgenerateincrementsinprice.Asteepdemandcurvecorrespondstoacomparativelymoreinelasticdemand,hencecapacityexpansionwillhavehighimpactonpricereduction.Themodelsobtainedfornodes3and5aremorecomplexbecausetheyinvolvebothfactorsandinthecaseofbus5aninteractiontermisalsopresent.Thesepredictivemodelsareusefulinexaminingnetworkbehavior.Forexample,theequationforbus3:y=)]TJ/F15 11.955 Tf 9.299 0 Td[(82:66)]TJ/F15 11.955 Tf 10.607 0 Td[(18:92xdem+1:49xlinecap,canbeusedtoshowthatforalinecapacityof25MW,anegativepricedierentialwilloccuronlyaslongastheabsolutevalueoftheslopeofthedemandcurveissmallerthan2.4.Thatis,forlesselasticconsumerdemandabsolutevalueofslopehigherthan2.4thepricedierentialwillbecomepositiveincreasingnodalprice.Eventhoughconductingabus-by-busanalysis,as72

PAGE 84

presentedabove,maybecomecumbersomeforverylargesizenetworks,itcanbeaccomplishedforthosebusesdeemedcritical"inthenetwork.8.1.3ImpactofRiskPreferenceonGEPWeexaminedhowGEPoutcomesvarywithrespecttogeneratorriskpreferencesiseeEquation4.4.Foragivenpowernetwork,thiscouldservetoestablishtheivaluesatwhichthegeneratorsbegintoswitchtheirinvestmentchoices.Figure8.3showsthedierentexpansionplanschosenfordierentvaluesoftheriskpreferencesfordemandcorrespondingtoyear1ofTable8.1.Notethat,highervaluesofriskpreferenceiindicateshigherlevelofriskaversion.Resultsshowthatthoughgener-ator1'sexpansionplandoesnotchange,theinvestmentdecisionsatanetworklevelvarywithincreasingriskaversion.Intheanalysisabove,thesameivaluewasassumedforbothgenerators.However,wecanalsouseourmodeltoobservehowdierentriskpreferencesofgeneratorscanhaveanimpactontheexpansionplans.GeneratorsorISOcanalsoexaminethechangesinGEPpatternwhentheriskpreferenceschangeovertheyears.Forexample,ageneratorwhomayhaverecentlyinvestedinaplant,mightbeaversetoriskinthesubsequentfewyears.Suchvariationsinriskpreferencescanbeconsideredbyourmodel.73

PAGE 85

Figure8.3.ProtsandExpansionPlansversusRiskPreferences74

PAGE 86

CHAPTER9CONCLUDINGREMARKSElectricitymarketrestructuringgaverisetoanewrealmofissuesthatneededtobeaddressed,thatwerenotseenintheresearchpresentedintheeraofregulatedmarkets.Agreatamountofresearchhasbeendevotedtoissueslikemarketde-sign,marketpowerassessment,nancialtransmissionrights,capacitymarkets,andancillarymarkets,however,theresearchingenerationexpansionplanninginrestruc-turedmarketshassignicantroomforexploration.Thisresearchpresentedanovelmethodologytoaidthisexploration.9.1AdvancesMadebythisResearchinGEPAsexplainedearlier,currently,overfteenstatesintheUShaverestructureden-ergymarkets,wheregeneratorsarerequiredtocompeteinanopenmarkettosupplypowerintotheelectricitygrid.TheEnergyInformationAdministrationEIAfore-caststhatseveralpowerplantsaretobeconstructedintheserestructuredmarkets,leadingtoinvestmentsofbillionsofdollarsinthenexttwodecadestosatisfytherisingdemandforelectricity.Thisissue,referredtoasgenerationcapacityexpansion,hasbeenverywellstudiedinregulatedmarkets.However,modelsdevelopedunderregu-latedsettingshadanoptimizationstructureconsideringonlyasingledecisionmaker,andarerenderedobsoleteunderthenewrestructuredenergymarketparadigm.Intherestructuredmarkets,thisissue,aspresentedinthisdissertation,needstobemodeledasagame-theoreticproblemsinceitrequiresthesimultaneousconsideration75

PAGE 87

ofseveralcompetingdecisionmakers.Oneoftheprimaryresearchcontributionsofthisdissertationisthedevelopmentofacomprehensivegame-theoreticmodelwhichconsidersseveralimportantrestructuredmarketfeatureslikecompetitivebehaviorofthegenerators,transmissionlineconstraints,voltage¤trelatedconstraints,multipleyearplanninghorizon,powerdemandstochasticity,andriskduetovolatili-tiesinprot.Suchacomprehensivegame-theoreticmodelwithalltheaforementionedfeaturesisasignicantadvancementintheareaofgenerationcapacityexpansioninrestructuredelectricitymarkets.Ourmodelexaminesthenon-cooperativecompetitionofgeneratorsattwotiers.Atonetierthegenerationinvestmentgameisexaminedandatanothertierthesupplyfunctiongameatthepowernetworkoperationallevelisexamined.Wepresentanovelsolutionalgorithmforthetwo-tiermodelthatshowshowthesetiersinteracttoobtainamulti-yeargenerationexpansionplan.Usingasampleelectricpowernetworktheapplicabilityofthemethodologyisdemonstrated.Therearesomeimportantfeaturesthatcanbeincludedinthemodeltoenhanceitsapplicability.FeaturessuchasreliabilityandcapacitymarketscanbeincorporatedintheDC-OPFmodel,usingthestrategypresentedin[23,85].Furthermore,theOPFmodelthatweadoptinthisresearchisalinearizedDC-OPFversionalaHobbsetal.[46,76],andwaschosenonlytosimplifythecomputationofsupplyfunctionequilibria.Theadvantagesofusingsuchlinearizedsupplyfunctionswerediscussedingreatdetailin[86].OnecouldalsoreplacetheDC-OPFwithAC-OPFmodelandstep-functionbidding,asinourpreviouswork[87].Themodelpresentedinthisresearchcanbebenecialforallpowermarketcon-stituents:generators,consumers,andISO.Thematrixgameapproachweadoptinthisresearchallowsgeneratorstoassesstheprotabilityofseveralinvestmentalterna-tivesbyincorporatingriskpreferencesandCVaR.Webelievethattheconsideration76

PAGE 88

ofriskpreferencesandCVaRmakestheinvestmentdecisionsgeneratedbyourmodelmorerobust.Consumerscanuseourmodeltostudyhowdierentexpansionplansadoptedbygeneratorswillaectnodalprices,whiletheISOcantestdierentmarketdesignsaimedatmaximizingsocialwelfare.9.2PracticalApplicationsofMatrixGamesThemultiplayerelectricpowermarketproblemwesolvedinChapter5aswellasthemultiplayerGEPproblemwesolvedinChapter8serveasexcellentexamplesofreallifeapplicationofmatrixgames.Outcomesofsuchgamesdeterminethenatureofhourlyanddailypowertransactionsaswellastheabilityofamarkettomeetdemandgrowthoverseveralyears.Hence,theabilitytoaccuratelyobtainNEformatrixgamesallowsforbetterassessmentofmarketperformanceandecientmarketdesign,whichtranslatetostablepowermarketoperationswithlimitedpricespikes.Solutionsofrelativelylargematrixgamesofsize625625resultingfromthesamplepowernetworkproblemindicatethealgorithmspotentialtotacklereallifepowernetworks,whichcanbemagnitudeslargerinsize.Thoughthenumericalresultsarepromisingandencouragefurtherexplorationofouralgorithmsperformance,atheoreticalproofofconvergenceandoptimalityisrequired.Webelievethatsuchaproofcanbeconstructedfollowingthelogicusedin[33],andwearecurrentlyworkingondevelopingsuchaproof.Workinprogressandfutureresearchdirectionsincludethetheoreticalconvergenceanalysisofthereinforcementlearningalgorithm.Aninterestinglineofresearchwillbetheapplicationofthemultiplayermatrixgameapproachinothereldsthathaveoligopolisticcompetition.Someexamplesofsucholigopolisticmarketsincluderetailsales,homeandautoinsurance,mortgagelending,serviceindustrieslikeairlines,hotels,andentertainments.77

PAGE 89

CHAPTER10FUTURERESEARCHDIRECTIONSTherestructuringoftheelectricityindustryhasgivenrisetoanewrealmofexcit-ingaswellasextremelychallengingmodelingissuesattheintersectionofoperationsresearch,public-policy,economics,andriskmanagement.Suchmodelingissues,duetotheirinherentcomplexitieswithseveraldynamicelementscanbebestmodeledandsolvedusingcomputationaloptimizationapproaches.Computationalmodelingaordsthedistinctadvantageofbeingabletohandleextremelycomplexsystemswithdata-richenvironments.EnergymarketsarenotedbyNationalScienceFoundationandinliteratureascomplexsystemsthatneedtobemodeledusingcomputationalapproachestobeabletocomprehensivelycaptureallthestochasticdynamics.More-over,computationalmodelshelpinfurtheringourfundamentalunderstandingofthecomplexinteractionsofmultilevel,multi-scalesystemssuchasenergymarkets,whichmaynotbepossibleviatraditionalmodelingapproachesTheresearchpresentedinthisdissertationwillhelpinjointlyaddressingtwoemergingareasofcriticalnationalimportance:generationcapacityexpansioninre-structuredderegulatedenergymarketstomeetthegrowingenergydemandsdis-cussedsofar,andenvironmentalemissioncontrolviacarboncap-and-tradeCTprogramsandrenewableportfoliostandardsRPS,aimedatreducingthenegativeimpactofelectricpowergenerationonclimatechange.Theobjectiveofthischapteristobrieydiscusshowtheseareasarecloselyintertwinedandexplaintheneedfor78

PAGE 90

thedevelopmentofacomprehensivestochasticoptimizationmodeltojointlyaddresstheseissues.10.1Cap-and-TradeProgramsandRenewablePortfolioStandardsElectricitygeneratorsintheUSarethesinglelargestindustrialcontributorsofCO2andareoneoftheleadingcausesoftheclimatechangecrisis.Duetotheneedforaerceurgencyinreversingdetrimentalhumaneectsontheearth'sclimate,countriesaroundtheworldaswellassomeregionsintheUShaveenactedcarboncap-and-tradeCTprogramsforemissionsreductions.Cap-and-tradesystemshavehistoricallybeenusedasaneectivemarketmecha-nismtolimittheemissionofpollutantslikeNOXandSO2[88].SincetheinceptionofsuchaprogramintheU.S.in1995,therehavebeensignicantemissionsreduc-tionsandfarrangingenvironmentalaswellashumanhealthbenets,atalowerthanexpectedcompliancecost.Economistsandpolicyanalystsbelievethatimplementa-tionofasimilarcap-and-tradesystemforCO2shouldbeacentralelementofanyemissionscontrolpolicy[89].Acap-and-tradesystemwouldestablishCO2emissionlimitseitheratanupstreamlevelforproducersoffuels,oratadownstreamlevelforindustrialconsumersoffuelsincludingelectricitygenerators.RegulatedentitieswillbuyallowancesthatwillpermitthemtoreleaseacertainamountofCO2withinaspeciedperiodoftime.Iftheemissionsexceedallowances,entitiesneedtopurchasemoreallowancesorpayapenaltyintermsofincreasedpriceofallowancepurchaseforthenextperiod.Ontheotherhand,ifanentitydoesnotuseallofitsallowances,theycanbebankedforfutureorsoldinanopensecondarymarket[89,90].Bygraduallyloweringthecapontotalemissions,regulatedentitieswillbeforcedtoin-vestincleanersourcesofenergyandgreenertechnologies.Dierentvariationsofthecap-and-tradesystemhavebeenoperationalinEurope[91].79

PAGE 91

Recently,therstCO2CTprogramofUS,theRegionalGreenhouseGasInitia-tiveRGGI,becamefullyfunctioningin10NortheasternandMid-AtlanticStates.Also,theWesternClimateInitiativeandtheMidwesternGreenhouseGasAccord,comprisingoveradozenstates,aresettocommenceearlynextdecade.Withtheimplementationofsuchinitiatives,perWorldResourcesInstituteWRI,almost50%ofthepopulationofUSwillresideinstateswithCTprograms,outofwhichseveralmarketsarerestructured.RenewableportfoliostandardsRPS,perEnergyEciencyandRenewableEn-ergydivisionofDepartmentofEnergy,isastatepolicythatrequireselectricityproviderstoobtainaminimumpercentageoftheirpowerfromrenewableenergyresourcesbyacertaindate[92].Withtheenforcementofrenewableportfoliostan-dardsRPS,electricutilitiesinover24statesinthecomingdecadearerequiredtoproduceasignicantpercentageofelectricityusingrenewableenergysourceslikebio-fuels,solar,wind,andgeothermal.Sincecarbon-basedfuelsarethecurrentprimarysourcesforelectricitygeneration,theimplementationofCTandRPSisexpectedtotriggerafundamentaltransformationinthetechnologiesusedtoproduceelectricityinthecomingdecades.10.2JointModelforGEPandEmissionsControlSinceelectricitygenerationandclimatechangecrisisareinterrelated,thegener-ationcapacityexpansionplanningproblemandenvironmentalemissioncontrolarecloselyconnected.Toaddressthiscriticalsocietalchallenge,similartothemodelpre-sentedinthisdissertation,acomprehensivestochasticoptimizationgame-theoreticmodelneedstobedeveloped,whichwillcaptureatmultipleinterconnectedtiers:thecompetitionamonggeneratorsforcapacityinvestments,thecompetitionforal-lowancesintheCTmarkets,andtheoptimizationofelectricpowerowwhilemeeting80

PAGE 92

RPS.Solvingsuchcomplexmodelsisalmostimpossiblewithtraditionaloptimiza-tionapproaches.Therefore,toaddressthestochasticdynamicsinvolvedinsolvingthisproblem,asimulation-basedoptimizationcomputationalsolutionmethodologysimilartotheonedevelopedinthisresearchmustbeformulated.10.3FurtherPolicyAnalysisandPlanningApplicationsSeveralissuespresentedbelowcanbeaddressedbasedonthemodelspresentedinthiswork.1.Examiningtheeectofallowancepricesonelectricitymarketprices,2.CO2levelsoveralongtermplanninghorizon,3.eectsofdierentallowanceallocationmethods:auctioning,grandfathering,orhybridmodels,4.examiningeectsofdierentriskattitudesofinvestorsonexpansions,5.examinationofportfolioofgenerationplantsoveralongtermhorizon,6.eectofRPSongenerationexpansiononastatebystatebasis,and7.eectoflarge-scaleintroductionofmicrogridsasapotentialexpansionalterna-tive.81

PAGE 93

REFERENCES[1]S.Stoft.PowerSystemEconomics.IEEEPress,NewJersey,2002.[2]A.J.WoodandB.F.Wollenberg.PowerGenerationOperationandControl.JohnWileySons,Inc,NewYork,1994.[3]K.Bhattacharya,M.H.J.Bollen,andJ.E.Daalder.Operationofrestructuredpowersystems.Kluweracademicpublisher,Boston,2001.[4]M.Ventosa,A.Baillo,A.Ramos,andM.Rivier.Electricitymarketmodelingtrends.EnergyPolicy,33:897{913,2005.[5]C.J.Day,B.F.Hobbs,andJ.S.Pang.Oligopolisticcompetitioninpowernetworks:Aconjecturedsupplyfunctionapproach.IEEETransactionsonPowerSystems,173:597{607,2002.[6]J.BoucherandY.Smeers.Alternativemodelsofrestructuredelectricitysystems,part1:Nomarketpower.OperationsResearch,496:821{838,2001.[7]V.NanduriandT.K.Das.Asurveyofcriticalresearchareasintheenergyseg-mentofrestructuredelectricitymarkets.ElectricalPowerandEnergySystems,31:181{191,2009.[8]NationalEnergyPolicy.Reportofthenationalenergypolicydevelopmentgroup.Technicalreport,2001.[9]AnnualEnergyOutlook.Annualenergyoutlook,withprojectionsto2030.Tech-nicalreport,EnergyInformationAdministration,2007.[10]V.NanduriandT.K.Das.AreinforcementlearningapproachforobtainingtheNashequilibriumofmultiplayermatrixgames.IIETransactions,41:158{167,2009.[11]T.K.Das,A.Gosavi,S.Mahadevan,andN.Marchalleck.Solvingsemi-Markovdecisionproblemsusingaveragerewardreinforcementlearning.ManagementScience,454:560{574,1999.82

PAGE 94

[12]A.Gosavi,N.Bandla,andT.K.Das.Areinforcementlearningapproachtoairlineseatallocationformultiplefareclasseswithoverbooking.IIEtransactions,Specialissueonadvancesonlarge-scaleoptimizationforlogistics,productionandmanufacturingsystems,349:729{742,2002.[13]A.Gosavi.Aconvergentreinforcementlearningalgorithmforsolvingsemi-Markovdecisionproblemsunderaveragecostforaninnitetimehorizon.Euro-peanJournalofOperationalResearch,155:654{674,2003.[14]P.MasseandR.Gilbrat.Applicationoflinearprogrammingtoinvestmentsinelectricpowerindustry.ManagementSciences,3:149{166,1957.[15]J.A.Bloom.Long-rangegenerationplanningusingdecompositionandproba-bilisticsimulation.IEEETransactionsonPowerSystems,101:797{802,1982.[16]B.Mo,J.Hegge,andI.Wangensteen.Stochasticgenerationexpansionplanningbymeansofstochasticdynamicprogramming.IEEETransactionsonPowerSystems,6:662{668,May1991.[17]A.B.Borison,P.A.Morris,andS.Oren.Astate-of-the-worlddecompositionapproachtodynamicsanduncertainityinelectricutilitygenerationexpansionplanning.OperationsResearch,325:1052{1068,1984.[18]J.J.C.Meza,M.B.Yilidirim,andA.S.M.Masud.Amodelforthemulti-periodmultiobjectivepowergenerationexpansionproblem.IEEETransactionsonPowerSystems,222:871{878,May2007.[19]B.F.Hobbs.Optimizationmethodsforelectricutilityresourceplanning.Euro-peanJournalofOperationalResearch,83:1{20,1995.[20]J.ZhuandM-.Y.Chow.Areviewofemergingtechniquesongenerationexpan-sionplanning.IEEETransactionsonPowerSystems,12:1722{1728,Novem-ber1997.[21]S.T.JenkinsandD.S.Joy.WienautomaticsystemplanningpackageWASP-anelectricutilityoptialgenerationexpansionplanningcomputercode.Technicalreport,OakRidgeNationalLaboratory,ORNL4945,1974.[22]ElectricPowerResearchInstitute.ElectricgenerationexpansionanalysissystemEGEAS.Technicalreport,EPRI,PaloAlto,CA,1982.[23]A.S.Chuang,F.Wu,andP.Varaiya.Agame-theoreticmodelforgenerationexpansionplanning:Problemformulationandnumericalcomparisons.IEEETransactionsonPowerSystems,164:885{891,2001.83

PAGE 95

[24]F.MurphyandY.Smeers.Generationcapacityexpansioninimperfectlycom-petitiverestructuredelectricitymarkets.OperationsResearch,53:646{661,July-August2005.[25]P.Kaymaz,J.Valenzula,andC.SPark.Transmissioncongestionandcompe-titiononpowergenerationexpansion.IEEETransactionsonPowerSystems,22:156{163,2007.[26]P.JirutitijaroenandC.Singh.Reliabilityandcosttradeoinmultiareapowersystemgenerationexpansionusingdynamicprogrammingandglobaldecompo-sition.IEEETransactionsonPowerSystems,213:1432{1441,August2006.[27]J-.H.Kim,J-.B.Park,J-.K.Park,andS-.K.Joo.Amarket-basedanalysisonthegenerationexpansionplanningstrategies.ISAPConferenceProceedings,pages458{463,2005.[28]S.K.K.Ng,C.W.Lee,andJ.Zhong.Agametheoreticapproachtostudystrategicinteractionbetweentransmissionandgenerationexpansionplanning.IEEEConferenceProceedings,2006.[29]A.EhrenmannandY.Smeers.Capacityexpansioninnonregulatedelectricitymarkets.Proceedingsofthe39thHawaiiInternationalConferenceinSystemSciences,2006.[30]A.Botterud,M.Ilic,andI.Wangensteen.Optimalinvestmentsinpowergener-ationundercentralizedanddecentralizeddecisionmaking.IEEETransactionsonPowerSystems,201:254{263,Feb2005.[31]B.F.Hobbs.Linearcomplementaritymodelsofnash-cournotcompetitioninbilateralandPOOLCOpowermarkets.IEEETransactionsonPowerSystems,16:194{202,2001.[32]J.Nash.Non-cooperativegames.TheAnnalsofmathematics.,54:286{295,1951.[33]J.Li,T.K.Das,andK.M.Ramachandran.LearningNashequilibriumpoliciesinaveragerewardirreduciblestochasticgames.Workingpaper,DepartmentofIMSE,USF,2009.[34]J.HuandM.P.Wellman.Multiagentreinforcementlearning:Theoreticalframe-workandanalgorithm.InProc.15thInternationalConf.onMachineLearning,Madison,WI,1998.[35]V.S.Borkar.ReinforcementlearninginMarkovianevolutionarygames.AdvancesinComplexSystems,5:55{72,2002.84

PAGE 96

[36]J.FilarandK.Vrieze.CompetitiveMarkovDecisionProcesses.SpringerVerlag,NewYork,1997.[37]R.McKelveyandA.McLennan.Computationofequilibriainnitegames.InJ.RustH.Amman,D.Kendrick,editor,HandbookofComputationalEconomics,pages87{142.Elsevier,1996.[38]C.E.LemkeandJ.T.HowsonJr.Equilibriumpointsofbimatrixgames.SIAMJ.ofApp.Math.,12:413{423,1964.[39]J.Rosenmuller.Onthegeneralizationofthelemke-howsonalgorithmtononco-operativen-persongames.SIAMJ.ofApp.Math.,211:73{79,1971.[40]R.Wilson.Computingequilibriaofn-persongames.SIAMAppliedMath,21:80{87,1971.[41]L.Mathiesen.Computationalexperienceinsolvingequilibriummodelsbyasequenceoflinearcomplementarityproblems.OperationsResearch,336:1225{1250,1985.[42]P.T.HarkerandJ.S.Pang.Finitedimensionalvariationalinequalityandnonlin-earcomplementarityproblems:Asurveyoftheory,algorithmsandapplications.MathematicalProgramming,48:161{220,1990.[43]R.D.McKelvey,A.M.McLennan,andT.L.Turocy.GAMBIT:Softwaretoolsforgametheory,version0.2007.01.30.2007.[44]K.H.LeeandR.Baldick.Solvingthree-playergamesbythematrixapproachwithanapplicationtoanelectricpowermarket.IEEETransactionsonPowerSystems,184:166{172,2003.[45]B.F.Hobbs.LinearcomplementaritymodelsofNash-CournotcompetitioninbilateralandPOOLCOpowermarkets.IEEETransactionsOnPowerSystems,16:194{202,May2001.[46]B.F.Hobbs,B.M.Carolyn,andJ.S.Pang.Strategicgamminganalysisforelec-tricpowersystems:AnMPECapproach.IEEETran.PowerSyst.,15:638{645,2000.[47]J.Yao,S.Oren,andI.Adler.ComputingCournotequilibriaintwosettlementelectricitymarketswithtransmissionconstraints.Proceedingsof37thHawaiiinternationalconferenceonsystemsciences,pages1{9,2003.[48]T.C.Price.Usingco-evolutionaryprogrammingtosimulatestrategicbehaviorinmarkets.J.Evol.Econ.,7:219{254,1997.85

PAGE 97

[49]L.B.Cunningham,R.Baldick,andM.L.Baughman.Anempiricalstudyofappliedgametheory:Transmissionconstrainedcournotbehavior.IEEETrans-actionsonPowerSystems,171:166{172,2002.[50]R.W.Ferrero,M.Shahidehpour,andV.C.Ramesh.Transactionanalysisinderegulatedpowersystemsusinggametheory.IEEETransactionsonPowerSystems,123:1340{1347,1999.[51]S.Stoft.Usinggametheorytostudymarketpowerinsimplenetworks.IEEETutorialonGameTheoryinElectricPowerMarkets,pages33{40,1999.[52]Z.Luo,J.S.Pang,andD.Ralph.MathematicalProgramswithEquilibriumConstraints.CambridgeUniversityPress,Boston,1996.[53]C.L.Su.Equilibriumproblemswithequilibriumconstraints:Stationarities,algorithms,andapplications.DissertationfromtheDepartmentofManagementSciences,StanfordUniversity,2005.[54]D.RalphandY.Smeers.Epecsasmodelsforelectricitymarkets.PowerSystemsConferenceandExposition,Atlanta,GA,2006.[55]J.B.Cardell,C.C.Hitt,andW.W.Hogan.Marketpowerandstrategicin-teractioninelectricitynetworks.ResourceandEnergyEconomics,19:109{137,1997.[56]V.GountisandA.G.Bakirtzis.Biddingstrategiesforelectricityproducersinacompetitiveelectricitymarketplace.IEEETransactionsonPowerSystems,19:356{365,2004.[57]C.W.RichterJr.andG.Sheble.Geneticalgorithmevolutionofbiddingstrate-giesforthecompetitivemarketplace.IEEETransactionsonPowerSystems,13:256{261,1998.[58]C.W.RichterJr.,G.Sheble,andD.Ashlock.Comprehensivebiddingstrate-gieswithgeneticprogramming/niteautomata.IEEETransactionsonPowerSystems,144:1207{1212,1999.[59]P.Attaviriyanupap,H.Kita,E.Tanaka,andJ.Hasegawa.Newbiddingstrategyformulationfordayaheadenergyandreservemarketsbasedonevolutionaryprogramming.ElectricalPowerandEnergySystems,27:157{167,2005.[60]F.WenandA.K.David.Optimalbiddingstrategiesandmodelingofimper-fectinformationamongcompetitivegenerators.IEEETransactionsonPowerSystems,161:15{21,2001.86

PAGE 98

[61]R.RajaramanandF.Alvarado.Optimalbiddingstrategiesinelectricitymarketsunderuncertainenergyandreserveprices.Technicalreport,PowerSystemsEngineeringResearchCenter,Paper03-05,2003.[62]C.Li,A.J.Svoboda,X.Guan,andH.Singh.Revenueadequatebiddingstrate-giesincompetitiveelectricitymarkets.IEEETransactionsonPowerSystems,14:492{497,1999.[63]A.Seeger.RecentAdvancesinoptimization.LectureNotesinEconomicsandMathematicalSystems.Springer,2004.[64]K.G.Murthy.LinearComplementarity,Linear,andNonlinearProgramming,chapterTheLCPanditsGeometry,pages1{61.InternetEdition,1999.[65]K.H.LeeandR.Baldick.Tuningofdiscretizationinbimatrixgameapproachtopowersystemmarketanalysis.IEEETransactionsonPowerSystems,18:830{836,MAY2003.[66]J.Yao.Cournotequilibriumintwosettlementelectricitymarkets:formulationandcomputation.PhDthesis,UniversityofCalifornia,Berkeley,2006.[67]A.Gosavi.Aconvergentreinforcementlearningalgorithmforsolvingsemi-Markovdecisionproblemsunderaveragecostforaninnitetimehorizon.Inpress,EuropeanJournalofOperationalResearch,2003.[68]M.L.Puterman.MarkovDecisionProcesses.JohnWileyandSons,NewYorkChichesterBrisbaneTorontoSingapore,1994.[69]H.RobbinsandS.Monro.Astochasticapproximationmethod.Ann.Math.Statist.,22:400{407,1951.[70]A.Gosavi.Simulation-BasedOptimization:ParametricOptimizationTech-niquesandReinforcementLearning.OperationsResearch/ComputerScienceIn-terfacesSeries.Springer,2003.[71]R.S.SuttonandA.G.Barto.ReinforcementLearning:Anintroduction.MITPress,Cambridge,MA,1998.[72]T.R.RockafellarandS.Uryasev.Optimizationofconditionalvalueatrisk.JournalofRisk,2:21{41,2000.[73]T.R.RockafellarandS.Uryasev.Conditionalvalue-at-riskforgenerallossdistributions.JournalofBankingandFinance,267:21{41,2002.[74]J.SuandF.Wu.Evaluationofgenerationexpansioninvestmentundercom-petitivemarketenvironment.IEEEPowerandEnergySocietyGMProceedings,pages2136{2140,2005.87

PAGE 99

[75]C.AcerbiandD.Tasche.Expectedshortfall:Anaturalcoherentalternativetovalue-at-risk.EconomicNotes,312:379{388,2002.[76]C.A.Berry,B.F.Hobbs,W.A.Meroney,R.P.ONeill,andW.R.StewartJr.Understandinghowmarketpowercanariseinnetworkcompetition:Agametheoreticapproach.UtilitiesPolicy,8:139{158,1999.[77]L.S.Shapley.Stochasticgame,1953.InH.WKuhn,editor,ClassicsinGameTheory.PrincetonUniversityPress,1997.[78]M.L.Littman.Markovgamesasaframeworkformulti-agentreinforcementlearning.InInProceedingsoftheEleventhInternationalConferenceonMachineLearning,pages151{163,SanFrancisco,CA,1994.[79]J.HuandM.P.Wellman.NashQ-learningforgeneral-sumstochasticgames.JournalofMachineLearningResearch,4:1039{1069,2003.[80]L.P.Kaelbling,M.L.Littman,andA.W.Moore.Reinforcementlearning:Asurvey.JournalofArticialIntelligenceResearch,4:237{285,1996.[81]M.BowlingandM.M.Veloso.Ananalysisofstochasticgametheoryformul-tiagentreinforcementlearning.TechnicalReportCMU-CS-00-165,ComputerScienceDepartment,CarnegieMellonUniversity,2000.[82]C.Darken,J.Chang,andJ.Moody.Learningrateschedulesforfasterstochas-ticgradientsearch.InD.A.WhiteandD.A.Sofge,editors,NeuralNetworksforSignalProcessing2-Proceedingsofthe1992IEEEWorkshop.IEEEPress,Piscataway,NJ,1992.[83]PowerWorldSimulator.Version8.0Glover/SarmaBuild11/02/01,Power-WorldCorporation,2004SouthWrightStreet,Urbana,IL-61801,http://www.powerworld.com.[84]W.B.Arthur.Out-of-equilibriumeconomicsandagent-basedmodeling.InK.JuddandL.Tesfatsion,editors,HandbookofComputationalEconomics,vol-ume2.Elsevier/North-Holland,2005.[85]G.L.DoormanandA.Botterud.Analysisofgenerationinvestmentunderdif-ferentmarketdesigns.IEEETransactionsonPowerSystems,23:859{867,August2008.[86]R.Baldick,R.Grant,andE.Kahn.Theoryandapplicationoflinearsupplyfunctionequilibriuminelectricitymarkets.JournalofRegulatoryEconomics,25:143{167,2004.88

PAGE 100

[87]V.NanduriandT.K.Das.Areinforcementlearningmodeltoassessmarketpowerunderauction-basedenergypricing.IEEETransactionsonPowerSys-tems,221:85{95,2007.[88]A.D.Ellerman.Arecap-and-tradeprogramsmoreenviromentallyeectivethanconventionalregulation.InJ.FreemanandC.D.Kolstad,editors,Movingtomarketsinenviromentalregulation:LessonsfromTwentyYearsofExperience,chapter3,pages48{62.OxfordUniversityPressUS,2006.[89]R.N.Stavins.AU.S.cap-and-tradesystemtoaddressglobalclimatechange.TheHamiltonProject,DiscussionPaper2007-13.Washington,D.C.:TheBrookingsInstitution,,2007.[90]MarketAdvisoryCommittee.Recommendationsfordesigningagreenhousegascap-and-tradesystemforcalifornia.RecommendationsoftheMarketAdvisoryCommitteetotheCaliforniaAirResourcesBoard,http://www.climatechange.ca.gov/market advisory committee/index.html,2007.[91]EuropeanUnionEmissionTradingSchemeEUETS.http://ec.europa.eu/environment/climat/emission,2009.[92]RenewablePortfolioStandards.http://apps1.eere.energy.gov/states/maps/renewable portfolio states.cfm,2009.89

PAGE 101

ABOUTTHEAUTHORVishnutejaVishnuNanduri,receivedhisPh.D.inIndustrialEngineeringin2009andM.S.inIndustrialEngineeringin2005,bothfromUniversityofSouthFlorida.HereceivedaB.E.inIndustrialEngineering&ManagementfromVisweswariahTech-nologicalUniversity,India,in2002.Currently,heisanAssistantProfessorintheDe-partmentofIndustrial&ManufacturingEngineeringattheUniversityofWisconsin-Milwaukee.HisMaster'sThesiswontheFirstplaceintheInstituteofIndustrialEngineer'sIIEGraduateResearchAwardnationalcompetition.HewontheBestpaperawardintheEnergysponsoredsessionsofInstituteforOperationsResearchandManage-mentSciencesINFORMSatWashingtonD.C.Between2005-2009heservedastheProjectManagerfortheNSFfundedGK-12programatUSFStudents,Teachers,andResourcesintheSciences,STARS.Hismethodologicalareasofinterestarereinforcementlearning,game-theoreticmodeling,stochasticoptimization,andappliedprobability.Majorareasofapplicationofhisresearchareenergyandtheenvironment.


xml version 1.0 encoding UTF-8 standalone no
record xmlns http:www.loc.govMARC21slim xmlns:xsi http:www.w3.org2001XMLSchema-instance xsi:schemaLocation http:www.loc.govstandardsmarcxmlschemaMARC21slim.xsd
leader nam 2200397Ka 4500
controlfield tag 001 002063947
005 20100318123048.0
007 cr mnu|||uuuuu
008 100318s2009 flu s 000 0 eng d
datafield ind1 8 ind2 024
subfield code a E14-SFE0003031
035
(OCoLC)558721573
040
FHM
c FHM
049
FHMM
090
T56 (Online)
1 100
Nanduri, Vishnuteja.
0 245
Generation capacity expansion in restructured energy markets
h [electronic resource] /
by Vishnuteja Nanduri.
260
[Tampa, Fla] :
b University of South Florida,
2009.
500
Title from PDF of title page.
Document formatted into pages; contains 89 pages.
Includes vita.
502
Dissertation (Ph.D.)--University of South Florida, 2009.
504
Includes bibliographical references.
516
Text (Electronic dissertation) in PDF format.
520
ABSTRACT: With a significant number of states in the U.S. and countries around the world trading electricity in restructured markets, a sizeable proportion of capacity expansion in the future will have to take place in market-based environments. However, since a majority of the literature on capacity expansion is focused on regulated market structures, there is a critical need for comprehensive capacity expansion models targeting restructured markets. In this research, we develop a two-level game-theoretic model, and a novel solution algorithm that incorporates risk due to volatilities in profit (via CVaR), to obtain multi-period, multi-player capacity expansion plans. To solve the matrix games that arise in the generation expansion planning (GEP) model, we first develop a novel value function approximation based reinforcement learning (RL) algorithm.Currently there exist only mathematical programming based solution approaches for two player games and the N-player extensions in literature still have several unresolved computational issues. Therefore, there is a critical void in literature for finding solutions of N-player matrix games. The RL-based approach we develop in this research presents itself as a viable computational alternative. The solution approach for matrix games will also serve a much broader purpose of being able to solve a larger class of problems known as stochastic games. This RL-based algorithm is used in our two-tier game-theoretic approach for obtaining generation expansion strategies. Our unique contributions to the GEP literature include the explicit consideration of risk due to volatilities in profit and individual risk preference of generators. We also consider transmission constraints, multi-year planning horizon, and multiple generation technologies.The applicability of the twotier model is demonstrated using a sample power network from PowerWorld software. A detailed analysis of the model is performed, which examines the results with respect to the nature of Nash equilibrium solutions obtained, nodal prices, factors affecting nodal prices, potential for market power, and variations in risk preferences of investors. Future research directions include the incorporation of comprehensive cap-and-trade and renewable portfolio standards components in the GEP model.
538
Mode of access: World Wide Web.
System requirements: World Wide Web browser and PDF reader.
590
Advisor: Tapas K. Das, Ph.D.
653
Deregulated electricity markets
Generation expansion planning
Matrix games
Reinforcement learning
Conditional value-at-risk
690
Dissertations, Academic
z USF
x Industrial and Management Systems Engineering
Doctoral.
773
t USF Electronic Theses and Dissertations.
4 856
u http://digital.lib.usf.edu/?e14.3031