Documente Academic
Documente Profesional
Documente Cultură
Slidesfrom
R.S.SuttonandA.G.Barto
ReinforcementLearning:AnIntroduction
http://www.cs.ualberta.ca/~sutton/book/thebook.html
http://rlai.cs.ualberta.ca/RLAI/RLAIcourse/RLAIcourse.html
TheAgentEnvironmentInterface
Agentandenvironmentinteractatdiscretetimesteps : t 0,1, 2, K
Agentobservesstateatstep t : st S
producesactionatstep t : at A(st )
getsresultingreward : rt 1
andresultingnextstate : st 1
...
st
at
rt+1
st+1
at+1
rt+2
st+2
at+2
rt+3 s
t+3
...
at+3
TheAgentLearnsaPolicy
Policyatstept, t :
amappingfromstatestoactionprobabilities
t (s, a) probabilitythat at awhenst s
Reinforcementlearningmethodsspecifyhowtheagent
changesitspolicyasaresultofexperience.
Roughly,theagentsgoalistogetasmuchrewardasit
canoverthelongrun.
GettingtheDegreeofAbstractionRight
Timestepsneednotrefertofixedintervalsofrealtime.
Actionscanbelowlevel(e.g.,voltagestomotors),orhigh
level(e.g.,acceptajoboffer),mental(e.g.,shiftinfocus
ofattention),etc.
Statescanbelowlevelsensations,ortheycanbe
abstract,symbolic,basedonmemory,orsubjective(e.g.,
thestateofbeingsurprisedorlost).
AnRLagentisnotlikeawholeanimalorrobot,which
consistofmanyRLagentsaswellasothercomponents.
Theenvironmentisnotnecessarilyunknowntotheagent,
onlyincompletelycontrollable.
Rewardcomputationisintheagentsenvironmentbecause
theagentcannotchangeitarbitrarily.
GoalsandRewards
Isascalarrewardsignalanadequatenotionofagoal?
maybenot,butitissurprisinglyflexible.
Agoalshouldspecifywhatwewanttoachieve,nothow
wewanttoachieveit.
Agoalmustbeoutsidetheagentsdirectcontrolthus
outsidetheagent.
Theagentmustbeabletomeasuresuccess:
explicitly;
frequentlyduringitslifespan.
Returns
Supposethesequenceofrewardsafterstep tis :
rt 1 , rt 2 , rt 3 , K
Whatdowewanttomaximize?
Ingeneral,
wewanttomaximizethe expectedreturn, ERt ,foreachstept.
Episodictasks:interactionbreaksnaturallyinto
episodes,e.g.,playsofagame,tripsthroughamaze.
Rt rt 1 rt 2 L rT ,
whereTisafinaltimestepatwhichaterminalstateis
reached,endinganepisode.
ReturnsforContinuingTasks
Continuingtasks:interactiondoesnothavenaturalepisodes.
Discountedreturn:
Rt rt 1 rt 2 2 rt 3 L k rt k 1 ,
k 0
where , 0 1,isthediscountrate.
shortsighted0 1farsighted
AnExample
Avoidfailure:thepolefallingbeyond
acriticalangleorthecarthittingendof
track.
Asanepisodictaskwhereepisodeendsuponfailure:
reward 1foreachstepbeforefailure
return numberofstepsbeforefailure
Asacontinuingtaskwithdiscountedreturn:
Ineithercase,returnismaximizedby
avoidingfailureforaslongaspossible.
AnotherExample
Gettothetopofthehill
asquicklyaspossible.
Returnismaximizedbyminimizing
numberofstepsreachthetopofthehill.
AUnifiedNotation
Inepisodictasks,wenumberthetimestepsofeach
episodestartingfromzero.
Weusuallydonothavedistinguishbetweenepisodes,so
st
st, j
wewriteinsteadofforthestateatsteptof
episodej.
Thinkofeachepisodeasendinginanabsorbingstatethat
alwaysproducesrewardofzero:
R
Wecancoverallcasesbywriting t rt k 1 ,
k 0
where canbe1onlyifazerorewardabsorbingstateisalwaysreached.
TheMarkovProperty
Bythestateatstept,thebookmeanswhateverinformationis
availabletotheagentatsteptaboutitsenvironment.
Thestatecanincludeimmediatesensations,highlyprocessed
sensations,andstructuresbuiltupovertimefromsequencesof
sensations.
Ideally,astateshouldsummarizepastsensationssoastoretain
allessentialinformation,i.e.,itshouldhavetheMarkov
Property:
Prst 1 s,r
t 1 r st ,at ,rt , st 1 ,at 1 ,K ,r1 ,s0 ,a0
Prst 1 s,r
t 1 r st ,at
MarkovDecisionProcesses
IfareinforcementlearningtaskhastheMarkovProperty,itis
basicallyaMarkovDecisionProcess(MDP).
Ifstateandactionsetsarefinite,itisafiniteMDP.
TodefineafiniteMDP,youneedtogive:
stateandactionsets
onestepdynamicsdefinedbytransitionprobabilities:
Psas Prst 1 s st s,at aforalls, sS, a A(s).
rewardprobabilities:
Rsas Ert 1 st s,at a,s t 1 sforalls, sS, a A(s).
AnExampleFiniteMDP
RecyclingRobot
Ateachstep,robothastodecidewhetheritshould(1)activelysearchforacan,
(2)waitforsomeonetobringitacan,or(3)gotohomebaseandrecharge.
Searchingisbetterbutrunsdownthebattery;ifrunsoutofpowerwhile
searching,hastoberescued(whichisbad).
Decisionsmadeonbasisofcurrentenergylevel:high,low.
Reward=numberofcanscollected
ValueFunctions
Thevalueofastateistheexpectedreturnstartingfrom
thatstate;dependsontheagentspolicy:
State valuefunctionforpolicy :
k
V (s) E Rt st s E rt k 1 st s
k 0
Thevalueoftakinganactioninastateunderpolicy
istheexpectedreturnstartingfromthatstate,takingthat
action,andthereafterfollowing:
Action valuefunctionforpolicy :
k
Q (s, a) E Rt s t s, at a E rt k 1 s t s,at a
k 0
BellmanEquationforaPolicy
Thebasicidea:
Rt rt 1 rt 2 2 rt 3 3 rt 4 L
rt 1 rt 2 rt 3 rt 4 L
2
rt 1 Rt 1
So:
V (s) E Rt st s
E rt 1 V st 1 st s
Or,withouttheexpectationoperator:
V (s) (s,a) PssRss V ( s)
MoreontheBellmanEquation
V (s) (s,a) PssRss V ( s)
Thisisasetofequations(infact,linear),oneforeachstate.
Thevaluefunctionforisitsuniquesolution.
Backupdiagrams:
forV
forQ
Gridworld
Actions:north,south,east,west;deterministic.
Ifwouldtakeagentoffthegrid:nomovebutreward=1
Otheractionsproducereward=0,exceptactionsthatmove
agentoutofspecialstatesAandBasshown.
Statevaluefunction
forequiprobable
randompolicy;
=0.9
OptimalValueFunctions
ForfiniteMDPs,policiescanbepartiallyordered:
ifandonlyif V (s) V (s)foralls S
Thereisalwaysatleastone(andpossiblymany)policiesthat
isbetterthanorequaltoalltheothers.Thisisanoptimal
policy.Wedenotethemall*.
Optimalpoliciessharethesameoptimalstatevaluefunction:
Optimalpoliciesalsosharethesameoptimalactionvalue
function:
Q (s,a) max Q (s, a)foralls Sanda A(s)
Thisistheexpectedreturnfortakingactionainstates
andthereafterfollowinganoptimalpolicy.
BellmanOptimalityEquationforV*
Thevalueofastateunderanoptimalpolicymustequal
theexpectedreturnforthebestactionfromthatstate:
aA(s)
Therelevantbackupdiagram:
V
istheuniquesolutionofthissystemofnonlinearequations.
BellmanOptimalityEquationforQ*
Therelevantbackupdiagram:
Q
istheuniquesolutionofthissystemofnonlinearequations.
WhyOptimalStateValueFunctionsareUseful
V
Anypolicythatisgreedywithrespecttoisanoptimalpolicy.
V
Therefore,given,onestepaheadsearchproducesthe
longtermoptimalactions.
E.g.,backtothegridworld:
WhatAboutOptimalActionValueFunctions?
*
Q
Given,theagentdoesnoteven
havetodoaonestepaheadsearch:
SolvingtheBellmanOptimalityEquation
FindinganoptimalpolicybysolvingtheBellmanOptimality
Equationrequiresthefollowing:
accurateknowledgeofenvironmentdynamics;
wehaveenoughspaceantimetodothecomputation;
theMarkovProperty.
Howmuchspaceandtimedoweneed?
polynomialinnumberofstates(viadynamicprogramming
methods;Chapter4),
BUT,numberofstatesisoftenhuge(e.g.,backgammonhas
about10**20states).
Weusuallyhavetosettleforapproximations.
ManyRLmethodscanbeunderstoodasapproximatelysolvingthe
BellmanOptimalityEquation.
TDPrediction
PolicyEvaluation(thepredictionproblem):
foragivenpolicy,computethestatevaluefunction V
ThesimplestTDmethod, TD(0) :
V(st ) V(st ) rt 1 V (st1 ) V(st )
target:anestimateofthereturn
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
24
SimplestTDMethod
V(st ) V(st ) rt 1 V (st1 ) V(st )
st
rt 1
st 1
TT
TT
T
T
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
TT
T
T
TT
25
Example:DrivingHome
State
ElapsedTime
(minutes)
leavingoffice
0
Predicted
TimetoGo
30
Predicted
TotalTime
30
reachcar,
raining
exithighway
35
40
20
15
35
behindtruck
30
10
40
homestreet
40
43
arrivehome
43
43
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
26
DrivingHome
Changesrecommendedby
MonteCarlomethods=1)
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
Changesrecommended
byTDmethods(=1)
27
AdvantagesofTDLearning
TDmethodsdonotrequireamodeloftheenvironment,
onlyexperience
TDmethodscanbefullyincremental
Youcanlearnbeforeknowingthefinaloutcome
Lessmemory
Lesspeakcomputation
Youcanlearnwithoutthefinaloutcome
Fromincompletesequences
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
28
RandomWalkExample
ValueslearnedbyTD(0)
after
variousnumbersofepisodes
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
29
TDandMContheRandomWalk
Dataaveragedover
100sequencesofepisodes
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
30
OptimalityofTD(0)
BatchUpdating:traincompletelyonafiniteamountofdata,
e.g.,trainrepeatedlyon10episodesuntilconvergence.
ComputeupdatesaccordingtoTD(0),butonlyupdate
estimatesaftereachcompletepassthroughthedata.
ForanyfiniteMarkovpredictiontask,underbatchupdating,
TD(0)convergesforsufficientlysmall.
ConstantMCalsoconvergesundertheseconditions,butto
adifferenceanswer!
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
31
RandomWalkunderBatchUpdating
Aftereachnewepisode,allpreviousepisodesweretreatedasabatch,
andalgorithmwastraineduntilconvergence.Allrepeated100times.
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
32
LearningAnActionValueFunction
Estimate Q forthecurrentbehaviorpolicy .
Aftereverytransitionfromanonterminalstate st , dothis :
Q st , at Qst , at rt 1 Qst 1 ,at 1 Qst ,at
Ifst 1isterminal, thenQ(st 1, at 1 ) 0.
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
33
Sarsa:OnPolicyTDControl
Turnthisintoacontrolmethodbyalwaysupdatingthe
policytobegreedywithrespecttothecurrentestimate:
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
34
WindyGridworld
undiscounted,episodic,reward=1untilgoal
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
35
ResultsofSarsaontheWindyGridworld
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
36
Cliffwalking
greedy=0.1
R.S.SuttonandA.G.Barto:ReinforcementLearning:AnIntroduction
37