4.2 Q-learning(advanced)

Q-learning

We use off-policy in the Q-learning. We apply greedy action for target policy, and $\epsilon$-greedy action for behavior policy.

equation for policies

target policy

Since we use greedy action for target policy, we need the best action for every future action(we use delta function!!)
$\mathrm{target} : p(a_{t+1}|s_{t+1}) = \delta(a_{t+1} - a_{t+1}^*), \\a_{t+1}^* = \argmax\limits_{a_{t+1}} Q(s_{t+1}, a_{t+1})$
We have learn this idea(delta function) at 3.1 - optimal policy.

behavior policy

The behavior policy follows $\epsilon$-greedy action.

final equation

from the final equation, we now can sample a data(TD-target).
$R_t + \gamma \max\limits_{a_{t+1}}Q(s_{t+1},a_{t+1})$