2.3 Bellman equation

Bellman equation is about method of representing state value function and action value function.

Bellman equation for value state function

Let’s change the form of state value function(using bayesian rule).
Recall Bayesian rule:
$p(x,y) = p(x|y)p(y)$
$p(x,y|z) = p(x|y,z)p(y|z)$
The state value function becomes:

We can find the blue box is actually action value function.
So the state value function becomes:
$V(s_t) = \int\limits_{a_t} Q(s_t, a_t) p(a_t|s_t)da_t$
For example, in the Q-learning, the state value function is expectation of all Q in the current state.

Let's change the equation in different way.

As we learned, if we already have information about $s_{t+1}$, we don't need $s_t, a_t$ anymore.
$G_t$ can be expressed as $R_t + \gamma R_{t+1}$Therefore, the equation becomes:
$V(s_t)\\=\int\limits_{a_t,s_{t+1}} \int\limits_{a_{t+1}:a_\infin} (R_t + \gamma G_{t+1}) p(a_{t+1},...|s_{t+1})da_{t+1}:a_\infin p(a_t, s_{t+1}|s_t)da_t,s_{t+1}$

If we look at the equation, we know that
$\int\limits_{a_{t+1}:a_\infin} G_{t+1} p(a_{t+1},...|s_{t+1})da_{t+1}:a_\infin = V(s_{t+1})$
The final equation is:
$\begin{aligned}V(s_t)&=\int\limits_{a_t,s_{t+1}} (R_t + \gamma V(s_{t+1})) p(a_t, s_{t+1}|s_t)da_t,s_{t+1}\\ &=\int\limits_{a_t,s_{t+1}} (R_t + \gamma V(s_{t+1})) \colorbox{lightgreen}{$p(s_{t+1}|s_t,a_t)$}\colorbox{aqua}{$p(a_t|s_t)$}da_t,s_{t+1}\end {aligned}$
This equation form has an advantage in being able to see the policy(blue box). The green box is called transition probability.

Bellman equation for action value function

Now, we will change the form of action value function.

We have applied same principle to changing equation. As we can see the equation, we can express Q with $V(s_{t+1})$.

Let's change the action value function again.

Again, by changing the equation, we reduced a calculation for integration, and also found the policy(blue box)and the transition probability(green box) in the equation.