Per step reward function

[sagnik119]

In the training part, is it assumed that the reward is 0 in all states except the terminal state? If this is so, what will the changes in the code be if each step in an episode has a reward for a state transition? Would the rewards in the code be replaced by returns in that case?

[YuriCat]

Thanks for your great question!

I used simple Monte Carlo returns in the current code.
If you'd like to use per-step rewards (along with TD(λ), etc.), you would calculate the value target recursively, and then calculate the advantage for the policy gradient method.
The following code will help you:
https://github.com/DeNA/HandyRL/blob/master/handyrl/losses.py#L20

The simulation code during episode generation would also need to be changed.
I simply used the average of the estimated values for each step in the simulation, but you need to consider rewards here, and the estimated reward output from the model will be necessary as well.