def calc_policy_loss(self, model, policy_params, policy_entropy, rollout):
actions = rollout.batch_tensor('<STR_LIT>')<EOL>advantages = rollout.batch_tensor('<STR_LIT>')<EOL>fixed_logprobs = rollout.batch_tensor('<STR_LIT>')<EOL>model_logprobs = model.logprob(actions, policy_params)<EOL>advantages = (advantages - advantages.mean()) / (advantages.std() + <NUM_LIT>)<EOL>policy_loss = -advantages * torch.exp(model_logprobs - fixed_logprobs)<EOL>return policy_loss.mean() - policy_entropy * self.entropy_coef<EOL>
Policy gradient loss - calculate from probability distribution

Calculate surrogate loss - advantage * policy_probability / fixed_initial_policy_probability

Because we operate with logarithm of -probability (neglogp) we do
- advantage * exp(fixed_neglogps - model_neglogps)