SAC (Soft Actor Critic)是一种对随机策略进行非策略优化的算法，在随机策略优化和ddpg方法之间架起了一座桥梁。它并不是TD3的直接继承者(几乎是同时发布的)，但它包含了短双q技巧，并且由于SAC政策的固有随机性，它也最终受益于目标政策平滑之类的东西。

SAC的一个中心特征是熵正则化。该策略被训练成最大化期望收益和熵之间的权衡，熵是该策略中随机性的度量。这与探索-利用的权衡关系密切:熵的增加会导致更多的探索，从而加速以后的学习。它还可以防止策略过早地收敛到一个坏的局部最优。

Entropy-Regularized Reinforcement Learning

Entropy is a quantity which, roughly speaking, says how random a random variable is. If a coin is weighted so that it almost always comes up heads, it has low entropy; if it’s evenly weighted and has a half chance of either outcome, it has high entropy.

Let x be a random variable with probability mass or density function P. The entropy H of x is computed from its distribution P according to

$$H(P) = \mathop{\text{E}}\limits_{x\sim P}[-\log P(x)].$$

在熵正则化强化学习中，agent在每个时间步上获得的奖励与策略在该时间步上的熵成正比。这将RL问题改变为:

$$\pi^* = \arg \max_{\pi} \mathop{\text{E}}\limits_{\tau \sim \pi}{ \sum_{t=0}^{\infty} \gamma^t \bigg( R(s_t, a_t, s_{t+1}) + \alpha H\left(\pi(\cdot|s_t)\right) \bigg)},$$

其中$\alpha > 0$是权衡系数。(注意:我们在这里假设一个无限视界贴现设置，我们将在本页面的其余部分做同样的事情。)现在我们可以在这个设置中定义稍微不同的值函数。$V^{\pi}$被修改为包含每个时间步的熵加成:

$$V^{\pi}(s) = \mathop{\text{E}}\limits_{\tau \sim \pi} \Bigg [ \left. \sum_{t=0}^{\infty} \gamma^t \bigg( R(s_t, a_t, s_{t+1}) + \alpha H\left(\pi(\cdot|s_t)\right) \bigg) \right| s_0 = s \Bigg ]$$

$Q^{\pi}$被修改为包含除了第一个时间步之外的每个时间步的熵奖励:

$$Q^{\pi}(s,a) = \mathop{\text{E}}\limits_{\tau \sim \pi}{ \Bigg [\left. \sum_{t=0}^{\infty} \gamma^t R(s_t, a_t, s_{t+1}) + \alpha \sum_{t=1}^{\infty} \gamma^t H\left(\pi(\cdot|s_t)\right)\right| s_0 = s, a_0 = a}\Bigg ]$$

有了这些定义，$V^{\pi}$和$Q^{\pi}$之间是这样连接的:

$$V^\pi(s) = \mathop{\text{E}}\limits_{a \sim \pi}[Q^\pi(s,a)] + \alpha H(\pi(\cdot | s))$$

$Q^{\pi}$的Bellman方程为:

$$Q^{\pi}(s,a) = \mathop{\text{E}}\limits_{s' \sim P, a' \sim \pi}{R(s,a,s') + \gamma\left(Q^{\pi}(s',a') + \alpha H\left(\pi(\cdot|s')\right) \right)} \\ = \mathop{\text{E}}\limits_{s' \sim P}{R(s,a,s') + \gamma V^{\pi}(s')}.$$

Soft Actor-critic

SAC同时学习一个策略$pi{\ θ}$和两个$q$函数$Q{\phi1}$，$Q{\phi_2}$。目前标准的SAC有两种变体:一种使用固定的熵正则化系数$\alpha$，另一种通过在培训过程中改变$\alpha$来强制熵约束。

Algorithm