《强化学习：原理与Python实现》更新与勘误

（2019年8月第1版第1次印刷）

Gym 0.25版本更新了API，GitHub代码已相应更新。代码以GitHub上为准。

行数计算方法

本勘误文档中，行数计算“第$i$$i$行”（$i=0,1,2,\dots$$i=0,1,2,\ldots$）是从0开始计数的。小节标题、公式、内联代码、注意、本章要点记入行数，章标题、图、表、代码清单及它们的题注不计入行数。空行不计入行数。

$$$\newcommand{\sfA}{\unicode{x1d608}} \newcommand{\sfR}{\unicode{x1d619}} \newcommand{\sfS}{\unicode{x1d61a}} \newcommand{\sfa}{\unicode{x1d622}} \newcommand{\sfs}{\unicode{x1d634}} \newcommand{\sfx}{\unicode{x1d639}} \newcommand{\bftheta}{\pmb{\unicode{x3B8}}} \newcommand{\E}{\textrm{E}}$

第5页第19行

记为$\text{𝘙}$$\sfR$

改为

记为$R$$R$

第11页第17行

动作空间是Dicrete(3)

改为

动作空间是Discrete(3)

第17页第6行

作者注： “轨道”又称为“轨迹”。本书中这两个词混用。

第20页第2行

作者注： 这种带折扣的回报定义既可以用于回合制任务，也可以用于连续性任务，是一种统一的表示。

第20页第9行

为$\overline{R}=\underset{t\to +\mathrm{\infty }}{lim}\text{E}\left[\frac{1}{t}\sum _{\tau =0}^t{R}_{\tau }\right]$$\bar{R}=\lim\limits_{t\to+\infty}\E\left[\frac{1}{t}\sum\limits_{\tau=0}^{t}R_\tau\right]$

改为

为$\overline{R}=\underset{t\to +\mathrm{\infty }}{lim}\text{E}\left[\frac{1}{t}\sum _{\tau =1}^t{R}_{\tau }\right]$$\bar{R}=\lim\limits_{t\to+\infty}\E\left[\frac{1}{t}\sum\limits_{\tau=1}^{t}R_\tau\right]$

第20页第24-27行

$q_{\pi }(\text{吃}\mid \text{饿})={\text{E}}_{\pi }[G_t\mid {\text{𝘚}}_t=\text{饿},{\text{𝘈}}_t=\text{吃}]$$q_\pi\left(吃\mid饿\right)=\E_\pi\left[G_t\mid\sfS_t=饿,\sfA_t=吃\right]$

$q_{\pi }(\text{不}\text{吃}\mid \text{饿})={\text{E}}_{\pi }[G_t\mid {\text{𝘚}}_t=\text{饿},{\text{𝘈}}_t=\text{不}\text{吃}]$$q_\pi\left(不吃\mid饿\right)=\E_\pi\left[G_t\mid\sfS_t=饿,\sfA_t=不吃\right]$

$q_{\pi }(\text{吃}\mid \text{饱})={\text{E}}_{\pi }[G_t\mid {\text{𝘚}}_t=\text{饱},{\text{𝘈}}_t=\text{吃}]$$q_\pi\left(吃\mid饱\right)=\E_\pi\left[G_t\mid\sfS_t=饱,\sfA_t=吃\right]$

$q_{\pi }(\text{不}\text{吃}\mid \text{饱})={\text{E}}_{\pi }[G_t\mid {\text{𝘚}}_t=\text{饱},{\text{𝘈}}_t=\text{不}\text{吃}]$$q_\pi\left(不吃\mid饱\right)=\E_\pi\left[G_t\mid\sfS_t=饱,\sfA_t=不吃\right]$

改为

$q_{\pi }(\text{饿},\text{吃})={\text{E}}_{\pi }[G_t\mid {\text{𝘚}}_t=\text{饿},{\text{𝘈}}_t=\text{吃}]$$q_\pi\left(饿,吃\right)=\E_\pi\left[G_t\mid\sfS_t=饿,\sfA_t=吃\right]$

$q_{\pi }(\text{饿},\text{不}\text{吃})={\text{E}}_{\pi }[G_t\mid {\text{𝘚}}_t=\text{饿},{\text{𝘈}}_t=\text{不}\text{吃}]$$q_\pi\left(饿,不吃\right)=\E_\pi\left[G_t\mid\sfS_t=饿,\sfA_t=不吃\right]$

$q_{\pi }(\text{饱},\text{吃})={\text{E}}_{\pi }[G_t\mid {\text{𝘚}}_t=\text{饱},{\text{𝘈}}_t=\text{吃}]$$q_\pi\left(饱,吃\right)=\E_\pi\left[G_t\mid\sfS_t=饱,\sfA_t=吃\right]$

$q_{\pi }(\text{饱},\text{不}\text{吃})={\text{E}}_{\pi }[G_t\mid {\text{𝘚}}_t=\text{饱},{\text{𝘈}}_t=\text{不}\text{吃}]$$q_\pi\left(饱,不吃\right)=\E_\pi\left[G_t\mid\sfS_t=饱,\sfA_t=不吃\right]$

第22页图2-2b)中图注

$v_{\pi }(\text{𝘴})$$v_\pi(\sfs)$

改为

$v_{\pi }({\text{𝘴}}^{\prime })$$v_\pi(\sfs')$

第2.2节和第2.3节

有观点认为：用动作价值表示状态价值的表达式，或是用状态价值表示动作价值的表达式，不能称为Bellman方程；而用动作价值表示动作价值的表达式，或是用状态价值表示状态价值的表达式，才可以称为Bellman方程。

第22页第18行

$q_{\pi }(\text{𝘴},\text{𝘢})=\gamma \sum _{{\text{𝘴}}^{\prime },r}p({\text{𝘴}}^{\prime },r\mid \text{𝘴},\text{𝘢})[r+\sum _{{\text{𝘢}}^{\prime }}\pi ({\text{𝘢}}^{\prime }|{\text{𝘴}}^{\prime })q_{\pi }({\text{𝘴}}^{\prime },{\text{𝘢}}^{\prime })]$${q_\pi}\left(\sfs,\sfa\right)=\gamma \sum\limits_{\sfs',r}{p\left(\sfs',r\mid\sfs,\sfa\right)\left[r+\sum\limits_{\sfa'}{\pi\left(\sfa'|\sfs'\right){q_\pi}\left(\sfs',\sfa'\right)}\right]}$

改为

$q_{\pi }(\text{𝘴},\text{𝘢})=\sum _{{\text{𝘴}}^{\prime },r}p({\text{𝘴}}^{\prime },r\mid \text{𝘴},\text{𝘢})[r+\gamma \sum _{{\text{𝘢}}^{\prime }}\pi ({\text{𝘢}}^{\prime }|{\text{𝘴}}^{\prime })q_{\pi }({\text{𝘴}}^{\prime },{\text{𝘢}}^{\prime })]$${q_\pi}\left(\sfs,\sfa\right)=\sum\limits_{\sfs',r}{p\left(\sfs',r\mid\sfs,\sfa\right)\left[r+\gamma \sum\limits_{\sfa'}{\pi\left(\sfa'|\sfs'\right){q_\pi}\left(\sfs',\sfa'\right)}\right]}$

第23页第19行

$\left[\begin{array}{cccccc}1& 0& x-1& -x& 0& 0\\ \\ 0& 1& 0& 0& -y& y-1\\ \\ -\gamma & 0& 1& 0& 0& 0\\ \\ (\alpha -1)\gamma & -\alpha \gamma & 0& 1& 0& 0\\ \\ -\beta \gamma & (\beta -1)\gamma & 0& 0& 1& 0\\ \\ 0& -\gamma & 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{v}_{\pi }\left(\text{饿}\right)\\ \\ v_{\pi }\left(\text{饱}\right)\\ \\ q_{\pi }(\text{饿},\text{不}\text{吃})\\ \\ q_{\pi }(\text{饿},\text{吃})\\ \\ q_{\pi }(\text{饱},\text{不}\text{吃})\\ \\ q_{\pi }(\text{饱},\text{吃})\end{array}\right]=\left[\begin{array}{c}0\\ \\ 0\\ \\ 2\\ \\ -4\alpha +3\\ \\ 4\beta -2\\ \\ -1\end{array}\right]$$\begin{bmatrix}1&0&x-1&-x&0&0\\\\0&1&0&0&-y&y-1\\\\-\gamma&0&1&0&0&0\\\\\left(\alpha-1\right)\gamma&-\alpha\gamma&0&1&0&0\\\\-\beta\gamma&\left(\beta-1\right)\gamma&0&0&1&0\\\\0&-\gamma&0&0&0&1\end{bmatrix}\begin{bmatrix}v_\pi\left(饿\right)\\\\v_\pi\left(饱\right)\\\\q_\pi\left(饿,不吃\right)\\\\q_\pi\left(饿,吃\right)\\\\q_\pi\left(饱,不吃\right)\\\\q_\pi\left(饱,吃\right)\end{bmatrix}=\begin{bmatrix}0\\\\0\\\\2\\\\-4\alpha+3\\\\4\beta-2\\\\-1\end{bmatrix}$

改为

$\left[\begin{array}{cccccc}1& 0& x-1& -x& 0& 0\\ \\ 0& 1& 0& 0& -y& y-1\\ \\ -\gamma & 0& 1& 0& 0& 0\\ \\ (\alpha -1)\gamma & -\alpha \gamma & 0& 1& 0& 0\\ \\ -\beta \gamma & (\beta -1)\gamma & 0& 0& 1& 0\\ \\ 0& -\gamma & 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{v}_{\pi }\left(\text{饿}\right)\\ \\ v_{\pi }\left(\text{饱}\right)\\ \\ q_{\pi }(\text{饿},\text{不}\text{吃})\\ \\ q_{\pi }(\text{饿},\text{吃})\\ \\ q_{\pi }(\text{饱},\text{不}\text{吃})\\ \\ q_{\pi }(\text{饱},\text{吃})\end{array}\right]=\left[\begin{array}{c}0\\ \\ 0\\ \\ -2\\ \\ 4\alpha -3\\ \\ -4\beta +2\\ \\ 1\end{array}\right]$$\begin{bmatrix}1&0&x-1&-x&0&0\\\\0&1&0&0&-y&y-1\\\\-\gamma&0&1&0&0&0\\\\\left(\alpha-1\right)\gamma&-\alpha\gamma&0&1&0&0\\\\-\beta\gamma&\left(\beta-1\right)\gamma&0&0&1&0\\\\0&-\gamma&0&0&0&1\end{bmatrix}\begin{bmatrix}v_\pi\left(饿\right)\\\\v_\pi\left(饱\right)\\\\q_\pi\left(饿,不吃\right)\\\\q_\pi\left(饿,吃\right)\\\\q_\pi\left(饱,不吃\right)\\\\q_\pi\left(饱,吃\right)\end{bmatrix}=\begin{bmatrix}0\\\\0\\\\-2\\\\4\alpha-3\\\\-4\beta+2\\\\1\end{bmatrix}$

第24页代码清单2-1

代码改为

xxxxxxxxxx
import sympy
from sympy import symbols
sympy.init_printing()
v_hungry, v_full = symbols('v_hungry v_full')
q_hungry_eat, q_hungry_none, q_full_eat, q_full_none = \
        symbols('q_hungry_eat q_hungry_none q_full_eat q_full_none')
alpha, beta, x, y, gamma = symbols('alpha beta x y gamma')
system = sympy.Matrix((
        (1, 0, x-1, -x, 0, 0, 0),
        (0, 1, 0, 0, -y, y-1, 0),
        (-gamma, 0, 1, 0, 0, 0, -2),
        ((alpha-1)*gamma, -alpha*gamma, 0, 1, 0, 0, 4*alpha-3),
        (-beta*gamma, (beta-1)*gamma, 0, 0, 1, 0, -4*beta+2),
        (0, -gamma, 0, 0, 0, 1, 1) )) # 标准形式的系数矩阵
sympy.solve_linear_system(system,
        v_hungry, v_full,
        q_hungry_none, q_hungry_eat, q_full_none, q_full_eat)

第24页第2行~第11行

求解有误，更改如下

$v_{\pi }\left(\text{饿}\right)=\frac{1}{\mathrm{\Delta }}(\alpha \gamma xy-3\alpha \gamma x+4\alpha x-\beta \gamma xy-2\beta \gamma y+\gamma x+2\gamma -x-2)$$v_\pi\left(饿\right)=\frac{1}{\Delta}\left(\alpha\gamma xy-3\alpha\gamma x+4\alpha x-\beta\gamma xy-2\beta\gamma y+\gamma x+2\gamma-x-2\right)$

$v_{\pi }\left(\text{饱}\right)=\frac{1}{\mathrm{\Delta }}(\alpha \gamma xy+\alpha \gamma x-\beta \gamma xy+2\beta \gamma y-4\beta y-\gamma y-\gamma +y+1)$$v_\pi\left(饱\right)=\frac{1}{\Delta}\left(\alpha\gamma xy+\alpha\gamma x-\beta\gamma xy+2\beta\gamma y-4\beta y-\gamma y-\gamma+y+1\right)$

$q_{\pi }(\text{饿},\text{不}\text{吃})=\frac{1}{\mathrm{\Delta }}(\alpha {\gamma }^{2}xy-\alpha {\gamma }^{2}x+2\alpha \gamma x-\beta {\gamma }^{2}xy-2\beta \gamma y+{\gamma }^{2}x-\gamma x+2\gamma -2)$$q_\pi\left(饿,不吃\right)=\frac{1}{\Delta}\left(\alpha\gamma^2xy-\alpha\gamma^2x+2\alpha\gamma x-\beta\gamma^2xy-2\beta\gamma y+\gamma^2x-\gamma x+2\gamma-2\right)$

$q_{\pi }(\text{饿},\text{吃})=\frac{1}{\mathrm{\Delta }}(\alpha {\gamma }^{2}xy-\alpha {\gamma }^{2}x-\alpha {\gamma }^{2}y+\alpha {\gamma }^2+2\alpha \gamma x+\alpha \gamma y-5\alpha \gamma +4\alpha -\beta {\gamma }^{2}xy+\beta {\gamma }^{2}y-$$q_\pi\left(饿,吃\right)=\frac{1}{\Delta}\left(\alpha\gamma^2xy-\alpha\gamma^2x-\alpha\gamma^2y+\alpha\gamma^2+2\alpha\gamma x+\alpha\gamma y-5\alpha\gamma+4\alpha-\beta\gamma^2xy+\beta\gamma^2y-\right.$

$3\beta \gamma y+{\gamma }^{2}x-{\gamma }^2-\gamma x+4\gamma -3)$$\left.3\beta\gamma y+\gamma^2x-\gamma^2-\gamma x+4\gamma-3\right)$

$q_{\pi }(\text{饱},\text{不}\text{吃})=\frac{1}{\mathrm{\Delta }}(\alpha {\gamma }^{2}xy-\alpha {\gamma }^{2}x+2\alpha \gamma x-\beta {\gamma }^{2}xy+\beta {\gamma }^{2}x+\beta {\gamma }^{2}y-\beta {\gamma }^2-\beta \gamma x-3\beta \gamma y+$$q_\pi\left(饱,不吃\right)=\frac{1}{\Delta}\left(\alpha\gamma^2xy-\alpha\gamma^2x+2\alpha\gamma x-\beta\gamma^2xy+\beta\gamma^2x+\beta\gamma^2y-\beta\gamma^2-\beta\gamma x-3\beta\gamma y+\right.$

$5\beta \gamma -4\beta -{\gamma }^{2}y+{\gamma }^2+\gamma y-3\gamma +2)$$\left.5\beta\gamma-4\beta-\gamma^2{y}+\gamma^2+\gamma{y}-3\gamma+2\right)$

$q_{\pi }(\text{饱},\text{吃})=\frac{1}{\mathrm{\Delta }}(\alpha {\gamma }^{2}xy+\alpha \gamma x-\beta {\gamma }^{2}xy+\beta {\gamma }^{2}y-3\beta \gamma y-{\gamma }^{2}y+\gamma y-\gamma +1)$$q_\pi\left(饱,吃\right)=\frac{1}{\Delta}\left(\alpha\gamma^2xy+\alpha\gamma x-\beta\gamma^2xy+\beta\gamma^2y-3\beta\gamma y-\gamma^2y+\gamma y-\gamma+1\right)$

其中

$\mathrm{\Delta }=(1-\gamma )(1-(1-\alpha x-\beta y)\gamma )$$\Delta=\left(1-\gamma\right)\left(1-\left(1-\alpha{x}-\beta{y}\right)\gamma\right)$

第25页第2行

如果对于任意$\text{𝘴}\in \mathcal{S}$$\sfs\in\mathcal{S}$，都$v_{\pi }\left(\text{𝘴}\right)\le v_{\ast }\left(\text{𝘴}\right)$${v_{\pi}}\left(\sfs\right)\le{v_*}\left(\sfs\right)$，则称

改为

如果对于任意$\text{𝘴}\in \mathcal{S}$$\sfs\in\mathcal{S}$都满足$v_{\pi }\left(\text{𝘴}\right)\le v_{\ast }\left(\text{𝘴}\right)$${v_{\pi}}\left(\sfs\right)\le{v_*}\left(\sfs\right)$，则称

第26页第2行和第7行

$\gamma \sum _{{\text{𝘴}}^{\prime }}p({\text{𝘴}}^{\prime },r\mid \text{𝘴},\text{𝘢})v_{\ast }\left({\text{𝘴}}^{\prime }\right)$$\gamma\sum_{\sfs'}{p\left(\sfs',r\mid\sfs,\sfa\right)}v_*\left(\sfs'\right)$

（共2处）

改为

$\gamma \sum _{{\text{𝘴}}^{\prime }}p({\text{𝘴}}^{\prime }\mid \text{𝘴},\text{𝘢})v_{\ast }\left({\text{𝘴}}^{\prime }\right)$$\gamma\sum_{\sfs'}{p\left(\sfs'\mid\sfs,\sfa\right)}v_*\left(\sfs'\right)$

第27页第0~5行

所有的下标$\pi$$\pi$改为下标$\ast$$\ast$（共16处）。

第27页代码清单2-2

代码改为

xxxxxxxxxx
import sympy
from sympy import symbols
sympy.init_printing()
alpha, beta, gamma = symbols('alpha beta gamma')
v_hungry, v_full = symbols('v_hungry v_full')
q_hungry_eat, q_hungry_none, q_full_eat, q_full_none = \
        symbols('q_hungry_eat q_hungry_none q_full_eat q_full_none')
xy_tuples = ((0, 0), (1, 0), (0, 1), (1, 1))
for x, y in xy_tuples: # 分类讨论
    system = sympy.Matrix((
            (1, 0, x-1, -x, 0, 0, 0),
            (0, 1, 0, 0, -y, y-1, 0),
            (-gamma, 0, 1, 0, 0, 0, -2),
            ((alpha-1)*gamma, -alpha*gamma, 0, 1, 0, 0, 4*alpha-3),
            (-beta*gamma, (beta-1)*gamma, 0, 0, 1, 0, -4*beta+2),
            (0, -gamma, 0, 0, 0, 1, 1) ))
    result = sympy.solve_linear_system(system,
            v_hungry, v_full,
            q_hungry_none, q_hungry_eat, q_full_none, q_full_eat, simplification=True)
    msgx = 'v(饿) = q(饿,{}吃)'.format('' if x else '不')
    msgy = 'v(饱) = q(饱,{}吃)'.format('不' if y else '')
    print('==== {}, {} ==== x = {}, y = {} ===='.format(msgx, msgy, x, y))
    display(result)

第27页第10行~第29页第8行

求解有误，更改如下

接下来进一步分析这个方程组。比较最优价值的方程组和一般策略的价值方程组可以发现，最优策略价值方程组的解正是在一般策略方程组的解中$(x,y)$$\left(x,y\right)$分别取(0,0)、(0,1)、(1,0)、(1,1)得到。所以，我们用一般策略价值方程组的解的形式来表示最优价值方程组的解，并比较其中$q_{\ast }(\text{饱}$$q_\ast\left(\right.饱$,$\text{不}\text{吃})$$不吃\left.\right)$和$q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,吃\right)$的大小，以及$q_{\ast }(\text{饿},\text{不}\text{吃})$$q_\ast\left(饿,不吃\right)$和$q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,吃\right)$的大小。在比较时，注意到$x,y\in [0,1]$$x,y\in\left[0,1\right]$并且已经假设$\alpha ,\beta ,\gamma \in (0,1)$$\alpha,\beta,\gamma\in\left(0,1\right)$。所以可以验证

$\mathrm{\Delta }=(1-\gamma )(1-(1-\alpha x-\beta y)\gamma )>0$$\Delta=\left(1-\gamma\right)\left(1-\left(1-\alpha{x}-\beta{y}\right)\gamma\right)>0$

进而比较大小可以只比较分子部分。

首先来比较$q_{\ast }(\text{饿},\text{不}\text{吃})$$q_\ast\left(饿,不吃\right)$和$q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,吃\right)$的大小。注意到$q_{\ast }(\text{饿},\text{不}\text{吃})\le q_{\ast }(\text{饿},\text{不}\text{吃})$$q_\ast\left(饿,不吃\right)\le{q_\ast}\left(饿,不吃\right)$等价于

$\alpha {\gamma }^{2}xy-\alpha {\gamma }^{2}x+2\alpha \gamma x-\beta {\gamma }^{2}xy-2\beta \gamma y+{\gamma }^{2}x-\gamma x+2\gamma -2\le$$\alpha\gamma^2xy-\alpha\gamma^2x+2\alpha\gamma x-\beta\gamma^2xy-2\beta\gamma y+\gamma^2x-\gamma x+2\gamma-2\le$

$\alpha {\gamma }^{2}xy-\alpha {\gamma }^{2}x-\alpha {\gamma }^{2}y+\alpha {\gamma }^2+2\alpha \gamma x+\alpha \gamma y-5\alpha \gamma +4\alpha -\beta {\gamma }^{2}xy+\beta {\gamma }^{2}y-3\beta \gamma y+{\gamma }^{2}x-{\gamma }^2-\gamma x+4\gamma -3$$\alpha\gamma^2xy-\alpha\gamma^2x-\alpha\gamma^2y+\alpha\gamma^2+2\alpha\gamma x+\alpha\gamma y-5\alpha\gamma+4\alpha-\beta\gamma^2xy+\beta\gamma^2y-3\beta\gamma y+\gamma^2x-\gamma^2-\gamma x+4\gamma-3$

即

$-\alpha {\gamma }^{2}y+\alpha {\gamma }^2+\alpha \gamma y-5\alpha \gamma +4\alpha +\beta {\gamma }^{2}y-\beta \gamma y-{\gamma }^2+2\gamma -1\ge 0$$-\alpha\gamma^2y+\alpha\gamma^2+\alpha\gamma y-5\alpha\gamma+4\alpha+\beta\gamma^2y-\beta\gamma y-\gamma^2+2\gamma-1\ge0$

两边除以$(1-\gamma )$$\left(1-\gamma\right)$可得

$\alpha \gamma y-\alpha \gamma +4\alpha -\beta \gamma y+\gamma -1\ge 0$$\alpha\gamma{y}-\alpha\gamma+4\alpha-\beta\gamma y+\gamma-1\ge0$

注意到$1-\alpha +(\alpha -\beta )y>0$$1-\alpha+\left(\alpha-\beta\right)y>0$，上述不等式等价于

$\gamma \ge \frac{1-4\alpha }{1-\alpha +(\alpha -\beta )y}$$\gamma\ge\frac{1-4\alpha}{1-\alpha+\left(\alpha-\beta\right)y}$.

接下来比较$q_{\ast }(\text{饱},\text{不}\text{吃})$$q_\ast\left(饱,不吃\right)$和$q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,吃\right)$的大小。注意到$q_{\ast }(\text{饱},\text{不}\text{吃})\le q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)\le{q_\ast}\left(饱,吃\right)$等价于 $\alpha {\gamma }^{2}xy-\alpha {\gamma }^{2}x+2\alpha \gamma x-\beta {\gamma }^{2}xy+\beta {\gamma }^{2}x+\beta {\gamma }^{2}y-\beta {\gamma }^2-\beta \gamma x-3\beta \gamma y+5\beta \gamma -4\beta -{\gamma }^{2}y+{\gamma }^2+\gamma y-3\gamma +2\le$$\alpha{\gamma^2}xy-\alpha{\gamma^2}x+2\alpha\gamma{x}-\beta{\gamma^2}xy+\beta{\gamma^2}x+\beta{\gamma^2}y-\beta{\gamma^2}-\beta\gamma{x}-3\beta\gamma{y}+5\beta\gamma-4\beta-{\gamma^2}y+\gamma^2+\gamma{y}-3\gamma+2\le$

$\alpha {\gamma }^{2}xy+\alpha \gamma x-\beta {\gamma }^{2}xy+\beta {\gamma }^{2}y-3\beta \gamma y-{\gamma }^{2}y+\gamma y-\gamma +1$$\alpha{\gamma^2}xy+\alpha\gamma{x}-\beta{\gamma^2}xy+\beta{\gamma^2}y-3\beta\gamma{y}-{\gamma^2}y+\gamma{y}-\gamma+1$

即

$\alpha {\gamma }^{2}x-\alpha \gamma x-\beta {\gamma }^{2}x+\beta \gamma x+\beta {\gamma }^2-5\beta \gamma +4\beta -{\gamma }^2+2\gamma -1\ge 0$$\alpha{\gamma^2}x-\alpha\gamma{x}-\beta{\gamma^2}x+\beta\gamma{x}+\beta\gamma^2-5\beta\gamma+4\beta-\gamma^2+2\gamma-1\ge{0}$

两边除以$(1-\gamma )$$\left(1-\gamma\right)$可得

$-\alpha \gamma x+\beta \gamma x-\beta \gamma +4\beta +\gamma -1\ge 0$$-\alpha\gamma{x}+\beta\gamma{x}-\beta\gamma+4\beta+\gamma-1\ge{0}$

注意到$1-\beta +(\beta -\alpha )x>0$$1-\beta+\left(\beta-\alpha\right)x>0$，上述不等式等价于

$\gamma \ge \frac{1-4\beta }{1-\beta +(\beta -\alpha )x}$$\gamma\ge\frac{1-4\beta}{1-\beta+\left(\beta-\alpha\right)x}$.

综合上述分析，有以下四种情况。

情况I：$q_{\ast }(\text{饿},\text{不}\text{吃})>q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,不吃\right)>{q_\ast}\left(饿,吃\right)$且$q_{\ast }(\text{饱},\text{不}\text{吃})\le q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)\le{q_\ast}\left(饱,吃\right)$。这时有$v_{\ast }\left(\text{饿}\right)=q_{\ast }(\text{饿},\text{不}\text{吃})$$v_\ast\left(饿\right)=q_\ast\left(饿,不吃\right)$且$v_{\ast }\left(\text{饱}\right)=q_{\ast }(\text{饱},\text{吃})$$v_\ast\left(饱\right)=q_\ast\left(饱,吃\right)$，以及$x=0$$x=0$且$y=0$$y=0$。相应条件简化为$\gamma <\frac{1-4\alpha }{1-\alpha }$$\gamma<\frac{1-4\alpha}{1-\alpha}$且$\gamma \ge \frac{1-4\beta }{1-\beta }$$\gamma\ge\frac{1-4\beta}{1-\beta}$，最优价值简化为

$q_{\ast }(\text{饿},\text{吃})=\frac{1}{1-\gamma }(-\alpha \gamma +4\alpha +\gamma -3)$$q_\ast\left(饿,吃\right)=\frac{1}{1-\gamma}\left(-\alpha\gamma+4\alpha+\gamma-3\right)$

$v_{\ast }\left(\text{饿}\right)=q_{\ast }(\text{饿},\text{不}\text{吃})=\frac{-2}{1-\gamma }$$v_\ast\left(饿\right)=q_\ast\left(饿,不吃\right)=\frac{-2}{1-\gamma}$

$v_{\ast }\left(\text{饱}\right)=q_{\ast }(\text{饱},\text{吃})=\frac{1}{1-\gamma }$$v_\ast\left(饱\right)=q_\ast\left(饱,吃\right)=\frac{1}{1-\gamma}$

$q_{\ast }(\text{饱},\text{不}\text{吃})=\frac{1}{1-\gamma }(\beta \gamma +4\beta -\gamma +2)$$q_\ast\left(饱,不吃\right)=\frac{1}{1-\gamma}\left(\beta\gamma+4\beta-\gamma+2\right)$

情况II：$q_{\ast }(\text{饿},\text{不}\text{吃})\le q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,不吃\right)\le{q_\ast}\left(饿,吃\right)$且$q_{\ast }(\text{饱},\text{不}\text{吃})\le q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)\le{q_\ast}\left(饱,吃\right)$。这时有$v_{\ast }\left(\text{饿}\right)=q_{\ast }(\text{饿},\text{吃})$$v_\ast\left(饿\right)=q_\ast\left(饿,吃\right)$且$v_{\ast }\left(\text{饱}\right)=q_{\ast }(\text{饱},\text{吃})$$v_\ast\left(饱\right)=q_\ast\left(饱,吃\right)$，以及$x=1$$x=1$且$y=0$$y=0$。相应条件简化为$\gamma \ge \frac{1-4\alpha }{1-\alpha }$$\gamma\ge\frac{1-4\alpha}{1-\alpha}$且$\gamma \ge \frac{1-4\beta }{1-\alpha }$$\gamma\ge\frac{1-4\beta}{1-\alpha}$，最优价值简化为

$q_{\ast }(\text{饿},\text{不}\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{II}}}(-\alpha {\gamma }^2+2\alpha \gamma +{\gamma }^2+\gamma -2)$$q_\ast\left(饿,不吃\right)=\frac{1}{\Delta_\mathrm{II}}\left(-\alpha\gamma^2+2\alpha\gamma+\gamma^2+\gamma-2\right)$

$v_{\ast }\left(\text{饿}\right)=q_{\ast }(\text{饿},\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{II}}}(-3\alpha \gamma +4\alpha +3\gamma -3)$$v_\ast\left(饿\right)=q_\ast\left(饿,吃\right)=\frac{1}{\Delta_\mathrm{II}}\left(-3\alpha\gamma+4\alpha+3\gamma-3\right)$

$q_{\ast }(\text{饱},\text{不}\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{II}}}(-\alpha {\gamma }^2+2\alpha \gamma +4\beta \gamma -4\beta +{\gamma }^2-3\gamma +2)$$q_\ast\left(饱,不吃\right)=\frac{1}{\Delta_\mathrm{II}}\left(-\alpha\gamma^2+2\alpha\gamma+4\beta\gamma-4\beta+\gamma^2-3\gamma+2\right)$

$v_{\ast }\left(\text{饱}\right)=q_{\ast }(\text{饱},\text{吃})=\frac{1}{1-\gamma }$$v_\ast\left(饱\right)=q_\ast\left(饱,吃\right)=\frac{1}{1-\gamma}$

其中

${\mathrm{\Delta }}_{\mathrm{II}}=(1-\gamma )(1-(1-\alpha )\gamma )$$\Delta_\mathrm{II}=\left(1-\gamma\right)\left(1-\left(1-\alpha\right)\gamma\right)$.

情况III：$q_{\ast }(\text{饿},\text{不}\text{吃})>q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,不吃\right)>{q_\ast}\left(饿,吃\right)$且$q_{\ast }(\text{饱},\text{不}\text{吃})>q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)>{q_\ast}\left(饱,吃\right)$。这时有$v_{\ast }\left(\text{饿}\right)=q_{\ast }(\text{饿},\text{不}\text{吃})$$v_\ast\left(饿\right)=q_\ast\left(饿,不吃\right)$且$v_{\ast }\left(\text{饱}\right)=q_{\ast }(\text{饱},\text{不}\text{吃})$$v_\ast\left(饱\right)=q_\ast\left(饱,不吃\right)$，以及$x=0$$x=0$且$y=1$$y=1$。相应条件简化为$\gamma <\frac{1-4\alpha }{1-\beta }$$\gamma<\frac{1-4\alpha}{1-\beta}$且$\gamma <\frac{1-4\beta }{1-\beta }$$\gamma<\frac{1-4\beta}{1-\beta}$，最优价值简化为

$v_{\ast }\left(\text{饿}\right)=q_{\ast }(\text{饿},\text{不}\text{吃})=\frac{-2}{1-\gamma }$$v_\ast\left(饿\right)=q_\ast\left(饿,不吃\right)=\frac{-2}{1-\gamma}$

$q_{\ast }(\text{饿},\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{III}}}(-4\alpha \gamma +4\alpha +\beta {\gamma }^2-3\beta \gamma -{\gamma }^2+4\gamma -3)$$q_\ast\left(饿,吃\right)=\frac{1}{\Delta_\mathrm{III}}\left(-4\alpha\gamma+4\alpha+\beta\gamma^2-3\beta\gamma-\gamma^2+4\gamma-3\right)$

$v_{\ast }\left(\text{饱}\right)=q_{\ast }(\text{饱},\text{不}\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{III}}}(2\beta \gamma -4\beta -2\gamma +2)$$v_\ast\left(饱\right)=q_\ast\left(饱,不吃\right)=\frac{1}{\Delta_\mathrm{III}}\left(2\beta\gamma-4\beta-2\gamma+2\right)$

$q_{\ast }(\text{饱},\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{III}}}(\beta {\gamma }^2-3\beta \gamma -{\gamma }^2+1)$$q_\ast\left(饱,吃\right)=\frac{1}{\Delta_\mathrm{III}}\left(\beta\gamma^2-3\beta\gamma-\gamma^2+1\right)$

其中

${\mathrm{\Delta }}_{\mathrm{III}}=(1-\gamma )(1-(1-\beta )\gamma )$$\Delta_\mathrm{III}=\left(1-\gamma\right)\left(1-\left(1-\beta\right)\gamma\right)$.

情况IV：$q_{\ast }(\text{饿},\text{不}\text{吃})\le q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,不吃\right)\le{q_\ast}\left(饿,吃\right)$且$q_{\ast }(\text{饱},\text{不}\text{吃})>q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)>{q_\ast}\left(饱,吃\right)$。这时有$v_{\ast }\left(\text{饿}\right)=q_{\ast }(\text{饿},\text{吃})$$v_\ast\left(饿\right)=q_\ast\left(饿,吃\right)$且$v_{\ast }\left(\text{饱}\right)=q_{\ast }(\text{饱},\text{不}\text{吃})$$v_\ast\left(饱\right)=q_\ast\left(饱,不吃\right)$，以及$x=1$$x=1$且$y=1$$y=1$。相应条件简化为$\gamma \ge \frac{1-4\alpha }{1-\beta }$$\gamma\ge\frac{1-4\alpha}{1-\beta}$且$\gamma <\frac{1-4\beta }{1-\alpha }$$\gamma<\frac{1-4\beta}{1-\alpha}$，最优价值简化为

$q_{\ast }(\text{饿},\text{不}\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{IV}}}(2\alpha \gamma -\beta {\gamma }^2-2\beta \gamma +{\gamma }^2+\gamma -2)$$q_\ast\left(饿,不吃\right)=\frac{1}{\Delta_\mathrm{IV}}{\left(2\alpha\gamma-\beta\gamma^2-2\beta\gamma+\gamma^2+\gamma-2\right)}$

$v_{\ast }\left(\text{饿}\right)=q_{\ast }(\text{饿},\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{IV}}}(-2\alpha \gamma +4\alpha -3\beta \gamma +3\gamma -3)$$v_\ast\left(饿\right)=q_\ast\left(饿,吃\right)=\frac{1}{\Delta_\mathrm{IV}}{\left(-2\alpha\gamma+4\alpha-3\beta\gamma+3\gamma-3\right)}$

$v_{\ast }\left(\text{饱}\right)=q_{\ast }(\text{饱},\text{不}\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{IV}}}(2\alpha \gamma +\beta \gamma -4\beta -2\gamma +2)$$v_\ast\left(饱\right)=q_\ast\left(饱,不吃\right)=\frac{1}{\Delta_\mathrm{IV}}{\left(2\alpha\gamma+\beta\gamma-4\beta-2\gamma+2\right)}$。

$q_{\ast }(\text{饱},\text{吃})=\frac{1}{{\mathrm{\Delta }}_{\mathrm{IV}}}(\alpha {\gamma }^2+\alpha \gamma -3\beta \gamma -{\gamma }^2+1)$$q_\ast\left(饱,吃\right)=\frac{1}{\Delta_\mathrm{IV}}{\left(\alpha\gamma^2+\alpha\gamma-3\beta\gamma-\gamma^2+1\right)}$

其中

${\mathrm{\Delta }}_{\mathrm{IV}}=(1-\gamma )(1-(1-\alpha -\beta )\gamma )$$\Delta_\mathrm{IV}=\left(1-\gamma\right)\left(1-\left(1-\alpha-\beta\right)\gamma\right)$.

第29页第13行

$v\left(\text{𝘴}\right)\ge r(\text{𝘴},\text{𝘢})+\gamma \sum _{{\text{𝘴}}^{\prime }}p({\text{𝘴}}^{\prime }\mid \text{𝘴},\text{𝘢})v_{\ast }\left({\text{𝘴}}^{\prime }\right),\text{𝘴}\in \mathcal{S}$$v\left(\sfs\right)\ge r\left(\sfs,\sfa\right)+\gamma\sum\limits_{\sfs'} {p\left(\sfs'\mid\sfs,\sfa\right){v_*}\left(\sfs'\right)},\quad\sfs\in\mathcal{S}$

改为

$v\left(\text{𝘴}\right)\ge r(\text{𝘴},\text{𝘢})+\gamma \sum _{{\text{𝘴}}^{\prime }}p({\text{𝘴}}^{\prime }\mid \text{𝘴},\text{𝘢})v\left({\text{𝘴}}^{\prime }\right),\text{𝘴}\in \mathcal{S},\text{𝘢}\in \mathcal{A}$$v\left(\sfs\right)\ge r\left(\sfs,\sfa\right)+\gamma\sum\limits_{\sfs'}{p\left(\sfs'\mid\sfs,\sfa \right){v}\left(\sfs'\right)},\quad\sfs\in\mathcal{S},\sfa\in\mathcal{A}$

作者注：用线性规划求解最优状态价值的详细证明可见《Markov Decision Processes: Discrete Stochastic Dynamic Programming》（Martin Puterman）的第6.9节。证明大致如下：当$c\left(\text{𝘴}\right)$$c\left(\sfs\right)$为Markov决策过程的初始状态分布时，原问题和对偶问题的目标都是Markov决策过程的平均回合奖励，原问题的最优解是最优状态价值。对$c\left(\text{𝘴}\right)$$c\left(\sfs\right)$做灵敏度分析可知，无论$c\left(\text{𝘴}\right)$$c\left(\sfs\right)$取什么分布，对偶问题均有解（这利用到对偶问题的可行域就是带折扣的状态动作对组成的分布所在的空间），所以原问题的最优解都是不变的。也就是说，Markov决策过程的最优状态价值和初始状态分布无关。进一步，可以在原问题中放宽${\sum }_{\text{𝘴}}c\left(\text{𝘴}\right)=1$$\sum\nolimits_{\sfs}c\left(\sfs\right)=1$这个限制，原问题的解依然不变，只是优化目标进行了放缩。

第29页第21行

$v_{\pi }\left(\text{饿}\right)=10$$v_\pi\left(饿\right)=10$，$v_{\pi }\left(\text{饱}\right)=8\frac{3}{4}$$v_\pi\left(饱\right)=8\frac{3}{4}$

改为

$v_{\ast }\left(\text{饿}\right)=\frac{35}{11}$$v_\ast\left(饿\right)=\frac{35}{11}$，$v_{\ast }\left(\text{饱}\right)=5$$v_\ast\left(饱\right)=5$

第29页第23行

$q_{\pi }(\text{饿},\text{不}\text{吃})=7\frac{2}{3}$$q_\pi\left(饿,不吃\right)=7\frac{2}{3}$，$q_{\pi }(\text{饿},\text{吃})=10$$q_\pi\left(饿,吃\right)=10$，$q_{\pi }(\text{饱},\text{不}\text{吃})=6$$q_\pi\left(饱,不吃\right)=6$，$q_{\pi }(\text{饱},\text{吃})=8\frac{3}{4}$$q_\pi\left(饱,吃\right)=8\frac{3}{4}$

改为

$q_{\ast }(\text{饿},\text{不}\text{吃})=\frac{6}{11}$$q_\ast\left(饿,不吃\right)=\frac{6}{11}$，$q_{\ast }(\text{饿},\text{吃})=\frac{35}{11}$$q_\ast\left(饿,吃\right)=\frac{35}{11}$，$q_{\ast }(\text{饱},\text{不}\text{吃})=\frac{21}{11}$$q_\ast\left(饱,不吃\right)=\frac{21}{11}$，$q_{\ast }(\text{饱},\text{吃})=5$$q_\ast\left(饱,吃\right)=5$

第30页第4行~第25行

求解有误，更改如下

情况I：$q_{\ast }(\text{饿},\text{不}\text{吃})>q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,不吃\right)>{q_\ast}\left(饿,吃\right)$且$q_{\ast }(\text{饱},\text{不}\text{吃})\le q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)\le{q_\ast}\left(饱,吃\right)$，即$\gamma <\frac{1-4\alpha }{1-\alpha }$$\gamma<\frac{1-4\alpha}{1-\alpha}$且$\gamma \ge \frac{1-4\beta }{1-\beta }$$\gamma\ge\frac{1-4\beta}{1-\beta}$。这种情况的最优策略是

${\pi }_{\ast }\left(\text{饿}\right)=\text{不}\text{吃}$$\pi_\ast\left(饿\right)=不吃$，${\pi }_{\ast }\left(\text{饱}\right)=\text{吃}$$\pi_\ast\left(饱\right)=吃$.

即饿时不吃，饱时吃。

情况II：$q_{\ast }(\text{饿},\text{不}\text{吃})\le q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,不吃\right)\le{q_\ast}\left(饿,吃\right)$且$q_{\ast }(\text{饱},\text{不}\text{吃})\le q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)\le{q_\ast}\left(饱,吃\right)$，即$\gamma \ge \frac{1-4\alpha }{1-\alpha }$$\gamma\ge\frac{1-4\alpha}{1-\alpha}$且$\gamma \ge \frac{1-4\beta }{1-\alpha }$$\gamma\ge\frac{1-4\beta}{1-\alpha}$。这种情况的最优策略是

${\pi }_{\ast }\left(\text{饿}\right)=\text{吃}$$\pi_\ast\left(饿\right)=吃$，${\pi }_{\ast }\left(\text{饱}\right)=\text{吃}$$\pi_\ast\left(饱\right)=吃$.

即一直吃。

情况III：$q_{\ast }(\text{饿},\text{不}\text{吃})>q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,不吃\right)>{q_\ast}\left(饿,吃\right)$且$q_{\ast }(\text{饱},\text{不}\text{吃})>q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)>{q_\ast}\left(饱,吃\right)$，即$\gamma <\frac{1-4\alpha }{1-\beta }$$\gamma<\frac{1-4\alpha}{1-\beta}$且$\gamma <\frac{1-4\beta }{1-\beta }$$\gamma<\frac{1-4\beta}{1-\beta}$。这种情况的最优策略是

${\pi }_{\ast }\left(\text{饿}\right)=\text{不}\text{吃}$$\pi_\ast\left(饿\right)=不吃$，${\pi }_{\ast }\left(\text{饱}\right)=\text{不}\text{吃}$$\pi_\ast\left(饱\right)=不吃$.

即一直$\text{不}\text{吃}$$不吃$。

情况IV：$q_{\ast }(\text{饿},\text{不}\text{吃})\le q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,不吃\right)\le{q_\ast}\left(饿,吃\right)$且$q_{\ast }(\text{饱},\text{不}\text{吃})>q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)>{q_\ast}\left(饱,吃\right)$，即$\gamma \ge \frac{1-4\alpha }{1-\beta }$$\gamma\ge\frac{1-4\alpha}{1-\beta}$且$\gamma <\frac{1-4\beta }{1-\alpha }$$\gamma<\frac{1-4\beta}{1-\alpha}$。这种情况的最优策略是

${\pi }_{\ast }\left(\text{饿}\right)=\text{吃}$$\pi_\ast\left(饿\right)=吃$，${\pi }_{\ast }\left(\text{饱}\right)=\text{不}\text{吃}$$\pi_\ast\left(饱\right)=不吃$.

即$\text{饿}$$饿$时$\text{吃}$$吃$，$\text{饱}$$饱$时$\text{不}\text{吃}$$不吃$。

对于一组特定的数值，求解则更加直接。例如，当$\alpha =\frac{2}{3}$$\alpha=\frac{2}{3}$，$\beta =\frac{3}{4}$$\beta=\frac{3}{4}$，$\gamma =\frac{4}{5}$$\gamma=\frac{4}{5}$时，2.3.2节已经求得了最优动作价值，此时动作价值满足$q_{\ast }(\text{饿},\text{不}\text{吃})<q_{\ast }(\text{饿},\text{吃})$$q_\ast\left(饿,不吃\right)<q_\ast\left(饿,吃\right)$且$q_{\ast }(\text{饱},\text{不}\text{吃})<q_{\ast }(\text{饱},\text{吃})$$q_\ast\left(饱,不吃\right)<q_\ast\left(饱,吃\right)$。所以，它对应的最优策略为${\pi }_{\ast }\left(\text{饿}\right)={\pi }_{\ast }\left(\text{饱}\right)=\text{吃}$$\pi_\ast\left(饿\right)=\pi_\ast\left(饱\right)=吃$.

第31页第12行

回合奖励为各步奖励之。

改为

回合奖励为各步奖励之和。

第32页代码清单2-4

代码改为

xxxxxxxxxx
def play_once(env, policy):
    total_reward = 0
    state = env.reset()
    while True:
        loc = np.unravel_index(state, env.shape)
        print('状态 = {}, 位置 = {}'.format(state, loc), end=' ')
        action = np.random.choice(env.nA, p=policy[state])
        state, reward, done, _ = env.step(action)
        print('动作 = {}, 奖励 = {}'.format(action, reward))
        total_reward += reward
        if done:
            break
    return total_reward

第32页第10行

Bellmn期望方程为

改为

Bellman期望方程为

第33页第0行

Bellan期望方程：

改为

Bellman期望方程：

第33页代码清单2-6

题注改为

用Bellman方程求解状态价值和动作价值

代码改为

xxxxxxxxxx
def evaluate_bellman(env, policy, gamma=1.):
    a, b = np.eye(env.nS), np.zeros((env.nS))
    for state in range(env.nS - 1):
        for action in range(env.nA):
            pi = policy[state][action]
            for p, next_state, reward, done in env.P[state][action]:
                a[state, next_state] -= (pi * gamma * p)
                b[state] += (pi * reward * p)
    v = np.linalg.solve(a, b)
    q = np.zeros((env.nS, env.nA))
    for state in range(env.nS - 1):
        for action in range(env.nA):
            for p, next_state, reward, done in env.P[state][action]:
                q[state][action] += ((reward + gamma * v[next_state]) * p)
    return v, q

第34页第2行

$v\left(\text{𝘴}\right)-\gamma \sum _{{\text{𝘴}}^{\prime },r}p({\text{𝘴}}^{\prime },r|\text{𝘴},\text{𝘢})v_{\ast }\left({\text{𝘴}}^{\prime }\right)\ge \sum _{{\text{𝘴}}^{\prime },r}rp({\text{𝘴}}^{\prime },r|\text{𝘴},\text{𝘢})\text{𝘴}\in \mathcal{S}$$v\left(\sfs\right)-\gamma\sum\limits_{\sfs',r}{p\left(\sfs',r|\sfs,\sfa\right)v_*\left(\sfs'\right)}\ge\sum\limits_{\sfs',r}{rp\left(\sfs',r|\sfs,\sfa\right)}\qquad\sfs\in\mathcal{S}$

改为

$v\left(\text{𝘴}\right)-\gamma \sum _{{\text{𝘴}}^{\prime },r}p({\text{𝘴}}^{\prime },r|\text{𝘴},\text{𝘢})v\left({\text{𝘴}}^{\prime }\right)\ge \sum _{{\text{𝘴}}^{\prime },r}rp({\text{𝘴}}^{\prime },r|\text{𝘴},\text{𝘢})\text{𝘴}\in \mathcal{S},\text{𝘢}\in \mathcal{A}$$v\left(\sfs\right)-\gamma\sum\limits_{\sfs',r}{p\left(\sfs',r|\sfs,\sfa\right)v\left(\sfs'\right)}\ge\sum\limits_{\sfs',r}{rp\left(\sfs',r|\sfs,\sfa\right)}\qquad\sfs\in\mathcal{S},\sfa\in\mathcal{A}$

第34页第4页

计算这个动态规划问题

改为

计算这个线性规划问题

第35页第25行

可以应用下列线性规划求解最优动作价值

改为

可以应用下列线性规划求解最优状态价值

第36页第2行

$v\left(\text{𝘴}\right)\ge r(\text{𝘴},\text{𝘢})+\gamma \sum _{{\text{𝘴}}^{\prime }}p({\text{𝘴}}^{\prime }|\text{𝘴},\text{𝘢})v_{\ast }\left({\text{𝘴}}^{\prime }\right),\text{𝘴}\in \mathcal{S}$$v\left(\sfs\right)\ge r\left(\sfs,\sfa\right)+\gamma\sum\limits_{\sfs'}{p\left(\sfs'|\sfs,\sfa\right){v_*}\left(\sfs'\right)} ,\quad \sfs\in{\cal S}$

改为

$v\left(\text{𝘴}\right)\ge r(\text{𝘴},\text{𝘢})+\gamma \sum _{{\text{𝘴}}^{\prime }}p({\text{𝘴}}^{\prime }|\text{𝘴},\text{𝘢})v\left({\text{𝘴}}^{\prime }\right),\text{𝘴}\in \mathcal{S},\text{𝘢}\in \mathcal{A}$$v\left(\sfs\right)\ge r\left(\sfs,\sfa\right)+\gamma \sum\limits_{\sfs'}{p\left(\sfs'|\sfs,\sfa\right){v}\left(\sfs'\right)},\quad\sfs\in\mathcal{S},\sfa\in \mathcal{A}$

第38页第11行和第14行

这里的$d$$d$都是指$d_{\mathrm{\infty }}$$d_\infty$（共3处）。

第38页第21行

$d(t\left({\text{𝘹}}^{\prime }\right),t\left({\text{𝘹}}^{″}\right))<d({\text{𝘹}}^{\prime },{\text{𝘹}}^{″})$$d\left(t\left(\sfx'\right),t\left(\sfx''\right)\right)<d\left(\sfx',\sfx''\right)$

改为

$d(t\left({\text{𝘹}}^{\prime }\right),t\left({\text{𝘹}}^{″}\right))<\gamma d({\text{𝘹}}^{\prime },{\text{𝘹}}^{″})$$d\left(t\left(\sfx'\right),t\left(\sfx''\right)\right)<\gamma d\left(\sfx',\sfx''\right)$

第38页第24~25行

用动作价值表示动作价值的形式

改为

用状态价值表示状态价值的形式

第39页第0行

$v_{\ast }\left({\text{𝘴}}^{\prime }\right)$$v_{*}\left(\sfs'\right)$

改为

$v\left({\text{𝘴}}^{\prime }\right)$$v\left(\sfs'\right)$

第39页第5行~第7行

$|t_{\pi }\left(v^{\prime }\right)\left(\text{𝘴}\right)-t_{\pi }\left(v^{″}\right)\left(\text{𝘴}\right)|\le \gamma \sum _{\text{𝘢}}\pi (\text{𝘢}|\text{𝘴})\sum _{{\text{𝘴}}^{\prime }}p({\text{𝘴}}^{\prime }|\text{𝘴},\text{𝘢})\underset{{\text{𝘴}}^{\prime }}{max}|v^{\prime }\left({\text{𝘴}}^{\prime }\right)-v^{″}\left({\text{𝘴}}^{\prime }\right)|=\gamma d(v^{\prime },v^{″})$$\left|t_\pi\left(v'\right)\left(\sfs\right)-t_\pi\left(v''\right)\left(\sfs\right)\right|\le\gamma\sum\limits_\sfa{\pi\left(\sfa|\sfs\right)\sum\limits_{\sfs'}{p\left(\sfs'|\sfs,\sfa\right)}}\mathop{\max}\limits_{\sfs'} \left|v'\left(\sfs'\right)-v''\left(\sfs'\right)\right|=\gamma d\left(v',v''\right)$

考虑到$v^{\prime },v^{″}$$v',v''$是任取的，所以有

$d(t_{\pi }\left(v^{\prime }\right),t_{\pi }\left(v^{″}\right))\le \gamma d(v^{\prime },v^{″})$$d\left(t_\pi\left(v'\right),t_\pi\left(v''\right)\right)\le\gamma d\left(v',v''\right)$

改为

$|t_{\pi }\left(v^{\prime }\right)\left(\text{𝘴}\right)-t_{\pi }\left(v^{″}\right)\left(\text{𝘴}\right)|\le \gamma \sum _{\text{𝘢}}\pi (\text{𝘢}|\text{𝘴})\sum _{{\text{𝘴}}^{\prime }}p({\text{𝘴}}^{\prime }|\text{𝘴},\text{𝘢})\underset{{\text{𝘴}}^{\prime }}{max}|v^{\prime }\left({\text{𝘴}}^{\prime }\right)-v^{″}\left({\text{𝘴}}^{\prime }\right)|=\gamma d_{\mathrm{\infty }}(v^{\prime },v^{″})$$\left|t_\pi\left(v'\right)\left(\sfs\right)-t_\pi\left(v''\right)\left(\sfs\right)\right|\le\gamma\sum\limits_\sfa{\pi\left(\sfa|\sfs\right)\sum\limits_{\sfs'}{p\left(\sfs'|\sfs,\sfa\right)}}\mathop{\max}\limits_{\sfs'}\left|v'\left(\sfs'\right)-v''\left(\sfs'\right)\right|=\gamma d_\infty\left(v',v''\right)$

考虑到$\text{𝘴}$$\sfs$是任取的，所以有

$d_{\mathrm{\infty }}(t_{\pi }\left(v^{\prime }\right),t_{\pi }\left(v^{″}\right))\le \gamma d_{\mathrm{\infty }}(v^{\prime },v^{″})$$d_\infty\left(t_\pi\left(v'\right),t_\pi\left(v''\right)\right)\le\gamma d_\infty\left(v',v''\right)$

第39页第12行~第13行

$\le \underset{\text{𝘢}}{max}|f^{\prime }\left({\text{𝘢}}^{\prime }\right)-f^{″}\left({\text{𝘢}}^{\prime }\right)|$$\le\max_\sfa\left|f'\left(\sfa'\right)-f''\left(\sfa'\right)\right|$

改为（共2处）

$\le \underset{\text{𝘢}}{max}|f^{\prime }\left(\text{𝘢}\right)-f^{″}\left(\text{𝘢}\right)|$$\le\max_\sfa\left|f'\left(\sfa\right)-f''\left(\sfa\right)\right|$

第39页第15行~第19行

$t_{\ast }\left(v^{\prime }\right)\left(\text{𝘴}\right)-t_{\ast }\left(v^{″}\right)\left(\text{𝘴}\right)$${t_*}\left(v'\right)\left(\sfs\right)-{t_*}\left(v''\right)\left(\sfs\right)$

$=\underset{\text{𝘢}\in \mathcal{A}}{max}[r(\text{𝘴},\text{𝘢})+\gamma \sum _{{\text{𝘴}}^{\prime }\in \mathcal{S}}p({\text{𝘴}}^{\prime }\mid \text{𝘴},\text{𝘢})v^{\prime }\left({\text{𝘴}}^{\prime }\right)]-\underset{\text{𝘢}\in \mathcal{A}}{max}[r(\text{𝘴},\text{𝘢})+\gamma \sum _{{\text{𝘴}}^{\prime }\in \mathcal{S}}p({\text{𝘴}}^{\prime }\mid \text{𝘴},\text{𝘢})v^{″}\left({\text{𝘴}}^{\prime }\right)]$$=\mathop{\max}\limits_{\sfa\in\mathcal{A}}\left[r\left(\sfs,\sfa\right)+\gamma\sum\limits_{\sfs'\in{\mathcal{S}}}{p\left(\sfs'\mid\sfs,\sfa\right)v'\left(\sfs'\right)}\right]-\mathop{\max}\limits_{\sfa\in{\mathcal{A}}}\left[r\left(\sfs,\sfa\right)+\gamma\sum\limits_{\sfs'\in{\mathcal{S}}}p\left(\sfs'\mid\sfs,\sfa\right)v''\left(\sfs'\right)\right]$

$\le \underset{\text{𝘢}\in \mathcal{A}}{max}\left|\gamma \sum _{{\text{𝘴}}^{\prime }\in \mathcal{S}}p({\text{𝘴}}^{\prime }\mid \text{𝘴},\text{𝘢})(v^{\prime }\left({\text{𝘴}}^{\prime }\right)-v^{″}\left({\text{𝘴}}^{\prime }\right))\right|$$\le\mathop{\max}\limits_{\sfa\in\mathcal{A}}\left|\gamma\sum\limits_{\sfs'\in{\mathcal{S}}}{p\left(\sfs'\mid\sfs,\sfa\right)\left(v'\left(\sfs'\right)-v''\left(\sfs'\right)\right)}\right|$

$\le \gamma |v^{\prime }\left({\text{𝘴}}^{\prime }\right)-v^{″}\left({\text{𝘴}}^{\prime }\right)|,$$\le\gamma\left|v'\left(\sfs'\right)-v''\left(\sfs'\right)\right|,$

进而易知$|t_{\ast }\left(v^{\prime }\right)\left(\text{𝘴}\right)-t_{\ast }\left(v^{″}\right)\left(\text{𝘴}\right)|\le \gamma |v^{\prime }\left({\text{𝘴}}^{\prime }\right)-v^{″}\left({\text{𝘴}}^{\prime }\right)|\le \gamma d(v^{\prime },v^{″})$$\left|t_*\left(v'\right)\left(\sfs\right)-{t_*}\left(v''\right)\left(\sfs\right)\right| \le\gamma\left|v'\left(\sfs'\right)- v''\left(\sfs'\right)\right|\le\gamma d\left(v',v''\right)$，所以$t_{\ast }$$t_*$是压缩映射。

改为

$t_{\ast }\left(v^{\prime }\right)\left(\text{𝘴}\right)-t_{\ast }\left(v^{″}\right)\left(\text{𝘴}\right)$${t_*}\left(v'\right)\left(\sfs\right)-{t_*}\left(v''\right)\left(\sfs\right)$

$=\underset{\text{𝘢}\in \mathcal{A}}{max}[r(\text{𝘴},\text{𝘢})+\gamma \sum _{{\text{𝘴}}^{\prime }\in \mathcal{S}}p({\text{𝘴}}^{\prime }\mid \text{𝘴},\text{𝘢})v^{\prime }\left({\text{𝘴}}^{\prime }\right)]-\underset{\text{𝘢}\in \mathcal{A}}{max}[r(\text{𝘴},\text{𝘢})+\gamma \sum _{{\text{𝘴}}^{\prime }\in \mathcal{S}}p({\text{𝘴}}^{\prime }\mid \text{𝘴},\text{𝘢})v^{″}\left({\text{𝘴}}^{\prime }\right)]$$=\mathop{\max}\limits_{\sfa\in\mathcal{A}}\left[r\left(\sfs,\sfa\right)+\gamma\sum\limits_{\sfs'\in\mathcal{S}}{p\left(\sfs'\mid\sfs,\sfa\right)v'\left(\sfs'\right)}\right]-\mathop{\max}\limits_{\sfa\in\mathcal{A}}\left[r\left(\sfs,\sfa\right)+\gamma\sum\limits_{\sfs'\in\mathcal{S}}p\left(\sfs'\mid\sfs,\sfa\right)v''\left(\sfs'\right)\right]$

$\le \underset{\text{𝘢}\in \mathcal{A}}{max}\left|\gamma \sum _{{\text{𝘴}}^{\prime }\in \mathcal{S}}p({\text{𝘴}}^{\prime }\mid \text{𝘴},\text{𝘢})(v^{\prime }\left({\text{𝘴}}^{\prime }\right)-v^{″}\left({\text{𝘴}}^{\prime }\right))\right|$$\le\mathop{\max}\limits_{\sfa\in\mathcal{A}}\left|{\gamma\sum\limits_{\sfs'\in{\mathcal{S}}}{p\left(\sfs'\mid\sfs,\sfa\right)\left(v'\left(\sfs'\right)-v''\left(\sfs'\right)\right)}}\right|$

$\le \gamma \underset{\text{𝘢}\in \mathcal{A}}{max}\left|\sum _{{\text{𝘴}}^{\prime }\in \mathcal{S}}p({\text{𝘴}}^{\prime }|\text{𝘴},\text{𝘢})\right|\underset{{\text{𝘴}}^{\prime }\in \mathcal{S}}{max}|v^{\prime }\left({\text{𝘴}}^{\prime }\right)-v^{″}\left({\text{𝘴}}^{\prime }\right)|$$\le\gamma\mathop{\max}\limits_{\sfa\in\mathcal{A}}\left|{\sum\limits_{\sfs'\in\mathcal{S}}{p\left(\sfs'|\sfs,\sfa\right)}}\right|\mathop{\max}\limits_{\sfs'\in\mathcal{S}}\left|v'\left(\sfs'\right)-v''\left(\sfs'\right)\right|$

$\le \gamma d_{\mathrm{\infty }}(v^{\prime },v^{″})$$\le\gamma{d}_\infty\left(v',v''\right)$

进而易知$\left|t_{\ast }\left(v^{\prime }\right)\left(\text{𝘴}\right)-t_{\ast }\left(v^{″}\right)\left(\text{𝘴}\right)\right|\le \gamma d_{\mathrm{\infty }}(v^{\prime },v^{″})$$\left|{{t_*}\left(v'\right)\left(\sfs\right)-{t_*}\left(v''\right)\left(\sfs\right)}\right|\le\gamma d_\infty\left(v',v''\right)$。所以$t_{\ast }$$t_*$是压缩映射。

第39页第21行

如果$x\in \mathcal{X}$$x\in\mathcal{X}$使得$t\left(x\right)=x$$t\left(x\right)=x$，则称$x$$x$是

改为

如果$\text{𝘹}\in \mathcal{X}$$\sfx\in\mathcal{X}$使得$t\left(\text{𝘹}\right)=\text{𝘹}$$t\left(\sfx\right)=\sfx$，则称$\text{𝘹}$$\sfx$是

第39页第22行

（fix point）

改为

（fixed point）

第42页第7行

对于两个确定性的策略

改为

对于策略

第42页第8行

$v_{\pi }\left(\text{𝘴}\right)\le q_{\pi }(\text{𝘴},{\pi }^{\prime }\left(\text{𝘴}\right)),\text{𝘴}\in \mathcal{S}$$v_\pi\left(\sfs\right)\le{q}_\pi\left(\sfs,\pi'\left( \sfs\right)\right),\quad\sfs\in\mathcal{S}$

改为

$v_{\pi }\left(\text{𝘴}\right)\le \sum _{\text{𝘢}}{\pi }^{\prime }(\text{𝘢}|\text{𝘴})q_{\pi }(\text{𝘴},\text{𝘢}),\text{𝘴}\in \mathcal{S}$$v_\pi\left(\sfs\right)\le\sum\limits_\sfa{\pi'\left(\sfa|\sfs\right)q_\pi\left(\sfs,\sfa\right)},\quad\sfs\in\mathcal{S}$

作者注：这里的修改使得策略改进定理不仅可以用于确定性策略的改进。这样修改后，相关的结论就可以用于后续章节中（如第60~61页的4.1.3节）。

第42页第12行~第43页第11行

证明更新如下

不等式（3-1）等价于

$v_{\pi }\left(\text{𝘴}\right)={\text{E}}_{{\pi }^{\prime }}[v_{\pi }\left({\text{𝘚}}_t\right)\mid {\text{𝘚}}_t=\text{𝘴}]\le {\text{E}}_{{\pi }^{\prime }}[q_{\pi }({\text{𝘚}}_t,{\text{𝘈}}_t)\mid {\text{𝘚}}_t=\text{𝘴}],\text{𝘴}\in \mathcal{S}$$v_\pi\left(\sfs\right)=\E_{\pi'}\left[v_\pi\left(\sfS_t\right)\mid\sfS_t=\sfs\right]\le\E_{\pi'}\left[q_\pi\left(\sfS_t,\sfA_t\right)\mid\sfS_t=\sfs\right],\quad\sfs\in\mathcal{S}$

其中的期望是针对用策略${\pi }^{\prime }$$\pi'$生成的轨迹中，选取${\text{𝘚}}_t=\text{𝘴}$$\sfS_t=\sfs$的那些轨迹而言的。进而有

${\text{E}}_{{\pi }^{\prime }}[v_{\pi }\left({\text{𝘚}}_{t+\tau }\right)|{\text{𝘚}}_t=\text{𝘴}]={\text{E}}_{{\pi }^{\prime }}[{\text{E}}_{{\pi }^{\prime }}[v_{\pi }\left({\text{𝘚}}_{t+\tau }\right)|{\text{𝘚}}_{t+\tau }]|{\text{𝘚}}_t=\text{𝘴}]\le {\text{E}}_{{\pi }^{\prime }}[{\text{E}}_{{\pi }^{\prime }}[q_{\pi }({\text{𝘚}}_{t+\tau },{\text{𝘈}}_{t+\tau })|{\text{𝘚}}_{t+\tau }]|{\text{𝘚}}_t=\text{𝘴}]={\text{E}}_{{\pi }^{\prime }}[q_{\pi }({\text{𝘚}}_{t+\tau },{\text{𝘈}}_{t+\tau })|{\text{𝘚}}_t=\text{𝘴}],\text{𝘴}\in \mathcal{S},\tau =0,1,2,\dots$$\E_{\pi'}\left[v_\pi\left(\sfS_{t+\tau}\right)|\sfS_t=\sfs\right]=\E_{\pi'}\left[\E_{\pi'}\left[v_\pi\left(\sfS_{t+\tau}\right)|\sfS_{t+\tau}\right]|\sfS_t=\sfs\right]\le\E_{\pi'}\left[\E_{\pi'}\left[q_\pi\left(\sfS_{t+\tau},\sfA_{t+\tau}\right)|\sfS_{t+\tau}\right]|\sfS_t=\sfs\right]=\E_{\pi'}\left[q_\pi\left(\sfS_{t+\tau},\sfA_{t+\tau}\right)|\sfS_t=\sfs\right],\quad\sfs\in\mathcal{S},\tau=0,1,2,\ldots$

考虑到

${\text{E}}_{{\pi }^{\prime }}[q_{\pi }({\text{𝘚}}_{t+\tau },{\text{𝘈}}_{t+\tau })|{\text{𝘚}}_t=\text{𝘴}]={\text{E}}_{{\pi }^{\prime }}[R_{t+\tau +1}+\gamma v_{\pi }\left({\text{𝘚}}_{t+\tau +1}\right)|{\text{𝘚}}_t=\text{𝘴}],\text{𝘴}\in \mathcal{S},\tau =0,1,2,\dots$$\E_{\pi'}\left[q_\pi\left(\sfS_{t+\tau},\sfA_{t+\tau}\right)|\sfS_t=\sfs\right]=\E_{\pi'}\left[R_{t+\tau+1}+\gamma{v_\pi}\left(\sfS_{t+\tau+1}\right)|\sfS_t=\sfs\right],\quad\sfs\in\mathcal{S},\tau=0,1,2,\ldots$

所以

${\text{E}}_{{\pi }^{\prime }}[v_{\pi }\left({\text{𝘚}}_{t+\tau }\right)|{\text{𝘚}}_t=\text{𝘴}]\le {\text{E}}_{{\pi }^{\prime }}[R_{t+\tau +1}+\gamma v_{\pi }\left({\text{𝘚}}_{t+\tau +1}\right)|{\text{𝘚}}_t=\text{𝘴}],\text{𝘴}\in \mathcal{S},\tau =0,1,2,\dots$$\E_{\pi'}\left[v_\pi\left(\sfS_{t+\tau}\right)|\sfS_t=\sfs\right]\le\E_{\pi'}\left[R_{t+\tau+1}+\gamma{v_\pi}\left(\sfS_{t+\tau+1}\right)|\sfS_t=\sfs\right],\quad\sfs\in\mathcal{S},\tau=0,1,2,\ldots$

进而有

$v_{\pi }\left(\text{𝘴}\right)={\text{E}}_{{\pi }^{\prime }}[v_{\pi }\left({\text{𝘚}}_t\right)|{\text{𝘚}}_t=\text{𝘴}]$$v_\pi\left(\sfs\right)=\E_{\pi'}\left[v_\pi\left(\sfS_t\right)|\sfS_t=\sfs\right]$

$\le {\text{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma v_{\pi }\left({\text{𝘚}}_{t+1}\right)|{\text{𝘚}}_t=\text{𝘴}]$$\le\E_{\pi'}\left[R_{t+1}+\gamma{v_\pi}\left(\sfS_{t+1}\right)|\sfS_t=\sfs\right]$

$\le {\text{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma {\text{E}}_{{\pi }^{\prime }}[R_{t+2}+\gamma v_{\pi }\left({\text{𝘚}}_{t+2}\right)|{\text{𝘚}}_t=\text{𝘴}]|{\text{𝘚}}_t=\text{𝘴}]$$\le\E_{\pi'}\left[R_{t+1}+\gamma\E_{\pi'}\left[R_{t+2}+\gamma{v_\pi}\left(\sfS_{t+2}\right)|\sfS_t=\sfs\right]|\sfS_t=\sfs\right]$

$={\text{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{v}_{\pi }\left({\text{𝘚}}_{t+2}\right)\mid {\text{𝘚}}_t=\text{𝘴}]$$=\E_{\pi'}\left[R_{t+1}+\gamma{R}_{t+2}+\gamma^2v_\pi\left(\sfS_{t+2}\right)\mid\sfS_t=\sfs\right]$

$\le {\text{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+{\gamma }^3{v}_{\pi }\left({\text{𝘚}}_{t+3}\right)|{\text{𝘚}}_t=\text{𝘴}]$$\le\E_{\pi'}\left[R_{t+1}+\gamma{R}_{t+2}+\gamma^2R_{t+3}+\gamma^3v_\pi\left(\sfS_{t+3}\right)|\sfS_t=\sfs\right]$

$\cdots$$\cdots$

$\le {\text{E}}_{{\pi }^{\prime }}[R_{t+1}+\gamma R_{t+2}+{\gamma }^2{R}_{t+3}+{\gamma }^3{R}_{t+4}+\cdots |{\text{𝘚}}_t=\text{𝘴}]$$\le\E_{\pi'}\left[R_{t+1}+\gamma{R}_{t+2}+\gamma^2{R_{t+3}}+\gamma^3R_{t+4}+\cdots|{\sfS_t}=\sfs\right]$

$={\text{E}}_{{\pi }^{\prime }}[G_t|{\text{𝘚}}_t=\text{𝘴}]$$=\E_{\pi'}\left[G_t|\sfS_t=\sfs\right]$

$=v_{{\pi }^{\prime }}\left(\text{𝘴}\right)$$=v_{\pi'}\left(\sfs\right)$

第48页3.5节

作者注：Gym 0.19版本将FrozenLake-v0改为FrozenLake-v1。

作者注：Gym 0.22删除了DiscreteEnv类，所以env.unwrapped.nS和env.unwrapped.nA不再可用。建议用env.observation_space.n替换env.unwrapped.nA，用env.action_space.n替换env.unwrapped.nA。

第49页第12行

这个函数使用theta作

改为

这个函数使用tolerant作

第52页代码清单3-10

代码改为

xxxxxxxxxx
policy_vi, v_vi = iterate_value(env)
print('状态价值函数 =')
print(v_vi.reshape(4, 4))
print('最优策略 =')
print(np.argmax(policy_vi, axis=1).reshape(4, 4))
episode_rewards = [play_policy(env, policy_vi) for _ in range(100)]
print("价值迭代 平均奖励：{}".format(np.mean(episode_rewards)))

第54页第15行

$\frac{1}{c}{\sum }_{i=1}^c{g}_c$$\frac{1}{c}\sum\nolimits_{i=1}^c{g_c}$

改为

$\frac{1}{c}{\sum }_{i=1}^c{g}_i$$\frac{1}{c}\sum\nolimits_{i=1}^c{g_i}$

第56页第10行

同时满足$\{{\alpha }_k:k=1,2,\dots \}$$\left\{\alpha_k:k=1,2,\ldots\right\}$

改为

$\{{\alpha }_k:k=1,2,\dots \}$$\left\{\alpha_k:k=1,2,\ldots\right\}$同时满足

第57页算法4-2第1步，第58页算法4-4第1步（共2处）

初始化动作价值估计

改为

初始化状态价值估计

第58页最后一行

态${\text{𝘴}}_{\text{起}\text{始}}$$\sfs_{起始}$

改为

态${\text{𝘴}}_{\text{开}\text{始}}$$\sfs_{开始}$

第60页第17~20行

证明：对于某个$\epsilon$$\varepsilon$柔性策略，用

$\begin{array}{c}\pi (\text{𝘢}\mid \text{𝘴})=\{\begin{array}{cc}1-\epsilon +\frac{\epsilon }{\left|\mathcal{A}\right|},& \text{𝘢}=\arg \underset{{\text{𝘢}}^{\prime }}{max}{q}_{\pi }(\text{𝘴},{\text{𝘢}}^{\prime })\\ \frac{\epsilon }{\left|\mathcal{A}\right|},& \text{𝘢}\ne \arg {\underset{\text{𝘢}}{max}}^{\prime }{q}_{\pi }(\text{𝘴},{\text{𝘢}}^{\prime })\end{array}\end{array}$$\pi\left(\sfa\mid\sfs\right)=\left\{\begin{matrix}1-\varepsilon+\frac{\varepsilon}{\left|\mathcal{A}\right|},&\sfa=\arg\max_{\sfa'}q_\pi\left(\sfs,\sfa'\right)\\ \frac{\varepsilon}{\left|\mathcal{A}\right|},&\sfa\ne\arg\max_\sfa'q_\pi\left(\sfs,\sfa'\right)\end{matrix}\right.$

改进后的策略为${\pi }^{\prime }$$\pi'$，则有

$q_{\pi }(\text{𝘴},\text{𝘢})=\sum _{\text{𝘢}}{\pi }^{\prime }(\text{𝘢}|\text{𝘴})q_{\pi }(\text{𝘴},\text{𝘢})=\frac{\epsilon }{\left|\mathcal{A}\left(\text{𝘴}\right)\right|}\sum _{\text{𝘢}}{q}_{\pi }(\text{𝘴},\text{𝘢})+(1-\epsilon )\underset{\text{𝘢}}{max}{q}_{\pi }(\text{𝘴},\text{𝘢})$${q_\pi}\left(\sfs,\sfa\right)=\sum\limits_\sfa{\pi'\left(\sfa|\sfs\right){q_\pi}\left(\sfs,\sfa\right)}=\frac{\varepsilon}{{\left|{{\mathcal{A}}\left(\sfs\right)}\right|}}\sum\limits_\sfa{{q_\pi }\left(\sfs,\sfa\right)}+\left(1-\varepsilon\right)\mathop{\max }\limits_\sfa{q_\pi}\left(\sfs,\sfa\right)$

改为

证明：考虑在$\epsilon$$\varepsilon$柔性策略$\pi$$\pi$上进行如下方式改进得到的$\epsilon$$\varepsilon$柔性策略${\pi }^{\prime }$$\pi'$：

$\begin{array}{c}{\pi }^{\prime }(\text{𝘢}|\text{𝘴})=\{\begin{array}{cc}1-\epsilon +\frac{\epsilon }{\left|\mathcal{A}\right|},& \text{𝘢}=\arg \underset{{\text{𝘢}}^{\prime }}{max}{q}_{\pi }(\text{𝘴},{\text{𝘢}}^{\prime })\\ \frac{\epsilon }{\left|\mathcal{A}\right|},& \text{𝘢}\ne \arg {\underset{\text{𝘢}}{max}}^{\prime }{q}_{\pi }(\text{𝘴},{\text{𝘢}}^{\prime })\end{array}\end{array}$$\pi'\left(\sfa|\sfs\right)=\left\{\begin{matrix}1-\varepsilon+\frac{\varepsilon}{\left|\mathcal{A}\right|},&\sfa=\arg\max_{\sfa'}q_\pi\left(\sfs,\sfa'\right)\\ \frac{\varepsilon}{\left|\mathcal{A}\right|},&\sfa\ne\arg\max_\sfa'q_\pi\left(\sfs,\sfa'\right)\end{matrix}\right.$

在3.2.2节策略改进定理的证明过程中，我们已经证明，只要

$\sum _{\text{𝘢}}{\pi }^{\prime }(\text{𝘢}|\text{𝘴})q_{\pi }(\text{𝘴},\text{𝘢})\ge v_{\pi }\left(\text{𝘴}\right),\text{𝘴}\in \mathcal{S}$$\sum\limits_\sfa{\pi'\left(\sfa|\sfs\right){q_\pi}\left(\sfs,\sfa\right)}\ge{v_\pi}\left(\sfs\right),\quad\sfs\in\mathcal{S}$

就有$\pi \le {\pi }^{\prime }$$\pi\le\pi'$。接下来验证上述不等式。考虑到

$\sum _{\text{𝘢}}{\pi }^{\prime }(\text{𝘢}\mid \text{𝘴})q_{\pi }(\text{𝘴},\text{𝘢})=\frac{\epsilon }{\left|\mathcal{A}\left(\text{𝘴}\right)\right|}\sum _{\text{𝘢}}{q}_{\pi }(\text{𝘴},\text{𝘢})+(1-\epsilon )\underset{\text{𝘢}}{max}{q}_{\pi }(\text{𝘴},\text{𝘢})$$\sum\limits_\sfa\pi'\left(\sfa\mid\sfs\right)q_\pi\left(\sfs,\sfa\right)=\frac{\varepsilon}{\left|\mathcal{A}\left(\sfs\right)\right|}\sum\limits_\sfa{q_\pi}\left(\sfs,\sfa\right)+\left(1-\varepsilon\right)\max\limits_\sfa{q_\pi}\left(\sfs,\sfa\right)$