4、练习目标函数的另一种形式回答 首先分析和理解这样一个metric。它从S0∼dS_0\sim dS0​∼d开始然后A0,R1,S1,A1,R2,S2,......A_0,R_1,S_1,A_1,R_2,S_2,......A0​,R1​,S1​,A1​,R2​,S2​,......At∼π(St)A_t\sim\pi(S_t)At​∼π(St​)并且Rt1,St1∼p(Rt1∣St,At)R_{t1},S_{t1}\sim p(R_{t1}|S_t,A_t)Rt1​,St1​∼p(Rt1​∣St​,At​)p(St1∣St,At)p(S_{t1}|S_t,A_t)p(St1​∣St​,At​)然后我们知道这个metric和average value相同因为J(θ)E[∑t0∞γtRt1]∑s∈Sd(s)E[∑t0∞γtRt1∣S0s]∑s∈Sd(s)vπ(s)vˉπ\begin{aligned} J(\theta) \mathbb{E}\left[\sum_{t0}^\infty\gamma^tR_{t1}\right] \sum_{s\in\mathcal{S}}d(s)\mathbb{E}\left[\sum_{t0}^\infty\gamma^tR_{t1}|S_0s\right]\\ \sum_{s\in\mathcal{S}}d(s)v_{\pi}(s) \\ \bar{v}_{\pi} \end{aligned}J(θ)E[t0∑∞​γtRt1​]​s∈S∑​d(s)E[t0∑∞​γtRt1​∣S0​s]s∈S∑​d(s)vπ​(s)vˉπ​​