a survey for RL
• A finite set of states St summarizing the information the agent senses from the environment at every time step t ∈ {1, ..., T}.
• A set of actions At which the agent can perform at each time step t ∈ {1, ..., T} to interact with the environment.
• A set of transition probabilities between subsequent states which render the environment stochastic. Note: the probabilities are usually not explicitly modeled but the result of the stochastic nature of the financial asset’s price process.
• A reward (or return) function Rt which provides a numerical feedback value rt to the agent in response to its action At−1 = at−1 in state St−1 = st−1.
• A policy π which maps states to concrete actions to be carried out by the agent. The policy can hence be understood as the agent’s rules for how to choose actions.
• A value function V which maps states to the total (discounted) reward the agent can expect from a given state until the end of the episode (trading period) under policy π.
Given the above framework, the decision problem is formalized as finding the optimal policy π = π ∗ , i.e., the mapping from states to actions, corresponding to the optimal value function V ∗ - see also Dempster et al. (2001); Dempster and Romahi (2002):
V ∗ (st) = max at E[Rt+1 + γV ∗ (St+1)|St = st ].(1)
Hereby, E denotes the expectation operator, γ the discount factor, and Rt+1 the expected immediate reward for carrying out action At = at in state St = st . Further, St+1 denotes the next state of the agent. The value function can hence be understood as a mapping from states to discounted future rewards which the agent seeks to maximize with its actions.
To solve this optimization problem, the Q-Learning algorithm (Watkins, 1989) can be applied, extending the above equation to the level of state-action tuples:
Q ∗ (st , at) = E[Rt+1 + γ max at+1 Q ∗ (St+1, at+1)|St = st , At = at ].(2)
Hereby, the Q-value Q∗ (st , at) equals to the immediate reward for carrying out action At = at in state St = st plus the discounted future reward from carrying on in the best way possible.
The optimal policy π ∗ (the mapping from states to actions) then simply becomes:
π ∗ (st) = max at Q ∗ (st , at) .(3)
i.e., in every state St = st , choose the action At = at that yields the highest Q-value. To approximate the Q-function during (online) learning, an iterative optimization is carried out with α denoting the learning rate - see also Sutton and Barto (1998) for further details:
Q ∗ (st , at) ← (1 − α) Q ∗ (st , at) + α (rt+1 + γ max at+1 Q ∗ (st+1, at+1) ) . (4)
a survey for RL的更多相关文章
- (转)Applications of Reinforcement Learning in Real World
Applications of Reinforcement Learning in Real World 2018-08-05 18:58:04 This blog is copied from: h ...
- 论文笔记系列-Neural Network Search :A Survey
论文笔记系列-Neural Network Search :A Survey 论文 笔记 NAS automl survey review reinforcement learning Bayesia ...
- (zhuan) 一些RL的文献(及笔记)
一些RL的文献(及笔记) copy from: https://zhuanlan.zhihu.com/p/25770890 Introductions Introduction to reinfor ...
- A Survey of Visual Attention Mechanisms in Deep Learning
A Survey of Visual Attention Mechanisms in Deep Learning 2019-12-11 15:51:59 Source: Deep Learning o ...
- Generalizing from a Few Examples: A Survey on Few-Shot Learning 小样本学习最新综述 | 三大数据增强方法
目录 原文链接:小样本学习与智能前沿 01 Transforming Samples from Dtrain 02 Transforming Samples from a Weakly Labeled ...
- 知识图谱顶刊综述 - (2021年4月) A Survey on Knowledge Graphs: Representation, Acquisition, and Applications
知识图谱综述(2021.4) 论文地址:A Survey on Knowledge Graphs: Representation, Acquisition, and Applications 目录 知 ...
- SharePoint 2010 Survey的Export to Spreadsheet功能怎么不见了?
背景信息: 最近用户报了一个问题,说他创建的Survey里将结果导出成Excel文件(Export to spreadsheet)的按钮不见了. 原因排查: 正常情况下,这个功能只存在于SharePo ...
- 中间值为什么为l+(r-l)/2,而不是(l+r)/2
二分法的算法中,我们看到一些代码里取中间值: MID=l+(r-l)/2; 为什么是这个呢?不就是(l+r)/2吗?为什么要多此一举呢? 其实还是有不一样的,看看他们的区别吧: l,r是指针的时候只能 ...
- SharePoint Tricks - Survey
1. SharePoint 2010中,在Survey的问题框中输入HTML代码可以用于插入图片或者链接,具体方法为: 1.1 在问题框中输入html, 1.2 在New Form和Edit Form ...
随机推荐
- c# 库间关系
- ps色彩混合
http://tieba.baidu.com/p/2032536851?pn=1 HSB 这是一种颜色的表示方法:其中"H"表示色相,"S"表示饱和度,&quo ...
- 洛谷P3200 [HNOI2009]有趣的数列(Catalan数)
P3200 [HNOI2009]有趣的数列 题目描述 我们称一个长度为2n的数列是有趣的,当且仅当该数列满足以下三个条件: (1)它是从1到2n共2n个整数的一个排列{ai}: (2)所有的奇数项满足 ...
- [Xcode 实际操作]七、文件与数据-(10)NSkeyedArchiver存储和解析数据,Swift对象的归档和恢复归档
目录:[Swift]Xcode实际操作 本文将演示如何使用归档的方法,对模型对象进行持久化工作. 在项目名称上点击鼠标右键,弹出右键菜单,选择[New File]新建文件命令, 在弹出的模板选项窗口中 ...
- [转] 【iOS基础知识】之判断NSString是否为整数、浮点数
//判断是否为整形: - (BOOL)isPureInt:(NSString*)string{ NSScanner* scan = [NSScannerscannerWithString:string ...
- Java Servlet图片上传至指定文件夹并显示图片
在学习Servlet过程中,针对图片上传做了一个Demo,实现的功能是:在a页面上传图片,点击提交后,将图片保存到服务器指定路径(D:/image):跳转到b页面,b页面读取展示绝对路径(D:/ima ...
- jar包冲突问题
这两天在启动一个新项目的时候,项目一直启动不了,报StackOverFlow; java.util.concurrent.ExecutionException: java.lang.StackOver ...
- Django之Form的ModelForm
form与model的终极结合. from django import forms class BookForm(forms.ModelForm): class Meta: model = model ...
- 图片旋转js代码
function rotateImage(imgId) { imageToRotate = document.getElementById(imgId); imageToRotate.style.fi ...
- hdu1166-敌兵布阵-分块
把区间分成√n份降低复杂度. #include<bits/stdc++.h> #define inf 0x3f3f3f3f ; ; using namespace std; int t,n ...