Deep Q Learning

使用gym的CartPole作为环境，使用QDN解决离散动作空间的问题。

一、导入需要的包和定义超参数

import tensorflow as tf
import numpy as np
import gym
import time
import random
from collections import deque
 
#####################  hyper parameters  ####################
 
# Hyper Parameters for DQN
GAMMA = 0.9 # discount factor for target Q
INITIAL_EPSILON = 0.5 # starting value of epsilon
FINAL_EPSILON = 0.01 # final value of epsilon
REPLAY_SIZE = 10000 # experience replay buffer size
BATCH_SIZE = 32 # size of minibatch

二、DQN构造函数

1、初始化经验重放buffer；

2、设置问题的状态空间维度，动作空间维度；

3、设置e-greedy的epsilon；

4、创建用于估计q值的Q网络，创建训练方法。

5、初始化tensorflow的session

def __init__(self, env):
    # init experience replay
    self.replay_buffer = deque()
    # init some parameters
    self.time_step = 0
    self.epsilon = INITIAL_EPSILON
    self.state_dim = env.observation_space.shape[0]
    self.action_dim = env.action_space.n
 
    self.create_Q_network()
    self.create_training_method()
 
    # Init session
    self.session = tf.InteractiveSession()
    self.session.run(tf.global_variables_initializer())

三、创建神经网络

创建一个3层全连接的神经网络，hidden layer有20个神经元。

def create_Q_network(self):
    # network weights
    W1 = self.weight_variable([self.state_dim,20])
    b1 = self.bias_variable([20])
    W2 = self.weight_variable([20,self.action_dim])
    b2 = self.bias_variable([self.action_dim])
    # input layer
    self.state_input = tf.placeholder("float",[None,self.state_dim])
    # hidden layers
    h_layer = tf.nn.relu(tf.matmul(self.state_input,W1) + b1)
    # Q Value layer
    self.Q_value = tf.matmul(h_layer,W2) + b2
 
def weight_variable(self,shape):
    initial = tf.truncated_normal(shape)
    return tf.Variable(initial)
 
def bias_variable(self,shape):
    initial = tf.constant(0.01, shape = shape)
    return tf.Variable(initial)

定义cost function和优化的方法，使“实际”q值(y)与当前网络估计的q值的差值尽可能小，即使当前网络尽可能接近真实的q值。

def create_training_method(self):
    self.action_input = tf.placeholder("float",[None,self.action_dim]) # one hot presentation
    self.y_input = tf.placeholder("float",[None])
    Q_action = tf.reduce_sum(tf.multiply(self.Q_value,self.action_input),reduction_indices = 1)
    self.cost = tf.reduce_mean(tf.square(self.y_input - Q_action))
    self.optimizer = tf.train.AdamOptimizer(0.0001).minimize(self.cost)

从buffer中随机取样BATCH_SIZE大小的样本，计算y(batch中(s,a)在当前网络下的实际q值)

if done: y_batch.append(reward_batch[i])

else : y_batch.append(reward_batch[i] + GAMMA * np.max(Q_value_batch[i]))

def train_Q_network(self):
    self.time_step += 1
    # Step 1: obtain random minibatch from replay memory
    minibatch = random.sample(self.replay_buffer,BATCH_SIZE)
    state_batch = [data[0] for data in minibatch]
    action_batch = [data[1] for data in minibatch]
    reward_batch = [data[2] for data in minibatch]
    next_state_batch = [data[3] for data in minibatch]
 
    # Step 2: calculate y
    y_batch = []
    Q_value_batch = self.Q_value.eval(feed_dict={self.state_input:next_state_batch})
    for i in range(0,BATCH_SIZE):
      done = minibatch[i][4]
      if done:
        y_batch.append(reward_batch[i])
      else :
        y_batch.append(reward_batch[i] + GAMMA * np.max(Q_value_batch[i]))
 
    self.optimizer.run(feed_dict={
      self.y_input:y_batch,
      self.action_input:action_batch,
      self.state_input:state_batch
      })

四、Agent感知环境的接口

每次决策采取的动作，得到环境的反馈，将(s, a, r, s_, done)存入经验重放buffer。当buffer中经验数量大于batch_size时开始训练。

def perceive(self,state,action,reward,next_state,done):
    one_hot_action = np.zeros(self.action_dim)
    one_hot_action[action] = 1
    self.replay_buffer.append((state,one_hot_action,reward,next_state,done))
    if len(self.replay_buffer) > REPLAY_SIZE:
      self.replay_buffer.popleft()
 
    if len(self.replay_buffer) > BATCH_SIZE:
      self.train_Q_network()

五、决策(选取action)

两种选取方式greedy和e-greedy。

  def egreedy_action(self,state):
    Q_value = self.Q_value.eval(feed_dict = {
      self.state_input:[state]
      })[0]
    if random.random() <= self.epsilon:
        self.epsilon -= (INITIAL_EPSILON - FINAL_EPSILON) / 10000
        return random.randint(0,self.action_dim - 1)
    else:
        self.epsilon -= (INITIAL_EPSILON - FINAL_EPSILON) / 10000
        return np.argmax(Q_value)
 
  def action(self,state):
    return np.argmax(self.Q_value.eval(feed_dict = {
      self.state_input:[state]
      })[0])

Agent完整代码：

DQN

import tensorflow as tf
import numpy as np
import gym
import time
import random
from collections import deque
 
#####################  hyper parameters  ####################
 
# Hyper Parameters for DQN
GAMMA = 0.9 # discount factor for target Q
INITIAL_EPSILON = 0.5 # starting value of epsilon
FINAL_EPSILON = 0.01 # final value of epsilon
REPLAY_SIZE = 10000 # experience replay buffer size
BATCH_SIZE = 32 # size of minibatch
 
###############################  DQN  ####################################
 
class DQN():
  # DQN Agent
  def __init__(self, env):
    # init experience replay
    self.replay_buffer = deque()
    # init some parameters
    self.time_step = 0
    self.epsilon = INITIAL_EPSILON
    self.state_dim = env.observation_space.shape[0]
    self.action_dim = env.action_space.n
 
    self.create_Q_network()
    self.create_training_method()
 
    # Init session
    self.session = tf.InteractiveSession()
    self.session.run(tf.global_variables_initializer())
 
  def create_Q_network(self):
    # network weights
    W1 = self.weight_variable([self.state_dim,20])
    b1 = self.bias_variable([20])
    W2 = self.weight_variable([20,self.action_dim])
    b2 = self.bias_variable([self.action_dim])
    # input layer
    self.state_input = tf.placeholder("float",[None,self.state_dim])
    # hidden layers
    h_layer = tf.nn.relu(tf.matmul(self.state_input,W1) + b1)
    # Q Value layer
    self.Q_value = tf.matmul(h_layer,W2) + b2
 
  def create_training_method(self):
    self.action_input = tf.placeholder("float",[None,self.action_dim]) # one hot presentation
    self.y_input = tf.placeholder("float",[None])
    Q_action = tf.reduce_sum(tf.multiply(self.Q_value,self.action_input),reduction_indices = 1)
    self.cost = tf.reduce_mean(tf.square(self.y_input - Q_action))
    self.optimizer = tf.train.AdamOptimizer(0.0001).minimize(self.cost)
 
  def perceive(self,state,action,reward,next_state,done):
    one_hot_action = np.zeros(self.action_dim)
    one_hot_action[action] = 1
    self.replay_buffer.append((state,one_hot_action,reward,next_state,done))
    if len(self.replay_buffer) > REPLAY_SIZE:
      self.replay_buffer.popleft()
 
    if len(self.replay_buffer) > BATCH_SIZE:
      self.train_Q_network()
 
  def train_Q_network(self):
    self.time_step += 1
    # Step 1: obtain random minibatch from replay memory
    minibatch = random.sample(self.replay_buffer,BATCH_SIZE)
    state_batch = [data[0] for data in minibatch]
    action_batch = [data[1] for data in minibatch]
    reward_batch = [data[2] for data in minibatch]
    next_state_batch = [data[3] for data in minibatch]
 
    # Step 2: calculate y
    y_batch = []
    Q_value_batch = self.Q_value.eval(feed_dict={self.state_input:next_state_batch})
    for i in range(0,BATCH_SIZE):
      done = minibatch[i][4]
      if done:
        y_batch.append(reward_batch[i])
      else :
        y_batch.append(reward_batch[i] + GAMMA * np.max(Q_value_batch[i]))
 
    self.optimizer.run(feed_dict={
      self.y_input:y_batch,
      self.action_input:action_batch,
      self.state_input:state_batch
      })
 
  def egreedy_action(self,state):
    Q_value = self.Q_value.eval(feed_dict = {
      self.state_input:[state]
      })[0]
    if random.random() <= self.epsilon:
        self.epsilon -= (INITIAL_EPSILON - FINAL_EPSILON) / 10000
        return random.randint(0,self.action_dim - 1)
    else:
        self.epsilon -= (INITIAL_EPSILON - FINAL_EPSILON) / 10000
        return np.argmax(Q_value)
 
  def action(self,state):
    return np.argmax(self.Q_value.eval(feed_dict = {
      self.state_input:[state]
      })[0])
 
  def weight_variable(self,shape):
    initial = tf.truncated_normal(shape)
    return tf.Variable(initial)
 
  def bias_variable(self,shape):
    initial = tf.constant(0.01, shape = shape)
    return tf.Variable(initial)

训练agent：

train.py

from DQN import DQN
import gym
import numpy as np
import time
 
ENV_NAME = 'CartPole-v1'
EPISODE = 3000 # Episode limitation
STEP = 300 # Step limitation in an episode
TEST = 10 # The number of experiment test every 100 episode
 
def main():
  # initialize OpenAI Gym env and dqn agent
  env = gym.make(ENV_NAME)
  agent = DQN(env)
 
  for episode in range(EPISODE):
    # initialize task
    state = env.reset()
    # Train
    ep_reward = 0
    for step in range(STEP):
      action = agent.egreedy_action(state) # e-greedy action for train
      next_state,reward,done,_ = env.step(action)
      # Define reward for agent
      reward = -10 if done else 1
      ep_reward += reward
      agent.perceive(state,action,reward,next_state,done)
      state = next_state
      if done:
        #print('episode complete, reward: ', ep_reward)
        break
    # Test every 100 episodes
    if episode % 100 == 0:
      total_reward = 0
      for i in range(TEST):
        state = env.reset()
        for j in range(STEP):
          #env.render()
          action = agent.action(state) # direct action for test
          state,reward,done,_ = env.step(action)
          total_reward += reward
          if done:
            break
      ave_reward = total_reward/TEST
      print ('episode: ',episode,'Evaluation Average Reward:',ave_reward)
 
if __name__ == '__main__':
  main()

reference:

https://www.cnblogs.com/pinard/p/9714655.html

https://github.com/ljpzzz/machinelearning/blob/master/reinforcement-learning/dqn.py

强化学习_Deep Q Learning(DQN)_代码解析的更多相关文章

强化学习9-Deep Q Learning
之前讲到Sarsa和Q Learning都不太适合解决大规模问题,为什么呢? 因为传统的强化学习都有一张Q表,这张Q表记录了每个状态下,每个动作的q值,但是现实问题往往极其复杂,其状态非常多,甚至是连 ...
强化学习(十二) Dueling DQN
在强化学习(十一) Prioritized Replay DQN中,我们讨论了对DQN的经验回放池按权重采样来优化DQN算法的方法,本文讨论另一种优化方法,Dueling DQN.本章内容主要参考了I ...
【转载】强化学习(十一) Prioritized Replay DQN
原文地址: https://www.cnblogs.com/pinard/p/9797695.html ------------------------------------------------ ...
强化学习(十一) Prioritized Replay DQN
在强化学习(十)Double DQN (DDQN)中,我们讲到了DDQN使用两个Q网络,用当前Q网络计算最大Q值对应的动作,用目标Q网络计算这个最大动作对应的目标Q值,进而消除贪婪法带来的偏差.今天我 ...
强化学习（四）—— DQN系列（DQN, Nature DQN, DDQN, Dueling DQN等）
1 概述在之前介绍的几种方法,我们对值函数一直有一个很大的限制,那就是它们需要用表格的形式表示.虽说表格形式对于求解有很大的帮助,但它也有自己的缺点.如果问题的状态和行动的空间非常大,使用表格表示难 ...
强化学习10-Deep Q Learning-fix target
针对 Deep Q Learning 可能无法收敛的问题,这里提出了一种 fix target 的方法,就是冻结现实神经网络,延时更新参数. 这个方法的初衷是这样的: 1. 之前我们每个(批)记忆都 ...
强化学习（3）-----DQN
看这篇https://blog.csdn.net/qq_16234613/article/details/80268564 1.DQN 原因:在普通的Q-learning中,当状态和动作空间是离散且维 ...
强化学习 - Q-learning Sarsa 和 DQN 的理解
本文用于基本入门理解. 强化学习的基本理论 : R, S, A 这些就不说了. 先设想两个场景: 一. 1个 5x5 的格子图, 里面有一个目标点, 2个死亡点二. 一个迷宫, 一个出发点, ...
转：强化学习（Reinforcement Learning）
机器学习算法大致可以分为三种: 1. 监督学习(如回归,分类) 2. 非监督学习(如聚类,降维) 3. 增强学习什么是增强学习呢? 增强学习(reinforcementlearning, RL)又叫 ...

随机推荐

PHP中的常用正则表达式集锦
PHP中的常用正则表达式集锦: 匹配中文字符的正则表达式: [\u4e00-\u9fa5] 评注:匹配中文还真是个头疼的事,有了这个表达式就好办了匹配双字节字符(包括汉字在内):[^\x00-\xf ...
CTP 下单返回错误: 没有报单权限和字段错误需要注意的问题
没有报单权限一般被认为期货公司没有开权限, 但是更多的问题是没有填写 BrokerId, InvestorId 下单字段错误注意一个容易忽略的地方: a. order 应该全部设为0, b. orde ...
DataGridView DataSource 如何实现排序
将数据绑定在下面的类中就可以实现排序 public class SortableBindingList<T> : BindingList<T> { private ArrayL ...
IT兄弟连 JavaWeb教程 Servlet会话跟踪 Cookie技术原理
Cookie使用HTTPHeader传递数据.Cookie机制定义了两种报头,Set-Cookie报头和Cookie报头.Set-Cookie报头包含于Web服务器的响应头(ResponseHeade ...
chmod 详解
http://man.linuxde.net/chmod chmod u+x,g+w f01 //为文件f01设置自己可以执行,组员可以写入的权限 chmod u=rwx,g=rw,o=r f01 c ...
day04 Calendar类
缺少mscvr100.dll
最后使用百度电脑专家修复好的!
[BZOJ5219]最长路径
Description 在Byteland一共有n个城市,编号依次为1到n,它们之间计划修建n(n-1)/2条单向道路,对于任意两个不同的点i和 j,在它们之间有且仅有一条单向道路,方向要么是i到j, ...
Codeforces 997D(STL+排序)
D. Divide by three, multiply by two time limit per test 1 second memory limit per test 256 megabytes ...
洛谷1072（gcd的运用）
已知正整数a0,a1,b0,b1,设某未知正整数x满足: 1． x 和 a0 的最大公约数是 a1: 2． x 和 b0 的最小公倍数是b1. Hankson 的“逆问题”就是求出满足条件的正整数 ...

强化学习_Deep Q Learning(DQN)_代码解析