a survey for RL

• A finite set of states S_t summarizing the information the agent senses from the environment at every time step t ∈ {1, ..., T}.

• A set of actions A_t which the agent can perform at each time step t ∈ {1, ..., T} to interact with the environment.

• A set of transition probabilities between subsequent states which render the environment stochastic. Note: the probabilities are usually not explicitly modeled but the result of the stochastic nature of the financial asset’s price process.

• A reward (or return) function Rt which provides a numerical feedback value rt to the agent in response to its action A_t−1 = at−1 in state S_t−1 = s_t−1.

• A policy π which maps states to concrete actions to be carried out by the agent. The policy can hence be understood as the agent’s rules for how to choose actions.

• A value function V which maps states to the total (discounted) reward the agent can expect from a given state until the end of the episode (trading period) under policy π.

Given the above framework, the decision problem is formalized as finding the optimal policy π = π ∗ , i.e., the mapping from states to actions, corresponding to the optimal value function V ∗ - see also Dempster et al. (2001); Dempster and Romahi (2002):

　　V ^∗ (s_t) = max at E[R_t+1 + γV ^∗ (S_t+1)|S_t = s_t ].(1)

Hereby, E denotes the expectation operator, γ the discount factor, and Rt+1 the expected immediate reward for carrying out action At = at in state St = st . Further, St+1 denotes the next state of the agent. The value function can hence be understood as a mapping from states to discounted future rewards which the agent seeks to maximize with its actions.

To solve this optimization problem, the Q-Learning algorithm (Watkins, 1989) can be applied, extending the above equation to the level of state-action tuples:

　　Q ^∗ (s_t , a_t) = E[R_t+1 + γ max a_t+1 Q ^∗ (S_t+1, a_t+1)|S_t = s_t , A_t = a_t ].(2)

Hereby, the Q-value Q∗ (st , at) equals to the immediate reward for carrying out action At = at in state St = st plus the discounted future reward from carrying on in the best way possible.

The optimal policy π ^∗ (the mapping from states to actions) then simply becomes:

　　π ^∗ (s_t) = max a_t Q ^∗ (s_t , a_t) .(3)

i.e., in every state St = st , choose the action At = at that yields the highest Q-value. To approximate the Q-function during (online) learning, an iterative optimization is carried out with α denoting the learning rate - see also Sutton and Barto (1998) for further details:

　　Q ^∗ (s_t , a_t) ← (1 − α) Q ^∗ (s_t , a_t) + α (r_t+1 + γ max a_t+1 Q ^∗ (s_t+1, a_t+1) ) . (4)