## Deep Deterministic Policy Gradient Agents

The deep deterministic policy gradient (DDPG) algorithm is a model-free, online, off-policy reinforcement learning method. A DDPG agent is an actor-critic reinforcement learning agent that searches for an optimal policy that maximizes the expected cumulative long-term reward.

For more information on the different types of reinforcement learning agents, see Reinforcement Learning Agents.

DDPG agents can be trained in environments with the following observation and action spaces.

Observation SpaceAction Space
Continuous or discreteContinuous

DDPG agents use the following actor and critic representations.

CriticActor

Q-value function critic Q(S,A), which you create using `rlQValueRepresentation`

Deterministic policy actor π(S), which you create using `rlDeterministicActorRepresentation`

During training, a DDPG agent:

• Updates the actor and critic properties at each time step during learning.

• Stores past experiences using a circular experience buffer. The agent updates the actor and critic using a mini-batch of experiences randomly sampled from the buffer.

• Perturbs the action chosen by the policy using a stochastic noise model at each training step.

### Actor and Critic Functions

To estimate the policy and value function, a DDPG agent maintains four function approximators:

• Actor π(S|θ)— The actor, with parameters θ, takes observation S and returns the corresponding action that maximizes the long-term reward.

• Target actor πt(S|θt) — To improve the stability of the optimization, the agent periodically sets the target actor parameters θt to the latest actor parameter values.

• Critic Q(S,A|ϕ) — The critic, with parameters ϕ, takes observation S and action A as inputs and returns the corresponding expectation of the long-term reward.

• Target critic Qt(S,A|ϕt) — To improve the stability of the optimization, the agent periodically sets the target critic parameters ϕt to the latest critic parameter values.

Both Q(S,A) and Qt(S,A) have the same structure and parameterization, and both π(S) and πt(S) have the same structure and parameterization.

When training is complete, the trained optimal policy is stored in actor π(S).

For more information on creating actors and critics for function approximation, see Create Policy and Value Function Representations.

### Agent Creation

You can create and train DDPG agents at the MATLAB® command line or using the Reinforcement Learning Designer app.

For more information on creating agents using Reinforcement Learning Designer, see Create Agents Using Reinforcement Learning Designer.

At the command line, you can create a DDPG agent with default actor and critic representations based on the observation and action specifications from the environment. To do so, perform the following steps.

1. Create observation specifications for your environment. If you already have an environment interface object, you can obtain these specifications using `getObservationInfo`.

2. Create action specifications for your environment. If you already have an environment interface object, you can obtain these specifications using `getActionInfo`.

3. If needed, specify the number of neurons in each learnable layer or whether to use an LSTM layer. To do so, create an agent initialization option object using `rlAgentInitializationOptions`.

4. If needed, specify agent options using an `rlDDPGAgentOptions` object.

5. Create the agent using an `rlDDPGAgent` object.

Alternatively, you can create actor and critic representations and use these representations to create your agent. In this case, ensure that the input and output dimensions of the actor and critic representations match the corresponding action and observation specifications of the environment.

1. Create an actor using an `rlDeterministicActorRepresentation` object.

2. Create a critic using an `rlQValueRepresentation` object.

3. Specify agent options using an `rlDDPGAgentOptions` object.

4. Create the agent using an `rlDDPGAgent` object.

For more information on creating actors and critics for function approximation, see Create Policy and Value Function Representations.

### Training Algorithm

DDPG agents use the following training algorithm, in which they update their actor and critic models at each time step. To configure the training algorithm, specify options using an `rlDDPGAgentOptions` object.

• Initialize the critic Q(S,A) with random parameter values ϕ, and initialize the target critic parameters ϕt with the same values: ${\varphi }_{t}=\varphi$.

• Initialize the actor π(S) with random parameter values θ, and initialize the target actor parameters θt with the same values: ${\theta }_{t}=\theta$.

• For each training time step:

1. For the current observation S, select action A = π(S) + N, where N is stochastic noise from the noise model. To configure the noise model, use the `NoiseOptions` option.

2. Execute action A. Observe the reward R and next observation S'.

3. Store the experience (S,A,R,S') in the experience buffer. The length of the experience buffer is specified in the `ExperienceBufferLength` property of the `rlDDPGAgentOptions` object.

4. Sample a random mini-batch of M experiences (Si,Ai,Ri,S'i) from the experience buffer. To specify M, use the `MiniBatchSize` property of the `rlDDPGAgentOptions` object.

5. If S'i is a terminal state, set the value function target yi to Ri. Otherwise, set it to

`${y}_{i}={R}_{i}+\gamma {Q}_{t}\left({S}_{i}\text{'},{\pi }_{t}\left({S}_{i}\text{'}|{\theta }_{t}\right)|{\varphi }_{t}\right)$`

The value function target is the sum of the experience reward Ri and the discounted future reward. To specify the discount factor γ, use the `DiscountFactor` option.

To compute the cumulative reward, the agent first computes a next action by passing the next observation S'i from the sampled experience to the target actor. The agent finds the cumulative reward by passing the next action to the target critic.

6. Update the critic parameters by minimizing the loss L across all sampled experiences.

`$L=\frac{1}{M}\sum _{i=1}^{M}{\left({y}_{i}-Q\left({S}_{i},{A}_{i}|\varphi \right)\right)}^{2}$`
7. Update the actor parameters using the following sampled policy gradient to maximize the expected discounted reward.

`$\begin{array}{l}{\nabla }_{\theta }J\approx \frac{1}{M}\sum _{i=1}^{M}{G}_{ai}{G}_{\pi i}\\ {G}_{ai}={\nabla }_{A}Q\left({S}_{i},A|\varphi \right)\text{ }\text{where}\text{\hspace{0.17em}}A=\pi \left({S}_{i}|\theta \right)\\ {G}_{\pi i}={\nabla }_{\theta }\pi \left({S}_{i}|\theta \right)\end{array}$`

Here, Gai is the gradient of the critic output with respect to the action computed by the actor network, and Gπi is the gradient of the actor output with respect to the actor parameters. Both gradients are evaluated for observation Si.

8. Update the target actor and critic parameters depending on the target update method. For more information see Target Update Methods.

For simplicity, the actor and critic updates in this algorithm show a gradient update using basic stochastic gradient descent. The actual gradient update method depends on the optimizer you specify using `rlRepresentationOptions`.

### Target Update Methods

DDPG agents update their target actor and critic parameters using one of the following target update methods.

• Smoothing — Update the target parameters at every time step using smoothing factor τ. To specify the smoothing factor, use the `TargetSmoothFactor` option.

• Periodic — Update the target parameters periodically without smoothing (`TargetSmoothFactor = 1`). To specify the update period, use the `TargetUpdateFrequency` parameter.

• Periodic Smoothing — Update the target parameters periodically with smoothing.

To configure the target update method, create a `rlDDPGAgentOptions` object, and set the `TargetUpdateFrequency` and `TargetSmoothFactor` parameters as shown in the following table.

Update Method`TargetUpdateFrequency``TargetSmoothFactor`
Smoothing (default)`1`Less than `1`
PeriodicGreater than `1``1`
Periodic smoothingGreater than `1`Less than `1`

## References

[1] Lillicrap, Timothy P., Jonathan J. Hunt, Alexander Pritzel, Nicolas Heess, Tom Erez, Yuval Tassa, David Silver, and Daan Wierstra. “Continuous Control with Deep Reinforcement Learning.” ArXiv:1509.02971 [Cs, Stat], September 9, 2015. https://arxiv.org/abs/1509.02971.