rollout

R2026b

Executes rollout for a vectorized environment with a given policy

Since R2026b

Syntax

[NextObservation,Observation,Action,Reward,IsDone,policy] = rollout(venv,policy,NumSteps)

[NextObservation,Observation,Action,Reward,IsDone,policy] = rollout(venv,policy,NumSteps,MaxStepsPerEpisode=Nmax)

Description

Use rollout to execute a specified number of simulation steps of a given policy on all instances of a vectorized environment.

[NextObservation,Observation,Action,Reward,IsDone,policy] = rollout(venv,policy,NumSteps) executes NumSteps simulation steps using the policy policy on each instance of the vectorized environment venv. The function generates a total of NumSteps*venv.NumEnv samples.

[NextObservation,Observation,Action,Reward,IsDone,policy] = rollout(venv,policy,NumSteps,MaxStepsPerEpisode=Nmax) also specifies the maximum steps for each environment instance. The function resets all environment instances that reach or exceed this value.

Examples

Simulate Policy on Vectorized Environment

This example shows how to use the rollout function to execute six simulation steps of a given policy on all instances of a vectorized environment.

First, create a vectorized version of the cart-pole environment with continuous action space. This environment has four states.

venv = rlVectorEnv(@(info) rlPredefinedEnv("CartPole-Continuous"))

venv = 
  rlVectorEnv with properties:

    NumEnv: 2

By default, the vectorized environment has two instances.

Create a PPO agent for that environment.

agent = rlPPOAgent(getObservationInfo(venv),getActionInfo(venv));

Extract the exploration policy from the PPO agent.

policy = getExplorationPolicy(agent);

Simulate the policy on all instances of the vectorized environment for 6 steps.

[NextObs,Obs,Act,Rwd,Isd,policy] = rollout(venv,policy,6);

Display the sizes of the observation batch.

Obs

Obs = 1×1 cell array
    {4×1×12 double}

The batch dimension is the number of environment instances multiplied by the number of steps.

Display the is-done batch.

Isd

Isd = 1×12 uint8 row vector

   0   0   0   0   0   3   0   0   0   0   0   3

Within a custom training loop, you can use the rollout function to simulate a policy against a vectorized environment.

Input Arguments

venv — Vectorized environment

rlVectorEnv object

Vectorized environment, specified as an rlVectorEnv object.

Example: venv = rlVectorEnv(@(info) rlPredefinedEnv("DoubleIntegrator-Discrete")) creates the vector environment venv from the predefined double-integrator environment with discrete action space.

policy — Reinforcement learning policy

reinforcement learning policy object

Reinforcement learning policy, specified as one of the following objects:

• rlMaxQPolicy

• rlEpsilonGreedyPolicy

• rlDeterministicActorPolicy

• rlAdditiveNoisePolicy

• rlStochasticActorPolicy

• rlHybridStochasticActorPolicy

For more information on reinforcement learning policies, see Create Actors, Critics, and Policy Objects.

Example: policy = getExplorationPolicy(rlPPOAgent(rlNumericSpec([2 1]),rlNumericSpec([1 1]))) extracts the object that implements the exploration policy from a default PPO agent and assigns it to the variable policy.

NumSteps — Number of simulation steps

positive integer scalar

Number of simulation steps to take for each environment instance, specified as a positive integer scalar.

Example: 128

Nmax — Maximum number of simulation steps

500 (default) | positive integer scalar

Maximum number of simulation steps to take for each environment instance, specified as a positive integer scalar. If you do not specify this argument, the function uses the default value of 500.

Example: 1000

Output Arguments

NextObservation — Batch of next observations

cell array

Batch of next observations, returned as an array whose first dimensions are specified in getObservationInfo(venv). The batch dimension is equal to venv.NumEnv multiplied by NumSteps.

Observation — Batch of current observations

cell array

Batch of current observations, returned as an array whose first dimensions are specified in getObservationInfo(venv). The batch dimension is equal to venv.NumEnv multiplied by NumSteps.

Action — Batch of actions

cell array

Batch of actions, returned as an array whose first dimensions are specified in getActionInfo(venv). The batch dimension is equal to venv.NumEnv multiplied by NumSteps.

Reward — Batch of rewards

row vector

Batch of actions, returned as a row vector with length equal to venv.NumEnv multiplied by NumSteps.

IsDone — Is-done batch

row vector

Is-done batch, returned as an uint8 row vector with length equal to venv.NumEnv multiplied by NumSteps.

policy — Updated policy

policy object

Updated policy, returned as the same policy object as the policy input argument. Typically, the policy updates its noise parameters during simulation.

Version History

Introduced in R2026b