Mixed DataType in action space

For mixed type of Data in action space, which of them will be used rlFiniteSetSpec or rlNumericSpec. I have seen a query regarding the observation space, but does it holds for action space too?

https://www.mathworks.com/matlabcentral/answers/706758-a-mix-of-rlnumericspec-and-rlfinitesetspec-objects-observation-for-a-rl-environment

Answer 1

[Shaik]

Hi,

The choice between rlFiniteSetSpec and rlNumericSpec for the action space depends on the nature of the action you want to define in your reinforcement learning (RL) environment.

• rlFiniteSetSpec: Use this specification when the action space consists of a discrete set of possible actions. Each action in the set can be represented by a unique value or identifier. For example, if your action space consists of a finite set of discrete actions, such as "move forward," "turn left," or "turn right," you can use rlFiniteSetSpec to define the action space.
• rlNumericSpec: Use this specification when the action space is continuous or can be represented by a range of numerical values. If your action space requires specifying a continuous value, such as controlling the speed or position of a robotic arm, you can use rlNumericSpec to define the action space.

When defining the action space in an RL environment, you need to choose the appropriate specification based on the characteristics of the actions you want to represent. You can have a mix of rlFiniteSetSpec and rlNumericSpec objects within the action space if your environment requires both discrete and continuous actions.

Regarding the link you provided, it refers to a query about the observation space, but the same concept applies to the action space as well. The rlFiniteSetSpec and rlNumericSpec objects can be used to define both observation and action spaces in an RL environment, depending on the type of data they represent.

In summary, consider the nature of your action space (discrete or continuous) and use either rlFiniteSetSpec or rlNumericSpec accordingly to define the appropriate specification for your action space in the RL environment.

[Sania Gul]

Thankyou Shaik. I am working on "wareHouse Robot" given in MATLAB Reinforcement Learning on Ramp. There the actions i.e. The translational and Rotational forces applied on robot to avoid the obstacles are continuous. and these actions are then used to generate the continuous observations including 6 parameters" x', 'y', cos theta', 'sin theta', 'velocity' and 'omega' (angular velocity) as given in its Simulink model given in Figure below.Now, I want the rotations (sin theta) and (cos theta) to be discrete only (1,0,-1), while others should be continuous, but if I make Rotational Force discrete, it would make the states 'x' and 'y' discrete too as they are generated by both Translational and Rotational Forces as shown in next Figure. What should I do?

[Shaik]

If you want to keep the rotational forces discrete while keeping the states 'x' and 'y' continuous, one possibility is to modify the state representation. One approach could be to represent the angle θ as an integer value (e.g., in degrees or radians) rather than using sin(θ) and cos(θ).

Here's an example of how you could modify the state representation:

1. Define a maximum and minimum angle, θ_max and θ_min, respectively, in degrees or radians, depending on your preference. For this example, let's assume that θ can vary between -180 and 180 degrees.
2. Define the rotation angle, θ, as an integer value between θ_min and θ_max. You can use the 'discretize' function in MATLAB to obtain the corresponding integer value. For example:

theta_bins = linspace(-180, 180, num_bins); % divide the range [-180, 180] into num_bins equally spaced intervals

theta = discretize(atan2(sin_theta, cos_theta)*180/pi, theta_bins);

Here, num_bins is the number of intervals you want to divide the range into. The atan2 function is used to obtain the angle from sin_theta and cos_theta, and then this angle is converted to degrees using the *180/pi scaling factor.

1. Update the state representation to use the new rotation angle θ instead of sin(θ) and cos(θ). You would need to modify the observation space to include the new state variables, and also the action space to properly map the discrete action space to the continuous control inputs.

You can experiment with different values of num_bins to see how it affects the discretization of the rotation angle. A larger value of num_bins will result in a finer discretization of the angle, at the cost of increased computational complexity and training time.

[Sania Gul]

Thank you sooo much. No words of gratitude are enough for this guidance. Stay happy. :-)

[Shaik]

If my answer is helpful, can you accept it please?

[Sania Gul]

I have already done it :-)

[Shaik]

Thanks Sania! :)