Based on this information, it appears that this issue is likely caused due to the following reasons:
1) Large input dimensions of the images.
2) Limited memory availability on the GPU card.
3) Size and complexity of the network architecture.
Furthermore, the Faster R-CNN method processes the entire input image without resizing. This is in contrast to the 'RCNNObjectDetector' method which performs inherent cropping and resizing of regions to be comparable to the input dimensions of the network.
A couple of workarounds that you may consider to continue using the 'trainFasterRCNNObjectDetector' method are mentioned below:
1) Setting the 'SmallestImageDimension' parameter of the 'trainFasterRCNNObjectDetector' to a value of 400. However, this parameter may need to be adjusted to account for your current GPU specifications. Setting this parameter will resize the images during training and help avoid 'Out of memory' errors.
2) Using a smaller network in addition to setting the 'SmallestImageDimension' parameter as shown in this example: https://www.mathworks.com/help/vision/examples/object-detection-using-faster-r-cnn-deep-learning.html
3) Updating the GPU card. To train a network like AlexNet using the Faster R-CNN method, a GPU with more memory might be required.
I believe that the above suggestions should help resolve this issue.