Problems with text generation using a bilstmLayer

I have been exploring various methods for text generation using deep learning models and from some reading I have done bilstm networks seem to be able to learn dependencies better since they look in both temporal directions. I have seen people get good results using them in Python but when I try to implement them in Matlab the network only generates a mess of characters. The training progress UI shows the network reaching ~90% accuracy and almost 0 loss very quickly. However, the trained net fails to produce anything other than random strings of characters. I tried it in my own code and then in Mathworks generate text example below. Changing the lstmLayer to a bilstmLayer does not yield any meaningful results. Is there a different way the net must be used to generate predictions or is it just not suitable for this task?

https://www.mathworks.com/help/deeplearning/examples/generate-text-using-deep-learning.html