Reinforcement Learning Simulations slows down significantly over time

Hello,

i am currently use double Deep Q-Learning for process design (episodic task) and i wanted to do a deeper analysis of the convergence behaviour of my algorithm. Thats why i tried to train the algorithm for 300k Iterations instead of 50k (averaging 4 (forming-)steps each Episode) and recognized, that the computation time for one Iteration trebled over time (13 Iterations per min in the beginning / 4 Iterations per min after 150k Iterations) even though the average number of steps did not change or decreased.

The pc i am using does not overheat ,so no throttling, and there is enough free RAM ,so no memory leak.

As a precaution i deactivated the Trainingsfigure, but that had no influence on the calculation time as well.

Does anybode have an idea what could caus the increase in computing time?

Tanks in advance and king regards

Niklas

[Emmanouil Tzorakoleftherakis]

What settings are you using for DQN? Particularly mini batch and experience buffer size may play a role.

[Niklas Reinisch]

Buffersize is 10.000 and mini batch size is 64.

So the decrease in speed is happening way after the buffer is full.