Here is how I would approach your problem. First we write
a*r*dr/dt*log(a*r*dr/dt) = a*b/r^6
Now define w:
w = lambertw(a*b/r^6)
a*r*dr/dt = exp(lambertw(a*b/r^6))
dr/dt = 1/(a*r)*exp(lambertw(a*b/r^6))
Now finally you have a differential equation in the form that Matlab's ode functions can evaluate numerically, provided you have the lambertw function available.