Neural Network (NARX) performance interpretation - how small is small?

I have built a neural network to forecast currency pair prices (trend) a few steps ahead. How do I interpret the performance plot results? For instance, the error histogram, the NMSE (what is considered small for currency pairs? How do I assess if these metrics are optimal?

Answer 1

[Greg Heath]

If you naively model a function with a straight line, the MSE is just the variance of the function and varies with the scale of the function.

In previous MATLAB posts I have used the notations

      MSE00 = vart1 = mean(var(target'),1)

and use it as a standard for measuring MSE.

Using this NAIVE model as a comparative standard, the normalized MSE is defined by

       NMSE = MSE/mean(var(target',1))

which is related to the statistical RSQUARE via

              RSQUARE = 1 - NMSE

For reasonable models both NMSE and RSQUARE lie within the closed interval [ 0,1].

For most regression and classification models, I use a goal

         NMSEgoal = 0.01 % Regression & Classification

However, in order to obtain that goal for CLOSED LOOP time series, I use the OPEN LOOP goals

          NMSEgoal = 0.005 or 0.001

I don't know of anyone who complains when Rsquare >= 0.99 or equivalently, NMSE <= 0.01.

Hope this helps.

Thank you for formally accepting my answer

Greg