RBF network not generalizing well

Hi all,

I'm trying RBF for a classification problem. The input matrix is of size 40,000 x 2,500 and output matrix is of size 25 x 2,500. Output matrix consists of 0 and 1 only.

As stated in most examples, RBF can give 100% accuracy in training data, so is happening in my case. However, when I test the network, the accuracy is very poor. I've tried a wide range of SPREAD values, but none of them helped in generalizing the network.

The input dataset values lie between -960.0 to +960.0.

Few SPREAD values that I've tried includes:

 5e-9, 5e-6, 0.005, 0.01, 0.25, 1.0, 5.0, 100.0, 450.0

Can anyone suggest how testing accuracy of RBF can be improved?

So far, I've used newrb and kept number of neurons in hidden layer as 250. Will increasing neurons in hidden layer help?

Thanks in advance !!

Answer 1

[Greg Heath]

Ntrn ~ 0.7*2500 = 1750 samples span, AT MOST, a 1,749 dimensional input space.

Therefore, an input dimension of 40,000 is ridiculously large for a practical model of the data.

Consequently, the first thing you need to do is reduce the input dimension to a more practical size.

Hope this helps.

Greg

Answer 2

[John D'Errico]

I'm not an expert in neural nets, so I cannot offer specific advice about the NN tools in MATLAB. However, I am an expert on modeling.

Whenever you fit a model using a training and validation set, there are several things to consider. Data will have signal in it, as well as noise. No matter how much data you have, you might still have a large amount of corrupting noise. So you have a training set, and a validation set to see how well your model does.

Let me give an example. Pictures always make things easy to follow.

nt = 1000;
xt = rand(nt,1);
yt = 2 + xt + randn(size(xt))/5;
nv = 250;
xv = rand(nv,1);
yv = 2 + xv + randn(size(xv))/5;
plot(xt,yt,'bo',xv,yv,'rx')
grid on

So in blue, the training data, in red, the validation data.

Suppose I start out with a sophisticated, complex model, that fits the training data exactly?

spl = spline(xt,yt);
hold on
xpred = linspace(0,1,3000);
ypred = ppval(spl,xpred);
plot(xpred,ypred,'b-')

Ok. I'm not sure you can see it, but the curve passes exactly through the blue data points. It fits the training set EXACTLY. Of course, I had to restrict the plot limits on y, since the curve is so highly oscillatory.

Would you have any expectation that this model will fit the validation set well? God, I hope not.

std(yv - ppval(spl,xv))
ans =
        1.92881347645068

In fact, even though I know the noise standard deviation was 0.2, the standard deviation of the residuals for the validation set was HUGE, roughly 10 times as large.

In fact, the true signal here was a very simple one. The sophisticated model ended up chasing noise, to no good end. It was a waste of CPU cycles.

p = polyfit(xt,yt,1)
p =
          1.04034582451399          1.97520900998711

std(yt - polyval(p,xt))
ans =
         0.200663205871802

std(yv - polyval(p,xv))
ans =
         0.180740826588139

In fact, the standard deviation of the residuals for both the training and validation set is roughly the same, and both are of the same size as what I know to be the true noise level. While much of the time we won't know how much noise is in a complex set of data, it is important the two sets yield roughly the same standard deviation.

So the idea is to start out SMALL. Can you build a simple model that will predict your data well? A measure is that your model also predicts the validation set roughly just as well. If the two predictions are comparable, then it suggests that you are at least encapsulating some basic signal in the data into your model.

Now, you might increase the complexity of the model. Does it improve the fit? Does the quality of prediction also improve by a commensurate amount for the validation set? As long as these things continue to happen as you increase the model complexity, you can be somewhat happy. However, when you get to the point where your increased complexity drives the residuals for the training set down to zero, yet the validation set gets worse in predictive ability, so the two no longer track each other, you have gone TOO far.

Always, start small and simple, not big. What you have done is exactly that. Increasing the model complexity is exactly the wrong thing to do.

[Greg Heath]

"Always, start small and simple, not big. What you have done is exactly that. Increasing the model complexity is exactly the wrong thing to do."

NO, NO, NO!

It is EXACTLY the RIGHT THING to do! My approach as been successfully used for 35 years:

 Loop over a succession of more complex models.   
 Stop when the decrease in mse becomes insignificant 
 (e.g., less than 1 pct).

Greg

Answer 3

[Akshay Joshi]

Thanks Greg, John !!