f= 100*(Y-X^2)^2+ (1-X)^2;
(Y-X^2) involves X to degree 2, and then you square that, (Y-X^2)^2, so your f is going to have X to degree 4. You substitute something+h in for X in f, so you are going to end up with h^4 . You differentiate with respect to h, getting a cubic in h. solve() of that for h is going to have three solutions.
Your test function ends up having only X^2 so the subs() gives you something in h^2, differentiate and you get something in h^1 which has only a single solution.
What can you do? Well, you can do the double-differentiate test to determine which values are local minima or maxima. However eventually at some input function you would run into there being multiple local minima or maxima, and then you would be stuck.
You should be reviewing the entire algorithm to figure out what the algorithm says to do if there are multiple solutions for h.