Histogram gives a wrong result？

I was debugging the codes to find what was wrong, but I just found histogram can give wrong results.

This is an example. Accoring to the document (Histogram - Histogram plot - MATLAB), it should give the all statistical values as 5 except the last one, but instead, there are wrong variances. (Due to the floating number?)

Is there an accurate way to do the histogram?

a=0:0.01:1; figure; h=histogram(a,0:0.05:1);

Answer 1

[John D'Errico]

Never assume that two mathematically identical floating point numbers will, if computed using different expressions, produce the the identical result. You tripped over that when you assumed 0:.01:1 and 0:0.05:1 would in fact "line up".

a=0:0.01:1; figure; h=histogram(a,0:0.05:1);

The crux of the problem can be seen in this call to intersect.

intersect(a,0:0.05:1)

ans = 1×19

0 0.0500 0.1000 0.2000 0.2500 0.3500 0.4000 0.4500 0.5000 0.5500 0.6000 0.6500 0.7000 0.7500 0.8000 0.8500 0.9000 0.9500 1.0000

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

setdiff(0:0.05:1,a)

ans = 1×2

0.1500 0.3000

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

Do you see that 0.15 is not in the intersection, that is, it did not appear in both vectors? As well, 0.30 is also missing.

b = 0:0.05:1;
a([16,31])

ans = 1×2

0.1500 0.3000

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

b([4,7])

ans = 1×2

0.1500 0.3000

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

It certainly looks as if both numbers appear in each vector. And they look identical. But they are not in fact identical. Again, compute them in different ways, and you run the risk they will not be the same.

num2str(a([16,31]),55)

ans = '0.1499999999999999944488848768742172978818416595458984375      0.299999999999999988897769753748434595763683319091796875'

num2str(b([4,7]),55)

ans = '0.15000000000000002220446049250313080847263336181640625        0.3000000000000000444089209850062616169452667236328125'

It looks like in all other cases, the two vector computations did generate the same set of numbers, since intersect would have caught more cases otherwise. But there are tiny, imperceptible differences. How far apart are those numbers?

num2hex(a([16,31]))

ans = 2×16 char array
    '3fc3333333333333'
    '3fd3333333333333'

num2hex(b([4,7]))

ans = 2×16 char array
    '3fc3333333333334'
    '3fd3333333333334'

As you can see, they actually differ by one single bit, down at the level of the least significant bit. So what went wrong is those two elements ended up in the wrong bins, fairly arbitrarily. And this caused the counts for successive bins to be unexpectedly different. Not wrong. Just different from what you expected. The difference arises because of the floating point representations of the numbers you created, and then passed to histogram, not because of anything histogram could be expected to do.

Can you "fix" that? Not easily in some automatic way. Yes, you can often use tolerances. But histogram does not employ tolerances when it computes frequencies of elements landing in bins, nor should it do so. And there is no flag you can set in the call to histogram to fix it. Histogram cannot understand the subtle difference here, nor can/should it worry about least significant bits.

I suppose, by understanding what is happening, in this specific case you might have tried this, which offsets the bin boundaries subtly by just enough the counts come out "correct".

 h=histogram(a,-1e-12:0.05:1.05);

And indeed, now the counts per bin are correct, with the final bin catching only that last element at 1. It worked because I understood the root cause of the problem, and that led me to a solution which would fix it for this specific case.

Answer 2

[Matt J]

Due to the floating number?

Correct. There is no reason to expect exact boundaries, or for data to fall exactly within those boundaries, when the inputs are generated with floating point math. One solution would be to recast the operation in terms of integers,

a=0:100; figure; h=histogram(a,0:5:100);
xticklabels(str2double(xticklabels)/100)

Warning: Graphics acceleration hardware is unavailable. Graphics quality and performance might be diminished. See MATLAB System Requirements.

[Tianlun]

So there is no accurate way to do a histogram. My code is just an example. With real data, there is no way to normalize them into integer.

[Steven Lord]

In case anyone is wondering, that last bin being higher than the rest is not a bug. Every bin except for the last bin includes its left edge but excludes its right edge. The last bin includes both its left and right edges. That way each point in the range covered by the edges is part of one and only one bin.

The first bin covers [0, 5) and so will count a = 0, a = 1, a = 2, a = 3, and a = 4.

The last bin covers [95, 100] and so will count a = 95, a = 96, a = 97, a = 98, a = 99, and a = 100.

If you update the edges list so 100 is its own bin:

a = 0:100;
histogram(a, 0:5:105)

Warning: Graphics acceleration hardware is unavailable. Graphics quality and performance might be diminished. See MATLAB System Requirements.

[Steven Lord]

So there is no accurate way to do a histogram.

Sure there is, as long as you realize that many, many numbers you can type cannot be stored in double precision. Do you expect a = 0:0.1:1 to exactly contain the numbers one-tenth, two-tenths, three-tenths, etc.?

a = 0:0.1:1;
tentimes = 10*a;
shouldThisBeAllZero = tentimes - (0:10)

shouldThisBeAllZero = 1×11

1.0e-15 * 0 0 0 0.4441 0 0 0 0 0 0 0

<mw-icon class=""></mw-icon>

<mw-icon class=""></mw-icon>

If you believe the 4th element of shouldThisBeAllZero not being 0 is incorrect, please try this little experiment. Find something to write with and something to write on (ideally compatible things; pencil and paper not pencil and whiteboard.)

Step 1: Using long division (like you learned in school) divide 1 by 3. Call the result x. You are allowed to write as many decimal places of the result as you want, but only those you explicitly write can be used in step 2. No using 0.3 repeating to get "an infinite" number of places.

Step 2: Multiply x by 3. Call the result y.

In exact arithmetic we know (1/3)*3 is exactly 1. But the x value you defined in step 1 is not one third. It is slightly smaller than one third because you rounded off one third to fit it into x. If you've written one more decimal place in step 1 you'd have an x that's closer to one third than the x you actually used in step 2. Therefore y will not be 1. The value stored in y will be slightly smaller than 1.

This is exactly the same scenario, only the number in the computer is stored in base 2 and the computer stores the number to a certain number of binary places. Integer values (up to a certain point) can be exactly represented. So can numbers that are [waves hands a bit] "the sum of a small number of relatively closely spaced powers of 2". 0.5, 0.25, and 0.75 (2^-1+2^-2) are examples of those types of numbers; 0.1 and 0.3 aren't.

[Matt J]

So there is no accurate way to do a histogram?

histogram(a,e) is giving you perfectly accurate results. It is your input data (both a and e) that are inaccurate, because of floating point approximation. To avoid this issue, choose edge data e(i) that are well-separated from your a(j). By well-separated, I mean, greater than floating point precision thresholds.