calculation error for floor function

9 vues (au cours des 30 derniers jours)
Nishanth Kumar
Nishanth Kumar le 6 Fév 2017
Commenté : Shlomo Geva le 29 Oct 2017
Hello all, when I was trying for >>floor(2.3 * 50) , I was getting 114 where I suppose to get 115. Is this due to precision issue? and How to fix it?
  1 commentaire
Shlomo Geva
Shlomo Geva le 29 Oct 2017
Well, check this code: floor(p/65536) floor() does not handle correctly uint32 values of p > 4294934528 which is well within uint32 range. It is not even returning a value in the expected range 0 to 65535
>> p=uint32([4294934527, 4294934528, 2^32-1]);floor(p/65536)
ans =
1×3 uint32 row vector
65535 65536 65536

Connectez-vous pour commenter.

Réponses (2)

Sebastian
Sebastian le 6 Fév 2017
If you have the Symbolic Math Toolbox, try this:
x = vpa(2.3 * 50)
x =
115.0
x = floor(x)
x =
115

Jan
Jan le 6 Fév 2017
Yes, this is an effect of the limited precision of the IEEE754 floating point standard. See http://matlab.wikia.com/wiki/FAQ#Why_is_0.3_-_0.2_-_0.1_.28or_similar.29_not_equal_to_zero.3F . You find a lot of corresponding question, when you search e.g. for "faq 6.1".

Community Treasure Hunt

Find the treasures in MATLAB Central and discover how the community can help you!

Start Hunting!

Translated by