calculation error for floor function
9 vues (au cours des 30 derniers jours)
Afficher commentaires plus anciens
Hello all, when I was trying for >>floor(2.3 * 50) , I was getting 114 where I suppose to get 115. Is this due to precision issue? and How to fix it?
1 commentaire
Shlomo Geva
le 29 Oct 2017
Well, check this code: floor(p/65536) floor() does not handle correctly uint32 values of p > 4294934528 which is well within uint32 range. It is not even returning a value in the expected range 0 to 65535
>> p=uint32([4294934527, 4294934528, 2^32-1]);floor(p/65536)
ans =
1×3 uint32 row vector
65535 65536 65536
Réponses (2)
Sebastian
le 6 Fév 2017
If you have the Symbolic Math Toolbox, try this:
x = vpa(2.3 * 50)
x =
115.0
x = floor(x)
x =
115
0 commentaires
Jan
le 6 Fév 2017
Yes, this is an effect of the limited precision of the IEEE754 floating point standard. See http://matlab.wikia.com/wiki/FAQ#Why_is_0.3_-_0.2_-_0.1_.28or_similar.29_not_equal_to_zero.3F . You find a lot of corresponding question, when you search e.g. for "faq 6.1".
0 commentaires
Voir également
Catégories
En savoir plus sur Numbers and Precision dans Help Center et File Exchange
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!