error decimal aproximation, extra decimal
3 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
why in Matlab 0.94-1=-0.0600000000000001? Why is there an extra decimal 1?
0 Kommentare
Antworten (1)
James Tursa
am 11 Mai 2020
Bearbeitet: James Tursa
am 11 Mai 2020
Welcome to the world of floating point arithmetic! See this link:
Bottom line ... 0.94 cannot be represented exactly in IEEE double precision. That trailing 1 is actually part of more digits in an exact floating point binary to decimal conversion:
>> fprintf('%.60f\n',0.94-1)
-0.060000000000000053290705182007513940334320068359375000000000
The 0.94 all by itself converted into decimal is:
>> fprintf('%.60f\n',0.94)
0.939999999999999946709294817992486059665679931640625000000000
0 Kommentare
Siehe auch
Kategorien
Mehr zu Get Started with MATLAB finden Sie in Help Center und File Exchange
Produkte
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!