Double Precision is rounding at the 5th digit?
4 Ansichten (letzte 30 Tage)
Ältere Kommentare anzeigen
Brian Rose
am 24 Apr. 2019
Bearbeitet: Stephen23
am 24 Apr. 2019
(Newbie question)
I am trying to do some basic math and I am running into what appears to be a rounding or precision problem. MATLAB uses doubles, which should be good out to 15 digits. It appears that MATLAB is rounding at less than 4. What am I missing? Note: I do not have access to the Symbolic toolkit, and frankly can't understand why any additional code would be needed for this basic equaiton.
value1 = double(0.999969482421875)
eps(value1)
value2 = double(32767) ./ double(32768)
Output is
value1 =
1.0000
ans =
1.1102e-16
value2 =
1.0000
0 Kommentare
Akzeptierte Antwort
Weitere Antworten (0)
Siehe auch
Kategorien
Mehr zu Numeric Types finden Sie in Help Center und File Exchange
Produkte
Community Treasure Hunt
Find the treasures in MATLAB Central and discover how the community can help you!
Start Hunting!