Accuracy Of Tcl Calculations
Posted by Graham_Tasker (Graham_Tasker), 8 March 2005Graham :
Back in December I can remenber that we discussed the accuracy of Tcl Calculations, but I cannot remenber what was said, so this question :
All Tcl data is stored as strings, so when you do a calculation using the expr command that data is first converted into a numeric type. But what 'type'? 1.234 would be converted to a float, but would it be a long float (64 bits) or a float (32 bits)? Would 123 be converted to an Integer or a Float? What about the convertion back, who many digits would be written to the string? How much accuracy is lost during this process?
This quetion relates back to the problem is representing any floating point number on a computer.
Thanks : Graham Tasker
Posted by admin (Graham Ellis), 8 March 2005It's using double precision floats internally, but the conversion to and from strings degrades it from that a little - see the following example.
If all of the inputs to en expr are integers, integer arithmetic is performed. So 123 will only be converted to a float if the other value in a division / multiply / etc is a float.
Other Forum Posts
Follow us on ...
PH: 01144 1225 708225 • FAX: 01144 1225 793803 • EMAIL: email@example.com • WEB: http://www.wellho.net • SKYPE: wellho