| |||||||||||
Accuracy Of Tcl Calculations Posted by Graham_Tasker (Graham_Tasker), 8 March 2005 Graham : Back in December I can remenber that we discussed the accuracy of Tcl Calculations, but I cannot remenber what was said, so this question : All Tcl data is stored as strings, so when you do a calculation using the expr command that data is first converted into a numeric type. But what 'type'? 1.234 would be converted to a float, but would it be a long float (64 bits) or a float (32 bits)? Would 123 be converted to an Integer or a Float? What about the convertion back, who many digits would be written to the string? How much accuracy is lost during this process? This quetion relates back to the problem is representing any floating point number on a computer. Thanks : Graham Tasker Posted by admin (Graham Ellis), 8 March 2005 It's using double precision floats internally, but the conversion to and from strings degrades it from that a little - see the following example.Code:
If all of the inputs to en expr are integers, integer arithmetic is performed. So 123 will only be converted to a float if the other value in a division / multiply / etc is a float. Graham This page is a thread posted to the opentalk forum
at www.opentalk.org.uk and
archived here for reference. To jump to the archive index please
follow this link.
|
Other Forum Posts Follow us on ... | ||||||||||
PH: 01144 1225 708225 • FAX: 01144 1225 793803 • EMAIL: info@wellho.net • WEB: http://www.wellho.net • SKYPE: wellho |