DeskCalc: Limit precision of large magnitude results

This targets a problem where a numbers with large numbers of non-decimal
significant digits took a long time to round after converting to scientific
notation because they are rounded one character a time.

To solve this, after converting to scientific notation lop off everything
after 40 characters greatly reducing the amount of further rounding
needed.

An example I used to test this was to calculate 10,000! which gives
a result with 35660 significant non-decimal digits (aka a lot). By loping
off numbers after 40 characters before rounding to fit the operation goes
from ~10 seconds to complete to under a second.

I chose 40 as a max as it is large enough to ensure that the result will
get rounded with some leeway provided for font width variations.
Worse-case scenario is the result is off by 1 in the last place.

Numbers with large numbers of significant decimal digits get rounded
by MAPM so aren't a problem.
This commit is contained in:
John Scipione
2013-06-26 19:19:21 -04:00
parent f9d2a8cb25
commit 31650fd465
+8 -3
View File
@@ -244,10 +244,15 @@ ExpressionTextView::SetValue(BString value)
}
}
// add the exponent
offset = value.CountChars() - 1;
if (exponent != 0)
if (exponent != 0) {
value.Truncate(40);
// truncate to a reasonable precision
// while ensuring result will be rounded
offset = value.CountChars() - 1;
value << "E" << exponent;
// add the exponent
} else
offset = value.CountChars() - 1;
// reduce the number of digits until the string fits or can not be
// made any shorter