WeBWorK Problems

significant digits

Re: significant digits

by Alex Jordan -
Number of replies: 0
OK, but be careful about tolerance=>0. At a minimum, only use it this way if you explicitly define the answer to be a terminating decimal. If you defined $ans to be something like "0.2/0.4", then in binary with machine rounding that might be "0.001100/0.011001" (with the rounding happening much further down the line) which is only 0.48 in decimal.

So in this example, you might be sure that you are defining the answer to be 0.5, but it comes out as something different. Usually the tolerance being greater than 0 absorbs this issue. But with tolerance=>0, the student would type in 0.5 and be marked wrong.