WeBWorK Main Forum

num_cmp Mystery

Re: num_cmp Mystery

by Davide Cervone -
Number of replies: 0

I think Davide could shed some light on this situation.

Here's how the zero-level values work: when $a is compared to $b, if either value is below the $zeroLevel value, which is 1E-14 by default, then if the difference between the two is less than $zeroLevelTol, which is 1E-12 by default, then they are equal. That is, if abs($a) < $zeroLevel || abs($b) < $zeroLevel, then the two are considered equal if abs($a - $b) < $zeroLevelTol, and are not equal otherwise.

In the case discussed here, the correct value is given by pi/6*500*cos(pi/2), which evaluates to 1.60305891143483e-14, and the test value is 500*cos(pi/2), which is 3.06161699786838e-14, neither of these is less than $zeroLevel, so the usual relative tolerance rules are used, and they are not equal by that measure (since they would have to agree on their first 3 or 4 significant digits).

On the other hand, when testing against 100*cos(pi/2), which is 6.12323399573677e-15, this value is less than $zeroLevel, and so the rules above come into play, and we must compare the difference, which is -9.90735511861148e-15, to the $zeroLevelTol of 1E-14 in absolute value, and since it is less than the tolerance, these are considered equal.

When testing against the answer of 0 itself, since that is less than $zeroLevel, the zero-level rules are used, and since the difference between pi/6*500*cos(pi/2) and 0 is 1.60305891143483e-14, and that is less than the $zeroLevelTol of 1E-12, these are also equal.

Finally, for 260*cos(pi/2), which is 1.59204083889156e-14, neither is smaller than $zeroLevel, and so the usual relative tolerance is used, and this don't match 1.60305891143483e-14 to enough digits, so they aren't equal. On the other hand, 262*cos(pi/2) does match to enough digits, so that does match pi/6*500*cos(pi/2).

So that is how it works, and why these examples do what they do.