WeBWorK Problems

Issue with tolerance on answer checking

Issue with tolerance on answer checking

by Paul Seeburger -
Number of replies: 2
Ok, I believe that the large numbers in this expression are causing it to not reliably mark incorrect answers.

The expression is ".1(10000+x)" and the number "1000" was marked as correct for this expression.

Why was this the default behavior, and how can I make sure that at least an x was included in the answer?

Would adding some large test values do the trick?

Here's the code:

$rhs = Compute("$dnr*($rInit + x)");

ANS( $rhs -> cmp());

Ok, I've actually gone out and tested this idea about adding a test point and it seemed to work, but I am still not sure why it needed me to do this here.  When should I be sure to add test points?

Here's the code that made it realize that "1000" was incorrect in this case.

$rhs = Compute("$dnr*($rInit + x)")->with(test_at => [[10000]]);

ANS( $rhs -> cmp());

Thanks!
In reply to Paul Seeburger

Re: Issue with tolerance on answer checking

by Paul Pearson -
Hi Paul,

You can check how the answer evaluator is working by setting the diagnostics flag to be true:

ANS( $answer->cmp(diagnostics=>1) );

In some cases, you can use this information to help you choose an appropriate domain for function evaluation:

http://webwork.maa.org/wiki/FormulaTestPoints

A better approach is probably to think about the problem as if you were in a real analysis course (i.e., think in terms of deltas and epsilons) and set the domain and tolerance accordingly.

The default domain for variables in the context is [-2,2], by default five randomly selected points are chosen from the domain [-2,2] when functions are evaluated, and the default error (as a percentage) for numerical comparisons is 0.1.  This means that the value of the student's answer has to be within 0.1 percent (or 1/1000) of the correct answer for each of the randomly chosen points in the domain.  In webwork lingo, the default tolerance is 'relative' with value 0.1, and this default can be changed in the course configuration menu (although I do not advise it).  So, if the defaults are used and the correct answer is the number 1000, then the student answer must be in the interval 1000 +/- 0.001*1000 or equivalently in [999,1001].  If the correct answer is the constant function 1000, then the student's answer is compared to the correct answer for five random points in the domain.

If an 'absolute' tolerance of 0.01 is used when the correct answer is 1000, then the student answer must be in the interval 1000 +/- 0.01 or equivalently in [999.99,1000.01].

Over the domain [-2,2], the expression 0.1(10000+x) takes values between 999.8 and 1000.2.  The default numerical tolerance for the value 999.8 at the left endpoint of the domain is the interval [998.8,1000.8].  The default numerical tolerance for the value 1000.2 at the right endpoint of the domain is [999.8,1001.8].  If the student enters 1000, then their answer is always in the envelope bounded below by the line connecting the point (-2,998.8) to (2,999.8) and bounded above by the line connecting the point (-2,1000.8) to (2,1001.8).  Thus, the student answer 1000 is marked correct no matter what five points are randomly chosen from the domain.

I would recommend enlarging the domain or using an absolute numerical tolerance when the answer is something like 0.1(10000+x). There are certainly other good solutions to this problem.

Best regards,

Paul Pearson
In reply to Paul Pearson

Re: Issue with tolerance on answer checking

by Alex Jordan -
Shifting the domain to be centered around -10000 might be a good idea too, for the same reasons.