WeBWorK Problems

Changing Precision in Answer Checker

Changing Precision in Answer Checker

by Jim Fischer -
Number of replies: 8

Is there still an easy way to change the “required” default precision of 4-5 digits (OR equivalent percent accuracy) in WW to something less stringent, like 2 digits?


Would the modification be different depending on Math Objects or other code?

In reply to Jim Fischer

Re: Changing Precision in Answer Checker

by Arnold Pizer -
Hi Jim,

The default precision is .1 meaning .1 percent which is the allowed relative error.  If you want to change this for a course, use "Course Configuration" and select "PG - Problem Display/Answer Checking" and adjust "Allowed error, as a percentage, for numerical comparisons".

If you want to change this for all courses on your system, edit localOverrides.conf adding a line 
$pg{ansEvalDefaults}{functRelPercentTolDefault} = .1;
where you should change .1 to whatever value you want (e.g. 1).

This works for all numerical answers larger than the "zeroLevel" (10^{-14}?).  

Arnie
In reply to Jim Fischer

Re: Changing Precision in Answer Checker

by Alex Jordan -
If it's a MathObject based problem, and you are happy to only modify that problem, then the context you are using has a flag called tolType that you can set to either absolute or relative:

Context()->flags->set(tolType=>absolute);
Context()->flags->set(tolType=>relative);

There is a second flag, tolerance, that you set to a numerical value:

Context()->flags->set(tolType=>0.01);

And this works as you would think: answers can be within +/- 0.01 if the tolType is absolute, or within */÷ 1.01 if the tolType is relative.

Now, using relative tolerance and a power of 10 is close to asking for so many significant digits, but the demand is stronger on answers with low leading digits than with high. For example, if tolType is relative and tolerance is 0.001, then:
  • If an answer is Real(1.004), then "1" will not be accepted as correct; roughly anything from 1.003 to 1.005 would be correct. So it's almost like demanding four significant digits.
  • But if an answer is Real("9.004"), then "9" is counted as correct. Even Real("9.009") would allow "9" from a student. So at this extreme you can sometimes get away with only being correct to two significant digits.
So it's not exactly the same as saying you demand four significant digits. It would be interesting to see if a tolType=>sigfig could be developed, where tolerance would be the required number of significant digits and work exactly as you'd want it
In reply to Alex Jordan

Re: Changing Precision in Answer Checker

by Alex Jordan -
Also, you can just set these flags for one particular answer checker rather than the whole context, by putting the flag assignments in as arguments to cmp. For example:

ANS($ans->cmp(tolType=>absolute,tolerance=>0.5));
In reply to Alex Jordan

Re: Changing Precision in Answer Checker

by Jim Fischer -
Thanks for all the suggestions. I think it would have helped if I gave more context to the situation. We have statistics instructors who are having some issue with precision in problems due to the difference between using tables and using formulas to produce answers. So far, the above suggestions are not working. Below is a sample problem that we would like to adjust so that a student will get the green bar of glory regardless of whether they use table or formula. We think that about 2 digits accuracy would suffice.


----------------------------------------------------------------------

##DESCRIPTION
## DBsubject('Statistics')
## DBchapter('Continuous Random Variables and Their Probability Distributions')
## DBsection('The Normal Probability Distribution')
## KEYWORDS('')
## Author('N.Spencer Sitton')
## Institution('NAU')
##ENDDESCRIPTION

# File Created: 07/15/05; Modified 08/18.10
# modified by J. Hagood
# Location: Northern Arizona University

DOCUMENT();

loadMacros("PGstandard.pl",
"PGnauGraphics.pl",
"PGnauStats.pl"
);

TEXT(&beginproblem);

@list = ( 'Length of skateboards in a skateshop', 'The shelf life of a batteries produced by one major company',
'Fluorescent lighbulbs have lifetimes that ', 'Scores on a standardized exam',
'Length of snowboards in a boardshop' );
@list2 = ('the length of skateboards in a skateshop', 'shelf lives of batteries produced by one major company',
'lifetimes of fluorescent lightbulbs', 'the scores on a standardized exam',
'the length of snowboards in a boardshop' );
@mean = ( random( 30, 32, .1 ), random( 3, 5, .1 ), random( 5, 7, .1 ),
random( 500, 550, 1), random( 150, 153, .1 ) );
@sd = ( random( .1, 1, .1 ), random(.1, 1, .1 ), random(1, 1.6, .1 ),
random( 50, 100, 1 ), random( .1, 1, .1 ) );
$ind = random( 0, 4, 1 );
@unit = ( 'in', 'years', 'years', '', 'cm' );
$shade = list_random( 0, 1 );
$fact = list_random( -1, 1 );
@val = ( .67 * $sd[$ind], $sd[$ind], 2 * $sd[$ind] );
@zscore = ( .67, 1, 2 );
@prop = ( .25, .34, .475 );
$s1 = list_random( 0, 1 );
push @slice, $s1, 2;
$a1 = $prop[$slice[1]];
$a2 = $prop[$slice[0]];
$z = $val[$slice[1]];
$y = $val[$slice[0]];
$z1 = $zscore[$slice[1]];
$z2 = $zscore[$slice[0]];

if( $fact == -1 ){$a = $mean[$ind] - $z;
$b = $mean[$ind] - $y;
$newz1 = -$z1;
$newz2 = -$z2;
if( $shade == 0 ){
$ans = $a1 - $a2;
}else{$ans = 1 - ( $a1 - $a2 );
}
}else{ $a = $mean[$ind] + $y;
$b = $mean[$ind] + $z;
$newz1 = $z2;
$newz2 = $z1;
if( $shade == 0 ){
$ans = $a1 - $a2;
}else{$ans = 1 - ( $a1 - $a2 );
}
}

@tmp = split ' ', $list2[$ind];
$tmp[0] = ucfirst $tmp[0];
for($i = 1; $i < scalar @tmp; $i++){
$val = $tmp[$i];
if (length $val > 2 && $val ne 'the'){
$val = ucfirst ($val);
$tmp[$i] = $val;
}
}

$title = join ' ', @tmp;

$diagram = DrawNormalDist( $newz1, $newz2, $a, $b, outside => $shade, title => $title );

BEGIN_TEXT

$list[$ind] are normally distributed with a mean of $mean[$ind] $unit[$ind] and a standard deviation of
$sd[$ind] $unit[$ind]. The figure below shows the distribution of $list2[$ind].
Calculate the shaded area under the curve. $BBOLD Express your answer in decimal form accurate to at least two decimal places. $EBOLD $BR
Answer: \{ans_rule( 15 )\}
$PAR
\{Plot($diagram, tex_size=>900)\}

END_TEXT

ANS( num_cmp( $ans ));

ENDDOCUMENT();








In reply to Jim Fischer

Re: Changing Precision in Answer Checker

by Alex Jordan -
Ah! I used to have this problem too, but have since just told my students to stop using tables :)

Here's something that could work. Behind the scenes, code the problem to produce multiple answers: $tableBasedAns and $techBasedAns

Then load parserOneOf.pl, and use
ANS(OneOf($tableBasedAns,$techBasedAns)->cmp());
In reply to Alex Jordan

Re: Changing Precision in Answer Checker

by Gavin LaRose -
Hi all,

For what it's worth, for this type of problem I've also just worked out what the answer is from the table that my students are most likely to use and set the precision required so that it allows that answer.

Gavin

In reply to Gavin LaRose

Re: Changing Precision in Answer Checker

by Jim Fischer -
Thanks again for the suggestions. Using formulas wont work as the instructors want the students to use the tables.

We did use Arnie's suggestion for a course-wide change and that helped some. However, the instructors would prefer to edit those problems individually and retain accuracy or precision for other types of problems in the course.

So with the above problem I posted, what specifically should we write into the code to accomplish our task?


In reply to Jim Fischer

Re: Changing Precision in Answer Checker

by Alex Jordan -
Another issue that statistics instructors will encounter is when, say, questions are about a 95% confidence interval or involve a z-score of 2. Some questions programmed in the OPL use the Empirical Rule (68-95-99.7 Rule), and some use table- or erf-based answers.

Questions about the tail area of a Normal curve are particularly bad because the Empirical Rule puts 2.5% in a tail when z=2, but actually it's 2.275...%. So that's quite an error tolerance you'd be trying to universally impart. If you did so successfully, you'd probably have a hugely unnecessary tolerance in other versions of the problem.

As I see it, in a lot of these stats problems, there are often three answers that students might come to:
  • one based on a Normal table (where rounding has occurred)
  • one based on exact values using a calculator or stats software
  • sometimes, one based on the Empirical Rule

If an instructor has no preference, they should calculate all three in the code for the problem, and use the OneOf construct from my previous post.

If an instructor has a preference, a great approach would be to still calculate all three possibilities, then use the preference as "the" answer, and then use answerHints.pl to give custom feedback for the other answers. I'm not sure it's quite right for the student to be told they are outright wrong when they submit one of the other two approaches, which can leave them feeling confused if they followed a textbook or online example without error. With answerHints.pl, they could get a feedback message like "Ah, it looks like you used technology to find your answer, but I want you to do this kind of problem using the Normal table."

Here's an example of such a problem, with all the constructs. I'm afraid if your instructors are modifying OPL problems, the variety of coding techniques used in OPL stats problems is wide, and they'll have to treat them as they come. But maybe the pieces here can be cut and adapted. Also, this uses Davide Cervone's contextPercent.pl, which I believe is still not distributed, so I've attached it.

DOCUMENT();

loadMacros(
"PGstandard.pl",
"MathObjects.pl",
"PGML.pl",
"answerHints.pl",
"PGnumericalmacros.pl",
"PGstatisticsmacros.pl",
"contextPercent.pl",
"parserOneOf.pl",
"PGcourse.pl",
);

##############################################

Context("Percent"); #answers may be entered as "0.3" or "30%" for example; by default error tolerance in this context is absolute to 0.0005=0.05%. Below, based on the answer, I'll change it to something appropriate given the answer.

$mu = random(65, 75, 1);
$sigma = random(4, 10, 1);

do {
$examscore = random(60,80,1);
$z = ($examscore - $mu) / $sigma;
} until (abs($z) > 0.2);


$techAns = normal_prob(-infty, $z, mean=>0, deviation=>1);

#set context settings for an appropriate tolerance and number of displayed decimals
Context()->flags->set(tolerance=>abs(10**floor(log($techAns)/log(10))/200));
Context()->flags->set(decimalPlaces=>abs(floor(log($techAns)/log(10))));

$roundedz = round($z*100)/100;

$tableAns = normal_prob(-infty, $roundedz, mean=>0, deviation=>1);

$empAns = $ableAns; #in case the Empirical Rule doesn't apply
if ( grep( $roundedz, (-3, -2, -1, 0, 1 ,2 , 3) ) )
{%emp = (-3 => 0.0015,
-2 => 0.025,
-1 => 0.16,
0 => 0.5,
1 => 0.84,
2 => 0.975,
3 => 0.9985);
$empAns = $emp{$roundedz};
};


$allInclusinveAns = OneOf(
Percent($techAns),
Percent($tableAns),
Percent($empAns)); #in case an instructor is not picky, use this in the ANS call

##############################################
TEXT(beginproblem());

BEGIN_PGML

Scores from an exam taken by thousands of people were Normally distributed with mean [`[$mu]`] and standard deviation [`[$sigma]`]. One person scored a [`[$examscore]`]; what percentage of test takers did they score higher than?

[____________]


END_PGML
##############################################

#if the instructor is not picky
#ANS($allInclusinveAns->cmp());

#if say, the instructor wants the table-based answer
ANS(Percent($tableAns)->cmp()->withPostFilter(AnswerHints(
Percent($tableAns) => "Correct!",
#give feedback for correct first, or else one of the later messages might appear even when correct
Percent($techAns) => "It looks like you used technology; use the table of Normal values instead.",
Percent($empAns) => "It looks like you used the Empirical Rule; use the table of Normal values instead.",
)));


##############################################

BEGIN_PGML_SOLUTION

Walk the student through a solution here...

END_PGML_SOLUTION

ENDDOCUMENT();