WeBWorK Main Forum

Answer checker rejects correct answer 14 times

Answer checker rejects correct answer 14 times

by Dave Rosoff -
Number of replies: 6
. . . and then accepts on the 15th click. I couldn't believe my student who came to me with this story until I printed out her past answers. The visual evidence is so wacky that I scanned and uploaded it. You can view the past answers (student's name is redacted, of course) here. The correct set of answers for this problem is 

CONTINUOUS; X < 7.5; 0.1469; 0.1469; 0.7062

This correct set of answers was rejected 14 times, as you can see, before WeBWorK accepted her 15th click. Note the timestamps on the submissions. If you look at the entire problem history (not just the last 15 clicks) you can see that she has been trained to expect this behavior. WeBWorK changed its mind in the other direction as well, rejecting correct answer-parts that had been accepted in previous submissions.

Here is the entire source of the offending problem. I have heard reports of this behavior, not all of them as believable, on other problems as well. Any advice or suggestions you all might be able to give will be very greatly appreciated. Thanks!

Dave

## DESCRIPTION
## DBsubject('Introduction to Statistics')
## DBchapter('Part II From Exploration to Inference')
## DBsection('Chapter 10: Introducing Probability')
## KEYWORDS('statistics', 'probability')
## TitleText1('The Basic Practice of Statistics')
## EditionText1('5e')
## AuthorText1('David Moore')
## Section1('Chapter 10: Introducing Probability')
## Problem1('')
## Author('R Cruz')
## Institution('The College of Idaho')
## Date: 2009/03

DOCUMENT();        # This should be the first executable line in the problem.

loadMacros(
  "PGstandard.pl",
  "MathObjects.pl",
  "PGstatisticsmacros.pl", 
  "answerHints.pl",
  "contextInequalities.pl"
);

TEXT(beginproblem());

######################################
#  Setup

Context("Inequalities-Only");
Context()->variables->add(X=>'Real');

$mu = random(7, 10, 0.1);
$sigma = random(1, 2, 0.1);
$length1 = $mu - random(1,3,1);
$length2 = $mu + random(1,3,0.5);

$zscore1 = int(100 * (-0.005 + ($length1 - $mu) / $sigma))/100;
$zscore2 = int(100 * (0.005 + ($length2 - $mu) / $sigma))/100;
$pval1 = 1-uprob(abs($zscore1));
$pval2 = 1-uprob(abs($zscore2));

@event = ("a fish chosen has a length of over $length1 inches",
 "a fish chosen has a length of less than $length1 inches",
 "a fish chosen has a length equal to $length1 inches");
@ans_event = (Compute("X>$length1"), Compute("X<$length1"), 
             Compute("X=$length1"));
@ans_prob = ($pval1, 1-$pval1, 0.0);
$choose = random(0,2,1);

######################################
#  Main text

BEGIN_TEXT
The length, \(X\), of a fish from a particular mountain lake in Idaho is normally
distributed with \(\mu = $mu\) inches and \(\sigma = $sigma\) inches.

$PAR
(a)  Is \(X\) a discrete or continuous random variable? (Type: DISCRETE or CONTINUOUS)
$BR
ANSWER: \{ans_rule(20)\}

$PAR
(b)   Write the event ''$event[$choose]'' in terms of \(X\): \{ans_rule(20)\}.

$PAR
(c)  Find the probability of this event: \{ans_rule(10)\} $PAR

$PAR
(d)    Find the probability that the length of a chosen fish was greater than $length2 inches: \{ans_rule(15)\}.

$PAR
(e)    Find the probability that the length of a chosen fish was between $length1 and $length2 inches: \{ans_rule(15)\}.
END_TEXT

######################################
#  Answers  %%%THESE NEED CHANGING

$showPartialCorrectAnswers = 1;

Context()->strings->add("DISCRETE"=>{},"CONTINUOUS"=>{});

$ans_a = String("CONTINUOUS");
ANS($ans_a->cmp);

##Answer checker not working as expected.  It uses this line instead of the one in the checker. 
##Had to add these lines to get it to give the desired hint.

Context()->{error}{msg}{"Operands of '*' are not of compatible types"} 
                         = "Enter the event: X = a, X < b, X > c, d < X < e";
Context()->variables->add(P=>'Real',p=>'Real');

$ans_b = $ans_event[$choose];
ANS($ans_b->cmp->withPostFilter(AnswerHints(
    sub {
         my($correct,$student,$ansHash) = @_;
         $ansHash->{ans_message} = "nter the event: X = a, X < b, X > c, d < X < e" 
        if $ans->{ans_message} eq 
        "Operands of '*' are not of compatible types"; return $ansHash;}
))); 

$ans_c = Real($ans_prob[$choose])->with(tolType=>'absolute', tolerance=>'0.0005');
ANS($ans_c->cmp->withPostFilter(AnswerHints(
    sub {
         my($correct,$student,$ansHash) = @_;
         return abs($correct - $student) <= 0.003;
         } => ["Close! Try keeping more accuracy in your calculation.", score => 0],
     sub {
         my ($correct,$student,$anshash) = @_;
         return $student < 0 || $student > 1;
         } => ["Probabilities must be between 0 and 1."]
))); 

$ans_d = Real(1-$pval2)->with(tolType=>'absolute', tolerance=>'0.0002');
ANS($ans_d->cmp->withPostFilter(AnswerHints(
    sub {
         my($correct,$student,$ansHash) = @_;
         return abs($correct - $student) <= 0.003;
         } => ["Close! Try keeping more accuracy in your calculation.", score => 0],
     sub {
         my ($correct,$student,$anshash) = @_;
         return $student < 0 || $student > 1;
         } => ["Probabilities must be between 0 and 1."]
)));

$ans_e = Real($pval1 - (1-$pval2))->with(tolType=>'absolute',tolerance=>'0.0002');
ANS($ans_e->cmp->withPostFilter(AnswerHints(
    sub {
         my($correct,$student,$ansHash) = @_;
         return abs($correct - $student) <= 0.003;
         } => ["Close! Try keeping more accuracy in your calculation.", score => 0],
     sub {
         my ($correct,$student,$anshash) = @_;
         return $student < 0 || $student > 1;
         } => ["Probabilities must be between 0 and 1."]
)));

ENDDOCUMENT();       # This should be the last executable line in the problem.


In reply to Dave Rosoff

Re: Answer checker rejects correct answer 14 times

by Davide Cervone -
Your scan seems to be a black-and-white one. Can you do a color scan so that we can see which answers were marked correct and which incorrect? Thanks.

I have my suspicions that it is the answers hints that are the culprit but I will need to investigate a bit before I can say for sure.

Davide
In reply to Davide Cervone

Re: Answer checker rejects correct answer 14 times

by Dave Rosoff -
Hi Davide,

Thanks for your reply. Here's a screen cap (I don't know why I wanted to print and scan before, or why I didn't notice the Insert Image button).

vUBqW

Said button doesn't seem to be working. Here is the hosted image. Thanks for looking into this. I just spoke with another prof who has had similar problems but in a different course. She said her problems may also have had hints, so I hope you are onto something!

Best,
Dave
In reply to Dave Rosoff

Re: Answer checker rejects correct answer 14 times

by Davide Cervone -
Thanks for the updated image. Pretty strange stuff.

A couple of questions:

1. Are you aware of this in any other problems, or just this one?

2. Is this happening for other students on this same problem?

3. Have you yourself been able to reproduce this behavior yourself?

4. When the answers are not accepted, does one of the answer hints show up?

5. Can you provide the seed for this student's version of the problem?

I'll check into it when I get the chance, but it may not be immediately.

Davide
In reply to Davide Cervone

Re: Answer checker rejects correct answer 14 times

by Dave Rosoff -
Hi Davide,

I answer your questions below.

>1. Are you aware of this in any other problems, or just this one?

In other problems as well. 

>2. Is this happening for other students on this same problem?

Unclear. I worry that it may be, but they are not confident enough to keep clicking and just give up. Then it would naturally not be reported.

>3. Have you yourself been able to reproduce this behavior yourself?

Yes. Not the exact pattern of acceptance and rejection, but similar flipping of correctitude and incorrect marking.

>4. When the answers are not accepted, does one of the answer hints show up?

I will try to find out.

>5. Can you provide the seed for this student's version of the problem?

Yes, it is 179.

>I'll check into it when I get the chance, but it may not be immediately.

Of course. Thanks again for your assistance. I've been losing sleep over this for a few weeks. I'll try to get back to you soon with info about the answer hints.

Dave

In reply to Dave Rosoff

Re: Answer checker rejects correct answer 14 times

by Dave Rosoff -
Hi Davide,

Answer hints are, as far as I know, present in all of the problems that are misbehaving. Thanks again for any insight you can provide into this.

Dave
In reply to Dave Rosoff

Re: Answer checker rejects correct answer 14 times

by Dave Rosoff -
Hi Davide, and anyone else who might be listening,

I'm now able to report that this is happening with problems that do not have answer hints. My only guess is that there is some kind of collision happening; the SQL server, say, might be busy when another request comes in, and this behavior is the result? It doesn't seem very likely, but then again neither does this entire scenario. I hope very much that you can spare a moment to provide some insight. Thanks again for taking the time.

Best,
Dave