WeBWorK Main Forum

formula marked wrong sporadically

formula marked wrong sporadically

by Gavin LaRose -
Number of replies: 5

I have a problem with the following code: 

  Context("Numeric");
  Context()->flags->set(reduceConstants=>0);
  Context()->variables->add(s=>"Real");
  Context()->variables->set(s=>{limits=>[0,10]});

  $ans = "(1/5)*s^(5) - 2*s^(3) + 8*s + C";
  $fans = FormulaUpToConstant($ans);

  BEGIN_TEXT
  answer prompt: \{ans_rule(55)\}
  END_TEXT

  ANS( $fans->cmp() );

Today I found a student who submitted the answer s^5/5-2s^3+8s+c for this problem and had a recorded score of zero. Resubmitting it without changing the answer I am unable to get WeBWorK to check it as anything other than correct. As an experiment, I've tried specifying test points to see if I can find some set that give non-zero errors, but haven't succeeded.

That is, WeBWorK evidentally managed to find some test point at which, for reasons that are not obvious to me, the student's and the correct answers disagreed to the required precision.

My first reaction is that this is more than a little disturbing. Any insight on what could be going on would be greatly appreciated. As a side note, the problem is used in a gateway/quiz assignment, but I wouldn't expect that to have an impact on the answer evaluation.

Thanks,
Gavin

In reply to Gavin LaRose

Re: formula marked wrong sporadically

by Paul Pearson -
Hi Gavin,

Try using variables->are() rather than variables->add(), such as
Context()->variables->are(s=>"Real");

The answer checker should be more reliable by restricting the variables in the domain from {x, s, adaptive parameter} to {s, adaptive parameter}. In particular, the algorithm that finds the values for the adaptive parameter should be much more reliable when there is only one independent variable.

Also, you may want to look into adding test points for function evaluation. I think this is possible to do with FormulaUpToConstant() MathObjects, but I'd like to hear what Davide Cervone has to say on the topic since I'm not 100% certain about how it would be done.

Best regards,

Paul


In reply to Paul Pearson

Re: formula marked wrong sporadically

by Davide Cervone -
Paul:

You should be able to set the test_at or test_points values for the FormulaUpToConstant() object directly, and the internals will use them for the testing. At least that is my recollection, and a quick view of the code suggests that is the case (though it is a pretty complicated object, and so things are not always as clear as they might be).

Davide
In reply to Gavin LaRose

Re: formula marked wrong sporadically

by Davide Cervone -
Does the problem also include any non-standard graders? I recently had a student complain that WeBWorK hadn't recorded their correct answer, and it turned out that it was due to a flaw in a grader that the problem included.

Did the past answers show the answer as marked correct (i.e., in green) for then the student submitted it?

By "resubmitting" was that the "check" button or the "submit for student" button?

Davide
In reply to Gavin LaRose

Re: formula marked wrong sporadically

by Alexander Basyrov -
Hi Gavin,

I suspect that the algorithm that finds the value of the arbitrary parameter is numeric in its nature. With the values of s being in interval [0, 10] and the correct answer containing s^5 it seems to be possible for webwork to select the values of s that do not result in a good numeric result for the arbitrary parameter C.

You might want to restrict the 's' interval to, say, [0, 2] to see how that changes things.

-- Alex
In reply to Alexander Basyrov

Re: formula marked wrong sporadically

by Gavin LaRose -
Hi Paul, Davide, Alex, et al.,

Thanks for the good comments and suggestions. I have a suspicion that the problem comes from some domain issue having to do with the adaptive parameter, which is consistent with your ideas, but don't have good data to support that. Specific responses to some of the questions here:

  • I tried explicitly setting test points in the problem to extreme values, and was unable to get anything that showed a non-zero difference between the student and correct answers (other than is noted below);
  • The independent variables as written (and with the answer submitted) are {C, c, s, x}, plus the adaptive parameter; it may be that eliminating one of the variables would help, but it doesn't seem as if that should be necessary;
  • The problem doesn't use any non-standard graders, and the past answer is reported as incorrect. I've been testing it since with the "check answers" button.

The only time when I see (inconsequential, but non-zero) differences between the student's and correct answers is with the adaptive parameters, which can have (inconsequential) errors that result in differences of on the order of 10^(-12).

The biggest issue is, of course, that this is for all intents and purposes an irreproducible error: I don't know what test points were used on the submission that was marked incorrect. [Side note: in that the test points vary between submissions now, it would be nice to have them logged somewhere; maybe in the answer log.]

I'd love to hear additional thoughts on this.

Thanks
Gavin