WeBWorK Problems

Can I turn off comparison of an answer to the previous submitted answer?

Can I turn off comparison of an answer to the previous submitted answer?

by Alex Jordan -
Number of replies: 7
We have developed a lot of custom answer checkers for pre-college algebra problems. I'll say first, that it is quite possible that these custom checkers simply have not been programmed in the best way possible. But putting that aside, the stage where the answer checker compares an answer to the previous submitted answer is causing issues for us. My question is just this: Is there a way to turn that comparison off? Preferably by just locally adding in something to the cmp() call?

The best "solution" I've got so far is to put
$ansHash->{prev_ans}='';
at the beginning of the custom checker, to wipe memory of the previous answer. This band-aids the issues I describe below, but I think it would be better just to turn off this comparison in these kinds of problems.

----

There are two ways in which this is causing trouble. One is that a student might submit something like "4x/2" and be told their answer is incorrect because the custom checker is examining the form of the answer. Then they try the correct answer "2x", and it is marked correct, and the message "this is equivalent to your previous answer" is given, presumably because the "4x/2" is compared to the "2x" without any custom method of comparison. A feedback message like "this is equivalent to your previous answer" is undesirable in this situation.

The second issue is something I cannot fully explain, but I am convinced it has to do with this comparison to the previous answer. Again using one of our complicated custom checkers, a student might submit an answer. What happens sometimes is that a WeBWorK error message (not a student feedback message) arises:

Please inform your instructor that an error occurred while checking your answer at [PG]/lib/Value/AnswerChecker.pm line 254

And experimentation (that I won't go into, unless someone asks for it) is suggesting that this is stemming from there being no definition of what it means to compare the answer and the previous answer. That is, I think the defined($equal) in AnswerChecker.pm line 250 is working out to false.
In reply to Alex Jordan

Re: Can I turn off comparison of an answer to the previous submitted answer?

by Davide Cervone -
Is there a way to turn that comparison off?

Your idea of setting $ansHash->{prev_ans}='' seems like a good solution to me. This will certainly disable the check, and has no adverse side-effects.

The check is done in an AnswerEvaluator post-filter. It would also be possible to remove all the post filters via 

ANS($ans->cmp(...)->withPostFilter("erase"));
This will remove all post-filters, and there is a second one that cleans up after one of the pre-filters (the one that checks for blank answers). It is not essential, however, so erasing them both would also be OK.

It would be possible to add an option to cmp() that disables the check, I suppose. That check has come up several times recently, and it needs to be improved. Currently it converts the previous answer to a Formula, and that may not end up being the proper class for the student answer, so things are compared when they shouldn't be. Also, the check doesn't use the typeMatch() call to decide if the two answers can be compared. So there are several things that need fixing there.

You say that the custom checker isn't used during the checking of the previous answer, but actually it is. But some checkers have a post-processor (cmp_postprocess()) that does additional checking, and that is not being called during the previous-answer check. For example, parallel vectors are checked in cmp_postprocess(), and whether fractions are reduced or not. It may be that cmp_postprocess() should be part of the previous-answer check as well.

The error message that you cite is generated when there is a custom answer checker and it throws an error. (Note that the fact that you are getting this shows that the checker actually is running.) Usually, this is accompanied by the error being put into the results table as the answer message, but during the previous-answer check, the answer message is discarded; but the warning remains. You are right that it probably shouldn't be issued, but it is caused by the custom checker dying somewhere. It is probably because the previous or current answers are not the proper type of objects that the custom checker expects, and so it uses them in an invalid way. Or you may be making assumptions about the correct answer being the instructor's answer, but during the previous-answer check, the check is between two student answers, so the "correct" answer is actually one of the student answers.

In any case, that error message indicates a problem with the custom checker itself. You have correctly identified defined($equal) a being false in this case ($equal is undefined if the custom checker fails to run due to an error in the checker). It may be that when I add the typeMatch() check before calling the custom checker, that will take care of your particular problems. Alternatively, you could look for where the checker is crashing due to the use with two student answers.

In reply to Davide Cervone

Re: Can I turn off comparison of an answer to the previous submitted answer?

by Alex Jordan -
Thanks Davide,

This helps me understand the entire answer checking routine better. I'm OK with the current state of the problem, although there is still room for improvement. I have some further observations that I'll offer in case they help you decide how to improve this post filter if you do that at some point in the future.
  • A student was getting the WeBWorK error message upon their very first submission if the answer was a certain incorrect answer (it was -x^10/35). Other incorrect answers that have the same "shape" and for which one would expect to be Computed into the same class (like -x^9/35) did not yield the error message if submitted on the first submission. This just strikes me as odd since there is no previous answer yet.
  • I noticed that I could check -x^9/35, not get the error; check -x^10/35, not get the error; and check -x^10/35 again, and _get_ the error. Actually this was the thing that clued me in to this having to do with the previous answer check.
  • I could only reproduce this while acting as the student. If I was acting as myself, using the student's seed, I could not reproduce this. Do any of these post-filters use randomization that does not respect the seed? (Incidentally, we had an issue last year where some problems would hang forever for some students, but using the same seed, the problem would not hang for instructors. I changed a Compute to a Formula, or maybe vice versa, and the issue went away. But I'm wondering if there is seed-independent randomization in play somewhere in the system.)
  • I added
    $ansHash->{prev_ans}=''
    to the beginning of the custom checker, and thus far all is well. But if I attach an AnswerHint post filter, I start getting the same WeBWorK error message. (The question here is basically asking the student to multiply x^5*x^2, and the x^10 response is a common mistake that an AnswerHint would be good for.) Does AnswerHint make use of prev_ans?
In reply to Alex Jordan

Re: Can I turn off comparison of an answer to the previous submitted answer?

by Davide Cervone -
Do any of these post-filters use randomization that does not respect the seed?

Not that I recall. I'm pretty sure that is not the case (though the Value library does have its own random number routines in case they are used outside of WeBWorK, but they are supposed to use the WeBWorK one when available. I suppose it is possible that that isn't working entirely correctly).

I changed a Compute to a Formula, or maybe vice versa, and the issue went away.

The only way I could see that making an effect is changing Formula() to Compute() in the case where the result is constant, so that this changes it from a constant-valued formula to a plain constant. The former would involve random points while the latter would not. But even then, the random point generation is pretty careful not to get into infinite loops, so I'd be surprised if that were the cause.

Does AnswerHint make use of prev_ans?

No. Only the Formula cmp_preprocess() and cmp_postprocess() functions do.

I am thinking that in this case the error message may not eb coming from the previous answer checker. I'd have to see the full problem code to be able to say anything more useful about what you are seeing.
In reply to Davide Cervone

Re: Can I turn off comparison of an answer to the previous submitted answer?

by Alex Jordan -
A year later. A few days ago I posted something about having trouble with postFilters, and now I think it has more to do with this old thread.

I've hammered it down to this MWE below. There is no custom answer checker here, so I haven't blanked out prev_ans as discussed in this thread. This MWE may seem silly---it's just meant to illustrate the issue.

A few days ago I did not think this had to do with ImplicitPlane, but I was unable to make an MWE without using ImplicitPlane. So the issue may be tied to ImplicitPlane after all. I can't tell.

I tested this both on our production server and on another one on a "clean" develop branch. Note that you may need to submit something twice to see the error messages appear. The error messages indicate that within the postFilter, the active context is not ImplicitPlane anymore. It claims 'y' is not defined in the context.

DOCUMENT();

loadMacros(
"PGstandard.pl",
"MathObjects.pl",
"PGML.pl",
"parserImplicitPlane.pl",
);

##############################################

Context("ImplicitPlane");

$ans = ImplicitPlane("y=x");

##############################################

TEXT(beginproblem());

BEGIN_PGML

Enter [`y=x`].

[_______________].

END_PGML

##############################################


ANS($ans->cmp() -> withPostFilter(sub {
my $ansHash = shift;
$ansHash->{ans_message} = Formula("y");
return $ansHash;
}));


ENDDOCUMENT();


In reply to Alex Jordan

Re: Can I turn off comparison of an answer to the previous submitted answer?

by Davide Cervone -
This turns out to be a bug in the MathObjects Formula object that causes the current context to be incorrectly set after the main answer evaluator runs but before the post-filters run. This means that the post-filters don't have the ImplicitPlane context set, and are using the default Numeric context instead. That is the origin of the error message about the missing "y", since that context doesn't have a variable "y".

I have submitted a pull request to patch the problem.

In the meantime, adding a Context() call (with no context name specified) in the post-filter before the Formula() call is made will reset the context to the correct one.
In reply to Davide Cervone

Re: Can I turn off comparison of an answer to the previous submitted answer?

by Nathan Wallach -
I ran into some issues with adaptive parameters, and at times was getting unwanted "This answer is equivalent to the one you just submitted" messages and sometimes the "Please inform your instructor that an error occurred while checking your answer" message. The second error message (and Google) lead me to this thread.

I tried using withPostFilter("erase") to avoid the issue... and discovered that with WW 2.15 using the "erase" postfilter approach causes an error message on problems loading (assuming the past answer is still blank/non-existent) and whenever the problem is submitted with an empty input box. 

On problem load I see: 
     PG question processing error messages

     PG warning messages

     ------
     Evaluation error in new process: Answer AnSwEr0001:

     BLANK
     ::
     The answer is blank


It seems that it is now somewhat dangerous to use withPostFilter("erase").

It would be nice if there was a setting which could be sent to cmp which would disable the "produce_equivalence_message" filter (cmp_postfilter in lib/Value/AnswerChecker.pm) and the postfilter in macros/PGfunctionevaluators.pl rather than Alex's hack of setting $ansHash->{prev_ans}=''. 

I opened an issue at:  https://github.com/openwebwork/pg/issues/495

Sample code which hits the "blank" warning on load:

DOCUMENT();
loadMacros(
  "PGstandard.pl",
  "MathObjects.pl",
);
Context("Numeric");
$aSoln = Compute("e^x");
BEGIN_TEXT
\{ $aSoln->ans_rule(20) \}
END_TEXT
ANS( $aSoln->cmp()->withPostFilter("erase") );
ENDDOCUMENT();
The problems seems to be that it is no longer safe to remove the "blank_postfilter" and a less than convenient approach would be to force it back by hand as follows: 
DOCUMENT();
loadMacros(
  "PGstandard.pl",
  "MathObjects.pl",
);
Context("Numeric");
$aSoln = Compute("e^x");
BEGIN_TEXT
\{ $aSoln->ans_rule(20) \}
END_TEXT
ANS( $aSoln->cmp()->withPostFilter("erase")->withPostFilter(
  # Reinstall code of "blank_postfilter" manually as "erase" removed it.
  # Since I did not find a method to reload it by name, the subroutine is just given.
  sub {
    my $rh_ans=shift;
    $rh_ans->{_filter_name} = 'blank_postfilter';
    return($rh_ans) unless defined($rh_ans->{error_flag}) and $rh_ans->{error_flag} eq 'BLANK';
    $rh_ans->{error_flag} = undef;
    $rh_ans->{error_message} = '';
    $rh_ans->{done} =1;    # no further checking is needed.
    $rh_ans;
  }
));
ENDDOCUMENT();
In reply to Alex Jordan

Re: Can I turn off comparison of an answer to the previous submitted answer?

by Nathan Wallach -
I created a pull request with a small change to PG to enable easily disabling the equivalence message by passing "bypass_equivalence_test => 1" as part of the arguments to the cmp call.


See: https://github.com/openwebwork/pg/pull/497

Hopefully it will make its way into the develop branch and then into WW 2.16.
People with access to modify the main PG code on their server can just make the minor change in the PR to lib/Value/AnswerChecker.pm and try it out.