WeBWorK Problems

Matrix custom answer checker not returning Value->Error messages

Re: Matrix custom answer checker not returning Value->Error messages

by Davide Cervone -
Number of replies: 3

There is a flag, showEqualErrors that controls whether the MathObject answer checker reports errors that occur during the equality comparison (which is when the checker is called). For most object types, this is set to 1, but for the Matrix class, it is 0 by default. I no longer remember why that was set to 0, but it has been that way for the past 14 years. So Value->Error() inside a Matrix checker routine will stop the answer checker from running, but will not report the error in the answer table. Again, I don't remember why that was done, but it has been the case for quite some time.

There are several possible solutions.

  1. Set showEqualErrors=>1 in the cmp() call for the answers that could have error message.

  2. Use something like

        ANS($answer->cmp(checker => sub {
          my ($correct, $student,$ans) = @_;
          $correct->cmp_Error($ans,'testing');
        }));
    

    for the error messages rather than Value->Error();.

  3. Modify pg/lib/Value/AnswerChecker.pm to change the setting for showEqualErrors to 1 in the Value::Matrix package.

  4. Use parserCustomization.pl to do something like

        $context{Matrix} = Context("Matrix");
        Context->{cmpDefaults}{Matrix}{showEqualErrors} = 1;
        Context("Numeric");
    

    (The middle line could also be used in a .pg file to set this for all Matrix answers)

Note that, other than 2 above, these may cause other unwanted error messages to show up. I think the reason they were turned off is that if the student enters a Matrix of the wrong size, the message will be "Can't compare matrices of different dimensions" or something like that, rather than the better "Matrix dimension is not correct", but I suppose you could use the Context()->{msg} remapping mechanism to fix that as well. There might be other situations that I'm not able to think of at this point.

In any case, I don't see how the problem would have been working in the past without having done one of these things, since the code that controls it has been there for more than a decade.

In reply to Davide Cervone

Re: Matrix custom answer checker not returning Value->Error messages

by Alex Jordan -

Thanks Davide, this will let me start pretty printing the answer hash to explore why the real problem stopped working. I was (too) optimistic that my real issue is related to the error messages not coming through. Hopefully I am not too optimistic that I can sleuth out the bug in the real problem now.

In reply to Alex Jordan

Re: Matrix custom answer checker not returning Value->Error messages

by Davide Cervone -
OK, good luck. If you need me to look at it, let me know.
In reply to Davide Cervone

Re: Matrix custom answer checker not returning Value->Error messages

by Alex Jordan -
I got it, and thanks again. Once I could explore the elements of the custom answer checker by printing error messages, I tracked it down to our use of `digits` tolType (https://github.com/openwebwork/pg/pull/441). We've been using it as the default for Numeric since early 2020, so I'm unsure why it was only now that problems stopped working. But it could be that only some random seeds trigger to the issue.

As a consequence of using it as the default for Numeric, it was inherited by Matrix. And at some level when matrices are multiplied and then compared with other matrices, this tolType leads to declaring some things not equal when they should be declared equal.

In my situation, a matrix Formula was evaluated at random values for the variables, leading to a comparison of something with all zero entries with something else that *should* have had zero entries, but the decimal arithmetic and numerical matrix multiplication was making something with entries that were slightly off. Off by enough to get a comparison like "0 == 0.000000001" which returned false.

This reveals that this tolType needs to be amended to handle comparisons within the $zeroLevel better.