WeBWorK Problems

Matrix custom answer checker not returning Value->Error messages

Matrix custom answer checker not returning Value->Error messages

by Alex Jordan -
Number of replies: 7

I have some matrix problems that use a custom answer checker. Suddenly this term, they do not work. It's possible that they have not been used since the server was upgraded to 2.15, and it's possible they worked in 2.14. I checked on other 2.15 servers I have access to and I get the same behavior.

My normal debugging routine would be to start fiddling with the custom answer checker, getting it to pretty print the answer hash. But I can't get it to print any feedback error message at all when the answer is a Matrix.

I've whittled it down to the silly problem below. My expectation is that the answer checker should simply report 'testing' as the feedback error message. But it does not do that when the answer is a Matrix. It seems specific to Matrix. If the answer is just, say, Real(1), then the custom answer checker behaves as expected and does return the 'testing' error message.

I've tried disabling pre filters and post filters and there is no change. So I'm not sure what is going on. Does anyone have insight? Also, I'm curious if this problem works on a version earlier than 2.15 if someone has access to one.

Of course I still have my original issue of why my custom answer checker stopped working. But I wonder if resolving this issue with Value->Error first might also resolve my original issue.


DOCUMENT();

loadMacros(
      "PGstandard.pl",
      "MathObjects.pl",
    );

Context("Matrix");
#Context("Numeric");
#Context()->lists->set(Matrix => {open => "[", close => "]"});
#Context()->parens->set("[" => {type => "Matrix", close => "]"});
   
$answer = Matrix([[1,1],[1,1]]);
# submit the above as the answer and yet I do not see the expected feedback message

#change the answer to Real(1), submit "1" as an answer, and see the feedback message appear
#$answer = Real(1);      

BEGIN_TEXT

Enter: $answer $PAR

\{ ans_rule(60) \}

END_TEXT

ANS($answer->cmp(checker => sub {Value->Error('testing');}));

ENDDOCUMENT();



In reply to Alex Jordan

Re: Matrix custom answer checker not returning Value->Error messages

by Danny Glin -

On my 2.14 installation I see the same behaviour: with Matrix I get no feedback.  If I change it to Real I get the feedback message.

In reply to Danny Glin

Re: Matrix custom answer checker not returning Value->Error messages

by Alex Jordan -

Thanks for checking Danny. This adds to the mystery as I will post about on the main thread.

In reply to Alex Jordan

Re: Matrix custom answer checker not returning Value->Error messages

by Alex Jordan -
Danny sees this happening on a 2.14 WW server. I'm sure I used these problems back when our server was 2.14.

But also, three of our instructors used these problems in the Fall and there was no issue. In fact, I just logged in to a Fall course and found a student who had one of these questions with a 100%. I check their submitted answer and it incorrectly comes back as incorrect. So I'm sure these were recently working on our 2.15.

I can find nothing with our WW configuration that changed this recently to blame for the change. But since it's happening on other servers (clean, unadulterated 2.15 servers and Danny's 2.14) I suspect it's something else. some things that I know changed with the /server/ are in October (possibly past the point these exercises are used in a course) more resources were allocated (more RAM, more CPU). And in December, out IT migrated the server to a VxRail VMware environment (I don't know what that means, just copy-pasting what they told me). I'm now wondering if during any of that, something at a level deeper than the WW installation was upgraded.

It's a significant issue for us, because most of our linear algebra exercises at this point in the term are failing. Danny, what OS and perl version are on your server? I have:

Ubuntu 18.04.5 LTS (GNU/Linux 4.15.0-128-generic x86_64)
This is perl 5, version 26, subversion 1 (v5.26.1) built for x86_64-linux-gnu-thread-multi

Can anyone else replicate the issue in my sample problem? Or, if you cannot replicate it, that could help figure this out too if your OS or perl is different from mine.
In reply to Alex Jordan

Re: Matrix custom answer checker not returning Value->Error messages

by Davide Cervone -

There is a flag, showEqualErrors that controls whether the MathObject answer checker reports errors that occur during the equality comparison (which is when the checker is called). For most object types, this is set to 1, but for the Matrix class, it is 0 by default. I no longer remember why that was set to 0, but it has been that way for the past 14 years. So Value->Error() inside a Matrix checker routine will stop the answer checker from running, but will not report the error in the answer table. Again, I don't remember why that was done, but it has been the case for quite some time.

There are several possible solutions.

  1. Set showEqualErrors=>1 in the cmp() call for the answers that could have error message.

  2. Use something like

        ANS($answer->cmp(checker => sub {
      my ($correct, $student,$ans) = @_;
      $correct->cmp_Error($ans,'testing');
    }));

    for the error messages rather than Value->Error();.

  3. Modify pg/lib/Value/AnswerChecker.pm to change the setting for showEqualErrors to 1 in the Value::Matrix package.

  4. Use parserCustomization.pl to do something like

        $context{Matrix} = Context("Matrix");
    Context->{cmpDefaults}{Matrix}{showEqualErrors} = 1;
    Context("Numeric");

    (The middle line could also be used in a .pg file to set this for all Matrix answers)

Note that, other than 2 above, these may cause other unwanted error messages to show up. I think the reason they were turned off is that if the student enters a Matrix of the wrong size, the message will be "Can't compare matrices of different dimensions" or something like that, rather than the better "Matrix dimension is not correct", but I suppose you could use the Context()->{msg} remapping mechanism to fix that as well. There might be other situations that I'm not able to think of at this point.

In any case, I don't see how the problem would have been working in the past without having done one of these things, since the code that controls it has been there for more than a decade.

In reply to Davide Cervone

Re: Matrix custom answer checker not returning Value->Error messages

by Alex Jordan -

Thanks Davide, this will let me start pretty printing the answer hash to explore why the real problem stopped working. I was (too) optimistic that my real issue is related to the error messages not coming through. Hopefully I am not too optimistic that I can sleuth out the bug in the real problem now.

In reply to Alex Jordan

Re: Matrix custom answer checker not returning Value->Error messages

by Davide Cervone -
OK, good luck. If you need me to look at it, let me know.
In reply to Davide Cervone

Re: Matrix custom answer checker not returning Value->Error messages

by Alex Jordan -
I got it, and thanks again. Once I could explore the elements of the custom answer checker by printing error messages, I tracked it down to our use of `digits` tolType (https://github.com/openwebwork/pg/pull/441). We've been using it as the default for Numeric since early 2020, so I'm unsure why it was only now that problems stopped working. But it could be that only some random seeds trigger to the issue.

As a consequence of using it as the default for Numeric, it was inherited by Matrix. And at some level when matrices are multiplied and then compared with other matrices, this tolType leads to declaring some things not equal when they should be declared equal.

In my situation, a matrix Formula was evaluated at random values for the variables, leading to a comparison of something with all zero entries with something else that *should* have had zero entries, but the decimal arithmetic and numerical matrix multiplication was making something with entries that were slightly off. Off by enough to get a comparison like "0 == 0.000000001" which returned false.

This reveals that this tolType needs to be amended to handle comparisons within the $zeroLevel better.