** Using WeBWorK **

Is there an answer evaluator that will handle this? I brief look through the Webwork library doesn't bring up any similar looking problems.

I thought that equation_cmp might work, but that seems to compare one equation with another equation, instead of checking if entered numbers satisfy a given equation.

Thanks, Tom

<| Post or View Comments |>

In general, one would like answer evaluators that test whether a student's answer satisfies some conditions rather than just being a specific number, formula, etc. Clearly some programming is required to do this, but writing ad hoc answer evaluators from scratch, though do-able, is time consuming, and trying to use existing evaluators with filters is not flexible enough.

The new mathematical-expression parser shipping with Webwork 2.1 simplifies things a lot. (My thanks to the developers; I know Davide Cervone is involved. Any others?) Based on a sample answer evaluator "vector_cmp" in extensions/8-answer.pg in the parser documentation (located in docs/parser under the webwork2 directory) I tried to devise a generic answer evaluator that accepts a plug-in for checking the student's answer for some condition. Using this, a solution to Tom's problem might look like:

BEGIN_TEXT
Enter a point on the graph of ( x + 2y = 3 )
$BR
(x,y) = {ans_rule()}
END_TEXT

sub check{
my $stu_ans = shift(); #expects a point
my $x = $stu_ans -> extract(1); #extract the first coordinate
my $y = $stu_ans -> extract(2);
$x + 2*$y == 3;
}

ANS( generic_cmp("(-3,3)", type=>"Point", checker => ~~&check));

The first argument to generic_cmp is the professor's preferred answer, which by default will show up as the correct solution. The type of the expected answer (must be a recognized parser type) and the plug-in checker must be specified, and there are a few other optional parameters I've found useful.

(In this example, the checker routine could be improved: e.g.,by testing that the point has exactly two coordinates, and using fuzzy comparison.)

The main advantage is that generic_cmp does the work of making sure that the student's answer is syntactically correct and, if necessary, evaluating it before the plug-in checker gets it. Also, the checker can use the existing parser methods to manipulate the student's answer. (When the parser is better documented this will get easier.) So the checker routine can be relatively simple. Or it can be quite complicated if the conditions one is testing for are.

I've found it possible to write ad hoc answer evaluators using this technique without much pain, and I'm currently using some of the problems in a course. So far my students haven't broken it. It could also be used to write the specific answer evaluator Tom wanted.

I think an answer evaluator of this type is needed, but I haven't spotted one before. A discussion on how it should be done would be interesting.

e-mail me if you would like to look at the source.

Bob Byerly

<| Post or View Comments |>

Thanks for posting your approach to the answer checker. You mention that you would like to add "fuzzy" checking, but I wanted to point out that you already have it. I assume that $stu_ans is getting the Point object that the student entered, and it turns out that the entries in the Point will be Real objects, and these already do fuzzy checking automatically. So $x + 2*$y == 3 is already doing the fuzzy checking, since == is overloaded to call the Real comparison that does the fuzzy check. So you get that for free (which is one of the nice things about the new parser).

Even if $x and $y were not Reals, you could force fuzzy comparison by using $x + 2*$y == Real(3) in this case.

There is another way to make new answer checkers using the new parser, and that is to take advantage of the parser's built in answer checkers. You can do that by making a subclass of one of the parser classes, and override the methods that do the comparison within the answer checker. I've written a general-purpose checker for a point satisfying an equation, and will post it to the CVS repository as soon as the development server is up and running again. I'll post an example here shortly to let you know the basics about how it works.

Davide

(PS, you ask if anyone else is involed with the parser, and the answer is no, it's pretty much my baby. So any flaws and misfeatures are entirely my fault.)

<| Post or View Comments |>

Bob

<| Post or View Comments |>

Here is an example of how to make a class that handles questions of the form "Give a point (x,y) that satisfies the equation x^2-3y = 2x".

DOCUMENT();        # This should be the first executable line in the problem.

loadMacros(
PG.pl,
PGbasicmacros.pl,
PGanswermacros.pl,
"Parser.pl",
);

TEXT(&beginproblem);

##############################################
#
# Define the SolutionFor class
#

#
# Syntactic sugar for creating SolutionFor objects
#
sub SolutionFor {SolutionFor::Point->new(@_)}

#
# Make a subclass of the Point class
#
package SolutionFor::Point;
our @ISA = qw(Value::Point Value);

sub new {
my $self = shift; my $class = ref($self) || $self;
Parser::BOP::equality::Allow; # enable equality operator
my $f = main::Formula(shift); # get professor's formula
my $p = main::Point(shift); # get professor's point
$p->{f} = $f; # save the formula object
$p->{F} = $f->perlFunction; # and an executable version
$p->{isValue} = 1; # mark this as a Value object
return bless $p, $class;
}

#
# Evaluate the formula on the given point
#
sub f {
my $self = shift;
&{$self->{F}}(shift->value);
}

#
# Use the Point's defaults, but turn off coordinate hints
# (since a wrong coordinate isn't detectable)
#
sub cmp_defaults {(
shift->SUPER::cmp_defaults,
showCoordinateHints => 0,
)}

#
# Override the standard <=> comparison.
# Test if the formula's equality operation returns true or false.
# (Since we are implementing <=> here, we need
# to return 0 when true and 1 when false.)
#
sub compare {
my ($l,$r,$flag) = @_;
$r = Value::makeValue($r);
return ($l->f($r)) ? 0 : 1;
}

package main;

##############################################
#
# The setup
#

Context("Vector")->variables->are(x=>'Real',y=>'Real');

$f = SolutionFor("x+y=3","(1,2)");

##############################################
#
# The problem text
#

Context()->texStrings;
BEGIN_TEXT
Suppose ($f->{f}).
A solution for this equation is ((x,y)) = {ans_rule(15)}
END_TEXT
Context()->normalStrings;

##############################################
#
# The answer

ANS($f->cmp);
$showPartialCorrectAnswers = 1;

##############################################

ENDDOCUMENT(); # This should be the last executable line in the problem.

I hope the comments in the code make it clear what is happening (though a look through pg/lib/value/AnswerChecker.pl might help, since this is where the main cmp() method is defined that is used by all the parser objects).

The basic idea is that we make a subclass of Value::Point (the main Point object class), and inherit its cmp() method, but override its compare() method. This is the method called by the overloaded == operator (which is used by cmp() to decide when the students answer matches the professor's answer).

The SolutionFor class creates a point that is the professor's answer, and stores the equation as part of the data for that point, and makes it into a SolutionFor object. When a point is compared to that object, the comparison is true when the poitn satisfies the associated equation, and false otherwise. For example:

  $f = SolutionFor("x^2=y^3","(1,1)");
  if ($f == Point(1,-1)) {warn "yes"} else {warn "no"}
  if ($f == Point(0,0)) {warn "yes"} else {warn "no"}

would generate a warnig mesage of "no" followed by a warning of "yes". Since arrays are promoted to point automatically, you could even do

  $f = SolutionFor("x^2=y^3","(1,1)");
  if ($f == [1,-1]) {warn "yes"} else {warn "no"}

though perhaps that's not best practice.

In all other ways, the SolutionFor acts like a point (since it inherits all the other Point methods). So you can get its TeX string from $f->TeX, and so on. Finally, since the Formula object for the equation is stored in the SolutionFor object, you can use $f->{f} to access it (as in the example above in order to print its TeX form as part of the text of the problem). You can also use $f->f($p) to evaluate the equation on the point $p; this will return 1 if the point satisfies the equality and 0 otherwise.

So not only do you get an answer checker from this, you actually get a fully functional parser object that you can use in more sophisticated ways.

If you wanted to use something like this in practice, you would need to do some error checking and so on, but this is a simple example to show you the right way to do it. This one also leaves the equality operator defined, so the student could include them in his or her answer without generating an error, though the answer would be marked wrong in most cases.

I will post a more fully featured version on the CVS repostiory as soon as the server is up again. That one does the error checking, works for points or for single real and complex answers, lets you specify the order of the variables in the student's answer, and doesn't allow the equality operator in the student's answer. (This one only works where the equation has two ore more variables, and the variables in the student answer must be in alphabetical order).

Finally, because of an oversight when I wrote the Parser package originally, you will need to get the latest version of the parser in order for this to work. (It turns out I hadn't done the overloading properly so you can't override the compare() method in the original version of the parser). Again, I will post the updated code to the CVS server as soon as it is available again.

Davide

<| Post or View Comments |>

Be sure to update your copy of the parser files from teh latest CVS version, as there were some changes that I had to make in order to allow subclasses to override the overloaded operators, and parserSolutionFor.pl relies on this ability. You will need to restart your server after updating the files.

Davide

<| Post or View Comments |>

I thought I would go ahead and make available the source code for generic_cmp. It can be downloaded here. Davide could no doubt have done the thing much better, and I would be happy to get suggestions for improvements. (I will almost certainly continue tweaking it; and there are some extensions I have in mind.)

I think the plug-in approach still has some value, as it allows the problem writer (so far, just me!) to write custom answer evaluators while concentrating (mostly!) on just the mathematical conditions the student answer should satisfy, and not have to worry about syntax checking or answer hashes. (The plug-in is passed a reference to the answer hash, though, so it can pass back custom error messages if the problem writer desires.)

Davide's example does illustrate that there are two levels of understanding a complex piece of software such as the parser package. At the developer's level, one would have a clear mental picture of the relationships among the different classes, what is inherited from what, and what is overridden. And, of course, the data-structures.

At a somewhat lower level (my level), one would understand what the different classes are supposed to represent, and what the main methods for each class are supposed to accomplish. As most of the classes of the parser package are supposed to represent standard types of mathematical objects, this level of knowledge is relatively easy to attain (although sometimes there are surprises when things don't work exactly as you expect!)

It speaks well of the design of the parser package that someone with my level of understanding can still use it to write answer evaluators, particularly with a model before me, and indeed to find ways to streamline the process. I would hesitate to attempt to emulate Davide's example without a better understanding of the parser. (Davide, if I were to want to attain such an understanding by a "top-down" reading of the source code, where would you suggest I start?)

Here are a few examples of the ways I've used generic_cmp:

1) Illustrate that a particular set of vectors does not span a vector space by giving a vector not in the span. (An example of this is given in the source code. This one can be easily solved by guessing, but then I've found students are motivated to ask how one might find it in general.)

2) Show that a set v1, ..., vn of vectors is linearly dependent by finding a particular non-trivial solution to x1*v1 + ... + xn*vn = 0. (Infinitely many correct answers, but they all satisfy easily an easily specifiable condition.)

3) Enter a basis for the following subspace ... (For subspaces of R^n this answer evaluator already exists. I wanted to ask such questions for more general vector spaces such as spaces of polynomials. The existing answer evaluator could be used as a back-end.)

Evaluators such as these would occur in only a few problems in a linear algebra course, where one is encouraging students to understand new concepts by playing with particular examples of them. One would therefore not want to spend a large amount of time writing custom evaluators for such problems (and, except in example 3, I didn't. Like many of us, I have to write problems to a deadline.)

Answer evaluators I've wanted in the past and didn't have, but now feel more confident that I could get with not too much work include: (1) evaluator for checking when a student is asked to enter, but not evaluate, a definite integral for computing a particular area, volume, or whatever (I assume I could use the numerical quadrature techniques in the WebWork library for this,) (2) evaluators for asking a student to enter a function in a particular form (e.g., no logs of products or powers). (I think I could get this fairly easily when I understand the data-structures in the parser class a little better. A sample from Davide would make it even easier.)

In short I feel a lot less constrained about the types of problems I can pose in WebWork now.

I don't know what the situation is with regard to contributing code to the WebWork project. If anyone but me finds this useful, I would be willing to do so.

Bob Byerly

<| Post or View Comments |>

Thanks for posting your generic_cmp code. I agree with you that the "plug-in" approach is a good one, and didn't mean to suggest that you shouldn't continue to work on your answer checker.

Looking through the code for your generic_cmp, things looked pretty good to me (I recognized some of the code), but there were a few things I was going to point out to you. For example, you really don't need to pass the type of the correct answer, since you can get that from the parsed object itself (for example, using the ->class method). Also, you might want to check if the student is previewing rather than submitting answers before you give an error message that tells the student something about whether his answer is correct or not (the one for the type mismatch already does this).

Some technical things: the protectHTML routine is available within the Value package as Value::protectHTML(string), so you call that rather than repeating its definition yourself. Also, the stuff at the end of the routine that handles the error message is also available from the Value package as a method of any parser object. So in your code, you could use $v->cmp_error($ans) to replace the stuff in the "else" clause.

I was going to write up a modified version for you, but as I was doing it, I realized that some things like the length check and the type matching, and so on, are already being done by the parser's built-in answer checker, and so I thought that perhaps it would be better to subclass that and replace the part that does the equality check. So I wrote that instead.

It worked, but I was not satisfied with the number of routines that had to be overridden in order to get at what amounted to only one line of code in the parser's cmp_equal method (the line that performs the overloaded == operator to see of the professor's and student's values are equal). I have wanted to replace that line in other circumstances, so I realized that I really needed to make one more level of modularization and pull that one line out into a separate method that could be overridden by subclasses. Following Bob's lead, however, I made the new method look for a user-supplied checker, and if there wasn't one, use the == operator. This lets you override the standard equality check without having to subclass anything, so it gives you the functionality of Bob's generic_cmp directly within the parser's native answer checker. E.g.:

      ANS(Vector(1,2,3)->cmp(checker=>sub {
        my ($correct,$student,$ans) = @_;
        return 0 if $correct->length != $student->length;
        return 0 if norm($student) == 0; # don't accept zero vector
        return ($correct . $student) == 0;
      },
      showCoordinateHints => 0);

would do a check to see if the student's answer is perpendicular to the given vector, rather than equal to it. Note that you can pass additional options to the answer checker, like showCoordinateHints above (we don't want these when the answer is wrong, since we can't tell whether an individual coordinate is right or not). Any of the standard options for the answer checker of the object class of the professor's answer can be specified in this way.

If you want to try it, you will need to get the updated pg/lib/Value/AnswerChecker.pm file (and restart the server). I also added a new file in pg/macros called answserCustom.pl that implements a simple shell around this new feature. So if you load that macro file, you can do something like:

    ANS(custom_cmp("<1,2,3>",sub {
        my ($correct,$student,$ans) = @_;
        return 0 if norm($student) == 0;
        return ($correct . $student) == 0;
    }, showCoordinateHints => 0);

You can specify some additional parameters that control wether your checker is called only when the two vectors are the same length (true by default), and so on. See the comments in the answerCustom.pl file for details.

If you specify a custom checker for a List object (or a Union object, which is actually implemented as a list of intervals), then the checker will be called on the individual elements of the list to decide if they are equal. So you don't have to write the code to process unordered vs. ordered lists, or work out partial credit, or report appropriate messages. You can just concentrate on the checking of entries in the list, and the parser answer checker does the rest.

If you want, however, you can specify your own list-based checker that gets passed the two lists, rather than the individual elements of the lists. This would give you the ability to handle the lists in any way that you choose. You can do this by using the list_checker=>code option to the List(...)->cmp method, or by loading answerCustom.pl and calling custom_list_cmp(). Again, see the documentation in that file for details.

One nice thing about the implementation in the parser is that it traps errors in the user-supplied code and can report them in a better way. This is quite difficult to do in a macro file, but is easy to do in a preloaded module file. Your checker code can call Value::Error(message) to generate an error (and then die); the error message will appear in the Messages area at the top of the screen rather than cause a pink screen of death.

Here is an example problem that uses a custom checker to ask for three distinct points that lie on a given implicit curve. The answer checker tests if the point actually IS a solution, and also checks if the student has given that point already, and issues an error message if he did.

DOCUMENT();        # This should be the first executable line in the problem.

######################################################################
#
# Example of using custom_cmp to install custom answer
# checkers for various object types.
#
######################################################################

loadMacros(
PG.pl,
PGbasicmacros.pl,
PGanswermacros.pl,
"Parser.pl",
"answerCustom.pl",
);

TEXT(&beginproblem);

##############################################
#
# The setup
#

Context("Vector")->variables->are(x=>'Real',y=>'Real');
Parser::BOP::equality::Allow; # allow in professor's formula

#
# Student answers must satisfy this equation
#
$f = Formula("x^2 + y^2 = 1");

Context()->operators->remove('='); # don't allow in student answers

##############################################
#
# The problem text
#

Context()->texStrings;
BEGIN_TEXT
Find three distinct solutions to the equation \($f\): \{ans_rule(30)\}
END_TEXT
Context()->normalStrings;

##############################################
#
# The answer

@student_points = (); # store points the student got right
$F = $f->perlFunction; # a function to evaluate the formula

sub checkSolutions {
my ($correct,$student,$ans) = @_;
#
# Check that the student point IS a solution.
#
return 0 unless &{$F}($student->value);
#
# Check that it hasn't already been used.
#
#
foreach $p (@student_points) {
Value::Error("You can't use the same point more than once")
if $p == $student;
}
#
# Save the student's point for future checks.
#
push(@student_points,$student);
return 1;
}

ANS(custom_cmp("(1,0),(0,1),(sqrt(2)/2,sqrt(2)/2)",~~&checkSolutions));
$showPartialCorrectAnswers = 1;

##############################################

ENDDOCUMENT(); # This should be the last executable line in the problem.

Hope that makes things easier for you.

Davide

<| Post or View Comments |>

Looks like you've done a lot of work on this. Thanks. I can definitely see some of the advantages of your approach, and I will trying in the next few weeks to see if I can adapt my examples to it. I may have specific suggestions or questions as I do this. I will be glad not to have problems that rely too much on locally produced macro files (been through too much of that with TeX).

Meanwhile, I'll be trying to digest your message on the organization of the parser. I've already tried playing with a few examples of analyzing the parse tree and find it not too hard, but I agree that some better methods for getting access to it will make things much easier.

This has been a long thread, but I think it has resulted in the addition of a potentially quite useful class of answer evaluators to WebWork.

Bob

<| Post or View Comments |>