## Forum archive 2000-2006

### Nandor Sieben - user defined functions in the parser (a bug?) ### Nandor Sieben - user defined functions in the parser (a bug?)

by Arnold Pizer -
Number of replies: 0 user defined functions in the parser (a bug?) topic started 7/11/2005; 2:42:04 PMlast post 8/12/2005; 10:07:15 PM Nandor Sieben - user defined functions in the parser (a bug?) 7/11/2005; 2:42:04 PM (reads: 1527, responses: 14) We'd like to create the following problem. A function f(x)=sqrt(x+1)-2 is randomly created (the 2 and the 1 are randomly chosen). Then a randomly chosen g is defined to something like g(x) = f(x+1). The student would see the graphs of f and g and we ask for the formula of g in terms of f. So I need to be able to evaluate if an answer like "f(x-1)+2" is correct or not (not correct in this case). We need an evaluator that can handle user defined functions. I was hoping that the parser can do this. This is what I tried: $x=Formula('x'); Formula('sqrt(x+1)-2')->perlFunction('fff');$g = fff($x+1);$f = Formula('sqrt(x+1)-2'); $ff =$f->perlFunction; $h = &$ff($x+1); Both g and h works. I could use them as the correct answer. But I am going to get the student answer as a string. So now I have to do the same thing but somehow with the string "f(x+1)". I can't do it. The parser documentation mentions the use of eval() (page1 UsingParser.txt). I can see that this might be useful but eval() is not allowed in pg files. I tried this constructs. None of the work. #$k = Formula("fff($x+1)"); # error #$k = Formula("fff(x+1)"); # error Thse were my best guesses. I would think that fff now behaves like sin. Unfortunately this is not the case. What is the difference between fff and sin? # $k = Formula("$ff($x+1)"); # error #$k = Formula("&$ff($x+1)"); # error # $k = Formula("$f($x+1)"); # makes a product instead of function evaluation Am I trying the impossible? I hope I don't have to parse the student answer on my own. Nandor <| Post or View Comments |> Bob Byerly - Re: user defined functions in the parser 7/11/2005; 3:44:21 PM (reads: 1729, responses: 0) There is a sample file in the parser documentation: that gives an example of how to add a single-variable function to the parser. This sounds to me like what you want. (Disclaimer: I haven't actually tried it.) Bob <| Post or View Comments |> Nandor Sieben - Re: user defined functions in the parser 7/11/2005; 5:31:39 PM (reads: 1703, responses: 0) Well it helped. I managed to make it work. I have to say this though: it looks like black magic, it's way beyond my understanding. I have a few questions: I don't have a "ParserTables.pl" but I have "parserTables.pl". Is this a typo? Is this file needed at all for this example? I can comment it out with seemingly no effect. Is Context('Numeric'); needed for this example? I can comment it out with seemingly no effect. Could this context be the default? Nandor <| Post or View Comments |> Nandor Sieben - Re: user defined functions in the parser (a bug?) 7/11/2005; 6:37:35 PM (reads: 1692, responses: 0) There is a problem. Everything seems to be working fine until I try to do something like this for input: 2-f(x). The problem is with the negative sign in front of f(x). The error message apperas in the Messages field of the answer preview: syntax error at (eval 72952) line 5, near "2-f" No error message if I write 2-1f(x). Could this be a bug? ##DESCRIPTION # Name of the file: unknownfunc.pg # File Created: 7/1/05 # Problem Author: Nandor Sieben # Location: Northern Arizona University # Course: # Recommended trials: # Recommended value: # ##ENDDESCRIPTION ##KEYWORDS('function evaluator, string evaluator') DOCUMENT(); loadMacros("PGstandard.pl", "Parser.pl", # "parserTables.pl", ); TEXT(&beginproblem) ; # Context('Numeric'); package MyFunction; our @ISA = qw(Parser::Function::numeric); sub f { shift; my$x = shift; return CORE::sqrt($x+1)-2; } package main; sub f { Parser::Function->call('f',@_); } Context()->functions->add( f => {class => 'MyFunction' , TeX => 'f'}, );$x = Formula('x'); $a = random(-2,2,1);$g = Formula("f(x-$a)")->reduce;$g->{limits} =[$a-1,$a+5]; BEGIN_TEXT $BBOLD Unknownfunc$EBOLD $PAR (g(x)= {$g->TeX } ) $PAR (g(x)=) { ans_rule(30) } END_TEXT ANS($g->cmp); ENDDOCUMENT(); <| Post or View Comments |> Davide P. Cervone - Re: user defined functions in the parser (a bug?) 7/11/2005; 10:16:32 PM (reads: 1690, responses: 0) I don't have a "ParserTables.pl" but I have "parserTables.pl". Is this a typo? Yes, it is a typo (the file was developed on Mac OS X, which has case-insenstitive filenames, so I don't always catch this type of error). Is this file needed at all for this example? I can comment it out with seemingly no effect. No, it is not needed in your problem. It is used in the extension example file to handle the table of output that illustrates the different ways that the function works in different contenxts. Is Context('Numeric'); needed for this example? I can comment it out with seemingly no effect. Could this context be the default? While it is not strictly needed, I recommend always setting the context to the one you want, as this forces you to think about what context you really need. If you are in the habit of doing this, you will not forget to put Context("Vector") when you need that. Yes, Context("Numeric") is the default. <| Post or View Comments |> Davide P. Cervone - Re: user defined functions in the parser (a bug?) 7/11/2005; 10:21:56 PM (reads: 1688, responses: 0) Everything seems to be working fine until I try to do something like this for input: 2-f(x). The problem is with the negative sign in front of f(x). ... Could this be a bug? Yes, it is a bug, but a subtle one having to do with perl more than the parser. For function comparisons, the parser converts the formulas to equivalent perl functions and calls those (for speed) rather than using the parse tree to evaluate them. The expression 2-f(x) is converted to essentially 2-f($x), but in this case perl thinks the -f is the file-test operation is-a-plain-file rather than the difference between 2 and the value of f at$x. I have fixed this in the Parser, but you will have to update your copy of pg/lib/Parser/Context/Default.pm for it to take effect. Davide <| Post or View Comments |> Davide P. Cervone - Re: user defined functions in the parser (a bug?) 7/11/2005; 10:57:23 PM (reads: 1679, responses: 0) I managed to make it work. I have to say this though: it looks like black magic, it's way beyond my understanding. But it is all necessary black magic. You do seem to have gotten all the pieces together correctly. I was getting a similar example together to post, but you beat me to it. Here is what I was going to recommend, together with some comments on what the different steps do.   DOCUMENT(); loadMacros( "PGstandard.pl", "Parser.pl", "PGcourse.pl", ); TEXT(beginproblem()); Context("Numeric"); package f; our @ISA = qw(Parser::Function::numeric); my $a = 1; my$b = -2; sub f { my $self = shift; my$x = shift; sqrt($x+$a)+$b; } package main; sub f {Parser::Function->call('f',@_)} Context()->functions->add(f => {class => 'f', TeX => 'f'});$g = Formula("f(x+1)"); BEGIN_TEXT $$f(x+1)$$ = \{ans_rule(30)\} END_TEXT ANS($g->cmp); ENDDOCUMENT(); Here is what the important lines do:  package f; our @ISA = qw(Parser::Function::numeric); The first line starts a new package (called 'f' in this case, but it could be anything) and the second line makes it a subclass of the Parser::Function::numeric class, which is the one that implements functions of the form R->R. This makes this a real-valued function of one real variable, which is what we want. Our function will inherit all the behavior of that class (i.e., how to evaluate itself, how to print its TeX form and its perl form, and so on). This saves us a lot of work.  sub f { my$self = shift; my $x = shift; sqrt($x+$a)+$b; } This makes the function that actually computes the value of the function we care about. We could have lots of additional functions here (that are R->R functions), but we only need one for our purposes. (The actual Parser::Function::numeric class that we are taking advantage of defines sin, cos, asin, log, and so on.)  package main; This ends the package and goes back to the "normal" namespace.  sub f {Parser::Function->call('f',@_)} This makes the f(x) function work in perl expressions. It calls the Parser's function-calling mechanism to look up the function 'f' and call it with the appropriate parameters. One of the consequences of this is that if you call f with a formula as its argument, instead of having f try to evaluate itself on the formula, you get back a formula containing f applied to your original formula. E.g.,f(Formula("x+1")) produces the same thing as Formula("f(x+1)"). This is not what would have happened without f being defined this way (see my next message).  Context()->functions->add(f => {class => 'f', TeX => 'f'}); This adds the function f to the current context so that it can actually be included in parsed formulas. Here, "f =>" tells the Parser that the name of the function as it will appear in the student's answer (or the string to Formula) is 'f'. The class => 'f' says that the package that should be used to handle the funciton is package f, which we defined above, and TeX => 'f' says that it's TeX form should a plain f followed by its arguments. (Without this, the f would appear in \rm, which is how most named functions like sin and tan are supposed to be represented). Once this command is given, "f(...)" can appear in the strings passed to Formula() or in student answers that use this context. So setting $g = Formula("f(x+1)") is now allowed, and would not be if the Context()->functions->add had not been given. The remainder of code should be self-evident. An slight modification to this is the following:  DOCUMENT(); loadMacros( "PGstandard.pl", "Parser.pl", "PGcourse.pl", ); TEXT(beginproblem()); Context("Numeric"); my$x = Formula('x'); my $a = 1; my$b = -2; $f = sqrt($x+$a)+$b; package f; our @ISA = qw(Parser::Function::numeric); sub f {shift; $main::f->eval(x=>shift)} package main; sub f {Parser::Function->call('f',@_)} Context()->functions->add(f => {class => 'f', TeX => 'f'});$g = f($x+1); BEGIN_TEXT (f(x+1)) = {ans_rule(30)} END_TEXT ANS($g->cmp); ENDDOCUMENT(); There are two main changes in this example: first, the actual value of the function is not hard-coded into the subroutine in the package, but is defined in a Parser formula outside the package. (The subroutine asks the formula stored in $f to evaluate itself with x equal to the value passed to the subroutine.) Second, rather than calling Formula() directly, the values of $f and $g are produced automatically by the overloaded operators acting on the formula held in $x, which is set to Formula('x') earlier in the file, so once this formula is generated, all the other formulas can be produced simply by writing what look like normal perl expressions. I include this example mostly as a sample of how to create formulas in this non-traditional way. You should use whichever form seems most natural to you (or most convenient at the time). Notice that formulas produced in this way need not be assigned to variables in order to be useful. For example, the answer checker could have been produced as  ANS(f($x+1)->cmp); instead. The addition of named functions to the context is, perhaps, more cumbersome than necessary. (On the other hand, it was not possible at all with the previous parser). I have some ideas about how to simplify the situation. Davide <| Post or View Comments |> Davide P. Cervone - Re: user defined functions in the parser (a bug?) 7/11/2005; 11:51:06 PM (reads: 1667, responses: 0) I should probably also explain why your attempts outlined in your original posting didn't work, and what the various commands actually did. $x = Formula('x'); This makes $x contain a formula whose value is the variable 'x'. This is useful in creating more complicated formulas as described in the previous message. It is used that way to create $g and $h below, but not in the other commands.  Formula('sqrt(x+1)-2')->perlFunction('fff'); This is equivalent to  sub fff {my$x = shift; return sqrt($x+1)-2} (actually, there is some error checking for the number of parameters passed to the function, and the return value is converted to a Real object, but those things would just get in the way here). $g = fff($x+1); This is a little more interesting. Since $x is a formula, $x+1 is the same as Formula("x+1"), and so you are passing that formula to the subroutine defined above. So the value of $x inside the subroutine is the formula x+1. Fortunately, the operators and functions have been overloaded to handle formula objects, and the result of fff will be a formula equivalent to Formula("sqrt((x+1)+1)-2"). (Note that this is not equal to Formula("fff(x+1)"), which would, in fact, produce an error, since the function fff is not defined in the current Context().) If you printed the value of $g->string, you would get sqrt(x+1+1)-2 not fff(x+1). $f = Formula('sqrt(x+1)-2'); This is straight-forward, it just assigns a formula to $f. Note that this is just a Formula object, not a perl function. $ff = $f->perlFunction; This assignes an anonymous code reference to $ff, and is essentially the same as  $ff = sub {my$x = shift; return sqrt($x+1)-2} Note that this is not a parser object; it is a perl code reference. $h = &$ff($x+1); This works much like the assignment to $g above. Again, $x+1 is the same as Formula("x+1"), and that formula is passed as the argument to the function that is pointed to by the variable $ff. As before, the function's local variable is then the Formula x+1 and the overloaded sqrt, plus and minus make its return value equivalent to Formula("sqrt((x+1)+1)-2"). So $h and $g turn out to be the essentially the same thing (a Parser Formula object for the expression sqrt(x+1+1)-2). Now for the ones that produced errors: $k = Formula("fff($x+1)"); Because$x is a Formula object, it stringifies as "(x)", so the above (after variable substitution) is  $k = Formula("fff((x)+1)"); but since there is no function fff defined in the context, you will get a run-time error when this command is executed saying something like "The function 'fff' is not defined in this context", which is perfectly true. $k = Formula("fff(x+1)"); This produces the same error for the same reason. I would think that fff now behaves like sin. This is incorrect, because sin has been added to the Numeric context, while fff hasn't. It would be disasterous to make every function that perl knows about be a valid function in the student's answer. For example, do you want students to be able to enter "ans_rule(10)" as a valid answer? Obviously not, so you need a way of telling the Parser what functions are actually valid ones for student answers, and what ones aren't. That is where the Context object comes in; it is what defines the things that are valid in formulas, and what those things mean. What is the difference between fff and sin? The difference is that sin has been defined (in PGcommonFunctions.pl, which is loaded by Parser.pl) to be  sub sin {Parser::Formula->call('sin',@_)} and has been added to the Numeric (and other contexts) with the equaivalent of  Context("Numeric")->functions->add(sin=>{class=>"Parser::Functions::numeric"}); and in pg/lib/Parser/Functions/numeric.pm there is a package Parser::Functions::numeric that contains a definition like sub sin {shift; CORE::sin(@_)}. These are exactly the things you ended up needing to supply when you defined your own f correctly in your later example. The fact that the Parser knows about sin has nothing to do with the fact that there is a perl sin() function by default.  $k = Formula("$ff($x+1)"); Since $ff is a code reference, its string version is something like CODE(0x944d9c) so after variable substitution, this is  $k = Formula("CODE(0x944d9c)((x)+1)"); which will produce an error since CODE is not defined in the Parser's Context. $k = Formula("&$ff($x+1)"); Note that function evaluations are not performed inside double quotes (only variable substitution), so this is essentially  $k = Formula("&CODE(0x944d9c)((x)+1)"); and this time the Parser will complain about the '&', since that has no defined meaning in the Context. $k = Formula("$f($x+1)"); # makes a product instead of function evaluation In this case, since $f contains a Formula object, and those stringify as the formula in parentheses, this becomes $k = Formula("(sqrt(x+1)-2)((x)+1)"); or a product, as you suggested. Note that a Formula object is not a function, it is just a mathematical expression. You can perform subsitution on it if you want, but not using function notation like that. For example,  $f->replace(x=>$x+1); would produce a result equivalent to Formula("sqrt((x+1)+1)-2"). Note also that perlFuncion produces just that, a perl function, and does not add anything to the Context. The reason you might want this is, for example, to pass the function to the graphing routines to produce on-the-fly-graphics (as your problem suggested), or to make it easier to evaluate the formula at various points, etc. Registering a function for use within parsed strings is a completely different action. Hope this clarifies some of the issues with functions, formulas and the Parser. Davide <| Post or View Comments |> Davide P. Cervone - Re: user defined functions in the parser (a bug?) 7/12/2005; 12:07:38 AM (reads: 1656, responses: 0) One final note. You should be aware that the student does not have to enter the answer as f(x+1), or use f in any way in his or her answer. For example, he or she could enter sqrt(x+2)-2 and get the correct answer. So if the student recognizes the curve, he or she may enter the formula directly without the composition. One possible solution to this would be to disable the sqrt() in the student answers so the only way to specify the correct answer would be via f(). For example, you could put  Context()->functions->disable('All'); just before the line that adds your function to the Context. This will make f be the only function allowed in the student's answer, which should be sufficient to prevent the student from specifying the composition without it. Davide <| Post or View Comments |> Nandor Sieben - Re: user defined functions in the parser (a bug?) 7/12/2005; 12:15:11 AM (reads: 1696, responses: 1) Thank you very much for the explanation and the fix. Where exactly do I find the latest version of pg/lib/Parser/Context/Default.pm ? The exact cvs command would be best. Nandor <| Post or View Comments |> Nandor Sieben - 7/12/2005; 12:15:48 AM (reads: 1677, responses: 0) <| Post or View Comments |> Davide P. Cervone - Re: user defined functions in the parser (a bug?) 7/12/2005; 7:12:30 AM (reads: 1938, responses: 0) Where exactly do I find the latest version of pg/lib/Parser/Context/Default.pm ? The exact cvs command would be best. If you got your original copy of WeBWorK via CVS, then the easiest thing to do is just cd to the pg directory and do cvs update, which will update everything in pg that has changed since you got it (this is generally safe to do in pg). Of you could cd to the pd/lib/Parser/Context directory and to cvs update Default.pm to get just that file.  cd path-to-pg/pg/lib/Parser/Context cvs update Default.pm If you didn't use CVS to get WeBWorK originally, you could go to and right-click (on a PC, or control-click on a Mac) on the "download" link to get a menu that contains something like "Save linked file as..." and use that to save it to you local disk, then move it to your server. Alternatively, you could follow the instructions at for setting up anonymous CVS access, and then use the command  wwcvs -d :ext:anoncvs@cvs.webwork.rochester.edu:/webwork/cvs/system \ checkout pg/lib/Parser/Context/Default.pm to get Default.pm (it will make a pg/lib/Parser/Context directory in the current directory in which to put it). <| Post or View Comments |> Davide P. Cervone - Re: user defined functions in the parser (a bug?) 7/12/2005; 11:25:00 PM (reads: 1696, responses: 2) OK, after having this discussion about adding functions to the Parser, I decided to make it a bit easier, so I have just added parserFunction.pl to pg/macros that should make it easier to define a function within the current context given a Formula for its result. To create a function that can be used in Formula() calls and by students in their answers, use the parserFunction() routine, as in the following examples:  loadMacros("parserFunction.pl"); parserFunction(f => "sqrt(x+1)-2"); $x = Formula('x'); parserFunction(f => sqrt($x+1)-2); parserFunction("f(x)" => "sqrt(x+1)-2"); parserFunction("f(x,y)" => "sqrt(x*y)"); The first parameter to parserFunction is the name of the function or the name with its argument list. If no argument list is given, the names of the variables are taken from the formula for the function, and are taken to be in alphabetical order. The second argument is the formula used to compute the value of the function. It can be either a string or a Parser Formula object. After issuing one of the parserFunction commands above, the function 'f' will have been added to the current Context and will be allowed within Formula() calls and in student answers that use this Context. In addition, a perl function called f() will be defined that also computes the value of f, so you can call this within the body of the problem if you want to evaluate 'f'. This function will act like the other Parser-aware functions, and return a Formula object if it is passed a Formula as one of its arguments. This should make it easier to create problems that use custom functions within student answers, at least when those functions are computed by a simple formula. Davide <| Post or View Comments |> Bob Byerly - Re: user defined functions in the parser (a bug?) 8/12/2005; 1:54:05 PM (reads: 1815, responses: 1) Davide, parserFunction should be very useful, not only for custom functions in student answers, but also because there are lots of variants of names of elementary functions in various textbooks. (This fall, I'll need to teach from a textbook using Yet Another Notation for the inverse trig functions.) While we're on the subject, is there an easy way to add a differentiation method to a user-defined function? I'm looking at the examples in pg/lib/Parser/Differentiation.pm and I think I can figure it out, but thought I'd check to see if you had any comments first. I presume there's a reason this is part of the Parser package and not the Value package as I would have expected. Would it be hard to add this capability to parserFunction? Thanks, Bob <| Post or View Comments |> Davide P. Cervone - Re: user defined functions in the parser 8/12/2005; 10:07:15 PM (reads: 2040, responses: 0) If you want to have more than one name for a given function, it is probably better to use  Context()->functions->add(newname => {alias => 'oldname'}); rather than a parserFunction, as this will be more efficient. But you're right, it could be used to do that.  is there an easy way to add a differentiation method to a user-defined function? I'm looking at the examples in pg/lib/Parser/Differentiation.pm and I think I can figure it out, but thought I'd check to see if you had any comments first. Since the parserFunctions are produced from Formula objects originally, they can take advantage of the Parser's differentiation feature to produce the derivative for them automatically, using the chain rule. I have modified the parserFunction.pl file to implement this, so you don't have to worry about defining your own differentiation formulas. So if you did  parserFunction(f => "1/x"); $F = Formula("f(x^3)")->D; then$F would contain the equivalent of (-1/(x^3)^2)*3*x^2. Note, however, that I have only implemented differentiation for single-variable functions. It would be possible to form the matrix of partial derivatives and use the multivariable chain rule, but I didn't go to the trouble of implementing that right now. It's something to add in the future.  I presume there's a reason this is part of the Parser package and not the Value package as I would have expected. The Value package is for constants of the various types. The Parser package is for formulas. Since the differentiation is done algebraically on formulas, the differentiation routines are part of the Parser package. The reason for the confusion is probably that the Formula object is part of the Value package as well. This was done so that all the things that the user normally has to handle are all in the Value package (and they have a consistent interface and common set of methods). The Value package is what overloads the operators, and so forth, so it is the Value package that lets formulas be added to each other to create new formulas and lets you compare two formulas for equality. The Parser package creates and manipulates formulas, but the Value::Formula package is the one that actually USES the formulas. The differentiation is done by asking the various nodes of the parse tree to differentiate themselves. Since these nodes are in the Parser package (e.g., Parser::BOP::add for the addition operator), the differentiation information must be in the Parser package. The Differentiation.pm file adds new methods to the various Parser objects so that they know how to differentiate themselves. Originally, there was a strong distinction between Value and Parser, and the Value package could be used independent of the Parser (except for the Formula object). As features are being added, however, that distinction is no longer so sharp, and the Value package would no longer operate completely without the Parser. It was a good idea at the time, but turns out to have caused more trouble than it was probably worth. (One of the main problems is that there are really two types of Context objects, one in Value and one in Parser. The Parser one is a subclass that extends the Value one to include the new items that the Parser needs. The Value package was supposed to remain unaware of this extension, but that has not turned out to be practical. So there is confusion in some parts of the Value code about what Context information should be looked at.) Anyway, the differentiation should work for you now. Davide <| Post or View Comments |>