## Forum archive 2000-2006

### Nandor Sieben - undef as result for function evaluators ### Nandor Sieben - undef as result for function evaluators

by Arnold Pizer -
Number of replies: 0 undef as result for function evaluators topic started 5/31/2005; 7:56:44 PMlast post 6/2/2005; 6:59:58 PM Nandor Sieben - undef as result for function evaluators 5/31/2005; 7:56:44 PM (reads: 1573, responses: 10) Suppose the correct answer in a problem is the function ln(-x). Then the function evaluator cannto use the default [0,1) interval to check if the student answer is correct. It needs to use something like [-10,0). Now if the student tries the answer ln(x) then the evaluator rejects this as an incorrect answer (this is good) but it also gives a warning that ln(x) cannot be evaluated at some negative number (this is bad). Is there any way to avoid this warning? Is there any reason that undef is not a completely legal output for function evaluation that does not cause any errors? In fact if the correct answer is x/|x| then I'd like to compare the student answer at the points -2, -1, 0, 1, 2 and I'd like to make sure that the student function and the correct function both evaluate to undef. Currently I'd get an error since 0 is not in the domain. Having undef as a valid value that can be compared would make the choice of the test points simpler, perhaps even more robust. An interval of [-20,20] with 30 test points would work in almost all cases even if the answer in not definded on the interval [-2,infty). Nandor <| Post or View Comments |> Davide P. Cervone - Re: undef as result for function evaluators 5/31/2005; 9:55:46 PM (reads: 1837, responses: 0) Nador: I was always bothered by the ln(-x) and ln(x) problem that you mentioned as well, and so the new Parser does not generate an error message if the student's answer does not evaluate at one of the points that the professor's answer does (it is quietly marked as incorrect). There are situations where this is not really the right thing to do (and that is why the original answer checkers report errors), but I think the number of instances where that is the case is so much smaller than the number of times this IS what you want to do, I chose to not report those errors. It would be possible to add an object class to the Parser for the indefined value (similar to the one for infinity), but I'd need to think more about some of the implications. It could not be used with adapting parameters, for example. Some things to consider: If the professor's function were sqrt(x-30), for example, it would produce undef all the time on your interval; should that match sqrt(x-40)? I suppose one could require that some portion of the points produce something other than undef. If the professor's answer were ln(x) and the natural range of x's in the problem are all positive, but the professor didn't think to restrict the interval to (0,20], a student who entered ln(abs(x)) would be marked wrong, since this would not produce undef on the negative values while ln(x) would. Granted, this is the professor's mistake for not restricting the x's to the correct range, but most problem writers don't think about these issues, and it is not always easy to tell what the right range really IS. Also, but making it work in more cases, you are making it even less likely that they will think about it, and are more likely to be caught by these situations. Another posibility would be to report errors if the professor's answer is undefined unless the professor specifically indicates a point or range of points to check for undefined. That way, the problem writer must actively request that undefined values be accepted, and so will have to have throught about it before it is allowed. As for [-20,20] working most of the time, I think you will find that this is not the case. There are lots of chain-rule and exponential growth situations where this large a range can cause problems. As some of the developers are fond of saying, the function comparison algorithm has no real right to work, and the fact that it does as well as it does is remarkable. The issues involved are extremely subtle and counter-intuitive at times, and a change of this type would require pretty extensive testing to earn the level of trust that the current method has. It's certainly worth trying out. I'll put it on my list of things to try in the new Parser. Davide <| Post or View Comments |> Nandor Sieben - Re: undef as result for function evaluators 6/1/2005; 12:16:46 AM (reads: 1805, responses: 0) > If the professor's function were sqrt(x-30), for example, it would > produce undef all the time on your interval; should that match > sqrt(x-40)? I suppose one could require that some portion of the > points produce something other than undef. Well, the [-20,20] interval certainly doesn't work in this situation. The suggested solution should work though. Maybe give a warning if more than half of the tested values are undef. The default input set could be something like {-1000,1000} U [-20,20]. I realize nothing is going to work completely but allowing undef instead of error might give more flexibility even when the default needs to be changed. It might be a good idea not to have any defaults and force the instructor to define a domain. >If the professor's answer were ln(x) and the natural range of x's in > the problem are all positive, but the professor didn't think to > restrict the interval to (0,20], a student who entered ln(abs(x)) > would be marked wrong, since this would not produce undef on the > negative values while ln(x) would. It's arguable but I believe ln(abs(x)) is an incorrect answer in this case and it should be rejected. Nandor <| Post or View Comments |> Davide P. Cervone - Re: undef as result for function evaluators 6/1/2005; 7:42:53 AM (reads: 1811, responses: 1) It's arguable but I believe ln(abs(x)) is an incorrect answer in this case and it should be rejected. If the natural range of x's in the problem are all positive (as I stipulated), then ln(abs(x)) = ln(x) for all x, so the functions are equal and I don't see why you think this should be rejected.  I realize nothing is going to work completely but allowing undef instead of error might give more flexibility even when the default needs to be changed. I agree that it is certainly worth looking into, and I've been thinking about how to do it using the new Parser. I only meant to point out that the implications need to be carefully considered. Function comparison is so central to the operation of WeBWorK that changes to it can't be taken lightly. Davide <| Post or View Comments |> Gavin LaRose - Re: undef as result for function evaluators 6/1/2005; 8:03:44 AM (reads: 2142, responses: 0) A similar issue came up when I was working on the gateway/quiz testing module that I've been working on in the past year. In this case (I claim, at least that) it's undesirable to have the evaluator tell the student (e.g., when previewing) if the student answer isn't correct because of a domain error. My work-around was to add a flag to the fun_cmp() evaluator that tells it not to report domain errors. That is, we assume that the evaluator is defined for a domain on which the correct answer is well-defined, so if the student's answer isn't defined there, it's wrong. But the actual error message isn't reported. Gavin <| Post or View Comments |> Nandor Sieben - Re: undef as result for function evaluators 6/1/2005; 11:24:09 AM (reads: 1788, responses: 1) > If the natural range of x's in the problem are all positive (as I > stipulated), then ln(abs(x)) = ln(x) for all x, so the functions are > equal and I don't see why you think this should be rejected. If we want to be absolutely precise then we define a function together with it's domain like f:[0,inf)-> R, f(x)=ln(x). This function is the same as g:[0,inf), g(x)=ln(|x|). We don't wanna use this type of precision in WebWork and we use the standard practice of defining a function with it's formula with the assumption that the domain is the largest subset of R or of C for which the expression makes sense. With this practice f(x)=ln(x) and g(x)=ln(|x|) are not the same. The natural domain for x should not be an extra condition that hides somewhere in the background. The student's answer has the opportunity to encode the information about the natural domain by producing undef outside the natural domain. So why not live with this opportunity if we can. If we asked the questiion this way: Find a formula for f:[0,inf)->R that satidfies blah blah. Then I agree that both ln(x) and ln(|x|) are perfectly correct. I admit that this is splitting hair a little bit but hey we are mathematicians we like splitting hair :-) Nandor <| Post or View Comments |> Davide P. Cervone - Re: undef as result for function evaluators 6/1/2005; 4:57:35 PM (reads: 2084, responses: 0) I disagree with your claim concerning the natural domain of the function, since I said that in the statement of the problem, the assumption is that x is positive, so the largest domain under consideration is already (0,inf). (We may have said that x represents the length of a side of a box or something.) So the natural domain of any function of x considered in the problem can be no bigger than (0,inf). Even if the function could be defined on a lerger domain, that is immaterial to the problem, and the student is entirely correct in disregarding the the values of x outside this range. There is no reason why the student's function must match the professor's function outside the range of x's used in the problem. For example, if the question were what is the length of the diagonal of a square of edge length x, both sqrt(2) x and sqrt(2) abs(x) are correct answers. Since squares only have positive edge lengths, the fact that these functions disagree on negative values of x makes no difference, and the student's answer is correct: any square of edge length x DOES have diagonal of length sqrt(2)abs(x).  If we asked the questiion this way: Find a formula for f:[0,inf)->R that satidfies blah blah. I claim that we have asked it in this way by setting up a situation where we all know that x is positive. Although the explicit notation may not be used, many problems come with implicit domains that are not all of R. Just because the function can be evaluated at non-existent x's, why should it be marked wrong? If the professor's answer is x+1 and the student answers (x^2-1)/(x-1), what do you want to result to be? The current function checker will mark this correct, and I see no hope of coming up with a general algorithm to mark these types of answers as wrong, even if I wanted to (and I'm not sure I do). You claim that WeBWorK should not be concerned with the niceties of domains, but, in fact, your request would require WeBWorK (and the students using it) to be more concerned about domains. That is, they will always have to be thinking about whether their version of the function is defined in more places than the problem calls for, and that is a level of abstraction I don't expect from standard calculus students. Davide <| Post or View Comments |> John Jones - Re: undef as result for function evaluators 6/1/2005; 6:30:27 PM (reads: 1792, responses: 0) Hi, I think the biggest gain webwork can make is to not generate error messages when evaluating student functions. I think it is bad form for webwork to expose internals to students. If the correct answer evaluates and the student's doesn't, then the student answer should be marked wrong. This is one of my favorite parts of Davide's Parser. Almost all of the time, this will be the right thing, and most exceptions can be handled by forcing the domain or evaluation points. If you want to be extra kind to the student, in addition to marking the answer wrong the message could say something along the lines of "the function you answered does not have the same domain as the correct answer". This will make sense to more students than when they enter "sqrt(x)" and get a message of "Error: Can't take sqrt of -0.11". I think that if you want to know if the student knows the proper domain of a function, you should just ask it as a separate part of the question. Embedding this in the function checking is likely to lead to irritated students. It is one thing if they make this kind of error on paper homework, get an explanatory comment and partial credit. It creates different feelings when the machine just says "wrong" with no other help and it seems like a subtle difference was the reason. It is still an interesting idea to allow the correct answer to sometimes be undef. I think its biggest use would be to allow an answer checker to work on essentially a discontinuous domain. On the other hand, since you can hard-wire the evaluation points, that could also be used to check ln(abs(x)) where both positive and negative numbers would be sampled. John <| Post or View Comments |> Nandor Sieben - Re: undef as result for function evaluators 6/1/2005; 7:01:59 PM (reads: 1796, responses: 0) > I claim that we have asked it in this way by setting up a situation > where we all know that x is positive. Although the explicit notation > may not be used, many problems come with implicit domains that are not > all of R. Just because the function can be evaluated at non-existent > x's, why should it be marked wrong? I agree with this, it should not be marked wrong by webwork. On the other hand if I graded it by hand I would write a comment that altough sqrt(2)|x| is defendable since our assumption about x is there somewhere, sqrt(2)x is the better good answer on a painfully pedantric level. > You claim that WeBWorK should not be concerned with the niceties of > domains, No no, I don't want to say this at all. I think everybody should get into the habit of not using the default values for function evaluation and think through what the appropriate domain should be. I suggest the following: 1. Extend the codomain of real valued functions to R U {undef} . 2. Use undef as any other value when comparing functions, without error or warning messages. 3. Report an error or a warning if a large percentage of the test points give undef during function comparison. 4. Change the deafults: 4a. make the interval larger and include negative numbers as well 4b. use more than 3 test points These would be nice and hopefully without problems but I think I'd be happy if I didn't get a warning if the submitted function does not evaluate at some of the test points. Nandor <| Post or View Comments |> Davide P. Cervone - Re: undef as result for function evaluators 6/1/2005; 10:13:10 PM (reads: 1790, responses: 0) Your suggestions  Extend the codomain of real valued functions to R U {undef}.Use undef as any other value when comparing functions, without error or warning messages.Report an error or a warning if a large percentage of the test points give undef during function comparison.Change the deafults:make the interval larger and include negative numbers as welluse more than 3 test points are good ones, and I'll look into adding an "Undefined" object to the parser. For your information, the last few are already part of the Parser. The Parser's default domain is [-2,2], and it automatically discards points where the professor's function is not defined (unless it can't find enough points where it IS defined); if the studen'ts function is onjt defined at one of the points where the professor's IS defined, the student's answer is marked as wrong with no error message. The default number of points can be set in the global.conf or course.conf file, so you can use more if you want. The Parser's default is funcNumOfPoints + 2 (I thought that 3 was too small as well, so I used 5). Davide <| Post or View Comments |> Davide P. Cervone - Re: undef as result for function evaluators 6/2/2005; 6:59:58 PM (reads: 1790, responses: 0) OK, I've implemented the undefined value checking that we have been discussing as part of the function comparison in the new Parser's answer checker. This is an optional feature that you need to enable explicitly by setting the checkUndefinedPoints flag in the Context() or in the formula itself. When this is done, test points where the function is undefined will be retained and the student's answer will have to be undefined at the same points. If the points are selected at random, the points where the function is undefined are kept in addition to the ones where it is defined, so the required number of test points where the function is defined will still need to be able to be found. The number of undefined points can be up to the number of defined points. This number can be reduced by setting the max_undefined flag. If you want to guarantee that a specific undefined point be tested, you can provide that point using the test_points field of the formula. For example:  Context()->flags->set(checkUndefinedPoints=>1); $f = Formula("(x^2-1)/(x-1)");$f->{test_points} = [[-1],,,]; ANS($f->cmp); will guarantee that the singularity at x=1 is tested, so the answer x+1 will not be marked as correct. If an answer matches at all the test points where the functions are both defined, but some of the undefined points differ between the two functions, the answer checker will generate an error message indicating that the domains of the functions don't agree, as in John's suggestion. (This is only generated when the functions match except for that.) This can be controlled by setting the showDomainErrors flag in the formula's cmp() call:  ANS($f->cmp(showDomainErrors=>0)); The default is to show these errors. I hope that satisfies the requirements we have discussed. I think it will be a useful addition to the answer-checker repertoire. Davide <| Post or View Comments |>