Below I have a question about WeBWorK's capabilities, followed by an announcement, followed by a request for input to a checklist I am building.
- Most problems involve randomization. Every problem in a set uses a 4-digit seed which determines the "random" variables used in the problem. Would it be possible to have WeBWorK execute a procedure that would compile a problem using every single seed, one at a time? And would this be an outrageous computational load if it were run on a library of say 1000 problems?
We'd be looking for compiling errors, like accidental division by 0. We'd be looking for TeX errors in the hardcopy version. We'd also want the procedure to take the literal text of the displayed correct answer and submit that as an answer, checking that the displayed correct answer at least actually counts as correct. (These issues might only arise for one seed in a thousand.) We'd also check that different seeds do in fact tend to lead to different problems.
- Of course there is a lot more to be done when debugging a library for say, preparation for the OPL. One thing to do is to tidy up the code for readability. I'm working with a colleague on a script that can be run on a pg file to automatically indent and space out the code appropriately.
- Then I have this checklist, and I would like to know if anyone can add to it. It's hard for me to imagine a way to automate checking most of these items - I think they mostly need to be checked by hand from problem to problem.
- Does the problem use Math Objects if it's at all feasible to use them?
- Is the problem's displayed correct answer actually mathematically correct?
- Does the problem accept the displayed correct answer as correct? (would be automated via item 1)
- Does the problem accept alternative expressions for the correct answer? (For example I just cleaned a problem that would accept 26/3, but not 26/3+ln(8)-ln(8))
- Is the problem appropriately randomized enough? (would be automated via item 1)
- Does the problem appropriately allow/disallow decimal approximations to correct numerical answers? Ditto with improper fractions and mixed numbers.
- Is the expected correct answer of the appropriate Math Object type?
- Are MathType type warnings turned off if they provide too big of a hint?
- Are incorrect answers counted as incorrect?
- Are incorrect answers that are not even of the right type given appropriate type warning messages?
- Is the wording of the problem acceptable?
- Are there any typos in the display of the problem on-screen or in the hard copy?
- Do you have any other observations for improvement of this problem, such as its layout on screen?