One of my colleagues (who is involved with problem authoring at our institution) posed the following question:
Has anyone done research on the optimal level of randomization for online homework problems in mathematics?
Having too few values for a coefficient (less than 5, say) runs the risk that adjacent students doing a quiz might see the same problem, and features like "Show me another" become problematic.
Having too many values makes it increasingly likely that one set of values breaks the problem, and makes it much harder to ensure that all versions of a given problem have similar difficulty.
If anyone has thought about this, please let us know!
Personally, I construct WeBWorK problems in reverse - starting from the solutions in order to come up with the problem. In this way, one can ensure that problems are not "broken" by an errant randomization. It also helps to ensure that sufficient randomizations are possible so that students are unlikely to encounter the exact same problem.
Also, the Show Me Another feature simply will not work if there are insufficient randomizations possible, so that should not be a concern.
Thanks. We've found that some randomization scenarios are easier to implement than others (and more useful). There are a lot of problems where the randomization is pretty artificial, like "evaluate the integral of a*cos(x)" that I wouldn't really consider random, and of course, lots of well-randomized problems as well. For calculus it's not too hard to get a reasonable degree of randomization.
Harder to accomplish, even working backwards, are problems like computing eigenvalues and eigenvectors. Let's say they start with a 3x3 matrix, and you want the outcome to be two (random integer) eigenvalues, one repeated, with the repeated eigenvalue having a two-dimensional eigenspace. Ideally the eigenvectors should have some degree of randomness to them as well.
I can easily create a matrix with these properties, but the hard part is ensuring that all students will encounter a comparable level of difficulty when computing and factoring the characteristic polynomial, especially if they are expected to do so without the aid of technology.
Harder to accomplish, even working backwards, are problems like computing eigenvalues and eigenvectors. Let's say they start with a 3x3 matrix, and you want the outcome to be two (random integer) eigenvalues, one repeated, with the repeated eigenvalue having a two-dimensional eigenspace. Ideally the eigenvectors should have some degree of randomness to them as well.
I can easily create a matrix with these properties, but the hard part is ensuring that all students will encounter a comparable level of difficulty when computing and factoring the characteristic polynomial, especially if they are expected to do so without the aid of technology.
You're right, problems such as the one you mentioned stray into territory where you want to be able to accept multiple answers. In these cases, working backwards can help to cap the complexity, but you also have an issue with accepting a variety of student responses (as a 2-d eigenspace can have many correct eigenvalues...)
I've taken the liberty of coding up an example for the problem you've suggested - perhaps you'll find it useful. I don't know how specific you'd want to be when grading student responses, so I've included the requirements that the eigenvectors for 2-d eigenspace be orthonormal. These lines in the answer checker may be removed - but the check for orthogonality should be replaced by a check for linearly dependent eigenvectors.
Thanks for this. I like the way the code has been set up. I think there is one typo: the dot product should be $u . $w in the check for orthogonality.
I'll hand this off to the students working on linear algebra problems this summer. Probably I'll have them replace orthogonality with independence -- students will solve this type of problem by finding the basic eigenvectors (i.e. basic solutions to the usual homogeneous system) and these are usually not orthogonal.
I seem to recall that there is a macro file for linear algebra that contains functions like a basis checker. That might be a good replacement for the orthogonality check.
I'll hand this off to the students working on linear algebra problems this summer. Probably I'll have them replace orthogonality with independence -- students will solve this type of problem by finding the basic eigenvectors (i.e. basic solutions to the usual homogeneous system) and these are usually not orthogonal.
I seem to recall that there is a macro file for linear algebra that contains functions like a basis checker. That might be a good replacement for the orthogonality check.
LOL, yes. I did some last-minute rearranging of the variables, and must have missed that line. (It should be `until ( $v . $w == 0 );` right there at the beginning...)
I've not written any linear algebra problems (except this one, of course), so I can't speak to the available contexts or macros - but if all you want is a check for independence, then checking that the cross product of the $sVec variables is not a zero vector would be all you'd need. (e.g. `norm($sVec1 x $sVec2) != 0` etc)
It's more work, but I recommend always coding an algorithmic solution. This way, you can (almost) confirm that the level of difficulty in the solution steps that a student would go through is pretty much the same. For example, with the 3x3 matrix problem, you could code steps for writing and simplifying the determinant polynomial, with checks along the way that no values have gotten too cumbersome. You could code the search for eigenvectors one step at a time, making sure that the requisite row reductions never need too-large coefficients.
As a Perl trick, I find it useful to encompass blocks of code in:
do { CODEBLOCK } until (condition);
as a way to work in those checks that values have not grown too cumbersome.
To gauge how random your versions really are, if you find the problem in the Library Browser, you can repeatedly hit the re-randomize icon to see. Otherwise you can manually type in new seeds using the editor before clicking to View.
Anyway, having an algorithmically coded walk-through solution carries side benefits in the long term. I think it is worth the effort if you can afford the time, and it helps put the randomization in the Goldilocks zone.
As a Perl trick, I find it useful to encompass blocks of code in:
do { CODEBLOCK } until (condition);
as a way to work in those checks that values have not grown too cumbersome.
To gauge how random your versions really are, if you find the problem in the Library Browser, you can repeatedly hit the re-randomize icon to see. Otherwise you can manually type in new seeds using the editor before clicking to View.
Anyway, having an algorithmically coded walk-through solution carries side benefits in the long term. I think it is worth the effort if you can afford the time, and it helps put the randomization in the Goldilocks zone.
Thanks for the suggestions.
I don't have the time (or the coding expertise) myself, but we've got some summer students writing problems for us. I'll pass this suggestion along to the person managing that project.
I don't have the time (or the coding expertise) myself, but we've got some summer students writing problems for us. I'll pass this suggestion along to the person managing that project.