Below is an e-mail discussion I had with Michael Gage of Rochester about completion time data, if you would like details.
Daniel Smaltz
Worcester Polytechnic Institute 2009
Daniel Smaltz wrote:
Hello,
Thanks very much for both of your replies of January 21 in response to our question about measuring how much time a student spends taking a WeBWorK placement test. In the time since your reply, we have looked into this question fairly thoroughly and found it to be an interesting line of inquiry. We obtained our time-based data from the "transaction.log" file, which ended up being relatively easy to edit using Microsoft Excel, allowing us to sidestep the use of database commands per se. We defined a student's "completion time" as the time from the submission of their first answer to the time of their last answer. We found some interesting trends -- we noticed that while many of our students completed their placement test in one brief sitting, many others logged out of their first session and resumed hours or days later. We tried a variety of methods for finding a correlation between completion time and a student's ultimate score, since we thought that students with a stronger math background might finish their test more quickly. On the contrary, we found essentially no correlation between completion time and total score. Seen from another perspective, however, this is encouraging: Even if we allow students unlimited time to complete the tests, their scores will not be correlated with their completion time, but with other factors -- hopefully their mathematics background!
I will post our e-mail discussion on the main WeBWorK forum to see if anyone else has analyzed completion time for a WeBWorK assignment in the same or similar ways. Again, thanks very much for your advice.
Cheers,
Daniel Smaltz and Shannon Leary
Worcester Polytechnic Institute
Hi,
We don't automatically timestamp homework all submissions in the database. (It's been proposed as a feature -- but not yet implemented.) One thing that is recorded is the number of incorrect attempts before the student gets the answer correct -- and that might correlate with the information you want. It's pretty easy to get from the WeBWorK instructors pages.
There is a log file attached to the apache server that timestamps all activities, but it might take a significant amount of work to extract the data you want from this log file.
As to other schools using WeBWorK for placement and literature on the subject -- I suggest that you post this message on the main forum at http://wwrk.maa.org/moodle/course/view.php?id=3 -- you'll get responses from a number of schools using WeBWorK for placement tests and you might get clever ideas for extracting the amount of time spent from the data that is collected. We haven't used WeBWorK in this way at the University of Rochester, so others in the WeBWorK community will be more knowledgeable.
The gateway test mode does record additional information beyond the homework mode which might make it easier to get the data you want. Gavin LaRose at the University of Michigan is our expert on that subject. He also uses WeBWorK for placement.
So I suggest as next steps that you (1) -- post this message to the main forum. and (2) contact Gavin (glarose@umich.edu) and see if he has further suggestions.
Take care,
Mike
On Jan 21, 2008, at 1:54 PM, Heinricher, Arthur C. wrote:
Dear Professor Gage,
Worcester Polytechnic Institute recently began using WeBWorK for an internet-based calculus placement test that was given to entering freshmen this past summer. The students take the test on-line before arriving on campus for their first year and the results are used to recommend a first calculus course at WPI. As part of a required university project, we are working with Professor Heinricher to conduct a validation study of this placement test.
We want to determine how much time a student actually spent taking the placement test. We know that WeBWorK logs the time a student submits an answer for any question so it should be possible to determine how much time a student actually spent on the test. (A student who takes 30 minutes to get a good score is probably better prepared for calculus than a student who spends 3 days to get the same good score.) Is there a simple procedure (or maybe not so simple procedure?) for retrieving these times?
We have also begun a literature review to find other schools who use WeBWorK in the same way. If you know of any examples and could point us toward them, we would be most grateful.
Thanks in advance for any help you can provide.
Best regards,
Shannon Leary and Daniel Smaltz
Worcester Polytechnic Institute