Is it possible to see the distribution of answers to a specific question across all students? Having this capability would help me identify the common mistakes
and address those in class.
The "Answer log" page gives this information, but only to a single student at a time. When you have 150 students, checking each one separately would take a while.
Yoav
I've thought about this myself. However, most problems in WeBWorK (at least what I consider good problems) involve a random element in that coefficients or some other aspect of the problem is randomly chosen.
I have never figured out how to parse such a table so that it makes any sense in terms of finding patterns of mistakes.
On the other hand, I have thought about using something similar to "Answer Hints" and code what I would expect students to answer based on known common mistakes. It would then be possible to count how many students' answers matched the corresponding predicted mistake answers.
However, this would require significant work, both to the problem library and to the supporting WeBWorK system. The "pg" file defining problems would have to include some type of coded method (like ANS) that would define expected answers based on common errors. Then the student evaluation routine of WeBWorK would parse the same PG files but would interpret them to compute predicted erroneous answers and then check the student answers in the database for matches to these answers.
I have settled on giving students feedback for common mistakes within the problem using the AnswerHints macros. But this does not let me, as instructor, know which/how many students are encountering these issues.
Best wishes,
D. Brian Walton
I have never figured out how to parse such a table so that it makes any sense in terms of finding patterns of mistakes.
On the other hand, I have thought about using something similar to "Answer Hints" and code what I would expect students to answer based on known common mistakes. It would then be possible to count how many students' answers matched the corresponding predicted mistake answers.
However, this would require significant work, both to the problem library and to the supporting WeBWorK system. The "pg" file defining problems would have to include some type of coded method (like ANS) that would define expected answers based on common errors. Then the student evaluation routine of WeBWorK would parse the same PG files but would interpret them to compute predicted erroneous answers and then check the student answers in the database for matches to these answers.
I have settled on giving students feedback for common mistakes within the problem using the AnswerHints macros. But this does not let me, as instructor, know which/how many students are encountering these issues.
Best wishes,
D. Brian Walton
You can actually enter wildcards in the fields at the top of the Show Past Answers form to see more than one student or more than one problem at a time.
For example, entering "*" in the student field will get you the answers for all the student for that problem, or entering "*" in the problem number field will get you all the problems for a given student. I don't remember if they use regex syntax (with * handled as a special case), or file glob wildcards, but expect the former.
Davide
For example, entering "*" in the student field will get you the answers for all the student for that problem, or entering "*" in the problem number field will get you all the problems for a given student. I don't remember if they use regex syntax (with * handled as a special case), or file glob wildcards, but expect the former.
Davide
Thanks guys,
I have found a slightly different way to analyze the answers - download the
file "answer_log" and use my own script to do it.
Here are two lines from the file:
[Sun Aug 19 00:40:19 2012] |yfreund|multi-Step_questions|1|0000000 1345351219 1 1 0
[Sun Aug 19 00:41:01 2012] |yfreund|multi-Step_questions|1|1000000 1345351261 {0,2,3,4,5,6,7,8} 1 0
Most of the fields in each line are self explanatory. But what is 1345351261 ?
Is it maybe the random seed for this procedure?
Is it possible to add to the log file the correct answers?
Actually, now that I think of it. There is a simple solution, I can
ask the students to type in the problem parameters, this is not much work for the student, and would provide me with the input they saw as part of the log file. From this input I can compute the correct answer.
Yoav
I have found a slightly different way to analyze the answers - download the
file "answer_log" and use my own script to do it.
Here are two lines from the file:
[Sun Aug 19 00:40:19 2012] |yfreund|multi-Step_questions|1|0000000 1345351219 1 1 0
[Sun Aug 19 00:41:01 2012] |yfreund|multi-Step_questions|1|1000000 1345351261 {0,2,3,4,5,6,7,8} 1 0
Most of the fields in each line are self explanatory. But what is 1345351261 ?
Is it maybe the random seed for this procedure?
Is it possible to add to the log file the correct answers?
Actually, now that I think of it. There is a simple solution, I can
ask the students to type in the problem parameters, this is not much work for the student, and would provide me with the input they saw as part of the log file. From this input I can compute the correct answer.
Yoav
I'm guessing that that mystery field is some kind of universal time stamp. I notice that its ones digit always agrees with the ones digit of the seconds field of the time that the answer was submitted.
The time stamp field is a Unix timestamp, measuring "the number of non-leap seconds since whatever time the system considers to be the epoch," which on most systems is 00:00:00 UTC, January 1, 1970.
To wit:
$ perl -e 'print scalar(localtime("1345351219")),"\n";'
Sun Aug 19 00:40:19 2012
Gavin
What are the fields?
I think I have determined:
timestamp | user | assignment | question_number | ? | UTC_seconds | answer
What is the field where the question mark is? I've looked at the values in that field from the answer log that I have downloaded, and it appears to be something coded in binary, but I'm not sure what.
1s and 0s for if each answer within the question was scored as fully correct or not.
Up until now, I had been using that field to determine the %correct, by dividing the number of 1's by the length of the string (I always use equal weighting).
But I noticed on a recent assignment of a MathObject List answer, that that field only had a single '0' or '1', even though the assignment was giving partial credit.
Is there anywhere that that "partial credit" is recorded by answer (with time stamp)?
BACKGROUND (for the curious):
I have been using my own Python script to parse the answer logs to only accept answers that were submitted on webwork BEFORE they turned in a PDF copy of their assignment, so that students cannot turn in the PDF on time, but then update their webwork answers before I download the grades.
I cannot just have the assignment close at a particular time, because students are permitted to turn in the assignment late with a late penalty. So most students turn in the assignment on time, but some are still working on the assignment up to 3 weeks later. My script ensures that, once the assignment is closed and I grade everyone all at the same time, I only count the answers in the answer log BEFORE the timestamp of when the students turned in their PDF copy.
But I noticed on a recent assignment of a MathObject List answer, that that field only had a single '0' or '1', even though the assignment was giving partial credit.
Is there anywhere that that "partial credit" is recorded by answer (with time stamp)?
BACKGROUND (for the curious):
I have been using my own Python script to parse the answer logs to only accept answers that were submitted on webwork BEFORE they turned in a PDF copy of their assignment, so that students cannot turn in the PDF on time, but then update their webwork answers before I download the grades.
I cannot just have the assignment close at a particular time, because students are permitted to turn in the assignment late with a late penalty. So most students turn in the assignment on time, but some are still working on the assignment up to 3 weeks later. My script ensures that, once the assignment is closed and I grade everyone all at the same time, I only count the answers in the answer log BEFORE the timestamp of when the students turned in their PDF copy.
The scores string saved to the answer log hasn't changed. If the problem only has one answer, then the score string will be a single '0' or '1' even if the problem gives partial credit. For example, if you have a list answer and a score of 0.5 (50%) is awarded, then the answer log will have '0' recorded. The score of 0.5 is recorded for the user problem in the database, but it does not have a timestamp associated.
The past answer database table also records the timestamp and the same score string. That would be a more reliable place to get this information because it is easier to parse than a text log file. Python has database libraries that can handle this nicely. However, it still has the exact same score string that won't reflect partial credit.
This design with the score string is quite old, and certainly did not have partial credit in mind. It is an ugly limitation of the answer log and past answer database table.
Thank you very much for that information! Sorry, I wasn't trying to suggest that anything changed with the software when I said "up until now". I was just indicating why I was inquiring about the partial credit data to hopefully make my intent clear.