We're trying to implement placement exams in WeBWorK. We aren't planning to go live yet (we probably will sometime during 2011, though I'm not ready to predict whether that will be January or July).
I'd like to describe what we're trying to accomplish and what we've done just in case any of the well-informed readership of this forum see any icebergs we should steer away from!
We have two goals:
a) to do more accurate placement than we've done in the past (we'll be able to ask better questions than the multiple choice questions we've asked in the past, and we'll have access to data that we hope will make it possibly continually improve the accuracy of the exams).
b) to reach out to high schools in the state. In Oregon a very high percentage of college bound high school students go to _some_ state institution - not necessarily ours, and the state universities and community colleges share what are supposed to be the same lower division math courses with the same numbers. We hope to give high school students (and teachers) a clearer idea of college mathematics expectations by allowing them to take placement exams online (these won't count except for their own diagnostic purposes - they'd be taken with guest logins).
We've created a placement exam webwork course. My colleague Mike Price has been writing 15 minute gateway exams for the various placement levels we have (we have 4 levels, the top level placement we give is readiness for our calculus course for the physical sciences). He is also writing a number of practice exams for each level. Students will be able to take practice exams on their own, and will have to take the placement exam proctored if they want the score to count.
There are several technical issues:
1) We need to have all students who might legitimately need placement exam scores here to be students in the placement exam course. This is at least 26,000 people. (Our undergraduate population is about 18,000 but there are various other categories that come in here.)
I have a script which takes a csv and adds any new students using wwdb to access the database.
2) All those students need to have all the exams and practice exams assigned to them.
I plan to use the "assign to all students" for the initial assignment (this works, although I estimate it will take roughly 30 minutes per assignment, but we'll do this in the early morning) and then once I've done that, write a script that uses wwdb to assign the exams to the new students as I add them.
3) I'd like to figure out how to report to the student immediately after he or she finishes the placement test (e.g. whether passed, and information about areas of weakness). Ideally this would be on the web page since I'd like this to be available to guest logins (who won't have email).
4) We'll need to generate a report for the registrar's office on placement scores. Probably we'll want to only report changes from the last report.
5) We'll need a way to retire students out of the system I guess, though it is hard to worry hard about that at the moment.
Sorry to be so long-winded here. I'm mostly concerned about whether I'm asking for trouble by creating a course with 26,000 students, and I'm wishing I had better ways to interact with the database (as far as adding students and adding assignments to the students) than wwdb.
Oh, and I have one specific question for anyone who knows the answer. What does the psvn (problem set version number) do? If I'm assigning problem sets by importing the set_user table into the database with wwdb, do I do any harm by always using 1 for the psvn?
thanks for reading this tome, and for any advice,
Hal
Hi Hal,
This sounds very impressive. We use WeBWorK for placement testing, but ours is mostly for our College in the University and only tests enrolled students. Thus we see "only" between 6000 and 7000 students a year for the placement testing.
We did have the complication that we were putting in the WeBWorK test to replace an existing pencil and paper test, for which there was already a scoring and reporting infrastructure. As a result, the reports that we're generating for the powers that be are formatted to look like the reports that were generated for the old scantron graded test. (Insert shudder here.)
Responding to some of your questions and commenting on a couple of other issues we've seen:
(4) [Generating reports and incremental updates for the registrar] - I wrote a command line script to generate these reports, which also maintains a separate cached GDBM database of results. This means that it's very fast to look up cached results, and easy to find incremental updates (we look only for results that aren't already cached).
Related to this, we also had to address the issue of a student who ran overtime on the placement test, and therefore has a recorded score of zero (or no recorded score) on the test. Our work-around was to have the results generating script go back through the answer log to find results for students who don't have recorded scores in the database. This is very slow for large answer logs, however, and argues strongly for truncating the answer log periodically.
My script is a little idiosyncratic (see comment about results formatting above), but if it would be useful to you I'm happy to forward it with some notes suggesting which parts might be useful to you.
(5) [Retiring students from the system] - We do this every spring (our placement testing is primarily in the summer).
Re: PSVN - theoretically it's a unique identifier for each student's problem set. Some problems take advantage of this to get the same random seed for successive problems in the set. In the database definition it's an auto_incremented integer data type, so there shouldn't be a need to set it to any value (if you just insert NULL or leave it out the database will give the correct value for it).
Gavin
This sounds very impressive. We use WeBWorK for placement testing, but ours is mostly for our College in the University and only tests enrolled students. Thus we see "only" between 6000 and 7000 students a year for the placement testing.
We did have the complication that we were putting in the WeBWorK test to replace an existing pencil and paper test, for which there was already a scoring and reporting infrastructure. As a result, the reports that we're generating for the powers that be are formatted to look like the reports that were generated for the old scantron graded test. (Insert shudder here.)
Responding to some of your questions and commenting on a couple of other issues we've seen:
(4) [Generating reports and incremental updates for the registrar] - I wrote a command line script to generate these reports, which also maintains a separate cached GDBM database of results. This means that it's very fast to look up cached results, and easy to find incremental updates (we look only for results that aren't already cached).
Related to this, we also had to address the issue of a student who ran overtime on the placement test, and therefore has a recorded score of zero (or no recorded score) on the test. Our work-around was to have the results generating script go back through the answer log to find results for students who don't have recorded scores in the database. This is very slow for large answer logs, however, and argues strongly for truncating the answer log periodically.
My script is a little idiosyncratic (see comment about results formatting above), but if it would be useful to you I'm happy to forward it with some notes suggesting which parts might be useful to you.
(5) [Retiring students from the system] - We do this every spring (our placement testing is primarily in the summer).
Re: PSVN - theoretically it's a unique identifier for each student's problem set. Some problems take advantage of this to get the same random seed for successive problems in the set. In the database definition it's an auto_incremented integer data type, so there shouldn't be a need to set it to any value (if you just insert NULL or leave it out the database will give the correct value for it).
Gavin
Hal,
At WPI we've been using WeBWorK for freshman calculus placement for
several years. The number of students is much smaller - only about
900 per year, but we have implemented some of the features you
are interested in. Below I've responded to your five technical issues.
1, 2. Enrollment of students and assignment.
When a freshman officially enrolls, they get a computer account username
and password. This information is used to automatically enroll the
student in the placement test course and make assignments. This
is done by directly writing information to the WW mysql database. The
programming is in C, and was not done by me but by one of our outstanding computer support people.
3. Reporting results to the student.
The last problem in the placement test has two parts and uses
Davide Cervone's compoundProblem package. Answering the first
part of the problem sets a value in the WW database that says
the student is done. The student can then go on to the second part
of the problem, which contains a link to an external web site. If the
student clicks on the link, external programming is triggered that
collects information from the WW database, generates a recommendation,
and saves information to an external (to WW) database. It also sets
a field in the WW database that indicates the student has been processed.
The recommendation appears on the screen and is also e-mailed to the student.
If a student chooses not to view the recommendation immediately, there
is a chron job that runs early every morning to process students who
are done with the exam but chose not to see the recommendation
immediately. This generates the recommendation e-mail and writes
the information to the external database.
4. Reports to registrar.
Students, faculty, and academic advising staff can use the web site to
view the recommendation for a particular student or generate summary
reports. More complicated reports can be generated by accessing the
mysql database where the placement info is stored.
5. Retiring students
I have a model placement test course that I use to generate the
placement test course each year. After the end of a placement
cycle, the current course is deactivated, meaning that no more
students are added to that course. Before the next cycle begins, I
create a new placement course.
In the external data base, there is a different table for each cycle.
I'm not sure how much of our experience can be translated to your
very different situation, but it does indicate that at least part of
what you are interested in is possible.
Regards,
Bill Farr
Worcester Polytechnic Institute
At WPI we've been using WeBWorK for freshman calculus placement for
several years. The number of students is much smaller - only about
900 per year, but we have implemented some of the features you
are interested in. Below I've responded to your five technical issues.
1, 2. Enrollment of students and assignment.
When a freshman officially enrolls, they get a computer account username
and password. This information is used to automatically enroll the
student in the placement test course and make assignments. This
is done by directly writing information to the WW mysql database. The
programming is in C, and was not done by me but by one of our outstanding computer support people.
3. Reporting results to the student.
The last problem in the placement test has two parts and uses
Davide Cervone's compoundProblem package. Answering the first
part of the problem sets a value in the WW database that says
the student is done. The student can then go on to the second part
of the problem, which contains a link to an external web site. If the
student clicks on the link, external programming is triggered that
collects information from the WW database, generates a recommendation,
and saves information to an external (to WW) database. It also sets
a field in the WW database that indicates the student has been processed.
The recommendation appears on the screen and is also e-mailed to the student.
If a student chooses not to view the recommendation immediately, there
is a chron job that runs early every morning to process students who
are done with the exam but chose not to see the recommendation
immediately. This generates the recommendation e-mail and writes
the information to the external database.
4. Reports to registrar.
Students, faculty, and academic advising staff can use the web site to
view the recommendation for a particular student or generate summary
reports. More complicated reports can be generated by accessing the
mysql database where the placement info is stored.
5. Retiring students
I have a model placement test course that I use to generate the
placement test course each year. After the end of a placement
cycle, the current course is deactivated, meaning that no more
students are added to that course. Before the next cycle begins, I
create a new placement course.
In the external data base, there is a different table for each cycle.
I'm not sure how much of our experience can be translated to your
very different situation, but it does indicate that at least part of
what you are interested in is possible.
Regards,
Bill Farr
Worcester Polytechnic Institute