AIM-WeBWorK Problem Library Group report

Related Documentation

• Automated Problem Checking

---

Problem Library Group Goals:

• Come up with procedures for users to contribute problems and macros back to the community. Currently this can be done in a number of ways:
  • ask Rochester for a CVS repository; this is particularly relevant for problem files;
  • have the problems or macros in a file or files that one distributes oneself;
  • have the file or files in the National Problem Library (this requires that one know how to tell !WeBWorK where to look for any macro files that are there--which should be documented somewhere); or
  • have the problems or macros folded into the generic !WeBWorK distribution.
• Come up with documentation on how to get material into the National Problem Library.
• Come up with a way to track problem statistics for problems that are distributed and widely used.

The National Problem Library

• Currently, problems are added to the NPL by sending problems to the NPL maintainers. Could we add a link to have a problem in a course submitted to the NPL (e.g., similar to the link to report bugs in existing problems/the !WeBWorK code base)? This could be done as a bug report in bugzilla as a "feature request" with included code, and may part of the problem editor redesign.
• If things get checked into the NPL, there should be some sort of standardization so that the problems work (without having to do things like telling !WeBWorK where macro directories are). Currently, in global.conf there is a list of directories that are searched for macros.
• When the NPL is installed, a slew of problems is dumped into some directory or another, a command adds the meta data for those problems to the !WeBWorK database, and there's a tweak to global.conf to make sure that !WeBWorK knows where the files are.
• For submission to the NPL, there is a style guide somewhere, but it's out of date and not well publicized. Further, it turned out when the NPL was started that there was not a huge amount of consensus about what the form of the problems should be.
• If the indexing tags are in problems that are submitted to the NPL, inclusion is really easy. Problems that don't have tags are harder to deal with. It would be possible to have an 'unclassified' tag for some values, and then the problems would all be in the system and someone else could go through and tag them at some point.

Implementation Outline

Contributing Problems and Macros

We're most interested in the case of users who have some (modest) number of problems or macro files that they want to contribute to the code base.

The most logical way of doing this is to have some electronic drop boxes (that would be for problem and macro submissions, respectively) that would use the same login as the discussion board, which would drop into some contributed directory, with subdirectories by login name. If possible, it might be nice to have some sort of File Manager interface that would allow the user to manage their uploaded files independently. In the absence of that, the file or files would just be dropped into a directory and the user could submit a comment the desired file structure in which the file(s) should reside. Alternately, the user could submit a tar file that unpacks to give the user's desired directory structure.

Then the maintainer of the contributions would move them into the CVS directory tree. This could be done manually, or there could be a cron job that moves things across; there should be some check on the size of the file or its mime-type to be sure that someone doesn't start using this as a convenient mechanism to distribute their pr0n or warez. After being moved and committed to the CVS, they would be in a parallel contrib directory in the pg and Libraries modules in the CVS:

 pg
 contrib lib macros
 username1 username2

 Libraries
 contrib NPL
 OSU DCDS username1 username2 ASU Dartmouth et al.

Then, how do people have access to these problems? They get them automatically when the grab stuff from the CVS, and then have to reset the paths in their global.conf file to look at them (if they dare to do that).

Tracking Problem Statistics

Data what we want to track (for each problem in the problem library):

• the number of times that the problem has been used (number of students that have submitted an answer to the problem), and
• the level of difficulty of the problem, as measured by the success index or number of incorrect submissions (etc.).

We could get these data by logging statistics in a new database table when the set is scored. This could index problems by relative path from each course template directory (which has ramifications if 10 instructors all create problems called prob1.pg with the same relative path below their templates directory). The NPL problems already have unique paths that are defined in the Library and which start with "Library".

Then, the admin course could include a "view local stats" option, which would also allow viewing of the national stats for the NPL problems (perhaps only after the local statistics have been submitted?), and a submit local stats to the NPL option.

We want these statistics to be distributed with the library, which requires that there be some mechanism for the statistics to be downloaded when the library is unpacked on the local server (or obtained from the CVS). Then there could be an option to get latest statistics (for the problems that have been viewed, or for the entire library of problems) from the NPL, and adjacent to that a (larger) button to submit local stats to the NPL.