OPL Problem Statistics

This article is under construction. Use the information herein with caution until this message is removed.

In a future version of WeBWorK (probably 2.12) the Library Browser will optionally display local and global data about problems, specifically the number of individuals who have attempted the problem, the average number of attempts on the problem and the average status earned on the problem. Local data represents usage at your institution and global data represents the sum of all local data contributions.

The display

The display in the Library Browser looks like

where obviously the data displayed in the image above is not real.

Information for Instructors

Statistics are only displayed for OPL problems for which local and/or global data exists (you will not see all 0's). Data is not collected nor displayed for non OPL problems. The display of local and/or global data can be disabled for an individual course or for all courses. If you are not seeing any data please ask your WeBWorK administrator to enable the display of data.

Following is a description of the data displayed.

GLOBAL Usage

Global data on problem usage is contributed by many institutions using WeBWorK all over the world. The Usage figure is the total number of individuals who have attemped this problem at least once. A high figure represents a problem which has been assigned to many students and is both popular with instructors and likely bug free.

GLOBAL Attempts

The Attempts figure is the global average of the number of attempts (both correct and incorrect) individuals take on this problem. A high figure may represent a difficult problem. Note that problems with multiple parts may have higher average attempts since many students will submit an answer to each part before continuing and each such submittal counts as an attempt.

GLOBAL Status

The Status figure is the global average of the Status individuals have earned on this problem. The Status is the percentage correct (from 0% to 100%) recorded for the problem. A low figure may represent a difficult problem. The Status is often fairly high since many students will work on a problem until they get it correct or nearly so.

Reviewing a problem and looking at both the average Attempts and average Status should give instructors valuable information about the difficulty of the problem.

LOCAL Usage

Local data on problem usage is generated and maintained by your institution. The Usage figure is the total number of local individuals who have attemped this problem at least once. A high figure represents a problem which has been assigned to many students and is both popular with instructors and likely bug free. Local data is generated when your systems admin runs the script update-OPL-statistics.

LOCAL Attempts

The Attempts figure is the local average of the number of attempts (both correct and incorrect) individuals at your institution take on this problem. A high figure may represent a difficult problem. Note that problems with multiple parts may have higher average attempts since many students will submit an answer to each part before continuing and each such submittal counts as an attempt.

LOCAL Status

The Status figure is the local average of the Status individuals at your institution have earned on this problem. The Status is the percentage correct (from 0% to 100%) recorded for the problem. A low figure may represent a difficult problem. The Status is often fairly high since many students will work on a problem until they get it correct or nearly so.

Reviewing a problem and looking at both the average Attempts and average Status should give instructors valuable information about the difficulty of the problem.

Information for WeBWorK Administrators

The data that we are requesting you generate and contribute goes in a MySQL table OPL_local_statistics where entries look like:

+--------------------------------------------------+--------------------+------------------+----------------+ | source_file | students_attempted | average_attempts | average_status | +--------------------------------------------------+--------------------+------------------+----------------+ | Library/LoyolaChicago/Precalc/Chap4Sec2/Q02.pg | 2 | 3 | .5 | +--------------------------------------------------+--------------------+------------------+----------------+

so, as you can see, there is no student identifying data being requested.

In order to get this project (which is a small and independent part of the WeBWorK "Big Data" project) off the ground, we need global data. Hence this personal request for data from your institution.

If you are interested in contributing (and we hope you are), the process is very easy. First download two scripts with the following commands:

wget --no-check-cert https://raw.githubusercontent.com/goehle/webwork2/dbscript/bin/update-OPL-statistics wget --no-check-cert https://raw.githubusercontent.com/goehle/webwork2/dbscript/bin/upload-OPL-statistics

You can put them in /opt/webwork/webwork2/bin/ if you want. They should run on any recent version of WeBWorK but have only been tested on 2.7, 2.9 and 2.10. Run perl update-OPL-statistics which generates the OPL_local_statistics table. In the future you should probably run update-OPL-statistics at the end of every semester. It harvests data from all closed (due date has passed) homework sets and is written so that old data (e.g. from courses deleted since the last time the script was run or from courses that are reused by deleting old students and adding new ones or just old courses that remain on the server) is saved and new data is properly handled. It does not retrieve data from archived courses.

Then maybe you should contact your IRB office and tell them that you are planning on contributing data and show them the OPL_local_statistics table (or the sample above) so they can see that no student identifying data is involved. After receiving their blessing, to upload the data, run the command perl upload-OPL-statistics

That's it. Your local data will get sent to a server that Geoff Goehle (who has written most of the code for this project) is maintaining. The resulting OPL_global_statistics table will be distributed as part of the OPL.

If you are interested in seeing how this actually works in the Library Browser, do the following on your development system.

git checkout origin/develop git pull http://github.com/goehle/webwork2.git dbscript Update localOverrides.conf from localOverrides.conf.dist and run apachectl restart

Then run update-OPL-statistics if you haven't run that yet. Also it's a good idea to first update the OPL if you haven't done that in awhile. Either turn off the global data option (in localOverrides.conf) or create a fake OPL_global_statistics table, e.g a copy of OPL_local_statistics (maybe very soon with your help we will have a real OPL_global_statistics table available). Note that if no data is available for a problem you are looking at in the Library Browser, you will not see labels with 0's displayed.

Thanks for any help you can give us with this and feel free to write with any questions, comments or suggestions.

Sincerely,

Arnie