# OPL Problem Statistics

In WeBWorK (beginning with version 2.12) the Library Browser optionally displays local and global data about problems, specifically the number of individuals who have attempted the problem, the average number of attempts on the problem and the average status earned on the problem. The purpose is to provide instructors with useful information when selecting problems. Local data represents usage at your institution and global data represents the sum of all local data contributions. This page provides information on OPL Problem Statistics for Instructors and WeBWorK Administrators.

## The Display

The display in the Library Browser looks like

## Information for Instructors

Statistics are only displayed for OPL problems for which local and/or global data exists (you will not see all 0's). Data is not collected nor displayed for non OPL problems. The display of local and/or global data can be disabled for an individual course or for all courses. If you are not seeing any data please ask your WeBWorK administrator to enable the display of data.

Following is a description of the data displayed.

### GLOBAL Usage

Global data on problem usage is contributed by many institutions using WeBWorK all over the world. The Usage figure is the total number of individuals from contributing institutions who have attempted this problem at least once. A high figure represents a problem which has been assigned to many students and is both popular with instructors and likely bug free.

### GLOBAL Attempts

The Attempts figure is the global average of the number of attempts (both correct and incorrect) individuals take on this problem. A high figure may represent a difficult problem. Note that problems with multiple parts may have higher average attempts since many students will submit an answer to each part before continuing and each such submittal counts as an attempt.

### GLOBAL Status

The Status figure is the global average of the Status individuals have earned on this problem. The Status is the percentage correct (from 0% to 100%) recorded for the problem. A low figure may represent a difficult problem. The Status is often fairly high since many students will work on a problem until they get it correct or nearly so.

Reviewing a problem and looking at both the average Attempts and average Status should give instructors valuable information about the difficulty of the problem.

### LOCAL Usage

Local data on problem usage is generated and maintained by your institution. The Usage figure is the total number of local individuals who have attemtped this problem at least once. A high figure represents a problem which has been assigned to many students and is both popular with instructors at your institution and likely bug free. Local data is generated when your systems admin runs the standalone script update-OPL-statistics or, assuming the display of local data is enabled, the script OPL-update.

### LOCAL Attempts

The Attempts figure is the local average of the number of attempts (both correct and incorrect) individuals at your institution take on this problem. A high figure may represent a difficult problem. Note that problems with multiple parts may have higher average attempts since many students will submit an answer to each part before continuing and each such submittal counts as an attempt.

### LOCAL Status

The Status figure is the local average of the Status individuals at your institution have earned on this problem. The Status is the percentage correct (from 0% to 100%) recorded for the problem. A low figure may represent a difficult problem. The Status is often fairly high since many students will work on a problem until they get it correct or nearly so.

Reviewing a problem and looking at both the local average Attempts and local average Status should give instructors valuable information about the difficulty of the problem for students at your institution.

### Enabling and Disabling the display of OPL Problem statistics

The display of both global and local OPL Problem statistics in the Library Browser is enabled by default for all courses in the `defaults.config` file:

```\$problemLibrary{showLibraryGlobalStats} = 1;
\$problemLibrary{showLibraryLocalStats} = 1;
```

Either or both can be disabled by setting the above values to zero (0) in the `localOverrides.conf` file.

Either or both can be enabled or disabled for an individual course by setting the above values to one (1) or zero (0) in the course's `course.conf` file.

No OPL Problem statistics data will be displayed unless `update-OPL-statistics` is run. If you enable either of the above options and run `OPL-update` then `update-OPL-statistics` will automatically be run.

### Generating Global OPL Problem statistics

Global OPL Problem statistics data is contained in a file `OPL_global_statistics.sql` which is distributed with the OPL. Downloading the current version of the OPL with `git` will automatically retrieve the latest version of this file. Assuming `\$problemLibrary{showLibraryGlobalStats} = 1` is set for the server, then whenever the script `update-OPL-statistics` is run, the file `OPL_global_statistics.sql` will be processed and a MySQL table `OPL_global_statistics` will be created which contains the global data for display.

### Generating Local OPL Problem statistics

This data is generated in one of two ways. First, if `\$problemLibrary{showLibraryLocalStats} = 1` is set for the server, then whenever the script `OPL-update` is run to update the OPL, the OPL statistics will be generated by calling `update-OPL-statistics`. The second method is to run the script `update-OPL-statistics` directly. Either method creates a MySQL table `OPL_local_statistics` which contains the local data for display. All scripts are found in the standard directory `/opt/webwork/webwork2/bin/`. While the script `update-OPL-statistics` can be be run at any time and as often as one wants, it is recommended that script be run at the end of every term or semester after WeBWorK assignments due dates have passed. Note that no data will be collected for assignments whose due date has not passed at the time the script is run nor will any data be collected from archived courses. This is the reason we recommend running the script `update-OPL-statistics` at the end of semesters.

#### How Accurate Local Data is Maintained

Data is collected from all closed (due date has passed) homework sets and this is done in such a way that old data (e.g. from courses deleted since the last time the script was run or from courses that are reused by deleting old students and adding new ones or just old courses that remain on the server) is saved and new data is properly appended.

Specifically the script `update-OPL-statistics` first checks whether the MySQL table `OPL_problem_user` exists and creates it if it does not exist. If this table exists, new data will be appended to it. Maintaining this table is the key to maintaining accurate local OPL problem statistics data at your institution. If you bring a new server online, even if you choose not to bring over the whole `webwork` database, you should definitely transfer over the `OPL_problem_user` table.

The `OPL_problem_user` table contains rows with the following type of data

```+--------------+-----------+--------+------------+------------+--------------------------------------------------+--------+-----------+-------------+---------------+
| course_id    | user_id   | set_id | due_date   | problem_id | source_file                                      | status | attempted | num_correct | num_incorrect |
+--------------+-----------+--------+------------+------------+--------------------------------------------------+--------+-----------+-------------+---------------+
| myTestCourse | admin     | test   | 1447649940 |          1 | Library/Rochester/setDerivatives7Log/mec1.pg     |      1 |         1 |           1 |             1 |
+--------------+-----------+--------+------------+------------+--------------------------------------------------+--------+-----------+-------------+---------------+
```

This is a subset of the data contained in the CourseName_problem_user, CourseName_problem and CourseName_set tables. After the `OPL_problem_user` table is updated with new information, the `update-OPL-statistics` script reads through the `OPL_problem_user` table and collects information on usage and average status and average number of attempts and puts this information in a new table `OPL_local_statistics` (first deleting an old version if it exists). For example, if the `OPL_problem_user` table contains only the information above, then the `OPL_local_statistics` table would look like

```+--------------------------------------------------+--------------------+------------------+----------------+
| source_file                                      | students_attempted | average_attempts | average_status |
+--------------------------------------------------+--------------------+------------------+----------------+
| Library/Rochester/setDerivatives7Log/mec1.pg     |                  1 |                2 |              1 |
+--------------------------------------------------+--------------------+------------------+----------------+
```

This sample table shows only one student attempted the problem and they got it 100% correct after 2 attempts (not very realistic but it illustrates the point). The Library Browser uses the information in the `OPL_local_statistics` table to display the the local information illustrated in the screen shot above. Global information comes from the `OPL_global_statistics` table which has exactly the same structure as the `OPL_local_statistics` table.

### Contributing your Local Data to WeBWorK's Global Database

The global statistics displayed come from voluntary contributions of institutions all over the world using WeBWorK. Obviously the more institutions that contribute data, the more accurate the data will be and the more valuable the data will be to instructors when selecting problems. Contributing your Local Data to WeBWorK's Global Database is very easy and consists of running the script `upload-OPL-statistics` which basically sends a compressed version of your `OPL_local_statistics` table to WeBWorK's repository.

The `upload-OPL-statistics` script resides in the standard directory `/opt/webwork/webwork2/bin/`. First run `update-OPL-statistics` to make sure your local statistics are up to date. The `upload-OPL-statistics` script creates three files: server_name-xxxx-data.tar.gz, server_name-xxxx-desc.txt, and server_name-xxxx-opl.sql so you should run the command in a directory for which you have write permission, e.g. a temp directory. In that directory run the command

``` perl upload-OPL-statistics
```

Note if you do not have `/opt/webwork/webwork2/bin/` in your environmental search path, you will have to use the full path name of the script, `perl /opt/webwork/webwork2/bin/upload-OPL-statistics` The script will request some basic information

• if you have uploaded data from this server before
• approximately what years does this data span
• approximately how many classes are included

Probably the most important thing to mention as an additional comment is, if you have previously uploaded data, should this upload replace your previous upload (the usual case) or does this upload consist of all new data (e.g. if your old MySQL OPL_problem_user table was removed or not moved to a new server). See the above section How Accurate Local Data is Maintained for more information.

Finally you will be given the choice of

• Reenter above information
• Cancel

That's it. Assuming you choose to go ahead and upload data, your data will be sent to the global WeBWorK repository and will be combined with data from all other contributing institutions.

#### IRB (Institutional Review Board) considerations

The data that is being contributed consists of the data contained in your `OPL_local_statistics` table, namely gross usage and averages. Absolutely no individual data is being transferred, much less any individually identifiable data. In particular the individual student data in the `OPL_problem_user` table is never transferred to the WeBWorK repository. Under the guidelines published by the U.S. Department of Health & Human Services (see http://www.hhs.gov/ohrp/policy/checklists/decisioncharts.html#c1), this research is not research involving human subjects, and 45 CFR part 46 does not apply. Specifically, looking at chart 1, one sees "Is the information individually identifiable (i.e. the identity of the subject is or may readily be ascertained by the investigator or associated with the information)?" and if the answer is NO, the conclusion is "The research is not research involving human subjects, and 45 CFR part 46 does not apply". In our case, not only is the data not individually identifiable, there is no individual data at all. However, different IRB offices view things in their own and different ways. You should probably contact your IRB office and tell them that you are planning on contributing data and show them the OPL_local_statistics table (or the sample above) so they can see themselves that no student identifying data is involved.

## Contributing Data even if you are not yet using WeBWorK Version 2.12 or later

We hope many institutions will choose to contribute data even if they are not yet using WeBWorK Version 2.12. Currently (May 2015) we have data on approximately 23,000 of the 30,000 problems in the OPL. This is a good start, but we obviously want and need more data.

The `update-OPL-statistics` and `upload-OPL-statistics` scripts are contained in the standard WeBWorK 2.12 (and later) master branch distribution but not in earlier distributions. You can download them directly with the following commands:

```wget --no-check-cert https://raw.githubusercontent.com/openwebwork/webwork2/master/bin/update-OPL-statistics
```

Note that these scripts are also contained in the master branch of webwork2 (in the webwork2/bin/ directory) so you can get them from that location if you prefer.

### Running the scripts

You can put the scripts in a temp directory. Also the user running the scripts has to have the WEBWORK_ROOT environmental variable set (see e.g. Installation_Manual_for_2.10_on_Ubuntu_14.04#Configuring_the_Shell). The scripts should work on any recent version of WeBWorK but have only been tested on 2.7 and later versions. Note that it is a good idea to first update the OPL if you haven't done that in awhile (see e.g. Installation_Manual_for_2.10_on_Ubuntu_14.04#Updating_the_OPL). After updating the OPL if necessary, first run the command

```perl update-OPL-statistics
```

which generates the OPL_local_statistics table.

The `upload-OPL-statistics` script creates three files: server_name-xxxx-data.tar.gz, server_name-xxxx-desc.txt, and server_name-xxxx-opl.sql so you should run the command in a directory for which you have write permission, e.g. a temp directory. Now in that directory run the command

```perl upload-OPL-statistics
```

That's it. Your local data will get sent to a server that Geoff Goehle (who has written most of the code for this project) is maintaining.

-- Main.ArnoldPizer - 18 May 2016