Gateway Tests and Quizzes

From version 2.2 of WeBWorK "GatewayQuiz" assignments have been supported. These assignments are distinguished by the following characteristics:

• They have a fixed time limit;
• They are versioned, that is, the assignment may be taken by a student some number of times (the maximum number is determined by the instructor, and could of course be one), and a different version of the assignment is generated for the student each time;
• They may require proctor authorization to be taken and graded;
• They present more than one question from the assignment on a single page; and
• They allow problems to be described as a group of problems from which one (or more) problem(s) is(are) drawn.

As of version 2.4 of WeBWorK there are additional features for GatewayQuiz assignments:

• They allow restriction of the grade and related data that is shown to students when they have completed an assignment;
• They allow GatewayQuiz assignments to be available only from computers with specific IP addresses (e.g., a computer lab). This is a feature that is available to all assignments, but may be of particular use for GatewayQuiz assignments; and
• They allow proctored GatewayQuiz assignments to have a set-level proctor password defined that allows the assignment to be started, but not graded.

The effect of these features and their use are described below.

Setting up a Gateway Test/Quiz

Any assignment may be made into a Gateway Test/Quiz assignment by selecting the set on the Instructor Tools page and clicking "Edit Set." Among the "General Information" items is the "assignment type" of the assignment, which may be set to homework, the default, gateway/quiz, or proctored gateway/quiz. If either of the latter two types is selected and the set is saved, it is then possible to edit other parameters characteristic of Gateway Test/Quiz assignments.

These other characteristics include

• Set Password: (for proctored assignments only, and for WeBWorK 2.4 and later only) a password for the set that is the required password to start the assignment. This password will not authorize grading, for which a regular proctor login is required. See the proctored tests and proctors and users sections below for additional information.
• Test Time Limit: the length of time the student is allowed to work on each version of the assignment. Note that the expiry of the time limit has implications for the recording of student answers, as noted below in the Due Dates section of the documentation.
• Cap Test Time at Set Due Date: a flag (available in WeBWorK 2.4 and later) that allows the test time to be restricted, as described below in the discussion of the GatewayQuiz due dates.
• Number of Graded Submissions Per Test: the number of times a student may submit each version of the assignment for a grade.
• Time Interval for New Versions: students may be restricted to a fixed number of assignment versions in a given time interval, for example, two per day. To implement this, we specify the time interval and the number of versions per interval. Note that the time interval is a rolling interval, so that specifying one version per hour means that a student who starts a test/quiz at 11:58 may start a second at 12:58:01. To implement a daily restriction (e.g., two tests per day) it is generally sufficient to specify the number of tests in a 12 or 18 hour interval. In WeBWorK 2.4, entering "0" for the time interval for new versions gives an infinite interval.
• Number of New Versions per Time Interval: this is explained above; entering "0" puts no limit on the number of allowed versions. If the time interval for new versions is infinite, this gives the total number of test/quizzes that may be taken.
• Order Problems Randomly: if this is true, the problems on the test/quiz will be presented in a random order. It is worth noting that this has implications for the manner in which student results are displayed.
• Number of Problems per Page: how many of the questions in the test/quiz to show on one page. For example, specifying "3" for a 5 question quiz will result in a first page of 3 questions and a second of 2. Specifying "0" problems per page puts all problems on one page.
• Show score on finished assignments: (available in WeBWorK 2.4 and later) whether students should be able to see their score on versions of the test/quiz on which they have no remaining attempts, or no remaining time. This is described below in the Showing Student Scores and Work section of the documentation.
• Show student work on finished assignments: (available in WeBWorK 2.4 and later) similarly, whether students should see the problems and their work after finishing a test version. This is also described below in the Showing Student Scores and Work section of the documentation.

There are also several settings in the global.conf file that determine the behavior of GatewayQuiz assignments. These include settings in the permissionLevels hash, view_proctored_tests, view_hidden_work and view_ip_restricted_sets, and the gatewayGracePeriod variable.

Using a Gateway/Quiz

Once created, gateway/quiz assignments may be used largely as regular assignments are. Because of their nature as versioned (and possibly proctored) sets, however, there are a number of features that are unique to or characteristic of this assignment type. These include:

• WeBWorK URLs of Gateway/Quiz Assignments
• Proctored Gateway Test/Quiz Assignments
  • Proctors and WeBWorK Users
• Due and Answer Dates for Gateway/Quiz Assignments
• Set Listing Display of Gateway/Quiz Assignments
• Restarting Timed Tests and Other Timing Details
• Scoring Gateway/Quiz Assignments
• Showing Student Scores and Work
• Set Definition Files for Gateway/Quiz Assignments

In addition, it may be useful to create a gateway/quiz assignment by selecting problems from problem groups, and to restrict access to such assignments by IP address (e.g., lab location).

WeBWorK URLs of Gateway/Quiz Assignments

A WeBWorK problem set may be obtained directly by going to the URL http://webworkhost/webwork2/courseName/setName/. To go to a gateway/quiz assignment, it is necessary to specify that the assignment is a gateway/quiz in the URL: the address for a gateway/quiz assignment is thus http://webworkhost/webwork2/courseName/quiz_mode/setName/ for unproctored gateway/quiz assignments, and http://webworkhost/webwork2/courseName/proctored_quiz_mode/setName/ for proctored assignments.

This allows one to set up an easy access page on the computers in a proctored lab or a link on a course webpage that sends students directly to the proctored assignment that they are working on.

Up to: table of contents || using gateways

Proctored Gateway/Quiz Assignments

Proctored assignments require proctor authorization at two points in the test/quiz process: before the set may be started, and before it may be graded. A proctor is any member of the class with a permission level of proctor or better (permission levels are defined in the global.conf file). Note that in WeBWorK 2.4 and later, the proctor permission level is divided into two levels, login_proctor and grade_proctor. This is considered in the proctor users section, below.

Depending on how a WeBWorK system is set up, it may be desirable to hide proctored gateway/quiz assignments from students' view on the ProblemSets listing page, because they will not be able to actually start the assignment without a proctor being present. This is controlled by the view_proctored_tests setting in the permissionLevels hash in the WeBWorK global.conf file. If you find that proctored assignments are disturbingly invisible for your students, you probably want to reset this in your WeBWorK configuration.

Up to: table of contents || using gateways

Proctors and WeBWorK Users

In a WeBWorK course there are users with different permission levels, student, proctor (or login_proctor and grade_proctor, in WeBWorK 2.4 and later), ta and professor. These levels are in an ascending tree: thus a user whose permission level is ta has all the abilities of a proctor or student, but not all of those of a professor. In many cases where proctored assignments are used, it will be useful to create a set of users who are proctors, giving them proctor (or grade_proctor) permissions, but in any event all ta and professor users in the course will be able to proctor assignments as well.

The one exception to this rule is when a set-level login proctor password is given for the assignment. When this password exists, the only user able to proctor the start of a test is the implicitly defined set-level proctor.

Another aspect of user management that pertains to proctors is a user's status in the course. All students and instructors in a WeBWorK course have a status of Enrolled; other valid statuses are Audit, Drop and, new in WeBWorK 2.4, Proctor. These status designations determine how students interact with the system: Audit students' scores are not included in score reports, Drop students cannot log in, etc. Proctor users are similar to Drop students, in that they are unable to log in and work assignments, though if their permission level is set to login_proctor or grade_proctor they will be able to authorize proctored tests.

Up to: table of contents || using gateways

Due and Answer Dates for Gateway/Quiz Assignments

For a homework assignment, students must finish the assignment before its due date, and answers and solutions to the homework set are available to students after the answer date for the assignment. For gateway/quiz assignments, where a student may be able take multiple versions of the assignment, we talk about the due (and answer) date(s) for the template set, and the due (and answer) date(s) for the set version that a student is taking. The due date for the template set is the due date that the instructor sets up when editing the set in the Instructor Tools section of WeBWorK. This determines the latest time that a student may start a version of the assignment.

The set version that the student starts (before the template set's due date, of course) then has a due date that is determined by the time limit on the test. Thus if the template set is due at 3/21/2007 at 11:00am and has a time limit of 30 minutes, a student can start a test at any time up to 11:00am on 3/21/2007. If the students starts a test at 10:05am, that set version will be due at 10:35am, because the time limit is 30 minutes. (In WeBWorK 2.4 and later,) A gateway/quiz assignment can be configured to cap the time limit for a test at the due date for the template set. If this is selected, a student starting a version of our example test at 10:35am will only be allowed 25 minutes to work on it. If it is not selected, the student starting at 10:35am will still be allowed to work on the test 30 minutes, and so will not have to finish until 11:05am (even though the due date for the template set is before that).

Note that regardless of the cap the time limit setting for the assignment, the student has a grace period after her/his test is due in which the system will still record her/his answers. In addition, the course answer log records information about the student's work, as described in the second paragraph below.

The answers available date for the template set determines the relative answer available date for versions of the set that are taken. Suppose that for our example set the answers available date is 3/21/2007 at 11:15am. Then for all set versions, answers will be available 15 minutes after they are completed. Thus the student who starts a test at 10:05am will find answers are available for that set version at 10:50am. In most cases the answers available date will be set to be the same as the assignment due date, and restriction of availability of answers will be accomplished by using the show work settings available in WeBWorK 2.4 and later.

One ramification of the test time limit is that a student who does not submit her/his test before it is due will not have any of her/his answers recorded. The student's score is recorded as a 0, and none of her/his answers are saved as the "last answers submitted." It is, however, possible to get a record of the student's work by using the Show Past Answers feature of assignments. A Professor (or TA, or, more generally, anyone with view_answers permission as defined in the global.conf file) when viewing the set will have a Show Past Answers button available, which will show the student's progress on the quiz problems, including all preview, new page, and submission information. This includes a submission that is sent after time has expired.

Up to: table of contents || using gateways

Set Listing Display of Gateway/Quiz Assignments

By default, all Gateway/Quiz assignments that are assigned to a student appear at the bottom of the ProblemSets list s/he sees when logging in to WeBWorK. Available gateway/quiz assignments will be indicated with a link called "Take SetName test" (if they are not proctored, or if the view_proctored_tests setting is configured appropriately, as indicated above). This link will start a new version of the gateway/quiz for the student, if s/he has available versions (that is, if s/he hasn't completed all of her/his versions for the given time interval; otherwise, an error will be returned). If the student has a version of the assignment that s/he has not completed (that is, which s/he can still submit for a grade, because s/he hasn't submitted it as many times as s/he is allowed and still has time left on the assignment), clicking the "Take SetName test" link will cause the student to resume work on that version of the test.

Once a student has completed a version of a gateway/quiz assignment, the version will show up on the ProblemSets page as a link "SetName (test N)", where N indicates the Nth test that s/he has taken. Note that once a student has completed a proctored test, that version of the test is redefined for the student as an unproctored test. Thus a student is able to review completed proctored gateway/quiz assignments without needing a proctor. Note that completed proctored assignments therefore appear on the ProblemSets page regardless of the value of view_proctored_tests in the permisionLevels hash in global.conf.

Up to: table of contents || using gateways

Restarting Timed Tests and Other Timing Details

As noted above, a test that has been started but which has not been submitted and which has time left may be reentered. It is worth observing, however, that the timer for the assignment does not stop if the student leaves the assignment, logs out of WeBWorK, etc.

The official deadline for a timed test is the due time that is stored on the server, not the time that is shown on the timer on the student's page. In general these two measures will be close to the same. When a student approaches the deadline for the test, various warnings will appear on the student's computer. If a test is submitted after its time limit has been exceeded, it will be checked but not graded: that is, the grade that is recorded for that version of the assignment in WeBWorK will be a zero (though the student's work may be logged), as noted above in the due dates section of the documentation. The time that the test is considered to have been submitted is the time when the "Grade Test" button has been pressed. Thus, if a student is taking a proctored test and clicks "Grade Test," s/he will not be penalized if the proctor is not immediately available to authorize the grading of the test.

To provide some flexibility in the time deadline, there is a gatewayGracePeriod variable in the WeBWorK global.conf file that specifies the time (in seconds) after the deadline for any given set version that the test will still be considered on time. Thus, if this is set to zero there is no grace period; if it is set to 30, students will be allowed to submit the test version up to 30 seconds after the server thinks they have run out of time. The default setting for the grace period is 120 seconds (two minutes).

Up to: table of contents || using gateways

Scoring Gateway/Quiz Assignments

Gateway/Quiz assignments are scored (on the Scoring page of the WeBWorK system) the same way that regular assignments are. However, only the best score over all versions is reported. Thus, if a student takes ten versions of a gateway/quiz assignment, only the best of those scores will be reported when scoring the assignment. (Note also that any test that a student does not submit before the test time limit expires is recorded with a score of zero.)

This is not the case when viewing "Student Progress," or when students view their grades, however. There, if the instructor views progress by set, all versions of the assignment will be visible (though there is the option on that page to restrict to only the best assignment). Similarly, viewing results by student will show all versions of gateway/quiz assignments that the student has taken, and a student who views her/his Grades will see all versions that s/he has taken.

In early versions of the Gateway/Quiz module, students who have completed no versions of a gateway/quiz assignment do not appear when student results are viewed by set. In version 2.4 and later, students who are not assigned the gateway/quiz do not appear, but all others do, with students who have taken no versions of the assignment being indicated in the results listing.

There is one more subtlety on the Grades and StudentProgress pages. When viewing students' scores by problem on gateway/quiz assignments for which the problems are ordered randomly, there are two orders in which the problem scores could be presented. The first is the instructor's order, that is, the order in which the problems are defined in the set, and the second is the test/quiz order, that is, the order in which the problems are shown on a particular test/quiz. On the student Grades page, and on the instructor's StudentProgress page when viewing results for a student, problem results are shown in the test/quiz order.

Up to: table of contents || using gateways

Showing Student Scores and Work

Starting in version 2.4 of WeBWorK, there are two settings for gateway/quiz assignments that determine the amount of information about a completed set that is shown to students. These are show scores on finished assignments and show work on finished assignments. Both of these may be set to "yes" (the default), "no," or "only after set answer date."

If they are set to "no," then students will not be able to view their score, or work, on a set version that they have completed. Thus setting show scores to "no" will mean that when students finish a set version they will get the message "your score was recorded," but they will not be able to see how they did on the test. Similarly, if show work is set to "no" when they finish the set version they will not be able to see the problems in their set version nor their work on the assignment. These restrictions extend to generation of hardcopies of an assignment and viewing of grades through the Grades link on the front WeBWorK page.

If they are set to "only after set answer date," then student scores or work will only be available to the student after the answers available date for the template set. (See due dates for an explanation of the usual relationship between the template set and set version.)

The effect of these settings is influenced by the view_hidden_work setting in the permissionLevels hash in the global.conf file. By default, anyone with a permission level of TA or higher (that is, with the usual permission settings, a permission level of 5 or higher) is able to view work and scores on gateway/quiz assignments even if the set is configured to prevent those things from being displayed.

Up to: table of contents || using gateways

Set Definition Files for Gateway/Quiz Assignments

Gateway/Quiz assignments may be imported from set definition files in the same manner as other assignments. To specify the gateway/quiz parameters, however, additional options must be indicated in the definition file. These are largely self explanatory, as is suggested in the following example file:

openDate = 01/24/2007 at 8:00 AM
dueDate = 02/23/2007 at 11:59 PM
answerDate = 02/23/2007 at 11:59 PM
paperHeaderFile = integralGWpaperHeader.pg
assignmentType = gateway
attemptsPerVersion = 1
timeInterval = 43200
versionsPerInterval = 0
versionTimeLimit = 1800
problemRandOrder = 1
problemsPerPage = 0
hideScore = N
hideWork = N
capTimeLimit = 0
restrictIP = RestrictTo
restrictLocation = ProctoredLabs
relaxRestrictIP = No
#
problemList=
(etc.)

All time limits and intervals are given in seconds. Note some restriction variables have specific valid values:

• assignmentType may be gateway or proctored_gateway (just omit assignmentType for non-gateway sets);
• probRandOrder and capTimeLimit may be 0= or =1 (for don't/do display problems in a random order or cap the time available on the test);
• hideScore and hideWork may be N=, =Y, or BeforeAnswerDate;
• restrictIP may be No, RestrictTo or DenyFrom;
• relaxRestrictIP may be No, AfterAnswerDate, or AfterVersionAnswerDate.

It may also be worth observing that the input data for the hideScore and hideWork parameters are the negation of the data that are selected on the set editor page.

When importing sets from a set definition file, it is not currently possible to set IP address restrictions on sets to more than one location. The value given by restrictLocation must be a valid location as defined in the WeBWorK environment into which the set is being imported. It is also not currently possible to create a set-level proctor password from a set definition file.

Up to: table of contents || using gateways

Selecting Problems from Problem Groups

The following is an overview of the process of selecting problems from groups; for more detailed information, see the ProblemGroupHowTo.

Of particular interest for gateway/quiz assignments is the ability to select problems from a group. The group of problems from which a problem on an assignment may be selected is just another problem set, which may be a set that is assigned to users or may be a set that is created just for the purposes of defining the group. To select a problem from this group, the "Source File" for the problem is specified as group:setName. Then when a user who is assigned the gateway/quiz starts a new set version, the version will include a randomly selected problem from the indicated set.

For example, suppose that the set chainRule includes the problems chain/prob1.pg, chain/prob2.pg, etc., through chain/prob9.pg, and that the set DerivativeGateway includes four problems, gateway/prob1.pg, gateway/prob2.pg, group:chainRule, and group:chainRule. Then when a user begins a new DerivativeGateway assignment, the set version that s/he gets will include the problems gateway/prob1.pg, gateway/prob2.pg, and two (different) problems randomly selected from the nine that make up the chainRule problem set.

Up to: table of contents || using gateways

Restricting Access to Sets by IP Address

Sets (both homework and gateway/quiz assignments) may be configured to restrict access to students working on computers with specific IP addresses, or to deny access to specific addresses. This is accomplished in two steps. First, a location (or locations) is (are) defined through the Course Administration course: for example, the locations defined could be "EH-B069" for a computer lab in East Hall, room B069, or "Proctored Gateway Lab", for the lab in which proctored tests are administered. Second, the set is changed by editing it to restrict access by location.

To define locations, the WeBWorK administrator selects "Manage Locations" in the Admin Course. This allows her or him to add, edit, and delete locations. Each location can have any number of IP addresses associated with it, either specified individually (e.g., as 192.168.15.101), as a range of values (e.g., 192.168.15.101-192.168.15.125), or by an IP mask (e.g., 192.168.15.0/24).

Once locations are defined, a WeBWorK professor can edit her/his sets to restrict access by location. To do this s/he selects "Restrict To" or "Deny From" from the "Restrict Access by IP" option when editing the set. After saving this change, a list of the available locations appears, from which s/he can select the desired restrictions (and then, of course, save the set again to save the changes).

Note that if no locations are defined, the "Restrict Access by IP" option is unavailable to professors. In addition, note that the view_ip_restricted_sets entry in the permissionLevels hash defined in the global.conf file determines whether a user can view a set from a computer which would otherwise be denied access to it. The default setting allows TAs and professors to ignore these restrictions.

Finally, it is possible to relax the IP address restrictions by setting a value for "Relax IP restrictions when?". Possible values are "Never," "After set answer date," (that is, after answers come available for the set, or, for gateway/quiz assignments, after they come available for the template set (c.f., due and answer dates for Gateway/Quiz assignments), or "After version answer date." "After version answer date" makes answers available after the answer date for a specific set version (as described in due and answer dates) of a gateway/quiz, which may in general be before the answer date of the template set. "After version answer date" and "After set answer date" are regarded as the same thing for non-gateway/quiz assignments.

Up to: table of contents || using gateways

-- Main.GavinLaRose - 06 Apr 2007