WeBWorK Main Forum

problems with gateway exams

problems with gateway exams

by Sheldon Newhouse -
Number of replies: 7
We are having some difficulties in administering proctored gateway exams in WebWork.
I would appreciate any suggestions.

1. We tried having 120 or so students log in from different locations withing a period of 5 minutes and the system ground to a halt and many could not log in.
This is running on a virtual server (via VMware) with a large set of resources(I think 6 processors and 16 GB RAM). The logs on the server showed no big overload in memory or cpu usage.
This seems strange since we have no problems with 400-500 students logged doing HW. Of course, they don't all log in at the same time.
Any ideas about this one?
One fix is to have the exams given with staggered logins, but, if possible, we would like to have the students take the quiz at the same time.

2. One idea I had was to print hardcopies of the quizzes for the students and have them work these on worksheets. Since since each student only needs to log in when he finishes the quiz, we could easily arrange for staggered logins during a fixed time period.
However, as a "Professor", I have not been able to print out individualized quizzes for some 250 students at the same time using the quiz format. I have no problem doing this with the HW format.

I get the following error message:

3 errors occured while generating hardcopy:

  • Can't generate hardcopy for set ''DGW' for user 'new1' -- set is not assigned to that user.
  • Can't generate hardcopy for set ''DGW' for user 'new2' -- set is not assigned to that user.
  • Can't generate hardcopy for set ''DGW' for user 'new3' -- set is not assigned to that user.

A hardcopy file was generated, but it may not be complete or correct. Please check that no problems are missing and that they are all legible. If not, please inform your instructor.
Download Hardcopy

When I look at the hardcopy, all I get is my own version of the quiz.

Of course, I don't want to "act as each student" to print the exam out.

Any ideas?

In reply to Sheldon Newhouse

Re: problems with gateway exams

by Gavin LaRose -
A note about the second of these issues:

Printing hardcopies of gateway/quiz assignments is tricky. This is a function of their nature: to allow students to take multiple versions of the assignment, each version is itself a newly assigned copy of the assignment. As a result, until the student starts the gateway/quiz, s/he has no actual gateway/quizzes on which s/he is working. This is different from a homework assignment, where once the set is assigned it exists for the student to work. For gateway/quizzes, assigning the set creates a template for the student from which the gateway/quiz assignments that s/he takes are modeled. As a result, there's probably no good way with the existing system to generate a copy of the gateway/quiz for each assignment to hand out.

It's interesting that this hasn't come up before. The solution is probably to recode part of the GatewayQuiz and ProblemSetDetail modules so that they allow "pre-creation" of a version of the set for all students, with a specified open and due date. This doesn't exist yet (obviously), but wouldn't be too difficult to implement.

In reply to Gavin LaRose

Re: problems with gateway exams

by Geoff Goehle -
I've been using the workaround of generating an exam using my account and printing that out for each student.  Everyone gets the same copy, but its easier to grade... 

In a related problem, the printing format is kind of off for gateway exams.  It prints all of the problems with a strange numbering style and it doesn't leave much room to input answers by hand (unless they are pretty short).  
In reply to Gavin LaRose

Re: problems with gateway exams

by Sheldon Newhouse -
In this regard, it seems that there are two types of quizzes/exams.

The first type is the gateway type in which one wants the student to log in as often as necessary during a specified time interval and keep taking new versions of the quiz until some threshhold of passing is achieved.

The second type is like a standard exam in which there is a definite time limit and the student only gets to login during that time interval and takes the same exam(in my case, the student only gets to log in once). This is like a replacement for the usual written exam given in a class.

Let me consider our implementation of the second type and describe some quirks and work arounds.

1, The student logs in and starts the exam.
A proctor login is then required and the student gives that login and password. The student then can see the exam.

We hand out blank worksheets on which the students can do the work.

2. The student finishes the entire exam on the worksheet before submitting it for grading. Since, we can't print the individual exams for the students (many students, and different seeds for the problems), we instruct the students to copy each problem onto the worksheet before doing it. This also allows us to give credit for work if there are errors in some of the WW problems.

3. The student hits the 'Grade' button and enters the grading proctor login.

4. The exam is graded, correct answers are shown and incorrect answers are shown.

5. We instruct the students to write down the problem numbers and parts which were wrong.
This is because, when they try to return to doing the exam and hit preview, the set of 'green stripes' or 'red stripes' disappears and the the students no longer can see which problems were originally right or wrong.
I would like to see this fixed. That is, KEEP at least the red stripes.
6. The students now try to fix the wrong problems, and repeat the process above up to four times.

This seems to work. It would be better if we could

a. print the individualized exams on the worksheets so the students don't have to copy them.

b. Have WW keep the red stripes through multiple submissions for grading.

Any suggestions on this matter are welcome.


In reply to Sheldon Newhouse

Re: problems with gateway exams

by Arnold Pizer -
Hi Sheldon,

In regard to your first problem, some people think that one bottleneck with WeBWorK might be MySQL, specifically the number of allowed connections.

Here's a snippet of advice that Gavin has given in the past:

This may or may not be in any way related, but...  We've had to explicitly increase the number of allowed connections to mysql to avoid running out under load here.  I've set max_connections=5000 in the my.cnf file.  The default for mysql5 seems to be 100, or 151 if one is running mysql5.1.  I would expect this to be indicated by "Too manyconnections" errors somewhere.

and another from Stephen DeSanto from MAA

I hear reports of slowdowns from time to time during particularly high-traffic times, but I
think MySQL is the bottleneck, and allowing more connections (and increasing PHP 
memory/threads/etc past the minimum as well) seems to help alleviate some of that.

Not sure if this is relevent, but it's worth a look.

In reply to Arnold Pizer

Re: problems with gateway exams

by Sheldon Newhouse -
We did increase the number of connections--to 5000--or so, but it did not completely solve the problem. I set up a mock exam to last 5 minutes and had some 120 students try to take it. Some got in in 2-3 minutes, some 4 minutes, some not at all. I wonder if the problem is related to the "double login effect" since the students have to also login as a proctor to start the exam.

We will try more staggered logins, with 20-30 at a time, and report back on the effect.

In reply to Sheldon Newhouse

Re: problems with gateway exams

by Danny Glin -
We've been running gateway quizzes for 30 (and sometimes 60) students starting simultaneously.  With our hardware it generally doesn't take more than a few seconds for the students to get in.  We have had some problems in the past, so I can offer a few additional thoughts:
  • How many questions are on a given quiz?  WeBWorK needs to do (at least) one database write for each question for each student starting the quiz, as well as reading a question file for each question.  Our quizzes tend to be 5 or 6 questions.  If yours is much longer it may help to display fewer questions per page (though I haven't personally experimented with this).

  • What are MaxClients and ServerLimit set to in your httpd.conf file?  The most requests that apache can handle simultaneously is the lesser of these values (though I don't know why they shouldn't always be equal).  If mysql is taking a few seconds to process a request, then all of the apache processes could be tied up waiting for mysql queries, which means that later http requests have to wait for an apache process to free up before they can start querying the database.  Here's a sample scenario where this might bog things down:

    Suppose MaxClients is set to 20, and an apache process takes 10 seconds to create the quiz.
    If 120 people all click on the quiz at the same time, the first 20 get an apache process right away.  The next 20 have to wait until the first 20 requests complete before they can even start, and so on.  Thus the last 20 people would have to wait something like a full minute before apache even accepts their connection.

    Do be careful how high you set MaxClients though, as each process takes up some amount of memory (my experience shows that with MaxRequestsPerChild set to 100, each apache process balloons up to about 100MB of RAM).  My rule of thumb is take the amount of system memory that you're willing to devote to apache, divide by 100MB and set MaxClients/ServerLimit to that value.  For 120 students, you may actually need significantly more than 120 apache processes to handle all requests simultaneously, since each quiz request needs to retrieve the actual quiz page, as well as any images, css files, etc. required to render it.  I would guess that with 120 students, MaxClients=300 would probably be enough to prevent apache processes from being the bottleneck.

  • You can get some info on how many connections mysql is using (and has used) by running
    mysql> show status like '%conn%';
    From my understanding, two of these values are of particular interest here:
    • Threads_connected, which shows the current number of connections
    • Max_used_connections, which shows the maximum number of simultaneous connections since the server was started.
Let me know if any of these things make a difference.

In reply to Danny Glin

Re: problems with gateway exams

by Arnold Pizer -

Speaking of Apache connections, I assume you are using lighttpd to serve static images, etc. This takes much less resources than Apache. See e.g. http://webwork.maa.org/wiki/Installation_Manual_for_2.4_on_Ubuntu_10.04#Implement_Optional_B_.28lighttpd.29