WeBWorK Main Forum

Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
Number of replies: 23
I have a server with 1 Gb of RAM.

Even with just 4 students grading their exams at the same time, previewing
exam answers can take FIVE MINUTES!!!

Grading an exam can take 5 - 10 MINUTES!!!

What can cause such slowness?

All I can think of is that I import lot of macros in each problem because
I don't want to bother pruning each question to keep ONLY what I need.
Is that bad?  

Is that the problem or could something else be causing the slowness?

cs
In reply to Christian Seberino

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Gavin LaRose -
Hi Christian,

I think there must be something else going on with your server. The macro loads nominally slow things down, but shouldn't cause the problems you report unless there's some looping issue with the macros or problems that you're using. We have a testing lab with about 30 machines, which in peak times are hitting the server regularly, and we don't see wait times that are even seconds long (we do have a much bigger server, of course, but I think the observation is still valid).

Can you do some investigation on the server to see if there is some additional evidence of what's causing the slow down? e.g., top, uptime, and vmstat might give some information about where things are bogging down.

Gavin

In reply to Gavin LaRose

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
I don't see anything odd with top or uptime.  I'm on a Linode.com Xen virtual machine.  

Here is vmstat output.........

% vmstat
procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa
 0  0  17764 476380  30080 417984    3    3     7     3    7    3  0  0 100  0

Does that look ok?

-----------------------

Are there certain combinations of macros that do indeed cause looping issues?

Since Webwork doesn't crash, I assume these looping issues are somehow
handled by Webwork eventually?


In reply to Christian Seberino

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Gavin LaRose -
Hi Christian,

I'm assuming that this is from a time when your server wasn't returning results from a gateway test. I don't see anything odd there, though I'm a little surprised that there are swapping events (for comparison, my server shows the following, albeit in an unloaded state:

procs -----------memory---------- ---swap-- -----io---- --system-- -----cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 0  0 313632 5243360 766216 16049680    0    0     0     2    0    0  0  0 100  0  0

but I don't know if that's different enough that I would worry about it.)

I don't know of any combinations of the standard macros that would cause looping issues. The only cases where I've seen a problem is when I've gotten a problem with an infinite loop, and that generates timeouts from the server.

If you load the problems that you have in the test into a homework set and have multiple sessions submit them, do you get the same behavior? Can you shorten the gateway/quiz assignment to only one problem and elicit the same response?

Gavin

In reply to Gavin LaRose

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
Actually the printout was from a random time NOT when having Webwork trouble.  
In reply to Gavin LaRose

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
I'll see if 4 students simultaneously with homework and/or an exam with 1 problem has the same issue this week and report back.  I also make all the suggestions on this thread so far so I'll also report back if that fixed the problem somehow.
In reply to Gavin LaRose

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
Server still slow...I tested it today.  Hard to tell what the problem is.

I did a "normal" quiz with about 20 questions but not 1 question yet.

My server is 1 year old.....I will build a new Webwork server with the latest
source and test the fresh server and give you my report soon.

cs
In reply to Christian Seberino

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Alex Jordan -
It may be worth noting that for certain questions, notably those that actually make use the statistics macro, they can take a long time to process. The algorithm that computes areas of the bell curve (and the inverse process) is pretty accurate, but computation heavy.

Just a thought - probably not the issue.
In reply to Christian Seberino

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Jason Aubrey -
Christian,

What are the specs on your vm? Also, about how many students do you having hitting it when a gateway quiz comes due?

The grading/rendering of a gateway quiz is more resource intensive than grading/rendering a single problem, and it's probably less efficient than it could be.

We had gateways due at midnight for about 1/2 of a semester and probably at least 1000 kids working on them up to the deadline. Even our very big server was having trouble with it. (Grading/loading was taking a long time, probably not 5-10 minutes, but more like up to 5 minutes...)

Changing the due dates to 8am fixed the problem. I'd also check the apache error log and use the webwork2/bin/check_timing_log.pl script to get general timing stats on how long problems are taking to render.

Hope this helps,
Jason
In reply to Jason Aubrey

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
I have a Linode.com Xen virtual machine with 1 Gb of RAM.  It is also running a Django/Apache web server (http://phil4.com) serving some static content in addition to Webwork but that is it.

On quiz day I limit simultaneous exams to 4 students.  That is why it is soo odd since people here said I can expect about 20 students per Gb of RAM.

Here is one of the worst from timing.log....I guess it was closer to 4 min...

[Mon Mar 18 19:56:15 2013] 18660 1363654570 - [/webwork2/physics_1402/quiz_mode/exam_2,v1/] [runTime = 258.202 sec sql_si
ngle]

In reply to Christian Seberino

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Michael Gage -
Are you limiting the number of simultaneous child clients allowed by the server? I've seen thrashing when there are too many clients alive simultaneously even if many of them are not actually serving requests.  The WW footprint is pretty large and it is a good idea to keep the number of clients small enough so their footprints can all fit into available memory simultaneously.
In reply to Michael Gage

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Gavin LaRose -
Hi all,

In the far distant past I also had trouble with MySQL timing out because it didn't have enough clients available to service requests. This was corrected by adding a line
  max_connections=5000
to the /etc/my.cnf file. That said, 5000 is certainly overkill, and we had the trouble many years ago (but have been setting max_connections ever since).

Gavin

In reply to Gavin LaRose

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Arnold Pizer -
Hi,

Another thing to check is what engine MySQL is using? In the instructions http://webwork.maa.org/wiki/Installation_Manual_for_2.5_on_Ubuntu_12.04#MySQL_Security_and_Performance_Issuses
you will see:

 The default engine is InnoDB as of MySQL 5.5.5 (MyISAM before 5.5.5) but (at least on some hardware) InnoDB seems to be 50-100 times slower than MyISAM. So we will change the default MySQL engine from InnoDB to MyISAM. Note  that this change only applies to new tables, tables already constructed will continue to use InnoDB.  But we haven't haven't created any WeBWorK tables so we don't have to change the engine for any existing tables.  ...

You already have a tables set up, so you should check what engine they are using and if they are using InnoDB, I would suggest switching to MyISAM. Googling "mysql change engine" should bring up documentation on how to do this.

Arnie
In reply to Arnold Pizer

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
I seem to have MyISAM.  That just seems to be
what my Ubuntu server defaulted to without my doing anything special.
In reply to Gavin LaRose

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
Did you have HUGE numbers of students to need that MySQL tweak?  Would that be needed if I have speed issues with just 4 students?
In reply to Michael Gage

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
Are these the Apache settings you are referring to for that?  Should I make these values smaller?

MaxClients           20
MaxRequestsPerChild  100

In reply to Christian Seberino

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Michael Gage -
Make MaxClients 4 or 5 (for 1 Gig of memory), restart the server (make sure all of the child processes disappear before restarting) and see if that improves things.  Students might have to wait 1 second before obtaining a process but that would be better than having lots of thrashing as underutilized processes are swapped in and out of memory.

-- Mike

In reply to Christian Seberino

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Danny Glin -
A couple more things to check:

A couple people have already mentioned apache settings (in httpd.conf).  There are a couple of things that are critical:

MaxRequestsPerChild - If this is set higher than about 100, you'll eventually see apache processes eating up all of your available memory.

MaxClients - This shouldn't be more than your available memory can support.  In my experience, each apache process can grow to about 100MB of memory, so with 1GB of memory I'd be hesitant to push this past 10 (in fact I personally would probably set it around 5).

Here is my first thought at appropriate settings for 1GB of RAM (excerpted from httpd.conf):
<IfModule prefork.c>
StartServers 5
MinSpareServers 5
MaxSpareServers 5
ServerLimit 5
MaxClients 5
MaxRequestsPerChild 100
</IfModule>

The only other thought I had is: which display mode are you using? If you are using Images, then the system is running a bunch of things through LaTeX each time a quiz is generated/previewed/submitted.  This can be very taxing on the server.  I would recommend changing to something more client-side (MathJax or jsMath depending on which version of WW you are running).
In reply to Danny Glin

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Lars Jensen -
Hi Christian,

We had some slowdown on our server and used the recommendation of Alex Basyrov here to configure the apache SizeLimit module. Try add this to the end of the <Perl>...</Perl> section of webwork.apache2-config:

use Apache2::SizeLimit;

$Apache2::SizeLimit::MAX_PROCESS_SIZE = 120000;
$Apache2::SizeLimit::MAX_UNSHARED_SIZE = 120000;
$Apache2::SizeLimit::CHECK_EVERY_N_REQUESTS = 20;

We have not had any issues since.

Lars.
In reply to Danny Glin

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
I looked at my global.conf and it seems Webwork
has 3 display modes by default..

images, jsMath and MathJax.

Do you comment out "images" to FORCE everyone to
use the other 2?  
(Since the docs say MathJax is strongly recommended, I'm going
to comment out everything but that and see what happens.)

cs

In reply to Christian Seberino

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Arnold Pizer -
Hi Christian,

I don't think you have to comment out "images" to FORCE everyone to
use the other 2.  Making MathJax the default will do it.  99.9% (which in your case is 100%) will use the default display mode whatever it is.  Years ago at Rochester we had images mode set as the default and also jsMath. I set up both to run under lighttpd and we had that set up for a full semester with maybe 1200 students using the system. No one ever reported a problem even though jsMath will NOT run under lighttpd (something I discovered the following semester).  So I assume virtually no student tried jsMath. 

Note that I believe that MathJax does run under lighttpd but I have not tried that. Probably running MathJax under lighttpd should be a standard set up for large installations.

Arnie   
In reply to Arnold Pizer

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Christian Seberino -
I see now that Webwork for me is defaulting to jsMath.  Could that be the problem?  I know "images" is bad but is MathJax soo much better that I should
default to that instead?

cs
In reply to Christian Seberino

Re: Why my exam grading so slow? (Does importing unnecessary macro files slow things down a lot?)

by Davide Cervone -
Yes. jsMath will use lots of small image files to make the equations when the user doesn't have its fonts installed, and that means lots of file accesses, each in a separate child process, and that could easily swamp your server, causing thrashing. MathJax will used web-based fonts instead which means fewer file accesses. They should also be cached by the browser, which will also help.