WeBWorK Main Forum

RAM requirement for 2.17

RAM requirement for 2.17

by Andras Balogh -
Number of replies: 8

There was a discussion more than 5 years ago mentioning that 16GB of RAM should be fine for 2,000 students. Is that still the case? 

https://webwork.maa.org/moodle/mod/forum/discuss.php?d=6388

We have almost daily crashes with 24 GB of RAM and about 3,000 students on WeBWorK version 2.17. (Many high schools are using our WeBWorK server.)

Last semester with version 2.16, we had maybe 500 fewer students, with crashes only for things that I was able to identify and correct as mistakes.

Right now, what I see is that 101 apache processes wanted to take up 24.5 GB of RAM. Nine of those apache processes use between 400 and 570 MB each.

Unfortunately, oom-killer always chooses mysqld to kill, probably because that uses the largest amount of memory, which is 2.3GB.



In reply to Andras Balogh

Re: RAM requirement for 2.17

by Alex Jordan -

Did you do the kind of configuring described here?

https://webwork.maa.org/wiki/Installation_Manual_for_2.17_on_Ubuntu#Edit_mpm_prefork.conf

In reply to Alex Jordan

Re: RAM requirement for 2.17

by Andras Balogh -

I thought so. "5 MaxRequestWorkers per 1 GB of memory" and the mpm_prefork.conf file has

<IfModule mpm_prefork_module>
        StartServers                     5
        MinSpareServers           5
        MaxSpareServers          10
        MaxRequestWorkers         100
        MaxConnectionsPerChild   50
</IfModule>

MaxRequestWorkers  100 => 100/5 = 20 GB and the server has 24 GB.


In reply to Andras Balogh

Re: RAM requirement for 2.17

by Danny Glin -
WeBWorK is not great at cleaning up memory after processing problems, which is why the memory used by apache processes tends to grow, and the settings you quote above are necessary.

Those guidelines are pretty good for most scenarios. There are a couple of confounding factors that may be at play here:

  • When a gateway quiz is served to a student, WeBWorK processes all of the problems at once, which can cause an individual apache process to eat up more memory than you would otherwise see.  This means that gateway quizzes with a large number of problems cause apache to use more memory.  If a large number of students access a quiz in a short period of time, then this effect will compound before the process gets killed off (after serving 50 requests as per the configuration).
  • Since your system has a larger amount of RAM and you have set MaxRequestWorkers to a higher value, there is room for more ballooning of memory usage, so the values may need to be tweaked.

If you lower the value of MaxConnectionsPerChild you will likely see the amount of memory taken by the apache processes decrease, as they will be serving less requests before being killed off.  The tradeoff is that there is some overhead in starting new apache processes to replace them, which will lead to a performance hit.  I don't have any gauge of how significant that hit will be.

Another factor to consider is if you are using lighttpd to serve static content, then a larger proportion of the requests handled by apache will be memory intensive.  If that's the case then it is probably a good idea to lower MaxConnectionsPerChild.

Summary:

  • Check to see if one (or more) of your courses is running gateway quizzes with a large number of questions, and where many students start the quiz at the same time.
  • If you lower MaxConnectionsPerChild, you will likely see less memory used by each process, but more processor usage and disk I/O as new apache processes are started.
  • If you lower MaxRequestWorkers then the total memory used by all apache processes will decrease (since there will be less processes), but it will also reduce the number of concurrent requests your server can handle.  If there are more requests to the server than there are apache workers, then new requests will have to wait until an apache worker becomes available.
In reply to Danny Glin

Re: RAM requirement for 2.17

by Andras Balogh -

Thank you for the details answer.

Yes, more than half of the courses are using gateway quizzes regularly (chapter tests and three-part final exams).

We decreased MaxConnectionsPerChild from 50 to 40 for now.

I also read that omm_killer kills the process with the largest memory gain, unless it is prioritized. Since the mysqld process is using the most amount of memory, it was killed all the time. So we decreased mysqld's  OOM Score. 

In reply to Andras Balogh

Re: RAM requirement for 2.17

by Danny Glin -

Another thing that can make a big difference to server load is to allow students a time window to complete a quiz.

For example, if your quiz is from 1pm to 2pm with a 60 minute time limit, then all students will try to start the quiz at exactly 1pm.  Even making the quiz open from 12:55 until 2:05 with a 60 minute time limit prevents all students from starting the quiz at the same time, which could mean that your server never actually has to start 100 workers.

Of course whether this is feasible (or even applicable) will depend on your individual scenario.

In reply to Danny Glin

Re: RAM requirement for 2.17

by Andras Balogh -

Does it matter for the RAM when taking gateway quizzes whether all problems are shown on one page or it is one problem per page?

In reply to Andras Balogh

Re: RAM requirement for 2.17

by Nathan Wallach -

You may also want to set up Apache2::SizeLimit to have Apache kill workers that get very big.

See: https://webwork.maa.org/moodle/mod/forum/discuss.php?d=5008#p15317

In reply to Nathan Wallach

Re: RAM requirement for 2.17

by Andras Balogh -

Thanks, I will try it. This should be better than what we are doing right now. Right now the oom-killer is killing the apache processes once we set the mysqld's priority higher.