WeBWorK Problems

OOM kills the mysqld process

OOM kills the mysqld process

by Sukhjit Singh Sehra -
Number of replies: 4

Hi Everyone, 

I use LTI authentication to navigate from D2L to Webwork for homework-type assignments with four sessions throughout the day. At the start of each session, the student group (around 100+) almost simultaneously clicks a link to the webwork assignment. We have had an odd issue with the third and fourth lab sessions for the two weeks. The MySQL of Webwork gets swamped and stops responding, and some students cannot log into Webwork. I checked OOM killer logs with

> grep "Killed process" /var/log/messages and found that: 

 Out of memory: Killed process 1911 (mysqld)

Certainly, the Webwork is running out of memory ( current RAM is 16 GB). I reviewed the MySQL settings and found that the maximum user limit is 300, with each connection using approximately 9.906MB of memory (max should be 3123.875 MB). Despite only having 107 students in this week’s lab session, we still experienced this issue. Any suggestions?


With mpm_prefork.conf:


In reply to Sukhjit Singh Sehra

Re: OOM kills the mysqld process

by Danny Glin -
I'm assuming that you pasted an image of your mpm_prefork.conf. Inline images don't seem to work on these forums, so can you post it as an attachment?

Since you have mpm_prefork.conf I'm assuming that you're running a version of WeBWorK earlier than 2.18. As of 2.18 WeBWorK no longer runs under apache, so the issues with memory usage will be different (and hopefully less of an issue).

It's likely not MySQL that is eating up all of the memory on your server. Since MySQL runs under a single process there is not a linear relationship between the number of connections and the memory used. In fact I've never noticed the memory usage grow as the number of connections increases.

I suspect that what is happening is that the apache processes are eating up all of the memory, which causes the OOM killer to kick in. Because MySQL is the single process with the highest memory usage it's the first to get killed. You should be able to adjust the oom_score for MySQL so that it isn't the first thing to get killed, but that's a band-aid since you're still dealing with a situation where your server is out of memory.

Also, does your server have swap space? I believe it's pretty standard on fresh linux installations, but I've heard of cloud providers who don't allocate any swap by default on newly-created VMs. Swapping leads to terrible server performance, but it's a lot better than the oom killer indiscriminately killing processes (as you've experienced).

Once you've reposted your mpm_prefork.conf we can give better suggestions about those settings, but the short answer is reducing MaxRequestWorkers and MaxConnectionsPerChild will lead to less total memory usage by apache.
In reply to Danny Glin

Re: OOM kills the mysqld process

by Sukhjit Singh Sehra -

Webwork 2.16 has been running for 2 years now, and the last time we saw this issue, we upgraded the RAM to 16GB and adjusted the mpm_prefork file (MaxRequestWorkers = 40, MaxConnectionsPerChild =50). I have now adjusted MySql Max_connections = 150 ( from 300 to 150 for sanity check as students as the student batch at one-time slot is always below 110 ) and oom_score=-1000 so OS does not kill it.

MPM PREFORK SETTINGS

<IfModule mpm_prefork_module>

    StartServers             5

    MinSpareServers       3

    MaxSpareServers       5

    MaxRequestWorkers     40

    MaxConnectionsPerChild   50

</IfModule>

MYSQL SETTINGS:

key_buffer_size = 8.000 MB

innodb_buffer_pool_size =128.000 MB 

innodb_log_buffer_size =16.000 MB 

sort_buffer_size=  0.250 MB 

read_buffer_size=  0.125 MB 

 read_rnd_buffer_size = 0.250 MB 

join_buffer_size = 0.250 MB 

thread_stack = 1.000 MB 

binlog_cache_size=   0.031 MB 

 tmp_table_size = 8.000 MB 

MEMORY PER CONNECTION= 9.906 MB 

max_connections =   150



In reply to Sukhjit Singh Sehra

Re: OOM kills the mysqld process

by Andras Balogh -
We had memory problem due to quiz submission bug in 2.17.
What we did was we prioritized mysqld so that it is not the first one that gets killed. I think by default OOM kills the process that has the largest memory footprint in order to get the largest gain. You can set it to have an exception for mysqld.
In reply to Sukhjit Singh Sehra

Re: OOM kills the mysqld process

by Nathan Wallach -

In addition to the changes to the Linux OOM settings, and to mpm_prefork.conf (lowering MaxConnectionsPerChild) you may want to try to use $Apache2::SizeLimit to limit the size of Apache server processes.

The idea is to prevent any individual apache process from getting to large. Since you are encountering problems, make sure that it polls the size often by using a small value of $Apache2::SizeLimit::CHECK_EVERY_N_REQUESTS 

To tune the size of $Apache2::SizeLimit::MAX_PROCESS_SIZE  keep an eye on the initial size of an Apache process and how it grows when many request are made (ex. when loading many problems). You want to allow moderate growth in size, but not to allow any process to get very big, as multiple oversized Apache processes are what is triggering the OOM condition.

See: https://webwork.maa.org/moodle/mod/forum/discuss.php?d=5008#p15317 and search the forums for older posts on this.