apache prefork memory consumption

apache prefork memory consumption

by Balagopal Pillai -
Number of replies: 4
 Our department operates a new webwork server (12 cores, 128 GB ram, debian 8, webwork 2.11) that is getting loaded now with more courses and there is one more on order (20 cores, 256 GB ram) to serve  a large 
course in fall (800+ students) So I was looking at the resource requirements for apache. Please see below from a test setup -

apache with ssl no webwork enabled on config -

 17835 www-data  20   0  277716  15616   8912 S   0.0  0.8   0:00.66 /usr/sbin/apach

apache with ssl  with webwork  apache config enabled - 

22130 www-data  20   0  489012 112412   1944 S   0.0  5.5   0:00.00 /usr/sbin/apach

           As you could see, enabling webwork bumps up the resident size to over 100 MB per process. On the production webwork server, I see that it grows continuously to a gigabyte and beyond. I have somewhat worked around it by dialing down MaxSpareServers,KeepAliveTimeout  and MaxConnectionsPerChild, adjusting MaxRequestWorkers and ServerLimit
by assuming 200 MB per apache process (instead of the 50 MB mentioned in one of the install guides) and with Apache2::SizeLimit using 600 MB virtual size as the limit for graceful termination of apache process (turns 
out to be 200 - 250 MB resident size) But if the apache process could use less than 50 MB resident size, it might be possible for me to squeeze in more courses on the server. 

         Please see the pmap output below. I could see a large anon allocation below apache that might be the reason for this ballooned up resident size -

0000558760782000       0       0       0 rw--- apache2
0000558760786000      12       8       8 rw---   [ anon ]
0000558760786000       0       0       0 rw---   [ anon ]
0000558761d48000  105160  105096  105096 rw---   [ anon ]

        I did some testing starting with a blank apache webwork config file and adding all lines up to 
eval "use lib '$pg_dir/lib'"; die $@ if $@;   Here is the process -

21945 www-data  20   0  324636  19200   1976 S   0.0  0.9   0:00.00 /usr/sbin/apach

        Then adding require Apache::WeBWorK; and restart apache. Please see below. This adds about
85 MB -

22136 www-data  20   0  480216 105604   1880 S   0.0  5.2   0:00.00 /usr/sbin/apach

        Then adding the rest of the lines back adds another few megabytes to the resident size -

22396 www-data  20   0  489004 112432   1972 S   0.0  5.5   0:00.00 /usr/sbin/apach

         Is there something in the webwork config I could modify to get the resident sizes to reasonable values and stop the ballooning effect when students access the courses? Thanks very much.

In reply to Balagopal Pillai

Re: apache prefork memory consumption

by Danny Glin -
The reason that WeBWorK adds 85MB initially to each apache process has to do with mod perl. All of the WeBWorK code is compiled into each apache process. This leads to better performance in serving WeBWorK pages, but also means that there is a large memory footprint.

As for the apache processing growing in memory usage over time, this is a result of a memory leak somewhere in the system. So far no one has been able to pin it down, or even determine if it can be fixed. The current practice is to keep the value of MaxConnectionsPerChild low enough that the processes don't get too big. On my system with MaxConnectionsPerChild set at 100 my apache processes rarely get over 200MB, with them usually dying off at just over 100MB (as an aside, at our WeBWorK administrators course we noticed that apache processes in Red Hat systems tended to consume about half as much memory as those in Ubuntu, so there may be something in the default denial config which causes this).

It also depends on how you use WeBWorK. Pages that load a large number of problems simultaneously cause a more pronounced ballooning of memory usage. The two main culprits are the Library Browser and gateway quizzes. You also have to consider how many simultaneous requests your server will be receiving. For a class of 800 this can vary wildly. We have used WeBWorK with classes this size, and for homework assignments which are due late at night you may never see more than a few students hitting the system at the exact same time. The other extreme is if for example you assign a gateway quiz which everyone must start at the same time. This will cause a huge server load, and you will actually risk overwhelming your system.

In short, if you set MaxConnectionsPerChild to something like 50 or 100, then your apache processes shouldn't get too large unless you are giving gateway quizzes with a large number of questions.
In reply to Danny Glin

Re: apache prefork memory consumption

by Davide Cervone -
The growth might not be entirely due to a "leak" per se. Not everything that WeBWorK uses is loaded initially, and so individual files are loaded as the process runs. This includes macro files used by the problems, but also the problem files themselves. I don't know how the mod_perl process handles these, but if they are also cached, that would account for the prices growing as time passes and it loads different problems which load more macro files, without there being an actual memory leak in the WeBWorK system itself. Just a thought.
In reply to Davide Cervone

Re: apache prefork memory consumption

by Balagopal Pillai -
Thanks. Centos 7 also ships with apache 2.4.x, like debian and ubuntu. Could this low memory consumption on redhat systems attribute to older versions of apache and mod_perl? (a wild guess) I will test it out with centos 6 and 7. Here are the settings I have now on the 128 GB server. This keeps the resident size mostly under control -

KeepAliveTimeout 2

 StartServers                    2
        MinSpareServers          2
        MaxSpareServers         5
        MaxRequestWorkers        600 (assuming 120 GB available for webwork)
        ServerLimit             600
        MaxConnectionsPerChild   100

$Apache2::SizeLimit::MAX_PROCESS_SIZE = 600000; (virtual size)
$Apache2::SizeLimit::MAX_UNSHARED_SIZE = 600000;
$Apache2::SizeLimit::CHECK_EVERY_N_REQUESTS = 20;

From what I can see, google chrome is the worst case keeping 6 connections active. With an 800+ student class, if at least half the students come in at the same time, theoretically  I could exhaust all 256 GB of ram assuming the ~200 MB prefork process size. Would an ngnix ssl proxy solution work in this case? Say chrome hits nginx with 6 connections, but nginx engages only one prefork apache connection. Would this break LTI? Thanks again for the responses
and help.  

In reply to Balagopal Pillai

Re: apache prefork memory consumption

by Balagopal Pillai -
I did some testing today. The proxying with nginx worked like a charm. But looks like I may not need it. Looking at established connections, every connection
to a test assignment with 32 questions from Chrome starts one established connection to apache and 4 or 5 to lighttpd. With the nginx proxy, I can see two connections to apache from nginx - most likely proxy connection in one direction and back. Since lighty doesn't mind lots and lots of connections to serve static content (I bumped up the server.max-fds ), I may be able to get away  with using one apache prefork connection per student with a very short keep alive  even if the professors decide to do a gateway quiz. MaxRequestWorkers and ServerLimit are set to 1200 assuming 240 GB of ram for webwork and 200 MB per prefork process. Thanks.