Our webwork server crashes pretty frequently. This week it crashed twice.
It definitely happens on nights when HW is due the next day. We've tried to stagger these due dates between our classes, but the problem has gotten worse.
Sometimes I can ssh in after a crash and restart apache, and that works. But often I can't ssh in, and the machine has to be rebooted at the switch the next morning.
Just wondering if others have this problem. Will adding more RAM help? Other ideas?
Also, what are the plans for the MAA webwork hosting referred to in the following post (3rd paragraph)?
http://www.maa.org/columns/launchings/launchings_04_09.html
I'm liking that idea a lot.
Many thanks!
Andy
Have you tweaked your apache settings in httpd.conf? There is a memory leak somewhere in mod perl which makes the default apache settings prone to running out of memory.
Take a look at the following posts:
http://wwrk.maa.org/moodle/mod/forum/discuss.php?d=129#p6963
http://wwrk.maa.org/moodle/mod/forum/discuss.php?d=157#p7480
They describe more reasonable settings which will most likely prevent your servers from running out of memory.
Take a look at the following posts:
http://wwrk.maa.org/moodle/mod/forum/discuss.php?d=129#p6963
http://wwrk.maa.org/moodle/mod/forum/discuss.php?d=157#p7480
They describe more reasonable settings which will most likely prevent your servers from running out of memory.
Hi,
This term we are using WeBWork for about 10 classes compared
to about 6 last term. While I did not see specific errors, the system
just started to crawl to a halt from time to time. Indeed, memory
leaks seemed to be the issue which I concluded by watching the
system memory completely disappear over the course of the day.
I conlclude that it was the nightly apache restart that cleared the
issue.
As a fix I now have a hourly cron job restart httpd, which solved the issue, at least this term. I will look into our apache settings also to see if the tweaks noted in the previous posts help.
It may be useful to add a note or section in the administrators
section of the wiki about this issue.
- Hans
This term we are using WeBWork for about 10 classes compared
to about 6 last term. While I did not see specific errors, the system
just started to crawl to a halt from time to time. Indeed, memory
leaks seemed to be the issue which I concluded by watching the
system memory completely disappear over the course of the day.
I conlclude that it was the nightly apache restart that cleared the
issue.
As a fix I now have a hourly cron job restart httpd, which solved the issue, at least this term. I will look into our apache settings also to see if the tweaks noted in the previous posts help.
It may be useful to add a note or section in the administrators
section of the wiki about this issue.
- Hans
I have had this experience when moving from a small to large number of users (100 to 1000). You need to balance the number of apache children with the amount of RAM you have. I seem to remember that 60MB per child was about right.
Hi Andrew,
Did you look in the apache error.log ? I have seen these symptoms on more than one webwork server. If the problem is frequent segmentation faults, resulting in blank web pages and unresponsiveness, the problem may be that you are using a buggy version of the DBD::mysql perl module - see this post:
http://wwrk.maa.org/moodle/mod/forum/discuss.php?d=186
The fix I used is to restart apache gracefully every 8 hours with a crontab entry. That fixed my problem completely.
Lars.
Did you look in the apache error.log ? I have seen these symptoms on more than one webwork server. If the problem is frequent segmentation faults, resulting in blank web pages and unresponsiveness, the problem may be that you are using a buggy version of the DBD::mysql perl module - see this post:
http://wwrk.maa.org/moodle/mod/forum/discuss.php?d=186
The fix I used is to restart apache gracefully every 8 hours with a crontab entry. That fixed my problem completely.
Lars.
Thanks for all your comments and suggestions- we'll try restarting apache regularly!
Actually I looked at /var/log/cron, and we already have apache being restarted every 6 hours.
Still have to find where the apache crash log is located. ("locate error.log" doesn't turn up anything.)
We have about 500 users total, but some are in large classes all with sets due at the same time. Is it really OK to set
MaxClients 15
(currently we have it as 150).
The comment in httpd.conf says extra users will be locked out (not sure what that actually means). Does this just make the server respond more slowly?
Thanks.
Still have to find where the apache crash log is located. ("locate error.log" doesn't turn up anything.)
We have about 500 users total, but some are in large classes all with sets due at the same time. Is it really OK to set
MaxClients 15
(currently we have it as 150).
The comment in httpd.conf says extra users will be locked out (not sure what that actually means). Does this just make the server respond more slowly?
Thanks.
Setting MaxClients at 15 (or less if you have less RAM) is a good idea. If you set it higher then if at some point 20 students make requests nearly simultaneously you will spawn 20 processors -- each taking up 100+Megs of memory -- and they won't go away. Eventually this results in a lot of virtual memory swapping which just wastes time. If MaxClients is set at 15 then 5 of the 20 students might have to wait an extra 0.5 seconds before their request is even considered so they will notice a slower response on that request, but there is no long term impact resulting in excessive memory swapping.
One other thing. I have noticed on our machine that using
apachectl graceful
to restart the apache children does NOT clear out the memory usage. I have to use apachectl stop followed by apachectl start to clear the memory. (I'm not sure why.) This causes an interruption of service to anyone who is using the server that instant. At worst the interruption might last a second or two and require the user to repeat a request.
Hope this helps.
Mike
One other thing. I have noticed on our machine that using
apachectl graceful
to restart the apache children does NOT clear out the memory usage. I have to use apachectl stop followed by apachectl start to clear the memory. (I'm not sure why.) This causes an interruption of service to anyone who is using the server that instant. At worst the interruption might last a second or two and require the user to repeat a request.
Hope this helps.
Mike
At Indiana University Bloomington we use Apache::SizeLimit in combination with the MaxClients setting to solve this problem, when using Apache1.
To determine a value for MaxClients, we find out how much RAM is available when everything is running EXCEPT for Apache, then divide that by 120M, and then round down and subtract 3, 4, or 5.
Then we configure Apache::SizeLimit to kill Apache children when they exceed 100M.
Also, at 5 am each morning we stop Apache, rotate the log files, and then restart Apache.
Under normal circumstances, this has solved the problem even for servers that have as many as 1,400 users with twice/thrice weekly assignments.
The only failures are usually due to inadvertent Denial of Service attacks due to deficiencies in some browsers.
Sincerely,
Bill Wheeler
To determine a value for MaxClients, we find out how much RAM is available when everything is running EXCEPT for Apache, then divide that by 120M, and then round down and subtract 3, 4, or 5.
Then we configure Apache::SizeLimit to kill Apache children when they exceed 100M.
Also, at 5 am each morning we stop Apache, rotate the log files, and then restart Apache.
Under normal circumstances, this has solved the problem even for servers that have as many as 1,400 users with twice/thrice weekly assignments.
The only failures are usually due to inadvertent Denial of Service attacks due to deficiencies in some browsers.
Sincerely,
Bill Wheeler