WeBWorK Main Forum

Memory Usage & Problem Connecting with mySQL server

Memory Usage & Problem Connecting with mySQL server

by Thomas Hagedorn -
Number of replies: 6

We've been getting the following error occasionally in some of our courses late at night. The server restarts at 5am and the errors then seem to be gone. We have about 500 users on a 4-processor 2gB system. We've haven't had this problem before this semester, but we upgraded from 2.1 to 2.4.(latest stable version as of Jan.1) in January. We ran a similar version (2.4.(x-1)) last semester on another server with more memory, but never saw this problem (again, we rebooted every morning). In the forum, I didn't see anyone having this problem with mySQL and the table "set_user". I looked at the answers to similar sounding problems involving mySQL and the table "problem_user" but don't think they apply here (the values in global.conf are all set using single-quotes, the values for max_user and wait_timeout seem reasonable, and the errors don't show up after a reboot).

Could this error simply be Webwork running out of memory?

-Tom


WeBWorK error

An error occured while processing your request. For help, please send mail to this site's webmaster (root@fermat.TCNJ.EDU), including all of the following information as well as what what you were doing when the error occured.

Mon Feb 18 02:35:30 2008

Warning messages

Error messages

error instantiating DB driver WeBWorK::DB::Driver::SQL for table set_user: DBI connect('webwork','webworkWrite',...) failed: Can't connect to local MySQL server through socket '/var/lib/mysql/mysql.sock' (2) at /local/webwork2/lib/WeBWorK/DB/Driver/SQL.pm line 65
at /local/webwork2/lib/WeBWorK.pm line 286

Call stack

The information below can help locate the source of the problem.

  • in Carp::croak called at line 208 of /local/webwork2/lib/WeBWorK/DB.pm

  • in WeBWorK::DB::init_table called at line 200 of /local/webwork2/lib/WeBWorK/DB.pm

  • in WeBWorK::DB::init_table called at line 169 of /local/webwork2/lib/WeBWorK/DB.pm

  • in WeBWorK::DB::new called at line 286 of /local/webwork2/lib/WeBWorK.pm

Request information

Method

GET

URI

/webwork2/Linear_Algebra_Conjura_S08/

HTTP Headers

User-Agent

Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; MathPlayer 2.10b; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; SLCC1; .NET CLR 2.0.50727; Media Center PC 5.0; .NET CLR 3.0.04506)

Accept

*/*

Connection

Keep-Alive

UA-CPU

x86

Referer

http://webwork.tcnj.edu/webwork/

Accept-Encoding

gzip, deflate

Cookie

__utma=258717071.118389005.1179681984.1201199102.1201290702.26; __utmz=258717071.1195560589.18.1.utmccn=(direct)|utmcsr=(direct)|utmcmd=(none)

Accept-Language

en-us

Host

webwork.tcnj.edu

In reply to Thomas Hagedorn

Re: Memory Usage & Problem Connecting with mySQL server

by Sam Hathaway -
Have you checked the MySQL error logs? Can you make sure MySQL is even running when this error occurs? Also check to see if it's accepting connections at all -- use ``mysql'' to connect and see if that works. I think one gets a different error if MySQL has reached its connection limit, and this message is pretty much just libmysql saying "I tried to open this socket and no one was listening".
In reply to Sam Hathaway

Re: Memory Usage & Problem Connecting with mySQL server

by Thomas Hagedorn - 
Dear Sam,

Thanks for the reply. We had another crash last night and I was
able to inspect the system live. The error message was indeed
caused by mySQL not running. But it appears that memory is still
the problem as the system had run out of memory, and as part of
it's automatic recovery, the OS starts killing off user processes
to try to recover. One of those processes it killed off was mySQL.
As soon as mySQL was restarted, the system was fine. We are putting
more memory in the system today and we'll see if that helps.

Thanks,
Tom

(Edited by Michael Gage - original submission Tuesday, 19 February 2008, 11:39 AM) -- to break long line

In reply to Thomas Hagedorn

Re: Memory Usage & Problem Connecting with mySQL server

by Michael Gage -
Hi Tom,

How many simultaneous clients do you allow? We keep the number
pretty low to keep too many WeBWorK processes starting up. Figure
150Meg or more for each process. (I've seen large processes as well -- due to
some memory leaks in which case you want the processes to start more often.)

Here is part of our httpd.conf file for the apache server for reference:

Timeout 1200
KeepAlive On
#MaxKeepAliveRequests 100
MaxKeepAliveRequests 25
KeepAliveTimeout 10
StartServers 5
MinSpareServers 5
MaxSpareServers 10
#sam# Default is 150, but was previously set to 10. Trying 50 for now to 
#sam# see if there'll be any problems.
#mike# changed back to 10 clients then inched back up to 12.
MaxClients 12
MaxRequestsPerChild 100

Hope this helps.


-- Mike

In reply to Michael Gage

Re: Memory Usage & Problem Connecting with mySQL server

by Thomas Hagedorn -
Hi Mike,

We have MaxClients =150, and MaxRequestsPerChild =0. I'm not sure what you mean by "simultaneous clients". From the 'ps aux' command data below, it seems we had 10 copies of apache running and each using about 9% of system memory. Does each of these count as a client? How few clients can we get away with?

-Tom

Here's the information from the 'ps aux' command:


wwserver 24950 0.4 8.1 160440 83532 ? S 18:00 1:08 /local/apache/bin/httpd
wwserver 24951 0.3 7.5 145856 77860 ? S 18:00 0:55 /local/apache/bin/httpd
wwserver 24952 0.4 7.7 169824 79052 ? S 18:00 1:02 /local/apache/bin/httpd
wwserver 24953 0.4 8.6 159736 88196 ? S 18:00 1:05 /local/apache/bin/httpd
wwserver 24954 0.4 9.5 170316 97520 ? S 18:00 1:03 /local/apache/bin/httpd
wwserver 24957 0.3 7.6 157696 78140 ? S 18:00 0:58 /local/apache/bin/httpd
wwserver 24960 0.3 9.4 165616 97188 ? S 18:00 0:58 /local/apache/bin/httpd
wwserver 24961 0.4 9.8 183220 100496 ? S 18:00 1:10 /local/apache/bin/httpd
wwserver 24962 0.3 8.6 164016 88884 ? S 18:00 0:57 /local/apache/bin/httpd
wwserver 24963 0.3 8.1 144380 83248 ? S 18:00 0:59 /local/apache/bin/httpd
In reply to Thomas Hagedorn

Re: Memory Usage & Problem Connecting with mySQL server

by Michael Gage -
10 processes -- each using 10% of memory is probably risky. Try setting
MaxClients to 8 and MaxRequestsPerChild to 100 (don't forget to restart the server)

See if that stops the memory problems. Because each WeBWorK process is pretty large you don't want to start too many of them or the system will spend a lot of time swapping them in and out of memory. Keeping the number of clients low keeps the number of processes low.

If you get more memory for the machine you can inch the number of MaxClients upward for faster service.
In reply to Thomas Hagedorn

Re: Memory Usage & Problem Connecting with mySQL server

by Davide Cervone -
There were some changes to the MathObjects library to address memory leak issues about 5 months ago. These are not yet included in an official release, so you would not have gotten them if you updated to 2.4.0. Perhaps you should try updating your pg directory to HEAD rather than 2.4 and see if that improves your memory situation. You can do that without updating webwork2, as the two directories are pretty independent.

Davide