WeBWorK Main Forum

system crash Can anyone help?

system crash Can anyone help?

by joel robbin -
Number of replies: 6
WeBWorK crashed at UW last night.

------------------------------------------------------------

Last night the following error message appeared:

Wed Oct 03 23:30:27 2007

Warning messages


Error messages

error instantiating DB driver WeBWorK::DB::Driver::SQL for table problem_user: DBI connect('webwork','webwork',...) failed: Too many connections at /usr/local/webwork2/lib/WeBWorK/DB/Driver/SQL.pm line 72

at /usr/local/webwork2/lib/WeBWorK.pm line 238

Call stack

The information below can help locate the source of the problem.

  • in Carp::croak called at line 221 of /usr/local/webwork2/lib/WeBWorK/DB.pm
  • in WeBWorK::DB::new called at line 238 of /usr/local/webwork2/lib/WeBWorK.pm


-----------------------------------------------------------

This morning I got the following error message when

I tried to sign on.



Thu Oct 04 06:41:46 2007

Warning messages


Error messages

error instantiating DB driver WeBWorK::DB::Driver::SQL for table problem_user: DBI connect('webwork','webworkRead',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (111) at /usr/local/webwork2/lib/WeBWorK/DB/Driver/SQL.pm line 62

at /usr/local/webwork2/lib/WeBWorK.pm line 238

Call stack

The information below can help locate the source of the problem.

  • in Carp::croak called at line 221 of /usr/local/webwork2/lib/WeBWorK/DB.pm
  • in WeBWorK::DB::new called at line 238 of /usr/local/webwork2/lib/WeBWorK.pm


In reply to joel robbin

Re: system crash Can anyone help?

by Fuzzy Rogers -
Joel-

I had this same problem a few days ago. My guess is that your MySQL server has a low default for the max_connections and probably a long timeout period. You can check this by accessing mysql via something like...
mysql -u root -p webwork
and entering the mysql password. Then give a
show global variables;
and check you max_connections and wait_timeout .
Often the default for max_connections is 100 and wait_timeout is 28800 (8 hours) (typical linux). Some online forums say max_connections can easily be extended to 500 or 600, but each new connection eats RAM, so be forewarned that you may run into more serious crashes (possible kernel panics) if your RAM runs out. You can change these variables on the fly with a mysql command like...
set global max_connections = 400;
set global wait_timeout = 300;
but on a restart, the values will return to their defaults. So you have to edit
/etc/my.cnf and put the appropriate lines in their. Under the general heading [mysqld]
max_connections 400
wait_timeout 300

I'm not positive of the implications of reducing the wait_timeout, so use that with knowing I'm in experimental mode at the moment. On our machine, we ran into a low RAM problem yesterday, so I added 2 GB (now 5GB, on a 32-bit system, not sure how linux will use that extra 1 GB). We'll see how the system performs over the next few days.
Our specs: WW 1.9 - 1000 students with precreated images
WW 2.3 - 1500 students
RedHat linux AS 4, 5 GB RAM, Dell PowerEdge 2850 (dual 2.8 single cores)

Hope that helps. On my previous thread about this crash I'll post any new information I come across regarding mysql, ram, and webwork.
Take care,
Fuzzy
In reply to Fuzzy Rogers

Re: system crash Can anyone help?

by Rob Owen -
I'm working on the same system as Joel, and I'm now gravely worried that the SQL tables for the entire class are damaged. One bright spark is that this appears confined to the single class, every other class seems ok; but I have no idea how to go about 1) analyzing the table for damage, 2) figuring out what that damage is, 3) fixing the damage and 4) (by far the most important) preventing such crashes in the future. If anyone can offer specific suggestions on what to do or look for, I'd be greatly appreciative.

Thanks,

-- Robert
In reply to Rob Owen

Re: system crash Can anyone help?

by Gavin LaRose -
Hi Rob,

At some point I was having trouble with periodic database table corruption when we were running on Solaris servers, and ended up putting in a script to check and repair the mySQL database tables on a regular basis. It's been a while since that's been an issue for us, though, as the Linux boxes we're now running on don't seem to have the same problem.

The check/repair commands I was using were
mysql> check table [tablename];
and, when there were errors reported,
mysql> repaire table [tablename];

I have a script that I was using to do this as a cronjob; if that's useful I'm happy to provide it for your modification and experimentation.

Gavin
In reply to Fuzzy Rogers

Re: system crash Can anyone help?

by joel robbin -
Hi Fuzzy --
How many students are using your system? Perhaps
our server is just overloaded.
-- Joel

In reply to joel robbin

Re: system crash Can anyone help?

by Fuzzy Rogers -
About 2500 students on one DELL Poweredge 2850 Dual 2.8 GHz single core xeons (I think xeons) with fast scsi hard drives (not sure if they're 10K or 15K rpms) and 5 GB RAM.
WW 2.3- 1100 students
WW 1.9 - 1400 students

You can imagine we're pushing the limits of what's possible with that hardware. I've noticed that my apache (1.33 compiled by me) has a memory leak so I restart it once a day with a cron job. Currently, at 11am top shows that httpd & mysqld & linux are taking up 3.5 GB of memory with 1.5 GB available. This drops significantly in the evening as students log on, but so far the swap hasn't been hit too hard and no students have since received the mysql "too many connections" error

Also, for the 1.9 courses I precreate the images to speed things up. The reason we still use 1.9 is because of formatting issues in the .pg files between 1.9 and 2.

According to mysql, the max. connections I've had is 264 (max set to 500), with a timeout set to 600.

Take care,
Fuzzy
In reply to Fuzzy Rogers

Re: system crash Can anyone help?

by Fuzzy Rogers -
Whoops- should have read my earlier post... got my numbers backwards:

Our specs: WW 1.9 - 1000 students, 3 lectures, with precreated images
WW 2.3 - 1500 students , 6 lectures

Take care,
Fuzzy