WeBWorK Main Forum

Webwork Database is not starting

Webwork Database is not starting

by Bianca Sosnovski -
Number of replies: 5

Hi everyone,

Yesterday, our Webwork stopped running, meaning that the our Webwork's website  was unreachable.

After contacting our IT about it, the server was checked and it had an error about memory. Restarting the server made our Webwork to run again just fine but by the end of the day nobody could access it.

Last night until this morning, we were able to see the Webwork's website but trying to access any of the accounts showed the following error:

Error messages

error instantiating DB driver WeBWorK::DB::Driver::SQL for table problem_user: DBI connect('webwork','webworkWrite',...) failed: Too many connections at /opt/webwork/webwork2/lib/WeBWorK/DB/Driver/SQL.pm line 70. at /opt/webwork/webwork2/lib/WeBWorK.pm line 317.

Call stack

The information below can help locate the source of the problem.

  • in Carp::croak called at line 269 of /opt/webwork/webwork2/lib/WeBWorK/DB.pm
  • in WeBWorK::DB::init_table called at line 228 of /opt/webwork/webwork2/lib/WeBWorK/DB.pm
  • in WeBWorK::DB::new called at line 317 of /opt/webwork/webwork2/lib/WeBWorK.pm
I spoke with IT again this morning and when checking the server it showed the error message about corrupt memory. IT then switched our Webwork to a different server that was available, thus changing the memory.

We still able to see the website but then the error changed to the following:

Error messages

error instantiating DB driver WeBWorK::DB::Driver::SQL for table locations: DBI connect('webwork','webworkWrite',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /opt/webwork/webwork2/lib/WeBWorK/DB/Driver/SQL.pm line 70. at /opt/webwork/webwork2/lib/WeBWorK.pm line 317.

Call stack

The information below can help locate the source of the problem.

  • in Carp::croak called at line 269 of /opt/webwork/webwork2/lib/WeBWorK/DB.pm
  • in WeBWorK::DB::init_table called at line 228 of /opt/webwork/webwork2/lib/WeBWorK/DB.pm
  • in WeBWorK::DB::new called at line 317 of /opt/webwork/webwork2/lib/WeBWorK.pm
After checking the listed files above in /opt/webwork/webwork2/lib, it seems to me that the database cannot be started, which is confirmed when I try to access it directly in the server:

root@webwork:~# mysql

ERROR 2002 (HY000): Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2)

I tried the following commands but no success:

mysql start

/etc/init.d/mysql start

Any suggestions how to fix this issue.

Thank you.




In reply to Bianca Sosnovski

Re: Webwork Database is not starting

by Danny Glin -
There are a couple of things that will affect how you start the database.
The first thing to note is that the command to start the DB needs to be run as root, so if you're not logged in as root then you will have to preface the command with "sudo".
The command you indicated: "/etc/init.d/mysql start" is the old way of starting services in linux. This will likely work in versions of Ubuntu earlier than 16, and versions of RHEL/CentOS before 7. In those cases the preferred command is "service mysql start".

It's also possible that the script for starting the DB goes by a different name, so depending on your system it may be "mysqld" or "mariadb" instead of "mysql".

If you are on a more recent version of linux (Ubuntu 16 or newer, RHEL/CentOS 7 or newer), then the command you want is
systemctl start mysql

(where "mysql" may need to be replaced with "mysqld" or "mariadb").

Can you please tell us what version of linux you are using, and any error messages you get when running these commands? It's also worth checking that your server has not run out of disk space.
In reply to Danny Glin

Re: Webwork Database is not starting

by Bianca Sosnovski -
Hi Danny,

Thank you for your reply.

Yes, our server is out of disk space.

I just found in /var/log/apache2/error.log the following error message about "No space left on device":

[Tue Apr 13 18:53:15.679117 2021] [perl:error] [pid 1056] [client 69.196.253.30:17460] [/webwork2/MA119-S21-EStelmach-PN15/Exam_3_Review_Section_15-21] error instantiating DB driver WeBWorK::DB::Driver::SQL for table location_addresses: DBI connect('webwork','webworkWrite',...) failed: Can't connect to local MySQL server through socket '/var/run/mysqld/mysqld.sock' (2) at /opt/webwork/webwork2/lib/WeBWorK/DB/Driver/SQL.pm line 70.\n at /opt/webwork/webwork2/lib/WeBWorK.pm line 317.\n * in Carp::croak called at line 269 of /opt/webwork/webwork2/lib/WeBWorK/DB.pm\n * in WeBWorK::DB::init_table called at line 228 of /opt/webwork/webwork2/lib/WeBWorK/DB.pm\n * in WeBWorK::DB::new called at line 317 of /opt/webwork/webwork2/lib/WeBWorK.pm
[Tue Apr 13 18:53:15.680052 2021] [log_config:warn] [pid 1056] (28)No space left on device: [client 69.196.253.30:17460] AH00646: Error writing to /var/log/apache2/access.log
[Tue Apr 13 18:53:15.680115 2021] [log_config:warn] [pid 1058] (28)No space left on device: [client 69.196.253.30:46643] AH00646: Error writing to /var/log/apache2/access.log
[Tue Apr 13 18:53:16.112863 2021] [log_config:warn] [pid 1057] (28)No space left on device: [client 66.249.64.22:57713] AH00646: Error writing to /var/log/apache2/access.log, referer: https://webwork.qcc.cuny.edu/webwork2/jfunk/
[Tue Apr 13 18:53:16.874148 2021] [log_config:warn] [pid 1059] (28)No space left on device: [client 143.48.84.164:51270] AH00646: Error writing to /var/log/apache2/access.log
[Tue Apr 13 18:53:17.151804 2021] [log_config:warn] [pid 1059] (28)No space left on device: [client 143.48.84.164:51270] AH00646: Error writing to /var/log/apache2/access.log, referer: https://webwork.qcc.cuny.edu/webwork2/
[Tue Apr 13 18:53:17.269344 2021] [log_config:warn] [pid 1102] (28)No space left on device: [client 108.5.56.110:65051] AH00646: Error writing to /var/log/apache2/access.log
[Tue Apr 13 18:53:17.345046 2021] [log_config:warn] [pid 1102] (28)No space left on device: [client 108.5.56.110:65051] AH00646: Error writing to /var/log/apache2/access.log, referer: https://webwork.qcc.cuny.edu/webwork2
[Tue Apr 13 18:53:17.462479 2021] [log_config:warn] [pid 1102] (28)No space left on device: [client 108.5.56.110:65052] AH00646: Error writing to /var/log/apache2/access.log, referer: https://webwork.qcc.cuny.edu/webwork2

I guess that is the main issue that initiated the server not being able to be accessed at all.
I'll have to ask IT take care of the issue of no space on device and hope they know more about it.
Any information that you may know about how to resolve the issue is welcome.

Answering your questions, our server is running on Ubuntu 20.04.1 LTS.
I'm running the commands as root. Also systemctl start mysql doesn't start the database:

root@webwork:~# systemctl start mysql
Job for mysql.service failed because the control process exited with error code.
See "systemctl status mysql.service" and "journalctl -xe" for details.

Thank you again!
In reply to Bianca Sosnovski

Re: Webwork Database is not starting

by Danny Glin -

If your server is a virtual machine then your IT may simply be able to increase the size of the disk.  Otherwise you'll have to delete some files to make space.  Here are a few general suggestions:

I believe the WeBWorK timing log is enabled by default, which means that there is likely a large file at /opt/webwork/webwork2/logs/timing.log if your server has been around for a long time.  You can safely delete that file.

WeBWorK does not automatically delete images or pdfs that are generated for students after they are used.  This means that there could be a lot of stuff in /opt/webwork/webwork2/htdocs/tmp.  Since these are all temporary files they can be deleted.  WeBWorK will recreate any files that are needed.

The apache logs can get quite large as well.  I believe that logrotate is already set up on most systems to rotate these logs (i.e. only keep the log for a fixed amount of time), but you can verify whether this is the case.

In reply to Danny Glin

Re: Webwork Database is not starting

by Arnold Pizer -
There are instructions about setting up log rotation for the timing log at  
https://webwork.maa.org/wiki/Installation_Manual_for_2.15_on_Ubuntu_20.04_Server#Implement_Optional_E_.28Logrotate.29


Also look at https://webwork.maa.org/wiki/Clean_Out_Temporary_Files for information on setting up a cron job to remove temporary files automatically.



In reply to Arnold Pizer

Re: Webwork Database is not starting

by Bianca Sosnovski -
Our Webwork server is back running again.

Following your suggestions, I deleted /opt/webwork/webwork2/logs/timing.log that was about 190 Mb in size. That didn't solve the issue. But just in case, I created the configuration file for timing.log in logrotate.

The latest installation of the server has option A implemented that removes temporary files (pdf, images, etc) with a cron job. Not something that would solve the issue neither.

I found out that the /opt/webwork/webwork2/logs/debug.log was huge, 97Gb in size. It was turned on because of another issue that I reported in February (https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4955#p16130). 
After deleting debug file, I was able to restart the database and Webwork is running again.

I still plan to keep the debug on for a bit longer, just in case, because of the other issue above reported in the forum. I also configured file in logrotate for the debug to keep compressed weekly backups of the debug for up to 4 weeks to avoid the file to become so big as before.

I also requested our college IT to increase the disk space  in the Virtual Machine for the server. Probably, IT will do it tomorrow morning.

Thank you so much for all the help.