WeBWorK Main Forum

WeBWorK is always restarting.....

WeBWorK is always restarting.....

by Dongchuan Luo -
Number of replies: 9
I maintain a WeBWorK 2.12 running on CentOS 7. I noticed that there are a lot of "WeBWorK is starting....." statements in /var/log/messages file, 3 or 4 in each minute.

Jan 12 10:40:04 webwork2 httpd: webwork.apache2-config: WeBWorK server is starting
Jan 12 10:40:04 webwork2 httpd: webwork.apache2-config: WeBWorK root directory set to /opt/webwork/webwork2 in webwork2/conf/webwork.apache2-config
Jan 12 10:40:04 webwork2 httpd: webwork.apache2-config: The following locations and urls are set in webwork2/conf/site.conf
Jan 12 10:40:04 webwork2 httpd: webwork.apache2-config: PG root directory set to /opt/webwork/pg
Jan 12 10:40:04 webwork2 httpd: webwork.apache2-config: WeBWorK server userID is apache
Jan 12 10:40:04 webwork2 httpd: webwork.apache2-config: WeBWorK server groupID is apache
Jan 12 10:40:04 webwork2 httpd: webwork.apache2-config: The webwork url on this site is https://webwork.act.utoronto.ca/webwork2
Jan 12 10:40:04 webwork2 httpd: WebworkSOAP::WSDL: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
Jan 12 10:40:04 webwork2 httpd: WebworkSOAP::WSDL: rpc_url set to https://webwork.act.utoronto.ca/webwork2_rpc
Jan 12 10:40:04 webwork2 httpd: WebworkWebservice: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
Jan 12 10:40:29 webwork2 httpd: webwork.apache2-config: WeBWorK server is starting


I'd like to know whether it is normal?

Thanks

Dongchuan
In reply to Dongchuan Luo

Re: WeBWorK is always restarting.....

by Danny Glin -
I just checked on my CentOS 7 server (running WW 2.13), and I'm seeing similar behaviour, though not every minute.

Looking at the code, it looks like this is expected. Every time a new child process starts, it will log this information. The recommended apache configuration for WeBWorK involves killing apache processes after 100 requests, so new processes will be starting often.

If the large number of messages are a problem, you can comment out the lines that start with "print" in /opt/webwork/webwork2/conf/webwork.apache2.4-config.dist.

In reply to Dongchuan Luo

Re: WeBWorK is always restarting.....

by Lars Jensen -
Hi Dongchuan,

We don't see logs like yours. We're on a debian server running WeBWorK-2.13.

Lars.
In reply to Lars Jensen

Re: WeBWorK is always restarting.....

by Dongchuan Luo -
Hi Lars,

The frequence in my case is too high. I guess it has relationship with the number of user and the configuration of Apache, such as MaxRequestWorkers and so on. There are 3400 students to use it in this winter term. Anybody can give some suggestion about configuration for Apache?

Thanks a lot.

Dongchuan
In reply to Dongchuan Luo

Re: WeBWorK is always restarting.....

by Dongchuan Luo -
I used the Apache2.4 on centos 7, its module is prefork.

Thanks
In reply to Dongchuan Luo

Re: WeBWorK is always restarting.....

by Bob Carmichael -
I'm sure you've seen these settings but here they are:


Can you post your current configuration? e.g.,

StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxRequestWorkers 150
MaxConnectionsPerChild 0

Are you using NewRelic or do you have monitoring to see the load on the server? If MaxConnectionsPerChild is low and traffic is high, you might see the behavior you are experiencing, but it's hard to know without more visibility into your server activity. The only other explanation I can think of would be a hard crash of the process during a request, which would hopefully show up in the error log.
In reply to Bob Carmichael

Re: WeBWorK is always restarting.....

by Dongchuan Luo -
Hi Bob,

My current configuration is:

Timeout 1200
MaxRequestWorkers 80
MaxConnectionsPerChild 100
ExtendedStatus On

I monitored the system and found the load is not heavy, there is always 5, 6 G free memory(total memory is 8G) and the cpu usage keeps around 50%, occasionally reach 75%. According to my monitor there is no Apache restarting.

Thanks

Dongchuan
In reply to Dongchuan Luo

Re: WeBWorK is always restarting.....

by Danny Glin -
Everything is working as expected, including the log messages.

Since you didn't post the value of MaxSpareServers, I'm assuming you're using the default value, which is 10.

What that means is that your apache server will run somewhere between 5 (MinSpareServers) and 10 (MaxSpareServers) apache processes waiting for connections. If there are more simultaneous requests, the server will start more processes to serve them, up to a maximum of 80 (MaxRequestWorkers). Once those requests are completed, it will kill off the extra processes until it is down below 10 spares (MaxSpareServers).

In addition, after a given process has served 100 requests (MaxConnectionsPerChild), it will be killed off and replaced by a new process. This is important for WeBWorK, because processing a pg problem adds a significant amount of memory to an apache process, so each time a problem is processed, the memory footprint of the process increases.

Each time a new process is started for one of the reasons above, you will see those entries in your log file. So for example, if the server is idle, and then 80 students click on a problem at about the same time, apache will start 70 additional processes to handle all of the requests, leading to 70 new log entries for WeBWorK starting.

I'm guessing that Lars doesn't see this on his Debian server because it is configured not to log messages from individual apache processes to /var/log/messages.