I am trouble shooting my install on our backup server, 2.12 on Ubuntu 16.04 LTS. Everything is running smoothly with the exception of the Rendering of problems within HW set editor.
I can select a problem and edit it, but if I click on Render All Problems, the system times out. I get an error box (actually two). The first says
Webwork2InstructorXMLHandler - Timeout
The second gives an option to ignore future error messages. In any case, the system never compiles the problems.
Any suggestions on where to start, for example one of the conf files? The OPL , PG and Mathjax are all up to date. I have restarted apache2 with no improvments.
The first thing to check with these sorts of issues is that $server_root_url is set correctly in site.conf, though if the individual render buttons are working, then I suspect that this is not the cause.
There are two places to check for error messages:
- The javascript console in your web browser. This differs from browser to browser, but is usually among the developer tools. This may give you a more meaningful error message.
- The apache error logs.
Hopefully one of these will give more detail which could help in tracking down the issue.
Also, have you tried a different browser? It's possible that this could be an issue with a specific browser.
Thanks. Not only will a single problem render fine, if I select automatically render all problems at the bottom of the hmwk set editor page and select save, the system will render all problems in the set. As you suspected, the url address is correct in site.conf. So the issue again is that if I click on render all problems button, the system locks up and I have to restart apache2. None of the problems render and I get time out errors.
Here is one of the errors I found in error.log within var/log/apache2
Here is one of the errors I found in error.log within var/log/apache2
[Tue Mar 28 11:36:03.283189 2017] [perl:error] [pid 28455] [client 66.190.230.112:62914] [/webwork2/instructorXMLHandler] Error message for command: renderProblem \n<br/>faultcode: 404 \n<br/>faultstring: WebworkWebservice: Can't authenticate -- session may have timed out. \n<br/>End error message<br/>\n * in WeBWorK::ContentGenerator::instructorXMLHandler::content called at line 233 of /opt/webwork/webwork2/lib/WeBWorK/ContentGenerator.pm\n * in WeBWorK::ContentGenerator::go called at line 382 of /opt/webwork/webwork2/lib/WeBWorK.pm, referer: http://mathdev.oit.edu/webwork2/3_Spring_Fischer_M112_2017/instructor/sets2/HW1.2_Radian_Formulas/?effectiveUser=fischerj&user=fischerj&key=3fSXGaTgJTMrpbmrxAHEOtQ0SX5Zz4Dw
Error message for command: renderProblem
<br/>faultcode: 404
<br/>faultstring: WebworkWebservice: Can't authenticate -- session may have timed out.
<br/>End error message<br/>
I also experienced this problem, usually only with problem sets that contain a larger number of problems. In such situation I have also seen lines as the following in the apache error log:
[Wed Mar 29 13:08:27.437553 2017] [core:notice] [pid 2529] AH00052: child pid 3871 exit signal Segmentation fault (11)
I do not know whether this is related. I am however not using 2.12, but merged
https://github.com/goehle/webwork2/tree/locbug
into the the master branch:
https://github.com/openwebwork/webwork2/tree/master
[Wed Mar 29 13:08:27.437553 2017] [core:notice] [pid 2529] AH00052: child pid 3871 exit signal Segmentation fault (11)
I do not know whether this is related. I am however not using 2.12, but merged
https://github.com/goehle/webwork2/tree/locbug
into the the master branch:
https://github.com/openwebwork/webwork2/tree/master
Are you using some form of external authentication, or just WeBWorK usernames and passwords?
Also, based on the previous post, have you tried this with a homework set that only has a couple of problems? It's possible that the apache processes are running out of resources trying to complete the request.
The error message and the fact that things render when you click the box at the bottom of the homework page suggests that this may be a deeper problem with the XMLHandler and the xmlrpc transport that is used for AJAX like rendering of each problem on the Homework page.
(I _think_ that when you click the box at the bottom of the page an submit the rendering is done all at once on the server and then passed back in a single page.) The javaScript render button requests that each problem to rendered individually via AJAX calls. This mechanism has not been as well tested and I have found it to fail on certain specific problems -- usually ones that involve matrices -- and I haven't yet had time to investigate the roots of the problem.
I don't know if that is the cause of this particular issue. Is there any sign that the errors occur only with certain problems? or as Danny has suggested, only when you try to render many problems together?
The sigfault is quite possibly related and indicates some fundamental problem -- I've seen sigfaults most often with graphics errors and occasionally with database communication errors.
I created a set and added one problem at a time while using the Rendering All Problems button after each addition. The feature worked fine until I got to 5 problems, then it would time out as described above. For now, I am fine with using the automatically render problems method by clicking the box at the bottom of the page and selecting save. However, it would be nice to see if there is a fix for my installation in case it is specific to my server or installation.
How much memory does this server have?
I have one more theory: when you use the automatically render problems method, then if Mike is correct it pre-renders all of the problems and serves the page back as a whole, which means that all of the work is done by one apache process. When you click the "render all" button, I believe that each problem sends its own request to the web server, which means that each problem is rendered by a separate apache process. If there is some issue with this server being able to handle many concurrent requests, then you would see this behaviour.
I agree with Danny that this is a possibility. I have midterms this week so I won't be able to do testing until the weekend. The segment faults might also becaused by server overload.
Thanks, this makes sense. The server I am using is older and possibly slower than our production server.
Is there an apache or other setting that I can adjust that might help remedy the situation?
Is there an apache or other setting that I can adjust that might help remedy the situation?
Here is more information. The webwork server is older and has 4 gb memory. Here are the settings in mpm-prefork.conf :
<IfModule mpm_prefork_module>
StartServers 5
MinSpareServers 5
MaxSpareServers 10
MaxRequestWorkers 5
MaxConnectionsPerChild 50
Instructions from the installation manual for 2.12 say the following:
For WeBWorK a rough rule of thumb is 5 MaxRequestWorkers per 1 GB of memory MaxRequestWorkers 5 MaxConnectionsPerChild 50
So I guess I need to change MaxRequestWorkers to 20.
What if any other parameters should I change to improve server speed? I noticed that when my class was in the computer lab and we had say 20 seats running webwork , the system was very very slow.
Later Edit: Tried changing the last two numbers, then restart apache2. Settled with
MaxRequestWorkers 25
MaxConnectionsPerChild 200
The editor now can render all with assignment as big as 15 problems. It timed out on a larger set with several graphics questions. But was able to render almost all, then I could render the last couple by clicking on individual render icon.
So the change was definitely good, I still wonder what numbers to use for optimal performance. I suspect the slowness in the computer lab will improve as well.
Here is one way to conserve the apache resources -
dial down MaxConnectionsPerChild to the minimum. This should help with reducing the chances of apache processes balooning up with memory consumption. Assuming 200MB per process and leaving memory for things like mysql, about 18 would be right for MaxRequestWorkers with 4 GB ram. Dial down StartServers, MinSpareServers and MaxSpareServers. Again helps with less chance of balooning of apache processes. At the end two more important things - switch over to lighttpd for static content and change keepalive to off. Last thing would be to run something like collectd on this server and you will have a better idea on how many apache processes got created, how much memory used, number of tcp connections etc so that could tweak and tune as needed. Hope this helps
I have the same problem when rendering all problems, but here is the errors I found in error.log
---------------
There were a lot of errors
Errors:
500 Can't connect to ebwork.cegepmontpetit.ca:443 at /opt/webwork/webwork2/lib/WebworkClient.pm line 293.
End Errors
xmlrpcCall to renderProblem returned no result for
[Fri Nov 03 09:07:37.491972 2017] [perl:error] [pid 19901] [client 192.168.145.63:51551] [/webwork2/instructorXMLHandler] xmlrpcCall to renderProblem returned no result for$
---------------
There were a lot of errors
Errors:
500 Can't connect to ebwork.cegepmontpetit.ca:443 at /opt/webwork/webwork2/lib/WebworkClient.pm line 293.
End Errors
xmlrpcCall to renderProblem returned no result for
[Fri Nov 03 09:07:37.491972 2017] [perl:error] [pid 19901] [client 192.168.145.63:51551] [/webwork2/instructorXMLHandler] xmlrpcCall to renderProblem returned no result for$
In the error you have "ebwork.cegepmontpetit.ca:443". I'm guessing that is supposed to have a w at the beginning. Check site.conf to see if this is what you have set for $server_root_url.
Danny paste error
I have the correct URL https://mydomaine set for $server_root_url.
I don't have the error message every time I check "Render All" on Homework problem set editor.
I have the correct URL https://mydomaine set for $server_root_url.
I don't have the error message every time I check "Render All" on Homework problem set editor.