Installation

WeBWorK 2.18 Crashing

Re: WeBWorK 2.18 Crashing

by Danny Glin -
Number of replies: 6
When you say that it hung up, what exactly are you seeing? Does the page time out after some period of time?

If you catch the system locked up again, try connecting directly to the webwork service via http://webwork.yourschool.edu:8080/webwork2. Depending on your firewall rules you might not be able to access this from a browser, so from the command line via ssh you can try
curl http://localhost:8080/webwork2

If that returns the html of the webwork landing page, then the webwork service is running properly and the problem is likely somewhere else.

Also check the status of the apache2 service, and try restarting it. It's possible that apache is the thing that crashed and not WeBWorK.

The other thing to check is that the disk is not full, though if this were the case I would expect that you would be seeing other symptoms as well.
In reply to Danny Glin

Re: WeBWorK 2.18 Crashing

by Mark Watney -

Yes, when I say that it hangs up, I mean that the page times out after some period of time.  Nobody can access WeBWorK via web browser, despite systemctl reporting that the webwork2 service is up and running.  

I will try accessing the URL you suggested next time it hangs up, which should be fairly soon; I have crontab scheduling a mid day reboot and a student reported 36 minutes after the reboot that it hung up again.  This has gone from hang ups every few weeks to multiple times per day now.  

I can also check the status of apache2 via systemctl, but I suspect this will be up and running since I'm able to see the "It works!" apache landing page at http://webwork.myschool.edu, even when http://webwork.myschool.edu/webwork2 hangs up. 

In reply to Mark Watney

Re: WeBWorK 2.18 Crashing

by Danny Glin -

I forgot that you mentioned that the apache landing page still works, so that means that apache is still running and doing its job.

I wonder if the individual webwork processes are getting tied up by some request that doesn't finish, and once all of them are stuck there are none left to serve new requests.  The "top" command will show you information about running processes.  Hit "shift-M" to sort by memory usage, or "shift-P" to sort by processor usage.

Next time it happens, try stopping the webwork2 service, and then double-check to see if any webwork processes are still running.  You can use "ps -ef |grep webwork".  When you stop a service it tries to stop all related processes, but I've seen cases where some processes still end up running.

In reply to Danny Glin

Re: WeBWorK 2.18 Crashing

by Mark Watney -
Just did some testing at the most recent hang up. Accessing http://webwork.myschool.edu:8080/webwork2 directly (which only works when on my school's vpn due to firewall rules) actually did bring up the webwork login page, whereas http://webwork.myschool.edu/webwork2 was still timing out. I also verified that I'm able to access direct with the curl command, which did indeed return the html of the landing page.

Running 'ps -ef |grep webwork' yielded the following output:
www-data    1377       1  0 12:26 ?        00:00:02 /opt/webwork/webwork2/bin/webwork2
www-data    1379    1377  0 12:26 ?        00:00:04 /opt/webwork/webwork2/bin/webwork2
www-data    1380    1377  0 12:26 ?        00:00:03 /opt/webwork/webwork2/bin/webwork2
www-data    1381    1377  0 12:26 ?        00:00:01 /opt/webwork/webwork2/bin/webwork2
www-data    1383    1377  0 12:26 ?        00:00:02 /opt/webwork/webwork2/bin/webwork2
www-data    1384    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
www-data    1386    1377  0 12:26 ?        00:00:01 /opt/webwork/webwork2/bin/webwork2
www-data    1387    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
www-data    1389    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
www-data    1390    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
www-data    1391    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
www-data    1394    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
www-data    1395    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
www-data    1397    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
www-data    1398    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
www-data    1401    1377  0 12:26 ?        00:00:00 /opt/webwork/webwork2/bin/webwork2
wwadmin     4192    4176  0 13:42 pts/0    00:00:00 grep --color=auto webwork
Am I to take this to mean that there were 16 processes running?  I'm not sure what to make of this time stamp (note: I ran this command at 13:42).  Upon stopping the webwork2 service via systemctl, only the wwadmin process on the last line above remained. 


                                    

                
                
                
                                        
                
                
                                            

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to Mark Watney
                                        
Re: WeBWorK 2.18 Crashing

                                        

                                            by Glenn Rice - 
                                        

                                

                            

                
                            

                                        

                                        

                
                                

                                    

                                        
The STIME is the fifth column.  That is the start time of the process.
There will be as many processes as you have set for the number of workers in conf/webwork2.mojolicious.yml.  Those processes are the workers.

                                    

                
                
                
                                        
                
                
                                            

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

        

                

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to Mark Watney
                                        
Re: WeBWorK 2.18 Crashing

                                        

                                            by Danny Glin - 
                                        

                                

                            

                
                            

                                        

                                        

                
                                

                                    

                                        
Based on this it looks like webwork is running properly, but connections are not getting through for some reason.
It looks like your institution uses Cloudflare to proxy off-campus connections to your WeBWorK server.  You may need to engage your IT folks to see if there is anything showing up in the logs for the public-facing IP address that proxies to your WeBWorK server.

                                    

                
                
                
                                        
                
                
                                            

                                                    
                                            

                
                                        
                                

                            

                        

                    

                
        

                

                    

                        
                
                        

                            

                                        

                                                
                                        

                                

                                        In reply to Danny Glin
                                        
Re: WeBWorK 2.18 Crashing

                                        

                                            by Mark Watney - 
                                        

                                

                            

                
                            

                                        

                                        

                
                                

                                    

                                        
My institution does indeed use Cloudflare to proxy the requests to our server.  However, when a user is on campus and connected to wifi with the proper credentials, it bypasses Cloudflare.  Yet in this scenario the hang up is identical for the on-campus user, so would that rule out something funny going on with proxied connections?