Forum archive 2000-2006

Nandor Sieben - monitoring bad processes during development

Nandor Sieben - monitoring bad processes during development

by Arnold Pizer -
Number of replies: 0
inactiveTopicmonitoring bad processes during development topic started 6/17/2005; 1:34:38 PM
last post 6/22/2005; 10:39:52 PM
userNandor Sieben - monitoring bad processes during development  blueArrow
6/17/2005; 1:34:38 PM (reads: 1070, responses: 2)
We have a 3GH dual processor machine with 4G ram for webwork. We use this machine to create problems. There are about 8 people working on this machine. We also have a few summer classes using the same machine (not a good idea, I know now). An infinite loop or other mistakes in a pg file or a macro package is unavoidable during development. Unfortunately the result of this is offen a creation of a process that takes over the machine. A process can eat up 80% of the memory. When this happens, the system load can go up to 16 from the usual less than 1. The machine becomes unusable at this time. Since these processes are started by the web server (nobody) there is no way for me to know who has the infinite loop. I wrote a cron police job that catches these runaway processes and shuts them down. The police job runs every minute but still doesn't catch everything in time. It makes the problem better, but the system still could require a couple of minutes to recover. So this solution is not optimal. Is there another way to fix this? How could I limit the amount of memory a single process started by nobody can use? Could webwork call every script with a memory usage limitation?

Nandor

<| Post or View Comments |>

userBob Byerly - Re: monitoring bad processes during development  blueArrow
6/17/2005; 3:45:12 PM (reads: 1375, responses: 0)
See the discussion in http://webhost.math.rochester.edu/webworkdocs/discuss/msgReader$3168 Bill Wheeler has some helpful remarks.

ulimit unfortunately does not seem to be fully functional on Linux systems, but you might want to take a look at the man page for the perl module Apache::Resource. (On our machine man Apache::Resource does it. You may have to install this module.) Based on this, we tried adding the lines 

PerlSetEnv PERL_RLIMIT_AS 100:120
PerlModule Apache::Resource
to our httpd.conf file. (Parameters need to be adjusted according to how much memory you have.)I think this helps; at least we haven't had any crashes clearly caused by memory leaks since we put it in. I'd be very interested in hearing others' experience with this.

Bob

<| Post or View Comments |>

userMichael Gage - Re: monitoring bad processes during development  blueArrow
6/22/2005; 10:39:52 PM (reads: 1342, responses: 0)
In addition there is a constant called TIMEOUT defined in WeBWorK::PG::Local which gives the maximum time allowed for rendering a given problem.

I've just reworked the code there a bit, so that the value of TIMEOUT can be defined in WeBWorK::Constants (if you download the very latest version of files in the CVS).

The original value of TIMEOUT was set to 5 minutes, but I've reset the default value to 60 seconds. If you don't set it at all you get only 10 seconds in order to remind you to set it. :-)

This may help with your problem pertaining to infinite loops in problems.

-- Mike

<| Post or View Comments |>