WeBWorK Main Forum

Library browser causes slow-down

Library browser causes slow-down

by Lars Jensen -
Number of replies: 16
Hi everyone,

It seems that working in the library browser for extended periods of time causes the library browser to slow down. Initially, as I build a set, the browser refreshes quickly as I change between libraries and add problems to a set. But after having added problems from several locations and refreshed several times, the library browser begins to slow down dramatically, and eventually becomes very slow, taking several minutes to refresh a small page.

What causes this behavior, and is there any remedy?

Thanks,
Lars.
In reply to Lars Jensen

Re: Library browser causes slow-down

by D. Brian Walton -
I just wanted to say that my school has been suspicious of this behavior for a few months, but we have not been able to validate the hypothesis yet. If we are seeing the same behavior, then the entire server slows down and not just the library browser. We see extended response times for all users of the server at those times, and it does not necessarily match when many students are active.

In fact, last summer we had only instructors working and saw the slowdowns. At the time we thought it was just a server issue and mostly resolved the problem by adjusting the number of available threads and increasing the amount of memory assigned to the virtual server.

This was going to be one of my summer activities, trying to replicate the issue and pin down the problem.

D. Brian Walton
James Madison University
In reply to D. Brian Walton

Re: Library browser causes slow-down

by Arnold Pizer -
Hi,

When investigating this, it would be good to report the OS name and version, the version of Perl and especially the version of MySQL you are using.

Thanks,

Arnie


In reply to Arnold Pizer

Re: Library browser causes slow-down

by Lars Jensen -
Hi Arnold,

Our version numbers are:

Ubuntu 10.04.3 LTS (lucid)
Perl v. 5.10.1
Mysql Ver 14.14 Distrib 5.1.41

Lars.
In reply to Lars Jensen

Re: Library browser causes slow-down

by Alexander Basyrov -
My two cents.

Our installation:
Debian Linux 6 (squeeze)
perl v5.10.1
mysql Ver 14.14 Distrib 5.1.61

On the installation that we use at UW-Stout, any display of problems through Library Browser causes apache process to increase in its size (e.g., RAM use). The more problems are displayed on one screen, the larger the increase in the size of the apache process.

If one views about 20 problems, the process size grows from about 65m to 110m (I am talking about 'resident' size, 'virtual' size is obviously larger).

If about 50 problems are viewed through the Library Browser, the process size grows to about 206m.

We had the situation when an instructor attempted to view about 500 problems at once, which made apache process to grow beyond 2g limit, so it was killed by kernel and generated a 'segfault' complaint from the kernel.

My first guess would be that repeated use of Library Browser causes a lot of apache processes to become really large, which consumes all of available RAM on the server; the server starts to use swap, and the intensive use of swap slows the server down.

We are using Apache::SizeLimit to terminate any 'huge' apache processes to avoid this type of situation.

Depending on the apache server configuration, it might be quite easy to replicate the problem: one would repeatedly view problems through Library Browser until most of apache processes consume all of available RAM.

In reply to Alexander Basyrov

Re: Library browser causes slow-down

by Michael Gage -
Thanks, Alex.  That makes some sense.  Have you or someone else had a chance to try this with the new Library Browser 3? (in the trunk version of webwork now)  I'm curious whether that also causes the child process to increase in size.
(I'm pretty sure that LibraryBrowser2 will still have the same bloating problem, but LibraryBrowser3 uses a different mechanism.)

We haven't observed this bloating systematically here, but we also restart the server every 24 hours just in case some process has grown large.

-- Mike
In reply to Michael Gage

Re: Library browser causes slow-down

by Alexander Basyrov -
Mike,

Yes, we also restart apache server every 24 hours -- just in case.

I did not have a chance to work with Library Browser 3, as we did not update webwork code for a while. If and when I have a chance to play with the new Browser, I'll try to report back on the memory usage.

-- Alex
In reply to Michael Gage

Re: Library browser causes slow-down

by Lars Jensen -
Hi Mike,

We're running the svn trunk version, but I only see Library browser and Library Browser 2 not Library Browser 3. How do I access Library browser 3?

Thanks,
Lars.
In reply to Lars Jensen

Re: Library browser causes slow-down

by Michael Gage -
You can get library browser 3 from the github.com/openwebwork repository.
I have this version working on test.webwork.maa.org but you will need to do a bit of work on the configuration files when you install it, so plan to spend a little time, and don't risk it on a machine where being down for a few hours would be a disaster.

You can use svn to get the latest version from github.com/openwebwork

For instructions of using svn with github go to 

http://webwork.maa.org/wiki/Github#Download_WeBWorK_via_svn_to_an_svn_repository

the instructions are pretty straightforward.


There is a README in the webwork2/conf directory for working with the new configuration files.  

It's best to get new versions of both the webwork2 repository and the pg repository from github.com/openwebwork.

Complete docs still need to be written which is part of why the newest version hasn't been pushed to the svn repository yet. (Exam season is the other reason :-) )


--Mike

In reply to Alexander Basyrov

Re: Library browser causes slow-down

by Jason Aubrey -
Hi Alex,

Could you post your apache config stanzas for Apache::SizeLimit. I think this is a good safeguard and others might like to implement it. In fact, maybe we should make it part of the standard installation instructions...

Thanks,
Jason
In reply to Jason Aubrey

Re: Library browser causes slow-down

by Lars Jensen -
Hi Alex and Jason,

When trying to install Apache2::SizeLimit (ver. 0.96) from CPAN, I get an error - something like "can't find apache include dir." Anyone knows how to fix this?

Lars.
In reply to Lars Jensen

Re: Library browser causes slow-down

by Jason Aubrey -
Hi Lars,

Looking at the installation documentation in the tar file for Apache2::SizeLimit, it looks like you have to pass it some info about apache:

$ tar zxvf Apache-SizeLimit-0.XX.tar.gz
$ cd Apache-SizeLimit-0.XX

$ perl Makefile.PL
For static mod_perl use -httpd /path/to/httpd
For dynamic mod_perl use -apxs /path/to/apxs
$ make
$ sudo make install

I don't know how to do that in cpan (it's probably easy), so I'd just suggest downloading the tarball and doing the steps above.

Hope this helps,
Jason
In reply to Jason Aubrey

Re: Library browser causes slow-down

by Danny Glin -
My experience is that if apxs is installed and is in the path, then the installers can usually find it.

In the Redhat world, this means installing the httpd-devel package.  I assume that there's an analogous apache libraries/development package for Ubuntu.

Danny
In reply to Jason Aubrey

Re: Library browser causes slow-down

by Alexander Basyrov -
I did not have to separately install Apache2::Size limit, it seemed to be already available as part of mod_perl installation.

Anyways, the following are the edits that I did to webwork.apache2-config file:

after the line with
my $webwork_dir = "whatever"
add
# size limiter for Apache2
use Apache2::SizeLimit;

then somewhere close to the end of the <Perl> section (I have it just above the </Perl> line), insert the actual configuration:

# sizes are in KB
$Apache2::SizeLimit::MAX_PROCESS_SIZE = 120000;
$Apache2::SizeLimit::MAX_UNSHARED_SIZE = 120000;
$Apache2::SizeLimit::CHECK_EVERY_N_REQUESTS = 20;
</Perl>


The maximum's are really what one needs to carefully check. Those are the 'virtual' process sizes, on the 64 bit installation they might be 4 times the real resident size.

Then, in the /etc/apache/apache2.conf file I added
PerlCleanupHandler Apache2::SizeLimit
to turn the use of Apache2::SizeLimit on.




In reply to Alexander Basyrov

Re: Library browser causes slow-down

by Lars Jensen -
Hi Alex,

Thanks! The SizeLimit module was indeed already on my system - it is in /usr/lib/perl5/Apache2/SizeLimit.pm so that was probably why I couldn't install it. I configured it according to your specifications, and restarted apache without errors.

Lars.
In reply to Lars Jensen

Re: Library browser causes slow-down

by Alexander Basyrov -
Hi Lars,

Great!

To see if the configuration really does what it is supposed to do, one can look through the error logs of apache for lines similar to 

[Wed Apr 11 14:15:21 2012] (19620) Apache2::SizeLimit httpd process too big, exiting at SIZE=315752/120000 KB SHARE=5856/0 KB UNSHARED=309896/120000 KB REQUESTS=61 LIFETIME=5166 seconds
If one is really curious, it should be possible to track the pages (through the access logs) that such killed processes had generated to see if there is a pattern.

There is also a program called monit, http://mmonit.com/monit/ that could be set up to monitor server resources in almost real time. It could be set up to send an email when available RAM get below certain percentage, etc. This could be a way to get more information about what is happening with your webwork server.

-- Alex