** Using WeBWorK **

Brian Camp and I have set up a webwork server using Webwork 2.3 ( patched branch ). We use Apache 2.0.54, Perl 5.8.6, on Fedora Core 4.

We have been having issues with Apache and Memory.

Our server is dual processor, dual core Xeons, with 1GB RAM and 3.5GB swap.

On two occasions we have checked the server and found apache misbehaving.

Saturday, an instructor called saying WW was not working. We checked into it, and apache had used all available RAM, as well as all available swap. The professor had been using Course administration to select problems for a homework set. He had selected 'show all' from the drop down, so he was trying to view all 500 problems in the library. Apparently the server had not responded so he tried again and again until he contacted us. When we checked apache was using all available RAM as well as nearly all available swap. We restarted apache. And then I followed the advice in the below threads:

http://webhost.math.rochester.edu/webworkdocs/discuss/msgReader$3168#3189 http://webhost.math.rochester.edu/webworkdocs/discuss/msgReader$4481

I changed our httpd.conf to:

KeepAlive On

MaxKeepAliveRequests 10

KeepAliveTimeout 15

<IfModule prefork.c>
StartServers 8
MinSpareServers 5
MaxSpareServers 20
ServerLimit 70
MaxClients 160
MaxRequestsPerChild 100
</IfModule>

<IfModule worker.c>
StartServers 2
MaxClients 150
MinSpareThreads 25
MaxSpareThreads 75
ThreadsPerChild 25
MaxRequestsPerChild 100
</IfModule>

PerlSetEnv PERL_RLIMIT_AS 100:120
PerlModule Apache2::Resource

Looking through the apache error log, we were able to identify when the problem started, and from that point on the logs are littered with: [Sun Dec 03 18:39:01 2006] [info] (104)Connection reset by peer: core_output_filter: writing data to the network [Sun Dec 03 18:39:02 2006] [info] (32)Broken pipe: core_output_filter: writing data to the network [Sun Dec 03 18:39:02 2006] [info] server seems busy, (you may need to increase StartServers, or Min/MaxSpareServers), spawning 16 children, there are 0 idle, and 42 total children

We then checked the webwork logs and found one user had managed to submit an answer to a problem 405 times in about 2 minutes. We tried recreating the error by logging in and doing the same problem, with the answer they submited but never caused the server to go crazy.

Any idea what might be the problem? Any recommendations on apache configuration?

Thanks for your time,

James Elliott & Brian Camp

<| Post or View Comments |>

Timeout 1200
KeepAlive On
#MaxKeepAliveRequests 100
MaxKeepAliveRequests 50
KeepAliveTimeout 10
StartServers 5
MinSpareServers 5  
MaxSpareServers 10
#sam# Default is 150, but was previously set to 10. Trying 50 for now to
#sam# see if there'll be any problems.
#mike# there were some problems -- trying 25 clients -- (see Oct 22, 2006)-- some problems timed out
#Mike# still problems -- reducing it to 15
#mike# now to 10
MaxClients 10
MaxRequestsPerChild 300 

As you can see some experimentation is necessary. A webwork child takes about 50Meg of memory, and it appears that it works best if you keep the total amount of memory that can be claimed by the children to less than the physical RAM available.

Hope this helps.

<| Post or View Comments |>

Thanks for the quick response!

I had another apache question that I think is related to this. What sorts of modules should apache be loaded with? For example, have you tinkered with this on hosted or hosted2 to try and make your server more efficient for webwork?

Here are the modules that we currently are loading:

> LoadModule access_module modules/mod_access.so
> LoadModule auth_module modules/mod_auth.so
> LoadModule auth_anon_module modules/mod_auth_anon.so
> LoadModule auth_dbm_module modules/mod_auth_dbm.so
> LoadModule auth_digest_module modules/mod_auth_digest.so
> LoadModule ldap_module modules/mod_ldap.so
> LoadModule auth_ldap_module modules/mod_auth_ldap.so
> LoadModule include_module modules/mod_include.so
> LoadModule log_config_module modules/mod_log_config.so
> LoadModule logio_module modules/mod_logio.so
> LoadModule env_module modules/mod_env.so
> LoadModule mime_magic_module modules/mod_mime_magic.so
> LoadModule cern_meta_module modules/mod_cern_meta.so
> LoadModule expires_module modules/mod_expires.so
> LoadModule deflate_module modules/mod_deflate.so
> LoadModule headers_module modules/mod_headers.so
> LoadModule usertrack_module modules/mod_usertrack.so
> LoadModule setenvif_module modules/mod_setenvif.so
> LoadModule mime_module modules/mod_mime.so
> LoadModule dav_module modules/mod_dav.so
> LoadModule status_module modules/mod_status.so
> LoadModule autoindex_module modules/mod_autoindex.so
> LoadModule asis_module modules/mod_asis.so
> LoadModule info_module modules/mod_info.so
> LoadModule dav_fs_module modules/mod_dav_fs.so
> LoadModule vhost_alias_module modules/mod_vhost_alias.so
> LoadModule negotiation_module modules/mod_negotiation.so
> LoadModule dir_module modules/mod_dir.so
> LoadModule actions_module modules/mod_actions.so
> LoadModule speling_module modules/mod_speling.so
> LoadModule userdir_module modules/mod_userdir.so
> LoadModule alias_module modules/mod_alias.so
> LoadModule rewrite_module modules/mod_rewrite.so
> LoadModule proxy_module modules/mod_proxy.so
> LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
> LoadModule proxy_http_module modules/mod_proxy_http.so
> LoadModule proxy_connect_module modules/mod_proxy_connect.so
> LoadModule cache_module modules/mod_cache.so
> LoadModule suexec_module modules/mod_suexec.so
> LoadModule disk_cache_module modules/mod_disk_cache.so
> LoadModule file_cache_module modules/mod_file_cache.so
> LoadModule mem_cache_module modules/mod_mem_cache.so
> LoadModule cgi_module modules/mod_cgi.so

Initially we had not been storing images generated on the fly so that they would have to be regenerated. I think this may have led to our difficulty with the instructor browsing the problem library.

They had picked something like 518 problems they wanted to view. Images was their default display option. This by itself was not a problem but then they would then click on something else causing the page to want to redraw all 518 problems. Is this a bug in the problem library? It would seem that there might be a better way to select libraries and problem sets and so on without the display needing to refresh every time a button is clicked. On the other hand, I am trying to discourage the instructor from having to view such massive numbers of problems all at the same time :)

Thanks again for all of the help, Brian

<| Post or View Comments |>

Lars.

<| Post or View Comments |>

-- Mike

<| Post or View Comments |>

<| Post or View Comments |>

Just tried to change the "Assignment type" setting of a problem set, and it turns out that it can't be changed - the only setting that "sticks" is "homework." No matter what I choose from the drop-down menu, once I click the "Save changes" button, the setting returns to "homework."

Lars.

<| Post or View Comments |>

I took an existing homework course. Changed homework to gateway/quiz. Pressed "Save Changes" at the top of the page.

The return page had extra blanks to fill in regarding the parameters for the quiz.

Gavin LaRose is the principal author on this module, we might have to consult him.

<| Post or View Comments |>

That's what I did as well, except that the "gateway/quiz" setting switched back to "homework" instantly as I clicked "Save changes."

You are running development 2.2, though, right? I'm running plain 2.2.

Lars.

<| Post or View Comments |>

I've never seen this problem, but I'm running versions of WeBWorK that are well behind the curve. It sounds as if either the assignment type isn't being saved to the mySQL database, or the set detail page is automatically giving the default answer rather than what's in the database. Do you have access to the mySQL database? The easiest thing to do would be to check the actual assignment type in the database to see if it's correct.

<| Post or View Comments |>

Yes, I do have mysql access. And I found the setName_set table with table entry "assignment_type", and it does indeed say "gateway." However when I login to the course and look at the same setting after editing the course in the library browser, it is still listed as "homework" and none of the extra blanks that Mike refered to pops up.

Lars.

<| Post or View Comments |>

cvs status ProblemSetDetail.pm

will give you the version you are currently using.

cvs update -r 1.37 ProblemSetDetail.pm

will update just that one file.

Hope this helps.

-- Mike

<| Post or View Comments |>

Tried this - it didn't help...

Lars.

<| Post or View Comments |>

I'm not sure I completely understand what you're seeing here. I think you've described two apparently buggy behaviours: (1) when you go to the set detail page, select "gateway", and press "save", it returns with the set detail page but the assignment type reset to "homework" (even though the database indicates that the assignment type has been saved as "gateway"), and (2) after resetting the assignment type to "gateway", if you enter the Library browser the assignment type in the database gets reset as "homework". (Please let me know if that description isn't correct.)

I don't understand (1), because I've never seen that behavior (I've only seen the sequence of events that Mike indicated) and unfortunately don't have the time this morning to look at it to try and figure out what's going on. I'll be able to look at it next week, at which point I'll grab the latest version of WeBWorK out of the CVS and see if I can duplicate your problem. (2) might be more understandable; I haven't worked at all with the Library browser, so it may be that it creates/works with assignments without specifying an "assignment_type". I'd expect that this should result in the type not being changed, but I could possibly imagine some way that it might end up resetting the type unexpectedly.

Gavin

<| Post or View Comments |>

Your statement (1) above is exactly correct, and contains everything. I'm not sure that (2) adds anything.

So to summarize, here's what's going on:

(i) Enter the library browser

(ii) select a set from the "Select a Set from this Course" drop-down menu and click the "Edit Target Set" button

(iii) Select "gateway/quiz" from the "Assignment type" drop-down menu, and click "Save changes"

After carrying out (iii), (a) the web page instantly refreshes, listing the "Assignment type" as "homework" (on the drop-down menu), and (b) the mysql database correctly changes assignment_type to "gateway."

None of the "blanks" Mike mentions show up on the web page.

Hope this clarifies the behavior.

Lars.

<| Post or View Comments |>

Can you do the experiment you describe while watching the apache error log? I'm curious if any errors are being reported to the error log that aren't appearing on the screen. An error such as "such and such is not defined at line ..... of ....." would be very helpful.

-- Mike

<| Post or View Comments |>

<| Post or View Comments |>

Is there an "Assignment Type" selector on the "Library Browser" page? I can't see one in the version of WeBWorK (2.1 or so) that I'm running on my development server. (Sam or Mike, did you add one?) I just did the following to see if I could duplicate your problem:

• Entered the Library Browser,
• Created a new set, and then added problems to it. I don't have an "Assignment Type" option on the Library Browser page, so I then
• Entered Instructor Tools, selected the set, and clicked "Edit one set",
• Changed the assignment type to "gateway/quiz", saved and got the gateway parameters, which I then corrected, and finally
• Went back to the Library Browser and added a problem, saving the set afterwards.

Through all of this, once I'd set the assignment type to "gateway/quiz" it remained that way without changing. I'll have to get the latest version of WeBWorK to see if there's a difference in that.

Lars, you said you're running WeBWorK 2.2, right? Can you give me access to the course you're working with so that I can see the behavior first-hand?

Thanks,
Gavin

<| Post or View Comments |>

The plot thickens! I've tested the behavior you report on your server, and agree that the bug is clearly defined: changing the Assignment Type in the Set Detail page does not work. I can't look at the database to see what's going on there, but the Set Detail page doesn't show any change to the gateway/quiz assignment type.

So, I upgraded my development server to the latest version from the CVS, and I cannot duplicate the bug on my machine. So it's got to be something odd with either (a) the specific version of WeBWorK that you're running, assuming that it isn't the most current version from CVS, or (b) something with your set-up.

More anon,
Gavin

<| Post or View Comments |>

The problem appears to be a very odd database data error. After selecting a 'gateway' assignment type in the Set Detail page and saving the set, the assignment_type field in the mySQL database is correctly defined to be 'gateway'. However, when revisiting the Set Detail page, the data from the database isn't being correctly read into the set object: a dump of the fields and their values is as follows.

  version_last_attempt_time =>
  version_time_limit => 0
  versions_per_interval =>
  time_interval =>
  version_creation_time => gateway
  assignment_type =>
  attempts_per_version =>
  problem_randorder =>

Looking in mySQL, the assignment_type field is defined to be 'gateway', version_creation_time is '0', and version_time_limit is 'null'. So somehow these are getting scrambled in the course of reading the data into the set object that's being used by the Set Detail page (which is generated by ProblemSetDetail.pm).

I'm not sure why this is. If someone else has some insight, it's welcome. We'll follow up when we have more information. The Set.pm file in ...lib/WeBWorK/DB/Record/ is fine.

Gavin

<| Post or View Comments |>

This could be caused by the columns of the set table being ordered incorrectly. Lars, could you run the following SQL query:

show fields from coursename_set (replace coursename with the name of your course.)

You'd do this in the mysql console for the webwork database. Enter it with:

mysql -u webworkRead -p webwork

Thanks.
-sam

<| Post or View Comments |>

This is the result of the

show fields from test1_set

query. The name of the course is test1

Lars.

set_id  	text  	YES  	MUL  	NULL  	 
set_header 	text 	YES 	  	NULL 	 
hardcopy_header 	text 	YES 	  	NULL 	 
open_date 	text 	YES 	  	NULL 	 
due_date 	text 	YES 	  	NULL 	 
answer_date 	text 	YES 	  	NULL 	 
published 	text 	YES 	  	NULL 	 
version_last_attempt_time 	bigint(20) 	YES 	  	NULL 	
version_time_limit 	int(11) 	YES 	  	NULL 	 
versions_per_interval 	int(11) 	YES 	  	NULL 	 
time_interval 	int(11) 	YES 	  	NULL 	 
version_creation_time 	bigint(20) 	YES 	  	NULL 	 
assignment_type 	text 	YES 	  	NULL 	 
attempts_per_version 	int(11) 	YES 	  	NULL 	 
problem_randorder 	int(11) 	YES 	  	NULL

<| Post or View Comments |>

set_id                   
set_header               
hardcopy_header          
open_date                
due_date                 
answer_date              
published                
assignment_type          
attempts_per_version     
time_interval            
versions_per_interval    
version_time_limit       
version_creation_time    
problem_randorder        
version_last_attempt_time

Looks like yours is out of order for some reason. I'm not sure why this would have happened, but you can fix it by modifying the file lib/WeBWorK/DB/Schema/SQL.pm:

Replace each occurence of:

my $stmt = "SELECT * FROM `$sqlTable` $where";

(This line appears twice, once in the gets subroutine, and once in the getAll subroutine.)

With:

my @fieldnames = $self->sqlFieldnames();
my $fieldnames = join(", ", @fieldnames);
my $stmt = "SELECT $fieldnames FROM `$sqlTable` $where";

I haven't tested this yet, but it shouldn't do any damage. I'll test it tomorrow.
-sam

<| Post or View Comments |>

The above replacement resulted in this error when I tried to login to the course:

Warning messages

* DBD::mysql::st execute failed: Operand should contain 1 column(s) at /opt/webwork2/lib/WeBWorK/DB/Schema/SQL.pm line 229.

Error messages

DBD::mysql::st execute failed: Operand should contain 1 column(s) at /opt/webwork2/lib/WeBWorK/DB/Schema/SQL.pm line 229.

Call stack

The information below can help locate the source of the problem.

* in WeBWorK::DB::Schema::SQL::gets called at line 202 of /opt/webwork2/lib/WeBWorK/DB/Schema/SQL.pm

* in WeBWorK::DB::Schema::SQL::get called at line 1039 of /opt/webwork2/lib/WeBWorK/DB.pm

* in WeBWorK::DB::getUser called at line 179 of /opt/webwork2/lib/WeBWorK/Authen.pm

* in WeBWorK::Authen::verify called at line 227 of /opt/webwork2/lib/WeBWorK.pm

Lars.

<| Post or View Comments |>

Sorry about that. I should not have put parens around $fieldnames. I've fixed it above and tested the code this time.
-sam

<| Post or View Comments |>

This bug is fixed in lib/WeBWorK/DB/Schema/SQL.pm version 1.31 (for HEAD) and version 1.29.2.2 (for the rel-2-2-dev branch).

If you're running 2.2.x, you can update to the tip of the 2.2 dev branch as follows:

cvs update -r rel-2-2-dev

(Don't forget to restart the web server after you do this.)

HTH.
-sam

<| Post or View Comments |>

Thanks! The "Assignment type" now sticks and the extra blanks show up!

Is there (going to be) a rel-2-2-patches branch?

Lars.

<| Post or View Comments |>

If you would like to get in touch with us at the meetings before then, we will be checking the message boards. We hope to see lots of people there.

Jeff Holt and John Jones
Posted by Michael Gage on 1/9/06; 9:18:30 PM
from the dept.