Hi all. I've just setup a new webwork install and am running into the occasional error I can't explain. Looks like the connection to apache dies mid upload severing the connection.
Here is what the client sees:
WeBWorK error
An error occured while processing your request. For help, please send
mail to this site's webmaster (webmaster@localhost), including all of
the following information as well as what what you were doing when the
error occured.
Wed Jan 14 17:32:01 2009
Warning messages
Error messages
Internal apreq error
Call stack
The information below can help locate the source of the problem.
in Apache2::Request::upload called at line 221 of
/usr/local/webwork2/lib/WeBWorK.pm
Request information
Method POST
URI /webwork2/JSwift_137/01_Diff_Review/4/
HTTP Headers
User-Agent Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_6;
en-us) AppleWebKit/525.27.1 (KHTML, like Gecko) Version/3.2.1
Safari/525.27.1
Accept text/xml,application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Connection keep-alive
Referer http://webwork.math.nau.edu/webwork2/JSwift_137/01_Diff_Review/4/?effectiveUser=aea74&displayMode=jsMath&user=aea74&key=zDHya5EJtU3Ax2BPTakzATeqIVlxXI5L
Content-Length 2015
Accept-Encoding gzip, deflate
Content-Type multipart/form-data;
boundary=----WebKitFormBoundaryPprMHPUrdg9j0Aen
Accept-Language en-us
Cookie jsMath=font%3Asymbol%2Cwarn%3A0
Host webwork.math.nau.edu
And here is what I have in the apache log:
[Wed Jan 14 17:14:45 2009] [error] [client 24.156.36.114] (104)Connection reset by peer: ap_get_brigade failed during prefetch, referer: http://balance-webwork.math.nau.edu/webwork2/JSwift_137/02_Int_Review/13/
[Wed Jan 14 17:14:45 2009] [error] [client 24.156.36.114] [/webwork2/JSwift_137/02_Int_Review/13/] Internal apreq error\n * in Apache2::Request::upload called at line 221 of /usr/local/webwork2/lib/WeBWorK.pm, referer: http://balance-webwork.math.nau.edu/webwork2/JSwift_137/02_Int_Review/13/
[Wed Jan 14 17:29:02 2009] [error] [client 134.114.9.95] (104)Connection reset by peer: ap_get_brigade failed during prefetch, referer: http://webwork.math.nau.edu/webwork2/JSwift_137/01_Diff_Review/4/?user=hsm9&effectiveUser=hsm9&key=9I8Piv99rhNgu9pnxBFuqdCSMpxTMEva&displayMode=jsMath&showOldAnswers=1
[Wed Jan 14 17:29:02 2009] [error] [client 134.114.9.95] [/webwork2/JSwift_137/01_Diff_Review/4/] Internal apreq error\n * in Apache2::Request::upload called at line 221 of /usr/local/webwork2/lib/WeBWorK.pm, referer: http://webwork.math.nau.edu/webwork2/JSwift_137/01_Diff_Review/4/?user=hsm9&effectiveUser=hsm9&key=9I8Piv99rhNgu9pnxBFuqdCSMpxTMEva&displayMode=jsMath&showOldAnswers=1
[Wed Jan 14 17:43:17 2009] [error] [client 97.124.38.218] (70007)The timeout specified has expired: ap_get_brigade failed during prefetch, referer: http://webwork.math.nau.edu/webwork2/JSwift_137/feedback/
Does anyone have any ideas why the timeouts would be expiring? The old webwork server is still up and functioning and does not seem to have this problem.
Thanks
Thinking that this might have something to do with the round robin nature of hitting two different machines (with shared class space and where both are hitting the same mysql server) I added the persistent and quiescent options to our ldirectord.cf.
virtual=webwork.math:80
real=server1:80 ipip
real=server2:80 ipip
service=http
request="index.html"
receive="Northern Arizona University Department of Mathematics and Statistics"
scheduler=rr
checktype=negotiate
persistent=6000
quiescent=no
One of our professors is still getting reports from students about problems. Reports are intermittent about webwork hanging when going through the loadbalancer, but fine when accessing an individual machine. Like I said, I just addded the persistent line, but is anyone else running a simular setup or have any advice?
Thanks
virtual=webwork.math:80
real=server1:80 ipip
real=server2:80 ipip
service=http
request="index.html"
receive="Northern Arizona University Department of Mathematics and Statistics"
scheduler=rr
checktype=negotiate
persistent=6000
quiescent=no
One of our professors is still getting reports from students about problems. Reports are intermittent about webwork hanging when going through the loadbalancer, but fine when accessing an individual machine. Like I said, I just addded the persistent line, but is anyone else running a simular setup or have any advice?
Thanks
I've been seeing the same apreq error in our log files, but have yet to hear a student complain about it, so I don't have a good indication of what was happening when it occurred, and what the net result was.
We are also using a load-balanced setup with a shared disk space for the Webwork file system. Have you seen any more of these errors since you made your connections persistent in the load balancer?
We are also using a load-balanced setup with a shared disk space for the Webwork file system. Have you seen any more of these errors since you made your connections persistent in the load balancer?
I've never been able to get the apreq errors, but I've not received any more reports about it so it looks like that problem might have been solved. But this brings me to an even more interesting problem.
A professor reported that when checking a problem, the webwork servers would hang when accessed via the loadbalancer, but not when accessed individually. So accessing webwork.math (loadbalancer) hangs, while performing the same operation on bakura.math or kuat.math (individual machines) does not.
After getting this report we of course tried to replicate the problem, but had no luck. On campus the loadbalancer does not hang. But when I went home, I noticied the same operation that worked on campus through the loadbalancer, does hang firefox3 on xp. To make things even stranger, google chrome, and ie6 do no hang when going through the loadbalaner or either of the individual machines.
So at this point I'm pretty confused. On campus firefox3 works fine. Off campus it hangs, but only when going through the loadbalancer, and other browsers are unaffected.
If anyone has any ideas at all, I'm all ears.
A professor reported that when checking a problem, the webwork servers would hang when accessed via the loadbalancer, but not when accessed individually. So accessing webwork.math (loadbalancer) hangs, while performing the same operation on bakura.math or kuat.math (individual machines) does not.
After getting this report we of course tried to replicate the problem, but had no luck. On campus the loadbalancer does not hang. But when I went home, I noticied the same operation that worked on campus through the loadbalancer, does hang firefox3 on xp. To make things even stranger, google chrome, and ie6 do no hang when going through the loadbalaner or either of the individual machines.
So at this point I'm pretty confused. On campus firefox3 works fine. Off campus it hangs, but only when going through the loadbalancer, and other browsers are unaffected.
If anyone has any ideas at all, I'm all ears.
I see this in the logs too (have not seen any student complaints yet). There is no load balancing in our setup - just a single server running under CentOS 5 with Apache 5. I upgraded libapreq2 to version 2.12(for an unrelated reason), but still see the error messages.