Apache Memory Problems | topic started 12/3/2006; 9:46:01 PM last post 12/12/2006; 11:07:42 PM |
|
|
|
William Wheeler - Re: Apache Memory Problems 12/12/2006; 11:07:42 PM (reads: 315, responses: 0) |
Dear James and Brian, Re: >We then checked the webwork logs and found one user had managed to submit an answer to a problem 405 times in about 2 minutes. We tried recreating the error by logging in and doing the same problem, with the answer they submited but never caused the server to go crazy. I see this situation several times each semester. This is an inadvertent Denial of Service (DoS) attack caused by a "bug" in the operating system and browser combination on the computer the student is using. What seems to be the case is that the "Submit" button/mouse combination and/or the "Enter" key act as "repeating" keys, like the letter keys on the keyboard. (Hold down a letter key and watch how fast it repeats.) So if the student holds down the mouse button while the mouse pointer is on the Submit key or holds down the "Enter" key, then the browser will repeatedly submit the form as rapidly as possible. I've seen submission rates approaching 20 submissions per second. I've seen these attacks with both "GET" and "POST" requests. Each resubmission breaks the computer's network connection to the Apache server. That's the source of the "Connection reset" and "Broken pipe" messages. These attacks rapidly overwhelm Apache, because it has to assign each request to a new child or an old one that isn't otherwise assigned. (Note: When a connection is broken, it doesn't stop the WeBWorK process that is running; that process will run to completion, at which point the Apache child tries to send the WeBWorK output back to the student's computer; but the child discovers that the "pipe" is broken when it tries to write to the pipe.) Because the rate of submissions is faster than WeBWorK can process the submissions, WeBWorK falls behind and Apache has to create one new child after another. If the attack lasts several minutes, then Apache will overflow both RAM and Swap. The server will appear to be non-responsive, because it is spending almost all of its CPU cycles swapping. If left alone for a long time, the server will eventually recover on its own. But that may take hours. So the timely response is to stop and restart Apache. (This may take several minutes.) These inadvertent DoS attacks were a frequent problem with the old Windows Millenium version of Windows. I see the problem less frequently now. Last month I saw one attack from a Mac and one attack from a Windows PC. I've never been able to recreate this phenomenon on my workstations. But I've spoken with students who were sources of attacks. They usually describe the computer's screen as appearing to "shake rapidly". I usually caution the students to stop using the computers that generated the attacks. Sincerely, Bill Wheeler, Indiana University, Bloomington |