Hi Holly,
At the University of Michigan we're running similar numbers of students as are you; in the Fall we have one class of 1500, plus classes of 700 and 800 running simultaneously on our server. For reasons other than performance we have the course of 1500 split into sections, but all sections have the same deadlines. If the performance issue is the size of the mySQL tables, our experience won't be directly relevant to yours, but I would be surprised if that were the bottleneck. Sam or Mike may have insight on that particular issue, as they've spent more time looking at where the code slows down.
We run WeBWorK (and all of our other instructional technology applications) on a 4 dual core Xeon 2.8GHz system with 12GB memory, running RHEL. We've had no performance issues other than when we had MaxRequestsPerChild set too high in the apache configuration (which caused the apache child processes to quietly slurp up memory until they used up all 12GB, which resulted in everything grinding to a halt). This is 64bit, but I don't know if any specific work has been done to take full advantage of that.
My bet is that a server load of any Linux distribution will work fine to run WeBWorK. We've used RHEL and SUSE, as well as Solaris before we converted to Linux, all with no trouble.
Gavin