could be used with WeBWorK to serve the moderate-size audience you describe (and to supplement the existing Preview). DragMath is an open-source Java applet which provides a "drag and drop" expression editor.
It is not clear whether DragMath could be enabled at the level of an individual course (e.g., via its PGcourse.pl file) or if enabling it would need to be system-wide.
It may be worth considering whether WeBWorK might benefit by emulating STACK (http://www.stack.bham.ac.uk/) by using a two-stage process for processing an answer:
1) answer is checked for valid syntax (and displayed as 2D math)
2) correctness is checked and feedback is provided