I got it working.
I added the real certificate to both the proxy and the webwork server.
webwork2.math.umb.edu is a cname that points to the proxy so I added a local webwork2.math.umb.edu on the webwork server in /etc/hosts so it can call itself without going through the proxy, using the routable address (not localhost, although that might work.)
I changed $server_root_url in site.conf to be https://webwork2.math.umb.edu
Voila
I added the real certificate to both the proxy and the webwork server.
webwork2.math.umb.edu is a cname that points to the proxy so I added a local webwork2.math.umb.edu on the webwork server in /etc/hosts so it can call itself without going through the proxy, using the routable address (not localhost, although that might work.)
I changed $server_root_url in site.conf to be https://webwork2.math.umb.edu
Voila