WeBWorK Main Forum

SSL with 2.18 and hypnotoad

SSL with 2.18 and hypnotoad

by Alex Jordan -
Number of replies: 30

I'm installing WeBWorK on an Ubuntu 22 server and I'm having trouble with SSL and serving via hypnotoad.

I installed certificates using certbot. They are at:
Certificate is saved at: /etc/letsencrypt/live/webwork.mysite/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/webwork.mysite/privkey.pem
where of course "webwork.mysite" is the actual domain.


In the mojolicious yml file, I have:

server_user: www-data
server_group: www-data

I don't know if I should have done this, but I set the certificate to be readable by all, and the key readable by owner and group, which are root:www-data.


For now since https is not working, I have
redirect_http_to_https: 0


I have:
hypnotoad:
  listen:
    - http://*:80
    - https://*:443?cert=/etc/letsencrypt/live/webwork.mysite/fullchain.pem&key=/etc/letsencrypt/live/webwork.mysite/privkey.pem

  ...

  #proxy: 1


I have commented out the tree lines from the service file:

#User=www-data
#Group=www-data
#Environment="MOJO_REVERSE_PROXY=1"

So I think I have done everything per the draft installation instructions. Now I can visit the site with a web browser using http://, but visiting with https:// gives an error in Firefox:
Secure Connection Failed
An error occurred during a connection to webwork.mysite. PR_END_OF_FILE_ERROR


Is it clear what I'm doing wrong, or what I could look into?

In reply to Alex Jordan

Re: SSL with 2.18 and hypnotoad

by Glenn Rice -

This is most likely a permissions issue.  The server runs as the www-data user.  So both of your certificate files need to be readable by the www-data user.  The ownership and permissions you gave should work.  However, are all of the directories leading up to the files readable by the www-data user also?

In reply to Glenn Rice

Re: SSL with 2.18 and hypnotoad

by Alex Jordan -
That was it. It was one of two things that I changed at the same time. The \etc\letsencrypt\live folder was readable by www-data, but not executable. And then the other thing was that the files /etc/letsencrypt/live/webwork.mysite/fullchain.pem and /etc/letsencrypt/live/webwork.mysite/privkey.pem are actually symlinks to where the real file is, which had a folder in its path neither readable nor executable by www-data, and I updated that. Now it works. Thanks!
In reply to Alex Jordan

Re: SSL with 2.18 and hypnotoad

by Wesley Burr -

Thanks for this post, and the note at the end, Alex. This was my issue as well - letsencrypt being a symlink file which wasn't permission'd properly. I've added a short note to the wiki for the manual install about this for future us-types.

In reply to Wesley Burr

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -

This was helpful, as I've just moved our server over to letsencrypt as well. I had to chown both the live and archive folders, and chmod to make them executable.

I am also curious how you ran CertBot: I decided to go with the 'certonly --standalone' option. This is OK, but you can't have hypnotoad listening on 80 any more, or else the renewal dry run fails.

We've always had our servers listen on both 80 and 443, and then redirect any traffic over 80 to 443.

Are you both set up to listen on 443 only, or did you find a way to let certbot run and still have hypnotoad listen on port 80?

In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Glenn Rice -

That is interesting.  I proxy hypnotoad via apache2 on my servers (since I have other sites that also run on the same server), and I have port 80 redirected to port 443 as well.  It works fine with that.  I am not sure why it is an issue when serving directly with hypnotoad.

For a workaround you could just temporarily disable forwarding of port 80 to port 443 when you update the certificate.

In reply to Glenn Rice

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -
Right. But the appeal of using CertBot is that it automates the renewal process. To keep listening/forwarding port 80 I would have to manually reconfigure hypnotoad to stop listening on port 80, let CertBot run, and then manually re-reconfigure.
And letsencrypt certificates renew every 3 months, I think.
In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Glenn Rice -

Yeah, I know it isn't convenient.

In reply to Glenn Rice

Re: SSL with 2.18 and hypnotoad

by Glenn Rice -
By the way, things were much worse with previous versions of WeBWorK 2 with certbot. I found that I had to complete remove webwork2 from the apache2 configuration to get certbot to run.
In reply to Glenn Rice

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -

My production server is still on 2.17, running Apache. I use CertBot there with the webroot option. (It took me a bit to figure out what the right for is.)

You put a file in the root folder (I think this is /var/www/html for me) and CertBot uses that for the challenge process.

I set it once and it seems to be chugging along just fine. I should probably do the same on 2.18.

In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Glenn Rice -

Unfortunately, I don't think that the webroot option will work for 2.18 when serving directly with hypnotoad.

In order for the webroot option to work, you need to pass certbot the directory on the server that is served as the html root.  Then certbot creates a .well-known subdirectory there, and a temporary file in that subdirectory.  Then the Let’s Encrypt validation server makes a request to your server to get that file.

If you are serving directly with hypnotoad then no matter what directory you pass as webroot, the Let's Encrypt validation server request for the above file will fail, because the webwork2 app does not serve that file.

In reply to Glenn Rice

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -

Oh! Thanks for saving me the trouble of discovering that the hard way.

So it sounds like my options are to either proxy by Apache and use webroot, or figure out a way to script the renewal process with hypnotoad.

(Or just have hypnotoad listen only on 443. Is there anything inherently bad about doing it that way?)

Maybe Alex Jordan found a better solution. I'll ask the next time I talk to him.

In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Glenn Rice -

If you are using apache2 with webwork 2.18, there is no need to use webroot.  You can use certbot's apache2 method.  Webwork 2.18 does not cause the issue that previous versions of webwork did.  The issue was mod_perl2.

There is nothing wrong with only listening on port 443.  It is a convenience for your users to redirect port 80 to port 443, but not necessary.  However, It can be a bit confusing for your users, if they enter a url and get taken to the unsecured port 80 and get the site is not found.

In reply to Glenn Rice

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -
Ah, right. I remember having that annoyance with mod_perl2.

I think I'll leave things as is. My 2.18 server exists mostly as a PreTeXt backend, and for a handful of people who want playgrounds to try writing new questions. Traffic for the former is all over https, and the latter consist of people who should know to use https.

Once I move my production server to 2.18 I'm also not concerned, since 99% of the traffic is students who can only get in via LTI.
In reply to Glenn Rice

Re: SSL with 2.18 and hypnotoad

by Alex Jordan -

What I found is that if I am serving directly with hypnotoad, it has occupied (in some sense) port 80, and apache won't run simultaneously because of the port being in use. certbot attempts to fire up apache to do its thing, but it can't. So to use certbot, I do something like:

systemctl stop webwork2
systemctl start apache2 (or httpd)
certbot renew
systemctl stop apache2 (or httpd)
systemctl start webwork2

with a brief outage of service for webwork2. And this works. (And it's a separate issue, but the new certs may end up in file trees that are not readable by the www-data user (or whichever user hypnotoad is run by) and a follow-up step is to change ownership of some intermediate folders to those certs, and check on their permissions).

In reply to Alex Jordan

Re: SSL with 2.18 and hypnotoad

by Arnold Pizer -

There is some documentation on this at https://webwork.maa.org/wiki/WeBWorK_2.18_Ubuntu_Server_22.04_LTS_Amazon_Machine_Image#Renewing_the_certificate

In reply to Arnold Pizer

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -
I've hit another problem with Hypnotoad, SSL, and CertBot: CertBot ran its auto-renewal on Monday, and that took my server down!
The reason is that the new certicates get saved in /etc/letsencrypt/archive/my-site/ and they don't get saved with the same permissions as the old ones.

So I hit similar permissions issues to the first time around.

In my case: the old certificates were owned by www-data. The new ones are initially owned by root. I need to chown to www-data before my site will load.
In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -
What's weird is that the symlinks in /etc/letsencrypt/live that point to the actual files in the archive folder are all owned by root.
In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Alex Jordan -

Check the permissions/owner on every intermediate folder in the symlinks that points to the certs. That was the issue for me with this. Twice now. 

In reply to Alex Jordan

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -
The symlinks look like -> ../../archive/site-name/filename

Both live/ and archive/ are owned by www-data
live/site-name/ is owned by www-data
archive/site-name/ is owned by www-data

But the symlinks are owned by root, and the newly created files are owned by root.

Should I make the symlinks owned by www-data, or make all the folders owned by root?
In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Alex Jordan -
Probably root should own them, but there should be a group that includes the hypnotoad user, and that group has read access to these folders and everything inside.



In reply to Alex Jordan

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -
To the best of my knowledge there is no hypnotoad user (at least not by that name). I think hypnotoad (that is, the webwork2 service) runs under the www-data user and group. Right now that is what owns all the /etc/letsencrypt folders.
In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Alex Jordan -

Right, by "hypnotoad user" I did not mean a user named hypnotoad. I mean the user that the webwork2 service is running under. Probably www-data in this case.

So you no longer have an issue? If you do still, did you check permissions on each and every intermediate directory?

In reply to Alex Jordan

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -

Maybe not? I mean, I guess I could force an early renewal to see what happens, but otherwise, I won't know until the automatic renewal runs again in a few months.

In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -

Certificate renewal took down my server again.

New certificates are installed in the archive folder with root as the owner, and I have to chown to the www-data user to get the site back up.

Have you had any luck with automatic renewal? Or are you just setting yourself a reminder to do it manually every 3 months?

In reply to Arnold Pizer

Re: SSL with 2.18 and hypnotoad

by Sean Fitzpatrick -

Arnold, I think these instructions need to be updated to make it clear that there's an issue with automatic renewal: the renewed certificates are owned by root, and can't be accessed by www-data. Unless there's a better way to configure things to avoid this issue, the instructions are going to need to explain how to update ownership each time (and perhaps run everything as a scheduled process for those, like me, who will forget to do it manually).

In reply to Sean Fitzpatrick

Re: SSL with 2.18 and hypnotoad

by Alex Jordan -
Folders in /etc/letsencrypt/ are owned by root, with permissions 700. At least that is how certbot makes them.

I think that when apache is spinning up and loading its config, it's acting as root at that early stage, and so it can see the cert files down inside the appropriate folder with 700 permission. Later apache drops its root access and the www-data or apache user is running apache or httpd, but it already has the certificate files to work with. So while the www-data or apache user cannot see into those 700 files, it doesn't matter anymore.

By contrast, I think hypnotoad is not running as root at any time, or at least not when it wants to access the certificate files. So it cannot get to them. Currently we have to adjust permissions/ownership so that whoever the hypnotoad user is can get to them. It's just a bother when certbot renews, and resets the permissions/ownership.

Is there anything we can do with the mojolicious configuration to briefly give it root access to read certificate files the same way apache/httpd works?
In reply to Alex Jordan

Re: SSL with 2.18 and hypnotoad

by Glenn Rice -
You are correct about apache2 starting as root (binding to port 80 or 443 or both) and reading the certificates before it switches to the www-data user.

For webwork2, hypnotoad does start as root also so that it can bind to port 80 or 443 or both. However, unlike apache2, it doesn't read the certificates until after the Mojolicious::Plugin::SetUserGroup has switched the user and group to www-data. I don't think there is any way to change it so that the certificate files can be read before that happens.

I have been thinking about implementing optional endpoints in the webwork2 app that would work for renewing letsencrypt certificates with the certbot webroot method. I think it is possible to do that. Then automatic renewal would work without needing to stop the webwork2 app, and since the certificates renewal is done by the webwork2 app the permissions of the certificate files would not change.
In reply to Glenn Rice

Re: SSL with 2.18 and hypnotoad

by Glenn Rice -
I added pull request https://github.com/openwebwork/webwork2/pull/2321 for this. I was mistaken on the file permissions. It should have been obvious that the webwork2 app would not do the certificate renewal (certbot does). So adding the certbot renewal routes isn't quite enough, since certbot will create the new certificates with root permissions. Fortunately, certbot has a --post-hook option that makes it possible to automate fixing permissions after the certificates are renewed. The webwork2 app also has to be reloaded in the post hook so that it reads and starts using the new certificates.

In any case, this is all documented in the pull request, and provides an automatic certificate renewal option for those serving webwork2 directly via hypnotoad.