There is some documentation on this at https://webwork.maa.org/wiki/WeBWorK_2.18_Ubuntu_Server_22.04_LTS_Amazon_Machine_Image#Renewing_the_certificate
I've hit another problem with Hypnotoad, SSL, and CertBot: CertBot ran its auto-renewal on Monday, and that took my server down!
The reason is that the new certicates get saved in /etc/letsencrypt/archive/my-site/ and they don't get saved with the same permissions as the old ones.
So I hit similar permissions issues to the first time around.
In my case: the old certificates were owned by www-data. The new ones are initially owned by root. I need to chown to www-data before my site will load.
The reason is that the new certicates get saved in /etc/letsencrypt/archive/my-site/ and they don't get saved with the same permissions as the old ones.
So I hit similar permissions issues to the first time around.
In my case: the old certificates were owned by www-data. The new ones are initially owned by root. I need to chown to www-data before my site will load.
What's weird is that the symlinks in /etc/letsencrypt/live that point to the actual files in the archive folder are all owned by root.
Check the permissions/owner on every intermediate folder in the symlinks that points to the certs. That was the issue for me with this. Twice now.
The symlinks look like -> ../../archive/site-name/filename
Both live/ and archive/ are owned by www-data
live/site-name/ is owned by www-data
archive/site-name/ is owned by www-data
But the symlinks are owned by root, and the newly created files are owned by root.
Should I make the symlinks owned by www-data, or make all the folders owned by root?
Both live/ and archive/ are owned by www-data
live/site-name/ is owned by www-data
archive/site-name/ is owned by www-data
But the symlinks are owned by root, and the newly created files are owned by root.
Should I make the symlinks owned by www-data, or make all the folders owned by root?
Probably root should own them, but there should be a group that includes the hypnotoad user, and that group has read access to these folders and everything inside.
To the best of my knowledge there is no hypnotoad user (at least not by that name). I think hypnotoad (that is, the webwork2 service) runs under the www-data user and group. Right now that is what owns all the /etc/letsencrypt folders.
Right, by "hypnotoad user" I did not mean a user named hypnotoad. I mean the user that the webwork2 service is running under. Probably www-data in this case.
So you no longer have an issue? If you do still, did you check permissions on each and every intermediate directory?
Maybe not? I mean, I guess I could force an early renewal to see what happens, but otherwise, I won't know until the automatic renewal runs again in a few months.
Certificate renewal took down my server again.
New certificates are installed in the archive folder with root as the owner, and I have to chown to the www-data user to get the site back up.
Have you had any luck with automatic renewal? Or are you just setting yourself a reminder to do it manually every 3 months?
Arnold, I think these instructions need to be updated to make it clear that there's an issue with automatic renewal: the renewed certificates are owned by root, and can't be accessed by www-data. Unless there's a better way to configure things to avoid this issue, the instructions are going to need to explain how to update ownership each time (and perhaps run everything as a scheduled process for those, like me, who will forget to do it manually).
Folders in /etc/letsencrypt/ are owned by root, with permissions 700. At least that is how certbot makes them.
I think that when apache is spinning up and loading its config, it's acting as root at that early stage, and so it can see the cert files down inside the appropriate folder with 700 permission. Later apache drops its root access and the www-data or apache user is running apache or httpd, but it already has the certificate files to work with. So while the www-data or apache user cannot see into those 700 files, it doesn't matter anymore.
By contrast, I think hypnotoad is not running as root at any time, or at least not when it wants to access the certificate files. So it cannot get to them. Currently we have to adjust permissions/ownership so that whoever the hypnotoad user is can get to them. It's just a bother when certbot renews, and resets the permissions/ownership.
Is there anything we can do with the mojolicious configuration to briefly give it root access to read certificate files the same way apache/httpd works?
You are correct about apache2 starting as root (binding to port 80 or 443 or both) and reading the certificates before it switches to the www-data user.
For webwork2, hypnotoad does start as root also so that it can bind to port 80 or 443 or both. However, unlike apache2, it doesn't read the certificates until after the Mojolicious::Plugin::SetUserGroup has switched the user and group to www-data. I don't think there is any way to change it so that the certificate files can be read before that happens.
I have been thinking about implementing optional endpoints in the webwork2 app that would work for renewing letsencrypt certificates with the certbot webroot method. I think it is possible to do that. Then automatic renewal would work without needing to stop the webwork2 app, and since the certificates renewal is done by the webwork2 app the permissions of the certificate files would not change.
For webwork2, hypnotoad does start as root also so that it can bind to port 80 or 443 or both. However, unlike apache2, it doesn't read the certificates until after the Mojolicious::Plugin::SetUserGroup has switched the user and group to www-data. I don't think there is any way to change it so that the certificate files can be read before that happens.
I have been thinking about implementing optional endpoints in the webwork2 app that would work for renewing letsencrypt certificates with the certbot webroot method. I think it is possible to do that. Then automatic renewal would work without needing to stop the webwork2 app, and since the certificates renewal is done by the webwork2 app the permissions of the certificate files would not change.
I added pull request https://github.com/openwebwork/webwork2/pull/2321 for this. I was mistaken on the file permissions. It should have been obvious that the webwork2 app would not do the certificate renewal (certbot does). So adding the certbot renewal routes isn't quite enough, since certbot will create the new certificates with root permissions. Fortunately, certbot has a --post-hook option that makes it possible to automate fixing permissions after the certificates are renewed. The webwork2 app also has to be reloaded in the post hook so that it reads and starts using the new certificates.
In any case, this is all documented in the pull request, and provides an automatic certificate renewal option for those serving webwork2 directly via hypnotoad.
In any case, this is all documented in the pull request, and provides an automatic certificate renewal option for those serving webwork2 directly via hypnotoad.
This looks like exactly the thing, thanks! I'll add any further comments over on GitHub.