We had an our campus a Ransomware attack last Thursday and our IT people still have not got our WW back online yet. I am seeking advice on best practices to have a back up for remote sever as this is the week before finals week and is creating tons of stress on everyone.
https://webwork.maa.org/wiki/Backup_and_Disaster_Recovery describes one backup strategy that would allow you to (relatively) quickly rebuild your WeBWorK server somewhere else. That's probably a good place to start.
Thanks for this link. Our server is fine since it sits behind a firewall, our network people are very slow to getting all the ports set so students can login using authentication on campus but not off campus. Is there a way to make this process easier for them to get back up?
If the students can login on campus, but not off campus, then your system administrators have not opened up the ports for off campus access. The only thing that will fix that is your system administrators opening up the ports. I don't think there is anything else that can be done.
The Backup and Disaster Recovery page is a great resource--thanks.
I am hoping to save time in the event of a major event that would require a new server to be set up--- if I was interested in backing up the entire drive (/dev/sda), or at least all of the relevant folders that WW relies on, what folders should I be sure to include? I figure I would backup /opt/webwork as well as the SSL certificate folders/symlinks. (Certainly there are files/folders that are build-specific but I believe I would be able to restore an identical OS.)
I am hoping to save time in the event of a major event that would require a new server to be set up--- if I was interested in backing up the entire drive (/dev/sda), or at least all of the relevant folders that WW relies on, what folders should I be sure to include? I figure I would backup /opt/webwork as well as the SSL certificate folders/symlinks. (Certainly there are files/folders that are build-specific but I believe I would be able to restore an identical OS.)
With the attitude that you could spin up a new, independent WW server somewhere else, what you need to have is the courses folder (typically /opt/webwork/courses) and a backup of the database. Maybe also the webwork2/conf folder. Certainly backing up all of /opt/webwork is enough for the non-database components. And maybe you really do want all that if you have customized WeBWorK or PG in nontrivial ways.
What I meant to suggest is that I would be interested in a backup method that would allow me to extract the contents of a .tar.gz archive onto the root directory of the server and then with as few interactions as possible, have a working webserver once again. I realize this would require backing up operating system files that have to do with systemctl, etc. Is this even feasible or realistic? Were this to be feasible, I envision restoring the webserver system files first and the courses/student data second.
I recommend making a tar ball of the entire server file system. A good reference on this is https://help.ubuntu.com/community/BackupYourSystem/TAR.
In my experience making a tarball of an entire server while it is live has been unreliable. Because you are archiving a large number of files it takes a long time. Files are changing while the archive is being created, so there is a possibility of inconsistencies in the archive. I would particularly worry about things like the database not being properly preserved.
If you go this route my suggestion would be to do a full system backup once, and then back up the WeBWorK data regularly (nightly?). I would only update the system backup if system-level changes are made (e.g. package upgrades, software patches). This would mean a two-step restore (restore the OS, then restore the WW files), but it allows you to test your OS backup and make sure that it maintains integrity.
If your WeBWorK server is a virtual machine, then I recommend looking at VM snapshots. These have built-in protections to make sure that the snapshot can be restored to a fully functional VM.
It's also worth emphasizing that whenever possible a backup should be hosted off-site (subject to privacy rules). Many of the ransomware attacks I have heard of have taken down multiple services at the same institution, so backups living on another server in the same infrastructure are not necessarily safe.
The solution that has been easiest for us is to install WeBWorK on a VM. We're using the Proxmox hypervisor and there's a nightly backup process.
If something goes wrong, we just restore the VM image from backup and everything is back up and running in a couple of minutes.
(Full disclosure: I've got a colleague on the computer science side of the department who assists with this.)