Recently, we lost some of our problem sets the professors created and some student answers. I still have not figured out what went wrong.
We are running webwork on Ubuntu 14.04 LTS and the webwork version is 2.10. For mysql, the output of mysql -V is:
mysql Ver 14.14 Distrib 5.5.43, for debian-linux-gnu (x86_64) using readline 6.3
The backup I am running is to backup all the files in /opt/webwork and all the files in /var/lib/mysql ( and some other configuration files for apache2, mysql ...) to another system with NFS mount every four hours.
I had been running this kind of backup for a long time and could recover the webwork system on another system with the same version of mysql/apache2/webwork/OS without problem.
But recently, someone complaint that he could not log in. So I reboot the server. Then everything looked fine except that: The files in /var/lib/mysql were not updated since about eight days ago. Which means that I lost all the data for those eight days. I then went back to the backups and found that all those backups only backed up the same data again and again which is the data from eight days ago.
What puzzles me is that during that eight days period, everything looked fine from the webwork web interface. But the files for the database were not updated. Where were those data saved to during that time? (from now on, I think I will also use mysqldump to backup the databases.)
I googled around and not sure if this "hidden corruptions" problem is relevant: https://www.percona.com/blog/2006/06/17/using-myisam-in-production/.
What could be the problem? Can someone help?