Installation

Answer encryption problem

Answer encryption problem

by Steve Plemmons -
Number of replies: 13

 

We have been running webwork for a fairly large installation on 32-bit ubuntu. The number of users has gotten large enough that occasionally we are running out of memory that is limited by the 32-bit operating system. To fix this, I installed the 64-bit version from LiveDVD, archived the courses, moved them to the new 64-bit version, then unarchived them there.

Everything worked like a charm except for the fact that students cannot view their answers that were submitted on the old system becuase they are base64_encoded in the database. The student gets a pink screen with the following error message:

Warning -- there may be something wrong with this question. Please inform your instructor including the warning messages below.

WeBWorK Warnings

WeBWorK has encountered warnings while processing your request. If this occured when viewing a problem, it was likely caused by an error or ambiguity in that problem. Otherwise, it may indicate a problem with the WeBWorK system itself. If you are a student, report these warnings to your professor to have them corrected. If you are a professor, please consult the warning output below for more information.

Warning messages

  • problem fetching answers -- possibly left over from base64 days. Not to worry -- press preview or submit and this will go away permanently for this question. Storable binary image v49.97 more recent than I am (v2.7) at ../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al) line 415, at /opt/webwork/webwork2/lib/WeBWorK/Utils.pm line 837

So I created a new user on the new system and answered the same question. The answer is indeed in clear text in the database and not base64_encoded.

This is no problem at the end of the semester since we will be starting new courses and the system will work fine, but since we are trying to do this in the middle of the semester to aleviate poor server performance when we reach the memory limit, this becomes a problem. The pink screen with the error message will be trouble enough, not to mention the fact that the student will not be able to view their old answers when using the system to study, etc.

Any ideas on a work around for this?

Thanks,

Steve Plemmons

In reply to Steve Plemmons

Re: Answer encryption problem

by Michael Gage -
If the student submits their answer (any answers) this message will disappear and not reappear again. Even without this the student can see their old answers, but the annoying warning will remain until the data has been re-saved in the new format.

It's not the change from 32 bit to 64 bit that is involved, but an internal change from a homegrown method of storing past answers to a CPAN "storable" solution made in changing from 2.4.7 to 2.4.9

The message (as it says) is merely an alert that there was an automatic change in how data was stored so that if there were discrepancies in data instructors, users and developers would have some idea of where to look for the problem.

In fact the change over seems to have been remarkably smooth and no known problems with the automatic conversion of the database have been reported.

If you wish you can comment out the warning line around line 839 in
Utils.pm -- I think that will allow your students to view the old answers with out the warning sign. You'll want to do some testing however since I can't guarantee exactly what will happen.

-- Mike

In reply to Michael Gage

Re: Answer encryption problem

by Nandor Sieben -
I am getting the same error message even though I have the exact same webwork version installed on two different servers running different Linux distributions.

I used rsync to copy /opt/webwork/courses from server 1 to server 2.
Then I used mysqldump to dump the webwork database on server 1 which I restored at server 2. This seems to be working except the error message I get
when I try to work on a problem on server 2. What could be the problem? Do I
need copy other directories from server 1 to server 2?

Error message:
problem fetching answers -- possibly left over from base64 days. Not to worry -- press preview or submit and this will go away permanently for this question. Byte order is not compatible at ../../lib/Storable.pm (autosplit into ../../lib/auto/Storable/thaw.al) line 415, at /opt/webwork/webwork2/lib/WeBWorK/Utils.pm line 837

The dump command used on server 1:
mysqldump --add-drop-database -u webworkWrite -psecret -f -v -q --opt -B webwork | gzip -9 > data.mysql.gz

Restore on server 2:
gunzip  < data.mysql.gz | mysql -u webworkWrite -psecret webwork

On both servers:
Webwork Version 2.5.0
At revision 7146.

server 1: 
redhat
/proc/cpuinfo:
  cpu family      : 15 
  model           : 6
  model name      : Intel(R) Xeon(TM) CPU 3.00GHz
uname -m: x86_64
mysql -v: Server version 5.1.61 Source distribution

server 2: 
Ubuntu
/proc/cpuinfo:
  cpu family      : 6
  model           : 23
  model name      : Intel(R) Core(TM)2 Duo CPU     E8500  @ 3.16GHz
uname -m:  i686
mysql -v: Server version: 5.1.63-0ubuntu0.10.04.1 (Ubuntu)
 
In reply to Nandor Sieben

Re: Answer encryption problem

by Michael Gage -
Hi Nandor,

This is a warning message, not a fatal error, so if you submit this problem you won't see it again.

The warning message was put in as we switched the storage of answers and other data from the adhoc ascii code that we used with GDBM in WeBWorK 1.9 and for a number of years in WeBWorK2.  We switched to using one of the 
packages (Storable I think) that safely encodes object data for storage in ascii 
format.

The message was put in because if something is being done to upgrade the data for the course it's best to know about it in case something goes wrong. (So far there have been no errors reported -- so perhaps we can dispense with the warning message.)

I'm not sure why you are seeing this message on server2 but not on server1.  I would expect you to see that error message whenever you look at an old course for the first time.  I would not expect you to see it on a new course
or after an answer has been submitted and recorded -- and the data storage upgraded to the new format.

If you are seeing this only on the new server then perhaps the warning message is serving a purpose and there is something about the msyqldump command that is messing up Storable's encryption of the data.

Hope this info helps.

-- Mike
In reply to Michael Gage

Re: Answer encryption problem

by Nandor Sieben -
Mike,

I experience scoring problems as well so I think there is a more serious issue.
There is no upgrade going on on either of the machines. They have the exact same version of webwork. Both are fresh installations. There is no old student data. I try to explain in more details what I did. 

1. create a new class with a new set on server 1. 
2. partially solve the first problems on server 1.
3. rscync the courses directory from server 1 to server 2.
4. dump the database data on server 1.
5. restore the database data to server 2.
6. server 2 has the course, the set and the partial score for problem 1. So far so good.
7. When I try to work on the first problem on server 2, I get the warning message.
8. When I ignore the warning message on server 2 and solve the problem again, the score is not set correctly on server 2. 
In reply to Michael Gage

Re: Answer encryption problem

by Nandor Sieben -
Could it be that the storable package creates encryption that is not transferable
between different versions of Linux (32 vs. 64 bit)?

The documentation of the storable package: 
http://search.cpan.org/~ams/Storable-2.30/Storable.pm
mentions "network order". I do not fully understand this but it seems that
network order is needed for compatibility. Is network order used by webwork?

The "64 bit data in perl 5.6.0 and 5.6.1" section of the documentation also could be relevant.

How can I check the version number of the storable package on my two servers?

In reply to Nandor Sieben

Re: Answer encryption problem

by Arnold Pizer -
Hi Nandor,

This isn't an answer to your question, but as a test what what happens if you archive a course on the "old" server and unarchive it on the new one? If that works, you could use that method to transfer courses.

Arnie
In reply to Arnold Pizer

Re: Answer encryption problem

by Nandor Sieben -
Arnie, I did not try that. Am I right to assume that I need to do that class by class?

The reason I am doing this is to have a backup server that can take over any time the main server is down. Replicating the whole database seems like the easiest way to do this because this transfers everything in a single simple step.
I could do this replication several times a day in a cron job.

In reply to Arnold Pizer

Re: Answer encryption problem

by Nandor Sieben -
Arnie,

I tried your suggestion. I archived the course on server 1 and unarchived on server 2. The result is exactly same as I got with mysqldump. I do see the course and it is seemingly good, but I do get the same warning message when I try to work on a problem. 

This shows that the problem is not mysqldump. The archive procedure produces the same error.

I think it is a compatibility issue with the storable library. I think the first thing to try is to use the compatibility library with "network order" if it is not used that way already. 
In reply to Nandor Sieben

Re: Answer encryption problem

by Michael Gage -
Hi Nandor,

This is a possibility.  I'm not in Rochester at the moment so I can't easily check in to it.  WeBWorK just uses Storable straight out of the box -- we didn't do any configuration or anything else special.

It does appear that the error message is real and not just a warning in your case.  

In the perl library (/usr/local/lib/perl5 perhaps? ) you'll
find a Storable.pm module with the version number.  You can also use the
perldoc command but you will have to look up the switch that gives you the version number (man perldoc should tell you).  

Let us know what you find out.

-- Mike

In reply to Michael Gage

Re: Answer encryption problem

by Nandor Sieben -
Mike, I think a possible solution could be to simply use newtwork order in the storable module. It seems that this isn't a configuration issue but the use of the appropriate commands in storable. Could you point me to the webwork file that uses the storable module.
In reply to Nandor Sieben

Re: Answer encryption problem

by Michael Gage -
It's used in decode/encodeAnswers in WeBWorK/Utils.pm.  That's the one causing your error.

Also in AchievementEvaluator.pm (same kind of use).

It's also used in DelayedMailer.mail-sender.pm and RestrictedMailer.pm - although I'm not sure whether that is relevant to your current problem.

Let us know how the fix works.  Or if you want to get in on the current
development stuff pull the repos on github.com/openwebwork,  
add your patch and send a pull request to github.com/openwebwork when you are done.

(Right this second I think webwork2-dev and pg-dev are more stable 
then webwork2 and pg since they contain the latest bug fixes.  I'll 
be moving them over to webwork2 and pg once I get back to Rochester.
Don't want to make the change while I'm away.)  

From the point of view of the code you'll be working with on this problem it doesn't matter much which repo you pull.

(There is github information on the wiki -- just search for github or read some of the posts in webwork.maa.org/planet )

Good luck.

-- Mike

In reply to Michael Gage

Re: Answer encryption problem

by Nandor Sieben -
I think I have found the solution:

Line 853 of Utils.pm should be
   return Storable::nfreeze( \@ordered_hash); 
instead of
   return Storable::freeze( \@ordered_hash); 

This fixes my problem.

The explanation is at http://search.cpan.org/~ams/Storable-2.30/Storable.pm which says:
"Surprisingly, the routines to be called are named freeze and thaw. If you wish to send out the frozen scalar to another machine, use nfreeze instead to get a portable image."

I can't find the file AchievementEvaluator.pm so I am not sure whether this needs to be fixed at other places. I searched for freeze in files I did not find any.

I don't feel confident enough about patches, so I hope someone can apply this fix to both the current and the development versions.

In reply to Nandor Sieben

Re: Answer encryption problem

by Arnold Pizer -
Hi,

Looking around a bit, there doesn't seem to be any reason not to use nfreeze instead of freeze (see e.g.  http://blogs.perl.org/users/ovid/2010/10/storable-freeze-versus-nfreeze.html).

As Mike says, it is only used in two modules:

wwadmin@wwserver:/opt/webwork/from_openwebwork_dev_repository$ grep -R freeze *
webwork2/lib/WeBWorK/Utils.pm:  return Storable::freeze( \@ordered_hash);
webwork2/lib/WeBWorK/AchievementEvaluator.pm:use Storable qw(freeze thaw);
webwork2/lib/WeBWorK/AchievementEvaluator.pm:   #update counter, freeze localData and store
webwork2/lib/WeBWorK/AchievementEvaluator.pm:   $userAchievement->frozen_hash(freeze($localData));
webwork2/lib/WeBWorK/AchievementEvaluator.pm:    #freeze globalData and store
webwork2/lib/WeBWorK/AchievementEvaluator.pm:    $globalUserAchievement->frozen_hash(freeze($globalData));


Thanks Nandor.

Arnie