UsingWW: Michael Gage - Migrating to WeBWorK 2.0 version 3 (8/25/03)

$inactiveTopic$ Migrating to WeBWorK 2.0 version 3 (8/25/03)

topic started 8/25/2003; 3:55:42 PM
last post 8/25/2003; 3:55:42 PM

$user$ Michael Gage - Migrating to WeBWorK 2.0 version 3 (8/25/03) $blueArrow$
8/25/2003; 3:55:42 PM (reads: 2803, responses: 0)

Instructions for migrating to WeBWorK 2.0 (ver 3.0 8/25/03)

These instructions assume that you are already running WeBWorK 1.8 or 1.9, and details how to install WeBWorK 2.0 so that it runs in parallel for the same courses.

WeBWorK2 is pretty stable from the student users point of view. The tools used by the instructor are now reasonably stable and complete. Those features that are missing -- such as upload of a classlist can be accomplished by switching to the webwork 1.9 tools for that particular task. The two sets of tools use the same database, so the operations can be done using either the old or the new tools.

For the forseeable future we'll ask that anyone wishing to try out WeBWorK 2.0 obtain the scripts via a CVS (as per the instructions in this document) so that they can easily update to obtain the latest bug fixes as they are made.

A second step in the migration (not yet described in detail) will allow you to use a mysql database instead of the GDBM database currently in use. Using the mysql database will increase the options available for the instructor, as well as the speed on large data sets, however it will no longer be possible to use the webwork 1.9 tools which will work only with the GDBM database.

In order to use WeBWorK 2.0 you MUST have mod_perl enabled on your apache server. (If you don't know what this means, then wait a bit before trying out WeBWorK 2.0 -- or consult with your local UNIX computer guru to have your copy of the apache webserver upgraded.)

I'll assume three things:

your current webwork 1.8 system has the structure .../topDir/webwork/system/.... and that system directory has sub directories such as cgi, courses,courseScripts, scripts, and so forth.
Similarly your course will be contained in a structure ../courses/myCourse/... and the myCourse directory will contain webworkCourse.ph, html, templates, and so forth. The templates directory will contain a macros directory (which will probably be empty). (In many cases, following the standard setup for WeBWorK1.8, the course directory will be at ../topDir/webwork/courses/myCourse, that is the courses and sytem directory are siblings.
Finally the links to your courses are of the form http:\//myMachine/myCourse/. For example you might have a directory called ../htdocs/ which is the home directory for your apache server and which contains links to the html directory of myCourse: ls -l myCourse ---> ...../courses/myCourse/html/

Let's begin

First we set up the ssh key, then download and create the directories linked to the cvs repository. (Full details are at http://webwork3.math.rochester.edu/cvs.html)

Setup ssh key files

These files are needed in order to automatically access the webwork server via ssh.

ssh (Secure Shell) is a replacement for telnet which does not send passwords as clear text.

Set up ssh key file(see details at http://webwork3.math.rochester.edu/cvs.html ) -- this seems to be the step that often causes difficulties. See subsequent posts below for troubleshooting.

We want to create the file ~/.ssh/webwork-anoncvs containing (approximately) the following text

Making sure that the contents of this file are exactly correct seems to be difficult when copying the text from the screen. Instead use option-click (or alt-click or control-click -- whatever works in your browser) to copy the key file directly to your desktop or download directory. The url for the key file is http://webwork3.math.rochester.edu/webwork-anoncvs . Move or copy the downloaded file to ~/.ssh/webwork-anoncvs (no extension) and make sure that the permissions are set correctly as below.

Set the permissions for this file webwork-anoncvs to 600 or it will not work. Secret keys are not supposed to be generally readable!

Add these lines to the ~/.ssh/config file . You may have to create it.


					Host webwork3.math.rochester.edu

					User anoncvs

					IdentityFile ~/.ssh/webwork-anoncvs

					ForwardAgent no

					ForwardX11 no

					Compression yes

Set the environment variable CVS_RSH to ssh . In bash, type export CVS_RSH=ssh . In tcsh, type setenv CVS_RSH ssh . Since you need this variable set whenever you want to use the webwork CVS repositories, you might want to add it to your .bashrc, .profile or .cshrc file (depending on your shell).

Warning -- Experts logging in to webwork3 under their own assigned user names in order to upload patches to code should not set up an anonymous account. Depending on the order things are written in the config file, the anoncvs user name might become the default instead of your standard user name. See http://webwork3.math.rochester.edu/cvs.html for setting up non-anonymous user accounts.

Downloading the webwork 2.0 files

cd topDir/webwork # (get to your webwork directory)
In order to install the webwork-modperl scripts you will need write permissions in this directory.
The address of the CVS repository has changed! It's now on webwork3.math.rochester.edu
cvs -d :ext:anoncvs@webwork3.math.rochester.edu:/webwork/cvs/system checkout webwork-modperl
This will create a directory webwork-modperl inside topDir/webwork
Next we create the directory for pg
cvs -d :ext:anoncvs@webwork3.math.rochester.edu:/webwork/cvs/system checkout pg

Set permissions on webwork2 (webwork-modperl) files and directories

We need to make that the webwork2 files can be read by the apache daemon

The directories have 744 permissions when they are created. We need to make sure that the directories can all be executed and read by the apache daemon. A few directories (the log directories and tmp directories) also require write privileges.

Add execution privileges to sub directories for users in the group -- this allows access to the files in the directory.
1. While still in topDir/webwork type:
  1. chmod -R go+X webwork-modperl
  2. chmod -R go+X pg
  3. This adds execution permissions to all of the directories in the webwork-modperl tree and the pg tree, allowing access to the files in the webwork-modperl directories. The capitalization is important (as is usual in unix commands).
  4. We are using relatively lax security, since anyone can now read these files. ("Tightening the permission structure" -- to be written)
Determine the groups which your apache server belongs.
1. One possibility is to type
2. ps -aux |grep httpd
3. to get a listing of processes which are running. Look for something like
4. wwhttpd 72270 0.0 0.1 1896 1384 ?? I 1:00AM 0:00.01 /usr/local/sbin/httpd
  wwhttpd 72274 0.0 0.1 1908 1392 ?? S 1:02AM 0:00.04 /usr/local/sbin/httpd
5. This tells me that the apache daemon is named wwhttpd on our machine
6. (Troubleshooting: Finding the webserver's group name: http://webhost.math.rochester.edu/webworkdocs/discuss/msgReader$1730)
7. Next type
  1. groups wwhttpd
    webwork MTH161 MTH141A MTH140A MTH140L MTH141 MTH142 MTH143 ....
8. Our daemon belongs to many course groups and also to one general group webwork. For WeBWorK 2.0 the daemon only needs to belong to one group -- in our case webwork
9. You may need to consult with whomever has been setting up groups for your WeBWorK 1.8 course. Troubleshooting.
Set the group for the webwork-modperl files
1. From the topDir/webwork directory set the group for the directory webwork-modperl to webwork chgrp -R webwork webwork-modperl
  ls -l
  drwxr-xr-x 14 gage webwork 512 May 31 12:33 webwork-modperl/
  
  chgrp -R webwork pg
Set write permission for the logs directory and the tmp directory
1. % cd webwork-modperl % ls -l
  total 24
  drwxr-xr-x 2 gage webwork 512 May 31 12:33 CVS/
  drwxr-xr-x 3 gage webwork 512 May 31 12:33 bin/
  drwxr-xr-x 3 gage webwork 512 May 31 12:33 conf/
  drwxr-xr-x 4 gage webwork 512 May 31 12:33 courses/
  drwxr-xr-x 3 gage webwork 512 May 31 12:33 doc/
  drwxr-xr-x 4 gage webwork 512 May 31 12:33 htdocs/
  drwxr-xr-x 5 gage webwork 512 May 31 12:33 lib/
  drwxr-xr-x 3 gage webwork 512 May 31 12:33 logs/
  drwxr-xr-x 4 gage webwork 512 May 31 12:33 tests/
  drwxr-xr-x 3 gage webwork 512 May 31 12:33 tmp/
2. % chmod g+w logs tmp # make the directory writeable for the group webwork
3. % ls -l # check that it worked!
  drwxrwxr-x 3 gage webwork 512 May 31 12:33 logs/
  drwxrwxr-x 3 gage webwork 512 May 31 12:33 tmp/

Configure WeBWorK 2.0

-- to make sure that it knows where to find files and databases. This is done in the configuration (conf) directory.

% cd conf %ls
CVS/
gdbm.conf.dist
global.conf.dist
httpd.conf.INSERTME
httpd-wwmp-header.conf.dist
httpd-wwmp.conf.dist
snippets/
sql.conf.dist
templates/
Make local copies of the global.conf.dist, gdbm.conf.dist and sql.conf.dist files.
1. cp global.conf.dist global.conf cp gdbm.conf.dist gdbm.conf cp sql.conf.dist sql.conf nano global.conf (# or pico global.conf might work, or use emacs or vi :-) )
2. The variable $webworkRoot will point to the directory topDir/webwork/webwork-modperl which you have created. You should not need to change any of these values when using the default configuration. If you change the directory structure of WeBWorK for some reason you may also have to make changes here.
Initially you will only need to make one change (if your directory structure is as described in the instructions above).
1. Change addresses for oldProf in the structure
2. %webworkURLs = ( root => "$webworkURLRoot", home => "/webwork2_files/index.html", htdocs => "/webwork_files", docs => "http://webhost.math.rochester.edu/webworkdocs/docs", oldProf => "/webwork-old/profLogin.pl", );
3. to
4. %webworkURLs = ( root => "$webworkURLRoot", home => "/webwork2_files/index.html", htdocs => "/webwork2_files", docs => "http://webhost.math.rochester.edu/webworkdocs/docs", oldProf => "http://your.host.edu/cgi-bin/system/cgi/profLogin.pl", );
5. The oldProf url should just like the url that WeBWorK 1.8 uses on your system when you go to the "professor Page" to edit classlists, butild sets and so forth.
6. The structure for the external programs used by webwork may also need adjusting %externalPrograms = ( mkdir => "/bin/mkdir", tth => "/usr/local/bin/tth", pdflatex => "/usr/local/bin/pdflatex", latex => "/usr/local/bin/latex", dvipng => "/usr/local/bin/dvipng -mode ljfivemp -D600 -Q6 -x1000.5 -bgTransparent", gif2eps => "$webworkDirs{bin}/gif2eps", png2eps => "$webworkDirs{bin}/png2eps", gif2png => "$webworkDirs{bin}/gif2png", );
7. You can insure that the external programs have been installed on your machine by typing
  1. which tth /usr/local/bin/tth which pdflatex /usr/local/bin/pdflatex which dvipng /usr/local/bin/dvipng

Configure the Apache server

We will tell the apache server where to find the webwork 2.0 files. It helps to have someone who has set up and restarted the apache server help out at this point.

First find the apache config file. Usually called httpd.conf and found perhaps in /etc/httpd/ or in our case /usr/local/etc/apache
1. On MacOS X the main file is at /private/etc/httpd/httpd.conf, but you can add the new code lines to /private/etc/httpd/users/username.conf where username is your login name on the mac. These user configuration files are included into the main httpd.conf file when apache starts up.
Check that httpd.conf contains these two lines (uncommented). The lines are in two different sections of http.conf
LoadModule perl_module libexec/httpd/libperl.so AddModule mod_perl.c
If these lines are there, but commented out with a #, just uncomment them. If they are not there at all then you first need to build an apache daemon which will run mod_perl. That is beyond the scope of this note.
Assuming that all is well, we need to add these two code fragments to the httpd.conf file. (These directions are also available on the page how to use CVS with WeBWorK2.) You will see similar code fragments in the file already.
The fragments you need are in the file conf/httpd.conf.INSERTME but you will need to edit them first. You can edit a copy of that file and then cut and paste the contents into http.conf.
The WEBWORK_DIR should be replaced by ...topDir/webwork as described above.
<IfModule mod_perl.c>
PerlFreshRestart On
<Location /webwork2>
SetHandler perl-script
PerlHandler Apache::WeBWorK
PerlSetVar webwork_root WEBWORK_DIR/webwork-modperl
PerlSetVar pg_root /WEBWORK_DIR/pg
<Perl>
use lib 'WEBWORK_DIR/webwork-modperl/lib';
use lib 'WEBWORK_DIR/pg/lib';
</Perl>
</Location>
</IfModule>
We also need to define url link for the accessing images and such like.
# This alias is important.
Alias /webwork2_files/ /WEBWORK_DIR/webwork-modperl/htdocs/
The alias allows you to find installation wide information stored in the webwork-modperl/htdocs directory. This includes the index.html file and courses which contains links to the html directories of each of your courses.
Finally here is a little test module that will help with trouble shooting which is worth installing.
# status test
<Location /ww_status>
SetHandler perl-script
PerlHandler Apache::Status
</Location>

Restart the Apache server

Before doing this find the location of the apache error log.

You can search httpd.conf for ErrorLog. Our error log is stored at /ww/logs/error_log on one machine and /var/log/error_log on another.

In a separate window issue the command


				tail -f  /ww/logs/error_log

(use your own path to the error log) to continuously view the last 20 lines or so of errors issued by the apache server.

The restart command is called apachectl (apache control) and is probably in /usr/local/sbin You need super user privileges to start and stop the server, hence the use of the sudo command.

Tell the apache server to shutdown and restart gracefully.


					sudo /usr/local/sbin/apachectl graceful

You can also use the commands apachectl start and apachectl stop

In the window showing the contents of the error_log you should see a flurry of activity. The last messages should not be error messages. If they are, try to decipher them. They might indicate a typo in http.conf which you have to correct.

Another possibility is that one or more perl modules have not been installed. (Mail::Sender.pm is the most likely culprit). These modules are all common perl modules and can be obtained and installed from CPAN. ("Instructions on installing CPAN modules" to be written.)

Refer to the error log listing whenever something funny happens. There may be a useful error message.

Testing the server

Open a browser go the url http://your.host.edu/ww_status. If this works it should give a number of links to information about your apache server, including which modules are installed. This means that mod_perl is working. Your version of perl should be at least 5.6 in order to insure compatibility with WeBWorK2.0.

Next type the url http://your.host.edu/webwork2_files. This should give a response of Welcome to WeBWorK2.0 at the University of Rochester

For the fun of it go to topDirs/webwork/webwork-modperl/htdocs and edit the file index.html, replacing the University of Rochester by your own institution. This is just an html file, you can add anything else you want. Test that it works. You don't have to restart the server.

Now type http://your.host.edu/webwork2 This should redirect you to the file above. If this works then the locations pointed to by the entries in the http.conf file are correct.

Here is a review of the connections which have been made in the http.conf file. (See diagram )

<Location /webwork2>: Tells us that webwork URL for all dynamic actions is http://your.host.edu/webwork2
PerlSetVar webwork_root WEBWORK_DIR/webwork-modperl: Tells perl the path to your webwork-modperl directory, which contains the bulk of the WeBWorK2 files
PerlSetVar pg_root /WEBWORK_DIR/pg: Tells perl the path to your pg directory, which contains the macros and .pm files which define the .pg language for writing problems in WeBWorK1.8 and WeBWorK2.0
use lib 'WEBWORK_DIR/webwork-modperl/lib';: Insures that perl looks for files in the webwork-modperl library tree
use lib 'WEBWORK_DIR/pg/lib';: Insures that perl also looks for files in the pg library tree
Alias /webwork2_files/ /WEBWORK_DIR/webwork-modperl/htdocs/: Tells us that the address for static files the webwork htdocs directory are reached with the url http://your.host.edu/webwork2_files/...; The fragment containing the directory defines the permissions which decide who is allowed to see these files.

Trouble shooting apache/perl modules

Keep the window to the error log open and check it often!

Otherwise mysterious errors print relatively helpful messages to the error log.

Missing modules

You will probably be missing some perl modules that are used by WeBWorK. The error message in the error log file will tell you which modules cannot be found. The modules are all available on CPAN.

For example:


						[Wed Aug 13 10:12:10 2003] [error] Can't locate Apache/Request.pm in @INC
						(@INC
						contains: /home/webwork/pg/lib /home/webwork/webwork-modperl/lib
						/usr3/Linux/loc .............

This indicates that the module Apache::Request is missing from your perl/apache installation. You may have to scroll back through your error log output in order to see the first line which is the most important one. (This module is often missing on Mac OS X systems, it's less likely to be missing on a linux system.)

The lines after the first one indicate the directory in which perl searched for the missing modules. Sometimes the problem is that this search path is not complete. In that case you will need to consult a local perl/apache/unix guru to reconfigure the search paths.

Three more modules which are likely to be missing are

Time::HiRes
Mail::Sender and
MIME::Base64

On many systems you can install modues using CPAN with the command sequence:


					sudo perl -MCPAN -e shell
					cpan>  install Time::Res
					etc.

We have had some trouble installing Time::HiRes on some systems, depending on the environment. You can email us (gage@math.rochester.edu) or better yet post to the list (http://webhost.math.rochester.edu/webworkdocs/discuss/edit/newTopic) if you run into trouble with this one. With enough examples we should be able to write a FAQ to guide you through the installation of this module.

Hooking up to your current WeBWork 1.8 course

We now know that WeBWorK2.0 system is working and connected. It remains to connect WeBWorK2.0 to the WeBWorK1.8 courses which you already have set up.

There are two connection points to WeBWorK2.0

Dynamic URL:A URL of the form http://your.host.edu/webwork2/myCourse will access myCourse if it is in the directory topDir/webwork/webwork-modperl/courses/myCourse In fact that directory already contains a minimal course called test_course so http://your.host.edu/webwork2/test_course should give you a login page. (If you enter a login name you will get an error, because no database of users has been built for this course yet. Just getting the login message is enough to confirm that this link is correctly set up.)
Passive URL: The second link is for URLs of the form http://your.host.edu/webwork2_files/myCourse which will access the links in topDir/webwork/webwork-modperl/htdocs/courses. The htdocs/courses directory contains symlinks to the html directory in each course. In our case
1. ls -l htdocs/courses
2. should give
3. test_course --> topDir/webwork/webwork-modperl/courses/test_course/html

Notice that the first directory, topDir/webwork/webwork-modperl/courses corresponds to the directory ../courses mentioned as the second of the three assumptions I made at the top of this page. It is the directory that contains all of your WeBWorK1.8 courses. (diagram) We will make a symlink between the two directories. While in the directory webwork-modperl type

mv courses courses.save

ln -s .../courses courses

We moved the webwork-modperl/courses directory out of the way (saving it) so that we could construct a symlink between the webwork-modperl/courses link (alias) and the .../courses directory containing the WeBWorK 1.8 courses. The "test_course" URL will no longer work, since it is now in courses.save instead of in courses.

At this point webwork 2.0 knows the directory in which to find the courses.

Next we hook up the passive URL link which searches for html files associated with the course. These are in the course's html directory.

Transfer to webwork_modperl/htdocs (cd webwork-modperl/htdocs):

If necessary create a directory courses (mkdir courses) and transfer to the courses directory.

Create a symlink using


				ln -s ......./courses/myCourse/html myCourse

The URL http://your.host.edu/webwork2_files/courses/myCourse/index.html will now find the file ..../courses/myCourse/html/index.html

Testing the hook-up

Type the url http://your.host.edu/webwork2_files/myCourse (making the obvious substitutions for your server and your course)

You should get the login page for your existing WeBWorK1.8 course! If you don't, then there might be a problem with the symlink between the directory ..topDir/webwork/webwork-modperl/htdocs/courses and the .../htdocs directory. Check also that the .../htdocs directory has a symlink called myCourse which points to .../courses/myCourse/html. It is a common mistake to forget the last html in the link.

The other connection that might be at fault is the Alias entry in http.conf (Alias /webwork2_files/ ...topDir/webwork/webwork-modperl/htdocs/ )

Assuming all is well so far type the url http://your.host.edu/webwork2/myCourse where myCourse is the name of one of your existing courses. You should get a login screen and be able to view your course. If this doesn't work at all, check the <Location /webwork2> directive in the httpd.conf file. Check also the permissions on the files and directories in webwork-modperl. Remember that the apache server must be able to execute the directories in order to read the files inside.

If you get an error screen saying something like: "Can't read .../macros/IO.pl" then either the PerlSetVar pg_root topDir/webwork/pg is not pointing to the topDir/webwork/pg directory or the server does not have permission to read the files in that directory (Again remember that it must be able to execute the directories in order to read the files inside.)

The access to the professor pages is still being developed. When you click on the "instructor" link in the left maargin you will be taken to a page where you can use the "professor pages" from WeBWorK 1.8 to edit your course (the third and fourth links) or you can use the second link to edit problem sets. The first link, to edit users is not yet very functional. If the third and fourth links don't work check the value given to oldProf in the global.conf file.

You can report bugs using the bug link in the upper left corner. The first time you do this you will need to login and receive a password via e-mail. Once logged in bugzilla will set a cookie and you will be taken directly to the page for reporting bugs.

Speeding up large classes.

You don't HAVE to make any configuration changes in your course directory. However one change will speed up the listing of problem sets in large courses:

Inside your course myCourse (this directory contains webworkCourse.ph, templates, DATA, etc.) create a file course.conf

Inside course.conf place the lines $dbLayout{set}->{params}->{globalUserID} = "yourID"; # yourID is the instructor's loginID for the course (or whoever is responsible for building sets and maintaing the course.) $dbLayout{problem}->{params}->{globalUserID} = "yourID";

There are other customization which can be made in this file to override those in global.conf, but we'll come to those later.

Test results on my site:

After restarting the server:

http://webwork3.math.rochester.edu:11002/ww_status: This gives me output starting with Embedded Perl version v5.6.1 for Apache/1.3.27 (Unix) mod_perl/1.27 process 91936 , running since Mon Jun 16 13:48:52 2003 and followed by several links.; This assures me that mod_perl is working and reading the httpd.conf file that I just modified.
http://webwork3.math.rochester.edu:11002/webwork2_files/index.html: This should return Welcome to WeBWorK at the University of Rochester which is the contents of the file WEBWORK_ROOT/htdocs/index.html . This indicates that the link to WEBWORK_ROOT/htdocs is set up properly.; You can modify this file to name your institution and add anything else you like.
http://webwork3.math.rochester.edu:11002/webwork2: This should be redirected to the address at webwork2_files above and will give the same output. This indicates that the dispatcher for WeBWorK2 is working.
http://webwork.math.rochester.edu:11002/webwork2/mth143: mth143 is a course in the /shared-courses/ directory. (This location for courses was specified in global.conf.); I get a login page for the course.

My test server runs on the non-standard port number 11002. You should get similar results on your server by replacing webwork3.math.rochester.edu by the address of your server and leaving off the :11002 entirely. This gives a default port number of :80 which is almost certainly the port number used by your server.

Sample errors on my site:

Can't open file /home/gage/webwork/pg/macros/IO.pl for reading Can't open file /home/gage/webwork/pg/macros/PG.pl for reading Can't open file /home/gage/webwork/pg/macros/dangerousMacros.pl for reading: I had forgotten to check out the pg directory! It's also possible that the permissions on the pg directories and subdirectories don't allow the server to access the files. Check (1) that the pg directory exists (2) that the permissions are reasonable for the server. and (3) That the address of PG_ROOT in the httpd.conf file is pointing to the pg directory.

Post comments

Comments

Let me know what worked and what didn't:

Troubleshooting responses to test: ssh anoncvs@webwork3.math.rochester.edu: http://webhost.math.rochester.edu/webworkdocs/discuss/msgReader$1705
Trouble shooting responses to test: cvs -d :ext:anoncvs@webwork3.math.rochester.edu:/webwork/cvs/system checkout webwork: http://webhost.math.rochester.edu/webworkdocs/discuss/msgReader$1706
Problems finding group name of the apache server: http://webhost.math.rochester.edu/webworkdocs/discuss/msgReader$1730

-- Mike Gage

<| Post or View Comments |>

Using WeBWorK

Forum archive 2000-2006

Michael Gage - Migrating to WeBWorK 2.0 version 3 (8/25/03)