Installation

Batch-editing problem file locations

Batch-editing problem file locations

by Andree Chea -
Number of replies: 2

Hello,

I have recently upgraded (reinstalled) a WeBWorK installation for my old high school to 2.4-dev using CVS from a version that I forgot to check before reformatting (it had DvipngDepths database, if that helps...). I exported the webwork database and courses folder from the old installation and installed it to the new installation and WeBWorK seemed to have no problems reading them (I executed wwdb_upgrade for good measure), but we haven't done much other than log in.

There is one thing that we want to do differently with this installation than with the previous one: namely, use our own custom problem files. What is the recommended way to do this? In our old installation, we had two symbolic links in each course folder (Library and Local) that both pointed to /opt/ProblemLibrary. The reason for Local and Library being identical was that we had some set definition files from ASU that used Local instead of Library, and we didn't want to change the Local link to Library back then (so we 'shortcutted'). If I'm correct in my assumption that we upload custom problem files to Local, then that means that I'll have to change all of the problem urls in each course from Local/something/something to Library/something/something. If we didn't want to use our own custom files, then I could have linked /opt/ProblemLibrary to /opt/webwork/libraries/database_problems (or whatever the NPL name is) and not worry about it. But since we do, I have to "fix" these issues (and it's also less to fix in the future).

The best solution that I see is to write a script that fixes all possible "Local" problems, which includes symlinks, def files, and the mysql database (afaik), which thus frees the Local symlink for local problem libraries:

for each course
changes Library symlink to /opt/webwork/libraries/ProblemLibrary, which points to database_problems
change Local to (1)

for each def file
change all instances of Local to Library

for all assignments (in mysql database)
change all problem URL instances of Local to Library (2)

I have a few questions in this script:
1) Where should Local point to (that holds our custom problems)?
2) What tables should I be editing in the webwork database?

I'm not so experienced with bash scripts, so I'm reading up on loops, sed, mysql etc. I also am going to test this not-yet-written script on my laptop first (from the backups) before running on the live server. Is this the best solution for our problem? TIA.

Andree
In reply to Andree Chea

Re: Batch-editing problem file locations

by Gavin LaRose -
Hi Andree,

I think what you've outlined here sounds good. I have all of our local problem libraries in /opt/webwork/problembanks, which is similar to your setup. If I were you I would be tempted to put them in /opt/webwork/libraries/local_problems or something similar.

I believe the only tables in the database with path information are the course_problem and course_problem_user tables. Depending on how your courses are used (and how many you have), however, it may be possible to only fix a few of them and not worry about the rest. We have a lot of courses per semester (on the order of 100), so I tend to archive and remove them after the term and not worry if there are broken links. If you can connect to mysql with the mysql client, you could get a list of all affected problems with

use webwork;
select set_id, problem_id, source_file from `COURSENAME_problem` where source_file like '%Local%';

If you run

select set_id, problem_id, source_file from `COURSENAME_problem` where source_file like '%Local%' into outfile '/location/of/filelist.txt';

you'll get a list of the files in a convenient file that might then expedite writing a mysql script to correct the paths. Again, though, it or may not be the case that having to edit the mysql tables is practically necessary.

Gavin
In reply to Gavin LaRose

Re: Batch-editing problem file locations

by Andree Chea -
Thanks for the pointers. I've written and tested the scripts on my laptop and it appears to be working (and I learned a few things in the process!), and will test on the webwork server within a few days. My scripts are here for whoever stumbles across this thread in the future and wants it (be sure to test first!):
#!/bin/sh

# Update Problem Library Locations
# by Andree Chea

# 20080526

# This script updates problem library locations, by changing prefixes whereever a problem file may be
# A detailed description as per this forum post: http://wwrk.maa.org/moodle/mod/forum/discuss.php?d=532
# This thread was found in the process: http://wwrk.maa.org/moodle/mod/forum/discuss.php?d=311

# changes symlinks in each course to the correct directory
# from http://webwork.maa.org/wiki/CVS_Access_to_Problem_Libraries

for each in /opt/webwork/courses/*/templates
do
 #change Library path
 cd $each

 course_name=`echo "$each" | awk -F/ '{ print $5 }'`

 echo $course_name

 if [ -h "Library" ] #if exists and is a symbolic link
 then
 rm -f Library
 ln -s /opt/webwork/libraries/ProblemLibrary Library
 fi

 if [ -h "Local" ] #local
 then
 rm -f Local
 ln -s /opt/webwork/libraries/local_problems Local
 fi

 # get list of def files containing Local at beginning
 # grep --files-with-matches --basic-regexp "^Local/*" `find . -type f`
 # http://www.linuxdevcenter.com/pub/a/linux/lpt/09_22.html - xargs with spaces in file names, not working...

 for file in $( find -P . -name "*.def" -type f | xargs grep --files-with-matches --basic-regexp "^Local/*" )
 do
 sed -e "s/^Local\//Library\//g" "$file" > tmp_file #replace all instances of ^Local\/ with Library/
 mv tmp_file "$file"
 done
done


exit
---
#!/usr/bin/php

<?php
 // Updates problem library locations in mysql
 // By Andree Chea

 // 20080602

 // To be used with update_pl_locations, which edits the courses directory
 // Should edit problem and problem_user tables


 // updates path results so that Local is library
 function updatePaths ( $m_table_list )
 {
 //can be problem_user or problem, doesn't matter
 //m_table_list is raw mysql output of SHOW TABLES
 //must do each table separately because of mysql error 1052

 while ( $table_name = mysql_fetch_array($m_table_list))
 {
 $query = "SELECT set_id, problem_id, source_file FROM " . $table_name[0] . " WHERE source_file LIKE '%Local%'";
 $results = mysql_query($query);
 //results has list of afflicted problems
 //change each row (Local to Library) and update
 echo $table_name[0] . " ... ";
 $count = 0;
 $fail = 0;
 while ($row = mysql_fetch_object($results))
 {
 $path_url = str_replace ( "Local/" , "Library/", $row->source_file ); //find and replace
 if (strcmp($path_url, $row->source_file) != 0)
 {
 //update
 $query = "UPDATE " . $table_name[0] . " SET source_file = \"" . $path_url .
 "\" WHERE ( set_id = \"" . $row->set_id . "\" AND problem_id = " . $row->problem_id . ")";
 //echo $query . "\n";
 $status = mysql_query($query);
 //var_dump($row);

 //update counters
 $count++;
 if (!($status))
 $fail++;

 }
 }

 echo $count . ($fail ? " failed: " . $fail : "" ) . "\n";
 }
 }

 function mysqlToString ( $results )
 {
 // http://www.webmasterworld.com/databases_sql_mysql/3276117.htm

 $text = array();
 $string = '';
 while(list($val) = mysql_fetch_array($results))
 {
 if ($val)
 {
 $text[] = $val;
 }
 };
 if (!empty($text))
 {
 $string = implode(", ",$text);
 return $string;
 }
 else
 return "";
 }

 $mysqlUser = "root";
 $mysqlPass = "testpass0";
 $mysqlDatabase = "webwork";

 // open connection to MySQL server
 $connection = mysql_connect(localhost,$mysqlUser,$mysqlPass)
 or die ('Unable to connect to MySQL!');
 // select database for use
 mysql_select_db($mysqlDatabase) or die ('Unable to select database');

 $query = "SHOW TABLES";
 /*$results = mysql_query($query); #list of all courses

 while ($course = mysql_fetch_array ($results))
 {
 echo $course[0] . "\n";
 }*/

 $result = mysql_query($query . " LIKE '%_problem'");
 //$table_list = mysqlToString($result);
 updatePaths ($result);
 $result = mysql_query($query . " LIKE '%_problem_user'");
 //$table_list .= ", " . mysqlToString($result);
 // table_list has all problem and problem_user tables, but can't do because of MYSQL error 1052
 //echo $table_list;
 updatePaths ($result);

 mysql_close($connection);
?>

edit: the forum software messed with the indents...