Achievement Evaluator

The achievement evaluator is the system by which WeBWorK checks to see if achievements have been earned or not. Every time a student submits a problem WeBWorK goes through the list of assigned achievements for that student. For each achievement it loads up a perl script, called the achievement evaluator, and runs that script (in a Safe container). If the script returns 1 then the achievement is awarded and if it returns 0 then the achievement is not awarded. This means that all of the heavy lifting for assigning achievements is done by these perl script achievement evaluators.

Editing Achievement Evaluators

Each achievement as an evaluator file which can be set using the edit tab of the Achievement Editor page. The evaluator itself can be edited using the "Edit Evaluator" link. This will bring you to a text editor page where you can make changes, save, or save as a new editor which can replace the existing one or be used in a new achievement. The evaluators are assigned to a particular achievement using the edit tab and the evaluator files are all stored in the achievements directory in the templates folder. At heart an achievement evaluator is nothing more than a perl script that returns a true or false value. For example, here is the evaluator script for an achievement which is awarded when a student finishes a homework set (100% on every problem) between midnight and 2am.

my @timeData = localtime(time);

#test to see if it is between midnight and 2am return 0 if it isn't
if ($timeData[2] > 1) {
     return 0;
}

#if it is check to see if we have finished the set return 0 if we haven't
foreach my $problemRecord (@setProblems) {
     if ($problemRecord->status != 1) {
          return 0;
     }
}
    
#if we got this far then the student should be awarded the achievement
return 1;

The evaluator system is very flexible. You can implement any achievement which can be tested using a perl script. Note: The functions available to you inside the perl script are somewhat limited because of the Safe environment. They can be expanded by editing the AchievementEvaluator.pm file.

Evaluator Environment Variables

When the achievement evaluator is run there are a number of environment variables available for use. Most of these variables are read only, but some of them will persist across evaluator calls. The available variables are listed below.

• $problem : The problem data. Changes to this variable will not be saved! This variable contains the problem data. It is a hash pointer to a standard WeBWorK problem data structure. The following variables are a sample of those stored in $problem.
  • $problem->status : the score of the current problem
  • $problem->problem_id : the id of the current problem
  • $problem->set_id : the id of the set containing the problem
  • $problem->num_correct : the number of correct attempts
  • $problem->num_incorrect : the number of incorrect attempts
  • $problem->max_attempts : the maximum number of allowed attempts

• $set : The set data. Changes to this variable will not be saved! This variable contains the set data. It is a hash pointer to a standard WeBWorK set data structure. It contains the following values (not all values shown).
  • $set->open_date : when the set was open
  • $set->due_date : when the set is due

• @setProblems : The problem data for all the problems from this set. Changes to this variable will not be saved! This is an array of problem hashes. Each element of the array has the same hash keys as the $problem variable above

• $counter : The users counter associated to this achievement. Changes to this variable will be saved! If this achievement has a counter associated to it (i.e. solve 20 problems) then this is where you store the students counter for this achievement. This variable will initially start as undefined. If you do not store your counter data here students will not be able to see their progress on the Achievements page.

• $maxCounter : The goal for the $counter variable for this achievement. Changes to this variable will not be saved! If this achievement has a counter associated to it then this variable contains the goal for the counter. Your achievement should return 1 when $counter >= $maxCounter. This variable is set using the edit tab of the Achievement Editor page.

• $tags : This contains the metadata for the problem. This is a hash pointer to a standard tags structure. Note: These values are not super stable and will change as the library tagging is updated. It contains the following values (not all values shown).
  • $tags->DBsubject : The subject tags for the problem
  • $tags->DBchapter : The chapter tags for the problem
  • $tags->DBsection : The section tags for the problem

• $localData : This is a hash which stores data for this user and achievement. Changes to this variable will be saved! This hash will persist from evaluation to evaluation. You can store whatever you like in here and it can be accessed next time the evaluator is run. Two things to keep in mind. First, The data in this hash will not be accessible by other achievements. Second, the first time a variable is accessed it will be undefined.

• $globalData : This is a hash which stores data for this user and all achievements. Changes to this variable will be saved! This hash will persist from evaluation to evaluation and, like $localData, you can store whatever you like in here. This data will be accessible from every achievement and is unique to the user.
  • There are two variables stored in this hash that are maintained by the system.
    • $globalData->completeSets : This is the number of sets which the student has earned 100% on.
    • $globalData->completeProblems : This is the number of problems which the student has earned 100% on.
  • Warning: The achievements are always evaluated in the order they are listed the Achievement Editor page. To make matters more complicated, achievements which have already been earned are not evaluated at all. The up-shot of this is that when modifying variables in $globalData you need to either write your code so it doesn't matter which order the evaluators are run, or you need to pay very close attention to which evaluators are run and when.
  • The Achievement Items are kept track of using this hash.

In addition to the above variables there are two special variables available to "Level" achievements. Level achievements behave differently than other achievements. When a level achievement is earned the system uses the icon and title of the level achievement as the icon and title of that students level. There are two additional variables available to level achievements.

• $achievementPoints : The number of achievement points the current user has. Changes to this variable will not be saved.
• $nextLevelPoints : The number of points needed to earn the a level. Changes to this variable will be saved if a new level is earned. These variables are used to construct the users level progress bar.

Creating New Achievements

Achievements can be created using the Create tab on the Achievement Editor page. You define the achievement ID when you create the achievement and then you will need to set the other fields using the Edit tab (or by clicking on the pencil). You will need to define the following:

• Achievement ID: This can not be renamed. It is used to identify achievements and to sort them. (Which is why a lot of the achievement ids start with numbers.)
• Achievement Name: This is the visible title attached to the achievement.
• Achievement Description: This is the description of the conditions needed to earn the achievement.
• Achievement Category: This is not visible to students but it has a special purpose. Achievements are sorted (for presentation and evaluation) first by category and then alphabetically according to their achievement id. (Which is why categories also tend to start with number.) There are two special categories that deserve to be singled out.
  • Achievements in the "secret" category will not be visible to the students until after they are earned and are for fun/surprise achievements.
  • Achievements in the "level" category are used to determine/control a students Level. They are not visible to students in the usual way. If a student earns a new level the achievement name and icon become the new level name and icon.
• Achievement Icon: These icon files are all located in the [TMPL]/html/achievements directory. If you put in the wrong file name then you will end up with broken image links. If you don't put in any filename into the Icon File entry then you will get a default icon.
  • There are extra icons in the [TMPL]/html/achievements directory that are not used for any of the default achievements.
• Achievement Points: This is for the number of achievement points the student will gain by earning the achievement.
• Counter: This is where the value of $maxCounter is set.
• Achievement Evaluator: This is where the name of the evaluator file corresponding to the achievement is kept. These files are kept in the [TMPL]/templates/achievements directory.

Content Achievements

The achievements work best when they are more closely tied to a particular course. One good way to do this are to have "content" achievements which reward students for mastering a particular topic. For example you can have an achievement for solving a particularly hard Related Rates problem or doing 20 derivatives. These achievements are not included by default because they are course specific. However, they are not difficult to create. For example, the following achievement checks to see if students have solved a certain number of a specific type of problem.

#define valid problems using this nested hash
my %validproblems = (
       'SetNameHere' => { 
           '1' => 1, 
           '2' => 1, 
           '3' => 1, }, 
       'AnotherSetNameHere' => { 
           '2' => 1, 
           '5' => 1, 
           '7' => 1, }, #etc...
       );

#check and see if this problem was solved
if ($validproblems{$problem->set_id} &&
       $validproblems{$problem->set_id}{$problem->problem_id} &&
       $problem->status == 1 &&
       $problem->num_correct == 1) {
           #update counter;
           $counter++;
   }

if ($counter >= $maxCounter) {
   return 1;
} else {
   return 0;
}

All you need to do to customize this achievement for your course is to specify which problems count toward the achievement using the %validproblems hash. There are icons and evaluators for calculus themed content achievements included in the model course (although these achievements are not in the default import list).