Just-In-Time

Introduction

Just-In-Time type sets are a new category of assignment, similar to gateway/quizzes or the traditional homework set. The key feature of Just-In-Time type sets is that they are adaptive. This is accomplished via a tree structure, where every problem may have an ordered collection of "child" problems that are opened up when a student is having trouble. Initially the student will be shown a collection of questions, as usual. However, each problem in the set can (but does not have to) have a collection of associated child problems. If a student runs out of attempts or passes a certain threshold of incorrect attempts then the child problems will be presented to the student. The student will then complete those problems, which will often contain review and practice for the concept presented in the parent problem. The students grades on the child problems can count towards the parent problem, but they don’t have to. The child problems can have their own child problems, if additional review is needed on a particular subtopic. After completing the child problems the student can proceed to the next top level problem from the original collection. If they are unable to complete the child problems then there is an option for the instructor to be notified. An example of this is described below. This new structure is extremely powerful, but designing effective just-in-time sets requires more effort from an instructor than is required for a traditional homework set. From a technical point of view WeBWorK treats these homework sets much the same as it treats any homework set when it comes to assigning to students, importing, exporting, ect... Details on how to set up and manage Just-In-Time type sets are also described here, as are the technical specifications of these sets.

Example

An example of a Just-In-Time homework set with two top level problems is shown below:

• Problem 1: This problem tests to see if students can give the exponential representation of a complex number. Child problems will be opened after a student runs out of attempts.
  Max Attempts: 3
  Att. to Open Children: max attempts
  Counts for Parent Grade: n/a
  
• Problem 1.1: This problem contains a review of Complex Numbers delivered via embedded video.
  Max Attempts: unlimited
  Att. to Open Children: n/a
  Counts for Parent Grade: no
  
• Problem 1.2: This problem walks students through converting a complex number to exponential form using the scaffolding problem type.
  Max Attempts: unlimited
  Att. to Open Children: n/a
  Counts for Parent Grade: no
  
• Problem 1.3: This problem is a practice problem asking students to convert a complex number to an exponential one.
  Max Attempts: unlimited
  Att. to Open Children: n/a
  Counts for Parent Grade: no
  
• Problem 1.4: This problem reevaluates if a student can convert a complex number to exponential form. It is very similar to Problem 1. If students get this problem right it will replace the grade for Problem 1.
  Max Attempts: 3
  Att. to Open Children: n/a
  Counts for Parent Grade: yes
  
• Problem 2: This problem tests if students can do a basic Fourier Series.
  Max Attempts: 5
  Att. to Open Children: max attempts
  Counts for Parent Grade: n/a
  
• Problem 2.1: This problem first tests if a if a student can do integration by parts. It has unlimited attempts but child problems will be opened after 4 attempts.
  Max Attempts: unlimited
  Att. to Open Children: 4
  Counts for Parent Grade: no
  
• Problem 2.1.1: This contains a text review of integration by parts.
  Max Attempts: unlimited
  Att. to Open Children: n/a
  Counts for Parent Grade: no
  
• Problem 2.1.2: Integration by parts walkthrough problem. This has a bunch of subparts to help students with the integral.
  Max Attempts: unlimited
  Att. to Open Children: n/a
  Counts for Parent Grade: no
  
• Problem 2.1.3: Final integration by parts practice problem.
  Max Attempts: unlimited
  Att. to Open Children: n/a
  Counts for Parent Grade: no
  
• Problem 2.2: Review of Fourier Series and Fourier Coefficients. This has an embedded slideshow with audio.
  Max Attempts: unlimited
  Att. to Open Children: n/a
  Counts for Parent Grade: no
  
• Problem 2.3: Practice computing Fourier Coefficients.
  Max Attempts: unlimited
  Att. to Open Children: n/a
  Counts for Parent Grade: no
  
• Problem 2.4: Re-evaluate if a student can compute a basic Fourier series. This is the first of two problems. It counts 50% for score of Problem 2.
  Max Attempts: 5
  Att. to Open Children: n/a
  Counts for Parent Grade: yes
  
• Problem 2.5: Re-evaluate if a student can compute a basic Fourier series. This is the second of two problems. It counts 50% for score of Problem 2.
  Max Attempts: 5
  Att. to Open Children: n/a
  Counts for Parent Grade: yes
  

A student working through this set would, at first, only be greeted with Problems 1 and 2. If the **Restrict Problem Progression** flag is set then they will need to start with problem one, otherwise they could try either problem. If a student completes problems 1 and 2 with the given number of attempts then they will be done. They won't even be shown the child problems. If a student runs out of attempts in Problem 1 then they will be shown the child problems 1.1 through 1.3. These problems start with a review of the concept, then a practice problem. Neither of these problems have restricted attempts, but they also don't count toward a replacement for the parent problem grade. However if the student finishes Problem 1.3 in the given number of attempts then that grade will replace the grade of Problem 1 in their homework. In particular from the point of view of scoring this Just-In-Time type set will only have two problems.

Moving on, if the student fails to finish the second problem then they will be presented problems 2.1 through 2.5. If they get problem 2.1 wrong enough times then they will be shown problems 2.1.1 through 2.1.3. These problems all review the concept of problem 2.1 (integration by parts), which is important for completing Problem 2. However none of them counts towards the grade of Problem 2 and if a student completes Problem 2.1 successfully then they are never shown. Problems 2.2 and 2.3 contain review and practice for the topic of Problem 2. In this case the two problems which count towards the grade for Problem 2 are Problems 2.4 and 2.5, each of which as a weight of 50%.

When it comes to scoring this set will look like it only has two problems, Problem 1 and Problem 2. The score for each of the problems will be the larger of the score on the problem and the score coming from its child problems.

Just-In-Time Set Parameters

Just-In-Time type sets have configuration options which are specific to this type of set. There are two classes of options, options for the set and options for individual problems.

Terminology

Top Level Problem 

This is a problem which is not the child of another problem, e.g. Problem 1,2,3,etc... The score of a student on a Just-In-Time set is determined using the adjusted status' of the Top Level Problems.

Adjusted Status 

The adjusted status of a problem is the larger of its status and the weighted average of the status of its child problems with the counts for parent flag enabled. When there are multiple levels the adjusted status is computed recursively.

Child Problems 

These are the problems which are opened when a student gets a certain number of unsuccessful attempts on a problem. For example, the child problems of Problem 1 might be Problems 1.1, 1.2, and 1.3.

Parent Problem 

This is the problem to which a child problem is attached. For example the parent of Problem 1.3.2 is Problem 1.3.

Finish a Problem 

This happens when a student either earns an adjusted status of 100% or runs out of attempts on all problems which contribute to the adjusted status.

Set Options

These are the set wide options specific to this type of homework. Many of the other set wide options, such as due date, open date and answer date, will be present. If conditional release and/or reduced credit is enabled then these configuration options will also be present.

Restrict Problem Progression 

This option controls whether students are allowed to proceed to the next parent problem before finishing a problem. Here finishing means that that they either have an adjusted status of 100% on the problem, or they have run out attempts on that problem and all child problems which count for its score.

Email Notification 

When this option is enabled all instructors set to receive feedback emails will be sent a notification when a student finishes a problem, e.g. runs out of attempts on a problem, and all child problems which count for that problems score, and do not have an adjusted status of 100%. This is to let the instructor know that the review for that problem was unsuccessful.

Problem Options

These are options for individual problems. The usual problem variables, such as weight and max attempts, will be present as well.

Att. to Open Children 

This controls how many incorrect attempts are needed before the child problems for this problem become visible to a student. If this is set to 0 then child problems are always visible. If this is set to -1 or a number higher than max attempts then the child problems will become visible after a student runs out of attempts. If this is set to -1 and max attempts is set to unlimited then the child problems will never be open.

Counts For Parent 

This controls whether this problems score counts towards the score of its parent. In general the adjusted status of a problem is the larger of its status and the weighted average of the status of its child problems with this flag enabled.

Editing Just-In-Time Sets

In order to edit Just-In-Time sets you must use the Problem Set Detail 2 editor. This can be found by visiting the set, either on the Problem Set page or the Problem Set Detail page and clicking on the link in the left hand navigation pane which looks like --Set Name. (This is similar to how one accesses different versions of the PG Problem Editor.) Once you reach this page it should look something like below. (Note the link in the left hand pane.)

This portion of the page should look pretty familiar. You set parameters here as you normally would. Just like with Gateway sets if you select just-in-time as the set type you will need to save before the changes are registered. The main differences are how you order problems. The problem section of the page looks like this.

Notice the nesting in the problem list. This, together with the problem numbers, should indicate the child-parent relationship between the problems. Because just-in-time type sets can have so many problems you can simplify things by clicking the - icon to hide the children of a particular problem and you can click the + icon to show the children of a particular problem. This is very similar to how viewing a directory structure in list mode works. You can expand or collapse all of the child problems by using the buttons at the top.

In order to reorder problems you drag a problem by clicking on the arrow icon. When you move a problem you will be shown a little preview box indicating where the problem will go. You can change whether a problem is a child or not by dragging the problem slightly to the right or left. (When you do so the preview box should become indented.) If you delete a problem there may be gaps in your numbers. This is good if you delete something while a set is active (not recommended) but otherwise you can renumber the problems consecutively using the Renumber button. When focusing on an individual problem you will see something like this.

This is very similar to the previous problem set detail editor in that you use the text fields to set the problem parameters. The main difference is how rendering problems work. In this editor you click the picture icon to render a problem. If you click it again it will hide the problem. The method used to render the problem is determined by the drop down box. The other icons are to used to view the problem and to edit it, as usual. If you want to render all of the problems at the same time you can use the button at the top of the list. Be warned this will take up a lot of vertical space.

Managing Just-In-Time Sets

• Managing Just-In-Time sets is done using the Hmwk Sets Editor 2 page. You can do everything you would normally do with an assignment including assigning users, importing, exporting, changing visibility, etc... The one main difference is that importing and exporting Just-In-Time sets uses the new .def file format described in Set Definition Files. This is only supported by Hmwk Sets Editor 2, so if you use a different set editor it will fail.

• In order to override set parameters for a specific user you should visit the Problem Set Detail 2 page and click the Edit individual versions of ... link. This will take you to a page where you can select the user. Then you will be taken to a version of Problem Set Detail 2 which contains student specific information.

• Adding problems to Just-In-Time sets is done the same as it is for regular or gateway sets. You add problems using the library browser, or the problem editors, or the "add blank problems" field as you normally would. New problems are always added to the end of the set as a new top level problem, so you will most likely have to reorder them after you add them.

Gotchas

• When using the ProblemSetDetail2 page you still need to click save after you have made all your changes.
• If you change a Just-In-Time type set to another homework type all of the problems will remain but you will lose the nested structure of your set.
• You have to have an email address in WeBWorK for the Email Instructor feature to work.