WeBWorK Main Forum

LTI homework grades transfer issues

LTI homework grades transfer issues

by Lars Jensen -
Number of replies: 10

Dear Colleagues,

Our experimentation on the transfer of grades between Canvas and webwork shows that errors occur during the transfer when using the $LTIGradeMode = "homework" setting.

The experimentation we have done consists of switching back and forth between the "course" setting and the "homework" setting. The "course" setting seem to be more robust, so if I switch my setting from $LTIGradeMode = "homework" to $LTIGradeMode = "course" the correct homework average seems to show in Canvas. But if I switch in the other direction, back to $LTIGradeMode = "homework," the individual assignment grades in canvas don't generally match those in webwork. Transferred scores does seem to increase as more problems are completed, but the values are off, and I cannot make sense of the values that are transferred. 

My conclusion is that I think there is a bug in the grades  transfer under the $LTIGradeMode = "homework"  setting.

Has anyone else had issues with the robustness of the $LTIGradeMode = "homework" setting? Any insights?

Sincerely,
Lars.

In reply to Lars Jensen

Re: LTI homework grades transfer issues

by Alex Jordan -

In the past, I have observed issues that come down to decimal rounding. In a set/student where there is a discrepancy:

  1. How many points are possible in the problem set?
  2. What score does the student have for the set when viewing it in WeBWorK?
  3. What score does Canvas report the student having? Include details about how the Canvas graded item is set up (for how many points, scored as a percentage, etc.)
And with all experimentation, are you allowing enough opportunity for WeBWorK to send updated grades to Canvas since the last change?


In reply to Alex Jordan

Re: LTI homework grades transfer issues

by Lars Jensen -

Hi Alex,

This is worse than a rounding issue. In webwork I have 2 sets each of 16 points. But that shouldn't really matter since only the percentage is supposed to transfer.

The grades reported by webwork for two sets are 25% and 10%. In Canvas they report as 18 out of 100  and 15 out of 100. In canvas each assignments are set up as 100 point assignments.

Yes, I can see the grades are changing but to the wrong values. I also set $LTIMassUpdateInterval = 10 for the testing.

My experiments are easy to repeat (if you have Canvas available).

Sincerely,
Lars.

In reply to Lars Jensen

Re: LTI homework grades transfer issues

by Alex Jordan -
What is the problem score breakdown in each set? Especially the one that leads to a score of 10% on a 16-pt problem set? Understanding that may (or may not) shed light on possible rounding errors and misuse of WW data by Canvas.

If you change the Canvas sets to be out of 16 pts to match WW, does that change anything? I haven't used Canvas firsthand, and I'm not familiar with the configuration options for a graded item. But if it's possible, some experimentation with different schemes might shed light. It may be that in some schemes the data that WW passes back is not used as intended.

Here is a related thread, but your discrepancies are larger.
https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4934&parent=14961

In reply to Alex Jordan

Re: LTI homework grades transfer issues

by Lars Jensen -
Hi Alex,

Now, after a delay, both sets in canvas report 18 out of 100 (which happens to be the same value reported in the webwork "Homework Totals" column). Both of these two set scores are wrong, of course.

The problem are scored like this in webwork:
set 1: "100" on problems 1,2,3,5, and "0" on problems 6 and 7 and "." on the rest of the 16.
set 2: "100" on problems 1. "67" on problem 2, and "0" on problems 3, 5, 6 and "." on the rest of the 16.

If on canvas I grade the problems out of a point score of 16, Canvas then reports 2.88 out of 16 for the first set, and 18 out of 16 for the second set

Sincerely,
Lars.
In reply to Lars Jensen

Re: LTI homework grades transfer issues

by Lars Jensen -

...a strange behavior I just noticed is that if there are two webwork sets, and the student clicks on one in Canvas, but after having been logged into webwork, decides to work on the other set, then the grade is not transferred instantly, like it is the case if the students had worked on the first set. So the transfer of the grade depends on which webwork set the students decides to work on....

In reply to Lars Jensen

Re: LTI homework grades transfer issues

by Alex Jordan -
If you are using 'homework' mode, then the link between Canvas and a particular problem set is not established until the first time the student clicks the set-specific ELT link in Canvas. This is a problem with using 'homework' mode. Your student needs to go back to Canvas, click the ELT link that specifically takes them to the other set before grade passback will start.

Once that connection is made, then grades will update after that no matter how the student reached the problem set. Although, I'm unsure how frequently.

I have a plan to deal with this issue. The outline is that if a student clicks a set-specific ELT link while using 'homework' mode, they will reach a WeBWorK page without any navigation options. (Ideally in a frame in the LMS.) So now to reach the second homework set, the student would pretty much *have* to go through the LMS. This is just an idea for now, no implementation work yet.
In reply to Lars Jensen

Re: LTI homework grades transfer issues

by Alex Jordan -

> both sets in canvas report 18 out of 100 (which happens to be the same value reported in the webwork "Homework Totals" column).

OK, this suggests that the overall homework score is reaching Canvas and being used for the score on each graded item in Canvas. Earlier, with the 18 and 15, perhaps an update of the 15 score had just not yet  happened.

You reported switching back and forth between 'course' and 'homework' $LTIGradeMode. When you do this, are you changing the URLs for the External Learning Tool links? Say you were using 'homework' mode. Then in Canvas that requires setting up a separate ELT for each homework set, each one using a different URL to the WW course that goes straight to that homework set. And then in the Canvas course, inserting one link for each problem set, each link using one of the ELTs. (So it's a lot of setup work.)

Before we continue, I just want to confirm that you understand all these things about how to use 'homework' mode. The fact that you are getting the overall WeBWorK scores showing up in the Canvas grades suggests maybe something is not set up right on that front.

In reply to Alex Jordan

Re: LTI homework grades transfer issues

by Lars Jensen -
Hi Alex,

Yes, of course. I change the links, adding the setname to the end of the URL entered into the external tool, and each canvas assignment points directly to a different webwork set.

When I start from scratch with a new webwork student and a new canvas course, things seems to work mostly fine in the "homework" mode. It is first when I begin switching back and forth between the two modes that the canvas "homework" grades gets messed up.

Sincerely,
Lars.
In reply to Lars Jensen

Re: LTI homework grades transfer issues

by Alex Jordan -
I don't claim to fully understand WeBWorK and LTI, but I wonder if you can't be switching back and forth like this. Because the first time a student clicks a link from the LMS, that establishes something in the LMS for how it receives and handles whatever data WeBWorK will be sending it. Once it is established, I wouldn't be surprised if switching modes can leave the wrong things in place in the LMS. And you find yourself seeing the overall course grade where you are expecting to see a set-specific grade.

Was this all happening in a test course? If so, you might try:
* put it in 'homework' mode if it is not already
* create a new Canvas course
* create new ELT objects for those two problem sets, and put links in the course that use them
* while logged in to the Canvas course as the same user who has 25% and 10% on those two problem sets, click each link to a WW set. This will initialize the connection between that user in Canvas and each WW set.

And then do you see good scores in Canvas? If so, then it confirms things work when you don't start switching back and forth.
In reply to Alex Jordan

Re: LTI homework grades transfer issues

by Lars Jensen -

Hi Alex,

After more experimentation, I don't think it is the switching of the modes that messes things up in the "homework" mode as much as what I described above, namely that if the student clicks one set in Canvas, and then, after having been logged into webwork, decides to work on a different set, then the grade is not reported to webwork. Nothing is transferred, and grades gets messed up in canvas.

Lars.