WeBWorK Main Forum

Quiz timeout not preventing further graded submissions

Quiz timeout not preventing further graded submissions

by Alex Jordan -
Number of replies: 14

I never really used Gateway quizzes until recently as part of rearranging things for the virus response. We are seeing an issue with quizzes, and I wonder if it is a real issue or just me not understanding something about quiz settings.

In March, I set up a quiz. It had a 120 minute time limit and a close date of March 18. Just now (a month later) I logged in as one of the students (who never changed their original password). I went into the quiz (test/version 1) and went to a question where I had verified they had it wrong. The message was there at the top as expected about 0 minutes remaining. I entered a correct answer, scrolled down, and the "Grade test" button was showing. This itself came as a surprise. I clicked it. Then after logging in as instructor, I saw that here quiz score had indeed gone up by one (in Scoring Tools, Student Progress, and the page that shows points for each problem for this student).

So the time limit is seemingly ineffective. I was prompted to do this by a colleague who built a simpler test quiz and discovered the same thing happening.

Is anyone else able to reproduce this? Either like my colleague, by conducting a controlled experiment. Or like I did, logging in to an old course as a student with an incomplete quiz and seeing what happens? (It may be important to actually be logged in as a student.)

We are using version 2.15. If it is something about my quiz settings, the set details page has the following settings:

Opens: 03/17/2020 at 12:00am
Reduced Scoring Date: 03/18/2020 at 11:59pm
Closes: 03/18/2020 at 11:59pm
Answers Available: 03/18/2028 at 11:59pm
Visible to Students: Yes
Reduced Scoring Enabled: No
Hide Hints from Students: Yes
Assignment Type: gateway/quiz
Test Time Limit: 120
Cap Test Time at Set Close Date: No
Number of Graded Submissions per Test: 0 (infinity)
Time Interval for New Test Versions: 0 (infinite)
Number of Tests per Time Interval: 1
Order Problems Randomly: No
Number of Problems per Page: 1
Show Scores on Finished Assignments: No
Show Problems on Finished Tests: No

In reply to Alex Jordan

Re: Quiz timeout not preventing further graded submissions

by Andrew Dabrowski -
See also
https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4751
There Danny Glin said that quizzes haven't been well tested. I have abandoned them I just use regular assignments.
In reply to Andrew Dabrowski

Re: Quiz timeout not preventing further graded submissions

by Danny Glin -
There are certain aspects of gateway quizzes that haven't been well tested. With that being said we've recently run several large-scale exams (>600 students) using the WeBWorK quiz mode with very few complications. Of course some of that is based on lessons learned from previous experience.

I've never encountered what Alex describes, but as Glenn mentions it may have to do with allowing unlimited attempts, which is something we've never done.
In reply to Alex Jordan

Re: Quiz timeout not preventing further graded submissions

by Glenn Rice -

I suspect this has something to do with the setting "Number of Graded Submissions per Test" that you have set to 0 (which means unlimited).  This allows the student to change answers and grade the quiz again.  Set that to 1, and this won't happen.

I agree that submissions should not be permitted after the test time limit even with this setting though.  If that setting is set to 1 (or probably even a finite number), then submissions aren't permitted after the test time limit.

I may be wrong though, and something else might be going on.

I will try to do some testing and look at the code to see what is going on later.

In reply to Glenn Rice

Re: Quiz timeout not preventing further graded submissions

by Alex Jordan -

I'd come to the same place. That setting is something the faculty here largely want to use in order to avoid students who accidentally hit the Grade button before they were really finished. It's more of a problem with our student population than it should be.

It is true that if I change that setting to 1, then log in as the student, then there is no Grade button. Of course, the reason for that probably has nothing to do with the timer, but rather that a submission attempt has already been recorded.

I've been looking into the code in Gateway.pm that provides the Grade button. You may have more success debugging than I am having.

I do wonder if it could be something specific to our system. So if anyone on another installation can verify this happening, that is useful information.

In reply to Alex Jordan

Re: Quiz timeout not preventing further graded submissions

by Glenn Rice -
I just did some testing on this. If I set the "Number of Graded Submissions per Test" to 1, then even if the student never hits "Grade Test", after the time limit is up (and the gateway grace period, that defaults to 120 seconds, is over) the exam can no longer be graded.

If that setting is set to more than 1 (or to 0), however, the student can continue to hit "Grade Test" until all submissions are used up, even after the time limit is up.
In reply to Alex Jordan

Re: Quiz timeout not preventing further graded submissions

by Alex Jordan -
I experimented with changing the settings on Number of Graded Submissions per Test. I changed this setting for my old student, changing it in the specific "test 1" for her, not the "parent" quiz set.

When I changed it from 0 to 1000, no change in behavior. That is, I see a clock that says I have 0 minutes remaining, I see the quesitons, and I have access to the Grade button.

When I changed it down to 1, then there is a change. No clock. Questions are not visible. There is no grade button.

I change it back to 0 and the bad behavior returns.

Given what happened with 1000, it's not exactly about using 0 as the setting. It's happening as long as the student has attempts remaining.
In reply to Alex Jordan

Re: Quiz timeout not preventing further graded submissions

by Alex Jordan -
My colleague tried another experiment. Similar to mine, he opened a quiz from an old course. Except in his case, it was an old course from Fall. And of importance here: we upgraded to 2.15 over winter break. In the Fall course, the quiz did not initially exhibit the bad behavior. Except then he unassigned the quiz from a student, then reassigned it, then changed its date settings and gave it a 5-minute duration, then logged in as the student and took the quiz. After time expired, he was still able to submit answers. In other words, the bad behavior had returned.

To me this is suggesting that there is something off with the database where the quiz settings are stored. Even though we are using 2.15, the Fall course uses database entries that predate the upgrade. And there was no bad behavior. Only after reassigning the quiz (and rewriting to the database) does the issue reappear.
In reply to Alex Jordan

Re: Quiz timeout not preventing further graded submissions

by Glenn Rice -
It is not just as long as the student has attempts remaining. As I observed in my comment above, when the setting is set to 1, then if a quiz is started and the student never clicks "Grade Test", once the time is up, no submissions can be made. So the 1 submission is still available, but it still shuts down and there is no clock, no grade button, and questions are not visible.
In reply to Glenn Rice

Re: Quiz timeout not preventing further graded submissions

by Alex Jordan -

Sorry, our posts crossed. (The forum waits 30 minutes before sending out a post.)

More experiments, delving into the code, and inserting some debugging messages, and some things clear up while others become murkier. The can_recordAnswers subroutine checks if the most recent submit time is in between the open date and the close date (with a grace period added to the close date). If you never submit at all, there is no most recent submit time, and that leads to what you found, with no access to submit answers. Good.

However if there was a most recent submit time back when the set was still legally open to the student, then that time (which was a month ago in my original posted example) is used to determine can_recordAnswers. And I find myself logged in as the student, able to submit an answer from something that closed a month ago. And I can do so repeatedly as long as I don't navigate away from the quiz page. Yet if I leave the quiz and go straight back in, then everything is now unavailable. Because this time loading the page, the most recent submission was five minutes ago, well outside the legal window.

There are still several things I do not understand, but this business with the most recent submit time is a clue. And it may be that my colleagues experiment misled him, because the student he tested with had tried to submit already past the legal submission window.


In reply to Alex Jordan

Re: Quiz timeout not preventing further graded submissions

by Glenn Rice -
The can_recordAnswers method uses the version_last_attempt_time for the submission time if it is set. The reason it does this, according to the comments preceding the method, is for proctored gateway exams. Once the student hits "Grade Test" then the proctor may take some time to get there to enter the proctor username and password. So the student's submission time is used to determine when the quiz was really submitted. However, it seems that it is not taking into account quizzes that are not proctored. It seems that it should check to see if this is a proctored quiz, and in that case use the version_last_attempt_time, and otherwise use the current time.
In reply to Glenn Rice

Re: Quiz timeout not preventing further graded submissions

by Alex Jordan -

Thanks Glenn, you got it. I tested and it works as expected.

In reply to Alex Jordan

Re: Quiz timeout not preventing further graded submissions

by Julie Tremblay -

Hi, i’m new to this forum, but i signed in because i was looking for a solution to the same problem you described. Can you elaborate with the modifications you did in the code? I think I succeeded but i’m not sure if I did the right thing.

I changed this line in the can_recordAnswer module:

my $submitTime = ( defined($Set->version_last_attempt_time()) && 

         $Set->version_last_attempt_time()  
         && $Set->{'assignment_type'} eq 'proctored_gateway' ) ? 
 
         $Set->version_last_attempt_time() : $timeNow;

Thank you for your insight!



In reply to Julie Tremblay

Re: Quiz timeout not preventing further graded submissions

by Glenn Rice -
That is pretty much it.  I also made the same change in can_checkAnswers.  But the more important place for this issue is in the can_recordAnswer subroutine.