WeBWorK Main Forum

"Unable to retrieve your permissions" error

"Unable to retrieve your permissions" error

by David Brodbeck -
Number of replies: 16

A few months ago we migrated to WebWork 2.18, on Ubuntu 22 LTS. This was a clean install -- we didn't attempt to copy the old database, we just imported any needed courses as tgz and upgraded them. (None of these were used directly, they were just templates for new courses.) We're using Canvas as our LMS, LTI with grade passback.

The system worked fine through the summer term, but now that it's under heavier load I'm getting this error frequently in webwork2.log, and students are complaining they can't submit answers:

Unable to retrieve your permissions, perhaps due to a collision between your request and that of another user (or possibly an unfinished request of yours). Please press the BACK button on your browser and try again. at /opt/webwork/webwork2/lib/WeBWorK/ContentGenerator.pm line 116.

Not every request fails, but once it starts failing for a student it tends to keep failing. I did a web search and saw some references to this problem from 2020 or so, but nothing that provided a fix. Any ideas on how to troubleshoot this? How can I reduce these request collisions?


In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by David Brodbeck -
More info: I think it's related to this database error:
mariadbd[71496]: 2024-10-04 17:25:28 102011 [Warning] Aborted connection 102011 to db: 'webwork' user: 'webworkWrite' host: 'localhost' (Got an error reading communication packets)

But I'm not sure why the connections are aborting.

The server has plenty of memory, so it isn't an OOM killer situation. CPU-wise it seems pretty lightly loaded.
In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by David Brodbeck -
This seems to be a red herring. I'm seeing "unable to retrieve permissions" errors that aren't correlated with db errors.

I tried increasing the mojolicious worker timeouts, but I'm still seeing the errors.

EDIT: Don't think this is timeout-related -- the timing log shows very short runtimes (on the order of 50 ms) for transactions that fail with this error.
In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by Alex Jordan -

At my institutional server, in the mariadb logs, I have the same kinds of "Aborted connection" errors logged. There are dozens of these per hour, up to and including today.

2024-10-11 14:47:14 1094266 [Warning] Aborted connection 1094266 to db: 'REDACTED' user: 'REDACTED' host: 'localhost' (Got an error reading communication packets)

I've not had any complaints from users though about students are complaining they can't submit answers. And I've not had reports of "Unable to retrieve your permissions...".  In webwork2.log, I did find that error, but the last instance was in April 2024. But also, I did not have debugging enabled in webwork2.mojolicious.yml. I just turned it on. I have new entries like the above in the mariadb error log, but still no new entries about this in webwork2.log.

Did you correlate the times of your mariadb error log entries with times of this issue in webwork2.log?


In reply to Alex Jordan

Re: "Unable to retrieve your permissions" error

by David Brodbeck -
There were some correlations, but a lot of "Unable to retrieve" errors didn't correlate, so I concluded the aborted connections were unrelated.

I also checked to make sure I'm not hitting the mariadb connection limit. It's set to 600 and Max_used_connections is 243, so it would seem not.
In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by David Brodbeck -
Things I've tried so far:
- Checking the mariadb tables -- no problems.
- Reducing accepts in the mojolicious settings.
- Increasing accepts
- Increasing graceful_timeout
- Increasing inactivity_timeout
- Increasing heartbeat_timeout (Logically speaking, it does not seem to be a timeout issue, since the transactions that show this error are very short.)
- Reducing workers and spares (on the theory it's a race condition and fewer processes competing for the DB will help)
- Disabling antivirus endpoint protection on the server

Nothing seems to make a difference, I'm still getting "unable to retrieve permissions" errors. This is becoming an absolute fiasco. It's happened over 4000 times in the last week, affecting over 600 different students. This same server seemed to run fine under lighter load during summer term. I'm running out of knobs I can try tweaking, so I'm hoping for some suggestions.
In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by David Brodbeck -
Bump. Still getting complaints about this. These errors do not show up in the debug log, only the webwork2.log, which makes it hard to figure out what the cause could be. Has anyone else dealt with this problem? Is there any way to troubleshoot it further? I'm not sure what to do since downgrading back to 2.16 isn't going to work with the new database.
In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by Glenn Rice -

I am sorry that we haven't responded to this.  I think that none of us have really encountered this issue.  My best guess is that there is some sort of database corruption.  The error message you are getting would only happen if the WeBWorK::Authz module is unable to retrieve the permission level for the user from the database.

Can you run "git status" from the /opt/webwork/webwork2 directory and ensure that there aren't any local modifications?

We will probably need more information on your setup as well to be able to assist you.  How are you serving webwork2?  Are you serving directly, or are you proxying in some way?  Is there a load balancer involved?


In reply to Glenn Rice

Re: "Unable to retrieve your permissions" error

by David Brodbeck -

Yeah, there was one other thread about it in 2020, but it petered out without a resolution:

https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4958


There is only one modification, a cherry-picked change from a later branch to allow Certbot to request certificates:

ubuntu@ip-172-31-45-160:/opt/webwork/webwork2$ git status
On branch main
Your branch is up to date with 'origin/main'.
Changes not staged for commit:
  (use "git add <file>..." to update what will be committed)
  (use "git restore <file>..." to discard changes in working directory)
modified:   lib/Mojolicious/WeBWorK.pm
no changes added to commit (use "git add" and/or "git commit -a")
ubuntu@ip-172-31-45-160:/opt/webwork/webwork2$ git diff
diff --git a/lib/Mojolicious/WeBWorK.pm b/lib/Mojolicious/WeBWorK.pm
index 5d5441286..bb5effc8b 100644
--- a/lib/Mojolicious/WeBWorK.pm
+++ b/lib/Mojolicious/WeBWorK.pm
@@ -240,6 +240,16 @@ sub startup ($app) {
                                        . 'It must consist entirely of digits.  SOAP endpoints NOT enabled.');
                }
        }
+       # Letsencrypt renewal route. Cherrypicked from https://github.com/openwebwork/webwork2/pull/2321/commits/2bd380a5906ac63537612acc2bfa78268312f240
+       if ($config->{enable_certbot_webroot_routes}) {
+               $r->any(
+                       "/.well-known/*static" => sub ($c) {
+                               my $file = "$ce->{webworkDirs}{tmp}/.well-known/" . $c->stash('static');
+                               return $c->reply->file($file) if -r $file;
+                               return $c->render(data => 'File not found', status => 404);
+                       }
+               );
+       }
        # Note that these routes must come last to support the case that $webwork_url is '/'.

There's no load balancer. I'm serving it directly via hypnotoad/mojolicious, as described in the setup instructions. We've never used a proxy or load balancer; before this version we served webwork via apache24 and mod_perl.


In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by Glenn Rice -

Those changes shouldn't cause a problem.

What version of WeBWorK were the courses that you unarchived created on?  Depending on the version, there may be some UTF-8 issues.  You could try the upgrade-database-to-utf8mb4.pl script.  If you run the script with no arguments it will show the available options.  The script will make a backup of the database before making any changes.

In reply to Glenn Rice

Re: "Unable to retrieve your permissions" error

by David Brodbeck -
2.16. I did run "upgrade course" on each unarchived course before using it as a template for a new course.

I've now run "upgrade-database-to-utf8mb4.pl --all", I'll see if it makes any difference.
In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by David Brodbeck -
Didn't help, I'm still getting the error.

Could this be related to LTI authentication? We use it for all students (but not for instructors.)
In reply to Glenn Rice

Re: "Unable to retrieve your permissions" error

by David Brodbeck -
Still getting complaints about this. Some students are getting 0s because they can't access their homework and I'm under a lot of pressure to fix it. Some people with the problem report they can successfully log in on iOS but not on their desktop, but not everyone has an iOS device, so it's not an acceptable workaround.
In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by Danny Glin -

The only thing I can add is that I'm seeing both of these errors (the "unable to retrieve permissions" and the database aborted connections) on my production server, though they are not that frequent, and so far I haven't heard of any student that has not been able to log in (I suspect that in my case when a student runs into such an error they refresh the page and are able to log in, so they don't bother to report it).

You've already tried most of the things that I would suggest based on the suspicion that this has to do with collisions of database requests.  The one thing that wasn't mentioned in your previous post was confirming that clients is set to 1.  I suspect that you left that alone since the documentation explicitly says not to change it.

The one thing that I will mention from my system is that I have been seeing processed being killed by the out-of-memory killer because they are taking up too much RAM.  Increasing "accepts" in the config will exacerbate this, as the memory used by an individual process will increase with each new connection that it accepts.  It's possible that this is completely unrelated, but it's another thing that has been happening on my server.

In reply to Danny Glin

Re: "Unable to retrieve your permissions" error

by David Brodbeck -

Yes, clients is still set to 1.

I double checked the system logs for any OOM-killer activations, and didn't find any. The system is well-provisioned (64 GB) because I sized it with 2.16 in mind, and 2.18 uses less RAM.

I did see some segfaults:

[Sat Oct 19 05:35:35 2024] traps: /opt/webwork/we[2491281] general protection fault ip:7fa055e6c3fe sp:7ffd3f6d5740 error:0 in libc.so.6[7fa055def000+195000]

While somewhat concerning I don't think this is related, because the timestamps don't match the errors in webwork2.log.

The complaint I'm hearing from students is they either can't view problems or can't submit answers, and get that "unable to retrieve permissions" error instead. If they try to use their back button as instructed they get a Webwork login page, which of course doesn't work because we have LTI authentication for students.

This seems to be mainly affecting our largest classes (600-700 students), but I can't tell if that's because it's related to load or if it's because there are just more students in those classes and so it's more likely one of them will encounter it.

We occasionally got this with 2.16 but for some reason it's exploded in frequency under 2.18.

In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by David Brodbeck -

So, I think I've been barking up the wrong tree.

I was aware that I was getting complaints, and that the logs showed this error occurring in a lot of classes.

However, it wasn't until today that I made the connection that all my complaints are from one specific course, even though it isn't even the one that's seeing this error the most often. I got in touch with the instructor from one of the other courses that's logging these errors, and he said he wasn't aware of any problems.

So, I'm going to stop pursuing this as a technical issue for now and see if I can figure out if there's something else going on with either how the instructor is linking to the problems, or how his students are reacting to the error. Sorry to lead everyone on a bit of a wild goose chase.

In reply to David Brodbeck

Re: "Unable to retrieve your permissions" error

by David Brodbeck -
OK, to close the loop on this:

The ultimate problem was the prof was not checking "Load This Tool In A New Tab" in Canvas. Canvas was loading Webwork in an iframe. This didn't *cause* the "unable to retrieve your permissions" error, but it did mean when students got the error and followed the instructions to click the BACK button, they would be unable to re-authenticate properly.

The prof changed their Canvas problem set links and students are now able to successfully complete their homework. The issue with needing to open in a new window was actually known and documented, but no one had connected it with this particular problem before.

As far as I can tell there's nothing different in the behavior of 2.18 vs. 2.16 here; it just seemed that way because of the timing of our upgrade.

Thanks for your patience.