This is a great question. I think there are two issues with the load that gateway/quiz assignments impose on the system. The first is that when a student starts an assignment, WeBWorK has to create a new version of the assignment for them. I'm not sure why this seems to be a heavy lift, but it appears to be. This could be resolved by pre-creating that the first version of the assignment fo the student when the global set is assigned to the student. The only slightly tricky part of this would be to then fork the creation of the next version when the first version is started, so that it can proceed in the background when the server has processor time to deal with it.
The second issue, of processing all of the problems on the quiz when it's submitted, is one that depends a little bit on the set up of the quiz (e.g., I think that if the quiz has multiple pages only those on the current page are processed with a preview---but all will be processed on submit). I don't have a sense of the degree to which this load is a significant issue.
Gavin
Given that problems are typically customized per student with random parameters - and that the "seed" range is pretty large, I suspect pre-computing and storing cached "rendered" problems would be both CPU and storage intensive, and it would not help once answers are being submitted. I suspect that load peaks occur both at the start of a quiz (many students opening the quiz in a short time period) and at the end (many submissions very close to to the deadline).
I think that the issue is the current "single build in" PG renderer per Apache process, as rendering each PG file takes time, and causes the memory ballooning issue which the current generation of the WW server software experiences.
I think that the goal should be to improve WW so that it can easily provide more or less "power" without requiring provisioning a fixed large amount of resources based on expected peak loads. Anyone running in the cloud or with access to a Kubernetes cluster or the like would hopefully be able to then provision for "typical" demand as a standard, but either schedule or "auto-scale" for higher demand when necessary.
I think that the best long-term direction will be to separate the "PG renderer" out of the Apache process, using the new "standalone render" or something similar. Hopefully that will allow somewhat parallel processing of multi-problem rendering requests - or at least provide a means to better control the memory issues which probably is the bigger problem than the CPU resource issue. (Less memory ballooning would allow more Apache processes to run, so many more students could be served in parallel.)
This is something which is not likely to be ready for WW 2.16, but could be a major goal for development in the near future.
I think that the issue is the current "single build in" PG renderer per Apache process, as rendering each PG file takes time, and causes the memory ballooning issue which the current generation of the WW server software experiences.
I think that the goal should be to improve WW so that it can easily provide more or less "power" without requiring provisioning a fixed large amount of resources based on expected peak loads. Anyone running in the cloud or with access to a Kubernetes cluster or the like would hopefully be able to then provision for "typical" demand as a standard, but either schedule or "auto-scale" for higher demand when necessary.
I think that the best long-term direction will be to separate the "PG renderer" out of the Apache process, using the new "standalone render" or something similar. Hopefully that will allow somewhat parallel processing of multi-problem rendering requests - or at least provide a means to better control the memory issues which probably is the bigger problem than the CPU resource issue. (Less memory ballooning would allow more Apache processes to run, so many more students could be served in parallel.)
This is something which is not likely to be ready for WW 2.16, but could be a major goal for development in the near future.
---
Posted also to the GitHub issue requesting feature requests for Gateway quizzes: https://github.com/openwebwork/webwork2/issues/1156 with a reference to this forum discussion.