UsingWW: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

Hello all,

I'm in the process of setting up a WW server and am hoping for your help on an installation issue I am having. I believe that apache is up and running, but when I visit the public IPv4 for the AWS, the connection times out.

I am using the instructions at https://webwork.maa.org/wiki/WeBWorK_2.15_Ubuntu_Server_20.04_LTS_Amazon_Machine_Image and am trying to start the server but something is preventing apache2 from serving up the pages. I think the apache2 server is running from the output of service apache2 status (included below). To my knowledge, inbound and outbound rules are set correctly: I'm allowing SSH through my home IP only, allow ports 8443 and 443 inbound, and allow ports 80 and 8080 outbound. This is all set within the AWS EC2 interface. (I have not adjusted iptables or any other firewall within Ubuntu.)

Notably, I have commented out the "ping" line within /etc/apache2/conf-enabled/webwork.conf (following the idea in post https://webwork.maa.org/moodle/mod/forum/discuss.php?d=4842) because Ubuntu let me know no packets were sent back. I have also tried setting the apache version as 2.4.41 in site.conf and I have also picked out three instances of aws.apizer.org in localOverrides.conf and replaced them with the public IPv4 address of the Amazon AWS instance. I have also set my email and the timezone in a few places, and as far as I know, there are no other edits to the .conf files. What could the issue be?   Besides these outputs, is there other information that would be helpful?

Output from apache2ctl restart:

root@server:/home/ubuntu# apache2ctl restart
webwork.apache2-config: WeBWorK server is starting
webwork.apache2-config: WeBWorK root directory set to /opt/webwork/webwork2 in webwork2/conf/webwork.apache2-config
webwork.apache2-config: The following locations and urls are set in webwork2/conf/site.conf
webwork.apache2-config: PG root directory set to /opt/webwork/pg
webwork.apache2-config: WeBWorK server userID is www-data
webwork.apache2-config: WeBWorK server groupID is wwdata
webwork.apache2-config: The webwork url on this site is http://localhost/webwork2
webwork.apache2-config: The webwork smtp server address is localhost
webwork.apache2-config:     The webwork smtp server port is
webwork.apache2-config:     The webwork smtp server protocol is 'not ssl'
WebworkSOAP::WSDL: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
WebworkSOAP::WSDL: rpc_url set to http://localhost/webwork2_rpc
WebworkWebservice: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
webwork.apache2-config: WeBWorK server is starting
webwork.apache2-config: WeBWorK root directory set to /opt/webwork/webwork2 in webwork2/conf/webwork.apache2-config
webwork.apache2-config: The following locations and urls are set in webwork2/conf/site.conf
webwork.apache2-config: PG root directory set to /opt/webwork/pg
webwork.apache2-config: WeBWorK server userID is www-data
webwork.apache2-config: WeBWorK server groupID is wwdata
webwork.apache2-config: The webwork url on this site is http://localhost/webwork2
webwork.apache2-config: The webwork smtp server address is localhost
webwork.apache2-config:     The webwork smtp server port is
webwork.apache2-config:     The webwork smtp server protocol is 'not ssl'
WebworkSOAP::WSDL: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
WebworkSOAP::WSDL: rpc_url set to http://localhost/webwork2_rpc
WebworkWebservice: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config

Output of /var/log/apache2/error.log after issuing apache2ctl restart (LogLevel is set to "trace1" most verbose setting)

[Mon Jan 18 12:25:22.631470 2021] [mpm_prefork:notice] [pid 834] AH00173: SIGHUP received. Attempting to restart
[Mon Jan 18 12:25:23.902654 2021] [watchdog:debug] [pid 834] mod_watchdog.c(452): AH010033: Watchdog: Running with WatchdogInterval 1000ms
[Mon Jan 18 12:25:23.902676 2021] [watchdog:debug] [pid 834] mod_watchdog.c(461): AH02974: Watchdog: found parent providers.
[Mon Jan 18 12:25:23.902681 2021] [watchdog:debug] [pid 834] mod_watchdog.c(507): AH02977: Watchdog: found child providers.
[Mon Jan 18 12:25:23.902685 2021] [watchdog:debug] [pid 834] mod_watchdog.c(515): AH02978: Watchdog: Looking for child (_default_).
[Mon Jan 18 12:25:23.902694 2021] [watchdog:debug] [pid 834] mod_watchdog.c(515): AH02978: Watchdog: Looking for child (_singleton_).
[Mon Jan 18 12:25:23.907511 2021] [mpm_prefork:notice] [pid 834] AH00163: Apache/2.4.41 (Ubuntu) mod_apreq2-20090110/2.8.0 mod_perl/2.0.11 Perl/v5.30.0 configured -- resuming normal operations
[Mon Jan 18 12:25:23.907522 2021] [mpm_prefork:info] [pid 834] AH00164: Server built: 2020-08-12T19:46:17
[Mon Jan 18 12:25:23.907532 2021] [core:notice] [pid 834] AH00094: Command line: '/usr/sbin/apache2'
[Mon Jan 18 12:25:23.907537 2021] [core:debug] [pid 834] log.c(1568): AH02639: Using SO_REUSEPORT: yes (1)
[Mon Jan 18 12:25:23.907541 2021] [mpm_prefork:debug] [pid 834] prefork.c(914): AH00165: Accept mutex: none (default: sysvsem)
[Mon Jan 18 12:25:23.911307 2021] [watchdog:debug] [pid 1003] mod_watchdog.c(566): AH02980: Watchdog: nothing configured?
[Mon Jan 18 12:25:23.911685 2021] [watchdog:debug] [pid 1004] mod_watchdog.c(566): AH02980: Watchdog: nothing configured?
[Mon Jan 18 12:25:23.912733 2021] [watchdog:debug] [pid 1005] mod_watchdog.c(566): AH02980: Watchdog: nothing configured?
[Mon Jan 18 12:25:23.913512 2021] [watchdog:debug] [pid 1006] mod_watchdog.c(566): AH02980: Watchdog: nothing configured?
[Mon Jan 18 12:25:23.913981 2021] [watchdog:debug] [pid 1007] mod_watchdog.c(566): AH02980: Watchdog: nothing configured?

Check Apache 2 status:

root@server:/home/ubuntu# service apache2 status
● apache2.service - The Apache HTTP Server
     Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
     Active: active (running) since Mon 2021-01-18 12:11:36 PST; 18min ago
       Docs: https://httpd.apache.org/docs/2.4/
    Process: 818 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
   Main PID: 834 (/usr/sbin/apach)
      Tasks: 6 (limit: 2368)
     Memory: 152.3M
     CGroup: /system.slice/apache2.service
             ├─ 834 /usr/sbin/apache2 -k start
             ├─1003 /usr/sbin/apache2 -k start
             ├─1004 /usr/sbin/apache2 -k start
             ├─1005 /usr/sbin/apache2 -k start
             ├─1006 /usr/sbin/apache2 -k start
             └─1007 /usr/sbin/apache2 -k start

Jan 18 12:11:36 ip-172-31-34-46 apachectl[832]: webwork.apache2-config: WeBWorK server userID is www-data
Jan 18 12:11:36 ip-172-31-34-46 apachectl[832]: webwork.apache2-config: WeBWorK server groupID is wwdata
Jan 18 12:11:36 ip-172-31-34-46 apachectl[832]: webwork.apache2-config: The webwork url on this site is http://localhost/webwork2
Jan 18 12:11:36 ip-172-31-34-46 apachectl[832]: webwork.apache2-config: The webwork smtp server address is localhost
Jan 18 12:11:36 ip-172-31-34-46 apachectl[832]: webwork.apache2-config:     The webwork smtp server port is
Jan 18 12:11:36 ip-172-31-34-46 apachectl[832]: webwork.apache2-config:     The webwork smtp server protocol is 'not ssl'
Jan 18 12:11:36 ip-172-31-34-46 apachectl[832]: WebworkSOAP::WSDL: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
Jan 18 12:11:36 ip-172-31-34-46 apachectl[832]: WebworkSOAP::WSDL: rpc_url set to http://localhost/webwork2_rpc
Jan 18 12:11:36 ip-172-31-34-46 apachectl[832]: WebworkWebservice: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
Jan 18 12:11:36 ip-172-31-34-46 systemd[1]: Started The Apache HTTP Server.

It looks like Apache has indeed started..

Thanks in advance!

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Danny Glin - Tuesday, 19 January 2021, 1:03 PM

Here are a couple of ways to check if it is a local firewall issue:

From the command line you can use curl (or wget) to check if a local user can connect:

curl http://localhost/webwork2

This should return the webwork landing page.

Another thing to try is to temporarily turn off the linux firewall. I believe that Ubuntu uses firewalld, so you should be able to run

systemctl stop firewalld

If this allows you to connect, then it indicates that firewalld is blocking traffic on those ports, and you will have to add rules to allow it.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Nathan Wallach - Tuesday, 19 January 2021, 1:59 PM

I think you need to set $server_root_url as explained in https://webwork.maa.org/wiki/Installation_Manual_for_2.15_on_Ubuntu_20.04_Server and not leave it as localhost.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Miguel-Angel Manrique - Tuesday, 19 January 2021, 4:41 PM

Great thank you for these suggestions!

The curl command did return a "Page generated at *date at *time* ..." WeBWorK message. It looks like apache could serve the pages all along! I also tried disabling firewalld as suggested, but it was not loaded. Curiously, the firewalld package was not installed (and IP tables had no rules set in it). I even tried removing all of the security rules from the AWS settings.

I also tried replacing localhost with my server's public IP address. Unfortunately the server still wasn't accessible.

I am going to try installing Ubuntu onto a fresh server, following the instructions https://webwork.maa.org/wiki/Installation_Manual_for_2.15_on_Ubuntu_20.04_Server

I am thinking I will leave the AWS instance installed and see about sorting through it at a later time.

Thank you for looking at my question.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Arnold Pizer - Tuesday, 19 January 2021, 5:28 PM

Hi,

Here are a few comments.

Re "I have also tried setting the apache version as 2.4.41 in site.conf" Look at https://webwork.maa.org/moodle/mod/forum/discuss.php?d=3155, specifically

Re: Problems starting webwork after package upgrade

by Nathan Wallach - Monday, 5 October 2020, 6:09 AM

At present, in WW 2.15 and apparently back to WW 2.11, the setting
$server_apache_version = '2.4';
in site.conf should work, but not forms like
$server_apache_version = '2.4.29';
with 3 numeric parts to the version number. This is due to a bug in the code adding support for this setting. Hopefully this will be fixed in WW 2.16, and maybe in a hotfix to WW 2.15.
See: https://github.com/openwebwork/webwork2/pull/1145

Also I would suggest you do a diff between you new conf files and the version contained in the ami to make sure your changes are correct and that no extraneous errors crept in. And finally double check your security rules.

Good lucck,

Arnie

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Miguel-Angel Manrique - Tuesday, 19 January 2021, 8:08 PM

Thank you all! I have gotten WW to fire up!

If it helps anyone in the future, this is what did it in the end:

Amazon AWS inbound rules

In Amazon EC2 management I got it by selecting "Anywhere" in the dropdown:

Amazon AWS inbound rules setting

Later I can try seeing if $server_apache_version set to 2.4.29 breaks the setup.

Thank you all again!

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by pb math - Sunday, 18 April 2021, 10:32 AM

I started another thread before seeing yours, I have a very similar issue but I can access WebWork it's just that every interaction takes the page about 2min to load, while just going to the ip address (without webwork) works as quick as i would expect.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Arnold Pizer - Sunday, 18 April 2021, 11:17 AM

Hi,

I haven't seen this even with very minimal resources assigned to the server but maybe it is a resource issue. I would try running "top" in a terminal window when using WeBWorK and looking a memory usage, etc. Also try
wwadmin@wwserver:~$ cd /opt/webwork/webwork2/logs/
wwadmin@wwserver:/opt/webwork/webwork2/logs$ tail -f timing.log
and do something with WeBWorK. This will tell you how much time WeBWorK operations are taking. Obviously they should be quick.

There are also probably AWS methods for determining server load, etc but I am not knowledgeable about them.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by pb math - Sunday, 18 April 2021, 11:29 AM

Thanks,
The system is not under any serious load, peaked at 35%ram and CPU was barely doing anything.
The timing logs all showed very quick times, eg0.015sec and so on.
It's almost as if I click, then there is just a pause for a mine or two then all of a sudden the page loads like it should. I did nothing other than follow the instructions for AWS.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Arnold Pizer - Sunday, 18 April 2021, 1:13 PM

I'm at a loss here and just reaching for things. I would try restarting apache

sudo service apache2 restart

and also looking at the apache error log while using WeBWorK

tail -f /var/log/apache2/error.log

Maybe even reboot the server.

Good luck and please report back when you find out what's going on.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by pb math - Sunday, 18 April 2021, 1:24 PM

So, restarting apache and restarting the server via aws had zero effect, had a search in the error logs like you recomended while clicking on things and see:
[Sun Apr 18 10:58:35.963632 2021] [core:notice] [pid 665] AH00094: Command line: '/usr/sbin/apache2'
[Sun Apr 18 13:17:37.690712 2021] [mpm_prefork:notice] [pid 665] AH00169: caught SIGTERM, shutting down
[Sun Apr 18 13:17:41.284952 2021] [mpm_prefork:notice] [pid 1336] AH00163: Apache/2.4.41 (Ubuntu) mod_apreq2-20090110/2.8.0 mod_perl/2.0.11 Perl/v5.30.0 configured -- resuming normal operations
[Sun Apr 18 13:17:41.285000 2021] [core:notice] [pid 1336] AH00094: Command line: '/usr/sbin/apache2'
[Sun Apr 18 13:17:45.289422 2021] [mpm_prefork:error] [pid 1336] AH00161: server reached MaxRequestWorkers setting, consider raising the MaxRequestWorkers setting

Clicking on things didn't issue any new errors, but then again everything takes minutes to do anything, I am literally the only person using it, don't even have a course created just taking forever to even click into course admin, so I can't imagine I am putting the server under any stress. However the error above talks about max workers ? Should I do something?

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Arnold Pizer - Sunday, 18 April 2021, 1:41 PM

I agree it's hard to understand why you are running out of workers. Look at

https://webwork.maa.org/wiki/Installation_Manual_for_2.15_on_Ubuntu_20.04_Server#Edit_mpm_prefork.conf

Not sure what would cause a SIGTERM but obviously something did. Rebooting the server can't hurt.

What is the response from

http://yourserver.yourschool.edu

http://yourserver.yourschool.edu:8080

http://yourserver.yourschool.edu:8080/webwork2_files

Quick or not?

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by pb math - Sunday, 18 April 2021, 2:13 PM

So I am just using the IP I don't have a name DNS yet, from my browser if I type in:

http://ip_add = super quick, I see the apache page
http://ip_add:8080 = super quick, I see the apache page
http://ip_add:8080/webwork2_files = super quick, I see webwork placeholder page, Exploring This page sits at the top level hddocs.....

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Arnold Pizer - Sunday, 18 April 2021, 2:28 PM

But
http://yourserver.yourschool.edu/webwork2
http://yourserver.yourschool.edu/webwork2/admin
http://yourserver.yourschool.edu/webwork2/mytestcourse

are slow? Did you look at MaxRequestWorkers?

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by pb math - Sunday, 18 April 2021, 2:36 PM

All of those are taking 2+min

I increased the MaxRequest Workers from 5 to 25, made zero difference, since have reverted it back to what it was, 5.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Arnold Pizer - Sunday, 18 April 2021, 3:33 PM

Here is one thing to try. In /opt/webwork/webwork2/lib/WeBWorK/Constants.pm set

$WeBWorK::Debug::Enabled = 1;

and look at the resulting log (in /opt/webwork/webwork2/logs)

Remember to set it back to 0 since it can get huge and use up all your disk space.

I assume you are still getting errors in the apache errror.log. It seems like apache is trying to do something, then timing out and finally processing the request. Not sure how to log what apache is trying to do. Googling may help.

Sorry I can't really help. Haven't seen this reported before.

Arnie

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by pb math - Sunday, 18 April 2021, 4:35 PM

Appreciate the help,
I tried that and nothing of note was written in the logs, as in nothing when I went to go the the /webwork2 page, all I seen was a few took .02 sec for so and so. But im still waiting 2min for the page to load.
Perhaps someone else reading might know.

One thing I do notice is, In AWS when I click on my public IP it says page cannot be found,(https://IP) but when I change this to HTTP it works (the apache page) Now looking inside of site.conf I see $ser_root_url='http://localhost' , I assume AWS is running https could this be causing an issue?

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Arnold Pizer - Sunday, 18 April 2021, 4:51 PM

No, https access is not set up (e.g. you need your own official certificates but there are also directions for using self signed ones). See https://webwork.maa.org/wiki/WeBWorK_2.15_Ubuntu_Server_20.04_LTS_Amazon_Machine_Image#Set_up_WeBWorK_to_use_SSL

Maybe https times out and it reverts to http.

Arnie

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by pb math - Monday, 19 April 2021, 5:58 AM

Thanks again Arnie, in case anyone else is reading , still no luck. Has anyone setup the AWS image and had everything just work ? Since other than security settings for http/ssh I did nothing other than 'launch' the instance.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Alex Jordan - Monday, 19 April 2021, 11:22 AM

Not that it should matter, but can you confirm that you are on the "master" branch of webwork2 and pg, which is presently version 2.15?

Is it possible to give a link (even with IP address) to the instance? (Also not much hope that would help but maybe I/we see something there as a clue.)

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by pb math - Monday, 19 April 2021, 11:39 AM

"Not that it should matter, but can you confirm that you are on the "master" branch of webwork2 and pg, which is presently version 2.15?" - sorry not 100% sure what you mean ? or how to check

So, today I terminated the AWS image, deleted everything and started fresh, following https://webwork.maa.org/wiki/Installing_from_WW2.15_Ubuntu20.04_Server_Virtual_Machine_Image

Same issue persists, the server is, http://ec2-3-133-102-106.us-east-2.compute.amazonaws.com

I'm happy to give anyone ssh access, as it is just a test case with nothing sensitive, let me know, since I am well out of my dept in terms of server stuff. Given that I have followed the instructions and taken the official image I cannot see how I have done anything wrong. I am based in Ireland, so I had to change my AWS default from here to Ohio, if that makes any difference, my security settings are SSH and HTTP added, with 0.0.0.0/0 as the IP, I have tried this with 'anywhere' too and it makes no difference. As you can see the apache page works fine, but webwork nope, all the settings are default so I am happy to let anyone play and see if they can shed some light.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by Danny Glin - Monday, 19 April 2021, 1:17 PM

A couple of observations:

The serving of the WeBWorK pages themselves is not the problem. If you fetch a page like http://ec2-3-133-102-106.us-east-2.compute.amazonaws.com/webwork2/myTestCourse/ using a tool such as wget or curl it is almost instant, so the problem isn't with WeBWorK processing pages.
Looking at the Browser Console in Firefox (under "Web Developer"), it shows the following:
Loading failed for the <script> with source “http://aws.apizer.org:8080/webwork2_files/mathjax/MathJax.js?config=TeX-MML-AM_HTMLorMML-full”.
It looks like your server is configured to get its javascript from aws.apizer.org, which appears to not be responding. My guess is that the page tries to load this javascript, and then times out. Once that dependency times out then it loads the rest of the page.

Most likely in your configuration you need to find the mention of aws.apizer.org, and change it to the name of your server. Since it is coded for port 8080 it is likely expecting lighttpd to be set up to serve static content, so if you set it to port 8080 then you will need to make sure that you have lighttpd running and configured properly.

Re: Ubuntu 20.04 Amazon AWS image: connection timeout, apache does not serve pages

by pb math - Monday, 19 April 2021, 3:16 PM

Hi Danny et al,
Solved!!!
And egg on my face! I read the instructions for installation and got to editing the config files, and it talks about emailing the admin etc none of which I wanted and on previous (iso) installs by the time I got this far all was done, of course as you pointed out replacing aws.aspizer.org is mentioned and the 'solution'
Total egg on my face sorry for wasting the communities time and thank you for everyone help!