Installation

Fresh WW Installation - 404 when attempting to load site

Fresh WW Installation - 404 when attempting to load site

by Tony Box -
Number of replies: 19
I've got a new installation of WW 2.13 that I've manually set up. I'm running it an Amazon Linux AMI.

I've got all perl (5.16.3) and Apache (Apache 2.2.34) modules installed properly and have all directories and permissions set correctly (at least according to the instructions and in comparison to my currently working WW server of an older version).

Apache starts just fine and and loads all of the perl modules:

Starting httpd: webwork.apache2-config: webwork_server_admin_email for reporting bugs has been set to atsi@up.edu in site.conf
webwork.apache2-config: WeBWorK server is starting
webwork.apache2-config: WeBWorK root directory set to /opt/webwork/webwork2 in webwork2/conf/webwork.apache2-config
webwork.apache2-config: The following locations and urls are set in webwork2/conf/site.conf
webwork.apache2-config: PG root directory set to /opt/webwork/pg
webwork.apache2-config: WeBWorK server userID is apache
webwork.apache2-config: WeBWorK server groupID is apache
webwork.apache2-config: The webwork url on this site is https://webwork.atsi.cool/webwork2
webwork.apache2-config: The webwork smtp server address is
webwork.apache2-config: The webwork smtp server port is
webwork.apache2-config: The webwork smtp server protocol is 'not ssl'
WebworkSOAP::WSDL: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
WebworkSOAP::WSDL: rpc_url set to https://webwork.atsi.cool/webwork2_rpc
WebworkWebservice: webwork_directory set to /opt/webwork/webwork2 via $WeBWorK::Constants::WEBWORK_DIRECTORY set in webwork.apache2-config
Despite no errors, I cannot load the webwork page. If I attempt to go to https://webwork.atsi.cool/webwork2 I get a 404 error. I can see the requests hitting the apache server by watching the access logs so that leads me to believe there's something wrong with the apache redirects.

I've got a symlink set in /etc/httpd/conf.d/webwork.conf -> /opt/webwork/webwork2/conf/webwork.apache2-config.conf (a copy of the .dist file).

Is there something super simple I'm missing? There seems to be a pretty large gap somewhere which makes me think it's some small thing I missed but I can't for the life of me figure out why it isn't working.

Thanks!
In reply to Tony Box

Re: Fresh WW Installation - 404 when attempting to load site

by Michael Gage -
You are reaching a copy of webwork.apache2-config since all of those report messages come from that file.

Do you see anything in the apache error log?

Do you have the admin course set up? (That's the next step and you won't be able to use webwork until it's set up. ) I don't expect a 404 error even if the admin course has not been set up.

Is there a reason you are using apache 2.2.x instead of apache 2.4.x?

That's all I've got off the top of my head. I don't immediately see errors in urls.
You are using https consistently in the urls?

The redirect occurs in apache2-config.conf so you may be able to insert a print statement there that will leave a message in the apache log about where things are going astray.

-- Mike

In reply to Michael Gage

Re: Fresh WW Installation - 404 when attempting to load site

by Tony Box -
Hi Mike,

I stuck with Apache 2.2 because I ran into issues installing libapreq2 and perl-libapreq2 with Apache 2.4. Neither of them seem to install correctly with them having an Apache 2.2 dependency.

I'm actually trying to figure out if I can get them installed right now with 2.4. Yum and CPAN won't install them. Attempting to manually install doesn't seem to work either as it reports the Apache binary is incorrect.

I'll try to do some print debugging in the apache2-config.conf to see if I can find where it isn't getting to.
In reply to Tony Box

Re: Fresh WW Installation - 404 when attempting to load site

by Danny Glin -
On my CentOS 7 system yum had no problems installing libapreq2 and perl-libapreq2 with apache 2.4, so it is possible to do with yum depending on the system.

A couple more things to check:
Make sure mod-perl is actually loading with apache. You can do this by enabling perl-status in your apache config, then going to https://yourserver/perl-status.

If your system uses SElinux, check that SElinux isn't somehow blocking things. SElinux often supersedes file permissions when it comes to security.
In reply to Danny Glin

Re: Fresh WW Installation - 404 when attempting to load site

by Tony Box -
Thanks for the info Danny. I'm going to hold off on Apache 2.4 now since the Amazon AMI's yum repo's apparently don't support it and libapreq2 :(

I went to the perl-status page and confirmed mod_perl is loaded properly. SElinux is not running on this server.
In reply to Michael Gage

Re: Fresh WW Installation - 404 when attempting to load site

by Tony Box -
More info:

I have HTTPS set as the $server_root_url in site.conf -- I think that's the only place, yeah?

I tried printing stuff in the apache2-config file but it doesn't seem to help too much. All of the variables it populates seem correct (as you can see from my previous post). For the record I'm using mod_perl 2.0.7

One interesting thing, if I try to go to https://webwork.atsi.cool/webwork2_rpc I get a 400 response from the server instead of a 404.

Man, so annoying... especially since I can see traffic hitting the apache logs, it just isn't redirecting it properly.
In reply to Tony Box

Re: Fresh WW Installation - 404 when attempting to load site

by Danny Glin -
A couple more thoughts:

Are you using virtual hosts? If so, the configuration would have to be bound to the correct vhost.

I think you mentioned that your web server(s) were behind a load balancer. Have you tried connecting directly to the web server (if that's not already what you're doing)?
In reply to Danny Glin

Re: Fresh WW Installation - 404 when attempting to load site

by Michael Gage -
The snippet that defines the handoff to webwork (in webwork.apache2-config) is 

# Define the location that is handled by the Apache::WeBWorK module, and tell
# Perl where to find the libraries Apache::WeBWorK needs to run.
# 
$Location{$webwork_url} = {
 SetHandler => "perl-script",
 PerlHandler => "Apache::WeBWorK",
};

$webwork_url should contain "/webwork2".

Is there another directive referencing "/webwork2" somewhere in the httpd.conf file?

Unfortunately I don't know how to set a runtime print statement within that snippet. I believe using

Error "message"

will check compile time errors of the apache directive.

 

In reply to Michael Gage

Re: Fresh WW Installation - 404 when attempting to load site

by Tony Box -
Not using Vhosts. I changed the urls and things to bypass the load balancer for now. I can still successfully hit the server as indicated by apache access_logs.

Michael -- If I take the perl-generated apache <Location> directive, hard code it, and move it out of the block of code that mod_perl uses to generate the conf, it works!

<Location "/webwork2">
SetHandler perl-script
PerlHandler Apache::WeBWorK
</Location>

It also looks like the $Directory{} stuff is not working either. Redirects don't work properly to webwork_courses or webwork_htdocs URLs.

Styling isn't working but that's a different issue that I can tackle.

For now it seems pretty clear that mod_perl is not generating the <Location> or <Directory> directives correctly, or something...

You can check the perl-status page if you'd like: https://webwork.atsi.cool/perl-status
In reply to Tony Box

Re: Fresh WW Installation - 404 when attempting to load site

by Michael Gage -
I'm afraid I couldn't see anything immediately. There is a reference at:


in particular

Apache2::PerlSections->dump


might be helpful.

I've known implementations of mod_perl distributed as ports to be faulty in the past. People have rebuilt mod_perl from CPAN to fix this. The fact that you are having trouble installing apache 2.4 on this distro makes a problem of this sort slightly more likely.

I worry that even if you get around the current problem by replacing <Perl> sections you might run into other difficulties with a faulty mod_perl implementation.

Let us know if you discover something.

-- Mike
In reply to Michael Gage

Re: Fresh WW Installation - 404 when attempting to load site

by Tony Box -
Alright so I've tried some other things but am still having problems.

First, I attempted a fresh install on Debian stretch. Got everything installed but when I attempted to start apache2, I'd get an error along the lines of "segmentation fault (core dumped) Action "start" failedfollowed by a bunch of memory addresses.

That seemed like a dead-end road so I switched to Ubuntu and tried there, being that Ubuntu seems to be the most supported by WebWork. Same thing happened... this time I found a guide on how to get more debug info from a segmentation fault beyond just memory addresses. Here is what I got from that after attempting to start apache2 again:

[New LWP 16912]
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/sbin/apache2 -k start'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x00007fcbd39d21a1 in Perl_sighandler () from /usr/lib/x86_64-linux-gnu/libperl.so.5.22
(gdb) bt full
#0 0x00007fcbd39d21a1 in Perl_sighandler () from /usr/lib/x86_64-linux-gnu/libperl.so.5.22
No symbol table info available.
#1 <signal handler called>
No locals.
#2 0x00007fcbcb2710dc in PerlIOEncode_getarg () from /usr/lib/x86_64-linux-gnu/perl/5.22/auto/PerlIO/encoding/encoding.so
No symbol table info available.
#3 0x00007fcbd3a6ae4e in PerlIOBase_dup () from /usr/lib/x86_64-linux-gnu/libperl.so.5.22
No symbol table info available.
#4 0x00007fcbcb2735d8 in PerlIOEncode_dup () from /usr/lib/x86_64-linux-gnu/perl/5.22/auto/PerlIO/encoding/encoding.so
No symbol table info available.
#5 0x00007fcbd39f09b3 in Perl_fp_dup () from /usr/lib/x86_64-linux-gnu/libperl.so.5.22
No symbol table info available.
#6 0x00007fcbd3a67d36 in PerlIO_clone () from /usr/lib/x86_64-linux-gnu/libperl.so.5.22
No symbol table info available.
#7 0x00007fcbd3a054b5 in perl_clone () from /usr/lib/x86_64-linux-gnu/libperl.so.5.22
No symbol table info available.
#8 0x00007fcbd3d0b4ad in modperl_interp_new () from /usr/lib/apache2/modules/mod_perl.so
No symbol table info available.
#9 0x00007fcbd3d0c14b in modperl_tipool_init () from /usr/lib/apache2/modules/mod_perl.so
No symbol table info available.
#10 0x00007fcbd3d0a92d in ?? () from /usr/lib/apache2/modules/mod_perl.so
No symbol table info available.
#11 0x0000557fe5242d53 in ap_run_post_config ()
No symbol table info available.
#12 0x0000557fe521e49b in main ()
No symbol table info available.
Not sure what else I can do at this point. I may try a fresh ubuntu 16.04 server and use the install script to see if that gets the same unfortunate result.
In reply to Tony Box

Re: Fresh WW Installation - 404 when attempting to load site

by Michael Gage -
Does it work if you are not including the 
webwork.apache2-config

file? The error messages look like something that is happening way

before WeBWorK gets loaded.


You might try searching for mod_perl installation errors on the distro you are using. In the past people have sometimes had to load apreq from CPAN because there was something messed up in their distro. Haven't heard of this recently however.



In reply to Michael Gage

Re: Fresh WW Installation - 404 when attempting to load site

by Tony Box -
It does work if I don't include webwork's apache .conf file.

So I tried using the install script on a brand new installation of Ubuntu just to see if that worked... and it does! (I actually only used the script to do apache/perl setup) So there's clearly something wrong with one or more of the perl modules that I am installing using "apt install" that is causing Apache to crash on startup.

For the record, I used this fork https://github.com/apizer/ww_install of the install script since it has been updated more recently to support WW 2.13

I guess now I need to find the culprit perl module that is borking things up, if that indeed is what it is. The thing I'm still hung up on is that I'm installing exactly the same perl modules as listed in https://github.com/apizer/ww_install/blob/master/distros/ubuntu/1604.pm

The question is should I install ALL of those perl modules using CPAN? Is that what the script does? It seems to have the name of the aptitude package listed which makes it confusing to me if it's using apt or CPAN to install.
In reply to Tony Box

Re: Fresh WW Installation - 404 when attempting to load site

by Danny Glin -
I took a look at the code for the installer. Anything that has the name of an aptitude package is being installed with apt. Anything that has CPAN instead of a package name is installed using CPAN.

It sounds like the culprit might be a package that you are installing using CPAN and the script installs using apt, or vice versa. This may help you narrow down which package is causing the problem.

The most complex things used by WeBWorK are mod_perl, Apache2::Request and mod_apreq. The install script gets these from the aptitude packages libapache2-mod-perl2, libapache2-request-perl and libapache2-mod-apreq2 respectively. If you installed any of these using CPAN, I'd recommend trying the aptitude packages instead and seeing if that fixes the problem.
In reply to Danny Glin

Re: Fresh WW Installation - 404 when attempting to load site

by Tony Box -
Good lord, I think I finally figured it out.

After quadruple checking that I had all of the correct perl mods installed, some using CPAN and others using Aptitude, and still not having it work, I decided to go through the install script, chunk by chunk.

Seems as though mpm_prefork never loaded properly. Without mpm_prefork, the apache server would keep getting those segmentation fault errors. I did notice that mpm_event was loaded which prevents you from enabling mpm_prefork. Is that the proper setup? Have mpm_event disabled and mpm_prefork enabled?


In reply to Tony Box

Re: Fresh WW Installation - 404 when attempting to load site

by Michael Gage -
The basics of setting up apache are not my strong suit but I'm pretty sure that it requires the mpm_prefork to be enabled. Danny @dglin and others will be able to give you a more authoritative answer.

I'm sorry this installation was such a bear. Thanks for hanging in there and debugging it for the rest of us. Hopefully this thread will alert others with a similar problem to check the mpm_prefork issue.

-- Mike

In reply to Tony Box

Re: Fresh WW Installation - 404 when attempting to load site

by Arnold Pizer -
Hi Tony,

Yes, you should have mpm_event disabled and mpm_prefork enabled. See http://webwork.maa.org/wiki/Installation_Manual_for_2.13_on_Ubuntu_16.04#Configuring_Apache
"The default multi-processing module (MPM) for Apache is the event module but WeBWorK uses the prefork module. So we disable the event module and enable the prefork module."

Arnie
In reply to Arnold Pizer

Re: Fresh WW Installation - 404 when attempting to load site

by Tony Box -
Jeeze. I legitimately never saw that tutorial despite looking for a 2.13 install guide on the website. It's linked under a heading on this page (http://webwork.maa.org/wiki/Get_WeBWorK) named Old Installation Guides, which links to another page titled "Old installation guides" http://webwork.maa.org/wiki/Old_Installation_Guides. This is very confusing since it clearly holds the most current installation guides.

Can someone change that heading and page's name? The heading should be "Installation Guides" and the page should probably be called "Manual Installation Guides" or something to that effect.

And to clarify things even more, this page should be changed to something like "Automatic Installation Script Guide": http://webwork.maa.org/wiki/WeBWorK_Installer_Guide

That would lend a much clearer picture as to what all of those pages are.
In reply to Tony Box

Re: Fresh WW Installation - 404 when attempting to load site

by Arnold Pizer -
Hi Tony,

Excellent ideas. Thanks for the feedback. If you have the time, take a look and see if you think things are clearer.

Arnie
In reply to Arnold Pizer

Re: Fresh WW Installation - 404 when attempting to load site

by Tony Box -
WAY better! This should be very helpful for new people trying to install WW.

Thanks!