Latest ISP Config has probles with vhosts

Hello,

after working for months with ispconfig and doing all the latest stable updates, I encounter the problem that vhost files do not get generated anymore.

All vhosts, also after a resync of the websites, get created as *.vhost.err files.

So I started to debug:

I commented the server.sh cronjob, put ISPconfig in debug/dev mode, removed the .err files, resynced the websites and runned the script manually. the result:

13.03.2016-16:31 - DEBUG - Set Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
13.03.2016-16:31 - DEBUG - Found 7 changes, starting update process.
13.03.2016-16:31 - DEBUG - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
13.03.2016-16:31 - DEBUG - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
13.03.2016-16:31 - DEBUG - Add server alias: helpdesk.mydomain.com
13.03.2016-16:31 - DEBUG - Writing the vhost file: /etc/apache2/sites-available/mydomain.com.vhost
13.03.2016-16:31 - DEBUG - Apache status is: running
13.03.2016-16:31 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
13.03.2016-16:31 - DEBUG - Restarting httpd: service httpd restart
13.03.2016-16:31 - DEBUG - Apache restart return value is: 1
13.03.2016-16:31 - DEBUG - Apache online status after restart is: running


******

warning comes for all subdomains, I shorted it here


*****

13.03.2016-16:31 - WARNING - Apache did not restart after the configuration change for website mydomain.com. Reverting the configuration. Saved non-working config as /etc/apache2/sites-available/mydomain.com.vhost.err
13.03.2016-16:31 - WARNING - Reason for Apache restart failure: httpd: unrecognized service
13.03.2016-16:31 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
13.03.2016-16:31 - DEBUG - Restarting httpd: service httpd restart
13.03.2016-16:31 - DEBUG - Processed datalog_id 244
13.03.2016-16:31 - DEBUG - Calling function 'ssl' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
13.03.2016-16:31 - DEBUG - Calling function 'update' from plugin 'apache2_plugin' raised by event 'web_domain_update'.
13.03.2016-16:31 - DEBUG - Writing the vhost file: /etc/apache2/sites-available/maildev.mydomain.com.vhost
13.03.2016-16:31 - DEBUG - Apache status is: running
13.03.2016-16:31 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
13.03.2016-16:31 - DEBUG - Restarting httpd: service httpd restart
13.03.2016-16:31 - DEBUG - Apache restart return value is: 1
13.03.2016-16:31 - DEBUG - Apache online status after restart is: running

13.03.2016-16:31 - DEBUG - Calling function 'restartHttpd' from module 'web_module'.
13.03.2016-16:31 - DEBUG - Restarting httpd: service httpd restart
13.03.2016-16:31 - DEBUG - Processed datalog_id 250
13.03.2016-16:31 - DEBUG - Remove Lock: /usr/local/ispconfig/server/temp/.ispconfig_lock
finished.

 

So I dont know why i cannot post the Logs etc. (Forum is telling me I cannot post links), I add all this to a file here

 

If you "enable" one of the .err files by renaming them to "something.vhost" without overwriting any valid one and then issue

Code:

$apachectl -t

should give you the error shown.

 

has no error, just tried it.

When trying to enable it(a2ensite testsite.vhost), it tells me:
ERROR: Site testsite.vhost does not exist!

Well the file itself exists. the shell also autocompletes it with pressing TAB.

 

the a2ensite command does add the extenion on its own, just leave the .vhost or copy that file temporarily to sites-enabled folder and delete it afterwards.
Just want to let it parsed by your webserver without restart in order to find any complaints for missing modules, syntax errors or similar

 

So when you say without restart, I should just try to reload the apache with the manually copied file?

 

Ok just did this.

Created the symlink manually, reloaded the apache. Tada, it works fine

Now I am confused, what problem ISPconfig could have with it.

 

Hmm basically according to the logs ISPConfig did everything right, it just had an issue finding the httpd / apache process after the change.
So for some reason apache took longer than expected to restart.
How long does it take if you restart your webserver including serving websites again?
Did you modify / tune some settings?

It should not exceed 7 seconds.
If for some reason it takes longer than that its unusal and should be followed by examination but can be "silenced"
by modifying
/usr/local/ispconfig/server/plugins-available/apache2_plugin.inc.php on line 1530
either increase the value for the sleep() call or increase the loops ( for( ... ) ) the script does from 5 to a higher amount. But this is no real solution to a possible issue with the basic configuration thus considered as a temporary workaround.

 

Hi I didn't changed anything on the apache, but can post the conf tomorrow.

Could it be something about the sites itself? It doesn't have that much traffic, but an old system with alot of .htaccess sites with overwrides.

 

So does a restart need that long to restart?
while .htaccess files do need some sort of additional time before a request is processed it should not matter in your case - at least I doubt that is an issue here.

 

Hello,

the apache restart takes 22 seconds, without any error or password.

Just stopping it takes allready 7 seconds, starting it manually 15 seconds.

Htop shows a load of 0.35 at 4 Xeon V3 Cores and 4 GB Ram (Virtual Server). As soon as Apache is running, all websites run fast and smoothly.

apache2conf:

# This is the main Apache server configuration file. It contains the
# configuration directives that give the server its instructions.
# See http://httpd.apache.org/docs/2.4/ for detailed information about
# the directives and /usr/share/doc/apache2/README.Debian about Debian specific
# hints.
#
#
# Summary of how the Apache 2 configuration works in Debian:
# The Apache 2 web server configuration in Debian is quite different to
# upstream's suggested way to configure the web server. This is because Debian's
# default Apache2 installation attempts to make adding and removing modules,
# virtual hosts, and extra configuration directives as flexible as possible, in
# order to make automating the changes and administering the server as easy as
# possible.

# It is split into several files forming the configuration hierarchy outlined
# below, all located in the /etc/apache2/ directory:
#
# /etc/apache2/
# |-- apache2.conf
# | `-- ports.conf
# |-- mods-enabled
# | |-- *.load
# | `-- *.conf
# |-- conf-enabled
# | `-- *.conf
# `-- sites-enabled
# `-- *.conf
#
#
# * apache2.conf is the main configuration file (this file). It puts the pieces
# together by including all remaining configuration files when starting up the
# web server.
#
# * ports.conf is always included from the main configuration file. It is
# supposed to determine listening ports for incoming connections which can be
# customized anytime.
#
# * Configuration files in the mods-enabled/, conf-enabled/ and sites-enabled/
# directories contain particular configuration snippets which manage modules,
# global configuration fragments, or virtual host configurations,
# respectively.
#
# They are activated by symlinking available configuration files from their
# respective *-available/ counterparts. These should be managed by using our
# helpers a1enmod/a2dismod, a2ensite/a2dissite and a2enconf/a2disconf. See
# their respective man pages for detailed information.
#
# * The binary is called apache2. Due to the use of environment variables, in
# the default configuration, apache2 needs to be started/stopped with
# /etc/init.d/apache2 or apache2ctl. Calling /usr/bin/apache2 directly will not
# work with the default configuration.


# Global configuration
#

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE! If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the Mutex documentation (available
# at <URL:http://httpd.apache.org/docs/2.4/mod/core.html#mutex>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
#ServerRoot "/etc/apache2"

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
Mutex file:${APACHE_LOCK_DIR} default

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 100

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 5


# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here. If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog ${APACHE_LOG_DIR}/error.log

#
# LogLevel: Control the severity of messages logged to the error_log.
# Available values: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the log level for particular modules, e.g.
# "LogLevel info ssl:warn"
#
LogLevel warn

# Include module configuration:
IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf

# Include list of ports to listen on
Include ports.conf


# Sets the default security model of the Apache2 HTTPD server. It does
# not allow access to the root filesystem outside of /usr/share and /var/www.
# The former is used by web applications packaged in Debian,
# the latter may be used for local directories served by the web server. If
# your system is serving content from a sub-directory in /srv you must allow
# access here, or in any related virtual host.
<Directory />
Options FollowSymLinks
AllowOverride None
Require all denied
</Directory>

<Directory /usr/share>
AllowOverride None
Require all granted
</Directory>

<Directory /var/www/>
Options Indexes FollowSymLinks
AllowOverride None
Require all granted
</Directory>

#<Directory /srv/>
# Options Indexes FollowSymLinks
# AllowOverride None
# Require all granted
#</Directory>

<Location /server-status>
SetHandler server-status
Order allow,deny
Allow from all
</Location>


# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives. See also the AllowOverride
# directive.
#
AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<FilesMatch "^\.ht">
Require all denied
</FilesMatch>


#
# The following directives define some format nicknames for use with
# a CustomLog directive.
#
# These deviate from the Common Log Format definitions in that they use %O
# (the actual bytes sent including headers) instead of %b (the size of the
# requested file), because the latter makes it impossible to detect partial
# requests.
#
# Note that the use of %{X-Forwarded-For}i instead of %h is not recommended.
# Use mod_remoteip instead.
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

# Include of directories ignores editors' and dpkg's backup files,
# see README.Debian for details.

# Include generic snippets of statements
IncludeOptional conf-enabled/*.conf

# Include the virtual host configurations:
IncludeOptional sites-enabled/

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

#include phpmyadmin
Include /etc/phpmyadmin/apache.conf



Is there any other file you need to get a better view on it?

Thanks guys

 

if you did no attempts to tune your apache server I don't think it's really apaches fault entirely.

After doing a restart, what does the following output?

Code:

journalctl -n
journalctl -xe 
systemctl status httpd.service 

Do you have by chance some strange vServer situation networkwise?
Do you have a valid hostname? Does your /etc/resolv.conf have valid & reachable nameservers only?
Are there used IPs in /etc/hosts ( ipv4/v6 ) which do not resolv properly?

What does

Code:

apache2ctl -M

do output?

 

Hi,

those commands do not work, as far as I researched, systemd isn't available for ubuntu 14.04

apache2ctl -M:


root@faris:/tmp# apache2ctl -M
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-ispconfig.conf:62
Loaded Modules:
core_module (static)
so_module (static)
watchdog_module (static)
http_module (static)
log_config_module (static)
logio_module (static)
version_module (static)
unixd_module (static)
access_compat_module (shared)
actions_module (shared)
alias_module (shared)
auth_basic_module (shared)
auth_digest_module (shared)
authn_core_module (shared)
authn_file_module (shared)
authz_core_module (shared)
authz_host_module (shared)
authz_svn_module (shared)
authz_user_module (shared)
autoindex_module (shared)
cgi_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_svn_module (shared)
deflate_module (shared)
dir_module (shared)
env_module (shared)
expires_module (shared)
fastcgi_module (shared)
fcgid_module (shared)
filter_module (shared)
headers_module (shared)
include_module (shared)
mime_module (shared)
mpm_prefork_module (shared)
negotiation_module (shared)
pagespeed_module (shared)
perl_module (shared)
php5_module (shared)
proxy_module (shared)
proxy_http_module (shared)
python_module (shared)
rewrite_module (shared)
setenvif_module (shared)
socache_shmcb_module (shared)
ssl_module (shared)
status_module (shared)
suexec_module (shared)
suphp_module (shared)


Something seems wrong with the ispconfig vhost.

hostname returns the main-domain-name (setup by the hoster by default)

Nameservers are also given by the hoster and are set in the isp config system preferences and in the /etc/resolv.con
nameserver 158.181.50.240
nameserver 158.181.50.245


/etc/hosts is also fine, showing the localhost and the main domain pointed to the server ip

 

Did you test your nameservers?

Code:

dig @158.181.50.240 google.com

At least they're not available from the outside.

Code:

NameVirtualHost has no effect and will be removed in the next release

has no really bad effect and defnitly not causing such a slow startup/stop time.

Do you really use all of these modules?

Code:

autoindex_module (shared)
dav_module (shared)
dav_fs_module (shared)
dav_svn_module (shared)
negotiation_module (shared)
pagespeed_module (shared)
perl_module (shared)
php5_module (shared)
proxy_module (shared)
proxy_http_module (shared)
python_module (shared)
status_module (shared)
suexec_module (shared)
suphp_module (shared)

Try disableing as many of these modules until you see an error ( apache2ctl -t to test configuration ).
I don't really think there is an issue with these module but I'd start with the proxy and pagespeed modules.
If you didn't secure your proxy_module ( Disable proxy requests ) anyone could use your webserver as anonymous proxy ...

Since the systemd commands are not available for you, what about the good old /var/log/apache2/error.log ?

 

Hi,
the nameserver work fine:

Code:

root@faris:/tmp# dig @158.181.50.240 google.com

; <<>> DiG 9.9.5-3ubuntu0.5-Ubuntu <<>> @158.181.50.240 google.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 58094
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 5

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;google.com.                    IN      A

;; ANSWER SECTION:
google.com.             182     IN      A       172.217.19.238

;; AUTHORITY SECTION:
google.com.             152161  IN      NS      ns3.google.com.
google.com.             152161  IN      NS      ns1.google.com.
google.com.             152161  IN      NS      ns4.google.com.
google.com.             152161  IN      NS      ns2.google.com.

;; ADDITIONAL SECTION:
ns1.google.com.         325038  IN      A       216.239.32.10
ns2.google.com.         325038  IN      A       216.239.34.10
ns3.google.com.         325038  IN      A       216.239.36.10
ns4.google.com.         325038  IN      A       216.239.38.10

;; Query time: 2 msec
;; SERVER: 158.181.50.240#53(158.181.50.240)
;; WHEN: Mon Mar 14 15:08:57 CET 2016
;; MSG SIZE  rcvd: 191

I removed all proxmodules, pagespeed. Python and Perl are needed by a Website.

After doing that, I found out that the proxy stuff is used by openproject... So I had to enable it again.

Well looking through the logs (actually is there a trick to tail -f the file and restart the server without exiting the tail?)

Code:

[Mon Mar 14 15:17:13.866430 2016] [mpm_prefork:notice] [pid 5888] AH00169: caught SIGTERM, shutting down
[Mon Mar 14 15:17:22.574764 2016] [ssl:warn] [pid 6059] AH01906: RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Mar 14 15:17:22.574869 2016] [ssl:warn] [pid 6059] AH01909: RSA certificate configured for op.mydomain.info:443 does NOT include an ID which matches the server name
[Mon Mar 14 15:17:22.575723 2016] [ssl:warn] [pid 6059] AH01906: RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Mar 14 15:17:22.575761 2016] [ssl:warn] [pid 6059] AH01909: RSA certificate configured for mydomain.info:8080 does NOT include an ID which matches the server name
[Mon Mar 14 15:17:22.576187 2016] [suexec:notice] [pid 6059] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Mon Mar 14 15:17:26.700516 2016] [auth_digest:notice] [pid 6063] AH01757: generating secret for digest authentication ...
[Mon Mar 14 15:17:26.755079 2016] [:notice] [pid 6079] FastCGI: process manager initialized (pid 6079)
[Mon Mar 14 15:17:26.950893 2016] [:notice] [pid 6063] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Mon Mar 14 15:17:26.950966 2016] [:notice] [pid 6063] mod_python: using mutex_directory /tmp 
[Mon Mar 14 15:17:26.993950 2016] [ssl:warn] [pid 6063] AH01906: RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Mar 14 15:17:26.994046 2016] [ssl:warn] [pid 6063] AH01909: RSA certificate configured for op.mydomain.info:443 does NOT include an ID which matches the server name
[Mon Mar 14 15:17:26.994947 2016] [ssl:warn] [pid 6063] AH01906: RSA server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Mar 14 15:17:26.994991 2016] [ssl:warn] [pid 6063] AH01909: RSA certificate configured for mydomain.info:8080 does NOT include an ID which matches the server name
[Mon Mar 14 15:17:27.296325 2016] [mpm_prefork:notice] [pid 6063] AH00163: Apache/2.4.7 (Ubuntu) SVN/1.8.8 mod_fastcgi/mod_fastcgi-SNAP-0910052141 mod_fcgid/2.3.9 PHP/5.5.9-1ubuntu4.14 mod_python/3.3.1 Python/2.7.6 OpenSSL/1.0.1f mod_perl/2.0.8 Perl/v5.18.2 configured -- resuming normal operations
[Mon Mar 14 15:17:27.296514 2016] [core:notice] [pid 6063] AH00094: Command line: '/usr/sbin/apache2'

looks good, only the openproject seems to have issues with the openssl certificate. (It's not a bought one, just normal openssl)

I try to find some issues in the openprject community about that.

 

Well looks normal?

Code:

[Mon Mar 14 15:17:22.576187 2016] [suexec:notice] [pid 6059] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec)
[Mon Mar 14 15:17:26.700516 2016] [auth_digest:notice] [pid 6063] AH01757: generating secret for digest authentication ...

We have 4 seconds of gambling here - make sure you really need auth_digest
( Also webdav support, which needs auth_digest if I remember correctly )

I still don't know what's taking 7-8 seconds to shutdown the server though - probably need some debug level logging enabled.

If this turns out be expected behaviour and is just doing slow - I'd say it's save for you to hack ISPConfig to just wait a lil longer for your apache beeing ready again. vServer, many modules, maybe shutdown of mod_proxy can be the cause here.

 

Auth_digest is used by svn authentication over webdav.

Ok atleast nothing seems to be damaged.

I will hack the /usr/local/ispconfig/server/plugins-available/apache2_plugin.inc.php as you suggested and monitor the server.

Actually, the vServer is from a well let's say quite expensive hoster in germany, known for their quality in doing that.

Should I contact them with some possible problems?

Thank you anyway for your help.

 

Hehe yeah, I already did a quick research about that
Well I'm not quite familiar with the actual code of the auth_digest module, but slow key generation could be a sign of lack of entropy - maybe your system needs more load =) Honestly I don't know.
But due to lack of hardware vServers do have much less entropy sources

Code:

cat /proc/sys/kernel/random/entropy_avail

Maybe this tut can help you https://www.howtoforge.com/helping-...in-enough-entropy-with-rng-tools-debian-lenny

I'd still analyze what causes apache to take so long to shutdown.

 

Hi,

entropy is always about 800 - 900.

But don't get me wrong, even after reading your link and the definition of entropy I still dont really get what it is

Well for the apache shutdown: It seems it's only openproject. Removing the vhost file and disable the mods needed, apache restarts within 3 seconds.

What I didn't found yet, is if the delay caused by openproject is somehow normal or maybe too much.