Hi, Today i am unable to access ISPConfig control panel. Over the years i have installed ISPConfig only twice. I am not able to setup Certbot or any certificate for my websites and it wasn't a problem for accessing ISPConfig, so far. Today i noticed that i am not able to access the ISPConfig panel on my server. Browser says: Code: The connection has timed out All other services, are working normally and even same domain is working fine on other port 80 tail apache error.log: Code: [Sun May 26 13:57:24.722904 2024] [autoindex:error] [pid 1186:tid 140063362963136] [client 1.2.3.4:52588] AH01276: Cannot serve directory /var/www/apps/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm,index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm,standard_index.html) found, and server-generated directory index forbidden by Options directive [Sun May 26 14:09:18.546484 2024] [proxy_fcgi:error] [pid 1186:tid 140063514031808] [client 4.3.2.1:10148] AH01071: Got error 'Primary script unknown' [Sun May 26 14:31:19.327766 2024] [mpm_event:notice] [pid 1107:tid 140063665059712] AH00492: caught SIGWINCH, shutting down gracefully [ N 2024-05-26 14:31:19.5259 4805/T1 age/Wat/WatchdogMain.cpp:1377 ]: Starting Passenger watchdog... [ N 2024-05-26 14:31:19.5618 4808/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core... [ N 2024-05-26 14:31:19.5625 4808/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode. [ N 2024-05-26 14:31:19.5692 4808/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 4808 [Sun May 26 14:31:19.589508 2024] [suexec:notice] [pid 4802:tid 140032510748544] AH01232: suEXEC mechanism enabled (wrapper: /usr/lib/apache2/suexec) [ N 2024-05-26 14:31:19.6249 4808/T8 age/Cor/CoreMain.cpp:670 ]: Signal received. Gracefully shutting down... (send signal 2 more time(s) to force shutdown) [ N 2024-05-26 14:31:19.6250 4808/T1 age/Cor/CoreMain.cpp:1245 ]: Received command to shutdown gracefully. Waiting until all clients have disconnected... [ N 2024-05-26 14:31:19.6258 4808/Ta Ser/Server.h:901 ]: [ApiServer] Freed 0 spare client objects [ N 2024-05-26 14:31:19.6258 4808/Ta Ser/Server.h:558 ]: [ApiServer] Shutdown finished [ N 2024-05-26 14:31:19.6258 4808/T8 Ser/Server.h:901 ]: [ServerThr.1] Freed 0 spare client objects [ N 2024-05-26 14:31:19.6258 4808/T8 Ser/Server.h:558 ]: [ServerThr.1] Shutdown finished [ N 2024-05-26 14:31:19.7172 4830/T1 age/Wat/WatchdogMain.cpp:1377 ]: Starting Passenger watchdog... [ N 2024-05-26 14:31:19.7447 4833/T1 age/Cor/CoreMain.cpp:1340 ]: Starting Passenger core... [ N 2024-05-26 14:31:19.7448 4833/T1 age/Cor/CoreMain.cpp:256 ]: Passenger core running in multi-application mode. [ N 2024-05-26 14:31:19.7489 4833/T1 age/Cor/CoreMain.cpp:1015 ]: Passenger core online, PID 4833 [Sun May 26 14:31:19.782153 2024] [:notice] [pid 4825:tid 140032510748544] mod_python: Creating 8 session mutexes based on 0 max processes and 25 max threads. [Sun May 26 14:31:19.782191 2024] [:notice] [pid 4825:tid 140032510748544] mod_python: using mutex_directory /tmp [Sun May 26 14:31:19.828626 2024] [mpm_event:notice] [pid 4825:tid 140032510748544] AH00489: Apache/2.4.59 (Debian) mod_fcgid/2.3.9 Phusion_Passenger/6.0.17 OpenSSL/3.0.11 mod_python/3.5.0+git20211031.e6458ec Python/3.11.2 mod_perl/2.0.12 Perl/v5.36.0 configured -- resuming normal operations [Sun May 26 14:31:19.828698 2024] [core:notice] [pid 4825:tid 140032510748544] AH00094: Command line: '/usr/sbin/apache2' [ N 2024-05-26 14:31:20.1587 4808/T1 age/Cor/TelemetryCollector.h:531 ]: Message from Phusion: End time can not be before or equal to begin time [ N 2024-05-26 14:31:20.1856 4808/T1 age/Cor/CoreMain.cpp:1325 ]: Passenger core shutdown finished [ E 2024-05-26 14:31:22.3354 4833/T7 age/Cor/SecurityUpdateChecker.h:521 ]: A security update is available for your version (6.0.17) of Phusion Passenger(R). We strongly recommend upgrading to version 6.0.22. [ E 2024-05-26 14:31:22.3355 4833/T7 age/Cor/SecurityUpdateChecker.h:526 ]: Additional security update check information: - [Fixed in 6.0.19] [CVE-2023-38545] A vulnerability existed in libcurl before 8.4.0 which was the library used for Passenger proxy functionality. Exploiting this vulnerability would require two preconditions. First a SOCKS5 proxy to be configured for Passenger licensing, anonymous telemetry, or security update check which is not the default but is possible. Second the attacker would need to cause Passenger to use an attacker-controlled URL when performing these requests. Causing Passenger to use non-standard urls requires that the attacker already have code execution on the Passenger host, or control of the Passenger config. If exploited this vulnerability could lead to code execution, due to buffer overflow. UFW Status Code: 8080/tcp (v6) ALLOW Anywhere (V6) 8080/tcp (v6) ALLOW Anywhere (V6) nc Code: nc -z 1.2.3.4 80 example.com [1.2.3.4] 80 (http) open nc -z 1.2.3.4 8080 example.com [1.2.3.4] 8080 (http-alt) Connection timed out. How to find which is blocking my request
Run the test script and post the result: https://forum.howtoforge.com/threads/please-read-before-posting.58408/ And regarding this: See: https://forum.howtoforge.com/threads/lets-encrypt-error-faq.74179/
Btw. The most likely reason for your issue is an external firewall or router in front of your server blocking port 8080 and preventing the LE test succeeds, which then prevents LE certs to be issued. This firewall can be in a data center, e.g. most cloud providers block ports, so if this system is in the cloud, then check the control panel of the data center and open the ports there. Plus disable let#s encrypt check in ISPConfig for the LE issue 8see let#s encrypt Error FAQ that I linked above).
if a firewall or router is the culprit, this could be easily checked using tools like this https://dnschecker.org/port-scanner.php I assume it was you doing the restart manually or ispconfig byrequest/update/cron? Though is a strong indicator either the firewall does not reject but drop ; the timeout value is set too low in a script and your server has lack of resoureces to response in a timely manner or simple as @till said. check netstat -tulpen and your ufw status see this as addition, not OR, the logs for ispconfig / letsencrypt may help finding another possible issue, aswell as tills instructions!
first glance reaction; firewall looks ok. however one can assume you have a complete ispconfig setup including amavis or rspamd. However your lack of resources might start the OOM reaper on needed services. If you are using a full featured ispconfig with database, clamav and such, aim for 4gb of memory at least for smoother operations. Does it work for a while after rebooting? Does cat /var/log/syslog | grep OOM or dmesg | grep OOM reveal something? could be an configuration issue with apache / php / execution method of the panel still, but lack of resources is a valid reason too
ISPConfig seems to be running fine, and Apache is listening on port 8080, so your issue is likely an external firewall problem or router in front of the server that blocks access to port 8080. 1GB RAM is low, it is better to use 2GB RAM + 2GB swap for a small system. If you need more performance, using 4GB RAM or more will be useful like @ztk.me mentioned.
>>> inputs below if a firewall or router is the culprit, this could be easily checked using tools like this https://dnschecker.org/port-scanner.php >>> port 8080 is open as per dnschecker I assume it was you doing the restart manually or ispconfig byrequest/update/cron? >>> not sure of this Though is a strong indicator either the firewall does not reject but drop ; the timeout value is set too low in a script and your server has lack of resoureces to response in a timely manner or simple as @till said. check netstat -tulpen and your ufw status i >>> already posted above and is in allow mode see this as addition, not OR, the logs for ispconfig / letsencrypt may help finding another possible issue, aswell as tills instructions! >>> can you specify which file is should check for ispconfig
no output for dmesg but got this for syslog Code: 2024-05-26T13:07:03.750906+05:30 server1 clamd[890]: LibClamAV debug: * Submodule MYDOOMLOG:#011On 2024-05-26T13:28:17.451882+05:30 server1 clamd[890]: LibClamAV debug: * Submodule MYDOOMLOG:#011On 2024-05-26T13:49:13.168792+05:30 server1 clamd[890]: LibClamAV debug: * Submodule MYDOOMLOG:#011On 2024-05-26T14:49:43.417357+05:30 server1 clamd[890]: LibClamAV debug: * Submodule MYDOOMLOG:#011On
An increase in RAM is on the cards, but my server is just a joke running debian (provided by vultr). i have just one client website running, a small website. i am using mail server only for my personal usage. Just got about 4 clients using SSH for running python programs. They are running between 9:15 to 15:30 +5:30 UST here is the ps output Code: USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND _rspamd 1380 0.0 4.6 248464 46104 ? S 13:48 0:06 rspamd: controller process (localhost:11334) ispconf+ 12831 0.0 4.6 317596 46052 ? Ss 16:42 0:00 /usr/bin/php-cgi -d disable_classes= -d disable_functions= -d magic_quotes_gpc=off -d open_basedir= -d session.save_path=/usr/local/ispconfig/interface/temp _rspamd 1381 0.0 4.4 247108 43508 ? S 13:48 0:03 rspamd: normal process (localhost:11333) root 899 0.1 3.7 659436 36944 ? Ssl 13:48 0:37 /usr/bin/python3 /usr/bin/fail2ban-server -xf start mysql 1036 0.0 2.9 1092656 28696 ? Ssl 13:48 0:16 /usr/sbin/mariadbd ispconf+ 12832 0.0 2.4 317924 24368 ? S 16:42 0:00 /usr/bin/php-cgi -d disable_classes= -d disable_functions= -d magic_quotes_gpc=off -d open_basedir= -d session.save_path=/usr/local/ispconfig/interface/temp clamav 890 0.2 2.3 1609032 22992 ? Ssl 13:48 1:07 /usr/sbin/clamd --foreground=true root 259 0.0 2.0 82468 19940 ? Ss 13:48 0:04 /lib/systemd/systemd-journald ntpsec 966 0.0 1.9 84876 19236 ? SLs 13:48 0:01 /usr/sbin/ntpd -p /run/ntpd.pid -c /etc/ntpsec/ntp.conf -g -N -u ntpsec:ntpsec
you are right i entered http:// instead of https:// https://server1.example.com:8080 and http://server1.example.com got the same results. however entering https://server1.example.com i got this on the browser Code: =')) { error_reporting(E_ALL & ~E_NOTICE & ~E_DEPRECATED & ~E_STRICT & ~E_USER_NOTICE & ~E_USER_DEPRECATED); } else { error_reporting(E_ALL & ~E_NOTICE & ~E_STRICT & ~E_USER_NOTICE); } break; default: header('HTTP/1.1 503 Service Unavailable.', TRUE, 503); echo 'The application environment is not set correctly.'; exit(1); // EXIT_ERROR } /* *--------------------------------------------------------------- * SYSTEM DIRECTORY NAME *--------------------------------------------------------------- * * This variable must contain the name of your "system" directory. * Set the path if it is not in the same directory as this file. */ $system_path = 'system'; /* *--------------------------------------------------------------- * APPLICATION DIRECTORY NAME *--------------------------------------------------------------- * * If you want this front controller to use a different "application" * directory than the default one you can set its name here. The directory * can also be renamed or relocated anywhere on your server. If you do, * use an absolute (full) server path. * For more info please see the user guide: * * https://codeigniter.com/user_guide/general/managing_apps.html * * NO TRAILING SLASH! */ $application_folder = 'application'; /* *--------------------------------------------------------------- * VIEW DIRECTORY NAME *--------------------------------------------------------------- * * If you want to move the view directory out of the application * directory, set the path to it here. The directory can be renamed * and relocated anywhere on your server. If blank, it will default * to the standard location inside your application directory. * If you do move this, use an absolute (full) server path. * * NO TRAILING SLASH! */ $view_folder = ''; /* * -------------------------------------------------------------------- * DEFAULT CONTROLLER * -------------------------------------------------------------------- * * Normally you will set your default controller in the routes.php file. * You can, however, force a custom routing by hard-coding a * specific controller class/function here. For most applications, you * WILL NOT set your routing here, but it's an option for those * special instances where you might want to override the standard * routing in a specific front controller that shares a common CI installation. * * IMPORTANT: If you set the routing here, NO OTHER controller will be * callable. In essence, this preference limits your application to ONE * specific controller. Leave the function name blank if you need * to call functions dynamically via the URI. * * Un-comment the $routing array below to use this feature */ // The directory name, relative to the "controllers" directory. Leave blank // if your controller is not in a sub-directory within the "controllers" one // $routing['directory'] = ''; // The controller class file name. Example: mycontroller // $routing['controller'] = ''; // The controller function you wish to be called. // $routing['function'] = ''; /* * ------------------------------------------------------------------- * CUSTOM CONFIG VALUES * ------------------------------------------------------------------- * * The $assign_to_config array below will be passed dynamically to the * config class when initialized. This allows you to set custom config * items or override any default config values found in the config.php file. * This can be handy as it permits you to share one application between * multiple front controller files, with each file containing different * config values. * * Un-comment the $assign_to_config array below to use this feature */ // $assign_to_config['name_of_config_item'] = 'value of config item'; // -------------------------------------------------------------------- // END OF USER CONFIGURABLE SETTINGS. DO NOT EDIT BELOW THIS LINE // -------------------------------------------------------------------- /* * --------------------------------------------------------------- * Resolve the system path for increased reliability * --------------------------------------------------------------- */ // Set the current directory correctly for CLI requests if (defined('STDIN')) { chdir(dirname(__FILE__)); } if (($_temp = realpath($system_path)) !== FALSE) { $system_path = $_temp.DIRECTORY_SEPARATOR; } else { // Ensure there's a trailing slash $system_path = strtr( rtrim($system_path, '/\\'), '/\\', DIRECTORY_SEPARATOR.DIRECTORY_SEPARATOR ).DIRECTORY_SEPARATOR; } // Is the system path correct? if ( ! is_dir($system_path)) { header('HTTP/1.1 503 Service Unavailable.', TRUE, 503); echo 'Your system folder path does not appear to be set correctly. Please open the following file and correct this: '.pathinfo(__FILE__, PATHINFO_BASENAME); exit(3); // EXIT_CONFIG } /* * ------------------------------------------------------------------- * Now that we know the path, set the main path constants * ------------------------------------------------------------------- */ // The name of THIS file define('SELF', pathinfo(__FILE__, PATHINFO_BASENAME)); // Path to the system directory define('BASEPATH', $system_path); // Path to the front controller (this file) directory define('FCPATH', dirname(__FILE__).DIRECTORY_SEPARATOR); // Name of the "system" directory define('SYSDIR', basename(BASEPATH)); // The path to the "application" directory if (is_dir($application_folder)) { if (($_temp = realpath($application_folder)) !== FALSE) { $application_folder = $_temp; } else { $application_folder = strtr( rtrim($application_folder, '/\\'), '/\\', DIRECTORY_SEPARATOR.DIRECTORY_SEPARATOR ); } } elseif (is_dir(BASEPATH.$application_folder.DIRECTORY_SEPARATOR)) { $application_folder = BASEPATH.strtr( trim($application_folder, '/\\'), '/\\', DIRECTORY_SEPARATOR.DIRECTORY_SEPARATOR ); } else { header('HTTP/1.1 503 Service Unavailable.', TRUE, 503); echo 'Your application folder path does not appear to be set correctly. Please open the following file and correct this: '.SELF; exit(3); // EXIT_CONFIG } define('APPPATH', $application_folder.DIRECTORY_SEPARATOR); // The path to the "views" directory if ( ! isset($view_folder[0]) && is_dir(APPPATH.'views'.DIRECTORY_SEPARATOR)) { $view_folder = APPPATH.'views'; } elseif (is_dir($view_folder)) { if (($_temp = realpath($view_folder)) !== FALSE) { $view_folder = $_temp; } else { $view_folder = strtr( rtrim($view_folder, '/\\'), '/\\', DIRECTORY_SEPARATOR.DIRECTORY_SEPARATOR ); } } elseif (is_dir(APPPATH.$view_folder.DIRECTORY_SEPARATOR)) { $view_folder = APPPATH.strtr( trim($view_folder, '/\\'), '/\\', DIRECTORY_SEPARATOR.DIRECTORY_SEPARATOR ); } else { header('HTTP/1.1 503 Service Unavailable.', TRUE, 503); echo 'Your view folder path does not appear to be set correctly. Please open the following file and correct this: '.SELF; exit(3); // EXIT_CONFIG } define('VIEWPATH', $view_folder.DIRECTORY_SEPARATOR); /* * -------------------------------------------------------------------- * LOAD THE BOOTSTRAP FILE * -------------------------------------------------------------------- * * And away we go... */ require_once BASEPATH.'core/CodeIgniter.php';
That's not ISPConfig. You either access a wrong server or you did not installed a clean server but having installed another app on port 8080 already.
My assumption would be a misconfigured php configuration for your vhosts. If this is an older instance or a configuration got disrupted due to lack of resources it could be this or just a bug a while ago ( or whatever ) and there is a missmatch now. is running, but is it configured for your ispconfig panel correctly. This is stuff I'd look for if I had more time but I think @till knows his things much better and could give you the valid pointers. tldr, compare the ispconfig vhost configs and check if paths given makes sense. Though I guess the error-log should show something if upstream is not available. If you are not sure if you induced the restat of apache, it's a thing to monitor then. The OOM you found is not the OOM I was looking for, though it might have been logged a too long while ago or in other logs. Also ufw spams dmesg unfortunally. If you can afford some downtime, disable clamav, rspamd and redis if you do not need redis for other things. but clamav alone can lead to more memory consumption than you have, unfortunally.
it was not showing when accessing port 8080 but 80 , which is actually another subdomain. so the issue remains. sorry for confusing.
