Issues updating from Ubuntu 18.04 to Ubuntu 20.04

I'm currently running the perfect server under Ubuntu 18.04 LTS with ISPConfig 3.2.2. All has been running well for several years. Last night, I tried updating to Ubuntu 20.04 LTS and things didn't go well. Fortunately, I have a couple of spare hard drives, so I cloned the working drive onto them, swapped out the working drive with one of the new clones, and tried updating the clone.
The first issue was when it was trying to update the myphpadmin database. Got errors saying it couldn't connect to the database. Eventually, I figured out that mysqld wasn't running. I connected, started mysqld, and that seemed to get past that issue.
I've been updating Roundcube as new versions came out. The update process was, I think, trying to update Roundcube to 1.3.6 and failing because I'm running version 1.4.11. I finally told it to abort that update, and it moved on, only to pitch fits at the end of the update when it tried to install it again. As a result, I'm not sure the update completed successfully. Is there a better way around this?
The end result was a bootable system, but trying to connect to Roundcube resulted in a white screen.
I think I need to try it again, going through the 20.04 server configuration to make sure there's not something missing.
Would I be better off to wait until 22.04 comes out next year, make a new, clean installation, and copy the vmail folders from the 18.04 configuration to the new one?
Thanks,
Bob

 

Ubuntu 18.04 dist upgrade to 20.04 assumes the OS is plain 18.04. If you have installed roundcube from outside Ubuntu 18.04 there is no guarantee the upgrade goes successfully. Same for all othere software. Examine the Ubuntu upgrade instructions and preconditions, I'm sure they are documented somewhere.
When you have successfully done the upgrade to 20.04, follow the ISPConfig Perfect server guider for 20.04 to install the necessary packages and do the configurations. Then do

Code:

ispconfig_update.sh --force

to force ISPConfig to configure for applications properly.
If your testing shows dist-upgrade does not work, another way is to install a new empty ISConfig 3.2 on Ubuntu 20.04 and use ISPConfig Migration Tool to copy the data from old server to this new. You need two servers running at the same time.

 

I initially installed Roundcube when I first built the server, but have been updating it from outside since then. I guess that's the problem there. I may try backing up the Roundcube database, uninstalling it via apt, doing the upgrade to 20.04, and then putting Roundcube back.
The only problem I can see with having an empty ISPConfig 3.2 on a new 20.04 server and migrating is the hardware involved - I'd be installing it on different hardware than what I would want to run it on long-term. It is an option, and one I can try if all else fails.
Thanks,
Bob

 

Basically I think RC won't affect much of your server upgrade as it is merely a website. So long you have your database and mail server are backup properly, you shouldn't be worried in upgrading to Ubuntu 20.04. You can surely rebuild RC after you finished with the upgrade.

 

Well, I'm almost there. I've upgraded the server and gotten almost everything working again. Except for SMTP authentication and ISPConfig.
I can receive email and it shows up in my Roundcube inbox. When I try to send email, I get a SMTP Error 535: Authentication failed. I send email out through Google's SMTP server, so it may be failing there. I believe I made that setting in ISPConfig, but when I go to ISPConfig, I just get a blank screen, no login prompt - nothing.

Is there something that didn't get put in the right place for the ISPConfig web page to come up? The two ispconfig files are in the Apache sites-enabled folder. There's nothing in the /var/www/html folder, which is the root web folder.

Here is master.cf:

Code:

#
# Postfix master process configuration file.  For details on the format
# of the file, see the master(5) manual page (command: "man 5 master" or
# on-line: http://www.postfix.org/master.5.html).
#
# Do not forget to execute "postfix reload" after editing this file.
#
# ==========================================================================
# service type  private unpriv  chroot  wakeup  maxproc command + args
#               (yes)   (yes)   (no)    (never) (100)
# ==========================================================================
smtp      inet  n       -       y       -       -       smtpd
#smtp      inet  n       -       y       -       1       postscreen
#smtpd     pass  -       -       y       -       -       smtpd
#dnsblog   unix  -       -       y       -       0       dnsblog
#tlsproxy  unix  -       -       y       -       0       tlsproxy
submission inet n       -       y       -       -       smtpd
  -o syslog_name=postfix/submission
  -o smtpd_tls_security_level=encrypt
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
#  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
smtps     inet  n       -       y       -       -       smtpd
  -o syslog_name=postfix/smtps
  -o smtpd_tls_wrappermode=yes
  -o smtpd_sasl_auth_enable=yes
  -o smtpd_client_restrictions=permit_sasl_authenticated,reject
#  -o smtpd_reject_unlisted_recipient=no
#  -o smtpd_client_restrictions=$mua_client_restrictions
#  -o smtpd_helo_restrictions=$mua_helo_restrictions
#  -o smtpd_sender_restrictions=$mua_sender_restrictions
#  -o smtpd_recipient_restrictions=
#  -o smtpd_relay_restrictions=permit_sasl_authenticated,reject
#  -o milter_macro_daemon_name=ORIGINATING
#628       inet  n       -       y       -       -       qmqpd
pickup    unix  n       -       y       60      1       pickup
cleanup   unix  n       -       y       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
#qmgr     unix  n       -       n       300     1       oqmgr
tlsmgr    unix  -       -       y       1000?   1       tlsmgr
rewrite   unix  -       -       y       -       -       trivial-rewrite
bounce    unix  -       -       y       -       0       bounce
defer     unix  -       -       y       -       0       bounce
trace     unix  -       -       y       -       0       bounce
verify    unix  -       -       y       -       1       verify
flush     unix  n       -       y       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       y       -       -       smtp
relay     unix  -       -       y       -       -       smtp
#       -o smtp_helo_timeout=5 -o smtp_connect_timeout=5
showq     unix  n       -       y       -       -       showq
error     unix  -       -       y       -       -       error
retry     unix  -       -       y       -       -       error
discard   unix  -       -       y       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       y       -       -       lmtp
anvil     unix  -       -       y       -       1       anvil
scache    unix  -       -       y       -       1       scache
#
# ====================================================================
# Interfaces to non-Postfix software. Be sure to examine the manual
# pages of the non-Postfix software to find out what options it wants.
#
# Many of the following services use the Postfix pipe(8) delivery
# agent.  See the pipe(8) man page for information about ${recipient}
# and other message envelope options.
# ====================================================================
#
# maildrop. See the Postfix MAILDROP_README file for details.
# Also specify in main.cf: maildrop_destination_recipient_limit=1
#
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/bin/maildrop -d vmail ${extension} ${recipient} ${user} ${nexthop} ${sender}
#
# ====================================================================
#
# Recent Cyrus versions can use the existing "lmtp" master.cf entry.
#
# Specify in cyrus.conf:
#   lmtp    cmd="lmtpd -a" listen="localhost:lmtp" proto=tcp4
#
# Specify in main.cf one or more of the following:
#  mailbox_transport = lmtp:inet:localhost
#  virtual_transport = lmtp:inet:localhost
#
# ====================================================================
#
# Cyrus 2.1.5 (Amos Gouaux)
# Also specify in main.cf: cyrus_destination_recipient_limit=1
#

And here is main.cf

Code:

# See /usr/share/postfix/main.cf.dist for a commented, more complete version


# Debian specific:  Specifying a file name will cause the first
# line of that file to be used as the name.  The Debian default
# is /etc/mailname.
myorigin = /etc/mailname

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no

# appending .domain is the MUA's job.
append_dot_mydomain = no

# Uncomment the next line to generate "delayed mail" warnings
#delay_warning_time = 4h

readme_directory = /usr/share/doc/postfix

# TLS parameters
smtpd_tls_cert_file = /etc/postfix/smtpd.cert
smtpd_tls_key_file = /etc/postfix/smtpd.key
smtpd_use_tls = yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache

# See /usr/share/doc/postfix/TLS_README.gz in the postfix-doc package for
# information on enabling SSL in the smtp client.

smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = fileserver.thecovey.net
alias_maps = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases
alias_database = hash:/etc/aliases, hash:/var/lib/mailman/data/aliases
mydestination = fileserver.thecovey.net, localhost, localhost.localdomain
mynetworks = 127.0.0.0/8, 192.168.1.0/24
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = ipv4
html_directory = /usr/share/doc/postfix/html
virtual_alias_domains =
virtual_alias_maps = hash:/var/lib/mailman/data/virtual-mailman, proxy:mysql:/etc/postfix/mysql-virtual_forwardings.cf, proxy:mysql:/etc/postfix/mysql-virtual_email2email.cf
virtual_mailbox_domains = proxy:mysql:/etc/postfix/mysql-virtual_domains.cf
virtual_mailbox_maps = proxy:mysql:/etc/postfix/mysql-virtual_mailboxes.cf
virtual_mailbox_base = /var/vmail
virtual_uid_maps = mysql:/etc/postfix/mysql-virtual_uids.cf
virtual_gid_maps = mysql:/etc/postfix/mysql-virtual_gids.cf
sender_bcc_maps = proxy:mysql:/etc/postfix/mysql-virtual_outgoing_bcc.cf
smtpd_sasl_auth_enable = yes
smtpd_sasl_authenticated_header = yes
smtpd_restriction_classes = greylisting
greylisting = check_policy_service inet:127.0.0.1:10023
smtpd_recipient_restrictions = permit_mynetworks, permit_sasl_authenticated, reject_unauth_destination, reject_rbl_client zen.spamhaus.org, check_recipient_access mysql:/etc/postfix/mysql-virtual_recipient.cf, check_recipient_access mysql:/etc/postfix/mysql-virtual_policy_greylist.cf
smtpd_tls_security_level = may
transport_maps = hash:/var/lib/mailman/data/transport-mailman, proxy:mysql:/etc/postfix/mysql-virtual_transports.cf
relay_domains = mysql:/etc/postfix/mysql-virtual_relaydomains.cf
relay_recipient_maps = mysql:/etc/postfix/mysql-virtual_relayrecipientmaps.cf
smtpd_sender_login_maps = proxy:mysql:/etc/postfix/mysql-virtual_sender_login_maps.cf
proxy_read_maps = $local_recipient_maps $mydestination $virtual_alias_maps $virtual_alias_domains $sender_bcc_maps $virtual_mailbox_maps $virtual_mailbox_domains $relay_recipient_maps $relay_domains $canonical_maps $sender_canonical_maps $recipient_canonical_maps $relocated_maps $transport_maps $mynetworks $smtpd_sender_login_maps
smtpd_helo_required = yes
smtpd_helo_restrictions = permit_sasl_authenticated, permit_mynetworks, check_helo_access regexp:/etc/postfix/helo_access, reject_invalid_hostname, reject_non_fqdn_hostname, reject_invalid_helo_hostname, reject_unknown_helo_hostname, check_helo_access regexp:/etc/postfix/blacklist_helo
smtpd_sender_restrictions = check_sender_access regexp:/etc/postfix/tag_as_originating.re , permit_mynetworks, permit_sasl_authenticated, check_sender_access mysql:/etc/postfix/mysql-virtual_sender.cf, check_sender_access regexp:/etc/postfix/tag_as_foreign.re
smtpd_client_restrictions = check_client_access mysql:/etc/postfix/mysql-virtual_client.cf
smtpd_client_message_rate_limit = 100
maildrop_destination_concurrency_limit = 1
maildrop_destination_recipient_limit = 1
virtual_transport = dovecot

header_checks = pcre:/etc/postfix/header_checks
mime_header_checks = pcre:/etc/postfix/mime_header_checks
nested_header_checks = pcre:/etc/postfix/nested_header_checks
body_checks = pcre:/etc/postfix/body_checks

owner_request_special = no
smtp_tls_security_level = may
smtpd_tls_mandatory_protocols = !SSLv2, !SSLv3
smtpd_tls_protocols = !SSLv2,!SSLv3
smtp_tls_protocols = !SSLv2,!SSLv3
smtpd_tls_exclude_ciphers = RC4, aNULL
smtp_tls_exclude_ciphers = RC4, aNULL
dovecot_destination_recipient_limit = 1
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
content_filter = amavis:[127.0.0.1]:10024
receive_override_options = no_address_mappings
smtp_generic_maps = hash:/etc/postfix/generic

home_mailbox = Maildir/

# Postfix configuration to relay e-mail through ISP

relayhost = [smtp.gmail.com]:587
smtp_sasl_auth_enable = yes
smtp_sasl_password_maps = hash:/etc/postfix/sasl_passwd
smtp_sasl_security_options =
# Enable STARTTLS encryption
smtp_use_tls = yes

# where to find CA certificates
smtp_tls_CAfile = /etc/ssl/certs/ca-certificates.crt

# Postfix configuration as SMTP server

#smtpd_sasl_local_domain =
#smtpd_sasl_security_options =
broken_sasl_auth_clients = yes
smtpd_sasl_auth_enable = yes

message_size_limit = 0

 

You need to force update ISPConfig reconfiguring all services after upgrading Ubuntu.

 

I did that - ispconfig_update.sh --force. Seemed to complete without errors after I installed curl and a couple of PHP packages.

 

Have you done

 

I figured it out. The roundcube configuration wasn't passing the username and password when it tried to connect. Once I updated config.inc.php to include this:

$config['smtp_user'] = '%u';
$config['smtp_pass'] = '%p';​

I was able to send email.

 

ISPConfig's web interface still doesn't come up, however.

 

Is your web server up? Are other websites on that server working properly? If you try to access it locally, have you tried to access it externally? Have you try accessing via ip address instead?

 

Did you read #8 and did you run the update with reconfigure services AFTERWARDS?

 

I executed the "ispconfig_update --force" afterwards, if that's what you mean.
Bob

 

Here's the last few lines of the apache error log. Seems to have some problems with certificate stapling, whatever that is. I don't have any other websites - I use this as an email consolidator and file server for our home network. I tried connecting via the IP address, got a warning about the certificate, and then a blank screen when I proceeded. The last line of the log says the root of the web server doesn't have an index file in it, which I mentioned earlier. I have anonymized the lines since I'm posting it publicly.

Code:

[Mon Sep 06 10:08:11.565016 2021] [ssl:error] [pid 61805] AH02604: Unable to configure certificate ********.thecovey.net:8080:0 for stapling
[Mon Sep 06 10:08:11.567077 2021] [ssl:warn] [pid 61805] AH01906: ********.thecovey.net:8081:0 server certificate is a CA certificate (BasicConstraints: CA == TRUE !?)
[Mon Sep 06 10:08:11.567354 2021] [ssl:error] [pid 61805] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: emailAddress=*******@triad.rr.com,CN=*******.thecovey.net,OU=IT Department,O=Home,L=City,ST=State,C=US / issuer: <redacted>,C=US / serial: 948493D8D6B0057E / notbefore: Nov 19 03:48:43 2016 GMT / notafter: Nov 17 03:48:43 2026 GMT]
[Mon Sep 06 10:08:11.567416 2021] [ssl:error] [pid 61805] AH02604: Unable to configure certificate ********.thecovey.net:8081:0 for stapling
[Mon Sep 06 10:08:11.567697 2021] [:error] [pid 61805] python_init: Python version mismatch, expected '2.7.17', found '2.7.18'.
[Mon Sep 06 10:08:11.567896 2021] [:error] [pid 61805] python_init: Python executable found '/usr/bin/python'.
[Mon Sep 06 10:08:11.567914 2021] [:error] [pid 61805] python_init: Python path being used '/usr/lib/python2.7:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload'.
[Mon Sep 06 10:08:11.567416 2021] [ssl:error] [pid 61805] AH02604: Unable to configure certificate ********.thecovey.net:8081:0 for stapling
[Mon Sep 06 10:08:11.567697 2021] [:error] [pid 61805] python_init: Python version mismatch, expected '2.7.17', found '2.7.18'.
[Mon Sep 06 10:08:11.567896 2021] [:error] [pid 61805] python_init: Python executable found '/usr/bin/python'.
[Mon Sep 06 10:08:11.567914 2021] [:error] [pid 61805] python_init: Python path being used '/usr/lib/python2.7:/usr/lib/python2.7/plat-x86_64-linux-gnu:/usr/lib/python2.7/lib-tk:/usr/lib/python2.7/lib-old:/usr/lib/python2.7/lib-dynload'.
[Mon Sep 06 10:08:11.567966 2021] [:notice] [pid 61805] mod_python: Creating 8 session mutexes based on 150 max processes and 0 max threads.
[Mon Sep 06 10:08:11.567986 2021] [:notice] [pid 61805] mod_python: using mutex_directory /tmp
[Mon Sep 06 10:08:11.605602 2021] [mpm_prefork:notice] [pid 61805] AH00163: Apache/2.4.41 (Ubuntu) mod_fcgid/2.3.9 OpenSSL/1.1.1f mod_python/3.3.1 Python/2.7.18 configured -- resuming normal operations
[Mon Sep 06 10:08:11.605683 2021] [core:notice] [pid 61805] AH00094: Command line: '/usr/sbin/apache2'
[Mon Sep 06 10:10:02.303655 2021] [autoindex:error] [pid 61813] [client ::1:37218] AH01276: Cannot serve directory /var/www/html/: No matching DirectoryIndex (index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm,index.html,index.cgi,index.pl,index.php,index.xhtml,index.htm,standard_index.html) found, and server-generated directory index forbidden by Options directive

 

I've made some progress with email. Whatever was supposed to populate dh.pem created an empty file. I found a thread where Till mentioned how to populate it using Openssl. When I did that, Dovecot is much happier and sends and receives email happily. Still no web interface to ISPConfig, however.

 

Did you opt to have ISPConfig installed on the server (the panel)

 

The server has been running ISPConfig for years. I was running version 3.2.2 on Ubuntu 18.04, and was upgrading the server to Ubuntu 20.04. I didn't reinstall it, no. I could access it on port 8080 prior to the OS upgrade.

 

last chance: did you follow #8?

 

I think he did not though we all know that following every steps in Ubuntu 20.04 tutorial is necessary as well reinstalling ISPConfig and reconfiguring all services thereafter.

 

I was rapidly coming to the conclusion that even though I had an up-to-date version of ISPConfig installed before the update, that it needed to be reinstalled after the update.
I appreciate all the help and the work behind the Perfect Server and all the other instructions you guys have built. It would be nice if the upgrade process was a little more documented. It's the "we all know" part that isn't necessarily communicated. When I went through the 20.04 tutorial, I found that everything that had been done by the 18.04 tutorial was already done and there were just a few odds and ends of things that needed to be done. My assumption - and it was a bad one, and my fault - was that ISPConfig was still properly configured. It isn't, and I'll fix that.
Thanks,
Bob