ISPConfig could be great but its riddled with issues. How to fix?

Discussion in 'Installation/Configuration' started by yucktoufoo, Feb 14, 2011.

  1. yucktoufoo

    yucktoufoo Member

    I've installed a multiserver setup (Centos 5.5) as per the guides (which are very good) but it seems I get different results every time.

    Firstly, built a BIND DNS server and the monitor page repeatedly tells me MyDNS isn't running, also drops the zone files in the wrong place for a chrooted bind install.

    Then built a mailserver, setup my domains to find I am unable to add a fetchmail setup with an alias address as the destination mailbox. Perhaps there is some golden rule about not doing this but I modified the database by hand and everything seems to work as expected.

    Then built a secondary mailserver only to find I am unable to host the same domain on both servers (i do not wish the two servers to be mirrors, i just want to host a single domain on both servers).

    Then I see I am unable to receive mail at the latest mail server I built, getting the error

    Code:
    pipe_command: execvp /usr/bin/maildrop: No such file or directory
    I thought that path was just incorrect, and that is should be /var/spool/postfix/maildrop but when I started to poke around in the server table and found the following in the [MAIL] section...

    Code:
    pop3_imap_daemon=courier
    mail_filter_syntax=maildrop
    
    Well I don't know why it thinks courier is used, its a brand new build and courier has NEVER been installed. I guess it should read...

    Code:
    pop3_imap_daemon=dovecot
    mail_filter_syntax=sieve
    
    I also noted that removing uninstalling and reinstalling ispconfig on the remote mailserver results in another server added to the UI. This seems a bit silly to me if the server has the same hostname and ip addresses.

    There are loads of other things I've had to manually modify in the database but that's really scary as some of these things can easily be "un-done" by innocently updating an alias record in the UI or by an ispconfig update.

    I think its odd that someone would store an entire ini file in a single column in a database table. Everything is also done with raw SQL in the PHP scripts, now I've never developed anything in PHP by in my past experience, if your database engine supports stored procedures then you should use them. It would certainly make fudging the database a bit easier.

    I am also amazed that there are no foreign key constraints on the database. That could be a real problem for a big installation with lots of managed hosts.

    I did head over to the bugtracker site for ispconfig but I wasnt able to add any new tickets and any existing tickets in the system didn't look like they were being worked on.

    Now, I assume its bad mojo and deemed UNSAFE to edit the content of the dbispconfig database manually, I know this but I have to make changes to get it to work properly. I'd like other peoples thoughts on how likely it is that these modifications will be overwritten by ispconfig updates....

    As for my alias problem, I am thinking of adding a trigger to the mail_get table to ensure that the destination address is always forced to the correct alias address of someone updates the row.

    Add a stored procedure or trigger to parse the config column of the server table to replace any instances of 'MyDNS' with 'bind' instead as I do not use MyDNS. I will probably do the same again to fix my maildrop issue.

    I'd like to know how data flows between the various instances of mysql. I will do some digging through the PHP code later but if someone has any quick answers on this I would be greatful. Presumably the remote installations connect back to main ISPConfig database (as I've only allowed MySQL from client to server), push some stuff into the central database and pull any changes down at the same time?

    On this basis, when I make the changes to the config column, am I better to do this on the remote server or the central ISPConfig server?

    I really don't know what I will do to work around hosting the same email domain on two different servers, any pointers greatly appreciated.
     
  2. till

    till Super Moderator Staff Member ISPConfig Developer

    They are all being worked. If you see our blog you can see that we release new bugfix releases every 1-2 months. Yyou can post new items after you registered yourself in the bugtracker.

    Thats not odd at all and it works fine. Its much better to use a ini file instead of having a new column for every entry and the ini file is stored in mysql and not the filesystem as this includes it into the internal replication mechanism of ispconfig.

    Yes it is unsafe and will break your setup. If you think that there is something missing, feel free to make a feature request in the bugtracker.

    Yes. This makes the setup very fault tolerant. The master server keeps a transaction log and when a slave is offline for a while, it will sync itself with the master when it come back online.

    Regarding your mail issues, please look at the setup guides and install your system exactly as described there. If you install packages in different locations or outside the path, then they can not be recognized by the installer.

    This is not nescessary if you setup the server correctly before you install ispconfig. The issues that you describe are all caused by a incompatible setup which makes it impossible for the ispconfig installer to recognize the system components. For a correct setup, please read the perfect setup guides.

    Thats covered in this guide:

    http://www.howtoforge.com/installin...tabase-cluster-on-debian-5.0-with-ispconfig-3

    ISPConfig is doing all the configuration mirroring for as many mirrors as you need. Then a shared filesystem that is used for the /var/vmail directory is used for data mirroring.
     
  3. yucktoufoo

    yucktoufoo Member

    I will check this out. I registered for the bug tracker but was unable to find any options for creating a new ticket.


    I've worked on possibly some of the largest database installations in the UK and disagree with this comment but I guess we will have to agree to disagree ;)


    I will go back over the guide but I have followed it several times now and am 99.9998% sure that I did everything as per the guide, certainly for the DNS servers which think they have MyDNS installed. The latest mailserver was a 32bit build rather than 64bit, perhaps this is where I have problems?


    Hmm, I've followed that guide numerous times now so Im fairly sure I am doing everything correctly. How exactly does ispconfig determine if mydns is installed?




    I dont have time to check that out but will have a look later this evening but I am not sure this is what I want, I do not want to mirror any of the mail data. The second server is internal and has no knowledge of the external server. The internal server should getmail from the external and simply inject it into the local server under the same recipient domain.... is this not supported?
     
  4. till

    till Super Moderator Staff Member ISPConfig Developer

    There should be a "Add New Task" Button in the upper menu after you logged in.

    I think so :) I guess these were not mysql setups? Mysql has limits in the overall length of column names for a database table and we also needed a flexible solution.

    Thats should not make a difference. Please check with the command:

    which dovecot

    if the dovecot binary is in the path.

    Thats not supported, at least not without mirroring as ispconfig ensures for consistency that a domain can only be added once per cluster.

    Why do you want to use getmail for this setup instead of a postfix transport? With a postfix transport, you setup a transport on the external server for the domain, then add the allowed email addresses as relay recipients. Then add the domain as email domain for the internal server in ispconfig and add mailboxes.
     
  5. yucktoufoo

    yucktoufoo Member

    Forgot to query... Does this include not being able to set an alias/forward address as a destination for a new fetchmail setup as I think thats a bug/missing feature rather than an implementation thing.
     
  6. till

    till Super Moderator Staff Member ISPConfig Developer

    Adding a fetchmail job for the alias of a mailbox instead of the mailbox itself makes the list much larger as users tend to have many aliases for a single mailbox and so might become confusing. Thats why we have not added it.

    But there is no technical reason why it should not work for aliases as well.
     
  7. yucktoufoo

    yucktoufoo Member

    A mixture of MySQL, MSSQL & Postgres & DB2 and in some cases all flavours together. Some of these others have stupidly impossible limits on naming too :)


    Ahh.. now see, here's the thing, which is not installed. Presumably that answers my question although does this mean if dovecot is not found, ispconfig automatically assumes that courier is installed instead?

    That leads me to think there are two problems here. ISPConfig should complain if which is not installed and also probably shouldn't just default to courier.


    There are a few reasons for this. Initially I had no access to the external mail server to be able to configure transports so the internal server was simply configured to getmail from the external instead. Now that I am building a new external server I could do it via transport as you said but this would also require ports to be opened on our firewall which I was trying to avoid.


    Aside from all of these small problems, where is the best place to get more documentation for under the bonnet of ispconfig? its incredibly difficult to support something when you don't understand how it works.

    I purchased the user manual which will give me some bed time reading this week but I am looking for more in-depth knowledge of what happens behind the scenes.
     
  8. yucktoufoo

    yucktoufoo Member

    This is what I thought, for me it is not a problem to have big lists because I host a very controlled set of domains but perhaps if I revisit my getmail/transports setup as per your previous posting this issue will be null and void anyway.
     
  9. till

    till Super Moderator Staff Member ISPConfig Developer

    Yes, we should add that. As courier is the default pop3/imap daemon. Same with mydns and BIND.

    We should add that. Even if I've never seen a system yet where the Linux base utils had been missing.

    Most infos are here in the forum. We do not have a full technical documentation available yet.

    The basics are:

    ISPConfig consists of two, mostly independant parts. The Server and the interface (/usr/local/ispconfig/interface and /usr/local/ispconfig/server). Server and interface communicate trogh the mysql db.

    For the configuration, the most interesting part is the server. It is a very modular system consisting of modules and plugins (/usr/local/ispconfig/server/mods-available and /usr/local/ispconfig/server/plugins-available). If a plugin or module is enabled in your setup, then it has a symlink from the ...-enabled directory to the ...-available directory.
    The modules corresposnd to the interface modules like dns, mail, web and database. So a server cluster can split into these parts. The modules provide / raise events like "mail_user_insert" or "mail_user_update" and the plugins can subscribe to these events and get called when such a event occurs. So the plugins do the actual config work by creating directories, copying files and executing commands etc.

    So as an (very simplified) example, the difference between a mydns, powerdns and bind setup is basically which server plugin is enabled.

    mydns: no plugin as mydns reads directly out of the db and needs no further configuration.
    powerdns: Powerdns has its own mysql database as the ispconfig db layout is not compatible, so the powerdns plugin just syncs the records in the powerdns db with the ones in the ispconfig db.
    Bind: The bind plugin writes / updates BIND config files when a DNS record gets changed. The place for the BIND config directrory can be configured in ispconfig under System > server config.
     
  10. yucktoufoo

    yucktoufoo Member

    That system is a VPS provided by bytemark which I re-imaged this morning and would also have expected which to have been installed so yeah, probably should check for this.


    I can't help but think that the whole process would have been much easier had I have used debian over CentOS. Either way it was a worth excersize and I've learnt a little in the process so can't be all bad.

    Thanks very much for your assistance. I look forward to the day when there is some official tech documents but in the meantime I will take on board what you have said, between that and poking at the code im sure it will all become clearer over time.
     
  11. till

    till Super Moderator Staff Member ISPConfig Developer

    Debian is much easier indeed. ISPConfig is developed on Debain so ist the most easy to install and stable platform for ispconfig.
     
  12. yucktoufoo

    yucktoufoo Member

    Just to re-cap....

    Till is absolutely correct, with the exception of the alias things everything else was due to install/distro issues so I take back my unjustified rant ;)

    Now my only outstanding issue is after upgrading everything to 3.0.3.2 using the update shell script.

    The DNS server is switching between red and green in the monitor page. When it's red it tells me there is a problem in the system log.

    When the status is green the error is still in the log but I notice that if I select memory/cpu/ANY stats for the DNS server:

    Code:
    No data about the memory usage available at the moment. Please check again later.
    

    The error in the system log reads...

    Code:
    Replication failed. Error: (server) in mysql server: (localhost) Unknown column 'dbversion' in 'field list' # SQL: REPLACE INTO server 
    
    That's a little odd because I have checked the local and remote database and both server tables have the dbversion column.

    I guess this was most likely caused by forgetting to update the dns server before updating the central server but that error should be clear now.it doesnt seem to cause any issues other than the warning in the monitor page. My zone file updates are going out as expected.

    I also have something outstanding in the job queue for a few days

    Code:
    2011-02-10 21:31 	cmname01.agrinet.co.uk 	Update 	server
    
    Whether this is realted or not I don't know but I also notice after the upgrade that the servers listed in the monitor page do not always display their ISPConfig Database version.
    
    
    Till, do you have any suggestions.
     
    Last edited: Feb 15, 2011
  13. till

    till Super Moderator Staff Member ISPConfig Developer

    The system stops the replication in case of a fatal error like this until the admin confirms that the error is fixed. The confirmation is done by clicking on the "Delete" bitton that is in the column of the error in the system log in the ispconfig monitor.
     
  14. yucktoufoo

    yucktoufoo Member

    Thanks, thats sorted that but does the monitor behaviour sound correct.

    Basically, if a server is red then thats fine, I can get details about the error/load/disk/memory

    If the server status is green it seems to switch between having data for load/disk/cpu and not not having any.

    I noticed, when the servers are not displaying their ISPConfig version number is when I cannot get any data about them.

    Surely that's not normal? Again this is an assumption but I guess the central server gets notified about load/disk/memory and that data should always be visible until the next db sync?

    What I mean is, once disk data for a server is sent to the central database it should continue to exist right? It would never be deleted by the replication system or anything before the next update comes in from the monitored host?
     
  15. yucktoufoo

    yucktoufoo Member

    Till are you about?

    I am still struggling with DNS server monitoring. The interface repeatedly tells me that the DNS server is offline when it is most definitely up and running.

    I've checked the INI config which is correctly set to named and I and perform lookups at the DNS server from the control panel server


    Code:
    [root@CMISPC01 ~]# dig @CMNAME01 .
    
    ; <<>> DiG 9.3.6-P1-RedHat-9.3.6-4.P1.el5_4.2 <<>> @CMNAME01 .
    ; (1 server found)
    ;; global options:  printcmd
    ;; Got answer:
    ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 49027
    ;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 0
    
    ;; QUESTION SECTION:
    ;.                              IN      A
    
    ;; AUTHORITY SECTION:
    .                       9744    IN      SOA     a.root-servers.net. nstld.verisign-grs.com. 2011021701 1800 900 604800 86400
    
    ;; Query time: 2 msec
    ;; SERVER: 192.168.0.108#53(192.168.0.108)
    ;; WHEN: Fri Feb 18 14:51:39 2011
    ;; MSG SIZE  rcvd: 92
    
    
    What am I missing here?
     
  16. falko

    falko Super Moderator ISPConfig Developer

    What are the outputs of
    Code:
    netstat -tap
    and
    Code:
    netstat -uap
    ?
     
  17. yucktoufoo

    yucktoufoo Member

    Hi Falko, here is the output requested....

    Code:
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
    tcp        0      0 0.0.0.0:3306                0.0.0.0:*                   LISTEN      2850/mysqld
    tcp        0      0 127.0.0.1:587               0.0.0.0:*                   LISTEN      2921/exim
    tcp        0      0 127.0.0.1:465               0.0.0.0:*                   LISTEN      2921/exim
    tcp        0      0 192.168.0.108:53            0.0.0.0:*                   LISTEN      19214/named
    tcp        0      0 127.0.0.1:953               0.0.0.0:*                   LISTEN      19214/named
    tcp        0      0 127.0.0.1:25                0.0.0.0:*                   LISTEN      2921/exim
    tcp        0      0 192.168.0.108:46650         192.168.0.109:3306          TIME_WAIT   -
    tcp        0      0 ::1:587                     :::*                        LISTEN      2921/exim
    tcp        0      0 ::1:465                     :::*                        LISTEN      2921/exim
    tcp        0      0 ::1:53                      :::*                        LISTEN      19214/named
    tcp        0      0 :::22                       :::*                        LISTEN      2763/sshd
    tcp        0      0 ::1:953                     :::*                        LISTEN      19214/named
    tcp        0      0 ::1:25                      :::*                        LISTEN      2921/exim
    tcp        0    296 ::ffff:192.168.0.108:22     ::ffff:192.168.0.10:51610   ESTABLISHED 20183/sshd: foo
    Active Internet connections (servers and established)
    Proto Recv-Q Send-Q Local Address               Foreign Address             State       PID/Program name
    udp        0      0 192.168.0.108:53            0.0.0.0:*                               19214/named
    udp        0      0 0.0.0.0:35896               0.0.0.0:*                               2999/avahi-daemon:
    udp        0      0 0.0.0.0:68                  0.0.0.0:*                               2401/dhclient
    udp        0      0 0.0.0.0:5353                0.0.0.0:*                               2999/avahi-daemon:
    udp        0      0 :::56985                    :::*                                    2999/avahi-daemon:
    udp        0      0 ::1:53                      :::*                                    19214/named
    udp        0      0 :::5353                     :::*                                    2999/avahi-daemon:
    
     
  18. till

    till Super Moderator Staff Member ISPConfig Developer

    It seems as if your DNS server is not listening on 127.0.0.1:53. ISPConfig uses TCP port 53 on localhost (127.0.0.1) to check if DNS is running.

    Please take a look in the named.conf file and enable BIND to listen on IP 127.0.0.1 too.
     
  19. yucktoufoo

    yucktoufoo Member

    Oh ok, I see. Perhaps that might be changed in a future release? Is it not better to monitor the DNS (or any monitored) service from the control panel or at least to provide that option. Testing against the loopback doesn't give you any indication that the DNS service is available to the world.

    I will modify my configuration now, thanks for your help :)
     

Share This Page