Backup (DNS and Mail)

I ran into a small problem when I want to add a slave zone (dns_slave_add) I did set all the parameters found in the file dns_slave.tform.php

Code:

 'server_id' => 1,
 'origin' => $origin,
 'ns' => $ns,
 'xfer' => $xfer,
 'active' => 'Y'

But I get an error: ns_error_regex
I will search to code if I can find something that I did wrong, but maybe someone in here has more experience.

 

Sure, here they are
[server_id] => 1
[origin] => domain.com.
[ns] => ns2.nameserver.com
[xfer] => 10.178.34.110
[active] => Y

I changed the origin and ns, but in the same format

 

Thank you for your reply.
I'm sorry that didn't resolve the error

 

I think I found the problem in dns_slave.tform.php for ns I see 'type' => 'ISIP' does this mean there can only be an IP not a domain?

 

Thanks, it was indeed the IP issue.
I have managed to complete the script, it works fine, master syncs the slave, so it does everything I expected so far.
Are you still interested in the script?

 

Code:

<?php
/*
Copyright (c) 2020 B. Wubben

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Script for updating the Slave DNS zone records (create and remove)



*/
// Config the Secondary DNS servers here where
// array (ServerID (does not have to be the ispconfig ID,actually it's not used),host or IP,Port,loginname,loginpass,Check certificate)
$secondaryservers=array(
    array(1,'','8080','','','N')
);

/* Soap Settings */
// Local
// Local username
$username_local = 'localuser';
// Local password
$password_local = 'localpassword';
// Local Soap location
$soap_location_local = 'https://localhost:8080/remote/index.php';
// Local Soap URI
$soap_uri_local = 'https://localhost:8080/remote/';
// Local connection timeout in seconds
$soap_timeout_local = 4;
// The IP where I comunnicate with the other Nameservers
$my_notify_ip = 'myip';
// Owner of the records
$my_client_id = 1;

// Remote
// Remote connection timeout in seconds
$soap_timeout_remote = 8;

/* End Soap settings */

// Functions
function SoapConnect($location,$uri,$checkSSL,$timeout = 60) {
    try {
            if ($checkSSL != 'Y') {
                    $con = new SoapClient(null, array('location' => $location,
                            'uri'      => $uri,
                               'trace' => 1,
                            'exceptions' => 1,
                            'connection_timeout'=> $timeout,
                            'stream_context'=> stream_context_create(array('ssl'=> array('verify_peer'=>false,'verify_peer_name'=>false)))
                               ));

            } else {
                    $con = new SoapClient(null, array('location' => $location,
                            'uri'      => $uri,
                            'trace' => 1,
                            'exceptions' => 1,
                            'connection_timeout'=> $timeout
                               ));
        }

    } catch (SoapFault $e) {
        $con = NULL;
    }
    return $con;
}
// End functions

// First get the remote DNS record
$errorserver=array();
$dns_zone_remote=array();
foreach ($secondaryservers as $secondaryserver) {
    if ($secondaryserver[2]!='') {
        $http_s='https://'.$secondaryserver[1].':'.$secondaryserver[2];
    } else {
        $http_s='https://'.$secondaryserver[1];
    }
    $client_remote = SoapConnect($http_s.'/remote/index.php',$http_s.'/remote/',$secondaryserver[5],$soap_timeout_remote);
    if (!is_null($client_remote)) {
        try {
                if($session_id = $client_remote->login($secondaryserver[3],$secondaryserver[4])) {
                        echo ("Logged in successfull on ".$secondaryserver[1]." Session ID:".$session_id." \r\n");
                }

                // Get all the master zone list local
                $dns_zones = $client_remote->dns_zone_get($session_id,-1);
            foreach ($dns_zones as $dns_zone) {
                // Check if I'm the notify server otherwise it has no use to act as secondary server
                if ((array_key_exists('xfer',$dns_zone)) && ($dns_zone['xfer'] == $my_notify_ip) && ($dns_zone['active'] == 'Y')) {
                    $dns_zone_remote[] = $dns_zone;
                }
            }
                // logout
                if($client_remote->logout($session_id)) {
                        echo ("Remote logged out from ".$secondaryserver[1].". \r\n");
                }
        } catch (SoapFault $e) {
                echo $client_remote->__getLastResponse();
                echo("SOAP Error (".$secondaryserver[1].") : ".$e->getMessage()."\r\n");
            // add this server to the errorlist
            $errorserver[]=$secondaryserver[1];
        }
    } else {
        // add this server to the errorlist
        $errorserver[]=$secondaryserver[1];
    }
}

// Make a local connection
$client_local = SoapConnect($soap_location_local,$soap_uri_local,'N',$soap_timeout_local);
// try to login
try {
    if($session_id = $client_local->login($username_local, $password_local)) {
        echo ("Local logged in successfull Session ID:".$session_id.". \r\n");
    }


    // Get all the master zone list local
    $dns_slave_records = $client_local->dns_slave_get($session_id,-1);

    // Add all the zone records from remote to local slave, we don't have to check for dupication in the primary zone, ISPConfig will do that for us
    foreach ($dns_zone_remote as $dns_zone) {
        // Check if the zone is allready in the list
        if (array_search($dns_zone['origin'], array_column($dns_slave_records, 'origin')) === false) {
            // Check what server this record is from, so we know it's nameserver hostname
            $key = array_search($dns_zone['xfer'], array_column($secondaryservers,1));
            $ns = $secondaryservers[$key][1];
            echo ("NS : $ns \r\n");
            // Add this record to the slave zone
            echo ($dns_zone['origin']." not found, add it \r\n");
            $params = array(
                'server_id' => 1,
                'origin' => $dns_zone['origin'],
                'ns' => $ns,
                'xfer' => '',
                'active' => 'Y'
            );
            $res = $client_local->dns_slave_add($session_id,$my_client_id,$params );
            if ($res == -1) {
                echo ($dns_zone['origin']." add error $res \r\n");
            } else {
                echo ($dns_zone['origin']." added to slave \r\n");
            }
        }
    }

    // Now delete if its not in the list
    foreach ($dns_slave_records as $dns_zone) {
        if (array_search($dns_zone['origin'], array_column($dns_zone_remote, 'origin')) === false) {
            // Check if the record is not from an error server (server down = don't delete)
            if (in_array($dns_zone['ns'],$errorserver) === false) {
                // Delete the record
                echo ("Delete zone ".$dns_zone['origin']."\r\n");
                $res = $client_local->dns_slave_delete($session_id,$dns_zone['id']);
                            if ($res == -1) {
                                    echo ($dns_zone['origin']." delete error $res \r\n");
                            } else {
                                    echo ($dns_zone['origin']." deleted from slave \r\n");
                            }
            }
        }
    }


 
    // logout
    if($client_local->logout($session_id)) {
        echo ("Local logged out. \r\n");
    }
} catch (SoapFault $e) {
    echo $client_local->__getLastResponse();
    die('SOAP Error: '.$e->getMessage());
}
echo ("DNS Update Done \r\n");
?>

What needs to be configure is:
the remote server(s) $secondaryservers you can add as many ISPConfig servers you want, the server the script is running on will be the backup of that server, the 'N' at the end stands for do not check certificate (https), 'Y' means certificate must be ok.
$username_local and $password_local this needs to be the local soap login
$my_notify_ip this is the IP where the primary name server is sending the updates to (this IP is used to match the records, if it's not set correctly the slave records will be deleted)
$my_client_id this is the user id where the backup records are assigned to (I have made a server account that's suspended) and assigned the ID, so that nobody can delete the secondary records.
optional
$soap_timeout_local and $soap_timeout_remote, this is the timeout for the soap connection (note: this does not work correctly in PHP 5)

Please let me know if you are going to use the script..

 

Forgot to mention, this script can be run from the crontab

 

Ok, here is the script for mail back also, the configuration is almost the same, there is 1 extra setting.

Code:

<?php
/*
Copyright (c) 2020 B. Wubben

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.

Script for updating the Slave DNS zone records (create and remove)



*/
// Config the backup mail servers here where
// array (ServerID (does not have to be the ispconfig ID, but must be unique),IP,Port,loginname,loginpass,Check certificate,SMTP forward IP/Host)
$secondaryservers=array(
    array(1,'','8080','','','N','')
);

/* Soap Settings */
// Local
// Local username
// Local username
$username_local = 'localuser';
// Local password
$password_local = 'localpassword';
// Local Soap location
$soap_location_local = 'https://localhost:8080/remote/index.php';
// Local Soap URI
$soap_uri_local = 'https://localhost:8080/remote/';
// Local connection timeout in seconds
$soap_timeout_local = 4;
// The IP where I comunnicate with the other Nameservers
$my_notify_ip = 'myip';
// Owner of the records
$my_client_id = 4;
// Remote
// Remote connection timeout in seconds
$soap_timeout_remote = 8;

/* End Soap settings */

// Functions
function SoapConnect($location,$uri,$checkSSL,$timeout = 60) {
    try {
            if ($checkSSL != 'Y') {
                    $con = new SoapClient(null, array('location' => $location,
                            'uri'      => $uri,
                               'trace' => 1,
                            'exceptions' => 1,
                            'connection_timeout'=> $timeout,
                            'stream_context'=> stream_context_create(array('ssl'=> array('verify_peer'=>false,'verify_peer_name'=>false)))
                               ));

            } else {
                    $con = new SoapClient(null, array('location' => $location,
                            'uri'      => $uri,
                            'trace' => 1,
                            'exceptions' => 1,
                            'connection_timeout'=> $timeout
                               ));
        }

    } catch (SoapFault $e) {
        $con = NULL;
    }
    return $con;
}
// End functions

// First get the remote email domains and accounts
$errorserver=array();
$maildomain_remote=array();
foreach ($secondaryservers as $secondaryserver) {
    if ($secondaryserver[2]!='') {
        $http_s='https://'.$secondaryserver[1].':'.$secondaryserver[2];
    } else {
        $http_s='https://'.$secondaryserver[1];
    }
    $client_remote = SoapConnect($http_s.'/remote/index.php',$http_s.'/remote/',$secondaryserver[5],$soap_timeout_remote);
    if (!is_null($client_remote)) {
        try {
                if($session_id = $client_remote->login($secondaryserver[3],$secondaryserver[4])) {
                        echo ("Logged in successfull on ".$secondaryserver[1]." Session ID:".$session_id." \r\n");
                }

                // Get all the email domains
                $mail_domains = $client_remote->mail_domain_get($session_id,-1);
            $marr = array();
            foreach ($mail_domains as $mail_domain) {
                if (strtoupper($mail_domain['active']) == 'Y') {
                    $marr[] = $mail_domain['domain'];
                }
            }
            $maildomain_remote[$secondaryserver[6]] = $marr;
                // logout
                   if($client_remote->logout($session_id)) {
                        echo ("Remote logged out from ".$secondaryserver[1].". \r\n");
                }
        } catch (SoapFault $e) {
                echo $client_remote->__getLastResponse();
                echo("SOAP Error (".$secondaryserver[1].") : ".$e->getMessage()."\r\n");
            // add this server to the errorlist
            $errorserver[]=$secondaryserver[6];
        }
    } else {
        // add this server to the errorlist
        $errorserver[]=$secondaryserver[1];
    }
}


// Make a local connection
$client_local = SoapConnect($soap_location_local,$soap_uri_local,'N',$soap_timeout_local);
// try to login
try {
    if($session_id = $client_local->login($username_local, $password_local)) {
        echo ("Local logged in successfull Session ID:".$session_id.". \r\n");
    }


    // get the transprt list
    $mail_transports = $client_local->mail_transport_get($session_id,-1);
    // Get all the mail domains, we have to check later if this server is not allready serving this domain
    $maildomain_local = $client_local->mail_domain_get($session_id,-1);
    // Get the relay recipient list
    $mailrecipient_local = $client_local->mail_relay_recipient_get($session_id,-1);
    // Add all the mail domains from remote that's not in the mail transport list, also add the relay_recipient
    // first we loop thru all servers
    foreach ($maildomain_remote as $key => $server) {
        // now loop to all the maildomains
        foreach($server as $maildomain) {
            // search array
            if (array_search($maildomain, array_column($mail_transports, 'domain')) === false) {
                if (array_search($maildomain, array_column($maildomain_local, 'domain')) === false) {
                // add the record
                    $rec = array (
                        'server_id' => 1,
                        'domain' => $maildomain,
                        'transport' => 'smtp:['.$key.']',
                        'sort_order' => 5,
                        'active' => 'y'
                    );
                    $res = $client_local->mail_transport_add($session_id,$my_client_id,$rec);
                                   if ($res == -1) {
                                        echo ($maildomain." add error $res \r\n");
                                } else {
                                        echo ($maildomain." added to mail transport \r\n");
                                }
                    $rec = array (
                        'server_id' => 1,
                        'source' => '@'.$maildomain,
                        'active' => 'y'
                    );
                                    $res = $client_local->mail_relay_recipient_add($session_id,$my_client_id,$rec);
                                       if ($res == -1) {
                                            echo ($maildomain." add error $res \r\n");
                                    } else {
                                            echo ($maildomain." added to recipient \r\n");
                                    }

                } else {
                    echo ($maildomain." is allready beeing served on this server \r\n");
                }
            }

        }
    }
    // now loop thru the mail transport to delete what's  not in the maildomain remote
    foreach ($mail_transports as $mail_transport) {
        if (in_array($mail_transport['domain'],array_merge(...array_values($maildomain_remote)),true) == false) {
            $smtp_host = preg_match('~\[([^()]+)\]~',$mail_transport['transport'],$hm);
            if ((empty($hm)) || (in_array($hm[1],$errorserver) == false)) {
                // remove the mailtransport and relay recipient
                                $res = $client_local->mail_transport_delete($session_id,$mail_transport['transport_id']);
                                if ($res == -1) {
                                    echo ($mail_transport['domain']." (mailtransport) delete error $res \r\n");
                                } else {
                                        echo ($mail_transport['domain']." removed from mail transport \r\n");
                                }
                // find th relay recipient record
                $recipient_rec = array_search('@'.$mail_transport['domain'],array_column($mailrecipient_local,'source'));
                if ($recipient_rec != false) {
                    // found, now delete it
                                        $res = $client_local->mail_relay_recipient_delete($session_id,$mailrecipient_local[$recipient_rec]['relay_recipient_id']);
                                        if ($res == -1) {
                                                echo ($mail_transport['domain']." (relay recipient) delete error $res \r\n");
                                        } else {
                                                echo ($mail_transport['domain']." removed from relay recipient \r\n");
                                        }

                }
            }
        }
    }
    // logout
    if($client_local->logout($session_id)) {
        echo ("Local logged out. \r\n");
    }
} catch (SoapFault $e) {
    echo $client_local->__getLastResponse();
    die('SOAP Error: '.$e->getMessage());
}
echo ("Mail backup update Done \r\n");
?>

 

The explanation of the settings you can find in post #31
There is on more item added to the "$secondaryservers" array: SMTP forward IP/Host, this is the IP or hostname the SMTP server will forward the mails to (for example: public IP or MX1 record)

Please let me know if you are going to use the code, you can run this from the cron. both scripts can run as often as you like, it will check for changes, if there are non, the script won't do anything.
If the remote server is not responding, it will not delete the records (otherwise the backup would not make sense at all)
This is the same for the DNS script.

With these 2 scripts you can make one ISPConfig backup of another ISPConfig server but they will be standalone and won't share customer details, except DNS and mail settings, but not login details etc.

Enjoy

 

Fascinating thread with a great finish! I am working intensely on backup and recovery right now and need to post another thread on this topic. But I absolutely will go through the code and other notes in this thread for an education.

Fun thoughts...
I think what @PDJ has been describing is "ISPConfig for ISPConfig" Where ISPC serves as a controller for common websites, PDJ has a couple ISPC instances where it seems a centralized controller could be used to keep them in sync - only for specific components. Picture a UI, in or like the ISPConfig UI, that has tabs and fields to administer a collection of other ISPConfig instances. ISPC already supports multiple role tiers: There is the ISPC administrator, who sells to resellers, and they sell to clients. This is just one level above that. That would be a cool module ... for a very limited audience!

 

Thank you for your reply.
The idea behind this is actually de-central, if you have 2 or more ISPConfigs on different locations and you don't want to merge them (different customers, or whatever reason you have) via those 2 scripts the servers can be their backups for vital services (DNS and mail), problem with DNS going down and you don't have a backup, because of cashing, for some customers it can be down for 24 to 48 hours, even if your server was down for less then a minute, so you need a backup DNS.
For mail, it is possible that someone is sending a mail to on of your clients when the server is down, or unreachable, because of the backup the mail will be sent to the other server, kept there until the server is reachable again.
That's what those 2 script will accomplish, everything can be done by hand, but those 2 scripts will automate this.

If you look like centralized maintenance or user database, I think it's better to use the build in synchronization of ISPConfig, I think that would work way better for you.

 

May be but personally I am not really convinced with this method.

 

You mean with the scripts I wrote?
Why not? did I do something that's already in ISPConfig, or are you not convinced about backup of DNS and mail?

 

I do not agree completely, yes most mail servers will queue a mail if it can't be delivered, but there are mailserver that do not queue and let the sender know the mail can't be delivered.
Some servers doe queue, but give up after 5 minutes or an hour. (hotmail for example gives up pretty soon)
Most queue for a longer time, but just for the ones who do not queue (long enough) there is this backup for.
Also, you lay the service of your hosting in the hands of another, when your server is down, you hope the IT department will back your service up by queuing the mail, if they don't, who is to blame, the mailserver that does not queue (long enough) or your webserver that shouldn't be down...
Many hosting companies do have backup mail.
That's my opinion.
But therefore there are 2 separate scripts, they run independent, if you only want to use the automatic slave zone creation, that will work.

P.S. this script works for me, but there should be some tweaks for common use, for example, every slave record that's not on the master will be deleted, for me ok, but it's easy to adjust the script a bit so that it only deletes the records made by the script (actually the script's user) if you need the adjustments, I'm happy to do so.