Best practices to "secure" external backups

Discussion in 'General' started by ChuckSC, Jan 2, 2016.

  1. ChuckSC

    ChuckSC New Member

    Hi everyone,

    I'm running a server with ISPConfig and the websites' files and databases are locally backed up under /var/backup. In case of a major issue with my main hosting, I need to recursively grab what's there to store it on a backup server elsewhere. I found the easiest way to do just that was to run a cron from the backup server:
    Code:
    30 4 * * 2 scp -i ~/.ssh/id_rsa -r [email protected]:/var/backup /home/week`date +\%V`
    So, while this is working just fine and it's easy to operate, there is an inherent safety issue here. If somehow my backup server is hacked, that key being used to run the SCP command gives root access to the main server. That's not the best option and defeats the purpose of an external backup.

    What would you say are my options here, if any? How can I secure the access to the main server from the external backup server? My understanding is that files in /var/backup are accessible from root only on the main server. Your feedback on this is more than welcome, thank you.
     
  2. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    Why did you store the private-key on the backup-server?
    You can also create a script /usr/local/ispconfig/server/scripts/backup_dir_mount.sh (root.root and 0700) and use this script to mount the backupspache to /var/backup. something like
    #!/bin/sh
    sshfs backup@backup-server:/backups /var/backup
     
  3. ChuckSC

    ChuckSC New Member

    Thank you for your feedback, florian030.

    It appeared to be the easiest solution, while not the safest obviously. This is one of techniques listed in my hosting provider's doc, but they also recommend to add command="/root/bin/backup.sh" before the key to limit the access via a script (I was not able to figure out that part unfortunately.)

    OK, that's also an option I looked at. However, I need to enter the password of the backup server on the main server in that case (or use -oIdentityFile=...), is that correct?
    So, I don't see if that's necessarily better. That would mean in case of a hack on my main server, that person could then log into the backup server as well. What am I missing here?

    (On a side note, in that thread it is mentioned that you should umount the directory after the backup job, why is that?)

    Thanks for bearing with me, I'm trying to grasp all the aspects of this.
     
  4. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    Usually you add something like command= to the public-key on the remote server and not to your private key. You can´t use the public-key on the backup-server to ssh into your server.
    I think, you mixed the ssh-keys for logins without a password.
    If you access the remote-server over sshfs, i would setup a ssh-login without password so you can mount your backup-space into your server.
     
  5. ChuckSC

    ChuckSC New Member

    Correct, this is what I attempted to do at first. Public key of the backup server onto the main server. The ran SCP from the backup server using its own private key.
    I was just unable to figure out the content of the .sh script listed in command= on the public key on the main server to limit access to certain commands like SCP on specific directories.
    Alright, in that case I will have to: 1. add the public key of my main server in the authorized_keys of the backup server 2. use the private key of my main server on the sshfs command to mount remotely.
    Is that correct? If so, this would potentially give access via other means to the backup server from the main server using the same set of key?

    Thank again for your assistance and feedback.
     
  6. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    command=foo in authorized_keys(2) limits the ssh-connection to the command foo.

    if you add the public-key to authorized_keys, your private-key will be used for the login (try ssh remoteruser@remoteserver). if ssh works, you can also use "ssfhs remoteruser@remoteserver:/ /mnt" to mount the remote to /mnt.

    if someone gets your private ssh-key, he can login to the external server (but you have the same problem with a password).
     
  7. ChuckSC

    ChuckSC New Member

    Thank you for the clarification, florian030. Basically, using ssfhs is not safer than the method I used with scp in the event of a compromised private key.

    In that case, using "command=" in front of the public key of the backup server (in authorized_keys on the main server) is the only solution to secure the process a bit? That way if the private key of the backup server is compromised, it would limit its use on the main server to cause harm, is that correct?
     
  8. florian030

    florian030 ISPConfig Developer ISPConfig Developer

    command=foo set the command, that the user can run on the remote-server.
     
  9. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    In your setup, compromised private key on your backup server results in root access to your main server. In Florian's setup, compromised private key on the main server results in non-root access to your backups server. And in either case, you probably have one root compromise on your hands to obtain the private key in the first place.

    Another thing to keep in mind is security of the backup data itself. You should probably use disk-level encryption if there's a chance the disks will ever come in anyone else's hands to protect your and your customers' data. (Think: failed disk you have to send back to the vendor; physical theft/misplacement/shipping mishaps; and later resale/repurposing of equipment.)

    If your backup server performs backups for more than one machine, you might want to make sure that the backups/data from one is secure from another. Eg. if server1 is hacked, they can ssh as [email protected] and access files there, so make sure server2 uses [email protected] account, and that backup1 and backup2 cannot read each others files. If you want to really be safe, use file-level encryption in addition to filesystem permissions (which would even protect for data loss against a compromised root account on your backup server).
     
  10. Jesse Norell

    Jesse Norell ISPConfig Developer Staff Member ISPConfig Developer

    A different approach which I'm playing with here is using a cheap NAS for the backup server. Create a separate iSCSI LUN for each server, so each server mounts their own iSCSI target. On the server to be backed up, setup LUKS encryption on the device (iSCSI LUN), then make a filesystem and use your backup system of choice (tar, dirvish, etc.). You can save the LUKS encryption key as a file on the server (and back it up somewhere, of course!), as it only protects the data which is already available right on that server. iSCSI can use mutual username/passwords, and even limit what iSCSI initiator can access a LUN, so one server can't access another's iSCSI target.
     
    Last edited: Jan 6, 2016
  11. vk3heg

    vk3heg Member

    What I'm doing here is reverse to how the op is doing it. My server is the one that starts the backup, does the scp to a remote backup server as a normal user. (Have exchanged key's so no need for password when the backup script runs).

    I have direct access to the backup server behind me on the shelf. It's the production server that is remote.
     

Share This Page