"/etc/cron.d" or "/etc/cron.daily" or "crontab -e"

Discussion in 'Server Operation' started by danhansen@denmark, Dec 9, 2014.

  1. danhansen@denmark

    danhansen@denmark Member HowtoForge Supporter

    Hi,


    Ubuntu Server 14.04.1
    SAMBA filer server
    RSYNC

    I'm building a file server. No problems with the file server, but I'm trying to figure out which cron level I'm going to use. I wan't to make a few backup scripts run using cron. After reading several pages/sites I'm learned that "crontab -e" is a editor used on user level to add these rutines. I wan't to make these backup scripts run at root level (admin) Isn't that pretty normal when it's a backup of the whole data disk onto a backupdisk!? I want to backup the whole content of the data disk (RAID1 - md1), all users data and admin home directory as well.

    Here's one of the suggestions I found:
    The simplest method for adding a cron job is to edit, as root, /etc/crontab. In your case the entry would look something like this:
    # 0 9,15 * * * root sh /path/to/your/backup_script
    That runs the job at 0900 and 1500.
    Don't put scripts into the /etc/cron.* directories. To run a job daily, create the script somewhere else (I use /usr/local/sbin for all my root scripts), then put a symlink to the file in /etc/cron.daily.


    And this suggestion, which say a lot I think:
    cron also reads /etc/crontab, which is in a slightly different format (see crontab(5)). Additionally, cron reads the files in /etc/cron.d: it treats the files in /etc/cron.d as in the same way as the /etc/crontab file (they follow the special format of that file, i.e. they include the user field). However, they are independent of /etc/crontab: they do not, for example, inherit environment variable settings from it. The intended purpose of this feature is to allow packages that require finer control of their scheduling than the /etc/cron.{daily,weekly,monthly} directories to add a crontab file to /etc/cron.d.
    So, if you need environment variables to be loaded, use /etc/crontab, if you don't, then you can go with a file in /etc/cron.d. Also a reason for putting a file in /etc/cron.d can be, as pointed out in the man page, a finer scheduling control, which you also have in /etc/crontab (but that includes the environment variables).


    It's just that I can't find "/etc/crontab", may be an older version. I'm not sure which one to chose and exactly how to use it. Do I use "/etc/cron.d" og do I use one of the others e.g. daily.cron ?? And is it OK to edit the file directly using VI??

    Backups script:
    I've made a script using Rsync to make a backup from "/md1/home/ --> /sdc1/"

    Script name and location:
    "/home/admin/rsync_dailybackup.sh"

    Content:
    #!/bin/bash
    rsync -av /md1/home/ /sdc1/


    Cron rutine - Make the backup script run every morning at 06.00am:
    # 0 6 * * * root sh /home/admin/rsync_dailybackup.sh


    Question:
    1. What about the "--delete" flag?? rsync -av --delete /Directory1/ /Directory2/
    2. Do I need "root sh" in the Cron command line??


    Looking forward to hear from some of you CRON guru's ;)

    .
     
    Last edited: Dec 9, 2014
  2. florian030

    florian030 Well-Known Member HowtoForge Supporter

    You can add --delete to you script? Or did i missed something?

    root sh is not needed i crontab. Just make the script executable (chmod +x).
     
  3. srijan

    srijan New Member HowtoForge Supporter

    Hi Dan,

    Hope you are well.
    See the best way I will suggest it to make th cron with root user as
    and the content must be like this.
    Keep in mind that the rsync_dailybackup.sh must have execute permissions, if not then make it as
    After making the changes restart the cron service.
     
  4. danhansen@denmark

    danhansen@denmark Member HowtoForge Supporter

    Hi Florian,

    Thanks! No, you are quite right. The reason I ask is because I'm unfamiliar with rsync and trying to find the way it's usually done, you know "the common way"..
    And, is there a chance that the flag "--delete" will delete a file some day which was not intended!? It's a scary flag but actually I thought rsync would do this automatically! If a file is removed from the source, then you don't wan't it anymore, right?? But I read somewhere that it works both ways!?!? And because of this I'm not sure I know the consequences of "--delete". In case it works "both ways" and e.g. a file does not get fully "copied/rsync'ed" to the source drive and then nest time the script is executed rsync makes problems!?!? I haven't thought it through, hoping you guys know the facts of this tool ;)


    Hi Srijan,
    Thanks I'm well and you too I hope ;)

    So if even though I want to make a cron rutine at admin/root level it's OK to use the crontab -e (editor) !? I was about to modify the cron.d file manually!! I read somewhere that by using crontab -e you are setting the cron rutine as the current user. So if I've claimed su rights as I always do working the system, then it should be OK to use crontab -e ?!

    Regarding rsync I found a pretty nice ToDO here: https://help.ubuntu.com/community/rsync
    Regarding the script, when making backups to 2 disks, is this the way to do it or should I make a real script with loops/rutines which makes backup1 first and then backup2 last? How will rsync handle this little script:

    Code:
    #!/bin/bash
    rsync --delete -avv /home/ /backupdisk1/
    rsync --delete -avv /home/ /backupdisk2/
    echo "MyServer 06.00 - Daily Backup Successful: $(date)" >> /home/admin/logs/mybackup.log
    
    BTW I made a stupid mistake! I believed that I had to use the system path to the source, and the whole path. Like this:
    Code:
    I've made a script using Rsync to make a backup from "/md1/home/ --> /sdc1/"
    
    rsync -av /md1/home/ /sdc1/
    
    I quickly learned that I had to use the mounted paths. As you might remember we mounted the backupdisks as "/media/backupdisk1" & "/media/backupdisk2" back then. So the way I've made it in the script above using "/home/" and "/backupdisk1/" is OK!? It's not usually done in another way. Does it work at all when trying to use e.g. "/md1" or e.g. "sdd1/some/destination/" ??

    I learned that the slash after the source was pretty important too ;)


    .
     
    Last edited: Dec 9, 2014
  5. florian030

    florian030 Well-Known Member HowtoForge Supporter

    To sync everything from /md1/home to /sdc1 (incl. all directorys), use

    rsync OPTIONS /md1/home/ /sdc1

    BTW: i prefer --delete-after instead of --delete.

    And you should also have a look at the permissions like user and group.
     
  6. danhansen@denmark

    danhansen@denmark Member HowtoForge Supporter

    Hi Florian,


    What's the difference? Didn't know that flag!?! Sounds interesting ;)

    I'm not sure I explained it properly. What I meant was that it doesn't work when I'm using the "system path" including the drives. md1 is a RAID1 configuration which is mounted " / " (root) or what its called. And the 2 backup drives are mounted as "backupdisk1" and "backupdisk2". In fstab its setup to "/media/backupdisk1" and "/media/backupdisk2". The only way I could make it work was using the mount points, like this:

    Code:
    rsync -av /home/ /backupdisk1/

    Here's my "/etc/fstab"
    Code:
    UUID=23e37e48-fee8-3e6d-af42-53b23916e813 /               ext4    errors=remount-ro 0       1
    # swap was on /dev/md0 during installation
    UUID=f4ccdcfe-7760-4c12-be5a-c8c5aa7df3d7 none            swap    sw              0       0
    
    UUID=59918EAB7F03D5A5 /media/backupdisk1 ntfs defaults,uid=1000,rw  0       0
    UUID=c11b3a15-1f40-ff38-b9dc-61b4ac84d7fc /media/backupdisk2 ext4 rw,user,exec  0       0

    And another one showing the mountpoints:
    Code:
    # df -kh
    Filesystem      Size  Used Avail Use% Mounted on
    /dev/md1        1.8T  1.5G  1.7T   1% /
    none            4.0K     0  4.0K   0% /sys/fs/cgroup
    udev            1.8G  4.0K  1.8G   1% /dev
    tmpfs           365M  968K  364M   1% /run
    none            5.0M     0  5.0M   0% /run/lock
    none            1.8G     0  1.8G   0% /run/shm
    none            100M     0  100M   0% /run/user
    /dev/sdb1       932G   94M  932G   1% /media/backupdisk1
    /dev/sdd1       917G   72M  871G   1% /media/backupdisk2

    So when using the disk in the commands I get errors. Like in this example:
    Code:
    rsync -av /md1/home/ /sdc1/backups/
    This works just fine! Isn't it the way to do it??? Do you use the disk titles or what its called? "MD" being "MultiDisk" part of an array and "sdc1" when defining a command in/for rsync? E.g.:
    Code:
    rsync -av /md1/source/ /sdc1/destination/

    Sample from Ubuntu Docs:

    Local Backup
    Code:
    sudo rsync -azvv /home/path/folder1/ /home/path/folder2
    Backup Over Network
    Code:
    sudo rsync --dry-run --delete -azvv -e ssh /home/path/folder1/ [email protected]:/home/path/folder2

    And a question regarding the shell-script and making it executable! I've noticed that sometimes there's issues when using this command "chmod +x .....". Why I'm having issues sometimes I don't know. But there's 3 different ways to make a file executable according to my understanding. So why, what and how!? Is it just "the same difference" ;) Here's what I know:

    Your suggestion:
    Code:
    chmod +x /home/admin/rsync_dailybackup.sh 
    From Ubuntu Docs:
    Code:
    chmod 700 /home/admin/rsync_dailybackup.sh 
    https://help.ubuntu.com/community/Beginners/BashScripting

    And this one with another flag "a" ???:
    Code:
    chmod a+x /home/admin/rsync_dailybackup.sh 


    .
     
    Last edited: Dec 11, 2014
  7. florian030

    florian030 Well-Known Member HowtoForge Supporter

    From the manpage:

    --delete delete extraneous files from dest dirs
    --delete-after receiver deletes after transfer, not during

    You can sync between to directories / mountpoints. I.e. /home to /backup. I never tried it with and umounted partition or by using the volume-name.

     
  8. danhansen@denmark

    danhansen@denmark Member HowtoForge Supporter

    Hi Florian,

    1.
    Thanks...


    2.
    I was just because you showed a sample using volumename etc. I ´guess it was because I used it myself in a sample earlier on!? Anyway, lets not use anymore time on that one. I get it, I think ;)


    3.
    What's the advantage? That there will be no corrupt files in case of a bad connection?? Anyway, it sound better to me! If you can explain any better, and you've got the nerves, please do :D


    4.
    A quote from an earlier post from you:
    Not sure what to think about this one!? What is it you were thinking on? Please explain, because I read this:

    From man page
    Code:
    -a, --archive               archive mode; equals -rlptgoD (no -H,-A,-X)
    Code:
    And these flags does this:
            Descend recursively into all directories (-r),
            copy symlinks as symlinks (-l),
            preserve file permissions (-p),
            preserve modification times (-t),
            preserve groups (-g),
            preserve file ownership (-o), and
            preserve devices as devices (-D). 

    5.
    Question:
    If you are using the -z flag and compress the backups, is it then data which is compressed or is it only the files that is compressed during transfer!? I think it's the last thing but I'm not sure. I read about it, but as always a lot of samples and other explanations make it impossible to understand. I think you'll have to have quite a bit of knowhow before these man pages make sense.
    Here's a sample of what I'm talking about:
    Code:
    -r, --recursive
                  This  tells  rsync  to  copy  directories recursively.  See also
                  --dirs (-d).
    
                  Beginning with rsync 3.0.0, the recursive algorithm used is  now
                  an  incremental  scan that uses much less memory than before and
                  begins the transfer after the scanning of the first few directo‐
                  ries  have  been  completed.  This incremental scan only affects
                  our recursion algorithm, and does  not  change  a  non-recursive
                  transfer.  It is also only possible when both ends of the trans‐
                  fer are at least version 3.0.0.
    
                  Some options require rsync to know the full file list, so  these
                  options  disable the incremental recursion mode.  These include:
                  --delete-before,   --delete-after,    --prune-empty-dirs,    and
                  --delay-updates.   Because of this, the default delete mode when
                  you specify --delete is now --delete-during when  both  ends  of
                  the  connection are at least 3.0.0 (use --del or --delete-during
                  to request this improved deletion mode  explicitly).   See  also
                  the  --delete-delay  option  that  is a better choice than using
                  --delete-after.
    

    What about --delete-delay ??? Is there any reason this should be better? It depends on what is rsync'ed I guess

    Code:
           --delete
                  This tells rsync to delete extraneous files from  the  receiving
                  side  (ones  that  aren’t on the sending side), but only for the
                  directories that are being synchronized.  You  must  have  asked
                  rsync to send the whole directory (e.g. "dir" or "dir/") without
                  using a wildcard for the  directory’s  contents  (e.g.  "dir/*")
                  since  the wildcard is expanded by the shell and rsync thus gets
                  a request to transfer individual files, not  the  files’  parent
                  directory.   Files  that are excluded from the transfer are also
                  excluded from being deleted unless you use the --delete-excluded
                  option  or  mark  the rules as only matching on the sending side
                  (see the include/exclude modifiers in the FILTER RULES section).
    
                  Prior to rsync 2.6.7, this option would have  no  effect  unless
                  --recursive  was  enabled.  Beginning with 2.6.7, deletions will
                  also occur when --dirs (-d) is enabled, but only for directories
                  whose contents are being copied.
    
                  This  option can be dangerous if used incorrectly!  It is a very
                  good idea to first try a run using the --dry-run option (-n)  to
                  see what files are going to be deleted.
    
                  If the sending side detects any I/O errors, then the deletion of
                  any files at the destination  will  be  automatically  disabled.
                  This  is  to  prevent temporary filesystem failures (such as NFS
                  errors) on the sending side from causing a massive  deletion  of
                  files  on  the  destination.   You  can  override  this with the
                  --ignore-errors option.
    
                  The  --delete  option  may  be  combined   with   one   of   the
                  --delete-WHEN    options    without   conflict,   as   well   as
                  --delete-excluded.   However,  if  none  of  the   --delete-WHEN
                  options  are  specified,  rsync  will choose the --delete-during
                  algorithm  when  talking  to  rsync  3.0.0  or  newer,  and  the
                  --delete-before  algorithm  when talking to an older rsync.  See
                  also --delete-delay and --delete-after.
    
           --delete-before
                  Request that the file-deletions on the receiving  side  be  done
                  before the transfer starts.  See --delete (which is implied) for
                  more details on file-deletion.
    
                  Deleting before the transfer is helpful  if  the  filesystem  is
                  tight for space and removing extraneous files would help to make
                  the transfer possible.   However,  it  does  introduce  a  delay
                  before the start of the transfer, and this delay might cause the
                  transfer to timeout  (if  --timeout  was  specified).   It  also
                  forces rsync to use the old, non-incremental recursion algorithm
                  that requires rsync to scan all the files in the  transfer  into
                  memory at once (see --recursive).
    
           --delete-during, --del
                  Request  that  the  file-deletions on the receiving side be done
                  incrementally as the transfer happens.  The per-directory delete
                  scan is done right before each directory is checked for updates,
                  so it behaves like a more efficient  --delete-before,  including
                  doing  the  deletions  prior  to  any per-directory filter files
                  being updated.  This option was first  added  in  rsync  version
                  2.6.4.   See  --delete  (which  is  implied) for more details on
                  file-deletion.
    
           --delete-delay
                  Request that the file-deletions on the receiving  side  be  com‐
                  puted  during  the  transfer  (like  --delete-during),  and then
                  removed after the transfer completes.  This is useful when  com‐
                  bined with --delay-updates and/or --fuzzy, and is more efficient
                  than using --delete-after (but  can  behave  differently,  since
                  --delete-after  computes  the deletions in a separate pass after
                  all updates are done).  If the number of removed files overflows
                  an  internal  buffer,  a  temporary  file will be created on the
                  receiving side to hold the names (it is removed while  open,  so
                  you  shouldn’t  see it during the transfer).  If the creation of
                  the temporary file fails, rsync will try to fall back  to  using
                  --delete-after  (which  it  cannot do if --recursive is doing an
                  incremental scan).  See --delete (which  is  implied)  for  more
                  details on file-deletion.
    
           --delete-after
                  Request  that  the  file-deletions on the receiving side be done
                  after the transfer has completed.  This is  useful  if  you  are
                  sending  new per-directory merge files as a part of the transfer
                  and you want their exclusions to  take  effect  for  the  delete
                  phase  of the current transfer.  It also forces rsync to use the
                  old, non-incremental recursion algorithm that requires rsync  to
                  scan  all  the  files  in  the transfer into memory at once (see
                  --recursive).  See --delete (which is implied) for more  details
                  on file-deletion.
    

    6.
    Regarding the crontab -e command. I tried adding logged in as SU (root) as I always do making things on the system. When adding the line to the file and exiting it I got this message. Maybe it's just a comfimation on a CRON being activated!?!?:

    Code:
    # crontab -e
    no crontab for root - using an empty one
    crontab: installing new crontab

    7.
    Scripts need "#!" to begin with:

    Code:
    #!/bin/sh
    or
    Code:
    #!/bin/bash


    Thanks in advance ;)

    .
     
    Last edited: Dec 15, 2014

Share This Page