There are several "good" standard ways to perform backups for your system now. The "best" seems to be rdiff-backup - although I have been successfully using rsync for a long time and so I haven't felt the need to change. There is also the incremental tar backup method. I won't describe that here, either, but I have included a script (at the end) that I use for such purposes.
Rsync
If you want to learn the basics of "rsync" then you can first read
this website
which will open your eyes. Outlined there is a "rotating" backup method
cleverly exploiting "hard links" in the Linux filesystem.
In Debian you can install rsync by installing the package "rsync"
on the client machine. It is useful to also install this package
on the server machine and then set the following lines in
/etc/default/rsync
RSYNC_ENABLE=true RSYNC_OPTS='--address=127.0.0.1 --port=873'This starts up the rsync server daemon listening on loopback to port 873. Although the rsync daemon is not necessary (we will use rsync over SSH instead) I have found that I have problems backing up largish (~250GB) systems without it (I have no idea why this daemon helps when using the SSH method since the two methods seem to have nothing to do with each other, but it does).
If you want to use the rsync daemon for real (instead of rsync over SSH) then you will need to leave off the "--address=127.0.0.1" to have the daemon listen on all network interfaces (or, alternatively keep this entry and use an SSH Tunnel).
Although the scripts below are written for rsync over SSH, it is trivial to modify them to use the rsync daemon instead. This can be extremely handy since then you will not be required to SSH into the remote machine (good for FTP sites, also good for security reasons). Check the manpages for instructions concerning the rsync daemon.
Tip: if you are backing up to/from FAT32 partitions then you SHOULD mount these partitions using the "iocharset=utf8" option. Otherwise any international characters will be converted into "?" - and rsync does not handle question marks very well!!!
To get the flavor for rsync, first here is a simple backup script
that simply SYNCS the source to the destination - this is nothing fancy
(if either SRC or DEST is remote (using SSH) then this WILL NOT WORK in a cron job.
You need to "source" the keychain variables first! Check the next script):
#!/bin/sh
# this script automatically syncs SRC to DEST
# don't forget the trailing "/" on SRC
# this can be a valid ssh path like
# yourname@ma.utexas.edu:work/ OR something like
# yourname@ma.utexas.edu:/home/yourname/work/
SRC="/home/stirling/movies/"
# don't forget the trailing "/" on DEST
DEST="/disks/sdc1/movies/"
# list your exclusions here - REQUIRED even if empty
EXCLUDEFILE="excludethisjunk.txt"
# check to see if the source even exists (don't worry if its remote)
echo $SRC | grep ":"
if [ ! $? -eq 0 ]
then
if [ ! -d $SRC ]
then
echo "error: directory $SRC does not exist"
exit 1
fi
fi
# the source exists at this point (or it's remote)
# check to see if the destination even exists (don't worry if its remote)
echo $DEST | grep ":"
if [ ! $? -eq 0 ]
then
if [ ! -d $DEST ]
then
echo "error: directory $DEST does not exist"
exit 1
fi
fi
rsync -vapu --delete -e ssh --exclude-from=$EXCLUDEFILE $SRC $DEST
if [ ! $? -eq 0 ]
then
echo "Failed!!!"
exit 1
fi
echo "Success!!!"
exit 0
I have written a couple of (what I think are) improvements to the techniques outlined in the "rotating backup" article above. Most notably I don't like having my weekly/monthly/halfyearly backup schedule being determined by actual dates/times - this is because I deal with several machines that sometimes go for weeks without being "on", and I don't want them to miss their monthly or halfyearly backups because they are not powered. In this script I can make it a CRON job and run it as often as I like - each run corresponds to "1 day".
Here is the script file that I use. This script file can take a "source" directory and back it up to a "destination" directory, although the "destination" directory MUST be locally mounted (because I'm rotating backups - obviously you can get around this by mounting with NFS, SMB, or my favorite: SHFS/LUFS networking - see my "no promises" tar script included at the VERY end if you want an idea how to write an rsync script that mounts first). The backups are kept on a "today, yesterday, week, month, half-year" schedule (again these increments of "time" are really determined by how often you run the script - each run corresponds to "1 day").
Also included is an option for an "exclusions file" so that certain things can be left out of the backup (like web caches or other worthless things). YOU MUST HAVE AN EXCLUSIONS FILE, EVEN IF IT IS EMPTY. Furthermore there is the ability to use the "keychain" utility for "passwordless SSH"ing (handy if the SRC directory is on a different machine and you want to run the backup in the middle of the night). You can read my blurb about keychain for guidance about that.
The format of the EXCLUDEFILE is as follows: assuming that
SRC="/home/johnsmith/work/" and
EXCLUDEFILE="/home/johnsmith/excludethisjunk.txt"
then in "excludethisjunk.txt" you can put a list of files/directories that you don't want
(pathnames are RELATIVE TO SRC!!!). For example, if I put in the following lines
junkdirectory1
junkfile
Then the directory /home/johnsmith/work/junkdirectory1 and the
file /home/johnsmith/work/junkfile will be omitted from the backup. Notice
that I DID NOT use the full pathname - this WILL NOT WORK. All files/directories
put in your EXCLUDEFILE are relative to your SRC directory (this is an rsync
thing, not a shortcoming in my script).
For completeness I have included below this script ANOTHER script that uses incremental tar backups (but otherwise is very similar). I prefer the "rsync" way, but "tar" backups can be useful for some things.
#!/bin/sh
# this program creates a "daily" rsync backup of the data specified
# in SRC (note the trailing "/", which is necessary for rsync)
# it puts the result in DEST
# IT IS ABSOLUTELY NECESSARY THAT "DEST" BE A LOCALLY MOUNTED DIRECTORY
# NOTE: "daily" doesn't really mean anything - each "day" corresponds
# to a single run (this script is intended to be run every night, but it
# doesn't have to)
# Don't forget the trailing /
# Note that SRC can be a remote system like
# johnsmith@machine.ma.utexas.edu:backup_this_directory/
SRC="/home/johnsmith/work/"
# DEST MUST be a LOCAL directory... don't forget the trailing /
DEST="/disks/data/backup/work/"
# This is the file which contains the exclusions (files/directories to leave out)
# MUST BE CREATED (even if empty)
EXCLUDEFILE="/home/johnsmith/excludethisjunk.txt"
# we need to "source" the keychain information
# so that we don't need to put in a password in
# order to make use of ssh tools (like scp)
# IF YOU ARE NOT USING SSH THEN YOU CAN IGNORE THIS
# OTHERWISE LOOK UP HOW TO USE KEYCHAIN
#KEYCHAINVARS="${HOME}/.keychain/yourkeychainfile-sh"
#. $KEYCHAINVARS
# These are the number of days since the last backup to wait
# in order to make a new backup
DAYSINWEEK=7
DAYSINMONTH=30
DAYSINHALFYEAR=180
# These are the files which store the days since last backup
DAYSSINCELASTWEEKFILE=${DEST}days_since_last_week_backup
DAYSSINCELASTMONTHFILE=${DEST}days_since_last_month_backup
DAYSSINCELASTHALFYEARFILE=${DEST}days_since_last_halfyear_backup
DEST_TEMP=${DEST}temp
DEST_TODAY=${DEST}today
DEST_YESTERDAY=${DEST}yesterday
WEEK_CANDIDATE=${DEST}week.candidate
DEST_WEEK=${DEST}week
MONTH_CANDIDATE=${DEST}month.candidate
DEST_MONTH=${DEST}month
HALFYEAR_CANDIDATE=${DEST}halfyear.candidate
DEST_HALFYEAR=${DEST}halfyear
# First check to make sure that DEST is not a remote directory
# this is not allowed
echo $DEST | grep ":"
if [ $? -eq 0 ]
then
echo "No remote destinations allowed!!!"
exit 1
fi
# check to see if the source even exists (don't worry if its remote)
echo $SRC | grep ":"
if [ ! $? -eq 0 ]
then
if [ ! -d $SRC ]
then
echo "error: directory $SRC does not exist"
exit 1
fi
fi
# the source exists at this point (or it's remote)
# check to see if the destination exists... if not, then create it
if [ ! -d $DEST ]
then
mkdir $DEST
fi
# still check to see if the destination directory exists, and if so, make
#it writeable
if [ -d $DEST ]
then
chmod u+w $DEST
cd $DEST
chmod u+w *
else
echo "Cannot create destination directory"
exit 1
fi
# We are going to rotate the halfyear backups first, if necessary
# first see if it is necessary
if [ -f $DAYSSINCELASTHALFYEARFILE ]
then
DAYSSINCELASTHALFYEAR=`cat $DAYSSINCELASTHALFYEARFILE`
else
DAYSSINCELASTHALFYEAR=$DAYSINHALFYEAR
fi
DAYSSINCELASTHALFYEAR=`expr $DAYSSINCELASTHALFYEAR + 1`
if [ $DAYSSINCELASTHALFYEAR -gt $DAYSINHALFYEAR ]
then
# we are going to rotate... reset the count variable
DAYSSINCELASTHALFYEAR=0
# we need to make sure that the candidate even exists
if [ -d $HALFYEAR_CANDIDATE ]
then
# rotate the files
# start by deleting the halfyear backup
if [ -d $DEST_HALFYEAR ]
then
rm -r $DEST_HALFYEAR
fi
# then rotate the candidate to "halfyear"
mv $HALFYEAR_CANDIDATE $DEST_HALFYEAR
# put a timestamp into the log file
echo -n "Halfyear backup: " >> "${DEST}log"
date >> "${DEST}log"
else
echo "No halfyear candidate exists: nothing to do"
fi
fi
# write the day count out
echo $DAYSSINCELASTHALFYEAR > $DAYSSINCELASTHALFYEARFILE
# We are going to rotate the monthly backups next, if necessary
# first see if it is necessary
if [ -f $DAYSSINCELASTMONTHFILE ]
then
DAYSSINCELASTMONTH=`cat $DAYSSINCELASTMONTHFILE`
else
DAYSSINCELASTMONTH=$DAYSINMONTH
fi
DAYSSINCELASTMONTH=`expr $DAYSSINCELASTMONTH + 1`
if [ $DAYSSINCELASTMONTH -gt $DAYSINMONTH ]
then
# we are going to rotate... reset the count variable
DAYSSINCELASTMONTH=0
# we need to make sure that the candidate even exists
if [ -d $MONTH_CANDIDATE ]
then
# rotate the files
# start by rotating the "month" to "halfyear.candidate"
if [ -d $DEST_MONTH ]
then
if [ ! -d $HALFYEAR_CANDIDATE ]
then
mv $DEST_MONTH $HALFYEAR_CANDIDATE
else
rm -r $DEST_MONTH
fi
fi
# then rotate the candidate to "month"
mv $MONTH_CANDIDATE $DEST_MONTH
# put a timestamp into the log file
echo -n "Monthly backup: " >> "${DEST}log"
date >> "${DEST}log"
else
echo "No monthly candidate exists: nothing to do"
fi
fi
# write the day count out
echo $DAYSSINCELASTMONTH > $DAYSSINCELASTMONTHFILE
# We are going to rotate the weekly backups next, if necessary
# first see if it is necessary
if [ -f $DAYSSINCELASTWEEKFILE ]
then
DAYSSINCELASTWEEK=`cat $DAYSSINCELASTWEEKFILE`
else
DAYSSINCELASTWEEK=$DAYSINWEEK
fi
DAYSSINCELASTWEEK=`expr $DAYSSINCELASTWEEK + 1`
if [ $DAYSSINCELASTWEEK -gt $DAYSINWEEK ]
then
# we are going to rotate... reset the count variable
DAYSSINCELASTWEEK=0
# we need to make sure that the candidate even exists
if [ -d $WEEK_CANDIDATE ]
then
# rotate the files
# start by rotating the "month" to "month.candidate"
if [ -d $DEST_WEEK ]
then
if [ ! -d $MONTH_CANDIDATE ]
then
mv $DEST_WEEK $MONTH_CANDIDATE
else
rm -r $DEST_WEEK
fi
fi
# then rotate the candidate to "week"
mv $WEEK_CANDIDATE $DEST_WEEK
# put a timestamp into the log file
echo -n "Weekly backup: " >> "${DEST}log"
date >> "${DEST}log"
else
echo "No weekly candidate exists: nothing to do"
fi
fi
# write the day count out
echo $DAYSSINCELASTWEEK > $DAYSSINCELASTWEEKFILE
# NOW FOR THE DAILY BACKUPS
# start by copying "today" backup to "temp" (just copy links, remember)
# if it exists
# first delete "temp" from any unfinished backups
if [ -d $DEST_TEMP ]
then
rm -r $DEST_TEMP
fi
if [ -d $DEST_TODAY ]
then
cp -lax $DEST_TODAY $DEST_TEMP
fi
# NOW perform the rsync
rsync -vapu --delete -e ssh --exclude-from=$EXCLUDEFILE $SRC $DEST_TEMP
if [ ! $? -eq 0 ]
then
echo "backup failed"
exit 1
fi
# rotate the files
# start by rotating the "yesterday" to the "week candidate"
if [ -d $DEST_YESTERDAY ]
then
if [ ! -d $WEEK_CANDIDATE ]
then
mv $DEST_YESTERDAY $WEEK_CANDIDATE
else
rm -r $DEST_YESTERDAY
fi
fi
# now rotate "today" to "yesterday"
if [ -d $DEST_TODAY ]
then
mv $DEST_TODAY $DEST_YESTERDAY
fi
# now finally rotate the backup just done "temp" to "today"
if [ -d $DEST_TEMP ]
then
mv $DEST_TEMP $DEST_TODAY
touch $DEST_TODAY
fi
# put a timestamp on the backup
date >> "${DEST}log"
# change the backup directory back to nonwriteable
cd $DEST
chmod a-w *
chmod a-w $DEST
echo "Success!!!"
exit 0
Incremental TAR backups
The following script gives a similar "tar" incremental backup from a "source" directory
to a "destination" directory (both MUST be locally mounted - although you can get
around this by mounting using NFS, SMB, or SHFS/LUFS networking). I should probably
put an "EXCLUDEFILE" option in here (as above using rsync), but I've been too lazy.
#!/bin/sh
# this program creates a "daily" tar incremental backup of the data specified
# in SRC it puts the result in DEST
# NOTE: Here "daily" doesn't mean anything - it just corresponds to
# each time the program is run (this batch script is written to be run
# each night, but that's not necessary)
# everything must be locally mounted
SRC="/home/stirling/work/"
# don't forget the trailing "/" in DEST
DEST="/disks/sdd1/backup/work/"
# These are the number of days since the last full backup to wait
# in order to make a new full backup
DAYSINWEEK=7
# These are the files which store the days since last backup
DAYSSINCELASTWEEKFILE=${DEST}days_since_last_week_backup
DEST_TEMP=${DEST}temp.tgz
DEST_DAY_PART=${DEST}day
DEST_WEEK=${DEST}week.tgz
DEST_FILELIST=${DEST}filelist
DEST_FILELIST_TEMP=${DEST}filelist.temp
# check to see if the source even exists
if [ -d $SRC ]
then
# check to see if the destination exists... if not, then create it
if [ ! -d $DEST ]
then
mkdir $DEST
fi
# still check to see if the destination directory exists, and if so, make
#it writeable
if [ -d $DEST ]
then
chmod u+w $DEST
cd $DEST
chmod u+w *
else
echo "Cannot create destination directory"
exit 1
fi
# We are going to do the weekly full backup, if necessary
# first see if it is necessary
if [ -f $DAYSSINCELASTWEEKFILE ]
then
DAYSSINCELASTWEEK=`cat $DAYSSINCELASTWEEKFILE`
else
DAYSSINCELASTWEEK=$DAYSINWEEK
fi
DAYSSINCELASTWEEK=`expr $DAYSSINCELASTWEEK + 1`
if [ $DAYSSINCELASTWEEK -gt $DAYSINWEEK ]
then
# we are going to rotate backups... reset the count variable
DAYSSINCELASTWEEK=0
# we need to make sure to delete any partial old backups
if [ -f $DEST_TEMP ]
then
rm $DEST_TEMP
fi
if [ -f $DEST_FILELIST_TEMP ]
then
rm $DEST_FILELIST_TEMP
fi
# now make the temporary full backup (complete with a filelist)
tar zcvpf $DEST_TEMP -g $DEST_FILELIST_TEMP $SRC >> /dev/null
if [ ! $? -eq 0 ]
then
echo "backup failed"
exit 1
fi
# we have a successful full backup - we need to remove the old backups
if [ -f $DEST_WEEK ]
then
rm $DEST_WEEK
fi
if [ -f $DEST_FILELIST ]
then
rm $DEST_FILELIST
fi
mv $DEST_TEMP $DEST_WEEK
mv $DEST_FILELIST_TEMP $DEST_FILELIST
if [ -f "${DEST_DAY_PART}1.tgz" ]
then
rm ${DEST_DAY_PART}*
fi
# put a timestamp into the log file
echo -n "Weekly backup: " >> ${DEST}log
date >> ${DEST}log
fi
# write the day count out
echo $DAYSSINCELASTWEEK > $DAYSSINCELASTWEEKFILE
# We are going to do the daily backups, if necessary
# first see if it is necessary
if [ $DAYSSINCELASTWEEK -gt 0 ]
then
DEST_DAY="${DEST_DAY_PART}${DAYSSINCELASTWEEK}.tgz"
# we need to make sure to delete any partial old backups
if [ -f $DEST_TEMP ]
then
rm $DEST_TEMP
fi
if [ -f $DEST_FILELIST_TEMP ]
then
rm $DEST_FILELIST_TEMP
fi
# now make the daily incremental backup (using the filelist)
cp $DEST_FILELIST $DEST_FILELIST_TEMP
tar zcvpf $DEST_TEMP -g $DEST_FILELIST_TEMP $SRC >> /dev/null
if [ ! $? -eq 0 ]
then
echo "backup failed"
exit 1
fi
if [ -f $DEST_FILELIST ]
then
rm $DEST_FILELIST
fi
mv $DEST_TEMP $DEST_DAY
mv $DEST_FILELIST_TEMP $DEST_FILELIST
touch $DEST_DAY
touch $DEST_FILELIST
# put a timestamp on the backup
date >> ${DEST}log
fi
# change the backup directory back to nonwriteable
cd $DEST
chmod a-w *
chmod a-w $DEST
else
echo "error: directory $SRC does not exist"
exit 1
fi
echo "Success!!!"
exit 0
"No Promises" tar script which first mounts using SHFS
Here is a VERY ROUGH script that mounts using SHFS before performing a tar
backup (as we've seen, some things like destination directories need to be
locally mounted for any of the above to work). You should have
SHFS/LUFS networking before trying to use this - you WILL NEED TO MODIFY IT!!!
It's JUST AN EXAMPLE:
#!/bin/sh
# this program backs up the "source" directory (REMOTE_DIR) on a remote machine to
# a file on the local machine using incremental TAR backups
# first the remote directory is mounted using the SHFS system tools
# don't forget the trailing "/" in DEST
DEST="/disks/sdc1/backup/laptop_junk/"
# REMOTE_DIR IS THE SOURCE DIRECTORY - put in any valid SHFS pathname
REMOTE_DIR="stirling@laptop"
# this is where REMOTE_DIR will be mounted
# DO NOT MODIFY THIS - MODIFY ONLY REMOTE_DIR as the "source"
SRC="${DEST}mnt/"
# check to see if the destination exists... if not, then create it
if [ ! -d $DEST ]
then
mkdir $DEST
fi
# make sure that the local mount directory exists
if [ -d $DEST ]
then
if [ ! -d $SRC ]
then
mkdir $SRC
fi
if [ ! -d $SRC ]
then
echo "cannot create temporary mount"
exit 1
fi
else
echo "cannot create destination"
exit 1
fi
# mount the laptop directory
shfsmount -o ro $REMOTE_DIR $SRC
if [ ! $? -eq 0 ]
then
echo "Cannot mount laptop"
exit 1
fi
# NOW JUST USE THE OLD TAR INCREMENTAL PROGRAM (EVERYTHING IS LOCALLY MOUNTED)
# this program creates a "daily" tar incremental backup of the data specified
# in SRC it puts the result in DEST
# NOTE: Here "daily" doesn't mean anything - it just corresponds to
# each time the program is run (this batch script is written to be run
# each night, but that's not necessary)
# These are the number of days since the last full backup to wait
# in order to make a new full backup
DAYSINWEEK=7
# These are the files which store the days since last backup
DAYSSINCELASTWEEKFILE=${DEST}days_since_last_week_backup
DEST_TEMP=${DEST}temp.tgz
DEST_DAY_PART=${DEST}day
DEST_WEEK=${DEST}week.tgz
DEST_FILELIST=${DEST}filelist
DEST_FILELIST_TEMP=${DEST}filelist.temp
# check to see if the source even exists
if [ -d $SRC ]
then
# check to see if the destination exists... if not, then create it
if [ ! -d $DEST ]
then
mkdir $DEST
fi
# still check to see if the destination directory exists, and if so, make
#it writeable
if [ -d $DEST ]
then
chmod u+w $DEST
cd $DEST
chmod u+w *
else
echo "Cannot create destination directory"
exit 1
fi
# We are going to do the weekly full backup, if necessary
# first see if it is necessary
if [ -f $DAYSSINCELASTWEEKFILE ]
then
DAYSSINCELASTWEEK=`cat $DAYSSINCELASTWEEKFILE`
else
DAYSSINCELASTWEEK=$DAYSINWEEK
fi
DAYSSINCELASTWEEK=`expr $DAYSSINCELASTWEEK + 1`
if [ $DAYSSINCELASTWEEK -gt $DAYSINWEEK ]
then
# we are going to rotate backups... reset the count variable
DAYSSINCELASTWEEK=0
# we need to make sure to delete any partial old backups
if [ -f $DEST_TEMP ]
then
rm $DEST_TEMP
fi
if [ -f $DEST_FILELIST_TEMP ]
then
rm $DEST_FILELIST_TEMP
fi
# now make the temporary full backup (complete with a filelist)
tar zcvpf $DEST_TEMP -g $DEST_FILELIST_TEMP $SRC >> /dev/null
if [ ! $? -eq 0 ]
then
echo "backup failed"
exit 1
fi
# we have a successful full backup - we need to remove the old backups
if [ -f $DEST_WEEK ]
then
rm $DEST_WEEK
fi
if [ -f $DEST_FILELIST ]
then
rm $DEST_FILELIST
fi
mv $DEST_TEMP $DEST_WEEK
mv $DEST_FILELIST_TEMP $DEST_FILELIST
if [ -f "${DEST_DAY_PART}1.tgz" ]
then
rm ${DEST_DAY_PART}*
fi
# put a timestamp into the log file
echo -n "Weekly backup: " >> ${DEST}log
date >> ${DEST}log
fi
# write the day count out
echo $DAYSSINCELASTWEEK > $DAYSSINCELASTWEEKFILE
# We are going to do the daily backups, if necessary
# first see if it is necessary
if [ $DAYSSINCELASTWEEK -gt 0 ]
then
DEST_DAY="${DEST_DAY_PART}${DAYSSINCELASTWEEK}.tgz"
# we need to make sure to delete any partial old backups
if [ -f $DEST_TEMP ]
then
rm $DEST_TEMP
fi
if [ -f $DEST_FILELIST_TEMP ]
then
rm $DEST_FILELIST_TEMP
fi
# now make the daily incremental backup (using the filelist)
cp $DEST_FILELIST $DEST_FILELIST_TEMP
tar zcvpf $DEST_TEMP -g $DEST_FILELIST_TEMP $SRC >> /dev/null
if [ ! $? -eq 0 ]
then
echo "backup failed"
exit 1
fi
if [ -f $DEST_FILELIST ]
then
rm $DEST_FILELIST
fi
mv $DEST_TEMP $DEST_DAY
mv $DEST_FILELIST_TEMP $DEST_FILELIST
touch $DEST_DAY
touch $DEST_FILELIST
# put a timestamp on the backup
date >> ${DEST}log
fi
# change the backup directory back to nonwriteable
cd $DEST
chmod a-w *
chmod a-w $DEST
else
echo "error: directory $SRC does not exist"
exit 1
fi
echo "Success!!!"
exit 0
This page has been visited 1,249 (2 today) times since April 8, 2005