Backups

I've got and tried several backup solutions, often things labelled as backup are actually file sync which is not what I want and all of them have weaknesses (and strengths).

• TimeMachine: Free but Mac only and when it works it's great but if it decides a backup is corrupt there is virtually no way to recover and the only realistic option is to delete the existing backup and start again.
• Rsync: Free and not strictly backup but can be used for non-incremental backups, I've used it on my Raspberry Pi projects where the data volume is low and regular full backups are not too painful, all that is needed is to added the current date to the tar archive filename.
• Rsnapshot: Free and great for local incremental backups e.g. to a USB disk or locally mounted drive, it does remote backups but only pulling from a server rather than pushing to it, does not have encryption and not easy to see when the backups were made.
• Duplicati 2.0: Free, this is fairly reliable despite being listed as in beta and not being updated recently but it also suffers occasionally from corrupt backups with the difference being that a mechanism to rebuild the database is part of the software and it has worked every time I've needed it. The error messaging is very opaque and the user interface is not to my liking. It is however very flexible offering encryption and scheduling. Due to changes in MacOs it now lacks a tray icon so it's less easy to know if it's running and I found that it would stop for reasons unknown.
• iDrive: Paid, I've had this on my Macbook for nearly a year and honestly it just works and has not gone wrong yet, it has user encryption and scheduling.
• pCloud: Paid and although it claims to be backup it is really only sync, I only use this as network storage to share and transfer files across devices. It mounts as a local drive on Macbook, Windows and Linux.
• BackBlaze: Backblaze has two paid offerings:
  • The first one is a local client for Macbook and PC with a nice UI as I recall but despite being offered as a backup solution is actually a sync solution and does not offer no-knowledge encryption.
  • The second one is plain storage which can be accessed with, for example, Duplicati, does require logging into the Backblaze site, creating buckets and generating pairs of keys which need to be transferred to which ever program is being used to access them.
• UrBackup: Free, this has a client/server approach unlike the others so I ran the server on an Asustor 1102T and had 2 Raspberry Pi's one Macbook, and one Linux desktop backup to it with no difficulty.

Restic backups to multiple locations

I continue to look for a reliable, cross platform, incremental backup solution with encryption and happened upon Restic which has existed since August 2015 but is on version 0.15.2 as of May 2023. It's less easy to set up than the others I've mentioned but the documentation is excellent and once you get it going it's extremely fast. What started out as an attempt to script a backup to a local hard disk got out of control and ended up as the script below which I will explain.

Before Restic can make any backups the destinations have to be initialised which is done as follows in increasing level of complexity:

Initialising a restic repository on a local disk

# restic init --repo /local/backup 

A password will be requested to encrypt the backup and should be as decent as you think it needs to be and remembered. Create a file called ~/.restic_pw containing only this password. We'll need it later.

Initialising a restic repository on a NAS over sftp

# restic --repo sftp:user@host:/remote/backup init

Again a password will be needed, use the same one as above. user is a user on the remote server, host is the IP address or hostname of the remote server and /remote/backup is the path on the server.

To allow automation of the backup later a passwordless ssh key needs to be created locally and copied to the remote location. This page is an excellent source of all information pertaining to ssh but all that is needed for now is to generate a key using the following command; hit enter when asked for the location to store the key in the default location and twice more when asked for a password to leave it blank.

ssh-keygen -t ed25519

Then run

# ssh-copy-id -i ~/.ssh/id_ed25519.pub user@host

Where user and host match the ones used to initialise the restic repository. A password will be requested one time but the next execution of ssh user@host should land you in the remote host without a password being asked for.

Initialising a restic repository on Backblaze B2

Setting up Backblaze bucket is better described in one or both of these references ^{1 2} and then a file needs to be created with some environment variables that restic will need. It doesn't matter where this file is or what it is named, I created it as ~/.restic_env.

export B2_ACCOUNT_ID="...." <-- this one seems to be only digits
export B2_ACCOUNT_KEY="....." <-- this one looks like a complex password
export RESTIC_REPOSITORY="b2:bucket-name"<-- needs to match the name used during bucket setup
export RESTIC_PASSWORD_FILE=~/.restic_pw

Initialising the restic repository is done in a similar way as before, if all the environment variables are set correctly then everything is ready for backups to happen.

# restic --repo b2:bucket-name init

The restic backup script

Having prepared the destinations all that's left is to make use of them which is done with the script below. I didn't necessarily want to back up to all destinations every time the script was run so learning from this page I added some command line options to allow me to specify which location to use or to do all of them. Usage is described in the script but in short to back up to the NAS:

# resticlocal --nas

#!/bin/bash

# Path to the executable
RESTIC=/home/restic/bin/restic

# All known backup destinations
# These must exist and be intialised

# Asustor NAS box
NAS="sftp:stuart@192.168.178.41:/volume1/Restic/snowdog"
# 1TB local HDD
LOCAL=/backup/restic/snowdog
# Backblaze bucket name
REMOTE=b2:bucket-name

usage()
{
    echo "usage: resticlocal --nas | --local | --b2 | -all | --snapshots"
}

case $1 in
        -n | --nas )        DEST=$NAS
                                ;;
        -l | --local )      DEST=$LOCAL
                                ;;
        -b | --b2 )     DEST=$REMOTE
                                ;;
        -a | --all )        DEST="ALL"
                                ;;
        -s | --snapshots )  DEST="SNAP"
                                ;;
        * )                     usage
                                exit 1
esac

# Locations to backup
SRC=( /home/stuart /etc /home/maxwell )

# Locations and files to exclude 
EXCLUDES=( "Downloads" ".cache" ".config" ".pcloud" "Music" "Pictures"  \
    ".mozilla" ".local" "pCloudDrive" "www/hugo/*/public" ".wine")

# echo "Backing up" $SRC "to" $DEST "excluding" ${EXCLUDES[@]}

# PW=--password-file="/home/stuart/.restic_pw"

# Add an --exclude prefix to each of the excluded directories
# above and concatenate them so they can be added to the command line
exclude_opts=()
for item in "${EXCLUDES[@]}" 
do
  exclude_opts+=(--exclude "$item" )
done

# Make Backblaze environment variables available
source /home/stuart/.restic_env

# Runtime options
OPTIONS="--exclude-caches"

case $DEST in
    $NAS | $LOCAL | $REMOTE )
        # run the specified backup for all source locations
        for src_to_backup in "${SRC[@]}"
        do
            $RESTIC -r $DEST backup $src_to_backup --exclude-caches --verbose "${exclude_opts[@]}"
        done
        ;;
    "ALL" )
        # run all backups
        for src_to_backup in "${SRC[@]}"
        do
            $RESTIC -r $NAS backup $src_to_backup $OPTIONS "${exclude_opts[@]}"
            $RESTIC -r $LOCAL backup $src_to_backup $OPTIONS "${exclude_opts[@]}"
            $RESTIC -r $REMOTE backup $src_to_backup $OPTIONS "${exclude_opts[@]}"
        done
        ;;
    "SNAP" )
        # show all snapshots
        $RESTIC -r $NAS snapshots
        $RESTIC -r $LOCAL snapshots
        $RESTIC -r $REMOTE snapshots
esac