The Xymon test that reports on backup status runs every 45 minutes.  But
the status of an individual backup does not change very frequently - we
try to back most things up a few times a day. This change makes the
individual backup statuses valid for three hours, rather than the one
hour they were previously. This is to avoid getting purple dots when we
we reboot a backup server and interrupt the 45 minute check, which then
won't run again for another 45 minutes causing a 90 minute gap between
reports for some hosts and hence purple dots.

For machines like nest-backup and cerebro-backup we have lots of backup
tasks for the same host spread across several zpools, so
move-machine-to-zpool.sh can't be used to migrate the contents of a
failing zpool/disk. This adds a script to move an individual ZFS which is
the target of a backup task to another zpool.

It assumes all necessary parent ZFSes already exist on the target. If
they don't it fails.

It does not yet clean up the old ZFS as it's not had a lot of use.

We have started putting extra configuration for sshing to a host in a
file in the /etc/chem-zfs-backup-server/zfs-rsync.d/$HOSTNAME directory.
This updates the backup migration script to copy that as well as the
main config file for the machine. I've chosen to copy the entire
directory to catch other files we might want to add in future. There is
often an 'exclude' file in there that's autogenerated by the prepare
scripts, but copying that won't do any damage; it's just redundant
because it will be regenerated when the backup runs.

1. The export is done via set sharenfs which means we shouldn't need
   to manually manage exports

2. This part of the script does not work because it tries to unexport
   the old export but by looking up the db record that we have already
   updated to refer to the new zpool.

This adds a new config file which allows setting the command to use for
'rsync' and global options for that command. This is motivated by the
need to use an alternative rsync command on Jammy machines, as the
system one is too slow.

The option for global rsync arguments was added as a way to add the
'--trust-sender' flag to all backups to turn off certain checks that we
suspect to be the cause of the slowdown, but it didn't help enough to
fix the speed problem. Instead we are going to use our own package of an
older rsync from before the checking code was added, which of course
doesn't support --trust-sender so the global args are left blank.

This was unintentionally removed in 53f5ba

Custom options need to be a file passed via -F because we want
to specify options for both ssh and scp. They don't have a compatible
set of CLI options but both take -F.

This supercedes 53f5ba49 ; I had only deployed SSHOPTIONS for one host which
I've updated.

This also removes the SSHPORT option, which had only been used in the config
for one host which I've updated.

These are static files provided by our package, not config files.

These are not config files, and we should not be modifying the package-provided
versions of these files. I'm leaving symlinks behind to make sure we don't break
all our existing backups though!

I've checked all our live backup servers and made sure these aren't in use
(and updated some backup configs to achieve this in some cases)

This could/should probably supercede the specific option for SSHPORT
as I think usage of that is minimal or perhaps even zero, but we'd
have to check if that's in use and make suitable updates to config
files before removing it.

We had been raising a failure report if we had never seen a succesful backup
for a host. However, when we have a host with more than one backup task, we
can have the situation where one backup is working OK but the other has never
completed correctly. This let to a green report as we had a non-zero number
of rows, but we require number_of_good_backups == number_of_tasks !

tempfile is a debian-ism and jammy warns us that use is deprecated.
mktemp has been available on all our debian/ubuntu machines for a long time
via coreutils (e.g. it was definitely in wheezy and trusty, and I'm pretty
sure since before then too)

The incremental backups on some cluster head nodes are growing quite
large, and most of the churn is the uncompressed MySQL dumpfile which
changes with every backup and can be over 1GB. This commit compresses
that data, which reduces the size of the file by 90% on at least one
machine.

This can happen for a number of reasons:

- child ZFS has a quota bigger than its parent
- we have over-provisioned such that the sum of quotas is bigger than the disk
- an individual ZFS has a quota bigger than the disk

..and convert such things to human-friendly versions if required.

This is to facilitate extra checks where we want to make numeric
calculations involving quotas and other similar properties.

As of b4b89ed3 we need the prepare script to exit zero on success.

We only use the stdout from this command, not the return code, for
determining if the target is a xen VM.

This has been silently masking failures, which is a Bad Thing. The only
errors I've spotted so far are ones hopefully fixed by ad91da90, which
has the effect that we've been backing up more things than we needed
(which is better than the opposite possibility, at least!)

c714a331 introduced an annoying bug here. If the filenames piped to xargs include
an apostrophe, xargs will complain and stop. We have thus, in practice, been
including all files that appear in the list after

/usr/share/sounds/ubuntu/ringtones/Sam's Song.ogg

!! Also, we've not been properly handling filenames with spaces in, and perhaps
other filenames too. We thus null-terminate the filenames for ourself.

Because I want to get to the point where we can have a sensible return code
from the prepare script, this commit also adds some set -e commands to try
to ensure errors bubble up. This is a little tedious to achieve due to all
the subshells in this script.

For the same reason, we split off the "diff" command into a "set +e" block
because diff returns non-zero if differences are found. We do not consider
that to be an error!

We think the intention of the old version of this block is

"if mysql is running, dump the databases".

The check has been buggy for a long long time: it reads the contents
of my.cnf ... which nowadays just has some !includedir directives.
This lead to setting

SOCKET=""

and, it turns out, [ -S "" ] returns true.

The main intention of this part of the prepare script is to backup
vaguely normal machines running a simple mysql database, such as
small group webservers. We thus don't need to consider every
eventuality.

On servers we configure this via ansible.

On workstations it is not having the desired effect because the stock
logrotate.conf includes the line ...

and so the simple grep thinks it's already configured! I'll add an
ansible task in due course to actually configure this.

Now that we use pygrub, these dirs are populated by quite a lot of files
that we don't want to back up but are dynamically built by the kernel/related
packages so do not make it into the the prepared "excludes" file

i.e. we can now simply

delete from host where hostname='example.ch.private.cam.ac.uk';

without having to chase the foreign keys.

I've made the equivalent change on our live backup servers with
an ad hoc script.

i.e. send the ZFS, update db records and copy the config files. The
sql inserts should broadly mirror those done when setting up a new
backup, though with field values matching those in the source database
rather than just using the defaults.

This script has so only been tested for the case of moving a "simple"
backup where a host has a single backup task and no special config.
It's quite likely there'll be bugs to fix for other cases that
we'll find in due course.

I'm about to add a script to send a backup to a different backup server.
It's thus probably best if the script names describe their functions in a
little more detail