tempfile is a debian-ism and jammy warns us that use is deprecated.
mktemp has been available on all our debian/ubuntu machines for a long time
via coreutils (e.g. it was definitely in wheezy and trusty, and I'm pretty
sure since before then too)

The incremental backups on some cluster head nodes are growing quite
large, and most of the churn is the uncompressed MySQL dumpfile which
changes with every backup and can be over 1GB. This commit compresses
that data, which reduces the size of the file by 90% on at least one
machine.

As of b4b89ed3 we need the prepare script to exit zero on success.

We only use the stdout from this command, not the return code, for
determining if the target is a xen VM.

This has been silently masking failures, which is a Bad Thing. The only
errors I've spotted so far are ones hopefully fixed by ad91da90, which
has the effect that we've been backing up more things than we needed
(which is better than the opposite possibility, at least!)

c714a331 introduced an annoying bug here. If the filenames piped to xargs include
an apostrophe, xargs will complain and stop. We have thus, in practice, been
including all files that appear in the list after

/usr/share/sounds/ubuntu/ringtones/Sam's Song.ogg

!! Also, we've not been properly handling filenames with spaces in, and perhaps
other filenames too. We thus null-terminate the filenames for ourself.

Because I want to get to the point where we can have a sensible return code
from the prepare script, this commit also adds some set -e commands to try
to ensure errors bubble up. This is a little tedious to achieve due to all
the subshells in this script.

For the same reason, we split off the "diff" command into a "set +e" block
because diff returns non-zero if differences are found. We do not consider
that to be an error!

We think the intention of the old version of this block is

"if mysql is running, dump the databases".

The check has been buggy for a long long time: it reads the contents
of my.cnf ... which nowadays just has some !includedir directives.
This lead to setting

SOCKET=""

and, it turns out, [ -S "" ] returns true.

The main intention of this part of the prepare script is to backup
vaguely normal machines running a simple mysql database, such as
small group webservers. We thus don't need to consider every
eventuality.

On servers we configure this via ansible.

On workstations it is not having the desired effect because the stock
logrotate.conf includes the line ...

and so the simple grep thinks it's already configured! I'll add an
ansible task in due course to actually configure this.

Now that we use pygrub, these dirs are populated by quite a lot of files
that we don't want to back up but are dynamically built by the kernel/related
packages so do not make it into the the prepared "excludes" file

prepare-nondebian does not work on RedHat machines running MySQL as the
paths are different, so providing a fixed version. prepare-nondebian has
historically been used more widely than just RedHat, hence the decision
to provide a RedHat-specific version and not just edit it.

As of focal a bunch of top-level dirs are symlinks (eg /lib -> /usr/lib) but
the deb packages still deploy files to the symlink rather than the real dir.
Thus if we just take the contents of all the *.list files we end up not
excluding lots of files that are in fact provided by debs

The default prepare script now uses appropriate options if it detects Postgres
12 or higher. The error handling still needs work though.

Update prepare script for newer Postgres

Looks like Subiquity sets up a swapfile at /swap.img on Ubuntu 20.04 by default
and we certainly don't want to back that up.

We don't need these backed up, and having them in the list leads to
an error due to not having show_compatibility_56 enabled

Resolves #3

On a new machine, there's nothing to remove and so rm will return
an error. I'd like our prepare scripts not to return errors
unnecessarily!

Ignore /usr/share/doc, /usr/share/texlive - takes a long time to enumerate contents which we then exclude anyway due to being deb-provided

Tidy global_ignore - remove some long-since unneeded lines that should have been host-specific in the first place

e.g. /DATA for apps, /home/web for atm webserver. Also /run was in twice...

Our backup logs are generally full of

+ ssh -p 22 -o ConnectTimeout=10 root@openvpn2020.ch.cam.ac.uk '
  umask 077
  FILELIST=`tempfile`
  CONFLIST=`tempfile`
  mkdir -p /var/adm/backup
  # Make logrotate use datestamps
  if ! grep -q dateext /etc/logrotate.conf ; then sed  -i '\''s/^include/dateext\ninclude/'\'' /etc/logrotate.conf ; fi
  # Which packages are installed?
  dpkg --get-selections | awk '\'' { print $1 ; } '\'' >/var/adm/backup/packages
  cat /var/lib/dpkg/info/*.list | while read F ; do [ -f "$F" ] && echo "$F" ; done | sort > $FILELIST
  awk '\''/Description:/ { flag = 0 } ; flag == 1 { print $1 ; }  ; /Conffiles:/ { flag = 1  } ; '\'' </var/lib/dpkg/status |  sort >$CONFLIST
  diff -u $FILELIST $CONFLIST | grep ^-/ | sed s/^-// >/var/adm/backup/package-files
  rm
 '
rm: missing operand

because we're not escaping the args to that final rm command! This means we gradually fill
up /tmp, especially on machines that are infrequently rebooted (e.g. calculon)

prepare-olddebian : does not do package file excludes
prepare-database: does not do package file excludes or dump databases

These are for use on machines where we can no longer reinstate them by
quickly and easily installing a new VM from the network and then
layering the backup on top, usually because the OS is obsolete.

This seems to be used for machines with postgres. It was also updated
to put exclude and log files in the new locations.

Split logs for tasks with same target machine but different target filesystems into
different logfiles for easier debugging.

prepare-nondebian script creates a fake excludes file for the machine now if
one doesn't exist, because I keep forgetting to do it myself for Redhat
machines and then the zfs-rsync script fails.