This was unintentionally removed in 53f5ba

Custom options need to be a file passed via -F because we want
to specify options for both ssh and scp. They don't have a compatible
set of CLI options but both take -F.

This supercedes 53f5ba49 ; I had only deployed SSHOPTIONS for one host which
I've updated.

This also removes the SSHPORT option, which had only been used in the config
for one host which I've updated.

These are static files provided by our package, not config files.

These are not config files, and we should not be modifying the package-provided
versions of these files. I'm leaving symlinks behind to make sure we don't break
all our existing backups though!

I've checked all our live backup servers and made sure these aren't in use
(and updated some backup configs to achieve this in some cases)

This could/should probably supercede the specific option for SSHPORT
as I think usage of that is minimal or perhaps even zero, but we'd
have to check if that's in use and make suitable updates to config
files before removing it.

We had been raising a failure report if we had never seen a succesful backup
for a host. However, when we have a host with more than one backup task, we
can have the situation where one backup is working OK but the other has never
completed correctly. This let to a green report as we had a non-zero number
of rows, but we require number_of_good_backups == number_of_tasks !

tempfile is a debian-ism and jammy warns us that use is deprecated.
mktemp has been available on all our debian/ubuntu machines for a long time
via coreutils (e.g. it was definitely in wheezy and trusty, and I'm pretty
sure since before then too)

The incremental backups on some cluster head nodes are growing quite
large, and most of the churn is the uncompressed MySQL dumpfile which
changes with every backup and can be over 1GB. This commit compresses
that data, which reduces the size of the file by 90% on at least one
machine.

This can happen for a number of reasons:

- child ZFS has a quota bigger than its parent
- we have over-provisioned such that the sum of quotas is bigger than the disk
- an individual ZFS has a quota bigger than the disk

..and convert such things to human-friendly versions if required.

This is to facilitate extra checks where we want to make numeric
calculations involving quotas and other similar properties.

As of b4b89ed3 we need the prepare script to exit zero on success.

We only use the stdout from this command, not the return code, for
determining if the target is a xen VM.

This has been silently masking failures, which is a Bad Thing. The only
errors I've spotted so far are ones hopefully fixed by ad91da90, which
has the effect that we've been backing up more things than we needed
(which is better than the opposite possibility, at least!)

c714a331 introduced an annoying bug here. If the filenames piped to xargs include
an apostrophe, xargs will complain and stop. We have thus, in practice, been
including all files that appear in the list after

/usr/share/sounds/ubuntu/ringtones/Sam's Song.ogg

!! Also, we've not been properly handling filenames with spaces in, and perhaps
other filenames too. We thus null-terminate the filenames for ourself.

Because I want to get to the point where we can have a sensible return code
from the prepare script, this commit also adds some set -e commands to try
to ensure errors bubble up. This is a little tedious to achieve due to all
the subshells in this script.

For the same reason, we split off the "diff" command into a "set +e" block
because diff returns non-zero if differences are found. We do not consider
that to be an error!

We think the intention of the old version of this block is

"if mysql is running, dump the databases".

The check has been buggy for a long long time: it reads the contents
of my.cnf ... which nowadays just has some !includedir directives.
This lead to setting

SOCKET=""

and, it turns out, [ -S "" ] returns true.

The main intention of this part of the prepare script is to backup
vaguely normal machines running a simple mysql database, such as
small group webservers. We thus don't need to consider every
eventuality.

On servers we configure this via ansible.

On workstations it is not having the desired effect because the stock
logrotate.conf includes the line ...

and so the simple grep thinks it's already configured! I'll add an
ansible task in due course to actually configure this.

Now that we use pygrub, these dirs are populated by quite a lot of files
that we don't want to back up but are dynamically built by the kernel/related
packages so do not make it into the the prepared "excludes" file

i.e. we can now simply

delete from host where hostname='example.ch.private.cam.ac.uk';

without having to chase the foreign keys.

I've made the equivalent change on our live backup servers with
an ad hoc script.

i.e. send the ZFS, update db records and copy the config files. The
sql inserts should broadly mirror those done when setting up a new
backup, though with field values matching those in the source database
rather than just using the defaults.

This script has so only been tested for the case of moving a "simple"
backup where a host has a single backup task and no special config.
It's quite likely there'll be bugs to fix for other cases that
we'll find in due course.

I'm about to add a script to send a backup to a different backup server.
It's thus probably best if the script names describe their functions in a
little more detail

https://tickets.ch.cam.ac.uk/rt/Ticket/Display.html?id=211460

e.g. RT 211460, 211465. spri-musuem-rt-2025 had been set up with an adhoc script
which was not properly tidying up after itself due to bailing early on a "set -e"
error.

This will let us use zfs_target as the name of a subtest which
in turn means we would be able to separately log and graph multiple
backup targets associated with a single host.

This change does not affect the current parsing performed when
we input data into postgres: it uses non-anchored regexps to
identify SpaceUsed etc so prepending extra text won't change
anything

In some versions of backup_queue (I think just on splot4 now), we use
backup_log.isrunning as part of the logic to determine if a task should be
enqueued. The problem is that scheduler.pl makes three writes on the table:

1) an insert when the task is queued (a trigger sets isrunning='t' here)
2) an update to set started_processing when the task begins (a trigger
   sets isrunning='f' here!!!!)
3) an update to set ended_processing when the task finishes (a trigger
   again sets isrunning='f' here)

Thus, being careful to only set isrunning='f' when a backup task is finished
(i.e. when we set ended_processing=now() in scheduler.pl) seems sensible, and
empirically does seem to lead to the right backup_queue without duplicates.

This commit will only affect new setups of backup servers; the change has been
deployed to live servers with an ad hoc script I've run.

I think we only see this on splot4 because it has a very different definition of
the backup_queue view to a) the one defined in this file, b) the one that's on
all the other backup servers. If I just try to replace the view on splot4, though,
any attempt to select from it just times out so there may be other relations on
splot4 that need updating too.

NB the obvious thing missing on splot4 is

WHERE ((backup_log.backup_task_id = a.backup_task_id) AND (backup_log.ended_processing IS NULL))) < 1))

which feels like a hack but nonetheless ensures in practice that we don't get
duplicate queued tasks.

We don't always need the role data, if the presumption is that we'll
be doing a pg_restore in conjunction with an ansible role which creates
all required roles. But, having a copy of the role data will never hurt!
It also gives us a straightforward way of restoring a database to a
standalone postgres instance without having to have provisioned a
dedicated VM with the relevant ansible roles.

At present we use myriad one-off per host scripts to do a pg_dump,
and they all do (or probably should do) the same thing. In combination
with setting options in the host's backup config file, I think
this single script covers all our routine pg backups.