Hmm. My newly-created backup was failing due to this dir missing, but it surely
isn't the first backup that has been added when we've had the current
version of the prepare script in place. Yet I see us calling mkdir in
the collection of old prepare scripts, and there's no attempt to create
it in the script that creates new backups ....

We were referencing a db field that doesn't exist.

Closes #6

When running psql commands to insert rows in the database, psql normally
returns an message about what it did, eg "INSERT 0 1" if it inserted a
row. This can be suppressed with -q . Several of the scripts use psql
commands to get primary keys from the database, inserting the row if
necessary. This can lead to the host id variable in the script being set
to 'INSERT 0 1 <thehostid>' which causes problems when this variable is
used in other SQL commands.

This always used to work; I suspect the thing that changed is our
upgrading to Postgres 16 on the backup servers, but I'm struggling to
see how as Postgres 13 seems to behave the same for me.

We had ended up, somehow, with a few hosts on one backup server
which appeared twice in `host` - one with disabled=f and some backup
tasks as expected, and one with disabled=t. I manually (necessarily)
deleted the latter before adding this constraint on the live servers.

(all servers listed in zfs_backup_server.conf have had the db table
manually updated)

I think the intention here was perhaps:

- create new host, marked as disabled
- finish setting up backups
- once done, mark host as enabled

...except the script runs as "set -e", so if something goes wrong
we just never get as far as enabling the host, which means not only
do no backups run but no failure reports get sent to xymon so we
don't even notice the failure. This is not good.

Given the entry of a row in the `host` table doesn't do much in and
of itself, I see no reason why we shouldn't just mark the host as
initially enabled. We won't try to actually perform a backup until
a `backup_task` has been created. Perhaps this leads to a brief
transient behaviour where xymon reports a backup as failing whilst
the script is still running - but OTOH the xymon report for a new
machine will always be red for "a while" until the first backup
has actually run OK.

We don't need to record the ssh host key in most cases given that we
generally deploy signed ssh host keys, but I suspect we might have the
occasional backup target where that doesn't apply (e.g. clusters?)

Regardless, if we can't scan the host key the right behaviour is for
the script to continue on and set up the backup. If the backup then
fails due to the absent host key, we will be alerted and take suitable
action. Right now, the failure mechanism is that we silently don't finish
setting up the backup, the backup never gets enabled, and we don't
realise we don't have a backup - eek.

For machines like nest-backup and cerebro-backup we have lots of backup
tasks for the same host spread across several zpools, so
move-machine-to-zpool.sh can't be used to migrate the contents of a
failing zpool/disk. This adds a script to move an individual ZFS which is
the target of a backup task to another zpool.

It assumes all necessary parent ZFSes already exist on the target. If
they don't it fails.

It does not yet clean up the old ZFS as it's not had a lot of use.

We have started putting extra configuration for sshing to a host in a
file in the /etc/chem-zfs-backup-server/zfs-rsync.d/$HOSTNAME directory.
This updates the backup migration script to copy that as well as the
main config file for the machine. I've chosen to copy the entire
directory to catch other files we might want to add in future. There is
often an 'exclude' file in there that's autogenerated by the prepare
scripts, but copying that won't do any damage; it's just redundant
because it will be regenerated when the backup runs.

1. The export is done via set sharenfs which means we shouldn't need
   to manually manage exports

2. This part of the script does not work because it tries to unexport
   the old export but by looking up the db record that we have already
   updated to refer to the new zpool.

This adds a new config file which allows setting the command to use for
'rsync' and global options for that command. This is motivated by the
need to use an alternative rsync command on Jammy machines, as the
system one is too slow.

The option for global rsync arguments was added as a way to add the
'--trust-sender' flag to all backups to turn off certain checks that we
suspect to be the cause of the slowdown, but it didn't help enough to
fix the speed problem. Instead we are going to use our own package of an
older rsync from before the checking code was added, which of course
doesn't support --trust-sender so the global args are left blank.

Custom options need to be a file passed via -F because we want
to specify options for both ssh and scp. They don't have a compatible
set of CLI options but both take -F.

This supercedes 53f5ba49 ; I had only deployed SSHOPTIONS for one host which
I've updated.

This also removes the SSHPORT option, which had only been used in the config
for one host which I've updated.

These are static files provided by our package, not config files.

These are not config files, and we should not be modifying the package-provided
versions of these files. I'm leaving symlinks behind to make sure we don't break
all our existing backups though!

This could/should probably supercede the specific option for SSHPORT
as I think usage of that is minimal or perhaps even zero, but we'd
have to check if that's in use and make suitable updates to config
files before removing it.

i.e. we can now simply

delete from host where hostname='example.ch.private.cam.ac.uk';

without having to chase the foreign keys.

I've made the equivalent change on our live backup servers with
an ad hoc script.

i.e. send the ZFS, update db records and copy the config files. The
sql inserts should broadly mirror those done when setting up a new
backup, though with field values matching those in the source database
rather than just using the defaults.

This script has so only been tested for the case of moving a "simple"
backup where a host has a single backup task and no special config.
It's quite likely there'll be bugs to fix for other cases that
we'll find in due course.

I'm about to add a script to send a backup to a different backup server.
It's thus probably best if the script names describe their functions in a
little more detail

https://tickets.ch.cam.ac.uk/rt/Ticket/Display.html?id=211460

e.g. RT 211460, 211465. spri-musuem-rt-2025 had been set up with an adhoc script
which was not properly tidying up after itself due to bailing early on a "set -e"
error.

In some versions of backup_queue (I think just on splot4 now), we use
backup_log.isrunning as part of the logic to determine if a task should be
enqueued. The problem is that scheduler.pl makes three writes on the table:

1) an insert when the task is queued (a trigger sets isrunning='t' here)
2) an update to set started_processing when the task begins (a trigger
   sets isrunning='f' here!!!!)
3) an update to set ended_processing when the task finishes (a trigger
   again sets isrunning='f' here)

Thus, being careful to only set isrunning='f' when a backup task is finished
(i.e. when we set ended_processing=now() in scheduler.pl) seems sensible, and
empirically does seem to lead to the right backup_queue without duplicates.

This commit will only affect new setups of backup servers; the change has been
deployed to live servers with an ad hoc script I've run.

I think we only see this on splot4 because it has a very different definition of
the backup_queue view to a) the one defined in this file, b) the one that's on
all the other backup servers. If I just try to replace the view on splot4, though,
any attempt to select from it just times out so there may be other relations on
splot4 that need updating too.

NB the obvious thing missing on splot4 is

WHERE ((backup_log.backup_task_id = a.backup_task_id) AND (backup_log.ended_processing IS NULL))) < 1))

which feels like a hack but nonetheless ensures in practice that we don't get
duplicate queued tasks.

We don't always need the role data, if the presumption is that we'll
be doing a pg_restore in conjunction with an ansible role which creates
all required roles. But, having a copy of the role data will never hurt!
It also gives us a straightforward way of restoring a database to a
standalone postgres instance without having to have provisioned a
dedicated VM with the relevant ansible roles.

At present we use myriad one-off per host scripts to do a pg_dump,
and they all do (or probably should do) the same thing. In combination
with setting options in the host's backup config file, I think
this single script covers all our routine pg backups.

we were just passing the hostname. Adding extra args should
not impact any existing script, but will let us write better/
more maintainable/deduplicated PRE scripts

This came about because a disk has failed on nest-backup, which only has
subdirectory backups of nest-filestore-0 and so move-machine.sh was not
going to be helpful - it assumes all tasks for a machine are on the same
zpool which isn't true there. In this case I did the move by hand, but
have sketched out the steps in the script in the hope that next time we
have to do this we'll do it by looking at the script and running bits by
hand, then improve the script a bit, and continue until it's usable.

I don't think there's a sensible default quota; the value for
a workstation will be very different to a tiny VM, for example.

This is needed on focal if a client is to be able to access snapshots over NFS.
From the docs I don't see why we didn't also need this option on xenial,
but empirically, we need it on focal. (e.g. RT-207229)

The generation of the command to unexport NFS filesystems could generate
an invalid command. Leading spaces were not being stripped, and in cases
where there is more than one backup target for a machine we need to
unexport every target. Because we also had 'set -e' in operation at this
point, the script would fail there and never clean up the moved ZFS. I
don't mind if we fail to unexport; if that's subsequently a problem for
removing the ZFS then the script will fail at that point.

This change makes the script generate better exportfs -u commands and
not exit if they fail.

The code used to open a database connection for each thread and leave
them open for as long as the scheduler ran. This worked reasonably well
until we moved to PostgreSQL 13 on Focal, although the scheduler would
fail if the database was restarted because there was no logic to
reconnect after a connection dropped.

On Focal/PG13 the connection for the 'cron' thread steadily consumes
memory until it has exhausted everything in the machine. This appears to
be a Postgres change rather than a Perl DBI change: the problem can be
reproduced by sitting in psql and running 'select * from backup_queue'
repeatedly. Once or twice a minute an instance of this query will cause
the connection to consume another MB of RAM which is not released until
the database connection is closed. The cron thread runs that query every
two seconds. My guess is it's something peculiar about the view that
query selects from - the time interval thing is interesting.
This needs more investigation.

But in the meantime I'd like to have backup servers that don't endlessly
gobble RAM, so this change makes the threads connect to the database
only when they need to, and closes the connection afterwards. This
should also make things work better over database restarts but that's
not been carefully tested.

Can't say "keys $foo", must say "keys %{$foo}" to get the keys of a hash
pointed to by a hash reference $foo. This broke with the perl in focal.

NB this is so we can put things in LOGFILE at earlier points
than the script does at present