FAQ | This is a LIVE service | Changelog

Commit 3941a9df authored by Dr Adam Thorn's avatar Dr Adam Thorn
Browse files

Partial fix for behaviour where we see multiple backups for one task running at once

In some versions of backup_queue (I think just on splot4 now), we use
backup_log.isrunning as part of the logic to determine if a task should be
enqueued. The problem is that scheduler.pl makes three writes on the table:

1) an insert when the task is queued (a trigger sets isrunning='t' here)
2) an update to set started_processing when the task begins (a trigger
   sets isrunning='f' here!!!!)
3) an update to set ended_processing when the task finishes (a trigger
   again sets isrunning='f' here)

Thus, being careful to only set isrunning='f' when a backup task is finished
(i.e. when we set ended_processing=now() in scheduler.pl) seems sensible, and
empirically does seem to lead to the right backup_queue without duplicates.

This commit will only affect new setups of backup servers; the change has been
deployed to live servers with an ad hoc script I've run.

I think we only see this on splot4 because it has a very different definition of
the backup_queue view to a) the one defined in this file, b) the one that's on
all the other backup servers. If I just try to replace the view on splot4, though,
any attempt to select from it just times out so there may be other relations on
splot4 that need updating too.

NB the obvious thing missing on splot4 is

WHERE ((backup_log.backup_task_id = a.backup_task_id) AND (backup_log.ended_processing IS NULL))) < 1))

which feels like a hack but nonetheless ensures in practice that we don't get
duplicate queued tasks.
parent 67d141b5
......@@ -376,9 +376,12 @@ CREATE FUNCTION update_ended_processing() RETURNS trigger
declare
begin
update backup_task set last_ended=new.ended_processing where backup_task_id=new.backup_task_id and last_ended<new.ended_processing;
update backup_task set isrunning='f' where backup_task_id=new.backup_task_id and isrunning;
return new;
-- NB we also update the backup_log table to set started_processing; we only want to do the following updates if we have ended_processing
if new.ended_processing is not null then
update backup_task set last_ended=new.ended_processing where backup_task_id=new.backup_task_id;
update backup_task set isrunning='f' where backup_task_id=new.backup_task_id and isrunning;
end if;
return new;
end
$$;
......
  • For posterity, I've now also updated the definition of backup_queue on splot4 to match the other backup servers. It queries backup_log, and due to historical legacy that table was rather huge on splot4. I've done much housekeeping and trimmed ~4e6 rows, and splot4 can now query its (new) backup_queue view in a sensible time, O(second).

Supports Markdown
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment