zback issueshttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues2021-10-24T14:58:23Zhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/29Feature: pre and post hooks2021-10-24T14:58:23ZJ. MarshallFeature: pre and post hooksNot 100% sure if this is even necessary for most applications; for example postgres already does transactional commits and so in theory can't get inconsistent. However, for things like backing up VMs that are ZFS backed it might make sen...Not 100% sure if this is even necessary for most applications; for example postgres already does transactional commits and so in theory can't get inconsistent. However, for things like backing up VMs that are ZFS backed it might make sense - additionally if you have some esoteric thing that you want to dump out of something before taking a snapshot that might be useful.
Pre and post snapshot taking is the thing.J. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/28Bug : snapshot matching is crap and stops working as soon as things get out o...2021-10-11T16:46:31ZJ. MarshallBug : snapshot matching is crap and stops working as soon as things get out of syncNot sure how I managed to make this, its a little embarrassing.Not sure how I managed to make this, its a little embarrassing.J. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/26BUG: init and resend specification not matching substring correctly2021-09-30T14:15:57ZJ. MarshallBUG: init and resend specification not matching substring correctlyIn theory this feature was supposed to ease the case where multiple destinations are specified and allow you to pick one. However, two problems exist, one is that this is not clear at all, even I forgot about it and the second is that it...In theory this feature was supposed to ease the case where multiple destinations are specified and allow you to pick one. However, two problems exist, one is that this is not clear at all, even I forgot about it and the second is that it requires the whole send string when it should be able to accept a substring.https://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/25Bug: Destination does not exist doesn't cancel send2021-09-03T13:40:48ZJ. MarshallBug: Destination does not exist doesn't cancel sendNoticed when zpool not imported - errors out on remote dataset lookup but this apparently does not cancel teh send job, which only fails after trying to zfs recv into nothing.Noticed when zpool not imported - errors out on remote dataset lookup but this apparently does not cancel teh send job, which only fails after trying to zfs recv into nothing.J. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/23Bug: Complete crapout if host not in known_hosts2021-07-08T10:59:44ZJ. MarshallBug: Complete crapout if host not in known_hosts11:49:45::DEBUG: Destination is remote
11:49:48::DEBUG: Server 'nas-zsrv1.cri.camres.org' not found in known_hosts
11:49:53::DEBUG: Server 'nas-zsrv1.cri.camres.org' not found in known_hosts
11:49:54::DEBUG: Server 'nas-zsrv1.cri.camres....11:49:45::DEBUG: Destination is remote
11:49:48::DEBUG: Server 'nas-zsrv1.cri.camres.org' not found in known_hosts
11:49:53::DEBUG: Server 'nas-zsrv1.cri.camres.org' not found in known_hosts
11:49:54::DEBUG: Server 'nas-zsrv1.cri.camres.org' not found in known_hosts
11:49:57::DEBUG: Server 'nas-zsrv1.cri.camres.org' not found in known_hosts
Oops, unhandled type 3 ('unimplemented')J. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/22Feature: Pull support2021-07-07T11:58:07ZJ. MarshallFeature: Pull supportNot a massive priority for our use case but given the way things are structured it shouldn't be that hard to simply use a remote dataset as an argument and pull instead of push...Not a massive priority for our use case but given the way things are structured it shouldn't be that hard to simply use a remote dataset as an argument and pull instead of push...J. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/21Bug: configure script does not recognise tag as it has not been added yet2021-07-07T10:32:36ZJ. MarshallBug: configure script does not recognise tag as it has not been added yetPretty obvious in hindsightPretty obvious in hindsightJ. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/18Bug: Error reading SSH protocol banner2021-12-06T09:36:09ZJ. MarshallBug: Error reading SSH protocol bannerThis is happening due to too many sends being in the unauthenticated state at the same time. One thing that can be done to mitigate this is to add a random start delay in each send process - I don't want to have to tweak the sshd config ...This is happening due to too many sends being in the unauthenticated state at the same time. One thing that can be done to mitigate this is to add a random start delay in each send process - I don't want to have to tweak the sshd config on the other end as this is kind of against the spirit of things.
Things that have been tried so far that make no difference - adjusting banner_timeout on the SSH client to 120 and introducing an auth_timeout. I will leave these in place as this is probably good for when a server is just a bit overloaded, but the way sshd works is it starts probabilistically rejecting connections when there are more than 10 unathed at the same time, so the connection gets dropped and no timeout will help.
I could introduce a reconnect, which is probably a good thing to be doing anyway, as there could be a variety of reasons for a connection to fail.
I am leaving the code as is for now though as the most annoying part of this is that the error does not appear to be being caught by any of the exception handling - this should count as a send fail and I should be getting an email, so something is fucked somewhere along the line.J. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/17Feature: Use delegated ZFS permissions and zback user account for better secu...2020-02-21T17:53:41ZJ. MarshallFeature: Use delegated ZFS permissions and zback user account for better securityThe idea here would be for a central backup server that pools storage from several servers that for whatever reason should not have root access to the central backup server. The major reason would be passwordless SSH keys; if a client ge...The idea here would be for a central backup server that pools storage from several servers that for whatever reason should not have root access to the central backup server. The major reason would be passwordless SSH keys; if a client gets compromised it no longer has root access to the main backup server that can potentially server many clients.J. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/15Bug: SSH local user config not loaded2021-07-08T10:59:45ZJ. MarshallBug: SSH local user config not loadedLoading user `~/.ssh/config` allows for aliases as well as resolveable hostsLoading user `~/.ssh/config` allows for aliases as well as resolveable hostsJ. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/14NTH: Implement send/recv resume support2019-12-09T13:37:21ZJ. MarshallNTH: Implement send/recv resume supportThis is a complicated one - only pertains if a send has been interrupted. In an ideal world, this would happen if the program had encountered a problem while sending and then had shut down gracefully, removing the send tag. However if th...This is a complicated one - only pertains if a send has been interrupted. In an ideal world, this would happen if the program had encountered a problem while sending and then had shut down gracefully, removing the send tag. However if the program crashes then the send tag will still be there.
Needs some thought.J. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/13NTH: Send limits2020-07-31T12:02:31ZJ. MarshallNTH: Send limitsNice to have:
Send limits - do a dry send on the snapshot iteration, if it is above a certain limit then abandon save and warn.
TDB: How to set this - additional dataset property, but what?Nice to have:
Send limits - do a dry send on the snapshot iteration, if it is above a certain limit then abandon save and warn.
TDB: How to set this - additional dataset property, but what?J. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/12NTH: Status output2019-12-09T13:36:34ZJ. MarshallNTH: Status outputNice to have:
Status output - nicely formatted table of what datasets are managed, what their current status is (sending, sent, what else?), set properties etc.
TBD: decide what else to display, how to formatNice to have:
Status output - nicely formatted table of what datasets are managed, what their current status is (sending, sent, what else?), set properties etc.
TBD: decide what else to display, how to formatJ. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/7Feature: TLS encryption2019-12-09T13:34:17ZJ. MarshallFeature: TLS encryptionActually pretty straightforward, can just wrap a socket in TLS and netcat can do SSLActually pretty straightforward, can just wrap a socket in TLS and netcat can do SSLJ. MarshallJ. Marshallhttps://gitlab.developers.cam.ac.uk/jm2193/zback/-/issues/4Closing socket doesn't terminate remote netcat2019-12-09T13:31:48ZJ. MarshallClosing socket doesn't terminate remote netcatMeans that a local orderly close could leave phantom threads and indeed has.Means that a local orderly close could leave phantom threads and indeed has.J. MarshallJ. Marshall