Determine the fall out from deleting user accounts after they have been suspended for more than a year

We currently never delete accounts. After speaking with the CISO, we agreed that we should delete the accounts after they have been suspended for a year.

This raises some questions:

• Can we recover documents, etc from deleted users?
• Should we transfer documents, etc to a "grandfather" user?
• Do documents, etc shared by users become inaccessible to people shared with as soon as the user is suspended or only when deleted? If the former, that's better since we get some warning about important documents in danger of deletion. If the latter we need to have to recovery plan.

changed milestone to %DevOps Sprint 106

added 1 deleted label

added priority2 Medium label

changed the description

changed time estimate to 1d

added spike workflowSprint Ready labels and removed 1 deleted label

changed title from Delete user accounts after they have been suspended for more than a year to Determine the fall out from deleting user accounts after they have been suspended for more than a year

changed the description

Modified in sprint planning to become a spike investigating the practical consequences.

changed milestone to %DevOps Sprint 107

changed milestone to %DevOps Sprint 108

changed milestone to %DevOps Sprint 109

changed milestone to %DevOps Sprint 110

changed milestone to %DevOps Sprint 111

changed milestone to %DevOps Sprint 112

changed milestone to %DevOps Sprint 113

changed milestone to %DevOps Sprint 114

Google Admin API user resource has suspended flag and suspensionReason but no suspension date.

We may need to get this from somewhere else to determine who to delete. Not sure lookup has a cancelled timestamp and you'd need an authenticated connection to see cancelled accounts.

Maybe if we have suspended with lastLoginTime being unset (they can have no data to delete if they've never signed in) or > 2 years ago?

This may be useful: https://gitlab.developers.cam.ac.uk/uis/sysdev/devgroup/ibis/ibis/-/issues/11

doh! you've just done that the other way around!

changed milestone to %DevOps Sprint 115

assigned to @rjg21

changed milestone to %DevOps Sprint 116

added workflowIn Progress label and removed workflowSprint Ready label

Who do we delete?

This may be the more fundamental initial question; Is it currently possible to determine who to delete from Google Workspace?

There's possibly an easy case of, they are suspended and have never logged in or at least not in the last 2 years.

The Google Workspace user resource only has a suspended flag and suspensionReason so it is not possible to tell when a user was suspended from this. It does, however have a deletionTime for seeing how long it was deleted - not sure that's useful.

Lookup's API Person resource has a cancelled flag which maps to Jackdaw's DELETED == 'Y' (again no date).

Using transactions with Lookup's API you can get people who had changes between two transactions but the data in the response is the latest for the person. So the person being cancelled doesn't indicate when (or even if) they were cancelled during that period.

Also, the current sync tool implementation uses Lookup's LDAP personality which doesn't return a cancelled attribute, nor any cancelled people.

Jackdaw has a Cancelled timestamp (at least in UI).

So determining who to delete may be blocked by https://gitlab.developers.cam.ac.uk/uis/sysdev/devgroup/ibis/ibis/-/issues/11.

How do we delete them?

Admin SDK - Directory API has a users.delete endpoint that can take a users email or uid.

from googleapiclient import discovery
directory_service = discovery.build('admin', 'directory_v1', credentials=creds)
directory_service.users().delete(userKey=google_id)

What happens to their documents, calendars, etc?

Personal Drive and Calendar

These are actually deleted 20 days after the account is deleted.

Shared Drives and Calendars

All files in a shared drive are owned collectively, meaning if one person leaves, other shared drive members can still access the files. This prevents files from being deleted when their owner’s account is deleted.

Shared calendars aren't deleted, though may be problematic if the deleted user is the owner and no one else has been given "Manage Sharing" permissions.

Suspended accounts

Everything is kept for suspended accounts and calendar events are still visible to their shared attendees.

How do we do recovery?

Account restoration

According to this answer, you can’t restore a user if:

• it’s been longer than 20 days since you deleted the user account
• the username has been given to someone else as a primary address or email alias — or if it’s used for a group

Admin SDK - Directory API's users.list endpoint has a showDeleted query parameter that if set to "true" returns the users deleted in the last 5 days (reference).

Admin SDK - Directory API's users.undelete can restore an account (given uid not email).

There appears to be discrepancy between the 5 and 20 days, with some suggestion that the 5 got extended to 20 but docs are out of date.

Are we expecting to automate the undeleting of an account within 20 days of being deleted, a year after being suspended? It seems that these would be rare cases.

It is unknown what happens if you try to create a new account for an account that has recently been deleted. It may prevent account creation, or just create a new one with same attributes. Again, I guess these are going to be very infrequent.

Document recovery

Manual recovery, within 20 days of deletion, requires restoring the account first, transferring ownership of the files, then deleting the account again. There appears to be no way to change ownership of documents without having to undelete first.

The Admin Console has a transfer all files of one user to another function.

A Data Transfer API exists and may be able to create transfers of user data.

Conclusion

What are we trying to achieve? Why are we deleting the users?

I've investigated document retention and recovery but are we are deleting these accounts, not just to get licences back, but to remove data for GDPR reasons?

If so then we should not care about their data - we want it gone. So as long as we are deleting the correct accounts there should be no need for recovery or transfer.

In which case, we need to:

1. expose a cancelled date in the Lookup API
2. change the sync to use the API instead of LDAP
3. build a list of accounts to delete from:
   • those suspended
   • and a cancelled date > 2 years
   • (and not logged in within 2 years - as a safety?)
4. delete accounts through Google API

If an account is recreated within 20 days of it being deleted and the user.insert fails then we can manually undelete them in the Console and the sync will no longer attempt to create them.

Thanks, @rjg21. Happy for this issue to be closed as being done at your convenience but I'll leave it open for a short while to let others comment.

It seems to me that "What are we trying to achieve? Why are we deleting the users?" is the nub of the issue. From a technical reason we'd only want to delete to a) get seats back and b) free up storage. (I'll leave aside the legal and political reasons for now.)

Perhaps as a start we could get some numbers on a) how much storage our suspended users have and b) how many suspended users have either never signed in or have no data. As a quick win we could delete suspended users with no data or who have never signed in with little effect since they will be re-created if they ever come back.

Downloaded from admin console, will analyse and get stats tomorrow.

G Suite is moving to Google Workspace and they are making it very restrictive in the free license, so I wouldn't be surprised if they start saying no to our increase number of seats. They are also going to start restricting the amount of space available in the whole instance (at this moment they are saying 100TB, but waiting to hear from them)

We are currently approaching 1PB!

You can see now why that is a problem...

See https://support.google.com/a/answer/10403871

Institutions with greater than 20,000 students, faculty, and staff or 20,000 monthly active users that follow our Terms of Service and Acceptable Use Policy will be provided with additional storage. This process will roll out by the end of 2021.

Always going to be over that figure.

From last run:

Total Google users: 96007
Suspended Google users: 32805

Oddly this 32805 doesn't exactly match what I've downloaded from the console in two separate places...

This took way to long pulling different sources of google data together, and I believe storage is actually MiB when they say MB.

Suspended (non-admin) users in 'cam.ac.uk' only:

31558 of 32786 are using no storage (96.25 %)
Remaining 1228 are using 75.74 TiB in total

25 of 32786 have logged in within last 20 days (0.08 %)
1055 of 32786 have logged in over 20 days ago but within last 6 months (3.22 %)
2064 of 32786 have logged in over 6 months ago but within last year (6.30 %)
1853 of 32786 have logged in over a year ago but within last 2 years (5.65 %)
2570 of 32786 have logged over 2 years ago (7.84 %)
25219 of 32786 have never logged in (76.92 %)

Deleting suspended users who haven't logged in or within the last 2 years would restore:
   Accounts = 27789
   Storage = 2.67 TiB

very useful @rjg21, it would allow us to do a big purge.

@rjw57 has stated he's happy to close this issue, I've added a follow-up issue to look at scripting the deletion of people who have been suspended but who have never logged in #29 (closed).

With #29 (closed) and #28 (closed) raised, I think this issue can be closed.

added workflowReview Required label and removed workflowIn Progress label

By chance, I just came across the fact that the suspension date may be retrievable from the Audit/Reports API:

https://developers.google.com/admin-sdk/reports/v1/appendix/activity/admin-user-settings#SUSPEND_USER

Had a go at trying to prod this API. Struggling to get authorised usage. Appears that Domain Wide Delegation is needed - sigh - but even turning that on and impersonating my super admin, still failed. Will spin off an issue to investigate for real.

#28 (closed)

changed milestone to %DevOps Sprint 117

mentioned in issue #29 (closed)

closed