Synchronise Lookup groups.

This causes Lookup groups and their members to be synchronised with groups in Google. As for people, there are various configuration options to control which groups to synchronise, and enforce limits on the number of changes the script will perform.

The group name and email in Google are based on the Lookup groupID and groupName, noting that the email is the group's unique identifier in Google:

email = {groupID}@{groups_domain} name = {groupName}@groups.lookup.cam.ac.uk

where groups_domain is a configurable secondary domain or domain alias for groups in Google.

The rationale for this choice is that it will allow for easier searching by Lookup groupID and groupName (assuming that searches operate from the start of these fields), while using a separate namespace for group identifiers to avoid possible conflicts with users and groups from other sources.

Note: The main sync.py script has become quite long, and could probably use refactoring, but keeping it as it is for now to make it easier to diff.

Closes #10 (closed)

gsuitesync/gapiutil.py

@@ -114,3 +114,53 @@ def list_all_in_list(directory_service, list_cb, *, item_ids=[], id_key='key', b
                 break
 
     return resources
+
+
+def get_all_in_list(directory_service, get_cb, *, item_ids=[], id_key='key', batch_size=1000,
+                    retries=2, retry_delay=5, **kwargs):
+    """
+    Wrapper for Google Client SDK get()-style callables, operating on a list of items. Invokes the
+    "get_cb" Google API method for each item in the "item_ids" list, returning a list resources.
+    The key used to identify the items in Google is specified by the "id_key" argument.
+
+    This is equivalent to calling "get_cb" for each item in the "item_ids" list, and collecting all
+    the results in a list, except that it uses the Google batch processing API to reduce the number
+    of API calls.
+
+    """
+    # Invoke get_cb for each item in the item_ids list, using the batch processing API. The
+    # Directory API supports a maximum batch size of 1000. See:
+    # https://developers.google.com/admin-sdk/directory/v1/guides/batch
+    resources = []
+
+    # Batch response handler
+    def handle_batch_response(request_id, response, exception):
+        if exception:
+            raise exception
+        resources.append(response)
+
+    for i in range(0, len(item_ids), batch_size):
+        batch_item_ids = item_ids[i:i+batch_size]
+
+        # Form the batch request
+        batch = directory_service.new_batch_http_request()
+        for item_id in batch_item_ids:
+            request = get_cb(**kwargs, **{id_key: item_id})
+            batch.add(request, callback=handle_batch_response)
+
+        # Process the batch request
+        while True:
+            try:
+                batch.execute()
+            except HttpError as err:
+                if (err.resp.status == 503 and retries > 0):
+                    retries -= 1
+                    LOG.warn('503: Service unavailable - retrying')
+                    sleep(retry_delay)
+                    continue
+                if retries == 0:
+                    LOG.error('503: Service unavailable - retry count exceeded')
+                raise
+            break
+
+    return resources