Sync Tool

Tool for performing all Education Space Scheduling and Modelling (ESSM) synchronisation operations.

Uses TermTime and Booker APIs to sync rooms, buildings, bookings, etc between these systems.

Configuration is provided by one or more configuration files. This allows separate of services (say Booker production and staging configuration) and allows secrets to be held separately from non-secret configuration. These config files are deep merged together to create the initial state.

Example non-secret and secret files show required entries.

Installation

The command-line tool can be installed directly from the git repository:

$ pip3 install git+https://gitlab.developers.cam.ac.uk/uis/devops/essm/sync-tool.git

For developers, the script can be installed from a cloned repo using pip:

$ cd /path/to/this/repo
$ pip3 install -e .

Usage

The tool can be invoked from the command line:

$ essmsync

By default this will log what will be done. To actually perform the sync:

$ essmsync --really-do-this

To specify configuration files to be loaded the use the --configuration or -c arguments either multiple times or with a comma separated list of files (or a combination of both).

# multiple `-c` arguments
$ essmsync -c termtime-test.yaml -c booker-staging.yaml
# or comma separated
$ essmsync -c termtime-test.yaml,booker-staging.yaml

If not configuration files are specified then the tool will attempt to load a configuration.yaml and secrets.yaml.

A list of operations (see below) can be specified at the end of the command line.

See the output of essmsync --help for more information.

Docker Usage

Running sync-tool directly from repository:

$ docker run --rm -ti -v $(pwd):/usr/src/app \
    registry.gitlab.developers.cam.ac.uk/uis/devops/essm/sync-tool/master \
    -c termtime-test.yaml -c booker-staging.yaml \
    test(inside,docker)

Alternatively, an image can be build locally:

$ cd /path/to/repo
$ docker build -t essmsync .
# Assuming configurations files in another location
$ cd /path/to/configuration
$ docker run --rm -ti -v $(pwd):/usr/src/app essmsync \
    -c termtime-test.yaml -c booker-staging.yaml \
    test(local,docker)

Tests can then be run by overriding the entrypoint with tox:

$ docker run --rm -ti --entrypoint=tox essmsync

Operations

List of operations can be found in operations.md

The operations for the sync tool to perform can be specified either in the operations list in the configuration root or appended to the command line. If both, then the command line options are appended to the configuration list.

$ cat configuration.yaml
...
operations:
  - test(do,first)
...
$ essmsync --quiet -c configuration.yaml "test(do,next)" test "test(and,last)"
Test called with ('do', 'first')
Test called with ('do', 'next')
Test called without arguments
Test called with ('and', 'last')

Operations are categorised in to 'modules' based on the service being used, or core for non-service specific operations and comparison for comparison operations. The module to use can be specified before the operation, separated by a colon or dot, e.g. module:operation or module.operation.

If no module is specified then all modules are searched for the operation. If a single module has the specified operation then this module is assumed. If more than one module has the specified operation then a error is raised - the operation must be prefixed with the module, as above.

Some operations take arguments that are specified within parenthesis. e.g. operation(arg1,arg2) or module:operation(arg1,arg2)

Note that when using operations with arguments you may need to enclose the command line argument within quotes to avoid conflicts with your shell's parser.

Dumping results

Results from API queries (or any part of the state) can be dumped out in either yaml or json using the dump_yaml (shorthand dump) or dump_json operations. Both these take the 'key' in the state to dump and an optional flag as whether to include the parents of the key in the output. dump_json also optionally can be instructed to indent the output.

# Examples use the termtime get_rooms operation then dump the result

# dump yaml including parent keys (the default)
$ essmsync --quiet "termtime:get_rooms" "dump(termtime.rooms)"
$ essmsync --quiet "termtime:get_rooms" "dump_yaml(termtime.rooms)"

# dump yaml not including parent keys
$ essmsync --quiet "termtime:get_rooms" "dump(termtime.rooms,false)"
$ essmsync --quiet "termtime:get_rooms" "dump_yaml(termtime.rooms,false)"

# dump json including parent keys
$ essmsync --quiet "termtime:get_rooms" "dump_json(termtime.rooms,true)"

# dump json not including parent keys (the default)
$ essmsync --quiet "termtime:get_rooms" "dump_json(termtime.rooms)"

# dump json not including parent keys with indent of 4 spaces
$ essmsync --quiet "termtime:get_rooms" "dump_json(termtime.rooms,false,4)"

The first example, produces yaml that could be included as a configuration file in another call to essmsync.