GitLab

Configure GitLab CI

Closes #37

See merge request !41

project where the test suite isn't containerised so the config simply
overrides the Auto-DevOps test job and runs tox.

Add clause to export_sms_csv.sh that excludes corrupted clips

Add a property to the collection listing all the media ids for the
collection in the order they should be in. This custom property is not
modified by adding/removing videos to the corresponding JWP playlist
unlike sms_{,failed_}media_ids and is used by the media platform to
reconcile videos with playlists.

in a table named corrupted_clips.

use media id as primary key when determining create/update

We previously used the clip id as a primary key for determining if a
video needed to be updated or created. Unfortunately this meant that our
previous change which modified our clip id selection resulted in
duplicate videos being uploaded with the same media id and differing
clip id.

Compounding this error was the fact that these videos had identical
publication dates set which meant that, in practice, they came back from
the JWP API in a semi-random order depending on the vagaries of their
database.

Move to using media id as the primary key which should stop this
happening in future. We still need to tidy up the videos in JWP. The
best approach would be to simply delete all the videos with multiple
clip ids.

Empirically, we should always use MPEG4 is available since it is often
the highest resolution. Tweak the order of prferred formats/qualities to
reflect this.

widen range of acceptable export formats

Widen the range of formats we try to upload to JWP. Some media items had
neither an audio or video archive export.

Modify the CSV dump script to dump *all* clips and define a preferred
ordering for formats, video then audio and within each decreasing
quality options. Explicitly log items with no media at all (there are
some!) or ones where we could not determine a "best" format.

IMPORTANT: This changes the default strip-leading path option from 1 to
0 to allow for non-archive media items to be downloaded. This needs to
be combined with a change to gen_jwplayer_feed.sh to strip the
"sms-archive02/" from the base URL.

support empty collections

Some collections may be empty, in which case the media ids prop will be
blank. In this case we should parse it as the empty list but we don't
since ''.split(',') returns ['']. Catch this case.

Closes #32

Modify the SMS export to use the associated collection ACL for a media
item if it does not otherwise have an ACL. This should fix
uisautomation/sms-webapp#218.

genupdatejob: make sanitize a little more forgiving

Initially we had to truncate video descriptions because the JWP client
libraries would try to pass all parameters as GET requests, even when
using HTTP POST. That has been fixed by later versions of the JWP
client. Additionally, we want to allow for newlines in descriptions
since they are usually important.

Update the sanitisiation function to only strip non-newline control
characters and to no-longer truncate strings.

BUGFIX: Wrong media type imported

Synchronisation of SMS collections to JW channels

genupdatejob: check failed_media_ids so as not to re-run failed jobs

/channels/videos/create (and delete) operation.