Gracefully fail in the face of network blips
When in front of a real user, the IAR backend returned a 500 response. The traceback is below.
The ultimate cause appears to be a network blip where we couldn't react the token validation endpoint. This flow needs to be reworked.
- Investigate if requests supports an exponential-backoff-type retry strategy.
- Provide a production-ready un-authenticated token introspection endpoint like the Google tokeninfo endpoint which can be more resliant.
{
insertId: "q3gxsxg42ipaq2"
labels: {
compute.googleapis.com/resource_name: "fluentd-gcp-v2.0.10-lfnfq"
container.googleapis.com/namespace_name: "iar"
container.googleapis.com/pod_name: "iar-66b7ddb7c9-pthdg"
container.googleapis.com/stream: "stderr"
}
logName: "projects/information-asset-register/logs/iar-backend-app"
receiveTimestamp: "2018-02-28T10:12:08.097789557Z"
resource: {
labels: {
cluster_name: "iar"
container_name: "iar-backend-app"
instance_id: "915970991488558301"
namespace_id: "iar"
pod_id: "iar-66b7ddb7c9-pthdg"
project_id: "information-asset-register"
zone: "europe-west2-a"
}
type: "container"
}
severity: "ERROR"
textPayload: "Traceback (most recent call last):
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
response = get_response(request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
response = self.process_exception_by_middleware(e, request)
File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
response = wrapped_callback(request, *callback_args, **callback_kwargs)
File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
return view_func(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/rest_framework/viewsets.py", line 95, in view
return self.dispatch(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 494, in dispatch
response = self.handle_exception(exc)
File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 454, in handle_exception
self.raise_uncaught_exception(exc)
File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 482, in dispatch
self.initial(request, *args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 399, in initial
self.perform_authentication(request)
File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 325, in perform_authentication
request.user
File "/usr/local/lib/python3.6/site-packages/rest_framework/request.py", line 222, in user
self._authenticate()
File "/usr/local/lib/python3.6/site-packages/rest_framework/request.py", line 374, in _authenticate
user_auth_tuple = authenticator.authenticate(self)
File "/usr/src/app/assets/authentication.py", line 78, in authenticate
token = self.validate_token(bearer)
File "/usr/src/app/assets/authentication.py", line 133, in validate_token
timeout=2, data={'token': token})
File "/usr/src/app/assets/authentication.py", line 46, in _request
return _request.__session.request(*args, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests_oauthlib/oauth2_session.py", line 360, in request
headers=headers, data=data, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 490, in send
raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError("(104, 'ECONNRESET')",))
"
timestamp: "2018-02-28T10:12:07Z"
}