FAQ | This is a LIVE service | Changelog

Gracefully fail in the face of network blips

When in front of a real user, the IAR backend returned a 500 response. The traceback is below.

The ultimate cause appears to be a network blip where we couldn't react the token validation endpoint. This flow needs to be reworked.

  1. Investigate if requests supports an exponential-backoff-type retry strategy.
  2. Provide a production-ready un-authenticated token introspection endpoint like the Google tokeninfo endpoint which can be more resliant.
 {
 insertId: "q3gxsxg42ipaq2"  
 labels: {
  compute.googleapis.com/resource_name: "fluentd-gcp-v2.0.10-lfnfq"   
  container.googleapis.com/namespace_name: "iar"   
  container.googleapis.com/pod_name: "iar-66b7ddb7c9-pthdg"   
  container.googleapis.com/stream: "stderr"   
 }
 logName: "projects/information-asset-register/logs/iar-backend-app"  
 receiveTimestamp: "2018-02-28T10:12:08.097789557Z"  
 resource: {
  labels: {
   cluster_name: "iar"    
   container_name: "iar-backend-app"    
   instance_id: "915970991488558301"    
   namespace_id: "iar"    
   pod_id: "iar-66b7ddb7c9-pthdg"    
   project_id: "information-asset-register"    
   zone: "europe-west2-a"    
  }
  type: "container"   
 }
 severity: "ERROR"  
 textPayload: "Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/exception.py", line 35, in inner
    response = get_response(request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 128, in _get_response
    response = self.process_exception_by_middleware(e, request)
  File "/usr/local/lib/python3.6/site-packages/django/core/handlers/base.py", line 126, in _get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/local/lib/python3.6/site-packages/django/views/decorators/csrf.py", line 54, in wrapped_view
    return view_func(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/viewsets.py", line 95, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 494, in dispatch
    response = self.handle_exception(exc)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 454, in handle_exception
    self.raise_uncaught_exception(exc)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 482, in dispatch
    self.initial(request, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 399, in initial
    self.perform_authentication(request)
  File "/usr/local/lib/python3.6/site-packages/rest_framework/views.py", line 325, in perform_authentication
    request.user
  File "/usr/local/lib/python3.6/site-packages/rest_framework/request.py", line 222, in user
    self._authenticate()
  File "/usr/local/lib/python3.6/site-packages/rest_framework/request.py", line 374, in _authenticate
    user_auth_tuple = authenticator.authenticate(self)
  File "/usr/src/app/assets/authentication.py", line 78, in authenticate
    token = self.validate_token(bearer)
  File "/usr/src/app/assets/authentication.py", line 133, in validate_token
    timeout=2, data={'token': token})
  File "/usr/src/app/assets/authentication.py", line 46, in _request
    return _request.__session.request(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests_oauthlib/oauth2_session.py", line 360, in request
    headers=headers, data=data, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/requests/adapters.py", line 490, in send
    raise ConnectionError(err, request=request)
requests.exceptions.ConnectionError: ('Connection aborted.', OSError("(104, 'ECONNRESET')",))
"  
 timestamp: "2018-02-28T10:12:07Z"  
}