We have been able to fix these issues because of our resumable queue. Restarting ob2 will re-enqueue any interrupted requests, which means we can just restart the autograder whenever somebody complains of not being able to access their repository.
I[2016-01-21 14:17:02,305][github3] Building a url from ('https://api.github.com', 'orgs', 'Berkeley-CS162')
I[2016-01-21 14:17:02,308][requests.packages.urllib3.connectionpool] Starting new HTTPS connection (1): api.github.com
I[2016-01-21 14:17:02,668][github3] Attempting to get JSON information from a Response with status code 200 expecting 200
I[2016-01-21 14:17:02,668][github3] JSON was returned
I[2016-01-21 14:17:02,668][github3] Building a url from ('https://api.github.com/orgs/Berkeley-CS162', 'repos')
I[2016-01-21 14:17:03,211][github3] Attempting to get JSON information from a Response with status code 201 expecting 201
I[2016-01-21 14:17:03,211][github3] JSON was returned
I[2016-01-21 14:17:03,212][github3] Building a url from ('https://api.github.com/repos/Berkeley-CS162/REDACTED', 'collaborators')
I[2016-01-21 14:17:03,212][github3] Missed the cache building the url
I[2016-01-21 14:17:03,307][github3] JSON was not returned
I[2016-01-21 14:17:03,307][github3] Building a url from ('https://api.github.com/repos/Berkeley-CS162/REDACTED', 'collaborators', 'REDACTED')
I[2016-01-21 14:17:03,308][github3] Missed the cache building the url
E[2016-01-21 14:17:03,420][root] [repomanager] Error occurred while processing queue
Traceback (most recent call last):
File "ob2/util/resumable_queue.py", line 107, in run
self.process_job(operation, payload)
File "ob2/repomanager/__init__.py", line 18, in process_job
_assign_repo(*payload)
File "ob2/util/github_api.py", line 90, in _assign_repo
repr(fq_repo_name))
AssertionError: Unable to add member u'REDACTED' to u'Berkeley-CS162/REDACTED'
I feel like this may be caused by eventual consistency semantics in GitHub's API. There should be a better way of fixing temporary errors like this, which doesn't require a manual restart of ob2.
This bug has occurred 2 times out of 160 successful
assign_reporequests. The assign_repo action is able to create the repository, but fails to add the student's GitHub account as a collaborator.Workaround
We have been able to fix these issues because of our resumable queue. Restarting ob2 will re-enqueue any interrupted requests, which means we can just restart the autograder whenever somebody complains of not being able to access their repository.
Details
I feel like this may be caused by eventual consistency semantics in GitHub's API. There should be a better way of fixing temporary errors like this, which doesn't require a manual restart of ob2.