Resource View & DataStore Fields for Loading Objects#229
Conversation
- Load resource views from ckanapi dump.
- Load resource datastore fields from ckanapi dump.
- Update the cli help text.
- Update the cli help text.
- Update the cli help text.
| r = ckan.call_action(thing_create, args, | ||
| requests_kwargs=requests_kwargs) | ||
| except ValidationError as e: | ||
| reply(act, 'ValidationError', e.error_dict, stdout) |
There was a problem hiding this comment.
In the validation error case I think we want to only catch the case where the table already exists and then skip applying the fields. We could return a value to include in the reply that indicates that datastore fields were not applied to some resources, but I don't think this should be treated as an error.
There was a problem hiding this comment.
cuz we are in a loop of the resources, how would we return which ones did not get created? just as a list of resource ids?
There was a problem hiding this comment.
Okay I kinda made something for the extra output for datastore_fields and resource_views. So it would display what was created, updated, and skipped.
There was a problem hiding this comment.
Okay I THINK I did it, so catch ValidationError. If there is no existing table, then raise the error and the worker process will scream for that line. If there is an existing table, I just append the resource id and string error to the "skipped" tables which outputs in the worker process log
- Update the cli help text.
- Update the cli help text.
- New arg.
- New arg.
- New arg.
|
@wardi okay I believe I have achieved everything here now |
| reply(act, 'NotFound', obj) | ||
| else: | ||
| reply(act, None, r.get('name',r.get('id'))) | ||
| log_obj = {} |
There was a problem hiding this comment.
How about we switch to always returning a dict with id and name values and optionally these other things, instead of sometimes returning a string and sometimes a dict, depending on options passed
| requests_kwargs = None | ||
| if arguments['--insecure']: | ||
| requests_kwargs = {'verify': False} |
There was a problem hiding this comment.
This repeated pattern suggests we should add requests_kwargs to RemoteCKAN.__init__ so that we don't have to pass them on every call_action
| # exceptions handled in load_things_worker | ||
| # raise normal exception for non-existing tables | ||
| raise e | ||
| skipped.append('%s: %s' % (rid, str(e))) |
There was a problem hiding this comment.
I'm pretty sure the str(e) happens automatically as part of the string formatting
| skipped.append('%s: %s' % (rid, str(e))) | |
| skipped.append('%s: %s' % (rid, e)) |
|
Tests can be added by creating mocks for the actions. We should at least cover the happy path and exceptions from each action to make sure they're handled. |
2 participants
Adds the ability to load the resource_views and datastore_fields from the CKANAPI dump options.
cc: @wardi