issues registering custom dataset

[geldouche13]

We have tried to upload our own dataset but were faced with some problems like register_dataset not being recognized by our system. Is there a clear way to easily upload a custom dataset. Thank you so much

[nberth]

Hi,

Are you trying to implement a new plugin as documented here? In principle the register_dataset function is part of the deepconcolic.datasets module, so just adding from deepconcolic.datasets import register_dataset should do it. Otherwise, please could you provide more details about the error?

Best,

[geldouche13]

The error looks like this. we put the load_fridge_data in the utils folder

[nberth]

Oh I see. Actually there's no persistence in DeepConcolic w.r.t datasets: to add support for a custom dataset, the idea is to create a plugin script that is looked up by DeepConcolic's main scripts (e.g. deepconcolic.main, deepconcolic.eval_classifier, deepconcolic.fuzzer, etc) whenever they start. Such a plugin must then call deepconcolic.datasets.register_dataset with some name for the dataset, which is then available as argument to the command line flag `--dataset' for the aforementioned main scripts.

Plugin scripts are searched within any directory listed in the colon-separated environment variable DC_PLUGINS_PATH if this variable is defined (I have no idea how that works on windows, I guess there's something similar), or else within the ./dc_plugins directory if it exists (note the latter is relative to the current working directory).

Hope that clarifies the issue,

[geldouche13]

Hello again! Thank you so much for your detailed replies. We did as suggested but faced another problem. It is giving us an error and not enabling us to import the register_dataset function.

[nberth]

Hi again,

I think you are still using the tutorial branch, which integrates some "safety" features to ensure the command we provided for the tutorial were executed from within the DeepConcolic directory. To regain more freedom, you need to pull the master branch.

Best,

[geldouche13]

We kept paving down that path and it worked! you had a point, we were indeed working on the tutorial branch. Thank you so much for all your help. You just saved our lives