Issues with notebooks 02a_... and 02b_...

[expectopatronum]

Hi,
I am trying now for a couple of days to run the notebook 02a_prepareData.ipynb (I converted it to a Python script, but the code is the same). I have a few issues/questions:

• when running the script I run out of memory (my machine has 32GB). is that expected? how much memory does the machine have that you were using?
• This is the output printed at the beginning, do the numbers look right?
• 02a_... and 02b_... look very similar, are they doing different things? Or more precise: is 02a doing something that is not done by 02b? it seems like a few things are created in both notebooks.

I am looking forward to your answers!
Best regards
Verena

[expectopatronum]

Hi,
after cleaning up the code a bit I noticed another thing that confuses me: In this notebook chunkSize is set to 64, but later in many places 256 is used (e.g. save_chunks_file = '../score_feat/data.chunks256.pkl', csv_train_file = path/'train256.csv'`, ...). Does this number mean something different or should it be a variable?
Sorry for the questions but it is not clear what is happening and I am worried that I'll overwrite half of the data when rerunning the script with a different nr of chunks.

Thanks and best regards
Verena