Hello,
I found the repo and blog post very interesting and useful, especially the tensor serialization utility.
I've encountered similar problems with RAM usage, but in my case I have to use dictionaries to store data, or even dictionaries of list.
Can you confirm that the issues you presented may happen with dictionaries too? I also happen to see many "too many open files errors" due to this.
Secondly, how can I serialize dictionaries in a torch Tensor instead of using lists?
I could work with lists too, but it would be very complicated to retrieve the correct Tensor in the main processes then (I find dictionaries to be more flexible on this point of view).
Hello,
I found the repo and blog post very interesting and useful, especially the tensor serialization utility.
I've encountered similar problems with RAM usage, but in my case I have to use dictionaries to store data, or even dictionaries of list.
Can you confirm that the issues you presented may happen with dictionaries too? I also happen to see many "too many open files errors" due to this.
Secondly, how can I serialize dictionaries in a torch Tensor instead of using lists?
I could work with lists too, but it would be very complicated to retrieve the correct Tensor in the main processes then (I find dictionaries to be more flexible on this point of view).