dumping embeddings for closed-set experiments

[joocjun]

Hello!

I've been trying to dump the embeddings for the closed-set experiments. Unfortunately except for enwiki-0 corpus, which has a 0_valid file, other corpora seem to have a problem with the dumping process. The problem seems to occur while collating the 'is_valid' attribute in the datamodule part(dimension errors). If I'm not mistaken this particular line seems to be the source of trouble.

NPM/dpr_scale/datamodule/utils.py

Line 138 in d284f40

is_valid = [i for i, _id in enumerate(input_ids) if _id not in [0, 2]]

The non-uniform lengths generated for each instance seem to be causing a dimension error.

At your earliest convenience, could check whether this is a valid error to be fixed??

Thank you