Cannot evaluate the model

[bajracharya-kshitij]

I tried running python eval.py as mentioned in README.md and ran into RuntimeError: Expected object of backend CUDA but got backend CPU for argument #3 'index. I managed to solve this by looking at the train.py and replacing

sentence, len_sentence, question = batch.src[0], batch.src[1], batch.trg[0]

with

sentence, len_sentence, question = batch.src[0].to(device), batch.src[1].to(device), batch.trg[0].to(device)

But now I have another problem. I debugged the code and found out that the evaluation loop runs for 4 iterations and it prints a lot of <unk>s but on the 5th iteration, it doesn't go beyond the line:

pred = model(sentence, len_sentence, answer=answer)

It is stuck forever in the loop. I'm new to this and I'm trying to learn how this works by looking at your code. Please help.

[MrSchnappi]

the same question that I have encountered as well. Have you solved this question?

[bajracharya-kshitij]

No, I haven't found anything yet. Please let me know if you do.

[jamontol]

Same issue as well. It gets stuck on 8th iteration with a lot of <unk> tokens.

Problem with iterations is in layers.py, line 113:

while out.item() in self.special_tokens_ids:
out = torch.multinomial(probs, num_samples=1)

when probs is a tensor with only one non-zero element it gets trapped on while loop because it always yields the same value

[francescovannoni]

i solved adding this in layers.py

        out, probs = sample_sequence(out, self.top_k, self.top_p, self.temperature, False)
        if t < self.min_len_sentence and out.item() in self.special_tokens_ids:
            count = 0
            probs_size = list(probs.size())
            probs_size = probs_size[0]
            for o in range (0, probs_size):
                if(probs[o] != 0):
                    count = count +1
            if(count == 1):
                return outputs
            while out.item() in self.special_tokens_ids:
                out = torch.multinomial(probs, num_samples=1)

i don't know if it is correct. The idea is to check if probs is a tensor with only one non-zero element and in that case to return output to avoid infinite loop

[bajracharya-kshitij]

@francescovannoni Hi, could you tell me where exactly you added this in layers.py?

[francescovannoni]

As you can see in the message below i added the part from (if t < self.min_len_sentence and out.item() in self.special_tokens_ids:) (line 111) to (while out.item() in self.special_tokens_ids:) line 113. In the middle i added the part which start with count = 0