Skip to content

RuntimeError: The size of tensor a (1024) must match the size of tensor b (1031) at non-singleton dimension 3 in generate.py #3

Open
Open
RuntimeError: The size of tensor a (1024) must match the size of tensor b (1031) at non-singleton dimension 3 in generate.py#3

Description

@RadiantCrystal
RadiantCrystal
opened on Sep 12, 2024

Hi Authors,

I am constantly getting this issue while running generate.py and while making efficient = True. Also I am seeing the generation time is very very high. Could you help me to resolve this issue?

Exception in thread Thread-932 (do_normal_task):
Traceback (most recent call last):
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/threading.py", line 1016](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/threading.py#line=1015), in _bootstrap_inner
    self.run()
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
    _threading_Thread_run(self)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/threading.py", line 953](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/threading.py#line=952), in run
    self._target(*self._args, **self._kwargs)
  File "/home/rima/RAD/utils/logits_processor.py", line 67, in do_normal_task
    reward_scores_part, self._past_key_values[gpu_id], self._attention_mask[gpu_id] = self.get_reward(
  File "[/home/rima/RAD/utils/logits_processor.py", line 213](http://127.0.0.1:8976/lab/tree/RAD/RAD/utils/logits_processor.py#line=212), in get_reward
    reward_scores, past_key_values = self.helper(inputs.input_ids, attention_mask, position_ids, past_key_values, gpu)
  File "/home/rima/RAD/utils/logits_processor.py", line 219, in helper
    _, reward_logits, past_key_values = reward_model(input_ids=input_ids,
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "[/tmp/ipykernel_1151105/2675989202.py", line 34](http://127.0.0.1:8976/tmp/ipykernel_1151105/2675989202.py#line=33), in forward
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1526), in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward
    transformer_outputs = self.transformer(
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 888](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=887), in forward
    outputs = block(
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1526), in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 390, in forward
    attn_outputs = self.attn(
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 331](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=330), in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 202, in _attn
    attn_weights = torch.where(causal_mask, attn_weights.to(attn_weights.dtype), mask_value)
RuntimeError: The size of tensor a (1024) must match the size of tensor b (1031) at non-singleton dimension 3
Exception in thread Thread-930 (do_normal_task):
Traceback (most recent call last):
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/threading.py", line 1016](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/threading.py#line=1015), in _bootstrap_inner
    self.run()
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
    _threading_Thread_run(self)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/threading.py", line 953](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/threading.py#line=952), in run
    self._target(*self._args, **self._kwargs)
  File "/home/rima/RAD/utils/logits_processor.py", line 67, in do_normal_task
    reward_scores_part, self._past_key_values[gpu_id], self._attention_mask[gpu_id] = self.get_reward(
  File "[/home/rima/RAD/utils/logits_processor.py", line 213](http://127.0.0.1:8976/lab/tree/RAD/RAD/utils/logits_processor.py#line=212), in get_reward
    reward_scores, past_key_values = self.helper(inputs.input_ids, attention_mask, position_ids, past_key_values, gpu)
  File "/home/rima/RAD/utils/logits_processor.py", line 219, in helper
    _, reward_logits, past_key_values = reward_model(input_ids=input_ids,
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "[/tmp/ipykernel_1151105/2675989202.py", line 34](http://127.0.0.1:8976/tmp/ipykernel_1151105/2675989202.py#line=33), in forward
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1526), in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward
    transformer_outputs = self.transformer(
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 888](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=887), in forward
    outputs = block(
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1526), in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 390, in forward
    attn_outputs = self.attn(
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 331](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=330), in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 202, in _attn
    attn_weights = torch.where(causal_mask, attn_weights.to(attn_weights.dtype), mask_value)
RuntimeError: The size of tensor a (1024) must match the size of tensor b (1031) at non-singleton dimension 3
Exception in thread Thread-933 (do_normal_task):
Traceback (most recent call last):
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/threading.py", line 1016](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/threading.py#line=1015), in _bootstrap_inner
Exception in thread Thread-931 (do_normal_task):
Traceback (most recent call last):
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/threading.py", line 1016, in _bootstrap_inner
    self.run()
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 766](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/ipykernel/ipkernel.py#line=765), in run_closure
    self.run()
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
    _threading_Thread_run(self)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/threading.py", line 953](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/threading.py#line=952), in run
    _threading_Thread_run(self)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/threading.py", line 953, in run
    self._target(*self._args, **self._kwargs)
  File "[/home/rima/RAD/utils/logits_processor.py", line 67](http://127.0.0.1:8976/lab/tree/RAD/RAD/utils/logits_processor.py#line=66), in do_normal_task
    self._target(*self._args, **self._kwargs)
  File "/home/rima/RAD/utils/logits_processor.py", line 67, in do_normal_task
    reward_scores_part, self._past_key_values[gpu_id], self._attention_mask[gpu_id] = self.get_reward(
  File "[/home/rima/RAD/utils/logits_processor.py", line 213](http://127.0.0.1:8976/lab/tree/RAD/RAD/utils/logits_processor.py#line=212), in get_reward
    reward_scores_part, self._past_key_values[gpu_id], self._attention_mask[gpu_id] = self.get_reward(
  File "/home/rima/RAD/utils/logits_processor.py", line 213, in get_reward
    reward_scores, past_key_values = self.helper(inputs.input_ids, attention_mask, position_ids, past_key_values, gpu)
  File "[/home/rima/RAD/utils/logits_processor.py", line 219](http://127.0.0.1:8976/lab/tree/RAD/RAD/utils/logits_processor.py#line=218), in helper
    reward_scores, past_key_values = self.helper(inputs.input_ids, attention_mask, position_ids, past_key_values, gpu)
  File "/home/rima/RAD/utils/logits_processor.py", line 219, in helper
    _, reward_logits, past_key_values = reward_model(input_ids=input_ids,
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    _, reward_logits, past_key_values = reward_model(input_ids=input_ids,
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1526), in _call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "[/tmp/ipykernel_1151105/2675989202.py", line 34](http://127.0.0.1:8976/tmp/ipykernel_1151105/2675989202.py#line=33), in forward
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1526), in _call_impl
    return forward_call(*args, **kwargs)
  File "/tmp/ipykernel_1151105/2675989202.py", line 34, in forward
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return forward_call(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=1073), in forward
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    transformer_outputs = self.transformer(
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    return forward_call(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 1074, in forward
    return self._call_impl(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1526), in _call_impl
    transformer_outputs = self.transformer(
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return forward_call(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 888](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=887), in forward
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    outputs = block(
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    return forward_call(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 888, in forward
    return self._call_impl(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1526), in _call_impl
    outputs = block(
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    return forward_call(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 390](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=389), in forward
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return forward_call(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 390](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=389), in forward
    attn_outputs = self.attn(
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518, in _wrapped_call_impl
    attn_outputs = self.attn(
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1518](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1517), in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527, in _call_impl
    return self._call_impl(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1527](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/torch/nn/modules/module.py#line=1526), in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 331, in forward
    return forward_call(*args, **kwargs)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 331](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=330), in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 202, in _attn
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "[/home/rima/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py", line 202](http://127.0.0.1:8976/lab/tree/RAD/anaconda3/envs/llava/lib/python3.10/site-packages/transformers/models/gpt2/modeling_gpt2.py#line=201), in _attn
    attn_weights = torch.where(causal_mask, attn_weights.to(attn_weights.dtype), mask_value)
RuntimeError: The size of tensor a (1024) must match the size of tensor b (1031) at non-singleton dimension 3
    attn_weights = torch.where(causal_mask, attn_weights.to(attn_weights.dtype), mask_value)
RuntimeError: The size of tensor a (1024) must match the size of tensor b (1031) at non-singleton dimension 3

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions