RuntimeError: CUDA error: unspecified launch failure

[Facerman-cloud]

Skipping evaluating episode.
Error executing job with overrides: ['habitat.dataset.data_path=data/datasets/partnr_episodes/v0_0/val_mini.json.gz', 'evaluation.agents.agent_0.planner.plan_config.llm.inference_mode=hf', 'evaluation.agents.agent_0.planner.plan_config.llm.generation_params.engine=meta-llama/Meta-Llama-3.1-8B-Instruct']
Traceback (most recent call last):
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/examples/planner_demo.py", line 186, in run_eval
run_planner(config, dataset)
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/examples/planner_demo.py", line 405, in run_planner
eval_runner.reset()
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/evaluation/evaluation_runner.py", line 196, in reset
self.reset_planners()
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/evaluation/decentralized_evaluation_runner.py", line 89, in reset_planners
planner.reset()
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/planner/llm_planner.py", line 106, in reset
agent.reset()
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/agent/agent.py", line 86, in reset
self.tools[tool_name].skill.reset([0]) # type: ignore
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/tools/motor_skills/nav/oracle_nav_skill.py", line 108, in reset
super().reset(batch_idxs)
File "/media/taiic/5E08BBF708BBCC71/jiawei/partnr-planner/habitat_llm/tools/motor_skills/skill.py", line 201, in reset
self.recurrent_hidden_states = torch.zeros(
RuntimeError: CUDA error: unspecified launch failure
CUDA kernel errors might be asynchronously reported at some other API call, so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1
Compile with TORCH_USE_CUDA_DSA to enable device-side assertions.
Set the environment variable HYDRA_FULL_ERROR=1 for a complete stack trace.

Please tell me that how to solve this promble

[xavierpuigf]

Hi, @Facerman-cloud . Could you check what is the value of self.device and self.env.device here:
https://github.com/facebookresearch/partnr-planner/blob/main/habitat_llm/tools/motor_skills/skill.py#L201

[Facerman-cloud]

Hi, @Facerman-cloud . Could you check what is the value of self.device and self.env.device here: https://github.com/facebookresearch/partnr-planner/blob/main/habitat_llm/tools/motor_skills/skill.py#L201
Teacher, thank you for your guidance. I will pay attention to the value of self. device and self. env. device. And could you please take a look at the #31 ?