Request for Performance Results with LLaMA-3.1-8B

[EEcol1]

I'm trying to reproduce the baseline performance. It appears that the baseline results reported in the paper use the llama-3.1-70B model for the human agent. However, due to limited computing resources, I'm unable to run the llama-3.1-70B model. Instead, I would like to know the performance results when both the human and robot agents use the llama-3.1-8B model as the planner.

If possible, could you share the results for this configuration?

[Facerman-cloud]

Hello, may I ask if you can run the 8B model? What is the model of the device you are using?

[EEcol1]

As the number of episodes increased, a memory overflow occurred, so I created a shell script(.sh) to run the episodes in batches of five.

#!/bin/bash

for start in {0..999..5}
do
  end=$((start + 4))

  if [ $end -ge 1000 ]; then
    end=999
  fi

  episodes="["
  for ((i=start; i<=end; i++)); do
    if [ $i -ne $start ]; then
      episodes+=","
    fi
    episodes+="$i"
  done
  episodes+="]"

  echo "Running episodes $episodes..."
  python -m habitat_llm.examples.planner_demo --config-name baselines/decentralized_zero_shot_react_summary.yaml \
    habitat.dataset.data_path="data/datasets/partnr_episodes/v0_0/val.json.gz" \
    evaluation.agents.agent_0.planner.plan_config.llm.inference_mode=hf \
    evaluation.agents.agent_1.planner.plan_config.llm.inference_mode=hf \
    evaluation.agents.agent_0.planner.plan_config.llm.generation_params.engine=meta-llama/Meta-Llama-3-8B-Instruct \
    evaluation.agents.agent_1.planner.plan_config.llm.generation_params.engine=meta-llama/Meta-Llama-3-8B-Instruct \
    +episode_indices="$episodes"
done

My computer specifications are as follows:

OS: Ubuntu 20.04
CPU: Intel i7-12700KF
GPU: NVIDIA RTX A6000
RAM: 64GB

[Hobeom]

The performance results for your configuration, where ReAct-8B agents use LLaMA-3.1-8B in a decentralized setting, are already reported in the paper.