Errors with Llama models

[kjain1810]

Hi
I am trying to run the script in examples folder with Llama-2-7B and Llama-3.2-3B models but facing the following errors. I am running this on A100 GPUs

Llama-2-7B:

Traceback (most recent call last):
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/examples/offline_inference.py", line 59, in <module>
    llm_engine = LLMEngine.from_system_config(system_config)
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/engine/llm_engine.py", line 13, in from_system_config
    engine = PipelineParallelLLMEngine(config)
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/engine/pipeline_parallel_llm_engine.py", line 49, in __init__
    super().__init__(config)
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/engine/base_llm_engine.py", line 86, in __init__
    self._init_workers_ray()
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/engine/base_llm_engine.py", line 197, in _init_workers_ray
    self._run_workers(
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/engine/base_llm_engine.py", line 421, in _run_workers
    all_outputs = ray.get(all_outputs, timeout=0)
  File "/opt/conda/envs/ptca/lib/python3.10/site-packages/ray/_private/auto_init_hook.py", line 21, in auto_init_wrapper
    return fn(*args, **kwargs)
  File "/opt/conda/envs/ptca/lib/python3.10/site-packages/ray/_private/client_mode_hook.py", line 103, in wrapper
    return func(*args, **kwargs)
  File "/opt/conda/envs/ptca/lib/python3.10/site-packages/ray/_private/worker.py", line 2782, in get
    values, debugger_breakpoint = worker.get_objects(object_refs, timeout=timeout)
  File "/opt/conda/envs/ptca/lib/python3.10/site-packages/ray/_private/worker.py", line 929, in get_objects
    raise value.as_instanceof_cause()
ray.exceptions.RayTaskError(TypeError): ray::RayWorker.execute_method() (pid=69210, ip=10.0.0.5, actor_id=7b9c0a4ffe454e78af73b8cc01000000, repr=<sarathi.engine.ray_utils.RayWorker object at 0x152ebc4b82b0>)
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/engine/ray_utils.py", line 32, in execute_method
    return executor(*args, **kwargs)
  File "/opt/conda/envs/ptca/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/utils/threading_utils.py", line 17, in synced_method
    return method(self, *args, **kwargs)
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/worker/base_worker.py", line 134, in init_model
    self.model_runner = ModelRunner(
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/model_executor/model_runner.py", line 46, in __init__
    self.model = get_model(self.config.model_config)
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/model_executor/model_loader.py", line 48, in get_model
    model_class = _get_model_architecture(model_config.hf_config)
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/model_executor/model_loader.py", line 38, in _get_model_architecture
    for arch in architectures:
TypeError: 'NoneType' object is not iterable

Llama-3.2-3B:

Traceback (most recent call last):
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/examples/offline_inference.py", line 32, in <module>
    model_config = ModelConfig(
  File "<string>", line 11, in __init__
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/config/config.py", line 64, in __post_init__
    self.max_model_len = get_and_verify_max_len(self.hf_config, self.max_model_len)
  File "/mnt/azureml/cr/j/ad6dadea4df14bbaaf4656365fc84ac3/exe/wd/sarathi-serve/sarathi/utils/hf_utils.py", line 100, in get_and_verify_max_len
    if rope_scaling["type"] == "yarn":
KeyError: 'type'