Hello,
Thank you for your excellent research and code.
Inspired by your code, I tried to run the NoLiMa test by loading a local file instead of using a server.
When serving the same model with vllm and loading it locally, I expected the same result if the generation config is the same. However, the same result was not achieved. (The performance was slightly reduced.) The code is as follows. Did I miss something?
load part
local_model = AutoModelForCausalLM.from_pretrained(MODEL_PATH, trust_remote_code=True, torch_dtype=torch.bfloat16,)
local_tokenizer = AutoTokenizer.from_pretrained(MODEL_PATH, use_fast=True)
generation part
def format_messages(messages):
return self.tokenizer.apply_chat_template(
messages,
tokenize=False,
add_generation_prompt=True
)
async def generate_content(messages, **kwargs):
prompt = format_messages(messages)
inputs = self.tokenizer(prompt, return_tensors='pt')
output_result = self.api.generate(**inputs,max_new_tokens=max_tokens,temperature=temperature,top_p=top_p,do_sample=False,repetition_penalty=1.0,pad_token_id=self.tokenizer.eos_token_id)
generated_text = self.tokenizer.decode(output_result[0][inputs['input_ids'].shape[1]:],skip_special_tokens=True,clean_up_tokenization_spaces=False)
prompt_tokens = len(inputs['input_ids'][0])
completion_tokens = len(self.encode(generated_text))
total_tokens = prompt_tokens + completion_tokens
usage = {
"prompt_tokens": prompt_tokens,
"completion_tokens": completion_tokens,
"total_tokens": total_tokens
}
return {
"choices": [{
"message": {
"role": "assistant",
"content": generated_text
},
}],
"usage": usage
}
completion = await generate_content(messages)
Hello,
Thank you for your excellent research and code.
Inspired by your code, I tried to run the NoLiMa test by loading a local file instead of using a server.
When serving the same model with vllm and loading it locally, I expected the same result if the generation config is the same. However, the same result was not achieved. (The performance was slightly reduced.) The code is as follows. Did I miss something?
load part
generation part