Retokenization drift

[phisiart]

(non-student here)

@dataclass
class VLLMCompletion:
    text: str
    token_ids: list[int]
    finish_reason: str | None

During my RL training runs, I observed that sometimes VLLMCompletion.token_ids are different from the result of my tokenize_prompt_and_output() which uses the HuggingFace tokenizer. This is especially obvious when VLLMCompletion.token_ids has a length of 512 (hitting the max) but the re-tokenized labels is longer.

This seems to be retokenization drift.

Am I thinking in the right direction? If so, I think that in future iterations of this course, it should be required that we directly use VLLMCompletion.token_ids and avoid retokenization.