Cannot decode > context length tokens due to RoPE max seq len?

[yuzu-ido]

Hi team! Thank you so much for publishing the course assignments and offering low resource options in the handout. I'm an alum but this course didn't exist back in my day :) I learned so much and had fun following along.

Q: when I try to decode more than context_len (== 256) tokens, it crashes in the RoPE step due to max_seq_len. The 7.2 "generate" problem says the deliverable is at least 256 tokens or <|endoftext|>. Is this expected or did I misunderstand RoPE? Is max_seq_len != context_len?

For context - I achieved 1.65 val loss with 5000 steps and pretty coherent TinyStories output. I think due to memory constraints (M4 Mac 16GB), my implementation took ~2s / step instead of ~0.43s / step (36 min / 5000 steps) as suggested in the handout for M4 Max 36GB.
Decoding 200 tokens looks like this:
Prompt: "hi my name is Sally and"
Response: ' I like to play with you. Can I play with you?"\nSally smiled and said, "Yes, Tim! Let's play together!" They played all day, and Tim learned that it's good to share and be kind to others.\n'

Thanks so much again!

[marcelroed]

Hi! max_seq_len is indeed the maximum length you expect your model to be able to generate. Since RoPE needs to generate a cache of positional encodings, we limit this up-front, but it can be as long as you want to ensure you don't get limited during generation. Keep in mind you can change this after training, since the weights/generation quality won't be affected by this parameter.

[yuzu-ido]

you can change this after training, since the weights/generation quality won't be affected by this parameter.
Makes sense, super helpful! Thank you.