Cannot decode > context length tokens due to RoPE max seq len?

[yuzu-ido]

Hi team! Thank you so much for publishing the course assignments and offering low resource options in the handout. I'm an alum but this course didn't exist back in my day :) I learned so much and had fun following along.

Q: when I try to decode more than context_len (== 256) tokens, it crashes in the RoPE step due to max_seq_len. The 7.2 "generate" problem says the deliverable is at least 256 tokens or <|endoftext|>. Is this expected or did I misunderstand RoPE? Is max_seq_len != context_len?

For context - I achieved 1.65 val loss with 5000 steps and pretty coherent TinyStories output. I think due to memory constraints (M4 Mac 16GB), my implementation took ~2s / step instead of ~0.43s / step (36 min / 5000 steps) as suggested in the handout for M4 Max 36GB.
Decoding 200 tokens looks like this:
Prompt: "hi my name is Sally and"
Response: ' I like to play with you. Can I play with you?"\nSally smiled and said, "Yes, Tim! Let's play together!" They played all day, and Tim learned that it's good to share and be kind to others.\n'

Thanks so much again!