All tests in `test_tokenizer.py` pass despite faulty tokenizer

[harrisonstropkay]

The encode_iterable method that students implement in the Tokenizer class accepts an iterable of strings. A straightforward but incorrect implementation is to iterate over the strings and yield from the output of encode for each string independently.

This approach can split pretokens across iterable boundaries. For example, consider the iterable:

["test\n", "\n", "string"]

This is similar to what you might get when iterating over a Python file pointer for a text file like:

test

string

If encode_iterable processes each string independently, the two newline characters will be tokenized separately rather than as the combined pretoken \n\n.

Currently, tests/test_tokenizer.py does not cover this edge case. I added a test for this scenario on this branch:

https://github.com/harrison-f-stropkay/assignment1-basics/tree/multi-newline-test-case

I'd open a PR, but I don't believe I have the necessary permissions.

[marcelroed]

Thanks! Yes, unfortunately we get a lot of PRs of people accidentally creating permanently accessible PRs that contain their full solutions, so I've had to limit PRs to contributors only.

I will check and merge this in the next release!