Do not split UTF-8 characters between "Q"-encoded-words

[link2xt]

Partially addresses #40.

Multiple commits, don't squash.

[link2xt]

This does not enforce 76 character limit strictly and only splits newlines after overflow, but is an improvement over current state anyway. The test does not rely on exact line lengths, so this can be improved later and the test will ensure that there is no regression wrt. fix for character boundaries.

[mdecimus]

I think it would be more idiomatic (and likely more performant) to replace the current self.text.as_bytes().iter().enumerate() + text.is_char_boundary() combination with self.text.char_indices().

char_indices() allows iterating over valid UTF-8 character boundaries and avoids the need to manually check boundaries using is_char_boundary on each iteration.

[link2xt]

I thought about it, but I want to iterate over bytes rather than characters. Internally char_indices() just uses chars() iterator.

Regarding performance, checking for UTF-8 boundary is very cheap, it just looks at the byte value internally using this private function:
https://github.com/rust-lang/rust/blob/8708f3cd1f96d226f6420a58ebdd61aa0bc08b0a/library/core/src/num/mod.rs#L1073-L1077
Iterating over bytes might actually be more performant than iterating to the end of char and then rewinding back and looping over all char bytes again, but in any case I think it's better to not do such low-level optimizations without measurements.

[mdecimus]

In that case you can replace:

self.text.is_char_boundary(pos)

with

pos == 0 || ch.is_utf8_char_boundary()

this avoids the extra array access done internally by self.text.is_char_boundary(pos).

[link2xt]

is_utf8_char_boundary is private. And is_char_boundary is inlined, so compiler should optimize array access.
See also comments in the standard library:
https://github.com/rust-lang/rust/blob/8708f3cd1f96d226f6420a58ebdd61aa0bc08b0a/library/core/src/str/mod.rs#L360-L383
Anyway, I don't see much point in optimizing the code without measurements. I will be likely changed in the future anyway to have strict 76 character boundary, which I don't try to fix. My only goal is to fix the bug with UTF-8 characters being broken between encoded-words.

[mdecimus]

Yes, is_utf8_char_boundary is inlined but self.text.is_char_boundary(pos) is accessing the array by pos on each iteration and you already have ch with the value needed.

If is_utf8_char_boundary is private then an alternative would be pos == 0 || (ch as i8) >= -0x40.

[mdecimus]

Thanks!