UTF-8 files are not decoded before searching

[karenetheridge]

$ cat 'fooಠbar' > test
$ ack 'foo.bar' test
# ack 'foo..bar' test
$ ack 'foo...bar' test
fooಠbar

It would appear that the file is treated as UTF-8 bytes, so attempts to match a non-ascii character against . are not successful if the utf8 representation uses more than one octet.

Should files be assumed to be UTF-8 encoded? Or perhaps just when characters are seen that are not in the [\x00-\x7F] range? Or should a command-line argument be required to specify the encoding?

[n1vux]

It would appear that the file is treated as UTF-8 bytes, so attempts to match a non-ascii character against . are not successful if the utf8 representation uses more than one octet.

Well, it's treated as extended-ASCII bytes as characters, which might contain a UTF-8 multi-octet sequence, but we're mostly unaware of that. (If the first few bytes have high-bit or really low values, ASCII Control codes, we infer it's a binary file and skip it when directory scanning.)

Should files be assumed to be UTF-8 encoded?

Maybe? Is it harmless, if a file is in some extended-ascii code-page?

Or perhaps just when characters are seen that are not in the [\x00-\x7F] range?

While DWIM is nice, DWIM is not really practical with UTF since UTF BOMs are a standard observed mostly in the breach. (Files authored on a system with a default encoding often omit the BOM for that encoding. Your `fooಠbar` test file doesn't get a BOM on most systems; and wouldn't on many an Asian UTF-16/32 system either.) Any other DWIM than BOM is fraught as it's merely probabilistic. Do a smattering of high-bit-set octets indicate the file is UTF-8, or perhaps UTF-16(LE/BE), or UTF-32, or an Extended ASCII "code-page" unspecified, or is it a binary file we should be ignoring anyway? DWIM would promise more, work for some few UTF-8 files, and multiply edge-cases where it fails anyway.

Or should a command-line argument be required to specify the encoding?

We previously had had a workaround (documented in these tickets) with `perl -C -M'open=:std,:encoding(UTF-16LE)' ~/bin/ack` etc., but it looks like optimizations using `systread()` have nixed that? (*Since the workaround wasn't added to a FAQ but was only in bug-reply comments, I guess failure to create test-cases for it isn't horrible but ...*) So yes * probably a CLI argument/option* to specify encoding, which probably forces downshift from `sysread` buffering as well as using appropriate `PerlIO :encoding` layers. (or it could decode each sysread buffer, but that is fraught, e.g. when a buffer boundary occurs in the middle of a multi-octet UTF character; it's awkward enough working with line-endings across buffers!). [And no, we do not presume all practical files will fit in memory, so just slurp isn't the obvious win in this glorious big RAM future (*or it was glorious before RAM became $Unobtanium*!); ack can be a filter, should start producing output when it can; and can run on an RPi or whatever with minimal RAM.]