Sanitize Unicode normalization form for file and directory names

[florianraith]

Add a unicode_normalization option for crawlers

Ran into this while downloading a course and then trying to AirDrop the folder to my iPad: everything under the umlaut directories (Übung/, Werbung für Hochschulgruppen/, ...) just silently disappeared, while the ASCII-named folders transferred fine.

Turns out it's a Unicode normalization thing. PFERD writes names in NFC (ü = one code point), which is totally fine locally since APFS (apple file system) doesn't care about the difference. But AirDrop re-normalizes paths to NFD (u + combining mark) when rebuilding the file tree on the other device, and for entries whose path crosses an NFC directory name that step fails and the files just get dropped. So even though the actual filenames are pure ASCII (ss20_aufgaben.pdf), the umlaut parent directory is enough to break it.

This adds a unicode_normalization crawler option to control the form names are written in:

[crawl:something]
unicode_normalization = nfd

Accepts none (default), nfc, nfd, nfkc and nfkd. Default is none, so nothing changes unless you opt in (setting it to nfd fixes the AirDrop problem for me).

The normalization happens in OutputDirectory right before the path is used, so the report, the names on disk and the cleanup all agree on the same form (otherwise cleanup would think everything changed and re-download the whole tree every run). Heads up that switching it on for an already-downloaded folder can cause a one-time re-download since the old names are still in the previous form (mentioned that in the docs).

nfd is probably a sensible default for macOS/iOS, but I didn't want to cause any breaking changes, so for this PR it's opt-in only.

[I-Al-Istannen]

Wat. So Apple decided they are special and unlike everything else they want NFD? And instead of transcoding that they just silently drop all your data?

[florianraith]

Wat. So Apple decided they are special and unlike everything else they want NFD? And instead of transcoding that they just silently drop all your data?

It appears so, but I haven't done any heavy research on this. While for the past few years when I owned a Mac I never encountered any issue with PFERD files, when I recently got my first iPad and tried to AirDrop some files from Mac to iPad the encoding issue popped up. So it might as well just be an issue with iPad (iOS?) and not with Apple/Mac in general

[florianraith]

with defaultnfc

with unicode_normalization = nfd

[I-Al-Istannen]

I am not ecstatic, but the change looks relatively non-invasive, so I think it should be fine. But WTF :P

[florianraith]

I am not ecstatic, but the change looks relatively non-invasive, so I think it should be fine. But WTF :P

neither was I while figuring out why some files were vanishing on my ipad :D

[I-Al-Istannen]

neither was I while figuring out why some files were vanishing on my ipad :D

Understandable...

Thanks!