When initialising the Unicode filter, the user should have the ability to specify which line endings to use:
# Unix line endings (default)
>>> f.Unicode(convert_newlines='\n').apply('Foo\nBar\rBaz\r\n')
'Foo\nBar\nBaz\n'
# Windows line endings
>>> f.Unicode(convert_newlines='\r\n').apply('Foo\nBar\rBaz\r\n')
'Foo\r\nBar\r\nBaz\r\n'
# Custom line endings
>>> f.Unicode(convert_newlines='|').apply('Foo\nBar\rBaz\r\n')
'Foo|Bar|Baz'
# Line endings unmodified
>>> f.Unicode(convert_newlines=False).apply('Foo\nBar\rBaz\r\n')
'Foo\nBar\rBaz\r\n'
Note that this change potentially conflicts with the normalize argument, so we'll probably need to make a couple of additional changes to support this new functionality:
- Add a
remove_unprintables argument.
- Rename
normalize to normalize_unicode.
- Deprecate the
normalize argument.
When initialising the
Unicodefilter, the user should have the ability to specify which line endings to use:Note that this change potentially conflicts with the
normalizeargument, so we'll probably need to make a couple of additional changes to support this new functionality:remove_unprintablesargument.normalizetonormalize_unicode.normalizeargument.