Skip to content

Allow specifying line ending style to Unicode filter #34

Description

@todofixthis

When initialising the Unicode filter, the user should have the ability to specify which line endings to use:

# Unix line endings (default)
>>> f.Unicode(convert_newlines='\n').apply('Foo\nBar\rBaz\r\n')
'Foo\nBar\nBaz\n'

# Windows line endings
>>> f.Unicode(convert_newlines='\r\n').apply('Foo\nBar\rBaz\r\n')
'Foo\r\nBar\r\nBaz\r\n'

# Custom line endings
>>> f.Unicode(convert_newlines='|').apply('Foo\nBar\rBaz\r\n')
'Foo|Bar|Baz'

# Line endings unmodified
>>> f.Unicode(convert_newlines=False).apply('Foo\nBar\rBaz\r\n')
'Foo\nBar\rBaz\r\n'

Note that this change potentially conflicts with the normalize argument, so we'll probably need to make a couple of additional changes to support this new functionality:

  • Add a remove_unprintables argument.
  • Rename normalize to normalize_unicode.
  • Deprecate the normalize argument.

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions