Skip to content

String serialization bug #5

Description

@edinhodiluviano

Hello,
I've been using your lib these last few days and I noticed a strange behavior with the string packing.
Digging a bit, there seems to have an inconsistency when serializing them.
The root cause is: when packing, the first byte(s) is the number of chars in the string (ds.py line 390). When unpacking, the first byte(s) is the number of bytes that represents the string (ds.py lines 395 and 396). This is a problem whenever we have multiple bytes utf-8 chars.

I don't actually know which approach is correct (number of chars or bytes). But if you tell me the right one, I'd be happy to send a PR with tests and a fix ; )

To reproduce:

import ueosio
ds = ueosio.DataStream()
ds.pack_string("µ")
ds.stream.seek(0)
ds.unpack_string()

Full example with output:

Python 3.8.11 (default, Jul  3 2021, 17:53:42) 
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ueosio
>>> ds = ueosio.DataStream()
>>> ds.pack_string("µ")
>>> ds.stream.seek(0)
0
>>> print(ds.stream.read())
b'\x01\xc2\xb5'
>>> ds.stream.seek(0)
0
>>> ds.unpack_string()
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/edinho/groups/eosdetroit/facings/eospyo/.venv/lib/python3.8/site-packages/ueosio/ds.py", line 425, in unpack_string
    return self.unpack_bytes().decode('utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 0: unexpected end of data

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions