Hello,
I've been using your lib these last few days and I noticed a strange behavior with the string packing.
Digging a bit, there seems to have an inconsistency when serializing them.
The root cause is: when packing, the first byte(s) is the number of chars in the string (ds.py line 390). When unpacking, the first byte(s) is the number of bytes that represents the string (ds.py lines 395 and 396). This is a problem whenever we have multiple bytes utf-8 chars.
I don't actually know which approach is correct (number of chars or bytes). But if you tell me the right one, I'd be happy to send a PR with tests and a fix ; )
To reproduce:
import ueosio
ds = ueosio.DataStream()
ds.pack_string("µ")
ds.stream.seek(0)
ds.unpack_string()
Full example with output:
Python 3.8.11 (default, Jul 3 2021, 17:53:42)
[GCC 7.5.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import ueosio
>>> ds = ueosio.DataStream()
>>> ds.pack_string("µ")
>>> ds.stream.seek(0)
0
>>> print(ds.stream.read())
b'\x01\xc2\xb5'
>>> ds.stream.seek(0)
0
>>> ds.unpack_string()
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/home/edinho/groups/eosdetroit/facings/eospyo/.venv/lib/python3.8/site-packages/ueosio/ds.py", line 425, in unpack_string
return self.unpack_bytes().decode('utf8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xc2 in position 0: unexpected end of data
Hello,
I've been using your lib these last few days and I noticed a strange behavior with the string packing.
Digging a bit, there seems to have an inconsistency when serializing them.
The root cause is: when packing, the first byte(s) is the number of chars in the string (ds.py line 390). When unpacking, the first byte(s) is the number of bytes that represents the string (ds.py lines 395 and 396). This is a problem whenever we have multiple bytes utf-8 chars.
I don't actually know which approach is correct (number of chars or bytes). But if you tell me the right one, I'd be happy to send a PR with tests and a fix ; )
To reproduce:
Full example with output: