`json.decoder.JSONDecodeError` on sync (mangled event JSON)

[emdete]

Description

When trying to login from a web client I see this snipped in the log on the server and the client hangs forever:

Android clients work well. Older Synapse worked as well with this setup. the server in question was setup several years ago already. it seems to be related to an update to synapse. i am running matrix-synapse 1.139.2-1 on a Devuan system.

Steps to reproduce

• Start a web-element somewhere
• Point homeserver to my server
• Try to log in

Homeserver

my homeserver

Synapse Version

1.139.2

Installation Method

Other (please mention below)

Database

pg 13, single database, ported from sqlite, never restored from backup.

Workers

Single process

Platform

Devuan Ceres (unstable)

6.3.0-2-amd64, Debian 6.3.11-1

bare metal

Configuration

none of those

Relevant log output

2025-11-02 18:32:53,722 - synapse.http.server - 151 - ERROR - GET-7049- Failed handle request via 'SyncRestServlet': <XForwardedForRequest at 0x7f1b09112d50 method='GET' uri='/_matrix/client/v3/sync?filter=8&timeout=0&org.matrix.msc4222.use_state_after=true&_cacheBuster=1762108373545' clientproto='HTTP/1.1' site='800
8'>
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/synapse/http/server.py", line 339, in _async_render_wrapper
    callback_return = await self._async_render(request)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/synapse/http/server.py", line 564, in _async_render
    callback_return = await raw_callback_return
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/synapse/rest/client/sync.py", line 268, in on_GET
    sync_result = await self.sync_handler.wait_for_sync_for_user(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<6 lines>...
    )
    ^
  File "/usr/lib/python3/dist-packages/synapse/handlers/sync.py", line 364, in wait_for_sync_for_user
    res = await self.response_cache.wrap(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/usr/lib/python3/dist-packages/synapse/util/caches/response_cache.py", line 314, in wrap
    return await make_deferred_yieldable(result)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1195, in __iter__
    result.raiseException()
    ~~~~~~~~~~~~~~~~~~~~~^^
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 455, in raiseException
    raise self.value.with_traceback(self.tb)
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
        cast(Failure, result).throwExceptionIntoGenerator, gen
    )
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/synapse/util/caches/response_cache.py", line 292, in cb
    return await callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/synapse/handlers/sync.py", line 465, in _wait_for_sync_for_user
    result = await self.current_sync_for_user(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        sync_config, since_token, full_state=full_state
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/synapse/handlers/sync.py", line 533, in current_sync_for_user
    sync_result = await self.generate_sync_result(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        sync_config, since_token, full_state
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/synapse/handlers/sync.py", line 1724, in generate_sync_result
    ) = await self._generate_sync_entry_for_rooms(sync_result_builder)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/synapse/handlers/sync.py", line 2238, in _generate_sync_entry_for_rooms
    await concurrently_execute(handle_room_entries, room_entries, 10)
  File "/usr/lib/python3/dist-packages/synapse/util/async_helpers.py", line 259, in concurrently_execute
    await yieldable_gather_results(
    ...<2 lines>...
    )
  File "/usr/lib/python3/dist-packages/synapse/util/async_helpers.py", line 308, in yieldable_gather_results
    raise dfe.subFailure.value from None
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
        cast(Failure, result).throwExceptionIntoGenerator, gen
    )
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/synapse/util/async_helpers.py", line 245, in _concurrently_execute_inner
    await maybe_awaitable(func(value))
  File "/usr/lib/python3/dist-packages/synapse/handlers/sync.py", line 2227, in handle_room_entries
    await self._generate_room_entry(
    ...<6 lines>...
    )
  File "/usr/lib/python3/dist-packages/synapse/handlers/sync.py", line 2746, in _generate_room_entry
    state = await self.compute_state_delta(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<7 lines>...
    )
    ^
  File "/usr/lib/python3/dist-packages/synapse/handlers/sync.py", line 1165, in compute_state_delta
    state = await self.store.get_events(list(state_ids.values()))
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 618, in get_events
    events = await self.get_events_as_list(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<4 lines>...
    )
    ^
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 674, in get_events_as_list
    event_entry_map = await self.get_unredacted_events_from_cache_or_db(
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        set(event_ids), allow_rejected=allow_rejected
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 911, in get_unredacted_events_from_cache_or_db
    missing_events: Dict[str, EventCacheEntry] = await delay_cancellation(
                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^
        get_missing_events_from_cache_or_db()
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1187, in __iter__
    yield self
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1853, in _inlineCallbacks
    result = context.run(
        cast(Failure, result).throwExceptionIntoGenerator, gen
    )
  File "/usr/lib/python3/dist-packages/twisted/python/failure.py", line 467, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 897, in get_missing_events_from_cache_or_db
    raise e
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 890, in get_missing_events_from_cache_or_db
    db_missing_events = await self._get_events_from_db(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        missing_events_ids - missing_events.keys(),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 1368, in _get_events_from_db
    event_ids_to_fetch = await _fetch_event_ids_and_get_outstanding_redactions(
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        event_ids
        ^^^^^^^^^
    )
    ^
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 1353, in _fetch_event_ids_and_get_outstanding_redactions
    row_map = await self._enqueue_events(event_ids_to_fetch)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 1538, in _enqueue_events
    row_map = await events_d
              ^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/twisted/internet/defer.py", line 1187, in __iter__
    yield self
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 1286, in _fetch_event_list
    row_dict = self.db_pool.new_transaction(
        conn,
    ...<5 lines>...
        events_to_fetch,
    )
  File "/usr/lib/python3/dist-packages/synapse/storage/database.py", line 821, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/synapse/storage/databases/main/events_worker.py", line 1618, in _fetch_event_rows
    event_json = json.loads(event.json)
  File "/usr/lib/python3.13/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ~~~~~~~~~~~~~~~~~~~~~~~^^^
  File "/usr/lib/python3.13/json/decoder.py", line 345, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.13/json/decoder.py", line 363, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

Anything else that would be useful to know?

The whole thing started with element-web not being able to show pics (or media) while android element-x could do so. That's why I started to create a new session which didn't work at all. Existing sessions seem to work but suffer from not being able to show/download media.

[MadLittleMods]

I've created a PR to update the error here to point out the event ID of the event causing the problem, #19169

This doesn't solve the issue at all but makes it easier to jump into the database and inspect the event in question.

[emdete]

I changed the code (synapse/storage/databases/main/events_worker.py#L1612) to ignore such decode errors. Instead, I log the content:

7b22617574685f6576656e7473223a5b5b22243135323332363333353331313130394f6250446b3a74616c6b2e707572692e736d222c7b22736861323536223a223670475641583

Assuming these are byte nibbles I decoded it et voila:

{"auth_events":[["$15..PDk:talk.puri.sm",{"sha256":"6p..obM6JA"}],["$15..58ntolg:matri..

(I shortened the IDs with .. here). it seems there are some garbage entries in my database?

After the change the clients now start up properly.

Any idea how these entries get into the database?

Let me know if I can help investigating the issue and how.

[MadLittleMods]

Malformed event

For my own reference, here is how to decode what you posted:

hex_string = "7b22617574685f6576656e7473223a5b5b22243135323332363333353331313130394f6250446b3a74616c6b2e707572692e736d222c7b22736861323536223a22367047564158"
decoded_bytes = bytes.fromhex(hex_string)
decoded_text = decoded_bytes.decode("utf-8")
print(decoded_text)
# {"auth_events":[["$152326335311109ObPDk:talk.puri.sm",{"sha256":"6pGVAX

The auth_events definitely look messed up as they should be a list of strings (event ID's).

Is the event sensitive? Can you post the full thing so we can have a better frame of reference (which type, exact shape, etc)?

Normal well-formed events

Comparing to some normal events:

m.room.create:

{"auth_events":[],"prev_events":[],"type":"m.room.create","sender":"@user-1-alice:hs1","content":{"room_version":"10","creator":"@user-1-alice:hs1"},"depth":1,"room_id":"!nUkSuiRyvhAVWIHvjC:hs1","state_key":"","origin_server_ts":1763148315175,"hashes":{"sha256":"pcAQWOm0GZ7TeyogyL6P5qEre+SCcSQQwGISlJl2d8U"},"signatures":{"hs1":{"ed25519:a_UDzP":"365q2SuudvH6y9vKOKE93bcVgB0Qfo78kGxJCnsU8MdAyhoqtU/zKAPQcbBhiKJte3V8qFCQI3reEZs66yUbBw"}},"unsigned":{"age":3171}}

m.room.join_rules:

{"auth_events":["$1l1OKndjBbRLVkAcf7Ta4PZsL-JAEIRJdm0LT2B7VbA","$uJI3D26ZzAjrhCBf8lJnmu1xA9UcBQQlqR6qADuYeOE","$1sR2OiG_GIXBgf9UmhS_oAz7VR_sJXErOIjXlVn3Gtk"],"prev_events":["$uJI3D26ZzAjrhCBf8lJnmu1xA9UcBQQlqR6qADuYeOE"],"type":"m.room.join_rules","sender":"@user-1-alice:hs1","content":{"join_rule":"public"},"depth":4,"room_id":"!nUkSuiRyvhAVWIHvjC:hs1","state_key":"","origin_server_ts":1763148315483,"hashes":{"sha256":"W+bA3yqp0jPM3qaZsWWluHz/wxvB/ef3nbOJZAENQ2k"},"signatures":{"hs1":{"ed25519:a_UDzP":"CPUHL1mLfutJrzmV2VPjLqW937svEMQ2T4T9xNUfxy2kQz9iuJuDdNiMN0opP6hyoxE5bdm9tOwE2l6F9e0rDw"}},"unsigned":{"age":2863}}

m.room.message:

{"auth_events":["$ln6INDAdTPxIcM8prdn1jpgIhaPX69lD-FRi4F8f7Bc","$92YoLMgRHtrfbonRmuLs4FlMVpGrr7giQPd66M6a0Bg","$6P8ksuEtF6XADGjXB-1GVA9RKyN7VJ1zVIi_WQK8hWs"],"prev_events":["$ivkoiUyC5lAF8Y94r2_fNzccjGS1wC3z7AAs-rg4RrA"],"type":"m.room.message","sender":"@user-1-alice:hs1","content":{"body":"Filler message 300 to increase history size.","msgtype":"m.text"},"depth":305,"room_id":"!wacmIZBraTxforZbzf:hs1","origin_server_ts":1763148338107,"hashes":{"sha256":"VIZNZODqINKVDzlo+LMJRTFDJAwZdOfYSrnNcLWxpAM"},"signatures":{"hs1":{"ed25519:a_UDzP":"t87d6kjx7Wt2OLmlyAFZItoB4wWt43M0oJPufuLwgVePZAP/t90vf81/CzWN7FiIX4US7wSGbimq9JjSJsC4Bg"}},"unsigned":{"age":245}}

[emdete]

so you mean the encoded string is correct? but why does it throw it into an json-decoder without doing the .fromhex
decode step? that error happens in the line mentioned above, the string given to loads() is that hex string. here is what i see in the log now:

2025-11-14 07:29:12,494 - synapse.storage.databases.main.events_worker - 1615 - WARNING - fetch_events-859- json is '\x7b22617574685f6576656e7473223a5b5b22243135323332363333353331313130394f6250446b3a74616c6b2e707572692e736d222c7b22736861323536223a223670475641583341704c6c4455374d5739334757734e6a616533735445635745513134534a6f624d364a41227d5d2c5b222431353536373133303035386e746f6c673a6d61747269782e737765657462612e7365222c7b22736861323536223a2242334d4557485a4846624f2f54464c31584862586d6951356b36614759645065732f3853745538424b6e51227d5d2c5b222431353536343339393931313935333538415759664d3a74616c6b2e707572692e736d222c7b22736861323536223a225970796b57494f66464d6d2b68536344773141716f64646a7576366e5a6a465462776f576c7579564b7941227d5d5d2c22636f6e74656e74223a7b22616c6961736573223a5b5d7d2c226465707468223a32333338372c226576656e745f6964223a222431353536373133323837323562774c63753a6d61747269782e737765657462612e7365222c22686173686573223a7b22736861323536223a22476f625631416161376935444b4e73784849375032527851756a323677713373584f364d554c526f742b30227d2c226f726967696e223a226d61747269782e737765657462612e7365222c226f726967696e5f7365727665725f7473223a313535363731333238373035302c22707265765f6576656e7473223a5b5b2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365222c7b22736861323536223a226a563577434b69797a71783076316d4a3567354c357a6f684e31495232536f39666a594c2b536b61566d45227d5d5d2c22707265765f7374617465223a5b5d2c22726f6f6d5f6964223a22214253715248677643744973476974746b42473a74616c6b2e707572692e736d222c2273656e646572223a2240636f6469666c6f773a6d61747269782e737765657462612e7365222c227369676e617475726573223a7b226d61747269782e737765657462612e7365223a7b22656432353531393a615f5352536b223a224a6f35715065614d4e553155652f48362b706d4c6274636e674b336730753757775a51545a36652b35705364714e303737494276312b7450797a7a5536655053467158315a796e433462417575797656396351734477227d7d2c2273746174655f6b6579223a226d61747269782e737765657462612e7365222c2274797065223a226d2e726f6f6d2e616c6961736573222c22756e7369676e6564223a7b226167655f7473223a313535363731333238373035302c227265706c616365735f7374617465223a2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365227d7d'
2025-11-14 07:29:12,521 - synapse.storage._base - 244 - WARNING - GET-78280- Tried to decode ''\\x7b22617574685f6576656e7473223a5b5b22243135323332363333353331313130394f6250446b3a74616c6b2e707572692e736d222c7b22736861323536223a223670475641583341704c6c4455374d5739334757734e6a616533735445635745513134534a6f624d364a41227d5d2c5b222431353536373133303035386e746f6c673a6d61747269782e737765657462612e7365222c7b22736861323536223a2242334d4557485a4846624f2f54464c31584862586d6951356b36614759645065732f3853745538424b6e51227d5d2c5b222431353536343339393931313935333538415759664d3a74616c6b2e707572692e736d222c7b22736861323536223a225970796b57494f66464d6d2b68536344773141716f64646a7576366e5a6a465462776f576c7579564b7941227d5d5d2c22636f6e74656e74223a7b22616c6961736573223a5b5d7d2c226465707468223a32333338372c226576656e745f6964223a222431353536373133323837323562774c63753a6d61747269782e737765657462612e7365222c22686173686573223a7b22736861323536223a22476f625631416161376935444b4e73784849375032527851756a323677713373584f364d554c526f742b30227d2c226f726967696e223a226d61747269782e737765657462612e7365222c226f726967696e5f7365727665725f7473223a313535363731333238373035302c22707265765f6576656e7473223a5b5b2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365222c7b22736861323536223a226a563577434b69797a71783076316d4a3567354c357a6f684e31495232536f39666a594c2b536b61566d45227d5d5d2c22707265765f7374617465223a5b5d2c22726f6f6d5f6964223a22214253715248677643744973476974746b42473a74616c6b2e707572692e736d222c2273656e646572223a2240636f6469666c6f773a6d61747269782e737765657462612e7365222c227369676e617475726573223a7b226d61747269782e737765657462612e7365223a7b22656432353531393a615f5352536b223a224a6f35715065614d4e553155652f48362b706d4c6274636e674b336730753757775a51545a36652b35705364714e303737494276312b7450797a7a5536655053467158315a796e433462417575797656396351734477227d7d2c2273746174655f6b6579223a226d61747269782e737765657462612e7365222c2274797065223a226d2e726f6f6d2e616c6961736573222c22756e7369676e6564223a7b226167655f7473223a313535363731333238373035302c227265706c616365735f7374617465223a2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365227d7d'' as JSON and failed
2025-11-14 07:29:12,522 - synapse.storage.databases.main.events_worker - 1388 - ERROR - GET-78280- Unable to parse json from event: $155671328725bwLcu:matrix.sweetba.se

the first line is mine, all the rest comes from synapse. the second and third line isn't shown without my try-except-log change.

is it possible that this event is double encoded by accident? can i check the db manually? where can i find that thingie?

[MadLittleMods]

so you mean the encoded string is correct?

I'm explaining how to interpret what you posted and comparing it to a normal event JSON string (mostly for my own reference).

Things appear mangled on multiple levels.

The string includes literal single quotes in it

In the code, event.json should be a str containing JSON. It looks like it is actually a literal string containing: '\x7b2...' (single quotes and all, bad ❌)

What's with \x?

The \x in the string itself isn't necessarily wrong (it's just an escape sequence). For example, print('\x61') is the same as print("a")

But it doesn't mean much here and only applies to the first character (\x7b -> {) if this is Python escape sequence.

print("'\x7b22617574685f6576656e7473223a5b5b22243135323332363333353331313130394f6250446b3a74616c6b2e707572692e736d222c7b22736861323536223a223670475641583341704c6c4455374d5739334757734e6a616533735445635745513134534a6f624d364a41227d5d2c5b222431353536373133303035386e746f6c673a6d61747269782e737765657462612e7365222c7b22736861323536223a2242334d4557485a4846624f2f54464c31584862586d6951356b36614759645065732f3853745538424b6e51227d5d2c5b222431353536343339393931313935333538415759664d3a74616c6b2e707572692e736d222c7b22736861323536223a225970796b57494f66464d6d2b68536344773141716f64646a7576366e5a6a465462776f576c7579564b7941227d5d5d2c22636f6e74656e74223a7b22616c6961736573223a5b5d7d2c226465707468223a32333338372c226576656e745f6964223a222431353536373133323837323562774c63753a6d61747269782e737765657462612e7365222c22686173686573223a7b22736861323536223a22476f625631416161376935444b4e73784849375032527851756a323677713373584f364d554c526f742b30227d2c226f726967696e223a226d61747269782e737765657462612e7365222c226f726967696e5f7365727665725f7473223a313535363731333238373035302c22707265765f6576656e7473223a5b5b2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365222c7b22736861323536223a226a563577434b69797a71783076316d4a3567354c357a6f684e31495232536f39666a594c2b536b61566d45227d5d5d2c22707265765f7374617465223a5b5d2c22726f6f6d5f6964223a22214253715248677643744973476974746b42473a74616c6b2e707572692e736d222c2273656e646572223a2240636f6469666c6f773a6d61747269782e737765657462612e7365222c227369676e617475726573223a7b226d61747269782e737765657462612e7365223a7b22656432353531393a615f5352536b223a224a6f35715065614d4e553155652f48362b706d4c6274636e674b336730753757775a51545a36652b35705364714e303737494276312b7450797a7a5536655053467158315a796e433462417575797656396351734477227d7d2c2273746174655f6b6579223a226d61747269782e737765657462612e7365222c2274797065223a226d2e726f6f6d2e616c6961736573222c22756e7369676e6564223a7b226167655f7473223a313535363731333238373035302c227265706c616365735f7374617465223a2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365227d7d'")
->
'{22617574685f6576656e7473223a5b5b22243135323332363333353331313130394f6250446b3a74616c6b2e707572692e736d222c7b22736861323536223a223670475641583341704c6c4455374d5739334757734e6a616533735445635745513134534a6f624d364a41227d5d2c5b222431353536373133303035386e746f6c673a6d61747269782e737765657462612e7365222c7b22736861323536223a2242334d4557485a4846624f2f54464c31584862586d6951356b36614759645065732f3853745538424b6e51227d5d2c5b222431353536343339393931313935333538415759664d3a74616c6b2e707572692e736d222c7b22736861323536223a225970796b57494f66464d6d2b68536344773141716f64646a7576366e5a6a465462776f576c7579564b7941227d5d5d2c22636f6e74656e74223a7b22616c6961736573223a5b5d7d2c226465707468223a32333338372c226576656e745f6964223a222431353536373133323837323562774c63753a6d61747269782e737765657462612e7365222c22686173686573223a7b22736861323536223a22476f625631416161376935444b4e73784849375032527851756a323677713373584f364d554c526f742b30227d2c226f726967696e223a226d61747269782e737765657462612e7365222c226f726967696e5f7365727665725f7473223a313535363731333238373035302c22707265765f6576656e7473223a5b5b2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365222c7b22736861323536223a226a563577434b69797a71783076316d4a3567354c357a6f684e31495232536f39666a594c2b536b61566d45227d5d5d2c22707265765f7374617465223a5b5d2c22726f6f6d5f6964223a22214253715248677643744973476974746b42473a74616c6b2e707572692e736d222c2273656e646572223a2240636f6469666c6f773a6d61747269782e737765657462612e7365222c227369676e617475726573223a7b226d61747269782e737765657462612e7365223a7b22656432353531393a615f5352536b223a224a6f35715065614d4e553155652f48362b706d4c6274636e674b336730753757775a51545a36652b35705364714e303737494276312b7450797a7a5536655053467158315a796e433462417575797656396351734477227d7d2c2273746174655f6b6579223a226d61747269782e737765657462612e7365222c2274797065223a226d2e726f6f6d2e616c6961736573222c22756e7369676e6564223a7b226167655f7473223a313535363731333238373035302c227265706c616365735f7374617465223a2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365227d7d'

Or even without the extra layer of quotes

print('\x7b22617574685f6576656e7473223a5b5b22243135323332363333353331313130394f6250446b3a74616c6b2e707572692e736d222c7b22736861323536223a223670475641583341704c6c4455374d5739334757734e6a616533735445635745513134534a6f624d364a41227d5d2c5b222431353536373133303035386e746f6c673a6d61747269782e737765657462612e7365222c7b22736861323536223a2242334d4557485a4846624f2f54464c31584862586d6951356b36614759645065732f3853745538424b6e51227d5d2c5b222431353536343339393931313935333538415759664d3a74616c6b2e707572692e736d222c7b22736861323536223a225970796b57494f66464d6d2b68536344773141716f64646a7576366e5a6a465462776f576c7579564b7941227d5d5d2c22636f6e74656e74223a7b22616c6961736573223a5b5d7d2c226465707468223a32333338372c226576656e745f6964223a222431353536373133323837323562774c63753a6d61747269782e737765657462612e7365222c22686173686573223a7b22736861323536223a22476f625631416161376935444b4e73784849375032527851756a323677713373584f364d554c526f742b30227d2c226f726967696e223a226d61747269782e737765657462612e7365222c226f726967696e5f7365727665725f7473223a313535363731333238373035302c22707265765f6576656e7473223a5b5b2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365222c7b22736861323536223a226a563577434b69797a71783076316d4a3567354c357a6f684e31495232536f39666a594c2b536b61566d45227d5d5d2c22707265765f7374617465223a5b5d2c22726f6f6d5f6964223a22214253715248677643744973476974746b42473a74616c6b2e707572692e736d222c2273656e646572223a2240636f6469666c6f773a6d61747269782e737765657462612e7365222c227369676e617475726573223a7b226d61747269782e737765657462612e7365223a7b22656432353531393a615f5352536b223a224a6f35715065614d4e553155652f48362b706d4c6274636e674b336730753757775a51545a36652b35705364714e303737494276312b7450797a7a5536655053467158315a796e433462417575797656396351734477227d7d2c2273746174655f6b6579223a226d61747269782e737765657462612e7365222c2274797065223a226d2e726f6f6d2e616c6961736573222c22756e7369676e6564223a7b226167655f7473223a313535363731333238373035302c227265706c616365735f7374617465223a2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365227d7d')
->
{22617574685f6576656e7473223a5b5b22243135323332363333353331313130394f6250446b3a74616c6b2e707572692e736d222c7b22736861323536223a223670475641583341704c6c4455374d5739334757734e6a616533735445635745513134534a6f624d364a41227d5d2c5b222431353536373133303035386e746f6c673a6d61747269782e737765657462612e7365222c7b22736861323536223a2242334d4557485a4846624f2f54464c31584862586d6951356b36614759645065732f3853745538424b6e51227d5d2c5b222431353536343339393931313935333538415759664d3a74616c6b2e707572692e736d222c7b22736861323536223a225970796b57494f66464d6d2b68536344773141716f64646a7576366e5a6a465462776f576c7579564b7941227d5d5d2c22636f6e74656e74223a7b22616c6961736573223a5b5d7d2c226465707468223a32333338372c226576656e745f6964223a222431353536373133323837323562774c63753a6d61747269782e737765657462612e7365222c22686173686573223a7b22736861323536223a22476f625631416161376935444b4e73784849375032527851756a323677713373584f364d554c526f742b30227d2c226f726967696e223a226d61747269782e737765657462612e7365222c226f726967696e5f7365727665725f7473223a313535363731333238373035302c22707265765f6576656e7473223a5b5b2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365222c7b22736861323536223a226a563577434b69797a71783076316d4a3567354c357a6f684e31495232536f39666a594c2b536b61566d45227d5d5d2c22707265765f7374617465223a5b5d2c22726f6f6d5f6964223a22214253715248677643744973476974746b42473a74616c6b2e707572692e736d222c2273656e646572223a2240636f6469666c6f773a6d61747269782e737765657462612e7365222c227369676e617475726573223a7b226d61747269782e737765657462612e7365223a7b22656432353531393a615f5352536b223a224a6f35715065614d4e553155652f48362b706d4c6274636e674b336730753757775a51545a36652b35705364714e303737494276312b7450797a7a5536655053467158315a796e433462417575797656396351734477227d7d2c2273746174655f6b6579223a226d61747269782e737765657462612e7365222c2274797065223a226d2e726f6f6d2e616c6961736573222c22756e7369676e6564223a7b226167655f7473223a313535363731333238373035302c227265706c616365735f7374617465223a2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365227d7d

Decoding further (just to pick it apart)

Let's use the same strategy from my last post to pick it apart a bit further:

hex_string = "7b22617574685f6576656e7473223a5b5b22243135323332363333353331313130394f6250446b3a74616c6b2e707572692e736d222c7b22736861323536223a223670475641583341704c6c4455374d5739334757734e6a616533735445635745513134534a6f624d364a41227d5d2c5b222431353536373133303035386e746f6c673a6d61747269782e737765657462612e7365222c7b22736861323536223a2242334d4557485a4846624f2f54464c31584862586d6951356b36614759645065732f3853745538424b6e51227d5d2c5b222431353536343339393931313935333538415759664d3a74616c6b2e707572692e736d222c7b22736861323536223a225970796b57494f66464d6d2b68536344773141716f64646a7576366e5a6a465462776f576c7579564b7941227d5d5d2c22636f6e74656e74223a7b22616c6961736573223a5b5d7d2c226465707468223a32333338372c226576656e745f6964223a222431353536373133323837323562774c63753a6d61747269782e737765657462612e7365222c22686173686573223a7b22736861323536223a22476f625631416161376935444b4e73784849375032527851756a323677713373584f364d554c526f742b30227d2c226f726967696e223a226d61747269782e737765657462612e7365222c226f726967696e5f7365727665725f7473223a313535363731333238373035302c22707265765f6576656e7473223a5b5b2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365222c7b22736861323536223a226a563577434b69797a71783076316d4a3567354c357a6f684e31495232536f39666a594c2b536b61566d45227d5d5d2c22707265765f7374617465223a5b5d2c22726f6f6d5f6964223a22214253715248677643744973476974746b42473a74616c6b2e707572692e736d222c2273656e646572223a2240636f6469666c6f773a6d61747269782e737765657462612e7365222c227369676e617475726573223a7b226d61747269782e737765657462612e7365223a7b22656432353531393a615f5352536b223a224a6f35715065614d4e553155652f48362b706d4c6274636e674b336730753757775a51545a36652b35705364714e303737494276312b7450797a7a5536655053467158315a796e433462417575797656396351734477227d7d2c2273746174655f6b6579223a226d61747269782e737765657462612e7365222c2274797065223a226d2e726f6f6d2e616c6961736573222c22756e7369676e6564223a7b226167655f7473223a313535363731333238373035302c227265706c616365735f7374617465223a2224313535363731333237313234614b6747573a6d61747269782e737765657462612e7365227d7d"
decoded_bytes = bytes.fromhex(hex_string)
decoded_text = decoded_bytes.decode("utf-8")
print(decoded_text)
# {"auth_events":[["$152326335311109ObPDk:talk.puri.sm",{"sha256":"6pGVAX3ApLlDU7MW93GWsNjae3sTEcWEQ14SJobM6JA"}],["$15567130058ntolg:matrix.sweetba.se",{"sha256":"B3MEWHZHFbO/TFL1XHbXmiQ5k6aGYdPes/8StU8BKnQ"}],["$1556439991195358AWYfM:talk.puri.sm",{"sha256":"YpykWIOfFMm+hScDw1Aqoddjuv6nZjFTbwoWluyVKyA"}]],"content":{"aliases":[]},"depth":23387,"event_id":"$155671328725bwLcu:matrix.sweetba.se","hashes":{"sha256":"GobV1Aaa7i5DKNsxHI7P2RxQuj26wq3sXO6MULRot+0"},"origin":"matrix.sweetba.se","origin_server_ts":1556713287050,"prev_events":[["$155671327124aKgGW:matrix.sweetba.se",{"sha256":"jV5wCKiyzqx0v1mJ5g5L5zohN1IR2So9fjYL+SkaVmE"}]],"prev_state":[],"room_id":"!BSqRHgvCtIsGittkBG:talk.puri.sm","sender":"@codiflow:matrix.sweetba.se","signatures":{"matrix.sweetba.se":{"ed25519:a_SRSk":"Jo5qPeaMNU1Ue/H6+pmLbtcngK3g0u7WwZQTZ6e+5pSdqN077IBv1+tPyzzU6ePSFqX1ZynC4bAuuyvV9cQsDw"}},"state_key":"matrix.sweetba.se","type":"m.room.aliases","unsigned":{"age_ts":1556713287050,"replaces_state":"$155671327124aKgGW:matrix.sweetba.se"}}

Pretty-printing this event to analyze we can see how things look:

Edit: I initially thought this event format was weird but it's just the event format from room version 1 and 2.

{
  "auth_events": [
    [
      "$152326335311109ObPDk:talk.puri.sm",
      {
        "sha256": "6pGVAX3ApLlDU7MW93GWsNjae3sTEcWEQ14SJobM6JA"
      }
    ],
    [
      "$15567130058ntolg:matrix.sweetba.se",
      {
        "sha256": "B3MEWHZHFbO/TFL1XHbXmiQ5k6aGYdPes/8StU8BKnQ"
      }
    ],
    [
      "$1556439991195358AWYfM:talk.puri.sm",
      {
        "sha256": "YpykWIOfFMm+hScDw1Aqoddjuv6nZjFTbwoWluyVKyA"
      }
    ]
  ],
  "content": {
    "aliases": []
  },
  "depth": 23387,
  "event_id": "$155671328725bwLcu:matrix.sweetba.se",
  "hashes": {
    "sha256": "GobV1Aaa7i5DKNsxHI7P2RxQuj26wq3sXO6MULRot+0"
  },
  "origin": "matrix.sweetba.se",
  "origin_server_ts": 1556713287050,
  "prev_events": [
    [
      "$155671327124aKgGW:matrix.sweetba.se",
      {
        "sha256": "jV5wCKiyzqx0v1mJ5g5L5zohN1IR2So9fjYL+SkaVmE"
      }
    ]
  ],
  "prev_state": [],
  "room_id": "!BSqRHgvCtIsGittkBG:talk.puri.sm",
  "sender": "@codiflow:matrix.sweetba.se",
  "signatures": {
    "matrix.sweetba.se": {
      "ed25519:a_SRSk": "Jo5qPeaMNU1Ue/H6+pmLbtcngK3g0u7WwZQTZ6e+5pSdqN077IBv1+tPyzzU6ePSFqX1ZynC4bAuuyvV9cQsDw"
    }
  },
  "state_key": "matrix.sweetba.se",
  "type": "m.room.aliases",
  "unsigned": {
    "age_ts": 1556713287050,
    "replaces_state": "$155671327124aKgGW:matrix.sweetba.se"
  }
}

This event is from 2019-05-01 according to origin_server_ts and unsigned.age_ts in the event and was created by the local homeserver matrix.sweetba.se.

Nowadays, it appears like the homeserver is running the latest Synapse 1.142.0 (according to https://matrix.sweetba.se/_matrix/federation/v1/version).

Affected by previous Synapse bugs

I think you're being hit by matrix-org/synapse#5312 (comment) which mentions event_json was affected JSON data being migrated incorrectly when using synapse_port_db. I also think this is the problem because it also mentions the byte encoding stuff.

Even if you used the suggested fix on that issue, I'm not sure it would completely solve your situation although we may get valid JSON at-least to get past this problem.

Be very cautious of following those instructions as they were written in the context of many years ago and may not apply today. Please back up your database before attempting anything. Manually editing the database is generally not recommended. We don't provide support or warranty for people who mess things up.

But with more of this context now, perhaps you can self-service a bit and report back.

[MadLittleMods]

As an update, in terms of the event format looking weird, @realtyem pointed out that this is just the event format from room version 1 and 2 (v1/v2 rooms) which I was just unfamiliar with (before my time).

The event is still mangled and serialized/encoded badly in the database as some sort of byte format, but the underlying JSON itself is an expected old event format. Doing the byte conversion carefully definitely seems like the path to victory (as mentioned in my previous reply) ⏩