Error: internal error in Neon module: called Result::unwrap() on an Err value: SendError { .. }

[richvdh]

Sometimes every call to Seshat.commit() (and possibly other APIs) starts throwing an error, until the application is restarted:

Error: internal error in Neon module: called `Result::unwrap()` on an `Err` value: SendError { .. }
    at vector://vector/webapp/bundles/7301ce335b9a023b7ddd/init.js:1:1557039

This ends up making the application unusable: element-hq/element-web#33501

[richvdh]

#162 (landed in Seshat 4.1.0) aimed to change the error message to help diagnose this; however, we're still seeing the old error message with more recent versions of Seshat, so it looks like the error is happening in a different code path.

Possibly it's happening in neon::event::Channel::drop. I note that that code has been changed in more recent versions of neon (neon-bindings/neon@e856654), so it may be that updating neon would help. That doesn't entirely add up though, because the comment there suggests that send() should only fail during VM shutdown... and we're seeing the problem at other times.

It's probably not helped by the fact that we are building neon with napi-4 (and not napi-6), which means that we get a new instance of Channel which is then dropped for each request (see https://docs.rs/neon/latest/neon/context/trait.Context.html#method.channel). But still, that doesn't explain the root problem here.

[richvdh]

So, we think we're actually getting a std::sync::mspc::SendError, rather than a neon::event::SendError.

More specifically, we think is happening is that seshat's writer thread is dying; that will then cause operations such as commit to panic because they try to send over an mspc channel whose Receiver has gone away.

What we don't know yet: why the writer thread is dying, and what to do about it when it does.

[richvdh]

#174 is intended to confirm the above hypothesis.

[richvdh]

... and #176 will further improve the diagnostics, on the assumption that the previous hypothesis is correct.

[richvdh]

Per element-hq/element-web#33501 (comment), the errors with #176 have the format:

Error: internal error in Neon module: Could not send message of type Write to writer thread: Writer thread panicked with message: called `Result::unwrap()` on an `Err` value: IOError(IOError { path: Some("d2a04c78e494411484bc3cf976a302b9.pos"), err: Os { code: 5, kind: PermissionDenied, message: "Access is denied" } })