Invalid memory address after lot of concurrency

[vlourme]

Hey, this might not be directly related to rio.

I'm continuing my tests on my HTTP proxy, actually just trying to splice in io_uring, using the new exported Fd().
When doing a lot of splice, I run into a invalid memory address or nil pointer reference:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x642382]

goroutine 31 [running]:
github.com/brickingsoft/rio/pkg/liburing/aio.(*Channel).Complete(0xc004ba2300?, 0x786a59712480?, 0x91d30?, {0x926940?, 0xc009a5ed60?})
	/root/go/pkg/mod/github.com/vlourme/rio@v0.0.0-20250509065627-533b82f20637/pkg/liburing/aio/channel.go:107 +0x22
github.com/brickingsoft/rio/pkg/liburing/aio.(*Poller).prepareSQE(0xc003ae4000, 0xc005484f00)
	/root/go/pkg/mod/github.com/vlourme/rio@v0.0.0-20250509065627-533b82f20637/pkg/liburing/aio/poller.go:738 +0x2b7
github.com/brickingsoft/rio/pkg/liburing/aio.(*Poller).process1(0xc003ae4000)
	/root/go/pkg/mod/github.com/vlourme/rio@v0.0.0-20250509065627-533b82f20637/pkg/liburing/aio/poller.go:563 +0x2b9
github.com/brickingsoft/rio/pkg/liburing/aio.(*Poller).process(0xc003ae4000)
	/root/go/pkg/mod/github.com/vlourme/rio@v0.0.0-20250509065627-533b82f20637/pkg/liburing/aio/poller.go:496 +0xc9
github.com/brickingsoft/rio/pkg/liburing/aio.newPoller.func1({0x0, 0xb100, 0x0, 0x0, 0x0, 0x0, 0x0, {0x0, 0x0, 0x0}, ...}, ...)
	/root/go/pkg/mod/github.com/vlourme/rio@v0.0.0-20250509065627-533b82f20637/pkg/liburing/aio/poller.go:229 +0x7dd
created by github.com/brickingsoft/rio/pkg/liburing/aio.newPoller in goroutine 30
	/root/go/pkg/mod/github.com/vlourme/rio@v0.0.0-20250509065627-533b82f20637/pkg/liburing/aio/poller.go:112 +0x22a
exit status 2

The error can be triggered easily by running multiple connection, each one handled in a single goroutine, and by doing something like conn.WriteTo(dst) or a splice fd.Splice(dest.Fd(), 4096) (from aio)

This will work for maybe 500 connections and later crash. I'm not sure if rio is supposed to be thread-safe, IIRC, io_uring is more like a thread-per-core idea.

[vlourme]

I confirm, it's working with a single thread. Performance are quite terrible doing that. It seems like ReusePort is not working, I can't bind same port multiple time and do a true thread-per-core listener

[wamshawn]

reuse port was fixed. see v1.6.10.

For invalid memory address.
I will test it later.

the reason is that rio use sqe.userdata to transfer pointers between userspace and the kernel, and the point type is uintptr which go do not trace, so it maybe gc before iouring.Enter completed.

keepalive aio.Operation and aio.Channel to make sure the pointer will not gc before enter.

For single thread or multi threads.
Many people have brought this issue back.

The answer from iouring author is that single thread is recommended by iouring, because it is good for batch enter, so there is a submit and wait func. or use 2 threads, one submit and another wait cqe.

rio use iouring setup flags to decide using one thread or two threads.

[wamshawn]

I use k6 and wrk to test 10,000 conns. it passed.

[wamshawn]

@vlourme
For posix supporting, the Fd func in rio.Conn was removed, but also can use it from *rio.TCPConn *rio.UDPConn and *rio.UnixConn

[vlourme]

@vlourme
For posix supporting, the Fd func in rio.Conn was removed, but also can use it from *rio.TCPConn *rio.UDPConn and *rio.UnixConn

No problem. I always cast to TCPConn. I'll retry later using connections thread that prepare only and another thread that submit (maybe with a mutex?).

I was just using aio.Splice for now, but it does not support one connection per thread because it submit directly in the Splice.

Thank you for your reactivity by the way!

[wamshawn]

No problem. I always cast to TCPConn. I'll retry later using connections thread that prepare only and another thread that submit (maybe with a mutex?).

When use aio.poller, mutex is not required, cause synchronize using chan.

If you use a customized poller, note that iouring is not thread safe, such as get sqe and submit, make sure that prep and submit in one thread, or the submit thread forbid prep and wait all prep done, then submit and allow prep thread to prep. note, the linked sqe can not be used cause get sqe is not thread safe.

You can use two wait group, prepWG and submitWG.
prepWG : add before prep and done after prep.
submitWG : add before submit and done after submit.

prep thread: submitWG.Wait then prep.
sumitWG: submitWG.Add and prepWG.Wait.

I was just using aio.Splice for now, but it does not support one connection per thread because it submit directly in the Splice.

aio.splice is create op and send it to poller processing thread with chan, and wait result via op.channel.

Try to add runtime.KeepAlive(src) before last return in func (fd *Fd) Splice(src *Fd, remain int64) (n int64, err error).
I can not be sure that src fd was lost cause there is a loop.

In my words, splice is not good enough, for example, it can not handle TLS. it only transfer data to another fd.
If just transfer data, I will ask axe for a prep_recvsend_multishot op, which is recv msg and send msg to another fd in iouring, return a number about result. so there is no pipe.

[vlourme]

I'm doing some more tests using thread-per-core, connection are handled in a single core, no go-routine.
I have 24 cores so 24 connections handled at same time, this is just for benchmarking and testing, I doubt I get better performance like that.

I created a small package that fill a map threadId -> *liburing.Ring and I'm now trying to splice:
(full example: https://pastes.dev/LMl2Pa2pVr)

// Copy copies data from srcFd to dstFd using a pipe to splice
func Copy(ringId int, srcFd, dstFd int) {
	log.Info().Int("id", ringId).Int("srcFd", srcFd).Int("dstFd", dstFd).Msg("Copying")
	ring := rings[ringId] // One ring created for each core
	pipe := pipes[ringId] // One pipe created for each core
	log.Info().Int("pipe0", pipe[0]).Int("pipe1", pipe[1]).Msg("Pipes")

	for _, fd := range []int{srcFd, dstFd, pipe[0], pipe[1]} {
		_, err := sys.Fcntl(fd, syscall.F_GETFD, 0)
		if err != nil {
			log.Error().Int("fd", fd).Err(err).Msg("Invalid FD")
		}
	}

	if err := syscall.SetNonblock(srcFd, true); err != nil {
		log.Error().Int("fd", srcFd).Err(err).Msg("Error setting nonblock")
	}
	if err := syscall.SetNonblock(dstFd, true); err != nil {
		log.Error().Int("fd", dstFd).Err(err).Msg("Error setting nonblock")
	}

	sqe := ring.GetSQE()
	sqe.PrepareSplice(srcFd, -1, pipe[1], -1, sys.MaxSpliceSize, unix.SPLICE_F_NONBLOCK)
	sqe.SetFlags(liburing.IOSQE_IO_LINK)

	sqe = ring.GetSQE()
	sqe.PrepareSplice(pipe[0], -1, dstFd, -1, sys.MaxSpliceSize, unix.SPLICE_F_NONBLOCK)

	submitted, err := ring.SubmitAndWait(2)
	if err != nil {
		log.Error().Err(err).Msg("Error submitting")
	}
	log.Info().Int("submitted", int(submitted)).Msg("Submitted")

	ring.ForEachCQE(func(cqe *liburing.CompletionQueueEvent) {
		if cqe.Res < 0 {
			log.Error().Str("error", unix.Errno(-cqe.Res).Error()).Msg("CQE")
		}

		log.Info().Int("cqe", int(cqe.Res)).Msg("CQE")
	})
}

I get -9 (bad fd) on my srcFd which is a rio.Dial created connection, I'll keep debugging.

What do you think of this strategy? thread-per-core and "ring-per-core"? With more rewriting, multishot accept and multishot read, it could eventually go full async later.

Also, I know splice does not handle TLS, but it's alright in case, the HTTP/S proxy is just doing tunnel and copying bytes, I don't need TLS support. Let me know about prep_recvsend_multishot, it could be very interesting.

[user451421541757324]

I run into a invalid memory address or nil pointer reference

I get the same error under the same conditions: the code works under low concurrency, but then under stress, it crashes.

[wamshawn]

I run into a invalid memory address or nil pointer reference

I get the same error under the same conditions: the code works under low concurrency, but then under stress, it crashes.

Maybe there is no runtime.Keepalive() in your testcase, when using iouring directly.