Cache flush missing after `metal_io_block_write` in `rpmsg_virtio_send_offchannel_raw`

[Sunil-Hegde]

In rpmsg_create_ept when an endpoint message is written in rpmsg_virtio_send_offchannel_raw using metal_io_block_write, there is no cache flush operation carried out which results in other core seeing stale/garbage values.

[arnopo]

Hello @Sunil-Hegde

The flush is down in rpmsg_virtio_enqueue_buffer (https://github.com/OpenAMP/open-amp/blob/main/lib/rpmsg/rpmsg_virtio.c#L151)

Did you detect an issue?

[Sunil-Hegde]

@arnopo thankyou

My bad, I was running this sample on zephyr and I have to manually invalidate before i send the virtqueue_notification like this sys_cache_data_invd_range((void *)SHM_START_ADDR+0x35, 0x40);

Earlier I used a flush but that probably worked because there is a memory barrier inside the flush function.

So I was wondering if invalidate is required once an endpoint is registered on the host device?

[arnopo]

Please could you give more context about the issue.

• memory mapping used
• what is the SHM_START_ADDR+0x35 address
• on which side ( host/ remote) you have an issue, in which direction ( RX/TX)
• in which function have you to add sys_cache_data_invd_range((void *)SHM_START_ADDR+0x35, 0x40); to fix your issue?

Do you have a look around VDEV_STATUS_ADDR that is directly managed by zephyr without cache management ?

[Sunil-Hegde]

Hello, Sorry for the late response.

• memory mapping used is cacheable
• SHM_START_ADDR+0x35 corresponds to RPMSG_NS_EPT_ADDR
• I have issue on the host side in RX. Remote writes data to the shared memory but host still looks at the cache memory.
• I added sys_cache_data_invd_range() right after rpmsg_register_ept() by changing SHM_START_ADDR+0x35 to shm_io->virt + RPMSG_NS_EPT_ADDR
• I also added a flush after VDEV_STATUS_ADDR like this sys_cache_data_flush_range((void*)VDEV_STATUS_ADDR, sizeof(status));

[arnopo]

Hello, Sorry for the late response.

• memory mapping used is cacheable
• SHM_START_ADDR+0x35 corresponds to RPMSG_NS_EPT_ADDR

Seems here that there is a confusion here

The RPMSG_NS_EPT_ADDR is the address of an endpoint. It is not a physical address but an ID on the rpmsg bus. You cannot use it as reference for a cache management

• I have issue on the host side in RX. Remote writes data to the shared memory but host still looks at the cache memory.

-Are you sure that the remote has effectively written the data? or does the remote is blocked on the virtio status waiting VIRTIO_CONFIG_STATUS_DRIVER_OK?

• I added sys_cache_data_invd_range() right after rpmsg_register_ept() by changing SHM_START_ADDR+0x35 to shm_io->virt + RPMSG_NS_EPT_ADDR

• SHM_START_ADDR+0x35 seems to correspond to the vring_0 that contains the message that should be received by the host

• I suppose that if you add sys_cache_data_invd_range(SHM_START_ADDR, 0x100); in receive_message() function it should work, you confirm?

• Could you confirm that the openAMP lib is compibled with VIRTIO_USE_DCACHE enabled on both HOST and remote?

• I also added a flush after VDEV_STATUS_ADDR like this sys_cache_data_flush_range((void*)VDEV_STATUS_ADDR, sizeof(status));

Yes the flush seems missing in the Zephyr sample, and also an invalidate on ipc_virtio_get_status remote side)

[Sunil-Hegde]

Understood, since I saw metal writes I assumed that is the region i should be invalidating.

• I am using M4 core as the remote and it does write to the memory. But when I inspect the same memory addresses via the A53 core, all values are zeroes. I have also checked the VDEV_STATUS_ADDR address and remote is not stuck at VIRTIO_CONFIG_STATUS_DRIVER_OK once the cache flush is done on the A53 core.
• Actually I had modified the sample to only send and receive one message which worked fine with sys_cache_data_invd_range((void *)SHM_START_ADDR+0x35, 0x40);, I tried 0x100, SHM_SIZE/2 and SHM_SIZE, I have to invalidate the whole shared memory region for 100 messages.
• Also, a cache invalidate only inside receive_message() does not work and I assume it is because the host needs to know what is written before ipm callback is triggered?
• Hence I added BUFFER_INVALIDATE((void *)(shm_io->virt), shm_io->size); after rpmsg_register_ept()
• Yes OpenAMP is compiled with VIRTIO_USE_DCACHE enabled on host but I am not enabling anything related to cache on the remote since I dont have cache on M4 core.

[arnopo]

Could you share you code on github , specifying on which board you are running the sample?

[Sunil-Hegde]

these these are the changes I made. I am working on TI Sitara AM62 SK board and was trying to figure out why the sample was not working.

board: sk_am62/am6254/a53 and sk_am62/am6254/m4

[arnopo]

Regarding code in common.h

#define VDEV_START_ADDR		DT_REG_ADDR(DT_CHOSEN(zephyr_ipc_shm))
#define VDEV_SIZE		DT_REG_SIZE(DT_CHOSEN(zephyr_ipc_shm))

#define VDEV_STATUS_ADDR	VDEV_START_ADDR
#define VDEV_STATUS_SIZE	0x400

#define SHM_START_ADDR		(VDEV_START_ADDR + VDEV_STATUS_SIZE)
#define SHM_SIZE		(VDEV_SIZE - VDEV_STATUS_SIZE)
#define SHM_DEVICE_NAME		"sramx.shm"

#define VRING_COUNT		2
#define VRING_RX_ADDRESS	(VDEV_START_ADDR + SHM_SIZE - VDEV_STATUS_SIZE)
#define VRING_TX_ADDRESS	(VDEV_START_ADDR + SHM_SIZE)
#define VRING_ALIGNMENT		4
#define VRING_SIZE		16

If I have well computed the VRING_TX_ADDRESS value:
VRING_TX_ADDRESS = VDEV_START_ADDR + VDEV_SIZE.
This would put the vring 0 is out of the zephyr_ipc_shm memory region
That could explain your issue...

[Sunil-Hegde]

actually VRING_TX_ADDRESS = VDEV_START_ADDR + SHM_SIZE where SHM_SIZE is VDEV_SIZE - 0x400 so this is still inside the range right?

[arnopo]

You are right I confused VDEV_START_ADDR with SHM_START_ADDR
So your mapping is:
0xc9CB00000 0x9CB00400 vdev_status
0x9CB00400 to 0x9CBFF7FF Virtio/RPmsg buffers
0x9CBFF800 to 0x9CBFFBFF vring TX
0x9CBFFC00 to 0x9CBFFFFF Vring Rx

To make it work, if you invalidate from SHM_START_ADDR + 0x35, that is, from address 0x9CB00400, then the issue would seem to be related to the buffers. could you confirm it?

I have no setup to reproduce, so it is quite difficult to me to help you to debug the issue. I can just give you few guidance to help you to debug

you can for instance to add traces in the openamp library to understand at which level the issue occurs?

• check if rpmsg_virtio_rx_callback is called.
  if not called, the issue is likely related to the mailbox.
• check if virtqueue_get_buffer return a buffer
  If no buffer is returned, cache issue likely concerns in the vrings
• check the content of the buffer
  If there is a cache issue on the buffer, the RPMsg header would probably be incorrect, and the message would not be processed.

Actually I had modified the sample to only send and receive one message which worked fine with sys_cache_data_invd_range((void *)SHM_START_ADDR+0x35, 0x40);, I tried 0x100, SHM_SIZE/2 and SHM_SIZE, I have to invalidate the whole shared memory region for 100 messages.

I can not see where you add this in the code you shared

[Sunil-Hegde]

Thank you very much, I will look through the issue more and will update regarding the same

I can not see where you add this in the code you shared

Actually I added a buffer invalidate in rpmsg_virtio.c, I will add it back in the sample itself

[github-actions]

This issue has been marked as a stale issue because it has been open (more than) 45 days with no activity.