Anything special I need to do to get RoCC commands to work in a Firesim simulation?

[pepper-lees]

Hi,

Wasn't sure whether it's more appropriate to put this question in Rocket Chip's or Firesim's repo, but I figured Rocket Chip is more appropriate.

For context, I'm working on a RoCC module which aims to add message-driven computation to the Chipyard ecosystem. It's based on ucb-bar/chipyard#1380. It works fine in Verilator, but I am now attempting to run it on a Firesim FPGA simulation running Fedora.

#include <stdio.h>
#include "sage.h"

int main(void) {

  printf("Hello world!\n");

  qPut(1, 200); // This line runs fine (program prints "Done!" and exits if the following lines are commented out

  int i = qPollAll(); // Program halts here
  printf("qpoll result=%d\n", i);

  printf("Done!\n");
}

These are the function definitions of qPut and qPollAll. Note that qPut does not wait for a return value.

static inline void qPut(unsigned long destination, unsigned long data) {
  ROCC_INSTRUCTION_SS(CUSTOM_INSTR, destination, data, FUNCT_QPUT);
}

static inline unsigned int qPollAll() {
  unsigned long result; // 0 or 1
  ROCC_INSTRUCTION_D(CUSTOM_INSTR, result, FUNCT_QPOLL);
  return result;
}

For some reason, it runs fine when calling qPut, i.e. a custom instruction that does not wait for a response; but when it runs a custom instruction that does require a response, like qPollAll, it hangs indefinitely.

After a while, it outputs some call traces.

[root@fedora-riscv ~]# ./test
Hello world!
[  744.782724] rcu: INFO: rcu_sched detected stalls on CPUs/tasks:
[  744.782752] rcu:     3-...0: (5 ticks this GP) idle=b25c/1/0x4000000000000000 softirq=2969/2972 fqs=1928
[  744.782783]  (detected by 2, t=5253 jiffies, g=4569, q=4 ncpus=4)
[  744.782805] Task dump for CPU 3:
[  744.782818] task:test            state:R  running task     stack:0     pid:311   ppid:246    flags:0x00000008
[  744.783121] Call Trace:
[  744.783197] [<ffffffff800036b0>] ret_from_fork+0x0/0xc
[  754.662737] BUG: workqueue lockup - pool cpus=3 node=0 flags=0x0 nice=0 stuck for 30s!
[  754.662794] Showing busy workqueues and worker pools:
[  754.662810] workqueue events_power_efficient: flags=0x80
[  754.662827]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[  754.662850]     pending: do_cache_clean
[  754.662870] workqueue mm_percpu_wq: flags=0x8
[  754.663128]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[  754.663151]     pending: vmstat_update
[  785.382715] BUG: workqueue lockup - pool cpus=3 node=0 flags=0x0 nice=0 stuck for 61s!
[  785.382771] Showing busy workqueues and worker pools:
[  785.382787] workqueue events_power_efficient: flags=0x80
[  785.382803]   pwq 0: cpus=0 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[  785.382827]     pending: do_cache_clean
[  785.382846] workqueue mm_percpu_wq: flags=0x8
[  785.383106]   pwq 6: cpus=3 node=0 flags=0x0 nice=0 active=1/256 refcnt=2
[  785.383128]     pending: vmstat_update

Is there something special I need to do in order for RoCC instructions to work properly? Again, these instructions/functions all work perfectly fine in Verilator.

[jerryz123]

I suspect your system has entered some deadlock condition. Are your custom instructions non-blocking? Are there scenarios you imagine where some custom instruction may become starved for a necessary resource for completion?

[pepper-lees]

Are there scenarios you imagine where some custom instruction may become starved for a necessary resource for completion?

At the moment, the only instruction which can potentially be starved and lead to a deadlock is qWait/qWaitAny, which waits for a message to come in through the NoC before responding. When the instruction comes in, it enters a "waiting for input" state and stalls until a message arrives.

That's what I thought was causing my process to stall, i.e. a programming error leading to qWait being called after a message was sent. But then I tried with qPoll, which is supposed to respond instantly.
The qPoll instruction is hardcoded to send io.resp.valid = true instantly, no matter what.

Are your custom instructions non-blocking?

I was under the impression that RoCC instructions were designed to always be blocking. Doesn't it always stall the processor until io.resp.valid = true - or am I missing/misunderstanding something?

[jerryz123]

I looked through the code... the qPoll instruction responds immediately, so you have a data hazard on the rd field that you aren't handling, since you read and write reg_rd on the same cycle in this case.

[pepper-lees]

Ah, thank you for catching that, I think I see it. Since reg_rd is only used for the qWait command, i suppose I only need to read from it inside the is (sWaitingForInput) block; otherwise, it makes sense to just set io.resp.bits.rd = cmd.bits.inst.rd.

I'll give that a try! I don't really understand how it worked fine in Verilator but not on the real system, though. Could it have been some kind of fluke that it worked in the Verilator simulation? Like where I'm technically doing undefined behavior, and I'm just lucky that it worked in Verilator but it isn't guaranteed to work?

[pepper-lees]

This feels like a hack but I think it's all I need to change - io.resp.bits.rd := Mux(state === sWaitingForInput, reg_rd, cmd.bits.inst.rd)

[pepper-lees]

Thanks, that fixed qPoll!! I'm still running into some snags with the other instructions, but I'll do some more digging before coming back for help.