Support for Jetson Orin (NX or Nano)

[WilliamTambellini]

Hi nvidia @tongyuantongyu
1 Could you please confirm that TRTLLM is not supported (anymore? yet?) on Jetson Nano ?
2 Will the '0.12 jetson' branch be maintained?
Best
Ref:
https://github.com/NVIDIA/TensorRT-LLM/tree/v0.12.0-jetson
https://nvidia.github.io/TensorRT-LLM/reference/support-matrix.html

[tongyuantongyu]

Jetson architectures are not supported in the main branch.

@karljang, could you answer, or route this question to someone who knows about Jetson support better?

[karljang]

Hi @WilliamTambellini,
The TRTLLM team is currently concentrating on data center use cases, and edge device support is not included in our roadmap. I’ve had discussions with the leads about this.
cc. @laikhtewari.

[karljang]

Additionally, there’s another effort focused on enabling LLMs on edge devices, and we hope to make it public soon.

[Thor-x86]

Hi Karl,

Thanks for supporting customers who working on edge inference. Just to let you know that optimization on edge is way impactful than data center due to RAM and compute power constraint. TensorRT somehow makes it possible. This is why I chose Nvidia over competitors, and maybe other users has a same reason.

[karljang]

Check out this link: https://github.com/NVIDIA/TensorRT-Edge-LLM! They’ve finally made it!