RuntimeError: Exporting the operator grid_sampler to ONNX opset version 9 is not supported

[choozhenbo]

I would like to convert the TPS-resnet-bilstm-ctc model in ONNX format, but shows the error of "Exporting the operator grid_sampler to ONNX opset version 9 is not supported"
This is the command and error messages:
python tools/torch2onnx.py configs/tps_resnet_bilstm_ctc.py /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/vedastr/ckpt/tps_resnet_bilstm_ctc.pth /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/tps_resnet_bilstm_ctc.onnx
2022-06-18 15:54:46,461 - INFO - Use GPU 0
2022-06-18 15:54:46,461 - INFO - Set cudnn deterministic False
2022-06-18 15:54:46,461 - INFO - Set cudnn benchmark True
2022-06-18 15:54:46,461 - INFO - Set seed 1111
2022-06-18 15:54:46,461 - INFO - Build model
2022-06-18 15:54:46,786 - INFO - GResNet init weights
2022-06-18 15:54:46,982 - INFO - CTCHead init weights
2022-06-18 15:54:48,754 - INFO - Load checkpoint from /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/vedastr/ckpt/tps_resnet_bilstm_ctc.pth
Convert to Onnx with constant input shape 3,32,100 and opset version 9
Traceback (most recent call last):
File "tools/torch2onnx.py", line 92, in
main()
File "tools/torch2onnx.py", line 83, in main
do_constant_folding=args.do_constant_folding,
File "/home/tham/vedastr/tools/volksdep/converters/torch2onnx.py", line 62, in torch2onnx
dynamic_axes=dynamic_axes)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/init.py", line 320, in export
custom_opsets, enable_onnx_checker, use_external_data_format)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 111, in export
custom_opsets=custom_opsets, use_external_data_format=use_external_data_format)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 729, in _export
dynamic_axes=dynamic_axes)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 501, in _model_to_graph
module=module)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 216, in _optimize_graph
graph = torch._C._jit_pass_onnx(graph, operator_export_type)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/init.py", line 373, in _run_symbolic_function
return utils._run_symbolic_function(*args, **kwargs)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 1028, in _run_symbolic_function
symbolic_fn = _find_symbolic_in_registry(domain, op_name, opset_version, operator_export_type)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 982, in _find_symbolic_in_registry
return sym_registry.get_registered_op(op_name, domain, opset_version)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/symbolic_registry.py", line 125, in get_registered_op
raise RuntimeError(msg)
RuntimeError: Exporting the operator grid_sampler to ONNX opset version 9 is not supported. Please feel free to request support or submit a pull request on PyTorch GitHub

[choozhenbo]

@hxcai do you know where torch.nn.functional.grid_sample is assigned as TPS is included in the model?

[choozhenbo]

@hxcai I found the supported opset version from https://github.com/onnx/onnx/blob/main/docs/Operators.md . But the python tools/torch2onnx.py configs/tps_resnet_bilstm_ctc.py /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/vedastr/ckpt/tps_resnet_bilstm_ctc.pth /home/tham/Desktop/convert/text-recognition-tps-resnet-bilstm-ctc/tps_resnet_bilstm_ctc.onnx --opset_version=16 command shows the error:
Convert to Onnx with constant input shape 3,32,100 and opset version 16
Traceback (most recent call last):
File "tools/torch2onnx.py", line 92, in
main()
File "tools/torch2onnx.py", line 83, in main
do_constant_folding=args.do_constant_folding,
File "/home/tham/vedastr/tools/volksdep/converters/torch2onnx.py", line 62, in torch2onnx
dynamic_axes=dynamic_axes)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/init.py", line 320, in export
custom_opsets, enable_onnx_checker, use_external_data_format)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 111, in export
custom_opsets=custom_opsets, use_external_data_format=use_external_data_format)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/utils.py", line 707, in _export
_set_opset_version(opset_version)
File "/home/tham/anaconda3/envs/vedastr/lib/python3.6/site-packages/torch/onnx/symbolic_helper.py", line 849, in _set_opset_version
raise ValueError("Unsupported ONNX opset version: " + str(opset_version))
ValueError: Unsupported ONNX opset version: 16

[hxcai]

@choozhenbo The onnx opset version 16 is not supported by pytorch yet, if you want to support grid_sample, you should implement the operation by yourself.

[choozhenbo]

@hxcai do you have clue how to implement the operation? I tried to trace the location for the implementation of the operation but ended up I have no clue.

[hxcai]

@choozhenbo you can go to this issue to find the solution. You need to add onnx and trt plugin.