tensorflow.python.framework.errors_impl.NotFoundError: <username>/.local/lib/python3.6/site-packages/nvtx/plugins/tf/lib/nvtx_ops.cpython-36m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

[xintin]

I am trying to use nvtx Plugin binding for the Python. It is a Linux machine with below config.

1. cmake/3.16.4
2. cuda/10.0.130
3. cudnn/7.4.1.5
4. openmpi/4.0.2
5. Tensorfow 1.14.0
6. Python 3.6.9

I did python3 -m pip install nvtx-plugins and now while importing as import nvtx.plugins.tf as nvtx_tf, I am getting the mentioned error. Tried manually searching for the dynamic file but could not find it.

I am referring https://nvtx-plugins.readthedocs.io/en/latest/templates/examples.html#tensorflow-session-example

[DEKHTIARJonathan]

Could you please add the full error that you obtain. Thanks

[xintin]

>>> import nvtx.plugins.tf
Traceback (most recent call last): File "<stdin>", line 1, in <module> File "/home/username/.local/lib/python3.6/site-packages/nvtx/plugins/tf/__init__.py", line 33, in <module> import nvtx.plugins.tf.ops File "/home/username/.local/lib/python3.6/site-packages/nvtx/plugins/tf/ops.py", line 29, in <module> nvtx_tf_ops = load_library('lib/nvtx_ops' + get_ext_suffix()) File "/home/username/.local/lib/python3.6/site-packages/nvtx/plugins/tf/ext_utils.py", line 54, in load_library library = _load_library.load_op_library(filename) File "/home/username/.local/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library lib_handle = py_tf.TF_LoadLibrary(library_filename) tensorflow.python.framework.errors_impl.NotFoundError: /home/username/.local/lib/python3.6/site-packages/nvtx/plugins/tf/lib/nvtx_ops.cpython-36m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory

If the above helps, Thanks.

[akemisetti]

.

I am also facing the same problem. Here is my stacktrace

Python 3.6.10 |Anaconda, Inc.| (default, Mar 23 2020, 23:13:11)
[GCC 7.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.

import tensorflow as tf
import nvtx.plugins.tf as nvtx_tf
Traceback (most recent call last):
File "", line 1, in
File "/fmrienv/envs/fmrienv/lib/python3.6/site-packages/nvtx/plugins/tf/init.py", line 33, in
import nvtx.plugins.tf.ops
File "/fmrienv/envs/fmrienv/lib/python3.6/site-packages/nvtx/plugins/tf/ops.py", line 29, in
nvtx_tf_ops = load_library('lib/nvtx_ops' + get_ext_suffix())
File "/fmrienv/envs/fmrienv/lib/python3.6/site-packages/nvtx/plugins/tf/ext_utils.py", line 54, in load_library
library = _load_library.load_op_library(filename)
File "/fmrienv/envs/fmrienv/lib/python3.6/site-packages/tensorflow_core/python/framework/load_library.py", line 61, in load_op_library
lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /fmrienv/envs/fmrienv/lib/python3.6/site-packages/nvtx/plugins/tf/lib/nvtx_ops.cpython-36m-x86_64-linux-gnu.so: cannot open shared object file: No such file or directory
--
I have installed the latest version of nvtx_plugins using pip install nvtx_plugins
Name: nvtx-plugins
Version: 0.1.7
Summary: Python bindings for NVTX
Home-page: https://github.com/NVIDIA/nvtx-plugins/
Author: Ahmad Kiswani, Roni Forte, Jonathan Dekhtiar, Yaki Tebeka
Author-email: akiswani@nvidia.com, rforte@nvidia.com, jdekhtiar@nvidia.com, ytebeka@nvidia.com
License: Apache2
Location: /fmrienv/envs/fmrienv/lib/python3.6/site-packages
Requires: wrapt, tensorflow
Required-by:
--
There is no folder called lib in the plugins directory. Here are the contents
fmrienv/envs/fmrienv/lib/python3.6/site-packages/nvtx/plugins/tf ls
base_callbacks.py estimator.py ext_utils.py init.py keras ops.py package_info.py pycache
--
My tf version is 1.15.0

Thanks
Anil

[DEKHTIARJonathan]

@xintin @akemisetti please provide a docker container to reproduce your issue. Unfortunately, I can't reproduce your issue, nor the CI suite we have in place.

[xintin]

@DEKHTIARJonathan,

Sorry for the late comment, I was off this work a bit.

My python version is Python 3.6.9, tf version is 2.3.0 now.
I did pip install nvtx-plugins as mentioned here.

Followed by import nvtx.plugins.tf as nvtx_tf, and I am getting the above-mentioned error. I tried building from the source too but the same error persists. Can you help me here?

Issue: While installing nvtx-plugins, I cannot see lib folder being created under .local/lib/python3.6/site-packages/nvtx/plugins/tf/

Are you able to reproduce the same error?

Thanks,

[jiangyi15]

I got the same problems, I found that in https://github.com/NVIDIA/nvtx-plugins/blob/master/setup_utils.py#L413
there is no 'else' cases. That may be the reason.

[jainmilan]

Hi, I am facing same issue. Do we any solution for this now?