Does not work with Tensorflow 1.8

[BourbonCreams]

I get this error when running source build_tools.py

I know you said your code is supported by Tensorflow 1.4 but I thought it would be good for you to know.

[lukszamarcin]

I tested this repo with Tensorflow 1.8 on Jetson TX2 - works fine for me (~20 fps)

[BourbonCreams]

Oh, interesting. Do you know why I get that error then? When I run source build_tools.py a new terminal opens up and closes one second later. It shows what you can see above.

[lukszamarcin]

I'm not sure what you are trying to do so I will summarize what steps I took in order to make it run:
I use Jetson TX2 flashed with Jetpack 3.2 (comes with CUDA 9.0)

• build openCV 3.4.0 (I needed that specific version for other reasons but it should also work with others) - https://www.jetsonhacks.com/2018/05/28/build-opencv-3-4-with-cuda-on-nvidia-jetson-tx2/
• install tensorflow from whl available here: https://devtalk.nvidia.com/default/topic/1031300/tensorflow-1-7-wheel-with-jetpack-3-2-/
• cloned this repo
• run python3 run_objectdetection.py from the repo dir
  Works like a charm

[BourbonCreams]

Hey there. I'm trying to run the code on my Jetson TX2 using the onboard camera. I have flashed it with Jetpack 3.2, installed OpenCV 3.4 and TensorFlow 1.8 but when I run python run_objectdetection.py (I'm using python 2.7) I get this error:

VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
[...]
Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline) in icvStartPipeline, file /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp, line 450 [...]

I set my config.yaml file to receive input from the onboard camera by setting the following parameter:

VIDEO_INPUT: "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)120/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"                 # Input Must be OpenCV readable

Among others, I have installed with OpenCV using the following 3 flags:
WITH_GSTREAMER=ON -D WITH_FFMPEG=ON -D WITH_LIBV4L=ON -D

Any hint about that error?
Thanks

[lukszamarcin]

Can you try replacing your VIDEO_INPUT with the following?
VIDEO_INPUT: "video_input : nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)600, height=(int)600,format=(string)I420, framerate=(fraction)5/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"

[BourbonCreams]

Thanks @xxmarl, unfortunately I get the same error. This is the output:

VIDEOIO ERROR: V4L: device video_input : nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)600, height=(int)600,format=(string)I420, framerate=(fraction)5/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...

Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in icvStartPipeline, file /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp, line 450
Traceback (most recent call last):
  File "run_objectdetection.py", line 180, in <module>
    detection(model, config)
  File "run_objectdetection.py", line 35, in detection
    vs = WebcamVideoStream(config.VIDEO_INPUT,config.WIDTH,config.HEIGHT).start()
  File "/home/nvidia/project/realtime_object_detection/rod/helper.py", line 177, in __init__
    self.stream.set(cv2.CAP_PROP_FRAME_WIDTH, self.width)
cv2.error: /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp:450: error: (-2) GStreamer: unable to start pipeline
 in function icvStartPipeline

[BourbonCreams]

This is the output of cv2.getBuildInformation().

print(cv2.getBuildInformation())

General configuration for OpenCV 3.4.0 =====================================
  Version control:               unknown

  Platform:
    Timestamp:                   2018-06-15T12:19:00Z
    Host:                        Linux 4.4.38-tegra aarch64
    CMake:                       3.5.1
    CMake generator:             Unix Makefiles
    CMake build tool:            /usr/bin/make
    Configuration:               RELEASE

  CPU/HW features:
    Baseline:                    NEON FP16
      required:                  NEON
      disabled:                  VFPV3

  C/C++:
    Built as dynamic libs?:      YES
    C++ Compiler:                /usr/bin/c++  (ver 5.4.0)
    C++ flags (Release):         -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -O3 -DNDEBUG  -DNDEBUG
    C++ flags (Debug):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wundef -Winit-self -Wpointer-arith -Wshadow -Wsign-promo -Wuninitialized -Winit-self -Wno-narrowing -Wno-delete-non-virtual-dtor -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -fvisibility-inlines-hidden -g  -O0 -DDEBUG -D_DEBUG
    C Compiler:                  /usr/bin/cc
    C flags (Release):           -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -O3 -DNDEBUG  -DNDEBUG
    C flags (Debug):             -fsigned-char -W -Wall -Werror=return-type -Werror=non-virtual-dtor -Werror=address -Werror=sequence-point -Wformat -Werror=format-security -Wmissing-declarations -Wmissing-prototypes -Wstrict-prototypes -Wundef -Winit-self -Wpointer-arith -Wshadow -Wuninitialized -Winit-self -Wno-narrowing -Wno-comment -fdiagnostics-show-option -pthread -fomit-frame-pointer -ffunction-sections -fdata-sections    -fvisibility=hidden -g  -O0 -DDEBUG -D_DEBUG
    Linker flags (Release):
    Linker flags (Debug):
    ccache:                      NO
    Precompiled headers:         YES
    Extra dependencies:          dl m pthread rt cudart nppc nppial nppicc nppicom nppidei nppif nppig nppim nppist nppisu nppitc npps cublas cufft -L/usr/local/cuda-9.0/lib64
    3rdparty dependencies:

  OpenCV modules:
    To be built:                 calib3d core cudaarithm cudabgsegm cudacodec cudafeatures2d cudafilters cudaimgproc cudalegacy cudaobjdetect cudaoptflow cudastereo cudawarping cudev dnn features2d flann highgui imgcodecs imgproc ml objdetect photo python2 python_bindings_generator shape stitching superres video videoio videostab
    Disabled:                    java js world
    Disabled by dependency:      -
    Unavailable:                 python3 ts viz
    Applications:                apps
    Documentation:               NO
    Non-free algorithms:         NO

  GUI: 
    GTK+:                        YES (ver 2.24.30)
      GThread :                  YES (ver 2.48.2)
      GtkGlExt:                  NO
    VTK support:                 NO

  Media I/O: 
    ZLib:                        /usr/lib/aarch64-linux-gnu/libz.so (ver 1.2.8)
    JPEG:                        /usr/lib/aarch64-linux-gnu/libjpeg.so (ver )
    WEBP:                        build (ver encoder: 0x020e)
    PNG:                         /usr/lib/aarch64-linux-gnu/libpng.so (ver 1.2.54)
    TIFF:                        /usr/lib/aarch64-linux-gnu/libtiff.so (ver 42 / 4.0.6)
    JPEG 2000:                   /usr/lib/aarch64-linux-gnu/libjasper.so (ver 1.900.1)
    OpenEXR:                     build (ver 1.7.1)

  Video I/O:
    DC1394:                      YES (ver 2.2.4)
    FFMPEG:                      YES
      avcodec:                   YES (ver 56.60.100)
      avformat:                  YES (ver 56.40.101)
      avutil:                    YES (ver 54.31.100)
      swscale:                   YES (ver 3.1.101)
      avresample:                NO
    GStreamer:                   
      base:                      YES (ver 1.8.3)
      video:                     YES (ver 1.8.3)
      app:                       YES (ver 1.8.3)
      riff:                      YES (ver 1.8.3)
      pbutils:                   YES (ver 1.8.3)
    libv4l/libv4l2:              1.10.0 / 1.10.0
    v4l/v4l2:                    linux/videodev2.h
    gPhoto2:                     NO

  Parallel framework:            pthreads

  Trace:                         YES (built-in)

  Other third-party libraries:
    Lapack:                      NO
    Eigen:                       NO
    Custom HAL:                  YES (carotene (ver 0.0.1))

  NVIDIA CUDA:                   YES (ver 9.0, CUFFT CUBLAS)
    NVIDIA GPU arch:             62
    NVIDIA PTX archs:

  Python 2:
    Interpreter:                 /usr/bin/python2.7 (ver 2.7.12)
    Libraries:                   /usr/lib/aarch64-linux-gnu/libpython2.7.so (ver 2.7.12)
    numpy:                       /usr/local/lib/python2.7/dist-packages/numpy/core/include (ver 1.14.3)
    packages path:               lib/python2.7/dist-packages

  Python (for build):            /usr/bin/python2.7

  Java:
    ant:                         /usr/bin/ant (ver 1.9.6)
    JNI:                         /usr/lib/jvm/java-8-oracle/include /usr/lib/jvm/java-8-oracle/include/linux /usr/lib/jvm/java-8-oracle/include
    Java wrappers:               NO
    Java tests:                  NO

  Matlab:                        NO

  Install to:                    /usr/local

In the Video I/O section GStreamer seems to be installed correctly.

[BourbonCreams]

I could run it successfully on my laptop which has OpenCV 3.4, Tensorflow 1.4.0 and an NVIDIA GeForce 950M (it ran at about 3/4 FPS and I was using the laptop's camera).

Any idea on how to make it run on the TX2 given the information above?

[naisy]

Hi @BourbonCreams,

=== About laptop ===
Tensorflow 1.4.0 doesn't support CUDA9.
Perhaps your Tensorflow is running on cpu only. (and build without cpu architecture optimization.)

try:
pip install tensorflow-gpu
on your ubuntu laptop.

=== About TX2 ===
try:
sudo apt-get install libxine2

check onboard cam with code:

import cv2

src = "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"
vid = cv2.VideoCapture(src)

if not vid.isOpened():
    raise IOError(("Couldn't open video file or webcam."))

print("Press 'q' to Exit")
while True:
    retval, cv_bgr = vid.read()
    if not retval:
        break
    cv2.imshow('camera', cv_bgr)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break

vid.release()

config.yml

### USB Webcam on PC ###
#video_input: 0 
### USB Webcam on TX2 ###
#video_input: 1 
### Onboard camera on TX2 ###
video_input: "nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink" 

[BourbonCreams]

Hi @naisy, thank you for your support.

=== About laptop ===
I am not too concerned about my laptop, I would need this model to work on the Jetson so I'm not worried about using my GPU with Tensorflow (you're right, I have it installed without GPU support).

=== About TX2 ===
I installed the library you told me to install and a re-installed OpenCV with the flag -D WITH_XINE=ON but it still does not work. This is the output I get from python run_objectdetection.py:

VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
(ERROR)icvOpenAVI_XINE(): Unable to open source 'nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink'

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...

Socket read error. Camera Daemon stopped functioning.....
gst_nvcamera_open() failed ret=0
OpenCV Error: Unspecified error (GStreamer: unable to start pipeline
) in icvStartPipeline, file /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp, line 450
Traceback (most recent call last):
  File "run_objectdetection.py", line 180, in <module>
    detection(model, config)
  File "run_objectdetection.py", line 35, in detection
    vs = WebcamVideoStream(config.VIDEO_INPUT,config.WIDTH,config.HEIGHT).start()
  File "/home/nvidia/project/realtime_object_detection/rod/helper.py", line 177, in __init__
    self.stream.set(cv2.CAP_PROP_FRAME_WIDTH, self.width)
cv2.error: /home/nvidia/opencv/modules/videoio/src/cap_gstreamer.cpp:450: error: (-2) GStreamer: unable to start pipeline
 in function icvStartPipeline

After running your script the video stream works but on Terminal I still see this cheeky output:

nvidia@tegra-ubuntu:~$ python cameraTrial.py 
VIDEOIO ERROR: V4L: device nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink: Unable to query number of channels
(ERROR)icvOpenAVI_XINE(): Unable to open source 'nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink'

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...

Press 'q' to Exit

The good news is that if I plug in an external webcam and set VIDEO_INPUT:1 in the config.yml file then everything works (I get around 27 fps, which is good). I'm not sure why the onboard camera does not want to work. I have run other algorithms such as Faster R-CNN and YOLO from DarkNet using the onboard camera. For YOLO I also needed to specify the camera with this parameter:
"nvcamerasrc ! video/x-raw(memory:NVMM), width=(int)1280, height=(int)720,format=(string)I420, framerate=(fraction)30/1 ! nvvidconv flip-method=0 ! video/x-raw, format=(string)BGRx ! videoconvert ! video/x-raw, format=(string)BGR ! appsink"
but it won't work for SSD_with_MobileNet. I am happy to run this with the webcam, I don't particularly need the onboard camera, but it would good for future reference to solve this issue so if guys have any other idea I would gladly help you solve this problem with your guidance.

Thanks,
Lorenzo

[naisy]

Hi @BourbonCreams,

This is my log. Something error, but onboard camera worked.
python a

(ERROR)icvOpenAVI_XINE(): Unable to initialize video driver.

Available Sensor modes : 
2592 x 1944 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
2592 x 1458 FR=30.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10
1280 x 720 FR=120.000000 CF=0x1109208a10 SensorModeType=4 CSIPixelBitDepth=10 DynPixelBitDepth=10

NvCameraSrc: Trying To Set Default Camera Resolution. Selected sensorModeIndex = 1 WxH = 2592x1458 FrameRate = 30.000000 ...

Press 'q' to Exit
Could not initialize OpenGL for RasterGLSurface, reverting to RasterSurface.

Maybe, you need to install something package.
I always install these packages for build OpenCV, Tensorflow and pip on TX2.

apt-get install -y python-dev libjpeg-dev libxslt-dev libxml2-dev libffi-dev libcurl4-openssl-dev libssl-dev libblas-dev liblapack-dev gfortran libpng12-dev libfreetype6-dev

apt-get install -y zip curl locate libprotobuf-dev build-essential cmake libeigen3-dev libatlas-base-dev gfortran wget libavformat-dev libavcodec-dev libswscale-dev libavresample-dev ffmpeg pkg-config unzip qtbase5-dev libgtk-3-dev libdc1394-22 libdc1394-22-dev libjpeg-dev libpng12-dev libtiff5-dev libjasper-dev libavcodec-dev libavformat-dev libswscale-dev libxine2-dev libgstreamer1.0-dev libgstreamer-plugins-base1.0-dev libv4l-dev libtbb-dev libfaac-dev libmp3lame-dev libopencore-amrnb-dev libopencore-amrwb-dev libtheora-dev libvorbis-dev libxvidcore-dev v4l-utils liblapacke-dev libopenblas-dev checkinstall libgdal-dev libgphoto2-dev

I have just finished building OpenCV 3.4.1(JetPack3.2.1/Python2.7) which enabled the Tracking API.
Onboard camera worked with this package. (Even JetPack 3.2 should work)
https://github.com/naisy/JetsonTX2/tree/JetPack3.2_python2.7/JetPack3.2/python2.7/binary/
dpkg -i opencv-3.4.1.deb