gen2-gaze-estimation crashes/failures (after working once)

[ramonbroox]

After working thrice, interspersed with some random failures, now the demo does not work.
$ python3 main.py -cam

Upgraded DepthAI SDK to: depthai-2.20.2.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
Python 3.8.10 (default, Nov 14 2022, 12:59:47) [GCC 9.4.0] on linux
Ubuntu 20.04.5 LTS

Result 1:

Creating pipeline...
Creating Color Camera...
Creating Face Detection Neural Network...
Creating Landmarks Detection Neural Network...
Creating Head Pose Neural Network...
Creating Gaze Estimation Neural Network...
Traceback (most recent call last):
  File "main.py", line 385, in <module>
    with depthai.Device(create_pipeline()) as device:
depthai.XLinkWriteError: Couldn't write data to stream: '__rpc_main' (X_LINK_ERROR)

Result 2:

Creating pipeline...
Creating Color Camera...
Creating Face Detection Neural Network...
Creating Landmarks Detection Neural Network...
Creating Head Pose Neural Network...
Creating Gaze Estimation Neural Network...
terminate called without an active exception
Stack trace (most recent call last):
#29   Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in 
#28   Object "python3", at 0x5faa2d, in _start
#27   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f4b23db4082, in __libc_start_main
#26   Object "python3", at 0x6b840c, in Py_BytesMain
#25   Object "python3", at 0x6b8081, in Py_RunMain
#24   Object "python3", at 0x67fbf6, in PyRun_SimpleFileExFlags
#23   Object "python3", at 0x67dad0, in 
#22   Object "python3", at 0x67da2e, in 
#21   Object "python3", at 0x67d9b0, in 
#20   Object "python3", at 0x68e266, in PyEval_EvalCode
#19   Object "python3", at 0x569d89, in _PyEval_EvalCodeWithName
#18   Object "python3", at 0x571142, in _PyEval_EvalFrameDefault
#17   Object "python3", at 0x5f6705, in _PyObject_MakeTpCall
#16   Object "/home/mon/.local/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7f4b0071e728, in 
#15   Object "python3", at 0x5a6ca6, in 
#14   Object "python3", at 0x59c543, in 
#13   Object "python3", at 0x5f52b1, in PyObject_Call
#12   Object "python3", at 0x50ba57, in 
#11   Object "python3", at 0x5f6705, in _PyObject_MakeTpCall
#10   Object "python3", at 0x5f5b38, in PyCFunction_Call
#9    Object "/home/mon/.local/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7f4b0071f02a, in 
#8    Object "/home/mon/.local/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7f4b0075a1b1, in 
#7    Object "/home/mon/.local/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7f4b00929aad, in dai::Device::Device(dai::Pipeline const&, dai::DeviceInfo const&)
#6    Object "/home/mon/.local/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7f4b00935a31, in dai::DeviceBase::tryStartPipeline(dai::Pipeline const&)
#5    Object "/home/mon/.local/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7f4b0093fde5, in dai::DeviceBase::startPipelineImpl(dai::Pipeline const&)
#4    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f4b226723f6, in std::terminate()
#3    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f4b2267238b, in 
#2    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7f4b22666910, in 
#1    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f4b23db2858, in abort
#0    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7f4b23dd300b, in gsignal
Aborted (Signal sent by tkill() 51192 1000)
Aborted (core dumped)

[ramonbroox]

Looks like an exception thrown at startPipelineImpl

(gdb) bt
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50
#1  0x00007ffff7dd5859 in __GI_abort () at abort.c:79
#2  0x00007ffff6689911 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff669538c in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff66953f7 in std::terminate() ()
   from /lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007fffd495ebd6 in dai::DeviceBase::startPipelineImpl(dai::Pipeline const&) ()
   from /home/mon/.local/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so
#6  0x00007fffd4954822 in dai::DeviceBase::tryStartPipeline(dai::Pipeline const&) ()

[ramonbroox]

Now, setting DEPTHAI_LEVEL=debug has started working again 10 or so times until I've had it crash again. x-D

python3 main.py -cam -laz
[2023-02-08 16:46:43.261] [debug] Python bindings - version: 2.13.3.0 from 2021-12-01 03:26:17 +0200 build: 2021-12-01 03:15:55 +0000
[2023-02-08 16:46:43.262] [debug] Library information - version: 2.13.3, commit: 0cc6284767ae21e8747d26562cb058d181e0ca34 from 2021-12-01 03:23:55 +0200, build: 2021-12-01 03:08:42 +0000
[2023-02-08 16:46:43.273] [debug] Initialize - finished
Creating pipeline...
Creating Color Camera...
Creating Face Detection Neural Network...
Creating Landmarks Detection Neural Network...
Creating Head Pose Neural Network...
Creating Gaze Estimation Neural Network...
[2023-02-08 16:46:43.351] [debug] Resources - Archive 'depthai-bootloader-fwp-0.0.15.tar.xz' open: 3ms, archive read: 76ms
^[[A[2023-02-08 16:46:43.618] [debug] Resources - Archive 'depthai-device-fwp-48fca4a443d841221c94c70d5c05e7e946168636.tar.xz' open: 2ms, archive read: 353ms
[2023-02-08 16:46:43.954] [debug] Device - OpenVINO version: 2021.4
[1844301071567C0F00] [5.434] [system] [info] Memory Usage - DDR: 0.12 / 358.55 MiB, CMX: 2.05 / 2.50 MiB, LeonOS Heap: 6.72 / 78.59 MiB, LeonRT Heap: 2.88 / 23.84 MiB
[1844301071567C0F00] [5.434] [system] [info] Temperatures - Average: 33.39 °C, CSS: 34.16 °C, MSS 32.97 °C, UPA: 32.73 °C, DSS: 33.69 °C
[1844301071567C0F00] [5.434] [system] [info] Cpu Usage - LeonOS 6.82%, LeonRT: 1.61%
[2023-02-08 16:46:46.542] [debug] Schema dump: {"connections":[{"node1Id":10,"node1Output":"out","node2Id":12,"node2Input":"in"},{"node1Id":11,"node1Output":"out","node2Id":10,"node2Input":"in"},{"node1Id":7,"node1Output":"out","node2Id":9,"node2Input":"in"},{"node1Id":8,"node1Output":"out","node2Id":7,"node2Input":"in"},{"node1Id":4,"node1Output":"out","node2Id":6,"node2Input":"in"},{"node1Id":5,"node1Output":"out","node2Id":4,"node2Input":"in"},{"node1Id":2,"node1Output":"out","node2Id":3,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":2,"node2Input":"in"},{"node1Id":0,"node1Output":"preview","node2Id":1,"node2Input":"in"}],"globalProperties":{"calibData":null,"cameraTuningBlobSize":null,"cameraTuningBlobUri":"","leonCssFrequencyHz":700000000.0,"leonMssFrequencyHz":700000000.0,"pipelineName":null,"pipelineVersion":null,"xlinkChunkSize":-1},"nodes":[[0,{"id":0,"ioInfo":{"inputConfig":{"blocking":false,"name":"inputConfig","queueSize":8,"type":3},"inputControl":{"blocking":true,"name":"inputControl","queueSize":8,"type":3},"isp":{"blocking":false,"name":"isp","queueSize":8,"type":0},"preview":{"blocking":false,"name":"preview","queueSize":8,"type":0},"raw":{"blocking":false,"name":"raw","queueSize":8,"type":0},"still":{"blocking":false,"name":"still","queueSize":8,"type":0},"video":{"blocking":false,"name":"video","queueSize":8,"type":0}},"name":"ColorCamera","properties":{"boardSocket":0,"colorOrder":0,"fp16":false,"fps":30.0,"imageOrientation":-1,"initialControl":{"aeLockMode":false,"aeRegion":{"height":0,"priority":0,"width":0,"x":0,"y":0},"afRegion":{"height":0,"priority":0,"width":0,"x":0,"y":0},"antiBandingMode":0,"autoFocusMode":3,"awbLockMode":false,"awbMode":0,"brightness":0,"chromaDenoise":0,"cmdMask":0,"contrast":0,"effectMode":0,"expCompensation":0,"expManual":{"exposureTimeUs":0,"frameDurationUs":0,"sensitivityIso":0},"lensPosition":0,"lumaDenoise":0,"saturation":0,"sceneMode":0,"sharpness":0,"wbColorTemp":0},"inputConfigSync":false,"interleaved":false,"ispScale":{"horizDenominator":0,"horizNumerator":0,"vertDenominator":0,"vertNumerator":0},"previewHeight":300,"previewKeepAspectRatio":true,"previewWidth":300,"resolution":0,"sensorCropX":-1.0,"sensorCropY":-1.0,"stillHeight":-1,"stillWidth":-1,"videoHeight":-1,"videoWidth":-1}}],[1,{"id":1,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"cam_out"}}],[2,{"id":2,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":5,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0},"passthrough":{"blocking":false,"name":"passthrough","queueSize":8,"type":0}},"name":"NeuralNetwork","properties":{"blobSize":1344256,"blobUri":"asset:__blob","numFrames":8,"numNCEPerThread":0,"numThreads":0}}],[3,{"id":3,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"face_nn"}}],[4,{"id":4,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":5,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0},"passthrough":{"blocking":false,"name":"passthrough","queueSize":8,"type":0}},"name":"NeuralNetwork","properties":{"blobSize":532736,"blobUri":"asset:__blob","numFrames":8,"numNCEPerThread":0,"numThreads":0}}],[5,{"id":5,"ioInfo":{"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"XLinkIn","properties":{"maxDataSize":5242880,"numFrames":8,"streamName":"landmark_in"}}],[6,{"id":6,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"landmark_nn"}}],[7,{"id":7,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":5,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0},"passthrough":{"blocking":false,"name":"passthrough","queueSize":8,"type":0}},"name":"NeuralNetwork","properties":{"blobSize":3839808,"blobUri":"asset:__blob","numFrames":8,"numNCEPerThread":0,"numThreads":0}}],[8,{"id":8,"ioInfo":{"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"XLinkIn","properties":{"maxDataSize":5242880,"numFrames":8,"streamName":"pose_in"}}],[9,{"id":9,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"pose_nn"}}],[10,{"id":10,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":5,"type":3},"out":{"blocking":false,"name":"out","queueSize":8,"type":0},"passthrough":{"blocking":false,"name":"passthrough","queueSize":8,"type":0}},"name":"NeuralNetwork","properties":{"blobSize":3784320,"blobUri":"asset:__blob","numFrames":8,"numNCEPerThread":0,"numThreads":0}}],[11,{"id":11,"ioInfo":{"out":{"blocking":false,"name":"out","queueSize":8,"type":0}},"name":"XLinkIn","properties":{"maxDataSize":5242880,"numFrames":8,"streamName":"gaze_in"}}],[12,{"id":12,"ioInfo":{"in":{"blocking":true,"name":"in","queueSize":8,"type":3}},"name":"XLinkOut","properties":{"maxFpsLimit":-1.0,"metadataOnly":false,"streamName":"gaze_nn"}}]]}
[2023-02-08 16:46:46.542] [debug] Asset map dump: {"map":{"/node/10/__blob":{"alignment":64,"offset":0,"size":3784320},"/node/2/__blob":{"alignment":64,"offset":8156864,"size":1344256},"/node/4/__blob":{"alignment":64,"offset":7624128,"size":532736},"/node/7/__blob":{"alignment":64,"offset":3784320,"size":3839808}}}
terminate called without an active exception
Stack trace (most recent call last):
[2023-02-08 16:46:48.081] [debug] Log thread exception caught: Couldn't read data from stream: '__log' (X_LINK_ERROR)
[2023-02-08 16:46:48.081] [debug] Timesync thread exception caught: Couldn't read data from stream: '__timesync' (X_LINK_ERROR)
#29   Object "[0xffffffffffffffff]", at 0xffffffffffffffff, in 
#28   Object "python3", at 0x5faa2d, in _start
#27   Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fdee8eef082, in __libc_start_main
#26   Object "python3", at 0x6b840c, in Py_BytesMain
#25   Object "python3", at 0x6b8081, in Py_RunMain
#24   Object "python3", at 0x67fbf6, in PyRun_SimpleFileExFlags
#23   Object "python3", at 0x67dad0, in 
#22   Object "python3", at 0x67da2e, in 
#21   Object "python3", at 0x67d9b0, in 
#20   Object "python3", at 0x68e266, in PyEval_EvalCode
#19   Object "python3", at 0x569d89, in _PyEval_EvalCodeWithName
#18   Object "python3", at 0x571142, in _PyEval_EvalFrameDefault
#17   Object "python3", at 0x5f6705, in _PyObject_MakeTpCall
#16   Object "/home/mon/Luxonis/venv/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7fdec72c8ff8, in 
#15   Object "python3", at 0x5a6ca6, in 
#14   Object "python3", at 0x59c543, in 
#13   Object "python3", at 0x5f52b1, in PyObject_Call
#12   Object "python3", at 0x50ba57, in 
#11   Object "python3", at 0x5f6705, in _PyObject_MakeTpCall
#10   Object "python3", at 0x5f5b38, in PyCFunction_Call
#9    Object "/home/mon/Luxonis/venv/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7fdec72ca6b6, in 
#8    Object "/home/mon/Luxonis/venv/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7fdec72ecb94, in 
#7    Object "/home/mon/Luxonis/venv/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7fdec742c14a, in dai::Device::Device(dai::Pipeline const&, dai::DeviceInfo const&)
#6    Object "/home/mon/Luxonis/venv/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7fdec7435d71, in dai::DeviceBase::tryStartPipeline(dai::Pipeline const&)
#5    Object "/home/mon/Luxonis/venv/lib/python3.8/site-packages/depthai.cpython-38-x86_64-linux-gnu.so", at 0x7fdec743af83, in dai::DeviceBase::startPipelineImpl(dai::Pipeline const&)
#4    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7fdec90313f6, in std::terminate()
#3    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7fdec903138b, in 
#2    Object "/lib/x86_64-linux-gnu/libstdc++.so.6", at 0x7fdec9025910, in 
#1    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fdee8eed858, in abort
#0    Object "/lib/x86_64-linux-gnu/libc.so.6", at 0x7fdee8f0e00b, in gsignal
Aborted (Signal sent by tkill() 14741 1000)
Aborted (core dumped)