diff --git a/cpp/uvc/AGENTS.md b/cpp/uvc/AGENTS.md index aa751f4f8..a506fdeb7 100644 --- a/cpp/uvc/AGENTS.md +++ b/cpp/uvc/AGENTS.md @@ -9,6 +9,7 @@ This is the standalone-only C++ reference for turning an RVC4 device into a USB - You need the OAK device to appear as a standard USB camera on a host machine. - You want a standalone-only C++ app that runs fully on RVC4. - You need a reference for combining a DepthAI pipeline with the `uvc-gadget` framework. +- You need host UVC clients to adjust camera settings such as exposure mode, exposure time, brightness, or gain. - You want to study the packaging and runtime requirements for configfs- and `/dev`-heavy OAK apps. ## Do Not Use This Example When @@ -29,7 +30,8 @@ This is the standalone-only C++ reference for turning an RVC4 device into a USB - `Runs on:` RVC4 standalone only - `Requires:` RVC4 device; configfs and `/dev` access; `uvc-gadget` submodule content; USB gadget-capable runtime environment - `Input:` one detected camera stream from the connected device camera list -- `Output:` MJPEG UVC stream over USB +- `Output:` continuous MJPEG UVC video stream plus MJPEG still-image responses over USB +- `Controls:` standard UVC auto-exposure mode, exposure time, brightness, and gain; optional extension-unit control for max ISO - `Models:` none - `Visualizer / UI:` none; the host sees a USB UVC camera @@ -37,19 +39,22 @@ This is the standalone-only C++ reference for turning an RVC4 device into a USB - [src/uvc_example.cpp](src/uvc_example.cpp): full pipeline, UVC callback wiring, and stream lifecycle - [src/uvc_example.hpp](src/uvc_example.hpp): UVC gadget header bridge +- [src/uvc_controls.cpp](src/uvc_controls.cpp): UVC control registration and mapping into `dai::CameraControl` +- [src/uvc_controls.hpp](src/uvc_controls.hpp): control registration interface shared with the main app - [CMakeLists.txt](CMakeLists.txt): C++ build and `uvc-gadget` linkage - [oakapp.toml](oakapp.toml): device permissions, mounts, build steps, and packaged runtime contract -- [uvc-start.sh](uvc-start.sh): configfs gadget setup, bind/unbind behavior, restart loop, and shutdown handling +- [uvc-start.sh](uvc-start.sh): configfs gadget setup, control and extension-unit advertisement, bind/unbind behavior, restart loop, and shutdown handling - [README.md](README.md): standalone run instructions ## Architecture -- The app builds the `uvc-gadget` submodule and the local `uvc_example` binary. -- [uvc-start.sh](uvc-start.sh) configures the USB gadget through configfs, binds the UDC, and launches `/app/uvc_example`. +- The app builds the `uvc-gadget` submodule, the local `uvc_example` binary, and the `src/uvc_controls.cpp` control bridge. +- [uvc-start.sh](uvc-start.sh) configures the USB gadget through configfs, advertises camera-terminal and processing-unit controls, optionally creates an extension unit, binds the UDC, and launches `/app/uvc_example`. - The C++ app registers a buffer callback for the UVC gadget runtime. - A `dai::Device` and `dai::Pipeline` are created on-device. - The first detected connected camera is used as the video source. -- The camera output is encoded as MJPEG and exposed to the UVC buffer callback through an output queue. +- The camera creates a 1920x1080 MJPEG path for continuous UVC video and a 3840x2160 MJPEG path for still-image requests. +- `registerExampleControls(...)` binds host UVC control writes to `dai::CameraControl` messages on the camera input control queue. - UVC stream on/off events call back into `depthai_control_pipeline_cb(...)`, which starts or stops camera streaming via `CameraControl`. ## Data Flow @@ -57,17 +62,20 @@ This is the standalone-only C++ reference for turning an RVC4 device into a USB - `device camera -> Camera node -> NV12 output -> MJPEG VideoEncoder -> outputQueue` - `outputQueue -> depthai_uvc_get_buffer(...) -> UVC gadget buffers -> host USB UVC stream` - `UVC stream events -> depthai_control_pipeline_cb(...) -> camera inputControl queue` +- `host UVC control write -> registerExampleControls(...) handlers -> camera inputControl queue` ## Modification Guide -- `Safe to change:` output resolution, encoder profile, UVC gadget strings, restart-loop behavior, selected camera source -- `Requires care:` configfs gadget structure, UDC bind/unbind sequencing, allowed device mounts, C/C++ linkage with `uvc-gadget`, stream-on/stream-off control semantics -- `Likely to break if changed blindly:` USB gadget configuration, callback timing, library copy steps in [oakapp.toml](oakapp.toml), or runtime access to `/dev` and `/sys/kernel/config` +- `Safe to change:` output resolution, still-image resolution, encoder profiles, UVC gadget strings, control defaults/ranges, restart-loop behavior, selected camera source +- `Requires care:` configfs gadget structure, UDC bind/unbind sequencing, advertised control selectors and extension-unit layout, allowed device mounts, C/C++ linkage with `uvc-gadget`, stream-on/stream-off control semantics +- `Likely to break if changed blindly:` USB gadget configuration, callback timing, keeping [uvc-start.sh](uvc-start.sh) and [src/uvc_controls.cpp](src/uvc_controls.cpp) in sync, library copy steps in [oakapp.toml](oakapp.toml), or runtime access to `/dev` and `/sys/kernel/config` ## Common Adaptations - `To change the exported UVC mode:` edit `create_frame` usage in [uvc-start.sh](uvc-start.sh) - `To change the camera resolution or format:` edit the `requestOutput(...)` and encoder setup in [src/uvc_example.cpp](src/uvc_example.cpp) +- `To change still-image behavior:` edit the still encoder path in [src/uvc_example.cpp](src/uvc_example.cpp) and the advertised still dimensions in [uvc-start.sh](uvc-start.sh) +- `To add or remove host-visible controls:` update the configfs-advertised `bmControls` and extension-unit setup in [uvc-start.sh](uvc-start.sh) together with the handlers in [src/uvc_controls.cpp](src/uvc_controls.cpp) - `To support a different camera selection policy:` replace `device->getConnectedCameras()[0]` logic in [src/uvc_example.cpp](src/uvc_example.cpp) - `To strip this back to a Visualizer-based C++ baseline:` compare against [cpp/camera_stream](https://github.com/luxonis/oak-examples/tree/main/cpp/camera_stream) @@ -76,6 +84,7 @@ This is the standalone-only C++ reference for turning an RVC4 device into a USB - This example is RVC4 standalone only. - It depends on the `uvc-gadget` submodule being present and buildable. - The runtime needs privileged-ish access patterns: writable configfs, `/dev` mounted into the container, and allowed device access. +- UVC control support depends on kernel configfs attributes being present; the extension-unit control is optional and is skipped when the kernel does not expose the needed extension path. - The code currently selects the first connected camera only. - The app pauses the DepthAI pipeline until the host starts the UVC stream. @@ -83,8 +92,9 @@ This is the standalone-only C++ reference for turning an RVC4 device into a USB - [uvc-start.sh](uvc-start.sh) is the real runtime entrypoint; [src/uvc_example.cpp](src/uvc_example.cpp) is only one part of the standalone behavior. - The gadget setup is torn down and rebound on stop or restart, so gadget lifecycle is part of normal operation here. +- [uvc-start.sh](uvc-start.sh) and [src/uvc_controls.cpp](src/uvc_controls.cpp) are a matched pair: the shell script advertises controls in configfs, and the C++ file implements what those controls do. - [oakapp.toml](oakapp.toml) copies shared libraries into `/usr/lib` manually after build; changing the build layout may require updating those copy steps. -- `VideoSaver` exists in [src/uvc_example.cpp](src/uvc_example.cpp) but is not part of the active pipeline path. +- [oakapp.toml](oakapp.toml) builds against the `luxonis/depthai-library` image, so build/runtime assumptions should be checked there before changing dependencies. ## Related Examples diff --git a/cpp/uvc/CMakeLists.txt b/cpp/uvc/CMakeLists.txt index c7c5e5acb..c94c2352f 100644 --- a/cpp/uvc/CMakeLists.txt +++ b/cpp/uvc/CMakeLists.txt @@ -13,6 +13,7 @@ find_package(depthai REQUIRED) message(STATUS "Found depthai: ${depthai_DIR}") add_executable(uvc_example + src/uvc_controls.cpp src/uvc_example.cpp ) diff --git a/cpp/uvc/oakapp.toml b/cpp/uvc/oakapp.toml index 1bd516f31..8c0064f74 100644 --- a/cpp/uvc/oakapp.toml +++ b/cpp/uvc/oakapp.toml @@ -5,7 +5,7 @@ # Application metadata identifier = "com.example.streaming.uvc" -app_version = "3.0.0" +app_version = "4.0.0" # Command to run when the container starts entrypoint = ["bash", "-c", "/app/uvc-start.sh start"] @@ -51,6 +51,6 @@ api_url = "https://registry-1.docker.io" service = "registry.docker.io" oauth_url = "https://auth.docker.io/token" auth_type = "repository" -auth_name = "luxonis/oakapp-base" -image_name = "luxonis/oakapp-base" -image_tag = "1.2.6-cpp" +auth_name = "luxonis/depthai-library" +image_name = "luxonis/depthai-library" +image_tag = "f9fb5eae82c60509f996c1e9b3adf5cb95965e98" diff --git a/cpp/uvc/src/uvc_controls.cpp b/cpp/uvc/src/uvc_controls.cpp new file mode 100644 index 000000000..41ddc7744 --- /dev/null +++ b/cpp/uvc/src/uvc_controls.cpp @@ -0,0 +1,256 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright (c) 2026 Luxonis, Inc. + * + * Contact: + */ + +#include +#include +#include +#include +#include +#include + +#include "depthai/depthai.hpp" +#include "uvc_controls.hpp" + +extern "C" { +#include "uvcgadget/control.h" +#include +} + +struct uvc_control gExposureModeControl{}; +struct uvc_control gExposurePriorityControl{}; +struct uvc_control gExposureTimeAbsoluteControl{}; +struct uvc_control gBrightnessControl{}; +struct uvc_control gGainControl{}; +struct uvc_control gExtensionControl{}; + +std::shared_ptr gInputQueue{nullptr}; + +constexpr uint8_t kUvcAeModeManual = 0x01; +constexpr uint8_t kUvcAeModeAuto = 0x02; +constexpr uint8_t kUvcAeModeShutterPriority = 0x04; +constexpr uint8_t kUvcAeModeAperturePriority = 0x08; +constexpr uint32_t kDefaultExposureTimeUs = 20000; +constexpr uint32_t kDefaultSensitivityIso = 800; +constexpr int16_t kDepthaiBrightnessMin = -10; +constexpr int16_t kDepthaiBrightnessMax = 10; +constexpr int16_t kDepthaiGainMin = 1; +constexpr int16_t kDepthaiGainMax = 1600; + +uint8_t gAeMode = kUvcAeModeAperturePriority; +uint8_t gAePriority = 0; +uint32_t gExposureTimeUs = kDefaultExposureTimeUs; +uint32_t gSensitivityIso = kDefaultSensitivityIso; + +int sendDepthaiControl(const std::shared_ptr& ctrl, const char* reason) { + if(gInputQueue == nullptr) { + std::cerr << "DepthAI control queue is not ready for " << reason << "." << std::endl; + return -EAGAIN; + } + + gInputQueue->send(ctrl); + return 0; +} + +template +int readControlValue(const uint8_t* data, uint16_t size, T& value) { + if(size != sizeof(T)) { + return -EINVAL; + } + + std::memcpy(&value, data, sizeof(T)); + return 0; +} + +int uvcSetAeModeControl(struct uvc_stream*, + const struct uvc_control*, + const uint8_t* data, + uint16_t size, + void*) { + uint8_t aeMode = 0; + const int ret = readControlValue(data, size, aeMode); + if(ret < 0) { + return ret; + } + if (gAeMode == aeMode) { + // No mode change, so no need to send a control to DepthAI + return 0; + } + + switch(aeMode) { + case kUvcAeModeManual: { + auto ctrl = std::make_shared(); + gAeMode = aeMode; + ctrl->setManualExposure(gExposureTimeUs, gSensitivityIso); + + return sendDepthaiControl(ctrl, "manual exposure mode"); + } + case kUvcAeModeAuto: + case kUvcAeModeShutterPriority: + case kUvcAeModeAperturePriority: { + auto ctrl = std::make_shared(); + gAeMode = aeMode; + ctrl->setAutoExposureEnable(); + + return sendDepthaiControl(ctrl, "auto exposure mode"); + } + default: + return -EOPNOTSUPP; + } +} + +int uvcSetExposureTimeAbsoluteControl(struct uvc_stream*, + const struct uvc_control*, + const uint8_t* data, + uint16_t size, + void*) { + uint32_t exposureTime100us = 0; + const int ret = readControlValue(data, size, exposureTime100us); + if(ret < 0) { + return ret; + } + + gExposureTimeUs = std::max(1u, exposureTime100us * 100u); + std::cout << "UVC camera terminal: exposure time absolute set to " + << exposureTime100us << " (100 us units), mapped to " + << gExposureTimeUs << " us" << std::endl; + + std::cout << "gAeMode = 0x" << std::hex << static_cast(gAeMode) << std::dec << std::endl; + auto ctrl = std::make_shared(); + if(gAeMode == kUvcAeModeManual || gAeMode == kUvcAeModeShutterPriority) { + ctrl->setManualExposure(gExposureTimeUs, gSensitivityIso); + } else { + ctrl->setAutoExposureEnable(); + } + + return sendDepthaiControl(ctrl, "absolute exposure time"); +} + +int uvcSetBrightnessControl(struct uvc_stream*, + const struct uvc_control*, + const uint8_t* data, + uint16_t size, + void*) { + int16_t brightness = 0; + const int ret = readControlValue(data, size, brightness); + if(ret < 0) { + return ret; + } + + brightness = std::clamp(brightness, kDepthaiBrightnessMin, kDepthaiBrightnessMax); + std::cout << "UVC processing unit: brightness set to " << brightness << std::endl; + + auto ctrl = std::make_shared(); + ctrl->setBrightness(static_cast(brightness)); + return sendDepthaiControl(ctrl, "brightness"); +} + +int uvcSetGainControl(struct uvc_stream*, + const struct uvc_control*, + const uint8_t* data, + uint16_t size, + void*) { + int16_t gain = 0; + const int ret = readControlValue(data, size, gain); + if(ret < 0) { + return ret; + } + + gSensitivityIso = std::clamp(gain, kDepthaiGainMin, kDepthaiGainMax); + std::cout << "UVC processing unit: gain set to " << gain << std::endl; + + auto ctrl = std::make_shared(); + ctrl->setManualExposure(gExposureTimeUs, gSensitivityIso); + return sendDepthaiControl(ctrl, "gain"); +} + +int uvcSetExtensionControl(struct uvc_stream*, + const struct uvc_control*, + const uint8_t* data, + uint16_t size, + void*) { + uint32_t value = 0; + const int ret = readControlValue(data, size, value); + if(ret < 0) { + return ret; + } + + auto ctrl = std::make_shared(); + ctrl->setAutoExposureMaxISO(value); + sendDepthaiControl(ctrl, "max-iso"); + + std::cout << "UVC extension unit: value set to 0x" + << std::hex << static_cast(value) << std::dec << std::endl; + return 0; +} + +void cleanupExampleControls() { + uvc_control_deinit(&gExposureModeControl); + uvc_control_deinit(&gExposurePriorityControl); + uvc_control_deinit(&gExposureTimeAbsoluteControl); + uvc_control_deinit(&gBrightnessControl); + uvc_control_deinit(&gExtensionControl); + gInputQueue.reset(); +} + +int registerExampleControls(struct uvc_stream* stream, std::shared_ptr inputQueue) { + static const struct uvc_control_ops aeModeOps{nullptr, uvcSetAeModeControl}; + static const struct uvc_control_ops exposureTimeAbsoluteOps{nullptr, uvcSetExposureTimeAbsoluteControl}; + static const struct uvc_control_ops brightnessOps{nullptr, uvcSetBrightnessControl}; + static const struct uvc_control_ops gainOps{nullptr, uvcSetGainControl}; + static const struct uvc_control_ops extensionOps{nullptr, uvcSetExtensionControl}; + + gInputQueue = inputQueue; + + int ret = uvc_stream_register_control_uint8( + stream, + &gExposureModeControl, + UVC_CONTROL_SECTION_CAMERA_TERMINAL, + UVC_CT_AE_MODE_CONTROL, + kUvcAeModeAuto, 0x00, 0x00, kUvcAeModeManual | kUvcAeModeAuto, kUvcAeModeAuto, + &aeModeOps + ); + ret |= uvc_stream_register_control_uint32( + stream, + &gExposureTimeAbsoluteControl, + UVC_CONTROL_SECTION_CAMERA_TERMINAL, + UVC_CT_EXPOSURE_TIME_ABSOLUTE_CONTROL, + 20000, 1, 20000, 1, 20000, + &exposureTimeAbsoluteOps + ); + ret |= uvc_stream_register_control_int16( + stream, + &gBrightnessControl, + UVC_CONTROL_SECTION_PROCESSING_UNIT, + UVC_PU_BRIGHTNESS_CONTROL, + 0, kDepthaiBrightnessMin, kDepthaiBrightnessMax, 1, 0, + &brightnessOps + ); + ret |= uvc_stream_register_control_int16( + stream, + &gGainControl, + UVC_CONTROL_SECTION_PROCESSING_UNIT, + UVC_PU_GAIN_CONTROL, + kDefaultSensitivityIso, kDepthaiGainMin, kDepthaiGainMax, 1, kDefaultSensitivityIso, + &gainOps + ); + ret |= uvc_stream_register_control_uint32( + stream, + &gExtensionControl, + UVC_CONTROL_SECTION_EXTENSION_UNIT, + 1, + 6000, 50, 6000, 50, 6000, + &extensionOps + ); + + if(ret == -ENOENT) { + std::cout << "No extension unit ID found in configfs. Skipping extension control." << std::endl; + } else if(ret < 0) { + return ret; + } + + return 0; +} diff --git a/cpp/uvc/src/uvc_controls.hpp b/cpp/uvc/src/uvc_controls.hpp new file mode 100644 index 000000000..16efdf5f5 --- /dev/null +++ b/cpp/uvc/src/uvc_controls.hpp @@ -0,0 +1,19 @@ +/* SPDX-License-Identifier: MIT */ +/* + * Copyright (c) 2026 Luxonis, Inc. + * + * Contact: + */ + +#pragma once + +#include + +#include "depthai/depthai.hpp" + +extern "C" { +#include "uvcgadget/stream.h" +} + +int registerExampleControls(struct uvc_stream* stream, std::shared_ptr inputQueue); +void cleanupExampleControls(); diff --git a/cpp/uvc/src/uvc_example.cpp b/cpp/uvc/src/uvc_example.cpp index 92c85caf3..8218b63c7 100644 --- a/cpp/uvc/src/uvc_example.cpp +++ b/cpp/uvc/src/uvc_example.cpp @@ -15,6 +15,7 @@ #include "depthai/pipeline/MessageQueue.hpp" #include "depthai/pipeline/datatype/ImgFrame.hpp" #include "depthai/pipeline/datatype/MessageGroup.hpp" +#include "uvc_controls.hpp" #include "uvc_example.hpp" extern "C" { @@ -128,23 +129,6 @@ int main() { return 1; } - stream = uvc_stream_new(fc->video); - if (!stream) { - std::cerr << "Failed to create UVC stream." << std::endl; - video_source_destroy(src); - return 1; - } - - /* Register the callback to control the pipeline on UVC events: - * - UVC_EVENT_STREAMON - * - UVC_EVENT_STREAMOFF - * - UVC_EVENT_DISCONNECT - */ - uvc_events_register_cb(stream, depthai_control_pipeline_cb); - - uvc_stream_set_event_handler(stream, &events); - uvc_stream_set_video_source(stream, src); - // Create device std::shared_ptr device = std::make_shared(); @@ -173,15 +157,36 @@ int main() { stillSource->link(stillEncoder->input); stillBitstreamQueue = stillEncoder->bitstream.createOutputQueue(1, false); + stream = uvc_stream_new(fc->video); + if (!stream) { + std::cerr << "Failed to create UVC stream." << std::endl; + video_source_destroy(src); + return 1; + } + + /* Register the callback to control the pipeline on UVC events: + * - UVC_EVENT_STREAMON + * - UVC_EVENT_STREAMOFF + * - UVC_EVENT_DISCONNECT + */ + uvc_events_register_cb(stream, depthai_control_pipeline_cb); + + uvc_stream_set_event_handler(stream, &events); + uvc_stream_set_video_source(stream, src); + uvc_stream_init_uvc(stream, fc); + + const int controlRet = registerExampleControls(stream, inputQueue); + if(controlRet < 0) { + std::cerr << "Failed to register example UVC controls: " + << std::strerror(-controlRet) << " (" << -controlRet << ")." << std::endl; + } + // Start pipeline pipeline.start(); std::cout << "Started the pipeline" << std::endl; std::cout << "Press Ctrl+C to stop" << std::endl; depthai_control_pipeline_cb(0); // Pause the pipeline until UVC stream is started by the host - /* Register the UVC events and init it */ - uvc_stream_init_uvc(stream, fc); - /* Main capture loop */ events_loop(&events); @@ -190,6 +195,7 @@ int main() { pipeline.wait(); uvc_stream_delete(stream); + cleanupExampleControls(); video_source_destroy(src); events_cleanup(&events); configfs_free_uvc_function(fc); diff --git a/cpp/uvc/uvc-gadget b/cpp/uvc/uvc-gadget index a2280d0b8..eb9cb7727 160000 --- a/cpp/uvc/uvc-gadget +++ b/cpp/uvc/uvc-gadget @@ -1 +1 @@ -Subproject commit a2280d0b829302361fae6899b2a6017d4d9f80a6 +Subproject commit eb9cb772703cb42d885487142410c22091f39c09 diff --git a/cpp/uvc/uvc-start.sh b/cpp/uvc/uvc-start.sh index eaec06fe5..1d242af82 100644 --- a/cpp/uvc/uvc-start.sh +++ b/cpp/uvc/uvc-start.sh @@ -118,7 +118,7 @@ create_uvc() { mkdir "functions/$FUNCTION" # write to fule function_name PRODUCT var - echo "Luxonis UVC Camera" > "functions/$FUNCTION/function_name" + echo $PRODUCT > "functions/$FUNCTION/function_name" # create_frame "$FUNCTION" 640 360 uncompressed u # create_frame "$FUNCTION" 1280 720 uncompressed u @@ -151,6 +151,49 @@ create_uvc() { ln -s header/h class/ss cd ../../../ + if [ -w "functions/$FUNCTION/control/terminal/camera/default/bmControls" ]; then + # Advertise: + # - Auto-Exposure Mode (selector 0x02) + # - Auto-Exposure Priority (selector 0x03) + # - Exposure Time Absolute (selector 0x04) + # - Exposure Time Relative (selector 0x05) + echo '0x1e' > "functions/$FUNCTION/control/terminal/camera/default/bmControls" + else + log " Camera terminal bmControls attribute not present, using kernel defaults" + fi + + if [ -w "functions/$FUNCTION/control/processing/default/bmControls" ]; then + # Advertise Brightness and gain + cat << EOF > functions/$FUNCTION/control/processing/default/bmControls +1 +2 +EOF + else + log " Processing unit bmControls attribute not present, using kernel defaults" + fi + + # Include an Extension Unit if the kernel supports that + if [ -d functions/$FUNCTION/control/extensions ]; then + mkdir functions/$FUNCTION/control/extensions/xu.0 + pushd functions/$FUNCTION/control/extensions/xu.0 + + # Set the bUnitID of the Processing Unit as the XU's source + echo 2 > baSourceID + + # Set this XU as the source for the default output terminal + cat bUnitID > ../../terminal/output/default/bSourceID + + # Flag some arbitrary controls. This sets alternating bits of the + # first byte of bmControls active. + echo 0x01 > bmControls + echo 1 > bNumControls + + # Set the GUID + echo -e -n "\x73\xa1\x9a\xdc\xb9\x77\x45\xa6\x96\x34\xdc\xbf\x2a\x7a\x8d\x2b" > guidExtensionCode + + popd + fi + echo 3072 > "functions/$FUNCTION/streaming_maxpacket" ln -s "functions/$FUNCTION" configs/c.1