Video Sidechain

Sidechaining is an audio technique consistent in making an audio signal (the control signal) impact how another is processed. One common application of this is sidechain compression, which allows the level of one audio track to trigger a volume reduction on another.

This plugin for DaVinci Resolve brings this concept to the video domain. Specifically, it enables using perceptual lightness measurements of a region of interest in a video to impact how one or more audio tracks are processed in a synchronized manner.

Prerequisites

• DaVinci Resolve Studio. Tested on version 20.
• Python 3 with the numpy package. Tested on Python 3.10.12 with numpy 1.24.2.

Setup

1. Combine the Fuses and Scripts directories with the ones in your DaVinci Resolve extensions folder (tutorials here and here)
2. Restart DaVinci Resolve afterwards so that it makes them available.

Workflow

The overall process can be summarized as follows:

1. Pass along the video region of interest to an AverageLightness tool in Fusion.
2. Run a provided Fusion script to produce per-frame lightness data.
3. Convert lightness data to an audio control signal.
4. Import the generated audio clip into the Resolve timeline and sync it with the original video on its own muted audio track.
5. In Fairlight, use the created track as the sidechain control signal on effects / plugins that support it to impact how other tracks / buses are processed.

A more step-by-step explanation of this process, specifically to implement lightness-based sidechain compression using the tools provided here:

1. Open the relevant clip in the Fusion page of DaVinci Resolve.
2. Select the tool producing the video to analyze (e.g. MediaOut).
3. Right click on it and run Script ⮞ Video Sidechain ⮞ 1 ROI to create helper tools designed to speed up the process of providing a region of interest.
4. Adjust the created mask and make any changes / replace the associated with tracking information so that the new merge tool only contains the visual information of interest. Non relevant regions of the image must be transparent.
5. If the video is not in Rec. 709 color space, add a color space transform tool to make it so before the AverageLightness tool.
6. You can optionally introduce a crop tool before the created AverageLightness to improve the performance of the next step if the mask is known to be constrained to a section of the image.
7. Right click on the created AverageLightness tool, run Script ⮞ Video Sidechain ⮞ 2 Export, choose an output path and export.
8. Run python3 video_sidechain.py path/to/export.json control.wav.
9. Import the generated .wav file back into DaVinci Resolve, align it with the video clip on an unused audio track and group them to help keeping them in sync.
10. Mute the audio track with the generated audio clip.
11. Open the Fairlight page.
12. Select the track or bus you want to trigger based on the new control track and open its "Dynamics" track effect.
13. Activate the compressor, select the new muted control track in the Sidechain section and turn the sidechain on.
14. Set Ratio=Max; Knee=0; Mix=0; Attack=Min; Hold=0; Release=Min. This makes the response immediate.
15. Adjust the compresson threshold and Make Up to tweak its volume when "off" and when "on".
16. Tweak compression settings as needed to adjust the intensity and the effect response.

That setup will make high lightness values make the controlled track to lower in volume. Other behaviors can be achieved by transforming the input to the AverageLightness tool so that brighter values represent other features of the video.

The [video_sidechain.py] script has options to tweak its output, which can be queried with the -h option. Use the --sample-rate option to change the audio format to match your needs. The --normalize option maximizes the difference between low and high volume, to increase the effect.

License

This tool is shared under the GPLv3 license: link.