Ilpo Viertola$^{1,2}$, Giulio Cengarle$^1$, Gouthaman KV$^1$, Daniel Arteaga$^1$, Lie Lu$^1$, Esa Rahtu$^2$
$^1$Dolby Laboratories, $^2$Tampere University
This work was conducted during an internship at Dolby Laboratories.
Preprint
ArXiv link coming soon
Code
A link to the code repository for generating DegradedMix will be provided here pending legal clearance.
We introduce Spot, Separate, and Enhance (SSE), the first multimodal, user-guided generative model for audio remixing and enhancement. SSE enhances video content by rebalancing the audio, removing unwanted audio sources and reducing reverberation, guided by both video and textual descriptions. To support its training and evaluation, we propose DegradedMix, a new dataset built on the audio remixing benchmark MuddyMix [1]. We also adopt evaluation metrics from generative modeling, which better capture the creative nature of remixing than standard reconstruction-based metrics. SSE outperforms existing baselines in both controllability and remixing quality, as shown by extensive experiments.
Figure 1. Overview of Spot, Separate, and Enhance (SSE) model.
SSE is multimodal user-guided generative network for audio enhancement. Given audio, video, and text, each modality is first encoded with specific encoder networks. Then, audio and visual features are aligned. Text is used to guide the generation process in the diffusion transformer via cross-attention. After inference, the generated latent sequence is decoded into waveform representation. SSE is able to rebalance the audio, remove backround noise, and suppress reverberations.
We conduct a pairwise preference test using UGC. 10 participants took part in the study. Videos are shown in pairs and participants select the one with audio that they prefer more. All the videos used are presented below.
Table 1. Pairwise preference test results for UGC enhancement. Preference rate is the proportion of trials in which the first condition was preferred over the second, with 95% confidence intervals. p-values are from a two-sided binomial test against chance (50%).
| Comparison (A vs. B) | Wins (A-B) | Pref. rate (%) | p-value |
|---|---|---|---|
| SSE-L vs. SSE-S | 68-22 | 75.6 | < 0.001 |
| SSE-L vs. Original audio | 72-18 | 80.0 | < 0.001 |
| SSE-L vs. VisAH [1] | 80-10 | 88.9 | < 0.001 |
| SSE-S vs. Original audio | 47-43 | 52.2 | 0.752 |
| SSE-S vs. VisAH [1] | 57-33 | 63.3 | 0.015 |
| Original audio vs. VisAH [1] | 62-28 | 68.9 | < 0.001 |
Original audio
VisAH [1]
SSE-S
SSE-L
We evaluate perceptual audio quality of the enhanced audios using the MUSHRA protocol (ITU-R BS.1534), a multi-stimulus subjective listening test with hidden reference and anchor. Ten listeners rate the enhanced and baseline outputs alongside a hidden copy of the reference signal and a low-pass filtered anchor, all on a continuous 0–100 quality scale.
Using internal professional quality video dataset with multi-channel audio, we generate *Input audio *****by rebalancing the separate audio stems following MuddyMix [1]. We do not add other degradations to keep the comparison fair, since VisAH was trained using only gain-based audio degradations. The professionally mixed mono audio is used as Reference. To create Anchor, we low-pass filter Reference with 3.5 kHz cut-off frequency.