Official implementation of the EUSIPCO 2025 paper "Soft Disentanglement in Frequency Bands for Neural Audio Codecs" by Benoît Giniès, Xiaoyu Bie, Olivier Fercoq and Gaël Richard (LTCI, Télécom Paris, Institut polytechnique de Paris).
🎧 Audio examples: https://soft-disentangled-codec.github.io/
This code is built on the Descript Audio Codec (DAC) implementation (Kumar et al., High-Fidelity Audio Compression with Improved RVQGAN, NeurIPS 2023). The encoder, decoder, residual vector quantizer, discriminator, losses and the training loop are adapted from DAC's code (MIT License). We thank the authors for releasing it.
The codec is a cascade of DAC-like branches, each operating at its own sampling rate. In the paper we use two branches, at 16 kHz and 32 kHz:
- the 16 kHz branch encodes S16kHz and outputs d̂16kHz, with Ŝ16kHz = d̂16kHz (the [0 – 8 kHz] band);
- the 32 kHz branch encodes the residual S32kHz − U(d̂16kHz), where U is a sinc upsampling, and Ŝ32kHz = U(d̂16kHz) + d̂32kHz (the [0 – 16 kHz] band).
The tokens of each branch therefore mainly carry the information of its own frequency band, while
nothing prevents the 32 kHz branch from correcting the lower band (soft disentanglement).
Each branch has 4 codebooks of 1024 entries at 50 Hz: 2 kbps for the 16 kHz output and 4 kbps in total for the 32 kHz output.
The implementation (softdis_codec.SoftDisentangledCodec) supports any number of branches.
Training is done in cascade: (1) the 16 kHz branch is trained alone, (2) it is frozen and the 32 kHz branch is trained, (3) both branches are finetuned jointly, the reconstruction losses being averaged over branches and the VQ losses summed (Eq. 3 of the paper).
softdis_codec/
model/codec.py # SoftDisentangledCodec: the multi-branch codec
model/discriminator.py # one DAC discriminator (MPD + MRD) per branch
nn/ # layers, residual vector quantizer and losses (from DAC)
conf/
softdis_codec_16-32/ # configurations of the three training stages of the paper model
dac_baselines/ # configurations of the DAC baselines (16 kHz / 2 kbps, 32 kHz / 4 kbps)
scripts/
train.py # cascade training (one call per stage)
save_test_set.py # exports the test excerpts at every branch rate
reconstruct.py # encodes/decodes audio (and inpainting experiment)
evaluate.py # mel/STFT/waveform losses, SI-SDR, ViSQOL (Tables I and V)
band_sdr.py # SDR per frequency band and spectrograms (Tables III, IV and Fig. 2)
git clone https://github.com/Soft-Disentangled-Codec/Soft-Disentangled-Codec.git
cd Soft-Disentangled-Codec
pip install -r requirements.txt # versions used in the paper (Python 3.11)
pip install -e .ViSQOL (only needed by scripts/evaluate.py) has to be built from source with its Python bindings,
following the instructions of https://github.com/google/visqol. Without it, evaluate.py skips this metric.
All commands below are run from the root of the repository.
We train on MUSDB18 (train set mixtures) and MTG-Jamendo (audio), and validate/test on the 50 mixtures of the MUSDB18 test set. The data loader reads every audio file (wav, flac, mp3, ...) found recursively in the configured folders and resamples it to the rate of each branch, so only the mixtures of MUSDB18 must be visible. With MUSDB18-HQ, for instance:
for split in train test; do
mkdir -p data/musdb18/$split
for track in /path/to/musdb18hq/$split/*/; do
ln -s "$track/mixture.wav" "data/musdb18/$split/$(basename "$track").wav"
done
done
ln -s /path/to/mtg-jamendo/audio data/mtg-jamendoThe data paths are set at the end of conf/softdis_codec_16-32/base.yml
(train/build_dataset.folders, val/..., test/...); edit them if your data is elsewhere.
The three stages are run one after the other, with the same save_path (default runs/softdis_codec_16-32):
export CUDA_VISIBLE_DEVICES=0
python scripts/train.py --args.load conf/softdis_codec_16-32/stage1_16k.yml # 16 kHz branch
python scripts/train.py --args.load conf/softdis_codec_16-32/stage2_32k.yml # 32 kHz branch (16 kHz frozen)
python scripts/train.py --args.load conf/softdis_codec_16-32/stage3_finetune.yml # joint finetuningEach stage starts from the best checkpoint of the previous one (best_16000, then best_32000), selected
on the validation mel loss. Checkpoints are written to runs/softdis_codec_16-32/{latest,best_<stage>,<stage>_<N>k}/,
and TensorBoard logs (losses and audio samples) to runs/softdis_codec_16-32/<stage>/logs.
The configurations reproduce the paper setting (22k, 44k and 30k iterations with batch sizes 64, 32 and 16,
on a single GPU). Any entry can be overridden from the command line, e.g. --batch_size 16 --save_path runs/test.
Multi-GPU training uses torchrun, as in DAC:
torchrun --nproc_per_node gpu scripts/train.py --args.load conf/softdis_codec_16-32/stage1_16k.ymlThe baselines of the paper are the original DAC model retrained on the same data at 16 kHz (2 kbps)
and 32 kHz (4 kbps). They are trained with the official DAC repository and the configurations of
conf/dac_baselines:
git clone https://github.com/descriptinc/descript-audio-codec.git && cd descript-audio-codec
pip install -e ".[dev]"
python scripts/train.py --args.load ../Soft-Disentangled-Codec/conf/dac_baselines/dac_16khz_2kbps.yml --save_path runs/dac_16khz
python scripts/train.py --args.load ../Soft-Disentangled-Codec/conf/dac_baselines/dac_32khz_4kbps.yml --save_path runs/dac_32khz(adapt the data paths of these configurations to the location of your data).
import torch
from audiotools import AudioSignal
from softdis_codec import SoftDisentangledCodec
# From a training checkpoint folder ...
model, _ = SoftDisentangledCodec.load_from_folder("runs/softdis_codec_16-32/best_finetune", package=False)
# ... or from a weights file
# model = SoftDisentangledCodec.load("softdis_codec_16-32.pth")
model.eval()
signal = AudioSignal("input.wav").to_mono()
x = [signal.clone().resample(sr).audio_data for sr in model.sample_rates] # the input at 16 and 32 kHz
with torch.no_grad():
out = model(x)
s_16, s_32 = out["audio"] # reconstructions Ŝ_16kHz and Ŝ_32kHz
d_16, d_32 = out["branch_audio"] # branch outputs d̂_16kHz and d̂_32kHz
codes_16, codes_32 = out["codes"] # tokens of each branch, [batch x 4 x frames]
# Decoding from the tokens
s_16, s_32 = model.decode([codes_16, codes_32], lengths=[a.shape[-1] for a in x])
# Decoding the 16 kHz band only
(s_16,) = model.decode([codes_16])1. Test set. Export the 1000 test excerpts of 10 s (drawn from the MUSDB18 test set) at 16 and 32 kHz:
python scripts/save_test_set.py --args.load conf/softdis_codec_16-32/base.yml --output samples/testThis writes samples/test/16000/sample_XXXX.wav and samples/test/32000/sample_XXXX.wav.
2. Reconstructions. Encode and decode the test set (samples/recons/{16000,32000}/), and run the
inpainting experiment of Section VI, where the 32 kHz branch receives U(S16kHz) instead of S32kHz
(samples/inpainting/{16000,32000}/):
python scripts/reconstruct.py --checkpoint runs/softdis_codec_16-32/best_finetune --input samples/test --output samples/recons
python scripts/reconstruct.py --checkpoint runs/softdis_codec_16-32/best_finetune --input samples/test --output samples/inpainting --inpaintingreconstruct.py also accepts any folder of audio files, which are then resampled to the rate of each branch.
3. Reconstruction metrics (Table I).
python scripts/evaluate.py --reference samples/test/16000 --estimate samples/recons/16000
python scripts/evaluate.py --reference samples/test/32000 --estimate samples/recons/32000The mean ± std of each metric is printed and per-file values are saved to <estimate>/metrics.csv.
For the DAC baselines, generate the reconstructions with scripts/get_samples.py of the DAC repository
(--input samples/test/16000 or samples/test/32000) and evaluate them the same way.
4. Disentanglement: SDR per frequency band (Table III) and spectrograms (Fig. 2).
python scripts/band_sdr.py --reference samples/test --recons samples/recons --figures 5This compares d̂32kHz = Ŝ32kHz − U(Ŝ16kHz) to S32kHz in the [0 – 8 kHz] and [8 – 16 kHz]
bands, and Ŝ32kHz to S32kHz around the 8 kHz cut-off ([7.9 – 8.1 kHz]) and over the whole band.
The SDR is computed on STFT magnitudes (Hann window of 2048 samples, hop of 512) bin by bin, then averaged
over the bins of the band and over the files. Figures are saved to samples/recons/figures/.
5. Inpainting (Tables IV and V).
# Table IV: SDR per band of the 32 kHz branch output
python scripts/band_sdr.py --reference samples/test --recons samples/inpainting
# Table V: U(Ŝ_16kHz), U∘D(Ŝ^inp_32kHz) and Ŝ^inp_32kHz compared to S_32kHz
python scripts/evaluate.py --reference samples/test/32000 --estimate samples/inpainting/16000 --csv samples/inpainting/u_s16.csv
python scripts/evaluate.py --reference samples/test/32000 --estimate samples/inpainting/32000 --through_sample_rate 16000 --csv samples/inpainting/ud_s32.csv
python scripts/evaluate.py --reference samples/test/32000 --estimate samples/inpainting/32000 --csv samples/inpainting/s32.csvThe MUSHRA listening test (Table II) was run with webMUSHRA and is not part of this repository.
@inproceedings{ginies2025soft,
title = {Soft Disentanglement in Frequency Bands for Neural Audio Codecs},
author = {Gini{\`e}s, Beno{\^\i}t and Bie, Xiaoyu and Fercoq, Olivier and Richard, Ga{\"e}l},
booktitle = {Proc. European Signal Processing Conference (EUSIPCO)},
pages = {11--15},
year = {2025}
}Please also cite DAC, on which this code is built:
@inproceedings{kumar2023high,
title = {High-Fidelity Audio Compression with Improved {RVQGAN}},
author = {Kumar, Rithesh and Seetharaman, Prem and Luebs, Alejandro and Kumar, Ishaan and Kumar, Kundan},
booktitle = {Advances in Neural Information Processing Systems (NeurIPS)},
year = {2023}
}This code is built on the Descript Audio Codec implementation and uses audiotools and julius (sinc resampling). This work was funded by the European Union (ERC, HI-Audio, 101052978). Views and opinions expressed are however those of the author(s) only and do not necessarily reflect those of the European Union or the European Research Council. Neither the European Union nor the granting authority can be held responsible for them.
MIT (see LICENSE), including the parts adapted from DAC (Copyright (c) 2023-present, Descript).