Proceedings of the International Workshop on Advances in Deep Learning for Image Analysis and Computer Vision (IWADIC 2025)

<Previous Article In Volume
Next Article In Volume>

VAE-based Generative Modeling for Music Audio

Authors
Wantong Zhang1, *
1University of California, Berkeley, USA
*Corresponding author. Email: wtzhang@berkeley.edu
Corresponding Author
Wantong Zhang
Available Online 24 April 2026.
DOI
10.2991/978-94-6239-648-7_101How to use a DOI?
Keywords
Variational autoencoder; β-VAE; Timbre; Log-mel spectrogram; Audio synthesis
Abstract

This project studies unsupervised timbre representation learning using a lightweight convolutional Variational Autoencoder (VAE) trained on log-mel spectrograms of monophonic instrument notes. Timbre is a multidimensional perceptual attribute shaped by harmonic structure, spectral envelope, and transient behavior, and compact latent embeddings of timbre are useful for controllable synthesis and analysis. Building on the VAE framework and the β-VAE objective, a baseline CNN β-VAE is re-implemented and evaluated on the nsynth-mini dataset. Qualitative inspection of reconstructions shows that the baseline captures coarse energy patterns but suffers from harmonic oversmoothing and transient smearing, particularly in high-frequency partials. To address these limitations under constrained computation, two minimal modifications are introduced: U-Net-style skip connections to improve time–frequency detail propagation and a mel-axis gradient difference loss to penalize frequency-domain blurring. Comparative results indicate visibly sharper harmonic stacks and better localized onsets without adversarial training. The report concludes with practical considerations for stable training and directions for extending the approach to richer perceptual objectives and more complex musical textures.

Copyright
© 2026 The Author(s)
Open Access
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Download article (PDF)

<Previous Article In Volume
Next Article In Volume>
Volume Title
Proceedings of the International Workshop on Advances in Deep Learning for Image Analysis and Computer Vision (IWADIC 2025)
Series
Advances in Computer Science Research
Publication Date
24 April 2026
ISBN
978-94-6239-648-7
ISSN
2352-538X
DOI
10.2991/978-94-6239-648-7_101How to use a DOI?
Copyright
© 2026 The Author(s)
Open Access
Open Access This chapter is licensed under the terms of the Creative Commons Attribution-NonCommercial 4.0 International License (http://creativecommons.org/licenses/by-nc/4.0/), which permits any noncommercial use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license and indicate if changes were made.

Cite this article

risenwbib
TY  - CONF
AU  - Wantong Zhang
PY  - 2026
DA  - 2026/04/24
TI  - VAE-based Generative Modeling for Music Audio
BT  - Proceedings of the International Workshop on Advances in Deep Learning for Image Analysis and Computer Vision (IWADIC 2025)
PB  - Atlantis Press
SP  - 943
EP  - 950
SN  - 2352-538X
UR  - https://doi.org/10.2991/978-94-6239-648-7_101
DO  - 10.2991/978-94-6239-648-7_101
ID  - Zhang2026
ER  -