Small Vocabulary Singing Voice Synthesis using
Neural Network Characterization of Vocal Features
Angelina A. Aquino
Abstract—A project is proposed on the design and implemen-
tation of a system to characterize acoustic and phonemic features
of utterances in sung speech over a limited vocabulary and
range of pitches, to find optimal parameters from the given data
by backpropagation in a neural network, and to subsequently
synthesize song by generating specified word-pitch sequences
within the trained vocabulary and vocal range.
I. I NTRODUCTION
In digital signal processing, acoustic signals and speech
signals are typically characterized independently, using distinct
methods of analysis. However, singing is an audio signal
which consists of both notation, a tonal component, and lyrics,
a linguistic component. The synthesis of a singing voice
therefore requires the capability to analyze and reconstruct
(a) the fundamentals and harmonics which produce a given
musical note, (b) the formant frequencies which differentiate
the phonemes in a given word, and (c) the effect of speaker
characteristics, such as gender, on the formant frequencies [1].
Singing is produced by organic variations in the vocal tract,
creating a signal with continuously changing properties. The
variation in these features over time may be difficult to model
using rational expressions. However, a neural network is able
to analyze large amounts of data and use various computational
layers to formulate accurate models without the need for
a mathematical expression relating input variables [2]. The
modeling of a spoken or sung signal can therefore be made
more natural by characterizing its continuous properties with
a neural network.
II. M ETHODOLOGY
A. Data Collection and Feature Extraction
A set of five test words (‘happy’, ‘birth’, ‘day’, ‘to’, ‘you’)
will be recorded at equal tempo by two groups of speakers at
two specific vocal ranges: ten female speakers, recording each
set of words from C4 to B4 , and ten male speakers, recording
from C3 to B3 .
Recordings will be analyzed using 20-millisecond frames.
For each frame, the current word and phoneme will be
specified. The fundamental frequency will be extracted using
high-time liftering of the signal cepstrum, while the four most
significant formants and their corresponding magnitudes will
be extracted using low-time liftering [3].
B. Parameter Training using Neural Networks
The neural network will have three nodes at the input layer:
the word to be sung, the note at which it is to be sung, and
the gender of the speaker. The output at the final layer will
be the fundamental and formant frequencies and amplitudes
at each short-time window.
The extracted feature sets will be normalized and regular-
ized. These will then be used to train the weights and biases of
the neural network through backpropagation, with a sigmoid
activation function for all nodes [4]. The number of nodes and
hidden layers to be used will be optimized as needed.
C. Synthesis by Concatenation
The outputs of the neural network will be used to produce
short-time segments with varying formant frequencies, which
will be concatenated to produce single words sung at the
desired pitch. Multiple words may then be concatenated to
produce continuous phrases of song.
III. D ELIVERABLES
A. Halfway Point
By the second week of the project, it is expected that record-
ing will have been accomplished for all speakers, features will
have been extracted for all recordings, and a functional (if not
optimized) neural network will be presentable.
B. Final Output
By the end of the project duration, a fully-functional system
is expected, capable of creating audio samples with continuous
phrases of song, with specific word and pitch sequences.
R EFERENCES
[1] J. Kaur and V. Narang, “Variation of pitch and formants in different age
group,” International Journal of Multidisciplinary Research and Modern
Education, vol. 1, no. 1, pp. 517–521, 2015.
[2] V. Maini and S. Sabri, “Machine learning for humans,” 2017. https:
//medium.com/machine-learning-for-humans.
[3] Sakshat-Virtual-Labs, “Cepstral analysis of speech,” 2011. http://iitg.vlab.
co.in/?sub=59&brch=164&sim=615&cnt=1.
[4] I. Goodfellow, Y. Bengio, and A. Courville, Deep Learning. MIT Press,
2016. http://www.deeplearningbook.org.