| <— 6_4_Semantic_segmentation.md | Зміст | 6_6_Text_image_representations.md —> |
6.5 Speech recognition
Speech recognition consists of converting a sound sample into a sequence of words. There have been plenty of approaches to this problem historically, but a conceptually simple and recent one proposed by Radford et al. [2022] consists of casting it as a sequence-to-sequence translation and then solving it with a standard attentionbased Transformer, as described in § 5.3.
Their model first converts the sound signal into a spectrogram, which is a one-dimensional series $T×D$, that encodes at every time step a vector of energies in $D$ frequency bands. The associated text is encoded with the BPE tokenizer (see § 3.2).
The spectrogram is processed through a few 1D convolutional layers, and the resulting representation is fed into the encoder of the Transformer. The decoder directly generates a discrete sequence of tokens, that correspond to one of the possible tasks considered during training. Multiple objectives are considered for training: transcription of English or non-English text, translation from any language to English, or detection of non-speech sequences, such as background music or ambient noise.
This approach allows leveraging extremely large data sets that combine multiple types of sound sources with diverse ground truth.
It is noteworthy that even though the ultimate goal of this approach is to produce a translation as deterministic as possible given the input signal, it is formally the sampling of a text distribution conditioned on a sound sample, hence a synthesis process. The decoder is in fact extremely similar to the generative model of § 7.1.