Wise teachers train better DNN acoustic models
详细信息 查看全文
- 作者：Ryan Price ; Ken-ichi Iso ; Koichi Shinoda
- 关键词：Soft targets ; Deep neural networks ; Online speech recognition ; Speaker ; adaptive features ; Model compression
- 刊名：EURASIP Journal on Audio, Speech, and Music Processing
- 出版年：2016
- 出版时间：December 2016
- 年：2016
- 卷：2016
- 期：1
- 全文大小：1,015 KB
- 刊物主题：Signal, Image and Speech Processing;
- 出版者：Springer International Publishing
- ISSN：1687-4722
文摘
Automatic speech recognition is becoming more ubiquitous as recognition performance improves, capable devices increase in number, and areas of new application open up. Neural network acoustic models that can utilize speaker-adaptive features, have deep and wide layers, or more computationally expensive architectures, for example, often obtain best recognition accuracy but may not be suitable for the given budget of computational and storage resources or latency required by the deployed system. We explore a straightforward training approach which takes advantage of highly accurate but expensive-to-evaluate neural network acoustic models by using their outputs to relabel training examples for easier-to-deploy models. Experiments on a large vocabulary continuous speech recognition task offer relative reductions in word error rate of up to 16.7 % over training with the hard aligned labels by effectively making use of large amounts of additional untranscribed data. Somewhat remarkably, the approach works well even when only two output classes are present. Experiments on a voice activity detection task give relative reductions in equal error rate of up to 11.5 % when using a convolutional neural network to relabel training examples for a feedforward neural network. An investigation into the hidden layer weight matrices finds that soft target-trained networks tend to produce weight matrices having fuller rank and slower decay in singular values than their hard target-trained counterparts, suggesting that more of the network’s capacity is utilized for learning additional information giving better accuracy.