HIERARCHICAL MULTITASK LEARNING WITH CTC

Ramon Sanabria and Florian Metze

Carnegie Mellon UniversityLanguage Technologies Institute, School of Computer Science

Pittsburgh, PA; U.S.A.{ramons|fmetze}@cs.cmu.edu

ABSTRACTIn Automatic Speech Recognition, it is still challenging to learn

useful intermediate representations when using high-level (or ab-stract) target units such as words. For that reason, when only afew hundreds of hours of training data are available, character orphoneme-based systems tend to outperform word-based systems. Inthis paper, we show how Hierarchical Multitask Learning can en-courage the formation of useful intermediate representations. Weachieve this by performing Connectionist Temporal Classificationat different levels of the network with targets of different granular-ity. Our model thus performs predictions in multiple scales for thesame input. On the standard 300h Switchboard training setup, ourhierarchical multitask architecture demonstrates improvements oversingletask architectures with the same number of parameters. Ourmodel obtains 14.0% Word Error Rate on the Switchboard subset ofthe Eval2000 test set without any decoder or language model, outper-forming the current state-of-the-art on non-autoregressive Acoustic-to-Word models.

Index Terms— hierarchical multitask learning, ASR, CTC

1. INTRODUCTION

Automatic Speech Recognition (ASR) systems, traditionally, werecomposed of three elements: a phoneme-based Acoustic Model(AM), a word-based Language Model (LM), and a pronunciationlexicon that maps a sequence of phonemes to words [1]. This setupwas predominating for systems based on Gaussian Mixture Models(GMMs) and Hidden Markov Models (HMMs) [2]. More recently,systems that used a single loss-function based on the predicted se-quence (i.e., end-to-end models) showed similar performance byusing characters as target units. More specifically, these approachesuse either a Sequence-to-Sequence (S2S) architecture [3] or a Con-nectionist Temporal Classification (CTC) loss-function [4]. By notrequiring a pronunciation lexicon anymore, they simplify the ASRpipeline and obtain competitive results.

To simplify even more the ASR pipeline, Soltau et al. suggestusing words instead of characters as target units [5]. Even thoughthis technique forces a closed vocabulary setting, experiments showcompetitive results on a large training corpus. Audhkhasi et al. testthe same approach on Switchboard, a smaller dataset, and show thatthe model can only predict reliably words that frequently appear inthe training set [6]. Concerning this issue, multiple approaches useintermediate level target units (between characters and words) suchas subwords [7] that provide a more balanced solution.

In this work, we introduce a new Hierarchical Multitask Learn-ing architecture (HMTL) (see Fig. 1) that maintains the modular-ity of traditional models by using a Multitask Learning (MTL) [8]

BiLSTM

…

CTC Loss

BiLSTM n<latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="NtnmDLJ1NgRz18QxoCqIYjLd6rc=">AAAB3XicbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy5bsA9oh5JJ77SxmcyQ3BHK0F/gxoUi/i13/hvTx0JbDwQ+zknIvSfKlLTk+9/e1vbO7t5+6aB8WDk6PqmeVto2zY3AlkhVaroRt6ikxhZJUtjNDPIkUtiJJvfzvPOMxspUP9I0wzDhIy1jKTg5q6kH1Zpf9xdimxCsoAYrNQbVr/4wFXmCmoTi1vYCP6Ow4IakUDgr93OLGRcTPsKeQ80TtGGxGHTGLp0zZHFq3NHEFu7vFwVPrJ0mkbuZcBrb9Wxu/pf1copvw0LqLCfUYvlRnCtGKZtvzYbSoCA1dcCFkW5WJsbccEGum7IrIVhfeRPa1/XArwdNH0pwDhdwBQHcwB08QANaIADhBd7g3XvyXr2PZV1b3qq3M/gj7/MHwtmLnw==</latexit><latexit sha1_base64="NtnmDLJ1NgRz18QxoCqIYjLd6rc=">AAAB3XicbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy5bsA9oh5JJ77SxmcyQ3BHK0F/gxoUi/i13/hvTx0JbDwQ+zknIvSfKlLTk+9/e1vbO7t5+6aB8WDk6PqmeVto2zY3AlkhVaroRt6ikxhZJUtjNDPIkUtiJJvfzvPOMxspUP9I0wzDhIy1jKTg5q6kH1Zpf9xdimxCsoAYrNQbVr/4wFXmCmoTi1vYCP6Ow4IakUDgr93OLGRcTPsKeQ80TtGGxGHTGLp0zZHFq3NHEFu7vFwVPrJ0mkbuZcBrb9Wxu/pf1copvw0LqLCfUYvlRnCtGKZtvzYbSoCA1dcCFkW5WJsbccEGum7IrIVhfeRPa1/XArwdNH0pwDhdwBQHcwB08QANaIADhBd7g3XvyXr2PZV1b3qq3M/gj7/MHwtmLnw==</latexit><latexit sha1_base64="ck1/v/JpOJELH0iIdgPbhaWErXY=">AAAB6HicbVBNT8JAEJ3iF+IX6tHLRmLiibRe9Ej04hESCyTQkO0yhZXtttndmpCGX+DFg8Z49Sd589+4QA8KvmSSl/dmMjMvTAXXxnW/ndLG5tb2Tnm3srd/cHhUPT5p6yRTDH2WiER1Q6pRcIm+4UZgN1VI41BgJ5zczf3OEyrNE/lgpikGMR1JHnFGjZVaclCtuXV3AbJOvILUoEBzUP3qDxOWxSgNE1TrnuemJsipMpwJnFX6mcaUsgkdYc9SSWPUQb44dEYurDIkUaJsSUMW6u+JnMZaT+PQdsbUjPWqNxf/83qZiW6CnMs0MyjZclGUCWISMv+aDLlCZsTUEsoUt7cSNqaKMmOzqdgQvNWX10n7qu65da/l1hq3RRxlOINzuAQPrqEB99AEHxggPMMrvDmPzovz7nwsW0tOMXMKf+B8/gDV0Yzu</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit>

1<latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="2yIKfhykAKv0MhUqDiS4SywkL1Y=">AAAB3XicbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy5bsA9oh5JJ77SxmcyQ3BHK0F/gxoUi/i13/hvTx0JbDwQ+zknIvSfKlLTk+9/e1vbO7t5+6aB8WDk6PqmeVto2zY3AlkhVaroRt6ikxhZJUtjNDPIkUtiJJvfzvPOMxspUP9I0wzDhIy1jKTg5qxkMqjW/7i/ENiFYQQ1WagyqX/1hKvIENQnFre0FfkZhwQ1JoXBW7ucWMy4mfIQ9h5onaMNiMeiMXTpnyOLUuKOJLdzfLwqeWDtNIncz4TS269nc/C/r5RTfhoXUWU6oxfKjOFeMUjbfmg2lQUFq6oALI92sTIy54YJcN2VXQrC+8ia0r+uBXw+aPpTgHC7gCgK4gTt4gAa0QADCC7zBu/fkvXofy7q2vFVvZ/BH3ucPaQSLYg==</latexit><latexit sha1_base64="2yIKfhykAKv0MhUqDiS4SywkL1Y=">AAAB3XicbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy5bsA9oh5JJ77SxmcyQ3BHK0F/gxoUi/i13/hvTx0JbDwQ+zknIvSfKlLTk+9/e1vbO7t5+6aB8WDk6PqmeVto2zY3AlkhVaroRt6ikxhZJUtjNDPIkUtiJJvfzvPOMxspUP9I0wzDhIy1jKTg5qxkMqjW/7i/ENiFYQQ1WagyqX/1hKvIENQnFre0FfkZhwQ1JoXBW7ucWMy4mfIQ9h5onaMNiMeiMXTpnyOLUuKOJLdzfLwqeWDtNIncz4TS269nc/C/r5RTfhoXUWU6oxfKjOFeMUjbfmg2lQUFq6oALI92sTIy54YJcN2VXQrC+8ia0r+uBXw+aPpTgHC7gCgK4gTt4gAa0QADCC7zBu/fkvXofy7q2vFVvZ/BH3ucPaQSLYg==</latexit><latexit sha1_base64="1enTr8x+dPZYKeA2su7IqvnUdLk=">AAAB6HicbVBNT8JAEJ3iF+IX6tHLRmLiibRe9Ej04hESCyTQkO0yhZXtttndmpCGX+DFg8Z49Sd589+4QA8KvmSSl/dmMjMvTAXXxnW/ndLG5tb2Tnm3srd/cHhUPT5p6yRTDH2WiER1Q6pRcIm+4UZgN1VI41BgJ5zczf3OEyrNE/lgpikGMR1JHnFGjZVa3qBac+vuAmSdeAWpQYHmoPrVHyYsi1EaJqjWPc9NTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90EOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7EheKsvr5P2Vd1z617LrTVuizjKcAbncAkeXEMD7qEJPjBAeIZXeHMenRfn3flYtpacYuYU/sD5/AF5XYyx</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit>

Char

PRO

J

BiLS

TM

BiLSTM

Subword10k

PRO

J

BiLSTM

Task

-spe

cific

C

TC L

oss

BiLSTM

BiLSTM

BiLS

TM

Subword1k

PRO

J

Task

-spe

cific

C

TC L

oss

Task-specific C

TC Loss

PRO

J

BiLSTM

Subword300

Task-specific C

TC Loss

Shared Encoder

Speech Features

Fig. 1. Our Hierarchical Multitask Learning (HMTL) Model learnsto recognize word-level units with a cascade of fine-to-coarse gran-ularity CTC auxiliary objectives, progressing from recognizingcharacter-level units to almost word-level units at the Subword10KCTC loss.

strategy without giving up the simplicity of end-to-end approaches.Our method exploits the inherent compositionality of different unitrepresentations. By providing auxiliary loss-functions, we hypoth-esize that the network is forced to learn useful intermediate repre-sentations. The experimental observation that each module in a deepneural network architecture learns (hierarchically) specific represen-tations of the input signal inspired the design of this model.

Our experiments show that this technique outperforms compa-rable singletask learning architectures (STL) and traditional MTLtechniques. We also establish the current state-of-the-art on non-autoregressive acoustic-to-word models (A2W) [9] on the 300 hoursSwitchboard subset without exhaustive hyper-parameter optimiza-tion.

2. RELATED WORK

During the last decade of development on end-to-end speech recog-nition systems [10, 11], the ASR community started paying moreattention to different types of target units. In this direction, Soltau etal. propose a CTC model that uses words as target units without any

Published in the 2018 IEEE Workshop on Spoken Language Technology (SLT 2018), Athens, Greece c© IEEE 2018

arX

iv:1

807.

0710

4v5

[cs

.CL

] 1

4 Ja

n 20

19

LM [5]. Later, Audhkhasi et al. port this idea to smaller and publicdatasets achieving competitive results [9, 6].

The high number of dataset-specific hyper-parameter tuningneeded (e.g., number of layers, optimizers, learning rate scheduler)for A2W models makes difficult the incorporation of target units asan extra hyper-parameter. It is still an open question which is theoptimal unit length or the number of targets for end-to-end models.Small target units give more flexibility, and they are complicated todecode. On the other hand, big units are less flexible and usuallyeasier and faster to decode. In [12] we observe that each type oftarget unit contributes differently to the final output. As a possiblesolution, Chan et al. and Liu et al. propose to learn the best possibledecomposition of units [13, 14].

In a different direction, some approaches propose to combine thehypothesis of various models that use different sets of target units.For instance, Hori et al. suggest to combine the predictions of aword-based LM with the ones from a character-based during decod-ing [15]. The results obtained show improvements over character-only decoding. Another example is [12], where we show that bycombining hypothesis [16] from various models that use differenttargets units, the final output recovers some of the mistakes done bythe individual models.

Another potential solution to take into consideration differentunit representations is to use MTL techniques. Recently, the ASRcommunity started showing interest in MTL approaches. One exam-ple of this recent interest is [17]. In this work, Kim et al. combinesa S2S with a CTC-loss by using MTL techniques. In this case, theCTC-loss was only used as an auxiliary loss-function to enforce amonotonic alignment between the speech signal and the output labelsequence. Another ASR task where MTL is applied is in multilin-gual speech recognition. In this direction, we propose [18]. Thiswork shows that in a common MTL setting, the network can learnuseful intermediate representations for multiple languages. By usingvarious CTC-losses, our architecture was able to learn useful inter-mediate representations common among all inputs.

MTL has also been used hierarchically to provide low-level su-pervision to deep neural networks. In this direction, Søgaard andGoldberg propose an architecture that learns more fundamental Nat-ural Language Processing tasks to guide the internal representationof a neural network [19]. Focused on ASR, Fernandez et al. usemono phone prediction as an auxiliary task to improve digit recog-nition using CTC [20] . In this case, the auxiliary loss helps thenetwork to learn an intermediate phonetic representation. More re-cently, Toshniwal et al. and Krishna et al. port this idea to a morerealistic dataset and use characters and workpieces as higher levelunits respectively [21, 22]. There are three fundamental differencesbetween [20, 21, 22] and our work. First, we analyze and showimprovements in different multitask learning architectures. Second,our method takes advantage of the inherent compositionality of theunits rather than the acoustic characteristics of the signal. The fact ofnot considering acoustics, at the same time, allows us to not rely onphoneme annotations. Working with phonemes requires a dictionarywhich maps a word to its phonetic transcriptions. Such transcrip-tions are expensive and prone to error. Third, our method can use ahigher number of auxiliary tasks.

3. UNIT SELECTION

Different methods have been proposed to create intermediate targetrepresentations between character and words. For instance, Liu et al.use bigram and trigram units constructed from characters by a CTCmodel [14]. Another example is [7]. In this work, Sennrich et al. use

PROJ

Subword1kChar

BiLSTM

Shared Encoder

Speech Features

PROJ

Subword300

BiLSTM

…

BiLSTM n<latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="NtnmDLJ1NgRz18QxoCqIYjLd6rc=">AAAB3XicbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy5bsA9oh5JJ77SxmcyQ3BHK0F/gxoUi/i13/hvTx0JbDwQ+zknIvSfKlLTk+9/e1vbO7t5+6aB8WDk6PqmeVto2zY3AlkhVaroRt6ikxhZJUtjNDPIkUtiJJvfzvPOMxspUP9I0wzDhIy1jKTg5q6kH1Zpf9xdimxCsoAYrNQbVr/4wFXmCmoTi1vYCP6Ow4IakUDgr93OLGRcTPsKeQ80TtGGxGHTGLp0zZHFq3NHEFu7vFwVPrJ0mkbuZcBrb9Wxu/pf1copvw0LqLCfUYvlRnCtGKZtvzYbSoCA1dcCFkW5WJsbccEGum7IrIVhfeRPa1/XArwdNH0pwDhdwBQHcwB08QANaIADhBd7g3XvyXr2PZV1b3qq3M/gj7/MHwtmLnw==</latexit><latexit sha1_base64="NtnmDLJ1NgRz18QxoCqIYjLd6rc=">AAAB3XicbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy5bsA9oh5JJ77SxmcyQ3BHK0F/gxoUi/i13/hvTx0JbDwQ+zknIvSfKlLTk+9/e1vbO7t5+6aB8WDk6PqmeVto2zY3AlkhVaroRt6ikxhZJUtjNDPIkUtiJJvfzvPOMxspUP9I0wzDhIy1jKTg5q6kH1Zpf9xdimxCsoAYrNQbVr/4wFXmCmoTi1vYCP6Ow4IakUDgr93OLGRcTPsKeQ80TtGGxGHTGLp0zZHFq3NHEFu7vFwVPrJ0mkbuZcBrb9Wxu/pf1copvw0LqLCfUYvlRnCtGKZtvzYbSoCA1dcCFkW5WJsbccEGum7IrIVhfeRPa1/XArwdNH0pwDhdwBQHcwB08QANaIADhBd7g3XvyXr2PZV1b3qq3M/gj7/MHwtmLnw==</latexit><latexit sha1_base64="ck1/v/JpOJELH0iIdgPbhaWErXY=">AAAB6HicbVBNT8JAEJ3iF+IX6tHLRmLiibRe9Ej04hESCyTQkO0yhZXtttndmpCGX+DFg8Z49Sd589+4QA8KvmSSl/dmMjMvTAXXxnW/ndLG5tb2Tnm3srd/cHhUPT5p6yRTDH2WiER1Q6pRcIm+4UZgN1VI41BgJ5zczf3OEyrNE/lgpikGMR1JHnFGjZVaclCtuXV3AbJOvILUoEBzUP3qDxOWxSgNE1TrnuemJsipMpwJnFX6mcaUsgkdYc9SSWPUQb44dEYurDIkUaJsSUMW6u+JnMZaT+PQdsbUjPWqNxf/83qZiW6CnMs0MyjZclGUCWISMv+aDLlCZsTUEsoUt7cSNqaKMmOzqdgQvNWX10n7qu65da/l1hq3RRxlOINzuAQPrqEB99AEHxggPMMrvDmPzovz7nwsW0tOMXMKf+B8/gDV0Yzu</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit><latexit sha1_base64="IqW2N36QktBwntGQavLhE1DLrrE=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1FipKQflilt1FyDrxMtJBXI0BuWv/jBmaYTSMEG17nluYvyMKsOZwFmpn2pMKJvQEfYslTRC7WeLQ2fkwipDEsbKljRkof6eyGik9TQKbGdEzVivenPxP6+XmvDGz7hMUoOSLReFqSAmJvOvyZArZEZMLaFMcXsrYWOqKDM2m5INwVt9eZ20r6qeW/Wa15X6bR5HEc7gHC7BgxrU4R4a0AIGCM/wCm/Oo/PivDsfy9aCk8+cwh84nz/XEYzy</latexit>

1<latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="hP+6LrUf2d3tZaldqaQQvEKMXyw=">AAAB2XicbZDNSgMxFIXv1L86Vq1rN8EiuCozbnQpuHFZwbZCO5RM5k4bmskMyR2hDH0BF25EfC93vo3pz0JbDwQ+zknIvSculLQUBN9ebWd3b/+gfugfNfzjk9Nmo2fz0gjsilzl5jnmFpXU2CVJCp8LgzyLFfbj6f0i77+gsTLXTzQrMMr4WMtUCk7O6oyaraAdLMW2IVxDC9YaNb+GSS7KDDUJxa0dhEFBUcUNSaFw7g9LiwUXUz7GgUPNM7RRtRxzzi6dk7A0N+5oYkv394uKZ9bOstjdzDhN7Ga2MP/LBiWlt1EldVESarH6KC0Vo5wtdmaJNChIzRxwYaSblYkJN1yQa8Z3HYSbG29D77odBu3wMYA6nMMFXEEIN3AHD9CBLghI4BXevYn35n2suqp569LO4I+8zx84xIo4</latexit><latexit sha1_base64="2yIKfhykAKv0MhUqDiS4SywkL1Y=">AAAB3XicbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy5bsA9oh5JJ77SxmcyQ3BHK0F/gxoUi/i13/hvTx0JbDwQ+zknIvSfKlLTk+9/e1vbO7t5+6aB8WDk6PqmeVto2zY3AlkhVaroRt6ikxhZJUtjNDPIkUtiJJvfzvPOMxspUP9I0wzDhIy1jKTg5qxkMqjW/7i/ENiFYQQ1WagyqX/1hKvIENQnFre0FfkZhwQ1JoXBW7ucWMy4mfIQ9h5onaMNiMeiMXTpnyOLUuKOJLdzfLwqeWDtNIncz4TS269nc/C/r5RTfhoXUWU6oxfKjOFeMUjbfmg2lQUFq6oALI92sTIy54YJcN2VXQrC+8ia0r+uBXw+aPpTgHC7gCgK4gTt4gAa0QADCC7zBu/fkvXofy7q2vFVvZ/BH3ucPaQSLYg==</latexit><latexit sha1_base64="2yIKfhykAKv0MhUqDiS4SywkL1Y=">AAAB3XicbZBLSwMxFIXv+Ky1anXrJlgEV2XGjS4FNy5bsA9oh5JJ77SxmcyQ3BHK0F/gxoUi/i13/hvTx0JbDwQ+zknIvSfKlLTk+9/e1vbO7t5+6aB8WDk6PqmeVto2zY3AlkhVaroRt6ikxhZJUtjNDPIkUtiJJvfzvPOMxspUP9I0wzDhIy1jKTg5qxkMqjW/7i/ENiFYQQ1WagyqX/1hKvIENQnFre0FfkZhwQ1JoXBW7ucWMy4mfIQ9h5onaMNiMeiMXTpnyOLUuKOJLdzfLwqeWDtNIncz4TS269nc/C/r5RTfhoXUWU6oxfKjOFeMUjbfmg2lQUFq6oALI92sTIy54YJcN2VXQrC+8ia0r+uBXw+aPpTgHC7gCgK4gTt4gAa0QADCC7zBu/fkvXofy7q2vFVvZ/BH3ucPaQSLYg==</latexit><latexit sha1_base64="1enTr8x+dPZYKeA2su7IqvnUdLk=">AAAB6HicbVBNT8JAEJ3iF+IX6tHLRmLiibRe9Ej04hESCyTQkO0yhZXtttndmpCGX+DFg8Z49Sd589+4QA8KvmSSl/dmMjMvTAXXxnW/ndLG5tb2Tnm3srd/cHhUPT5p6yRTDH2WiER1Q6pRcIm+4UZgN1VI41BgJ5zczf3OEyrNE/lgpikGMR1JHnFGjZVa3qBac+vuAmSdeAWpQYHmoPrVHyYsi1EaJqjWPc9NTZBTZTgTOKv0M40pZRM6wp6lksaog3xx6IxcWGVIokTZkoYs1N8TOY21nsah7YypGetVby7+5/UyE90EOZdpZlCy5aIoE8QkZP41GXKFzIipJZQpbm8lbEwVZcZmU7EheKsvr5P2Vd1z617LrTVuizjKcAbncAkeXEMD7qEJPjBAeIZXeHMenRfn3flYtpacYuYU/sD5/AF5XYyx</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit><latexit sha1_base64="l9eImvYcFOKpzEDji/n9jPDeWb8=">AAAB6HicbVBNS8NAEJ3Ur1q/qh69LBbBU0lEqMeiF48t2A9oQ9lsJ+3azSbsboQS+gu8eFDEqz/Jm//GbZuDtj4YeLw3w8y8IBFcG9f9dgobm1vbO8Xd0t7+weFR+fikreNUMWyxWMSqG1CNgktsGW4EdhOFNAoEdoLJ3dzvPKHSPJYPZpqgH9GR5CFn1Fip6Q3KFbfqLkDWiZeTCuRoDMpf/WHM0gilYYJq3fPcxPgZVYYzgbNSP9WYUDahI+xZKmmE2s8Wh87IhVWGJIyVLWnIQv09kdFI62kU2M6ImrFe9ebif14vNeGNn3GZpAYlWy4KU0FMTOZfkyFXyIyYWkKZ4vZWwsZUUWZsNiUbgrf68jppX1U9t+o1ryv12zyOIpzBOVyCBzWowz00oAUMEJ7hFd6cR+fFeXc+lq0FJ585hT9wPn8Aep2MtQ==</latexit>

BiLSTM BiLSTM BiLSTM

PROJ

Subword10k

PROJ

Task-specific CTC Loss

Task-specific CTC Loss

Task-specific CTC Loss

Task-specific CTC Loss

… …

Fig. 2. The more common Block Multitask Learning (BMTL) modellearns intermediate representation using its shared encoder. Thetask-specific modules will use this intermediate representation to dis-criminate each unit.

the Byte-Pair Encoding (BPE) algorithm [23] to overcome the fixedlength restriction applied in [14]. By using BPE, they create unitsaccording to the frequency of repetition fixing the number of targets.This approach has been extensively used by the Machine Translationcommunity (MT) and for LM [24]. The inherent compositionalityof BPE sets (i.e., units with fewer operations form units with moreBPE operations) makes this technique an appropriate candidate forHMTL, the model proposed in this work.

3.1. Byte-Pair Encoding Algorithm

Gage propose the BPE algorithm as a compression algorithm [23].This technique iteratively replaces pairs of most frequently co-occurrent bytes creating a new byte. A table that maps the corre-spondence of the new unit and its composition is needed to recoverthe original form of the data.

We apply this algorithm to natural languages by treating eachtarget unit (e.g., a character) as a byte. More specifically, in our case,our initial unit set (i.e., our first set of bytes) are characters (see Sec-tion 3.3). Each BPE step/operation performs a unit merging. For thatreason, the number of BPE operations roughly defines the number ofunits of the target set. For instance, if two targets in our vocabularyare ‘HELL’ and ‘O’ and they appear together frequently, by applyinga BPE operation we can merge them in a new target unit: ‘HELLO’.This operation can be repeated an arbitrary number of times untilconverging to a target set that contains all words of the vocabulary.

3.2. Subword Units

Subwords are the units created from characters by using BPE oper-ations. More concretely, we use [7] to create our Subword units. Togenerate different units we start with a character set. We also use aunique character (in our case ‘@’) to determine the position of theSubword inside a word. We do not use a dedicated space characterbecause we define word boundaries by the absence of the token ‘@’within a unit.

2

Method UtteranceOriginal you know it’s no not even cold weather

Character y o u k n o w i t ’ s n o n o t e v e n c o l d we a t h e r

Subword 300 you know it’s no not even co@ ld w@ ea@ther

Subword 1k you know it’s no not even co@ ld wea@ therSubword 10k you know it’s no not even cold weather

Table 1. Utterance sw02054-B 032568-033031 from the Switch-board dataset decomposed into characters, 300, 1,000 and 10,0000BPE operations. The character ‘@’ denotes that the unit is in themiddle of a word.

We create three different sets of subword units: Subword300(s300), Subword1k (s1k) and Subword10k (s10k). The numberafter ‘Subword’ defines the number of BPE operations used to cre-ate that set, that is roughly the number of units that the set has.More precisely, taking the blank symbol for CTC computation(see Section 4.1.1) into consideration, each set contains 388, 1,088and 10,088 target units respectively. Table 1 shows an utterancefrom Switchboard dataset (see Section 5) decomposed using dif-ferent BPE operations. Note that as the number of BPE operationsincreases, the amount of units that represents whole words also in-creases. For that reason, we can consider s10k an approximation ofa word target set.

3.3. Characters

The character set is the base set from where we construct the Sub-word units. It contains alpha-numerical characters, punctuation andthe blank symbol: 48 symbols in total.

Because we can use the position symbol ‘@’ to find word bound-aries when using subword units (i.e., s300, s1k, s10k) and to stressthe compositionality of our model, we do not include a ‘space’ repre-sentation in our character set. Therefore, the character representationof our model predicts strings of characters without spaces as we cansee in Table 1.

4. MULTITASK LEARNING ARCHITECTURE

In this section, we introduce two different architectures that acceptauxiliary loss-functions. First, we present Block Multitask Learn-ing (BMTL) (see Fig. 2), the conventional approach for multitasklearning with CTC inspired by our previous work [18]. Second, weintroduce our new HMTL architecture (see Fig. 1) that is inspiredby [21, 20, 22] and takes advantage of the compositionality of sub-words. Although both models use CTC as auxiliary loss-functions,we can port them to S2S models.

4.1. Singletask Learning Baseline

The STL baseline architecture is similar to the approach that we pro-pose in [25, 12]. The acoustic encoder reads a sequence of acousticfeatures vectors X = (x0 · · ·xT ), where X ∈ RF×T , F ∈ N+

is the dimensionality of the feature vector and T ∈ N+ the numberof frames of the audio sequence. Then, a Bidirectional Long-shortTerm Memory (BiLSTM) [26] takes X as input and returns a matrixH ∈ R(2×H)×T where H ∈ N+ is the dimensionality of one direc-tion of a hidden state of the BiLSTM. Then, each hidden state is pro-jected by a function F : R(2×H)×T → R|L

′|×T , where L′ = L ∪ ∅

is the extended character set, L is our original character set, and ∅ isthe blank symbol. Note that |L′| = |L|+ 1.

Finally, we apply a softmax function over each row S :

R|L′|×T → {H ∈ R|L

′|×T } where 0 < Hl′,t < 1, creatinga probability distribution over the extended unit target set L′ foreach frame. More formally,

∑|L′|l′=0 Hl′,t = 1. Then, CTC uses

this matrix and dynamic programming techniques to compute theprobability of generating all possible correct sequences.

4.1.1. CTC

Let z = (z1, · · · , zU ) ∈ N+U be a ground truth sequence whereU ∈ N+, U ≤ T and zu < |L|. In this paper, z is the transcriptionof an input sequence. Let us define a many to one mapping functionB : N+T → N+U , where T ∈ N+ is the input length of the featurematrix X . B will therefore map a sequence p = (p1, · · · , pT ) ∈N+T , where pt < |L′|, to the ground truth sequence z.

This mapping B is also referred as the squash function, as it re-moves all blank symbols of the path and squashes multiple repeatedcharacters into a single one (e.g., B(AA∅AAABB) = AAB). Notethat we do not squash characters that are separated by the blank sym-bol because this allows to predict repeated characters. Then, let usdefine the probability of the sequence p as

P (p|X) =

T∏t=1

P (pt|x) (1)

where P (pt|x) is the probability of observing the label pt at time t.To calculate the probability of an output sequence z we sum over allpossible sequences:

P (z|X) =∑

p∈B−1(z)

P (p|X) (2)

To sum over all sequences, we use a technique inspired by the tradi-tional dynamic programming method used in HMMs: the forward-backward algorithm [2]. We additionally force the appearance ofblank symbols in the sequence p by augmenting it with blank sym-bols between each of the labels of z as well as at the beginning andthe end of the sequence.

Given a sequence of speech features X , we can now computethe probability that the network correctly predicts the ground truthsequence z. In the following subsections, we present decodingstrategies that process the output of the system trained with CTC tocreate a linguistically reasonable transcription.

4.1.2. Greedy Search

To create a transcription without adding any linguistic informationwe can greedily search the best sequence p ∈ N+T :

argmaxp

T∏t=1

P (pt|X) (3)

The mapping of the path to a transcription z is straight forward. Wecan map both sequences by applying the squash function: z = B(p).

4.1.3. Shallow Fusion

We can combine a unit-specific LM with the probability distributiongiven by a CTC-trained model. Let us assume that the alphabet of the

3

character based LM is equal to L. We want to find a transcription thathas a high probability based on the acoustics from the CTC modelas well as the LM. Since we have to sum over all possible sequencesp for a transcription z, our goal is to solve the following equation:

argmaxz

∑B−1(z)

T∏t=1

PAM (pt|X)× PLM (pt|B(p1:t−1)) (4)

Where PAM is the probability of the AM and PLM is the probabilityof the LM. Note that we can not estimate a useful probability for theblank label ∅ with the LM, so we set P ′LM (∅|B(p1:t−1)) = 1. Tonot favor a sequence of blank symbols, we apply an insertion bonusb ∈ R for every pt 6= ∅ as:

PLM (k|B(p1:t−1)) =

{PLM (k|B(p1:t−1)) · b, if k 6= ∅1, if k = ∅

(5)

Where PLM (k|B(p1:t−1)) is provided by the unit-specific LM. Be-cause it is unfeasible to calculate an exact solution to equation (5) bybrute force, we apply beam search similar to [27].

Shallow fusion is compatible with any target unit. More con-cretely, we apply this approach to subword, character or word unitsystems. Note that this technique is independent of the architectureof the AM. For instance, in this work, we apply shallow fusion toSTL or MTL architecture.

4.2. Block Multi Task Learning

BMTL (see Fig. 2) is a general architecture for multitask learning.This approach shares a n BiLSTM layer encoder until before theunit-specific modules. After that, a unit-specific module process thehidden representations of the shared encoder. We compose this mod-ule with one BiLSTM layer and a projection layer that maps the hid-den states of the BiLSTM layer to the dimensionality of each sub-word/character unit set plus one.

More formally, BMTL can be written as,

e0 = Shared Encoder(X)

Lchar = softmax(WcharBiLSTMchar(e0))

Ls300 = softmax(Ws300BiLSTMs300(e0))

Ls1k = softmax(Ws1kBiLSTMs1k(e0))

Ls10k = softmax(Ws10kBiLSTMs10k(e0))

(6)

Where Shared Encoder function is a n layer BiLSTM thatis shared across all unit-specific modules, e0 is the hidden rep-resentation generated by the shared encoder, BiLSTMn withn ∈ {char, s300, s1k, s10k}, is a one layer unit-specific BiL-STM, W ∈ R|L

′n|×(2×H) represents a unit-specific linear trans-

formation (feedforward network) that maps the bidirectional hiddendimensionality of the unit-specific BiLSTM to the actual target unitdimension plus one, X ∈ RF×T is the input sequence featurevectors and Ln ∈ R|L

′n|×T is the probability matrix of each frame

over each target unit set n plus one. Note that the output matrix Ln

is then processed either for CTC during training or for a decoderduring testing.

Because the model needs to generate different output representa-tions of the same input, we hypothesize that the shared encoder willlearn a more general and flexible encoding of the audio features. Fi-nally, we assume that the subword/character-specific modules willlearn how to discriminate each unit separately using the representa-tion generated by the shared encoder.

4.3. Hierarchical Multi Task Learning

HMTL (see Fig. 1), similar to BMTL, shares an encoder composedof n BiLSTM layers. The fundamental difference between BMTLand HMTL is that HMTL will learn intermediate representations ina fine-to-coarse fashion after each layer. More concretely, followingthe encoder, a cascade of BiLSTM layers are stacked on the model.After each layer, we connect an auxiliary task-specific module. ABiLSTM and a linear projection layer form this task-specific mod-ule that maps the hidden representation of each layer to the specificnumber of units of each target set plus one. The projection made bythe task-specific module allows us to compute CTC in different partsof the network.

More formally, we can define the main structure of HMTL as

e0 = Shared Encoder(X)

e1 = BiLSTM1(e0)

e2 = BiLSTM2(e1)

e3 = BiLSTM3(e2)

(7)

Then, we will obtain a probability distribution by using unit-specific module as

Lchar = softmax(WcharBiLSTMchar(e0))

Ls300 = softmax(Ws300BiLSTMs300(e1))

Ls1k = softmax(Ws1kBiLSTMs1k(e2))

Ls10k = softmax(Ws10kBiLSTMs10k(e3))

(8)

Note that we use the same name convention as in Section 4.2.We hypothesize that this training strategy will help the model to

learn how to exploit the inherent compositionality of each subwordsunit.

5. RESULTS

In this work, we use the 300 hours Switchboard subset for train-ing the AM and the Fisher transcripts dataset for training the unit-specific LM for shallow fusion. We use the 2000 HUB5 Eval2000(LDC2002S09) for evaluation. The corpus is composed of Englishtelephone conversations and is divided into the Switchboard subset,which more closely resembles the training data, and the Callhomesubset. We extracted 43-dimensional filter bank, and pitch featuresvectors with Cepstral Mean Normalization using Kaldi [28]. Weuse the Tensorflow branch of EESEN [4] to develop the rest of thepipeline.

5.1. Architecture

All MTL architectures presented in this section (i.e., HMTL andBMTL) have a two-layer shared encoder, and each layer has 320cells. We perform a 3-fold data augmentation by sub-sampling.We triple the number of sequences by reducing the frame rate ofeach sample by 3. The frames dropped during sub-sampling areconcatenated to the middle frame for providing additional context.We also add a projection of 340 cells between each layer of theshared encoder. The rest of the architecture is described in Section 4.Our models are decoded using greedy search or shallow fusion witha unit-specific LM composed by two unidirectional LSTM layers.The results presented in this section have not been properly hyper-parameter tuned, which may leave room for more improvement.

4

% WER STL HMTL STL BMTL STL (LM) HMTL (LM) STL (LM) BMTL (lm)Subword300 20.6 / 33.5 17.6 / 30.4 20.6 / 33.5 18.7 / 31.8 14.2 / 26.3 13.1 / 24.7 14.2 / 26.3 13.5 / 25.6Subword1k 19.0 / 30.8 15.3 / 26.9 20.5 / 32.5 16.7 / 28.6 15.2 / 26.8 12.7 / 23.8 15.5 / 27.5 13.1 / 24.5

Subword10k 17.4 / 28.5 14.0 / 25.5 17.8 / 29.2 14.6 / 25.8 15.6 / 26.7 12.5 / 23.7 16.2 / 27.3 13.1 / 24.7

Table 2. % WER summary of the models presented in this paper on the Eval2000 test set. The AM is trained on the 300 hours subset of theSwitchboard training set, while the LM is trained on the Switchboard and Fisher training sets. The results on the Switchboard subset (swbd)are on the left side of the slash and the results on the Callhome (cllhm) subset are on the right side of the slash. STL stands for singletasklearning model, HMTL stands for Hierarchical Multitask Learning, BMTL stands for Block Multitask Learning and S300, S1k and S10k arethe different subword target units used. All models have a two layer shared encoder. Each STL model has the same number of parametersand configuration than the model on the right. Note that, except in Subword300 unit set, HMTL and BMTL do not have the same number ofparameters so they can not be compared.

Model Type LM % WER (swbd / cllhm)Zeyer et al. [29] S2S NO 13.5 / 27.4Zeyer et al. [29] S2S YES 11.8 / 25.7

Ours CTC NO 14.0 / 27.1Ours CTC YES 12.5 / 23.7

Zenkel et al. [12] CTC NO 17.8 / 29.0Zenkel et al. [12] CTC YES 14.7 / 26.2

Audhkhasi et al. [9] CTC NO 14.6 / 23.6

Table 3. Comparison table for different approaches of A2W models.All models have been trained on 300h Switchboard and tested on theEval2000 test set. The results on the Switchboard subset (swbd) arein the left side of the slash and the results from Callhome subset(cllhm) are in the right side of the slash.

5.2. Evaluation of multitask learning architectures

In Table 2, we show a comparison between the STL and the MTLmodels presented in this paper. In this table, each pair of STL andMTL model have the same number of parameters. Note that, exceptin the case of Subword300, HMTL and BMTL does not have thesame amount of parameters (see Section 4), so they are not compa-rable.

In the left side of Table 2, we summarize all results obtainedusing only the AM without considering the LM during decoding.Those models are also known as A2W. We observe that in general,the addition of auxiliary tasks (while maintaining the same numberof parameters) helps concerning WER. We can see that the relativeimprovements in HMTL (14-20% relative WER) are higher than inBMTL (9-18% relative WER).

The right side of Table 2 summarizes all results obtained com-bining the AM and the LM probabilities by shallow fusion. Wecan see that shallow fusion generally helps regarding WER. The im-provements of the addition of a LM, however, banish as the numberof units increases. This behavior that we also observe in [12], isconsistent in MTL models.

The improvement of shallow fusion in an intermediate represen-tation (Subword1k in STL(lm) in the next-to-last column of Table 2)is comparable to HMTL (Subword1k in HMTL in the first column ofTable 2). Considering these results, we hypothesize that the additionof a higher order unit supervision on upper layers provides linguisticknowledge to the AM.

Finally, in Table 3, we compare existing A2W models that, to thebest of our knowledge, achieve the best results in the Eval2000 testset. We can see that HMTL outperforms previous approaches basedon CTC in the Switchboard subset. Audhkhasi et al. [9], however,

achieve a better score in the Callhome subset than our model. Thisimprovement is most probably an indicator that HMTL is not ableto generalize as good as [9] across domains. To improve the gen-eralization of HMTL, we can apply regularization techniques suchas Dropout. The approach presented by Zeyer et al. [29], however,outperforms other methods, with the downside that is autoregressive.

6. CONCLUSIONS

In this paper, we present a CTC-based Acoustic-to-Word (A2W)model with Hierarchical Multitask Learning (HMTL). This approachencourages the formation of useful intermediate representations byusing targets with a different number of Byte-Pair Encoding (BPE)operations at different levels in a single network. At the lower lay-ers, the model predicts few and general targets, while at the higherlayers, the model predicts highly specific and abstract targets, suchas words.

By using the Subword10k representation, after shallow fusionwith a unit-specific LM, our HMTL model improves from 15.6%WER (STL model with the same number of parameters) to 12.5%WER. By contrast, a more traditional Multitask Learning Technique(MLT), called Block Multitask Learning (BMTL), using again Sub-word10k improves from 16.2% WER (STL model with the samenumber of parameters) to 13.1% WER. None of the experiments re-quired any fine parameter-tuning.

Given that the presented model generates different-granularityoutputs, we are attempting to combine the individual hypotheses [16]and develop a decoding scheme that jointly processes the differentoutputs, and integrates them directly into a word hypothesis.

7. ACKNOWLEDGMENTS

We gratefully acknowledge the support of NVIDIA Corporation withthe donation of the Titan X Pascal GPU used for this research. Theauthors would also like to thank the anonymous reviewers, KarenLivescu, Thomas Zenkel, Siddharth Dalmia, Jindrich Libovicky,Ozan Caglayan, Amanda Duarte, Elizabeth Salesky, Desmond El-liott and Pranava Madhyastha for sharing their insights.

8. REFERENCES

[1] Dan Jurafsky and James Martin, Speech & language process-ing, Pearson Education, 2000.

[2] Lawrence Rabiner and B Juang, “An introduction to hiddenmarkov models,” ASSP Magazine, 1986.

5

[3] Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prab-havalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron JWeiss, Kanishka Rao, Katya Gonina, et al., “State-of-the-artspeech recognition with sequence-to-sequence models,” in In-ternational Conference on Acoustics, Speech and Signal Pro-cessing (ICASSP). IEEE, 2017.

[4] Yajie Miao, Mohammad Gowayyed, and Florian Metze,“Eesen: End-to-end speech recognition using deep rnn modelsand wfst-based decoding,” in Workshop on Automatic SpeechRecognition and Understanding (ASRU). IEEE, 2015.

[5] Hagen Soltau, Hank Liao, and Hasim Sak, “Neural speech rec-ognizer: Acoustic-to-word LSTM model for large vocabularyspeech recognition,” in Interspeech. ISCA, 2017.

[6] Kartik Audhkhasi, Bhuvana Ramabhadran, George Saon,Michael Picheny, and David Nahamoo, “Direct acoustics-to-word models for english conversational speech recognition,” inInterspeech. ISCA, 2017.

[7] Rico Sennrich, Barry Haddow, and Alexandra Birch, “Neuralmachine translation of rare words with subword units,” in An-nual Meeting of the Association for Computational Linguistics.ACL, 2016.

[8] Rich Caruana, “Multitask learning,” Machine Learning, 1997.

[9] Kartik Audhkhasi, Brian Kingsbury, Bhuvana Ramabhadran,George Saon, and Michael Picheny, “Building competi-tive direct acoustics-to-word models for english conversationalspeech recognition,” in International Conference on Acoustics,Speech and Signal Processing (ICASSP). IEEE, 2018.

[10] Dzmitry Bahdanau, Jan Chorowski, Dmitriy Serdyuk, Phile-mon Brakel, and Yoshua Bengio, “End-to-end attention-basedlarge vocabulary speech recognition,” in International Con-ference on Acoustics, Speech and Signal Processing (ICASSP).IEEE, 2016.

[11] Alex Graves, Santiago Fernandez, Faustino Gomez, and Jur-gen Schmidhuber, “Connectionist temporal classification: la-belling unsegmented sequence data with recurrent neural net-works,” in International Conference on Machine learning.ACM, 2006.

[12] Thomas Zenkel, Ramon Sanabria, Florian Metze, and AlexWaibel, “Subword and crossword units for CTC acoustic mod-els,” in International Conference on Acoustics, Speech andSignal Processing (ICASSP). ISCA, 2017.

[13] Quoc V. Le Navdeep Jaitly William Chan, Yu Zhang, “La-tent sequence decompositions,” International Conference onLearning Representations (ICLR), 2016.

[14] Hairong Liu, Zhenyao Zhu, Xiangang Li, and SanjeevSatheesh, “Gram-CTC: Automatic unit selection and targetdecomposition for sequence labelling,” in International Con-ference on Machine Learning (ICML), 2017.

[15] Takaaki Hori, Shinji Watanabe, and John R Hershey, “Multi-level language modeling and decoding for open vocabularyend-to-end speech recognition,” in Workshop AutomaticSpeech Recognition and Understanding Workshop (ASRU).IEEE, 2017.

[16] Jonathan G Fiscus, “A post-processing system to yield re-duced word error rates: Recognizer output voting error reduc-tion (rover),” in Workshop on Automatic Speech Recognitionand Understanding (ASRU). IEEE, 1997.

[17] Suyoun Kim, Takaaki Hori, and Shinji Watanabe, “Joint CTC-attention based end-to-end speech recognition using multi-tasklearning,” in International Conference on Acoustics, Speechand Signal Processing (ICASSP). IEEE, 2017.

[18] Siddharth Dalmia, Ramon Sanabria, Florian Metze, andAlan W Black, “Sequence-based multi-lingual low resourcespeech recognition,” in International Conference on Acoustics,Speech and Signal Processing (ICASSP). IEEE, 2018.

[19] Anders Søgaard and Yoav Goldberg, “Deep multi-task learn-ing with low level tasks supervised at lower layers,” in An-nual Meeting of the Association for Computational Linguistics(ACL). ACL, 2016.

[20] Santiago Fernandez, Alex Graves, and Jurgen Schmidhuber,“Sequence labelling in structured domains with hierarchical re-current neural networks,” in International Joint Conference onArtificial Intelligence (IJCAI), 2007.

[21] Shubham Toshniwal, Hao Tang, Liang Lu, and Karen Livescu,“Multitask learning with low-level auxiliary tasks for encoder-decoder based speech recognition,” in Interspeech. ISCA,2017.

[22] Kalpesh Krishna, Shubham Toshniwal, and Karen Livescu,“Hierarchical multitask learning for CTC-based speech recog-nition,” arXiv preprint arXiv:1807.06234, 2018.

[23] Philip Gage, “A new algorithm for data compression,” in TheC Users Journal. R & D Publications, Inc., 1994.

[24] Tomas Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le,Stefan Kombrink, and Jan Cernocky, “Subword language mod-eling with neural networks,” preprint (http://www. fit. vutbr.cz/imikolov/rnnlm/char. pdf), 2012.

[25] Thomas Zenkel, Ramon Sanabria, Florian Metze, Jan Niehues,Matthias Sperber, Sebastian Stuker, and Alex Waibel, “Com-parison of decoding strategies for CTC acoustic models,” inInterspeech. ISCA, 2017.

[26] Sepp Hochreiter and Jurgen Schmidhuber, “Long short-termmemory,” Neural Computation, 1997.

[27] Kyuyeon Hwang and Wonyong Sung, “Character-level incre-mental speech recognition with recurrent neural networks,” inInternational Conference on Acoustics, Speech and Signal Pro-cessing (ICASSP). IEEE, 2016.

[28] Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Bur-get, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, PetrMotlicek, Yanmin Qian, Petr Schwarz, et al., “The kaldi speechrecognition toolkit,” in Workshop on Automatic Speech Recog-nition and Understanding (ASRU). IEEE, 2011.

[29] Albert Zeyer, Kazuki Irie, Ralf Schluter, and Hermann Ney,“Improved training of end-to-end attention models for speechrecognition,” in Interspeech. ISCA, 2018.

6