U %‰ddŠã@sddlmZmZddlmZmZmZddlZddlm Z dddgZ Gdd „d ej j ƒZ Gd d „d ej j ƒZGd d „d eƒZGdd„dej j eƒZGdd„dej j ƒZGdd„dej j ƒZGdd„dej j ƒZeeeeeeeeeeeeeeeeeeeeeedœdd„Zeedœdd„ZdS)é)ÚABCÚabstractmethod)ÚListÚOptionalÚTupleN)ÚEmformerÚRNNTÚemformer_rnnt_baseÚemformer_rnnt_modelcsJeZdZdZeddœ‡fdd„ Zejejeejejfdœdd„Z ‡Z S) Ú_TimeReductionzÂCoalesces frames along time dimension into a fewer number of frames with higher feature dimensionality. Args: stride (int): number of frames to merge for each output frame. N)ÚstrideÚreturncstƒ ¡||_dS©N)ÚsuperÚ__init__r )Úselfr ©Ú __class__©ú:/tmp/pip-unpacked-wheel-lbdmvq91/torchaudio/models/rnnt.pyrs z_TimeReduction.__init__©ÚinputÚlengthsr c Csr|j\}}}|||j}|dd…d|…dd…f}|j|jdd}||j}| ||||j¡}| ¡}||fS)aForward pass. B: batch size; T: maximum input sequence length in batch; D: feature dimension of each input sequence frame. Args: input (torch.Tensor): input sequences, with shape `(B, T, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``input``. Returns: (torch.Tensor, torch.Tensor): torch.Tensor output sequences, with shape `(B, T // stride, D * stride)` torch.Tensor output lengths, with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in output sequences. NÚtrunc)Z rounding_mode)Úshaper ÚdivZreshapeÚ contiguous) rrrÚBÚTÚDZ num_framesZT_maxÚoutputrrrÚforwards  z_TimeReduction.forward) Ú__name__Ú __module__Ú __qualname__Ú__doc__ÚintrÚtorchÚTensorrr!Ú __classcell__rrrrr sr cs^eZdZdZd eeeeddœ‡fdd„ Zej e e ej e ej e ej fdœd d „Z ‡ZS) Ú _CustomLSTMa­Custom long-short-term memory (LSTM) block that applies layer normalization to internal nodes. Args: input_dim (int): input dimension. hidden_dim (int): hidden dimension. layer_norm (bool, optional): if ``True``, enables layer normalization. (Default: ``False``) layer_norm_epsilon (float, optional): value of epsilon to use in layer normalization layers (Default: 1e-5) Fçñh㈵øä>N)Ú input_dimÚ hidden_dimÚ layer_normÚlayer_norm_epsilonr csŒtƒ ¡tjj|d|| d|_tjj|d|dd|_|rjtjj||d|_tjjd||d|_ ntj  ¡|_tj  ¡|_ ||_ dS)Né©ZbiasF)Zeps) rrr'ÚnnÚLinearÚx2gÚp2gÚ LayerNormÚc_normÚg_normZIdentityr-)rr,r-r.r/rrrrCs   z_CustomLSTM.__init__)rÚstater cCs|dkrD| d¡}tj||j|j|jd}tj||j|j|jd}n|\}}| |¡}g}| d¡D]€}|| |¡}|  |¡}|  dd¡\} } } } |   ¡} |   ¡} |   ¡} |   ¡} | || | }|  |¡}| |  ¡}| |¡qdtj|dd} ||g}| |fS)aÒForward pass. B: batch size; T: maximum sequence length in batch; D: feature dimension of each input sequence element. Args: input (torch.Tensor): with shape `(T, B, D)`. state (List[torch.Tensor] or None): list of tensors representing internal state generated in preceding invocation of ``forward``. Returns: (torch.Tensor, List[torch.Tensor]): torch.Tensor output, with shape `(T, B, hidden_dim)`. List[torch.Tensor] list of tensors representing internal state generated in current invocation of ``forward``. Né)ÚdeviceÚdtyperr0)Zdim)Úsizer'Úzerosr-r;r<r4Zunbindr5r8ÚchunkZsigmoidÚtanhr7ÚappendÚstack)rrr9rÚhÚcZ gated_inputÚoutputsZgatesZ input_gateZ forget_gateZ cell_gateZ output_gater rrrr!Vs,      z_CustomLSTM.forward)Fr+©r"r#r$r%r&ÚboolÚfloatrr'r(rrrr!r)rrrrr*7sûú þr*c @steZdZeejejeejejfdœdd„ƒZeejejee e ejeejeje e ejfdœdd„ƒZ dS)Ú _TranscriberrcCsdSrr)rrrrrrr!Šsz_Transcriber.forward©rrÚstatesr cCsdSrr)rrrrKrrrÚinferŽsz_Transcriber.inferN) r"r#r$rr'r(rr!rrrLrrrrrI‰s$ûrIcsºeZdZdZddddddœeeeeeeeeeeeeeeedd œ‡fd d „Ze j e j e e j e j fd œd d„Z e j je j e j eeee j e e j e j eee j fdœdd„ƒZ‡ZS)Ú_EmformerEncodera½Emformer-based recurrent neural network transducer (RNN-T) encoder (transcription network). Args: input_dim (int): feature dimension of each input sequence element. output_dim (int): feature dimension of each output sequence element. segment_length (int): length of input segment expressed as number of frames. right_context_length (int): length of right context expressed as number of frames. time_reduction_input_dim (int): dimension to scale each element in input sequences to prior to applying time reduction block. time_reduction_stride (int): factor by which to reduce length of input sequence. transformer_num_heads (int): number of attention heads in each Emformer layer. transformer_ffn_dim (int): hidden layer dimension of each Emformer layer's feedforward network. transformer_num_layers (int): number of Emformer layers to instantiate. transformer_left_context_length (int): length of left context. transformer_dropout (float, optional): transformer dropout probability. (Default: 0.0) transformer_activation (str, optional): activation function to use in each Emformer layer's feedforward network. Must be one of ("relu", "gelu", "silu"). (Default: "relu") transformer_max_memory_size (int, optional): maximum number of memory elements to use. (Default: 0) transformer_weight_init_scale_strategy (str, optional): per-layer weight initialization scaling strategy. Must be one of ("depthwise", "constant", ``None``). (Default: "depthwise") transformer_tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) çÚrelurÚ depthwiseF)Útransformer_dropoutÚtransformer_activationÚtransformer_max_memory_sizeÚ&transformer_weight_init_scale_strategyÚtransformer_tanh_on_memN)r,Ú output_dimÚsegment_lengthÚright_context_lengthÚtime_reduction_input_dimÚtime_reduction_strideÚtransformer_num_headsÚtransformer_ffn_dimÚtransformer_num_layersÚtransformer_left_context_lengthrQrRrSrTrUr cs|tƒ ¡tjj||dd|_t|ƒ|_||}t|||| ||| | | ||| ||d |_ tj ||¡|_ tj  |¡|_ dS)NFr1)ÚdropoutÚ activationZleft_context_lengthrXZmax_memory_sizeZweight_init_scale_strategyZ tanh_on_mem) rrr'r2r3Ú input_linearr Útime_reductionrÚ transformerÚ output_linearr6r.)rr,rVrWrXrYrZr[r\r]r^rQrRrSrTrUZtransformer_input_dimrrrr°s0 ý ôz_EmformerEncoder.__init__rc CsF| |¡}| ||¡\}}| ||¡\}}| |¡}| |¡} | |fS)a¥Forward pass for training. B: batch size; T: maximum input sequence length in batch; D: feature dimension of each input sequence frame (input_dim). Args: input (torch.Tensor): input frame sequences right-padded with right context, with shape `(B, T + right context length, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``input``. Returns: (torch.Tensor, torch.Tensor): torch.Tensor output frame sequences, with shape `(B, T // time_reduction_stride, output_dim)`. torch.Tensor output input lengths, with shape `(B,)` and i-th element representing number of valid elements for i-th batch element in output frame sequences. )rarbrcrdr.) rrrÚinput_linear_outÚtime_reduction_outÚtime_reduction_lengthsÚtransformer_outÚtransformer_lengthsÚoutput_linear_outÚlayer_norm_outrrrr!Üs    z_EmformerEncoder.forwardrJc CsN| |¡}| ||¡\}}|j |||¡\}}} | |¡} | | ¡} | || fS)aRForward pass for inference. B: batch size; T: maximum input sequence segment length in batch; D: feature dimension of each input sequence frame (input_dim). Args: input (torch.Tensor): input frame sequence segments right-padded with right context, with shape `(B, T + right context length, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``input``. state (List[List[torch.Tensor]] or None): list of lists of tensors representing internal state generated in preceding invocation of ``infer``. Returns: (torch.Tensor, torch.Tensor, List[List[torch.Tensor]]): torch.Tensor output frame sequences, with shape `(B, T // time_reduction_stride, output_dim)`. torch.Tensor output input lengths, with shape `(B,)` and i-th element representing number of valid elements for i-th batch element in output. List[List[torch.Tensor]] output states; list of lists of tensors representing internal state generated in current invocation of ``infer``. )rarbrcrLrdr.) rrrrKrerfrgrhriZtransformer_statesrjrkrrrrLùs# ü  z_EmformerEncoder.infer)r"r#r$r%r&rHÚstrrGrr'r(rr!ÚjitÚexportrrrLr)rrrrrM˜s>$ïî,"ûrMc sxeZdZdZd eeeeeeeeddœ ‡fdd„ Zd ej ej e e e ej e ej ej e e ej fd œd d „Z ‡ZS)Ú _PredictoraRecurrent neural network transducer (RNN-T) prediction network. Args: num_symbols (int): size of target token lexicon. output_dim (int): feature dimension of each output sequence element. symbol_embedding_dim (int): dimension of each target token embedding. num_lstm_layers (int): number of LSTM layers to instantiate. lstm_hidden_dim (int): output dimension of each LSTM layer. lstm_layer_norm (bool, optional): if ``True``, enables layer normalization for LSTM layers. (Default: ``False``) lstm_layer_norm_epsilon (float, optional): value of epsilon to use in LSTM layer normalization layers. (Default: 1e-5) lstm_dropout (float, optional): LSTM dropout probability. (Default: 0.0) Fr+rNN) Ú num_symbolsrVÚsymbol_embedding_dimÚnum_lstm_layersÚlstm_hidden_dimÚlstm_layer_normÚlstm_layer_norm_epsilonÚ lstm_dropoutr c s†tƒ ¡tj |ˆ¡|_tj ˆ¡|_tj ‡‡‡‡fdd„t |ƒDƒ¡|_ tjj |d|_ tj  ˆ|¡|_tj |¡|_||_dS)Ncs(g|] }t|dkrˆnˆˆˆˆd‘qS)r)r.r/)r*)Ú.0Úidx©rsrtrurqrrÚ Hsúüz'_Predictor.__init__..)Úp)rrr'r2Z EmbeddingÚ embeddingr6Úinput_layer_normZ ModuleListÚrangeÚ lstm_layersZDropoutr_r3ÚlinearÚoutput_layer_normrv) rrprVrqrrrsrtrurvrryrr9s ùÿ z_Predictor.__init__)rrr9r cCs–| dd¡}| |¡}| |¡}|}g}t|jƒD]:\} } | ||dkrJdn|| ƒ\}} | |¡}| | ¡q2| |¡} | | ¡} |  ddd¡||fS)a#Forward pass. B: batch size; U: maximum sequence length in batch; D: feature dimension of each input sequence element. Args: input (torch.Tensor): target sequences, with shape `(B, U)` and each element mapping to a target symbol, i.e. in range `[0, num_symbols)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``input``. state (List[List[torch.Tensor]] or None, optional): list of lists of tensors representing internal state generated in preceding invocation of ``forward``. (Default: ``None``) Returns: (torch.Tensor, torch.Tensor, List[List[torch.Tensor]]): torch.Tensor output encoding sequences, with shape `(B, U, output_dim)` torch.Tensor output lengths, with shape `(B,)` and i-th element representing number of valid elements for i-th batch element in output encoding sequences. List[List[torch.Tensor]] output states; list of lists of tensors representing internal state generated in current invocation of ``forward``. r:rNé) Zpermuter|r}Ú enumeraterr_rAr€r)rrrr9Zinput_tbZ embedding_outZinput_layer_norm_outZlstm_outZ state_outZ layer_idxZlstmZlstm_state_outZ linear_outZoutput_layer_norm_outrrrr!Xs       z_Predictor.forward)Fr+rN)NrFrrrrro(s,÷ö#üûrocs\eZdZdZd eeeddœ‡fdd„ Zejejejeje ejejejfdœdd „Z ‡Z S) Ú_Joinera@Recurrent neural network transducer (RNN-T) joint network. Args: input_dim (int): source and target input dimension. output_dim (int): output dimension. activation (str, optional): activation function to use in the joiner. Must be one of ("relu", "tanh"). (Default: "relu") rON)r,rVr`r cs\tƒ ¡tjj||dd|_|dkr4tj ¡|_n$|dkrJtj ¡|_nt d|›ƒ‚dS)NTr1rOr@zUnsupported activation ) rrr'r2r3r€ZReLUr`ZTanhÚ ValueError)rr,rVr`rrrr“s z_Joiner.__init__©Úsource_encodingsÚsource_lengthsÚtarget_encodingsÚtarget_lengthsr cCs:| d¡ ¡| d¡ ¡}| |¡}| |¡}|||fS)a›Forward pass for training. B: batch size; T: maximum source sequence length in batch; U: maximum target sequence length in batch; D: dimension of each source and target sequence encoding. Args: source_encodings (torch.Tensor): source encoding sequences, with shape `(B, T, D)`. source_lengths (torch.Tensor): with shape `(B,)` and i-th element representing valid sequence length of i-th batch element in ``source_encodings``. target_encodings (torch.Tensor): target encoding sequences, with shape `(B, U, D)`. target_lengths (torch.Tensor): with shape `(B,)` and i-th element representing valid sequence length of i-th batch element in ``target_encodings``. Returns: (torch.Tensor, torch.Tensor, torch.Tensor): torch.Tensor joint network output, with shape `(B, T, U, output_dim)`. torch.Tensor output source lengths, with shape `(B,)` and i-th element representing number of valid elements along dim 1 for i-th batch element in joint network output. torch.Tensor output target lengths, with shape `(B,)` and i-th element representing number of valid elements along dim 2 for i-th batch element in joint network output. r‚r:)Z unsqueezerr`r€)rr‡rˆr‰rŠZjoint_encodingsZactivation_outr rrrr!s"  z_Joiner.forward)rO) r"r#r$r%r&rlrr'r(rr!r)rrrrr„ˆs  úr„c sdeZdZdZeeeddœ‡fdd„ Zdej ej ej ej e e e ej e ej ej ej e e ej fdœdd„Z ejjej ej e e e ej e ej ej e e ej fd œd d „ƒZejjej ej e ej ej fd œd d„ƒZejjej ej e e e ej e ej ej e e ej fdœdd„ƒZejjej ej ej ej e ej ej ej fdœdd„ƒZ‡ZS)raXtorchaudio.models.RNNT() Recurrent neural network transducer (RNN-T) model. Note: To build the model, please use one of the factory functions. Args: transcriber (torch.nn.Module): transcription network. predictor (torch.nn.Module): prediction network. joiner (torch.nn.Module): joint network. N)Ú transcriberÚ predictorÚjoinerr cs tƒ ¡||_||_||_dSr)rrr‹rŒr)rr‹rŒrrrrrÓs z RNNT.__init__)ÚsourcesrˆÚtargetsrŠÚpredictor_stater c CsL|j||d\}}|j|||d\}}}|j||||d\}}}||||fS)aForward pass for training. B: batch size; T: maximum source sequence length in batch; U: maximum target sequence length in batch; D: feature dimension of each source sequence element. Args: sources (torch.Tensor): source frame sequences right-padded with right context, with shape `(B, T, D)`. source_lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``sources``. targets (torch.Tensor): target sequences, with shape `(B, U)` and each element mapping to a target symbol. target_lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``targets``. predictor_state (List[List[torch.Tensor]] or None, optional): list of lists of tensors representing prediction network internal state generated in preceding invocation of ``forward``. (Default: ``None``) Returns: (torch.Tensor, torch.Tensor, torch.Tensor, List[List[torch.Tensor]]): torch.Tensor joint network output, with shape `(B, max output source length, max output target length, output_dim (number of target symbols))`. torch.Tensor output source lengths, with shape `(B,)` and i-th element representing number of valid elements along dim 1 for i-th batch element in joint network output. torch.Tensor output target lengths, with shape `(B,)` and i-th element representing number of valid elements along dim 2 for i-th batch element in joint network output. List[List[torch.Tensor]] output states; list of lists of tensors representing prediction network internal state generated in current invocation of ``forward``. )rr©rrr9©r‡rˆr‰rŠ)r‹rŒr) rrŽrˆrrŠrr‡r‰r rrrr!Ùs(,þ ý ü üz RNNT.forward)rŽrˆr9r cCs|j |||¡S)a¸Applies transcription network to sources in streaming mode. B: batch size; T: maximum source sequence segment length in batch; D: feature dimension of each source sequence frame. Args: sources (torch.Tensor): source frame sequence segments right-padded with right context, with shape `(B, T + right context length, D)`. source_lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``sources``. state (List[List[torch.Tensor]] or None): list of lists of tensors representing transcription network internal state generated in preceding invocation of ``transcribe_streaming``. Returns: (torch.Tensor, torch.Tensor, List[List[torch.Tensor]]): torch.Tensor output frame sequences, with shape `(B, T // time_reduction_stride, output_dim)`. torch.Tensor output lengths, with shape `(B,)` and i-th element representing number of valid elements for i-th batch element in output. List[List[torch.Tensor]] output states; list of lists of tensors representing transcription network internal state generated in current invocation of ``transcribe_streaming``. )r‹rL)rrŽrˆr9rrrÚtranscribe_streamings#zRNNT.transcribe_streaming)rŽrˆr cCs | ||¡S)aÆApplies transcription network to sources in non-streaming mode. B: batch size; T: maximum source sequence length in batch; D: feature dimension of each source sequence frame. Args: sources (torch.Tensor): source frame sequences right-padded with right context, with shape `(B, T + right context length, D)`. source_lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``sources``. Returns: (torch.Tensor, torch.Tensor): torch.Tensor output frame sequences, with shape `(B, T // time_reduction_stride, output_dim)`. torch.Tensor output lengths, with shape `(B,)` and i-th element representing number of valid elements for i-th batch element in output frame sequences. )r‹)rrŽrˆrrrÚ transcribeAszRNNT.transcribe)rrŠr9r cCs|j|||dS)aApplies prediction network to targets. B: batch size; U: maximum target sequence length in batch; D: feature dimension of each target sequence frame. Args: targets (torch.Tensor): target sequences, with shape `(B, U)` and each element mapping to a target symbol, i.e. in range `[0, num_symbols)`. target_lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``targets``. state (List[List[torch.Tensor]] or None): list of lists of tensors representing internal state generated in preceding invocation of ``predict``. Returns: (torch.Tensor, torch.Tensor, List[List[torch.Tensor]]): torch.Tensor output frame sequences, with shape `(B, U, output_dim)`. torch.Tensor output lengths, with shape `(B,)` and i-th element representing number of valid elements for i-th batch element in output. List[List[torch.Tensor]] output states; list of lists of tensors representing internal state generated in current invocation of ``predict``. r‘)rŒ)rrrŠr9rrrÚpredict^s!z RNNT.predictr†cCs"|j||||d\}}}|||fS)a¶Applies joint network to source and target encodings. B: batch size; T: maximum source sequence length in batch; U: maximum target sequence length in batch; D: dimension of each source and target sequence encoding. Args: source_encodings (torch.Tensor): source encoding sequences, with shape `(B, T, D)`. source_lengths (torch.Tensor): with shape `(B,)` and i-th element representing valid sequence length of i-th batch element in ``source_encodings``. target_encodings (torch.Tensor): target encoding sequences, with shape `(B, U, D)`. target_lengths (torch.Tensor): with shape `(B,)` and i-th element representing valid sequence length of i-th batch element in ``target_encodings``. Returns: (torch.Tensor, torch.Tensor, torch.Tensor): torch.Tensor joint network output, with shape `(B, T, U, output_dim)`. torch.Tensor output source lengths, with shape `(B,)` and i-th element representing number of valid elements along dim 1 for i-th batch element in joint network output. torch.Tensor output target lengths, with shape `(B,)` and i-th element representing number of valid elements along dim 2 for i-th batch element in joint network output. r’)r)rr‡rˆr‰rŠr rrrÚjoins#ü z RNNT.join)N)r"r#r$r%rIror„rr'r(rrrr!rmrnr“r”r•r–r)rrrrrÅsF  úù Cû$üû"ú)r,Ú encoding_dimrprWrXrYrZr[r\r]rQrRr^rSrTrUrqrrrtrurvr cCsTt||||||||| | | | | ||d}t||||||||d}t||ƒ}t|||ƒS)a Builds Emformer-based recurrent neural network transducer (RNN-T) model. Note: For non-streaming inference, the expectation is for `transcribe` to be called on input sequences right-concatenated with `right_context_length` frames. For streaming inference, the expectation is for `transcribe_streaming` to be called on input chunks comprising `segment_length` frames right-concatenated with `right_context_length` frames. Args: input_dim (int): dimension of input sequence frames passed to transcription network. encoding_dim (int): dimension of transcription- and prediction-network-generated encodings passed to joint network. num_symbols (int): cardinality of set of target tokens. segment_length (int): length of input segment expressed as number of frames. right_context_length (int): length of right context expressed as number of frames. time_reduction_input_dim (int): dimension to scale each element in input sequences to prior to applying time reduction block. time_reduction_stride (int): factor by which to reduce length of input sequence. transformer_num_heads (int): number of attention heads in each Emformer layer. transformer_ffn_dim (int): hidden layer dimension of each Emformer layer's feedforward network. transformer_num_layers (int): number of Emformer layers to instantiate. transformer_left_context_length (int): length of left context considered by Emformer. transformer_dropout (float): Emformer dropout probability. transformer_activation (str): activation function to use in each Emformer layer's feedforward network. Must be one of ("relu", "gelu", "silu"). transformer_max_memory_size (int): maximum number of memory elements to use. transformer_weight_init_scale_strategy (str): per-layer weight initialization scaling strategy. Must be one of ("depthwise", "constant", ``None``). transformer_tanh_on_mem (bool): if ``True``, applies tanh to memory elements. symbol_embedding_dim (int): dimension of each target token embedding. num_lstm_layers (int): number of LSTM layers to instantiate. lstm_layer_norm (bool): if ``True``, enables layer normalization for LSTM layers. lstm_layer_norm_epsilon (float): value of epsilon to use in LSTM layer normalization layers. lstm_dropout (float): LSTM dropout probability. Returns: RNNT: Emformer RNN-T model. )r,rVrWrXrYrZr[r\r]rQrRr^rSrTrU)rqrrrsrtrurv)rMror„r)r,r—rprWrXrYrZr[r\r]rQrRr^rSrTrUrqrrrtrurvÚencoderrŒrrrrr ­s:Añø )rpr cCs2tdd|dddddddd d d d d dddddddS)z¸Builds basic version of Emformer RNN-T model. Args: num_symbols (int): The size of target token lexicon. Returns: RNNT: Emformer RNN-T model. éPiér0é€éiégš™™™™™¹?ZgeluérrPTiégü©ñÒMbP?g333333Ó?)r,r—rprWrXrYrZr[r\r]rQrRr^rSrTrUrqrrrtrurv)r )rprrrr s. ë)ÚabcrrÚtypingrrrr'Ztorchaudio.modelsrÚ__all__r2ÚModuler r*rIrMror„rr&rHrlrGr r rrrrÚsJ  ,R`=ké `