U %d" @sddlZddlmZmZmZddlZdgZejejdddZdejejejejejeejeejddd Z e ej j d d d Z ee eeeed ddZeeeeeejejdddZGdddej j ZGdddej j ZGdddej j ZGdddeZdS)N)ListOptionalTupleEmformer)lengthsreturncCsF|jd}tt|}tj||j|jd||| dk}|S)Nr)devicedtype) shapeinttorchmaxitemZarangerr expand unsqueeze)r batch_size max_length padding_maskr>/tmp/pip-unpacked-wheel-lbdmvq91/torchaudio/models/emformer.py_lengths_to_padding_mask s r) utterance right_contextsummaryrmemsleft_context_keyrc Cs|d|d|d}|d}|dkr6d}nR|t||d} |dk rd|dnd} ||d| | } t| d}|S)Nrr )r)sizer rr r) rrrrrrTBrright_context_blocks_lengthZleft_context_blocks_lengthZklengthsrrr_gen_padding_masks  r!) activationrcCsH|dkrtjS|dkr$tjS|dkr6tjStd|dS)NreluZgeluZsiluzUnsupported activation )r nnZReLUZGELUZSiLU ValueError)r"rrr_get_activation_module's   r&)weight_init_scale_strategy num_layersrcCs`|dkrddt|DS|dkr4ddt|DS|dkrNddt|DStd|dS)NcSsg|]}dqSNr).0_rrr 4sz*_get_weight_init_gains.. depthwisecSsg|]}dt|dqS)?r mathsqrtr* layer_idxrrrr,6sZconstantcSsg|]}dtdqS)r.r/r2rrrr,8sz-Unsupported weight_init_scale_strategy value )ranger%)r'r(rrr_get_weight_init_gains2sr6) col_widthscol_masknum_rowsrrcs@t|t|kstdfddt||D}tj|ddS)Nz0Length of col_widths must match that of col_maskcs4g|],\}}|r tj|dntj|dqS)r)r Zoneszeros)r*Z col_widthZ is_ones_colrr9rrr,Bsz-_gen_attention_mask_block..r dim)lenAssertionErrorzipr cat)r7r8r9rZ mask_blockrr<r_gen_attention_mask_block=s  rCcs<eZdZdZdeeeeeeedfdd Ze j e j e e j e j fd d d Z e j e j ee j e j d d dZ de j e j e j e j e j e j ee j ee j e e j e j e j e j fd ddZe j e j e j e j e j e j e e j e j fdddZe jje j e j e j e j e j e j e j e e j e j e j e j fdddZZS)_EmformerAttentiona_Emformer layer attention module. Args: input_dim (int): input dimension. num_heads (int): number of attention heads in each Emformer layer. dropout (float, optional): dropout probability. (Default: 0.0) weight_init_gain (float or None, optional): scale factor to apply when initializing attention module parameters. (Default: ``None``) tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) negative_inf (float, optional): value to use for negative infinity in attention weights. (Default: -1e8) NFח input_dim num_headsdropoutweight_init_gain tanh_on_mem negative_infcst||dkr,td|d|d||_||_||_||_||_|j|jd|_t j j |d|dd|_ t j j ||dd|_ t j j ||dd|_|rt j jj|j j|d t j jj|j j|d dS) Nrz input_dim (z") is not a multiple of num_heads (z).gr4T)Zbias)Zgain)super__init__r%rHrIrJrLrMscalingr r$Linearemb_to_key_value emb_to_queryout_projinitZxavier_uniform_Zweight)selfrHrIrJrKrLrM __class__rrrOXs  z_EmformerAttention.__init__)inputrrc CsX|j\}}}|dd}|d||}t||g}||jddd\}} || fS)Nrr r4chunksr>)r rr rBrRchunk) rVrYrrr+summary_lengthZright_ctx_utterance_blockZmems_right_ctx_utterance_blockkeyvaluerrr_gen_key_valuevs  z!_EmformerAttention._gen_key_value)attention_weightsattention_maskrrcCs|}||d|j}|d}|d|j}|dk r|||j|d}||ddtj |j}|||j|d}tj j j |dd |}tj j j|t|j|jdS)Nrr r4r=)ptraining)floatZ masked_fillrrMrrIviewtor boolr$Z functionalZsoftmaxZtype_asrJre)rVrarbrZattention_weights_floatrrattention_probsrrr_gen_attention_probs~s z'_EmformerAttention._gen_attention_probs) rrrrrrbrleft_context_valrc s|d|d|d|d} t|||g} t|||gjddd\} } |dk r|dk r| t||d} t| d|d| || |d| dg} t| d|d| || |d| dg} fdd| | | fD\}}}t|j | dd}t ||||||} |||}t||}|j j| jjfkst| dd| j}|}|d}|d| |}|| |d}jrt|}ntj|ddd }||| | fS) Nr rr4rZcs4g|],}|djjjddqS)rcrr ) contiguousrgrIrH transpose)r*ZtensorrrVrrr,sz4_EmformerAttention._forward_impl..i )minr)rrSr rBrRr\rr ZbmmrPrnr!rkr rIrHr@rmrgrTrLtanhclamp)rVrrrrrrbrrlrqueryr^r_r Zreshaped_queryZ reshaped_keyZreshaped_valuerarrj attentionZoutput_right_context_memsr]output_right_context output_memsrror _forward_implsN $       z _EmformerAttention._forward_impl)rrrrrrbrc Cs,|||||||\}}} } ||ddfS)acForward pass for training. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; S: number of summary elements; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. summary (torch.Tensor): summary elements, with shape `(S, B, D)`. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. attention_mask (torch.Tensor): attention mask for underlying attention module. Returns: (Tensor, Tensor): Tensor output frames corresponding to utterance and right_context, with shape `(T + R, B, D)`. Tensor updated memory elements, with shape `(M, B, D)`. Nrc)rx) rVrrrrrrboutputrwr+rrrforwards"z_EmformerAttention.forward)rrrrrrrlrc Cs|d|d|d}|d|d|d|d} t|| jtj|jd} d| dd|df<|j|||||| ||d\} } } }| | | |d|dd||d|ddfS)aForward pass for inference. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; S: number of summary elements; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. summary (torch.Tensor): summary elements, with shape `(S, B, D)`. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. left_context_key (torch.Tensor): left context attention key computed from preceding invocation. left_context_val (torch.Tensor): left context attention value computed from preceding invocation. Returns: (Tensor, Tensor, Tensor, and Tensor): Tensor output frames corresponding to utterance and right_context, with shape `(T + R, B, D)`. Tensor updated memory elements, with shape `(M, B, D)`. Tensor attention key computed for left context and utterance. Tensor attention value computed for left context and utterance. rr rTrcN)rrl)rr r;rhrirrx)rVrrrrrrrlZ query_dimZkey_dimrbryrwr^r_rrrinfers&)( z_EmformerAttention.infer)rENFrF)NN)__name__ __module__ __qualname____doc__r rfrrirOr Tensorrr`rkrxrzjitexportr| __classcell__rrrWrrDKsd"   J %rDc seZdZdZd'eeeeeeeeeeeed fd d Z eee j e e j d d d Ze e j ee j e j e j fdddZe j e j ee j e e j e e j dddZe j e j e j e j dddZe j e j ee j e j fdddZe j e j e j ee j e j fdddZe j e j e j e j ee j ee j e j fdddZe j e j e j e j ee e j ee j e j e e j fdd d!Ze j e j e j e j e j ee j e j e j fdd"d#Ze jje j e j e j ee e j e j ee j e j e e j e j fd$d%d&ZZS)(_EmformerLayera$Emformer layer that constitutes Emformer. Args: input_dim (int): input dimension. num_heads (int): number of attention heads. ffn_dim: (int): hidden layer dimension of feedforward network. segment_length (int): length of each input segment. dropout (float, optional): dropout probability. (Default: 0.0) activation (str, optional): activation function to use in feedforward network. Must be one of ("relu", "gelu", "silu"). (Default: "relu") left_context_length (int, optional): length of left context. (Default: 0) max_memory_size (int, optional): maximum number of memory elements to use. (Default: 0) weight_init_gain (float or None, optional): scale factor to apply when initializing attention module parameters. (Default: ``None``) tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) negative_inf (float, optional): value to use for negative infinity in attention weights. (Default: -1e8) rEr#rNFrF) rHrIffn_dimsegment_lengthrJr"left_context_lengthmax_memory_sizerKrLrMc stt|||| | | d|_tj||_tjj||dd|_ t |} tj tj |tj ||| tj|tj ||tj||_tj ||_tj ||_||_||_||_||_|dk|_dS)NrGTZ kernel_sizeZstrideZ ceil_moder)rNrOrDrur r$ZDropoutrJ AvgPool1d memory_opr&Z SequentialZ LayerNormrQpos_fflayer_norm_inputlayer_norm_outputrrrrHuse_mem) rVrHrIrrrJr"rrrKrLrMZactivation_modulerWrrrOPs6      z_EmformerLayer.__init__)rrrcCsbtj|j||j|d}tj|j||j|d}tj|j||j|d}tjd|tj|d}||||gS)Nr:r r{)r r;rrHrZint32)rVrrZ empty_memoryrrl past_lengthrrr _init_state~s z_EmformerLayer._init_state)statercCs|ddd}t|j|}t|jt||j}|d|j|d}|d|j|d}|d|j|d}|||fS)Nrr r4)rrqrrr0ceilr)rVrrZpast_left_context_lengthZpast_mem_lengthpre_memslc_keylc_valrrr _unpack_states z_EmformerLayer._unpack_state)next_knext_v update_lengthrrrcCst|d|g}t|d|g}t|d|g|j d|d<||jd|jd|d<||jd|jd|d<|d||d<|S)Nr r4rr)r rBrr r)rVrrrrrZnew_kZnew_vrrr _pack_states"z_EmformerLayer._pack_state) rc_outputrrrcCs4||t||g}|||}||}|Sr))rJr rBrr)rVrrrresultrrr_process_attention_outputs z(_EmformerLayer._process_attention_output)rrrcCs8|t||g}||dd|d|dfSNr)rr rBr)rVrrrrrr_apply_pre_attention_layer_normsz._EmformerLayer._apply_pre_attention_layer_normcCs2||||}||dd|d|dfSr)rr)rVrrrrrr_apply_post_attention_ffnsz(_EmformerLayer._apply_post_attention_ffn)rrrrrbrc Csp|dkrtd|jr6||dddddd}ntdj|j|jd}|j ||||||d\}}||fS)Nz;attention_mask must be not None when for_inference is Falser r4rr{)rrrrrrb) r%rrpermuter emptyrhr rru) rVrrrrrbrrnext_mrrr_apply_attention_forwards  z'_EmformerLayer._apply_attention_forward)rrrrrrc Cs|dkr|j|d|jd}||\}}}|jr`||dddddd} | dd} ntdj |j |jd} |j j |||| |||d\} } } } | | | |d||}| | |fS)Nr r:r4rr{)rrrrrrrl)rrrrrrrr rrhr rur|r)rVrrrrrrrrrrrrrrrr_apply_attention_infers$ z%_EmformerLayer._apply_attention_inferc CsB|||\}}||||||\}} ||||\} } | | | fS)a1Forward pass for training. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. attention_mask (torch.Tensor): attention mask for underlying attention module. Returns: (Tensor, Tensor, Tensor): Tensor encoded utterance frames, with shape `(T, B, D)`. Tensor updated right context frames, with shape `(R, B, D)`. Tensor updated memory elements, with shape `(M, B, D)`. )rrr) rVrrrrrblayer_norm_utterancelayer_norm_right_contextrrwoutput_utterancervrrrrzs$ z_EmformerLayer.forward)rrrrrrc CsF|||\}}||||||\}} } ||||\} } | | | | fS)a2Forward pass for inference. B: batch size; D: feature dimension of each frame; T: number of utterance frames; R: number of right context frames; M: number of memory elements. Args: utterance (torch.Tensor): utterance frames, with shape `(T, B, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``utterance``. right_context (torch.Tensor): right context frames, with shape `(R, B, D)`. state (List[torch.Tensor] or None): list of tensors representing layer internal state generated in preceding invocation of ``infer``. mems (torch.Tensor): memory elements, with shape `(M, B, D)`. Returns: (Tensor, Tensor, List[torch.Tensor], Tensor): Tensor encoded utterance frames, with shape `(T, B, D)`. Tensor updated right context frames, with shape `(R, B, D)`. List[Tensor] list of tensors representing layer internal state generated in current invocation of ``infer``. Tensor updated memory elements, with shape `(M, B, D)`. )rrr) rVrrrrrrrrrw output_staterrvrrrr|s)  z_EmformerLayer.infer)rEr#rrNFrF)r}r~rrr rfstrrrirOr rrrrrrrrrrrrrzrrr|rrrrWrr=s.&         / rc seZdZdejjeeeedfdd ZejejdddZ eee edd d Z ejejdd d Z ejeje ejejfd ddZejjdejejee e eje ejeje e ejfdddZZS) _EmformerImplr)emformer_layersrrright_context_lengthrcsJt|dk|_tjj||dd|_||_||_||_ ||_ ||_ dS)NrTr) rNrOrr r$rrrrrrr)rVrrrrrrWrrrONs  z_EmformerImpl.__init__)rYrcCs|jd}t||j|j}g}t|dD].}|d|j}||j}||||q0||||jdt|SNrr ) r r0rrrr5appendr rB)rVrYrnum_segsZright_context_blocksseg_idxstartendrrr_gen_right_contextds  z _EmformerImpl._gen_right_context)rutterance_lengthrc Cst||j}|j}|j}||}||}t||j|d}t|d|j|} |j|} |jrt||jd} |d} | || | |||| ||| ||| g } n||| ||| ||| g} | Sr) r0rrrrrrqrr)rVrrrrclcZrc_startZrc_endZ seg_startZseg_endZ rc_lengthZm_startZ mem_lengthr7rrr_gen_attention_mask_col_widthsos:   z,_EmformerImpl._gen_attention_mask_col_widthsc Cs*|d}t||j}g}g}g}|jr`d}ddt|D}ddt|D} |||g} n"d}ddt|D}d} ||g} t|D]z} || |} t| ||j|j } | | t| |t |j|| |j|j }| || dk rt| | d|j }| |qdt d d| Dt j}|S) Nr cSsg|] }|dkqS))r rr*idxrrrr,sz5_EmformerImpl._gen_attention_mask..cSsg|] }|dkqS))rrrrrrrr,scSsg|] }|dkqS))r rrrrrrr,sr cSsg|]}t|qSr)r rB)r*maskrrrr,s)rr0rrrr5rrCrrrrqr rBrhri)rVrYrrZrc_maskZ query_maskZ summary_maskZnum_colsZrc_q_cols_maskZ s_cols_maskZmasks_to_concatrr7Z rc_mask_blockZquery_mask_blockZsummary_mask_blockrbrrr_gen_attention_masksL         z!_EmformerImpl._gen_attention_mask)rYrrc Cs|ddd}||}|d|d|j}||}|jrf||ddddddddntdj |j |j d}|}|j D]}||||||\}}}q|ddd|fS)aGForward pass for training and non-streaming inference. B: batch size; T: max number of input frames in batch; D: feature dimension of each frame. Args: input (torch.Tensor): utterance frames right-padded with right context frames, with shape `(B, T + right_context_length, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid utterance frames for i-th batch element in ``input``. Returns: (Tensor, Tensor): Tensor output frames, with shape `(B, T, D)`. Tensor output lengths, with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in output frames. r rr4Nrcr{) rrrrrrrr rrhr rr) rVrYrrrrbrrylayerrrrrzs  ( z_EmformerImpl.forwardN)rYrstatesrcCs$|d|j|jks:td|j|jd|dd|ddd}|d|j}||d}|d|}tj||jdd}|jr||ddddddnt dj |j |j d }|} g} t |jD]<\} } | | |||dkrdn|| |\} }} }| | q| ddd|| fS) aForward pass for streaming inference. B: batch size; D: feature dimension of each frame. Args: input (torch.Tensor): utterance frames right-padded with right context frames, with shape `(B, segment_length + right_context_length, D)`. lengths (torch.Tensor): with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in ``input``. states (List[List[torch.Tensor]] or None, optional): list of lists of tensors representing internal state generated in preceding invocation of ``infer``. (Default: ``None``) Returns: (Tensor, Tensor, List[List[Tensor]]): Tensor output frames, with shape `(B, segment_length, D)`. Tensor output lengths, with shape `(B,)` and i-th element representing number of valid frames for i-th batch element in output frames. List[List[Tensor]] output states; list of lists of tensors representing internal state generated in current invocation of ``infer``. r zIPer configured segment_length and right_context_length, expected size of z# for dimension 1 of input, but got .rr4N)rqr{)rrrr@rr rsrrrrhr r enumeraterr|r)rVrYrrZright_context_start_idxrrZoutput_lengthsrryZ output_statesr3rrrrrr|s0     z_EmformerImpl.infer)rrr)N)r}r~rr r$ ModuleListr rOrrrrrrrzrrrr|rrrrWrrMs, $0"#rcsDeZdZdZd eeeeeeeeeeeeeed fd d Z Z S) raImplements the Emformer architecture introduced in *Emformer: Efficient Memory Transformer Based Acoustic Model for Low Latency Streaming Speech Recognition* [:footcite:`shi2021emformer`]. Args: input_dim (int): input dimension. num_heads (int): number of attention heads in each Emformer layer. ffn_dim (int): hidden layer dimension of each Emformer layer's feedforward network. num_layers (int): number of Emformer layers to instantiate. segment_length (int): length of each input segment. dropout (float, optional): dropout probability. (Default: 0.0) activation (str, optional): activation function to use in each Emformer layer's feedforward network. Must be one of ("relu", "gelu", "silu"). (Default: "relu") left_context_length (int, optional): length of left context. (Default: 0) right_context_length (int, optional): length of right context. (Default: 0) max_memory_size (int, optional): maximum number of memory elements to use. (Default: 0) weight_init_scale_strategy (str or None, optional): per-layer weight initialization scaling strategy. Must be one of ("depthwise", "constant", ``None``). (Default: "depthwise") tanh_on_mem (bool, optional): if ``True``, applies tanh to memory elements. (Default: ``False``) negative_inf (float, optional): value to use for negative infinity in attention weights. (Default: -1e8) Examples: >>> emformer = Emformer(512, 8, 2048, 20, 4, right_context_length=1) >>> input = torch.rand(128, 400, 512) # batch, num_frames, feature_dim >>> lengths = torch.randint(1, 200, (128,)) # batch >>> output, lengths = emformer(input, lengths) >>> input = torch.rand(128, 5, 512) >>> lengths = torch.ones(128) * 5 >>> output, lengths, states = emformer.infer(input, lengths, None) rEr#rr-FrF) rHrIrr(rrJr"rrrr'rLrMc sVt| | tj f ddt|D}tj|| ddS)Ncs.g|]&}t | d qS))rJr"rrrKrLrM)rr2 r"rJrrHrrrMrIrrLZweight_init_gainsrrr,Usz%Emformer.__init__..)rrr)r6r r$rr5rNrO)rVrHrIrr(rrJr"rrrr'rLrMrrWrrrOCs zEmformer.__init__)rEr#rrrr-FrF) r}r~rrr rfrrrirOrrrrWrr#s0&)N)r0typingrrrr __all__rrr!rr$Moduler&r rfr6rirrCrDrrrrrrrs<    sW