U %d?@sddlZddlmZmZmZddlZddlmZmZddlm Z m Z e e Z GdddejZGddde ZGd d d e ZGd d d e ZGd dde ZGddde ZGddde ZGddde ZGddde ZGddde ZeeeeeefeedddZeeeeeeeeeeeeeedddZd1eeefeeeeeeeeeed" d#d$Zeeed%d&d'ZGd(d)d)e Z eee ed*d+d,Z!Gd-d.d.e Z"Gd/d0d0ej#j$Z%dS)2N)ListOptionalTuple)nnTensor)Module Parameterc@s eZdZdZeedddZdS) LayerNormzLayer norm with transpose)inputreturncCs8|dd}tj||j|j|j|j}|dd}|S)N) transposer functional layer_normnormalized_shapeweightbiasZeps)selfr xrI/tmp/pip-unpacked-wheel-lbdmvq91/torchaudio/models/wav2vec2/components.pyforwards  zLayerNorm.forwardN)__name__ __module__ __qualname____doc__rrrrrrr sr csVeZdZdZeeeeeeedfdd Ze ee e e ee fdddZ Z S)ConvLayerBlockz$Convolution unit of FeatureExtractor in_channels out_channels kernel_sizestriderrcs6t||_||_||_tj|||||d|_dS)N)rr r!r"r)super__init__r!r"rrConv1dconv)rrr r!r"rr __class__rrr$s zConvLayerBlock.__init__rlengthr cCsh||}|jdk r||}tj|}|dk r`tj||j|jddd}t t ||}||fS)a# Args: x (Tensor): Shape: ``[batch, in_channels, in_frame]``. length (Tensor or None, optional): Shape ``[batch, ]``. Returns: Tensor: Shape ``[batch, out_channels, out_frames]``. Optional[Tensor]: Shape ``[batch, ]``. Nfloor)Z rounding_mode) r&rrrgelutorchdivr!r"maxZ zeros_like)rrr*rrrr-s    zConvLayerBlock.forward) rrrrintboolrrr$rrr __classcell__rrr'rrsrcsJeZdZdZejdfdd Zeeee eeefdddZ Z S)FeatureExtractorzoExtract features from audio Args: conv_layers (nn.ModuleList): convolution layers ) conv_layerscst||_dSN)r#r$r5)rr5r'rrr$Ns zFeatureExtractor.__init__r)cCsJ|jdkrtd|d}|jD]}|||\}}q"|dd}||fS)a Args: x (Tensor): Input Tensor representing a batch of audio, shape: ``[batch, time]``. length (Tensor or None, optional): Valid length of each input sample. shape: ``[batch, ]``. Returns: Tensor: The resulting feature, shape: ``[batch, frame, feature]`` Optional[Tensor]: Valid length of each output sample. shape: ``[batch, ]``. zNExpected the input Tensor to be 2D (batch, time), but received {list(x.shape)}r,)ndim ValueError unsqueezer5r)rrr*layerrrrrUs    zFeatureExtractor.forward) rrrrr ModuleListr$rrrrr3rrr'rr4Fs  r4cs2eZdZdZeeedfdd ZddZZS)FeatureProjectionzLayer that connects FeatureExtractor and Encoder Projects features to encoder dimension. Args: in_features (int): Input feature dim. out_features (int): Output feature dim. dropout (float): Dropout probability. ) in_features out_featuresdropoutcs4tt||_t|||_t||_dSr6) r#r$rr rLinear projectionDropoutr@)rr>r?r@r'rrr$}s  zFeatureProjection.__init__cCs"||}||}||}|S)z Args: x (Tensor): Feature Tensor. shape: ``[batch, frame, in_feature]`` Returns: Tensor: Projected features. ``[batch, frame, out_feature]``. )rrBr@rrrrrrs   zFeatureProjection.forward rrrrr1floatr$rr3rrr'rr=rs  r=cs:eZdZdZeeedfdd ZddZddZZS) ConvolutionalPositionalEmbeddingaPositional embedding which is placed at the beginning of Transformer. Args: embed_dim (int): Feature dimension of the input Tensor. kernel_size (int): The number of frames to be use. groups (int): The number of groups in feature dimensions. ) embed_dimr!groupscsZt||_tj||||d|d|_tjj|jddd|_|ddkrPdnd|_dS)Nr7)rr r!paddingrIr)namedimrr,) r#r$rHrr%r&utilsZ weight_norm num_remove)rrHr!rIr'rrr$s z)ConvolutionalPositionalEmbedding.__init__cCsL|jjD]:}|jdkr |jjdkr td|jjtj j |jq |S)Nztorch.nn.utils.weight_normZ WeightNormzRemoving weight_norm from %s) r&Z_forward_pre_hooksvaluesrr(r_LGwarningr.rrMZremove_weight_norm)rhookrrr__prepare_scriptable__s z7ConvolutionalPositionalEmbedding.__prepare_scriptable__cCsR|dd}||}|jdkr4|dd|j f}tjj|}|dd}|S)z Args: x (Tensor): shape ``[batch, frame, feature]``. Returns: Tensor: The resulting feature. Shape ``[batch, frame, feature]``. r r r.N)rr&rNr.rrr-rDrrrrs    z(ConvolutionalPositionalEmbedding.forward) rrrrr1r$rSrr3rrr'rrGs  rGcsDeZdZdZd eeedfdd Zd eeeeddd Z Z S) SelfAttentionaMultihead Self Attention module Args: embed_dim (int): Total dimension of the model. num_heads (int): The number of heads. dropout (float, optional): Dropout probabiliry on attn_output_weights. Default: ``0.0`` rH num_headsr@cst||}|||kr4td|d|d||_||_tj||_||_ |j d|_ tj ||dd|_ tj ||dd|_ tj ||dd|_tj ||dd|_dS)Nz `embed_dim (z#)` is not divisible by `num_heads (z)`gT)r)r#r$r9rHrWr.rrCr@head_dimscalingrAk_projv_projq_projout_proj)rrHrWr@rXr'rrr$s   zSelfAttention.__init__N)rattention_maskr c Cs>|jdks|jd|jkr4td|jd|jd|\}}}|dk r||d||f}||kr|td|d |d|||j|jf}||j| dd}| |j| d ddd} | |j| dd} |j || } |dk r| |7} tjjj| d d } || } | | } | dd|||} || } | S) aS Args: x (Tensor): shape: ``[batch_size, sequence_length, embed_dim]``. attention_mask (Tensor or None, optional): shape: ``[batch_size, 1, sequence_length, sequence_length]`` Returns: Tensor: The resulting tensor. shape: ``[batch, sequence_length, embed_dim]`` r7z9The expected input shape is (batch, sequence, embed_dim==z ). Found .Nr,z%The expected attention mask shape is z. Found rr rL)r8shaperHr9sizerWrXr\viewrrZZpermuter[rYr.rrZsoftmaxr@Zreshaper]) rrr^ batch_sizer*rHZshape_rbqkvweightsoutputrrrrs,    zSelfAttention.forward)rU)N) rrrrr1rFr$rrrr3rrr'rrTs rTcs4eZdZdZeeeedfdd ZddZZS) FeedForwardz4Layer that follows attention layer in encoder layer. io_featuresintermediate_featuresintermediate_dropoutoutput_dropoutcsBtt|||_t||_t|||_t||_dSr6) r#r$rrAintermediate_denserCro output_denserp)rrmrnrorpr'rrr$s   zFeedForward.__init__cCs:||}tjj|}||}||}||}|S)z Args: x (Tensor): shape: `(batch, sequence_length, io_features)` Returns: x (Tensor): shape: `(batch, sequence_length, io_features)` )rqr.rrr-rorrrprDrrrr,s     zFeedForward.forwardrErrr'rrks rkcsBeZdZdZeeeedfdd Zd ee edddZ Z S) EncoderLayerzLA layer unit in encoder. Combines multihead self attention and feed forward. attentionr@layer_norm_first feed_forwardcsHt||_t||_t|j|_||_ ||_ t|j|_ dSr6) r#r$rurrCr@r rHrrvrwfinal_layer_norm)rrur@rvrwr'rrr$?s  zEncoderLayer.__init__Nrr^cCsp|}|jr||}|||}||}||}|jrN||||}n||}||||}|S)z Args: x (Tensor): shape: `(batch, sequence_length, embed_dim)` attention_mask (Tensor or None, optional): shape: `(batch, 1, sequence_length, sequence_length)` )rvrrur@rwrx)rrr^ZresidualrrrrNs     zEncoderLayer.forward)N) rrrrrrFr2r$rrrr3rrr'rrs<srscspeZdZeeeeedfdd ZedddZdee edd d Z dee ee e e ed d d Z ZS) Transformerpos_conv_embedr@layersrv layer_dropcs@t||_t|j|_||_||_t ||_ ||_ dSr6) r#r$r|rr rHrrvr~rCr@r})rr|r@r}rvr~r'rrr$ks  zTransformer.__init__)rcCs,|||}|jr||}||}|Sr6)r|rvrr@rDrrr _preprocess{s   zTransformer._preprocessNrycCsN||}|jD](}|jr.td|jks|||}q|jsJ||}|S)Nr,) rr}Ztrainingr.randitemr~rvr)rrr^r;rrrrs    zTransformer.forward)rr^ num_layersr cCs|dk r:d|kr"t|jks:ntdt|jdg}||}|jD]4}|||}|||dk rNt||krN|SqN|S)Nrz!`num_layers` must be between [1, ])lenr}r9rappend)rrr^rretr;rrrget_intermediate_outputss     z$Transformer.get_intermediate_outputs)N)NN)rrrrrFr2r$rrrrr1rrr3rrr'rrzjs(  rzcseZdZeedfdd Zd eeeeeeefdddZdeeeeddd Z deeeee e ed d d Z Z S)Encoder)feature_projection transformercst||_||_dSr6)r#r$rr)rrrr'rrr$s zEncoder.__init__N)featureslengthsr cCs||}d}|dk r|j\}}}tj||jd|||dddfk}d||<d|ddddddfj|jd}||d||}||fS)NdevicerUgdtyper,)rrbr.arangerexpandtor)rrrrmaskremax_len_rrrrs  (&zEncoder._preprocesscCs"|||\}}|j||d}|S)N)r^)rr)rrrrrrrrrszEncoder.forward)rrrr cCs"|||\}}|jj|||dS)N)r^r)rrr)rrrrrmasksrrrextract_featuresszEncoder.extract_features)N)N)NN)rrrrr$rrrrrr1rrr3rrr'rrs.   r) norm_modeshapesrr c Cs|dks tg}d}t|D]f\}\}}}d} |dkrP|dkrPtj||dd} n|dkrdt|dd } |t|||||| d |}qtt|S) a Args: norm_mode (str): Either "group_norm" or "layer_norm". If "group_norm", then a single normalization is applied in the first convolution block. Otherwise, all the convolution blocks will have layer normalization. This option corresponds to "extractor_mode" from fairseq. Expected values are "group_norm" for Base arch, and "layer_norm" for Large arch. shapes (list of tuple of int): Configuration of convolution layers. List of convolution configuration, i.e. ``[(output_channel, kernel_size, stride), ...]`` This option corresponds to "conv_feature_layers" from fairseq. Expected values are ``[(512, 10, 5)] + [(512, 3, 2)] * 4 + [(512, 2, 2)] * 2`` for all the architectures. bias (bool): Whether to include bias term to each convolution operation. This option corresponds to "conv_bias" from fairseq. Expected values are False for Base arch, and True for Large arch. See Also: * Original implementation https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L666-L733 * "extractor_mode" - Def and base: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L38-L45 - Large: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L52 * "conv_feature_layers" - Def, base and large: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L94-L100 * "conv_bias" - Def and base: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L101-L103 - Large: https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L61 ) group_normrr,NrrT)Z num_groupsZ num_channelsZaffiner)rZelementwise_affiner) AssertionError enumeraterZ GroupNormr rrr4r<) rrrblocksrir r!r"Z normalizationrrr_get_feature_extractors8,  r)r>rH dropout_inputpos_conv_kernelpos_conv_groupsrrWattention_dropoutff_interm_featuresff_interm_dropoutr@rvr~r c  Cst|||} t|||}t}t|D]8}t|||d}t||| | d}|t|| | |dq(t || || | d}t | |S)a Args: in_features (int): The number of input features. embed_dim (int): The dimension of embedding. This option corresponds to "encoder_embed_dim" from fairseq. Expected values are 768 for Base arch, and 1024 for Large arch. dropout_input (float): The dropout probability applied after the input feature is projected to ``embed_dim``. This option corresponds to "dropout_input" from fairseq. Expected values are 0.1 for both Base and Large arch. pos_conv_kernel (int): The kernel size of convolutional positional embeddings. This option corresponds to "conv_pos" from fairseq. Expected values are 128 for both Base and Large arch. pos_conv_groups (int): The number of groups of convolutional positional embeddings. This option corresponds to "conv_pos_groups" from fairseq. Expected values are 16 for both Base and Large arch. num_layers (int): The number of self attention layers in transformer block. This option corresponds to "encoder_layers" from fairseq. Expected values are 12 for Base and 24 for Large arch. num_heads (int): The number of heads in self attention layers. This option corresponds to "encoder_attention_heads" from fairseq. Expected values are 12 for Base and 16 for Large arch. attention_dropout (float): The dropout probability applied after softmax in self-attention layer. This option corresponds to "attention_dropout" from fairseq. Expected values are 0.1 for Base and 0.0 for Large arch. ff_interm_features (int): The dimension of hidden features in feed forward layer. This option corresponds to "encoder_ffn_embed_dim" from fairseq. Expected values are 3072 for Base and 4096 for Large arch. ff_interm_dropout (float): The dropout probability applied in feedforward layer. This option correspinds to "activation_dropout" from fairseq. Expected values are 0.1 for both Base and Large arch. dropout (float): The dropout probability applied at the end of feed forward layer. This option corresponds to "dropout" from fairseq. Expected values are 0.1 for Base and 0.0 for Large arch. layer_norm_first (bool): Control the order of layer norm in transformer layer and each encoder layer. If True, in transformer layer, layer norm is applied before features are fed to encoder layers. In encoder layer, two layer norms are applied before and after self attention. If False, in transformer layer, layer norm is applied after features are fed to encoder layers. In encoder layer, two layer norms are applied after self attention, before and after feed forward. This option corresponds to "layer_norm_first" from fairseq. Expected values are False for Base and True for Large arch. layer_drop (float): Probability to drop each encoder layer during training. This option corresponds to "layerdrop" from fairseq. Expected values are 0.1 for both Base and Large arch. See Also: * "encoder_embed_dim" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L49-L51 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L64 * "dropout_input" - Def, base and large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L75-L78 * "conv_pos" - Def, base and large NOTE: The description is wrong. https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L204-L207 - Usage https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L756 * "conv_pos_groups" - Def, base and large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L208-L211 * "encoder_layers" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L46-L48 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L63 * "encoder_attention_heads" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L55-L57 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L66 * "attention_dropout" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L66-L68 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L60 * "encoder_ffn_embed_dim" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L52-L54 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L65 * "activation_dropout" - Def https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L69-L71 - Base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/finetuning/base_960h.yaml#L55 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/finetuning/vox_960h.yaml#L55 * "dropout" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L63-L65 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L59 * "layer_norm_first" - Def and base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L91-L93 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/pretraining/wav2vec2_large_librivox.yaml#L53 * "layerdrop" - Def https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/fairseq/models/wav2vec/wav2vec2.py#L72-L74 - Base https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/finetuning/base_960h.yaml#L54 - Large https://github.com/pytorch/fairseq/blob/425c36eafff535fe7337f8bdd5ace22ebacc78cb/examples/wav2vec/config/finetuning/vox_960h.yaml#L54 rVrlrtr{) r=rGrr<rangerTrkrrsrzr)r>rHrrrrrWrrrr@rvr~rZpos_convZencoder_layersrrurwrrrr _get_encoder!s@   rstaticrUF) rb padding_mask mask_prob mask_length mask_type mask_other min_masks no_overlap min_spacer c  s |\} } t| | fd} t|| t|td} t|| } g} t| D]d}|dk r| || }t||t|td}t||}n| }| }|dkrt|f|n|dkrtj ||dd|fdnr|dkrtj |||fdt t dtn8|d krDtj||fdtntd |td krrt||dd <|r:gfd d }d |fg}t}tddD]}tjdd|Dtjd}d |||k<|}|d krq.||}tjj|}||\}}||||||qtn`t}|||kr\||d}tjt ||f|ddtfddttD| t|kqLtdd| D}t| D]L\}t|kr td tjt j d f|ddd| |f<q| S)aComputes random mask spans for a given shape. Args: shape (int, int): The shape for which to compute masks. The first element is batch size and second is the number of frames. padding_mask (Tensor or None): The padding mask of the same dimension as shape, which will prevent masking padded elements. mask_prob (float): Probability for each token to be chosen as start of the span to be masked. This will be multiplied by number of timesteps divided by length of mask span to mask approximately this percentage of all elements. However due to overlaps, the actual number will be smaller (unless no_overlap is True). mask_type (str): How to compute mask lengths. Options: [``static``, ``uniform``, ``normal``, ``poisson``]. ``static``: Fixed size ``uniform``: Sample from uniform distribution [mask_other, mask_length*2] ``normal``: Sample from normal distribution with mean ``mask_length`` and stdev ``mask_other``. ``poisson``: Sample from possion distribution with lambda = ``mask_length``. min_masks (int): Minimum number of masked spans. no_overlap (bool): If false, will switch to an alternative recursive algorithm that prevents spans from overlapping. min_space (int): How many frames to keep unmasked between spans (Only used if no_overlap is True). Returns: (Tensor): The mask indices of dimension `[batch, frame]`. Fr,Nruniformr7rcnormalpoissonzunknown mask selection: rcstj|||ddfddt|Dg}||krZ||df|||kr|||f|S)N)r,rc3s|]}|VqdSr6r).0rZ span_startrr sz9_compute_mask_indices..arrange..r,)r.randintextendrr)ser*Z keep_lengthZ new_parts)mask_idcrrrarrangesz&_compute_mask_indices..arrangeT)reversecSsg|]\}}||qSrr)rrrrrr $sz)_compute_mask_indices..r)Z num_samples replacementcs*g|]"}t|D]}||qqSr)r)rjoffset)rrrrr6scSsg|] }t|qSr)r)rmrrrr;s)!r.fullr1rFrr0rlongsumrrrmaximumZonesroundr ExceptionminsortedZtensor distributionsZ categoricalZ CategoricalsamplepoprZ multinomialrruniquer index_selectrb)rbrrrrrrrrreframerZ all_num_maskZ mask_idcsrszZnum_maskrpartsZ min_lengthr*ZlensZl_sumZprobscrrZmin_lenr)rrrr_compute_mask_indicess~#         r)r rr cCs8|j\}}}tj||jd|||dddfk}|S)a&Generate the padding mask given the padded input and the lengths Tensors. Args: input (Tensor): The padded Tensor of dimension `[batch, max_len, frequency]`. lengths (Tensor): The lengths Tensor of dimension `[batch,]`. Returns: (Tensor): The padding mask. rN)rbr.rrr)r rrerrrrrr_get_padding_maskLs (rcsTeZdZdZeeeeeeeeeeeeed fdd Ze e e e dddZ Z S) MaskGeneratoraAGenerate the masks for masked prediction. Args: encoder_embed_dim (int): The dimension of the transformer embedding output. mask_prob (float): Probability for each token to be chosen as start of the span to be masked. This will be multiplied by number of timesteps divided by length of mask span to mask approximately this percentage of all elements. However due to overlaps, the actual number will be smaller (unless no_overlap is True). mask_selection (str): How to choose the mask length. Options: [``static``, ``uniform``, ``normal``, ``poisson``]. mask_other (float): Secondary mask argument (used for more complex distributions). mask_length (int): The lengths of the mask. no_mask_overlap (bool): Whether to allow masks to overlap. mask_min_space (int): Minimum space between spans (if no overlap is enabled). mask_channel_prob (float): The probability of replacing a feature with 0. mask_channel_selection (str): How to choose the mask length for channel masking. Options: [``static``, ``uniform``, ``normal``, ``poisson``]. mask_channel_other (float): Secondary mask argument for channel masking(used for more complex distributions). mask_channel_length (int): Minimum space between spans (if no overlap is enabled) for channel masking. no_mask_channel_overlap (bool): Whether to allow channel masks to overlap. mask_channel_min_space (int): Minimum space between spans for channel masking(if no overlap is enabled). ) encoder_embed_dimrmask_selectionrrno_mask_overlapmask_min_spacemask_channel_probmask_channel_selectionmask_channel_othermask_channel_lengthno_mask_channel_overlapmask_channel_min_spacecsvt||_||_||_||_||_||_||_| |_ | |_ | |_ | |_ | |_ tt||_tjj|jdSr6)r#r$rrrrrrrrrrrrrr. FloatTensormask_embeddingrinituniform_)rrrrrrrrrrrrrrr'rrr$qs zMaskGenerator.__init__)rrr c Cs|j\}}}|jdkrXt||f||j|j|j|jd|j|jd }||j }|j ||<nd}|j dkrt||fd|j |j |j |j|j|jd}||j dd|d}d||<||fS)a Args: x (Tensor): The encoded representations after feature extraction module. padding_mask (Tensor or None): The padding mask of the same dimension as shape, which will prevent masking padded elements. Returns: Tensor: The feature representations after masking. Tensor: The generated mask indices. rr7)rrrN)rrr,r )rbrrrrrrrrrrrrrrrrr:r)rrrBTCZ mask_indicesZmask_channel_indicesrrrrs<      zMaskGenerator.forward) rrrrr1rFstrr2r$rrrr3rrr'rrZs  r)proj_xtargetlabel_embeddingsr c Csd}t|d|}|dd|dd}||kd}|d}tj||gdd}tj| | dd |}||}| rt d|dd|<| dd}|S)aCompute the logits of the embeddings. Args: proj_x (Tensor): The projected masked representations of dimension `[batch, frame, final_dim]`. target (Tensor): The target Tensor of dimension `[batch, frame, final_dim]`. label_embeddings (Parameter): The trainable embeddings of target of dimension `[num_class, final_dim]`. Returns: (Tensor): The logits of the inputs. g?rr,r raz-infN) r.rrr:rrcallcatZcosine_similarityrFZtype_asanyr) rrrZ logit_tempposZnegsZ neg_is_postargetsZlogitsrrr_compute_logitss  rcsLeZdZdZeeeeedfdd ZeeeeeeefdddZ Z S)LogitGeneratoraGenerate the logits of masked and unmasked inputs. Args: encoder_embed_dim (int): The dimension of the transformer embedding output. num_classes (int): The number of classes in the labels. final_dim (int): Project final representations and targets to `final_dim`. skip_masked (bool): If True, skip computing losses over masked frames. skip_nomask (bool): If True, skip computing losses over unmasked frames. )r num_classes final_dim skip_masked skip_nomaskcsLttt|||_tjj|jtj |||_ ||_ ||_ dSr6) r#r$rr.rrrrrrA final_projrr)rrrrrrr'rrr$s  zLogitGenerator.__init__)rlabelmask_mmask_ur c Csf||}|jrd}n||}||}t|||j}|jr@d} n||} ||} t| | |j} || fS)a/ Args: x (Tensor): The feature representation of the last transformer layer. label (Tensor): The label Tensor of dimension `[batch, frame]`. mask_m (Tensor): The masked indices of dimension `[batch, frame]`. mask_u (Tensor): The unmasked indices of dimension `[batch, frame]`. Returns: Tensor: The logits of masked frames. Tensor of dimension `[masked_frame, final_dim]`. Tensor: The logits of unmasked frames. Tensor of dimension `[unmasked_frame, final_dim]`. N)rrrrr) rrrrrrZlogit_mZproj_x_mZlabel_mZlogit_uZproj_x_uZlabel_urrrrs zLogitGenerator.forward) rrrrr1r2r$rrrr3rrr'rrs rc@s$eZdZeddZeddZdS) GradMultiplycCs||_||}|Sr6)scalenew)ctxrrresrrrrs zGradMultiply.forwardcCs||jdfSr6)r)rZgradrrrbackwardszGradMultiply.backwardN)rrr staticmethodrrrrrrrs r)rrUrFr)&loggingtypingrrrr.rrZtorch.nnrr getLoggerrrPr rr4r=rGrTrkrsrzrrr1r2rrFrrrrrrZautogradZFunctionrrrrrs|  1,'7L .=1 K 3  ~f 6