U ‰dæZã@srddlZddlmZddlmmZddlmmZddlmZddl m Z m Z ddl Z Gdd„dej ƒZ dS)éN)Únn)ÚTensor)ÚOptionalÚTuplec sÖeZdZejZdgZdeeee e e eee ddœ ‡fdd„ Z d d „Z e d d „ƒZ ejjd d„ƒZe dd„ƒZdeeeeee eee eeeefdœdd„Zdeeeeee eee eeeefdœdd„Z‡ZS)ÚMultiheadAttentionÚ batch_firstçTFN) Ú embed_dimÚ num_headsÚdropoutÚbiasÚ add_bias_kvÚ add_zero_attnÚkdimÚvdimrÚreturnc sú| | dœ} tt|ƒj||||||||| f | Žtj|j|jfd|i| —Ž|_tj|j|jfd|i| —Ž|_tj|j |jfd|i| —Ž|_ tj|j|jfd|i| —Ž|_ t   ¡|_tjj ¡|_tjj ¡|_tjj ¡|_tjj ¡|_tjj ¡|_dS)N)ÚdeviceÚdtyper )ÚsuperrÚ__init__rZLinearr Úlinear_QrÚlinear_KrÚlinear_VÚout_projÚnnqZFloatFunctionalÚq_scaling_productÚtorchÚaoÚ quantizationZ QuantStubÚquant_attn_outputÚquant_attn_output_weightsZ DeQuantStubÚ dequant_qÚ dequant_kÚ dequant_v) Úselfr r r r r rrrrrrZfactory_kwargs©Ú __class__©úK/tmp/pip-unpacked-wheel-ua33x9lu/torch/nn/quantizable/modules/activation.pyr;s* þý zMultiheadAttention.__init__cCsdS)NZQuantizableMultiheadAttentionr')r$r'r'r(Ú _get_nameUszMultiheadAttention._get_namec Cs„t|ƒ|jkst‚t|dƒs$tdƒ‚||j|j|j|jdk |jdk |j |j |j ƒ}|j|_|j |_ |j |_ |jj|j_|jj|j_|jr¸|j}d}||j}|j||…dd…f}|dk rÔtj |||…|j¡}tj ||j¡|j_||j_|j}|}||j}|j||…dd…f}|dk rt ‚||j| | …<| } | |j} ||j| | …dd…f<|jdk rœt|dkƒsŽt ‚||j| | …<| } ||j| d…dd…f<|jdk rht| dkƒsÔt ‚| |j| d…<n„t   |¡|_t   |¡|_t   |¡|_|jdkr.d|j_d|j_d|j_n:||jd|j…<||j|j|jd…<| |j|jdd…<|S)zæUtility to convert the quantized MHA back to float. The motivation for this is that it is not trivial to conver the weights from the format that is used in the quantized version back to the float. Nrr+)r.r r r r1r2rrrrr5r/rr7Ú dequantizer3rZ _weight_biasr4r rrrr6Úallr8r9r:) r$ÚfpÚwÚbZwQZbQZwKZbKZwVZbVr?r@r'r'r(rB˜shý            zMultiheadAttention.dequantizecCs¨tjjj|ddddd}|jdk r^|j d¡}tj|dd\}}t |||tj ¡}t |d|ƒ|j dk r¤|j d¡}tj|dd\}}t |||tj ¡}t |d|ƒ|S)NFT)Úmappingr,Zremove_qconfigZconvert_custom_config_dictr2)Z reduce_ranger3) rrrÚconvertr2Ú _parametersÚpopZ_choose_qparams_per_tensorÚquantize_per_tensorZquint8Úsetattrr3)r=r>Z convertedr2ZscZzpr3r'r'r(Ú from_observedÞs( ý  ÿ    ÿ  z MultiheadAttention.from_observed)ÚqueryÚkeyÚvalueÚkey_padding_maskÚ need_weightsÚ attn_maskÚaverage_attn_weightsrc Cs| |||||||¡S)aÁ Note:: Please, refer to :func:`~torch.nn.MultiheadAttention.forward` for more information Args: query, key, value: map a query and a set of key-value pairs to an output. See "Attention Is All You Need" for more details. key_padding_mask: if provided, specified padding elements in the key will be ignored by the attention. When given a binary mask and a value is True, the corresponding value on the attention layer will be ignored. When given a byte mask and a value is non-zero, the corresponding value on the attention layer will be ignored need_weights: output attn_output_weights. attn_mask: 2D or 3D mask that prevents attention to certain positions. A 2D mask will be broadcasted for all the batches while a 3D mask allows to specify a different mask for the entries of each batch. Shape: - Inputs: - query: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``. - key: :math:`(S, N, E)`, where S is the source sequence length, N is the batch size, E is the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``. - value: :math:`(S, N, E)` where S is the source sequence length, N is the batch size, E is the embedding dimension. :math:`(N, S, E)` if ``batch_first`` is ``True``. - key_padding_mask: :math:`(N, S)` where N is the batch size, S is the source sequence length. If a ByteTensor is provided, the non-zero positions will be ignored while the position with the zero positions will be unchanged. If a BoolTensor is provided, the positions with the value of ``True`` will be ignored while the position with the value of ``False`` will be unchanged. - attn_mask: 2D mask :math:`(L, S)` where L is the target sequence length, S is the source sequence length. 3D mask :math:`(N*num_heads, L, S)` where N is the batch size, L is the target sequence length, S is the source sequence length. attn_mask ensure that position i is allowed to attend the unmasked positions. If a ByteTensor is provided, the non-zero positions are not allowed to attend while the zero positions will be unchanged. If a BoolTensor is provided, positions with ``True`` is not allowed to attend while ``False`` values will be unchanged. If a FloatTensor is provided, it will be added to the attention weight. - average_attn_weights: If true, indicates that the returned ``attn_weights`` should be averaged across heads. Otherwise, ``attn_weights`` are provided separately per head. Note that this flag only has an effect when ``need_weights=True.``. Default: True (i.e. average weights across heads) - Outputs: - attn_output: :math:`(L, N, E)` where L is the target sequence length, N is the batch size, E is the embedding dimension. :math:`(N, L, E)` if ``batch_first`` is ``True``. - attn_output_weights: If ``average_attn_weights=True``, returns attention weights averaged across heads of shape :math:`(N, L, S)`, where N is the batch size, L is the target sequence length, S is the source sequence length. If ``average_weights=False``, returns attention weights per head of shape :math:`(N, num_heads, L, S)`. )Ú _forward_impl)r$rNrOrPrQrRrSrTr'r'r(Úforwardùs 8 ÿzMultiheadAttention.forwardcCsØd}d} |jr(dd„|||fDƒ\}}}| ¡\} } } |j| ksDt‚| d¡| d¡krl| d¡| d¡kspt‚|j|j} | |j|jks”tdƒ‚t| ƒd}| |¡}| |¡}| |¡}|j   ||¡}|dk rê|j t j ks,|j t jks,|j t jks,|j t jks,|j t jks,td |j ¡ƒ‚|j t jkrPt d¡| t j¡}| ¡d kr–| d¡}t| ¡ƒd| d¡| d¡gkrêtd ƒ‚nT| ¡d krØt| ¡ƒ| |j| d¡| d¡gkrêtd ƒ‚ntd  | ¡¡ƒ‚|dk r|j t jkrt d¡| t j¡}|jdk rð|jdk rð|dkrÊ| dkrÊ|j}|dk sXt‚|j}|dk slt‚t  || d| d¡g¡}t  || d| d¡g¡}|dk r²t |d¡}|dk rît |d¡}n$|dksÜtdƒ‚| dkstdƒ‚n |jdkst‚|jdkst‚|  ¡ !| | |j| ¡ "dd¡}|dk rZ|  ¡ !d| |j| ¡ "dd¡}|dk r„|  ¡ !d| |j| ¡ "dd¡}|dk rÀ| d¡| |jks¨t‚| d ¡| ks¼t‚|}| dk rü|  d¡| |jksät‚|  d ¡| ksøt‚| }| d¡}|dk r8| d¡| ks$t‚| d¡|ks8t‚|j#r$|d7}t  $| d¡df| ¡d d…¡}|j%rŽt  &|| '¡| (¡|j ¡}t j||gdd}t  $| d¡df| ¡d d…¡}|j%ræt  &|| '¡| (¡|j ¡}t j||gdd}|dk rt |d¡}|dk r$t |d¡}| )|¡}| *|¡}| +|¡}t  ,|| "dd ¡¡}t| ¡ƒ| |j| |gksxt‚|dk rª|j t jkr¢| -|tdƒ¡n||7}|dk rö| !| |j| |¡}| .| d¡ d ¡tdƒ¡}| !| |j| |¡}tj/|dd}tj0||j0|j1d}t  ,||¡}t| ¡ƒ| |j| | gksFt‚|jr`| !| | |j¡}n| "dd¡  ¡ !| | |j¡}| 2|¡}| 3|¡}| 4|¡}|rÌ| !| |j| |¡}|rÄ|j5dd}||fS|dfSdS)NcSsg|]}| dd¡‘qS©ré)Ú transpose)Ú.0Úxr'r'r(Ú Esz4MultiheadAttention._forward_impl..rrXz(embed_dim must be divisible by num_headsgà¿zDOnly float, byte, and bool types are supported for attn_mask, not {}zZByte tensor for attn_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.r+z,The size of the 2D attn_mask is not correct.éz,The size of the 3D attn_mask is not correct.z)attn_mask's dimension {} is not supportedzaByte tensor for key_padding_mask in nn.MultiheadAttention is deprecated. Use bool tensor instead.rWz#bias cannot be added to static key.z%bias cannot be added to static value.éÿÿÿÿ)Údimz-inf)ÚpÚtraining)6rÚsizer r/r ÚfloatrrrrZ mul_scalarrrZfloat32Zfloat64Zfloat16Zuint8ÚboolÚformatÚwarningsÚwarnÚtor_Z unsqueezeÚlistÚ RuntimeErrorr2r3ÚcatÚrepeatÚnnFÚpadÚ contiguousÚviewrYrÚzerosZ is_quantizedrKZq_scaleZ q_zero_pointr!r"r#ZbmmZ masked_fill_Z masked_fillZsoftmaxr rarrr Zmean)r$rNrOrPrQrRrSrTZstatic_kZstatic_vZtgt_lenZbszZembed_dim_to_checkZhead_dimZscalingÚqÚkÚvr2r3Zsrc_lenZk_zerosZv_zerosZattn_output_weightsZ attn_outputr'r'r(rU4sð ,       ÿ ÿ ÿ þ   $ *               $$       "  þÿ "    z MultiheadAttention._forward_impl) rTFFNNFNN)NTNT)NTNT)Ú__name__Ú __module__Ú __qualname__rrr.Z __constants__Úintrcrdrr)Ú classmethodrArZjitZunusedrBrMrrrrVrUÚ __classcell__r'r'r%r(r sp-üü ? E ùù ?ùùr)rrZtorch.nn.functionalZ functionalrmZtorch.nn.quantizedZ quantizedrrÚtypingrrrfrr'r'r'r(Ús