U d@sDddlZddlmZmZddlZddlmZddlmZddlm Z m Z m Z ddl m Z ddlmZd d lmZGd d d eZGd ddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdddeZGdd d eZGd!d"d"eZGd#d$d$eZGd%d&d&eZ Gd'd(d(eZ!Gd)d*d*eZ"Gd+d,d,eZ#Gd-d.d.eZ$Gd/d0d0eZ%Gd1d2d2eZ&Gd3d4d4eZ'Gd5d6d6eZ(Gd7d8d8eZ)Gd9d:d:eZ*Gd;d<dd>eZ,Gd?d@d@eZ-GdAdBdBeZ.GdCdDdDeZ/dS)EN)OptionalTuple)Tensor)NonDynamicallyQuantizableLinear) constant_xavier_normal_xavier_uniform_) Parameter)Module) functionalcsjeZdZUdZdddgZeed<eed<eed<deeeddfdd Ze e d d d Z d dZ Z S) ThresholdaThresholds each element of the input Tensor. Threshold is defined as: .. math:: y = \begin{cases} x, &\text{ if } x > \text{threshold} \\ \text{value}, &\text{ otherwise } \end{cases} Args: threshold: The value to threshold at value: The value to replace with inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. Examples:: >>> m = nn.Threshold(0.1, 20) >>> input = torch.randn(2) >>> output = m(input) thresholdvalueinplaceFN)rrrreturncs$tt|||_||_||_dSN)superr__init__rrr)selfrrr __class__?/tmp/pip-unpacked-wheel-ua33x9lu/torch/nn/modules/activation.pyr.szThreshold.__init__inputrcCst||j|j|jSr)Frrrrrrrrforward5szThreshold.forwardcCs |jr dnd}d|j|j|S)N, inplace=Truezthreshold={}, value={}{})rformatrrrZ inplace_strrrr extra_repr8s zThreshold.extra_repr)F __name__ __module__ __qualname____doc__ __constants__float__annotations__boolrrrr$ __classcell__rrrrr s  rcsVeZdZUdZdgZeed<d edfdd Zeeddd Z e d d d Z Z S)ReLUaApplies the rectified linear unit function element-wise: :math:`\text{ReLU}(x) = (x)^+ = \max(0, x)` Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/ReLU.png Examples:: >>> m = nn.ReLU() >>> input = torch.randn(2) >>> output = m(input) An implementation of CReLU - https://arxiv.org/abs/1603.05201 >>> m = nn.ReLU() >>> input = torch.randn(2).unsqueeze(0) >>> output = torch.cat((m(input),m(-input))) rFrcstt|||_dSr)rr/rrrrrrrr]sz ReLU.__init__rcCstj||jdSNr0)rZrelurrrrrrasz ReLU.forwardrcCs|jr dnd}|SNz inplace=Truer!r0r#rrrr$dszReLU.extra_repr)F r&r'r(r)r*r-r,rrrstrr$r.rrrrr/?s r/csheZdZUdZdddgZeed<eed<eed<deeedfd d Ze e d d d Z ddZ Z S)RReLUaApplies the randomized leaky rectified liner unit function, element-wise, as described in the paper: `Empirical Evaluation of Rectified Activations in Convolutional Network`_. The function is defined as: .. math:: \text{RReLU}(x) = \begin{cases} x & \text{if } x \geq 0 \\ ax & \text{ otherwise } \end{cases} where :math:`a` is randomly sampled from uniform distribution :math:`\mathcal{U}(\text{lower}, \text{upper})`. See: https://arxiv.org/pdf/1505.00853.pdf Args: lower: lower bound of the uniform distribution. Default: :math:`\frac{1}{8}` upper: upper bound of the uniform distribution. Default: :math:`\frac{1}{3}` inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/RReLU.png Examples:: >>> m = nn.RReLU(0.1, 0.3) >>> input = torch.randn(2) >>> output = m(input) .. _`Empirical Evaluation of Rectified Activations in Convolutional Network`: https://arxiv.org/abs/1505.00853 lowerupperr?UUUUUU?F)r8r9rcs$tt|||_||_||_dSr)rr7rr8r9r)rr8r9rrrrrszRReLU.__init__rcCst||j|j|j|jSr)rZrrelur8r9trainingrrrrrrsz RReLU.forwardcCs |jr dnd}d|j|j|S)Nr r!zlower={}, upper={}{})rr"r8r9r#rrrr$szRReLU.extra_repr)r:r;Fr%rrrrr7is '  r7cs|eZdZUdZdddgZeed<eed<eed<deeeeeeedd fd d Z e e d d dZ e dddZ ZS)HardtanhaApplies the HardTanh function element-wise. HardTanh is defined as: .. math:: \text{HardTanh}(x) = \begin{cases} \text{max\_val} & \text{ if } x > \text{ max\_val } \\ \text{min\_val} & \text{ if } x < \text{ min\_val } \\ x & \text{ otherwise } \\ \end{cases} Args: min_val: minimum value of the linear region range. Default: -1 max_val: maximum value of the linear region range. Default: 1 inplace: can optionally do the operation in-place. Default: ``False`` Keyword arguments :attr:`min_value` and :attr:`max_value` have been deprecated in favor of :attr:`min_val` and :attr:`max_val`. Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Hardtanh.png Examples:: >>> m = nn.Hardtanh(-2, 2) >>> input = torch.randn(2) >>> output = m(input) min_valmax_valr?FN)r>r?r min_value max_valuercs`tt||dk r$td|}|dk r:td|}||_||_||_|j|jks\tdS)Nz>keyword argument min_value is deprecated and rename to min_valz>keyword argument max_value is deprecated and rename to max_val) rr=rwarningswarnr>r?rAssertionError)rr>r?rrBrCrrrrs  zHardtanh.__init__rcCst||j|j|jSr)rZhardtanhr>r?rrrrrrszHardtanh.forwardr3cCs |jr dnd}d|j|j|S)Nr r!zmin_val={}, max_val={}{})rr"r>r?r#rrrr$s zHardtanh.extra_repr)r@rAFNN)r&r'r(r)r*r+r,r-rrrrr6r$r.rrrrr=s(  r=cs6eZdZdZd edfdd ZedddZZS) ReLU6aApplies the element-wise function: .. math:: \text{ReLU6}(x) = \min(\max(0,x), 6) Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/ReLU6.png Examples:: >>> m = nn.ReLU6() >>> input = torch.randn(2) >>> output = m(input) Fr0cstt|dd|dS)Ng@)rrGrr1rrrrszReLU6.__init__r3cCs|jr dnd}|Sr4r0r#rrrr$szReLU6.extra_repr)F) r&r'r(r)r-rr6r$r.rrrrrGsrGc@s eZdZdZeedddZdS)SigmoidaApplies the element-wise function: .. math:: \text{Sigmoid}(x) = \sigma(x) = \frac{1}{1 + \exp(-x)} Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Sigmoid.png Examples:: >>> m = nn.Sigmoid() >>> input = torch.randn(2) >>> output = m(input) rcCs t|Sr)torchZsigmoidrrrrr!szSigmoid.forwardNr&r'r(r)rrrrrrrI srIcsJeZdZUdZdgZeed<d eddfdd Zeedd d Z Z S) HardsigmoidaApplies the Hardsigmoid function element-wise. Hardsigmoid is defined as: .. math:: \text{Hardsigmoid}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ 1 & \text{if~} x \ge +3, \\ x / 6 + 1 / 2 & \text{otherwise} \end{cases} Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Hardsigmoid.png Examples:: >>> m = nn.Hardsigmoid() >>> input = torch.randn(2) >>> output = m(input) rFNrrcstt|||_dSr)rrLrrr1rrrrDszHardsigmoid.__init__rcCst||jSr)rZ hardsigmoidrrrrrrHszHardsigmoid.forward)F r&r'r(r)r*r-r,rrrr.rrrrrL%s rLc@s eZdZdZeedddZdS)TanhaApplies the Hyperbolic Tangent (Tanh) function element-wise. Tanh is defined as: .. math:: \text{Tanh}(x) = \tanh(x) = \frac{\exp(x) - \exp(-x)} {\exp(x) + \exp(-x)} Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Tanh.png Examples:: >>> m = nn.Tanh() >>> input = torch.randn(2) >>> output = m(input) rcCs t|Sr)rJtanhrrrrrasz Tanh.forwardNrKrrrrrOLsrOcsVeZdZUdZdgZeed<d edfdd Zeeddd Z e d d d Z Z S)SiLUaApplies the Sigmoid Linear Unit (SiLU) function, element-wise. The SiLU function is also known as the swish function. .. math:: \text{silu}(x) = x * \sigma(x), \text{where } \sigma(x) \text{ is the logistic sigmoid.} .. note:: See `Gaussian Error Linear Units (GELUs) `_ where the SiLU (Sigmoid Linear Unit) was originally coined, and see `Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learning `_ and `Swish: a Self-Gated Activation Function `_ where the SiLU was experimented with later. Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/SiLU.png Examples:: >>> m = nn.SiLU() >>> input = torch.randn(2) >>> output = m(input) rFr0cstt|||_dSr)rrQrrr1rrrrsz SiLU.__init__rcCstj||jdSr2)rZsilurrrrrrsz SiLU.forwardr3cCs|jr dnd}|Sr4r0r#rrrr$szSiLU.extra_repr)Fr5rrrrrQds rQcsVeZdZUdZdgZeed<d edfdd Zeeddd Z e d d d Z Z S)MishawApplies the Mish function, element-wise. Mish: A Self Regularized Non-Monotonic Neural Activation Function. .. math:: \text{Mish}(x) = x * \text{Tanh}(\text{Softplus}(x)) .. note:: See `Mish: A Self Regularized Non-Monotonic Neural Activation Function `_ Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Mish.png Examples:: >>> m = nn.Mish() >>> input = torch.randn(2) >>> output = m(input) rFr0cstt|||_dSr)rrRrrr1rrrrsz Mish.__init__rcCstj||jdSr2)rZmishrrrrrrsz Mish.forwardr3cCs|jr dnd}|Sr4r0r#rrrr$szMish.extra_repr)Fr5rrrrrRs rRcsJeZdZUdZdgZeed<d eddfdd Zeedd d Z Z S) Hardswisha'Applies the hardswish function, element-wise, as described in the paper: `Searching for MobileNetV3`_. .. math:: \text{Hardswish}(x) = \begin{cases} 0 & \text{if~} x \le -3, \\ x & \text{if~} x \ge +3, \\ x \cdot (x + 3) /6 & \text{otherwise} \end{cases} Args: inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Hardswish.png Examples:: >>> m = nn.Hardswish() >>> input = torch.randn(2) >>> output = m(input) .. _`Searching for MobileNetV3`: https://arxiv.org/abs/1905.02244 rFNrMcstt|||_dSr)rrSrrr1rrrrszHardswish.__init__rcCst||jSr)rZ hardswishrrrrrrszHardswish.forward)FrNrrrrrSs rScsdeZdZUdZddgZeed<eed<deeddfdd Ze e d d d Z e d ddZ Z S)ELUanApplies the Exponential Linear Unit (ELU) function, element-wise, as described in the paper: `Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs) `__. ELU is defined as: .. math:: \text{ELU}(x) = \begin{cases} x, & \text{ if } x > 0\\ \alpha * (\exp(x) - 1), & \text{ if } x \leq 0 \end{cases} Args: alpha: the :math:`\alpha` value for the ELU formulation. Default: 1.0 inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/ELU.png Examples:: >>> m = nn.ELU() >>> input = torch.randn(2) >>> output = m(input) alpharrAFNrUrrcstt|||_||_dSr)rrTrrUrrrUrrrrrsz ELU.__init__rcCst||j|jSr)rZelurUrrrrrrsz ELU.forwardr3cCs|jr dnd}d|j|SNr r!z alpha={}{}rr"rUr#rrrr$szELU.extra_repr)rAFr&r'r(r)r*r+r,r-rrrr6r$r.rrrrrTs rTcsdeZdZUdZddgZeed<eed<deeddfdd Ze e d d d Z e d ddZ Z S)CELUa.Applies the element-wise function: .. math:: \text{CELU}(x) = \max(0,x) + \min(0, \alpha * (\exp(x/\alpha) - 1)) More details can be found in the paper `Continuously Differentiable Exponential Linear Units`_ . Args: alpha: the :math:`\alpha` value for the CELU formulation. Default: 1.0 inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/CELU.png Examples:: >>> m = nn.CELU() >>> input = torch.randn(2) >>> output = m(input) .. _`Continuously Differentiable Exponential Linear Units`: https://arxiv.org/abs/1704.07483 rUrrAFNrVcstt|||_||_dSr)rr[rrUrrWrrrr(sz CELU.__init__rcCst||j|jSr)rZcelurUrrrrrr-sz CELU.forwardr3cCs|jr dnd}d|j|SrXrYr#rrrr$0szCELU.extra_repr)rAFrZrrrrr[ s r[csXeZdZUdZdgZeed<deddfdd Zeedd d Z e d d d Z Z S)SELUayApplied element-wise, as: .. math:: \text{SELU}(x) = \text{scale} * (\max(0,x) + \min(0, \alpha * (\exp(x) - 1))) with :math:`\alpha = 1.6732632423543772848170429916717` and :math:`\text{scale} = 1.0507009873554804934193349852946`. .. warning:: When using ``kaiming_normal`` or ``kaiming_normal_`` for initialisation, ``nonlinearity='linear'`` should be used instead of ``nonlinearity='selu'`` in order to get `Self-Normalizing Neural Networks`_. See :func:`torch.nn.init.calculate_gain` for more information. More details can be found in the paper `Self-Normalizing Neural Networks`_ . Args: inplace (bool, optional): can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/SELU.png Examples:: >>> m = nn.SELU() >>> input = torch.randn(2) >>> output = m(input) .. _Self-Normalizing Neural Networks: https://arxiv.org/abs/1706.02515 rFNrMcstt|||_dSr)rr\rrr1rrrrZsz SELU.__init__rcCst||jSr)rZselurrrrrr^sz SELU.forwardr3cCs|jr dnd}|Sr4r0r#rrrr$aszSELU.extra_repr)Fr5rrrrr\5s !r\csXeZdZUdZdgZeed<deddfdd Zeedd d Z e d d d Z Z S)GLUa3Applies the gated linear unit function :math:`{GLU}(a, b)= a \otimes \sigma(b)` where :math:`a` is the first half of the input matrices and :math:`b` is the second half. Args: dim (int): the dimension on which to split the input. Default: -1 Shape: - Input: :math:`(\ast_1, N, \ast_2)` where `*` means, any number of additional dimensions - Output: :math:`(\ast_1, M, \ast_2)` where :math:`M=N/2` Examples:: >>> m = nn.GLU() >>> input = torch.randn(4, 2) >>> output = m(input) dimNr^rcstt|||_dSr)rr]rr^rr^rrrr|sz GLU.__init__rcCst||jSr)rZglur^rrrrrsz GLU.forwardr3cCs d|jS)Nzdim={}r"r^rrrrr$szGLU.extra_repr)r_ r&r'r(r)r*intr,rrrr6r$r.rrrrr]fs r]csXeZdZUdZdgZeed<deddfdd Zeedd d Z ed d d Z Z S)GELUa3Applies the Gaussian Error Linear Units function: .. math:: \text{GELU}(x) = x * \Phi(x) where :math:`\Phi(x)` is the Cumulative Distribution Function for Gaussian Distribution. When the approximate argument is 'tanh', Gelu is estimated with: :math:: \text{GELU}(x) = 0.5 * x * (1 + \text{Tanh}(\sqrt(2 / \pi) * (x + 0.044715 * x^3))) Args: approximate (string, optional): the gelu approximation algorithm to use: ``'none'`` | ``'tanh'``. Default: ``'none'`` Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/GELU.png Examples:: >>> m = nn.GELU() >>> input = torch.randn(2) >>> output = m(input) approximatenoneN)rgrcstt|||_dSr)rrfrrg)rrgrrrrsz GELU.__init__rcCstj||jdS)N)rg)rZgelurgrrrrrsz GELU.forwardr3cCs d|jS)Nzapproximate={})r"rgrcrrrr$szGELU.extra_repr)rh) r&r'r(r)r*r6r,rrrr$r.rrrrrfs rfcsXeZdZUdZdgZeed<deddfdd Zeedd d Z e d d d Z Z S) HardshrinkaApplies the Hard Shrinkage (Hardshrink) function element-wise. Hardshrink is defined as: .. math:: \text{HardShrink}(x) = \begin{cases} x, & \text{ if } x > \lambda \\ x, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} Args: lambd: the :math:`\lambda` value for the Hardshrink formulation. Default: 0.5 Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Hardshrink.png Examples:: >>> m = nn.Hardshrink() >>> input = torch.randn(2) >>> output = m(input) lambd?Nrjrcstt|||_dSr)rrirrjrrjrrrrszHardshrink.__init__rcCst||jSr)rZ hardshrinkrjrrrrrszHardshrink.forwardr3cCs d|jS)Nz{})r"rjrcrrrr$szHardshrink.extra_repr)rk r&r'r(r)r*r+r,rrrr6r$r.rrrrris ricsdeZdZUdZddgZeed<eed<deeddfdd Ze e d d d Z e d ddZ Z S) LeakyReLUa?Applies the element-wise function: .. math:: \text{LeakyReLU}(x) = \max(0, x) + \text{negative\_slope} * \min(0, x) or .. math:: \text{LeakyRELU}(x) = \begin{cases} x, & \text{ if } x \geq 0 \\ \text{negative\_slope} \times x, & \text{ otherwise } \end{cases} Args: negative_slope: Controls the angle of the negative slope. Default: 1e-2 inplace: can optionally do the operation in-place. Default: ``False`` Shape: - Input: :math:`(*)` where `*` means, any number of additional dimensions - Output: :math:`(*)`, same shape as the input .. image:: ../scripts/activation_images/LeakyReLU.png Examples:: >>> m = nn.LeakyReLU(0.1) >>> input = torch.randn(2) >>> output = m(input) rnegative_slope{Gz?FN)rprrcstt|||_||_dSr)rrorrpr)rrprrrrrszLeakyReLU.__init__rcCst||j|jSr)rZ leaky_relurprrrrrrszLeakyReLU.forwardr3cCs|jr dnd}d|j|S)Nr r!znegative_slope={}{})rr"rpr#rrrr$szLeakyReLU.extra_repr)rqF)r&r'r(r)r*r-r,r+rrrr6r$r.rrrrros  roc@s eZdZdZeedddZdS) LogSigmoidaApplies the element-wise function: .. math:: \text{LogSigmoid}(x) = \log\left(\frac{ 1 }{ 1 + \exp(-x)}\right) Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/LogSigmoid.png Examples:: >>> m = nn.LogSigmoid() >>> input = torch.randn(2) >>> output = m(input) rcCs t|Sr)rZ logsigmoidrrrrrszLogSigmoid.forwardNrKrrrrrr srrcsdeZdZUdZddgZeed<eed<deeddfdd Zeed d d Z e d ddZ Z S)SoftplusanApplies the Softplus function :math:`\text{Softplus}(x) = \frac{1}{\beta} * \log(1 + \exp(\beta * x))` element-wise. SoftPlus is a smooth approximation to the ReLU function and can be used to constrain the output of a machine to always be positive. For numerical stability the implementation reverts to the linear function when :math:`input \times \beta > threshold`. Args: beta: the :math:`\beta` value for the Softplus formulation. Default: 1 threshold: values above this revert to a linear function. Default: 20 Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Softplus.png Examples:: >>> m = nn.Softplus() >>> input = torch.randn(2) >>> output = m(input) betarrN)rtrrcstt|||_||_dSr)rrsrrtr)rrtrrrrr@szSoftplus.__init__rcCst||j|jSr)rZsoftplusrtrrrrrrEszSoftplus.forwardr3cCsd|j|jS)Nzbeta={}, threshold={})r"rtrrcrrrr$HszSoftplus.extra_repr)rrurdrrrrrs"s rscsXeZdZUdZdgZeed<deddfdd Zeedd d Z e d d d Z Z S) SoftshrinkaApplies the soft shrinkage function elementwise: .. math:: \text{SoftShrinkage}(x) = \begin{cases} x - \lambda, & \text{ if } x > \lambda \\ x + \lambda, & \text{ if } x < -\lambda \\ 0, & \text{ otherwise } \end{cases} Args: lambd: the :math:`\lambda` (must be no less than zero) value for the Softshrink formulation. Default: 0.5 Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Softshrink.png Examples:: >>> m = nn.Softshrink() >>> input = torch.randn(2) >>> output = m(input) rjrkNrlcstt|||_dSr)rrvrrjrmrrrriszSoftshrink.__init__rcCst||jSr)rZ softshrinkrjrrrrrmszSoftshrink.forwardr3cCs t|jSr)r6rjrcrrrr$pszSoftshrink.extra_repr)rkrnrrrrrvLs rvc seZdZUdZdgZeejed<eejed<ddd fd d Z d d Z fddZ deeeeee eee e eeefdddZZS)MultiheadAttentionaY Allows the model to jointly attend to information from different representation subspaces as described in the paper: `Attention Is All You Need `_. Multi-Head Attention is defined as: .. math:: \text{MultiHead}(Q, K, V) = \text{Concat}(head_1,\dots,head_h)W^O where :math:`head_i = \text{Attention}(QW_i^Q, KW_i^K, VW_i^V)`. ``forward()`` will use a special optimized implementation if all of the following conditions are met: - self attention is being computed (i.e., ``query``, ``key``, and ``value`` are the same tensor. This restriction will be loosened in the future.) - Either autograd is disabled (using ``torch.inference_mode`` or ``torch.no_grad``) or no tensor argument ``requires_grad`` - training is disabled (using ``.eval()``) - dropout is 0 - ``add_bias_kv`` is ``False`` - ``add_zero_attn`` is ``False`` - ``batch_first`` is ``True`` and the input is batched - ``kdim`` and ``vdim`` are equal to ``embed_dim`` - at most one of ``key_padding_mask`` or ``attn_mask`` is passed - if a `NestedTensor `_ is passed, neither ``key_padding_mask`` nor ``attn_mask`` is passed If the optimized implementation is in use, a `NestedTensor `_ can be passed for ``query``/``key``/``value`` to represent padding more efficiently than using a padding mask. In this case, a `NestedTensor `_ will be returned, and an additional speedup proportional to the fraction of the input that is padding can be expected. Args: embed_dim: Total dimension of the model. num_heads: Number of parallel attention heads. Note that ``embed_dim`` will be split across ``num_heads`` (i.e. each head will have dimension ``embed_dim // num_heads``). dropout: Dropout probability on ``attn_output_weights``. Default: ``0.0`` (no dropout). bias: If specified, adds bias to input / output projection layers. Default: ``True``. add_bias_kv: If specified, adds bias to the key and value sequences at dim=0. Default: ``False``. add_zero_attn: If specified, adds a new batch of zeros to the key and value sequences at dim=1. Default: ``False``. kdim: Total number of features for keys. Default: ``None`` (uses ``kdim=embed_dim``). vdim: Total number of features for values. Default: ``None`` (uses ``vdim=embed_dim``). batch_first: If ``True``, then the input and output tensors are provided as (batch, seq, feature). Default: ``False`` (seq, batch, feature). Examples:: >>> multihead_attn = nn.MultiheadAttention(embed_dim, num_heads) >>> attn_output, attn_output_weights = multihead_attn(query, key, value) batch_firstbias_kbias_vrHTFNr3c s| | d} tt|||_|dk r*|n||_|dk r<|n||_|j|koT|j|k|_||_||_| |_ |||_ |j ||jkst d|jdkrt t j||ff| |_t t j||jff| |_t t j||jff| |_|ddn@t t jd||ff| |_|dd|dd|dd|rPt t jd|f| |_n |d dt||fd |i| |_|rt t jd d |ff| |_t t jd d |ff| |_n d|_|_||_|dS) Ndevicedtypez(embed_dim must be divisible by num_headsFin_proj_weight q_proj_weight k_proj_weight v_proj_weight in_proj_biasbiasr)rrwr embed_dimkdimvdim_qkv_same_embed_dim num_headsdropoutrxZhead_dimrFr rJemptyrrrZregister_parameterr~rrout_projryrz add_zero_attn_reset_parameters) rrrrrZ add_bias_kvrrrrxr|r}factory_kwargsrrrrs<        zMultiheadAttention.__init__cCs|jrt|jnt|jt|jt|j|jdk rTt|jdt|jj d|j dk rht |j |j dk r|t |j dS)NrH) rr r~rrrrrrrryrrzrcrrrrs         z$MultiheadAttention._reset_parameterscs$d|krd|d<tt||dS)NrT)rrw __setstate__rstaterrrrszMultiheadAttention.__setstate__)querykeyrkey_padding_mask need_weights attn_maskaverage_attn_weightsrcCs\|dk}d} |s&d|} n||k s6||k rSsz.MultiheadAttention.forward..z,some Tensor argument is neither CUDA nor CPUcSsg|] }|jqSr)Z requires_gradrrrrrUszhgrad is enabled and at least one of query or the input/output projection weights or biases requires_gradzKMultiheadAttention does not support NestedTensor outside of its fast path. z"The fast path was not hit because rrcSsg|]}|ddqSrr transposerrrrrpscSsg|]}|ddqSrrrrrrrssT) r<rrrZuse_separate_proj_weightrrrr)r<rrrr)r^rr}r~r<rxryrzrrrZ is_nestedrweightrrJZ overridesZhas_torch_functionallZis_grad_enabledanyZ_native_multi_head_attentionrrrFrrZmulti_head_attention_forwardrrr)rrrrrrrrZ is_batchedZwhy_not_fast_pathZ tensor_argsZ any_nestedZ attn_outputZattn_output_weightsrrrrs4           zMultiheadAttention.forward) rHTFFNNFNN)NTNT)r&r'r(r)r*rrJrr,rrrr-rrr.rrrrrwts2 6* rwcsZeZdZUdZdgZeed<deeddfdd Ze e d d d Z e d d dZ Z S)PReLUaApplies the element-wise function: .. math:: \text{PReLU}(x) = \max(0,x) + a * \min(0,x) or .. math:: \text{PReLU}(x) = \begin{cases} x, & \text{ if } x \geq 0 \\ ax, & \text{ otherwise } \end{cases} Here :math:`a` is a learnable parameter. When called without arguments, `nn.PReLU()` uses a single parameter :math:`a` across all input channels. If called with `nn.PReLU(nChannels)`, a separate :math:`a` is used for each input channel. .. note:: weight decay should not be used when learning :math:`a` for good performance. .. note:: Channel dim is the 2nd dim of input. When input has dims < 2, then there is no channel dim and the number of channels = 1. Args: num_parameters (int): number of :math:`a` to learn. Although it takes an int as input, there is only two values are legitimate: 1, or the number of channels at input. Default: 1 init (float): the initial value of :math:`a`. Default: 0.25 Shape: - Input: :math:`( *)` where `*` means, any number of additional dimensions. - Output: :math:`(*)`, same shape as the input. Attributes: weight (Tensor): the learnable weights of shape (:attr:`num_parameters`). .. image:: ../scripts/activation_images/PReLU.png Examples:: >>> m = nn.PReLU() >>> input = torch.randn(2) >>> output = m(input) num_parametersr?N)rinitrcs<||d}||_tt|ttj|f|||_dS)Nr{) rrrrr rJrZfill_r)rrrr|r}rrrrrs zPReLU.__init__rcCst||jSr)rZprelurrrrrrsz PReLU.forwardr3cCs d|jS)Nznum_parameters={})r"rrcrrrr$szPReLU.extra_repr)rrNN)r&r'r(r)r*rer,r+rrrr6r$r.rrrrrs 0rc@s eZdZdZeedddZdS)SoftsignaApplies the element-wise function: .. math:: \text{SoftSign}(x) = \frac{x}{ 1 + |x|} Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Softsign.png Examples:: >>> m = nn.Softsign() >>> input = torch.randn(2) >>> output = m(input) rcCs t|Sr)rZsoftsignrrrrrszSoftsign.forwardNrKrrrrrsrc@s eZdZdZeedddZdS) TanhshrinkaApplies the element-wise function: .. math:: \text{Tanhshrink}(x) = x - \tanh(x) Shape: - Input: :math:`(*)`, where :math:`*` means any number of dimensions. - Output: :math:`(*)`, same shape as the input. .. image:: ../scripts/activation_images/Tanhshrink.png Examples:: >>> m = nn.Tanhshrink() >>> input = torch.randn(2) >>> output = m(input) rcCs t|Sr)rZ tanhshrinkrrrrrszTanhshrink.forwardNrKrrrrrsrcsfeZdZUdZdgZeeed<deeddfdd ZfddZ e e d d d Z d d Z Z S)Softmina4Applies the Softmin function to an n-dimensional input Tensor rescaling them so that the elements of the n-dimensional output Tensor lie in the range `[0, 1]` and sum to 1. Softmin is defined as: .. math:: \text{Softmin}(x_{i}) = \frac{\exp(-x_i)}{\sum_j \exp(-x_j)} Shape: - Input: :math:`(*)` where `*` means, any number of additional dimensions - Output: :math:`(*)`, same shape as the input Args: dim (int): A dimension along which Softmin will be computed (so every slice along dim will sum to 1). Returns: a Tensor of the same dimension and shape as the input, with values in the range [0, 1] Examples:: >>> m = nn.Softmin() >>> input = torch.randn(2, 3) >>> output = m(input) r^Nr`cstt|||_dSr)rrrr^rarrrrszSoftmin.__init__cs t|t|dsd|_dSNr^rrhasattrr^rrrrr"s  zSoftmin.__setstate__rcCstj||jddSNZ _stacklevel)rZsoftminr^rrrrr'szSoftmin.forwardcCsdj|jdSNz dim={dim})r^rbrcrrrr$*szSoftmin.extra_repr)Nr&r'r(r)r*rrer,rrrrr$r.rrrrrs   rcsleZdZUdZdgZeeed<deeddfdd ZfddZ e e d d d Z e d d dZ ZS)SoftmaxaApplies the Softmax function to an n-dimensional input Tensor rescaling them so that the elements of the n-dimensional output Tensor lie in the range [0,1] and sum to 1. Softmax is defined as: .. math:: \text{Softmax}(x_{i}) = \frac{\exp(x_i)}{\sum_j \exp(x_j)} When the input Tensor is a sparse tensor then the unspecifed values are treated as ``-inf``. Shape: - Input: :math:`(*)` where `*` means, any number of additional dimensions - Output: :math:`(*)`, same shape as the input Returns: a Tensor of the same dimension and shape as the input with values in the range [0, 1] Args: dim (int): A dimension along which Softmax will be computed (so every slice along dim will sum to 1). .. note:: This module doesn't work directly with NLLLoss, which expects the Log to be computed between the Softmax and itself. Use `LogSoftmax` instead (it's faster and has better numerical properties). Examples:: >>> m = nn.Softmax(dim=1) >>> input = torch.randn(2, 3) >>> output = m(input) r^Nr`cstt|||_dSr)rrrr^rarrrrVszSoftmax.__init__cs t|t|dsd|_dSrrrrrrrZs  zSoftmax.__setstate__rcCstj||jddSr)rsoftmaxr^rrrrr_szSoftmax.forwardr3cCsdj|jdSrrbrcrrrr$bszSoftmax.extra_repr)N)r&r'r(r)r*rrer,rrrrr6r$r.rrrrr-s %  rc@s eZdZdZeedddZdS) Softmax2da|Applies SoftMax over features to each spatial location. When given an image of ``Channels x Height x Width``, it will apply `Softmax` to each location :math:`(Channels, h_i, w_j)` Shape: - Input: :math:`(N, C, H, W)` or :math:`(C, H, W)`. - Output: :math:`(N, C, H, W)` or :math:`(C, H, W)` (same shape as input) Returns: a Tensor of the same dimension and shape as the input with values in the range [0, 1] Examples:: >>> m = nn.Softmax2d() >>> # you softmax over the 2nd dimension >>> input = torch.randn(2, 3, 12, 13) >>> output = m(input) rcCs0|dks |dks tdtj|dddS)Nrz-Softmax2d requires a 3D or 4D tensor as inputrr)r^rFrrrrrrr|s zSoftmax2d.forwardNrKrrrrrfsrcsfeZdZUdZdgZeeed<deeddfdd ZfddZ e e d d d Z d d Z Z S) LogSoftmaxaApplies the :math:`\log(\text{Softmax}(x))` function to an n-dimensional input Tensor. The LogSoftmax formulation can be simplified as: .. math:: \text{LogSoftmax}(x_{i}) = \log\left(\frac{\exp(x_i) }{ \sum_j \exp(x_j)} \right) Shape: - Input: :math:`(*)` where `*` means, any number of additional dimensions - Output: :math:`(*)`, same shape as the input Args: dim (int): A dimension along which LogSoftmax will be computed. Returns: a Tensor of the same dimension and shape as the input with values in the range [-inf, 0) Examples:: >>> m = nn.LogSoftmax() >>> input = torch.randn(2, 3) >>> output = m(input) r^Nr`cstt|||_dSr)rrrr^rarrrrszLogSoftmax.__init__cs t|t|dsd|_dSrrrrrrrs  zLogSoftmax.__setstate__rcCstj||jddSr)rZ log_softmaxr^rrrrrszLogSoftmax.forwardcCsdj|jdSrrbrcrrrr$szLogSoftmax.extra_repr)Nrrrrrrs   r)0rDtypingrrrJrZlinearrZ torch.nn.initrrr Ztorch.nn.parameterr moduler r!r rrr/r7r=rGrIrLrOrQrRrSrTr[r\r]rfrirorrrsrvrwrrrrrrrrrrrsN     2*AE')$*.,1!(*2*(B/9