U ‰d¹*ã@sÆddlZddlZddlmZddlmZddlmZddl m Z ddl m Z dd lm Z mZdd lmZmZmZGd d „d eƒZGd d„deƒZeeeeefZGdd„deƒZGdd„deƒZdS)éN)Ú Parameteré)ÚModule)Ú CrossMapLRN2dé)Ú functional)Úinit)ÚTensorÚSize)ÚUnionÚListÚTuplecsveZdZUdZddddgZeed<eed<eed<eed<deeeed d œ‡fd d „ Ze e d œdd„Z dd„Z ‡Z S)ÚLocalResponseNormauApplies local response normalization over an input signal composed of several input planes, where channels occupy the second dimension. Applies normalization across channels. .. math:: b_{c} = a_{c}\left(k + \frac{\alpha}{n} \sum_{c'=\max(0, c-n/2)}^{\min(N-1,c+n/2)}a_{c'}^2\right)^{-\beta} Args: size: amount of neighbouring channels used for normalization alpha: multiplicative factor. Default: 0.0001 beta: exponent. Default: 0.75 k: additive factor. Default: 1 Shape: - Input: :math:`(N, C, *)` - Output: :math:`(N, C, *)` (same shape as input) Examples:: >>> lrn = nn.LocalResponseNorm(2) >>> signal_2d = torch.randn(32, 5, 24, 24) >>> signal_4d = torch.randn(16, 5, 7, 7, 7, 7) >>> output_2d = lrn(signal_2d) >>> output_4d = lrn(signal_4d) ÚsizeÚalphaÚbetaÚkç-Cëâ6?çè?çð?N©rrrrÚreturncs*tt|ƒ ¡||_||_||_||_dS©N)ÚsuperrÚ__init__rrrr©Úselfrrrr©Ú __class__©úB/tmp/pip-unpacked-wheel-ua33x9lu/torch/nn/modules/normalization.pyr/s zLocalResponseNorm.__init__©ÚinputrcCst ||j|j|j|j¡Sr)ÚFZlocal_response_normrrrr©rr"rrr Úforward6sÿzLocalResponseNorm.forwardcCsdjf|jŽS©Nz){size}, alpha={alpha}, beta={beta}, k={k}©ÚformatÚ__dict__©rrrr Ú extra_repr:szLocalResponseNorm.extra_repr)rrr) Ú__name__Ú __module__Ú __qualname__Ú__doc__Ú __constants__ÚintÚ__annotations__Úfloatrr r%r+Ú __classcell__rrrr r s  rcsleZdZUeed<eed<eed<eed<deeeedd œ‡fd d „ Zeed œd d„Ze dœdd„Z ‡Z S)rrrrrrrrNrcs*tt|ƒ ¡||_||_||_||_dSr)rrrrrrrrrrr rDs zCrossMapLRN2d.__init__r!cCst ||j|j|j|j¡Sr)Ú_cross_map_lrn2dÚapplyrrrrr$rrr r%KsÿzCrossMapLRN2d.forward©rcCsdjf|jŽSr&r'r*rrr r+OszCrossMapLRN2d.extra_repr)rrr) r,r-r.r1r2r3rr r%Ústrr+r4rrrr r>s rcs†eZdZUdZdddgZeedfed<eed<e ed<de ee dd œ‡fd d „ Z dd œd d„Z e e dœdd„Zed œdd„Z‡ZS)Ú LayerNormaõ Applies Layer Normalization over a mini-batch of inputs as described in the paper `Layer Normalization `__ .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated over the last `D` dimensions, where `D` is the dimension of :attr:`normalized_shape`. For example, if :attr:`normalized_shape` is ``(3, 5)`` (a 2-dimensional shape), the mean and standard-deviation are computed over the last 2 dimensions of the input (i.e. ``input.mean((-2, -1))``). :math:`\gamma` and :math:`\beta` are learnable affine transform parameters of :attr:`normalized_shape` if :attr:`elementwise_affine` is ``True``. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. .. note:: Unlike Batch Normalization and Instance Normalization, which applies scalar scale and bias for each entire channel/plane with the :attr:`affine` option, Layer Normalization applies per-element scale and bias with :attr:`elementwise_affine`. This layer uses statistics computed from input data in both training and evaluation modes. Args: normalized_shape (int or list or torch.Size): input shape from an expected input of size .. math:: [* \times \text{normalized\_shape}[0] \times \text{normalized\_shape}[1] \times \ldots \times \text{normalized\_shape}[-1]] If a single integer is used, it is treated as a singleton list, and this module will normalize over the last dimension which is expected to be of that specific size. eps: a value added to the denominator for numerical stability. Default: 1e-5 elementwise_affine: a boolean value that when set to ``True``, this module has learnable per-element affine parameters initialized to ones (for weights) and zeros (for biases). Default: ``True``. Attributes: weight: the learnable weights of the module of shape :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``. The values are initialized to 1. bias: the learnable bias of the module of shape :math:`\text{normalized\_shape}` when :attr:`elementwise_affine` is set to ``True``. The values are initialized to 0. Shape: - Input: :math:`(N, *)` - Output: :math:`(N, *)` (same shape as input) Examples:: >>> # NLP Example >>> batch, sentence_length, embedding_dim = 20, 5, 10 >>> embedding = torch.randn(batch, sentence_length, embedding_dim) >>> layer_norm = nn.LayerNorm(embedding_dim) >>> # Activate module >>> layer_norm(embedding) >>> >>> # Image Example >>> N, C, H, W = 20, 5, 10, 10 >>> input = torch.randn(N, C, H, W) >>> # Normalize over the last three dimensions (i.e. the channel and spatial dimensions) >>> # as shown in the image below >>> layer_norm = nn.LayerNorm([C, H, W]) >>> output = layer_norm(input) .. image:: ../_static/img/nn/layer_norm.jpg :scale: 50 % Únormalized_shapeÚepsÚelementwise_affine.çñh㈵øä>TN)r:r;r<rcs˜||dœ}tt|ƒ ¡t|tjƒr*|f}t|ƒ|_||_||_ |j rtt t j |jf|Žƒ|_ t t j |jf|Žƒ|_n| dd¡| dd¡| ¡dS)N©ÚdeviceÚdtypeÚweightÚbias)rr9rÚ isinstanceÚnumbersÚIntegralÚtupler:r;r<rÚtorchÚemptyrArBÚregister_parameterÚreset_parameters)rr:r;r<r?r@Úfactory_kwargsrrr r¤s     zLayerNorm.__init__r7cCs"|jrt |j¡t |j¡dSr)r<rÚones_rAÚzeros_rBr*rrr rJ·s zLayerNorm.reset_parametersr!cCst ||j|j|j|j¡Sr)r#Z layer_normr:rArBr;r$rrr r%¼sÿzLayerNorm.forwardcCsdjf|jŽS)NzF{normalized_shape}, eps={eps}, elementwise_affine={elementwise_affine}r'r*rrr r+ÀsÿzLayerNorm.extra_repr)r=TNN)r,r-r.r/r0r r1r2r3ÚboolÚ_shape_trrJr r%r8r+r4rrrr r9Vs H ÿÿr9csŠeZdZUdZddddgZeed<eed<eed<eed<deeeedd œ‡fd d „ Z dd œd d„Z e e dœdd„Z e d œdd„Z‡ZS)Ú GroupNormaŸApplies Group Normalization over a mini-batch of inputs as described in the paper `Group Normalization `__ .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The input channels are separated into :attr:`num_groups` groups, each containing ``num_channels / num_groups`` channels. :attr:`num_channels` must be divisible by :attr:`num_groups`. The mean and standard-deviation are calculated separately over the each group. :math:`\gamma` and :math:`\beta` are learnable per-channel affine transform parameter vectors of size :attr:`num_channels` if :attr:`affine` is ``True``. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. This layer uses statistics computed from input data in both training and evaluation modes. Args: num_groups (int): number of groups to separate the channels into num_channels (int): number of channels expected in input eps: a value added to the denominator for numerical stability. Default: 1e-5 affine: a boolean value that when set to ``True``, this module has learnable per-channel affine parameters initialized to ones (for weights) and zeros (for biases). Default: ``True``. Shape: - Input: :math:`(N, C, *)` where :math:`C=\text{num\_channels}` - Output: :math:`(N, C, *)` (same shape as input) Examples:: >>> input = torch.randn(20, 6, 10, 10) >>> # Separate 6 channels into 3 groups >>> m = nn.GroupNorm(3, 6) >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm) >>> m = nn.GroupNorm(6, 6) >>> # Put all 6 channels into a single group (equivalent with LayerNorm) >>> m = nn.GroupNorm(1, 6) >>> # Activating the module >>> output = m(input) Ú num_groupsÚ num_channelsr;Úaffiner=TN)rQrRr;rSrcs˜||dœ}tt|ƒ ¡||dkr,tdƒ‚||_||_||_||_|jrttt j |f|Žƒ|_ tt j |f|Žƒ|_ n|  dd¡|  dd¡| ¡dS)Nr>rz,num_channels must be divisible by num_groupsrArB)rrPrÚ ValueErrorrQrRr;rSrrGrHrArBrIrJ)rrQrRr;rSr?r@rKrrr rös    zGroupNorm.__init__r7cCs"|jrt |j¡t |j¡dSr)rSrrLrArMrBr*rrr rJ s zGroupNorm.reset_parametersr!cCst ||j|j|j|j¡Sr)r#Z group_normrQrArBr;r$rrr r%sÿzGroupNorm.forwardcCsdjf|jŽS)Nz8{num_groups}, {num_channels}, eps={eps}, affine={affine}r'r*rrr r+sÿzGroupNorm.extra_repr)r=TNN)r,r-r.r/r0r1r2r3rNrrJr r%r8r+r4rrrr rPÅs * ÿ ÿrP)rGrDZtorch.nn.parameterrÚmodulerZ _functionsrr5Úrr#rr r Útypingr r r rr1rOr9rPrrrr Ús     1o