U dK@sddlmZddlmZmZddlmZGdddeZGdd d eZ Gd d d eeZ Gd d d eZ GdddeeZ GdddeZ GdddeeZdS))Tensor) _LazyNormBase _NormBase) functionalcsjeZdZdeeeeeddfdd Zdd Zd d Zd d Z ddZ fddZ e e dddZ ZS) _InstanceNormh㈵>皙?FN) num_featuresepsmomentumaffinetrack_running_statsreturnc s*||d}tt|j|||||f|dS)N)devicedtype)superr__init__) selfr r r rrrrZfactory_kwargs __class__A/tmp/pip-unpacked-wheel-ua33x9lu/torch/nn/modules/instancenorm.pyrs  z_InstanceNorm.__init__cCstdSNNotImplementedErrorrinputrrr_check_input_dimsz_InstanceNorm._check_input_dimcCstdSrrrrrr_get_no_batch_dimsz_InstanceNorm._get_no_batch_dimcCs||ddS)Nr)_apply_instance_normZ unsqueezeZsqueezerrrr_handle_no_batch_inputsz$_InstanceNorm._handle_no_batch_inputc Cs.t||j|j|j|j|jp"|j |j|j Sr) FZ instance_norm running_mean running_varZweightZbiasZtrainingrr r rrrrr"s z"_InstanceNorm._apply_instance_normc s|dd}|dkr|jsg} dD]} || } | |kr"| | q"t| dkr|djddd| D|jjd| D]} || qzt t | |||||||dS) Nversion)r%r&raUnexpected running stats buffer(s) {names} for {klass} with track_running_stats=False. If state_dict is a checkpoint saved before 0.4.0, this may be expected because {klass} does not track running stats by default since 0.4.0. Please remove these keys from state_dict. If the running stats are actually needed, instead set track_running_stats=True in {klass} to enable them. See the documentation of {klass} for details.z and css|]}d|VqdS)z"{}"N)format).0krrr 9sz6_InstanceNorm._load_from_state_dict..)namesklass) getrappendlenr(joinr__name__poprr_load_from_state_dict) rZ state_dictprefixZlocal_metadatastrictZ missing_keysZunexpected_keysZ error_msgsr'Zrunning_stats_keysnamekeyrrrr4$s2      z#_InstanceNorm._load_from_state_dict)rrcCs.||||kr$||S||Sr)rdimr!r#r"rrrrforwardBs  z_InstanceNorm.forward)r r FFNN)r2 __module__ __qualname__intfloatboolrrr!r#r"r4rr: __classcell__rrrrrs( rc@s eZdZdZddZddZdS)InstanceNorm1da Applies Instance Normalization over a 2D (unbatched) or 3D (batched) input as described in the paper `Instance Normalization: The Missing Ingredient for Fast Stylization `__. .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the number of features or channels of the input) if :attr:`affine` is ``True``. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. By default, this layer uses instance statistics computed from input data in both training and evaluation modes. If :attr:`track_running_stats` is set to ``True``, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. .. note:: :class:`InstanceNorm1d` and :class:`LayerNorm` are very similar, but have some subtle differences. :class:`InstanceNorm1d` is applied on each channel of channeled data like multidimensional time series, but :class:`LayerNorm` is usually applied on entire sample and often in NLP tasks. Additionally, :class:`LayerNorm` applies elementwise affine transform, while :class:`InstanceNorm1d` usually don't apply affine transform. Args: num_features: number of features or channels :math:`C` of the input eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: ``False``. track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: ``False`` Shape: - Input: :math:`(N, C, L)` or :math:`(C, L)` - Output: :math:`(N, C, L)` or :math:`(C, L)` (same shape as input) Examples:: >>> # Without Learnable Parameters >>> m = nn.InstanceNorm1d(100) >>> # With Learnable Parameters >>> m = nn.InstanceNorm1d(100, affine=True) >>> input = torch.randn(20, 100, 40) >>> output = m(input) cCsdSNrrr rrrr!sz InstanceNorm1d._get_no_batch_dimcCs"|dkrtd|dSN)rz'expected 2D or 3D input (got {}D input)r9 ValueErrorr(rrrrrs zInstanceNorm1d._check_input_dimNr2r;r<__doc__r!rrrrrrAKsBrAc@s$eZdZdZeZddZddZdS)LazyInstanceNorm1daqA :class:`torch.nn.InstanceNorm1d` module with lazy initialization of the ``num_features`` argument of the :class:`InstanceNorm1d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, L)` or :math:`(C, L)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: ``False``. track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: ``False`` Shape: - Input: :math:`(N, C, L)` or :math:`(C, L)` - Output: :math:`(N, C, L)` or :math:`(C, L)` (same shape as input) cCsdSrBrr rrrr!sz$LazyInstanceNorm1d._get_no_batch_dimcCs"|dkrtd|dSrCrErrrrrs z#LazyInstanceNorm1d._check_input_dimN)r2r;r<rHrA cls_to_becomer!rrrrrrIsrIc@s eZdZdZddZddZdS)InstanceNorm2da Applies Instance Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper `Instance Normalization: The Missing Ingredient for Fast Stylization `__. .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size) if :attr:`affine` is ``True``. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. By default, this layer uses instance statistics computed from input data in both training and evaluation modes. If :attr:`track_running_stats` is set to ``True``, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. .. note:: :class:`InstanceNorm2d` and :class:`LayerNorm` are very similar, but have some subtle differences. :class:`InstanceNorm2d` is applied on each channel of channeled data like RGB images, but :class:`LayerNorm` is usually applied on entire sample and often in NLP tasks. Additionally, :class:`LayerNorm` applies elementwise affine transform, while :class:`InstanceNorm2d` usually don't apply affine transform. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, H, W)` or :math:`(C, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: ``False``. track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: ``False`` Shape: - Input: :math:`(N, C, H, W)` or :math:`(C, H, W)` - Output: :math:`(N, C, H, W)` or :math:`(C, H, W)` (same shape as input) Examples:: >>> # Without Learnable Parameters >>> m = nn.InstanceNorm2d(100) >>> # With Learnable Parameters >>> m = nn.InstanceNorm2d(100, affine=True) >>> input = torch.randn(20, 100, 35, 45) >>> output = m(input) cCsdSNrDrr rrrr!sz InstanceNorm2d._get_no_batch_dimcCs"|dkrtd|dSN)rDz'expected 3D or 4D input (got {}D input)rErrrrrs zInstanceNorm2d._check_input_dimNrGrrrrrKsCrKc@s$eZdZdZeZddZddZdS)LazyInstanceNorm2daA :class:`torch.nn.InstanceNorm2d` module with lazy initialization of the ``num_features`` argument of the :class:`InstanceNorm2d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, H, W)` or :math:`(C, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: ``False``. track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: ``False`` Shape: - Input: :math:`(N, C, H, W)` or :math:`(C, H, W)` - Output: :math:`(N, C, H, W)` or :math:`(C, H, W)` (same shape as input) cCsdSrLrr rrrr!)sz$LazyInstanceNorm2d._get_no_batch_dimcCs"|dkrtd|dSrMrErrrrr,s z#LazyInstanceNorm2d._check_input_dimN)r2r;r<rHrKrJr!rrrrrrO srOc@s eZdZdZddZddZdS)InstanceNorm3da Applies Instance Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper `Instance Normalization: The Missing Ingredient for Fast Stylization `__. .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension separately for each object in a mini-batch. :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size C (where C is the input size) if :attr:`affine` is ``True``. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. By default, this layer uses instance statistics computed from input data in both training and evaluation modes. If :attr:`track_running_stats` is set to ``True``, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. .. note:: :class:`InstanceNorm3d` and :class:`LayerNorm` are very similar, but have some subtle differences. :class:`InstanceNorm3d` is applied on each channel of channeled data like 3D models with RGB color, but :class:`LayerNorm` is usually applied on entire sample and often in NLP tasks. Additionally, :class:`LayerNorm` applies elementwise affine transform, while :class:`InstanceNorm3d` usually don't apply affine transform. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, D, H, W)` or :math:`(C, D, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: ``False``. track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: ``False`` Shape: - Input: :math:`(N, C, D, H, W)` or :math:`(C, D, H, W)` - Output: :math:`(N, C, D, H, W)` or :math:`(C, D, H, W)` (same shape as input) Examples:: >>> # Without Learnable Parameters >>> m = nn.InstanceNorm3d(100) >>> # With Learnable Parameters >>> m = nn.InstanceNorm3d(100, affine=True) >>> input = torch.randn(20, 100, 35, 45, 10) >>> output = m(input) cCsdSNrNrr rrrr!vsz InstanceNorm3d._get_no_batch_dimcCs"|dkrtd|dSN)rNz'expected 4D or 5D input (got {}D input)rErrrrrys zInstanceNorm3d._check_input_dimNrGrrrrrP2sCrPc@s$eZdZdZeZddZddZdS)LazyInstanceNorm3daA :class:`torch.nn.InstanceNorm3d` module with lazy initialization of the ``num_features`` argument of the :class:`InstanceNorm3d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, D, H, W)` or :math:`(C, D, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters, initialized the same way as done for batch normalization. Default: ``False``. track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics and always uses batch statistics in both training and eval modes. Default: ``False`` Shape: - Input: :math:`(N, C, D, H, W)` or :math:`(C, D, H, W)` - Output: :math:`(N, C, D, H, W)` or :math:`(C, D, H, W)` (same shape as input) cCsdSrQrr rrrr!sz$LazyInstanceNorm3d._get_no_batch_dimcCs"|dkrtd|dSrRrErrrrrs z#LazyInstanceNorm3d._check_input_dimN)r2r;r<rHrPrJr!rrrrrrTsrTN)ZtorchrZ batchnormrrrr$rrArIrKrOrPrTrrrrs  DL'M'M