U d@s ddlmZmZddlZddlmZddlmZmZmZddl m Z ddl m Z dd l mZdd lmZdd lmZGd d d eZGdddeZGdddeeZGdddeZGdddeeZGdddeZGdddeeZGdddeZGdddeeZGdddeZdS) )OptionalAnyN)Tensor) ParameterUninitializedParameterUninitializedBuffer) functional)init) SyncBatchNorm)LazyModuleMixin)ModulecseZdZUdZdZdddddgZeed<eed<eed<e ed<e ed<deeee e d d fd d Z d dddZ d dddZ ddZ ddZfddZZS) _NormBasez+Common base of _InstanceNorm and _BatchNormrtrack_running_statsmomentumeps num_featuresaffineh㈵>皙?TNrrrrrreturnc s||d}tt|||_||_||_||_||_|jrftt j |f||_ tt j |f||_ n| dd| dd|jr|dt j|f||dt j|f||||dt jd dt jid d |D|n$|dd|dd|dd|dS) Ndevicedtypeweightbias running_mean running_varnum_batches_trackedrrcSsi|]\}}|dkr||qSr.0kvr"r">/tmp/pip-unpacked-wheel-ua33x9lu/torch/nn/modules/batchnorm.py 9sz&_NormBase.__init__..)r)superr__init__rrrrrrtorchemptyrrZregister_parameterZregister_bufferzerosZonestensorlongitemsreset_parameters selfrrrrrrrfactory_kwargs __class__r"r'r*s6      z_NormBase.__init__rcCs*|jr&|j|jd|jdSNr )rrZzero_rZfill_r r3r"r"r'reset_running_statsAs  z_NormBase.reset_running_statscCs*||jr&t|jt|jdSN)r:rr Zones_rZzeros_rr9r"r"r'r1Is z_NormBase.reset_parameterscCstdSr;)NotImplementedErrorr3inputr"r"r'_check_input_dimOsz_NormBase._check_input_dimcCsdjf|jS)Nzj{num_features}, eps={eps}, momentum={momentum}, affine={affine}, track_running_stats={track_running_stats})format__dict__r9r"r"r' extra_reprRsz_NormBase.extra_reprc sf|dd}|dks|dkrF|jrF|d} | |krFtjdtjd|| <tt||||||||dS)Nversionrr rr!)getrr+r.r/r)r_load_from_state_dict) r3Z state_dictprefixZlocal_metadatastrictZ missing_keysZunexpected_keysZ error_msgsrCZnum_batches_tracked_keyr5r"r'rEXs  z_NormBase._load_from_state_dict)rrTTNN)__name__ __module__ __qualname____doc___versionZ __constants__int__annotations__floatboolr*r:r1r?rBrE __classcell__r"r"r5r'rs6 &rcs>eZdZd eeeeeddfdd Zeedd d ZZ S) _BatchNormrrTNrc s*||d}tt|j|||||f|dSNr)r)rRr*r2r5r"r'r*ws  z_BatchNorm.__init__r>rc Cs|||jdkrd}n|j}|jrb|jrb|jdk rb|jd|jdkr\dt|j}n|j}|jrnd}n|jdko|jdk}t ||jr|jr|jnd|jr|jr|jnd|j |j |||j S)Nr ?T)r?rtrainingrr add_rOrrF batch_normrrr)r3r>exponential_average_factor bn_trainingr"r"r'forwards6      z_BatchNorm.forward)rrTTNN) rHrIrJrMrOrPr*rr]rQr"r"r5r'rRvsrRcsVeZdZUeed<eed<dddfdd Zddfd d Zddd d ZZS) _LazyNormBaserrrrTNr7cs||d}tt|jd||ddf|||_||_|jrPtf||_tf||_|jrtf||_ tf||_ t j ddt j idd|D|_dS)NrrFrcSsi|]\}}|dkr||qSr!r"r#r"r"r'r(sz*_LazyNormBase.__init__..)r)r)r^r*rrrrrrrrr+r.r/r0r )r3rrrrrrr4r5r"r'r*s2       z_LazyNormBase.__init__cs |s|jdkrtdS)Nr)has_uninitialized_paramsrr)r1r9r5r"r'r1sz_LazyNormBase.reset_parameterscCs|r|jd|_|jrZt|jts*tt|jts:t|j |jf|j |jf|j r|j |jf|j |jf| dSr8)r_shaperr isinstancerrAssertionErrorrZ materializerrrr1r=r"r"r'initialize_parameterss z#_LazyNormBase.initialize_parameters)rrTTNN) rHrIrJrrNr*r1rcrQr"r"r5r'r^s r^c@seZdZdZddZdS) BatchNorm1da Applies Batch Normalization over a 2D or 3D input as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{\sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the number of features or channels of the input). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, L)` slices, it's common terminology to call this Temporal Batch Normalization. Args: num_features: number of features or channels :math:`C` of the input eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` Shape: - Input: :math:`(N, C)` or :math:`(N, C, L)`, where :math:`N` is the batch size, :math:`C` is the number of features or channels, and :math:`L` is the sequence length - Output: :math:`(N, C)` or :math:`(N, C, L)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm1d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm1d(100, affine=False) >>> input = torch.randn(20, 100) >>> output = m(input) cCs.|dkr*|dkr*td|dSNrz'expected 2D or 3D input (got {}D input)Zdim ValueErrorr@r=r"r"r'r?(s zBatchNorm1d._check_input_dimNrHrIrJrKr?r"r"r"r'rdsArdc@seZdZdZeZddZdS)LazyBatchNorm1da6A :class:`torch.nn.BatchNorm1d` module with lazy initialization of the ``num_features`` argument of the :class:`BatchNorm1d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` cCs.|dkr*|dkr*td|dSrergr=r"r"r'r?Ks z LazyBatchNorm1d._check_input_dimN)rHrIrJrKrd cls_to_becomer?r"r"r"r'rj/srjc@seZdZdZddZdS) BatchNorm2da Applies Batch Normalization over a 4D input (a mini-batch of 2D inputs with additional channel dimension) as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, H, W)` slices, it's common terminology to call this Spatial Batch Normalization. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` Shape: - Input: :math:`(N, C, H, W)` - Output: :math:`(N, C, H, W)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm2d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm2d(100, affine=False) >>> input = torch.randn(20, 100, 35, 45) >>> output = m(input) cCs"|dkrtd|dSNz!expected 4D input (got {}D input)rgr=r"r"r'r?s zBatchNorm2d._check_input_dimNrir"r"r"r'rlRsBrlc@seZdZdZeZddZdS)LazyBatchNorm2da6A :class:`torch.nn.BatchNorm2d` module with lazy initialization of the ``num_features`` argument of the :class:`BatchNorm2d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` cCs"|dkrtd|dSrmrgr=r"r"r'r?s z LazyBatchNorm2d._check_input_dimN)rHrIrJrKrlrkr?r"r"r"r'rosroc@seZdZdZddZdS) BatchNorm3da Applies Batch Normalization over a 5D input (a mini-batch of 3D inputs with additional channel dimension) as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over the mini-batches and :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are set to 1 and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done over the `C` dimension, computing statistics on `(N, D, H, W)` slices, it's common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, D, H, W)` eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` Shape: - Input: :math:`(N, C, D, H, W)` - Output: :math:`(N, C, D, H, W)` (same shape as input) Examples:: >>> # With Learnable Parameters >>> m = nn.BatchNorm3d(100) >>> # Without Learnable Parameters >>> m = nn.BatchNorm3d(100, affine=False) >>> input = torch.randn(20, 100, 35, 45, 10) >>> output = m(input) cCs"|dkrtd|dSNz!expected 5D input (got {}D input)rgr=r"r"r'r?s zBatchNorm3d._check_input_dimNrir"r"r"r'rpsCrpc@seZdZdZeZddZdS)LazyBatchNorm3da6A :class:`torch.nn.BatchNorm3d` module with lazy initialization of the ``num_features`` argument of the :class:`BatchNorm3d` that is inferred from the ``input.size(1)``. The attributes that will be lazily initialized are `weight`, `bias`, `running_mean` and `running_var`. Check the :class:`torch.nn.modules.lazy.LazyModuleMixin` for further documentation on lazy modules and their limitations. Args: eps: a value added to the denominator for numerical stability. Default: 1e-5 momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` cCs"|dkrtd|dSrqrgr=r"r"r'r? s z LazyBatchNorm3d._check_input_dimN)rHrIrJrKrprkr?r"r"r"r'rssrsc sfeZdZdZdeeeeeeeddfdd Z d d Z d d Z e e d ddZ edddZZS)r aApplies Batch Normalization over a N-Dimensional input (a mini-batch of [N-2]D inputs with additional channel dimension) as described in the paper `Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift `__ . .. math:: y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta The mean and standard-deviation are calculated per-dimension over all mini-batches of the same process groups. :math:`\gamma` and :math:`\beta` are learnable parameter vectors of size `C` (where `C` is the input size). By default, the elements of :math:`\gamma` are sampled from :math:`\mathcal{U}(0, 1)` and the elements of :math:`\beta` are set to 0. The standard-deviation is calculated via the biased estimator, equivalent to `torch.var(input, unbiased=False)`. Also by default, during training this layer keeps running estimates of its computed mean and variance, which are then used for normalization during evaluation. The running estimates are kept with a default :attr:`momentum` of 0.1. If :attr:`track_running_stats` is set to ``False``, this layer then does not keep running estimates, and batch statistics are instead used during evaluation time as well. .. note:: This :attr:`momentum` argument is different from one used in optimizer classes and the conventional notion of momentum. Mathematically, the update rule for running statistics here is :math:`\hat{x}_\text{new} = (1 - \text{momentum}) \times \hat{x} + \text{momentum} \times x_t`, where :math:`\hat{x}` is the estimated statistic and :math:`x_t` is the new observed value. Because the Batch Normalization is done for each channel in the ``C`` dimension, computing statistics on ``(N, +)`` slices, it's common terminology to call this Volumetric Batch Normalization or Spatio-temporal Batch Normalization. Currently :class:`SyncBatchNorm` only supports :class:`~torch.nn.DistributedDataParallel` (DDP) with single GPU per process. Use :meth:`torch.nn.SyncBatchNorm.convert_sync_batchnorm()` to convert :attr:`BatchNorm*D` layer to :class:`SyncBatchNorm` before wrapping Network with DDP. Args: num_features: :math:`C` from an expected input of size :math:`(N, C, +)` eps: a value added to the denominator for numerical stability. Default: ``1e-5`` momentum: the value used for the running_mean and running_var computation. Can be set to ``None`` for cumulative moving average (i.e. simple average). Default: 0.1 affine: a boolean value that when set to ``True``, this module has learnable affine parameters. Default: ``True`` track_running_stats: a boolean value that when set to ``True``, this module tracks the running mean and variance, and when set to ``False``, this module does not track such statistics, and initializes statistics buffers :attr:`running_mean` and :attr:`running_var` as ``None``. When these buffers are ``None``, this module always uses batch statistics. in both training and eval modes. Default: ``True`` process_group: synchronization of stats happen within each process group individually. Default behavior is synchronization across the whole world Shape: - Input: :math:`(N, C, +)` - Output: :math:`(N, C, +)` (same shape as input) .. note:: Synchronization of batchnorm statistics occurs only while training, i.e. synchronization is disabled when ``model.eval()`` is set or if ``self.training`` is otherwise ``False``. Examples:: >>> # With Learnable Parameters >>> m = nn.SyncBatchNorm(100) >>> # creating process group (optional) >>> # ranks is a list of int identifying rank ids. >>> ranks = list(range(8)) >>> r1, r2 = ranks[:4], ranks[4:] >>> # Note: every rank calls into new_group for every >>> # process group created, even if that rank is not >>> # part of the group. >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]] >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1] >>> # Without Learnable Parameters >>> m = nn.BatchNorm3d(100, affine=False, process_group=process_group) >>> input = torch.randn(20, 100, 35, 45, 10) >>> output = m(input) >>> # network is nn.BatchNorm layer >>> sync_bn_network = nn.SyncBatchNorm.convert_sync_batchnorm(network, process_group) >>> # only single gpu per process is currently supported >>> ddp_sync_bn_network = torch.nn.parallel.DistributedDataParallel( >>> sync_bn_network, >>> device_ids=[args.local_rank], >>> output_device=args.local_rank) rrTN)rrrrr process_grouprc s0||d} tt|j|||||f| ||_dSrS)r)r r*rt) r3rrrrrrtrrr4r5r"r'r*s  zSyncBatchNorm.__init__cCs"|dkrtd|dS)Nrz*expected at least 2D input (got {}D input)rgr=r"r"r'r?s  zSyncBatchNorm._check_input_dimcCs|ddkrtddS)Nr rz9SyncBatchNorm number of input channels should be non-zero)sizerhr=r"r"r'_check_non_zero_input_channelssz,SyncBatchNorm._check_non_zero_input_channelsrTc CsV|jstd|||||jdkr2d}n|j}|jr~|jr~|jdk sRt|j d|jdkrxd|j }n|j}|jrd}n|j dko|j dk}|jr|jr|j nd}|jr|jr|j nd}|o|j}|rt jjj}|jr|j}t j|}|dk}|s(t||||j|j|||jS|s2tt||j|j|||j||| SdS)Nz0SyncBatchNorm expected input tensor to be on GPUrUr rVT)Zis_cudarhr?rvrrWrr rbrXitemrrr+Z distributedgroupZWORLDrtZget_world_sizerYrZrrrsync_batch_normapply) r3r>r[r\rrZ need_syncrtZ world_sizer"r"r'r]sd           zSyncBatchNorm.forwardc Cs|}t|tjjjjrtj|j|j|j |j |j |}|j r`t |j |_ |j|_W5QRX|j|_|j|_|j|_t|dr|j|_|D]\}}|||||q~|S)a Helper function to convert all :attr:`BatchNorm*D` layers in the model to :class:`torch.nn.SyncBatchNorm` layers. Args: module (nn.Module): module containing one or more :attr:`BatchNorm*D` layers process_group (optional): process group to scope synchronization, default is the whole world Returns: The original :attr:`module` with the converted :class:`torch.nn.SyncBatchNorm` layers. If the original :attr:`module` is a :attr:`BatchNorm*D` layer, a new :class:`torch.nn.SyncBatchNorm` layer object will be returned instead. Example:: >>> # Network with nn.BatchNorm layer >>> module = torch.nn.Sequential( >>> torch.nn.Linear(20, 100), >>> torch.nn.BatchNorm1d(100), >>> ).cuda() >>> # creating process group (optional) >>> # ranks is a list of int identifying rank ids. >>> ranks = list(range(8)) >>> r1, r2 = ranks[:4], ranks[4:] >>> # Note: every rank calls into new_group for every >>> # process group created, even if that rank is not >>> # part of the group. >>> process_groups = [torch.distributed.new_group(pids) for pids in [r1, r2]] >>> process_group = process_groups[0 if dist.get_rank() <= 3 else 1] >>> sync_bn_module = torch.nn.SyncBatchNorm.convert_sync_batchnorm(module, process_group) qconfig)rar+nnmodulesZ batchnormrRr rrrrrZno_gradrrrrr hasattrr{Znamed_childrenZ add_moduleconvert_sync_batchnorm)clsmodulertZ module_outputnamechildr"r"r'rs4#   z$SyncBatchNorm.convert_sync_batchnorm)rrTTNNN)N)rHrIrJrKrMrOrPrrr*r?rvrr] classmethodrrQr"r"r5r'r %s,gQr )typingrrr+rZtorch.nn.parameterrrrr rYr Z _functionsr ryZlazyr rrrrRr^rdrjrlrorprsr"r"r"r's$      hA/I#H!I!