U d@sddlZddlZddlmZmZejjejjhZejj j ejj j hZ e e ZddddZddddZdd Zd d Zdd dZdddZdddZdddZd ddZd!ddZdS)"N)DictAny)returncCs2t|tkrtj||_ntj||d_dSNr)type_supported_typestorchnn Parameterweight)moduler r C/tmp/pip-unpacked-wheel-ua33x9lu/torch/ao/quantization/_equalize.pyset_module_weight s rcCs2t|tkrtj||_ntj||d_dSr)rrrr r bias)r rr r rset_module_biass rcCs t|tkr|jS|djSdSr)rrr r r r rget_module_weights rcCs t|tkr|jS|djSdSr)rrrrr r rget_module_biass rFcCs*|jdd|D]}|||\}}q|S)z/ Applies 'torch.max' over the given axises Treverse)sortmaxinput axis_listZkeepdimaxis_r r r max_over_ndim!s rcCs*|jdd|D]}|||\}}q|S)z/ Applies 'torch.min' over the given axises Tr)rminrr r r min_over_ndim)s r cCsT|j}tt|}||t||}t||}|d||ksLtd||S)zC finds the range of weights associated with a specific channel rzKDimensions of resultant channel range does not match size of requested axis)ndimlistrangeremover rsizeAssertionError)rrZsize_of_tensor_dimrZminsZmaxsr r r channel_range1s    r'c Cst|tkst|tkr.tdt|dt|t|}t|}||||krZtdt|}t||}t||}|d7}t ||} t | } || }dg|j } ||| |<dg|j } ||| |<t | | } t | | } || }|| }t ||t||t ||dS)z Given two adjacent tensors', the weights are scaled such that the ranges of the first tensors' output channel are equal to the ranges of the second tensors' input channel zmodule type not supported: z_Number of output channels of first arg do not match number input channels of second argg& .>r(N)r_all_supported_types ValueErrorrr% TypeErrorrr'rsqrtZ reciprocalr!Zreshaperr) Zmodule1Zmodule2Z output_axisZ input_axisZweight1Zweight2rZ weight1_rangeZ weight2_rangeZscaling_factorsZinverse_scaling_factorsZsize1Zsize2r r rcross_layer_equalization>s0         r.-C6?Tc Cs|st|}i}i}dd|D}|D] \}}||kr,|||<d||<q,t|||s|D]R} t|| d|| d<t|| d|| d<t|| d|| dq^qN|S)a) Given a list of adjacent modules within a model, equalization will be applied between each pair, this will repeated until convergence is achieved Keeps a copy of the changing modules from the previous iteration, if the copies are not that different than the current modules (determined by converged_test), then the modules have converged enough that further equalizing is not necessary Implementation of this referced section 4.1 of this paper https://arxiv.org/pdf/1906.04721.pdf Args: model: a model (nn.module) that equalization is to be applied on paired_modules_list: a list of lists where each sublist is a pair of two submodules found in the model, for each pair the two submodules generally have to be adjacent in the model to get expected/reasonable results threshold: a number used by the converged function to determine what degree similarity between models is necessary for them to be called equivalent inplace: determines if function is inplace or not cSsh|]}|D]}|q qSr r ).0pairnamer r r szequalize..Nrr()copydeepcopyZ named_modules convergedr.) modelZpaired_modules_list thresholdZinplaceZname_to_moduleZprevious_name_to_moduleZname_setr2r r1r r requalizejs   r9cCs|||krtdtd}d|kr2dS|D]4}t||}t||}||}|t|7}q:t||kS)a` Tests for the summed norm of the differences between each set of modules being less than the given threshold Takes two dictionaries mapping names to modules, the set of names for each dictionary should be the same, looping over the set of names, for each name take the differnce between the associated modules in each dictionary zIThe keys to the given mappings must have the same set of names of modulesgNF) keysr+rZtensorvaluesrsubZnormbool)Z curr_modulesZ prev_modulesr8Z summed_normsr2Z curr_weightZ prev_weight differencer r rr6s       r6)F)F)r)rr()r/T)r/)rr4typingrrr ZConv2dZLinearrZ intrinsicZ ConvReLU2dZ LinearReLUZ_supported_intrinsic_typesunionr*rrrrrr r'r.r9r6r r r rs    , '