U d?^@s UddlZddlZddlmZddlmZmZddlmZddl m Z m Z ddl m Z mZddlmZmZmZmZmZmZmZmZmZddlmZddlZddlZejjjdgejjjdgejjj dgejjj!dgejjj"d gejjj#d gejjj$d giZ%ejjj"ejjj#ejjj$hZ&ejjjd gejjjd gejjj d gejjj!d gejjj"d gejjj#d gejjj$d giZ'dXe(d ddZ)ddZ*eeee(eee(fee(effdddZ+dYeejj,eee(ejj,fe ee(ee(e-ffe.e(ed ddZ/eed ddZ0ddZ1eedddZ2ee.ed d!d"Z3e(ed#d$d%Z4eeeed&d'd(Z5e eee d)d*d+Z6ejj,ed,d-d.Z7ejj,e e(eed/d0d1Z8e(eeee(ejj,fe ee(ee(e-ffeeefd2d3d4Z9eee(ejj,feee.fe.d5d6d7Z:eee;d&d8d9ZZ>e>d?d@ej.e=dge?e=d gie>d?dAe;ed?dBe;ed?dCe;ed?dDe;e=dgie>d?dEe;ed?ej@e;ed?dFe;e=dgie>d?dGe;e=dgie>d?ejAe;e=dgie>d?dHe;eeee-ejCfeegee;fffeDdI<iZEeee-ejCfeegee;ffeDdJ<eeee-ejCfeegee;ffd&dKdLZFee.d&dMdNZGee.d&dOdPZHdZeee(ej,feeej,eeedQdRdSZIe eedTfeedUdVdWZJdS)[N) is_per_tensoris_per_channel)is_activation_post_process) GraphModulemap_arg)GraphNode) CallableOptionalListDictAnySetTupleUnionType) namedtupleT)returncCstd}td}dddddd}i}d }|D]}t|||<q0g}|jD]} t| j} |rl| d d } t| j} |r| |kr|| } t| j} |r| | } | rd | d d} | | }|rd| d d} | d d } t| j }|r|d d }t| j }t || | | ||fD] \}}t||t|||<q|| | | ||gqLd}d}||j||7}|D]}||j||7}qp|r|d7}|S)zyReturns a printable representation of the ops in the graph of g. If shorten is True, tries to abbreviate fields. zz ZplchdrZgt_prmZcl_funZcl_modZcl_meth) placeholderget_attr call_function call_module call_method)nameoptargetargskwargsactivation_post_processZobszz t|jstd|\}}t|}t|}|j}|||fS)Nz)Only per tensor quantization is supported)rqschemeAssertionErrorcalculate_qparamsfloatintdtype)r"scale zero_pointrBr:r:r;get_per_tensor_qparamsls  rE)r"rc Cs|j}d}t|dr|j}d}|tjtjfkrd}|\}}t|jrht |j }||||d}tj }qt |}t |}|||d}tj }nr|tjkrd}d}d|i}nV|tjkr|tjtjtjfkrd}tj}tjjjd k} || d }ntd |dS|||fS) z Given an activation_post_process module, return node_type(e.g. call_function), quantize op(e.g. quantize_per_tensor) and a dictionary of extracted qparams from the module N compute_dtyper)_scale_ _zero_point_Z_axis__dtype_)rGrHrIrtorIZfbgemm)rIZ_reduce_range_z?Unsupported activation_post_process in get_quantize_node_info: )rBhasattrrFtorchZquint8qint8r?rr=rAch_axisZquantize_per_channelr@Zquantize_per_tensorfloat16Zfloat32Zquantize_per_tensor_dynamicbackends quantizedZenginewarningswarn) r"rBrF quantize_op node_typerCrDrNqparamsZ reduce_ranger:r:r;get_quantize_node_infots8        rW_output) in_node obs_moduleobs_nodemodulesquantized_graphnode_name_to_scopeis_input output_prefixrcCs|r\t|j}|r|dnd} d} |D]&} | jdkr&| jtjjjkr&| } qNq&| rV| } d} n|} |} | r| j|kr|| j\} }nd} |d}|}t |}|dk st d||\}}}|g}| D]<\}}|dkrt ||| | ||}| |q| |q|||t|iS)a  Add quantization nodes (eg. quantize_per_tensor/per_channel) for given node to graph with the qparams calculated from activation_post_process (obs_module). The observer node (obs_node) is used to find the FQN of the user of act_post_process. e.g. Given input `node` in `node = self.conv(x)`, insert node: `quantized_node = torch.quantize_per_tensor(x, self._scale_0, self._zer_point_0, self._dtype_0)` where self._scale_0, self._zero_point_0 and self._dtype_0 are calculated from `obs_module` rNr_inputr$z7Expecting quantize node info not to be None, observer: )rGrH)listusersrrrLnn functionallinearrrWr>itemscreate_getattr_from_valuer/ create_nodetuple)rYrZr[r\r]r^r_r`rcZfirst_linear_use_or_first_useZ linear_noder5prefix module_path_ root_modulegraphZmaybe_quantize_node_inforUrTrVinputskeyvalueZ qparam_noder:r:r; quantize_nodes<     rscCsDt}||i}dD]$}||i}t|}||O}qt|S)a_ Get all the unique custom module keys in the custom config dict e.g. Input: custom_config_dict = { "float_to_observed_custom_module_class": { "static": { CustomModule1: ObservedCustomModule }, "dynamic": { CustomModule2: DynamicObservedCustomModule }, "weight_only": { CustomModule3: WeightOnlyObservedCustomModule }, }, } Output: # extract all the keys in "static", "dynamic" and "weight_only" dict [CustomModule1, CustomModule2, CustomModule3] )ZstaticZdynamicZ weight_only)setgetkeysrb)Zcustom_config_dictZcustom_config_dict_keyZfloat_custom_module_classesZcustom_module_mappingZ quant_modeZquant_mode_custom_module_configZ quant_mode_custom_module_classesr:r:r;get_custom_module_class_keyss    rwcCs6|tjkrtjjjS|tjkr(tjjjStd|dS)Nz&can't get linear prepack op for dtype:)rLrOopsrQZlinear_prepack_fp16rMZlinear_prepack Exception)rBr:r:r;get_linear_prepack_op_for_dtypes     rz)conv_oprcCsVtjjjtjjjtjjjtjjjtjjj tjjj i}| |d}|sRt d ||S)NzDidn't find prepack op for {})rLrdreconv1drxrQZconv1d_prepackconv2dZconv2d_prepackconv3dZconv3d_prepackrur>r0)r{Z prepack_opsZ prepack_opr:r:r;get_qconv_prepack_ops r)r{has_relurcCstjjjtjjjtjjjtjjjtjjj tjjj itjjjtjjjtjjjtjjjtjjj tjjj id}|| |}|st d |||S)N)TFz4Can't find corresponding quantized conv op for {} {})rLrdrer|rxrQZ conv1d_relur}Z conv2d_relur~Z conv3d_relurur>r0)r{rZqconv_opZqconvr:r:r; get_qconv_op s$ r)rkrcs&ddtjjdfdd }|S)N.rm)modulecs>tdfdd }d}||}t||r:|d7}||}q|S)Nics t|SN)r)rrkr:r; get_attr_name%szOget_new_attr_name_with_prefix..get_new_attr_name..get_attr_namerr)rArK)rrr attr_namerr:r;get_new_attr_name$s  z8get_new_attr_name_with_prefix..get_new_attr_name)r*rLrdModule)rkrr:rr;get_new_attr_name_with_prefix!s  rnodercCs|g}|g}|r|}t|jt|j}|D]H}t|tsDq4|jdkrTdS|||jdkrr|j t ks4||q4q |S)a Starting from a target node, trace back until we hit inpu or getattr node. This is used to extract the chain of operators starting from getattr to the target node, for example def forward(self, x): observed = self.observer(self.weight) return F.linear(x, observed) collect_producer_nodes(observed) will either return a list of nodes that produces the observed node or None if we can't extract a self contained graph without free variables(inputs of the forward function). rNr) poprbr r!values isinstancerrr/rgetattr)rr(ZfrontierZall_argsargr:r:r;collect_producer_nodes/s    r)rootproducer_nodesrcslt|dkstd|t}ifdd}|D]}||||<q6|||dt||}|S)a- Construct a graph module from extracted producer nodes from `collect_producer_nodes` function Args: root: the root module for the original graph producer_nodes: a list of nodes we use to construct the graph Return: A graph module constructed from the producer nodes rz'list of producer nodes can not be emptycst|fddS)Ncs|Srr:renvr:r;[zDgraph_module_from_producer_nodes..load_arg..)r)arr:r;load_argZsz2graph_module_from_producer_nodes..load_arg)r'r>reverserZ node_copyoutputr)rrrorZ producer_nodeZ graph_moduler:rr; graph_module_from_producer_nodesJs   r)rrcCs^dd|Ddd|DB}t|dks>td|t|dkrVtt|nd}|S)z Returns the unique device for a module, or None if no device is found. Throws an error if multiple devices are detected. cSsh|] }|jqSr:device.0pr:r:r; gsz/assert_and_get_unique_device..cSsh|] }|jqSr:rrr:r:r;rhsrzMprepare only works with cpu or single-device CUDA modules, but got devices {}rN) parametersbuffersr'r>r0nextiter)rZdevicesrr:r:r;assert_and_get_unique_devicebsr)rrorkrrrcCs>t|}||}t|}||tj||d|d|}|S)z Given a value of any type, creates a getattr node corresponding to the value and registers the value as a buffer to the module. rr)rrZregister_bufferrLZtensorri)rrorkrrrrrZ attr_noder:r:r;rhps  rh) node_namerCrDr\r]r^rc Cs@|d}||\}}t|||d|} t|||d|} | | fS)z Create getattr nodes in the quantized graph for scale and zero point values. The nodes are registered with the root_module of the model. r$rGrH)rh) rrCrDr\r]r^rnrlrmZ scale_nodeZzero_point_noder:r:r;create_qparam_nodes}s  r)rr\cacherc Cs|r||kr||Sd}t|ts*d}n|jdkrrall_node_args_have_no_tensorsr operatorgetitemrrbrA) rr\rr9Zfound_one_tensorrZlist_elZ!this_list_el_args_have_no_tensorsZthis_arg_args_have_no_tensorsr:r:r;rsj               rcCsttdt|jS)z2 Returns all node arg indices after first r)rbranger'r rr:r:r;all_node_args_except_firstsr) arg_indicesrcstttdfdd }|S)zu Constructs a function that takes a node as arg and returns the arg_indices that are valid for node.args rcsfddDS)Ncsg|]}|tjkr|qSr:)r'r )rrrr:r; sz=return_arg_list..arg_indices_func..r:rrrr;arg_indices_funcsz)return_arg_list..arg_indices_func)rr rA)rrr:rr;return_arg_listsrNodeInfoz op targetrZ masked_fillZpermuterepeatZreshaper transpose unsqueezeZ unsqueeze_viewNON_OBSERVABLE_ARG_DICTEMPTY_ARG_DICTcCst|j|j}t|tS)z Returns a dict with of non float tensor types as keys and values which correspond to a function to retrieve the list (which takes the node as an argument) )rrrrrur)rinfor:r:r;(get_non_observable_arg_indexes_and_typessrcCs|jdko|jdkS)zd Returns true if this node results in an integer, even if some of the args are Tensors. rr)rrrr:r:r;node_return_type_is_int srcCs&|jdko |jtko |jddk}|S)z Returns True if this node is a node that takes a Tensor as input and output some meta information about the Tensor, e.g. shape, size etc. rrr)rrrr )rr9r:r:r;is_get_tensor_info_node(s r)rr\target_module_typetarget_functional_typercCsj|jD]Z\}}|jdkr@|dk r@t|t|j|r@|S|jdkr |dk r |j|kr |Sq dS)a% Gets the next module that matches what is needed in is_target_module_type if it exists Args: node: The node whose users we want to look at target_module_type: Module type that we want to check target_functional_type: Functional type that we want to check rNr)rcrgrrr)r)rr\rruserrmr:r:r;maybe_get_next_module0s r.)r]create_node_argsold_nodercCs|j|}|j|_|S)zU Creates `new_node` and copies the necessary metadata to it from `old_node`. )riZ stack_trace)r]rrZnew_noder:r:r;'create_node_from_old_node_preserve_metaIs r)T)rX)NN)Kr%rLZtorch.nnrdZtorch.ao.quantization.utilsrrZtorch.ao.quantization.quantizerZtorch.fxrrZtorch.fx.graphrrtypingr r r r r rrrr collectionsrrrRrer|r}r~rfZ layer_normZ group_normZ instance_normZWEIGHT_INDEX_DICTZNON_QUANTIZABLE_WEIGHT_OPSZBIAS_INDEX_DICTr)r<rErWrtypeboolrsrwrzrrrrrrrhrrrArrrr@rrrrB__annotations__rrrrrrr:r:r:r;s$  ,   B./ >   (K"   4%,0