U dWk@sddlZddlZddlZddlZddlmZddlmmZddl m Z ddl m Z m Z mZmZmZmZmZmZddlmZmZddlmZmZddlmZmZmZmZmZm Z ddl!m"Z"d d Z#d4d d Z$d5ddZ%ddZ&ddZ'd6ddZ(d7ddZ)ddZ*ddZ+d8ddZ,dd Z-d!d"Z.d9d#d$Z/dej0ddfd%d&Z1d:d'd(Z2d;d)d*Z3dd2d3Z7dS)?N) _FusedModule))get_default_dynamic_quant_module_mappings(get_default_static_quant_module_mappings2get_default_static_quant_reference_module_mappingsget_default_qat_module_mappings$get_default_qconfig_propagation_listno_observer_set_has_special_act_post_process_get_special_act_post_process)get_qparam_dict)has_no_children_ignoring_parametrizations) DeQuantStub QuantWrapper)add_module_to_qconfig_obs_ctrdefault_dynamic_qconfigfloat16_dynamic_qconfig!float_qparams_weight_only_qconfig&float_qparams_weight_only_qconfig_4bitactivation_is_memoryless)type_before_parametrizationscCs t|tjjjpt|tjjjSN) isinstancetorchao quantizationZ ObserverBaseZFakeQuantizeBasemodulerB/tmp/pip-unpacked-wheel-ua33x9lu/torch/ao/quantization/quantize.pyis_activation_post_processsr c Cs|t||}|||}t|d|}tjjj||t||}||_| D]V\}}|rj|d|n|} |dks||dgksRt ||dgksRt |||| qRdS)aThis is a helper function for `propagate_qconfig_` Args: module: input module qconfig_dict: dictionary that maps from name of submodule to quantization configuration qconfig_parent: quantization config of parent module, we will fallback to this config when there is no specified config for current module prefix: corresponding prefix of the current module, used as key in qconfig_dict prepare_custom_config_dict: dictionary for custom handling of modules see docs for :func:`~torch.ao.quantization.prepare_fx` Return: None, module is modified inplace with qconfig attached qconfig.NZnon_traceable_module_nameZnon_traceable_module_class) getrgetattrrrrr"Zassert_valid_qconfigrnamed_childrentype_propagate_qconfig_helper) r qconfig_dictZqconfig_parentprefixprepare_custom_config_dictZmodule_qconfigZqconfig_with_device_checknamechild module_prefixrrrr($s&   r(cCs*|dkr i}|dkri}t|||ddS)aPropagate qconfig through the module hierarchy and assign `qconfig` attribute on each leaf module Args: module: input module qconfig_dict: dictionary that maps from name or type of submodule to quantization configuration, qconfig applies to all submodules of a given module unless qconfig for the submodules are specified (when the submodule already has qconfig attribute) prepare_custom_config_dict: dictionary for custom handling of modules see docs for :func:`~torch.ao.quantization.prepare_fx` Return: None, module is modified inplace with qconfig attached N)r+)r()rr)r+rrrpropagate_qconfig_Ls r/cCs ||S)z3Forward hook that calls observer on the output activation_post_process)selfinputoutputrrr_observer_forward_hookbsr5cCs||dS)z7Forward pre hook that calls observer on the output rr0)r2r3rrr_observer_forward_pre_hookgsr6FcCsTt|dstd|r4|t}|jj|jddn|t}|j j|jdddS)Nr1zGExpect activation_post_process attribute already attached to the moduleF)last) hasattrAssertionErrorregister_forward_pre_hookr6_forward_pre_hooks move_to_endidregister_forward_hookr5_forward_hooks)rpre_hookhandlerrr%register_activation_post_process_hookls   rBc s|dkrt}|dkri}dkr`t|}t|dksDtd|t|dkr\tt|ndd ddddd fd d }|D]\}}t|t j fkrqqt|t j t j fkrֈ|rԈ|j|_qt|tr|r||qt|rt|} ||| q|dk r.get_activation_post_processcSst|do|jdk S)Nr"r8r")mrrrneeds_observationsz(add_observer_..needs_observationcs>|r:t|ts:|d|j|t|t|jddS)zn Adds an activation post process module and register a pre or post hook that calls the module r1r@N)rrZ add_moduler"rBr)rIrFrErGrJrrinsert_activation_post_processsz5add_observer_..insert_activation_post_process)N)N)rget_unique_devices_lenr9formatnextiterr&rnnZDropoutnnqZFloatFunctionalZ QFunctionalr"r1rrr r from_floatsetattrr add_observer_r rZ Sequential) rqconfig_propagation_listZnon_leaf_module_listrEcustom_module_class_mappingdevicesrMr,r-rFZobserved_childrrLrrWwsP           rWcCs$dd|Ddd|DBS)NcSsh|] }|jqSrrE.0prrr sz&get_unique_devices_..cSsh|] }|jqSrr[r\rrrr_s) parametersbuffersrrrrrNsrNcCsDt|r t|dr |jr t|S|D]\}}t||j|<q(|S)a{Wrap the leaf child module in QuantWrapper if it has a valid qconfig Note that this function will modify the children of module inplace and it can return a new module which wraps the input module as well. Args: module: input module with qconfig attributes for all the leaf modules that we want to quantize Return: Either the inplace modified module with submodules wrapped in `QuantWrapper` based on qconfig or a new `QuantWrapper` module which wraps the input module, the latter case only happens when the input module is a leaf module and we want to quantize it. r")r r8r"rr&add_quant_dequant_modules)rr,r-rrrrbs rbcCstjd|dkri}|di}|s2t|}|}|dkrDt}t|ddtdd| Dspt dt ||||d|S) aPrepares a copy of the model for quantization calibration or quantization-aware training. Quantization configuration should be assigned preemptively to individual submodules in `.qconfig` attribute. The model will be attached with observer or fake quant modules, and qconfig will be propagated. Args: `model`: input model to be modified in-place `inplace`: carry out model transformations in-place, the original module is mutated `allow_list`: list of quantizable modules `observer_non_leaf_module_list`: list of non-leaf modules we want to add observer `prepare_custom_config_dict`: customization configuration dictionary for prepare function .. code-block:: python # Example of prepare_custom_config_dict: prepare_custom_config_dict = { # user will manually define the corresponding observed # module class which has a from_float class method that converts # float custom module to observed custom module "float_to_observed_custom_module_class": { CustomModule: ObservedCustomModule } } z!quantization_api.quantize.prepareNZ%float_to_observed_custom_module_classr)css|]}t|do|jVqdS)r"NrH)r]rIrrr szprepare..zNone of the submodule got qconfig applied. Make sure you passed correct configuration through `qconfig_dict` or by assigning the `.qconfig` attribute directly on submodules)rY) r_C_log_api_usage_oncer$copydeepcopyrr/anymoduleswarningswarnrW)modelinplaceZ allow_listobserver_non_leaf_module_listr+rYrXrrrprepares&     rqcsDtdrtjrtddfdd }|dd|dddS)Nr1Fcs^|r jnj}|rtnt}t}|D]\}}||kr*||q*|D]}||qJdSr)r;r?r6r5setitemsaddpop)r@Zhook_mapZ observer_hookZhandle_ids_to_removeZ handle_idhook_fnrrr remove_hooks's  z5_remove_activation_post_process..remove_hooksTrK)F)r8r r1delattr)rrwrrr_remove_activation_post_processs   rycCs0|D] }t|qt|dr$|`t|dS)zClean up the qconfig left in the module so that new qconfig can be propagated. Args: module: module to be cleaned up r"N)children_remove_qconfigr8r"ry)rr-rrrr{5s    r{cCs\tjd|dkrt}|s(t|}|t|dd||f|t||dd|S)aQuantize the input float model with post training static quantization. First it will prepare the model for calibration, then it calls `run_fn` which will run the calibration step, after that we will convert the model to a quantized model. Args: model: input float model run_fn: a calibration function for calibrating the prepared model run_args: positional arguments for `run_fn` inplace: carry out model transformations in-place, the original module is mutated mapping: correspondence between original module types and quantized counterparts Return: Quantized model. z"quantization_api.quantize.quantizeNTro) rrfrgrrhrievalrqconvert)rnrun_fnrun_argsmappingrorrrquantizeDs   rc Csrtjd|dkr|tjkrHtjttjttjttj ttj ttj ti}nt|tj kr|tjt tjt tjt tj t tj t tj t i}n@|tjkrtjttjti}n$|tjkrtjti}ntd|npt|tr.|tjkrt}n@|tj krt }n0|tjkrt}n |tjkr t}ntdt|tt|t|}|dkr>t}|sNt |}|!t"||t#||dd|S)avConverts a float model to dynamic (i.e. weights-only) quantized model. Replaces specified modules with dynamic weight-only quantized versions and output the quantized model. For simplest usage provide `dtype` argument that can be float16 or qint8. Weight-only quantization by default is performed for layers with large weights size - i.e. Linear and RNN variants. Fine grained control is possible with `qconfig` and `mapping` that act similarly to `quantize()`. If `qconfig` is provided, the `dtype` argument is ignored. Args: model: input model qconfig_spec: Either: - A dictionary that maps from name or type of submodule to quantization configuration, qconfig applies to all submodules of a given module unless qconfig for the submodules are specified (when the submodule already has qconfig attribute). Entries in the dictionary need to be QConfig instances. - A set of types and/or submodule names to apply dynamic quantization to, in which case the `dtype` argument is used to specify the bit-width inplace: carry out model transformations in-place, the original module is mutated mapping: maps type of a submodule to a type of corresponding dynamically quantized version with which the submodule needs to be replaced z*quantization_api.quantize.quantize_dynamicNzTDon't know how to quantize with default settings for {}. Provide full qconfig pleasez.Unknown dtype specified for quantize_dynamic: Tr|)$rrfrgqint8rSZLinearrZLSTMZGRUZLSTMCellZRNNCellZGRUCellZfloat16rZquint8Z EmbeddingBagrZ EmbeddingZquint4x2r ValueErrorrPrrr RuntimeErrorstrdictzip itertoolsrepeatrrhrir}r/r~)rnZ qconfig_specZdtyperroZdefault_qconfigrrrquantize_dynamic`s|             rcCsltjd|jstd|dkr(t}|s6t|}t|ddt ||dddt |t | dd|S) a Prepares a copy of the model for quantization calibration or quantization-aware training and converts it to quantized version. Quantization configuration should be assigned preemptively to individual submodules in `.qconfig` attribute. Args: model: input model to be modified in-place mapping: dictionary that maps float modules to quantized modules to be replaced. inplace: carry out model transformations in-place, the original module is mutated z%quantization_api.quantize.prepare_qatz1prepare_qat only works on models in training modeNrdTF)rroremove_qconfig)rpro) rrfrgZtrainingr9rrhrir/r~rqrrvalues)rnrrorrr prepare_qats   rcCsLtjd|st|}|t|dd||f|t|dd|S)agDo quantization aware training and output a quantized model Args: model: input model run_fn: a function for evaluating the prepared model, can be a function that simply runs the prepared model or a training loop run_args: positional arguments for `run_fn` Return: Quantized model. z&quantization_api.quantize.quantize_qatTr|)rrfrgrhriZtrainrr~)rnrrrorrr quantize_qats    rTcCs<tjd|st|}t||d||d|r8t||S)agConverts submodules in input module to a different module according to `mapping` by calling `from_float` method on the target module class. And remove qconfig at the end if remove_qconfig is set to True. Args: `module`: prepared and calibrated module `mapping`: a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules `inplace`: carry out model transformations in-place, the original module is mutated `convert_custom_config_dict`: custom configuration dictionary for convert function .. code-block:: python # Example of convert_custom_config_dict: convert_custom_config_dict = { # user will manually define the corresponding quantized # module class which has a from_observed class method that converts # observed custom module to quantized custom module "observed_to_quantized_custom_module_class": { ObservedCustomModule: QuantizedCustomModule } } z!quantization_api.quantize.convertT)ro is_referenceconvert_custom_config_dict)rrfrgrhri_convertr{)rrrorrrrrrr~s  r~c Cs|dkr|rtnt}|dkr$i}|di}|s>t|}i}|D]>\}}t|tsxt||krxt ||d||t |||||<qJ| D]\} } | |j | <q|S)aConverts submodules in input module to a different module according to `mapping` by calling `from_float` method on the target module class Args: module: input module mapping: a dictionary that maps from source module type to target module type, can be overwritten to allow swapping user defined Modules inplace: carry out model transformations in-place, the original module is mutated is_reference: a flag to enable quantized reference module NZ)observed_to_quantized_custom_module_classT) rrr$rhrir&rrrr swap_modulersrc) rrrorrrYZreassignr,modkeyvaluerrrrs,      rc CsJ|}t|drF|jdk rFd}t||krD|t||}d}nnt||kr|t|}t|dr|jr|jdk szt|j}||jt|}|||}n ||}d}|rF|j D]}| |q|j D]} | t k r|| qt|} t| dkstd| t| dkr2tt| nd} | rF|| |S) a Swaps the module if it has a quantized counterpart and it has an `observer` attached. Args: mod: input module mapping: a dictionary that maps from nn module to nnq module Return: The corresponding quantized module of `mod` r"NFT _IS_REFERENCEr zQswap_module only works with cpu or single-device CUDA modules, but got devices {}r)r8r"rZ from_observedrr9Zweightr rUr;rr:r?r5r>rNrOrPrQrRrD) rrrYnew_modZswappedZqmodZweight_post_processZweight_qparamsZ pre_hook_fnrvrZrErrrr6s@          rcCsZdd}t|dr$|j|||d<|D](\}}|rD|||n|}t|||q,dS)a,Traverse the modules and save all observers into dict. This is mainly used for quantization accuracy debug Args: mod: the top module we want to save all observers prefix: the prefix for the current module target_dict: the dictionary used to save all the observers cSs|dkr |S|dS)Nr!r#r)r*rrr get_prefixpsz%get_observer_dict..get_prefixr1N)r8r1r&get_observer_dict)r target_dictr*rr,r-r.rrrrhs  r)Nr!N)NN)F)NNNN)FNNN)NF)NF)F)NFTFN)NFFN)r!)8rhrrlrZtorch.nnrSZtorch.nn.quantizedZ quantizedrTZtorch.nn.intrinsicrZ+torch.ao.quantization.quantization_mappingsrrrrrrr r utilsr r Ztorch.ao.quantization.stubsrrZtorch.ao.quantization.qconfigrrrrrrZtorch.nn.utils.parametrizerr r(r/r5r6rBrWrNrbrqryr{rrrrrr~rrrrrrrsb  (    (  V 8  U   ( (2