U df @s ddlmZmZmZmZmZmZmZddlZddl m Z ddl m Z ddl mZmZmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZmZmZm Z ddl!m"Z"ddl!m#Z#ej$j%ddddZ&ej$j%ddddZ'd8e e(eee)efeee)efe dddZ*Gddde+Z,Gddde+Z-Gddde Z.d9ej$j%ee(eee)efeee)efeee)efe(edd d!Z/d:ej$j%ee(eee)efeee)efe d"d#d$Z0d;ej$j%eee)efeee)efe d%d&d'Z1de e(eee)efe(e(ee)efee)efej$j%d/d0d1Z4d?e e(eee)efe(ee)efee)efej$j%d2d3d4Z5d@e e(eee)efej$j%d5d6d7Z6dS)A)DictAnyListCallableTupleOptionalSetN) GraphModule)Tracer)TargetNodeArgument) _FusedModule)fuse)prepare)convert) get_tensorrt_backend_config_dict)ObservedGraphModule))check_is_valid_convert_custom_config_dict&check_is_valid_fuse_custom_config_dict)check_is_valid_prepare_custom_config_dictcheck_is_valid_qconfig_dict)graph_pretty_str)get_custom_module_class_keys)modelreturncCs*t|ts&tdtt|dddS)Nz,input model must be a GraphModule, Got type:z Please make zsure to follow the tutorials.) isinstancer ValueErrorstrtype)rr!E/tmp/pip-unpacked-wheel-ua33x9lu/torch/ao/quantization/quantize_fx.py_check_is_graph_modules  r#cCsbg}|D],\}}t|tjjjr0||q t|q |D]}|j|=tjj |j|<q>dS)z1 Swap FloatFunctional with FXFloatFunctional N) Znamed_childrenrtorchnn quantizedZFloatFunctionalappend_swap_ff_with_fxffZ_modulesZFXFloatFunctional)rZmodules_to_swapnamemoduler!r!r"r("s  r() graph_moduleis_qatfuse_custom_config_dictbackend_config_dictrcCst|t||||S)z Internal helper function to fuse modules in preparation for quantization Args: graph_module: GraphModule object from symbolic tracing (torch.fx.symbolic_trace) )r#r)r+r,r-r.r!r!r"_fuse_fx1s r/cs(eZdZdZeedfdd ZZS)Scopea/ Scope object that records the module path and the module type of a module. Scope is used to track the information of the module that contains a Node in a Graph of GraphModule. For example:: class Sub(torch.nn.Module): def forward(self, x): # This will be a call_method Node in GraphModule, # scope for this would be (module_path="sub", module_type=Sub) return x.transpose(1, 2) class M(torch.nn.Module): def __init__(self): self.sub = Sub() def forward(self, x): # This will be a call_method Node as well, # scope for this would be (module_path="", None) x = x.transpose(1, 2) x = self.sub(x) return x ) module_path module_typecst||_||_dSN)super__init__r1r2)selfr1r2 __class__r!r"r5Ys zScope.__init__)__name__ __module__ __qualname____doc__rrr5 __classcell__r!r!r7r"r0Asr0cs>eZdZdZeejjedfdd Z ddZ ddZ Z S) ScopeContextManagerz A context manager to track the Scope of Node during symbolic tracing. When entering a forward function of a Module, we'll update the scope information of the current module, and when we exit, we'll restore the previous scope information. )scopecurrent_modulecurrent_module_pathcs8t|j|_|j|_||_||j_t||j_dSr3)r4r5r2prev_module_typer1prev_module_pathr?r )r6r?r@rAr7r!r"r5es  zScopeContextManager.__init__cCsdSr3r!)r6r!r!r" __enter__oszScopeContextManager.__enter__cGs|j|j_|j|j_dSr3)rCr?r1rBr2)r6argsr!r!r"__exit__rs  zScopeContextManager.__exit__) r9r:r;r<r0r$r%Modulerr5rDrFr=r!r!r7r"r>_s r>c seZdZeeeedfdd Zejj ee dddZ ejj ede fe e dfeee fe dfd d Zdeee edfeeefeeee ed fd d ZZS)QuantizationTracer)skipped_module_namesskipped_module_classescs2t||_||_tdd|_i|_d|_dS)NT)r4r5rIrJr0r?node_name_to_scopeZrecord_stack_traces)r6rIrJr7r!r"r5ys   zQuantizationTracer.__init__)mmodule_qualified_namercCs>|jdrt|tjj p<||jkpr?r4 call_module)r6rMrQrErRrNr7r!r"rSs zQuantizationTracer.call_moduleN)kindtargetrErRr) type_exprrcs2t||||||}|jj|jjf|j|j<|Sr3)r4 create_noder?r1r2rLr))r6rTrUrErRr)rVnoder7r!r"rWs  zQuantizationTracer.create_node)NN)r9r:r;rrrr5r$r%rGboolrPrrrrSr r rr rWr=r!r!r7r"rHxs*      rHF)r qconfig_dictr,prepare_custom_config_dictequalization_qconfig_dictr.is_standalone_modulerc Cs.|dkr i}|dkri}t|t|t||dg}|dg}t||s|dg} |dd| D7}|dg} |dd| D7}t|d } || 7}|d g} t||} t|| |}| D]}t||t ||qt ||||}t |||| j ||||d }| D]}t||t ||q|S) a= Internal helper function for prepare_fx Args: `model`, `qconfig_dict`, `prepare_custom_config_dict`, `equalization_qonfig_dict`: see docs for :func:`~torch.ao.quantization.prepare_fx` `is_standalone_module`: a boolean flag indicates whether we are quantizing a standalone module or not, a standalone module is a submodule of the parent module that is not inlined in the forward graph of the parent module, the way we quantize standalone module is described in: :func:`~torch.ao.quantization._prepare_standalone_module_fx` NZnon_traceable_module_nameZnon_traceable_module_classZstandalone_module_namecSsg|] }|dqSrr!.0configr!r!r" sz_prepare_fx..Zstandalone_module_classcSsg|] }|dqSr^r!r_r!r!r"rbsZ%float_to_observed_custom_module_classpreserved_attributes)r[r\r.r]) rrgetr(rrHr tracesetattrgetattrr/rrL)rrZr,r[r\r.r]rIrJZstandalone_module_name_configsZstandalone_module_class_configsZfloat_custom_module_classesrcZtracerr+ attr_namepreparedr!r!r" _prepare_fxst   rj)rrZr,r[r.rcCst|||||ddS)a [Internal use only] Prepare a standalone module, so that it can be used when quantizing the parent module. standalone_module means it a submodule that is not inlined in parent module, and will be quantized separately as one unit. How the standalone module is observed is specified by `input_quantized_idxs` and `output_quantized_idxs` in the prepare_custom_config for the standalone module Returns: * model(GraphModule): prepared standalone module. It has these attributes: * `_standalone_module_input_quantized_idxs(List[Int])`: a list of indexes for the graph input that is expected to be quantized, same as input_quantized_idxs configuration provided for the standalone module * `_standalone_module_output_quantized_idxs(List[Int])`: a list of indexs for the graph output that is quantized same as input_quantized_idxs configuration provided for the standalone module T)r.r])rj)rrZr,r[r.r!r!r"_prepare_standalone_module_fxsrk)rr-r.rcCsdtjdt|tj|}t}|r:t|dg}|D]}t||t ||q>t |d||S)aZ Fuse modules like conv+bn, conv+bn+relu etc, model must be in eval mode. Fusion rules are defined in torch.quantization.fx.fusion_pattern.py Args: * `model`: a torch.nn.Module model * `fuse_custom_config_dict`: Dictionary for custom configurations for fuse_fx, e.g.:: fuse_custom_config_dict = { # Attributes that are not used in forward function will # be removed when constructing GraphModule, this is a list of attributes # to preserve as an attribute of the GraphModule even when they are # not used in the code, these attributes will also persist through deepcopy "preserved_attributes": ["preserved_attr"], } Example:: from torch.ao.quantization import fuse_fx m = Model().eval() m = fuse_fx(m) z$quantization_api.quantize_fx.fuse_fxrcF) r$_C_log_api_usage_oncerfxZsymbolic_tracesetrdrfrgr/)rr-r.r+rcrhr!r!r"fuse_fx,s   rp)rrZr[r\r.rcCstjdt||d|||S)a] Prepare a model for post training static quantization Args: * `model`: torch.nn.Module model, must be in eval mode * `qconfig_dict`: qconfig_dict is a dictionary with the following configurations:: qconfig_dict = { # optional, global config "": qconfig?, # optional, used for module and function types # could also be split into module_types and function_types if we prefer "object_type": [ (torch.nn.Conv2d, qconfig?), (torch.nn.functional.add, qconfig?), ..., ], # optional, used for module names "module_name": [ ("foo.bar", qconfig?) ..., ], # optional, matched in order, first match takes precedence "module_name_regex": [ ("foo.*bar.*conv[0-9]+", qconfig?) ..., ], # optional, used for matching object type invocations in a submodule by # order # TODO(future PR): potentially support multiple indices ('0,1') and/or # ranges ('0:3'). "module_name_object_type_order": [ # fully_qualified_name, object_type, index, qconfig ("foo.bar", torch.nn.functional.linear, 0, qconfig?), ], # priority (in increasing order): # global, object_type, module_name_regex, module_name, # module_name_object_type_order # qconfig == None means fusion and quantization should be skipped for anything # matching the rule } * `prepare_custom_config_dict`: customization configuration dictionary for quantization tool:: prepare_custom_config_dict = { # optional: specify the path for standalone modules # These modules are symbolically traced and quantized as one unit "standalone_module_name": [ # module_name, qconfig_dict, prepare_custom_config_dict ("submodule.standalone", None, # qconfig_dict for the prepare function called in the submodule, # None means use qconfig from parent qconfig_dict {"input_quantized_idxs": [], "output_quantized_idxs": []}), # prepare_custom_config_dict {} # backend_config_dict, TODO: point to README doc when it's ready ], "standalone_module_class": [ # module_class, qconfig_dict, prepare_custom_config_dict (StandaloneModule, None, # qconfig_dict for the prepare function called in the submodule, # None means use qconfig from parent qconfig_dict {"input_quantized_idxs": [0], "output_quantized_idxs": [0]}, # prepare_custom_config_dict {}) # backend_config_dict, TODO: point to README doc when it's ready ], # user will manually define the corresponding observed # module class which has a from_float class method that converts # float custom module to observed custom module # (only needed for static quantization) "float_to_observed_custom_module_class": { "static": { CustomModule: ObservedCustomModule } }, # the qualified names for the submodule that are not symbolically traceable "non_traceable_module_name": [ "non_traceable_module" ], # the module classes that are not symbolically traceable # we'll also put dynamic/weight_only custom module here "non_traceable_module_class": [ NonTraceableModule ], # By default, inputs and outputs of the graph are assumed to be in # fp32. Providing `input_quantized_idxs` will set the inputs with the # corresponding indices to be quantized. Providing # `output_quantized_idxs` will set the outputs with the corresponding # indices to be quantized. "input_quantized_idxs": [0], "output_quantized_idxs": [0], # Attributes that are not used in forward function will # be removed when constructing GraphModule, this is a list of attributes # to preserve as an attribute of the GraphModule even when they are # not used in the code, these attributes will also persist through deepcopy "preserved_attributes": ["preserved_attr"], } * `equalization_qconfig_dict`: equalization_qconfig_dict is a dictionary with a similar structure as qconfig_dict except it will contain configurations specific to equalization techniques such as input-weight equalization. * `backend_config_dict`: a dictionary that specifies how operators are quantized in a backend, this includes how the operaetors are observed, supported fusion patterns, how quantize/dequantize ops are inserted, supported dtypes etc. The structure of the dictionary is still WIP and will change in the future, please don't use right now. Return: A GraphModule with observer (configured by qconfig_dict), ready for calibration Example:: import torch from torch.ao.quantization import get_default_qconfig from torch.ao.quantization import prepare_fx float_model.eval() qconfig = get_default_qconfig('fbgemm') def calibrate(model, data_loader): model.eval() with torch.no_grad(): for image, target in data_loader: model(image) qconfig_dict = {"": qconfig} prepared_model = prepare_fx(float_model, qconfig_dict) # Run calibration calibrate(prepared_model, sample_inference_data) z'quantization_api.quantize_fx.prepare_fxFr$rlrmrj)rrZr[r\r.r!r!r" prepare_fxTs rr)rrZr[r.rcCstjdt||d||dS)a  Prepare a model for quantization aware training Args: * `model`: torch.nn.Module model, must be in train mode * `qconfig_dict`: see :func:`~torch.ao.quantization.prepare_fx` * `prepare_custom_config_dict`: see :func:`~torch.ao.quantization.prepare_fx` * `backend_config_dict`: see :func:`~torch.ao.quantization.prepare_fx` Return: A GraphModule with fake quant modules (configured by qconfig_dict), ready for quantization aware training Example:: import torch from torch.ao.quantization import get_default_qat_qconfig from torch.ao.quantization import prepare_fx qconfig = get_default_qat_qconfig('fbgemm') def train_loop(model, train_data): model.train() for image, target in data_loader: ... float_model.train() qconfig_dict = {"": qconfig} prepared_model = prepare_fx(float_model, qconfig_dict) # Run calibration train_loop(prepared_model, train_loop) z+quantization_api.quantize_fx.prepare_qat_fxT)r.rq)rrZr[r.r!r!r"prepare_qat_fxs% rsT)r+ is_referenceconvert_custom_config_dictr]_remove_qconfigrZr.rc Cs^|dkr i}t|t|t|||||||d}|dg}|D]} t|| t|| qB|S)ze `is_standalone_module`: see docs in :func:`~torch.ao.quantization.prepare_standalone_module_fx` N)Z_remove_qconfig_flagZconvert_qconfig_dictr.rc)r#rrrdrfrg) r+rtrur]rvrZr.r&rcrhr!r!r" _convert_fx"s"  rw)r+rtrurvrZr.rcCs tjdt||||||dS)a Convert a calibrated or trained model to a quantized model Args: * `graph_module`: A prepared and calibrated/trained model (GraphModule) * `is_reference`: flag for whether to produce a reference quantized model, which will be a common interface between pytorch quantization with other backends like accelerators * `convert_custom_config_dict`: dictionary for custom configurations for convert function:: convert_custom_config_dict = { # user will manually define the corresponding quantized # module class which has a from_observed class method that converts # observed custom module to quantized custom module "observed_to_quantized_custom_module_class": { "static": { ObservedCustomModule: QuantizedCustomModule }, "dynamic": { ObservedCustomModule: QuantizedCustomModule }, "weight_only": { ObservedCustomModule: QuantizedCustomModule } }, # Attributes that are not used in forward function will # be removed when constructing GraphModule, this is a list of attributes # to preserve as an attribute of the GraphModule even when they are # not used in the code "preserved_attributes": ["preserved_attr"], } * `_remove_qconfig`: Option to remove the qconfig attributes in the model after convert. * `qconfig_dict`: qconfig_dict with either same keys as what is passed to the qconfig_dict in `prepare_fx` API, with same values or `None`, or additional keys with values set to `None` For each entry whose value is set to None, we skip quantizing that entry in the model:: qconfig_dict = { # used for object_type, skip quantizing torch.nn.functional.add "object_type": [ (torch.nn.functional.add, None), (torch.nn.functional.linear, qconfig_from_prepare) ..., ], # sed for module names, skip quantizing "foo.bar" "module_name": [ ("foo.bar", None) ..., ], } * `backend_config_dict`: A configuration for the backend which describes how operators should be quantized in the backend, this includes quantization mode support (static/dynamic/weight_only), dtype support (quint8/qint8 etc.), observer placement for each operators and fused operators. Detailed documentation can be found in torch/ao/quantization/backend_config/README.md Return: A quantized model (GraphModule) Example:: # prepared_model: the model after prepare_fx/prepare_qat_fx and calibration/training quantized_model = convert_fx(prepared_model) z'quantization_api.quantize_fx.convert_fx)rvrZr.)r$rlrmrw)r+rtrurvrZr.r!r!r" convert_fxCsN rx)r+rtrurcCst|||ddS)a| [Internal use only] Convert a model produced by :func:`~torch.ao.quantization.prepare_standalone_module_fx` and convert it to a quantized model Returns a quantized standalone module, whether input/output is quantized is specified by prepare_custom_config_dict, with input_quantized_idxs, output_quantized_idxs, please see docs for prepare_fx for details T)r])rw)r+rtrur!r!r"_convert_standalone_module_fxs ry)NN)NNNF)NN)NN)NNN)NN)NFTNN)FNTNN)FN)7typingrrrrrrrr$Ztorch.fxr Ztorch.fx._symbolic_tracer Z torch.fx.noder r r Ztorch.nn.intrinsicrrnrrZ fx.convertrZbackend_configrZfx.graph_modulerZfx.qconfig_utilsrrrrZfx.utilsrrr%rGr#r(rYrr/objectr0r>rHrjrkrprrrsrwrxryr!r!r!r"s$            = X ( + # 2   #   [