U ‰dß ã @s ddlmZddlmZmZmZmZmZmZm Z ddl Z ddl m Z ddl mZdd„Zdd „Zee e jeeeeeefed œd d „Ze jjeed œdd„ZdS)é)Ú OrderedDict)ÚAnyÚCallableÚDictÚListÚSetÚTupleÚUnionN)Ú _BatchNorm)ÚPackedSequencecCstdd„| ¡DƒƒS)Ncss|]}t|tƒVqdS©N)Ú isinstancer )Ú.0Úmod©rúA/tmp/pip-unpacked-wheel-ua33x9lu/torch/distributed/fsdp/_utils.pyÚ sz&_contains_batchnorm..)ÚanyÚmodules)ÚmodulerrrÚ_contains_batchnorm sÿrcCs&| ¡D]}t|tƒrddi|_qdS)NZmixed_precision)rr r Z_wrap_overrides)rrrrrÚ#_override_batchnorm_mixed_precisions  r)ÚfnÚ containerÚreturncs2ttjttttttft dœ‡‡fdd„ ‰ˆ|ƒS)zFRecursively apply to all tensor in different kinds of container types.)Úxrcs°t |¡rˆ|ƒSt|tƒrF| ¡}| ¡D]\}}ˆ|ƒ||<q,|St|tƒr^ˆ|jƒ|St|tƒr~‡fdd„| ¡DƒSt|t t t fƒr¨t |ƒ‡fdd„|DƒƒS|SdS)Ncsi|]\}}|ˆ|ƒ“qSrr)rÚkeyÚvalue©ÚapplyrrÚ &sz4_apply_to_tensors..apply..c3s|]}ˆ|ƒVqdSr r)rÚelrrrr(sz3_apply_to_tensors..apply..) ÚtorchZ is_tensorr rÚ __class__Úitemsr ÚdataÚdictÚlistÚtupleÚsetÚtype)rZodrr©rrrrrs     z _apply_to_tensors..apply) r r"ÚTensorrrrrrr r)rrrr+rÚ_apply_to_tensorss*r-)Ú root_moduleÚ module_fnÚ return_fncs6tjjtdœ‡‡fdd„ ‰ˆ|df|ž|Ž|||ŽS)ae Performs a pre-order traversal of the modules in the hierarchy rooted at ``root_module``, applying ``module_fn`` at each module and finally returning a value using ``return_fn``. The traversal constructs the full module prefix name (e.g. "module.submodule." just like in model state dict) and makes that available to ``module_fn``. )rÚprefixcsNˆ||f|ž|Ž| ¡D].\}}|dk r||d}ˆ||f|ž|ŽqdS)NÚ.)Znamed_children)rr1ÚargsÚkwargsZsubmodule_nameÚ submoduleÚ new_prefix©Úfr/rrr8<s  z_apply_to_modules..fÚ)r"ÚnnÚModuleÚstr)r.r/r0r3r4rr7rÚ_apply_to_modules.sr=)Ú collectionsrÚtypingrrrrrrr r"Ztorch.nn.modules.batchnormr Ztorch.nn.utils.rnnr rrr,r-r:r;r=rrrrÚs $  þ ý