U *-e/@sddlmZddlZddlmZddlmZddlmZddl m Z ddl m Z m Z ddlmZddlmZdeje e d d d Zdejjee d d dZdaeddZddZejje ejjdddZejjejjdddZdejedddZdS))contextmanagerN)distributed_c10d) ShardedTensor) ShardingSpecChunkShardingSpec) ShardingPlan)Sharder)tensor sharding_specreturnc Cs|std|dk r|nt}t|}t|}dg|}tj|||f|dt|D]h\}} || dkrtd|d|d| dd||| dkr^td |d|d | dd|q^|j |||d } | S) ab Given a :class:`torch.Tensor`, it shards that tensor according to the provided ``sharding_spec``. ``src_rank`` denotes the source rank which would be used as the ground truth of the data which would be scattered as shards across the rest of the ranks. Args: tensor (:class:`torch.Tensor`): Tensor needs to be sharded. sharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification describing how to shard the Tensor. Keyword args: src_rank (int, optional): The source rank which is used as the ground truth of the data for the parameter that would be sharded and scattered across the rest of the ranks. Default: 0. process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. Returns: A :class:`ShardedTensor` sharded from the given tensor. .. warning:: Only :class:`torch.distributed._shard.sharding_spec.ChunkShardingSpec` is currently supported as the ``sharding_spec``. z'input tensor is not a contiguous TensorN)grouprz src_rank=z on rank: z does not match with src_rank=rzsharding_spec=z# does not match with sharding_spec=src_rank process_group) is_contiguous ValueErrorr_get_default_groupdistZget_world_sizeZget_rankZall_gather_object enumerateshard) r r rrZpgZ world_sizeZ current_rankZ gathered_listidxentrystr]/var/www/html/Darija-Ai-Train/env/lib/python3.8/site-packages/torch/distributed/_shard/api.py _shard_tensors$     r)module param_namer cCst||s"t|d|dt||}t|tjs^tdt|j d|dt|j | svtd|dt ||||}| |t |dS) a. Given a :class:`torch.nn.Module`, a ``param_name`` for a parameter in that module, it shards that parameter according to the provided ``sharding_spec``. ``src_rank`` denotes the source rank which would be used as the ground truth of the data which would be scattered as shards across the rest of the ranks. This method replaces ``module.param_name`` with a :class:`torch.distributed._sharded_tensor.ShardedTensor` Args: module (:class:`torch.nn.Module`): Module whose parameter needs to be sharded. param_name (str): Name of the parameter of ``module`` that needs to be sharded. sharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification describing how to shard the Tensor. Keyword args: src_rank (int, optional): The source rank which is used as the ground truth of the data for the parameter that would be sharded and scattered across the rest of the ranks. Default: 0. process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. .. warning:: Only :class:`torch.distributed._shard.sharding_spec.ChunkShardingSpec` is currently supported as the ``sharding_spec``. z has no attribute ``z Expected .z to be a Tensor, but found zparam: z is not a contiguous TensorN)hasattrAttributeErrorZ _get_namegetattr isinstancetorchTensorrtype__name__rrZregister_parameternn Parameter)rrr rrr rrrrshard_parameterHs#   &r+ccs*tdk rtd|az |VW5daXdS)zV Context manager to set the process group with which to load a ShardedTensor. NzNProcessGroup already set by previous "load_with_process_group" context manager)_CURRENT_PROCESS_GROUP RuntimeError)rrrrload_with_process_group}s r.cCstdkrtStSdS)z Retrieves the current process group set by ``load_with_process_group``. If not set, it just returns the default group. N)r,rrrrrr_get_current_process_groupsr/)rresharding_specr csfdd}|||S)a Hook a module with output resharding in the forward pass according to the given ``resharding_spec``. Args: module (:class:`torch.nn.Module`): Module whose output needs to be resharded. resharding_spec (:class:`torch.distributed._shard.sharding_spec.ShardingSpec`): The specification describing how the output of the module will be resharded. Returns: A :class:`torch.nn.Module` object with reshard API hooked. cst|tr|S|S)N)r$rZreshard)_module_inputoutputr0rr hook_funcs  z"_reshard_output..hook_funcZregister_forward_hook)rr0r5rr4r_reshard_outputs  r7)rr cCsdd}|||S)a Hook a module with local shards collection in the forward pass. This API is typically used to convert a sharded representation back to data parallel representation. In particular, it returns the local tensor for this Shard. If the size along the sharding dimension for the local tensor is 1, this dimension is removed from the final result. For example a [4, 16] ShardedTensor across 4 ranks is typically a local Tensor of size [16] across each rank and not [1, 16] across each rank. Args: module (:class:`torch.nn.Module`): Module whose output is ShardedTensor and the local tensor value needs to be returned. Returns: A :class:`torch.nn.Module` object with collection API hooked. cSsHt|trD|}|j}t|tr@||jdkr@||jj}|SdS)Nr)r$r local_tensorZ_sharding_specrsizedimZsqueeze)r1r2r3r8r rrrr5s  z'_collect_local_shard..hook_funcr6)rr5rrr_collect_local_shards r;)rplancCsg}|jD]\}}t|tr||q|jD]\}}t|tr|d\}}} |D]$} || r\td|d| dq\| |} t | | |||dq6t|tr|d\} }} |dkrt d| |} | | }| | }||_ q6td|d q6|jd k rV|jD]<\}}t|trB| |} t| |ntd |d q|jd k r|jD]}| |} t| qhd S) aa Shards a given module according to the provided sharding `plan`. This method first shards all the parameters according to the given sharding `plan`. Then if `output_plan` and `return_local_tensor` are specified in the sharding `plan`, it will tag the output of modules according `output_plan`, convert the module's output back to data parallel according to `return_local_tensor`. Needs to be called on all ranks in an SPMD fashion. Args: module (:class:`torch.nn.Module`): The module to apply sharding to plan (:class:`torch.distributed._shard.sharding_plan.ShardingPlan`): The ShardingPlan which specified param name to ShardingSpec to apply to each parameter. Keyword args: src_rank (int, optional): The source rank which is used as the ground truth of the data for the module that would be sharded and scattered across the rest of the ranks. Default: 0. process_group (ProcessGroup, optional): The process group to work on. If None, the default process group will be used. r z7ShardingPlan is in-valid, trying to shard a parameter: z1, but there's already a Sharder entry for module z`, parameter sharding should not conflict with the submodule tree that a Sharder is working with!rz1Module path must not be empty for custom sharder!z:Only `ShardingSpec` and `Sharder` are supported to shard ''Nz5Only `ShardingSpec` is supported as output_plan for ')r<itemsr$r appendr rpartition startswithr-Z get_submoduler+KeyErrorrmod_name TypeErrorZ output_planr7Zreturn_local_tensorr;)rr<rrZ sharder_pathsnamespec module_path_rZ sharder_pathmodZparent_mod_pathrDZ parent_modZ sharded_modZ output_specrrr shard_modulesJ                rK)rN)rN)rN) contextlibrr%Ztorch.distributed distributedrZtorch.nnr)rZ'torch.distributed._shard.sharded_tensorrr rrZ sharding_planrZsharderr r&rModulestrr+r,r.r/r7r;rKrrrrsJ        : 3   #