U d##@sddlmZmZddlmZmZmZmZmZddl m Z Gddde Z Gddde Z Gdd d e ZGd d d e ZGd d d e ZGdddeZGdddZeegefZGdddZeZdS))ABCabstractmethod)AnyCallableDictOptionalTuple)Storec@seZdZdZdS)RendezvousErrorz/Represents the base type for rendezvous errors.N__name__ __module__ __qualname____doc__rrL/tmp/pip-unpacked-wheel-ua33x9lu/torch/distributed/elastic/rendezvous/api.pyr sr c@seZdZdZdS)RendezvousClosedErrorz#Raised when a rendezvous is closed.Nr rrrrrsrc@seZdZdZdS)RendezvousTimeoutErrorz2Raised when a rendezvous did not complete on time.Nr rrrrrsrc@seZdZdZdS)RendezvousConnectionErrorz>Raised when the connection to a rendezvous backend has failed.Nr rrrrrsrc@seZdZdZdS)RendezvousStateErrorz1Raised when the state of a rendezvous is corrupt.Nr rrrrrsrc@seZdZdZeedddZeeee e fdddZ ee dddZ ed d Z ee dd d Zeedd dZe dddZdS)RendezvousHandlerzMain rendezvous interface. Note: Distributed Torch users normally **do not** need to implement their own ``RendezvousHandler``. An implementation based on C10d Store is already provided, and is recommended for most users. returncCsdS)z+Returns the name of the rendezvous backend.Nrselfrrr get_backend*szRendezvousHandler.get_backendcCsdS)aMain entry-point into the rendezvous barrier. Blocks until the rendezvous is complete and the current process is included in the formed worker group, or a timeout occurs, or the rendezvous was marked closed. Returns: A tuple of :py:class:`torch.distributed.Store`, ``rank``, and ``world size``. Raises: RendezvousClosedError: The rendezvous is closed. RendezvousConnectionError: The connection to the rendezvous backend has failed. RendezvousStateError: The rendezvous state is corrupt. RendezvousTimeoutError: The rendezvous did not complete on time. Nrrrrrnext_rendezvous.sz!RendezvousHandler.next_rendezvouscCsdS)aChecks whether the rendezvous has been closed. A closed rendezvous means all future attempts to re-rendezvous within same job will fail. ``is_closed()`` and :py:meth:`set_closed` have semantics of eventual propagation and should not be used for synchronization. The intention is that if at least one node decides the job is finished, it will close the rendezvous, and other nodes will soon observe this and stop running as well. Nrrrrr is_closedGszRendezvousHandler.is_closedcCsdS)zMarks the rendezvous as closed.Nrrrrr set_closedUszRendezvousHandler.set_closedcCsdS)aXReturns the number of nodes who arrived late at the rendezvous barrier, hence were not included in the current worker group. Callers should periodically call this method to check whether new nodes are waiting to join the job and if so admit them by calling :py:meth:`next_rendezvous()` (re-rendezvous). Nrrrrrnum_nodes_waitingYsz#RendezvousHandler.num_nodes_waitingcCsdS)aReturns the run id of the rendezvous. The run id is a user-defined id that uniquely identifies an instance of a distributed application. It typically maps to a job id and is used to allow nodes to join the correct distributed application. Nrrrrr get_run_idcszRendezvousHandler.get_run_idcCsdS)a Closes all resources that were open for the rendezvous. Example:: rdzv_handler = ... try: store, rank, world_size = rdzv_handler.next_rendezvous() finally: rdzv_handler.shutdown() NrrrrrshutdownlszRendezvousHandler.shutdownN)r r rrrstrrrr intrboolrrrr r!rrrrr!s    rc@sreZdZdZeeeeedddZd eeedddZdee e e e dd d Z dee ee edd d Z dS)RendezvousParametersa>Holds the parameters to construct a :py:class:`RendezvousHandler`. Args: backend: The name of the backend to use to handle the rendezvous. endpoint: The endpoint of the rendezvous, usually in form [:]. run_id: The id of the rendezvous. min_nodes: The minimum number of nodes to admit to the rendezvous. max_nodes: The maximum number of nodes to admit to the rendezvous. **kwargs: Additional parameters for the specified backend. )backendendpointrun_id min_nodes max_nodescKsj|s td|dkr$td|d||krBtd|d|d||_||_||_||_||_||_dS)N7The rendezvous backend name must be a non-empty string.z(The minimum number of rendezvous nodes (z) must be greater than zero.z(The maximum number of rendezvous nodes (zK) must be greater than or equal to the minimum number of rendezvous nodes (z).) ValueErrorr&r'r(r)r*config)rr&r'r(r)r*kwargsrrr__init__s  zRendezvousParameters.__init__N)keydefaultrcCs|j||S)zBReturns the value for ``key`` if ``key`` exists, else ``default``.)r.get)rr1r2rrrr3szRendezvousParameters.getcCs|||}|dkst|tr"|St|trF|dkr8dS|dkrpdSn*t|trp|dkr`dS|dkrpdStd|d dS) z,Returns the value for ``key`` as a ``bool``.Nr,TrF)1truetyesy)0falsefnon%The rendezvous configuration option 'z+' does not represent a valid boolean value.)r3 isinstancer$r#r"lowerr-rr1r2valuerrr get_as_bools       z RendezvousParameters.get_as_boolcCsL|||}|dkr|Sz t|WStk rFtd|dYnXdS)z,Returns the value for ``key`` as an ``int``.Nr>z+' does not represent a valid integer value.)r3r#r-rArrr get_as_ints   zRendezvousParameters.get_as_int)N)N)N) r r rrr"r#r0rr3rr$rCrDrrrrr%ys r%c@sReZdZUdZeeefed<ddddZeedddd Z e e d d d Z dS) RendezvousHandlerRegistryz@Represents a registry of :py:class:`RendezvousHandler` backends. _registryNrcCs i|_dS)N)rFrrrrr0sz"RendezvousHandlerRegistry.__init__)r&creatorrcCsn|s tdz|j|}Wntk r2d}YnX|dk r`||kr`td|d|d|d||j|<dS)zRegisters a new rendezvous backend. Args: backend: The name of the backend. creater: The callback to invoke to construct the :py:class:`RendezvousHandler`. r+NThe rendezvous backend 'z' cannot be registered with 'z$' as it is already registered with ''.)r-rFKeyError)rr&rGZcurrent_creatorrrrregisters  z"RendezvousHandlerRegistry.register)paramsrc Csxz|j|j}Wn0tk r@td|jd|jjdYnX||}||jkrttd|d|jd|S)z,Creates a new :py:class:`RendezvousHandler`.rHz-' is not registered. Did you forget to call `z`?z(' does not match the requested backend 'rI)rFr&rJr-rKr r RuntimeError)rrLrGhandlerrrrcreate_handlers z(RendezvousHandlerRegistry.create_handler) r r rrrr"RendezvousHandlerCreator__annotations__r0rKr%rrOrrrrrEs rEN)abcrrtypingrrrrrZtorch.distributedr Exceptionr rrrrrr%rPrEZrendezvous_handler_registryrrrrs XT;