U d'@s^ddlZddlZddlmZmZddlmZmZmZm Z m Z m Z m Z ddl mmmmZddlmZmZddlmZddlmZddlmZmZddlmZdd lm Z dd l!m"Z"dd l#m$Z$e$Z%eGd d d Z&GdddZ'e ee(dfe ee(dddZ)e e e e(e e*fdddZ+e&e ee(dfe eee*efdddZ,dS)N) dataclassfield)AnyCallableDictListOptionalTupleUnion)eventsmetrics) WorkerSpec)LocalElasticAgent)SignalExceptionStd)ChildFailedError)RendezvousParameters)parse_rendezvous_endpoint) get_loggerc@seZdZUdZeed<eed<eed<dZeed<dZeed<dZ eed <d Z eed <e e d Z eeefed <dZeed<dZeed<dZeed<dZeed<dZeeed<ejZeeeeeffed<ejZeeeeeffed<e e d Zeeefed<ddZdS) LaunchConfiga Creates a rendezvous config. Args: min_nodes: Minimum amount of nodes that the user function will be launched on. Elastic agent ensures that the user function start only when the min_nodes amount enters the rendezvous. max_nodes: Maximum amount of nodes that the user function will be launched on. nproc_per_node: On each node the elastic agent will launch this amount of workers that will execute user defined function. rdzv_backend: rdzv_backend to use in the rendezvous (zeus-adapter, etcd). rdzv_endpoint: The endpoint of the rdzv sync. storage. rdzv_configs: Key, value pair that specifies rendezvous specific configuration. rdzv_timeout: Legacy argument that specifies timeout for the rendezvous. It is going to be removed in future versions, see the note below. The default timeout is 900 seconds. run_id: The unique run id of the job (if not passed a unique one will be deduced from run environment - flow workflow id in flow - or auto generated). role: User defined role of the worker (defaults to "trainer"). max_restarts: The maximum amount of restarts that elastic agent will conduct on workers before failure. monitor_interval: The interval in seconds that is used by the elastic_agent as a period of monitoring workers. start_method: The method is used by the elastic agent to start the workers (spawn, fork, forkserver). log_dir: base log directory where log files are written. If not set, one is created in a tmp dir but NOT removed on exit. redirects: configuration to redirect stdout/stderr to log files. Pass a single ``Std`` enum to redirect all workers, or a mapping keyed by local_rank to selectively redirect. tee: configuration to "tee" stdout/stderr to console + log file. metrics_cfg: configuration to initialize metrics. ..note: `rdzv_timeout` is a legacy argument that will be removed in future. Set the timeout via `rdzv_configs['timeout']` min_nodes max_nodesnproc_per_noderun_idZ default_rolerole rdzv_endpointZetcd rdzv_backend)default_factory rdzv_configs rdzv_timeout max_restartsmonitor_intervalspawn start_methodNlog_dir redirectstee metrics_cfgcCs4d}|jdkr|j|jd<nd|jkr0||jd<dS)Nir timeout)r!r)selfZdefault_timeoutr.B/tmp/pip-unpacked-wheel-ua33x9lu/torch/distributed/launcher/api.py __post_init__Ws   zLaunchConfig.__post_init__)__name__ __module__ __qualname____doc__int__annotations__rstrrrrrdictrrrr!r#r%floatr'r(rrNONEr)r r*r+r0r.r.r.r/rs$ )        rc@s2eZdZdZeeeedfdddZddZ dS)elastic_launcha Launches an torchelastic agent on the container that invoked the entrypoint. 1. Pass the ``entrypoint`` arguments as non ``kwargs`` (e.g. no named parameters)/ ``entrypoint`` can be a function or a command. 2. The return value is a map of each worker's output mapped by their respective global rank. Usage :: def worker_fn(foo): # ... def main(): # entrypoint is a function. outputs = elastic_launch(LaunchConfig, worker_fn)(foo) # return rank 0's output return outputs[0] # entrypoint is a command and ``script.py`` is the python module. ouptuts = elestic_launch(LaunchConfig, "script.py")(args) ouptuts = elestic_launch(LaunchConfig, "python")("script.py") N)config entrypointcCs||_||_dSN)_config _entrypoint)r-r<r=r.r.r/__init__zszelastic_launch.__init__cGst|j|jt|Sr>) launch_agentr?r@list)r-argsr.r.r/__call__szelastic_launch.__call__) r1r2r3r4rr rr7rArEr.r.r.r/r;_s  r;)r=rDreturncCsFt|tr|jSt|tr>|tjkr8tdd|DdS|SndSdS)aRetrive entrypoint name with the rule: 1. If entrypoint is a function, use ``entrypont.__qualname__``. 2. If entrypoint is a string, check its value: 2.1 if entrypoint equals to ``sys.executable`` (like "python"), use the first element from ``args`` which does not start with hifen letter (for example, "-u" will be skipped). 2.2 otherwise, use ``entrypoint`` value. 3. Otherwise, return empty string. css|]}|ddkr|VqdS)r-Nr.).0argr.r.r/ s z'_get_entrypoint_name..rN) isinstancerr1r7sys executablenext)r=rDr.r.r/_get_entrypoint_names   rO)rdzv_parametersrFcCsX|jdkrdS|j}|}|s(tdt|dd\}}|dkrPtd|d||fS)NZstatic)NNzKEndpoint is missing in endpoint. Try to add --master_addr and --master_portr ) default_portzport is missing in endpoint: z. Try to specify --master_port)backendendpointstrip ValueErrorr)rPrS master_addr master_portr.r.r/_get_addr_and_ports  rX)r<r=rDrFc Cs|js*ttj}td|||_t||}td|d|j d|j d|j d|jd|j d|j d |jd |jd |jd |jd |jdtf|j |j |j|j |j d|j}t|\}}t|j|j |t|t||j|j|j|j||d }t||j|jd} d} zzLt !t "|j| #} t$%| &| 'r^t(|| j)d| j*WWjSt(k rYnNt+k rd} t$%| ,Yn&t-k rt$%| ,YnXW5| r|jXdS)Nz1config has no run_id, generated a random run_id: zDStarting elastic_operator with launch configs: entrypoint : z min_nodes : z max_nodes : z nproc_per_node : z run_id : z rdzv_backend : z rdzv_endpoint : z rdzv_configs : z max_restarts : z monitor_interval : z log_dir : z metrics_cfg :  )rRrSrrr) rZlocal_world_sizer=rD rdzv_handlerr#r%r)r*rVrW)specr'r(T)namefailuresF).rr7uuidZuuid4r5loggerwarningrOinforrrrrrr#r%r(r+rrXr rtuple rdzv_registryZget_rendezvous_handlerr)r*rr'rZshutdownr Zinitialize_metricsZ MetricsConfigrunr recordZget_event_succeededZ is_failedrr]Z return_valuesrZget_event_failed Exception) r<r=rDrZentrypoint_namerPrVrWr[ZagentZ shutdown_rdzvresultr.r.r/rBst b    rB)-rLr^Z dataclassesrrtypingrrrrrr r Z-torch.distributed.elastic.rendezvous.registryZ distributedZelasticZ rendezvousregistryrcZtorch.distributed.elasticr r Z*torch.distributed.elastic.agent.server.apir Z:torch.distributed.elastic.agent.server.local_elastic_agentrZ)torch.distributed.elastic.multiprocessingrrZ0torch.distributed.elastic.multiprocessing.errorsrZ$torch.distributed.elastic.rendezvousrZ*torch.distributed.elastic.rendezvous.utilsrZ'torch.distributed.elastic.utils.loggingrr_rr;r7rOr5rXrBr.r.r.r/s8$      C(