U d@sddlZddlZddlmZddlmZmZeejeejfZdee e e ejdddZ dee e e ejdd d Z ee dd d d Z dS)N)inf)UnionIterable@F) parametersmax_norm norm_typeerror_if_nonfinitereturnc st|tjr|g}dd|D}t|}tt|dkrFtdS|djjtkrfdd|D}t|dkr|dnt t |}n"t t fdd|D}|rt | |rtdd ||d }tj|d d }|D]}|j||jjq|S) aAClips gradient norm of an iterable of parameters. The norm is computed over all gradients together, as if they were concatenated into a single vector. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized max_norm (float or int): max norm of the gradients norm_type (float or int): type of the used p-norm. Can be ``'inf'`` for infinity norm. error_if_nonfinite (bool): if True, an error is thrown if the total norm of the gradients from :attr:`parameters` is ``nan``, ``inf``, or ``-inf``. Default: False (will switch to True in the future) Returns: Total norm of the parameter gradients (viewed as a single vector). cSsg|]}|jdk r|qSNgrad.0pr s z#clip_grad_norm_..rgcs$g|]}|jqSr)r detachabsmaxtor)devicerrr'scs$g|]}t|jqSr)torchnormr rrrrrrrr*szThe total norm of order z for gradients from `parameters` is non-finite, so it cannot be clipped. To disable this error and scale the gradients by the non-finite norm anyway, set `error_if_nonfinite=False`gư>g?)r) isinstancerTensorfloatlenZtensorr rrrstackr logical_orisnanisinf RuntimeErrorclamprZmul_r) rrrr ZnormsZ total_normZ clip_coefZclip_coef_clampedrrrrclip_grad_norm_ s*    &"  r'cCstjdddt||||S)zClips gradient norm of an iterable of parameters. .. warning:: This method is now deprecated in favor of :func:`torch.nn.utils.clip_grad_norm_`. z[torch.nn.utils.clip_grad_norm is now deprecated in favor of torch.nn.utils.clip_grad_norm_.) stacklevel)warningswarnr')rrrr rrrclip_grad_norm;s r,)r clip_valuer cCsFt|tjr|g}t|}tdd|D]}|jjj| |dq(dS)aClips gradient of an iterable of parameters at specified value. Gradients are modified in-place. Args: parameters (Iterable[Tensor] or Tensor): an iterable of Tensors or a single Tensor that will have gradients normalized clip_value (float or int): maximum allowed value of the gradients. The gradients are clipped in the range :math:`\left[\text{-clip\_value}, \text{clip\_value}\right]` cSs |jdk Sr r )rrrrXz"clip_grad_value_..)minrN)rrrrfilterr dataZclamp_)rr-rrrrclip_grad_value_Is r3)rF)rF)r*rZ torch._sixrtypingrrrZ_tensor_or_tensorsrboolr'r,r3rrrrs.  3