U d@sZddlmZmZmZmZddlZddlmmZ ddlm Z ej j Gddde ZdS))ListDictOptionalTupleN)Tensorc @sbeZdZdeeeeeefeeeeeed ddZee ed d d Z ee ed d dZ dS)_FunctionalAdamMbP?g?g+?:0yE>F) paramslrbetaseps weight_decayamsgradmaximizeforeach_allow_empty_param_listc Csd|kstd|d|ks,td|d|dkrDdksXntd|dd|dkrpdksntd|dd|kstd ||||d|d|d |_||_||_||_tjt tj t t tj ffi|_ t |dkr| std d |i|_dS) Nr zInvalid learning rate: {}zInvalid epsilon value: {}rg?z%Invalid beta parameter at index 0: {}z%Invalid beta parameter at index 1: {}zInvalid weight_decay value: {})r rbeta1beta2rz%optimizer got an empty parameter listr ) ValueErrorformatdefaultsrrrtorchjitZannotaterrstrstatelen param_group) selfr r rrrrrrrr"K/tmp/pip-unpacked-wheel-ua33x9lu/torch/distributed/optim/functional_adam.py__init__s. $z_FunctionalAdam.__init__)paramgradc CsR|g}g}g}g}g}g}g} |dk r:||||||jkri|j|<|j|} td| d<tj|tjd| d<tj|tjd| d<|jrtj|tjd| d<|j|} || d|| d|jr|| d| | dtPtj |||||| |j|j |j d|j d |j d |j d |j d |j d W5QRXdS)zo Similar to step, but operates on a single parameter and optionally a gradient tensor. Nr stepZ memory_formatexp_avg exp_avg_sqmax_exp_avg_sqrrr rrrrrrr rrr) appendrrtensor zeros_likepreserve_formatrno_gradFadamrrr) r!r%r&r params_with_gradgradsexp_avgs exp_avg_sqsmax_exp_avg_sqs state_stepsrr"r"r# step_param<sP       z_FunctionalAdam.step_param) gradientsc Cs|jd}g}g}g}g}g}g}t|t|krXtddt|ddt|t|jd|D]\} } | dk rh|| || | |jkri|j| <|j| } td| d<tj| tj d| d <tj| tj d| d <|j rtj| tj d| d <|j| } || d || d |j r6|| d || dqht Pt j |||||||j |j|jd |jd |jd|jd|jd|jdW5QRXdS)Nr zEthe gradients passed in does not equal to the size of the parameters!zParams length: z. zGradients length: r r'r(r)r*r+rrr rrr,)r rrzipr-rrr.r/r0rr1r2r3rrr) r!r;r r4r5r6r7r8r9r%Zgradientrr"r"r#r'lsb         z_FunctionalAdam.stepN)rr r r FFFF) __name__ __module__ __qualname__rrfloatrboolr$rr:r'r"r"r"r#rs*  *0r)typingrrrrrZtorch.optim._functionalZoptimZ _functionalr2rrscriptobjectrr"r"r"r#s