U (dZ @s,ddlmZddlmZmZmZmZddlZddlm m Z ddlm Z m Z ddl mZmZddlmZddlmZmZdd lmZd d lmZmZd d lmZd d lmZdddddddgZddZej !dGddde j"Z#d/e e e e ee$e$ee$e%e%ee ee d ddZ&ej !dGddde j"Z'Gdd d e j"Z(Gd!dde j"Z)ee$e$ee$ee$ee$e%eee*ee)d" d#d$Z+d%eiZ,Gd&ddeZ-Gd'ddeZ.Gd(ddeZ/dd)d*ee-e*ee)d+d,dZ0dd)d*ee.e*ee)d+d-dZ1dd)d*ee/e*ee)d+d.dZ2dS)0)partial)OptionalCallableListAnyN)nnTensor)MLPPermute)StochasticDepth)ImageClassificationInterpolationMode)_log_api_usage_once) WeightsEnumWeights)_IMAGENET_CATEGORIES)_ovewrite_named_paramSwinTransformerSwin_T_WeightsSwin_S_WeightsSwin_B_Weightsswin_tswin_sswin_bc Cs8|jdd\}}}t|ddd|dd|df}|S)Nrr )shapeFpad)xHW_r$G/tmp/pip-unpacked-wheel-vx7f76es/torchvision/models/swin_transformer.py_patch_merging_pads r&csFeZdZdZejfeedejfdfdd Z e dddZ Z S) PatchMergingzPatch Merging Layer. Args: dim (int): Number of input channels. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. .)dim norm_layercsDtt|||_tjd|d|dd|_|d||_dS)Nr Fbias)super__init__rr(rLinear reductionnorm)selfr(r) __class__r$r%r.,s  zPatchMerging.__init__r cCst|}|dddddddddf}|dddddddddf}|dddddddddf}|dddddddddf}t||||gd}||}||}|S)z Args: x (Tensor): input tensor with expected layout of [..., H, W, C] Returns: Tensor with layout of [..., H/2, W/2, 2*C] .rNr r)r&torchcatr1r0)r2r Zx0x1Zx2Zx3r$r$r%forward3s      zPatchMerging.forward) __name__ __module__ __qualname____doc__r LayerNormintrModuler.rr: __classcell__r$r$r3r%r'%s$r') input qkv_weight proj_weightrelative_position_bias window_size num_heads shift_sizeattention_dropoutdropoutqkv_bias proj_biasc ! CsP|j\} } } }|d| |d|d}|d| |d|d}t|ddd|d|f}|j\}}}}|d|krd|d<|d|krd|d<t|dkrtj||d |d fdd}||d||d}|| ||d|d||d|d|}|dddddd| ||d|d|}t ||| }|| d| dd|||ddddd}|d|d|d}}}|||d }| | d d }||}t|dkrR| ||f}d|d f|d |d f|d d ff}d|d f|d |d f|d d ff}d}|D]>}|D]2} |||d|d| d| df<|d7}q>q6|||d|d||d|d}|dddd||d|d}|d|d}||dktd |dktd}|| d|||| d| d}||dd}|d || d| d}tj|d d}tj||d}| | dd| d| d|}t ||| }tj||d}|| ||d||d|d|d|}|dddddd| |||}t|dkr(tj||d|dfdd}|d d d | d | d d f}|S)aE Window based multi-head self attention (W-MSA) module with relative position bias. It supports both of shifted and non-shifted window. Args: input (Tensor[N, H, W, C]): The input tensor or 4-dimensions. qkv_weight (Tensor[in_dim, out_dim]): The weight tensor of query, key, value. proj_weight (Tensor[out_dim, out_dim]): The weight tensor of projection. relative_position_bias (Tensor): The learned relative position bias added to attention. window_size (List[int]): Window size. num_heads (int): Number of attention heads. shift_size (List[int]): Shift size for shifted window attention. attention_dropout (float): Dropout ratio of attention weight. Default: 0.0. dropout (float): Dropout ratio of output. Default: 0.0. qkv_bias (Tensor[out_dim], optional): The bias tensor of query, key, value. Default: None. proj_bias (Tensor[out_dim], optional): The bias tensor of projection. Default: None. Returns: Tensor[N, H, W, C]: The output tensor after shifted window attention. rr)rr )ZshiftsZdimsr r*gr6NgYrC)r()p)rrrsumr7ZrollviewpermuteZreshapeZlinearsizematmulZ transposeZ new_zeros unsqueezeZ masked_fillfloatZsoftmaxrL contiguous)!rDrErFrGrHrIrJrKrLrMrNBr!r"CZpad_rZpad_br r#Zpad_HZpad_WZ num_windowsqkvqkvattnZ attn_maskZh_slicesZw_slicescounthwr$r$r%shifted_window_attentionGs^    ,.0..$($$((, $rec sLeZdZdZd eeeeeeeeeedfdd Ze ddd Z Z S) ShiftedWindowAttentionz/ See :func:`shifted_window_attention`. TrC)r(rHrJrIrMrNrKrLc stt|dks"t|dkr*td||_||_||_||_||_t j ||d|d|_ t j |||d|_ t td|ddd|dd||_t|jd} t|jd} ttj| | dd} t| d} | dddddf| dddddf} | ddd} | dddddf|jdd7<| dddddf|jdd7<| dddddfd|jdd9<| d d }|d |t jj|jd d dS) Nr z.window_size and shift_size must be of length 2rOr+rrZij)Zindexingr6relative_position_index{Gz?Zstd)r-r.len ValueErrorrHrJrIrKrLrr/r]proj Parameterr7zerosrelative_position_bias_tableZarangestackZmeshgridflattenrUrZrSrTZregister_bufferinit trunc_normal_)r2r(rHrJrIrMrNrKrLZcoords_hZcoords_wZcoordsZcoords_flattenZrelative_coordsrgr3r$r%r.s2 & ,((, zShiftedWindowAttention.__init__r5c Cs~|jd|jd}|j|j}|||d}|dddd}t||jj |j j ||j|j |j |j |j|jj|j jd S)z Args: x (Tensor): Tensor with layout of [B, H, W, C] Returns: Tensor with same layout as input, i.e. [B, H, W, C] rrr6r )rJrKrLrMrN)rHrorgrTrUrZrXrer]weightrlrIrJrKrLr,)r2r NrGr$r$r%r:s" zShiftedWindowAttention.forward)TTrCrC) r;r<r=r>r@rboolrYr.rr:rBr$r$r3r%rfs  +rfc sreZdZdZddddejefeeeeeee e e e e dej fe dej fd fdd Z e dd d ZZS) SwinTransformerBlocka Swin Transformer Block. Args: dim (int): Number of input channels. num_heads (int): Number of attention heads. window_size (List[int]): Window size. shift_size (List[int]): Shift size for shifted window attention. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob: (float): Stochastic depth rate. Default: 0.0. norm_layer (nn.Module): Normalization layer. Default: nn.LayerNorm. attn_layer (nn.Module): Attention layer. Default: ShiftedWindowAttention @rC.) r(rIrHrJ mlp_ratiorLrKstochastic_depth_probr) attn_layerc stt|| ||_| ||||||d|_t|d|_| ||_t|t |||gt j d|d|_ |j D]:} t| t jrtt j| j| jdk rtt jj| jddqtdS)N)rKrLrow)Zactivation_layerZinplacerLgư>ri)r-r.rnorm1rar stochastic_depthnorm2r r@rZGELUmlpmodules isinstancer/rrZxavier_uniform_rtr,Znormal_) r2r(rIrHrJryrLrKrzr)r{mr3r$r%r. s&    "  zSwinTransformerBlock.__init__r5cCs8|||||}|||||}|S)N)r~rar}rrr2r r$r$r%r:-szSwinTransformerBlock.forward)r;r<r=r>rr?rfr@rrYrrAr.rr:rBr$r$r3r%rws(  #rwcsreZdZdZd eeeeeeeeeeeeeeeede j feede j fd fdd Z d d Z Z S) ra Implements Swin Transformer from the `"Swin Transformer: Hierarchical Vision Transformer using Shifted Windows" `_ paper. Args: patch_size (List[int]): Patch size. embed_dim (int): Patch embedding dimension. depths (List(int)): Depth of each Swin Transformer layer. num_heads (List(int)): Number of attention heads in different layers. window_size (List[int]): Window size. mlp_ratio (float): Ratio of mlp hidden dim to embedding dim. Default: 4.0. dropout (float): Dropout rate. Default: 0.0. attention_dropout (float): Attention dropout rate. Default: 0.0. stochastic_depth_prob (float): Stochastic depth rate. Default: 0.0. num_classes (int): Number of classes for classification head. Default: 1000. block (nn.Module, optional): SwinTransformer Block. Default: None. norm_layer (nn.Module, optional): Normalization layer. Default: None. rxrCN.) patch_size embed_dimdepthsrIrHryrLrKrz num_classesr)blockc stt|| |_| dkr$t} | dkr:ttjdd} g} | t tj d||d|df|d|dfdt ddddg| |t |}d}t t|D]}g}|d|}t ||D]R| t||d}|| ||||fdd |D||||| d |d7}q| tj ||t|dkr| t|| qtj | |_|dt|d}| ||_td|_t|| |_|D]@}t|tjrtjj|jd d |jdk rtj|jqdS) Ngh㈵>)ZepsrOrr)Z kernel_sizeZstrider cs$g|]}ddkrdn|dqS)r rr$).0rdZi_layerr$r% ysz,SwinTransformer.__init__..)rHrJryrLrKrzr)rhri)r-r.rrrwrrr?appendZ SequentialZConv2dr rSrangerjrYr'featuresr1ZAdaptiveAvgPool2davgpoolr/headrrrrrsrtr,Zzeros_)r2rrrrIrHryrLrKrzrr)rZlayersZtotal_stage_blocksZstage_block_idZi_stageZstager(Zsd_probZ num_featuresrr3rr%r.Fsl         zSwinTransformer.__init__cCsH||}||}|dddd}||}t|d}||}|S)NrrOrr )rr1rUrr7rqrrr$r$r%r:s     zSwinTransformer.forward)rxrCrCrCrNN)r;r<r=r>rr@rYrrrrAr.r:rBr$r$r3r%r3s.M) rrrrIrHrzweightsprogresskwargsreturnc KsX|dk rt|dt|jdtf||||||d|} |dk rT| |j|d| S)Nr categories)rrrrIrHrz)r)rrjmetarZload_state_dictZget_state_dict) rrrrIrHrzrrrmodelr$r$r%_swin_transformers  rrc @sFeZdZedeeddejdeddddd d d id d dZ e Z dS)rz7https://download.pytorch.org/models/swin_t-704ceda3.pthZ crop_sizeZ resize_size interpolationibrrUhttps://github.com/pytorch/vision/tree/main/references/classification#swintransformer ImageNet-1KguV^T@glW@zacc@1zacc@5YThese weights reproduce closely the results of the paper using a similar training recipe.Z num_paramsZmin_sizeZrecipeZ_metricsZ_docsurlZ transformsrN r;r<r=rrr rZBICUBIC _COMMON_METAZ IMAGENET1K_V1DEFAULTr$r$r$r%rs*c @sFeZdZedeeddejdeddddd d d id d dZ e Z dS)rz7https://download.pytorch.org/models/swin_s-5e29d889.pthrrirrrrgCT@gףp= X@rrrrNrr$r$r$r%rs*c @sFeZdZedeeddejdeddddd d d id d dZ e Z dS)rz7https://download.pytorch.org/models/swin_b-68c6b09e.pthrri<;rrrgh|?T@g)\(X@rrrrNrr$r$r$r%rs*T)rr)rrrrc KsBt|}tfddgdddddgddddgddgd ||d |S) a Constructs a swin_tiny architecture from `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows `_. Args: weights (:class:`~torchvision.models.Swin_T_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_T_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_T_Weights :members: r*`r rO g?rrrrIrHrzrr)rverifyrrrrr$r$r%rs    c KsBt|}tfddgdddddgddddgd d gd ||d |S) a Constructs a swin_small architecture from `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows `_. Args: weights (:class:`~torchvision.models.Swin_S_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_S_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_S_Weights :members: r*rr rOrrrrg333333?r)rrrrr$r$r%r)s    c KsBt|}tfddgdddddgddddgddgd ||d |S) a Constructs a swin_base architecture from `Swin Transformer: Hierarchical Vision Transformer using Shifted Windows `_. Args: weights (:class:`~torchvision.models.Swin_B_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.Swin_B_Weights` below for more details, and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.swin_transformer.SwinTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.Swin_B_Weights :members: r*r r rg?r)rrrrr$r$r%rMs    )rCrCNN)3 functoolsrtypingrrrrr7Ztorch.nn.functionalrZ functionalrrZops.miscr r Zops.stochastic_depthr Ztransforms._presetsr rutilsrZ_apirrZ_metar_utilsr__all__r&ZfxwraprAr'r@rYrerfrwrrvrrrrrrrrr$r$r$r%s~       * d L9k  $ $