U (d{ @sUddlZddlmZddlmZddlmZmZmZm Z m Z m Z ddl Z ddl mZddlmZmZddlmZmZddlmZd d lmZmZd d lmZd d lmZmZd ddddddddddg ZGddde Z GdddeZ!Gdddej"Z#Gdddej"Z$Gd d d ej"Z%e&e&e&e&e&e ee'ee%d! d"d#Z(d$eiZ)e e*efe+d%<e)d&d'd(Z,Gd)ddeZ-Gd*ddeZ.Gd+ddeZ/Gd,ddeZ0Gd-ddeZ1ed.e-j2fd/dd0d1e e-e'ee%d2d3dZ3ed.e.j2fd/dd0d1e e.e'ee%d2d4dZ4ed.e/j2fd/dd0d1e e/e'ee%d2d5dZ5ed.e0j2fd/dd0d1e e0e'ee%d2d6dZ6dd0d1e e1e'ee%d2d7dZ7d@e&e&d:e*e'd:d;dlm9Z9e9e-j2j:e.j2j:e/j2j:e0j2j:d?Z;dS)AN) OrderedDict)partial)AnyCallableList NamedTupleOptionalDict)Conv2dNormActivationMLP)ImageClassificationInterpolationMode)_log_api_usage_once) WeightsEnumWeights)_IMAGENET_CATEGORIES)handle_legacy_interface_ovewrite_named_paramVisionTransformerViT_B_16_WeightsViT_B_32_WeightsViT_L_16_WeightsViT_L_32_WeightsViT_H_14_Weightsvit_b_16vit_b_32vit_l_16vit_l_32vit_h_14c@sVeZdZUeed<eed<eed<ejZedej fed<ej Z edej fed<dS)ConvStemConfig out_channels kernel_sizestride. norm_layeractivation_layerN) __name__ __module__ __qualname__int__annotations__nnZ BatchNorm2dr%rModuleZReLUr&r.r.I/tmp/pip-unpacked-wheel-vx7f76es/torchvision/models/vision_transformer.pyr! s r!cs:eZdZdZdZeeedfdd ZfddZZ S)MLPBlockzTransformer MLP block.r )in_dimmlp_dimdropoutcsdtj|||gtjd|d|D]:}t|tjr$tj|j |j dk r$tjj |j ddq$dS)N)r&Zinplacer3ư>std) super__init__r,ZGELUmodules isinstanceLinearinitZxavier_uniform_weightbiasnormal_)selfr1r2r3m __class__r.r/r8-s    zMLPBlock.__init__c s|dd}|dks|dkrxtdD]R} dD]H} |d| dd| } |d| d| } | |kr,|| || <q,q$t|||||||dS)Nversionr )r=r>Zlinear_r.)getrangepopr7_load_from_state_dict) r@Z state_dictprefixZlocal_metadatastrictZ missing_keysZunexpected_keysZ error_msgsrDitypeZold_keyZnew_keyrBr.r/rJ6s"  zMLPBlock._load_from_state_dict) r'r(r)__doc___versionr*floatr8rJ __classcell__r.r.rBr/r0(s r0c sZeZdZdZeejddfeeeeee de jj fdfdd Z e j dd d ZZS) EncoderBlockzTransformer encoder block.r4Zeps.) num_heads hidden_dimr2r3attention_dropoutr%csVt||_|||_tj|||dd|_t||_|||_ t ||||_ dS)NT)r3Z batch_first) r7r8rUln_1r,ZMultiheadAttentionself_attentionDropoutr3ln_2r0mlp)r@rUrVr2r3rWr%rBr.r/r8Ys    zEncoderBlock.__init__inputcCsjt|dkd|j||}|j|||dd\}}||}||}||}||}||S)NrF2Expected (batch_size, seq_length, hidden_dim) got F)querykeyvalueZ need_weights) torch_assertdimshaperXrYr3r[r\)r@r^x_yr.r.r/forwardns    zEncoderBlock.forwardr'r(r)rOrr, LayerNormr*rQrrcr-r8TensorrjrRr.r.rBr/rSVs rSc s^eZdZdZeejddfeeeeeeee de jj fdfdd Z e j dd d ZZS) Encoderz?Transformer Model Encoder for sequence to sequence translation.r4rT.) seq_length num_layersrUrVr2r3rWr%c s~tttd||jdd|_t||_ t } t |D] } t ||||||| d| <qBt | |_|||_dS)Nrg{Gz?r5Zencoder_layer_)r7r8r, Parameterrcemptyr? pos_embeddingrZr3rrHrS Sequentiallayersln) r@rorprUrVr2r3rWr%rurMrBr.r/r8}s    zEncoder.__init__r]cCs<t|dkd|j||j}||||S)NrFr_)rcrdrerfrsrvrur3)r@r^r.r.r/rjs zEncoder.forwardrkr.r.rBr/rnzs rncseZdZdZddddeejdddfeeeeeeeeee ee de jj fe e ed fd d Ze je jd d d Ze jdddZZS)rz;Vision Transformer as per https://arxiv.org/abs/2010.11929.iNr4rT.) image_size patch_sizerprUrVr2r3rW num_classesrepresentation_sizer%conv_stem_configsc  stt|t||dkd||_||_||_||_||_ ||_ | |_ | |_ | |_ | dk rt} d}t| D]:\}}| d|t||j|j|j|j |jd|j}qx| dtj||dd| |_ntjd|||d |_||d }ttdd||_|d7}t|||||||| |_||_t}| dkrRt || |d <n,t || |d <t!|d <t | | |d <t||_"t#|jtjr|jj$|jjd|jjd}tj%j&|jj't()d|d|jj*dk rbtj%+|jj*nj|jj,dk rbt#|jj,tjrbtj%j-|jj,j'dt()d|jj,jd|jj,j*dk rbtj%+|jj,j*t.|j"d rt#|j"j/tj r|j"j/j0}tj%j&|j"j/j't()d|dtj%+|j"j/j*t#|j"j1tj rtj%+|j"j1j'tj%+|j"j1j*dS)Nrz&Input shape indivisible by patch size!rFZ conv_bn_relu_) in_channelsr"r#r$r%r& conv_lastr)r}r"r#)r}r"r#r$r head pre_logitsZactr5rwg@)Zmeanr6)2r7r8rrcrdrxryrVr2rWr3rzr{r%r,rt enumerateZ add_moduler r"r#r$r&ZConv2d conv_projrqzeros class_tokenrnencoderrorr;ZTanhheadsr:r}r<Z trunc_normal_r=mathsqrtr>Zzeros_r~r?hasattrrZ in_featuresr)r@rxryrprUrVr2r3rWrzr{r%r|Zseq_projZ prev_channelsrMZconv_stem_layer_configroZ heads_layersZfan_inrBr.r/r8s            zVisionTransformer.__init__)rgreturnc Csx|j\}}}}|j}t||jkdt||jkd||}||}||}|||j||}|ddd}|S)NzWrong image height!zWrong image width!rr r) rfryrcrdrxrreshaperVpermute) r@rgnchwpZn_hZn_wr.r.r/_process_input s z VisionTransformer._process_input)rgcCs^||}|jd}|j|dd}tj||gdd}||}|dddf}||}|S)Nrrre)rrfrexpandrccatrr)r@rgrZbatch_class_tokenr.r.r/rj!s    zVisionTransformer.forward)r'r(r)rOrr,rlr*rQrrrcr-rr!r8rmrrjrRr.r.rBr/rs.   i) ryrprUrVr2weightsprogresskwargsrc Ks|dk rTt|dt|jd|jdd|jddks>tt|d|jdd|dd}tf||||||d|} |r| |j|d | S) Nrz categoriesmin_sizerrrx)rxryrprUrVr2)r)rlenmetaAssertionErrorrIrZload_state_dictZget_state_dict) ryrprUrVr2rrrrxmodelr.r.r/_vision_transformer4s$    rr _COMMON_METAz(https://github.com/facebookresearch/SWAGz:https://github.com/facebookresearch/SWAG/blob/main/LICENSE)recipelicensec @seZdZedeeddedddddd d id d d Zedeeddej de dddddd iddd Z edeeddej de ddddddd iddd Z eZ dS)rz9https://download.pytorch.org/models/vit_b_16-c867db91.pthr crop_sizei(rrzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_b_16 ImageNet-1KgS㥛DT@g1ZW@zacc@1zacc@5 These weights were trained from scratch by using a modified version of `DeIT `_'s training recipe.  num_paramsrr_metrics_docsurlZ transformsrz>https://download.pytorch.org/models/vit_b_16_swag-9ac1b537.pthr resize_size interpolationi^-)rrg~jtSU@giX@ These weights are learnt via transfer learning by end-to-end fine-tuning the original `SWAG `_ weights on ImageNet-1K data. rrrrzAhttps://download.pytorch.org/models/vit_b_16_lc_swag-4e70ced5.pth+https://github.com/pytorch/vision/pull/5793gbX9xT@gQ X@ These weights are composed of the original frozen `SWAG `_ trunk weights and a linear classifier learnt on top of them trained on ImageNet-1K data. rrrrrNr'r(r)rrr r IMAGENET1K_V1rBICUBIC_COMMON_SWAG_METAIMAGENET1K_SWAG_E2E_V1IMAGENET1K_SWAG_LINEAR_V1DEFAULTr.r.r.r/r_sn c @s@eZdZedeeddedddddd d id d d ZeZdS)rz9https://download.pytorch.org/models/vit_b_32-d86f8d99.pthrri1BrzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_b_32rg|?5^R@gW@rrrrN r'r(r)rrr rrrr.r.r.r/rs  c @seZdZedeedddeddddd d d id d dZedeeddej de dddddd idddZ edeeddej de ddddddd idddZ eZ dS)rz9https://download.pytorch.org/models/vit_l_16-852ce7e3.pthr)rri#rzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_l_16rg|?5^S@gFԨW@ra These weights were trained from scratch by using a modified version of TorchVision's `new training recipe `_. rrz>https://download.pytorch.org/models/vit_l_16_swag-4f3808c9.pthri0)rrgjtV@gT㥛ĠX@rrzAhttps://download.pytorch.org/models/vit_l_16_lc_swag-4d563306.pthrgMbXIU@g^I [X@rrNrr.r.r.r/rsn c @s@eZdZedeeddedddddd d id d d ZeZdS)rz9https://download.pytorch.org/models/vit_l_32-c7638314.pthrri[ErzNhttps://github.com/pytorch/vision/tree/main/references/classification#vit_l_32rg|?5>S@gGzDW@rrrrNrr.r.r.r/r s  c @szeZdZedeeddejdedddddd id d d Z ed eeddejdeddddddd iddd Z e Z dS)rz>https://download.pytorch.org/models/vit_h_14_swag-80465313.pthri%)rrrgS#V@g#~jX@rrrrzAhttps://download.pytorch.org/models/vit_h_14_lc_swag-c1eb923e.pthrri@%rgZd;OmU@gQnX@rrN) r'r(r)rrr rrrrrrr.r.r.r/r"sPZ pretrained)rT)rr)rrrrc Ks(t|}tfddddd||d|S)a Constructs a vit_b_16 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_B_16_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_B_16_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_B_16_Weights :members:   ryrprUrVr2rr)rverifyrrrrr.r.r/rWs c Ks(t|}tfddddd||d|S)a Constructs a vit_b_32 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_B_32_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_B_32_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_B_32_Weights :members: rrrr)rrrrr.r.r/rxs c Ks(t|}tfddddd||d|S)a Constructs a vit_l_16 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_L_16_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_L_16_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_L_16_Weights :members: rr)rrrrr.r.r/rs c Ks(t|}tfddddd||d|S)a Constructs a vit_l_32 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_L_32_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_L_32_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_L_32_Weights :members: rrrrrr)rrrrr.r.r/rs c Ks(t|}tfddddd||d|S)a Constructs a vit_h_14 architecture from `An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale `_. Args: weights (:class:`~torchvision.models.ViT_H_14_Weights`, optional): The pretrained weights to use. See :class:`~torchvision.models.ViT_H_14_Weights` below for more details and possible values. By default, no pre-trained weights are used. progress (bool, optional): If True, displays a progress bar of the download to stderr. Default is True. **kwargs: parameters passed to the ``torchvision.models.vision_transformer.VisionTransformer`` base class. Please refer to the `source code `_ for more details about this class. .. autoclass:: torchvision.models.ViT_H_14_Weights :members: rriir)rrrrr.r.r/r s bicubicFzOrderedDict[str, torch.Tensor])rxry model_stateinterpolation_mode reset_headsrcCsh|d}|j\}}}|dkr,td|j||dd} | |krd|d8}| d8} |ddddddf} |ddddddf} | ddd} tt|} | | |krtd| | d|| d|| | } ||} tjj | | |d d }|d|| }|ddd}t j | |gdd }||d<|rdt }| D]\}}|d s@|||<q@|}|S) aThis function helps interpolating positional embeddings during checkpoint loading, especially when you want to apply a pre-trained model on images with different resolution. Args: image_size (int): Image size of the new model. patch_size (int): Patch size of the new model. model_state (OrderedDict[str, torch.Tensor]): State dict of the pre-trained model. interpolation_mode (str): The algorithm used for upsampling. Default: bicubic. reset_heads (bool): If true, not copying the state of heads. Default: False. Returns: OrderedDict[str, torch.Tensor]: A state dict which can be loaded into the new model. zencoder.pos_embeddingrz%Unexpected position embedding shape: r NrzPseq_length is not a perfect square! Instead got seq_length_1d * seq_length_1d = z and seq_length = T)sizemodeZ align_cornersrr)rf ValueErrorrr*rrrr,Z functionalZ interpolatercrritems startswith)rxryrrrrsrrorVZnew_seq_lengthZpos_embedding_tokenZpos_embedding_imgZ seq_length_1dZnew_seq_length_1dZnew_pos_embedding_imgZnew_pos_embeddingZmodel_state_copykvr.r.r/interpolate_embeddingssF     r) _ModelURLs)rrrr)rF)s      .$& !IJ5" " " " $ O