U 9%ea@sdZddlZddlZddlZddlmZddlmZm Z m Z ddl Z ddl Z ddl m Z ddlmZmZmZddlmZdd lmZdd lmZmZmZdd lmZmZmZmZmZm Z d d l!m"Z"e#e$Z%dZ&dZ'd ddgZ(dZ)dZ*dgZ+eGdddeZ,eGdddeZ-eGdddeZ.eGdddeZ/ddZ0ddZ1dIe j2e3e4e j2d!d"d#Z5Gd$d%d%e j6Z7Gd&d'd'e j6Z8Gd(d)d)e j6Z9Gd*d+d+e j6Z:Gd,d-d-e j6Z;Gd.d/d/e j6ZGd4d5d5e j6Z?Gd6d7d7e j6Z@Gd8d9d9e j6ZAGd:d;d;e j6ZBGdZDd?ZEed@eDGdAdBdBeCZFedCeDGdDdEdEeCZGedFeDGdGdHdHeCZHdS)Jz" PyTorch Swinv2 Transformer model.N) dataclass)OptionalTupleUnion)nn)BCEWithLogitsLossCrossEntropyLossMSELoss)ACT2FN)PreTrainedModel) find_pruneable_heads_and_indicesmeshgridprune_linear_layer) ModelOutputadd_code_sample_docstringsadd_start_docstrings%add_start_docstrings_to_model_forwardloggingreplace_return_docstrings) Swinv2Configrz(microsoft/swinv2-tiny-patch4-window8-256@iz Egyptian catc@sbeZdZUdZdZejed<dZe e ejed<dZ e e ejed<dZ e e ejed<dS)Swinv2EncoderOutputa Swinv2 encoder's outputs, with potential hidden states and attentions. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nlast_hidden_state hidden_states attentionsreshaped_hidden_states) __name__ __module__ __qualname____doc__rtorch FloatTensor__annotations__rrrrrr%r%i/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/transformers/models/swinv2/modeling_swinv2.pyrBs rc@steZdZUdZdZejed<dZe ejed<dZ e e ejed<dZ e e ejed<dZ e e ejed<dS)Swinv2ModelOutputaV Swinv2 model's outputs that also contains a pooling of the last hidden states. Args: last_hidden_state (`torch.FloatTensor` of shape `(batch_size, sequence_length, hidden_size)`): Sequence of hidden-states at the output of the last layer of the model. pooler_output (`torch.FloatTensor` of shape `(batch_size, hidden_size)`, *optional*, returned when `add_pooling_layer=True` is passed): Average pooling of the last layer hidden-state. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nr pooler_outputrrr)rrr r!rr"r#r$r(rrrrrr%r%r%r&r'ds r'c@seZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<edd ZdS) Swinv2MaskedImageModelingOutputa Swinv2 masked image model outputs. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `bool_masked_pos` is provided): Masked image modeling (MLM) loss. reconstruction (`torch.FloatTensor` of shape `(batch_size, num_channels, height, width)`): Reconstructed pixel values. hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. NlossreconstructionrrrcCstdt|jS)Nzlogits attribute is deprecated and will be removed in version 5 of Transformers. Please use the reconstruction attribute to retrieve the final output instead.)warningswarn FutureWarningr+selfr%r%r&logitss z&Swinv2MaskedImageModelingOutput.logits)rrr r!r*rr"r#r$r+rrrrpropertyr1r%r%r%r&r)s r)c@steZdZUdZdZeejed<dZ ejed<dZ ee ejed<dZ ee ejed<dZ ee ejed<dS)Swinv2ImageClassifierOutputa Swinv2 outputs for image classification. Args: loss (`torch.FloatTensor` of shape `(1,)`, *optional*, returned when `labels` is provided): Classification (or regression if config.num_labels==1) loss. logits (`torch.FloatTensor` of shape `(batch_size, config.num_labels)`): Classification (or regression if config.num_labels==1) scores (before SoftMax). hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, sequence_length, hidden_size)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs. attentions (`tuple(torch.FloatTensor)`, *optional*, returned when `output_attentions=True` is passed or when `config.output_attentions=True`): Tuple of `torch.FloatTensor` (one for each stage) of shape `(batch_size, num_heads, sequence_length, sequence_length)`. Attentions weights after the attention softmax, used to compute the weighted average in the self-attention heads. reshaped_hidden_states (`tuple(torch.FloatTensor)`, *optional*, returned when `output_hidden_states=True` is passed or when `config.output_hidden_states=True`): Tuple of `torch.FloatTensor` (one for the output of the embeddings + one for the output of each stage) of shape `(batch_size, hidden_size, height, width)`. Hidden-states of the model at the output of each layer plus the initial embedding outputs reshaped to include the spatial dimensions. Nr*r1rrr)rrr r!r*rr"r#r$r1rrrrr%r%r%r&r3s r3cCsR|j\}}}}|||||||||}|ddddddd|||}|S)z2 Partitions the given input into windows. rrr shapeviewpermute contiguous) input_feature window_size batch_sizeheightwidth num_channelswindowsr%r%r&window_partitions$rDcCsN|jd}|d|||||||}|ddddddd|||}|S)z? Merges windows to produce higher resolution features. r7rrr r4r5r6r8)rCr>r@rArBr%r%r&window_reverses $rEF)input drop_probtrainingreturncCsd|dks |s|Sd|}|jdfd|jd}|tj||j|jd}||||}|S)aF Drop paths (Stochastic Depth) per sample (when applied in main path of residual blocks). Comment by Ross Wightman: This is the same as the DropConnect impl I created for EfficientNet, etc networks, however, the original name is misleading as 'Drop Connect' is a different form of dropout in a separate paper... See discussion: https://github.com/tensorflow/tpu/issues/494#issuecomment-532968956 ... I've opted for changing the layer and argument names to 'drop path' rather than mix DropConnect as a layer name and use 'survival rate' as the argument. rFrr)r)dtypedevice)r9ndimr"ZrandrKrLZfloor_div)rGrHrIZ keep_probr9Z random_tensoroutputr%r%r& drop_paths rPcsPeZdZdZd eeddfdd ZejejdddZ e d d d Z Z S) Swinv2DropPathzXDrop paths (Stochastic Depth) per sample (when applied in main path of residual blocks).N)rHrJcst||_dSN)super__init__rH)r0rH __class__r%r&rT s zSwinv2DropPath.__init__rrJcCst||j|jSrR)rPrHrIr0rr%r%r&forwardszSwinv2DropPath.forward)rJcCs d|jS)Nzp={})formatrHr/r%r%r& extra_reprszSwinv2DropPath.extra_repr)N) rrr r!rfloatrTr"TensorrYstrr[ __classcell__r%r%rUr&rQ srQcsHeZdZdZd fdd Zd eejeeje ej dddZ Z S) Swinv2EmbeddingszW Construct the patch and position embeddings. Optionally, also the mask token. Fcstt||_|jj}|jj|_|r@tt dd|j nd|_ |j rjtt d|d|j |_nd|_t|j |_t|j|_dS)Nr)rSrTSwinv2PatchEmbeddingspatch_embeddings num_patches grid_size patch_gridr Parameterr"zeros embed_dim mask_tokenZuse_absolute_embeddingsposition_embeddings LayerNormnormDropouthidden_dropout_probdropout)r0configuse_mask_tokenrcrUr%r&rTs    zSwinv2Embeddings.__init__N) pixel_valuesbool_masked_posrJc Cs||\}}||}|\}}}|dk rb|j||d}|d|} |d| || }|jdk rv||j}||}||fS)Nr7?) rbrlsizeriexpand unsqueezeZtype_asrjro) r0rrrs embeddingsoutput_dimensionsr?Zseq_len_Z mask_tokensmaskr%r%r&rY.s    zSwinv2Embeddings.forward)F)N) rrr r!rTrr"r# BoolTensorrr]rYr_r%r%rUr&r`sr`csLeZdZdZfddZddZeeje ej e e fdddZ Z S) raz This class turns `pixel_values` of shape `(batch_size, num_channels, height, width)` into the initial `hidden_states` (patch embeddings) of shape `(batch_size, seq_length, hidden_size)` to be consumed by a Transformer. cst|j|j}}|j|j}}t|tjj r8|n||f}t|tjj rR|n||f}|d|d|d|d}||_||_||_||_ |d|d|d|df|_ t j ||||d|_dS)Nrr) kernel_sizeZstride)rSrT image_size patch_sizerBrh isinstance collectionsabcIterablercrdrConv2d projection)r0rpr~rrB hidden_sizercrUr%r&rTKs  "zSwinv2PatchEmbeddings.__init__cCs||jddkr