U d@s(ddlZddlZddlZddlmZmZmZmZddlZddlm Z ddl m Z ddl m Z ddlmZdd lmZd d lmZejejd Zd e e ee d ddZGddde ZGdddeZGdddeZGdddeZGddde ZGdddeZGdddeZGdddeZdS)!N)ListTupleOptionaloverload)Tensor)Module) Parameter)PackedSequence)init)_VF)RNN_TANHRNN_RELU)tensor permutationdimreturncCs |||SN)Z index_select)rrrr8/tmp/pip-unpacked-wheel-ua33x9lu/torch/nn/modules/rnn.pyapply_permutationsrc seZdZUddddddddd g Zd gZeed<eed<eed<eed<eed<eed<e ed<eed<eed <d4eeeeeee eedd fdd Z fddZ ddddZ fddZ ddddZeeeddddZeeeeeeefdd d!Zd5eeeeefedd#d$d%Zeeeed&d'd(Zeeed)d*d+Zedd,d-Zfd.d/Zeeeedd0d1Zfd2d3ZZS)6RNNBasemode input_size hidden_size num_layersbias batch_firstdropout bidirectional proj_size all_weightsrTFrN) rrrrrrr r!r"rc s| | d} tt|_|_|_|_|_|_t |_ |_ | _ |rZdnd} t |tjrd|kr~dkrnn t |trtd|dkr|dkrtd||| dkrtd| |krtd|d krd |}n>|d krd |}n,|d kr |}n|dkr|}n td|g_g_t|D]t| D]r}| dkr\| n|}dkrn|n|| }ttj||ff| }ttj||ff| }ttj|f| }ttj|f| }d}j dkr|r||||f}n||f}n6ttj| |ff| }|r |||||f}n |||f}|dkr8dndddg}|rV|ddg7}j dkrl|dg7}fdd|D}t||D]\}}t||qj|j|qHq:fddjD_ !dS)Ndevicedtyper rrzbdropout should be a number in range [0, 1] representing the probability of an element being zeroedzdropout option adds dropout after all but last recurrent layer, so non-zero dropout expects num_layers greater than 1, but got dropout={} and num_layers={}zEproj_size should be a positive integer or zero to disable projectionsz,proj_size has to be smaller than hidden_sizeLSTMGRUr rrzUnrecognized RNN mode: r_reverseweight_ih_l{}{}weight_hh_l{}{} bias_ih_l{}{} bias_hh_l{}{}weight_hr_l{}{}csg|]}|qSrformat.0xlayersuffixrr rsz$RNNBase.__init__..csg|]}fdd|qS)cst|rt|SdSrhasattrgetattrwnselfrryz-RNNBase.__init__...rr5r?r@rrr:ys)"superr__init__rrrrrrfloatr r!r" isinstancenumbersNumberbool ValueErrorwarningswarnr3_flat_weights_names _all_weightsranger torchemptyzipsetattrextendappend _flat_weightsflatten_parametersreset_parameters)rArrrrrrr r!r"r&r'factory_kwargsnum_directionsZ gate_size directionreal_hidden_sizeZlayer_input_sizeZw_ihZw_hhZb_ihZb_hhZ layer_paramsZw_hrZ param_namesnameparam __class__r8rAr9rrF's   $            zRNNBase.__init__cs@t|dr*||jkr*|j|}||j|<tt|||dS)NrO)r<rOindexrXrEr __setattr__)rAattrvalueidxrarrre~s  zRNNBase.__setattr__rcCsXt|jt|jkrdS|jD]}t|tsdSq|jd}|j}|jD]:}t|jtr~|jj|kr~|jjr~tj j |jsJdSqJt dd|jD}t|t|jkrdStj |ddlm m m}tftr@|jrdnd}|jdkr |d7}t|j||j||j|j|j|j|jt|j W5QRXW5QRXdS)zResets parameter data pointer so that they can use faster code paths. Right now, this works only if the module is on the GPU and cuDNN is enabled. Otherwise, it's a no-op. Nrcss|]}|VqdSr)Zdata_ptr)r5prrr sz-RNNBase.flatten_parameters..r)r r)lenrXrOrHrr'dataZis_cudarRbackendsZcudnnZ is_acceptablesetZcudaZ device_ofZtorch.backends.cudnn.rnnrnnZno_gradZ_use_cudnn_rnn_flatten_weightrr"Z_cudnn_rnn_flatten_weightrZget_cudnn_moderrrrrKr!)rAwZfirst_fwr'fwZunique_data_ptrsrpZ num_weightsrrrrYsF        zRNNBase.flatten_parameterscs2tt|}fddjD_|S)Ncsg|]}fdd|qS)cst|rt|SdSrr;r>r@rrrBrCz+RNNBase._apply...rrDr@rrr:sz"RNNBase._apply..)rEr_applyrOrXrY)rAfnretrar@rrsszRNNBase._applycCs@|jdkrdt|jnd}|D]}t|| |q&dSNrg?rmathsqrt parametersr Zuniform_rAZstdvweightrrrrZs zRNNBase.reset_parametersinput batch_sizesrcCs\|dk r dnd}||kr0td|||j|dkrXtd|j|ddS)Nr r z%input must have {} dimensions, got {}z?input.size(-1) must be equal to input_size. Expected {}, got {})r RuntimeErrorr3rsize)rAr~rZexpected_input_dimrrr check_inputs zRNNBase.check_inputcCsr|dk rt|d}n|jr&|dn|d}|jr:dnd}|jdkr\|j|||jf}n|j|||jf}|SNrrr )intrrr!r"rrrAr~rZ mini_batchr\expected_hidden_sizerrrget_expected_hidden_sizes z RNNBase.get_expected_hidden_sizeExpected hidden size {}, got {})hxrmsgrcCs(||kr$t||t|dSr)rrr3list)rArrrrrrcheck_hidden_sizes zRNNBase.check_hidden_sizer~hiddenrcCs(||||||}|||dSr)rrr)rAr~rrrrrrcheck_forward_argss  zRNNBase.check_forward_args)rrcCs|dkr |St||SrrrArrrrrpermute_hiddenszRNNBase.permute_hiddencCs~d}|jdkr|d7}|jdkr(|d7}|jdk r:|d7}|jdk rL|d 7}|jdkr^|d 7}|jdk rp|d 7}|jf|jS) N{input_size}, {hidden_size}rz, proj_size={proj_size}rz, num_layers={num_layers}T , bias={bias}Fz, batch_first={batch_first}z, dropout={dropout}z, bidirectional={bidirectional})r"rrrr r!r3__dict__rAsrrr extra_reprs      zRNNBase.extra_reprcstt|d|kr"|d_d|kr0d_tjddtrHdSj}jrXdnd}g_ g_t |D]&t |D]}|dkrdnddd d d d g}fd d|D}j rjdkrj|g7_j |n,j|ddg7_j |ddq~jdkrjj|ddg|ddg7_j |dd|ddgq~j|ddg7_j |ddq~qpfddj D_ dS)Nr#r"rr rr+r,r-r.r/r0r1csg|]}|qSrr2r4r7rrr:sz(RNNBase.__setstate__..r)rcsg|]}fdd|qS)cst|rt|SdSrr;r>r@rrrB$rCz1RNNBase.__setstate__...rrDr@rrr:$s)rEr __setstate__rPr"rHstrrr!rOrQrrVrX)rAdrr\r]weightsrarcrrs>   &$zRNNBase.__setstate__csfddjDS)Ncsg|]}fdd|DqS)csg|]}t|qSr)r=)r5r|r@rrr:(sz2RNNBase.all_weights...r)r5rr@rrr:(sz'RNNBase.all_weights..)rPr@rr@rr#&szRNNBase.all_weightscs2tt|}|jdd|_|jdd|_|Sr)rEr_replicate_for_data_parallelrXrO)rAZreplicararrr*sz$RNNBase._replicate_for_data_parallel)rTFr$FrNN)r)__name__ __module__ __qualname__ __constants__Z__jit_unused_properties__r__annotations__rrKrGrFrerYrsrZrrrrrrrrrrpropertyrr r#r __classcell__rrrarrsf  W 0    &rcseZdZdZfddZeejjd e e e e e e fdddZ eejjd e e e e e e fdddZ d d dZ ZS) RNNaApplies a multi-layer Elman RNN with :math:`\tanh` or :math:`\text{ReLU}` non-linearity to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: h_t = \tanh(x_t W_{ih}^T + b_{ih} + h_{t-1}W_{hh}^T + b_{hh}) where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is the input at time `t`, and :math:`h_{(t-1)}` is the hidden state of the previous layer at time `t-1` or the initial hidden state at time `0`. If :attr:`nonlinearity` is ``'relu'``, then :math:`\text{ReLU}` is used instead of :math:`\tanh`. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` would mean stacking two RNNs together to form a `stacked RNN`, with the second RNN taking in outputs of the first RNN and computing the final results. Default: 1 nonlinearity: The non-linearity to use. Can be either ``'tanh'`` or ``'relu'``. Default: ``'tanh'`` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` batch_first: If ``True``, then the input and output tensors are provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: ``False`` dropout: If non-zero, introduces a `Dropout` layer on the outputs of each RNN layer except the last layer, with dropout probability equal to :attr:`dropout`. Default: 0 bidirectional: If ``True``, becomes a bidirectional RNN. Default: ``False`` Inputs: input, h_0 * **input**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of the input sequence. The input can also be a packed variable length sequence. See :func:`torch.nn.utils.rnn.pack_padded_sequence` or :func:`torch.nn.utils.rnn.pack_sequence` for details. * **h_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden state for the input sequence batch. Defaults to zeros if not provided. where: .. math:: \begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{out} ={} & \text{hidden\_size} \end{aligned} Outputs: output, h_n * **output**: tensor of shape :math:`(L, D * H_{out})` for unbatched input, :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features `(h_t)` from the last layer of the RNN, for each `t`. If a :class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output will also be a packed sequence. * **h_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state for each element in the batch. Attributes: weight_ih_l[k]: the learnable input-hidden weights of the k-th layer, of shape `(hidden_size, input_size)` for `k = 0`. Otherwise, the shape is `(hidden_size, num_directions * hidden_size)` weight_hh_l[k]: the learnable hidden-hidden weights of the k-th layer, of shape `(hidden_size, hidden_size)` bias_ih_l[k]: the learnable input-hidden bias of the k-th layer, of shape `(hidden_size)` bias_hh_l[k]: the learnable hidden-hidden bias of the k-th layer, of shape `(hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` .. note:: For bidirectional RNNs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when ``batch_first=False``: ``output.view(seq_len, batch, num_directions, hidden_size)``. .. note:: ``batch_first`` argument is ignored for unbatched inputs. .. include:: ../cudnn_rnn_determinism.rst .. include:: ../cudnn_persistent_rnn.rst Examples:: >>> rnn = nn.RNN(10, 20, 2) >>> input = torch.randn(5, 3, 10) >>> h0 = torch.randn(2, 3, 20) >>> output, hn = rnn(input, h0) csjd|krtd|dd|_|jdkr.d}n |jdkr>d}ntd|jtt|j|f||dS) Nr"=proj_size argument is only supported for LSTM, not RNN or GRU nonlinearitytanhrrelurzUnknown nonlinearity '{}')rLpoprr3rErrF)rAargskwargsrrarrrFs  z RNN.__init__Nr~rrcCsdSrrrAr~rrrrforwardsz RNN.forwardcCsdSrrrrrrrsc Csx|}t|tr(|\}}}}t|d}nd}|dk}|jrBdnd} |s|| }|dk r|dkr|td|d|d}n(|dk r|dkrtd|d|jr|dn|d}d}d}|dkr |jrdnd} t j |j | ||j |j |jd}n |||}|dk s&t|||||jd ksP|jd ksPt|dkr|jd krt|||j|j|j |j|j|j|j } n(t|||j|j|j |j|j|j|j } nZ|jd krt||||j|j|j |j|j|j } n&t||||j|j|j |j|j|j } | d} | d} t|trNt| |||}||| |fS|sh| | } | d} | || |fS) Nrr rr 7For unbatched 2-D input, hx should also be 2-D but got -D tensor5For batched 3-D input, hx should also be 3-D but got r'r&rr)rHr rrr unsqueezerrr!rRzerosrrr'r&rAssertionErrorrrrrnn_tanhrXrr trainingrnn_relusqueezerAr~r orig_inputrsorted_indicesunsorted_indicesmax_batch_size is_batched batch_dimr\resultoutputr output_packedrrrrs               )N)N)Nrrr__doc__rFrrR _jit_internal_overload_methodrrrrr rrrrarr3sf $$rc seZdZdZfddZeeeeeeefdddZ eeeefeeddd Z eeefeeeeefd d d Z e e jjdeeeeefeeeeeffdddZe e jjdeeeeefeeeeeffdddZdddZZS)r(aW$Applies a multi-layer long short-term memory (LSTM) RNN to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: \begin{array}{ll} \\ i_t = \sigma(W_{ii} x_t + b_{ii} + W_{hi} h_{t-1} + b_{hi}) \\ f_t = \sigma(W_{if} x_t + b_{if} + W_{hf} h_{t-1} + b_{hf}) \\ g_t = \tanh(W_{ig} x_t + b_{ig} + W_{hg} h_{t-1} + b_{hg}) \\ o_t = \sigma(W_{io} x_t + b_{io} + W_{ho} h_{t-1} + b_{ho}) \\ c_t = f_t \odot c_{t-1} + i_t \odot g_t \\ h_t = o_t \odot \tanh(c_t) \\ \end{array} where :math:`h_t` is the hidden state at time `t`, :math:`c_t` is the cell state at time `t`, :math:`x_t` is the input at time `t`, :math:`h_{t-1}` is the hidden state of the layer at time `t-1` or the initial hidden state at time `0`, and :math:`i_t`, :math:`f_t`, :math:`g_t`, :math:`o_t` are the input, forget, cell, and output gates, respectively. :math:`\sigma` is the sigmoid function, and :math:`\odot` is the Hadamard product. In a multilayer LSTM, the input :math:`x^{(l)}_t` of the :math:`l` -th layer (:math:`l >= 2`) is the hidden state :math:`h^{(l-1)}_t` of the previous layer multiplied by dropout :math:`\delta^{(l-1)}_t` where each :math:`\delta^{(l-1)}_t` is a Bernoulli random variable which is :math:`0` with probability :attr:`dropout`. If ``proj_size > 0`` is specified, LSTM with projections will be used. This changes the LSTM cell in the following way. First, the dimension of :math:`h_t` will be changed from ``hidden_size`` to ``proj_size`` (dimensions of :math:`W_{hi}` will be changed accordingly). Second, the output hidden state of each layer will be multiplied by a learnable projection matrix: :math:`h_t = W_{hr}h_t`. Note that as a consequence of this, the output of LSTM network will be of different shape as well. See Inputs/Outputs sections below for exact dimensions of all variables. You can find more details in https://arxiv.org/abs/1402.1128. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` would mean stacking two LSTMs together to form a `stacked LSTM`, with the second LSTM taking in outputs of the first LSTM and computing the final results. Default: 1 bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` batch_first: If ``True``, then the input and output tensors are provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: ``False`` dropout: If non-zero, introduces a `Dropout` layer on the outputs of each LSTM layer except the last layer, with dropout probability equal to :attr:`dropout`. Default: 0 bidirectional: If ``True``, becomes a bidirectional LSTM. Default: ``False`` proj_size: If ``> 0``, will use LSTM with projections of corresponding size. Default: 0 Inputs: input, (h_0, c_0) * **input**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of the input sequence. The input can also be a packed variable length sequence. See :func:`torch.nn.utils.rnn.pack_padded_sequence` or :func:`torch.nn.utils.rnn.pack_sequence` for details. * **h_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden state for each element in the input sequence. Defaults to zeros if (h_0, c_0) is not provided. * **c_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{cell})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{cell})` containing the initial cell state for each element in the input sequence. Defaults to zeros if (h_0, c_0) is not provided. where: .. math:: \begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{cell} ={} & \text{hidden\_size} \\ H_{out} ={} & \text{proj\_size if } \text{proj\_size}>0 \text{ otherwise hidden\_size} \\ \end{aligned} Outputs: output, (h_n, c_n) * **output**: tensor of shape :math:`(L, D * H_{out})` for unbatched input, :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features `(h_t)` from the last layer of the LSTM, for each `t`. If a :class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output will also be a packed sequence. When ``bidirectional=True``, `output` will contain a concatenation of the forward and reverse hidden states at each time step in the sequence. * **h_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state for each element in the sequence. When ``bidirectional=True``, `h_n` will contain a concatenation of the final forward and reverse hidden states, respectively. * **c_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{cell})` for unbatched input or :math:`(D * \text{num\_layers}, N, H_{cell})` containing the final cell state for each element in the sequence. When ``bidirectional=True``, `c_n` will contain a concatenation of the final forward and reverse cell states, respectively. Attributes: weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer `(W_ii|W_if|W_ig|W_io)`, of shape `(4*hidden_size, input_size)` for `k = 0`. Otherwise, the shape is `(4*hidden_size, num_directions * hidden_size)`. If ``proj_size > 0`` was specified, the shape will be `(4*hidden_size, num_directions * proj_size)` for `k > 0` weight_hh_l[k] : the learnable hidden-hidden weights of the :math:`\text{k}^{th}` layer `(W_hi|W_hf|W_hg|W_ho)`, of shape `(4*hidden_size, hidden_size)`. If ``proj_size > 0`` was specified, the shape will be `(4*hidden_size, proj_size)`. bias_ih_l[k] : the learnable input-hidden bias of the :math:`\text{k}^{th}` layer `(b_ii|b_if|b_ig|b_io)`, of shape `(4*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the :math:`\text{k}^{th}` layer `(b_hi|b_hf|b_hg|b_ho)`, of shape `(4*hidden_size)` weight_hr_l[k] : the learnable projection weights of the :math:`\text{k}^{th}` layer of shape `(proj_size, hidden_size)`. Only present when ``proj_size > 0`` was specified. weight_ih_l[k]_reverse: Analogous to `weight_ih_l[k]` for the reverse direction. Only present when ``bidirectional=True``. weight_hh_l[k]_reverse: Analogous to `weight_hh_l[k]` for the reverse direction. Only present when ``bidirectional=True``. bias_ih_l[k]_reverse: Analogous to `bias_ih_l[k]` for the reverse direction. Only present when ``bidirectional=True``. bias_hh_l[k]_reverse: Analogous to `bias_hh_l[k]` for the reverse direction. Only present when ``bidirectional=True``. weight_hr_l[k]_reverse: Analogous to `weight_hr_l[k]` for the reverse direction. Only present when ``bidirectional=True`` and ``proj_size > 0`` was specified. .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` .. note:: For bidirectional LSTMs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when ``batch_first=False``: ``output.view(seq_len, batch, num_directions, hidden_size)``. .. note:: For bidirectional LSTMs, `h_n` is not equivalent to the last element of `output`; the former contains the final forward and reverse hidden states, while the latter contains the final forward hidden state and the initial reverse hidden state. .. note:: ``batch_first`` argument is ignored for unbatched inputs. .. include:: ../cudnn_rnn_determinism.rst .. include:: ../cudnn_persistent_rnn.rst Examples:: >>> rnn = nn.LSTM(10, 20, 2) >>> input = torch.randn(5, 3, 10) >>> h0 = torch.randn(2, 3, 20) >>> c0 = torch.randn(2, 3, 20) >>> output, (hn, cn) = rnn(input, (h0, c0)) cstt|jd||dS)Nr()r()rEr(rFrArrrarrrFsz LSTM.__init__r}cCsT|dk rt|d}n|jr&|dn|d}|jr:dnd}|j|||jf}|Sr)rrrr!rrrrrrget_expected_cell_sizeszLSTM.get_expected_cell_sizercCsD|||||d|||d||d|||ddS)Nrz"Expected hidden[0] size {}, got {}rz"Expected hidden[1] size {}, got {})rrrr)rAr~rrrrrrs zLSTM.check_forward_args)rrrcCs(|dkr |St|d|t|d|fS)NrrrrrrrrszLSTM.permute_hiddenNrcCsdSrrrrrrrsz LSTM.forwardcCsdSrrrrrrrsc Cs|}d}t|tr0|\}}}}|d}t|}nNd}|dk}|jrJdnd} |s\|| }|jrl|dn|d}d}d}|dkr|jrdnd} |jdkr|jn|j } t j |j | || |j |jd} t j |j | ||j |j |jd} | | f}n|dkr|rR|ddks&|ddkrd|dd|dd}t|nj|ddksv|ddkrd |dd|dd}t||dd|ddf}|||}|||||dkr t|||j|j|j |j|j|j|j }n&t||||j|j|j |j|j|j }|d}|dd}t|trnt||||}||||fS|s|| }|dd|ddf}||||fSdS) Nrr rr rz=For batched 3-D input, hx and cx should also be 3-D but got (z-D, z -D) tensorsz?For unbatched 2-D input, hx and cx should also be 2-D but got ()rHr rrrrrr!r"rrRrrr'r&rrrrZlstmrXrr rr)rAr~rrrrrrrrr\r^Zh_zerosZc_zerosrrrrrrrrrs~         $" $"     )N)N)N)rrrrrFrrrrrrrrrRrrrr rrrrarr(s.     r(cseZdZdZfddZeejjd e e e e e e fdddZ eejjd e e e e e e fdddZ d d dZ ZS) r*aApplies a multi-layer gated recurrent unit (GRU) RNN to an input sequence. For each element in the input sequence, each layer computes the following function: .. math:: \begin{array}{ll} r_t = \sigma(W_{ir} x_t + b_{ir} + W_{hr} h_{(t-1)} + b_{hr}) \\ z_t = \sigma(W_{iz} x_t + b_{iz} + W_{hz} h_{(t-1)} + b_{hz}) \\ n_t = \tanh(W_{in} x_t + b_{in} + r_t * (W_{hn} h_{(t-1)}+ b_{hn})) \\ h_t = (1 - z_t) * n_t + z_t * h_{(t-1)} \end{array} where :math:`h_t` is the hidden state at time `t`, :math:`x_t` is the input at time `t`, :math:`h_{(t-1)}` is the hidden state of the layer at time `t-1` or the initial hidden state at time `0`, and :math:`r_t`, :math:`z_t`, :math:`n_t` are the reset, update, and new gates, respectively. :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product. In a multilayer GRU, the input :math:`x^{(l)}_t` of the :math:`l` -th layer (:math:`l >= 2`) is the hidden state :math:`h^{(l-1)}_t` of the previous layer multiplied by dropout :math:`\delta^{(l-1)}_t` where each :math:`\delta^{(l-1)}_t` is a Bernoulli random variable which is :math:`0` with probability :attr:`dropout`. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` num_layers: Number of recurrent layers. E.g., setting ``num_layers=2`` would mean stacking two GRUs together to form a `stacked GRU`, with the second GRU taking in outputs of the first GRU and computing the final results. Default: 1 bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` batch_first: If ``True``, then the input and output tensors are provided as `(batch, seq, feature)` instead of `(seq, batch, feature)`. Note that this does not apply to hidden or cell states. See the Inputs/Outputs sections below for details. Default: ``False`` dropout: If non-zero, introduces a `Dropout` layer on the outputs of each GRU layer except the last layer, with dropout probability equal to :attr:`dropout`. Default: 0 bidirectional: If ``True``, becomes a bidirectional GRU. Default: ``False`` Inputs: input, h_0 * **input**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, H_{in})` when ``batch_first=False`` or :math:`(N, L, H_{in})` when ``batch_first=True`` containing the features of the input sequence. The input can also be a packed variable length sequence. See :func:`torch.nn.utils.rnn.pack_padded_sequence` or :func:`torch.nn.utils.rnn.pack_sequence` for details. * **h_0**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` or :math:`(D * \text{num\_layers}, N, H_{out})` containing the initial hidden state for the input sequence. Defaults to zeros if not provided. where: .. math:: \begin{aligned} N ={} & \text{batch size} \\ L ={} & \text{sequence length} \\ D ={} & 2 \text{ if bidirectional=True otherwise } 1 \\ H_{in} ={} & \text{input\_size} \\ H_{out} ={} & \text{hidden\_size} \end{aligned} Outputs: output, h_n * **output**: tensor of shape :math:`(L, H_{in})` for unbatched input, :math:`(L, N, D * H_{out})` when ``batch_first=False`` or :math:`(N, L, D * H_{out})` when ``batch_first=True`` containing the output features `(h_t)` from the last layer of the GRU, for each `t`. If a :class:`torch.nn.utils.rnn.PackedSequence` has been given as the input, the output will also be a packed sequence. * **h_n**: tensor of shape :math:`(D * \text{num\_layers}, H_{out})` or :math:`(D * \text{num\_layers}, N, H_{out})` containing the final hidden state for the input sequence. Attributes: weight_ih_l[k] : the learnable input-hidden weights of the :math:`\text{k}^{th}` layer (W_ir|W_iz|W_in), of shape `(3*hidden_size, input_size)` for `k = 0`. Otherwise, the shape is `(3*hidden_size, num_directions * hidden_size)` weight_hh_l[k] : the learnable hidden-hidden weights of the :math:`\text{k}^{th}` layer (W_hr|W_hz|W_hn), of shape `(3*hidden_size, hidden_size)` bias_ih_l[k] : the learnable input-hidden bias of the :math:`\text{k}^{th}` layer (b_ir|b_iz|b_in), of shape `(3*hidden_size)` bias_hh_l[k] : the learnable hidden-hidden bias of the :math:`\text{k}^{th}` layer (b_hr|b_hz|b_hn), of shape `(3*hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` .. note:: For bidirectional GRUs, forward and backward are directions 0 and 1 respectively. Example of splitting the output layers when ``batch_first=False``: ``output.view(seq_len, batch, num_directions, hidden_size)``. .. note:: ``batch_first`` argument is ignored for unbatched inputs. .. include:: ../cudnn_persistent_rnn.rst Examples:: >>> rnn = nn.GRU(10, 20, 2) >>> input = torch.randn(5, 3, 10) >>> h0 = torch.randn(2, 3, 20) >>> output, hn = rnn(input, h0) cs*d|krtdtt|jd||dS)Nr"rr*)r*)rLrEr*rFrrarrrFsz GRU.__init__NrcCsdSrrrrrrrsz GRU.forwardcCsdSrrrrrrrsc Cs|}t|tr,|\}}}}|d}t|}nd}|dk}|jrFdnd} |s|| }|dk r|dkrtd|d|d}n(|dk r|dkrtd|d|jr|dn|d}d}d}|dkr|jrdnd} t j |j | ||j |j |jd}n |||}|||||dkr^t|||j|j|j |j|j|j|j } n&t||||j|j|j |j|j|j } | d} | d} t|trt| |||}||| |fS|s| | } | d} | || |fSdS) Nrr rr rrrr)rHr rrrrrrr!rRrrrr'r&rrrZgrurXrr rrrrrrrsn              )N)N)Nrrrrarr*sm $$r*cs|eZdZUdddgZeed<eed<eed<eed<eed<deeeeddfdd Ze d d d Z dd d dZ Z S) RNNCellBaserrr weight_ih weight_hhN)rrr num_chunksrcs||d}tt|||_||_||_ttj|||ff||_ ttj|||ff||_ |rttj||f||_ ttj||f||_ n| dd| dd|dS)Nr%bias_ihbias_hh)rErrFrrrr rRrSrrrrZregister_parameterrZ)rArrrrr&r'r[rarrrFs   zRNNCellBase.__init__ricCsJd}d|jkr |jdk r |d7}d|jkr<|jdkr<|d7}|jf|jS)NrrTrrrz, nonlinearity={nonlinearity})rrrr3rrrrrs zRNNCellBase.extra_reprcCs@|jdkrdt|jnd}|D]}t|| |q&dSrvrwr{rrrrZs zRNNCellBase.reset_parameters)NN) rrrrrrrKrrFrrrZrrrrarrs   rcs^eZdZUdZddddgZeed<deeeedd fd d Z de e e e d d dZ Z S)RNNCellarAn Elman RNN cell with tanh or ReLU non-linearity. .. math:: h' = \tanh(W_{ih} x + b_{ih} + W_{hh} h + b_{hh}) If :attr:`nonlinearity` is `'relu'`, then ReLU is used in place of tanh. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` nonlinearity: The non-linearity to use. Can be either ``'tanh'`` or ``'relu'``. Default: ``'tanh'`` Inputs: input, hidden - **input**: tensor containing input features - **hidden**: tensor containing the initial hidden state Defaults to zero if not provided. Outputs: h' - **h'** of shape `(batch, hidden_size)`: tensor containing the next hidden state for each element in the batch Shape: - input: :math:`(N, H_{in})` or :math:`(H_{in})` tensor containing input features where :math:`H_{in}` = `input_size`. - hidden: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the initial hidden state where :math:`H_{out}` = `hidden_size`. Defaults to zero if not provided. - output: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the next hidden state. Attributes: weight_ih: the learnable input-hidden weights, of shape `(hidden_size, input_size)` weight_hh: the learnable hidden-hidden weights, of shape `(hidden_size, hidden_size)` bias_ih: the learnable input-hidden bias, of shape `(hidden_size)` bias_hh: the learnable hidden-hidden bias, of shape `(hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` Examples:: >>> rnn = nn.RNNCell(10, 20) >>> input = torch.randn(6, 3, 10) >>> hx = torch.randn(3, 20) >>> output = [] >>> for i in range(6): hx = rnn(input[i], hx) output.append(hx) rrrrTrN)rrrrrcs4||d}tt|j|||fddi|||_dS)Nr%rr)rErrFr)rArrrrr&r'r[rarrrF.s  zRNNCell.__init__rcCs|dks td|d|dk}|s:|d}|dkrbtj|d|j|j|jd}n|sp|dn|}|j dkrt |||j |j |j|j}n<|j dkrt |||j |j |j|j}n|}td |j |s|d}|S) Nrr z6RNNCell: Expected input to be 1-D or 2-D but received rr rrrrzUnknown nonlinearity: {})rrrrRrrrr'r&rrZ rnn_tanh_cellrrrrZ rnn_relu_cellrr3rrAr~rrrurrrr4sD       zRNNCell.forward)TrNN)N)rrrrrrrrrKrFrrrrrrrarrs 5  rcsVeZdZdZd eeeddfdd Zd eee eefe eefddd Z Z S) LSTMCella A long short-term memory (LSTM) cell. .. math:: \begin{array}{ll} i = \sigma(W_{ii} x + b_{ii} + W_{hi} h + b_{hi}) \\ f = \sigma(W_{if} x + b_{if} + W_{hf} h + b_{hf}) \\ g = \tanh(W_{ig} x + b_{ig} + W_{hg} h + b_{hg}) \\ o = \sigma(W_{io} x + b_{io} + W_{ho} h + b_{ho}) \\ c' = f * c + i * g \\ h' = o * \tanh(c') \\ \end{array} where :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` Inputs: input, (h_0, c_0) - **input** of shape `(batch, input_size)` or `(input_size)`: tensor containing input features - **h_0** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the initial hidden state - **c_0** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the initial cell state If `(h_0, c_0)` is not provided, both **h_0** and **c_0** default to zero. Outputs: (h_1, c_1) - **h_1** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the next hidden state - **c_1** of shape `(batch, hidden_size)` or `(hidden_size)`: tensor containing the next cell state Attributes: weight_ih: the learnable input-hidden weights, of shape `(4*hidden_size, input_size)` weight_hh: the learnable hidden-hidden weights, of shape `(4*hidden_size, hidden_size)` bias_ih: the learnable input-hidden bias, of shape `(4*hidden_size)` bias_hh: the learnable hidden-hidden bias, of shape `(4*hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` On certain ROCm devices, when using float16 inputs this module will use :ref:`different precision` for backward. Examples:: >>> rnn = nn.LSTMCell(10, 20) # (input_size, hidden_size) >>> input = torch.randn(2, 3, 10) # (time_steps, batch, input_size) >>> hx = torch.randn(3, 20) # (batch, hidden_size) >>> cx = torch.randn(3, 20) >>> output = [] >>> for i in range(input.size()[0]): hx, cx = rnn(input[i], (hx, cx)) output.append(hx) >>> output = torch.stack(output, dim=0) TNrrrrcs.||d}tt|j|||fddi|dS)Nr%rr))rErrFrArrrr&r'r[rarrrFs zLSTMCell.__init__rcCs|dks td|d|dk}|s:|d}|dkrjtj|d|j|j|jd}||f}n$|s|dd|ddfn|}t |||j |j |j |j}|s|dd|ddf}|S)Nrz7LSTMCell: Expected input to be 1-D or 2-D but received rr rrr)rrrrRrrrr'r&rZ lstm_cellrrrrr)rAr~rrrrurrrrs*   $zLSTMCell.forward)TNN)N) rrrrrrKrFrrrrrrrrarrWs;rcsFeZdZdZd eeeddfdd Zd eeeeddd Z Z S) GRUCellai A gated recurrent unit (GRU) cell .. math:: \begin{array}{ll} r = \sigma(W_{ir} x + b_{ir} + W_{hr} h + b_{hr}) \\ z = \sigma(W_{iz} x + b_{iz} + W_{hz} h + b_{hz}) \\ n = \tanh(W_{in} x + b_{in} + r * (W_{hn} h + b_{hn})) \\ h' = (1 - z) * n + z * h \end{array} where :math:`\sigma` is the sigmoid function, and :math:`*` is the Hadamard product. Args: input_size: The number of expected features in the input `x` hidden_size: The number of features in the hidden state `h` bias: If ``False``, then the layer does not use bias weights `b_ih` and `b_hh`. Default: ``True`` Inputs: input, hidden - **input** : tensor containing input features - **hidden** : tensor containing the initial hidden state for each element in the batch. Defaults to zero if not provided. Outputs: h' - **h'** : tensor containing the next hidden state for each element in the batch Shape: - input: :math:`(N, H_{in})` or :math:`(H_{in})` tensor containing input features where :math:`H_{in}` = `input_size`. - hidden: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the initial hidden state where :math:`H_{out}` = `hidden_size`. Defaults to zero if not provided. - output: :math:`(N, H_{out})` or :math:`(H_{out})` tensor containing the next hidden state. Attributes: weight_ih: the learnable input-hidden weights, of shape `(3*hidden_size, input_size)` weight_hh: the learnable hidden-hidden weights, of shape `(3*hidden_size, hidden_size)` bias_ih: the learnable input-hidden bias, of shape `(3*hidden_size)` bias_hh: the learnable hidden-hidden bias, of shape `(3*hidden_size)` .. note:: All the weights and biases are initialized from :math:`\mathcal{U}(-\sqrt{k}, \sqrt{k})` where :math:`k = \frac{1}{\text{hidden\_size}}` On certain ROCm devices, when using float16 inputs this module will use :ref:`different precision` for backward. Examples:: >>> rnn = nn.GRUCell(10, 20) >>> input = torch.randn(6, 3, 10) >>> hx = torch.randn(3, 20) >>> output = [] >>> for i in range(6): hx = rnn(input[i], hx) output.append(hx) TNrcs.||d}tt|j|||fddi|dS)Nr%rr )rErrFrrarrrFs zGRUCell.__init__rcCs|dks td|d|dk}|s:|d}|dkrbtj|d|j|j|jd}n|sp|dn|}t |||j |j |j |j}|s|d}|S)Nrz6GRUCell: Expected input to be 1-D or 2-D but received rr rr)rrrrRrrrr'r&rZgru_cellrrrrrrrrrrs(    zGRUCell.forward)TNN)N) rrrrrrKrFrrrrrrrarrs=r)r) rxrMrItypingrrrrrRrmodulerZ parameterr Z utils.rnnr r,r rrrZ _rnn_implsrrrrr(r*rrrrrrrrs8      O8+bY