U %d ;@sddlZddlmZmZmZddlZddlmmZ ddlmZm Z dddddgZ Gd ddej Z Gd ddej ZGd ddej ZGd ddej ZGd ddej ZdS)N)ListOptionalTuple)nnTensorResBlock MelResNet Stretch2dUpsampleNetworkWaveRNNcs:eZdZdZd eddfdd Zeeddd ZZS) ralResNet block based on *Efficient Neural Audio Synthesis* [:footcite:`kalchbrenner2018efficient`]. Args: n_freq: the number of bins in a spectrogram. (Default: ``128``) Examples >>> resblock = ResBlock() >>> input = torch.rand(10, 128, 512) # a random spectrogram >>> output = resblock(input) # shape: (10, 128, 512) N)n_freqreturnc sRtttj||dddt|tjddtj||dddt||_dS)NF in_channels out_channels kernel_sizebiasTZinplace)super__init__r SequentialConv1d BatchNorm1dReLUresblock_model)selfr  __class__=/tmp/pip-unpacked-wheel-lbdmvq91/torchaudio/models/wavernn.pyrs  zResBlock.__init__specgramrcCs|||S)zPass the input through the ResBlock layer. Args: specgram (Tensor): the input sequence to the ResBlock layer (n_batch, n_freq, n_time). Return: Tensor shape: (n_batch, n_freq, n_time) )rrr#r r r!forward(s zResBlock.forward)r __name__ __module__ __qualname____doc__intrrr% __classcell__r r rr!rs  csBeZdZdZd eeeeeddfdd Zeed d d ZZS) raMelResNet layer uses a stack of ResBlocks on spectrogram. Args: n_res_block: the number of ResBlock in stack. (Default: ``10``) n_freq: the number of bins in a spectrogram. (Default: ``128``) n_hidden: the number of hidden dimensions of resblock. (Default: ``128``) n_output: the number of output dimensions of melresnet. (Default: ``128``) kernel_size: the number of kernel size in the first Conv1d layer. (Default: ``5``) Examples >>> melresnet = MelResNet() >>> input = torch.rand(10, 128, 512) # a random spectrogram >>> output = melresnet(input) # shape: (10, 128, 508) r N) n_res_blockr n_hiddenn_outputrrcsftfddt|D}tjtj||ddttjddf|tj|ddf|_dS) Ncsg|] }tqSr )r).0_r0r r! Isz&MelResNet.__init__..FrTrr)rrr) rrrangerrrrrmelresnet_model)rr/r r0r1rZ ResBlocksrr4r!rDs  zMelResNet.__init__r"cCs ||S)zPass the input through the MelResNet layer. Args: specgram (Tensor): the input sequence to the MelResNet layer (n_batch, n_freq, n_time). Return: Tensor shape: (n_batch, n_output, n_time - kernel_size + 1) )r7r$r r r!r%Ss zMelResNet.forward)r-r r r r.r&r r rr!r4scs:eZdZdZeeddfdd ZeedddZZS) r aUpscale the frequency and time dimensions of a spectrogram. Args: time_scale: the scale factor in time dimension freq_scale: the scale factor in frequency dimension Examples >>> stretch2d = Stretch2d(time_scale=10, freq_scale=5) >>> input = torch.rand(10, 100, 512) # a random spectrogram >>> output = stretch2d(input) # shape: (10, 500, 5120) N) time_scale freq_scalercst||_||_dSN)rrr9r8)rr8r9rr r!rms zStretch2d.__init__r"cCs||jd|jdS)zPass the input through the Stretch2d layer. Args: specgram (Tensor): the input sequence to the Stretch2d layer (..., n_freq, n_time). Return: Tensor shape: (..., n_freq * freq_scale, n_time * time_scale) )Zrepeat_interleaver9r8r$r r r!r%ss zStretch2d.forwardr&r r rr!r _s c sPeZdZdZd eeeeeeeddfdd Zeeeefd d d Z Z S) r aUpscale the dimensions of a spectrogram. Args: upsample_scales: the list of upsample scales. n_res_block: the number of ResBlock in stack. (Default: ``10``) n_freq: the number of bins in a spectrogram. (Default: ``128``) n_hidden: the number of hidden dimensions of resblock. (Default: ``128``) n_output: the number of output dimensions of melresnet. (Default: ``128``) kernel_size: the number of kernel size in the first Conv1d layer. (Default: ``5``) Examples >>> upsamplenetwork = UpsampleNetwork(upsample_scales=[4, 4, 16]) >>> input = torch.rand(10, 128, 10) # a random spectrogram >>> output = upsamplenetwork(input) # shape: (10, 128, 1536), (10, 128, 1536) r-r r.N)upsample_scalesr/r r0r1rrc std}|D] }||9}q||_|dd||_t||||||_t|d|_g} |D]d} t| d} tj ddd| ddfd| fdd} t jj | j d| dd| | | | q^tj| |_dS)NrrF)rrrpaddingr?)rr total_scaleindentrresnetr resnet_stretchrZConv2dtorchinitZ constant_Zweightappendrupsample_layers) rr=r/r r0r1rrAupsample_scaleZ up_layersZscaleZstretchconvrr r!rs,      zUpsampleNetwork.__init__r"cCsf||d}||}|d}|d}||}|ddddd|j|j f}||fS)aPass the input through the UpsampleNetwork layer. Args: specgram (Tensor): the input sequence to the UpsampleNetwork layer (n_batch, n_freq, n_time) Return: Tensor shape: (n_batch, n_freq, (n_time - kernel_size + 1) * total_scale), (n_batch, n_output, (n_time - kernel_size + 1) * total_scale) where total_scale is the product of all elements in upsample_scales. rN)rC unsqueezerDsqueezerHrB)rr#Z resnet_outputZupsampling_outputr r r!r%s     &zUpsampleNetwork.forward)r-r r r r.) r'r(r)r*rr+rrrr%r,r r rr!r s c s~eZdZdZdeeeeeeeeeeedd fdd Zeeed d d Ze j j dee ee ee efd ddZZS)r a<WaveRNN model based on the implementation from `fatchord `_. The original implementation was introduced in *Efficient Neural Audio Synthesis* [:footcite:`kalchbrenner2018efficient`]. The input channels of waveform and spectrogram have to be 1. The product of `upsample_scales` must equal `hop_length`. Args: upsample_scales: the list of upsample scales. n_classes: the number of output classes. hop_length: the number of samples between the starts of consecutive frames. n_res_block: the number of ResBlock in stack. (Default: ``10``) n_rnn: the dimension of RNN layer. (Default: ``512``) n_fc: the dimension of fully connected layer. (Default: ``512``) kernel_size: the number of kernel size in the first Conv1d layer. (Default: ``5``) n_freq: the number of bins in a spectrogram. (Default: ``128``) n_hidden: the number of hidden dimensions of resblock. (Default: ``128``) n_output: the number of output dimensions of melresnet. (Default: ``128``) Example >>> wavernn = WaveRNN(upsample_scales=[5,5,8], n_classes=512, hop_length=200) >>> waveform, sample_rate = torchaudio.load(file) >>> # waveform shape: (n_batch, n_channel, (n_time - kernel_size + 1) * hop_length) >>> specgram = MelSpectrogram(sample_rate)(waveform) # shape: (n_batch, n_channel, n_freq, n_time) >>> output = wavernn(waveform, specgram) >>> # output shape: (n_batch, n_channel, (n_time - kernel_size + 1) * hop_length, n_classes) r-r.r N) r= n_classes hop_lengthr/n_rnnn_fcrr r0r1rc s:t||_|dr |dn|d|_||_| d|_||_||_tt |j|_ d} |D] } | | 9} q`| |jkrt d| d|t |||| | ||_t||jd||_tj||dd|_tj||j|dd|_tjdd|_tjdd|_t||j||_t||j||_t||j|_dS) Nr>rz/Expected: total_scale == hop_length, but found z != T)Z batch_firstr)rrr_padrPn_auxrOrNr+mathlog2n_bits ValueErrorr upsamplerZLinearfcZGRUrnn1rnn2rrelu1relu2fc1fc2fc3) rr=rNrOr/rPrQrr r0r1rArIrr r!rs,    zWaveRNN.__init__)waveformr#rcs|ddkstd|ddks,td|d|d}}|d}tjd|j|j|jd}tjd|j|j|jd}|\}}| dd}| dd}fddt d D}|d d d d |d|df}|d d d d |d|df} |d d d d |d|d f} |d d d d |d |d f} tj | d ||gd d} | } | } | |\} }| | } | } tj | | gd d} | |\} }| | } tj | | gd d} | } | } tj | | gd d} | } | } | } | dS)aPass the input through the WaveRNN model. Args: waveform: the input waveform to the WaveRNN layer (n_batch, 1, (n_time - kernel_size + 1) * hop_length) specgram: the input spectrogram to the WaveRNN layer (n_batch, 1, n_freq, n_time) Return: Tensor: shape (n_batch, 1, (n_time - kernel_size + 1) * hop_length, n_classes) rz*Require the input channel of waveform is 1z*Require the input channel of specgram is 1r)dtypedevicer>csg|]}j|qSr rTr2irr r!r5'sz#WaveRNN.forward..r.NrRr<Zdim)sizeAssertionErrorrLrEzerosrPrcrdrYZ transposer6catrKrZr[r\r_r]r`r^ra)rrbr#Z batch_sizeh1h2auxZaux_idxZa1Za2a3Za4xresr3r rhr!r% s>    """"      zWaveRNN.forward)r#lengthsrcs|j}|j}tjj|jjf}|\}|dk rF|jj}g}| \}}}tj d|j f||d} tj d|j f||d} tj |df||d} fddt dD} t |D]8|ddddf} fdd| D\}}}}tj | | |gdd} | } | d| \}} | | d } tj | |gdd}|d| \}} | | d } tj | |gdd} t| } tj | |gdd} t| } | }tj|dd}t|d} d | d jd d } || qt|dd d |fS) aInference method of WaveRNN. This function currently only supports multinomial sampling, which assumes the network is trained on cross entropy loss. Args: specgram (Tensor): Batch of spectrograms. Shape: `(n_batch, n_freq, n_time)`. lengths (Tensor or None, optional): Indicates the valid length of each audio in the batch. Shape: `(batch, )`. When the ``specgram`` contains spectrograms with different durations, by providing ``lengths`` argument, the model will compute the corresponding valid output lengths. If ``None``, it is assumed that all the audio in ``waveforms`` have valid length. Default: ``None``. Returns: (Tensor, Optional[Tensor]): Tensor The inferred waveform of size `(n_batch, 1, n_time)`. 1 stands for a single channel. Tensor or None If ``lengths`` argument was provided, a Tensor of shape `(batch, )` is returned. It indicates the valid length in time axis of the output Tensor. Nr)rdrccs6g|].}ddj|j|dddfqS)Nrrerf)rqrr r!r5qsz!WaveRNN.infer..rRcs"g|]}|ddddfqSr:r )r2a)rgr r!r5wsrjrr>r@)rdrcrEr functionalpadrSrYrArkrmrPr6rnrZr[rKr\FZrelur_r`raZsoftmaxZ multinomialfloatrWrGstackZpermute)rr#rurdrcoutputZb_sizer3Zseq_lenrorprsZ aux_splitZm_tZa1_tZa2_tZa3_tZa4_tinpZlogitsZ posteriorr )rqrgrr!inferDs@      z WaveRNN.infer)r-rMrMr.r r r )N)r'r(r)r*rr+rrr%rEZjitexportrrr~r,r r rr!r s0 *7)rUtypingrrrrEZtorch.nn.functionalrrwryr__all__Modulerrr r r r r r r!s #+!G