U (dC @sUddlZddlZddlZddlZddlZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl Z ddl Z ddlZ ddlZddlZddlmZmZmZmZmZmZmZmZmZmZddlmZddlZddlZddl m!Z!ddl"m#Z#m$Z$dZ%dUee&e'ee(ddd d Z)dVe'e'e(dd d dZ*ee(e(e(gdfdddZ+dWe'e(e'dddZ,e'e'ee-dddZ.dXe'ee'e-dddZ/dYe'e(e'dddZ0e'ee'd d!d"Z1dZe'e'ee'ee'e(dd#d$d%Z2d[e'e-ee'd'd(d)Z3d\e'e'e-ee'd*d+d,Z4d]e(ee&ee&fd-d.d/Z5d^e'e'ee'ee'd0d1d2Z6e'e'ee'dd3d4d5Z7ej8ej9d6Z:ee'e(fe;d7<e'e'ee'dd3d8d9Zej>ej>d<Z?ee'ed=effe;d><d?d?d@dAZ@ee'eee'ee'ffe;dB<e'ee'ee'ee'fdCdDdEZAd_e'ee'e-e'dFdGdHZBd`e'ee'e-e'dFdIdJZCdae'e'ee'ee'ee'e-ddKdLdMZDee'dNdOdPZEedQe'e&ZFdbeFee'eeFee'eFdRdSdTZGdS)cN) AnyCallableListIterableOptionalTypeVarDictIOTupleIterator)urlparse)tqdm)#_download_file_from_remote_location_is_remote_location_availablezpytorch/vision)content destinationlengthreturnc CsXt|dD}t|d.}|D]"}|s&q|||t|qW5QRXW5QRXdS)Nwbtotal)openr writeupdatelen)rrrfhpbarchunkr>/tmp/pip-unpacked-wheel-vx7f76es/torchvision/datasets/utils.py_save_response_content!s  r!)urlfilename chunk_sizerc sLtjtjj|dtid&ttfddd|jdW5QRXdS)N User-Agentheaderscs SNreadrr%responserr 2z_urlretrieve..r/)r)urllibrequesturlopenRequest USER_AGENTr!iterr)r#r$r%rr,r _urlretrieve0sr6)rcs$tdtddfdd}|S)NzSThe function `gen_bar_update` is deprecated since 0.13 and will be removed in 0.15.rcs0jdkr|r|_||}|jdSr))rrn)count block_sizeZ total_sizeZprogress_bytesrrr bar_update9sz#gen_bar_updater..bar_update)warningswarnr )r;rr:r gen_bar_updater5s   r>)fpathr%rc sdtjftjdkrtddnt}t|d*tfdddD]}||qBW5QRX|S)N) F)Zusedforsecurityrbcs Sr)r*rr%frr r.Hr/zcalculate_md5..r/) hashlibmd5sys version_infodictrr5r hexdigest)r@r%rGrrrDr calculate_md5Bs " rL)r@rGkwargsrcKs|t|f|kSr))rL)r@rGrMrrr check_md5MsrN)r@rGrcCs&tj|sdS|dkrdSt||S)NFT)ospathisfilerN)r@rGrrr check_integrityQs  rRrA)r#max_hopsrc Cs|}dtd}t|dD]V}tjtjj||d4}|j|ksL|jdkr`|W5QRS|j}W5QRXqtd|d|d|ddS) NHEAD)ZMethodr&r'z Request to z exceeded z( redirects. The last redirect points to .)r4ranger0r1r2r3r#RecursionError)r#rSZ initial_urlr(_r-rrr _get_redirect_urlYs rZ)r#rcCsBt|}td|jdkrdStd|j}|dkr8dS|dS)Nz(drive|docs)[.]google[.]comz/file/d/(?P[^/]*)id)r rematchnetlocrPgroup)r#partsr]rrr _get_google_drive_file_idisra)r#rootr$rGmax_redirect_hopsrc Cs8tj|}|stj|}tj||}tj|ddt||rRtd|dStrdt ||nt ||d}t |}|dk rt ||||Sz"td|d|t ||Wnntjjtfk r}zF|ddd kr |d d }td |d|t ||n|W5d}~XYnXt||s4td dS)aDownload a file from a url and place it in root. Args: url (str): URL to download file from root (str): Directory to place downloaded file in filename (str, optional): Name to save the file under. If None, use the basename of the URL md5 (str, optional): MD5 checksum of the download. If None, do not check max_redirect_hops (int, optional): Maximum number of redirect hops allowed Texist_okz$Using downloaded and verified file: N)rSz Downloading  to httpszhttps:zhttp:z;Failed download. Trying https -> http instead. Downloading zFile not found or corrupted.)rOrP expanduserbasenamejoinmakedirsrRprintrrrZradownload_file_from_google_driver6r0errorURLErrorOSErrorreplace RuntimeError)r#rbr$rGrcr@file_iderrr download_urlvs2         rvF)rbprefixrcsBtjfddtD}|dkr>fdd|D}|S)a List all directories at a given root Args: root (str): Path to directory whose folders need to be listed prefix (bool, optional): If true, prepends the path to each result, otherwise only returns the name of the directories found cs&g|]}tjtj|r|qSr)rOrPisdirrk.0prbrr szlist_dir..Tcsg|]}tj|qSrrOrPrkrzdr|rr r}srOrPrilistdir)rbrw directoriesrr|r list_dirs  r)rbsuffixrwrcsDtjfddtD}|dkr@fdd|D}|S)aList all files ending with a suffix at a given root Args: root (str): Path to directory whose folders need to be listed suffix (str or tuple): Suffix of the files to match, e.g. '.png' or ('.jpg', '.png'). It uses the Python "str.endswith" method and is passed directly prefix (bool, optional): If true, prepends the path to each result, otherwise only returns the name of the files found cs0g|](}tjtj|r|r|qSr)rOrPrQrkendswithryrbrrr r}s zlist_files..Tcsg|]}tj|qSrr~rr|rr r}sr)rbrrwfilesrrr list_filess r)r%rcCst||}d}|st|}qt|g|}z(td|}|dk rL|dnd}Wntk rjd}YnX||fS)Nz3Google Drive - (?P<api_response>.+?) api_response) iter_contentnext itertoolschainr\searchdecodeUnicodeDecodeError)r-r%rZ first_chunkr]rrrr _extract_gdrive_api_responses   r)rtrbr$rGc Cstj|}|s|}tj||}tj|ddt||rZtd|rFdndd|dSd}t|d d }t }|j ||dd }|j D]\} } | d r| } qqt|\} } | d krdnd} | dk r|j |t|| ddd }t|\} } | dkr td|dt| |W5QRXt|jdkrtt@t|,}|}td|rntd|W5QRXW5QRX|rt||std|ddS)afDownload a Google Drive file from and place it in root. Args: file_id (str): id of file to be downloaded root (str): Directory to place downloaded file in filename (str, optional): Name to save the file under. If None, use the id of the file. md5 (str, optional): MD5 checksum of the download. If None, do not check TrdzUsing downloaded z and verified zfile: Nzhttps://drive.google.com/ucdownload)r[export)paramsstreamZdownload_warningzVirus scan warningt)ZconfirmzQuota exceededzThe daily quota of the file z} is exceeded and it can't be downloaded. This is a limitation of Google Drive and can only be overcome by trying again later.i(z5]*\s*>|(&(?:[\w\d]+|#\d+|#x[a-f\d]+);)zWe detected some HTML elements in the downloaded file. This most likely means that the download triggered an unhandled API response by GDrive. Please report this to torchvision at https://github.com/pytorch/vision/issues including the response: z&The MD5 checksum of the download file z does not match the one on record.Please delete the file and try again. If the issue persists, please report this to torchvision at https://github.com/pytorch/vision/issues.)rOrPrirkrlrRrmrJrequestsSessiongetcookiesitems startswithrrsr!statst_size contextlibsuppressrrr+r\rr<r=rN)rtrbr$rGr@r#rsessionr-keyvaluetokenrrrtextrrr rnsJ          rn) from_pathto_path compressionrc Cs<t||rd|ddnd}||W5QRXdS)Nzr:rUr)tarfiler extractall)rrrtarrrr _extract_tars$r).bz2.xz_ZIP_COMPRESSION_MAPc Cs8tj|d|rt|ntjd}||W5QRXdS)Nr)r)zipfileZipFiler ZIP_STOREDr)rrrziprrr _extract_zip*sr).tarz.zip_ARCHIVE_EXTRACTORS)r.gzr._COMPRESSED_FILE_OPENERS)rr)rr)z.tbzz.tbz2z.tgz_FILE_TYPE_ALIASES)filercCst|j}|s td|d|d}|tkr>|ft|S|tkrP||dfS|tkrt|dkr|d}|tkr||||fS|d|fStt tt tBt tB}td|d|d dS) aDetect the archive type and/or compression of a file. Args: file (str): the filename Returns: (tuple): tuple of suffix, archive type, and compression Raises: RuntimeError: if file has no suffix or suffix is not supported zFile 'zP' has no suffixes that could be used to detect the archive type and compression.NrUz&Unknown compression or archive type: 'z'. Known suffixes are: 'z'.) pathlibPathsuffixesrsrrrrsortedset)rrrZsuffix2Zvalid_suffixesrrr _detect_file_typeAs$     r)rrremove_finishedrc Cst|\}}}|s"td|d|dkrB|||dk r<|nd}t|}||d*}t|d}||W5QRXW5QRX|rt||S)aDecompress a file. The compression is automatically detected from the file name. Args: from_path (str): Path to the file to be decompressed. to_path (str): Path to the decompressed file. If omitted, ``from_path`` without compression extension is used. remove_finished (bool): If ``True``, remove the file after the extraction. Returns: (str): Path to the decompressed file. z*Couldn't detect a compression from suffix rVNrrCr) rrsrrrrrr+rOremove) rrrr archive_typerZcompressed_file_openerZrfhZwfhrrr _decompressls " rc Cst|dkrtj|}t|\}}}|sNt|tj|tj||d|dSt|}|||||rpt ||S)aDExtract an archive. The archive type and a possible compression is automatically detected from the file name. If the file is compressed but not an archive the call is dispatched to :func:`decompress`. Args: from_path (str): Path to the file to be extracted. to_path (str): Path to the directory the file will be extracted to. If omitted, the directory of the file is used. remove_finished (bool): If ``True``, remove the file after the extraction. Returns: (str): Path to the directory the file was extracted to. Nr)r) rOrPdirnamerrrkrjrrrr)rrrrrrZ extractorrrr extract_archives   r)r# download_root extract_rootr$rGrrcCshtj|}|dkr|}|s(tj|}t||||tj||}td|d|t|||dS)Nz Extracting rf)rOrPrirjrvrkrmr)r#rrr$rGrarchiverrr download_and_extract_archives  r)iterablercCsdddd|DdS)N'z', 'cSsg|] }t|qSr)str)rzitemrrr r}sz#iterable_to_str..)rk)rrrr iterable_to_strsrT)rarg valid_values custom_msgrcCst|tjjs:|dkrd}nd}|jt||d}t||dkrF|S||kr||dk r\|}nd}|j||t|d}t||S)Nz'Expected type str, but got type {type}.z:Expected type str for argument {arg}, but got type {type}.)typerzPUnknown value '{value}' for argument {arg}. Valid values are {{{valid_values}}}.)rrr) isinstancetorchZ_sixZstring_classesformatr ValueErrorr)rrrrmsgrrr verify_str_argsr)N)r")r?)N)rA)NNrA)F)F)r")NN)NF)NF)NNNF)NNN)Hbz2rgziprFrlzmarOos.pathrr\rHrr0 urllib.errorurllib.requestr<rtypingrrrrrrrr r r urllib.parser rrZtorch.utils.model_zoor Z_internally_replaced_utilsrrr4bytesrintr!r6r>rLboolrNrRrZrarvrrrrnr ZIP_BZIP2ZIP_LZMAr__annotations__rrrrrrrrrrrrrrrr s0      4E(&"+ '