U 9%e?@sddlZddlZddlmZddlmZmZmZmZm Z m Z ddl Z ddl m Z mZddlmZmZddlmZddlmZmZmZerddlZeeZe d d d ZGd d d ZGd ddeZdS)N)Path)DictListOptionalTypeTypeVarUnion) CONFIG_NAMEPYTORCH_WEIGHTS_NAME)hf_hub_downloadis_torch_available)HfApi)SoftTemporaryDirectoryloggingvalidate_hf_hub_argsT ModelHubMixin)boundc@seZdZdZddddeeefeeeee eedddZ eddd d Z e e dddddddd eeeeefe e eeeeee feeeefe eeed d dZe eeeeeeeeefe eee e eeee fed ddZe ddddddddddd eeeee eeeeeeee eeeeefeeeeefeeeeefed ddZdS)ra A generic mixin to integrate ANY machine learning framework with the Hub. To integrate your framework, your model class must inherit from this class. Custom logic for saving/loading models have to be overwritten in [`_from_pretrained`] and [`_save_pretrained`]. [`PyTorchModelHubMixin`] is a good example of mixin integration with the Hub. Check out our [integration guide](../guides/integrations) for more instructions. NF)configrepo_id push_to_hub)save_directoryrrrreturncKst|}|jddd||t|tr>|tt||r|| }|dk rZ||d<|dkrh|j }|j fd|i|SdS)a Save weights in local directory. Args: save_directory (`str` or `Path`): Path to directory in which the model weights and configuration will be saved. config (`dict`, *optional*): Model configuration specified as a key/value dictionary. push_to_hub (`bool`, *optional*, defaults to `False`): Whether or not to push your model to the Huggingface Hub after saving it. repo_id (`str`, *optional*): ID of your repository on the Hub. Used only if `push_to_hub=True`. Will default to the folder name if not provided. kwargs: Additional key word arguments passed along to the [`~ModelHubMixin._from_pretrained`] method. T)parentsexist_okNrr) rmkdir_save_pretrained isinstancedictr write_textjsondumpscopynamer)selfrrrrkwargsr'X/var/www/html/Darija-Ai-API/env/lib/python3.8/site-packages/huggingface_hub/hub_mixin.pysave_pretrained s  zModelHubMixin.save_pretrainedrrcCstdS)aD Overwrite this method in subclass to define how to save your model. Check out our [integration guide](../guides/integrations) for instructions. Args: save_directory (`str` or `Path`): Path to directory in which the model weights and configuration will be saved. NNotImplementedError)r%rr'r'r(rLs zModelHubMixin._save_pretrained)force_downloadresume_downloadproxiestoken cache_dirlocal_files_onlyrevision) clspretrained_model_name_or_pathr-r.r/r0r1r2r3rc Ks|} d} tj| rPtt| kr2tj| t} qttdt|  nVt | t rz"t t | t|||||||d } Wn(t jjk rttdYnX| dk rt| ddd} t| } W5QRX| d| i|jft | |||||||d | S) a Download a model from the Huggingface Hub and instantiate it. Args: pretrained_model_name_or_path (`str`, `Path`): - Either the `model_id` (string) of a model hosted on the Hub, e.g. `bigscience/bloom`. - Or a path to a `directory` containing model weights saved using [`~transformers.PreTrainedModel.save_pretrained`], e.g., `../path/to/my_model_directory/`. revision (`str`, *optional*): Revision of the model on the Hub. Can be a branch name, a git tag or any commit id. Defaults to the latest commit on `main` branch. force_download (`bool`, *optional*, defaults to `False`): Whether to force (re-)downloading the model weights and configuration files from the Hub, overriding the existing cache. resume_download (`bool`, *optional*, defaults to `False`): Whether to delete incompletely received files. Will attempt to resume the download if such a file exists. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint, e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`. The proxies are used on every request. token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. By default, it will use the token cached when running `huggingface-cli login`. cache_dir (`str`, `Path`, *optional*): Path to the folder where cached files are stored. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, avoid downloading the file and return the path to the local cached file if it exists. model_kwargs (`Dict`, *optional*): Additional kwargs to pass to the model during initialization. Nz not found in rfilenamer3r1r-r/r.r0r2z not found in HuggingFace Hub.rzutf-8)encodingr)model_idr3r1r-r/r.r2r0)ospathisdirr listdirjoinloggerwarningrresolverstrr requests exceptionsRequestExceptionopenr!loadupdate_from_pretrained)r4r5r-r.r/r0r1r2r3 model_kwargsr: config_filefrr'r'r(from_pretrainedWsJ,    zModelHubMixin.from_pretrained) r4r:r3r1r-r/r.r2r0rc KstdS)aOverwrite this method in subclass to define how to load your model from pretrained. Use [`hf_hub_download`] or [`snapshot_download`] to download files from the Hub before loading them. Most args taken as input can be directly passed to those 2 methods. If needed, you can add more arguments to this method using "model_kwargs". For example [`PyTorchModelHubMixin._from_pretrained`] takes as input a `map_location` parameter to set on which device the model should be loaded. Check out our [integration guide](../guides/integrations) for more instructions. Args: model_id (`str`): ID of the model to load from the Huggingface Hub (e.g. `bigscience/bloom`). revision (`str`, *optional*): Revision of the model on the Hub. Can be a branch name, a git tag or any commit id. Defaults to the latest commit on `main` branch. force_download (`bool`, *optional*, defaults to `False`): Whether to force (re-)downloading the model weights and configuration files from the Hub, overriding the existing cache. resume_download (`bool`, *optional*, defaults to `False`): Whether to delete incompletely received files. Will attempt to resume the download if such a file exists. proxies (`Dict[str, str]`, *optional*): A dictionary of proxy servers to use by protocol or endpoint (e.g., `{'http': 'foo.bar:3128', 'http://hostname': 'foo.bar:4012'}`). token (`str` or `bool`, *optional*): The token to use as HTTP bearer authorization for remote files. By default, it will use the token cached when running `huggingface-cli login`. cache_dir (`str`, `Path`, *optional*): Path to the folder where cached files are stored. local_files_only (`bool`, *optional*, defaults to `False`): If `True`, avoid downloading the file and return the path to the local cached file if it exists. model_kwargs: Additional keyword arguments passed along to the [`~ModelHubMixin._from_pretrained`] method. Nr+) r4r:r3r1r-r/r.r2r0rKr'r'r(rJs/zModelHubMixin._from_pretrainedz!Push model using huggingface_hub.) rcommit_messageprivate api_endpointr0branch create_prallow_patternsignore_patternsdelete_patterns) rrrOrPrQr0rRrSrTrUrVrc  Csrt||d} | j||ddj}tD} t| |}|j||d| j|d||||| | | d W5QRSQRXdS)a Upload model checkpoint to the Hub. Use `allow_patterns` and `ignore_patterns` to precisely filter which files should be pushed to the hub. Use `delete_patterns` to delete existing remote files in the same commit. See [`upload_folder`] reference for more details. Args: repo_id (`str`): ID of the repository to push to (example: `"username/my-model"`). config (`dict`, *optional*): Configuration object to be saved alongside the model weights. commit_message (`str`, *optional*): Message to commit while pushing. private (`bool`, *optional*, defaults to `False`): Whether the repository created should be private. api_endpoint (`str`, *optional*): The API endpoint to use when pushing the model to the hub. token (`str`, *optional*): The token to use as HTTP bearer authorization for remote files. By default, it will use the token cached when running `huggingface-cli login`. branch (`str`, *optional*): The git branch on which to push the model. This defaults to `"main"`. create_pr (`boolean`, *optional*): Whether or not to create a Pull Request from `branch` with that commit. Defaults to `False`. allow_patterns (`List[str]` or `str`, *optional*): If provided, only files matching at least one pattern are pushed. ignore_patterns (`List[str]` or `str`, *optional*): If provided, files matching any of the patterns are not pushed. delete_patterns (`List[str]` or `str`, *optional*): If provided, remote files matching any of the patterns will be deleted from the repo. Returns: The url of the commit of your model in the given repository. )Zendpointr0T)rrPr)rmodel) rZ repo_typeZ folder_pathrOr3rSrTrUrVN)rZ create_reporrrr)Z upload_folder)r%rrrOrPrQr0rRrSrTrUrVapitmpZ saved_pathr'r'r(rs 4  zModelHubMixin.push_to_hub)__name__ __module__ __qualname____doc__rrCrrrboolr)r classmethodrrrrrNrJrrr'r'r'r(rs   ,  R0c @sjeZdZdZeddddZedddeeeee eefe ee e e e ee dfee d d d Z dS) PyTorchModelHubMixinaL Implementation of [`ModelHubMixin`] to provide model Hub upload/download capabilities to PyTorch models. The model is set in evaluation mode by default using `model.eval()` (dropout modules are deactivated). To train the model, you should first set it back in training mode with `model.train()`. Example: ```python >>> import torch >>> import torch.nn as nn >>> from huggingface_hub import PyTorchModelHubMixin >>> class MyModel(nn.Module, PyTorchModelHubMixin): ... def __init__(self): ... super().__init__() ... self.param = nn.Parameter(torch.rand(3, 4)) ... self.linear = nn.Linear(4, 5) ... def forward(self, x): ... return self.linear(x + self.param) >>> model = MyModel() # Save model weights to local directory >>> model.save_pretrained("my-awesome-model") # Push model weights to the Hub >>> model.push_to_hub("my-awesome-model") # Download and initialize weights from the Hub >>> model = MyModel.from_pretrained("username/my-awesome-model") ``` Nr*cCs,t|dr|jn|}t||tdS)z7Save weights from a Pytorch model to a local directory.moduleN)hasattrratorchsave state_dictr )r%rZ model_to_saver'r'r(rGsz%PyTorchModelHubMixin._save_pretrainedcpuF) map_locationstrict) r:r3r1r-r/r.r2r0rgrhc  Ksvtj|r$tdtj|t} nt|t|||||||d } |f| } tj| t | d}| j || d| | S)zs