U ad2l@sXdZddlZddlZddlmZddlmZddlmZddl m Z ddl m Z ddl mZmZmZmZmZmZdd lmZdd lmZdd lmZmZdd lmZd dZGddde ZddZGdddZ Gddde Z!Gddde Z"Gddde Z#d,ddZ$d-d d!Z%d"d#Z&d$d%Z'Gd&d'd'e(Z)Gd(d)d)e(Z*Gd*d+d+Z+dS).a Title: Dumpscript management command Project: Hardytools (queryset-refactor version) Author: Will Hardy (http://willhardy.com.au) Date: June 2008 Usage: python manage.py dumpscript appname > scripts/scriptname.py $Revision: 217 $ Description: Generates a Python script that will repopulate the database using objects. The advantage of this approach is that it is easy to understand, and more flexible than directly populating the database, or using XML. * It also allows for new defaults to take effect and only transfers what is needed. * If a new database schema has a NEW ATTRIBUTE, it is simply not populated (using a default value will make the transition smooth :) * If a new database schema REMOVES AN ATTRIBUTE, it is simply ignored and the data moves across safely (I'm assuming we don't want this attribute anymore. * Problems may only occur if there is a new model and is now a required ForeignKey for an existing model. But this is easy to fix by editing the populate script. Half of the job is already done as all ForeingKey lookups occur though the locate_object() function in the generated script. Improvements: See TODOs and FIXMEs scattered throughout :-) N)apps) ContentType)ObjectDoesNotExist) BaseCommand)router) AutoField BooleanField DateField DateTimeField FileField ForeignKey) Collector)timezone) force_str smart_str) signalcommandc Cs|jj}|}|jjj}|}t||}t|drft|jdrft|jjdrf|jj}|jjj}t||}q$t|j}|D]j}||}|dk rtt|t j rt |st |}t d|||<qtt|tttfsttd|||<qtd||||||f} | S)z Is called every time an object that will not be exported is required. Where orm_obj is the referred object. We postpone the lookup to locate_object() which will be run on the generated script _metapknameNdateutil.parser.parse("%s")%sz5 importer.locate_object(%s, "%s", %s, "%s", %s, %s ) )r object_namerrgetattrhasattrmake_clean_dict__dict__ isinstancedatetimerZis_awareZ make_awareStrToCodeChanger isoformatstrintfloat) Zorm_objZ the_classZoriginal_classpk_nameZoriginal_pk_nameZpk_value clean_dictkeyvoutputr(T/tmp/pip-unpacked-wheel-portabvm/django_extensions/management/commands/dumpscript.pyorm_item_locator1s8  $      r*cs,eZdZdZfddZeddZZS)Commandz-Dumps the data as a customised python script.cs2t||jddd|jdddddd dS) Nappname+)nargsz --autofield store_falseskip_autofieldTz#Include Autofields (like pk fields))actiondestdefaulthelp)super add_arguments add_argument)selfparser __class__r(r)r6Zs zCommand.add_argumentscOsJ|d}t|}i}t|||j|j|d}|jt||jddS)Nr,)modelscontextstdoutstderroptions ) get_modelsScriptr>r?writer )r8argsr@ app_labelsr<r=scriptr(r(r)handlebszCommand.handle)__name__ __module__ __qualname__r4r6rrH __classcell__r(r(r:r)r+Ws r+cstfg}|sBtD]&}|fddt|jD7}q|S|D]P}d|krv|dd\}}|t||qF|fddt|D7}qF|S)z Get a list of models for the given app labels, with some exceptions. TODO: If a required model is referenced, it should also be included. Or at least discovered with a get_or_create() call. csg|]}|kr|qSr(r(.0mZEXCLUDED_MODELSr(r) szget_models...csg|]}|kr|qSr(r(rMrPr(r)rQs) rrZget_app_configsZget_app_configlabelrBsplitappendZ get_model)rFr<Zapp app_labelZ model_namer(rPr)rBzs  $"rBc@s2eZdZdZd ddZddZdd ZeeZdS) Codez A snippet of python script. This keeps track of import statements and can be output to a string. In the future, other features such as custom indentation might be included in this class. NcCs*|s tj}|stj}||_||_||_dSN)sysr>r?indent)r8r\r>r?r(r(r)__init__sz Code.__init__cCsN|jr:|jt|jtdg|jdg|j|jdSt|j|jdSdS)z0 Return a string representation of this script.  num_indentsN)importsr?rDrepr import_linesflatten_blockslinesr\r8r(r(r)__str__s"z Code.__str__cCs"|jrdd|jDSgSdS)z4 Take the stored imports and converts them to lines cSsg|]\}}d||fqS)zfrom %s import %sr()rNr%valuer(r(r)rQsz)Code.get_import_lines..N)raitemsrfr(r(r)get_import_linesszCode.get_import_lines)rYNN) rIrJrK__doc__r]rgrjpropertyrcr(r(r(r)rXs  rXcsBeZdZdZd fdd ZddZeeZddZeeZ Z S) ModelCodezJ Produces a python script that can recreate data for a given model class. Ncs:tjd||d||_|dkr$i}||_||_g|_dS)Nrr\r>r?)r5r]modelr=r@ instances)r8ror=r>r?r@r:r(r)r]szModelCode.__init__cCs|jjt|jjiS)zo Return a dictionary of import statements, with the variable being defined as the key. )rorIrrJrfr(r(r) get_importsszModelCode.get_importsc Cszg}t|jjD]D\}}t||d|j|j|j|jd}|j ||j r||j 7}q|j D]}|j r`||j 7}q`|S) Return a list of lists or strings, representing the code body. Each list is a block, each string is a statement. rS)instanceidr=r>r?r@) enumerateroZ_default_managerall InstanceCoder=r>r?r@rprV waiting_listre)r8codecounteritemrsr(r(r) get_liness     zModelCode.get_lines)NNNN) rIrJrKrkr]rqrlrar|rerLr(r(r:r)rms  rmcsXeZdZdZdfdd ZdddZeeZdd Zd d Z dd d Z dddZ Z S)rwzM Produces a python script that can recreate data for a given model instance. Nc stjd||di|_||_||_|jj|_|dkr:i}||_d|jjj |f|_ d|_ d|_ t |jjj|_i|_|jjjD]J}z|jjjjsWqWntk rYnXt t|j|j|j|<qdS)z, We need the instance in question and an id rrnN%s_%sF)r5r]rar@rsr;ror=rZdb_table variable_nameskip_me instantiatedlistfieldsrxmany_to_many_waiting_list many_to_many remote_fieldZthroughZ auto_createdAttributeErrorrrrv)r8rsrtr=r>r?r@fieldr:r(r)r]s(  zInstanceCode.__init__FcCsjg}|rgS||7}||7}|r<||j|d7}|rV|d|j|jf||j|d7}|S)ai Return a list of lists or strings, representing the code body. Each list is a block, each string is a statement. force (True or False): if an attribute object cannot be included, it is usually skipped to be processed later. With 'force' set, there will be no waiting: a get_or_create() call is written instead. forcez!%s = importer.save_or_locate(%s) )skip instantiateget_waiting_listrVr~get_many_to_many_lines)r8r code_linesr(r(r)r| s   zInstanceCode.get_linescsjdk rjSjj}tj|jd}t|d}|jjgddtdd|j Dg}dd|D}fd d|D d d krjj j j }d jjtj|f}dj|<d _nd_jS) a$ Determine whether or not this object should be skipped. If this model instance is a parent of a single subclassed instance, skip it. The subclassed instance will create this parent instance for us. TODO: Allow the user to force its creation? N)rs)usingF)Zcollect_relatedcSsg|] }t|qSr()r)rNir(r(r)rQ@sz%InstanceCode.skip..cSsg|] }|jjqSr()rparents)rNsor(r(r)rQAscsg|]}j|kqSr()ro)rNprfr(r)rQBsTrSr})rrsr;rZ db_for_writer ZcollectsumdatavaluescountrrrrorIrr=)r8clsr collectorZ sub_objectsZsub_objects_parentsr#r%r(rfr)r0s    zInstanceCode.skipcCs\g}|jsX|d|j|jjfd|_|jjjj}d|jjt |j|f}|j|j |<|S)z Write lines for instantiation z %s = %s()Tr}) rrVr~rorIrsrrrrr=)r8rr#r%r(r(r)rNs  zInstanceCode.instantiatec Csg}|jd}t|jD]~}z>t|j||j||d}|d|j|j|f|j |Wqt k r~|j |YqYqt k rYqYqXq|S)z= Add lines for any waiting fields that can be completed now. r0)rr0z %s.%s = %s) r@rrxget_attribute_valuersr=rVr~rremove SkipValueDoLater)r8rrr0rrhr(r(r)r^s   zInstanceCode.get_waiting_listc Csg}|jD]\}}t|D]}zZ|jjj}d|jjt||f}d|j |}| d|j |j|f|j| |Wqt k r|rt|} |j|j d|jj<| d|j |j| f|j| |YqXqq|r| d|S)zF Generate lines that define many to many relations for this instance. r}rz %s.%s.add(%s)__extra_importsz%s.%s.add( %s )r^)rrirrrrr;rIrr=rVr~rKeyErrorr*rJr) r8rrerZ rel_itemsZrel_itemr#r%rh item_locatorr(r(r)rus$   z#InstanceCode.get_many_to_many_lines)NNNN)F)F)F) rIrJrKrkr]r|rlrerrrrrLr(r(r:r)rws $ rwcsJeZdZdZd fdd ZddZddZeeZd d e j Z Z S) rCzN Produces a complete python script that can recreate data for the given apps. NcsPtj||di|_||_|dkr(i}||_t||jd<i|jd<||_dS)N)r>r?__avaliable_modelsr)r5r]rar<r=setr@)r8r<r=r>r?r@r:r(r)r]s zScript.__init__c sg}t|}|}|}|dkr|}|d}t||dr^t|jjjd} || n ||t|}||kr|d8}|dkrfdd|D} || 7}| |dd<qq|}q|S)z Work an an appropriate ordering for the models. This isn't essential, but makes the script look nicer because more instances can be defined on their first try. rrror=r>r?r@rSc s$g|]}t|jjjdqS)r)rmr>r?r@rMr=r8r(r)rQsz(Script._queue_models..N)lenpopcheck_dependenciesrmr>r?r@rV) r8r<r= model_queueZnumber_remaining_modelsZ MAX_CYCLESZallowed_cyclesZ previous_number_remaining_modelsro model_classZmissing_modelsr(rr) _queue_modelss*    zScript._queue_modelscCs |jg}|j|j|jdD]T}d|jj|jjf}|j || d|| |j | d| |j q|jD]\}d|jj|jjf}|j || d||j D]"}|js|jr| |jddqqz|dd |d d|jd D]\}}|d d ||fq|S) rr)r=zProcessing model: %s.%s z # r^zRe-processing model: %s.%s TrrSz # Initial Importsrz from %s import %s) FILE_HEADERstriprr<r=rorJrIr?rDrVrcrerprxrr|insertri)r8ryrmsgrorsr%rhr(r(r)r|s(          zScript.get_linesa #!/usr/bin/env python # This file has been automatically generated. # Instead of changing it, create a file called import_helper.py # and put there a class called ImportHelper(object) in it. # # This class will be specially casted so that instead of extending object, # it will actually extend the class BasicImportHelper() # # That means you just have to overload the methods you want to # change, leaving the other ones intact. # # Something that you might want to do is use transactions, for example. # # Also, don't forget to add the necessary Django imports. # # This file was generated with the following command: # %s # # to restore it, run # manage.py runscript module_name.this_script_name # # example: if manage.py is at ./manage.py # and the script is at ./some_folder/some_script.py # you must make sure ./some_folder/__init__.py exists # and run ./manage.py runscript some_folder.some_script import os, sys from django.db import transaction class BasicImportHelper: def pre_import(self): pass @transaction.atomic def run_import(self, import_data): import_data() def post_import(self): pass def locate_similar(self, current_object, search_data): # You will probably want to call this method from save_or_locate() # Example: # new_obj = self.locate_similar(the_obj, {"national_id": the_obj.national_id } ) the_obj = current_object.__class__.objects.get(**search_data) return the_obj def locate_object(self, original_class, original_pk_name, the_class, pk_name, pk_value, obj_content): # You may change this function to do specific lookup for specific objects # # original_class class of the django orm's object that needs to be located # original_pk_name the primary key of original_class # the_class parent class of original_class which contains obj_content # pk_name the primary key of original_class # pk_value value of the primary_key # obj_content content of the object which was not exported. # # You should use obj_content to locate the object on the target db # # An example where original_class and the_class are different is # when original_class is Farmer and the_class is Person. The table # may refer to a Farmer but you will actually need to locate Person # in order to instantiate that Farmer # # Example: # if the_class == SurveyResultFormat or the_class == SurveyType or the_class == SurveyState: # pk_name="name" # pk_value=obj_content[pk_name] # if the_class == StaffGroup: # pk_value=8 search_data = { pk_name: pk_value } the_obj = the_class.objects.get(**search_data) #print(the_obj) return the_obj def save_or_locate(self, the_obj): # Change this if you want to locate the object in the database try: the_obj.save() except: print("---------------") print("Error saving the following object:") print(the_obj.__class__) print(" ") print(the_obj.__dict__) print(" ") print(the_obj) print(" ") print("---------------") raise return the_obj importer = None try: import import_helper # We need this so ImportHelper can extend BasicImportHelper, although import_helper.py # has no knowlodge of this class importer = type("DynamicImportHelper", (import_helper.ImportHelper, BasicImportHelper ) , {} )() except ImportError as e: # From Python 3.3 we can check e.name - string match is for backward compatibility. if 'import_helper' in str(e): importer = BasicImportHelper() else: raise import datetime from decimal import Decimal from django.contrib.contenttypes.models import ContentType try: import dateutil.parser from dateutil.tz import tzoffset except ImportError: print("Please install python-dateutil") sys.exit(os.EX_USAGE) def run(): importer.pre_import() importer.run_import(import_data) importer.post_import() def import_data():  )NNNN)rIrJrKrkr]rr|rlrejoinr[argvrrLr(r(r:r)rCs-  rCrYcs:d}|s dSt|tr"||Sdfdd|DS)zg Take a list (block) or string (statement) and flattens it into a string with indentation. z r^rAcsg|]}t|dqS)rS)rd)rNliner_r(r)rQsz"flatten_blocks..)rr r)rer`Z INDENTATIONr(r_r)rdzs   rdFTc Cszt||j}Wn*tk r:td|jj|jfYnX|rTt|trTtn*t|trr|dk rrt t |St|t rt t |St|t rP|dk rP|jjtkrd|j|jfS|jjj}d|jjt||f}||kr||}|dkrtd|S|j|dks|r8|j|d|jj<t|} | Std|jj|jfn.t|ttfrv|dk rvd |St |SdS) zI Get a string version of the given attribute's value, like repr() might. z+Could not find object for %s.%s, ignoring. Nz3ContentType.objects.get(app_label="%s", model="%s")r}rrrz (FK) %s.%s r)rrrrr;rIrrrrbboolr rr rrorrWrrrJrr*rr r r) r{rr=rr0rhr#r%r~rr(r(r)rs8         rcCsd|kr|}|d=|S|S)N_state)copy)Zthe_dictr$r(r(r)rs rcCs~dd|D|jdg}|jjD].}|js,q |jjj|kr |jj|krHq dSq |jjD] }|jsdqX|jjj|krXdSqXdS)zK Check that all the depenedencies for this model are already in the queue. cSsg|] }|jjqSr()rorIrMr(r(r)rQsz&check_dependencies..rFT)rIrrrror)rorZavaliable_modelsZ allowed_linksrr(r(r)rs   rc@seZdZdZdS)r8 Value could not be parsed or should simply be skipped. NrIrJrKrkr(r(r(r)rsrc@seZdZdZdS)rrNrr(r(r(r)rsrc@seZdZddZddZdS)rcCs ||_dSrZrb)r8stringr(r(r)r]szStrToCodeChanger.__init__cCs|jSrZrrfr(r(r)__repr__szStrToCodeChanger.__repr__N)rIrJrKr]rr(r(r(r)rsr)rY)FT),rkrr[Z django.appsrZ"django.contrib.contenttypes.modelsrZdjango.core.exceptionsrZdjango.core.management.baserZ django.dbrZdjango.db.modelsrrr r r r Zdjango.db.models.deletionr Z django.utilsrZdjango.utils.encodingrrZ"django_extensions.management.utilsrr*r+rBrXrmrwrCrdrrr Exceptionrrrr(r(r(r)s8         &#"$,$l  9