U 3dz@s&dZddlmZddlmZddlZddlmZmZddl m Z ddl m Z dd l mZdd l mZGd d d ed ZGdddeed ZGdddeZGdddeZGdddeZGdddeZGdddeed ZGdddeZGdddeZGdddeZeeeedded ZdS)!zZLosses and corresponding default initial estimators for gradient boosting decision trees. )ABCMeta)abstractmethodN)expit logsumexp) TREE_LEAF)_weighted_percentileDummyClassifierDummyRegressorc@sdeZdZdZdZddZeddZeddd Zed d Z dddZ eddZ eddZ dS) LossFunctionaMAbstract base class for various loss functions. Parameters ---------- n_classes : int Number of classes. Attributes ---------- K : int The number of regression trees to be induced; 1 for regression and binary classification; ``n_classes`` for multi-class classification. FcCs ||_dSN)Kself n_classesr?/tmp/pip-unpacked-wheel-zrfo1fqw/sklearn/ensemble/_gb_losses.py__init__#szLossFunction.__init__cCsdS)z-Default ``init`` estimator for loss function.Nrrrrrinit_estimator&szLossFunction.init_estimatorNcCsdS)adCompute the loss. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves). sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nrryraw_predictions sample_weightrrr__call__*szLossFunction.__call__cKsdS)NCompute the negative gradient. Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. Nrrrrkargsrrrnegative_gradient:szLossFunction.negative_gradient皙?rc Cs||} | } d| |<t|jtkdD](} ||| | ||||dd| f|q0|dd| f||jddddfj| dd7<dS)azUpdate the terminal regions (=leaves) of the given tree and updates the current predictions of the model. Traverses tree and invokes template method `_update_terminal_region`. Parameters ---------- tree : tree.Tree The tree object. X : ndarray of shape (n_samples, n_features) The data array. y : ndarray of shape (n_samples,) The target labels. residual : ndarray of shape (n_samples,) The residuals (usually the negative gradient). raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. sample_weight : ndarray of shape (n_samples,) The weight of each sample. sample_mask : ndarray of shape (n_samples,) The sample mask to be used. learning_rate : float, default=0.1 Learning rate shrinks the contribution of each tree by ``learning_rate``. k : int, default=0 The index of the estimator being updated. rNZaxis) applycopynpwhereZ children_leftr_update_terminal_regionvaluetake) rtreeXrresidualrr sample_mask learning_ratekterminal_regionsZmasked_terminal_regionsleafrrrupdate_terminal_regionsHs$)   &z$LossFunction.update_terminal_regionsc CsdS)z=Template method for updating terminal regions (i.e., leaves).Nr rr+r1r2r,rr-rrrrrr(s z$LossFunction._update_terminal_regioncCsdS)aLReturn the initial raw predictions. Parameters ---------- X : ndarray of shape (n_samples, n_features) The data array. estimator : object The estimator to use to compute the predictions. Returns ------- raw_predictions : ndarray of shape (n_samples, K) The initial raw predictions. K is equal to 1 for binary classification and regression, and equal to the number of classes for multiclass classification. ``raw_predictions`` is casted into float64. Nr)rr, estimatorrrrget_init_raw_predictionssz%LossFunction.get_init_raw_predictions)N)r!r) __name__ __module__ __qualname____doc__is_multi_classrrrrr r3r(r6rrrrr s     A r ) metaclasscs0eZdZdZfddZddZddZZS)RegressionLossFunctionz)Base class for regression loss functions.cstjdddS)Nr)superrr __class__rrrszRegressionLossFunction.__init__cCs t|drt|dstddS)zMake sure estimator has the required fit and predict methods. Parameters ---------- estimator : object The init estimator to check. fitpredictzNThe init parameter must be a valid estimator and support both fit and predict.Nhasattr ValueErrorrr5rrrcheck_init_estimatorsz+RegressionLossFunction.check_init_estimatorcCs||}|ddtjS)Nr"r>)rDreshapeastyper&float64)rr,r5Z predictionsrrrr6s z/RegressionLossFunction.get_init_raw_predictions)r7r8r9r:rrIr6 __classcell__rrrArr=s r=c@s<eZdZdZddZdddZddZdd d Zd dZdS)LeastSquaresErrorzLoss function for least squares (LS) estimation. Terminal regions do not need to be updated for least squares. Parameters ---------- n_classes : int Number of classes. cCs tddS)Nmeanstrategyr rrrrrsz LeastSquaresError.init_estimatorNcCsH|dkrt||dSd|t|||dSdS)arCompute the least squares loss. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves). sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nrr>)r&rOravelsumrrrrrszLeastSquaresError.__call__cKs ||S)aTCompute half of the negative gradient. Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples,) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. rRrrrrr s z#LeastSquaresError.negative_gradientr!rc Cs*|dd| f|||7<dS)aLeast squares does not need to update terminal regions. But it has to update the predictions. Parameters ---------- tree : tree.Tree The tree object. X : ndarray of shape (n_samples, n_features) The data array. y : ndarray of shape (n_samples,) The target labels. residual : ndarray of shape (n_samples,) The residuals (usually the negative gradient). raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. sample_weight : ndarray of shape (n,) The weight of each sample. sample_mask : ndarray of shape (n,) The sample mask to be used. learning_rate : float, default=0.1 Learning rate shrinks the contribution of each tree by ``learning_rate``. k : int, default=0 The index of the estimator being updated. N)rDrR) rr+r,rr-rrr.r/r0rrrr3s(z)LeastSquaresError.update_terminal_regionsc CsdSrrr4rrrr("s z)LeastSquaresError._update_terminal_region)N)r!r) r7r8r9r:rrr r3r(rrrrrNs   *rNc@s2eZdZdZddZd ddZddZd d ZdS) LeastAbsoluteErrorzLoss function for least absolute deviation (LAD) regression. Parameters ---------- n_classes : int Number of classes cCs tdddSNquantile?rQrWr rrrrr9sz!LeastAbsoluteError.init_estimatorNc CsJ|dkrt||Sd|t|t||SdS)atCompute the least absolute error. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves). sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nr>)r&absrRrOrSrrrrr<szLeastAbsoluteError.__call__cKs|}d||dkdS)aCompute the negative gradient. 1.0 if y - raw_predictions > 0.0 else -1.0 Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. rrr>rTrrrrr Ssz$LeastAbsoluteError.negative_gradientc CsZt||kd} |j| dd}|j| dd|j| dd} t| |dd|j|ddf<dS)z1LAD updates terminal regions to median estimates.rr#2 percentileN)r&r'r*rr)) rr+r1r2r,rr-rrterminal_regiondiffrrrr(ds z*LeastAbsoluteError._update_terminal_region)N)r7r8r9r:rrr r(rrrrrU0s  rUcsFeZdZdZdfdd ZddZddd Zdd d Zd d ZZ S)HuberLossFunctionahHuber loss function for robust regression. M-Regression proposed in Friedman 2001. Parameters ---------- alpha : float, default=0.9 Percentile at which to extract score. References ---------- J. Friedman, Greedy Function Approximation: A Gradient Boosting Machine, The Annals of Statistics, Vol. 29, No. 5, 2001. ?cst||_d|_dSr)r@ralphagammarrbrArrrs zHuberLossFunction.__init__cCs tdddSrVr rrrrrsz HuberLossFunction.init_estimatorNc Cs|}||}|j}|dkrX|dkr@tt||jd}ntt|||jd}t||k}|dkrtd||d}t|t|||d}|||jd} nZtd||||d}t|||t|||d}|||} | S)aCompute the Huber loss. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. NdrXrr) rRrcr&r]rZrbrrSshape) rrrrr_rc gamma_maskZsq_lossZlin_losslossrrrrs4"zHuberLossFunction.__call__c Ks|}||}|dkr2tt||jd}ntt|||jd}t||k}tj|jdftjd}||||<|t ||||<||_ |S)aCompute the negative gradient. Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. NrerZdtype) rRr&r]rZrbrzerosrfrLsignrc) rrrrrr_rcrgr-rrrr s z#HuberLossFunction.negative_gradientc  Cst||kd} |j| dd}|j} |j| dd|j| dd} t| |dd} | | } | tt| tt| | |j |df<dS)Nrr#r[r\) r&r'r*rcrrOrkZminimumrZr))rr+r1r2r,rr-rrr^rcr_ZmedianZdiff_minus_medianrrrr(s z)HuberLossFunction._update_terminal_region)ra)N)N r7r8r9r:rrrr r(rMrrrArr`zs  ) r`csDeZdZdZdfdd ZddZddd Zd d Zd d ZZ S)QuantileLossFunctionzLoss function for quantile regression. Quantile regression allows to estimate the percentiles of the conditional distribution of the target. Parameters ---------- alpha : float, default=0.9 The percentile. racst||_|d|_dS)Nre)r@rrbr]rdrArrrs zQuantileLossFunction.__init__cCstd|jdS)NrWrY)r rbrrrrrsz#QuantileLossFunction.init_estimatorNcCs|}||}|j}||k}|dkrX|||d||||jd}nD|t||||d|t|||||}|S)aCompute the Quantile loss. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nr>r)rRrbrSrfr&)rrrrr_rbmaskrhrrrrs$ zQuantileLossFunction.__call__cKs,|j}|}||k}||d||S)rr>)rbrR)rrrrrbrnrrrr "s z&QuantileLossFunction.negative_gradientc Cs\t||kd} |j| dd|j| dd} |j| dd}t| ||j} | |j|df<dS)Nrr#)r&r'r*rr]r)) rr+r1r2r,rr-rrr^r_valrrrr(3s z,QuantileLossFunction._update_terminal_region)ra)NrlrrrArrms   rmc@s0eZdZdZeddZeddZddZdS) ClassificationLossFunctionz-Base class for classification loss functions.cCsdS)aTemplate method to convert raw predictions into probabilities. Parameters ---------- raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. Returns ------- probas : ndarray of shape (n_samples, K) The predicted probabilities. Nrrrrrr_raw_prediction_to_probaKsz3ClassificationLossFunction._raw_prediction_to_probacCsdS)aTemplate method to convert raw predictions to decisions. Parameters ---------- raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. Returns ------- encoded_predictions : ndarray of shape (n_samples, K) The predicted encoded labels. Nrrqrrr_raw_prediction_to_decision[sz6ClassificationLossFunction._raw_prediction_to_decisioncCs t|drt|dstddS)zMake sure estimator has fit and predict_proba methods. Parameters ---------- estimator : object The init estimator to check. rC predict_probazTThe init parameter must be a valid estimator and support both fit and predict_proba.NrErHrrrrIksz/ClassificationLossFunction.check_init_estimatorN)r7r8r9r:rrrrsrIrrrrrpHs   rpcsZeZdZdZfddZddZdddZd d Zd d Zd dZ ddZ ddZ Z S)BinomialDevianceaBinomial deviance loss function for binary classification. Binary classification is a special case; here, we only need to fit one tree instead of ``n_classes`` trees. Parameters ---------- n_classes : int Number of classes. cs.|dkrtd|jj|tjdddSNrz-{0:s} requires 2 classes; got {1:d} class(es)r>r?rGformatrBr7r@rrrArrrszBinomialDeviance.__init__cCs tddSNZpriorrPr rrrrrszBinomialDeviance.init_estimatorNc Cs\|}|dkr.dt||td|Sd|t|||td|SdS)aCompute the deviance (= 2 * negative log-likelihood). Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. Nr)rRr&rOZ logaddexprSrrrrrszBinomialDeviance.__call__cKs|t|S)aPCompute half of the negative gradient. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. )rrRrrrrr s z"BinomialDeviance.negative_gradientc Cst||kd} |j| dd}|j| dd}|j| dd}t||} t|||d||} t| dkrd|j|ddf<n| | |j|ddf<dS)zMake a single Newton-Raphson step. our node estimate is given by: sum(w * (y - prob)) / sum(w * prob * (1 - prob)) we take advantage that: y - prob = residual rr#r>u?j/ N)r&r'r*rSrZr) rr+r1r2r,rr-rrr^ numerator denominatorrrrr(s z(BinomialDeviance._update_terminal_regioncCsZtj|jddftjd}t||dddf<|dddf|dddf8<|S)Nrrrir>r&ZonesrfrLrrRrrZprobarrrrrs$z)BinomialDeviance._raw_prediction_to_probacCs||}tj|ddSNr>r#rrr&Zargmaxrrrrrss z,BinomialDeviance._raw_prediction_to_decisioncCs`||}|dddf}ttjj}t||d|}t|d|}|ddtj S)Nr>r" rtr&finfofloat32epscliplogrJrKrLrr,r5probasZproba_pos_classrrrrrr6s  z)BinomialDeviance.get_init_raw_predictions)N r7r8r9r:rrrr r(rrrsr6rMrrrArruzs  !rucs`eZdZdZdZfddZddZddd Zdd d Zd dZ ddZ ddZ ddZ Z S)MultinomialDeviancezMultinomial deviance loss function for multi-class classification. For multi-class classification we need to fit ``n_classes`` trees at each stage. Parameters ---------- n_classes : int Number of classes. Tcs*|dkrtd|jjt|dS)Nz#{0:s} requires more than 2 classes.rwrrArrrs  zMultinomialDeviance.__init__cCs tddSryr rrrrr sz"MultinomialDeviance.init_estimatorNcCsjtj|jd|jftjd}t|jD]}||k|dd|f<q&tjd||jddt|dd|dS)aCompute the Multinomial deviance. Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. rriNr"r>r#)weights) r&rjrfrrLrangeZaveragerSr)rrrrYr0rrrrszMultinomialDeviance.__call__rc Ks,|tt|dd|ft|ddS)aCompute negative gradient for the ``k``-th class. Parameters ---------- y : ndarray of shape (n_samples,) The target labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. k : int, default=0 The index of the class. Nr>r#)r& nan_to_numexpr)rrrr0kwargsrrrr (s z%MultinomialDeviance.negative_gradientc Cst||kd} |j| dd}|j| dd}|j| dd}t||} | |jd|j9} t|||d||} t| dkrd|j|ddf<n| | |j|ddf<dS)z"Make a single Newton-Raphson step.rr#r>r{r|N)r&r'r*rSrrZr)r}rrrr(;s  z+MultinomialDeviance._update_terminal_regionc Cs*tt|t|ddddtjfSr)r&rrrZnewaxisrqrrrrrWs z,MultinomialDeviance._raw_prediction_to_probacCs||}tj|ddSrrrrrrrs^s z/MultinomialDeviance._raw_prediction_to_decisioncCs@||}ttjj}t||d|}t|tj}|S)Nr>) rtr&rrrrrrKrL)rr,r5rrrrrrr6bs  z,MultinomialDeviance.get_init_raw_predictions)N)r)r7r8r9r:r;rrrr r(rrrsr6rMrrrArrs    rcsZeZdZdZfddZddZdddZd d Zd d Zd dZ ddZ ddZ Z S)ExponentialLossaExponential loss function for binary classification. Same loss as AdaBoost. Parameters ---------- n_classes : int Number of classes. References ---------- Greg Ridgeway, Generalized Boosted Models: A guide to the gbm package, 2007 cs.|dkrtd|jj|tjdddSrvrwrrArrryszExponentialLoss.__init__cCs tddSryr rrrrrszExponentialLoss.init_estimatorNcCs`|}|dkr.ttd|d |Sd|t|td|d |SdS)aCompute the exponential loss Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble. sample_weight : ndarray of shape (n_samples,), default=None Sample weights. N@?rr>)rRr&rOrrSrrrrrs zExponentialLoss.__call__cKs$d|d}|t| |S)aUCompute the residual (= negative gradient). Parameters ---------- y : ndarray of shape (n_samples,) True labels. raw_predictions : ndarray of shape (n_samples, K) The raw predictions (i.e. values from the tree leaves) of the tree ensemble at iteration ``i - 1``. rr)r&rrR)rrrry_rrrr s z!ExponentialLoss.negative_gradientc Cst||kd} |j| dd}|j| dd}|j| dd}d|d} t| |t| |} t|t| |} t| dkrd|j|ddf<n| | |j|ddf<dS)Nrr#rrr{r|)r&r'r*rSrrZr)) rr+r1r2r,rr-rrr^rr~rrrrr(s   z'ExponentialLoss._update_terminal_regioncCs^tj|jddftjd}td||dddf<|dddf|dddf8<|S)Nrrrirr>rrrrrrrs$z(ExponentialLoss._raw_prediction_to_probacCs|dktS)Nr)rRrKintrqrrrrssz+ExponentialLoss._raw_prediction_to_decisioncCsd||}|dddf}ttjj}t||d|}dt|d|}|ddtj S)Nr>rXr"rrrrrr6s  z(ExponentialLoss.get_init_raw_predictions)NrrrrArrjs  r)Z squared_errorZabsolute_errorZhuberrWZdevianceZlog_lossZ exponential)r:abcrrZnumpyr&Z scipy.specialrrZ tree._treerZ utils.statsrdummyr r r r=rNrUr`rmrprurrZLOSS_FUNCTIONSrrrrs6      jJuY2}su