U 3d @sDdZddlZddlZddlmZddlmZmZm Z m Z m Z m Z m Z mZmZddlmZmZmZmZmZddlmZdd lmZdd lmZGd d d ZGd ddeZGdddeZGdddeZGdddeZ GdddeZ!GdddeZ"GdddeZ#GdddeZ$GdddeZ%eeee e!e"e$e%dZ&dS) z This module contains loss classes suitable for fitting. It is not part of the public API. Specific losses are used for regression, binary classification or multiclass classification. Nxlogy) CyHalfSquaredErrorCyAbsoluteError CyPinballLossCyHalfPoissonLossCyHalfGammaLossCyHalfTweedieLossCyHalfTweedieLossIdentityCyHalfBinomialLossCyHalfMultinomialLoss)Interval IdentityLinkLogLink LogitLinkMultinomialLogit) check_scalar)ReadonlyArrayWrapper)_weighted_percentilec@seZdZdZdZdZdZdddZddZd d Z dd d Z dddZ d ddZ d!ddZ d"ddZd#ddZd$ddZejdfddZdS)%BaseLossaBase class for a loss function of 1-dimensional targets. Conventions: - y_true.shape = sample_weight.shape = (n_samples,) - y_pred.shape = raw_prediction.shape = (n_samples,) - If is_multiclass is true (multiclass classification), then y_pred.shape = raw_prediction.shape = (n_samples, n_classes) Note that this corresponds to the return value of decision_function. y_true, y_pred, sample_weight and raw_prediction must either be all float64 or all float32. gradient and hessian must be either both float64 or both float32. Note that y_pred = link.inverse(raw_prediction). Specific loss classes can inherit specific link classes to satisfy BaseLink's abstractmethods. Parameters ---------- sample_weight : {None, ndarray} If sample_weight is None, the hessian might be constant. n_classes : {None, int} The number of classes for classification, else None. Attributes ---------- closs: CyLossFunction link : BaseLink interval_y_true : Interval Valid interval for y_true interval_y_pred : Interval Valid Interval for y_pred differentiable : bool Indicates whether or not loss function is differentiable in raw_prediction everywhere. need_update_leaves_values : bool Indicates whether decision trees in gradient boosting need to uptade leave values after having been fit to the (negative) gradients. approx_hessian : bool Indicates whether the hessian is approximated or exact. If, approximated, it should be larger or equal to the exact one. constant_hessian : bool Indicates whether the hessian is one for this loss. is_multiclass : bool Indicates whether n_classes > 2 is allowed. FTNcCsB||_||_d|_d|_||_ttj tjdd|_|jj |_ dS)NF) closslinkapprox_hessianconstant_hessian n_classesrnpinfinterval_y_trueinterval_y_pred)selfrrrr"6/tmp/pip-unpacked-wheel-zrfo1fqw/sklearn/_loss/loss.py__init__}szBaseLoss.__init__cCs |j|SzuReturn True if y is in the valid range of y_true. Parameters ---------- y : ndarray )rincludesr!yr"r"r#in_y_true_rangeszBaseLoss.in_y_true_rangecCs |j|S)zuReturn True if y is in the valid range of y_pred. Parameters ---------- y : ndarray )r r&r'r"r"r#in_y_pred_rangeszBaseLoss.in_y_pred_rangercCsj|dkrt|}|jdkr4|jddkr4|d}t|}t|}|dk rTt|}|jj|||||dS)aJCompute the pointwise loss value for each input. Parameters ---------- y_true : C-contiguous array of shape (n_samples,) Observed, true target values. raw_prediction : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Raw prediction values (in link space). sample_weight : None or C-contiguous array of shape (n_samples,) Sample weights. loss_out : None or C-contiguous array of shape (n_samples,) A location into which the result is stored. If None, a new array might be created. n_threads : int, default=1 Might use openmp thread parallelism. Returns ------- loss : array of shape (n_samples,) Element-wise loss function. Nrry_trueraw_prediction sample_weightloss_out n_threads)r empty_likendimshapesqueezerrloss)r!r,r-r.r/r0r"r"r#r5s  z BaseLoss.losscCs|dkr8|dkr&t|}t|}qPtj||jd}n|dkrPtj||jd}|jdkrr|jddkrr|d}|jdkr|jddkr|d}t|}t|}|dk rt|}|jj||||||dS)aCompute loss and gradient w.r.t. raw_prediction for each input. Parameters ---------- y_true : C-contiguous array of shape (n_samples,) Observed, true target values. raw_prediction : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Raw prediction values (in link space). sample_weight : None or C-contiguous array of shape (n_samples,) Sample weights. loss_out : None or C-contiguous array of shape (n_samples,) A location into which the loss is stored. If None, a new array might be created. gradient_out : None or C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) A location into which the gradient is stored. If None, a new array might be created. n_threads : int, default=1 Might use openmp thread parallelism. Returns ------- loss : array of shape (n_samples,) Element-wise loss function. gradient : array of shape (n_samples,) or (n_samples, n_classes) Element-wise gradients. Ndtyperr)r,r-r.r/ gradient_outr0) rr1r7r2r3r4rr loss_gradient)r!r,r-r.r/r8r0r"r"r#r9s.&    zBaseLoss.loss_gradientcCs|dkrt|}|jdkr4|jddkr4|d}|jdkrV|jddkrV|d}t|}t|}|dk rvt|}|jj|||||dS)aCompute gradient of loss w.r.t raw_prediction for each input. Parameters ---------- y_true : C-contiguous array of shape (n_samples,) Observed, true target values. raw_prediction : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Raw prediction values (in link space). sample_weight : None or C-contiguous array of shape (n_samples,) Sample weights. gradient_out : None or C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) A location into which the result is stored. If None, a new array might be created. n_threads : int, default=1 Might use openmp thread parallelism. Returns ------- gradient : array of shape (n_samples,) or (n_samples, n_classes) Element-wise gradients. Nrr)r,r-r.r8r0)rr1r2r3r4rrgradient)r!r,r-r.r8r0r"r"r#r: s"   zBaseLoss.gradientcCs|dkr2|dkr&t|}t|}qDt|}n|dkrDt|}|jdkrf|jddkrf|d}|jdkr|jddkr|d}|jdkr|jddkr|d}t|}t|}|dk rt|}|jj||||||dS)aCompute gradient and hessian of loss w.r.t raw_prediction. Parameters ---------- y_true : C-contiguous array of shape (n_samples,) Observed, true target values. raw_prediction : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Raw prediction values (in link space). sample_weight : None or C-contiguous array of shape (n_samples,) Sample weights. gradient_out : None or C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) A location into which the gradient is stored. If None, a new array might be created. hessian_out : None or C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) A location into which the hessian is stored. If None, a new array might be created. n_threads : int, default=1 Might use openmp thread parallelism. Returns ------- gradient : arrays of shape (n_samples,) or (n_samples, n_classes) Element-wise gradients. hessian : arrays of shape (n_samples,) or (n_samples, n_classes) Element-wise hessians. Nrr)r,r-r.r8 hessian_outr0)rr1r2r3r4rrgradient_hessian)r!r,r-r.r8r;r0r"r"r#r<>s2'       zBaseLoss.gradient_hessiancCstj|j||dd|d|dS)a{Compute the weighted average loss. Parameters ---------- y_true : C-contiguous array of shape (n_samples,) Observed, true target values. raw_prediction : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Raw prediction values (in link space). sample_weight : None or C-contiguous array of shape (n_samples,) Sample weights. n_threads : int, default=1 Might use openmp thread parallelism. Returns ------- loss : float Mean or averaged loss function. Nr+)weights)raverager5)r!r,r-r.r0r"r"r#__call__szBaseLoss.__call__cCstj||dd}dt|jj}|jjtj kr8d}n|jjrJ|jj}n |jj|}|jj tjkrjd}n|jj r||jj }n |jj |}|dkr|dkr|j |S|j t |||SdS)a#Compute raw_prediction of an intercept-only model. This can be used as initial estimates of predictions, i.e. before the first iteration in fit. Parameters ---------- y_true : array-like of shape (n_samples,) Observed, true target values. sample_weight : None or array of shape (n_samples,) Sample weights. Returns ------- raw_prediction : numpy scalar or array of shape (n_classes,) Raw predictions of an intercept-only model. rr=axis N) rr>finfor7epsr lowrZ low_inclusivehighZhigh_inclusiverclip)r!r,r.Zy_predrDZa_minZa_maxr"r"r#fit_intercept_onlys     zBaseLoss.fit_intercept_onlycCs t|S)zpCalculate term dropped in loss. With this term added, the loss of perfect predictions is zero. )rZ zeros_liker!r,r.r"r"r#constant_to_optimal_zerosz!BaseLoss.constant_to_optimal_zeroFcCsv|tjtjfkr td|d|jr2||jf}n|f}tj|||d}|jr^tjd|d}ntj|||d}||fS)auInitialize arrays for gradients and hessians. Unless hessians are constant, arrays are initialized with undefined values. Parameters ---------- n_samples : int The number of samples, usually passed to `fit()`. dtype : {np.float64, np.float32}, default=np.float64 The dtype of the arrays gradient and hessian. order : {'C', 'F'}, default='F' Order of the arrays gradient and hessian. The default 'F' makes the arrays contiguous along samples. Returns ------- gradient : C-contiguous array of shape (n_samples,) or array of shape (n_samples, n_classes) Empty array (allocated but not initialized) to be used as argument gradient_out. hessian : C-contiguous array of shape (n_samples,), array of shape (n_samples, n_classes) or shape (1,) Empty (allocated but not initialized) array to be used as argument hessian_out. If constant_hessian is True (e.g. `HalfSquaredError`), the array is initialized to ``1``. zCValid options for 'dtype' are np.float32 and np.float64. Got dtype=z instead.)r3r7order)r)r3r7) rZfloat32float64 ValueError is_multiclassremptyrZones)r!Z n_samplesr7rLr3r:Zhessianr"r"r#init_gradient_and_hessians  z"BaseLoss.init_gradient_and_hessian)N)NNr)NNNr)NNr)NNNr)Nr)N)N)__name__ __module__ __qualname____doc__need_update_leaves_valuesdifferentiablerOr$r)r*r5r9r:r<r?rHrJrrMrQr"r"r"r#r>s::    4 F 8 E  * rcs"eZdZdZdfdd ZZS)HalfSquaredErroraHalf squared error with identity link, for regression. Domain: y_true and y_pred all real numbers Link: y_pred = raw_prediction For a given sample x_i, half squared error is defined as:: loss(x_i) = 0.5 * (y_true_i - raw_prediction_i)**2 The factor of 0.5 simplifies the computation of gradients and results in a unit hessian (and is consistent with what is done in LightGBM). It is also half the Normal distribution deviance. Ncs"tjttd|dk|_dS)Nrr)superr$rrrr!r. __class__r"r#r$szHalfSquaredError.__init__)NrRrSrTrUr$ __classcell__r"r"r\r#rX srXcs4eZdZdZdZdZd fdd Zd ddZZS) AbsoluteErroraAbsolute error with identity link, for regression. Domain: y_true and y_pred all real numbers Link: y_pred = raw_prediction For a given sample x_i, the absolute error is defined as:: loss(x_i) = |y_true_i - raw_prediction_i| FTNcs(tjttdd|_|dk|_dS)NrYT)rZr$rrrrr[r\r"r#r$3szAbsoluteError.__init__cCs&|dkrtj|ddSt||dSdS)Compute raw_prediction of an intercept-only model. This is the weighted median of the target, i.e. over the samples axis=0. NrrA2)rZmedianrrIr"r"r#rH8sz AbsoluteError.fit_intercept_only)N)N rRrSrTrUrWrVr$rHr_r"r"r\r#r`"s  r`cs4eZdZdZdZdZd fdd Zd dd ZZS) PinballLossaQuantile loss aka pinball loss, for regression. Domain: y_true and y_pred all real numbers quantile in (0, 1) Link: y_pred = raw_prediction For a given sample x_i, the pinball loss is defined as:: loss(x_i) = rho_{quantile}(y_true_i - raw_prediction_i) rho_{quantile}(u) = u * (quantile - 1_{u<0}) = -u *(1 - quantile) if u < 0 u * quantile if u >= 0 Note: 2 * PinballLoss(quantile=0.5) equals AbsoluteError(). Additional Attributes --------------------- quantile : float The quantile to be estimated. Must be in range (0, 1). FTN?csFt|dtjddddtjtt|dtdd|_|dk|_ dS) NquantilerrZneither)Z target_typeZmin_valZmax_valZinclude_boundaries)rgrYT) rnumbersRealrZr$rfloatrrr)r!r.rgr\r"r#r$as zPinballLoss.__init__cCs8|dkr tj|d|jjddSt||d|jjSdS)raNdrrb)rZ percentilerrgrrIr"r"r#rHqs zPinballLoss.fit_intercept_only)Nrf)Nrdr"r"r\r#reDs recs,eZdZdZdfdd ZdddZZS) HalfPoissonLossaHalf Poisson deviance loss with log-link, for regression. Domain: y_true in non-negative real numbers y_pred in positive real numbers Link: y_pred = exp(raw_prediction) For a given sample x_i, half the Poisson deviance is defined as:: loss(x_i) = y_true_i * log(y_true_i/exp(raw_prediction_i)) - y_true_i + exp(raw_prediction_i) Half the Poisson deviance is actually the negative log-likelihood up to constant terms (not involving raw_prediction) and simplifies the computation of the gradients. We also skip the constant term `y_true_i * log(y_true_i) - y_true_i`. Ncs*tjttdtdtjdd|_dS)NrYrTF)rZr$rrrrrrr[r\r"r#r$szHalfPoissonLoss.__init__cCs"t|||}|dk r||9}|S)Nrr!r,r.termr"r"r#rJsz(HalfPoissonLoss.constant_to_optimal_zero)N)NrRrSrTrUr$rJr_r"r"r\r#rlsrlcs,eZdZdZdfdd ZdddZZS) HalfGammaLossaVHalf Gamma deviance loss with log-link, for regression. Domain: y_true and y_pred in positive real numbers Link: y_pred = exp(raw_prediction) For a given sample x_i, half Gamma deviance loss is defined as:: loss(x_i) = log(exp(raw_prediction_i)/y_true_i) + y_true/exp(raw_prediction_i) - 1 Half the Gamma deviance is actually proportional to the negative log- likelihood up to constant terms (not involving raw_prediction) and simplifies the computation of the gradients. We also skip the constant term `-log(y_true_i) - 1`. Ncs*tjttdtdtjdd|_dS)NrYrF)rZr$r rrrrrr[r\r"r#r$szHalfGammaLoss.__init__cCs$t| d}|dk r ||9}|SNr)rlogrmr"r"r#rJsz&HalfGammaLoss.constant_to_optimal_zero)N)Nror"r"r\r#rpsrpcs,eZdZdZdfdd Zd ddZZS) HalfTweedieLossaHalf Tweedie deviance loss with log-link, for regression. Domain: y_true in real numbers for power <= 0 y_true in non-negative real numbers for 0 < power < 2 y_true in positive real numbers for 2 <= power y_pred in positive real numbers power in real numbers Link: y_pred = exp(raw_prediction) For a given sample x_i, half Tweedie deviance loss with p=power is defined as:: loss(x_i) = max(y_true_i, 0)**(2-p) / (1-p) / (2-p) - y_true_i * exp(raw_prediction_i)**(1-p) / (1-p) + exp(raw_prediction_i)**(2-p) / (2-p) Taking the limits for p=0, 1, 2 gives HalfSquaredError with a log link, HalfPoissonLoss and HalfGammaLoss. We also skip constant terms, but those are different for p=0, 1, 2. Therefore, the loss is not continuous in `power`. Note furthermore that although no Tweedie distribution exists for 0 < power < 1, it still gives a strictly consistent scoring function for the expectation. N?csvtjtt|dtd|jjdkr@ttj tj dd|_ n2|jjdkr`tdtj dd|_ ntdtj dd|_ dSN)powerrYrFrT) rZr$r rjrrrvrrrrr!r.rvr\r"r#r$s   zHalfTweedieLoss.__init__cCs|jjdkrtj||dS|jjdkr8tj||dS|jjdkrTtj||dS|jj}tt|dd|d|d|}|dk r||9}|SdS)Nr)r,r.rr)rrvrXrJrlrprmaximum)r!r,r.prnr"r"r#rJs(   (z(HalfTweedieLoss.constant_to_optimal_zero)Nrt)Nror"r"r\r#rss rscs"eZdZdZdfdd ZZS)HalfTweedieLossIdentityanHalf Tweedie deviance loss with identity link, for regression. Domain: y_true in real numbers for power <= 0 y_true in non-negative real numbers for 0 < power < 2 y_true in positive real numbers for 2 <= power y_pred in positive real numbers for power != 0 y_pred in real numbers for power = 0 power in real numbers Link: y_pred = raw_prediction For a given sample x_i, half Tweedie deviance loss with p=power is defined as:: loss(x_i) = max(y_true_i, 0)**(2-p) / (1-p) / (2-p) - y_true_i * raw_prediction_i**(1-p) / (1-p) + raw_prediction_i**(2-p) / (2-p) Note that the minimum value of this loss is 0. Note furthermore that although no Tweedie distribution exists for 0 < power < 1, it still gives a strictly consistent scoring function for the expectation. Nrtcstjtt|dtd|jjdkr@ttj tj dd|_ n2|jjdkr`tdtj dd|_ ntdtj dd|_ |jjdkrttj tj dd|_ ntdtj dd|_ dSru) rZr$r rjrrrvrrrrr rwr\r"r#r$s    z HalfTweedieLossIdentity.__init__)Nrtr^r"r"r\r#rzsrzcs4eZdZdZd fdd Zd ddZddZZS) HalfBinomialLossaHalf Binomial deviance loss with logit link, for binary classification. This is also know as binary cross entropy, log-loss and logistic loss. Domain: y_true in [0, 1], i.e. regression on the unit interval y_pred in (0, 1), i.e. boundaries excluded Link: y_pred = expit(raw_prediction) For a given sample x_i, half Binomial deviance is defined as the negative log-likelihood of the Binomial/Bernoulli distribution and can be expressed as:: loss(x_i) = log(1 + exp(raw_pred_i)) - y_true_i * raw_pred_i See The Elements of Statistical Learning, by Hastie, Tibshirani, Friedman, section 4.4.1 (about logistic regression). Note that the formulation works for classification, y = {0, 1}, as well as logistic regression, y = [0, 1]. If you add `constant_to_optimal_zero` to the loss, you get half the Bernoulli/binomial deviance. Ncs*tjttddtdddd|_dS)NrrrrrrT)rZr$r rrrr[r\r"r#r$Gs zHalfBinomialLoss.__init__cCs0t||td|d|}|dk r,||9}|Srqrrmr"r"r#rJOsz)HalfBinomialLoss.constant_to_optimal_zerocCsx|jdkr"|jddkr"|d}tj|jddf|jd}|j||dddf<d|dddf|dddf<|S)a=Predict probabilities. Parameters ---------- raw_prediction : array of shape (n_samples,) or (n_samples, 1) Raw prediction values (in link space). Returns ------- proba : array of shape (n_samples, 2) Element-wise class probabilities. rrrr6N)r2r3r4rrPr7rinverse)r!r-Zprobar"r"r# predict_probaVs   zHalfBinomialLoss.predict_proba)N)N)rRrSrTrUr$rJr~r_r"r"r\r#r{,s r{csJeZdZdZdZdfdd ZddZdd d Zd d ZdddZ Z S)HalfMultinomialLossaCategorical cross-entropy loss, for multiclass classification. Domain: y_true in {0, 1, 2, 3, .., n_classes - 1} y_pred has n_classes elements, each element in (0, 1) Link: y_pred = softmax(raw_prediction) Note: We assume y_true to be already label encoded. The inverse link is softmax. But the full link function is the symmetric multinomial logit function. For a given sample x_i, the categorical cross-entropy loss is defined as the negative log-likelihood of the multinomial distribution, it generalizes the binary cross-entropy to more than 2 classes:: loss_i = log(sum(exp(raw_pred_{i, k}), k=0..n_classes-1)) - sum(y_true_{i, k} * raw_pred_{i, k}, k=0..n_classes-1) See [1]. Note that for the hessian, we calculate only the diagonal part in the classes: If the full hessian for classes k and l and sample i is H_i_k_l, we calculate H_i_k_k, i.e. k=l. Reference --------- .. [1] :arxiv:`Simon, Noah, J. Friedman and T. Hastie. "A Blockwise Descent Algorithm for Group-penalized Multiresponse and Multinomial Regression". <1311.6529>` TNcs<tjtt|dtdtjdd|_tdddd|_dS)Nr|rTFr) rZr$r rrrrrr )r!r.rr\r"r#r$szHalfMultinomialLoss.__init__cCs |j|ot|t|kSr%)rr&rallZastypeintr'r"r"r#r)sz#HalfMultinomialLoss.in_y_true_rangecCstj|j|jd}t|jj}t|jD]6}tj||k|dd||<t|||d|||<q*|j |dddf dS)zCompute raw_prediction of an intercept-only model. This is the softmax of the weighted average of the target, i.e. over the samples axis=0. r6rr@rN) rzerosrr7rCrDranger>rGrZreshape)r!r,r.outrDkr"r"r#rHs z&HalfMultinomialLoss.fit_intercept_onlycCs |j|S)a=Predict probabilities. Parameters ---------- raw_prediction : array of shape (n_samples, n_classes) Raw prediction values (in link space). Returns ------- proba : array of shape (n_samples, n_classes) Element-wise class probabilities. )rr})r!r-r"r"r#r~s z!HalfMultinomialLoss.predict_probarcCs||dkr2|dkr&t|}t|}qDt|}n|dkrDt|}t|}t|}|dk rdt|}|jj||||||dS)aKCompute gradient and class probabilities fow raw_prediction. Parameters ---------- y_true : C-contiguous array of shape (n_samples,) Observed, true target values. raw_prediction : array of shape (n_samples, n_classes) Raw prediction values (in link space). sample_weight : None or C-contiguous array of shape (n_samples,) Sample weights. gradient_out : None or array of shape (n_samples, n_classes) A location into which the gradient is stored. If None, a new array might be created. proba_out : None or array of shape (n_samples, n_classes) A location into which the class probabilities are stored. If None, a new array might be created. n_threads : int, default=1 Might use openmp thread parallelism. Returns ------- gradient : array of shape (n_samples, n_classes) Element-wise gradients. proba : array of shape (n_samples, n_classes) Element-wise class probabilities. N)r,r-r.r8 proba_outr0)rr1rrgradient_proba)r!r,r-r.r8rr0r"r"r#rs&$    z"HalfMultinomialLoss.gradient_proba)Nr)N)NNNr) rRrSrTrUrOr$r)rHr~rr_r"r"r\r#rls"  r)Z squared_errorZabsolute_errorZ pinball_lossZ poisson_lossZ gamma_lossZ tweedie_lossZ binomial_lossZmultinomial_loss)'rUrhZnumpyrZ scipy.specialrZ_lossrrrrr r r r r rrrrrrutilsrZutils._readonly_array_wrapperrZ utils.statsrrrXr`rerlrprsrzr{rZ_LOSSESr"r"r"r#s> ,    P"; @.@