U 3d-@sH ddlmZddlmZddlmZddlmZddlmZddlZ ddl m Z ddl Z ddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlm Z ddlm!Z!ddlm"Z"ddlm#Z#ddlm$Z$ddlm%Z%ddlm&Z&ddlm'Z'ddlm(Z(ddlm)Z)ddlm*Z*ddlm+Z+ddlm,Z,ddlm-Z-dd lm.Z.dd!lm/Z/dd"lm0Z0dd#lm1Z1dd$lm2Z2dd%lm3Z3dd&lm4Z4dd'lm5Z5dd(lm6Z6dd)lm7Z7dd*lm8Z8dd+lm9Z9dd,lm:Z:dd-lm;Z;dd.lmZ>dd1lm?Z?dd2lm@Z@dd3lmAZAdd4lmBZBdd5lmCZCdd6lmDZDdd7lmEZEdd8lmFZFdd9lmGZGdd:lmHZHdd;lImJZJe4e6e8e>ee:e;ee9d?d>ee'd?d>e(e)d@ZKe e"ee"dAdBee dCdDe%dEdFe=ee=dAdGe.eEeeEdCdDe0e@eBe,ee-dHdIee-dJdIe5ee-dKdJdLee,dKdMee-dKdHdLee@dKdMeeBdKdMee0dKdMee-dNdJdLee,dNdMee-dNdHdLee@dNdMeeBdNdMee0dNdMee-dOdJdLee,dOdMee-dOdHdLee@dOdMeeBdOdMee0dOdMee-dPdJdLee,dPdMee-dPdHdLee@dPdMeeBdPdMee0dPdMe$dQ+ZLdRdSZMeDeMe*dTZNe&e2e3ee3dCdDe/e#eCeeCdKdMeeCdPdMeeCdNdMeeCdOdUdVeeCdKdUdVeeCdOdWdVeeCdKdWdVeeCdJdXe!ee!dKdMee!dPdMee!dNdMe1eFeGeHdYZOePZQeQReOeQReLeQReKeQReNdZd[d\d]d^d_d`dadbdcdddeh ZSdfdgdhdidjdkdldmdndodpdqdrdsdtdudvdwdxhZTeSUeTZVdqdrdsdtdudphZWdjdldihZXdvdwdxdfdqdrdsdtdudpdldmdndohZYdydzdvdwdxdqdrdsdtdudpd{d|d}d~ddddddddddddddddadh ZZdddhZ[dddjdkdhdgdidldmdodnd`dbdedddchZ\dddddd{d|d}d~ddddddddddddddddZd[d\d]d^d_hZ]dddddddddh Z^ddddddddpd_dsddddddddddddddddhZ_dddddydzdvdwdxdqdrdtdud{d|d}d~ddddddddddddddhZ`ddddhZaddddhZbddZcddZde jefdege_ddZhe jefdege`ddZie jefdegejeQeVddZkeddZle jefdegejeQeVddZme jefdegejeLeVddZne jefdeOddZoddge jpe jpgfddge jqe jqgfddge jqe jpgfddge jpdgfddge jqdgfgZre jefdeeOseKse jefderddÄZte jefdeLse jefdere jqddHgddHdgfe jpddHgddHdgfgddƄZue jefdeLsddȄZveddʄZwedd̄Zxe jefdegejeQeVejeOdd΄Zye jefdege^e]BddЄZze jefdege^dd҄Z{e jefdege^ddԄZ|eddքZ}e jefdege]dd؄Z~e jefdege[ddڄZe jefdege[dd܄Ze jefdege[e]ddބZeddZddZe jefdegeWddZe jefdegeWeXBddZe jefdegeWddZddZe jefdegeWddZeddZe jefdegejeQejeKeaddZe jefdegejeQejeKeaeSddZe jefdegejeQejeKeaeVddZe jefdege]e\Be^BeaddZeddZe jefdege]dhddZe jefdege\e^BddZe jefdegejeOeVddZe jefdeLddZe jefde!dAfe#dAfe,dCfee-ddIdCfe0dCfe?dAfe@dCfeBdCfeDdAfg e jefdeegddZdS()partial) signature)product)chain) permutationsN)make_multilabel_classification)LabelBinarizer)type_of_target) _num_samples)check_random_state)shuffle)assert_allclose)assert_almost_equal)assert_array_equal)assert_array_less)ignore_warnings)accuracy_score)average_precision_score)balanced_accuracy_score)brier_score_loss)cohen_kappa_score)confusion_matrix)coverage_error)d2_tweedie_score)d2_pinball_score)d2_absolute_error_score) det_curve)explained_variance_score)f1_score) fbeta_score) hamming_loss) hinge_loss) jaccard_score)%label_ranking_average_precision_score)label_ranking_loss)log_loss) max_error)matthews_corrcoef)mean_absolute_error)mean_absolute_percentage_error)mean_squared_error)mean_tweedie_deviance)mean_poisson_deviance)mean_gamma_deviance)median_absolute_error)multilabel_confusion_matrix)mean_pinball_loss)precision_recall_curve)precision_score)r2_score) recall_score) roc_auc_score) roc_curve) zero_one_loss) ndcg_score) dcg_score)top_k_accuracy_score)_average_binary_scoreZvariance_weighted)Z multioutput)powergffffff?)r&r(r*r0r.r)rr3mean_normal_deviancer,r-mean_compound_poisson_deviancerrrT)adjustedF normalizecOs0t||dt||jddddtjfS)Nfloataxis)rastypesumnpZnewaxis)argskwargsrKE/tmp/pip-unpacked-wheel-zrfo1fqw/sklearn/metrics/tests/test_common.pysrM)Z samplewise)betag?weighted)averagerOrQmicromacrosamples)+rr adjusted_balanced_accuracy_scoreunnormalized_accuracy_scoreunnormalized_confusion_matrixnormalized_confusion_matrix(unnormalized_multilabel_confusion_matrix/unnormalized_multilabel_confusion_matrix_sampler r7unnormalized_zero_one_lossr"r2r4rf2_score f0.5_scorematthews_corrcoef_scoreweighted_f0.5_scoreweighted_f1_scoreweighted_f2_scoreweighted_precision_scoreweighted_recall_scoreweighted_jaccard_scoremicro_f0.5_scoremicro_f1_scoremicro_f2_scoremicro_precision_scoremicro_recall_scoremicro_jaccard_scoremacro_f0.5_scoremacro_f1_scoremacro_f2_scoremacro_precision_scoremacro_recall_scoremacro_jaccard_scoresamples_f0.5_scoresamples_f1_scoresamples_f2_scoresamples_precision_scoresamples_recall_scoresamples_jaccard_scorerc OsNt||\}}}t|t|}t||tj|tjd|fdtjgdgS)a' The dimensions of precision-recall pairs and the threshold array as returned by the precision_recall_curve do not match. See func:`sklearn.metrics.precision_recall_curve` This prevents implicit conversion of return value triple to an higher dimensional np.array of dtype('float64') (it will be of dtype('object) instead). This again is needed for assert_array_equal to work correctly. As a workaround we pad the threshold array with NaN values to match the dimension of precision and recall arrays respectively. rZconstant)Z pad_widthmodeZconstant_values)r1lenrHarraypadrFZfloat64nan)rIrJZ precisionZrecallZ thresholdsZpad_threshholdsrKrKrL(precision_recall_curve_padded_thresholdss  r})r6r1rZovr)rQZ multi_classZovo)Zmax_fpr)rr$r%unnormalized_log_lossr!rr5weighted_roc_aucsamples_roc_auc micro_roc_aucZ ovr_roc_aucZweighted_ovr_roc_auc ovo_roc_aucweighted_ovo_roc_aucpartial_roc_aucr weighted_average_precision_scoresamples_average_precision_scoremicro_average_precision_scorer#r8r9r:rrrsrtrurvrwrr[r$r#r9r8rrrrr5rrrrrr"r2r4rr]r^r6r1rrXrYr`rarbrcrdrerfrgrhrirjrkrlrmrnrorprqrZrrr:r7r%r~rWr r\r(r.r*r3rr)r0rrr_r&r=rrVr!r-r,r>rrrcCs2tt||d}||7}||7}||fS)zMake targets strictly positiverC)absmin)y1y2offsetrKrKrL_require_positive_targetssrcCs6ttBttBtBttks ttt@tks2tdSN)SYMMETRIC_METRICSNOT_SYMMETRIC_METRICSsetTHRESHOLDED_METRICS"METRIC_UNDEFINED_BINARY_MULTICLASS ALL_METRICSAssertionErrorrKrKrKrLtest_symmetry_consistency'srnamecCstd}|jdddd}|jdddd}|tkr>t||\}}|jdddd}|jdddd}t|}|tkr|tkrt||||||d|dqdstd nt||||||d|ddS) NrrNsizerz%s is not symmetricerr_msgFz This case is currently unhandled) r randintMETRICS_REQUIRE_POSITIVE_YrrMETRIC_UNDEFINED_BINARYMULTILABELS_METRICSr r)r random_statey_truey_predZ y_true_binZ y_pred_binmetricrKrKrLtest_symmetric_metric4s*rc Cstd}|jdddd}|jdddd}|tkr>t||\}}t|}tt(t||||||t d|W5QRXdS)NrrNrrz%s seems to be symmetric) r rrrrpytestraisesrr ValueError)rrrrrrKrKrLtest_not_symmetric_metricSs rc Cstd}|jdddd}|jdddd}|tkr>t||\}}t||dd\}}t,t|}t||||||d|dW5QRXdS)NrrNrrr %s is not sample order invariantr)r rrrr rrr )rrrry_true_shuffley_pred_shufflerrKrKrLtest_sample_order_invarianceesrc Cstd}|jdddd}|jdddd}|j|jd}t|||dd\}}}tD]*}t|}t||||||d|dqPtD]*}t|}t||||||d|dqt D]H}t|}t||||||d|dt||||||d|dqdS)NrrNrrrrr) r rnormalshaper rrr THRESHOLDED_MULTILABEL_METRICSMULTIOUTPUT_METRICS) rrry_scorerrZy_score_shufflerrrKrKrL7test_sample_order_invariance_multilabel_and_multioutput{sH rc Cstd}|jdddd}|jdddd}|tkr>t||\}}t|}t|}t|t|}}t|jdt|jdt |d}t |d} t |d} t |d} t t |} | ||} t | ||| d|d t | ||| d |d t | || | d |d t | ||| d |d t | ||| d |d t | || | d |d t | ||| d |d t | || | d|d t | ||| d|d t t| || W5QRXt t| | |W5QRXt t| || W5QRXt t| | |W5QRXt t| || W5QRXt t| | | W5QRX|ttBtBkrt t| | | W5QRXW5QRXdS)NrrNrrrC)rC)rCrz,%s is not representation invariant with listrz3%s is not representation invariant with np-array-1dz7%s is not representation invariant with np-array-columnz@%s is not representation invariant with mix np-array-1d and listzK%s is not representation invariant with mix np-array-1d and np-array-columnzD%s is not representation invariant with mix list and np-array-column)r rrrlistrHrzrndimreshaperrr rrrrrr)rrrrZy1_listZy2_listZy1_1dZy2_1dZ y1_columnZ y2_columnZy1_rowZy2_rowrmeasurerKrKrL&test_format_invariance_with_1d_vectorss                   rc Cs>td}|jdddd}|jdddd}tddg|}tddg|}d}ddg}tt|}|||} |} |tkrt| |d} | ||} t| | d |d | | d | d } t| | d |d |t kr0t| |d } | ||} t| | d |d | | d | d } t| | d |d W5QRXdS)NrrNrreggsspam pos_label+{0} failed string vs number invariance testrO2{0} failed string object vs number invariance test)labelsz,{0} failed string vs number invariance test) r rrHrzrCLASSIFICATION_METRICSMETRICS_WITH_POS_LABELrrformatrFMETRICS_WITH_LABELS) rrrry1_strZy2_str pos_label_strZ labels_strrmeasure_with_number metric_strmeasure_with_strmeasure_with_strobjrKrKrL7test_classification_invariance_string_vs_numbers_labelssN      rc Cstd}|jdddd}|jdddd}tddg|}d}tt|}|tkr|}|tkrnt||d}|||}|||} t || d |d || d |} t || d |d nFt t|||W5QRXt t|| d |W5QRXW5QRXdS) NrrNrrrrrrrrr)r rrHrzrrrrrrrrFrrr) rrrrrrrrrrrrKrKrL4test_thresholded_invariance_string_vs_numbers_labelsWs<     rrCrzy_true, y_scorec Cs<|tkr|g}|g}tjtdd|||W5QRXdS)Nzcontains (NaN|infinity)match)rrrr)rrrrKrKrL)test_regression_thresholded_inf_nan_inputs rc Cst|s,d}t|r&d}qHd}nd}t|rDd}nd}d|d|}tjt|d|||W5QRXdS) z{check that classification metrics raise a message mentioning the occurrence of non-finite values in the target vectors.rNaNzinfinity or a value too largerzInput z contains rN)rHisfiniteallisnananyrrr)rrrZ input_nameZunexpected_valuerrKrKrL!test_classification_inf_nan_inputsrc CsBdddgdddg}}d}tjt|d|||W5QRXdS) zocheck that classification metrics raise a message of mixed type data with continuous/binary target vectors.abg?g?333333?zJClassification metrics can't handle a mix of binary and continuous targetsrN)rrr)rrrrrKrKrL+test_classification_binary_continuous_inputs rcCsJt|}|tkrddg}nddg}t|ddD]\}}||g|gq.dS)NrCrNrrepeat)rrr)rrvaluesijrKrKrLcheck_single_samples  rcCsLt|}tddgddD].\}}}}|t||ggt||ggqdS)NrrCr)rrrHrz)rrrrklrKrKrLcheck_single_sample_multioutputsrcCs t|dSr)rrrKrKrLtest_single_samples rcCs t|dSr)rrrKrKrLtest_single_sample_multioutputsrc Csptddddgddddgddddgg}tddgddgddgg}t|}tt|||W5QRXdS)NrCrrHrzrrrr)rrrrrKrKrL(test_multioutput_number_of_output_differs ( rcCstd}|jdddd}|jdddd}t|}|||}tdD]D}||jd}t||dd|f|dd|f|d|dqBdS) NrrN)rrrrCz'%s is not dimension shuffling invariantr)r uniformrrangeZ permutationrr )rrrrrerror_permrKrKrL=test_multioutput_regression_invariance_to_dimension_shufflings   rcCsd}d}td|d|dd\}}td|d|dd\}}t|dg|gg}t|dg|gg}t|}t|}t|}t|}dd|D} d d|D} tD]t} t| } t| t rd | _ | | _ | ||} t | ||| d | d t | | | | d | d t | ||| d| d qdS)Nr2rCrT n_features n_classesr n_samplesallow_unlabeledcSsg|] }t|qSrKr.0rrKrKrL -sz=test_multilabel_representation_invariance..cSsg|] }t|qSrKrrrKrKrLr.stmpzO%s failed representation invariance between dense and sparse indicator formats.rz\%s failed representation invariance between dense array and list of list indicator formats.zW%s failed representation invariance between dense and list of array indicator formats.)rrHvstackspZ coo_matrixrrr isinstancer __module____name__r r)rrrrrZy1_sparse_indicatorZy2_sparse_indicatorZy1_list_array_indicatorZy2_list_array_indicatorZy1_list_list_indicatorZy2_list_list_indicatorrrrrKrKrL)test_multilabel_representation_invariancesf        rc Csndgdgddggdddgggdgtjgddggddg}t|}|D]$}tt|||W5QRXqDdS)NrCrNrrK)rrCobjectdtyper)rZmultilabel_sequencesrseqrKrKrL+test_raise_value_error_multilabel_sequencesZs r c Csd}d}td}|jd||fd}|jd||fd}|j|jd}t|}|tkrV|n|}|||dd} |||dd} td| dd d t| | |d |d dS) NrNrrrTr@F0We failed to test correctly the normalize optionr Failed with )r rrrrrrr rrrrrrrmetricspredmeasure_normalizedmeasure_not_normalizedrKrKrL+test_normalize_option_binary_classificationks(rc Csd}d}td}|jd||fd}|jd||fd}|j||fd}t|}|tkrX|n|}|||dd} |||dd} td| dd d t| | |d |d dS) NrrrrTr@Fr r rr)r rrrrrr rrKrKrL/test_normalize_option_multiclass_classifications(rc Csd}d}td}td|dd|d\}}td|dd|d\}}|j|jd}|dg|7}|dg|7}t|}|tkrz|n|} ||| dd} ||| d d} td | dd d t| | |d |d dS)NrdrrCT)rrrrrrr@Fr r rr)r rrrrrrr ) rrrrrrrrrrrrrKrKrL/test_normalize_option_multilabel_classificationsD  rc s\j\}}||dd}t|fddt|D||dd} t| ||dd} t| t|tjdtd} t| dkr||dd} t| tj|| d n||dd} t| d|r||d d} t| tfd dt|Dt t ||d dW5QRXt t ||d dW5QRXdS)NrRcs.g|]&}dd|fdd|fqSrrKrrry_pred_binarizey_true_binarizerKrLrsz$_check_averaging..rSrTr)rEr rP)weightsrUcsg|]}||qSrKrKrrrKrLrsunknowngarbage) rr rZravelrHZmeanrGintrQrrr)rrrrr is_multilabelrrZ label_measureZ micro_measureZ macro_measurerZweighted_measureZsample_measurerKrrL_check_averagingsF     r!cCsZt|d}t|}|tkr2t||||||n$|tkrNt||||||ntddS)NZ multilabelz2Metric is not recorded as having an average option)r startswithrMETRICS_WITH_AVERAGINGr!"THRESHOLDED_METRICS_WITH_AVERAGINGr)rrrrrrr rrKrKrLcheck_averagings*r%c Cszd\}}td}|jd||fd}|jd||fd}|j||fd}t|}||}||} t||||| |dS)N)rrrr)r rrrZfitZ transformr%) rrrrrrrZlbrrrKrKrLtest_averaging_multiclasss   r&c Cshd\}}td|d|dd\}}|dd}|dd}tdjd|fd}|}|} t||||| |dS) N)(rrCrFrrrr)rr rr%) rrrryrrrrrrKrKrLtest_averaging_multilabel.s   r)cCs<td}td}td}|}|}t||||||dSNrr)rHzerosr%rrrrrrrKrKrL$test_averaging_multilabel_all_zeroesCs    r.cCs>td}td}|}|}ddd}t|||||dddS)Nr+rTcSstt|||Sr)r;r2)rrrQrKrKrLrMTs z=test_averaging_binary_multilabel_all_zeroes..T)r )rT)rHr,r!)rrrrZ binary_metricrKrKrL+test_averaging_binary_multilabel_all_zeroesNs   r/cCs<td}td}td}|}|}t||||||dSr*)rHonesr%r-rKrKrL"test_averaging_multilabel_all_onesas    r1c Cstjd}|jddt|d}|dkr4t|ddn|}|||dd}t||||tjt|ddd |d ||||d}t t "t||t d |||fW5QRX|||| d}t||d |||fd |tj ||dd tj ||dd dd} t|| d|d |ddd} t|} d| ddd<|ddd} |ddd} || | | d}|||| d}t||d|||fd |dsdD]&}t||||||dd|d qdt|t|t|d}tj t |d|||t||gdW5QRXdS)NrrC rr:)r) sample_weight)rzAFor %s sample_weight=None is not equivalent to sample_weight=onesrzQUnweighted and weighted scores are unexpectedly almost equal (%s) and (%s) for %szVWeighted scores for array and list sample_weight input are not equal (%s != %s) for %srDz.Weighting %s is not equal to repeating samplesrNzeZeroing weights does not give the same result as removing the corresponding samples (%s != %s) for %sZ unnormalized)rNrz/%s sample_weight is not invariant under scalingzJFound input variables with inconsistent numbers of samples: \[{}, {}, {}\]r)rHrandom RandomStaterryrr r0rrrrtolistrcopyr"rr Zhstack)rrrrrngr3Zunweighted_scoreZweighted_scoreZweighted_score_listZrepeat_weighted_scoreZsample_weight_subsetZsample_weight_zeroedZ y1_subsetZ y2_subsetZweighted_score_subsetZweighted_score_zeroedZscaling error_messagerKrKrLcheck_sample_weight_invariancels        r:cCsBd}td}|j|fd}|j|fd}t|}t||||dS)Nrrr)r random_samplerr:)rrrrrrrKrKrL(test_regression_sample_weight_invariances r<cCspd}td}|jdd|fd}|jdd|fd}|j|fd}t|}|tkr^t||||nt||||dS)NrrrNr)r rr;rrr:)rrrrrrrrKrKrL$test_binary_sample_weight_invariances r=c Csd}td}|jdd|fd}|jdd|fd}|j|dfd}t|}|tkrt| }||jdddd}t ||||nt ||||dS)NrrrrrrDrC) r rr;rrrHexprGrr:) rrrrrrrtempZ y_score_normrKrKrL(test_multiclass_sample_weight_invariances  r@c Cstd}tdddddd\}}tdddddd\}}t||g}t||g}|jdd|jd}t|}|tkrt||||nt||||dS) NrrCr2rFrrr) r rrHrrrrrr:) rrrZyaZybrrrrrKrKrL(test_multilabel_sample_weight_invariances,   rAc Cstddddgddddgg}tddddgddddgg}tdddg}tdddg}tddddg}tj|dd\}}tD]f}||g||gfD]P\}} |tkr| jdkrqt|} | || |dd} | || dd} t| | |qqdS) NrCrrNrT)Zreturn_inverse)rrQrR)rHrzuniquer#rrrr) Zy_true_multilabelZy_pred_multilabelZy_true_multiclassZy_pred_multiclassrrZinverse_labelsrrrrZ score_labelsscorerKrKrLtest_no_averaging_labels+s  rDc Cstd}d\}}|jdd||fd}|jdd||fd}t|}|||}tt||D]8}|dd|f} |dd|f} || | } t|| qXdS)NrrrrNr)r rrrrr rrrrrrrrCr y_score_perm y_true_perm current_scorerKrKrL-test_multilabel_label_permutations_invarianceEs  rJc Cstd}d\}}|jdd||fd}|j|jd}d||ddkdf<d||ddkdf<t|}|||}tt||D]\}|dd|f} |dd|f} || | } |tkrt | st | dkst q~t || q~dS)NrrErNrrCrg.A) r rrrrGrrrr)rHrrrrFrKrKrL?test_thresholded_multilabel_multioutput_permutations_invarianceZs   rKcCsd\}}td}|||}t| }||jdddd}|jd||d}t|}|||}tt ||D]T} tj |t d} t || t | <|dd| f} t| |} || | } t|| qpdS)N)rrrrrDrCrr)r ZrandrHr>rGrrrrrr,rZarangerZtaker)rrrrrr?rrrCrZ inverse_permrGrHrIrKrKrL.test_thresholded_metric_permutation_invariance{s     rL metric_namec Csltjd}tjdgddgdtd}|jdd|jd}d }tjt |d t |||W5QRXdS) N*rrrrNrrrz7Labels in y_true and y_pred should be of the same type.r) rHr4r5rzrrrrr TypeErrorr)rMr8rrrrKrKrL"test_metrics_consistent_type_errors  rPzmetric, y_pred_threshold dtype_y_strc Cstjd}tjdgddgd|d}|jdd|jd}|sVtjddg|d|}d }d }t|jd j}|d krz|n|} t j t | d |||W5QRXdS)NrNrrrrNrrrzy_true takes value in {'eggs', 'spam'} and pos_label is not specified: either make y_true take value in {0, 1} or {-1, 1} or pass pos_label explicitzHpos_label=1 is not a valid label. It should be one of \['eggs', 'spam'\]rrCr) rHr4r5rzrrr parametersdefaultrrr) rZy_pred_thresholdrQr8rrZerr_msg_pos_label_NoneZerr_msg_pos_label_1Zpos_label_defaultrrKrKrL test_metrics_pos_label_error_strs rT) functoolsrinspectr itertoolsrrrZnumpyrHZ scipy.sparsesparserrZsklearn.datasetsrZsklearn.preprocessingrZsklearn.utils.multiclassr Zsklearn.utils.validationr r Z sklearn.utilsr Zsklearn.utils._testingr rrrrZsklearn.metricsrrrrrrrrrrrrrrr r!r"r#r$r%r&r'r(r)r*r+r,r-r.r/r0r1r2r3r4r5r6r7r8r9r:Zsklearn.metrics._baser;ZREGRESSION_METRICSrr}Z CURVE_METRICSrdictrupdaterZMETRIC_UNDEFINED_MULTICLASSunionrr#r$rrZMETRICS_WITH_NORMALIZE_OPTIONrrrrrZMETRICS_WITHOUT_SAMPLE_WEIGHTrrrmarkZ parametrizesortedrrrrrrrrinfr|Zinvalids_nan_infrrrrrrrrrrrr rr intersectionrr!r%r&r)r.r/r1r:r<r=r@rArDrJrKrLrPstrrrTrKrKrKrLs~                                                           %                                 9         "     %#"%     * v 5  +            K     / 6      a