src.eval.classification_evaluator

Classes

ClassificationEvaluator(**kwargs)

class src.eval.classification_evaluator.ClassificationEvaluator(**kwargs)
Author:

Alberto M. Esmoris Pena

Class to evaluate classification-like predictions against expected/reference classes.

Variables:

  • metrics (list of str) – The name of the metrics for overall evaluation.

  • class_metrics (list) – The name of the metrics for the class-wise evaluation.

  • class_names (list) – The name for each class.

  • metricf (list) – The functions to compute each metric. The metric \(j\) will be computed as \(f_j(y, \hat{y})\).

  • class_metricf (list) – The function to compute class-wise metrics. The metric \(j\) for class \(i\) will be computed as \(f_j(y, \hat{y}, i)\).

  • report_path (str) – The path to write the global evaluation report.

  • class_report_path (str) – The path to write the class-wise evaluation report.

  • confusion_matrix_report_path (str) – The path to write the confusion matrix report.

  • confusion_matrix_plot_path (str) – The path to write the plot representing the confusion matrix.

  • confusion_matrix_normalization_strategy (str or None) – The type of normalization strategy to be applied to the confusion matrix when plotting it. Either None or a strign from ["row", "col", "full"].

  • class_distribution_report_path (str) – The path to write the class distribution report.

  • class_distribution_plot_path (str) – The path to write the plot representing the class distribution.

  • ignore_classes (list of str) – The list of classes that must be ignored when computing the evaluations. In other words, those points that are labeled (not predicted) as one of the ignored classes will not be considered when calculating the evaluation metrics.

  • ignore_predictions (bool) – Whether to ignore the classes also in the predictions (False) or not (True, default). Note that, in the general case, it is not recommended to ignore the classes also in the predictions.

  • nthreads (int) – How many threads to compute the metrics in parallel (i.e., one thread per metric at most). Note that -1 means using as many threads as available cores.

static extract_eval_args(spec)

Extract the arguments to initialize/instantiate a ClassificationEvaluator from a key-word specification.

Parameters:

spec – The key-word specification containing the arguments.

Returns:

The arguments to initialize/instantiate a ClassificationEvaluator.

__init__(**kwargs)

Initialize/instantiate a ClassificationEvaluator.

Parameters:

kwargs – The attributes for the ClassificationEvaluator.

eval(yhat, y=None, **kwargs)

Evaluate predicted classes (\(\hat{y}\)) against expected/reference classes (\(y\)).

Parameters:

  • yhat – The predictions to be evaluated.

  • y – The expected/reference values to evaluate the predictions against.

Returns:

The evaluation of the classification.

Return type:

ClassificationEvaluation

__call__(x, **kwargs)

Evaluate with extra logic that is convenient for pipeline-based execution.

See src.eval.evaluator.Evaluator.eval().

static metrics_from_names(names)

Obtain a list of metrics that can be evaluated for vectors of classes y (expected), and yhat (predicted).

Parameters:

names – The names of the metrics. Currently supported metrics are Overall Accuracy “OA”, Mean Accuracy “mAcc”, Precision “P”, Recall “R”, F1 score “F1”, Intersection over Union “IoU”, weighted Precision “wP”, weighted Recall “wR”, weighted F1 score “wF1”, weighted Intersection over Union “wIoU”, Matthews Correlation Coefficient “MCC”, and Kohen’s Kappa score “Kappa”.

Returns:

List of metrics such that metric_i(y, yhat) can be invoked.

Return type:

list

static class_metrics_from_names(names)

Obtain a list of class-wise metrics that can be evaluated for vectors of classes \(\pmb{y}\) (expected), and \(\pmb{\hat{y}}\) (predicted).

Parameters:

names – The names of the class-wise metrics. Currently supported class-wise metrics are Accuracy “Acc”, Precision “P”, Recall “R”, F1 score “F1”, and Intersection over Union “IoU”.

Returns:

List of metrics such that metric_i(y, yhat) can be invoked.

Return type:

list

static get_indices_from_names(all_names, target_names)

Find the indices of the target names with respect to all names.

Parameters:

  • all_names (list) – The list of all names.

  • target_names (list) – The list of target names.

Returns:

The indices of target_names in the all_names list

Return type:

np.ndarray of int

static find_ignore_mask(y, indices)

Find the boolean ignore mask (True to ignore, False otherwise).

For any point whose class (given by y) matches any target index (given by indices) a true must be stored in the corresponding element of the boolean mask.

Parameters:

  • y (np.ndarray) – A vector of point-wise classes.

  • indices (np.ndarray) – The class indices to search for.

Returns:

The boolean mask specifying what points must be ignored.

Return type:

np.ndarray of bool

static remove_indices(y, mask)

Preserve only those points for which the boolean mask is False (True means it must be ignored).

Parameters:

  • y – The points to be filtered by the mask.

  • mask – The mask defining the removal filter.

Returns:

The input points without the ignored ones.