src.eval.regression_evaluator

Classes

RegressionEvaluator(**kwargs)

class src.eval.regression_evaluator.RegressionEvaluator(**kwargs)

Author:: Alberto M. Esmoris Pena

Class to evaluate regression-like predictions against expected/reference classes.

Internally, the regression evaluator works with key-value maps governing the information. They are described below:

– quantities

The keys are feature names (from the original point cloud), the values are vectors with the point-wise features in the components.

– errors

The keys are the case names (renamed when cases_renames has been specified), the values are dictionaries such that:

– fname_ref: The name of the reference feature in the point cloud.
– fname_pred: The name of the predicted feature in the point cloud.
– error: A dictionary whose keys are the error names ("e", "se", "ae") and whose values are vectors of point-wise errors.

– metrics

The keys are the case names (renamed when cases_renames has been specified), the values are dictionaries whose keys are metric names and whose values are metric values.

– outers

The keys are the case names (renamed when cases_renames has been specified), the values are dictionaries whose keys are the names of the outer features and whose values are subdictionaries. These subdictionaries have as keys the error names ("e", "se", "ae") and whose values are correlation dictionaries. Each correlation dictionary has keys corresponding to correlation measurements ("pearson", "spearman") and whose values are tuples such that (name, correlation value, p-value).

– distribution

The keys are the case names (renamed when cases_renames has been specified), the values are subdictionaries. Each subdictionary has the estimator name as a key (including a special case "ref" that is not associated to the estimation/predicted value but to the reference value), the values are the vectors with the quantiles (typically percentiles).

Variables:

metrics (list of str) – The name of the metrics for overall evaluation.
cases (list of list of str) – A list whose elements are lists of at least two elements. The first element of each inner list is a string that gives the name of the reference attribute and the successive one give the name of the prediction attributes.
cases_renames (list of list of str) – A list whose elements are lists of strings for the elements of the inner lists in cases (ignoring the first one). They will be used to rename the metrics on a case-wise basis.
outer_correlations (dict of dict of dict) –
A dictionary whose keys are the names of the cases (considering case_renames if given) and whose values are other dictionaries. The children dictionaries have as keys the name of a feature in the point cloud that is called the outer feature. The value of each children dictionary is another dictionary of three entries:
```
{
    "metrics": ["E", "SE", "AE"],
    "correlations": ["pearson", "spearman"],
    "frenames": [
        "case_feat_E_r", "case_feat_E_rho",
        "case_feat_SE_r", "case_feat_SE_rho",
        "case_feat_AE_r", "case_feat_AE_rho"
    ]
}
```
outlier_filter (str or None) – The outlier filter to be applied, if any. It can be either "stdev" (to filter values that are below or above many times the standard deviation), "iqr" (to filter values that are outside [Q1-kIQR, Q3+kIQR], where k is a given value and IQR means interquartile range), "topk", "botk", "extremek" (to filter the k best, worst, and best and worst simultaneously cases), "topp", "botp", "extremep" (to filter the p percentage of best, worst, and best and worst simultaneously cases).
outlier_param (float or int) – The parameter, for those outlier filters that are based on a parameter (e.g., "stdev", "iqr”, "topk", "botp", etc.).
regression_report_path (str) – The path to write the report with the regression evaluation metrics.
outer_report_path (str) – The path to write the report with the correlation of the regression metrics and given features.
distribution_report_path (str) – The path to write the report describing the distribution of each attribute (includes references and predictions).
regression_pcloud_path (str) – The path to write the point cloud with the point-wise regression metrics.
regression_plot_path (str) – The path where the plot with the regression evaluation will be written. It represents the error on the y-axis and the predictions on the x-axis.
regression_hist2d_path – The path where the 2D histogram with the regression evaluation will be written. It represents the error on the y-axis and the predictions on the x-axis.
residual_plot_path (str) – Like the regression_plot_path but the x-axis represents the references instead of the predictions.
residual_hist2d_path (str) – Like the regression_hist2d_path but the x-axis represents the references instead of the predictions.
scatter_plot_path (str) – The path where the plot with the references on the x-axis and the predictions on the y-axis will be written.
scatter_hist2d_path (str) – The path where the 2D histogram with the references on the x-axis and the predictions on the y-axis will be written.
qq_plot_path (str) – The path where the plot with the quartiles of the references on the x-axis and the quartiles of the predictions on the y-axis will be written.
summary_plot_path (str) – The path where the plot summarizing the mean and standard deviation for each error measurement for each case.
nthreads (int) – How many threads to compute the metrics in parallel (i.e., one thread per metric at most). Note that -1 means using as many threads as available cores.
metric_funs (list of callable) – The functions (callables) that compute the metrics. They are derived from the requested metrics (names).

static extract_eval_args(spec)

Extract the arguments to initialize/instantiate a RegressionEvaluator from a key-word specification.

Parameters:: spec – The key-word specifiaction containing te arguments.
Returns:: The arguments to initialize/instantiate a RegressionEvaluator.

__init__(**kwargs)

Initialize/instantiate a RegressionEvaluator.

Parameters:: kwargs – The attributes for the RegressionEvaluator.

eval(pcloud)

Evaluate the predicted values (\(\hat{y}\)) against expected/reference values (\(y\)).

Parameters:: pcloud (PointCloud) – The point cloud to be evaluated.
Returns:: The evaluation of the regression/s.
Return type:: RegressionEvaluation.

__call__(pcloud, **kwargs)

Evaluate with extra logic that is convenient for pipeline-based execution.

See Evaluator.eval().

Parameters:: pcloud – The point cloud that must be evaluated with respect to the specified regressions and references.

generate_metric_functions()

Generate a list of callables that receive as input the reference values and a predictions to yield an aggregated error or correlation measurement. Alternatively, the functions receive a third argument known has the calculus space (cs) where intermediate values can be cached to avoid redundant computations.

Returns:: List of callables to compute error and correlation measurements from an input vector of reference values and another input vector with the predicted values. Besides, the third argument provides a cache known as calculus space (cs) to avoid redundant computations.
Return type:: list of callable

determine_error_measurements()

Determine what error measurements must be computed depending on the requested metrics and outer correlations.

Returns:: The value for the member attribute self.error_measurements, which is a list with the names of the necessary error measurements.
Return type:: list

extract_quantities(pcloud)

Extract the quantities from the point cloud

Parameters:: pcloud (PointCloud) – The point cloud from where the quantities must be extracted.
Returns:: A dictionary with the quantities. The keys are the names of the quantitative features, the values are the vectors where each component yields the value of the feature for a point.
Return type:: dict

quantify_error(quantities)

Quantify the error, squared error or absolute error as necessary.

Returns:: The dictionary with the errors. The keys correspond to the names of the regression values with a subdictionary for each. The keys for each subdictionary are "e" for the error (raw difference between expected and predicted), "se" for the squared error, and "ae" for the absolute error. Any error can be given as a np.ndarray with the error values or None if the error measurement is not necessary.
Return type:: dict

compute_metrics(quantities, errors)

Compute the aggregated error and correlation metrics.

Returns:: The dictionary with metrics. The keys correspond to the name of each case and the values are subdictionaries. Each subdictionary has as key the name of the metric and as value the metric itself.
Return type:: dict

compute_outer_correlations(quantities, errors)

Compute correlation metrics with respect to a third variable that is not necessarily involved in the error computation. For example, check how the norm of the estimated gradient norm correlates with error in the estimated Gaussian curvature.

Returns:: The dictionary with metrics. The keys correspond to the name of each case and the values are subdictionaries. Each subdictionary has as keys the names of the outer feature whose correlation must be computed. The values of the subdictionary are dictionaries whose keys give the name for the error type and whose values are pairs (correlation, p-value) with the correlation value and the associated p-value.
Return type:: dict

compute_qq_distribution(quantities)

Compute the QQ distribution considering the percentiles for each requested case.

Returns:: The dictionary with the QQ distributions. The keys correspond to the name of the case’s reference. The values are a subdictionary with a special key "ref" that contains the percentiles of the reference values. The other keys in the subdictionary are the names of the features whose percentiles were computed.
Return type:: dict

static me_fun(y, yhat, cs): Compute the Mean Error

static mse_fun(y, yhat, cs): Compute the Mean Squared Error

static rmse_fun(y, yhat, cs): Compute the Root Mean Squared Error

static mae_fun(y, yhat, cs): Compute the Mean Absolute Error

static maxe_fun(y, yhat, cs): Compute the Max Error

static maxse_fun(y, yhat, cs): Compute the Max Squared Error

static rmaxse_fun(y, yhat, cs): Compute the Root Max Squared Error

static maxae_fun(y, yhat, cs): Compute the Max Absolute Error

static mine_fun(y, yhat, cs): Compute the Min Error

static minse_fun(y, yhat, cs): Compute the Min Squared Error

static rminse_fun(y, yhat, cs): Compute the Root Min Squared Error

static minae_fun(y, yhat, cs): Compute the Min Absolute Error

static mee_fun(y, yhat, cs): Compute the Median Error

static mese_fun(y, yhat, cs): Compute the Median Squared Error

static rmese_fun(y, yhat, cs): Compute the Root Median Squared Error

static meae_fun(y, yhat, cs): Compute the Median Absolute Error

static deve_fun(y, yhat, cs): Compute the Standard Deviation of the Error

static devse_fun(y, yhat, cs): Compute the Standard Deviation of the Squared Error

static rdevse_fun(y, yhat, cs): Compute the Root of the Standard Deviation of the Squared Error

static devae_fun(y, yhat, cs): Compute the Standard Deviation of the Absolute Error

static rangee_fun(y, yhat, cs): Compute the Range of the Error

static rangese_fun(y, yhat, cs): Compute the Range of the Squared Error

static rrangese_fun(y, yhat, cs): Compute the Root of the Range of the Squared Error

static rangeae_fun(y, yhat, cs): Compute the Range of the Absolute Error

static q1e_fun(y, yhat, cs): Compute the First Quartile of Error

static q1se_fun(y, yhat, cs): Compute the First Quartile of Squared Error

static rq1se_fun(y, yhat, cs): Compute the Root of the First Quartile of Squared Error

static q1ae_fun(y, yhat, cs): Compute the First Quartile of Absolute Error

static q3e_fun(y, yhat, cs): Compute the Third Quartile of Error

static q3se_fun(y, yhat, cs): Compute the Third Quartile of Squared Error

static rq3se_fun(y, yhat, cs): Compute the Root of the Third Quartile of Squared Error

static q3ae_fun(y, yhat, cs): Compute the Third Quartile of Absolute Error

static ske_fun(y, yhat, cs): Compute the Skewness of Error

static skse_fun(y, yhat, cs): Compute the Skewness of Squared Error

static skae_fun(y, yhat, cs): Compute the Skewness of Absolute Error

static kue_fun(y, yhat, cs): Compute the Kurtosis of Error

static kuse_fun(y, yhat, cs): Compute the Kurtosis of Squared Error

static kuae_fun(y, yhat, cs): Compute the Kurtosis of Absolute Error

static pearson_fun(y, yhat, cs)

Compute the Pearson correlation coefficient.

Parameters:

y – The references.
yhat – The predictions.
cs – The computing space

Returns:

The Pearson correlation coefficient between the references and the predictions.

static spearman_fun(y, yhat, cs)

Compute the Spearman correlation coefficient.

Parameters:

y – The references.
yhat – The predictions.
cs – The computing space.

Returns:

The Spearman correlation coefficient between the references and the predictions.

filter_outliers(e, se, ae)

Filter the outlier values from the given errors.

Parameters:

e (np.ndarray or None) – The error (raw differences).
se (np.ndarray or None) – The squared error.
ae (np.ndarray or None) – The absolute error.

Returns:

A tuple with the three error vectors without the outliers.

Return type:

tuple

stdev_outlier_filter(e, se, ae): Apply a filter based on removing those values lying outside the interval centered at the mean and governed by k times the standard deviation.

iqr_outlier_filter(e, se, ae): Apply a filter based on removing those values lying outside the interval governed by the first and third quartiles expanded k times the interquartile range.

topk_outlier_filter(e, se, ae): Apply a filter based on removing the k highest errors.

botk_outlier_filter(e, se, ae): Apply a filter based on removing the k lowest errors.

extremek_outlier_filter(e, se, ae): Apply a filter based on removing the k lowest and highest errors.

topp_outlier_filter(e, se, ae): Apply a filter based on removing the p percentage of highest errors.

botp_outlier_filter(e, se, ae): Apply a filter based on removing the p percentage of lowest errors.

extremep_outlier_filter(e, se, ae): Apply a filter based on removing the p percentage of highest and lowest errors.

eval_args_from_state(state)

Obtain the arguments to call the RegressionEvaluator from the current pipeline’s state.

Parameters:: state (SimplePipelineState) – The pipeline’s state.
Returns:: The dictionary of arguments for calling RegressionEvaluator.
Return type:: dict

has_report_paths()

Check whether at least one report path has been specified.

Returns:: True if at least one report path has been specified, False otherwise.
Return type:: bool

has_plot_paths()

Check whether at least one plot path has been specified.

Returns:: True if at least one plot path has been specified, False otherwise.
Return type:: bool