.. _Evaluators page: Evaluators ************ Evaluators are components that can be used to evaluate the data, the model, or the predictions. The evaluations are typically represented through text reports, point clouds, and plots. An :class:`.Evaluator` component is typically used inside pipelines to assess the performance of a machine learning model or to understand the insights of a neural network. Readers should be familiar with :ref:`pipelines ` to understand how to include evaluators in their workflows. .. _Classification evaluator section: Classification evaluator ========================= The :class:`.ClassificationEvaluator` assumes there is a labeled point cloud and that some predictions have been computed for that point cloud. It will consider the predictions and reference labels at the current pipeline's state and will evaluate them in different ways. A :class:`.ClassificationEvaluator` can be defined inside a pipeline using the JSON below: .. code-block:: json { "eval": "ClassificationEvaluator", "class_names": ["Ground", "Vegetation", "Building", "Urban furniture", "Vehicle"], "metrics": ["OA", "P", "R", "F1", "IoU", "wP", "wR", "wF1", "wIoU", "MCC", "Kappa"], "class_metrics": ["P", "R", "F1", "IoU"], "report_path": "*report/global_eval.log", "class_report_path": "*report/class_eval.log", "confusion_matrix_report_path" : "confusion_matrix.log", "confusion_matrix_plot_path" : "confusion_matrix.svg", "confusion_matrix_normalization_strategy": null, "class_distribution_report_path": "class_distribution.log", "class_distribution_plot_path": "class_distribution.svg", "nthreads": 1 } The JSON above defines a :class:`.ClassificationEvaluator` that will consider many metrics, from the overall accuracy to the Cohen's kappa score, to evaluate the predicted point clouds. Some metrics will also be considered to compute class-wise scores. On top of that, the confusion matrix and the distribution of the points among the classes will be analyzed. All the evaluations can be exported as a report, typically a text file containing the data or a plot for quick visualization. **Arguments** .. _Classification evaluator class_names: -- ``class_names`` A list with the names for the classes. These names will be used to represent the classes in the plots and the reports. .. _Classification evaluator metrics: -- ``metrics`` The metrics to evaluate the classification. Supported class metrics are: * ``"OA"`` Overall accuracy. * ``"P"`` Precision. * ``"R"`` Recall. * ``"F1"`` F1 score (harmonic mean of precision and recall). * ``"IoU"`` Intersection over union (also known as Jaccard index). * ``"wP"`` Weighted precision (weights by the number of true instances for each class). * ``"wR"`` Weighted recall (weights by the number of true instances for each class). * ``"wF1"`` Weighted F1 score (weights by the number of true instances for each class). * ``"wIoU"`` Weighted intersection over union (weights by the number of true instances for each class). * ``"MCC"`` Matthews correlation coefficient. * ``"Kappa"`` Cohen's kappa score. .. _Classification evaluator class_metrics: -- ``class_metrics`` The metrics to evaluate the classification in a class-wise way. Supported class metrics are: * ``"P"`` Precision. * ``"R"`` Recall. * ``"F1"`` F1 score (harmonic mean of precision and recall). * ``"IoU"`` Intersection over union (also known as Jaccard index). .. _Classification evaluator ignore_classes: -- ``ignore_classes`` Optional list of classes to be ignored when computing the evaluation metrics. Any point whose label matches a class in this list will be excluded. The classes must be given as a list of strings that matches those from ``class_names``, i.e., ``ignore_classes`` :math:`\subseteq` ``class_names``. .. _Classification evaluator ignore_predictions: -- ``ignore_predictions`` Whether to also ignore the classes in the predictions (``true``) or not (``false``, default). Note that it is strongly recommended to avoid ignoring the predictions, as it can lead to wrong interpretations of the model performance. Yet, it can be useful in some circumstances. .. _Classification evaluator report_path: -- ``report_path`` Path to write the evaluation of the classification to a text file. .. _Classification evaluator class_report_path: -- ``class_report_path`` Path to write the class-wise evaluation of the classification to a text file. .. _Classification evaluator confusion_matrix_report_path: -- ``confusion_matrix_report_path`` Path to write the confusion matrix to a text file. .. _Classification evaluator confusion_matrix_plot_path: -- ``confusion_matrix_plot_path`` Path to write the plot representing a confusion matrix to a file. .. _Classification evaluator confusion_matrix_normalization_strategy: -- ``confusion_matrix_normalization_strategy`` The strategy to normalize the confusion matrix (when plotting it), if any. It can be either ``"row"`` to apply a row-wise normalization (i.e., by references, recall), ``"col"`` to apply a column-wise normalization (i.e., by predictions, precision), or ``"full"`` to normalize considering the total number of samples. When not given (i.e., ``null``), no normalization will be applied. .. _Classification evaluator class_distribution_report_path: -- ``class_distribution_report_path`` Path to write the class distribution report to a text file. .. _Classification evaluator class_distribution_plot_path: -- ``class_distribution_plot_path`` Path to write the plot representing the class distribution to a text file. .. _Classification evaluator nthreads: -- ``nthreads`` The number of threads to use for the parallel computation of evaluation metrics. At most one thread per requested metric can be used. If set to ``-1`` then as many threads as available cores will be used. **Output** The output is illustrated considering the March 2018 point clouds from the `Hessigheim dataset `_ to compute the classification's evaluation. The table below represents the confusion matrix exported as a CSV report. The rows represent the true labels, while the columns represent the predictions. .. csv-table:: :file: ../csv/classif_eval_confmat.csv :widths: 20 20 20 20 20 :header-rows: 1 The image below represents the confusion matrix as a figure. The information in the image is the same than the one in the table but in a different format. .. figure:: ../img/classif_eval_confmat.png :scale: 18 :alt: Figure representing a confusion matrix The confusion matrix exported by the classification evaluator. .. _Advanced classification evaluator section: Advanced classification evaluator =================================== The :class:`.AdvancedClassificationEvaluator` uses the :class:`.ClassificationEvaluator` many times (see :ref:`documentation `), one per node in the analysis domain. It can be defined inside a pipeline using the JSON below: .. code-block:: json { "eval": "AdvancedClassificationEvaluator", "class_names": ["vegetation", "other"], "metrics": ["OA", "P", "R", "F1", "IoU", "wP", "wR", "wF1", "wIoU", "MCC", "Kappa"], "class_metrics": ["P", "R", "F1", "IoU"], "report_path": "*report/global_eval.log", "plot_path": "*report/global_plot.svg", "class_report_path": "*report/class_eval.log", "class_plot_path": "*report/class_eval.svg", "confusion_matrix_report_path" : "*report/confusion_matrix.log", "confusion_matrix_plot_path" : "*report/confusion_matrix.svg", "confusion_matrix_normalization_strategy": "row", "class_distribution_report_path": "*report/class_distribution.log", "class_distribution_plot_path": "*report/class_distribution.svg", "nthreads": 1, "domain_name": "PWE cut", "filters": [ { "name": "pwe0_1", "x": 0.1, "conditions": [ { "value_name": "classification", "condition_type": "equals", "value_target": 2, "action": "discard" }, { "value_name": "PointWiseEntropy", "condition_type": "less_than_or_equal_to", "value_target": 0.1, "action": "preserve" } ] }, { "name": "pwe0_3", "x": 0.3, "conditions": [ { "value_name": "classification", "condition_type": "equals", "value_target": 2, "action": "discard" }, { "value_name": "PointWiseEntropy", "condition_type": "less_than_or_equal_to", "value_target": 0.3, "action": "preserve" } ] }, { "name": "pwe0_6", "x": 0.6, "conditions": [ { "value_name": "classification", "condition_type": "equals", "value_target": 2, "action": "discard" }, { "value_name": "PointWiseEntropy", "condition_type": "less_than_or_equal_to", "value_target": 0.6, "action": "preserve" } ] }, { "name": "pwe0_9", "x": 0.9, "conditions": [ { "value_name": "classification", "condition_type": "equals", "value_target": 2, "action": "discard" }, { "value_name": "PointWiseEntropy", "condition_type": "less_than_or_equal_to", "value_target": 0.9, "action": "preserve" } ] }, { "name": "pwe1", "x": 1.0, "conditions": [ { "value_name": "classification", "condition_type": "equals", "value_target": 2, "action": "discard" }, { "value_name": "PointWiseEntropy", "condition_type": "less_than_or_equal_to", "value_target": 1.0, "action": "preserve" } ] } ] } The JSON above defines a :class:`.AdvancedClassificationEvaluator` that will consider many metrics, for a global evaluation and a class-wise evaluation as well. It will also consider the confusion matrices and the class distributions for each of the four requested nodes in the domain of the ``"PointWiseEntropy"``. The confusion matrices will be normalized by references (row-wise). All the evaluations will be exported as a text report but also plots will be generated. **Arguments** -- ``class_names`` See :ref:`classification evaluator documentation ` . -- ``metrics`` See :ref:`classification evaluator documentation ` . -- ``class_metrics`` See :ref:`classification evaluator documentation ` . -- ``ignore_classes`` See :ref:`classification evaluator documentation ` . -- ``report_path`` See :ref:`classification evaluator documentation ` . -- ``class_report_path`` See :ref:`classification evaluator documentation ` . -- ``confusion_matrix_report_path`` See :ref:`classification evaluator documentation ` . -- ``confusion_matrix_plot_path`` See :ref:`classification evaluator documentation ` . -- ``confusion_matrix_normalization_strategy`` See :ref:`classification evaluator documentation ` . -- ``class_distribution_report_path`` See :ref:`classification evaluator documentation ` . -- ``class_distribution_plot_path`` See :ref:`classification evaluator documentation ` . -- ``nthreads`` See :ref:`classification evaluator documentation ` . -- ``domain_name`` The name that must be given to the domain (it will be used in the headers of CSV-like output but also in the titles and labels of the generated figures). -- ``filters`` A list with the filters that must be applied, each filter must have a format like shown below: .. code-block:: json { "name": "filter_name", "x": 0.5, "conditions": [ { "value_name": "feature_name", "condition_type": "less_than_or_equal_to", "value_target": 0.5, "action": "preserve" } ] } -- ``name`` A name for the filter so it can be identified internally when reporting errors. -- ``x`` The value in the domain that corresponds to this node. .. _Advanced evaluator conditions: -- ``conditions`` A list with many conditions specified as dictionaries. -- ``value_name`` The name of the feature (must exist in the point cloud) that must be considered by the filter. It can also be the standard ``"classification"`` attribute. -- ``condition_type`` The type of relational that governs the condition. It is one of those that can be specified as :ref:`the condition types of an advanced input `. -- ``value_target`` See :ref:`value target of an advanced input `. -- ``action`` Whether to ``"preserve"`` the points that satisfy the condition or to ``"discard"`` them. **Output** The output is illustrated using the `PNOA-II dataset `_ (in the region of Galicia, northwest Spain). It corresponds to an advanced evaluation like in the example shown above considering the PointWiseEntropy but analyzing linearly spaced nodes from :math:`0.1` to :math:`1.0` with a step of :math:`0.1`. The table below represents the global evaluation metrics exported as a CSV report. The rows represent the metrics for the different point-wise entropy cuts. For each analysis the cut value, the number of points and the many evaluation and correlation metrics are included in the output. .. csv-table:: :file: ../csv/advanced_classif_global_eval.csv :widths: 6 8 7 7 7 7 7 7 7 :header-rows: 1 The image below represents the class-wise evaluation on different nodes of the point-wise entropy domain. It can be seen that, no matter the metric and the class, the point-wise entropy is a good uncertainty measurement because the greater it is, the lower the evaluation metric. .. figure:: ../img/advanced_classif_classwise_eval.png :scale: 42 :alt: Figure representing the advanced class-wise evaluation. The class-wise evaluation as a function of the point-wise entropy exported by the advanced classification evaluator. .. _Classification uncertainty evaluator section: Classification uncertainty evaluator ====================================== The :class:`.ClassificationUncertaintyEvaluator` can be used to get insights on what points are more problematic for a given model when solving a particular point-wise classification task. The evaluation will be more detailed when there is more data available (e.g., reference labels) but it can also be computed solely from the predicted probabilities. A :class:`.ClassificationUncertaintyEvaluator` can be defined inside a pipeline using the JSON below: .. code-block:: json { "eval": "ClassificationUncertaintyEvaluator", "class_names": ["Ground", "Vegetation", "Building", "Urban furniture", "Vehicle"], "include_probabilities": true, "include_weighted_entropy": true, "include_clusters": true, "weight_by_predictions": false, "num_clusters": 10, "clustering_max_iters": 128, "clustering_batch_size": 1000000, "clustering_entropy_weights": true, "clustering_reduce_function": "mean", "gaussian_kernel_points": 256, "report_path": "uncertainty/uncertainty.las", "plot_path": "uncertainty/" } The JSON above defines a :class:`.ClassificationUncertaintyEvaluator` that will export a point cloud and many plots to the `uncertainty` directory. **Arguments** -- ``class_names`` A list with the names for the classes. These names will be used to represent the classes in the plots and the reports. -- ``ignore_classes`` Optional list of classes to be ignored when computing the uncertainty metrics. Any point whose label matches a class in this list will be excluded. The classes must be given as a list of strings that matches those from ``class_names``, i.e., ``ignore_classes`` :math:`\subseteq` ``class_names``. -- ``probability_eps`` A value representing the zero. It can be useful to avoid NaNs when computing the logarithms of the likelihoods, i.e., :math:`\log_2(0)`. If it is exactly zero, then logarithms of zero might arise. Otherwise, the zeroes will be replaced by this value or the minimum greater than zero likelihood, whatever is smaller. -- ``include_probabilities`` Whether to include the probabilities in the output point cloud (True) or not (False). -- ``include_weighted_entropy`` Whether to include the weighted entropy in the evaluation (True) or not (False). The weighted entropy considers either the distribution of reference or predicted labels to compensate for unbalanced class distributions. -- ``include_clusters`` Whether to include the cluster-wise entropy in the evaluation (True) or not (False). Note that the cluster-wise entropy might take too long to compute depending on how it is configured. -- ``weight_by_predictions`` Whether to compute the weighted entropy considering the predictions instead of the reference labels (True) or not (False, by default). -- ``num_clusters`` How many clusters must be computed for the cluster-wise entropy. -- ``clustering_max_iters`` How many iterations are allowed at most when computing the clustering algorithm (KMeans). -- ``clustering_batch_size`` How many points per batch must be considered when computing the batch KMeans. -- ``clustering_entropy_weights`` Whether to use point-wise entropy as the sample weights for the KMeans clustering (True) or not (False). -- ``clustering_reduce_function`` What function must be used to reduce all the entropies in a given cluster to a single one that will be assigned to all points in the cluster. Supported reduce functions are: * ``"mean"`` Select the mean entropy value. * ``"median"`` Select the median of the entropy distribution. * ``"Q1"`` Select the first quartile of the entropy distribution. * ``"Q3"`` Select the third quartile of the entropy distribution. * ``"min"`` Select the min entropy value. * ``"max"`` Select the max entropy value. -- ``gaussian_kernel_points`` How many points will be considered to evaluate each gaussian kernel density estimation. -- ``report_path`` Path to write the point cloud with the computed uncertainty metrics. -- ``plot_path`` Path to the directory where the many plots representing the computed uncertainty metrics will be written. **Output** The output is illustrated considering the March 2018 point clouds from the `Hessigheim dataset `_ to compute the classification's uncertainty evaluation. Below, an example of one of the figures that can be generated with the :class:`.ClassificationUncertaintyEvaluator`. It clearly illustrates that the point-wise classification of vehicles is problematic. .. figure:: ../img/pwise_entropy_fig.png :scale: 14 :alt: Figure with four plots representing the point-wise entropy. Visualization of the point-wise entropy outside the point cloud. Below, an example of the point cloud representing the uncertainty metrics. In the general case, it can be seen that a high class ambiguity is associated with misclassified points. Thus, even in the absence of labeled point clouds, the uncertainty metrics can be used to understand the problems of a model when classifying previously unseen data. .. figure:: ../img/uncertainty_pcloud.png :scale: 36 :alt: Figure representing the class ambiguity metric and the success/fail point-wise mask in the 3D point cloud. Visualization of a point cloud representing the class ambiguity and the success/fail point-wise mask on previously unseen data, respectively. Red means failed classification and gray means successfully classified. Regression evaluator ======================== The :class:`.RegressionEvaluator` assesses the relationship between numerical quantities assuming there is a reference one, therefore enabling error measurements. A :class:`.RegressionEvaluator` can be defined inside a pipeline using the JSON below: .. code-block:: json { "eval": "RegressionEvaluator", "metrics": [ "MSE", "RMSE", "MAE", "MaxSE", "RMaxSE", "MaxAE", "MinSE", "RMinSE", "MinAE", "MeSE", "RMeSE", "MeAE", "DevSE", "RDevSE", "DevAE", "RangeSE", "RRangeSE", "RangeAE", "Q1SE", "RQ1SE", "Q1AE", "Q3SE", "RQ3SE", "Q3AE", "SkSE", "SkAE", "KuSE", "KuAE", "Pearson", "Spearman" ], "cases": [ ["sage_gauss_curv", "full_gauss_r0_4", "cc_Gauss_curv_r0_4"], ["sage_mean_curv", "full_mean_r0_4", "cc_Mean_curv_r0_4"], ["sage_shape_index", "shape_index_r0_4"], ["sage_umbildev", "umbilic_dev_r0_4"], ["sage_minabscurv", "minabscurv_r0_4"], ["sage_maxabscurv", "maxabscurv_r0_4"] ], "cases_renames":[ ["GaussC", "ccGaussC"], ["MeanC", "ccMeanC"], ["Shpidx"], ["Umbdev"], ["minabsC"], ["maxabsC"] ], "outer_correlations": { "GaussC": { "abs_algdist_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["GaussC_absad_SE_r", "GaussC_absad_SE_rho", "GaussC_absad_AE_r", "GaussC_absad_AE_rho"] }, "linear_norm_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["GaussC_lnorm_SE_r", "GaussC_lnorm_SE_rho", "GaussC_lnorm_AE_r", "GaussC_lnorm_AE_rho"] }, "full_gradnorm_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["GaussC_gnorm_SE_r", "GaussC_gnorm_SE_rho", "GaussC_gnorm_AE_r", "GaussC_gnorm_AE_rho"] } }, "ccGaussC": { "abs_algdist_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["ccGaussC_absad_SE_r", "ccGaussC_absad_SE_rho", "ccGaussC_absad_AE_r", "ccGaussC_absad_AE_rho"] }, "linear_norm_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["ccGaussC_lnorm_SE_r", "ccGaussC_lnorm_SE_rho", "ccGaussC_lnorm_AE_r", "ccGaussC_lnorm_AE_rho"] }, "full_gradnorm_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["ccGaussC_gnorm_SE_r", "ccGaussC_gnorm_SE_rho", "ccGaussC_gnorm_AE_r", "ccGaussC_gnorm_AE_rho"] } }, "MeanC": { "abs_algdist_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["MeanC_absad_SE_r", "MeanC_absad_SE_rho", "MeanC_absad_AE_r", "MeanC_absad_AE_rho"] }, "linear_norm_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["MeanC_lnorm_SE_r", "MeanC_lnorm_SE_rho", "MeanC_lnorm_AE_r", "MeanC_lnorm_AE_rho"] }, "full_gradnorm_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["MeanC_gnorm_SE_r", "MeanC_gnorm_SE_rho", "MeanC_gnorm_AE_r", "MeanC_gnorm_AE_rho"] } }, "ccMeanC": { "abs_algdist_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["ccMeanC_absad_SE_r", "ccMeanC_absad_SE_rho", "ccMeanC_absad_AE_r", "ccMeanC_absad_AE_rho"] }, "linear_norm_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["ccMeanC_lnorm_SE_r", "ccMeanC_lnorm_SE_rho", "ccMeanC_lnorm_AE_r", "ccMeanC_lnorm_AE_rho"] }, "full_gradnorm_r0_4":{ "metrics": ["SE", "AE"], "correlations": ["pearson", "spearman"], "frenames": ["ccMeanC_gnorm_SE_r", "ccMeanC_gnorm_SE_rho", "ccMeanC_gnorm_AE_r", "ccMeanC_gnorm_AE_rho"] } }, "Shpidx": { "abs_algdist_r0_4":{ "metrics": ["AE"], "correlations": ["pearson", "spearman"], "frenames": ["Shpidx_absad_AE_r", "Shpidx_absad_AE_rho"] }, "full_gradnorm_r0_4":{ "metrics": ["AE"], "correlations": ["pearson", "spearman"], "frenames": ["Shpidx_gnorm_AE_r", "Shpidx_gnorm_AE_rho"] } }, "Umbdev": { "abs_algdist_r0_4":{ "metrics": ["AE"], "correlations": ["pearson", "spearman"], "frenames": ["Umbdev_absad_AE_r", "Umbdev_absad_AE_rho"] }, "full_gradnorm_r0_4":{ "metrics": ["AE"], "correlations": ["pearson", "spearman"], "frenames": ["Umbdev_gnorm_AE_r", "Umbdev_gnorm_AE_rho"] } }, "minabsC": { "abs_algdist_r0_4":{ "metrics": ["AE"], "correlations": ["pearson", "spearman"], "frenames": ["minabsC_absad_AE_r", "minabsC_absad_AE_rho"] }, "full_gradnorm_r0_4":{ "metrics": ["AE"], "correlations": ["pearson", "spearman"], "frenames": ["minabsC_gnorm_AE_r", "minabsC_gnorm_AE_rho"] } }, "maxabsC": { "abs_algdist_r0_4":{ "metrics": ["AE"], "correlations": ["pearson", "spearman"], "frenames": ["maxabsC_absad_AE_r", "maxabsC_absad_AE_rho"] }, "full_gradnorm_r0_4":{ "metrics": ["AE"], "correlations": ["pearson", "spearman"], "frenames": ["maxabsC_gnorm_AE_r", "maxabsC_gnorm_AE_rho"] } } }, "outlier_filter": null, "outlier_param": 0, "regression_report_path": "*/report/regression_eval.log", "outer_report_path": "*/report/outer_eval.log", "distribution_report_path": "*/report/regression_distrb.log", "regression_pcloud_path": "*/report/regression_eval.las", "regression_plot_path": "*/plot/regression_plot.png", "regression_hist2d_path": "*/plot/regression_hist2d.png", "residual_plot_path": "*/plot/residual_plot.png", "residual_hist2d_path": "*/plot/residual_hist2d.png", "scatter_plot_path": "*/plot/scatter_plot.png", "scatter_hist2d_path": "*/plot/scatter_hist2d.png", "qq_plot_path": "*/plot/qq_plot.png", "summary_plot_path": "*/plot/summary_plot.png", "nthreads": -1 } The JSON above defines a :class:`.RegressionEvaluator` that will consider the analytical Gaussian and mean curvatures, shape index, umbilical deviation and absolute curvatures computed with `Sagemath `_. These reference values will be used to compute the Mean Squared Error (``"MSE"``), Root Mean Squared Error (``"RMSE"``), Mean Absolute Error (``"MAE"``), the Median Absolute Error (``"MeAE"``), and many more metrics for the curvature values computed with the :ref:`geometric eatures miner++ ` of the VL3D++ framework. Moreover, the correlations between the errors on the estimated curvatures and some arbitrary geometric properties will be computed too. Note that, in this case, there is no outlier filtering at all. **Arguments** -- ``metrics`` A list with the names of the metrics that can be computed. Metrics based on error (i.e., raw diferences) are ``["ME", "MaxE", "MinE", "MeE", "DevE", "RangeE", "Q1E", "Q3E", "SkE", "KuE"]``, those based on the squared error are ``["MSE", "MaxSE", "MinSE", "MeSE", "DevSE", "RangeSE", "Q1SE", "Q3SE", "SkSE", "KuSE"]``, those based on the root of some aggregation of the squared error ``["RMSE", "RMaxSE", "RMinSE", "RMeSE", "RDevSE", "RRangeSE", "RQ1SE", "RQ3SE"]``, and those based on the absolute error are ``["MAE", "MaxAE", "MinAE", "MeAE", "DevAE", "RangeAE", "Q1AE", "Q3AE", "SkAE", "KuAE"]``, . Note that the metric name format convention is ``"Mx"`` mean of x, ``"Maxx"`` max of x, ``"Minx"`` min of x, ``"Mex"`` median of x, ``"Devx"`` standard deviation of x, ``"Rangex"`` range of x, ``"Q1x"`` first quartile of x, ``"Q3x"`` third quartile of x, ``"Skx"`` skewness of x, ``"Kux"`` kurtosis of x. Besides, ``["Pearson", "Spearman"]`` correlations can be computed, which automatically involves computing associated :math:`p`-values too. -- ``cases`` A list of lists. Each inner list uses the first element to specify the reference attribute and following elements specify the different predicted attributes whose error must be measured with respect to the references. -- ``cases_renames`` A list of lists. Each inner list must have as many elements as each inner in ``cases`` minus one. Each element must be a string uniquely naming the metrics computed for the case. -- ``outer_correlations`` A dictionary whose keys are the case names (as specified through ``cases_renames``) and whose values are another dictionary. Each subdictionary has as keys the names of the features (that must be available in the point cloud at the current time in the pipeline) whose correlations must be computed. The values of the each subdictionary are another dictionary with three elements: -- ``metrics`` A list specifying the error metrics that must be considered for the correlations. Supported ones are the raw error ``"E"``, the squared error ``"SE"``, and the absolute error ``"AE"``. -- ``correlations`` A list specifying the correlations that must be computed. Supported ones are ``"pearson"`` (linear relationship) and ``"spearman"`` (monotonic relationship). -- ``frenames`` For each error metric and each correlation (iterated with nested loops in the same order as mentioned) the name of the resulting measurement. -- ``outlier_filter`` What outlier filtering strategy must be applied. If not given, then outliers are not filtered. Supported strategies are :math:`k` times the standard deviation (``"stdev"``), :math:`k` times the interquartile range minus the first quartiles and plus the third quartile (``"iqr"``), discard top :math:`k` errors (``"topk"``), discard bottom :math:`k` errors (``"botk"``), discard top and bot k errors (``extremek``), discard :math:`p` percentage of greatest errors (``"topp"``), discard :math:`p` percentage of least errors (``"botp"``), and discard :math:`p` percentage of greateast and least errors (``"extremp"``). -- ``outlier_param`` The parameter governing the outlier filter (e.g., :math:`k` for the ``"stdev"`` outlier filter. -- ``regression_report_path`` Path to write the report with the regression metrics to a text file. -- ``outer_report_path`` Path to write the report with the correlations to a text file. -- ``distribution_report_path`` Path to write the report with the percentiles for each feature (including references, predictions, and arbitrary features involved in the correlations). -- ``regression_pcloud_path`` Path to write the point cloud with the point-wise error measurements. -- ``regression_plot_path`` Path to write the residual scatter plot with the predictions on the :math:`x`-axis and the error on the :math:`y`-axis. -- ``regression_hist2d_path`` Path to write a 2D histogram representation of the residuals. It is similar to the plot generated for ``regression_plot_path`` but accounting for the density through the color. -- ``residual_plot_path`` Path to write the residual scatter plot with the references on the :math:`x`-axis and the error on the :math:`y`-axi. -- ``residual_hist2d_path`` Path to write a 2D histogram representation of the residuals. It is similar to the plot generated for ``residual_plot_path`` but accounting for the density through the color. -- ``scatter_plot_path`` The scatter plot for the different features involved in the error metrics and correlations. -- ``scatter_hist2d_path`` Path to write a 2D histogram representation of the correlation between pairs of features. It is similar to the plot generated by ``scatter_plot_path`` but accounting for the density through the color. -- ``qq_plot_path`` Path to write the QQ plot for each pair (reference, prediction) involved in the computation of the error metrics. -- ``summary_plot_path`` Path to write a plot summarizing the mean and standard deviation of each regression. -- ``nthreads`` How many threads must be used in parallel computations. **NOTE** that currently, parallelism is not supported for :class:`.RegressionEvaluator`. **Output** The output is illustrated considering an hyperbolic paraboloid and a torus. The table below represents the mean absolute error (MAE), median absolute error (MeAE), standard deviation of absolute error (DevAE), max absolute error (MaxAE), mean squared error (MSE), median squared error (MeSE), standard deviation of squared error (DevSE), max squared error (MaxSE), Pearson correlation coefficient, and Spearman correlation coefficient. .. csv-table:: :file: ../csv/regression_eval.csv :widths: 10 9 9 9 9 9 9 9 9 9 9 :header-rows: 1 The image belows represents the point-wise mean curvature and shape index for the hyperbolic paraboloid and the torus, respectively. Besides the corresponding absolute and squared errors. .. figure:: ../img/regression_eval.png :scale: 50 :alt: Figure representing the point-wise errors for the mean curvature and the shape index on an hyperbolic paraboloid and a torus, respectively. The top row shows the point-wise mean curvature and its absolute error on an hyperbolic paraboloid. The bottom row shows the point-wise shape index and its squared error on a torus. Deep learning model evaluator ============================== The :class:`.DLModelEvaluator` assumes there is a deep learning at the current pipeline's state that can be used to process the point cloud at the current pipeline's state. Instead of returning the output point-wise predictions, the values of the output layer and some internal feature representation will be returned to be visualized directly in the point cloud. Note that the internal feature representation might need an enormous amount of memory as it scales depending on how many features are generated by the architecture at the studied layer. A :class:`.DLModelEvaluator` can be defined inside a pipeline using the JSON below: .. code-block:: json { "eval": "DLModelEvaluator", "pointwise_model_output_path": "pwise_out.las", "pointwise_model_activations_path": "pwise_activations.las" } The JSON above defines a :class:`.DLModelEvaluator` that will export the values of the output layer to the file `pwise_out.las` and a representation of the features in the hidden layers to the file `pwise_activations.las`. **Arguments** -- ``pointwise_model_output_path`` Where to export the point cloud with the point-wise outputs of the neural network. -- ``pointwise_model_activations_path`` Where to export the point cloud with the internal features of the neural network. **Output** The output is illustrated considering the March 2018 point clouds from the `Hessigheim dataset `_ to compute the deep learning model evaluation. The figure below illustrates four different features extracted by the neural network. They are taken as the activated outputs of the last layer before the softmax . .. figure:: ../img/dl_activations.png :scale: 50 :alt: Figure representing some features generated by the neural network in the point cloud. Visualization of some features used by a PointNet-based neural network for point-wise classification. .. _Raster grid evaluator: Raster grid evaluator ===================== The :class:`.RasterGridEvaluator` can be used to evaluate the point cloud on a grid. This grid can later be exported to a GeoTIFF file that extends the grid with geographic information. Therefore, the GeoTIFF can be used to evaluate the features or classifications in the point cloud, e.g., loading the GeoTIFF in a GIS software to compare the rasterized point cloud with satellite image or maps in general. The GeoTIFFs are generated using the `RasterIO library `_. .. code-block:: json { "eval": "RasterGridEvaluator", "crs": "+proj=utm +zone=29 +ellps=GRS80 +towgs84=0,0,0,0,0,0,0 +units=m +no_defs +type=crs", "plot_path": "*geotiff/", "conditions": null, "xres": 1.0, "yres": 1.0, "grid_iter_step": 1024, "grids": [ { "fnames": ["Vegetation"], "reduce": "mean", "empty_val": "nan", "target_relational": "equals", "oname": "vegetation_mean" }, { "fnames": ["Vegetation"], "reduce": "max", "empty_val": "nan", "target_relational": "equals", "oname": "vegetation_max" }, { "fnames": ["Prediction"], "target_val": 2, "target_relational": "equals", "reduce": "binary_mask", "count_threshold": 3, "empty_val": "nan", "oname": "vegetation_mask" }, { "fnames": ["Ground", "Vegetation", "Other"], "reduce": "mean", "empty_val": "nan", "target_relational": "equals", "oname": "GVO_mean" }, { "fnames": ["Ground", "Vegetation", "Other"], "reduce": "max", "empty_val": "nan", "target_relational": "equals", "oname": "GVO_max" }, { "fnames": ["PointWiseEntropy"], "reduce": "mean", "empty_val": "nan", "target_relational": "equals", "oname": "pwise_entropy_mean" }, { "fnames": ["PointWiseEntropy"], "reduce": "max", "empty_val": "nan", "target_relational": "equals", "oname": "pwise_entropy_max" } ] } The JSON above defines a :class:`.RasterGridEvaluator` that generates many GeoTIFFs using the EPSG:25829 coordinate reference system (specified using PROJ syntax). The GeoTIFFs are exported to the *geotiff* subdirectory, considering the output prefix of the pipeline. The cell size is :math:`1\,\mathrm{m}` along each axis. Some GeoTIFFs use a single color channel to represent a continuous value. One particular GeoTIFF generates a binary mask for each cell with :math:`1` when there are at least :math:`3` points classified as vegetation and :math:`0` otherwise. The GeoTIFFs that consider the likelihood for Ground, Vegetation, and Other classes will export each likelihood in a different color channel. **Arguments** -- ``crs`` The coordinate reference system specification. See the `RasterIO documentation `_ for more information about CRS specification. -- ``plot_path`` The directory where the GeoTIFF files will be stored. -- ``conditions`` A list with many conditions specified as dictionaries. These conditions will be applied to filter the entire point cloud before computing any raster or grid. See :ref:`the advanced evaluator conditions documentation ` for further details. -- ``xres`` The cell resolution along the x-axis. -- ``yres`` The cell resolution along the y-axis. -- ``grid_iter_step`` How many max rows per iteration. It can be tuned to improve the efficiency but also to prevent memory exhaustion. -- ``radius_expr`` An optional specification to define the computation of the radius for the ball-like neighborhoods. The variable ``"l"`` represents the max cell size and it is the default radius expression. Note that any expression less than ``"sqrt(2)*l/2"`` (half of the cell's hypotenuse) will potentially ignore some points inside the cell boundary. Also, values greater than the previous one will increase the "smooth" effect through more overlapped neighborhoods. -- ``grids`` A list with potentially many grid specifications. A grid can be specified with a dictionary-like style: .. code-block:: json { "fnames": ["feat1", "feat2"], "reduce": "mean", "empty_val": "nan", "target_relational": "equals", "target_val": 2, "count_threshold": 3, "oname": "my_geotiff" } The ``fnames`` list must specify the name of the involved features. It can be null but only when the reduce strategy is ``"binary_mask"`` (binary occupation mask, i.e., whether a cell contains at least one point or not), ``"recount"`` (number of points per cell) or ``"relative_recount"`` (normalized number of points per cell). The optional ``conditions`` list must specify the conditional filters that must be applied to the data for a particular grid. Its specification is the same as for :ref:`the advanced evaluator conditions `. The ``reduce`` string must refer to a strategy to reduce many values per cell to a single one, e.g., ``"mean"``, ``"median"``, ``"min"``, ``"max"``, ``"binary_mask"``, ``"recount"``, and ``"relative_recount"``. The ``empty_val`` value will be assigned to represent the cells with no points. They can be numbers or the string ``"nan"`` (not a number). The ``target_relational`` governs how the values will be compared against the target. Supported relationals are ``"equals"`` (:math:`=`), ``"not_equals"`` (:math:`\neq`), ``"less_than"`` (:math:`<`), ``"less_than_or_equal_to"`` (:math:`\leq`), ``"greater_than"`` (:math:`>`), ``"greater_than_or_equal_to"`` (:math:`\geq`), ``"in"`` (:math:`\in`), and ``"not_in"`` (:math:`\notin`). Note that it must be given even if not used. The ``target_val`` the value that must be searched when using a binary mask strategy. The ``count_threshold`` governs how many times the target value must be found to consider a :math:`1` for the binary mask. The ``dem`` overloads the previous specifications and instead computes a Digital Elevation Model (DEM) assuming the :math:`(x, y)` plane with elevation values corresponding to :math:`z`. Note that since all the other arguments are ignored, only the global conditions of the :class:`.RasterGridEvaluator` are applied. Post-processing interpolations are still possible but generally not especially useful as the DEM already involves a 2D grid interpolation. The DEM is computed through the :class:`.DEMGenerator` class. It supports selecting its internal interpolation strategy (either ``"linear"``, ``"nearest"``, or ``"cubic"``), by default it is .. code-block:: json "dem": { "interpolation": "linear" } The ``interpolator`` can post-process the generated raster to interpolate some cells considering the values in the others through radial basis functions. It is based on the :class:`GridInterpolator2D` class. Below an example of a potential interpolator specificaiton for a particular grid: .. code-block:: json "interpolator": { "iterations": 2, "domain": { "strategy": "polygonal_contour_target", "channel": 0, "erosions": 2, "dilations": 3, "polygonal_approximation": 0, "target_val": "nan", "target_relational": "equals" }, "interpolation": { "smoothing": 0.0, "kernel": "linear", "neighbors": 64, "epsilon": null, "degree": null, "clip_interval": [0, null] } } The arguments of the ``interpolator`` are: -- ``iterations`` How many times apply the entire interpolation strategy to the grid. -- ``domain`` The specification of the first stage where the cells that must be interpolated are determined. Its arguments are: -- ``strategy`` The strategy to determine the cells to be interpolated. It can be: ``"all"`` to interpolate all the cells in the grid, ``"target"`` to interpolate only the cells in the grid that satisfy the given relational, ``"polygonal_contour"`` to interpolate all the cells in the grid inside the region closed by the contour of the cells that do not satisfy the given relational, or ``"polygonal_contour_target"`` to interpolate only the cells that satisfy the given relational and are inside the region closed by the contour of the cells that do not satisfy the given relational. -- ``channel`` The channel (default 0) that must be selected when the grid contains many values per cell to check the relational. -- ``erosions`` How many morphological erosions perform before computing the contour (see `scikit-image documentation about erosion `_). -- ``dilations`` How many morphological dilations perform after the erosions but before computing the contour (see `scikit-image documentation about dilation `_). -- ``polygonal_approximation`` The tolerance for the polygonal approximation using the `Douglas-Peucker algorithm `_. Note that if zero is given there is no polygonal approximation at all. (see `scikit-image documentation about approximate polygons `_). -- ``target_val`` The target value for the relational check. Typically, it will be a number. However, it can be a list given the two endpoints of the interval for ``"inside_open"`` and ``"inside_close"`` relationals or a list representing the items of a set for ``"in"`` and ``"not_in"`` relationals. -- ``target_relational`` The relational itself. It can be either ``"equals"`` (:math:`=`), ``"not_equals"`` (:math:`\neq`), ``"less_than"`` (:math:`<`), ``"less_than_or_equal_to"`` (:math:`\leq`), ``"greater_than"`` (:math:`>`), ``"greater_than_or_equal_to"`` (:math:`\geq`), ``"inside_open"`` (:math:`\in (a, b)`), ``"inside_close"`` (:math:`\in [a, b]`), ``"in"`` (:math:`\in`), or ``"not_in"`` (:math:`\notin`). -- ``interpolation`` The specification of the second stage where the cells that must be interpolated are effectively interpolated. -- ``kernel`` See the kernel parameter in the `SciPy documentation `_. -- ``smoothing`` See the smoothing parameter in the `SciPy documentation `_. -- ``neighbors`` See the neighbors parameter in the `SciPy documentation `_. -- ``epsilon`` See the epsilon parameter in the `SciPy documentation `_. -- ``degree`` See the degree parameter in the `SciPy documentation `_. -- ``clip_interval`` If given, it must be a list of exactly two elements. The first element is the min value such that any value in the grid that is below it will be set to the min value. The second element is the max value such that any value in the grid that is above it will be set to the max value. One of the elements in this list can be null. In that chase, no clipping to that endpoint will be applied. The ``hillshading`` specification, as a post-processing step to improve the visibility of certain landforms in the output raster. Below an example of the hillshading post-processing with default values: .. code-block:: json "hillshading": { "azimuth": 30, "altitude": 30 } The arguments for the hillshading specification are: -- ``azimuth`` The angle of the light source with respect to the surface. It must be inside :math:`[0, 360)` degrees. -- ``altitude`` The angle where the light source is located. It must be inside :math:`[0, 90]` degrees. The ``oname`` name for the output GeoTIFF file corresponding to the grid specification. -- ``reverse_rows`` Boolean flag to control whether to reverse the rows of the grid (True) or not (False). The default is the reversed order, i.e., True. -- ``nthreads`` The number of threads for the parallel computation of the grids. By default, just one thread is used. The value -1 implies using as many threads as available cores. **Output** The output is illustrated considering some MLS points clouds acquired in the region of Pontevedra, Galicia, northwest Spain. The points sum up to :math:`6.15 \times 10^{8}` from a dataset of :math:`3.51 \times 10^{9}` points. .. figure:: ../img/raster_grid_qgis_eval.png :scale: 50 :alt: Figure representing the output GeoTIFFs. Visualization of the output GeoTIFFs on QGIS as an overlay to the Openstreetmap of the region of Pontevedra in Galicia, northwest Spain. The red color represents the mean ground likelihood in the neighborhood, green the vegetation likelihood, and blue any other class (e.g., buildings and powerlines).