.. _example_intra_sflnet: SFLNet-like model for point-wise aneurysm detection ****************************************************** Data ====== The point clouds used in this example are from the `IntrA dataset `_ which can be downloaded from `Google drive `_. The point clouds were converted from ASCII to LAS format using the `LASTools `_ inside the bash script below: .. code-block:: bash #!/bin/bash # AUTHOR: Alberto M. Esmoris Pena # BRIEF: Script to generate .laz point clouds from .ad point clouds for adf in $(ls ad/*.ad); do outf=$(sed 's/\.ad/\.laz/g' <<< $adf | sed 's/ad\//las\//g'); txt2las64 -itxt $adf \ -add_attribute 9 "nx" "normal x" \ -add_attribute 9 "ny" "normal y" \ -add_attribute 10 "nz" "normal z" \ -iparse xyz012c -o ${outf}; done The training dataset was composed by merging the following point cloud using the `CloudCompare software `_: *AN1, AN11, AN116, AN117, AN119-1, AN119-2, AN128, AN129, AN134, AN135, AN136, AN138, AN140, AN142, AN149, AN152, AN153, AN155, AN157, AN158, AN159, AN160, AN162, AN164, AN165, AN166, AN167, AN168-1, AN170, AN172, AN175, AN178, AN180, AN181, AN182-1, AN182-2, AN182-3, AN183, AN185, AN186-1, AN186-2, AN187, AN188, AN189, AN19, AN190, AN192, AN193-1, AN193-2, AN194, AN195, AN196-1, AN196-2, AN197, AN198-1, AN198-2, AN199, AN2, AN200, AN201, AN202, AN203, AN205, AN206, AN207, AN208, AN209, AN210, AN211, AN212, AN213, AN214, AN215, AN216, AN217, AN218, AN219, AN23, AN25, AN26, AN27, AN28, AN3, AN31, AN32, AN34, AN42-1, AN42-2, AN42-3, AN44, AN54-1, AN54-2, AN58, AN6, AN9-1, AN9-2*. The resulting training point cloud looks as shown in the image below: .. figure:: ../../img/intra_training007.png :scale: 70 :alt: Figure representing the referenced training point cloud. Visualization of the training point cloud. The healthy vessel is represented with blue color and the aneurysm with red color. Note that the vessels can experiment dramatic changes of geometric scale. The validation of the model was computed on the following point clouds (that were not used to train the model): *AN121, AN125, AN40, AN204, AN55, AN120, AN191, AN144-2, AN148, AN151, AN161, AN163-1, AN137, AN139, AN163-2, AN168-2, AN171, AN173, AN174, AN177*. JSON ====== Training JSON ---------------- The JSON below was used to train the model: .. code-block:: json { "in_pcloud": [ "/ext4/medical_data/IntrA/annotated/training007.laz" ], "out_pcloud": [ "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/*" ], "sequential_pipeline": [ { "train": "ConvolutionalAutoencoderPwiseClassifier", "training_type": "base", "fnames": ["ones"], "random_seed": null, "model_args": { "fnames": ["ones"], "num_classes": 3, "class_names": ["vessel", "aneurysm", "perimeter"], "pre_processing": { "pre_processor": "hierarchical_fpspp", "support_strategy_num_points": 16384, "to_unit_sphere": false, "support_strategy": "fps", "support_strategy_fast": false, "receptive_field_oversampling": { "min_points": 64, "strategy": "knn", "k": 16, "radius": 0.5 }, "center_on_pcloud": true, "neighborhood": { "type": "sphere", "radius": 100.0, "separation_factor": 0.8 }, "num_points_per_depth": [4096, 1024, 512, 256, 64], "fast_flag_per_depth": [false, false, false, false, false], "num_downsampling_neighbors": [1, 32, 32, 32, 32], "num_pwise_neighbors": [32, 32, 32, 32, 32], "num_upsampling_neighbors": [1, 32, 32, 32, 32], "nthreads": -1, "training_receptive_fields_distribution_report_path": "*/training_eval/training_receptive_fields_distribution.log", "training_receptive_fields_distribution_plot_path": "*/training_eval/training_receptive_fields_distribution.svg", "training_receptive_fields_dir": null, "receptive_fields_distribution_report_path": "*/training_eval/receptive_fields_distribution.log", "receptive_fields_distribution_plot_path": "*/training_eval/receptive_fields_distribution.svg", "_receptive_fields_dir": "*/training_eval/receptive_fields/", "training_support_points_report_path": "*/training_eval/training_support_points.las", "support_points_report_path": "*/training_eval/support_points.las" }, "feature_extraction": { "type": "LightKPConv", "operations_per_depth": [2, 1, 1, 1, 1], "feature_space_dims": [64, 64, 128, 256, 512, 1024], "bn": true, "bn_momentum": 0.98, "activate": true, "sigma": [100.0, 100.0, 125.0, 150.0, 175.0, 200.0], "kernel_radius": [100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "num_kernel_points": [15, 15, 15, 15, 15, 15], "deformable": [false, false, false, false, false, false], "W_initializer": ["glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform"], "W_regularizer": [null, null, null, null, null, null], "W_constraint": [null, null, null, null, null, null], "A_trainable": [true, true, true, true, true ,true], "A_regularizer": [null, null, null, null, null, null], "A_constraint": [null, null, null, null, null, null], "A_initializer": ["ones", "ones", "ones", "ones", "ones", "ones"], "_unary_convolution_wrapper": { "activation": "relu", "initializer": "glorot_uniform", "bn": true, "bn_momentum": 0.98, "feature_dim_divisor": 2 }, "hourglass_wrapper": { "internal_dim": [2, 2, 4, 16, 32, 64], "parallel_internal_dim": [8, 8, 16, 32, 64, 128], "activation": ["relu", "relu", "relu", "relu", "relu", "relu"], "activation2": [null, null, null, null, null, null], "regularize": [true, true, true, true, true, true], "W1_initializer": ["glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform"], "W1_regularizer": [null, null, null, null, null, null], "W1_constraint": [null, null, null, null, null, null], "W2_initializer": ["glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform"], "W2_regularizer": [null, null, null, null, null, null], "W2_constraint": [null, null, null, null, null, null], "loss_factor": 0.1, "subspace_factor": 0.125, "feature_dim_divisor": 4, "bn": false, "bn_momentum": 0.98, "out_bn": true, "out_bn_momentum": 0.98, "out_activation": "relu" } }, "features_alignment": null, "downsampling_filter": "strided_lightkpconv", "upsampling_filter": "mean", "upsampling_bn": true, "upsampling_momentum": 0.98, "upsampling_hourglass": { "activation": "relu", "activation2": null, "regularize": true, "W1_initializer": "glorot_uniform", "W1_regularizer": null, "W1_constraint": null, "W2_initializer": "glorot_uniform", "W2_regularizer": null, "W2_constraint": null, "loss_factor": 0.1, "subspace_factor": 0.125 }, "conv1d": false, "conv1d_kernel_initializer": "glorot_normal", "output_kernel_initializer": "glorot_normal", "model_handling": { "summary_report_path": "*/model_summary.log", "training_history_dir": "*/training_eval/history", "_features_structuring_representation_dir": "*/training_eval/feat_struct_layer/", "kpconv_representation_dir": "*/training_eval/kpconv_layers/", "skpconv_representation_dir": "*/training_eval/skpconv_layers/", "lkpconv_representation_dir": "*/training_eval/lkpconv_layers/", "slkpconv_representation_dir": "*/training_eval/slkpconv_layers/", "class_weight": [1.0, 1.0, 0.0], "training_epochs": 400, "batch_size": 64, "training_sequencer": { "type": "DLSequencer", "random_shuffle_indices": true, "augmentor": { "transformations": [ { "type": "Rotation", "axis": [0, 0, 1], "angle_distribution": { "type": "uniform", "start": -3.141592, "end": 3.141592 } }, { "type": "Rotation", "axis": [0, 1, 0], "angle_distribution": { "type": "uniform", "start": -3.141592, "end": 3.141592 } }, { "type": "Rotation", "axis": [1, 0, 0], "angle_distribution": { "type": "uniform", "start": -3.141592, "end": 3.141592 } }, { "type": "Scale", "scale_distribution": { "type": "uniform", "start": 0.98, "end": 1.02 } }, { "type": "Jitter", "noise_distribution": { "type": "normal", "mean": 0, "stdev": 0.03 } } ] } }, "prediction_reducer": { "reduce_strategy" : { "type": "MeanPredReduceStrategy" }, "select_strategy": { "type": "ArgMaxPredSelectStrategy" } }, "checkpoint_path": "*/checkpoint.weights.h5", "checkpoint_monitor": "loss", "learning_rate_on_plateau": { "monitor": "loss", "mode": "min", "factor": 0.1, "patience": 2000, "cooldown": 5, "min_delta": 0.01, "min_lr": 1e-6 } }, "compilation_args": { "optimizer": { "algorithm": "Adam", "learning_rate": { "schedule": "exponential_decay", "schedule_args": { "initial_learning_rate": 1e-2, "decay_steps": 512, "decay_rate": 0.96, "staircase": false } } }, "loss": { "function": "class_weighted_categorical_crossentropy" }, "metrics": [ "categorical_accuracy" ] }, "architecture_graph_path": "*/model_graph.png", "architecture_graph_args": { "show_shapes": true, "show_dtype": true, "show_layer_names": true, "rankdir": "TB", "expand_nested": true, "dpi": 300, "show_layer_activations": true } }, "autoval_metrics": null, "training_evaluation_metrics": null, "training_class_evaluation_metrics": null, "training_evaluation_report_path": null, "training_class_evaluation_report_path": null, "training_confusion_matrix_report_path": null, "training_confusion_matrix_plot_path": null, "training_class_distribution_report_path": null, "training_class_distribution_plot_path": null, "training_classified_point_cloud_path": null, "training_activations_path": null }, { "writer": "PredictivePipelineWriter", "out_pipeline": "*/pipe/SFLNET.pipe", "include_writer": false, "include_imputer": false, "include_feature_transformer": false, "include_miner": false, "include_class_transformer": false, "include_clustering": false, "ignore_predictions": false } ] } Preparation JSON ------------------- The JSON below was used to modify the model before using it for final predictions. The rationale behind this is that the model operates point cloud by point cloud, while it is trained on a merged point cloud. Thus, it does not require :math:`16384` input neighborhoods, in this case :math:`256` input neighborhoods are enough. .. code-block:: json { "in_pcloud": [ "/ext4/medical_data/IntrA/annotated/training007.laz" ], "out_pcloud": [ "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/*" ], "sequential_pipeline": [ { "train": "ConvolutionalAutoencoderPwiseClassifier", "pretrained_model": "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/pipe/SFLNET.model", "training_type": "base", "fnames": ["ones"], "random_seed": null, "model_args": { "fnames": ["ones"], "num_classes": 3, "class_names": ["vessel", "aneurysm", "perimeter"], "pre_processing": { "pre_processor": "hierarchical_fpspp", "support_strategy_num_points": 256, "to_unit_sphere": false, "support_strategy": "fps", "support_strategy_fast": false, "receptive_field_oversampling": { "min_points": 64, "strategy": "knn", "k": 16, "radius": 0.5 }, "center_on_pcloud": true, "neighborhood": { "type": "sphere", "radius": 100.0, "separation_factor": 0.8 }, "num_points_per_depth": [4096, 1024, 512, 256, 64], "fast_flag_per_depth": [false, false, false, false, false], "num_downsampling_neighbors": [1, 32, 32, 32, 32], "num_pwise_neighbors": [32, 32, 32, 32, 32], "num_upsampling_neighbors": [1, 32, 32, 32, 32], "nthreads": -1, "training_receptive_fields_distribution_report_path": null, "training_receptive_fields_distribution_plot_path": null, "training_receptive_fields_dir": null, "receptive_fields_distribution_report_path": null, "receptive_fields_distribution_plot_path": null, "_receptive_fields_dir": null, "training_support_points_report_path": null, "support_points_report_path": null }, "feature_extraction": { "type": "LightKPConv", "operations_per_depth": [2, 1, 1, 1, 1], "feature_space_dims": [64, 64, 128, 256, 512, 1024], "bn": true, "bn_momentum": 0.98, "activate": true, "sigma": [100.0, 100.0, 125.0, 150.0, 175.0, 200.0], "kernel_radius": [100.0, 100.0, 100.0, 100.0, 100.0, 100.0], "num_kernel_points": [15, 15, 15, 15, 15, 15], "deformable": [false, false, false, false, false, false], "W_initializer": ["glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform"], "W_regularizer": [null, null, null, null, null, null], "W_constraint": [null, null, null, null, null, null], "A_trainable": [true, true, true, true, true ,true], "A_regularizer": [null, null, null, null, null, null], "A_constraint": [null, null, null, null, null, null], "A_initializer": ["ones", "ones", "ones", "ones", "ones", "ones"], "_unary_convolution_wrapper": { "activation": "relu", "initializer": "glorot_uniform", "bn": true, "bn_momentum": 0.98, "feature_dim_divisor": 2 }, "hourglass_wrapper": { "internal_dim": [2, 2, 4, 16, 32, 64], "parallel_internal_dim": [8, 8, 16, 32, 64, 128], "activation": ["relu", "relu", "relu", "relu", "relu", "relu"], "activation2": [null, null, null, null, null, null], "regularize": [true, true, true, true, true, true], "W1_initializer": ["glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform"], "W1_regularizer": [null, null, null, null, null, null], "W1_constraint": [null, null, null, null, null, null], "W2_initializer": ["glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform", "glorot_uniform"], "W2_regularizer": [null, null, null, null, null, null], "W2_constraint": [null, null, null, null, null, null], "loss_factor": 0.1, "subspace_factor": 0.125, "feature_dim_divisor": 4, "bn": false, "bn_momentum": 0.98, "out_bn": true, "out_bn_momentum": 0.98, "out_activation": "relu" } }, "features_alignment": null, "downsampling_filter": "strided_lightkpconv", "upsampling_filter": "mean", "upsampling_bn": true, "upsampling_momentum": 0.98, "upsampling_hourglass": { "activation": "relu", "activation2": null, "regularize": true, "W1_initializer": "glorot_uniform", "W1_regularizer": null, "W1_constraint": null, "W2_initializer": "glorot_uniform", "W2_regularizer": null, "W2_constraint": null, "loss_factor": 0.1, "subspace_factor": 0.125 }, "conv1d": false, "conv1d_kernel_initializer": "glorot_normal", "output_kernel_initializer": "glorot_normal", "model_handling": { "summary_report_path": "*/model_summary.log", "training_history_dir": null, "_features_structuring_representation_dir": "*/training_eval/feat_struct_layer/", "kpconv_representation_dir": null, "skpconv_representation_dir": null, "lkpconv_representation_dir": null, "slkpconv_representation_dir": null, "class_weight": [1.0, 1.0, 0.0], "training_epochs": 0, "batch_size": 64, "training_sequencer": { "type": "DLSequencer", "random_shuffle_indices": true, "augmentor": { "transformations": [ { "type": "Rotation", "axis": [0, 0, 1], "angle_distribution": { "type": "uniform", "start": -3.141592, "end": 3.141592 } }, { "type": "Rotation", "axis": [0, 1, 0], "angle_distribution": { "type": "uniform", "start": -3.141592, "end": 3.141592 } }, { "type": "Rotation", "axis": [1, 0, 0], "angle_distribution": { "type": "uniform", "start": -3.141592, "end": 3.141592 } }, { "type": "Scale", "scale_distribution": { "type": "uniform", "start": 0.98, "end": 1.02 } }, { "type": "Jitter", "noise_distribution": { "type": "normal", "mean": 0, "stdev": 0.03 } } ] } }, "prediction_reducer": { "reduce_strategy" : { "type": "MeanPredReduceStrategy" }, "select_strategy": { "type": "ArgMaxPredSelectStrategy" } }, "checkpoint_path": "*/checkpoint.weights.h5", "checkpoint_monitor": "loss", "learning_rate_on_plateau": { "monitor": "loss", "mode": "min", "factor": 0.1, "patience": 2000, "cooldown": 5, "min_delta": 0.01, "min_lr": 1e-6 } }, "compilation_args": { "optimizer": { "algorithm": "Adam", "learning_rate": { "schedule": "exponential_decay", "schedule_args": { "initial_learning_rate": 1e-2, "decay_steps": 512, "decay_rate": 0.96, "staircase": false } } }, "loss": { "function": "class_weighted_categorical_crossentropy" }, "metrics": [ "categorical_accuracy" ] }, "architecture_graph_path": "*/model_graph.png", "architecture_graph_args": { "show_shapes": true, "show_dtype": true, "show_layer_names": true, "rankdir": "TB", "expand_nested": true, "dpi": 300, "show_layer_activations": true } }, "autoval_metrics": null, "training_evaluation_metrics": null, "training_class_evaluation_metrics": null, "training_evaluation_report_path": null, "training_class_evaluation_report_path": null, "training_confusion_matrix_report_path": null, "training_confusion_matrix_plot_path": null, "training_class_distribution_report_path": null, "training_class_distribution_plot_path": null, "training_classified_point_cloud_path": null, "training_activations_path": null }, { "writer": "PredictivePipelineWriter", "out_pipeline": "*/prepared_pipe/SFLNET.pipe", "include_writer": false, "include_imputer": false, "include_feature_transformer": false, "include_miner": false, "include_class_transformer": false, "include_clustering": false, "ignore_predictions": false } ] } Validation JSON ------------------ The following JSON was used to validate the model on unseen data: .. code-block:: json { "in_pcloud": [ "/ext4/medical_data/IntrA/annotated/las/AN121-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN125-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN40-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN204-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN55-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN120-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN191-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN144-2-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN148-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN151-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN161-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN163-1-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN137-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN139-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN163-2-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN168-2-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN171-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN173-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN174-_norm.laz", "/ext4/medical_data/IntrA/annotated/las/AN177-_norm.laz" ], "out_pcloud": [ "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN121/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN125/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN40/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN204/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN55/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN120/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN191/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN144-2/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN148/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN151/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN161/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN163-1/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN137/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN139/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN163-2/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN168-2/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN171/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN173/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN174/*", "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/preds/AN177/*" ], "sequential_pipeline": [ { "predict": "PredictivePipeline", "model_path": "/ext4/medical_data/IntrA/vl3d/out/DL_SFLNETPP_X/T1/prepared_pipe/SFLNET.pipe" }, { "writer": "ClassifiedPcloudWriter", "out_pcloud": "*predicted.las" }, { "eval": "ClassificationEvaluator", "class_names": ["vessel", "aneurysm", "perimeter"], "ignore_classes": ["perimeter"], "metrics": ["OA", "P", "R", "F1", "IoU", "wP", "wR", "wF1", "wIoU", "MCC", "Kappa"], "class_metrics": ["P", "R", "F1", "IoU"], "report_path": "*report/global_eval.log", "class_report_path": "*report/class_eval.log", "confusion_matrix_report_path" : "*report/confusion_matrix.log", "confusion_matrix_plot_path" : "*plot/confusion_matrix.svg", "confusion_matrix_normalization_strategy": "row", "class_distribution_report_path": "*report/class_distribution.log", "class_distribution_plot_path": "*plot/class_distribution.svg" }, { "eval": "ClassificationUncertaintyEvaluator", "class_names": ["vessel", "aneurysm", "perimeter"], "ignore_classes": ["perimeter"], "include_probabilities": true, "include_weighted_entropy": false, "include_clusters": false, "weight_by_predictions": false, "num_clusters": 0, "clustering_max_iters": 0, "clustering_batch_size": 0, "clustering_entropy_weights": false, "clustering_reduce_function": null, "gaussian_kernel_points": 256, "report_path": "*uncertainty/uncertainty.las", "plot_path": "*uncertainty/" } ] } Quantification ================ The table below shows the evaluation metrics for each validation point cloud. .. csv-table:: :file: ../../csv/intra_example_validation.csv :widths: 9 8 8 8 8 8 8 8 8 8 8 8 :header-rows: 1 Visualization ================ The figure below shows the results of the model on validation data never seen before. .. figure:: ../../img/intra_example_visualization.png :scale: 50 :alt: Figure representing the references, predictions, errors, and point-wise probabilities. Visualization of the reference, predictions, errors, and point-wise probabilities on some validation point clouds never seen before. The red color represents an aneurysm, the blue one a healthy vessel. Application ============= This example has two main applications: #. **Baseline model** for point-wise aneurysm detection in 3D point clouds with deep learning. #. Show how to apply deep learning models for **3D medical point clouds**.