model.deeplearn.sequencer package

Submodules

model.deeplearn.sequencer.dl_abstract_sequencer module

class model.deeplearn.sequencer.dl_abstract_sequencer.DLAbstractSequencer(X, y, batch_size, **kwargs)

Bases: PyDataset

Author:

Alberto M. Esmoris Pena

A deep learning sequencer governs how the input data is fed to a neural network. The abstract sequencer for deep learning models provides common logic for different sequencers. It cannot be used directly because it is an abstract class, hence it must be extended by a concrete implementation.

Variables:

  • X – The input data.

  • y – The input reference values.

  • arch (Architecture) – The neural network architecture.

  • batch_size (int) – The number of elements per batch.

  • random_shuffle_indices (bool) – Flag governing whether the indexing of the elements (i.e., the order in which they are given) must be shuffled or not (typically at the end of each epoch).

  • getitem_method – The method that must be called by DLAbstractSequencer.__getitem__(). It might change depending on whether training or predictive mode is enabled.

  • on_epoch_end_method – The method that must be called by DLAbstractSequencer.on_epoc_end(). It might change depending on whether training or predictive mode is enabled.

  • Irandom – A cache for the random shuffle of elements’ indices.

  • shuffled – A cache to track whether the elements have been shuffled (True) or not (False).

__init__(X, y, batch_size, **kwargs)

Initialize the member attributes of the DLAbstractSequencer.

Parameters:

  • X – The input data.

  • y – The input reference values.

  • batch_size (int) – The number of elements per batch.

  • kwargs – The key-word specification to parametrize the sequencer.

on_epoch_end()

Logic to handle the dataset between epochs.

Note that this method can work for training, in which case it calls the DLAbstractSequencer.on_epoch_training() method (or its child/derived class override), but it can also work for predictions, in which case it calls the DLAbstractSequencer.on_epoch_predict().

See DLAbstractSequencer.on_epoch_end_training(), DLAbstractSequencer.enable_training_mode(), See DLAbstractSequencer.on_epoch_end_predict(), and DLAbstractSequencer.enable_predict_mode(),

Returns:

Nothing at all, but the internal state of the sequencer is updated.

set_input_data(X, y)

Set the input data for the sequencer. Note that this method must be overridden for any non-abstract implementation that has a different logic for setting the input data that this baseline method.

Parameters:

  • X – The input data.

  • y – The input reference values.

Returns:

Nothing at all, but the internal state of the sequencer is updated with the new input data.

get_input_data()

Get the input data of the sequencer.

Returns:

A 2-tuple with the input training data (first element) and the corresponding references (second element)

Return type:

tuple

augment(batch_X)

Augment the given batch. Note that, by default, no data augmentation is applied. However, some sequencers can override this method to provide their own data augmentation logic. Note also that, by default, the augment method is not called. Thus, sequencers providing data augmentation logic should also call this method (typically inside their getitem_training method). The data augmentation logic must be provided inside the augment method so it can be properly handled by decorators like DLOfflineSequencer.

Parameters:

batch_X – The input batch to be augmented.

Returns:

The input batch as received.

abstractmethod getitem_training(idx)

Method that provides the logic for DLAbstractSequencer.__getitem__() in training contexts. It must be overridden by any derived class of DLAbstractSequencer that aims to being concrete (non-abstract, i.e., actually callable).

Parameters:

idx (int) – The index identifying the batch that must be obtained.

Returns:

The batch corresponding to the given index as a tuple (X, y).

Return type:

tuple

on_epoch_end_training()

Method that provides the logic for DLAbstractSequencer.on_epoch_end() in training contexts. It can be overridden by any derived class of DLAbstractSequencer to change ts behavior.

The default logic to handle the dataset between epochs provided by DLAbstractSequencer is the random shuffle of the data so the input batches are given in a different order at each training epochs.

See DLAbstractSequencer.init_random_indices() and DLAbstractSequencer.apply_random_indices() methods.

Returns:

Nothing at all, but the internal state of the sequencer is updated.

enable_training_mode()

Enable the training mode of the sequencer so it can be used for training.

See DLAbstractSequencer.__getitem__() and DLAbstractSequencer.getitem_training().

Returns:

Nothing at all but the DLAbstractSequencer state is updated to operate on training mode.

abstractmethod getitem_predict(idx)

Method that provides the logic for DLAbstractSequencer.__getitem__() in prediction contexts. It must be overridden by any derived class of DLAbstractSequencer that aims to being concrete (non-abstract, i.e., actually callable).

Parameters:

idx (int) – The index identifying the batch that must be obtained.

Returns:

The batch corresponding to the given index but without references.

on_epoch_end_predict()

Method that provides the logic for DLAbstractSequencer.on_epoch_end() in prediction contexts. It can be overridden by any derived class of DLAbstractSequencer to change its behavior.

The default logic to handle the dataset between epochs provided by DLAbstractSequencer is to do nothing.

Returns:

Nothing at all, but the internal state of the sequencer is updated.

enable_predict_mode()

Enable the predict mode of the sequencer so it can be used for predictions.

See DLAbstractSequencer.__getitem__() and DLAbstractSequencer.getitem_predict().

Returns:

Nothing at all but the DLAbstractSequencer state is updated to operate on predict mode.

abstractmethod init_random_indices()

Method that must provide the logic to initialize the random indices to shuffle the data.

See DLAbstractSequencer.on_epoch_end_training().

Returns:

Nothing at all but the internal state of the DLAbstractSequencer is updated (e.g., the self.Irandom cache).

abstractmethod apply_random_indices()

Method that must provide the logic to apply the random shuffle based on the random indices computed by the DLAbstractSequencer.init_random_indices() method.

See DLAbstractSequencer.on_epoch_end_training() and DLAbstractSequencer.init_random_indices().

Returns:

Nothing at all but the internal state of the DLAbstractSequencer is updated (e.g., the self.X and self.y data).

extract_input_batch(start_idx, end_idx)

Extract the input batch inside the given indexing interval.

Note that this method can be overridden by any derived class of DLAbstractSequencer that needs to change its behavior.

Parameters:

  • start_idx – Start point of the indexing interval (inclusive).

  • end_idx – End point of the indexing interval (exclusive).

Returns:

The extracted input batch inside the indexing interval.

Return type:

list or np.ndarray

extract_reference_batch(start_idx, end_idx)

Extract the reference batch inside the given indexing interval.

Note that this method can be overridden by any derived class of DLAbstractSequencer that needs to change its behavior.

Parameters:

  • start_idx – Start point of the indexing interval (inclusive).

  • end_idx – End point of the indexing interval (exclusive).

Returns:

The extracted reference batch inside the indexing interval.

Return type:

np.ndarray

post_process_output(z_rf)

Post-process the outputs computed on the receptive fields. There might be no need for a post-processing at all, but some sequencers will need it (e.g., DLSparseShadowSequencer).

Parameters:

z_rf (np.ndarray or tf.Tensor or list) – The outputs computed on the receptive fields

Returns:

The post-processed output.

Return type:

np.ndarray or tf.Tensor or list

model.deeplearn.sequencer.dl_offline_sequencer module

class model.deeplearn.sequencer.dl_offline_sequencer.DLOfflineSequencer(X, y, batch_size, **kwargs)

Bases: DLAbstractSequencer

Author:

Alberto M. Esmoris Pena

A deep learning sequencer that decorates a backbone sequencer and serializes the data so it can be read and fed directly into a neural network. This sequencer allows using training datasets that cannot be hold entirely in memory.

The internal structure of an offline storage file (ofs) is:

– ofs["data"]

Group containing the chunks of training data.

—ofs["data"]["chunk_idx"]

Group containing the training data of the chunk with given index. Note that "chunk_idx" ranges from 0 to number of chunks, i.e., \(0, 1, 2, \ldots, \text{number of chunks}\).

—ofs["data"]["chunk_idx"]["elem_idx"]

All the training data elements in the chunk. Note that "elem_idx" identifies one of the input elements. For example, assume a neural network that receives two inputs, e.g., the structure space and the feature space. Then, the tensor representing the structure space will have "elem_idx" = "X0" and the tensor representing the feature space "elem_idx" = "X1". For convenience the elements are named as \(X_0, \ldots, X_{\text{number of data elements}-1}\).

—ofs["references"]

Group containing the chunks of training references.

—ofs["references"]["chunk_idx"]

Group containing the training references of the chunk with given index. Note that "chunk_idx" ranges from 0 to number of chunks, i.e., \(0, 1, 2, \ldots, \text{number of chunks}\).

—ofs["references"]["chunk_idx"]["elem_idx"]

The training reference elements in the chunk. For standard point-wise classification tasks there is only one single element named "Y0". In other cases, the elements will be indexed as \(Y_0, \ldots, Y_{\text{number of reference elements}-1}\).

—ofs["ordered_keys"]

The keys for the data elements given in the same order as they must yielded by the sequencer so they can be straightforward feed into the neural network.

Variables:

  • offline_storage (str) – Path to the online storage file.

  • chunk_size (int) – How many batches per chunk. Note that each chunk must be small enough to be stored in the available memory, i.e., each chunk will be held in memory during training.

  • chunk_randomization (bool) – When enabled (True), the offline training sequencer will iterate sequentially over the chunks in the first pass but with a different random order in passes after the first one. If disabled (False, default), the offline sequencer will iterate always in a sequential way (i.e., for all passes).

  • batch_randomization (bool) – When enabled (True), the offline training sequencer will iterate sequentially over the elements of a batch in the first pass buth with a different random order in passes after the first one. If disabled (False, default), the offline sequencer will iterate always in a sequential way (i.e., for all passes). Note that the passes refer to calling __getitem__ for all the dataset once, i.e., chunk-wise passes not batch-wise passes.

  • offline_pcloud (None or list of str) – Paths to the many point clouds that must be pre-processed and included into the offline storage file. Note that only the neural network pre-processor will be applied, i.e., previous components of the pipeline (wrt the deep learning model) are applied to the original input point cloud but not to the extra point clouds specified through this argument.

  • disable_offline_storage_writing (bool) – Whether to allow writing operations in the offline storage (typically inserting new data, False by default) or not (True).

  • mh – The model handler handling the deep learning model for which the offline sequencer (optional in general but necessary for most real training pipelines) must yield training data and references.

  • instantiated (bool) – Flag tracking whether the offline sequencer has been instantiated (True, after calling __init__) or not (False, at the beginning of __init__).

__init__(X, y, batch_size, **kwargs)

Initialize the member attributes of DLOfflineSequencer.

Parameters:

  • X – The input data.

  • y – The input reference values.

  • batch_size (int) – The number of elements per batch.

  • kwargs – The key-word specification to parametrize the sequencer.

set_input_data(X, y)

Delegate the logic to the backbone.

See DLAbstractSequencer.set_input_data().

get_input_data()

Delegate the logic to the backbone.

See DLAbstractSequencer.get_input_data().

getitem_training(idx)

See DLAbstractSequencer.getitem_training().

on_epoch_end_training()

See DLAbstractSequencer.on_epoch_end_training().

getitem_predict(idx)

Delegate the logic to the backbone.

See DLAbstractSequencer.getitem_predict().

on_epoch_end_predict()

Delegate the logic to the backbone.

See DLAbstractSequencer.getitem_predict().

is_offline_storage_open()

Check whether the offline storage is currently open.

Returns:

True if offline storage is currently open, false otherwise.

Return type:

bool

open_offline_storage()

Open the offline storage.

Returns:

Nothing, but the offline storage is assigned to the member cache variable self.osf (offline storage file).

are_pclouds_loaded_in_offline_storage()

Check whether offline point clouds have been loaded into the offline storage (True) or not (False).

Returns:

True if offline point clouds are loaded in the offline storage, False otherwise.

Return type:

bool

load_pclouds_in_offline_storage()

Load the point clouds (the one already loaded in the backbone and those whose paths are given in the offline_pcloud list). Note that if disable_offline_storage_writing is set to True, no point cloud will be loaded into the offline storage at all.

Returns:

Nothing, but the offline storage file will contain training data derived from the point clouds in the offline sequencer.

load_backbone_pcloud_in_offline_storage()

Load the training data from the point cloud currently handled by the backbone sequencer into the offline storage file.

Returns:

Nothing, but the point cloud in the backbone will be loaded into the offline storage file (osf).

insert_chunk_into_offline_storage_file(chunk_idx, chunk, chunk_ref)

Insert the given chunk into the offline storage file.

Parameters:

  • chunk_idx (int) – Index of the chunk to be inserted.

  • chunk (List of list) – The data of the chunk to be inserted. Note that chunk[k][j] reads k-th element of the j-th batch.

  • chunk_ref (list) – The references of the chunk to be inserted. Note that chunk_ref[j] reads j-th batch.

Returns:

The index for the next chunk.

Return type:

int

load_pcloud_in_backbone(pcloud_path)

Load the point cloud in the given path in the backbone, after pre-processing it with the pre-processor in the model’s architecture.

Parameters:

pcloud_path – The path of the point cloud to be loaded (after pre-processing) in the backbone.

Returns:

Nothing, but the backbone is updated with the point cloud in the given path.

is_offline_storage_file_initialized()

Check whether the offline storage file (osf) has been initialized (i.e., it has the expected HDFS groups associated to the corresponding keys: references, data, ordered_keys).

Returns:

True if the offline storage file has been initialized, False otherwise.

Return type:

bool

initialize_offline_storage_file()

Initialize the offline storage file registering groups "data" and "references" and the dataset "ordered_keys".

Returns:

Nothing at all, but the offline storage file is modified.

getitem_training_from_offline_storage(idx)

Get the next training data item from the offline storage to feed it into the neural network.

Parameters:

idx – The index of the batch that must be obtained. Note that for the offline sequencer this index is ignored and, instead, it is tracked internally considering chunk and batch indices. Consequently, calling offline_sequencer[i] will yield the same result as calling offline_sequencer[j] even if \(i \neq j\).

Returns:

The next batch as a tuple (X, y), where X is the training data itself and y are the references.

Return type:

tuple

load_current_chunk_in_cache()

Load the current chunk in the offline sequencer cache.

Returns:

Nothing, but the member attributes self.current_chunk and self.current_ref_chunk are updated internally.

init_random_indices()

See DLAbstractSequencer.init_random_indices().

apply_random_indices()

See DLAbstractSequencer.apply_random_indices().

post_process_output(z_rf)

Some backbones might need to post-process the output after computing the predictions. This method delegates the post-processing logic to the backbone.

See DLAbstractSequencer.post_process_output().

model.deeplearn.sequencer.dl_sequencer module

class model.deeplearn.sequencer.dl_sequencer.DLSequencer(X, y, batch_size, **kwargs)

Bases: DLAbstractSequencer

Author:

Alberto M. Esmoris Pena

A deep learning sequencer that governs how the input data is fed to a neural network. It handles two main points:

  1. Memory management: Explicitly take control of the memory handling

    logic to prevent undesired scenarios, e.g., InternalError, or memory exhaustion due to TensorFlow loading the full array of batches into the GPU memory.

  2. Data augmentation: Provides rotations around an arbitrary axis,

    jitter, and scaling through SimpleDataAugmentor. Besides, it also enables random shuffle of the indices to unbias the training process wrt the initial input order.

See DLAbstractSequencer.

Variables:

  • X – The input data.

  • y – The input reference values.

  • arch (Architecture) – The neural network architecture.

  • batch_size (int) – The number of elements per batch.

__init__(X, y, batch_size, **kwargs)

Initialize the member attributes of the DLSequencer.

Parameters:

  • X – The input data.

  • y – The input reference values.

  • batch_size (int) – The number of elements per batch.

  • kwargs – The key-word specification to parametrize the sequencer.

augment(batch_X)

See DLAbstractSequencer.augment().

getitem_training(idx)

See DLAbstractSequencer.getitem_training()

getitem_predict(idx)

See DLAbstractSequencer.getitem_predict().

on_epoch_end_training()

See DLAbstractSequencer.on_epoch_end_training().

find_augmentation_elements(batch_X)

Find the indices of the elements in the batch that must be considered for data augmentation. Note that these indices will depend on the underlying neural network architecture.

Parameters:

batch_X (list or np.ndarray) – The input batch that must be augmented.

Returns:

List of indices representing the elements that must be considered for data augmentation.

Return type:

list of int

init_random_indices()

See DLAbstractSequencer.init_random_indices().

apply_random_indices()

See DLAbstractSequencer.apply_random_indices().

model.deeplearn.sequencer.dl_sparse_shadow_sequencer module

class model.deeplearn.sequencer.dl_sparse_shadow_sequencer.DLSparseShadowSequencer(X, y, batch_size, **kwargs)

Bases: DLAbstractSequencer

Author:

Alberto M. Esmoris Pena

A deep learning sequencer that governs how the input data is fed to a neural network. It handles two main points:

  1. Memory management: Explicitly take control of the memory handling

    logic to prevent undesired scenarios or memory exhaustion due to TensorFlow loading the full array of batches into the GPU memory.

  2. Shadow tensors: The data for models based on sparse grids

    (e.g., SpConv3DPwiseClassif) must be converted from ragged tensors to shadow tensors. Ragged tensors are implemented through the tf.RaggedTensor class while the shadow tensors are implemented as regular tensors through the tf.Tensor class. The shadow tensors are regular through padding to the max size. The layers of the model must know how to ignore the padding in the computations.

See DLAbstractSequencer.

Variables:

  • X – The input data.

  • y – The input reference values.

  • arch (Architecture) – The neural network architecture.

  • batch_size (int) – The number of elements per batch.

  • total_elems (int) – Maximum number of input elements (i.e., in the full batch)

  • max_depth (int) – The max depth of the model.

  • start (list of int) – The start row-index for each element in the batch. It is used to separate padding data from real data. The list of start indices is computed when initializing the sequencer.

  • max_rows (list of int) – The max number of rows (i.e., the number of rows of the element with most rows) at each depth.

__init__(X, y, batch_size, **kwargs)

Initialize the member attributes of the DLSparseShadowSequencer.

Parameters:

  • X – The input data.

  • y – The input reference values.

  • batch_size (int) – The number of elements per batch.

  • kwargs – The key-word specification to parametrize the sequencer.

set_input_data(X, y)

See DLAbstractSequencer.set_input_data() and DLSparseShadowSequencer.prepare_data().

getitem_training(idx)

See DLAbstractSequencer.getitem_training().

getitem_predict(idx)

See DLAbstractSequencer.getitem_predict().

on_epoch_end_training()

See DLAbstractSequencer.on_epoch_end_training().

init_random_indices()

See DLAbstractSequencer.init_random_indices().

apply_random_indices()

See DLAbstractSequencer.apply_random_indices().

prepare_data()

Prepare the input data (self.X and self.y) unrolling the elements per depth. Also compute the max number of rows among all elements and the start row-index for each element in the batch.

Returns:

Nothing at all, but the internal state of the DLSparseShadowSequencer is updated.

prepare_indexing_maps(start_idx, end_idx, batch_X)

Update the given batch (batch_X) so the indexing maps \((h_1, \ldots, h_{t^*})\), \((h^D_1, \ldots, h^D_{t^*-1}\), and \((h^U_1, \ldots, h^U_{t^*-1}\) apply the corresponding offset such that the batch can be unified into a single hash table. This property can be achieved by making sure that the index zero of the second element is transformed to be the last plus one index of the first element and applying this inductively to all subsequent elements in the batch.

Parameters:

  • start_idx – The start index for the current batch (inclusive).

  • end_idx – The end index for the current batch (exclusive)

  • batch_X – The current batch (note references aer not needed here).

Returns:

Nothing at all but the indexing maps are updated in place, i.e, inside the current batch (batch_X).

extract_input_batch(start_idx, end_idx)

See DLAbstractSequencer.extract_input_batch().

extract_reference_batch(start_idx, end_idx)

See DLAbstractSequencer.extract_reference_batch().

post_process_output(z_rf)

Post-process the output to remove the padding. See DLAbstractSequencer.post_process_output()).

Module contents

author:

Alberto M. Esmoris Pena

The sequencer package contains the logic for Keras 2 Sequence, Keras 3 PyDataset, or similar components that can be used to govern how the input data is fed into the neural network.