src.model.deeplearn.layer.point_transformer_layer

Classes

PointTransformerLayer(*args, **kwargs)

class src.model.deeplearn.layer.point_transformer_layer.PointTransformerLayer(*args, **kwargs)

Author:

Alberto M. Esmoris Pena

A point transformer layer receives batches of \(R\) points with \(D_{\text{in}}\) features each, and \(\kappa\) known neighbors in the same space. These inputs are used to compute an output feature space of \(R\) points with \(D_{\text{out}}\) features each. In doing so, the indexing tensor \(\mathcal{N} \in \mathbb{Z}^{K \times R \times \kappa}\) is used to link \(\kappa\) neighbors for each of the \(R\) input points in each of the \(K\) input batches.

The layer applies a feature space transformation

\[\pmb{\hat{f}}_{i*} = \sum_{\pmb{x}_{j*} \in \mathcal{N}(\pmb{x}_{i*})}{ \sigma \left( \gamma\left( \psi(\pmb{f}_{j*}) - \phi(\pmb{f}_{i*}) + \delta(\pmb{x}_{i*}, \pmb{x}_{j*}) \right) \right) \odot \left( \alpha(\pmb{f}_{j*}) + \delta(\pmb{x}_{i*}, \pmb{x}_{j*}) \right) } ,\]

where the positional encoding is

\[\delta(\pmb{x}_{i*}, \pmb{x}_{j*}) = \tilde{\sigma}_{\theta} \left( \sigma_{\theta}\left( \left(\pmb{x}_{j*} - \pmb{x}_{i*}\right) \pmb{\Theta} \oplus \pmb{\theta} \right) \pmb{\widetilde{\Theta}} \oplus \pmb{\tilde{\theta}} \right) ,\]

\(\odot\) is the Hadamard product, \(\oplus\) is the vector broadcast sum,

\[\phi(\pmb{f}_{i*}) = \sigma_{\phi}\left( \pmb{f}_{i*} \pmb{\Phi} \oplus \pmb{\phi} \right) ,\]

\[\psi(\pmb{f}_{j*}) = \sigma_{\psi}\left( \pmb{f}_{i*} \pmb{\Psi} \oplus \pmb{\psi} \right) ,\]

\[\alpha(\pmb{f}_{j*}) = \sigma_{\alpha}\left( \pmb{f}_{j*} \pmb{A} \oplus \pmb{a} \right) ,\]

\[\gamma(\pmb{x}) = \sigma_{\gamma} \left( \pmb{x} \pmb{\Gamma} \oplus \pmb{\gamma} \right) ,\]

with \(\sigma_{\phi}, \sigma_{\psi}, \sigma_{\alpha}, \sigma_{\gamma}, \sigma_{\theta}, \tilde{\sigma}_{\theta}\) typically a component-wise ReLU activation, and

\[\sigma(\pmb{x})_p = \dfrac{ e^{x_p} }{ \sum_{q=1}^{D_{\text{out}}}{e^{x_q}} }\]

a softmax activation.

The parameter space of the layer is represented through

\[\pmb{\Phi}, \pmb{\Psi}, \pmb{A} \in \mathbb{R}^{D_{\text{in}} \times D_{\text{out}}} \,,\; \pmb{\Gamma} \in \mathbb{R}^{D_{\text{out}} \times D_{\text{out}}}\]

weight matrices,

\[\pmb{\Theta} \in \mathbb{R}^{n_x \times D_{\text{out}}} \,,\; \pmb{\widetilde{\Theta}} \in \mathbb{R}^{D_{\text{out}} \times D_{\text{out}}}\]

positional weight matrices (with \(n_x\) being the dimensionality of the structure space, e.g., \(n_x=3\) for a 3D point cloud), and

\[\pmb{\phi}, \pmb{\psi}, \pmb{a}, \pmb{\gamma}, \pmb{\theta}, \pmb{\tilde{\theta}} \in \mathbb{R}^{D_{\text{out}}}\]

weight and positional weight vectors.

Note that the structure space of the point cloud is represented through \(\pmb{X} \in \mathbb{R}^{R \times n_x}\), whose rows \(\pmb{x}_{i*} \in \mathbb{R}^{n_x}\) represent point-wise coordinates in an \(n_x\)-dimensional Euclidean space. Besides, the feature space is represented with \(\pmb{F} \in \mathbb{R}^{R \times n_f}\) (where \(n_f = D_{\text{in}}\)), whose rows \(\pmb{f}_{i*} \in \mathbb{R}^{n_f}\) represent point-wise features.

This layer is inspired in the PointTransformer v1 paper (https://doi.org/10.48550/arXiv.2012.09164).

Variables:

• Dout (int) – The output dimensionality of the point transformer.

• built_Phi (bool) – Whether the weights matrix \(\pmb{\Phi}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_Psi (bool) – Whether the weights matrix \(\pmb{\Psi}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_A (bool.) – Whether the weights matrix \(\pmb{A}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_Gamma (bool) – Whether the weights matrix \(\pmb{\Gamma}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_Theta (bool) – Whether the weights matrix \(\pmb{\Theta}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_ThetaTilde (bool) – Whether the weights matrix \(\pmb{\widetilde{\Theta}}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_phi (bool) – Whether the weights vector \(\pmb{\phi}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_psi (bool) – Whether the weights vector \(\pmb{\psi}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_a (bool.) – Whether the weights vector \(\pmb{a}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_gamma (bool) – Whether the weights vector \(\pmb{\gamma}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_theta (bool) – Whether the weights vector \(\pmb{\theta}\) is built. Initially it is false, but it will be updated once the layer is built.

• built_thetaTilde (bool) – Whether the weights vector \(\pmb{\tilde{\theta}}\) is built. Initially it is false, but it will be updated once the layer is built.

• Phi_initializer – The initializer for the matrix of weights \(\pmb{\Phi}\).

• Phi_regularizer – The regularizer for the matrix of weights \(\pmb{\Phi}\).

• Phi_constraint – The constraint for the matrix of weights \(\pmb{\Phi}\).

• Psi_initializer – The initializer for the matrix of weights \(\pmb{\Psi}\).

• Psi_regularizer – The regularizer for the matrix of weights \(\pmb{\Psi}\).

• Psi_constraint – The constraint for the matrix of weights \(\pmb{\Psi}\).

• A_initializer – The initializer for the matrix of weights \(\pmb{A}\).

• A_regularizer – The regularizer for the matrix of weights \(\pmb{A}\).

• A_constraint – The constraint for the matrix of weights \(\pmb{A}\).

• Gamma_initializer – The initializer for the matrix of weights \(\pmb{\Gamma}\).

• Gamma_regularizer – The regularizer for the matrix of weights \(\pmb{\Gamma}\).

• Gamma_constraint – The constraint for the matrix of weights \(\pmb{\Gamma}\).

• Theta_initializer – The initializer for the matrix of weights \(\pmb{\Theta}\).

• Theta_regularizer – The regularizer for the matrix of weights \(\pmb{\Theta}\).

• Theta_constraint – The constraint for the matrix of weights \(\pmb{\Theta}\).

• ThetaTilde_initializer – The initializer for the matrix of weights \(\pmb{\widetilde{\Theta}}\).

• ThetaTilde_regularizer – The regularizer for the matrix of weights \(\pmb{\widetilde{\Theta}}\).

• ThetaTilde_constraint – The constraint for the matrix of weights \(\pmb{\widetilde{\Theta}}\).

• phi_initializer – The initializer for the vector of weights \(\pmb{\phi}\).

• phi_regularizer – The regularizer for the vector of weights \(\pmb{\phi}\).

• phi_constraint – The constraint for the vector of weights \(\pmb{\phi}\).

• psi_initializer – The initializer for the vector of weights \(\pmb{\psi}\).

• psi_regularizer – The regularizer for the vector of weights \(\pmb{\psi}\).

• psi_constraint – The constraint for the vector of weights \(\pmb{\psi}\).

• a_initializer – The initializer for the vector of weights \(\pmb{a}\).

• a_regularizer – The regularizer for the vector of weights \(\pmb{a}\).

• a_constraint – The constraint for the vector of weights \(\pmb{a}\).

• gamma_initializer – The initializer for the vector of weights \(\pmb{\gamma}\).

• gamma_regularizer – The regularizer for the vector of weights \(\pmb{\gamma}\).

• gamma_constraint – The constraint for the vector of weights \(\pmb{\gamma}\).

• theta_initializer – The initializer for the vector of weights \(\pmb{\theta}\).

• theta_regularizer – The regularizer for the vector of weights \(\pmb{\theta}\).

• theta_constraint – The constraint for the vector of weights \(\pmb{\theta}\).

• thetaTilde_initializer – The initializer for the vector of weights \(\pmb{\widetilde{\theta}}\).

• thetaTilde_regularizer – The regularizer for the vector of weights \(\pmb{\widetilde{\theta}}\).

• thetaTilde_constraint – The constraint for the vector of weights \(\pmb{\widetilde{\theta}}\).

__init__(Dout=320, built_Phi=False, built_Psi=False, built_A=False, built_Gamma=False, built_Theta=False, built_ThetaTilde=False, built_phi=False, built_psi=False, built_a=False, built_gamma=False, built_theta=False, built_thetaTilde=False, Phi_initializer=None, Phi_regularizer=None, Phi_constraint=None, Psi_initializer=None, Psi_regularizer=None, Psi_constraint=None, A_initializer=None, A_regularizer=None, A_constraint=None, Gamma_initializer=None, Gamma_regularizer=None, Gamma_constraint=None, Theta_initializer=None, Theta_regularizer=None, Theta_constraint=None, ThetaTilde_initializer=None, ThetaTilde_regularizer=None, ThetaTilde_constraint=None, phi_initializer=None, phi_regularizer=None, phi_constraint=None, psi_initializer=None, psi_regularizer=None, psi_constraint=None, a_initializer=None, a_regularizer=None, a_constraint=None, gamma_initializer=None, gamma_regularizer=None, gamma_constraint=None, theta_initializer=None, theta_regularizer=None, theta_constraint=None, thetaTilde_initializer=None, thetaTilde_regularizer=None, thetaTilde_constraint=None, sigma_activation=<function softmax>, sigma_phi_activation=<function relu>, sigma_psi_activation=<function relu>, sigma_alpha_activation=<function relu>, sigma_gamma_activation=<function relu>, sigma_theta_activation=<function relu>, sigma_theta_tilde_activation=<function relu>, **kwargs)

See Layer and Layer.__init__().

build(dim_in)

Build the weight vectors and matrices.

See Layer and layer.Layer.build().

call(inputs, training=False, mask=False)

Compute the PointTransformerLayer on an input batch.

Parameters:

inputs –

The input such that:

– inputs[0]

is the structure space tensor representing the geometry of the many receptive fields in the batch.

\[\mathcal{X} \in \mathbb{R}^{K \times R \times n_x}\]

– inputs[1]

is the feature space tensor representing the features of the many receptive fields in the batch.

\[\mathcal{F} \in \mathbb{R}^{K \times R \times n_f}\]

– inputs[2]

is the indexing tensor representing the neighborhoods of \(\kappa\) neighbors for each input point, in the same space.

\[\mathcal{N} \in \mathbb{Z}^{K \times R \times \kappa}\]

Returns:

The output feature space \(\mathcal{\widehat{F}} \in \mathbb{R}^{K \times R \times D_{\mathrm{out}}}\).

get_config()

Return necessary data to serialize the layer.

classmethod from_config(config)

Use given config data to deserialize the layer.