$\renewcommand{\Diff}{\mathcal{D}}\newcommand{\dist}{\mathrm{dist}}\renewcommand{\Imm}{\mathcal{I}}\newcommand{\Shape}{\mathcal{S}}\newcommand{\R}{\mathbb{R}}\newcommand{\vol}{\operatorname{vol}}\newcommand{\Vol}{\mathrm{Vol}}\newcommand{\Var}{V}$

<h4> Parameterization Invariant Representations<br> for Efficient Shape Learning </h4><hr> <p> <b>Emmanuel Hartman$^1$</b>, Emery Pierson$^2$, Martin Bauer$^3$, Nicolas Charon$^1$<br> Zan Ahmad$^4$, Yashil Sukurdeep$^4$, Shiyi Chen$^4$, Minglang Yin$^5$ <br> <br>$^1$Department of Mathematics, University of Houston<br> $^2$ LIX, Ecole Polytechnique, France<br> $^3$Department of Mathematics, Florida State University<br> $^4$Department of Applied Mathematics, Johns Hopkins University<br> $^5$Department of Biomedical Engineering, Johns Hopkins University<br></p><hr><p> 10 February 2025 </p> <div class="row"> <div class="col-md-6" align="left" markdown="1"> <img src="figs/nsf.png" width="20%" /> </div> <div class="col-md-6" align="right" markdown="1"> <img src="figs/ESI_Logo.png" width="60%" /> </div> </div>

<h4 align="left">Shape Learning</h4><hr> <div class="row" align="left"> <div class="col-md-8" align="left" markdown="1"> <p>Consider the space of parameterized objects $\Imm$ acted on by a group of reparameterizations $\Diff$. <br>The goal of shape learning is to approximate a function $f:\Imm \to \mathcal{Y}$ which is <b>invariant</b> to the action of $\Diff$. i.e. $$\text{for any } q\in\Imm \text{ and } \gamma\in\Diff, \quad f(q)=f(q\circ\gamma).$$</p> <p class="fragment fade-left">How can we ensure our approximation of $f$ is parametrization invariant?<br></p><br> <p class="fragment fade-left">How can we represent data with a fixed dimension? Especially as triangular mesh representations of surfaces have differing simplicial structures and numbers of vertices.</p> </div> <div class="col-md-4" align="center" markdown="1"> <img src="figs/classification.png" width="90%"/> <img src="figs/reconstruction1.jpg" width="75%"/> </div> </div>

<h4 align="left">Parameterization Invariant Latent Space Representations</h4><hr> <div class="row"> <div class="col-md-8" align="left" markdown="1"> <p>In this talk we will consider the approach of parameterization invariant latent space representations. In this approach, we construct $E:\Imm\to \mathcal{L} = \R^m$ so that $E$ is invariant to the action of $\Diff$. <br>For a well constructed representation we can learn a function $g:\mathcal{L}\to \mathcal{Y}$ such that $$g\circ E \approx f.$$</p> <p class="fragment fade-left">Why decompose the problem in this way?</p> <div class="col-md-1" align="left" markdown="1"></div> <div class="col-md-11" align="left" markdown="1"> <p class="fragment fade-left">+ $g\circ E$ is invariant to the action of $\mathcal{D}$</p> <p class="fragment fade-left">+ $E$ can be designed to not depend on the dimensionality of the discetization of data from $\Imm$</p> </div> </div> <div class="col-md-4" markdown="1" align="center"> <div class="r-stack"> <span class="fragment current-visible"></span> <span class="fragment current-visible"><img src="figs/InvarianceExplainer.png" width="100%"/></span> <span class="fragment current-visible"><img src="figs/InvariantFVS.png" width="100%"/></span> </div></div> </div>

<h4 align="left">PointNet[1] and similar methods[2,3,...] for point clouds</h4><hr> <div class="row"> <p>In the context of 3D pointclouds, we consider $\Imm = \{q:\{1,2,..,n\} \to \R^3\}$ and the group of reparameterizations $\Diff = S_n$ is the symmetric group on $n$ elements.</p><br><br> <div class="col-md-7" align="left" markdown="1"> <p>To learn a parameterization invariant representation of pointclouds PointNet learns a function $m: \R^3 \to \R^{1024}$ and considers $E:\Imm \to \R^{1024}$ given by $$E(q) = \frac{1}{n}\sum_{i=1}^n m\circ q(i).$$ </p><br> <p>By the symmetry of addition, $E$ is invariant to the action of $S_n$. Alternatively, other symmetric functions (i.e max pooling) can be used.</p> </div> <div class="col-md-5" align="left" markdown="1"> <img src="figs/PointNet_Overview.png" width="100%"/> </div> <p align="left">_{$^1$ Qi et. al. "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation."} <p align="left">_{$^2$ Wang et. al. "Dynamic Graph CNN for Learning on Point Clouds"} <p align="left">_{$^3$ Guerrero et. al. "PCPNET: Learning Local Shape Properties from Raw Point Clouds"} </div>

<h4 align="left">Pitfalls for Using These Approaches on Continuous Data</h4><hr> <div class="row"><p>For the remainder of the talk, we will consider surface data. In this context, we fix a 2-manifold $M$ and consider parameterized objects as $\Imm = \operatorname{Imm}(M,\R^3)$ and the group of reparameterizations as $\Diff= \operatorname{Diff}(M)$.</p><br> <div class="col-md-9" align="left" markdown="1"><br> <p class="fragment fade-left">One could apply PointNet (or similar models) to this data by sampling points on $M$ and operating directly on the resulting pointcloud.</p><br><br> <div class="col-md-11" align="left" markdown="1"> <p class="fragment fade-left">+ This approach ignores the continuous information and depends on the sampling method used.<br><br></p><br> <p class="fragment fade-left">+ Moreover, it is invariant to the action of $S_n$ on a point cloud but it is NOT invariant to the action of $\operatorname{Diff}(M)$.<br></p> </div> </div> <div class="col-md-3" markdown="1"><img src="figs/MeshVerts.png" width="60%"/> </div> <div class="col-md-9" align="left" markdown="1"> <p class="fragment fade-left">Despite this, state-of-the-art frameworks (i.e. 3DCoded$^1$) exist using this approach for continuous data.<br></p> </div> </div> <p align="left">_{$^1$Groueix, et al. "3D-CODED : 3D Correspondences by Deep Deformation."}<br>

<h4 align="left">Parameterization Invariant Representations from Geometric Measure Theory$^{1,2,3,4}$</h4><hr> <div class="row"> <div class="col-md-8" markdown="1" align="left"> <p>For any $q\in \Imm$ we can associate a varifold $\mu_q\in\mathcal{M}(\mathbb R^3\times S^{2})$ as the push forward measure $\mu_q:= (q,n_q)_*\operatorname{vol}_q$ where $n_q$ is the unit normal map of $q$. In particular, for any $\gamma\in\Diff$, $\mu_{q\circ\gamma} = \mu_q$.</p><br> <p class="fragment" data-fragment-index="1"><br>Unfortunately, to represent a varifold with a fixed finite dimension requires solving a quantization problem.</p><br> <p class="fragment" data-fragment-index="2"> <br>To avoid this problem, we will first need a distance function on the space of varifolds. We will equip $\mathcal{M}(\mathbb R^3\times S^{2})$ with the an RKHS norm $\|\cdot\|_{V}$ where for $p,q\in\Imm$ the scalar product between $\mu_q$ and $\mu_p$ is given by: \begin{equation*}\label{equ:norm_var} \langle \mu_{q},\mu_{p}\rangle_{V}=\iint_{M \times M}e^{-\alpha||q(x)-p(y)||^2}\langle n_q(x), n_p(y)\rangle^2\vol_{q}(x) \vol_{p}(y). \end{equation*}</p> </div> <div class="col-md-4" markdown="1" align="right"><img src="figs/Face_fullsample.png" width="40%" /><img src="figs/Face_downsampled.png" width="40%" /><hr><img src="figs/varifold_similar.png" width="100%" /></div> <p align="left">_{$^1$Charon & Trouvé. "The varifold representation of nonoriented shapes for diffeomorphic registration."}<br> _{$^2$Kaltenmark, et al. "A general framework for curve and surface comparison and registration with oriented varifolds."}<br> _{$^3$Feydy et al. "Optimal transport for diffeomorphic registration"}<br> _{$^4$ Roussillon & Glaunès. "Representation of surfaces with normal cycles and application to surface registration."}<br></p> </div>

<h4 align="left">Varifold Gradient For Representing Shapes</h4><hr> <div class="row" align = "left"> <p> To construct parameterization invariant representations of shapes with fixed dimension we consider the following setup. Consider a template $\mathcal{T}\in \Imm$ discretized by a triangular mesh with $M$ vertices and define the following operator: </p> <div class="col-md-12" markdown="1" align="center"> <p>$$\begin{align}\operatorname{VariGrad}:\Imm&\to\R^{3M}\\ q&\mapsto \nabla_{\vec{0}}\|\mu_q - \mu_{\mathcal{T} + (\cdot)}\|_V^2 = C - 2 \nabla_{\vec{0}}\langle\mu_q , \mu_{\mathcal{T}+(\cdot)}\rangle.\end{align}$$<br></p> <div> <div class="col-md-6" markdown="1" align="left"> <p class="fragment fade-left">+ $\operatorname{VariGrad}(q)$ is a vector field on $\mathcal{T}$ and thus has fixed dimension $3M$ independent of the dimensionality of our mesh representation of $q$. <br><br></p> <p class="fragment fade-left">+ Intuitively, this gives a first order approximation of how to deform $\mathcal{T}$ into the same shape as $q$. <br><br></p> <p class="fragment fade-left">+ Due to the properties of varifolds, it is parameterization blind and robust to imaging noise. <br><br></p> </div> <div class="col-md-6" markdown="1" align="center"><br><img src="figs/VarigradVisualization.png" width="100%"/></div> </div>

<h4 align="left">VariGrad for Surface Reconstruction</h4><hr> <div class="row"> <div class="col-md-12" markdown="1" align="left"> <p>Consider $\Imm_F \subset \Imm$ as the space of all surfaces of a specific modality which can be represented by a mesh with a fixed mesh structure $F$. Our goal will be to learn $f:\Imm \to \Imm_F$ such that $f(q)$ is the same shape as $q$.</p><br><br> </div> <div class="col-md-8" markdown="1" align="left"> <p>We attempt to learn such a function the DFAUST[1] dataset which consists of ~40,000 scans. This dataset includes mesh representations of each of the scans with a fixed mesh structure $F$.</p><br><br> <p>We train the models with ~80% of the registered DFAUST to minimize the mean square error of the vertex error.</p> </div> <div class="col-md-4" markdown="1" align="left"> <img src="figs/VariGrad_AutoEncode.png" width="100%"/> </div> </div> <p align="left">_{$^{1}$ Bogo, et al. "Dynamic FAUST: Registering Human Bodies in Motion"}<br>

<h4 align="left">VariGrad Results</h4><hr> <div class="row"> <div class="col-md-6" markdown="1" align="left"> <p>We test the model on the unseen testing set of registered data as well as an unseen testing set of unregistered scans of human bodies.</p> <span class="fragment fade-left"><img src="figs/NumericalResults.png" width="66%"/></span> <span class="fragment fade-left"><img src="figs/QualitativeResults.png" width="66%"/></span></div> <div class="col-md-6" markdown="1" align="left"><span class="fragment fade-left"><p>As an additional test of generalizability of both models, we select 100 unseen surfaces and resample each surface 100 times. </p><img src="figs/ReparamResults.png" width="100%"/><span class="fragment fade-left"><p>We test both the mean error on the whole dataset as well as the mean variance within each class.</p><img src="figs/Numerical2.png" width="80%"/></span></span><br></div> </div> <p align="left">_{$^{13}$Groueix, et al. "3D-CODED : 3D Correspondences by Deep Deformation."}<br>

<h4 align="left">VariGrad for Clustering and Classification</h4><hr> <div class="row"> <p>We consider a dataset of 3D surfaces representing left atrial appendages (LAA) of a large cohort (n=543) of patients. Several works have shown that morphological changes in the LAA are correlated to ischemic stroke[1,2].</p> <br><p>Previous works (i.e. [1]) have considered LAAs which are manually classified as one of four morphological types and shown there is significant difference in relative stroke rates of these clusters. </p> <div class="col-md-8" markdown="1" align="left"><br> <p>We perform two experiments using the VariGrad representations of the LAA data set:</p><br><br> <p>+ we perform automatic clustering of the samples into 4 clusters and compare the relative rates of stroke across each cluster.</p><br><br> <p>+ we train a support vector classifier on the varigrad representations of 143 LAAs supervised with ground truth labels. We test the performance of this classifier on the remaining 400 patients to validate the model. </p> </div> <div class="col-md-4" markdown="1" align="center"> <br> <img src="figs/LAA_varigrad_examples.gif" width="100%" /> <br> </div> </div> <div class="row" align = "left"> <p align="left">_{$^1$ Biase et al. "Does the Left Atrial Appendage Morphology Correlate With the Risk of Stroke in Patients With Atrial Fibrillation?: Results From a Multicenter Study"<br>} <p align="left">_{$^2$ Ahmad et al. "Elastic shape analysis computations for clustering left atrial appendage geometries of atrial fibrillation patients"} </div>

<h4 align="left">VariGrad for Clustering and Classification Results</h4><hr> <div class="row"> <div class="col-md-4" markdown="1" align="left"> <div class="fragment fade-left"> <p><b>Clustering:</b></p> <p>We perform automatic clustering based on the $L^2$ distance between VariGrad representations of the dataset. This produces similar clusters to the manual clustering with similar relative stroke rates between clusters.</p> <img src="figs/HeartClusterMeans.png" width="80%" /> </div> </div> <div class="col-md-4" markdown="1" align="left"> <div class="fragment fade-left"> <p><b>Classification:</b></p> <p>The SVM classifier produces an accuracy of $83.5\%$ with $10.25\%$ false positives and $6.25\%$ false negatives. <br><br> Additionally, we can visualize the disciminant direction of the SVM model as a vector field on the template. This vector field can be interpreted as the type of deformation that leads to higher risk of ischemic stroke. </p> </div> </div> <div class="col-md-4" markdown="1" align="center"> <div class="fragment fade-left"> <img src="figs/LAA_varigrad_examples.gif" width="100%" /> <br><br> <img src="figs/regression_LAA.gif" width="100%" /> </div> </div> </div>

<h4 align="left">Conclusion + Future Directions</h4><hr> <p align="left"> We propose using the gradient of the varifold distance as a parameterization invariant feature representations of surface data. <br><br> </p><div class="row"> <div class="col-md-8" markdown="1" align="left"> <p class="fragment fade-left"> + We demonstrate the strength of this method in deep learning applications.</p> <p class="fragment fade-left"> + Moreover, we show the sucess of these features for classic learning methods and discuss the interpretability of these results.</p> <p class="fragment fade-left"> - Show the sensitivity of the VariGrad representation with respect to the choice of template object.</p> <p class="fragment fade-left"> - Characterize the kernel of the VariGrad operator to understand what shapes the VariGrad operator cannot distinguish.</p> <hr> <p>_{This talk is based on:<br><a href="https://diglib.eg.org/handle/10.2312/3dor20231150">https://diglib.eg.org/handle/10.2312/3dor20231150</a><br> <a href="https://arxiv.org/pdf/2411.03475">https://arxiv.org/pdf/2411.03475</a><br> A Pytorch implementation VariGrad model for shape graphs is available at:<br> <a href="https://github.com/emmanuel-hartman/Pytorch_VariGrad">https://github.com/emmanuel-hartman/Pytorch_VariGrad</a><br> The slides for this talk are available at: <a href="https://emmanuel-hartman.github.io/Talks/Slides/VariGrad_Vienna/talk.html">https://emmanuel-hartman.github.io/Talks/Slides/VariGrad_Vienna/talk.html.</a>}</p> </div> <div class="col-md-4" markdown="1" align="center"> <img src="figs/VariGradOnPhoneScan.png" width="100%"/> <br><br> <center><p class="fragment fade-left">Thank you to the organizers for inviting me to speak and thank you for your attention!</p></center>