PATCHNETS: PATCH-BASED GENERALIZABLE DEEP IMPLICIT 3D SHAPE REPRESENTATIONS - EDGAR TRETSCHK AYUSH TEWARI VLADISLAV GOLYANIK MICHAEL ZOLLHÖFER ...

PatchNets: Patch-Based Generalizable
Deep Implicit 3D Shape Representations
   Edgar Tretschk Ayush Tewari      Vladislav Golyanik
  Michael Zollhöfer Carsten Stoll   Christian Theobalt

Introduction
 Geometry representations for neural networks:
  – Mesh:
      Integration into network? Graph convolutions?
      Generating meshes?
  – Point cloud:
      No continuous surface
  – Voxel grid:
      Computationally expensive

                                                       2

Related Work
 Global approaches, e.g.:
  – DeepSDF (Park et al. 2019)
  – OccupancyNetwork (Mescheder et al. 2019)
  –  PatchNets generalize better
 Patch-based approaches, e.g.:
  – AtlasNet (Groueix et al. 2018)
  – Structured Implicit Functions (Genova et al. 2019)
  – Deep Structured Implicit Functions (Genova et al. 2020)
  –  PatchNets are more flexible
                                                              3

Introduction

               4

Method
           Deep implicit representation (DeepSDF):

                                Single
                                                      8x FC
                                Object:

                                Multiple
                                Objects:

Park et al. (2019)                                            5

Method
State-of-the-Art Global Approach                                           Our Patch-Based Approach

                                                                  latent
latent

             latent

                      latent

                                                                                        latent
                                                                               latent
                                                       SDF(x′′)
                                             SDF(x′)
                                   SDF(x)

                                                                                                                        SDF(x′′)
                               …                                                                 …

                                                                                                              SDF(x′)
                                                                                                     SDF(x)
 x           x′       x′′                                          x           x′       x′′

                                                 x′′                            x′′                            x′′
              x′′

                                            x′                             x′                             x′
         x′

                                                                           x                            x
         x                                  x

                                                                                                                                   6

Method: Optimization
 Training data: 1300 shapes from ShapeNet
   – Preprocessing: generate (x,y,z,SDF) samples from mesh
 Losses: reconstruction loss & guidance losses for patch extrinsics
 Auto-decoder:
   – No encoder  per-object patch parameters are free variables
        Optimized during training (jointly with shared network weights)
   – Test time: optimize patch parameters, while keeping network weights fixed

                             Patch 1          Patch 2          Patch 3
                              Scale            Scale            Scale
             Patch
           Extrinsics
                             Position
                             Rotation
                                             Position
                                             Rotation
                                                               Position
                                                               Rotation
                                                                            …
                              Latent          Latent            Latent
                                                                                 7

Results: Surface Reconstruction
 Quantitative results on 13 ShapeNet categories
                                  IoU     Chamfer       F-score
        OccNet (Mescheder et al. 2019)   77.8   0.049    81.9
        SIF (Genova et al. 2019)         66.0   0.118    59.0
        DSIF (Genova et al. 2020)        90.0   0.040    92.2
        DeepSDF (Park et al. 2019)       77.4   0.075    89.9
        Global-patch Baseline            76.5   0.111    80.6
        Ours                             92.1   0.044    94.8

                                                                  8

Results: Surface Reconstruction

ShapeNet
(Chang et al. 2015)

                                                   9

Results: Surface Reconstruction

Dynamic FAUST
(Bogo et al. 2017)

                                                   10

Results: Generalization Across Categories

                                            11

Results: Generalization from Less Training Data

                                                  12

Results: Generalization from Less Training Data

                                                  13

object
              latent
                ObjectNet

     extrinsics             latent

     extrinsics             latent

     extrinsics             latent
        x

                  latent
                                     Applications: Object Priors

        x′

                  latent
        x′′

                  latent
                   …

              SDF(x)
              SDF(x′)
              SDF(x′′)
15

Applications: Object Priors (Interpolation)

                                              16

Applications: Object Priors (Depth Completion)

                     latent

                                  latent

                                               latent

                                                                          latent

                                                                                   latent

                                                                                            latent
                                                                                                     …

                                                                                                                              SDF(x′′)
object

                                                                                                                    SDF(x′)
latent

                                                                                                           SDF(x)
         ObjectNet

                     extrinsics

                                  extrinsics

                                               extrinsics
                                                                           x       x′       x′′

Global Baseline                                             Ours (Unrefined)                             Ours (Refined)

                                                                                                                                         17

Applications: Articulated Deformations

                                         19

Applications: Large Scenes (Preliminary)

Ground-truth                          Patches
(ICL-NUIM,
Handa et al. 2014)                    (Ours)

    DeepSDF                              Ours

                                                           21

Limitations
 SDF is only well-defined for watertight shapes
 Only modifying extrinsics cannot model large
  non-rigid deformations
 Generalizability at test time comes at the cost
  of having to optimize for patch parameters
   – 1 shape in 10 minutes, 650 shapes in 71 minutes
     (due to batching)

                                                       23

Conclusion
 Novel deep implicit shape representation
   – Replaces global latent vector with local patches
 Results:
   – Higher quality
   – Improved generalizability
   – Requires less training data
 Numerous applications:
   – Interpolation
   – Partial point cloud completion
   – Articulated deformations

                                                        24

For more details and results, check out our
 paper, supplemental video and material!

Thank you for your attention!

                                              25

References
 Bogo et al. (2017), Dynamic FAUST: Registering human bodies in motion (CVPR 2017)
 Chang et al. (2015), ShapeNet: An information-rich 3D model repository (arXiv 2015)
 Genova et al. (2019), Learning Shape Templates with Structured Implicit Functions (ICCV
  2019)
 Genova et al. (2020), Local Deep Implicit Functions for 3D Shape (CVPR 2020)
 Groueix et al. (2018), A papier-mache approach to learning 3D surface generation (CVPR
  2018)
 Handa et al. (2014), A Benchmark for RGB-D Visual Odometry, 3D Reconstruction and SLAM
  (ICRA 2014)
 Park et al. (2019), DeepSDF: Learning Continuous Signed Distance Functions for Shape
  Representation (CVPR 2019)
 Mescheder et al. (2019), Occupancy Networks: Learning 3D Reconstruction in Function
  Space (CVPR 2019)
                                                                                            26