PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space

Date: 2017
Journal: CVPR

1 Introduction

Exploiting local structure has proven to be important for the success of convolutional architectures

CNN takes data defined on regular grids as the input an is able to progressively capture features at increasingly larger scales along a multi resolution hierarchy

PointNet++ is a hierarchical neural network that process a set of points sampled in a metric space in a hierarchical fashion

First partition the set of points into overlapping local regions by distance metric of the underlying space

Two issues are addressed by PointNet++

1. How to generate the partitioning of the point set
2. How to abstract sets of points or local features through a local feature learner

PointNet++ apples PointNet recursively on a nested partitioning of the input set

Unlike CNNs, where smaller kernels often enhance performance, point cloud data can be sparse, making small scales inadequate

PointNet++ addresses this by using multi-scale neighborhoods, adapting to different scales during training, and achieving superior results on 3D point cloud benchmarks

2. Problem Statement

3. Method

Extension of PointNet with added hierarchical structure

3.1 Review of PointNet

Invariant to point permutations and can arbitrarily approximate any continuous set function

Lacks the ability to capture local context at different scale

3.2 Hierarchical Point Set Feature Learning

Use a hierarchical grouping of points and progressively abstract larger and larger local region along the hierarchy

Hierarchical structure is composed by a number of set abstraction levels

Three key layers: Sampling layer, Grouping layer, PointNet layer

Sampling layer

Iterative farthest point sampling to choose a subset of points

Generates receptive fields in a data dependent manner

Grouping layer

Grouping input point set ($N \times (d~ +~C)$) matrix into output ($N' \times K \times (d~+~C')$)

$K$ is the number of points in neighborhood

PointNet layer

Local feature is abstracted by its centroid, and that encode the centroid’s neighbourhood

Output size of $N' \times (d~+~C')$

3.3 Robust Feature Learning under Non-Uniform Sampling Density

MSG Multi-scale grouping

Capture multi scale patterns by applying grouping layers with different scales followed by according PointNet to extract features of each scale

Concatenated to form a multi scale feature

Optimize with random input dropout

MRG Multi-resolution grouping

The # og centroid points is usually large at the lowest level, which cause time cost increase

Use multi resolution grouping

One vector from summarizing features at each subregion from lower level

One vector from directly processing raw points in local region

3.4 Point Feature Propagation for set Segmentation