I initially set out to use PointNet/PointNet++ to predict amodal 3D bounding boxes on point-clouds, utilizing the SUN RGB-D 3D dataset. My approach was similar to that of Faster R-CNN. However, I did not want to descritize the search space and therefore attempted to use the points in the point-clouds as anchors for the bounding boxes. This approach ended up being too challenging for the network to easily learn. Frustum PointNet presents a very interesting solution to this problem and leverages the 2D rgb image to descritize the search space.
During this project, I gained a deep understand of PointNets and wrote of own implement of PointNet++ in PyTorch so I could learn PyTorch and its CUDA interoperability bridge.