Structure-from-Motion using Dense CNN Features with Keypoint Relocalization

Structure from Motion (SfM) using imagery that involves extreme appearance changes is yet a challenging task due to a loss of feature repeatability. Using feature correspondences obtained by matching densely extracted convolutional neural network (CNN) features significantly improves the SfM reconstruction capability. However, the reconstruction accuracy is limited by the spatial resolution of the extracted CNN features which is not even pixel-level accuracy in the existing approach. Providing dense feature matches with precise keypoint positions is not trivial because of memory limitation and computational burden of dense features. To achieve accurate SfM reconstruction with highly repeatable dense features, we propose an SfM pipeline that uses dense CNN features with relocalization of keypoint position that can efficiently and accurately provide pixel-level feature correspondences. Then, we demonstrate on the Aachen Day-Night dataset that the proposed SfM using dense CNN features with the keypoint relocalization outperforms a state-of-the-art SfM (COLMAP using RootSIFT) by a large margin.

In this decade, the performance of local features, namely, SIFT [11] and its variants [16,21,22,23,24] are validated on 3D reconstruction as well as many other tasks [25,26,27]. The local features give promising matches for well-textured surfaces/objects but significantly drop its performance for matching weaklytextured objects [28], repeated patterns [29], extreme changes of viewpoints [21,30,31] and illumination change [32,33] because of degradation in repeatability of feature point (keypoint) extraction [21,31]. This problem can be mitigated by using densely detected * Correspondence: widya.a.aa@m.titech.ac.jp Tokyo Institute of Technology, O-okayama, Meguro-ku, 152-8550 Tokyo, Japan Full list of author information is available at the end of the article features on a regular grid [34,35] but their merit is only demonstrated in image retrieval [32,36] or image classification tasks [26,34] that use the features for global image representation and do not require oneto-one feature correspondences as in SfM.
Only recently, SfM with densely detected features are presented in [37]. DenseSfM [37] uses convolutional neural network (CNN) features as densely detected features, i.e., it extracts convolutional layers of deep neural network [38] and converts them as feature descriptors of keypoints on a grid pattern (Section 3.1). As the main focus of [37] is camera localization, the SfM architecture including neither dense CNN feature description and matching nor its 3D reconstruction performance is not studied in detail.
Contribution. In this work, we first review the details of the SfM pipeline with dense CNN feature extraction and matching. We then propose a keypoint relocalization that uses the structure of convolutional layers (Section 3.2) to overcome keypoint inaccuracy on the grid resolution and computational burden of dense feature matching. Finally, the performance of SfM with dense CNN feature using the proposed keypoint relocalization is evaluated on Aachen Day-Night [37] dataset and additionally on Strecha [39] dataset. Pipeline of the proposed SfM using dense CNN features with keypoint relocalization. Our SfM starts from dense feature extraction (Section 3.1), feature matching (Section 3.2), the proposed keypoint relocalization (Section 3.3), feature verification using RANSAC with multiple homographies (Section 3.4) followed by 3D reconstruction (Section 3.5).

Related work
SfM and VisualSLAM. The state-of-the-art SfM is divided into a few mainstream pipelines: incremental (or sequential) [4,6,40], global [8,9,41], and hybrid [10,42]. In VisualSLAM approaches, namely, LSD-SLAM [43] and DTAM [44] repeat camera pose estimation based on selected keyframe and (semi-)dense reconstruction using the pixel-level correspondences in real-time. These methods are particularly designed to work with video streams, i.e., short baseline camera motion, but not with general wide-baseline camera motion.
Recently, [37] introduces CNN-based DenseSfM that adopts densely detected and described features. But, their SfM uses fixed poses and intrinsic parameters of reference images in evaluating the performance of query images localization. They also do not address keypoint inacuraccy of CNN features. Therefore, it remains as an open challenge.
Feature points. The defacto standard local feature, SIFT [11], is capable of matching images under viewpoint and illumination changes thanks to scale and rotation invariant keypoint patches described by histograms of the oriented gradient. ASIFT [21] and its variants [30,31] explicitly generate synthesized views in order to improve repeatability of keypoint detection and description under extreme viewpoint changes.
An alternative approach to improve feature matching between images across extreme appearance changes is to use densely sampled features from images. Densely detected features are often used in multi-view stereo [45] with DAISY [46], or image retrieval and classification [35,47] with Dense SIFT [34]. However, dense features are not spotlighted in the task of oneto-one feature correspondence search under unknown camera poses due to its loss of scale, rotation invariant, inaccuracy of localized keypoints, and computational burden.
Those studies motivate us to adopt CNN architecture for extracting features from images and matching them for SfM as it efficiently outputs multi-resolution features and has potential to be improved by better training or architecture.

The pipeline: SfM using dense CNN features with keypoint relocalization
Our SfM using densely detected features mimics the state-of-the-art incremental SfM pipeline that consists of feature extraction (Section 3.1), feature matching (Section 3.2 to 3.4), and incremental reconstruction (Section 3.5). Figure 1 overviews the pipeline. In this section, we describe each component while stating the difference to the sparse keypoint based approaches.

Dense feature extraction
Firstly, our method densely extracts the feature descriptors and their locations from the input image. In the same spirit of [53,54], we input images in a modern CNN architecture [38,55,56] and use the convolutional layers as densely detected keypoints on a regular grid, i.e., cropping out the fully connected and softmax layers. In the following, we chose VGG-16 [38] as the base network architecture and focus on the description tailored to it, but this can be replaced with other networks with marginal modification. As illustrated in Figure 2, VGG-16 [38] is composed of five max-pooling layers and 16 weight layers. We extract the max-pooling layers as dense features. As can be seen in Figure 2, the conv1 max-pooling layer is not yet the same resolution as the input image. We, therefore, also extract conv1 2, one layer before the conv1 max-pooling layer, that has pixel-level accuracy.

Tentative matching
Given multi-level feature point locations and descriptors, tentative matching uses upper max-pooling layer (lower spatial resolution) to establish initial correspondences. This is motivated by that the upper maxpooling layer has a larger receptive field and encodes more semantic information [48,57,58] which potentially gives high matchability across appearance changes. Having the lower spatial resolution is also advantageous in the sense of computational efficiency.
For a pair of images, CNN descriptors are tentatively matched by searching their nearest neighbors (L2 distances) and refined by taking mutually nearest neighbors. Note that the standard ratio test [11] removes too many feature matches as neighborhood features on a regularly sampled grid tend to be similar to each other.
We perform feature descriptor matching for all the pairs of images or shortlisted images by image retrieval, e.g., NetVLAD [53].

Keypoint relocalization
The tentative matching using the upper max-pooling layers, e.g., conv5, generates distinctive correspondences but the accuracy of keypoint position is limited by their spatial resolution. This inaccuracy of keypoints can be mitigated by a coarse-to-fine matching from the extracted max-pooling layer up to conv1 2 layer utilizing extracted intermediate max-pooling layers between them.
For example, the matched keypoints found on the conv3 layer are transferred to the conv2 (higher spatial resolution) and new correspondences are searched only in the area constrained by the transferred keypoints. This can be repeated until reaching conv1 2 layer. However, this naive coarse-to-fine matching generates too many keypoints that may lead to a problem in computational and memory usage in incremental SfM step, especially, bundle adjustment.
To generate dense feature matches with pixel-level accuracy while preserving their quantity, we propose a method of keypoint relocalization as follows.
For each feature point at the current layer, we retrieve the descriptors on the lower layer (higher spatial resolution) in the corresponding K × K pixels [1] . The feature point is relocalized at the pixel position that has the largest descriptor norm (L2 norm) in the K×K pixels. This relocalization is repeated until it reaches the conv1 2 layer which has the same resolution as the input image (see also Figure 3). [1] We use K = 2 throughout the experiments.

Feature verification using RANSAC with multiple
homographies Using all the relocated feature points, we next remove outliers from a set of tentative matches by Homography-RANSAC. We rather use a vanilla RAN-SAC instead of the state-of-the-art spatial verification [59] by taking into account the spatial density of feature correspondences. To detect inlier matches lying on several planes, Homography-RANSAC is repeated while excluding the inlier matches of the best hypothesis. The RANSAC inlier/outlier threshold is set to be loose to allow features off the planes.

3D reconstruction
Having all the relocalized keypoints filtered by RAN-SAC, we can export them to any available pipelines that perform pose estimation, point triangulation, and bundle adjustment.
Dense matching may produce many confusing feature matches on the scene with many repetitive structures, e.g., windows, doors, pillars, etc. In such cases, we keep only the N best matching image pairs for each image in the dataset based on the number of inlier matches of multiple Homography-RANSAC.

Experiments
We implement feature detection, description and matching (Sections 3.1 to 3.4) in MATLAB with third-party libraries (MatConvNet [60] and Yael library [61]). Dense CNN features are extracted using the VGG-16 network [38]. Using conv4 and conv3 max-pooling layers, feature matches are computed by the coarse-to-fine matching followed by multiple Homography-RANSAC that finds at most five homographies supported by an inlier threshold of 10 pixels. The best N pairs based on multiple Homography-RANSAC of every image are imported to COLMAP [6] with the fixed intrinsic parameter option for scene with many repetitive structures. Otherwise, we use all the image pairs.
In our preliminary experiments, we tested other layers having the same spatial resolution, e.g., using conv4 3 and conv3 3 layers in the coarse-to-fine matching but we observed no improvement in 3D reconstruction. As a max-pooling layer has a half depth dimension in comparison with the other layers at the same spatial resolution, we chose the max-pooling layer as the dense features for efficiency.
In the following, we evaluate the reconstruction performance on Aachen Day-Night [37] and Strecha [39] dataset. We compare our SfM using dense CNN features with keypoint relocalization to the baseline COLMAP with DoG+RootSIFT features [6]. In addition, we also compare our SfM to SfM using dense CNN without keypoint relocalization [37]. All experiments are tested on a computer equipped with a 3.20GHz Intel Core i7-6900K CPU with 16 threads and a 12GB GeForce GTX 1080Ti.

Results on Aachen Day-Night dataset
The Aachen Day-Night dataset [37] is aimed for evaluating SfM and visual localization under large illumination changes such as day and night. It includes 98 subsets of images. Each subset consists of 20 day-time images and one night-time image, their reference camera poses, and 3D points [2] . For each subset, we run SfM and evaluate the estimated camera pose of the night image as follows. First, the reconstructed SfM model is registered to the reference camera poses by adopting a similarity transform obtained from the camera positions of day-time images. We then evaluate the estimated camera pose of the night image by measuring positional (L2 distance) and angular (acos( )) error. Table 1 shows the number of reconstructed cameras. The proposed SfM with keypoint relocalization (conv1 2) can reconstruct 96 night images that are twice as many as that of the baseline method using COLMAP with DoG+RootSIFT [6]. This result validates the benefit of densely detected features that can provide correspondences across large illumination changes as they have smaller loss in keypoint detection repeatablity than a standard DoG. On the other hand, both methods with sparse and dense features work well for reconstructing day images. The difference between with and without keypoint localization can be seen more clearly in the next evaluation. Figure 4 shows the percentages of night images reconstructed (y-axis) within certain positional and angular error threshold (x-axis). Similarly, Table 2 shows [2] Although the poses are carefully obtained with manual verification, the poses are called as "reference poses" but not ground truth. the reconstruction percentages of night images for varying distance error thresholds with a fixed angular error threshold at ten degrees. As can be seen from both evaluations, the proposed SfM using dense CNN features with keypoint relocalization outperforms the baseline DoG+RootSIFT [6] by a large margin. The improvement by the proposed keypoint relocalization is significant when the evaluation accounts for pose accuracy. Notice that the SfM using dense CNN without keypoint relocalization [37] performs worse than the baseline DoG+RootSIFT [6] at small thresholds, e.g., below 3.5 meters position and 2 o angular error. This indicates that the proposed keypoint relocalization gives features at more stable and accurate positions and provides better inlier matches for COLMAP reconstruction which results 3D reconstruction in higher quality. Figure 5 illustrates the qualitative comparison result between our method and the baseline DoG+Root-SIFT [6].

Results on Strecha dataset
We additionally evaluate our SfM using dense CNN with the proposed keypoint relocalization on all six subsets of Strecha dataset [39] which is a standard benchmark dataset for SfM and MVS. Position and angular error between the reconstructed cameras and the ground truth poses are evaluated. In our SfM, we take only feature matches from the best N = 5 image pairs for each image to suppress artifacts from confusing image pairs. The mean average position and angular errors resulted by our SfM are 0.59 meters and 2.27 degrees. Although these errors are worse than those of the stateof-the-art COLMAP with DoG+RootSIFT [6] which are 0.17 meters and 0.90 degrees, the quantitative evaluation on the Strecha dataset demonstrated that our SfM does not overfit to specific challenging tasks but works reasonably well for standard (easy) situations.

Conclusion
We presented a new SfM using dense features extracted from CNN with the proposed keypoint relocalization to improve the accuracy of feature positions sampled on a regular grid. The advantage of our SfM has demonstrated on the Aachen Day-Night dataset that includes images with large illumination changes. The result on the Strecha dataset also showed that our SfM works for standard datasets and does not overfit to a particular task although it is less accurate than the state-of-the-art SfM with local features. We wish the proposed SfM becomes a milestone in the 3D reconstruction, in particularly, challenging situations.