Supervoxel-based segmentation of 3D imagery with optical flow integration for spatiotemporal processing

2.1 Normalized cuts and gPb-UCM

Normalized cuts [2] gained immense popularity in 2D image segmentation by treating the image as a graph and computing hard partitions. The gPb-UCM framework [1] leveraged this approach in a soft manner to introduce globalization into their contour detection process. This led to a drastic reduction in the oversegmentation arising out of gradual texture or brightness changes in the previous approaches. Being a computationally expensive method, the last decade saw the emergence of several techniques to speed up the underlying spectral decomposition process in [1]. These techniques range from various optimization techniques like multilevel solvers [8, 9], to systems implementations exploiting GPU parallelism [3], and to approximate methods like reduced order normalized cuts [4, 10].

2.2 3D volumetric image segmentation

Segmentation techniques applied to 3D volumetric images can be found in the literature of medical imaging (usually MRI) and 2D+time video sequence segmentation. In 3D medical image segmentation, unsupervised techniques like region growing [11] have been well studied. In [12], normalized cuts were applied to MRIs but gained little attention. Recent literature mostly focuses on supervised techniques [13, 14]. In video segmentation, there are two streams of works. On the one hand, Galasso et al. and Khoreva et al.’s studies [15–18] are frameworks that rely on segmenting each frame into superpixels in the first place. Therefore, they are not applicable to general 3D volumes. On the other hand, a variety of other methods treat video sequences as spatiotemporal volumes and try to segment them into supervoxels. These methods are mostly extensions of popular 2D image segmentation approaches, including graph cuts [19, 20], SLIC [21], mean shift [22], graph-based methods [6], and normalized cuts [9, 23]. Besides these, temporal superpixels [24–26] are another set of effective approaches that extract high-quality spatiotemporal volumes. A comprehensive review of supervoxel methods can be found in [27, 28]. All of this development in the area of 3D supervoxels has led to a general consensus in the video segmentation community that supervoxels have favorable properties which can be leveraged later in the pipeline. These characteristics are summarized as follows: (i) supervoxel boundaries should stick to meaningful image boundaries; (ii) regions within one supervoxel should be homogeneous while inter-supervoxel differences should be substantially larger; (iii) it is important that supervoxels have regular topologies; (iv) a hierarchy of supervoxels is favored as different applications have different supervoxel granularity preferences.

2.3 Deep neural networks

Recently, deep learning, a re-branding of neural networks (NNs), has seen successful application in numerous areas, especially in computer vision and natural language processing. Typical approaches, such as support vector machines (SVMs), Gaussian mixture models, hidden Markov models, and conditional random fields, mostly have “shallow” architectures that have only one to two processing layers between the features and the output. Deep neural networks, e.g., convolutional neural networks (CNNs) and recurrent neural networks (RNNs), usually have many layers which allow for a rich variety of highly complex, nonlinear, and hierarchical features to be learned from the data. Video segmentation problems involve several subproblems which can be classified as semantic segmentation [29] and general object segmentation [30, 31]. For video object segmentation problems, convolutional neural network (CNN)-based approaches have made great advances in both unsupervised learning [30, 32–35] and semi-supervised learning [36–38]. CNNs have recently been integrated with gPb-UCM. Convolutional oriented boundaries (COB) [39, 40] is a generic CNN architecture that allows end-to-end learning of multiscale oriented contours and provides state of the art contours and region hierarchies. It uses a pipeline similar to gPb-UCM but replaces the contour probability maps by CNN maps from VGGNet [41] and ResNet [42]. We introduced transfer learning to 3D UCM without the intermediate supervised CNN layers as in COB. Our very preliminary investigation into using transfer learning (re-application of discriminative deep learning features) in 3D UCM was unsuccessful in outperforming unsupervised 3D UCM. However, due to rapid developments that are ongoing in this space, a definitive conclusion cannot be reached at this time. Since the basic approach of adding deep network features to our 3D UCM pipeline is extremely straightforward, we plan to revisit this in the future to see if significant improvements can be achieved over and beyond our current unsupervised learning UCM pipeline.

3.1 Local gradient feature extraction

The UCM framework begins with gradient-based edge detection to quantify the presence of boundaries. Most gradient-based edge detectors in 2D [43, 44] can be extended to 3D for this purpose. The 3D gradient operator used in this work is based on the mPb detector proposed in [1, 45] which has been empirically shown to have superior performance in 2D.

The building block of the 3D mPb gradient detector is an oriented gradient operator G(x,y,z,θ,φ,r) described in detail in Fig. 1. To be more specific, in a 3D volumetric or spatiotemporal intensity field, we place a sphere centered at each voxel to denote its neighborhood. An equatorial plane specified by its normal vector \(\vec {t}(\theta,\varphi)\) splits the sphere into two half spheres. We compute the intensity histograms for both half spheres, denoted as g and h. Then we define the gradient magnitude in the direction \(\vec {t}(\theta,\varphi)\) as the χ² distance between g and h:

$$ \chi^{2}(\mathbf{g},\mathbf{h})=\frac{1}{2}\sum_{i}\frac{(g(i)-h(i))^{2}}{g(i)+h(i)}. $$

(1)

In our experiments, θ and φ take values in \(\left \{ 0,\frac {\pi }{4},\frac {\pi }{2},\frac {3\pi }{4}\right \} \) and \(\left \{ -\frac {\pi }{4},0,\frac {\pi }{4}\right \} \) respectively and in one special case, \(\varphi =\frac {\pi }{2}\). Therefore, we compute local gradients in 13 different directions. Neighborhood values of 2, 4, and 6 voxels were used for r. Equal weights α _r were used to combine gradients across different scales. Also, as is standard, we always apply an isotropic Gaussian smoothing filter with σ=3 voxels before any gradient operation.

3.2 Globalization using a reduced order eigensystem

The globalization core of gPb-UCM is driven by nonlinear dimensionality reduction (closely related to spectral clustering). The local cues obtained from the gradient feature detection phase are globalized (and therefore emphasize the most salient boundaries in the image) by computing generalized eigenvectors of a graph Laplacian (obtained from the normalized cuts principle) [2]. However, this approach depends on solving a sparse eigensystem at the scale of the number of pixels in the image. Thus, as the size of the image grows larger, the globalization step becomes the computational bottleneck of the entire process. This problem is even more severe in the 3D setting because the voxel cardinality far exceeds the pixel cardinality of our 2D counterparts. An efficient approach was proposed in [4] to reduce the size of the eigensystem while maintaining the quality of the eigenvectors used in globalization. We generalize this method to 3D so that our approach becomes scalable to handle sizable datasets.

In the following, we describe the globalization steps: (i) graph construction and oriented intervening contour cue, (ii) reduced order normalized cuts and eigenvector computation, (iii) scale-space gradient computation on the eigenvector image, and (iv) the combination of local and global gradient information. For the most part, this pipeline mirrors the transition from mPb to gPb with the crucial difference being the adoption of reduced order normalized cuts.

3.2.1 Graph construction and the oriented intervening contour cue

In 2D, the normalized cuts approach begins with sparse graph construction obtained by connecting pixels that are spatially close to each other. gPb-UCM [1] specifies a sparse symmetric affinity matrix W using the intervening contour cue [5] which is the maximal value of mPb along a line connecting the two pixels i,j at the ends of relation W _ij . However, this approach does not utilize all of the useful information obtained from the previous gradient feature detection step. Figure 2 describes a potential problem and our resolution. To improve the accuracy of the affinity matrix, we take the direction vector of the maximum gradient magnitude into consideration when calculating the pixel-wise affinity value. This new variant is termed the oriented intervening contour cue. For any spatially close voxels i and j, we use \(\bar {ij}\) to denote the line segment connecting i and j. \(\vec {d}\) is defined as the unit direction vector of \(\bar {ij}\). Assume P is a set of voxels that lie close to \(\bar {ij}\). For any p∈P, \(\vec {n}\) is the unit direction vector associated with its mPb value. We define the affinity value W _ij between i and j as follows:

$$ W_{ij}=\exp\left(-\max_{p\in P}\{mPb(p)|\langle\vec{d},\vec{n}\rangle|\}/\rho\right) $$

(4)

where 〈·,·〉 is the inner product operator of the vector space and ρ is a scaling constant. In our experiments, the set P contains the voxels that are at most 1 voxel away from \(\bar {ij}\). ρ is set to 0.1. In the affinity matrix W, each voxel is connected to voxels that fall in the 5×5×5 cube centered at that voxel. Thus the graph defined by W is very sparse.

3.2.3 Scale space gradient computation on the eigenvector image

We solve for the generalized eigenvectors \(\{\vec {x_{0}},\vec {x_{1}},\ldots,\vec {x_{n}}\}\) of the system in (6) corresponding to the smallest eigenvalues \(\left \{\lambda _{0}^{\prime },\lambda _{1}^{\prime },\ldots,\lambda _{n}^{\prime }\right \}\). As stated in [4], λ _i in (5) will be equal to \(\lambda _{i}^{\prime }\) and \(L\vec {x_{i}}\) will match \(\vec {v_{i}}\) modulo an irrelevant scale factor, where \(\vec {v_{i}}\) are the eigenvectors of the original eigensystem (5). Similar to the 2D scenario [1], eigenvectors \(\vec {v_{i}}\) carry surface information. Figure 3 shows several example eigenvectors obtained from two types of 3D volumetric datasets. In both cases, the eigenvectors distinguish salient aspects of the original image. Based on this observation, we apply the gradient operator mPb defined in (3) to the eigenvector images. The outcome of this procedure is denoted as sPb because it represents the spectral component of the boundary detector, following the convention established in [1]:

$$ sPb(x,y,z)=\sum_{i=1}^{K}\frac{1}{\sqrt{\lambda_{i}}}{mPb}_{\vec{v_{i}}}(x,y,z). $$

(7)

Note that this weighted summation starts from i=1 because λ₀ always equals 0 and \(\vec {v_{0}}\) is a vanilla image. The weighting by \(1/\sqrt {\lambda _{i}}\) is inspired by the mass-spring system in mechanics [1, 46]. In our experiments, we use 16 eigenvectors, i.e., K=16.

3.2.4 The combination of local and global gradient information

The last step is to combine local cues mPb and global cues sPb. mPb tries to capture local variations while sPb aims to obtain salient boundary surfaces. By linearly combining them together, we get a globalized boundary detector gPb:

$$ gPb(x,y,z)=\omega mPb(x,y,z)+(1-\omega)sPb(x,y,z). $$

(8)

In practice, we use equal weights for mPb and sPb. After obtaining the gPb values, we apply a post-processing step of non-maximum suppression [43] to get thinned boundary surfaces when the resulting edges from mPb are too thick. Figure 4 shows some examples of mPb, sPb and gPb.

3.3 Supervoxel agglomeration

At this point, 2D gPb-UCM proceeds with the oriented watershed transform (OWT) [1, 47, 48] to create a hierarchical segmentation of the image resulting in the ultrametric contour map. However, we find that the same strategy does not work well in 3D. The reasons are twofold. First, because of the presence of irregular topologies, it is more difficult to approximate boundary surfaces with square or triangular meshes in 3D than to approximate boundary curves with line segments in 2D. Second, following OWT, during superpixel merging, only the information of the pixels on boundaries are used during greedy boundary removal. This is not a robust design especially when we take into account the fragmentation of boundary surfaces in 3D.

In practice, we increase the granularity parameter τ by a factor of 1.1 in each iteration. This agglomeration process iteratively progresses until no edge meets the merging criterion. The advantage of graph-based methods is that they make use of the information in all voxels in the merged regions. Furthermore, as shown in the experiments below, we see that it overcomes the weakness of fragmented supervoxels of previous graph-based methods. This is because traditional graph-based methods are built on voxel-level graphs.

Finally, we obtain a supervoxel hierarchy represented by a bottom-up tree structure. This is the final output of the 3D UCM algorithm. The granularity of the segmentation is a user-driven choice guided by the application.

4.1 Optical flow computation

Optical flow has a complex relationship to the actual 3D motion field since it represents apparent motion caused by intensity change [49, 50]. Optical flow has been a central topic in computer vision since the 1980s and has considerably matured since then [51]. The basic approach consists of estimating a vector field (velocity vectors at each grid point) which is subsequently related to object motion.

In general, optical flow can be divided into two types, sparse optical flow and dense optical flow [51, 52]. Briefly, the sparse optical flow method only computes flow vectors at some “interesting” pixels in the image, while the dense optical flow method estimates the flow field at every pixel and is more accurate [53, 54]. We use dense optical flow here since we wish to integrate with scene segmentation.

4.2 Supervoxel segmentation

In recent work on video segmentation problems, motion estimation has emerged as a key ingredient for state of the art approaches [7, 55, 56] for both deep learning methods and typical methods. We introduced optical flow estimation to the pipeline of 3D OF UCM. On top of 3D UCM, we first compute optical flow and then consider to employ it as the source of an additional cue to guide the segmentation.

Given a video sequence, we compute the optical flow and combine RGB channels with the magnitude of the optical flow as input to 3D UCM. We use the state of the art Classic+NL-fast method [57] to estimate the optical flow. Figure 5 shows the examples of optical flow computed for video sequences from [31, 58]. This is repeated across multiple frames followed by interpolation to bring the set of frames into register [59]. We used color coding in [60] for optical flow in order to aid visualization. Subsequently, 3D UCM is applied to obtain supervoxel segmentation of spatiotemporal imagery.

Since our input data has additional optical flow information (a motion channel), we usually achieve better segmentation results especially for some small regions. As optical flow provides complementary information to RGB images, 3D OF UCM improves the overall performance. Datasets, evaluation metrics, and comparison results are presented in Section 5.

5.1 Experimental setup

5.2 Qualitative comparisons

We present some example segmentation results for three datasets: IBSR, BuffaloXiph, and DAVIS. To compare GBH, SWA, and 3D UCM, we present some slices from both the IBSR and BuffaloXiph datasets in Fig. 6. It shows that, through different levels, there are significant differences in the segmentation results of all three methods. However, it is difficult to say which method is the best compared with the ground truth. It should be noted that GBH has a fragmentation problem in the IBSR and BuffaloXiph datasets. On the other hand, SWA has a clean segmentation in IBSR but suffers from fragmentation in video sequences. In contrast, 3D UCM has the most regular segmentation.

And in Figs. 7 and 8, from top to bottom, we show fine-level to coarse-level supervoxel segmentation comparison of these methods on IBSR and BuffaloXiph datasets respectively. The results show that all four methods differ markedly in the kind of segmentations they obtain. For brain frames the IBSR dataset, the visual comparison shows that 3D UCM performs well especially on the globalization aspect, such as having no oversegmentation in the background. As can be seen, GBH and SWA both have a fragmentation problem in the BuffaloXiph datasets through different levels. In contrast, 3D UCM has the most regular segmentation and 3D OF UCM has the finest segmentation even in small detailed regions. In general, 3D UCM generates meaningful and compact supervoxels at all levels of the hierarchy while GBH and SWA have a fragmentation problem at the lower levels. To demonstrate the improved supervoxel agglomeration of 3D OF UCM compared to 3D UCM, we examine Fig. 9 which shows the original images, optical flow images, segmentation from 3D UCM, and 3D OF UCM. The optical flow images generated prior to segmentation extract the boundaries of different objects and improve the supervoxel segmentation. For instance, the two legs of the person in the center of the image in Fig. 9 are grouped as the same supervoxel object by 3D OF UCM but segmented as different objects by 3D UCM. Also for the snowboard frames, 3D OF UCM generates finer supervoxels for both the moving object and the background.

And in the DAVIS dataset, we selected some examples of optical flow and segmentation results of 3D OF UCM in Fig. 10. It shows that 3D OF UCM generates fine supervoxels and avoids unnecessary oversegmentation of the background for the moving object because of information from the optical flow motion channel. The result can be regarded as early video processing and be employed to other vision tasks, e.g., semantic segmentation and object segmentation. To sum up, 3D UCM leverages the strength of a high-quality boundary detector and, based on that, 3D OF UCM leverages additional flow field information to perform better. Hence, from qualitative comparisons on two benchmark datasets, it suffices to say that 3D OF UCM generates more meaningful and compact supervoxels at all levels of the hierarchy than three other methods.

5.3 Quantitative measures

As both the IBSR and BuffaloXiph datasets are densely labeled, we are able to compare the supervoxel methods on a variety of characteristic measures. Besides, we re-examined our 3D OF UCM experiments to quantify improvement by using optical flow as a complementary cue. In the previous experiment integrating optical flow with 3D UCM, we only compute optical flow for the first 15 frames and added flow vectors to the original RGB channels. In the next experiment, we process optical flow estimation for the complete video in the BuffaloXiph dataset, and the results show more improvements. We posit the improvements mostly benefit from consistency of motion flow in video. For performance comparisons between 3D OF UCM and 3D UCM, the main improvements are demonstrated on quantitative measures. From the figures of quantitative measures, we notice that, except at very small granularity, 3D OF UCM perform better than 3D UCM on boundary quality, region quality, and supervoxel compactness.

5.3.1 Boundary quality

The precision-recall curve is the most recommended measure and has found widespread use in comparing image segmentation methods [1, 45]. This was introduced into the world of video segmentation in [16]. We use it as the boundary quality measure in our benchmarks. It measures how well the machine generated segments stick to ground truth boundaries. More importantly, it shows the tradeoff between the positive predictive value (precision) and true positive rate (recall). For a set of machine generated boundary pixels S _b and human labeled boundary pixels G _b , precision and recall are defined as follows:

$$ \text{precision}\equiv\frac{|S_{b}\cap G_{b}|}{|S_{b}|},\quad\text{recall}\equiv\frac{|S_{b}\cap G_{b}|}{|G_{b}|}. $$

(11)

We show the precision-recall curves on IBSR and BuffaloXiph datasets in Fig. 11a, d respectively. 3D UCM performs the best on IBSR and is the second best on BuffaloXiph while 3D OF UCM perform the best. GBH does not perform well on BuffaloXiph while SWA is worse on IBSR. One limitation of 3D OF UCM and 3D UCM is that there is an upper limit of its boundary recall because it is based on supervoxels. GBH and SWA can have arbitrarily fine segmentations. Thus they can achieve a recall rate arbitrarily close to 1, although the precision is usually low in these situations. Since our 3D OF UCM adds a motion channel to 3D UCM, its boundary quality is better than 3D UCM.