 Research Paper
 Open Access
 Published:
Phase disambiguation using spatiotemporally modulated illumination in depth sensing
IPSJ Transactions on Computer Vision and Applications volume 12, Article number: 1 (2020)
Abstract
Phase ambiguity is a major problem in the depth measurement in either timeofflight or phase shifting. Resolving the ambiguity using a low frequency pattern sacrifices the depth resolution, and using multiple frequencies requires a number of observations. In this paper, we propose a phase disambiguation method that combines temporal and spatial modulation so that the high depth resolution is preserved while the number of observation is kept. A key observation is that the phase ambiguities of temporal and spatial domains appear differently with respect to the depth. Using this difference, the phase can disambiguate for a wider range of interest. We develop a prototype to show the effectiveness of our method through realworld experiments.
Introduction
Depth measurement is widely used in applications such as augmented reality, factory automation, robotics, and autonomous driving. In the computer vision field, there are two wellknown techniques for measuring scene depth using active illumination. One is the timeofflight camera, which uses temporally modulated illumination to measure the travel time of light; the other is the phase shifting, which uses temporally modulated illumination to find the correspondence between the projector and the camera for triangulation.
A common problem is how to resolve the periodic ambiguity of the phase because either measurement gives the phase that is defined between 0 to 2π. Typical solution is to use multiple frequencies to resolve the phase ambiguity. However, the phase ambiguity still exists in the frequency of the greatest common divisor, which requires several measurements to obtain a wider range of interest. Another possible approach is to use a low frequency that sacrifices the depth resolution. The aim of this study is to resolve the phase ambiguity in fewer observations, where both the wider range of interest and the better resolution of the depth are guaranteed.
A key observation of this paper is that the phase ambiguities of the timeofflight (ToF) and the phase shifting appear differently on the depth domain. Since the temporal phase is proportional to the depth, the depth candidates from the phase appear at equal intervals along with the depth. On the other hand, the spatial phase is defined as the disparity domain; hence, the depth candidates appear at gradually increasing intervals. Based on this difference, the phase ambiguity can be resolved by combining temporal and spatial modulation. Because the candidate depth that satisfies both measured phases seldom appears, the number of phase can be reduced to one for each frequency. In this paper, we discuss ordinary ToF and phase shifting in the same framework. We show that precise depth can be measured in a wide range by combining temporal and spatial modulation. We also reveal the resolution and the range of interest theoretically, analyze the recoverability, and build a prototype to show the effectiveness of our method via realworld experiments.
This paper extends its preliminary version [1] with the following differences. Extensions have been made to (1) reveal the depth resolution and the range of interest of our proposed method, (2) develop an efficient implementation, and (3) confirm that the unrecoverable depth due to ambiguity seldom exists by simulation.
The rest of the paper is organized as follows. The related work is discussed in Section 2, a brief review of the ordinary timeofflight and phase shifting algorithms are provided in Section 3, a spatiotemporal modulation technique is proposed in Section 4, the resolution and range of interest of our method is analyzed in Section 5, experiments with a prototype system is shown in Section 6, and we conclude with some discussions in Section 7.
Related work
Active depth measurements have been widely studied in the computer vision field. Earlier work used a projectorcamera system to convert the projector’s pixel index into multiple projection images based on the gray code [2]. The phase shifting approach [3] recovers subpixel correspondences by detecting the phase of the sinusoid. Gupta and Nayer [4] unwrapped the phase from slightly different frequencies so that it became robust to indirect light transport with a small budget of projection numbers. Mirdehghan et al. [5] proposed an optimal code for the structured light technique. The timeofflight method is another way to measure depth. It emits amplitude modulated light, and a delayed signal is detected that corresponds to the scene depth [6]. Because the range of interest and the depth resolution are tradeoffs, a better resolution is obtained by limiting the range of interest [7]. We combine these techniques to realize both better resolution and wider range of interest.
Another problem regarding the ToF is multipath interference due to indirect light transport. Recovering the correct depth of multipath scenes has been broadly studied using a parametric model [8, 9], Ksparsity [10, 11], frequency analysis [12], and datadriven approaches [13–15]. Because the scene depth can be recovered by the firstreturning photon, the depth can be obtained after recovering lightinflight imaging [16–21]. Multipath interference is mitigated by combining ToF and projector. Naik et al. [22] combined the ToF camera and a projectorcamera system to mitigate a multipath that uses directglobal separation [23]. Similar ideas are implemented with the ToF projectors that can modulate both spatially and temporally [24, 25]. In both cases, directglobal separation is utilized to mitigate multipath interference. We also use a similar system for phase disambiguation not only for mitigating multipath.
To obtain fine resolution, Gupta et al. [26] proposes the optimal code for ToF modulation. GutierrezBarragan et al. [27] proposes an optimization approach for designing practical coding functions under hardware constraints. Kadambi et al. [28] uses the polarization cue to recover the smooth surface. Our method is more fundamental layer; hence, these techniques can be incorporated with our method to boost the resolution. An interferometer can also obtain micrometer resolution of a small size object. Interferometry gives micrometer resolution [29] in a carefully controlled environment. Li et al. [30] recover microresolution ToF using the superheterodyne technique. Maeda et al. [31] leverages the heterodyne technique to the polarization imaging to obtain the accurate depth.
Phase unwrapping is a subproblem in the depth measurement. The phase has to be unwrapped with either the phase shifting or the ToF; otherwise, the estimated depth have 2π ambiguity. The number of observations can be reduced by sacrificing the spatial resolution. The projector’s coordinates can be obtained from a single image using a color code [32], a wave grid pattern [33], and a lightfield ToF [34]. Our method falls into this class but does not sacrifice the spatial resolution nor require many patterns. Our method leverages the asymmetric relations of spatial and temporal wrapping to solve the ambiguity of the phase.
Depth measurement techniques using modulated illumination
Before explaining our method, we briefly review the ToF and phase shifting methods. We respectively explain them as the phase measurements using temporally or spatially modulated light.
Temporal modulation (timeofflight)
The ToF camera emits the temporally modulated light as shown in Fig. 1a. It measures the amplitude decay and phase delay of the modulated light, and the phase delay corresponds to the time it takes for the light to make a round trip.
The ToF camera measures the correlation between the signals emitted and those received. For each frequency, the phase delay is calculated from the correlations with N_{T} reference signals, which are temporally shifted. For the kth signal, the correlation i_{k}(x) at the camera pixel x is represented as
where \(g\left ({t} + \frac {2\pi k}{N_{T}}\right)\) is the reference signal with the shifted phase 2πk/N_{T}, s is the returned signal, the ∗ operator represents the correlation, A is the amplitude decay, ϕ_{T} is the phase delay, and O is the ambient light. In the case of N_{T}=4, the phase ϕ_{T} and the amplitude A of the returned signal can be recovered by a direct conversion method from multiple observations while changing the phase \(\frac {2\pi k}{N_{T}}\) as
The depth d is obtained as
where ω_{T} is the modulation frequency and c is the speed of light.
Spatial modulation (phase shifting)
The phase shifting spatially modulates the projection pattern. Finding the correspondences between the projector and camera pixels is the main part of the spatial phase shifting. The idea is to project the sinusoidal pattern as shown in Fig. 1b and measure the phase of the sinusoid for each pixel, which corresponds to the projector’s pixel coordinates.
The observed intensity of the camera I_{l}(x) for lth shift is represented a
where ϕ_{S} is the spatial phase of the projection pattern due to disparity. There are three unknown parameters, which are the offset O, the amplitude A(x), and the phase ϕ_{S}(x); therefore, they can be recovered from N_{S}≥3 observations while changing the phase of the pattern. In the case of N_{S}=4, the spatial phase ϕ_{S} and the amplitude A can be recovered in the same way as the ToF as
From the estimated disparity, the scene depth can be recovered using the triangulation theory. For example, when the parallel stereo is assumed, the depth is inversely proportional to the disparity as
where \({x}  \frac {{\phi }_{S}}{{\omega _{S}}({x})}\) is the disparity, ω_{S} is the spatial angular frequency of the projection pattern, f is the focal length, and b is the baseline of the procam system. Here, x represents the horizontal pixel position.
Phase ambiguity and depth resolution
A common problem in both temporal and spatial methods is 2π ambiguity, where the phase is wrapped when the depth exceeds the maximum depth of interest. A naive approach is using a low frequency to avoid the phase ambiguity. However, a tradeoff exists between the range of interest and the depth resolution. While the phase ambiguity does not appear at a lower frequency, the depth resolution becomes low as shown in Fig. 2a. With a higher frequency, the depth resolution improves while the phase ambiguity becomes significant, and the depth cannot be uniquely recovered for a wide range of interest as shown in Fig. 2b.
The phase ambiguity is usually relaxed by using multiple frequencies in either a temporal or a spatial domain. However, multiple captures are required, and it sacrifices realtime possibility as shown in Fig. 2c. We propose a hybrid approach of disambiguation that can take advantage of a different nature in temporal and spatial modulation.
Proposed method
We propose a hybrid method of temporal and spatial modulation as shown in Fig. 1c. The phase ambiguity can be resolved by using both temporal and spatial phases instead of using multiple frequencies in either domain.
Spatiotemporal phase disambiguation
Our key idea is that the depth candidates from the ambiguity of the temporal and spatial phases are different. In the case of the temporal phase, the intervals of the depth candidates are constant along the depth because the depth is proportional to the phase, as shown in Eq. (5). On the other hand, the spatial phase is defined in the disparity domain. Because the depth is inversely proportional to the disparity (as shown in Eq. (9)), the intervals of depth candidates increase along with the depth. Figure 3 shows the phase observations along with the scene depth. Multiple depth candidates correspond to a single phase. The depth candidates appear at the same interval for the temporal phase, while the intervals of the spatial phase increase. This difference is a key feature of our method to resolve the phase ambiguity.
Depths that satisfy both temporal and spatial phases seldom appear. The unwrapped phase is not restricted by the greatest common divisor, and the set of temporal and spatial phases is unique for the wider range of interest. The candidate depths can be respectively obtained from the following equations as
The integer pair (n_{T},n_{S}) that satisfies d_{T}=d_{S} seldom exists. Therefore, the phase ambiguity problem can be resolved using phases of different domains.
Phase recovery and depth estimation
Defining I_{0} as the irradiance, the emitted signal from the projector with the kth temporal shift and the lth spatial shift I(p,t,k,l) can be expressed as
where t is time and p is the projector’s pixel. The returned signal r(x,t,k,l) at the camera pixel x is represented as
where κ is the reflectance of target object, o(x) is the ambient light, ϕ_{T}(x) is the phase delay corresponding to the round trip time, and ϕ_{S}(x) is the phase corresponding to the disparity (x−p). The intensity is the correlation with the reference signal \(g_{{\omega _{T}}}(t)\) [35] as
where T is the exposure time. The temporal phase ϕ_{T} and spatial phase ϕ_{S} are obtained from 8 observations with N_{T}=4 and N_{S}=4 as
Now, we have two phases: the temporal phase ϕ_{T} and the spatial phase ϕ_{S}. Depth estimation from the two phases is similar to the unwrapping problem in both the multifrequency phase shifting and the ToF, and it can be solved by searching a lookup table [4]. The observed phases should respectively equal to the phases computed from the same depth, the computed phase ϕ_{T} ~(d),ϕ_{S} ~(d) is obtained as
A lookup table is built for each horizontal pixel position x of the camera because the spatial phase depends on the pixel position. The table \(\mathcal {T}_{{x}}\) at the horizontal position x consists of the vector \(\Phi _{D_{i}, {x}} = [{\Tilde {\phi _{T}}}(D_{i}), {\Tilde {\phi _{S}}}(D_{i}, {x})]\) of the candidate depth D_{i} as
For each pixel, the depth can be estimated by searching the lookup table as
Efficient implementation In practice, building the look up table for each horizontal pixel position is not necessary. Although the spatial phase and corresponding depth depends on the position of camera pixel, the disparity does not depend on the position of the camera pixel. The depth of all camera pixels can be obtained by only one look up table by building from the pair of temporal phase and the disparity after converting the measured phase to the disparity. The disparity is obtained from the measured spatial phase ϕ_{S} and pixel position x as
where δ represents the disparity and \(\tilde {d}\) is the wrapped depth. The table \(\mathcal {T'}\) consists of the vector \(\Phi _{D_{i}}' = [{\Tilde {\phi _{T}}}(D_{i}), {\Tilde {\delta }}(D_{i})]\) of the candidate depth D_{i} as
where δ~ is the computed disparity from candidate depths. For each pixel, the depth can be estimated by searching the lookup table as
Analysis of the proposed method
Depth resolution The resolution is better than ToF in a near range and better than phase shifting in a far range.
The resolution of ordinary ToF and phase shifting is respectively represented as [6, 25]
where A and B are the number of photoelectrons that the sensor can accumulate and represents the amplitude and the DC component, respectively. We suppose that A and B are the parameters of the hardware and are independent from the scene. However, the returned light is influenced by the light falloff in real; hence, a future work is expected to include this effect to analyze more accurately.
Figure 4 shows the depth resolution of ToF and phase shifting along with the depth according to Eqs. (25) and (26). The resolution of ToF is constant at any depth while the resolution of phase shifting is proportional to the square of the depth. The proposed method achieves the resolution that is close to the better resolution of either phase shifting or timeofflight as shown in Fig. 4.
The depth d_{cross} is defined by the depth where the resolution of ToF is equal to the resolution of phase shifting. In the range near than d_{cross}, the resolution of our method is better than ToF and close to phase shifting. In the range far than d_{cross}, the resolution of our method is better than phase shifting and close to ToF. The depth d_{cross} is given as
When we want to improve the resolution of pure ToF, the maximum range of this system should be designated shorter than d_{cross}.
Range of interest The range of interest (ROI) of the proposed method is determined by the relative relation between the temporal and the spatial frequencies.
Nearest range When the spatial frequency is too high compared with the temporal frequency, the phase ambiguity problem cannot be resolved because multiple candidate depths exist within the resolution of the ToF, as shown in Fig. 5a. The spatial frequency varies depending on the depth because the projection is perspective. As the distance is shorter, the spatial frequency is higher. This property gives the nearest ROI of the proposed method. The nearest ROI d_{min} is where the wrapping distance of spatial phase is equal to the resolution of the ToF at the given temporal and spatial frequencies as
where \(\phantom {\dot {i}\!}{d}_{S}_{n_{S}=n_{S}^{'}}\) is the unwrapped depth and \(\phantom {\dot {i}\!}{d}_{S}_{n_{S}=n_{S}'  1}\) is the neighbor depth candidate from Eq. (11). Substituting Eq. (17) and transforming the expression, the minimum depth of the range of interest d_{min} can be obtained as^{Footnote 1}
Farthest range When the spatial frequency is too low compared with the temporal frequency, the phase ambiguity problem cannot be resolved because multiple candidate depths exist within the resolution of the spatial phase shifting, as shown in Fig. 5b. Because the resolution of the spatial phase shifting is inversely proportional to the depth, the farthest ROI d_{max} is determined. The farthest ROI d_{max} is where the wrapping distance of temporal phase is equal to the resolution of the phase shifting as
where \(\phantom {\dot {i}\!}{d}_{T}_{n_{T}=n_{T}'}\) is the unwrapped depth and \(\phantom {\dot {i}\!}{d}_{T}_{n_{T}=n_{T}'1}\) is the neighbor depth candidate from Eq. (11). Substituting Eq. (16), Eq. (26), and transforming the expression, the farthest ROI d_{max} can be obtained as^{Footnote 2}
Unrecoverable point There are few unrecoverable depths in the proposed method. Figure 6 shows that the pair of temporal and spatial phases corresponding to the depth. The vertical axis is the temporal phase, and the horizontal axis is the spatial phase. The color of the curves represents the depth. The intersections of the curves are unrecoverable depth because different depths have the same phase pair. This is a limitation of this method; however, these points generally appeared sparsely in the image hence can be estimated by looking at neighbor pixels of the image.
We confirm that the unrecoverable points seldom exists via simulation. We evaluate the percentage of unrecoverable pixels in an image using an indoor dataset [36]. Temporal phases and spatial phases were respectively rendered, and the depth image is estimated by our method from these phase images. The temporal frequency is set to 50 MHz, and spatial frequency is 1/0.6 mm^{−1}. One hundred scenes were selected from the dataset randomly.
The results are shown in Fig. 7. Depths of some pixels cannot be recovered due to multiple candidates. The average ratio of the uncovered pixel in each image is less than 5%. These points exist sparsely in the image; hence, it is possible to select the candidate by looking around their pixels.
Brightness of the pattern One may think that the temporal phase cannot be obtained if the spatial pattern is completely black. Because the spatial sinusoidal pattern is projected, all the pixels have a chance to obtain the photons unless the spatial pattern is extremely low. A possible solution is to add the constant value to the spatial pattern so that there are no pixels that are always black. In this case, the observation Eq. (14) is rewritten as
where A_{S} and O_{S} (0<O_{S}−A_{S} and O_{S}+A_{S}≤1) are the amplitude and offset of the spatial modulation, respectively. Analogous to Eq. (14), both phases can be obtained by the same equations as Eq. (15) in the N_{T},N_{S}=4 case. So, it is not necessary to increase the number of observations.
Experiment
We demonstrated the effectiveness of our proposed method with realworld experiments.
Hardware prototype We developed a hardware prototype that can illuminate a scene with a spatiotemporal modulated pattern. Our prototype was built onto a ToF camera (Texas Instruments OPT8241CDKEVM). The light source was replaced with a laser diode and a DMD system that can project the spatial pattern. The light source was an 830nm laser diode (Hamamatsu Photonics L927742), and its emission was synchronized with the ToF sensor. The light emitted by the diode was collimated and expanded through lenses, and then reflected onto a DMD device (Texas Instruments DLP6500) that had 1920×1080 pixels. Finally, the spatiotemporal pattern was projected onto the scene through a projection lens, as shown in Fig. 8.
First, the measurement system was calibrated in a standard way for the procam systems using a reference board [37]. The phase of the ToF on each pixel was then calibrated to share the same coordinates as the procam system. A white plane board was captured while its position was moved for the phase calibration. For each measurement of the board, the pair of the raw phase and the groundtruth depth was obtained because the depth of the board was measured by the ordinary phase shifting. The parameter to recover the depth from the phase was calibrated by line fitting.
Result First, we measured a white planar board and placed it at approximately 350 mm from the camera and slightly slanted it, as shown in Fig. 9a. The temporal frequency was 60 MHz, and the period of the spatial pattern was 60 pixels on the projection image. The baseline between the camera and the projector was approximately 70 mm, and the focal length of the projection lens was 35 mm.
The depths were obtained by an ordinary ToF with a single low frequency, phase shifting with single high frequency, and our method for the comparison. Figure 9b shows the estimated depth images. Both the ToF and our method recover the global depth. The depth image with phase shifting cannot be visualized because it has multiple depth candidates. The crosssection of the red line is shown in Fig. 9c. While the depth measured by the ordinary ToF is noisy and there are many depth candidates due to phase ambiguity in the phase shifting, our method recovers a smooth surface while resolving the phase ambiguity. The region near the edge is not correctly disambiguated because the resolution of the temporal measurement exceeds the interval of the phase shifting. The ToF resolution near the edge is lower than what we expected because the illumination is very low near the edge. However, decreasing the spatial frequency might have mitigated it.
Finally, we measured a plaster bust and placed it approximately 400 mm from the camera, as shown in Fig. 10a. The estimated depth images are shown in Fig. 10b. The crosssection of the depth is shown in Fig. 10c. Our method recovers a unique and smooth depth.
Conclusion
We developed a depth sensing method that uses spatiotemporally modulated illumination. We showed that the phase ambiguities of the temporal and spatial modulations are different, so it is possible to effectively resolve the ambiguities while reducing the observations and preserving the depth resolution.
Our proposed method inherits not only the strength of timeofflight camera and active stereo using projectorcamera system but also the weakness of them. While the proposed method can archive better resolution and wider range of interest, it may suffer from occlusion, which scarifies the ToF camera’s potential. However, in practice, the current ToF camera is not a coaxial setup and it does not much suffer from occlusion. If the spatialtemporal projector is configured in the microbaseline setup similarly to a ToF camera, the system does not much suffer from occlusion.
In this paper, depths of the ToF measurement are defined as the distance between a camera and a target; on the other hand, depths of the projectorcamera system of phase shifting is defined as the distance between a center of baseline and a target. In practice, the difference should be correct for implementation although this is not affected to our key idea. Indeed, this model mismatch is absorbed by calibration step to build a look up table.
Our hardware prototype has some limitations. Because the DMD produces the sinusoidal pattern by controlling the mirrors on and off, it can make artifacts to the ToF. We ignored this effect, but it should be considered to control the DMD or to use a solid spatial light modulator appropriately. The quality of the spatiotemporally modulated illumination of our prototype is not very high. The temporal phase contains a systematic distortion, and the spatial resolution of the projector is currently limited to 64 pixels on the DMD, corresponding to 4 pixels on the camera, because the pattern is blurred. This might be due to the collimation and the alignment accuracy of the optics or the diffraction on the DMD. The light source cannot emit a spatial pattern that is equal to or less than the camera pixel’s size, resulting in diminished phase shifting. In future implementations, we will develop a better light source unit to improve the temporal phase measurements and generate higher spatial resolutions.
Appendix
Derivation of Eq. (30)
We reshow Eq. (29) for the derivation of Eq. (30) as
\(\phantom {\dot {i}\!}{d}_{S}_{n_{S}=n_{S}'1}\) is the neighbor depth candidate as
The unwrapped depth \(\phantom {\dot {i}\!}{d}_{S}_{n_{S}=n_{S}'}\) that satisfy Eq. (29) is the minimum depth of the range of interest d_{min} as
Substituting Eqs. (A.1) and (A.2) to Eq. (29),
Substituting Eq. (A.2) to the denominator part,
Multiplying both sides of the equation by \(\frac {{b}{f}}{{d_{\text {min}}}}  \frac {2\pi }{{\omega _{S}}}\) and rearranging the equation,
Solving the quadratic equation for d_{min}, we obtain
where the other solution is always negative and the out of range of d_{min}>0.
Derivation of Eq. 32
Substituting Eqs. (10) and (26) to Eq. (31),
Rearranging the equation,
Substituting Eq. (25) to Eq. (A.9) to cancel A and B,
Therefore,
because d_{max}>0.
Availability of data and materials
The datasets generated during and/or analyzed during the current study are available from the corresponding author on reasonable request.
Abbreviations
 ToF:

Timeofflight
 DMD:

Digital mirror device
References
 1
Kushida T, Tanaka K, Takahito A, Funatomi T, Mukaigawa Y (2019) Spatiotemporal phase disambiguation in depth sensing In: Proc. ICCP. https://doi.org/10.1109/iccphot.2019.8747338.
 2
Inokuchi S, Sato K, Matsuda F (1984) Range imaging system for 3D object recognition In: Proc. International Conference on Pattern Recognition, 806–808.. IEEE Computer Society Press.
 3
Salvi J, Fernandez S, Pribanic T, Llado X (2010) A state of the art in structured light patterns for surface profilometry. Pattern Recog 43. https://doi.org/10.1016/j.patcog.2010.03.004.
 4
Gupta M, Nayer S (2012) Micro phase shifting In: Proc. CVPR, 813–820.. IEEE. https://doi.org/10.1109/CVPR.2012.6247753.
 5
Mirdehghan P, Chen W, Kutulakos KN (2018) Optimal structured light à la carte In: Proc. CVPR. https://doi.org/10.1109/cvpr.2018.00654.
 6
Lange R, Seitz P (2001) Solidstate timeofflight range camera. IEEE J Quantum Electron 37(3):390–397.
 7
Yasutomi K, Usui T, Han S. m., Takasawa T, Keiichiro K, Kawahito S (2016) A submillimeter range resolution timeofflight. IEEE Trans Electron Devices 63(1):182–188.
 8
Heide F, Xiao L, Kolb A, Hullin MB, Heidrich W (2014) Imaging in scattering media using correlation image sensors and sparse convolutional coding,. Opt Express 22(21):26338–50.
 9
Kirmani A, Benedetti A, Chou PA (2013) Spumic: simultaneous phase unwrapping and multipath interference cancellation in timeofflight cameras using spectral methods In: IEEE International Conference on Multimedia and Expo (ICME), 1–6. https://doi.org/10.1109/icme.2013.6607553.
 10
Freedman D, Krupka E, Smolin Y, Leichter I, Schmidt M (2014) SRA: Fast Removal of General Multipath for ToF Sensors In: Proc. ECCV, 1–15. https://doi.org/10.1007/9783319105901_16.
 11
Qiao H, Lin J, Liu Y, Hullin MB, Dai Q (2015) Resolving transient time profile in ToF imaging via logsum sparse regularization. Opt Lett 40(6):918–21.
 12
Kadambi A, Schiel J, Raskar R (2016) Macroscopic interferometry: rethinking depth estimation with frequencydomain timeofflight In: Proc. CVPR, 893–902. https://doi.org/10.1109/cvpr.2016.103.
 13
Marco J, Hernandez Q, Muñoz A, Dong Y, Jarabo A, Kim MH, Tong X, Gutierrez D (2017) DeepTof: offtheshelf realtime correction of multipath interference in timeofflight imaging. ACM Trans Graph 36(6):219–121912. https://doi.org/10.1145/3130800.3130884.
 14
Tanaka K, Mukaigawa Y, Funatomi T, Kubo H, Matsushita Y, Yagi Y (2018) Material classification from timeofflight distortions. IEEE TPAMI. https://doi.org/10.1109/tpami.2018.2869885.
 15
Su S, Heide F, Wetzstein G, Heidrich W (2018) Deep endtoend timeofflight imaging In: Proc. CVPR. https://doi.org/10.1109/cvpr.2018.00668.
 16
Velten A, Willwacher T, Gupta O, Veeraraghavan A, Bawendi MG, Raskar R (2012) Recovering threedimensional shape around a corner using ultrafast timeofflight imaging. Nat Commun 3(745). https://doi.org/10.1038/ncomms1747.
 17
Heide F, Hullin MB, Gregson J, Heidrich W (2013) Lowbudget transient imaging using photonic mixer devices. ACM ToG 32(4):1.
 18
Kitano K, Okamoto T, Tanaka K, Aoto T, Kubo H, Funatomi T, Mukaigawa Y (2017) Recovering temporal PSF using ToF camera with delayed light emission. IPSJ Trans Comput Vis Appl 9(15). https://doi.org/10.1186/s4107401700263.
 19
Kadambi A, Whyte R, Bhandari A, Streeter L, Barsi C, Dorrington A, Raskar R (2013) Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM ToG 32(6):1–10.
 20
O’Toole M, Heide F, Xiao L, Hullin MB, Heidrich W, Kutulakos KN (2014) Temporal frequency probing for 5D transient analysis of global light transport. ACM ToG 33(4):1–11.
 21
O’Toole M, Heide F, Lindell D, Zang K, Diamond S, Wetzstein G (2017) Reconstructing transient images from singlephoton sensors In: Proc. CVPR. https://doi.org/10.1109/cvpr.2017.246.
 22
Naik N, Kadambi A, Rhemann C, Izadi S, Raskar R, Bing Kang S (2015) A light transport model for mitigating multipath interference in timeofflight sensors In: Proc. CVPR, 73–81. https://doi.org/10.1109/cvpr.2015.7298602.
 23
Nayar SK, Krishnan G, Grossberg MD, Raskar R (2006) Fast separation of direct and global components of a scene using high frequency illumination. ACM ToG 25(3):935–944.
 24
Whyte R, Streeter L, Cree MJ, Dorrington AA (2015) Resolving multiple propagation paths in time of flight range cameras using direct and global separation methods. Opt Eng 54:54–549. https://doi.org/10.1117/1.OE.54.11.113109.
 25
Agresti G, Zanuttigh P (2018) Combination of spatiallymodulated ToF and structured light for MPIfree depth estimation In: ECCV Workshop on 3D Reconstruction in the Wild.. IEEE. https://doi.org/10.1007/9783030110093_21.
 26
Gupta M, Velten A, Nayar SK, Breitbach E (2018) What are optimal coding functions for timeofflight imaging?. ACM ToG 37(2):13–11318. https://doi.org/10.1145/3152155.
 27
GutierrezBarragan F, Reza S, Velten A, Gupta M (2019) Practical coding function design for timeofflight imaging In: Proc. CVPR. https://doi.org/10.1109/cvpr.2019.00166.
 28
Kadambi A, Taamazyan V, Shi B, Raskar R (2015) Polarized 3D: highquality depth sensing with polarization cues In: Proc. ICCV, 3370–3378. https://doi.org/10.1109/iccv.2015.385.
 29
Gkioulekas I, Levin A, Durand F, Zickler T (2015) Micronscale light transport decomposition using interferometry. ACM ToG 34(4):37–13714.
 30
Li F, Willomitzer F, Rangarajan P, Gupta M, Velten A, Cossairt O (2018) Shtof: micro resolution timeofflight imaging with superheterodyne interferometry In: Proc. ICCP. https://doi.org/10.1109/iccphot.2018.8368473.
 31
Maeda T, Kadambi A, Schechner YY, Raskar R (2018) Dynamic heterodyne interferometry In: Proc. ICCP.. IEEE. https://doi.org/10.1109/ICCPHOT.2018.8368471.
 32
Sagawa R, Kawasaki H, Furukawa R, Kiyota S (2011) Dense oneshot 3D reconstruction by detecting continuous regions with parallel line projection In: Proc. ICCV. https://doi.org/10.1109/iccv.2011.6126460.
 33
Sagawa R, Sakashita K, Kasuya N, Kawasaki H, Furukawa R, Yagi Y (2012) Gridbased active stereo with singlecolored wave pattern for dense oneshot 3D scan In: 3DIMPVT, 363–370. https://doi.org/10.1109/3DIMPVT.2012.41.
 34
Jayasuriya S, Pediredla A, Sivaramakrishnan S, Molnar A, Veeraraghavan A (2015) Depth fields: extending light field techniques to timeofflight imaging In: 2015 International Conference on 3D Vision, 1–9. https://doi.org/10.1109/3DV.2015.8.
 35
Heide F, Heidrich W, Hullin M, Wetzstein G (2015) Doppler timeofflight imaging. ACM ToG 34(4):36–13611.
 36
McCormac J, Handa A, Leutenegger S, J.Davison A (2017) SceneNet RGBD: can 5m synthetic images beat generic ImageNet pretraining on indoor segmentation?https://doi.org/10.1109/iccv.2017.292.
 37
Zhang Z (2000) A flexible new technique for camera calibration. TPAMI 22:1330–1334. https://doi.org/10.1109/34.888718.
Acknowledgements
We thank all the people who gave us various insightful and constructive comments.
Funding
This work is partly supported by JST CREST JPMJCR1764 and JSPS Kaken grant JP18H03265 and JP18K19822.
Author information
Affiliations
Contributions
TK contributed to the concept, conducted experiments, and wrote the manuscript; KT and TA contributed to the concept and optical design and edited the manuscript; and TF and YM supervised the project and improved the representation. The authors reviewed and approved the final manuscript.
Corresponding author
Ethics declarations
Competing interests
The authors declare that they have no competing interests.
Additional information
Publisher’s Note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Kushida, T., Tanaka, K., Aoto, T. et al. Phase disambiguation using spatiotemporally modulated illumination in depth sensing. IPSJ T Comput Vis Appl 12, 1 (2020). https://doi.org/10.1186/s4107402000063x
Received:
Accepted:
Published:
Keywords
 Timeofflight camera
 Phase shifting
 Computational photography