# Recovering temporal PSF using ToF camera with delayed light emission

- Kazuya Kitano
^{1}Email author, - Takanori Okamoto
^{1}, - Kenichiro Tanaka
^{1}, - Takahito Aoto
^{1}, - Hiroyuki Kubo
^{1}, - Takuya Funatomi
^{1}and - Yasuhiro Mukaigawa
^{1}

**9**:15

https://doi.org/10.1186/s41074-017-0026-3

© The Author(s) 2017

**Received: **31 March 2017

**Accepted: **9 May 2017

**Published: **2 June 2017

## Abstract

Recovering temporal point spread functions (PSFs) is important for various applications, especially analyzing light transport. Some methods that use amplitude-modulated continuous wave time-of-flight (ToF) cameras are proposed to recover temporal PSFs, where the resolution is several nanoseconds. Contrarily, we show in this paper that sub-nanosecond resolution can be achieved using pulsed ToF cameras and an additional circuit. A circuit is inserted before the illumination so that the emission delay can be controlled by sub-nanoseconds. From the observations of various delay settings, we recover temporal PSFs of the sub-nanosecond resolution. We confirm the effectiveness of our method via real-world experiments.

## Keywords

## 1 Introduction

When an image is recorded by a general camera, various optical phenomena are collectively recorded. Light rays leaving from the light source change in intensity by repeating reflection and scattering, and the final intensity is recorded by the camera.

In recent years, temporal point spread function (PSF) of a scene, which is a response to the impulsive light, is attracting attention because it can be used for analysis of interreflection and scattering. By analyzing temporal PSFs, how the light transports in the scene can be analyzed in detail.

Temporal PSF of the scene can be obtained using an interferometer [1], holography [2], and femtosecond-pulsed laser [3], which requires complex optics and expensive devices. Heide et al. [4] firstly recover a temporal PSF in an inexpensive way using an amplitude-modulated continuous wave ToF camera. Using a continuous wave ToF camera, the propagation of light in units of nanoseconds can be visualized [5–7].

In this paper, we propose a method for estimating the temporal PSF using a *pulsed* ToF camera. Using pulsed ToF cameras, low-resolution temporal PSFs can be straightforwardly obtained, e.g., tens of nanoseconds, which is insufficient for analyzing scattering that is finished within 1 ns. To achieve sub-nanosecond resolution, we introduce an additional circuit for delaying the light emission. By delaying the light emission in sub-nanoseconds, we recover temporal PSFs via computation and achieve sub-nanosecond resolution.

Our contributions are twofold. Firstly, the impulse response of the scene can be estimated even with the pulsed ToF camera. Secondly, we achieve sub-nanosecond resolution. We show a simple PSF recovery by delaying the light emission and computation.

## 2 PSF estimation using ToF camera

In this section, we explain the method that estimates temporal PSF using pulsed ToF camera. Before explaining the method, we start from the basic theory of the pulsed ToF camera.

### 2.1 Principle of ToF camera operation

*d*is obtained as

*τ*is the measured round trip time of light and

*c*is the speed of light. Because the observation of the ToF camera is independent for each pixel, the camera pixel

*p*is omitted for simplicity. In this paper, we assume a pulsed ToF camera, which emits a square pulsed light for a few tens of nanoseconds as shown in Fig. 1. Synchronizing with the light emission, two images

*I*

_{1}and

*I*

_{2}exposed at the same time as the light emission width are obtained. From these two images, the round trip time

*τ*can be obtained as

*T*is the width of light emission and exposure. Moreover, because the observable range of the depth is limited to the width of the square wave, the start of the exposure is shifted to change the region of interest. In such a case, Eq. (2) can be written as

where *t*
_{offset} is the shift of the exposure.

### 2.2 Distortion of reflected light due to PSF

*r*(

*t*) of the scene is spread along with the time domain; therefore, the waveform of the reflected light is no longer the square wave.

*l*(

*t*) and the temporal PSF

*r*(

*t*). Observation

*i*of the ToF camera is then represented as

where ⊗ is the convolution operator and *g*(*t*) is the exposure function of the ToF camera representing the exposure sensitivity at a certain time *t*. We assume that *g*(*t*) is a binary function, whether the exposure is performed or not, but in general, it can take a continuous value.

### 2.3 PSF estimation with delayed light emission

Temporal PSF varies depending on the surface shape and translucency of the object; it has important information about the material and structure of the object. In order to analyze the properties of the object in detail, we are interested in recovering the temporal PSF with high resolution.

*l*(

*t*) and the exposure

*g*(

*t*) can be regarded as the delta function

*δ*(

*t*). Appropriately shifting the start of the exposure, the temporal PSF can be directly obtained as

where *s* is the variable of integration on the time domain and *t* corresponds to the shift of exposure. However, it is difficult to shorten the emission and exposure width to the infinitesimal time because of the limitation of the logical gates. Using the typical ToF cameras, only tens of nanoseconds resolution can be achieved.

*blurred*PSF \(\bar {r}(t)\) can be observed as

where *g*
^{∗} is a exposure function whose time axis is inverted. Because \(\bar {r}(t)\) is convoluted by illumination and exposure, the original PSF *r*(*t*) can be estimated by deconvolution.

### 2.4 Numerical implementation

*i*

_{ j }at

*j*-th delay setting is given by

*g*

_{ j }is the

*j*-th exposure function corresponding to the

*j*-th delay of light emission. Descretizing Eq. (7),

**g**

_{ j }is a vector representing

*j*-th exposure,

**L**is a convolution matrix of emitted wave, and

**r**is a vector of discretized temporal PSF, whose resolution is the same as the width of the delay. Superposing all the observations

**i**

and *m* is the number of observations.

**G**

**L**is a known matrix. Therefore, if the number of the observation is sufficiently large, the temporal PSF

**r**can be estimated by the least squares method as

where (**G**
**L**)^{+} is the pseudo-inverse matrix.

**G**

**L**is ill-conditioned hence the calculation as shown in Eq. (11) is unstable. However, using the property that the intensity of light does not have a negative value, it is expected to estimate robustly against the instability of calculation. Estimation of temporal PSF can be estimated by the least square method with the non-negative constraint as

Since Eq. (12) is a convex optimization problem, the optimal solution can be obtained in a polynomial time.

## 3 Experiments

## 4 Conclusion

In this paper, we propose a method to estimate temporal PSF with high temporal resolution by combining a pulsed ToF camera and a simple delay circuit. We use the delay circuit to control the light emission timing in units of sub-nanoseconds and recover the temporal PSF by a least square method with the non-negative constraint. We have conducted some real-world experiments and confirm the effectiveness of our method. We are interested in improving more higher temporal resolution in the future research so that the subsurface scattering can be fully analyzed.

## Declarations

### Acknowledgments

A part of this work was supported by JSPS KAKENHI Grant Number JP15H05918.

### Authors’ contributions

KK planned and executed experiments and wrote the manuscript. TO executed preliminary experiments. KT advised, helped experiment, and wrote the manuscript. TA created the supplemental and helped to write the manuscript. HK and TF are co-supervisors and edited the manuscript. YM is a supervisor and edited the manuscript. All authors reviewed and approved the final manuscript.

### Competing interests

The authors declare that they have no competing interests.

### Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

**Open Access** This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

## Authors’ Affiliations

## References

- Gkioulekas I, Levin A, Durand F, Zickler T (2015) Micron-scale light transport decomposition using interferometry. ACM Trans Graph (ToG) 34(4): 37–13714.View ArticleMATHGoogle Scholar
- Kakue T, Tosa K, Yuasa J, Tahara T, Awatsuji Y, Nishio K, Ura S, Kubota T (2012) Digital light-in-flight recording by holography by use of a femtosecond pulsed laser. IEEE J Sel Top Quantum Electron 18(1): 479–485.View ArticleGoogle Scholar
- Velten A, Raskar R, Wu D, Jarabo A, Masia B, Barsi C, Joshi C, Lawson E, Bawendi M, Gutierrez D (2013) Femto-photography: capturing and visualizing the propagation of light. ACM Trans Graph (ToG) 32(4): 44–1448.View ArticleGoogle Scholar
- Heide F, Hullin MB, Gregson J, Heidrich W (2013) Low-budget transient imaging using photonic mixer devices. ACM Trans Graph (ToG) 32(4): 45–14510.MATHGoogle Scholar
- Peters C, Klein J, Hullin MB, Klein R (2015) Solving trigonometric moment problems for fast transient imaging. Proc ACM SIGGRAPH Asia 34(6): 220–122011.Google Scholar
- Kadambi A, Whyte R, Bhandari A, Streeter L, Barsi C, Dorrington A, Raskar R (2013) Coded time of flight cameras: sparse deconvolution to address multipath interference and recover time profiles. ACM Trans Graph (ToG) 32(6): 167–116710.View ArticleGoogle Scholar
- O’Toole M, Heide F, Xiao L, Hullin MB, Heidrich W, Kutulakos KN (2014) Temporal frequency probing for 5D transient analysis of global light transport. ACM Trans Graph (ToG) 33(4): 87–18711.Google Scholar