nature methods: doi:10.1038/nmeth...(a) 3-d rendering of the tracking microscope. (b) design of...

Supplementary Figure 1

Design and operation of the MPC-based tracking microscope.

Nature Methods: doi:10.1038/nmeth.4429

(a) 3-D rendering of the tracking microscope. (b) Design of behavior chamber wall with terraced layers of PDMS. The terraced design allows fluorescent light from the brain to reach the imaging objective (NA = 1.0) without obstruction. (c) A custom Model Predictive Control (MPC) implementation is used to keep the brain within the field of view of the DIFF microscope while the animal moves freely in the behavior arena. (d) Schematic of axial motion cancellation. During each piezo Z sweep (black line), each structured fluorescence image is analyzed in real-time to compute an estimate for its axial location (red line) within the targeted brain volume (red shaded region). Near the end of each sweep, the estimated axial brain position is used to adjust the range of the next piezo Z sweep to recenter the brain. (e) Example of closed-loop live Z tracking. As the targeted brain volume shifts axially (red line and shaded region), the piezo Z sweeps are adjusted to center the brain within the next axial sweep.



Image-processing pipeline for fish tracking.

Each image undergoes background subtraction and image filtering, followed by feature detection to locate the eyes and yolk. The location and orientation of the brain is determined by the position of the eyes and yolk (Methods). Scale bars, 1 mm. All data were collected from an awake and freely swimming 6 dpf larval zebrafish.



Prediction of stage and fish motion optimizes tracking performance.

(a) We model the stage as a linear time invariant (LTI) system that transforms target stage velocity (control input, vinput) to actual stage velocity (voutput). (b) To build a predictive model of stage motion, stage velocities (black) were measured in response to white noise input (red). (c) The impulse response function (red) was solved by ordinary least squares regression, using the preceding 100 ms of control input (vinput) as regressors and the actual stage velocity (voutput) as the response variable. Integrating the impulse response function with respect to time yields the impulse response function for position (red). Every 4 ms, our MPC controller uses this LTI model to select the optimal control input (vinput) that minimizes the predicted future error between the stage position and brain position. (d) The direction of forward fish motion and current forward velocity is estimated from the past 6 time steps of the fish trajectory (-20 ms to 0 ms, red). Based on this history, the fish position is projected 7 time steps into the future (+4 ms to +28 ms, blue). (e) Stage position more closely tracks fish position (black) when fish motion prediction is enabled (blue) than when prediction is disabled (red). (f,g) Cumulative distribution of tracking error with fish motion prediction (blue), without motion prediction (red), or with the actual future fish position (gray). Tracking performance with actual future fish position represents the hypothetical performance of MPC control in the case of perfect motion prediction. (h) Stage position more closely tracks fish position (black) with an MPC controller (blue) relative to a PID controller (red). (i,j) Cumulative distribution of tracking error using MPC (blue) and PID (red) controllers, measured across all time points (i) or while the fish is moving (j). (k) Tracking performance visualized during the replay of a movement bout using MPC or PID control. Scale bar, 500 µm. For cumulative distribution of tracking error, n = 130,621 NIR images (all time) and n = 30,610 NIR images (in motion). All data were collected from an awake and freely swimming 6 dpf larval zebrafish.



Joint distribution of tracking error and fish velocity.

Across an imaging session, tracking error is < 100 µm 91.4 % of the time. During motion (Methods), tracking error is < 100 µm 55.1 % of the time, and < 200 µm 78.0 % of the time. n = 7 fish, 3,426,748 images (all time), and 650,653 images (in motion) from awake and freely swimming 5-7 dpf larval zebrafish.



Distributions of bout kinematics with and without motion cancellation for two behavioral contexts.

The positions of zebrafish larvae were recorded by a NIR camera during free swimming behavior in either a stationary behavior chamber (blue and black), or in the tracking microscope with motion cancellation enabled and fluorescent excitation light off (magenta and grey) or on (green). Fish were monitored in two behavioral contexts: spontaneous behavior in the absence of paramecia (black and grey) and prey capture in the presence of paramecia (blue, magenta, and green). For each swim bout, twelve kinematic parameters were calculated (Methods). Each panel shows the distribution of one kinematic parameter. n = 11 fish, 19,250 bouts (magenta, prey capture with tracking enabled), n = 2 fish, 2,826 bouts (green prey capture with tracking enabled and fluorescent excitation light on), n = 9 fish, 14,743 bouts (gray, spontaneous swimming with tracking enabled), n = 16 fish, 49,223 bouts (blue, prey capture with stationary chamber), n = 7 fish, 11,109 bouts (black spontaneous swimming with stationary chamber). All data were collected from awake and freely swimming 6-7 dpf larval zebrafish.



Motion cancellation and DIFF imaging do not affect prey capture rate.

Larvae were placed in a behavioral chamber containing 20-30 paramecia with tracking ON or OFF, and with the fluorescence excitation light ON or OFF. A box plot of prey capture rate is shown for all three conditions (Methods). Differences between conditions were not statistically significant (p = 0.570 for tracking ON and fluorescence excitation OFF, p = 0. 0.398 for tracking ON and fluorescence excitation ON, Mann-Whitney U test). n = 16 larvae (tracking OFF), n = 7 larvae (tracking ON with fluorescence excitation OFF), and n = 5 larvae (tracking ON with fluorescence excitation ON). Red horizontal line indicates the median, black box spans the first and third quartiles, and whiskers extend to a maximum of 1.5 × IQR beyond the box. Individual data points are shown in gray. All data were collected from awake and freely swimming 6-7 dpf larval zebrafish.



Effects of binning and averaging on shot noise and resolution of DIFF optically sectioned images.

Top, optical sections of the brain using 1 x 1 binning. Bottom, optical sections of the brain using 2 x 2 binning. Left to right, images obtained by averaging increasing numbers of frames (1x, 5x, 10x, and 20x). An ROI of the brain is shown below each whole brain section (yellow box, position of the ROI in brain). Scale bar: 50 μm for whole brain sections and 20 μm for ROIs. To avoid activity-related changes in fluorescence, image data was collected from an anaesthetized 5 dpf elavl3:GCaMP6s larval zebrafish.



Optical sectioning by DIFF and HiLo.


The raw images required for DIFF and HiLo optical sectioning were collected with the following interleaved sequence: (DIFF

structured image A and HiLo structured image ), followed by (HiLo unstructured image), followed by (DIFF structured image B), repeated across an entire imaging session. All images were collected with the same camera gain (21 dB). To ensure all images had the same average brightness, light source was attenuated by two-fold for relative to the structured images (to counteract the two-fold

increase in the number of “on” DMD pixels). DIFF optical sectioning was performed using and . HiLo optical sectioning was performed using and . Top row, DIFF algorithm applied to and (Methods and Supplementary Notes). Middle and bottom

rows, HiLo algorithm48

applied to and with shot noise correction enabled (middle) or disabled (bottom). Left column, single optically sectioned images. Middle and left columns, temporally averaged images across 5 timepoints (middle) and 20 timepoints (left). To avoid activity related changes in fluorescence, data shown were collected from an anaesthetized 5 dpf larval zebrafish.



Effect of tissue scattering on axial and lateral resolution of DIFF microscopy.

(a) 1 μm fluorescent beads were injected (Methods) into the brain of a 4 dpf nacre-/-

fish. The larva was anaesthetized and imaged at 5 dpf. Beads were distributed throughout the fish brain. The outline of the brain is shown in yellow. Top, dorsal view of the brain. Bottom, sagittal view of the brain. Left, images were acquired with high gain to facilitate visualization of the bead locations within the outline of the fish brain. Right, images were acquired with lower gain to accurately measure bead PSFs without saturated pixels. Measured lateral (b) and axial (c) FWHM of 65 beads as a function of each bead’s axial location within the brain.



Paired pulse imaging with global shutter sCMOS minimizes motion blur.

(a) The diagram depicts the relative timing of pulsed LED illumination, DMD pattern generation, camera exposure, and camera readout during two paired image acquisition cycles. Each image pair consists of two differentially patterned images. The DMD pattern is switched during the 60 µs interframe interval of the camera. LED pulses are 45 µs each and occur immediately before and after the 60 µs interframe interval. (b) To minimize motion blur, the fluorescence illumination is pulsed for 45 µs per image using a pulse-gated high current drive circuit. (c) Examples of motion blur for fast moving samples. Assuming a residual velocity of 5 mm/s (after motion cancelation), simulated images of the right optic tectum (elavl3::GCaMP6s) are shown assuming pulsed excitation of either 10 ms (left) or 150 µs (right). Scale bar, 50 µm. (d) Spatial shift measured between image pairs acquired with a 45 µs illumination pulse per image with a 60 µs interframe interval between pulse pairs. To minimize motion artifacts, only image pairs with shift < 1 µm are used for analysis of neural activity. n = 57,643 image pairs (all time) and n = 11,112 image pairs (in motion) from an awake and freely swimming 6 dpf larval zebrafish. (e) Photobleaching rate was measured during DIFF imaging in an anaesthetized 6 dpf elavl3:GCaMP6s fish. Imaging was performed at 200 fps, corresponding to 100 paired images and 100 paired LED pulses per second. The red curve was obtained by least squares fitting of a double exponential model: ( ) ( ) ( ) where = 0.94, = 890.4 min,

= 0.06, and = 3.1 min. To avoid activity-related changes in fluorescence, photobleaching data was collected from an anaesthetized larval zebrafish.



Mean fluorescence of an ROI spanning a single neuron measured over 16 heat pulses.

Raw fluorescence traces (F, black) across 16 consecutive heat pulses (pink, 5 s duration, 30 s interval) of a GCaMP6s-expressing neuron (a) and a Kaede-expressing neuron (b). All data were collected from awake and freely swimming 6 dpf larval zebrafish.



Neuronal responses to heat pulses in neighboring cells in an elavl3:GCaMP6s fish.

(a) Mean activity (∆F/F) of an elavl3:GCaMP6s fish (Fig. 4) to heat pulses (5 s duration, 30 s interval). ROI 1 (yellow dot) corresponds to ROI 7 from Fig. 4. ROIs 2-8 (cyan dots) are non-responsive neighboring cells. Scale bars: 100 μm (left) and 20 μm (right). (b) Single trial activity (∆F/F) of selected neurons in response to individual heat pulses (red lines). (c) Event-triggered average activity (∆F/F, mean ± s.e.m. are shown in black and gray, n = 18 heat pulses) of the same neurons aligned to the onset of heat (red line). (d) Fish speed (mm/s) across the heat pulses (red lines) from (b). (e) Zoom of the fish speed from (d) across a single trial. (f) Event-triggered average speed (mm/s, mean ± s.e.m. are shown in black and gray, n = 18 heat pulses) of the same fish aligned to the onset of heat (red line).



Calcium dynamics of pre-motor and heat-responsive neurons.

Calcium responses to right turns or heat pulses (5 s duration, 30 s interval) are shown for two neurons. Left, event-triggered average activity (∆F/F, mean ± s.e.m. are shown in black and gray, n = 49 turns) of a hindbrain neuron (top) and habenula neuron (bottom) aligned to right turns (dotted line). Right, event-triggered average activity (∆F/F, mean ± s.e.m. are shown in black and gray, n = 18 heat pulses) of the same hindbrain neuron (top) and habenula neuron (bottom) aligned to onset of heat (red). Baseline fluorescence (F) is defined as mean fluorescence during a 5 s interval prior to event onset (-10 s to -6 s for activity aligned to turns, -6 s to -1 s for activity aligned to heat onset). All data were collected from an awake and freely swimming 6 dpf elavl3:GCaMP6s larval zebrafish.



Regression maps across multiple z planes.

Whole brain activity maps of an elavl3:GCaMP6s larval zebrafish navigating a thermal gradient (same animal as in Fig. 6h,i). Bout angle (a), bout speed (b), absolute temperature (c), and relative temperature (d) were each used as a regressor to generate a

regression map showing , the measured linear relationship between neural activity and the regressor. Each panel shows one Z plane with axial depth within the targeted volume indicated at the bottom of each panel. Scale bar: 100 μm. All data were collected from an awake and freely swimming 6 dpf larval zebrafish.



Regression maps of behavior and stimulus parameters.

(a,b) Left, an elavl3:GCaMP6s fish navigating a linear thermal gradient (same animal as in Fig. 6h,i). Colors represent values of bout speed (a) and relative temperature (b) that were used as regressors for ordinary least squares regression analysis of neural activity.

Right, whole brain maps of , the linear relationship between neural activity and each regressor. (c) Whole brain activity maps of an elavl3:GCaMP6s fish (different animal from a,b) responding to heat pulses (5 s duration and 30 s interval). Bout angle was used as a regressor to generate a regression map showing the measured linear relationship between neural activity and bout angle. Each panel shows one Z plane with axial depth within the targeted volume indicated below. All data were collected from awake and freely swimming 6 dpf larval zebrafish.


SUPPLEMENTARY NOTES

Supplementary Note 1: DIFF microscopy

DIFF algorithm

To simplify presentation, we will first explain DIFF optical sectioning under the assumption that camera pixels are small relative to the grating period. In this case, we perform optical sectioning with DIFF as follows:

𝐼𝐼𝐷𝐷 = 𝜅𝜅(𝐼𝐼𝐴𝐴 − 𝐼𝐼𝐵𝐵)

𝐼𝐼𝑈𝑈 = 𝐼𝐼𝐴𝐴 + 𝐼𝐼𝐵𝐵

𝐼𝐼𝐿𝐿𝐿𝐿 = LPσ(𝐼𝐼𝐷𝐷)

𝐼𝐼𝐻𝐻𝐻𝐻 = HPσ(𝐼𝐼𝑈𝑈)

𝐼𝐼𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 = 𝐼𝐼𝐻𝐻𝐻𝐻 + 𝜂𝜂𝐼𝐼𝐿𝐿𝐿𝐿

Each step of this optical sectioning pipeline is explained below.

𝐼𝐼𝐴𝐴(𝑖𝑖, 𝑗𝑗) and 𝐼𝐼𝐵𝐵(𝑖𝑖, 𝑗𝑗) are the raw structured fluorescence images collected in DIFF microscopy. This image pair is obtained by sequentially illuminating the sample with a pair of complementary grating patterns, preferably using a global shutter sensor so the images can be illuminated by a “double shot” strategy with minimal time delay (Supplementary Fig. 10).

𝐼𝐼𝑈𝑈 is an unstructured image that plays a similar role as the HiLo unstructured image 𝐼𝐼𝑢𝑢, but with some distinctions (in the following discussion, we will use lower case subscripts to distinguish HiLo images such as 𝐼𝐼𝑢𝑢 from partially related DIFF images such as 𝐼𝐼𝑈𝑈). In HiLo, 𝐼𝐼𝑢𝑢 is one of the raw images, whereas the DIFF image 𝐼𝐼𝑈𝑈 is computed. In DIFF, a pair of images illuminated with complementary patterns are combined to obtain an unstructured image containing all the photons from both images. This has favorable implications for image brightness and hence signal to noise (SNR) in the formation of 𝐼𝐼𝐻𝐻𝐻𝐻 below.

𝐼𝐼𝐷𝐷 is a difference image that plays a similar role as the absolute difference image 𝐼𝐼𝑑𝑑 in the HiLo algorithm. Both HiLo and DIFF generate a version of this image as a digitally computed intermediate image.

In DIFF, 𝜅𝜅(𝑝𝑝) simply flips the sign of 𝐼𝐼𝐷𝐷 for any pixel 𝑝𝑝 = (𝑖𝑖, 𝑗𝑗) where grating pattern B is brighter than grating pattern A. That is, we define

𝜅𝜅(𝑝𝑝) = � 1 grating A(𝑝𝑝) ≥ grating B(𝑝𝑝)−1 grating A(𝑝𝑝) < grating B(𝑝𝑝)

Multiplication by 𝜅𝜅(𝑝𝑝) plays the same role as applying the absolute value operation in HiLo (i.e. 𝜅𝜅(𝐼𝐼𝐴𝐴 − 𝐼𝐼𝐵𝐵) versus |𝐼𝐼𝑠𝑠 − 𝐼𝐼𝑢𝑢|), but with the distinction that it is a linear operation and it is unaffected by shot noise. In comparison, the absolute value step in HiLo is a nonlinear operation that stochastically applies sign flipping based on shot noise in the raw images. Since the absolute


difference of two Poisson distributed signals has a positive expected value, the absolute value operation in HiLo has the effect of introducing a positive bias. While this bias can be corrected, it adds complexity and potential inaccuracy (e.g. requires estimating the true Poisson means at each pixel). The advantage of using 𝜅𝜅 is that it provides linear, unbiased, shot noise-invariant sign flipping. It is true that our approach requires knowledge of where each grating pattern is brighter than the other, but this is considerably weaker than requiring knowledge of the precise illumination intensities of the two grating patterns. In practice, a mean projection of 𝐼𝐼𝐴𝐴 − 𝐼𝐼𝐵𝐵 across an entire tracking dataset can be used to determine the sign of 𝜅𝜅 at each pixel. We find no evidence for instability in the location of the patterns, at least on the timescale of a tracking experiment. For example, computing the mean projection of 𝐼𝐼𝐴𝐴 − 𝐼𝐼𝐵𝐵 based on the first 10 minutes of an one hour imaging experiment is indistinguishable from the mean projection of 𝐼𝐼𝐴𝐴 − 𝐼𝐼𝐵𝐵 based on the last 10 minutes of imaging data.

LPσ and HPσ denote 2-D low-pass and high-pass filters with a Gaussian kernel with standard deviation 𝜎𝜎. The parameter 𝜎𝜎 should be at least half the grid period 𝑝𝑝. We use 𝜎𝜎 = 0.56𝑝𝑝.

𝐼𝐼𝐿𝐿𝐿𝐿 and 𝐼𝐼𝐻𝐻𝐻𝐻 contain the low frequency in-focus and high frequency in-focus components of the optically sectioned image. These images serve the same role as the HiLo images 𝐼𝐼𝑙𝑙𝐿𝐿 and 𝐼𝐼ℎ𝐻𝐻. However, by design the DIFF images 𝐼𝐼𝐿𝐿𝐿𝐿 and 𝐼𝐼𝐻𝐻𝐻𝐻 are both twice as bright as the HiLo images 𝐼𝐼𝑙𝑙𝐿𝐿 and 𝐼𝐼ℎ𝐻𝐻 obtained with the same total excitation power (i.e. assuming 𝐼𝐼𝐴𝐴, 𝐼𝐼𝐵𝐵, 𝐼𝐼𝑠𝑠, and 𝐼𝐼𝑢𝑢 have same mean brightness). The image 𝐼𝐼𝐻𝐻𝐻𝐻 is twice as bright as 𝐼𝐼ℎ𝐻𝐻 because it is the sum of two images 𝐼𝐼𝐴𝐴 and 𝐼𝐼𝐵𝐵 which each have the same mean brightness as a HiLo unstructured image 𝐼𝐼𝑢𝑢. The image 𝐼𝐼𝐿𝐿𝐿𝐿 is twice as bright as 𝐼𝐼𝑙𝑙𝐿𝐿 because 𝐼𝐼𝐷𝐷 = 𝜅𝜅(𝐼𝐼𝐴𝐴 − 𝐼𝐼𝐵𝐵) contains twice as much in-plane modulation as the corresponding HiLo absolute difference image 𝐼𝐼𝑑𝑑 = |𝐼𝐼𝑠𝑠 − 𝐼𝐼𝑢𝑢|. We closely follow the established approach29,47,48 to model the grating pattern version of HiLo as follows:

𝐼𝐼𝑢𝑢(�⃗�𝑝) = 𝐼𝐼𝐻𝐻𝑖𝑖(𝑝𝑝) + 𝐼𝐼𝐿𝐿𝑢𝑢𝑜𝑜(𝑝𝑝)

𝐼𝐼𝑠𝑠(𝑝𝑝) = 𝐼𝐼𝐻𝐻𝑖𝑖(𝑝𝑝)�1 + 𝑀𝑀 sin�𝑘𝑘�⃗ ⋅ 𝑝𝑝�� + 𝐼𝐼𝐿𝐿𝑢𝑢𝑜𝑜(𝑝𝑝)

where 𝑀𝑀 is the modulation depth and the grating is modeled as a sinusoidal modulation to simplify presentation. Then the corresponding model for DIFF is:

𝐼𝐼𝐴𝐴(𝑝𝑝) = 𝐼𝐼𝐻𝐻𝑖𝑖(𝑝𝑝)�1 + 𝑀𝑀 sin�𝑘𝑘�⃗ ⋅ 𝑝𝑝�� + 𝐼𝐼𝐿𝐿𝑢𝑢𝑜𝑜(𝑝𝑝)

𝐼𝐼𝐵𝐵(𝑝𝑝) = 𝐼𝐼𝐻𝐻𝑖𝑖(𝑝𝑝)�1−𝑀𝑀 sin�𝑘𝑘�⃗ ⋅ 𝑝𝑝�� + 𝐼𝐼𝐿𝐿𝑢𝑢𝑜𝑜(𝑝𝑝)

The HiLo rectified subtraction step29 yields

|𝐼𝐼𝑠𝑠(𝑝𝑝)− 𝐼𝐼𝑢𝑢(𝑝𝑝)| = �𝐼𝐼𝐻𝐻𝑖𝑖(𝑝𝑝)�𝑀𝑀 sin�𝑘𝑘�⃗ ⋅ 𝑝𝑝��= 𝐼𝐼𝐻𝐻𝑖𝑖(𝑝𝑝)𝑀𝑀�sin�𝑘𝑘�⃗ ⋅ 𝑝𝑝��

whereas the DIFF rectified subtraction step yields


𝜅𝜅�𝐼𝐼𝐴𝐴(𝑝𝑝)− 𝐼𝐼𝐵𝐵(𝑝𝑝)� = 𝜅𝜅 �𝐼𝐼𝐻𝐻𝑖𝑖(𝑝𝑝)�𝑀𝑀 sin�𝑘𝑘�⃗ ⋅ 𝑝𝑝� − �−𝑀𝑀 sin�𝑘𝑘�⃗ ⋅ 𝑝𝑝��

= 2𝜅𝜅 �𝐼𝐼𝐻𝐻𝑖𝑖(𝑝𝑝)�𝑀𝑀 sin�𝑘𝑘�⃗ ⋅ 𝑝𝑝��

= 2𝐼𝐼𝐻𝐻𝑖𝑖(𝑝𝑝)𝑀𝑀�sin�𝑘𝑘�⃗ ⋅ 𝑝𝑝��

Hence the DIFF image 𝐼𝐼𝐷𝐷 has twice the intensity of the HiLo image 𝐼𝐼𝑑𝑑.

𝐼𝐼𝐷𝐷𝐷𝐷𝐷𝐷𝐷𝐷 contains the final optically sectioned image, and 𝜂𝜂 controls the relative contribution of 𝐼𝐼𝐿𝐿𝐿𝐿 and 𝐼𝐼𝐻𝐻𝐻𝐻. This step is identical to HiLo. For images collected with camera binning enabled we use 𝜂𝜂 = 3, which we determined by measuring the modulation depth with a thin fluorescent sheet as a calibration sample. As M decreases as a function of tissue depth, 𝜂𝜂 should ideally be adjusted to maintain a consistent ratio of high and low frequency information. Thus, in the current implementation, low frequency information may be underestimated in the ventral parts of the brain.

The optical sectioning thickness of DIFF is related to the illumination grating period. A finer grating period leads to thinner sectioning because the patterns quickly become defocused away from the focal plane. Conversely, larger patterns lead to thicker axial sectioning because the grating pattern retains significant contrast over a larger axial range before becoming defocused. However, there is a trade off in that the modulation depth tends to decrease as the grating period is reduced to near the diffraction limit. We selected a grating period of 6.05 µm because it allowed us to roughly match the optical sectioning thickness to a single cell layer in the brain.

To excite the sample with complementary DIFF grating patterns, we configure the DMD to generate two complementary vertical grating patterns each with a period of 8 DMD pixels (i.e. 4 columns of on pixels and 4 columns of off pixels in an alternating fashion across the DMD). The patterns are strictly complementary, i.e. each on pixel in the A pattern is off in the B pattern and vice versa. The DMD pixel pitch is 7.56 µm and we image the DMD plane into the sample with a demagnification ratio of 10 X. Thus, our patterns have a grating period of 6.05 µm in the sample. Our fluorescence imaging camera has a pixel pitch of 3.45 µm and the detection system has a magnification of 7X, so the image of the grating on the camera has a period of 12.27 camera pixels. With binning disabled, the DIFF sectioning procedure described above works without modification. However, when 2x2 binning is enabled in the camera during live tracking, the grating period becomes 6.14 (binned) camera pixels, which is close enough to the binned camera resolution that subtle brightness variations are observed in 𝐼𝐼𝐿𝐿𝐿𝐿 as the pixel pitch becomes slightly more or less aligned to the periodicity of the grating. We found that we could computationally reproduce this aliasing effect by extracting the grating phase across the image, and then running a mock DIFF-sectioning calculation on a simulated uniform thin fluorescent sheet. In fact, this modulation even helps to confirm that the grating phase has been fit with high accuracy. In practice, we compute the mean projection of 𝜅𝜅(𝐼𝐼𝐴𝐴 − 𝐼𝐼𝐵𝐵) over an entire imaging dataset, fit a model 𝐼𝐼𝑚𝑚 that approximates the projection without measurement noise, and then define a correction image 𝐼𝐼𝑐𝑐 = 𝐿𝐿𝑃𝑃𝜎𝜎(𝐼𝐼𝑚𝑚) which can be applied to an image 𝐼𝐼𝐿𝐿𝐿𝐿 to obtain the alias-corrected image 𝐼𝐼𝐿𝐿𝐿𝐿′ = 𝐼𝐼𝐿𝐿𝐿𝐿 𝐼𝐼𝑐𝑐⁄ . To confirm that this correction procedure works, we performed DIFF optical sectioning on a thin fluorescent sheet, obtained an image 𝐼𝐼𝐿𝐿𝐿𝐿 that was in excellent


agreement with 𝐼𝐼𝑐𝑐, and then followed the correction procedure to obtain an image 𝐼𝐼𝐿𝐿𝐿𝐿′ = 𝐼𝐼𝐿𝐿𝐿𝐿 𝐼𝐼𝑐𝑐⁄ that was uniform and free of aliasing issues.

The maximum frame rate of DIFF imaging is determined by the sCMOS camera. Using the entire sensor with 2 × 2 binning, the maximum frame rate of our camera is 200 Hz. Relative to the camera, the other components of the microscope are not rate limiting. The DMD operates at 10 kHz, the pulsed illumination has switching times of 5 µs, and the piezo stage allows for 10 axial sweeps per second. Thus, in the current implementation, the sCMOS camera effectively determines the imaging speed.

SNR and image brightness in DIFF and HiLo

For the same average excitation power, the use of two structured and complementary illumination patterns in DIFF improves the SNR of the optically sectioned image relative to HiLo. This can be demonstrated theoretically (see above section on DIFF microscopy), and empirically (Supplementary Fig. 8). Experimentally, we interleaved the patterns required to perform both DIFF and HiLo optical sectioning and then quantified the SNR of the resulting optically sectioned images (Supplementary Fig. 8). The single frame SNR for DIFF and HiLo with shot noise correction was measured based on the brightest cells in the focal plane and found to be 10.6 for DIFF and 3.5 for HiLo. The single frame SNR was estimated for each pixel based on its statistics across time (mean and standard deviation based on 200 timepoints) and then the pixel-level SNR values were averaged across the brightest pixels to obtain a consensus SNR.


Supplementary Note 2: Image registration

Reference brain volumes for online and offline registration

At the beginning of an experiment, a series of sweeps over the entire piezo Z adjustment range are performed to facilitate both online and offline registration of moving images to a reference volume acquired from the same animal.

i) Coarse reference brain volume for online Z tracking

To facilitate live Z tracking, a coarse reference volume consisting of 50 focal planes spanning the entire brain is collected while the fish is stationary. The piezo Z stage moves the objective with a symmetric bidirectional scan (triangle wave) with a range of 400 μm and a step size of 8 μm per structured fluorescence image pair (i.e. 8 μm per DIFF-sectioned image), yielding 2 volumes per second. The X-Y tracking system is enabled during this process to maintain the brain within the field of view as the animal swims around the chamber. Sweeps are performed continuously until an entire sweep is collected while the animal is stationary, which typically takes < 1 min. This sweep (1009 μm x 761 μm x 400 μm, 1024 x 772 x 50 voxels) is used as the reference volume for online Z tracking (described below).

ii) High resolution reference brain volume for offline registration

To facilitate high resolution offline registration, a second set of bidirectional sweeps are recorded with a range of 400 μm and a step size of 2 μm per structured fluorescence image pair (i.e. 2 μm per DIFF-sectioned image), which matches the Z spacing used during functional imaging. The tracking system is enabled during this process to maintain the brain within the field of view. High resolution sweeps are continuously recorded for 1-2 min, until an entire sweep is collected while the animal is stationary. After trimming the Z dimension, this process results in a reference volume of roughly 1009 μm x 761 μm x 250 μm (1024 x 772 x 125 voxels).

Online Z tracking by GPU-accelerated live registration

To implement closed loop Z scanning adjustment, each moving image is analyzed in real-time to determine its approximate location in the brain. To ensure low latency, a downsampled registration pipeline is implemented on a dedicated GPU (distinct from the NIR tracking GPU). We implement an established rotation invariant form of the phase correlation technique49 to rapidly detect the best matching reference plane regardless of the heading of the animal in the fluorescence image. Briefly, we downsample the moving image by 8-fold on each axis to obtain a 128 x 128 (padded) downsampled image. We then high-pass with an 11 x 11 pixel Gaussian filter (NPP, NVIDIA), calculate the Fourier transform of the image (cuFFT, NVIDIA), and then calculate the magnitude of each complex Fourier coefficient. We then extract a semicircular subset of the resulting real-valued image by converting to polar coordinates, with 𝑟𝑟 ranging from 1.0 to 13.4 in increments of 0.1 and 𝜃𝜃 ranging from −𝜋𝜋

2 to 𝜋𝜋

2 in increments of 0.025 (1.4°).

Finally, we use FFT-based convolution to compute the dot product between the polar image and the corresponding precomputed polar image for each reference plane, for all circular permutations 𝛥𝛥𝜃𝜃 on the 𝜃𝜃 axis of the polar image. The best matching Z plane is obtained from


argmax𝛥𝛥𝜃𝜃,𝑧𝑧

�𝑃𝑃𝑚𝑚𝐿𝐿𝑚𝑚(𝑟𝑟, 𝜃𝜃 + 𝛥𝛥𝜃𝜃)𝑃𝑃𝑟𝑟𝑟𝑟𝑟𝑟𝑧𝑧 (𝑟𝑟,𝜃𝜃 )𝑟𝑟 ,𝜃𝜃

where 𝑃𝑃𝑚𝑚𝐿𝐿𝑚𝑚(𝑟𝑟, 𝜃𝜃) and 𝑃𝑃𝑟𝑟𝑟𝑟𝑟𝑟𝑧𝑧 (𝑟𝑟, 𝜃𝜃) are the polar images of the moving image and reference image 𝑧𝑧, respectively. This procedure also provides the yaw (modulo 𝜋𝜋), but this information is not used during live tracking.

High resolution offline registration

Each DIFF-sectioned fluorescence image (the “moving image”) is registered to a high resolution reference brain volume collected from the same animal (described above). An initial registration is obtained by optimizing a 3-D rigid transformation mapping the moving image to a (possibly tilted) plane within the reference brain volume. This planar surface is then finely subdivided into a deformable surface that is locally adjusted within the reference volume using a regularized piecewise affine transform (see below for details).

High resolution offline registration: 6 Degree of Freedom (6-DoF) rigid transform

For each pixel location 𝑝𝑝𝑚𝑚𝐿𝐿𝑚𝑚𝐻𝐻𝑖𝑖 = [𝑖𝑖, 𝑗𝑗]𝑇𝑇 in the 2-D moving image, we associate a 3-D point in the

reference volume 𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟𝐻𝐻𝑖𝑖 = �𝑝𝑝𝑥𝑥 ,𝑝𝑝𝑦𝑦 ,𝑝𝑝𝑧𝑧�

𝑇𝑇 according to

𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟𝐻𝐻𝑖𝑖 (�⃗�𝑥) = 𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟

𝐻𝐻𝑖𝑖 �𝑡𝑡𝑥𝑥 , 𝑡𝑡𝑦𝑦 , 𝑡𝑡𝑧𝑧 ,𝜙𝜙,𝜃𝜃,𝜓𝜓� = 𝑅𝑅𝜓𝜓−1𝑅𝑅𝜃𝜃−1𝑅𝑅𝜙𝜙−1 �𝑖𝑖𝑗𝑗0� + �

𝑡𝑡𝑥𝑥𝑡𝑡𝑦𝑦𝑡𝑡𝑧𝑧�

where �⃗�𝑥 = �𝑡𝑡𝑥𝑥 , 𝑡𝑡𝑦𝑦 , 𝑡𝑡𝑧𝑧 ,𝜙𝜙,𝜃𝜃,𝜓𝜓�𝑇𝑇 are the parameters of the 6-DoF transform. The parameters 𝑡𝑡𝑥𝑥, 𝑡𝑡𝑦𝑦,

and 𝑡𝑡𝑧𝑧 represent translations, and 𝑅𝑅𝜙𝜙, 𝑅𝑅𝜃𝜃 , and 𝑅𝑅𝜓𝜓 represent rotation matrices for roll, pitch, and yaw (heading), respectively.

We optimize the 6-DoF correlation function

𝑓𝑓(�⃗�𝑥) = cori,j�𝐼𝐼𝑚𝑚𝐿𝐿𝑚𝑚�𝑝𝑝𝑚𝑚𝐿𝐿𝑚𝑚

𝐻𝐻𝑖𝑖 �, 𝐼𝐼𝑟𝑟𝑟𝑟𝑟𝑟 �𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟𝐻𝐻𝑖𝑖 (�⃗�𝑥)��

where 𝐼𝐼𝑚𝑚𝐿𝐿𝑚𝑚:ℤ2 → ℝ is the moving image, 𝐼𝐼𝑟𝑟𝑟𝑟𝑟𝑟:ℝ3 → ℝ is the reference volume extended to ℝ3 using trilinear interpolation, and cor

i,j(⋅) is the Pearson correlation taken over all pixel coordinates

[𝑖𝑖, 𝑗𝑗]𝑇𝑇 in the 2-D moving image. We use a three-step procedure to locate the global solution �⃗�𝑥∗ =�𝑡𝑡𝑥𝑥∗ , 𝑡𝑡𝑦𝑦∗ , 𝑡𝑡𝑧𝑧∗,𝜙𝜙∗, 𝜃𝜃∗,𝜓𝜓∗�𝑇𝑇 of the 6-DoF registration problem argmax

𝑥𝑥 𝑓𝑓(�⃗�𝑥).

First, we solve argmax

�𝑜𝑜𝑥𝑥,𝑜𝑜𝑦𝑦 ,𝜓𝜓�𝑇𝑇∈𝛺𝛺1

𝑓𝑓�𝑡𝑡𝑥𝑥 , 𝑡𝑡𝑦𝑦,𝜓𝜓�

over a gridded domain 𝛺𝛺1 ⊂ ℝ3, taking advantage of the fact that this simpler objective function can be optimized by FFT-based cross correlation.


Second, we solve argmax𝑥𝑥∈𝛺𝛺2

𝑓𝑓(�⃗�𝑥)

by block coordinate descent over a gridded domain 𝛺𝛺2 ⊂ ℝ6.

Finally, we iteratively solve argmax

𝑥𝑥 𝑓𝑓(�⃗�𝑥)

for �⃗�𝑥 ∈ ℝ6 using the Broyden–Fletcher–Goldfarb–Shanno (BFGS) algorithm.

High resolution offline registration: Non-rigid registration with piecewise affine transform

To account for non-rigid motion within the brain, we subdivide the plane defined by the 6-DoF rigid transform into a regularly spaced grid of points. These points induce a piecewise affine transformation linking the pixels of the moving image to a continuous, deformable 3-D surface in the reference brain volume. The locations of these points are adjusted by iterative optimization using the limited memory Broyden-Fletcher-Goldfarb-Shanno (L-BFGS) algorithm.

A 2-D grid of control points 𝑃𝑃𝑚𝑚𝐿𝐿𝑚𝑚 are defined in the moving image, spaced by 64 pixels in 𝑥𝑥 and 𝑦𝑦. These control points are fixed. A corresponding set of control points 𝑃𝑃𝑟𝑟𝑟𝑟𝑟𝑟 are defined in the reference brain volume, with initial values set by the optimal solution 𝑥𝑥∗ of the 6-DoF problem (described above). If 𝑝𝑝𝑚𝑚𝐿𝐿𝑚𝑚 = [𝑖𝑖, 𝑗𝑗]𝑇𝑇 is a control point in the moving image, then the corresponding control point in the reference brain volume is initialized as

𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟𝐻𝐻𝑖𝑖 (�⃗�𝑥∗) = 𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟

𝐻𝐻𝑖𝑖 �𝑡𝑡𝑥𝑥∗ , 𝑡𝑡𝑦𝑦∗ , 𝑡𝑡𝑧𝑧∗,𝜙𝜙∗,𝜃𝜃∗,𝜓𝜓∗� = 𝑅𝑅𝜓𝜓∗−1𝑅𝑅𝜃𝜃∗

−1𝑅𝑅𝜙𝜙∗−1 �

𝑖𝑖𝑗𝑗0� + �

𝑡𝑡𝑥𝑥∗𝑡𝑡𝑦𝑦∗

𝑡𝑡𝑧𝑧∗�

We define a triangulation over the control points by connecting each control point to its four immediate neighbors and connecting 𝑝𝑝𝑚𝑚𝐿𝐿𝑚𝑚

𝐻𝐻𝑖𝑖 to 𝑝𝑝𝑚𝑚𝐿𝐿𝑚𝑚𝐻𝐻+1,𝑖𝑖+1 for all 𝑖𝑖, 𝑗𝑗. For a moving image of 1024 x

772 pixels, we generate 221 control points and 384 triangles. Since each moving image pixel belongs to a triangle (𝑝𝑝𝑚𝑚𝐿𝐿𝑚𝑚

1 , 𝑝𝑝𝑚𝑚𝐿𝐿𝑚𝑚2 ,𝑝𝑝𝑚𝑚𝐿𝐿𝑚𝑚

3 ), it can be mapped into the reference brain volume by interpolating its location within the plane defined by the corresponding points �𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟1 ,𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟2 ,𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟3 �. Each triangle defines an affine transform, and collectively the triangles define a piecewise affine transform.

Given the locations of the control points 𝑃𝑃𝑟𝑟𝑟𝑟𝑟𝑟 = �𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟�, we compute the similarity of the moving image and the piecewise affine interpolation of the reference image by their dot product:

𝑓𝑓�𝑃𝑃𝑟𝑟𝑟𝑟𝑟𝑟� = �HP𝜎𝜎�𝐼𝐼𝑚𝑚𝐿𝐿𝑚𝑚(𝑖𝑖, 𝑗𝑗)�HP𝜎𝜎 �𝐼𝐼𝑟𝑟𝑟𝑟𝑟𝑟�𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟𝐻𝐻𝑖𝑖 ��

𝐻𝐻,𝑖𝑖

Where HP𝜎𝜎(⋅) denotes a Gaussian high-pass filter with 𝜎𝜎 = 20.

We solve


𝑃𝑃𝑟𝑟𝑟𝑟𝑟𝑟∗ = argmax𝑃𝑃𝑟𝑟𝑟𝑟𝑟𝑟

𝑓𝑓�𝑃𝑃𝑟𝑟𝑟𝑟𝑟𝑟� − 𝜆𝜆��𝐷𝐷𝐻𝐻𝐻𝐻𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟𝐻𝐻𝑖𝑖 �

2

2+ �𝐷𝐷𝑖𝑖𝑖𝑖𝑝𝑝𝑟𝑟𝑟𝑟𝑟𝑟

𝐻𝐻𝑖𝑖 �2

2�

𝐻𝐻,𝑖𝑖

where 𝐷𝐷𝐻𝐻𝐻𝐻 and 𝐷𝐷𝑖𝑖𝑖𝑖 represent second order difference operators, and 𝜆𝜆 = 1 is a regularization penalty. We use L-BFGS, storing only the past 𝑚𝑚 = 10 steps.

Projection of moving images into the reference brain and handling of missing data

After registration, each 2-D DIFF-sectioned moving image 𝐼𝐼𝑚𝑚𝐿𝐿𝑚𝑚 is associated with a non-rigid surface within the reference brain volume defined by a piecewise affine transform. In other words, for each pixel [𝑖𝑖, 𝑗𝑗]𝑇𝑇 in moving image 𝐼𝐼𝑚𝑚𝐿𝐿𝑚𝑚 collected in sweep 𝑘𝑘, there is a corresponding point 𝑝𝑝𝐻𝐻𝑖𝑖 = �𝑝𝑝𝑥𝑥 ,𝑝𝑝𝑦𝑦, 𝑝𝑝𝑧𝑧�

𝑇𝑇 in the 3-D coordinate system of the reference brain volume. We want to place

the measured pixel intensity 𝐼𝐼𝑚𝑚𝐿𝐿𝑚𝑚(𝑖𝑖, 𝑗𝑗) into a volume 𝑉𝑉𝑘𝑘 representing all the brain data collected in sweep 𝑘𝑘. We treat each sweep as a timepoint, so after registration the moving data should be represented by a time series of volumes 𝑉𝑉1,𝑉𝑉2, … ,𝑉𝑉𝑖𝑖, where 𝑛𝑛 is the number of sweeps in the entire tracking experiment. As a practical matter, we represent each volume 𝑉𝑉1 discretely, with the same resolution as the reference brain volume. In general, the point 𝑝𝑝𝐻𝐻𝑖𝑖 = �𝑝𝑝𝑥𝑥 ,𝑝𝑝𝑦𝑦, 𝑝𝑝𝑧𝑧�

𝑇𝑇 is located at a 3-

D location within a cube of 8 nearest discrete voxels in 𝑉𝑉𝑘𝑘 . We could simply round the coordinates 𝑝𝑝𝑥𝑥 ,𝑝𝑝𝑦𝑦 ,𝑝𝑝𝑧𝑧 to the nearest integers, but for greater accuracy, we update all 8 nearest voxels in proportion to the proximity of 𝑝𝑝𝐻𝐻𝑖𝑖 to each of the 8 neighbors. We define 𝛼𝛼𝑥𝑥 = 𝑝𝑝𝑥𝑥 − ⌊𝑝𝑝𝑥𝑥⌋, 𝛼𝛼𝑦𝑦 =𝑝𝑝𝑦𝑦 − �𝑝𝑝𝑦𝑦�, and 𝛼𝛼𝑧𝑧 = 𝑝𝑝𝑧𝑧 − ⌊𝑝𝑝𝑧𝑧⌋ and define each voxel update weight as the product 𝑢𝑢𝑥𝑥𝑢𝑢𝑦𝑦𝑢𝑢𝑧𝑧, where 𝑢𝑢𝛾𝛾 ∈ �𝛼𝛼𝛾𝛾 , 1− 𝛼𝛼𝛾𝛾� (there are 8 cases corresponding to the 8 neighboring voxels). The other issue is that due to sample motion, there is no assurance that any image data will be collected near every voxel. Similarly, some voxels may be oversampled, for example if brain motion causes a point that was already scanned to be encountered again in the current sweep. Therefore, for every update we make to 𝑉𝑉𝑘𝑘 , we also apply a corresponding update to a normalizing volume 𝑁𝑁𝑘𝑘, where instead of using the value of the pixel 𝐼𝐼𝑚𝑚𝐿𝐿𝑚𝑚(𝑖𝑖, 𝑗𝑗), we simply use a value of 1 to mark that data is available there in the volume. After all moving images from a given sweep 𝑘𝑘 have been applied to 𝑉𝑉𝑘𝑘 and 𝑁𝑁𝑘𝑘 in this way, we compute an intensity 𝑉𝑉𝑘𝑘/𝑁𝑁𝑘𝑘 at every voxel where at least some data is available. Alternatively, we compute a projection over multiple time points 𝑉𝑉𝑘𝑘 ,𝑉𝑉𝑘𝑘+1, … before normalizing by a projection over 𝑁𝑁𝑘𝑘,𝑁𝑁𝑘𝑘+1, etc. In visualization or data analysis contexts that cannot accommodate missing data, we perform simple missing data estimation by setting 𝑉𝑉𝑘𝑘�𝑝𝑝𝑥𝑥 , 𝑝𝑝𝑦𝑦, 𝑝𝑝𝑧𝑧� =𝑉𝑉𝑘𝑘−1�𝑝𝑝𝑥𝑥 ,𝑝𝑝𝑦𝑦 ,𝑝𝑝𝑧𝑧� and 𝑁𝑁𝑘𝑘�𝑝𝑝𝑥𝑥 ,𝑝𝑝𝑦𝑦 ,𝑝𝑝𝑧𝑧� = 𝑁𝑁𝑘𝑘−1�𝑝𝑝𝑥𝑥 , 𝑝𝑝𝑦𝑦,𝑝𝑝𝑧𝑧�.


Supplementary Note 3: Computer hardware

Tracking microscope computer

The tracking microscope is controlled by a custom rack-mounted computer (RSV-L4500, Rosewill) with an 8-core Haswell-E processor overclocked to 4 GHz (i7-5960X, Intel), an X99 series motherboard (X99-Deluxe, Asus) with 64 GB of RAM (G.SKILL), two GPUs (GTX 1080, NVIDIA), and an M.2 solid state drive (SSD) (SHPM2280P2/480G, Kingston) for the operating system (Windows 10, Microsoft). During tracking and imaging, NIR and fluorescence data is striped across a 10-drive JBOD array of 6 TiB hard drives (ST6000NM0024, Seagate) for a total capacity of 60 TiB during data acquisition.

Image registration and data analysis servers

Image registration and data analysis is performed on two custom rack-mounted computers (RSV-L4500, Rosewill). Each server has a 8-core Broadwell-E processor overclocked to 4 GHz (i7-6900K, Intel), an X99 series motherboard with dual 10 Gb Ethernet (X99-E-10G, Asus) with 128 GB of RAM (G.SKILL), four GPUs (GTX 1080 or GTX 1080 Ti, NVIDIA), and an M.2 solid state drive (SSD) (SHPM2280P2/480G, Kingston) for the operating system (Ubuntu 16.04, Canonical). Each server uses ZFS (Oracle) to implement a 10-drive software RAID-6 array of 8 TiB hard drives (ST8000NM0055, Seagate) for a total capacity of 80 TiB per server (before considering parity data and overhead). The servers are connected by a 10 Gb Ethernet switch (XS716T, NETGEAR).


REFERENCES 47. Ford, Tim N., Daryl Lim, and Jerome Mertz. "Fast optically sectioned fluorescence HiLo endomicroscopy." Journal of biomedical optics 17.2 (2012): 0211051-0211057. 48. "Biomicroscopy Lab." Biomicroscopy Lab. N.p., n.d. Web. 31 May 2017. http://biomicroscopy.bu.edu/media/resources/

49. Reddy, B. Srinivasa, and Biswanath N. Chatterji. "An FFT-based technique for translation, rotation, and scale-invariant image registration." IEEE transactions on image processing 5.8 (1996): 1266-1271.


nature methods: doi:10.1038/nmeth...(a) 3-d rendering of the tracking microscope. (b) design of...

Documents