Enhancing Speckle Statistics for Imaging inside Scattering Media
←
→
Page content transcription
If your browser does not render page correctly, please read the page content below
Research Article 1
Enhancing Speckle Statistics for Imaging inside
Scattering Media
W EI -Y U C HEN1 , M ATTHEW O’TOOLE2 , A SWIN C. S ANKARANARAYANAN1 , AND A NAT L EVIN3,*
1 ECE Department, Carnegie Mellon University, 5000 Forbes Avenue Pittsburgh, PA 15213
arXiv:2203.14214v1 [physics.optics] 27 Mar 2022
2 RoboticsInstitute, Carnegie Mellon University, 5000 Forbes Avenue Pittsburgh, PA 15213
3 EE Department, Technion, Israel, Technion City, Haifa 3200003, Israel
* Corresponding author: anat.levin@ee.technion.ac.il
Compiled March 29, 2022
We exploit memory effect speckle correlations for the imaging of incoherent linear (single-photon) flu-
orescent sources behind scattering tissue. While memory effect based imaging techniques have been
heavily studied in the past, for thick scattering layers and complex illumination patterns these correla-
tions are weak, limiting the practice applicability of the idea. In this work we introduce a Spatial Light
Modulator (SLM) between the tissue sample and the imaging sensor and capture multiple modulations of
the speckle pattern. We show that by correctly designing the modulation pattern and the reconstruction
algorithm we can greatly enhance statistical correlations in the data. We exploit this to demonstrate the
reconstruction of mega-pixel wide fluorescent patterns behind scattering tissue.
http://dx.doi.org/10.1364/ao.XX.XXXXXX
1. INTRODUCTION is due to the fact that speckle correlations have been shown [11]
to be stronger when the sources are further than the scattering
One of the main barriers on our ability to see fluorescent sources layer, and hence they span a smaller range of angles relative to
deep inside tissue is the fact that the light they emit is scattered the sample. Also, the contrast of the observed speckle pattern
through the tissue. Given a sparse set of incoherent sources decays as more independent emitters are present, and hence,
inside the tissue, a microscope usually observe a noisy speckle mostly simple and sparse emitter layouts were recovered.
pattern that carry little resemblance to the actual sources. In this work we exploit strategies for maximizing the amount
Despite their noise-like appearance, speckles have strong of ME correlation we can extract from speckle images. To this
statistical properties, such as the memory effect (ME), implying end we build on a simple observation: if we could image the
that speckle patterns generated by nearby sources are correlated same layout of fluorescent sources through different tissue lay-
shifted versions of each other. This fascinating property was ers we would obtain independent speckle images, leading to
exploited in the past to expand our ability to see deep inside independent auto-correlations. Averaging such independent
scattering tissue, by observing that due to the memory effect auto-correlations can boost the signal to noise ratio of the de-
the auto-correlation of a speckle pattern generated by multiple tected illuminator pattern. While some temporal dynamics is
independent sources is equivalent to the auto-correlation of the present in live tissue, sequential images of the same tissue are
latent source layout [1–10]. This algorithm has drawn a lot of still highly correlated. Rather, we use a programmable Spatial
interest as it allows recovering latent illuminators completely Light Modulator (SLM) mask in the optical path imaging the tis-
invisible to the naked eye, purely by exploiting speckle statistics. sue, and use it to modulate the field, leading to different speckle
Despite its potential, there are still major challenges to solve be- images. We discuss various forms of modulation and arrive at
fore the idea can apply to realistic biomedical imaging scenarios. a spatial form of modulation we term translating interferometry,
that can lead to uncorrelated speckle measurements, as well as
The main barrier is that the information available by speckle maximize ME correlation.
statistics is limited, and to circumvent around it, experimental To further maximize the amount of information we can ex-
demonstrations involved various simplifications. For exam- tract from speckle data, we follow an idea recently proposed
ple, while in realistic biomedical imaging scenarios fluorescent by [11], which argue that when light sources are located inside
sources of interest are located inside the scattering sample rather the sample rather than far behind it, the speckles from each
than far behind it, many earlier demonstrations of speckle corre- source have a limited support and do not spread over the full
lation based see-through algorithms considered sources located sensor. Thus, rather than computing a global auto-correlation,
a few centimeters beyond the sample rather than inside it. This they compute local auto-correlations in the form of a Ptychogra-
Research Article 2
Reference Input speckle images
Single image reconstruction Our reconstruction
(Validation camera) (Main camera)
Fig. 1. Reconstructing a wide-range fluorescent bead target from modulated speckles. We reconstruct the layout of fluorescent
beads spread behind a chicken breast tissue slice, whose thickness was measured at about ∼ 150µm. The beads are attached to the
tissue, separated only by a 150µm cover glass. The beads spread over a field of view of 300µm × 300µm, occupying a one mega pixel
image. While the memory effect correlations in a single image capture are too noisy to provide good reconstruction, we optically
modulate the speckle field, capturing 54 shots with different modulations. This modulation allows us to amplify statistical correla-
tions, leading to accurate reconstruction of a complex illuminator pattern, despite high degradation and limited speckle contrast
in the input images. The lower part of the figure includes the full images, while at the top we zoom on two sub windows for high
resolution visualization.
phy algorithm [12–16]. As shown in [11] these local correlations with τ the displacement between the source to the observation
can boost the signal to noise ratio of the detect correlation by a point τ = v − i1 , and α ≈ − 3
2L , where L is the tissue thickness.
few orders of magnitude. This model assumes we image the volume with a microscope
Overall we demonstrate the reconstruction of wide, com- whose sensor plane is conjugate to the plane of the illuminators
plex fluorescent bead patterns beyond scattering tissue. Our ap- i2 , i1 . That is, the depth of the sensor plane is selected such
proach captures only a few dozen images of the tissue, compared that in the absence of the scattering layer it would focus on the
to hundred of images used by recent approaches imaging linear sources i1 , i2 and provide a sharp image of them. Obviously due
fluorescent sources behind scattering layers [17, 18]. Compared to scattering in the medium, even if the objective aims to focus
1 2
to recent wavefront shaping approaches [19, 20] that only facili- at the right depth, the images ui (v), ui (v) involve speckles.
tate imaging of a local neighborhood governed by the limited However, by setting the sensor plane to be conjugate to the
extent of the memory effect, our approach recovers mega-pixel illumination plane [22] simplify the tilt-shift relation.
large images over a wide field of view, as demonstrated in Fig. 1. A camera sensor only measures the intensity of the speckle
pattern which we denote by
2. PRINCIPLE 2
Si (v) = ui (v) . (3)
A. A review of memory effect and its see-through application
In the presence of multiple incoherent illuminators we observe
We start with a quick review of the memory effect (ME) and its n n 2
application for seeing inside scattering media. an intensity image I (v) = ∑n Si (v) = ui (v) . If ME correla-
tion exists speckle intensities from nearby sources are shifted
The memory effect states that speckle fields generated by
versions of each other, and since we deal with intensity images,
nearby sources are correlated shifted versions of each other. Let 1 2
1 2 phase adjustments are not required Si (v) ≈ Si (v + ∆). We
i2 , i1 denote the position of two such sources and ui (v), ui (v)
denote by S0 (v) the speckle from an illumination at the center of
the fields they generate, where v denotes a sensor coordinate.
the frame. With this notation we can express the sum of speckles
It can be shown that for sufficiently small displacements ∆ =
from incoherent sources as
i2 − i1 :
1 2
ui (v) ≈ eiφ(v) ui (v + ∆), (1) I = S0 (v) ∗ O, (4)
where φ(v) denote a phase correction. Osnabrugge et al.[21] where O is a binary image denoting the location of the illu-
argue that when the illumination arises from points inside the mination sources, and ∗ denotes convolution. In see-through
scattering media, the phase correction can be approximated as a algorithms our goal is to recover a latent illuminator pattern O
tilt. Bar et al.[22] offer a simple model for this tilt, stating that from an input speckle image I.
We now filter I and S0 to locally have a zero mean
ui (v) ≈ eikα ui (v + ∆),
1 2
(2) Ī = I − g ∗ I, S̄0 = S0 (v) − g ∗ S0 , (5)
Research Article 3
(d) Translating
(c) Random (e) Translating
(a) Ground truth (b) No modulation [11] interferometry (phase
modulation interferometry
uncorrected)
Fig. 2. Comparing speckle auto-correlation with different modulation approaches. The two rows compare illuminator layouts
with different complexity, while the columns evaluate different modulation strategies. All results use the same number of shot im-
ages. (a) Ground truth. (b) Without any modulation, reconstructed auto-correlation is noisy, and when the target is complex (2nd
row) it is almost unrecognizable. (c) Random modulation can improve contrast, but still contains noise. (d) translating interferome-
try greatly reduce the noise, but larger displacements at the outer image regions are degraded by the phase ramp. (e) After phase
correction, translating interferometry can clearly recover the auto-correlation.
where g is a low pass filter. We note that Eq. (4) also holds if we To analyze the contrast of the speckle correlation we intro-
replace I, S0 with Ī, S̄0 , and we can express Ī = S̄0 ∗ O. Since S̄0 duce the following notation. We denote by Γ∆ the set of all
is a random zero mean signal, S̄0 ? S̄0 ≈ δ is a sharply peaked displacements ∆ such that our latent pattern includes a pair of
impulse function [2]. With this approximation [1, 2] derive the illuminators (n, m) separated by ∆ :
relationship:
Γ∆ = {∆|∃(n, m), ∆ = im − in } , (7)
0 0
Ī ? Ī = (S̄ ? S̄ ) ∗ (O ? O) ≈ O ? O, (6)
and by Γ̄∆ the list of all other displacements. We denote the
where ? denotes correlation and ∗ denotes convolution. Thus, speckle auto-corr by C Ī , which is defined for a displacement ∆
the auto correlation of the input speckle intensity is equivalent as:
to the auto-correlation of the desired latent image O. Hence one C Ī (∆) = ∑ Ī (v) Ī (v + ∆). (8)
can recover O from Ī ? Ī using a phase retrieval algorithm [2]. v
Intuitively, the speckle correlation has good contrast if C Ī (∆) is
B. Improving speckle correlation contrast
high for displacements corresponding to real illuminator posi-
The idea beyond Eq. (6) is very compelling because it suggests tions ∆ ∈ Γ∆ , and low for all other displacements ∆ ∈ Γ̄∆ . We
that latent illuminators O can be recovered from noisy speckle define the correlation contrast using the following metric:
images I, despite the fact that these input images carry no simi-
larity to the latent source layout. Yet, its practical applicability is h i2
∑∆∈Γ∆ E C Ī (∆)
1
|Γ∆ |
limited. In practice the algorithm was mostly demonstrated us-
Θ C Ī = h i (9)
ing far field illumination patterns, namely with sources located 1
∑ E |C Ī
( ∆ )| 2
|Γ̄∆ | ∆∈Γ̄∆
a few centimeters beyond the scattering medium. In realistic
biomedical imaging applications one is usually interested in de- One way to increase the correlation contrast that was used e.g.
tecting fluorescent illumination sources located inside the tissue in [1], is to capture multiple images of the latent pattern O behind
rather than far beyond it. As analyzed in [11], and the corre- different scattering layers, so that effectively we measure It =
1 2
lation between Si , Si only holds for very small displacements St0 ? O with different speckle patterns St0 . The auto-correlation is
∆ = i − i . In Fig. 2(b), we demonstrate the auto-correlation
2 1
then evaluated as the average of the individual auto-correlations
of a speckle image Ī composed of a sparse layout of sources.
1
∑ C Ī (∆),
We can see that as more incoherent sources are included the Ī1 ,..., ĪT
C̄ (∆) = t
(10)
auto-correlation is very noisy and does not resemble the desired T t
layout of O ? O. Our goal in this work is to improve the contrast
of this auto-correlation by capturing multiple modulations of with C Īt = Īt ? Īt as defined in Eq. (8). It is easy to see the
the speckle signal. following, and we provide a formal proof in appendix.
Research Article 4
maintain the memory effect correlation. To this end we design an
interferometric setup that allows us to measure the interference
between ui and a shifted copy of it, which we name translating
interferometry. This leads to a measurement of the form
Sti = ui (v)ui (v + dt )∗ , (13)
where dt denotes the displacement vector. Since this signal does
not average complex field entries with different phases it does
not suffer from the correlation reduction anticipated in Eq. (12).
When several incoherent sources are present we will acquire an
incoherent summation
Fig. 3. Contrast improvement for different modulations. We
∑ Sti
n
compare speckle correlation contrast as a function of the num- It = . (14)
ber of images, for different modulation approaches. The two n
images show correlation contrast measurements for two differ- This interferometric measurement is already a zero mean signal
ent tissue slices. The graphs are noisy due to the finite speckle and there is no need to subtract the mean as with the intensity
spread in an image, but still demonstrate clear trends. Without measurements of Eq. (5).
modulation, multiple images only reduce read and photon We describe the acquisition setup leading to this measure-
noise of the image, which does not lead to a significant con- ment in Sec. 3.A.1 below. Here we start by analyzing the advan-
trast improvement. Multiple images with random modula- tages it offers. In the appendix we prove the following:
tions or translating interferometry with no phase correction
can improve contrast, but this increases quickly saturates as Claim 2 For displacements dt1 , dt2 whose distance kdt1 − dt2 k is
memory effect degrades. On the other hand, translating inter- larder than the speckle grain, the signals Sti1 , Sti2 are uncorrelated.
ferometry with phase correction can achieve a higher contrast, The fact that different displacements lead to uncorrelated speckle
and in agreement with theory, has nearly linear rate of contrast measurements means that according to Claim 1 we could av-
improvement. erage correlations od different measurements m and improve
correlation contrast.
The auto-correlation of the translating interferometry mea-
Claim 1 If the speckle patterns St0 are uncorrelated with each other surements relates to the auto-correlation of the hidden illumina-
for different t values, than replacing C Ī with C̄ T in the correlation tor pattern O, but unlike pure intensity speckles, with the above
contrast of Eq. (9), increases the contrast linearly with the number of modulations a phase correction is needed, which we derive in
measurements T. the following claim, and prove in appendix.
Ī ,..., Ī
Θ C̄ 1 T = T · Θ C Ī (11) Claim 3 Using the translating interferometry measurements of
Eq. (13), the speckle auto-correlation C It = It ? It is equivalent to the
While this is a promising idea, when the sources are located
auto-correlation of the latent pattern C O = O ? O, times a phase ramp
inside the tissue it is not possible to move the scattering layer
correction
and image the same illuminator layout. While speckles in a
C It (∆) ≈ C O (∆)e− jkα . (15)
live tissue decorrelate over time, without very long acquisition
times, it is usually not possible to acquire a large number of Given this conclusion we average the auto-correlation of the
independent speckle patterns. Rather, in this work we would different translating interferometry measurements, applying the
like to modify the speckle patterns by adjusting the optics. phase ramp correction of Eq. (15):
Intuitively, to create different speckle intensity images, we
1
∑ e jkα C I (∆)
can put a random phase mask in the optical path between the Ī1 ,..., ĪT
C̄ = t t
(16)
sample to the imaging sensor. If we put this mask in the Fourier T t
plane it would translate into a convolution of the fields ui (v)
The phase corrected averaging in Eq. (16) is subject to a single
with the Fourier transform of the mask, that we denote as ht .
n unknown parameter α. In our implementation we tune it to
This would lead into an intensity image It = ∑n Sti with
maximize the visual quality of the results.
n n 2 In Fig. 2(d-e) we show the auto-correlation obtained by av-
Sti = ui ∗ ht . (12) eraging translating interferometry measurements It (Eq. (14))
with and without the phase ramp correction of Eq. (16). Both
In Fig. 2(c), we compare the auto-correlation of a single speckle approaches reduce noise and improve the correlation contrast
image to the average auto-correlation with 54 random masks when compared with random modulations (Eq. (12)) or just
hk . Averaging random masks rejects noise and improves the with the auto-correlation of a single speckle image. However,
correlation contrast, but it is still noisy. To understand why this the phase correction further improves the contrast, especially
modulation is sub-optimal we recall the tilt-shift correlation in at larger displacements ∆. In Fig. 2(d-e) we average 18 translat-
n n
Eq. (2). If the fields ui would follow a pure shift, then ui ∗ ht ing interferometry measurements It . Note that as we explain
would also be shifted versions of each other. However, according in Sec. 3.A.1, each interferometric measurement It is extracted
to Eq. (2), fields from different sources vary by phase, and hence using 3 shots, so the 18 measurements in Fig. 2(d-e) are acquired
1
a convolution with ht largely degrades the correlation and Sti (v) using a total of 54 shots. This is compared against 54 indepen-
2
would differ from Sti (v + ∆). dent measurements captured by the random modulation ap-
Our goal in this work is to change the optical path such that proach. 18 translating interferometry modulations are superior
we can capture multiple independent speckle patterns, and yet over 54 random modulations.
Research Article 5
pattern and the optimized latent image. This leads into a Pty-
Table 1. Summary of different speckle modulation ap- chography style cost. For a local window w p we express the
proaches. local auto-correlations of the latent image
Ī1 ,..., ĪT
Sti C̄
No modulation [11] ui (v)
2
Ī ? Ī
CO
w p (∆) = ∑ O(v)O(v + ∆)∗ (17)
v∈ w p
2 1
Random modulation ui (v) ∗ h t T ∑t Īt ? Īt
Translating interf.
and the average auto-correlation of the observed speckle mea-
ui (v) ui (v + d t )∗ 1
T ∑t It ? It surements
(phase uncorrected)
Translating interf. u i (v ) u i (v + dt ) ∗ 1
∑t e jkα It ? It 1
∑ e jkα C wI
T I ,...,IT
C̄ w1 p (∆) = t t
p
(∆) (18)
T t
Algorithm 1. reconstruction pipeline for translating interferom- 1
etry.
=
T ∑ e jkα ∑ t
It (v) It (v + ∆)∗
t v∈ w p
1. Capture interference signal It as discussed in Sec. 3.A.1
2. Calculate local correlations, correct phase and average local with this notation we optimize for a latent image O minimizing
correlations, using Eq. (18). 2
3. Solve for latent image explaining local correlations, Eq. (19). min ∑ C̄ w p − C O
wp , (19)
O p
where we sum over multiple overlapping local windows w p . We
We note that the translating interferometry measurements note that in practice the definition of the local windows in [11]
used here are similar to those used in shearing interferometry is a bit more involved than the one in Eq. (19) and we provide
[23]. However shearing interferometry usually uses smaller the precise optimization cost in appendix.
displacements to obtain the local gradient of the wave, while the In the experimental section below we show that moving from
displacements we use here are larger than the speckle grain size full-frame correlations to local ones has a major impact on noise
so that we obtain independent speckles. elimination and improving the resulting reconstruction.
Table 1 summarize the different modulation approaches eval- Another advantage of the local cost discussed in [11], is that it
uated in this paper. In Fig. 3 we plot the correlation contrast allows recovering patterns larger than the extent of the memory
of Eq. (9) as a function of the number of averaged images T. effect. As mentioned above, ME correlations of the form of
We start by capturing multiple images without any modulation. Eq. (2) only hold for small displacements ∆. When matching the
This only reduces read and photon noise, which does not trans- full-frame auto-correlation (Eq. (6)) of I and O we rely on the
late into a real improvement in correlation contrast. When we fact that ME correlation exists between any two sources in our
randomly modulate the wave (Eq. (12)), the contrast increases latent pattern. This assumption largely limits the the range of
but it eventually saturates as the convolution reduced the ME recoverable illuminator patterns to patterns lying within the ME
correlation. By using our translating interferometry (Eq. (13)), range. The local cost of Eq. (19) only relies on local correlations
the contrast is much improved. When using the phase correction between sources in the same local window. At the same time,
this contrast is linearly increasing as predicted by the theory of the overall extant of the illuminator pattern O can be larger than
Claim 1. This suggests that the measured speckle signals St we these local windows.
generated are indeed uncorrelated for different displacements The full algorithm is summarized in Algorithm 1.
dt1 , dt2 . The graphs in Fig. 3 demonstrate correlation contrast we
captured through two different tissue slices. Beyond each slice
we generated the source layout in the lower row of Fig. 2 us- 3. RESULTS
ing the translating laser setup described below. As we evaluate A. Experiment setup
speckles through real tissue we note that: 1) The exact amount Fig. 4 illustrates our acquisition setup, including an imaging
of correlation we measure in each tissue slice can vary; and 2) arm and a validation arm. The imaging arm consists of an
As each tissue layer generates a speckle spread with a limited objective and a tube lens, followed by a second relay system
support, we only average a finite number of speckle pixels and which allows us to place a spatial light modulator (SLM) at the
the graphs are noisy. Despite these issues, the graphs estimated Fourier plane. The image of the modulated field is collected by
from difference slices demonstrate consistent trends. the main camera. The objective attempts to image incoherent
sources beyond chicken breast tissue slices. The target and the
C. Exploiting local support validation objectives are mounted on z-axis translation stages,
The previous section aims to increasing the auto-correlation facilitating accurate control over focusing in both imaging and
contrast by averaging multiple measurements. Recently [11] validation arms. A second validation camera images the beads
suggested a complementary approach for noise reduction. They from the other end of the tissue, allowing the capture of a clear
observe that when the light sources are inside the sample rather unscattered image of the illuminator layout, which is used to
than far behind it, the speckle pattern scattered from a single asses reconstruction quality. Note that this validation camera
source have local support and the scattered light usually does not does not provide any input to the algorithm. The validation arm
spread over the entire sensor. Therefore they argue that comput- contains a second laser source for simulating incoherent sources
ing the full-frame auto-correlation using the entire image cor- as described next.
rupts the correlation with a lot of additional noise. Rather than We conduct experiments using two kinds of targets: fluo-
searching for a latent pattern O explaining the full-frame auto- rescent beads and shifting laser dots. In the first case we used
correlation as in Eq. (6), they suggest that one should only try Spherotech Fluorescent Nile Red Particles 0.4 − 0.6µm, FP-0556-
to match between the local correlations of the observed speckle 2. The beads are attached on a microscope cover glass right
Research Article 6
Fig. 4. Experiment setup with laser dots or fluorescent beads target. L: lens. Obj: objective lens. BP: bandpass filter. QWP: quarter
wave plate. BS: beam splitter. P-BS: polarized beam splitter. DM: dichromic mirror. We image two kind of targets behind a scatter-
ing tissue: laser dots or fluorescent. For the laser dot target (colored in red), we use mount a fibered laser on a 2D translation stage
to create the pattern we need, and use a objective-tube-lens pair to shrink and focus the dot behind the tissue. For the fluorescent
target (colored in green), we use a laser in front of the tissue to excite the fluorescent beads behind. In either cases, the target is
observed by two pathes. The path to the right directly image the magnified beads by the validation sensor to observe the ground
truth image, and the path to the left is first scattered by the tissue, magnified by another objective-tube-lens pair, modulated by the
SLM in the Fourier plane of a 4f system, and eventually imaged on the main sensor plane. To interfere modulated and unmodu-
lated waves, we horizontally polarize the light by a beam-splitter, rotate the polarization state by 45 degrees with a quarter wave
plate, modulate the horizontal polarized wave by the SLM, rotate the wave again and analyze the wave by the beam-splitter to
interfere two polarization states. Note that when using a fluorescent target, we need to use a dichromic mirror and band-pass filters
to suppress the brightness of excitation laser.
φk ∈ {0, π3 , 2π ◦
behind a chicken breast tissue slice. The separation between 3 }. With our 45 polarization the SLM modulates
the beads and the tissue is as low as the 150µm thickness of the only part of the wave, leading to a measurement
cover glass. The beads are excited with a 532nm laser from the
front side of the tissue. The excitation light scatters through the 2
tissue, illuminates the beads and the emitted light scatters back Ŝti,k = uin (v) + e jφk uin (v + dt ) , (20)
through the tissue to the camera. We filtered fluorescent light
using a 10nm bandpass filter centered at 580nm. Using phase-shifting interferometry (PSI) [24], we can extract
In the shifting laser mode we simulated incoherent sources in the interference signal desired in Eq. (13) as:
controllable positions. For that we imaged the diffused output of
a fibered laser to generate a point focused exactly at the back of 2
the tissue (we use the validation camera for accurate focusing). Sti = ∑ e jφ
k ui (v) + e jφk ui (v + dt ) = ui (v)ui (v + dt )∗ , (21)
The main camera at the other side of the tissue captured the k
intensity scattered from this spot. We then translate the fiber out-
put on an xy stage to generate spots at other positions behind the In the presence of multiple incoherent sources, emission from
tissue. We capture a sequence of images at each source position different sources do not interfere. Thus the measured intensity in
n
and sum their intensities, thus simulating incoherent summation each shot is equivalent to Îtk = ∑n Ŝti ,k . With the phase shifting
n
from multiple sources. This setup allows us to control the layout interferometry in Eq. (21) we extract It = ∑n Sti .
and complexity of the sources, which is useful for analyzing our
algorithm with patterns of varying complexities.
B. Fluorescent bead results
Throughout the experimental section we visualize the fluo-
rescent bead captures with a green colormap and the shifting We start by demonstrating our setup on a fluorescent bead target.
laser captures with a red colormap. In Fig. 1, we reconstruct a 1MB image, corresponding to a field of
view of 300µm × 300µm. The random beads were spread behind
A.1. Interferometric measurements a ~150µm thick tissue. Note that all thickness measurements
While most of our setup is similar to the one used by [11], we in this paper are approximated due to the limited resolution
also use a SLM (Holoeye LETO-GAEA 1920x1080) in the Fourier of the clipper. The bead layout is totally unrecognizable from
plane of the imaging arm, which we use to modulate the scat- the captured speckle input, but our translating interferometry
tered light. To interfere the modulated and unmodulated wave, framework achieves a clear reconstruction from 54 shots. Fig. 5
we use a polarizing beamsplitter to horizontally polarize the demonstrates another reconstruction of beads behind a ~150µm
wave, rotate the polarization state by 45◦ with a quarter wave- thick tissue.
plate, use the SLM to modulate the horizontal polarized part of While the reference and the reconstruction have the same
the wave, rotate the polarization state again and use polarizing layout they have different brightness and resolution. The res-
beamsplitter again to interfere the modulated and unmodulated olution of the reference is subject to the diffraction limit. The
waves. reconstruction algorithm on the other hand encourages sparse
To capture the translating interferometry measurements of results and hence recovered dots tend to be narrower. The bright-
Eq. (13) we place on the SLM a phase ramp whose frequency and ness variation is partially attributed to the fact that the reference
orientation matches the translation dt we want to implement. is captured by a different camera from a different direction, but
We capture K = 3 images of this phase ramp plus a global phasor also due to imperfect convergence of our optimization.
Research Article 7
Validation camera Main camera Our reconstruction
Fig. 5. Reconstruction results for fluorescent beads behind ~150µm-thick tissues: With our translating interferometry, we can
clearly reconstruct fluorescent beads target from 54 shot images captured by the main camera. The reconstruction is compared
against a reference image from the validation camera observing the beads directly.
C. Shifting laser results standard full frame auto-correlation used in previous work [1, 2];
By shifting the laser source behind the target we can simulate and (ii), the modulation approach.
latent illuminator patterns of arbitrary shape and complexity, As discussed by [11] and reviewed in Sec. 2.C, the local ap-
rather than only sparse beads. In Fig. 6 we successfully recover proach detects correlations with a higher SNR compared to the
target patterns through ~150µm thick tissue. Note that the line full frame approach and indeed it leads to better reconstructions.
structure in these targets is created as a sequence of dots. We also show that the translating interferometry modulation
leads to better results compared to simpler modulation alterna-
D. Comparing reconstruction and modulation approaches tives.
In Fig. 7, we evaluate two components of our algorithm: (i) The In the top part of Fig. 2 we reconstruct a simple dot pattern
usage of local correlations [11] discussed in Sec. 2.C, versus the captured with the scanning laser approach. The auto-correlation
Research Article 8
Validation camera Main camera Our reconstruction
Fig. 6. Reconstruction results from our shifting laser setup. Going beyond random bead layout, we shift a laser behind a ~150µm-
thick tissue slice to simulate incoherent source layouts. From a noisy speckle input we reconstruct hidden illuminators with com-
pelling structures.
of the same pattern was evaluated in the lower row of Fig. 2. to compare reconstructions through different tissue thicknesses.
Considering the full frame approach, a single shot leads to totally For the thickest layer the reconstruction failed.
unrecognizable results, which is only slightly improved given As discussed in [11], thicker tissue leads to larger speckle
54 random modulations. Using our translating interferometry spread and weaker memory effect correlation. We used the
without phase correction the result is much improved, but some fact that the scanning laser setup captures speckle patterns by
dots are still missing. Including the phase ramp correction fills- different point sources separately, to evaluate statistical correla-
in the missing dots. On the other hand, the local support results tion through the different tissue layers. In the top part of Fig. 9
are recognizable even from a single shot (no modulation), and we plot the correlation we measured between speckle patterns
they are good with all types of modulation. generated by different sources, as a function of the displace-
In the lower part we reconstruct a denser fluorescent bead ment between the sources. As expected, as the tissue thickness
target. As explained in [11], The increased source density is increases the correlation decays faster as a function of displace-
more challenging to reconstruct as the contrast of the incoherent ment, explaining the reconstruction failure in the lowest row.
speckle image decreases. The full-frame approach failed recon- These results demonstrate the limitation of our method: while
structing this target with any modulation approach and the we can improve correlation contrast, our approach is still based
output of the phase retrieval algorithm is a single dot. The local on the existence of some memory effect correlation and will fail
support algorithm fails with a single shot (no modulation). The when this correlation is too weak.
random modulation reconstructed only a subset of the beads,
the translating interferometry without phase ramp correction
4. DISCUSSION
reconstruct a bigger subset of the beads, and the best results are
obtained when phase correction is included. In this work we demonstrate the reconstruction of fluorescent il-
luminator patterns attached to a scattering tissue. Despite heavy
E. Evaluating reconstruction vs. number of shots scattering that completely distort the captured images we can
exploit memory effect correlations in the measured speckles to
Fig. 3 numerically evaluates the correlation contrast improve-
detect the hidden illuminator layout. While such correlations are
ment as a function of the number of shot images. In Fig. 8 we
inherently weak, we suggest a modulation scheme which allows
visually compare reconstructions using an increasing number of
us to extract multiple uncorrelated speckle measurements from
input shots, and demonstrate how the improved contrast trans-
the same sources. By averaging the correlations of such measure-
lates into better reconstruction quality. The target to reconstruct
ments we increase the signal to noise ratio in the data and largely
in this case is the same as the bottom part in Fig. 7.
boost the reconstruction quality. We combine these modulated
measurements with a recent algorithm [11] seeking a latent pat-
F. Tissue thickness tern whose local correlations agree with local correlations in
In Fig. 9 we used the scanning laser setup to create the same the measured data. The local correlations provide additional
illumination layout behind different tissue slices. This allows us improvement in SNR. Moreover, since it only assumes local cor-
Research Article 9
Random Translating Translating
Ground truth No modulation interferometry
modulation interferometry
(phase uncorrected)
Full-frame
Local support
Full-frame
Local support
Fig. 7. Comparing reconstruction and modulation approaches. Top part: sparse, simple shifting laser target. Lower part: chal-
lenging fluorescent beads target. Local support correlations [11] are stronger than standard full-frame auto-correlations [1, 2], and
our translating interferometry improves over a single shot (with no modulation) and over simple random modulation. For the
simple target on the top, the full-frame algorithm can recover the image given the improved correlation provided by translating
interferometry modulations, but fails to do so from the noisier correlations provided by other modulation strategies. The local
correlation approach which is more robust to noise can recover the target even with the simpler modulations. For the challenging
target at the lower part, the full-frame algorithm fails completely using all types of modulations. The local correlations algorithm
can reconstruct the target using translating interferometry modulations. However given random modulations or even translating
interferometry without phase correction, it can only reconstruct a subset of the beads.
relations between the speckles emitted by nearby sources rather tissue layers. The second challenge for memory effect based
than global full-frame correlations between any two sources in correlations is that we assume speckle variation is observed in
the image, it allow us to reconstruct wide patterns, much behind the captured data. As more and more independent sources are
the limited extent of the memory effect. Overall we demonstrate present, the speckle contrast decays and the captured intensity
the reconstruction of mega-pixel wide patterns, limited only by images are smoother. Naturally, when speckle contrast is lower
the sensor size. than the photon noise in the data no correlations can be detected.
Despite the advance offered by this approach it is inherently Hence memory effect based techniques are inherently limited to
dependent on the existence of some memory effect correlation, simple, sparse illuminator layouts.
and when such correlations are too weak to be measured our ap- An alternative approach for seeing through scattering tissue
proach will fail as well. Two major factors that decrease the mem- is based on wavefront shaping optics. Rather than post-process
ory effect correlations are the tissue thickness and the number of the speckle data, it attempts to modulate the incoming excitation
independent incoherent sources. In our current implementation light and/or the outgoing emission to undo the tissue aberra-
we recovered sources behind 150µm which is much beyond the tion in optics. In theory this approach carries the potential to
penetration depth of a standard microscope, yet there is a need extend to thicker tissue layers and to correct complex patterns.
for imaging techniques which can propagate through thicker In practice, efficiently finding a proper modulation mask is a
Research Article 10
1 image 6 images Input from main
Our reconstruction
camera
~100µm
18 images 54 images
~150µm
Fig. 8. Evaluating reconstruction vs. number of shots. We vi-
sualize the quality of the reconstruction given no modulation
(single shot), and with an increasing number of modulation
~200µm
patterns. As evaluated numerically in Fig. 3 the correlation
contrast increases linearly with the number of shots and in
agreement the visual quality of the reconstruction improves.
The reference speckle layout in this experiment is equivalent
to the validation camera image in the lower part of Fig. 7.
challenging task. Recently [20] has manages to recover such a
modulation efficiently using linear (single photon) fluorescent
feedback from a sparse set of beads. However, even after recover-
ing a good modulation mask, the area they could correct with it
is limited due to the limited extent of the memory effect. In con-
trast, our approach can recover wide-field-of-view, full-frame
images, as it only relies on local memory effect correlations.
Supplemental document. See Appendix for supporting content.
REFERENCES
Fig. 9. Compare reconstructions with different tissue thick-
1. O. Katz, P. Heidmann, M. Fink, and S. Gigan, “Non-invasive single- ness. Top panel: we use the shifting laser acquisition to cap-
shot imaging through scattering layers and around corners via speckle ture the same illumination pattern through three different
correlation,” Nat. Photonics (2014).
tissue slices of increasing thicknesses. As tissue thickness in-
2. J. Bertolotti, E. G. van Putten, C. Blum, A. Lagendijk, W. L. Vos, and
crease the spread of the speckles is wider and the statistical
A. P. Mosk, “Non-invasive imaging through opaque scattering layers,”
(2012). correlations are weaker. With the ~200µm-thick tissue the cor-
3. K. T. Takasaki and J. W. Fleischer, “Phase-space measurement for relations are too wreak and reconstruction fails. Lower panel:
depth-resolved memory-effect imaging,” Opt. Express (2014). As the shifting laser setup allows capturing speckles from in-
4. E. Edrei and G. Scarcelli, “Optical imaging through dynamic turbid dividual sources independently, we can empirically compute
media using the fourier-domain shower-curtain effect,” Optica (2016). the decay of speckle correlation as a function of the distance
5. E. Edrei and G. Scarcelli, “Memory-effect based deconvolution mi- between the sources. Indeed for thicker tissue slices the cor-
croscopy for super-resolution imaging through scattering media,” Sci. relation decays faster as a function of source separation. This
Reports (2016). explains the reconstruction failure for the ~200µm-thick tissue
6. M. Hofer, C. Soeller, S. Brasselet, and J. Bertolotti, “Wide field fluores-
example.
cence epi-microscopy behind a scattering medium enabled by speckle
correlations,” Opt. Express (2018).
7. T. Wu, J. Dong, X. Shao, and S. Gigan, “Imaging through a thin scatter-
ing layer and jointly retrieving the point-spread-function using phase- a point-spread function of a memory effect based scattering imaging
diversity,” Opt. Express (2017). system,” Opt. Lett. (2020).
8. T. Wu, J. Dong, and S. Gigan, “Non-invasive single-shot recovery of 9. X. Wang, X. Jin, and J. Li, “Blind position detection for large field-of-viewResearch Article 11
scattering imaging,” Photon. Res. (2020).
10. J. Chang and G. Wetzstein, “Single-shot speckle correlation fluores-
cence microscopy in thick scattering tissue with image reconstruction
priors,” J. Biophotonics (2018).
11. M. Alterman, C. Bar, I. Gkioulekas, and A. Levin, “Imaging with local
speckle intensity correlations: theory and practice,” ACM TOG (2021).
12. M. Zhou, A. Pan, R. Li, Y. Liang, J. Min, T. Peng, C. Bai, and B. Yao,
“Retrieval of non-sparse object through scattering media beyond the
memory effect,” J. Opt. (2020).
13. D. F. Gardner, S. Divitt, and A. T. Watnik, “Ptychographic imaging of
incoherently illuminated extended objects using speckle correlations,”
Appl. Opt. (2019).
14. G. Li, Y. Wanqin, W. Haichao, and G. Situ, “Image transmission
through scattering media using ptychographic iterative engine,” Appl.
Sci. (2019).
15. L. Li, X. Wen, R. Song, J.-T. Jiang, H.-L. Zhang, X.-B. Liu, and L. Wei,
“Imaging correlography using ptychography,” Appl. Sci. (2019).
16. N. Shekel and O. Katz, “Using fiber-bending-generated speckles for
improved working distance and background rejection in lensless micro-
endoscopy,” Opt. Lett. 45, 4288–4291 (2020).
17. A. Boniface, J. Dong, and S. Gigan, “Non-invasive focusing and imaging
in scattering media with a fluorescence-based transmission matrix,”
Nat. Commun. (2020).
18. L. Zhu, F. Soldevila, C. Moretti, A. d’Arco, A. Boniface, X. Shao, H. B.
de Aguiar, and S. Gigan, “Large field-of-view non-invasive imaging
through scattering layers using fluctuating random illumination,” arXiv
preprint arXiv:2017.08158 (2021).
19. A. Boniface, B. Blochet, J. Dong, and S. Gigan, “Noninvasive light fo-
cusing in scattering media using speckle variance optimization,” Optica
(2019).
20. D. Aizik, I. Gkioulekas, and A. Levin, “Fluorescent wavefront shap-
ing using incoherent iterative phase conjugation,” arXiv preprint
arXiv:2111.00000 (2022).
21. G. Osnabrugge, R. Horstmeyer, I. N. Papadopoulos, B. Judkewitz, and
I. M. Vellekoop, “Generalized optical memory effect,” Optica (2017).
22. C. Bar, M. Alterman, I. Gkioulekas, and A. Levin, “Single scattering
modeling of speckle correlation,” in ICCP, (2021).
23. M. Riley and M. Gusinow, “Laser beam divergence utilizing a lateral
shearing interferometer,” Appl. Opt. 16, 2753–2756 (1977).
24. P. Hariharan, B. Oreb, and T. Eiju, “Digital phase-shifting interferometry:
a simple error-compensating phase calculation algorithm,” Appl. Opt.
26, 2504–2506 (1987).
25. D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,”
ICLR (2015).
26. J. M. Rodenburg, A. C. Hurst, A. G. Cullis, B. R. Dobson, F. Pfeiffer,
O. Bunk, C. David, K. Jefimovs, and I. Johnson, “Hard-x-ray lensless
imaging of extended objects,” Phys. Rev. Lett. 98 (2007).Research Article 12
Imaging Through Scattering Media using Speckle Modulation
Supplementary Appendix
A. APPENDIX Claim 2 For displacements dt1 , dt2 whose distance kdt1 − dt2 k is
larder than the speckle grain, the signals Sti1 , Sti2 are uncorrelated, so
A. Analyzing correlation properties in translating interferome-
that
try measurements h i ∗
h i h i ∗
E Sti1 · Sti2 − E Sti1 · E Sti2 =0 (29)
In this section we prove Claims 1–3 of the main paper, analyzing
the various properties of translating interferometry measure-
Proof: Our derivation is based on the assumption that the
ments.
speckle fields ui have zero means, and the speckle values in
Claim 1 We define correlation contrast different sensor positions v are independent random variables.
For a non zero displacement dt we get
h i2
∑∆∈Γ∆ E C Ī1 ,..., ĪT (∆)
1
|Γ∆ |
h i h i
E Sti E ui (v) ui (v + dt )
Θ C Ī1 ,..., ĪT = h i (22) =
1 Ī1 ,..., ĪT
Γ̄
| ∆| ∑ ∆ ∈ Γ̄ ∆
E |C ( ∆ )| 2 h i h i
= E ui (v) E ui (v + dt ) = 0. (30)
If the speckle patterns St0 are uncorrelated with each other for differ-
ent t values, than the correlation contrast increases linearly with the In a similar way
number of measurements T. h ∗
i h ∗ ∗
i
E Sti1 Sti2 = E ui (v ) u i (v + dt1 ) ui (v ) ui (v + dt2 ) (31)
Proof: The claim is based on the observation that for dis- h i
placements ∆ ∈ Γ∆h that correspond to an actual illumina- = E |ui (v)|2 ui (v + dt1 )∗ ui (v + dt2 ) (32)
i
tors displacement E C Ī1 ,..., ĪT (∆) is a positive quantity, while
h i h i∗ h i
= E |ui (v)|2 E ui (v + dt1 ) E ui (v + dt2 ) (33)
for ∆ ∈ Γ̄∆ , which do not correspond to a displacement be-
tween two illuminators, no correlation exists and in expectation = 0. (34)
h i
E C Ī1 ,..., ĪT (∆) = 0. Where Eq. (33) follows again from the assumption that speckle
With this understanding we note that expectation is linear at different pixel positions are independent. Eq. (30) and Eq. (34)
and hence recalling the definition of C Ī1 ,..., ĪT (∆) in Eq. (10) of prove the desired Eq. (29).
the main paper, the numerator of Eq. (22) is independent of the We introduce the following claim, which we will use to prove
number of measurements T: Claim 3 of the main paper.
1 2
h i 1 h i h i
Claim 3 For speckle fields ui (v), ui (v) satisfying the tilt shift re-
E C Ī1 ,..., ĪT (∆) = ∑ E C Īt (∆) = E C Ī1 (∆) . (23)
T t lationship of Eq. (2), the translating interferometry measurements
1 2
Sti (v) and Sti (v) defined in Eq. (13) are shifted versions of each
We now move to express the denominator. Using again the other, times a globally constant phasor, which is independent of pixel
definition of the average correlation in Eq. (10) of the main paper, position v. This is expressed by the relationship:
we express it as
Sti (v) ≈ Sti (v + ∆)e− jkα ,
1 2
(35)
h 1 i h ∗i
E |C (∆)| = 2 ∑ E C Īt1 (∆) · C Īt2 (∆)
Ī1 ,..., ĪT 2
(24) with ∆ = i2 − i1 .
T (t ,t )
1 2
Proof: From Eq. (2), we have ui (v) ≈ e jkα ui (v + ∆).
1 2
1 h ∗i
= 2 ∑ E C Īt (∆) · C Īt (∆) (25) Substituting into Eq. (13) we have,
T t
1 h i h i∗
+ 2 ∑ E C Īt1 (∆) · E C Īt2 (∆) (26) 1 1 1
T (t 6=t )
1 2
Sti (v) = ui (v) ui (v + dt ) ∗
e jkα ui (v + ∆)e− jkα ui (v + ∆ + dt )∗
2 2
1 h i
≈
+ 2 ∑ E |C Īt (∆)|2 (27)
T t
e− jkα ui (v + ∆)ui (v + ∆ + dt )∗
2 2
=
1 h i
e− jkα Sti (v + ∆)
2
= E |C Ī1 (∆)|2 (28) =
T
Where Eq. (26) follows from the assumption that for t1 6= t2 the
speckles St01 , St02 are uncorrelated with each other, and Eq. (27)
from the fact that for displacements ∆ ∈ Γ̄∆ the correlation has Claim 4 Using the translating interferometry measurements of
zero expectation. Eq. (13), the speckle auto-correlation C It = It ? It is equivalent to the
From Eq. (28) we conclude that as we increase the number of auto-correlation of the latent pattern C O = O ? O, times a phase ramp
measurements the denominator scales as 1/T. As a result the correction
correlation contrast in Eq. (22) scales linearly with T. C It (∆) ≈ C O (∆)e− jkα . (36)Research Article 13
Proof: Using Claim 3 for the specific case i1 = i and i2 = 0 Full Frame Correlation
we get Sti (v) ≈ St0 (v + i)e− jkα . Summing over all sources
Sti (v) we have
Ground Truth
It (v) ≈ St0 ∗ Õ.
with
Õ(v) = O(v)e− jkα . (37)
This is due to the fact that O is non zero only at positions v =
in for one of the sources, so effectively Õ has for each sensor
position i the global phasor of Eq. (35). As in the standard
derivation of the speckle auto-correlation we assume St0 ? St0 = δ,
Input
and hence
It ? It ≈ Õ ? Õ. (38)
or equivalently
C It (∆) ≈ C Õ (∆). (39)
Local Correlation
Hence we are left with the need of computing C Õ (∆). We note
that by the Wiener-Khinchin theorem, C Õ (∆) is the inverse
2
Fourier transform of F (Õ) . However as Õ is obtained by
multiplying O with a phase ramp (see Eq. (37)), their Fourier
transforms are related via a shift:
F (Õ)(ω) = F (O)(ω + αdt ). (40)
Fig. 10. Local versus global auto-correlation. The orientation
The shift relation holds also for their absolute values of the auto-correlation evaluated in three different local win-
2 2 dows of the image matches the orientation of the arc in the
F (Õ)(ω) = |F (O)(ω + αdt )| (41)
corresponding region of the latent image. By contrast, the auto-
correlation of the full frame is much nosier, and decays for
Hence C Õ (∆) and C O (∆) are related via a tilt:
large displacements due to limited ME.
C Õ (∆) = C O (∆)e− jkα . (42)
this relationship does not hold and the approach reduces to the
Substituting Eq. (42) in Eq. (39) proves the desired Eq. (36).
baseline full-frame auto-correlation algorithm.
The algorithm searches for a latent image O such that the auto-
correlation in its local windows will match the auto-correlation
B. Optimizing using local support in the local windows of the input image I. We define w∆ and
For sources located inside the scattering medium, speckle pat- wτ to be binary windows with support T∆ , Tτ , respectively, and
terns emerging from a single source have local support and do w̄2τ = wτ ? wτ —note that, from its definition, w̄2τ is non-binary.
not spread over the entire sensor. To take advantage over this Then, we recover O by solving the optimization problem:
property, [11] suggest to match the local speckle correlations in
the image, rather than the full-frame auto-correlation. We review min ∑ p k T1 ∑t e jkα · It,wτp ? It,w∆p − Ow̄2τ
p
? Ow∆p k2 , (43)
O
this algorithm below.
For motivation, consider Fig. 10 that we re-plot from [11]. It where It,wτp , It,w∆p , Ow̄2τ , Ow∆p denote windows of a given size
p
visualize speckles produced by latent incoherent illuminators
cropped from the input and latent images, centered around
in a double arc layout. Computing auto-correlation at small
the p-th pixel.
subwindows of the speckle image reveals the local orientation
Eq. (43) uses windows of three different sizes, and we use
of the arc in the latent image. By contrast, when computing the
Fig. 11 to visualize their different roles: Each wτp is a small win-
auto-correlation of the full frame, the correlation is considerably
dow around pixel p whose support is equivalent to the expected
noisier even for small displacements. Correlations between far
support size of the speckle pattern due to a single illuminator.
illuminators are even harder to detect due to the limited ME
w∆p is a larger window around the same pixel, corresponding
range.
to the maximal displacement T∆ for which we expect to find
The optimization algorithm takes as input two threshold
correlation, as dictated by the ME range.
parameters Tτ , T∆ . It assumes that speckles from one illuminator
We note, additionally, that the window cropped from O
are spread over pixels in a window of size Tτ around it, and
should be wider than that from I. This is because speckle at
that ME correlation holds for displacements |∆| < T∆ . The
a certain pixel can arise from an illuminator within a window
thresholds Tτ , T∆ are free parameters that can be fine-tuned to
around it. For example, in Fig. 11, no illuminator is located
improve reconstruction quality, and [11] show that performance
inside the cyan subwindow of O, but part of the speckle pattern
are not too sensitive to their exact values. The algorithm offers
of a neighboring source is leaking into the corresponding cyan
improved performance compared to the baseline full-frame auto-
subwindow of I. As a result Owτp ? Ow∆p is a zero image, even
correlation algorithm in situations where Tτ < T∆ , namely when
the support from one illuminator is lower than the ME range. For though Iwτp ? Iw∆p detects three impulses. It is easy to prove that
thick scattering slices, where high-order scattering is dominant, this can be addressed using the larger, non-binary window w̄2τResearch Article 14
I O efficiently, e.g., using a GPU based fast Fourier transform. For
initialization, we set the latent image to random noise; we have
observed empirically that the optimization is fairly insensitive
to initialization. Finally, we note that even though we could
place a window w p around every pixel of I, the empirical corre-
lation is insensitive to small displacements of the central pixel
p. Therefore, in practice, we consider windows only at strides
Tτ /2, which helps reduce computational complexity.
We note that the optimization problem of Eq. (43) is similar
to ptychography algorithms [26]. However, we emphasize that
Iwτ ? Iw∆ Ow τ ? Ow ∆ Ow̄2τ ? Ow∆ previous ptychographic approaches for extending the ME range
recover the latent illuminators from multiple image measure-
ments, captured by sequentially exciting different areas on the
scattering sample [12–16]. By contrast, our algorithm recovers
the latent illuminators from a fixed number of full-frame shots.
C. Additional results
I O In Fig. 12 we show additional reconstruction results using flu-
orescent bead targets as well as laser dots targets. Both used
tissue slices of thickness around ~150µm.
Iwτ ? Iw∆ Ow τ ? Ow ∆ Ow̄2τ ? Ow∆
Fig. 11. Local window selection for optimization. We con-
sider local subwindows wτ (light green and cyan frames)
whose support is equivalent to the speckle support size. Each
such window is correlated with a wider window w∆ (yellow
and blue frames) around it, whose support is equivalent to
the ME range. As speckle inside window wτ can arise from
a source outside wτ , Owτ ? Ow∆ may not match Iwτ ? Iw∆ . To
overcome this, we use an extended non-binary sub-window
w̄2τ = wτ ? wτ for O, whose support is indicated by dashed
lines.
in the latent image, indicated in Fig. 11 using dashed lines: in
this case, Ow̄2τ p
? Ow∆p correctly detects the same three impulses
as Iwτp ? Iw∆p .
The motivation for the cost of Eq. (43) is that, even if two illu-
minators in the latent pattern O are at a distance larger than the
ME range T∆ , they can be recovered if there exists a sequence of
illuminators between them, where each two consecutive illumi-
nators in the sequence are separated by a distance smaller than
T∆ . For example, in Fig. 11, the illuminators outside the yellow
and cyan w∆ windows are recovered thanks to the intermediate
illuminators.
The optimization problem in Eq. (43) is no longer a phase
retrieval problem as in standard full-frame auto-correlation al-
gorithms. We minimize it using the ADAM gradient-based opti-
mizer [25]. Gradient evaluation is described in [11], and reduces
to a sequence of convolution operations that can be performedResearch Article 15
Validation camera Main camera Single image reconstruction Our reconstruction
Fig. 12. Additional reconstruction results for fluorescent beads and laser dots behind 150µm-thick tissues Top panel: fluorescent
beads results. Lower panel: laser dots results.You can also read
