## Abstract

An inexpensive method to convert a microscope into an imaging spectrometer is presented. Unlike current microscope-based spectrometers which use specialized optics or scanning mechanisms, our system only requires at most two image captures with a 3-chip CCD camera and a lightly-tinted color filter to output the color signal of a sample at each pixel. Basis spectra are obtained by principal components analysis applied to an ensemble of color signals of commercially-available dyes observed with different dichroic mirrors. A transformation matrix from channel values to spectral coefficients is derived. Minimum negativity constraint is applied to eliminate negative parts of the reconstructed fluorescence spectrum. The technique is demonstrated on fluorescence microspheres (fluorospheres) and chlorophyll from plant leaf.

©2002 Optical Society of America

## 1. Introduction

Systems that convert a microscope into an imaging spectrometer, delivering the emittance, reflectance or color signal of a sample at pixel resolution, normally require affixing special optics to the microscope, fitting a scanning mechanism, or taking a large number images of the sample [1–4]. Hence such spectral imaging systems tend to be expensive. Here, we demonstrate a simple system for estimating the fluorescence spectrum at an arbitrary (pixel) location of a fluorescence image from its captured image color using just two or three image captures taken with minimal equipment. Our assemblage consists only of a 3-chip CCD camera attached to the fluorescence microscope and 1 to 2 lightly-tinted color filters. Basis spectra or eigenspectra are derived from an ensemble of known fluorescence color signals using Principal Components Analysis (PCA). Compared with Singular Value Decomposition, PCA normally yields fewer sets of eigenspectra that are needed to reconstruct color signals since the ensemble is mean-centered and the eigenvalues are more representative of the real variances of the ensemble. Fluorescence spectrum is estimated from its image color by mapping its color channel values (e.g. red, green and blue signals) to eigenspectra coefficients; Section 2.1 derives the transformation matrix. Estimating a spectrum from only a few measurement points is an underdetermined problem. However, since most fluorophores have very similar spectral characteristics, we assume that a weighted sum of the first few eigenspectra (ranked in descending order of eigenvalues) will give a good first estimate of the actual spectrum. The number of color channels that can be obtained limits the number of eigenspectra that can be used for spectrum recovery. This limitation and the fact that eigenspectra, in general, have negative values, causes the estimated spectrum to have negative values. We improve the estimate by employing a minimum negativity constraint, discussed in Section 2.2, that allows us to: (1) compute additional coefficients, and (2) put correction terms to the initial estimated coefficients derived from image color. Our approach is similar to [5] but with the addition of a minimum negativity constraint to ensure an all-positive reconstructed fluorescence color signal. Because our technique requires only at most two image captures, sample exposure time is minimal and the fluorescence spectra of live and motile samples can be examined before severe photobleaching sets in.

## 2. Method

#### 2.1 Mapping from RGB to eigenspectra coefficients

Typically, fluorescent samples are excited with short wavelength light and observed with a filter that excludes the excitation wavelengths. The observed color signal C(λ) of a fluorescent sample is the product of its fluorescence emittance spectrum E(λ) and the effective transmission spectrum of the filter used, *F _{eff}(λ)*, i.e.,

where *F _{eff}(λ)* is the product of the transmission curve of a dichroic mirror

*F*and a barrier filter

_{DM}(λ)*F*,

_{B}(λ)A color camera viewing the sample generates a camera channel output *Q _{m}* given by

where *S _{m}(λ)* is the spectral sensitivity for the

*m*-th channel. For a 3-channel color camera, Q

_{1}, Q

_{2}, Q

_{3}corresponds to the color components Red, Green and Blue (RGB), respectively.

We use an existing ensemble of color signals from known fluorescence emittance spectra of dyes and transmittances of common dichroic mirrors. We also limit our ensemble of *C(λ)*’s to those which are smooth and unimodal or at most, bimodal. This restriction is valid for biological samples since majority of the recorded spectra of fluorescing marine organisms or terrestial plants are unimodal and only few species have two peaks or secondary shoulders [6]. The eigenspectra *e _{i}(λ)* is obtained by applying the PCA on the ensemble. Our assumptions ensure that only a few eigenspectra are needed for its spectral reconstruction.

The reconstructed color signal of C(λ), C̃(λ)in terms of the first few significant eigenspectra is:

where *C _{mean}(λ)* is the mean of the spectral ensemble and N is the number of eigenspectra used.

Coefficients *a _{i}* are computed by

Combining Eqs (3) and (4) we get:

or **Q** = **Ta** + **Q _{mean}** where

**Q**is the vector at the left side of Eq (6),

**Q**is the second term in the right side and

_{mean}**T** is the transformation matrix that maps *a _{i}* to the

*Q*of the image. Since we wish to recover

_{m}*C(λ)*,

**a**’s are unknown.

For a colored image the camera signals **Q** are the RGB components. Hence, the only unknown variable in Eq. (6) is the coefficient vector **a**. By matrix operation, **a** can be found using

where **T ^{-1}** is the inverse of

**T**. It is important to note that the inversion matrix

**T**in Eq. (8) is defined only if

^{-1}**T**is a square matrix. Because the size of

**T**is equal to

*M*×

*N*,

**T**exists only if

^{-1}*N*and

*M*are equal.

To increase *M* the sample is image-captured with a lightly-tinted colored filter placed before the camera. A color camera normally has *M* = 3, (for R, G, B) but with the filter inserted, the fluorescent sample is effectively imaged under 6 independent channels (*M* = 6). Changing filters and recapturing images further increase *M* by multiples of 3. The filters to be used must not be spectrally flat, as in the case of neutral density filters, because this will merely cause RGB values to be scaled by a constant and will lead to singularities in **T ^{-1}**. Thus only lightly-tinted colored filters are suitable because, in effect, they substantially alter the spectral sensitivities of each of the camera channels.

Figure 1 shows the proposed spectral imaging system. A 3CCD camera is attached to a microscope and in-between them are changeable colored filters. A personal computer which stores the pre-computed inversion matrix **T ^{-1}**, digitizes the N/3 captured images of fluorescent samples.

For each pixel value, **a** is computed using Eq. (8). We have found that 5 basis spectra give the least residual error in the calculation of **T ^{-1}** which implies that only two colored images are needed for spectral recovery, one taken by the colored CCD camera and another taken with a lightly-tinted color filter inserted before the camera.

#### 2.2 Minimum-negativity constraint

Since only the first *M* = *N* eigenvectors are utilized, *C*̃ can erroneously contain negative values which may be due to the lack of higher-order eigenvectors. The negative values may be minimized by recovering the lost high-order eigenvectors i.e. by increasing N. Furthermore, the first *N* computed coefficients may still be incremented to produce an all-positive *C*̃. We minimize a function ** f** defined as the sum of the squared value of the negative parts of

*C*̃ plus an adjustment term which uses n ≥ N eigenvectors:

where *α _{I}*’s are the correction coefficients and

*H(λ)*is the Heaviside step function:

*H(λ)* excludes the positive values of *C*̃ allowing the summation in Eq. (9) to include only the negative values. The *α _{j}*’s are derived by getting the partial derivative of Eq. (9) with respect to the unknown

*α*and setting it to zero:

_{j}Performing Eq. 11 up to **n**
^{th}
*α _{j}* will lead to

or **B α** = **M** where “∙” in Eq. (12) represents the dot product performed over the negative parts of *C*̃ only. **B** is the square matrix of dot products of * e*’s, and

**M**is the column matrix on the right side of Eq. (12). Because the vector

**α**is the only unknown, it is computed as

The all-positive reconstructed spectrum is then given by

Reconstruction error is measured using Normalized Mean Square Error (NMSE) defined as $\mathit{NMSE}=\sqrt{\frac{\left\{{\displaystyle \sum _{\lambda}}{\left[{C}_{m}\left(\lambda \right)-{C}_{\mathit{rec}}\left(\lambda \right)\right]}^{2}\right\}}{\left\{{\left[{\displaystyle \sum _{\lambda}}{C}_{m}\left(\lambda \right)\right]}^{2}\right\}}}$ and correlation coefficient $\mu =\frac{\u3008\left({C}_{m}-{C}_{\mathit{mean}}\right)\left({C}_{\mathit{rec}}-{C}_{\mathit{mean}}\right)\u3009}{\sqrt{\u3008\left({C}_{m}-{C}_{\mathit{mean}}\right)\left({C}_{m}-{C}_{\mathit{mean}}\right)\u3009\u3008\left({C}_{\mathit{rec}}-{C}_{\mathit{mean}}\right)\left({C}_{\mathit{rec}}-{C}_{\mathit{mean}}\right)\u3009}}$ where *C _{m}* and

*C*is the measured (true) and average color signal spectrum of the fluorescent sample, respectively. Perfect reconstruction corresponds to

_{mean}*NMSE*= 0. Because

*μ*is the normalized measure of the strength of the linear relationship between the measured and the reconstructed spectrum,

*μ*= 0 for uncorrelated spectra and

*μ*= 1 for equivalent ones. Correctly-positioned but oppositely-valued peaks corresponds to

*μ*= -1. The procedure only has one free parameter to adjust,

*n*, which is the number of correction coefficients to recover and it may be increased beyond N until NMSE is closest to zero, μ is closest to 1 and f in Eq. (9) is lowest.

## 3. Experiment

The image is produced with an Olympus BH2 fluorescence microscope, captured by a Hitachi HV-C20 3CCD camera and digitized by a Matrox Framegrabber. The spectral sensitivities of the camera channels and the relative transmittance of the colored filter are posted at: http://www.nip.upd.edu.ph/ipl. The camera is pre-calibrated to give a linear response. We demonstrate our technique using two samples: (1) 0.04 μm carboxylate-modified fluorospheres (Molecular Probes) having excitation/emission at 505/515 nm, and (2) Chlorophyll b in plant leaf (*Bambusa Multiplex Rivierorum*). Both are viewed under a UV objective (0.65 numerical aperture, 40×) and a “G-excitation” dichroic mirror which allows the reflection of excitation light starting at 545 nm and the transmission of fluorescent emission beginning at 590 nm. The recovered color signal from the fluorosphere is clipped at the cutoff of the G-excitation dichroic mirror. We did not use the next available dichroic mirror (B-excitation) because it excludes the excitation wavelength of the fluorosphere. Calculations were performed using MATLAB.

Emittances of 121 fluorescent dyes were taken from Molecular Probes [7]. Together with the transmittances of three different dichroic mirrors, a total of 363 color signals are computed. Camera channel sensitivities were obtained from the manufacturer. All spectra are discretized at 1 nm-resolution. PCA is performed on the ensemble of color signals to yield 301 eigenspectra and the **T ^{-1}** was computed via Eq. (7) using the first 5 eigenspectra and the camera spectral sensitivity data.

The reference chlorophyll color signal was measured using SPECIM Imspector V9 Imaging Spectrometer while the reference fluorosphere spectrum was obtained from Molecular Probes [7]. To limit noise effects, the RGB from a 10×10 block representation of the sample image is averaged and used to recover the fluorescence color signal. Figure 2 shows images of Chlorophyll b (2a) and fluorospheres (2b) samples. Right column of Fig. 2 presents the measured spectra (black), the reconstructed spectra using 5 coefficients (blue) without minimum negativity, and the corrected spectra using 10 coefficients (red) with minimum negativity constraint applied. The recovered peaks are accurate to within ±5 nm of the actual peak. Figure 3 shows the dependence of *f*, NMSE and μ values for the two samples with the number n of recovered coefficients.

## 4. Conclusions

We have demonstrated an inexpensive system for estimating the spectra at any arbitrary pixel location of a fluorescence image at a spectral resolution that is dependent on the information content of the spectral data ensemble. The technique uses a linear mapping of color camera outputs to the basis coefficients. Minimum negativity constraint is utilized to improve the estimate and reduce negative parts of the recovered spectra.

## Acknowledgement

This work received financial support from the University of the Philippines Creative & Research Scholarship Fund.

## References and Links

**1. **S. Kawata, K. Sasaki, and S. Minami’ “Component Analysis of Spatial and Spectral Patterns in Multispectral Images,” J Opt Soc Am A. **4**, 2101–2106 (1987). [CrossRef] [PubMed]

**2. **E. Schrock et al, “Multicolor Spectral Karyotyping of Human Chromosomes,” Science **273**, 494–497 (1996). [CrossRef] [PubMed]

**3. **D. Youvan et al. “Fluorescence Imaging Micro-Spectrophotometer (FIMS),” Biotechnology et alia , **1**, 1–16, (1997).

**4. **B. Ford, M. R. Descour, and R. M. Lynch, “Large-image-format computed tomography imaging spectrometer for fluorescence microscopy,” Opt. Express **9**, 444–453 (2001). http://www.opticsexpress.org/abstract.cfm?URI=OPEX-9-9-444 [CrossRef] [PubMed]

**5. **F. Imai and R. Berns. “Spectral Estimation Using Trichromatic Digital Cameras,” .Proc. of the International Symposium on Multispectral Imaging and Color Reproduction for Digital Archives, 42–49, 1999.

**6. **P. Herring, “The Spectral Characteristics of Luminous Marine Organisms,” Proc Roy Soc Lond **B220**, 183–217 (1993).

**7. **R. Haugland, “Handbook of Fluorescent Probes and Research Chemicals, 6^{th} ed.,” Molecular Probes, Inc., 1996.