US20110090332A1
2011-04-21
12/927,602
2010-11-18
The invention relates to an apparatus and processes for optical measurement and detection with real-time closed-loop controls, which enable higher levels of performance. The invention is especially suitable for applications such as spectroscopy; microscopy; biochemical assays; processes and reactions on miniaturized formats (such as those involving micro-/nano-plates, micro-formats & micro-arrays, chemistry-on-chip, lab-on-chip, micro-channels and micro-fluidics, where dimensions are on micron scale and volumes are in the sub-nanoliter range). Such “intelligent sensing” allows higher data quality and reliability, higher measurement and analysis throughput and lower cost.
The invention uses fast real-time adaptive digital signal processing and controls directly at the point where data is sensed. Through real-time adaptive control of sensors, chemical/opto-mechanical/opto-electronic processes and other components during the measurement process, consistently higher quality results and higher reliability are achieved. This invention furthermore includes an improved image sensor architecture that enables very high intra-array dynamic range at fast frame rates and low noise performance.
Get notified when new applications in this technology area are published.
G01J3/02 » CPC main
Spectrometry; Spectrophotometry; Monochromators; Measuring colours Details
G01J3/0286 » CPC further
Spectrometry; Spectrophotometry; Monochromators; Measuring colours; Details Constructional arrangements for compensating for fluctuations caused by temperature, humidity or pressure, or using cooling or temperature stabilization of parts of the device; Controlling the atmosphere inside a spectrometer, e.g. vacuum
G01J3/2803 » CPC further
Spectrometry; Spectrophotometry; Monochromators; Measuring colours; Investigating the spectrum using photoelectric array detector
H04N5/341 » CPC further
Details of television systems; Transforming light or analogous information into electric information using solid-state image sensors [SSIS] Extracting pixel data from an image sensor by controlling scanning circuits, e.g. by modifying the number of pixels having been sampled or to be sampled
H04N5/374 » CPC further
Details of television systems; Transforming light or analogous information into electric information using solid-state image sensors [SSIS]; SSIS architecture; Circuitry associated therewith Addressed sensors, e.g. MOS or CMOS sensors
H04N5/3765 » CPC further
Details of television systems; Transforming light or analogous information into electric information using solid-state image sensors [SSIS]; SSIS architecture; Circuitry associated therewith; Addressing circuits Timing or clock signal generating circuits
H04N7/18 IPC
Television systems Closed circuit television systems, i.e. systems in which the signal is not broadcast
This application is a continuation of co-pending application Ser. No. 10/380,570, which was a §371 submission of international application PCT/EP01/11027, filed 24 Sep. 2001, and which claims the benefit of the filing date of DE 100 47 299.0, filed 25 Sep. 2000.
This invention relates to an image sensor device, an apparatus and a method for optical measurements, in particular for optimizing optical measurements through real-time closed-loop control.
Current biotechnology instrumentation: Imaging and spectroscopy is most commonly used in applications such as microscopy; readers for microplates, gel plate electrophoresis, microscope slides and chips; and capillary electrophoresis. Typically the biochemical process is monitored or a result is detected by optical measurement, primarily using fluorescence spectroscopy or chemiluminescence. Common detectors are Cooled scientific CCD and CID cameras, or Photomultiplier Tubes (PMT's).
The most common format for automating biochemistry reactions and assays is the microtiterplate, which provide 96,384, and recently over one thousand locations (vials or wells) for holding samples or reagents. The dimensions of these plates are on a macro scale of centimeters (approx. 12 cm×8 cm), and reagent volumes in the microliter range. Cameras, or PMT with fiber optic or scanning laser designs are typically used for detection. Microscopy cameras are intended for mounting onto standard microscopes, with connection to a computer. For spectroscopic measurements, optical filters are typically used. Capillary and gel electrophoresis commonly use laser-induced fluorescence or radioactive labeling of molecules, the former using either CCD cameras together with spectrographs or PMT's with optical filters. The target units (samples) to be simultaneously measured currently number up to 96, and can have spacing of close to 100 microns. There are 2-dimensional capillary array designs of similar spacings in the literature. New miniature formats for higher throughput such as nano-plates, BioChips & Arrays, Chemistry-on-a-chip are now emerging. To analyze these, “biochip readers” are available which image the biochip with either scanning lasers and PMT's or cooled scientific CCD cameras.
In general, current instrumentation uses one-time factory alignment and calibration, since mechanical tolerances are acceptable (examples are focusing of optics, spatial location of images, spectral calibration). The sensor or camera has pre-defined operating settings, which it uses to acquire and transmit raw data (most commonly in the form of images) to a host computer. This host then has the opportunity to process the data and perform controls of the process. Most instruments however do not use adaptive real-time controls for performing detection. If it is done, the reaction times are slow because of communication loops, data volumes, and other tasks.
The disadvantage of the current instrumentation is that they do not have capability to rapidly adapt (in real-time) to changes that are sensed, and to perform closed loop control based on this information. They therefore have non-optimal performance, notably limited operating range, non-optimum reaction times, and performance which degrades with time. High throughput, micron-scale dimensions pose alignment problems. As the density and amount of samples increase, dimensions decrease (e.g. micron-scale biochips), and cost reduction is demanded, the existing designs are not well suited.
Systems based on individual sensors or small arrays such as Photodiodes, Avalanche Photodiodes and PMTs have the inherent disadvantage of lower throughput when compared to large array sensors which afford more parallelism.
Scientific CCD Devices and cameras: Used for imaging, microplates and biochips. The major advantages of these devices are low readout noise and low dark current when cooled (enabling long exposure times). Scientific devices allow “on-chip pixel binning”, which enables virtually noiseless summation. “Back-thinned” or “Back-illuminated” devices with high quantum efficiency are available. The main disadvantages are high cost, slow speed serial readout (no random pixel access). The charge binning capability has been used in biotechnology to achieve higher signal levels at low noise, to decrease data rate output of the sensor, to provide programmable detection of wavelength bands (in spectroscopic applications). Multiple reading of the charge packet from a pixel is a technique used in so-called “skipper CCD's” in the astronomy field to reduce read noise.
Scientific CID Devices and cameras: Used in scientific imaging and spectroscopy applications. The major advantage of these devices is the ability to perform non-destructive pixel reading, and random access to pixels, reportedly allowing dynamic range up to ˜109. The main disadvantages are high cost, slow speed of the random accessing. This feature of CID devices has been used in biotechnology to achieve high dynamic range, limited however in speed.
Video rate CCD Devices and cameras: These devices and cameras are commonly used for machine vision applications. Most are designed for video standards, and are not suitable for analytical measurements. Progressive scan devices are most suitable for measurement applications, and are commonly used in imaging applications such as microscopy, particularly when cooled. General advantages are fast speed, electronic shuttering, high resolution, low cost. Disadvantages are high noise, low dynamic range, limited or no pixel binning, higher defect rates.
CMOS Image Sensors: Current devices are targeting consumer/commercial imaging, and have integrated logic functionality and architecture which restrict the control of the sensor. Disadvantages are high noise, low dynamic range, fixed readout timing, higher defect rates. However, their positive features are low cost, high integration, and improving performance as the technology develops. CMOS sensors can also provide similar advantages as CID devices, namely non-destructive pixel reading, and random access to pixels, allowing dynamic range up to ˜109. Chemistry has been performed directly on the surface of a CMOS sensor array, thereby using the device as a disposable.
Intelligent Cameras: In the machine vision field, there exist cameras with integrated data processors. These commonly are video cameras and are not suitable for analytical needs of biotechnology. Typically the data processing functions and possibilities for adaptive real-time control of the sensor are fixed or limited.
Current state-of-the-art cameras and detection systems are generally limited by the following disadvantages:
The object of the present invention is to provide an image sensor device, an apparatus and a method for optical measurements, in particular for optimizing optical measurements through real-time closed-loop control. The object of the present invention is to solve one or several of the above problems.
This object is solved with the features of the claims.
The advantages of this invention over previous approaches are as follows:
“Sample-Based Detection”
Internet-Ready Appliances and Instrumentation
The present invention can be used in a number of target markets including, but not limited to:
The invention will now be described with reference to preferred embodiments and the drawings, in which:
FIG. 1 is a diagram of a system according to this invention, in particular illustrating how the apparatus is positioned and used in a typical micro-format application,
FIG. 2 illustrates a first preferred embodiment of an apparatus according to this invention for CCD image sensors,
FIG. 3 illustrates a second preferred embodiment of an apparatus according to this invention for CMOS, CID and PAF image sensors,
FIG. 4 shows a diagram of the general architecture of the preferred embodiment of the PAF image sensor, and
FIG. 5 shows a preferred embodiment of a cooling package which can be used for cooling an apparatus shown in any of FIGS. 2 to 4.
The present invention provides an intelligent detector:
The present invention particularly provides an apparatus comprising of electro-optics, electronics, firmware and software and processes which enables higher levels of optimization for optical measurements. Achieving higher performance and reliability is enabled for applications such as Image Processing, Spectroscopy, Microscopy, Chemical and biochemical process controls. The said apparatus is especially suitable for assays, processes and reactions in miniaturized formats with dimensions in the micron scale and sample volumes in sub-nanoliter scale. Furthermore, the apparatus has a high level of integration, compactness and Internet-capability, and can operate independent of host computers (PCs).
The said apparatus allows fast closed-loop digital control of image sensor(s) and any or all chemical, mechanical, opto-mechanical and opto-electronic components and processes which may affect the signals to be optimized. This is achieved by programmable high-speed processing (for example using real-time embedded microcontroller, hardware signal processing logic and/or DSP systems) employed directly at the image sensor. Sensor output data is immediately processed and evaluated. Furthermore, the processor has direct control of image sensor parameters (such as integration time, pixel binning, readout pattern, etc) via direct interface (bus or I/O) to digital logic which drives the sensor, as well as direct I/O control of any external parameters (dashed lines in FIG. 1).
When said apparatus is used with available CCD, CMOS and CID image sensors, performance can be significantly improved, thereby enabling applications that were not previously possible. Furthermore, an improved sensor architecture (hereto referred to as “PAF Image Sensor”), which enables fast frame rates, low noise, and very high intra-array dynamic range is a part of this invention. The programmability of said apparatus provides the platform for the development of application-specific control and data-processing algorithms (intellectual property), for example for biotechnology assays.
A diagram of a system according to this invention, in particular how the apparatus is positioned and used in a typical micro-format application is illustrated in FIG. 1.
Referring to FIG. 1, the system comprises a host computer and/or network and/or local display 1, which is connected via a bus to an apparatus 2, which comprises an intelligent detection and control system 20 and an image sensor 30. The dashed lines and arrows indicate capability of high speed real-time adaptive control of other parts of the system, which will be described hereunder. The image sensor(s) 30 is also under the control of said apparatus 2. A target object 3, here portrayed as a Biochip or micro-array, typically has features in the micron range, and may be transported into and out of the system manually or by automated (robotic) transport 6. Said target object may be illuminated and/or excited by optical sources 11, which may be programmable, optionally using programmable spatial light modulation means 12, via an optical system 5, which comprises optics and/or electro-optics. The system further may include means for micropositioning 4 the apparatus with respect to the target object 3. Additionally an interface 7 for fluidic, electrical and/or mechanical interaction may be connected to a system 8 comprising sensors and actuators for process controls, e.g. temperature, pH, voltages and/or currents. The system further may comprise process means 9 comprising chemical process control actuators and a supply means 10 for reagents. Detection (imaging) is performed by said apparatus 2 via said optical system 5. Based upon the data previously acquired, said apparatus 2 can perform real-time adaptive control of all other subsystems which affect the measurement, with the goal of optimizing the result within the time frame required. Said apparatus performs data processing directly at the sensor. Some parameters that can be modified might be (and is not limited to): Mechanical alignment; Focus; Exposure time; Illumination; Voltages and currents; temperature (8); Flow rates of reagents (9); Sensor parameters such as pixel binning, image sensor readout pattern, noise optimization.
The apparatus 2 has the capability of autonomously performing the detection/control task using any desired optimizing method (algorithm), provided that constraints required by the application are met. Said algorithms are fully programmable and can be defined and changed by the host 1 at any time, and are executed at high speed. Within the “measurement time” required by the host, said apparatus is able optimize the system to deliver the highest quality data.
The first preferred embodiment of said apparatus 2 for CCD image sensors 30 is illustrated by FIG. 2. A real-time micro-controller embedded system provides flexibility, programmability, and easy host interface. It handles communications protocols with the host using industry standards via a host communication interface 22, multi-tasking, operation sequences over longer time frames, and slower controls via a low speed control interface 23. This includes Internet protocols and the full implementation of a web server in the said apparatus. Said micro-controller 21 acts as host (master) to optional digital signal processor(s) (DSP) 24 and to optional hardware signal processing (typically implemented in FPGA or ASIC technology) 25. The apparatus enables distributed data processing in three optional stages. Firstly, the said hardware data processing is the fastest, allowing in-line data processing with algorithm times on the order of 1-100 ns. Secondly, the said DSP performs data processing, with high speed real-time feedback control of the CCD image sensor 30, as well as other external actuators and sensors which affect performance via a high-speed I/O control interface 26. Such algorithms execute in the micro-second time frame, and can operate on significant amounts of data. Thirdly, the said micro-controller system 21 can perform data processing algorithms. One or more CCD Image Sensors 30 perform the optical acquisition function, whereby said apparatus accommodates multiple output devices. The operation of said CCD is driven by a bank of clock drivers 27, the number and organization of which is determined by the specific image sensor. Clock voltage levels 15 and optimization for fast or slow clock rates are programmable as is cooling control 40. The clock waveforms and readout mode of said CCD are generated by timing logic 25 (typically implemented in FPGA or ASIC technology). The readout mode is programmable. The output(s) of said CCD are amplified by a bank of signal processing chains 28, whereby gains can be programmed, and either low noise, high resolution or high speed modules can be used. A Data Handler 29 accepts multiple data streams from said CCD, and ensures high bandwidth interface to said DSP. Said DSP 24 and hardware signal processing 25 perform application-specific data processing, in-line data calibration/normalization/correction (also using pre-stored calibration data), and transmits the resulting high quality, minimized result to a host computer.
The second preferred embodiment of said apparatus for CMOS, CID and PAF image sensors is illustrated by FIG. 3. The description of the parts and their functions which are the same as in FIG. 2 is omitted here. Currently available CMOS and CID image sensors have a high level of integration, with pixel address decoder, driving circuits, readout timing generators, amplifier(s), A/D converters possibly on-chip. Since most are designed for specific modes of operation (e.g. video imaging), their architecture limit flexibility in controlling the operation of the device, which in turn limits the performance that can be achieved. The PAF Image Sensor is a part of this invention which overcomes these limitations. It allows the intelligent detector more control of the sensor, and integrates additional circuitry on-chip in order to achieve high dynamic range at high frame rates.
The image sensor 30 is interfaced to said micro-controller 21, via programmable logic 25 if necessary. Said image sensor delivers digital data, which passes through the high bandwidth DSP interface 29.
The present invention further provides an improved “Pixel-Almost-Full Image Sensor”:
An improved architecture for an image sensor which, when used in the said intelligent detector, enables achievement of high intra-array dynamic range at speeds which are enabling to micro-format biotechnology applications. Such speeds are not otherwise available.
Said PAF Image Sensor (Pixel-Almost-Full; PAF) integrates on-chip Circuitry and Logic for monitoring (via non-destructive reading) of all pixels, detection of pixels which are close to full-well, resetting of these individual pixels as necessary, and output of their location and values. To achieve higher frame rates, the pixel array is modularly segmented, such that each segment is monitored by its own Segment Control Logic and circuitry. On-chip integration of this function alleviates need for off-chip logic or DSP resources. By multiple reading of individual pixels within the exposure time, the apparent capacity of the pixel and hence dynamic range is increased.
In the preferred embodiment, the technology for implementation of said PAF image sensor is CMOS process, for the following reasons:
The following definitions are solely given for the purpose of a better understanding of the invention. However, the scope and meaning of the below terms shall not be restricted to these definitions.
1-39. (canceled)
40. An apparatus for optical measurements, real-time imaging, sensing, detection, or controlling, comprising:
a single or a plurality of image sensors;
digital logic means to drive said image sensors as well as a direct I/O control;
a processor mechanism, to derive a result from an image, use the result to determine optimal sensor parameters and feed back the optical sensor parameters to the single or plurality of image sensors in an autonomous and continuous fashion until the result is optimized, for performing high-speed closed-loop digital control of said image sensors, the processor mechanism performing high-speed real-time control of image sensor parameters via direct interface to said digital logic means; and
at least one signal optimization system, comprising at least one of a chemical, mechanical, opto-mechanical or opto-electronic component or process which affect any signals to be optimized, said digital logic means to drive at least one of said chemical, mechanical, opto-mechanical and opto-electronic components and processes;
wherein said processor mechanism is placed directly at said image sensor such that output data from said image sensor is immediately processed and evaluated.
41. The apparatus according to claim 40, wherein said image sensor parameters comprise at least one of integration time, pixel binning, sensor readout pattern and timing.
42. The apparatus according to claim 40, wherein said processor mechanism comprises firmware, software or software algorithms, which are re-programmable and changeable at any time by an external host computer.
43. The apparatus according to claim 40, wherein said processor mechanism comprises an embedded micro-controller system and software.
44. The apparatus according to claim 40, wherein said processor mechanism comprises an electronics and software system based on a single or a plurality of Digital Signal Processors (DSP).
45. The apparatus according to claim 40, wherein said processor mechanism comprises:
a Digital Signal Processor (DSP) for performing data processing and high-speed digital control functions, and
an embedded micro-controller system for performing communications and multi-tasking functions and being a host to said DSP.
46. The apparatus according to claim 40, wherein said image sensor comprises one or a plurality of image sensor devices.
47. The apparatus according to claim 40, wherein said image sensor comprises one or a plurality of Charge Coupled Devices (CCD).
48. The apparatus according to claim 40, wherein said image sensor comprises a Complimentary Metal Oxide Semiconductor (CMOS) image sensor.
49. The apparatus according to claim 40, wherein said image sensor comprises a Charge Injection Device (CID) image sensor.
50. The apparatus according to claim 46, wherein said processor mechanism additionally comprises hardware logic for high-speed signal processing, said hardware logic performing real-time in-line data correction, such that pixel values are processed with no decrease in pixel rate during data acquisition, and wherein functionality and algorithms implemented in said hardware logic are programmable and changeable at any time by an external host.
51. The apparatus according to claim 50, wherein said hardware logic performs real-time correction of variations in an optical response of individual pixels or of binned sub-areas of pixels of said image sensor devices, and wherein said hardware logic uses calibration data pre-stored in said apparatus to perform data processing algorithms.
52. The apparatus according to claim 46, wherein said apparatus comprises hardware and software necessary for Ethernet or Internet communications, or wherein said apparatus can be controlled and programmed, and can transfer data, over said Ethernet or Internet.
53. The apparatus according to claim 46, further comprising cooling means for cooling said image sensor devices, said cooling means comprise at least one package in which one or several image sensor devices are thermo-electrically cooled and sealed in a hermetic manner, said package configured to accommodate off-the-shelf non-cooled image sensors packaged in standard I.C. packages, and further allowing for the image sensor device to be removed or replaced.