Systems and methods for image data compression

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

The current application claims priority to U.S. Provisional Patent Application No. 61/776,751 entitled “System and Methods for Image Data Compression,” filed Mar. 11, 2013, the disclosure of which is incorporated herein by reference.

FIELD OF THE INVENTION

The present invention is generally related to array cameras and more specifically to data compression of image data captured by focal planes within an array camera for bandwidth reduction.

BACKGROUND OF THE INVENTION

In a typical camera, light enters through an opening (aperture) at one end of the camera and is directed to a focal plane by a lens stack. The lens stack creates an optical channel that forms an image of a scene upon the focal plane. The focal plane includes an array of light sensitive pixels, which are part of a sensor that generates signals upon receiving light via the optical channel. Commonly used sensors include CCD (charge-coupled device) sensors and CMOS (complementary metal-oxide-semiconductor) sensors.

Traditional cameras typically use a single focal plane to capture single images, one at a time. The image data from each pixel of the focal plane is then sent directly from the focal plane to a processor. The processor can manipulate the image data, such as to encode or modify the image data.

SUMMARY OF THE INVENTION

Systems and methods for image data compression in an array camera are disclosed. In accordance with embodiments of the invention, an array camera includes a processor, a camera module, and a compression module. The camera module includes an imager array and an optic array of lens stacks. Each focal plane of the imager array includes rows of pixels that also form columns of pixels and is contained within a region of the imager array that does not contain pixels from another focal plane. An image is formed on each active focal plane by a separate lens stack in said optic array of lens stacks. A compression module is configured to receive image data from the array of pixels of a focal plane in a sequentially linear group of pixel data, analyze the received image data based upon truncation rules, compress the received data image based upon the analysis to generate compressed image data, and generate a bit mask identifying truncated image data in the compressed image data.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A conceptually illustrates an array camera in accordance with an embodiment of the invention.

FIG. 1B conceptually illustrates a focal plane from an imager array in accordance with an embodiment of the invention.

FIG. 2 conceptually illustrates an optic array and an imager array in an array camera module in accordance with an embodiment of the invention.

FIG. 3 is an architecture diagram of an imager array in accordance with an embodiment of the invention.

FIG. 4 illustrates a process for transmitting compressed image data in accordance with an embodiment of the invention.

FIG. 5 illustrates a process for decompressing image data compressed in accordance with an embodiment of the invention.

FIG. 6 illustrates image data transmitted as packets in accordance with an embodiment of the invention.

FIG. 7 illustrates transmission of image data as packets compatible with the MIPI CSI-2 standard interface format in accordance with an embodiment of the invention.

FIGS. 8a, 8b, and 8c conceptually illustrate lines of image data transmitted by interleaving image data from multiple focal planes using modulo processes in accordance with an embodiment of the invention.

DETAILED DESCRIPTION

Turning now to the drawings, systems and methods for compressing image data captured by individual pixels from focal planes of an imager array in an array camera in accordance with embodiments of the invention are illustrated. Array cameras are discussed in U.S. patent application Ser. No. 13/106,797 entitled “Architectures for Imager Arrays and Array Cameras” and U.S. patent application Ser. No. 12/952,106 entitled “Capturing and Processing of Images Using Monolithic Camera Array with Heterogeneous Imagers” the disclosures of both applications are hereby incorporated by reference in their entirety. In several embodiments, an array camera includes a camera module constructed using an imager array and a lens stack array, which combine to form a plurality of cameras. The cameras capture image data, which can be provided via an interface to a processor that can perform processes including synthesizing high resolution images from the individual images captured by the cameras in the camera module using super-resolution processing. Super-resolution processing is discussed in U.S. patent application Ser. No. 12/967,807 entitled “Systems and Methods for Synthesizing High Resolution Images Using Super-Resolution Processes”, filed Dec. 14, 2010, the disclosure of which is hereby incorporated by reference in its entirety.

In several embodiments, compressed image data is truncated image data along with at least one associated bitmask. In certain embodiments, image data is truncated by removing redundant or unwanted image data generated by one or more focal planes while a bitmask is generated that notes the truncation for reference in conjunction with the truncated image data. A processor using a decompression process can utilize a bit mask and compressed image data to reconstruct image data captured by the focal planes of an array camera. In many embodiments, the image data is truncated using a compression module to identify redundant image data. While much of the discussion that follows relates to truncation of image data captured by a single focal plane, similar processes can be utilized to truncate image data captured by multiple focal planes (e.g. image data from different focal planes that sample similar portions of the object space of a scene). Accordingly, truncation processes in accordance with embodiments of the invention should be understood as not limited to application image data captured by a single focal plan.

In a number of embodiments, digital logic is utilized to analyze image data in a compression module to determine whether the image data can be compressed by truncating redundant image data. The digital logic can be configured in accordance with any criteria for data compression, including (but not limited to) whether a pixel value is redundant with respect to neighboring pixel values or simply to omit a certain amount of image data to comply with certain bandwidth constraints. In several embodiments, redundancy can be determined based upon adjacent pixel values having the same intensity value, intensity values within a predetermined threshold distance, and/or intensity values within a threshold based upon the noise characteristics of the sensor. Each pixel value can be analyzed to determine if the image data should be transmitted or omitted. The decision can be stored in a digital memory element as either a 1 or a 0 as part of a bitmask. After the image data is truncated, the truncated image data along with the bitmask can be sent to a processor. The processor can perform a decompression process by utilizing the bitmask and truncated image data to generate an image representative of the scene captured by the focal plane from which truncated image data is generated. In many embodiments, the decompression process can configure the processor to use a bitmask to decompress the truncated image data to produce a complete set of image data corresponding to every pixel location within a focal plane.

Systems and methods for compressing image data captured by pixels within a focal plane of an imager array in accordance with embodiments of the invention are discussed further below.

System Architecture

Array cameras in accordance with many embodiments of the invention can include a camera module, compression module and a processor. The camera module can include an array of cameras. A camera module can also include an imager array, which is a sensor that includes an array of focal planes. Each focal plane includes an array of pixels used to capture an image formed on the focal plane by a lens stack. The focal plane can be formed of, but is not limited to, traditional CIS (CMOS Image Sensor), CCD (charge-coupled device), high dynamic range sensor elements, multispectral sensor elements and various alternatives thereof. In many embodiments, the pixels of each focal plane have similar physical properties and receive light through the same lens stack. Furthermore, the pixels in each focal plane may be associated with the same color filter. In a number of embodiments, at least one of the focal planes includes a Bayer-pattern filter. In several embodiments, the focal planes are independently controlled. In other embodiments, the operation of the focal planes in the imager array is controlled via a single set of controls.

An array camera in accordance with an embodiment of the invention is illustrated in FIG. 1A. The array camera 100 includes a camera module 102 that is configured to transmit image data to a processor 108. The processor 108 is connected to a memory 110. The camera module 102 includes an array of cameras 104. The cameras 104 in the camera module 102 are formed from the combination of a lens stack and a focal plane. The camera module 102 can include an optic array of lens stacks and an imager array of focal planes. These multiple cameras 104 may be active or inactive at any given time. Array cameras are discussed in U.S. patent application Ser. No. 13/106,797 entitled “Architectures for Imager Arrays and Array cameras” and U.S. patent application Ser. No. 12/952,106 entitled “Capturing and Processing of Images using Monolithic Camera Array with Heterogeneous Imagers”, the disclosures of both applications are hereby incorporated by reference in their entirety. The image data captured by these multiple cameras is sent from the focal planes of each camera to a processor via a compression module 106 resident in the camera module. In certain embodiments, the compression module is integrated within the imager array. The compression module 106 compresses the image data by truncating the image data and generating a bitmask describing the truncation of the image data, thereby compressing image data sent from each focal plane to the processor 108.

A single focal plane of the array camera of FIG. 1A in accordance with an embodiment of the invention is illustrated in FIG. 1B. The focal plane 150 is composed of a square array of pixels 152. Each pixel of the focal plane can generate image data as a pixel value based upon the intensity of light incident upon the pixel and the exposure and analog gain settings of the pixel. That image data can be compressed using a compression module that truncates redundant image data.

In many embodiments, the array camera captures images using a plurality of cameras, which can have different imaging characteristics. The array camera can separately control each of the cameras to obtain enhanced image capture and/or to enhance processes such as (but not limited to) super-resolution processes that may be applied to the captured images. For example, each pixel of a focal plane can capture different wavelengths of light, or can capture the intensity of light, varying exposure times, start times, or end times relative to each pixel of a different focal plane. Once the array camera has commenced capturing image data using the pixels on the imager array, the focal planes can commence transmitting the image data captured using the pixels. The image data can be compressed using a compression module and the compressed image data transmitted to the processor.

Although specific array camera system architectures are discussed above, any of a variety of system architectures for array cameras can be utilized as appropriate to the requirements of a specific application in accordance with embodiments of the invention. Camera modules in accordance with embodiments of the invention are discussed further below.

Camera Modules

Camera modules in accordance with embodiments of the invention can be constructed from an imager array and an optic array. A camera module in accordance with an embodiment of the invention is illustrated in FIG. 2. The camera module 200 includes an imager array 230 including an array of focal planes 240 along with a corresponding optic array 210 including an array of lens stacks 220. Within the array of lens stacks, each lens stack 220 creates an optical channel that forms an image of the scene on an array of light sensitive pixels within a corresponding focal plane 240. Each pairing of a lens stack 220 and focal plane 240 forms a single camera 104 within the camera module. Each pixel within a focal plane 240 of a camera 104 generates image data that can be sent from the camera 104 to the processor 108. In many embodiments, the lens stack within each optical channel is configured so that pixels of each focal plane 240 sample the same object space or region within the scene. In several embodiments, the lens stacks are configured so that the pixels that sample the same object space do so with sub-pixel offsets to provide sampling diversity that can be utilized to recover increased resolution through the use of super-resolution processes.

In several embodiments, color filters in individual cameras can be used to pattern the camera module with π filter groups as further discussed in U.S. Provisional Patent Application No. 61/641,165 entitled “Camera Modules Patterned with pi Filter Groups” filed May 1, 2012, the disclosure of which is incorporated by reference herein in its entirety. These cameras can be used to capture data with respect to different colors, or a specific portion of the spectrum. In contrast to applying color filters to the pixels of the camera, color filters in many embodiments of the invention are included in the lens stack. For example, a green color camera can include a lens stack with a green light filter that allows green light to pass through the optical channel. In many embodiments, the pixels in each focal plane are the same and the light information captured by the pixels is differentiated by the color filters in the corresponding lens stack for each filter plane. Although a specific construction of a camera module with an optic array including color filters in the lens stacks is described above, camera modules including π filter groups can be implemented in a variety of ways including (but not limited to) by applying color filters to the pixels of the focal planes of the camera module similar to the manner in which color filters are applied to the pixels of a conventional color camera. In several embodiments, at least one of the cameras in the camera module can include uniform color filters applied to the pixels in its focal plane. In many embodiments, a Bayer filter pattern is applied to the pixels of one of the cameras in a camera module. In a number of embodiments, camera modules are constructed in which color filters are utilized in both the lens stacks and on the pixels of the imager array.

Although specific array camera modules are described above, any of a variety of techniques can be utilized to construct array cameras in accordance with embodiments of the invention. Imager arrays that can be utilized in the construction of monolithic array camera modules in accordance with embodiments of the invention are discussed further below.

Imager Arrays

An imager array in which the image capture settings of a plurality of focal planes can be independently configured in accordance with an embodiment of the invention is illustrated in FIG. 3. The imager array 300 includes a focal plane array core 302 that includes an array of focal planes 304 and all analog signal processing, pixel level control logic, signaling, and analog-to-digital conversion (ADC) circuitry. The imager array also includes focal plane timing and control circuitry 306 that is responsible for controlling the capture of image data using the pixels. In a number of embodiments, the focal plane timing and control circuitry utilizes reset and read-out signals to control the integration time of the pixels. In other embodiments, any of a variety of techniques can be utilized to control integration time of pixels and/or to capture image data using pixels. In many embodiments, the focal plane timing and control circuitry 306 provides flexibility of image data capture control, which enables features including (but not limited to) high dynamic range imaging, high speed video, and electronic image stabilization. In various embodiments, the imager array includes power management and bias generation circuitry 308. The power management and bias generation circuitry 308 provides current and voltage references to analog circuitry such as the reference voltages against which an ADC would measure the signal to be converted against. In many embodiments, the power management and bias circuitry also includes logic that turns off the current/voltage references to certain circuits when they are not in use for power saving reasons. In several embodiments, the imager array includes dark current and fixed pattern noise (FPN) correction circuitry 310 that increases the consistency of the black level of the image data captured by the imager array and can reduce the appearance of row temporal noise and column fixed pattern noise. In several embodiments, each focal plane includes reference pixels for the purpose of calibrating the dark current and FPN of the focal plane and the control circuitry can keep the reference pixels active when the rest of the pixels of the focal plane are powered down in order to increase the speed with which the imager array can be powered up by reducing the need for calibration of dark current and FPN.

In many embodiments, a single self-contained chip imager includes focal plane framing circuitry 312 that packages the data captured from the focal planes into a container file and can prepare the captured image data for transmission. In several embodiments, the focal plane framing circuitry includes information identifying the focal plane and/or group of pixels from which the captured image data originated. In a number of embodiments, the imager array also includes an interface for transmission of captured image data to external devices. Image data provided to the interface can be directed through a compression module 314 that compresses the image data using any of a variety of compression processes discussed in more detail below. In the illustrated embodiment, the interface is a MIPI CSI 2 output interface (as specified by the non-profit MIPI Alliance, Inc.) supporting four lanes that can support read-out of video at 30 fps from the imager array and incorporating data output interface circuitry 316, interface control circuitry 318. Typically, the bandwidth of each lane is optimized for the total number of pixels in the imager array and the desired frame rate. The use of various interfaces including the MIPI CSI 2 interface to transmit image data captured by an array of imagers within an imager array to an external device in accordance with embodiments of the invention is described in U.S. Pat. No. 8,305,456, entitled “Systems and Methods for Transmitting Array Camera Data”, issued Nov. 6, 2012, the disclosure of which is incorporated by reference herein in its entirety.

Although specific components of an imager array architecture are discussed above with respect to FIG. 3, any of a variety of imager arrays can be constructed in accordance with embodiments of the invention that enable the capture of images of a scene at a plurality of focal planes in accordance with embodiments of the invention. The compression and transmission of image data captured by a plurality of focal planes in an imager array and a bitmask that enables a processor to construct reconstruct the image data from the compressed image data in accordance with embodiments of the invention is discussed further below.

Image Data Compression

Image data in accordance with many embodiments of the invention can be compressed utilizing a compression module that truncates image data from a row of pixels. The image data can be truncated according to digital logic resident in the compression module that detects for redundancies among image data read out from pixels within rows (or columns) of pixels, such as but not limited to truncating redundant pixel values from neighboring pixels along a row (or column) of pixels such that less than all of the redundant values from neighboring pixels are transmitted from a focal plane to a processor. A bitmap can also be generated by the compression module and sent to the processor that receives the compressed image data such that the processor using a decompression process can reconstruct the uncompressed image data.

A truncation rule can be set in accordance with any constraint to determine whether to truncate or transmit pixel values. In certain embodiments, a truncation rule can require that a certain amount of image data must be omitted to achieve a particular output bandwidth constraint. A particular embodiment can dictate that at most image data from 75% of the pixels of a focal plane can be transmitted to achieve a bandwidth usage target. In other embodiments, any of a variety of criteria can be utilized to control the compression process to achieve a target bandwidth.

In many embodiments, image data can be compressed by truncating redundant or unwanted image data such that only image data from pixels that contain important information is transmitted to a processor. Compression modules can store image data as pixel values in a shift register from pixels across a row of pixels temporarily, allowing for digital logic to quickly analyze image data relative to neighboring pixels to determine whether pixel values from certain pixels should be truncated. The shift register can be implemented as a cascade of flip flops that share the same clock signal. Shifting at a transition of the clock signal causes the image data stored in the shift register to shift by one position, enabling the compression module to load in and shift out pixel values one pixel value at a time. This also enables simultaneous access to pixel intensity values from adjacent pixels. In many embodiments, intensity values from adjacent pixels from a row or column of pixels are analyzed. In many embodiments, a compression module with a shift register 3 words deep can store pixel values from three different pixels, thereby providing access to values from pixels in the present clock cycle's pixel, N, as well as pixels from the previous two consecutive clock cycles N₋₁ and N₋₂. Digital logic resident in the compression module can then analyze the pixel value stored at the clock cycle N₋₁ for comparison to pixel value stored at clock cycle N and N₋₂.

A number of redundancy criteria can be applied by the digital logic so as to determine if the pixel value at clock cycle N₋₁ is worth transmitting. In many embodiments, a truncation rule can be utilized to determine a threshold of redundancy beyond which pixel values are truncated. In certain embodiments, the truncation rule can utilize the following relationship:
K=(2*N₋₁)−N−N₋₂
Where only when K>threshold, is the pixel value at clock cycle N₋₁ transmitted. This configurable threshold could be set to, for example, the noise floor of the focal plane or alternatively be additionally made a function of the intensity value of pixel N₋₁ such that both the noise floor and photon shot noise can be used to set the threshold. If a particular pixel has K value less than the threshold, the image data from the pixel can be considered to be of no significance as being indistinguishable from noise. In other embodiments, any of a variety of criterion can be utilized to determine whether to truncate a pixel based upon the intensity values of one or more adjacent pixels in a row or column of pixels read out from a focal plane.

In a number of embodiments, a compression module can employ digital logic to determine whether compression of image data by truncation is necessary by determining if there is a compression advantage. In numerous embodiments, a compression advantage determines if transmitting compressed image data is advantageous relative to transmitting non compressed image data. This type of compression advantage can determine whether the combination of the truncated image data and the bitmask would require less bandwidth to transmit than merely uncompressed image data. In certain embodiments, a compression advantage is present when the number of pixel values truncated multiplied by the number of bits required to store a pixel value is larger than the number of bits in the bitmask.

In several embodiments, a bitmask can be created to record truncation of image data to be used by a processor using a decompression process to generate an image representative of the scene captured by the focal plane from which truncated image data is generated. In certain embodiments, a bitmask represents whether image data as pixel values from particular pixels is truncated as either a 1 or a 0 associated with the particular pixel in the bitmask. Thereby, in many embodiments, image data from a row of 1000 pixels would require memory of 1000 bits in size to store information in a bitmask. The location of bits within the bitmask indicates the location of each pixel in which image is truncated.

A focal plane can utilize one or multiple bitmasks in transmitting compressed image data. A bitmask can be inserted into a stream of truncated image data sent to a processor periodically, such as but not limited to being inserted with each row of truncated image data, or once upon successful transmission of image data associated with an entire focal plane.

A process for image data compression in accordance with an embodiment of the invention is illustrated in FIG. 4. The compression process 400 includes receiving (402) image data as pixel values from pixels in a row of pixels of a focal plane at a compression module. The image data can be analyzed (404) by digital logic resident on the compression module for compression. The analysis can be made in accordance with a truncation rule, such as (but not limited) to the various truncation rules described above. The compression can be achieved by truncating (406) image data from a row of pixels in accordance with the analysis. A bit mask can be generated (408) that notates the truncation of image data. The compressed image data, as truncated image data and a bitmask, can be sent (410) to a processor for reconstruction of a scene captured by the focal plane using a decompression process. In many embodiments, the decompression process can configure the processor to use the truncated image data to generate pixel values associated with pixel locations (as represented within image data used to generate an image) found using the bitmask to be missing an associated pixel value due to truncation.

A decompression process in accordance with an embodiment of the invention is illustrated in FIG. 5. The decompression process 500 includes receiving (502) compressed image data from a row (or column of pixels) and extracting a bit mask that can be used to decompress (504) truncated image data within the compressed image data based upon a known truncation rule, such as (but not limited to) the truncation rules described above. In several embodiments, the process (optionally) involves encoding the decompressed image data using an image encoding format such as (but not limited to) the lossless JPEG image encoding formats specified by the Joint Photographic Experts Group and/or the image compression format specified in U.S. Provisional Patent Application Ser. No. 61/767,520 entitled “Systems and Methods for Generating Captured Light Field Image Data Using Captured Light Fields”, filed Feb. 21, 2013, the disclosure of which is incorporated by reference herein in its entirety. The (encoded) image data can then be stored for use in additional processing.

Although specific embodiments of array camera image data compression are discussed above, many other implementations of image data compression are possible in accordance with various embodiments of the invention. While the above description contains many specific embodiments of the invention, these should not be construed as limitations on the scope of the invention, but rather as an example of one embodiment thereof.

Packetizing Image Data Captured by Multiple Focal Planes

In several embodiments, image data from a plurality of focal planes can be packetized by inserting the image data and/or additional data describing the image data into a packet in such a way that a processor can reconstruct images of a scene from the received image data. A conceptual illustration of a packet including image data and additional data describing the image data transmitted by an imager array in accordance with an embodiment of the invention is illustrated in FIG. 6. The packet 602 includes a packet header 604 and a packet footer 606. The packet 602 also includes a number of lines 608 of image data, where each line 608 of image data includes a line header 610 and a line footer 612. In many embodiments, the packet header 604 and/or the line header 610 contain additional data that describes the image data in such a way that a device receiving the packet can reconstruct a plurality of images using image data including the lines of image data contained within the packet. The number of lines of image data included within the packet typically depends upon the requirements of a specific application. In many embodiments, the packet can contain all of the lines of image data for a single captured light field. The term light field can be used to describe a number of two dimensional images that are captured of a scene from different perspectives. In other embodiments, the lines of image data of a captured light field can be divided and sent in multiple packets. In many embodiments, image data captured by individual pixels or groups of pixels from within a line of pixels from different focal planes can be interleaved within a packet of image data.

In a number of embodiments, the packet header 604 contains embedded data. In many embodiments, the embedded data describes the camera module from which image data is generated in such a way that a processor can determine the structure of the image data in the packet and reconstruct images from the data received from the camera module. In several embodiments, a packet header 604 includes embedded data such as (but not limited to) the number of focal planes in an imager array, the timing of the image capture per focal plane, the identity of the particular focal planes being read out, the total number of pixels in a focal plane, the resolution of an image taken by a focal plane, the timing of the pixel read outs and the gain for the focal plane. As discussed below, the embedded data described above need not be included in a packet header and some or all of the information can be transmitted accompanying image data in different ways including but not limited to locating the additional data elsewhere in the packet and/or transmitting the additional data in a separate packet. Embedded data describing imaging data in accordance with embodiments of the invention is discussed further below.

In the illustrated embodiment, the lines 608 of image data include line headers 610. The line header identifies the focal plane or focal planes and pixels in the imager array that captured the image data contained within the line of image data. A processor can utilize the line header to identify the specific image data contained within the line 610 of image data. In various embodiments, a line header 610 includes information such as (but not limited to) the identify of the focal plane that captured the image data within the line and/or the identity of the specific pixels(s) or group of pixels used to capture the image data contained within the line of data, and a timestamp. Stated another way, a line of image data within a packet formatted in accordance with embodiments of the invention need not correspond to image data captured using a single line of pixels in a single focal plane. Indeed, packets of image data in accordance with embodiments of the invention can include lines of image data containing image data captured by different lines of pixels and/or from different focal planes. Inclusion of the additional data describing the line of image data in the line header allows a processor to receive and process image data from multiple images multiplexed into a single packet or stream of packets. Different types of embedded data that can be included in line headers (or elsewhere) in accordance with embodiments of the invention are discussed further below.

Each line 608 of image data can include a line footer 612 to indicate that the line of image data 608 associated with the preceding line header 610 has ended. Also, each packet 602 can include a packet footer 606 to indicate that the image data associated with the previous packet header 604 has ended. In many embodiments, the imager array is configured to generate multiple packets 602 to contain the image data captured by the focal planes and each packet includes multiple lines of image data.

Due to the manner in which image data is captured by different sets of pixels in different focal planes as data is transmitted by the imager array, the processor typically cannot predict the order in which it will receive image data from the imager array. In many embodiments, the processor has no knowledge of the focal plane or focal planes that captured the image data contained within a line of image data without reference to the packet header and/or the line header for the line of image data. However, in other embodiments the imager array imposes constraints with respect to the order in which image data is captured by specific focal planes (see for example the discussion below with respect to FIGS. 8a-8c) and a processor can rely upon the predetermined order of image data capture to reconstruct the image data. While imposing constraints on the order in which image data is captured can reduce the flexibility of the image array with respect to the manner in which image data is captured from different focal planes, the predictable manner in which image data is received from the imager array can result in the reduction in the amount of additional data transmitted in conjunction with the image data by removing information that identifies the focal plane and/or pixels that captured the image data. In many embodiments, the manner in which the imager array is constrained to capture image data enables the packet header, the packet footer, the line header and/or the line footer illustrated in FIG. 6 to be eliminated.

Although the inclusion of specific pieces of information within packet headers and/or line headers is described above, any information that enables the reconstruction of multiple images from image data multiplexed into a single packet or stream of packets can be incorporated into a packet of image data in accordance with embodiments of the invention. Transmission of image data compatible with the MIPI interface format is discussed further below.

Image Data Transmission Compatible with the MIPI Interface Format

In several embodiments, imager arrays transmit image data and additional data describing the image data in a manner that is compatible with an existing interface format for the transmission of image data by a conventional camera including a single focal plane. A conceptual illustration of image data and additional data describing the image data transmitted as packets compatible with the MIPI CSI-2 standard interface format (MIPI interface format) in accordance with an embodiment of the invention is illustrated in FIG. 7. The conceptual illustration can be read as involving transmissions from left to right in the X direction 720 and from top to bottom in the Y direction 722. The transmission begins with a MIPI frame blanking interval 702. A MIPI frame start (MFS) 710 indicator is then sent by the imager array, followed by a portion of the MIPI header 712. A packet of data generated in accordance with embodiments of the invention is inserted within the standard MIPI container as embedded data. Accordingly, the first line of data within the MIPI container can include a packet header 724 containing information concerning the focal planes that generated the image data (see discussion above).

The transmission of the first line of the MIPI container is completed by the transmission of a MIPI footer 706. There is a pause during the MIPI line blanking interval 708, and then the next portion of the MIPI header 712 is transmitted. The next line of the MIPI container includes a line of image data 714. In embodiments where the order in which the lines of image data transmitted by the imager array is not predetermined, the line of image data can be preceded by a line header and followed by a line footer. In embodiments where the lines of image data are transmitted in a predetermined order (see for example the discussion of FIGS. 8a-8c), a line header and/or line footer may not be utilized.

The process of transmitting a MIPI footer, pausing during a MIPI line blanking interval, transmitting a portion of the MIPI header, and transmitting lines of image data within the MIPI container continues until all the lines of image data in the packet are transmitted. In several embodiments, an embedded packet footer is transmitted in the MIPI container to indicate that the transmission of the packet is complete. Following the transmission of the packet, the transmission of the MIPI container is completed by transmitting a MIPI footer 706 and a MIPI frame end 716. Although the packet illustrated in FIG. 7 involves transmitting one line of image data between line blanking intervals of the MIPI container, in many embodiments the packet header and the lines of image data do not correspond with the line blanking intervals of the MIPI container. Stated another way, a single line blanking interval of the MIPI container can contain image data from two or more lines of image data. Accordingly, the line headers and/or line footers are utilized to identify the individual lines of image data within the container.

As can readily be appreciated, the process illustrated in FIG. 7 involves formatting a packet of data including image data and additional data describing the image data generated in accordance with embodiments of the invention within a conventional MIPI container. In this way, an imager array can utilize an interface standard developed to enable the transmission of image data captured by a single focal plane to enable transmission of a packet of data containing image data captured by a plurality of focal planes (i.e. a light field). In other embodiments, similar processes can be utilized to transmit packets formatted in the manner outlined above using other containers and/or interface formats including (but not limited to) a CameraLink interface format, a USB interface format, or a Firewire interface format.