REFRIGERATOR WITH OPERATING PARAMETER ADJUSTMENT

Description

BACKGROUND

US 2014/0137587 A1 discloses a method for storing food items within a refrigerator having a chilled chamber that includes establishing an identity of each food item within the chilled chamber. A preferred storage condition of the chilled chamber is determined based upon the identities of the food items, and a current storage condition of the chilled chamber can be adjusted toward the preferred storage condition.

U.S. Pat. No. 11,852,404 B2 discloses a refrigeration appliance system including a camera that captures images of objects entering and exiting the interior space of a refrigeration appliance and processing the images to identify the objects in the image using a trained machine learning model. The system may also process the images to determine a volume or quantity within the object. Using this determined information, the system may then create, update, or alter a log of objects within the refrigeration appliance and change a function of the refrigeration appliance based on one or more items in the log of the contents.

Both of the aforementioned patent documents have limitations when it comes to determining the items or objects being loaded into or removed from the refrigerator. US 2014/0137587 A1 discloses cameras directed inwardly toward a fresh food chamber. Images taken by these cameras may not locate items that are blocked from the camera's view by other items in the fresh food chamber. U.S. Pat. No. 11,852,404 B2 discloses a camera that obtains visual images of objects entering or exiting the interior of the refrigeration appliance as they pass through the plane of the door opening while a door of the refrigeration appliance is open. Sometimes, especially when items are loaded into drawers that when open extend beyond the plane of the door opening, capturing images of the items in the drawers can be difficult or confused with other items in the drawer while the drawer is being closed.

SUMMARY

In view of the foregoing, a refrigerator includes a main body and at least one storage compartment. The main body defines a refrigerated enclosure and a main body opening leading to at least a portion of the refrigerated enclosure. The storage compartment is within the refrigerated enclosure and defines a compartment volume able to be closed off from at least one other portion of the refrigerated enclosure allowing a storage condition within the compartment volume to be controlled separately from other portions of the refrigerated enclosure. At least one camera mounts to the main body and is positioned to have a field of view forward of the main body opening. At least one computing device is in operable connection with the at least one camera and is configured to perform the following functions: (1) instruct the at least one camera to capture images of items being placed into or removed from the at least one storage compartment, (2) analyze the images captured by the at least one camera and determine whether the images contain an object of interest, (3) assign a direction of movement for the object of interest being at least one of into or out of the compartment volume, (4) update an inventory for the at least one storage compartment based on the object of interest and the direction of movement for the object of interest, and (5) provide instructions to at least one environmental control device in electrical communication with the at least one computing device. The instructions provided to the at least one environmental control device control the at least one environmental control device to change the storage condition within the compartment volume based on the inventory in the compartment volume.

A method of controlling a refrigerator includes instructing at least one camera to capture images forward of a main body opening of a main body of the refrigerator of items being placed into or removed from at least one storage compartment. The images are analyzed to determine whether the images contain an object of interest. The method further includes assigning a direction of movement for the object of interest being at least one of into or out of the at least one storage compartment. An inventory for the at least one storage compartment is updated based on the object of interest and the direction of movement for the object of interest. Instructions are then provided via at least one computing device to at least one environmental control device in electrical communication with the at least one computing device. The instructions control the at least one environmental control device to change a storage condition within the at least one storage compartment based on the inventory in the at least one storage compartment.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a perspective view of a refrigerator with a door shown in an open position.

FIG. 2 is a front view of the refrigerator with the door shown in an open position.

FIG. 3 is a block diagram of the refrigerator.

FIG. 4 is an example of an analyzed image containing an object of interest.

FIG. 5 is a perspective view of the refrigerator with the door and a drawer shown in an open position.

FIG. 6 is a perspective view of a similar refrigerator with a drawer in an open position.

DETAILED DESCRIPTION

Referring to FIG. 1, a refrigerator 102 includes a main body 104 and a door 106 pivotally mounted to and movable with respect to the main body 104. If desired, more than one door, e.g., similar to known French door style refrigerators, could be provided to selectively cover all or a portion of a main body opening 108 leading to a refrigerated enclosure 110 when the door 106 is closed against the main body 104. Shelves 112 can be positioned within the refrigerated enclosure 110 to facilitate organization of items to be loaded into the refrigerator 102.

The refrigerator 102 includes at least one storage compartment within the refrigerated enclosure 110 that is able to be closed off from other portions of the refrigerated enclosure 110 so that a storage condition within the respective storage compartment can be controlled separately from the remainder of the refrigerated enclosure 110. These storage conditions can include temperature, gas content, air flow, lighting, and humidity allowing the aforementioned storage conditions within the respective storage compartment to be different than the storage condition of the remainder of the refrigerated enclosure 110. This is useful, for example, when fresh vegetables are stored in the aforementioned storage compartment and the temperature, gas content, air flow, lighting, or humidity may be different in the aforementioned storage compartment as compared to the portion of the refrigerated enclosure 110 having the shelves 112 as an example.

In view of the foregoing, the refrigerator 102 includes least one storage compartment, e.g., a first storage compartment 114, within the refrigerated enclosure 110 that can define a compartment volume, e.g., a first compartment volume 116, able to be closed off from at least one other portion of the refrigerated enclosure 110. The storage condition, e.g., temperature, gas content, air flow, lighting, humidity, within the first compartment volume 116 can be controlled separately from at least one other portion of the refrigerated enclosure 110 in a manner that will be described in more detail below. Multiple similar type storage compartments can be provided in the refrigerated enclosure 110. For example, a second storage compartment 118 can define a second compartment volume 122 able to be closed off from at least one other portion of the refrigerated enclosure 110 allowing the storage condition within the second compartment volume 122 to be controlled separately from the at least one other portion of the refrigerated enclosure 110. Also, a third storage compartment 124 can define a third compartment volume 126 able to be closed off from at least one other portion of the refrigerated enclosure 110 allowing a storage condition within the third compartment volume 126 to be controlled separately from the at least one other portion of the refrigerated enclosure 110.

As illustrated in FIGS. 1 and 2, a plurality of cameras 150 (see FIG. 3) mount to the main body 104 and the door 106. Each of the plurality of cameras 150 are positioned to have a field of view forward of the main body opening 108 for capturing images of objects moving into and out of the refrigerated enclosure 110 when the door 106 is open. The plurality of cameras 150 includes a first set of cameras 160a, 160b mounted on the main body 104 associated with the first storage compartment 114, a second set of cameras 162a, 162b mounted on the main body 104 associated with the second storage compartment 118 and a third set of cameras 164a, 164b mounted on the main body 104 associated with the third storage compartment 124.

Additional cameras can be provided and associated with other portions of the refrigerated enclosure 110. For example, and as seen in FIG. 2, a fourth set of cameras 166a, 166b, 168a, 168b are mounted on the main body 104 and associated with the portion of the refrigerated enclosure 110 above the first storage compartment 114. Also, a fifth set of cameras 174a, 174b can mount to the door 106 and be associated with an upper door shelf 176 on the door 106, a sixth set of cameras 180a, 180b can mount to the door 106 and be associated with an middle door shelf 182 on the door 106 and a seventh set of cameras 184a, 184b can mount to the door 106 and be associated with a lower door shelf 186 on the door 106.

With reference to FIGS. 1 and 2, another example of a storage compartment within the refrigerated enclosure 110 that can define a fourth compartment volume 190 able to be closed off from at least one other portion of the refrigerated enclosure 110 is shown as the upper door shelf 176 on the door 106 where a lid 192 can selectively close of the fourth compartment volume 190. The lid 192, for example, can pivot about a horizontal axis with respect to the door 106 to selectively close of the fourth compartment volume 190 from at least one other portion of the refrigerated enclosure 110. Lids could also be provided elsewhere within the refrigerated enclosure 110 to selectively close off other compartment volumes within the refrigerated enclosure 110.

As shown between FIGS. 1 and 2, the cameras with a suffix “a” are positioned at a left side of the main body 104 or door 106 when the door 106 is open, and the cameras with a suffix “b” are positioned at a right side. Instead of having a field of view towards the interior of the refrigerator 102, each camera among the plurality of cameras 150 (see FIG. 3) has a field of view forward of the main body opening 108, i.e., directed away from the interior of the refrigerator 102. With this construction, the plurality of cameras 150 is configured to capture images of an item being loaded into the refrigerated enclosure 110 prior to the item passing through the main body opening 108.

With reference to FIG. 3, the refrigerator 102 includes at least computing device, e.g., a computing device 200, in operable connection with the plurality of cameras 150 via a bus 202. The computing device 200 is configured to instruct each of the cameras of the plurality of cameras 150 to capture images of items being placed into or removed from the refrigerated enclosure 110. The computing device 200 is also configured to analyze the images captured by the plurality of cameras 150 and determine whether the images contain an object of interest, which is an item or items within the respective images that is being placed into or removed from the refrigerator 102. The computing device 200 is further configured to assign a direction of movement for the object of interest being at least one of into or out of the refrigerated enclosure 110. The computing device 200 is also configured to update an inventory for the refrigerated enclosure 110 based on the object of interest and the direction of movement for the object of interest. In certain instances, the computing device 200 is configured to provide instructions to at least one environmental control device in electrical communication with the at least one computing device 200 based on the inventory in at least one of the first compartment volume 116, the second compartment volume 122, the third compartment volume 126 and the fourth compartment volume 190.

The computing device 200 providing instructions to at least one environmental control device based on the inventory in the first compartment volume 116 will be described in more detail with the understanding that the computing device 200 can provide instructions to at least one environmental control device based on the inventory in the second compartment volume 122, the third compartment volume 126 or the fourth compartment volume 190 in a similar manner. The computing device 200 instructs at least one camera, including but not limited to the first set of cameras 160a, 160b, which can capture images of items being placed into or removed from the first storage compartment 114. The computing device 200 then analyzes the images captured by the first set of cameras 160a, 160b, for example, and determines whether the images contain an object of interest. The computing device 200 assigns a direction of movement for the object of interest being at least one of into or out of the first compartment volume 116 and updates an inventory for the first storage compartment 114 based on the object of interest and the direction of movement for the object of interest. The computing device 200 can then provide instructions to at least one environmental control device, examples of which will be described in more detail below. The environmental control devices are in electrical communication with the computing device 200. The instructions provided to the at least one environmental control device control the at least one environmental control device to change the storage condition within the first compartment volume 116 based on the inventory in the first compartment volume 116.

With reference to FIG. 3, the refrigerator 102 includes a cooling system 214 that can be similar to those found in known refrigerators, and therefore will be depicted schematically. The cooling system 214 and its individual components can be in electrical communication with the computing device via the bus 202. The cooling system 214 can include temperature sensors 216 for sensing temperatures within the refrigerator 102 and air blower fans 218 for supplying cooled air to locations within the refrigerated enclosure 110, which can also include the first compartment volume 116, the second compartment volume 122 and the third compartment volume 126. Multiple air blower fans 218 can be provided. The cooling system 214 also a compressor 222 compressing refrigerant, a condenser 224 condensing the compressed refrigerant, an expansion valve 226 decompressing the condensed refrigerant, and an evaporator 228 evaporating the decompressed refrigerant. The cooling system 214 and each of its individual components can be in electrical communication with the computing device 200 via the bus 202. The cooling system 214 or individual components thereof can make up at least one of the aforementioned environmental control devices that can change the storage condition within the first compartment volume 116. For example, operation of air blower fans 218, or one such fan in fluid communication with the first compartment volume 116 via an air duct 232 (depicted schematically in FIG. 2), can be operated to control air flow into the first compartment volume 116.

With reference to FIG. 3, the refrigerator 102 can also include a damper control mechanism 240 as an example of at least one of the aforementioned environmental control devices that can change the storage condition within the first compartment volume 116. The damper control mechanism 240, which can be in the form of a motor or similar actuator, can control (e.g., open and close) a damper 242 within the air duct 232 (FIG. 2) leading to the first compartment volume 116. The damper control mechanism 240 is in electrical communication with the computing device 200 via the bus 202 and is operably connected with the damper 242.

With reference to FIG. 3, the refrigerator 102 can also include a filtration system 250 as an example of at least one of the aforementioned environmental control devices that can change the storage condition within the first compartment volume 116. For example, the filtration system 250, which can be in electrical communication with the computing device 200 via the bus 202, can be activated when particular food items are detected as being located in the first compartment volume 116. The filtration system 250 may include a filter 252 to remove gases (e.g., ethylene, carbon-dioxide, and methane) and odors caused by food that may be decomposing. As such, a gas sensor 254 may be part of the filtration system 250 to detect these gases. Moreover, the filtration system 250 may include a filter shield 256, which can cover a portion of the filter 252, and a filter shield control mechanism 258, which can move the filter shield 256 with respect to the filter 252 based on the inventory in the first compartment volume 116. The filtration system 250 may be located in any of the first compartment volume 116, the second compartment volume 122, the third compartment volume 126, and the fourth compartment volume 190 or simply in fluid communication with any of the first compartment volume 116, the second compartment volume 122, the third compartment volume 126 and the fourth compartment volume 190.

With reference to FIG. 3, the refrigerator 102 can also include a light source 270 as an example of at least one of the aforementioned environmental control devices that can change the storage condition within the first compartment volume 116. The light source 270, which can be in electrical communication with the computing device 200 via the bus 202, can be positioned with respect to the first compartment volume 116 to direct light into the first compartment volume 116, for example. Light from the light source 270 can assist with increasing the useful life of food items within the first compartment volume 116. The computing device 200 can adjust the wavelength or the intensity of light emitted from the light source 270 into the first compartment volume 116 based on the inventory in the first compartment volume 116.

With reference to FIG. 3, the refrigerator 102 can also include a gas source controller 280, which can be in fluid communication with a gas source 282 and the first compartment volume 116, as an example of at least one of the aforementioned environmental control devices that can change the storage condition within the first compartment volume 116. The gas source controller 280 is configured for controlling a flow of gas from the gas source 282 into the first compartment volume 116. The gas source 282 can include canisters containing nitrogen gas, oxygen gas, carbon dioxide gas, ozone gas, argon gas, or combinations thereof. The gas source 282 may be positioned at any suitable location within refrigerator 102. The gas source controller 280, which is in electrical communication with the computing device 200 via the bus 202, can include an electrically operated valve or pump to selectively control the delivery of gas from the gas source 282. The flow of gas from the gas source 282 into the first compartment volume 116 can assist with increasing the useful life of food items within the first compartment volume 116.

A sensor, such as the gas sensor 254 that is part of the filtration system 250 described above, a similar sensor that could be used in conjunction with the gas source controller 280 can be another example of at least one of the aforementioned environmental control devices that can change the storage condition within the first compartment volume 116. In such an instance, the sensor, which can sense for particular gases, as an example, is in electrical communication with the computing device 200 via the bus 202 and is positioned within the first compartment volume 116. The storage condition within the first compartment volume 116 can be changed by the computing device 200 providing instructions to the sensor to power down or off when no items are detected within the first compartment volume 116, which can save energy.

With reference to FIG. 3, the refrigerator 102 can also include a humidity regulator 290, which can be in fluid communication with the first compartment volume 116, as an example of at least one of the aforementioned environmental control devices that can change the storage condition within the first compartment volume 116. The humidity regulator 290, which can be in electrical communication with the computing device 200 via the bus 202, is configured for regulating and adjusting an amount of water vapor within the first compartment volume 116. The humidity regulator 290 can adjust a humidity of the first compartment volume 116 by regulating a flow of air between a volume of relatively high humidity air and a volume of relatively low humidity air. The humidity regulator 290 can include a valve positioned between such volumes and selectively adjustable to adjust a flow of air between the volumes. As an example, the refrigerated enclosure 110 outside of the first compartment volume 116 can have a relatively low humidity relative to the first compartment volume 116. The humidity regulator 290 can adjust the humidity within the first compartment volume 116 by selectively adjusting a flow of air between the refrigerated enclosure 110 outside of the first compartment volume 116 and the first compartment volume 116.

With reference to FIG. 3, the refrigerator 102 can also include an auxiliary heating/cooling system 294, which can be in fluid communication with the first compartment volume 116, as an example of at least one of the aforementioned environmental control devices that can change the storage condition within the first compartment volume 116. In an example, the auxiliary heating/cooling system 294 can be similar to the cooling system 214 described above and can include similar components, e.g., a compressor, a condenser, an expansion valve and an evaporator. Where the cooling system 214 described above is configured to cool the refrigerated enclosure 110, which is relatively larger than the first compartment volume 116, the auxiliary heating/cooling system 294 is configured to cool the first compartment volume 116 separately from the refrigerated enclosure 110. As another example, the auxiliary heating/cooling system 294 may include components of a cooling circuit while utilizing the compressor 222 of the cooling system 214. For example, the auxiliary heating/cooling system 294 may include one or more additional evaporators that are similar to the evaporator 228, and one or more valves may be provided in the cooling circuit to control the flow of coolant fluid through the one or more additional evaporators. The one or more valves can be electronically controlled, e.g., by being in electrical communication with the bus 202 and thus the computing device 200, to open or close the valves to control the flow of coolant fluid through the one or more additional evaporators and thus change the storage condition within the first compartment volume 116. The auxiliary heating/cooling system 294 could also be configured to heat the first compartment volume 116 separately from the refrigerated enclosure 110. As one example, the auxiliary heating/cooling system 294 could a small resistance heater located in or in fluid communication with the first compartment volume 116.

To accomplish the tasks described above, the computing device 200 can include a processor 302, a memory 304, and a communication interface 308. The communication interface 308 provides software and hardware to facilitate data input and output between the components of the computing device 200 and other components, networks, and data sources. The computing device 200 can also include a session controller module 312, a synchronizer module 314, an object detection module 318, an object tracking module 322, a direction module 324, an object merge module 326, and an inventory update module 328.

The session controller module 312 is configured to start a session and to instruct each camera in the plurality of cameras 150 to begin capturing images based on a start trigger signal, which can be generated by a trigger switch 334, received by the session controller module 312. The plurality of cameras 150 begin capturing images after the session controller module 312 receives the start trigger signal, and the plurality of cameras 150 cease capturing images after the session controller module 312 receives a stop trigger signal, which can also be generated by the trigger switch 334. Each camera in the plurality of cameras 150 can be configured to compress captured images, to assign a session identification (ID), to assign a camera identification (ID) and to assign a frame identification (ID) for each captured image and send the compressed images along with their respective session ID, camera ID and frame ID to the synchronizer module 314, which can occur during or after a session, which is the time between opening and closing of the door 106. A session can be relatively long and may include the transferring of multiple items into and out of the refrigerator 102.

The synchronizer module 314 is configured to receive captured images and the session ID, camera ID and frame ID associated with each captured image from the plurality of cameras 150, and to group the captured images into frame sets based on the frame ID associated with each captured image. Each frame set is made up of respective captured images that were captured at the same time from different cameras among the plurality of cameras 150. The synchronizer module 314 can group captured images from the plurality of cameras 150 based on the frame ID associated with each image. The same frame ID among different captured images indicates the images were taken at the same time albeit from different cameras. The synchronizer module 314 receives the session ID and the camera ID, in addition to the frame ID for each captured image received from each camera in the plurality of cameras 150. The session ID is associated with the session assigned by the session controller module 312, the camera ID is associated with a particular camera among the plurality of cameras 150 that captured the captured image, and the frame ID, as mentioned above, is associated with a time at which the captured image was captured.

The object detection module 318 employs a computer vision algorithm that identifies objects within an image or video. More particularly, the object detection module 318 is configured to determine whether each captured image in a respective frame set contains an object of interest 342 (FIG. 4), to define a bounding box 344 around each object of interest 342 located in the respective captured image and to assign a class identification to the object of interest 342. Class identifications assigned by the object detection module 318 indicate an inventory type of the object of interest 342. The inventory type may indicate a category of ingredient, food product, meal, dish, or other object type. FIG. 4 is an example of one image that was captured by a camera among the plurality of cameras 150, and many more similar images from the same and different cameras are analyzed in the object detection module 318 and bounding boxes (similar to the bounding box 344) can be generated around different objects of interest.

The object detection module 318 includes a computer vision model that outputs the bounding box 344 (FIG. 4) around the object of interest 342 (FIG. 4) and the class identification for the object of interest 342 for captured images containing the object of interest 342. In an example, the computer vision model can be YOLOv5, however the object detection module 318 may include additional or alternative computer vision models for determining object presence and class identification in the refrigerator 102. The updated group of images, which may include only captured images that contain an object of interest, may include a lesser number of images as compared to the frame set received by object detection module 318 from the synchronizer module 314. The first captured image that contains an object of interest in the updated group of images can indicate the start of a sequence, which indicates one hand motion in or out of the refrigerator. The last captured image that contains an object of interest in the updated group of images can indicate the end of the sequence. Where the object detection module 318 identifies no object of interest, for example the object detection module 318 identifies an empty hand without any object to be loaded into or removed from the refrigerator, the last captured image containing the object of interest can indicate the end of the sequence. Each sequence, or detected motion in or out of the refrigerator 102, can contain multiple objects of interest.

The object tracking module 322 is configured to analyze sequential captured images from each camera in the plurality of cameras 150 tracking objects of interest among the sequential captured images per camera. For example, the object tracking module 322 can compare a first n images captured in the updated group of images during a sequence to a last n images captured in the updated group of images during the sequence to track movement of objects of interest during the sequence. Determination of the temporal sequence of captured images is based on the frame ID associated with each image. For example, captured images having a higher frame ID can be captured later in time as compared to those with a lower frame ID. Determination of the camera that captured a respective captured image is based on the camera ID associated with the respective captured image. In this regard, the object tracking module 322 associates an object of interest with itself across sequential images per camera, thereby tracking movement of the object of interest over time.

The object tracking module 322 can also be configured to assign track identifications to associate the object of interest in one analyzed image to the object of interest in other analyzed images per camera in the plurality of cameras 150. Track identifications assigned by the object tracking module 322 are unique to each camera in the plurality of cameras 150, and unique to each object of interest captured by the plurality of cameras 150 in sequential captured images. For example, the object tracking module 322 is configured to assign a different track identification to an object of interest that appears as a lemon in the captured images than the track identification for an object of interest that appears as a cup of yogurt in the captured images.

The object tracking module 322 can employ an observation-centric simple online and real-time tracking (OCSORT) algorithm for analyzing the sequential captured images from each camera in the plurality of cameras 150. The object tracking module 322 receives the updated group of images with bounding boxes and class IDs from the object detection module 318 and the OCSORT algorithm can process the updated group of images, and then output a further updated group of images with the bounding boxes, class IDs and additionally include associated track IDs for each object located in each captured image in the further updated group of images.

The direction module 324 employs a computer algorithm to provide real-time direction prediction of moving objects within video streams. Leveraging input parameters such as object location, e.g., x, y coordinates, associated with the object of interest and door angles, e.g., the angle of the door 106 with respect to the main body 104, the direction module 324 can track changes in bounding box area or size and x, y coordinates. As for the door angle, the door 106 mounts to the main body 104 via an upper hinge 362 and a lower hinge 364, and a door angle sensor 366, which could include a rotary encoder having a shaft that rotates as the door 106 pivots with respect to the main body 104, can output the relative angular position of the door 106 with respect to the main body 104. Operating across the entire video, the direction module 324 can yield predictions for each camera among the plurality of cameras 150.

The direction module 324 can be configured to assign a direction to each sequence to determine if the object of interest is moving into, out of or internally within particular locations within the refrigerated enclosure 110 including the first compartment volume 116, the second compartment volume 122 and the third compartment volume 126. The direction module 324 can employ a machine learning algorithm, e.g., a vision transformer such as CSWin Transformer or another machine learning algorithm capable of performing the functions described herein. Alternatively, the direction module 324 can employ a rule-based algorithm. Moreover, the direction module 324 could employ both a rule-based algorithm and a machine learning algorithm.

When employing the rule-based algorithm and/or the machine learning algorithm, the direction module 324 can receive the sensed door angle and the object location associated with the object of interest as input parameters. For example, the sensed door angle and the object location associated with the object of interest for the first n analyzed images and the last n analyzed images during a sequence can be received by the direction module 324 the object detection module 318 or the object tracking module 322. The number n can be three images; however, a fewer or greater number could be used. Because the analyzed images are those that contain the object of interest, the first n analyzed images are indicative of the object entering a particular camera's field of view and the last n images are indicative of the object leaving the particular camera's field of view.

The object merge module 326 employs a computer algorithm that consolidates the direction predictions obtained from multiple cameras among the plurality of cameras 150. The object merge module 326 algorithm processes predictions from each camera, identifying unique inventory changes, and combines the results to produce a comprehensive and consolidated set of predictions. The object merge module 326 algorithm receives predictions from multiple cameras, which can be organized as a dictionary (cam_to_preds), where each camera is associated with a list of predictions. Each prediction can be represented as a tuple (track_id, class_id, change), where track_id is a unique identifier for the object of interest 342, class_id indicates the class of the object of interest 342, and change signifies the detected directional change in the object of interest 342, e.g., into, out of or no change with respect to the refrigerated enclosure 110. The algorithm within the object merge module 326 can then iterate through the predictions for each camera associating the track_id's across multiple cameras according to the movement of the object of interest, creating a unique identifier (inventory_change_id) for each inventory change based on the class identification for the object of interest 342 and the detected change. The object merge module 326 algorithm can then count the occurrences of each unique inventory change within each camera's predictions and then combines the counts of each unique inventory change across all cameras. If a particular inventory change is observed in multiple cameras, the object merge module 326 algorithm retains the maximum count, increasing the likelihood of an accurate representation of the overall occurrence. The consolidated results are stored in a format that captures the class_id, change, and the count of occurrences. The final result is a list of lists, each containing the class_id (food), inventory change, and the number of inventory changes for that item observed across all cameras.

The inventory update module 328 is configured to communicate with the memory 304 to update an inventory status for the refrigerated enclosure 110 including the first compartment volume 116, the second compartment volume 122 and the third compartment volume 126 based on each inventory change and the direction assigned thereto.

FIG. 5 depicts the first storage compartment 114 as a drawer that is movable between a closed position and an open position. When in the open position the drawer extends forwardly from main body opening 108. At least the first set of cameras 160a, 160b mounted on the main body 104 are positioned to have a field of view to include a drawer opening leading to the drawer even though the drawer extends forwardly from main body opening 108. As mentioned above, each camera of the plurality of cameras 150 is positioned to have a field of view forward of the main body opening 108 when the door is in the open position. Such a configuration allows for the detection of objects of interest prior to the object of interest having to cross the plane of the main body opening 108, which is depicted in FIG. 4.

FIG. 6 depicts a refrigerator 502 including a main body 504 and two doors 506 pivotally mounted to and movable with respect to the main body 104 in a French door style arrangement. As shown in FIG. 6, with the doors 506 closed covering a main body opening (not visible with the doors closed), the main body 504 and two doors 506 define at least a portion of a refrigerated enclosure (not visible in FIG. 6, but similar to known refrigerators). The refrigerator 502 includes at least one storage compartment, e.g., an upper drawer 514, within the refrigerated enclosure that is able to be closed off from other portions of the refrigerated enclosure so that a storage condition within the upper drawer 514 can be controlled separately from the remainder of the refrigerated enclosure.

A plurality of cameras 520 (only one schematically depicted in FIG. 6) mount to the main body 504. Similar to the embodiment discussed above, each of the plurality of cameras 520 are positioned to have a field of view forward of the main body opening for capturing images of objects moving into and out of the refrigerated enclosure. However, the camera 520 depicted in FIG. 6 is capable of capturing images of objects moving into and out of the upper drawer 514 when the two doors 506 are closed. The refrigerator 502 can include at least one computing device similar to the computing device 200 as well as the other components depicted in FIG. 3. Accordingly, further description of the refrigerator 502 will be omitted for the sake of brevity.

It will be appreciated that various of the above-disclosed embodiments and other features and functions, or alternatives or varieties thereof, may be desirably combined into many other different systems or applications. Also that various presently unforeseen or unanticipated alternatives, modifications, variations or improvements therein may be subsequently made by those skilled in the art which are also intended to be encompassed by the following claims.