REFRIGERATOR AND CONTROLLING METHOD THEREOF

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of International Application No. PCT/KR2025/007277 designating the United States, filed on May 28, 2025, in the Korean Intellectual Property Receiving Office and claiming priority to Korean Patent Application No. 10-2024-0100774, filed on Jul. 30, 2024, in the Korean Intellectual Property Office, the disclosures of each of which are incorporated by reference herein in their entireties.

BACKGROUND

1. Field

One or more example embodiments of the disclosure relate to a refrigerator and a controlling method thereof, and more particularly, to a refrigerator that captures food products that are entered in and/or taken from the refrigerator and manages information on the food products and a controlling method thereof.

2. Description of Related Art

With developments in electronic technology, electronic devices of various types are being used in daily life. Among these electronic devices, there may be smart refrigerators that employ a neural network model, and the like.

In general, a refrigerator may be a home appliance that can keep food products fresh for a long period by including a storage compartment for storing food products, and a cool air supplying device for supplying cool air to the storage compartment.

Specifically, recent refrigerators may capture an inside of a main body by including a camera in the main body that includes the storage compartment. Further, the refrigerator may obtain information on the food product currently stored in the refrigerator based on the captured image, and provide a user with obtained information on the food product.

SUMMARY

According to an aspect of at least one embodiment of the disclosure, a refrigerator includes a camera, at least one memory storing instructions and at least one processor, wherein at least one instruction from among the instructions that, when executed by the at least one processor individually or collectively, causes the refrigerator to control the camera to capture an object based on the object being entered in or taken from the refrigerator, provide, based on obtaining a capture image about the object entered in or taken from the refrigerator, a guide for obtaining an additional image, obtain, based on detecting a signal for capturing the object corresponding to the provided guide, the additional image through the camera and store the capture image and the additional image as data on the object being entered or taken.

In addition, according to an aspect of at least one embodiment of the disclosure, a method of controlling a refrigerator includes capturing an image on an object being entered in or taken from the refrigerator, providing, based on obtaining a capture image on the object being entered in or taken from the refrigerator, a guide for obtaining an additional image, obtaining the additional image based on detecting a signal for capturing the object corresponding to the provided guide and storing the capture image and the additional image as data on the object being entered or taken.

BRIEF DESCRIPTION OF THE DRAWINGS

Example embodiments of the disclosure will be more clearly understood from the following detailed description taken in conjunction with the accompanying diagrams.

FIG. 1 is a diagram illustrating an operation of a refrigerator according to at least one embodiment of the disclosure;

FIG. 2 is a block diagram illustrating a configuration of a refrigerator according to at least one embodiment of the disclosure;

FIG. 3 is a detailed block diagram illustrating a refrigerator according to at least one embodiment of the disclosure;

FIG. 4 is a diagram illustrating an operation for obtaining data on an object that is entered in or taken from a refrigerator according to at least one embodiment of the disclosure;

FIG. 5 is a diagram illustrating an operation for obtaining data on an object that is entered in or taken from a refrigerator according to at least one embodiment of the disclosure;

FIG. 6 is a diagram illustrating an operation for obtaining data on an object that is entered in or taken from a refrigerator according to at least one embodiment of the disclosure;

FIG. 7 is a diagram illustrating an operation for obtaining an image about an object that is entered in or taken from a refrigerator according to at least one embodiment of the disclosure;

FIG. 8 is a diagram illustrating a user interface (UI) providing a guide of a refrigerator according to at least one embodiment of the disclosure;

FIG. 9 is a diagram illustrating a UI registering an additional image according to a guide of a refrigerator according to at least one embodiment of the disclosure;

FIG. 10 is a diagram illustrating a UI providing data on an object that is entered in or taken from a refrigerator according to at least one embodiment of the disclosure;

FIG. 11 is a diagram illustrating a UI providing data on an object that is entered in or taken from a refrigerator according to at least one embodiment of the disclosure;

FIG. 12 is a diagram illustrating an operation for extracting features of an object that is entered in or taken from a refrigerator according to at least one embodiment of the disclosure;

FIG. 13 is a diagram illustrating an example of a refrigerator receiving data from various electronic devices according to at least one embodiment of the disclosure;

FIG. 14 is a flowchart illustrating an operation for managing data on an object that is entered in or taken from a refrigerator according to at least one embodiment of the disclosure; and

FIG. 15 is a flowchart illustrating an operation for managing data on an object that is entered in or taken from a refrigerator according to at least one embodiment of the disclosure.

DETAILED DESCRIPTION

Terms used in describing various embodiments of the disclosure are general terms selected that are currently widely used considering their function herein. However, the terms may change depending on intention, legal or technical interpretation, emergence of new technologies, and the like of those skilled in the related art. Further, in certain cases, there may be terms arbitrarily selected, and in this case, the meaning of the term will be disclosed in greater detail in the corresponding description. Accordingly, the terms used herein are not to be understood simply as its designation but based on the meaning of the term and the overall context of the disclosure.

Various embodiments of the disclosure and terms used herein are not intended to limit the technical features described in the disclosure to specific embodiments, and it is to be understood as including various modifications, equivalents, or alternatives of the corresponding embodiments.

With respect to the description of the drawings, like reference numerals may be used for like or related elements.

A singular form of a noun corresponding to an item may include one or a plurality of items above, unless otherwise specified.

In the disclosure, expressions such as “at least one of”, when preceding a list of elements, modify the entire list of elements and do not modify the individual elements of the list, and phrases such as “A or B”, “at least one of A and B”, “at least one of A or B”, “at least one from among A or B,” “A, B, or C”, “at least one of A, B, and C”, and “at least one of A, B, or C” may respectively include any one or all possible combinations of the items listed together with the relevant phrase from among the phrases.

Terms such as “1st”, “2nd”, “first” or “second” may be used to simply distinguish a relevant element from another relevant element, and not limit the relevant elements in other aspects (e.g., importance or order).

When a certain (e.g., first) element is indicated as being “coupled with/to” or “connected to” another (e.g., second) element, together with or without terms such as “operatively” or “communicatively”, it is to be understood as the certain element being coupled with/to the another element directly (e.g., via wire), wirelessly, or through a third element.

Terms such as “have” or “include” are used herein to designate a presence of a characteristic, number, step, operation, element, component, or a combination thereof, and not to preclude a presence or a possibility of adding one or more of other characteristics, numbers, steps, operations, elements, components or a combination thereof.

When a certain element is described as “coupled”, “combined”, “supported”, or “contacted” with another element, the above may include not only the elements being directly coupled, combined, supported, or contacted, but also being indirectly coupled, combined, supported, or contacted through the third element.

When a certain element is described as positioned “on” another element, the above may include not only the certain element being contacted to the another element, but also other element being present between the certain element and the another element.

A refrigerator according to an embodiment may include a main body.

The “main body” may include an inner case, an outer case disposed at an outer side of the inner case, and an insulation material provided between the inner case and the outer case.

The “inner case” may include at least one from among a case, a plate, a panel, or a liner that form a storage compartment. The inner case may be formed as one body or formed with a plurality of plates being assembled. The “outer case” may form an exterior of the main body, and may be coupled to the outer side of the inner case for the insulating material to be disposed between the inner case and the outer case.

The “insulating material” may insulate an inside of the storage compartment and an outside of the storage compartment for temperature inside of the storage compartment to be maintained at an appropriately set temperature without being affected by the external environment of the storage compartment. According to an embodiment, the insulating material may include a foam insulating material. The foam insulating material may be formed by injecting and foaming urethane foam mixed with polyurethane and a foaming agent between the inner case and the outer case.

According to an embodiment, the insulating material may include a vacuum insulating material additionally in addition to the foam insulating material, or the insulating material may be formed with only the vacuum insulating material rather than the foam insulating material. The vacuum insulating material may include a core material, and a covering material that contains the core material and seals the inside in a vacuum state or at a pressure close to the vacuum state. However, the insulating material is not limited to the above-described foam insulating material or the vacuum insulating material, and may include various materials that may be used for heat insulation.

The “storage compartment” may include a space limited by the inner case. The storage compartment may further include an inner case that limits a space corresponding to the storage compartment. In the storage compartment, various products such as food products, medical products, cosmetic products, and the like may be stored, and the storage compartment may be formed such that at least one side may be opened to input and output products.

The refrigerator may include one or more storage compartments. When two or more storage compartments are formed in the refrigerator, the two or more storage compartments may have different uses from each other and may be maintained at different temperatures. To this end, each storage compartment may be separated from each other by a partition wall that includes an insulating material.

The storage compartment may be provided to be maintained within an appropriate temperature range according to use, and may include a “refrigerating compartment”, a “freezing compartment” or a “temperature changeable compartment” which are distinguished according to use thereof and/or a temperature range thereof. The refrigerating compartment may be maintained at an appropriate temperature to keep products refrigerated, and the freezing compartment may be maintained at an appropriate temperature to keep products frozen. “Refrigeration” may mean chillingly cooling to an extent that products are not frozen, and as an example, the refrigerating compartment may be maintained between a temperature range of zero Celsius (0° C.) and seven degrees above zero Celsius (7° C.). “Freezing” may mean freezing products or cooling the products to maintain a frozen state, and as an example, the freezing compartment may be maintained between a temperature range of twenty degrees below zero Celsius (−20° C.) and one degree below zero Celsius (−1° C.). The temperature changeable compartment may be used based on a user selection or any one of the refrigerating compartment or freezing compartment regardless of the user selection.

The storage compartment, in addition to designations such as the “refrigerating compartment”, the “freezing compartment”, and the “temperature changeable compartment”, may be referred in various designations such as a “vegetable compartment”, a “fresh compartment”, a “cooling compartment”, and a “ice forming compartment”, and the terms such as the “refrigerating compartment”, the “freezing compartment”, and the “temperature changeable compartment” used below is to be understood as a meaning that includes the storage compartments having uses and temperature ranges respectively corresponding thereto.

According to an embodiment, the refrigerator may include at least one door configured to open and close one side of the storage compartment that is opened. The at least one door may be provided to open or close one or more storage compartments, respectively, or one door may be provided to open and close a plurality of storage compartments. The door may be installed at a front surface of the main body to be rotatable or slidable.

The “door” may be configured to seal (or cover) the storage compartment when the door is closed. The door may include an insulating material like the main body to heat insulate the storage compartment when the door is closed.

According to an embodiment, the door may include a door outside plating that forms a front surface of the door, a door inside plating that forms a back surface of the door and faces the storage compartment, an upper cap, a lower cap, and a door insulating material provided inside of the above components of the door.

At an edge of the door inside plating, a gasket for sealing the storage compartment may be provided as the door is closely contacted at the front surface of the main body when closed. The door inside plating may include a dyke which is protruded toward a back direction for a door basket which may store products to be mounted.

According to an embodiment, the door may include a door body, and a front panel that is separably coupled to a front side of the door body and forms the front surface of the door. The door body may include a door outside plating that forms a front surface of the door body, a door inside plating that forms a back surface of the door body and faces the storage compartment, an upper cap, a lower cap, and a door insulating material provided inside of the above components of the door.

The refrigerator may be divided into to a French door type, a side-by-side type, a bottom mounted freezer (BMF), a top mounted freezer (TMF), a one-door refrigerator, or the like according to an arrangement of the door and the storage compartment.

According to an embodiment, the refrigerator may include a cool air supplying device configured to supply cool air to the storage compartment.

The “cool air supplying device” may include a machine that may generate cool air and cool the storage compartment by guiding the cool air, an equipment, an electronic device and/or a system that combines one or more of the above.

According to an embodiment, the cool air supplying device may generate cool air through a refrigeration cycle that includes a compression process, a condensation process, an expansion process, and an evaporation process of a refrigerant. To this end, the cool air supplying device may include a refrigeration cycle device having a compressor, a condenser, an expander, and an evaporator which may drive the refrigeration cycle. According to an embodiment, the cool air supplying device may include a semiconductor such as a thermoelectric device. The thermoelectric device may cool the storage compartment with exothermic action and cooling action through a Peltier effect.

According to an embodiment, the refrigerator may include a machinery room that is provided for at a portion of components belonging to the cool air supplying device to be disposed.

The “machinery room” may be provided to be partitioned and heat insulated from the storage compartment to prevent heat generated from the components disposed in the machinery room from being transferred to the storage compartment. An inside of the machinery room may be configured to communicate with an outside of the main body to radiate heat from the components disposed inside of the machinery room.

According to an embodiment, the refrigerator may include a dispenser provided at the door to provide water and/or ice. The dispenser may be provided at the door to be accessible to users without opening the door.

According to an embodiment, the refrigerator may include an ice making device provided to generate ice. The ice making device may include an ice making tray which stores water, an ice removing device which separates ice from the ice making tray, and an ice bucket which stores the ice generated from the ice making tray.

According to an embodiment, the refrigerator may include a controller configured to control the refrigerator.

The “controller” may include at least one memory configured to store and/or retain program and/or data related to controlling the refrigerator, and at least one processor configured to output control signals related to controlling the cool air supplying device and the like according to the program and/or data stored and/or retained from the at least one memory.

The at least one memory (hereinafter, referred to as “memory”) may store or record various information, data, instructions, program and the like necessary in an operation of the refrigerator. The memory may retain temporary data generated while generating control signals related to controlling configurations included in the refrigerator. The memory may include at least one or a combination of a volatile memory or a non-volatile memory.

The at least one processor (hereinafter, referred to as “processor”) may control an overall operation of the refrigerator. The processor may control elements of the refrigerator by executing the program stored in the memory. The processor may include a separate neural processing unit (NPU) which performs operations of an artificial intelligence model. In addition, the processor may include a central processing unit (CPU), a graphics dedicated processor (GPU), and the like. The processor may generate control signals for controlling an operation of the cool air supplying device. For example, the processor may receive temperature information of the storage compartment from a temperature sensor, and generate a cooling control signal for controlling an operation of the cool air supplying device based on the temperature information of the storage compartment.

In addition, the processor may process a user input of a user interface according to the program and/or data retained and/or stored in the memory, and control an operation of the user interface. The user interface may be provided using an input interface and an output interface. The processor may receive a user input through the user interface. In addition, the processor may transfer a display control signal for displaying an image in the user interface and image data to the user interface in response to the user input.

The processor and the memory may be provided integrally or provided separately. The processor may include one or more processors. For example, the processor may include a main processor and at least one sub processor. The memory may include one or more memory.

According to an embodiment, the refrigerator may include at least one processor that controls all the configurations included in the refrigerator and a plurality of processors and a plurality of memories which individually control the configurations of the refrigerator. For example, the refrigerator may include at least one processor and at least one memory which controls an operation of the cool air supplying device according to an output of the temperature sensor. In addition, the refrigerator may separately include at least one processor and at least one memory which controls an operation of the user interface according to a user input.

A communication interface may communicate with external devices such as a server, a mobile device, other home appliances, and the like through a peripheral access point (AP). The access point (AP) may connect a local area network (LAN) to which the refrigerator and/or a user device is connected to a wide area network (WAN) to which the server is connected. The refrigerator and/or the user device may be connected to the server through the wide area network (WAN).

The input interface may include, for example but not limited to, a key, a touchscreen, a microphone, and the like. The input interface may receive a user input and transmit the user input to the processor.

The output interface may include, for example but not limited to, a display, a speaker, and the like. The output interface may output various notifications, messages, information, and the like generated from the processor.

The meaning of a refrigerator “providing” a guide (or guide information) in the disclosure may include not only displaying the guide through a display included in the refrigerator, but also displaying the guide through a display of a user terminal by transmitting the guide to the user terminal communicatively connected with the refrigerator.

The term “and/or” may include a combination of a plurality of related elements described or any element from among the plurality of related elements described.

The term “module” or “part” used in the disclosure may perform at least one function or operation, and may be implemented with hardware or software, or implemented with a combination of hardware and software. In addition, a plurality of “modules” or a plurality of “parts”, except for a “module” or a “part” which needs to be implemented with a specific hardware, may be integrated in at least one module and implemented as at least one processor.

Various elements and areas of the drawings have been schematically illustrated. Accordingly, the technical spirit of the disclosure is not limited by relative sizes and distances illustrated in the accompanied drawings.

In the disclosure, the term “user” may refer to a person using an electronic device and/or a device (e.g., artificial intelligence electronic device) using the electronic device.

The refrigerator according to an embodiment of the disclosure will be described in detail with reference to the attached drawings below.

FIG. 1 is a diagram illustrating an operation of the refrigerator according to at least one embodiment of the disclosure.

Referring to FIG. 1, a refrigerator 100 may perform capturing of an object that is entered or taken. For example, if a user enters a food product 10 in the refrigerator 100, the refrigerator may obtain a capture image of the food product 10 by detecting movement of the food product 10.

According to an embodiment, the refrigerator 100 may obtain, based on information on classification of an object not being obtainable based on the obtained capture image, an additional image by providing a guide for obtaining the additional image. The refrigerator 100 may store data on an object based on the capture image and the additional image.

Here, the “capture image” may mean an image obtained while proceeding with capturing of an object. For example, it may mean an image captured by the refrigerator 100, or it may mean a capture image obtained from an external device (e.g., a user terminal device, a server device, etc.).

Here, the “additional image” may mean an image obtained by proceeding with additional capturing of the object. For example, the capture image may mean an image obtained by automatically capturing an object being entered in or taken from the refrigerator, and the additional image may mean an image obtained by proceeding with additional capturing according to a capture guide after having obtained the capture image.

The additional image may include an image of various capture angles to an extent that information on classification of an object being entered in or taken from the refrigerator may be sufficiently secured. For example, if a food product being entered or taken is identified as a canned beverage, the refrigerator 100 may additionally request an image on a front surface, a lateral surface, and/or a side surface of the canned beverage.

The “capture image” in the disclosure may be a concept that includes the additional image, and may be differently described as a capture image, an image, an additional capture image, a food product image, image data, and the like.

In FIG. 1, the refrigerator 100 may be illustrated as a general refrigerator for a typical household use, but is not limited thereto, and may be various types of a refrigerator such as, for example but not limited to, a kimchi refrigerator, an alcohol refrigerator, a cosmetic product refrigerator, a freezer, and the like.

FIG. 2 is a block diagram illustrating a configuration of a refrigerator according to at least one embodiment of the disclosure.

Referring to FIG. 2, the refrigerator 100 may include a camera 110, one or more memories 120 (hereinafter, referred to as “memory”), and one or more processors 130. The refrigerator 100 may be a device that keeps things such as food products or medical products of sorts cool or at a pre-set temperature or lower to prevent from spoiling.

An object that is entered in or taken from (hereinafter, referred to as “entered/taken” for brevity of description) the refrigerator 100 may include an object that needs to be kept at a pre-set temperature or lower such as a food product and a cosmetic product, and for convenience of description, mainly a food product will be described below as an example.

The camera 110 may capture an object and generate the capture image, and here, the capture image may include a moving image and/or a still image. The “image” in the disclosure may be a concept that includes an image that is output on the display and an image frame captured by the camera 110.

Specifically, the camera 110 may capture the storage compartment inside the main body of the refrigerator 100 and a door bin (or a door basket, a pantry, etc.) area of the door. The camera 110 may be provided on at least one from among an upper end area, a lower end area, or a lateral surface area inside the main body to capture the inside of the main body and the door bin area of the door. In addition, the camera 110 may capture an outside of the refrigerator 100 by being provided at the outside of the refrigerator 100. That is, the camera 110 may be implemented as not only one camera, but also as a plurality of cameras according to an embodiment.

The refrigerator 100 may include the camera 110 at an upper end area of the main body to capture at least a portion of the storage compartment and the door bin of the door. However, the embodiment is not limited thereto, and the camera 110 may be positioned at another area (e.g., a back surface area, a lower end area, a lateral surface area, etc.) inside the main body, or provided in plurality.

In addition, the camera 110 may provide a captured image to the processor 130 to manage entering/taking of food products.

In addition, the camera 110 may be implemented as a wide angle camera to capture a wide viewing angle, but is not limited thereto.

The memory 120 may store an operating system (OS) for controlling the overall operation of elements of the refrigerator 100 and instructions associated with the elements of the refrigerator 100, computer programs including instructions, data on objects, or the information on classification of objects.

Specifically, the memory 120 may store various configurations for managing the entering/taking of food products. In addition, the memory 120 may store a food product database (DB) which stores information (e.g., types of food products, amount of food products, expiration of food products, storage location of the food products, etc.) on food products stored in the refrigerator 100.

In addition, according to an embodiment, the memory 120 may store a trained neural network model (e.g., a food product classifying model, etc.) for obtaining feature information corresponding to the food product entered in or taken from the refrigerator 100. Additionally or alternatively, the memory 120 may store a trained neural network model (e.g., an entering/taking recognition model, etc.) for recognizing the food products entered in or taken from the refrigerator 100.

However, the embodiment is not limited thereto, and various neural network models, databases (DBs), and the like may be stored in the memory 120 of the refrigerator 100, or stored in an external device such as a server.

The memory 120 may store a trained neural network model for converting a capture image with respect to the food product being entered in/taken from the refrigerator 100. Additionally or alternatively, the memory 120 may store a trained neural network model (e.g., a feature extraction model or a feature matching model) for analyzing a feature of an object being entered in/taken from the refrigerator 100 and obtaining the information on classification of the object that is being entered/taken based on the analyzed feature of the object.

Additionally or alternatively, the memory 120 may store an image of an actual inside of the refrigerator 100 or an image that captures an object being entered/taken. The memory 120 may be implemented in a form of a memory embedded in the refrigerator 100 according to data storage use, and/or implemented in a form of a memory attachable to and/or detachable from the refrigerator 100.

For example, data for driving of the refrigerator 100 may be stored in the memory embedded in the refrigerator 100, and data for an expansion function of the refrigerator 100 may be stored in the memory attachable to and/or detachable from the refrigerator 100.

The memory embedded in the refrigerator 100 may be implemented in a form such as a volatile memory (e.g., a dynamic RAM (DRAM), a static RAM (SRAM), or a synchronous dynamic RAM (SDRAM)), a non-volatile memory (e.g., a one-time programmable ROM (OTPROM), a programmable ROM (PROM), an erasable and programmable ROM (EPROM), an electrically erasable and programmable ROM (EEPROM), a mask ROM, a flash ROM, a flash memory (e.g., NAND flash or NOR flash), a hard disk drive (HDD) or a solid state drive (SSD)), and the like.

The memory attachable to and/or detachable from the refrigerator 100 may be implemented in a form such as a memory card (e.g., a compact flash (CF), a secure digital (SD), a micro secure digital (micro-SD), a mini secure digital (mini-SD), an extreme digital (xD), a multi-media card (MMC), etc.), an external memory (e.g., universal serial bus (USB) memory) connectable to a USB port, or the like.

According to an embodiment, the one or more processors 130 may control the overall operation of the refrigerator 100 according to at least one instruction stored in the memory 120. Specifically, the one or more processors 130 may control the overall operation of the refrigerator 100 by being connected with each configuration of the refrigerator 100.

According to an embodiment, the one or more processors 130 may be implemented as a digital signal processor (DSP) that processes digital signals, a microprocessor, an artificial intelligence (AI) processor, or a time controller (TCON). However, the embodiment is not limited thereto, and the one or more processors 130 may include one or more from among a central processing unit (CPU), a micro controller unit (MCU), a micro processing unit (MPU), a controller, an application processor (AP), a communication processor (CP), an advanced reduced instruction set computing (RISC) machine (ARM) processor, or an artificial intelligence (AI) processor, or may be defined by a corresponding term.

In addition, the one or more processors 130 may be implemented with a system on chip (SoC) or a large scale integration (LSI) in which a processing algorithm is embedded, and may be implemented in a form of a field programmable gate array (FPGA). The one or more processors 130 may perform various functions by executing computer executable instructions stored in the memory 120.

The one or more processors 130 may include one or more from among a central processing unit (CPU), a graphics processing unit (GPU), an accelerated processing unit (APU), a many integrated core (MIC), a digital signal processor (DSP), a neural processing unit (NPU), a hardware accelerator, or a machine learning accelerator.

The one or more processors 130 may control one or a random combination from among other elements of the electronic device, and perform an operation associated with communication or data processing. The one or more processors 130 may execute one or more programs or instructions stored in the memory 120. For example, the one or more processors 130 may perform, by executing one or more instructions stored in the memory 120, a method according to an embodiment of the disclosure.

When a method according to an embodiment of the disclosure includes a plurality of operations, the plurality of operations may be performed by one processor, or performed by the plurality of processors.

For example, when a first operation, a second operation, and a third operation are performed by a method according to an embodiment, the first operation, the second operation, and the third operation may all be performed by a first processor, or the first operation and the second operation may be performed by the first processor (e.g., a generic-purpose processor) and the third operation may be performed by a second processor (e.g., an artificial intelligence dedicated processor).

The one or more processors 130 may be implemented as a single core processor that includes one core, or implemented as one or more multicore processors that include a plurality of cores (e.g., a homogeneous multicore or a heterogeneous multicore).

If the one or more processors 130 are implemented as multicore processors, each of the plurality of cores included in the multicore processors may include a memory inside the processor such as a cache memory and an on-chip memory, and a common cache shared by the plurality of cores may be included in the multicore processors.

Each of the plurality of cores (or a portion from among the plurality of cores) included in the multicore processors may independently read and perform a program command for implementing a method according to one or more embodiments, or read and perform a program command for implementing a method according to an embodiment of the disclosure due to a whole (or a portion) of the plurality of cores being interconnected.

According to an embodiment of the disclosure, the one or more processors 130 may control, based on an object being entered in or taken from the refrigerator 100, the camera 110 to capture the object being entered/taken. The one or more processors 130 may provide, based on obtaining a capture image about the object being entered in or taken from the refrigerator 100, a guide for obtaining an additional image based on the obtained capture image.

Here, the “guide” may mean a guide or guideline for capturing an image. For example, the guide may include guide information on a capture angle to obtain the information on classification of an object that is entered or taken based on the capture image. The guide information included in the guide may be output in voice (or audio) and/or displayed as an image or text.

If a lateral surface of a beverage can that is entered in/taken from the refrigerator 100 is captured, the one or more processors 130 may provide a guide to the user to capture a front surface of the beverage can to obtain information (e.g., trademark and type of beverage, etc.) on classification of the beverage can.

According to an embodiment, the one or more processors 130 may obtain, based on a signal for capturing an object corresponding to the provided guide being detected, an additional image through the camera 110.

For example, when the user adjusts a position of the object for the object (or a requested portion such as a front surface of the object) to be visible to the camera 110 according to a request by the guide, the one or more processors 130 may obtain an additional image about the object through the camera 110 by detecting a position change of the object as a signal for capturing the object.

The camera 110 may include one or more cameras, and if a plurality of cameras are included, each camera 110 may capture an object being entered/taken from different positions from one another. For example, the one or more processors 130 may capture, when the entering/taking of an object is recognized, the object from different directions from one another through the plurality of cameras 110, and register the one or more capture images as data on the object being entered/taken.

According to an embodiment of the disclosure, the one or more processors 130 may store the capture image and the additional image as data on the object being entered or taken. For example, the one or more processors 130 may register, based on a capture image which captures a canned beverage being obtained, the additional image for obtaining the information on classification of the canned beverage as data on the corresponding canned beverage together with the capture image.

Here, the “data on the object being entered/taken” may mean various information on the objects being entered in/taken from the refrigerator. For example, the data on the object being entered/taken may include data of various types for obtaining the information on classification of objects such as, for example, and without limitation, a capture image about the object being entered in/taken from the refrigerator 100, an additional image, a generated image, data on the object obtained from the external device, and the like.

Here, the “information on classification of objects (or an object)” may mean various information on the classification of objects (or object). For example, the information on classification of an object may include at least one from among a name of the object, a type of the object, a number of the object, information for purchasing an ingredient, and information on entered or taken time of the object.

For example, the one or more processors 130 may obtain information such as shopping for purchasing an ingredient as the information on classification of an object by referencing data on the object being entered/taken (e.g., recent purchase list information on ingredients).

FIG. 3 is a detailed block diagram illustrating the refrigerator according to at least one embodiment of the disclosure.

Referring to FIG. 3, the refrigerator 100 according to an embodiment of the disclosure may include the camera 110, the memory 120, the one or more processors 130, an output device 140, a communication device 150, a microphone 160, and a sensor 170. Parts that overlap with the above-described description below may be omitted or abridged.

The output device 140 may provide various feedback. Specifically, the output device 140 may include, as shown in FIG. 3, a speaker 141, a display 142, a light emitting diode (LED) 143, and/or the like, but the above is merely one embodiment, and another output device (e.g., a haptic providing device, etc.) may be further included and any one of the above described examples of the output device 140 may be omitted.

The speaker 141 may provide various audible feedback through audio by being provided inside or outside of the refrigerator 100. For example, if the capture image about the food product entered in/taken from the refrigerator is insufficient for obtaining food product information, the one or more processors 130 may output an audio such as “please capture while rotating the food product slightly” through the speaker 141.

The LED 143 may be provided in the storage compartment inside the refrigerator 100 or inside the door and provide various visual feedback through indicators of various shapes, blinking, and the like.

The display 142 may provide various visual feedback to the user by being positioned on at least a portion of area(s) from among one or more doors of the refrigerator. Specifically, the display 142 may display a user register user interface (UI) requesting whether to store the capture image in the data on the object being entered/taken.

The user register UI may include an area for the user to directly register a trademark name and a type of the object with respect to the capture image about the object being entered in/taken from the refrigerator 100. The user register UI may be referred in various terms such as, for example, and without limitation, a user DB register, a user database register, a user registered information, and the like.

The display 142 may display an object classification UI including the information on classification of an object. For example, if two orange juices are entered in the refrigerator 100, the display 142 may display a UI including information on product names of the orange juices, a date on which the orange juices are entered, an expiration date of the orange juices, and a number of orange juices remaining in the storage compartment.

The information included in the object classification UI may be referred to in various terms such as, for example, and without limitation, food product information, food product classification information, a food product list, product information, and the like. The above will be described in detail in FIG. 10.

The communication device 150 may perform communication with an external server and/or an external terminal device. Specifically, the communication device 150 may receive, in order to obtain information on a food product, an image including the food product or information on the food product from the external server.

In addition, the communication device 150 may receive, in order to obtain information on the food product, an image including the food product, information on the food product, or the like from a user terminal. An operation for performing communication with an external device will be described in detail in FIG. 13.

The communication device 150 may include a wired and/or wireless input and/or output interface (or, input and/or output terminal) according to various standards. For example, one or more connection interfaces of the communication device 150 may include various interfaces such as, for example, and without limitation, a high-definition multimedia interface (HDMI), a mobile high-definition link (MHL), a universal serial bus (USB), a display port (DP), Thunderbolt, a video graphics array (VGA) port, an RGB port, a D-subminiature (D-SUB), a digital visual interface (DVI), an AP-based Wi-Fi (e.g., a wireless LAN network), Bluetooth, ZigBee, a wired/wireless local area network (LAN), a wide area network (WAN), Ethernet, IEEE 1394, Audio Engineering Society/European Broadcasting Union (AES/EBU), Optical, Coaxial, or the like.

The microphone 160 may obtain an audio signal and convert the audio signal to an electric signal, and may be provided in the inside or at the outside of the refrigerator 100. Specifically, the microphone 160 may receive an audio signal that include a user voice. Information on the entering or taking and the information on classification of an object (e.g., types of food products, expiration of food products, etc.) may be included in the user voice.

The sensor 170 may detect an operating state (e.g., power or temperature) of the refrigerator 100, or an external environmental state (e.g., user state), and generate an electric signal or a data value corresponding to the detected state. The sensor 170 may include one or more sensors, and when a plurality of sensors are provided, the plurality of sensors may be provided at different positions in the refrigerator 100. Specifically, the one or more processors 130 may respectively measure temperature of the one or more storage compartments in the refrigerator 100 through a sensing value obtained through the sensor 170.

Additionally or alternatively, the one or more processors 130 may recognize that the user approaches through the sensor 170, and control for the camera 110 to be in a ready state. Additionally or alternatively, the one or more processors 130 may detect an opening of the door through the sensor 170, and operate the camera 110.

For example, the one or more processors 130 may detect, through the sensor 170, an object corresponding to the guide provided based on a movement of the user as a signal for capturing. In this case, the one or more processors 130 may obtain an additional image about the object by controlling the camera 110.

FIG. 4 is a diagram illustrating an operation for obtaining data on an object that is entered in or taken from the refrigerator according to at least one embodiment of the disclosure.

The one or more processors 130 may obtain the information on classification of an object being entered/taken by detecting the object being entered/taken by a motion of the user.

For example, the one or more processors 130 may obtain, when a movement of the user entering a food product 20 in the refrigerator is detected (410), a capture image 21 about the food product 20 being entered by controlling the camera 110. The obtained capture image may be input to a trained neural network model (e.g., food product classifying model) (420).

If a result of the food product classifying model is recognized as not being subject to existing food product classification or if the result indicates that the food product 20 is not recognized (430), the capture image may be input to a trained neural network model (e.g., feature matching model), and a similarity may be identified by comparing the capture image with pre-stored data (e.g., data on objects being entered or taken) (440).

The one or more processors 130 may select data on an object being entered or taken that has a similarity, with respect to the capture image, that is greater than or equal to a pre-set level from among the pre-stored data on the objects being entered or taken and obtain the information on classification of an object based on the selected data on the objects being entered or taken.

For example, if there is data on the object being entered or taken having a similarity of greater than or equal to a pre-set level, the one or more processors 130 may obtain food product classification information of “food product name ABC and tomato juice” if “matching is successful” using user database (DB) matching (451).

Conversely, if there is no data on the object being entered or taken having a similarity of greater than or equal to the pre-set level, the one or more processors 130 may not obtain food product classification information by “failing in matching” (452). However, the above is merely one embodiment, and whether or not matching is successful may be determined by another standard and/or method.

The one or more processors 130 may provide, based on not being able to obtain the information on classification of an object based on the selected data on the object being entered or taken, a guide for obtaining an additional image based on the obtained capture image.

The one or more processors 130 may request, based on not being able to obtain the information on the classification of an object based on the selected data on the object being entered or taken, an additional registration directly to the user in addition to or alternatively to providing the guide for obtaining the additional image.

Specifically, the refrigerator 100 may request to the user whether to store the capture image in the selected data on the object being entered or taken through the speaker 141 or the display 142, and when a user input corresponding to the request is received, proceed with additional registration by storing the capture image in the selected data on the object being entered or taken.

The one or more processors 130 may perform, even when the information on classification of an object is obtained based on the selected data on the object being entered or taken, an operation for obtaining an additional image if the additional image is required.

For example, the one or more processors 130 may provide the guide for obtaining an additional image if the classification of the food product being entered or taken is not correctly recognized or if a result from food product matching is incorrect (e.g., the information on classification of the object obtained through the food product classifying model and/or the feature matching model does not correspond to the food product 20 being entered).

The one or more processors 130 may obtain, when a signal for capturing the object corresponding to the guide is detected, the additional image through the camera 110, and store the capture image and the additional image as data on the object being entered or taken. Because the operation for providing the guide for obtaining the additional image and the operations following thereafter have been described above, redundant descriptions thereof may be omitted.

The one or more processors 130 may analyze a feature of the object being entered in or taken from the refrigerator 100 based on the capture image. For example, the capture image may be input to a feature analysis model and a feature of the object being entered/taken may be analyzed. However, the above is not limited thereto, and the feature analysis model may be included in various trained neural network models (e.g., feature extraction model).

The one or more processors 130 may obtain the information on classification of an object being entered or taken based on the feature of the analyzed object and store in the memory 120.

FIG. 5 is a diagram illustrating an operation for obtaining data on an object that is entered in or taken from the refrigerator according to at least one embodiment of the disclosure.

According to an embodiment, the refrigerator 100 may generate a plurality of same images for generating a guide image associated with the obtained capture image. Referring to FIG. 5, the refrigerator 100 may analyze a feature of an image 31 which captured a canned beverage, and generate a plurality of sample images 32-1, 32-2, 32-3, and 32-4 by inputting data 31′ obtained from analysis of the feature of the image 31 in a trained neural network model 500.

According to an embodiment, a sample image that does not correspond to the capture image may be generated as the guide image based on a similarity of the generated plurality of sample images with the capture image. Description for the above will be described in detail in FIG. 6.

According to an embodiment, the plurality of sample images may be generated based on data on a capture angle of an object being entered or taken. The refrigerator 100 may obtain an angle at which the object is entered in or taken out from an actual refrigerator 100 based on an image and an angle value, and generate an image of a different capture angle from the obtained capture angle as the plurality of sample images 32-1, 32-2, 32-3, and 32-4.

For example, if a capture angle of the image that captures the canned beverage is (15,0,50), a sample image 32-1 of a different capture angle (e.g., (34,20,50), etc.) from the capture angle of the capture image may be generated.

Here, a first number in the capture angle indicates a “pan angle” by which the camera is rotated left or right, and ‘15’ in the capture angle (15, 0, 50) may mean that the camera has rotated by 15 degrees from an original point to a right side.

A second number in the capture angle indicates a “tilt angle” by which the camera is rotated up and down, and ‘0 ’ in the capture angle (15, 0, 50) may mean that the camera has not rotated in a vertical direction.

A third number in the capture angle indicates a “zoom level” which is an extent by which the camera is zoomed in or zoomed out, and ‘50’ in the capture angle (15, 0, 50) may mean that the capture image is zoomed in by about 50%.

FIG. 6 is a diagram illustrating an operation for obtaining data on an object that is entered in or taken from the refrigerator according to at least one embodiment of the disclosure.

According to an embodiment, a sample image 32-3 having a high similarity with the data 31′ that is obtained from analysis of the feature of the capture image 31 of the canned beverage from among the plurality of sample images 32-1, 32-2, 32-3, and 32-4 may be removed from an additional capture viewpoint. That is, based on the similarity of the plurality of sample images with the capture image, a sample image that does not correspond to the capture image (e.g., a sample image having a low similarity with the data 31′) may be generated as the guide image.

For example, the one or more processors 130 may remove, based on the capture angle of the capture image being (15, 0, 50) by analyzing the image which captures the actual canned beverage, the sample image 32-3 having the capture angle of (15, 0, 50) from among the plurality of sample images from the guide image. The one or more processors 130 may generate sample images 32-1, 32-2, and 32-4 that do not have the capture angle of (15, 0, 50) as guide images 33-1, 33-2, and 33-4.

The refrigerator 100 may respectively compare values of capture angles of plurality of sample images 32-1, 32-2, 32-3, and 32-4 to identify similarities between the plurality of sample images 32-1, 32-2, 32-3, and 32-4 and the data 31′ obtained from analysis of the feature of the capture image.

For example, if the capture angle of the data 31′ obtained from analysis of the feature of the capture image is (15, 0, 50), a sample image most approximate to “pan angle 15, tilt angle 0, zoom level 50%” from among the sample images may be identified as having the highest similarity and removed from the guide image.

FIG. 7 is a diagram illustrating an operation for obtaining an image about an object that is entered in or taken from the refrigerator according to at least one embodiment of the disclosure.

According to an embodiment, the refrigerator 100 may divide a category of objects and make an additional request an image of an area required for each category of an object being entered/taken.

For example, the refrigerator 100 may request, when the object being entered/taken is detected as a canned beverage 40, an additional image capturing of a front surface 41-1, a lateral surface 41-2, and a back surface 41-3 as images of areas required in the category of canned beverages.

For example, if the front surface 41-1 of the canned beverage is captured by the user performing an operation of entering the canned beverage in the refrigerator 100, the refrigerator 100 may provide an audio (e.g., voice) such as “please also additionally capture the lateral surface 41-2, and the back surface 41-3” and/or a guide in an image form.

However, the embodiment is not limited thereto, and additional images of various surfaces such as a bottom surface may be requested according to the category of the object. For example, if chocolate with an expiration disclosed at a bottom surface is entered in the refrigerator 100 by the user, the refrigerator 100 may provide a voice such as “please also additionally capture the bottom surface” and/or a guide in an image form.

In another embodiment, if “orange” is entered in the refrigerator 100 by the user, the refrigerator 100 may identify that the food product being entered is an orange by using only the capture image in a specific one direction. The refrigerator 100 may not request the additional capturing to the user if the information on classification of an object being entered/taken is obtainable from the capture image.

The refrigerator 100 may be pre-set with images of areas that need to be captured for each category of an object as described above. However, the embodiment is not limited thereto, and the refrigerator 100 may analyze, in real time, the feature of the object being captured while being entered/taken and providing a guide requesting an image of a required area.

FIG. 8 is a diagram illustrating a UI providing a guide of the refrigerator according to at least one embodiment of the disclosure.

According to an embodiment, the refrigerator 100 may generate guide images 53-1, 53-2, 53-3, and 53-4 based on an obtained capture image. The user may register an additional capture image corresponding to each guide image by clicking a capture button 801. If a registration progress rate of the image that requires additional capturing is 45% (806), information on the registration progress rate may be provided to the user on the UI.

The refrigerator 100 may provide, based on identifying that a similarity of the additional capture image 51-1 corresponding to the guide image 53-1 with the guide image 53-1 is greater than or equal to a pre-set level, feedback 802 of being correctly captured according to the guide to the user.

The refrigerator 100 may use, in addition to the actually captured image, the guide image as a capture image 51-3 through a user input such as dragging. The refrigerator 100 may provide feedback 804 indicating the guide image being considered as captured, similar to the feedback 802 of being correctly captured according to the guide.

Conversely, the refrigerator 100 may provide, based on identifying that the similarity between a capture image 51-4 corresponding to the guide image 53-4 and the guide image 54-4, feedback 805 of requesting a re-capturing together with feedback of not being correctly captured according to the guide to the user.

The refrigerator 100 may provide, based on there being an image 51-2 that is not yet captured corresponding to the guide image 53-2, feedback 803 of not yet being captured to the user.

However, the above is merely one embodiment, and an additional capture image may be registered by another method and the progress rate thereof may be displayed. Examples of a display method by another method will be described in detail in FIG. 9.

FIG. 9 is a diagram illustrating a UI registering an additional image according to a guide of the refrigerator according to at least one embodiment of the disclosure.

Referring to FIG. 9, the refrigerator 100 may provide, if obtaining the information on classification of an object based on the capture image is insufficient, a guide such as “please capture by rotating the food product by degrees” (901) to the user. For example, the guide may indicate a certain degree at which the food product needs to be rotated.

The user may proceed with additional capturing by clicking a capture button 902 and register additional images 61-1, 61-2, and 61-4. The refrigerator 100 may provide, to the user, guide image 63-1, 63-2, 63-2, and 63-4 for additional capturing until additional registration is carried out to an extent that is sufficient for obtaining the information on classification of an object.

The refrigerator 100 may provide, to the user, a UI for the registration progress rate by taking into consideration an extent to which obtaining the information on classification of an object is sufficient and an extent of the additionally captured image.

For example, if there are four guide images for additional capturing, and there are three registered images which are captured that correspond to each guide image among the four guide images, the refrigerator 100 may display information that the registration progress rate is 75% (903) through, for example, a circular progress bar.

As described above, the refrigerator 100 may provide the user with various information associated with the image capturing and/or the progress rate using a progress bar that is easy to understand. However, the embodiment is not limited thereto, and the method for providing information on the registration progress rate may be variously modified.

The UI providing the guide of the refrigerator described in FIG. 8 and FIG. 9 may be displayed in the display 142 of the refrigerator 100. However, the embodiment is not limited thereto, and the refrigerator 100 may transmit information on the guide of the refrigerator 100 to an external device and have the information displayed on a display of the external device.

FIG. 10 is a diagram illustrating a UI providing data on an object that is entered in or taken from the refrigerator according to at least one embodiment of the disclosure.

According to an embodiment, the refrigerator 100 may store, when the capture image and additional images 71-1, 71-2, 71-3, and 71-4 are obtained by the user, the obtained images as data on the object being entered/taken and provide the user with the information on classification of the object.

Referring to FIG. 10, the information on classification of the object may include information on the product name of the object, the name of the food product, and the entered date ([Brand Name], [Food/Ingredient Name], [Entered date]). However, the embodiment is not limited thereto, and may include various information such as a taken date of the food product, the number of food products, or shopping information of the food product.

For example, the refrigerator 100 may obtain information on classification of the object 142′ including a product name “AA”, a product type “coffee”, and an entered date “today” of the object being entered/taken based on the capture image and the additional images 71-1, 71-2, 71-3, and 71-4 and display the information 142′ in the display 142.

In still another embodiment, the refrigerator 100 may delete information on the food product being taken from the UI that displays the information on classification of the object if the food product with the product name of “AA” and product type of “coffee” is identified as having been taken. However, if two or more same food products are entered, the information on classification of the object may not be deleted and a number of food products remaining in the refrigerator 100 may be updated and displayed.

However, the above is merely one embodiment, and the information on classification of the object may be provided in an audio form through the speaker 141. Here, the information on classification of the object may include at least one from among information on the name of the object, the type of the object, the number of the object, and the entered or taken time of the object, but is not limited thereto.

FIG. 11 is a diagram illustrating a UI providing data on an object that is entered in or taken from the refrigerator according to at least one embodiment of the disclosure.

According to an embodiment, the refrigerator 100 may display, in the display 142, a list of capture images in a form of a function that shows an entered/taken history or a user history together therewith and provide the same to the user. The user may use the function of showing the entered/taken history or the user history together therewith and directly register similar capture images as information on a specific food product.

For example, if images 142-1 captured at 01:40 AM and 01:39 AM are “AA Coffee”, the user may select the images 142-1 captured at 01:40 AM and 01:39 AM and register the selected images as image data about “AA Coffee”.

However, the above is merely one embodiment, and the refrigerator 100 may automatically identify similar images 142-1 and display a UI requesting whether to register the above in the food product DB to the user.

In FIG. 10, although only the UI providing the information on classification of the object has been shown, but the embodiment is not limited thereto, and the refrigerator 100 may display a UI that includes notification information on the object being entered/taken.

For example, the refrigerator 100 may analyze, based on the information on the object being entered or taken, an entering or taking pattern of the object being entered or taken, and output the notification information on the object being entered or taken based on the analyzed entering or taking pattern through the speaker 141 and/or the display 142.

Here, the “notification information on the object being entered or taken” may mean various notification information associated with the object being entered/taken. For example, the notification information on the object being entered/taken may include a re-purchase notification on the object, a notification on an object similar with the object, a consumption notification on the object, and a recipe notification information on the object. However, the embodiment is not limited thereto, and may include various information useful to the user based on the pattern of the object being entered or taken.

The UI providing data on the object being entered/taken described in FIG. 10 and FIG. 11 may be displayed in the display 142 of the refrigerator 100. However, the above is not limited thereto, and the refrigerator 100 may transmit the data on the object being entered/taken to an external device and display the data on the object being entered/taken in a display of the external device.

FIG. 12 is a diagram illustrating an operation for extracting features of an object that is entered in or taken from the refrigerator according to at least one embodiment of the disclosure.

According to an embodiment, the refrigerator 100 may input the capture image in a trained neural network model (e.g., an image conversion model) and extract the feature of the object based on the output image.

For example, if the capture image is an image obtained from an external device, the refrigerator 100 may input (1210) the capture image in the image conversion model and output (1220) as an image which is converted to an image having quality of a pre-set level (e.g., a level of a specification supported by the camera of the refrigerator) based on the quality of the image being greater than or equal to the pre-set level. The refrigerator 100 may extract (1230) the feature of the object based on the converted image. An illustrative example of obtaining the capture image from the external device will be described in detail in FIG. 13.

Here, the “quality” of the image may include image quality, brightness, feature, and the like of the image. Accordingly, the refrigerator 100 may convert an image of high image quality to an image of low image quality and then, extract the feature information on the object based on the converted image of low image quality.

FIG. 13 is a diagram illustrating an example of a refrigerator receiving data from various electronic devices according to at least one embodiment of the disclosure.

According to an embodiment, the refrigerator 100 may perform communication with various external devices such as the user terminal device and the server device through the communication device 150. For example, the refrigerator 100 may receive an image captured from a user terminal device 200 and use the received image as the capture image.

The refrigerator 100 may perform communication directly with the user terminal device 200 through the communication device 150, but the above is merely one embodiment, and may perform communication with an external user terminal through a server.

However, as described in FIG. 12, if the received image is identified as having quality greater than or equal to a pre-set level, the refrigerator 100 may use the received image in a form that is converted to the image having quality of low image quality.

In another embodiment, the refrigerator 100 may receive data on the object from a server device 300. For example, the refrigerator 100 may receive a food product image of an affiliated company and information on the product name from the server device 300. Accordingly, the refrigerator 100 may automatically register information on the food product in the user food product DB based on the information received from the server 300.

FIG. 14 is a flowchart illustrating an operation for managing data on an object that is entered in or taken from the refrigerator according to at least one embodiment of the disclosure.

The refrigerator 100 may obtain, based on an object being entered in or taken from the refrigerator 100, a capture image by capturing the object (S1410). The refrigerator 100 may capture the object by detecting movement of the object being entered/taken.

The refrigerator 100 may provide, based on obtaining the capture image about the object being entered in or taken from the refrigerator, the guide for obtaining an additional image based on the obtained capture image (S1420). Because an example method(s) of providing the guide based on the obtained capture image has been described above, redundant descriptions thereof will be omitted.

The refrigerator 100 may obtain the additional image when the signal for capturing the object corresponding to the provided guide is detected (S1430). The refrigerator 100 may proceed with additional capturing by detecting the movement of the object being entered/taken. The refrigerator 100 may register the image obtained through additional capturing as the additional image.

The refrigerator 100 may store the capture image and the additional image as data on the object being entered or taken (S1440). The operation for storing the capture image and the additional image as data on the object being entered/taken by the refrigerator 100 may be included in the operation for registering as the additional image.

The refrigerator 100 may analyze the feature of the object based on the data on the object being entered in/taken out from the refrigerator 100, and obtain the information on classification of an object based on the analyzed feature of the object.

FIG. 15 is a flowchart illustrating an operation for managing data on an object that is entered in or taken from the refrigerator according to at least one embodiment of the disclosure.

The refrigerator 100 may, based on not being able to obtain the information on classification of an object, compare data (e.g., pre-stored data) on the object being entered or taken with the capture image and identify a similarity therebetween (S1510). The refrigerator 100 may identify the similarity by comparing various feature information such as the form, the angle, and the like of the object included in the image.

The refrigerator 100 may select data on the object being entered or taken having a similarity with the capture image that is greater than or equal to the pre-set level from among the data on the objects being entered or taken (S1520). The refrigerator 100 may select data on the object being entered/taken having a high similarity level and identify the selected data as data on the object that is actually entered/taken.

The refrigerator 100 may obtain the information on classification of an object based on the selected data on the object being entered or taken (S1530).

As described above, by obtaining the information on classification of an object being entered in/taken from the refrigerator 100, the user may be able to conveniently manage the food products stored in the refrigerator 100. Ultimately, user experience may be enhanced.

An order of the flowcharts of the various embodiments described above are merely at least one embodiment, and the order of each step in the flowcharts may be changed, and the order in the flowcharts may be performed simultaneously.

According to an embodiment of the disclosure, the one or more processors 130 may control to process input data according to a pre-defined operation rule or an artificial intelligence model stored in the memory 120. The pre-defined operation rule or the artificial intelligence model may be characterized by being created through learning.

Here, being created through learning may mean a pre-defined operation rule and/or an artificial intelligence model of a desired characteristic being created by applying a learning algorithm to a plurality of training data. The learning may be carried out in a device itself in which the artificial intelligence according to the disclosure is performed, or carried out through a separate server/system.

The artificial intelligence model (e.g., a first object detecting network and a second object detecting network), may be configured with a plurality of neural network layers. At least one layer may have at least one weight value, and perform a layer operation through an operation result of a previous layer and at least one defined operation. Examples of the neural network may include a Convolutional Neural Network (CNN), a Deep Neural Network (DNN), a Recurrent Neural Network (RNN), a Restricted Boltzmann Machine (RBM), a Deep Belief Network (DBN), a Bidirectional Recurrent Deep Neural Network (BRDNN), a Deep-Q Networks, and a Transformer, and the neural network in the disclosure may not be limited to the above-described examples except for when specified.

The learning algorithm may be a method for training a predetermined target device to make a decision and/or prediction on its own using the plurality of training data. Examples of the learning algorithm may include a supervised learning, an unsupervised learning, a semi-supervised learning, or a reinforcement learning, and the learning algorithm of the disclosure is not limited to the above-described examples unless otherwise specified.

The control method described in FIG. 14 and FIG. 15 may be performed by a device having various configurations such as that described in FIG. 2 and/or FIG. 3 above, but is not necessarily limited thereto, and may be performed by a device having various configurations.

The various embodiments described in the above may be implemented in an embodiment individually, or the at least one embodiment may be combined as a whole or partially with one another and implemented together in one device.

The various embodiments of the disclosure may be implemented as software stored in a machine-readable storage media which may be mounted in or connected to smartphones or user terminal devices and other various electronic devices (e.g., a computer).

Specifically, a non-transitory computer-readable storage medium may be provided, the medium being stored with software for performing, based on an object being entered in or taken from the refrigerator, capturing of the object, providing, based on obtaining the capture image about the object being entered in or taken from the refrigerator, the guide for obtaining an additional image based on the obtained capture image, obtaining the additional image based on the signal for capturing the object corresponding to the provided guide being detected, and storing the capture image and the additional image as data on the object being entered or taken.

The device mounted with the non-transitory computer-readable storage medium may perform various operations such as identifying a tag corresponding to a user operation, checking importance for each of a plurality of frames, generating edited image, and the like described in the above-described various embodiments.

The term ‘non-transitory’ in the non-transitory computer-readable storage medium merely means that the storage medium is tangible and does not include a signal only, and the term does not differentiate data being semi-permanently stored or being temporarily stored in the storage medium.

Alternatively, a program for performing the method(s) according to the various embodiments described above may be distributed on-line through an application store. In the case of online distribution, at least a portion of a computer program product may be stored at least temporarily in the storage medium such as a server of a manufacturer, a server of an application store, or a memory of a relay server, or temporarily generated.

Each of the elements (e.g., a module or a program) according to various embodiments may be configured as a single entity or a plurality of entities, and a portion of sub-elements of the above-mentioned sub-elements may be omitted, or other sub-elements may be further included in the various embodiments. Alternatively or additionally, a portion of the elements (e.g., modules or programs) may be integrated into one entity to perform the same or similar functions performed by the respective elements prior to integration.

Operations performed by a module, a program, or another element, in accordance with various embodiments, may be executed sequentially, in a parallel, repetitively, or in a heuristic manner, or at least a portion of the operations may be executed in a different order, omitted or a different operation may be added.

While the disclosure has been illustrated and described with reference to example embodiments thereof, it will be understood that the example embodiments are intended to be illustrative, not limiting. It will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the true spirit and full scope of the disclosure, including the appended claims and their equivalents.