Robot Vacuum Automation Cleaner

Description

BACKGROUND

Robot vacuums are deployed to premises to relieve users from cleaning chores. Robot vacuums may be instructed to perform clean tasks of vacuuming and/or mopping waste in an entire or a section of a premise.

SUMMARY

The following summary presents a simplified summary of certain features. The summary is not an extensive overview and is not intended to identify key or critical elements.

Camera(s) from a premise may capture video feeds and may send them to a computer device for analysis and cleaning tasks. The computer device (e.g., video analyzer) may use a machine learning model to classify target objects from the video feeds and may generate cleaning tasks to clean up the target objects. The computer device may select one of a plurality of robot vacuums based on a criteria and may send the cleaning tasks to the selected robot vacuum. This process may efficiently target particular waste objects and may be implemented target particular waste objects in an automated manner.

These and other features and advantages are described in greater detail below.

BRIEF DESCRIPTION OF THE DRAWINGS

Some features are shown by way of example, and not by limitation, in the accompanying drawings. In the drawings, like numerals reference similar elements.

FIG. 1 shows an example communication network.

FIG. 2A, FIG. 2B, FIG. 2C and FIG. 2D show hardware elements of a computing device.

FIG. 3A shows an example system of video analyzer.

FIG. 3B shows an example of a cleaning task list.

FIG. 4 shows a flowchart showing an example of generating and maintaining cleaning tasks.

FIG. 5 shows a flowchart showing an example of pre-processing video feed.

FIG. 6 shows a flowchart showing an example of applying an AI model.

FIG. 7 shows a flowchart showing an example of generating cleaning tasks

FIG. 8 shows a flowchart showing an example of generating a map and routes for cleaning tasks

FIG. 9 shows a flowchart showing an example of tracking object movement and associated behavior.

FIG. 10 shows a flowchart showing an example of performing cleaning tasks.

DETAILED DESCRIPTION

The accompanying drawings, which form a part hereof, show examples of the disclosure. It is to be understood that the examples shown in the drawings and/or discussed herein are non-exclusive and that there are other examples of how the disclosure may be practiced.

FIG. 1 shows an example communication network 100 in which features described herein may be implemented. The communication network 100 may comprise one or more information distribution networks of any type, such as, without limitation, a telephone network, a wireless network (e.g., an LTE network, a 5G network, a WiFi IEEE 802.11 network, a WiMAX network, a satellite network, and/or any other network for wireless communication), an optical fiber network, a coaxial cable network, and/or a hybrid fiber/coax distribution network. The communication network 100 may use a series of interconnected communication links 101 (e.g., coaxial cables, optical fibers, wireless links, etc.) to connect multiple premises 102 (e.g., businesses, homes, consumer dwellings, offices, shopping malls, train stations, airports, etc.) to a local office 103 (e.g., a headend). The local office 103 may send downstream information signals and receive upstream information signals via the communication links 101. Each of the premises 102 may comprise devices, described below, to receive, send, and/or otherwise process those signals and information contained therein.

Throughout this disclosure, the “target object” may be referred to as “task object”.

The communication links 101 may originate from the local office 103 and may comprise components not shown, such as splitters, filters, amplifiers, etc., to help convey signals clearly. The communication links 101 may be coupled to one or more wireless access points 127 configured to communicate with one or more mobile devices 125 via one or more wireless networks. The mobile devices 125 may comprise smart phones, tablets or laptop computers with wireless transceivers, tablets or laptop computers communicatively coupled to other devices with wireless transceivers, and/or any other type of device configured to communicate via a wireless network.

The local office 103 may comprise an interface 104. The interface 104 may comprise one or more computing devices configured to send information downstream to, and to receive information upstream from, devices communicating with the local office 103 via the communications links 101. The interface 104 may be configured to manage communications among those devices, to manage communications between those devices and backend devices such as push server 105, content server 106, application server 107, model database server 122 and/or video analyzer 123 to manage communications between those devices and one or more external networks 109. The interface 104 may, for example, comprise one or more routers, one or more base stations, one or more optical line terminals (OLTs), one or more termination systems (e.g., a modular cable modem termination system (M-CMTS) or an integrated cable modem termination system (I-CMTS)), one or more digital subscriber line access modules (DSLAMs), and/or any other computing device(s). The local office 103 may comprise one or more network interfaces 108 that comprise circuitry needed to communicate via the external networks 109. The external networks 109 may comprise networks of Internet devices, telephone networks, wireless networks, wired networks, fiber optic networks, and/or any other desired network. The local office 103 may also or alternatively communicate with the mobile devices 125 via the interface 108 and one or more of the external networks 109, e.g., via one or more of the wireless access points 127.

The push server 105 may be configured to generate push notifications to deliver information to devices in the premises 102 and/or to the mobile devices 125. The content server 106 may be configured to provide content to devices in the premises 102 and/or to the mobile devices 125. This content may comprise, for example, video, audio, text, web pages, images, files, etc. The content server 106 (or, alternatively, an authentication server) may comprise software to validate user identities and entitlements, to locate and retrieve requested content, and/or to initiate delivery (e.g., streaming) of the content. The application server 107 may be configured to offer any desired service. For example, an application server may be responsible for collecting, and generating a download of, information for electronic program guide listings. Another application server may be responsible for monitoring user viewing habits and collecting information from that monitoring for use in selecting advertisements. Yet another application server may be responsible for formatting and inserting advertisements in a video stream being transmitted to devices in the premises 102 and/or to the mobile devices 125. The local office 103 may comprise additional servers and devices, such as model database server 122, video analyzer 123, additional push, content, and/or application servers, and/or other types of servers. The model database 122 may be configured to store and provide image model(s) of objects. The video analyzer 123 may analyze camera feed for objects (e.g., waste objects and/or non-waste objects) and instruct one or more robot vacuums (e.g., robot vacuums 121) for cleaning tasks (e.g., vacuuming the waste objects). The video analyzer may reside in different locations, such as local office 103 (video analyzer 123 as shown in FIG. 1), premises 102a (e.g., video analyzer 120 in FIG. 1) or inside robot vacuums 121 (e.g., video analyzer 221 in FIG. 2C). Although shown separately, the push server 105, the content server 106, the application server 107, the model database server 122, the video analyzer 123, and/or other server(s) may be combined. The servers 105-107, model database server 122, video analyzer 123 and/or other servers may be computing devices and may comprise memory storing data and also storing computer executable instructions that, when executed by one or more processors, cause the server(s) to perform steps described herein.

An example premises 102a may comprise an interface 120. The interface 120 may comprise circuitry used to communicate via the communication links 101. The interface 120 may comprise a modem 110, which may comprise transmitters and receivers used to communicate via the communication links 101 with the local office 103. The modem 110 may comprise, for example, a coaxial cable modem (for coaxial cable lines of the communication links 101), a fiber interface node (for fiber optic lines of the communication links 101), twisted-pair telephone modem, a wireless transceiver, and/or any other desired modem device. One modem is shown in FIG. 1, but a plurality of modems operating in parallel may be implemented within the interface 120. The interface 120 may comprise a gateway 111. The modem 110 may be connected to, or be a part of, the gateway 111. The gateway 111 may be a computing device that communicates with the modem(s) 110 to allow one or more other devices in the premises 102a to communicate with the local office 103 and/or with other devices beyond the local office 103 (e.g., via the local office 103 and the external network(s) 109). The gateway 111 may comprise a set-top box (STB), digital video recorder (DVR), a digital transport adapter (DTA), a computer server, and/or any other desired computing device.

The gateway 111 may also comprise one or more local network interfaces to communicate, via one or more local networks, with devices in the premises 102a. Such devices may comprise, e.g., display devices 112 (e.g., televisions, smart TVs), other devices 113 (e.g., a DVR or STB), personal computers 114, laptop computers 115, wireless devices 116 (e.g., wireless routers, wireless laptops, notebooks, tablets and netbooks, cordless phones (e.g., Digital Enhanced Cordless Telephone—DECT phones), mobile phones, mobile televisions, personal digital assistants (PDA), virtual reality devices), landline phones 117 (e.g., Voice over Internet Protocol—VoIP phones) and any other desired devices. Example types of local networks comprise Multimedia Over Coax Alliance (MoCA) networks, Ethernet networks, networks communicating via Universal Serial Bus (USB) interfaces, wireless networks (e.g., IEEE 802.11, IEEE 802.15, Bluetooth), networks communicating via in-premises power lines, and others. The lines connecting the interface 120 with the other devices in the premises 102a may represent wired or wireless connections, as may be appropriate for the type of local network used. One or more of the devices at the premises 102a may be configured to provide wireless communications channels (e.g., IEEE 802.11 channels) to communicate with one or more of the mobile devices 125, which may be on- or off-premises.

An example premises 102a may comprise camera(s) 118, monitor 119, video analyzer 120 and robot vacuum(s) 121. The camera(s) 118 may comprise one or more cameras lens and may capture video feed(s) in any resolution and/or frame rate. Each camera 118 may provide a surround view of a premise. The camera(s) 118 may send the captured video feed to the video analyzer 120/122/221 for analysis. The camera(s) 118 may be installed at fixed locations (e.g., permanently secured at ceiling, indoor/outdoor wall) and/or may be relocated by users (e.g., moving the camera from a dining table to a kitchen countertop). The camera(s) 118 may be standalone cameras or part of devices (e.g., a camera of a security system keypad, a camera of a smart TV). The monitor 119 is configured to provide status of the video analyzer 120/122/221 and robot vacuum(s) 121 to the users. The robot vacuum(s) 121 may be configured to receive instructions from the video analyzer 120/122/221 and perform cleaning tasks. The robot vacuum(s) 121 may operate in any premise, where the premise may be indoor or outdoor. The robot vacuum(s) 121 may report task status to the monitor 119.

The mobile devices 125, one or more of the devices in the premises 102a, and/or other devices may receive, store, output, and/or otherwise use assets. An asset may comprise a video, a game, one or more images, software, audio, text, webpage(s), and/or other content.

FIG. 2A shows hardware elements of a computing device 200 that may be used to implement any of the computing devices shown in FIG. 1 (e.g., the mobile devices 125, any of the devices shown in the premises 102a, any of the devices shown in the local office 103, any of the wireless access points 127, any devices with the external network 109) and any other computing devices discussed herein. The computing device 200 may comprise one or more processors 201, which may execute instructions of a computer program to perform any of the functions described herein. The instructions may be stored in a non-rewritable memory 202 such as a read-only memory (ROM), a rewritable memory 203 such as random access memory (RAM) and/or flash memory, removable media 204 (e.g., a USB drive, a compact disk (CD), a digital versatile disk (DVD)), and/or in any other type of computer-readable storage medium or memory. Instructions may also be stored in an attached (or internal) hard drive 205 or other types of storage media. The computing device 200 may comprise one or more output devices, such as a display device 206 (e.g., an external television and/or other external or internal display device) and a speaker 214, and may comprise one or more output device controllers 207, such as a video processor or a controller for an infra-red or BLUETOOTH transceiver. One or more user input devices 208 may comprise a remote control, a keyboard, a mouse, a touch screen (which may be integrated with the display device 206), microphone, camera, etc. The computing device 200 may also comprise one or more network interfaces, such as a network input/output (I/O) interface 210 (e.g., a network card) to communicate with an external network 209. The network I/O interface 210 may be a wired interface (e.g., electrical, RF (via coax), optical (via fiber)), a wireless interface, or a combination of the two. The network I/O interface 210 may comprise a modem configured to communicate via the external network 209. The external network 209 may comprise the communication links 101 discussed above, the external network 109, an in-home network, a network provider’s wireless, coaxial, fiber, or hybrid fiber/coaxial distribution system (e.g., a DOCSIS network), or any other desired network. The computing device 200 may comprise a location-detecting device, such as a global positioning system (GPS) microprocessor 211, which may be configured to receive and process global positioning signals and determine, with possible assistance from an external server and antenna, a geographic position of the computing device 200.

FIG. 2B shows hardware elements of a computing device 220, which is similar to the computing device 200 with the addition of at least one of video analyzer 221 and/or model database server 222. The video analyzer 221 may be software executed by the processor 201, and may have the same functionality as the video analyzer 123. The model database server 222 may be software executed by the processor 201, and may have the same functionality as the model database server 122.

FIG. 2C shows hardware elements of a computing device 240, which is similar to the computing device 200 with the addition of at least one of camera(s) 241, cleaning agent module 242, mopping module 243, vacuum module 244, mobility module 247, product containers 248 and/or mop 249. The computing device 240 may be used to implement robot vacuum(s) 121 shown in FIG. 1. The camera(s) 241 may comprise one or more cameras., where multiple cameras may provide a surround view for the robot vacuum avoiding obstacles in an area. The cleaning agent module 242 may be software executed by the processor 201. Each of the robot vacuum(s) 121 may comprise one or more product containers 248 for cleaning products (e.g., carpet stain cleaner, antibiotics, grease remover). Each of the product containers 248 may comprise a sensor to monitor the volume of the cleaning products. The cleaning agent module 242 may control releasing the cleaning products to a target cleaning area associated with the target object(s). The type of cleaning products to be released may depend on the target object(s) and/or target cleaning area. For example, carpet stain cleaner may be released to clean a carpet. The cleaning agent module 242 may send a warning message to user if the cleaning products of the one or more container(s) are running low or empty. Each of the robot vacuum(s) 121 may comprise a mop 249 for wet floor cleaning. The mop 249 may comprise a motor and a mop pad, where the motor may be configured to move or spin the mop pad in a motion (e.g., up-down, sideway or circular motion) and/or apply pressure to the mop pad for cleaning tasks. The mop 249 may be coupled with the product containers 248 (not shown in FIG. 2C) via a tube so that the cleaning products can be transferred from the product containers 248 to the mop 249. The mop pad may be made of microfiber, cotton and/or other synthetic blend material. The mopping module 243 and the mop 249 may perform mopping actions. The mopping module 243 may be software executed by the processor 201, and is configured to dampen a mop pad of the robot vacuum(s) 121 with the cleaning products and instruct the motor applying pressure to the mop pad for effective cleaning. The mobility module 247 may comprise at least one of motor(s), wheels or other means to provide mobility for the robot vacuum(s) 121. The mobility module 247 may be used so that the computing device 240 may travel to locations of the target object(s) according to routes (e.g., route 330).

As shown in FIG. 2D, the vacuum module 244 may perform vacuuming actions. The vacuum module 244 may comprise hardware and software, and may comprise controller 251, vacuum motor 252, airflow path 253, waste storage 254 and item storage 255. The controller 251 may be software executed by the processor 201, and may be configured to control the vacuum motor 252 and the airflow path 253. The controller 251 may be configured to receive status from the vacuum motor 252, the airflow path 253, the waste storage 254 and item storage 255. The vacuum motor 252 may be any type of electric motor, and may provide suction to draw target objects (such as dust, debris, and/or other small objects (e.g., toy blocks)) in the robot vacuum. The target objects may pass through the airflow path 253 to the waste storage 254 and item storage 255. The controller 251 may adjust the suction power by controlling power of the vacuum motor 252. The vacuum motor 252 may provide status information (e.g., on/off, rotations per minute, temperature, error message) to the controller 251. The airflow path 253 may comprise adjustable paths coupled to the waste storage 254 and the item storage 255 (not shown in FIG. 2D). The controller 251 may adjust the airflow path 253 based on the target objects so that the target objects may be collected at the waste storage 254 and the item storage 255 respectively. For example, the controller may adjust the airflow path 253 so that the airflow path 253 may lead the dust and/or debris into the waste storage 254. The controller may adjust the airflow path 253 so that the airflow path 253 may lead the small objects into the item storage 255. The airflow path 253 may comprise airflow sensor(s), and may report airflow rates (e.g., airflow rate associated with each respective path to the waste storage 254 or item storage 255) to the controller 251. Each of the waste storage 254 and item storage 255 may comprise a sensor to monitor the capacity of the corresponding storage.

Although FIG. 2A, FIG. 2B and FIG. 2C show example hardware configurations, one or more of the elements of the computing device 200 or 220 may be implemented as software or a combination of hardware and software. Modifications may be made to add, remove, combine, divide, etc. components of the computing device 200 or 220. Additionally, the elements shown in FIG. 2A, FIG. 2B and FIG. 2C may be implemented using basic computing devices and components that have been configured to perform operations such as are described herein. For example, a memory of the computing device 200 or 220 may store computer-executable instructions that, when executed by the processor 201 and/or one or more other processors of the computing device 200, 220 or 240, cause the computing device 200, 220 or 240 to perform one, some, or all of the operations described herein. Such memory and processor(s) may also or alternatively be implemented through one or more Integrated Circuits (ICs). An IC may be, for example, a microprocessor that accesses programming instructions or other data stored in a ROM and/or hardwired into the IC. For example, an IC may comprise an Application Specific Integrated Circuit (ASIC) having gates and/or other logic dedicated to the calculations and other operations described herein. An IC may perform some operations based on execution of programming instructions read from ROM or RAM, with other operations hardwired into gates or other logic. Further, an IC may be configured to output image data to a display buffer.

FIG. 3 shows example elements of video analyzer 300 that may be used to implement video analyzer 120/123 in FIG. 1 and the video analyzer 221 in FIG. 2B and FIG. 2C. The video analyzer 300 may comprise pre-processing module 302, artificial intelligence (AI) model inference module 304, task module 306, tracking & analysis module 308, map location module 310, signaling module 312 and cleaning task list 314. The video analyzer 300 may automatically classify target objects and generate cleaning tasks to clean up the target objects. No human intervention may be required.

The pre-processing module 302 may be configured to receive video feed(s) from camera(s) 118, and prepare the video feed(s) for analysis. The pre-processing module 302 may conduct as least one of: decoding video feed(s), resizing video feed(s), adjusting bitrates, normalizing frame rates and/or decomposing video feed(s) into image frame(s). The pre-processing module 302 may decode the video feed(s) so that the video feed(s) are in a compatible format for further pre-processing. The pre-processing module 302 may resize the video feed(s) to a pre-determined resolution, where the pre-determined resolution may be smaller than the resolution of the video feed(s). The pre-processing module 302 may adjust bitrates of video feed(s) to a pre-determined bitrate, where the pre-determined bitrate may be smaller than the bitrates of the video feed(s). The pre-processing module 302 may normalize the frame rates of the video feed(s) to a pre-determined frame rate, where the pre-determined frame rate may be smaller than the frame rates of the video feed(s). The pre-processed video feed(s) (e.g., video feed(s) after resizing resolution, adjusting bitrates and/or normalizing frame rates) may reduce the processing load by the video analyzer 120/123/221. The pre-processing module 302 may decompose the pre-processed video feed(s) into image frame(s) for further analysis.

The AI model inference module 304 may be configured to apply a trained AI model to infer information from the decomposed frame(s). The AI model inference module 304 may use an object image model to identify one or more objects in the video feed(s). The AI model inference module 304 may use the one or more images frames decomposed from the pre-processing module 302. For each image frame, the AI model inference module may generate region proposals (e.g., candidate bounding boxes) associated with the one or more objects, where the region proposals may contain edges of the one or more objects. The AI model inference module 304 may further extract features from each region proposal using a deep convolutional neural network. The features may include color, texture, size, dimension, and so on. Relevant features may have a correlation associated to the object image model. The AI model inference module 304 may, based on the object image model, classify features as one of the known classes, wherein the correlation may exceed a threshold (e.g., 90%). The object image model may be accessed from a model database server (e.g., model database 123/222). The known class may be an object or a person. The AI model inference module 304 may determine, based on the classified features associated with the region proposal, the position of the object or the person. The position may be pixel indices and/or spatial coordinates (e.g., cartesian coordinates). The determined objects may be target objects or non-target objects. The AI model inference module 304 may classify multiple objects as target objects and determine positions of the corresponding target objects. The target objects may be waste objects (e.g., dust, debris, small objects, liquid, excrements, other semi solid objects, etc.) and non-waste object (e.g., toy blocks). The target objects are subject to be picked up by the robot vacuum. Non-target objects may be objects not to be picked up by the robot vacuum, such as floors, walls, furniture, lamps, televisions, etc.

The task module 306 may schedule cleaning tasks based on the target objects classified from the AI model inference module 304. For example, the task module 306 may schedule a cleaning task of vacuuming a target object to the waste storage 254 if the target object is classified as solid waste object. The task module 306 may schedule a cleaning task of vacuuming a target object to the item storage 255 if the target object is classified as solid non-waste object. The size of the solid waste object or the solid non-waste object may be less than a threshold size (e.g., 2ft x 2ft) so that the robot vacuum(s) 121 can handle the size. The task module 306 may schedule a cleaning task of releasing (e.g., spraying) the cleaning product(s) to a target cleaning area associated with a target object if the target object is liquid. The cleaning task may further comprise mopping the target cleaning area after a pre-determined period of time (e.g., 1 minute). The clean tasks may be stored in cleaning task list 314.

As shown in FIG. 3B, the cleaning task list 314 shows an example list of 5 cleaning tasks. The cleaning task list 314 may comprise properties, such as task ID 322, task 324, task object 326, location 328, route 330, robot ID 332, status 334 and/or priority 336. The task ID 322 may indicate an identifier of each task. The task 324 may indicate a description of each task. The target object 326 indicates the determined target object, such as waste objects (e.g., dust, debris, small objects, liquid, etc.) and non-waste object (e.g., toy blocks). The location 328 may comprise 2-D, 3-D and/or polar coordinate of the target object. The location 328 may indicate a location of the target object. The route 330 may indicate a route object guiding the robot vacuum traveling from the current robot vacuum location (e.g., charging base) to the target object location. The robot ID 332 indicates an identifier of the robot vacuum selected for the cleaning task. Cleaning tasks may be assigned a priority (not shown in FIG. 3B) if multiple cleaning tasks are scheduled to a particular robot vacuum. The robot vacuum may perform the cleaning tasks according to the priority, where the cleaning task with the highest priority may be performed first and the cleaning task with the lowest priority may be performed the last. The status 334 may indicates the various status of the robot vacuum. For example, an “active” status may indicate an ongoing cleaning task. A “completed” status may indicate a finished cleaning task. A “suspend” status may indicate the cleaning task is suspended, which may be due to an interaction between the target object and human. A “resume” status may indicate a previously suspended cleaning task (e.g., cleaning task with a “suspend” status) is resumed. The status 334 is not limited to “active”, “completed”, “suspend” or “resume”. For example, the status 334 may indicate “error” for a problem occurred at the robot vacuum. The priority 336 may indicate a priority of cleaning tasks assigned to a particular robot vacuum. A priority with a lower value may indicate a higher priority than another priority with a higher value. In the example of FIG. 3B, task1 has a priority with the a value of 1 and may have a higher priority than task3 (task3 having a priority with a value of 3). In another case, a higher value of the priority may indicate a higher priority than a lower value of the priority.

The tracking & analysis module 308 may be configured to track movement of objects and robot vacuums across frames and analyzing object interactions over time. The tracking & analysis module 308 may determine and track locations of one or more object(s) and robot vacuums. The one or more object(s) may be non-target object(s). The tracking & analysis module 308 may determine that the non-target object(s) are blocking (e.g., intersecting) a current route for the robot vacuum, where the current route may direct the robot vacuum to the location of the target object(s). The tracking & analysis module 308 may notify map location module 310 for an updated route. The track & analysis module 308 may update the new location of the target object to the cleaning task list 314. Human may interact the target object(s). For example, a cleaning task of removing a toy block may be scheduled for the robot vacuum. A child may be interacting with the target object (e.g., playing with a toy block by lifting the toy block from a floor to a coffee table) after the scheduling. The video analyzer 300 may suspend the cleaning task if: i) the child is within a first threshold distance (e.g., 1 ft) from the target object, ii) the target object is located above the floor by a second threshold distance (e.g., 8 inches), and/or iii) the target object is located on the top of another non-waste object (e.g., coffee table). After a pre-determined period, the video analyzer 300 may resume the cleaning task if: i) the child is no longer within the first threshold distance from the target object, ii) the target object is no longer located above the floor by the second threshold distance (e.g., located on the floor), and/or iii) the target object is no longer located on the top of another non-waste object.

The map location module 310 may generate a map of the premise and routes for the robot vacuum(s). The map may comprise a floor plan with features such as doors, stairways, ramps, corridors, etc. The map location module 310 may use the pre-processed video feed(s) to generate the map of the premise. The pre-processed video feed(s) from 302 may be used to generate the map of the premise. For example, the map of the premise (e.g., premise 102a) may comprise a plurality of cameras 118, robot vacuums 121, target objects, non-target objects, and/or rooms (not shown in FIG. 1). The map may be generated to indicate the multiple rooms. A precision location may be determined by GPS signals, Wi-Fi positioning or Bluetooth beacons. For example, the locations of the cameras may be determined by Wi-Fi positioning, and the locations of the robot vacuums and target/non-target objects may be determined from the positions (from tracking & analysis module 308) and relative to the locations of the cameras. The map location module 310 may determine conditions of the robot vacuums 121, including calculating a distance between each of the robot vacuums 121 and the each of target objects, and/or determining a battery life, a capacity of the product container(s), a capacity of the waste storage and/or item storage for each of the robot vacuums 121. Distance, between each of the robot vacuums and the target object with the highest priority, may be calculated based on the locations of each robot vacuum and the target object (from the map of the premise). A query may be sent to each of the robot vacuums 121 for requesting the following conditions, including battery life, capacity of the product container(s), and/or capacity of the waste storage and/or item storage. A response message may be returned from each of the robot vacuums 121 to report the requested conditions. The robot vacuums 121 may be located throughout the premise (e.g., in different room). The map location module 310 may select one of the robot vacuums 121 for cleaning up target object(s). The selection may be based on at least one of the following factors:

shortest distance: the robot vacuum closest to the target object with the highest priority;

most battery life: the robot vacuum with the most battery capacity;

capacity of the product container(s): the robot vacuum with the most available product solutions;

capacity of the waste storage and item storage: the robot vacuum with the most capacity of the waste storage and/or item storage.

The map location module 310 may generate a route for the selected robot vacuum. The map location module 310 may avoid obstacles based on the classified object(s) (e.g., furniture and walls) and generate a shortest or fastest route from the location of the selected robot vacuum to the target object. The map location module 310 may store, according to the cleaning task, the route to the cleaning task list.

The signaling module 312 may send cleaning tasks from the cleaning task list 314 to the robot vacuum(s) 121. The signaling module 312 may send the updated location of the target object(s) to the robot vacuum(s) 121 if the target object(s) have been relocated. The signaling module 312 may notify the robot vacuum(s) 121 to suspend the cleaning tasks. The signaling module 312 may notify the robot vacuum(s) 121 to resume previously suspended cleaning task(s).

FIG. 4 is a flowchart showing an example method 400 of analyzing camera feed for objects and instructing one or more robot vacuums for cleaning tasks, according to embodiments of the disclosure. According to various embodiments, the example method may be implemented by the video analyzer 120/123/221 shown in FIG. 1 and FIG. 2B. At 402, video feed(s) may be received from camera(s) 118. At 404, the received video feed(s) may be pre-processed for analysis. The pre-processing the video feed(s) may comprise decoding video feed(s) 502, resizing video feed(s) 504, adjusting bitrates 506 and/or normalizing frame rates 508 as shown in FIG. 5, where steps 502, 504, 506 and 508 may be conducted in series or in parallel. The video feed(s) may be decoded so that the video feed(s) are in a compatible format for further pre-processing. The video feed(s) at 502 may resize to a pre-determined resolution, where the pre-determined resolution may be smaller than the resolution of the video feed(s). Bitrates of the video feed(s) at 504 may be adjusted to a pre-determined bitrate, where the pre-determined bitrate may be smaller than the bitrates of the video feed(s). The frame rates of the video feed(s) at 506 may be normalized to a pre-determined frame rate, where the pre-determined frame rate may be smaller than the frame rates of the video feed(s). The pre-processed video feed(s) (e.g., video feed(s) after resizing resolution, adjusting bitrates and/or normalizing frame rates) may reduce the processing load. The pre-processed video feed(s) at 510 may be decomposed into image frame(s) for further analysis.

At 406, a trained AI model is applied to infer information from the decomposed frames. An object image model may be used to identify one or more objects in the video feed(s). The one or more images frames decomposed at 510 may be used. For each image frame, region proposals (e.g., candidate bounding boxes) may be generated associated with the one or more objects at 602 (as shown in FIG. 6), where the region proposals may contain edges of the one or more objects. Features may be computed at 604 comprises extracting the features from each region proposal by using a deep convolutional neural network. The features may include color, texture, size, dimension, and so on. Relevant features may have a correlation associated to the object image model. At 606, the features may be, based on the object image model, classified as one of the known classes, wherein the correlation may exceed a threshold (e.g., 90%). The object image model may be accessed from a model database server (e.g., model database 123/222). The known class may be an object or a person. The position of the object or the person may be determined based on the classified features associated with the region proposal. The position may be pixel indices and/or spatial coordinates (e.g., cartesian coordinates). The determined objects may be target objects or non-target objects. The target objects may be waste objects (e.g., dust, debris, small objects, liquid, etc.) and non-waste object (e.g., toy blocks). The target objects are subject to be picked up by the robot vacuum. Non-target objects may be objects not to be picked up by the robot vacuum, such as furniture, lamps, televisions, etc.

At 408, cleaning tasks may be scheduled based on the target objects classified at 406. The step at 702 (shown in FIG. 7) may determine if the determined objects are target objects. The method can proceed to 704 if the determined objects are target objects to be cleaned up by the robot vacuum. The method may end if the determined objects are not the target objects. The step 704 may determine if the target objects are waste objects. The method can proceed to 706 if the target objects are waste objects. A cleaning task may be scheduled to clean up the non-waste objects at 708 (e.g., vacuuming the non-waste objects to the item storage 255) if the target objects are non-waste objects. The step 706 may determine if the waste objects are solid. A cleaning task may be scheduled to clean up the solid waste objects at 710 (e.g., vacuuming the waste objects to the waste storage 254) if the target objects are solid waste objects. A cleaning task may be scheduled to clean up the liquid waste objects at 712 if the waste objects are liquid. The cleaning task may comprise releasing (e.g., spraying) the cleaning product(s) to a target cleaning area associated with the liquid waste object(s). One or more types of cleaning products may be released based on the liquid waste object and/or the target cleaning area. For example, carpet stain remover may be released for carpet as the target cleaning area. Antibiotic cleaner may be released for blood or urine as the liquid waste object. The cleaning task may further comprise mopping the target cleaning area after a pre-determined period of time. The scheduled cleaning tasks may be stored in cleaning task list 314.

At 410, a map of the premise and routes for the robot vacuum(s) may be generated. The step 802 (as shown in FIG. 8) may determine if at least one active cleaning task does not include a route. The method may end if each active cleaning task already comprises a route. The method may proceed to 804 if at least one cleaning task does not include a route. The step 804 may determine if a map of the premise is generated. The map of the premise may comprise a floor plan with features such as doors, stairways, ramps, corridors, etc. The map of the premise may be generated at 806 if the map does not exist. The pre-processed video feed(s) at 404 may be used to generate the map of the premise. For example, the map of the premise (e.g., premise 102a) may comprise a plurality of cameras 118, robot vacuums 121, target objects, non-target objects, and/or rooms (not shown in FIG. 1). The map may be generated to indicate the multiple rooms. A precision location may be determined by GPS signals, Wi-Fi positioning or Bluetooth beacons. For example, the locations of the cameras may be determined by Wi-Fi positioning, and the locations of the robot vacuums and target/non-target objects may be determined from the positions in step 406 and relative to the locations of the cameras. At 808, conditions of the robot vacuums 121 may be determined, including calculating a distance between each of the robot vacuums 121 and the target object with the highest priority, and/or determining a battery life, a capacity of the product container(s), a capacity of the waste storage and/or item storage for each of the robot vacuums 121. Distance, between each of the robot vacuums and the target object with the highest priority, may be calculated based on the locations of each robot vacuum and the target object with the highest priority (from the map of the premise). A query may be sent to each of the robot vacuums 121 for requesting the following conditions, including battery life, capacity of the product container(s), and/or capacity of the waste storage and/or item storage. A response message may be returned from each of the robot vacuums 121 to report the requested conditions. At 810, one of the robot vacuums 121 may be selected for cleaning up target object(s). The robot vacuums 121 may be located throughout the premise (e.g., in different room). The selection may be based on at least one of the following factors:

shortest distance: the robot vacuum closest to the target object with the highest priority;

most battery life: the robot vacuum with the most battery capacity;

capacity of the product container(s): the robot vacuum with the most available product solutions;

capacity of the waste storage and item storage: the robot vacuum with the most capacity of the waste storage and/or item storage.

At 812, a route may be generated for the selected robot vacuum. Obstacles may be avoided based on the classified object(s). For example, the route may avoid classified furniture and wall(s). A shortest or fastest route may be generated from the location of the selected robot vacuum to the next target object. The route, according to the cleaning task, may be stored to the cleaning task list.

In the case of multiple target objects, a route may be determined by distances. For example, a first distance between each of the multiple target objects and the selected robot vacuum may be determined. One of the multiple target objects (e.g., target object 1) with the shortest first distance may be selected for a first part of the route, where the first part of the route may direct the vacuum robot to move from the vacuum robot’s current location to the location of the target object 1 for cleaning. A second distance between each of the multiple target objects (excluding target object 1) and the target object 1 may be determined. One of the multiple target objects (e.g., target object 2) with the shortest second distance may be selected for a second part of the route, where the second part of the route may direct the vacuum robot to move from the location of the target object 1 to the location of the target object 2 for cleaning. This process may continue until all target objects are selected for the route. A target object with a higher priority may be selected if the first or second distances are identical. For example, target object A and target object B may be located away from the robot vacuum (or another selected target object) by the same distance. The target object A may be selected if the target object A has a higher priority than the target object B.

In the case of multiple target objects, a route may be determined by priority of the target objects. For example, one of the multiple target objects (e.g., target object 1) with the highest priority may be selected for a first part of the route, where the first part of the route may direct the vacuum robot to move from the vacuum robot’s current location to the location of the target object 1 for cleaning. One of the multiple target objects (e.g., target object 2) with the second highest priority may be selected for a second part of the route, where the second part of the route may direct the vacuum robot to move from the location of the target object 1 to the location of the target object 2 for cleaning. This process may continue until all target objects are selected for the route. A target object with a shorter relative distance may be selected if the priorities of the target objects are identical. For example, target object A and target object B may have the same priority. The target object A may be selected if the target object A is located closer (e.g., shorter relative distance) to the robot vacuum (or another selected target object) than the target object B.

At 412, movement of objects across decomposed image frames (from 510) may be tracked and object interactions may be analyzed. The step 902 (as shown in FIG. 9) may determine whether the classified object is a target or non-target object. The method may proceed to 904 if the classified object is a non-target object. At 904, the method may determine one the non-target object is blocking (e.g., intersecting) a current route for the robot vacuum. A new route may be calculated if the non-target object is blocking the current route. The new route may be updated to the cleaning tasking list 314. Back to step 902, the method may proceed to 906 if the classified object is a target object. The step 906 may determine if the target object is relocated by comparing the current location against the location 328 indicated in the cleaning task list 314. A new route may be calculated if the target object is relocated. The new route and target object location may be updated to the cleaning task list 314. The step 908 may determine human interaction with the target object if a human is located within a threshold distance (e.g., 1ft) from the target object. The status 334 for the cleaning the cleaning task may be updated at 910 if the human interacts with the target object. For example, a toy block may be initially classified as a target object for the robot vacuum. A child may be interacting with the target object (e.g., a toy block). The cleaning task may be suspended by changing the status 334 as “suspended” if i) the child is within a first threshold distance (e.g., 1ft) from the target object, ii) the target object is located above the floor by a second threshold distance (e.g., 8 inches), and/or iii) the target object is located on the top of another non-waste object (e.g., coffee table). The cleaning task may be reactivated by changing the status 334 as “active” if i) the child is no longer within the first threshold distance from the target object, ii) the target object is no longer located above the floor by the second threshold distance (e.g., located on the floor), and/or iii) the target object is no longer located on the top of another non-waste object. The step 912 may determine if there is additional classified object for analysis. The method may return to 902 if there is additional classified object for analysis. The method may end if there is no additional classified object.

At 414, a determination may be made regarding whether there is new and/or updated cleaning task(s) to be sent to the corresponding robot vacuum(s) 121. A signal of the new and/or updated cleaning task(s) may be sent to the robot vacuum(s) 121 at 416 if there are new and/or updated cleaning task(s). For example, the cleaning tasks with “active” and “suspended” status (e.g., task2 and task3 and task4 in FIG. 3B) may be sent to the robot vacuum(s) 121. The signal may be a broadcast or unicast communication. The broadcast communication may comprise one message broadcasting all robot vacuum(s) 121, where the message may comprise all “active” and “suspended” cleaning tasks with task ID 322, task 324, task object 326, location 328, route 330, robot ID 332, status 334 and/or priority 336 as shown in FIG. 3B. The unicast communication may comprise multiple messages, where each of the multiple messages is sent to a particular robot vacuum and each of the multiple messages may comprise the “active” and “suspended” cleaning task(s) of the particular robot vacuum with task ID 322, task 324, task object 326, location 328, route 330, robot ID 332, status 334 and/or priority 336. The method may return to 402 for receiving new video feeds from the camera(s).

FIG. 10 is a flowchart showing an example method 1000 of running clean tasks according to embodiments of the disclosure. According to various embodiments, the example method may be implemented by the robot vacuum(s) 121 in FIG. 1.

At 1002, a signal of new and/or updated cleaning task(s) may be received. For example, the signal of new and/or updated cleaning task(s) may be received from video analyzer 120/123/221. The new and/or updated cleaning task(s) may comprise properties, such as task ID 322, task 324, task object 326, location 328, route 330, robot ID 332, status 334 and/or priority 336 as shown in FIG. 3B. Each of the cleaning task(s) may have a priority, where the priority may be assigned by the video analyzer 120/123/221. New cleaning task(s) may be saved, and properties of existing cleaning task(s) may, based on based on task ID, be modified by updated cleaning task(s). For example, an existing cleaning task may comprise properties of task ID (task2) and a status (“active”). An updated task may be received, wherein the updated task may comprise properties of task ID (task2) and a status (“suspended”). The properties of the existing cleaning task may be modified by the received updated cleaning task so that the status of the existing cleaning task may be changed to “suspended”.

The step 1004 may determine whether a cleaning task is available. The cleaning task with the highest priority may be selected if multiple cleaning tasks are scheduled. Based on the route 330, step 1005 may perform traveling to the location of the task object 326. For example, a robot vacuum, using mobility module 247 and based on the route 330, may travel to the location of the task object 326.

The step 1006 may determine whether the task object 326 is a liquid type object. A mopping action 1008 may be performed if the task object 326 is a liquid type object. For example, mopping module 243 and mop 249 (shown in FIG. 2C) may perform the mopping action.

Step 1010 may be performed if the task object 326 is not a liquid type object. The step 1010 may determine whether the task object 326 is a solid type object. A vacuuming action 1012 may be performed if the task object 326 is a solid type object. For example, vacuum module 244 (shown in FIG. 2C and 2D) may perform the vacuuming action. The solid type object may be collected in separate containers, based on the task object 326. Dust and/or debris may be collected into the waste storage 254. Small objects (e.g., toy blocks) may be collected into the item storage 255.

Step 1014 may be performed if the task object 326 is not a solid type object (e.g., ketchup, grease, feces, other semi-solid objects) . The step 1014 may determine whether the task object 326 requires cleaner. Action(s) of spraying cleaner, mopping and/or vacuuming may be performed if the task object 326 requires cleaner. For example, cleaning agent module 242 (as shown in FIG. 2C) may perform spraying cleaner action, mopping module 243 and mop 249 (shown in FIG. 2C) may perform the mopping action and/or vacuum module 244 (shown in FIG. 2C and 2D) may perform the vacuuming action. One or more types of cleaner may be released based on the task object. For example, grease remover may be released for ketchup or grease. Disinfectant cleaner may be released for feces. Based on task 324, mopping may be performed to wipe the combination of task object and cleaner and/or vacuuming action(s) may be performed to pick up loose particles.

Step 1018 may, based on the action status of step 1008, step 1012 or step1016, update the status 334. For example, the status 334 may change from “active” to “completed” if the step 1008, step 1012 or step 1016 is successfully completed. The status 334 may change from “active” to “error” if the step 1008, step 1012 or step 1016 is not successfully completed (e.g., jamming occurring at vacuum module 244). Further error description may be reported to the status 334. At 1020, a signal of the updated task may be sent. For example, the updated task may be sent to the video analyzer 120/123/221. The method may return to step 1002 for receiving the new/updated cleaning task(s).

Although examples are described above, features and/or steps of those examples may be combined, divided, omitted, rearranged, revised, and/or augmented in any desired manner. Various alterations, modifications, and improvements will readily occur to those skilled in the art. Such alterations, modifications, and improvements are intended to be part of this description, though not expressly stated herein, and are intended to be within the spirit and scope of the disclosure. Accordingly, the foregoing description is by way of example only, and is not limiting.