WORK MACHINE, INFORMATION PROCESSING DEVICE, AND COMPUTER READABLE MEDIUM

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application claims priority to Japanese Patent Application No. 2023-200126, filed Nov. 27, 2023, and International Patent Application No. PCT/JP2024/041424, filed Nov. 22, 2024, the entire content of each of which is incorporated herein by reference.

BACKGROUND

1. Technical Field

The present invention relates to a work machine and the like.

2. Description of Related Art

Work machines, such as excavators, operated by operators or operated autonomously have been known.

SUMMARY

According to one embodiment of the present disclosure, there is provided a work machine, which includes a processing device including a processor and a memory coupled with the processor, wherein the processing device is configured to acquire data that represents a state of the work machine and relates to a shape of a work target, in association with a motion of the work machine, and is configured to estimate the shape of the work target after the motion of the work machine based on the data acquired. In addition, according to another embodiment of the present disclosure, there is provided an information processing device, which includes a processor and a memory coupled with the processor, wherein the information processing device is configured to acquire data that represents a state of a work machine and relates to a shape of a work target, in association with a motion of the work machine, and is configured to estimate the shape of the work target after the motion of the work machine based on the data acquired.

Further, according to yet another embodiment of the present disclosure, there is provided a computer readable medium storing a program that causes an information processing device to execute a process that includes acquiring data that represents a state of a work machine and relates to a shape of a work target, in association with a motion of the work machine, and estimating the shape of the work target after the motion of the work machine based on the data acquired.

Furthermore, according to yet another embodiment of the present disclosure, there is provided a computer readable medium storing a program that causes a support device to execute a process that includes acquiring data that represents a state of a work machine and relates to a shape of a work target, in association with a motion of the work machine, estimating the shape of the work target after the motion of the work machine based on the data acquired, and causing a display to display the shape of the work target based on a result of the estimation of the shape of the work target.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating one example of an operation support system;

FIG. 2 is a top view illustrating one example of an excavator;

FIG. 3 is a diagram illustrating one example of a configuration related to remote operation of the excavator;

FIG. 4 is a diagram illustrating one example of a hardware configuration of the excavator;

FIG. 5 is a diagram illustrating one example of a hardware configuration of the information processing device;

FIG. 6 is a functional block diagram illustrating a first example of a functional configuration of the operation support system;

FIG. 7 is a functional block diagram illustrating a second example of the functional configuration of the operation support system;

FIG. 8 is a functional block diagram illustrating a third example of the functional configuration of the operation support system;

FIG. 9 is a diagram explaining a method of estimating a shape of a work target;

FIG. 10 is a diagram illustrating one example of measurement data and estimation data of a reaction force from the work target to the working part (bucket) during an excavating motion of the excavator;

FIG. 11 is a diagram illustrating another example of the measurement data and the estimation data of the reaction force from the work target to the working part (bucket) during an excavating motion of the excavator;

FIG. 12 is a diagram illustrating yet another example of the measurement data and the estimation data of the reaction force from the work target to the working part (bucket) during an excavating motion of the excavator;

FIG. 13 is a diagram illustrating one example of a relationship between certainty of the data representing the shape of the work target and a predetermined operation of the excavator;

FIG. 14 is a diagram illustrating another example of the relationship between the certainty of the data representing the shape of the work target and the predetermined operation of the excavator;

FIG. 15 is a diagram illustrating a first example of a process related to use of the data representing the shape of the work target;

FIG. 16 is a diagram illustrating a second example of the process related to use of the data representing the shape of the work target;

FIG. 17 is a diagram illustrating a third example of the process related to use of the data representing the shape of the work target;

FIG. 18 is a diagram illustrating a fourth example of the process related to use of the data representing the shape of the work target;

FIG. 19 is a flowchart schematically illustrating a first example of a process for acquiring the data representing the shape of the work target;

FIG. 20 is a diagram illustrating one example of an observation target region;

FIG. 21 is a diagram illustrating one example of an influence region;

FIG. 22 is a flowchart schematically illustrating a second example of the process for acquiring the data representing the shape of the work target; and

FIG. 23 is a flowchart schematically illustrating a third example of the process for acquiring the data representing the shape of the work target.

DETAILED DESCRIPTION

In the related art, for example, an operator who boards an excavator may have a blind spot on a part of the ground of a work target due to unevenness of soil, an attachment of the excavator, or the like. In addition, when the excavator is remotely operated, an operator needs to perform operation while viewing an image captured by a camera mounted on the excavator, and it may be difficult to grasp the shape of the ground of the work target depending on the state of the image of the camera. Therefore, the shape of the work target may be recognized based on the data from the camera or a distance measuring sensor (distance sensor) or the like, and the shape of the work target may be displayed on a display mounted on a cabin of the work machine or a display for remote controlling.

Further, for example, when autonomous operation of the excavator is performed, a trajectory of a working part (bucket) may be determined according to the shape of the soil on the ground of the work target. In this case, the shape of the work target is recognized based on the data of the camera, the distance measuring sensor, or the like, and the trajectory of the working part of the work machine is determined based on the shape of the work target.

However, for example, when the camera, the distance measuring sensor, or the like, which is mounted on the excavator, is used, occlusion of the image, where a part of the shape of the work target is blocked by the attachment of the excavator or unevenness of the soil and cannot be measured, may occur. Therefore, for example, there may be a hole in a part of the shape of the work target displayed on the display, which may cause an undesirable situation from the viewpoint of the operation support for the operator. In addition, for example, since the shape of the part of the work target cannot be measured, the certainty in the trajectory of the working part for autonomous operation may decrease. Further, for example, when the distance measuring sensor is not mounted in the excavator, a function of displaying the shape of the work target on the display or an autonomous driving function of the work target itself cannot be employed.

Therefore, in view of the related art above, it is desirable to provide a technology capable of grasping a shape of a work target of a work machine.

Embodiments of the present disclosure will be described with reference to drawings hereinafter.

[Overview of Operation Support System]

An overview of an operation support system SYS according to the present embodiment will be described with reference to FIGS. 1 to 3.

FIG. 1 is a diagram illustrating one example of the operation support system SYS. In FIG. 1, a left side view of an excavator 100 is illustrated. FIG. 2 is a top view illustrating one example of the excavator 100. FIG. 3 is a diagram illustrating one example of a configuration related to remote operation of the excavator 100. Hereinafter, a direction in the excavator 100 or a direction as viewed from the excavator 100 may be described by determining a direction in which the attachment AT extends (an upward direction in FIG. 2) as viewed in the top view of the excavator 100 as “front”.

As illustrated in FIG. 1, the operation support system SYS includes the excavator 100, an information processing device 200, and a sensor group 300.

The operation support system SYS cooperates with the excavator 100 using the information processing device 200 to support operation of the excavator 100. The operation support system SYS may include one excavator 100, or a plurality of excavators 100.

The excavator 100 is a work machine that is an operation-related support target in the operation support system SYS.

As illustrated in FIGS. 1 and 2, the excavator 100 includes a lower traveling body 1, an upper turning body 3, an attachment AT including a boom 4, an arm 5, and a bucket 6, and a cabin 10.

The lower traveling body 1 causes the excavator 100 to travel using a crawler 1C. The crawler 1C includes a left crawler 1CL and a right crawler 1CR. The crawler 1CL is hydraulically driven by a traveling hydraulic motor 1ML. Similarly, the crawler 1CR is hydraulically driven by a traveling hydraulic motor 1MR. This enables the lower traveling body 1 to travel by itself.

The upper turning body 3 is turnably (freely turnably) mounted on the lower traveling body 1 via a turning mechanism 2. For example, the upper turning body 3 turns with respect to the lower traveling body 1 by the turning mechanism 2 that is hydraulically driven by a turning hydraulic motor 2M.

The boom 4 is attached to the center of the front portion of the upper turning body 3 so as to be able to be raised and lowered around a rotation axis along the left-right direction. The arm 5 is attached to the distal end of the boom 4 so as to be rotatable around a rotation axis extending in the left-right direction. The bucket 6 is attached to the distal end of the arm 5 so as to be rotatable around a rotation axis extending in the left-right direction.

The bucket 6 is an example of an end attachment, and is used for example, for excavation work, slope protection work, land-leveling work, or the like.

The bucket 6 is attached to the distal end of the arm 5 so that the bucket can be replaced as appropriate in accordance with a type of operation of the excavator 100. Specifically, a bucket of a different type than the bucket 6, such as a relatively large bucket, a bucket for slopes, a bucket for dredging, or the like may be attached to the distal end of the arm 5 instead of the bucket 6. In addition, an end attachment other than the bucket, such as a stirrer, a breaker, a crusher, or the like, may be attached to the distal end of the arm 5. Further, for example, a spare attachment, such as a quick coupling, a tilt rotator, or the like, may be attached between the arm 5 and the end attachment.

The boom 4, the arm 5, and the bucket 6 are hydraulically driven by a boom cylinder 7, an arm cylinder 8, and a bucket cylinder 9, respectively.

The cabin 10 is a cockpit in which an operator boards and operates the excavator 100. The cabin 10 is mounted, for example, on the front left side of the upper turning body 3.

The excavator 100 is equipped with a communication device 60, and can communicate with the information processing device 200 via a predetermined communication line NW.

The communication line NW includes, for example, a local area network (LAN) of a work site. Moreover, the communication line NW may include a wide area network (WAN). The wide area network includes, for example, mobile communications networks having base stations as terminals, satellite communications networks using communication satellites, the Internet, and the like. In addition, the communication line NW may include, for example, a short-range communication line based on a wireless communication standard, such as WiFi, Bluetooth (registered trademark), and the like.

For example, the excavator 100 causes driven elements, such as the lower traveling body 1 (e.g., a pair of left and right crawlers 1CL and 1CR), the upper turning body 3, the boom 4, the arm 5, the bucket 6, and the like, to be operated in accordance with the operation of the operator who is in the cabin 10.

Moreover, the excavator 100 may be configured to be remotely operated (remote-controlled) from the outside of the excavator 100 instead of or in addition to being configured to be operable by the operator who is in the cabin 10. In the case where the excavator 100 is remotely operated, the inside of the cabin 10 may be in a state where no one is present. In addition, in the case where the excavator 100 is dedicated to remote operation, the cabin 10 may be omitted. Hereinafter, description will be made on the premise that the operation of the operator includes operation performed on an operation device 26 by the operator in the cabin 10, remote operation performed by an external operator, or both.

For example, as illustrated in FIG. 3, the remote operation includes an embodiment in which the excavator 100 is operated by an operation input related to an actuator of the excavator 100, which is performed by a remote operation support device 400 that can communicate with the excavator 100 via the communication line NW. The remote operation support device 400 may be provided separately from the information processing device 200, or may be the information processing device 200.

The remote operation support device 400 is installed, for example, in a management center or the like that manages the work of the excavator 100 from the outside. Moreover, the remote operation support device 400 may be a portable operation terminal. In this case, the operator can remotely operate the excavator 100 in the vicinity of the excavator 100, while directly checking the work status of the excavator 100 from the periphery of the excavator 100.

The excavator 100 may transmit, for example, an image representing the state of the surroundings including the area in front of the excavator 100 based on the captured image (referred to as a “peripheral image” hereinafter) output by an imaging device mounted on the excavator 100 to the remote operation support device 400 via a below-described communication device 60. Moreover, the excavator 100 may transmit the captured image output by the imaging device to the remote operation support device 400 via the communication device 60, and the remote operation support device 400 may process the captured image received from the excavator 100 to generate a peripheral image. Then, the remote operation support device 400 may display the peripheral image representing the state of the surroundings including the area in front of the excavator 100 on the display device thereof. Moreover, various information images (information screen images) displayed on the output device 50 (display device 50A) inside the cabin 10 of the excavator 100 may also be displayed on the display device of the remote operation support device 400. Thus, the operator using the remote operation support device 400 can remotely operate the excavator 100, while checking the display content displayed on the display device, such as the image representing the state of the surroundings of the excavator 100, information screen images, and the like. Then, the excavator 100 may cause the actuator to be operated in accordance with a remote operation signal representing the content of the remote operation received from the remote operation support device 400 via the communication device 60, thereby driving driven elements, such as the lower traveling body 1, the upper turning body 3, the boom 4, the arm 5, the bucket 6, and the like.

In addition, the remote operation may include an embodiment in which the excavator 100 is operated, for example, by a voice input, a gesture input, or the like of a person who is in the vicinity of the excavator 100 (e.g., a worker), which is externally input to the excavator 100. Specifically, the excavator 100 recognizes a voice uttered by a worker or the like in the vicinity or a gesture performed by the worker or the like through an audio input device (e.g., a microphone), a gesture input device (e.g., an imaging device), or the like mounted in the excavator 100. Then, the excavator 100 may cause the actuator to be operated in accordance with the content of the recognized voice, gesture, or the like, thereby driving the driven elements, such as the lower traveling body 1 (left and right crawlers 1C), the upper turning body 3, the boom 4, the arm 5, the bucket 6, and the like.

Moreover, the excavator 100 may cause the actuator to automatically be operated regardless of the content of the operation by the operator. Thus, the excavator 100 can realize a function of automatically operating at least a part of the drive elements, such as the lower traveling body 1, the upper turning body 3, the attachment AT, and the like, i.e., “automated driving function” or “machine control (MC) function”.

The automated driving function includes, for example, a semi-automated driving function (operation support-type MC function). The semi-automated driving function is a function of automatically operating driven elements (actuators) other than the driven element (actuator) to be operated in response to operation of an operator. Moreover, the automated driving function may include a fully automated driving function (fully automated MC function). The fully automated driving function is a function of automatically operating at least some of driven elements (hydraulic actuators) on the premise that there is no operation by an operator. When the fully automated driving function is enabled in the excavator 100, the inside of the cabin 10 may be in a state where no one is present. In addition, in the case where the excavator 100 is dedicated to fully automated driving, the cabin 10 may be omitted. Moreover, the semi-automated driving function, the fully automated driving function, or the like includes, for example, a rule-based automated driving function. The rule-based automated driving function is an embodiment of the automated driving function, in which content of operation of a driven element (actuator) that is a target of automated driving is automatically determined in accordance with a rule defined in advance. Moreover, the semi-automated driving function, the fully automated driving function, or the like may include an autonomous driving function. The autonomous driving function is an embodiment of the automated driving function, in which the excavator 100 autonomously makes various determinations, and content of the operation of a driven element (hydraulic actuator) that is a target of automated driving is determined according to the determination result.

Moreover, the work of the excavator 100 may be remotely monitored. In this case, a remote monitoring support device having the same function as the remote operation support device 400 may be provided. The remote monitoring support device is, for example, the information processing device 200. Thus, an observer who is a user of the remote monitoring support device can monitor the status of the work of the excavator 100 while checking the peripheral image displayed on the displayed device of the remote monitoring support device. Moreover, for example, when the observer determines that it is necessary from the viewpoint of safety, the observer can intervene in the operation performed by the operator of the excavator 100 or the automated driving, and urgently stop the excavator 100 by performing a predetermined input using the input device of the remote monitoring support device.

The information processing device 200 cooperates with the excavator 100 by communicating with the excavator 100, and provides support related to the operation of the excavator 100.

The information processing device 200 is, for example, a server device or a management terminal device installed in a management office at the work site of the excavator 100, or a management center or the like that is located at a place different from the work site of the excavator 100 and manages the operation status or the like of the excavator 100. The server device may be an on-premises server, a cloud server, or an edge server. The management terminal device may be, for example, a stationary terminal device, such as a desktop personal computer (PC) or the like, or a portable terminal device (portable terminal), such as a tablet terminal, a smartphone, a laptop PC, or the like. In the latter case, a worker of the work site, a supervisor who supervises the work, a manager who manages the work site, or the like can move within the work site while carrying the portable information processing device 200. In addition, in the latter case, an operator can bring, for example, the portable information processing device 200 into the cabin of the excavator 100.

The information processing device 200 acquires, for example, data related to the operation state from the excavator 100. Thus, the information processing device 200 can grasp the operation state of the excavator 100 and monitor the presence or absence of an anomaly of the excavator 100. Moreover, the information processing device 200 displays the data related to the operation state of the excavator 100, for example, on a below-described display device 208, thereby allowing a user to check the data. Moreover, the information processing device 200 can cause, for example, a learning model to learn the operation state of the excavator 100 to generate a learned model for supporting the operation of the excavator 100.

Further, the information processing device 200 may transmit a variety of data to the excavator 100, and the variety of data includes a program, reference data, and the like, used in the processing of the controller 30 of the excavator 100. Thus, the excavator 100 can perform a variety of processing related to the operation of the excavator 100 using the variety of data downloaded from the information processing device 200.

The sensor group 300 is disposed at a work site of the excavator 100.

For example, in the case where the operation support system SYS includes multiple excavators 100, the sensor group 300 is provided to each excavator 100. Moreover, in the case where multiple excavators 100 included in the operation support system SYS perform work at the same work site, one sensor group 300 may be shared by the multiple excavators 100.

The sensor group 300 includes sensors 300-1 to 300-M (M: an integer of 2 or more). The sensors 300-1 to 300-M measure a state of an object of the work site near the excavator 100, and acquire measurement data related to the state. The object of the work site includes a work target of the excavator 100. The work target is, for example, soil in a work region near the excavator 100. Moreover, the object of the work site includes, for example, a work machine in the vicinity of the excavator 100, such as another excavator, a bulldozer, or the like, and a work vehicle, such as a truck for transporting soil, in addition to the work target (soil in the work region) near the excavator 100. The state of the object includes a shape or characteristics of the object.

The sensors 300-1 to 300-M include, for example, distance measuring sensors (distance sensors). The distance measuring sensors include a light detecting and ranging (LiDAR) sensor, a millimeter-wave radar, an ultrasonic sensor, an infrared sensor, and the like. In addition, the sensors 300-1 to 300-M include, for example, a 3D camera capable of acquiring data related to a distance (depth) in addition to a two-dimensional image, such as stereo cameras, a time-of-flight (TOF), or the like. Further, the sensors 300-1 to 300-M may include a combination of the distance measuring sensor and the 3D camera. Thus, the sensor group 300 can acquire measurement data representing the shape of the object of the work site near the excavator 100. Hereinafter, a sensor capable of acquiring measurement data representing the shape of the object, such as the distance measuring sensor, the 3D camera, or the like may be referred to as a “geometry sensor” for convenience.

Moreover, the sensors 300-1 to 300-M may include a multi-wavelength spectral camera. The multi-wavelength spectral camera includes a multispectral camera, a hyperspectral camera, or the like. Thus, for example, the sensor group 300 can acquire measurement data representing characteristics of the object of the work site near the excavator 100, such as hardness, a moisture content, or the like of soil. Hereinafter, a sensor capable of acquiring measurement data representing characteristics of the object, such as a multi-wavelength spectral camera or the like may be referred to as a “physical sensor” for convenience.

For example, the sensors 300-1 to 300-M include multiple geometry sensors. Then, the multiple geometry sensors may be provided at different locations within the work site near the excavator 100 so that a sensing range of each of the multiple geometry sensors may overlap a sensing range of at least one of the other geometry sensors. Thus, for example, even in the case where occlusion occurs in measurement data of one geometry sensor and the measurement data representing a shape of an object cannot be obtained in a part of the sensing range, the measurement data representing the shape of the object in the same range may be acquired by another geometry sensor. Thus, the sensor group 300 can more reliably acquire the measurement data representing the shape of the object of the work site near the excavator 100.

Moreover, the sensors 300-1 to 300-M may include multiple physical sensors. Then, the multiple physical sensors may be provided at different locations within the work site near the excavator 100 so that a sensing range of each of the multiple physical sensors may overlaps a sensing range of at least one of the other physical sensors. Thus, for example, even in the case where occlusion occurs in the measurement data of one physical sensor and the measurement data representing characteristics of an object cannot be obtained in a part of the sensing range, the measurement data representing the characteristics of the object in the same range may be acquired by another geometry sensor. Thus, the sensor group 300 can more reliably acquire the measurement data representing the characteristics of the object of the work site near the excavator 100.

Further, the sensors 300-1 to 300-M may include a sensor having both the function of the geometry sensor and the function of the physical sensor (hereinafter, referred to as an “integrated sensor”). In this case, the sensors 300-1 to 300-M may include multiple integrated sensors. Then, the multiple integrated sensors may be provided at different locations within the work site near the excavator 100 so that a sensing region of each of the multiple integrated sensors overlaps a sensing region of at least one of the other integrated sensors.

Note that the sensor group 300 may simply include only one geometry sensor or physical sensor. Moreover, the operation support system SYS may simply include only one sensor capable of acquiring measurement data related to a state of an object of a work site near the excavator 100 instead of the sensor group 300.

The sensors 300-1 to 300-M may be fixed in the work site near the excavator 100, or may be mounted on a moving body that can move in the work site near the excavator 100. The moving body includes, for example, a work machine or a work vehicle that moves within the work site. Moreover, the moving body that can move within the work site may include, for example, a flying body, such as a drone that flies above the work site.

The outputs (measurement data) of the sensors 300-1 to 300-M are taken into the information processing device 200 via the communication line NW. The outputs of the sensors 300-1 to 300-M are, for example, directly taken into the information processing device 200 via the communication line NW. Moreover, the outputs of the sensors 300-1 to 300-M may be temporarily taken into the excavator 100 via the communication line NW, and then taken into the information processing device 200 via the excavator 100. In the case where the sensors 300-1 to 300-M are mounted on a predetermined device, such as the above-described moving body, the outputs of the sensors 300-1 to 300-M may be temporarily taken into the predetermined device, and then taken into the information processing device 200 via the device.

[Hardware Configuration of Operation Support System]

A hardware configuration of the operation support system SYS will be described with reference to FIG. 4 and FIG. 5, in addition to FIGS. 1 to 3.

A hardware configuration of the remote operation support device 400 may be the same as that of the information processing device 200. Thus, illustration as drawings and description of the hardware configuration of the remote operation support device 400 will be omitted.

<Hardware Configuration of Excavator>

FIG. 4 is a diagram illustrating one example of a hardware configuration of the excavator 100.

In FIG. 4, a path through which mechanical power is transmitted is indicated by a double line, a path through which high-pressure hydraulic fluid for driving the hydraulic actuator flows is indicated by a solid line, a path through which a pilot pressure is transmitted is indicated by a broken line, and a path through which an electric signal is transmitted is indicated by a dotted line.

The excavator 100 includes constituent components, such as a hydraulic drive system related to hydraulic driving of driven elements, an operation system related to operation of the driven elements, a user interface system related to exchange of information with a user, a communication system related to communication with the outside, a control system related to various controls, and the like.

<<Hydraulic Drive System>>

As illustrated in FIG. 4, the hydraulic drive system of the excavator 100 includes a hydraulic actuator HA that hydraulically drives each of the driven elements, such as the lower traveling body 1 (left and right crawler 1C), the upper turning body 3, the boom 4, the arm 5, the bucket 6, and the like, as described above. Moreover, the hydraulic drive system of the excavator 100 according to the present embodiment includes an engine 11, a regulator 13, a main pump 14, and a control valve 17.

The hydraulic actuator HA includes traveling hydraulic motors 1ML and 1MR, a turning hydraulic motor 2M, a boom cylinder 7, an arm cylinder 8, a bucket cylinder 9, and the like.

In the excavator 100, a part of or the entire hydraulic actuator HA may be replaced with an electric actuator. Specifically, the excavator 100 may be a hybrid excavator or an electric excavator.

The engine 11 is a prime mover of the excavator 100, and is a main power source in the hydraulic drive system. The engine 11 is, for example, a diesel engine that uses diesel oil as fuel. The engine 11 is mounted on, for example, a rear portion of the upper turning body 3. For example, the engine 11 rotates at a constant speed at a target rotational speed set in advance under direct or indirect control of a below-described controller 30, and drives the main pump 14 and a pilot pump 15.

Instead of or in addition to the engine 11, another prime mover (e.g., an electric motor) or the like may be mounted on the excavator 100.

The regulator 13 controls (adjusts) a discharge amount of the main pump 14 under the control of the controller 30. For example, the regulator 13 adjusts an angle of a swash plate of the main pump 14 (hereinafter, referred to as a “tilt angle”) in accordance with a control command from the controller 30.

The main pump 14 supplies a hydraulic fluid to the control valve 17 via a high-pressure hydraulic line. The main pump 14 is mounted on, for example, the rear portion of the upper turning body 3, similarly to the engine 11. The main pump 14 is driven by the engine 11 as described above. The main pump 14 is, for example, a variable displacement hydraulic pump. As described above, under the control of the controller 30, the tilt angle of the swash plate is adjusted by the regulator 13 to adjust the stroke length of the piston, thereby controlling a discharge flow rate and discharge pressure.

The control valve 17 drives the hydraulic actuator HA in accordance with the content of operation or remote operation performed on the operation device 26 by an operator, or the operation command corresponding to the automated driving function. The control valve 17 is mounted, for example, in the central portion of the upper turning body 3. The control valve 17 is connected to the main pump 14 via the high-pressure hydraulic line as described above, and selectively supplies the hydraulic fluid supplied from the main pump 14 to a respective hydraulic actuator in accordance with operation of the operator, or an operation command corresponding to the automated driving function. Specifically, the control valve 17 includes a plurality of control valves (direction switching valves) that control the flow rate and flow direction of the hydraulic fluid supplied from the main pump 14 to the respective hydraulic actuator HA.

<<Operation System>>

As illustrated in FIG. 4, the operation system of the excavator 100 includes a pilot pump 15, an operation device 26, a hydraulic control valve 31, a shuttle valve 32, and a hydraulic control valve 33.

The pilot pump 15 supplies a pilot pressure to various hydraulic devices via a pilot line 25. The pilot pump 15 is mounted, for example, in the rear portion of the upper turning body 3, similarly to the engine 11. The pilot pump 15 is, for example, a fixed displacement hydraulic pump, and is driven by the engine 11 as described above.

The pilot pump 15 may be omitted. In this case, after reducing, by a predetermined decompression valve, the pressure of the hydraulic fluid having relatively high pressure discharged from the main pump 14, the resultant hydraulic fluid having a relatively low pressure may be supplied to various hydraulic devices as a pilot pressure.

The operation device 26 is provided near the operator's seat in the cabin 10, and is used by an operator to operate various driven elements. Specifically, the operation device 26 is used by an operator to operate the hydraulic actuator HA to drive a respective driven element, and as a result, the operator can operate the driven element, which is a driving target, by the hydraulic actuator HA. The operation device 26 includes a pedal drive or a lever device for operating each of the driven elements (hydraulic actuator HA).

For example, as illustrated in FIG. 4, the operation device 26 is of a hydraulic pilot type. Specifically, the operation device 26 uses the hydraulic fluid supplied from the pilot pump 15 via the pilot line 25 and a pilot line 25A branched from the pilot line 25, and outputs a pilot pressure corresponding to the content of the operation to a pilot line 27A on the secondary side. The pilot line 27A is connected to one of the inlet ports of the shuttle valve 32, and is connected to the control valve 17 via a pilot line 27 connected to an outlet port of the shuttle valve 32. Thus, a pilot pressure corresponding to the content of operation related to various driven elements (hydraulic actuator HA) in the operation device 26 is input to the control valve 17 via the shuttle valve 32. Therefore, the control valve 17 can drive a respective hydraulic actuator HA in accordance with the content of operation performed on the operation device 26 by an operator or the like.

Moreover, the operation device 26 may be an electric type. In this case, the pilot line 27A, the shuttle valve 32, and the hydraulic control valve 33 are omitted. Specifically, the operation device 26 outputs an electric signal (hereinafter, referred to as an “operation signal”), and the operation signal is taken into the controller 30. Then, the controller 30 outputs a control command corresponding to the operation signal, i.e., a control signal corresponding to the content of the operation to be performed on the operation device 26, to the hydraulic control valve 31. As a result, a pilot pressure according to the content of the operation of the operation device 26 is input from the hydraulic control valve 31 to the control valve 17, and the control valve 17 can drive the respective hydraulic actuator HA in accordance with the content of the operation of the operation device 26.

Moreover, a control valve (direction switching valve) that is built in the control valve 17 and drive the hydraulic actuator HA may be an electromagnetic solenoid type. In this case, the operation signal output from the operation device 26 may be directly input to the control valve 17 (i.e., the electromagnetic solenoid type control valve).

As described above, a part of or the entire hydraulic actuator HA may be replaced with an electric actuator. In this case, the controller 30 may output a control command corresponding to the content of the operation of the operation device 26 or the content of remote control defined by the remote operation signal to the electric actuator or a driver or the like that drives the electric actuator. Moreover, in the case where the excavator 100 is remotely operated, the operation device 26 may be omitted.

The hydraulic control valve 31 is provided for each of the driven elements (hydraulic actuator HA) to be operated by the operation device 26, and for each of the driving directions (e.g., the upward direction and downward direction of the boom 4) of the driven element (hydraulic actuator HA). For example, two hydraulic control valves 31 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper turning body 3, the boom 4, the arm 5, the bucket 6, or the like. For example, the hydraulic control valve 31 is attached to the pilot line 25B between the pilot pump 15 and the control valve 17, and may be configured to be able to change the flow path area (e.g., a cross-sectional area through which the hydraulic fluid can flow). Thus, the hydraulic control valve 31 can output a predetermined pilot pressure to the pilot line 27B on the secondary side by using the hydraulic fluid of the pilot pump 15 supplied through the pilot line 25B. Therefore, the hydraulic control valve 31 can indirectly apply a predetermined pilot pressure corresponding to a control signal from the controller 30 to the control valve 17 through a shuttle valve 32 between the pilot line 27B and the pilot line 27. As a result, for example, the controller 30 can supply a pilot pressure according to the operation command corresponding to the automated driving function from the hydraulic control valve 31 to the control valve 17, thereby implementing a motion of the excavator 100 by the automated driving function.

Moreover, the controller 30 may control the hydraulic control valve 31 to realize remote control of the excavator 100. Specifically, the controller 30 outputs, to the hydraulic control valve 31, a control signal corresponding to the content of the remote operation designated by the remote operation signal received from the remote operation support device 400 by the communication device 60. Thus, the controller 30 can supply a pilot pressure corresponding to the content of the remote control from the hydraulic control valve 31 to the control valve 17, thereby implementing a motion of the excavator 100 based on the remote operation of the operator.

In the case where the operation device 26 is of an electric type, the controller 30 can directly supply a pilot pressure according to the content of operation (operation signal) of the operation device 26 from the hydraulic control valve 31 to the control valve 17, thereby implementing a motion of the excavator 100 based on the operation of the operator.

The shuttle valve 32 has two inlet ports and one outlet port, and outputs the hydraulic fluid having the higher pilot pressure, among the pilot pressures input from the two inlet ports, from the outlet port. Similarly to the hydraulic control valve 31, the shuttle valve 32 is provided for each of the driven elements (hydraulic actuator HA) to be operated by the operation device 26, and for each driving direction of the driven element (hydraulic actuator HA). For example, two shuttle valves 32 are provided for each double-acting hydraulic actuator HA for driving the lower traveling body 1, the upper turning body 3, the boom 4, the arm 5, the bucket 6, or the like. One of the two inlet ports of the shuttle valve 32 is connected to a pilot line 27A on the secondary side of the operation device 26 (specifically, the above-described lever device or pedal device included in the operation device 26), and the other inlet port is connected to a pilot line 27B on the secondary side of the hydraulic control valve 31. The outlet port of the shuttle valve 32 is connected to a pilot port of the control valve corresponding to the control valve 17 via the pilot line 27. The corresponding control valve is a control value that drives the hydraulic actuator that is an operation target of the above-described lever device or pedal device connected to one of the inlet ports of the shuttle valve 32. Thus, each of these shuttle valves 32 can apply the higher pilot pressure, among the pilot pressure of the pilot line 27A on the secondary side of the operation device 26 and the pilot pressure of the pilot line 27B on the secondary side of the hydraulic control valve 31, to the pilot port of the corresponding control valve. Specifically, the controller 30 can control the corresponding control valve by outputting the pilot pressure, which is higher than the pilot pressure on the secondary side of the operation device 26, from the hydraulic control valve 31, without relying on the operation of the operator to the operation device 26. Accordingly, the controller 30 can control the motions of the driven elements (the lower traveling body 1, the upper turning body 3, the boom 4, the arm 5, and the bucket 6) regardless of the operation state of the operator on the operation device 26, and can implement the automated driving function or the remote operation function.

The hydraulic control valve 33 is attached to the pilot line 27A connecting the operation device 26 and the shuttle valve 32. The hydraulic control valve 33 is, for example, configured to be able to change a flow path area of the hydraulic control valve 33. The hydraulic control valve 33 is operated in accordance with a control signal input from the controller 30. Therefore, in the case where the operation device 26 is operated by an operator, the controller 30 can forcibly reduce the pilot pressure output from the operation device 26. Thus, even when the operation device 26 is operated, the controller 30 can forcibly suppress or stop the operation of the hydraulic actuator HA corresponding to the operation of the operation device 26. Moreover, for example, even when the operation device 26 is operated, the controller 30 can reduce the pilot pressure output from the operation device 26 to lower the pilot pressure output from the hydraulic control valve 31. Thus, the controller 30 can assuredly apply the predetermined pilot pressure to the pilot port of the control valve inside the control valve 17 by controlling the hydraulic control valve 31 and the hydraulic control valve 33, for example, regardless of the content of the operation of the operation device 26. Accordingly, the controller 30 can more appropriately implement the automated driving function or remote operation function of the excavator 100, for example, by controlling the hydraulic control valve 33 in addition to the hydraulic control valve 31.

<<User Interface System>>

As illustrated in FIG. 4, the user interface system of the excavator 100 includes an operation device 26, an output device 50, and an input device 52.

The output device 50 outputs a variety of information to a user (e.g., an operator of the cabin 10 or an external operator of remote operation) of the excavator 100 or a person in the vicinity of the excavator 100 (e.g., a worker or a driver of a work vehicle), and the like.

For example, the output device 50 includes an illumination device, a display device 50A (see FIG. 7), or the like that outputs a variety of information in a visual manner. The illumination device is, for example, a warning lamp (indicator lamp) or the like. The display device 50A is, for example, a liquid crystal display, an organic electroluminescence (EL) display, or the like. For example, as illustrated in FIG. 2, the illumination device or the display device 50A may be provided inside the cabin 10 and output a variety of information to an operator or the like inside the cabin 10 in a visual manner. In addition, the illumination device or the display device 50A may be provided, for example, on a side surface or the like of the upper turning body 3, and output a variety of information to a worker or the like in the vicinity of the excavator 100 in a visual manner.

Moreover, the output device 50 may include an audio output device that output a variety of information in an auditory manner. The audio output device includes, for example, a buzzer, a speaker, or the like. The audio output device may be provided, for example, on an inside, or an outside, or both the inside and outside of the cabin 10, and may output a variety of information to an operator inside the cabin 10 or a person (a worker, etc.) in the vicinity of the excavator 100 in an auditory manner.

Further, the output device 50 may include a device that outputs a variety of information in a tactile manner, such as vibrations or the like of the operator's seat.

The input device 52 receives various inputs from a user of the excavator 100, and signals corresponding to the received inputs are taken into the controller 30. For example, as illustrated in FIG. 2, the input device 52 is provided inside the cabin 10, and receives an input from an operator or the like inside the cabin 10. In addition, the input device 52 may be provided, for example, on the side surface or the like of the upper turning body 3, and may receive an input from a worker or the like in the vicinity of the excavator 100.

For example, the input device 52 includes an operation input device that receives an input by mechanical operation from a user. The operation input device may include a touch panel mounted on the display device 50A, a touch pad provided in the periphery of the display device 50A, a button switch, a lever, a toggle, a knob switch provided on the operation device 26 (lever device), or the like.

Moreover, the input device 52 may include an audio input device that receives an audio input from a user. The audio input device includes, for example, a microphone.

Further, the input device 52 may include a gesture input device that receives a gesture input from a user. The gesture input device includes, for example, an imaging device capturing an image of a state of a gesture performed by a user.

Furthermore, the input device 52 may include a biometric input device that receives a biometric input from a user. The biometric input includes, for example, inputs of biometric information, such as fingerprints or an iris of a user.

<<Communication System>>

As illustrated in FIG. 4, the communication system of the excavator 100 according to the present embodiment includes a communication device 60.

The communication device 60 is connected to an external communication line NW, and communicates with a device provided separately from the excavator 100. The device provided separately from the excavator 100 may include a portable terminal device (mobile terminal) being brought into the cabin 10 by a user, as well as a device outside the excavator 100. The communication device 60 may include, for example, a mobile communication module that conforms to a standard, such as 4G (4th Generation), 5G (5^th Generation), or the like. Moreover, the communication device 60 may include, for example, a satellite communication module. Further, the communication device 60 may include, for example, a WiFi communication module, a Bluetooth (registered trademark) communication module, or the like. In the case where there are multiple connectable communication lines NW, the communication device 60 may include multiple communication devices according to the types of the multiple communication lines NW.

For example, the communication device 60 communicates with an external device, such as the information processing device 200, the remote operation support device 400, or the like outside the work site through a local communication line constructed at the work site. The local communication line is, for example, a mobile communication line by a local 5G (so-called local 5G) constructed at the work site or a local network by WiFi6.

Moreover, the communication device 60 may communicate with the information processing device 200, the remote operation support device 400, or the like through a wide communication line including the work site, i.e., a wide area network.

<<Control System>>

As illustrated in FIG. 4, the control system of the excavator 100 includes a controller 30. Moreover, the control system of the excavator 100 according to the embodiment of the present embodiment includes an operation pressure sensor 29, a sensor 40, and sensors S1 to S9.

The controller 30 performs various controls related to the excavator 100.

The function of the controller 30 may be implemented by any hardware, any combination of hardware and software, or the like. For example, as illustrated in FIG. 3, the controller 30 includes an auxiliary storage device 30A, a memory 30B, a central processing unit (CPU) 30C, and an interface device 30D, which are connected by a bus BS1.

The auxiliary storage device 30A is a nonvolatile storage unit, and stores a program to be installed and necessary files, data, and the like. The auxiliary storage device 30A is, for example, an electrically erasable programmable read-only memory (EEPROM), a flash memory, or the like.

For example, in the case where an instruction to start a program is issued, the memory 30B loads the program onto the auxiliary storage device 30A so that the CPU 30C can read the program. The memory 30B is, for example, a static random access memory (SRAM).

The CPU 30C executes, for example, the program loaded onto the memory 30B, and implements various functions of the controller 30 according to instructions of the program.

The interface device 30D functions as, for example, a communication interface for connecting to a communication line inside the excavator 100. The interface device 30D may include multiple different types of communication interfaces according to the types of the communication lines to be connected.

Moreover, the interface device 30D functions as an external interface for reading data from a recording medium and writing data to a recording medium. The recording medium is, for example, a dedicated tool connected by a cable detachably connected to a connector installed inside the cabin 10. Moreover, the recording medium may be, for example, a general-purpose recording medium, such as an SD memory card, a universal serial bus (USB) memory, or the like. Thus, the program that implements various functions of the controller 30 may be provided by, for example, a portable recording medium, and may be installed in an auxiliary storage device 30A of the controller 30. Moreover, the program may be downloaded from another computer (e.g., the information processing device 200) outside the excavator 100 through the communication device 60, and may be installed in the auxiliary storage device 30A.

Some of the functions of the controller 30 may be implemented by another controller (control device). Specifically, it may be an embodiment in which the functions of the controller 30 are distributed by a plurality of controllers mounted on the excavator 100.

The operation pressure sensor 29 detects a pilot pressure on the secondary side (pilot line 27A) of the hydraulic pilot type operation device 26, that is, a pilot pressure corresponding to the operation state of each of the driven elements (hydraulic actuator) in the operation device 26. A detection signal of the pilot pressure corresponding to the operation state of each of the driven elements (hydraulic actuator HA) in the operation device 26, which is obtained by the operation pressure sensor 29, is taken into the controller 30.

In the case where the operation device 26 is of an electric type, the operation pressure sensor 29 is omitted. This is because the controller 30 can grasp the operation state of each of the driven elements through the operation device 26 based on the operation signal taken in from the operation device 26.

The sensor 40 acquires, for example, measurement data regarding a shape of an object near the excavator 100.

For example, the sensor 40 is a geometry sensor capable of acquiring measurement data representing a shape of an object near the excavator 100, such as a distance measuring sensor, a 3D camera, or the like. Moreover, the sensor 40 may be an integrated sensor that has a function of a physical sensor capable of acquiring measurement data representing characteristics of an object near the excavator 100, such as a multi-wavelength spectral camera or the like, in addition to the function of the geometry sensor.

For example, as illustrated in FIG. 2, the sensor 40 includes sensors 40F, 40B, 40L, and 40R. The sensor 40F measures a state (shape or characteristics) of an object in front of the upper turning body 3. The sensor 40B measures a state of an object behind the upper turning body 3. The sensor 40L measures a state of an object on the left side of the upper turning body 3. The sensor 40R measures a state of an object on the right side of the upper turning body 3. Thus, the sensor 40 can measure the state of the object in a range over the entire circumference around the excavator 100 as a center, i.e., an angular direction of 360 degrees in a top view of the excavator 100. Hereinafter, the sensors 40F, 40B, 40L, and 40R may be collectively or individually referred to as a “sensor 40X”.

Output data of the sensor 40 (sensor 40X) (i.e., measurement data regarding the state of the object near the excavator 100) is taken into the controller 30 through a one-to-one communication line or an in-vehicle network. Therefore, for example, the controller 30 can grasp the state, such as the shape and characteristics of the object near the excavator 100 based on the output data of the sensor 40X.

Some or all of the sensors 40B, 40L, and 40R may be omitted.

The sensor S1 is attached to the boom 4, and measures a state of attitude of the boom 4. The sensor S1 outputs the measurement data representing the state of the attitude of the boom 4. The state of the attitude of the boom 4 is, for example, an attitude angle (hereinafter, referred to as a “boom angle”) around a rotation axis of a base end corresponding to a coupling portion of the boom 4 with the upper turning body 3. The sensor S1 includes, for example, a rotary potentiometer, a rotary encoder, an acceleration sensor, an angular acceleration sensor, a 6-axis sensor, an inertial measurement unit (IMU), or the like. Hereinafter, the same may apply to the sensors S2 to S4. In addition, the sensor S1 may include a cylinder sensor that detects an extension or contraction position of the boom cylinder 7. Hereinafter, the same may apply to the sensors S2 and S3. The output (the measurement data representing the state of the attitude of the boom 4) of the sensor S1 is taken into the controller 30. Thus, the controller 30 can grasp the state of the attitude of the boom 4.

The sensor S2 is attached to the arm 5, and measures a state of attitude of the arm 5. The sensor S2 outputs the measurement data representing the state of the attitude of the arm 5. The state of the attitude of the arm 5 is an attitude angle (hereinafter, referred to as an “arm angle”) around a rotational axis of a base end corresponding a coupling portion of the arm 5 with the boom 4. The output (measurement data representing the state of the attitude of the arm 5) of the sensor S2 is taken into the controller 30, Thus, the controller 30 can grasp the state of the attitude of the arm 5.

The sensor S3 is attached to the bucket 6, and measures a state of attitude of the bucket 6. The sensor S3 outputs the measurement data representing the state of the attitude of the bucket 6. The state of the attitude of the bucket 6 is, for example, an attitude angle (hereinafter, referred to as an “arm angle”) around a rotational axis of a base end corresponding to a coupling portion of the bucket 6 with the arm 5. The output (measurement data representing the state of the attitude of the bucket 6) of the sensor S3 is taken into the controller 30. Thus, the controller 30 can grasp the state of the attitude of the bucket 6.

The sensor S4 measures a state of attitude of the body (e.g., the upper turning body 3) of the excavator 100. The sensor S4 outputs the measurement data representing the state of the attitude of the body of the excavator 100. The state of the attitude of the body of the excavator 100 is, for example, an inclined state of the body with respect to a predetermined reference surface (e.g., a horizontal surface). For example, the sensor S4 is attached to the upper turning body 3, and measures inclination angles (hereinafter, referred to as a “front-rear inclination angle” and a “left-right inclination angle”) around two axes in the front-rear direction and the left-right direction of the excavator 100. The output (measurement data representing the state of the attitude of the body of the excavator 100) of the sensor S4 is taken into the controller 30. Thus, the controller 30 can grasp the state of the attitude (inclined state) of the body (upper turning body 3).

The sensor S5 is attached to the upper turning body 3, and measures a turning state of the upper turning body 3. The sensor S5 outputs measurement data representing the turning state of the upper turning body 3. The sensor S5 measures, for example, a turning angular speed or a turning angle of the upper turning body 3. The sensor S5 includes, for example, a gyro sensor, a resolver, a rotary encoder, or the like. The output (measurement data representing the turning state of the upper turning body 3) of the sensor S5 is taken into the controller 30. Thus, the controller 30 can grasp the turning state, such as the turning angle or the like, of the upper turning body 3.

The controller 30 can grasp (estimate) the position of the end (bucket 6) of the attachment AT based on the outputs of the sensors S1 to S5.

In the case where the sensor S4 includes a gyro sensor, a 6-axis sensor, IMU, or the like capable of detecting angular velocities about three axes, the turning state (e.g., the turning angular speed) of the upper turning body 3 may be detected based on the detection signal of the sensor S4. In this case, the sensor S5 may be omitted.

The sensor S6 measures a position of the excavator 100. The sensor S6 may measure the position in world (global) coordinates, or may measure the position in local coordinates in the work site. In the former case, the sensor S6 is, for example, a global navigation satellite system (GNSS) sensor. In the latter case, the sensor S6 is a transmitter/receiver that can communicate with a device serving as a reference of the position of the work site, and can output a signal corresponding to the position with respect to the reference. The output of the sensor S6 is taken into the controller 30.

The sensor S7 measures the pressure (cylinder pressure) in the hydraulic chamber of the boom cylinder 7. The sensor S7 includes, for example, a sensor that measures the cylinder pressure (rod pressure) of the rod-side oil chamber of the boom cylinder 7, and a sensor that measures the cylinder pressure (bottom pressure) of the bottom-side oil chamber of the boom cylinder 7. The output (measurement data of the cylinder pressure of the boom cylinder 7) of the sensor S7 is taken into the controller 30.

The sensor S8 measures the pressure (cylinder pressure) in an oil chamber of the arm cylinder 8. The sensor S8 includes, for example, a sensor that measures the cylinder pressure (rod pressure) of the rod-side oil chamber of the arm cylinder 8, and a sensor that measures the cylinder pressure (bottom pressure) of the bottom-side oil chamber of the arm cylinder 8. The output (measurement data of the cylinder pressure of the arm cylinder 8) of the sensor S8 is taken into the controller 30.

The sensor S9 measures the pressure (cylinder pressure) in an oil chamber of the bucket cylinder 9. For example, the sensor S9 includes a sensor that measures the cylinder pressure (rod pressure) of the rod-side oil chamber of the bucket cylinder 9 and a sensor that measures the cylinder pressure (bottom pressure) of the bottom-side oil chamber of the bucket cylinder 9. The output (measurement data of the cylinder pressure of the bucket cylinder 9) of the sensor S9 is taken into the controller 30.

The controller 30 can grasp the state of the load applied to the attachment AT based on the outputs of the sensors S7 to S9. The load applied to the attachment AT includes, for example, a reaction force applied from the work target (soil of the ground) to the bucket 6, a weight of the soil held in the bucket 6, and the like.

In addition to the sensors S1 to S9, the excavator 100 may be equipped with other sensors capable of grasping the state of the excavator 100. For example, the excavator 100 may include an orientation sensor capable of detecting the orientation of the excavator 100. The orientation sensor is, for example, an electronic compass including a geomagnetic sensor.

<Hardware Configuration of Information Processing Device>

FIG. 5 is a diagram illustrating one example of a hardware configuration of the information processing device 200.

The functions of the information processing device 200 are implemented by any hardware, any combination of hardware and software, or the like. For example, as illustrated in FIG. 5, the information processing device 200 includes an external interface 201, an auxiliary storage device 202, a memory 203, a CPU 204, a high-speed arithmetic device 205, a communication interface 206, an input device 207, a display device 208, and an audio output device 209. These are connected by a bus BS2.

The external interface 201 functions as an interface for reading data from a recording medium 201A or writing data to the recording medium 201A. The recording medium 201A includes, for example, a flexible disk, a compact disc (CD), a digital versatile disc (DVD), a Blu-ray (registered trademark) disc (BD), an SD memory card, a USB memory, or the like. Thus, the information processing device 200 can read a variety of data used in processing through the recording medium 201A to store the data in the auxiliary storage device 202 or install a program for implementing various functions.

The information processing device 200 may acquire a variety of data or programs used in the processing from an external device through the communication interface 206.

The auxiliary storage device 202 stores files, data or the like necessary for various processes, in addition to the installed various programs. The auxiliary storage device 202 includes, for example, a hard disk drive (HDD), a solid state disk (SSD), a flash memory, or the like.

When an instruction to start a program is issued, the memory 203 reads the program from the auxiliary storage device 202 and stores the program. The memory 203 includes, for example, a dynamic random access memory (DRAM) or SRAM.

The CPU 204 executes various programs loaded from the auxiliary storage device 202 onto the memory 203, thereby implementing various functions related to the information processing device 200 according to the programs.

The high-speed arithmetic device 205 performs arithmetic processing at a relatively high speed in conjunction with the CPU 204. The high-speed arithmetic device 205 includes, for example, a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or the like.

The high-speed arithmetic device 205 may be omitted depending on the speed required for arithmetic processing.

The communication interface 206 is used as an interface for communicably connecting to an external device. Thus, the information processing device 200 can communicate with, for example, an external device of the excavator 100 or the like through the communication interface 206. Moreover, the communication interface 206 may have a plurality of types of communication interfaces depending on a communication system or the like with a device to be connected.

The input device 207 receives various inputs from a user. The input device 207 includes an operation device for remote operations, which remotely operates the excavator 100.

The input device 207 includes, for example, an input device (hereinafter, referred to as an “operation input device”) that is configured to receive mechanical operation inputs from a user. The operation input device includes, for example, a button, a toggle, a lever, a keyboard, a mouse, a touch panel mounted on the display device 208, a touch pad provided separately from the display device 208, or the like.

Moreover, the input device 207 may include an audio input device capable of receiving a voice input from a user. The audio input device includes, for example, a microphone capable of collecting a voice of a user.

Further, the input device 207 may include a gesture input device capable of receiving a gesture input from a user. The gesture input device includes, for example, a camera capable of capturing an image of a state of a user's gesture.

Furthermore, the input device 207 may include a biometric input device capable of receiving a biometric input from a user. The biometric input device includes, for example, a camera capable of acquiring image data including information on a fingerprint or an iris of a user.

The display device 208 displays an information screen image or an operation screen image for a user of the information processing device 200. The display device 208 is, for example, a liquid crystal display, an organic electroluminescence (EL) display, or the like.

The audio output device 209 delivers a variety of information to a user of the information processing device 200 with sound. The audio output device 209 is, for example, a buzzer, an alarm, a speaker, or the like.

[Functional Configuration of Operation Support System]

The functional configuration of the operation support system SYS will be described with reference to FIGS. 6 to 14 in addition to FIGS. 1 to 5. Specifically, a functional configuration related to generation of a trajectory of the working part of the excavator 100 in the operation support system SYS will be described.

First Example

FIG. 6 is a functional block diagram illustrating a first example of the functional configuration of the operation support system SYS.

Hereinafter, the “trajectory of the working part of the excavator 100” is used to include both a path (i.e., a trajectory) along which the working part of the excavator 100 has moved, and a path along which the working part of the excavator 100 may move in the future. The working part corresponds to a distal end portion of the attachment AT that is used to alter the work target. Specifically, the working part is the bucket 6.

The excavator 100 includes a support device 150. In the present example, the support device 150 supports the excavator 100, which is moved by the autonomous driving function, to execute the work.

As illustrated in FIG. 6, the support device 150 includes a controller 30, a hydraulic control valve 31, a sensor 40, and sensors S1 to S9.

The controller 30 includes, as functional units, a motion log providing unit 301 and a work supporting unit 302.

In the case where a plurality of excavators 100 are included in the operation support system SYS, there may be an excavator 100 in which the controller 30 includes, among the motion log providing unit 301 and the work supporting unit 302, only the former, and an excavator 100 in which the controller 30 includes only the latter. In this case, the former excavator 100 only has a function of acquiring a motion log of the excavator, which is used for a work supporting function of the latter excavator 100, and providing the motion log of the excavator 100 to the information processing device 200. The same may apply to cases of a second example and a third example, which will be described later.

The information processing device 200 includes, as functional units, a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a teacher data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007.

The motion log providing unit 301 is a functional unit that acquires a motion log during a predetermined motion of the excavator 100 to provide the acquired motion log to the information processing device 200.

The predetermined motion includes, for example, an excavating motion, a boom-raising and turning motion, a boom-lowering and turning motion, a soil-dumping motion, a brooming motion, or the like used during excavation work. In addition, the predetermined motion may include an excavating motion, a soil-dumping motion, a sweeping motion, a leveling motion, a compacting motion, a brooming motion, or the like used during land-leveling work. Further, the predetermined motion may include a soil cutting motion, compacting motion, or the like used during slope protection work. The sweeping motion is, for example, a motion where the attachment AT is driven to push the bucket 6 forward along the ground to sweep the soil forward with the back of the bucket 6. In the sweeping motion, for example, the attachment AT perform a lowering motion of the boom 4 and an opening motion of the arm 5. The leveling motion is, for example, a motion where the attachment AT is driven to move a claw tip of the bucket 6 closer to the body substantially horizontally along the ground, thereby leveling the unevenness of the ground (the surface of the terrain). In the leveling motion, for example, the attachment AT performs a raising motion of the boom 4 and a closing motion of the arm 5. The compacting motion is, for example, a motion where the attachment AT is driven to push the back of the bucket 6 against the ground. In addition, the compacting motion may be a motion where the back of the bucket 6 is struck against the ground to press down the ground, while moving the bucket 6 up and down. Further, the compacting motion may be a motion where the bucket 6 is pushed forward along the ground so that soil is swept by the back of the bucket 6 to a predetermined position, followed by pressing the ground in the predetermined position with back of the bucket 6. In the compacting motion, for example, the attachment AT is driven to perform a lowering motion of the boom 4 when pressing the ground. The brooming motion is, for example, a motion where the upper turning body 3 is driven to turn the bucket 6 left and right with the bucket 6 being placed against the ground. Moreover, the brooming motion may be, for example, a motion where the attachment AT and the upper turning body 3 are driven to push the bucket 6 forward, while turning the bucket 6 left and right with the bucket 6 being placed against the ground. In the brooming motion, for example, the upper turning body 3 alternately repeats left and right turning motions. Moreover, in the brooming motion, for example, in addition to the alternate left and right turning motions of the upper turning body 3, the attachment AT may perform a lowering motion of the boom 4 and an opening motion of the arm 5, similarly to the case of the sweeping motion.

The motion log of the excavator 100 is time-series data representing a motion state of the excavator 100. For example, the motion log of the excavator 100 includes time-series data representing operation contents of an operator. The time-series data representing the operation contents of the operator may be, for example, time-series output data of the operation pressure sensor 29 corresponding to the hydraulic pilot type operation device 26 or time-series output data (data of operation signals) of the operation device 26 corresponding to the electric type operation device 26. Moreover, the motion log of the excavator 100 may be time-series output data of sensors S1 to S5 or time-series data representing the state of the attitude of the excavator 100 acquired from the output data of sensors S1 to S5.

For example, the motion log providing unit 301 acquires a motion log, when a relatively experienced operator who has a long history of operating the excavator 100 (hereinafter, referred to as a “skilled operator” for convenience) operates the excavator 100, and provides the acquired motion log to the information processing device 200. Thus, as described later, a learned model LM2, which can reproduce the motions of the excavator 100 under the operation of the skilled operator, can be generated by machine learning based on the motion log of the excavator 100.

Moreover, as described later, the learned model LM2 may be omitted, and only a learned model LM1 may be generated. In this case, the motion log of the excavator 100 to be provided to the information processing device 200 may include a motion log at the time of operation of the excavator 100 by an operator other than a skilled operator, or a motion log corresponding to the motions of the excavator 100 operated by the automated driving function. Moreover, a motion log of the excavator 100 for the learned model LM1 and a motion log of the excavator 100 for the learned model LM2 may be separately acquired.

The motion log providing unit 301 includes a motion log recording unit 301A, a motion log storage unit 301B, and a motion log transmission unit 301C.

The motion log recording unit 301A acquires a motion log during a predetermined motion of the excavator 100, and records the motion log in the motion log storage unit 301B. For example, the motion log recording unit 301A records a motion log of a predetermined motion in the motion log storage unit 301B every time the predetermined motion of the excavator 100 is executed.

The motion log of the excavator 100 is stored in the motion log storage unit 301B. For example, in the motion log storage unit 301B, the motion log and data of the time (date and time) when the predetermined motion is executed are associated with each other and stored for each predetermined motion executed by the excavator 100. The data of the time when the predetermined motion is executed includes both data of the start time and data of the end time of the predetermined motion executed by the excavator 100. In the case where a plurality of predetermined motions are defined, for each predetermined motion executed by the excavator 100, the motion log, the data of the time of the executed predetermined motion, and data of labeling information of the executed predetermined motion are associated with each other and stored in the motion log storage unit 301B. Hereinafter, the data associated with the motion log of the excavator 100 may be referred to as “auxiliary data” for convenience. For example, record data representing the correlation between the motion log and the auxiliary data is accumulated in the motion log storage unit 301B for each predetermined motion executed by the excavator 100, thereby constructing a database of the motion logs during the execution of the predetermined motions of the excavator 100.

The motion log of the motion log storage unit 301B, which has already been sent to the information processing device 200 by the below-described motion log transmission unit 301C, may be deleted later.

The motion log transmission unit 301C transmits the motion log during the execution of the predetermined motion of the excavator 100, and the auxiliary data associated with the motion log, both of which are stored in the motion log storage unit 301B, to the information processing device 200 via the communication device 60. Moreover, the motion log transmission unit 301C may also transmit the record data representing the correlation between the motion log of the excavator 100 and the auxiliary data for each predetermined motion executed by the excavator 100 to the information processing device 200.

For example, the motion log transmission unit 301C transmits the motion log of the excavator 100 and the auxiliary data, which have not yet been transmitted but are stored in the motion log storage unit 301B, to the information processing device 200 in response to a transmission request of the motion log of the excavator 100 received from the information processing device 200. Moreover, the motion log transmission unit 301C may automatically transmit the motion log of the excavator 100 and auxiliary data, which have not yet been transmitted but are stored in the motion log storage unit 301B, to the information processing device 200 at a predetermined timing. The predetermined timing is, for example, a timing of the termination of the operation of the excavator 100 (when the key switch is turned off), a timing of the start of the operation (when the key switch is turned on), or the like.

The log acquisition unit 2001 acquires a log during execution of a predetermined motion of the excavator 100.

The log during the execution of the predetermined motion of the excavator 100 includes a motion log during the execution of the predetermined motion of the excavator 100, and a state log of the work target. The state log of the work target includes data representing the state of the work target before and after the execution of the predetermined motion of the excavator 100. The state of the work target includes a shape of the work target (i.e., the landform of the ground region of the work target), characteristics of soil of the work target, and the like. The motion log during the execution of the predetermined motion of the excavator 100 is uploaded from the excavator 100. The state log of the work target during the execution of the predetermined motion of the excavator 100 is acquired based on the measurement data uploaded from the sensor group 300 and the auxiliary data (data of the time when the predetermined motion has been executed) uploaded from the excavator 100.

The simulator unit 2002 performs a computer simulation regarding the predetermined motion of the excavator 100 using virtual models of the excavator 100 and the work target (soil).

For example, the soil of the ground of the work target is modeled as a collection of fine particles using a distinct element method (DEM). Thus, the simulator unit 2002 can cause the virtual model of the excavator 100 to execute the predetermined motion, such as an excavating motion or the like, and can virtually reproduce the behavior of the entire soil of the work target, the reaction force from the soil, or the like, through the analysis of movements of individual fine particles.

The simulator unit 2002 acquires, as a log during the execution of the predetermined motion of the excavator 100, trajectory data of the working part of the excavator 100 and data of the state of the work target (soil) before and after the execution of the predetermined motion by a computer simulation. The former data corresponds to the motion log during the execution of the predetermined motion of the excavator 100 in the computer simulation, and the latter data corresponds to the state log of the work target during the execution of the predetermined motion of the excavator 100 in the computer simulation.

The simulator unit 2002 performs a large number of patterns of the computer simulations on the predetermined motion of the excavator 100 using various states of the work target (soil) and various trajectories of the working part of the excavator 100. Thus, the simulator unit 2002 can accumulate logs when the excavator 100 executes the predetermined motion under the different conditions of the work target and the working part in the computer simulations in the log storage unit 2003.

The log storage unit 2003 stores, in an accumulated manner, logs acquired by the log acquisition unit 2001 and the simulator unit 2002 during the execution of the predetermined motion of the excavator 100. For example, in the log storage unit 2003, the motion log of each predetermined motion that has actually been executed by the excavator 100 or executed by the computer simulation, and the state log of the work target are associated with each other, and are stored. In the log storage unit 2003, the logs acquired by the log acquisition unit 2001 and the logs acquired by the simulator unit 2002 may be stored in an identifiable manner, or may be mixed and stored in an unidentifiable manner.

The teacher data generation unit 2004 generates teacher data for machine learning based on the logs during the execution of the predetermined motion of the excavator 100, which are stored in the log storage unit 2003, and outputs a teacher dataset that is a collection of a variety of teacher data. The teacher data generation unit 2004 may automatically generate teacher data by batch processing, or may generate teacher data in accordance with an input from a user of the information processing device 200. The teacher data generation unit 2004 includes teacher data generation units 2004A and 2004B.

The teacher data generation unit 2004A generates a teacher dataset for generating a learned model LM1. The learned model LM1 infers the shape of the work target after the execution of the predetermined motion of the excavator 100 based on the predetermined input data. The shape of the work target refers to, for example, a landform of the ground of the work target, more specifically, the shape of the soil, such as undulations appearing on the ground of the work target, or the like.

The input data corresponding to the learned model LM1 includes data representing the state of the work target before the execution of the predetermined motion of the excavator 100 and the data of the trajectory (path) of the working part during the execution of the predetermined motion of the excavator 100. The state of the work target includes, for example, a shape of the work target. Moreover, the state of the work target may include characteristics of the soil of the work target. For example, the data representing the characteristics of the soil includes data of an angle of repose of the soil. Thus, the learned model LM1 can infer a more appropriate shape of the work target considering the angle of repose of the soil. In addition, the input data corresponding to the learned model LM1 may include data representing an excavation reaction force during the execution of the predetermined motion of the excavator 100. The excavation reaction force is a reaction force applied from the ground to the working part (specifically, the bucket 6) of the excavator 100. Thus, for example, even in a case where the bucket 6 abuts on a rock in the ground or the like and as a result, the soil cannot be excavated, the learned model LM1 can estimate an appropriate soil shape considering the excavation reaction force.

The teacher data is a combination of a type of the input data defined for the learned model LM1 and data (ground truth) representing a correct inference result corresponding to the input data. The ground truth is data representing the state of the work target after the execution of the predetermined motion of the excavator 100 corresponding to the input data included in the teacher data. In the case where a plurality of types of predetermined motions are defined, a learned model LM1 may be generated for each type of the predetermined motion. In this case, the teacher data generation unit 2004A generates a teacher dataset for each type of the predetermined motion.

The teacher dataset for generating the learned model LM1 is generated from, for example, both the logs acquired by the log acquisition unit 2001 and the logs acquired by the simulator unit 2002. Moreover, the teacher dataset for generating the learned model LM1 may be generated from, among the logs acquired by the log acquisition unit 2001 and the logs output from the simulator unit 2002, only the former logs. In this case, the simulator unit 2002 may be omitted. Further, the teacher dataset for generating the learned model LM1 may be generated from, among the logs acquired by the log acquisition unit 2001 and the logs output from the simulator unit 2002, only the latter logs. In this case, the sensor group 300 and the motion log providing unit 301 of the excavator 100 may be omitted. Moreover, the teacher dataset for generating the learned model LM1 may include a teacher dataset for final adjustment (for fine tuning). In this case, the base teacher dataset may be generated based on the logs output from the simulator unit 2002 because the base teacher dataset needs a large volume of data, and the teacher dataset for final adjustment may be generated based on the logs acquired by the log acquisition unit 2001. Hereinafter, these details of the method of generating the learned model LM1 may be the same for the below-described learned models LM2 to LM4, and may be incorporated into the method of generating the learned models LM2 to LM4.

The teacher data generation unit 2004B generates teacher data for generating a learned model LM2. The learned model LM2 infers a target trajectory of the working part for the predetermined motion of the excavator 100 based on the predetermined input data.

The input data corresponding to the learned model LM2 includes, for example, data representing a state of the work target near the excavator 100. The data representing the state of the work target includes, for example, data representing a shape of the work target as described above. In addition, the data representing the state of the work target may include data representing characteristics of soil of the work target, as described above. Further, the input data corresponding to the learned model LM2 may include data representing a target shape of the work target. In the case where the predetermined motion of the excavator 100 is a soil-dumping motion, the input data corresponding to the learned model LM2 may include data representing a weight or volume of the soil held in the bucket 6 before dumping.

The teacher data is a combination of a type of input data defined for the learned model LM2 and data (ground truth) representing a correct inference result corresponding to the input data. Specifically, the input data included in the teacher data includes data representing a state of the work target before the execution of the predetermined motion of the excavator 100. Moreover, the input data included in the teacher data may include data representing a target shape corresponding the work target. In the case where the predetermined motion of the excavator 100 is a soil-dumping motion, the input data included in the teacher data includes data representing a weight or volume of the soil held in the bucket 6 before dumping. Further, the ground truth included in the teacher data includes data representing a trajectory of the working part when the predetermined motion of the excavator 100 is executed by the operation of a skilled operator on the premise of the state of the work target corresponding to the input data. Specifically, the teacher data generation unit 2004B generates a teacher dataset based on logs at the time of the execution of the predetermined motion of the excavator 100 by a skilled operator, which are acquired by the log acquisition unit 2001. In the case where a plurality of types of predetermined motions are defined, a learned model LM2 may be generated for each type of the predetermined motion. In this case, the teacher data generation unit 2004B generates a teacher dataset for each type of the predetermined motion.

The machine learning unit 2005 causes the base learning model to be trained by machine learning based on the teacher data set generated by the teacher data generation unit 2004 to generate learned models LM1 and LM2. The learned model (base learning model) includes, for example, a neural network, such as a deep neural network (DNN) or the like.

The machine learning unit 2005 includes machine learning units 2005A and 2005B.

The machine learning unit 2005A causes the base learning model M1 to be trained by machine learning based on the teacher dataset output from the teacher data generation unit 2004A. Thus, the machine learning unit 2005A can generate a learned model LM1 capable of outputting (inferring) the state of the work target after the execution of the predetermined motion of the excavator 100 using, as inputs, the data of the state of the work target before the execution of the predetermined operation of the excavator 100, the data of the trajectory of the working part during the execution of the predetermined motion of the excavator 100, and the like. For example, the machine learning unit 2005A can optimize the learning model M1 and generate a learned model LM1 based on an error between the output data of the learning model M1 with respect to the input data included in the teacher data and the ground truth using an algorithm of the backpropagation. Hereinafter, the same may apply to the generation of the below-described learned models LM2 to LM4. Moreover, the machine learning unit 2005A may correct the learned model LM1 by causing the learned model LM1 to undergo additional learning to reduce an error between the inference result of the learned model LM1 and the actual measurement result of the sensor 40. In this case, the data of the inference result by the learned model LM1 and the data of the actual measurement result of the sensor 40 are uploaded from the excavator 100 onto the information processing device 200 and are utilized for additional learning.

The machine learning unit 2005B causes the base learning model M2 to be trained by machine learning based on the teacher dataset output from the teacher data generation unit 2004B. Thus, the machine learning unit 2005B can generate a learned model LM2 capable of outputting a target trajectory of the working part in the predetermined motion of the excavator 100 using, as an input, the data of the state of the work target near the excavator 100.

Moreover, the machine learning unit 2005B may generate the learned model LM2 by applying reinforcement learning to the learning model M2 instead of the supervised learning. In this case, the teacher data generation unit 2004B is omitted. For example, the machine learning unit 2005B causes the learning model M1 to undergo machine learning so as to maximize a predetermined reward related to the work efficiency based on the logs acquired by the log acquisition unit 2001 and the simulator unit 2002. At the time of the above machine learning, the machine learning unit 2005B works in conjunction with the simulator unit 2002, and causes the simulator unit 2002 to try many motion patterns, thereby causing the learning model M1 to undergo machine learning more efficiently.

In the learned model storage unit 2006, the learned models LM1 and LM2 output by the machine learning unit 2005 are stored. In the case where relearning or additional learning of the learned model LM1 is performed by the machine learning unit 2005A, the learned model LM1 of the learned model storage unit 2006 is updated. The same applies to a case where relearning or additional learning of the learned model LM2 is performed by the machine learning unit 2005B. In the case where the learned model LM1 is updated, the learned model LM1 before updating may be stored in the learned model storage unit 2006 or another storage unit in a reusable form. The same may apply to the learned model LM2 before updating in the case where the learned model LM2 is updated. Thus, for example, in the case where there is a problem in the learned model LM1 or learned model LM2 after updating, the learned model LM1 before updating or the learned model LM2 before updating can be restored and reused.

The distribution unit 2007 distributes the data of the learned models LM1 and LM2 to the excavator 100.

For example, when the learned model LM1 is generated or updated by the machine learning unit 2005A, the distribution unit 2007 distributes the recently generated or updated learned model LM1 to the excavator 100. Moreover, the distribution unit 2007 may distribute the latest learned model LM1 of the learned model storage unit 2006 to the excavator 100 in response to a signal received from the excavator 100 and requesting distribution of the learned model LM1. The same may apply to the learned model LM2.

The work supporting unit 302 is a functional unit for supporting the work of the excavator 100 operated by the autonomous driving function.

The work supporting unit 302 includes a learned model storage unit 302A, a work target shape acquisition unit 302B, a target trajectory generation unit 302C, and a motion control unit 302D.

In the learned model storage unit 302A, the learned models LM1 and LM2 distributed from the information processing device 200 and received via the communication device 60 are stored.

The work target shape acquisition unit 302B acquires data representing the shape of the work target of the excavator 100 based on the trajectory (path) of the working part during the execution of the predetermined motion of the excavator 100. Specifically, the work target shape acquisition unit 302B may acquire data representing the shape of the work target of the excavator 100 through estimation of a change in the landform (shape of soil) from the landform before the execution of the predetermined motion based on the trajectory of the working part during the execution of the predetermined motion of the excavator 100. Moreover, the work target shape acquisition unit 302B may acquire data representing the shape of the work target of the excavator 100 considering the measurement data of the characteristics of the soil of the work target provided by the sensor 40 (physical sensor) or the measurement data of the sensors S7 to S9 (i.e., data regarding the reaction force from the soil of the work target to the working part).

In this case, the work target shape acquisition unit 302B may interpolate the shape of the soil at a location that cannot be measured by the sensor 40, based on the trajectory of the working part during the execution of the predetermined motion of the excavator 100 on the premise of the measurement data of the shape of the work target provided by the sensor 40. Moreover, the work target shape acquisition unit 302B may acquire data representing the shape of the work target without using the sensor 40, while estimating a change in the shape of the soil with the initial state of the shape of the work target of the excavator 100 as a starting point, based on the trajectory of the working part at each execution of the predetermined motion of the excavator 100. In this case, the sensor 40 may be omitted. The initial state of the shape of the work target of the excavator 100 is, for example, distributed from the outside of the excavator 100 via the communication device 60, or acquired by inputs made by a user via the input device 52. Moreover, the initial state of the shape (initial shape) of the work target of the excavator 100 may be fixed, for example, to be a flat surface or the like at the same height as the ground on which the crawler of the excavator 100 is present.

For example, the work target shape acquisition unit 302B estimates the shape of the soil after the execution of the predetermined motion of the excavator 100 using the learned model LM1 based on the shape of the soil before the execution of the predetermined motion of the excavator 100 and the trajectory of the working part during the execution of the predetermined motion of the excavator 100. Then, the work target shape acquisition unit 302B may acquire data representing the current state of the work target by combining the estimation result and the measurement data of the state of the soil after the execution of the predetermined motion of the excavator 100 by the sensor 40. Specifically, the work target shape acquisition unit 302B performs the data integration by interpolating the location where the measurement of the shape of the soil cannot be performed by the sensor 40 within the observation target region near the excavator 100 with the estimation result. The observation target region refers to a region for which the data representing the shape of the work target of the excavator 100 is acquired by the work target shape acquisition unit 302B near the excavator 100. Thus, even in the case where measurement data of the work target cannot be obtained for a part of the observation target region by the sensor 40 due to occlusion or the like, the controller 30 can acquire data representing the shape of the work target of the observation target region including the shape of the work target at the location where occlusion has occurred.

The shape of the work target before the execution of the predetermined motion of the excavator 100, for example, corresponds the previous output of the work target shape acquisition unit 302B. The trajectory of the working part in the predetermined motion of the excavator 100 is acquired, for example, based on the outputs of the sensors S1 to S6.

In the case where there is an obstacle in the observation target region near the excavator 100, the work target shape acquisition unit 302B may exclude the region in which the obstacle is present from an estimation target of the shape of the work target based on the output of the sensor 40. The obstacle includes, for example, a moving body, such as a work machine, a work vehicle, or the like, or a feature, such as a utility pole, a fence, or the like. In the case where there are multiple excavators 100 operated in the same work site, the multiple excavators 100 may share the data representing the trajectory (path) of the working part during the execution of the predetermined motion. Thus, the work target shape acquisition unit 302B (learned model LM1) can also consider the change of the shape of the work target caused by the predetermined motion of another excavator 100. Thus, the work target shape acquisition unit 302B (learned model LM1) can more appropriately estimate the shape of the work target near the excavator 100. Some or all of the functions of the work target shape acquisition unit 302B may be transferred to an external device (e.g., the information processing device 200) having a relatively high processing capability. Thus, even when the processing capability of the controller 30 is insufficient, the controller 30 can acquire data representing the shape of the work target after the execution of the predetermined motion of the excavator 100 considering the trajectory of the working part during the execution of the predetermined motion of the excavator 100.

The target trajectory generation unit 302C generates a target trajectory for the predetermined motion of the excavator 100 based on the estimation result of the work target shape acquisition unit 302B (the state of the work target after the execution of the predetermined motion of the excavator 100).

For example, the target trajectory generation unit 302C generates a target trajectory of the working part in the predetermined motion of the excavator 100 using the learned model LM2 based on the shape of the work target acquired by the work target shape acquisition unit 302B.

The target trajectory generation unit 302C may generate a target trajectory of the excavator 100 corresponding to the state (prediction result) of the work target near the excavator 100 by applying any known method instead of the learned model LM2. In this case, the teacher data generation unit 2004C and the machine learning unit 2005B may be omitted. For example, the target trajectory generation unit 302C may generate data of a target trajectory of the working part of the excavator 100 using model predictive control (MPC) based on the shape of the work target acquired by the work target shape acquisition unit 302B. Moreover, the target trajectory generation unit 302C may generate data of a target trajectory of the working part of the excavator 100 by optimizing a reference trajectory of the working part of the excavator 100, which is defined in advance, based on the shape of the work target acquired by the work target shape acquisition unit 302B.

The motion control unit 302D causes the excavator 100 to execute the predetermined motion so that a predetermined part of the excavator 100 moves along the target trajectory generated by the target trajectory generation unit 302C. Specifically, the motion control unit 302D controls the hydraulic control valve 31 while grasping the position of the working part according to the outputs of the sensors S1 to S5, thereby causing the excavator 100 to execute the predetermined motion so that the working part of the excavator 100 moves along the target trajectory. Thus, the excavator 100 can autonomously proceed with the work while executing the predetermined motion according to the shape of the work target.

As described above, in the present example, the controller 30 acquires data that represents the state of the excavator 100 and is related to the shape of the work target, in association with the predetermined motion of the excavator 100. Then, the controller 30 estimates the shape of the work target (ground) of the excavator 100 based on the acquired data. Specifically, the controller 30 acquires data representing a trajectory (path) of the working part during the execution of the predetermined motion of the excavator 100, and estimates the shape of the work target (ground) after the execution of the predetermined motion of the excavator 100 based on the data. Thus, for example, even in the case where there is a location at which the sensor 40 cannot measure a shape of soil due to occlusion or the like, the controller 30 can interpolate the shape of the soil at such a location considering a change in the shape of soil due to the movement of the working part of the excavator 100. In addition, the shape of the work target of the excavator 100 can be estimated, without the sensor 40, based on the history of changes in the shape of soil due to the movement of the working part of the excavator 100. Thus, the controller 30 can more appropriately grasp the shape of soil of the work target of the excavator 100 within the observation target region, and as a result, can more appropriately generate a target trajectory of the working part of the excavator 100. Accordingly, the controller 30 can more appropriately execute the autonomous operation of the excavator 100.

Some or all of the functions of the work target shape acquisition unit 302B, the target trajectory generation unit 302C, and the motion control unit 302D may be transferred to the information processing device 200. Thus, the processing load of the excavator 100 can be reduced in the processing related to the generation of the target trajectory of the working part of the excavator 100 or the processing related to the control of the motion of the excavator 100. Hereinafter, the same may apply to a third example described later.

Second Example

FIG. 7 is a functional block diagram illustrating a second example of the functional configuration of the operation support system SYS.

Hereinafter, the same reference numerals are given to the same or corresponding components as those of the first example described above, and the description will be made focusing on the points different from the first example described above.

As in the above-described first example, the excavator 100 includes a support device 150. In the present example, the support device 150 supports a user who operates the excavator 100 to perform work, or who monitors the work of the excavator 100.

As illustrated in FIG. 7, in the present example, the support device 150 includes a controller 30, a sensor 40, a display device 50A, and sensors S1 to S9. In addition, in the case where the excavator 100 is remotely operated, the support device 150 may include a communication device 60.

The controller 30 includes, as functional units, a motion log providing unit 301 and a work supporting unit 302, as in the above-described first example.

The motion log providing unit 301 includes a motion log recording unit 301A, a motion log storage unit 301B, and a motion log transmission unit 301C, as in the above-described first example.

The information processing device 200 includes, as functional units, a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a teacher data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007, as in the above-described first example.

The present example is different from the above-described first example in that the teacher data generation unit 2004B, the machine learning unit 2005B, and the learned model LM2 are omitted in the information processing device 200.

The work supporting unit 302 is a functional unit configured to support a user who operates the excavator 100 to perform work, or who monitors the work of the excavator 100.

The work supporting unit 302 includes a learned model storage unit 302A, a work target shape acquisition unit 302B, and a display processing unit 302E. Specifically, the work supporting unit 302 is different from that of the above-described first example in that the target trajectory generation unit 302C and the motion control unit 302D are omitted and the display processing unit 302E is added.

The display processing unit 302E causes the display device 50A to display a screen image related to a work support for a user who operates the excavator 100 to perform work or who monitors the work of the excavator 100. Moreover, the display processing unit 302E transmits data corresponding the same screen image to the remote operation support device 400 or the remote monitoring support device via the communication device 60, thereby displaying the above screen image on the remote operation support device 400 or the remote monitoring support device.

For example, the display processing unit 302E causes the display device 50A to display an image representing the shape of the work target (ground) of the excavator 100 based on the output of the work target shape acquisition unit 302B (data of the shape of the work target after the execution of the certain motion of the excavator 100). In addition, the display processing unit 302E may transmit data of an image representing the shape of the work target of the excavator 100 to the remote operation support device 400 or the remote monitoring support device via the communication device 60, thereby displaying the above image on the remote operation support device 400 or a remote monitoring device. Thus, a user can operate the excavator 100 or monitor the work of the excavator 100, while grasping the shape of the work target of the excavator 100 through the image representing the shape of the work target of the excavator 100 displayed on the display device 50A or the like. In addition, the display device 50A or the like can update the image representing the shape of the work target of the excavator 100 according to the execution of the predetermined motion of the excavator 100. Thus, a user can more appropriately operate the excavator 100 or monitor the work of the excavator 100 while grasping the shape of the work target of the excavator 100 in real time.

As described above, in the present example, the controller 30 can more appropriately grasp the shape of soil of the work target of the excavator 100 within the observation target region, and as a result, a user can more appropriately grasp the shape of soil of the work target through the display of the image representing the shape of soil. Thus, a user can more appropriately operate the excavator 100 or more appropriately monitor the work of the excavator 100, while grasping the shape of soil of the work target.

In the case where the excavator 100 is remotely operated, some or all of the functions of the learned model storage unit 302A, the work target shape acquisition unit 302B, and the display processing unit 302E may be provided to the remote operation support device 400. In addition, the functions of the work target shape acquisition unit 302B may be transferred to the information processing device 200. Thus, the processing load of the excavator 100 can be reduced.

Third Example

FIG. 8 is a functional block diagram illustrating a third example of the functional configuration of the operation support system SYS. FIG. 9 is a diagram explaining a method of estimating a shape of a work target. Specifically, FIG. 9 is a diagram explaining a method of estimating the shape of the work target after repeating the predetermined motion of the excavator 100 several times (n times) with the initial shape of the work target as a starting point. FIGS. 10 to 12 are diagrams respectively illustrating one example, another example, and yet another example of measurement data and estimation data of the reaction force from the work target to the working part (bucket 6) during the excavating motion of the excavator 100. FIGS. 13 and 14 are diagrams respectively illustrating one example and another example of a relationship between the certainty of the estimation result of the shape of the work target of the excavator 100 and the predetermined motion of the excavator 100.

In FIGS. 13 and 14, the certainty of the estimation result of the shape of the work target of the excavator 100 is defined by a value between the lowest “0” and the highest “1”. In addition, in FIG. 13, the speed ratio refers to a ratio of the speed of the predetermined motion of the excavator 100 with respect to the speed in a normal state as a reference, that is, a state in which the motion of the excavator 100 is not restricted. In addition, in FIG. 14, the excavation depth ratio refers to a ratio of the excavation depth of the excavator 100 with respect to the excavation depth in a normal state as a reference, that is, a state in which the motion of the excavator 100 is not restricted. Each of the speed ratio and the excavation depth ratio is represented by “1” in a state where the motion of the excavator 100 is not restricted, and by a value smaller than “1” in the case where the motion of the excavator 100 is restricted.

Hereinafter, the same reference numerals are given to the same or corresponding components as those of the first example and the second example described above, and the description will be made focusing on the points different from the first example and the second example described above.

The excavator 100 includes a support device 150, as in the above-described first example and second example described above. In the present example, the support device 150 supports the work of the excavator 100 operated by the autonomous driving function, as in the above-described first example.

As illustrated in FIG. 8, the support device 150 includes a controller 30, a hydraulic control valve 31, and sensors S1 to S9, as in the above-described first example and second example. In the present example, unlike the above-described first example and second example, the sensor 40 is not mounted on the excavator 100, and the support device 150 does not include the sensor 40.

The controller 30 includes, as functional units, a motion log providing unit 301 and a work supporting unit 302, as in the above-described first example and second example.

The motion log providing unit 301 includes a motion log recording unit 301A, a motion log storage unit 301B, and a motion log transmission unit 301C, as in the above-described first example and second example.

The information processing device 200 includes, as functional units, a log acquisition unit 2001, a simulator unit 2002, a log storage unit 2003, a teacher data generation unit 2004, a machine learning unit 2005, a learned model storage unit 2006, and a distribution unit 2007, as in the above-described first example and second example.

The information processing device 200 of the present example is different from that of the above-described first example in that teacher data generation units 2004C and 2004D and machine learning units 2005C and 2005D are added, and learned models LM3 and LM4 are stored in the learned model storage unit 2006. In addition, the present example is different from the above-described first example in that a predetermined motion of the excavator 100 to be performed is limited to an embodiment of a motion to be in contact with soil of the work target. The predetermined motion of the excavator 100 to be performed in the present example includes, for example, an excavating motion. In addition, the predetermined motion of the excavator 100 to be performed in the present example may include, for example, a sweeping motion, a compacting motion, or a brooming motion.

The teacher data generation unit 2004 includes the teacher data generation units 2004A to 2004D.

The teacher data generation unit 2004C generates a teacher dataset for generating a learned model LM3. The learned model LM3 infers the reaction force applied from the work target to the working part based on the predetermined input data.

The input data corresponding to the learned model LM3 includes data representing the shape of the work target before the execution of the predetermined motion of the excavator 100 and data of the trajectory (path) of the working part during the execution of the predetermined motion of the excavator 100. In addition, the input data corresponding to the learned model LM3 may include data representing characteristics of the soil of the work target of the excavator 100.

The teacher data is a combination of a type of input data defined for the learned model LM3 and data representing a correct inference result (ground truth) corresponding to the input data. Specifically, the input data included in the teacher data includes data representing the shape of the work target before the execution of the predetermined motion of the excavator 100 and data representing the trajectory (path) of the working part during the execution of the predetermined motion of the excavator 100. In addition, the input data included in the teacher data may include data representing characteristics of soil of the work target during the execution of the predetermined motion of the excavator 100. Further, the ground truth included in the teacher data includes time-series data representing the reaction force applied from the work target to the working part during the execution of the predetermined motion of the excavator 100. In the case where a plurality of predetermined motions are defined, a learned model LM3 may be generated for each type of the predetermined motion. In this case, the teacher data generation unit 2004C generates a teacher dataset for each type of the predetermined motion.

The teacher data generation unit 2004D generates a teacher dataset for generating a learned model LM4. The learned model LM4 infers the certainty of the estimation result of the shape of the work target of the excavator 100. The estimation result of the shape of the work target of the excavator 100 includes the estimation result of the shape of the work target of the excavator 100 acquired by the work target shape acquisition unit 302B. In addition, the estimation result of the shape of the work target of the excavator 100 may include the initial shape of the work target recognized by the work target shape acquisition unit 302B. This is because the sensor 40 is not mounted on the excavator 100 in the present example, and the initial shape of the work target cannot be observed by the excavator 100. The certainty of the estimation result of the shape of the work target of the excavator 100 refers to a degree of certainty of the estimation result of the shape of the work target of the excavator 100. The certainty of the estimation result of the shape of the work target of the excavator 100 is defined so that the certainty increases, as a difference between the estimation result of the shape of the work target of the excavator 100 and the actual shape of the work target decreases, and the certainty decreases, as the difference increases. In addition, relatively high certainty of the estimation result of the shape of the work target of the excavator 100 means, in other words, a relatively low degree of uncertainty of the estimation result. Relatively low certainty of the estimation result means, in other words, a relatively high degree of uncertainty of the estimation result. Therefore, the certainty of the estimation result of the shape of the work target of the excavator 100 also means the degree of uncertainty of the estimation result of the shape of the work target of the excavator 100. Further, the certainty of the estimation result of the work target of the excavator 100 is defined, for example, for each of small regions obtained by dividing the observation target region having the shape of the work target. Thus, the certainty of the estimation result of the shape of the work target of the excavator 100 is, for example, represented as a vector defining the certainty of the estimation result of the shape for each of the small regions of the work target of the excavator 100.

For example, the learned model LM4 infers the certainty of the shape of the work target of the excavator 100 after the correction, on the premise that the estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100 is later corrected by the data obtained during the execution of the predetermined motion of the excavator 100. For example, as described later, the estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100 is corrected based on an error between the measurement result of the reaction force applied from the work target to the working part during the execution of the predetermined motion of the excavator 100 and the estimation result thereof. The shape of the work target before the execution of the predetermined motion of the excavator 100 is, for example, the shape of the work target immediately before the execution of the predetermined motion of the excavator 100, which is performed once. In addition, the shape of the work target before the execution of the predetermined motion of the excavator 100 may be the shape (i.e., the initial shape) of the work target of the excavator 100 before the start of the predetermined motion of the excavator 100, which will be repeatedly performed.

Specifically, the learned model LM4 may infer the certainty of the corrected estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100 with respect to the input data including the data representing the corrected estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100.

The input data of the learned model LM4 includes data representing the corrected estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100 and data obtained during the execution of the predetermined motion of the excavator 100. The data obtained during the execution of the predetermined motion of the excavator 100 includes, for example, data representing a trajectory of the working part during the execution of the predetermined motion of the excavator 100. In addition, the data obtained during the execution of the predetermined motion of the excavator 100 may include data representing a reaction force applied from the work target to the working part during the execution of the predetermined motion of the excavator 100. Further, the input data of the learned model LM4 may include data representing characteristics of soil of the work target of the excavator 100, as in the learned model LM1 or the like.

The teacher data is a combination of the input data of the learned model LM4 and data (ground truth) representing the correct inference result corresponding to the input data. The teacher data includes, as the ground truth, data (ground truth) representing correct certainty of the shape of the work target before the execution of the predetermined motion of the excavator 100.

The machine learning unit 2005 includes machine learning units 2005A to 2005D.

The machine learning unit 2005C causes the base learning model M3 to be trained by machine learning to generate a learned model LM3 based on the teacher dataset output from the teacher data generation unit 2004C. In addition, the machine learning unit 2005C may cause the learned model LM3 to undergo additional learning to correct the learned model LM3 to reduce an error between the output (inference result) of the learned model LM3 and the actual reaction force corresponding to the shape of the work target.

The machine learning unit 2005D causes the base learning model M4 to be trained by machine learning to generate a learned model LM4 based on the teacher dataset output by the teacher data generation unit 2004D. Moreover, the machine learning unit 2005D may cause the learned model LM4 to undergo additional learning to correct the learned model LM4 to reduce an error between the output (inference result) of the learned model LM4 and the actual certainty.

In the learned model storage unit 2006, the learned models LM1 to LM4, which are output from the machine learning unit 2005, are stored. In the case where relearning or additional learning of the learned model LM1 is performed by the machine learning unit 2005A, the learned model LM1 of the learned model storage unit 2006 will be updated. The same applies to the case where relearning or additional learning of each of the learned models LM2 to LM4 is performed by the respective machine learning unit 2005B to 2005D. In addition, the learned models LM1 to LM4 before updating may be stored in the learned model storage unit 2006 or another storage unit in a reusable form. Thus, for example, in the case where there is a problem in the learned models LM1 to LM4 after updating, the learned models LM1 to LM4 before updating can be restored and reused.

The distribution unit 2007 distributes the data of the learned models LM1 to LM4 to the excavator 100.

The method of distributing the learned models LM3 and LM4 may be the same as the above-described method of distributing the learned models LM1 and LM2.

The work supporting unit 302 is a functional unit configured to support the work of the excavator 100 performed by the autonomous driving function as in the above-described example 1.

The work supporting unit 302 includes the learned model storage unit 302A, the work target shape acquisition unit 302B, the target trajectory generation unit 302C, the motion control unit 302D, a reaction force estimation unit 302F, and a certainty estimation unit 302G.

In the learned model storage unit 302A, the learned models LM1 to LM4, which are distributed from the information processing device 200 and received via the communication device 60, are stored.

The reaction force estimation unit 302F estimates the reaction force applied to the working part at the predetermined portion of the excavator 100 based on the state of the work target before the execution of the predetermined motion of the excavator 100 and the trajectory of the working part during the execution of the predetermined motion of the excavator 100.

For example, the reaction force estimation unit 302F estimates the reaction force applied to the working part at the predetermined motion of the excavator 100 using the learned model LM3 based on the shape of the work target before the execution of the predetermined motion of the excavator 100 and the trajectory of the working part during the execution of the predetermined motion of the excavator 100.

The work target shape acquisition unit 302B estimates the shape of the work target of the excavator 100 based on the difference between the estimation result of the reaction force applied to the working part at the predetermined motion of the excavator 100, which is provided by the reaction force estimation unit 302F, and the actual measurement result of the reaction force, and acquires data representing the estimated shape. Specifically, the work target shape acquisition unit 302B corrects the estimated shape of the work target before the execution of the predetermined motion of the excavator 100 based on the difference (estimation error) between the estimation result of the reaction force applied to the working part at the predetermined motion of the excavator 100 and the actual measurement result of the reaction force. The measurement data of the reaction force of the excavator 100 is calculated, for example, from the measurement data of sensors S7 to S9. Then, the work target shape acquisition unit 302B estimates the shape of the work target after the execution of the predetermined motion of the excavator 100 based on the corrected estimate result of the shape of the work target before the execution of the predetermined motion of the excavator 100 and the trajectory of the working part during the execution of the predetermined motion of the excavator 100.

For example, as illustrated in FIG. 9, the work target shape acquisition unit 302B estimates the shape of the work target after each execution of the predetermined motion of the excavator 100 using the learned model LM1, with the data representing the initial shape of the work target, which is acquired in advance, as a starting point. Hereinafter, in order to distinguish from the actual initial shape, the initial shape corresponding to the data representing the initial shape of the work target of the excavator 100 may be referred to as a “provisional initial shape” for convenience. In addition, the reaction force estimation unit 302F estimates, using the learned model LM3, the reaction force applied to the working part of the excavator 100 based on the data representing the estimation result (estimated shape of soil) of the shape of the work target before each execution of the predetermined motion of the excavator 100 and the data representing the trajectory (path) of the working part in each execution. For the above estimation, the data representing the estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100 is the data representing the initial shape of the work target in the case of the first execution of the predetermined motion of the excavator 100. In addition, the trajectory of the working part in each execution may be data representing the target trajectory generated by the target trajectory generation unit 302C, or the measurement data of the trajectory based on the outputs of the sensors S1 to S3.

In addition, the work target shape acquisition unit 302B calculates the estimation error of the reaction force to the working part at each execution of the predetermined motion of the excavator 100 based on the estimation result (estimated reaction force) of the reaction force acquired by the reaction force estimation unit 302F and data of the actual measurement result of the reaction force.

Then, the work target shape acquisition unit 302B corrects the data representing the initial shape of the work target of the excavator 100 based on the estimation error of the reaction force applied to the working part at each execution of the predetermined motion of the excavator 100. Specifically, the work target shape acquisition unit 302B applies a known mathematical programming algorithm to correct the provisional initial shape of the work target to minimize the estimation error of the reaction force.

For example, there is a case where the certainty of the provisional initial shape of the soil of the work target is low, and the provisional initial shape may be set at a lower position than the height of the actual initial shape. As a result, as illustrated in FIG. 10, the working part may not abut on the soil at all and may strike only air during the predetermined motion of the excavator 100. Alternatively, as illustrated in FIG. 11, the working part of the excavator 100 may abut on the soil of the work target only at a part of the entire transport section to be in contact with the working part. Moreover, the value of the measured reaction force becomes smaller than the estimated reaction force throughout the predetermined motion of the excavator 100, and the estimation error obtained by subtracting the measured reaction force from the estimated reaction force becomes a positive and relatively large value. In this case, the work target shape acquisition unit 302B corrects the height position of the ground of the work target to become high at least in the small regions through which the working part passes during the predetermined motion of the excavator 100 within the observation target region of the work target of the excavator 100.

On the other hand, there is also a case where the certainty of the provisional initial shape of the soil of the work target is low, and the position of the provisional initial shape is set at the position higher than the height of the actual initial shape. As a result, as illustrated in FIG. 12, the working part may move along the trajectory that passes through a relatively deep position from the ground of the work target during the predetermined motion of the excavator 100. In addition, the value of the measured reaction force becomes larger than the reaction force estimated from the initial stage of the predetermined motion of the excavator 100. In the present example, the measured reaction force reaches the allowable upper limit value so that the excavator 100 stops the predetermined motion, and performs an avoidance operation to move the working part in the direction away from the work target (see the broken line portion in the figure). In this case, the work target shape acquisition unit 302B corrects the height position of the work target to be low at least in the small regions through which the working part passes during the predetermined motion of the excavator 100 (excluding the small regions through which the working part passes due to the avoidance motion) within the observation target region of the work target.

The work target shape acquisition unit 302B sequentially estimates the shape of the work target after each execution of the predetermined motion of the excavator 100 based on the data representing the trajectory (path) of the working part at each execution of the predetermined motion of the excavator 100 with the data representing the corrected initial shape of the work target as a starting point. Thus, the work target shape acquisition unit 302B can estimate the shape of the work target after the predetermined motion of the excavator 100 is executed for the n-th time, and can acquire the data representing the latest shape of the work target.

Each time the predetermined motion of the excavator 100 is completed, the work target shape acquisition unit 302B corrects the data representing the initial shape of the work target, estimates the latest shape of the work target based on the data representing the corrected data representing the shape of the work target, and acquires data representing the latest shape. Thus, even in the case where there is a deviation between the provisional initial shape and the actual initial shape at the start of work, the provisional initial shape can be sequentially corrected every time the predetermined motion of the excavator 100 is executed, with the provisional initial shape of the work target as a starting point. Accordingly, the work target shape acquisition unit 302B can sequentially reduce an error between the data representing the initial shape of the work target of the excavator 100 and data of the actual initial shape, and as a result, can estimate the latest shape of the work target of the excavator 100 with relatively high certainty. Therefore, for example, even in the case where the measurement data representing the initial shape cannot be acquired externally, or in the case where the measurement data representing the initial shape, which is externally acquired, has relatively low certainty for some reasons, the excavator 100 can use the data representing the shape of the work target with relatively high certainty.

The certainty estimation unit 302G estimates the certainty of the estimation result of the shape of the work target of the excavator 100. The estimation result of the shape of the work target of the excavator 100 may be the estimation result of the latest shape of the work target, which is estimated by the work target shape acquisition unit 302B, or the estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100. Moreover, the estimation result of the shape of the work target of the excavator 100 may be the provisional initial shape of the work target of the excavator 100. For example, the certainty estimation unit 302G estimates the certainty of the corrected shape of the work target before the execution of the predetermined motion of the excavator 100 based on the data representing the trajectory of the working part during the execution of the predetermined motion of the excavator 100 or the data representing the reaction force applied to the working part. This is because, as described above, the shape of the work target before the execution of the predetermined motion of the excavator 100 is corrected based on the data representing the trajectory of the working part during the execution of the predetermined motion of the excavator 100 or the data representing the reaction force applied to the working part. For the above estimation, the certainty of the estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100 is, for example, the certainty of the corrected provisional initial shape of the work target. Specifically, the certainty estimation unit 302G estimates, using the learned model LM4, the certainty of the corrected estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100. In addition, the certainty estimation unit 302G may estimate, on a rule basis, the certainty of the corrected estimation result of the shape of the work target before the execution of the predetermined motion of the excavator 100. For example, the certainty estimation unit 302G determines, per small region of the observation target region of the work target, whether the working part abuts on the work target (ground) during the execution of the predetermined motion of the excavator 100. Then, in the case where the working part abuts on the work target in the target small region, the certainty estimation unit 302G relatively largely increases the current value of the certainty. In the case where the working part does not abut on the work target, the certainty estimation unit 302G relatively slightly increases the current value of the certainty or maintains the current value of the certainty. This is because the shape of the work target before the execution of the predetermined motion of the excavator 100 is highly likely to be corrected mainly for the small region on which the working part abuts during the execution of the predetermined motion of the excavator 100.

The motion control unit 302D causes the excavator 100 to perform the predetermined motion so that the working part of the excavator 100 moves along the target trajectory generated by the target trajectory generation unit 302C. For the above motion control, the motion control unit 302D may restrict the motion of the excavator 100 based on the certainty of the estimation result of the shape of the work target of the excavator 100 estimated by the certainty estimation unit 302G. This is because, in the case where the certainty of the estimation result of the shape of the work target of the excavator 100 is relatively low, the reaction force applied to the working part becomes a problem due to the inconsistency between the estimation result of the shape of the work target of the excavator 100 and the target trajectory, and a large impact may be applied to the excavator 100. Specifically, the motion control unit 302D may calculate the average of the certainty (average certainty) of the small regions on the target trajectory on the top view of the excavator 100. In the case where the average certainty is relatively low, the motion control unit 302D may increase the degree of the restriction of the motion of the excavator 100. In the case where the average certainty is relatively high, the motion control unit 302D may decrease the degree of the restriction of the motion of the excavator 100.

For example, as illustrated in FIG. 13, the motion control unit 302D changes the speed ratio corresponding to the speed of the predetermined motion of the excavator 100 based on the average certainty. In the present example, the speed ratio is set to 1 in the range where the average certainty exceeds the threshold TH1 (<1), and is set to linearly decrease with respect to the decrease in the average certainty in the range where the average certainty is the threshold TH1 or less. Thus, in the case where the average certainty is the threshold TH1 or less, the motion control unit 302D restricts the motion speed of the excavator 100 so that the predetermined motion of the excavator 100 can be executed at a slower speed than the speed of the motion without the speed restriction. Therefore, even if the working part unintentionally abuts on the work target due to the inconsistency between the estimated result of the shape of the work target of the excavator 100 and the target trajectory, the impact applied to the excavator 100 can be suppressed to be relatively small.

For example, as illustrated in FIG. 14, the motion control unit 302D changes the excavation depth ratio corresponding to the excavating motion of the excavator 100 based on the average certainty. In the present example, the excavation depth ratio is set to 1 in the range where the average certainty exceeds the threshold TH2 (<1), and is set to linearly decrease with respect to the decrease in the average certainty in the range where the average certainty is the threshold TH2 or less. Thus, in the case where the average certainty is the threshold TH2 or less, the motion control unit 302D restricts the excavation depth at the excavating motion of the excavator 100 so that the movable range of the working part toward the work target side (ground side) can be restricted as compared to the range without the motion restriction. Therefore, even if the certainty of the estimation result of the shape of the work target of the excavator 100 is low and the height position of the actual ground is higher than the estimation, it is possible to suppress a situation in which the working part penetrates the ground into a very deep position and a relatively large impact is applied to the excavator 100. For example, the motion control unit 302D corrects the target trajectory generated by the target trajectory generation unit 302C, thereby restricting the excavation depth. In addition, the target trajectory generation unit 302C may generate the target trajectory of the excavation depth according to the average certainty, the motion control unit 302D may cause the excavator 100 to execute the predetermined motion based on the generated target trajectory, and as a result, the motion control unit 302D may restrict the excavation depth.

The motion control unit 302D may restrict, for the predetermined motion of the excavator 100 other than the excavating motion, the motion range of the working part in the vertical direction during the predetermined motion of the excavator 100 based on the average certainty.

As described above, in the present example, the controller 30 acquires the data that represents the state of the excavator 100 and is related to the shape of the work target in association with the predetermined motion of the excavator 100. Then, the controller 30 estimates the shape of soil of the work target (ground) of the excavator 100 based on the acquired data. Specifically, the controller 30 acquires the data representing the trajectory (path) during the execution of the predetermined motion of the excavator 100, and estimates the reaction force applied to the working part during the execution of the predetermined motion of the excavator 100 based on the acquired data. In addition, the controller 30 acquires the measurement data of the reaction force applied to the working part at the predetermined motion of the excavator 100, and corrects the shape of the work target before the execution of the predetermined motion of the excavator 100 based on a difference (estimation error) between the measurement data of the reaction force and the estimation data. Then, the controller 30 estimates the shape of the work target after the execution of the predetermined motion of the excavator 100 based on the data representing the corrected shape of the work target before the execution of the predetermined motion of the excavator 100 and the data representing the trajectory during the execution of the predetermined motion of the excavator 100. Thus, even in the case where the certainty of the shape of the work target before the execution of the predetermined motion of the excavator 100 is low, the shape of the work target before the execution of the predetermined motion of the excavator 100 can be corrected in accordance with the predetermined motion of the excavator 100. Thus, the controller 30 can more appropriately grasp the shape of the soil of the work target of the excavator 100 within the observation target region, and as a result, the target trajectory of the working part of the excavator 100 can be more appropriately generated. Therefore, the controller 30 can more appropriately execute the autonomous operation of the excavator 100.

Some or all of the functions of the reaction force estimation unit 302F and the certainty estimation unit 302G may be transferred to the information processing device 200. Thus, the processing load of the excavator 100 can be reduced.

OTHER EXAMPLES

The above-described first to third examples of the functional configuration of the operation support system SYS may be modified or changed as appropriate.

For example, in the above-described second example of the functional configuration of the operation support system SYS, the sensor 40 may be omitted from the excavator 100, the work target shape acquisition unit 302B may estimate the shape of the work target of the excavator 100 by the same method as in the above-described third example to acquire data representing the shape of the work target. In this case, the same functional unit as the reaction force estimation unit 302F of the above-described third example is added to the work supporting unit 302. In addition, in this case, the functions of the certainty estimation unit 302G and some of functions of the motion control unit 302D of the above-described third example, i.e., the function of restricting the predetermined motion of the excavator 100 based on the certainty of the estimation result of the shape of the work target of the excavator 100, may be added to the work supporting unit 302.

In addition, in the above-described third example of the functional configuration of the operation support system SYS, the functions of the certainty estimation unit 302G and some of the functions of the motion control unit 302D (the function of restricting the predetermined motion of the excavator 100 based on the certainty of the estimation result of the shape of the work target of the excavator 100) may be omitted.

[Specific Example of Process Related to Use of Data Representing Shape of Work Target]

Next, specific examples of the process related to use of the data representing the shape of the work target of the excavator 100, which is acquired by the work target shape acquisition unit 302B, will be described with reference to FIGS. 15 to 18.

First Example

FIG. 15 is a diagram illustrating a first example of the process related to the use of the data representing the shape of the work target. Specifically, FIG. 15 is a diagram illustrating one example of a process in which the controller 30 causes the excavator 100 to autonomously operate based on the data representing the shape of the work target, thereby performing excavation work in the work target region.

The excavation work is performed by repeating a series of motions including an excavating motion, a boom-raising and turning motion, a soil-dumping motion, and a boom-lowering and turning motion. The excavating motion is a motion of the attachment AT for excavating the soil of the work target with the bucket 6. For example, the excavating motion is implemented as combined motion of at least two or more driven elements of the boom 4, the arm 5, and the bucket 6. The boom-raising and turning motion includes the motions of the attachment AT and the upper turning body 3 for scooping up the soil, which has been excavated by the excavating motion, by the bucket 6 and carrying the soil to a soil dumping location away from the work target region. For example, the boom-raising and turning motion is implemented as a combined motion composed of the raising motion of the boom 4 and the turning motion of the upper turning body 3. The soil-dumping motion is a motion of the attachment AT for dumping the soil in the bucket 6 onto the ground of the soil dumping location. The soil-dumping motion is, for example, implemented as a combined motion composed of the opening motion of the arm 5 and the opening motion of the bucket 6. The boom-lowering and turning motion includes motions of the attachment AT and the upper turning body 3 for returning the bucket 6 to the work target region. The boom-lowering and turning motion is, for example, implemented as a combined motion composed of the lowering motion of the boom 4 and a turning motion of the upper turning body 3.

This flowchart starts, for example, when an input indicating the start of excavation work by the autonomous operation of the excavator 100 is received. The input may be performed, for example, by an operator or the like of the cabin 10 via the input device 52, or may be performed from the outside of the excavator 100 (by the information processing device 200 or the remote operation support device 400) via the communication device 60. Hereinafter, the same may apply to an input indicating the start of the loading work by the autonomous operation of the excavator 100.

As illustrated in FIG. 15, in step S10, the controller 30 sets a target region (work target region) of the excavation work. The target region of the excavation work is, for example, defined by a range in which the excavation work is performed in a plan view, and a depth of the excavation. Within the work target region, the range of the excavation work in a plan view corresponds to the observation target region of the work target shape acquisition unit 302B. The work target region may be set in accordance with the input from the operator or the like of the cabin 10 via the input device 52, or may be set by an input from the outside of the excavator 100 (for example, by the information processing device 200 or the remote operation support device 400) via the communication device 60.

When the process of step S10 is completed, the controller 30 proceeds to step S11.

In step S11, the controller 30 sets an initial shape of the work target. For example, in the case where the sensor 40 is mounted on the excavator 100, the initial shape of the work target is set based on the measurement data acquired by the sensor 40 before the start of work. In the case where the sensor 40 is not mounted on the excavator 100, the initial shape of the work target may be set to a provisional shape, which is defined in advance. The provisional shape is, for example, a flat shape obtained by extending the ground contact plane of the lower traveling body 1 of the excavator 100 to the work target region. In addition, the initial shape of the work target may be manually set in accordance with an input from an operator or the like. Further, the initial shape of the work target may be set based on the measurement data of the shape of the work target near the excavator 100, which is input from the outside (for example, by the information processing device 200 or the remote operation support device 400) via the communication device 60.

When the process of step S11 is completed, the controller 30 proceeds to step S12.

In step S12, the target trajectory generation unit 302C of the controller 30 generates a target trajectory of the working part of the excavator 100 based on the data representing the estimation result of the current shape of the work target.

In the case where the process proceeds from step S11 to step S12, the data representing the estimation result of the current shape of the work target is the data representing the initial shape of the work target set in S12. In addition, in the case where the process proceeds from step S16 to step S12, the data representing the estimation result of the current shape of the work target is data representing the latest shape of the work target acquired by the work target shape acquisition unit 302B.

When the process of step S12 is completed, the controller 30 proceeds to step S13.

In step S13, the motion control unit 302D of the controller 30 sets motion conditions of the excavating motion of the excavator 100. For example, the motion control unit 302D may set the speed of the excavating motion based on the certainty of the estimation result of the work target, which is estimated by the certainty estimation unit 302G. In addition, the motion control unit 302D may set the condition of the excavation depth of the excavating motion based on the certainty of the estimation result of the shape of the work target, which is estimated by the certainty estimation unit 302G.

When the process of step S13 is completed, the controller 30 proceeds to step S14.

In step S14, the motion control unit 302D causes the excavator 100 to execute the excavating motion based on the target trajectory generated in step S12 and the motion conditions set in step S13.

When the process of step S14 is completed, the controller 30 proceeds to step S15.

In step S15, the controller 30 performs a process for acquiring data (soil shape data) representing the shape of the work target in the observation target region defined based on the work target region set in step S10. Details of this process will be described later.

When the process of step S15 is completed, the controller 30 proceeds to step S16.

In step S16, the controller 30 determines whether or not a condition indicating the end of the work (work ending condition) is satisfied. For example, the controller 30 compares the setting content for the depth of the work target region and the height position of the soil in the work target region in the latest soil shape data acquired in S15, and determines that the work ending condition is satisfied when the excavation has progressed to the set depth of the work target region. When the work ending condition is not satisfied, the controller 30 returns to step S12, repeats the processing of step S12 to step S16, and ends the process of this flowchart when the work ending condition is satisfied.

As described above, in the present example, the controller 30 can cause the excavator 100 to autonomously perform the excavation work in the work target region.

Second Example

FIG. 16 is a diagram illustrating a second example of the process related to use of the data representing the shape of the work target. Specifically, FIG. 16 is a diagram illustrating one example of a process in which the controller 30 causes the excavator 100 to autonomously operate based on data representing the shape of the work target, thereby performing loading work of soil onto a truck.

The loading work onto the truck is performed by repeating a series of motions including an excavating motion, a boom-raising and turning motion, a soil-dumping motion, and a boom-lowering and turning motion. The excavating motion is a motion of the attachment AT for scooping the soil in the predetermined location of the ground by the bucket 6. For example, the excavating motion is implemented by a combined motion of at least two driven elements of the boom 4, the arm 5, and the bucket 6. The boom-raising and turning motion includes motions of the attachment AT and the upper turning body 3 for carrying the soil scooped by the bucket 6 onto the leading bed of the truck. For example, the boom-raising and turning motion is implemented as a combined motion of the raising motion of the boom 4 and the turning motion of the upper turning body 3. The soil-dumping motion is the motion of the attachment AT for dumping the soil of the bucket 6 onto the loading bed of the truck. The soil-dumping motion is implemented, for example, as a combined motion of the opening motion of the arm 5 and the opening motion of the bucket 6. The boom-lowering and turning motion includes the motions of the attachment AT and the upper turning body 3 for returning the bucket 6 to the predetermined location for scooping the soil. The boom-lowering and turning motion is implemented, for example, as a combined motion of the lowering motion of the boom 4 and the turning motion of the upper turning body 3. Moreover, in the present example, the description will be made on the premise that the position and shape of a pile of soil (soil pile) to be loaded, which is generated as a result of the excavation work, are known. This is because, during the excavation work, the position at which the excavator 100 dumps the soil is determined in advance with the excavator 100 as a reference, and a schematic shape of the pile of soil generated can be estimated considering the size or depth of the work target region.

This flowchart starts, for example, when an input indicating the start of loading work by the autonomous operation of the excavator 100 is received.

As illustrated in FIG. 16, in step S20, the controller 30 detects a relative position of a truck to be loaded with soil, with respect to the excavator 100. In the case where the absolute positions of the excavator 100 and the truck can be grasped, for example, the controller 30 detects a relative position of the truck with respect to the excavator 100 based on the absolute positions of the excavator 100 and the truck. In this case, the controller 30 detects a relative position of the truck with respect to the excavator 100, for example, based on the output of the GNSS sensor mounted on the excavator 100 and the position information of the truck received from the truck via the communication device 60. In addition, the controller 30 may receive the information of the relative position of the truck with respect to the excavator 100, which is acquired based on the outputs of the imaging device, the distance sensor, or the like installed in the work site, from an external device (e.g., the information processing device 200, etc.) via the communication device 60. Further, in the case where the sensor S6 is mounted on the excavator 100, the controller 30 recognizes the truck based on the output of the sensor S6 to detect a relative position of the truck with respect to the excavator 100.

When the process of step S20 is completed, the controller 30 proceeds to step S21.

In step S21, the controller 30 sets an initial shape of the work target. In the present example, the controller 30 estimates the position of the loading bed of the truck based on the position of the truck with respect to the excavator 100, which is detected at step S20, and sets the bottom surface of the loading bed of the truck as an initial shape with the loading bed of the truck as a work target.

In the case where the orientation of the truck when stopped with respect to the excavator 100 is not determined, in step S20, a relative orientation of the truck with respect to the excavator 100 may be detected. For example, in the case where the orientations of the excavator 100 and the truck can be grasped, the controller 30 detects a relative orientation of the truck with respect to the excavator 100 based on the orientations of the excavator 100 and the truck. In this case, the controller 30 detects the orientation of the truck with respect to the excavator 100, for example, based on the output of the sensor and the information of the orientation of the truck, which is received from the truck via the communication device 60. In addition, the controller 30 may receive information of a relative orientation of the truck with respect to the excavator 100, which is acquired based on the outputs of the imaging device, distance sensor, or the like installed in the work site, from an external device (e.g., the information processing device 200, etc.) via the communication device 60. Further, in the case where the sensor S6 is mounted on the excavator 100, the controller 30 may recognize the truck based on the output of the sensor S6, thereby detecting a relative orientation of the truck with respect to the excavator 100.

When the process of step S21 is completed, the controller 30 proceeds to step S22.

In step S22, the target trajectory generation unit 302C of the controller 30 generates a target trajectory of the working part of the excavator 100 based on the data representing the estimation result of the current shape of the work target.

In the case where the process proceeds from step S21 to step S22, the data representing the estimation result of the current shape of the work target is the data representing the initial shape of the work target set in step S22. In the case where the process proceeds from step S25 to step S22, the data representing the estimation result of the current shape of the work target is data representing the latest shape of the work target acquired by the work target shape acquisition unit 302B.

When the process of step S22 is completed, the controller 30 proceeds to step S23.

In step S23, the motion control unit 302D of the controller 30 causes the excavator 100 to execute a series of motions including a combination of an excavating motion, a boom-raising and turning motion, a soil-dumping motion, and a boom-lowering and turning motion in accordance with the target trajectory generated in S22.

When the process of step S23 is completed, the controller 30 proceeds to step S24.

In step S24, the controller 30 performs a process for acquiring data (soil shape data) representing the shape of the work target in the observation target region corresponding to the loading bed of the truck, which is detected in step S20. Details of this process will be described later.

When the process of step S24 is completed, the controller 30 proceeds to step S26.

In step S26, the controller 30 determines whether or not a condition indicating the end of the work (work ending condition) is satisfied. For example, the controller 30 determines that the loading work of the soil onto the truck is ended when the height of the soil from the bottom surface of the loading bed is relatively large with respect to a predetermined reference based on the data representing the shape of the work target. When the work ending condition is satisfied, the controller 30 ends the process of this flowchart. When the work ending condition is not satisfied, the controller 30 returns to step S22, and repeats the processing of step S22 to step S26.

In the present example, in step S24, in addition to the data representing the shape of the soil (soil shape data) on the loading bed of the truck, data representing the shape of the pile of the dumped soil (soil shape data) may be acquired as the data representing the shape of the work target. In this case, in step S22, the target trajectory generation unit 302C of the controller 30 generates a target trajectory of a series of motions based on both the soil shape data of the pile of the dumped soil and the soil shape data of the loading bed of the truck acquired in step S24.

As described above, in the present example, the controller 30 enables execution of the autonomous loading work in the work target region.

Third Example

FIG. 17 is a diagram illustrating a third example of the process related to use of the data representing the shape of the work target. Specifically, FIG. 17 is a diagram illustrating one example of a process in which the controller 30 causes the display device 50A or the like to display an image representing the shape of the work target based on the data representing the shape of the work target when the excavator 100 executes excavation work with an excavating motion.

This flowchart starts, for example, when an input indicating the start of excavation work, which is manually input by an operator, is received. The input may be, for example, made by an operator of the cabin 10 via the input device 52, for may be made by an operator who performs a remote operation via the remote operation support device 400. In the latter case, the content input in the remote operation support device 400 is received by the controller 30 via the communication device 60.

As illustrated in FIG. 17, the processing of steps S30 and S31 is the same as the processing of steps S10 and S11 of FIG. 15, and therefore the description will be omitted.

When the process of step S31 is completed, the controller 30 proceeds to step S32.

In step S32, the display processing unit 302E of the controller 30 causes the display device 50A or the display device of the remote operation support device 400 to display an image representing the initial shape of soil of the work target (initial shape image) based on the data representing the initial shape set in step S31.

The display processing unit 302E of the controller 30 transmits a display command including the data representing the initial shape to the remote operation support device 400 via the communication device 60, thereby causing the display device of the remote operation support device 400 to display the initial shape image. The same may apply to below-described step S42 of FIG. 18.

When the process of step S32 is completed, the controller 30 proceeds to step S33.

In step S33, the controller 30 performs monitoring of the motion of the excavator 100 based on the outputs of the sensors S1 to S9.

When the process of step S33 is completed, the controller 30 proceeds to step S34.

In step S34, the controller 30 determines whether or not the excavating motion has been executed and completed. When the excavating motion has been executed and completed, the controller 30 proceeds to step S35. In any other cases, the controller 30 proceeds to step S37.

In step S35, the controller 30 performs a process for acquiring data (soil shape data) representing the shape of the work target in the observation target region defined based on the work target region set in step S30. Details of this process will be described later.

When the process of step S35 is completed, the controller 30 proceeds to step S36.

In step S36, the display processing unit 302E of the controller 30 updates the image representing the shape of soil of the work target (soil shape image) displayed on the display device 50A or the display device of the remote operation support device 400 based on the soil shape data acquired in step S35.

The display processing unit 302E of the controller 30 transmits an update command including the data representing the latest soil shape data to the remote operation support device 400 via the communication device 60, thereby updating the soil shape image of the display device of the remote operation support device 400. Hereinafter, the same may apply to below-described step S46 of FIG. 18.

When the process of step S36 is completed, the controller 30 proceeds to step S37.

In step S37, the controller 30 determines whether or not a condition indicating the end of the display of the soil shape image (display ending condition) is satisfied. For example, the display ending condition is the receipt of a predetermined input from an operator or the like via the input device 52 or the input device of the remote operation support device 400. In addition, the display ending condition may be continuation of the state in which no excavating motion is executed for a certain period or longer. When the display ending condition is not satisfied, the controller 30 returns to step S33, and repeats processing of step S33 to step S37. When the display ending condition is satisfied, the controller 30 ends the process of this flowchart.

As described above, in the present example, the controller 30 can display the image representing the shape of the soil in the work target region on the display device 50A or display device of the remote operation support device 400, while updating the image representing the shape of the soil in the work target region in accordance with the progress of the excavation work in the work region.

Fourth Example

FIG. 18 is a diagram illustrating a fourth example of the process related to the data representing the shape of the work target. Specifically, FIG. 18 is a diagram illustrating one example of a process in which the controller 30 causes the display device 50A or the like to display an image representing the shape of the work target based on the data representing the shape of the work target when the excavator 100 executes loading work of soil onto a truck.

This flowchart starts, for example, when an input indicating the start of loading work, which is manually input by an operator, is received. The input may be, for example, made by an operator of the cabin 10 via the input device 52, or may be made by an operator who performs a remote operation via the remote operation support device 400. In the latter case, the content of the input in the remote operation support device 400 is received by the controller 30 via the communication device 60.

As illustrated in FIG. 18, the processing of steps S40 and S41 is the same as the processing of steps S20 and S21 of FIG. 16, and therefore the description will be omitted.

When the process of step S41 is completed, the controller 30 proceeds to step S42.

In step S42, the display processing unit 302E of the controller 30 causes the display device 50A or the display device of the remote operation support device 400 to display an image representing the initial shape of soil of the work target (initial shape image) based on the data representing the initial shape set in step S41.

When the process of step S42 is completed, the controller 30 proceeds to step S43.

In step S43, the controller 30 performs monitoring of the motion of the excavator 100 based on the outputs of the sensors S1 to S9.

When the process of step S43 is completed, the controller 30 proceeds to step S44.

In step S44, the controller 30 determines whether the soil-dumping motion has been executed and completed. When the soil-dumping motion has been executed and completed, the controller 30 proceeds to step S45. In any other cases, the controller 30 proceeds to step S47.

In step S45, the controller 30 performs a process for acquiring data (soil shape data) representing the shape of the work target in the observation target region corresponding to the loading bed of the truck detected in step S40. Details of this process will be described later.

When the process of step S45 is completed, the controller 30 proceeds to step S46.

In step S46, the display processing unit 302E of the controller 30 updates the image representing the shape of soil of the work target (soil shape image) displayed on the display device 50A or the display device of the remote operation support device 400 based on the soil shape data acquired in step S45.

When the process of step S46 is completed, the controller 30 proceeds to step S47.

In step S47, the controller 30 determines whether or not a condition indicating the end of the display of the soil shape image (display ending condition) is satisfied. For example, the display ending condition is the receipt of a predetermined input from an operator or the like via the input device 52 or the input device of the remote operation support device 400. In addition, the display ending condition may be continuation of the state in which no excavating motion is executed for a certain period or longer. When the display ending condition is not satisfied, the controller 30 returns to step S43, and repeats the processing of step S43 to step S47. When the display ending condition is satisfied, the controller 30 ends the process of this flowchart.

As described above, in the present example, the controller 30 can display the image representing the shape of the soil on the loading bed of the truck on the display device 50A or the display device of the remote operation support device 400, while updating the image representing the shape of the soil on the loading bed of the truck in accordance with the progress of the loading work of the soil onto the loading bed of the truck as the work target.

[Specific Examples of Process Related to Acquisition of Data Representing Work Target]

Specific examples of processing performed by the work target shape acquisition unit 302B will be described with reference to FIGS. 16 to 18.

First Example

FIG. 19 is a flowchart schematically illustrating a first example of a process for acquiring the data representing the shape of the work target. FIG. 20 is a diagram illustrating one example of the observation target region. FIG. 21 is a diagram illustrating one example of the influence region.

In the present example, the description will be made on the premise that the functional configuration of the operation support system SYS is the above-described first example (FIG. 6) or second example (FIG. 7), and the predetermined motion of the excavator 100 is an excavating motion.

<<Process for Acquiring Shape of Work Target>>

The flowchart of FIG. 19 is performed, for example, after the execution of the excavating motion of the excavator 100. Specifically, the flowchart of FIG. 19 corresponds to the processing of step S15 of FIG. 15 or the processing of step S35 of FIG. 17 described above. Hereinafter, the description will be made on the premise that, in this flowchart, the data representing the shape of the work target (soil shape) after the execution of the k-th excavating motion of the excavator 100 (k: positive integer) from the start of the excavation work, is acquired.

For example, as illustrated in FIG. 20, the observation target region TA near the excavator 100 is divided into a predetermined number N of grids. The observation target region TA is a region near the excavator 100 for which the work target shape acquisition unit 302B acquires data representing the soil shape. In the present example, as the data representing the soil shape, a height h_e^k of the soil and uncertainty s_e^k corresponding the estimation certainty of the height are acquired for each grid i (i=1 to N) of the observation target region TA. The height h_e^k of the soil and the uncertainty s_e^k are represented by the following equations (1) and (2), respectively.

h e k = [ h e , 1 k , h e , 2 k ,   … ,   h e , N k ] T ( 1 ) s e k = [ s e , 1 k , s e , 2 k   , … ,   s e , N k ] ( 2 )

In step S102, the work target shape acquisition unit 302B inputs the data of the soil shape before the execution of the excavating motion of the excavator 100. The data of the soil shape before the execution of the excavating motion of the excavator 100 corresponds to the data of the soil shape before the execution of the (k−1)th excavating motion (height h_e^k-1 of soil and uncertainty s_e^k-1).

In the case of the first excavating motion (k=1), the work target shape acquisition unit 302B inputs the initial values (height h_e⁰ of soil and uncertainty s_e⁰) of the data of the soil shape.

The initial values of the data of the soil shape may be acquired based on the outputs of the sensor 40 (geometry sensor), may be acquired from outside the excavator 100, or may be provisional values defined in advance. The provisional values defined in advance are, for example, zero (0) assuming that the entire (all grids i) observation target region TA is the same height as the ground on which the excavator 100 is located.

When the process of step S102 is completed, the work target shape acquisition unit 302B proceeds to step S104.

In step S104, the work target shape acquisition unit 302B inputs a log of the current excavating motion of the excavator 100 for each of the grids i of the observation target region TA. The log includes a height b^k of the blade edge (claw tip) of the bucket 6, an angle φ^k of the blade edge (claw tip) of the bucket 6, an excavation reaction force f^k, soil characteristics λ^k, and influence information κ^k. In addition, the log may include other information ω^k.

The height b^k of the blade edge of the bucket 6 refers to the height at which the blade edge of the bucket has passed through during the current excavating motion of the excavator for each of the grids of the observation target region TA and is represented by the following equation (3).

b k = [ b 1 k ,   b 2 k , … ,   b N k ] T ( 3 )

The height b^k of the blade edge of the bucket 6 may not be input for the grids i corresponding to the locations where the bucket 6 has not passed through, or an appropriate value may be input. The same applies to the angle φ^k of the blade edge of the bucket 6 and the excavation reaction force f^k described later. The height b^k of the blade edge of the bucket 6 is acquired, for example, based on the data of the trajectory of the bucket 6 during the current excavating motion of the excavator 100. The data of the trajectory of the bucket 6 is acquired, for example, based on the time-series data of the outputs of sensors S1 to S6 during the current excavating motion of the excavator 100.

The angle φ^k of the blade edge of the bucket 6 refers to an angle of the blade edge of the bucket 6 with respect to a predetermined reference surface (e.g., a horizontal surface) when the blade edge of the bucket 6 passes during the current excavating motion of the excavator 100 for each of the grids i of the observation target region TA, and is represented by the following equation (4).

φ k = [ φ 1 k , φ 2 k ,   … ,   φ N k ] T ( 4 )

The angle φ^k of the blade edge of the bucket 6 is acquired, for example, based on the data of the trajectory of the bucket 6 during the current excavating motion of the excavator 100 and the time-series data of the attitude angle of the bucket 6. The time-series data of the attitude angle of the bucket 6 is acquired, for example, based on the time-series data of the output of the sensor S3 during the current excavating motion of the excavator 100.

The excavation reaction force f^k refers to a reaction force applied from the ground to the bucket 6 by the current excavating motion of the excavator 100 for each of the grids i of the observation target region TA, and is represented by the following equation (5).

f k = [ f 1 k ,   f 2 k ,   … , f N k ] T ( 5 )

The excavation reaction force f^k is acquired, for example, based on the time-series data of the sensors S7 to S9 during the current excavating motion of the excavator 100.

The excavation reaction force for each of the grids i may be represented as a vector. In this case, the soil characteristics λ_i^k for each of the grids i of the observation target region TA is not one dimensional but three-dimensional.

The soil characteristics λ^k refer to characteristics of the soil for each of the grids i of the observation target region TA, and are represented by the following equation (6)

λ k = [ λ 1 k ,   λ 2 k ,   … ,   λ N k ] T ( 6 )

The soil characteristics λ^k are, for example, an angle of repose per each of the grids i of the observation target region TA. For example, the soil characteristics λ^k are defined in advance. In addition, the soil characteristics λ^k may be acquired based on the time-series data of the outputs of the sensor 40 or the outputs of the sensors S7 to S9.

There may be a plurality of types of the soil characteristics for each of the grids i. In this case, the soil characteristics λ_i^k for each of the grids i of the observation target region TA is not one dimensional but multi-dimensional.

The influence information κ^k indicates the presence or absence of the direct influence of the current excavating motion of the excavator 100 on the soil shape for each of the grids i of the observation target region TA. For example, the influence information κ^k refers to the presence or absence of the passage of the blade edge of the bucket 6 for each of the grids i of the observation target region TA, and is represented by the following equation (7).

κ k = [ κ 1 k ,   κ 2 k ,   … ,   κ N k ] T ( 7 )

For example, in the case where the blade edge of the bucket 6 passes through the grid i (i=1, . . . , N) of the observation target region TA during the current excavating motion, and the height of the blade edge of the bucket 6 during passage is equal to or lower than the height h_e,i^k-1 of the soil, the influence information κ_i^k is set to “+1” (κ_i^k=+1). On the other hand, in the case where the blade edge of the bucket 6 does not pass through the grid i (i=1, . . . , N) of the observation target region TA during the current excavating motion, or passes through but the height of the blade edge of the bucket 6 is higher than the height h_e,i^k-1, the influence information κ_i^k is set to “−1” (κ_i^k=−1).

Other information ω^k includes, for example, a weight ω^k and volume v^k of the soil held in the bucket 6 after the execution of the excavating motion of the excavator 100, and is represented by the following equation (8).

ω k = [ w k ,   ν k ] ( 8 )

When the process of step S104 is completed, the work target shape acquisition unit 302B proceeds to step S106.

In step S106, the work target shape acquisition unit 302B updates the uncertainty s_e,i^k-1 of the soil shape.

For example, the work target shape acquisition unit 302B sets a region (hereinafter, referred to as an “influence region”) Ω in which the shape of the soil (height h_e,i^k of the soil) is influenced by the current excavating motion of the excavator 100. Then, the work target shape acquisition unit 302B determines that the uncertainty s_e,i^k of the soil shape is increased in the grids i included in the influence region Ω, and performs updating according to the following equation (9).

s e . i k - 1 = s e . i k - 1 + c i ( i ∈ Ω ) ( 9 )

For example, as illustrated in FIG. 21, the influence region Ω is set as a range obtained by expanding the excavation region EA in the four directions by a predetermined amount. The excavation region EA is a region, within the observation target region TA, corresponding to a set of the grids i where the blade edge of the bucket 6 passes through during the current excavating motion of the excavator 100 and the height of the blade edge of the bucket 6 when passing through is equal to or lower than the height h_e,i^k-1 of the soil.

The increment c_i of the uncertainty s_e,i^k is, for example, the same value for each of the grids in the influence region Ω. In addition, the increment c_i of the uncertainty s_e,i^k may be set so that the increment is the largest in the excavation region EA and decreases as the distance from the excavation region increases within the influence region Ω.

When the process of step S106 is completed, the work target shape acquisition unit 302B proceeds to step S108.

In step S108, the work target shape acquisition unit 302B specifies the soil shape before the execution of the current excavating motion of the excavator 100 based on the output of the sensor 40. Thus, the work target shape acquisition unit 302B can specify the shape of the soil after the execution of the current excavating motion of the excavator 100 for the grids i in which the height of the soil can be measured by the sensor 40 within the observation target region TA.

For example, the work target shape acquisition unit 302B applies the Kalman filter to correct the shape of the soil after the execution of the current excavating motion of the excavator 100 based on the output of the sensor 40, thereby specifying the shape of the soil after the execution of the current excavating motion of the excavator 100.

Upon the application of the Kalman filter, the height h_e,i^k of the soil, and the uncertainty s_e,i^k are represented by the following equations (10) to (14) when the height zi of the soil is observed with the variance s_i for any grid i of the observation target region TA after the execution of the excavating motion of the excavator 100.

h _ e , i k = h e , i k - 1 ( 10 ) s _ e , i k = s e , i k - 1 + α i ( 11 ) ξ i = s _ e , i k ( s _ e , i k + s i ) - 1 ( 12 ) h e , i k = h _ e , i k + ξ i ( z i - h _ e , i k ) ( 13 ) s e , i k = ( 1 - ξ i ) ⁢ s _ e , i k ( 14 )

When the equations (10) to (14) are summarized by a function K, the equations (10) to (14) are represented by the following equation.

( h e , i k , s e , i k ) = K ⁡ ( h e , i k - 1 , s e , i k - 1 , z i , s i ) ( 15 )

Accordingly, the height z_l,i of the soil is measured with the variance s_l,i by the sensor 40 for the grid i of the observation target region TA, and the height h_l,i^k of the soil and the uncertainty s_l,i^k of the height based on the output of the sensor 40 are represented by the following equation (16).

( h l , i k , s l , i k ) = K ⁡ ( h e , i k - 1 , s e , i k - 1 , z l , i , s l , i ) ( 16 )

Thus, the work target shape acquisition unit 302B can acquire (specify), based on the output of the sensor 40, the height h_l,i^k of the soil and the uncertainty s_l,i^k of the height after the execution of the current excavating motion of the excavator 100.

The height h_l^k of the soil based on the output of the sensor 40 for each of the grids i of the observation target region TA is represented by the following equation (17).

h l k = { h l , 1 k , h l , 2 k , … , h l , N k } ( 17 )

There is a case where the sensor 40 cannot measure the height for all of the grids of the observation target region TA due to occlusion or the like. Thus, the height h_l,i^k of the soil and the uncertainty s_l,i^k of the height are acquired by the equation (16) only for the grids i in which the height of the soil can be measured by the sensor 40. The height h_l,i^k of the soil and the uncertainty s_l,i^k for the grids, in which the height of the soil cannot be measured by the sensor 40, are maintained, for example, at the previous values (the height h_l,i^k-1 of the soil, and the uncertainty s_l,i^k-1). In addition, the height h_l,i^k of the soil and the uncertainty s_l,i^k for the grids, in which the height of the soil cannot be measured by the sensor 40, may be designated to a value indicating that it is unknown (“unknown”). Hereinafter, measurability information m_l^k indicating whether or not the sensor 40 can measure the soil shape for each of the grids i of the observation target region TA is used. For example, the measurability information m_l^k is represented by the following equations (18) to (20).

m l k = { m l , 1 k , m l , 2 k , … , m l , N k } ( 18 )

• • Grid i in which the sensor 40 can measure the soil shape

m l , i k = + 1 ( 19 )

• • Grid i in which the sensor 40 cannot measure the soil shape

m l , i k = - 1 ( 20 )

When the process of step S108 is completed, the work target shape acquisition unit 302B proceeds to step S110.

In step S110, the work target shape acquisition unit 302B infers the soil shape (z_p^k of the soil and the uncertainty s_p^k of the height) for each of the grids i using the factor g corresponding to the learned model LM1.

( z p k , s p k ) = g ⁡ ( h e k - 1 , s e k - 1 , h l k , s l k , m l k , b k , φ k , f k , λ k , κ k , ω k ) ( 21 )

The function g is configured, for example, based mainly on a deep neural network (DNN). In addition, it is also possible to use U-Net with the input and output in the form of images. In this case, the above-described other information ω^k of the equation (8) does not employ the image format, and therefore may be expanded into the image format, or may be directly input into the intermediate layer.

Then, as in the step S108, the work target shape acquisition unit 302B applies the Kalman filter to estimate the shape of the soil after the execution of the current excavating motion of the excavator 100 based on the inference result of the factor g. The height h_p,i^k of the soil and the uncertainty s_p,i^k of the height for each of the grids i of the observation target region based on the inference result of the factor g are represented by the following equation (22).

( h p , i k , s p , i k ) = K ⁡ ( h e , i k - 1 , s e , i k - 1 , z p , i , s p , i ) ( 22 )

When the process of step S110 is completed, the work target shape acquisition unit 302B proceeds to step S112.

In step S112, the work target shape acquisition unit 302B generates the data (the height h_e^k of the soil and the uncertainty s_e^k) representing the final shape of the soil after the execution of the excavating motion of the excavator 100 based on the outputs from the processing of both step S108 and step S110. Specifically, the work target shape acquisition unit 302B integrates the data of the shape of the soil based on the output of the sensor 40, which is output by the processing of step S108, and the data of the shape of the soil based on the inference result of the function g, which is output by the processing of step S110.

For example, the work target shape acquisition unit 302B determines the data (the height h_l,i^k of the soil and the uncertainty s_l,i^k) of the soil shape specified based on the output of the sensor 40 in the processing of step S108 as the data (the height h_e,i^k of the soil and the uncertainty s_e,i^k) of the final soil shape.

However, as described above, there may be a grid i in which the shape of the soil is not specified based on the output of the sensor 40 in the processing of step S108 due to occlusion caused by the sensor 40, i.e., the grid i for which the value of the measurability information m_l^k is “−1”. Therefore, for such a grid i, the work target shape acquisition unit 302B sets the data (the height h_p,i^k of the soil, and the uncertainty s_p,i^k) of the soil shape specified based on the inference result of the function g by the processing of step S110 as the data of the final soil shape.

Specifically, the work target shape acquisition unit 302B generates the data of the final soil shape according to the following equations (23) and (24).

• • In the case of

m l , i k = + 1

( h e , i k , s e , i k ) = ( h l , i k , s l , i k ) ( 23 )

• • In the case of

m l , i k = - 1

( h e , i k , s e , i k ) = ( h p , i k , s p , i k ) ( 24 )

For the grids i in which the sensor 40 can measure the soil shape after the execution of the current excavating motion of the excavator 100, the data of the soil shape specified based on the inference result of the function g by the processing of step S110 may be used as the data of the final soil shape.

When the process of step S110 is completed, the work target shape acquisition unit 302B proceeds to step S112.

In step S112, the work target shape acquisition unit 302B outputs the data (the height h_e^k of the soil and the uncertainty s_e^k) of the soil shape generated in step S110.

Thus, for example, the target trajectory generation unit 302C can generate the target trajectory of the working part of the excavator 100, i.e., the bucket 6 (the blade edge of the bucket 6), based on the data of the soil shape output from the work target shape acquisition unit 302B.

In addition, for example, the display processing unit 302E can generate an image representing the soil shape of the observation target region TA near the excavator 100 based on the data of the soil shape output from the work target shape acquisition unit 302B. Thus, the display processing unit 302E can display the image representing the soil shape of the observation target region TA near the excavator 100 on the display device 50A, or can transmit the image to the remote operation support device 400 or the remote monitoring support device via the communication device 60 to display the image on the display device of the remote operation support device 400 or the remote monitoring support device. Further, the display processing unit 302E may reflect the uncertainty s_e^k on the image representing the shape of the soil of the observation target region TA near the excavator 100. For example, the display processing unit 302E indicates uncertainty s_e^k for each of the grids i of the observation target region TA near the excavator 100 with a color of the portion corresponding to the grid i in the image representing the shape of the soil. Thus, the display processing unit 302E can display the image representing the shape of the soil of the observation target region TA near the excavator 100 so that the difference in uncertainty can be identifiable for each of the grids i of the observation target region TA near the excavator 100.

The work target shape acquisition unit 302B may output, in addition to the data of the soil shape, other data obtained during the process of this flowchart. For example, the work target shape acquisition unit 302B may output the measurability information m_l^k. Thus, the controller 30 can identify, within the observation target region TA, the grids i in which the sensor 40 can measure the soil shape, and the grids i in which the sensor 40 cannot measure the soil shape. Therefore, for example, when the display processing unit 302E displays the image representing the soil shape of the observation target region TA on the display devices 50A or the like, the grids i to which the output of the sensor 40 is reflected, and the grids onto which the output of the sensor 40 is not reflected can be identifiably displayed.

When the processing of step S114 is completed, the work target shape acquisition unit 302B ends the process of this flowchart.

As described above, in the present example, the work target shape acquisition unit 302B can estimate the soil shape after the excavating motion from the soil shape before the execution of the excavating motion using the function g, considering the trajectory (path) of the bucket 6 during the excavating motion of the excavator 100, the excavation reaction force, the characteristics of the soil, and the like.

In the present example, even in the case where the predetermined motion is a soil-dumping motion, the work target shape acquisition unit 302B can generate and output the data of the soil shape after the soil-dumping motion of the excavator 100 in the same process as above. In this case, for example, the other information ωk is represented by the following equation (25).

ω k = [ w k , v k , w k - 1 , v k - 1 , ρ ] ( 25 )

As represented by the equation (25), the other information ω^k includes a weight W^k-1 and volume V^k-1 of the soil held in the bucket 6 before the execution of the soil-dumping motion of the excavator 100, and a weight W^k and volume V^k of the soil held in the bucket 6 after the execution of the soil-dumping motion. In addition, as represented by the equation (25), in the case where the dumping location is the loading bed of the truck, the other information ω^k may include the positions p of the four sides of the loading bed of the truck in a plan view. Thus, the work target shape acquisition unit 302B can estimate the soil shape on the loading bed considering the shape of the loading bed of the truck.

In addition, the work target shape acquisition unit 302B can generate and output the data of the soil shape after the excavating motion of the excavator 100 or the soil-dumping motion by the same process as above, for the work executed by a combination of the excavating motion and the soil-dumping motion of the excavator 100.

<<Specific Example of Method of Generating Learned Model>>

Specific examples of the method of generating the learned model LM1 will be described.

In the present example, a method of generating a function g corresponding the learned model LM1 used in the above-described process of FIG. 16 will be described.

The teacher data cj (j=1 to L) of the teacher dataset D is represented, for example, by the following equation (26).

c j = { h r j , h ^ e j , s ^ e j , h l j , s l j , m l j , b j , φ j , f j , λ j , κ j , ω j } ( 26 )

The teacher data c^j is a combination of input data h{circumflex over ( )}_e^j,s{circumflex over ( )}_e^j,h_l^j,s_l^j,m_l^j,b^j,φ^j,f^j, λ^j,κ^j, ω^j, and the height h_r^j of the soil after the excavating motion of the excavator 100 serving as the ground truth. The input data h{circumflex over ( )}_e^j,s{circumflex over ( )}_e^j,h_l^j,s_l^j,m_l^j,b^j, φ^j,f^j,λ^j,ω^j corresponds to the input data (h_e^k-1,s_e^k-1,h_l^k,s_l^k,m_l^k,b^k,φ^k,f^k,λ^k,κ^k,ω^k) of the function g of the equation (21).

As represented by the following equation (27), the function g has a parameter W, and machine learning is performed by optimizing the parameter W with teacher dataset D.

( z p j , s p j , ) = g ⁢ ( h ^ e j , s ^ e j , h l j , s l j , m l j , b j , φ j , f j , λ j , κ j , ω j ; W ) ( 27 )

For example, the function g corresponding to the learned model LM1 is generated by optimizing the parameter W so that the loss function E(W) of the following equation (28) becomes the minimum.

E ⁡ ( W ) = - ∑ j log ⁢ N ⁡ ( h r j | z p j , s p j ; W ) ( 28 )

As described above, the teacher dataset D may be generated from the log acquired by the log acquisition unit 2001, generated from the log acquired by the simulator unit 2002, or acquired from both of the log acquired by the log acquisition unit 2001 and the log acquired by the simulator unit 2002.

In the simulator unit 2002, for example, particle simulation, such as DEM or the like, is employed, and the height h_r^j of the soil is acquired by ray tracing of the geometry sensor, such as LIDAR or the like, virtually arranged with respect to the positions of the particles, as described above.

In addition, as described above, the teacher dataset D may include a base teacher dataset generated from the log acquired by the simulator unit 2002, and a teacher dataset for fine tuning, which is generated from the log acquired by the log acquisition unit 2001. In this case, the number of pieces of the teacher data included in the teacher dataset for fine tuning may be relatively small.

In association with the acquisition of the input data h_l^j, S_l^j, and m_l^j, part of the measurement data of the sensor group 300 or the measurement data of the virtual geometry sensor arranged in the virtual space of the simulator unit 2002 may be virtually masked. Thus, even if no occlusion is caused by the sensor group 300 or by the virtual geometry sensor arranged in the virtual space of the simulator unit 2002, measurement data in which occlusion has virtually occurred can be acquired. In addition, the occlusion region in the observation target region TA may be specifically calculated by ray tracing of the geometry sensor arranged in the virtual space of the simulator unit 2002. Further, the occlusion region may be changed while changing the position of the geometry sensor arranged in the virtual space of the simulator unit 2002. As a result, it is possible to realize robust machine learning for the function g.

In addition, the input data h{circumflex over ( )}_e^j and s{circumflex over ( )}_e^j corresponds to the previous output of the function g (inference result), and may be acquired by using a plurality of excavating motions repeated in time series. In addition, from the viewpoint of inhibition of an increase in the number of data points, the learning may be substituted by learning while mixing a virtually large noise. As a result, it is possible to realize robust machine learning for the function g.

As described above, the information processing device 200 generates the teacher dataset D including the teacher data c^j, and can generate the function g corresponding to the learned model LM1 by machine learning based on the teacher dataset D.

Second Example

FIG. 22 is a flowchart schematically illustrating a second example of the process for acquiring the data representing the shape of the work target.

In the present example, the description will be made on the premise that the functional configuration is the above-described third example (FIG. 8) of the functional configuration of the operation support system SYS or a modification example of the above-described second example (FIG. 7) of the functional configuration of the operation support system SYS, and the predetermined motion of the excavator 100 is an excavating motion.

The flowchart of FIG. 22 is performed, for example, after the execution of the excavating motion of the excavator 100. The flowchart of FIG. 22 corresponds to the above-described processing of step S15 of FIG. 15 or the processing of step S35 of FIG. 17. Hereinafter, in the present flowchart, the description will be made on the premise that data representing the shape of the work target (soil shape) after the execution of k-th excavating motion of the excavator 100 from the start of the excavation work.

As in the above-described first example, in the present example, the description will be made on the premise that the observation target region TA near the excavator 100 is divided into the predetermined number N of grids (see FIG. 20). In addition, in the present example, the description will be made on the premise that the data representing the shape of the work target (soil shape data) after the execution of k-th excavating motion is represented by the height h_e^k of the soil.

As illustrated in FIG. 22, in step S202, the controller 30 acquires data representing the trajectory x of the working part during each excavating motion of the excavator 100. For example, in the case of k=1, the controller 30 acquires data of the trajectory x¹ of the working part during the initial (first) excavating motion of the excavator 100. For example, in the case of k≥2, the controller 30 acquires the data of trajectories x¹ to x^k of the working part during the first to k-th excavating motions of the excavator 100, respectively.

For example, the controller 30 acquires, as the data representing the trajectory x^m during the m-th (1≤m≤k) excavating motion of the excavator 100, the data representing the target trajectory of the working part during the m-th excavating motion of the excavator 100, which is generated by the target trajectory generation unit 302C. In addition, the controller 30 may acquire, as the data representing the trajectory x^m during the m-th (1≤m≤k) excavating motion of the excavator 100, the measurement data of the trajectory (path) of the working part based on outputs of the sensors S1 to S5.

After the start of the excavation work of the excavator 100 and until the completion of the excavation work, the result of the processing of step S202 may be retained, for example, in the auxiliary storage device 30A or memory 30B of the controller 30. Thus, at step S202 after the execution of the k-th excavating motion, the controller 30 can easily and promptly acquire the data of the trajectories x¹ to x^k using the result of the processing of step S202 after the execution of the previous excavating motions.

When the process of step S202 is completed, the controller 30 proceeds to step S204.

In step S204, the work target shape acquisition unit 302B of the controller 30 acquires the time-series data of the measurement result (measured reaction force) F_o of the reaction force to the working part during each excavating motion of the excavator 100 based on the outputs of the sensors S7 to S9. For example, in the case of k=1, the work target shape acquisition unit 302B acquires the time-series data of the measured reaction force F_o¹ during the initial (first) excavating motion of the excavator 100. For example, in the case of k≥2, the work target shape acquisition unit 302B acquires the time-series data of the measured reaction force F_o¹ to F_o^k during the first to k-th excavating motions of the excavator 100, respectively.

After the start of the excavation work of the excavator 100 and until the completion of the excavation work, the result of the processing of step S204 may be retained, for example, in the auxiliary storage device 30A or memory 30B of the controller 30. Thus, at step S204 after the execution of the k-th excavating motion, the work target shape acquisition unit 302B can easily and promptly acquire the data of the measured reaction forces F_o¹ to F_o^k using the result of the processing of step S204 after the execution of the previous excavating motions.

When the process of step S204 is completed, the controller 30 proceeds to step S206.

In step S206, the reaction force estimation unit 302F of the controller 30 acquires the time-series data of the estimation result (estimated reaction force) F_e of the reaction force to the working part during each excavating motion of the excavator 100. For example, in the case of k=1, the work target shape acquisition unit 302B acquires the time-series data of the estimated reaction force F_e¹ during the initial (first) excavating motion of the excavator 100. For example, in the case of k≥2, the work target shape acquisition unit 302B acquires time-series data of the estimated reaction forces F_e¹ to F_e^k during the first and k-th excavating motions of the excavator 100, respectively.

For example, the reaction force estimation unit 302F acquires the time-series data of the estimated reaction force F_e^m for the m-th excavating motion according to the following equation (29) using the function g corresponding to the following learned model LM3.

F e m = g ⁢ ( x m , h e m - 1 ) ( 29 )

In addition, the reaction force estimation unit 302F may acquire the time-series data of the estimated reaction force F_e^m for the m-th excavating motion according to the following equation (30) by introducing a parameter γ representing the characteristics of the soil of the work target.

F e m = g ⁢ ( x m , h e m - 1 | γ ) ( 30 )

Thus, the certainty of the estimated reaction force F_e^m can be improved.

When the process of step S206 is completed, the controller 30 proceeds to step S208.

In step S208, the work target shape acquisition unit 302B calculates a difference (estimation error) between the measured reaction force F_o^m and the estimated reaction force F_e^m acquired at step S202 and step S204, respectively, and correct the data of the initial shape of the work target based on the estimation error. The initial shape is set by the above-described processing of step S11 of FIG. 15 or processing of step S31 of FIG. 17, and the data of the initial shape is represented by the height h_e⁰ of the soil. Specifically, the work target shape acquisition unit 302B corrects the initial shape (the height h_e⁰ of the soil) of the work target so that the estimation error becomes the minimum.

For example, in the case of k=1, i.e., after the execution of the initial (first) excavating motion, the work target shape acquisition unit 302B acquires the data of the corrected initial shape (the height h{tilde over ( )}_e⁰ of the soil) according to the following equation (31) on the premise of the above equation (29).

h ˜ e 0 = argmin h e 0 ⁢  g ⁢ ( x 1 , h e 0 ) - F o 1  ( 31 )

For example, in the case of k≥2, i.e., in the case of the execution of the second or later excavating motions, the work target shape acquisition unit 302B acquires the data of the corrected initial shape (the height h{tilde over ( )}_e⁰ of the soil) according to the following equation (32) on the premise of the above equation (29).

h ˜ e 0 = argmin h e 0 ⁢ ∑ m = 1 k  g ⁢ ( x m , h e m - 1 ) - F o m  ( 32 )

For example, the work target shape acquisition unit 302B calculates the data of the shape (the height h_e^m of the soil) of the work target after the execution of the m-th excavating motion according to the following equation (33) using the function r corresponding to the learned model LM1.

h e m = r ⁡ ( h e m - 1 , x m ) ( 33 )

In addition, the work target shape acquisition unit 302B may calculate the data of the shape (the height h_e^m of the soil) of the work target after the execution of the m-th excavating motion according to the following equation (34) by introducing the parameter γ representing the characteristics of the soil of the work target, as in the case of the function g.

h e 𝔪 = r ⁡ ( h e m - 1 , x m | γ ) ( 34 )

Thus, the certainty of the shape (the height h_e^m of the soil) of the work target can be improved.

When the process of step S208 is completed, the controller 30 proceeds to step S210.

In step S210, the certainty estimation unit 302G of the controller 30 estimates the certainty of the initial shape of the work target corrected by the processing of step S208.

For example, the certainty estimation unit 302G estimates the certainty c^k of the corrected initial shape of the work target after the execution of the k-th excavating motion according to the following equation (35) using the function w corresponding to the learned model LM4.

c k = w ⁢ ( x 1 : k , h e 0 , F o 1 : k ) ( 35 )

The trajectory x^1:k represents the trajectories of the working part in the first to k-th excavating motions, and the measured reaction force F_o^1:k represents the measurement result of the reaction force to the working part during the first to k-th excavating motions.

In addition, the work target shape acquisition unit 302B may estimate the certainty c^k of the corrected initial shape of the work target after the execution of the k-th excavating motion according to the following equation (36) by introducing the parameter γ representing the characteristics of the soil of the work target, as in the case of the function g or r.

c k = w ⁢ ( x 1 : k , h e 0 , F o 1 : k | γ ) ( 36 )

Thus, the estimation certainty of certainty c^k of the corrected initial shape of the work target after the execution of the k-th excavating motion can be improved.

When the process of step S210 is completed, the controller 30 proceeds to step S212.

In step S212, the work target shape acquisition unit 302B estimates the latest shape of the work target after the execution of the k-th excavating motion, and acquires the data representing the latest shape (soil shape data).

Specifically, the work target shape acquisition unit 302B estimates the latest shape of the work target based on the trajectories x⁰ to x^k of the first to k-th excavating motions, with the initial shape (the height h{tilde over ( )}_e⁰ of the soil) of the work target corrected at step S208 as a starting point.

For example, in the case of k=1, the work target shape acquisition unit 302B acquires the data representing the shape (the height h_e¹ of the soil) of the work target after the execution of the first excavating motion according to the following equation (37) using the function r corresponding to the learned model LM1.

h e 1 = r ⁢ ( h ~ e 0 , x 1 ) ( 37 )

In addition, in the case of k≥2, the work target shape acquisition unit 302B acquires the data representing the shape (the height h_e^k of the soil) of the work target after the execution of the k-th excavating motion according to the following equation (38) using the function r corresponding to the learned model LM1.

h e 1 = r ⁢ ( h ~ e 0 , x 1 ) ( 38 ) ⋮ h e k = r ⁢ ( h e k - 1 , x k )

When the process of step S212 is completed, the controller 30 ends the process of this flowchart.

As described above, in the present example, the controller 30 can correct the initial shape of the work target based on the difference (estimation error) between the estimated reaction force and the measured reaction force, and estimate the latest shape of the work target after the execution of the excavating motion of the excavator 100 with the corrected initial shape of the work target as a starting point.

Third Example

FIG. 23 is a flowchart schematically illustrating a third example of the process for acquiring the data representing the shape of the work target.

In the present example, the description will be made on the premise that the functional configuration of the operation support system SYS is the first example (FIG. 6), the second example (FIG. 7), the third example (FIG. 8), or a modification example of the second example of the functional configuration of the operation support system SYS described above, and the predetermined motion of the excavator 100 is a soil-dumping motion.

The flowchart of FIG. 23 is executed, for example, after the soil-dumping motion of the excavator 100. Specifically, the flowchart of FIG. 23 corresponds to the above-described processing of step S24 of FIG. 16 or the above-described processing of step S45 of FIG. 18. Hereinafter, in this flowchart, the description will be made on the premise that data representing the shape of the work target (soil shape) after the execution of a k-th soil-dumping motion of the excavator 100 from the start of the loading motion is acquired.

In the present example, the description will be made on the premise that the observation target region TA near the excavator 100, i.e., a loading bed of a truck, is divided into the predetermined number N of grids (see FIG. 20), as in the above-described first example or second example. In addition, in the present example, the description will be made on the premise that the data representing the shape of the work target (soil shape data) after the execution of the k-th excavating motion is represented by the height b_e^k of the soil with the bottom surface of the loading bed of the truck as a reference.

As illustrated in FIG. 23, in step S302, the controller 30 estimates a weight w^k of the soil held in the bucket 6 before the execution of the k-th soil-dumping motion based on the outputs of the sensors S7 to S9.

When the process of step S302 is completed, the controller 30 proceeds to step S304.

In step S304, the controller 30 acquires data representing the trajectory x^k of the working part during the execution of the k-th soil-dumping motion of the excavator 100.

For example, the controller 30 acquires, as the data representing the trajectory x^k of the working part during the execution of k-th soil-dumping motion of the excavator 100, data representing a target trajectory of the working part during the k-th soil-dumping motion of the excavator 100, which is generated by the target trajectory generation unit 302C. In addition, the controller 30 may acquire, as the data representing the trajectory x^k during the execution of k-th soil-dumping motion of the working part, measurement data of the trajectory (path) of the working part during the k-th soil-dumping motion of the excavator 100 based on the outputs of the sensors S1 to S5.

When the process of step S304 is completed, the controller 30 proceeds to step S306.

In step S306, the work target shape acquisition unit 302B of the controller 30 estimates the latest shape of the work target after the execution of the k-th soil-dumping motion of the excavator 100, i.e., the shape of the soil on the loading bed of the truck, and acquires the data representing the latest shape.

For example, the work target shape acquisition unit 302B acquires the data representing the shape (height b_e^k of the soil) of the soil on the loading bed of the truck after the execution of the k-th soil-dumping motion of the excavator 100 according to the following equation (39) using the function a corresponding to the learned model LM1.

b e k = a ⁢ ( b e k - 1 , x k , w k ) ( 39 )

In addition, the work target shape acquisition unit 302B may acquire the data representing the shape (height b_e^k of the soil) of the soil on the loading bed of the truck after the execution of the k-th soil-dumping motion of the excavator 100 according to the following equation (40) by introducing the parameter γ representing the characteristics of the soil.

b e k = a ⁡ ( b e k - 1 , x k , w k | γ ) ( 40 )

Thus, the certainty of the estimation of the shape of the soil on the loading bed of the truck after the execution of the k-th soil-dumping motion of the excavator 100 can be improved.

When the process of step S306 is completed, the controller 30 ends the process of this flowchart.

As described above, in the present example, the controller 30 can estimate the latest shape of the work target after the execution of the soil-dumping motion of the excavator 100 based on the shape of the work target before the execution of the soil-dumping motion of the excavator 100 and the trajectory of the working part during the soil-dumping motion of the excavator 100.

[Functions]

Next, functions of the work machine, the information processing device, and the program according to the present embodiment will be described.

In a first aspect of the present embodiment, the work machine includes a processing device. The work machine is, for example, the above-described excavator 100. The processing device is, for example, the above-described controller 30. Specifically, the processing device is configured to acquire data that represents a state of the work machine and relates to a shape of a work target, in association with a motion of the work machine, and is configured to estimate the shape of the work target after the motion of the work machine based on the acquired data.

In addition, in the first aspect of the present embodiment, the information processing device may be configured to acquire data that represents a state of the work machine and relates to a shape of a work target, in association with a motion of the work machine, and is configured to estimate the shape of the work target after the motion of the work machine based on the acquired data. The information processing device is, for example, the above-described controller 30, the information processing device 200, or the remote operation support device 400.

Further, in the first aspect of the present embodiment, the program may cause the information processing device to execute a first procedure and a second procedure. Specifically, in the first procedure, data that represents a state of the work machine and relates to a shape of a work target, in association with a motion of the work machine, may be acquired. In the second procedure, the shape of the work target of the work machine may be estimated based on the data acquired in the first procedure.

Thus, the work machine or the information processing device (hereinafter, referred to as the “work machine or the like”) can estimate the state of the work target, for example, considering a change in the shape of the work target in accordance with the motion of the work machine. Therefore, the work machine or the like can more appropriately grasp the shape of the work target.

In addition, in a second aspect of the present embodiment on the premise of the above-described first aspect, the work machine may include a first acquisition device. The first acquisition device is, for example, the above-described sensors S1 to S3. Specifically, the first acquisition device may acquire data related to a trajectory of a working part of the work machine. The processing device or information processing device (hereinafter, referred to as the “processing device or the like”) may estimate the shape of the work target based on the data related to the trajectory of the working part during execution of a predetermined motion.

Thus, the work machine or the like can estimate the state of the work target from the trajectory of the working part, considering the change in the shape of the work target in view of the positional relationship between the working part and the work target.

In addition, in a third aspect of the present embodiment on the premise of the above-described second aspect, the work machine may include a second acquisition device. The second acquisition device is, for example, the above-described sensors S7 to S9. Specifically, the second acquisition device may acquire data related to a reaction force from the work target to the working part, or data related to a weight of soil held by the working part. The processing device or the like may estimate the shape of the work target based on the data related to the reaction force to the working part or the data related to the weight of the soil held by the working part during the execution of the predetermined motion.

Thus, the work machine or the like can recognize, for example, any contact with rocks or the like in the ground from the reaction force to the working part during the execution of the predetermined motion of the work machine, or can recognize characteristics of the soil, such as hardness or the like. In addition, the work machine or the like can recognize, for example, the amount of the soil to be dumped onto the work target from the weight of the soil held in a bucket serving as the working part. Thus, the work machine or the like can more appropriately estimate the shape of the work target considering these recognition results.

In addition, in a fourth aspect of the present embodiment on the premise of the above-described second aspect, the work machine may include a measuring device. The measuring device is, for example, the above-described sensor 40. Specifically, the measuring device may measure the shape of the work target near the work machine. The processing device or the like may estimate the shape of the work target based on measurement data of the shape of the work target and the data related to the trajectory of the working part during the execution of the predetermined motion of the working part.

Thus, the work machine or the like can estimate the shape of the work target, for example, considering the change in the shape of the work target from the trajectory of the working part with reference to the shape of the work target at a certain point as a starting point.

In addition, in a fifth aspect of the present embodiment on the premise of the fourth aspect, the processing device or the like may estimate the shape of the work target after the execution of the predetermine motion based on measurement data of the shape of the work target before the execution of the predetermined motion and measurement data of the shape of the work target after the execution of the predetermined motion.

Thus, the work machine or the like can estimate the shape of the work target, which is in an unmeasurable location due to occlusion or the like, after the execution of the predetermined motion of the work machine from the trajectory of the working part using the measurement data of the measuring device as a base.

In addition, in a sixth aspect of the present embodiment on the premise of the above-described fourth or fifth aspect, the processing device or the like may acquire data representing the shape of the work target based on a result of the estimation of the shape of the work target and measurement data of the shape of the work target acquired by the measuring device after an output of the result of the estimation.

Thus, the processing device or the like can more appropriately estimate the data representing the shape of the work target, for example, using a result of machine learning based on a difference between the estimation result of the shape of the work target and the measurement result of the measuring device after the output of the estimation result.

In addition, in a seventh aspect of the present embodiment on the premise of the above-described third embodiment, the processing device or the like may estimate a reaction force from the work target to the working part during the execution of the predetermined motion based on data representing the shape of the work target before the execution of the predetermined motion and the data related to the trajectory of the working part during the execution of the predetermined motion of the work machine. In addition, the processing device or the like may correct data representing the shape of the work target before the execution of the predetermined motion based on a result of the estimation of the reaction force from the work target to the working part during the execution of the predetermined motion, and data related to an actual reaction force from the work target to the working part during the execution of the predetermined motion. The processing device or the like may estimate the shape of the work target after the execution of the predetermined motion based on the corrected data representing the shape of the work target before the execution of the predetermined motion and the data related to the trajectory of the working part during the execution of the predetermined motion.

Thus, for example, even in the case where certainty of the data representing the shape of the work target before the execution of the predetermined motion of the work machine is relatively low, the processing device or the like can correct the data based on the estimation error of the estimation result of the reaction force during the execution of the predetermined motion of the work machine. Therefore, the processing device or the like can estimate the shape of the work target after the execution of the predetermined motion of the work machine with relatively high certainty.

In addition, in an eighth aspect of the present embodiment on the premise of the above-described seventh embodiment, the processing device or the like may correct data representing an initial shape of the work target based on a result of the estimation of the reaction force from the work target to the working part during the execution of each of a plurality of predetermined motions with the initial shape of the work target as a starting point, and the data related to the actual reaction force from the work target to the working part during the execution of the predetermined motion. The processing device or the like may estimate the shape of the work target after the execution of the plurality of predetermined motions based on the corrected data representing the initial shape of the work target and data related to the trajectory of the working part during the execution of each of the plurality of predetermined motions.

Thus, for example, even in the case where certainty of the data representing the initial shape of the work machine is relatively low, the processing device or the like can correct the data based on the estimation error of the estimation result of the reaction force during the execution of the predetermined motion of the work machine. Therefore, the processing device or the like can estimate the shape of the work target after the execution of the predetermined motion of the work machine with relatively high certainty.

In addition, in a ninth aspect of the present embodiment on the premise of the above-described seventh or eighth aspect, the processing device or the like may restrict a motion speed, a motion range, or both a motion speed and a motion range of the predetermined motion of the work machine based on data representing uncertainty of the shape of the work target corresponding to a position within the work target in the data representing the shape of the work target before the execution of the predetermined motion of the work machine.

Thus, for example, in the case where uncertainty in the shape of the data representing the shape of the work target before the execution of the predetermined motion of the work target is relatively high, the processing device or the like may decrease the motion speed of the predetermined motion of the work machine or may decrease the motion range of the predetermined motion of the work machine. Accordingly, for example, even in the case where the predetermined motion of the work machine is executed on the premise of the data representing the shape of the work target before the execution of the predetermined motion of the work target with relatively low certainty, it is possible to avoid a situation where the reaction force from the work target to the working part of the work machine becomes excessive.

In addition, in a tenth aspect of the present embodiment on the premise of any one of the above-described first to ninth aspects, the processing device or the information processing device may estimate the shape of the work target based on characteristics of soil of the work target, or characteristics of soil added to the work target according to the motion of the work machine.

Thus, the work machine or the like can more appropriately estimate the shape of the work target considering the characteristics of the soil.

In addition, in an eleventh aspect of the present embodiment on the premise of the above-described tenth aspect, the characteristics of the soil may include an angle of response of the soil, a moisture content of the soil, a granularity of the soil, or any combination of the foregoing.

Thus, the work machine or the like can more appropriately estimate the shape of the work target considering the angle of repose of the soil, the moisture content of the soil, or the granularity of the soil.

In addition, in a twelfth aspect of the present embodiment on the premise of any one of the above-described first to eleventh aspects, the predetermined motion of the work machine may include a plurality of motions. The processing device or the information processing device may estimate the shape of the work target according to the predetermined motion to be executed by the work machine among the plurality of predetermined motions.

Thus, the work machine or the like can estimate the state of the work target according to the predetermined motion to be executed by the work machine.

In addition, in a thirteenth aspect of the present embodiment on the premise of any one of the above-described second to ninth aspects, the working part of the work machine may be a bucket. The predetermined motion of the work machine may be an excavating motion or a soil-dumping motion.

Thus, the work machine or the like can more appropriately estimate the shape of the soil of the work target considering the change in the shape of the work target according to the excavating motion or the soil-dumping motion of the work machine.

In addition, in a fourteenth aspect of the present embodiment on the premise of any one of the above-described first to thirteenth aspects, the work machine may include a control device that is configured to control the motion of the work machine based on a result of the estimation of the shape of the work target. In addition, the information processing device may include a control unit that is configured to control the motion of the work machine based on a result of the estimation of the shape of the work target.

Thus, the work machine or the like can control the motion of the work machine according to the shape of the work target.

In addition, in a fifteenth aspect of the present embodiment on the premise of any one of the above-described first to fourteenth aspects, the work machine may include a display device that is configured to display an image representing the shape of the work target based on a result of the estimation of the shape of the work target. The display device is, for example, the above-described output device 50.

In addition, in the fifteenth aspect of the present embodiment on the premise of any one of the above-described first to fourteenth aspects, the program may cause a support device to execute the first to third procedures. Specifically, in the first procedure, the data that represents the state of the work machine and relates to the shape of the work target in association with the motion of the work machine may be acquired. In the second procedure, the shape of the work target of the work machine may be estimated based on the data acquired by the first procedure. In the third procedure, an image representing the shape of the work target may be displayed on a display based on a result of the estimation of the shape of the work target. The support device is, for example, the remote operation support device 400.

Thus, the work machine or the like can provide the estimation result of the shape of the work target to an operator or the like.

In addition, in a sixteenth aspect of the present embodiment on the premise of the above-described fifteenth aspect, the processing device or the like may may estimate the shape of the work target after the motion of the work machine according to the motion of the work machine, may acquire data representing the shape of the work target after the motion of the work machine, and may acquire data representing uncertainty of the shape of the work target corresponding to a position within the work target in the data acquired. The display device may be configured to display the image representing the shape of the work target so that a difference in uncertainty corresponding to the position within the work target is identifiable.

In addition, in the sixteenth aspect of the present embodiment on the premise of the above-described fifteenth aspect, the program may cause the support device to execute a fourth procedure in addition to the second procedure. In the second procedure, the shape of the work target of the work machine after the motion of the work machine is estimated according to the motion of the work machine, and the data representing the shape of the work target after the motion of the work machine is acquired based on the result of the estimation of the shape of the work target. In the fourth procedure, data representing uncertainty of the result of the estimation of the shape of the work target corresponding to a position within the work target in the data acquired in the second procedure is acquired. The program may cause the support device to execute the third procedure, in which the image representing the result of the estimation of the shape of the work target is displayed on the display so that a difference in the uncertainty corresponding to the position within the work target is identifiable.

Thus, the work machine or the like can allow a user to visually recognize the estimation result of the shape of the work target, and at the same time, can allow the user to recognize a difference in uncertainty of the estimation result corresponding to the position within the range to be observed.

Although the embodiments have been described in detail above, the present disclosure is not limited to the specific embodiments, and various modifications and changes can be made within the scope of the gist described in the claims.