CONTROL SYSTEM, CONTROL METHOD AND MACHINE, STORAGE MEDIUM AND PROGRAM PRODUCT

Description

CROSS-REFERENCE TO RELATED APPLICATIONS

The present application is based on and claims priority to CN Patent Application No. 202410415004.X filed on Apr. 7, 2024, the disclosure of which is incorporated by reference herein in its entirety.

TECHNICAL FIELD

The present disclosure relates to the technical field of smart construction machine, in particular to a control system, a control method and machine, a storage medium and a program product.

BACKGROUND

At present, smart construction machine has received more and more attention and has made some progress. Current smart construction machine substantially uses a sequential architecture of perception-planning-control.

SUMMARY

According to an aspect of the present disclosure, a control system for a construction machine is provided. The control system comprises: an environment perception module configured to receive environment information and/or control result information, output perception result information based on the environment information and/or the control result information, receive a system adjustment instruction, and adjust a state of the environment perception module based on the system adjustment instruction; a decision-making planning module configured to receive an operation task and the perception result information, generate a motion trajectory based on the operation task and the perception result information, output the motion trajectory, predict the motion trajectory to obtain prediction result information, receive feedback result information corresponding to the motion trajectory, and output the system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information; and a control performing module configured to receive the motion trajectory, perform the operation task based on the motion trajectory, obtain the feedback result information and the control result information during the performing of the operation task, output the feedback result information to the decision-making planning module, and output the control result information to the environment perception module.

In some embodiments, the environment perception module is further configured to output the perception result information directly to the control performing module; and the control performing module is further configured to receive the perception result information and perform an operation corresponding to the perception result information based on the perception result information.

In some embodiments, the decision-making planning module is configured to calculate and obtain an adjustment parameter based on the prediction result information and the feedback result information, and output the system adjustment instruction based on the adjustment parameter, wherein the system adjustment instruction comprises the adjustment parameter, and the adjustment parameter is configured to adjust the state of the environment perception module; and the environment perception module is configured to extract the adjustment parameter from the system adjustment instruction and adjust the state of the environment perception module based on the adjustment parameter.

In some embodiments, the feedback result information comprises a reaction result of an operation object of the construction machine.

In some embodiments, the perception result information comprises the environment information; or the perception result information comprises a logical judgment result generated based on the environment information and/or the control result information. In some embodiments, the environment perception module comprises an environment perception element and a multi-degree-of-freedom platform, wherein the environment perception element is installed on the multi-degree-of-freedom platform; and the adjustment parameter comprises an internal parameter value of the environment perception element and/or a parameter of the multi-degree-of-freedom platform.

In some embodiments, the environment perception module is further configured to transmit the environment information to the decision-making planning module, wherein the environment information is first data information with an element coordinate system of the environment perception module as a coordinate system; and the decision-making planning module is further configured to convert the first data information into second data information with a reference coordinate system of the construction machine as a coordinate system based on a transformation matrix between the element coordinate system and the reference coordinate system, and generate the motion trajectory based on the operation task and the second data information.

In some embodiments, the decision-making planning module is further configured to adjust the motion trajectory based on the perception result information of the environment perception module, and output the motion trajectory adjusted to the control performing module; and the control performing module is further configured to control motion of the construction machine based on the motion trajectory adjusted.

In some embodiments, the decision-making planning module is further configured to adjust the motion trajectory based on the feedback result information of the control performing module, and output the motion trajectory adjusted to the control performing module; and the control performing module is further configured to control motion of the construction machine based on the motion trajectory adjusted.

In some embodiments, the decision-making planning module is further configured to send a control instruction to the control performing module to control an operational state or operational speed of the control performing module to make the operational state or operational speed of the control performing module match an adjusted state of the environment perception module.

According to another aspect of the present disclosure, a control method for a construction machinery is provided. The control method comprises: receiving, by an environment perception module, environment information and/or control result information, and outputting perception result information based on the environment information and/or the control result information; receiving, by a decision-making planning module, an operation task and the perception result information, generating a motion trajectory based on the operation task and the perception result information, outputting the motion trajectory, and predicting the motion trajectory to obtain predicting result information; receiving, by a control performing module, the motion trajectory, performing the operation task based on the motion trajectory, obtaining feedback result information and the control result information during the performing of the operation task, outputting the feedback result information to the decision-making planning module, and outputting the control result information to the environment perception module; receiving, by the decision-making planning module, the feedback result information corresponding to the motion trajectory, and outputting a system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information; and receiving, by the environment perception module, the system adjustment instruction, and adjusting a state of the environment perception module based on the system adjustment instruction.

In some embodiments, the control method further comprises: outputting the perception result information directly to the control performing module by the environment perception module; and receiving, by the control performing module, the perception result information, and performing an operation corresponding to the perception result information based on the perception result information.

In some embodiments, the outputting, by the decision-making planning module, the system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information comprises: calculating and obtaining, by the decision-making planning module, an adjustment parameter based on the prediction result information and the feedback result information, wherein the adjustment parameter is configured to adjust the state of the environment perception module; and outputting, by the decision-making planning module, the system adjustment instruction based on the adjustment parameter, wherein the system adjustment instruction comprises the adjustment parameter; and the adjusting, by the environment perception module, the state of the environment perception module based on the system adjustment instruction comprises: extracting, by the environment perception module, the adjustment parameter from the system adjustment instruction; and adjusting, by the environment perception module, the state of the environment perception module based on the adjustment parameter.

In some embodiments, the feedback result information comprises a reaction result of an operation object of the construction machine.

In some embodiments, the perception result information comprises the environment information; or the perception result information comprises a logical judgment result generated based on the environment information and/or the control result information.

In some embodiments, the control method further comprises: transmitting the environment information to the decision-making planning module by the environment perception module, wherein the environment information is first data information with an element coordinate system of the environment perception module as a coordinate system; and converting, by the decision-making planning module, the first data information into second data information with a reference coordinate system of the construction machine as a coordinate system based on a transformation matrix between the element coordinate system and the reference coordinate system, and generating the motion trajectory based on the operation task and the second data information.

In some embodiments, the control method further comprises: adjusting, by the decision-making planning module, the motion trajectory based on the perception result information of the environment perception module, and outputting the motion trajectory adjusted to the control performing module; and controlling, by the control performing module, motion of the construction machine based on the motion trajectory adjusted.

In some embodiments, the control method further comprises: adjusting, by the decision-making planning module, the motion trajectory based on the feedback result information of the control performing module, and outputting the motion trajectory adjusted to the control performing module; and controlling, by the control performing module, motion of the construction machine based on the motion trajectory adjusted.

In some embodiments, the control method further comprises: sending a control instruction to the control performing module by the decision-making planning module to control an operational state or operational speed of the control performing module to make the operational state or operational speed of the control performing module match an adjusted state of the environment perception module.

According to another aspect of the present disclosure, a control system for a construction machinery is provided. The control system comprises: a memory; and a processor coupled to the memory, wherein the processor is configured to, based on instructions stored in the memory, perform the control method as described previously.

According to another aspect of the present disclosure, a construction machine is provided. The construction machine comprises the control system as described previously.

According to another aspect of the present disclosure, a computer-readable storage medium is provided. The computer-readable storage medium has computer program instructions stored thereon that, when executed by a processor, implement the control method as described previously.

According to another aspect of the present disclosure, a computer program product is provided. The computer program product comprises instructions that, when executed by a processor, cause the processor to perform the control method as described previously.

According to another aspect of the present disclosure, a computer program is provided. The computer program comprises instructions that, when executed by a processor, cause the processor to perform the control method as described previously.

Other features and advantages of the present disclosure will become clear from the following detailed description of exemplary embodiments of the present disclosure with reference to the accompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings which constitute part of this specification, describe the embodiments of the present disclosure, and together with this specification, serve to explain the principles of the present disclosure.

The present disclosure may be more explicitly understood from the following detailed description with reference to the accompanying drawings, in which:

FIG. 1 is a structural block view schematically showing a control system for a construction machine according to some embodiments of the present disclosure;

FIG. 2 is a flowchart showing a control method for a construction machine according to some embodiments of the present disclosure;

FIG. 3 is a schematic view showing an architecture system of a smart unmanned excavator according to some embodiments of the present disclosure;

FIG. 4 is a schematic view showing a joint coordinate model of a working device of a smart unmanned excavator according to some embodiments of the present disclosure;

FIG. 5 is a flowchart showing a control method for a smart unmanned excavator according to some embodiments of the present disclosure;

FIG. 6 is a structural block view schematically showing a control system for a construction machine according to other embodiments of the present disclosure;

FIG. 7 is a structural block view schematically showing a control system for a construction machine according to other embodiments of the present disclosure.

DETAILED DESCRIPTION

Various exemplary embodiments of the present disclosure will now be described in detail with reference to the accompanying drawings. It should be noted that: unless additionally specified, the relative arrangements, numerical expressions and numerical values of the components and steps expounded in these examples do not limit the scope of the present disclosure.

At the same time, it should be understood that, for ease of description, the dimensions of various parts shown in the accompanying drawings are not drawn according to actual proportional relations.

The following descriptions of at least one exemplary embodiment which are in fact merely illustrative, shall by no means serve as any delimitation on the present disclosure as well as its application or use.

The techniques, methods, and devices known to a common technical person in the relevant art may not be discussed in detail, but where appropriate, the techniques, methods, and devices should be considered as part of the description.

Among all the examples shown and discussed here, any specific value shall be construed as being merely exemplary, rather than as being restrictive. Thus, other examples in the exemplary embodiments may have different values.

It is to be noted that: similar reference numerals and letters present similar items in the following accompanying drawings, and therefore, once an item is defined in one accompanying drawing, it is not necessary to make further discussion on the same in the subsequent accompanying drawings.

In the related art, smart construction machine focuses on smart operation, and since the tasks, objects and environments of a construction machine operation are more intricate and volatile than the driving task and road environment of self-driving automobile, the sequential architecture of perception-planning-control might no longer meet the operation requirements in some circumstances. For example, when a loader works, a walking mechanism and a bucket move at the same time to complete a combined action. If decision-making is not well-planned, the bucket will block the visual field of the sensor pointing to the traveling direction. During operation of the excavator, the slewing mechanism, the jib, the arm and the bucket move at the same time to complete a complex operation action. No matter where the vehicle-mounted sensor is installed, it will move along with the machine. In some circumstances, the sensor no longer points to the traveling direction or the operation direction, and so forth. Therefore, the sequential architecture of perception-planning-control of the construction machine in the related art might lead to relatively poor coordination of the construction machine during the process of performing operations.

In view of this, the embodiment of the present disclosure provides a control system or control method for a construction machine to improve the coordination of the construction machine during the process of performing operations.

FIG. 1 is a structural block view schematically showing a control system for a construction machine according to some embodiments of the present disclosure. As shown in FIG. 1, the control system comprises: an environment perception module 110, a decision-making planning module 120 and a control performing module 130. For example, the control system may be applied in smart construction machine such as an excavator and a loader and unmanned smart systems.

The environment perception module 110 is configured to receive environment information and/or control result information, output perception result information based on the environment information and/or the control result information, receive a system adjustment instruction, and adjust a state of the environment perception module based on the system adjustment instruction.

That is, the environment perception module 110 may receive the environment information and output the perception result information based on the environment information; or the environment perception module 110 may receive the control result information and output the perception result information based on the control result information; or the environment perception module 110 may receive the environment information and the control result information, and output the perception result information based on the environment information and the control result information.

For example, when the construction machine just starts working, the environment perception module may receive the environment information and output the perception result information based on the environment information; the decision-making planning module generates and outputs a motion trajectory based on an operation task and the perception result information; the control performing module performs the operation task based on the motion trajectory, obtains control result information and outputs the control result information to the environment perception module, so that the environment perception module may receive the control result information. At this time, since the environment perception module has been receiving the environment information, the environment perception module receives the environment information and/or the control result information.

In some embodiments, the environment perception module comprises an environment perception element and a multi-degree-of-freedom platform. The environment perception element is installed on the multi-degree-of-freedom platform. For example, the environment perception element comprises, but is not limited to, a perception element (also referred to as a sensor) such as a camera, a laser radar or a millimeter-wave radar. The multi-degree-of-freedom platform may be a controllable platform or a multi-degree-of-freedom structure. For example, the multi-degree-of-freedom platform comprises, but is not limited to, some structures on the construction machine or an individually provided device.

For example, an installation location and an installation method of the environment perception element are determined according to the operation task.

If an internal parameter of the environment perception element may be changed (for example, a focal length or a frequency may be changed), a program interface may be provided to the decision-making planning module to control the change of its internal parameter.

For example, the environment perception element is installed on the multi-degree-of-freedom platform, so that the decision-making planning module can command an actuator to change a base point position and a perspective of the environment perception element within a certain range according to the operation task.

In some embodiments, the environment information comprises: environment image information shot by a camera and/or three-dimensional information (for example, the three-dimensional information comprises distance information, direction information or the like) detected by a radar. The control result information which is the change of own attitude and state of the construction machine in operation, and output by the control performing module, comprises, for example, attitude information (for example, a boom angle) of a boom or the like.

In the embodiment of the present disclosure, the environment perception module 110 is configured to receive at least one of the environment information or the control result information, and output the perception result information to the decision-making planning module 120 based on the at least one of the environment information or the control result information.

For example, the perception result information comprises the environment information. That is, the environment perception module may output the environment information as the perception result information directly to the decision-making planning module.

For another example, the perception result information comprises: a logical judgment result generated based on the environment information and/or the control result information. That is, the environment perception module may generate the logical judgment result based on the environment information and/or the control result information. For example, if the environment perception module determines that there is an obstacle ahead of the construction machine (for example, an excavator) according to the environment information and/or the control result information, the environment perception module generates a logical judgment result that there is an obstacle ahead. Then, the environment perception module outputs the logical judgment result as the perception result information to the decision-making planning module.

For another example, the environment perception module may also output the environment information and the above-described logical judgment result together as the perception result information to the decision-making planning module.

In the embodiment of the present disclosure, the environment perception module 110 is further configured to receive a system adjustment instruction from the decision-making planning module 120 and a state of the environment perception module based on the system adjustment instruction. This facilitates the environment perception module to achieve a relatively favorable perception function. For example, the state of the environment perception module comprises: an internal parameter value of the environment perception element and/or a parameter of the multi-degree-of-freedom platform. That is, the environment perception module 110 may adjust the internal parameter value (for example, a focal length or the like) of the environment perception element and/or the parameter of the multi-degree-of-freedom platform (for example, an azimuth angle of the multi-degree-of-freedom platform, comprising a horizontal angle and/or a pitch angle, and the like) based on the system adjustment instruction. For another example, the state of the environment perception module further comprises a position or attitude of the environment perception element, or the like.

In some embodiments, the environment perception module is an environment perception circuit. That is, the environment perception module may be implemented by a circuit. In other embodiments, the environment perception module comprises a software part, or is implemented by combining software and hardware.

The decision-making planning module 120 is configured to receive an operation task and the perception result information, generate a motion trajectory based on the operation task and the perception result information, output the motion trajectory, predict the motion trajectory to obtain prediction result information, receive feedback result information corresponding to the motion trajectory, and output the system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information.

For example, an operation task (for example, a task such as excavation, loading, or the like) can be input to the decision-making planning module, so that the decision-making planning module receives the operation task. The decision-making planning module also receives the perception result information output by the environment perception module, generates a motion trajectory based on the operation task and the perception result information, and outputs the motion trajectory to the control performing module, so that the control performing module performs the operation task based on the motion trajectory.

The decision-making planning module may also predict the motion trajectory to obtain the prediction result information. For example, if the operation task is to lift the boom, the decision-making planning module predicts the motion trajectory of lifting the boom, and predicts whether a perception range (for example, a photographing range) of the environment perception element installed on the boom will be reduced. The decision-making planning module may also receive the feedback result information corresponding to the motion trajectory. The feedback result information is fed back by the control performing module.

In some embodiments, the feedback result information comprises a reaction result of an operation object of the construction machine. For example, the operation task is material excavation and loading, and the operation object is material. The reaction result of the operation object to the bucket is presented by the pressure change of the boom cylinder, and the pressure change condition of the cylinder is fed back to the decision-making planning module, so that the decision-making planning module may determine whether the current boom is confronted with a foreign object during the excavation process according to the pressure change condition of the cylinder.

The decision-making planning module may output the system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information. For example, if the decision-making planning module predicts that the perception range of the environment perception element installed on the boom (for example, the jib of the excavator) will be reduced, and the current state of the boom (for example, the height of the boom, or whether it is confronted with an obstacle, or the like) is determined according to the reaction result of the operation object of the construction machine, the system adjustment instruction can be output to the environment perception module, so that the environment perception module adjusts own state, for example, changing the focal length or angle, so as to increase the perception range.

In some embodiments, the decision-making planning module is a decision-making planning circuit. That is, the decision-making planning module may be implemented by a circuit. In other embodiments, the decision-making planning module comprises a software part, or is implemented by combining software and hardware.

The control performing module 130 is configured to receive the motion trajectory (as described previously, the motion trajectory is a trajectory generated after comprehensively considering the operation task and the perception result), perform the operation task based on the motion trajectory, obtain the feedback result information and the control result information during the performing of the operation task, output the feedback result information to the decision-making planning module, and output the control result information to the environment perception module.

In some embodiments, the control result information is part of the feedback result information. For example, taking an excavator as an example, the control result information comprises: the attitude information (for example, a jib angle, or the like) of the working device.

In some embodiments, the feedback result information comprises a reaction result of an operation object of the construction machine. For example, the feedback result information comprises not only the above-described control result information (for example, the attitude information of the boom), but also the information such as the reaction result of the operation object (for example, a pressure of an oil cylinder of a working device detected by a pressure sensor). The control performing module outputs the feedback result information to the decision-making planning module.

In other embodiments, the control performing module controls the machine to complete the operation task based on the motion trajectory and the reaction of the operation object.

In some embodiments, the control performing module is a control performing circuit. That is, the control performing module is implemented by a circuit. For example, the control performing module comprises a control unit and a performing mechanism. In other embodiments, the control performing module comprises a software part, or is implemented by combining software and hardware.

So far, a control system for a construction machine according to some embodiments of the present disclosure has been provided. The control system comprises: an environment perception module configured to receive environment information and/or control result information, output perception result information based on the environment information and/or the control result information, receive a system adjustment instruction, and adjust a state of the environment perception module based on the system adjustment instruction; a decision-making planning module configured to receive an operation task and the perception result information, generate a motion trajectory based on the operation task and the perception result information, output the motion trajectory, predict the motion trajectory to obtain prediction result information, receive feedback result information corresponding to the motion trajectory, and output the system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information; and a control performing module configured to receive the motion trajectory, perform the operation task based on the motion trajectory, obtain the feedback result information and the control result information during the performing of the operation task, output the feedback result information to the decision-making planning module, and output the control result information to the environment perception module. The above-described control system can improve the coordination of the construction machine during the process of performing an operation.

In addition, the above-described control system can enable the construction machine to take into account a corresponding motion of the environment perception module and its potential influence on the perception function and takes appropriate compensation measures when planning and performing a machine operation behavior during a smart operation process, thereby improving the coordination of the system.

In some embodiments, the decision-making planning module is further configured to adjust the motion trajectory based on the prediction result information and the feedback result information to facilitate environment perception. For example, if the decision-making planning module determines that an obstacle which blocks the perception of an ambient condition by the environment perception module is currently encountered based on the prediction result information and the feedback result information, the motion trajectory of the construction machine is adjusted so as to facilitate better perceiving an ambient condition by the environment perception module.

In some embodiments, the environment perception module 110 is further configured to output the perception result information directly to the control performing module. The control performing module 130 is further configured to receive the perception result information and perform an operation corresponding to the perception result information based on the perception result information. That is, some perception results may be directly applied to the control performing module not via the decision-making planning module. This can realize a response to the environment in time and improve the response speed of the control system.

For example, during the process of emergency avoidance, if the environment perception module determines that there is an obstacle ahead in a traveling direction of the construction machine according to the environment information, the perception result information about the presence of the obstacle is directly output to the control performing module, and the control performing module performs an avoidance operation based on the perception result information, for example, stopping traveling. This can improve the response speed of the control system and reduce the possibility of collision with the obstacle.

In some embodiments, the decision-making planning module 120 is configured to calculate and obtain an adjustment parameter based on the prediction result information and the feedback result information, and output the system adjustment instruction based on the adjustment parameter, wherein the system adjustment instruction comprises the adjustment parameter, and the adjustment parameter is configured to adjust the state of the environment perception module. The environment perception module 110 is configured to extract the adjustment parameter from the system adjustment instruction and adjust the state of the environment perception module based on the adjustment parameter. In this way, the decision-making planning module may adjust the state of the environment perception module, so that appropriate compensation measures can be taken for a corresponding motion of the environment perception module and its potential influence on the perception function, thereby improving the coordination of the system. For example, the adjustment parameter comprises: an internal parameter value (for example, a focal length, or the like) of the environment perception element and/or a parameter of the multi-degree-of-freedom platform. For example, the internal parameter value of the environment perception element comprises the focal length or the like. For another example, the parameter of the multi-degree-of-freedom platform comprises a horizontal angle and/or pitch angle of the multi-degree-of-freedom platform.

In some embodiments, the environment perception module 110 is further configured to transmit the environment information to the decision-making planning module, wherein the environment information is first data information with an element coordinate system of the environment perception module as a coordinate system. The decision-making planning module 120 is further configured to convert the first data information into second data information with a reference coordinate system of the construction machine as a coordinate system based on a transformation matrix between the element coordinate system and the reference coordinate system, and generate the motion trajectory based on the operation task and the second data information.

That is, the environment information detected by the environment perception module is data with own element coordinate system of the environment perception module as a coordinate system, while data processed by the decision-making planning module is data with the reference coordinate system as a coordinate system. Therefore, after receiving the environment information from the environment perception module, the decision-making planning module is required to convert the first data information with the element coordinate system as a coordinate system into the second data information with the reference coordinate system as a coordinate system based on the transformation matrix about the coordinate systems, and then generate a motion trajectory based on the operation task and the second data information.

In some embodiments, the transformation matrix between the coordinate system of each environment perception element and the reference coordinate system of the machine is measured and calculated in real time, so that a detection result of each environment perception element may be expressed to the reference coordinate system of the machine in time through appropriate spatial transformation and temporal alignment.

In some embodiments, the coordinate parameters (for example, an offset and an angle) between the element coordinate system and the reference coordinate system of the environment perception module may be obtained through physical measurement, multi-sensor detection fusion and other means, so as to obtain the transformation matrix and calibrate the environment perception module.

In some embodiments, the decision-making planning module 120 is further configured to adjust the motion trajectory based on the perception result information of the environment perception module, and output the motion trajectory adjusted to the control performing module 130. The control performing module 130 is further configured to control motion of the construction machine based on the motion trajectory adjusted. This can improve the strain capacity of the construction machine, and also improve the safety of the device.

For example, if the perception result information comprises the information that there is a pedestrian or obstacle ahead in a traveling direction of the construction machine, the decision-making planning module adjusts the motion trajectory (for example, a traveling trajectory of the construction machine) based on the perception result information, and outputs the motion trajectory adjusted to the control performing module. Then, the control performing module adjusts the motion trajectory of the construction machine based on the motion trajectory adjusted, so as to avoid collision with the pedestrian or obstacle as much as possible.

For another example, if the environment perception module perceives that there is a pedestrian or obstacle ahead in a traveling direction of the construction machine during a performing process of the construction machine according to a motion trajectory (such condition might affect the perception range of the environment perception module), such perception result information is sent to the decision-making planning module, so that the decision-making planning module adjusts the motion trajectory based on the perception result information and outputs the motion trajectory adjusted to the control performing module, and the control performing module controls the motion of the construction machine based on the motion trajectory adjusted, thereby changing the motion trajectory. By changing the motion trajectory, it is possible to improve the environment perception condition of the environment perception module.

In some embodiments, the decision-making planning module 120 is further configured to adjust the motion trajectory based on the feedback result information of the control performing module, and output the motion trajectory adjusted to the control performing module. The control performing module is further configured to control motion of the construction machine based on the motion trajectory adjusted. This can improve the strain capacity of the construction machine, thereby also improving the safety of the device.

For example, the feedback result information comprises information that a pressure of a bucket cylinder of an excavator exceeds a threshold value, so that the decision-making planning module determines that the bucket is confronted with a hard object based on the feedback result information, and adjusts the motion trajectory, and outputs the motion trajectory adjusted to the control performing module. The control performing module adjusts the motion trajectory of the construction machine based on the motion trajectory adjusted, for example, adjusts an excavation trajectory of the bucket, so as to bypass the hard object.

In some embodiments, the decision-making planning module 120 is further configured to send a control instruction to the control performing module 130 to control an operational state or operational speed of the control performing module 130 to make the operational state or operational speed of the control performing module 130 match an adjusted state of the environment perception module. This can improve the coordination of the construction machine.

For example, the decision-making planning unit may sacrifice some operation efficiency or other optimization indicators, so that the planning of the planned motion trajectory may cooperate with the normal work of the environment perception module, for example, planning that the construction machine is in a certain attitude or holds during the operation process. For example, if the pitch angle of the environment perception module changes relatively slowly, the decision-making planning module may reduce the operational speed of the actuator of the control performing module, so that the control performing module may match the adjusted state of the environment perception module.

In some embodiments, the environment perception module 110, the decision-making planning module 120 and the control performing module 130 contain software and/or hardware respectively. Specific hardware and software may be implemented in different methods. For example, perception algorithm, decision-making planning algorithm and advanced control algorithm may be deployed on one vehicle-mounted computing unit, or distributed in multiple vehicle-mounted computing units. In a case where the conditions of bandwidth, time delay, reliability and safety are satisfied, it is also possible to be partially deployed in computing units outside the vehicle, for example, control rooms of mines, construction sites and workshops (that is, edge-to-end deployment) or special computing centers (that is, cloud-edge-to-end deployment).

FIG. 2 is a flowchart showing a control method for a construction machine according to some embodiments of the present disclosure. As shown in FIG. 2, the control method comprises steps S202 to S210.

In step S202, by an environment perception module, environment information and/or control result information are received, and perception result information is output based on the environment information and/or the control result information.

In other embodiments, the environment perception module during its operation may also transmit its working state in addition to transmitting the perception result to a decision-making planning module.

In step S204, by a decision-making planning module, an operation task and the perception result information are received, a motion trajectory is generated based on the operation task and the perception result information, the motion trajectory is output, and the motion trajectory is predicted to obtain predicting result information.

In step S206, by a control performing module, the motion trajectory is received, the operation task is performed based on the motion trajectory, feedback result information and the control result information during the performing of the operation task are obtained, the feedback result information is output to the decision-making planning module, and the control result information is output to the environment perception module.

In step S208, by the decision-making planning module, the feedback result information corresponding to the motion trajectory is received, and a system adjustment instruction is output to the environment perception module based on the prediction result information and the feedback result information.

In step S210, by the environment perception module, the system adjustment instruction is received, and a state of the environment perception module is adjusted based on the system adjustment instruction.

So far, a control method for a construction machine according to some embodiments of the present disclosure is provided. In the control method, by the environment perception module, the perception result information is output based on the environment information and/or the control result information; by the decision-making planning module, the motion trajectory is generated and output based on an operation task and the perception result information, and the motion trajectory is predicted to obtain the prediction result information; by the control performing module, the operation task is performed based on the motion trajectory and the feedback result information and the control result information are obtained, and the feedback result information is output to the decision-making planning module, and the control result information is output to the environment perception module; by the decision-making planning module, a system adjustment instruction is output to the environment perception module based on the prediction result information and the feedback result information; and by the environment perception module, own state of the environment perception module is adjusted based on the system adjustment instruction. By way of the control method, the coordination of the construction machine during the process of performing an operation can be improved.

In some embodiments, the control method further comprises: the decision-making planning module adjusting the motion trajectory based on the prediction result information and the feedback result information to facilitate environment perception. In some embodiments, the control method further comprises: outputting the perception result information directly to the control performing module by the environment perception module; and receiving, by the control performing module, the perception result information, and performing an operation corresponding to the perception result information based on the perception result information. This can improve the response speed of the control system.

In some embodiments, the outputting, by the decision-making planning module, the system adjustment instruction to the environment perception module based on the prediction result information and the feedback result information comprises: calculating and obtaining, by the decision-making planning module, an adjustment parameter based on the prediction result information and the feedback result information, wherein the adjustment parameter is configured to adjust the state of the environment perception module; and outputting, by the decision-making planning module, the system adjustment instruction based on the adjustment parameter, wherein the system adjustment instruction comprises the adjustment parameter.

In some embodiments, the adjusting, by the environment perception module, the state of the environment perception module based on the system adjustment instruction comprises: extracting, by the environment perception module, the adjustment parameter from the system adjustment instruction; and adjusting, by the environment perception module, the state of the environment perception module based on the adjustment parameter.

In this way, the decision-making planning module adjusts the state of the environment perception module, so that appropriate compensation measures may be taken for a corresponding motion of the environment perception module and its potential influence on the perception function, thereby improving the coordination of the system.

In some embodiments, the feedback result information comprises a reaction result of an operation object of the construction machine.

In some embodiments, the perception result information comprises the environment information.

In other embodiments, the perception result information comprises a logical judgment result generated based on the environment information and/or the control result information.

In some embodiments, the control method further comprises: transmitting the environment information to the decision-making planning module by the environment perception module, wherein the environment information is first data information with an element coordinate system of the environment perception module as a coordinate system; and converting, by the decision-making planning module, the first data information into second data information with a reference coordinate system of the construction machine as a coordinate system based on a transformation matrix between the element coordinate system and the reference coordinate system, and generating the motion trajectory based on the operation task and the second data information. In some embodiments, the transformation matrix between the coordinate system of each environment perception element and the reference coordinate system of the machine is measured and calculated in real time, so that a detection result of each environment perception element may be expressed to the reference coordinate system of the machine in time through appropriate spatial transformation and temporal alignment.

In some embodiments, the control method further comprises: adjusting, by the decision-making planning module, the motion trajectory based on the perception result information of the environment perception module, and outputting the motion trajectory adjusted to the control performing module; and controlling, by the control performing module, motion of the construction machine based on the motion trajectory adjusted. That is, during a process of performing an operation trajectory, the decision-making planning module adjusts an operation trajectory of the machine in time according to an operational status of the environment perception module. This can improve the strain capacity of the construction machine, thereby also improving the safety of the device.

In some embodiments, the control method further comprises: adjusting, by the decision-making planning module, the motion trajectory based on the feedback result information of the control performing module, and outputting the motion trajectory adjusted to the control performing module; and controlling, by the control performing module, motion of the construction machine based on the motion trajectory adjusted. That is, during a process of performing an operation trajectory, the decision-making planning module adjusts operation planning in time according to an operational status of the control performing module. This can improve the strain capacity of the construction machine, thereby also improving the safety of the device.

In some embodiments, the control method further comprises: sending a control instruction to the control performing module by the decision-making planning module to control an operational state or operational speed of the control performing module to make the operational state or operational speed of the control performing module match an adjusted state of the environment perception module. This can improve the coordination of the construction machine.

In some embodiments, the decision-making planning module when planning the operation behavior of the construction machine calculates a motion trajectory of each environment perception element corresponding to the operation behavior and its influence on the perception function, so as to ensure that the environment perception element may provide sufficient quantity and quality of perception information in the planned motion as much as possible.

In other embodiments, during a process of performing an operation trajectory, the decision-making planning module may dynamically adjust the internal parameter value, position or attitude of the environment perception element according to actual conditions.

In the embodiment of the present disclosure, the construction machine comprises, but is not limited to, an excavator, a crane, a bulldozer, a loader or the like. The control system for a construction machine will be described below with a smart unmanned excavator as an example.

FIG. 3 is a schematic view showing an architecture system of a smart unmanned excavator according to some embodiments of the present disclosure.

As shown in FIG. 3, the environment perception module 110 (for example, an environment perception element) is installed on a vehicle body superstructure and/or a boom (for example, a jib) of an excavator 31. For example, the environment perception element may be fixed on a movable device on the machine body or the jib, and some structures on the machine body or the movable device may serve as a multi-degree-of-freedom platform (for example, a controllable platform or a multi-degree-of-freedom structural device). As mentioned previously, the multi-degree-of-freedom platform comprises, but is not limited to, some structures on the construction machine or an individually provided device. For example, the multi-degree-of-freedom platform is a jib 32 or a vehicle body superstructure 33. That is, the environment perception element may be installed on the jib 32 or on the top of the vehicle body superstructure 33. In this way, the environment perception module can adjust the position and the angle of view during operation of the excavator, and better perceive a target operation surface. The environment perception module collects the environment information of the target operation surface and sends the data to the decision-making planning module 120.

For example, the decision-making planning module 120 is installed on one or more vehicle-mounted computing units 34 on the excavator. For another example, when the conditions of bandwidth, time delay, reliability and safety are satisfied, the decision-making planning module 120 is partially deployed on the computing unit outside the vehicle, for example, the control room 37 of the mine, the construction site and the workshop (that is, edge-to-end deployment), or the special cloud computing center 38 (that is, cloud-edge-to-end deployment). The decision-making planning module receives the input of the operation task information, and outputs a motion trajectory to the control performing module 130 according to a perception result of the environment perception module. When the operation task of the excavator changes, for example, the slewing or excavation point changes, the decision-making planning module predicts the influence of the motion trajectory on the environment perception, calculates more appropriate perception parameters, and dynamically adjusts the position and angle of view of the environment perception element of the environment perception module, so that the environment perception module changes along with the motion of the excavator.

For example, the hardware of the decision-making planning module comprises, but is not limited to, a computing unit, such as an industrial computer, a board card, a central computer or the like.

The control performing module 130, as an interactive interface of command control performing data of the excavator, may control the excavator to complete an operation action such as traveling, excavation and motion of a working device according to the motion trajectory decision-making of the decision-making planning module, with an operation result as the input of the environment perception module. Furthermore, under particular operational conditions, such as emergency avoidance, the control performing module 130 may directly receive some perception results to implement a response to the environment in time. In addition, the control performing module 130 feeds back the reaction of the operation object and environment to the machine during the control performing process to the decision-making planning module 120.

FIG. 4 is a schematic view showing a joint coordinate model of a working device of a smart unmanned excavator according to some embodiments of the present disclosure.

FIG. 4 is a symmetrical plan view of the working device. A coordinate system involved in FIG. 4 uses a right-hand coordinate system. Therefore, FIG. 4 shows two of three coordinate axes of each coordinate system, and another coordinate axis perpendicular to these two coordinate axes is not shown. A coordinate system X₀Y₀Z₀ is a reference coordinate system, and O₀ is a coordinate origin. A point O is a link point between a jib and a excavator body. A coordinate system X₁Y₁Z₁ is a coordinate system at the point O. m1 is a horizontal offset of a position of the link point between the jib and the excavator from the reference coordinate origin, and n1 is a vertical offset of the position of the link point between the jib and the excavator from the reference coordinate origin.

A point P is a link point between an arm and the jib, a coordinate system X₂Y₂Z₂ is a coordinate system at the point P, a₂ is a length of the jib, and θ₂ is a joint angle of the jib. a₃ is a length of the arm, and θ₃ is a joint angle of the arm. A point Q is a link point between the arm and a bucket, a coordinate system X₃Y₃Z₃ is a coordinate system at the point Q, a₄ is a length of the bucket, and θ₄ is a joint angle of the bucket; φ is an orientation pitch angle of a bucket tooth. A coordinate system X₄Y₄Z₄ is a coordinate system at the bucket tooth.

F is a position where the environment perception module is installed on the jib, and I₀ is a distance from the installation position F of the environment perception module to the point O. β is a pitch angle of the multi-degree-of-freedom platform of the perception module. A coordinate system XgYgZg is a coordinate system at the point F.

FIG. 5 is a flowchart showing a control method for a smart unmanned excavator according to some embodiments of the present disclosure. As shown in FIG. 5, the control method comprises steps S501 to S517.

In step S501, attitude information of the excavator is received. For example, the attitude information of the excavator comprises a traveling state of the excavator, and the angle measuring device on the working device obtains the attitude information of the working device of the excavator.

In step S502, perception result information is received. For example, the decision-making planning module may receive a relevant parameter from the environment perception module.

In step S503, a geometric relationship (for example, a corresponding geometric relationship of the excavation point, the orientation pitch angle of the bucket tooth, the relative position and attitude of the working device relative to the excavator itself) is obtained. For example, an excavator attitude detection fusion system is established, and a three-dimensional coordinate model of the slewing device, the jib, the environment perception module (also referred to as the perception system), the arm and the bucket is established according to the excavator body superstructure containing four rigid rotary joints such as the slewing device, the jib, the arm and the bucket. In a certain operation range, a corresponding geometric relationship of the excavation point, the orientation pitch angle of the bucket tooth, the joint angle of the bucket, the joint angle of the boom and the joint angle of the arm relative to the position and attitude of the vehicle body is obtained, and the coordinates (Xg, Yg, Zg) of the installation position of the environment perception module, as well as the values of the horizontal angle α and the pitch angle β of the multi-degree-of-freedom platform of the environment perception module are solved, as shown in FIG. 4.

In step S504, a state of the excavator is planned or judged. For example, the decision-making planning module plans a working state of the excavator. For example, the working state of the excavator comprises traveling, slewing and excavation. Different working states will be described below respectively.

In step S505, the environment perception module is planned to be at the highest point. For example, when the excavator travels, the decision-making planning module formulates a planning control strategy, for example, lifting the working device high so that the perception element such as a camera and a laser radar can maintain sufficient forward vision.

In step S506, the attitude of the jib of the excavator is output. For example, given the coordinate values (Xg, Yg, Zg) of the environment perception module installed on the jib, the angle θ2 in FIG. 4 is obtained by solving, and the attitude of the jib is obtained, so that the attitude is output to the control performing module.

In step S507, the attitude is adjusted. For example, the control performing module adjusts the attitude of the working device and controls the working device to move to a planned point position.

In step S508, a parameter is defined. For example, the decision-making planning module defines a parameter of the environment perception module. For example, after the motion of the working device, the horizontal angle or the pitch angle of the environment perception module relative to the vehicle body coordinate system is changed, and the horizontal angle and the pitch angle of the environment perception module relative to the vehicle body coordinate system in the working state are defined. The values of the horizontal angle α and the pitch angle β of the multi-degree-of-freedom platform of the environment perception module are calculated, and the environment perception module is adjusted in real time. In step S509, the parameter is calculated. For example, the parameter such as a focal length of the environment perception module is calculated.

For example, when the excavator is in a slewing condition, the environment perception module rotates along with the excavator, and the information that a particular area or object to be perceived is within a perceivable range of the environment perception module becomes less, even beyond a coverage range of the environment perception module. By calculating and changing the parameter such as the focal length of the environment perception module, the decision-making planning module may achieve the purpose of expanding an angle range of the visual field, thereby obtaining more perception information.

In step S510, the perception range is obtained. That is, the perception range of the environment perception module on the target operation surface is obtained.

In step S511, a difference between the perception range and the operation range of the target area is compared.

For example, the three-dimensional information of a particular target perception object or area is compared, and the information of the visual field range of the environment perception module is compared with the information of the operation range of the target area in combination with the coordinates (Xg, Yg, Zg) and the visual field angle range of the environment perception module. The deviation value is calculated in real time during the slewing process, and the horizontal angle α of the environment perception module is dynamically adjusted. When the slewing condition is met, the environment perception module may obtain a great coverage time and space on the target operation surface, thereby improving the perception quality.

In step S512, an excavation point is planned. For example, when the excavator performs excavation, the decision-making planning module plans the coordinates of a next excavation point according to a perception result at a certain moment within the working range of the excavator.

In step S513, the coordinates (Xg, Yg, Zg) of the position of the environment perception module as shown in FIG. 4 are calculated.

In step S514, it is determined whether the excavation point is within the perception range. For example, whether the excavation point is within the perception range (that is, the visual field) of the environment perception module is determined by combining the internal parameter value of the environment perception element and the pitch angle and the horizontal angle of the multi-degree-of-freedom platform.

In step S515, the deviation is calculated. For example, if the excavation point is not within the visual field, the deviation between a center of the visual field of the environment perception module and the excavation point is calculated, and the internal parameter value of the environment perception element and the parameters (α, β) of the multi-degree-of-freedom platform are planned and adjusted.

In step S516, the internal parameter value of the environment perception element and/or the parameters of the multi-degree-of-freedom platform are calculated and output. For example, the decision-making planning module calculates and outputs the internal parameter value of the environment perception element and/or the parameters of the multi-degree-of-freedom platform, thereby outputting a corresponding system adjustment instruction.

In step S517, the internal parameter value of the environment perception element and/or the parameters of the multi-degree-of-freedom platform are adjusted. For example, the environment perception module receives the system adjustment instruction from the decision-making planning module and adjusts the environment perception attitude in real time, so as to better perceive the operation environment.

In addition, the control performing module receives the decision-making planning information, performs a corresponding action, and feeds back a performing result to the decision-making planning module, so as to perform control more accurately.

So far, the control method of the smart unmanned excavator according to some embodiments of the present disclosure has been provided. This can improve the coordination under the intricate and volatile operation tasks, objects and environments of the smart unmanned excavator. During the operation process of the smart unmanned excavator, a corresponding motion of the perception element and its potential influence on the perception function are considered while planning and performing a machine operation behavior, and appropriate compensation measures are taken, thereby improving the coordination of the system.

FIG. 6 is a structural block view schematically showing a control system for a construction machine according to other embodiments of the present disclosure. The control system comprises a memory 610 and a processor 620.

The memory 610 may be a magnetic disk, a flash memory or any other non-volatile storage medium. The memory is configured to store instructions in the embodiment corresponding to FIG. 2 and/or FIG. 5.

The processor 620 which is coupled to the memory 610, may be implemented as one or more integrated circuits, for example, a microprocessor or microcontroller. The processor 620 is configured to execute instructions stored in the memory, which can improve the coordination of the construction machine during the process of performing operations.

In an embodiment, as shown in FIG. 7, the control system 700 comprises a memory 710 and a processor 720. The processor 720 is coupled to the memory 710 through a bus 730. The control system 700 may also be connected to an external storage device 750 through a storage interface 740 to call external data, and may also be connected to a network or another computer system (not shown) through a network interface 760, which will not be described in detail here.

In this embodiment, data instructions are stored in the memory, and then the above-described instructions are processed by the processor, which can improve the coordination of the construction machine during the process of performing operations.

In some embodiments of the present disclosure, a construction machine is further provided. The construction machine comprises: the control system as described previously (for example, the control system shown in FIG. 1, FIG. 6 or FIG. 7). For example, the construction machine comprises, but is not limited to, an excavator, a crane, a bulldozer, a loader or the like.

In another embodiment, the present disclosure also provides a computer-readable storage medium (for example, a non-transitory computer-readable storage medium) having computer program instructions stored thereon that, when executed by a processor, implement the steps of the method in the embodiments corresponding to FIG. 2 and/or FIG. 5. Those skilled in the art will appreciate that the embodiments of the present disclosure may be provided as a method, device, or computer program product. Accordingly, the present disclosure may take the form of an entirely hardware embodiment, an entirely software embodiment, or a combination of software and hardware aspects. Moreover, the present disclosure may take the form of a computer program product embodied in one or more computer-usable non-transitory storage media (comprising but not limited to a disk memory, CD-ROM, an optical memory, or the like) containing computer usable program codes therein.

The present disclosure is described with reference to the flow charts and/or block views of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It will be understood that each step and/or block of the flow charts and/or block views as well as a combination of steps and/or blocks of the flow charts and/or block views may be implemented by a computer program instruction. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, an embedded processing machine, or other programmable data processing devices to produce a machine, such that the instructions executed by a processor of a computer or other programmable data processing devices produce a device for realizing a function designated in one or more steps of a flow chart and/or one or more blocks in a block view.

These computer program instructions may also be stored in a computer readable memory that may guide a computer or other programmable data processing device to operate in a particular manner, such that the instructions stored in the computer readable memory produce a manufacture comprising an instruction device. The instruction device realizes a function designated in one or more steps in a flow chart or one or more blocks in a block view.

These computer program instructions may also be loaded onto a computer or other programmable data processing devices, such that a series of operational steps are performed on a computer or other programmable device to produce a computer-implemented processing, such that the instructions executed on a computer or other programmable devices provide steps for realizing a function designated in one or more steps of the flow chart and/or one or more blocks in the block view.

In some embodiments of the present disclosure, a computer program product is also provided. The computer program product comprises instructions that, when executed by a processor, cause the processor to perform the control method described previously.

In some embodiments of the present disclosure, a computer program is also provided. The computer program comprises instructions that, when executed by a processor, cause the processor to perform the control method described previously.

Hitherto, the present disclosure has been described in detail. Some details well known in the art are not described in order to avoid obscuring the concept of the present disclosure. According to the above description, those skilled in the art would fully understand how to implement the technical solutions disclosed here.

Although some specific embodiments of the present disclosure have been described in detail by way of examples, those skilled in the art should understand that the above examples are only for an illustrative purpose, rather than limiting the scope of the present disclosure. It should be understood by those skilled in the art that modifications to the above embodiments may be made without departing from the scope and spirit of the present disclosure. The scope of the present disclosure is defined by the appended claims.