Method for Evaluating Golf Ball Hitting Process and Golf Club

Description

CROSS REFERENCE TO RELATED APPLICATION

This application claims priority to CN Patent Application No. 202411254440.X, filed Sep. 6, 2024 that is currently pending. This application is hereby incorporated by reference.

TECHNICAL FIELD

The present disclosure relates to the field of information processing, and specifically, embodiments of the present disclosure relate to a method for evaluating a golf ball hitting process and a golf club.

BACKGROUND OF THE INVENTION

The contact between the club head (specifically, the club face of the club head) and the ball is critical during the golf ball hitting process, as it determines a trajectory and a landing point of the ball. Golf learners often spend years to achieve a stable and good hitting contact, but due to the fast flight speed of the ball, the player cannot directly observe the moment of touch of the ball, so they need to use equipment to evaluate the contact position.

In order to obtain the contact position, related technologies adopt common cameras, emission analyzers or radar speed measurements and the like that are independent of the club as independent equipments to obtain the evaluation results of the hitting process. However, when using cameras and emission analyzers to obtain evaluation results, the system is relatively large and needs to be placed in front of the hitter, which brings a lot of inconvenience to the hitting; the system of using the radar speed measurements is also relatively large and needs to be placed behind the hitter, which can have an impact on the hitting process. In addition, there are at least the following technical defects in the relevant technical solutions: the corresponding equipment is expensive and has limited functions, and the accuracy of the evaluation results is poor.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present disclosure is to provide a method for evaluating a golf ball hitting process and a golf club. The embodiments of the present disclosure incorporate various sensors (including an image acquisition sensor, an inertial measurement unit, a microphone, etc.) as a part of the golf club, which can accurately measure the contact between the club head and the golf ball. At the same time, the embodiments of the present disclosure can also autonomously track and extremely accurately measure the swing path and speed, calculate the rotation and flight trajectory of the ball, and integrate the launch analysis, the ball touch measurements, and the swing analysis.

In the first aspect, the embodiments of the present disclosure provide a method for evaluating a golf ball hitting process, which operates on a golf club equipped with a processor and multiple types of sensors, comprising: obtaining multiple types of information collected in this hitting process, wherein the multiple types of information at least include video frame sequences, sound information and posture information; obtaining the evaluation results of the hitting process based on the multiple types of information, wherein the evaluation results of the hitting process comprise: a contact position of the ball with the club face, contact attributes of the club face with the ball and the club face with the ground, a motion trajectory of the club, and a rotation speed of the ball.

The embodiments of the present disclosure can collect multiple types of information in real-time by means of sensors integrated into the club, and evaluate the hitting process based on these information. This not only reduces costs but also improves the accuracy of the evaluation results, and reduces the impact of obtaining evaluation results on the hitter.

In some embodiments, the number of image acquisition sensors used to collect the video frame sequences is one, wherein obtaining the evaluation results of the hitting process based on the multiple types of information, comprising: obtaining a first image of the ball in contact with the club face from the video frame sequences; obtaining a homography matrix; converting the first image into an on-axis perspective contact diagram by means of the homography matrix; determining the pixel coordinate values of the contact point between the ball and the club face based on the on-axis perspective contact diagram to obtain the contact position of the ball with the club.

Some embodiments of the present disclosure could determine the contact position of the ball with the club head by means of images obtained by a camera, thereby improving the speed of data processing.

In some embodiments, determining the pixel coordinate values of the contact point between the ball and the club face based on the on-axis perspective contact diagram, comprising: selecting a target area on the on-axis perspective contact diagram, wherein the target area is a contact area between the club face and the ball on the on-axis perspective contact diagram; constructing a circular projection surface area in the vicinity of the target area of the on-axis perspective contact diagram according to the actual size and shape of the ball; taking a center position of the circular projection surface area as the contact position of the ball with the club face.

The embodiments of the present disclosure can determine the final contact position of the ball with the club head in combination with the actual size of the golf ball, thereby improving the accuracy of the obtained contact position.

In some embodiments, the image acquisition sensors used to collect the video frame sequences comprise a first image acquisition sensor and a second image acquisition sensor, the first image acquisition sensor and the second image acquisition sensor are disposed on the club with a predetermined rotation angle and a predetermined spatial displacement, wherein obtaining the evaluation results of the hitting process based on the multiple types of information, comprising: obtaining a first perspective video frame sequence collected by the first image acquisition sensor; obtaining a second perspective video frame sequence collected by the second image acquisition sensor; and obtaining three-dimensional coordinates of the contact position of the ball with the club face according to the first perspective video frame sequence and the second perspective video frame sequence.

The embodiments of the present disclosure can accurately obtain the three-dimensional coordinates of the contact position of the ball with the club head through the images of different perspectives shot by the two cameras.

In some embodiments, obtaining the evaluation results of the hitting process based on the multiple types of information, comprising: obtaining a deformation amount of the club in the hitting process; determining an initial motion trajectory of the club at least based on the video frame sequences; and obtaining a target motion trajectory of the club according to the deformation amount correcting the initial motion trajectory.

The embodiments of the present disclosure can determine the motion trajectory of the club through images collected by the image acquisition sensor and pre-estimated deformation amount, thereby improving the accuracy of the obtained evaluation results.

In some embodiments, determining the initial motion trajectory of the club face at least based on the video frame sequences, comprising: obtaining posture information at a first time by an posture information acquisition sensor to obtain a first position and a first direction; obtaining a club pose image shot at a time closest to the first time from the video frame sequences, and obtaining a second position and a second direction by the club pose image; and fusing the first position and the second position to obtain a target position at the first time, and fusing the first direction and the second direction to obtain a target direction at the first time, wherein the initial motion trajectory is a motion path determined by the target positions and the target directions at multiple times.

Some embodiments of the present disclosure can obtain a more accurate initial motion trajectory by fusing the pose information obtained by the images and using the pose information obtained by the IMU, thereby improving the accuracy of obtaining the motion trajectory of the club.

In some embodiments, obtaining the evaluation results of the hitting process based on the multiple types of information, comprising: identifying rotation markers on corresponding ball in the video frame sequences; estimating the rotation speed of the ball based on the rotation markers.

Some embodiments of the present disclosure can evaluate the rotation speed of the ball by analyzing the markers disposed on the spherical surface of the golf ball shot in the image, thereby improving the accuracy of this value.

In some embodiments, the contact attributes include a contact order and a contact intensity, wherein obtaining the evaluation results of the hitting process based on the multiple types of information, comprising: determining the contact order of the club with the ball and the club with the ground according to the attribute information of the sound information, or determining the contact intensity of the club with the ground or the lawn according to the attribute information of the sound information.

Some embodiments of the present disclosure can determine the contact attributes of the club with the ball as well as the club with the ground by analyzing the sound information collected by the microphone, thereby reducing costs and improving the speed of obtaining contact attribute results.

In some embodiments, obtaining the evaluation results of the hitting process based on the multiple types of information, comprising: determining a time T1 when the ball contacts the club by means of the sound information collected by a sound information acquisition sensor; screening out a target image shot at a time closest to the time T1 from the video frame sequences; screening out a second image of first contact between the club and the ground from the video frame sequences; confirming the relative shooting order of the target image and the second image in the video frame sequences so as to obtain the contact order of the club with the ball and the club with the ground.

Some embodiments of the present disclosure can obtain the contact order of the club with the ball and the club with the ground through images and sound information collected by the microphone, thereby improving the accuracy of the evaluation results.

In some embodiments, the video frame sequences are obtained by an event sensor.

Some embodiments of the present disclosure can obtain enough image frames by a high-speed camera (e.g., an event sensor), thereby improving the accuracy of obtaining the evaluation results by means of images and reducing the time for synchronizing different sensors to improve the data processing speed.

In the second aspect, some embodiments of the present disclosure provide a data acquisition and processing device mountable to a golf club, comprising: an image acquisition sensor configured to collect multi-frame images in the club hitting process to obtain video frame sequences; a posture information acquisition sensor configured to collect the posture information of the club; a sound information acquisition sensor configured to collect the sound information in the hitting process; and a computing processing unit configured to perform the method in any one of the embodiments of the first aspect.

In some embodiments, the data acquisition and processing device further comprising an ambient light sensor configured to obtain ambient light information of a scene in which the club is located; wherein an exposure parameter value of the image acquisition sensor is determined based on the ambient light information.

In the third aspect, embodiments of the present disclosure provide a golf club comprising the data acquisition and processing device of the second aspect and a club.

The embodiments of the present disclosure can collect multiple types of information through multiple sensors provided on the club, thereby obtaining different types of evaluation results, improving the universality of the technical solution.

In some embodiments, the posture information acquisition sensor, the sound information acquisition sensor and the computing processing unit included in the data acquisition and processing device are arranged inside the club.

The embodiments of the present disclosure can provide the sensors inside the club, at least to minimize the effect of the sensors on the balance and weight of the club as compared to providing it outside the club.

In some embodiments, at least a portion of the image acquisition sensor is located inside the club.

In some embodiments, an optical portion of the image acquisition sensor is located outside the club and the remaining portion is located inside the club.

Some embodiments of the present disclosure can only provide the optical portion of the image acquisition sensor outside the club, which can maximize the solution to the problem of affecting the hitting line of sight caused by providing the image acquisition sensor outside the club.

In the second aspect, some embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, can implement the method described in any one of the embodiments of the first aspect.

In the third aspect, some embodiments of the present disclosure provide a computer program product comprising a computer program, the computer program, when executed by a processor, implements the method described in any one of the embodiments of the first aspect.

BRIEF DESCRIPTION OF THE DRAWINGS

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings required to be used in the embodiments of the present disclosure will be briefly described below, it should be understood that the following drawings only illustrate some embodiments of the present disclosure and therefore should not be considered as limiting the scope, and it will be apparent to those skilled in the art that other relevant drawings can be obtained based on these drawings without inventive effort.

FIG. 1 is a first schematic view of a golf club according to an embodiment of the present disclosure;

FIG. 2 is a second schematic view of a golf club according to an embodiment of the present disclosure;

FIG. 3 is a flow chart of a method for evaluating the golf ball hitting process according to an embodiment of the present disclosure;

FIG. 4 is a schematic view of a club face composition according to an embodiment of the present disclosure;

FIG. 5a is a diagram illustrating a pre-and post-transformation comparison based on a homography matrix according to an embodiment of the present disclosure;

FIG. 5b is a schematic view illustrating a process of obtaining a contact position based on a size of a golf ball according to an embodiment of the present disclosure;

FIG. 6 is a schematic view of a golf club with two cameras according to an embodiment of the present disclosure;

FIG. 7 is a schematic diagram of a process for obtaining a position and a direction of any point on the motion trajectory of the club by fusing information of two types of sensors according to an embodiment of the present disclosure.

DETAILED DESCRIPTION OF THE INVENTION

The technical solutions in the embodiments of the present disclosure will be described below with reference to the drawings in the embodiments of the present disclosure.

It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined or explained in subsequent figures. Meanwhile, in the description of the present disclosure, the terms “first”, “second”, and the like are used only for distinguishing the description, and are not construed as indicating or implying relative importance.

To address at least some of the problems of the background art, embodiments of the present disclosure employ a high-speed camera integral with the club to directly view the contact of the ball with the club face, and can measure the coordinate values of the contact points of the ball with the club face very accurately by a visual algorithm homography (i.e., a homography matrix), and embodiments of the present disclosure can accurately measure the contact order and the contact intensity of the club head with the ball and the lawn by using a posture information acquisition sensor (e.g., an inertial measurement unit IMU) and a sound information sensor (e.g., a microphone), or obtain the motion trajectory of the club through the information of the corresponding sensor (because the sensor is fixed on the club, the collected information can reflect the motion trajectory of the club more accurately, namely, the direction angle, the position and the like of each time of the movement of the club are obtained), so that more complete evaluation results of the hitting process can be provided and the accuracy of the obtained evaluation results is improved.

That is, in the embodiments of the present disclosure, incorporating a camera and a sensor (e.g., an inertial measurement unit for collecting posture information and a microphone for collecting sound information) as a part of a golf club can accurately measure the contact between the club head and the golf ball. Meanwhile, the golf club of the embodiments of the disclosure can also track autonomously and measure the swing path and speed extremely accurately. In addition, the golf club of the embodiments of the present disclosure can also calculate the rotation and flight trajectory of the ball, and integrate the launch analysis, the ball touch measurement, and the swing analysis (i.e., all performed by the golf club).

It is not difficult to understand that the data acquisition and processing device provided in the embodiments of the present disclosure can be used as an accessory of the existing golf club, or it can be fixed to the club to use the club as a replacement for the club shaft, or it can be fixed on a completed golf club with the club and the club head as a dedicated practice club or a golf club equipped with new functions.

Referring to FIG. 1, FIG. 1 shows a golf club according to an embodiment of the present disclosure, which includes a grip 110, a club 120, a club head 130, and a data acquisition and processing device 140. It should be noted that, the data acquisition and processing device 140 according to the embodiment of present disclosure exemplarily comprises multiple types of sensors and a computing processing unit, and the embodiment of the present disclosure collects multiple types of information by means of the multiple types of sensors disposed on the club, so as to obtain the types of the evaluation results, thereby improving the universality of the technical solution.

For example, as shown in FIG. 2, in some embodiments of the present disclosure, the data acquisition and processing device 140 exemplarily comprises: an image acquisition sensor 141, a posture information acquisition sensor 142, a sound information acquisition sensor 143, and a computing processing unit 145, which are provided inside the club 120.

The embodiments of the present disclosure can provide the sensors inside the club, at least to minimize the effect of the sensors on the balance and weight of the club as compared to the sensors provided outside the club.

In some embodiments of the present disclosure, at least part of the image acquisition sensor 141 is located inside the club 120.

In some embodiments of the present disclosure, an optical portion (not shown) of the image acquisition sensor 141 is located outside the club 120 and the remaining portion is located inside the club. It should be noted that, FIG. 1 is only used to illustrate the approximate position relationship between the club and the data acquisition and processing device, and does not limit that the data acquisition and processing device must be located entirely inside the club.

It is not difficult to understand that, some embodiments of the present disclosure only provide the optical part of the image acquisition sensor outside the club, which can maximize the solution to the problem of affecting the hitting sight caused by providing the image acquisition sensor outside the club.

As shown in FIG. 2, in some embodiments of the present disclosure, the data acquisition and processing device further includes an ambient light sensor 144 configured to obtain ambient light information of a scene in which the club is located; wherein an exposure parameter value of the image acquisition sensor is determined based on the ambient light information.

For example, in some embodiments of the present disclosure, the ambient light information is collected by the ambient light sensor 144, and then the exposure parameter value of the image acquisition sensor is adjusted according to the ambient light information by the image acquisition sensor or by the computing processing unit, so as to obtain a collected image with better quality.

As shown in FIG. 2, the evaluation results of the hitting process obtained by using the data acquisition and processing device or the golf club according to the embodiment of the present disclosure exemplarily includes: the contact location of the ball with the club face, the contact attributes (the contact order or the contact intensity) of the club with the ball and the club with the ground, the motion trajectory of the club (determined by the direction and location of the various motion trajectory points), and the rotation speed of the ball.

A method for evaluating the golf ball hitting process executed by the data acquisition and processing device shown in FIG. 1 will be exemplarily explained below.

As shown in FIG. 3, embodiments of the present disclosure provide a method for evaluating the golf ball hitting process, operating on a golf club (shown in FIG. 1) equipped with a processor and multiple types of sensors, the method comprising:

S101, obtaining multiple types of information collected in the hitting process.

It should be noted that, the multiple types of information described in S101 at least include video frame sequences, sound information, and posture information. For example, in some embodiments of the present disclosure, the video frame sequences are obtained by shotting the hitting process by means of the image acquisition sensor (such as a camera) installed on the golf club, the sound information is obtained by collecting the sound information of the hitting process by means of the sound information acquisition sensor (such as a microphone) installed on the golf club, the posture information is obtained by collecting the posture signal of the hitting process by means of the posture information acquisition sensor (such as IMU) installed on the golf club.

For example, in some embodiments of the present disclosure, a high-speed camera capable of selecting a shooting rate greater than or equal to 1000 frames/second may be selected to collect the video frame sequences, in some embodiments of the present disclosure, a sensor capable of detecting a six-degree-of-freedom motion IMU may be selected as the sound information acquisition sensor, in some embodiments of the present disclosure, the sound information may be collected by the microphone, and the ambient light sensor 144 may be used to collect ambient light information to adjust the exposure parameter of the image acquisition sensor according to the ambient light information. A mechanical housing securely installs the camera and sensor onto the club through a designed rigid connection.

S102, obtaining the evaluation results of the hitting process based on the multiple types of information, wherein the evaluation results of the hitting process comprise: the contact location of the ball with the club face, the contact attributes of the club face with the ball and the club face with the ground, the motion trajectory of the club, and the rotation speed of the ball.

For example, some embodiments of the present disclosure perform a method for obtaining the evaluation results of the hitting process based on the multiple types of information by the computing processing unit 145 in FIG. 2. For example, the computing processing module includes a computing module (SoC based on ARM or QCOM) and a communication module, and the computing processing module is used for connecting various sensors.

It is not difficult to understand that, the embodiments of the present disclosure collect multiple types of information in real time by means of sensors integrated on the golf club, and evaluate the hitting process based on these information, thereby reducing costs and improving the accuracy of evaluation results.

The examples of obtaining the contact position of the ball with the club face (i.e., the club face included in the club head) by the image acquisition sensor will be exemplarily explained below, wherein the club face (TheClubFace) refers to a plane on the golf club that contacts the ball, i.e., a portion that directly impacts the ball when hitting the ball, and the design and material of the club face are important to the accuracy, strength, and rotation of the hitting ball.

As shown in FIG. 4, FIG. 4 is a schematic view of a club face composition, the club face shown includes: top edge 131, sweet spot 132, toe 133, leading edge 134, heel 135, and hosel 136, wherein the top edge (TopEdge) is the uppermost edge portion of the golf ball face, affecting the aerodynamic properties and flight stability of the ball; the sweet spot (Sweet Spot) of the club face is the optimal hitting position of the golf club, and when the club hits the ball at the right angle and path, the sweet spot can produce the optimal hitting effect, so that the ball flies straighter and farther; the hosel (Hosel) is the portion connecting the club shaft and the club head, the hosel ensures a stable connection between the club shaft and the club head and helps to transmit the forces when hitting the ball; toe (Toe) is the forward portion of the club head, and, relative to the heel, avoiding hitting the ball with the toe can reduce the likelihood of deviation from the target line when hitting the ball; the heel (Heel) is the rear portion of the club head, opposite the toe, where the ball should contact the center or sweet spot of the club face rather than the heel in the normal hitting process; the leading edge (Leading Edge) is the foremost edge of the club face and is also the portion of the club that first contacts the ball when hitting the ball, and its design and shape are critical to the accuracy and control of the hitting.

It should be noted that, the embodiments of the present disclosure can accurately determine the contact position of the ball with the club face, thereby can determine the effect of the ball hitting at this time, and finally can assist the hitter in improving the hitting skill.

For example, in some embodiments of the present disclosure, the number of image acquisition sensors to collect the video frame sequences is one, and the process of obtaining the evaluation results of the hitting process based on the multiple types of information in S102 exemplarily comprises:

the first step, obtaining a first image of the ball in contact with the club face from the video frame sequences.

For example, the first image is an image as shown in the left figure of FIG. 5a, which shows the process of contact of the golf ball 150 with the club face 137 during a hitting process.

The second step, obtaining a homography matrix.

How to obtain the homography matrix and the process of homography transformation will be exemplarily explained below. A first sub-step, matching feature points: the corresponding feature points are detected and matched in the image of the slanted perspective and the intended on-axis perspective (or referred to as a reference image), which can be accomplished by a feature detection algorithm such as SIFT, SURF or ORB. A second sub-step, calculating the homography matrix, and with enough matching points, the homography matrix can be estimated by using RANSAC algorithm or least square method. This matrix describes the transformation from the slanted perspective to the on-axis perspective of the club face.

It should be noted that, the homography matrix is used to describe the mapping relationship between two planes, and in particular, as shown in the right figure of FIG. 5a, the homography matrix is used to describe the mapping relationship between the plane 15 parallel to the paper surface and the club face 137 in FIG. 5a.

The third step, converting the first image into an on-axis perspective contact diagram by means of the homography matrix.

Some embodiments of the present disclosure may map all pixels in the slanted perspective image, i.e., the first image, to a new on-axis perspective using the calculated homography matrix, which typically involves image re-projection and interpolation, so as to ensure that the image at the new perspective is continuous and complete. It should be noted that in some embodiments of the present disclosure, the image may be post-processed, such as cropped, scaled, or color adjusted, to achieve a more visual effect.

In the on-axis perspective contact diagram shown on the right figure of FIG. 5b, this diagram shows the corresponding image after the first image of the left figure of FIG. 5a is converted into the on-axis perspective contact diagram.

The fourth step, determining the pixel coordinate values of the contact point of the ball with the club face according to the on-axis perspective contact diagram to obtain the contact position of the ball with the club.

That is, in some embodiments of the present disclosure, before employing the homography matrix transformation, the contact position of the ball with the club face is the position of the ball on the image shot at a slanted perspective, i.e., the projected position of the ball on the club face when the club face forms any angle with the horizontal plane in the actual hitting process; and obtaining the position of the ball on the image shot at the on-axis perspective after employing the homography matrix transformation, the image corresponding to the on-axis perspective, namely the on-axis perspective contact diagram, is the position of the projection of the ball on the surface when the club face is parallel to the paper surface.

For example, in some embodiments of the present disclosure, determining the pixel coordinate values of the contact points of the ball with the club face from the on-axis perspective contact diagram (the image corresponding to the right figure of FIG. 5b) in the fourth step exemplarily comprises: selecting a target area on the on-axis perspective contact diagram (the image corresponding to the right figure in FIG. 5b), wherein the target area (not shown, i.e., the contact area of the ball with the club face in the right figure in FIG. 5b) is the contact area of the club face with the ball on the on-axis perspective contact diagram; constructing a circular projection surface area in the vicinity of the target area of the on-axis perspective contact diagram according to the actual size and shape of the golf ball (a circular projection surface area 153 shown in the left figure of FIG. 5b, which is a circular area obtained by a perspective matrix according to the actual shape and size of the golf ball on the imaging plane of the on-axis perspective contact diagram); Taking the center position of the circular projection surface area (the center position in the left circular projection surface area shown in FIG. 5b, not shown) as the contact position of the ball with the club face. It is not difficult to understand that, the embodiments of the present disclosure can determine the final contact position of the ball with the club head in combination with the actual size of the golf ball, which can improve the accuracy of the obtained contact position.

The method and principle of obtaining the contact position of the ball with the club face by a homography will be exemplarily explained below with reference to FIG. 5a and FIG. 5b.

The left to right figures of FIG. 5a illustrate the process of straightening the club face by the homography in some embodiments of the present disclosure, the right to left figures of FIG. 5b illustrate the accurate estimation of the contact point of the club surface using the golf ball size (the size is mapped on the imaging plane of the on-axis perspective by the perspective matrix to obtain the circular projection surface area shown in FIG. 5b, i.e., the circular projection surface area 153) and the information of the club face in some embodiments of the present disclosure. It should be noted that, the surface of the club is parallel to the page surface in both the right figure of FIG. 5a and the right figure of FIG. 5b, and the right diagram of FIG. 5a shows a plane parallel to the page surface (i.e., a plane 152 parallel to the page surface). Some embodiments of the present disclosure straighten the club face of the left figure of FIG. 5a by the homography, which is a 2D transformation describing the mapping relationship between two planes in computer vision and image processing. It is commonly used to map points in one image to corresponding points in another image, these two images may be from different perspectives or shot by different cameras. When converting from the slanted perspective into the on-axis perspective, the homography can be used to correct perspective distortion in the image, which appears to be shot from the on-axis perspective. This is very common in document scanning, image soles inside buildings, etc.

The left figure of FIG. 5a is an image of the moment when the club contacts with the ball (or an image synthesized by an interpolation algorithm or the like) selected from the video frame sequences of the hitting process shot by the high-speed camera, the right figure of FIG. 5a is the on-axis perspective contact diagram obtained by multiplying the homography matrix obtained in advance by the matrix corresponding to the left figure of FIG. 5a, and it can be seen from this diagram that the ball is deformed when the club face is converted into a perspective parallel to the paper surface, so that it is unreasonable to estimate the contact point of the club with the ball by the deformed ball. In order to solve this technical problem, the embodiments of the present disclosure select a target area from the right figure of FIG. 5a, the target area is the area where the club contacts the ball (this area has less deformation during the homography transformation). Then, based on this area and the size of the golf ball, a circular virtual golf ball is generated (the size of the pixel image of the golf ball on this plane is determined based on the internal or external parameters of the high-speed camera) to obtain the left figure of FIG. 5b, the right figure of FIG. 5b is the same image as the right figure of FIG. 5a. The groove area is marked on the club face of the left figure of FIG. 5b, so after determining the contact position of the club face with the ball, the effect of this hitting can be determined by judging whether it is located in that area.

That is, some embodiments of the present disclosure have been able to determine the contact points of the club surface by calibrated perspective/distortion single camera (a single camera continuously shots a picture of the ball contacting the club to obtain the video frame sequences) using the following information and techniques: 1) precise ball size (all golf balls are the same size); 2) perspective transformation, converting the slanted perspective into an axial perspective (refer to FIG. 5a and FIG. 5b).

It is not difficult to understand that, some embodiments of the present disclosure determine the contact location of the ball with the club head (e.g., the ball with the club face located at the club head) from images obtained by a camera, thereby improving the speed of data processing.

A method of obtaining three-dimensional position points of the ball contacting the club face by two image acquisition sensors will be exemplarily described below with reference to FIG. 6.

As shown in FIG. 6, for example, in some embodiments of the present disclosure, the image acquisition sensors for collecting the video frame sequences include a first image acquisition sensor 146 and a second image acquisition sensor 147, the first image acquisition sensor 146 and the second image acquisition sensor 147 are disposed on the club 10 at predetermined rotation angles and spatial displacements, and the corresponding process of obtaining the evaluation results of the hitting process based on multiple types of information in S102 exemplarily includes: obtaining a first perspective video frame sequence collected by the first image acquisition sensor; obtaining a second perspective video frame sequence collected by the second image acquisition sensor; and obtaining three-dimensional coordinates of the contact position of the ball with the club face according to the first perspective video frame sequence and the second perspective video frame sequence.

That is, in other embodiments of the present disclosure, a dual camera configuration may be used, with the two cameras mounted at a certain rotation angle and a spatial displacement, so that stereosynthesizing the 3D position of the contact point of the ball relative to the club surface. The position, orientation of the camera and its design (focal length, field angle, resolution) are all optimized by end-to-end performance prediction. For example, two cameras may measure the relative position of the ball and club head through triangulation. For example, a pair of cameras may implement a 2D to 3D mapping or a 2D to 3D translation may also be implemented by a continuously changing position of one camera. This is helpful in estimating the position of the ball relative to the club head surface. The specific details are as follows: by using images shot from different perspectives, the points in the 3D scene and the position of the camera can be calculated using feature matching between the images, this process is called 3D Reconstruction, the 3D Reconstruction process comprising: the first step, feature detection and matching, i.e., firstly, key points (e.g., corners, edges, etc.) are detected in the input image, and these key points should be able to be stably identified among different images. Then, the key points are described through feature descriptors (such as SIFT, SURF, ORB and the like), and matched feature point pairs are found among different images; the second step, a Fundamental Matrix can be estimated by using the matched feature point pairs, wherein the Fundamental Matrix is a 3×3 matrix and describes the geometric relationship of corresponding points among different perspectives, in particular Epipolar Geometry; the third step, if the internal parameters (focal length, optical center, etc.) of the camera are known, the essential matrix can be calculated from the Fundamental Matrix; the fourth step, in computer view and three-dimensional reconstruction, “Triangulated to get the 3D points” refers to calculating three-dimensional coordinate points of the object from the images under multiple perspectives by a Triangulation method, which exemplarily involves the steps of camera calibration, feature extraction, feature matching and triangulation, and the like. For a specific three-dimensional reconstruction process, reference may be made to related prior art, and no redundant description is made herein for avoiding repetition.

Typically, an initial factory calibration is done during the product manufacturing process to obtain an “internal parameter” of camera attributes (e.g., focal length, center pixel), and an “external parameter” of camera orientation and position relative to a global coordinate system (e.g., club face). During the club's life, the camera may drift and move relative to the initial calibration. The external parameter may be calibrated periodically with respect to the club face, and the internal parameter may require the club to be sent to a service center or on site for calibration using calibration fittings.

It is not difficult to understand that, embodiments of the present application can accurately obtain three-dimensional coordinates of the contact location of the ball with the club head through images from different perspectives shot by the two cameras.

A process for measuring a swing trajectory according to some embodiments of the present disclosure will be will be exemplarily explained below.

For example, in some embodiments of the present disclosure, the process of obtaining evaluation results of the hitting process based on multiple types of information in S102 exemplarily comprises:

• • the first step, obtaining the deformation amount of the club in the hitting process.

The second step, determining an initial motion trajectory of the club based on at least the video frame sequences.

In some embodiments of the disclosure, if a high-speed camera is adopted, the initial motion trajectory is directly determined based on the image; if a low-speed camera is adopted, the IMU and the image information need to be fused to obtain an initial motion trajectory. For example, in some embodiments of the disclosure, the process of determining the initial motion trajectory of the club face based on at least the video frame sequences in the second step exemplarily comprises: obtaining posture information at the first time (belonging to any time in this hitting process) through an posture information acquisition sensor to obtain a first position, a first direction and a first speed; obtaining a club pose image shot at a time closest to the first time from the video frame sequences, and obtaining a second position and a second direction through the club pose image; and fusing (e.g., a fusion algorithm such as a weighted sum or an alternative) the first position and the second position to obtain a target position at the first time, and fusing the first direction and the second direction to obtain a target direction at the first time, wherein the initial motion trajectory is a motion path determined by the target positions and the target directions at multiple times. It is not difficult to understood that, some embodiments of the present disclosure obtain a more accurate initial motion trajectory by fusing pose information obtained by the image and pose information obtained by the IMU, thereby improving the accuracy of obtaining the club motion trajectory.

That is, the process of fusing visual information and posture information according to some embodiments of the present disclosure as shown in FIG. 7 exemplarily includes: shotting an image in the hitting process by using a camera, carrying out feature extraction and matching on the shot image, and carrying out pose estimation based on image features to obtain the second position and the second direction information; obtaining information of each axis collected by the IMU, and performing IMU data integration on the information to obtain the first position, the first direction and the first speed; and fusing the first position and the second position and fusing the first direction and the second direction through a fusion module to obtain a target position and a target direction (corresponding to a certain collection time). The real-time trajectory information of the image acquisition sensor and the IMU unit can be obtained by connecting these trajectory points, due to the image acquisition sensor and the IMU being arranged on the club of the golf, the motion trajectory of the club can be preliminarily reflected in real time through data collected by these sensors.

The third step, correcting the initial motion trajectory according to the deformation amount to obtain a target motion trajectory of the club.

In some embodiments of the disclosure, the motion trajectory of the club is determined through the images collected by the image acquisition sensor and the pre-estimated deformation amount, thereby improving the accuracy of the obtained evaluation results.

That is, some embodiments of the present disclosure determine the angle of swing by means of the deformation amount of the club, and the swing angle is in a positive correlation with the deformation amount of the club. Since the camera can not only measure the ball but also capture the external environment, some embodiments of the present disclosure can estimate the self-posture (position and direction, 6 degrees of freedom) of the golf club in real time using a Visual Inertial Odometer (VIO) method, so that it can measure the swing path and speed of the golf ball very accurately. In some embodiments of the present disclosure, the club shaft may flex during a swing and at the moment of hitting, the embodiments of the present disclosure model for this type of bending using known shaft models and the swing speed, or compensate for it in the algorithm when reporting the swing path by capturing and measuring it with a camera. For example, if the image acquisition sensor has a camera with 1000 frames per second, it may be possible to reduce or attenuate the effect of signals of the inertial measurement unit (IMU) when calculating the self-posture, typically the IMU reads data at a frequency of 200 to 1200 hertz, but consumes less power than the camera.

Visual-Inertial Odometer (VIO) is a common algorithm in Computer Vision (CV) by which the spatial position, angle, speed and angular velocity of the sensor itself can be deduced. Visual Inertial Odometer (VIO) is a position tracking technology that combines a visual camera and an inertial sensor (IMU). Visual Inertial Odometer (VIO) is a technique that uses one or more cameras, one or more inertial sensors IMU for sensor state measurements. The “state” herein refers to physical quantities such as the posture, the speed, etc. of an intelligent agent (e.g., a drone) in a particular degree of freedom. The Visual Inertial Odometer (VIO) is an important position tracking technology, and performs accurate state estimation on the swing trajectory of a golf club by fusing visual information (namely, the video frame sequences of the hitting process collected by the high-speed camera) and inertial information (namely, information collected and output by the inertial sensor IMU). For example, in some embodiments of the present disclosure, a method of estimating the real-time position, speed, and direction of the club exemplarily includes: obtaining the video frame sequences corresponding to the hitting process from the image acquisition sensor; obtaining accelerometer and gyroscope measurements from the inertial sensor IMU, including, for example, linear acceleration and angular velocity, etc.; collectively estimating a set of variable values related to the club by using the raw data collected by at least one of the visual sensor and the inertial sensor IMU, e.g., estimating the real-time position, speed, and direction of the club, etc. For example, in some embodiments of the present disclosure, if it is confirmed that the frequency of the video sequence collected by the visual information acquisition unit such as the image acquisition sensor is greater than a set threshold, the swing trajectory is estimated according to the information of the visual information acquisition unit; if it is confirmed that it is smaller than the set threshold, the swing attribute information needs to be collectively determined by combining the data collected by the visual information acquisition unit and the IMU unit.

The technical solution for obtaining the rotation speed of the ball will be exemplarily explained below.

In some embodiments of the present disclosure, the process of obtaining evaluation results of the hitting process based on multiple types of information in S102 exemplarily comprises:

• • the first step, identifying the rotation markers on the corresponding ball in the video frame sequences.

For example, in some embodiments of the present disclosure, images containing the balls are screened out from the video frame sequences by a computer vision algorithm and arranged in shooting order, and further rotation attribute markers are identified from each ball based on the computer vision algorithm. For example, these attribute markers may be dots or lines or other easily identifiable markers or features.

The second step, estimating the rotation speed of the ball based on the rotation markers.

For example, in some embodiments of the disclosure, the rotation attribute markers are a plurality of equally spaced markers distributed on the ball differently from one another, for example, the rotation speed of the ball is determined by identifying the frequency of occurrence of the same rotation attribute marker or the time interval of occurrence of adjacent or spaced markers.

It is not difficult to understand that, some embodiments of the present disclosure can evaluate the rotation speed of the ball by analyzing the marker provided on the spherical surface of the golf ball shot in the image, thereby improving the accuracy of this value.

That is, in some embodiments of the present disclosure, obtaining the evaluation results of the hitting process also includes obtaining the rotation and flight trajectory of the ball. In some embodiments of the present disclosure, due to the high frame rate of the cameras to which the image acquisition sensors correspond, the rotation speed of the ball (2000 to 9000 revolutions per minute) may be captured by capturing features or imprints on the ball. Some embodiments of the present disclosure can capture the flight path of a ball after it is hit by the club. Based on these images, the flight trajectory (direction and speed) thereof can be calculated.

The process of obtaining the contact attributes of the club with the ball and the club with the ground is exemplarily explained below. This contact is also a parameter that belongs to some embodiments of the present disclosure for obtaining the evaluation results of the hitting process.

For example, in some embodiments of the present disclosure, the contact attributes comprise an contact order and a contact intensity, wherein obtaining the evaluation results of the hitting process based on multiple types of information, comprising: determine the contact order of the club with the ball as well as the club with the ground, based on the attribute information of the sound information (e.g., where the attribute information exemplarily includes the characteristics of changes in timbre frequency over time), or determining the contact intensity of the club with the ground or the lawn based on the attribute information of the sound information (e.g., where the attribute information includes the strength and/or frequency spectrum information of the sound, such as the sound of grass being high-frequency and the sound of the ground being low-frequency).

Some embodiments of the present disclosure can determine the contact attributes of the club with the ball as well as the club with the ground, by analyzing the sound information collected by the microphone, thereby reducing costs and improving the speed of obtaining contact attribute results.

For example, in some embodiments of the present disclosure, obtaining the evaluation results of the hitting process based on multiple types of information, comprising: determining a time T1 when the ball is in contact with the club by means of the sound information collected by the sound information acquisition sensor (e.g., a microphone); screening out a target image shot at a time closest to the time T1 from the video frame sequences; screening out a second image of the first contact between the club and the ground from the video frame sequences; confirming the relative shooting order of the target image and the second image in the video frame sequences so as to obtain the contact order of the club with the ball and the ground.

Some embodiments of the present disclosure can obtain the contact order of the club with the ball and the club with the ground by means of images and sound information collected by the microphone, thereby improving the accuracy of the evaluation results.

It should be noted that, in some embodiments of the present disclosure, the video frame sequences are obtained by an event sensor. Some embodiments of the present disclosure can obtain enough image frames through a high-speed camera (e.g., an event sensor), which can improve the accuracy of the evaluation results obtained by images and reduce the time for synchronizing different sensors to improve the data processing speed.

Some embodiments of the present disclosure provide a computer-readable storage medium on which a computer program is stored, the program, when executed by a processor, can implement the method described in the above embodiments.

Some embodiments of the present disclosure provide a computer program product comprising a computer program, the computer program, when executed by a processor, implements the method described in the above embodiments.

In the several embodiments provided by the present disclosure, it should be understood that the disclosed apparatus and method may also be implemented in other manners. The apparatus embodiments described above are merely illustrative and, for example, the flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods and computer program products according to multiple embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, a program segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical functions. It should also be noted that, in some alternative implementations, the functions noted in the block may occur in a different order than indicated in the figures. For example, two consecutive blocks can actually be executed in parallel, and sometimes they can also be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagram and/or flowchart, as well as combinations of blocks in the block diagram and/or flowchart, can be implemented using dedicated hardware based systems that perform specified functions or actions, or can be implemented using a combination of dedicated hardware and computer instructions.

In addition, the functional modules in various embodiments of the present disclosure can be integrated together to form an independent part, or each module can exist separately, or two or more modules can be integrated to form an independent part.

The functions may be stored in a computer-readable storage medium if they are implemented in the form of software functional modules and sold or used as separate products. Based on such understanding, the technical solution of the present disclosure or a part thereof which contributes to the prior art and a part of the technical solution may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing a computer device (which may be a personal computer, a server, or a network device) to execute all or part of the steps of the method according to various embodiments of the present disclosure. And the aforementioned storage medium includes: various medias capable of storing program codes such as a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk, and so like.

The above description is only an example of the present disclosure, and is not intended to limit the scope of the present disclosure, and various modifications and changes may be made to the present disclosure by those skilled in the art. Any modification, equivalent replacement, or improvement and so like made within the spirit and principle of the present disclosure should be included in the protection scope of the present disclosure. It should be noted that: like reference numbers and letters refer to like items in the following figures, and thus, once an item is defined in one figure, it need not be further defined or explained in subsequent figures.

The above description is only for the specific embodiments of the present disclosure, but the scope of the present disclosure is not limited thereto, and any person skilled in the art can easily think of the changes or substitutions within the technical scope of the present disclosure, and shall cover the scope of the present disclosure. Therefore, the protection scope of the present disclosure shall be subject to the protection scope of the claims.

It should be noted that, in this document, relational terms such as first and second and so on are used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms “including”, “comprising” or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, a method, an article, or a device that comprises a list of elements not only includes those elements but also includes other elements not expressly listed, or but also includes elements inherent to such process, method, item, or device. Without further limitation, an element defined by the phrase “comprising a/an . . . ” does not exclude the presence of other identical elements in the process, method, article, or device that comprises the elements.