METHOD AND SYSTEM OF DETERMINING POSITION AND ORIENTATION OF A HELMET

Description

FIELD OF THE DISCLOSURE

The present disclosure relates to a system and a method of determining a position and orientation of a helmet and/or of an eyewear while moving on or in a vehicle. Specifically, the present disclosure relates to a method for determining a position and orientation of a helmet and/or of an eyewear while moving on or in a vehicle and to a tracking system for determining a position and orientation of a helmet/eyewear while moving on or in a vehicle. Further, the present disclosure relates to a computer program product comprising a non-transient computer-readable medium having stored thereon computer program code configured to control a processing system of the tracking system.

BACKGROUND OF THE DISCLOSURE

It is a common target to increase the safety on roads and in particular to reduce casualties or injuries for road users. This target has been addressed within the last years in particular by car manufacturers. Different systems like lane-keeping assistants, forward-collision warning or automatic emergency braking systems have been developed to achieve the above mentioned target. The safety of car and truck users has been increased over the last years substantially. Despite this technical development in the car industry, motorcycle safety has not increased to the same extend. For example, anti-block breaking systems (ABS) have only been introduced in the motorcycle industry in the last decade. Other safety systems from the car industry did not even find their way into modern motorcycles yet.

This discrepancy is difficult to comprehend, as non-powered two-wheeler, such as bicycles, powered two-wheelers, such as motorcycles, e-bikes, and e-scooters, pose an elevated risk of injury to their users compared to other forms of mobility, especially compared to cars. For example, in the United States in 2017, casualties in motorcycles were almost 27 times higher per mile driven than the number of casualties in cars.

Several safety aspects for cars are not that easy to implement on a two-wheeler. For example, head up displays in cars project information always at the same portion of a windshield of the car or in front of the windshield or on an additional dedicated display. The user of the car always looks through the same area of the windshield which enables him to read the information presented via the head up display in an easy manner during almost every driving situation which increases the driving safety, because the user of the car with the head up display does not need to lower his gaze. On the contrary, two-wheelers, for example e-scooters or high end motorcycles, do often not even have a windshield. Further, if two wheelers have a windshield, users of two-wheeler do not always look through the windshield while driving. In addition, users of two-wheelers move their head much more often compared to users of cars, which makes it, if the windshield of a two-wheeler comprises a head up display, much harder to present driving information to the user of the two-wheeler in a readable manner during usage of the two-wheeler.

Conventional systems in two-wheelers for displaying driving information like velocity or navigation information use fixedly mounted displays or tachometers which require the user of the two-wheeler to lower his gaze for reading relevant driving information. The lowering of the gaze of the user of the two-wheeler increases the risk and likelihood of accidents.

SUMMARY OF THE DISCLOSURE

It is an object of the present disclosure to provide a system and a method of determining a position and orientation of a helmet or of an eyewear while moving on or in a vehicle. In particular, it is an object of the present disclosure to provide a system and a method and a computer program product of determining a position and orientation of a helmet or of an eyewear while moving on or in a vehicle, which do not have at least some of the disadvantages of the prior art.

According to the present disclosure, these objects are addressed by the features of the independent claims. In addition, further advantageous embodiments follow from the dependent claims and the description.

According to the present disclosure, the above-mentioned objects are particularly achieved by a method of determining position and orientation of a helmet while moving on a vehicle. The method of the present disclosure comprises measuring, by an inertial measurement system of the helmet, inertial data of the helmet during movement of the helmet. The method further comprises measuring, by an inertial measurement system of the vehicle, inertial data of the vehicle during movement of the vehicle. The method of the present disclosure further comprise determining, by a processing system, the position and orientation of the helmet with respect to the vehicle, using the inertial data of the helmet and the inertial data of the vehicle.

In an embodiment, the method further comprises capturing by a camera arranged at the helmet, at least one digital image of the vehicle, and/or capturing by a camera arranged at the vehicle at least one digital image of the helmet. The method further comprises determining, by the processing system the position and orientation of the helmet with respect to the vehicle, further using the digital image(s) of the vehicle and/or of the helmet.

The camera as mentioned above and hereinafter may be any sensing device capable of generating data (e.g. digital image) indicative of distances and space. It thus includes any type of range sensor, for example a radar sensor or LIDAR (Light Detection and Ranging) sensor. Further, the sensing device may include a laser-based scanning system determining distances, for example with time-of-flight (TOF) measurements of a laser beam. By doing so, a two or three dimensional digital image of space can be generated by the data, which is processed by the processing system for determining the position and orientation of the helmet. In the following, by referring to a digital image or simply an image, the image could have been captured by such a camera or range sensor.

In an embodiment, the at least one captured digital image captured by the camera arranged at the helmet comprise at least one specific feature of the vehicle and/or wherein the at least one captured digital image of the camera arranged at the vehicle comprise at least one specific feature of the helmet, and wherein the processing system determines the position and orientation of the helmet with respect to the vehicle, further using the digital image(s) comprising the at least one specific feature of the vehicle and/or of the helmet.

In an embodiment, the method further comprises the processing system to determine an alternative feature of the vehicle and/or of the helmet, by performing the following steps. Capturing, using the camera arranged at the helmet or the vehicle, digital images which comprise the at least one specific feature and additional features of the vehicle and/or of the helmet. Determining, by the processing system the alternative feature from the images. The method according to this embodiment further comprises, using the alternative feature as specific feature for determining the position and orientation of the helmet with respect to the vehicle.

In an embodiment, the alternative feature is determined by a machine learning model or algorithm using the captured digital images comprising a specific feature and/or additional features. In an embodiment, the digital images are processed by the processing system using a neural network as machine learning model, which calculates the preferred alternative feature out of the possible additional features. In another embodiment, the alternative feature is determined, by the processing system, using feature descriptors like SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features) or ORB (Oriented FAST and rotated BRIEF).

In an embodiment, the method further comprises determining the position and orientation of the helmet by using at least one captured digital image captured by the camera, wherein the at least one digital image is processed by a position and orientation estimator on the processing system and/or wherein the position and orientation estimator might be a machine learning model, for instance a neural network.

In an embodiment, the method further comprises predicting the position and orientation of the helmet by using at least one captured digital image captured by the camera, wherein the at least one digital image is processed by a position and orientation estimator on the processing system, wherein the position and orientation estimator might be a machine learning model, for instance a neural network.

Whenever referring to a machine learning model/method above and hereinafter, it is understood that the method comprises any model generating an output for a given input. This also comprises any form of training and using a pre-trained model generating an output for a given input, pre-trained with historical data by optimizing the model—mostly, but not limited to, by minimizing a defined cost-function—according to input and expected output. The input might be inertial data and the output might be a determined position and orientation of an object or a prediction of a future position and orientation of an object. The input might be a digital image depicting at least part of an object and the output might be a determined position and orientation of an object or a prediction of a future position and orientation of an object. The input might also be a combination of inertial data and a digital image, combined by a weighted combination. The input might also be a combination of inertial data and a digital image, combined by the model itself with estimator based data integration. The object might be a helmet, an eyewear, a vehicle, an eye, a special feature or the surrounding environment that is part of a world reference system such as a road. In other cases, for an input of a digital image, the output might also be a determined/defined alternative feature at least partially depicted by the image, by selecting the alternative feature out of a plurality of features. The used machine learning model might comprise, but is not limited to, direct regression, general deep neural networks, convolutional neural networks, transformer based neural networks and recurrent neural networks or a weighted combination of these and further models.

In an embodiment, the at least one specific feature of the vehicle and/or the helmet is a fiducial marker, which is arranged at the vehicle or the helmet. A fiducial marker or fiducial is an object placed in the field of view of an imaging system that appears in the image produced, for use as a point of reference or a measure.

In an embodiment, the method further comprises measuring, by a global positioning system of the vehicle and/or of the helmet, global position data of the vehicle and/or of the helmet with respect to a world reference system. The method according to this embodiment further comprises determining, by the processing system, the position and orientation of the helmet with respect to the world reference system, further using the measured global position data of the vehicle and/or of the helmet. According to an embodiment, the global positioning system uses positioning information obtained via satellites. In another embodiment, the global positioning system uses image based global position data obtained, for example, from a vehicle Controller Area Network Bus (CAN-Bus) unit.

In an embodiment, an initial position of the helmet is determined by capturing, by the camera arranged at the helmet, at least one digital image of the vehicle, and/or by the camera arranged at the vehicle, at least one digital image of the helmet and determining, by the processing system, the initial position of the helmet with respect to the vehicle, using the digital image(s) of the vehicle and/or of the helmet. In an embodiment, the digital images may comprise the specific feature of the helmet or of the vehicle.

In an embodiment, the determination of the position and orientation of the helmet, with respect to the vehicle and/or with respect to the world reference system, is performed by the processing system, using a data integration method. The data is for example the captured digital images, captured data, the measured inertial data of the helmet and/or of the vehicle and of additional data.

In an embodiment, the determination of the position and orientation of the helmet, with respect to the vehicle and/or with respect to the world reference system, is performed by the processing system, using an estimator based data integration. In an embodiment, the estimator based data integration is performed using a linear filter, like Kalman-filter, a nonlinear filter, or a combination of the aforementioned. The estimator Kalman-Filter based data integration results in a smooth, fast, and accurate determination of the position and orientation of the helmet, with respect to the vehicle and/or with respect to the world reference system.

In an embodiment, the method further comprises determining, by the processing system, a display-position of data to be displayed on an augmented reality helmet mounted display, which is arranged at the helmet using the position and orientation of the helmet with respect to the vehicle and/or with respect to the world reference system. The method further comprises displaying, by the processing system, the data to a wearer of the helmet on the augmented reality helmet mounted display, using the determined display position. According to this embodiment, the helmet comprises the augmented reality helmet mounted display for displaying data to the wearer of the helmet, to the user of the vehicle. In an embodiment, the augmented reality display is fixedly arranged at the helmet, in another embodiment, the augmented reality display is detachable from the helmet.

In an embodiment, the method further comprises determining, by an eye tracking system arranged at the helmet, a gaze vector of at least one eye, preferably of both eyes, of the wearer of the helmet and determining, by the processing system, the position of the displayed data and/or the data type in the augmented reality helmet mounted display observed by the wearer, using the determined gaze vector. The eye tracking system is a system for determining a gaze vector of the user of the eye tracking system.

In an embodiment, the processing system comprises a processing system of the helmet, a processing system of the vehicle and/or a processing system of a remote server. In other words, the processing system may be arranged at the helmet, the vehicle or on the remote server or a combination of the aforementioned. Different calculations may be performed on the different processing systems. The processing system may further comprise a processing system of a mobile device, like a smart phone of the user, which is used, for example, in combination with the processing system of the helmet and/or of the vehicle to determine the position and orientation of the helmet or to update a neural network used to calculate a correction signal for the drift of the inertial measurement systems.

Depending on the embodiment, the processing system comprises a system on a chip (SoC), a central processing unit (CPU), and/or other more specific processing units such as a graphical processing unit (GPU), application specific integrated circuits (ASICs), reprogrammable processing units such as field programmable gate arrays (FPGAs), as well as processing units specifically configured to determine the position and orientation of the helmet with respect to the vehicle.

In a further aspect, in addition to a method for determining position and orientation of a helmet, the present disclosure relates to a tracking system for determining position and orientation of a helmet while moving on a vehicle. The tracking system comprising an inertial measurement system arranged at the helmet, an inertial measurement system arranged on the vehicle, and a processing system configured to perform the following steps: measure inertial data of the helmet during movement of the helmet, using the inertial measurement system of the helmet, measure inertial data of the vehicle during movement of the vehicle, using the inertial measurement system of the vehicle; and determine the position and orientation of the helmet with respect to the vehicle, using the inertial data of the helmet and the inertial data of the vehicle.

In an embodiment the tracking system further comprises a camera arranged at the helmet and/or a camera arranged at the vehicle and the processing system is further configure to capture at least one digital image of the vehicle, using the camera arranged at the helmet and/or at least one digital image of the helmet, using the camera arranged at the vehicle, and determine the position and orientation of the helmet with respect to the vehicle further using the digital image(s) of the vehicle and/or of the helmet.

In an embodiment, at least one captured digital image captured by the camera arranged at the helmet comprises at least one specific feature of the vehicle and/or wherein the at least one captured digital image of the camera arranged at the vehicle comprises at least one specific feature of the helmet. The at least one specific feature of the vehicle and/or of the helmet is further used, by the processing system, for the determination of the position and orientation of the helmet with respect to the vehicle.

In an embodiment, the tracking system further comprises a global positioning system, and the processing system is further configured to measure, using the global positioning system, global position data of the vehicle and/or the helmet with respect to a world reference system; and determine the position and orientation of the helmet with respect to the vehicle and/or with respect to the world reference system, further using the measured global position data of the vehicle and/or the helmet.

In an embodiment, the tracking system is configured to use a data integration method for determining the position and orientation of the helmet with respect to the vehicle and/or with respect to the world reference system.

In an embodiment, the tracking system is configured to use an estimator based data integration for determining the position and orientation of the helmet with respect to the vehicle and/or with respect to the world reference system.

In an embodiment, the tracking system further comprises an augmented reality display which is arranged at the helmet, and the processing system is further configured to determine a display-position of data to be displayed in the augmented reality helmet mounted display, using the position and orientation of the helmet with respect to the vehicle and/or with respect to the world reference system; and display, on the augmented reality display, the data to a wearer of the helmet, using the display-position.

In an embodiment, the tracking system further comprises an eye tracking system arranged at the helmet, and the processing system is further configured to determine, using the eye tracking system, a gaze vector of at least one eye, preferably of both eyes, of the wearer of the helmet; and determine the position of the displayed data and/or the data type in the augmented reality display observed by the wearer, using the determined gaze vector.

In addition to the system and method for determining position and orientation of a helmet while moving on a vehicle, the present disclosure relates to a computer program product comprising a non-transitory computer-readable medium having stored thereon computer program code configured to control a processing system of a tracking system such that the tracking system performs the steps according to the method as described above.

According to another aspect of the present disclosure, the above-mentioned objects are particularly achieved by a method of determining position and orientation of eyewear while moving on or in a vehicle. The features and advantages described above and hereinafter with respect to the aspect of determining the position and orientation of the helmet also apply to the aspect of determining position and orientation of eyewear while moving on or in a vehicle as presented hereinafter. The method comprises measuring, by an inertial measurement system of an eyewear, inertial data of the eyewear during movement of the eyewear. The method further comprises measuring, by an inertial measurement system of the vehicle, inertial data of the vehicle during movement of the vehicle. The method of the present disclosure further comprise determining, by a processing system, the position and orientation of the eyewear with respect to the vehicle, using the inertial data of the eyewear and the inertial data of the vehicle.

Eyewear, as understood above and hereinafter, can comprise any device worn by a human user—the wearer of the eyewear—and at least partially covers the line of sight of the user. It comprises devices such as glasses, goggles or monocles.

In an embodiment, the method further comprises capturing by a camera arranged at the eyewear, at least one digital image of the vehicle, and/or capturing by a camera arranged at the vehicle at least one digital image of the eyewear. The method further comprises determining, by the processing system the position and orientation of the eyewear with respect to the vehicle, further using the digital image(s) of the vehicle and/or of the eyewear.

In an embodiment, the method further comprises determining the position and orientation of the eyewear with respect to the vehicle by using at least one captured digital image captured by the camera, wherein the at least one digital image is processed by a position and orientation estimator in the processing system. The position and orientation estimator might be a machine learning model, for instance a neural network.

In an embodiment, the method further comprises predicting the position and orientation of the eyewear with respect to the vehicle by using at least one captured digital image captured by the camera, wherein the at least one digital image is processed by a position and orientation estimator in the processing system. The position and orientation estimator might be a machine learning model, for instance a neural network.

In an embodiment, the at least one captured digital image captured by the camera arranged at the eyewear comprise at least one specific feature of the vehicle and/or wherein the at least one captured digital image of the camera arranged at the vehicle comprise at least one specific feature of the eyewear, and wherein the processing system determines the position and orientation of the eyewear with respect to the vehicle, further using the digital image(s) comprising the at least one specific feature of the vehicle and/or of the eyewear.

In an embodiment, the method further comprises the processing system to determine an alternative feature of the vehicle and/or of the eyewear, by performing the following steps. Capturing, using the camera arranged at the eyewear or the vehicle, digital images which comprise the at least one specific feature and additional features of the vehicle and/or of the eyewear. Determining, by the processing system the alternative feature from the images. The method according to this embodiment further comprises, using the alternative feature as specific feature for determining the position and orientation of the eyewear with respect to the vehicle.

In an embodiment, the alternative feature is determined by a machine learning model or algorithm using the digital images comprising a specific feature and additional features. In an embodiment, the digital images are processed in the processing system by a neural network, which calculates the preferred alternative feature out of the possible additional features. In another embodiment, the alternative feature is determined, by the processing system, using feature descriptors like SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features) or ORB (Oriented FAST and rotated BRIEF).

In an embodiment, the at least one specific feature of the vehicle and/or the eyewear is a fiducial marker, which is arranged at the vehicle or the eyewear. A fiducial marker or fiducial is an object placed in the field of view of an imaging system that appears in the image produced, for use as a point of reference or a measure.

In an embodiment, the method further comprises measuring, by a global positioning system of the vehicle and/or of the eyewear, global position data of the vehicle and/or of the eyewear with respect to a world reference system. The method according to this embodiment further comprises determining, by the processing system, the position and orientation of the eyewear with respect to the world reference system, further using the measured global position data of the vehicle and/or of the eyewear. According to an embodiment, the global positioning system uses positioning information obtained via satellites. In another embodiment, the global positioning system uses global position data obtained, for example, from a vehicle Controller Area Network Bus (CAN-Bus) unit.

In an embodiment, an initial position of the eyewear is determined by capturing, by the camera arranged at the eyewear, at least one digital image of the vehicle, and/or by the camera arranged at the vehicle, at least one digital image of the eyewear and determining, by the processing system, the initial position of the eyewear with respect to the vehicle, using the digital image(s) of the vehicle and/or of the eyewear. In an embodiment, the digital images may comprise the specific feature of the eyewear or of the vehicle.

In an embodiment, the determination of the position and orientation of the eyewear, with respect to the vehicle and/or with respect to the world reference system, is performed by the processing system, using a data integration method. The data is for example the captured digital images, the measured inertial data of the eyewear and/or of the vehicle and/or of additional data.

In an embodiment, the determination of the position and orientation of the eyewear, with respect to the vehicle and/or with respect to the world reference system, is performed by the processing system, using an estimator based data integration. In an embodiment, the estimator based data integration is performed using a linear filter, like Kalman-filter, a nonlinear filter, or a combination of the aforementioned. The estimator Kalman-Filter based data integration results in a smooth, fast, and accurate determination of the position and orientation of the eyewear, with respect to the vehicle and/or with respect to the world reference system.

In an embodiment, the method further comprises determining, by the processing system, a display-position of data to be displayed on an augmented reality eyewear mounted display, which is arranged at the eyewear using the position and orientation of the eyewear with respect to the vehicle and/or with respect to the world reference system.

The method further comprises displaying, by the processing system, the data to a wearer of the eyewear on the augmented reality eyewear mounted display, using the determined display position. According to this embodiment, the eyewear comprises the augmented reality eyewear mounted display for displaying data to the wearer of the eyewear, to the user of the vehicle. In an embodiment, the augmented reality display is fixedly arranged at the eyewear, in another embodiment, the augmented reality display is detachable from the eyewear.

In an embodiment, the method further comprises determining, by an eye tracking system arranged at the eyewear, a gaze vector of at least one eye, preferably of both eyes, of the wearer of the eyewear and determining, by the processing system, the position of the displayed data and/or the data type in the augmented reality eyewear mounted display observed by the wearer, using the determined gaze vector. The eye tracking system is a system for determining a gaze vector of the user of the eye tracking system.

In an embodiment, the processing system comprises a processing system of the eyewear, a processing system of the vehicle and/or a processing system of a remote server. In other words, the processing system may be arranged at the eyewear, the vehicle or on the remote server or a combination of the aforementioned. Different calculations may be performed on the different processing systems. The processing system may further comprise a processing system of a mobile device, like a smart phone of the user, which is used, for example, in combination with the processing system of the eyewear and/or of the vehicle to determine the position and orientation of the eyewear or to update a neural network used to calculate a correction signal for the drift of the inertial measurement systems.

In a further aspect, the present disclosure relates to a tracking system for determining position and orientation of an eyewear while moving on or in a vehicle. The tracking system comprising an inertial measurement system arranged at the eyewear, an inertial measurement system arranged on the vehicle, and a processing system configured to perform the following steps: measure inertial data of the eyewear during movement of the eyewear, using the inertial measurement system of the eyewear, measure inertial data of the vehicle during movement of the vehicle, using the inertial measurement system of the vehicle; and determine the position and orientation of the eyewear with respect to the vehicle, using the inertial data of the eyewear and the inertial data of the vehicle.

In an embodiment the tracking system further comprises a camera arranged at the eyewear and/or a camera arranged at the vehicle and the processing system is further configure to capture at least one digital image of the vehicle, using the camera arranged at the eyewear and/or at least one digital image of the eyewear, using the camera arranged at the vehicle, and determine the position and orientation of the eyewear with respect to the vehicle further using the digital image(s) of the vehicle and/or of the eyewear.

In an embodiment, at least one captured digital image captured by the camera arranged at the eyewear comprises at least one specific feature of the vehicle and/or wherein the at least one captured digital image of the camera arranged at the vehicle comprises at least one specific feature of the eyewear. The at least one specific feature of the vehicle and/or of the eyewear is further used, by the processing system, for the determination of the position and orientation of the eyewear with respect to the vehicle.

In an embodiment, the tracking system further comprises a global positioning system, and the processing system is further configured to measure, using the global positioning system, global position data of the vehicle and/or the eyewear with respect to a world reference system; and determine the position and orientation of the eyewear with respect to the vehicle and/or with respect to the world reference system, further using the measured global position data of the vehicle and/or the eyewear.

In an embodiment, the tracking system is configured to use a data integration method for determining the position and orientation of the eyewear with respect to the vehicle and/or with respect to the world reference system.

In an embodiment, the tracking system is configured to use an estimator based data integration for determining the position and orientation of the eyewear with respect to the vehicle and/or with respect to the world reference system.

In an embodiment, the tracking system further comprises an augmented reality display which is arranged at the eyewear, and the processing system is further configured to determine a display-position of data to be displayed in the augmented reality eyewear mounted display, using the position and orientation of the eyewear with respect to the vehicle and/or with respect to the world reference system; and display, on the augmented reality display, the data to a wearer of the eyewear, using the display-position.

In an embodiment, the camera and the augmented reality display are comprised by one unit.

In an embodiment, the tracking system further comprises an eye tracking system arranged at the eyewear, and the processing system is further configured to determine, using the eye tracking system, a gaze vector of at least one eye, preferably of both eyes, of the wearer of the eyewear; and determine the position of the displayed data and/or the data type in the augmented reality display observed by the wearer, using the determined gaze vector.

In an embodiment, the camera, the augmented reality display and the eye tracking system are comprised by one unit. In an embodiment, the camera, the inertial measurement system, the augmented reality display and the eye tracking system are comprised by one unit.

In addition to the system and method for determining position and orientation of eyewear while moving on a vehicle, the present disclosure relates to a computer program product comprising a non-transitory computer-readable medium having stored thereon computer program code configured to control a processing system of a tracking system such that the tracking system performs the steps according to the method as described above.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will be explained in more detail, by way of example, with reference to the drawings in which:

FIG. 1: shows schematically a vehicle with a user and a tracking system according to a first exemplary embodiment,

FIG. 2: shows schematically a vehicle with a user and a tracking system according to a second exemplary embodiment,

FIG. 3: shows schematically a vehicle with a user and a tracking system according to a third exemplary embodiment,

FIG. 4: shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet according to a first exemplary embodiment,

FIG. 5: shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet according to a second exemplary embodiment,

FIG. 6: shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet according to a third exemplary embodiment,

FIG. 7: shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet according to a fourth exemplary embodiment,

FIG. 8: shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet according to a fifth exemplary embodiment,

FIG. 9: shows a flow diagram illustrating a sequence of steps for displaying data in an augmented reality helmet mounted display of the helmet according to a first exemplary embodiment,

FIG. 10: shows a flow diagram illustrating a sequence of steps for displaying data in an augmented reality helmet mounted display of the helmet according to a second exemplary embodiment,

FIG. 11: shows a flow diagram illustrating a sequence of steps for displaying data in an augmented reality helmet mounted display of the helmet according to a third exemplary embodiment,

FIG. 12: shows a flow diagram illustrating a sequence of steps for performing a determination of an alternative to a specific feature according to an exemplary embodiment,

FIG. 13: shows schematically a vehicle with a user and a tracking system according to a fourth exemplary embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENTS

FIG. 1 shows schematically a vehicle 110 with a user and a tracking system 100 according to a first exemplary embodiment. The vehicle 110 is according to this embodiment a powered two-wheeler. The vehicle 110 can, for example, be a powered or non-powered two-wheeler, a bicycle, a motorcycle, an e-scooter, an e-bike, a car, a jet ski, a snow mobile, ski, snowboard or any different kind of vehicle, where the user wears a helmet 120. The vehicle 110 comprises a global positioning system 111, an inertial measurement system 112, an engine 113, a specific feature 114 and a vehicle processing system 115b, and an alternative feature 114a. The global positioning system 111 is configured to measure global position data of the vehicle 110. The inertial measurement system 112 is configured to measure inertial data of the vehicle 110. The engine 113 is configured to power the vehicle 110. The specific feature 114 is arranged at or near the cockpit area of the vehicle 110 and is configured to be in an image frame of the captured digital images of a camera 122 arranged at a helmet 120. The alternative feature 114a is configured to act as an alternative to the specific feature 114.

The helmet 120 as shown in FIG. 1 comprises a battery 121, the camera 122, an inertial measurement system 123, an augmented reality helmet mounted display 124, a helmet processing system 115a and an eye tracking system 126. The battery 121 is arranged on the helmet 120 to provide electric energy to the different electric systems of the helmet 120. The camera 122 arranged at the helmet 120 is configured to capture images of the specific feature 114 and/or of the alternative feature 114a of the vehicle 110. The inertial measurement system 123 is configured to measure inertial data of the helmet 120. The augmented reality helmet mounted display 124 is configured to display data to the wearer of the helmet 120. The augmented reality helmet mounted display 124 is, for example fixedly or detachable arranged at the helmet 120. The eye tracking system 126 is configured to measure a gaze of the wearer of the helmet 120. In an embodiment, the world reference system 250 is a coordinate system which associates to each position in the world a dedicated coordinate position.

In the embodiment as shown in FIG. 1, the processing system 115 comprises the processing system 115a of the helmet 120 and the processing system 115b of the vehicle 110. The processing system 115 is configured to determine the position and orientation of the helmet 120 with respect to the vehicle 110 and/or with respect to the world reference system 250 and to display data to the wearer of the helmet 120 in the augmented reality helmet mounted display 124. The processing system 115 may further comprise a mobile device, like a smart phone, which in addition of in combination used to perform the calculations to determine the position and orientation of the helmet 120 and/or to display data to the wearer of the helmet 120. In other words, different calculation steps are, for example, performed on different systems of processing system 115, like the processing system 115a, the processing system 115b or the processing system of the mobile device.

Besides the world reference system 250, FIG. 1 further shows schematically a helmet reference system 230 and a vehicle reference system 240. The helmet reference system 230 is centred on the helmet 120 and moves therefore with the helmet 120. Visual data placed statically in the helmet reference system 230 moves statically with the movement of the helmet 120 from the point of view of the user of the helmet 120. The vehicle reference system 240 is centred on the vehicle 110 and moves therefore with the vehicle 110. Visual data placed statically in the vehicle reference system 240 moves statically with the movement of the vehicle 110 from the point of view of the user of the helmet 120. In addition, data placed statically in the world reference system 250 does not move with respect to the vehicle 110 or the helmet 120 from the point of view of the user of the helmet 120. It is of course possible that displayed data moves within his associated reference system 230, 240, 250 or even moves from one reference system 230, 240, 250 to another reference system 230, 240, 250.

The tracking system 100 determines, using the processing system 115, comprising the processing system 115a of the helmet 120 and/or the processing system 115b of the vehicle 110, the position and orientation of the helmet 120 with respect to the vehicle 110 and/or with respect to the world reference system 250. The communication between the processing system 115a of the helmet 110 and the processing system 115b of the vehicle 120 is performed in an embodiment via a wireless communication or a communication module 150 (shown in FIG. 3). In an embodiment, the collected data is sent from the helmet 120 to the processing system 115b, arranged only on the vehicle 110. In an embodiment, the collected data is sent from the vehicle 110 to the processing system 115a, arranged only on the vehicle 110. A combination is also conceivable.

The inertial measurement systems 112, 123 are configured to measure acceleration forces of the helmet 120 and of the vehicle 110. Such forces comprise according to an embodiment, linear acceleration forces and/or rotary acceleration forces. The movement of the user's head of the vehicle 110 results in the same movement of the helmet 120 which the user wears. The measurement of the inertial data of the helmet 120 enables therefore to make conclusions regarding a relative movement of the user's head, with respect to the vehicle 110.

According to an embodiment, different acceleration sensors of the inertial measurement system 112 of the vehicle 110 may be arranged at different positions of the vehicle 110 in order to increase the measurement accuracy. According to an embodiment, different acceleration sensors of the inertial measurement system 123 of the helmet 120 may be arranged at different positions of the helmet 120 in order to increase the measurement accuracy.

The inertial measurement system 116 of the vehicle 110 measures, for example, first an acceleration in a specific direction, a rotary acceleration with respect to a specific axis and a deceleration in a specific direction. With the data from the inertial measurement system 116 of the vehicle 110 it is therefore possible to determine the relative position and orientation of the vehicle 110. Simultaneously, the inertial measurement system 123 of the helmet 120 measures, for example, a first acceleration in specific direction, a rotary acceleration with respect to a specific axis and another acceleration in a further specific direction. With the data from the inertial measurement system 123 of the helmet 120 it is possible to determine the relative position and orientation of the helmet 120. Further, the combination of the measurements of the inertial measurement system 123 of the helmet 120 and of the measurements of the inertial measurement system 116 of the vehicle 110 enables to determine the position and orientation of the helmet 120 with respect to the vehicle 110.

The combination of measured data from the two inertial measurement systems 116, 123, by the processing system 115, enables advantageously to determine the position of the helmet 120 with respect to the vehicle 110 and, in particular to determine the position of the user's head within the helmet 120. With this information, it is possible to display data on a conventional data display system, for example, on a digital display of a two-wheeler, in an advantageous position for reading by the user of the two-wheeler, in dependence of the determined position and orientation of the user's head. The method enables therefore to display safety relevant data in an advantageous manner which increases the safety of the user of the vehicle 110.

In an embodiment, the camera 122 arranged at the helmet 120 is directed towards a cockpit of the vehicle 110 and/or is directed towards the rear of the vehicle 110 while using the vehicle 110.

The specific feature 114 of the vehicle 110 is, according to an embodiment, a part of the cockpit of the vehicle 110, like an edge, a shape, circular instruments, screws, a plurality of screws, or a feature of the cockpit or a combination of the aforementioned. The specific feature 114 of the vehicle 110 is, according to another embodiment, a part of the rear of the vehicle 110, like an edge, a shape of a feature of the rear of the vehicle 110, or a combination of the aforementioned. In an embodiment, the specific feature 114 is a fiducial marker.

FIG. 2 differs from FIG. 1 in that the vehicle 110 further comprises a camera 116 which is arranged fixedly on the rear of the vehicle 110 and which is directed towards the rear of the helmet 120. In another embodiment, the camera 116 may be arranged at or near the cockpit area of the vehicle 110 and may be directed towards the front of the helmet 120. The camera 116 is configured to capture images from the helmet and/or images of a specific feature 114 which is arranged on the helmet 120, for determining the position and orientation of the helmet 120 with respect to the vehicle 110.

In an embodiment, the fiducial marker as specific feature 114, is placed permanently on the vehicle 110 or the helmet 120 and is permanently used for the determination of the position and orientation of the helmet 120 with respect to the vehicle 110. In another embodiment, the fiducial marker is placed temporarily on the vehicle 110 or the helmet 120 and is used only temporarily for the determination of the position and orientation of the helmet 120 with respect to the vehicle 110. During this temporary usage of the fiducial marker an alternative feature 114a to the fiducial marker is determined as described below with reference to FIG. 12. The placement of the fiducial marker on the helmet 120 and/or on the vehicle 110 in the field of vision of the camera(s) 116, 122 is a simple and reliable method to obtain a high accuracy of the position and orientation of the helmet 120 with respect to the vehicle 110 and in addition to determine an advantageously alternative feature 114a to the specific feature 114.

FIG. 3 differs from FIG. 1 or FIG. 2 in that the tracking system 100 further comprises a communication module 150 arranged on the vehicle 120, which is configured to transmit and receive data/information to and from a remote server 160, wherein the remote server 160 comprises its own processing system 115c and forms part of the processing system 115 of the tracking system 100. The remote server 160 is in an embodiment a cloud server. The communication module 150 is, in an embodiment, connected to the processing system 115b of the vehicle 110 via cabling. In another embodiment, the communication module 150 is connected to the processing system 115b of the vehicle 110 and/or to the processing system 115a of the helmet 120 via a wireless connection. A combination is also conceivable. In an embodiment, the communication module 150 is further configured to receive and transmit data to the processing system 115a of the helmet 120. In other words, the communication module 150 is configured to perform the communication between the processing system 115a of the helmet 120, the processing system 115b of the vehicle 110 and/or the processing system 115c of the remote server 160 and/or of a mobile device acting as a further processing system.

In an embodiment, the wireless communication via the communication module 150 takes place using a mobile data network, such as Global System for mobile Communication (GSM), Code Division Multiple Access (CDMA) and Long Term Evolution (LTE) networks, and/or a close range wireless communication interface using a Wi-Fi network, Bluetooth, and/or other wireless network types and standards.

The calculations of the processing systems 115 for the determination of the position and orientation of the helmet 120 with respect to the vehicle 110 and/or with respect to the world reference system 250 are in an embodiment performed on the processing system 115a of the helmet 120, on the processing system 115b of the vehicle 110 and/or on the processing system 115c of the remote server 160. Further, the processing systems 115a, 115b, 115c or a combination of the processing systems 115a, 115b, 115c perform necessary calculations for initializing the tracking system 100, for determining a display position of data to be displayed in the augmented reality helmet mounted display and/or for determining a correction signal for the drift of the inertial measurement systems 112, 123 and for additional necessary steps.

Further types of data may be exchanged between the processing system 115a of the helmet 120, the processing system 115b of the vehicle and the processing system 115c of the remote server and a potential further mobile device such as a smart phone, tablet or laptop, For example, data indicative of the road pathway or data indicative of the local weather conditions might be uploaded to the processing system 115 from a remote device such as a mobile phone and/or a vehicle CAN-Bus unit. Furthermore, a human user might add data to be displayed at a particular point of the road, for instance a virtual brake point or a caution warning right in front of a sharp turn, with a mobile device such as a smart phone, tablet or laptop. Furthermore, the processing system 115 can calculate optimal trajectories based on measurement data and uploaded further types of data. By means of such further types of data, the augmented reality helmet mounted display can display the further types of data to the user.

FIG. 4 shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet 120 with respect to the vehicle 110. In the following paragraphs, described with reference to FIG. 4, a possible sequence of steps is described, performed by the processing system 115, for determining the position and orientation of the helmet 120 with respect to the vehicle 110.

In step S1a, the inertial measurement system 123 of the helmet 120 measures inertial data of the helmet 120. Inertial data of the helmet 120 comprises according to an embodiment linear acceleration data, rotary acceleration data and/or additional data. The inertial measurement system 123 of the helmet 120 is fixedly arranged on the helmet 120 which means that all of the movement of the helmet 120 is measured by the inertial measurement system 123 of the helmet 120. The measured inertial data of the inertial measurement system 123 of the helmet 120, which provides information on the relative movement of the helmet 120, is, in an embodiment, sent to the processing system 115a of the helmet 120.

In step S1b, which is, in an embodiment, performed simultaneously to the step S1a, the inertial measurement system 112 of the vehicle 110 measures inertial data of the vehicle 110. Inertial data of the vehicle 110 comprises, in an embodiment, linear acceleration data, rotary acceleration data and/or additional data. The inertial measurement system 112 of the vehicle 110 is fixedly arranged on the vehicle 110, which means that all of the movement of the vehicle 110 is measured by the inertial measurement system 116 of the vehicle 110. The measured inertial data of the inertial measurement system 116 of the vehicle 110 provides information on the relative movement of the vehicle 110. This information is, in an embodiment, sent to the processing system 115b of the vehicle 110.

In step S2, the processing system 115, comprising, for example, the processing system 115a of the helmet 120 and/or the processing system 115b of the vehicle 110, performs the determination of the position and orientation of the helmet 120 with respect to the vehicle 110 using the inertial data of the helmet 120 and the inertial data of the vehicle 110. The inertial data of the helmet 120 in combination with the inertial data of the vehicle 110 enables to calculate the position and orientation of the helmet 120 with respect to the vehicle 110 while moving on the vehicle 110, in particular relative movement of the helmet 120 with respect to the vehicle 110. With this information it is advantageously possible to display, for example, relevant driving information to the user of the vehicle 110 in an advantageous manner. In detail, the driving information is placed on the display in dependence on the position of the helmet 120 with respect to the vehicle 110. This increases the visibility of the information for the user and therefore the safety for the user of the vehicle 110.

FIG. 5 shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet 120 with respect to the vehicle 110. In the following paragraphs, described with reference to FIG. 5, a possible sequence of steps is described, performed by the processing system 115, for determining the position and orientation of the helmet 120 with respect to the vehicle 110. FIG. 5 shows in addition to the steps S1a, S1b and S2 the step S1c.

In step S1c, which is, in an embodiment, performed simultaneously to the steps S1a and S1b, the camera(s) 116, 122 capture digital images of the vehicle 110 and/or of the helmet 110. The images may also comprise only parts of the vehicle 110 or the helmet 120. An image frame of the camera(s) 116, 122 remains during the movement of the helmet 120 or during the movement of the vehicle 110 static with the helmet 120 or with the vehicle 110, because the camera(s) 116, 122 are fixedly arranged with the helmet 120 or with the vehicle 110. The position of the vehicle 110 and/or of the helmet 120 within the image frame of the camera(s) 116, 122 change due to movement of the helmet 120 or due to movement of the vehicle 110. It is therefore possible to determine the position of the helmet 120 with respect to the vehicle 110 using these images. For example, the captured images are processed by an end-to-end position and orientation estimator, like a neural network which regresses the position and orientation of the helmet 120 directly. The captured images are processed, for example directly by the neural network and the position and orientation of the helmet 120 with respect to the vehicle 110 is the output of the neural network. The calculation is performed using the processing system 115 (115a, 115b, and/or 115c).

Training data for the neural network comprises, for example as a ground truth, images of the vehicle 110 and/or the helmet 120 comprising position and orientation data of the helmet 120 with respect to the vehicle 110. The training of the neural network is performed on the processing system 115, preferably on the processing system 115c of the remote server. The updated neural network or an update for the neural network is, in an embodiment, sent back to the processing system 115a, 115b of the vehicle 110 and/or of the helmet 120 via the communication module 150. Running/operating of the neural network is performed on the processing system 115, preferably on the processing system 115a of the helmet 120 and/or on the processing system 115b of the vehicle 110 in order to reduce latencies.

In the step S2 of FIG. 5, the processing system 115, performs the determination of the position and orientation of the helmet 120 with respect to the vehicle 110 and with respect to the world reference system 250, further using the determined position and orientation of the helmet 120 with respect to the vehicle 110 out of the captured images obtained in step S1c. Combining the data from the inertial measurement systems 112, 123 of the helmet 120 and of the vehicle 110 with the data retrieved, by the processing system 115, from the at least one digital image increases the accuracy of the determination of the position and orientation of the helmet 120 with respect to the vehicle 110. The camera(s) 116, 122 enable in particular to increase the accuracy of the determination of movements of the helmet 120, because the measurements of the inertial measurement systems 112, 123 measure the position and orientation by integration of their measured data, thus over time, the integration causes errors to accumulate, known as drift. In other words, the drift of the inertial measurement systems 112, 123 is an accumulation of small errors in the measurements of the inertial measurement systems 112, 123 which gradually cause the integrated inertial data to become more and more inaccurate. The camera(s) 116, 122 provide orientation and position information directly, meaning there is no integration and drift. The camera(s) 116, 122 can therefore be used as drift correction for the measurements of the inertial measurement systems 112, 123. Therefore, the camera(s) 116, 122 enable, in an advantageous manner, to increase the accuracy of the determination of the orientation and position of the helmet 120 with respect to the vehicle 110, which enables to increase the overall safety for the user of the vehicle 110. The camera(s) 116, 122 enable to obtain a “correction signal” for the correction of drift of the inertial measurement systems 112, 123. With the correction signal it is possible to correct the measured inertial data, which increases the accuracy of the determination of the position and orientation of the helmet 120 with respect to the vehicle 110.

FIG. 6 shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet 120 with respect to the vehicle 110. In the following paragraphs, described with reference to FIG. 6, a possible sequence of steps is described, performed by the processing system 115, for determining the position and orientation of the helmet 120 with respect to the vehicle 110. FIG. 6 shows in addition to the steps S1a, S1b, S2 and S1c the step S2a. In the step S2a, the processing system 115, performs the determination of the position of the specific feature 114 using the captured images by the camera(s) 116, 122. The captured images of step S1c comprise the specific feature 114 of the helmet 120 and/or of the vehicle 110. The determination of the position of the specific feature 114 is, for example, performed by determining which pixel of the image(s) show the specific feature 114 and by determining the position of these pixel in a digital image coordinate system it is possible to determine the position and orientation of the helmet 120 with respect to the vehicle 110. The specific feature 114 is, for example, one or a plurality of screws of the cockpit of the vehicle 110. If, for example, the distance between two screws is known, it is possible to calculate the position of the helmet 120 with respect to the vehicle 110 by determining the position of the pixels showing the screws in the digital image. This calculation requires only geometrical algebra and is preferably performed on the processing system 115a of the helmet 120 and/or the processing system 115b of the vehicle 110.

In step S2, the determination of the position and orientation of the helmet 120 with respect to the vehicle 110 is performed by the processing system 115 using the determined position of the specific feature 114 of step S2a. According to this embodiment, it is in particular simple to calculate the correction signal for the drift of the inertial measurement systems 112, 123.

FIG. 7 shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet 120 with respect to the vehicle 110. In the following paragraphs, described with reference to FIG. 7, a possible sequence of steps is described, performed by the processing system 115, for determining the position and orientation of the helmet 120 with respect to the vehicle 110. FIG. 7 shows in addition to the steps S1a, S1b and S2 the steps S1c, S2b and S2c.

The step S1c is described above with reference to FIG. 6.

In step S2b, an alternative feature 114a is determined using the digital images captured in step S1c. The digital images comprise at least one specific feature 114 of the helmet 120 and/or of the vehicle 110 and additional features of the vehicle 110 and/or of the helmet 120 for example, circular instruments, screws or edges next to the specific feature 114. The alternative feature 114a is, for example, learned/determined by a neural network of the processing system 115 (115a, 115b, 115c). The digital images are processed by the neural network which calculates the preferred alternative feature 114a out of the possible additional features. In the training process, the neural network requires a ground truth to be able to learn the alternative features 114a. The ground truth for the training process is, in an embodiment, images comprising the specific feature 114 and/or associated inertial measurement data of the inertial measurement systems 112, 123 and/or associated position information of the camera.

In the step S2c, the alternative feature 114a is used instead of the specific feature 114. The specific feature 114 may now be removed from the vehicle 110 or the helmet 120.

In step S2, the determination of the position and orientation of the helmet 120 with respect to the vehicle 110 is performed by the processing system 115 using the alternative feature 114a. According to this embodiment, it is possible to provide an alternative or a combination for the calculation of the correction signal for the drift of the inertial measurement systems 112, 123 using the alternative feature 114a.

FIG. 8 shows a flow diagram illustrating a sequence of steps for performing a determination of a position and orientation of the helmet 120 with respect to the vehicle 110. In the following paragraphs, described with reference to FIG. 8, a possible sequence of steps is described, performed by the processing system 115, for determining the position and orientation of the helmet 120 with respect to the vehicle 110. FIG. 8 shows in addition to the steps S1a, S1b and S2 the step S1d.

In step S1d, which is according to an embodiment performed simultaneously to the step S1a, S1b and/or S1c, the global positioning system 111 of the vehicle 110 measures global position data of the vehicle 110. In a further embodiment, a global positioning system of the helmet 120 measures global position data of the helmet 120. In an embodiment, the global positioning system 111 uses positioning information obtained via satellites. In another embodiment, the global positioning system 111 uses image based information obtained, for example, from a vehicle CAN-Bus unit. The measured position data of the vehicle 110 and/or of the helmet 120 enables to determine the position of the vehicle 110 and/or of the helmet 120 with respect to the world reference system 250.

In step S2, the processing system 115, performs the determination of the position and orientation of the helmet 120 with respect to the vehicle 110 and with respect to the world reference system 250 further using the measures global position data from the global positioning system 111 of the vehicle 110 and/or of the helmet 120. The combination of the inertial data of the helmet 120, the inertial data of the vehicle 110 and the global position data enables to calculate the position and orientation of the helmet 120 with respect to the vehicle 110 while moving on the vehicle 110 and with respect to the world reference system 250 while the vehicle 110 with the helmet 120 moves in the world reference system 250. This enables to display, for example, navigation data to the user of the vehicle 110 in an advantageous manner which increases the safety of the usage of the vehicle 110 and increases the user experience of the vehicle 110.

With the global position data of the vehicle 110, in combination with a digital image of the camera(s) 122, 116 and/or in combination with the inertial data of the helmet 120 and the vehicle 110, it is not only possible to increase the accuracy of the determination of the position and orientation of the helmet 120 with respect to the vehicle 110, but also to determine the position of the helmet 120 and the vehicle 110 with respect to the world reference system 250. This enables to display navigation data to the user of the vehicle 110 in an advantageous manner, which increases the safety of the usage of the vehicle 110 and increases the usage experience of the vehicle 110.

FIG. 9 shows a flow diagram illustrating a sequence of steps for displaying data in an augmented reality helmet mounted display 124 of the helmet 120. In the following paragraphs, described with reference to FIG. 9, a possible sequence of steps is described, performed by the processing system 115, for displaying data in the augmented reality helmet mounted display 124. For the determination of the position and orientation of the helmet 120 (step S2), a combination of the steps S1a, S1b, S1d, S1c, S2a, S2b and/or S2c, as presented above, may be used (steps S2a, S2b, S2c are not shown in FIG. 9).

In step S3, the processing system 115, determines a display-position of data to be displayed on the augmented reality helmet mounted display 124 which is arranged at the helmet 120 using the position and orientation of the helmet 120 with respect to the vehicle 110 and/or with respect to the world reference system 250 determined in step S2. In an embodiment, the augmented reality helmet mounted display 124 is implemented in a visor of the helmet 120 or forms the visor of the helmet 120. The data to be displayed comprises, in an embodiment, navigation information, driving information like velocity, engaged gear etc. road information like ideal line, expected obstacles, or warning information like too high velocity for the next road segment. The display position is the position on the augmented reality helmet mounted display 124 for the data to be displayed. For example, conventional head up displays in the automotive industry project data in the lower part of the windshield of a car. The display position of the data to be displayed in the augmented reality helmet mounted display 124 may vary across the entire visor or field of vision of the user of the vehicle 110.

In step S4, the processing system 115, displays the data to the wearer of the helmet 120 on the augmented reality helmet mounted display 124, using the determined display position of step S3. The position of data to be displayed depends on the determined position and orientation of the helmet 120. In other words, different data may be displayed on the augmented reality helmet mounted display 124 in dependence of the position and orientation of the helmet 120. For example, data can be placed in the helmet reference system 230, this data will appear to the user to be moving exactly like the helmet 120 and remains static in the augmented reality helmet mounted display 124. Data placed in the helmet reference system 230 at 20 cm in front of the user's head will always be visible at 20 cm in front of the user's head, even if the helmet 120 changes orientation or position due to movement of the user. In addition, data can be placed in the vehicle reference system 240 which is centered on the vehicle 110. Data centered on the vehicle reference system 240 and displayed in the augmented reality helmet mounted display 124 remains static with respect to the vehicle 110. If the helmet 120 changes the position or orientation, the display position in the augmented reality display of this data will also change. For example, if the vehicle speed is associated to the vehicle reference system 240 and is placed always three meter in front of the vehicle 110, then it will always remain three meter in front of the vehicle 110 and can also disappear out of the user's field of view given certain movement of the helmet 120. In addition, data can be placed in the world reference system 250. Data centered in the world reference system 250 may remain static with respect to the world. If the helmet 120 and/or the vehicle 110 change(s) position and/or orientation, the display position in the augmented reality helmet mounted display 124 of this data will also change. For example, if navigation information, like an arrow, is placed in the world reference system 250, then it will always remain in the specific position in the world reference system 250 and can also disappear out of the user's field of view given certain movement of the head/helmet 120.

The knowledge of the position and orientation of the helmet 120, with respect to the vehicle 110 and/or with respect to the world reference system 250, enable to display the different data in the augmented reality helmet mounted display 124 in dependence on the associated reference system 230, 240, 250 and in dependence on the position and orientation of the helmet 120, with respect to the vehicle 110 and/or with respect to the world reference system 250.

In an embodiment, the determination of the position and orientation of the helmet 120 is continuously updated with data from the inertial measurement system 123 of the helmet 120, with data from the inertial measurement system 112 of the vehicle 110, with digital images of the at least one specific feature 114 of the helmet 120 and/or of the vehicle 110, and/or with global position data of the helmet and/or of the vehicle. The continuous updating enables to determine the position and orientation of the helmet 120 over time, which enables to change/adapt the position of displayed data to the user of the vehicle 110, in dependence on the change of the position and orientation of the helmet 120, with respect to the vehicle 110 and/or with respect to the world reference system 250. This increases the safety of the user of the vehicle 110.

FIG. 10 shows a flow diagram illustrating a sequence of steps for displaying data in an augmented reality helmet mounted display 124 of the helmet 120. In the following paragraphs, described with reference to FIG. 10, a possible sequence of steps is described, performed by the processing system 115, for displaying data in the augmented reality helmet mounted display 124. For the determination of the position and orientation of the helmet 120 (step S2), a combination of the steps S1a, S1b, S1d, S1c, S2a, S2b and/or S2c as presented above may be used (steps S2a, S2b, S2c are not shown in FIG. 10).

In step S3a, the processing system, determines a gaze vector of at least one eye, preferably of both eyes of the wearer of the helmet 120, using the eye tracking system 126. The eye tracking system 126 is a system for determining the gaze vector of the user of the eye tracking system 126. The gaze vector of the wearer of the helmet 120 determines the viewing direction of the wearer. Using both eyes for the determination of the gaze vector increases its accuracy.

In step S3, the processing system 115 determines the position of the displayed data in the augmented reality helmet mounted display 124 observed by the wearer, using the determined gaze vector. In other words, the display position in the augmented reality display 124 is further determined in dependence on the gaze vector.

In step S4, the processing system 115, displays the data to the wearer of the helmet 120 on the augmented reality helmet mounted display 124, further using the gaze vector of step S3a. For example, the displayed data, like vehicle velocity, moves in the augmented reality helmet mounted display 124 with the gaze movement of the user of the vehicle 110.

Further, the processing system 115 can determine the data type displayed in the augmented reality display 124 observed by the wearer of the helmet 120 using the determined gaze vector. For example, if the wearer of the helmet 120 does not look on the road, determined by the gaze vector, for a predetermined timespan, the processing system 115 may alert the user of the vehicle 110 by showing in the augmented reality display alerting information to the wearer of the helmet 120. With the eye tracking system 126, it is possible to determine the position of the displayed data and/or the data-type in the augmented reality helmet mounted display 124, which helps to show necessary information for a safe usage of the vehicle 110. In addition, if the wearer of the helmet 120 does not look on an obstacle on the road, the processing system 115 may alert the user of the vehicle 110 by showing alerts on the augmented reality helmet mounted display 124.

FIG. 11 shows a flow diagram illustrating a sequence of steps for displaying data in an augmented reality helmet mounted display 124 of the helmet 120. In this embodiment, determining the position and orientation of the helmet 120 is performed by the processing system 115 (115a, 115b and/or 115c) using a filter based on Kalman equations data integration as estimator. Input for the processing system 115 is inertial data of the inertial measurement system 112 of the vehicle 110, inertial data of the inertial measurement system 123 of the helmet 120, digital images captured by the camera 122 of the helmet 120 and global position data of the global positioning system 111. In the following paragraphs, described with reference to FIG. 11, a possible sequence of steps is described performed by the processing system 115 for determining the position and orientation of the helmet 120 and for displaying data in the augmented reality helmet mounted display 124.

In step S11, the processing system 115, determines the position of the helmet camera 122 with respect to the vehicle 110 or with respect to the vehicle reference system 240, using at least one captured image by the helmet camera 122. In another embodiment, the processing system 115, determines the position of the helmet 120 with respect to the vehicle 110, using at least one captured image by the vehicle camera 116. A combination of both is also conceivable.

In step S12, the processing system 115, determines the position and orientation of the vehicle 110 with respect to the world reference system 250, using the inertial data of the vehicle 110 measured by the inertial measurement system 112 of the vehicle 120 and the global position data measured by the global positioning system 111.

In step S13, the processing system 115, initializes the tracking system 100, by determining an initial position of the helmet 120 with respect to the vehicle 110 and to the world reference system 250. The initial position of the helmet 120 is determined using the inertial measurement data of the inertial measurement system 123 of the helmet 120, the inertial measurement data of the inertial measurement system 112 of the vehicle 110, the in step S11 determined position of the helmet camera 122 and the in step S12 determined position of the vehicle 110 with respect to the world reference system 250.

The initial position of the helmet 120 is, for example, the position of the helmet 120 on the vehicle 110 before the vehicle 110 is moved by the user. In another embodiment, the initial position of the helmet 120 is the first position of the helmet 120, when the helmet 120 enters the field of vision of the camera 116 arranged at the vehicle 110, or the first position of the helmet 120, when the specific feature 114 of the vehicle 110 enters the field of vision of the camera 122 arranged at the helmet 120. Starting from the initial position of the helmet 120, with respect to the vehicle 110 and/or with respect to the world reference system 250, the new position of the helmet 120, with respect to the vehicle 120 and/or with respect to the world reference system 250, due to continuous movement of the vehicle 110 and the helmet 120, can be updated by the data from the inertial measurement systems 112, 123, the images captured by the camera(s) 116, 122, and global position data. The determination of the initial position enables therefore to increase accuracy and velocity of the determination of the position and orientation of the helmet 120 with respect to the vehicle 110 and/or the world reference system 250.

In step S14a, the processing system 115, predicts a future position of the helmet 120 using Kalman-filter inspired equations. The input data for this step is inertial measurement data of the vehicle 110 measured by the inertial measurement system 112 of the vehicle 110, inertial measurement data of the helmet 120 measured by the inertial measurement system 123 of the helmet 120 and the in step S13 determined initial position of the helmet 120. Further input for this step is the in step S14b updated position of the helmet 120.

In step S14b, the processing system 115, updates the position of the helmet 120 using the Kalman filter inspired equations. The input data for this step is the in step S11 determined position of the helmet camera 122, the in step S12 determined position of the vehicle 110 in the world reference system 250 and the in step S14a predicted position of the helmet 120.

The steps S14a and S14b are performed iteratively. The predicted position of the position of the helmet 120 is used as input for updating the position of the helmet 120 with every new time step. Further, the updated position of the helmet 120 is used as input for predicting the position of the helmet 120 with every new time step.

In step S15, the processing system 115, determines the position of the helmet 120 with respect to the vehicle 110 and with respect to the world reference system 250.

In step S16, the processing system 115, displays data in the augmented reality helmet mounted display 124 to the wearer of the helmet 120, which is the driver of the vehicle 110, based on the determined position of the helmet 120 in step S15.

FIG. 12 shows a flow diagram illustrating a sequence of steps for performing a determination of the alternative 114a to the specific feature 114 according to an embodiment. In the following paragraphs, described with reference to FIG. 12, a possible sequence of steps is described for determining an alternative 114a to the specific feature 114.

In step S20, a specific feature 114 is arranged on the vehicle 110. The specific feature 114 is in an embodiment a fiducial marker. The specific feature 114 is, for example arranged during the installation of the tracking system 100 on the vehicle 110 and the helmet 120.

In step S21, the camera 122 of the helmet 120, captures digital images, which comprise the in step S20 arranged specific feature 114 and the additional features of the vehicle 110. The digital image shows for example the cockpit of the vehicle 110 with the dedicated specific feature 114. Further, the digital image shows additional features of the cockpit, for example, circular instruments, one or a plurality of screws or edges next to the specific feature 114.

In step S22, the processing system 115a, 115b, send the captured images to the processing system 115c of the remote server 160 for determining the alternative 114a feature using the image. The communication between the vehicle 110 and the remote server 160 is for example performed via the communication module 150.

In step S23, the processing system 115c of the remote server 160, determines the alternative feature 114a using the at least one image. Determining the alternative feature 114a of the vehicle 110 is in an embodiment done, on the remote server 160, using a neural network. The images are processed by the neural network and the alternative feature 114a is determined thereby. In another embodiment the alternative feature 114a is determined using feature descriptors like SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features) or ORB (Oriented FAST and rotated BRIEF) on the remote server 160. Other embodiments use other machine learning approaches.

In step S24, the processing system 115a of the helmet 120 and or the processing system 115b of the vehicle 110, receives the determined alternative feature 114a from the remote server 160. In other words, the information on the alternative feature 114a, determined by the neural network or another feature descriptor is sent from the remote sever 160 to the processing system 115a of the helmet 120 or the processing system 115b of the vehicle 110, for example, via the communication module 150.

In step S25, the processing system 115, uses the alternative feature 114a as specific feature 114 for determining the position and orientation of the helmet 120 with respect to the vehicle 110 and/or with respect to the world reference system 250. In an embodiment, the specific feature 114 is removed from the vehicle 110 after the alternative feature 114a is determined and used by the processing system 115.

In an embodiment, the specific feature 114 is arranged on the helmet 120 and the images are captured by the camera 116 of the vehicle 110. A combination is also conceivable. In an embodiment, the specific feature 114 is a fiducial marker.

The user of the vehicle 110 places a dedicated specific feature 114 like a marker on his vehicle during the installation of the system 100. Afterward he uses his vehicle 110 for a specific time period. The time period is according to an embodiment between five and hundred using hours, preferably between five and twenty using hours. During this specific time period the digital images, which comprise the specific features 114 and the additional features are sent to the remote server 160. The processing system 115c of the remote server 160 processes the images via the neural network and determines the alternative feature 114a. The information on the alternative feature 114a is afterwards sent back to the processing system 115 of the vehicle 110 and the processing system 115 of the vehicle 110 uses the alternative feature 114a instead of the former specific feature 114. The former specific feature 114 may be removed by the user from the vehicle 110, because it is no longer needed.

It should be noted that, in the description, the sequence of the steps has been presented in a specific order, one skilled in the art will understand, however, that the order of at least some of the steps could be altered, without deviating from the scope of the disclosure.

FIG. 13 shows schematically a vehicle 110 with a user and a tracking system 100 according to a fourth exemplary embodiment. It differs by the embodiments in FIGS. 1 to 3 in that an eyewear 120′ replaces the helmet 120 and the eyewear reference system 230′ replaces the helmet reference system 230. It is to be understood that all the combinations shown for a tracking system 100 comprising a helmet 120 as presented and described with respect to the FIGS. 1 to 3 equally apply to a tracking system 100 comprising eyewear 120′ instead of the helmet 120. Further, it is to be understood that all the flow diagrams of methods for determining a position and orientation of the helmet 120 presented and described with respect to the FIGS. 4 to 12 equally apply to a method for determining a position and orientation of the eyewear 120′ instead of the helmet 120. Further, it is to be understood that all the advantages presented and described with respect to the helmet 120 equally apply to the eyewear 120′.

The vehicle 110 is according to this embodiment a powered two-wheeler or in another embodiment a powered four-wheeler like a car, other vehicles like trucks, bicycles, etc. are also conceivable. The vehicle 110 comprises a global positioning system 111, an inertial measurement system 112, an engine 113, a specific feature 114 and a vehicle processing system 115b, and an alternative feature 114a. The global positioning system 111 is configured to measure global position data of the vehicle 110. The inertial measurement system 112 is configured to measure inertial data of the vehicle 110. The engine 113 is configured to power the vehicle 110. The specific feature 114 is arranged at or near the cockpit area of the vehicle 110 and is configured to be in an image frame of the captured digital images of a camera 122 arranged at an eyewear 120′. The alternative feature 114a is configured to act as an alternative to the specific feature 114.

The eyewear 120′ as shown in FIG. 13 comprises a battery 121, the camera 122, an inertial measurement system 123, an augmented reality eyewear mounted display 124, an eyewear processing system 115a and an eye tracking system 126. The battery 121 is arranged on the eyewear 120′ to provide electric energy to the different electric systems of the eyewear 120′. The camera 122 arranged at the eyewear 120′ is configured to capture images of the specific feature 114 and/or of the alternative feature 114a of the vehicle 110. The inertial measurement system 123 is configured to measure inertial data of the eyewear 120′. The augmented reality eyewear mounted display 124 is configured to display data to the wearer of the eyewear 120′. The augmented reality eyewear mounted display 124 is, for example fixedly or detachable arranged at the eyewear 120′. The eye tracking system 126 is for example configured to measure a gaze vector of the wearer of the eyewear 120′.

In an embodiment, the world reference system 250 is a coordinate system, which associates to each position in the world a dedicated coordinate position. Besides the world reference system 250, FIG. 13 further shows schematically an eyewear reference system 230′ and a vehicle reference system 240.

In an embodiment, the camera 122, the inertial measurement system 123, the augmented reality display 124, the eye tracking system 126 and the processing system 115a of the eyewear 120′ being a part of the processing system 115 (see FIG. 13) are comprised by one eyewear unit, for example specific glasses.