System and Method of a Digital Weapon Sight with Calculated Aimpoint Correction

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority under 35 U.S.C. § 119 (e) to U.S. Provisional Patent Application No. 63/618,317, filed Jan. 6, 2024, the disclosure of this prior application is considered part of this application and is hereby incorporated by reference in its entirety.

TECHNICAL FIELD OF THE INVENTION

The present disclosure relates to the technical field of aiming of telescopic weapon sights, and more particularly to the automatic calibration of digital aiming devices for firearms and other weapons systems.

BACKGROUND OF THE INVENTION

Currently, there exist a wide range of methods to provide a shooter with better aim and higher precision. A common solution is affixing a telescopic sight to a firearm to magnify the target and improve accuracy.

When using a telescopic weapon sight (“sights”/“telescopes”/“scopes”), the first thing a marksman does is a process referred to as “zeroing” the weapon sight to the weapon-what one may call an “aimpoint correction” step. Typically, a weapon is fitted with a sight via a mounting system such that the sight is rigidly attached to the weapon and positioned in-line with the weapon barrel. Adjustments are made to the sighting device to manipulate the position of lenses within the sight. The marksman then takes a series of careful shots at a target at a certain distance, then re-centers on the target, then manipulates knobs or buttons to move a reticle within the weapon sight's eyepiece display to more precisely indicate where a projectile is mean to impact. The marksman then takes another series of shots to confirm the manual manipulation.

In the case of zeroing a digital weapon sight is fundamentally the same. So it is that a fast, effective, hands-free method of zeroing a digital weapon sight is of great utility.

Digital weapon sights are increasingly common in military and consumer markets. They have been described in many disclosures, and have been described to use of a variety of sensors (accelerometers, gyroscopes, magnetometers, barometric sensors, microphones) including those that indicate when shots are fired and including features of storing various data. Despite the assortment of sensors, processors, memories often found in a digital weapon sight, there exists no system for digitally aligning the weapon sight and weapon in a fast, hands-free manner and can be deployed in small processors or FPGA's.

The utility of enhancing the reliability of the aiming of a weapon is well demonstrated by numerous efforts:

U.S. Pat. No. 12,026,912 B2 (Li) discloses a system where images are saved and analyzed to determine an aiming point. The utility of the claims is described as “relates generally to location recognition.” And though the algorithms for template matching may be useful, the keypoint extraction techniques and image registration using a SURF method are too complex to be employed by the small processors that are used digital weapon sights.

U.S. Pat. No. 10,378,857 (Li) uses a method to correct aimpoint by means of affine transformations and perspective corrections, and requires a certain paper target be employed. Given the complexity of calculations and required target, a simpler manner should be employed for smaller, low power devices in a tactical setting.

U.S. Pat. No. 8,651,381 (Rudich) specifies a device that “includes a video camera capable of capturing and tracking the path of a projectile,” specifying a system high-speed and high-definition enough “to record and save captured images of the flight of the projectile.” This implies power, memory and processing requirements that are not available in nearly any weapon sight system on the market even a decade after the disclosure was made.

None of the above inventions and patents, or others existing, either singly or in combination, is seen to describe the claimed invention herein.

SUMMARY OF THE INVENTION

This disclosure claims a digital weapon sight equipped with an aimpoint correction system wherein imagery data is stored to memory, both before and after a shot, that data is subsequently analyzed, and an on-screen reticle is automatically adjusted.

The digital weapon sight is an assembly of electronic components including an image sensor, a computational unit, herein referred to as “processor”; and a sensor used to detect that a shot has been fired; and digital memory. A graphical display of imagery data from the image sensor and overlay symbology is most often in common assembly with the other elements, and affixed to the weapon, however it may be remote, as in a helmet-mounted display system, or “heads-up display,” or “HUD.”

Specific to claims herein, the processor in the digital weapon sight is loaded with executable code or language-defined hardware that conducts a process whereby imagery data is captured and stored into memory immediately preceding a shot, and then again shortly after a shot. These two images are then digitally analyzed to determine where in the first image the shot landed. A reticle, digitally overlayed atop the imagery and nominally centered at the center of the imagery, is then shifted by an offset to indicate where a subsequent shot will land.

In an embodiment, the user enters a mode of operation where imagery data is continuously stored in addressable memory, then the user aims and takes a shot, and on-board sensors sense that ballistic event, triggering the processor to isolate a block of imagery data—the “pre-shot” image, captured at a time preceding the shot—and store it to memory. The user then aims a second time, this time aiming at the point of impact of the first shot. If the user fires another shot or if the user simply holds the weapon steady, on-board sensors trigger a second event, causing the processor to isolate and store a second image—the “post-shot” image. The pre-shot and post-shot imagery data are then passed to an algorithm in the processor in which the second image is registered to the first image. Several algorithms are detailed, and in embodiments the process is validated by comparing the outputs of a plurality of algorithms. The image registration determines a translational offset between the two images, and this offset indicates the place where the post-shot imagery data is found in the pre-shot imagery data, and is how much the on-screen reticle symbology should be shifted in order to correct for the misalignment of the reticle to the point of impact, so that the next bullet fired will land at the center of the reticle crosshairs. The process corrects the subsequent targeting of the weapon and weapon sight with minimal manual manipulation.

An embodiment is also detailed that includes a step where the point of impact is mathematically determined. The two sets of imagery data are the passed to an image registration algorithm to determine a translational offset, and then each are windowed based on this translational offset such that a pixel-level difference can be conducted. After some filtering, the maximal value of the difference between the images may be ascribed to be the calculated point of impact, and the location of this in the pre-shot image gives the translational offset to adjust the on-screen reticle. This embodiment includes a fallback: if analysis of the output of the registration/subtraction does not pass threshold criteria, the process reverts to the aforementioned process, and the user is prompted to aim at the point of impact, and post-shot imagery data is collected.

It is important to note that the method is designed to be used in nearly any situation where the user can determine the point of impact. It is also important to note that the user only needs to take careful aim single time, that is, at the point of impact. Further, the method and systems may be used in a production environment to enhance the precision of the manufacturing process. In this case, the alignment method may be conducted with button presses or an interface via a computer, rather than shooting a weapon.

In terms of algorithms, several are detailed: a frequency-domain method, an annular meth, and a spatial-domain method. The frequency-domain method uses the Fourier Transform to quickly calculate a cross-correlation, as multiplication of Fourier Transforms is equal to convolution of spatial domain data. The annular method pulls out a circular shape of pixels from the captured images and conducts either spatial or frequency-domain correlation. The spatial-domain method employs a convolution in the spatial domain, either finding the difference between the values of two pixels or multiplying two mean-adjusted pixel values for each step in the convolution.

With the use these algorithms, the goal of digitally aligning the weapon sight to the weapon (a “zero,” to use the nomenclature in the background section of this document) is able to be achieved, quickly, reliably, and with minimal user intervention.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an image of a digital weapon sight mounted to a weapon (a rifle), and a notional exemplary image of the correction of a digitally overlayed reticle.

FIG. 2 is a block diagram of a digital weapon sight and its functional components.

FIG. 3, comprising FIGS. 3A, 3B, 3C, and 3D, is a set of exemplary images from the user's point of view during the conduct of the aimpoint correction method according to a preferred embodiment. On the same sheet FIG. 3A, is an exemplary image showing the user's aiming view; FIG. 3B showing the view after a shot; FIG. 3C showing the view aiming at the point of impact; FIG. 3D showing the view after the reticle is shifted.

FIG. 4, comprising FIGS. 4A, 4B, 4C, 4D, and 4E, is a set of exemplary images captured, windowed, and exemplary image of image subtraction, during steps of an embodiment of the aimpoint correction method where the point of impact is determined algorithmically. FIG. 4A is an exemplary image showing the outline of a cropped area of imagery data; FIG. 4B showing an outline of a second cropped area of imagery data; FIG. 4C showing a cropped set of imagery data collected before a shot; FIG. 4D showing a cropped set of imagery data collected after a shot; and FIG. 4E showing the absolute value of the difference of two exemplary sets of cropped imagery data.

FIG. 5 is a flowchart of the preferred embodiment of the aimpoint correction method.

FIG. 6 is a flowchart of an embodiment of the aimpoint correction method where the point of impact is determined algorithmically.

FIG. 7 is a flowchart of the frequency-domain method of image registration process.

FIG. 8 is a flowchart of the annular method of image registration process.

FIG. 9, comprising FIGS. 9A and 9B, is a set of flowcharts of the spatial-domain method of image registration process. FIG. 9A is a flowchart showing the general spatial-domain method; FIG. 9B showing the spatial-domain method wherein a rotation step and a pre-processing step are conducted.

FIG. 10, comprising E1001-E1006, is a list of equations defining gradient operators.

FIG. 11 is two exemplary images of logically grouping of four image registration results produced by image registration with a Gradient Array Operator applied.

FIG. 12 is an exemplary image of storing imagery data to memory, where the imagery data is comprised of data from two successive frames.

The utility and novel technical solutions of the present invention will be more apparent by describing exemplary embodiments of the invention with reference to the figures provided. It should be appreciated that the embodiments described herein are exemplary but non-limiting, and are intended to illustrate the present invention through examination of preferred embodiments, and that inventions disclosed herein are not limited to the disclosed exemplary embodiments.

For clear and precise understanding of the present invention, specific details are described below. There will be a description of the relevant physical elements, and a description of multiple embodiments of the aimpoint correction method. The present invention includes within its scope all embodiments defined by the claims including alternatives, modifications, and logically equivalent methods and solutions.

Various embodiments described herein relate to a system of a digital weapon sight 100, exemplarily shown if FIG. 1 as affixed to a firearm 101. The aimpoint correction method pertains to and utilizes elements of the digital weapon sight described exemplarily here.

FIG. 2 illustrates a nonlimiting exemplary embodiment of the digital weapon sight in a functional and physical block diagram. Fundamental elements of the digital weapon sight 100 are: a housing 200; imaging optics 201 focusing a light to an image sensor 202; a computational unit, in this section and in claims referred to as the “processor” 203, configured by internal or external non-volatile memory 204 with instructions or configuration data to execute or implement in embedded hardware a series of processes and/or state machines to carry out the aimpoint correction method outlined herein; a ballistic event and orientation sensor 207; addressable digital memory 205; a user interface 208; an imagery output mechanism 210; and electronic power supply 213. In the preferred embodiment, the system includes a mounting bracket or rail mount 214 as a manner of affixing the weapon sight housing to a weapon, though this may be removable from and independent of the weapon sight housing. After-market rail mount accessories may affix to the housing 200 itself, and then to the weapon, in which case the manner of the affixing to the weapon may be the very shape of the housing 200. The wide arrows of FIG. 2 indicate the flow of high-bandwidth imagery data; the fine arrows indicate flow of low bandwidth orientation, ballistic event, and user interface data. Depending on chip architecture, the non-volatile memory 204 and/or the addressable memory 205 may be inside the same chip package as the processor 203.

The weapon sight 100 operates by capturing and displaying live video, with reticle symbology 300 overlayed upon imagery.

The image sensor 202 may be a color or monochrome CMOS or CCD image sensor or infrared focal plane array. When the image sensor 202 further comprises a color sensor array and imagery data is output in red, green, and blue constituent colors, the imagery data collected and stored is, in the preferred embodiment, stored as a grayscale value, achieved by a common equation of Gray=0.299R+0.587G+0.114B, where R, G, and B are the red, green, and blue component values. It is instructive here to point out that the imagery data saved is not necessarily raw data from the sensor, and may be processed by image enhancements widely known and used in industry to improve the quality of imagery, examples being histogram stretching, contrast enhancements, non-uniformity corrections, bad pixel replacements, etc. As such, the term “imagery data” is widely used herein, to indicate data derived from the image sensor.

The processor 203 may be a microcontroller, microprocessor, or FPGA or other such electronic computation device.

The user interface 208 may be buttons, knobs, keypad, and/or touchscreens. It is through the user interface 208 that the user controls the weapon sight's general function and may enter and terminate the aimpoint correction process. The user interface 208 may be remote and external from the housing 200.

The ballistic event and orientation sensor system 207 is comprised in the preferred embodiment of at least an accelerometer or gyroscope, though ideally is a 6- or 9-axis IMU (accelerometer, gyroscope, and magnetometer), and indicates the rotation with respect to gravity of the entire system as well as indicating to the processor 203 that weapon has been fired. In the preferred embodiment, the ballistic event and orientation sensor 207 is further comprised of a magnetometer, reports to the processor 203 acceleration and orientation data, the latter in the form of Euler angles of yaw, pitch, and roll. Alternatively, the processor may calculate orientation data such as roll algorithmically subsequent to direct magnetometer and gyroscope and accelerometer measurements. In other embodiments, it may also include a microphone, in communication with the processor 203, the two configured to determine that a shot has been fired from the weapon 101.

The digital memory 205 (or “memory”) may be Dynamic RAM such as Low Power DDR (LPDDR), HyperRAM, SRAM, PseudoSRAM, block memory, or other such addressable memory. This memory must have sufficient space to store imagery data collected via the output of the image sensor 202, and stored during the aimpoint correction process. The digital memory 205 may be located within the same integrated circuit package or on the same chip as the processor 203.

The power supply 213 may be removable batteries or rechargeable batteries or a connection to an external power source.

The weapon sight has an imagery output mechanism 210, most often comprising a digital display and viewing optics represented by 211 and 212. Alternatively, the imagery output mechanism may be comprised of a wired or wireless link to an external image rendering system for viewing, e.g. a heads-up display, or connector to an external display system, e.g., a laptop or personal computer.

In the general case embodiment, the aimpoint correction method may follow the flowchart outlined in FIG. 5. The imagery shown to the user through the imagery output mechanism would, in the preferred embodiment, follow along the lines of the exemplary images of FIG. 3. The images stored and correlated through image registration algorithms herein will be referred to as I₁, for “pre-shot” imagery data; and I₂^R, for “post-shot” imagery data; and I₂ for the “post-shot” imagery data after rotation is applied to match the orientation of I₁. Furthermore, I₁, I₂, and I₂^R nomenclature is preserved between the figures and this section, as they appear in both figures and also equations below. (The superscript R in I₂^R denotes that it may be in a rotated frame of reference relative to I₁.)

According to the exemplary flowchart in FIG. 5, the processor 203, when commanded through user interface, initiates the method at step 601. In the next step, 602, the processor 203 stores imagery data continuously to memory 205, during which time the user aims and fires a shot. The implementation of the data storage may be configured as a “ring buffer,” which is set up in digital logic such that data is continuously written to a section of memory 205, the oldest data overwritten by new data. FIG. 3A shows an exemplary imagery displayed to user via the imagery output mechanism 210 at this stage. The ballistic event and orientation sensor 207 reports to the processor 203 based on its detection of the shot. Subsequently at step 603 the processor 203 isolates in a known memory address at least one entire frame of imagery data I₁ and orientation data acquired immediately prior to the shot. FIG. 3B shows this stage in notional user view.

In FIG. 5, the step of 621, as the user takes aim at the point of impact of the first shot, the processor 203 acquires imagery data and ballistic event and orientation sensor data. In 622, in a manner depending on the embodiment, the processor 203 isolates the second image I₂^R, capturing the post shot imagery data and orientation data. In embodiments, the processor 203 may activate this isolation event when the user holds the weapon steady for a sustained period of time; or when the user fires a second shot, and the ballistic event is detected; or when the user presses a button otherwise engages the user interface 208. FIG. 3C shows this stage in a notional user view, where the user has adjusted aim to target the point of impact of the first shot.

In the preferred embodiment, a “time out” or other exit option is included, by which the processor 203 returns to default operation, indicated by the arrow from 602 to 641 or 621 to 641. In this case, if no ballistic event, steady indication, or user interaction is detected during steps 602 or 621 in a preset amount of time, the process is aborted. Alternatively, the processor 203 may be configured to wait for the user to engage the user interface 208 to abort the process.

As such, in the embodiment represented in FIG. 5, there are two data capture moments producing: a first image, the pre-shot image, I₁, and pre-shot orientation data; and also producing a second image, the post-shot image, I₂^R, and post-shot orientation data.

Still with reference to FIG. 5, with two images saved, the processor 203 may proceed to step 623, wherein, using an image registration algorithm, the translational offset between I₁ and I₂^R is calculated. The image registration algorithm may follow the flow of FIG. 7 (frequency-domain method), or FIG. 8 (annular method), or FIG. 9B (spatial-domain method). Embodiments using these algorithms will be detailed.

The result of the image registration algorithm is the translational offset between the images—in another sense, the location where the centroid of I₂^R is to be found within I₁. Hence the importance of the user aiming at the point of impact in 621; that is the centroid of I₂^R and I₂. Herein, a translational offset (or sets thereof) derived from the image registration algorithm is 2-dimensional and relates to what is effectively the Cartesian coordinate difference between the nominal reticle 300 and the adjusted reticle 305. In the preferred embodiment, a validation step 624 is included, which validates the results of the image registration. Validation is based on the manner of the image registration algorithm.

In embodiments, a plurality of image registration algorithms may be used, and the validation step of 624 employed to determine agreement between them. For example, the frequency domain method and spatial domain method may both be employed in 623, and the translational offset output of each the two algorithms agreeing with each other is validated in 624. In the preferred embodiment, using a plurality of image registration algorithms, the agreement heuristic is simply the distance between the two translational offset outputs. This must be of very low value, e.g., <1% of the size of the frame of data.

In embodiments using a spatial-domain or frequency-domain method, the validation in 624 may be based on the ratio of the value of a plurality of local maxima or minima found in the image registration. In embodiments employing a gradient array operator (to be detailed), the validation step 624 tests the level of agreement between several independent cross-correlation results.

In embodiments using the annular method, the validation check in 624 may be conducted by verifying that the phase of any potential valid solution of the image registration accords with the difference between the pre-shot orientation data and the post-shot orientation data, both collected by the processor 203 from the ballistic event and orientation sensor 207 in steps 602 and 622. In embodiments with an IMU serving as the ballistic event and orientation sensor, this can be seen a differential “roll” in Euler angles, and is used to rotate one image with respect to the other or as validation of an annular method, which will be discussed.

In the preferred embodiment, and iterative solution is employed to increase speed of calculations, wherein the “binning,” or averaging clusters of adjacent pixels, is employed prior to registration algorithm. This reduces image resolution, and therefore memory size and process time. For example, in a 640×480 sensor, the preferred bin is 8×8, reducing the entire image size by a factor of 64. In such an embodiment, parts of the image registration algorithm of FIG. 9A or FIG. 9B are run a plurality of times (preferred: two) at different levels of resolution. A coarse-resolution image registration of the entire imagery array precedes a fine-resolution image registration step, the latter conducted only over a windowed region of interest, the center of which is the result of the coarse-resolution registration step.

Passing validation, the reticle in the user's view is then adjusted by the translational offset, and results are stored in non-volatile memory 204 and/or cached processor memory in the process step 631 and the aimpoint correction method is complete.

Upon successful completion, imagery displayed to user is shown exemplarily in FIG. 3D, where the reticle 300, originally centered in the imagery, has been moved to a new position, and what is displayed is now an “adjusted reticle” 305, and will more precisely indicate location of the point of impact of subsequent shooting. This is also illustrated in FIG. 1, which in 102 is a notional image showing the effect of the aimpoint correction with the initial reticle 300, grayed out for illustrative clarity, the adjusted reticle 305, positioned at a point translated from the center by the translational offset. This adjusted reticle 305 and video data collected through the image sensor 203 is then displayed through the imagery output mechanism 210.

In an embodiment, the method may be employed in a weapon sight production environment to calibrate the weapon sight to its own housing 200. In this case, no shots need be fired, as the weapon itself is not used, but a fixture with a laser attached may be employed. Instead of acquiring imagery data based on ballistic events, the processor 203 may collect pre-shot and post-shot imagery data based on the user engaging the user interface 208.

In an embodiment, the method may be implemented in a manner such that the user fires a plurality of shots while aiming at the same specific point, then aims at the apparent centroid of the cluster (in step 621, FIG. 5). In another embodiment according to FIG. 5, an additional step of 632 may be added to allow the user to repeat the steps to repeat and refine the output of the aimpoint correction method. In this embodiment, if the second image capture is prompted by a ballistic and orientation sensor 207 event indication, initiating the isolation of I₂, I₂ becomes the new I₁, and then the processor 203 retrieves a new I₂ according to FIG. 5. Effectively, from the user's point of view, it would be like “chasing” the point of impact as the user keeps aiming and shooting at the previous shot's point of impact until aimpoint deviation is sufficiently corrected.

Embodiments include under-sampling the image sensor 202, and shifting its “active window”. For example, an image sensor 202 of 1400×1200 pixels may be sampled over an “active window” of 1280×960 pixels, which are displayed to the user. In this case, an alternative to adjusting the position of the reticle 300, is to adjust the position of the active window of the sensor. This is advantageous as it keeps the adjusted reticle overlay 305 at or near the center of the imagery seen by the user. If the translational offset between the two images I₁ and I₂ is too large, both such window adjustment and reticle adjustment may be employed.

It is also preferred that symbology (graphic symbols and icons) be placed over the imagery to assist the user in the process. This is not limited to instructions; should an IMU be used in place of the ballistic event and orientation sensor 207, the processor 203 may employ thresholds to the capturing of the post-shot image I₂: that it be within field of view of the first shot, and that the weapon sight be held sufficiently steady to get a clear image. These and other user benefits are important for usability of the method, but are not meant to limit the claims herein.

In what one can call an “Automatic Capture” embodiment of the aimpoint correction method, alternative to the case abovementioned and illustrated in FIG. 5, the user is not required to aim exactly at the point of impact as is done at step 621. Instead, the processer 203 calculates the point of impact algorithmically by capturing the post-shot imagery data I₂^R (when the weapon sight is appropriately steady, per the ballistic event and orientation sensor 207), rotating I₂^R to create I₂, and then determining the translational offset between the two images through image registration. Then I₂ and I₁ are both cropped based on the translational offset to cover only common elements, then an absolute value subtraction is conducted according to Eq. 1 between cropped I₁ and I₂ to determine the absolute difference of images (ADI). This reveals an image where the point of impact can be determined by finding the centroid of the region of peak value. The processor 203 then conducts a verification step to verify that the peak exists, is not too large, and no other peaks of similar prominence exist.

ADI ⁡ ( x , y ) = ❘ "\[LeftBracketingBar]" I 1 ( x , y ) - I 2 ( x , y ) ❘ "\[RightBracketingBar]" Eq . 1

An exemplary flowchart of the method is outlined in FIG. 6, which refers to steps in FIG. 5, and exemplary images are captured in FIG. 4. Note that these are images as captured by the processor 203 and stored to memory, and thus are not presented with reticle overlays. An exemplary flow follows the steps 601-602-603, but then, according to FIG. 6, instead of following the previously defined process, it may proceed to 611 where the post-shot image is captured, then to 612 where the image registration is conducted, in the same manner as 623. It is notably different and independent, however, and a fallback case, 623 may still be conducted.

FIG. 4A shows the pre-shot image, I₁, as captured by the processor 203. FIG. 4B shows I₂, the image captured after a shot. The processor 203, and then conducts the image registration algorithm (FIG. 7 or FIG. 8) to find the translational offset between I₁ and I₂. In step 613, there is a validation step mathematically similar to 624, wherein a plurality of outputs of the image registration algorithm may be compared. It is notably different and independent, however, and in a fallback case, 624 will still be conducted.

In step 614 each image is cropped to an area covering common pixels: the cropping window on I₁ is 401 and the cropping window on I₂ is 402. The resulting images are: FIG. 4C, a cropped version of FIG. 4A; and FIG. 4D, a cropped, rotated version of FIG. 4B. The processor 203 then determines the absolute difference of the images via Eq. 1, producing the difference array (ADI) between FIGS. 4C and 4D, illustrated notionally in FIG. 4E, in which one case see the point of impact and a lot of noise.

Following FIG. 6, in step 615, the processor 203 may conduct a subtraction of images, exemplarily according to Eq. 1. Noise filtering may be employed to suppress peaks in the difference found in step 615. Still following FIG. 6, in step 616 the point of impact location may be ascribed to the location of the ADI array that passes “constrainment” criteria.

The constrainment criteria are meant to guarantee that the peak is not too large and constrained locally. Referring to FIG. 4, this may be based on a locus of points, the “constraining locus” 410 applied to significant peak values of the difference array FIG. 4E to verify the values from the peak to the edge of 410 fall to below a threshold and/or are not continuous from peak to the constraining locus 410.

In the preferred embodiment, the processor 203 detects the point of impact through a process of firstly removing extended line segments of the difference image FIG. 4E, the finding one or a plurality of maximal values in the difference image, softening the image, and then finding a constrained peak. Such processing methods are known to those knowledgeable in the field, and are exemplified by gaussian blurring, and edge detection.

In the preferred embodiment, the processor 203 is configured to validate constrained peaks based on the value of the maximum; based on the value of the extreme maximum relative to other local maxima; based on the size of the constrainment locus containing the maximum where values of the constrainment locus is less than a value derived from the standard deviation of the image; based on the values of the image intersecting the constrainment locus.

In the preferred embodiment, two maximal values from the image are tested for constrainment and the values within their respective constrainment loci are compared to determine the certainty that multiple points in the image do not near equally satisfy the constrainment criteria.

For example, if values at the constraining locus 410 are larger than the threshold and extend through the constraining locus 410, for example, see 411, where there are elements of the array that intersect the constraining locus, then that peak will not be considered constrained. If all or a plurality of peaks violate this constraint, the process fails and reverts to the previously mentioned step 621 of FIG. 5 where the user aims at the point of impact.

Important preprocessing steps used in this “Automatic Capture” method may include gaussian blurring (known to those knowledgeable in image processing), to reduce the noise seen in FIG. 4E, caused by movement of objects in the scene and inaccuracy of the image registration algorithm, and which is illustrated in FIG. 4E. In that image, the constraining locus 410 is shown at two locations; 411 shows a point where shows the containment requirement is violated, showing the importance of filtering the output with a Gaussian blurring process implemented in as a part of 702.

In an embodiment, the image registration step 623 and/or 612 within the aimpoint correction method is executed according to a frequency-domain method, an exemplary flowchart of which is shown in FIG. 7. Referring to FIG. 7, the processor 203 may rotate I₂^R imagery data about its center in step 701, based on the difference in the orientation data between the two data capture moments creating the rotation-corrected image I₂. This step inherently re-sizes the image, reducing the image frame size about the center, nominally at a maximum by a factor of the square root of 2, to allow for a 45-degree rotation. As now I₂^R is rotated, we can call it I₂.

With continued reference to FIG. 7, the processor 203 may proceed to a zero-padding step 702. Zero-padding is a technique used in signal processing and image processing where one extends the size of an array (such as an image) by adding pixels of value zero around its borders. This is often done to match the dimensions of two data sets.

With continued reference to FIG. 7, the processor 203 may proceed to a Fourier Transformation step. In the preferred embodiment of the frequency-domain method, the manner of computing the Fourier Transformation in step 703 and its inverse in step 706, is a Fast Fourier Transform, where both the Fast Fourier Transform (FFT) and Inverse Fast Fourier Transform (IFFT) are very common approaches in digital systems processing.

With continued reference to FIG. 7, in 704, the processor calculates the complex conjugate of image I₂, and subsequently in 705 multiplies the complex conjugate of the FFT of I₂ with the FFT of I₁.

According to FIG. 7, the Inverse Fourier Transform is then calculated in step 706, and the magnitude of the result is determined in 707. The result of this is effectively equivalent to the cross-correlation of I₁ and I₂ in the spatial domain. Maximal values of the resultant array are found in the step 708, and indicate the translational offset between the two images. In the preferred embodiment, for subsequent validation step 624, a plurality of peaks is found in 708, and the validation step 624 uses the ratios between them to validate the image registration result.

Determination of the magnitude of the inverse transform in 706 is performed because the result of the Inverse Fast Fourier Transform (IFFT) is generally a complex-valued array, even if the input to the IFFT was purely real. Hence for each element, its magnitude must be determined, where any element may have the complex value z=a+bi, the magnitude is sqrt(a²+b²). The peaks of the magnitude, and best candidates for the translational offset, are found in step 708.

In any embodiment, either or both images may be cropped or remapped such that the number of pixels in row and column may be a power of 2, simplifying the Fourier Transform algorithm. These are known algorithms and techniques in the field of digital signal processing.

In an embodiment, the image registration step 623 and/or 612 within the aimpoint correction method may be executed according to FIG. 8, the annular method: wherein for all points (i,j) within the pre-shot imagery data I₁, an annulus of image points at a certain radius may be sampled from the imagery data, and those data used to create an annular set of data, a “pre-shot annulus” 803. A similar-sized annulus of data may be extracted from the post-shot imagery data I₂^R to create a “post-shot annulus” 804.

These data are compared through an image registration algorithm in 805, the output of which is an array, the extrema (maximum or minimum depending on the equation used) of which is the peak of the correlation for any particular value of (i,j) and which represents an angle of rotation. The peak correlation value and peak rotation that must match the a difference in rotation between the two for all values (i,j); to that end the processor 203 stores these correlation results to a local memory cache, in step 806, compare against each other as well. The primary examination is the ratio of the maximum to other local maxima, which in literature may be found as the “peak-to-side-lobe ratio”. In this step, or in alternate embodiments, in the step 624, the processor 203 may compare results against orientation data collected in 602 and 622, as the output of this annular method also includes phase data, and that should correspond to the difference in rotation, or “roll,” between data collected from the ballistic event and orientation sensor 207.

In an embodiment of the annular method, the processor 203 may conduct a spatial domain cross-correlation type image registration algorithm, in according to FIG. 9A to find the translational offset between I₁ and I₂. Cross-correlation is a technique used in image registration to find the spatial displacement or transformation between two images. Cross-correlation type registration methods are conducted by a series of steps: of defining a search area, using a slide window (also known as a ‘template’ or ‘patch’) to traverse the search area of the target image, and at each position of the sliding window calculating the sum of the differences between the between the pixel values of the sliding window and the corresponding region of the reference image.

The cross-correlation type methods of the spatial domain method take the forms of a Sum of Absolute Differences, wherein each convolution step involves a subtraction of pixel values and the summation of all the differences of all the overlapping pixels; a Sum of Squared Differences, wherein the convolution involves a subtraction of pixel values, the squaring of that difference, and the summation of all the differences of all the overlapping pixels; or a cross-correlation method wherein the convolution step involves the multiplication of the pixels, after each pixel is offset according to the average value of the data it is derived from.

The cross-correlation type methods may take the form of Normalized Sum of Absolute Difference (“NSAD”, Eq. 1), Normalized Sum of Squared Differences (“NSSD”, Eq. 2), Cross-Correlation, Zero-Mean Normalized Cross Correlation (“ZMNCC”, Eq. 3), Zero-Mean Sum of Squared Differences, or other similar in mathematical nature. These and others are well document in the science of image processing and statistics.

NSAD ⁡ ( u , v ) = 1 x ⁢ y ⁢ ∑ x , y ❘ "\[LeftBracketingBar]" I 1 ( u , v ) - I 2 ( x - u , y - v ) ❘ "\[RightBracketingBar]" Eq . 2 NSSD ⁡ ( u , v ) = 1 x ⁢ y ⁢ ∑ x , y ( I 1 ( u , v ) - I 2 ( x - u , y - v ) ) 2 Eq . 3 Eq . 4 ZMNCC ⁡ ( u , v ) = ∑ x , y [ I 1 ( x , y ) - avg ⁡ ( I 1 , u , v ) ] [ I 2 ( x - u , y - v ) - avg ⁡ ( I 2 ) ] sqrt ⁢ { ∑ x , y [ I 1 ( x , y ) - avg ⁡ ( I 1 , u , v ) ] 2 ⁢ ∑ x , y [ I 2 ( x - u , y - v ) - avg ⁡ ( I 2 ) ] 2 }

In Eq. 2 and Eq 3. the division by (1/x y) is a division of the number of pixels of the being summed at each slide position. In Eq. 4 avg(I₁, u, v) is the average of the I₁ image under the I₂ template, and avg (I₂) is the average of the entire template.

All the aforementioned equations output an array, the extreme value of which (maximum or minimum, depending on the algorithm; the NSAD the best correlation is at the minimum an NCC or ZMNCC the best correlation is at a maximum) is found in 904 and indicates a point where two images have highest correlation. Optimally, the entire output array need not be saved to memory as each element of the output array is calculated; only the extremum or a plurality of extrema need to be retained in memory.

In an embodiment, the cross-correlation algorithm is a Normalized Sum of Absolute Differences (“NSAD”) (Eq. 2), with an added feature that elements at or near the center of I₂ (near the point of impact) are excluded from the summation. The minimum of the NSAD output array reveals the best choice for where the centroid of I₂ is to be found in I₁.

In an embodiment of the annular method, the image registration algorithm is executed in a frequency-domain method. In this, a process following a modification of FIG. 7 is employed, wherein 701 and 702 are both bypassed, according to the arrow 750. The best location produced by 805 both has the highest degree of peak to side lobe ratio, and also accords with data captured by the ballistic event and orientation sensor.

In an embodiment, the image registration step 623 and/or 612 within the aimpoint correction method is executed according to the spatial domain cross-correlation method, according to FIG. 9B. The processor 203 may rotate I₂^R imagery data about its center in step 901, based on the difference in the orientation data between the two data capture moments creating the rotation-corrected image I₂. This step inherently re-sizes the image, reducing the image frame size about the center, nominally at a maximum by a factor of the square root of 2, to allow for a 45-degree rotation. Imagery data may be pre-processed for image enhancement in step 902, which may include normalization, histogram equalization, gradient detection, gradient amplification, gaussian smoothing, noise reduction, or convolutional image filters or processing steps known to those knowledgeable in the art. Such enhancements may be done before writing imagery data to memory, in step 602 and 622.

The processor 203 may then proceed according to FIG. 9B, to step 903 and run a cross-correlation type image registration algorithm to find the translational offset between I₁ and I₂.

The cross-correlation type methods of the spatial domain method may take the form of: a sum of absolute differences (see: the NSAD, Eq. 2), wherein each convolution step involves a subtraction of pixel values and the summation of all the differences of all the overlapping pixels; a sum of squared differences (see: the NSSD, Eq. 3), wherein the convolution involves a subtraction of pixel values, the squaring of that difference, and the summation of all the differences of all the overlapping pixels; or a cross-correlation method wherein the convolution step involves the multiplication of the pixels, after each pixel is offset according to the average value of the data it is derived from (see: the ZMNCC, Eq. 4). The similarity metric of the sum of absolute or squared difference methods is the minimum value of the output of the NSAD(u, v) or NSSD(u, v) data arrays; the similarity metric of cross-correlation approach is the maximum value of ZMNCC(u, v).

All the equations Eq.'s 2-4 output an array, the extreme value of which (maximum or minimum, depending on the algorithm) indicates a point where two images have highest correlation, and reveal the centroid of the I₂ image within I₁, this being the translational offset. Optimally, the entire output array need not be saved to memory as each element of the output array is calculated; only the extremum or a plurality of extrema need to be retained in memory.

A noise-reducing filter may be applied upon any correlation output array prior to searching for a maximum or minimum value. The values or a ratio between the values of the operative extremum and the second-most extreme value may passed to the validation step in 624. This validation level for this ratio depends on the algorithm chosen, but for NSAD should generally be >1.25. The output of step 904 reveals the best translational offset to apply as an aimpoint correction.

In the preferred embodiment of the spatial-domain cross-correlation method, the image registration algorithm in step 623 follows the specific flow of FIG. 9B with processing steps for improved image registration reliability. All imagery data is processed in step 902 with a “gradient array operator,” which is an array of first and/or second order partial derivatives of the imagery data, or absolute values thereof, and/or the raw imagery data, and which outputs a set of processed imagery data.

FIG. 10 shows the basic gradient equations, as a derivative of the image in either the x or y direction, in E1001. From this point, we can create a “absolute directional gradient operator” E1002, which is a linear combination of the partial derivatives in the x- and y-direction, but subject to an absolute value. In the case that the coefficients a and b of the linear combination are set to a=1 & b=0, the result is all positive gradients in the x-direction, and set all other values to zero. Per E1003, we see a similar absolute gradient, but this time in a diagonal direction (as opposed to x-,y-cardinal directions).

The gradient array operator is thus an array of multiple types or orders of gradient operators: x-direction, y-direction, second order derivative, absolute value of the first order derivative, etc. Examples can be found in E1004, E1005, E1006, all of which can be considered as a gradient array operator. In E1004, the initial image plus five different derivates constitute a “Multi-Level Gradient Array”. Between E1005 and E1006, all orientations of the Absolute Gradient Operator E1002 are found; E1006 is fundamentally equivalent to E1005, rotated 45 degrees. In the preferred embodiment, the gradient array operator takes the form of E1006, the Diagonal Method of the ADG Array.

Turning back to FIG. 9B, from one image, the Gradient Array Operator produces a multiple of processed images, which are sent to a cross-correlation based image registration algorithm. The utility of this method is as follows: it is beneficial for speed and memory to concatenate each pixel of the four outputs of the gradient array operator to a single data element, and then run the image registration algorithm a single time as opposed to the four registrations. With this optimization, in each internal step of the cross-correlation, the single data element is deconstructed into four elements and independent cross-correlations are run on those.

The x,y locations of the four extrema of each cross-correlation algorithm are then put through an agreement analysis step that groups and averages the “agreeing results” together and discards erroneous results, and then the number of “agreeing results” is fed to a validation step, for reference see 905 of FIG. 9B.

To expand on this, refer to FIG. 11, where 1101 and 1105 both illustrate a possible set of outputs (x,y locations of NSAD minima), labeled a,b,c,d. In 1101, the radius 1103 centered at the average point 1102 of all four points contains only one point; in 1105, the same size radius 1107 centered at 1106, which is the average of points a, b, and c contains a,b,c—the removal of point d reveals that a, b, and c “agree” with each other. In this case, three results “agree.” Thusly, “agreeing results” are those that occur within a certain distance of each other or within a certain distance of common point. From logically grouping and averaging these, the translational offset (or failure to agree) is determined, the image registration is complete, and the process proceeds to 624, where the validation check in this embodiment relies on the number of agreeing results.

If the number of agreeing results is not sufficient—for a in the preferred embodiment with a gradient array operator of type E1006 this number is optimally three or four-then the process proceeds to step 641 and the process aborts with a failure. In embodiments, the user may select the degree of agreeing results.

If the number of agreeing results is sufficient, then the process proceeds to step 631, where results are stored in non-volatile memory 204 and the reticle display 300 is adjusted to indicate the subsequent point of impact. At this stage, imagery displayed to user is shown in FIG. 3D, where the reticle 300, originally centered in the imagery, has been moved to a new position, and what is displayed is now an “adjusted reticle” 305.

Finally, in a further optimization to reduce memory requirements, in an embodiment, the processor 203 is interfaces with the ballistic event and orientation sensor 207 in such a manner that a signal from latter, indicating the firing of the weapon, interrupts the storing of the imagery data. Such an implementation is exemplarily shown in FIG. 12, where the stream of imagery data 1204 is stored in a buffer 1205, and an interrupt 1208 is an input to a logic block 1206. The logic block 1206, within the processor 203, is configured halt storing the data when directed by the interrupt. The buffer should be of a size to remove any data that may be disturbed by the firing of the weapon, and depends on the delay between the weapon fire and the interrupt signal being processed. Any data output from the buffer 1207 is therefore data collected before the firing of the weapon, and is stored in memory, represented by 1201, which is comprised of data from two successive frames: where 1203 is data from a previous frame and 1202 is data from the most current frame before the interrupt. In such an embodiment, the ballistic event sensor 207 may be comprised of at least an inertial measurement unit (IMU) and the IMU is set to deliver an interrupt based on an accelerometer threshold, set by the processor 203.