GUNSHOT DETECTION USING MULTICHANNEL ANALYSIS

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of U.S. Provisional Application No. 63/729,990, filed Dec. 10, 2024, and U.S. Provisional Application No. 63/787,010, filed Apr. 11, 2025, the contents of which are incorporated by reference herein.

This application is a continuation-in-part of U.S. application Ser. No. 18/660,490, filed May 10, 2024, which is a continuation-in-part of U.S. application Ser. No. 18/642,901, filed Apr. 23, 2024, which claims the benefit of U.S. Provisional Application No. 63/561,823, filed Mar. 6, 2024; U.S. Provisional Application No. 63/613,114, filed Dec. 21, 2023; U.S. Provisional Application No. 63/541,324, filed Sep. 29, 2023; U.S. Provisional Application No. 63/465,896, filed May 12, 2023; and U.S. Provisional Application No. 63/461,339, filed Apr. 24, 2024.

This application is a continuation-in-part of U.S. application Ser. No. 18/642,901, filed Apr. 23, 2024, which claims the benefit of U.S. Provisional Application No. 63/561,823, filed Mar. 6, 2024; U.S. Provisional Application No. 63/613,114, filed Dec. 21, 2023; U.S. Provisional Application No. 63/541,324, filed Sep. 29, 2023; U.S. Provisional Application No. 63/465,896, filed May 12, 2023; and U.S. Provisional Application No. 63/461,339, filed Apr. 24, 2024.

BACKGROUND

Gunshot detection can enhance safety by enabling a response to firearm-related incidents. These systems can utilize sensors to detect a gunshot. The sensors can be distributed throughout an environment to determine the location of the gunshot within the environment.

SUMMARY

One innovative aspect of the subject matter described in this specification can be embodied in methods that include the actions of determining, by a device that includes an infrared sensor and an acoustic sensor, whether infrared data captured by the infrared sensor satisfies a saturation threshold; selecting, from a plurality of classifiers, at least one classifier using a result of the determination whether the infrared data satisfies the saturation threshold; accessing at least one feature from one or more of the infrared data captured by the infrared sensor or acoustic data captured by the acoustic sensor; determining, using the at least one classifier and the at least one feature, whether the at least one feature represents a possible gunshot event in an environment that includes the device; and performing an action using a result of the determination whether the at least one feature represents a possible gunshot event in the environment.

One innovative aspect of the subject matter described in this specification can be embodied in systems that include an infrared sensor; a plurality of classifiers; one or more processors; and one or more storage devices on which are stored instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising: receive, from the infrared sensor, infrared data; determine whether an infrared feature from the infrared data satisfies a saturation threshold; select, from the plurality of classifiers, at least one classifier using a result of the determination whether the infrared feature from the infrared data satisfies the saturation threshold; access at least one feature from one or more of the infrared data or acoustic data captured by an acoustic sensor; determine, using the at least one classifier and the at least one feature, whether the at least one feature represents a possible gunshot event in an environment that includes the infrared sensor; and perform an action using a result of the determination whether the at least one feature represents a possible gunshot event in the environment.

One innovative aspect of the subject matter described in this specification can be embodied in computer storage media encoded with instructions that include the actions of determining, by a device that includes an infrared sensor and an acoustic sensor, whether infrared data captured by the infrared sensor satisfies a saturation threshold; selecting, from a plurality of classifiers, at least one classifier using a result of the determination whether the infrared data satisfies the saturation threshold; accessing at least one feature from one or more of the infrared data captured by the infrared sensor or acoustic data captured by the acoustic sensor; determining, using the at least one classifier and the at least one feature, whether the at least one feature represents a possible gunshot event in an environment that includes the device; and performing an action using a result of the determination whether the at least one feature represents a possible gunshot event in the environment.

Other implementations of this aspect include corresponding computer systems, apparatus, computer program products, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

The foregoing and other implementations can each optionally include one or more of the following features, alone or in combination.

The at least one feature may include an infrared amplitude feature; determining whether the infrared amplitude feature satisfies the saturation threshold may include determining whether the infrared amplitude feature satisfies a peak amplitude threshold; and selecting at least one classifier may include, in response to determining that the infrared amplitude feature satisfies the peak amplitude threshold, selecting, from the plurality of classifiers that includes saturated classifiers and unsaturated classifiers, at least one saturated classifiers.

The at least one feature may include an infrared amplitude feature; determining whether the infrared amplitude feature satisfies the saturation threshold may include determining whether the infrared amplitude feature satisfies a peak amplitude threshold; and selecting at least one classifier may include, in response to determining that the infrared amplitude feature does not satisfy the peak amplitude threshold, selecting, from the plurality of classifiers that includes saturated classifiers and unsaturated classifiers, at least one unsaturated classifiers.

The infrared amplitude feature may include a sailboat feature; and selecting at least one classifier may include, in response to determining that the sailboat feature does not satisfy the peak amplitude threshold, selecting, from the plurality of classifiers that includes saturated classifiers and unsaturated classifiers, at least one unsaturated classifier.

Accessing the at least one feature from the one or more of the infrared data captured by the infrared sensor or the acoustic data captured by the acoustic sensor may include accessing at least an infrared peak feature from the infrared data and an acoustic peak feature from the acoustic data; determining, using the at least one classifier and the at least one feature, whether the at least one feature represents the possible gunshot event in the environment that includes the device may include: determining whether the infrared peak feature and the acoustic peak feature satisfy a peak timing threshold; and determining, in response to determining that the infrared peak feature and the acoustic peak feature satisfy the peak timing threshold and using the at least one classifier and the at least one feature, whether the at least one feature represents the possible gunshot event in the environment that includes the device.

Determining whether the at least one feature represents a possible gunshot event in the environment may include: providing the at least one feature to the at least one classifier; receiving from at least one of the classifiers a detection score that indicates a likelihood of a possible gunshot event in the environment; determining whether the detection score of the possible gunshot event in the environment satisfies a detection score threshold; and in response to determining that the detection score of the possible gunshot event in the environment satisfies the detection score threshold, determining that the at least one feature represents a possible gunshot event in the environment; and performing the action may include, in response to determining that the at least one feature represents a possible gunshot event in the environment, providing at least some data from the infrared data or the acoustic data to at least one solver.

The method may include determining, using the at least one solver, whether the possible gunshot event is likely an actual gunshot event in the environment; and selectively triggering an event or determining to skip triggering an event using a result of the determination whether the possible gunshot event is likely an actual gunshot event in the environment.

The method may include in response to determining that the possible gunshot event is likely an actual gunshot event in the environment, determining, using a weapon classifier, a predicted weapon type of the actual gunshot event; and selectively triggering an event using the predicted weapon type.

Determining the predicted weapon type of the actual gunshot event may include determining, using the weapon classifier and an earth mover distance feature, the predicted weapon type of the actual gunshot event.

The at least one classifier may include at least two classifiers, and determining whether the at least one feature represents a possible gunshot event in the environment may include: receiving, from each of the at least two classifiers, a corresponding detection score that indicates a likelihood of a possible gunshot event in the environment, generating an aggregated detection score by combining the corresponding detection scores from the at least two classifiers, determining whether the aggregated detection score satisfies a detections score threshold, and in response to determining that the aggregated detection score of the possible gunshot event in the environment satisfies the detection score threshold, determining that the at least one feature represents a possible gunshot event in the environment.

Determining whether the at least one feature represents a possible gunshot event in the environment may include: providing the at least one feature to the at least one classifier; receiving from at least one of the classifiers a detection score that indicates a likelihood of a possible gunshot event in the environment; and determining whether at least one detection score from the detection scores received from at least one of the at least one classifier indicates a detection of a likely false positive; and performing the action may include, in response to determining that at least one detection score from the detection scores received from at least one of the at least one classifier indicates a detection of a likely false positive, determining to not provide the detection of the possible gunshot event in the environment to at least one solver.

The at least one solver may include a muzzle solver or a flash solver, and determining the detection of the possible gunshot event in the environment uses one or more of the muzzle solvers and at least one acoustic feature from the acoustic data, or the flash solver and at least one infrared feature from the infrared data.

Determining whether the at least one feature represents a possible gunshot event in the environment may include: detecting, in at least some of the acoustic data, a first possible gunshot event and a second possible gunshot event, determining, using data for the first possible gunshot event and the second possible gunshot event, whether the second possible gunshot event satisfies an echo event threshold for the first possible gunshot event, and determining, in response to determining that the second possible gunshot event satisfies the echo event threshold for the first possible gunshot event, to not provide data for the second possible gunshot event to another processing engine.

The system may include the acoustic sensor, and the operations may include receiving, from the acoustic sensor, the acoustic data.

Other implementations of this aspect include corresponding computer systems, apparatus, computer program products, and computer programs recorded on one or more computer storage devices, each configured to perform the actions of the methods. A system of one or more computers can be configured to perform particular operations or actions by virtue of having software, firmware, hardware, or a combination of them installed on the system that in operation causes or cause the system to perform the actions. One or more computer programs can be configured to perform particular operations or actions by virtue of including instructions that, when executed by data processing apparatus, cause the apparatus to perform the actions.

The foregoing and other implementations can each optionally include one or more of the following features, alone or in combination.

The subject matter described in this specification can be implemented in various implementations and may result in one or more of the following advantages.

A system which receives multiple channels of acoustic and infrared information may reduce false positives from non-gunshot noises (e.g., fireworks, or car backfires) as the system analyzes both streams of information to match gunshot event characteristics. This can increase the likelihood of determining a true positive gunshot event and increase the safety of the environment monitored by the system.

A system which receives multiple channels of acoustic and infrared information may increase the probability of detecting a true positive gunshot event when one channel of information is compromised, e.g., in wind or rain. This can improve system robustness in outdoor, indoor, and cluttered environments.

A system which processes multiple channels of information in parallel may validate true positive gunshot events and screen false positive gunshot events in near-real time. The system can increase the speed of detection, response, or both, which can increase environment safety, decrease the personal and environmental costs of such a gunshot event, or both.

A system which receives multiple channels of acoustic and infrared information may increase the probability of detecting a true positive gunshot event by identifying features of the information which are likely indicative of a gunshot event versus features which are indicative of false positives, such as echoes. A system having a higher true positive detection rate and a lower false positive detection rate than other systems can be installed in environments having complex sound- and landscapes, such as concert venues, or outdoor spaces.

The details of one or more implementations of the subject matter described in this specification are set forth in the accompanying drawings and the description below. Other features, aspects, and advantages of the subject matter will become apparent from the description, the drawings, and the claims.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a flow diagram for gunshot detection using multichannel analysis.

FIG. 2 is a flow diagram for multichannel analysis of gunshot sensor information.

FIG. 3 is a system block diagram for a gunshot sensor device with multichannel analysis.

FIG. 4 is a block diagram for IR and acoustic gunshot detection.

FIG. 5 is a block diagram for gunshot sensor solver fusion.

FIG. 6 is a block diagram for a support vector machine (SVM).

FIG. 7 is a flow diagram for machine learning (ML) training for a support vector machine.

FIG. 8 is a system diagram for gunshot detection using multichannel analysis.

FIG. 9 is a flow diagram for gunshot detection using multichannel analysis.

FIG. 10 is a system diagram illustrating an example of a computing system.

Like reference numbers and designations in the various drawings indicate like elements.

DETAILED DESCRIPTION

To address the firearm-involved crime crises, emergency services, government agencies, law enforcement, and others demand timely, accurate detection of incidents involving firearms. The timely and accurate detection of gunshots enables rapid and appropriate responses to be mounted quickly. However, accurate gunshot detection is a highly complex problem. Natural environmental events such as lightning, earthquakes, and thunder; manmade events such as fire alarms, strobes, and sirens; or even highly amplified music and light shows at concerts all present challenges to accurate gunshot detection.

Techniques for gunshot detection using multichannel analysis are disclosed. Infrared and acoustic information is collected using a gunshot sensor device. Multiple IR and acoustic sensors are included in the gunshot sensor device. Each sensor provides a separate information channel for analysis. Features from both the infrared and acoustic information channels are extracted and analyzed. Infrared information channels direct features collected by the IR sensors through a flash detection classifier. Acoustic information channels direct features collected by the acoustic sensors through a shock detection classifier and a muzzle detection classifier. The classifiers compare features collected by the sensors with features stored in one or more trained machine learning models. The classifiers group possible gunshot events into those likely to be gunshots and those not likely to be. In some cases, events that appear likely to be gunshot events can be matched with details of weapon types. Location and range information can be calculated from the IR and acoustic feature information. The gunshot detection event data analyzed by the classifiers is forwarded into flash, shock, and muzzle solvers. The solvers work to further analyze the infrared and acoustic data and combine it into matched pairs in order to more accurately determine true gunshot events from false alerts. Information collected by IR and acoustic sensors can arrive at the gunshot sensor device at different times. Generally, IR light data arrives first, some from the initial explosion of gases released by the firing of the gun, and later, from the igniting of residue as it exits the gun barrel and combines with oxygen and heat in the air. Supersonic acoustic data can arrive next, generated by the passage of a bullet through the air. Finally, the sound of the muzzle blast, traveling at the relatively slower speed of sound, can be collected by the acoustic sensors. The light and sound waves collected by the sensors have features that can be quite distinct from other events a human might find to be similar. The combination of these features, including the timing between the features, can form a multi-dimensional fingerprint of sorts—a profile that can be compared to the multi-variate profile of gunshot events stored in a machine learning dataset. As the multi-sensor integration classifier completes its work on possible gunshot events, validation of true gunshots or false alerts can be reported to external systems for further investigation and action steps.

FIG. 1 is a flow diagram for gunshot detection using multichannel analysis. The flow 100 includes collecting 110 infrared information using a gunshot sensor device 116. The gunshot sensor device can include a single unit, where the single unit can include one or more sensors, a processing element, a communication element, a power source, and so on. The gunshot sensor device can be contained within a housing. The gunshot sensor device can be mounted in an indoor space or an outdoor space. The gunshot sensor device can be mounted on an interior wall or ceiling, on a pole or a post, etc. The gunshot sensor device can be mounted in an outdoor setting on a post, a pole, a vertical wall, a bridge or overpass, and the like. The collecting infrared information is performed by two or more infrared sensors. The infrared (IR) information can include near-IR (NIR) information. The infrared sensor can sense one or more infrared bands, such as far-infrared (FIR) band light with wavelengths between 50-1000 μm; a mid-infrared (MIR) band light with wavelengths between 3-50 μm; and so on. The IR sensor can sense near-infrared (NIR) band light with wavelengths between 0.76-3 μm. The two or more infrared sensors provide infrared information channels 114.

Each infrared sensor can provide a separate infrared information channel. When a gun is fired, a bright flash of light is produced, along with infrared radiation. The two or more infrared sensors can detect the light flash, even in low-light conditions or through smoke and haze. In addition, small particles of gunshot residue are produced when a gun is fired. The infrared sensors can detect these particles using infrared spectroscopy. Some of these particles are ignited as they exit the gun barrel, producing additional infrared light waves. As each infrared sensor detects light and gunshot residue events from a gunshot, the infrared information can be collected and forwarded for processing. Each infrared sensor can forward its data through a separate information channel to be processed. In some embodiments, the differences between the data collected from each of the two or more infrared sensors during the same event can be used in calculating the location and distance of the gunshot origin, as well as in classifying the event.

The flow 100 includes collecting 112 acoustic information using the gunshot sensor device 116. The collecting acoustic information is performed by two or more acoustic sensors. The acoustic sensors can include one or more of a microphone, a high SPL microphone, a transducer, etc. The two or more acoustic sensors can detect acoustic signals based on one or more ranges of sound pressure levels (SPLs). The acoustic sensor can detect high SPL acoustic signals such as those associated with a gunshot sound event. The acoustic sensor can detect muzzle blast sounds caused by hot, rapidly expanding gases emerging from a gun barrel. Gunshot information that is obtained from the acoustic sensors can be based on detecting shockwave and muzzle blast acoustics associated with a projectile fired in the vicinity of a gunshot sensor device. In some embodiments, each acoustic sensor within the gunshot sensor can receive the shockwave and/or muzzle blast at a slightly different time. The time difference information can be used to infer information about a gunshot, such as the approximate direction from which a projectile was fired, and/or the approximate speed of the projectile.

In some embodiments, the two or more infrared sensors and the two or more acoustic sensors are collocated in a gunshot sensor device unit. The placement of the infrared and acoustic sensors can create a field of detection of gunshot events around the gunshot sensor device. The field of detection can include a minimum angle of elevation extending down and away from the gunshot sensor device. The field of detection can include a maximum angle of elevation extending up and away from the gunshot sensor device.

The two or more acoustic sensors provide acoustic information channels 114. Gunshots can be identified by their high sound pressure level acoustic signatures and the associated muzzle flashes. The acoustic signatures include muzzle blasts and acoustic shock waves. The acoustic signatures can vary significantly from firearm to firearm and between types of firearms such as long arms or handguns. The sounds generated by gunshots can be collected by each of the two or more acoustic sensors and translated into data that can be forwarded through separate information channels for analysis.

In some examples, the flow 100 includes determining whether a signal from the gunshot sensor device 116 satisfies a saturation threshold. The flow 100 can determine whether an acoustic signal, an IR signal, or both, satisfies a saturation threshold. A signal which may be saturated can include a peak (e.g., a positive peak) which indicates the sensor was saturated. Without expressing limitation, a sensor can be saturated when the input signal exceeds the measurable range of the sensor. This can cause the sensor output signal to achieve a limit, e.g., stop responding proportionally to changes in the input. If an event produces more energy in a unit time than the sensor can measure in the unit time, the output signal shape may become saturated. A saturated signal shape can include a plateau at a maximum measurable level for a duration in the signal shape. For example, if a lightning event saturates a sensor, the signal shape may include a rapid increasing slope until the sensor is saturated and the output signal shape is a plateau. The signal shape may remain at a plateau for a duration. The signal shape may include a decreasing slope to a lower value following the saturation plateau.

The flow 100 includes extracting 120 features from both the infrared information channels and the acoustic information channels. In some embodiments, the features include an infrared information channel signal shape 122 and an acoustic channel signal shape 124. The signal shapes can be associated with a peak detected in the infrared information and the acoustic information. In some embodiments, the extracting features includes searching for peak amplitudes in the infrared information and in the acoustic information. In some embodiments, the peak amplitudes 126 include positive peaks and negative peaks. The extracted features can include additional time domain features such as slopes between positive and negative peak amplitude, slopes surrounding the one or more peaks, e.g., slopes on either side of a positive peak or a negative peak, ratios of the amplitudes of positive and negative peaks, a curvature feature (e.g., a feature representing a change in slope in a signal), one or more peak widths, an amount of energy before, during, or after a potential gunshot event, time delays between an infrared peak and an acoustic peak, squared values of the signal features, and earth mover distances (EMDs) to each of two class-average acoustic waveforms. In some embodiments, the frequency domain features can include a Fast Fourier Transform of samples around an acoustic peak, logarithms of a Fast Fourier Transform of samples around an acoustic peak, and so on. The extracted features can be extracted from acoustic signals, IR signals, or both.

In some examples, the flow 100 may use the determination whether the signal from the gunshot sensor device 116 satisfies a saturation threshold to determine whether extracting 120 the features uses one or more saturated classifier, or one or more unsaturated classifier. The flow 100 may determine whether the acoustic signal, the IR signal, or both, satisfy the saturation threshold. In some examples, if the flow 100 determines that the acoustic signal satisfies the saturation threshold, the flow 100 may use saturated acoustic classifiers to process the acoustic signal. In some examples, if the flow 100 determines that the IR signal satisfies the saturation threshold, the flow 100 may use saturated IR classifiers to process the IR signal.

The extracted features can include one or more EMD features. An EMD feature can be a feature comparing two distributions. The EMD feature can represent a minimal cost to transform one distribution into another. In some examples, the EMD feature can compare time-frequency representations (e.g., spectrograms) or feature distributions extracted from one or more signals, e.g., audio signals. In some examples, an EMD classifier may compute the EMD of one or more collected signal to a class-average signal representation of a weapon, e.g., a .22 caliber pistol, a .45 caliber pistol, or one or more long gun signal representations.

In some examples, the extracting 120 can extract a saturation feature from the data. The extracting 120 can include determining whether one or more of the extracted features satisfies a saturation threshold. The portion of the signal shape or the one or more of the extracted features can be compared to a peak amplitude threshold, e.g., a peak amplitude threshold representing 100% of a maximum signal from the sensor. If the portion of the signal shape or the one or more of the extracted features satisfies the peak amplitude threshold, the flow 100 can include processing saturated features through at least one saturated classifier. If the portion of the signal shape or the one or more of the extracted features does not satisfy the peak amplitude threshold, the flow 100 can include processing the features through at least one unsaturated classifier.

The flow 100 includes processing the features extracted from the infrared information channels through a flash detection classifier 130 and the features extracted from the acoustic information channels through a muzzle detection classifier 140. A classifier is created from an N-dimensional feature space, where the N-dimensional feature space results from a feature vector created from the features extracted from the infrared information and the acoustic information. In essence, the N-dimensional feature space is a set of multi-dimensional “fingerprints” of infrared and acoustic datapoints, with each datapoint fingerprint being related to a specific type or class of gunshot event. There are hundreds of different types of gun ammunition and thousands of types of guns. While both guns and ammunition can be broadly grouped into shotgun, handgun, and rifle categories, the number of combinations of the two is vast and ever changing. Thus, building a comprehensive library or datastore of infrared and acoustic features associated with each combination of firearm and ammunition is a challenging endeavor. Machine learning models, which can include flash detection and muzzle detection classifiers, are well suited to building and maintaining such datastores, including the additional aspects of indoor and outdoor environments, weather conditions, multiple shot scenarios, and so on. The infrared and acoustic classifiers can be trained with libraries of data collected from firearm and ammunition manufacturers, government, and private sources. The classifiers can be applied to the infrared information and to the acoustic information in series or in parallel. The classifiers can be used to improve classification of infrared data and acoustic data into one of two classes. The two classes can include a gunshot event class and a false event class. The two or more classifiers can be used when a single classifier does not result in sufficiently accurate classification on the data. The classifiers can be trained using actual gunshot event data and actual false alert event data. The classifiers can implement one or more non-linear surfaces in the N-dimensional feature space. The nonlinear surfaces can include a subset of surfaces within the N-dimensional feature space occupied by the infrared data and the acoustic data. The one or more non-linear surfaces can be selected. In some embodiments, the one or more non-linear surfaces can be selected through an adversarial process. The adversarial process can be controlled based on one or more of functions, heuristics, procedures, and so on. The adversarial process can be controlled algorithmically. The processing can provide detections of one or more possible gunshot events. The classifiers described herein can be trained on data determined to include saturated-or unsaturated signal shapes. A classifier trained on data determined to include saturated signal shapes can be a saturated classifier. A classifier trained on data determined to include unsaturated signal shapes can be an unsaturated classifier.

The flow 100 can include processing the features extracted from the acoustic information channels through a shock detection classifier 142. A shock wave from a gunshot event can have a very different acoustic waveform and signature from a muzzle wave. Having separate classifiers allows the different waveforms and signatures associated with shock waves and muzzle waves to be more accurately characterized before being forwarded to the solvers (described below). Some embodiments include processing the features extracted from the acoustic information channels through a muzzle detection classifier to augment the forwarding.

The flow 100 can include processing the features extracted from one or both of the information channels through a backfire classifier. A vehicle backfire can produce acoustic or infrared information that can be similar to a gunshot event. This can include bursts of acoustic information, flashes of infrared information, or both. A backfire classifier can distinguish the presence of a vehicle backfire in the information received from the acoustic information channel or the infrared information channel. Some embodiments include processing the features extracted from the acoustic information channels or the infrared information channels, or both, through a backfire classifier to augment the forwarding.

The flow 100 can include processing the features extracted from one or both of the information channels through a fireworks classifier. Fireworks can produce acoustic or infrared information that can be similar to a gunshot event. This can include bursts of acoustic information, flashes of infrared information, or both. A trained fireworks classifier can distinguish the presence of fireworks features in the information received from the acoustic information channel or the infrared information channel. Some embodiments include processing the features extracted from the acoustic information channels or the infrared information channels, or both, through a fireworks classifier to augment the forwarding.

The flow 100 can include processing the features extracted from one or both of the information channels through a sailboat classifier. In some examples, an event which produces high energy in a short time can cause a sensor signal shape to include a ‘sailboat shape.’ The sailboat shape of the detected signal can resemble a sailboat, e.g., due to aliasing or sampling artifacts. The sailboat shape of the detected signal can include a drop in signal amplitude, an increasing slope to a peak amplitude, and a decreasing slope which returns to the original signal amplitude. The sailboat classifier can determine the presence of a sailboat feature in the signal shape from the sensor information. The sailboat classifier can by an unsaturated classifier, and use unsaturated features in determining the presence of the sailboat features.

The flow 100 can include processing the features extracted from one or both of the information channels through a device impact classifier. In some examples, a device including an infrared sensor and an acoustic sensor can experience an impact. For instance, the device may experience an impact with an object, such as a person, an animal, or a thrown object. The device experiencing the impact with the object may cause a peak feature in the infrared data, the acoustic data, or both. The peak feature in the infrared data and the peak in the acoustic data may be substantially simultaneous in an impact event for the device.

The device impact classifier can determine whether an infrared peak feature in the infrared data and an acoustic peak feature in the acoustic data satisfies a timing threshold. The timing threshold may be a minimum distance between the acoustic peak feature and the infrared peak feature. In response to determining that the distance between the acoustic peak feature and the infrared peak feature does not satisfy the timing threshold, the system determines to not flag a possible gunshot event as the distance between the acoustic peak feature and the infrared peak feature is not sufficiently large. In some examples, the system performs an action in response to determining that the distance between the acoustic peak feature and the infrared peak feature does not satisfy the timing threshold, such as sending a notification of an impact event. In some examples, the system determines that the distance between the acoustic peak feature and the infrared peak feature does satisfy the timing distance threshold and the system determines that a possible gunshot event has occurred.

The flow 100 can include processing the features extracted from one or both of the information channels through an echo classifier. An event which produces high energy can generate reflections from reflective surfaces. An IR event can reflect from IR-reflective surfaces (e.g., glass surfaces), or an acoustic event can reflect from acoustically-reflective surfaces (e.g., hard surfaces). In some examples, the waveform of an echo in the acoustic signal or the infrared signal can have a similar shape to the waveform of the gunshot event. The echo classifier may compare features extracted from a possible gunshot event to feature extracted from a possible echo event. If the features satisfy an echo threshold, the flow 100 may determine that the possible echo event is a verified echo event. The flow 100 can determine whether a possible gunshot event may be an echo event using the extracted features and an echo classifier. In some examples, the echo classifier can by an unsaturated classifier, and use unsaturated features in determining the presence of the echo features.

The classifiers can determine a detection of a possible gunshot event or a possible false event. The classifiers can produce a detection of one or more possible gunshot event, or a detection of one or more possible false events. In some examples, one or more classifier may produce a detection of one or more possible gunshot event and one or more classifier may produce a detection of one or more possible false events from the same features. In some examples, the flow 100 includes not forwarding 150 the detection of the one or more possible false events to the at least two customized solvers.

In some examples, the classifier produces a detection score using the one or more features provided to the classifier. To determine a detection of a possible gunshot event, the classifier may include a detection score threshold. The classifier can determine whether the detection score satisfies the detection score threshold. If the detection score satisfies the detection score threshold, the classifier can determine a detection of the possible gunshot event. If the detection score does not satisfy the detection score threshold, the classifier can determine a detection of a false event.

In some examples, two or more classifiers can produce different detections when provided the same features. For example, one or more classifiers can determine a detection of a possible gunshot event and one or more classifiers, different from the classifiers which produced the detection of the possible gunshot event, can produce a possible false event from the same features. In some examples, the flow 100 can include the one or more classifiers determining a detection score, and aggregating the one or more detection scores into an aggregate detection score. The flow 100 can include determining whether the aggregate detection score satisfies the detection score threshold. If the aggregate detection score satisfies the detection score threshold, the classifier can determine a detection of the possible gunshot event. If the aggregate detection score does not satisfy the detection score threshold, the classifier can determine a detection of a false event.

The flow 100 includes forwarding 150 the detections of the one or more possible gunshot events to at least two customized solvers. The primary function of the flash and muzzle classifiers is to isolate possible gunshot events based on analysis of the acoustic and infrared data collected by the sensors included in the gunshot sensor device. As potential gunshot events are isolated, the various features of the infrared and acoustic data are compared to N-dimension feature profiles to determine whether a particular event is a gunshot event. In some embodiments, acoustic and infrared events that are strong matches to existing feature profiles can be augmented with classification data from the feature dataset. All isolated infrared and acoustic events determined to be likely gunshot events can be forwarded to two or more solvers for further analysis and classification. In some embodiments, one of the at least two customized solvers comprises a flash solver 152. In some embodiments, the flash solver selects one or more detections forwarded from the infrared information channels. The flash solver can accept all infrared flash detections from every infrared information channel. This can allow the flash solver to select the best infrared event or group of events to be analyzed. It can allow oldest and/or least likely detection events to be aged out from further analysis.

In some embodiments, one of the at least two customized solvers comprises a muzzle solver 154. In some embodiments, the muzzle solver selects one or more detections forwarded from the acoustic information channels. As with the flash solver, the muzzle solver can accept all acoustic muzzle detections from every acoustic information channel. This can allow the muzzle solver to select the best acoustic event or group of events to be analyzed. It can allow oldest and/or least likely detection events to be aged out from further analysis. In some embodiments, one of the at least two customized solvers comprises a shock solver 156. In some embodiments, the shock solver selects 160 at least two detections processed from the acoustic information channels. The shock solver can accept all acoustic shock detections from every acoustic information channel. This can allow the shock solver to select the best acoustic event or group of events to be analyzed. It can allow oldest and/or least likely detection events to be aged out from further analysis.

The flow 100 includes validating 170 at least one possible gunshot event, based on output from the at least two customized solvers. Processing of possible gunshot events from the flash solvers, muzzle solvers, and shock solvers can be combined to determine which events are gunshot events and which are not. A typical firearm discharge involves several phenomena. There is a muzzle flash, which emits visible and infrared light. A muzzle flash typically comprises multiple phases, which include a primary flash, an intermediate flash, and a secondary flash. The primary flash results as hot, highly compressed gases, which are the unburned propellants, exit the barrel along with the projectile. These gases expand rapidly, initially not mixing with the atmosphere, then rebound. This creates a high-temperature region that produces the intermediate flash. As oxygen mixes with the unburned propellants, ignition occurs, causing the secondary flash. A majority of the radiated energy corresponding to the muzzle flash occurs during the secondary flash, and most of the total radiated energy is in the infrared (IR) region. The flash solver can analyze the infrared flash data produced by various events captured by the infrared sensors, and can validate whether the collected event data is associated with a gunshot.

A shock wave following a gunshot event, which is caused by the supersonic travel of the expelled projectile/bullet, can occur. A supersonic bullet causes a distinctive shock wave pattern as it moves through the air. The shock wave expands as a cone behind the bullet, with the wave front propagating outward at the speed of sound. As the shock wave propagates, the nonlinear behavior of the air causes the pressure disturbance to form an “N” shape with a rapid onset, a ramp down to the minimum pressure, and then an abrupt offset. The shock solver can analyze the acoustic shockwave data produced by various events captured by the acoustic sensors, and can validate whether the collected event data is associated with a gunshot. Furthermore, a muzzle blast, which is the sound caused by the discharge of the firearm, can occur. A typical firearm uses a confined explosive charge to push the bullet or projectile down and out of the rifled gun barrel. The hot, rapidly expanding gases cause a muzzle blast to emerge from the barrel. The acoustic disturbance can have a duration in the range of 3 to 5 milliseconds and can propagate through the air at the speed of sound. The muzzle solver can analyze the acoustic data produced by various events captured by the acoustic sensors, and can validate whether the collected event data is associated with a gunshot. Rather than rely on one type of solver data, combining detection analyses from multiple solvers, flash, shock, and muzzle, can significantly improve the gunshot event validation process.

Various steps in the flow 100 may be changed in order, repeated, omitted, or the like without departing from the disclosed concepts. Various embodiments of the flow 100 can be included in a computer program product embodied in a non-transitory computer readable medium that includes code executable by one or more processors. Various embodiments of the flow 100, or portions thereof, can be included on a semiconductor chip and implemented in special purpose logic, programmable logic, and so on.

FIG. 2 is a flow diagram for multichannel analysis of gunshot sensor information. As mentioned above and throughout, a typical firearm discharge involves several phenomena. There is a muzzle flash, which emits visible and infrared light. A shock wave can follow, caused by the supersonic travel of the expelled projectile/bullet. Also, a muzzle blast, which is the sound caused by the discharge of the firearm, can occur. Thus, detecting IR energy, shock waves, and muzzle blasts are important aspects of gunshot detection. Multiple acoustic and infrared sensors that can be included in a gunshot sensor device can collect information generated by possible gunshot events. The gunshot sensor device can include two or more acoustic sensors and two or more infrared sensors. Each acoustic sensor and each infrared sensor can provide a separate information channel. The information channels can be used to process and analyze acoustic and infrared events collected by the acoustic and infrared sensors. In some embodiments, the two or more IR sensors and the two or more acoustic sensors can be paired within the gunshot sensor device.

The flow 200 comprises collecting 214 additional infrared information and additional acoustic information. Gunshot events can occur in rapid succession. More than one person can be involved so that multiple guns can be firing within a particular span of time. Automatic or semi-automatic guns can be used, so that many bullets can be fired from the same gun in a relatively short amount of time. A single shooter can fire a weapon multiple times. Acoustic and infrared events generated by gunfire can be echoed by walls, rocks, buildings, and other objects within an environment, and so on. Two or more infrared sensors can collect infrared information from each of the gunshot events encountered by the sensors. Two or more acoustic sensors can collect acoustic information from each of the gunshot events encountered by the sensors. In some embodiments, the flow 200 further comprises extracting additional features from each of the infrared information channels and each of the acoustic information channels. As mentioned above and throughout, acoustic and infrared features of each gunshot event can be extracted from the information collected by each sensor. Each sensor can provide acoustic or infrared information via acoustic or infrared information channels. In some embodiments, the flow 200 further comprises processing the additional features to provide detections of one or more additional possible gunshot events. The features extracted from the infrared information channels can be processed through a flash detection classifier and the features extracted from the acoustic information channels can be processed through a muzzle detection classifier. The processing provides detections of one or more possible gunshot events. The classifiers can be used to group infrared data and acoustic data into one of two classes. The two classes can include a gunshot event class and a false event class. The detections of the possible gunshot events can be forwarded to at least two customized solvers.

The flow 200 comprises analyzing 210 the output from the at least two customized solvers in a multi-sensor integration classifier. In some embodiments, the multi-sensor integration classifier analyzes a plurality 212 of gunshot events. Infrared and acoustic data collected by the infrared and acoustic sensors as part of a first possible gunshot event, as well as additional gunshot events, can be processed by flash detection classifiers and muzzle detection classifiers. Features identified by the classifiers from the plurality of gunshot events can then be forwarded to the solvers. In some embodiments, one of the at least two customized solvers comprises a flash solver. In some embodiments, the flash solver selects one or more detections forwarded from the infrared information channels. The flash solver can accept all infrared flash detections from every infrared information channel. This can allow the flash solver to select the best infrared event or group of events to be analyzed. It can allow oldest and/or least likely detection events to be aged out from further analysis. In some embodiments, one of the at least two customized solvers comprises a muzzle solver. In some embodiments, the muzzle solver selects one or more detections forwarded from the acoustic information channels. The muzzle solver can accept all acoustic muzzle detections from every acoustic information channel. This can allow the muzzle solver to select the best acoustic event or group of events to be analyzed. It can allow oldest and/or least likely detection events to be aged out from further analysis. In some embodiments, one of the at least two customized solvers comprises a shock solver. In some embodiments, the shock solver selects at least two detections processed from the acoustic information channels. The shock solver can accept all acoustic shock detections from every acoustic information channel. This can allow the shock solver to select the best acoustic event or group of events to be analyzed. It can allow oldest and/or least likely detection events to be aged out from further analysis.

The analyses from the customized flash, shock, and muzzle solvers can be combined and processed by the multi-sensor integration classifier. As stated above and throughout, gunshot events generate IR and acoustic information. In general, IR information reaches the IR sensors before shock or muzzle acoustic information reaches the acoustic sensors. The multi-sensor integration classifier can use solver data from the combination of flash, shock, and muzzle solvers across a time domain to determine whether a gunshot event has occurred. For example, a rifle fired from a hundred feet in front of a gunshot sensor device can first generate infrared information that is collected by the infrared sensors, then shock information generated as a supersonic rifle bullet passes through the air near the acoustic sensors. Following the infrared and shock information, muzzle blast acoustic information can be collected by the acoustic sensors. The wave patterns from the infrared and acoustic data can be analyzed and the combination of features of the data can be processed. The flash solver can indicate that the infrared features show that a gunshot event has occurred. The shock solver can indicate that acoustic features show that a gunshot event has occurred. The muzzle solver can indicate that acoustic features show that a gunshot event has occurred. The multi-sensor integration classifier can combine the information from all three solvers to validate that a gunshot event has indeed occurred.

The flow 200 includes validating at least one possible gunshot event, based on output from the at least two customized solvers. In some embodiments, the validating 220 is further based on an output from the multi-sensor integration classifier. In some embodiments, the flow 200 further comprises validating at least one of the one or more additional possible gunshot events. A plurality of gunshot events can be analyzed and processed by the detectors, classifiers, and solvers to validate one or more gunshot events. In some embodiments, the validating provides a gunshot event confirmation. In some embodiments, the validating provides a false alert indication. Gunshot events can generate IR light reflections and acoustic echoes which can be collected by the infrared and acoustic sensors included in the gunshot sensor device. The reflected infrared light and acoustic echoes can be analyzed by the detectors, solvers, and classifiers to differentiate between an additional gunshot event and echoes or reflections. In some embodiments, the flow 200 further comprises determining 230 a weapon type, based on an output of the multi-sensor integration classifier. In some embodiments, the analyzing determines a gunshot event range and a gunshot event direction. The flow 200 can include determining 230 the weapon type using a weapon classifier. The weapon classifier can process at least one feature from the infrared sensor, the acoustic sensor, or both, in determining a weapon type of the gunshot event. The weapon type can include a caliber, a round type, a handgun type, a rifle type, a shotgun type, or a combination of these.

In some embodiments, the flow 200 further comprises aging out older detections by at least one of the at least two customized solvers. In situations where a shooter is moving through an environment, the most recent gunshot event information can be more highly prioritized for analysis than the first gunshot event information collected by the infrared and acoustic sensors. Knowing where a shooter is located in real time can be prioritized above where the shooter was a few moments ago.

In some embodiments, the flow 200 further comprises reporting 240 the validation to an external system. The gunshot sensor device can report information upstream to a monitoring system, emergency warning system, or another suitable system. In some configurations, the gunshot sensing devices can include one or more microphones and one or more video cameras. In some embodiments, a video management system can be integrated and enabled to provide various notification capabilities to law enforcement, emergency services, the military, or other authorities and organizations. In some embodiments, the video camera and microphone are only activated after a gunshot sensing device detects a possible gunshot. Thus, if a gunshot is detected in the indoor environment, the microphone and video camera can be activated in an attempt to record audio and video of the scene, and possibly to record audio and video of the suspect(s) who discharged the firearm. In such embodiments, the gunshot sensing device enters a surveillance mode once a gunshot has been detected, and the gain of the microphone circuit can be adjusted to pick up additional sounds such as speech. The recorded audio and video information can be of tactical importance to law enforcement or the military so that trained personnel can better understand the extent of injuries and casualties and can assess the number of shooters on the scene. In some embodiments, the flow 200 further comprises reporting the weapon type to an external system.

Various steps in the flow 200 may be changed in order, repeated, omitted, or the like without departing from the disclosed concepts. Various embodiments of the flow 200 can be included in a computer program product embodied in a non-transitory computer readable medium that includes code executable by one or more processors. Various embodiments of the flow 200, or portions thereof, can be included on a semiconductor chip and implemented in special purpose logic, programmable logic, and so on.

FIG. 3 is a system block diagram for a gunshot sensor device with multichannel analysis. The gunshot sensor device can be placed at an indoor location such as a school, at an outdoor location such as a stadium, in public spaces such as parks, etc. The gunshot sensor device can enable gunshot detection using multi-channel analysis. Infrared information and acoustic information are collected using a gunshot sensor device. The collecting infrared information is performed by two or more infrared sensors in a gunshot sensor device. The collecting acoustic information is performed by two or more acoustic sensors in gunshot sensor device. Features are extracted from the infrared information and from the acoustic information. A gunshot event is identified.

The block diagram 300 includes a gunshot sensor device unit 310. The unit can contain components associated with the gunshot sensor device. The gunshot sensor device unit can comprise various materials and can provide protection against environmental hazards such as dust, dirt, precipitation, extreme temperatures, and the like. The unit can further provide protection against vandalism, tampering, etc. One or more components within the unit can be cooled. The block diagram 300 can include two or more infrared (IR) sensor 320. The IR sensors can be used to detect IR light in various IR bands. The IR sensors can detect IR light within the near-infrared (NIR) light band. The near-infrared (NIR) sensor can collect near-infrared information within an environment such as an indoor environment or an outdoor environment. The near-infrared sensor can detect near-infrared band light with wavelengths between 0.78 μm and 3 μm. The two or more infrared sensors can be oriented to cover different fields of view, overlapping fields of view, redundant fields of view, and so on. The near-infrared sensor can sense possible gunshot occurrences such as muzzle flashes. The two or more infrared sensors can provide separate infrared information channels 324. As an infrared sensor detects light and gunshot residue events from a gunshot, the infrared information can be collected and forwarded for processing. Each infrared sensor can forward its data through a separate information channel to be processed.

The infrared data can be forwarded to an information collector 330. The information collector can set up and configure the infrared sensors, test the sensors, provide power control to the sensors, operate the sensors such that NIR information can be collected from the sensors, and so on. The information collector can collect IR information continuously, periodically, based on a potential gunshot event, etc. The information collector is coupled to the classifiers and solvers. The block diagram 300 can include two or more acoustic sensors 322. The two or more acoustic sensors can detect acoustic signals based on one or more ranges of sound pressure levels (SPLs). The acoustic sensor can detect high SPL acoustic signals such as those associated with a gunshot sound event. The collecting acoustic information can be performed by a plurality of acoustic sensors that can be collocated in a single gunshot sensing device. The two or more acoustic sensors can provide separate acoustic information channels 326. As an acoustic sensor detects supersonic and subsonic events from a gunshot, the acoustic information can be collected and forwarded for processing. Each acoustic sensor can forward its data through a separate information channel to be processed. As mentioned above, the acoustic data can be forwarded to an information collector 330. The information collector can set up and configure the acoustic sensors, test the sensors, provide power control to the sensors, operate the sensors such that acoustic information can be collected from the sensors, and so on. The information collector can collect acoustic information continuously, periodically, based on a potential gunshot event, etc. The information collector is coupled to the classifiers and solvers.

The block diagram 300 includes classifiers and solvers 340 for the IR and acoustic information collected by the IR and acoustic sensors. The classifiers and solvers can include a feature extraction element 342 for IR information. The IR feature extraction element can extract features in proximity to the peak amplitudes. Proximity to the peak amplitudes can be based on an amount of time. The proximity can begin at least 20 ms before a peak amplitude. Other amounts of time associated with the peak amplitude can describe proximity to the peak. In some embodiments, the proximity extends at least 100 ms after a peak amplitude. The features that can be extracted from the IR signal can include time domain features. In some embodiments, the time domain features can include positive and negative peak amplitudes, slopes between positive and negative peak amplitudes, ratios of the amplitudes of positive and negative peaks, time delays between an infrared peak and a collected acoustic peak, and squared values of the signal features.

The classifiers and solvers can include a feature extraction element 342 for acoustic information. The multiple elements can be associated with an object such as a data object. The multiple elements can include the extracted features from the acoustic information. The features can be adjusted, normalized, offset, and so on. The features can include scaled features. The scaling of the features can include a reduction in amplitude by an amount or percentage, a compression of dynamic range of the features, etc. The scaling of the features can improve processing performance, can avoid saturating computations by one or more support vector machines (SVMs), etc. At least one feature vector can include an N-dimensional feature space. The N-dimensional space can be based on a number of acoustic features.

The block diagram 300 includes one or more classify elements 344. As mentioned above and throughout, a classifier is created from an N-dimensional feature space, where the N-dimensional feature space results from a feature vector created from the features extracted from the infrared information and the acoustic information. A gunshot class can be computed based on output from a trained support vector machine (SVM) and additional output from the support vector machine. The extracted IR features and the extracted acoustic features can be processed through support vector machines. The SVMs are trained to implement classifiers, where the classifiers are used to compute a gunshot class for the IR and acoustic events detected by a gunshot sensor device. A first gunshot class includes a true gunshot event, while a second gunshot class includes a negative gunshot or false gunshot event. By correctly classifying true gunshot events and false events, the false events can be rejected as false, and no law enforcement or other action need be initiated.

The classifiers and solvers include one or more solver elements 346. The solver elements can include two or more flash, shock, and muzzle solvers. The solvers work together in a multi-sensor integration classifier to pair infrared and acoustic events with one another based on time and wave pattern analysis. In some embodiments, flash, shock, and muzzle solver information can be combined so that the three patterns of solver data describe a single event. The combination of flash data patterns, shock data patterns, and muzzle data patterns can be compared to data from a training datastore 370 to determine whether an event can be confirmed as a gunshot event 350 or as a false alert indication 352. The training datastore comprises data that has been marked or labeled to indicate one or more expected inferences about that data. Data classifications can include temporal and spectral features associated with infrared information and acoustic information. Time domain features can include positive and negative peak amplitudes, slopes between positive and negative peak amplitude, ratios of the amplitudes of positive and negative peaks, time delays between an infrared peak and an acoustic peak, squared values of the signal features, and earth mover distances to each of two class-average acoustic waveforms. In some embodiments, the frequency domain features can include a Fast Fourier Transform of samples around an acoustic peak, logarithms of a Fast Fourier Transform of samples around an acoustic peak, and so on.

The block diagram 300 can include a communications device 360. The communications device can include a wired interface, a hybrid interface (e.g., wired and wireless), a wireless interface, and so on. The wireless interface can enable cable-free communication to one or more additional gunshot detection systems, to communication equipment, to a network such as a computer network or a cellular telephony network, and so on. The wireless interface can communicate using one or more wireless communication techniques including Wi-Fi, Bluetooth, Zigbee, near-field communication (NFC), and so on. The wireless interface may use a low power communication technique to reduce power consumption, to evade detection, to avoid interference with other wireless systems and services, etc. The cable-free communication to a network can include one or more power communication relay devices. A power communication relay device can detect a signal and retransmit the signal. The communication relay devices, or “repeaters”, can repeat a signal from the wireless interface to extend range, to transfer a signal from one wireless interface to another wireless interface, to convert the wireless signal to a wired signal, and the like. The block diagram 300 can include a power source 380. The power source can be distinct from the gunshot sensor device or can be integrated into, next to, or nearby the gunshot sensor device. The power source can include grid power, renewable power, battery power, and so on. A battery can include a rechargeable battery, a non-rechargeable battery, a single-use battery, and so on. A rechargeable battery may be recharged using a solar cell, a trickle charger, or the like. The battery can occupy less than 200 cubic centimeters of volume. The battery can include a nickel-cadmium battery, a sealed lead acid battery, a lithium iron phosphate battery, etc. Further embodiments can include powering the gunshot sensor device using a solar panel coupled to the single gunshot sensor device unit. The solar panel can be used to recharge the battery which powers the gunshot detector.

The block diagram 300 can include an external system 362. The external system can include a video monitoring device. The video monitoring device can include one or more cameras, where the cameras can include webcams, video cameras, and so on. The cameras can include cameras that detect various light wavelength bands such as visible light bands, infrared light bands, and the like. Detection of a gunshot event can initiate communications with various services such as law enforcement, the military, first responders, and the like. In some embodiments, the external system can provide data to a gunshot detection gateway. The notifying can include sending data as a stream or as packets, sending an email, sending a short message service (SMS) text, and the like.

FIG. 4 is a block diagram for IR and acoustic gunshot detection. The block diagram 400 includes two or more infrared sensors 410. The infrared sensors can be included in a gunshot sensor device. The two or more infrared sensors can collect infrared information and provide infrared information channels. The infrared sensors can include an analog-to-digital converter (ADC) which can convert the infrared analog signals to a digital form that can be processed by a microprocessor. In some embodiments, the ADC processor can be external to the infrared sensors. The block diagram 400 can include signal conditioning 420 for the infrared sensors. Signal conditioning can include amplification so increase the signal amplitude, especially for weak infrared signals; low-pass and high-pass filtering to remove noise and interference; offset removal to block DC components or actively subtract DC offsets, and so on. The block diagram 400 can include adaptive lighting cancel 430. Adaptive lighting cancel is a feature used in some infrared (IR) sensors to improve image quality in situations with high ambient light levels, such as headlights or streetlights. It works by automatically adjusting the intensity of the IR illuminators based on the amount of ambient light detected.

The block diagram 400 includes extracting features from the infrared information channels and processing the features extracted from the infrared information channels through a flash detection classifier 440. IR information such as near-IR information can be detected from naturally occurring sources, manmade sources, and so on. The near-IR information can detect motion. The processing can be accomplished using a trained support vector machine (SVM) classifier. The SVM classifier can be used to detect signal features in the IR information that can be associated with non-gunshot events. Noise peak (noise/pk) detection can be used to identify and locate significant peaks in an IR signal that are likely caused by noise or interference rather than the underlying signal of interest. Non-gunshot infrared noise events can include slow risetime or falltime events. The slow risetime or falltime events can be differentiated from fast risetime or falltime events associated with gunshots. Other signal features can be associated with non-gunshot events. The non-gunshot infrared noise events can include low amplitude events. Other aspects of the IR information associated with non-gunshot events can be handled by preprocessing the IR information. The non-gunshot infrared noise events can include non-stationary events. In some embodiments, the flash detection classifier can provide detection of one or more possible gunshot events.

The block diagram 400 includes two or more acoustic sensors 450. The acoustic sensors can be included in a gunshot sensor device. The two or more acoustic sensors can collect acoustic information and provide acoustic information channels. The acoustic sensors can include an analog-to-digital converter (ADC) which can convert the acoustic analog signals to a digital form that can be processed by a microprocessor. In some embodiments, the ADC processor can be external to the acoustic sensors. The block diagram 400 can include signal conditioning 460 for the acoustic sensors. Signal conditioning can include amplification so increase the signal amplitude, especially for weak acoustic signals; low-pass and high-pass filtering to remove noise and interference; offset removal to block DC components or actively subtract DC offsets, and so on.

The block diagram 400 includes extracting features from the acoustic information channels and processing the features extracted from the acoustic information channels through a shock detection classifier 470. A supersonic bullet causes a distinctive shock wave pattern as it moves through the air. The shock wave expands as a cone behind the bullet, with the wave front propagating outward at the speed of sound. As the shock wave propagates, the nonlinear behavior of the air causes the pressure disturbance to form an “N” shape with a rapid onset, a ramp down to the minimum pressure, and then an abrupt offset. Most common events, whether within an indoor or outdoor environment, do not cause a shock wave. For example, a door slamming, while perceived as loud to someone within its hearing, does not cause a shock wave such as that which would result from supersonic travel of a projectile. The processing can be accomplished using a trained support vector machine (SVM) classifier. The SVM classifier can be used to detect signal features in the acoustic information that can be associated with gunshot and non-gunshot events. Noise peak (noise/pk) detection can be used to identify and locate significant peaks in an acoustic signal that are likely caused by noise or interference rather than the underlying signal of interest. Non-gunshot acoustic noise events can include slow risetime or falltime events. The slow risetime or falltime events can be differentiated from fast risetime or falltime events associated with gunshots. Other signal features can be associated with non-gunshot events. The non-gunshot acoustic noise events can include low amplitude events. The non-gunshot acoustic noise events can include non-stationary events. In some embodiments, the shock detection classifier can provide detection of one or more possible gunshot events. The block diagram 400 can include post detection threshold updates 480 to one or more sensors included in the gunshot sensing device. A shock wave can comprise a higher intensity acoustic event, which could need to have a different level of attenuation in order to be processed effectively. For example, a muzzle detection peak can be smaller than a shock detection peak, and thresholds for the classifiers can be adjusted to take that into account.

Furthermore, a gunshot sensing device can be in a standby or low power mode until a potential gunshot sound, a motion, or a combination sound and motion event occurs. In the standby or low power mode, data buffering or storage can still be collecting and storing data in a circular buffer, but the system doesn't need to run any classifiers or solvers. When an event occurs, the data already present in the buffer and any subsequently arriving data can be sent to the classifier and solvers. Data can be held in a buffer such as a circular buffer, in direct memory access (DMA) storage, and so on. An engine such as a DMA engine within a gunshot sensing device can manage the data collection while a processor is in a sleep mode (e.g., the processor is not clocked; only needed peripherals and the DMA engine are clocked). A loud acoustic event that surpasses a minimum threshold sound pressure level (SPL) can occur. The high SPL event can cause the gunshot sensing device to move to a higher power mode. Additional data can then be collected and combined with previously collected, stored, or buffered data.

The IR, such as NIR, information can be processed for a potential gunshot event, where the processing identifies a region of interest within the IR information. Signal features can be extracted from the region of interest. A gunshot event can be identified. The IR information collection can include pre-event, event, and post-event information. The collected information can be buffered or stored. The IR information can be further scaled and vectorized. If a gunshot is detected, an alert can be sent. If the event is determined not to be a gunshot, the gunshot sensing device can return to a low power data collection mode. If motion is not subsequently detected for a predetermined amount of time (e.g., 10 minutes), regardless of whether a loud acoustic event occurred, the low power data collection mode can be exited, and the gunshot sensing device can remain in the ultra-low power motion detection mode. Another motion detection event can cause the sensor to repeat the above-described steps. Additionally, a motion detection alerting function can be enabled such that the sensor can transmit a “motion detected” alert to a receiving server or gateway. This alerting function can operate independently from the gunshot detection functionality.

The block diagram 400 includes extracting features from the acoustic information channels and processing the features extracted from the acoustic information channels through a muzzle detection classifier 490. A muzzle blast, which is the sound caused by the discharge of the firearm, can occur. A typical firearm uses a confined explosive charge to push the bullet or projectile down and out of the rifled gun barrel. The hot, rapidly expanding gases cause a muzzle blast to emerge from the barrel. The acoustic disturbance can have a duration in the range of 3 to 5 milliseconds and can propagate through the air at the speed of sound. The muzzle detection acoustic processing can be accomplished using a trained support vector machine (SVM) classifier. The SVM classifier can be used to detect signal features in the acoustic information that can be associated with gunshot and non-gunshot events. Noise peak (noise/pk) detection can be used to identify and locate significant peaks in an acoustic signal that are likely caused by noise or interference rather than the underlying signal of interest. Non-gunshot acoustic noise events can include slow risetime or falltime events. The slow risetime or falltime events can be differentiated from fast risetime or falltime events associated with gunshots. Other signal features can be associated with non-gunshot events. The non-gunshot acoustic noise events can include low amplitude events. The non-gunshot acoustic noise events can include non-stationary events. In some embodiments, the muzzle detection classifier can provide detection of one or more possible gunshot events.

FIG. 5 is a block diagram for gunshot sensor solver fusion. The block diagram 500 includes forwarding the detections of the one or more possible gunshot events to at least two customized solvers. In some embodiments, one of the at least two customized solvers comprises a flash solver 510. In some embodiments, the flash solver selects one or more detections forwarded from the infrared information channels. The flash solver can accept infrared flash detections from the two or more infrared information channels. The flash solver can select the best infrared event or group of events to be analyzed. The flash solver can allow oldest and/or least likely detection events to be aged out from further analysis. The flash solver can analyze the infrared flash information produced by various events captured by the infrared sensors, and can validate whether the collected event data is associated with a gunshot.

In some embodiments, one of the at least two customized solvers comprises a shock solver 520. In some embodiments, the shock solver selects at least two detections processed from the acoustic information channels. The shock solver can accept acoustic shock detections from the two or more acoustic information channels. The shock solver can select the best acoustic event or group of events to be analyzed. The shock solver can allow oldest and/or least likely detection events to be aged out from further analysis. The shock solver can analyze the acoustic shock information produced by various events captured by the acoustic sensors, and can validate whether the collected event data is associated with a gunshot.

In some embodiments, one of the at least two customized solvers comprises a muzzle solver 530. In some embodiments, the muzzle solver selects one or more detections forwarded from the acoustic information channels. The muzzle solver can select the best acoustic event or group of events to be analyzed. The muzzle solver can allow oldest and/or least likely detection events to be aged out from further analysis. The muzzle solver can analyze the acoustic muzzle information produced by various events captured by the acoustic sensors, and can validate whether the collected event data is associated with a gunshot.

The block diagram 500 further comprises analyzing the output from the at least two customized solvers in a multi-sensor integration classifier 540. In some embodiments, the multi-sensor integration classifier analyzes a plurality of gunshot event detections. The analyses from the customized flash, shock, and muzzle solvers can be combined and processed by the multi-sensor integration classifier. As stated above and throughout, gunshot events generate IR and acoustic information. The multi-sensor integration classifier can use solver data from the combination of flash, shock, and muzzle solvers across a time domain to determine whether a gunshot event has occurred. The feature information from the infrared and acoustic data can be analyzed and the combination of feature information can be processed. The flash solver can indicate whether the infrared features show that a gunshot event has occurred. The shock solver can indicate whether the acoustic features show that a gunshot event has occurred. The muzzle solver can indicate whether acoustic features show that a gunshot event has occurred. The multi-sensor integration classifier can combine the information from all three solvers to validate whether a gunshot event has indeed occurred. In some embodiments, the analyzing can determine a gunshot event range and a gunshot event direction. The gunshot sensor device can use information collected by the two or more sensors or can measure the angle of infrared light to calculate the distance from the device to the IR light source. In some embodiments, the two or more IR sensors can be arranged in an array to determine the direction of an IR source relative to the gunshot sensor device.

In some embodiments, the multi-sensor integration classifier 540 analysis can validate 550 the detection of at least one possible gunshot event, based on output from the at least two customized solvers. The validating can provide a gunshot event confirmation. The validating can provide a false alert indication. In some embodiments, the multi-sensor integration classifier validation can further comprise determining a weapon type 560. The determining a weapon type can be based on support vector machine (SVM) classifier information. The features of a possible gunshot event can be compared to known infrared flash and acoustic shock and muzzle gunshot event features to determine a weapon type. In some embodiments, an ammunition type can be determined, based on known infrared flash and acoustic shock and muzzle gunshot event features stored in the SVM classifier datastore.

The block diagram 500 can include reporting 570 the validation to an external system. In some embodiments, the reporting can include reporting the weapon type to an external system. The reporting can include the range and direction of the gunshot event. The reporting can include false alert indications. In some embodiments, the false alert indications can include a classification of the false alert, including thunder, lightning, camera flash, fire alarm strobe, door slam, and so on. The gunshot sensor device can report information upstream to a monitoring system, emergency warning system, or another suitable system. In some embodiments, the reporting can be integrated and enabled to provide various notification capabilities to law enforcement, emergency services, the military, or some authorities and organizations.

FIG. 6 is a block diagram for a support vector machine (SVM). A support vector machine (SVM) can be based on an algorithm, such as a machine learning (ML) algorithm. The machine learning algorithm can include a deep learning algorithm. The machine learning algorithm enables a machine learning technique, where the machine learning technique enables gunshot detection. The ML technique is enabled by the SVM. The SVM can include a class of kernel machine, where a kernel machine can be used for pattern analysis. The SVM can be used to process data, including infrared (IR) data such as near-infrared (NIR) data and acoustic data, in order to identify patterns, trends, clusters, and the like within the data. The SVM can be used to classify the data. Classifying the data can include assigning data to one of two classes based on inferences drawn on the data. The inferences that are drawn are based on a machine learning algorithm that has been trained. The inferences associated with the data can include identifying, by the SVM, that the data can be a member of one class such as a first class, or a member of another class such as a second class. The SVM can be executed on one or more processors, a network such as a neural network, one or more processor cores, and so on. The support vector machine enables gunshot detection with false alert rejection. Infrared information is collected using a gunshot sensor device, wherein the collecting infrared information is performed by at least one infrared sensor in a gunshot sensor device unit. Acoustic information is collected using the gunshot sensor device, wherein the collecting acoustic information is performed by at least one acoustic sensor collocated with the infrared sensor in the gunshot sensor device unit. Features are extracted from the infrared information and the acoustic information, wherein at least one of the features describes a first signal shape for the infrared information and at least one of the features describes a second signal shape for the acoustic information. The features extracted from the infrared information are processed through one or more trained support vector machines. The features extracted from the acoustic information are processed through at least one support vector machine. A gunshot class is computed, based on output from the one or more trained support vector machines and additional output from the at least one support vector machine.

The block diagram 600 can include a support vector machine (SVM) 610. The SVM can enable a machine learning algorithm, heuristic, procedure, and so on. The SVM can comprise one or more processors, processor cores, and the like. The processors can be configured in a network. The network can include a configurable network. In some embodiments, the SVM can be implemented using code. The code can include code generated by a high-level compiler such as a C, C++, or Python compiler; a special-purpose compiler; etc. The code can further include an assembly code or similar low-level code. The block diagram 600 can include a controller 620. The controller can be used to obtain and direct data to the SVM, to select and load a trained SVM model, etc. The controller can be used to enable further learning by the trained learning model. The block diagram 600 can include a trained ML model 630. The trained ML model can be based on an algorithm, where the algorithm has been trained by providing a training dataset for processing by the algorithm. The training dataset comprises data that has been marked or labeled to indicate one or more expected inferences about that data. The training dataset can include marked or labeled features, where the features 632 can be extracted from infrared information and from acoustic information.

The training dataset is used to adjust parameters such as weights and biases associated with the trained SVM. The SVM model can further include weights 634 and biases (not shown). The weights and the biases enable the trained ML model to make desired inferences about data. In some embodiments, the inferences can be based on classifications. The classifications can include temporal (time domain) features and spectral (frequency domain) features associated with the infrared information and the acoustic information. In some embodiments, the time domain features can include positive and negative peak amplitudes, slopes between positive and negative peak amplitudes, ratios of the amplitudes of positive and negative peaks, time delays between an infrared peak and an acoustic peak, squared values of the signal features, and earth mover distances to each of two class-average acoustic waveforms. In some embodiments, the frequency domain features can include a Fast Fourier Transform of samples around an acoustic peak, logarithms of a Fast Fourier Transform of samples around an acoustic peak, and so on.

The trained ML model can be executed on the SVM in order to process data. The processing of data can accomplish computing a gunshot class. The block diagram 600 includes data 640. The data includes the collected infrared information and the collected acoustic information. The processing of the data can include detectors 650 and solvers 652. The flash, shock, and muzzle detectors and solvers can analyze the infrared and acoustic data separately, then combine the analyses to determine the presence of a true gunshot event, an absence of a gunshot event or a false event, an alarm, a lightning flash, a reflection off a vehicle, etc. The determining can be based on a positive result or a negative result of the computing a gunshot class. In some embodiments, the determining can include evaluating the product of scaled and vectorized signal features and a weight vector, with an added bias, as a positive result or a negative result. The positive result and the negative result can be used for a variety of purposes. In some embodiments, a positive result can indicate a first classification, and a negative result can indicate a second classification. The first and second classifications can include a presence or confirmation of an event (e.g., a positive result), or an absence of an event (e.g., a negative result). The positive result and the negative result can be used to determine a true gunshot event, to reject a false alert, and so on. The positive result and the negative result can be used to differentiate between types of firearms. In some embodiments, the first classification can be a long gun bullet caliber classification, and the second classification can be a handgun bullet caliber classification. The long gun bullet caliber classification can be associated with a long gun such as a rifle, a shotgun, and so on. The long gun can include a bolt-action, semiautomatic, or automatic weapon, etc. The handgun can include a revolver or semiautomatic weapon, and the like.

FIG. 7 is a flow diagram for machine learning (ML) training for a support vector machine. Discussed previously, a support vector machine (SVM) can be based on a machine learning algorithm that can be used for pattern analysis of data. The pattern analysis can include classification of the data into one of two classes. The SVM must be trained in order for the SVM to accomplish the classification of the data. The training can be based on providing a training dataset, where the training dataset includes “labeled” or “tagged” data. The training dataset can include labeled features. The features are extracted from collected infrared information and from collected acoustic data. A label associated with the data can include a classification. The training data is applied to the SVM, and weights and biases associated with the SVM can be adjusted. The adjustment of the weights and biases associated with the SVM can continue until the SVM correctly classifies the training data as being a member of one of two classes. The adjustments can accomplish faster convergence of the SVM to determine the correct classification of the data. The training the SVM enables gunshot detection with false alert rejection. Infrared information is collected using a gunshot sensor device, wherein the collecting infrared information is performed by at least one infrared sensor in a gunshot sensor device unit. Acoustic information is collected using the gunshot sensor device, wherein the collecting acoustic information is performed by at least one acoustic sensor collocated with the infrared sensor in the gunshot sensor device unit. Features are extracted from the infrared information and the acoustic information, wherein at least one of the features describes a first signal shape for the infrared information and at least one of the features describes a second signal shape for the acoustic information. The features extracted from the infrared information are processed through one or more trained support vector machines. The features extracted from the acoustic information are processed through at least one support vector machine. A gunshot class is computed, based on output from the one or more trained support vector machines and additional output from the at least one support vector machine.

The flow 700 includes obtaining 710 one or more support vector machine (SVM) algorithms. An SVM algorithm can include a machine learning (ML) model. The ML model can be uploaded by a user, downloaded from a repository such as an online library or cloud storage, and so on. The ML model can include a general-purpose model, a previously trained model, and the like. The model can be associated with the Kernel Method for machine learning. The algorithm associated with the Kernel Method can be used for pattern analysis. The flow 700 can include obtaining 712 gunshot training data. The gunshot training data can include marked or labeled training data. The marked training data can include data that has been analyzed by human experts, a previously trained ML model, and so on. In a usage example, a collection of infrared flash information data has been analyzed by human experts. The human experts have classified some of the infrared flash information as containing gunshot flash events and some infrared flash information as containing false gunshot flash events. The human experts have further classified acoustic information and other sound recordings as containing gunshot events and other acoustic information and sound recordings as containing false acoustic events. The false event information can include flashes of sunlight such as reflections off of shiny surfaces, fireworks, strobes, etc. The false event information can further include vehicle backfires, door slams, thunder, earthquakes, etc. The marked training data can be applied to the ML algorithm in order to train or tune the algorithm to correctly classify gunshot events within the marked infrared and acoustic information. The ML algorithm can further classify false gunshot events.

The flow 700 includes running classifier design code 720. The classifier design code can access the training data, where the training data includes gunshot information data collected from firing of handguns, rifles, shotguns, and so on. The gunshot information can be based on one or more calibers such as common handgun calibers ranging from .22 to .45; rifle calibers such as common hunting rifle calibers such as .223, .243, .30-30, .30-06, and 7.62; common shotgun calibers including .410, 20 gauge, 16 gauge, and 12 gauge; and the like. In the flow 700, the classifier design code generates 722 matrices of features. Recall that the features can include infrared features and acoustic features. The features can include frequency domain features and time domain features. The infrared features and the acoustic features can be combined into at least one feature vector. More than one feature can result from combining the IR and the acoustic features. In some embodiments, at least one feature vector can include scaled features. In the flow 700, the running the classifier design code can optimize weights 724. The weights can be associated with a machine learning model, where the machine learning model can be executed on the SVM. The feature vector matrices and the optimized weights can be used to obtain a best overall classification rate. In order to obtain a best overall classification rate, careful selection of training data is performed. In some embodiments, the selecting can be performed by a user. Further techniques can be used to select training data. In some embodiments, the selecting can be performed by a trained ML model. The ML model for selecting can be executed on an SVM, on a neural network, and so on.

Training of machine learning models can be based on a variety of machine learning techniques. The machine learning techniques can include supervised training, unsupervised training, and so on. Supervised learning is based on providing a training dataset that has been marked or labeled. The labeling, which can be accomplished by human experts who have examined the dataset, indicates an inference that the supervised learning model is expected to make based on the dataset. Supervised learning of the SVM is based on applying the marked training data to the ML algorithm. The training accomplishes adjustments to the algorithm so that the algorithm can correctly classify the training data. The classifying can include detecting the presence or absence of a flash such as infrared flash. The adjustments that can be made to the ML algorithm for training purposes can include adjusting weights, applying a bias, and so on. The adjusting of the weights can include varying a weight for each computation associated with the SVM. The adjusting the weights can include an iterative technique where the adjusting the weights can continue throughout the processing of the training dataset. Biases associated with the SVM can be determined and adjusted. The bias can include the distance between a hyperplane separating clusters of points associated with classifications determined by an SVM.

The flow 700 includes supplementing 730 the training data. The marked training data can be supplemented with additional marked training data. The additional marked training data can include data generated by human experts. The additional data can include historical data. The additional marked training data can include synthetic data. The synthetic data can be generated using a generative technique, where the generative technique generates and marks data that can be substantially similar to the marked data generated by the human experts. The generative technique can generate more marked data faster than can the human experts. In the flow 700, the supplementing the training data includes generating false alerts 732. The false alerts, or “negative results,” can be used to improve inference by an ML model that IR information and/or acoustic information is associated with a false gunshot event.

The flow 700 includes updating 740 the one or more SVM models. The updating the SVM models can be based on updating a supervised learning model. The supplemental, marked training data can be applied to the supervised ML model. The supplemental training can be used to reoptimize the ML weights 742. The adjusting can further include adjusting the ML biases. The updating of the learning model can continue while the supplemental training data is available, until the ML model meets or exceeds a classification accuracy threshold, while convergence speed increases, and the like. The flow 700 includes promoting the one or more trained SVMs to production 750. The promoting can indicate that the ML model is ready for processing of “production” or real data. In some embodiments, the trained SVM can continue to learn based on the application of production data. The continued learning can be based on a semi-supervised training technique, an unsupervised training technique, etc.

Various events such as infrared (IR) flash events and acoustic events can be classified. Discussed previously and throughout, the IR events can be associated with lightning flashes, fireworks, strobes such as fire alarm strobes, and so on. The IR events can include flashes of sunlight such as flashes resulting from the sunlight reflecting off of water, vehicle windshields, building windows, mirrors, and other shiny surfaces. The acoustic events can include high sound pressure level (SPL) events such as thunder, vehicle backfires, door slams, heavily amplified music, etc. In some embodiments, the IR flash events and the acoustic events can be associated with gunshot events. The classifying can be used to determine a likelihood that an IR flash event and/or an acoustic event is associated a particular class. In a usage example, an IR flash event and an acoustic event are detected. By processing data associated with the flash event and the acoustic event, a determination can be made as to whether the flash event and the acoustic event may have been caused by a gunshot event or not. If not, then further classifying can be performed in order to determine another possible cause of the event.

Discussed previously, signal features can be extracted from collected infrared information and from collected acoustic information. The signal features surrounding at least one local peak can be extracted. The signal features can include time domain features such as positive peak amplitude, negative peak amplitude that follows an initial positive peak, pre-peak infrared level, and a ratio of a positive peak amplitude to a negative peak amplitude that follows an initial positive peak. The time domain features can include features associated with the peak, such as peak width at percentages showing a positive peak amplitude at 25%, 50%, 75%, and/or 97% values. The time domain features can include energy values associated with the peak. The time domain features associated with energy include peak pulse energy, pre-peak energy, post-peak energy, and/or the ratio of post-peak energy to pre-peak energy. The time domain features can further include rising and falling slopes, rising and falling edge counts, and so on. The features can further include frequency domain features. The frequency domain features can include a Fast Fourier Transform (FFT) of samples surrounding a local peak. The frequency domain features can include logarithms of a Fast Fourier Transform of samples surrounding a local peak. An FFT can be performed on samples surrounding the peak in batches of up to 25 ms. However, using a longer sampling time batch, such as 100 ms, can enable the testing of feature statistics over time. The classifying can be based on the time domain features, the frequency domain features, or both.

FIG. 8 is a system diagram for a system diagram for gunshot detection using multichannel analysis. The system 800 can include one or more processors 810 coupled to a memory 812 which stores instructions. The system 800 can include a display 814 coupled to the one or more processors 810 for displaying data, video streams, videos, intermediate steps, instructions, and so on. In some embodiments, one or more processors 810 are coupled to the memory 812 where the one or more processors, when executing the instructions which are stored, are configured to: collect infrared information using a gunshot sensor device, wherein the collecting infrared information is performed by two or more infrared sensors, and wherein the two or more infrared sensors provide infrared information channels; collect acoustic information using the gunshot sensor device, wherein the collecting acoustic information is performed by two or more acoustic sensors, and wherein the two or more acoustic sensors provide acoustic information channels; extract features from both the infrared information channels and the acoustic information channels; process the features extracted from the infrared information channels through a flash detection classifier and the features extracted from the acoustic information channels through a muzzle detection classifier, wherein the processing provides detections of one or more possible gunshot events; forward the detections of the one or more possible gunshot events to at least two customized solvers; and validate at least one possible gunshot event, based on output from the at least two customized solvers.

The system 800 includes SVM gunshot training data 840. The gunshot training data can include marked or labeled training data. The marked training data can include data that has been analyzed by human experts, a previously trained ML model, and so on. The training data can include a collection of infrared information 820. The infrared information can be analyzed and classified. Some of the infrared information can be classified as containing gunshot flash events and other infrared information as containing false gunshot flash events. The training data can include a collection of acoustic information 830 and other sound recordings containing gunshot events and other acoustic information and sound recordings containing false acoustic events. The false event information can include flashes of sunlight such as reflections off of shiny surfaces, fireworks, strobes, etc. The false event information can further include vehicle backfires, door slams, thunder, earthquakes, etc. The SVM training information can be applied to the ML algorithm in order to train or tune the algorithm to correctly classify gunshot events within the infrared and acoustic information. The ML algorithm can further classify false gunshot events infrared (IR) information.

The system 800 includes a collecting component 850. The collecting component 850 includes functions and instructions for collecting infrared information using a gunshot sensor device, wherein the collecting infrared information is performed by two or more infrared sensors 852, and wherein the two or more infrared sensors provide infrared information channels. The collecting component 850 includes functions and instructions for collecting acoustic information using the gunshot sensor device, wherein the collecting acoustic information is performed by two or more acoustic sensors 854, and wherein the two or more acoustic sensors provide acoustic information channels.

The system 800 includes an extracting component 860. The extracting component 860 includes functions and instructions for extracting features from both the infrared information channels and the acoustic information channels. In some embodiments, the features include an infrared information channel signal shape and an acoustic channel signal shape. The signal shapes can be associated with a peak detected in the infrared information and the acoustic information. In some embodiments, the extracting features includes searching for peak amplitudes in the infrared information and in the acoustic information. In some embodiments, the peak amplitudes include positive peaks and negative peaks. The extracted features can include additional time domain features such as slopes between positive and negative peak amplitude, ratios of the amplitudes of positive and negative peaks, time delays between an infrared peak and an acoustic peak, squared values of the signal features, and earth mover distances to each of two class-average acoustic waveforms. In some embodiments, the frequency domain features can include a Fast Fourier Transform of samples around an acoustic peak, logarithms of a Fast Fourier Transform of samples around an acoustic peak, and so on.

The system 800 includes a processing component 870. The processing component 870 includes functions and instructions for processing the features extracted from the infrared information channels through a flash detection classifier and the features extracted from the acoustic information channels through a muzzle detection classifier, wherein the processing provides detections of one or more possible gunshot events. The infrared and acoustic classifiers can be trained with libraries of data collected from firearm and ammunition manufacturers, government, and private sources. The classifiers can be applied to the infrared information and to the acoustic information in series or in parallel. The classifiers can be used to improve classification of infrared data and acoustic data into one of two classes. The two classes can include a gunshot event class and a false event class. The two or more classifiers can be used when a single classifier does not result in sufficiently accurate classification on the data. The classifiers can be trained using actual gunshot event data and actual false alert event data. The classifiers can implement one or more non-linear surfaces in the N-dimensional feature space. The nonlinear surfaces can include a subset of surfaces within the N-dimensional feature space occupied by the infrared data and the acoustic data. The one or more non-linear surfaces can be selected. The processing can provide detections of one or more possible gunshot events.

The system 800 includes a forwarding component 880. The forwarding component 880 includes functions and instructions for forwarding the detections of the one or more possible gunshot events to at least two customized solvers. In some embodiments, one of the at least two customized solvers comprises a flash solver. In some embodiments, the flash solver selects one or more detections forwarded from the infrared information channels. The flash solver can accept infrared flash detections from the plurality of infrared information channels. This can allow the flash solver to select the best infrared event or group of events to be analyzed. It can allow oldest and/or least likely detection events to be aged out from further analysis. In some embodiments, one of the at least two customized solvers comprises a muzzle solver. In some embodiments, the muzzle solver selects one or more detections forwarded from the acoustic information channels. As with the flash solver, the muzzle solver can accept acoustic muzzle detections from the plurality of acoustic information channels. This can allow the muzzle solver to select the best acoustic event or group of events to be analyzed. It can allow oldest and/or least likely detection events to be aged out from further analysis. In some embodiments, one of the at least two customized solvers comprises a shock solver. In some embodiments, the shock solver selects at least two detections processed from the acoustic information channels. The shock solver can accept all acoustic shock detections from every acoustic information channel. This can allow the shock solver to select the best acoustic event or group of events to be analyzed. It can allow oldest and/or least likely detection events to be aged out from further analysis.

The system 800 includes a validating component 890. The validating component 890 includes functions and instructions for validating at least one possible gunshot event, based on output from the at least two customized solvers. Processing of possible gunshot events from the flash solvers, muzzle solvers, and shock solvers can be combined to determine which events are gunshot events and which are not. The flash solver can analyze the infrared flash data produced by various events captured by the infrared sensors, and can validate whether the collected event data is associated with a gunshot. The shock solver can analyze the acoustic shockwave data produced by various events captured by the acoustic sensors, and can validate whether the collected event data is associated with a gunshot. The muzzle solver can analyze the acoustic data produced by various events captured by the acoustic sensors, and can validate whether the collected event data is associated with a gunshot. Rather than rely on one type of solver data, combining detection analyses from multiple solvers, flash, shock, and muzzle, can significantly improve the gunshot event validation process.

The system 800 can include a computer program product embodied in a non-transitory computer readable medium for gunshot analysis, the computer program product comprising code which causes one or more processors to perform operations of: collecting infrared information using a gunshot sensor device, wherein the collecting infrared information is performed by two or more infrared sensors, and wherein the two or more infrared sensors provide infrared information channels; collecting acoustic information using the gunshot sensor device, wherein the collecting acoustic information is performed by two or more acoustic sensors, and wherein the two or more acoustic sensors provide acoustic information channels; extracting features from both the infrared information channels and the acoustic information channels; processing the features extracted from the infrared information channels through a flash detection classifier and the features extracted from the acoustic information channels through a muzzle detection classifier, wherein the processing provides detections of one or more possible gunshot events; forwarding the detections of the one or more possible gunshot events to at least two customized solvers; and validating at least one possible gunshot event, based on output from the at least two customized solvers.

Each of the above methods may be executed on one or more processors on one or more computer systems. Embodiments may include various forms of distributed computing, client/server computing, and cloud-based computing. Further, it will be understood that the depicted steps or boxes contained in this disclosure's flow diagrams are solely illustrative and explanatory. The steps may be modified, omitted, repeated, or re-ordered without departing from the scope of this disclosure. Further, each step may contain one or more sub-steps. While the foregoing drawings and description set forth functional aspects of the disclosed systems, no particular implementation or arrangement of software and/or hardware should be inferred from these descriptions unless explicitly stated or otherwise clear from the context. All such arrangements of software and/or hardware are intended to fall within the scope of this disclosure.

The block diagrams and flow diagrams illustrations depict methods, apparatus, systems, and computer program products. The elements and combinations of elements in the block diagrams and flow diagrams show functions, steps, or groups of steps of the methods, apparatus, systems, computer program products and/or computer-implemented methods. Any and all such functions—generally referred to herein as a “circuit,” “module,” or “system”—may be implemented by computer program instructions, by special-purpose hardware-based computer systems, by combinations of special purpose hardware and computer instructions, by combinations of general-purpose hardware and computer instructions, and so on.

A programmable apparatus which executes any of the above-mentioned computer program products or computer-implemented methods may include one or more microprocessors, microcontrollers, embedded microcontrollers, programmable digital signal processors, programmable devices, programmable gate arrays, programmable array logic, memory devices, application specific integrated circuits, or the like. Each may be suitably employed or configured to process computer program instructions, execute computer logic, store computer data, and so on.

It will be understood that a computer may include a computer program product from a computer-readable storage medium and that this medium may be internal or external, removable and replaceable, or fixed. In addition, a computer may include a Basic Input/Output System (BIOS), firmware, an operating system, a database, or the like that may include, interface with, or support the software and hardware described herein.

Embodiments of the present invention are limited to neither conventional computer applications nor the programmable apparatus that run them. To illustrate: the embodiments of the presently claimed invention could include an optical computer, quantum computer, analog computer, or the like. A computer program may be loaded onto a computer to produce a particular machine that may perform any and all of the depicted functions. This particular machine provides a means for carrying out any and all of the depicted functions.

Any combination of one or more computer readable media may be utilized including but not limited to: a non-transitory computer readable medium for storage; an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor computer readable storage medium or any suitable combination of the foregoing; a portable computer diskette; a hard disk; a random access memory (RAM); a read-only memory (ROM); an erasable programmable read-only memory (EPROM, Flash, MRAM, FeRAM, or phase change memory); an optical fiber; a portable compact disc; an optical storage device; a magnetic storage device; or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain or store a program for use by or in connection with an instruction execution system, apparatus, or device.

It will be appreciated that computer program instructions may include computer executable code. A variety of languages for expressing computer program instructions may include without limitation C, C++, Java, JavaScript™, ActionScript™, assembly language, Lisp, Perl, Tcl, Python, Ruby, hardware description languages, database programming languages, functional programming languages, imperative programming languages, and so on. In some embodiments, computer program instructions may be stored, compiled, or interpreted to run on a computer, a programmable data processing apparatus, a heterogeneous combination of processors or processor architectures, and so on. Without limitation, embodiments of the present invention may take the form of web-based computer software, which includes client/server software, software-as-a-service, peer-to-peer software, or the like.

In some embodiments, a computer may enable execution of computer program instructions including multiple programs or threads. The multiple programs or threads may be processed approximately simultaneously to enhance utilization of the processor and to facilitate substantially simultaneous functions. By way of implementation, any and all methods, program codes, program instructions, and the like described herein may be implemented in one or more threads which may in turn spawn other threads, which may themselves have priorities associated with them. In some embodiments, a computer may process these threads based on priority or other order.

Unless explicitly stated or otherwise clear from the context, the verbs “execute” and “process” may be used interchangeably to indicate execute, process, interpret, compile, assemble, link, load, or a combination of the foregoing. Therefore, embodiments that execute or process computer program instructions, computer-executable code, or the like may act upon the instructions or code in any and all of the ways described. Further, the method steps shown are intended to include any suitable method of causing one or more parties or entities to perform the steps. The parties performing a step, or portion of a step, need not be located within a particular geographic location or country boundary. For instance, if an entity located within the United States causes a method step, or portion thereof, to be performed outside of the United States, then the method is considered to be performed in the United States by virtue of the causal entity.

While the invention has been disclosed in connection with preferred embodiments shown and described in detail, various modifications and improvements thereon will become apparent to those skilled in the art. Accordingly, the foregoing examples should not limit the spirit and scope of the present invention; rather it should be understood in the broadest sense allowable by law.

FIG. 9 is a flow diagram of a process 900 for gunshot detection using multichannel analysis. For example, the process 900 can be used by the gunshot detection multichannel analysis system monitoring an environment for gunshot events.

A system coupled to one or more sensors monitors an environment for a possible gunshot event. The system accesses data including infrared information, acoustic information, or both that were captured by at least one sensor in the environment. The system can receive one or more channels of information. In some examples, the system may receive more than one channel of infrared information, more than one channel of acoustic information, or a combination of these.

The system extracts, from the data, at least one feature. The system can extract one or more feature from the data including the acoustic information, the infrared information, or both. The one or more feature extracted from the data can include features determined by an extracting component of the system, features predetermined by the system, or both. Examples of the features which can be extracted from the data include a signal feature (e.g., a time domain feature, a frequency domain signal feature, a signal domain feature, or a combination of these). Examples of signal features can include a positive peak amplitude, a negative peak amplitude, a pre-peak feature, a ratio of a positive peak amplitude to a negative peak amplitude feature. The time domain features can include features associated with the peak, energy values associated with the peak, a peak pulse energy feature, a pre-peak energy feature, a post-peak energy feature, a ratio of post-peak energy to pre-peak energy feature, or a combination of these. Frequency domain features can include a Fast Fourier Transform (FFT) feature of samples surrounding a local peak.

The system determines, by a device that includes an infrared sensor and an acoustic sensor, whether infrared data captured by the infrared sensor satisfies a saturation threshold (902). The system extracts one or more features from the sensor signal. A feature can be a saturation feature which can indicate whether a portion of the sensor signal includes a saturation feature, e.g., a plateau at a maximum in the signal shape. The system can determine whether the saturation feature satisfies a saturation threshold indicating that a portion of the signal shape is saturated.

The system selects, from a plurality of classifiers, at least one classifier using a result of the determination whether the infrared data satisfies the saturation threshold (904). In response to determining that the data includes a saturation feature, the system can process the at least one feature through at least one saturated classifier. The system determines, using at least one saturated classifier and the at least one feature, a detection of a possible gunshot event in the environment. In some examples, the system can use at least one classifier disclosed herein that was trained on saturated data as the at least one saturated classifier. Examples include a saturated flash detection classifier, a saturated muzzle detection classifier, or a saturated weapon type classifier.

In response to determining that the data does not includes a saturation feature, or the saturation feature does not satisfy the saturation threshold, the system can process the at least one feature through at least one unsaturated classifier. The system determines, using at least one unsaturated classifier and the at least one feature, a detection of a possible gunshot event in the environment. In some examples, the system can use at least one classifier disclosed herein that was trained on unsaturated data as the at least one unsaturated classifier. Examples include an unsaturated flash detection classifier, an unsaturated muzzle detection classifier, an unsaturated weapon type classifier, or an unsaturated sailboat classifier.

The system accesses at least one feature from one or more of the infrared data captured by the infrared sensor or acoustic data captured by the acoustic sensor (906). The feature can correspond to the data from which the feature is extracted. The feature can be an infrared feature from infrared data or an amplitude feature from acoustic data.

The feature can be an amplitude feature. The feature can be an infrared amplitude feature or an acoustic amplitude feature. The system can determine whether the infrared amplitude feature, or the acoustic amplitude feature, satisfies the saturation threshold. The saturation threshold can be a peak amplitude threshold. In some examples, the feature can be a sailboat feature, e.g., an infrared feature can be the sailboat feature.

The at least one classifier can process the at least one feature. The classifier can process the at least one feature to determine a detection of a possible gunshot event, or a detection of a false event. In response to the classifier determines a detection of a possible gunshot event, the system provides the detection of the possible gunshot event to at least one solver.

The system determines, using the at least one classifier and the at least one feature, whether the at least one feature represents a possible gunshot event in an environment that includes the device (908). The system can include at least one flash solver, muzzle solver, shock solver, or a combination of these. The system can receive an output from the at least one solver to determine which events are gunshot events.

The system performs an action using a result of the determination whether the at least one feature represents a possible gunshot event in the environment (910). The system may report the possible gunshot event to an external system. The external system can be a monitoring system or an emergency warning system. The reporting action can provide at least one notification to one or more of law enforcement, emergency services, the military, or other authorities and organizations.

The order of operations in the process 900 described above is illustrative only, and can be performed in different orders. For example, the system can provide at least one feature to the at least one classifier in parallel. In examples in which the system determines at least one saturated feature and at least one unsaturated feature, the system can provide the at least one saturated feature to at least one saturated classifier, and the at least one unsaturated feature to at least one unsaturated classifier in parallel.

In some implementations, the process 900 can include additional operations, fewer operations, or some of the operations can be divided into multiple operations.

Optionally, the system can determine a first detection of a possible gunshot event from a first channel from at least one sensor and a second detection of a possible gunshot event from a second channel from at least one sensor are the same possible gunshot event. Determining whether the gunshot events from two different channels are the same possible gunshot event can increase gunshot event detection accuracy of the system. The system can determine whether a first timestamp of the first detection of the possible gunshot event and a second timestamp of the second detection of the possible gunshot event satisfy a timing threshold. For example, the system can determine a different between the first timestamp and the second timestamp and determine whether the difference satisfies the timing threshold. If the first timestamp and the second timestamp, or the difference, satisfy the timing threshold, the system can determine that the first detection and the second detection are likely the same detection.

Optionally, the system may use one or more classifier to determine the possible gunshot event and one or more solvers to determine an actual gunshot event. The system may determine whether at least one feature represents a possible gunshot event in the environment by providing the at least one feature to the at least one classifier. The system may receive from at least one of the classifiers a detection score that indicates a likelihood of a possible gunshot event in the environment. The system may determine whether the detection score of the possible gunshot event in the environment satisfies a detection score threshold, and in response to determining that the detection score of the possible gunshot event in the environment satisfies the detection score threshold, determine that the at least one feature represents a possible gunshot event in the environment. The system providing the action may include, in response to determining that the at least one feature represents a possible gunshot event in the environment, provide at least some data from the infrared data or the acoustic data to at least one solver. The system may determine, using the at least one solver, whether the possible gunshot event is likely an actual gunshot event in the environment. The system may selectively trigger an event or determining to skip triggering an event using a result of the determination whether the possible gunshot event is likely an actual gunshot event in the environment.

Optionally, the system can determine a weapon classification in response to determining the presence of a possible gunshot event in the environment which contains the device. In some examples, the system may, in response to determining that the possible gunshot event is likely an actual gunshot event in the environment, determine, using a weapon classifier, a predicted weapon type of the gunshot event. The system may selectively trigger an event using the predicted weapon type. The system may use an earth mover distance feature and the weapon classifier to determine the predicted weapon type.

Optionally, the system may determine an aggregated detection score using output from at least two classifiers. Determining an aggregated detection score can increase the accuracy of the detection of possible gunshot events by using the results from the at least two classifiers. The system may receive detection scores from each of the at least two classifiers. The detection scores may indicate a likelihood of a possible gunshot event in the environment. The system may generate an aggregated detection score by combining the detection scores from the at least two classifiers and determining whether the aggregated detection score satisfies the detections score threshold. The system may, in response to determining that the aggregated detection score of the possible gunshot event in the environment satisfies the detection score threshold, determining that the at least one feature represents a possible gunshot event in the environment.

Optionally, the system may determine whether a possible gunshot event may be a false positive. Determining whether the possible gunshot event is a false positive can reduce the number of actions performed by the system which may falsely notify the system or external systems. The system may use the detection scores from at least one of the classifiers to indicate a detection of a likely false positive if the detection score does not satisfy the gunshot event threshold. In some examples, the system performing the action can include, in response to determining that at least one detection score from the detection scores received from at least one of the at least one classifier indicates a detection of a likely false positive, determining to not provide the detection of the possible gunshot in the environment to at least one solvers.

Optionally, the system may detect two gunshot events and determine whether an echo from a first gunshot was detected. If the system determines that an echo event was received as data, the system may determine to not perform further processing of data related to the echo event. The system may detect, in at least some of the acoustic data, a first possible gunshot event and a second possible gunshot event. The system may determine, using data for the first possible gunshot event and the second possible gunshot event, whether the second possible gunshot event satisfies an echo event threshold for the first possible gunshot event. The system may determine, in response to determining that the second possible gunshot event satisfies the echo event threshold for the first possible gunshot event, to not provide data for the second possible gunshot event to another processing engine.

In this specification, the term “database” can broadly refer to any collection of data: the data does not need to be structured in any particular way, or structured at all, and it can be stored on storage devices in one or more locations. A database can be implemented on any appropriate type of memory.

This specification uses the term “configured to” in connection with systems, apparatus, and computer program components. That a system of one or more computers is configured to perform particular operations or actions means that the system has installed on it software, firmware, hardware, or a combination of them that in operation cause the system to perform those operations or actions. That one or more computer programs is configured to perform particular operations or actions means that the one or more programs include instructions that, when executed by data processing apparatus, cause the apparatus to perform those operations or actions. That special-purpose logic circuitry is configured to perform particular operations or actions means that the circuitry has electronic logic that performs those operations or actions.

In this specification the term “engine” can broadly refer to a software-based system, subsystem, or process that is programmed to perform one or more specific functions. Generally, an engine will be implemented as one or more software modules or components, installed on one or more computers in one or more locations. In some instances, one or more computers will be dedicated to a particular engine. In some instances, multiple engines can be installed and running on the same computer or computers.

Operations can occur substantially concurrently in that the operations need not be exactly concurrent but can overlap at least in part. For instance, a first operation can begin and sometime after that a second operation can begin while the first operation is still occurring. Execution of the two operations, whether by the same system or different systems, can be substantially concurrent. In some examples, two operations can execute substantially concurrently when they have the same start time, same end time, or both.

In this specification, the term likely can mean that there is a likelihood that something might occur and that the likelihood satisfies a likelihood threshold. For instance, when determining that an object is likely depicted in an image, a system would determine a likelihood that the object is depicted in the image. The system would then determine whether the likelihood satisfies, e.g., is greater than or equal to, a likelihood threshold by comparing the two values. If so, the system determines that the object is likely depicted in the image. If not, the system determines that the object is not likely depicted in the image.

As noted previously, the systems and methods disclosed above utilize data processing apparatus to implement aspects of the gunshot detection using multichannel analysis described herein. FIG. 10 shows an example of a computing device 1000 and a mobile computing device 1050 that can be used as data processing apparatuses to implement the techniques described here. The computing device 1000 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The mobile computing device 1050 is intended to represent various forms of mobile devices, such as personal digital assistants, cellular telephones, smart-phones, and other similar computing devices. The components shown here, their connections and relationships, and their functions, are meant to be examples only, and are not meant to be limiting.

The computing device 1000 includes a processor 1002, a memory 1004, a storage device 1006, a high-speed interface 1008 connecting to the memory 1004 and multiple high-speed expansion ports 1010, and a low-speed interface 1012 connecting to a low-speed expansion port 1014 and the storage device 1006. Each of the processor 1002, the memory 1004, the storage device 1006, the high-speed interface 1008, the high-speed expansion ports 1010, and the low-speed interface 1012, are interconnected using various busses, and may be mounted on a common motherboard or in other manners as appropriate. The processor 1002 can process instructions for execution within the computing device 1000, including instructions stored in the memory 1004 or on the storage device 1006 to display graphical information for a GUI on an external input/output device, such as a display 1016 coupled to the high-speed interface 1008. In some implementations, multiple processors and/or multiple buses may be used, as appropriate, along with multiple memories and types of memory. Also, multiple computing devices may be connected, with each device providing portions of the necessary operations (e.g., as a server bank, a group of blade servers, or a multi-processor system).

The memory 1004 stores information within the computing device 1000. In some implementations, the memory 1004 is a volatile memory unit or units. In some implementations, the memory 1004 is a non-volatile memory unit or units. The memory 1004 may be another form of computer-readable medium, such as a magnetic or optical disk.

The storage device 1006 is capable of providing mass storage for the computing device 1000. In some implementations, the storage device 1006 may be or contain a computer-readable medium, such as a floppy disk device, a hard disk device, an optical disk device, or a tape device, a flash memory or other similar solid state memory device, or an array of devices, including devices in a storage area network or other configurations. Instructions can be stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor 1002), perform one or more methods, such as those described above. The instructions can be stored by one or more storage devices such as computer-or machine-readable mediums (for example, the memory 1004, the storage device 1006, or memory on the processor 1002).

The high-speed interface 1008 manages bandwidth-intensive operations for the computing device 1000, while the low-speed interface 1012 manages lower bandwidth-intensive operations. Such allocation of functions is an example only. In some implementations, the high-speed interface 1008 is coupled to the memory 1004, the display 1016 (e.g., through a graphics processor or accelerator), and to the high-speed expansion ports 1010, which may accept various expansion cards (not shown). In the implementation, the low-speed interface 1012 is coupled to the storage device 1006 and the low-speed expansion port 1014. The low-speed expansion port 1014, which may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) may be coupled to one or more input/output devices, such as a keyboard, a pointing device, a scanner, or a networking device such as a switch or router, e.g., through a network adapter.

The computing device 1000 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a standard server 1020, or multiple times in a group of such servers. In addition, it may be implemented in a personal computer such as a laptop computer 1022. It may be implemented as part of a rack server system 1024. Alternatively, components from the computing device 1000 may be combined with other components in a mobile device (not shown), such as a mobile computing device 1050. Each of such devices may contain one or more of the computing device 1000 and the mobile computing device 1050, and an entire system may be made up of multiple computing devices communicating with each other.

The mobile computing device 1050 includes a processor 1052, a memory 1064, an input/output device such as a display 1054, a communication interface 1066, and a transceiver 1068, among other components. The mobile computing device 1050 may be provided with a storage device, such as a micro-drive or other device, to provide additional storage. Each of the processor 1052, the memory 1064, the display 1054, the communication interface 1066, and the transceiver 1068, are interconnected using various buses, and several of the components may be mounted on a common motherboard or in other manners as appropriate.

The processor 1052 can execute instructions within the mobile computing device 1050, including instructions stored in the memory 1064. The processor 1052 may be implemented as a chipset of chips that include separate and multiple analog and digital processors. The processor 1052 may provide, for example, for coordination of the other components of the mobile computing device 1050, such as control of user interfaces, applications run by the mobile computing device 1050, and wireless communication by the mobile computing device 1050.

The processor 1052 may communicate with a user through a control interface 1058 and a display interface 1056 coupled to the display 1054. The display 1054 may be, for example, a TFT (Thin-Film-Transistor Liquid Crystal Display) display or an OLED (Organic Light Emitting Diode) display, or other appropriate display technology. The display interface 1056 may comprise appropriate circuitry for driving the display 1054 to present graphical and other information to a user. The control interface 1058 may receive commands from a user and convert them for submission to the processor 1052. In addition, an external interface 1062 may provide communication with the processor 1052, so as to enable near area communication of the mobile computing device 1050 with other devices. The external interface 1062 may provide, for example, for wired communication in some implementations, or for wireless communication in some implementations, and multiple interfaces may be used.

The memory 1064 stores information within the mobile computing device 1050. The memory 1064 can be implemented as one or more of a computer-readable medium or media, a volatile memory unit or units, or a non-volatile memory unit or units. An expansion memory 1074 may be provided and connected to the mobile computing device 1050 through an expansion interface 1072, which may include, for example, a SIMM (Single In Line Memory Module) card interface. The expansion memory 1074 may provide extra storage space for the mobile computing device 1050, or may store applications or other information for the mobile computing device 1050. Specifically, the expansion memory 1074 may include instructions to carry out or supplement the processes described above, and may include secure information also. Thus, for example, the expansion memory 1074 may be provide as a security module for the mobile computing device 1050, and may be programmed with instructions that permit secure use of the mobile computing device 1050. In addition, secure applications may be provided via the SIMM cards, along with additional information, such as placing identifying information on the SIMM card in a non-hackable manner.

The memory may include, for example, flash memory and/or NVRAM memory (non-volatile random access memory), as discussed below. In some implementations, instructions are stored in an information carrier. The instructions, when executed by one or more processing devices (for example, processor 1052), perform one or more methods, such as those described above. The instructions can be stored by one or more storage devices, such as one or more computer-or machine-readable mediums (for example, the memory 1064, the expansion memory 1074, or memory on the processor 1052). In some implementations, the instructions can be received in a propagated signal, for example, over the transceiver 768 or the external interface 1062.

The mobile computing device 1050 may communicate wirelessly through the communication interface 1066, which may include digital signal processing circuitry where necessary. The communication interface 1066 may provide for communications under various modes or protocols, such as GSM voice calls (Global System for Mobile communications), SMS (Short Message Service), EMS (Enhanced Messaging Service), or MMS messaging (Multimedia Messaging Service), CDMA (code division multiple access), TDMA (time division multiple access), PDC (Personal Digital Cellular), WCDMA (Wideband Code Division Multiple Access), CDMA2000, or GPRS (General Packet Radio Service), among others. Such communication may occur, for example, through the transceiver 1068 using a radio-frequency. In addition, short-range communication may occur, such as using a Bluetooth, WiFi, or other such transceiver (not shown). In addition, a GPS (Global Positioning System) receiver module 1070 may provide additional navigation-and location-related wireless data to the mobile computing device 1050, which may be used as appropriate by applications running on the mobile computing device 1050.

The mobile computing device 1050 may communicate audibly using an audio codec 1060, which may receive spoken information from a user and convert it to usable digital information. The audio codec 1060 may likewise generate audible sound for a user, such as through a speaker, e.g., in a handset of the mobile computing device 1050. Such sound may include sound from voice telephone calls, may include recorded sound (e.g., voice messages, music files, etc.) and may include sound generated by applications operating on the mobile computing device 1050.

The mobile computing device 1050 may be implemented in a number of different forms, as shown in the figure. For example, it may be implemented as a cellular telephone 1080. It may be implemented as part of a smart-phone 1082, personal digital assistant, or other similar mobile device.

Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs (application specific integrated circuits), computer hardware, firmware, software, and/or combinations thereof. These various implementations can include implementation in one or more computer programs that are executable and/or interpretable on a programmable system including at least one programmable processor, which may be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device.

These computer programs (known as programs, software, software applications or code) include machine instructions for a programmable processor, and can be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms machine-readable medium and computer-readable medium refer to any computer program product, apparatus and/or device (e.g., magnetic discs, optical disks, memory, Programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term machine-readable signal refers to any signal used to provide machine instructions and/or data to a programmable processor.

To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having a display device (e.g., an OLED (organic light emitting diode) display or LCD (liquid crystal display) monitor) for displaying information to the user and a keyboard and a pointing device (e.g., a mouse or a trackball) by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form, including acoustic, speech, or tactile input.

The systems and techniques described here can be implemented in a computing system that includes a back end component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front end component (e.g., a client computer having a graphical user interface or a Web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such back end, middleware, or front end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.

In some embodiments, the computing system can be cloud based and/or centrally processing data. In such case anonymous input and output data can be stored for further analysis. In a cloud based and/or processing center set-up, compared to distributed processing, it can be easier to ensure data quality, and accomplish maintenance and updates to the calculation engine, compliance to data privacy regulations and/or troubleshooting.

A number of implementations have been described. Nevertheless, it will be understood that various modifications can be made without departing from the spirit and scope of the disclosure. For example, various forms of the flows shown above can be used, with operations re-ordered, added, or removed.

Implementations of the subject matter and the functional operations described in this specification can be implemented in digital electronic circuitry, in tangibly-embodied computer software or firmware, in computer hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Implementations of the subject matter described in this specification can be implemented as one or more computer programs, i.e., one or more modules of computer program instructions encoded on a tangible non-transitory program carrier for execution by, or to control the operation of, a data processing apparatus. Alternatively or in addition, the program instructions can be encoded on an artificially-generated propagated signal, e.g., a machine-generated electrical, optical, or electromagnetic signal, that is generated to encode information for transmission to a suitable receiver apparatus for execution by a data processing apparatus. One or more computer storage media can include a machine-readable storage device, a machine-readable storage substrate, a random or serial access memory device, or a combination of one or more of them.

The term “data processing apparatus” refers to data processing hardware and encompasses all kinds of apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can be or include special purpose logic circuitry, e.g., a field programmable gate array (“FPGA”) or an application-specific integrated circuit (“ASIC”). The apparatus can optionally include, in addition to hardware, code that creates an execution environment for computer programs, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.

A computer program, which may be referred to or described as a program, software, a software application, a module, a software module, a script, or code, can be written in any form of programming language, including compiled or interpreted languages, or declarative or procedural languages, and it can be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment. A computer program may, but need not, correspond to a file in a file system. A program can be stored in a portion of a file that holds other programs or data, e.g., one or more scripts stored in a markup language document, in a single file dedicated to the program in question, or in multiple coordinated files, e.g., files that store one or more modules, sub-programs, or portions of code. A computer program can be deployed to be executed on one computer or on multiple computers that are located at one site or distributed across multiple sites and interconnected by a communication network.

The processes and logic flows described in this specification can be performed by one or more programmable computers executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows can be performed by, and apparatus can be implemented as, special purpose logic circuitry, e.g., a field programmable gate array (“FPGA”) or an application-specific integrated circuit (“ASIC”).

Computers suitable for the execution of a computer program include, by way of example, general or special purpose microprocessors or both, or any other kind of central processing unit. Generally, a central processing unit will receive instructions and data from a read-only memory or a random access memory or both. The essential elements of a computer are a central processing unit for performing or executing instructions and one or more memory devices for storing instructions and data. Generally, a computer will include, or be operatively coupled to receive data from or transfer data to, or both, one or more mass storage devices for storing data, e.g., magnetic, magneto-optical disks, or optical disks. However, a computer need not have such devices. A computer can be embedded in another device, e.g., a mobile telephone, a smart phone, a headset, a personal digital assistant (“PDA”), a mobile audio or video player, a game console, a GPS receiver, or a portable storage device, e.g., a universal serial bus (“USB”) flash drive, to name just a few.

Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media and memory devices, including by way of example semiconductor memory devices, e.g., EPROM, EEPROM, and flash memory devices; magnetic disks, e.g., internal hard disks or removable disks; magneto-optical disks; and CD-ROM and DVD-ROM disks. The processor and the memory can be supplemented by, or incorporated in, special purpose logic circuitry.

To provide for interaction with a user, implementations of the subject matter described in this specification can be implemented on a computer having a display device, e.g., a liquid crystal display (“LCD”), an organic light emitting diode (“OLED”) or other monitor, for displaying information to the user and a keyboard and a pointing device, e.g., a mouse or a trackball or a touchscreen, by which the user can provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback, e.g., visual feedback, auditory feedback, or tactile feedback; and input from the user can be received in any form, including acoustic, speech, or tactile input. In some examples, a computer can interact with a user by sending documents to and receiving documents from a device that is used by the user; for example, by sending web pages to a web browser on a user's device in response to requests received from the web browser.

The computing system can include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. In some implementations, a server transmits data, e.g., an Hypertext Markup Language (“HTML”) page, to a client device, e.g., for purposes of displaying data to and receiving user input from a client device, which acts as a client. Data generated at the client device, e.g., a result of user interaction with the client device, can be received from the client device at the server.

While this specification contains many specific implementation details, these should not be construed as limitations on the scope of what may be claimed, but rather as descriptions of features that may be specific to particular implementations. Certain features that are described in this specification in the context of separate implementations can be implemented in combination in a single implementation. Conversely, various features that are described in the context of a single implementation can be implemented in multiple implementations separately or in any suitable subcombination. Moreover, although features may be described above as acting in certain combinations and even initially claimed as such, one or more features from a claimed combination can in some instances be excised from the combination, and the claimed combination may be directed to a subcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particular order, this should not be understood as requiring that such operations be performed in the particular order shown or in sequential order, or that all illustrated operations be performed, to achieve desirable results. In certain circumstances, multitasking and parallel processing may be advantageous. Moreover, the separation of various system modules and components in the implementations described above should not be understood as requiring such separation in all implementations, and it should be understood that the described program components and systems can generally be integrated together in a single software product or packaged into multiple software products.

Particular implementations have been described. Other implementations are within the scope of the following claims. For example, the operations recited in the claims, described in the specification, or depicted in the figures can be performed in a different order and still achieve desirable results. In some implementations, multitasking and parallel processing may be advantageous.