SYSTEMS AND METHODS FOR AUTOMATED MAPPING OF SPORTS DATA

Description

CROSS-REFERENCE TO RELATED APPLICATION

This application claims benefit to U.S. Provisional Patent Application No. 63/716,818 filed Nov. 6, 2024, the entire contents of which are incorporated herein by reference.

TECHNICAL FIELD

Various embodiments of the present disclosure relate generally to sports event data and, more particularly, to systems and methods for generating play graphics using sports event data.

INTRODUCTION

Play graphic generation often focuses on providing graphic representations with minimal functionality using basic previously known play information. This is done by using standard play information to show the relative locations of players and their expected respective movement during a play. The use of this technology, including player specific information, may refine and improve play graphic generation.

Unless otherwise indicated herein, the materials described in this section are not prior art to the claims in this application and are not admitted to be prior art, or suggestions of the prior art, by inclusion in this section.

SUMMARY

In some aspects, the techniques described herein relate to a method for generating sports play graphics, the method including: receiving, by a computing device, a first data set; comparing, by the computing device, the first data set to one or more template plays; determining, by the computing device, a representative play in response to comparing the first data set to the one or more template plays; receiving, by the computing device, at least one player performance data set; determining, by the computing device, a threat score associated with the representative play based on the at least one player performance data set for each of a plurality of players associated with the first data set; combining, by the computing device, the at least one player performance data set and the threat score associated with the representative play; generating, by the computing device, a sports play graphic in response to combining the at least one player performance data set and the threat score associated with the representative play; and outputting, by the computing device, the sports play graphic.

In some aspects, the techniques described herein relate to a method, wherein prior to generating the sports play graphic, the method includes translating, by the computing device, the first data set and the at least one player performance data set into a sports play graphic data set.

In some aspects, the techniques described herein relate to a method, wherein translating, by the computing device, the first data set and the at least one player performance data set into a sports graphic data set further includes: translating, by the computing device, the first data set from a first format to a second format; translating, by the computing device, the at least one player performance data set from a third format to a fourth format; and translating, by the computing device, the first data set in the second format and the at least one player performance data set in the fourth format into a fifth format.

In some aspects, the techniques described herein relate to a method, wherein prior to generating the sports play graphic, the method includes assigning, by the computing device, a unique ID for the representative play.

In some aspects, the techniques described herein relate to a method, wherein the sports play graphic includes at least one indicator representing the threat score for each player performance data set for each of a plurality of players.

In some aspects, the techniques described herein relate to a method, wherein the first data set includes at least one of a player tracking data, a player information data, a raw data, a subjective data, or an observed data.

In some aspects, the techniques described herein relate to a method, wherein the player tracking data includes a pre-play player location data and a post-play player location data.

In some aspects, the techniques described herein relate to a method, wherein the first data set includes at least one of an offensive player data set or a defensive player data set.

In some aspects, the techniques described herein relate to a method, wherein the player information data includes at least one of a player identification, a player position, a player status, or a player role.

In some aspects, the techniques described herein relate to a method, wherein the raw data includes at least one of a play concept, a play location, or a snap hash.

In some aspects, the techniques described herein relate to a system for generating sports play graphics, the system including: a memory storing instructions; a generative machine learning model trained to generate sports play graphics; a processor operatively connected to the memory and configured to execute instructions to perform operations including: receiving, by a computing device, a first data set; comparing, by the computing device, the first data set to one or more template plays; determining, by the computing device, a representative play in response to comparing the first data set to the one or more template plays; receiving, by the computing device, at least one player performance data set; determining, by the computing device, a threat score associated with the representative play based on the at least one player performance data set for each of a plurality of players associated with the first data set; combining, by the computing device, the at least one player performance data set and the threat score associated with the representative play; generating, by the computing device, a sports play graphic; and outputting, by the computing device, the sports play graphic.

In some aspects, the techniques described herein relate to a system, wherein prior to generating the sports play graphic, the operations further include translating, by the computing device, the first data set and the at least one player performance data set into a sports play graphic data set.

In some aspects, the techniques described herein relate to a system, the operations further including: translating, by the computing device, the first data set from a first format to a second format; translating, by the computing device, the at least one player performance data set from a third format to a fourth format; and translating, by the computing device, the first data set in the second format and the at least one player performance data set in the fourth format into a fifth format.

In some aspects, the techniques described herein relate to a system, wherein prior to generating the sports play graphic, the operations further include assigning, by the computing device, a unique ID for the determined representative play.

In some aspects, the techniques described herein relate to a system, wherein the sports play graphic includes at least one indicator representing the threat score for each player performance data set for each of a plurality of players.

In some aspects, the techniques described herein relate to a non-transitory computer readable medium configured to store processor readable instructions, wherein when executed by a processor, the instructions perform operations including: receiving, by a computing device, a first data set; comparing, by the computing device, the first data set to one or more template plays; determining, by the computing device, a representative play in response to comparing the first data set to the one or more template plays; receiving, by the computing device, at least one player performance data set; determining, by the computing device, a threat score associated with the representative play based on the at least one player performance data set for each of a plurality of players associated with the first data set; combining, by the computing device, the at least one player performance data set and the threat score associated with the representative play; generating, by the computing device, a sports play graphic; and outputting, by the computing device, the sports play graphic.

In some aspects, the techniques described herein relate to a non-transitory computer readable medium, wherein prior to generating the sports play graphic, the operations further include translating, by the computing device, the first data set and the at least one player performance data set into a sports play graphic data set.

In some aspects, the techniques described herein relate to a non-transitory computer readable medium, the operations further including: translating, by the computing device, the first data set from a first format to a second format; translating, by the computing device, the at least one player performance data set from a third format to a fourth format; and translating, by the computing device, the first data set in the second format and the at least one player performance data set in the fourth format into a fifth format.

In some aspects, the techniques described herein relate to a non-transitory computer readable medium, wherein prior to generating the sports play graphic, the operations further include assigning, by the computing device, a unique ID for the determined representative play.

In some aspects, the techniques described herein relate to a non-transitory computer readable medium, wherein the sports play graphic includes at least one indicator representing the threat score for each player performance data set for the plurality of players.

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate various exemplary embodiments and together with the description, serve to explain the principles of the disclosed aspects.

FIG. 1 depicts a block diagram illustrating a computing environment, according to example embodiments.

FIG. 2 depicts an exemplary flow diagram for generating play graphics, in accordance with an aspect of the disclosed subject matter.

FIG. 3 depicts an exemplary flow diagram for generating play graphics using format translations, in accordance with an aspect of the disclosed subject matter.

FIGS. 4A and 4B depict exemplary data used in generating play graphics, in accordance with an aspect of the disclosed subject matter.

FIGS. 5A and 5B depict additional exemplary data used in generating play graphics, in accordance with an aspect of the disclosed subject matter.

FIG. 6 depicts an exemplary sports graphic, in accordance with an aspect of the disclosed subject matter.

FIG. 7 depicts an example flowchart for generating tracking and/or event data, according to example embodiments.

FIGS. 8 and 9 depicts flow diagrams of an exemplary method for using the machine learning model to generate sports tracking data, according to example embodiments.

FIG. 10 depicts a flow diagram for training a machine learning model, in accordance with an aspect of the disclosed subject matter.

FIG. 11A depicts a block diagram illustrating a computing device, according to example embodiments.

FIG. 11B depicts another block diagram illustrating a computing device, according to example embodiments.

Notably, for simplicity and clarity of illustration, certain aspects of the figures depict the general configuration of the various embodiments. Descriptions and details of well-known features and techniques may be omitted to avoid unnecessarily obscuring other features. Elements in the figures are not necessarily drawn to scale; the dimensions of some features may be exaggerated relative to other elements to improve understanding of the example embodiments.

To facilitate understanding, identical reference numerals have been used, where possible, to designate identical elements that are common to the figures. It is contemplated that elements disclosed in one embodiment may be beneficially utilized on other embodiments without specific recitation.

DETAILED DESCRIPTION

Various aspects of the present disclosure relate generally to sports event data and, more particularly, to systems and methods for generating sports play graphics (also referred to herein as “play graphics” for simplicity) using sports event data and or sports tracking data. Various embodiments may use machine learning models to automatically generate play graphics relating to sporting events and the use thereafter. An event can refer to a particular play such as a pass or a run, but can also refer to an entire game. As discussed, existing solutions are unable to generate play graphics relating to sports plays.

According to aspects disclosed herein, a multimodal sports learning language model (LLM) may receive text, audio, video, or drawings as input information from a user or computing system. Such input information may be input by a user or by an automated system such as a tracking system, broadcast tracking system, and/or the like. The multimodal sports LLM may use preprocessed event streams to map corresponding metadata to the input information. This information may be used by the multimodal sports LLM to utilize sports tracking data (also referred to herein as “tracking data” for simplicity) that is stored and/or output to a user. The stored information may be in the form of graphics, data tables, audio, and/or text or a combination thereof. Information output by the multimodal sports LLM may be in the form of visualizations, retrieval systems, analyses, audio and/or textual commentary or a combination thereof. The output information may be play graphics data that is generated by the multimodal sports LLM based on historical event and/or tracking data that is associated with a determined play.

The following non-limiting example is introduced for discussion purposes. In the example, a system receives and stores a plurality of play data. The system may receive a user input for querying play data and accesses relevant database records from a database. The database records can include play-related data associated with the sporting event such as player information (e.g., location, position, role, status, etc.), player performance data, team(s), down and distance, and/or league related information. The system determines representative plays and contextual information from the input query. This information is then mapped to database records to generate a play graphic based on the received query. The system can output the generated play graphic to the client device.

Technical advantages of the disclosed techniques include improvements to machine learning outputs. For instance, certain aspects relate to determining representative plays and contextual information from an input that improve the performance, accuracy, and results of information to be mapped to play graphics. In doing so, disclosed techniques may provide improvements relative to existing solutions. The techniques disclosed herein improve machine learning outputs by, for example, filtering potential play graphic outputs based on template matching and further refining play graphic outputs based on player performance data after filtering the potential graphic options based on the template matching. By employing such a process that first filters the scope of potential graphics using template matching, the variables considered for machine learning outputs are reduced, thereby resulting in more accurate graphic generation and formatting.

The disclosed systems and methods provide technological improvements to computer-based sports data processing and visualization systems. In some aspects, the techniques transform raw sports data into structured, machine-readable formats that enable automated analysis and graphic generation. The computing systems may process large volumes of tracking data, player performance metrics, and/or contextual game information in real-time or near real-time (e.g., within approximately 1 minute, approximately 30 seconds, approximately 15 seconds, or approximately 5 seconds) to generate actionable visual representations.

In some cases, the disclosed systems and methods may utilize specialized machine learning models trained on sports-specific data sets to identify patterns and correlations that would not be apparent through manual analysis. The automated mapping of sports data may involve complex algorithmic processes that correlate player positions, movement trajectories, and performance statistics with template plays stored in databases. These processes may require significant computational resources and specialized data structures optimized for sports analytics. Such optimizations, as disclosed herein, may reduce the computational load required to generate outputs such as computational loads required for play graphic generation.

The generated play graphics may provide technical solutions to problems in sports analysis and coaching, such as the inability to quickly visualize complex player interactions and performance metrics in an intuitive format. In some aspects, the systems may automatically translate between different data formats and coordinate systems to ensure compatibility across various tracking technologies and data sources.

The computing devices disclosed herein may implement novel data processing pipelines that combine real-time tracking data with historical performance databases to generate predictive visualizations. These technical processes may involve multi-dimensional data analysis, statistical modeling, and dynamic graphic rendering that extends beyond conventional data display methods.

In some cases, the disclosed systems and methods may provide improvements to existing computer-based sports analysis tools by reducing processing time, increasing accuracy of play identification, and enabling automated generation of complex visual representations that previously required manual creation by human analysts.

The terminology used herein may be interpreted in its broadest reasonable manner, even though the terminology is being used in conjunction with a detailed description of certain specific examples of the present disclosure. Indeed, certain terms may even be emphasized above; however, any terminology intended to be interpreted in any restricted manner will be overtly and specifically defined as such in this Detailed Description section. Both the foregoing general description and the detailed description are exemplary and explanatory only and are not restrictive of the features.

As used herein, the terms “comprises,” “comprising,” “having,” including,” or other variations thereof, are intended to cover a non-exclusive inclusion such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements, but may include other elements not expressly listed or inherent to such a process, method, article, or apparatus.

In this disclosure, relative terms, such as, for example, “about,” “substantially,” “generally,” and “approximately” are used to indicate a possible variation of ±10% in a stated value.

The term “exemplary” is used in the sense of “example” rather than “ideal.” As used herein, the singular forms “a,” “an,” and “the” include plural reference unless the context dictates otherwise.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the embodiments disclosed herein. The specification and examples should be considered as exemplary only.

FIG. 1 is a block diagram illustrating a computing environment 100, according to example embodiments. The computing environment 100 may include a tracking system 102 (e.g., positioned at or in communication with one or more components positioned at a venue 106), an organization computing system 104, and one or more client devices 108 (collectively or individually referred to herein as “client device 108”) communicating via a network 105.

The network 105 may be of any suitable type, including individual connections via the Internet, such as cellular or Wi-Fi networks. In some embodiments, the network 105 may connect terminals, services, and mobile devices using direct connections, such as radio frequency identification (RFID), near-field communication (NFC), Bluetooth™, low-energy Bluetooth™ (BLE), Wi-Fi™, ZigBee™, ambient backscatter communication (ABC) protocols, USB, WAN, or LAN. Because the information transmitted may be personal or confidential, security concerns may dictate one or more of these types of connections be encrypted or otherwise secured. In some embodiments, however, the information being transmitted may be less personal, and therefore, the network connections may be selected for convenience over security.

The network 105 may include any type of computer networking arrangement used to exchange data or information. For example, the network 105 may be the Internet, a private data network, virtual private network using a public network and/or other suitable connection(s) that enables components in the computing environment 100 to send and receive information between the components of the computing environment 100.

The tracking system 102 may be positioned in the venue 106 and/or may be in communication (e.g., electronic communication, wireless communication, wired communication, etc.) with components located at the venue 106. For example, the venue 106 may be configured to host a sporting event that includes one or more agents 112. The tracking system 102 may be configured to capture the motions of one or more agents 112 (e.g., players) on the playing surface, as well as one or more other agents (e.g., objects) of relevance (e.g., ball, puck, referees, etc.). In some embodiments, the tracking system 102 may be an optically-based system using, for example, a plurality of fixed cameras, movable cameras, one or more panoramic cameras, etc. For example, a system of six calibrated cameras (e.g., fixed cameras), which project three-dimensional locations of players and a ball onto a two-dimensional overhead view of the playing surface may be used. In another example, a mix of stationary and non-stationary cameras may be used to capture motions of all agents on the playing surface as well as one or more objects or relevance. Utilization of such a tracking system (e.g., the tracking system 102) may result in many different camera views of the playing surface (e.g., high sideline view, free-throw line view, huddle view, face-off view, end zone view, etc.).

In some embodiments, the tracking system 102 may be used for a broadcast feed of a given match or game. For example, the tracking system 102 may be used to generate one or more first game files 110 to facilitate a broadcast feed of a given match. In such embodiments, each frame of the broadcast feed may be stored in the one or more first game files 110. A broadcast feed may be a feed that is formatted to be broadcast over one or more channels (e.g., broadcast channels, internet-based channels, etc.). The one or more first game files 110 may be converted from a first format (e.g., a format output by the one or more cameras or a different format than the format output by the one or more cameras) and may be converted into a second format (e.g., for broadcast transmission).

In some embodiments, the one or more first game files 110 may further be augmented with other event information corresponding to event data, such as, but not limited to, game event information (pass, made shot, turnover, etc.) and context information (current score, time remaining, etc.). Event data may be automatically identified using a machine learning trained to receive, as an input, the one or more first game files 110 or a subset thereof and output game information and/or context information based on the input. The machine learning model may be trained using supervised, semi-supervised, or unsupervised learning, in accordance with the techniques disclosed herein. The machine learning model may be trained by analyzing training data using one or more machine learning algorithms, as disclosed herein. The training data may include the one or more first game files 110 or simulated game files from historical games, simulated games, and/or the like and may include tagged and/or untagged data.

The tracking system 102 may be configured to communicate with the organization computing system 104 via the network 105. For example, the tracking system 102 may be configured to provide the organization computing system 104 with a broadcast stream of a game or event in real-time or near real-time via the network 105. As an example, the tracking system 102 may provide the one or more first game files 110 in a first format (e.g., corresponding to a format based on the components of the tracking system 102). Alternatively, or additionally, the tracking system 102 or the organization computing system 104 may convert the broadcast stream (e.g., one or more first game files 110) into a second format, from the first format. The second format may be based on the organization computing system 104. For example, the second format may be a format associated with a data store 118, discussed further herein.

The organization computing system 104 may be configured to process the broadcast stream of the game. The organization computing system 104 may include at least a web client application server 114, a tracking data system 116, the data store 118, a play-by-play module 120, a padding module 122, and/or a mapping module 124. Each of the tracking data system 116, the play-by-play module 120, the padding module 122, and the mapping module 124 may be comprised of one or more software modules. The one or more software modules may be collections of code or instructions stored on a media (e.g., memory of the organization computing system 104) that represent a series of machine instructions (e.g., program code) that implements one or more algorithmic steps. Such machine instructions may be the actual computer code the processor of the organization computing system 104 interprets to implement the instructions or, alternatively, may be a higher level of coding of the instructions that are interpreted to obtain the actual computer code. The one or more software modules may also include one or more hardware components. One or more aspects of an example algorithm may be performed by the hardware components (e.g., circuitry) itself, rather than as a result of the instructions.

The tracking data system 116 may be configured to receive broadcast data from the tracking system 102 and generate tracking data from the broadcast data. In some embodiments, the tracking data system 116 may apply an artificial intelligence and/or computer vision system configured to derive player-tracking data from broadcast video feeds.

To generate the tracking data from the broadcast data, the tracking data system 116 may, for example, map pixels corresponding to each player and object to dots and may transform the dots into a semantically meaningful event layer, which may be used to describe player attributes. For example, the tracking data system 116 may be configured to ingest broadcast video received from the tracking system 102. In some embodiments, the tracking data system 116 may further categorize each frame of the broadcast video into trackable and non-trackable clips. In some embodiments, the tracking data system 116 may further calibrate the moving camera based on the trackable and non-trackable clips. In some embodiments, the tracking data system 116 may further detect players within each frame using skeleton tracking. In some embodiments, the tracking data system 116 may further track and re-identify players over time. For example, the tracking data system 116 may re-identify players who are not within a line of sight of a camera during a given frame. In some embodiments, the tracking data system 116 may further detect and track an object across a plurality of frames. In some embodiments, the tracking data system 116 may further utilize optical character recognition techniques. For example, the tracking data system 116 may utilize optical character recognition techniques to extract score information and time remaining information from a digital scoreboard of each frame.

Such techniques assist in the tracking data system 116 generating tracking data from the broadcast feed (e.g., broadcast video data). For example, the tracking data system 116 may perform such processes to generate tracking data across thousands of possessions and/or broadcast frames. In addition to such process, the organization computing system 104 may go beyond the generation of tracking data from broadcast video data. Instead, to provide descriptive analytics, as well as a useful feature representation for the mapping module 124, the organization computing system 104 may be configured to map the tracking data to a semantic layer (e.g., events).

The tracking data system 116 may be implemented using a machine learning model. The machine learning model may be trained using supervised, semi-supervised, or unsupervised learning, in accordance with the techniques disclosed herein. The machine learning model may be trained by analyzing training data using one or more machine learning algorithms, as disclosed herein. The training data may include game files or simulated game files from historical games, simulated games, historical or simulated feature representations, and/or the like and may include tagged and/or untagged data. The tagged data may include position information, movement information, object information, trends, agent identifiers, agent re-identifiers, etc.

The play-by-play module 120 may be configured to receive play-by-play data from one or more third party systems. For example, the play-by-play module 120 may receive a play-by-play feed corresponding to the broadcast video data. In some embodiments, the play-by-play data may be representative of human generated data based on events occurring within the game. Even though the goal of computer vision technology is to capture all data directly from the broadcast video stream, the referee, in some situations, is the ultimate decision maker in the successful outcome of an event. For example, in basketball, whether a basket is a 2-point shot or a 3-point shot (or is valid, a travel, defensive/offensive foul, etc.) is determined by the referee. As such, to capture these data points, the play-by-play module 120 may utilize machine learning outputs and/or manually annotated data that may reflect the referee's ultimate adjudication. Such data may be referred to as the play-by-play feed.

To identify events within the generated tracking data, the tracking data system 116 may merge or align the play-by-play data with the raw generated tracking data (which may include the game and time fields). The tracking data system 116 may utilize a fuzzy matching algorithm, which may combine play-by-play data, optical character recognition data (e.g., shot clock, score, time remaining, etc.), and play/ball positions (e.g., raw tracking data) to generate the aligned tracking data.

Once aligned, the tracking data system 116 may be configured to perform various operations on the aligned tracking data. For example, the tracking data system 116 may use the play-by-play data to refine the player and ball positions and precise frame of the end of possession events (e.g., shot/rebound location). In some embodiments, the tracking data system 116 may further be configured to detect events, automatically, from the tracking data. In some embodiments, the tracking data system 116 may further be configured to enhance the events with contextual information.

For automatic event detection, the tracking data system 116 may include a neural network system trained to detect/refine various events in a sequential manner. For example, the tracking data system 116 may include an actor-action attention neural network system to detect/refine one or more of: shots, scores, points, rebounds, passes, dribbles, penalties, fouls, and/or possessions. The tracking data system 116 may further include a host of specialist event detectors trained to identify higher-level events. Exemplary higher-level events may include, but are not limited to, plays, transitions, presses, crosses, breakaways, post-ups, drives, isolations, ball-screens, offside, handoffs, off-ball-screens, and/or the like. In some embodiments, each of the specialist event detectors may be representative of a neural network, specially trained to identify a specific event type. More generally, such event detectors may utilize any type of detection approach. For example, the specialist event detectors may use a neural network approach or another machine learning classifier (e.g., random decision forest, SVM, logistic regression etc.).

While mapping the tracking data to events enables a player representation to be captured, to further build out the best possible player representation, the tracking data system 116 may generate contextual information to enhance the detected events. Exemplary contextual information may include defensive matchup information (e.g., who is guarding who at each frame, defensive formations), as well as other defensive information such as coverages for ball-screens or presses.

In some embodiments, to measure influence, the tracking data system 116 may use a measure referred to as an “influence score.” The influences score may capture the influence a player may have on each of the players on an opposing team on a scale of 0-100. In some embodiments, the value for the influence score may be based on sport principles, such as, but not limited to, proximity to player, distance from scoring object (e.g., basket, goal, boundary, etc.), gap closure rate, passing lanes, lanes to the scoring object, and the like.

The padding module 122 may be configured to create new player representations using mean-regression to reduce random noise in the features. For example, one of the profound challenges of modeling using potentially only limited games (e.g., 20-30 games) of data per player may be the high variance of low frequency events seen in the tracking data. Therefore, the padding module 122 may be configured to utilize a padding method, which may be a weighted average between the observed values and sample mean.

Accordingly, for each player, the tracking data system 116, the play-by-play module 120, and the padding module 122 may work in conjunction to generate a raw data set and a padded data set for each player.

The mapping module 124 may be configured or trained to generate a connection and/or association with prompts of a multimodal sports LLM and user inputs (e.g., audio, speech, drawings, video, etc.). For example, the mapping module 124 may be configured to receive a user input (e.g., audio/speech) requesting information relating to a play within a specific match (e.g., goal scored by Manchester United against Liverpool). The mapping module 124 may generate one or more connections and/or associations with the user input and the event stream (e.g., match between Manchester United against Liverpool). Based on the generated connections, the mapping module 124 may be configured to determine event data associated with the event stream and the user input. The mapping module 124 may output one or more graphics, text, audio, or a combination thereof based on the determined connections and/or associations with the user input.

In some embodiments, the mapping module 124 may include a separate mapping model tuned for each input type (e.g., audio, text, drawing, video, etc.). Given that each input is different from each other, there may be times that a single mapping model may have trouble determining connections and/or associations. In such scenarios, one or more individual mapping models may be employed for a single user input. For example, upon receiving a user input (e.g., speech and drawing), the mapping module 124 may utilize one or more mapping models for each input type received. The one or more mapping models may determine one or more connections and/or associations from the received inputs. Based on the determined one or more connections, the mapping module 124 may output one or more graphics and texts corresponding to the user inputs. The mapping module 124 is discussed further in conjunction with figures discussed below (e.g., FIGS. 2-9).

The data store 118 may be configured to store one or more second game files 126. Each of the one or more second game files 126 may include video data of a given match. For example, the video data may correspond to a plurality of video frames captured by the tracking system 102, the tracking data derived from the broadcast video as generated by the tracking data system 116, play-by-play data, enriched data, and/or padded training data. The one or more second game files 126 may be based, for example, on the one or more first game files 110 as discussed herein. The one or more second game files 126 may be in a different format than the one or more first game files 110. For example, a first format of the one or more first game files 110 or a subset thereof may be transformed into a second format to become the one or more second game files 126. The transformation may be performed automatically based on the type and/or content of the first format and the type and/or content of the second format.

The client device 108 may be in communication with the organization computing system 104 via the network 105. The client device 108 may be operated by a user. For example, the client device 108 may be a mobile device, a tablet, a desktop computer, or any computing system having the capabilities described herein. Users may include, but are not limited to, individuals such as, for example, subscribers, clients, prospective clients, or customers of an entity associated with the organization computing system 104, such as individuals who have obtained, will obtain, or may obtain a product, service, or consultation from an entity associated with the organization computing system 104.

The client device 108 may include at least an application 130. The application 130 may be representative of a web browser that allows access to a website or a stand-alone application. The client device 108 may access the application 130 to access one or more functionalities of the organization computing system 104. The client device 108 may communicate over the network 105 to request a webpage, for example, from the web client application server 114 of the organization computing system 104. For example, the client device 108 may be configured to execute the application 130 to access one or more determined connections and/or associations based on a user input generated by the mapping module 124. The content that is displayed to client device 108 may be transmitted from the web client application server 114 to the client device 108, and subsequently processed by the application 130 for display through a graphical user interface (GUI) of the client device 108.

Prediction Engine

A prediction engine (e.g., which may be part of the tracking data system 116) may be configured to predict an underlying formation of a team. Mathematically, the goal of a role-alignment procedure may be to find the transformation A: {U₁, U₂, . . . , U_n}×M→[R₁, R₂, . . . , R_K], which may map the unstructured set U of N player trajectories to an ordered set (e.g., a vector) of K role-trajectories R. Each player trajectory itself may be an ordered set of positions

U n = [ x s , n ] s = 1 S

for an agent n∈[1, N] and a frame s∈[1, S]. In some embodiments, M may represent the optimal permutation matrix that enables such an ordering. The goal of the prediction engine may be to find the most probable set of of two-dimensional (2D) probability density functions:

ℱ *= arg max ℱ P ⁡ ( ℱ | R ) P ⁡ ( x ) = ∑ n = 1 N P ⁡ ( x | n ) ⁢ P ⁡ ( n ) = 1 N ⁢ ∑ n = 1 N P n ( x )

In some embodiments, this equation may be transformed into one of entropy minimization where the goal is to reduce (e.g., minimize) the overlap (e.g., the KL-Divergence) between each role. As such, in some embodiments, the final optimization equation in terms of total entropy H may become:

ℱ *= arg max ℱ ∑ n = 1 N H ⁡ ( x | n )

The prediction engine may include a formation discovery module, a role assignment module, a template module, and/or the like each corresponding to a distinct phase of the prediction process. The formation discovery module may be configured to learn the distributions which maximize the likelihood of the data. The role assignment module may be configured to map each player position to a “role” distribution in each frame. Once the data has been aligned, the template module may be configured to map each learned formation a formation cluster template.

The organization computing system 104 may receive tracking data and/or event data for a plurality of events across a plurality of seasons or across a match. For each event, the pre-processing agent may divide the event into a plurality of segments based on the event information. In some embodiments, the pre-processing agent may divide the event into a plurality of segments based on various events that may occur throughout the game. For example, the pre-processing agent may divide the event into a plurality of segments based on one or more events that include, but may not be limited to, red cards, ejections, technical fouls, flagrant fouls, player disqualifications, substitutions, halves, periods, quarters, overtime, and the like. Generally, each segment of a plurality of segments associated with an event may include an interval of a requisite duration (e.g., at least one minute of play, at least two minutes of play, etc.). Such requisite duration may allow an organization computing system to detect a team's formation.

Each segment may include a set of tracking data associated therewith. The player tracking data may be captured by tracking system, which may be configured to record the (x, y) positions of the players at a high frame rate (e.g., 10 Hz). In some embodiments, the player tracking data may further include single-frame event-labels (e.g., pass, shot, cross) in each frame of player tracking data. These frames may be referred to as “event frames.” As shown, the initial player tracking data may be represented as a set U of N player trajectories. Each player trajectory itself may be an ordered set of positions

U n = [ x s , n ] s = 1 S

for an agent n∈[1, N] and a frame s∈[1, S].

In some embodiments, the pre-processing agent may normalize the raw position data of the players. For example, the pre-processing agent may normalize the raw position data of the players in each segment so that all teams in the player tracking data are attacking from left to right and have zero mean in each frame. Such normalization may result in the removal of translational effects from the data. This may yield the set

U ′ = { U 1 ′ , U 2 ′ , … , U n ′ } .

In some embodiments, the pre-processing agent may initialize cluster centers of the normalized data set for formation discovery with the average player positions. For example, average player positions may be represented by the set μ₀={μ₁, μ₂, . . . , μ₃}. The pre-processing agent may take the average position of each player in the normalized data and may initialize the normalized data based on the average player positions. Such initialization of the normalized data based on average player position may act as initial roles for each player to minimize data variance.

The organization computing system 104 may learn a formation template from the tracking data for each segment. For example, the formation discovery module may learn the distributions which maximize the likelihood of the data. The formation discovery module may structure the initialized data into a single (SN)×d vector, where S may represent the total number of frames, N may represent the total number of agents (e.g., ten outfielders in the case of soccer, five players in the case of basketball, fifteen players in the case of rugby, etc.) and d may represent the dimensionality of the data (e.g., d=2).

The formation discovery module may then initiate a formation discovery algorithm. For example, the formation discovery module may initialize a K-means algorithm using the player average positions and execute to convergence. Executing the K-means algorithm to convergence produces better results than conventional approaches of running a fixed number of iterations.

The formation discovery module may then initialize a Gaussian Mixture Model (GMM) using cluster centers of the last iteration of the K-means algorithm. By parametrizing the distribution as a mixture of K Gaussians (with K being equal to the number of “roles,” which is usually also equal to N, the number of players), the formation discovery module may be able to identify an optimal formation that maximizes the likelihood of the data x. In other words, GMM may be configured to identify ={P₁, P₂, . . . , P_K}, where may represent the optimal formation that maximizes the likelihood of the data x. Therefore, instead of stopping the process after the last iteration of the K-means algorithm, the formation discovery module may use GMM clustering, as the ellipse may better capture the shape of each player role compared to only a K-means clustering technique, which captures the spherical nature of each role's data cloud.

Further, GMMs are known to suffer from component collapse and become trapped in pathological solutions. Such collapse may result in non-sensible clustering, e.g., non-sensical outputs that may not be utilized. To combat this, the formation discovery module may be configured to monitor eigenvalues (λ_i) of each of the components or parameters of the GMM throughout the expectation maximization process. If the formation discovery module determines that the eigenvalue ratio of any component becomes too large or too small, the next iteration may run a Soft K-Means (e.g., a mixture of Gaussians with spherical covariance) update instead of the full-covariance update. Such process may be performed to ensure that the eventual clustering output is sensible. For example, the formation discovery module may monitor how the parameters of the GMM are converging; if the parameters of the GMM are erratic (e.g., “out of control”), the formation discovery module may identify such erratic behavior and then slowly return the parameters back within the solution space using a soft K-means update.

For retrieval tasks using large amounts of data, an embodiment of the system uses a hash-table is required by grouping similar plays together, such that when a query is made, only the “most-likely” candidates are retrieved. Comparisons can then be made locally amongst the candidates and each play in these groups are ranked in order of most similar. Previous systems attempted clustering plays into similar groups by using only one attribute, such as the trajectory of the ball. However, the semantics of a play are more accurately captured by using additional information, such as information about the players (e.g., identity, trajectory, etc.) and events (pass, dribble, shot, etc.), as well as contextual information (e.g., if team is winning or losing, how much time remaining, etc.). Thus, embodiments of the present system utilize information regarding the trajectories of the ball and the players, as well as game events and contexts, to create a hash-table, effectively learning a “playbook” of representative plays for a team or player's behavior. The playbook is learned by choosing a classification metric that is indicative of interesting or discriminative plays. Suitable classification metrics may include predicting the probability of scoring in soccer or basketball (e.g., expected point value (“EPV”), or expected goal value (“EGV”). Other predicted values can also be chosen for performance variables, such as probability of making a pass, probability of shooting, probability of moving in a certain direction/trajectory, or the probability of fatigue/injury of a player.

The classification metric is used to learn a decision-tree, which is a coarse-to-fine hierarchical method, where at each node a question is posed which splits the data into groups. A benefit of this approach is that the result may be interpretable and is multi-layered, which can act as “latent factors.”

Bottom-Up Approach

In an embodiment of the system, a bottom-up approach to learning the decision tree is used. Various features are used in succession to discriminate between plays (e.g., first use the ball, then the player who is closest to the ball, then the defender etc.). By aligning the trajectories, there is a point of reference for trajectories relative to their current position. This permits more specific questions while remaining general (e.g., if a player is in the role of “point guard”, what is the distance from his/her teammate in the role of “shooting guard”, as well as the distance from the defender in the role of “point guard”). Using this approach avoids the need to exhaustively check all distances, which is enormous for both basketball and soccer.

Top-Down Approach

In another embodiment of the system, a top-down approach to learning the decision tree is used. At a first step, all the plays are aligned to the set of templates. From this initial set of templates, the plays are assigned to a set of K groups (clusters), using all ball and player information, forming a Layer 1 of the decision tree. Back propagation is then used to prune out unimportant players and divide each cluster into sub-clusters (Layer 2). The approach continues until the leaves of the tree represent a dictionary of plays which are predictive of a particular task—e.g., goal-scoring (Layer 3).

Personalization Using Latent Factor Models

In addition to raw trajectory information, in embodiments of the system, the plays in the database are also associated with game event information and context information. The game events and contexts in the database for a play may be inferred directly from the raw positional tracking data (e.g., a made or missed basket), or may be manually entered. Role information for players (can also be either inferred from the positional tracking data or entered separately. In embodiments of the system, a model for the database can then be trained by crafting features which encode game specific information based on the positional and game data and then calculating a prediction value (between 0 and 1) with respect to a classification metric (e.g., expected point value).

If there are a sufficient number of examples, the database model can be personalized for a particular player or game situation using those examples. In practice, however, a specific player or game situation may not be adequately represented by plays in the database. Thus, embodiments of the system find examples which are similar to the situation of interest-whether that be finding players who have similar characteristics or teams who play in a similar manner. A more general representation of a player and/or team is used, whereby instead of using the explicit team identity, each player or team is represented as a distribution of specific attributes. Embodiments of the system use the plays in the hash-table/playbook that were learned through the distributive clustering processes described above.

Further, while various aspects are discussed with respect to a single sport, such aspects are described are merely illustrative examples. Disclosed techniques are by no means limited to any sport in particular. For example, the present aspects can be implemented for other sports or activities, such as soccer, football, basketball, baseball, hockey, cricket, rugby, tennis, team sports, individual sports, and so forth.

FIG. 2 depicts an exemplary method 200 for generating a play graphic, in accordance with an aspect of the disclosed subject matter. The method 200 may be used to generate a single play graphic to be output and for use by a user and/or the method 200 may generate a playbook including a plurality of different play graphics. For example, a National League Football (NFL) team may generate a playbook including a plurality of play graphics for each upcoming game within a season (e.g., pre-season, season, and playoff). The method 200 may generate a first play graphic based on a first data set, at least one player performance data set, and associated threat score(s), where the first data set may include formation data, player location, player role, player route, or the like. A representative play may be selected from a set of template plays based on at least the formation data and player location data within the first data set. Upon selection of a representative play, the method 200 may receive player performance data and associated threat score for inclusion into the play graphic. The method 200 may generate a play graphic based on the above data. Additionally, the method 200 may be performed repeatedly to generate a playbook including a plurality of play graphics based on respective first data sets, player data, player performance data, and/or threat score(s). Each time the method 200 is performed, the first data set may include modified formation data and/or modified player location data or use different formation data and/or different player location data already included within the first data set which, in turn, may select a different representative play from the set of template plays. In one embodiment, the method 200 may include generating a play graphic based on a combination of original data (e.g., formation data, player location data, etc.), modified data (e.g., modified formation data, modified player location data, etc.), and/or different data (e.g., different formation data, different player location data, etc.).

The method 200 may start with step 210, receiving a first data set. The first data set, as described in with respect to FIGS. 5A and 5B below, may include player information including, but not limited to, player location, player identification, player position, player status, player role, raw data, subjective data, and/or observed data. The player identification, player position, player status, and/or play role relate to each of the plurality of players on a team. Some of the player information may overlap with one another, however other player information may provide additional details. Raw data may include a play concept, a play location, and/or a snap hash. The first data set may be gathered manually by one or more users, using one or more techniques described above with respect to tracking systems, or a combination thereof. For example, the first data set may be generated based on tracking data and/or event data generated in accordance with the techniques discussed herein. The first data may be stored in the data store 118 as described with respect to FIG. 1 above.

For example, in an embodiment the first data set may be gathered manually by one or more users. One or more users may review broadcast images of the football game. The broadcast image may include some or all portions of the game between multiple teams. The one or more users may record information with respect to each player on the field at any given time. For instance, during a kickoff, each player from each side may be recorded with their respective locations to one or more of each player, the field, or a combination thereof. During the game, records of each play, player information, player performance information, or a combination thereof may be recorded and stored within a data table for later access.

In another example, in an embodiment the first data set may be gathered using one or more techniques associated with tracking systems (e.g., the tracking system 102 and/or tracking system 116) as described above. The tracking systems may track and record a plurality of events during a football game. Each event (e.g., kickoff, pass, run, touchdown, timeout, field goal, etc.) may be recorded and analyzed. Each event may be determined to include player information, player performance information, game information, or a combination thereof. The information gathered may then be placed into data structures for later access.

At step 220, the first data set may be automatically compared to one or more templates. The one or more templates may include a plurality of template plays based on historical information data. The one or more templates may be populated or pre-populated based on historical information and may be stored in and retrieved from a database or storage component (e.g., the data store 118). The historical information data may include, but is not limited to, play information, player information, team information, down and distance information, and/or game information. The historical information data may be used to generate a plurality of template plays.

A template play may be a play that has particular characteristics. For example, a template play may include the location of each or a subset of players on offense and/or defense in a particular formation (e.g., I-formation or 4-3 defense). The template play may also include a player route for each player moving during the play (e.g., wide receiver running a post pattern route or a defensive cornerback dropping into coverage). The template plays may be generated using one or more sports machine learning models as discussed above.

One or more template plays may be provided, generated (e.g., based on historical or simulated data), or automatically populated. One or more template plays may be generated by a template machine learning model trained based on historical or simulated data sets, historical or simulated tracking data, historical or simulated play information, and/or the like. Tracking data may include pre-play and/or post-play player location information, player routes or movement pre-play through post-play, and the like. The template machine learning model may be trained using supervised, semi supervised, or unsupervised training and/or in accordance with the machine learning training discussed herein. The template machine learning model may be trained by, for example, correlating historical or simulated data sets including formation data included therein with tracking or play information. The formation data may include formation information prior to a given play being performed. The template machine learning model may be trained by predicting the specifics of a given play based on the formation information and comparing the prediction to the actual given play (e.g., based on the tracking data or play information). The template machine learning model may be iteratively refined based on multiple iterations of such predictions and refinement.

A trained template machine learning model may generate a plurality of template plays. The plurality of template plays may be associated with (e.g., using metadata, filters, silos, reference values, etc.) one or more given teams, one or more given sports, one or more given coaches, one or more given payers, one or more given event types, and/or the like. The plurality of template plays may be stored in a database or storage device, as discussed herein.

According to an embodiment, at step 220, the comparison of the first data set to one or more template plays may involve multiple technical processes for determining similarity and alignment. The computing device may extract feature vectors from both the first data set and each template play, where these feature vectors may represent quantifiable characteristics such as player positions, formation structures, movement patterns, and contextual game state information.

In some aspects, the system may implement a multi-dimensional comparison algorithm that evaluates spatial relationships between players. The computing device may calculate Euclidean distances between corresponding player positions in the first data set and template plays, generating distance matrices that quantify positional similarities. The system may also compute angular relationships between players to assess formation geometry, using vector calculations to determine the relative orientations and spacing patterns within each play configuration.

The computing device may apply weighted scoring mechanisms where different aspects of the play data receive varying importance levels. For example, the system may assign higher weights to key player positions such as quarterbacks or ball carriers, while applying lower weights to secondary players. In some cases, the weighting may be dynamically adjusted based on the specific sport, game situation, or play type being analyzed.

In some aspects, the comparison process may utilize machine learning similarity models trained to recognize play patterns. These models may employ techniques such as cosine similarity calculations between feature vectors, where the system computes the cosine of the angle between multi-dimensional vectors representing the first data set and each template play. The resulting similarity scores may range from 0 to 1, indicating the degree of alignment between plays.

The computing device may implement clustering algorithms to group similar template plays before comparison, reducing computational overhead. In some cases, the system may use k-means clustering or hierarchical clustering to organize template plays into categories based on formation types, player movements, or strategic objectives. This pre-clustering may allow the comparison algorithm to focus on the most relevant subset of template plays rather than evaluating the entire database.

The system may also incorporate temporal sequence matching for plays that involve time-based elements. The computing device may analyze the progression of player movements over time, using dynamic time warping algorithms to align sequences of different lengths and identify similar movement patterns despite variations in timing or execution speed.

In some aspects, the comparison process may include contextual matching where the system evaluates game situation parameters such as down and distance, field position, score differential, and time remaining. The computing device may apply conditional logic to filter template plays based on these contextual factors before performing detailed similarity calculations.

The computing device may generate composite similarity scores by combining multiple comparison metrics through mathematical aggregation functions. These functions may include weighted averages, geometric means, or more complex scoring algorithms that account for the relative importance of different play characteristics. The system may then rank template plays based on these composite scores to identify the most similar matches to the first data set.

At step 230, determining a representative play in response to comparing the first data set and the one or more templates. The representative play is determined based on comparing the first data set (e.g., player location, player position, etc.) and the similar characteristics, as described above, of the one or more templates. Each representative play may be assigned a unique ID for retrieval by organization computing system 104. The unique ID may include text characters, numeric values, or the like. The assigned unique ID associated with each of the representative plays may be used at step 230 or in any other step for selecting the appropriate or most closely identified representative play from the one or more templates. It will be understood that the unique ID is an example only. Each representative play may be designated by any applicable designation technique such as an identifier, a value, a relationship, and/or the like.

According to an embodiment, at step 230, the first data set may be extrapolated into its respective characteristics. Values, attributes, and/or other applicable data may be associated with each applicable characteristic. As an example, predetermined or dynamically determined superset of characteristics may be used to apply the respective characteristics of the first data set. The superset of characteristics may include characteristics in addition to the respective characteristics of the first data set. Based on comparing the superset of characteristics and the respective characteristics of the first data set, a subset of missing characteristics may be identified. The subset of missing characters may be maintained as blanks (e.g., null values) or may be filled in. Such subset of missing characteristics may be filled in based on predetermined (e.g., default) values, attributes, and/or other applicable data. Alternatively, a machine learning model (e.g., a filling machine learning model) may populate values, attributes, and/or other applicable data based on one or more other respective characteristics of the first data set. For example, a first missing characteristic may be populated based on an output of a machine learning model that provides a values, attributes, and/or other applicable data for the first missing characteristic using correlations, overlaps, and/or predictions made based on historical or simulated training data, as discussed herein.

According to an embodiment, at step 230, a representative play may be output by a machine learning model (e.g., a play machine learning model). The play machine learning model may be trained using a training data set that includes historical or simulated characteristics associated with the first data set, based on historical, simulated, or actual template plays, based on historical or simulated plays (e.g., using tracking data and/or event data), and/or the like. The play machine learning model may receive, as inputs, the first data set. The template plays may be provided as inputs to the play machine learning model or may be otherwise accessible by the play machine learning model. Based on such input data and/or access, the play machine learning model may output a template play that most closely aligns with the characteristics of the first data set. For example, the play machine learning model may generate the similarity score or ranking of template plays based on their alignment with the characteristics of the first data set. In some embodiments, a higher score may indicate a better alignment with the characteristics, while in other embodiments a lower score may indicate a better alignment with the characteristics. Thus, the template play with the highest score or lowest rank may be output by the play machine learning model. The play machine learning model may apply a threshold barrier such that if no template play has a score that meets or exceeds the threshold barrier for the first data set, then the play machine learning model outputs a trigger action. The trigger action may be a request for additional data, may be an input to the filling machine learning model to generate updated fillers for the first data set, may be an input to the template machine learning model to generate one or more template plays or updated template plays, and/or the like. The play machine learning model may output a representative play once a corresponding template play (e.g., one with a highest score, lowest rank, and/or meets the threshold barrier) is identified.

At step 240, at least one player performance data set may be received. The player performance data set may include, but is not limited to, information relating player performance for one or more players during one or more plays against one or more teams over a period of time. For example, a player (e.g., quarterback) may have data associated with running a quarterback sneak against a plurality of teams over the player's career. Each time the player has run this play, information about the success of the play (e.g., down and distance, carry yards, score conversion, first down conversion, defensive formation, etc.) may be stored. This information may be gathered manually and/or using tracking systems as described above. This information may be stored in and/or retrieved from a database or storage medium (e.g., the data store 118).

At step 250, a threat score may be determined. The player performance data set as described above, may be used by a sports machine learning model to determine the threat score of each player based on the representative play selected at step 230. A threat score may be a metric that quantifies a threat level of a player, team, and/or action is to the opposing team or player (e.g., the likelihood of a given negative result to a team or player based on a given a player, team, or action). A threat score may be a measure that evaluates a likelihood of performance beyond. For example, a team's threat score might indicate defensive weaknesses, while a player's score could represent how effectively they move an object into dangerous areas. The threat score may be dynamic based on the opposing team, down and distance, time remaining, and/or location on the field. For example, a wide receiver running a longer route (e.g., flag or post pattern) may have a higher threat score when the defense is in a 4-3 formation and a lower threat score when the defense is in a nickel formation. Alternatively, the same wide receiver may have a higher threat score when playing against a certain team versus other teams or even individual matchups.

At step 260, the at least one player performance data set and the associated threat score are combined. For example, the at least one player performance data set and the associated threat score are associated in a temporary database (e.g., the data store 118) using joint reference values. The joint reference values may be predetermined or generated for each set of the at least one player performance data set and the associated threat score. According to an embodiment, the at least one player performance data set and the associated threat score may be associated creating new values and/or attributes values based on both the at least one player performance data set and the associated threat score when combined.

At step 270, a play graphic is generated using one or more sports machine learning models. The play graphic may include each player and their respective position of the field (e.g., 11 for offense and 11 for defense), the formation of both the offense and defense, the routes of each player (e.g., wide receivers passing routes and running back(s) running and/or blocking routes), field indicia (e.g., yardage markers, hash markers, line-of-scrimmage, etc.), and/or threat score indicia for each player. According to an embodiment, a play graphic may be output by a machine learning model (e.g., a play graphic machine learning model). The play graphic machine learning model may be trained using a training data set that includes historical or simulated characteristics associated with the first data set, the at least one player performance data set, and the associated threat score based on historical, simulated, or actual play graphics, based on historical or simulated play graphics (e.g., using tracking data and/or event data), and/or the like. The play graphic machine learning model may receive, as inputs, the first data set, the at least one player performance data set, and the associated threat score. The play graphics machine learning model may also receive or take into consideration both the offensive and defensive characteristic attributes separately and together. The play graphics may be provided as inputs to the play graphics machine learning model or may be otherwise accessible by the play graphics machine learning model. Based on such input data and/or access, the play graphics machine learning model may output a play graphic that most closely aligns with the characteristics of the first data set, the at least one player performance data set, and the associated threat score.

For example, the play graphics machine learning model may generate an initial play graphics based on the template play and the at least one player performance data set as described above. The play graphics machine learning model may then update the play graphic based on the associated threat score for each of the at least one player performance data received. The play graphics machine learning model may further update the play graphic based on game context, such as current score, time remaining, importance of the game (e.g., pre-season vs. season vs. playoff), injury information, or the like. The play graphics machine learning model may output a play graphic once one or more of the updates have been made. As described herein, the play graphics machine learning model may generate a play graphic by performing the play graphic process sequentially or simultaneously.

At step 280, the play graphic is output for display on a user device. The method 200 may be performed with more or less steps than those described herein and be repeated as necessary. The method 200 may output a single play graphic for use by a user and additionally output a playbook including a plurality of play graphics for use by the user. Each play graphic within the playbook may be organized and/or arranged in a manner consistent with user preferences. For example, user preferences may be stored within a database (e.g., the data store 118) for use by the organization computing system 104. The organization computing system 104 may automatically arrange the plurality of play graphics according to stored user preferences. Alternatively, in some embodiment the user may manually arrange the plurality of play graphics according to the user preference. According to an embodiment, user preference data may include one or more criteria (e.g., play formation, down and distance, time remaining, etc.) for play graphic organization. Each play graphic within the playbook may be separated into one or more sub-sections based on the user preferences and the one or more criteria, each sub-section may include one or more play graphics associated with the one or more criteria of each sub-section. According to an embodiment, a user may manually input arrangement data into the organization computing system 104 for organizing each play graphic within the playbook. The user may input arrangement data including historical playbook organization data, play formation, down and distance, time remaining, etc.).

FIG. 3 depicts an exemplary method 300 for translating information used in the method 200 as described above. At step 310, the method 300 may translate the first data set and the at least one play performance data set into a graphic data set. The first data set and the at least one player performance data set may be similar to that as described above with reference to method 200. At step 320, the first data set may be translated from a first format to a second format, where the second format may be different from the first format. At step 330, the at least one player performance data set may be translated from a third format to the fourth format, where the fourth format may be different from the third format. At step 340, both the first data set in the second format and the at least one player performance data in the fourth format may be translated to a fifth format, where the fifth format may be different from the second format and the fourth format.

For example, the first, second, third, fourth, and fifth formats are to be read by a system. Each of the data sets (e.g., first data set, player performance, and graphic data set) contain different information. The translation into each format (e.g., second, third, and fifth) are to modify the originating format into a common format for combining the information from each data set into a usable data set for performing the generation of a play graphic. Each format may be read by a specific system, thus requiring additional translations to be read and perform the play graphic generation. For example, the first format may include an informational format based on manually inputted and/or automatically generated data (e.g., tracking system data). The second format may include a machine-readable format that may be provided as an input to one or more machine learning models. Converting the first format to the second format may include adding known associations between different characteristics. These associations may be more easily analyzed by the machine learning models in comparison to the first format. The second format may include, for example, a JSON file, XML file, or the like. Likewise, the third format, the fourth format, and the fifth format may include similar formats as described with respect to the first format and the second format. At step 340, as described above, translating the second format and the fourth format into the fifth format may include combining two different machine-readable formats into a single machine-readable format for use by a machine learning model specific to outputting content (e.g., play graphic machine learning model).

FIGS. 4A and 4B depict exemplary player performance data. FIG. 4A includes a first player performance data set 410, a second player performance data set 420, and a third player performance data set 430. The first player performance data set 410 may include offensive performance for each play. The first player performance data set 410 may include efficiency metrics associated with each play based on a plurality of data factors (e.g., opposing team, down and distance, location on the field, etc.). The second player performance data set 420 may include a specific player and their associated performance data (e.g., success rate of a wide receiver against different defensive coverages). The third player performance data set 430 may include success rates for a particular offense based on an included player. For example, when a particular quarterback is included, run percentage, run success, run yards per play, pass percentage, pass success rate, and/or passing yard per play may all be included and represented in the third player performance data set 430. FIG. 4B includes a fourth player performance data set 440 and a fifth player performance data set 450. The fourth player performance data set 440 may include, for example, run percentages for a particular running back. The fourth player performance data set 440 data may include the percentages and the number of tries for each direction (e.g., F or B). For example, the fourth player performance data set 440 indicates out of 404 tries, the particular player has a slight preference (e.g., 204 vs. 200 or 50.5% vs. 49.5%) to run in the F direction versus the B direction. The fifth player performance data set 450 may include the percentages and the number of tries for each direction (e.g., right or left). For example, the fifth player performance data set 450 indicates out of 404 tries, the particular player has a preference (e.g., 220 vs. 184 or 54.46% vs. 45.54%) to run towards the right side versus the left side.

FIGS. 5A and 5B depict exemplary first data sets that may be stored in the data store 118. As described above, first data set may include, but is not limited to, player location, player position, player role, subjective data, and/or observed data. FIG. 5A includes a first data set 510 which may be stored in data store 118. The first data set 510 may include the names, player role, player position (e.g., QB, WR, etc.), hand on ground (HG), gap assignment (GAP), and/or coverage assignment (COV). FIG. 5B includes another exemplary first data set 520. The first data set 520 may include similar information as described above with reference to FIG. 5A. FIG. 5B may further include players in motion, play start and end time, formation, number of defensive players on the line-of-scrimmage, number of players in the box, and/or players rushed.

FIG. 6 depicts an exemplary play graphic 600, in accordance with an aspect of the disclosed subject matter. The play graphic 600 may include one or more offensive player(s) 605, one or more defensive player(s) 610, a threat score 615 corresponding to each of the one or more offensive player(s) 605 and the one or more defensive player(s) 610, a player route 620 corresponding to each of the one or more offensive player(s) 605 and the one or more defensive player(s) 610, and a field indicia 625. Each of the one or more offensive player(s) 605 may include a respective color, number, shape, indicia, or a combination thereof to identify each of the one or more offensive player(s) 605 of a respective team. Each of the one or more defensive player(s) 610 may include a respective color, number, shape, indicia, or a combination thereof to identify each of the one or more defensive player(s) 610 of a respective team. The threat score 615 may be associated with each of the one or more offensive player(s) 605 and/or the one or more defensive player(s) 610. The threat score 615 may include an indicia (e.g., graphic, text, color, shape, or a combination thereof) to signify the threat level of each of the one or more offensive player(s) 605 and each of the one or more defensive player(s) 610. For example, the threat score 615 may include a graphic with a range of colors (e.g., green to red) to indicate the level of threat each player is for that particular play, wherein one color, for example green, may represent a lower threat score 615 than a second color, for example red. The threat score 615, as described above, may be dynamic based on the current play, other players on the field, opposing team, down and distance, location on the field, weather etc. For example, weather conditions may influence the determination of the threat score 615 by affecting player performance capabilities and play execution probabilities. In some aspects, when determining the threat score 615 for each of the one or more offensive player(s) 605 and each of the one or more defensive player(s) 610, meteorological data such as wind speed, precipitation, temperature, and humidity levels may be incorporated to adjust each threat assessment. For example, strong crosswinds may reduce the threat score 615 for passing plays while increasing the threat score 615 for running plays. In addition, wet field conditions may affect player mobility and ball handling, thereby modifying individual player threat calculations. The computing device may dynamically update the threat score 615 based on real-time weather data to provide more accurate predictive play graphics that account for environmental factors impacting player and team performance. The player route 620 may include a respective color, shape, indicia, or a combination thereof to identify the particular moving route of each of the one or more offensive player(s) 605 or each of the one or more defensive player(s) 610. The player route 620 may indicate the route taken by each player during the performance of the play. The field indicia 625 may include a respective color, number, shape, indicia, or a combination thereof to identify one or more identifiable locations of the field. For example, the field indicia 625 may include one or more hash marks, one or more yard markers, and/or one or more direction indicators.

In an embodiment, the play graphic 600 may include video and/or animations of each player (e.g., the one or more offensive player(s) 605 and the one or more defensive player(s) 610). The animation of the play graphic 600 may include video and/or realistic 3D renderings (e.g., video game) of each component (e.g., the one or more offensive player(s) 605, the one or more defensive player(s) 610, etc.) moving within the play graphic 600. For example, the play graphic 600 may provide an animation of each of the one or more offensive player(s) 605 moving along their respective player route 620. In addition to each of the one or more offensive player(s) 605 being animated, each of the one or more defensive player(s) 610 may be similarly animated. For example, in response to the one or more offensive player(s) 605 moving along their player route 620, the one or more defensive player(s) 610 may similarly move in a defensive player route. The defensive player route may be in response to the one or more offensive player(s) 605 moving along their respective player route 620. Each route may be based on at least one of historical data, tracking data, event data, and/or predictive data. If the player route 620 is based on historical or event data, the playa graphic 600 may be animated in such a way to correspond to an actual play that took place between two different teams and their respective players, highlighting the outcome of the play as the play happened.

In an embodiment, the play graphic 600 may be based on predictive data. The one or more machine learning models disclosed herein may predict the outcome of each player (e.g., the one or more offensive player(s) 605 and the one or more defensive player(s) 610) based on historical information (e.g., archived data). For example, a player may have historical data including statistics against particular teams, players, at particular times in the season, or the game. Generating the play graphic 600 using predictive data may include displaying each of the one or more offensive player(s) 605 and their respective player route 620 based archived event data associated with the one or more offensive player(s) 605 and the one or more defensive player(s) 610 on the field. In an example, the predictive data may include the use of an offensive player's success on 1^st downs, within the redzone, in a particular formation, against a particular matchup, or the like to generate the play graphic 600 with each threat score 615 displayed. The play graphic 600 may then be animated in such a way to display the predictive outcome of the play with each particular player (e.g., offensive and defensive player) moving in a manner based on each of their respective historical and/or archived data sets.

In some aspects, the video and/or animations may be generated through a multi-stage computational process that leverages the sports play graphics system described herein. The computing device may utilize the generated sports play graphic as a foundational data structure, where each of the one or more offensive player(S) 605 and each of the one or more defensive player(s) 610 may represent a data object containing positional coordinates, movement vectors, and associated metadata including the threat score 615 and player route 620 information.

The animation generation process may begin with the computing device extracting temporal sequence data from the first data set and player performance data set. In some cases, the system may interpolate between discrete data points to create smooth movement trajectories for each player. The computing device may apply mathematical interpolation algorithms, such as spline interpolation or Bezier curve calculations, to generate intermediate positional coordinates between known player positions at different time intervals.

For historical or event data-based animations, the computing device may access stored tracking data that includes frame-by-frame positional information for each player throughout the duration of a play. The system may process this tracking data to extract movement patterns, acceleration profiles, and directional changes for each player. In some aspects, the computing device may synchronize multiple player movements by aligning temporal markers within the tracking data, ensuring that all player animations progress in coordination with the actual timing of the recorded play.

When generating predictive animations, the computing device may employ machine learning models trained on historical player movement patterns and performance statistics. These models may analyze the current game state, player positions, and contextual factors to predict probable movement trajectories for each player. The system may generate probability distributions for potential player movements and select the most likely paths based on the trained model outputs.

The rendering process may involve the computing device converting the calculated movement data into visual representations. In some cases, the system may utilize graphics processing algorithms to render 2D or 3D representations of players, field elements, and movement indicators. The computing device may apply texture mapping, lighting calculations, and perspective transformations to create realistic visual representations of the players and playing field.

For dynamic threat score visualization, the computing device may continuously update the visual indicators associated with each player based on changing game conditions throughout the animation sequence. The system may calculate updated threat scores at regular intervals during the animation and modify the visual properties of the threat score 615 indicators accordingly, such as adjusting color intensity, size, or opacity based on the calculated threat levels.

The computing device may implement frame rate optimization techniques to ensure smooth animation playback. In some aspects, the system may generate intermediate frames through motion blur calculations or temporal anti-aliasing to reduce visual artifacts during rapid player movements. The animation engine may also incorporate physics-based calculations to simulate realistic player interactions, collision detection, and momentum conservation during player movements.

FIG. 7 depicts an example flowchart for generating tracking and/or event data, in accordance with an aspect of the disclosed subject matter. At step 710, one or more inputs by a user or system (e.g., a user query or an auto generated query) may be received. The one or more inputs may include a description of a sporting action (e.g., a play), a question, a team or player, etc. and may be in a text format, audio format, visual format, event/tracking data format, or the like, as discussed herein (e.g., in reference to FIGS. 8 and 9). For example, the client device 108 may be executing the application 130 providing an interactive user interface. A user may make a selection to input a query (e.g., “show me the last touchdown scored between Kansas City Chiefs and Philadelphia Eagles in Super Bowl LIX”) using one or more input techniques.

At step 712, one or more metadata items related to the description may be extracted, for example, the mapping module 124 may extract metadata items (e.g., contextual items) from the user input (e.g., description) to generate one or more connections and/or associations relating to a game or sporting event. The one or more metadata items may correlate aspects of the description with features that can be mapped to an event stream. An event stream may be historical or simulated tracking and/or event data determined based on a broadcast or in-venue sports stream. Accordingly, at step 712, a user input query (e.g., description) may be translated into a format that allows mapping the input query to an event stream.

For example, at step 712, the mapping module 124 may use a generative model to convert the description received as a query into one or more metadata items associated with one or more sporting events. The metadata items may be specific items provided in the description (e.g., player, team, sporting event, etc.) and/or may be items identified by the generative model to be associated with the specific items provided in the description (e.g., specific plays, opponent information, types of event actions, types of tracking data, etc.). Accordingly, the generative model disclosed herein that is trained based on historical or simulated sport event information may be used to generate metadata items that meet a threshold correlation value to the description. In doing so, the generative model may exclude unrelated metadata items, allowing for faster and more efficient subsequent operations limited to the identified metadata items.

At step 714, the metadata items may be mapped to one or more event streams, as further discussed in reference to FIG. 9. The mapped event stream and/or contextual information associated with the mapped event stream may be provided to a multimodal sports LLM model. For example, after determining one or more contextual items, the mapping module 124 may determine one or more connections and/or associations to the event stream based on the determined contextual items. In doing so, the mapping module 124 may translate the user input query from a first format into a second format recognizable by one or more components and/or machine learning models. The second format may include the connections and/or associations to the event stream. Accordingly, at step 714, one or more event streams corresponding to the query may be identified based on the mapping.

The one or more event streams as well as the connections and/or associations determined at step 714 may be provided to a multimodal sports LLM model, as discussed above. The multimodal sports LLM model may be trained to determine content items from the event streams.

At step 716, the multimodal sports LLM model may apply the connections and/or associations identified based on the query to the one or more event streams. Applying the connections and/or associations to the event streams may include, for example, assigning a correlation score to subsets of the event streams. For example, the multimodal sports LLM model may assign attributes to each subset of the event streams. The attributes may be based on the tracking data and/or event data corresponding to each applicable subset of the event streams. The attributes may cluster the tracking data and/or event data by the actions (e.g., play types, players, teams, actions, events, scores, passes, etc.) performed therein. The attributes may be determined by identifying the actions performed in each respective subset of the event streams. The multimodal sports LLM model may then assign a correlation score to each subset of the event streams and the connections and/or associations identified based on the query. For example, the query may call for touchdowns scored in a given game. At step 714, connections and/or associations associated with a touchdown being scored may be identified. These connections and/or associations may, for example, include proximity of an offensive player to the endzone (e.g., based on tracking data), the movement of a ball and/or player in proximity to the endzone (e.g., based on tracking data), the accordance of a scoring event (e.g., based on tracking data or excitement data), or the like. The multimodal sports LLM model may assign a high correlation score to the subset of the event streams that indicate a goal scored or attempted based on the attributes associated with each respective subset of the event streams. The correlation score may be determined based on a degree of overlap or correlation between the attributes for a given subset of event stream and the connections and/or associations identified based on the query. For example, a subset of an event stream that is assigned a goal scored attribute may have a higher correlation score in comparison to a subset of an event stream that is assigned a pass made attribute based on respective tracking and/or event data.

At step 716, the multimodal sports LLM model may identify content items corresponding to the subset of event streams that have a correlation score higher than a threshold correlation score. Continuing the example above, a subset of an event stream that has attributes associated with a goal scored may have a correlation score higher than a threshold correlation score. Accordingly, video and/or audio content associated with that subset of the event stream may be identified by the multimodal sports LLM model as content items for output. The content items may further include a description of the subset of the event stream generated by the multimodal sports LLM model to describe the actions performed in that subset of the event stream. For example, the multimodal sports LLM model may translate the video and/or audio data in the subset of the event stream into a summary or analysis of the actions performed in that subset of the event stream (e.g., based on the audio/video feed, based on broadcast information, based on associated tracking data, based on associated event data, etc.).

Accordingly, at step 716, one or more content items that relate to the one or more mapped event streams (or subsets thereof) may be output by the multimodal sports LLM. As discussed herein, the multimodal sports LLM may be trained to output actual or generated event, data, tracking data, video content, audio content, summaries, analysis, and/or other content that correlate with the event streams mapped at step 714. As discussed above, mapped event streams may provide features, criteria, and/or boundaries for the information requested via the query, in a format that allows multimodal sports LLM to output a response to the query.

At step 716, the one or more content items output by the multimodal sports LLM may include actual or generated event, data, tracking data, video content, audio content, summaries, analysis, and/or other content in response to the user query. The actual or generated event, data, tracking data, video content, audio content, summaries, analysis, and/or other content may include player and/or object position information, movement information, trends, changes, plays, event actions, and/or the like in response to the user query.

At step 718, the actual or generated content items output by the multimodal sports LLM may be provided to the user (e.g., via a user device). The output may be provided as a visual display depicting the player and/or object position information, movement information, trends, changes, summaries, analysis, and/or the like in response to the user query. For example, the player and/or object information may be provided in a video format that depicts a play corresponding to the player and/or object information. The video may correspond to the identified subset of one or more event streams that exceed the correlation threshold and may progress from the beginning to an end of the play and may include indicators representing the player and/or object information. As another example, the player and/or object information may be provided in an image format. The image may depict player and/or object information over the course of a given play.

At step 718, the actual or generated content items may be formatted in a manner or order determined by the multimodal sports LLM based on the query. For example, where multiple subsets of event streams meet the correlation threshold, the multimodal sports LLM may identify a priority order for outputting the content streams generated based on the multiple subsets of event streams. The priority order may be determined by applying weights to each of the multiple event streams (and corresponding content streams). The weights may be generated by the multimodal sports LLM based on the description of the query. The multimodal sports LLM may be trained to determine such weights based on training data that includes historical or simulated event streams, subsets of event streams, queries, weights, content streams, and/or the like. Accordingly, the multimodal sports LLM may be trained to prioritize content streams that most correlate to the query and output the content streams in an order based on such prioritization (e.g., using the weights described above).

In addition, a user may input additional inputs (e.g., text, audio, drawing, etc.) to make further refinements of the inputted description. After each additional input, the system may further extract one or more additional metadata items relating to the refinements of the description. Upon determining the one or more additional metadata items, the system may perform steps similar to steps 714 to 718 as described above. This process (e.g., step 710 through step 718) may be repeated as necessary to produce a display as requested by the user.

FIGS. 8 and 9 depict flow diagrams of an exemplary method for using the machine learning model to generate sports tracking data, in accordance with an aspect of the disclosed subject matter. FIG. 8 describes an exemplary method 800 for using the machine learning model. The method 800 starts with receiving input(s) 801 (e.g., user inputs) from a client device (shown at inputs 805-825) at a multimodal sports LLM 830. The input(s) 801 may be in the form of a text description 805, an event stream 810, a drawing 815, an audio 820, or a video 825. For example, the client device 108 may execute the application 130 and provide a user interface for entering one or more inputs (e.g., input(s) 801). The user interface may include a dialogue box for inputting, via drag and drop, one or more of text, images, video clips, audio chunks, or the like. In addition, the dialogue box may include an interface object for capturing live audio and/or video from the user, for example, a user may capture an image of a broadcast event stream from a TV or user device. Additional inputs may be received throughout the method 800 to further refine the query.

Based on the received input(s) 801, the system may determine contextual and intentional information to describe the sports tracking information requested. The contextual and/or intentional information may be defined at the time of the received input(s) 801, for example defined by the user in addition to the query, and/or the contextual and/or intentional information may be inferred by the one or more models. For example, the user may input a query or a question using one or more input formats as described above, the query may include contextual and/or intentional information. The multimodal sports LLM 830 may include a mapping module (e.g., the mapping module 124 of FIG. 1) configured to use one or more mapping models based on the input type(s) to determine contextual and intentional information from the received input(s) 801. Each mapping model used may be trained for the specific input type, for example, a drawing interface may be mapped to a textual question, an event stream, tracking data, an image, video, or the like. The drawing mapping model may be configured to receive user input in a first format (e.g., drawings) and transform the received input into a second format (e.g., the one or more second game files 126) usable by the multimodal sports LLM 830. The input received by the multimodal sports LLM 830 may be configured to infer contextual and/or intentional information based on the inputted query. If the contextual and/or intentional information is received as part of the query, the multimodal sports LLM 830 may further compare the entered contextual and/or intentional information to the inferred contextual and/or intentional information to determine the accuracy of the multimodal sports LLM 830. The multimodal sports LLM 830 may use the comparison of the contextual and/or intentional information to modify the training of each additional multimodal sports LLM 830 to generate a more accurate inference for future inputs. In addition, the system may further include a conversation layer configured to clarify or request additional information from the user to determine the contextual and/or intentional information relating to the inputted query.

Based on the contextual and intentional information, the method 800 may determine sports tracking data that matches the inputs received using the multimodal sports LLM 830. The mapping module may determine, based on the contextual and intentional information, one or more connections and/or associations to one or more sports tracking data. The sports tracking data may include present and/or historical information related to individual players and/or teams. Upon the one or more sports tracking data being determined, the system may output the tracking and event data 835. In some embodiments, the tracking and event data 835 may be output to the user's client device (e.g., the client device 108). The system may present the sports tracking and event data in one or more of an audio and/or text commentary 840, one or more visualizations 845, one or more similar plays 850, and a play analysis 855. For example, based on a user query (e.g., text or audio) to display the highlights of Manchester United, the system may limit the highlight to goals scored against Liverpool. The mapping module may determine contextual and intentional information associated with the query and determine connections and associations to sports tracking data. Upon retrieving the sports tracking data from the tracking data system (e.g., the tracking data system 116 of FIG. 1) and/or the data store (e.g., the data store 118 of FIG. 1), the system may output one or more clips of each goal scored by Manchester United against Liverpool. In addition, the user may input further modifiers or queries to refine the contextual and intentional information to further narrow the selected sports tracking data.

For example, a user may be watching a broadcast event stream of a football game on a TV or other user device (e.g. the client device 108). The user may capture an image of the current play from the user device. The user may use the captured image as an input into method 800 to generate one or more visualizations 845. The method 800 may determine using the multimodal sports LLM 830 to determine contextual and intentional information related to the captured image. The method 800 may determine the particular play based on tracking data associated with the broadcast event stream to generate a play graphic (e.g., 600 of FIG. 6) as described above as the one or more visualizations 845. The play graphic may include each player, their respective routes, threat scores, or the like, providing a static or dynamic visualization. The play graphic may further be animated (e.g., dynamic) such that each player (e.g., offense and defense) may move according the actual events of the broadcast event stream. In an embodiment, the play graphic may be predictive to provide a dynamic or animated play graphic according to predictive data as described in further details below.

As another example, a user may input information (e.g., text, image, audio, etc.) to find a particular play. The user may capture an image via the client device 108 (e.g., TV, mobile phone, virtual reality (VR) headset, etc.) as an input for the method 800. The method 800 may determine a particular play associated with one or more teams, one or more individual players, point in the season (e.g., preseason, regular season, etc.). The multimodal sports LLM 830 may associate the contextual and intentional information determined from the input with the particular play. The method 800 may provide the play graphic via one or more visualizations 845. The play graphic may include one or more of an animation, highlighting one or more players of interest, or provide additional details associated with the play graphic. For example, the play graphic may include textual descriptions of the outcome of the play (e.g., play-by-play commentary), team information and statistics, player information and statistics, or the like. The play graphic may further provide an analysis of the play and one or more plays leading up to or following the particular play of interest. The additional plays may also be animated to provide additional context to the user.

FIG. 9 depicts an alternative technique for outputting sports tracking and event data. As similarly described in FIG. 8, method 900 may receive input(s) 901 (such as inputs 905-925). Such input(s) 901 may be, for example, user inputs. The input(s) 901 may be in the form of a text description 905, an audio 910, a drawing 915, a video 920, or event stream 925. Based on the received input(s) 901, a preprocessing procedure may occur. Preprocessing may include a text/audio model 960, a drawing model 965, or a video model 970. Preprocessing may augment the received input(s) 901 by determining what sports tracking data events are being requested via the user query. For example, the system may determine contextual information associated with the query entered by the user during the preprocessing procedure. In addition to contextual information, the system may identify intentional information associated with the query, for example a question posed by the user. With this additional information, the system can better map or match sports tracking data using a multimodal sports LLM 930. As described above, the system then outputs the tracking and event data 935 to a user device (e.g., the client device 108). The system may present the tracking and event data 935 in one or more of audio and/or the text commentary 940, the one or more visualizations 945, the similar plays 950, and the play analysis 955.

The preprocessing (960-970) may be incorporated into the mapping module (e.g., the mapping module 124 of FIG. 1) of the multimodal sports LLM 930 as described in FIG. 8 or as a separate component as described herein. For example, the preprocessing (960-970) may include one or more models configured to map text and/or audio inputs to an event stream (e.g., using a database lookup or other identification process). Each of the mapping models may be specific to the input type, for example, one or more for drawing inputs or one or more for video inputs. The mapping models incorporated into the preprocessing (960-970) may be configured to transform the received input(s) 901 from a first format (e.g., drawings or video) into a second format (e.g., event stream or tracking stream) for use by the multimodal sports LLM 930.

Accordingly, during such preprocessing, text and/or audio inputs are converted into a tracking and/or event data format that may be provided to the multimodal sports LLM 930 to more efficiently correlate with historical event and/or tracking data in order to generate output event and/or tracking data. Similarly, the preprocessing (960-970) may include mapping visual data (e.g., a drawing, video data, tracking data, etc.) to an event stream (e.g., using a lookup or other identification process). Visual data may be analyzed and converted into a tracking and/or event data format that may be provided to the multimodal sports LLM 930 to more efficiently correlate with historical event and/or tracking data in order to generate output event and/or tracking data.

As depicted in FIG. 9, the event stream 925 input data may not require preprocessing as such data may already be in a format that may be provided to the multimodal sports LLM 930 to more efficiently correlate with historical event and/or tracking data in order to generate output event and/or tracking data.

For example, a user requesting a play graphic (e.g., play the graphic 600 of FIG. 6) using a drawing as an input to the method 900. The method 900 may determine contextual and intentional information associated with the drawing to determine whether or not the particular play actually happened, using historical and/or archived data, or if the user is requesting to predict the outcome of the inputted play. If the particular play has happened, the play graphic may be generated as discussed above via one or more visualizations 945. The play graphic may include each team, player(s), player routes, as described above. The play graphic may be animated to display the outcome of the particular play. If the method 900 determines the particular play has not happened based on the contextual and intentional inputs, the method 900 may generate a predictive play graphic. The predictive play graphic may include substantially all of the same features as previously described. The predictive play graphic may be generated using historical and/or archived data associated with each team and/or individual player. The historical and/or archived data may include data associated with statistics and/or probabilities associated with each individual player and team based on historical events. For example, a quarterback may have statistics based on previous meetings with a particular team that includes a pass rating of 75%. Based on the input information, the predictive play graphic may include a higher threat level for the quarterback based on the opposing team and/or player matchups. Each of the statistics and probabilities may assist in developing the predictive play graphic for accurately displaying the outcome of a particular play.

In another example, a user may provide an input using an audio description (e.g., voice input) or play-by-play commentary to describe the play of interest. The method 900 may determine based on contextual and intentional information the play of interest has previous occurred. The method 900 may generate a play graphic (e.g., the play graphic 600 of FIG. 6) based on the input information. As similarly described above, the multimodal sports LLM 930 may generate and the one or more visualizations 945 may provide the play graphic based on the input information from the user. The play graphic may include each player and their respective positions on the field, each player's associated threat score, each player's associated player route, and any additional information requested based on the user input (e.g., player statistics, game information, play outcome, game outcome, etc.). The play graphic may be static and/or dynamic to provide a visual of the occurrence of the play as the play occurred.

FIG. 10 depicts a flow diagram for training a machine learning model, in accordance with an aspect of the disclosed subject matter. As shown in a flow diagram 1000 of FIG. 10, a training data 1012 may include one or more of stage inputs 1014 and known outcomes 1018 related to a machine learning model to be trained. The stage inputs 1014 may be from any applicable source including a component or set shown in the figures provided herein. The known outcomes 1018 may be included for machine learning models generated based on supervised or semi-supervised training. An unsupervised machine learning model might not be trained using the known outcomes 1018. The known outcomes 1018 may include known or desired outputs for future inputs similar to or in the same category as the stage inputs 1014 that do not have corresponding known outputs.

The training data 1012 and a training algorithm 1020 may be provided to a training component 1030 that may apply the training data 1012 to the training algorithm 1020 to generate a trained machine learning model 1050. According to an implementation, the training component 1030 may be provided comparison results 1016 that compares a previous output of the corresponding machine learning model to apply the previous result to re-train the machine learning model. The comparison results 1016 may be used by the training component 1030 to update the corresponding machine learning model. The training algorithm 1020 may utilize machine learning networks and/or models including, but not limited to a deep learning network such as Deep Neural Networks (DNN), Convolutional Neural Networks (CNN), Fully Convolutional Networks (FCN) and Recurrent Neural Networks (RCN), probabilistic models such as Bayesian Networks and Graphical Models, and/or discriminative models such as Decision Forests and maximum margin methods, or the like. The output of the flow diagram 1000 may be a trained machine learning model 1050.

A machine learning model disclosed herein may be trained by adjusting one or more weights, layers, and/or biases during a training phase. During the training phase, historical or simulated data may be provided as inputs to the model. The model may adjust one or more of its weights, layers, and/or biases based on such historical or simulated information. The adjusted weights, layers, and/or biases may be configured in a production version of the machine learning model (e.g., a trained model) based on the training. Once trained, the machine learning model may output machine learning model outputs in accordance with the subject matter disclosed herein. According to an implementation, one or more machine learning models disclosed herein may continuously update based on feedback associated with use or implementation of the machine learning model outputs.

Machine Learning for Team/Player Predictions

According to embodiments disclosed herein, a transformer neural network may receive inputs (e.g., tensor layers), where each input corresponds to a given player, team, or game. The transformer neural network may output generated predictions for one or more given players or teams based on such inputs. More specifically, the transformer neural network may output such generated predictions for a given player or team based on inputs associated with that given player or team and further based on the influence of one or more other players or teams. Accordingly, predictions provided by a transformer neural network, as discussed herein, may account for the influence of multiple players and/or teams when outputting a prediction for a given player and/or team.

The system described herein may include a machine learning system configured to generate one or more predictions. In some examples, the system may incorporate a transformer neural network, graphical neural network, a recurrent neural network, a convolutional neural network, and/or a feed forward neural network. The system may implement a series of neural network instances (e.g., feed forward network (FFN) models) connected via a transformer neural network (e.g., a graph neural network (GNN) model). Although a transformer neural network is generally discussed herein, it will be understood that any applicable GNN, or other neural network that may utilize graphical interpretations, may be used to perform the techniques discussed herein in reference to a transformer neural network.

The transformer-based neural network may include a set of linear embedding layers, a transformer encoder, and a set of fully connected layers. The set of linear embedding layers may map component tensors of received inputs into tensors with a common feature dimension. The transformer encoder may perform attention along the temporal and agent dimensions. The set of fully connected layers may map the output embeddings from a last transformer layer of the transformer encoder into tensors with requested feature dimension of each target metric.

The transformer-based neural network may be configured to receive input features through the set of linear embedding layers. The input features may be received at different resolutions and over a time-series. The input features may relate to player features, team features, and/or game features. Input features may be input into the linear embedding layers as a tuple of input tensors. For example, a tuple of three tensors may be provided where the first tensor corresponds to all players in a match, a second tensor corresponds to both teams in the match, and the third tensor corresponds to a match state.

Examining the set of linear embedding layers, the linear embedding layers may contain a linear block for each input tensor of the tuple, and each block may map an input tensor to a tensor with a common feature dimension D. The output of the linear embedding layer may be a tuple of tensors, with a common feature dimension, which can be concatenated along the temporal and agent dimension to form a single tensor.

The transformer encoder may be configured to receive the single tensor from the linear embedding layers. The transformer encoder may be configured to learn an embedding that is configured to generate predictions on multiple actions for each agent (e.g., each player and/or team). The transformer encoder may include a series of axial transformer encoder layers, where each layer alternatively applies attention along the temporal and agent dimensions. The transformer encoder may include layers that alternate between temporally applying attention to sequences of action events, and applying attention spatially across the set of players and teams at each event time-step. The transformer encoder may include axial encoder layers configured to accept a tensor from the linear layers and apply attention along the temporal dimension, then along the agent dimension.

The attention mechanism that is implemented by the transformer encoder layers may have a graphical interpretation on a dense graph where each element is a node, and the attention mask is the inverse of the adjacency matrix defining the edges between the nodes (the absence of an attention mask thus implies a fully-connected graph). In the case of the axial attention used here, with the attention mask on the temporal (row) dimension, the nodes in the graph can be arranged in a grid, and each node may be connected to all nodes in the same column, and to all previous nodes in the same row. Attention, in this case, may be message-passing where each node can accept messages describing the state of the nodes in its neighborhood, and then update its own state based on these messages. This attention scheme may mean that when making a prediction for a particular player, the model may consider (e.g., attend to) the nodes containing the previous states of the player along the time-series; and the state nodes of the other players, team and the current game state in the current time-step. It may not be necessary for the nodes to be homogeneous—beyond having the same feature dimension—and thus a node that represents a player can accept messages from a node that represents at team, or from the player's strength node. The model may therefore learn the interactions between agents, and ensure consistent predictions for each agent along the time-series. The output of the transformer encoder layers may be a tensor (e.g., an output embedding).

The final layers of the transformer-based neural network may be the fully connected layers. These layers may map the output embedding of the final transformer layer of the transformer encoder to the feature dimension of each target metric. The final layers may output a target tuple that contains tensors for each of a set of modeled actions for each player and/or team. For example, the modeled action may be an empirical estimate of distributions for sport statistics such as number of shots taken, number of goals, number of passes, etc.

The training of the transformer-based neural network may include choosing a corresponding loss function for the distribution assumption of each output target. For example, the loss function may be the Poisson negative log-likelihood for a Poisson distribution, binary cross entropy for a Bernouilli distribution, etc. The losses may be computed during training according to the ground truth value for each target in the training set, and the loss values may be summed, and the model weights may be updated from the total loss using an optimizer. The learning rate may have been adjusted on a schedule with cosine annealing, without warm restarts.

Sports Machine Learning

As discussed herein, one or more machine learning models may be trained to understand a sports language. Accordingly, machine learning models disclosed herein are sports machine learning models. Such sports machine learning models may be trained using sports related data (e.g., tracking data, event data, etc., as discussed herein). A sports machine learning model trained to understand a sports language based on sports related data may be trained to adjust one or more weights, layers, nodes, biases, and/or synapses based on the sports related data. A sports machine learning model may include components (e.g., a weights, layers, nodes, biases, and/or synapses) that collectively associate one or more of: a player with a team or league; a team with a player or league; a score with a team; a scoring event with a player; a sports event with a player or team; a win with a player or team; a loss with a player or team; and/or the like. A sports machine learning model may correlate sports information and statistics in a competition landscape. A sports machine learning model may be trained to adjust one or more weights, layers, nodes, biases, and/or synapses to associate certain sports statistics in view of a competition landscape. For example, a win indicator for a given team may automatically correlated with a loss indicator for an opposing team. As another example, a score static may be considered a positive attribution for a scoring team and a negative attribution for a team being scored upon. As another example, a given score may be ranked against one or more scores based on a relative position of the score in comparison to the one or more other scores.

A sports machine learning model may be trained based on sports tracking and/or event data, as discussed herein. Such data may include player and/or object position information, movement information, trends, and changes. For example, a sports machine learning model may be trained by modifying one or more weights, layers, nodes, biases, and/or synapses to associate given positions in reference to the playing surface of venue and/or in reference to none or more agents. As another example, a sports machine learning model may be trained by modifying one or more weights, layers, nodes, biases, and/or synapses to associate given movement or trends in reference to the playing surface of venue and/or in reference to none or more agents. As another example, a sports machine learning model may be trained by modifying one or more weights, layers, nodes, biases, and/or synapses to associate sporting events with corresponding time boundaries, teams, players, coaches, officials, and environmental data associated with a location of corresponding sporting events.

A sports machine learning model may be trained by modifying one or more weights, layers, nodes, biases, and/or synapses to associate position, movement, and/or trend information in view of a sports target. A sports target may be a score related target (e.g., a score, a goal, a shot, a shot count, a point, etc.), a play outcome (e.g., a pass, a movement of an object such as a ball, player positions, etc.), a player position, and/or the like. A sports machine learning model may be trained in view sports targets, play outcomes, player positions, and/or the like associated with a given sport (e.g., soccer, American football, basketball, baseball, tennis, golf, rugby, hockey, a team sport, an individual sport, etc.). For example, a soccer based sports machine learning model may be trained to correlate or otherwise associate player position information in reference to a soccer pitch. The soccer based sports machine learning model may further be trained to correlate or otherwise associate sports data in reference to a number of players and sports targets specific to soccer.

According to aspects, one or more given sports machine learning model types (e.g., generative learning, linear regression, logistic regression, random forest, gradient boosted machine (GBM), deep learning, graph neural networks (GNN) and/or a deep neural network) may be determined based on attributes of a given sport for which the one or more machine learning models are applied. The attributes may include, for example, sport type (e.g., individual sport vs. team sport), sport boundaries (e.g., time factors, player number factors, object factors, possession periods (e.g., overlapping or distinct), playing surface type (e.g., restricted, unrestricted, virtual, real, etc.) player positions, etc.

According to aspects, a sports machine learning model may receive inputs including sports data for a given sport and may generate a matrix representation based on features of the given sport. The sports machine learning model may be trained to determine potential features for the given sport. For example, the matrix may include fields and/or sub-fields related to player information, team information, object information, sports boundary information, sporting surface information, etc. Attributes related to each field or sub-field may be populated within the matrix, based on received or extracted data. The sports machine learning model may perform operations based on the generated matrix. The features may be updated based on input data or updated training data based on, for example, sports data associated with features that the model is not previously trained to associate with the given sport. Accordingly, sports machine learning models may be iteratively trained based on sports data or simulated data.

Machine Learning Models.

As used herein, a “machine learning model” generally encompasses instructions, data, and/or a model configured to receive input, and apply one or more of a weight, bias, classification, or analysis on the input to generate an output. The output may include, for example, a classification of the input, an analysis based on the input, a design, process, prediction, or recommendation associated with the input, or any other suitable type of output. A machine learning model is generally trained using training data, e.g., experiential data and/or samples of input data, which are fed into the model in order to establish, tune, or modify one or more aspects of the model, e.g., the weights, biases, criteria for forming classifications or clusters, or the like. Aspects of a machine learning model may operate on an input linearly, in parallel, via a network (e.g., a neural network), or via any suitable configuration.

The execution of the machine learning model may include deployment of one or more machine learning techniques, such as generative learning, linear regression, logistic regression, random forest, gradient boosted machine (GBM), deep learning, graphical neural network (GNN), and/or a deep neural network. Supervised and/or unsupervised training may be employed. For example, supervised learning may include providing training data and labels corresponding to the training data, e.g., as ground truth. Unsupervised approaches may include clustering, classification or the like. K-means clustering or K-Nearest Neighbors may also be used, which may be supervised or unsupervised. Combinations of K-Nearest Neighbors and an unsupervised cluster technique may also be used. Any suitable type of training may be used, e.g., stochastic, gradient boosted, random seeded, recursive, epoch or batch-based, etc.

While several of the examples herein involve certain types of machine learning, it should be understood that techniques according to this disclosure may be adapted to any suitable type of machine learning. It should also be understood that the examples above are illustrative only. The techniques and technologies of this disclosure may be adapted to any suitable activity.

FIG. 11A illustrates an architecture of a computing system 1100, according to example embodiments. The computing system 1100 may be representative of at least a portion of the organization computing system 104. One or more components of the computing system 1100 may be in electrical communication with each other using a system bus 1105. The computing system 1100 may include a processing unit (CPU or processor) 1110 and the system bus 1105 that couples various system components including a system memory 1115, such as a read only memory (ROM) 1120 and a random access memory (RAM) 1125, to the processor 1110. The computing system 1100 may include a cache of high-speed memory connected directly with, in close proximity to, or integrated as part of the processor 1110. The computing system 1100 may copy data from the system memory 1115 and/or a storage device 1130 to a cache 1112 for quick access by the processor 1110. In this way, the cache 1112 may provide a performance boost that avoids the processor 1110 delays while waiting for data. These and other modules may control or be configured to control the processor 1110 to perform various actions. Another system memory may be available for use as well. The system memory 1115 may include multiple different types of memory with different performance characteristics. The processor 1110 may include any general purpose processor and a hardware module or software module, such as service 1 1132, service 2 1134, and service 3 1136 stored in the storage device 1130, configured to control the processor 1110 as well as a special-purpose processor where software instructions are incorporated into the actual processor design. The processor 1110 may essentially be a completely self-contained computing system, containing multiple cores or processors, a bus, memory controller, cache, etc. A multi-core processor may be symmetric or asymmetric.

To enable user interaction with the computing system 1100, an input device 1145 may represent any number of input mechanisms, such as a microphone for speech, a touch-sensitive screen for gesture or graphical input, keyboard, mouse, motion input, speech and so forth. An output device 1135 (e.g., display) may also be one or more of a number of output mechanisms known to those of skill in the art. In some instances, multimodal systems may enable a user to provide multiple types of input to communicate with the computing system 1100. A communications interface 1140 may generally govern and manage the user input and system output. There is no restriction on operating on any particular hardware arrangement and therefore the basic features here may easily be substituted for improved hardware or firmware arrangements as they are developed.

The storage device 1130 may be a non-volatile memory and may be a hard disk or other types of computer readable media which may store data that are accessible by a computer, such as magnetic cassettes, flash memory cards, solid state memory devices, digital versatile disks, cartridges, a random access memories (RAMs) 1125, a read only memory (ROM) 1120, and hybrids thereof.

The storage device 1130 may include services 1132, 1134, and 1136 for controlling the processor 1110. Other hardware or software modules are contemplated. The storage device 1130 may be connected to the system bus 1105. In one aspect, a hardware module that performs a particular function may include the software component stored in a computer-readable medium in connection with the necessary hardware components, such as the processor 1110, the system bus 1105, the output device 1135, and so forth, to carry out the function.

FIG. 11B illustrates a computing system 1150 having a chipset architecture that may represent at least a portion of the organization computing system 104. The computing system 1150 may be an example of computer hardware, software, and firmware that may be used to implement the disclosed technology. The computing system 1150 may include a processor 1155, representative of any number of physically and/or logically distinct resources capable of executing software, firmware, and hardware configured to perform identified computations. The processor 1155 may communicate with a chipset 1160 that may control input to and output from the processor 1155. In this example, the chipset 1160 outputs information to an output 1165, such as a display, and may read and write information to a storage device 1170, which may include magnetic media, and solid-state media, for example. The chipset 1160 may also read data from and write data to a RAM 1175. A bridge 1180 for interfacing with a variety of user interface components 1185 may be provided for interfacing with the chipset 1160. Such the user interface components 1185 may include a keyboard, a microphone, touch detection and processing circuitry, a pointing device, such as a mouse, and so on. In general, inputs to the computing system 1150 may come from any of a variety of sources, machine generated and/or human generated.

The chipset 1160 may also interface with one or more communication interfaces 1190 that may have different physical interfaces. The one or more communication interfaces 1190 may include interfaces for wired and wireless local area networks, for broadband wireless networks, as well as personal area networks. Some applications of the methods for generating, displaying, and using the GUI disclosed herein may include receiving ordered data sets over the physical interface or be generated by the machine itself by the processor 1155 analyzing data stored in the storage device 1170 or the RAM 1175. Further, the machine may receive inputs from a user through user interface components 1185 and execute appropriate functions, such as browsing functions by interpreting these inputs using the processor 1155.

It may be appreciated that example the computing system 1100 and the computing system 1150 may have more than one processor 1110 or be part of a group or cluster of computing devices networked together to provide greater processing capability.

While the foregoing is directed to embodiments described herein, other and further embodiments may be devised without departing from the basic scope thereof. For example, aspects of the present disclosure may be implemented in hardware or software or a combination of hardware and software. One embodiment described herein may be implemented as a program product for use with a computer system. The program(s) of the program product define functions of the embodiments (including the methods described herein) and can be contained on a variety of computer-readable storage media. Illustrative computer-readable storage media include, but are not limited to: (i) non-writable storage media (e.g., read-only memory (ROM) devices within a computer, such as CD-ROM disks readably by a CD-ROM drive, flash memory, ROM chips, or any type of solid-state non-volatile memory) on which information is permanently stored; and (ii) writable storage media (e.g., floppy disks within a diskette drive or hard-disk drive or any type of solid state random-access memory) on which alterable information is stored. Such computer-readable storage media, when carrying computer-readable instructions that direct the functions of the disclosed embodiments, are embodiments of the present disclosure.

It will be appreciated to those skilled in the art that the preceding examples are exemplary and not limiting. It is intended that all permutations, enhancements, equivalents, and improvements thereto are apparent to those skilled in the art upon a reading of the specification and a study of the drawings are included within the true spirit and scope of the present disclosure. It is therefore intended that the following appended claims include all such modifications, permutations, and equivalents as fall within the true spirit and scope of these teachings.